SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We would like to thank Fergus Imrie, Tennison Liu, James Fox, and the anonymous reviewers for insightful comments and discussions on earlier drafts of this paper. AC and AJ gratefully acknowledge funding from AstraZeneca and the Cystic Fibrosis Trust respectively. This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab.
[ACHL19] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[Aka74] Hirotugu Akaike. A new look at the statistical model identification. IEEE transactions on automatic control, 19(6):716-723, 1974.
[AP20] Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias-variance decomposition. Advances in neural information processing systems, 33:11022-11032, 2020.
[ASS20] Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020.
[Bel21] Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-248, 2021.
[BFSO84] Leo Breiman, Jerome Friedman, Charles J. Stone, and R.A. Olshen. Classification and Regression Trees. Chapman and Hall/CRC, 1984.
[BH89] Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
[BHMM19] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.
[BHX20] Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4):1167-1180, 2020.
[BLLT20] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063-30070, 2020.
[BM21] Sebastian Buschjäger and Katharina Morik. There is no double-descent in random forests. arXiv preprint arXiv:2111.04409, 2021.
[BMM18] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pages 541-549. PMLR, 2018.
[BO96] Siegfried Bös and Manfred Opper. Dynamics of training. Advances in Neural Information Processing Systems, 9, 1996.
[Bre01] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[CI09] Jorge Cadima and Ian Jolliffe. On relationships between uncentred and column-centred principal component analysis. *Pakistan Journal of Statistics*, 25(4), 2009.
[CL21] Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. The Journal of Machine Learning Research, 22(1):5721-5750, 2021.
[CMBK21] Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve. Advances in Neural Information Processing Systems, 34:8898-8912, 2021.
[DB22] Yehuda Dar and Richard G Baraniuk. Double double descent: on generalization errors in transfer learning between linear regression tasks. SIAM Journal on Mathematics of Data Science, 4(4):1447-1472, 2022.
[DFO20] Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning. Cambridge University Press, 2020.
[PVG+11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaepau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[Ras18] Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808, 2018.
[RD98] Sarunas Raudys and Robert PW Duin. Expected classification error of the fisher linear classifier with pseudo-inverse covariance matrix. Pattern recognition letters, 19(5-6):385-392, 1998.
[RT19] Saharon Rosset and Ryan J Tibshirani. From fixed-x to random-x regression: Bias-variance decompositions, covariance penalties, and prediction error estimation. Journal of the American Statistical Association, 2019.
[Sch78] Gideon Schwarz. Estimating the dimension of a model. The annals of statistics, pages 461-464, 1978.
[SD96] Marina Skurichina and Robert PW Duin. Stabilizing classifiers for very small sample sizes. In Proceedings of 13th International Conference on Pattern Recognition, volume 2, pages 891-896. IEEE, 1996.
$\left[\mathrm{SGd}^{+}18\right]$  Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665, 2018.
[SGT18] Yitong Sun, Anna Gilbert, and Ambuj Tewari. On the approximation properties of random relu features. arXiv preprint arXiv:1810.04374, 2018.
[SKL92] Jon M Sutter, John H Kalivas, and Patrick M Lang. Which principal components to utilize for principal component regression. Journal of chemometrics, 6(4):217-225, 1992.
[THV22] Ningyuan Teresa, David W Hogg, and Soledad Villar. Dimensionality reduction, regularization, and generalization in overparameterized regressions. SIAM Journal on Mathematics of Data Science, 4(1):126-152, 2022.
[Tuk77] John W Tukey. Exploratory data analysis, volume 2. Reading, MA, 1977.
[Vap95] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1995.
[VCR89] F Vallet, J-G Cailton, and Ph Refregier. Linear and nonlinear extension of the pseudo-inverse solution for learning boolean functions. *Europhysics Letters*, 9(4):315, 1989.
[VL22] Tom Viering and Marco Loog. The shape of learning curves: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[WRR03] Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and principal component analysis. A practical approach to microarray data analysis, pages 91-109, 2003.
[WW75] Grace Wahba and Svante Wold. A completely automatic french curve: fitting spline functions by cross validation. Communications in Statistics-Theory and Methods, 4(1):1-17, 1975.
This Appendix is structured as follows: In Appendix A, we present an additional, broader, literature review. In Appendix B, we present additional background on the linear regression case study, including the proof of Proposition 1 (Appendix B.2) and an empirical study of the effect of excess features on the conditioning of PC feature matrices (Appendix B.3). In Appendix C, we present additional background on smoothers, including an analysis of the impossibility of double descent in prediction error in fixed design settings (Appendix C.2) and derivation of the smoothing matrices associated with linear regression, trees and boosting (Appendix C.3). In Appendix D, we present additional background and other definitions of effective parameter measures. Finally, in Appendix E, we further discuss the experimental setup and close by presenting additional results.
A Additional literature review
Double descent as a phenomenon. Although double descent only gained popular attention since [BHMM19]'s influential paper on the topic, the phenomenon itself had previously been observed:  $\mathrm{[LVM^{+}20]}$  provide a historical note highlighting that [VCR89] may have been the first to demonstrate double descent in min-norm linear regression; [BHMM19] themselves note that double descent-like phase transitions in neural networks had previously been observed in [BO96, ASS20, NMB  $^+$  18,  $\mathrm{SGd^{+}18}$ ]. As discussed above, in addition to linear regressions and neural networks, [BHMM19] also present double descent curves for trees and boosting. Since then, a rich literature on other deep double descent phenomena has emerged, including double descent in the number of training epochs[NKB  $^+$  21], sparsity [HXZQ22], data generation [LDB21] and transfer learning [DB22]. We also note that the occurrence of double descent has been studied for subspace regression methods themselves [DMLB20, THV22], but has not been linked to double descent in min-norm linear regressions as we do here. For a recent review of double descent more broadly see [VL22].
Theoretical understanding of double descent. In addition to the studies of double descent in linear regression discussed in the main text[ASS20, BLLT20, DLM20, BHX20, KSR $^{+}$ 21, HMRT22], we note that further theoretical studies of double descent in neural networks exist (see e.g. [DMB21] for a comprehensive review). These mainly focus on exact expressions of bias and variance terms by taking into account all sources of randomness in model training and data sampling [NMB $^{+}$ 18, AP20, dRBK20, LD21]. Finally, different to our peak-moving experiments in Sec. 3.2 and Appendix E.2 which show that the location of the second descent in all original non-deep experiments of [BHMM19] is determined by a change in the underlying method or training procedure, [CMBK21] show that one can "design your own generalization curve" in linear regression by controlling the data-generating process through the order by which newly (un)informative features are revealed, meaning that one could construct a learning curve with essentially arbitrary number of descents in this way. Similarly, we show in Fig. 6(b) in the main text and Fig. 13 in Appendix E.2 that multiple peaks can be achieved by switching between mechanisms for adding parameters multiple times. Note that [CMBK21] consider only the effect of data-generating mechanisms and arrive at the conclusion that the generalization curves observed in practice must arise due to interactions between typical data-generating processes and inductive biases of algorithms and "highlight that the nature of these interactions is far from understood" [CMBK21, p. 3] - with our paper, we contributed to the understanding of these interactions by studying the role of changes in inductive biases in the case of trees, boosting and linear regression.
Interpolation regime. Entangled in the double descent literature is the interpolation regime or the benign overfitting effect (e.g. [BMM18, MBB18, BLLT20, CL21]). That is, the observation that models with near-zero training error can achieve excellent test set performance. This insight was largely motivated by large deep learning architectures where this is commonplace [MBB18]. While this line of research does provide an important update to historic intuitions which linked perfect train set performance with poor generalization, it is entirely compatible with the ideas presented in this work. In particular, in Figure 20 we demonstrate that within the interpolation regime the number of effective parameters used on test examples and the predictive performance on those test examples can vary greatly. Therefore, benign overfitting may indeed result in excellent performance without contradicting the traditional notion of a U-shaped complexity curve. Additionally, note that while the double descent phenomenon is often presented as a consequence of benign overfitting in the interpolation regime, we show in Appendix E.2 that the location of the second descent is not
inherently linked to the interpolation regime at all in the non-deep experiments of [BHMM19] – but rather a consequence of the point of change between multiple mechanisms of increasing parameter count.
Other related works. Finally, we briefly discuss some other works that might provide further context on topics touched upon in this text. [BH89] study linear neural networks through the lens of principal component analysis. Recent theoretical work suggests that, when optimized using stochastic gradient descent, these networks tend to converge to low norm or rank solutions (e.g.  $\mathrm{[GWB^{+}17}$ , ACHL19]). Aspects of double descent, which were referred to as peaking in older literature, have been associated with the eigenvalues of the sample covariance matrix in a limited setting involving linear classifiers as far back as [RD98]. Much of this work focused on increasing the number of samples for a fixed model (e.g. [Dui95, SD96]). Prior to being rediscovered in the context of neural networks and larger models, more recent work investigated the peaking phenomenon in modern machine learning settings such as semi-supervised learning [KL16].
B More on linear regression
In this section, we first present a brief refresher on SVD, PCA and PCR to provide some further background to the discussion in the main text and the following subsections. We then present a proof of Proposition 1 in Appendix B.2 and finally empirically analyze the effects of excess features on the conditioning of the PCR feature matrix in Appendix B.3.
B.1 Background on SVD, PCA and PCR
In this section we provide a brief refresher on some of the mathematical tools that are fundamental to the insights presented in the main text. We begin with the singular value decomposition (SVD) which acts as a generalization of the standard eigendecomposition for a non-square matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$  (see e.g. [Kal96, DFO20]). The SVD consists of the factorization  $\mathbf{X} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^T$  where the columns of  $\mathbf{U} \in \mathbb{R}^{n \times n}$  and rows of  $\mathbf{V}^T \in \mathbb{R}^{d \times d}$  consist of the left and right singular vectors of  $\mathbf{X}$  respectively and both form orthonormal bases of  $\mathbf{X}$ . Then,  $\boldsymbol{\Sigma} \in \mathbb{R}^{n \times d}$  contains the so-called singular values  $\sigma_1 \geq \ldots \geq \sigma_{\min(n, d)}$  along the diagonal with zeros everywhere else. Analogous to eigenvalues and eigenvectors, the SVD satisfies  $\mathbf{X} \mathbf{V}_i = \sigma_i \mathbf{U}_i$  for all  $i \in 1, \ldots, \min(n, d)$ . We note that for any  $\sigma_i = 0$ , the corresponding  $\mathbf{V}_i$  lies in the nullspace of  $\mathbf{X}$  (while  $\mathbf{U}_i$  lies in the nullspace of  $\mathbf{X}^T$ ). The SVD results in the geometric intuition of factorizing the transformation  $\mathbf{X}$  into three parts consisting of a basis change in  $\mathbb{R}^d$ , followed by a scaling by the singular values that adds or removes dimensions, and then a further basis change in  $\mathbb{R}^n$ . We note that any columns of  $\mathbf{U}$  and  $\mathbf{V}$  corresponding to either null singular values (i.e.  $\sigma_i = 0$ ) or the  $\max(n, d) - \min(n, d)$  non-square dimensions of  $\mathbf{X}$  are redundant in the SVD. Therefore we can equivalently express the SVD in its so-called compact form with these vectors and their corresponding rows/columns in  $\boldsymbol{\Sigma}$  removed.
The SVD is intimately linked to dimensionality reduction through principal component analysis (PCA) [Pea01, FM23, WRR03]. Recall that PCA is a technique that finds a basis of a centered matrix  $\mathbf{X}$  by greedily selecting each successive basis dimension to maximize the variance within the data. Typically, this is presented as selecting each basis vector  $\mathbf{b}_i = \arg \max_{||\mathbf{b}|| = 1}||\hat{\mathbf{X}}_i\mathbf{b}||^2$  where  $\hat{\mathbf{X}}_i$  denotes the original (centered) data matrix  $\mathbf{X}$  with the first  $i - 1$  directions of variation removed (i.e.  $\hat{\mathbf{X}}_i = \mathbf{X} - \sum_{k = 1}^{i - 1}\mathbf{b}_k\mathbf{b}_k^T\mathbf{X}$ ). By expanding  $\| \hat{\mathbf{X}}_i\mathbf{b}\|^2 = \mathbf{b}^T\hat{\mathbf{X}}_i^T\hat{\mathbf{X}}\mathbf{b}^T$  we can notice that the basis vectors obtained are also the eigenvectors of  $\mathbf{X}^T\mathbf{X}$  which is proportional to the sample covariance matrix. The dimensionality reduction then occurs by selecting a subset of these new coordinates, ordered by highest variance, to represent a low-rank approximation of  $\mathbf{X}$ . The relationship to the SVD is illuminated by noticing that, for a centered  $\mathbf{X}$ , we can use the singular value decomposition such that
$$
\begin{array}{l} \mathbf {X} ^ {T} \mathbf {X} = (\mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {T}) ^ {T} (\mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {T}) \\ = \mathbf {V} \boldsymbol {\Sigma} ^ {T} \boldsymbol {\Sigma} \mathbf {V} ^ {T} \\ = \mathbf {V} \boldsymbol {\Sigma} ^ {2} \mathbf {V} ^ {T}. \\ \end{array}
$$
This reveals that  $\mathbf{V}$ , the orthonormal basis containing right singular vectors of  $\mathbf{X}$  and obtained through the SVD, produces exactly the eigenvectors of  $\mathbf{X}^T\mathbf{X}$  required for PCA. Then the PCA operation  $PCA_{k}(\mathbf{X}): \mathbb{R}^{n\times d} \to \mathbb{R}^{n\times k}$  (where  $k \leq d$  is the user-defined number of principal components
retained) may be obtained as  $PCA_{k}(\mathbf{X}) = \mathbf{X}\mathbf{V}_{k} = \mathbf{U}_{k}\pmb{\Sigma}_{k}$  where we overload notation such that the subscript of each matrix denotes keeping only the vectors corresponding to those first  $k$  principal components (ordered according to the magnitude of their singular values).
A natural idea that emerges from the description of PCA is to apply it in the supervised setting. Specifically, we might wish to first perform the unsupervised dimensionality reduction step of PCA followed by least squares linear regression on the transformed data. This is exactly the approach taken in principal component regression (PCR) [Ken57, Hot57]. Mathematically this can be expressed as solving  $\hat{\beta} = \min_{\beta} (\mathbf{y} - \mathbf{XV}_k \beta)^2$  for coefficients  $\beta \in \mathbb{R}^k$  and targets  $\mathbf{y} \in \mathbb{R}^n$ . Then the least squares solution takes the form  $\hat{\beta} = (\mathbf{V}_k^T \mathbf{X}^T \mathbf{XV}_k)^{-1} \mathbf{V}_k^T \mathbf{X}^T \mathbf{y}$ . The assumption of PCR is that the high variance components are most important for modeling the targets  $\mathbf{y}$ . While this is not necessarily always the case[7] [Jol82], PCR has historically been an important method in the statistician's toolbox (e.g. [HTF09]). In the main text we make use of the connection between PCR and min-norm least squares on underdetermined problems (i.e. a fat data matrix). In particular, we show that applying min-norm least squares in this setting is equivalent to applying PCR with all principal components retained. We provide a proof for this proposition in Appendix B.2. Of course, related mathematical connections between min-norm solutions and the singular value decomposition have been made in previous work (see e.g. [GVL13, Ch. 5.7]). The regularization effect of PCR also has a known connection to ridge regression where both methods penalize the coefficients of the principal components [HTF09, Ch. 3.5.1]. The former discards components corresponding to the smallest singular values while the latter shrinks coefficients in proportion to the size of the singular values[8].
B.2 Proof of Proposition 1
Proposition 1. [Min-norm least squares as dimensionality reduction.] For a full rank matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$  with  $n < d$  and a vector of targets  $\mathbf{y} \in \mathbb{R}^n$ , the min-norm least squares solution  $\hat{\beta}^{MN} = \{\min_{\boldsymbol{\beta}} ||\boldsymbol{\beta}||_2^2 : \mathbf{X}\boldsymbol{\beta} = \mathbf{y}\}$  and the least squares solution  $\hat{\beta}^{SVD} = \{\beta : \mathbf{B}\boldsymbol{\beta} = \mathbf{y}\}$  using the matrix of basis vectors  $\mathbf{B} \in \mathbb{R}^{n \times n}$ , constructed using the first  $n$  right singular vectors of  $X$ , are equivalent; i.e.  $\mathbf{x}^T\hat{\beta}^{MN} = \mathbf{b}^T\hat{\beta}^{SVD}$  for all  $\mathbf{x} \in \mathbb{R}^d$  and corresponding basis representation  $\mathbf{b} \equiv \mathbf{b}(\mathbf{x})$ .
Proof. We will use the compact singular value decomposition  $\mathbf{X} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T$  (i.e. with the  $d - n$  redundant column vectors dropped from the full  $\boldsymbol{\Sigma}$  and  $\mathbf{V}$ ). Specifically, the columns of  $\mathbf{U} \in \mathbb{R}^{n \times n}$  and  $\mathbf{V}^T \in \mathbb{R}^{n \times d}$  form orthonormal bases and  $\boldsymbol{\Sigma} \in \mathbb{R}^{n \times n}$  is a diagonal matrix with the singular values along the diagonal. The transformation of  $\mathbf{X}$  to  $\mathbf{B}$  is then given by  $\mathbf{X}\mathbf{V} = \mathbf{U}\boldsymbol{\Sigma} = \mathbf{B}$ . It is sufficient to show that for some test vector  $\mathbf{x} \in \mathbb{R}^d$  in the original data space and its projection to the reduced basis space  $\mathbf{b} = \mathbf{V}^T\mathbf{x} \in \mathbb{R}^n$ , then the predictions  $\hat{y}$  of both models are equal such that  $\mathbf{x}^T\hat{\beta}^{\mathrm{MN}} = \mathbf{b}^T\hat{\beta}^{\mathrm{SVD}}$ .
$$
\begin{array}{l} \hat {y} ^ {\mathrm {M N}} = \mathbf {x} ^ {T} \hat {\boldsymbol {\beta}} ^ {\mathrm {M N}} = \mathbf {x} ^ {T} \mathbf {X} ^ {T} (\mathbf {X} \mathbf {X} ^ {T}) ^ {- 1} \mathbf {y} \\ = \mathbf {x} ^ {T} (\mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {T}) ^ {T} ((\mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {T}) (\mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {T}) ^ {T}) ^ {- 1} \mathbf {y} \\ = \mathbf {x} ^ {T} \mathbf {V} \boldsymbol {\Sigma} \mathbf {U} ^ {T} \left(\mathbf {U} \boldsymbol {\Sigma} \boldsymbol {\Sigma} \mathbf {U} ^ {T}\right) ^ {- 1} \mathbf {y} \\ = \mathbf {x} ^ {T} \mathbf {V} \mathbf {B} ^ {T} (\mathbf {B} \mathbf {B} ^ {T}) ^ {- 1} \mathbf {y} \\ = \mathbf {b} ^ {T} \mathbf {B} ^ {T} (\mathbf {B} \mathbf {B} ^ {T}) ^ {- 1} \mathbf {y} \\ = \mathbf {b} ^ {T} \hat {\boldsymbol {\beta}} ^ {\mathrm {S V D}} \\ = \hat {y} ^ {\text {S V D}} \\ \end{array}
$$
Indeed, an alternative approach is to use standard variable selection techniques on the constructed dimensions of the transformed data (see e.g. [SKL92]).
This connection explains why the explicit regularization with the ridge penalty in [HMRT22] and the implicit regularization of min-norm solutions have a consistent shrinking effect on test loss.
Figure 10: The effect of  $P^{\phi}$  on the conditioning of the PCR feature matrix. Singular values  $\sigma_{k}$  (left) and  $\kappa(\mathbf{B}_{k})$  (right) derived from the random Fourier feature matrix  $\Phi$ , as a function of  $k$  and for different raw number of features  $P^{\phi}$ .
B.3 Understanding the impact of  $P^{ex}$ : Examining its impact on singular values of the PCR feature matrix
In this section, we empirically investigate how increasing  $P^{ex}$  for a fixed number of  $P^{PC}$  results in a gradual improvement in the quality of a basis of fixed dimension, revisiting the experimental setting of Sec. 3.2 in the main text.
In particular, we consider how excess features affect the conditioning of the feature matrix  $\mathbf{B}_k = PCA_k(\mathbf{X}) = \mathbf{X}\mathbf{V}_k = \mathbf{U}_k\Sigma_k$ , used in principal components regression, for different values of  $k \equiv P^{PC}$ . As noted in [PKB19], the condition number  $\kappa(\mathbf{A})$  of a matrix  $\mathbf{A}$  measures how much errors in  $\mathbf{y}$  impact solutions when solving a system of equations  $\mathbf{A}\beta = \mathbf{y}$ , and is given by  $\kappa(\mathbf{A}) = \frac{\sigma_{\max}(\mathbf{A})}{\sigma_{\min}(\mathbf{A})}$ , the ratio of maximal to minimal singular value of  $\mathbf{A}$ . Because in principal components regression  $\mathbf{B}_k$  is constructed from the top  $k \in \{1, \dots, n\}$  principal components, it is easy to see that  $\sigma_{\max}(\mathbf{B}_k) = \sigma_1$  and  $\sigma_{\min}(\mathbf{B}_k) = \sigma_k$  - the largest and k-th largest singular value of  $\mathbf{X}$  respectively (as defined in Appendix B.1) - so that in principal components regression we generally have  $\kappa(\mathbf{B}_k) = \frac{\sigma_1}{\sigma_k}$ .