No Double Descent in PCA: Training and Pre-Training in High Dimensions
Authors: Daniel Gedon
Antonio H. Ribeiro
Thomas B. Schön
Venue: Submitted to ICLR 2023 (2023)
Keywords: PCA
generalization theory
overparameterization
pre-training theory
encoder-decoder models
double descent
linear regression

Abstract: With the recent body of work on overparameterized models the gap between theory and practice in contemporary machine learning is shrinking. While many of the present state-of-the-art models have an encoder-decoder architecture, there is little theoretical work for this model structure. To improve our understanding in this direction, we consider linear encoder-decoder models, specifically PCA with linear regression on data from a low-dimensional manifold. We present an analysis for fundamental guarantees of the risk and asymptotic results for isotropic data when the model is trained in a supervised manner. The results are also verified in simulations. Furthermore, we extend our analysis to the popular setting where parts of the model are pre-trained in an unsupervised manner by pre-training the PCA encoder with subsequent supervised training of the linear regression. We show that the overall risk depends on the estimates of the eigenvectors in the encoder and present a sample complexity requirement through a concentration bound. The results highlight that using more pre-training data decreases the overall risk only if it improves the eigenvector estimates. Therefore, we stress that the eigenvalue distribution determines whether more pre-training data is useful or not.

Under review as a conference paper at ICLR 2023
No DOUBLE DESCENT IN PCA: TRAINING AND TRAINIG IN HIGH DIMENSIONS
Anonymous authors Paper under double-blind review
ABSTRACT
With the recent body of work on overparameterized models the gap between the- ory and practice in contemporary machine learning is shrinking. While many of the present state-of-the-art models have an encoder-decoder architecture, there is little theoretical work for this model structure. To improve our understanding in this direction, we consider linear encoder-decoder models, specifically PCA with linear regression on data from low-dimensional manifold. We present analy- sis for fundamental guarantees of the risk and asymptotic results for isotropic data when the model is trained in supervised manner. The results are also verified in simulations and compared with experiments from real-world genetics data. Fur- thermore, extend analysis to the popular setting where parts of the model are pre-trained in an unsupervised manner by pre-training the PCA encoder with subsequent supervised training of the linear regression. We show that the overall risk depends on the estimates of the eigenvectors in the encoder and present a sam- ple complexity requirement through a concentration bound. The results highlight that using more pre- pre-training data decreases the overall risk only if it improves the eigenvector estimates. Therefore, we stress that the eigenvalue distribution determines whether more pre-training data is useful or not.
INTRODUCTION
Many recent success stories of deep learning employ an encoder-decoder structure, where parts of the model are pre-trained in an unsupervised or self-supervised way. Examples can be found in computer vision (Caron et al., 2020; Chen et al., 2020; Goyal et al., 2021), natural language processing (Vaswani et al., 2017; Devlin et al., 2019; Raffel et al., 2020) or multi-modal models (Ramesh et al., 2021; Alayrac et al., 2022). Understanding the properties of this model structure might shed light on how to reliably build large-scale models.
We add to the theoretical understanding of encoder-decoder based models by studying model con- sisting of PCA and a linear regression head. We analyse this model for the supervised case and for the case where unsupervised pre-training is followed by supervised linear regression. Our model can be viewed as simplified, linear example of large pre-trained deep neural network in combination with linear probing (Devlin et al., 2019; Schneider et al., 2019). While linear models do not reveal the whole picture, they are studied as a tractable, first step towards deeper understanding. Indeed, re- search on linear models has previously provided important insights into relevant mechanisms (Saxe et al., 2014; Lampinen & Ganguli, 2019; Arora et al., 2019; Gidel et al., 2019; Pesme et al.. 2021).
We utilize data generated from a low-dimensional manifold, similar to Goldt al. (2020). This is motivated by the manifold hypothesis (Fefferman et al., 2016) which states that real-world high- dimensional data often have an underlying low-dimensional representation. Our PCA encoder can exploit this data structure effectively. While we keep the low-dimensional data structure fixed, we vary the number features w.r.t. the number of training data points which allows us to analyse what is often referred as overparameterization i.e. data features model parameters than training samples (Belkin et al., 2019). We do not consider parameter count, since for our model the number of parameters, i.c. the linear regressors, stay fixed due to the PCA encoding. Instead analyse high- dimensional settings. Studying overparameterization gives theoretical justification of the success of modern large-scale neural networks such Szegedy et al. (2016); Dosovitskiy et al. (2021).

---

Under review as a conference paper at ICLR 2023
Theoretical grounding is exceeded by the empirical success of machine learning and specifically deep learning methods through new model structures (Krizhevsky et al., 2017; He et al.. 2016; Vaswani et al., 2017) or training methods (Erhan et al., 2010; loffe & Szegedy, 2015; Ba et al., 2016). In recent years our theoretical understanding grew e.g. through the analysis of implicit regu- larization (Gunasekar al., 2017; Chizat & Bach, 2020; Smith et al., 2021). But also experimental work contributed to our understanding (Keskar et al., 2017: Zhang et al., 2017). The goal of this paper is extend our understanding of the successful encoder-decoder model structure through theoretical analysis of PCA-regression and by extensive numerical simulations. We generalize re- sults for linear regression and combine classical analysis of overparamterization with pre-training of model components. Our contributions can be summarized as:
In the supervised case, provide theoretical guarantees for the risk and parameter norm of the PCA-regression model. For isotropic data we extend the results to the limit where the number of data points and features m tend to infinity such that m/n + 7-
Through simulations, we confirm our theory for isotoopic data and explain the model be- havior on data from a low-dimensional manifold. Using geneties data, we validate our findings in a high- eigh-dimensiona real-world example.
We extend our analysis to the popular scenario of unsupervised pre-training of the en- coder and show that the correct estimation of feature covariance eigenvectors is crucial for low risk. These estimates are highly dependent on the data structure through the eigenvalue decay rate. The results provide link to known asymptotic results by Xu & Hsu (2019). We challenge the common wisdom that more pre-training data improves the overall risk and show that this is the case only if it improves the estimate of the cigenvectors in the encoder which is c.g. the case in data with rapidly decaying cigenvalues.
RELATED WORK
The study of overparameterized models offers natural route to gain the- oretical understanding when it comes to the successes of large models with good generalization properties (Neyshabur et al., 2015; Zhang et al., 2017). The double descent was discovered and analysed in early works (Krogh & Hertz, 1991; Geman et al., 1992; Opper, 1995) but the framing as 'double descent' (Belkin et al., 2019) boosted research in this direction even if generalization of large models was already studied before (Bartlett & Mendelson, 2002; Dziugaite & Roy, 2017; Belkin et al., 2018; Advani et al., 2020). We add to the understanding of machine learning models by analysing the neglected class of encoder-decoder models with the PCA reg-regressoon model.
Overparameetrizzttion
Analysis of pre-training The introduction of pre-training of neural networks for deep learning. Empirical work (Erhan et al., 2010; Raghu et al., 2019) but also theoretical work such for sample complexity (Tripuraneni al., 2020; Du et al., 2021) the out-of-distribution risk (Kumar et al., 2022) tried to understand the mechanisms. For unsupervised pre-training, con- trastive methods were studied (Wang & Isola, 2020; Von Kügelgen et al., 2021). Encoder-decoder based autoencoders are analysed for training dynamics (Nguyen et al., 2019; 2021) or overparame- terization (Radhakrishnan et al., 2019; 2020; Buhai et al., 2020; Zhang et al., 2020). In contrast, we study pre-trained PCA encoders and relate the risk to the covariance estimation of the encoder.
paradigm shift
Latent variable data generator We generate data via a linear latent variable data generator based on a low-dimensional manifold. The hidden manifold model (Goldt et al., 2020) and random feature model (Rahimi & Recht, 2007) present similar but nonlinear models. Goldt et al. (2022); Hu & Lu (2022) showed that these nonlinear models are asymptotically equivalent to linear Gaussian models under assumptions such as that the latent dimension d + oc. In contrast, we keep this dimension fixed. Asymptotic generalization results for this data generator are presented in Gerace et al. (2020); Mei & Montanari (2022). Different to our work where we exploit the low-dimensional structure with the PCA-regression model, they do not use this information by using Ridge or logistic regression.
PCA reggresssnonnn model Using PCA (Jolliffe, 1982) is common discussions focus on the choice of principle components (Breiman & Freedman, 1983) its use for dighndimensional data (Lee et al., 2012). PCA-regression is investigated in Xu & Hsu (2019) for general but fully known covari- ances in the asymptotic regime. Wu & Xu (2020) extend by showing that the misalignment of true
2

---

Under review as a conference paper at ICLR 2023
and estimated eigenvectors affect the risk. Huang al. (2022) use misalignment bounds (Loukas, 2017) to remove the known covariance assumption and obtain con-ssyptooiic risk bounds. Our work fills the gaps by providing asymptotic results for isotropic data. We generalize Loukas (2017) to obtain sample complexity for the covariance estimation in the PCA which is the missing piece quantify when the results from Xu & Hsu (2019) be used in practice with pre-training. turns out that the data covariance structure is crucial as Wainwright (2019) points out.
PROBLEM FORMULATION
Data generator We generate data ( ) 2=1 according latent variable data generator
2; = Dz; ei.
(1)
= 0 E
(2)
by mapping, the latent variable z; E Rd with D E mxd into the observed features 2; E IR and
with 0 E Fd into the observed outputs y, R. We create D randomly such that || 1 de² with e as correction factor to control the signal-to-noise ratio (SNR), defined in (27). Similarly, to control the outcome-noise-rese and output noise E ~ NO, of) are added. The latent variables are generated such that the singular values of the features have an exponential decay controlled by the decay rate a 2 0 according to 2, (0, 1a with X exp(-ia).
we create 0 such that E [|| =1ī r Feature noise e, (O, I,,)
Our theoretical results do not specifically require an exponential decay of the eigenvalues or spe- cific rate. However, fast decaying eigenvalues occur in many real-world examples, see Appendix B. We distinguish between two data generators:
1. Isotropic data. This is special case of (I), (2) with d = m. D = I. 0 and e 0 to generate isotropic features. It allows us to rewrite the data generator as
yi 0 at; + with 0; N² I.).
(4)
2. Latent variable data. We distinguish between 1) a 0 leading to isotropic but low- dimensional signal and 2) a >> which has dominant, but rapidly decaying eigenvalues of the feature covariance matrix. The latter data generator motivated since many real-world data sets have a low-dimensional signal manifold with rapidly decaying eigenvalues.
Note that while our latent variable data generator is similar to the latent space model from Hastie al. (2022), we use our PCA-regression model instead of direct regression from features to outputs. A graphical model of our data generator is provided in Figure and details are in Appendix C.
PCA
lin. reg.
Figure 1: Problem formulation. Left: Data generator. Right: PCA and linear regression model.
Model We use a linear model which resembles an encoder-decoder based architecture. The input data æ, is encoded into a lower d-dimensional space 2; via the use of PCA, where d is chosen during model selection. When comes the decoder we employ a linear regression model with parameters E IRd The resulting model is visualized in Figure and we can formulate it as
2 = VT xi.
(5)
0: 0 24
(6)
Collecting the features as rows the data matrix X [
x. we compute the principal
components V E m as the d first right singular vectors of the rank å reducing SVD X s ÚΣVT. Note that the estimated latent dimension d can be different from the true latent dimension d of the latent variable data generator. We refer to this construction as the PCA-regression model. In our numerical results we compare with a model which directly regresses the outcomes from the features, referred the direct regression model.
3

---

Under review as a conference paper at ICLR 2023
4 ANALYZING THE SUPERVISED CASE
Training the complete PCA-regression model in a supervised way represents a situation commonly encountered in high-dimensional real-world applications. Examples of using this model are in ex- ploratory statistical research (Massy, 1965), econometrics (Geweke, 1996), genetics (Wang & Ab- bott, 2008), robotics (Vijayakumar & Schaal, 2000) and many more.
4.1 THEORETICAL ANALYSIS
For our analysis, we are interested in closed form solutions for the risk and parameter norm in order obtain fundamental guarantees for the PCA-regression model. We decompose our solution into bias and variance terms similar to classic decompositions and interpret the results.
Bias-variance decomposition We stack all outputs in the vector y R" and all estimated latent variables as rows in the matrix 2 E Rndd The solution to the unregularized linear regression yields
0 i y xVx X y;
where (.)+ denotes the Moore-Penrose pseudoinverse. We can rewrite our data generator directly from features outputs y X + with 3 T 0 D+ I and 6 oN),, where o² = =o + || Following Appendix D.2 the solution becomes
-= (XB=)
(8)
Let the feature sample covariance be Ö = - and the true covariance be C. Define the orthogonal projectors E- VV and II = IIm $ where Tb is the projection onto the column space of the d first right singular vectors of X. Then, the risk of the PCA-regression model R(O) E(
Lemma 1.
[Yo
{xx))? and the parameter norm |0 0 1 are given by
0² Tr Civ
[(()] 3 = IICII + Eg
(9)
E2
3 $3 TT(V [|]3]
(10)
n
The proofs are Appendices D.3, D.5. both equations, the variance (second) term is controlled by the estimated singular vectors V. which project the covariances C, to a a d-dimensional subspace and therefore contain less noise. Hence, we expect the variance term to decrease constantly for larger y and that the PCA-regression model therefore avoids the "interpolation peak" at = 1 which linear regression has. The results generalize Lemma in Hastie al. (2022) for the risk of direct regression models since we obtain the same form when choosing d m, i.c. no dimensionality reduction.
Asymptotics for isotropic features Using results from random matrix theory, and Lemma I we derive asymptotics for the risk and parameter norm in the case of isotropic features I,..
Theorem 1. Assume isotropic features Then, as m, + co, such that - Y, the expected risk and parameter norm satisfy almost surely
Imm which implies d m and choose constant å.
ATB --immid,, mmdmmmmmmmm for <<1
E2
[((0]
oe m n
m
(dF,(s)+0g
(1I)
n
18T min(d, for <>1
Ee [1011
[ (s)
B B3 mminnd, m)/m for 7 <
(12)
BT 3 min(d, n)/m for 7 7>1'
with F., the Marčenko-Pant Pastur law (Marčenko Pastur, 1967) and the value R. that satisfies d = -5x dF... In both equations, the first term represents the variance and the last one the bias. m
The proofs are Appendices D.4, D.6. Again, we obtain the same risk when choosing d m as for direct regression models isotropic data, see Theorem in Hastie et al. (2022). Contrary to direct regression, the PCA-model will always have bias term since d < m, n in general.
4

---

Under review as a conference paper at ICLR 2023
4.2 NUMERICAL RESULTS
In this section we give numerical results for the different data generators and compare these results with those from the analysis above. We compare PCA-regression model with 1) the learnt direct regression model and 2) a model that predicts always zero which we denote as null risk.
Isotropic features We generate n 400 data points for training and testing according to our isotropic data generator (4), implying d = with o² and r 1. Each sample has - features where we vary 7 E [0.3, 20], i.e. from low-dimensional (7 < 1) to high-dimensional (? 1) features. We compute risk R(O) and parameter norm ||0| as in the definition of Lemma I and average over 200 realizations. The results are compared with analytical solutions from Theorem
2.5
dir
Parameter norm
2
pes, d=25
pea, d-50
d=100
Risk
2
analytical
1.5
0
0.5
2
10
20
0.5
5
10
20
7 = m
7 m
Figure 2: Supervised results on isotropic data: analysis vs simulation. Solid lines: analytical so- lutions (Theorem 1); xx: avg. simulation results; 'o": null risk. Left: Risk. Right: Parameter norm.
Figure 2 depicts the results for different values of d; we can make several observations: 1) The nu- merical results, i.e. the x marks, and the analytical solutions, i.e. the solid lines, align perfectly and therefore support our theoretical analysis. The expected decrease of the variance term and decom- position according to Theorem 1; 2) For sufficiently large d the results of the PCA model match the direct regression results in the limit of small and large y. For isotropic data, every singular direction equally important and the PCA requires sufficiently many components, i.e. large d to achieve reasonable results; 3) The ICA-reeressson model does not suffer from the singularity at = 1 as we predicted from Lemma The PCA alleviates the bad conditioning of the matrix X "X which has to be inverted for the least squares solution. Below we will see that ridge regression has similar effect; and, 4) The parameter norm is constantly decreasing for larger % We observe this for all models, which implies that we obtain smooth solutions which are beneficial to avoid overfitting.
nonzero bias term for all y can be observed in Figure where we show the bias-varias
Latent variable data We use the latent variable data generator with = 20, rg 11 20
0.
feature SNR Pr and @ in (33) generate 400 training and testing data points and average over 200 realizations. The results for the risks are depicted in Figure for eigenvalue decay of a = 0 (left) and for 0.25 (right). Corresponding plots for the parameter norm are in Figure 11.
We observe for regression results for small and large 7/ The plots for d 20 and å 40 overlay since both are larger than d and capture all information. However, for misspecified models with d < the solution obtained for the PCA-regression is suboptimal. Following Lemma 1, by choosing à < d we remove important eigendirections and therefore observe increased risk. Similar conclusions can be drawn for the results for data with a (right plot) but with less penalty on the risk for suboptimal d.
(left plot) if å 2 d, then the PCA-regression model approaches the direct
Real-world example: Geneties To visualize the PCA-regression model under high-dimensional inputs for real-world data example, we use the Diverse MAGIC wheat data set (Scott et al., 2021) from the National Institute for Applied Botany. The data set contains the genome sequence of 504 inbred wheat lines and multiple phenotypes. We split the data 252 training samples and equally many test samples. There are 1.1 million nucleotides in the genotype sequences which
5

---

Under review as a conference paper at ICLR 2023
Risk,
Risk, 0.25
10
2
0m
30
00
00
30
NN
M Mo
10
10
10
dic rng.
peu. da5
paa, d--15
10
peu, d=20
10-2 10
pea,
0.5
2
5
10
20
0.5
5
10
20
==
m/n
m/
Figure 3: Supervised risk on latent variable data: simulation. Left: Risk of models for data
generated with feature covariance eigenvalue decay of a = 0. Right: Results with 0.25.
binary encoded difference reference sequence. We subsample the genotypes uniformly for varying number of features m. As outcome we use one of the real-values phenotypes.
Figure 4 shows the median results 100
dir. reg. pen, des
alizations for different latent dimension d. We observe qualitative resemblance to the results for the latent variable model in Figure 1) The PCA-regression risk decreases monotoni- cally with increasing " and 2) higher values of d reach the lowest overall risk. Different is that the PCA-regression does not reach the same level as the direct regression for larger 7. However, this reasonable since the eigenvalue distribu- tion in the genetics example is heavy tailed (see Figure 16) which implies that the true latent di- mension would be much larger. Further, 2) the relationship between genotypes and phenotypes may not be linear in nature.
1.5
pea, d-40
pen,
Risk
K
0.5
1
2
5
10
20
7
m//n
Figure 4: Supervised risk for real-world exam- ple. Diverse MAGIC wheat genetics data set.
PRE-TRAINING THE PCA ENCODER
So far, we analysed the case when the complete model is trained supervised. Now we extend to the popular case of pre-training parts of the model in an unsupervised way. In this context we can view our model as simple. linear version of large pre-trained neural networks with linear probing. Our analysis therefore yields insights to their understanding. The pre-training extension requires generalization of our theory because we deal with different data sets of varying size.
5.1 GENERALISATION OF PROBLEM FORMULATION
First, we pre-train PCA on a so called pre-training data set { without output values ye. can therefore only be used for unsupervised pre-training. Second, we train only the linear regression head on the PCA features with the training data set {æ4, y1 / Note that the number of samples np in the pre-training data set differs from the number of samples in the training data set.
The
Data generator In this section, focus the latent variable data generator. We change our feature generation from (1) to simplify the theoretical analysis. We orthogonalise the signal z (gen- erated by (3)) and the noise e by introducing D such that D D_ 0. We use the following feature generator for both, the training and pre-training data set features
3;; Dz, Die,
(13)

---

Under review as a conference paper at ICLR 2023
Model The model is the same as in the supervised case. Since the PCA is performed on the pre- training data set, rename the first d estimated eigenvectors of the feature covariance matrix from the pre-training data H. We do so distinguish from the eigenvectors V estimated using the training data set as in the supervised case. Hence, in the pre-training case we obtain
2: = H xz
(14)
5.2 THEORETICAL ANALYSIS
As in section 4.1 we want establish a connection to the complete training risk as fundamental model guarantee. Since we extend the setting to pre-train the encoder on a different data set, we have deal with sample complexities for the estimation of cigenvectors in the PCA.
With the orthogonal feature generator (13), we recover the true latent variables from features
2 D* (z: ei),) D* x;;
(15)
Comparing it with the projection from the model in (14). we notice that the estimated latent space depends on how well H estimates D Hence, the risk analysis problem the case with pre- training turns into a sample complexity problem of the eigenvectors. Note that D+ -+=Di DT /c² with correction factor e for SNR control, see Appendix C.
Estimation of eigenvectors The sample complexity of eigenvectors is thoroughly studied by Loukas (2017). Here, we review some of their results and adapt them to our setting. The PCA loss of encoding into the (estimated) latent space is given by
L(D) E [|
I
a
1
Σ
812
(16)
i=d+1
m
m
((H) ||
H
x
12
8,
Σ j=1
h
(17)
Here, 8; the ith eigenvalue and h; is the ith eigenvector of the true feature covariance matrix. The difference of the PCA losses, quantifies how well a sample æ is projected with the estimated eigenvectors H into the latent space compared to projection with the true eigenvectors (D+)T.
Lemma 2. Define the loss of projecting sample with D or H as in (16), (17). Then, we can write the loss difference as L(H)- L(D) E
|| ||]]|]] and formulate it as
min(d,d)
C(H ( L(D)
Σ h1) .j)
8
(18)
=0 for d2d
=0 for ded
The result indicates that if we have perfect encoding (d = d), then only the first term remains. also all eigenvalues equal, then there is no loss difference and the estimation of the direction of eigenvectors H does not matter since we are dealing with the isotropic case. However, for more natural scenarios such as exponentially decaying eigenvalues, the eigenvalue difference is nonzero and correct estimation of the cigenvectors H is crucial for a small loss difference. If we are dealing with imperfect encoding, there is either an additional term due to misalignment of the estimated eigenvalues (d < d) or due to encoding of noise (d d). The proof is in Appendix F.I.
Theorem 2. Define a real t >> using Corollary 4.1 from Loukas (2017), and with kē sy(5; + TT(C)) from Corollary 4.3 in Loukas (2017), then we obtain the concentration inequality
(c((t) L(D) > t) =
4
min(d.d)
22
d
k
d
k2 $j (s,
(19)
<
t nip
8
(3:
3j 2
8,) )2
1=1
1=d 1=1
7

---

Under review as a conference paper at ICLR 2023
This theorem states, that in addition to the implications of Lemma 2, there two main scenarios where we obtain a lower right hand side and therefore tighter bound. 1) When the feature covariance matrix has rapidly decaying cigenvalues, ie. large 8; syl 2 0, since j >> i or 2) when have access to more pre-training samples np- The proof is Appendix F.2.
Connection the risk We define the risk between features and outcomes y in the same way as for the supervised case, see Lemma I. The goal is to obtain asymptotic results for the risk for different eigenvalue decays including our latent variable data generator. Xu & Hsu (2019) presents asymptotic results for polynomial and more general eigenvalue decays in the PCA-regression model. However, their analysis relies on the assumption that the eigenvectors fully known, i.e. H D* which is an unrealistic scenario.
A solution resolve this condition is to estimate the eigenvectors H from unlabeled data {x;) }- But it unclear under what conditions the estimate is sufficiently good. The eigenvector estimation is precisely what is done during the pre-training step. Theorem 2 provides a sample complexity for the eigenvector estimation quality and therefore provides the missing condition when the results from Xu & Hsu (2019) hold in practice. Choosing sufficiently small, we can quantify how many samples are necessary for the estimated eigenvectors to be close to the true ones. Hence, we provide conditions when the asymptotic risk results from Xu & Hsu (2019) can be used in practice.
However, if we do not have access to sufficiently many pre-training data samples, then we know that our estimated eigenvectors H are misaligned. These eigenvectors will project the features into misaligned latent space E. Finally, we perform linear regression from this misaligned space. Quan- tifying the additional error on the overall risk for misaligned linear regression is open problem.
5.3 NUMERICAL RESULTS
We present numerical results when using pre-training. We denote the relation of pre-training samples training samples as μ "² with μ 2 as we could use the training data set also for pre-training
We choose d = 20, r 1, of 0, Pa 1 and focus on = as the effects of misspecified models equal as without pre-training and elaborated Section 4.2. Experiments to confirm this behavior for pre-training are in Appendix G. We generate n 20200 training samples and ?p nu pre-training samples by varying μ E [1, 10] and average the computed risk over 100 realizations.
Risk with pre-training, a = 0
Risk with pre-training, a = 0.25
10
10
0.2
0.2
0.1
n
0.1
0.05
0.02
0.02
10
10
0.5
2
10
0.5
1
2
10
= m/
7 = m/n
Pre-training risk on latent variable data: simulation. On the x-axis increase the number of features m and therefore the degree of overparameterizztion 7y. On the y-axis we increase amount of pre-training np compared training data n. Left: Risk for latent variable data generated with feature covariance eigenvalue decay of a =. Right: Same setup but for =.25.
Figure 5:
In Figure 5 the risk for data with two different eigenvalue decay rates are depicted. We make three main observations: 1) Horizontally for const. in both plots the risk decreases similar to the supervised case Figure and therefore follows Lemma 1. 2) Vertically for y= =onst. in the right plot (a 0.25) the risk decreases expected from Theorem when using more pre-training samples. The effect most significant for large overparameterization 7. 3) Vertically for y const., in the left plot (a 0), notice that more pre-training data does not decrease the risk. Since
8

---

Under review as a conference paper at ICLR 2023
have perfect encoding d = d and two blocks of constant eigenvalues, the eigenvector estimation is by Lemma 2 almost perfect and barely improves with more pre-training, see Theorem 2. Therefore, using more pre-training data does also not improve the overall risk. The observation supports finding that more pre-training data decreases the risk only if it improves the eigenvector estimation. Hence, the eigenvalue distribution is crucial for the necessity of pre-training.
Figure shows horizontal slices of Figure 5 (right) and compares it with fully supervised models. We notice that all pre-trained models outperform the fully supervised models for 7 1. Interestingly, in the results for = 1 (blue 'x") we use the same amount of data to learn the PCA npp = n as in the fully supervised case (black triangles). While for the pre-trained model we use different data set of the same size to learn the regression, we use use the exact same data in the supervised case.
Figure 6: Pre-training risk for different μ:: simulation. Comparing horizontal slices of Figure 5 (right, a 0.25) for pre-trained models with different amount of pre-training data μ to I) a fully supervised direct regres- sion model and 2) fully supervised PCA- regression model, comparable to Figure
10-1
Risk
dir. reg.
pee pre-train. p=l train. p=2
pre-ttiin., jmd
1=10
10-2 2
pea, pre-train, train samples
0.5
1
2
5
10
7 = m/n
CONCLUSION
Limitations Our proofs the supervised case rely on random matrix theory for which we present asymptotic results for isotropic data. However, it is not trivial to find solutions for the general case, including our latent variable data generator, which requires more research. Similarly, it is an open question how to obtain closed form solution for the complete risk in the scenario with pre-training based on eigenvector alignment which leads to our sample complexity bounds. Furthermore, while observe key phenomena for real-world example, the data here is not approximately low-dimensional manifold as our latent variable data generator and hence not fully comparable.
Supervised case Our theoretical analysis generalizes the results for linear regression (Hastie et al., 2022) which is special case of PCA-regressie without dimensionality reduction (d = m)) In the non-asymptotic regime, Huang et (2022) describe similar results and hence they independently support our theory. Selecting the correct latent dimension d for data from a low dimensional mani- fold is crucial for the risk as Lemma 2 suggests. This is in line with the discovery of latent factors in variational autoencoders from the disentanglement literature (Higgins et al., 2017: Kumar et al., 2018). While results that PCA mitigates the "interpolation peak" due to its regularizing behavior may not surprise, they provide formal guarantees for the performance of a commonly used model on real-world data structures. Practitioners now rely these fundamental guarantees for model development, but more research is needed for general data structures.
Pre-training Our results from Figure 5 that a certain decay rate of the data covariance eigenvalues necessary for pre-training to have its expected effect (more pre-training data is better) may be surprising at first. However, from Theorem 2 it becomes clear that more pre-training data only helps to improve the eigenvector estimation. however, the eigenvectors are already estimated perfectly such as for two blocks of isotropic data (e.g. latent variable data with decay rate a =0,, then using more pre-training data has no effect. Hence, we provide a fundamental insight into the mechanisms of pre-training which highlight that we have to be aware of the data structure instead of following the general philosophy of adding more pre- mre-trannig data. Our results provide the missing link to Xu & Hsu (2019) when their asymptotic generalisation results be used in practice. We believe that our simple regressson model suitable for extensive studies of pre- pre-training phenomena. Therefore, this study lays the groundwork for future research and opens up many questions.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY
Code for reproducibility is attached as Jupyter notebook in the supplementary material and will be published online upon acceptance; all simulation parameters are explained in detail in the paper and copied in the code. All of our numerical simulations are run on Intel Core 17-6850K CPUs @ 3.60GHz in matter of minutes. The computationally most heavy experiment is for pre-training with large μ,, see Figure which takes for one run about minutes for the fine-grained grid that we show in the paper. Averaging over multiple runs for more accurate results increase the computational cost linearly.
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization enror in neural networks. Neural Networks, 132:428-444 2020. (Cited p. 2, 17)
Jean- Mann-Batistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022. (Cited on p. 1)
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient de- scent for deep linear neural networks. In International Conference on Learning Representations, 2019. (Cited on p.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv: 2016. (Cited on p. 2)
Peter I. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexitie: Risk bounds and
structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002. (Cited on p. 2)
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in lin- regression. Proceedings of the National Academy of Sciences, 117(48).30063-3000 2020. (Cited on 17)
Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems, 31, 2018. (Cited on 2)
Mikhail Belkin, Daniel J Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical ftas-variance trade-off. Proceedings of the National Academy of Sciences, 1 116(2)) 1558899-58544 2019. (Cited on 1, 17)
Leo Breiman and David Freedman. How many variables should be entered in regression equation? Journal of the American Statistical Association, 78(381):13 131- 1983, (Cited on p. 2)
Rares-Darius Buhai, Yoni Halpern, of the benefits of overparameterizzaion in learning latent variable models. In International Con- ference on Machine Learning, pp. 121 1-119. PMLR, 2020. (Cited on p. 2)
Yoon Kim, Andrej Risteski, and David Sontag. Empirical study
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020. (Cited on p.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020. (Cited on
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pp. 1305-1338. PMLR, 2020. (Cited on p. 2)
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high- dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11(2): 435-495, 2022. (Cited p. 17)
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, 2019.
Association for Computational Linguistics. (Cited on p. 1, 17)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 1616 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. (Cited on p. 1)
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021. (Cited on p. 2)
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http: ics. uci. edu/ml. /ml. (Cited on p. 18)
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceed- ings of the Thirty-Third Conference Uncertainty in Artificial Intelligence, UAI 2017, Sydney, Australia, August 11-15, 2017. AUAI Press, 2017. (Cited on p.
Stéphane Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in dou- ble descent: Bias and variance (s) in the lazy regime. In International Conference Machine Learning. pp. 2280-2290. PMLR, 2020. (Cited on p. 17)
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 201-208. JMLR Workshop and Conference Proceedings, 2010. (Cited p.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049, 2016. (Cited on p. 1)
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4(1):1-58, 1992. doi: 10. 2e11166//cco.11900 1992.4.1.1. (Cited on 2)
Federica Gerace, Bruno Loureiro, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Gener- alisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning, pp. 3452-345255 PMLR, 2020. (Cited on p. 2, 17)
John Geweke. Bayesian reduced rank regression in econometrics. Journal of econometrics, 75(1); 121-146, 1996. (Cited on p. 4)
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. Advances in Neural Information Processing Systems, 32, 2019. (Cited on p.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019. (Cited on p. 17)
Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Phys. Rev. X, 10: 041044, 2020. (Cited on p. 1, 2, 31)
Sebastian Goldt, Brumo Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zde- borová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, pp. 426-471. PMLR, 2022. (Cited p. 2)

---

Under review as a conference paper at ICLR 2023
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang. Vivek Mangalore Pai, Mannat Singh, Vitaliy Liptchinsky. Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self- supervised pretraining of visual features in the wild. arXiv preprint arXi::2103.01988, 2021. (Cited on 1)
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Sre- bro. Implicit regularization matrix factorization. Advances in Neural Information Processing Systems, 30, 2017. (Cited on p. 2)
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-98) 986, 2022. (Cited on p. 3, 4, 9, 17, 27, 30, 32)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016. (Cited on p. 2)
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. (Cited on p. 9)
Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 2022. (Cited on p.
Ningyuan (T.) Huang, David Hogg, and Soledad Villar. Dimensionality reduction, regularization, and generalization in overparameterized regressions. SIAM Journal Mathematics of Data
Science, 4(1):126-152) 2022. (Cited on p. 3, 9, 17)
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 148-456. PMLR, 2015, (Cited on 2)
Ian T Jolliffe. A note the use of principal components in regression. Journal of the Royal Statistical Society: Series C (Applied Statistics), 31 3103:300,,33, 1982. (Cited on p. 2)
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017. (Cited on p. 2)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Commuiccations of the ACM, 60(6):84-9)) 90, 2017. (Cited on p. 2)
Anders Krogh and John Hertz. A simple weight decay improve generalization. Advances in neural information processing systems, 4, 1991. (Cited on 2)
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018. (Cited on
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Con- ference on Learning Representations, 2022. (Cited on p. 2, 17)
Andrew K. Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and trans- fer learning in deep linear networks. In International Conference Learning Representations, 2019. (Cited on p. 1)
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://annn htwpy/aaw.eeu..coooeee cww.exdd//mii/tttt.ommmmmmmma 2010. (Cited on p. 18)
Young Kyung Lee, Eun Ryung Lee, and Byeong U Park. Principal component analysis in very high-dimensional spaces. Statistica Sinica, pp. 933-956, 2012. (Cited p. 2)
12

---

Under review as a conference paper at ICLR 2023
Daniel LeJeune, Hamid Javadi, and Richard Baraniuk. The implicit regularization of ordinary least squares ensembles. In International Conference on Artificial Intelligence and Statistics, pp. 3525- 3535. PMLR. 2020. (Cited on p. 17)
Andreas Loukas. How close are the eigenvectors of the sample and actual covariance matrices? In International Conference on Machine Learning, pp. 2223.. PMLR, 2017. (Cited on p. 3, 7, 34, 35)
Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions. Advances in Neural Information Processing Systems, 14:10144-10157, 2021. (Cited on p. 17, 32)
A Marčenko and L A Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457-483. 1967. (Cited on p. 4, 24)
William F Massy. Principal components regression in exploratory statistical research. Journal of the American Statistical Association, 60(309):234-255 256, 1965. (Cited on p. 4, 17)
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022. (Cited on p. 2. 31)
Fadia H Metwally. Simultaneous determination of nifuroxazide and drotaverine hydrochloride in pharmaceutical preparations by bivariate and multivariate spectral analysis. Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy, 69(2):343-344 2008. (Cited on p. 17)
Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization classification of high-dimensional noisy gaussian mixture. In Interna- tional Conference on Machine Learning, pp. 6874-6883, PMLR, 2020. (Cited on 32)
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, I(I); 67-83, 2020. (Cited on p. 17)
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003. 2021. (Cited p. 17)
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. / / //aaiiiv... org/ 11122. 6614. (Cited p. 2)
ICLR (Workshop). 2015. URL http:
Thanh V Nguyen, Raymond KW Wong, and Chinmay Hegde. On the dynamics of gradient descent for autoencoders. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2858-2867. PMLR, 2019. (Cited on 2)
Thanh V. Nguyen, Raymond W. Wong, and Chinmay Hegde. Benefits of jointly training autoen- coders: An improved neural tangent kernel analysis. IEEE Transactions on Information Theory, 67(7):469944622 2021. (Cited on p. 2)
Manfred Opper. Statistical mechanics of learning: Generalization. The handbook of brain theory and neural networks, pp. 922-925, 1995. (Cited on p. 2)
Scott Pesme, Loucas Pillaud- Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34:29218-292382 2021. (Cited on p. 1)
Adityamarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Memorization in overparam- eterized autoencoders. /CML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, 2019. (Cited p. 2)
13

---

Under review as a conference paper at ICLR 2023
Adiyaanaraaanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterizeed neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117 (44):27162-2711) 2020. (Cited on 2)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020. (Cited on p.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019. (Cited on p. 2)
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007. (Cited on p. 2, 31)
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021. (Cited on p. 1, 17)
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014. (Cited on p.
Steffen Schneider. Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised Pre-Traine Training for Speech Recognition. Proc. Interspeech 2019, pp. 3465-3469, 2019. (Cited on p. 1)
Michael F Scott, Nick Fradgley, Alison R Bentley, Thomas Brabbs, Fiona Corke, Keith A Gardner, Richard Horsnell, Phil Howell, Olufummilayo Ladejobi. Ian J Mackay, et al. Limited haplotype di- versity underlies polygenic trait architecture across 70 years of wheat breeding. Genome biology, 22(1):1-30, 2021. (Cited p. 5, 32)
Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regular- ization in stochastic gradient descent, In International Conference on Learning Representations, 2021. (Cited on p. 2)
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826. 2016. (Cited on p. 1)
Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into multi- class classification: A high-dimensional asymptotic view. Advances in Neural Information Pro- cessing Systems, 33:8907-8920, 2020. (Cited on p. 32)
Hien Tran, Jeongyeong Kim, Daeun Kim, Minyoung Choi, and Minha Choi. Impact of air pollution on cause-speeific mortality korea: Results from bayesian model averaging and principle com- ponent regression approaches. Science of The Total Environment, 636: 1030- 2018. (Cited on p. 17)
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in Neural Information Processing Systems, 33:7852-7862. 2020. (Cited on p. 2)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017. (Cited on p. 1)
Sethu Vijayakumar and Stefan Schaal. Locally weighted projection regression: An o(n) algorithm for incremental real time learning in high dimensional space. In Proceedings the seventeenth international conference on machine learning (ICML 2000), volume 1, pp. 288 288-293, 2000. (Cited on p. 4)
14

---

Under review as a conference paper at ICLR 2023
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-16467, 2021. (Cited on p. 2)
Martin J Wainwright. Hligh-dimensioal statistics: A non-asymptolic viewpoint, volume 48. Cam- bridge University Press, 2019. (Cited on p. 3)
Kai Wang and Diana Abbott. principal components regression approach to multilocus genetic association studies. Genet. Epidemiol., 32(2):108-118, 2008. (Cited on p. 4, 17)
Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classi- fication: All roads lead to interpolation. Advances in Neural Information Processing Systems, 34: 24164-24177 2021. (Cited on p. 17)
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9933 PMLR, 2020. (Cited on p. 2)
Denny Wu and Ji Xu. On the optimal weighted l_2 regularization in overparameterized linear re- gression. Advances in Neural Information Processing Systems, 33:10112-10113, 2020. (Cited on p. 2)
Xu and Daniel J Hsu. On the number of variables to use in principal component regression. Advances in neural information processing systems, 32, 2019. (Cited on p. 3, 8, 17)
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentationx, 2017. (Cited on p. 2)
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterriztion. In International Confer- ence on Learning Representations, 2020. (Cited on p. 2)
15

---

Under review as a conference paper at ICLR 2023
Appendix
CONTENTS
Additional related work
17
Eigenvalue distribution of real-world data sets
18
C Details on the data generator
19
D Proofs for the supervised case
21
D.I General
21
D.2 Linear regression
21
D.3 Parameter norm
22
D.4
Limiting parameter norm for isotropic features
22
D.5
Risk
24
D.6
Limiting risk for isotropic features
25
Additional numerical results for supervised case
27
E.1
Isotropic data: Bias variance decomposition
27
E.2 Latent variable data
................................
28
E.3 Connection to Ridge regression
30
E,4
Real-world example: Geneties
32
Proof's for the case with pre-training
33
F.1
Estimation of eigenvectors
33
F.2
Concentration inequality
34
Additional numerical results for the case with pre-training
36
16

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORK
This section complements the related work in Section
related work While the double descent has been observed in deep and state-of-the-at models (d' Ascoli et al., 2020; Nakkiran et al., 2021), most theoretical studies focus on simple models: Examples are found for linear regression (Bartlett et al., 2020; Muthukumar et al., 2020; Hastie et al., 2022), ensembles (LeJeune al. 2020; Loureiro et al., 2021), classification (Gerace et al., 2020; Wang et al., 2021; Deng et al., 2022), random features (Belkin et al., 2019; Mei & Montanari, 2022) or small neural networks trained using gradient descent (Goldt et al., 2019; Advani et al., 2020).
reeeparaeeeeiiatton--aiiiiiii
PCA- in applications The PCA-regression model is also known principle compo- nent regression (PCR) (Xu & Hsu, 2019) or PCA-OLS (Huang et al., 2022). The number of chosen principal components or eigenvectors is subject to model selection, see for example Xu & Hsu (2019) for an analysis if the true feature covariance matrix is fully known. Selecting is a crucial step. While we do not specify how to select d, we discuss the implication of model misspecification with d A d. When using supervised setup in Section 4, there are plenty examples when it comes to use PCA-regression models: Early work use the de-correlating property of PCA for their in- puts in small scale examples (Massy, 1965). Tran et al. (2018) uses 10 years of data from Scoul to analyse the impact of air pollution on the health of the population using PCA-regression. Wang & Abbott (2008) makes use of PCA-regression for genetic association to determine genetic variants of human diseases with a large number of features and few samples. Metwally (2008) uses the model spectrophotometry. When using pre-training as in Section 5, the PCA-regression model simplified, linear surrogate for large, nonlinear encoder-decoder models. Examples in this setting are the transformer based BERT model (Devlin et al., 2019) or DALL-E Ramesh al. (2021). In these models, parts of the model are pre-trained large corpus unlabeled data. The pre-trained model can then be used by other developers to fine-tune or adapt the last layer, see Kumar et al. (2022).
17

---

Under review as a conference paper at ICLR 2023
B EIGENVALUE DISTRIBUTION OF REAL-WORLD DATA SETS
In Figure 7 we plot the eigenvalue distribution four real-world data sets. Each of them has a low number of significant eigenvalues with sharp exponential decay. For some data sets such e.g. Steel Plates Fault there is even a low-dimensional data embedding up to about eigenvalue 12 visible. All data sets except MNIST were downloaded from the UCI Machine Learning Repository (Dua & Graff, 2017) through the OpenML interface.
MNIST digit 0
Spambase
106
105
10a
102
10
10-1
10
10-4
200
400
600
20
40
60
Breastcancer
Steel Plates Fault
10a
10a
10
10
10
10 10-7
10
0
10
20
30
0
10
20
30
Figure 7: Eigenvalue distribution of real-world data sets. Top left: Distribution for MNIST digit 0 of the test data set (LeCun et al., 2010). Notice that many of the 784 eigenvalues are almost zero. Top right: Complete features of the spambase data from UCI (Dua Graff, 2017). Bottom left: Breastcancer data set from UCI (Dua & Graff, 2017). Bottom right: Steel-plates-fat fault data set provided by Semeion, Research of Sciences of Communication, Via Sersale 117, 00128, Rome, Italy.
18

---

Under review as a conference paper at ICLR 2023
C DETAILS ON THE DATA GENERATOR
In this appendix we concentrate without loss of generality on the latent variable data generator with orthogonal features introduced in (2) and (13). In matrix notation when collecting all features in rows we can write the data generator as
X ZD +EDI.
(20)
= z0 te.
(21)
Singular value and eigenvalue decomposition Approximating the data matrix with an estimated singular values decomposition and reducing the rank d yields
X = ÚÊVT,
(22)
with estimated singular values Σ = diag(ō), (,, decomposition of the sample covariance matrix
oa)- Similarly we can define the eigenvalue
c= x x VEVEV =
1
(23)
with estimated cigenvalue matrix S diag ($1, 8d).
Covariance matrices
The feature covariance matrix can be written as
Z Z E E
C = E
x x
D DIJE
Z
E
E
D
VSVT,
(24)
by V. The eigenvalue matrix S can be written compare with the sample eigenvectors denoted
where V = D D1] are the true
A
0
S = diag( $1,
sm) = diag( (A1, ..... Ad. 1, 1)
(25)
Lm-dd
Signal-to-noise ratio control the SNR Pr of the features as
For the orthogonal latent variable data generator we can compute
E D
Tr(DADT) Tr(c2)) Tr(D D
PR E 1
(26)
m d
since - and D D Ia Here is correction factor which controls the
Tr(Am-d)
SNR. We define is as
c Pi Pm d)
(27)
d
Tr(A)
If non-orthogonal noise is used, then the first factor reduces to v Pdm/d. In the same way, we can compute the SNR Py of the outputs as
E
0
2
Tr(O AO)
2
Pg
(28)
o
03
with 2 = usually.
Implementation details for data generation For our latent variable orthogonal feature generator, generate the matrices D and D. by first sampling an auxiliary random variable A E R and then orthogonalizing it with a QR-cccompoiition
Ar, ~ ~o0, 1),
(29)
QR A,
(30)
19

---

Under review as a conference paper at ICLR 2023
where the columns of Q are orthogonal. Hence, we can define
D D=cQuda .Q...
(31)
D1 Q ddics
(32)
where the SNR-correction factor e is defined in (27). In order to hold that E [[|@]
x =1 = rg. we generate 0 as
rgg Ad)
0
(33)
20

---

Under review as a conference paper at ICLR 2023
D PROOFS FOR THE SUPERVISED CASE
In this appendix we detail and proof the results shown in section 4.1. After stating some notation and definition in D.I, first show the result for the linear regression solution in D.2. Subsequently, we will derive the result for the parameter norm in D.3 and its asymptotic in the isotropic case in D.4. Finally, the result for the risk is proven in D.5 and its asymptotic in the isotropic case is derived in D.6. For simplicity of notation, we will replace in the indices d by d-it is clear from context that we use d for the estimated latent space and d for the true one.
D.1 GENERAL
Note on notation in main text While in the main paper for the derivation of the linear regression solution in (8) we denote for simplicity the estimated singular value matrix of the data X as Σ, here we are more precise. We distinguish between the case of y > with m > n (left) and the case of
y with m (right)
81
O
Raxm
S
Rnxm
(34)
0
On
When truncating with d < min n) singular values, we obtain in both cases
Σg
E Rad
(35)
ôd
On-dxd
In (8) we write E instead of En and Σ instead of (Σ] En) in order to not overload notation and simplify reading without harming the results. In the following we will use the notation with subscript in order to highlight the zero rows or columns. Further, below we simplify the square matrix with only the first d singular values the diagonal as Eas to indicate its dimensions.
Sample covariance matrix its MMoore-Penrsss pseudoinverse as
We can define the feature sample covariance matrix Ó E m and
č Ò= x_x, n
=+=n(XTX)*
(36)
Using the definition of the truncated SVD, rewrite the sample covariance
Č = - VIVIOTTS = -VSTENT 1 -i VT = VSV
(37)
dd
- - Σ. = nV(()) ΣS V = =n -V VS-IVT
(38)
The above formulation also implies the following which will be useful
dd = VTCVV.
(39)
D.2 LINEAR REGRESSION
We consider the unregularized linear regression solution between the latent variables 2 and the outcome y:
Ô 0= y
(40)
= +
(41)
with ÚTÚ = and y = XB+e
i = -S " (X3+c) =
(42)
3 (2) UT T
(43)
21

---

Under review as a conference paper at ICLR 2023
Where we used X = ÚΣVT. Now we combine the singular value matrices. We indicate dimen- sions of combined matrices, Note that Σg and E are of different sizes.
o
B
B
0
0
82
dxd
dxm.
(44)
0
81
+
0
dxn
=
0]
T
B +
Σ
0]
Ü
(45)
Summarizing the matrices by truncating V and Ü yields the following solution for the regression parameter estimation
0
V 3+ E UJe
(46)
D.3
PARAMETER NORM
Here, prove the parameter norm part of Lemma 1.
Proof. In order to evaluate the parameter norm |012 = Ô we consider
0 0 - 30=3VVVV + Tr(e U
23 V
(47)
where the second term is scalar and hence equal to its trace. Now, make use of the cyclic property of the trace. Furthermore, define & = VVT as orthogonal projector.
θ θ - T/ T cc TU
VE
(48)
dd
Note that the properties of the orthogonal projector with T + and d ④b hold for our definition.
We take the expectation with respect to the noise
Egg θ 0 = 3 4/3 TÚÚ E. [Ee] 1 ÚΣ33)
(49)
= =3 4/3 o. TT U
(50)
B 43 o2 Tr (
(51)
using (39) we can write
of T C
Eg
θ
3 43
(52)
n
The second term uses the sample covariance matrix Ò projected down on the d dimensional eigen- vector space using V.
D.4 LIMITING PARAMETER NORM FOR ISOTROPIC FEATURES
Here, we prove the parameter norm part of Theorem I.
Proof. We can analyze the two terms in (52) independently in the limit of m, n + oo such that
- 7 (0, co) almost surely. Furthermore assume isotropic features Cov(x:)) C I Im-
22

---

Under review as a conference paper at ICLR 2023
First term We can write with the definition of our orthogonal projector.
3 $ 3 VVT
(53)
We can write VT with the SVD definition as V E/ X which yields
8 Φ 3 x X EJ X33
(54)
For the special case of i..d. matrix entries æ, (0,0 1) have by rotational invariance that the distribution of X and XP are equal for any orthogonal P E mxx
D 3 P X UΣ, EJ
XP3 TPB
(55)
Choose P such that P = 3e; with e, as the ith normal vector and then average over i
.,...,
B $ 3 3T (x U E x)/m
(56)
Now again the definition of X ÚΣVT yields
B 2 = 3 VT U TI (TVVVVV
(57)
= =3 B (V "E 7)mm
(58)
Using the same arguments as for the linear regression parameter solution by combining the singular value matrices, we obtain
B $ B 3T<(VV
(59)
Here again identify our orthogonal projector +
= 9 3 (r())mmm
(60)
Since & is symmetric positive definite and since its components V are orthogonal, all eigenvalues of 4 are equal to one, yielding
T 3 B Brank( (D)m
(61)
For m/n - y have to distinguish between 7 1 and 7 1. Therefore, obtain the final version for the first term of the limiting parameter norm:
13 minnd, for 7 << 1 B3 B min(d, n)mm for 7 >
T
43
(62)
Checking the results with considering all principal components, i.e. choosing d d= m (with m > n for > 1), we obtain
13 for 7 <<> 1 1
0 TT
for 7 > 1
which is the same results as for the case of direct regression between X and
Second term For the second term of the parameter norm we can write the trace as the sum over the eigenvalues 8; of Ò but limited the first d eigenvalues due to the projection using V
a² n
T((VTO+t)
o² n
(63)
81
dsF(s)
(64)
where the summation is rewritten as integral over the spectral measure Fg of Ò as and sy is the d largest eigenvalue of Ć. We know that in the limit m, + the spectral measure will almost surely converge to the Marčenko- Pastur distribution F,, which describes the distribution the cigenvalues of Ć
II(VTC+T)+
m 0e n
dF (s)
(65)
8
23

---

Under review as a conference paper at ICLR 2023
There are two steps to solve this integral. First, we need to find out the lower integral bound s and second, solve the integral itself. For 8 = -00, one can use the closed form solution of the Stieltjes transformation fz) of the Marčenko-Paasee distribution and evaluate it at O. However, there is no known closed form solution for general 8y- We therefore solve this part numerically.
Step obtain the lower bound sy We view the spectral measure Fs series of m impulses at 8; with magnitude 1/m because the sum is normalized to Since we only consider the largest cigenvalues, we know that their sum is d/ f/m, see Figure 8a. This sum is the same as the integral from sy over the Marčenko-Pastue distribution, see Figure 8b. Therefore we can find the lower integral bound sy by solving
d
dF,
(66)
18+ - 68 8_)) ds
(67)
2m
78
for sy numerically with $2 (1 = v/)² where 8t is the lowest/highest eigenvalue. Note that E [s_,s+]-
Step 2 solve integral of interest: Now we can solve the integral in (65) numerically from sy to the upper bound s.. Therefore, obtain solution for the second term, which is not based on data but the properties of our data matrix, especially y and d. This concludes the full proof for the asymptotics of the parameter norm.
F
F, (s)
8
1
1
m
772
77
A
d
IR
81
82
8m
8
(a)
(b)
Figure 8: Visualization of steps for variance term derivation. (a) spectral measure impulses and and lower integral bound of integral sy. (b) Marčenko- Pastur distribution (Marčenko & Pastur, 1967) for 7 = 0.3 with specific lower integration bound. The area under the distribution from that threshold is equal to d/m.
D.5 RISK
Here, we prove the risk part of Lemma
Proof. We define the risk as the expectation over the mean squared error, and then use yo = 3 ao +
e, ((x())))))))))))))))) = 0T and = V wo to obtain
R(O)
E(
[10 {²
(68)
Egg
( Eo + )
(69)
Euz
(.3 no + 0 2)2
(70)
Exo
(.3 xo + - OT V x0)2] [Tx)22
(71)
Emo
(13 v0)T T xo + c)²
(72)
(B VO)T C(B v cc
(73)
24

---

Under review as a conference paper at ICLR 2023
For simplicity we first rephrase the term in the bracket using the solution of the regression parameter
estimation in (46). We re-use our orthogonal projector o VVT and define another orthogonal
projector with II I + to obtain
B vo = 3
(74)
= 33-VVB--V
(75)
=Im - - Ue
(76)
II3 V da
(77)
Now we use this expression to take the expectation of the risk with respect to the noise. This yields
E [(0]]] 3 3 NCT33 + E. (Tr(e V CVS s) + R, [Ee
(78)
Here made use of the Trace since the expression is scalar. Hence, of the trace and pull the expectation inside
the cyclic property
B IcnB++ + Tr(V' CV S Ü E. [ee [ee"]er ( + E. [ee)
(79)
with E. [ee| of and Úū-
B ICII + of
+o?
(80)
using (39) for E
0² TT T B nons+
(81)
n
Again, similarly to the parameter norm, the second term here uses covariance matrices projected onto the d dimensional cigenvector space.
D.6 LIMITING RISK FOR ISOTROPIC FEATURES
Here, we prove the risk part of Theorem 1.
Proof. Since we use isotropic features, we have Im. Similar the limiting parameter norm we split the analysis for the two first parts of (81).
First term:
limiting bias
Using isotopic features and the definition of the orthogonal projector,
we have
3 ICM3 = =TIII II33
(82)
B (Im VVT)3
(83)
can the same arguments for the first term in the parameter norm. Namely, rewrite VT ŌJ X in terms of X, assume x; (O0 1) and by rotation invariance the distribution of X and XP are equal, where P is any orthogonal matrix. Then we choose P3 = Be, and average all 1,..., m.
=3 [m PT x S G
(84)
= 3 3 11 Tr(x ÜS ENO x)/m)
(85)
= B 3 (1 Tr(VVT)/ )/m)
(86)
= B 3 (1 Tr(3)/m)
(87)
B 8 (1 rank 3m/m)
(88)
25

---

Under review as a conference paper at ICLR 2023
in the limit of m, oo have the limiting bias as
y almost surely. We therefore obtain the final solution for
3 min(d, mmdmmmmmmmmmmmmmm for ? < 1
B PICT3=
(89)
(T 3 min min(d, (d,n)/m) for 7 >
Again, checking the results with considering all principal components, i.c. choosing d m (with m > for oy > 1), we obtain
0 B B 1 (1- for y>1 1
for 7 < 1
B 3
which is the same results as for the case of direct regression between X and Second term: limiting variance
Using isotropic features we have
Tr TV CVVT of
TVVTC*VV)
(90)
n
n
this the same form as the second term for the parameter norm and therefore yields the same numeric solution by solving
o² TVVTCVVTC+V)>
o 7 J9
(s)
(91)
n
26

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL NUMERICAL RESULTS FOR SUPERVISED CASE
E ISOTROPIC DATA: BIAS-VARIANCE DECOMPOSITION
In Figure 9 extend Figure 2. We additionally show the results for the PCA-regression model with d mt, which corresponds to PCA without compression and therefore a direct regression between input æ and output y. We compare the analytical results from Theorem of the PCA-model to 1) the direction regression model from simulations (solid line) and 2) the analytical solution for direct regression of isotropic data from Hastie et al. (2022). The authors give the solution for the risk in their Lemma and we derive the parameter norm in the same way.
0 0+000
for Y < 1
E2 R(0)
+ of +
(92)
B
+
for y <> 1
1-1
( g + for
for
<
Eg
|
(93)
We see in Figure 9 that the theory form Hastie et al. (2022) for direct regression, the numerical results for direct regression and our PCA-regression results without compression (d m) match and therefore further support our theory.
2.5
Parameter norm
Risk
2
1.5
ne. analticall d=m, analytical
000000
0
0.5
2
10
20
0.5
2
5
10
20
7 m
7 m/n
Figure 9: Supervised results isotropic data: analysis. We compare analytical solutions from Theorem 1 for the PCA-regression without compression (d m) with 1) analytical solution for direct regression and 2) simulations for direct regression
Complementary to Figure 2, we can decompose the risk and parameter norm according Theorem in bias and variance term. The results for this decomposition are shown in Figure 10. We can see that the bias term is nonzero for all 7 and increases for larger 7. Further, we observe a decrease of the variance term. In contrast, in the direct regression model, the variance term reaches peak at y = and therefore forms the classical variance decomposition trade-off for y < 1.
27

---

Under review as a conference paper at ICLR 2023
Risk, bias term
Parameter norm, bias term
10
10
10
100
10-1
10-2
10
0.5
2
10
20
0.5
5
10
20
Y = m/n
m/n
Risk, variance term
Parameter Norm, variance term
10
10
d=25
100
pea, d-50
100
pea, d=100
10-1
10-1 1
10-2
10
0.5
1
2
5
10
20
0.5
1
2
5
10
20
Y = m/n
7 = m/
Figure 10: Bias-variance decomposition of supervised results on isotropic data: analysis. Same as Figure but decomposed in the bias and variance terms of Theorem 1. In all plots the direct regression results are given comparison. Left: Risk. Right: Parameter norm. Top: Bias terms. Bottom: Variance terms.
E.2 LATENT VARIABLE DATA
Complementary result for parameter norm Complementary to the results the risk for a and a 0.25 in Figure 3, we show the results for the parameter norm in Figure 11. We observe that similarly to the isotropic case, the parameter norm decreases monotonically for larger y. This indicates simpler solution for larger y also in the latent variable data case.
Parameter norm,
Parameter norm, = 0.25
dir.
10
d=5
10-1
pea, d-15
d=20
10 10-2
2-40
10-2
10-3
10-3
10
10
0.5
1
2
5
10
20
0.5
2
5
10
20
7 = m
7 = m/
Figure 11: Supervised parameter norm latent variable data: simulation. Left: Parameter
norm of models for data generated with a = 0. Right: Equivalent results with 0.25. This figure complements Figure
28

---

Under review as a conference paper at ICLR 2023
Varying noise levels In Figure 12 we show the results for risk and parameter for different values of og- In the main text we only present results for o = but the results here show that conclusion hold also for additive output noise. Of course, the associated risk increases with the noise level but interestingly, the found solution as the same parameter norm.
Risk, a
Parameter norm, ar 0
109
100
0.66
10-1
10
10-2
10-2
9.
0.5
2
10
20
0.5
10
20
7 m
7
m7
Figure 12: Supervised results for different oA: simulation. We have the same experimental setup as in Figure with a but vary the amount of output noise. Full lines show the PCA-model with d d; dashed lines show the direct regression model for the specific noise level.
Details phenomenon for a > In Figure one could observe for a > that for a certain range of y 1, the optimal PCA-regrrssion model is worse than the direct regression model. We investigated the details of this observation the following ways, which we visualized in Figure 13:
1. We increased the range of y for higher values. Doing so, we can observe that results for direct regression (dashed line) and the PCA-regression model (x-marks) converge again,
2. We conduct more experiments with wider range of values for a. In all tested values, the same phenomenon is visible.
3. Analysing the uncertainty of our risk estimates over the 200 averaged simulations, note that the difference lies within the standard deviation of our risk estimates. This originates from the limited number of test samples (=400) estimate the risk.
Therefore, we conclude that this phenomenon is an artifact of our experimental setup.
29

---

Under review as a conference paper at ICLR 2023
Risk,
Risk, 0.25
10
2
0m
0m
0N
30
NN
X MN
10
7
10
10
dic rng.
peu. da5
pea, d--5
10
peu, d=20
10-2 2
pea,
0.5
2
5
10
20
0.5
1
2
5
10
20
50
m/n
m/n
Risk, 0.5
Risk, 1.0
10
10 2 0
10
10
10-2
10
10-3
1
0.5
5
10
50
0.5
1
2
5
10
20
50
7
m. /1
7
m/
Risk, 1.5
10 à d'ol w
10
10-2
Kix
10 3
0.5
20
50
7 m/n
Figure 13: Supervised risk on latent variable data: simulation. Top row a repetition of Fig- ure 3. Remaining rows for higher values of {0.5, 1, 1.5).
E.3 CONNECTION TO RIDGE REGRESSION
In the main paper, we focused on the unregularized linear regression problem. In this part we compare the PCA-regression model with the regularized Ridge regression solution and as the Ridge parameter
-= (X
y.
(94)
In the first part, we look at isotropic data where we have analytical solution. In the second part we compare numerical simulation for the latent variable data generator.
E.3.1 ISOTROPIC DATA
For isotropic data can compare the results from Theorem 1 for the asymptotic risk in the PCA- regression model with Ridge regression. Corollary 6 in Hastic et al. (2022) provides asymptotic results of Ridge regression for isotropic data. For completeness state the asymptotic Ridge
30

---

Under review as a conference paper at ICLR 2023
result:
Es [((O)) - 322mm (m) Am'(- o
(95)
1-7-2- (1-----00--2----- and m' (z) as the derivative w.r.t z. The optimal Ridge regu- 2/22 larization is achieved at X* =0²7/B 3 which then yields the optimal risk
with m(2)
E. [((0~)) om +
(96)
Note that the optimal Ridge regularization strength is function of and monotonically increases with y. Optimal regularization is not achieved by a single regularization value.
Figure 14 visualizes the comparison of the analytical solutions. The lowest risk for all is obtained for the optimal Ridge regression solution. Comparing the solutions from Ridge regularization with different with the solution from PCA-regression with different choices of d shows qualitative different behavior for isotropic data. While Ridge regression smoothens out the interpolation peak of direct regression with well tuned X, for PCA-regression we require a sufficiently large amount of eigenvectors, i.e. large d to obtain risk lower than the null risk. Overall, optimally tunes Ridge regression outperforms PCA-regression for all 7
rides In X--10, (dir. reg:)
100.4
ridgs In A=:3
A=-2
X=1
O
ridgo, In A=X*
Risk
gea, d=25
100.2
d=50
pes, d-100
10
0.1
0.2
0.5
2
10
20
7
m/n
Figure 14: PCA-regression comparison with ridge regression for isotropic data: analysis. Solid lines depict the Ridge regularized models. Note that the red solid line with very low À is equal the unregularized direct regression. Dashed lines depict the analytical PCA-regression model results.
E.3.2 LATENT VARIABLE DATA GENERATOR
In this setting, use the latent variable data generator without eigenvalue decay (a 0) and compare PCA-regression model without model misspecifications, i.e. d = d, to solutions using different Ridge parameters. We rely on numerical solutions in this section. The results are in Fig- 15. While for isotropic data, the comparison between both models shows qualitative different behavior, here for the latent variable data the results indicate clear connection between both models for large values of X. This observation is theoretically justified from the known relationship of both methods on the cigenvalues. Ridge regression lifts all eigenvalues S of the features by a value of A
X
X + Apm V ALa)V.
(97)
Here, V are the true, non-truncated eigenvectors. In contrast, the PCA-regression model cuts the eigenvalues off at a threshold chosen by d, which is clear in (7). The main difference is that with ridge regression, there smooth change of the risk controlled by the ridge parameter whereas with PCA-regression there is hard cut-off.
Figure 15 may imply that the optimal Ridge penalty is at À + oc as avoids the interpolation peak at no additional cost. Previous studies have concluded that finite Ridge regularization is better. Gerace et al. (2020) uses the hidden manifold model from Goldt et al. (2020) and Mei & Montanari (2022) use random features model by Rahimi & Recht (2007) which can be as two-layer neural network. Both studies conclude that finite achieves optimal regularization. However, we working with linearly separable data, which is closer to the latent space model in Hastie et al.
31

---

Under review as a conference paper at ICLR 2023
(2022). The difference to setup is, that for us both, the data generating process and the PCA- regression model have cow-dimensional embedding. The conclusion in Hastie et al. (2022) that the best Ridge regularization is À 0 and achieved in y > 1 may hold for as well but is difficult to proof with our numerical results in Figure 15.
Optimal penalty À oc for Ridge regularized problems was also observed in previous studies with different setups to ours. Both Mignacco et al. (2020) and Loureiro et al. (2021) study the classification of high-dimensional (isotropic) Gaussian-mixtures of balanced data from each mixture and show that large À are necessary to reach the Bayes-optimal performance. Thrampoulidis et al. (2020) studies a similar model for the classification of Gaussian mixtures as well as for multinomial logit model where they identify that the class averaging algorithm, which is equal to Ridge regression with A = oo, performs optimal in some settings.
100
100
Parameter norm
rigge In
rigge. In A-1 ridge. In X-3
nigee In X-6
Risk
10-1
10-1
pea, d-20
10-2 2
10-2
M.N
0.5
1
2
5
10
20
0.5
1
2
5
10
20
7 = /=m/n m/n
7 = m n
Figure 15: PCA-regression comparison with ridge regression for latent variable data: simu- lation. We highlight the similarity of the results obtained with large ridge parameter to our PCA- regression model. Left: Risk. Right: Parameter norm.
E.4 REAL-WORLD EXAMPLE: GENETICS
Background The Diverse MAGIC Wheat data set' is based on 16 founding wheat varieties which were listed between 1935 and 2004. These varieties were interbred to obtain new wheat varieties. From the resulting wheat types the genome of total of 502 wheats were sequenced. This genome sequence consists of = 1.1 M single nucleotide polymorphisms. Furthermore, phenotypes of the 502 wheat types were analysed, see Scott al. (2021).
Data processing The genotypes consist of binary features. The binary variables represent equality or difference to reference genotype. The phenotypes are real-values variables. We choose the phe- notype column named 'HET 2' in this example. Missing values for both, genotype and phenotype are replaced with the mean value of the variable. We select a subset of genotypes as inputs randomly uniform obtain the necessary m features. Then, normalize both, genotype and phenotype by z-transformation.
Data analysis In Figure 16 we plot the eigen- value distribution for the Diverse MAGIC Wheat data set, similar to the ones Appendix B. We observe that the eigenvalue distribution is heavy tailed. does not depict a clear example a low dimensional latent manifold. Therefore, ing the PCA-regression model will discard some useful information similar to the isotropic case.
Genetics data set
10
100
10-1
100
200
300
400
500
Figure 16:
Eigenvalue distribution of the Di-
verse MAGIC Wheat genetics data set.
httt:://ttweb. cs. uel aac. /us www.///s Cd verse/
32

---

Under review as a conference paper at ICLR 2023
F PROOFS FOR THE CASE WITH PRE-TRAINING
This appendix derives and proofs the results section 5.2. First, we derive the results for the estimation of the eigenvectors with the PCA loss in F.I. Then, show derive the concentration inequality for the PCA loss F.2.
F.1 ESTIMATION OF EIGENVECTORS
Here, we prove Lemma 2 for the loss difference for the projection into the true or estimated latent space from (18). We will look at both losses induced by the two projections separately,
Proof. First, define the loss for the projection into the true latent space as in (16)
L() = E [||=112 || x1
(98)
We can write the second term the following where scalars and apply the expectation on ææ
the cyclic property of the trace
E
[
E x ( D
(99)
[Tr( (D D T D*#)]]
(100)
-TrDD'VS (D
(101)
Since V [D
D1] defined in (24), we obtain
E
D +
x [
Tr
[Ta
0 0dxm-m]
S
0 Om-mxd
Ia
(102)
T+(A)
8t
(103)
Hence, the loss C(D) becomes
d
C(D)
8;
8;
2
8,
8i
(104)
i=1
1=
i=++1
Second, define the loss for the projection into the estimated latent space in (17)
C(H) = |
[
(105)
Again, we can write the second term the following by using the same arguments for the first term
E
H ( H) (hT
(106)
The last equality holds by switching to vector notation where factors can be combined to squared terms. Hence, the loss C(H) becomes
((H)
8
h e, 23.
(107)
Combining both solutions, the loss difference yields
m
((D)=
Σ S
h. v))
(108)
i=d+1
d
8; +
Σ
(h] 03)
23)
(109)
1=1
i=1
j=1
33

---

Under review as a conference paper at ICLR 2023
We can multiply the first term by |0y1 v; Io, = h vy
d m
- - Σ (,
(110)
1=1
i=1 j=1
Now we can combine terms but have to be careful due to different end indices of the sum
Eド= (h. us) 53) Ei
for d < d
-
j=1 u,) - 84)
for d d
-ー (he U;) 87) + ΣVt Σラ=1 (T.
Po for d d
(111)
which we can combine into
min(d.d)
d
((H) L(D) = Σ Σ ( 0j) ( -85) +
Ui P83
(112)
t=d j=1
=0 d2d
dsd
which concludes the proof.
F.2 CONCENTRATION INEQUALITY
Here, we prove the concentration incquality presented in Theorem
Proof. Write the loss difference (18) with the same argument in its derivation by including the factor |0, 22 = to the second summation yields
min(d,d)
à
((H) - L(D) Σ
Σ [ us) -s) j=l
h vj
8
+
( 03 )38
End 7=
i=d
(113)
Notice that s; - sy 2 for and 8; - s₁ S for removing the negative indices from the first summation
Therefore, we can upper bound by
min(d,d)
à
S Σ (hT - 8,1
Σ>
h
85
(114)
J>1
idd==1
i=d-=1
We simplify by denoting the three terms as
((H) L(D) = +
(115)
Now we define the probability that this upper bound on the loss difference is larger than a chosen real t. We upper bound this expression by applying the union bound
P(a Pla+b+e>> < P(a f) + P(b t) - P(e>
(116)
Recall Corollary 4.1 from Loukas (2017): We have that for any weights way and real > 0 that
4w;sk k2 mnt 2;)²
P
Σ
wyy
(h (hTT))) >1 S
(117)
where ki = EE ss) 8z). In accordance with this Corollary, we define
||::00
U 1
of and wij + 0 when s; + 8j and sgn(ss 8,)28; sgn(s;
8s s5l for i min(m, (d,d).d
St for d d, $j for d d,
wiy
(118)
0 otherwise
34

---

Under review as a conference paper at ICLR 2023
Hence, we obtain
P ((C(H)) L(D) 1) 2
4
(d
h2
k3 (st
d
(119)
1 pp
85)
(8:
83)2
++
i=d
i=d jm
with k sy(ر + Tr(C)) from Corollary 4.3 in Loukas (2017). This Corollary holds for our latent variable data generator. This concludes the proof.
35

---

Under review as a conference paper at ICLR 2023
G ADDITIONAL NUMERICAL RESULTS FOR THE CASE WITH TRE-TRAINING
While in Section 5.3 we concentrate on the well specified case with = d, here we show the effect of model misspecification. Specifically for the same data generator with d 20 choose d = {15, 20, 403. The results for the full risk over all μ in Figure 17 and slices of this figure are in Figure 18. We can observe a qualitatively similar behavior to model misspecification as when train fully supervised,
Section 4.2 or Figure 3: For < d, the risk is high and does not decrease significantly for larger y. For d 2 d, the risk decreases expected. Therefore, from observations the conclusions for model misspecificatiion from the supervised case translates to the case with pre-training.
= 0, d 15
0.25, d = 15
10
10
0.5
0.2
n
0.4
0.1
5
0.3
0.05
0.25
0.5
2
10
0.5
1
2
5
10
= m/n
= m n
a = 0, d =
Q = 0.25, d 20
10
10
0.2
0.2
= 7pl n
0.05
0.02
10-2
10 2
0.5
2
10
0.5
2
10
? m/n
7
m/n /n
0, d = 40
0.25, d = 40
10
10
0.2
0.2
M = Ttp 7p/n n
0.1
0.1
0.05
0.05
5
0.02
0.5
2
10
0.5
2
10
m/n
m. m/n
Figure 17: Pre-training risk on latent variable data: simulation. We use the latent variable data generator with d 20. In this simulations show the effect of model misspecification. Left: Risk for data with feature covariance eigenvalue decay of = Right: Same setup but for a = 0.25. Top row: à < d. Middle row: d = d. This is a repetition of Figure 5. Bottom row: d.
36

---

Under review as a conference paper at ICLR 2023
0,
15
a = 0.25, à 15
100
100
10
dic, tog.
10
pre-trimm pom
pre train 20M2
pre tramm 1om
pro-train. 10=10
pre-imain tnin samples
0.5
2
10
0.5
5
10
7 = m/
7 m n
d = 20
0.25, d = 20
0
10-1
10-1
10 10-2
10-2
0.5
2
5
0.5
10
7 m m/n
7 m/n
0
0, = 40
0.25, d = 40
10
10-1
10 10
10-2
0.5
2
5
10
0.5
2
5
10
7 = m/n
7 m/ /
Figure 18: Pre-training risk for different horizontal slices of Figure 17: simulation. Left: Risk
for data with feature covariance eigenvalue decay of 0. Right: Same setup but for a = 0.25. Top row: d < d. Middle row: = d. Bottom row: > d. The middle right figure is a repetition of Figure 6.
37


Reviews:
[{"review_id": "xpszsCuTv-", "reviewer": "Reviewer_eejS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is a theoretical paper studying a method for regression in high dimensions. In particular, it studies the use of PCA followed by linear regression. Their motivation for doing so is to better understand overparameterization in more general encoder-decoder models. The authors provide an explicit formula for the expected risk and prove the asymptotic behavior of this risk under a particular random covariate model. Empirically, these formulas do not show the \"double descent\" behavior that comes up in many other overparameterized models. The authors further study the effect of pre-training by considering a second regression method. They assume access to a (possibly large) pool of covariates $x_i$ without responses $y_i$. This large pool of covariates is used to estimate the top-$\\hat d$ eigenvectors of the distribution of the $x_i$; then, given a training $\\{(x_n, y_n)\\}$, regression is done in the space of these estimated top-$\\hat d$ eigenvectors. The authors prove a concentration inequality about how well these top-$\\hat d$ eigenvectors are estimated.", "summary_of_the_review": "Overall, I think the paper's current results need to either be presented significantly differently or some new results are needed to better connect the current results to the rest of machine learning practice / theory. that said, I'm not an expert in the area of PCA, so I'm open to being convinced that the results are meaningful as-is.", "strength_and_weaknesses": "The results and experiments in the paper are individually interesting, but I'm not totally clear on how they come together to tell a cohesive story about overparameterization in encoder-decoder models.\n\n1. What is the purpose of Figure 4 and the discussion of ridge regression? I don't think these are new results (e.g. they seem to be covered in Hastie et al. 2022), and it's not clear to me how they contribute to the discussion of PCA.\n2. Section 5.2 is supposed to be a theoretical analysis of the pre-trained PCA model. But the main result of this section (Theorem 2) is just about estimating covariance matrices from data. This definitely seems like a critical step along the way to understanding the risk of this pre-trained model, but this only gets us part of the way there. There is some discussion below Theorem 2 about connections to the risk, but these results seem like high-level qualitative discussions, rather than theory.\n3. I don't immediately see how the paper's results relate to PCA-followed-by-regression in practice or other machine learning models: (3i) all of the results are about a fixed $\\hat d$ in the paper. But in practice, one has to select $\\hat d$ somehow, and this selection will change the risk; it's hard to say if the title of the paper (\"No double descent in PCA\") will hold in this realistic case. (3ii) There's not any discussion about how the models the authors study relate to actual models in practice. E.g. a discussion along the lines of Section 1.2 of Hastie et al. 2022 would be really helpful, especially since I don't think the model studied here is actually used in practice (see point (3i)).\n4. In the conclusion, I'm not sure I see why real datasets would present such a challenge here. Could one not use a held-out set to estimate the risk and use subsampling to vary $\\gamma$? \n\nI also have a few smaller points about the paper's experiments\n1. Some conclusions are drawn about the relative order of the lines in on the right of Figure 3. I don't think these are valid conclusions from this plot. The differences seem to be on the order of about 0.01 or less, and (I think) the risk is estimated here by averaging over 400 test points. This doesn't seem like enough test points to make accurate inferences about differences this small.\n2. In Section 5.3, $\\hat d = d$ is used because \"the effects of misspecified models is elaborated in Section 4.2.\" But in Section 4.2, we are studying a different model. Why should we expect the results of that section to relate to the pre-training regime studied here?\n3. \"We see that the loss with pre-training is lower, which indicates that using different data sets and therefore more diverse data is advantageous.\" I don't think you can conclude this. Pre-training is giving more data overall. You would have to compare to both methods getting the same volume of data to make a conclusion like this; otherwise, I think a reasonable explanation for the behavior we are seeing here is that pre-training is getting more data.\n\n\nMisc smaller points\n- \"The variance (second) term is controlled...\" and \"The first term represents the variance and the last one the bias.\" Both of these statements are proceeded by two equations; which equation are these statements about (or is it both)?\n- Lemma 1 is about centered features, but I don't think centered features were previously discussed.\n- \"We compute risk and parameter norm as defined in Lemma 1\". Lemma 1 has two equations for risk / norm (one the actual risk and the other the expectation). Which one is being used here?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wknO_Ro557", "reviewer": "Reviewer_7G3J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the risk (generalization error) of the PCA least squares estimator and shows that dimension reduction can avoid the peaking in the risk curve. The analysis is divided into two parts: (i) precise bias-variance decomposition in the proportional limit for isotropic data, and (ii) non-asymptotic bound on the error due to estimating the low-dimensional projection in an unsupervised manner. Theoretical results are supported by some small-scale experiments.  ", "summary_of_the_review": "In my opinion this submission is below the acceptance bar due to the incremental theoretical contribution. I am happy to update my evaluation if the authors can adequately discuss the prior results and clarify the novelty in their theoretical analysis. ", "strength_and_weaknesses": "## Strength\n\nThe paper studies a relevant and fairly well-motivated problem: since dimension reduction in PCA can be interpreted as a form of \"regularization\", the least squares estimator on the truncated features might be able to avoid double descent. The main text is fairly easy to read, and the precise asymptotic results nicely illustrates the benefit of PCA close to the interpolation threshold. \n\n## Weaknesses\n\nMy main concern is that the theoretical results are somewhat incremental and underwhelming. \n\n1. The benefit of PCA in avoiding double descent has already been shown in [Teresa et al.]. The authors claim that the main improvement is the asymptotic result, but this is only shown for isotropic data, and the analysis follows from standard random matrix computation similar to that in [Hastie et al.]. Moreover, the observation that dimensionality reduction can suppress the peak by improving the stability of the pseudo-inverse is rather intuitive. \n\n2. While [Teresa et al.] does not provide the asymptotic risk formula, it highlights the role of alignment between the true coefficients and the features. This alignment is known to impact the performance of both ridge regression and PCR as shown in [Wu and Xu], but cannot be captured by the isotropic data model. In this submission the authors also considered a latent variable model with (anisotropic) decaying eigenvalues, but the analytical solution of the risk is not derived. \n\n3. The authors should also discuss the difference in the observed phenomena between the studied PCA-regression model and the PCR model in [Xu and Hsu] that assumes access to the population covariance, which can be obtained in an unsupervised manner. The precise risk of the population PCR estimator has been analyzed in [Wu and Xu] for general features and true coefficients, and conditions under which low-dimensional projection does not benefit the model performance are also given. \n\n4. The comparison between the PCA model and ridge regression is not quantitative. Note that the asymptotic risk of both estimators are available in the isotropic setting. Hence it would be nice to quantitatively compare the performance of the two models (e.g. under optimal truncation and regularization for a given SNR). \n\n5. [minor] The data-generating process that assumes a low-dimensional structure is sometimes termed the \"hidden manifold model\", for which the precise risk of two-layer networks has been studied in many prior works, see [Gerace et al.]. \n\n\nXu and Hsu 2019. On the number of variables to use in principal component regression.  \nHastie et al. 2019. Surprises in high dimensional ridgeless least squares interpolation.  \nTeresa et al. 2020. Dimensionality reduction, regularization, and generalization in overparameterized regressions.   \nWu and Xu 2020. On the optimal weighted $\\ell_2$ regularization in overparameterized linear regression.   \nGerace et al. 2020. Generalisation error in learning with random features and the hidden manifold model. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XMKOL_ZkFf", "reviewer": "Reviewer_JqQV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the supervised learning problem of fitting labels generated from a linear model with additive Gaussian noise $y_{i} = \\theta^{\\top}x_{i}+\\varepsilon_{i}$ for three generative models for the features $x_{i}\\in\\mathbb{R}^{m}$: (a) *Isotropic model*: i.i.d. Gaussian features $x\\sim\\mathcal{N}(0,I_{m})$; (b) *Latent model*: noisy features generated from a latent Gaussian vector $x_{i} = Dz_{i}+e_{i}$ with $z_{i}\\sim\\mathcal{N}(0,\\alpha I_{d})$; (c) *Latent model with orthogonal noise*: similar to the latent features model, but with the noise orthogonal to the features $x_{i} = Dz_{i}+D_{\\perp}e_{i}$ where $D^{\\top}D_{\\perp}=0$. Given a training set $(x_{i},y_{i})$, $i=1,\\cdots,n$, two scenarios are considered: the statistician either (1) fits the labels by first performing PCA on the features followed by linear regression on their projection on the top singular vectors; (2) performs PCA on a pre-training set $(x_{i})$ for $i=1,\\cdots,n_{p}$ and performs linear regression on the projection of the training data on the top singular vectors learned in the pre-training phase. In either cases, the number of principal vectors might not coincide with the dimension of the data latent space. This is compared with performing linear regression directly on the features.\n\nThe two main theoretical results are:\n\n- *Theorem 1*: an exact asymptotic characterisation of the generalisation error in terms of a bias-variance decomposition of model (1) in data model (a).\n- *Theorem 2*: a concentration bound for the training risk of estimating the eigenvectors of the optimal decoder.\n\nAnd the main conclusions of the work are:\n\n1. To show that performing PCA before regression has an implicit regularisation effect, mitigating for instance the interpolation peak in this model. Despite this, doing PCA might also harm the risk in the regime where one has more features than data.\n\n2. Pre-training the PCA features can lead to better estimation performance in the regime where one has more features than data. However, increasing the quantity of pre-training data only decreases the risk up to the point where the optimal decoder is perfectly learned.   \n", "summary_of_the_review": "Overall, I find the discussion in this work interesting. However, the fact that it is supported by incomplete theoretical results is deceiving, and hinders a finer understanding of the phenomenology reported. I think that this contribution could be of potential interest to the ICLR community if the theoretical part could be further developed.", "strength_and_weaknesses": "**Weaknesses**:\nThe major shortcoming of this work is that the theoretical results required to support the discussion are largely incomplete. As I discuss in the summary, this works deals with 3 data models and 2 estimation models. In the first part of the work, estimation model (1) is discussed in the context of data models (a) & (b), however the asymptotic results in Theorem 1 are only for data model (a), the rest of the plots relying on numerical observations. In the second part of the work, estimation model (2) is discussed in the context of data model (c), a simplified version of model (b) where the noise is orthogonal to the optimal decoder. Here, only a concentration bound on the estimation of the decoder is presented, allowing only for an informal discussion of the generalisation error and a direct comparison with the full supervised setting in the first part. While numerics might seen enough to draw the phenomenology, it has strong limitations regarding finer questions, for instance whether the observation that direct regression can outperform PCA in model (1)+(b) at $\\alpha>0$ (see **[Q2]** below.)\n\nA minor weakness is the lack of comparison with real data, which would give an interesting support to the relevance of the models.\n\n**Strengths**:\nThe overall discussion and conclusions of the paper are interesting, and have many potential follow ups. I believe that having a complete theory (even if heuristic) for at least Part I would make this an interest contribution for the ICLR community.\n\n\n**Questions**:\n- **[Q1]**: In Fig. 1 left, why are the analytical curves noisy? According to Theorem 1, they should be deterministic quantities of the model parameters.   \n\n- **[Q2]**: Are the authors confident of the observation in the top of Page 6, that for $\\alpha>0$ and certain $\\gamma>1$ the performance of regression on the top of PCA is worst than direct regression? Given that these are numerical results with noisy curves and in Fig. 3 (right) the difference is quite small, I believe this is not sufficient to support this claim. Can the authors provide an example (maybe larger $\\gamma$ or larger $\\alpha$) where this behaviour is pronounced?\n\n- **[Q3]**: In the discussion around Fig. 4, the authors compare PCA to a fixed $\\ell_2$ penalty. The figures suggest that the optimal regularisation here is actually $\\lambda = \\infty$. This is in contrast to other toy settings for studying overparametrisation, e.g. random features model, where the optimal regularization is finite [Mei & Montanari '22; Gerace et al. '20]. Is this observed in other cases (e.g. $\\hat{d}\\neq d$, $\\alpha > 0$, $\\sigma_{y}>0$)? Can the authors make sense of that?\n\nAs related side note, optimal $\\lambda = \\infty$ was observed in classification with balanced & isotropic Gaussian mixture data in [Mignacco et al. '20; Thrampoulidis et al. '20; Loureiro et al. '21], where it corresponds to the optimal Bayes-optimal plug-in estimator for this problem. However, for class imbalance [Mignacco et al. '20] and anisotropic mixtures [Loureiro et al. '21] the optimal regularizer is finite. Maybe there is an interesting connection to be drawn.\n\n\n**Comments**:\n- **[C1]**: Despite being employed by different authors, for the isotropic model the terminology over- and under-parametrised is misleading. Since the number of parameters in both the data model and the statistical model are proportional to the features dimension $m$, increasing the number of parameters actually decrease the sample complexity $\\gamma^{-1} = n/d$ of the problem, making it harder to learn (as shown in the right-side of the peak in Fig. 2).\n\n- **[C2]**: Although only the $\\mu>1$ (less pre-training data than training data) is considered here, the case $\\mu<1$ might also be of interest, since it is common in the context of transfer learning, where the estimation of a latent model for a big data set might be publicly available, and the statistician is not able to perform PCA on the combined data (either because she has not access to the pre-train data or because she lacks computational resources). Do the authors expect a different phenomenology in this case?\n\n- **[C3]**: The latent data model studied in this work is asymptotically equivalent to the random features / hidden manifold model studied in [Mei & Montanari '22; Gerace et al. '20]. Indeed, Gaussian equivalence [Mei & Montanari '22; Goldt et al. '22; Hu, Lu '20] asymptotically implies that:\n$$\nx_{i} = \\sigma(D z_{i}) \\asymp D z_{i} + \\kappa_{\\star}\\xi_{i}\n$$\nfor some constants $\\kappa_{1},\\kappa_{\\star}$ and an independent noise $\\xi\\sim\\mathcal{N}(0,I)$ (for simplicity assume $\\sigma$ odd), which is exactly the latent model in eq. (1). It would be nice if the authors comment on this connection and cite the related literature.\n\n\n**Smaller typos and suggestions:**\n- Transposes $^{\\top}$ are missing in a couple of equations, e.g. eqs. (2), (4).  \n- Just below eq. (7), it would be good to write $\\beta\\in\\mathbb{R}^{m}$ and the transformation of the noise explicitly for the sake of clarity.\n- $\\hat{y}_{0}$ in the expectation above eq. (9) is not defined. I guess the authors mean an expectation over a new sample $(x,y)$ with $\\hat{y} = \\hat{y}(x)$?\n- In the statement of Theorem 1 for the isotropic case ($d=m$) it would be clearer to keep only one of these two dimensions, or write explicitly $d=m$.\n- In figures 3 and 4, which curves are theoretical or numerical results? It would be nice if the authors could be more precise in the captions.\n- Not as related to the discussion in the paper, but since the authors mention in the related works. Double descent has discussed in the context of random features classification in [Gerace et al. '20] (before [Wang et al. '21]) and of ensembling methods also in [Ascoli et al. '20; Loureiro et al. '22]. The interpolation peak has been observed in a few precursors to [Belkin et al. '18], including in analytical results for least squares in the isotropic model in [Krogh, Hertz 91'; Opper '95] and numerically for neural networks in [Geman et al. '92].\n\n**References**\n\n[[Gerace et al. '20]](http://proceedings.mlr.press/v119/gerace20a.html): F Gerace, B Loureiro, F Krzakala, M Mezard, L Zdeborova. *Generalisation error in learning with random features and the hidden manifold model*. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3452-3462, 2020.\n\n[[Mignacco et al. '20]](http://proceedings.mlr.press/v119/mignacco20a.html): F Mignacco, F Krzakala, Y Lu, P Urbani, L Zdeborova. *The Role of Regularization in Classification of High-dimensional Noisy Gaussian Mixture*. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:6874-6883, 2020.\n\n[[Thrampoulidis et al. '20]](https://proceedings.neurips.cc/paper/2020/hash/6547884cea64550284728eb26b0947ef-Abstract.html): C Thrampoulidis, S Oymak, M Soltanolkotabi. *Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View*.  Part of Advances in Neural Information Processing Systems 33 (NeurIPS 2020).\n\n[[Loureiro et al. '21]](https://proceedings.neurips.cc/paper/2021/hash/543e83748234f7cbab21aa0ade66565f-Abstract.html) B Loureiro, G Sicuro, C Gerbelot, A Pacco, F Krzakala, L Zdeborová. *Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions*. Part of Advances in Neural Information Processing Systems 34 (NeurIPS 2021).\n\n[[Goldt et al. '22]](https://proceedings.mlr.press/v145/goldt22a.html): S Goldt, B Loureiro, G Reeves, F Krzakala, M Mezard, L Zdeborova. *The Gaussian equivalence of generative models for learning with shallow neural networks*. Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR 145:426-471, 2022.\n\n[[Hu, Lu '20]](https://arxiv.org/abs/2009.07669): H Hu, YM Lu. *Universality Laws for High-Dimensional Learning with Random Features*. arXiv: 2009.07669 [cs.IT]\n\n[[Ascoli et al. '20]](http://proceedings.mlr.press/v119/d-ascoli20a.html): S D’Ascoli, M Refinetti, G Biroli, F Krzakala. *Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime*. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2280-2290, 2020.\n\n[[Loureiro et al. '22]](https://proceedings.mlr.press/v162/loureiro22a.html): B Loureiro, C Gerbelot, M Refinetti, G Sicuro, F Krzakala. *Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension*. Proceedings of the 39th International Conference on Machine Learning, PMLR 162:14283-14314, 2022.\n\n[[Krogh, Hertz 91']](https://papers.nips.cc/paper/1991/hash/8eefcfdf5990e441f0fb6f3fad709e21-Abstract.html): A. Krogh, J. Hertz. *A Simple Weight Decay Can Improve Generalization*. Advances in Neural Information Processing Systems 4 (NIPS 1991)\n\n[[Geman et al. '92]](https://ieeexplore.ieee.org/document/6797087): S. Geman, E. Bienenstock, R. Doursat. Neural Networks and the Bias/Variance Dilemma. in Neural Computation, vol. 4, no. 1, pp. 1-58, Jan. 1992.\n\n[Opper '95]: M. Opper. *Statistical Mechanics of Learning : Generalization*. In The Handbook of Brain Theory and Neural Networks. pp. 922--925 (1995)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CXFIGWWx-N", "reviewer": "Reviewer_YsaP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the setting where response y and input x follow a linear transformation of the latent variable z, and the goal is to learn y from x through PCA regression. The paper shows that for fixed PCA dimension, there is no double-descent phenomena as the ratio between dim(x) and sample size n varies (the theorem is for isotropic data). Numerical experiments in multiple data settings verify the theoretical finding. The paper shift to discuss a pre-training setting where more data is allowed for the PCA step and built theoretical analysis over a slightly different data assumption. ", "summary_of_the_review": "The paper shows some interesting results on the asymptotic phenomena of supervised learning with high-dim inputs but reduced to low(and fixed)-dim space through dimension reduction. The analysis are based on somewhat simplified settings, but are consistent with numerical observations. Without judging too much on the significance and novelty of the paper, I'd put my rating as \"marginally above the acceptance threshold\". If more experience reviewer challenges the novelty/significance of the paper, my rating would as a result be less confident. ", "strength_and_weaknesses": "Strength:\nThe paper is relatively clear and considers a relatively simple setting for theoretical understanding. Within a simple setting, it reveals that if the implicit dimension is fixed, there is no double descent behavior as input dimension and sample size change. The 'limitations' paragraph in section 6 point out the weaknesses of the current work. \n\nOther weakness:\nThe analysis is based on a simplified setting (not a big deal), and the theorem from each section is based on a set of its own simplified assumptions. For example, assumption (15) is clearly for analytical purposes and I don't think it is necessary. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject