Title: Least Squares Regression Can Exhibit Under-Parameterized Double Descent
Venue: NeurIPS 2024
Accepted: yes
Keywords: Learning Theory

Generalization

Random Matrix Theory

High Dimensional Statistics
Abstract: The relationship between the number of training data points, the number of parameters, and the generalization capabilities of models has been widely studied. Previous work has shown that double descent can occur in the over-parameterized regime and that the standard bias-variance trade-off holds in the under-parameterized regime. These works provide multiple reasons for the existence of the peak. We postulate that the location of the peak depends on the technical properties of both the spectrum as well as the eigenvectors of the sample covariance. We present two simple examples that provably exhibit double descent in the under-parameterized regime and do not seem to occur for reasons provided in prior work.

Least Squares Regression Can Exhibit Under-Parameterized Double Descent

Abstract

The relationship between the number of training data points, the number of parameters, and the generalization capabilities of models has been widely studied. Previous work has shown that double descent can occur in the over-parameterized regime and that the standard bias-variance trade-off holds in the under-parameterized regime. These works provide multiple reasons for the existence of the peak. We postulate that the location of the peak depends on the technical properties of both the spectrum as well as the eigenvectors of the sample covariance. We present two simple examples that provably exhibit double descent in the under-parameterized regime and do not seem to occur for reasons provided in prior work.

1 Introduction

This paper demonstrates interesting new phenomena that suggest that our understanding of the relationship between the number of data points, the number of parameters, and the generalization error is incomplete, even for simple linear models. The classical bias-variance theory postulates that the generalization risk versus the number of parameters for a fixed number of training data points is U-shaped (Figure 1a 1 ). However, modern machine learning has shown that if we keep increasing the number of parameters, the generalization error eventually starts decreasing again [2, 3] (Figure 1b 2 ). This second descent has been termed as double descent and occurs in the over-parameterized regime , which is when the number of parameters exceeds the number of data points. Understanding the location and the cause of such peaks in the generalization error is of significant importance.

(b) Modern Double Descent.

under-parameterized

over-parameterized

ptimumModel

Test risk

Risk

"classical"

"modern"

Error

regime

interpolatingregime

Training risk:

Bias2

interpolation threshold

Model Complexity

Capacity of H

Many different theories have been postulated for the appearance of the peak. The prevalent theory is that when the model is under-parameterized, the learning is constrained. This constraint on the

Rishi Sonthalia

learning results in increased variance until the interpolation point. After this point, there exists a high dimensional space of solutions, and learning methods, such as gradient descent, pick solutions that generalize well. This conjecture has been empirically validated for deep neural networks. Due to the challenges of analyzing deep neural networks, theoretical understanding of this phenomenon has focused on linear models - linear regression [4-13] and kernelized regression [14-22]. These works show that there exists a peak at the boundary between the under and over-parameterized regimes . Hence validating the above postulated theory for their setting. Careful theoretical analysis then shows that the generalization error can be decomposed into various terms, one of which is the norm of the estimator. Specifically, it has been shown that the curve for the norm of the estimator versus the dimension of the data also exhibits double descent, with the peak occurring at the same point as the peak in the generalization error curve. In most cases, this is the only term in the decomposition that exhibits double descent. This leads to a second theory for the occurrence of the peak, that is, the peak in the generalization error for linear models occurs due to the norm of the estimator blowing up.

Contributions. Since understanding the reasons that peaks occur is of critical importance, it is crucial that we have a robust theory for their appearance. However, most work focuses on the overparameterized regime and ignores the under-parameterized regime. This is because it is commonly believed that the variance is monotonic in the under-parameterized regime. We show that this is not true and present two simple examples that exhibit double descent in the under-parameterized regime.

Why does the Peak Occur . We argue that the location of the peak depends on two factors: the alignment between the targets y and singular vectors V of the training data matrix and the spectrum of the data. We show that by modifying these quantities appropriately, we can move the peak into the underparameterized regime.

Modifying the Alignment . For the first example, we consider a spiked covariate model, where one eigendirection dominates, and the regression target only depends on the dominant eigendirection. For this model, we consider the ridge regularized problem with ridge parameter µ 2 and show that the ridge parameter µ 2 controls the alignment between the targets y and the singular vectors V . We show that for µ > 0 the peak occurs in the under-parameterized regime (Theorem 2). Specifically, when the ratio of the dimension to the number of training data points c is equal to (1 + µ 2 ) -1 ( c := d/n = 1 / (1 + µ 2 ) ).

Modifying the Spectrum For the second example, we consider training data that is a mixture of isotropic Gaussian vectors and vectors from along a fixed direction z . By varying the mixture proportions ( π 1 , π 2 ) , we can modify the spectrum of the covariance matrix. We show that the expected risk displays under-parameterized double descent (Theorem 4), with the peak occurring when c := d/n = π 1 ).

Norm of the estimator . We further analyze the first example and show that if we fix the number of training data points n and vary the dimension d of the problem, then for large values of µ , the risk curve does not display a double descent. However, the curve for the norm of the estimator does display descent. Thus, the peak in the norm of the estimator does not imply a peak in the generaliation error.

Organization. The rest of the paper is organized as follows. Section 2 presents a quick overview of prior work on double descent for linear models. Section 3 highlights two less-studied properties that influence the location of the peak. Section 4 presents the first of the two examples of underparameterized double descent. This model also shows that a double descent in the norm of the estimator does not translate to a double descent in the risk. Finally, Section 5 presents the second example of under-parameterized double descent.

2 Prior Work on Double Descent

In this section, we present the current prevailing theories for the occurrence of local maximums in the risk curve. Concretely, consider the following simple linear model that is a special case of the general models studied in [5, 8, 11, 23] amongst many other works. Let x i ∼ N (0 , I d ) and let β ∈ R d be a linear model with ∥ β ∥ = 1 . Let y i = β T x i + ξ i where ξ ∼ N (0 , 1) . Then, let β opt be the minimum norm solution to arg min ˜ β ∥ β T X trn -˜ β T X trn + ξ trn ∥ , where ξ trn ∈ R n × 1 . One important quantity is the aspect ratio of X . Specifically, for a d × n matrix, the aspect ratio is c := d/n . With this terminology, we see that a model is under-parameterized if c < 1 and over-parameterized when c > 1 .

Finally, the interpolation point, i.e., the point at which we can exactly fit the training data, is c = 1 , assuming we have full-rank data.

Then, the excess risk R and the expected norm of β opt can be expressed as follows:

In this model, there are a few important features that are ubiquitous in many prior double descent studies for linear models:

The peak happens at c = 1 , on the border between the under and over-parameterized regimes.

Further, at c = 1 the training error equals zero. Hence, this is the interpolation point.

The peak occurs due to the expected norm of the estimator β opt blowing up near the interpolation point.

This is further validated by works that study ridge regularized regression [23-26]. Works such as [23] have shown that optimally regularized regression no longer exhibits double descent. Further, increasing the amount of regularization from zero to the optimal amount of regularization results in the magnitude of the peak in the generalization getting smaller until a peak no longer exists. However, the location of the peak does not change by changing the amount of regularization. Further, Chen, Min, Belkin, and Karbasi [27] proved that double descent cannot take place in the under-parameterized regime for the above model.

Subsequently, works such as [10, 23, 28-31] show that there can be multiple descents in the overparameterized regime. Specifically, d'Ascoli, Sagun, and Biroli [30] show that the first peak in triple descent is due to the norm of the estimator peaking and that the second peak is due to the initialization of the random features. Their results, Figure 3 in [30], show that the peaks only occur if the model is over-parameterized. Further Chen, Min, Belkin, and Karbasi [27] show that by considering a variety of product data distributions, any shaped risk curve can be observed in the over-parameterized regime. Finally, Curth, Jeffares, and van der Schaar [32] says that the peak occurs at the point of effective dimensionality of the model and not the true dimensionality. Here, we see that there are three other reasons provided for the occurrence of peaks in the risk curve.

Regularization can reduce the effective dimensionality of the model and move the peak to the right into the over-parameterized regime [32].

For random feature models, we see that the random initialization results in a second peak in the over-parameterized regime [30].

Due to the data having a complex covariance structure, any shaped risk curve is possible in the over-parameterized regime [27].

Other works [33-36] have considered the problem for other loss models and shown a variety of different risk curves can exist. Table 1 summarizes some of the prior work.

Double Descent with Input Noise. There has also been prior work that studies double descent for models with input noise rather than output noise [24, 37, 38]. From these Sonthalia and Nadakuditi [24] and Kausik, Srivastava, and Sonthalia [37] consider the unregularized problem and show that the peak occurs at the boundary. Dhifallah and Lu [38] considers ridge regularization with isotropic Gaussian data and again sees that the peak occurs at the boundary.

3 Spectral Properties of the Data Affect the Peak Location

In this section, we identify two important spectral properties that govern the location of the peak. In later sections, we delve into two examples that modify these properties and move the peak into the under-parameterized regime. We begin with definitions and notations. Throughout the paper, we assume that training data X = [ x 1 . . . x n ] ∈ R d × n and y = [ y 1 . . . , y n ] ∈ R k × n . We are interested in the standard ridge regularized least squares problem.

In the unregularized case, the minimum norm solution is given by ˆ β T = yX † , where X † is the Moore-Penrose Pseudoinverse of X . Prior work on linear models has shown that a double descent in the risk is due to a double descent in the norm of the estimator. Suppose X = U Σ V T is the SVD, ˆ β ∈ R d × 1 , then using unitary invariance, we have that

where σ i is the i th singular value. Hence, this is controlled by

The alignment between y and V . Here V are the eigenvectors of the data gram matrix.

The spectrum of the gram matrix X T X .

Many prior works deal with the alignment in one of two ways. If y = β T X + ξ , with ξ having an isotropic distribution, then prior works either assume that β has an isotropic distribution [4, 5, 42] or they assume that X is isotropic Gaussian or that x i = ˇ Σ 1 / 2 z i , where z i is from an isotropic Gaussian [23, 43] and ˇ Σ is a deterministic matrix. For example, if β has an isotropic distribution, taking the expectation with respect to β, ξ we get that

This quantity is then studied by looking at the distribution of the spectrum as d, n →∞ .

Definition 1. Given a random matrix A , if λ 1 , . . . , λ k are its eigenvalues. Then the empirical spectral distribution (ESD) is the following sum of Dirac delta measure

and the limiting spectral distribution ν is a measure such that ν k → ν weakly almost surely.

In general, the limiting distribution ν c depends on the limiting aspect ratio ( i.e., d/n → c ) .

Definition 2. Given a measure ν c that is supported on the interval J ⊂ R , the Stieltjes transform is

One common assumption is that the limiting distribution of the empirical spectral distribution is the Marchenko-Pastur distribution [44]. Other limiting distributions have been considered in [5, 45, 46]. For the Marchenko-Pastur distribution, it is known (see, for example, Lemma 5 in [24]) that

Hence, the risk is governed by the value of the Stieltjes transform of the limiting spectrum at ζ = 0 . In particular, for the above example, the location of the peak of the risk as a function of c depended on the location of the peak of

Hence the risk depends on both the spectrum and the alignment between y and V . Thus, the peak occurs at c = 1 because of the following two conditions.

Alignment of y and right singular vectors of X

Assumption 1. If X = U Σ V T is the SVD, then yV is isotropic.

Stieltjes Transform Peak Assumption

Assumption 2. The function c ↦→ m ν c (0) =: G ( c ) has a local maximum at c = 1 .

In this paper, we show that violating either one of the above two assumptions can move the peak from the interpolation point into the under-parameterized regime.

4 Alignment Mismatch

In this section, we present the first example that exhibits double descent in the under-parameterized regime. This model violates Assumption 1.

4.1 Model Assumptions

For any k ≤ d , let β ∈ R d × k be fixed such that the operator norm || β T || is Θ(1) . Let X trn ∈ R d × n be the signal matrix and A trn ∈ R d × n be the noise matrix. Then, the ridge regularized least square estimator W opt is the minimum norm solution to

Given test data X tst + A tst , the mean squared generalization error is given by

Assumption 3. Let U ⊂ R d be a one dimensional space with a unit basis vector u . Then let X trn = σ trn uv T trn ∈ R d × n and X tst = σ tst uv T tst ∈ R d × n tst be the respective SVDs for the training data and test data matrices. We further assume that σ trn = O ( √ n ) and σ tst = O ( √ n tst ) .

There are no assumptions on the distribution of v trn , v tst besides having unit norm. First, we see that the data X + A has a spiked covariance, with the dominant eigendirection closely aligned with u . Since the targets only depend on X , we consider A to represent noise. Even with the rank 1 assumption, the model captures many different scenarios. If k = 1 , then the problem is similar to error-in-variables regression. If k = d and β = I , then this is the supervised denoising problem. If the columns of X trn are all ± u and β = u , then this captures the binary classification problem (with MSE loss) for two Gaussian clusters centered at u and -u with labels ± 1 .

Assumptions about A . The analysis works for general assumptions in [24]. For simplicity, we assume that the matrix A has I.I.D. entries drawn from a normalized Gaussian.

Assumption 4. The entries of the matrices A ∈ R d × n are I.I.D. from N (0 , 1 /d ) .

4.2 Expected Risk and Peak Location

We begin by providing a formula for the generalization error given by Equation 2 for the least squares solution given by Equation 1. All proofs are in Appendix E.

Theorem 1 (Generalization Error Formula) . Suppose the training data X trn and test data X tst satisfy Assumption 3 and the noise A trn , A tst satisfy Assumption 4. Let µ be the regularization parameter. Then for the under-parameterized regime (i.e., c < 1 ) for the solution W opt to Problem 1, the generalization error or risk given by Equation 2 is given by

Sketch. The proof proceeds in four key steps. First, we use the Sherman-Morrison formula for pseudoinverses [47]. Next, we decompose the error into constituent dependent quadratic forms. Through random matrix theory and concentration of measure arguments, we demonstrate that each quadratic form concentrates around a deterministic value characterized by the Stieltjes transform of the limiting empirical spectral distribution. Finally, we establish concentration bounds for the products and sums of these dependent forms, yielding the desired error rate.

Since the focus is on the under-parameterized regime, Theorem 1 only presents the underparameterized case. The over-parameterized case can be found in Appendix E.3. Due to the complexity of the expression, it is difficult to discern how the risk scales with respect to the training data signal strength σ 2 trn , the regularization strength µ , or the aspect ratio c . Since the focus of the paper is the scaling with respect to c , we present the connection between the risk curve and c in the main text. However, the shape of the risk curve with respect to the other parameters is also interesting and can be found in Appendix D.

To understand the shape of the risk curve as c varies, we first consider that data scaling regime . That is, fix d and change n . The following theorem 2 shows that the risk curve is theoretically guaranteed to have a peak at c = 1 1+ µ 2 .

Theorem 2 (Under-Parameterized Peak) . Let µ ∈ R > 0 , σ 2 trn = n = d/c and σ 2 tst = n tst , and d is sufficiently large, so that the error term o (1 /d ) is small, then the risk R ( c ) from Theorem 1, as a function of c , has a local maximum in the under-parameterized regime at c = 1 1+ µ 2 .

Theorem 2 contrasts with prior works, in which double descent occurs in the over-parameterized regime or on the boundary between the two regimes. We numerically verify the predictions from Theorems 1 and 2. Figure 2 shows that the theoretically predicted risk matches the numerical risk, thus verifying that double descent occurs in the under-parameterized regime. 3

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.001

0.002

0.003

0.004

0.005

Generalization Error

Under-Parameterized

Regime

Empirical

Theoretical

1

2 +1

Peak

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.00110

0.00112

0.00114

0.00116

0.00118

0.00120

Generalization Error

<- Under-Parameterized

Regime

Empirical

Theoretical

1

2 +1

Peak

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.00105

0.00106

0.00107

0.00108

0.00109

0.00110

0.00111

Generalization Error

Under

Parameterized

Regime

Empirical

Theoretical

1

2 +1

Peak

4.3 The Peak Occurs Due to Alignment Mismatch

We now show that the peak occurs due to a mismatch between the target vector and the right singular vectors of the input data. To begin, note that the ridge regularized problem can be written as follows

In this expression, y = β T ˆ X trn = ( β T u )[ v T trn 0 p ] . Hence the direction is given by ˆ v T trn = [ v T trn 0 p ] . The right singular vectors of ˆ X trn + ˆ A trn are more difficult to compute,thus we use proxies. Since ˆ X trn is rank 1, we use the right singular vectors of ˆ A trn as a proxy. Lemma 5 in the appendix, shows that if A = U Σ V T , then we can express ˆ A trn as ˆ U ˆ Σ ˆ V T , where ˆ U = U , ˆ Σ 2 = Σ 2 + µ 2 I , and ˆ V = [ V 1: p Σ ˆ Σ -1 µU ˆ Σ -1 ] ∈ R n + d × d . Here V 1: d are the first d columns of V . Then,

Since V 1: d came from a Gaussian random matrix, (ˆ v T trn V 1: d ) has isotropic entries. However, the diagonal matrix Σ ˆ Σ -1 results in the entries of y ˆ V not being isotropic. Note when µ = 0 , Σ ˆ Σ -1 = I , hence it is isotropic. Hence, µ controls the deviation from isotropy.

We use ˆ Σ T ˆ Σ I as a proxy for the spectrum of the sample covariance. By Lemma 5, we have that ˆ Σ T ˆ Σ = Σ T Σ+ µ 2 . Weknow that the limiting spectrum for Σ T Σ is the Marchenko-Pastur distribution for which the map G ( c ) = m ν c (0) has a maximum for c = 1 . Shifting the spectrum changes the magnitude of the peak but does not change the location. This suggests that the peak occurs due to the misalignment between the target vector and the right singular vectors of the input data.

Ablation experiment To verify that the location of the peak is due to the misalignment, we conduct two experiments. First, we solve the unregularized problem. However, we change the spectrum of the noise matrix A trn . That is, instead of using A trn = U Σ V T , we use ˜ A = U (Σ 2 + µ 2 I ) 1 / 2 V T . If the spectrum was the primary factor determining the location of the peak, the peak should occur at c = 1 / (1 + µ 2 ) . However, as seen in Figure 3a it still occurs at c = 1 . Second, we replace ˆ V with a uniformly random orthogonal matrix Q 4 . Clearly, yQ is now isotropic. Figure 3b shows that, in this case, the peak moves to the over-parameterized regime.

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

1

c

10 1

10 2

Generalization Risk

= 1

0.4

0.6

0.8

1.0

1.2

c

10 1

10 2

10 3

Generalization Risk

= 1

Connection to Prior Double Descent Theory Prior double descent theory postulates that the peak for the ridge regularized model occurs at the interpolation point for the unregularized model or further to the right into the over-parameterized regime. Hence, this model goes against prior expectations, with the peak moving to the left into the under-parameterized regime. However, there might still be some connection between the training error and the location of the peak. For example, the peak may correspond to a local minimum of the training loss. We explore this in Appendix C and see only a weak connection with the third derivative of the training error.

4.4 Peak in the Norm of the Estimator Does Not Imply a Peak in the Risk

Prior double descent theories suggest that double descent occurs due to the norm of the estimator increasing and then decreasing. This is true for the above model where we fixed d and varied n . However, the connection breaks if we fix n and vary d instead ( parameter scaling regime ). This difference between the two regimes is due to the normalization considered and has been observed before [30].

Figure 4 shows that for the parameter scaling regime, for small values of µ , we see underparameterized double descent. However, as we increase µ , the risk curve becomes monotonic. Nevertheless, as shown in Figure 5, for larger values of µ , there is still a peak in the curve for the norm of the estimator ∥ W opt ∥ 2 F . Hence, the curve for the norm of the estimator exhibits underparameterized double descent even if the risk does not. This is further highlighted in Figure 6. Here, we see that even though the variance is non-monotonic, the risk is dominated by the bias term. Thus, we show that a peak in the generalization error for linear models does not imply a peak in the norm of the estimator. The following theorem provides a local maximum in the E [ ∥ W opt ∥ 2 F ] curve for c < 1 .

Theorem 3 ( ∥ W opt ∥ F Peak) . If σ tst = √ n tst , σ trn = √ n and µ is such that p ( µ ) < 0 , then for fixed n that is sufficiently large enough, we have that E [ ∥ W opt ∥ F ] versus c = d/n curve has a local maximum in the under-parameterized regime at c = ( µ 2 +1) -1 .

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.002

0.004

0.006

0.008

0.010

Generalization Error

Under

Parameterized

Regime

Empirical

Theoretical

1

2 +1

Peak

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.002

0.004

0.006

0.008

0.010

Generalization Error

Under

Parameterized

Regime

Empirical

Theoretical

1

2 +1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.002

0.004

0.006

0.008

0.010

Generalization Error

Under

Parameterized

Regime

Empirical

Theoretical

1

2 +1

1

2

3

4

5

W

2

F

0.004

0.005

0.006

0.007

0.008

0.009

0.010

0.011

Generalization Error

0.0

0.2

0.4

0.6

0.8

1.0

c

(a)

µ

= 0

.

1

1.075

1.100

1.125

1.150

1.175

1.200

W

2

F

0.002

0.004

0.006

0.008

0.010

Generalization Error

0.0

0.2

0.4

0.6

0.8

1.0

c

(b)

µ

= 1

1.020

1.025

1.030

1.035

1.040

W

2

F

0.002

0.004

0.006

0.008

0.010

Generalization Error

0.0

0.2

0.4

0.6

0.8

1.0

c

(c)

µ

= 2

5 Shifting Local Maximum for Stieltjes Transform as a Function of c

In this section, we present the next example of under-parameterized double descent that violates Assumption 2. In particular, the maximum of the map c ↦→ m ν c (0) does not occur at c = 1 . We show that the maximum can be chosen to be any value in (0 , 1) . We consider the following mixture model. Let π 1 , π 2 be mixture weights such that π 1 + π 2 = 1 . Then, with probability π 1 , the data is sampled from N (0 , 1 d I ) and with probability π 2 , the data point is αz for fixed z ∈ R d and α ∼ N (0 , 1) . For this model, the uncentered covariance matrix is given by

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

0.00000

0.00002

0.00004

0.00006

0.00008

0.00010

0.00012

(

I

W

)

Xtst

2

F

Ntst

- Bias

<- Under Parameterized

Regime

Empirical

Theoretical

1

2 +1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

1.06

1.08

1.10

1.12

1.14

1.16

1.18

1.20

W

2

F

<- Under Parameterized

Regime

Emprircal

Theory

1

2 +1

(b)

∥

W

opt

∥

2

F

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.002

0.004

0.006

0.008

0.010

Generalization Error

Under Parameterized

Regime

Emprircal

Theory

1

2 +1

(c) Generalization Error

Then, the expected excess risk for a solution ˆ β compared to β is

Let X trn = [ A zv T ] ∈ R d × n , where the A ∈ R d × n -k with each column a data point sampled I.I.D. from N (0 , 1 d I ) and v ∈ R k is the vector with random coefficients in front of z . Let β ∈ R d be the target regressor function and let y T = β T X trn + ξ T trn , where ξ T trn has I.I.D. entries from a standard normal. Let β T opt = y T X T trn ( X trn X T trn ) -1 be the minimum norm. Then, Theorem 4 shows that the peak occurs when d/n = c = π 1 < 1 . To experimentally verify Theorem 4, we consider two cases, one where we enforce β ⊥ z and one where we do not. As seen in Figure 7, Theorem 4 is accurate for both cases. This suggests that the β ⊥ z assumption seems to only be needed for simplifying the proof.

0.4

0.6

0.8

1.0

1.2

d

Ntrn

10 0

10 1

10 2

10 3

Risk

Emprirical Risk

Theoretical Risk

0.4

0.6

0.8

1.0

1.2

d

Ntrn

10 0

10 1

10 2

10 3

Risk

Emprirical Risk

Theoretical Risk

Theorem 4 is quite surprising as it shows that the only peak in the risk curve occurs in the underparameterized regime. One might assume that is due to the low rankness of the data from the second mixture. While this is true, prior work does not indicate that this is the reason. Specifically, Huang, Hogg, and Villar [41] shows that projecting onto low-dimensional versions of the data acts as a regularizer and removes the peak altogether. Xu and Hsu [39], also looks at a similar problem, but they consider isotropic Gaussian data and project onto the first p components. In this case, the data is artificially high-dimensional (since only the first k coordinates are non-zero).

They again see a peak at the interpolation point ( n = p ). Wu and Xu [40] also looks at a version of Principal Component Regression in which the data dimension is reduced. That is, the data is not embedded in high-dimensional space anymore. Wu and Xu [40] sees a peak at the boundary between the under and over-parameterized regions. Finally, Sonthalia and Nadakuditi [24] and Kausik, Srivastava, and Sonthalia [37] look at the denoising problem for low-dimensional data and have peaks at c = 1 . Therefore, prior work does not immediately imply that low dimensional data results in under-parameterized double descent. If we had only low-dimensional data, then the peak 'should' move into the over-parameterized regime. This is because if the true dimensionality of the data is r < d . Then, one might think that the peak occurs when the number of training data points n equals r since that is the interpolation point 5 . We are in the over-parameterized regime since d > r = n .

We can understand this phenomenon as follows. The data from the second mixture does not affect the smallest eigenvalue of the covariance matrix. This is because the second mixture lives in a one-dimensional space. Hence, it only affects the top eigenvalue. Since the Stieltjes transform at 0 is dominated by the behavior of the smallest non-zero eigenvalue, data from the second mixture has very little effect on the Stieltjes transform of the ESD at 0. We expect the above intuition to hold, even when replacing rank 1 with rank r for any fixed small r .

Connection to Prior Double Descent Theory For this example, it is easy to show that there is a strong connection to prior double descent theory. Specifically, even though we cannot interpolate the data, the minimum training error will occur at c = π 1 . Further, we see that the blow-up in the excess risk is due to the norm of the estimator blowing up.

Additionally, we see that comparing with the result from [11] (which is the case when π 2 = 0 ), we see that the π 2 > 0 results in sifting the peak to π 1 and rescales the variance by π 1 . However, we also see an additional correction term in the overparameterized regime:

Here we see that the term depends on the alignment between the target β and the spike direction z .

6 Conclusion

This paper presents two simple models with double descent in the under-parameterized regime. While such peaks seem limited to special cases, understanding the cause is important for a complete understanding of the double descent phenomenon. Our analysis reveals that the location of peaks depends critically on two properties: the alignment between targets and the eigenvectors of the training data gram matrix and the behavior of the Stieltjes transform of the limiting empirical spectral distribution.

Wedemonstrate that violating either of these properties can shift the peak into the under-parameterized regime. The first model shows that ridge regularization can create a misalignment between targets and singular vectors, leading to a peak at c = 1 / (1 + µ 2 ) . The second model, using a mixture of isotropic Gaussian vectors and directional vectors, demonstrates that modifying the spectrum can result in a peak at c = π 1 . These findings challenge several prevailing theories about double descent. They show that peaks need not occur at or beyond the interpolation threshold and that a peak in the estimator's norm does not necessarily imply a peak in the generalization error.

Investigating the interaction between spectral properties and generalization in deep neural networks to provide a general theory of double descent is an important avenue for future work. Understanding whether similar phenomena occur in other architectures and the implications for model selection and regularization remain open questions.

References

Scott Fortmann-Roe. Understanding the Bias-Variance Tradeoff, June 2012. URL: http: //scott.fortmann-roe.com/docs/BiasVariance.html (cited on page 1).

Manfred Opper and Wolfgang Kinzel. Statistical Mechanics of Generalization. Models of Neural Networks III: Association, Generalization, and Representation , 1996 (cited on page 1).

Mikhail Belkin, Daniel J. Hsu, Siyuan Ma, and Soumik Mandal. Reconciling Modern MachineLearning Practice and the Classical Bias-Variance Trade-off. Proceedings of the National Academy of Sciences , 2019 (cited on page 1).

Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional Dynamics of Generalization Error in Neural Networks. Neural Networks , 2020 (cited on pages 2, 4).

Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: ridge regression and classification. The Annals of Statistics , 2018 (cited on pages 2-4).

Gabriel Mel and Surya Ganguli. A Theory of High Dimensional Regression with Arbitrary Correlations Between Input Features and Target Functions: Sample Complexity, Multiple Descent Curves and a Hierarchy of Phase Transitions. In Proceedings of the 38th International Conference on Machine Learning , 2021 (cited on page 2).

Vidya Muthukumar, Kailas Vodrahalli, and Anant Sahai. Harmless Interpolation of Noisy Data in Regression. 2019 IEEE International Symposium on Information Theory (ISIT) , 2019 (cited on page 2).

Peter Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign Overfitting in Linear Regression. Proceedings of the National Academy of Sciences , 2020 (cited on pages 2, 3).

Mikhail Belkin, Daniel J. Hsu, and Ji Xu. Two Models of Double Descent for Weak Features. SIAM Journal on Mathematics of Data Science , 2020 (cited on page 2).

Michal Derezinski, Feynman T Liang, and Michael W Mahoney. Exact Expressions for Double Descent and Implicit Regularization Via Surrogate Random Design. In Advances in Neural Information Processing Systems , 2020 (cited on pages 2, 3).

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in HighDimensional Ridgeless Least Squares Interpolation. The Annals of Statistics , 2022 (cited on pages 2, 3, 10).

Bruno Loureiro, Gabriele Sicuro, Cedric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborova. Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions. In Advances in Neural Information Processing Systems , 2021 (cited on page 2).

Chen Cheng and Andrea Montanari. Dimension Free Ridge Regression. arXiv preprint arXiv:2210.08571 , 2022 (cited on page 2).

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization Error of Random Feature and Kernel Methods: Hypercontractivity and Kernel Matrix Concentration. Applied and Computational Harmonic Analysis , 2022 (cited on page 2).

Song Mei and Andrea Montanari. The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve. Communications on Pure and Applied Mathematics , 75, 2021 (cited on page 2).

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A Mean Field View of the Landscape of Two-layer Neural Networks. Proceedings of the National Academy of Sciences of the United States of America , 2018 (cited on page 2).

Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate Shift in High-Dimensional Random Feature Regression. ArXiv , 2021 (cited on page 2).

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. Generalisation Error in Learning with Random Features and the Hidden Manifold Model. In Proceedings of the 37th International Conference on Machine Learning , 2020 (cited on page 2).

Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and Rich Regimes in Overparametrized Models. In Proceedings of Thirty Third Conference on Learning Theory , 2020 (cited on page 2).

Bruno Loureiro, C'edric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc M'ezard, and Lenka Zdeborov'a. Learning Curves of Generic Features Maps for Realistic Datasets with a Teacher-Student Model. In NeurIPS , 2021 (cited on page 2).

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of Lazy Training of Two-layers Neural Network. In Advances in Neural Information Processing Systems , 2019 (cited on page 2).

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When Do Neural Networks Outperform Kernel Methods? In Advances in Neural Information Processing Systems , 2020 (cited on page 2).

Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, and Tengyu Ma. Optimal Regularization can Mitigate Double Descent. In International Conference on Learning Representations , 2020 (cited on pages 2-4, 17).

Rishi Sonthalia and Raj Rao Nadakuditi. Training data size induced double descent for denoising feedforward neural networks and the role of training noise. Transactions on Machine Learning Research , 2023 (cited on pages 3-5, 10, 17, 20, 21, 24, 26, 31).

Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The Optimal Ridge Penalty for RealWorld High-Dimensional Data Can Be Zero or Negative Due to the Implicit Ridge Regularization. Journal of Machine Learning Research , 2022 (cited on page 3).

Fatih Furkan Yilmaz and Reinhard Heckel. Regularization-Wise Double Descent: Why it Occurs and How to Eliminate it. In IEEE International Symposium on Information Theory , 2022 (cited on page 3).

Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple Descent: Design Your Own Generalization Curve. Advances in Neural Information Processing Systems , 2021 (cited on page 3).

Lechao Xiao and Jeffrey Pennington. Precise learning curves and higher-order scaling limits for dot product kernel regression. arXiv preprint arXiv:2205.14846 , 2022 (cited on page 3).

Ben Adlam and Jeffrey Pennington. The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. In International Conference on Machine Learning , 2020 (cited on page 3).

Stéphane d'Ascoli, Levent Sagun, and Giulio Biroli. Triple Descent and the Two Kinds of Overfitting: Where and Why Do They Appear? In Advances in Neural Information Processing Systems , 2020 (cited on pages 3, 8).

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized Two-layers Neural Networks in High Dimension. The Annals of Statistics , 2021 (cited on page 3).

Alicia Curth, Alan Jeffares, and Mihaela van der Schaar. A u-turn on double descent: rethinking parameter counting in statistical learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL: https://openreview.net/forum?id= O0Lz8XZT2b (cited on page 3).

Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high-dimensional binary linear classification. Information and Inference: A Journal of the IMA , 11(2):435-495, 2022 (cited on page 3).

Ganesh Ramachandra Kini and Christos Thrampoulidis. Analytic study of double descent in binary classification: the impact of loss. In 2020 IEEE International Symposium on Information Theory (ISIT) , pages 2527-2532. IEEE, 2020 (cited on page 3).

Pragya Sur and Emmanuel J Candès. A modern maximum-likelihood theory for highdimensional logistic regression. Proceedings of the National Academy of Sciences , 116(29):14516-14525, 2019 (cited on page 3).

Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy gaussian mixture. In International conference on machine learning , pages 6874-6883. PMLR, 2020 (cited on page 3).

Chinmaya Kausik, Kashvi Srivastava, and Rishi Sonthalia. Generalization Error without Independence: Denoising, Linear Regression, and Transfer Learning, 2023 (cited on pages 3, 4, 10, 42).

Oussama Dhifallah and Yue Lu. On the inherent regularization effects of noise injection during training. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2665-2675. PMLR, 18-24 Jul 2021. URL: https://proceedings.mlr.press/ v139/dhifallah21a.html (cited on pages 3, 4, 14).

Ji Xu and Daniel J Hsu. On the number of variables to use in principal component regression. Advances in neural information processing systems , 2019 (cited on pages 3, 9).

Denny Wu and Ji Xu. On the Optimal Weighted \ ell _ 2 Regularization in Overparameterized Linear Regression. Advances in Neural Information Processing Systems , 2020 (cited on pages 3, 10).

Ningyuan Huang, David W. Hogg, and Soledad Villar. Dimensionality reduction, regularization, and generalization in overparameterized regressions. SIAM Journal on Mathematics of Data Science , 2022 (cited on pages 3, 9).

Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee. In International Conference on Learning Representations , 2020 (cited on page 4).

Yutong Wang, Rishi Sonthalia, and Wei Hu. Near-interpolators: rapid norm growth and the trade-off between interpolation and generalization. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 44834491. PMLR, February 2024. URL: https://proceedings.mlr.press/v238/ wang24k.html (cited on page 4).

Vladimir Marcenko and Leonid Pastur. Distribution of Eigenvalues for Some Sets of Random Matrices. Mathematics of The Ussr-sbornik , 1967 (cited on pages 4, 28).

Lucas Benigni and Sandrine Péché. Eigenvalue distribution of some nonlinear models of random matrices. Electronic Journal of Probability , 26:1-37, 2021 (cited on page 4).

Olivier Ledoit and Sandrine Péché. Eigenvectors of some large sample covariance matrix ensembles. Probability Theory and Related Fields , 151(1):233-264, 2011 (cited on page 4).

Carl D. Meyer Jr. Generalized Inversion of Modified Matrices. SIAM Journal on Applied Mathematics , 1973 (cited on pages 6, 22, 24).

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: an imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL: http://papers.neurips.cc/ paper/9015-pytorch-an-imperative-style-high-performance-deeplearning-library.pdf .

Chris M. Bishop. Training with Noise is Equivalent to Tikhonov Regularization. Neural Computation , 1995 (cited on page 17).

Friedrich Götze and Alexander Tikhomirov. The Rate of Convergence for Spectra of GUE and LUE Matrix Ensembles. Central European Journal of Mathematics , 2005 (cited on page 28).

Friedrich Götze and Alexander Tikhomirov. Rate of Convergence to the Semi-Circular Law. Probability Theory and Related Fields , 2003 (cited on page 28).

Friedrich Götze and Alexander Tikhomirov. Rate of Convergence in Probability to the Marchenko-Pastur Law. Bernoulli , 2004 (cited on page 28).

Z. Bai, Baiqi. Miao, and Jian-Feng. Yao. Convergence Rates of Spectral Distributions of Large Sample Covariance Matrices. SIAM Journal on Matrix Analysis and Applications , 2003 (cited on page 28).

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.02

0.03

0.04

0.05

0.06

0.07

Generalization Error

Under Parameterized

Regime

Emprircal

Theory

1

2 +1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.0071

0.0072

0.0073

0.0074

0.0075

0.0076

0.0077

Generalization Error

Under Parameterized

Regime

Emprircal

Theory

1

2 +1

0.0

0.5

1.0

1.5

2.0

2.5

c

=

d

/

Ntrn

10 0

10 1

10 2

Generalization Error

A Higher Rank for Denoising Model

One might be led to believe that the restriction that the data lie on a line embedded in high dimensional space is crucial to the appearance of this phenomenon. However, this is not true. As long as the rank of the data is relatively low, for the input noise setting, we can see this phenomenon. Hence, we extend our results beyond the one-dimensional case to the low-dimensional case. Due to space constraints, the conjectured formula for the risk is in Appendix F. 6 We verify the conjectured formula as well as the role of low dimensionality using MNIST data. Specifically, we project the data onto a r -dimensional subspace. We then add noise to the low-dimensional representation and then solve the denoising problem. The left two figures in Figure 8 show that the phenomenon exists for low-dimensional data. That is, we see that a peak occurs in the under-parameterized regime, and the location of the peak seems to occur at 1 µ 2 +1 . However, running the same experiment without projecting to a low-dimensional space (right figure) results in very different phenomena. We no longer see double descent at all. Hence we see that if a peak occurs, then its location does not depend on the dimension. However, the occurrence of the peak does depend on the dimension. Thus, we see that this complements the results in [38].

0.5

1.0

1.5

2.0

2.5

3.0

c

=

d

/

Ntrn

6

7

8

9

Generalization Error

1e

8+1.271e

3

0.5

1.0

1.5

2.0

2.5

3.0

c

=

d

/

Ntrn

1.22 × 10

3

1.23 × 10

3

1.24 × 10

3

1.25 × 10

3

1.26 × 10

3

1.27 × 10

3

Train Error

B Non-Linear Model

To show that under-parameterized peak occurs for non-linear models. We conducted an experiment on MNIST with a 1-hidden layer neural network with a width of 784. The network has no bias, so has 2 · 784 2 parameters. Let V be a random 10-dimensional subspace. We project our data onto V , add noise to the low dimensional representation, and train our network to remove this noise. We use full batch gradient descent with weight decay of 0 . 001 to train the model for 500 epochs with a learning rate of 2 × 10 -2 . We test it on the complete MNIST test data. Figure 9 shows the training error and generalization error as a function of the number of training data points. As seen in the figure, we have multiple peaks in the under-parameterized regime, and the peaks in the risk correspond to local minimums in the training error. Interestingly, the risk curve here exhibits 4 peaks!

There are, however, many crucial differences between the peaks for this neural network case and the linear model case. First, the location of the peak does not seem to depend on the strength of the ridge regularization. Second, the peak seems to directly correspond to the local minimums of the training error. Hence, while we see peaks in the under-parameterized regime, the mechanisms that create these peaks are likely to be different.

0.0

0.2

0.4

0.6

0.8

1.0

c

=

d

/

Ntrn

0.002

0.000

0.002

0.004

0.006

Empirical

Training Error

Training Error 3rd Derivative

1

2 +1

Minimum

0.0

0.2

0.4

0.6

0.8

1.0

c

=

d

/

Ntrn

0.00

0.02

0.04

0.06

0.08

Empirical

Training Error

Training Error 3rd Derivative

1

2 +1

Minimum

0

2

4

6

8

10

2

0.2

0.4

0.6

0.8

1.0

c

Minimum of 3rd Derivative

1

2 +1

2 =1

C Training Error

As seen in the prior section, the peak occurs in the interior of the under-parameterized regime and not on the border between the under-parameterized and over-parameterized regimes. We have also seen that it does not necessarily occur whenever there is a peak in the norm of the estimator. The final postulate for where the peak occurs is that it occurs when we first hit zero training error.

In this section, we explore the connection between the training error and the risk. Theorem 5 derives a formula for the training error in the under-parameterized regime.

Theorem 5 (Training Error) . Let τ be as in Theorem 1. The training error for c < 1 is given by

Since we are studying the ridge regularized problem, it is impossible for the training error to be exactly equal to 0. Hence, we may expect the peak to correspond to other features of the training error curve. Given the analytical formula for the training error, we can compute the derivatives. We found that the training error curve does not seem to signal the location of the peak in the generalization error curve.

Since we are studying the ridge regularized problem, it is impossible for the training error to be exactly equal to 0. Hence, we may expect the peak to correspond to other features of the training error curve. For example, the peak could correspond to a local minimum of the training error, or it could correspond to a point where the training error or its derivatives suddenly change. The first, a local minimum, is easy to see from the plot of the training error, but the second can be more subtle as we do not usually have access to the derivatives. However, since we have an analytical formula for the training error, we can compute the derivatives.

Figure 10 plots the location of the peak and the training error. Here, the figure shows that the training error curve does not seem to signal the location of the peak in the generalization error curve. However, it shows that for the data scaling regime, the peak roughly corresponds to a local minimum of the third derivative of the training error. While the minimum of the third derivative is difficult to interpret, as we can see from the third plot in Figure 10, the minimum seems to closely track the location of the peak.

D Regularization Trade-off

It has been seen in prior work that the amount of noise added to the input data can be viewed as a regularizer [24, 49]. In our setup, we have two different regularizers: the amount of noise added to the data (since we are dealing with linear models, this is equivalent to the strength of the signal σ trn ) and the strength of the ridge regularizer µ . It is interesting to analyze the trade-off between the two regularizers and the generalization error.

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

trn

/

Ntrn

0.001205

0.001210

0.001215

0.001220

0.001225

0.001230

0.001235

0.001240

Generalization Error

Empirical

Theoretical

Minimum

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

trn

/

Ntrn

0.00193

0.00194

0.00195

0.00196

0.00197

0.00198

0.00199

Generalization Error

Empirical

Theoretical

Minimum

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.00110

0.00112

0.00114

0.00116

0.00118

0.00120

Generalization Error

<- Under

Parameterized

Regime

Empirical

Theoretical

1

2 +1

Peak

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.002

0.004

0.006

0.008

0.010

Generalization Error

Under

Parameterized

Regime

Empirical

Theoretical

1

2 +1

Peak

Optimal σ trn First, we fix µ and determine the optimal σ trn . Figure 11 displays the generalization error versus σ 2 trn curve. The figure shows that the error is initially large but then decreases until the optimal generalization error. The generalization error when using the optimal σ trn is also shown in Figure 11. Here, unlike [23], picking the optimal value of σ trn does not mitigate double descent.

Proposition 1 (Optimal σ trn ) . The optimal value of σ 2 trn for c < 1 is given by

Additionally, it is interesting to determine how the optimal value of σ trn depends on both µ and c . Figure 12 shows that for small values of µ ∈ (0 . 1 , 0 . 5) , as c changes, there exists an (inverted) double descent curve for the optimal value of σ trn . However, for the data scaling regime, the minimum of this double descent curve does not match the location for the peak of the generalization error . Further, as the amount of ridge regularization increases, the optimal amount of noise regularization decreases proportionally; optimal σ 2 trn ≈ dµ 2 . Thus, for higher values of ridge regularization, it is preferable to have higher-quality data.

Optimal Value of µ We now explore the effect of fixing σ trn and then changing µ . Figure 13, shows a U shaped curve for the generalization error versus µ , suggesting that there is an optimal value of µ , which should be used to minimize the generalization error. Next, we compute the optimal value of µ using grid search and plot it against c . Figure 14 shows double descent for the optimal value of µ for small values of σ trn . Thus, for low SNR data, we see a double descent, but we do not for high SNR data.

Finally, for a given value of µ and c , we compute the optimal σ trn . Wethen compute the generalization error (when using the optimal σ trn ) and plot the generalization error versus µ curve. Figure 15 displays a very different trend from Figure 13. Instead of having a U -shaped curve, we have a

0

50

100

150

200

250

300

350

400

2

0

100

200

300

400

500

600

700

800

Optimal

2

trn

/

Ntrn

c

=0.10, slope = 0.10

c

=0.29, slope = 0.30

c

=0.48, slope = 0.50

c

=0.67, slope = 0.71

c

=0.86, slope = 0.91

c

=1.05, slope = 1.11

c

=1.24, slope = 1.30

c

=1.43, slope = 1.49

c

=1.62, slope = 1.68

c

=1.81, slope = 1.87

c

=2.00, slope = 2.06

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

1.0

1.5

2.0

2.5

3.0

optimal

2

trn

/

Ntrn

Risk peak

Minimum

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

c

=

d

/

Ntrn

0.50

0.75

1.00

1.25

1.50

1.75

2.00

optimal

2

trn

/

Ntrn

Risk peak

Minimum

0

2

4

6

8

10

0.002

0.004

0.006

0.008

0.010

Generalization Error

Theoretical

Empirical

Minimum

(a)

c

= 0

.

5

0

2

4

6

8

10

0.002

0.004

0.006

0.008

Generalization Error

Theoretical

Empirical

Minimum

(b)

c

= 2

0

1

2

3

4

5

c

0.1

0.2

0.3

0.4

0.5

Optimal

(a)

σ

trn

≈

0

.

2

0

1

2

3

4

5

c

0.2

0.4

0.6

0.8

1.0

Optimal

(b)

σ

trn

≈

0

.

3

0

1

2

3

4

5

c

1

2

3

4

5

Optimal

(c)

σ

trn

≈

1

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

10

3

1.2 × 10

3

1.4 × 10

3

1.6 × 10

3

1.8 × 10

3

2×10 3

Generalization Risk

c = 0.5

(a)

c

= 0

.

5

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

10

3

1.2 × 10

3

1.4 × 10

3

1.6 × 10

3

1.8 × 10

3

Generalization Risk

c = 2

(b)

c

= 2

monotonically decreasing generalization error curve. This suggests that we can improve generalization by using higher-quality training while compensating for this by increasing the amount of ridge regularization.

Interaction Between the Regularizers The optimal values of µ and σ trn are jointly computed using grid search for µ ∈ (0 , 100] and σ trn / √ n ∈ (0 , 10] . Figure 16 shows the results. Specifically, σ trn is at the highest possible value (so best quality data), and then the model regularizes purely using the ridge regularizer. This results in a monotonically decreasing generalization error curve. Thus, in the data scaling model, there is an implicit bias that favors one regularizer over the other. Specifically, the model's implicit bias is to use higher quality data while using ridge regularization to regularize the model appropriately . It is surprising that the two regularizers are not balanced.

0

2

4

6

8

10

1

c

=

Ntrn

d

9.6

9.8

10.0

10.2

10.4

Optimal

trn

/

Ntrn

0

2

4

6

8

10

1

c

=

Ntrn

d

5

10

15

20

25

30

Optimal

0

2

4

6

8

10

1

c

=

Ntrn

d

0.0009990

0.0009991

0.0009992

0.0009993

0.0009994

Generalization Error

0

1

2

3

4

5

c

=

d

Ntrn

4

5

6

7

8

9

10

Optimal

trn

0

1

2

3

4

5

c

=

d

Ntrn

6

7

8

9

10

Optimal

0

1

2

3

4

5

c

=

d

Ntrn

0.000

0.002

0.004

0.006

0.008

0.010

Generalization Error

(a)

0

.

1

µ

10

and

0

.

1

σ

trn

10

0

1

2

3

4

5

c

=

d

Ntrn

9.70

9.75

9.80

9.85

9.90

9.95

10.00

Optimal

trn

0

1

2

3

4

5

c

=

d

Ntrn

5

10

15

20

25

30

Optimal

0

1

2

3

4

5

c

=

d

Ntrn

0.000

0.002

0.004

0.006

0.008

0.010

Generalization Error

(b)

0

.

1

µ

100

and

0

.

1

σ

trn

10

Next we look at the trade-off between σ trn and µ for the parameter scaling regime. We again see, Figure 17, that the model implicitly prefers regularizing via ridge regularization and not via input data noise regularizer.

E Proofs

E.1 Linear Regression

We begin by noting,

Thus, we have,

Taking the expectation, with respect to ξ trn , we see that the last term vanishes.

Letting X trn = U X Σ X V T X . Wesee that using the rotational invariance of X , U X , V X are independent and uniformly random. Thus, s := β T opt U X is a uniformly random unit vector.

Thus, we see,

Similarly, we see,

Multiplying and dividing by d , normalizes the singular values squared of X trn so that the limiting distribution is the Marchenko Pastur distribution with shape c . Thus, we can estimate using Lemma 5 from Sonthalia and Nadakuditi [24] to get,

Finally, the cross-term has an expectation equal to zero. Thus,

The second term has an expectation equal to zero, and the first term is similar to before and has an expectation equal to min ( 1 , 1 c ) .

E.2 Proof for Output Noise Model

Theorem 4 (Under-parametrized Peak) . For the above model, if k/n → π 2 , and d/n → c , then the expected risk is given by R = { π 1 c π 1 -c c < π 1 π 1 ( π 1 c -π 1 + ( 1 -π 1 c ) ( ∥ β ∥ 2 d -( β T z ) 2 ∥ z ∥ 2 d )) c > π 1 .

Proof. We begin with c < π 1 . Here, since AA T is invertible, we can see that

Let us focus on the term without the ξ . Using the Sherman Morrison formula and the orthogonality of β and z , we see that

Then we have,

Thus, we get that

For this model, the uncentered covariance matrix is given by

Then, the expected excess risk for a solution ˆ β compared to β is

The first term is given by

Taking the expectation over ξ , we get that the expected excess risk is given by

Again, using the Sherman-Morrison formula, we get that

Suppose ν is the limiting distribution of the empirical spectral distribution for the non-zero eigenvalues.

Then using the concentration results from [24], we see that the error can be expressed

The second term of the expected risk is

Again, if we take the expectation over ξ and write as a trace, we get

Again, this can expressed as

Putting it all together, the expected excess risk is

If we then send n, k, d to infinity, while noting that ∥ v ∥ → ∞ , then we get then limiting risk

For the d > n -k case, AA T is no longer invertible. Hence, we need to replace the inverse with pseudoinverse. Hence, we have that

We now expand the pseudoinverse using Theorem 1 from [47]. To write is succinctly, let M = AA T , P = ( I -MM † ) , γ = z T Pz , and τ = z T M † z to get that

Note that P is the projection onto the orthogonal complement of the range of A . Hence we see that MP = M † P = 0 . Thus, multiplying through, and setting terms to zero, we see that

Using the fact that z T Pz = γ , z T M † z = τ and cancelling, we get

Thus, the first two terms in the error are

Let us look at the second term first. Here we see that

Thus, we see that

For the first term, we first note that

To compute the norm, we expand and get

Noting that P is a projection matrix, so PP = P and using z T PZ = γ , we get that this term simplifies to

Note that the subspace for P comes from a Gaussian random matrix. Hence is uniformly random. Hence

Similarly,

For the next two terms, we need to first simplify

Substituting in our formulas and noticing that

we get that

Then we have that

Using the cycling invariance of the trace and the fact that M † P = 0 , we get that

The last equality is obtained using cyclic invariance, the fact that P 2 = P , and γ = z T Pz . Using Lemma 6 for p = n -k and q = d , we get that asymptotically,

Note that we similarly get that asymptotically,

Thus, the contribution to the risk from this term is π 2 1 c -π 1 .

Finally, for the last term, we have that

Thus,

Note that ∥ v ∥ 2 ≈ k and hence this term is aympotitcally zero. Putting it all together, we get the needed result.

E.3 Proofs for Theorem 1

The proof structure closely follows that of [24].

E.3.1 Step 1: Decompose the error into bias and variance terms.

First, we decompose the error. Since we are not in the supervised learning setup, we do not have standard definitions of bias/variance. However, we will call the following terms the bias/variance of the model. First, we recall the following from [24].

Lemma 1 (Sonthalia and Nadakuditi [24]) . If A tst has mean 0 entries and A tst is independent of X tst and W , then

E.3.2 Step 2: Formula for W opt

Here, we compute the explicit formula for W opt in Problem 1. Let ˆ A trn = [ A trn µI ] , ˆ X trn = [ X trn 0] , and ˆ Y trn = ˆ X trn + ˆ A trn . Then solving arg min W ∥ X trn -WY trn ∥ 2 F + µ 2 ∥ W ∥ 2 F is equivalent to solving arg min W ∥ ˆ X trn -W ˆ Y trn ∥ 2 F . Thus, W opt = arg min W ∥ ˆ X trn -W ˆ Y trn ∥ 2 F = ˆ X trn ˆ Y † trn . Expanding this out, we get the following formula for ˆ W . Let ˆ u be the left singular vector and ˆ v trn be the right singular vectors of ˆ X trn . Note that the left singular does not change after ridge regularization, so ˆ u = u . Let ˆ h = ˆ v T trn ˆ A † trn , ˆ k = ˆ A † trn u , ˆ s = ( I -ˆ A trn ˆ A † trn ) u , ˆ t = ˆ v trn ( I -ˆ A † trn ˆ A trn ) , ˆ γ = 1 + σ trn ˆ v T trn ˆ A † trn u , ˆ τ = σ 2 trn ∥ ˆ t ∥ 2 ∥ ˆ k ∥ 2 + ˆ γ 2 .

̸

Proposition 2. If ˆ γ = 0 and A trn has full rank then

Proof. Here we know that u is arbitrary. We have that ˆ A trn has full rank. Thus, the rank of ˆ A trn is d , and the range of ˆ A trn is the whole space. Thus, u lives in the range of ˆ A trn . In this case, we want Theorem 3 from [47]. We define

Then we have,

Note that, by our assumptions, we have ˆ t = ˆ v trn ( I -ˆ A † trn ˆ A trn ) , and ( I -ˆ A † trn ˆ A trn ) is a projection matrix, thus

To compute W opt = ˆ X trn ( ˆ X trn + ˆ A trn ) † = σ trn u ˆ v T trn ( ˆ A trn + σ trn u ˆ v T trn ) † , using ˆ γ -1 = σ trn ˆ v T trn ˆ A † trn u = σ trn ˆ hu , we multiply this through.

Then we have,

and

Substituting back in and collecting like terms, we get,

We can then simplify the constants as follows.

and

This gives us the result.

E.3.3 Step 3: Decompose the terms into a sum of various trace terms.

For the bias and variance terms, we have the following two Lemmas.

Lemma 2. If W opt is the solution to Equation 1, then

Proof. To see this, note that we have n + M > M .

Note that ˆ γ = 1 + σ trn ˆ v T trn ˆ A † trn u . Thus, we have that σ trn ˆ v T trn ˆ A † trn u = ˆ γ -1 . Substituting this into the second term, we get,

For the third term, since ˆ k = ˆ A † trn u , ˆ k T ˆ A † trn u = ˆ k T ˆ k = ∥ ˆ k ∥ 2 . Substituting this into the expression, we get that

Since X tst = uv T tst , we get,

Simplify the constants using ˆ τ = σ 2 trn ∥ ˆ t ∥ 2 ∥ ˆ k ∥ 2 + ˆ γ 2 , we get,

Lemma 3 (Sonthalia and Nadakuditi [24]) . If the entries of A tst are independent with mean 0, and variance 1 /d , then we have that E A tst [ ∥ W opt A tst ∥ 2 ] = N tst d ∥ W opt ∥ 2 .

̸

Lemma 4. If ˆ γ = 0 and A trn has full rank, then we have that

Proof. We have

Where the last inequality is true due to the fact that ∥ u ∥ 2 = 1 .

E.3.4 Step 4: Estimate With Random Matrix Theory

Lemma 5. Let A be a p × q matrix and let ˆ A = [ A µI ] ∈ R p × q + p . Suppose A = U Σ V T be the singular value decomposition of A . If ˆ A = ˆ U ˆ Σ ˆ V T is the singular value decomposition of ˆ A , then ˆ U = U and if p < q

and

Here V 1: p are the first p columns of V .

Proof. Since p < q , we have that U ∈ R p × p , Σ ∈ R p × p are invertible. Here also consider the form of the SVD in which V T ∈ R p × q .

Westart by nothing that ˆ U ˆ Σ 2 ˆ U T = ˆ A ˆ A T = AA T + µ 2 I = U (Σ 2 + µ 2 I p ) U T . Thus, we immediately see that σ i ( ˆ A ) 2 = σ i ( A ) 2 + µ 2 and that ˆ U = U .

Finally, we see,

Lemma 6. Let A be a p × q matrix and let ˆ A = [ A µI ] ∈ R p × q + p . Suppose A = U Σ V T be the singular value decomposition of A . If ˆ A = ˆ U ˆ Σ ˆ V T is the singular value decomposition of ˆ A , then ˆ U = U and if p > q

Here we will denote the upper left q × q block by C . Further,

Proof. Since p > q , we have that U ∈ R p × p and we have that Σ ∈ R p × q . Here V T ∈ R q × q is invertible.

We start with nothing,

Thus, we immediately see that for i = 1 , . . . , p σ i ( ˆ A ) 2 = σ i ( A ) 2 + µ 2 and for i = p +1 , . . . , q , we have that σ i ( ˆ A ) 2 = µ 2 and that ˆ U = U .

Then, we see,

Note that Σ has 0 for the last p -q entries. Thus,

Similarly, due to the structure of ˆ Σ , we see,

Lemma 7. Suppose A is an p by q matrix such that p < q , the entries of A are independent and have mean 0, variance 1 /p , and bounded fourth moment. Let c = p/q . Let ˆ A = [ A µI ] ∈ R p × q + p . Let W p = ˆ A ˆ A T and let W q = ˆ A T ˆ A . Suppose λ p is a random non-zero eigenvalue from the largest p eigenvalues of W p , and λ q is a random non-zero eigenvalue of W q . Then

Proof. First, we note that the non-zero eigenvalues of W p and W q are the same. Hence we focus on W p . W p is nearly a Wishart matrix but is not normalized by the correct value. However, cW p does have the correct normalization.

Due to the assumptions on A , we have that the eigenvalues of cAA T converge to the MarchenkoPastur. Hence since the eigenvalues of cW p are

we can estimate them by estimating cσ i ( A ) 2 with the Marchenko-Pastur [44, 50-53]. In particular, we want the expectation of the inverse. We need to use the Stieljes transform. We know that if m c ( z ) is the Stieljes transform for the Marchenko-Pastur with shape parameter c , then if λ is sampled from the Marchenko-Pastur distribution, then

Thus, we have that the expected inverse of the eigenvalue can be approximated m ( -cµ 2 ) . We know that the Steiljes transform:

Thus, we have,

Canceling 1 /c from both sides, we get,

Then for the estimate of E [ 1 /λ 2 p ] , we need to compute the derivative of the m c ( z ) and evaluate it at -cµ 2 . Hence, we see,

Thus,

Canceling the 1 /c 2 from both sides, we get,

Multiplying out and simplifying

Lemma 8. Suppose A is an p by q matrix such that p > q , the entries of A are independent and have mean 0, variance 1 /p , and bounded fourth moment. Let c = p/q . Let ˆ A = [ A µI ] ∈ R p × q + p . Let W p = ˆ A ˆ A T and let W q = ˆ A T ˆ A . Suppose λ p is a random non-zero eigenvalue of W p , and λ q is a random eigenvalue from the largest q eigenvalues of W q . Then

Proof. First, we note that the non-zero eigenvalues of W p and W q are the same. Hence we focus on W p . Due to the assumptions on A , we have that the eigenvalues of A T A converge to the MarchenkoPastur with shape c -1 . Hence if λ p is one of the first q eigenvalues of W p , we see,

Then for the estimate of E [ 1 /λ 2 p ] , we need to compute the derivative of the m c -1 ( z ) and evaluate it at -µ 2 . Hence, we see,

This can be further simplified to

We will also need to estimate some other terms.

Lemma 9. Suppose A is an p by q matrix such that the entries of A are independent and have mean 0, variance 1 /p , and bounded fourth moment. Let ˆ A = [ A µI ] ∈ R p × q + p . Let W p = ˆ A ˆ A T and let W q = ˆ A T ˆ A . Suppose λ p , λ q are random non-zero eigenvalues of W p , W q from the largest min( p, q ) eigenvalues of W p , W q . Then

Proof. Notice that if λ is an eigenvalue of A (so unshifted).

Then use Lemmas 7, and 8 to finish the proof.

Bounding the Variance.

Lemma 10. Let η n be a uniform measure on n numbers a 1 , . . . , a n such that η n → η weakly in probability. Then for any bounded continuous function f

Then using the boundedness of f , we get,

Lemma 11. Let η n be a uniform measure on n numbers a 1 , . . . , a n such that η n → η weakly in probability. Let s be a uniformly random unit vector in R m independent of η n . Suppose n/m → ζ ∈ (0 , 1] . Then for any bounded function f ,

and

Proof. The first limit comes directly from weak convergence.

For the second, notice,

̸

Taking the expectation with respect to s we get,

Then using Lemma 10 for any fixed i , we have,

̸

Proof. Using weak convergence

̸

̸

Thus, as n →∞ , we have,

Then since

Thus, the variance goes to zero.

The interpretation of the above Lemma is that the variance of the sum decays to zero as m →∞ . Lemma 12. Suppose A is an p by q matrix such that the entries of A are independent and have mean 0, variance 1 /p , and bounded fourth moment. Let ˆ A = [ A µI ] ∈ R p × q + p . Let x ∈ R p and ˆ y ∈ R p + q be unit norm vectors such that ˆ y T = [ y T 0 p ] . Then

The variance of each above is o (1) .

Proof. Let us start with p < q .

Let ˆ A = ˆ U ˆ Σ ˆ V T , where ˆ Σ is p × p . Then we see,

Where ˆ U is uniformly random. Thus similar to [24], we can use Lemma 7 to get,

On the other hand, for p > q , we have that only the first q eigenvalues have the expectation in Lemma 8 The other p -q are equal to 1 µ 2 . Thus, we see,

Again let us first consider the case when p < q . Then we have,

Since ˆ y has zeros in the last p coordinates, we see,

Thus, we can use Lemma 9 to estimate this as,

The extra factor of c comes from the sum of p coordinates of a uniformly unit vector in q dimensional space. And for p > q , we have that the estimate is

For the variance term, use Lemma 11. For three of the cases, the limiting distribution is the MarchenkoPastur distribution. For the other case, the limiting measure is a mixture of the Marchenko-Pastur and a dirac delta at 1 /µ 2 .

The rest of the lemmas in this section are used to compute the mean and variance of the various terms that appear in the formula of W opt .

Lemma 13. We have that

and that V ( ∥ ˆ h ∥ 2 ) = o (1) .

Proof. Here we see that

Thus, using the Lemma 12 we get that if c < 1

and if c > 1

Lemma 14. We have

and that V ( ∥ ˆ k ∥ 2 ) = o (1) .

Proof. Since ˆ k = ˆ A † trn u , we have that

According to the Lemma 12, if c < 1

and if c > 1

Lemma 15. We have that

and we have that V ( ∥ ˆ t ∥ 2 ) = o (1)

Proof. Here we see that ˆ t = ˆ v trn ( I -ˆ A † trn ˆ A trn ) . Thus, we see that

If ˆ V ∈ R p + q × p + q , we have that

Then if p < q using Lemma 6 and the fact that the last p coordinates of ˆ v trn are 0, we see that

Then using Lemma 9 to estimate the middle diagonal matrix, we get that

Similarly for c > 1 , we have that

The variance of ˆ A † trn ˆ A trn is also o (1) using Lemma 11.

Lemma 16. We have that E A trn [ˆ γ ] = 1 and V ( γ ) = O ( σ 2 trn /d ) .

Proof. Noting that ˆ A = U ˆ Σ ˆ V T , we have that

Here ˆ a T = ˆ v T trn ˆ V and b = U T u . U is a uniformly random rotation matrix that is independent of ˆ Σ and ˆ V . Thus, taking the expectation with respect to A trn , we get that the expectation is equal to zero. For the variance, let us first consider the case when c < 1 . For this case, we have that

Thus, letting a T = v T trn V 1: d , we get that

Squaring and taking the expectation, we see that

Similarly for c > 1 , we have that

Lemma 17. We have that

and that V ( ρ ) = o (1) .

Proof. Here we have that

We first notice that

Thus using Lemmas 7 and 8, we see that if c < 1

and if c > 1

The variance being o (1) comes from Lemma 11 again.

Lemma 18. We have that

and the variance is o (1) .

Proof. Letting ˆ A = U ˆ Σ ˆ V T , we get that

Then again since U is uniformly random and independent of ˆ Σ and ˆ V , the expectation is equal to zero. The variance is computed similarly to Lemma 16.

E.3.5 Step 5: Putting it together

Lemma 19. We have that

and that V ( τ/σ 2 trn ) = o (1) .

Proof. Using the fact that all of the quantities concentrate, we can use the previous estimates. Specifically, we use that

Thus, since our variances decay, we can use the product of the expectations. Further,

Thus, since the variances individually go to 0, we see that the variance of the product also goes to 0 . Then using Lemma 15 and 14, we have that if c < 1

and V ( ∥ ˆ t ∥ 2 ∥ ˆ k ∥ 2 ) = o (1) . Then since

we have that using Lemma 16, that if c < 1

and that that variance is o (1) . If c > 1

Lemma 20. We have that

and that the variance is o (1) .

Proof. Similar to Lemma 19, we can multiply the expectations since the variances are small. For c < 1 , simplifying, we get that

and if c > 1 , we get that

and the variance decays since the variances decay individually.

Lemma 21. We have that

and that V ( ∥ W opt ∥ 2 F ) = o (1) .

Proof. Follows immediately from Lemmas 4, 17, 18, and 20.

Theorem 1 (Generalization Error Formula) . Suppose the training data X trn and test data X tst satisfy Assumption 3 and the noise A trn , A tst satisfy Assumption 4. Let µ be the regularization parameter. Then for the under-parameterized regime (i.e., c < 1 ) for the solution W opt to Problem 1, the generalization error or risk given by Equation 2 is given by

where

Proof. Rewriting ˆ γ 2 τ 2 as ˆ γ 2 /σ 4 trn τ 2 /σ 4 trn , we can the concentration from Lemmas 16 and 19. Then using Lemma 21 we get the needed result.

Theorem 6. For the over-parameterized case, we have that the generalization error is given by

Proof. Rewriting ˆ γ 2 τ 2 as ˆ γ 2 /σ 4 trn τ 2 /σ 4 trn , we can the concentration from Lemmas 16 and 19. Then using Lemma 21 we get the needed result.

E.4 Proof of Theorem 2

Theorem 2 (Under-Parameterized Peak) . Let µ ∈ R > 0 , σ 2 trn = n = d/c and σ 2 tst = n tst , and d is sufficiently large, so that the error term o (1 /d ) is small, then the risk R ( c ) from Theorem 1, as a function of c , has a local maximum in the under-parameterized regime at c = 1 1+ µ 2 .

Proof. First, we compute the derivative of the risk. We do so using SymPy and get the following expression.

We can then compute the limit as c → 0 + . Again using SymPy we see that

To find the critical point, we shall write the derivative as one fraction of the form

Let

Here we see that

and

We can see that the numerator is zero at c = (1 + µ 2 ) -1 . To evaluate the denominator at this point, we first get that

Thus, we see that the denominator is

Since T ((1 + µ 2 ) -1 , µ 2 < 4 , we have that

Hence the denominator is non-zero. Thus, we see that c = (1 + µ 2 ) -1 is a critical point.

Next we want to show that this point is a local maximum. To do so, we compute the second derivative and evaluate at c = (1 + µ 2 ) -1 . Using SymPy, we get that the value of the second derivative at this point is

To see that it is a maximum, we need to show that the above is negative. We begin by showing that the denominator is positive. Since 8 dµ 3 > 0 , we only need to look at the second term. Using arithmetic mean and geometric mean inequality, we see that

Hence the denominator is positive. To show that the numerator is negative, we have the following

Where we are allowed to square both sides because both quantities are non-negative. Thus, we see get the needed result.

E.5 Proof of Theorem 3

Theorem 3 ( ∥ W opt ∥ F Peak) . If σ tst = √ n tst , σ trn = √ n and µ is such that p ( µ ) < 0 , then for fixed n that is sufficiently large enough, we have that E [ ∥ W opt ∥ F ] versus c = d/n curve has a local maximum in the under-parameterized regime at c = ( µ 2 +1) -1 .

Proof. Here we note that the expression for the norm of W opt is given by Lemma 21. We follow the same proof structure as Theorem 2. Differentiating with respect to c , we see that the numerator is if the form

When the denominator is

Where is as is in the proof of Theorem 2. Again at when c = 1 / ( µ 2 +1) . We see that the denominator is positive because √ T < 2 . Hence again, we have a critical point c = (1 + µ 2 ) -1 .

E.6 Proof of Theorem 5

Theorem 5 (Training Error) . Let τ be as in Theorem 1. The training error for c < 1 is given by

and

Proof. Note that we have:

First, by Lemma 2, we have X trn -W opt X trn = ˆ γ ˆ τ X trn . Then, E [ ∥ X trn -W opt X trn ∥ 2 ] = ˆ γ 2 ˆ τ 2 E [ ∥ X trn ∥ 2 ] = ˆ γ 2 σ 2 trn ˆ τ 2 . Then, let us look at the E A trn [ ∥ W opt A trn ∥ 2 F ] term.

.

Then, we look at the Tr(( X trn -W opt X trn ) T W opt A trn ) term. By Lemma 2, we have X trn -W opt X trn = ˆ γ ˆ τ X trn . Then,

In conclusion, we have the training error:

Now we estimate the above terms using random matrix theory. Here we focus on the c < 1 case. For c < 1 , we note that

Thus, for c < 1

where a T = v T trn V 1: d . Taking the expectation, and using Lemma 9 we get that

Using Lemma 11, we see that the variance is o (1) . Similarly, we have that

Thus, again, using a similar argument, we see that

and again using Lemma 11, the variance is o (1) . Finally,

Thus,

Thus, using Lemma 9, we get that

and using Lemma 11, the variance is o (1) . Then, similar to the proof of Theorem 1, we can simplify the above expression to get the final result.

E.7 Proof of Proposition 1

Proposition 1 (Optimal σ trn ) . The optimal value of σ 2 trn for c < 1 is given by

Proof. Let σ := σ 2 trn and

Notice that only τ is a function of σ , ∥ ˆ h ∥ 2 2 , ∥ ˆ t ∥ 2 2 , and ∥ ˆ k ∥ 2 2 are all functions of µ . Then

The optimal σ ∗ satisfies ∂F ∂σ | σ = σ ∗ = 0 . Thus, we can solve the equation

Let α := ∥ ˆ t ∥ 2 2 ∥ ˆ k ∥ 2 2 , δ := d σ 2 tst N tst . Then

Notice that σ < 0 implies σ trn is an imaginary number, something we don't want. Thus, we look at the other expression.

Then multiplying through by d and τ

Then solving for σ , we get that

Then we use the random matrix theory lemmas to estimate this quantity.

F Proof of low-rank case

Similar to the proof in [37], we conducted the low rank case.

We begin by defining some notation. Let ˆ X trn = U Σ trn V T trn . Here U is d × r with U T U = I , Σ trn is r × r , and V trn is r × ( d + N ) . All of the following matrices are full rank.

ˆ X trn = U Σ V , by the singular value decomposition. Let ˆ X tst = UL .

Σ trn is r × r , with rank r .

10. Z is r × r , with rank r .

11.

H

1

is

r

×

r

, with rank

ˆ U is d × d unitary.

ˆ Σ is d × d .

For rank r data and r < N , with c = d N , the following is true.

We denote the minimum norm linear denoiser W opt by just W in this subsection. It is given by

The test error when X tst = UL is given by

where Q = V T ( I -ˆ A † trn ˆ A trn ) , H = V T trn ˆ A † trn ,1 K = -ˆ A † trn U Σ trn , Z = I + V T trn ˆ A † trn U Σ trn , H 1 = K T K + Z T ( QQ T ) -1 Z .

For c < 1 , we have that if d < N then

and if d > N then

When d < N then

if d > N then

r

.

We have that and

When d < N we have that

and when d > N , we have

When d < N , we have

when d > N this is estimated by

When d < N the test error R ( W,X tst ) for W = W opt is given by

when d > N this is estimated by

G Experiments

All experiments were conducted using Pytorch and run on Google Colab using an A100 GPU. For each empirical data point, we did at least 100 trials. The maximum number of trials for any experiment was 20000 trials.

For each configuration of the parameters, N trn , N tst , d, σ trn , σ tst , and µ . For each trial, we sampled u, v trn , v tst uniformly at random from the appropriate dimensional sphere. We also sampled new training and test noise for each trial.

For the data scaling regime, we kept d = 1000 and for the parameter scaling regime, we kept N trn = 1000 . For all experiments, N tst = 1000 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Both the abstract and introduction reference the main two Theorems (Theorem 4 and Theorem 2). Additionally, both put them in the context of prior work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We believe the main purpose of the paper is to show that a certain phenomenon exists and are very careful with our assumptions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All statements have corresponding detailed proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have very few experiments. However, for each one we have a section in the appendix with the needed details and have provided code as part of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the code as part of the submission. All data used is synthetic or already open source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have very few experiments. However, for each one we have a section in the appendix with the needed details and have provided code as part of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper is a theory paper about mean behavior under a variety of concentration results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Google Collab with an A100 was used.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines: The paper conforms with the code of ethics.

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work that helps build an understanding of existing phenomena.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical work

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credit pytorch.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This is a theoretical work

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This is a theoretical work

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is a theoretical work

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to understand the phenomenon of double descent in regression, and helps complement existing knowledge about the phenomenon by proving that double descent can occur even in the under-parametrized regime, going against previous intuition. They also prove that the peak in the norm of the estimator does not imply a peak in the risk.\n\nSTRENGTHS:\nQuality and Clarity: Contextualization relative to prior work is good, Table 1, for example, provides a very concise and insightful summary of existing results. \n\nOriginality: The paper considers an original viewpoint on the problem of double-descent, namely they realize that violating either Assumptions 1 or 2 can cause the peak to move into the under-parametrized region. To the best of my knowledge, this had not been noticed before. \n\nSignificance: This result is significant, as it addresses the prominent double-descent phenomenon in machine learning, which is an important primitive for understanding generalization and other properties of estimators operating on high-dimensional data. They also prove that a peak in the norm of the estimator does not imply a peak in risk, and this is important as it goes against some of the intuition provided in earlier works.\n\nWEAKNESSES:\nClarity: Although the contextualization relative to prior work is good, I find that the paper lacks in clarity. In particular, it is not exceptionally well written, and leaves some sections with much to be desired in terms of exposition. Specific examples are: \n\n- Table 1: What does 1/Low mean? The superscript 3 leads to nowhere.\n- Middle of page four: “Hence, this is controlled by 1. The alignment…, 2. The spectrum”. What exactly is being controlled here? And why exactly is it controlled by 1. and 2. ? This is not immediate to me, and I think this needs to be made more precise.\n- Over-use of italics in the introduction makes it hard to know what to focus on. I would recommend maximum one italicized sentence per paragraph.\n- Section 4.1: be more precise about the model, I don’t know what it is at this point. I later identify that $X + A$ represents the spiked data, but this must be made clear earlier on.  I find“Let A be the noise matrix” to not be clear enough.\n- Theorem 1: a comment on the proof technique for this theorem would be helpful, even if you have already mentioned it previously (or given the intuition). Also, the interpretation of Theorem 1 is not clear until we read Theorem 2, could you interpret Theorem 1 a bit more and specifically identify what it says that is not said in Theorem 2? Is Theorem 2 just a corollary of Theorem 1?\n- The first sentence of the abstract is not clear. Overall, the abstract may need to be rewritted in a more professional manner, not referring to previous works as \"believing\" in something, but something more precise.\n\nQUESTIONS:\nFigure 2: I see a clear peak, but I do not see an initial “descent”. Is there something I am missing here? Why does the model have best generalization at $c = 0$? I am guessing that the focus of the study is on the peak, although that is not the full picture of double descent as I am not seeing the initial descent. That is okay, but just make clear why there is no initial descent.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore the double descent phenomenon, postulating that the location of the peak (that separates the \"classical\" and the \"modern\" interpolating regime) depends on the properties of the spectrum and the eigenvectors of the sample covariance. In particular, the authors show that the violation of one of two assumptions (assumption 1: Alignment of y and right singular vectors of X; assumption 2: Stieljtes Transform Peak Assumption) can move the peak from the interpolation point into the under-parameterized regime. They also present two simple examples that exhibit double descent in the under-parameterized regime and do not seem to occur for reasons provided in prior work.\n\nSTRENGTHS:\n- The paper tackles a very important research topic, and tries to understand the reasons behind the location of the peak in double descent. \n- The work seems rigorous and the contributions relevant. \n- Overall, the paper is well-written and reasonably clear.\n\nWEAKNESSES:\n- The conclusions of the paper are very brief and, from my point of view, not very informative (see section 6 in the paper). In relation to this, I also perceive a certain imbalance in the weight of the two examples provided: while the first (the one related to \"Alignment Mismatch\") occupies 3 pages of the work, the second example (\"Shifting Local Maximum for Stieljtes Transform as a Function of c\") is addressed more hastily (one page).\n- The volume of information provided by the paper is very high. From this point of view, I think it would be positive to recapitulate and indicate clearly, and in a simple and intuitive way, the way in which the risk curves shown throughout the paper are created (Figures 2, 4 and 6). The same applies to Figure 3: what ablation experiments do the authors refer to?\n\nQUESTIONS:\n- What do the authors exactly mean by input and output noise?\n- In Table 1 (page 3), where are the footnotes related with numbers 3 and 4?\n- In Figure 3, what ablation experiments do the authors refer to?\n\nLIMITATIONS:\n- In my opinion, the authors do not sufficiently discuss the limitations of the work performed. In fact, in the NeurIPS Paper Checklist, they only state that \"We believe the main purpose of the paper is to show that a certain phenomenon exists and are very careful with our assumptions.\"", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors focus on the generalization performance of linear least squares regression and show the existence of double descent generalization curve  in the under-parameterization regime.\nIn particular, the authors argue, in the linear model in (1) under study (which is slightly different from standard linear models in the literature, but well motivated), that the generalization risk can have a peak in the under-paramererized regime that is due to the alignment between singular space of data and the target, and spectrum of data covariance, instead of the raw dimension ratio or the explosion of the estimator norm.\nSome numerical results are provided to support the theoretical analysis.\n\nSTRENGTHS:\nThe problem under study is of significance.\nThe message of this paper looks interesting. But I find it a bit hard to really understand the results and contribution, see my comments below.\n\nWEAKNESSES:\nWhile this paper looks interesting, it is a bit hard for me to really understand the results and contribution.\nI think making precise the dimension settings (relation between $n, d, n_{tst})$ will address this issue, see some of my detailed comments below.\nAnother issue is the contribution: while Theorem 1 is rather general, the discussion thereafter seems all special cases: For example Theorem 2 is a special case, and the results in Sec 4.3 and 4.5 are essentially numerical. The discussions in Sec 5 is interesting but again a very special setting (mixture model of multivariate Gaussian and a fixed direction) without any motivation. It is thus difficult for me to evaluate the significance of this work.\n\nQUESTIONS:\n* line 35: when summarizing the contribution of this work, it would be helpful to forward point to the corresponding theoretical result and/or definition, for the sake of a precise statement of the technical result or the definition (for example, the spiked covariate model).\n* it seems that the footnotes 3 and 4 are missing?\n* Equation after line 123: I am a bit confused here. What is the purpose here? Is $\\hat \\beta$ still the min norm solution, then what is $\\beta$?\n* To make Definition 1 more rigorous, perhaps say here that the convergence of ESD holds in a weak sense as $k \\to \\infty$, or something like that?\n* Theorem 1: perhaps say somewhere in the theorem that this result holds in the asymptotic setting as $n,d,n_{tst}$ going to infinity at the same pace?\n* Honestly, I do not understand this result. It seems to me that my previous comment is wrong, and that the result in Theorem 1 does NOT hold in the limit of $n,d,n_{tst} \\to \\infty$ together, or at least, $n_{tst}$ and $n$ can be much larger than $d$. Some specifications and discussions are needed here. \n* Theorem 2 looks interesting. Could the authors comment more on this? For example, note that taking $\\mu = 0$ is (more or less) similar to the ridgeless case in the literature. There, according to Theorem 2, we should have a at $c = 1$, as in accordance with \"classical\" double descent.  So, should we understand Theorem 2 as an extension of \"classical\" double descent to the regularized setting? Or is this due to the model in (1) and (2)?\n* line 207 -208: $\\hat \\Sigma^T \\hat \\Sigma = \\Sigma^T \\Sigma + \\mu^2$ a typo here?\n\n---\nI thank the authors for their detailed reply, which helps me better understand their theoretical results and their contribution I increase my score accordingly. \n\nI, nonetheless, feel that there are a few typos that need be fixed and clarifications needed, in the current version of the paper.\n\nLIMITATIONS:\nI do not see any potential negative social impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper considers the problem of linear least squares. Its main contribution is presenting two examples of double descent in the under-parameterized regime.

STRENGTHS:
- The paper is well written: Related works are sufficiently discussed (to my knowledge); the introduction is well-motivated and easy to follow; theorems are often followed by examples, figures, and illustrations helping the reader understand the results.
- The problem the paper investigates and the perspective the paper takes is quite interesting. While the mainstream research in the field focuses on double descent in the case of over-parameterization, the paper analyzes under-parametrization in-depth and presents several results that improve one's understanding of double descent.

WEAKNESSES:
- The paper takes unconventional notations that make the paper more challenging to penetrate. For example, it uses row vector notations and writes vector-matrix multiplication $\beta^\top X$ rather than the more common matrix-vector multiplication $X^\top \beta$. Sometimes I also found that the notations of singular vector $u$ and regularization parameter $\mu$ can be confusing as they look similar.
- The two examples the paper offered are indeed examples. The reason is that the paper's assumptions are quite strong. For example, Assumption 3 assumes the test and training data matrices are both of rank $1$, and Theorem 4 has the orthogonality assumption $\beta^\top z=0$ which greatly simplifies the model and analysis.

QUESTIONS:
I have no questions. It should be noted that I am not an expert in the exact area of double descent. I am not very familiar with the proof techniques used in the literature and am unable to make comments on technical depth.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show several facts about the double descent phenomena for the linear regression model with L2 loss and Frobenius norm regularization. They show taking different assumption from previous work moves the peak of the risk of the problem from the interpolation point into the under-parameterized regime. Provided theorems concretely describe the position of peak risk, including separating the risk into several terms: bias and the norm of the estimator. The authors also provide derivations (appendix) and experiments.\n\nSTRENGTHS:\nTheir analysis is thorough. Especially, reasoning change of trends of risks in Figure 4 through the term of the norm of the estimator supports the reliability of the results. They clarify proof steps in the appendix. Experiments provided to support each theorem also look thorough.\n\nWEAKNESSES:\nThe authors results which do not coincide with prior theory are based on different assumptions. Can the authors discuss how wider cases covered by their assumption?\n\nQUESTIONS:\nLine 190: Is the assumption \"$d$ is sufficiently large\" used only to assume $o\\left(1/d\\right)=0$ in the Equation between Lines 177 & 178? If so, clarifying it in the main text would be better.\n\n\n\nDoes $\\left\\|W_{opt}\\right\\|_F$ in Line 240 indicate its expectation?\n\n\n\nI recommend the authors to double check the overall text and equations. Here is a list of errata & typos.\n\nNote that from the equation between Line 119 and Line 120, $\\hat \\beta\\in\\\\mathbb{R}^{d\\times k}$.\n\n* Line 120: $\\hat\\beta = y X^\\dagger$ -> $\\hat \\beta = \\left(X^\\dagger\\right)^Ty^T$\n\n* Displayed Equation between Lines 123-124: The denominator seems wrong. The following should be correct:\n\n$$\n\\left\\|\\hat\\beta\\right\\|^2 = \\sum_{i=1}^{\\mathrm{rank}\\left(X\\right)}\\frac{\\left(y V\\right)_i^2}{\\sigma_i^2}\n$$\n\n* Line 129: It would be better to add a couple of words emphasizing $\\Sigma_n^{1/2}$ indicates the diagonal matrix $\\Sigma$ in $X=U\\Sigma V^T$ for readability. $\\Sigma_n^{1/2}z_i$ looks like a summation symbol. I first thought it is a typo of $\\sum_{i=1}^{1/2}z_i$.\n\n* Displayed equation between Lines 137-138: $z\\in \\mathbb{C}\\setminus J$ -> $\\zeta\\in \\mathbb{C}\\setminus J$\n\n* Lines 239-240: Is \"for $n$ large enough and $d=cn$\" a typo of \"for $n$ and large enough $d=cn$\"?\n\nLIMITATIONS:\nDoes the previous work (peak at interpolation point, using Assumptions 1 and 2) also rely on Assumptions 3 and 4? Or are you using just a different set (A 2 & 3 & 4) of assumptions which is a neither necessary nor sufficient condition of one of the previous work (A 1 & 2)? It should be stated more clearly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}