Title: Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors
Venue: NeurIPS 2024
Accepted: no
Keywords: minimum norm solution

ridgeless estimator

benign overfitting

double descent

overparameterization
Abstract: In recent years, there has been a significant growth in research focusing on minimum $\ell_2$ norm (ridgeless) interpolation least squares estimators. However, the majority of these analyses have been limited to an unrealistic regression error structure, assuming independent and identically distributed errors with zero mean and common variance. In this paper, we explore prediction risk as well as estimation risk under more general regression error assumptions, highlighting the benefits of overparameterization in a more realistic setting that allows for clustered or serial dependence. Notably, we establish that the estimation difficulties associated with the variance components of both risks can be summarized through the trace of the variance-covariance matrix of the regression errors. Our findings suggest that the benefits of overparameterization can extend to time series, panel and grouped data.

1

Abstract

In recent years, there has been a significant growth in research focusing on minimum ℓ 2 norm (ridgeless) interpolation least squares estimators. However, the majority of these analyses have been limited to an unrealistic regression error structure, assuming independent and identically distributed errors with zero mean and common variance. In this paper, we explore prediction risk as well as estimation risk under more general regression error assumptions, highlighting the benefits of overparameterization in a more realistic setting that allows for clustered or serial dependence. Notably, we establish that the estimation di ffi culties associated with the variance components of both risks can be summarized through the trace of the variance-covariance matrix of the regression errors. Our findings suggest that the benefits of overparameterization can extend to time series, panel and grouped data.

1 Introduction

Recent years have witnessed a fast growing body of work that analyzes minimum ℓ 2 norm (ridgeless) 13 interpolation least squares estimators [see, e.g., 2, 17, 27, and references therein]. Researchers in this 14 field were inspired by the ability of deep neural networks to accurately predict noisy training data 15 with perfect fits, a phenomenon known as 'double descent' or 'benign overfitting' [e.g., 3-5, 29, 22, 16 among many others]. They discovered that to achieve this phenomenon, overparameterization is 17 critical. 18

In the setting of linear regression, we have the training data { ( xi , yi ) ∈ R p × R : i = 1 , · · · , n } , where 19 the outcome variable yi is generated from 20

xi is a vector of features (or regressors), β is a vector of unknown parameters, and ε i is a regression 21 error. Here, n is the sample size of the training data and p is the dimension of the parameter vector β 22

.

In the literature, the main object for the theoretical analyses has been mainly on the out-of-sample 23 prediction risk. That is, for the ridge or interpolation estimator ˆ β , the literature has focused on 24

where x 0 is a test observation that is identically distributed as xi but independent of the training data. 25 For example, Dobriban and Wager [13], Wu and Xu [28], Richards et al. [23], Hastie et al. [17] 26 analyzed the predictive risk of ridge(less) regression and obtained exact asymptotic expressions under 27 the assumption that p / n converges to some constant as both p and n go to infinity. Overall, they 28 found the double descent behavior of the ridgeless least squares estimator in terms of the prediction 29

risk. Bartlett et al. [2], Kobak et al. [19], Tsigler and Bartlett [27] characterized the phenomenon of 30 benign overfitting in a di ff erent setting. 31

To the best of our knowledge, a vast majority of the theoretical analyses have been confined to a 32 simple data generating process, namely, the observations are independent and identically distributed 33 (i.i.d.), and the regression errors have mean zero, have the common variance, and are independent of 34 the feature vectors. That is, 35

This assumption, although convenient, is likely to be unrealistic in various real-world examples. For 36 instance, Liao et al. [21] adopted high-dimensional linear models to examine the double descent 37 phenomenon in economic forecasts. In their applications, the outcome variables include S&P firms' 38 earnings, U.S. equity premium, U.S. unemployment rate, and countries' GDP growth rate. As in 39 their applications, economic forecasts are associated with time series or panel data. As a result, it 40 is improbable that (1) holds in these applications. As another example, Spiess et al. [26] examined 41 the performance of high-dimensional synthetic control estimators with many control units. The 42 outcome variable in their application is the state-level smoking rates in the Abadie et al. [1] dataset. 43 Considering the geographical aspects of the U.S. states, it is unlikely that the regression errors 44 underlying the synthetic control estimators adhere to (1). In short, it is desirable to go beyond the 45 simple but unrealistic regression error assumption given in (1). 46

0

500

1000

1500

2000

p

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

error

train error (

c

=0)

train error (

c

=1/4)

train error (

c

=2/4)

train error (

c

=3/4)

test error (

c

=0)

test error (

c

=1/4)

test error (

c

=2/4)

test error (

c

=3/4)

To further motivate, we start with our own real-data example from American Community Survey 47 (ACS) 2018, extracted from IPUMS USA [24]. The ACS is an ongoing annual survey by the 48 US Census Bureau that provides key information about the US population. To have a relatively 49 homogeneous population, the sample extract is restricted to white males residing in California with at 50 least a bachelor's degree. We consider a demographic group defined by their age, the type of degree, 51 and the field of degree. Then, we compute the average of log hourly wages for each age-degree52 field group, treat each group average as the outcome variable, and predict group wages by various 53 group-level regression models where the regressors are constructed using the indicator variables of 54 age, degree, and field as well as their interactions. We consider 7 specifications ranging from 209 55 to 2,182 regressors. To understand the role of non-i.i.d. regressor errors, we add the artificial noise 56 to the training sample. See Appendix A for details regarding how to generate the artificial noise. 57 In the experiment, the constant c varies such that c = 0 corresponds to no clustered dependence 58 across observations but as a positive c gets larger, the noise has a larger share of clustered errors but 59 the variance of the overall regression errors remains the same regardless of the value of c . Figure 1 60 shows the in-sample (train) vs. out-of-sample (test) mean squared error (MSE) for various values 61 of c ∈ { 0 , 0 . 25 , 0 . 5 , 0 . 75 } . It can be seen that the experimental results are almost identical across 62 di ff erent values of c especially when p > n , suggesting that the double descent phenomenon might 63 be universal for various degrees of clustered dependence, provided that the overall variance of the 64 regression errors remains the same. It is our main goal to provide a firm foundation for this empirical 65 phenomenon. To do so, we articulate the following research questions: 66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

How to analyze the out-of-sample prediction risk of the ridgeless least squares estimator under general assumptions on the regression errors?

Why does not the prediction risk seem to be a ff ected by the degrees of dependence across observations?

To delve into the prediction risk, suppose that Σ : = E [ x 0 x ⊤ 0 ] is finite and positive definite. Then,

If Σ = I (i.e., the case of isotropic features), where I is the identity matrix, the mean squared error of the estimator defined by E [ ∥ ˆ β -β ∥ 2 ], where ∥ · ∥ is the usual Euclidean norm, is the same as the expectation of the prediction risk defined above. However, if Σ , I , the link between the two quantities is less intimate. One may regard the prediction risk as the Σ -weighted mean squared error of the estimator; whereas E [ ∥ ˆ β -β ∥ 2 ] can be viewed as an 'unweighted' version, even if Σ , I . In other words, regardless of the variance-covariance structure of the feature vector, E [ ∥ ˆ β -β ∥ 2 ] treats each component of β 'equally.' The mean squared error of the estimator is arguably one of the most standard criteria to evaluate the quality of the estimator in statistics. For instance, in the celebrated work by James and Stein [18], the mean squared error criterion is used to show that the sample mean vector is not necessarily optimal even for standard normal vectors (so-called 'Stein's paradox'). Many follow-up papers used the same criterion; e.g., Hansen [16] compared the mean-squared error of ordinary least squares, James-Stein, and Lasso estimators in an underparameterized regime. Both Σ -weighted and unweighted versions of the mean squared error are interesting objects to study. For example, Dobriban and Wager [13] called the former 'predictive risk' and the latter 'estimation risk' in high-dimensional linear models; Berthier et al. [6] called the former 'generalization error' and the latter 'reconstruction error' in the context of stochastic gradient descent for the least squares problem using the noiseless linear model. In this paper, we analyze both weighted and unweighted mean squared errors of the ridgeless estimator under general assumptions on the data-generating processes, not to mention anisotropic features. Furthermore, our focus is on the finite-sample analysis, that is, both p and n are fixed but p > n .

Although most of the existing papers consider the simple setting as in (1), our work is not the first paper to consider more general regression errors in the overparameterized regime. Chinot et al. [9], Chinot and Lerasle [8] analyzed minimum norm interpolation estimators as well as regularized empirical risk minimizers in linear models without any conditions on the regression errors. Specifically, Chinot and Lerasle [8] showed that, with high probability, without assumption on the regression errors, for the minimum norm interpolation estimator, ( ˆ β -β ) ⊤ Σ ( ˆ β -β ) is bounded from above by GLYPH<16> ∥ β ∥ 2 P i ≥ c · n λ i ( Σ ) ∨ P n i = 1 ε 2 i GLYPH<17> / n , where c is an absolute constant and λ i ( Σ ) is the eigenvalues of Σ in descending order. Chinot and Lerasle [8] also obtained the bounds on the estimation error ( ˆ β -β ) ⊤ ( ˆ β -β ). Our work is distinct and complements these papers in the sense that we allow for a general variance-covariance matrix of the regression errors. The main motivation of not making any assumptions on ε i in Chinot et al. [9] and Chinot and Lerasle [8] is to allow for potentially adversarial errors. We aim to allow for a general variance-covariance matrix of the regression errors to accommodate time series and clustered data, which are common in applications. See, e.g., Hansen [15] for a textbook treatment (see Chapter 14 for time series and Section 4.21 for clustered data).

The main contribution of this paper is that we provide exact finite-sample characterization of the variance component of the prediction and estimation risks under the assumption that X = [ x 1 , x 2 , · · · , xn ] ⊤ is left-spherical (e.g., xi 's can be i.i.d. normal with mean zero but more general); ε i 's can be correlated and have non-identical variances ; and ε i 's are independent of xi 's. Specifically, the variance term can be factorized into a product between two terms: one term depends only on the trace of the variance-covariance matrix, say Ω , of ε i 's; the other term is solely determined by the distribution of xi 's. Interestingly, we find that although Ω may contain non-zero o ff -diagonal elements, only the trace of Ω matters, as hinted by Figure 1, and further demonstrate our finding via numerical experiments. In addition, we obtain exact finite-sample expression for the bias terms when the regression coe ffi cients follow the random-e ff ects hypothesis [13]. Our finite-sample findings o ff er a distinct viewpoint on the prediction and estimation risks, contrasting with the asymptotic inverse relationship (for optimally chosen ridge estimators) between the predictive and estimation risks uncovered by Dobriban and Wager [13]. Finally, we connect our findings to the existing results on the prediction risk [e.g., 17] by considering the asymptotic behavior of estimation risk.

One of the limitations of our theoretical analysis is that the design matrix X is assumed to be left120 spherical, although it is more general than i.i.d. normal with mean zero. We not only view this as a 121 convenient assumption but also expect that our findings will hold at least approximately even if X 122 does not follow the left-spherical distribution. It is a topic for future research to formally investigate 123 this conjecture. 124

125

2 The Framework under General Assumptions on Regression Errors

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

We first describe the minimum ℓ 2 norm (ridgeless) interpolation least squares estimator in the overparameterized case ( p > n ). Define

so that y = X β + ε . The estimator we consider is

where A † denotes the Moore-Penrose inverse of a matrix A .

The main object of interest in this paper is the prediction and estimation risks of ˆ β under the data scenario such that the regression error ε i may not be i.i.d. Formally, we make the following assumptions.

Assumption 2.1. (i) y = X β + ε , where ε is independent of X , and E [ ε ] = 0. (ii) Ω : = E [ εε ⊤ ] is finite and positive definite (but not necessarily spherical).

We emphasize that Assumption 2.1 is more general than the standard assumption in the literature on benign overfitting that typically assumes that Ω ≡ σ 2 I . Assumption 2.1 allows for non-identical variances across the elements of ε because the diagonal elements of Ω can be di ff erent among each other. Furthermore, it allows for non-zero o ff -diagonal elements in Ω . It is di ffi cult to assume that the regression errors are independent among each other with time series or clustered data; thus, in these settings, it is important to allow for general Ω , σ 2 I . Below we present a couple of such examples.

Example 2.1 (AR(1) Errors) . Suppose that the regressor error follows an autoregressive process:

where ρ ∈ ( -1 , 1) is an autoregressive parameter, η i is independent and identically distributed with mean zero and variance σ 2 (0 < σ 2 < ∞ ) and is independent of X . Then, the ( i , j ) element of Ω is

Note that Ω i j , 0 as long as ρ , 0.

Example 2.2 (Clustered Errors) . Suppose that regression errors are mutually independent across clusters but they can be arbitrarily correlated within the same cluster. For instance, students in the same school may a ff ect each other and also have the same teachers; thus it would be di ffi cult to assume independence across student test scores within the same school. However, it might be reasonable that student test scores are independent across di ff erent schools. For example, assume that (i) if the regression error ε i belongs to cluster g , where g = 1 , . . . , G and G is the number of clusters, E [ ε 2 i ] = σ 2 g for some constant σ 2 g > 0 that can vary over g ; (ii) if the regression errors ε i and ε j ( i , j ) belong to the same cluster g , E [ ε i ε j ] = ρ g for some constant ρ g , 0 that can be di ff erent across g ; and (iii) if the regression errors ε i and ε j ( i , j ) do not belong to the same cluster, E [ ε i ε j ] = 0. Then, Ω is block diagonal with possibly non-identical blocks.

For vector a and square matrix A , let ∥ a ∥ 2 A : = a ⊤ Aa . Conditional on X and given A , we define

and we write Var = Var I and Bias = Bias I for the sake of brevity in notation.

The mean squared prediction error for an unseen test observation x 0 with the positive definite 157 covariance matrix Σ : = E [ x 0 x ⊤ 0 ] (assuming that x 0 is independent of the training data X ) and the mean 158 squared estimation error of ˆ β conditional on X can be written as: 159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

In what follows, we obtain exact finite-sample expressions for prediction and estimation risks:

We first analyze the variance terms for both risks and then study the bias terms.

3 The Variance Components of Prediction and Estimation Risks

3.1 The variance component of prediction risk

We rewrite the variance component of prediction risk as follows:

where positive definite symmetric matrices S : = Σ 1 / 2 and T : = Ω 1 / 2 are the square root matrices of the positive definite matrices Σ and Ω , respectively. To compute the above Frobenius norm of the matrix S X † T , we need to compute the alignment of the right-singular vectors of B : = S X † ∈ R p × n with the left-eigenvectors of T ∈ R n × n . Here, B is a random matrix while T is fixed. Therefore, we need the distribution of the right-singular vectors of the random matrix B .

Perhaps surprisingly, to compute the expected variance E X [Var Σ ( ˆ β | X )], it turns out that we do not need the distribution of the singular vectors if we make a minimal assumption (the left-spherical symmetry of X ) which is weaker than the assumption that { xi } n i = 1 is i.i.d. normal with E [ x 1 ] = 0.

Definition 3.1 (Left-Spherical Symmetry [10-12, 14]) . A random matrix Z or its distribution is called to be left-spherical if OZ and Z have the same distribution ( OZ d = Z ) for any fixed orthogonal matrix O ∈ O ( n ) : = { A ∈ R n × n : AA ⊤ = A ⊤ A = I } .

Assumption 3.2. The design matrix X is left-spherical.

For the isotropic error case ( Ω = I ), we have E X [Var Σ ( ˆ β | X )] = E X [Tr(( X ⊤ X ) † Σ )] directly from equation 3 since X † X †⊤ = ( X ⊤ X ) † . Moreover, for the arbitrary error, the left-spherical symmetry of X plays a critical role to factor out the same E X [Tr(( X ⊤ X ) † Σ )] and the trace of the variance-covariance matrix of the regression errors, Tr( Ω ), from the variance after the expectation over X .

Lemma 3.3. For a subset S ⊂ R m × m satisfying C -1 ∈ S for all C ∈ S , if matrix-valued random variables Z and AZ have the same distribution measure µ Z for any A ∈ S , then we have

for any function f ∈ L 1 ( µ Z ) and any probability density function ν on S .

Theorem 3.4. Let Assumptions 2.1, and 3.2 hold. Then, we have

Sketch of Proof. With B = Σ 1 / 2 X † and T = Ω 1 / 2 , we can rewrite the variance as follows:

from the singular value decompositions B = UDV ⊤ and T = UTDTV ⊤ T with orthogonal matrices U , V , UT , VT , and diagonal matrices D , DT . Then, we need to compute the alignment V ⊤ UT of the right-singular vectors of B with the left-eigenvectors of T because

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

0.00

0.25

0.50

0.75

1.00

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

2

Prediction Risk - AR(1) Errors

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

2.7

3.0

0.00

0.25

0.50

0.75

1.00

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

2

Estimation Risk - AR(1) Errors

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

2.7

3.0

where v ( i ) : = V : i , u ( j ) : = ( UT ) : j , γ i j : = ⟨ v ( i ) , u ( j ) ⟩ 2 ≥ 0, Γ ( X ) : = ( γ i j ) i , j ∈ R n × n and λ ( A ) ∈ R n is a vector where its elements are the eigenvalues of A .

Now, we want to compute the expected variance. To do so, from Lemma 3.3 with S = O ( n ) and the left-spherical symmetry of X , we can obtain

where ν is the unique uniform distribution (the Haar measure) over the orthogonal matrices O ( n ).

Here, we can show that E O ∼ ν [ Γ ( OX )] = 1 n J , where J is the all-ones matrix with Jij = 1( i , j = 1 , 2 , · · · , n ). Therefore, we have the expected variance as follows:

The proofs of Lemma 3.3 and Theorem 3.4 are in the supplementary appendix.

3.2 The variance component of estimation risk

For the expected variance E X [Var( ˆ β | X )] of the estimation risk, a similar argument still holds if plugging-in B = X † instead of B = Σ 1 / 2 X † .

Theorem 3.5. Let Assumptions 2.1, and 3.2 hold. Then, we have

where XX ⊤ / p = U Λ U ⊤ for some orthogonal matrix U ∈ O ( n ) .

3.3 Numerical experiments

In this section, we validate our theory with some numerical experiments of Examples 2.1 and 202 2.2, especially how the expected variance is related to the general covariance Ω of the regressor 203 error ε . In the both examples, we sample { xi } n i = 1 from N (0 , Σ ) with a general feature covariance 204 Σ = U Σ D Σ U ⊤ Σ for an orthogonal matrix U Σ ∈ O ( p ) and a diagonal matrix D Σ ≻ 0. In this setting, we 205 have rank( XX ⊤ ) = n and Λ † = Λ -1 almost everywhere. 206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

0.1

0.2

0.3

0.4

0.5

2

1

0.1

0.2

0.3

0.4

0.5

2

2

Prediction Risk - Clustered Errors

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.1

0.2

0.3

0.4

0.5

2

1

0.1

0.2

0.3

0.4

0.5

2

2

Estimation Risk - Clustered Errors

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

AR(1) Errors As shown in Example 2.1, when the regressor error follows an autoregressive process in equation 2, we have Ω i j = σ 2 ρ | i -j | / (1 -ρ 2 ) and Tr( Ω ) / n = σ 2 / (1 -ρ 2 ). Therefore, for pairs of ( σ 2 , ρ 2 ) with the same Tr( Ω ) / n , they are expected to yield the same variances of the prediction and estimation risk from Theorem 3.4 and 3.5 even though they have di ff erent o ff -diagonal elements in Ω . To be specific, the pairs ( σ 2 , ρ 2 ) on a line { ( σ 2 , ρ 2 ) : σ 2 /κ 2 + ρ 2 = 1 } have the same Tr( Ω ) / n and the same expected variance which gets larger for the line with respect to a larger κ 2 .

Figure 2 (left) shows the contour plots of E X [Var Σ ( ˆ β | X )] and 1 n Tr( Ω ) E X [Tr(( X ⊤ X ) † Σ )] for di ff erent pairs of ( σ 2 , ρ 2 ) in Example 2.1. They have di ff erent slopes -κ -2 according to the value of κ 2 = Tr( Ω ) / n . The right panel shows equivalent contour plots for estimation risk.

Clustered Errors Now consider the block diagonal covariance matrix Ω = diag( Ω 1 , Ω 2 , · · · , Ω G ) in Example 2.2, where Ω g is an ng × ng matrix with ( Ω g ) ii = σ 2 g and ( Ω g ) i j = ρ g ( i , j ) for each i , j = 1 , 2 , · · · , ng and g = 1 , 2 , · · · , G . Let n = P G g = 1 ng . We then have Tr( Ω ) / n = P G g = 1 Tr( Ω g ) / n = P G g = 1 ( ng / n ) σ 2 g . Therefore, given a partition { ng } G g = 1 of the n observations, the covariance matrices Ω with di ff erent { σ 2 g } G g = 1 have the same Tr( Ω ) / n if ( σ 2 1 , σ 2 2 , · · · , σ 2 G ) ∈ R G are on the same hyperplane n 1 n σ 2 1 + n 2 n σ 2 2 + · · · + nG n σ 2 G = κ 2 for some κ 2 > 0.

Figure 3 (left) shows the contour plots of E X [Var Σ ( ˆ β | X )] and 1 n Tr( Ω ) E X [Tr(( X ⊤ X ) † Σ )] for di ff erent pairs of ( σ 2 1 , σ 2 2 ) for a simple two-clusters example ( G = 2) of Example 2.2 with ( n 1 , n 2 ) = (5 , 15). Here, we use a fixed value of ρ 1 = ρ 2 = 0 . 05, but the results are the same regardless of their values, as shown in the appendix. Unlike Example 2.1, the hyperplanes are orthogonal to v = [ n 1 , n 2 ] ⊤ regardless of the value of κ 2 = Tr( Ω ) / n . Again, the right panel shows equivalent contour plots for estimation risk.

4 The Bias Components of Prediction and Estimation Risks

Our main contribution is to allow for general assumptions on the regression errors, and thus the bias 229 parts remain the same as they do not change with respect to the regression errors. For completeness, 230 in this section, we briefly summarize the results on the bias components. First, we make the following 231 assumption for a constant rank deficiency of X ⊤ X which holds, for example, each xi has a positive 232 definite covariance matrix and is independent of each other. 233

234

Assumption 4.1. rank( X ) = n almost everywhere.

4.1 The bias component of prediction risk 235

The bias term of prediction risk can be expressed as follows: 236

where ˆ Σ : = X ⊤ X / n . Now, in order to obtain an exact closed form solution, we make the following assumption:

Assumption 4.2. E β [ S β ( S β ) ⊤ ] = r 2 Σ I / p , where r 2 Σ : = E β [ ∥ β ∥ 2 Σ ] < ∞ and β is independent of X .

A similar assumption (see Assumption 4.4) has been shown to be useful to obtain closed-form expressions in the literature [e.g., 13, 23, 20, 7].

Under this assumption, since [Bias Σ ( ˆ β | X )] 2 = Tr[ S β ( S β ) ⊤ lim λ ↘ 0 λ 2 ( S -1 ˆ Σ S + λ I ) -2 ] from equation 4, we have the expected bias (conditional on X ) as follows:

where ˜ si are the eigenvalues of S -1 ˆ Σ S ∈ R p × p and rank( S -1 ˆ Σ S ) = rank( X ) = n almost everywhere under Assumption 4.1. This bias is independent of the distribution of X or the spectral density of S -1 ˆ Σ S , but only depending on the rank deficiency of the realization of X .

Finally, the prediction risk RP ( ˆ β ) can be summarized as follows:

Corollary 4.3. Let Assumptions 2.1, 3.2, 4.1, and 4.2 hold. Then, we have

4.2 The bias component of estimation risk

For the bias component of estimation risk, we can obtain a similar result with 4.1 as follows:

Assumption 4.4. E β [ ββ ⊤ ] = r 2 I / p , where r 2 : = E β [ ∥ β ∥ 2 ] < ∞ and β is independent of X .

Under Assumption 4.4, we have the expected bias (conditional on X ) as follows:

where si are the eigenvalues of ˆ Σ ∈ R p × p and rank( ˆ Σ ) = rank( X ) = n under Assumption 4.1.

Thanks to Theorem 3.5 and equation 5, we obtain the following corollary for estimation risk.

Corollary 4.5. Let Assumptions 2.1, 3.2, 4.1, and 4.4 hold. Then, we have

where F A ( s ) : = 1 n P n i = 1 1 { λ i ( A ) ≤ s } is the empirical spectral distribution of a matrix A and λ 1 ( A ) , λ 2 ( A ) , · · · , λ n ( A ) are the eigenvalues of A.

The proof of Corollary 4.5 is in the appendix.

4.2.1 Asymptotic analysis of estimation risk

To study the asymptotic behavior of estimation risk, we follow the previous approaches [13, 17]. First, we define the Stieltjes transform as follows:

Definition 4.6. The Stieltjes transform sF ( z ) of a df F is defined as:

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

281

282

283

10 0

10 1

10 2

=

p

/

n

10

2

10

1

10 0

10 1

10 2

Prediction Risk

variance

bias

bias (theory)

10 0

10 1

10 2

=

p

/

n

10

2

10

1

10 0

10 1

10 2

Estimation Risk

variance

bias

bias (theory)

We are now ready to investigate the asymptotic behavior of the mean squared estimation error with the following theorem:

Theorem 4.7. [25, Theorem 1.1] Suppose that the rows { xi } n i = 1 in X are i.i.d. centered random vectors with E [ x 1 x ⊤ 1 ] = Σ and that the empirical spectral distribution F Σ ( s ) = 1 p P p i = 1 1 { τ i ≤ s } of Σ converges almost surely to a probability distribution function H as p →∞ . When p / n → γ > 0 as n , p →∞ , then a.s., F XX ⊤ / n converges vaguely to a df F and the limit s ∗ : = lim z ↘ 0 sF ( z ) of its Stieltjes transform sF is the unique solution to the equation:

This theorem is a direct consequence of Theorem 1.1 in Silverstein and Bai [25]. Then, from Corollary 4.5, we can write the limit of estimation risk as follows:

Corollary 4.8. Let Assumptions 2.1, 3.2, 4.1, and 4.4 hold. Then, under the same assumption as Theorem 4.7, as n , p →∞ and p / n → γ , where 1 < γ < ∞ is a constant, we have

Here, the limit s ∗ of the Stieltjes transform sF is highly connected with the shape of the spectral 273 distribution of Σ . For example, in the case of isotropic features ( Σ = I ), i.e., dH ( τ ) = δ ( τ -1) d τ , we 274 have s ∗ iso = ( γ -1) -1 from 1 -1 γ = 1 1 + s ∗ iso . In addition, if Ω = σ 2 I , then the limit of the mean squared 275 error is exactly the same as the expression for γ > 1 in equation (10) of Hastie et al. [17, Theorem 1]. 276 This is because prediction risk is the same as estimation risk when Σ = I . 277 Remark 4.9 . Generally, if the support of H is bounded within [ cH , CH ] ⊂ R for some positive constants 278 0 < cH ≤ CH < ∞ , then we can observe the double descent phenomenon in the overparameterization 279 regime with lim γ ↘ 1 s ∗ = ∞ and lim γ →∞ s ∗ = 0 with s ∗ = Θ GLYPH<16> 1 γ -1 GLYPH<17> from the following inequalities: 280

In fact, a tighter lower bound is available:

where µ H : = E τ ∼ H [ τ ], i.e., the mean of distribution H . The proofs of equation 7 and equation 8 are given in the supplementary appendix.

We conclude this paper by plotting the 'descent curve' in the overparameterization regime in Figure 284 4. On one hand, the expected variance ( × ) perfectly matches its theoretical counterpart ( GLYPH<8> ) and goes 285 to zero as γ gets large. On the other hand, the bias term is bounded even if γ →∞ . The appendix 286 contains the experimental details for all the figures. 287

288

289

290

291

References

Alberto Abadie, Alexis Diamond, and Jens Hainmueller. Synthetic control methods for comparative case studies: Estimating the e ff ect of california's tobacco control program. Journal of the American Statistical Association , 105(490):493-505, 2010.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in 292 linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 293 2020. 294

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to 295 understand kernel learning. In International Conference on Machine Learning , pages 541-549. 296 PMLR, 2018. 297

298

299

300

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-o ff . Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

Raphaël Berthier, Francis Bach, and Pierre Gaillard. Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. Advances in Neural Information Processing Systems , 33:2576-2586, 2020.

Xin Chen, Yicheng Zeng, Siyue Yang, and Qiang Sun. Sketched ridgeless linear regression: The role of downsampling. In International Conference on Machine Learning , pages 5296-5326. PMLR, 2023.

Geo ff rey Chinot and Matthieu Lerasle. On the robustness of the minimum ℓ 2 interpolator. Bernoulli , 2023. forthcoming, available at https://www.bernoullisociety.org/ publications/bernoulli-journal/bernoulli-journal-papers .

Geo ff rey Chinot, Matthias Lö ffl er, and Sara van de Geer. On the robustness of minimum norm interpolators and regularized empirical risk minimizers. The Annals of Statistics , 50(4):2306 2333, 2022. doi: 10.1214 / 22-AOS2190. URL https://doi.org/10.1214/22-AOS2190 .

AP Dawid. Spherical matrix distributions and a multivariate model. Journal of the Royal Statistical Society: Series B (Methodological) , 39(2):254-261, 1977.

AP Dawid. Extendibility of spherical matrix distributions. Journal of Multivariate Analysis , 8 (4):559-566, 1978.

AP Dawid. Some matrix-variate distribution theory: Notational considerations and a Bayesian application. Biometrika , pages 265-274, 1981.

Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics , 46(1):247 -279, 2018. doi: 10.1214 / 17-AOS1549. URL https://doi.org/10.1214/17-AOS1549 .

Arjun K Gupta and Daya K Nagar. Matrix variate distributions , volume 104. CRC Press, 1999. 324

Bruce Hansen. Econometrics . Princeton University Press, 2022. 325

Bruce E Hansen. The risk of James-Stein and Lasso shrinkage. Econometric Reviews , 35(8-10): 326 1456-1470, 2016. 327

328

329

330

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

331

332

W. James and Charles Stein. Estimation with quadratic loss. In Proc. 4th Berkeley Sympos. Math. Statist. and Prob., Vol. I , pages 361-379. Univ. California Press, Berkeley, Calif., 1961.

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization. The Journal of Machine Learning Research , 21(1):6863-6878, 2020.

Zeng Li, Chuanlong Xie, and Qinwen Wang. Asymptotic normality and confidence intervals for prediction risk of the min-norm least squares estimator. In International Conference on Machine Learning , pages 6533-6542. PMLR, 2021.

Yuan Liao, Xinjie Ma, Andreas Neuhierl, and Zhentao Shi. Economic forecasts using many noises. arXiv preprint arXiv:2312.05593, 2023. URL https://arxiv.org/abs/2312. 05593 .

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75(4):667-766, 2022.

Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge (less) regression under general source condition. In International Conference on Artificial Intelligence and Statistics , pages 3889-3897. PMLR, 2021.

Steven Ruggles, Sarah Flood, Matthew Sobek, Daniel Backman, Annie Chen, Grace Cooper, Stephanie Richards, Renae Rodgers, and Megan Schouweiler. IPUMS USA: Version 15.0 [dataset]. https://doi.org/10.18128/D010.V15.0 , 2024. Minneapolis, MN: IPUMS.

Jack W Silverstein and ZD Bai. On the empirical distribution of eigenvalues of a class of large dimensional random matrices. Journal of Multivariate analysis , 54(2):175-192, 1995.

Jann Spiess, Guido Imbens, and Amar Venugopal. Double and single descent in causal inference with an application to high-dimensional synthetic control. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id= dL0GM9Wwtq .

Alexander Tsigler and Peter L. Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research , 24(123):1-76, 2023. URL http://jmlr.org/papers/v24/ 22-1398.html .

Denny Wu and Ji Xu. On the optimal weighted ℓ 2 regularization in overparameterized linear regression. Advances in Neural Information Processing Systems , 33:10112-10123, 2020.

Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Benign overfitting of constant-stepsize SGD for linear regression. In Conference on Learning Theory , pages 4633-4635. PMLR, 2021.

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

Appendix

A Details for drawing Figure 1

To draw Figure 1, we use a sample extract from American Community Survey (ACS) 2018. To have a relatively homogeneous population, the sample extract is restricted to white males residing in California with at least a bachelor's degree. We consider a demographic group defined by their age in years (between 25 and 70), the type of degree (bachelor's, master's, professional, and doctoral), and the field of degree (172 unique values). Then, we compute the average of log hourly wages for each age-degree-field group (all together 7,073 unique groups in the sample). We treat each group average as the outcome variable (say, ya , d , f ) and predict group wages by various group-level regression models where the regressors are constructed using the indicator variables of age, degree, and field as well as their interactions: that is,

For the regressors xa , d , f , we consider 7 specifications ranging from 209 to 2,183 regressors:

Spec. 1 ( p = 209): dummy variables for age (say, xa ) + dummy variables for the type of degree (say, xd ) + dummy variables for the field of degree (say, xf ),

Spec. 2 ( p = 391): Spec. 1 + all interactions between xd and xa ,

Spec. 3 ( p = 598): Spec. 1 + all interactions between xd and xf ,

Spec. 4 ( p = 778): Spec. 1 + all interactions between xd and xa + all interactions between xd and xf ,

Spec. 5 ( p = 1640): Spec. 1 + all interactions between xd and xa + all interactions between xa and xf ,

Spec. 6 ( p = 1754): Spec. 1 + all interactions between xd and xf + all interactions between xa and xf ,

Spec. 7 ( p = 2182): Spec. 1 + all three-way interactions among xa , xd and xf .

Here, the dummy variable are constructed using one-hot encoding. We randomly split the sample into the train and test samples with a ratio of 1 : 4. The resulting sample sizes are 1,415 and 5,658, respectively. To understand the role of non-i.i.d. regressor errors, we add the artificial noise to the training sample: that is, we compute the ridgeless least squares estimator using the training sample of (˜ ya , d , f , x ⊤ a , d , f ) ⊤ , where ˜ ya , d , f = ya , d , f + ua , d , f . Here, the artificial noise ua , d , f has the form

where ea , d , f ∼ N (0 , σ 2 ), independently across age ( a ), degree ( d ) and field ( f ); ef is the average of another independent N (0 , σ 2 ) variable within f (hence, ef is identical for each value of f ) and thus the source of clustered errors; and c ∈ { 0 , 0 . 25 , 0 . 5 , 0 . 75 } is a constant that will be varied across the experiment. As c gets larger, the noise has a larger share of clustered errors but the variance of the overall regression errors ( ua , d , f ) remains the same: in other words, var( ua , d , f ) = σ 2 for each value of c . Figure 1 was generated with σ = 0 . 5 by generating the artificial noise only once.

B Details for drawing Figures 2, 3, and 4

To draw Figure 2, 3, and 4, we sample { xi } n i = 1 from N (0 , Σ ) with Σ = U Σ D Σ U ⊤ Σ where U Σ is an orthogonal matrix random variable, drawn from the uniform (Haar) distribution on O ( p ), and D Σ is a diagonal matrix with its elements di = | zi | / P p i = 1 | zi | being sampled with zi ∼ N (0 , 1) for each i = 1 , 2 , · · · , p . With this general anisotropic Σ , the term E X [Tr( Λ -1 )] / p is somewhat larger than µ -1 H s ∗ iso = ( γ -1) -1 which is 1 in Figure 2 and 3 since µ H = 1 and γ = 2. For example, in Figure 2, when σ 2 = 1 , ρ 2 = 0, we have Tr( Ω ) / n = 1 but Tr( Ω ) E X [Tr( Λ -1 )] / ( np ) > 1.

In Figure 4, we fix n = 50 and use p = n γ for γ ∈ [1 , 100].

To compute the expectations of E X [Var( ˆ β | X )] and E X [Tr( Λ -1 )] over X , we sample NX sam407 ples of X 's, X 1 , X 2 , · · · , XNX . Moreover, to compute the expectation over ε in Var( ˆ β | Xi ) ≡ 408 Tr GLYPH<16> E ε [ ˆ β ˆ β ⊤ ] -E ε [ ˆ β ] E ε [ ˆ β ] ⊤ GLYPH<17> , we sample N ε samples of ε 's, ε 1 , ε 2 , · · · , ε N ε for each realization Xi . 409 To be specific, 410

where ˆ β i , j = arg min β {∥ b ∥ : Xib -yi , j = 0 } , yi , j = Xi β + ε j , and λ k ( XiX ⊤ i ) is the k -th eigenvalue of 411 XiX ⊤ i . We can do similarly for the variance part of the prediction risk. 412

Figure 5 shows an additional experimental result. 413

0.1

0.2

0.3

0.4

0.5

2

1

0.1

0.2

0.3

0.4

0.5

2

2

Prediction Risk - Clustered Errors

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.1

0.2

0.3

0.4

0.5

2

1

0.1

0.2

0.3

0.4

0.5

2

2

Estimation Risk - Clustered Errors

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.65

C Proofs omitted in the main text 414

Proof of Lemma 3.3. For a given A ∈ S , since A -1 ∈ S , we have Z d = A -1 Z : = ˜ Z and 415

This naturally leads to

where the first equality comes from Fubini's theorem and the integrability of f . □ 416

Proof of Theorem 3.4. Since ˆ β = X † y , we have Cov( ˆ β | X ) = X † Cov( y | X ) X †⊤ = X † Ω X †⊤ , which 417 leads to the following expression for the variance component of prediction risk: 418

where S = Σ 1 / 2 , T = Ω 1 / 2 , and B = S X † . Using the singular value decomposition (SVD) of B and T , 419 respectively, we can rewrite this as follows: 420

where B = UDV ⊤ and T = UTDTV ⊤ T with orthogonal matrices U , V , UT , VT , and diagonal matrices 421 D , DT . Now we need to compute the alignment V ⊤ UT of the right-singular vectors of B with the 422

left-eigenvectors of T . 423

427

428

429

430

431

432

433

434

435

436

437

438

439

where γ i j : = ⟨ V : i , ( UT ) : j ⟩ 2 ≥ 0, Γ ( X ) : = ( γ i j ) i , j ∈ R n × n and λ ( A ) ∈ R n is a vector with its element 424 λ i ( A ) as the i -th largest eigenvalue of A . 425

Therefore, we can rewrite the variance as Var Σ ( ˆ β | X ) = a ( X ) ⊤ Γ ( X ) b with 426

where v ( i ) : = V : i and u ( j ) : = ( UT ) : j . Note that the alignment matrix Γ ( X ) is a doubly stochastic matrix since P j γ i j = P i γ i j = 1 and 0 ≤ γ i j ≤ 1.

Now, we want to compute the expected variance. To do so, from Lemma 3.3 with S = O ( n ), we can obtain

where ν is the unique uniform distribution (the Haar measure) over the orthogonal matrices O ( n ). For an orthogonal matrix O ∈ O ( n ), we have

since S ( OX ) † = S X † O ⊤ = BO ⊤ = UD ( OV ) ⊤ . Here, ( OX ) † = X † O ⊤ follows from the orthogonality of O ∈ O ( n ). Since the Haar measure is invariant under the matrix multiplication in O ( n ), if we take the expectation over the Haar measure, then we have

Here, for a given j , we can choose a matrix O ( j ) ∈ O ( n ) such that its first column is u ( j ) and O ( j ) ⊤ u ( j ) = e 1, then ¯ Γ ( X ) i j is independent of j (say ¯ Γ ( X ) i j = α i ). Since Γ ( X ) is doubly stochastic, so is ¯ Γ ( X ) and we have P n j = 1 ¯ Γ ( X ) i j = n α i = 1 which yields ¯ Γ ( X ) i j = α i = 1 / n , regardless of the distribution of V ; thus, ¯ Γ ( X ) = 1 n J , where Jij = 1( i , j = 1 , 2 , · · · , n ).

Therefore, we have the expected variance as follows: 440

441

Proof of Corollary 4.5. Note that 442

Then, the desired result follows directly from equation 5. 443

□

□

Proof of equation 4. The bias term of the prediction risk can be expressed as follows: 444

where ˆ Σ = X ⊤ X / n . Here, the fourth equality comes from the equation 445

446

447

448

449

450

451

□

Proof of equation 7. The RHS of equation 6 is bounded above by R 1 1 + cH s ∗ dH ( τ ) = 1 1 + cH s ∗ , and thus 1 -1 γ ≤ 1 1 + cH s ∗ , which yields s ∗ ≤ c -1 H 1 γ -1 . We can similarly prove the other inequality in equation 7 with a lower bound 1 1 + CHs ∗ on the RHS of equation 6. □

Proof of equation 8. To further explore the inequalities equation 7, we rewrite equation 6 from Theorem 4.7 as follows:

Here, since g ( t ; s ) is convex with respect to t > 0 for a given s > 0, by Jensen's inequality, we then have

where µ H = E τ ∼ H [ τ ]. Therefore, the limit Stieltjes transform s ∗ in the anisotropic case should be larger 452 than µ -1 H s ∗ iso of the isotropic case to satisfy E τ ∼ H [ g ( τ ; s ∗ )] = 1 -γ -1 since g ( t ; s ) is a decreasing function 453 with respect to s ≥ 0 when t > 0. This leads to a tighter lower bound s ∗ ≥ µ -1 H s ∗ iso = µ -1 H ( γ -1) -1 than 454 equation 7 because µ H ≤ CH . □ 455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and / or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the last paragraph of Introduction.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational e ffi ciency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We explicitly mention the assumptions and provide a complete proof (see Appendix).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it a ff ects the main claims and / or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the code (supplementary material) and all the information needed to reproduce the main results (Appendix).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and / or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might su ffi ce, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with su ffi cient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code (supplementary material) and all the information needed to reproduce the main results (Appendix).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting / Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the details necessary to understand the results (Appendix).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: To support our claim, the experiments in the paper do not need error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train / test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide su ffi cient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our models are linear regression models which do not require much resources. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is a theoretical work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the e ffi ciency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is a theoretical work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing e ff ective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith e ff ort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We used some datasets and properly credited the creators of assets (cf. ACS 2018 [24]).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the code with the details as a supplementary material. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset / code / model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval / review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines prediction and estimation risk of ridgeless least squares estimator in the setting of a general error structure. The iid assumption on the error structure is often not valid in settings such as time series data , panel data, grouped data etc. The current paper introduces a theoretical framework which investigates the variance component estimation of both prediction and estimation risks in the above mentioned data settings. The benefits of overparametrization which has been seen in iid context has been shown to exist in the dependent error structure context as well.\n\nSTRENGTHS:\nFollowing are the strengths of the paper-\n\n- Investigation of prediction and estimation risk under non i.i.d. regressor errors with specific focus on time series and cluster data\n\n- Explicit quantification of the variance component of both the risks (as mentioned above) which depends on the trace of the error covariance matrix and the trace of a function of design matrix as a separable product.\n\n- Explicit analysis of the variance and bias term of both the risks (as mentioned above) in the high-dimensional asymptotics\n\n- Well constructed numerical experiments to support the theory\n\nWEAKNESSES:\nFollowing are the weaknesses of the paper\n\n- The theoretical results particularly the bias component analysis section could have been more rigorous and better written. There are some notational discrepancies and theoretical inconsistencies.\n\n- Some remarks following theorem 3.4 and 3.5 where the design matrix $X$ has a known distribution say Gaussian would have been useful  examples to get insight on the results proved in the theorems\n\n- Some notations such as $a(X)$ and $b$ used in theorem 3.4 have been clarified later in the appendix. It would be better to introduce them in the sketch of the proof if you are using them anyway there.\n\nQUESTIONS:\nI have the following queries for the authors\n\n- It would be interesting to see how do the prediction risk and the estimation risk behave in Figure 1 and 2 respectively as $n$ and $p$ increase? In other words can we infer any pattern from the results shown in Theorem 3.4 and 3.5 ?\n\n- In section 4 in the bias component analysis, before assumption 4.1 the authors mention each $x_i$ has a positive definite covariance matrix and is independent of each other. Why the dependent $x_i$'s are not considered?\n\n- In assumption 4.2, the expectation has subscript $\\beta$ and there is an assumption also that $\\beta$ is independent of X. Are the authors assuming $\\beta$ to be random. What does this mean?\n\n- What is $S$ in equation (4)? Did the authors define it earlier?\n\n- In equation (5) if $p >> n$, does this mean the bias go away?\n\n- In corollary 4.8, I thought the double asymptotics on $n$ and $p$ have already been used. Then what is the limit wrt $n$ mean in the second term on the right hand side?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper explores the prediction risk and estimation risk of the ridgeless least squares estimator under more general assumptions on regression errors. It highlights the benefits of overparameterization in a realistic setting that allows for clustered or serial dependence. The paper establishes that the estimation difficulties associated with the variance components of both risks can be summarized through the trace of the variance-covariance matrix of the regression errors. The findings suggest that the benefits of overparameterization can extend to time series, panel, and grouped data. The paper is a theoretical work that discusses various aspects of linear regression models, providing details on the assumptions and proofs for the theoretical results presented. It also includes information on the experimental setting and provides code and instructions for reproducing the main results.

STRENGTHS:
This study addresses an important research gap by considering more realistic assumptions on regression errors. It provides exact finite-sample characterizations of the variance components of prediction and estimation risks, includes numerical experiments that validate the theoretical results, and demonstrates the relationship between the expected variance and the covariance of the regression errors. Additionally, it analyzes the bias components of prediction and estimation risks, offers a comprehensive overview of linear regression models covering various theoretical aspects, and provides detailed proofs for the theoretical results, ensuring the validity of their claims.

WEAKNESSES:
Is it possible to provide validation on large-scale data?

QUESTIONS:
Refer to weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
The paper investigates the properties of minimum norm (ridgeless) interpolation least squares estimators, analyzing prediction risk and estimation risk under broader regression error assumptions, including clustered or serial dependence. This diverges from the typical assumption of i.i.d. errors with zero mean and common variance. The paper shows that the challenges in estimating the variance components of prediction and estimation risks can be captured by the trace of the variance-covariance matrix of the regression errors.

STRENGTHS:
1. The paper provides a more general theoretical analysis of minimum norm interpolation least squares estimators, going beyond the restrictive i.i.d. error assumption.

2. The paper suggests that the benefits of overparameterization can extend to a wider range of regression settings, including time series, panel, and grouped data.

WEAKNESSES:
While the paper examines broader error structures, it might not fully grasp the complexity of real-world regression challenges, which could involve even more intricate patterns of error dependence.

QUESTIONS:
1. Is it possible to remove the assumption that $\varepsilon$ is independent of $X$?

2, Is it overly restrictive to demand that the design matrix $X$ is left-spherical and has a rank of $n$ almost everywhere?

LIMITATIONS:
The authors have addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

comments: SUMMARY:
The paper considers the ridgeless least-squares estimator, and derives its prediction and estimation risk. One of the assumptions used is that the expectation of the noise variance matrix is finite and positive-definite. This is more general than the assumption that this expectation is some positive multiple of the identity matrix.

STRENGTHS:
- The paper has an easy-to-follow introduction that motivates the need to derive theoretical results under general assumptions on regression errors.
- Related works are sufficiently discussed. The most relevant papers are those of Chinot et al. [9] and Chinot and Lerasle [8], which are based on different noise assumptions that this paper makes.
- The technical presentation is clear with examples and figures to help the reader understand the notations and results.

WEAKNESSES:
The major concern I have is whether the paper makes sufficient technical contributions. Even with the more general assumption on noise (Assumption 2.1), the technical change in the proofs seems very small compared to prior work. For example, the proof of Theorem 3.4 is short and relatively straightforward (and this might further simplify if we make Gaussian assumptions on data rather than left-spherical assumptions. Gaussian assumptions are what I like to make personally). It is always nice to have short and concise proofs whenever possible, but this might also indicate that the paper is not very technically solid.

QUESTIONS:
N.A.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13