On the Role of Optimization in Double Descent: A Least Squares Study

1 Introduction . Deep Neural Networks optimized by Gradient Descent ( GD ) methods have shown amazing versatility across a large range of domains . One of their most intriguing features is their ability to perform better with scale . Indeed , some of the most impressive results [ see e.g . Brock et al. , 2021 , Brown et al. , 2020 , Senior et al. , 2020 , Schrittwieser et al. , 2020 , Silver et al. , 2017 , He et al. , 2016 and references therein ] have been obtained often by exploiting this fact , leading to neural network models that have at least as many parameters as the number of examples in the dataset they are trained on . Empirically , the limitation on the model size seems to be mostly imposed by hardware or compute . From a theoretical point of view , however , this property is quite surprising and counter-intuitive , as one would expect that in such extremely overparameterized regimes the learning would be prone to overfitting [ Hastie et al. , 2009 , Shalev-Shwartz and Ben-David , 2014 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently Belkin et al . [ 2019 ] proposed Double Descent ( DD ) phenomenon as an explanation . They argue that the classical view of overfitting does not apply in extremely overparameterized regimes , which were less studied prior to the emergence of the deep learning era . The classical view in the parametric learning models was based on error curves showing that the training error decreases monotonically when plotted against model size , while the corresponding test errors displayed a U-shape curve , where the model size for the bottom of the U-shape was taken to be a “ sweet spot ” [ cf . Belkin et al. , 2019 , Fig . 1A ] achieving an ideal trade-off between model size and generalization . Here , larger model sizes than the aforementioned ideal were thought to constitute the ‘ overfitting ’ regime , since the gap between the test error and the training error increased . The classical U-shape test error curve dwells in what is now called the under-parameterized regime , where the model size is smaller than the size of the dataset . Arguably , the restricted model sizes used in the past were tied to the available computing power . By contrast , it is common nowadays for model sizes to be larger than the amount of available data , which is called the over-parameterized regime . The divide between these two regimes is marked by a point where model size matches dataset size , which can be called the interpolation threshold [ cf . Belkin et al. , 2019 , Fig . 1B ] . The work of Belkin et al . [ 2019 ] argues that as model size grows beyond the interpolation threshold , one will observe a second descent of the test error that asymptotes in the limit to smaller values than those in the underparameterized regime , which indicates better generalization rather than overfitting . To some extent this was already known in the nonparametric learning where model complexity scales with the amount of data by design ( such as in nearest neighbor rules and kernels ) , yet one can generalize well and even achieve statistical consistency [ Györfi et al. , 2002 ] . This has lead to a growing body of works trying to identify the mechanisms behind DD , to which the current manuscript belongs too . We refer the reader to Section 2 , where the related literature is discussed . Similar to these works , our goal is also to understand the cause of DD . In this paper we explore a finite-time GD solution to least squares problems that allows us to work with analytic expressions for all the quantities involved . Fig . 1 provides a summary of our findings . In particular , it shows the behaviour of the risk ( or the expected loss of the algorithm ) in a linear regression setting with random inputs and noise-free labels , for which in Section 3 we prove a bound that has the form ( ( 1− αλ̂+min ) 2T + 1√ n ) ‖w ? ‖2 . Here labels are generated as Y = X > w ? for some unknown ground truth model w ? ∈ Rd and random inputsX ∈ Rd on a unit sphere ( ‖X‖M = √ X > MX = 1 for a positive matrixM ) . The factor α is a constant learning rate , and n is the number of examples in the training set . Note that the feature dimension d coincides with the number of parameters in this particular setting , hence d > n is the overparameterized regime . The quantity λ̂+min is the smallest positive eigenvalue of the sample covariance matrix of the inputs , and it is of special importance : We observe that the shape of the bound is controlled by λ̂+min via the exponential term , which increases as λ̂ + min decreases while it decreases as λ̂+min grows , resulting in a peaked shape of the bound . In Fig . 1 we observe a peaking behavior not only in the risk but also in the quantity that we label ‘ optimization error ’ which is this special term of the risk bound that is purely related to optimization . The peaking behaviour of the risk—the Mean Squared Error ( MSE ) in case of the squared loss—was observed and studied in a number of settings [ Belkin et al. , 2019 , Mei and Montanari , 2019 , Derezinski et al. , 2020 ] sometimes relating it to λ̂+min , however , to the best of our knowledge the connection between the peaking behavior and optimization so far received less attention . In the absence of label noise , we conclude that DD manifests due to the optimization process . On the other hand , when label noise is present , in addition to the optimization effect , λ̂+min also has an effect on the generalization error . Our contributions : Our main theoretical contribution is presented in Section 3 . In particular , Section 3.1 focuses on the noise-free least squares problem , Section 3.2 adds noise to the problem , and Section 3.3 deals with concentration of the sample-dependent λ̂+min around its population counterpart . The essential idea of the proof of our main result is outlined in Section 5 . Sections 4 and 6 provide some discussion on the implications of our findings and an empirical exploration of the question whether simple neural networks have a similar behaviour than that of the least squares setting . Notation : The linear algebra and analysis notations used in this work are defined in Appendix A . We briefly mention here that we denote column vectors and matrices with small and capital bold letters , respectively , e.g . α = [ α1 , α2 , . . . , αd ] > ∈ Rd and A ∈ Rd1×d2 . Singular values of a rectangular matrix A ∈ Rn×d are denoted by smax ( A ) = s1 ( A ) ≥ . . . ≥ sn∧d ( A ) = smin ( A ) . The rank of A is r = max { k | sk ( A ) > 0 } . The eigenvalues of a Positive Semi-Definite ( PSD ) matrix M ∈ Rd×d are non-negative and are denoted λmax ( M ) = λ1 ( M ) ≥ . . . ≥ λd ( M ) = λmin ( M ) , while λ+min ( M ) denotes the smallest positive ( non-zero ) eigenvalue . 2 Related Work . To the best of our knowledge DD was first mentioned by Vallet et al . [ 1989 ] , who observed the phenomenon for the Moore-Penrose pseudo-inverse solutions ( see [ Loog et al. , 2020 ] for a brief history ) . The recent literature revisited DD mainly considering ordinary least squares with the explicit solution given by the Moore-Penrose pseudo-inverse . These works have focused on instance-specific settings ( making distributional assumptions on the inputs ) while arguing when the analytic pseudoinverse solutions yield DD behaviour [ Belkin et al. , 2020 ] . This was later extended to a more general setting showcasing the control of DD by the spectrum of the feature matrix [ Derezinski et al. , 2020 ] . In this paper we also argue that the spectrum of the covariance matrix has a critical role in DD , however we take into account the effect of GD optimization , which was missed by virtually all the previous literature due to their focusing on analytic solutions . The effect of the smallest non-zero eigenvalue on DD , through a condition number , was briefly noticed by Rangamani et al . [ 2020 ] . In this work we carry out a more comprehensive analysis and show how the excess risk of GD is controlled the smallest eigenvalue . In particular , λ̂+min has a U-shaped behaviour as the number of features increases , and we give a high-probability characterization of this behavior when inputs are subgaussian . To some extent , this is a non-asymptotic manifestation of the Bai-Yin law , whose connection to DD in an asymptotic setting was noted by Hastie et al . [ 2019 , Theorem 1 ] . Some interest was also dedicated to the effect of bias and variance of DD [ Mei and Montanari , 2019 ] in the same pseudo-inverse setting , while a more involved fine-grained analysis was later carried out by Adlam and Pennington [ 2020 ] . In this work we focus on the influence of the optimization error , which is complementary to the bias-variance effects ( typically we care about it once optimization error is negligible ) . The DD behaviour was also observed beyond least squares , for instance in neural networks and other interpolating models [ Belkin et al. , 2019 ] . To some extent a formal connection to neural networks was first made by Mei and Montanari [ 2019 ] who studied the asymptotic behaviour of the risk under the random feature model , when n , dinput , dRF →∞ while having ndinput and dRF dinput fixed . Later on , with the popularity of the Neural Tangent Kernel ( NTK ) [ Jacot et al. , 2018 ] this connection became clearer as , within the NTK theory interpretation , shallow neural networks can be paralleled with kernelized predictors [ Bartlett et al. , 2021 ] . A detailed experimental study of DD in deep neural networks was carried out by [ Nakkiran et al. , 2019 ] , who showed that various forms of regularization mitigate DD . Our results imply that ` 2-regularization in least squares ( ridge regression ) makes λ̂+min well-controlled and therefore mitigates the peak . The same should carry over to the early-stopping in least squares [ Yao et al. , 2007 ] and to early stopping in shallow neural networks [ Ji et al. , 2021 , Richards and Kuzborskij , 2021 , Kuzborskij and Szepesvári , 2021 ] . In this work , we explain DD in least squares solution obtained by GD through the spectrum of the features , where optimization error has a visible role . While we do not present formal results for neural networks , we nevertheless empirically investigate on shallow neural networks whether our conclusions extend to neural nets as would be suggested by the NTK theory . 3 Excess Risk of the Gradient Descent Solution . We focus on the Gradient Descent ( GD ) algorithm , treated as a measurable map A : S × Rd → Rd where S = Zn is the space of size-n training sets . Consider a learning problem where hypotheses are parameterized by weight vectorsw ∈ Rd . The risk of a givenw is L ( w ) , and the empirical risk ofw over a sample S is L̂S ( w ) . GD is applied to minimize the empirical risk . Given a training set S ∈ S and an initialization point w0 ∈ Rd , we write AS ( w0 ) to indicate the output obtained recursively by the standard GD update rule with some fixed step size α > 0 , i.e . AS ( w0 ) = wT , where wt = wt−1 − α∇L̂S ( wt−1 ) , t = 1 , . . . , T . It is well-known that in the overparameterized regime ( d > n ) , GD is able to achieve zero empirical loss . Therefore , rather than focusing on the generalization gap L ( AS ( w0 ) ) − L̂S ( AS ( w0 ) ) it is more natural to compare the loss of AS ( w0 ) to that of the best possible predictor . Thus , we consider the excess risk defined as ET = L ( AS ( w0 ) ) − L ( w ? ) , w ? ∈ argmin w∈Rd L ( w ) . ( 1 )

Summary: This paper studies the double descent (DD) phenomena for least squares and proves upper bounds for excess risk of gradient descent. For both cases with and without label noise, the upper bound involves a term $E[(1-\alpha \hat{\lambda}\_{\min}^+)^{2T}]$, where $T$ is the training time, $\alpha$ is the learning rate, $\hat{\lambda}\_{\min}^+$ is the smallest positive eigenvalue of the covariance matrix of training inputs.  The most important observation in this paper is that the peak of the double descent curve around $d \approx n$ can be related to $\hat{\lambda}\_{\min}^+$. By a non-asymptotic version of Bai-Yin limit, $\hat{\lambda}\_{\min}^+$ could be very small when  $d \approx n$, and thus $E[(1-\alpha \hat{\lambda}\_{\min}^+)^{2T}]$ can be very close to $1$ when $T$ is finite.  Experiments show that $\hat{\lambda}\_{\min}^+$ for intermediate features correlates with the double descent curve of test error.