Title: On the Role of Optimization in Double Descent: A Least Squares Study
Authors: Ilja Kuzborskij
Csaba Szepesvari
Omar Rivasplata
Amal Rannen-Triki
Razvan Pascanu
Venue: NeurIPS 2021
Keywords: Double Descent
Optimization Error
Excess Risk
Generalization
Least Squares

Abstract: Empirically it has been observed that the performance of deep neural networks steadily improves with increased model size, contradicting the classical view on overfitting and generalization. Recently, the double descent phenomenon has been proposed to reconcile this observation with theory, suggesting that the test error has a second descent when the model becomes sufficiently overparameterized, as the model size itself acts as an implicit regularizer. In this paper we add to the growing body of work in this space, providing a careful study of learning dynamics as a function of model size for the least squares scenario. We show an excess risk bound for the gradient descent solution of the least squares objective. The bound depends on the smallest non-zero eigenvalue of the sample covariance matrix of the input features, via a functional form that has the double descent behaviour. This gives a new perspective on the double descent curves reported in the literature, as our analysis of the excess risk allows to decouple the effect of optimization and generalization error. In particular, we find that in the case of noiseless regression, double descent is explained solely by optimization-related quantities, which was missed in studies focusing on the Moore-Penrose pseudoinverse solution. We believe that our derivation provides an alternative view compared to existing works, shedding some light on a possible cause of this phenomenon, at least in the considered least squares setting. We empirically explore if our predictions hold for neural networks, in particular whether the spectrum of the sample covariance of features at intermediary hidden layers has a similar behaviour as the one predicted by our derivations in the least squares setting.


Review discussion:
I think that DD is still not well understood, and studying it is important and interesting.
Also, the connection between DD and certain parameters of the data or optimization is an interesting observation.
On the other hand, I think this paper makes a weak case for why these parameters actually affect DD, especially on the experimental part, and vaguely connects the theory presented in the paper with the experiments. In more details:
For the theoretical part:
1) I think the main idea is nice, studying linear regression in the case where d>n (more parameters than data examples), which is a less studied case in the literature.
2) Viewing section 2 as a whole, it is hard to grasp a single conclusion on how the excess risk behaves. It would have been much clearer if there was a single theorem that shows a bound on the excess risk when d goes to infinity. In both theorems 2 & 3, there are terms where it is not clear how they behave (e.g. E[||w^*||^2]_{I-M}).
3) To emphasize the previous point, two parameters are considered (n and d), and throughout the section it is unclear which one is considered to be larger. I assume the authors meant that d>n (since this is the over-parameterized case), but in line 163 (and Appendix F) it seems the E[||w^*||^2]_{I-M} term only goes to zero with large n, as well as the noise in Theorem 3. What happens in the over-parameterized regime when n is fixed and d goes to infinity? I think this is the interesting case when studying DD.
4) Section 3 is very unclear. What are the assumptions made in the equation after line 216? What are the main conclusions of this section? There are very long paragraphs, where the bottom line isn’t clear.
For the empirical part
1) I think this is the main weak point of the paper. The connection between the experiments and the theory is not well grounded.
Also, the experiments are very lacking and not convincing enough.
2) To emphasize, the theoretical part focuses on a certain eigenvalue of the data matrix. This eigenvalue has no connection at all to the optimization process and is fixed.
On the other hand, in the theoretical part the authors consider the eigenvalue of the output of some intermediary layer, which changes through the optimization process. Why not take the eigenvalue of the data itself? And if the authors already decide to take the eigenvalue of an intermediary layer, I suggest taking the eigenvalue of all layers and comparing them.
3) The only datasets used are MNIST and FashionMNIST which I think are very small and not very hard to learn, even with simple models. To give a more convincing argument I suggest using larger and harder datasets (e.g. CIFAR 10/100, ImageNet).
4) The models considered are very small, e.g. one hidden layer and three hidden layers. I suggest experimenting on more realistic models and see if this eigenvalue has the same effect.
Also, to experiment on other architectures such as ConvNets.
5) There are no experiments on regularization (e.g. L1/L2, Dropout, BatchNorm). Regularization techniques might mitigate the effect of this eigenvalue on the DD phenomenon. Without these experiments, it is hard to evaluate whether this phenomenon is due to the effects of this eigenvalue, or some other property that can be mitigated using regularization techniques.
6) In line 308 it says that ablation studies are in Appendix H. The only experiment in Appendix H is a three-layer network with or without skip connections. I suggest adding significantly more ablation studies to give a more convincing case.
Some more issues:
1) Why do the authors show the experiments for a different number of iterations? The reason is not explained in the paper, this is very confusing and takes three times as much space. I think the point of DD is looking at a trained model, if the authors want to emphasize some idea for an untrained model (e.g. after T=1000 iterations) then they should explain it.
2) There are confusing notational issues throughout the paper: the parameters alpha and T are shown in the introduction, but are defined only in section 2. Some notations are used in the main part of the paper, but are only defined in the appendix (e.g. ||A||_M norm). It is OK to define some notations in the appendix, but if something appears in the main part of the paper, then I think it should be defined there.
3) Line 131, what does it mean that \Delta and W_0 are independent? Each coordinate of W_0 is i.i.d Gaussian, so how can it be dependent on \Delta. Is it meant to be that \Delta is independent of d?
4) In all the theorems, why is W_0 upper case? This is a vector, and the notations state that vectors are lower case, also w and w^* are lower case.
5) The figures are extremely small and hard to read. I suggest the authors to remove the different number of iterations experiments and just show how the trained model behaves, this will save much space and will allow to give much bigger and clearer figures.
6) In lines 699-700 it states that “the deeper the network the harder it is to train them”, while Figure 6 shows the opposite, that deeper networks achieve smaller test error.
7) The title is “On the Role of Optimization in Double Descent..”, while the paper studies how a certain eigenvalue of the data matrix affects DD.
8) The "related work" section is in the appendix.
Although this is allowed, in my opinion, this is bad practice. This section is a crucial part of the paper and helps the reader to evaluate its contribution and novelty, based on previous works that were done in the same area.
Some typos and other minor issues:
1) Sometimes “optimization” is used and other times “optimisation”. Both are OK, but the authors should be consistent and choose one form.
2) Title of Section 3: parammeterization → parameterization
3) In figure 1, “The first row demonstrates behavior of (1)...” what is (1)? The hyperlink points to the introduction section.
4) First sentence of appendix F, should it be the over-parameterized case?
To improve the paper I suggest making a more thorough experimental study. This includes more realistic datasets, different architectures, Verify whether the conclusions hold with different regularization schemes, and conduct more ablation studies.
Also, to connect the empirical results to the theoretical study I suggest looking at different architectures and network depths, and calculate the eigenvalue for more intermediary layers (including for the data itself). It will also help if the theoretical part will have a single corollary or theorem on how the excess risk behaves when d->\infty, without any unclear terms such as E[||w^*||^2]_{I-M}.
-------------Update after rebuttal---------------
After reading the author's response my concerns remain, and I retain my score.
No
* Many thanks for your detailed comments. We disagree however on deeming the empirical part of our work as the main weak point.
While we agree that more questions can be answered with further and more extensive experiments, we think our work presents an interesting perspective on DD phenomena that merits attention of the neurips community.
* We agree with the reviewer that the fact that the eigenvalue we are reporting are subject to change through optimization, which makes it different that the theoretical setting studied in the previous sections.
However, the point of view we adopted (feature extractor + linear predictor) is the closest to the theoretical analysis.
For this reason, we report only the eigenvalue of the penultimate layer (and not any intermediary layer), and we report the eigenvalue at multiple stages of the optimization. This shows that the curve of the behavior of the smallest eigenvalue doesn't change with the optimization. We will make this clearer in the paper.
* It is known that a suitably chosen strength of regularization can mitigate the "peak" in the DD curves.
However, our focus is not on how to mitigate DD when it happens, but rather on proposing an explanation that when DD happens, it is due to an effect of the optimization. Note also that all the networks are trained with plain gradient descent and not SGD to avoid the effect of implicit regularization, and keep the experiments as close as possible to the least-squares setting studied in our theory.
* We will use your feedback to clarify obscure points and we will correct the pointed out typos. We will make an effort to declare the used notation, and will insert more cross-references for improving the readability. We will also improve the figures as per your suggestion.
* Comment that the role of $\mathbb{E}[||\mathbf{w}^{\star}||^2]_{\mathbf{I}-\mathbf{M}}$ is unclear..
* The role of term $\mathbb{E}[||\mathbf{w}^{\star}||^2]_{\mathbf{I}-\mathbf{M}}$ is explained in the main text: It corresponds to the squared length of the projection of $\mathbf{w}^{\star}$ onto the subspace not spanned by the data. This term is non-zero only when $d > n$, and decays as $1/\sqrt{n}$ when the tail of eigenvalues has a polynomial decay (see appendix F).
* The meaning of independence of $\Delta$ and $\mathbf{W}_0$ is that their joint distribution equals the product of the marginal distributions (i.e. this refers to "stochastic independence" and not to independence of the algebraic expressions), this will be emphasized in the revised paper. We write $\mathbf{W}$ when this is a random vector (i.e. it has a probability distribution) and we write $\mathbf{w}$ when this is a non-random vector, both cases in boldface to indicate that they are vectors. We will clarify this convention in the notation section.
Also note that our labels for matrices ($\mathbf{A}$, $\mathbf{M}$) do not conflict with the labels we use for random vectors ($\mathbf{W}$).
In the end, making explicit the space of each kind of object ($\mathbb{R}^{p \times q}$ with suitable positive integers $p$ and $q$) should rule out any confusion, then vectors and matrices are not too different from each other (think of the case $q=1$).  We will do our best to clarify our notation conventions this in the notation section, and to insert reminders throughout the paper, in the revised version.
* Regarding the effect of depth, your comment (point 6 under some more issues) made us realize that there is a typo in Figure 6. In this figure we report test accuracy and not test error. This will be corrected. Thank you very much.
* We agree that more experiments are needed in order to gain better understanding, with more datasets and more models with different modern components.
However, in this paper we chose to explore whether the insights from the least squares learning problem are transferable to neural networks. We believe that the observations drawn from our experiments are interesting to the community. Understanding the impact of other factors that can change the network behavior (and the smallest eigenvalue of the penultimate layer), such as initialization, the optimizer choice and regularization, is important but left to a future work.
This paper tries to study and explain the double descent phenomenon from a new perspective.
However, it does not provide much new insight for the phenomenon. The reasons are as follows:
1. The object lying in the center of the analysis---the smallest nonzero eigenvalue of the Gram matrix---is the well known cause of the double descent phenomenon in linear least squares problems (including the random feature models), and is also extensively studied. In the works focusing on minimal norm solutions or ridge regression solutions, the same quantity is also studied and identified to be the factor that leads to the DD phenomenon.
2. The upper bound for the excess risk only considers the smallest nonzero eigenvalue of the Gram matrix, which means it only controls the slowest component in the dynamics.
Hence, the bound becomes tight only when the number of iterations, T, becomes large. Before this, the components corresponding to big eigenvalues dominate the dynamics and the DD phenomenon does not appear. This is showed by the empirical results in the paper.
However, once T is sufficiently big such that the component of the smallest eigenvalue dominates the dynamics, the GD solution is already close to the minimal norm solution or the l2 regularized solutions studied in previous works.
Therefore, the so-called "the roll of optimization" does not make much sense. Especially considering that the loss function is actually convex and any reasonable optimization algorithm will converge to a global minimum.
3. For a linear least squares problem, the GD solutions and the corresponding population risk can be written down in close form.
Hence, the reviewer does not see the necessity to derive the excess risk bounds via indirect approach such as (\Delta, M)-admissible.
Also, the (\Delta, M)-admissible condition does not help extend the analysis to non-linear cases.
Some other comments:
1. Shown by numerical experiments on neural networks, it seems the peak of the generalization error happens when the width approximately equals the number of data. This is different that shown in Belkin et.al, in which the the generalization error peaks when the number of parameters = number of data * number of classes.
Hence, this is like a different kind of "double descent". It is possible to provide more explanation on this phenomenon using the techniques employed in the paper?
2. Some notations are confusing.
For example, (a) the excess risk is denoted by E(w^*) while w^* is actually a fixed point. It is better to use E(A_S(W_0)) to highlight the dependence of the excess risk with the GD solution.  (b) the data us denoted by a capital X, which it is mentioned that vectors are denoted by boldface lower case letters. This causes confusion.
No
Thank you for your comments. We disagree however that our work does not provide new insights. The connection between DD and optimization, and the connection between the smallest positive eigenvalue and DD, have not been made explicit in any work we are aware of. Experimentally we see that the smallest positive eigenvalue correlates with performance for small $T$ as well (even if the bound might not be tight), and seems predictive of the phenomena.
The role of this work is not to show that DD happens, this result is known as we specify in the paper as well, but to provide an alternative tools for understanding it.
Concerning the experiments and the size of the networks, Belkin et al. report the number of all the parameters in the network. We report the number of units in the penultimate layer. We will make this clearer in the paper. As stated in the general comments, we consider that the network is composed of a feature extractor and a linear predictor. This setting is the closest to our theoretical analysis. The theory predicts that the peak happens when the size of the inputs of the linear predictor matches the number of input data which is confirmed in our experiments. This corresponds to a much larger networks than the ones reported in Belkin's paper: With 1000 input points, we observe a peak for a single hidden layer of 1000 hidden units, which corresponds to 795,010. In Belkin's work, the peak is reported at 10,000, corresponding to the number of inputs times the number of classes.
We will improve the notation based on your comments. Thank you!
I like the idea of relating DD to the smallest positive eigenvalue $\hat{\lambda}\_{\min}^+$ of the covariance matrix, which seems to give a fairly new perspective on the DD curves.
But I'm not an expert in DD as I only read a few related papers.
Thus I think a detailed comparison with previous papers is important for me to have a proper evaluation for this paper.
However, the related work section is in the appendix, not in the main body of the paper.
Also, comparisons with some closely related papers are not detailed enough. The authors mentioned that [Derezinski et al., 2020], [Rangamani et al., 2020], [Hastie et al., 2019] also noted that $\hat{\lambda}_{\min}^+$ is important in explaining double descent, but the key difference to the current paper is not very clear to me if I only read the related work section. I would suggest the authors add more details:
1. What are the assumptions on the data distribution in previous works? How do they compare with those in the current paper?
2. How do those papers find the connection between DD and $\hat{\lambda}_{\min}^+$ without studying GD optimization?
3. If they prove double descent without considering GD optimization, does it mean optimization is not a crucial factor to make double descent occur? The authors claim that the current paper is superior as it considers GD optimization, but it should be justified that GD optimization is crucial in the first place.
Unlike some previous papers, this paper does not assume an explicit formula for the data distribution. This is an advantage of this paper as it can hold for broader settings.
However, this also makes it impossible to derive an exact formula for excess risk.
Alternatively, the authors study the upper bounds, but no lower bound is given. Lower bounds are especially important for explaining the peak because having an upper bound $\approx 1$ can be either because there is a peak or the upper bound is not tight enough. I would like to see the authors' argument about the tightness of the upper bounds. Empirical evaluations for those upper bounds may also help.
No
Thank you for your suggestions, we will rearrange the content of our paper and we will present its positioning within the literature in the main text (currently in appendix). We will also discuss the mentioned papers: note that all of them focus on the Moore-Penrose solution, not GD as considered in our paper, therefore the comparison would not be on the equal ground.
* Optimization influences DD:
* In our current paper we uncover that DD happens partly because of optimization, that is for a finite T, and this phenomenon was overlooked in the previous literature due to the focus on analytic solutions.
When T is infinite (as is effectively the case in the Moore-Penrose pseudoinverse), optimization-induced DD vanishes.
* Comment _"...tightness of the upper bounds. Empirical evaluations for those upper bounds may also help."_
* Note that empirical evaluations of the upper bound are given: See our figure 1.
* "Limitation: theoretical analysis cannot be extended easily to deep neural nets"
* This is possible by following NTK-type analysis, however, behaviour of the smallest positive eigenvalue in that setting is less understood.
After reading the authors' response, some of my concerns remain.
In particular, the authors clarified that DD also occurs for Moore-Penrose solutions, and thus DD happens only partly because of optimization. Potential motivation for studying DD with GD optimization could be that DD can be explained even in the absence of label noise.
However, as pointed out by Reviewer XaVR, the bound can only be tight for large T, which suggests that the theory in this paper may only hold when the solution found by GD is close to the Moore-Penrose solution.
Therefore, this paper does not have a significant contribution to our knowledge of DD.
For this reason and concerns raised by the other reviewers, I decided to change my score to 5.
We thank the reviewers for the time spent in our work and for their feedback.
We first provide a general response addressing some common feedback, and then we address each reviewers' feedback separately.
It seems that a common concern of reviewers g8ya and XaVR is about the related literature and the placement of our work in this context.
Accordingly, we will make an effort to address this concern. A common concern of the reviewers XaVR and rFdx is about the notation. We will make an effort to declare explicitly the notation choices (as indicated in the individual responses) and will use your feedback to improve our work in this regard.
We think our work presents a new perspective on DD by reasoning about the role of optimization in the DD curves.
While we agree that many more questions about DD need to be answered, we think our work presents a novel theoretical result and carefully scoped experimental results on which further work can be built.
Therefore we believe our work can be meaningful for the community, and we hope the reviewers consider raising their scores.

Meta-review:
role: PC
data: comment: Reviewers reached a consensus by which the paper is not ready for publication.  Several concerns were raised, most notably:

* Only an upper bound on the excess risk is derived, which is insufficient for establishing a double descent phenomenon (for that, a matching lower bound is required).

* While experiments indicate that the upper bound may be tight after many iterations, it seems that in the early phases of a run it is not.  This is particularly problematic since in late stages of a run the model may in fact be close to convergence, and at convergence optimization reaches the Moore-Penrose solution, which was heavily analyzed in prior work (hinting that the regime in which the analysis is valid is one that is already well understood).

* The experiments on deep neural networks are somewhat disconnected from the theory, and this warrants a more thorough empirical investigation.

I encourage the authors to make use of the above feedback in revising their paper towards future submission.

Decision: Accept (Poster)