Multiple Descent: Design Your Own Generalization Curve

1 Introduction . The main goal of machine learning methods is to provide an accurate out-of-sample prediction , known as generalization . For a fixed family of models , a common way to select a model from this family is through empirical risk minimization , i.e. , algorithmically selecting models that minimize the risk on the training dataset . Given a variably parameterized family of models , the statistical learning theory aims to identify the dependence between model complexity and model performance . The empirical risk usually decreases monotonically as the model complexity increases , and achieves its minimum when the model is rich enough to interpolate the training data , resulting in zero ( or near-zero ) training error . In contrast , the behaviour of the test error as a function of model complexity is far more complicated . Indeed , in this paper we show how to construct a model family for which the generalization curve can be fully controlled ( away from the interpolation threshold ) in both under-parameterized and over-parameterized regimes . Classical statistical learning theory supports a U-shaped curve of generalization versus model complexity [ 31 , 33 ] . Under such a framework , the best model is found at the bottom of the U-shaped curve , which corresponds to appropriately balancing under-fitting and over-fitting the training data . From the view of the bias-variance trade-off , a higher model complexity increases the variance while decreasing the bias . A model with an appropriate level of complexity achieves a relatively low bias while still keeping the variance under control . On the other hand , a model that interpolates the training data is deemed to over-fit and tends to worsen the generalization performance due to the soaring variance . Although classical statistical theory suggests a pattern of behavior for the generalization curve up to the interpolation threshold , it does not describe what happens beyond the interpolation threshold , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . commonly referred to as the over-parameterized regime . This is the exact regime where many modern machine learning models , especially deep neural networks , achieved remarkable success . Indeed , neural networks generalize well even when the models are so complex that they have the potential to interpolate all the training data points [ 61 , 10 , 32 , 34 ] . Modern practitioners commonly deploy deep neural networks with hundreds of millions or even billions of parameters . It has become widely accepted that large models achieve performance superior to small models that may be suggested by the classical U-shaped generalization curve [ 13 , 38 , 55 , 35 , 36 ] . This indicates that the test error decreases again once model complexity grows beyond the interpolation threshold , resulting in the so called double-descent phenomenon described in [ 9 ] , which has been broadly supported by empirical evidence [ 49 , 48 , 29 , 30 ] and confirmed empirically on modern neural architectures by Nakkiran et al . [ 46 ] . On the theoretical side , this phenomenon has been recently addressed by several works on various model settings . In particular , Belkin et al . [ 11 ] proved the existence of double-descent phenomenon for linear regression with random feature selection and analyzed the random Fourier feature model [ 50 ] . Mei and Montanari [ 44 ] also studied the Fourier model and computed the asymptotic test error which captures the double-descent phenomenon . Bartlett et al . [ 8 ] , Tsigler and Bartlett [ 56 ] analyzed and gave explicit conditions for “ benign overfitting ” in linear and ridge regression , respectively . Caron and Chretien [ 16 ] provided a finite sample analysis of the nonlinear function estimation and showed that the parameter learned through empirical risk minimization converges to the true parameter with high probability as the model complexity tends to infinity , implying the existence of double descent . Liu et al . [ 42 ] studied the high dimensional kernel ridge regression in the under- and over-parameterized regimes and showed that the risk curve can be double descent , bell-shaped , and monotonically decreasing . Among all the aforementioned efforts , one particularly interesting question is whether one can observe more than two descents in the generalization curve . d ’ Ascoli et al . [ 21 ] empirically showed a samplewise triple-descent phenomenon under the random Fourier feature model . Similar triple-descent was also observed for linear regression [ 47 ] . More rigorously , Liang et al . [ 41 ] presented an upper bound on the risk of the minimum-norm interpolation versus the data dimension in Reproducing Kernel Hilbert Spaces ( RKHS ) , which exhibits multiple descent . However , a multiple-descent upper bound without a properly matching lower bound does not imply the existence of a multiple-descent generalization curve . In this work , we study the multiple descent phenomenon by addressing the following questions : • Can the existence of a multiple descent generalization curve be rigorously proven ? • Can an arbitrary number of descents occur ? • Can the generalization curve and the locations of descents be designed ? In this paper , we show that the answer to all three of these questions is yes . Further related work is presented in Section 2 . Our Contribution . We consider the linear regression model and analyze how the risk changes as the dimension of the data grows . In the linear regression setting , the data dimension is equal to the dimension of the parameter space , which reflects the model complexity . We rigorously show that the multiple descent generalization curve exists under this setting . To our best knowledge , this is the first work proving a multiple descent phenomenon . Our analysis considers both the underparametrized and overparametrized regimes . In the overparametrized regime , we show that one can control where a descent or an ascent occurs in the generalization curve . This is realized through our algorithmic construction of a feature-revealing process . To be more specific , we assume that the data is in RD , where D can be arbitrarily large or even essentially infinite . We view each dimension of the data as a feature . We consider a linear regression problem restricted on the first d features , where d < D. New features are revealed by increasing the dimension of the data . We then show that by specifying the distribution of the newly revealed feature to be either a standard Gaussian or a Gaussian mixture , one can determine where an ascent or a descent occurs . In order to create an ascent when a new feature is revealed , it is sufficient that the feature follows a Gaussian mixture distribution . In order to have a descent , it is sufficient that the new feature follows a standard Gaussian distribution . Therefore , in the overparametrized regime , we can fully control the occurrence of a descent and an ascent . As a comparison , in the underparametrized regime , the generalization loss always increases regardless of the feature distribution . Generally speaking , we show that we are able to design the generalization curve . On the one hand , we show theoretically that the generalization curve is malleable and can be constructed in an arbitrary fashion . On the other hand , we rarely observe complex generalization curves in practice , besides carefully curated constructions . Putting these facts together , we arrive at the conclusion that realistic generalization curves arise from specific interactions between properties of typical data and the inductive biases of algorithms . We should highlight that the nature of these interactions is far from being understood and should be an area of further investigations . 2 Related Work . Our work is directly related to the recent line of research in the theoretical understanding of the double descent [ 11 , 34 , 60 , 44 ] and the multiple descent phenomenon [ 41 , 39 ] . Here we briefly discuss some other work that is closely related to this paper . Least Square Regression . In this paper we focus on the least square linear regression with no regularization . For the regularized least square regression , De Vito et al . [ 22 ] proposed a selection procedure for the regularization parameter . Advani and Saxe [ 1 ] analyzed the generalization of neural networks with mean squared error under the asymptotic regime where both the sample size and model complexity tend to infinity . Richards et al . [ 52 ] proved for least square regression in the asymptotic regime that as the dimension-to-sample-size ratio d/n grows , an additional peak can occur in both the variance and bias due to the covariance structure of the features . As a comparison , in this paper the sample size is fixed and the model complexity increases . Rudi and Rosasco [ 53 ] studied kernel ridge regression and gave an upper bound on the number of the random features to reach certain risk level . Our result shows that there exists a natural setting where by manipulating the random features one can control the risk curve . Over-Parameterization and Interpolation . The double descent occurs when the model complexity reaches and increases beyond the interpolation threshold . Most previous works focused on proving an upper bound or optimal rate for the risk . Caponnetto and De Vito [ 15 ] gave the optimal rate for least square ridge regression via careful selection of the regularization parameter . Belkin et al . [ 12 ] showed that the optimal rate for risk can be achieved by a model that interpolates the training data . In a series of work on kernel regression with regularization parameter tending to zero ( a.k.a . kernel ridgeless regression ) , Rakhlin and Zhai [ 51 ] showed that the risk is bounded away from zero when the data dimension is fixed with respect to the sample size . Liang and Rakhlin [ 40 ] then considered the case when d n , showed empirically the multiple descent phenomenon and proved a risk upper bound that can be small given favorable data and kernel assumptions . Instead of giving a bound , our paper presents an exact computation of risk in the cases of underparametrized and overparametrized linear regression , and proves the existence of the multiple descent phenomenon . Wyner et al . [ 59 ] analyzed AdaBoost and Random Forest from the perspective of interpolation . There has also been a line of work on wide neural networks [ 4–6 , 23 , 3 , 58 , 14 , 2 , 18 , 62 , 54 ] . Sample-wise Double Descent and Non-monotonicity . There has also been recent development beyond the model-complexity double-descent phenomenon . For example , regarding sample-wise non-monotonicity , Nakkiran et al . [ 46 ] empirically observed the epoch-wise double-descent and sample-wise non-monotonicity for neural networks . Chen et al . [ 19 ] and Min et al . [ 45 ] identified and proved the sample-wise double descent under the adversarial training setting , and Javanmard et al . [ 37 ] discovered double-descent under adversarially robust linear regression . Loog et al . [ 43 ] showed that empirical risk minimization can lead to sample-wise non-monotonicity in the standard linear model setting under various loss functions including the absolute loss and the squared loss , which covers the range from classification to regression . We also refer the reader to their discussion of the earlier work on non-monotonicity of generalization curves . Dar et al . [ 20 ] demonstrated the double descent curve of the generalization errors of subspace fitting problems . Fei et al . [ 28 ] studied the risk-sample tradeoff in reinforcement learning .

Summary: ================================= FINAL UPDATE AFTER DISCUSSION PHASE  As the authors addressed the points that came up during the discussion phase (which made me initially lower the score), I revert back to my initial scoring of accept.   =================================== UPDATE AFTER INITIAL DISCUSSION PHASE  The authors addressed my own points, but following the discussion with the other reviewers I decided to lower my score. It was pointed out that the data generation process depends on the model-dimension, and thus the resulting investigation is not about 'generalization curves' as claimed by the authors, but rather about the generalization behavior of different interpolation regimes. While I believe the contribution is still valuable, some explanations and need be changed (which may be possible in a camera ready version).  ( For example I don't believe anymore that the paper indeed answers the question: 'Can the existence of a multiple descent generalization curve be rigorously proven?'.)  ==================================  The paper analyses the behavior of generalization curves of linear regression wrt an increasing feature dimensionality. The paper constructs specific examples that can generate any ascending/ descending behavior of the curve, for both the under and overparametrized case. Their construction highlights that ascending/ descending behavior of the generalization curve is not inherent to the model family, but rather a result of the data distribution in combination with the model.