Interpolation can hurt robust generalization even when there is no noise

1 Introduction . Conventional statistical wisdom cautions the user who trains a model by minimizing a loss L ( θ ) : if a global minimizer achieves zero or near-zero training loss ( i.e. , it interpolates ) , we run the risk of overfitting ( i.e. , high variance ) and thus suboptimal prediction performance . Instead , regularization is commonly used to reduce the effect of noise and to obtain an estimator with better generalization . Specifically , regularization limits model complexity and induces worse data fit , for example via an explicit penalty term R ( θ ) . The resulting penalized loss L ( θ ) + λR ( θ ) explicitly imposes certain structural properties on the minimizer . This classical rationale , however , does seemingly not apply to overparameterized models : for example , large neural networks in practice exhibit good generalization performance on i.i.d . samples even if L ( θ ) vanishes and label noise is present [ 36 ] . Since interpolators are not unique in the overparameterized regime , it is crucial to study the specific implicit biases of interpolating estimators . In particular , for common losses , a large body of recent work analyzes the properties of the solutions found via gradient descent at convergence ( see e.g . [ 44 , 12 , 13 , 22 , 26 , 27 , 49 , 31 ] ) . For example , for linear and logistic regression , it is well-known that gradient descent converges to the min- ` 2-norm and max- ` 2-margin solutions , respectively [ 22 , 27 , 32 , 49 ] . These interpolating estimators also minimize the respective penalized loss L ( θ ) + λ‖θ‖22 in the limit of λ→ 0 [ 44 ] . A plethora of recent papers explicitly study generalization properties on min- ` 2-norm interpolators [ 15 , 18 , 23 , 6 , 33 , 34 , 35 ] and max- ` 2-margin solutions [ 14 , 34 , 46 ] , and show that the variance decreases as the overparameterization ratio increases beyond the interpolation threshold . While regularization with λ > 0 is commonly known to reduce the risk at the interpolation threshold [ 23 , 37 ] , many of these works are motivated by the second descent of the double descent phenomenon [ 7 ] which suggests that regularization becomes redundant with sufficient overparameterization . Hence , previous papers focus on highly overparameterized settings where the optimal regularization parameter satisfies λopt ≤ 0 [ 29 , 54 , 43 ] , implying that for large d/n , explicit regularization with λ > 0 is redundant or even detrimental for generalization . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Taking a step back , this narrative originated from theoretical and experimental findings that consider the standard test risk with identically distributed training and test data . However , this measure can not reflect the robust risk of models when the test data has a shifted distribution , is attacked by adversaries , or contains many samples from minority groups [ 19 , 21 , 40 , 55 ] . In fact , mounting empirical evidence suggests that regularization is indeed helpful for robust generalization , even in highly overparameterized regimes where the benefits for the standard risk are negligible [ 42 ] . This phenomenon is sometimes referred to as robust overfitting . In the presence of noise , the following intuition holds true : since the robust risk amplifies estimation errors , its variance is larger and hence regularization – such as early stopping – can be beneficial for generalization [ 47 ] . However , we observe that even when estimating entirely noiseless signals , robust overfitting persists ! We observe this phenomenon in experiments with shallow neural networks on sanitized image data depicted in Figure 1a and , in fact , even for linear models trained on highdimensional synthetic noiseless data . In particular , Figures 1b,1c show that min- ` 2-norm and robust max- ` 2-margin interpolators ( minimizers of the training loss for λ → 0 ) , achieve a higher robust risk than the corresponding regularized estimators that do not interpolate noiseless observations ( minimizers for λ > 0 ) . To date , our observations in the noiseless case can not be explained by prior work . On the contrary , they seem to contradict a simple intuition : if the min- ` 2-norm and robust max- ` 2-margin interpolators exhibit large risks as λ → 0 , the induced bias for a small ` 2-norm is potentially suboptimal and a larger penalty weight λ > 0 should only degrade performance . In this paper , we provide possible explanations that debunk this intuition in the high-dimensional overparameterized regime . We prove for isotropic Gaussian covariates that a strictly positive regularization parameter λ systematically improves robust generalization for linear and robust logistic regression . Empirically , we show that early stopping and other factors that lead to a non-interpolating estimator achieve a similar effect . Our results provide the first rigorous explanation of robust overfitting even in the absence of noise . In Section 2 , we formally define the setting that we use throughout our analysis . We then first present precise asymptotic expressions for the robust risk for linear regression in Section 3 that explicitly explain robust overfitting . Furthermore , in Section 4 , we consider classification with logistic regression and derive asymptotic results . 2 Risk minimization framework . In this section , we introduce the data generating process that we assume throughout our analysis , and define the standard and robust risks that we use as evaluation metrics . 2.1 Problem setting . This paper considers the supervised learning problem of estimating a mapping from d-dimensional real-valued features x ∈ Rd to a target y ∈ Y ⊆ R given a training set of labeled samples D = { ( xi , yi ) } ni=1 . We assume that the feature vectors xi are drawn i.i.d . from the marginal distribution P that we assume to be an isotropic Gaussian . We further focus on noiseless observations yi = 〈θ ? , xi〉 for regression tasks and yi = sgn ( 〈θ ? , xi〉 ) for classification tasks , respectively . However , the main results are more general and apply to noisy observations as well . For regression , we assume additive Gaussian noise with zero mean and σ2 variance , while for classification we flip a certain percentage of the training labels . This paper studies the high-dimensional asymptotic regime where d/n→ γ as both the dimensionality d and the number of samples n tend to infinity . This high-dimensional setting is widely studied as it can often – as in our experiments – yield precise predictions for the risk of the estimator when both the input dimension and the data set size are large [ 9 , 52 ] . It is also the predominant setting considered in previous theoretical works that discuss overparameterized linear models [ 2 , 14 , 15 , 23 , 25 , 24 , 50 ] . 2.2 Standard and robust risk . We now introduce the standard and robust evaluation metrics for regression and classification . Given a pointwise test loss ` test : R× R→ R , we define the standard ( population ) risk of an estimator θ̂ as R ( θ̂ ) : = EX∼P ` test ( 〈θ̂ , X〉 , 〈θ ? , X〉 ) , ( 1 ) where the expectation is taken over the marginal feature distribution P. Note that for any datadependent estimator θ̂ , this risk is fixed if conditioned on the training data . Our asymptotic bounds hold almost surely over draws of the training set . As standard in the literature , we choose the square loss ` test ( u , v ) = ( u − v ) 2 for regression and the 0-1 loss ` test ( u , v ) = 1sgn ( u ) 6=sgn ( v ) for classification . The broad application of ML models in real-world decision-making processes increases requirements on their generalization abilities beyond i.i.d . test sets . For example , in the image domain , classifiers should be robust and output the same prediction for perturbations of an image that do not change the ground truth label ( e.g. , imperceptible ` p-perturbations [ 19 ] ) . In this case , we say the perturbations are consistent and the estimator that achieves zero robust population risk also has zero standard population risk . For linear models in particular , one way to enforce consistency is to restrict perturbations to the space orthogonal to the ground truth , as proposed in [ 41 ] . Motivated by the imperceptibility assumption and ` p-adversarial attacks widely studied in the image domain , we consider the adversarially robust risk of a parameter θ with respect to consistent ` pperturbations R ( θ̂ ) : = EX∼P max δ∈Up ( ) ` test ( 〈θ̂ , X + δ〉 , 〈θ ? , X〉 ) , ( 2 ) with the perturbation set Up ( ) : = { δ ∈ Rd : ‖δ‖p ≤ and 〈θ ? , δ〉 = 0 } . In many scientific applications , security against adversarial attacks may not be the dominating concern ; one may instead require estimators to be robust against small distribution shifts . Earlier work [ 48 ] has pointed out that distribution shift robustness and adversarial robustness are equivalent for losses that are convex in the parameter θ . Similarly , in our setting , adversarial robustness against consistent ` p-perturbations implies distributional robustness against ` p-bounded mean shifts in the covariate distribution P ( see Appendix A.3 ) . 3 Min- ` 2-norm interpolation in robust linear regression In the context of regression , we illustrate overfitting of the robust risk in Equation ( 2 ) with the set of consistent ` 2-perturbations U2 ( ) . More precisely , we show that preventing min- ` 2-norm interpolation on noiseless samples via ridge regularization improves the robust risk . We refer the reader to Appendix C.1 for an intuitive explanation . Lastly , we note that due to the rotational invariance of the problem , our results hold for sparse and dense ground truths θ ? alike . 3.1 Interpolating and regularized estimator . We study linear ridge regression estimates defined as θ̂λ = arg min θ 1 n n∑ i=0 ( yi − 〈θ , xi〉 ) 2 + λ‖θ‖22 . ( 3 ) The min- ` 2-norm interpolator is the limit of the linear ridge regression estimate with λ→ 0 and is given by θ̂0 = arg min θ ‖θ‖2 such that 〈θ , xi〉 = yi for all i . ( 4 ) Note that the min- ` 2-norm interpolator is also the estimator that gradient descent on the unregularized loss converges to , while ridge regression with λ > 0 corresponds to early-stopped estimators [ 1 , 2 ] . Therefore , by proving that a ridge regularized estimator with λ > 0 significantly outperforms the min- ` 2-norm interpolator with λ→ 0 , we also show that early stopping benefits robust generalization . Whenever the goal is to achieve a low robust risk , a popular alternative to using the standard linear regression estimate in Equations ( 3 ) , ( 4 ) is to consider adversarially trained estimators [ 19 , 25 ] . However , ` 2-adversarial training in its usual form ( i.e. , with inconsistent perturbations ) prevents regression estimators from interpolating , and hence , has a similar effect to ` 2-regularization as we discuss in more detail in Appendix C.2 . On the other hand , training with consistent perturbations as defined in the robust risk is equivalent to full knowledge of the direction of θ ? and hence simply recovers the ground truth in the noiseless case . Since our goal is to reveal the shortcomings of interpolators compared to regularized estimators , we only analyze ridge estimators trained without perturbations .

Summary: The authors analyze the generalization performance of regularized adversarial linear regression and classification models for standard Gaussian data in the overparameterized setting.  The analysis allows for l^2 weight decay as well as l^\infty perturbation of the inputs.  Using similar methods to Hastie, Montanari, Rosset, and Tibshirani [18] and Javanmard and Soltanolkotabi [19], they are able to develop precise formula for the generalization error in the limit as n/d -> constant as n,d -> infinity.  They demonstrate that non-zero regularization has strictly better generalization in the robust setting.