Interpolation can hurt robust generalization even when there is no noise

1 Introduction . Conventional statistical wisdom cautions the user who trains a model by minimizing a loss L ( θ ) : if a global minimizer achieves zero or near-zero training loss ( i.e. , it interpolates ) , we run the risk of overfitting ( i.e. , high variance ) and thus suboptimal prediction performance . Instead , regularization is commonly used to reduce the effect of noise and to obtain an estimator with better generalization . Specifically , regularization limits model complexity and induces worse data fit , for example via an explicit penalty term R ( θ ) . The resulting penalized loss L ( θ ) + λR ( θ ) explicitly imposes certain structural properties on the minimizer . This classical rationale , however , does seemingly not apply to overparameterized models : for example , large neural networks in practice exhibit good generalization performance on i.i.d . samples even if L ( θ ) vanishes and label noise is present [ 36 ] . Since interpolators are not unique in the overparameterized regime , it is crucial to study the specific implicit biases of interpolating estimators . In particular , for common losses , a large body of recent work analyzes the properties of the solutions found via gradient descent at convergence ( see e.g . [ 44 , 12 , 13 , 22 , 26 , 27 , 49 , 31 ] ) . For example , for linear and logistic regression , it is well-known that gradient descent converges to the min- ` 2-norm and max- ` 2-margin solutions , respectively [ 22 , 27 , 32 , 49 ] . These interpolating estimators also minimize the respective penalized loss L ( θ ) + λ‖θ‖22 in the limit of λ→ 0 [ 44 ] . A plethora of recent papers explicitly study generalization properties on min- ` 2-norm interpolators [ 15 , 18 , 23 , 6 , 33 , 34 , 35 ] and max- ` 2-margin solutions [ 14 , 34 , 46 ] , and show that the variance decreases as the overparameterization ratio increases beyond the interpolation threshold . While regularization with λ > 0 is commonly known to reduce the risk at the interpolation threshold [ 23 , 37 ] , many of these works are motivated by the second descent of the double descent phenomenon [ 7 ] which suggests that regularization becomes redundant with sufficient overparameterization . Hence , previous papers focus on highly overparameterized settings where the optimal regularization parameter satisfies λopt ≤ 0 [ 29 , 54 , 43 ] , implying that for large d/n , explicit regularization with λ > 0 is redundant or even detrimental for generalization . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Taking a step back , this narrative originated from theoretical and experimental findings that consider the standard test risk with identically distributed training and test data . However , this measure can not reflect the robust risk of models when the test data has a shifted distribution , is attacked by adversaries , or contains many samples from minority groups [ 19 , 21 , 40 , 55 ] . In fact , mounting empirical evidence suggests that regularization is indeed helpful for robust generalization , even in highly overparameterized regimes where the benefits for the standard risk are negligible [ 42 ] . This phenomenon is sometimes referred to as robust overfitting . In the presence of noise , the following intuition holds true : since the robust risk amplifies estimation errors , its variance is larger and hence regularization – such as early stopping – can be beneficial for generalization [ 47 ] . However , we observe that even when estimating entirely noiseless signals , robust overfitting persists ! We observe this phenomenon in experiments with shallow neural networks on sanitized image data depicted in Figure 1a and , in fact , even for linear models trained on highdimensional synthetic noiseless data . In particular , Figures 1b,1c show that min- ` 2-norm and robust max- ` 2-margin interpolators ( minimizers of the training loss for λ → 0 ) , achieve a higher robust risk than the corresponding regularized estimators that do not interpolate noiseless observations ( minimizers for λ > 0 ) . To date , our observations in the noiseless case can not be explained by prior work . On the contrary , they seem to contradict a simple intuition : if the min- ` 2-norm and robust max- ` 2-margin interpolators exhibit large risks as λ → 0 , the induced bias for a small ` 2-norm is potentially suboptimal and a larger penalty weight λ > 0 should only degrade performance . In this paper , we provide possible explanations that debunk this intuition in the high-dimensional overparameterized regime . We prove for isotropic Gaussian covariates that a strictly positive regularization parameter λ systematically improves robust generalization for linear and robust logistic regression . Empirically , we show that early stopping and other factors that lead to a non-interpolating estimator achieve a similar effect . Our results provide the first rigorous explanation of robust overfitting even in the absence of noise . In Section 2 , we formally define the setting that we use throughout our analysis . We then first present precise asymptotic expressions for the robust risk for linear regression in Section 3 that explicitly explain robust overfitting . Furthermore , in Section 4 , we consider classification with logistic regression and derive asymptotic results . 2 Risk minimization framework . In this section , we introduce the data generating process that we assume throughout our analysis , and define the standard and robust risks that we use as evaluation metrics . 2.1 Problem setting . This paper considers the supervised learning problem of estimating a mapping from d-dimensional real-valued features x ∈ Rd to a target y ∈ Y ⊆ R given a training set of labeled samples D = { ( xi , yi ) } ni=1 . We assume that the feature vectors xi are drawn i.i.d . from the marginal distribution P that we assume to be an isotropic Gaussian . We further focus on noiseless observations yi = 〈θ ? , xi〉 for regression tasks and yi = sgn ( 〈θ ? , xi〉 ) for classification tasks , respectively . However , the main results are more general and apply to noisy observations as well . For regression , we assume additive Gaussian noise with zero mean and σ2 variance , while for classification we flip a certain percentage of the training labels . This paper studies the high-dimensional asymptotic regime where d/n→ γ as both the dimensionality d and the number of samples n tend to infinity . This high-dimensional setting is widely studied as it can often – as in our experiments – yield precise predictions for the risk of the estimator when both the input dimension and the data set size are large [ 9 , 52 ] . It is also the predominant setting considered in previous theoretical works that discuss overparameterized linear models [ 2 , 14 , 15 , 23 , 25 , 24 , 50 ] . 2.2 Standard and robust risk . We now introduce the standard and robust evaluation metrics for regression and classification . Given a pointwise test loss ` test : R× R→ R , we define the standard ( population ) risk of an estimator θ̂ as R ( θ̂ ) : = EX∼P ` test ( 〈θ̂ , X〉 , 〈θ ? , X〉 ) , ( 1 ) where the expectation is taken over the marginal feature distribution P. Note that for any datadependent estimator θ̂ , this risk is fixed if conditioned on the training data . Our asymptotic bounds hold almost surely over draws of the training set . As standard in the literature , we choose the square loss ` test ( u , v ) = ( u − v ) 2 for regression and the 0-1 loss ` test ( u , v ) = 1sgn ( u ) 6=sgn ( v ) for classification . The broad application of ML models in real-world decision-making processes increases requirements on their generalization abilities beyond i.i.d . test sets . For example , in the image domain , classifiers should be robust and output the same prediction for perturbations of an image that do not change the ground truth label ( e.g. , imperceptible ` p-perturbations [ 19 ] ) . In this case , we say the perturbations are consistent and the estimator that achieves zero robust population risk also has zero standard population risk . For linear models in particular , one way to enforce consistency is to restrict perturbations to the space orthogonal to the ground truth , as proposed in [ 41 ] . Motivated by the imperceptibility assumption and ` p-adversarial attacks widely studied in the image domain , we consider the adversarially robust risk of a parameter θ with respect to consistent ` pperturbations R ( θ̂ ) : = EX∼P max δ∈Up ( ) ` test ( 〈θ̂ , X + δ〉 , 〈θ ? , X〉 ) , ( 2 ) with the perturbation set Up ( ) : = { δ ∈ Rd : ‖δ‖p ≤ and 〈θ ? , δ〉 = 0 } . In many scientific applications , security against adversarial attacks may not be the dominating concern ; one may instead require estimators to be robust against small distribution shifts . Earlier work [ 48 ] has pointed out that distribution shift robustness and adversarial robustness are equivalent for losses that are convex in the parameter θ . Similarly , in our setting , adversarial robustness against consistent ` p-perturbations implies distributional robustness against ` p-bounded mean shifts in the covariate distribution P ( see Appendix A.3 ) . 3 Min- ` 2-norm interpolation in robust linear regression In the context of regression , we illustrate overfitting of the robust risk in Equation ( 2 ) with the set of consistent ` 2-perturbations U2 ( ) . More precisely , we show that preventing min- ` 2-norm interpolation on noiseless samples via ridge regularization improves the robust risk . We refer the reader to Appendix C.1 for an intuitive explanation . Lastly , we note that due to the rotational invariance of the problem , our results hold for sparse and dense ground truths θ ? alike . 3.1 Interpolating and regularized estimator . We study linear ridge regression estimates defined as θ̂λ = arg min θ 1 n n∑ i=0 ( yi − 〈θ , xi〉 ) 2 + λ‖θ‖22 . ( 3 ) The min- ` 2-norm interpolator is the limit of the linear ridge regression estimate with λ→ 0 and is given by θ̂0 = arg min θ ‖θ‖2 such that 〈θ , xi〉 = yi for all i . ( 4 ) Note that the min- ` 2-norm interpolator is also the estimator that gradient descent on the unregularized loss converges to , while ridge regression with λ > 0 corresponds to early-stopped estimators [ 1 , 2 ] . Therefore , by proving that a ridge regularized estimator with λ > 0 significantly outperforms the min- ` 2-norm interpolator with λ→ 0 , we also show that early stopping benefits robust generalization . Whenever the goal is to achieve a low robust risk , a popular alternative to using the standard linear regression estimate in Equations ( 3 ) , ( 4 ) is to consider adversarially trained estimators [ 19 , 25 ] . However , ` 2-adversarial training in its usual form ( i.e. , with inconsistent perturbations ) prevents regression estimators from interpolating , and hence , has a similar effect to ` 2-regularization as we discuss in more detail in Appendix C.2 . On the other hand , training with consistent perturbations as defined in the robust risk is equivalent to full knowledge of the direction of θ ? and hence simply recovers the ground truth in the noiseless case . Since our goal is to reveal the shortcomings of interpolators compared to regularized estimators , we only analyze ridge estimators trained without perturbations .

Summary: This paper studies how regularization affects the robust population risk in the overparameterization regime. The authors study the behavior of linear regression and logistic regression under sufficient statistical settings. In particular, even in the noiseless regime, the population robust risk benefits from the regularization, but not the population test risk. The argument is empirically verified.