Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds and Benign Overfitting

1 Introduction . The traditional understanding of machine learning suggests that models with zero training error tend to overfit , and explicit regularization is often necessary to achieve good generalization . Given the empirical success of deep learning models with zero training error ( Neyshabur et al . 2015 ; Zhang et al . 2017 ) and the ( re- ) discovery of the “ double descent ” phenomenon ( Belkin et al . 2019 ) , however , it has become clear that the textbook U-shaped learning curve is only part of a larger picture : it is possible for an overparameterized model with zero training loss to achieve low population error in a noisy setting . In an effort to understand how interpolation learning occurs , there has been much recent study of the testbed problem of linear regression with Gaussian features ( e.g . Hastie et al . 2019 ; Bartlett et al . 2020 ; Belkin et al . 2020 ; Ju et al . 2020 ; Muthukumar et al . 2020 ; Negrea et al . 2020 ; Tsigler and Bartlett 2020 ; Zhou et al . 2020 ) . Significant progress has been made in this setting , including nearly-matching necessary and sufficient conditions for consistency of the minimal ` 2 norm interpolator ( Bartlett et al . 2020 ) . Despite the fundamental role of uniform convergence in statistical learning theory , most of this line of work has used other techniques to analyze the particular minimal-norm interpolator.1 Instead of directly analyzing the population error of a learning algorithm , a uniform convergence-type argument would control the worst-case generalization gap over a class of predictors containing the typical outputs of a learning rule . Typically , this is done because for many algorithms – unlike the minimal Euclidean norm interpolator – it is difficult to exactly characterize the learned predictor , but we may be able to say e.g . that its norm is not too large . Since uniform convergence does not tightly depend on a specific algorithm , the resulting analysis can highlight the key properties that lead to good generalization : it can give bounds not only for , say , the minimal-norm interpolator , but also ⇤These authors contributed equally . 1Negrea et al . ( 2020 ) argue that Bartlett et al . ( 2020 ) ’ s proof technique is fundamentally based on uniform convergence of a surrogate predictor ; Yang et al . ( 2021 ) study a closely related setting with a uniform convergence-type argument , but do not establish consistency . We discuss both papers in more detail in Section 4 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for other interpolators with low norm ( e.g . Zhou et al . 2020 ) , increasing our confidence that low norm – and not some other property the particular minimal-norm interpolator happens to have – is key to generalization . In linear regression , practical training algorithms may not always find the exact minimal Euclidean norm solution , so it is also reassuring that all interpolators with sufficiently low Euclidean norm generalize . Nagarajan and Kolter ( 2019 ) , however , raised significant questions about the applicability of typical uniform convergence arguments to certain high-dimensional regimes , similar to those seen in interpolation learning . Following their work , Bartlett and Long ( 2020 ) , Negrea et al . ( 2020 ) , Zhou et al . ( 2020 ) , and Yang et al . ( 2021 ) all demonstrated the failure of forms of uniform convergence in various interpolation learning setups . To sidestep these negative results , Zhou et al . ( 2020 ) suggested considering bounds which are uniform only over predictors with zero training error . This weaker notion of uniform convergence has been standard in analyses of “ realizable ” ( noiseless ) learning at least since the work of Vapnik ( 1982 , Chapter 6.4 ) and Valiant ( 1984 ) . Zhou et al . demonstrated that at least in one particular noisy setting , such uniform convergence is sufficient for showing consistency of the minimal ` 2 norm interpolator , even though “ non-realizable ” uniform convergence arguments ( those over predictors regardless of their training error ) can not succeed . It remains unknown , however , whether these types of arguments can apply to more general linear regression problems and more typical asymptotic regimes , particularly showing rates of convergence rather than just consistency . In this work , we show for the first time that uniform convergence is indeed able to explain benign overfitting in general high-dimensional Gaussian linear regression problems . Similarly to how the standard analysis for learning with Lipschitz losses bounds generalization gaps through Rademacher complexity ( e.g . Shalev-Shwartz and Ben-David 2014 ) , our Theorem 1 ( Section 3 ) establishes a finitesample high probability bound on the uniform convergence of the error of interpolating predictors in a hypothesis class , in terms of its Gaussian width . This is done through an application of the Gaussian Minimax Theorem ; see the proof sketch in Section 7 . Combined with an analysis of the norm of the minimal ` 2 norm interpolator ( Theorem 2 in Section 4 ) , our bound recovers known consistency results ( Bartlett et al . 2020 ) , as well as proving a conjectured upper bound for larger-norm interpolators ( Zhou et al . 2020 ) . In addition , since we do not restrict ourselves to Euclidean norm balls but instead consider interpolators in an arbitrary compact set , our results allows for a wide range of other applications . Our analysis leads to a natural extension of the consistency result and notions of effective rank of Bartlett et al . ( 2020 ) for arbitrary norms ( Theorem 5 in Section 5 ) . As a demonstration of our general theory , in Section 6 we show novel consistency results for the minimal ` 1 norm interpolator ( basis pursuit ) in particular settings , which we believe are the first results of their kind . 2 Problem Formulation . Notation . We use k·kp for the ` p norm , kxkp = ( P i |xi|p ) 1/p . We always use maxx2S f ( x ) to be 1 when S is empty , and similarly minx2S f ( x ) to be 1 . We use standard O ( · ) notation , and a . b for inequality up to an absolute constant . For a positive semidefinite matrix A , the Mahalanobis ( semi- ) norm is kxk2 A : = hx , Axi . For a matrix A and set S , AS denotes the set { Ax : x 2 S } . Data model . We assume that data ( X , Y ) is generated as Y = Xw⇤ + ⇠ , Xi iid⇠ N ( 0 , ⌃ ) , ⇠ ⇠ N ( 0 , 2In ) , ( 1 ) where X 2 Rn⇥d has i.i.d . Gaussian rows X1 , . . . , Xn , d n , w⇤ is arbitrary , and ⇠ is Gaussian and independent of X . Though our proof techniques crucially depend on Xi being Gaussian , we can easily relax the assumption on the noise ⇠ to only being sub-Gaussian ; we assume Gaussian noise here for simplicity . The empirical and population loss are defined as , respectively , L̂ ( w ) = 1 n kY Xwk22 , L ( w ) = E ( x , y ) ( y hw , xi ) 2 = 2 + kw w⇤k2⌃ , where in the expectation y = hx , w⇤i+ ⇠0 with x ⇠ N ( 0 , ⌃ ) independent of ⇠0 ⇠ N ( 0 , 2 ) . For an arbitrary norm k·k , the minimal norm interpolator is ŵ = argmin L̂ ( w ) =0 kwk . For Euclidean norm specifically , the minimal norm interpolator can be written explicitly as ŵ = X ( XXT ) 1Y . If there is more than one minimal norm interpolator , all of our guarantees will hold for any minimizer ŵ . Speculative bound . Zhou et al . ( 2020 ) studied uniform convergence of low norm interpolators , sup kwkB , L̂ ( w ) =0 L ( w ) L̂ ( w ) . ( 2 ) Clearly , when B kŵk , this quantity upper-bounds the population risk of ŵ . Zhou et al . evaluated the asymptotic limit of ( 2 ) in one particular setting . But they further speculated that a bound of the following form may hold more generally : sup kwk2B , L̂ ( w ) =0 L ( w ) L̂ ( w ) B 2 n n + o ( 1 ) , ( ? ) where2 n ⇡ kxk2 . As discussed by Zhou et al. , a bound almost of this form is implied by results of Srebro et al . ( 2010 ) for general data distributions , except that approach gives a large leading constant and logarithmic factors . To show consistency of benign overfitting , though , we need ( ? ) to hold without even a constant multiplicative factor . Zhou et al . ask whether and when this holds , speculating that it might in broad generality . The goal of this paper is essentially to prove ( ? ) , at least for Gaussian data , and to use it to show consistency of the minimal norm interpolator . Our main result ( Theorem 1 ) can be thought of showing ( ? ) for Gaussian data with n = E kxk2 , as well as strengthening and significantly generalizing it . Our result is more general , as it applies to general compact hypothesis sets beyond just the Euclidean ball as in ( ? ) . But it also falls short of fully proving the speculative ( ? ) since our results are limited to Gaussian data , while there is no reason we are aware of to believe a tight uniform convergence guarantee of the form ( ? ) does not hold much more broadly . We leave extending our results beyond the Gaussian case open . 3 Generic Uniform Convergence Guarantee . To state our results , we first need to introduce some key tools . Definition 1 . The Gaussian width and the radius of a set S ⇢ Rd are W ( S ) : = E H⇠N ( 0 , Id ) sup s2S |hs , Hi| and rad ( S ) : = sup s2S ksk2 . The radius measures the size of a set in the Euclidean norm . The Gaussian width of a set S can be interpreted as the number of dimensions that a random projection needs to approximately preserve the norms of points in S ( Gordon 1988 ; Bandeira 2016 ) . These two complexity measures are connected by Gaussian concentration : Gaussian width is the expected value of the supremum of some Gaussian process , and the radius upper bounds the typical deviation of that supremum from its expected value . Definition 2 ( Covariance splitting ) . Given a positive semidefinite matrix ⌃ 2 Rd⇥d , we write ⌃ = ⌃1 ⌃2 if ⌃ = ⌃1 + ⌃2 , each matrix is positive semidefinite , and their spans are orthogonal . Note that ⌃ = ⌃1 ⌃2 effectively splits the eigenvectors of ⌃ into two disjoint parts . We can now state our generic bound . Section 7 sketches the proof ; all full proofs are in the appendix . Theorem 1 ( Main generalization bound ) . There exists an absolute constant C1 66 such that the following is true . Under the model assumptions in ( 1 ) , let K be an arbitrary compact set , and take any covariance splitting ⌃ = ⌃1 ⌃2 . Fixing 1/4 , let = C1 ✓q log ( 1/ ) n + q rank ( ⌃1 ) n ◆ . If n is large enough that 1 , then the following holds with probability at least 1 : sup w2K , L̂ ( w ) =0 L ( w ) 1 + n '' W ( ⌃1/22 K ) + ⇣ rad ( ⌃1/22 K ) + kw⇤k⌃2 ⌘ s 2 log ✓ 32 ◆ # 2 . In our applications , we consider K = { w 2 Rd : kwk B } for an arbitrary norm , with B based on a high-probability upper bound for kŵk . Depending on the application , the rank of ⌃1 will be 2When kxk2 concentrates , we need n to match its typical value . That is , n might be a high probability bound on kxk2 , or for ( sub ) Gaussian data , as in our case , n = E kxk2 . either constant or o ( n ) , so that ! 0 . The term kw⇤k⌃2 generally does not scale with n and hence is often negligible . As hinted earlier , we can think of the Gaussian width term and the radius term as bias and variance , respectively . To achieve consistency , we can expect that the Gaussian width should scale as p n. This agrees with the intuition that we need increasing norm to memorize noise when the model is not realizable . The radius term requires some care in our applications , but can be handled by the covariance splitting technique . As part of the analysis in the following sections , we will rigorously show in many settings that the dominant term in the upper bound is the Gaussian width . In these cases , our upper bound is roughly W ( ⌃1/22 K ) 2/n , which can be viewed as the ratio between the ( probabilistic ) dimension of our hypothesis class and sample size . We will also analyze how large K must be to contain any interpolators , allowing us to find consistency results .

Summary: This article deals with excess risk guarantees, under Gaussian noise, of linear regression with interpolators: estimators with zero training error. In particular, it highlights the role played by the norm of an interpolator in the generalization bounds. The proposed analysis covers a wide range of problems since it is valid for a large class of norms. In particular, the analysis covers the case of  $\ell_1$ norm interpolator and the  $\ell_2$ norm interpolator. These upper bounds were used to prove the consistency of the respective interpolators.