Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds and Benign Overfitting

1 Introduction . The traditional understanding of machine learning suggests that models with zero training error tend to overfit , and explicit regularization is often necessary to achieve good generalization . Given the empirical success of deep learning models with zero training error ( Neyshabur et al . 2015 ; Zhang et al . 2017 ) and the ( re- ) discovery of the “ double descent ” phenomenon ( Belkin et al . 2019 ) , however , it has become clear that the textbook U-shaped learning curve is only part of a larger picture : it is possible for an overparameterized model with zero training loss to achieve low population error in a noisy setting . In an effort to understand how interpolation learning occurs , there has been much recent study of the testbed problem of linear regression with Gaussian features ( e.g . Hastie et al . 2019 ; Bartlett et al . 2020 ; Belkin et al . 2020 ; Ju et al . 2020 ; Muthukumar et al . 2020 ; Negrea et al . 2020 ; Tsigler and Bartlett 2020 ; Zhou et al . 2020 ) . Significant progress has been made in this setting , including nearly-matching necessary and sufficient conditions for consistency of the minimal ` 2 norm interpolator ( Bartlett et al . 2020 ) . Despite the fundamental role of uniform convergence in statistical learning theory , most of this line of work has used other techniques to analyze the particular minimal-norm interpolator.1 Instead of directly analyzing the population error of a learning algorithm , a uniform convergence-type argument would control the worst-case generalization gap over a class of predictors containing the typical outputs of a learning rule . Typically , this is done because for many algorithms – unlike the minimal Euclidean norm interpolator – it is difficult to exactly characterize the learned predictor , but we may be able to say e.g . that its norm is not too large . Since uniform convergence does not tightly depend on a specific algorithm , the resulting analysis can highlight the key properties that lead to good generalization : it can give bounds not only for , say , the minimal-norm interpolator , but also ⇤These authors contributed equally . 1Negrea et al . ( 2020 ) argue that Bartlett et al . ( 2020 ) ’ s proof technique is fundamentally based on uniform convergence of a surrogate predictor ; Yang et al . ( 2021 ) study a closely related setting with a uniform convergence-type argument , but do not establish consistency . We discuss both papers in more detail in Section 4 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for other interpolators with low norm ( e.g . Zhou et al . 2020 ) , increasing our confidence that low norm – and not some other property the particular minimal-norm interpolator happens to have – is key to generalization . In linear regression , practical training algorithms may not always find the exact minimal Euclidean norm solution , so it is also reassuring that all interpolators with sufficiently low Euclidean norm generalize . Nagarajan and Kolter ( 2019 ) , however , raised significant questions about the applicability of typical uniform convergence arguments to certain high-dimensional regimes , similar to those seen in interpolation learning . Following their work , Bartlett and Long ( 2020 ) , Negrea et al . ( 2020 ) , Zhou et al . ( 2020 ) , and Yang et al . ( 2021 ) all demonstrated the failure of forms of uniform convergence in various interpolation learning setups . To sidestep these negative results , Zhou et al . ( 2020 ) suggested considering bounds which are uniform only over predictors with zero training error . This weaker notion of uniform convergence has been standard in analyses of “ realizable ” ( noiseless ) learning at least since the work of Vapnik ( 1982 , Chapter 6.4 ) and Valiant ( 1984 ) . Zhou et al . demonstrated that at least in one particular noisy setting , such uniform convergence is sufficient for showing consistency of the minimal ` 2 norm interpolator , even though “ non-realizable ” uniform convergence arguments ( those over predictors regardless of their training error ) can not succeed . It remains unknown , however , whether these types of arguments can apply to more general linear regression problems and more typical asymptotic regimes , particularly showing rates of convergence rather than just consistency . In this work , we show for the first time that uniform convergence is indeed able to explain benign overfitting in general high-dimensional Gaussian linear regression problems . Similarly to how the standard analysis for learning with Lipschitz losses bounds generalization gaps through Rademacher complexity ( e.g . Shalev-Shwartz and Ben-David 2014 ) , our Theorem 1 ( Section 3 ) establishes a finitesample high probability bound on the uniform convergence of the error of interpolating predictors in a hypothesis class , in terms of its Gaussian width . This is done through an application of the Gaussian Minimax Theorem ; see the proof sketch in Section 7 . Combined with an analysis of the norm of the minimal ` 2 norm interpolator ( Theorem 2 in Section 4 ) , our bound recovers known consistency results ( Bartlett et al . 2020 ) , as well as proving a conjectured upper bound for larger-norm interpolators ( Zhou et al . 2020 ) . In addition , since we do not restrict ourselves to Euclidean norm balls but instead consider interpolators in an arbitrary compact set , our results allows for a wide range of other applications . Our analysis leads to a natural extension of the consistency result and notions of effective rank of Bartlett et al . ( 2020 ) for arbitrary norms ( Theorem 5 in Section 5 ) . As a demonstration of our general theory , in Section 6 we show novel consistency results for the minimal ` 1 norm interpolator ( basis pursuit ) in particular settings , which we believe are the first results of their kind . 2 Problem Formulation . Notation . We use k·kp for the ` p norm , kxkp = ( P i |xi|p ) 1/p . We always use maxx2S f ( x ) to be 1 when S is empty , and similarly minx2S f ( x ) to be 1 . We use standard O ( · ) notation , and a . b for inequality up to an absolute constant . For a positive semidefinite matrix A , the Mahalanobis ( semi- ) norm is kxk2 A : = hx , Axi . For a matrix A and set S , AS denotes the set { Ax : x 2 S } . Data model . We assume that data ( X , Y ) is generated as Y = Xw⇤ + ⇠ , Xi iid⇠ N ( 0 , ⌃ ) , ⇠ ⇠ N ( 0 , 2In ) , ( 1 ) where X 2 Rn⇥d has i.i.d . Gaussian rows X1 , . . . , Xn , d n , w⇤ is arbitrary , and ⇠ is Gaussian and independent of X . Though our proof techniques crucially depend on Xi being Gaussian , we can easily relax the assumption on the noise ⇠ to only being sub-Gaussian ; we assume Gaussian noise here for simplicity . The empirical and population loss are defined as , respectively , L̂ ( w ) = 1 n kY Xwk22 , L ( w ) = E ( x , y ) ( y hw , xi ) 2 = 2 + kw w⇤k2⌃ , where in the expectation y = hx , w⇤i+ ⇠0 with x ⇠ N ( 0 , ⌃ ) independent of ⇠0 ⇠ N ( 0 , 2 ) . For an arbitrary norm k·k , the minimal norm interpolator is ŵ = argmin L̂ ( w ) =0 kwk . For Euclidean norm specifically , the minimal norm interpolator can be written explicitly as ŵ = X ( XXT ) 1Y . If there is more than one minimal norm interpolator , all of our guarantees will hold for any minimizer ŵ . Speculative bound . Zhou et al . ( 2020 ) studied uniform convergence of low norm interpolators , sup kwkB , L̂ ( w ) =0 L ( w ) L̂ ( w ) . ( 2 ) Clearly , when B kŵk , this quantity upper-bounds the population risk of ŵ . Zhou et al . evaluated the asymptotic limit of ( 2 ) in one particular setting . But they further speculated that a bound of the following form may hold more generally : sup kwk2B , L̂ ( w ) =0 L ( w ) L̂ ( w ) B 2 n n + o ( 1 ) , ( ? ) where2 n ⇡ kxk2 . As discussed by Zhou et al. , a bound almost of this form is implied by results of Srebro et al . ( 2010 ) for general data distributions , except that approach gives a large leading constant and logarithmic factors . To show consistency of benign overfitting , though , we need ( ? ) to hold without even a constant multiplicative factor . Zhou et al . ask whether and when this holds , speculating that it might in broad generality . The goal of this paper is essentially to prove ( ? ) , at least for Gaussian data , and to use it to show consistency of the minimal norm interpolator . Our main result ( Theorem 1 ) can be thought of showing ( ? ) for Gaussian data with n = E kxk2 , as well as strengthening and significantly generalizing it . Our result is more general , as it applies to general compact hypothesis sets beyond just the Euclidean ball as in ( ? ) . But it also falls short of fully proving the speculative ( ? ) since our results are limited to Gaussian data , while there is no reason we are aware of to believe a tight uniform convergence guarantee of the form ( ? ) does not hold much more broadly . We leave extending our results beyond the Gaussian case open . 3 Generic Uniform Convergence Guarantee . To state our results , we first need to introduce some key tools . Definition 1 . The Gaussian width and the radius of a set S ⇢ Rd are W ( S ) : = E H⇠N ( 0 , Id ) sup s2S |hs , Hi| and rad ( S ) : = sup s2S ksk2 . The radius measures the size of a set in the Euclidean norm . The Gaussian width of a set S can be interpreted as the number of dimensions that a random projection needs to approximately preserve the norms of points in S ( Gordon 1988 ; Bandeira 2016 ) . These two complexity measures are connected by Gaussian concentration : Gaussian width is the expected value of the supremum of some Gaussian process , and the radius upper bounds the typical deviation of that supremum from its expected value . Definition 2 ( Covariance splitting ) . Given a positive semidefinite matrix ⌃ 2 Rd⇥d , we write ⌃ = ⌃1 ⌃2 if ⌃ = ⌃1 + ⌃2 , each matrix is positive semidefinite , and their spans are orthogonal . Note that ⌃ = ⌃1 ⌃2 effectively splits the eigenvectors of ⌃ into two disjoint parts . We can now state our generic bound . Section 7 sketches the proof ; all full proofs are in the appendix . Theorem 1 ( Main generalization bound ) . There exists an absolute constant C1 66 such that the following is true . Under the model assumptions in ( 1 ) , let K be an arbitrary compact set , and take any covariance splitting ⌃ = ⌃1 ⌃2 . Fixing 1/4 , let = C1 ✓q log ( 1/ ) n + q rank ( ⌃1 ) n ◆ . If n is large enough that 1 , then the following holds with probability at least 1 : sup w2K , L̂ ( w ) =0 L ( w ) 1 + n '' W ( ⌃1/22 K ) + ⇣ rad ( ⌃1/22 K ) + kw⇤k⌃2 ⌘ s 2 log ✓ 32 ◆ # 2 . In our applications , we consider K = { w 2 Rd : kwk B } for an arbitrary norm , with B based on a high-probability upper bound for kŵk . Depending on the application , the rank of ⌃1 will be 2When kxk2 concentrates , we need n to match its typical value . That is , n might be a high probability bound on kxk2 , or for ( sub ) Gaussian data , as in our case , n = E kxk2 . either constant or o ( n ) , so that ! 0 . The term kw⇤k⌃2 generally does not scale with n and hence is often negligible . As hinted earlier , we can think of the Gaussian width term and the radius term as bias and variance , respectively . To achieve consistency , we can expect that the Gaussian width should scale as p n. This agrees with the intuition that we need increasing norm to memorize noise when the model is not realizable . The radius term requires some care in our applications , but can be handled by the covariance splitting technique . As part of the analysis in the following sections , we will rigorously show in many settings that the dominant term in the upper bound is the Gaussian width . In these cases , our upper bound is roughly W ( ⌃1/22 K ) 2/n , which can be viewed as the ratio between the ( probabilistic ) dimension of our hypothesis class and sample size . We will also analyze how large K must be to contain any interpolators , allowing us to find consistency results .

Summary: The authors study generalization bounds for minimum norm linear interpolants. They use techniques from uniform convergence to derive a general bound that applies to minimum norm linear interpolant, that depends on both the norm of the optimal linear predictor as well as the Gaussian width and radius of the corresponding norm ball. In the case of the Euclidean norm the authors re-derive the sufficient conditions previously obtained by Bartlett et al. (2020) by deriving an upper bound on the norm of the predictor. In the case of the l_1 norm, the authors provide a set of sufficient conditions for the minimum norm interpolant to be consistent and discuss the implications in two special cases.