On Optimal Interpolation in Linear Regression

1 Introduction . Establishing mathematical understanding for the good generalization properties of interpolating methods , i.e . methods that fit the training data perfectly , has attracted significant interest in recent years . Motivated by the quest to explain the generalization performance of neural networks which have zero training error , for example even on randomly corrupted data ( Zhang et al. , 2017 ) , this area of research has established results for a variety of models . For instance , in kernel regression , Liang and Rakhlin ( 2020 ) provide a data-dependent upper bound on the generalization performance of the minimum-norm interpolator . By analyzing the upper bound , they show that small generalization error of the minimum-norm interpolator occurs in a regime with favourable curvature of the kernel , particular decay of the eigenvalues of the kernel and data population covariance matrices and , importantly , in an overparametrized setting . In random features regression , Mei and Montanari ( 2019 ) showed that for large signal-to-noise ratio and in the limit of large overparametrization , the optimal regularization is zero , i.e . the optimal ridge regressor is an interpolator . Liang and Sur ( 2020 ) characterized the precise high-dimensional asymptotic generalization of interpolating minimum ` 1-norm classifiers and boosting algorithms which maximize the ` 1 margin . Bartlett et al . ( 2020 ) isolated a setting of benign overfitting in linear regression , dependent on notions of effective rank of the population covariance matrix , in which the minimum-norm interpolator has small generalization error . Similarly , this regime of benign overfitting occurs with large overparametrization . Linear models , in particular , provide a fundamental playground to understand interpolators . On the one hand , in overparametrized regimes , interpolators in linear models are seen to reproduce 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . stylized phenomena observed in more general models . For example , the double descent phenomenon , which was first empirically observed in neural networks ( Belkin et al. , 2019 ) , has also featured in linear regression ( Hastie et al. , 2019 ) . On the other hand , neural networks are known to be well-approximated by linear models in some regimes . For example , with specific initialization and sufficient overparametrization , two-layer neural networks trained with gradient descent methods are well-approximated by a first-order Taylor expansion around their initialization ( Chizat et al. , 2019 ) . This linear approximation can be split into a random features component and a neural-tangent component . The random features model , a two-layer neural network with randomly initialized first layer which is fixed during training , shares similar generalization behavior with the full neural network ( Bartlett et al. , 2021 ) , and as such , the random features model provides a natural stepping stone towards tackling a theoretical understanding of neural networks . A major focus of the interpolation literature has so far been to theoretically study if and when interpolating methods based on classical techniques such as ridge regression and gradient descent can have optimal or near-optimal generalization ( Bartlett et al. , 2021 ) . However the question of understanding which interpolators are best , and designing data-dependent schemes to implement them , seems to have received only partial attention . Work investigating which interpolators are optimal in linear regression includes Muthukumar et al . ( 2019 ) , where the authors constructed the best-possible interpolator , i.e . a theoretical device which uses knowledge of the true parameter and training noise vector to establish a fundamental limit on how well any interpolator in linear regression can generalize . When the whitened features are sub-Gaussian , this fundamental limit is lower bounded by a term proportional to n/d , up to an additive constant and with high probability , which is small only in the regime of large overparametrization . Here , n and d are the size and the dimension of the data . While this interpolator provides the best-possible generalization error , the interpolator is not implementable in general , as one would need access to the realization of the true data-generating parameter w ? and the realization of the noise in the training data . Rangamani et al . ( 2020 ) studied generalization of interpolators in linear regression and showed that the minimum-norm interpolator minimizes an upper bound on the generalization error related to stability . In ( Mourtada , 2020 ) , it was shown that the minimum-norm interpolator is minimax optimal over any choice of the true parameter w ? ∈ Rd , distributions of the noise with mean 0 and bounded variance , and for a fixed nondegenerate distribution of the features . Amari et al . ( 2021 ) computed the asymptotic risk of interpolating preconditioned gradient descent in linear regression and investigated the role of its implicit bias on generalization . In particular , they identified the preconditioning which leads to optimal asymptotic ( as d/n → γ > 1 with n , d → ∞ ) bias and variance , separately , among interpolators of the form w = PXT ( XPXT ) −1y for some matrix P , where X ∈ Rn×d is the data matrix , y ∈ Rn is the response vector . They showed that , within this class of interpolators , using the inverse of the population covariance matrix of the data as preconditioning achieves optimal asymptotic variance . However , the interpolator with optimal risk is not given . In this paper , we study the question of what is the optimal way to interpolate in overparametrized linear regression by procedures that do not use the realization of the true parameter generating the data , nor the realization of the training noise . The motivation for studying this question is twofold . First , in designing new ways to interpolate that are directly related to notions of optimality in linear models , we hope to provide a stepping stone to designing new ways to interpolate in more complex models , such as neural networks . Second , our results illustrate that there can be arbitrarily large differences in the generalization performance of interpolators , in particular considering the minimum-norm interpolator as a benchmark . This is a phenomenon that does not seem to have received close attention in the literature and may spark new interest in designing interpolators connected to optimality . We consider the family of interpolators that can be achieved as an arbitrary function f of the data , population covariance , signal-to-noise ratio and the prior covariance such that f is linear in the response variable y ( as the case for the Bayes optimal estimator in ridge regression ) . We call such interpolators response-linear achievable ( see Definition 3 ) . We also introduce a natural notion of optimality that assumes that the realization of true data-generating parameter and the realization of the noise in the training data are unknown . Within this class of interpolators and under this notion of optimality , we theoretically compute the optimal interpolator and show that this interpolator is achieved as the implicit bias of a preconditioned gradient descent with proper initialization . We refer to this interpolator as the optimal response-linear achievable interpolator . Could it be that the commonly used minimum-norm interpolator is good enough so that the benefit of finding a better interpolator is negligible ? We illustrate that the answer to this question is , in general , no . In particular , we construct an example in linear regression where the minimum-norm interpolator has arbitrarily worse generalization than the optimal response-linear achievable interpolator . Here , the variance ( hence also generalization error ) of the minimum-norm interpolator diverges to infinity as a function of the eigenvalues of the population covariance matrix , while the generalization error of the optimal response-linear achievable interpolator stays bounded , close to the optimal interpolator , i.e . the theoretical device of Muthukumar et al . ( 2019 ) which uses the value of the signal and noise . The optimal response-linear achievable interpolator uses knowledge of the population covariance matrix of the data ( similarly as in Amari et al . ( 2021 ) ) , the signal-to-noise ratio , and the covariance of the true parameter ( on which we place a prior distribution ) . Is it the case that the better performance of our interpolator is simply a consequence of this population knowledge ? We provide numerical evidence that shows that the answer to this question is , in general , no . In particular , we construct an algorithm to approximate the optimal response-linear achievable interpolator which does not require any prior knowledge of the population covariance or the signal-to-noise ratio and uses only the training data X and y , and we empirically observe that this new interpolator generalizes in a nearly identical way to the optimal response-linear achievable interpolator . Finally , we show that the concept of optimal response-linear achievable interpolation can be extended to more complex models by providing analogous results for a random features model under the same linear data-generating regime as also considered in ( Mei and Montanari , 2019 ) , for instance . 2 Problem setup . In this paper we investigate overparametrized linear regression . We assume there exists w ? ∈ Rd ( unknown ) so that yi = 〈w ? , xi〉+ ξi for i ∈ { 1 , . . . , n } , with i.i.d . noise ξi ∈ R ( unknown ) such that E ( ξi ) = 0 , E ( ξ2i ) = σ2 and i.i.d . features xi ∈ Rd that follow a distribution Px with mean E ( xi ) = 0 and covariance matrix E ( xixTi ) = Σ . We store the features in a random matrix X ∈ Rn×d with rows xi ∈ Rd , the response variable in a random vector y ∈ Rn with entries yi ∈ R , and the noise in a random vector ξ ∈ Rn with entries ξi ∈ R. Throughout the paper we assume that d ≥ n. We consider the whitened data matrix Z = XΣ− 1 2 ∈ Rn×d , whose rows satisfy E ( zizTi ) = Id , where Id ∈ Rd×d is the identity matrix . We place a prior on the true parameter in the form w ? ∼ Pw ? such that E ( w ? ) = 0 and E ( w ? w ? T ) = r 2 d Φ . Here , Φ is a positive definite matrix and r 2 is the signal . We sometimes abuse terminology and refer to Φ as the covariance of the prior even though r2 d Φ is the covariance matrix . Our results will be proved in general , but for the sake of exposition it can be assumed that xi ∼ N ( 0 , Σ ) , ξ ∼ N ( 0 , σ2In ) and w ? ∼N ( 0 , r 2 d Φ ) . We also define the signal-to-noise ratio δ = r2/σ2 and consider the squared error loss ` : ( x , y ) ∈ R2 7→ ( x − y ) 2 . Througout the paper , we assume the following two technical conditions hold . Assumption 1 . Px ( xi ∈ V ) = 0 for any linear subspace V of Rd with dimension smaller than d. Assumption 2 . For all Lebesgue measurable sets A ⊆ Rd , ν ( A ) > 0 implies Pw ? ( w ? ∈ A ) > 0 , where ν is the standard Lebesgue measure on Rd . Assumption 1 is needed only so that rank ( X ) = n with probability 1 ( for a proof see A.4 ) . A sufficient condition is that Px has a density on Rd . A sufficient condition for Assumption 2 is that Pw ? has a positive density on Rd . Now , our goal is to minimize the population risk r ( w ) = Ex̃ , ξ̃ ( ( 〈w , x̃〉 − ỹ ) 2 ) , or , equivalently , the excess risk r ( w ) − r ( w ? ) . Here , ( x̃ , ỹ , ξ̃ ) is a random variable which follows the distribution of ( x1 , y1 , ξ1 ) , . . . , ( xn , yn , ξn ) and is independent from them . Throughout the paper we write Ezg ( z , z̃ ) to denote the conditional expectation E ( g ( z , z̃ ) |z̃ ) , for two random variables z and z̃ and for a function g. The population risk satisfies r ( w ) = ( w − w ? ) TΣ ( w − w ? ) + σ2 = ‖w − w ? ‖2Σ + r ( w ? ) , ( 1 ) where ‖w‖2Σ = wTΣw . We define the bias and variance of w ∈ Rd by the decomposition Eξ , w ? r ( w ) = B ( w ) + V ( w ) , ( 2 ) where B ( w ) = Eξ , w ? ‖E ( w|w ? , X ) − w ? ‖2Σ V ( w ) = Eξ , w ? ‖w − E ( w|w ? , X ) ‖2Σ . ( 3 ) One of the main paradigms to minimize the ( unknown ) population risk is based on minimizing the empirical risk R ( w ) = 1n ∑n i=1 ( 〈w , xi〉 − yi ) 2 = 1 n ∑n i=1 ` ( w Txi , yi ) ( Vapnik , 1995 ) . In our setting , minimizing the empirical risk is equivalent to finding w ∈ Rd such that Xw = y .

Summary: This paper is concerned with inducing a certain type of best possible interpolator in linear regression. It is based on the notion of best possible interpolator, defined in Muthukumar et al., 2019., which is the best estimator (i.e., with minimum risk) among the class of interpolators. Based on that, the authors define a new theoretical estimator, the best estimator among interpolators that are also linear in the response variable.  They manage to induce a closed form of this estimator and prove that it is the limit of preconditional gradient descent under a specific initialization. In addition, using approximations for quantities that are usually unknown in practice (e.g.  population covariance matrix, signal to noise ratio), they turn this theoretical estimator into a fully empirical one and compare its performance to other frequently used estimators, showing that this estimator performs arbitrarily better in certain cases. Finally, they extend this notion of best possible theoretical estimator to the random features setting, where they again induce a closed form for this estimator and prove that it is the limit of preconditioned gradient descent under a specific initialization.