Ridge Regression: Structure, Cross-Validation, and Sketching

We study the following three fundamental problems about ridge regression : ( 1 ) what is the structure of the estimator ? ( 2 ) how to correctly use cross-validation to choose the regularization parameter ? and ( 3 ) how to accelerate computation without losing too much accuracy ? We consider the three problems in a unified large-data linear model . We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise . We study the bias of K-fold cross-validation for choosing the regularization parameter , and propose a simple bias-correction . We analyze the accuracy of primal and dual sketching for ridge regression , showing they are surprisingly accurate . Our results are illustrated by simulations and by analyzing empirical data . 1 INTRODUCTION . Ridge or ` 2-regularized regression is a widely used method for prediction and estimation when the data dimension p is large compared to the number of datapoints n. This is especially so in problems with many good features , where sparsity assumptions may not be justified . A great deal is known about ridge regression . It is Bayes optimal for any quadratic loss in a Bayesian linear model where the parameters and noise are Gaussian . The asymptotic properties of ridge have been widely studied ( e.g. , Tulino & Verdú , 2004 ; Serdobolskii , 2007 ; Couillet & Debbah , 2011 ; Dicker , 2016 ; Dobriban & Wager , 2018 , etc ) . For choosing the regularization parameter in practice , cross-validation ( CV ) is widely used . In addition , there is an exact shortcut ( e.g. , Hastie et al. , 2009 , p. 243 ) , which has good consistency properties ( Hastie et al. , 2019 ) . There is also a lot of work on fast approximate algorithms for ridge , e.g. , using sketching methods ( e.g. , el Alaoui & Mahoney , 2015 ; Chen et al. , 2015 ; Wang et al. , 2018 ; Chowdhury et al. , 2018 , among others ) . Here we seek to develop a deeper understanding of ridge regression , going beyond existing work in multiple aspects . We work in linear models under a popular asymptotic regime where n , p→∞ at the same rate ( Marchenko & Pastur , 1967 ; Serdobolskii , 2007 ; Couillet & Debbah , 2011 ; Yao et al. , 2015 ) . In this framework , we develop a fundamental representation for ridge regression , which shows that it is well approximated by a linear scaling of the true parameters perturbed by noise . The scaling matrices are functions of the population-level covariance of the features . As a consequence , we derive formulas for the training error and bias-variance tradeoff of ridge . Second , we study commonly used methods for choosing the regularization parameter . Inspired by the observation that CV has a bias for estimating the error rate ( e.g. , Hastie et al. , 2009 , p. 243 ) , we study the bias of CV for selecting the regularization parameter . We discover a surprisingly simple form for the bias , and propose a downward scaling bias correction procedure . Third , we study the accuracy loss of a class of randomized sketching algorithms for ridge regression . These algorithms approximate the sample covariance matrix by sketching or random projection . We show they can be surprisingly accurate , e.g. , they can sometimes cut computational cost in half , only incurring 5 % extra error . Even more , they can sometimes improve the MSE if a suboptimal regularization parameter is originally used . Our work leverages recent results from asymptotic random matrix theory and free probability theory . One challenge in our analysis is to find the limit of the trace tr ( Σ1 + Σ−12 ) −1/p , where Σ1 and Σ2 are p × p independent sample covariance matrices of Gaussian random vectors . The calculation requires nontrivial aspects of freely additive convolutions ( e.g. , Voiculescu et al. , 1992 ; Nica & Speicher , 2006 ) . Our work is connected to prior works on ridge regression in high-dimensional statistics ( Serdobolskii , 2007 ) and wireless communications ( Tulino & Verdú , 2004 ; Couillet & Debbah , 2011 ) . Among other related works , El Karoui & Kösters ( 2011 ) discuss the implications of the geometric sensitivity of random matrix theory for ridge regression , without considering our problems . El Karoui ( 2018 ) and Dicker ( 2016 ) study ridge regression estimators , but focus only on the risk for identity covariance . Hastie et al . ( 2019 ) study “ ridgeless ” regression , where the regularization parameter tends to zero . Sketching is an increasingly popular research topic , see Vempala ( 2005 ) ; Halko et al . ( 2011 ) ; Mahoney ( 2011 ) ; Woodruff ( 2014 ) ; Drineas & Mahoney ( 2017 ) and references therein . For sketched ridge regression , Zhang et al . ( 2013a ; b ) study the dual problem in a complementary finite-sample setting , and their results are hard to compare . Chen et al . ( 2015 ) propose an algorithm combining sparse embedding and the subsampled randomized Hadamard transform ( SRHT ) , proving relative approximation bounds . Wang et al . ( 2017 ) study iterative sketching algorithms from an optimization point of view , for both the primal and the dual problems . Dobriban & Liu ( 2018 ) study sketching using asymptotic random matrix theory , but only for unregularized linear regression . Chowdhury et al . ( 2018 ) propose a data-dependent algorithm in light of the ridge leverage scores . Other related works include Sarlos ( 2006 ) ; Ailon & Chazelle ( 2006 ) ; Drineas et al . ( 2006 ; 2011 ) ; Dhillon et al . ( 2013 ) ; Ma et al . ( 2015 ) ; Raskutti & Mahoney ( 2016 ) ; Gonen et al . ( 2016 ) ; Thanei et al . ( 2017 ) ; Ahfock et al . ( 2017 ) ; Lopes et al . ( 2018 ) ; Huang ( 2018 ) . The structure of the paper is as follows : We state our results on representation , risk , and biasvariance tradeoff in Section 2 . We study the bias of cross-validation for choosing the regularization parameter in Section 3 . We study the accuracy of randomized primal and dual sketching for both orthogonal and Gaussian sketches in Section 4 . We provide proofs and additional simulations in the Appendix . Code reproducing the experiments in the paper are available at https : //github . com/liusf15/RidgeRegression . 2 RIDGE REGRESSION . We work in the usual linear regression model Y = Xβ + ε , where each row xi of X ∈ Rn×p is a datapoint in p dimensions , and so there are p features . The corresponding element yi of Y ∈ Rn is its continous response ( or outcome ) . We assume mean zero uncorrelated noise , so Eε = 0 , and Cov [ ε ] = σ2In . We estimate the coefficient β ∈ Rp by ridge regression , solving the optimization problem β̂ = arg min β∈Rp 1 n ‖Y −Xβ‖22 + λ‖β‖22 , where λ > 0 is a regularization parameter . The solution has the closed form β̂ = ( X > X/n+ λIp ) −1 X > Y/n . ( 1 ) We work in a ” big data ” asymptotic limit , where both the dimension p and the sample size n tend to infinity , and their aspect ratio converges to a constant , p/n → γ ∈ ( 0 , ∞ ) . Our results can be interpreted for any n and p , using γ = p/n as an approximation . We recall that the empirical spectral distribution ( ESD ) of a p×p symmetric matrix Σ is the distribution 1p ∑p i=1 δλi where λi , i = 1 , . . . , p are the eigenvalues of Σ , and δx is the point mass at x . This is a standard notion in random matrix theory , see e.g. , Marchenko & Pastur ( 1967 ) ; Tulino & Verdú ( 2004 ) ; Couillet & Debbah ( 2011 ) ; Yao et al . ( 2015 ) . The ESD is a convenient tool to summarize all information obtainable from the eigenvalues of a matrix . For instance , the trace of Σ is proportional to the mean of the distribution , while the condition number is related to the range of the support . As is common , we will work in models where there is a sequence of covariance matrices Σ = Σp , and their ESDs converges in distribution to a limiting probability distribution . The results become simpler , because they depend only on the limit . By extension , we say that the ESD of the n× p matrix X is the ESD of X > X/n . We will consider some very specific models for the data , assuming it is of the form X = UΣ1/2 , where U has iid entries of zero mean and unit variance . This means that the datapoints , i.e. , the rows of X , have the form xi = Σ1/2ui , i = 1 , . . . , p , where ui have iid entries . Then Σ is the ” true ” covariance matrix of the features , which is typically not observed . These types of models for the data are very common in random matrix theory , see the references mentioned above . Under these models , it is possible to characterize precisely the deviations between the empirical covariance matrix Σ̂ = n−1X > X and the population covariance matrix Σ , dating back to the well known classical Marchenko-Pastur law for eigenvectors ( Marchenko & Pastur , 1967 ) , extended to more general models and made more precise , including results for eigenvectors ( see e.g. , Tulino & Verdú , 2004 ; Couillet & Debbah , 2011 ; Yao et al. , 2015 , and references therein ) . This has been used to study methods for estimating the true covariance matrix , with several applications ( e.g. , Paul & Aue , 2014 ; Bun et al. , 2017 ) . More recently , such models have been used to study high dimensional statistical learning problems , including classification and regression ( e.g. , Zollanvari & Genton , 2013 ; Dobriban & Wager , 2018 ) . Our work falls in this line . We start by finding a precise representation of the ridge estimator . For random vectors un , vn of growing dimension , we say un and vn are deterministic equivalents , if for any sequence of fixed ( or random and independent of un , vn ) vectors wn such that lim sup ‖wn‖2 < ∞ almost surely , we have |w > n ( un − vn ) | → 0 almost surely . We denote this by un vn . Thus linear combinations of un are well approximated by those of vn . This is a somewhat non-standard definition , but it turns out that it is precisely the one we need to use prior results from random matrix theory such as from ( Rubio & Mestre , 2011 ) . We extend scalar functions f : R→ R to matrices in the usual way by functional calculus , applying them to the eigenvalues and keeping the eigenvectors . If M = V ΛV > is a spectral decomposition of M , then we define f ( M ) : = V f ( Λ ) V > , where f ( Λ ) is the diagonal matrix with entries f ( Λii ) . For a fixed design matrix X , we can write the estimator as β̂ = ( Σ̂ + λIp ) −1Σ̂β + ( Σ̂ + λIp ) −1X > ε n . However , for a random design , we can find a representation that depends on the true covariance Σ , which may be simpler when Σ is simple , e.g. , when Σ = Ip is isotropic . Theorem 2.1 ( Representation of ridge estimator ) . Suppose the data matrix has the form X = UΣ1/2 , where U ∈ Rn×p has iid entries of zero mean , unit variance and finite 8 + c-th moment for some c > 0 , and Σ = Σp ∈ Rp×p is a deterministic positive definite matrix . Suppose that n , p→∞ with p/n→ γ > 0 . Suppose the ESD of the sequence of Σs converges in distribution to a probability measure with compact support bounded away from the origin . Suppose that the noise is Gaussian , and that β = βp is an arbitrary sequence of deterministic vectors , such that lim sup ‖β‖2 < ∞ . Then the ridge regression estimator is asymptotically equivalent to a random vector with the following representation : β̂ ( λ ) A ( Σ , λ ) · β +B ( Σ , λ ) · σ · Z p1/2 . Here Z ∼ N ( 0 , Ip ) is a random vector that is stochastically dependent only on the noise ε , and A , B are deterministic matrices defined by applying the scalar functions below to Σ : A ( x , λ ) = ( cpx+ λ ) −2 ( cp + c ′ p ) x , B ( x , λ ) = ( cpx+ λ ) −1cpx . Here cp : = c ( n , p , Σ , λ ) is the unique positive solution of the fixed point equation 1− cp = cp n tr [ Σ ( cpΣ + λI ) −1 ] . ( 2 ) This result gives a precise representation of the ridge regression estimator . It is a sum of two terms : the true coefficient vector β scaled by the matrixA ( Σ , λ ) , and the noise vectorZ scaled by the matrix B ( Σ , λ ) . The first term captures to what extent ridge regression recovers the ” signal ” . Morever , the noise term Z is directly coupled with the noise in the original regression problem , and thus also the estimator . The result would not hold for an independent noise vector Z . However , the coefficients are not fully explicit , as they depend on the unknown population covariance matrix Σ , as well as on the fixed-point variable cp . Some comments are in order : 1 . Structure of the proof . The proof is quite non-elementary and relies on random matrix theory . Specifically , it uses the language of the recently developed ” calculus of deterministic equivalents ” ( Dobriban & Sheng , 2018 ) , and results by ( Rubio & Mestre , 2011 ) . A general takeaway is that for n not much larger than p , the empirical covariance matrix Σ̂ is not a good estimator of the true covariance matrix Σ . However , the deviation of linear functionals of Σ̂ , can be quantified . In particular , we have ( Σ̂ + λI ) −1 ( cpΣ + λI ) −1 , in the sense that linear combinations of the entries of the two matrices are close ( see the proof for more details ) . 2 . Understanding the resolvent bias factor cp . Thus , cp can be viewed as a resolvent bias factor , which tells us by what factor Σ is multiplied when evaluating the resolvent ( Σ̂ + λI ) −1 , and comparing it to its naive counterpart ( Σ + λI ) −1 . It is known that cp is well defined , and this follows by a simple monotonicity argument , see Hachem et al . ( 2007 ) ; Rubio & Mestre ( 2011 ) . Specifically , the left hand side of ( 2 ) is decreasing in cp , while the right hand size is increasing in Also c′p is the derivative of cp , when viewing it as a function of z : = −λ . An explicit expression is provided in the proof in Section A.1 , but is not crucial right now . Here we discuss some implications of this representation . For uncorrelated features , Σ = Ip , A , B reduce to multiplication by scalars . Hence , each coordinate of the ridge regression estimator is simply a scalar multiple of the corresponding coordinate of β . One can use this to find the bias in each individual coordinate . Training error and optimal regularization parameter . This theorem has implications for understanding the training error , and optimal regularization parameter of ridge regression . As it stands , the theorem itself only characterizes the behavior og linear combinations of the coordinates of the estimator . Thus , it can be directly applied to study the bias Eβ̂ ( λ ) − β of the estimator . However , it can not directly be used to study the variance ; as that would require understanding quadratic functionals of the estimator . This seems to require significant advances in random matrix theory , going beyond the results of Rubio & Mestre ( 2011 ) . However , we show below that with additional assumptions on the structure of the parameter β , we can derive the MSE of the estimator in other ways . We work in a random-effects model , where the p-dimensional regression parameter β is random , each coefficient has zero mean Eβi = 0 , and is normalized so that Varβi = α2/p . This ensures that the signal strength E‖β‖2 = α2 is fixed for any p. The asymptotically optimal λ in this setting is always λ∗ = γσ2/α2 see e.g. , Tulino & Verdú ( 2004 ) ; Dicker ( 2016 ) ; Dobriban & Wager ( 2018 ) . The ridge regression estimator with λ = pσ2/ ( nα2 ) is the posterior mean of β , when β and ε are normal random variables . For a distribution F , we define the quantities θi ( λ ) = ∫ 1 ( x+ λ ) i dFγ ( x ) , ( i = 1 , 2 , . . . ) . These are the moments of the resolvent and its derivatives ( up to constants ) . We use the following loss functions : mean squared estimation error : M ( β̂ ) = E‖β̂ − β‖22 , and residual or training error : R ( β̂ ) = E [ ‖ ] Y −Xβ̂‖22 . Theorem 2.2 ( MSE and training error of ridge ) . Suppose β has iid entries with Eβi = 0 , Var [ βi ] = α2/p , i = 1 , . . . , p and β is independent of X and ε . Suppose X is an arbitrary n × p matrix depending on n and p , and the ESD of X converges weakly to a deterministic distribution F as n , p → ∞ and p/n → γ . Then the asymptotic MSE and residual error of the ridge regression estimator β̂ ( λ ) has the form lim n→∞ M ( β̂ ( λ ) ) = α2λ2θ2 + γσ 2 [ θ1 − λθ2 ] , ( 3 ) lim n→∞ R ( β̂ ( λ ) ) = α2λ2 [ θ1 − λθ2 ] + σ2 [ 1− γ ( 1 + λθ1 − λ2θ2 ) ] , ( 4 ) Bias-variance tradeoff . Building on this , we can also study the bias-variance tradeoff of ridge regression . Qualitatively , large λ leads to more regularization , and decreases the variance . However , it also increases the bias . Our theory allows us to find the explicit formulas for the bias and variance as a function of λ . See Figure 1 for a plot and Sec . A.3 for the details . As far as we know , this is one of the few examples of high-dimensional asymptotic problems where the precise form of the bias and variance can be evaluated . Bias-variance tradeoff at optimal λ∗ = γσ2/α2 . ( see Figure 6 ) This can be viewed as the ” pure ” effect of dimensionality on the problem , keeping all other parameters fixed , and has intriguing properties . The variance first increases , then decreases with γ . In the ” classical ” low-dimensional case , most of the risk is due to variance , while in the ” modern ” high-dimensional case , most of it is due to bias . This is consistent with other phenomena in proportional-limit asymptotics , e.g. , that the map between population and sample eigenvalue distributions is asymptotically deterministic ( Marchenko & Pastur , 1967 ) . Future applications . This fundamental representation may have applications to important statistical inference questions . For instance , inference on the regression coefficient β and the noise variance σ2 are important and challenging problems . Can we use our representation to develop debiasing techniques for this task ? This will be interesting to explore in future work .

Summary: This paper presents a theoretical study of ridge regression, focusing on the practical problems of correcting for the bias of the cross-validation based estimate of the optimal regularisation parameter, and quantification of the asymptotic risk of sketching algorithms for ridge regression, both in the p / n -> gamma in (0, 1) regime (n = # data points, p = # dimensions). The authors derive most of their results exploiting their (AFAICT) new asymptotic characterisation of the ridge regression estimator which may be of independent interest. The whole study is complemented by a series of numerical experiments.