Can we globally optimize cross-validation loss? Quasiconvexity in ridge regression

1 Introduction . Linear models , including LASSO and ridge regression , are widely used for data analysis across diverse applied disciplines . Linear models are often preferred since they are straightforward to apply in various senses . In particular , ( 1 ) their parameters are readily interpretable . ( 2 ) They have strong theoretical guarantees on quality . And ( 3 ) standard optimization tools are often assumed to find useful parameter and hyperparameter values . Despite their seeming simplicity , though , mysteries remain about the quality of inference in linear models . Consider cross-validation ( CV ) [ Stone , 1974 , Allen , 1974 ] , the de facto standard for hyperparameter selection across machine learning methods [ Musgrave et al. , 2020 ] . CV is an easy-to-evaluate proxy for the true out-of-sample loss . Is it a good proxy ? [ Homrighausen and McDonald , 2014 , 2013 , Chetverikov et al. , 2020 , Hastie et al. , 2020 , Patil et al. , 2021 ] give conditions under which the global minimum of the CV loss ( possibly with some mild corrections ) matches the optimum of the out-of-sample loss in LASSO and ridge regression . To complete the picture , we must understand whether standard methods for minimizing the CV loss find a global minimum . It would be easy to find a unique minimum of the CV loss if the CV loss were convex . Alas ( though perhaps unsurprisingly ) , we show below that in essentially every case of interest the CV loss is not convex . Indeed , the usual introductory cartoon of CV loss ( left panel of Fig . 1 ; see also Fig . 5.9 of Hastie et al . [ 2017 ] or Fig . 1 of Rad and Maleki [ 2020 ] ) is not convex . But the cartoon CV loss still exhibits a single global minimum and would be easy to globally minimize with popular approaches like gradient-based methods [ Do et al. , 2007 , Maclaurin et al. , 2015 , Pedregosa , 2016 , Lorraine et al. , 2020 ] or grid search [ Bergstra and Bengio , 2012 , Pedregosa et al. , 2011 , Hsu et al. , 2003 ] . Indeed , a ∗Alternate email : wtstephe @ gmail.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . more plausible possibility ( which holds for the typical cartoon CV loss ) is that the CV loss might be quasiconvex ; i.e . its level sets are convex . The benefit of quasiconvexity is that , in one dimension , any local optimum is a global optimum . Unfortunately , this cartoon need not hold in general , even in simple models like ℓ2-regularized linear regression ( i.e . ridge regression ) . Consider minimizing leave-one-out CV ( LOOCV ) loss as a function of the ridge regularization parameter ; we denote this loss by L. Wilson et al . [ 2020 , Fig . 1 ] detail a simulated example in which L can be non-quasiconvex . We first demonstrate that L can be non-quasiconvex in real-data examples ; see the middle and right panel of Fig . 1 , which we describe in detail in Section 3 . We next characterize which aspects of the covariate matrix and observed responses affect quasiconvexity . We prove that the norm of the responses , the scale of the singular values of the covariate matrix , and the right singular vectors of the covariate matrix all have no effect on the quasiconvexity of L. While this result places substantial constraints on what drives the quasiconvexity of L , we show that the quasiconvexity of L is unfortunately still a complex function of the remaining quantities . Our third contribution is to prove conditions under which L is guaranteed to be quasiconvex . In particular , we show that if ( 1 ) the covariate matrix has a singular value spectrum sufficiently close to uniform , ( 2 ) the least-squares estimator fits the training data sufficiently well , and ( 3 ) the left singular vectors of the covariate matrix are sufficiently regular , then L is guaranteed to be quasiconvex . While the conditions of our theory are deterministic , we show that they have natural probabilistic interpretations ; as a corollary to our theory , we demonstrate that many of our conditions are satisfied either empirically or theoretically by well-specified linear regression problems with i.i.d . sub-Gaussian covariates and moderate signal-to-noise ratios . Through empirical studies , we validate the conclusions of our theory and the necessity of our assumptions . 2 Setup and notation . For n ∈ { 1 , . . . , N } , we observe covariates xn ∈ RD and responses yn ∈ R. We are interested in learning a linear model between the covariates and responses , ⟨xn , θ⟩ ≈ yn , for some parameter θ ∈ RD . In ridge regression , i.e . ℓ2-regularized linear regression , we take some λ > 0 and estimate : θ̂ ( λ ) : = argmin θ∈RD N∑ n=1 ( ⟨xn , θ⟩ − yn ) 2 + λ 2 ∥θ∥22 . ( 1 ) The regularization parameter λ is typically chosen by minimizing the cross-validation ( CV ) loss . Here we study the leave-one-out CV ( LOOCV ) loss : L ( λ ) : = N∑ n=1 ( ⟨xn , θ̂\n ( λ ) ⟩ − yn ) 2 , ( 2 ) where θ̂\n ( λ ) is the solution to Eq . ( 1 ) with the nth datapoint left out . Let the covariate matrix X ∈ RN×D be the matrix with rows xn , and let the vector Y ∈ RN be the vector with entries yn . We consider the low to modest-dimensional case where D < N and assume the covariate matrix X is full-rank . We further assume X and Y have undergone standard data pre-processing , as described next . Condition 1 . Y is zero-mean , and X has zero-mean , unit variance columns . Equivalently , where 1 ∈ RN is the vector of all ones , 1TY = 0 and XT1 = 0 ∈ RD and for all d = 1 , . . . , D , ∑N n=1 x 2 nd = N . Preprocessing X and Y to satisfy Condition 1 represents standard best practice for ridge regression . First , using an unregularized bias parameter in Eq . ( 1 ) and setting Y to be zero-mean are equivalent ; we choose to make Y zero-mean , as it simplifies our analysis below . The conditions on the covariate matrix X are important to ensure the use of ℓ2-regularization is sensible . In particular , Eq . ( 1 ) penalizes all coordinates of θ equally . If e.g . some columns of X are measured in different scales or are centered differently , this uniform penalty will be inappropriate . 3 LOOCV loss is typically not convex and need not be quasiconvex . If the LOOCV loss L were convex or quasiconvex in λ , then any local minimum of L would be a global minimum , and we could trust gradient-based optimization methods or grid search methods to return a value near a global minimum . We next see that unfortunately L is typically not convex and is often not even quasiconvex . First we show that , in essentially all cases of interest , L is not convex . Proposition 1 . If λ = ∞ is not a minimum of L , then L is not a convex function . Proof . For the sake of contradiction , assume L is convex and λ = ∞ is not a minimum of L. This implies that there is some maximal λ∗ < ∞ such that L′ ( λ∗ ) = 0 . Let δ : = L′ ( λ∗ + 1 ) . By convexity , L′′ ≥ 0 , so we know that δ > 0 and that for λ ≥ λ∗ + 1 , we have L′ ( λ ) ≥ δ . Thus for λ ≥ λ∗ + 1 , we have L ( λ ) ≥ δ ( λ − λ∗ − 1 ) . So limλ→∞ L ( λ ) = ∞ . However , inspection of L shows limλ→∞ L ( λ ) = ∑N n=1 y 2 n < ∞ , which is a contradiction . We say that the result covers essentially all cases of interest : if L continues to decrease as λ → ∞ , then there is so little signal in the data that the zero model θ = 0 ∈ RD is the optimal predictor according to LOOCV . Although L is generally not convex , L ( λ ) might still be easy to optimize if it satisfies an appropriate generalized notion of convexity . To that end , we recall the definition of quasiconvexity . Definition 1 . A function f : Rp → R is quasiconvex if its level sets are convex . In one dimension ( i.e . p = 1 in Definition 1 ) , quasiconvexity guarantees that any local optimum is a global optimum , just as convexity does . This property is often the key consideration in practical optimization algorithms . Moreover , it is not unreasonable to hope that the CV loss is quasiconvex : typical illustrations of the CV loss are not convex but are quasiconvex ; see e.g . Hastie et al . [ 2015 , Fig . 5.9 ] , Rad and Maleki [ 2020 , Fig . 1 ] , or the left panel of Fig . 1 . Illustrations of the out-of-sample loss are also typically quasiconvex ; see e.g . Fig . 3.6 of Bishop [ 2006 ] . Unfortunately , we next demonstrate that the CV loss derived from real data analysis problems can be non-quasiconvex . Our first dataset contains N = 2,938 observations of life expectancy , along with D = 20 covariates such as country of origin or alcohol use ; see ? ? for a full description . In this case , after pre-processing according to Condition 1 , L for the full dataset is quasiconvex . But now consider some additional standard data pre-processing . Practitioners often perform principal component regression ( PCR ) with the aim of reducing noise in the estimated θ . That is , they take the singular value decomposition of X = USV and produce an N × R dimensional covariate matrix X ′ by retaining just the top R singular values of X : X ′ = U· , :RS : R. With this pre-processing , the resulting LOOCV curve L is non-quasiconvex for many values of R ; in the center panel of Fig . 1 we show one example for R = 15 . Our second dataset consists of recorded wine quality of N = 1,599 red wines . The goal is to predict wine quality from D = 11 observed covariates relating to the chemical properties of each wine ; see ? ? for a full description . We find that subsets of this dataset often exhibit non-quasiconvex L. In the right panel of Fig . 1 , we show L for a subset of size N = 50 . We see that this plot contains at least two local optima , with substantially different values of λ and substantially different values of the loss . A gradient-based algorithm initialized sufficiently far to the left would not find the global optimum , and grid search without sufficiently large values would not find the global optimum . Now we know that L can be non-quasiconvex for real data . Given the difficulty of optimizing a function with several local minima , we next seek to understand when L is quasiconvex or not .

Summary: This paper demonstrates that leave-one-out cross-validation (LOOCV) risk estimate for ridge regression can have multiple local minima (both in real data and in simulations). It then goes on to prove that under some restrictive set of conditions the LOOCV loss is quasiconvex and only has one local=global minimum.  The paper is easy to follow and fun to read. The fact that LOOCV curve can have local minima was pretty surprising to me (but I may be unaware of some relevant literature). Not sure it's a perfect fit specifically for NeurIPS, but I vote for acceptance.