Can we globally optimize cross-validation loss? Quasiconvexity in ridge regression

1 Introduction . Linear models , including LASSO and ridge regression , are widely used for data analysis across diverse applied disciplines . Linear models are often preferred since they are straightforward to apply in various senses . In particular , ( 1 ) their parameters are readily interpretable . ( 2 ) They have strong theoretical guarantees on quality . And ( 3 ) standard optimization tools are often assumed to find useful parameter and hyperparameter values . Despite their seeming simplicity , though , mysteries remain about the quality of inference in linear models . Consider cross-validation ( CV ) [ Stone , 1974 , Allen , 1974 ] , the de facto standard for hyperparameter selection across machine learning methods [ Musgrave et al. , 2020 ] . CV is an easy-to-evaluate proxy for the true out-of-sample loss . Is it a good proxy ? [ Homrighausen and McDonald , 2014 , 2013 , Chetverikov et al. , 2020 , Hastie et al. , 2020 , Patil et al. , 2021 ] give conditions under which the global minimum of the CV loss ( possibly with some mild corrections ) matches the optimum of the out-of-sample loss in LASSO and ridge regression . To complete the picture , we must understand whether standard methods for minimizing the CV loss find a global minimum . It would be easy to find a unique minimum of the CV loss if the CV loss were convex . Alas ( though perhaps unsurprisingly ) , we show below that in essentially every case of interest the CV loss is not convex . Indeed , the usual introductory cartoon of CV loss ( left panel of Fig . 1 ; see also Fig . 5.9 of Hastie et al . [ 2017 ] or Fig . 1 of Rad and Maleki [ 2020 ] ) is not convex . But the cartoon CV loss still exhibits a single global minimum and would be easy to globally minimize with popular approaches like gradient-based methods [ Do et al. , 2007 , Maclaurin et al. , 2015 , Pedregosa , 2016 , Lorraine et al. , 2020 ] or grid search [ Bergstra and Bengio , 2012 , Pedregosa et al. , 2011 , Hsu et al. , 2003 ] . Indeed , a ∗Alternate email : wtstephe @ gmail.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . more plausible possibility ( which holds for the typical cartoon CV loss ) is that the CV loss might be quasiconvex ; i.e . its level sets are convex . The benefit of quasiconvexity is that , in one dimension , any local optimum is a global optimum . Unfortunately , this cartoon need not hold in general , even in simple models like ℓ2-regularized linear regression ( i.e . ridge regression ) . Consider minimizing leave-one-out CV ( LOOCV ) loss as a function of the ridge regularization parameter ; we denote this loss by L. Wilson et al . [ 2020 , Fig . 1 ] detail a simulated example in which L can be non-quasiconvex . We first demonstrate that L can be non-quasiconvex in real-data examples ; see the middle and right panel of Fig . 1 , which we describe in detail in Section 3 . We next characterize which aspects of the covariate matrix and observed responses affect quasiconvexity . We prove that the norm of the responses , the scale of the singular values of the covariate matrix , and the right singular vectors of the covariate matrix all have no effect on the quasiconvexity of L. While this result places substantial constraints on what drives the quasiconvexity of L , we show that the quasiconvexity of L is unfortunately still a complex function of the remaining quantities . Our third contribution is to prove conditions under which L is guaranteed to be quasiconvex . In particular , we show that if ( 1 ) the covariate matrix has a singular value spectrum sufficiently close to uniform , ( 2 ) the least-squares estimator fits the training data sufficiently well , and ( 3 ) the left singular vectors of the covariate matrix are sufficiently regular , then L is guaranteed to be quasiconvex . While the conditions of our theory are deterministic , we show that they have natural probabilistic interpretations ; as a corollary to our theory , we demonstrate that many of our conditions are satisfied either empirically or theoretically by well-specified linear regression problems with i.i.d . sub-Gaussian covariates and moderate signal-to-noise ratios . Through empirical studies , we validate the conclusions of our theory and the necessity of our assumptions . 2 Setup and notation . For n ∈ { 1 , . . . , N } , we observe covariates xn ∈ RD and responses yn ∈ R. We are interested in learning a linear model between the covariates and responses , ⟨xn , θ⟩ ≈ yn , for some parameter θ ∈ RD . In ridge regression , i.e . ℓ2-regularized linear regression , we take some λ > 0 and estimate : θ̂ ( λ ) : = argmin θ∈RD N∑ n=1 ( ⟨xn , θ⟩ − yn ) 2 + λ 2 ∥θ∥22 . ( 1 ) The regularization parameter λ is typically chosen by minimizing the cross-validation ( CV ) loss . Here we study the leave-one-out CV ( LOOCV ) loss : L ( λ ) : = N∑ n=1 ( ⟨xn , θ̂\n ( λ ) ⟩ − yn ) 2 , ( 2 ) where θ̂\n ( λ ) is the solution to Eq . ( 1 ) with the nth datapoint left out . Let the covariate matrix X ∈ RN×D be the matrix with rows xn , and let the vector Y ∈ RN be the vector with entries yn . We consider the low to modest-dimensional case where D < N and assume the covariate matrix X is full-rank . We further assume X and Y have undergone standard data pre-processing , as described next . Condition 1 . Y is zero-mean , and X has zero-mean , unit variance columns . Equivalently , where 1 ∈ RN is the vector of all ones , 1TY = 0 and XT1 = 0 ∈ RD and for all d = 1 , . . . , D , ∑N n=1 x 2 nd = N . Preprocessing X and Y to satisfy Condition 1 represents standard best practice for ridge regression . First , using an unregularized bias parameter in Eq . ( 1 ) and setting Y to be zero-mean are equivalent ; we choose to make Y zero-mean , as it simplifies our analysis below . The conditions on the covariate matrix X are important to ensure the use of ℓ2-regularization is sensible . In particular , Eq . ( 1 ) penalizes all coordinates of θ equally . If e.g . some columns of X are measured in different scales or are centered differently , this uniform penalty will be inappropriate . 3 LOOCV loss is typically not convex and need not be quasiconvex . If the LOOCV loss L were convex or quasiconvex in λ , then any local minimum of L would be a global minimum , and we could trust gradient-based optimization methods or grid search methods to return a value near a global minimum . We next see that unfortunately L is typically not convex and is often not even quasiconvex . First we show that , in essentially all cases of interest , L is not convex . Proposition 1 . If λ = ∞ is not a minimum of L , then L is not a convex function . Proof . For the sake of contradiction , assume L is convex and λ = ∞ is not a minimum of L. This implies that there is some maximal λ∗ < ∞ such that L′ ( λ∗ ) = 0 . Let δ : = L′ ( λ∗ + 1 ) . By convexity , L′′ ≥ 0 , so we know that δ > 0 and that for λ ≥ λ∗ + 1 , we have L′ ( λ ) ≥ δ . Thus for λ ≥ λ∗ + 1 , we have L ( λ ) ≥ δ ( λ − λ∗ − 1 ) . So limλ→∞ L ( λ ) = ∞ . However , inspection of L shows limλ→∞ L ( λ ) = ∑N n=1 y 2 n < ∞ , which is a contradiction . We say that the result covers essentially all cases of interest : if L continues to decrease as λ → ∞ , then there is so little signal in the data that the zero model θ = 0 ∈ RD is the optimal predictor according to LOOCV . Although L is generally not convex , L ( λ ) might still be easy to optimize if it satisfies an appropriate generalized notion of convexity . To that end , we recall the definition of quasiconvexity . Definition 1 . A function f : Rp → R is quasiconvex if its level sets are convex . In one dimension ( i.e . p = 1 in Definition 1 ) , quasiconvexity guarantees that any local optimum is a global optimum , just as convexity does . This property is often the key consideration in practical optimization algorithms . Moreover , it is not unreasonable to hope that the CV loss is quasiconvex : typical illustrations of the CV loss are not convex but are quasiconvex ; see e.g . Hastie et al . [ 2015 , Fig . 5.9 ] , Rad and Maleki [ 2020 , Fig . 1 ] , or the left panel of Fig . 1 . Illustrations of the out-of-sample loss are also typically quasiconvex ; see e.g . Fig . 3.6 of Bishop [ 2006 ] . Unfortunately , we next demonstrate that the CV loss derived from real data analysis problems can be non-quasiconvex . Our first dataset contains N = 2,938 observations of life expectancy , along with D = 20 covariates such as country of origin or alcohol use ; see ? ? for a full description . In this case , after pre-processing according to Condition 1 , L for the full dataset is quasiconvex . But now consider some additional standard data pre-processing . Practitioners often perform principal component regression ( PCR ) with the aim of reducing noise in the estimated θ . That is , they take the singular value decomposition of X = USV and produce an N × R dimensional covariate matrix X ′ by retaining just the top R singular values of X : X ′ = U· , :RS : R. With this pre-processing , the resulting LOOCV curve L is non-quasiconvex for many values of R ; in the center panel of Fig . 1 we show one example for R = 15 . Our second dataset consists of recorded wine quality of N = 1,599 red wines . The goal is to predict wine quality from D = 11 observed covariates relating to the chemical properties of each wine ; see ? ? for a full description . We find that subsets of this dataset often exhibit non-quasiconvex L. In the right panel of Fig . 1 , we show L for a subset of size N = 50 . We see that this plot contains at least two local optima , with substantially different values of λ and substantially different values of the loss . A gradient-based algorithm initialized sufficiently far to the left would not find the global optimum , and grid search without sufficiently large values would not find the global optimum . Now we know that L can be non-quasiconvex for real data . Given the difficulty of optimizing a function with several local minima , we next seek to understand when L is quasiconvex or not .

Summary: ### Update  I have read the author response; I'd like to thank the authors for the detailed comments and clarifications.  - I am convinced from the authors' response that all of the theoretical issues I raised in my review can be fixed/clarified. Indeed, the response above does exactly this and need only be integrated into the main paper.  - I like the proposed experiments illustrating that GD on the LOO-CV loss can get stuck in local minima. This will be a nice demonstration of why the theoretical results matter in practice --- especially if the resulting test accuracies are quite different between the local min and the global solution.  In light of the response, I have decided to increase my score to a 6.  ==========  The authors conduct a theoretical and empirical investigation of the _quasi-convexity_ of the leave-one-out cross validation (CV) metric for $\ell_2$-regularized least-squares regression (aka ridge-regression). The main theoretical results are: i) counter-examples for general (quasi-)convexity of the CV metric, ii) an enumeration of dataset properties which do not effect quasi-convexity (scaling of the singular values of $X$, scaling of the targets $Y$, and the right singular vectors), and iii) sufficient conditions for quasi-convexity to hold.  The empirical component of the submission validates the sufficient conditions for quasi-convexity --- thus confirming the theoretical developments --- and examines sensitivity of the main result to their violation.  Interestingly, these experiments show that quasi-convexity is highly sensitive to the singular-value spectrum, but relatively robust to irregularity in the residuals and left singular vectors.