Optimizing for Generalization in Machine Learning with Cross-Validation Gradients

1 INTRODUCTION . The ultimate aim of a supervised learning method is generalization , that is , achieving good prediction ability on unseen test data given only a finite set of training data . The generalization capability of learning algorithms should be the primary criterion for model selection , yet an algorithm ’ s generalization capability is a somewhat elusive quantity that is challenging to optimize for . In this paper we introduce a method to optimize directly for the closest available proxy to generalization performance : cross-validation loss . We begin with a formal description of the overall goal in predictive learning , which also serves as an introduction to notation used throughout the paper . The task of predictive learning involves deriving a prediction function from a finite set of training data . More formally , suppose that ( x , y ) ∈ X × Y have some joint probability distribution . We have access to a finite dataset of N training examples zi ∈ Z = X × Y drawn i.i.d . from the joint distribution , denoted S = { z1 , z2 , . . . , zN } . We are given ( or specify ourselves ) a cost function c ( ŷ , y ) : Y × Y → R+ that quantifies the displeasure incurred when ŷ is predicted instead of y. Denoting the function space from input to outputs as F = YX , we define the loss of a function on a training example z = ( x , y ) as l ( f , z ) = c ( f ( x ) , y ) . Then , given a prediction function f ∈ F , we define the population risk as E z [ l ( f , z ) ] , and the target function f∗ ∈ F as the function that minimizes the population risk . The population risk represents how much loss we incur , on average , on the full joint distribution , and is the quantity we would like as small as possible . In this paper , we consider parametric prediction functions , that is , f is parameterized by a vector θ , denoted f ( x ; θ ) 1 . For example , in linear regression , X = Rn , Y = R , c ( ŷ , y ) = ( ŷ − y ) 2 , and F is the set of all affine functions parameterized as f ( x ; θ ) = θTx+ θ0 . We are then tasked with designing a learning algorithmA : ZN → F , which is a function that maps a dataset S to a prediction function . Without substantial knowledge of the actual joint distribution , or assumptions about the target function f∗ , it is extremely unlikely that A will ever reproduce the 1Nonparametric learning algorithms exist , e.g. , k-nearest neighbor , but are challenging to analyze with our method . exact target function . However , our goal is to minimize the population risk of the learning algorithm R ( A , S ) = E z [ l ( A ( S ) , z ) ] ( 1 ) which is a random variable that depends on S , our dataset . To make this problem of searching for learning algorithms tractable , we similarly parameterize our learning algorithm A by a vector α ∈ Rd , denoted Aα . These are known as the “ hyperparameters ” or “ meta-parameters ” of the learning algorithm , and can play many important roles : they can perform regularization , enforce sparsity , or even guide feature selection ( Hastie et al. , 2001 ) . The quantity we would then like to optimize is the expected population risk , or L ( α ) = E S [ R ( Aα , S ) ] . ( 2 ) It is impossible to exactly calculate ( 2 ) with a finite dataset S , as there are two expectations that both involve an unknown probability distribution . What we can do is construct a Monte Carlo estimate of the an algorithm ’ s expected population risk using a technique known as cross-validation . We first partition S into K partitions Tj , Vj , j = 1 , . . . , K ( that is , Tj ∩ Vj = ∅ and Tj ∪ Vj = [ n ] ) . Then our cross-validation risk , as a function of α , is Lcv ( α ) = 1 K K∑ j=1 1 |Vj | ∑ i∈Vj l ( Aα ( Tj ) , zi ) ( 3 ) and is readily calculated . We first apply the algorithm to each training set and then average the loss on each corresponding validation set . The first sum in ( 3 ) corresponds to the expectation in ( 2 ) , and the second sum corresponds to the expectation in ( 1 ) . Setting K = 1 reduces to simple outof-sample validation and an arbitrary K reduces to the common K-fold cross-validation estimate ( provided Tj form a partition of { 1 , . . . , N } and |Tj | = N − NK ) . Thus , this formulation can be viewed as a generalization of cross-validation . ( See Kohavi & John ( 1995 ) for a longer discussion about this general framework . ) In cases where the class of models to be used for learning are known , we have reduced the predictive learning problem to the problem of selecting of a hyperparameter vector α to minimize the cross-validation loss . Even in the simplest cases , however , the objective in ( 3 ) is nonconvex in α , and in many cases not even continuous ( e.g. , the 0 − 1 classification loss ) , which can make optimization of this quantity tricky . 1.1 SUMMARY OF RESULTS . Our first result is to demonstrate that we can find ∇αLcv ( α ) for many common convex machine learning algorithms ( e.g. , logistic regression , elastic-net regression , support vector machines ) , provided the cross-validation loss function is differentiable ( Section 3 ) . In those algorithms , α often plays the role of regularizer or defines a feature map ( in the case of SVM kernels ) . In the case where α is low-dimensional , ( 3 ) can be optimized by exhaustive search without incurring too much cost . However , if we want to design our machine learning algorithms with more expressive regularizations or feature maps , exhaustive search over our hyperparameter space becomes prohibitive . Our second contribution is to propose the cross-validation gradient method ( CVGM ) , which makes it possible to optimize cross-validation loss over high-dimensional hyperparameter spaces via gradient descent techniques ( Section 4 ) . We test the CVGM on an elastic-net regression problem to optimize two hyperparameters and on a more ambitious synthetic classification problem to optimize an entire neural network that serves as a kernel function . In this case , the parameters of the neural network are the hyperparameters ( Section 5 ) . 2 RELATED WORK . There have been many proposed approaches for the problem of hyperparameter optimization , which roughly fall into two camps based on whether or not they use gradients . 2.1 GRADIENT-FREE METHODS . Exhaustive Search Exhaustive search , also known as grid search , restricts the possible set of α to a ( finite ) set { α1 , . . . , αn } , usually by discretizing the parameter search space into a regular grid . Then one exhaustively computes ( 3 ) for each αi and chooses the argmin . The main disadvantage of exhaustive search is that its complexity ( to find an approximate minimum ) scales exponentially with the dimension d , making it prohibitive for practitioners to successfully apply exhaustive search to d greater than 5 or 6 . Random Search Random search for hyperparameters involves repeatedly specifying a probability distribution over Rd , sampling from it , and evaluating ( 3 ) . Quite unintuitively , random search can be more efficient than exhaustive search , even with a simple probability distribution . This is because , in practice , only a few of the hyperparameter dimensions matter ( Bergstra & Bengio , 2012 ) . Bayesian Optimization Bayesian regression allows us to predict a distribution over Lcv ( α ) , further allowing us to query α that maximize a surrogate function , e.g. , probability of improvement or expected improvement ( Močkus , 1975 ; Snoek et al. , 2012 ) . The regression is usually carried out with Gaussian processes ( GPs ) ( Rasmussen , 2004 ) . However , random search still remains a fierce competitor to the ( substantially more complicated ) Bayesian optimization approach . 2.2 GRADIENT-BASED METHODS . Implicit Differentiation Most learning algorithms Aα are solving some parameterized optimization problem , that is , optimizing some objective function . Under certain conditions , one can apply the well-known implicit function theorem ( Dontchev & Rockafellar , 2014 ) to the optimality conditions of the objective function , and calculate the gradients of the loss function . Larsen et al . ( 1998 ) were the first to propose this , in the context of neural networks , when the objective function includes a regularization term that is linear in the regularization parameters . Bengio ( 2000 ) further derived the gradients for a general ( unconstrained and differentiable ) training criterion along with an efficient way of calculating the gradient for a quadratic training criterion , and applied the algorithm to weight decays for linear regression . These results were then extended to support vector machines ( SVMs ) ( Chapelle et al. , 2002 ; Keerthi et al. , 2007 ) and applied to log-linear models ( Foo et al. , 2008 ) and ridge regression ( Pedregosa , 2016 ) . This paper seeks to generalize these methods and provide exact conditions under which Aα is actually differentiable . In short , when Aα is a convex optimization problem parameterized by α , under certain conditions that are satisfied by many common learning algorithms , we can find exact cross-validation gradients . Iterative Differentiation In addition to approaches based on implicit differentiation , there are also approaches based on iterative differentiation , i.e. , they unroll the optimization procedure in A to calculate gradients . Many large-scale machine learning algorithms perform a variation of gradient descent , and since gradient descent is a sequence of analytic updates to the parameters , Aα can be unrolled ( or “ reverse-mode ” differentiated ) with respect to α by recursively applying the chain rule to the updates in backwards order . Domke ( 2012 ) was the first to propose this , deriving backpropagation rules for the heavy-ball method and LBFGS . Since most large-scale machine learning problems in practice are ( approximately ) solved using variations of the stochastic subgradient method ( also known as SGD ) , the “ learning rate ” parameter has a large impact on the convergence and training speed of nonconvex models , e.g. , neural networks . Maclaurin et al . ( 2015 ) extended the results of Domke to the case of stochastic gradient methods , and as a result , the authors were able to update the learning rate throughout the learning process . The advantage of these methods are that they can be applied to any large-scale machine learning problem that uses a stochastic subgradient method . The main limitations of these methods , however , are that the use of finite precision arithmetic when recursively applying the chain rule can lead to inaccuracies in the gradient calculation and that one can encounter exploding or vanishing gradients from repeated application of the chain rule .

Summary: The paper considers the problem of automatic tuning of hyperparameters in machine learning models. To address this problem the authors propose to use the so called cross-validation gradients, which optimize a validation objective with respect to the hyperparameters of a model. This approach and the investigated setting falls into a class of optimization problems known as bilevel optimization. The main characteristic of this class of optimization problems is the nested structure, with an outer and inner optimization objectives/problems. The outer problem corresponds to the validation objective and it is defined via an optimal solution to the inner problem which corresponds to a training objective. The paper, however, fails to make a reference to a rather rich literature on bilevel optimization (e.g., see [3-5] and references therein).