Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

1 INTRODUCTION . Regularization hyperparameters such as weight decay , data augmentation , and dropout ( Srivastava et al. , 2014 ) are crucial to the generalization of neural networks , but are difficult to tune . Popular approaches to hyperparameter optimization include grid search , random search ( Bergstra & Bengio , 2012 ) , and Bayesian optimization ( Snoek et al. , 2012 ) . These approaches work well with low-dimensional hyperparameter spaces and ample computational resources ; however , they pose hyperparameter optimization as a black-box optimization problem , ignoring structure which can be exploited for faster convergence , and require many training runs . We can formulate hyperparameter optimization as a bilevel optimization problem . Let w denote parameters ( e.g . weights and biases ) and λ denote hyperparameters ( e.g . dropout probability ) . Let LT and LV be functions mapping parameters and hyperparameters to training and validation losses , respectively . We aim to solve1 : λ∗ = argmin λ LV ( λ , w∗ ) subject to w∗ = argmin w LT ( λ , w ) ( 1 ) ∗Equal contribution . 1The uniqueness of the argmin is assumed . Substituting the best-response function w∗ ( λ ) = argminw LT ( λ , w ) gives a single-level problem : λ∗ = argmin λ LV ( λ , w∗ ( λ ) ) ( 2 ) If the best-response w∗ is known , the validation loss can be minimized directly by gradient descent using Equation 2 , offering dramatic speed-ups over black-box methods . However , as the solution to a high-dimensional optimization problem , it is difficult to compute w∗ even approximately . Following Lorraine & Duvenaud ( 2018 ) , we propose to approximate the best-response w∗ directly with a parametric function ŵφ . We jointly optimize φ and λ , first updating φ so that ŵφ ≈ w∗ in a neighborhood around the current hyperparameters , then updating λ by using ŵφ as a proxy for w∗ in Eq . 2 : λ∗ ≈ argmin λ LV ( λ , ŵφ ( λ ) ) ( 3 ) Finding a scalable approximation ŵφ when w represents the weights of a neural network is a significant challenge , as even simple implementations entail significant memory overhead . We show how to construct a compact approximation by modelling the best-response of each row in a layer ’ s weight matrix/bias as a rank-one affine transformation of the hyperparameters . We show that this can be interpreted as computing the activations of a base network in the usual fashion , plus a correction term dependent on the hyperparameters . We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian follows a similar structure . We call our proposed networks Self-Tuning Networks ( STNs ) since they update their own hyperparameters online during training . STNs enjoy many advantages over other hyperparameter optimization methods . First , they are easy to implement by replacing existing modules in deep learning libraries with “ hyper ” counterparts which accept an additional vector of hyperparameters as input2 . Second , because the hyperparameters are adapted online , we ensure that computational effort expended to fit φ around previous hyperparameters is not wasted . In addition , this online adaption yields hyperparameter schedules which we find empirically to outperform fixed hyperparameter settings . Finally , the STN training algorithm does not require differentiating the training loss with respect to the hyperparameters , unlike other gradient-based approaches ( Maclaurin et al. , 2015 ; Larsen et al. , 1996 ) , allowing us to tune discrete hyperparameters , such as the number of holes to cut out of an image ( DeVries & Taylor , 2017 ) , data-augmentation hyperparameters , and discrete-noise dropout parameters . Empirically , we evaluate the performance of STNs on large-scale deep-learning problems with the Penn Treebank ( Marcus et al. , 1993 ) and CIFAR-10 datasets ( Krizhevsky & Hinton , 2009 ) , and find that they substantially outperform baseline methods . 2 BILEVEL OPTIMIZATION . A bilevel optimization problem consists of two sub-problems called the upper-level and lower-level problems , where the upper-level problem must be solved subject to optimality of the lower-level problem . Minimax problems are an example of bilevel programs where the upper-level objective equals the negative lower-level objective . Bilevel programs were first studied in economics to model leader/follower firm dynamics ( Von Stackelberg , 2010 ) and have since found uses in various fields ( see Colson et al . ( 2007 ) for an overview ) . In machine learning , many problems can be formulated as bilevel programs , including hyperparameter optimization , GAN training ( Goodfellow et al. , 2014 ) , meta-learning , and neural architecture search ( Zoph & Le , 2016 ) . Even if all objectives and constraints are linear , bilevel problems are strongly NP-hard ( Hansen et al. , 1992 ; Vicente et al. , 1994 ) . Due to the difficulty of obtaining exact solutions , most work has focused on restricted settings , considering linear , quadratic , and convex functions . In contrast , we focus on obtaining local solutions in the nonconvex , differentiable , and unconstrained setting . Let F , f : Rn × Rm → R denote the upper- and lower-level objectives ( e.g. , LV and LT ) and λ ∈ Rn , w ∈ Rm denote the upper- and lower-level parameters . We aim to solve : min λ∈Rn F ( λ , w ) ( 4a ) subject to w ∈ argmin w∈Rm f ( λ , w ) ( 4b ) 2We illustrate how this is done for the PyTorch library ( Paszke et al. , 2017 ) in Appendix G. It is desirable to design a gradient-based algorithm for solving Problem 4 , since using gradient information provides drastic speed-ups over black-box optimization methods ( Nesterov , 2013 ) . The simplest method is simultaneous gradient descent , which updates λ using ∂F/∂λ and w using ∂f/∂w . However , simultaneous gradient descent often gives incorrect solutions as it fails to account for the dependence of w on λ . Consider the relatively common situation where F doesn ’ t depend directly on λ , so that ∂F/∂λ ≡ 0 and hence λ is never updated . 2.1 GRADIENT DESCENT VIA THE BEST-RESPONSE FUNCTION . A more principled approach to solving Problem 4 is to use the best-response function ( Gibbons , 1992 ) . Assume the lower-level Problem 4b has a unique optimum w∗ ( λ ) for each λ . Substituting the best-response function w∗ converts Problem 4 into a single-level problem : min λ∈Rn F ∗ ( λ ) : = F ( λ , w∗ ( λ ) ) ( 5 ) If w∗ is differentiable , we can minimize Eq . 5 using gradient descent on F ∗ with respect to λ . This method requires a unique optimum w∗ ( λ ) for Problem 4b for each λ and differentiability of w∗ . In general , these conditions are difficult to verify . We give sufficient conditions for them to hold in a neighborhood of a point ( λ0 , w0 ) where w0 solves Problem 4b given λ0 . Lemma 1 . ( Fiacco & Ishizuka , 1990 ) Let w0 solve Problem 4b for λ0 . Suppose f is C2 in a neighborhood of ( λ0 , w0 ) and the Hessian ∂ 2f/∂w2 ( λ0 , w0 ) is positive definite . Then for some neighborhood U of λ0 , there exists a continuously differentiable function w∗ : U → Rm such that w∗ ( λ ) is the unique solution to Problem 4b for each λ ∈ U and w∗ ( λ0 ) = w0 . Proof . See Appendix B.1 . The gradient of F ∗ decomposes into two terms , which we term the direct gradient and the response gradient . The direct gradient captures the direct reliance of the upper-level objective on λ , while the response gradient captures how the lower-level parameter responds to changes in the upper-level parameter : ∂F ∗ ∂λ ( λ0 ) = ∂F ∂λ ( λ0 , w∗ ( λ0 ) ) ︸ ︷︷ ︸ Direct gradient + ∂F ∂w ( λ0 , w∗ ( λ0 ) ) ∂w∗ ∂λ ( λ0 ) ︸ ︷︷ ︸ Response gradient ( 6 ) Even if ∂F/∂λ 6≡ 0 and simultaneous gradient descent is possible , including the response gradient can stabilize optimization by converting the bilevel problem into a single-level one , as noted by Metz et al . ( 2016 ) for GAN optimization . Conversion to a single-level problem ensures that the gradient vector field is conservative , avoiding pathological issues described by Mescheder et al . ( 2017 ) . 2.2 APPROXIMATING THE BEST-RESPONSE FUNCTION . In general , the solution to Problem 4b is a set , but assuming uniqueness of a solution and differentiability of w∗ can yield fruitful algorithms in practice . In fact , gradient-based hyperparameter optimization methods can often be interpreted as approximating either the best-response w∗ or its Jacobian ∂w∗/∂λ , as detailed in Section 5 . However , these approaches can be computationally expensive and often struggle with discrete hyperparameters and stochastic hyperparameters like dropout probabilities , since they require differentiating the training loss with respect to the hyperparameters . Promising approaches to approximate w∗ directly were proposed by Lorraine & Duvenaud ( 2018 ) , and are detailed below . 1 . Global Approximation . The first algorithm proposed by Lorraine & Duvenaud ( 2018 ) approximates w∗ as a differentiable function ŵφ with parameters φ . If w represents neural net weights , then the mapping ŵφ is a hypernetwork ( Schmidhuber , 1992 ; Ha et al. , 2016 ) . If the distribution p ( λ ) is fixed , then gradient descent with respect to φ minimizes : Eλ∼p ( λ ) [ f ( λ , ŵφ ( λ ) ) ] ( 7 ) If support ( p ) is broad and ŵφ is sufficiently flexible , then ŵφ can be used as a proxy for w∗ in Problem 5 , resulting in the following objective : min λ∈Rn F ( λ , ŵφ ( λ ) ) ( 8 ) 2 . Local Approximation . In practice , ŵφ is usually insufficiently flexible to model w∗ on support ( p ) . The second algorithm of Lorraine & Duvenaud ( 2018 ) locally approximates w∗ in a neighborhood around the current upper-level parameter λ . They set p ( |σ ) to a factorized Gaussian noise distribution with a fixed scale parameter σ ∈ Rn+ , and found φ by minimizing the objective : E ∼p ( |σ ) [ f ( λ+ , ŵφ ( λ+ ) ) ] ( 9 ) Intuitively , the upper-level parameter λ is perturbed by a small amount , so the lower-level parameter learns how to respond . An alternating gradient descent scheme is used , where φ is updated to minimize equation 9 and λ is updated to minimize equation 8 . This approach worked for problems usingL2 regularization on MNIST ( LeCun et al. , 1998 ) . However , it is unclear if the approach works with different regularizers or scales to larger problems . It requires ŵφ , which is a priori unwieldy for high dimensional w. It is also unclear how to set σ , which defines the size of the neighborhood on which φ is trained , or if the approach can be adapted to discrete and stochastic hyperparameters . 3 SELF-TUNING NETWORKS . In this section , we first construct a best-response approximation ŵφ that is memory efficient and scales to large neural networks . We justify this approximation through analysis of simpler situations . Then , we describe a method to automatically adjust the scale of the neighborhood φ is trained on . Finally , we formally describe our algorithm and discuss how it easily handles discrete and stochastic hyperparameters . We call the resulting networks , which update their own hyperparameters online during training , Self-Tuning Networks ( STNs ) .

Summary: The paper deals with hyper-parameter optimization of neural networks. The authors formulate the problem as a bilevel optimization problem: minimizing the validation loss over the hyperparameters, subject to the parameters being at the minimum of the training loss. The authors propose an approximation of the so-called best-response function, that maps the hyperparameters to the corresponding optimal parameters (w.r.t the minimization of the training loss), allowing a formulate as a single-level optimization problem and the use gradient descent algorithm. The proposed