Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

1 INTRODUCTION . Regularization hyperparameters such as weight decay , data augmentation , and dropout ( Srivastava et al. , 2014 ) are crucial to the generalization of neural networks , but are difficult to tune . Popular approaches to hyperparameter optimization include grid search , random search ( Bergstra & Bengio , 2012 ) , and Bayesian optimization ( Snoek et al. , 2012 ) . These approaches work well with low-dimensional hyperparameter spaces and ample computational resources ; however , they pose hyperparameter optimization as a black-box optimization problem , ignoring structure which can be exploited for faster convergence , and require many training runs . We can formulate hyperparameter optimization as a bilevel optimization problem . Let w denote parameters ( e.g . weights and biases ) and λ denote hyperparameters ( e.g . dropout probability ) . Let LT and LV be functions mapping parameters and hyperparameters to training and validation losses , respectively . We aim to solve1 : λ∗ = argmin λ LV ( λ , w∗ ) subject to w∗ = argmin w LT ( λ , w ) ( 1 ) ∗Equal contribution . 1The uniqueness of the argmin is assumed . Substituting the best-response function w∗ ( λ ) = argminw LT ( λ , w ) gives a single-level problem : λ∗ = argmin λ LV ( λ , w∗ ( λ ) ) ( 2 ) If the best-response w∗ is known , the validation loss can be minimized directly by gradient descent using Equation 2 , offering dramatic speed-ups over black-box methods . However , as the solution to a high-dimensional optimization problem , it is difficult to compute w∗ even approximately . Following Lorraine & Duvenaud ( 2018 ) , we propose to approximate the best-response w∗ directly with a parametric function ŵφ . We jointly optimize φ and λ , first updating φ so that ŵφ ≈ w∗ in a neighborhood around the current hyperparameters , then updating λ by using ŵφ as a proxy for w∗ in Eq . 2 : λ∗ ≈ argmin λ LV ( λ , ŵφ ( λ ) ) ( 3 ) Finding a scalable approximation ŵφ when w represents the weights of a neural network is a significant challenge , as even simple implementations entail significant memory overhead . We show how to construct a compact approximation by modelling the best-response of each row in a layer ’ s weight matrix/bias as a rank-one affine transformation of the hyperparameters . We show that this can be interpreted as computing the activations of a base network in the usual fashion , plus a correction term dependent on the hyperparameters . We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian follows a similar structure . We call our proposed networks Self-Tuning Networks ( STNs ) since they update their own hyperparameters online during training . STNs enjoy many advantages over other hyperparameter optimization methods . First , they are easy to implement by replacing existing modules in deep learning libraries with “ hyper ” counterparts which accept an additional vector of hyperparameters as input2 . Second , because the hyperparameters are adapted online , we ensure that computational effort expended to fit φ around previous hyperparameters is not wasted . In addition , this online adaption yields hyperparameter schedules which we find empirically to outperform fixed hyperparameter settings . Finally , the STN training algorithm does not require differentiating the training loss with respect to the hyperparameters , unlike other gradient-based approaches ( Maclaurin et al. , 2015 ; Larsen et al. , 1996 ) , allowing us to tune discrete hyperparameters , such as the number of holes to cut out of an image ( DeVries & Taylor , 2017 ) , data-augmentation hyperparameters , and discrete-noise dropout parameters . Empirically , we evaluate the performance of STNs on large-scale deep-learning problems with the Penn Treebank ( Marcus et al. , 1993 ) and CIFAR-10 datasets ( Krizhevsky & Hinton , 2009 ) , and find that they substantially outperform baseline methods . 2 BILEVEL OPTIMIZATION . A bilevel optimization problem consists of two sub-problems called the upper-level and lower-level problems , where the upper-level problem must be solved subject to optimality of the lower-level problem . Minimax problems are an example of bilevel programs where the upper-level objective equals the negative lower-level objective . Bilevel programs were first studied in economics to model leader/follower firm dynamics ( Von Stackelberg , 2010 ) and have since found uses in various fields ( see Colson et al . ( 2007 ) for an overview ) . In machine learning , many problems can be formulated as bilevel programs , including hyperparameter optimization , GAN training ( Goodfellow et al. , 2014 ) , meta-learning , and neural architecture search ( Zoph & Le , 2016 ) . Even if all objectives and constraints are linear , bilevel problems are strongly NP-hard ( Hansen et al. , 1992 ; Vicente et al. , 1994 ) . Due to the difficulty of obtaining exact solutions , most work has focused on restricted settings , considering linear , quadratic , and convex functions . In contrast , we focus on obtaining local solutions in the nonconvex , differentiable , and unconstrained setting . Let F , f : Rn × Rm → R denote the upper- and lower-level objectives ( e.g. , LV and LT ) and λ ∈ Rn , w ∈ Rm denote the upper- and lower-level parameters . We aim to solve : min λ∈Rn F ( λ , w ) ( 4a ) subject to w ∈ argmin w∈Rm f ( λ , w ) ( 4b ) 2We illustrate how this is done for the PyTorch library ( Paszke et al. , 2017 ) in Appendix G. It is desirable to design a gradient-based algorithm for solving Problem 4 , since using gradient information provides drastic speed-ups over black-box optimization methods ( Nesterov , 2013 ) . The simplest method is simultaneous gradient descent , which updates λ using ∂F/∂λ and w using ∂f/∂w . However , simultaneous gradient descent often gives incorrect solutions as it fails to account for the dependence of w on λ . Consider the relatively common situation where F doesn ’ t depend directly on λ , so that ∂F/∂λ ≡ 0 and hence λ is never updated . 2.1 GRADIENT DESCENT VIA THE BEST-RESPONSE FUNCTION . A more principled approach to solving Problem 4 is to use the best-response function ( Gibbons , 1992 ) . Assume the lower-level Problem 4b has a unique optimum w∗ ( λ ) for each λ . Substituting the best-response function w∗ converts Problem 4 into a single-level problem : min λ∈Rn F ∗ ( λ ) : = F ( λ , w∗ ( λ ) ) ( 5 ) If w∗ is differentiable , we can minimize Eq . 5 using gradient descent on F ∗ with respect to λ . This method requires a unique optimum w∗ ( λ ) for Problem 4b for each λ and differentiability of w∗ . In general , these conditions are difficult to verify . We give sufficient conditions for them to hold in a neighborhood of a point ( λ0 , w0 ) where w0 solves Problem 4b given λ0 . Lemma 1 . ( Fiacco & Ishizuka , 1990 ) Let w0 solve Problem 4b for λ0 . Suppose f is C2 in a neighborhood of ( λ0 , w0 ) and the Hessian ∂ 2f/∂w2 ( λ0 , w0 ) is positive definite . Then for some neighborhood U of λ0 , there exists a continuously differentiable function w∗ : U → Rm such that w∗ ( λ ) is the unique solution to Problem 4b for each λ ∈ U and w∗ ( λ0 ) = w0 . Proof . See Appendix B.1 . The gradient of F ∗ decomposes into two terms , which we term the direct gradient and the response gradient . The direct gradient captures the direct reliance of the upper-level objective on λ , while the response gradient captures how the lower-level parameter responds to changes in the upper-level parameter : ∂F ∗ ∂λ ( λ0 ) = ∂F ∂λ ( λ0 , w∗ ( λ0 ) ) ︸ ︷︷ ︸ Direct gradient + ∂F ∂w ( λ0 , w∗ ( λ0 ) ) ∂w∗ ∂λ ( λ0 ) ︸ ︷︷ ︸ Response gradient ( 6 ) Even if ∂F/∂λ 6≡ 0 and simultaneous gradient descent is possible , including the response gradient can stabilize optimization by converting the bilevel problem into a single-level one , as noted by Metz et al . ( 2016 ) for GAN optimization . Conversion to a single-level problem ensures that the gradient vector field is conservative , avoiding pathological issues described by Mescheder et al . ( 2017 ) . 2.2 APPROXIMATING THE BEST-RESPONSE FUNCTION . In general , the solution to Problem 4b is a set , but assuming uniqueness of a solution and differentiability of w∗ can yield fruitful algorithms in practice . In fact , gradient-based hyperparameter optimization methods can often be interpreted as approximating either the best-response w∗ or its Jacobian ∂w∗/∂λ , as detailed in Section 5 . However , these approaches can be computationally expensive and often struggle with discrete hyperparameters and stochastic hyperparameters like dropout probabilities , since they require differentiating the training loss with respect to the hyperparameters . Promising approaches to approximate w∗ directly were proposed by Lorraine & Duvenaud ( 2018 ) , and are detailed below . 1 . Global Approximation . The first algorithm proposed by Lorraine & Duvenaud ( 2018 ) approximates w∗ as a differentiable function ŵφ with parameters φ . If w represents neural net weights , then the mapping ŵφ is a hypernetwork ( Schmidhuber , 1992 ; Ha et al. , 2016 ) . If the distribution p ( λ ) is fixed , then gradient descent with respect to φ minimizes : Eλ∼p ( λ ) [ f ( λ , ŵφ ( λ ) ) ] ( 7 ) If support ( p ) is broad and ŵφ is sufficiently flexible , then ŵφ can be used as a proxy for w∗ in Problem 5 , resulting in the following objective : min λ∈Rn F ( λ , ŵφ ( λ ) ) ( 8 ) 2 . Local Approximation . In practice , ŵφ is usually insufficiently flexible to model w∗ on support ( p ) . The second algorithm of Lorraine & Duvenaud ( 2018 ) locally approximates w∗ in a neighborhood around the current upper-level parameter λ . They set p ( |σ ) to a factorized Gaussian noise distribution with a fixed scale parameter σ ∈ Rn+ , and found φ by minimizing the objective : E ∼p ( |σ ) [ f ( λ+ , ŵφ ( λ+ ) ) ] ( 9 ) Intuitively , the upper-level parameter λ is perturbed by a small amount , so the lower-level parameter learns how to respond . An alternating gradient descent scheme is used , where φ is updated to minimize equation 9 and λ is updated to minimize equation 8 . This approach worked for problems usingL2 regularization on MNIST ( LeCun et al. , 1998 ) . However , it is unclear if the approach works with different regularizers or scales to larger problems . It requires ŵφ , which is a priori unwieldy for high dimensional w. It is also unclear how to set σ , which defines the size of the neighborhood on which φ is trained , or if the approach can be adapted to discrete and stochastic hyperparameters . 3 SELF-TUNING NETWORKS . In this section , we first construct a best-response approximation ŵφ that is memory efficient and scales to large neural networks . We justify this approximation through analysis of simpler situations . Then , we describe a method to automatically adjust the scale of the neighborhood φ is trained on . Finally , we formally describe our algorithm and discuss how it easily handles discrete and stochastic hyperparameters . We call the resulting networks , which update their own hyperparameters online during training , Self-Tuning Networks ( STNs ) .

Summary: The paper proposes a bilevel optimization approach for hyperparameter tuning. This idea is not new having been proposed in works prior to the current resurgence of deep learning (e.g., Do et al., 2007, Domke 2012, and Kunisch & Pock, 2013). However, the combination of bilevel optimization for hyperparameter tuning with approximation is interesting. Moreover, the proposed approach readily handles discrete parameters.