ING STRUCTURED BEST-RESPONSE FUNCTIONS
Year: 2019

Abstract:
Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, SelfTuning Networks (STNs).

heading: 1 INTRODUCTION
text: Regularization hyperparameters such as weight decay, data augmentation, and dropout (Srivastava et al., 2014) are crucial to the generalization of neural networks, but are difficult to tune. Popular approaches to hyperparameter optimization include grid search, random search (Bergstra & Bengio, 2012), and Bayesian optimization (Snoek et al., 2012). These approaches work well with low-dimensional hyperparameter spaces and ample computational resources; however, they pose hyperparameter optimization as a black-box optimization problem, ignoring structure which can be exploited for faster convergence, and require many training runs.
We can formulate hyperparameter optimization as a bilevel optimization problem. Let w denote parameters (e.g. weights and biases) and λ denote hyperparameters (e.g. dropout probability). Let LT and LV be functions mapping parameters and hyperparameters to training and validation losses, respectively. We aim to solve1:
λ∗ = argmin λ LV (λ,w∗) subject to w∗ = argmin w LT (λ,w) (1)
∗Equal contribution. 1The uniqueness of the argmin is assumed.
Substituting the best-response function w∗(λ) = argminw LT (λ,w) gives a single-level problem: λ∗ = argmin
λ LV (λ,w∗(λ)) (2)
If the best-response w∗ is known, the validation loss can be minimized directly by gradient descent using Equation 2, offering dramatic speed-ups over black-box methods. However, as the solution to a high-dimensional optimization problem, it is difficult to compute w∗ even approximately.
Following Lorraine & Duvenaud (2018), we propose to approximate the best-response w∗ directly with a parametric function ŵφ. We jointly optimize φ and λ, first updating φ so that ŵφ ≈ w∗ in a neighborhood around the current hyperparameters, then updating λ by using ŵφ as a proxy for w∗ in Eq. 2:
λ∗ ≈ argmin λ LV (λ, ŵφ(λ)) (3)
Finding a scalable approximation ŵφ when w represents the weights of a neural network is a significant challenge, as even simple implementations entail significant memory overhead. We show how to construct a compact approximation by modelling the best-response of each row in a layer’s weight matrix/bias as a rank-one affine transformation of the hyperparameters. We show that this can be interpreted as computing the activations of a base network in the usual fashion, plus a correction term dependent on the hyperparameters. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian follows a similar structure. We call our proposed networks Self-Tuning Networks (STNs) since they update their own hyperparameters online during training.
STNs enjoy many advantages over other hyperparameter optimization methods. First, they are easy to implement by replacing existing modules in deep learning libraries with “hyper” counterparts which accept an additional vector of hyperparameters as input2. Second, because the hyperparameters are adapted online, we ensure that computational effort expended to fit φ around previous hyperparameters is not wasted. In addition, this online adaption yields hyperparameter schedules which we find empirically to outperform fixed hyperparameter settings. Finally, the STN training algorithm does not require differentiating the training loss with respect to the hyperparameters, unlike other gradient-based approaches (Maclaurin et al., 2015; Larsen et al., 1996), allowing us to tune discrete hyperparameters, such as the number of holes to cut out of an image (DeVries & Taylor, 2017), data-augmentation hyperparameters, and discrete-noise dropout parameters. Empirically, we evaluate the performance of STNs on large-scale deep-learning problems with the Penn Treebank (Marcus et al., 1993) and CIFAR-10 datasets (Krizhevsky & Hinton, 2009), and find that they substantially outperform baseline methods.

heading: 2 BILEVEL OPTIMIZATION
text: A bilevel optimization problem consists of two sub-problems called the upper-level and lower-level problems, where the upper-level problem must be solved subject to optimality of the lower-level problem. Minimax problems are an example of bilevel programs where the upper-level objective equals the negative lower-level objective. Bilevel programs were first studied in economics to model leader/follower firm dynamics (Von Stackelberg, 2010) and have since found uses in various fields (see Colson et al. (2007) for an overview). In machine learning, many problems can be formulated as bilevel programs, including hyperparameter optimization, GAN training (Goodfellow et al., 2014), meta-learning, and neural architecture search (Zoph & Le, 2016).
Even if all objectives and constraints are linear, bilevel problems are strongly NP-hard (Hansen et al., 1992; Vicente et al., 1994). Due to the difficulty of obtaining exact solutions, most work has focused on restricted settings, considering linear, quadratic, and convex functions. In contrast, we focus on obtaining local solutions in the nonconvex, differentiable, and unconstrained setting. Let F, f : Rn × Rm → R denote the upper- and lower-level objectives (e.g., LV and LT ) and λ ∈ Rn,w ∈ Rm denote the upper- and lower-level parameters. We aim to solve:
min λ∈Rn
F (λ,w) (4a)
subject to w ∈ argmin w∈Rm f(λ,w) (4b)
2We illustrate how this is done for the PyTorch library (Paszke et al., 2017) in Appendix G.
It is desirable to design a gradient-based algorithm for solving Problem 4, since using gradient information provides drastic speed-ups over black-box optimization methods (Nesterov, 2013). The simplest method is simultaneous gradient descent, which updates λ using ∂F/∂λ and w using ∂f/∂w. However, simultaneous gradient descent often gives incorrect solutions as it fails to account for the dependence of w on λ. Consider the relatively common situation where F doesn’t depend directly on λ , so that ∂F/∂λ ≡ 0 and hence λ is never updated.

heading: 2.1 GRADIENT DESCENT VIA THE BEST-RESPONSE FUNCTION
text: A more principled approach to solving Problem 4 is to use the best-response function (Gibbons, 1992). Assume the lower-level Problem 4b has a unique optimum w∗(λ) for each λ. Substituting the best-response function w∗ converts Problem 4 into a single-level problem:
min λ∈Rn
F ∗(λ) := F (λ,w∗(λ)) (5)
If w∗ is differentiable, we can minimize Eq. 5 using gradient descent on F ∗ with respect to λ. This method requires a unique optimum w∗(λ) for Problem 4b for each λ and differentiability of w∗. In general, these conditions are difficult to verify. We give sufficient conditions for them to hold in a neighborhood of a point (λ0,w0) where w0 solves Problem 4b given λ0. Lemma 1. (Fiacco & Ishizuka, 1990) Let w0 solve Problem 4b for λ0. Suppose f is C2 in a neighborhood of (λ0,w0) and the Hessian ∂
2f/∂w2(λ0,w0) is positive definite. Then for some neighborhood U of λ0, there exists a continuously differentiable function w∗ : U → Rm such that w∗(λ) is the unique solution to Problem 4b for each λ ∈ U and w∗(λ0) = w0.
Proof. See Appendix B.1.
The gradient of F ∗ decomposes into two terms, which we term the direct gradient and the response gradient. The direct gradient captures the direct reliance of the upper-level objective on λ, while the response gradient captures how the lower-level parameter responds to changes in the upper-level parameter:
∂F ∗
∂λ (λ0) =
∂F ∂λ (λ0,w∗(λ0))︸ ︷︷ ︸
Direct gradient
+ ∂F
∂w (λ0,w∗(λ0))
∂w∗
∂λ (λ0)︸ ︷︷ ︸
Response gradient
(6)
Even if ∂F/∂λ 6≡ 0 and simultaneous gradient descent is possible, including the response gradient can stabilize optimization by converting the bilevel problem into a single-level one, as noted by Metz et al. (2016) for GAN optimization. Conversion to a single-level problem ensures that the gradient vector field is conservative, avoiding pathological issues described by Mescheder et al. (2017).

heading: 2.2 APPROXIMATING THE BEST-RESPONSE FUNCTION
text: In general, the solution to Problem 4b is a set, but assuming uniqueness of a solution and differentiability of w∗ can yield fruitful algorithms in practice. In fact, gradient-based hyperparameter optimization methods can often be interpreted as approximating either the best-response w∗ or its Jacobian ∂w∗/∂λ, as detailed in Section 5. However, these approaches can be computationally expensive and often struggle with discrete hyperparameters and stochastic hyperparameters like dropout probabilities, since they require differentiating the training loss with respect to the hyperparameters. Promising approaches to approximate w∗ directly were proposed by Lorraine & Duvenaud (2018), and are detailed below.
1. Global Approximation. The first algorithm proposed by Lorraine & Duvenaud (2018) approximates w∗ as a differentiable function ŵφ with parameters φ. If w represents neural net weights, then the mapping ŵφ is a hypernetwork (Schmidhuber, 1992; Ha et al., 2016). If the distribution p(λ) is fixed, then gradient descent with respect to φ minimizes:
Eλ∼p(λ) [f(λ, ŵφ(λ))] (7) If support(p) is broad and ŵφ is sufficiently flexible, then ŵφ can be used as a proxy for w∗ in Problem 5, resulting in the following objective:
min λ∈Rn F (λ, ŵφ(λ)) (8)
2. Local Approximation. In practice, ŵφ is usually insufficiently flexible to model w∗ on support(p). The second algorithm of Lorraine & Duvenaud (2018) locally approximates w∗ in a neighborhood around the current upper-level parameter λ. They set p( |σ) to a factorized Gaussian noise distribution with a fixed scale parameter σ ∈ Rn+, and found φ by minimizing the objective:
E ∼p( |σ) [f(λ+ , ŵφ(λ+ ))] (9) Intuitively, the upper-level parameter λ is perturbed by a small amount, so the lower-level parameter learns how to respond. An alternating gradient descent scheme is used, where φ is updated to minimize equation 9 and λ is updated to minimize equation 8. This approach worked for problems usingL2 regularization on MNIST (LeCun et al., 1998). However, it is unclear if the approach works with different regularizers or scales to larger problems. It requires ŵφ, which is a priori unwieldy for high dimensional w. It is also unclear how to set σ, which defines the size of the neighborhood on which φ is trained, or if the approach can be adapted to discrete and stochastic hyperparameters.

heading: 3 SELF-TUNING NETWORKS
text: In this section, we first construct a best-response approximation ŵφ that is memory efficient and scales to large neural networks. We justify this approximation through analysis of simpler situations. Then, we describe a method to automatically adjust the scale of the neighborhood φ is trained on. Finally, we formally describe our algorithm and discuss how it easily handles discrete and stochastic hyperparameters. We call the resulting networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs).

heading: 3.1 AN EFFICIENT BEST-RESPONSE APPROXIMATION FOR NEURAL NETWORKS
text: We propose to approximate the best-response for a given layer’s weight matrix W ∈ RDout×Din and bias b ∈ RDout as an affine transformation of the hyperparameters λ3:
Ŵφ(λ) =Welem + (V λ) row Whyper, b̂φ(λ) = belem + (Cλ) bhyper (10) Here, indicates elementwise multiplication and row indicates row-wise rescaling. This architecture computes the usual elementary weight/bias, plus an additional weight/bias which has been scaled by a linear transformation of the hyperparameters. Alternatively, it can be interpreted as directly operating on the pre-activations of the layer, adding a correction to the usual pre-activation to account for the hyperparameters:
Ŵφ(λ)x+ b̂φ(λ) = [Welemx+ belem] + [(V λ) (Whyperx) + (Cλ) bhyper] (11)
This best-response architecture is tractable to compute and memory-efficient: it requires Dout(2Din + n) parameters to represent Ŵφ and Dout(2 + n) parameters to represent b̂φ, where n is the number of hyperparameters. Furthermore, it enables parallelism: since the predictions can be computed by transforming the pre-activations (Equation 11), the hyperparameters for different examples in a batch can be perturbed independently, improving sample efficiency. In practice, the approximation can be implemented by simply replacing existing modules in deep learning libraries with “hyper” counterparts which accept an additional vector of hyperparameters as input4.

heading: 3.2 EXACT BEST-RESPONSE FOR TWO-LAYER LINEAR NETWORKS
text: Given that the best-response function is a mapping from Rn to the high-dimensional weight space Rm, why should we expect to be able to represent it compactly? And why in particular would equation 10 be a reasonable approximation? In this section, we exhibit a model whose best-response function can be represented exactly using a minor variant of equation 10: a linear network with Jacobian norm regularization. In particular, the best-response takes the form of a network whose hidden units are modulated conditionally on the hyperparameters.
Consider using a 2-layer linear network with weights w = (Q, s) ∈ RD×D × RD to predict targets t ∈ R from inputs x ∈ RD:
3We describe modifications for convolutional filters in Appendix C. 4We illustrate how this is done for the PyTorch library (Paszke et al., 2017) in Appendix G.
a(x;w) = Qx, y(x;w) = s>a(x;w) (12)
Suppose we use a squared-error loss regularized with an L2 penalty on the Jacobian ∂y/∂x, where the penalty weight λ lies in R and is mapped using exp to lie R+ :
LT (λ,w) = ∑
(x,t)∈D
(y(x;w)− t)2 + 1 |D| exp(λ) ∥∥∥∥ ∂y∂x (x;w) ∥∥∥∥2 (13)
Theorem 2. Let w0 = (Q0, s0), where Q0 is the change-of-basis matrix to the principal components of the data matrix and s0 solves the unregularized version of Problem 13 givenQ0. Then there exist v, c ∈ RD such that the best-response function5 w∗(λ) = (Q∗(λ), s∗(λ)) is:
Q∗(λ) = σ(λv + c) row Q0, s∗(λ) = s0,
where σ is the sigmoid function.
Proof. See Appendix B.2.
Observe that y(x;w∗(λ)) can be implemented as a regular network with weights w0 = (Q0, s0) with an additional sigmoidal gating of its hidden units a(x;w∗(λ)):
a(x;w∗(λ)) = Q∗(λ)x = σ(λv + c) row (Q0x) = σ(λv + c) row a(x;w0) (14)
This architecture is shown in Figure 1. Inspired by this example, we use a similar gating of the hidden units to approximate the best-response for deep, nonlinear networks.

heading: 3.3 LINEAR BEST-RESPONSE APPROXIMATIONS
text: The sigmoidal gating architecture of the preceding section can be further simplified if one only needs to approximate the best-response function for a small range of hyperparameter values. In particular, for a narrow enough hyperparameter distribution, a smooth best-response function can be approximated by an affine function (i.e. its first-order Taylor approximation). Hence, we replace the sigmoidal gating with linear gating, in order that the weights be affine in the hyperparameters. The following theorem shows that, for quadratic lower-level objectives, using an affine approximation to the best-response function and minimizing E ∼p( |σ) [f(λ+ , ŵφ(λ+ ))] yields the correct best-response Jacobian, thus ensuring gradient descent on the approximate objective F (λ, ŵφ(λ)) converges to a local optimum:
Theorem 3. Suppose f is quadratic with ∂2f/∂w2 0, p( |σ) is Gaussian with mean 0 and variance σ2I , and ŵφ is affine. Fix λ0 ∈ Rn and let φ∗ = argminφ E ∼p( |σ) [f(λ+ , ŵφ(λ+ ))]. Then we have ŵφ/∂λ(λ0) = ∂w ∗ /∂λ(λ0).
Proof. See Appendix B.3.

heading: 3.4 ADAPTING THE HYPERPARAMETER DISTRIBUTION
text: The entries of σ control the scale of the hyperparameter distribution on which φ is trained. If the entries are too large, then ŵφ will not be flexible enough to capture the best-response over the samples. However, the entries must remain large enough to force ŵφ to capture the shape locally around the current hyperparameter values. We illustrate this in Figure 2. As the smoothness of the loss landscape changes during training, it may be beneficial to vary σ.
To address these issues, we propose adjusting σ during training based on the sensitivity of the upperlevel objective to the sampled hyperparameters. We include an entropy term weighted by τ ∈ R+ which acts to enlarge the entries of σ. The resulting objective is:
E ∼p( |σ)[F (λ+ , ŵφ(λ+ ))]− τH[p( |σ)] (15)
This is similar to a variational inference objective, where the first term is analogous to the negative log-likelihood, but τ 6= 1. As τ ranges from 0 to 1, our objective interpolates between variational optimization (Staines & Barber, 2012) and variational inference, as noted by Khan et al. (2018). Similar objectives have been used in the variational inference literature for better training (Blundell et al., 2015) and representation learning (Higgins et al., 2017).
Minimizing the first term on its own eventually moves all probability mass towards an optimum λ∗, resulting in σ = 0 if λ∗ is an isolated local minimum. This compels σ to balance between shrinking to decrease the first term while remaining sufficiently large to avoid a heavy entropy penalty. When benchmarking our algorithm’s performance, we evaluate F (λ, ŵφ(λ)) at the deterministic current hyperparameter λ0. (This is a common practice when using stochastic operations during training, such as batch normalization or dropout.)

heading: 3.5 TRAINING ALGORITHM
text: We now describe the complete STN training algorithm and discuss how it can tune hyperparameters that other gradient-based algorithms cannot, such as discrete or stochastic hyperparameters. We use an unconstrained parametrization λ ∈ Rn of the hyperparameters. Let r denote the element-wise function which maps λ to the appropriate constrained space, which will involve a non-differentiable discretization for discrete hyperparameters.
Let LT and LV denote training and validation losses which are (possibly stochastic, e.g., if using dropout) functions of the hyperparameters and parameters. Define functions f, F by f(λ,w) = LT (r(λ),w) and F (λ,w) = LV (r(λ),w). STNs are trained by a gradient descent scheme which alternates between updatingφ for Ttrain steps to minimize E ∼p( |σ) [f(λ+ , ŵφ(λ+ ))] (Eq. 9) and updating λ and σ for Tvalid steps to minimize E ∼p( |σ)[F (λ+ , ŵφ(λ+ ))]− τH[p( |σ)] (Eq. 15). We give our complete algorithm as Algorithm 1 and show how it can be implemented in code in Appendix G. The possible non-differentiability of r due to discrete hyperparameters poses no problem. To estimate the derivative of E ∼p( |σ) [f(λ+ , ŵφ(λ+ ))] with respect to φ, we can use the reparametrization trick and compute ∂f/∂w and ∂ŵφ/∂φ, neither of whose computation paths involve the discretization r. To differentiate E ∼p( |σ)[F (λ + , ŵφ(λ + ))] − τH[p( |σ)] with respect to a discrete hyperparameter λi, there are two cases we must consider:
5This is an abuse of notation since there is not a unique solution to Problem 13 for each λ in general.
Algorithm 1 STN Training Algorithm Initialize: Best-response approximation parameters φ, hyperparameters λ, learning rates {αi}3i=1 while not converged do
for t = 1, . . . , Ttrain do ∼ p( |σ) φ← φ− α1 ∂∂φf(λ+ , ŵφ(λ+ )) for t = 1, . . . , Tvalid do ∼ p( |σ) λ← λ−α2 ∂∂λ (F (λ+ , ŵφ(λ+ ))− τH[p( |σ)]) σ ← σ−α3 ∂∂σ (F (λ+ , ŵφ(λ+ ))− τH[p( |σ)])
Case 1: For most regularization schemes, LV and hence F does not depend on λi directly and thus the only gradient is through ŵφ. Thus, the reparametrization gradient can be used.
Case 2: If LV relies explicitly on λi, then we can use the REINFORCE gradient estimator (Williams, 1992) to estimate the derivative of the expectation with respect to λi. The number of hidden units in a layer is an example of a hyperparameter that requires this
approach since it directly affects the validation loss. We do not show this in Algorithm 1, since we do not tune any hyperparameters which fall into this case.

heading: 4 EXPERIMENTS
text: We applied our method to convolutional networks and LSTMs (Hochreiter & Schmidhuber, 1997), yielding self-tuning CNNs (ST-CNNs) and self-tuning LSTMs (ST-LSTMs). We first investigated the behavior of STNs in a simple setting where we tuned a single hyperparameter, and found that STNs discovered hyperparameter schedules that outperformed fixed hyperparameter values. Next, we compared the performance of STNs to commonly-used hyperparameter optimization methods on the CIFAR-10 (Krizhevsky & Hinton, 2009) and PTB (Marcus et al., 1993) datasets.

heading: 4.1 HYPERPARAMETER SCHEDULES
text: Due to the joint optimization of the hypernetwork weights and hyperparameters, STNs do not use a single, fixed hyperparameter during training. Instead, STNs discover schedules for adapting the hyperparameters online, which can outperform any fixed hyperparameter. We examined this behavior in detail on the PTB corpus (Marcus et al., 1993) using an ST-LSTM to tune the output dropout rate applied to the hidden units.
The schedule discovered by an ST-LSTM for output dropout, shown in Figure 3, outperforms the best, fixed output dropout rate (0.68) found by a fine-grained grid search, achieving 82.58 vs 85.83 validation perplexity. We claim that this is a consequence of the schedule, and not of regularizing effects from sampling hyperparameters or the limited capacity of ŵφ.
To rule out the possibility that the improved performance is due to stochasticity introduced by sampling hyperparameters during STN training, we trained a standard LSTM while perturbing its dropout rate around the best value found by grid search. We used (1) random Gaussian perturbations, and (2) sinusoid perturbations for a cyclic regularization schedule. STNs outperformed both perturbation methods (Table 1), showing that the improvement is not merely due to hyperparameter stochasticity. Details and plots of each perturbation method are provided in Appendix F.
To determine whether the limited capacity of ŵφ acts as a regularizer, we trained a standard LSTM from scratch using the schedule for output dropout discovered by the ST-LSTM. Using this schedule, the standard LSTM performed nearly as well as the STN, providing evidence that the schedule itself (rather than some other aspect of the STN) was responsible for the improvement over a fixed dropout rate. To further demonstrate the importance of the hyperparameter schedule, we also trained a standard LSTM from scratch using the final dropout value found by the STN (0.78), and found that it did not perform as well as when following the schedule. The final validation and test perplexities of each variant are shown in Table 1.
Next, we show in Figure 3 that the STN discovers the same schedule regardless of the initial hyperparameter values. Because hyperparameters adapt over a shorter timescale than the weights, we find that at any given point in training, the hyperparameter adaptation has already equilibrated. As shown empirically in Appendix F, low regularization is best early in training, while higher regularization is better later on. We found that the STN schedule implements a curriculum by using a low dropout rate early in training, aiding optimization, and then gradually increasing the dropout rate, leading to better generalization.

heading: 4.2 LANGUAGE MODELING
text: We evaluated an ST-LSTM on the PTB corpus (Marcus et al., 1993), which is widely used as a benchmark for RNN regularization due to its small size (Gal & Ghahramani, 2016; Merity et al., 2018; Wen et al., 2018). We used a 2-layer LSTM with 650 hidden units per layer and 650- dimensional word embeddings. We tuned 7 hyperparameters: variational dropout rates for the input, hidden state, and output; embedding dropout (that sets rows of the embedding matrix to 0); DropConnect (Wan et al., 2013) on the hidden-to-hidden weight matrix; and coefficients α and β that control the strength of activation regularization and temporal activation regularization, respectively. For LSTM tuning, we obtained the best results when using a fixed perturbation scale of 1 for the hyperparameters. Additional details about the experimental setup and the role of these hyperparameters can be found in Appendix D.
We compared STNs to grid search, random search, and Bayesian optimization.6 Figure 4a shows the best validation perplexity achieved by each method over time. STNs outperform other meth-
6For grid search and random search we used the Ray Tune libraries (https://github.com/ ray-project/ray/tree/master/python/ray/tune). For Bayesian optimization, we used Spearmint (https://github.com/HIPS/Spearmint).
ods, achieving lower validation perplexity more quickly. The final validation and test perplexities achieved by each method are shown in Table 2. We show the schedules the STN finds for each hyperparameter in Figures 4b and 4c; we observe that they are nontrivial, with some forms of dropout used to a greater extent at the start of training (including input and hidden dropout), some used throughout training (output dropout), and some that are increased over the course of training (embedding and weight dropout).
4.3 IMAGE CLASSIFICATION
We evaluated ST-CNNs on the CIFAR-10 (Krizhevsky & Hinton, 2009) dataset, where it is easy to overfit with high-capacity networks. We used the AlexNet architecture (Krizhevsky et al., 2012), and tuned: (1) continuous hyperparameters controlling per-layer activation dropout, input dropout, and scaling noise applied to the input, (2) discrete data augmentation hyperparameters controlling the length and number of cut-out holes (DeVries & Taylor, 2017), and (3) continuous data augmentation hyperparameters controlling the amount of noise to apply to the hue, saturation, brightness, and contrast of an image. In total, we considered 15 hyperparameters.
We compared STNs to grid search, random search, and Bayesian optimization. Figure 5 shows the lowest validation loss achieved by each method over time, and
Table 2 shows the final validation and test losses for each method. Details of the experimental setup are provided in Appendix E. Again, STNs find better hyperparameter configurations in less time than other methods. The hyperparameter schedules found by the STN are shown in Figure 6.

heading: 5 RELATED WORK
text: Bilevel Optimization. Colson et al. (2007) provide an overview of bilevel problems, and a comprehensive textbook was written by Bard (2013). When the objectives/constraints are restricted to be linear, quadratic, or convex, a common approach replaces the lower-level problem with its KKT conditions added as constraints for the upper-level problem (Hansen et al., 1992; Vicente et al., 1994). In the unrestricted setting, our work loosely resembles trust-region methods (Colson et al., 2005), which repeatedly approximate the problem locally using a simpler bilevel program. In closely related work, Sinha et al. (2013) used evolutionary techniques to estimate the best-response function iteratively.
Hypernetworks. First considered by Schmidhuber (1993; 1992), hypernetworks are functions mapping to the weights of a neural net. Predicting weights in CNNs has been developed in various forms (Denil et al., 2013; Yang et al., 2015). Ha et al. (2016) used hypernetworks to generate weights for modern CNNs and RNNs. Brock et al. (2017) used hypernetworks to globally approximate a bestresponse for architecture search. Because the architecture is not optimized during training, they require a large hypernetwork, unlike ours which locally approximates the best-response.
Gradient-Based Hyperparameter Optimization. There are two main approaches. The first approach approximates w∗(λ0) using wT (λ0,w0), the value of w after T steps of gradient descent on f with respect to w starting at (λ0,w0). The descent steps are differentiated through to approximate ∂w∗/∂λ(λ0) ≈ ∂wT/∂λ(λ0,w0). This approach was proposed by Domke (2012) and used by Maclaurin et al. (2015), Luketina et al. (2016) and Franceschi et al. (2018). The second approach uses the Implicit Function Theorem to derive ∂w∗/∂λ(λ0) under certain conditions. This was first developed for hyperparameter optimization in neural networks (Larsen et al., 1996) and developed further by Pedregosa (2016). Similar approaches have been used for hyperparameter optimization in log-linear models (Foo et al., 2008), kernel selection (Chapelle et al., 2002; Seeger, 2007), and image reconstruction (Kunisch & Pock, 2013; Calatroni et al., 2015). Both approaches struggle with certain hyperparameters, since they differentiate gradient descent or the training loss with respect to the hyperparameters. In addition, differentiating gradient descent becomes prohibitively expensive as the number of descent steps increases, while implicitly deriving ∂w∗/∂λ requires using Hessian-vector products with conjugate gradient solvers to avoid directly computing the Hessian.
Model-Based Hyperparameter Optimization. A common model-based approach is Bayesian optimization, which models p(r|λ,D), the conditional probability of the performance on some metric r given hyperparameters λ and a dataset D = {(λi, ri)}. We can model p(r|λ,D) with various methods (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; 2015). D is constructed iteratively, where the next λ to train on is chosen by maximizing an acquisition functionC(λ; p(r|λ,D)) which balances exploration and exploitation. Training each model to completion can be avoided if assumptions are made on learning curve behavior (Swersky et al., 2014; Klein et al., 2017). These approaches require building inductive biases into p(r|λ,D) which may not hold in practice, do not take advantage of the network structure when used for hyperparameter optimization, and do not scale well with the number of hyperparameters. However, these approaches have consistency guarantees in the limit, unlike ours.
Model-Free Hyperparameter Optimization. Model-free approaches include grid search and random search. Bergstra & Bengio (2012) advocated using random search over grid search. Successive Halving (Jamieson & Talwalkar, 2016) and Hyperband (Li et al., 2017) extend random search by adaptively allocating resources to promising configurations using multi-armed bandit techniques. These methods ignore structure in the problem, unlike ours which uses rich gradient information. However, it is trivial to parallelize model-free methods over computing resources and they tend to perform well in practice.
Hyperparameter Scheduling. Population Based Training (PBT) (Jaderberg et al., 2017) considers schedules for hyperparameters. In PBT, a population of networks is trained in parallel. The performance of each network is evaluated periodically, and the weights of under-performing networks are replaced by the weights of better-performing ones; the hyperparameters of the better network are also copied and randomly perturbed for training the new network clone. In this way, a single model can experience different hyperparameter settings over the course of training, implementing a schedule. STNs replace the population of networks by a single best-response approximation and use gradients to tune hyperparameters during a single training run.

heading: 6 CONCLUSION
text: We introduced Self-Tuning Networks (STNs), which efficiently approximate the best-response of parameters to hyperparameters by scaling and shifting their hidden units. This allowed us to use gradient-based optimization to tune various regularization hyperparameters, including discrete hyperparameters. We showed that STNs discover hyperparameter schedules that can outperform fixed hyperparameters. We validated the approach on large-scale problems and showed that STNs achieve better generalization performance than competing approaches, in less time. We believe STNs offer a compelling path towards large-scale, automated hyperparameter tuning for neural networks.

heading: ACKNOWLEDGMENTS
text: We thank Matt Johnson for helpful discussions and advice. MM is supported by an NSERC CGSM award, and PV is supported by an NSERC PGS-D award. RG acknowledges support from the CIFAR Canadian AI Chairs program.

heading: A TABLE OF NOTATION

heading: B PROOFS
text: B.1 LEMMA 1
Because w0 solves Problem 4b given λ0, by the first-order optimality condition we must have:
∂f ∂w (λ0,w0) = 0 (16)
The Jacobian of ∂f/∂w decomposes as a block matrix with sub-blocks given by:[ ∂2f ∂λ∂w ∂2f ∂w2 ] (17)
We know that f is C2 in some neighborhood of (λ0,w0), so ∂f/∂w is continuously differentiable in this neighborhood. By assumption, the Hessian ∂2f/∂w2 is positive definite and hence invertible at (λ0,w0). By the Implicit Function Theorem, there exists a neighborhood V of λ0 and a unique continuously differentiable function w∗ : V → Rm such that ∂f/∂w(λ,w∗(λ)) = 0 for λ ∈ V and w∗(λ0) = w0.
Furthermore, by continuity we know that there is a neighborhood W1 ×W2 of (λ0,w0) such that ∂2f/∂w2 is positive definite on this neighborhood. Setting U = V ∩ W1 ∩ (w∗)−1(W2), we can conclude that ∂2f/∂w2(λ,w∗(λ)) 0 for all λ ∈ U . Combining this with ∂f/∂w(λ,w∗(λ)) = 0 and using second-order sufficient optimality conditions, we conclude that w∗(λ) is the unique solution to Problem 4b for all λ ∈ U .
B.2 LEMMA 2
This discussion mostly follows from Hastie et al. (2001). We letX ∈ RN×D denote the data matrix where N is the number of training examples and D is the dimensionality of the data. We let t ∈ RN denote the associated targets. We can write the SVD decomposition ofX as:
X = UDV > (18)
where U and V are N × D and D × D orthogonal matrices and D is a diagonal matrix with entries d1 ≥ d2 ≥ · · · ≥ dD > 0. We next simplify the function y(x;w) by setting u = s>Q, so that y(x;w) = s>Qx = u>x. We see that the Jacobian ∂y/∂x ≡ u is constant, and Problem 13 simplifies to standardL2-regularized least-squares linear regression with the following loss function:∑
(x,t)∈D
(u>x− t)2 + 1 |D| exp(λ) ‖u‖2 (19)
It is well-known (see Hastie et al. (2001), Chapter 3) that the optimal solution u∗(λ) minimizing Equation 19 is given by:
u∗(λ) = (X>X + exp(λ)I)−1X>t = V (D2 + exp(λ)I)−1DU>t (20)
Furthermore, the optimal solution u∗ to the unregularized version of Problem 19 is given by:
u∗ = V D−1U>t (21)
Recall that we defined Q0 = V >, i.e., the change-of-basis matrix from the standard basis to the principal components of the data matrix, and we defined s0 to solve the unregularized regression problem givenQ0. Thus, we require thatQ>0 s0 = u ∗ which implies s0 =D−1U>t.
There are not unique solutions to Problem 13, so we take any functions Q(λ), s(λ) which satisfy Q(λ)>s(λ) = v∗(λ) as “best-response functions”. We will show that our chosen functions Q∗(λ) = σ(λv+ c) rowQ0 and s∗(λ) = s0, where v = −1 and ci = 2 log(di) for i = 1, . . . , D, meet this criteria. We start by noticing that for any d ∈ R+, we have:
σ(−λ+ 2 log(d)) = 1 1 + exp(λ− 2 log(d)) = 1 1 + d−2 exp(λ) =
d2
d2 + exp(λ) (22)
It follows that:
Q∗(λ)>s∗(λ) = [σ(λv + c) row Q0]>s0 (23)
= diag  σ(−λ+ 2 log(d1)) ...
σ(−λ+ 2 log(dD))
Q0  > s0 (24)
= Q>0 diag  d21 d21+exp(λ)
... d2D
d2D+exp(λ)
  s0 (25)
= V diag  d21 d21+exp(λ)
... d2D
d2D+exp(λ)
 D−1U>t (26)
= V diag  d1 d21+exp(λ)
... dD
d2D+exp(λ)
 U>t (27)
= V [ (D2 + exp(λ)I)−1D ] U>t (28)
= v∗(λ) (29)
B.3 THEOREM 3
By assumption f is quadratic, so there existA ∈ Rn×n,B ∈ Rn×m,C ∈ Rm×m and d ∈ Rn, e ∈ Rm such that:
f(λ,w) = 1
2
( λ> w> )( A B B> C )( λ w ) + d>λ+ e>w (30)
One can easily compute that: ∂f
∂w (λ,w) = B>λ+Cw + e (31)
∂2f ∂w2 (λ,w) = C (32)
Since we assume ∂2f/∂w2 0, we must have C 0. Setting the derivative equal to 0 and using second-order sufficient conditions, we have:
w∗(λ) = −C−1(e+B>λ) (33) Hence, we find:
∂w∗
∂λ (λ) = −C−1B> (34)
We let ŵφ(λ) = Uλ+ b, and define f̂ to be the function given by:
f̂(λ,U , b,σ) = E ∼p( |σ) [f(λ+ ,U(λ+ ) + b)] (35) Substituting and simplifying:
f̂(λ0,U , b, σ) = E ∼p( |σ) [ 1
2 (λ0 + )
>A(λ0 + ) + (λ0 + ) >B(U(λ0 + ) + b)
+ 1
2 (U(λ0 + ) + b)
>C(U(λ0 + ) + b)
+ d>(λ0 + ) + e >(U(λ0 + ) + b) ] (36)
Expanding, we find that equation 36 is equal to: E ∼p( |σ) [ 1 + 2 + 3 + 4 ]
(37)
where we have: 1 = 1
2
( λ>0 Aλ0 + 2 >Aλ0 + >A ) (38)
2 = λ>0 BUλ0 + λ > 0 BU + λ > 0 Bb+ >BUλ0 + >BU + >Bb (39)
3 = 1
2 (λ>0 U >CUλ0 + λ0U >CU + λ0U >Cb+ >U>CUλ0
+ >U>CU + >U>Cb+ b>CUλ0 + b >CU + b>Cb) (40)
4 = d>λ0 + d> + e>Uλ0 + e>U + e>b (41) We can simplify these expressions considerably by using linearity of expectation and that ∼ p( |σ) has mean 0:
E ∼p( |σ) [ 1 ] = 1
2 λ>0 Aλ0 (42) E ∼p( |σ) [ 2 ] = λ>0 BUλ0 + λ > 0 Bb+ E ∼p( |σ) [ >BU ] (43)
E ∼p( |σ) [ 3 ] = 1
2 (λ>0 U >CUλ0 + λ0U >Cb+ E ∼p( |σ) [ >U>CU ] + b>CUλ0 + b >Cb) (44)
E ∼p( |σ) [ 4 ] = d>λ0 + e >Uλ0 + e >b (45)
We can use the cyclic property of the Trace operator, E ∼p( |σ)[ >] = σ2I , and commutability of expectation and a linear operator to simplify the expectations of 2 and 3 :
E ∼p( |σ) [ 2 ] = λ>0 BUλ0 + λ > 0 Bb+Tr [ σ2BU ] (46)
E ∼p( |σ) [ 3 ] = 1
2 (λ>0 U >CUλ0 + λ0U >Cb+ Tr [ σ2U>CU ] + b>CUλ0 + b >Cb) (47)
We can then differentiate f̂ by making use of various matrix-derivative equalities (Duchi, 2007) to find:
∂f̂ ∂b (λ0,U , b, σ) = 1 2 C>Uλ0 + 1 2 CUλ0 +B >λ0 + e+Cb (48)
∂f̂ ∂U (λ0,U , b, σ) = B >λ0λ > 0 + σ 2B> +Cbλ>0 + eλ > 0 +CUλ0λ > 0 + σ 2CU (49)
Setting the derivative ∂f̂/∂b(λ0,U , b, σ) equal to 0, we have:
b = −C−1(C>Uλ0 +B>λ0 + e) (50) Setting the derivative for ∂f̂/∂U(λ0,U , b, σ) equal to 0, we have:
CU(λ0λ > 0 + σ 2C) = −B>λ0λ>0 − σ2B> −Cbλ>0 − eλ>0 (51) Substituting the expression for b given by equation 50 into equation 51 and simplifying gives:
CU(λ0λ > 0 + σ 2I) = −σ2B> +CUλ0λ>0 (52) =⇒ σ2CU = −σ2B> (53)
=⇒ U = −C−1B> (54) This is exactly the best-response Jacobian ∂w∗/∂λ(λ) as given by Equation 34. Substituting U = C−1B into the equation 50 gives:
b = C−1B>λ0 −C−1B>λ0 −C−1e (55) This is w∗(λ0) − ∂w∗/∂λ(λ0), thus the approximate best-response is exactly the first-order Taylor series of w∗ about λ0.
B.4 BEST-RESPONSE GRADIENT LEMMA
Lemma 4. Under the same conditions as Lemma 1 and using the same notation, for all λ ∈ U , we have that:
∂w∗
∂λ (λ) = −
[ ∂2f
∂w2 (λ,w∗(λ))
]−1 ∂2f
∂λ∂w (λ,w∗(λ)) (56)
Proof. Define ι∗ : U → Rn×Rm by ι∗(λ) = (λ,w∗(λ)). By first-order optimality conditions, we know that: (
∂f ∂w ◦ ι∗ ) (λ) = 0 ∀λ ∈ U (57)
Hence, for all λ ∈ U :
0 = ∂
∂λ
( ∂f ∂w ◦ ι∗ ) (λ) (58)
= ∂2f
∂w2 (ι∗(λ))
∂w∗
∂λ (λ) +
∂2f
∂λ∂w (ι∗(λ)) (59)
= ∂2f
∂w2 (λ,w∗(λ))
∂w∗
∂λ (λ) +
∂2f
∂λ∂w (λ,w∗(λ)) (60)
Rearranging gives Equation 56.

heading: C BEST-RESPONSE APPROXIMATIONS FOR CONVOLUTIONAL FILTERS
text: We let L denote the number of layers, Cl the number of channels in layer l’s feature map, and Kl the size of the kernel in layer l. We let W l,c ∈ RCl−1×Kl×Kl and bl,c ∈ R denote the weight and bias respectively of the cth convolution kernel in layer l (so c ∈ {1, . . . , Cl}). For ul,c,al,c ∈ Rn, we define best-response approximations Ŵ l,cφ and b̂ l,c φ by:
Ŵ l,cφ (λ) = (λ >ul,c) W l,chyper +W l,c elem (61)
b̂l,cφ (λ) = (λ >al,c) bl,chyper + b l,c elem (62)
Thus, the best-response parameters used for modeling W l,c, bl are {ul,c,al,c,W l,chyper,W l,c elem, b l,c hyper, b l,c elem}. We can compute the number of parameters used as 2n + 2(|W l,c| + |bl,c|). Summing over channels c, we find the total number of parameters is 2nCl + 2p, where p is the total number of parameters in the normal CNN layer. Hence, we use twice the number of parameters in a normal CNN, plus an overhead that depends on the number of hyperparameters.
For an implementation in code, see Appendix G.

heading: D LANGUAGE MODELING EXPERIMENT DETAILS
text: Here we present additional details on the setup of our LSTM language modeling experiments on PTB, and on the role of each hyperparameter we tune.
We trained a 2-layer LSTM with 650 hidden units per layer and 650-dimensional word embeddings (similar to (Zaremba et al., 2014; Gal & Ghahramani, 2016)) on sequences of length 70 in minibatches of size 40. To optimize the baseline LSTM, we used SGD with initial learning rate 30, which was decayed by a factor of 4 based on the non-monotonic criterion introduced by Merity et al. (2018) (i.e., whenever the validation perplexity fails to improve for 5 epochs). Following Merity et al. (2018), we used gradient clipping 0.25.
To optimize the ST-LSTM, we used the same optimization setup as for the baseline LSTM. For the hyperparameters, we used Adam with learning rate 0.01. We used an alternating training schedule in which we updated the model parameters for 2 steps on the training set and then updated the hyperparameters for 1 step on the validation set. We used one epoch of warm-up, in which we
updated the model parameters, but did not update hyperparameters. We terminated training when the learning rate dropped below 0.0003.
We tuned variational dropout (re-using the same dropout mask for each step in a sequence) on the input to the LSTM, the hidden state between the LSTM layers, and the output of the LSTM. We also tuned embedding dropout, which sets entire rows of the word embedding matrix to 0, effectively removing certain words from all sequences. We regularized the hidden-to-hidden weight matrix using DropConnect (zeroing out weights rather than activations) (Wan et al., 2013). Because DropConnect operates directly on the weights and not individually on the mini-batch elements, we cannot use independent perturbations per example; instead, we sample a single DropConnect rate per mini-batch. Finally, we used activation regularization (AR) and temporal activation regularization (TAR). AR penalizes large activations, and is defined as:
α||m ht||2 (63)
wherem is a dropout mask and ht is the output of the LSTM at time t. TAR is a slowness regularizer, defined as: β||ht − ht+1||2 (64) For AR and TAR, we tuned the scaling coefficients α and β. For the baselines, the hyperparameter ranges were: [0, 0.95] for the dropout rates, and [0, 4] for α and β. For the ST-LSTM, all the dropout rates and the coefficients α and β were initialized to 0.05 (except in Figure 3, where we varied the output dropout rate).
E IMAGE CLASSIFICATION EXPERIMENT DETAILS
Here, we present additional details on the CNN experiments. For all results, we held out 20% of the training data for validation.
We trained the baseline CNN using SGD with initial learning rate 0.01 and momentum 0.9, on mini-batches of size 128. We decay the learning rate by 10 each time the validation loss fails to decrease for 60 epochs, and end training if the learning rate falls below 10−5 or validation loss has not decreased for 75 epochs. For the baselines—grid search, random search, and Bayesian optimization—the search spaces for the hyperparameters were as follows: dropout rates were in the range [0, 0.75]; contrast, saturation, and brightness each had range [0, 1]; hue had range [0, 0.5]; the number of cutout holes had range [0, 4], and the length of each cutout hole had range [0, 24].
We trained the ST-CNN’s elementary parameters using SGD with initial learning rate 0.01 and momentum of 0.9, on mini-batches of size 128 (identical to the baselines). We use the same decay schedule as the baseline model. The hyperparameters are optimized using Adam with learning rate 0.003. We alternate between training the best-response approximation and hyperparameters with the same schedule as the ST-LSTM, i.e. Ttrain = 2 steps on the training step and Tvalid = 1 steps on the validation set. Similarly to the LSTM experiments, we used five epochs of warm-up for the model parameters, during which the hyperparameters are fixed. We used an entropy weight of τ = 0.001 in the entropy regularized objective (Eq. 15). The cutout length was restricted to lie in {0, . . . , 24}while the number of cutout holes was restricted to lie in {0, . . . , 4}. All dropout rates, as well as the continuous data augmentation noise parameters, are initialized to 0.05. The cutout length is initialized to 4, and the number of cutout holes is initialized to 1. Overall, we found the ST-CNN to be relatively robust to the initialization of hyperparameters, but starting with low regularization aided optimization in the first few epochs.

heading: F ADDITIONAL DETAILS ON HYPERPARAMETER SCHEDULES
text: Here, we draw connections between hyperparameter schedules and curriculum learning. Curriculum learning (Bengio et al., 2009) is an instance of a family of continuation methods (Allgower & Georg, 2012), which optimize non-convex functions by solving a sequence of functions that are ordered by increasing difficulty. In a continuation method, one considers a family of training criteria Cλ(w) with a parameter λ, where C1(w) is the final objective we wish to minimize, and C0(w) represents the training criterion for a simpler version of the problem. One starts by optimizing C0(w) and then gradually increases λ from 0 to 1, while keeping w at a local minimum of Cλ(w) (Bengio et al., 2009). This has been hypothesized to both aid optimization and improve generalization. In this
section, we explore how hyperparameter schedules implement a form of curriculum learning; for example, a schedule that increases dropout over time increases stochasticity, making the learning problem more difficult. We use the results of grid searches to understand the effects of different hyperparameter settings throughout training, and show that greedy hyperparameter schedules can outperform fixed hyperparameter values.
First, we performed a grid search over 20 values each of input and output dropout, and measured the validation perplexity in each epoch. Figure 7 shows the validation perplexity achieved by different combinations of input and output dropout, at various epochs during training. We see that at the start of training, the best validation loss is achieved with small values of both input and output dropout. As we train for more epochs, the best validation performance is achieved with larger dropout rates.
Next, we present a simple example to show the potential benefits of greedy hyperparameter schedules. For a single hyperparameter—output dropout—we performed a fine-grained grid search and constructed a dropout schedule by using the hyperparameter values that achieve the best validation perplexity at each epoch in training. As shown in Figure 8, the schedule formed by taking the best output dropout value in each epoch yields better generalization than any of the fixed hyperparameter values from the initial grid search. In particular, by using small dropout values at the start of training, the schedule achieves a fast decrease in validation perplexity, and by using larger dropout later in training, it achieves better overall validation perplexity.
Figure 9 shows the perturbed values for output dropout we used to investigate whether the improved performance yielded by STNs is due to the regularization effect, and not the schedule, in Section 4.1.

heading: G CODE LISTINGS
text: In this section, we provide PyTorch code listings for the approximate best-response layers used to construct ST-LSTMs and ST-CNNs: the HyperLinear and HyperConv2D classes. We also provide a simplified version of the optimization steps used on the training set and validation set.
Listing 1: HyperLinear, used as a drop-in replacement for Linear modules
class HyperLinear(nn.Module):
def __init__(self, input_dim, output_dim, n_hparams): super(HyperLinear, self).__init__()
self.input_dim = input_dim self.output_dim = output_dim self.n_hparams = n_hparams self.n_scalars = output_dim
self.elem_w = nn.Parameter(torch.Tensor(output_dim, input_dim)) self.elem_b = nn.Parameter(torch.Tensor(output_dim))
self.hnet_w = nn.Parameter(torch.Tensor(output_dim, input_dim)) self.hnet_b = nn.Parameter(torch.Tensor(output_dim))
self.htensor_to_scalars = nn.Linear(self.n_hparams, self.n_scalars*2, bias=False)
self.init_params()
def forward(self, input, hnet_tensor): output = F.linear(input, self.elem_w, self.elem_b)
if hnet_tensor is not None: hnet_scalars = self.htensor_to_scalars(hnet_tensor) hnet_wscalars = hnet_scalars[:, :self.n_scalars] hnet_bscalars = hnet_scalars[:, self.n_scalars:]
hnet_out = hnet_wscalars * F.linear(input, self.hnet_w) hnet_out += hnet_bscalars * self.hnet_b
output += hnet_out
return output
Listing 2: HyperConv2d, used as a drop-in replacement for Conv2d modules
class HyperConv2d(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, padding, num_hparams, stride=1, bias=True): super(HyperConv2d, self).__init__()
self.in_channels = in_channels self.out_channels = out_channels self.kernel_size = kernel_size self.padding = padding self.num_hparams = num_hparams self.stride = stride
self.elem_weight = nn.Parameter(torch.Tensor( out_channels, in_channels, kernel_size, kernel_size)) self.hnet_weight = nn.Parameter(torch.Tensor( out_channels, in_channels, kernel_size, kernel_size)) if bias: self.elem_bias = nn.Parameter(torch.Tensor(out_channels)) self.hnet_bias = nn.Parameter(torch.Tensor(out_channels)) else: self.register_parameter(’elem_bias’, None) self.register_parameter(’hnet_bias’, None)
self.htensor_to_scalars = nn.Linear( self.num_hparams, self.out_channels*2, bias=False) self.elem_scalar = nn.Parameter(torch.ones(1))
self.init_params()
def forward(self, input, htensor): """ Arguments:
input (tensor): size should be (B, C, H, W) htensor (tensor): size should be (B, D)
""" output = F.conv2d(input, self.elem_weight, self.elem_bias,
padding=self.padding, stride=self.stride)
output *= self.elem_scalar if htensor is not None:
hnet_scalars = self.htensor_to_scalars(htensor) hnet_wscalars = hnet_scalars[:,
:self.out_channels].unsqueeze(2).unsqueeze(2) hnet_bscalars = hnet_scalars[:, self.out_channels:]
hnet_out = F.conv2d(input, self.hnet_weight, padding=self.padding, stride=self.stride) hnet_out *= hnet_wscalars if self.hnet_bias is not None:
hnet_out += (hnet_bscalars * self.hnet_bias).unsqueeze(2).unsqueeze(2)
output += hnet_out return output
def init_params(self): n = self.in_channels * self.kernel_size * self.kernel_size stdv = 1. / math.sqrt(n) self.elem_weight.data.uniform_(-stdv, stdv) self.hnet_weight.data.uniform_(-stdv, stdv) if self.elem_bias is not None:
self.elem_bias.data.uniform_(-stdv, stdv) self.hnet_bias.data.uniform_(-stdv, stdv)
self.htensor_to_scalars.weight.data.normal_(std=0.01)
Listing 3: Stylized optimization step on the training set for updating elementary parameters
# Perturb hyperparameters around current value in unconstrained # parametrization. batch_htensor = perturb(htensor, hscale)
# Apply necessary reparametrization of hyperparameters. hparam_tensor = hparam_transform(batch_htensor)
# Sets data augmentation hyperparameters in the data loader. dataset.set_hparams(hparam_tensor)
# Get next batch of examples and apply any input transformation # (e.g. input dropout) as dictated by the hyperparameters. images, labels = next_batch(dataset) images = apply_input_transform(images, hparam_tensor)
# Run everything through the model and do gradient descent. pred = hyper_cnn(images, batch_htensor, hparam_tensor) xentropy_loss = F.cross_entropy(pred, labels) xentropy_loss.backward() cnn_optimizer.step()
Listing 4: Stylized optimization step on the validation set for updating hyperparameters/noise scale
# Perturb hyperparameters around current value in unconstrained # parametrization, so we can assess sensitivity of validation # loss to the scale of the noise. batch_htensor = perturb(htensor, hscale)
# Apply necessary reparametrization of hyperparameters. hparam_tensor = hparam_transform(batch_htensor)
# Get next batch of examples and run through the model. images, labels = next_batch(valid_dataset) pred = hyper_cnn(images, batch_htensor, hparam_tensor) xentropy_loss = F.cross_entropy(pred, labels)
# Add extra entropy weight term to loss. entropy = compute_entropy(hscale) loss = xentropy_loss - args.entropy_weight * entropy loss.backward()
# Tune the hyperparameters. hyper_optimizer.step()
# Tune the scale of the noise applied to hyperparameters. scale_optimizer.step()

heading: H SENSITIVITY STUDIES
text: In this section, we present experiments that show how sensitive our STN models are to different meta-parameters.
In particular, we investigate the effect of using alternative schedules (Figure 10) for the number of optimization steps performed on the training and validation sets.
Additionally, we investigate the effect of using different initial perturbation scales for the hyperparameters, which are either fixed or tuned (Figure 11).

Review summaries:
The paper proposes a bilevel optimization approach for hyperparameter tuning. This idea is not new having been proposed in works prior to the current resurgence of deep learning (e.g., Do et al., 2007, Domke 2012, and Kunisch & Pock, 2013). However, the combination of bilevel optimization for hyperparameter tuning with approximation is interesting. Moreover, the proposed approach readily handles discrete parameters.

The paper deals with hyper-parameter optimization of neural networks. The authors formulate the problem as a bilevel optimization problem: minimizing the validation loss over the hyperparameters, subject to the parameters being at the minimum of the training loss. The authors propose an approximation of the so-called best-response function, that maps the hyperparameters to the corresponding optimal parameters (w.r.t the minimization of the training loss), allowing a formulate as a single-level optimization problem and the use gradient descent algorithm. The proposed