Title: STOCHASTIC HYPERPARAMETER OPTIMIZATION THROUGH HYPERNETWORKS

Abstract: Machine learning models are usually tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of both weights and hyperparameters. Our method trains a neural network to output approximately optimal weights as a function of hyperparameters. We show that our method converges to locally optimal weights and hyperparameters for sufficiently large hypernets. We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.

:

Standard practice in machine learning solves (1) by gradient-free optimization of hyperparameters, such as grid search, random search, or Bayesian optimization. Each set of hyperparameters is evaluated by reinitializing weights and training the model to completion. This is wasteful, since it trains the model from scratch each time, even if the hyperparameters change a small amount. Hyperband (Li et al., 2016) and freezethaw Bayesian optimization (Swersky et al., 2014) resume model training and do not waste this effort. Furthermore, gradient-free optimization scales poorly beyond 10 or 20 dimensions.

How can we avoid re-training from scratch each time?

We usually estimate the parameters with stochastic optimization, but the true optimal parameters are a deterministic function of the hyperparameters λ:

EQUATION

We propose to learn this function. Specifically, we train a neural network with inputs of hyperparameters, and outputs of an approximately optimal set of weights given the hyperparameters.

This provides two major benefits: First, we can train the hypernet to convergence using stochastic gradient descent, denoted SGD, without training any particular model to completion. Second, differentiating through the hypernet allows us to optimize hyperparameters with gradient-based stochastic optimization. The validation loss surface. The approximately optimal weights w φ * are output by a linear model fit at λ. The true optimal hyperparameter is λ * , while the hyperparameter estimated using approximately optimal weights is nearby at λ φ * .

2. TRAINING A NETWORK TO OUTPUT OPTIMAL WEIGHTS:

How can we train a neural network to output approximately optimal weights of another neural network? A neural net which outputs the weights of another neural net is called a hypernet (Ha et al., 2016) . The basic idea is that at each iteration, we ask a hypernet to output a set of weights given some hyperparameters: w = w φ (λ). Instead of updating weights w using the loss gradient ∂L(w) /∂w, we update the hypernet weights φ using the chain rule:

∂L(w φ ) ∂w φ ∂w φ

∂φ . We call this method hyper-training and contrast it with standard training methods in Figure 3 .

We call the function w * (λ) that outputs optimal weights for hyperparameters a best-response function (Fudenberg & Levine, 1998) . At convergence, we want our hypernet w φ (λ) to closely match the best-response function.

2.1. ADVANTAGES OF HYPERNET-BASED OPTIMIZATION:

We can compare hyper-training to other model-based hyperparameter schemes, such as Bayesian optimization. Bayesian optimization (Snoek et al., 2012) builds a model of the validation loss as a function of hyperparameters, usually using a Gaussian process (Rasmussen & Williams, 2006) to track uncertainty. This approach has several disadvantages compared to hyper-training.

First, obtaining data for standard Bayesian optimization requires optimizing models from initialization for each set of hyperparameters. In contrast, hyper-training never needs to fully optimize any one model.

Second, standard Bayesian optimization treats the validation loss as a black-box function:

LValid. (λ) = f (λ). In contrast, hyper-training takes advantage of the fact that the validation loss is a known, differentiable function:

LValid. (λ) = LValid.(wφ(λ)) This removes the need to learn a model of the validation loss. This function can also be evaluated stochastically by sampling points from the validation set.

What sort of parameters can be optimized by our approach? Hyperparameters typically fall into two broad categories: 1) Optimization hyperparameters such as learning rates and initialization schemes, and 2) Regularization or model architecture parameters. Hyper-training does not have inner optimization hyperparameters because there is no inner training loop, so we can not optimize these. We must still choose optimization parameters for the fused optimization loop, but this is also the case for any model-based hyperparameter optimization method. The method can be applied to unconstrained bi-level optimization problems which consist of an inner loss function, inner parameter, outer loss function, and an outer parameter. 11: return λ, w φ ( λ) Figure 3 : A comparison of standard hyperparameter optimization, and our first algorithm. Instead of updating weights w using the loss gradient ∂L(w) /∂w, we update hypernet weights φ using the chain rule:

∂L(w φ ) ∂w φ ∂w φ

∂φ . Also, instead of returning the best hyperparameters from a fixed set, our method uses gradient-based hyperparameter optimization. Here, hyperopt refers to a generic hyperparameter optimization.

2.2. LIMITATIONS OF HYPERNET-BASED OPTIMIZATION:

Hyper-training can handle discrete hyperparameters but does not offer special advantages for optimizing over continuous hyperparameters. The hyperparameters being optimized must affect the training loss -this excludes optimization hyperparameters like learning rate. Also, our approach only proposes making local changes to the hyperparameters and does not do uncertainty-based exploration. Uncertainty can be incorporated into the hypernet by using stochastic variational inference as in Blundell et al. (2015) , but we leave this for future work. Finally, it is not obvious how to choose the distribution over hyperparameters p(λ). If we do not sample a sufficient number of hyperparameters we may reach sub-optimal solutions. We approach this problem in section 2.4.

An obvious difficulty of this approach is that training a hypernet typically requires training several times as many parameters as training a single model. For example, training a fully-connected hypernet with a single hidden layer of H units to output D parameters requires training at least D × H hypernet parameters. Again, in section 2.4 we propose an algorithm that requires training only a linear model mapping hyperparameters to model weights.

2.3. ASYMPTOTIC CONVERGENCE PROPERTIES:

Algorithm 2 trains a hypernet using SGD, drawing hyperparameters from a fixed distribution p(λ). This section proves that Algorithm 2 converges to a local best-response under mild assumptions. In particular, we show that, for a sufficiently large hypernet, the choice of p(λ) does not matter as long as it has sufficient support. Notation as if w φ has a unique solution for φ or w is used for simplicity, but is not true in general.

Theorem 2.1. Sufficiently powerful hypernets can learn continuous best-response functions, which minimizes the expected loss for any hyperparameter distribution.

There exists φ * , such that for all λ ∈ support(p (λ)) , L

Train (w φ * (λ) , λ) = min w L Train (w, λ) and φ * = argmin φ E p(λ ) L Train (w φ (λ ), λ )

Proof. If w φ is a universal approximator (Hornik, 1991) and the best-response is continuous in λ (which allows approximation by w φ ), then there exists optimal hypernet parameters φ * such that for all hyperparameters λ, w φ * (λ) = argmin w LTrain(w, λ). Thus, LTrain(wφ * (λ) , λ) = min w LTrain(w, λ). In other words, universal approximator hypernets can learn continuous bestresponses.

Algorithm 2: Stochastic optimization of hypernet, then hyperparameters

1: initialize φ, λ 2: for T hypernet steps 3: x ∼ Training data, λ ∼ p (λ) 4: φ = φ -α∇ φ LTrain(x, w φ ( λ), λ) 5: for T hyperparameter steps 6: x ∼ Validation data 7: λ = λ -β∇λ LValid.(x, w φ ( λ)) 8: return λ, w φ ( λ)

Algorithm 3: Stochastic optimization of hypernet and hyperparameters jointly 1: initialize φ, λ 2: for T joint steps 3: 

x ∼ Training data, λ ∼ p(λ| λ) 4: φ = φ -α∇ φ LTrain(x, w φ ( λ), λ) 5: 6: x ∼ Validation data 7: λ = λ -β∇λ LValid.(x, w φ ( λ)) 8: return λ, w φ ( λ)

(w φ (λ), λ) for λ ∈ support(p (λ))} if support(p (λ)) is convex and LTrain(w, λ) = Ex∼Train[LPred(x, w)] + LReg(w, λ) with LReg(w, λ) = λ • L(w). Thus, φ * = argmin φ Ep(λ)[LTrain(wφ(λ), λ)].

In other words, if the hypernet learns the best-response it will simultaneously minimize the loss for every point in the support(p (λ)).

Thus, having a universal approximator and a continuous best-response implies for all λ ∈ support(p (λ)), LValid.(wφ * (λ)) = LValid.(w * (λ)) because w φ * (λ) = w * (λ). Thus, under mild conditions, we will learn a best-response in the support of the hyperparameter distribution.If the best-response is differentiable, then there is a neighborhood about each hyperparameter where the best-response is approximately linear. If we select the support of the hyperparameter distribution to be the neighborhood where the best-response is approximately linear then we can use linear regression to learn the best-response locally.

Theorem 2.1 holds for any p (λ). However in practice, we have a limited-capacity hypernet, and so should choose a p (λ) that puts most of its mass on promising hyperparameter values. This motivates the joint optimization of φ and p (λ). Concretely, we can introduce a "current" hyperparameter λ and define a conditional hyperparameter distribution p(λ| λ) which places its mass near λ. This allows us to use a limited-capacity hypernet, at the cost of having to re-train the hypernet each time we update λ.

In practice, there are no guarantees about the network being a universal approximator or the finitetime convergence of optimization. The optimal hypernet will depend on the hyperparameter distribution p(λ), not just the support of this distribution. We appeal to experimental results that our method is feasible in practice.

2.4. JOINTLY TRAINING PARAMETERS AND HYPERPARAMETERS:

Because in practice we use a limited-capacity hypernet, it may not be possible to learn a bestresponse for all hyperparameters. Thus, we propose Algorithm 3, which only tries to learn a bestresponse locally. We introduce a "current" hyperparameter λ, which is updated each iteration. We define a conditional hyperparameter distribution, p(λ| λ), which only puts mass close to λ.

Algorithm 3 combines the two phases of Algorithm 2 into one. Instead of first learning a hypernet that can output weights for any hyperparameter then optimizing the hyperparameters, Algorithm 3 only samples hyperparameters near the current best guess. This means the hypernet only has to be trained to estimate good enough weights for a small set of hyperparameters. The locally-trained hypernet can then be used to provide gradients to update the hyperparameters based on validation set performance.

How simple can we make the hypernet, and still obtain useful gradients to optimize hyperparameters? Consider the case used in our experiments where the hypernet is a linear function of the hyperparameters and the conditional hyperparameter distribution is p(λ| λ) = N ( λ, σ1) for some small σ. This hypernet learns a tangent hyperplane to a best-response function and only needs to make small adjustments at each step if the hyperparameter updates are sufficiently small. We can further restrict the capacity of a linear hypernet by factorizing its weights, effectively adding a bottleneck layer with a linear activation and a small number of hidden units. 

Hyperparameter λ:



Loss:



3. RELATED WORK:

Our work is closely related to the concurrent work of Brock et al. (2017) , whose SMASH algorithm also approximates the optimal weights as a function of model architectures, to perform a gradient-free search over discrete model structures. Their work focuses on efficiently evaluating the performance of a variety of discrete model architectures, while we focus on efficiently exploring continuous spaces of models.

Model-free approaches Model-free approaches only use trial-and-error to explore the hyperparameter space. Simple model-free approaches applied to hyperparameter optimization include grid search and random search (Bergstra & Bengio, 2012) . Hyperband (Li et al., 2016) combines bandit approaches with modeling the learning procedure.

Model-based approaches Model-based approaches try to build a surrogate function, which often allows gradient-based optimization or active learning. A common example is Bayesian optimization (Snoek et al., 2012) . Freeze-thaw Bayesian optimization can condition on partially-optimized model performance.

Optimization-based approaches Another line of related work attempts to directly approximate gradients of the validation loss with respect to hyperparameters. Domke (2012) proposes to differentiate through unrolled optimization to approximate best-responses in nested optimization and Maclaurin et al. (2015a) differentiate through entire unrolled learning procedures. DrMAD (Fu et al., 2016) approximates differentiating through an unrolled learning procedure to relax memory requirements for deep neural networks. HOAG (Pedregosa, 2016) finds hyperparameter gradients with implicit differentiation by deriving an implicit equation for the gradient with optimality conditions. Franceschi et al. (2017) study forward and reverse-mode differentiation for constructing hyperparameter gradients. Also, Feng & Simon (2017) establish conditions where the validation loss of best-responding weights are almost everywhere smooth, allowing gradient-based training of hyperparameters.

A closely-related procedure to our method is the T 1 -T 2 method of Luketina et al. (2016) , which also provides an algorithm for stochastic gradient-based optimization of hyperparameters. The convergence of their procedure to local optima of the validation loss depends on approximating the Hessian of the training loss with respect to parameters with the identity matrix. In contrast, the convergence of our method depends on having a suitably powerful hypernet.

Game theory Best-response functions are extensively studied as a solution concept in discrete and continuous multi-agent games (Fudenberg & Levine, 1998) . Games where learning a bestresponse can be applied include adversarial training (Goodfellow et al., 2014) , or Stackelberg competitions (Brückner & Scheffer, 2011) . For adversarial training, the analog of our method would be a discriminator which is trained while observing all of the generator's parameters.

4. EXPERIMENTS:

In our experiments, we examine the standard example of stochastic gradient-based optimization of neural networks, with a weight regularization penalty. Gradient-based methods explicitly use the gradient of a loss, while gradient-free methods do not (but can use the gradient of a surrogate loss that is learned). Our algorithm may be contrasted with gradient-free techniques like Bayesian optimization, gradient-based methods only handle hyperparameters affecting the training loss and gradient-based methods which can additionally handle optimization parameters. The best comparison for hyper-training is to gradient-based methods which only handle parameters affecting the training loss because other methods apply to a more general set of problems. In this case, the training and validation losses can be written as:

L Train (w, λ) = E x∼Train L Pred (x, w) + L Reg (w, λ) L Valid. (w) = E x∼Valid. L Pred (x, w)

In all experiments, Algorithms 2 or 3 are used to optimize weights of with mean squared error on MNIST (LeCun et al., 1998) with LReg as an L 2 weight decay penalty weighted by exp(λ). The elementary model has 7, 850 weights. All hidden units in the hypernet have a ReLU activation (Nair & Hinton, 2010) unless otherwise specified. Autograd (Maclaurin et al., 2015b) was used to compute all derivatives. For each experiment, the minibatch samples 2 pairs of hyperparameters and up to 1, 000 training data points. Adam was used for training the hypernet and hyperparameters, with a step size of 0.0001. All experiments were run on a 2012 MacBook pro.

4.1. LEARNING A GLOBAL BEST-RESPONSE:

Our first experiment, shown in Figure 1 , demonstrates learning a global approximation to a bestresponse function using Algorithm 2. To make visualization of the regularization loss easier, we use 10 training data points to exacerbate overfitting. We compare the performance of weights output by the hypernet to those trained by standard cross-validation (Algorithm 1). Thus, elementary weights were randomly initialized for each hyperparameter setting and optimized using Adam (Kingma & Ba, 2014) for 1, 000 iterations with a step size of 0.0001.

When training the hypernetwork, hyperparameters were sampled from a broad Gaussian distribution: p (λ) = N (0, 1.5). The hypernet has 50 hidden units which results in 400, 450 parameters of the hypernetwork.

The minimum of the best-response in Figure 1 is close to the true minimum of the validation loss, which shows a hypernet can satisfactorily approximate a global best-response function in small problems.

4.2. LEARNING A LOCAL BEST-RESPONSE:

Figure 5 shows the same experiment, but using the fused updates of Algorithm 3. This results in finding a best-response approximation whose minimum is the true minimum faster than the prior experiment. The conditional hyperparameter distribution is given by p(λ| λ) = N ( λ, 0.00001). The hypernet is a linear model, with only 15, 700 weights. We use the same optimizer as the global best-response to update both the hypernet and the hyperparameters.

Again, the minimum of the best-response at the end of training is the true optimum on the validation loss. This experiment shows that using only a locally-trained linear best-response function can give sufficient gradient information to optimize hyperparameters on a small problem. Algorithm 3 is also less computationally expensive than Algorithms 1 or 2.

4.3. HYPER-TRAINING AND UNROLLED OPTIMIZATION:

In order to compare hyper-training with other gradient-based hyperparameter optimization we trained a model with 7, 850 hyperparameters with a separate L 2 weight decay applied to each weight in a 1 layer (linear) model. The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiment. The weights for the model are factorized by selecting a hypernet with 10 hidden units. The factorized linear hypernet has 10 hidden units giving 164, 860 weights. This means each hypernet iteration 2 • 10 times as expensive as an iteration on just the model, because there is the same number of hyperparameters as model parameters. Hyper-training reaches sub-optimal solutions because of limitations on how many hyperparameters can be sampled for each update but overfits validation data less than unrolling. Standard Bayesian optimization cannot be scaled to this many hyperparameters. Thus, this experiment shows Algorithm 3 can effectively partially optimize thousands of hyperparameters.

4.4. OPTIMIZING WITH DEEPER NETWORKS:

In order to to see if we can optimize deeper networks with hyper-training we optimize models with 1, 2, and 3 layers with a separate L 2 weight decay applied to each weight. The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiment. The weights for each model are factorized by selecting a hypernet with 10 hidden units. Again, standard Bayesian optimization cannot be scaled to this many hyperparameters.

Figure 6 , right, shows that Algorithm 3 can scale to networks with multiple hidden layers and outperform hand-tuned settings. As more layers are added the difference between validation loss and testing loss decreases, and the model performs better on the validation set. Future work should compare other architectures like recurrent or convolutional networks. Additionally, note that more layers perform with lower training (not shown), validation, and test losses, as opposed to lower training loss but higher validation or test loss. This indicates that using weight decay on each weight could be a prior for generalization, or that hyper-training enforces another useful prior like the continuity of a best-response.

4.5. ESTIMATING WEIGHTS VERSUS ESTIMATING LOSS:

As mentioned above, our approach differs from Bayesian optimization in that we try to learn to predict optimal weights, while Bayesian optimization attempts to directly model the validation loss of optimized weights. In this final experiment, we untangle the reason for the better performance of our method: Is it because of a better inductive bias, or because our method can see more hyperparameter settings during optimization?

First, we constructed a hyper-training set: We optimized 25 sets of weights to completion, given randomly-sampled hyperparameters. We chose 25 samples since that is the regime in which we expect Gaussian process-based approaches to have the largest advantage. We also constructed a validation set of 10, 215 (optimized weight, hyperparameter) tuples generated in the same manner.

We then fit a Gaussian process (GP) regression model with an RBF kernel from sklearn on the hyper-training data. A hypernet is fit to the same dataset. However, this hypernet was trained to 0.6 0.7 0.8 0.9 1.0 1.1 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1

GP Mean:

Predicted Loss 0.6 0.7 0.8 0.9 1.0 1.1 0.4 0.5 0.6 0.7 0.8 0.9 1.0 7 : Comparing three approaches to predicting validation loss. First row: A Gaussian process, fit on a small set of hyperparameters and the corresponding validation losses. Second row: A hypernet, fit on the same small set of hyperparameters and the corresponding optimized weights. Third row: Our proposed method, a hypernet trained with stochastically sampled hyperparameters. Left: The distribution of predicted and true losses. The diagonal black line is where predicted loss equals true loss. Right: The distribution of differences between predicted and true losses. The Gaussian process often under-predicts the true loss, while the hypernet trained on the same data tends to over-predict the true loss. fit optimized training weights, not optimized validation loss. Finally, we optimize another hypernet using Algorithm 2, for the same amount of time as building the hyper-training set. The two hypernets were linear models and were trained with the same optimizer parameters as the 7, 850-dimensional hyperparameter optimization.

Figure 7 shows the distribution of prediction errors of these three models. We can see that the Gaussian process tends to underestimate loss. The hypernet trained with the same small fixed set of examples tends to overestimate loss. We conjecture that this is due to the hypernetwork producing bad weights in regions where it doesn't have enough training data. Because the hypernet must provide actual weights to predict the validation loss, poorly-fit regions will overestimate the validation loss. Finally, the hypernet trained with Algorithm 2 produces errors tightly centered around 0. The main takeaway from this experiment is a hypernet can learn more accurate surrogate functions than a GP for equal compute budgets because it views (noisy) evaluations of more points.

Code for all experiments will be made available upon publication.

5. CONCLUSIONS:

In this paper, we:

• Presented algorithms that efficiently learn a differentiable approximation to a best-response without nested optimization.

• Showed empirically that hypernets can provide a better inductive bias for hyperparameter optimization than Gaussian processes fit directly to the validation loss.

• Gave a theoretical justification that sufficiently large networks will learn the best-response for all hyperparameters it is trained against.

We hope that this initial exploration of stochastic hyperparameter optimization will inspire further refinements, such as hyper-regularization methods, or uncertainty-aware exploration using Bayesian hypernetworks.

Fabian Pedregosa. Hyperparameter optimization with approximate gradient. A separate L 2 weight decay is applied to the weights of each digit class, resulting in 10 hyperparameters. The weights w φ * are output by the hypernet for current hyperparameter λ, while random losses are for the best result of a random search. Hypernetwork-based optimization converges faster than random search or Bayesian optimization. We also observe significant overfitting of the hyperparameters on the validation set, which may be reduced be introducing hyperhyperparameters (parameters of the hyperparameter prior). The runtime includes the inner optimization for gradient-free approaches so that equal cumulative computational time is compared for each method.

Factors affecting this include removing the overhead of constructing tuples of hyperparameters and optimized weights, viewing more hyperparameter samples, or having a better inductive bias from learning weights.



Figures and Tables:

Figure 1: Training and validation loss of a neural net, estimated by crossvalidation (crosses) or by a hypernet (lines), which outputs 7, 850-dimensional network weights. The training and validation loss can be cheaply evaluated at any hyperparameter value using a hypernet. Standard cross-validation requires training from scratch each time.

Figure2: A visualization of exact (blue) and approximate (red) optimal weights as a function of given hyperparameters. Left: The training loss surface. Right: The validation loss surface. The approximately optimal weights w φ * are output by a linear model fit at λ. The true optimal hyperparameter is λ * , while the hyperparameter estimated using approximately optimal weights is nearby at λ φ * .

= hyperopt . . . , λ(i) , LValid. w (i) -α∇ w LTrain(x, w, λ) 7: λ i , w i = λ, w 8: for i = 1, . . . , T outer 9: if LValid. w (i) < LValid.(w) then Training data, λ ∼ p (λ) 6: φ = φ -α∇ φ LTrain(x, w φ (λ), λ) -β∇λ LValid.(x, w φ ( λ))

Figure 4: A side-by-side comparison of two variants of hyper-training. Algorithm 3 fuses the hypernet training and hyperparameter optimization into a single loop of SGD. Substituting φ * into the training loss gives Ep(λ)[LTrain(wφ * (λ), λ)] = Ep(λ)[minφ LTrain(wφ(λ), λ)]. By Jensen's inequality, min φ Ep(λ)[LTrain(wφ(λ), λ)] ≥ Ep(λ)[minφ LTrain(wφ(λ), λ)] where min φ is a convex function on the convex vector space of functions {L Train (w φ (λ), λ) for λ ∈ support(p (λ))} if support(p (λ)) is convex and LTrain(w, λ) = Ex∼Train[LPred(x, w)] + LReg(w, λ) with LReg(w, λ) = λ • L(w). Thus, φ * = argmin φ Ep(λ)[LTrain(wφ(λ), λ)]. In other words, if the hypernet learns the best-response it will simultaneously minimize the loss for every point in the support(p (λ)).

Train loss of optimized weights Train loss of hypernet weights Valid. loss of optimized weights Valid. loss of hypernet weights Optimal hyperparameter λ p(λ| λ)

Figure 5: The training and validation losses of a neural network, estimated by cross-validation (crosses) or a linear hypernet (lines). The hypernet's limited capacity makes it only accurate where hyperparameter distribution put mass.

Figure 6: Validation and test losses during hyperparameter optimization with a separate L 2 weight decay applied to each weight in the model. Thus, models with more parameters have more hyperparameters. Left: The 7, 850 dimensional hyperparameter optimization problem from having a linear model is solved with multiple algorithms. Hypernetwork-based optimization converges faster than unrolled optimization from Maclaurin et al. (2015a) but to a sub-optimal solution. Right: Hyper-training is applied different layer configurations in the model. The hand-tuned regularization parameters on the 784-10, 784-100-10, and 784-100-100-10 models have a validation losses of 0.434, 0.157 and 0.206 respectively.

Figure6, left, shows that Algorithm 3 converges more quickly than the unrolled reverse-mode optimization introduced inMaclaurin et al. (2015a)  with an implementation byFranceschi et al. (2017). Hyper-training reaches sub-optimal solutions because of limitations on how many hyperparameters can be sampled for each update but overfits validation data less than unrolling. Standard Bayesian optimization cannot be scaled to this many hyperparameters. Thus, this experiment shows Algorithm 3 can effectively partially optimize thousands of hyperparameters.

Figure8: Validation and test losses during hyperparameter optimization. A separate L 2 weight decay is applied to the weights of each digit class, resulting in 10 hyperparameters. The weights w φ * are output by the hypernet for current hyperparameter λ, while random losses are for the best result of a random search. Hypernetwork-based optimization converges faster than random search or Bayesian optimization. We also observe significant overfitting of the hyperparameters on the validation set, which may be reduced be introducing hyperhyperparameters (parameters of the hyperparameter prior). The runtime includes the inner optimization for gradient-free approaches so that equal cumulative computational time is compared for each method.

In International Conference on Machine Learning, pp. 737-746, 2016. Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning, volume 1. MIT press Cambridge, 2006. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012. Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint arXiv:1406.3896, 2014. A EXTRA EXPERIMENTS A.1 OPTIMIZING 10 HYPERPARAMETERS Here, we optimize a model with 10 hyperparameters, in which a separate L 2 weight decay is applied to the weights for each digit class in a linear regression model to see if we can optimize medium-sized models. The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiments. A linear hypernet is used, resulting in 86, 350 hyper-weights. Algorithm 3 is compared against random search and .Figure8, right, shows that our method converges more quickly and to a better optimum than either alternative method, demonstrating that medium-sized hyperparameter optimization problems can be solved with Algorithm 3.Hypernet Valid (w *( ))

Table content: <table><tr><td/><td/><td colspan="4">Optimizing 10 hyperparameters</td></tr><tr><td/><td>0.050</td><td/><td/><td colspan="2">Random search Valid</td></tr><tr><td/><td>0.048</td><td/><td/><td colspan="2">Bayesian opt. Valid.</td></tr><tr><td/><td>0.046</td><td/><td/><td colspan="2">Random search Test</td></tr><tr><td>Loss</td><td>0.044</td><td/><td/><td colspan="2">Bayesian opt. Test Hypernet Test (w *( ))</td></tr><tr><td/><td>0.042</td><td/><td/><td/></tr><tr><td/><td>0.040</td><td/><td/><td/></tr><tr><td/><td>0.038</td><td>0</td><td>25</td><td>50 Runtime in seconds 75</td><td>100 125</td></tr></table>

Peer review (iclr2018):
 *Summary* The paper proposes to use hyper-networks [Ha et al. 2016] for the tuning of hyper-parameters, along the lines of [Brock et al. 2017]. The core idea is to have a side neural network sufficiently expressive to learn the (large-scale, matrix-valued) mapping from a given configuration of hyper-parameters to the weights of the model we wish to tune. The paper gives a theoretical justification of its approach, and then describes several variants of its core algorithm which mix the training of the hyper-networks together with the optimization of the hyper-parameters themselves. Finally, experiments based on MNIST illustrate the properties of the proposed approach. While the core idea may appear as appealing, the paper suffers from several flaws (as further detailed afterwards): -Insufficient related work -Correctness/rigor of Theorem 2.1 -Clarity of the paper (e.g., Sec. 2.4) -Experiments look somewhat artificial -How scalable is the proposed approach in the perspective of tuning models way larger/more complex than those treated in the experiments? *Detailed comments* -"...and training the model to completion." and "This is wasteful, since it trains the model from scratch each time..." (and similar statement in Sec. 2.1): Those statements are quite debatable. There are lines of work, e.g., in Bayesian optimization, to model early stopping/learning curves (e.g., Domhan2014, Klein2017 and references therein) and where training procedures are explicitly resumed (e.g., Swersky2014, Li2016). The paper should reformulate its statements in the light of this literature. -"Uncertainty could conceivably be incorporated into the hypernet...". This seems indeed an important point, but it does not appear as clear how to proceed (e.g., uncertainty on w_phi(lambda) which later needs to propagated to L_val); could the authors perhaps further elaborate? -I am concerned about the rigor/correctness of Theorem 2.1; for instance, how is the continuity of the best-response exploited? Also, throughout the paper, the argmin is defined as if it was a singleton while in practice it is rather a set-valued mapping (except if there is a unique minimizer for L_train(., lambda), which is unlikely to be the case given the nature of the considered neural-net model). In the same vein, Jensen*s inequality states that Expectation[g(X)] >= g(Expectation[X]) for some convex function g and random variable X; how does it precisely translate into the paper*s setting (convexity, which function g, etc.)? -Specify in Alg. 1 that "hyperopt" refers to a generic hyper-parameter procedure. -More details should be provided to better understand Sec. 2.4. At the moment, it is difficult to figure out (and potentially reproduce) the model which is proposed. -The training procedure in Sec. 4.2 seems quite ad hoc; how sensitive was the overall performance with respect to the optimization strategy? For instance, in 4.2 and 4.3, different optimization parameters are chosen. -typo: "weight decay is applied the..." --> "weight decay is applied to the..." -"a standard Bayesian optimization implementation from sklearn": Could more details be provided? (there does not seem to be implementation there http://scikit-learn.org/stable/model_selection.html to the best of my knowledge) -The experimental set up looks a bit far-fetched and unrealistic: first scalar, than diagonal and finally matrix-weighted regularization schemes. While the first two may be used in practice, the third scheme is not used in practice to the best of my knowledge. -typo: "fit a hypernet same dataset." --> "fit a hypernet on the same dataset." -(Franceschi2017) could be added to the related work section. *References* (Domhan2014) Domhan, T.; Springenberg, T. & Hutter, F. Extrapolating learning curves of deep neural networks ICML 2014 AutoML Workshop, 2014 (Franceschi2017) Franceschi, L.; Donini, M.; Frasconi, P. & Pontil, M. Forward and Reverse Gradient-Based Hyperparameter Optimization preprint arXiv:1703.01785, 2017 (Klein2017) Klein, A.; Falkner, S.; Springenberg, J. T. & Hutter, F. Learning curve prediction with Bayesian neural networks International Conference on Learning Representations (ICLR), 2017, 17 (Li2016) Li, L.; Jamieson, K.; DeSalvo, G.; Rostamizadeh, A. & Talwalkar, A. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization preprint arXiv:1603.06560, 2016 (Swersky2014) Swersky, K.; Snoek, J. & Adams, R. P. Freeze-Thaw Bayesian Optimization preprint arXiv:1406.3896, 2014 ********* Update post rebuttal ********* I acknowledge the fact that I read the rebuttal of the authors, whom I thank for their detailed answers. My minor concerns have been clarified. Regarding the correctness of the proof, I am still unsure about the applicability of Jensen inequality; provided it is true, then it is important to see that the results seem to hold only for particular hyperparameters, namely regularization parameters (as explained in the new updated proof). This limitation should be exposed transparently upfront in the paper/abstract. Together with the new experiments and comparisons, I have therefore updated my rating from 5 to 6.