Revisiting Random Search for Neural Network Hyperparameter Optimization

The goal of most learning algorithms is to train some set of parameters θ over a set of training data X. However, many of these learning algorithms have associated quantities that are not learned during the course of training, but rather are fixed at the beginning of the training process. These quantities are called hyperparameters (denote them λ). In neural networks, these hyperparameters include the number of hidden units, the regularization penalty, and the method with which the weights are initialized. In this way, we can denote the result of training with hyperparameter choice λ as an output function f = Aλ(X). For neural networks, it is common practice to use stochastic gradient descent (SGD) as the learning algorithm A. However, the methods for finding optimal hyperparameters are not as straightforward. This paper will focus on this outer-loop optimization problem by describing a few existing methods, considering the advantages of each, and reproducing the results for the random search method introduced by Bergstra and Bengio [2]. 1 Problem Characterization The problem of hyperparameter optimization has the aim of minimizing the generalization error, EX∼GX [L(X;Aλ(X))], over the hyperparameter space Λ. This can be expressed as follows: λ∗ = arg min λ∈Λ EX∼GX [L(X;Aλ(X))] (1) An important thing to note is that for any problem in practice, there is no natural distribution but rather a finite set of samples taken from it to construct a training, validation, and test set. We should still consider the existence of a distribution GX however, since in practice any trained classifier may be used on new data not included in the datasets gathered for training and evaluation. This raises the question of how to compute the expectation. The standard technique is cross-validation [4], which is approximating the expectation with the mean over a validation set X that is drawn i.i. d. from GX . As long as the validation set is not seen by the learning algorithm during training (and under the i.i.d. assumption), cross-validation is unbiased. With this in mind, the optimization problem becomes: λ∗ ≈ arg min λ∈Λ 1 |X(valid)| ∑ X∈X(valid) L(X;Aλ(X)) (2) ≡ arg min λ∈Λ Ψ(λ) (3) ≈ arg min λ∈{λ(1),. ,λ(S)} Ψ(λ) (4) We’ll refer to Ψ from equations (3,4) as the hyperparameter response function, and in the future use it to refer to its use both validation and test datasets. Note that equation 4 suggests evaluating the response function at S trial points in the hyperparameter space, and simply choosing the one that minimizes the validation error. This approach is reasonable 2 Ben Rabe, brabe2@illinois.edu because we know very little about the behavior of the response function Ψ , so it cannot be solved in the same manner as more traditional optimization problems. Thus, the crux of the problem is in choosing the set of trial points {λ, . , λ}, methods for which will be described in the next section. 2 Comparison of Methods The first and most straightforward method is a manual search (”grad student descent”) that is effectively choosing values for hyperparameters sequentially based on the intuition of the researcher gleaned from prior experiments. There is essentially no barrier to entry with this method, although more experienced researchers will tune their model more effectively. This method also gives some sense of intuition into the hyperparameter response function Ψ . On the other hand, it is not reproducible or extensible. Another common and intuitive method is a grid search, in which a grid of reasonable hyperparameter values is iterated over in brute force. It is an embarrassingly parallel task, making it simple to implement and feasible if large amounts of resources are available. However, it suffers from the curse of dimensionality in the number of hyperparameters to optimize over. If each of the K hyperparameters may take values in a set L, then the number of trials needed in total is S = ∏K k=1 |L|. While this is the case, grid search was still the standard until random search and bayesian models began to be considered [6]. This brings us to random search. With this method, the value of each hyperparameter in each trial is drawn uniformly (or uniformly in log domain) from the same search space as a grid search. One of the most striking features of random search is its performance in high-dimensional hyperparameter spaces, especially those with low effective dimensionality. Low effective dimensionality is the property that Ψ is more sensitive to changes in some variables than others. In one dimension, for instance, we can think of this property as f(x1, x2) ≈ g(x1), where the function f is more sensitive in changes to x1 than x2. As shown in figure 1, the grid search doesn’t effectively cover the subspace formed by each dimension as it tests the same point in that dimension many times. Fig. 1: Illustration of grid search and random search performance in the case of low effective dimensionality. Taken from [2]. Revisiting Random Search 3 3 Interpreting Results of Random Experiments 3.1 Estimating Generalization The standard for evaluating performance of a model found by cross-validation is to consider the λ that minimizes Ψ (λ) as optimal, and report Ψ (λ) However, as discussed in the previous section, the training, validation, and test datasets are finite samples from an assumed natural distribution (extensible by assuming the inference task will be applied in practice on new data). As such, the test error is not monotone with the validation error i.e. the hyperparameter configuration that minimizes the validation error may not minimize the test error. We’d like to take this uncertainty in our choice of the ”best” model into account so that we not only report a test error, but have some idea of a variance in that error. Note that this is not helpful in a practical sense; choosing the minimizer on the validation data is still our best chance at minimizing test error. However, this can be useful as an analysis tool as we will discuss. A procedure for considering this variation is outlined in [2]. Consider the following definitions: Ψ (λ) = 1 |X(valid)| ∑ X∈X(valid) L(X;Aλ(X)) (5) Ψ (λ) = 1 |X(test)| ∑ X∈X(test) L(X;Aλ(X)) (6) V(λ) = Ψ (valid) (1− Ψ ) |X(valid)| − 1 (7) V(λ) = Ψ (test) (1− Ψ ) |X(test)| − 1 (8) Note that this variance is only valid because the loss we are considering in this case is the error rate which is 0-1 loss, so Bernoulli variance is correct. In light of the fact that X is a finite sample of GX , we can view the test error of the best model in the set {λ, . , λ} as a random variable z. We’ll model this random variable with a Gaussian mixture model with S mixture components. The mean and variance of each component are simply the mean test error and test variance respectively, or μs = Ψ (λ) and σ s = V(λ). For the weights of each component, we’d like some estimate on the probability that that component actually was the best model trained according to validation error. To do so via simulation, we’ll draw hypothetical validation scores Z from a normal distribution with mean Ψ (λ) and variance V(λ), and assign each component a weight ws equal to the fraction of simulation iterations where that component’s score was best. A formal description of the simulation strategy is as follows: ws = P ( Z < Z ′),∀s′ 6= s ) where (9) Z ∼ N (Ψ (λ),V(λ)) (10) Finally, bringing this all together gives the statistics of the random variable z representing the performance of the best model as follows: 4 Ben Rabe, brabe2@illinois.edu