Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space
Authors: Vlad Pushkarov
Yonathan Efroni
Mykola Maksymenko
Maciej Koch-Janusz
Venue:  (2020)
Keywords: deep learning
Hyperparameter optimization
dropout

Abstract: Hyperparameter optimization is both a practical issue and an interesting theoretical problem in training of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at a negligible additional computational cost, results can be improved by sampling \emph{nonlocal paths} instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in training, which can be mapped to an effective temperature. The usually independent instances of the model are coupled and allowed to exchange their hyperparameters throughout the training using the well established parallel tempering technique of statistical physics. Each simulation corresponds then to a unique path, or history, in the joint hyperparameter/model-parameter space. We provide empirical tests of our method, in particular for dropout and learning rate optimization. We observed faster training and improved resistance to overfitting and showed a systematic decrease in the absolute validation error, improving over benchmark results.

Under review as a conference paper at ICLR 2020
TRAINING DEEP NEURAL NETWORKS BY OPTIMIZING OVER NONLOCAL PATHS IN HYPERPARAMETER SPACE
Anonymous authors Paper under double-bling review
ABSTRACT
Hyperparameter optimization is both practical issue and an interesting theoret- ical problem in training of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at a negligible additional computational cost, results can be improved by sampling nonlocal paths instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in train- ing, which can be mapped to effective temperature. The usually independent instances of the model are coupled and allowed to exchange their hyperparameters throughout the training using the well established parallel tempering technique of statistical physics. Each simulation corresponds then to a unique path, or history, space. We provide empirical tests of
in the joint ppperparrmmtte/modell-aaaaeeee our method, in particular for dropout and learning rate optimization. We observed faster training and improved resistance to overfitting and showed a systematic de- crease in the absolute validation error, improving over benchmark results.
INTRODUCTION
The remarkable improvement in performance machine learning models has been paid for with an increased model complexity. Part of this complexity is directly related architectural choices, such as model topology and depth or increased overall number of parameters (weights), but an increasingly important aspect, from both theoretical and practical standpoint, number of hyperparameetrs. Though the classification may seem fluid, with structural features such as the number of layers being treated as hyperparameters as well, the difference is laid bare by the distinct nature of training, which involves double loop of optimizations. The inner loop finds the best model weights with hyperparameters held fixed to some arbitrary values using the performance on the training dataset, while the outer loop is responsible for finding the optimal value separate
the growth of the
of hyperparameters to ensure a robust performance on new data, i.e. generalization. dataset is used for this key task. Hyperparameters thus optimized may include structural features, regularization constants, and even dynamical characteristics of the training algorithm.
The practical importance of hyperparameters cannot be underestimated: the very same architecture may deliver state-of-the-art or mediocre results, depending on the choices made. Simultaneously as the number of hyperparameters grows, the search space becomes high-dimensional and thus ne- cessitates proper hyperparameter optimization (HO) procedures, which go beyond rule-of-thumb choices. This additionally complicated by the fact that computational resources and time are usu- ally limited. A number of HO approaches have been developed (Bergstra et al.) (01 T); the simplest ones involve grid- or random searches, but their main weakness is poor scaling and failure reuse information obtained from previous parameter choices. A broad family of methods addressing the latter issue arise in the Bayesian framework: here the the assignment of new hyperparameter val- ues to be evaluated is informed by the performance of the model with the previous assignments (Snoek et al. 2012). The high computational cost of evaluation inspires additional strategies, for instance bandit methods, to leverage cheaper but cruder results obtained with partial datasets. All of the above HO strategies, however, share a key characteristic: optimization of hyperparameters is decoupled from that of the weights, and the procedures amount to more or less clever sampling of the points in hyperparameter space, using the validation dataset only. A family of genetic HO algo- rithms overcome this issue by optimization in both weights and hyperparameter space, however, in

---

Under review as a conference paper at ICLR 2020
this case, in a greedy way (see for Daderberr al. (2017)) They cannot be thus guaranteed to explore space in an unbiased way, and the results may explicitly depend on initial conditions..
Here, propose a radically different strategy: that of coupling the usually independent copies of the model at different hyperparameter values during the training. The instances of the model are allowed to exchange their hyperparametes progresses, in effect tracing out a highly nonlocal trajectory in the product of hyperparameter and weight spaces. The final trained model cannot thus be thought of as being characterized by par- ticular point value of the hyperparameters, but rather by path or history. The exchange procedure based on the physical technique of parallel tempering (Swendsen & Wang 1986), which itself depends on mapping of the hyperparameter values to an effective "temperature" of the model. We demonstrate empirically, that this HO approach which leverages also the training dataset has number of desirable properties: it results in models more resilient to overfitting, achieving smaller overall errors, and achieving them faster. The method naturally parallelizable and the computational cost overhead over usual grid searches is negligible.
the optimization of the weights the training dataset
The goal of this work then twofold: and intuitive physical perspective on various types of hyperparameters, which can be interpreted as setting the level of "noise" during the training of the model. This, in turn, defines an effective pa- rameter- similar in spirit to temperature controlling the smoothness of the generalization function. Second, and much more important in applications, is to show that this perspective allows to move be- yond the usual paradigm of two decoupled weight/hyperparametee optimizations in well-motivated and controlled fashion, and ultimately to train more robust models.
first, to provide (or, in some instances, recapitulate) a unifying
This paper is organized as follows: in section 2 we review the interpretation of hyperparameters as controls of noise, and the mapping to an effective "temperature". In section 3 we introduce the main concepts of our path- based model optimization. In section 4 we show numerical tests of the approach on neural nets trained on EMNIST and CIFAR-10 datasets. The discussed examples of hyperparameters include the learning rate and drop-out rate, among others. Finally, in section 5 we discuss the implications, as well as potential generalizations of the method.
2 HYPERPARAMETERS AS CONTROLS OF SMOOTHNESS OF POTENTIAL LANDSCAPE
Training of modern machine learning models, in particular of deep neural networks (DNNs), is complex non-convex optimisation problem of minimizing the total objective function (otherwise called loss or energy):
1 N Σ 4xx, W),
[(x, W)
where 4xx;, W) captures deviations of the predictions of the model, parametrized by weights W, for the i-th data point x; in the training set of total size N. The standard choice of the optimisation method for this problem remains the Stochastic Gradient Descent (SGD) algorithm
W W.
7
Σ
Vw4(x; W.)
NB
VENs
with step size 7 and the gradient computed only on a batch Ng of the training set.
It is well established, that various forms of direct noise injection, whether to the inputs, weights or gradients, aid generalization performance (Sietsma Dow 1991 Neelakantan et al. 2015). From the theory point of view it has been shown that the effect of such methods is analogous - though not identical - to introducing particular types of regularization (Bishop) 1995 An 1996) smoothing the cost function. Furthermore, an often invoked intuition is that noise allows the algorithm to escape narrow minima of the cost function defined by the training data, thereby preventing overfitting. This is also true for non-white noise introduced "indirectly", such the one generated by the mini- batches in Eq 2 Aided by the results suggesting poor generalization for "deep" (Choromanska et al. 2015). , and, conversely, good generalization for "flat" minima of the loss function (Hinton &
van Camp 1993 Hochreiter & Schmidhuber 1995). was postulated in Zhang et al.] (2018) that strong performance of SGD is, in fact, due to the correlated nature of the induced batch noise.
2

---

Under review as a conference paper at ICLR 2020
75
10
10
oropoeig- 2400 2.30 qua (9%
Leaningg hasa Ty- 633
Lo Cardon CALAY (EAL
25c 53K
300C 125K 1938 175K 2006 Mat-balii Stass
25c 7SC 2006 125K
153K 173M 2006
25k 50k
15OK 175c 200K Min- Steps
10k 125K
Staes
Figure 1: Diffusion curves for model weights (calculated as Euclidean distance between weights vector initial time to and the vector at epoch f) for replicas of the model at different rate of Dropout, learning rate and 1.2 regularization. See discussion in the main text for details.
In what follows, we will greatly benefit from an intuitive if not always rigorous picture relating various kinds of noise to an effective parameter controlling smoothness of the potential landscape, similar in spirit to temperature. Common hyperparameters, such as the learning rate, dropout or batch size, can be treated on the same footing, as we now demonstrate.
Let us begin with the prototypical case of added Langevin noise, for which the temperature anal- ogy is exact (Seung al. [992]: the gradients are corrupted by white noise & with variance (E)(t))))))) 2T 21,0:, The naive GD can then be viewed as a diffusion process of particle in a complex potential landscape, with the weight updates a discretized version of the continuum equation:
aw
= - V + E(t).
(3)
0t
At long-times Eq. g converges to Gibbs probability distribution, from which weights W can be directly sampled:
P(W) 2 exp exp(
(4)
The inverse temperature 3 = 1/T is not arbitrary, but rather is defined by the variance of the noise Ş. Furthermore, the multiplicative prefactor 3 in Eq. coontolls the overall scale of variation of the potential landscape. Strong noise, corresponding to large variance, and therefore high temperature T, results in small 3 which "flattens" the landscape of L (W). Conversely, weak noise results in large B. amplifying potential differences. This is the central idea behind changing temperature in the familiar simulated annealing method (Kirkpatrick et al. 1983). An alternative way of phrasing the above is that the increased variance of the noise does not allow the finer details the potential landscape to be explored by the dynamics, effectively smoothing it. The crucial observation is that the latter perspective extends also to cases where the noise is not white. Though, strictly speaking, the variance of the noise is not the temperature anymore, it still performs an analogous function: effectively controls the roughness/smoottnes of the landscape. An immediate consequence is that higher variance facilitates diffusion of the model in such nigh-dimensional space, and improves ergodicity (see Fig.
As illustration, we consider common examples of hyperparameters from this perspective, and examine their influence on diffusion curves. For the learning rate y, its magnitude directly related to the size of the time step in the discretization of the Langevin equation, and by dimensional analysis it is analogous to increasing the noise variance (thus temperature) by a factor of y. In the case of SGD dynamics Eq. 2 the noise variance can be computed explicitly (Zhang et al., 2018). Even though the noise is highly correlated, i.e. not thermal, the variance scales overall as inverse batch size: [NV.l- and thus smaller batch size smoothens the potential. Similarly, dropout regularization has the effect of increasing the variance of neural outputs at training by factor of inverse dropout retention rate p (Hinton et al. 2012), therefore amplifying noise and improving diffusion, as seen in Fig. [ The case of L.2 regularization is different: an increased L2 naturally diminishes the magnitude of the potential landscape variations, and so the initial diffusion is faster (see Fig. []] however at later times the diffusion is effectively suppressed; the plateau value reached is, naturally, the lower the stronger the L2 regularization. Since, therefore, the models do not diffuse at different rates, L2 does not satisfy the basic requirements of our procedure, and do not expect systematic improvements for this hyperparameter. On the other hand, for parameters relating to direct noise injection (for instance gradients or weights), the relationship of large magnitude to
3

---

Under review as a conference paper at ICLR 2020
high "temperature" is natural, as described above, and their smoothing effect has been noted in the literature An (1996).
Since generically there are many hyperparameters, and results or intuitions such as above may not be readily available, it is imperative identify whether they are related to landscape smooth- ness. This is, fortunately, possible. The functional sig- nificance of the notion of smoothness lies precisely in determining the diffusion properties of the random walk, therefore it is natural to test for smoothing properties by performing diffusion experiment. To wit, sensitivity of the weight diffusion curves obtained from SGD training runs the value of hyperparameter in question implies relation to smoothness. To exemplify this we consider Batch Normalization (BN), By studying Lipschitz prop- erties Santurkar al. (2018) ultimately confirmed it is related to effective landscape smoothing. Alternatively,
be able to systematically
Diffission
LenetS Lanatz EN
200K
300K
400K
500K
600K
700K
Mini Biti-BBachhs Steps
Figure 2: Weight diffusion curves with and without Batch Normalization(BNN
comparing training with and without BN, it can be immediately shown improve weight diffusion (see Fig 2)
Training the model with different hyperparameters thus be intuitively thought of minimizing the objective at different intensities of noise, or, equivalently, in effective landscapes of varying degree of smoothness. We will now use this insight to construct new nonlocal hyperparameter optimization procedure.
REPLICA EXCHANGE OF HYPERPARAMETERS
Algorithm Training with replica exchange
INPUT:
Number of replicas M, Inverse "temperature" (hyperparameters)) 3 (Bi, , Number of steps for initialization AN, Number of SGD steps between exchanges AN
Exchange normalization parameter C'
Number of steps T
OUTPUT: Weight configurations W
(W1, W W WM) of the replicas, 1: Initialization: vk E M, initialize weights We for each replica and set = 0.
2: Vk M, perform SGD for AN; steps. Update
1 at each step.
3: Repeat:
4: Vk M. perform SGD for AN, steps to update W.. Set + each step.
5: Let L, = (C (Wi) . (W2) .... L (W)) be validation losses at time t.
6: Randomly select a pair (m, ) of replicas with adjacent temperatures.
7: if A CB3m Bx) [C (W... m) L (W,)) S then
8:
swap 3m and 3n
9: else
10:
swap 3m and B, with probability exp (-A). Finish if T.
11:
The core of our approach is endow the model with the ability change hyperparameters during training, and hence to optimize the model over a subset of paths in the combined weight and hy- perparameter space, as opposed to being confined to hyperplanes determined by fixed value of the hyperparameter. In the previous section we have noted that varying typical hyperparameters defines a family of optimization problems of monotonically changing smoothness of the effective landscape, and thus of ergodicity properties and complexity. This family is labelled by effective "tempera- ture" determined by the variance the noise induced in the training by particular hyperparameter choices.
Inspired by similar problems in statistical physics we use the parallel tempering (replica exchange) method (Swendsen & Wang. 1986). In this procedure, multiple Markov Chain Monte-Carlo
4

---

Under review as a conference paper at ICLR 2020
(MCMC) simulations, or replicas, are run in parallel at different temperatures, which define the levels of uncertainty in the objective function, ie. the energy. The temperatures are arranged in ascending manner forming ladder which the states of the replicas are swapped between neigh- boring values with Metropolis-Hastingg detailed balance not only for each chain individually, but also between the chains (see discussion below). The lower temperature chains' ergodicity is thus radically improved by the possibility of temporarily performing MC moves at higher temperatures, where the landscape looks flatter. Note that this results in non- non-greedy and highly nonlocal exploration. Indeed, parallel tempering is ef- ficient in systems with broken ergodicity the usual case for complex energy landscapes, where configuration space can effectively become partitioned into separate regions with low probability for inter-region transitions (for a concise review see Earl & Deem (2005). Applications beyond the original spin-glass simulations include protein folding (Fukunishi et al., 2002 and, in machine learning, training of Boltzmann machines (Desjardins et al. 2010).
acceptance criteria, ensuring that the whole setup satisfies
In our algorithm, the MCMC chains for individual replicas are replaced with multiple instances of the standard SGD dynamics distinguished by hyperparameter values, which is a more efficient way to approach Langevin-like equation for systems with large number of long-range correlated parameters. We assume, using the noise analogy discussed in the previous section, that the weight configurations W of the replicas follow Gibbs distribution P.,. (W) characterized by the effective inverse "temperature" Bm. determined by the value of the hyperparameter:
Pm (W)
exp (- (W)),
(5)
Zm
with Zm the partition function. The total system distribution is then a product over the M replicas:
M
Piull = P.W.)
The transition probability P P(W. Am ,,) of swapping configurations W and W between two replicas m and n should satisfy the detailed balance condition in the full system, which implies:
P (W, Bm; W (..))
(W', Bng W, Bm.) exp (-A),
where we used Gibbsianity and where:
(8) Note that the global detailed balance condition guarantees that in the long-time limit the joint prob- ability distribution Eq. 6 will be sampled faithfully. In particular, no dependence on initial state of the weights will survive. Following the standard Cetropolis-Hastinne with analytically computable acceptance probability:
A = (8m - B,) (W') [W)]
scheme, the exchange occurs
P (W, 9m, W B.)) = for A S O,
exp (- for A>0 0.
The resulting Algorithm 1 is described in the table above. It accepts M replicas of the system, a vector 3 of inverse temperatures, number of initialization and SGD steps between exchanges. Each one of the M realisations of the system is first run for AN, steps, until they achieve their relative equilibrium. Subsequently, exchanges are proposed every AN, steps. The constant C' introduced in the acceptance ratio is responsible for normalization of the exponential argument and may be required when the values of parameer are very low, or too high (see below).
An important practical issue is the selection of replica temperatures and spacing, so that exchange proposals are accepted with high probability. The theoretical answer depends on the properties of the energy landscape and fluctuations in the system; the basic idea is to ensure that energy his- tograms for neighbouring replicas have sufficiently large overlap. To this end, common choice for temperature ladder is a geometric progression. In what follows, for simplicity, we use an equal spacing between replicas and instead scale with a tunable parameter C' to control the exchange rate, which requires calibration. For such tuning, running of preliminary realisations of the system prior to a parallel tempering optimisation may be needed to approximate energy histograms. Here we run short training without exchanges to tune the acceptance ratio constant C, or to fine-tune the temperatures selection. is also worth noting that the acceptance ratio relates to the number of trainable parameters in the model. Increase in the number of parameters allows for more accessible states in parameter space, and induces smaller acceptance ratio.
5

---

Under review as a conference paper at ICLR 2020
25
32
Error (%6
Error (%)
23
30
(0.35..... 0.47)
YE (0.002.... 0.018)
m0.53
Y Y'00.002)
20K
40K
COK
80K
100K
120K
140K
28
20K
40K
6OK
BDK
100K
120K
140K
Mini-Batch Steps
Mini-Batch Stops
8.0
Enror (36
Enror (96)
7.5
(0.12.....
VE (O.006,
0.012)
p" 0.00
5
50K
100K
150K
200K
250K
200K
350K
7.0
50K
100K
150K
200K
250K
300K
350k
Mini-Batch Steps
Steps
Figure 3: Test error curves for LeNet-like architecture for replica exchange training over Dropout and Ieamn ing rate Y In the upper/lower row results for EMNIST CIFAR-10 datasets, The dark lines are running averages.
Another input argument is the swap step; here an important observation is that when parallel tempering used for MCMC sampling purposes, is necessary to satisfy the detailed bal- ance condition. For too small a step the system may not equi- librate sufficiently fast, and hence will not obey the Markov property.
Table 1: Architecture for EMNIST
Layer
Size
Activation
Inpul
32 x 32
Convolation
28 28
tanh
Avg Pooling
14 14
lanh
Convolation
10x 10x10x110
16
tanh
Avg Pooling
3
tanh
Convolation
1x
120
tanh
Fully Connected
tanh
EMPIRICAL RESULTS
Fully Connected
26
RBF
To validate the proposed approach we conducted a series of experiments: first, using small LeNet-like models, inves- tigated the effects various types of iyperparamettr-induuce "noise" weight diffusion and tested the associated idea of hyperparameter replica exchange. We then moved to deep ResNet architectures He al. (2016) verify applicability of our parallel-tempeerng--aase based algorithm to large scale models.
Table 2: Architecture for CIFAR-10
Layer
Activation
Input
32 x 32
Convolation
28 x 28
relu
Max Pooling
14 14 x
Convolation
10x
Max Pooling
5 5 16
Convolution
Fully Connected
84
relu
The expriments were performed on EMNIST-letters Cohen
Fully Connected
et al. (2017)) and CIFAR-10 (Krizhevsky & Hinton 2009)) datasets. EMNIST-letters consists of 124800 training images
and 20800 testing images of shape 28 x 28 pixels with 26 classes. CIFAR-10 has 50000 training images and 10000 test-
ing images with 10 classes. Each image has shape 32 x 32
pixels and 3 channels. The validation splits for both datasets were generated by random sampling of training dataset taking 10% out. All models were trained with mini-batch size of 128.
The LeNet-like models are summarized in tables [ and 2 32 x 32 pixels before the inputs are fed to the network. Each average pooling layer is multiplied by a learnable coefficient and added bias (one per feature map). The neurons between the average pooling layer and convolutional layer fully connected (every feature map from average pooling
For EMNIST we apply zero-pade aero-padding to

---

Under review as a conference paper at ICLR 2020
is connected every convolutional feature map), opposed to the original LeNet architecture from LeCun et al. (1998). In the output layer each neuron outputs the square of the Euclidean distance between its input vector and its weight vector. Afterwards apply normalized exponential function (RBF activation). For CIFAR-10 10 we used Max Pooling instead of Average Pooling and softmax activation on the output. Every simulation starts with learning rate of 0.1. The learning rate for EMNIST is annealed after 62K steps, and for CIFAR-10 10 after 25K steps. When considering dropout as the hyperparameter being varied, the exchanges are introduced after 25K and 10K steps for EMNIST and CIFAR-10, respectively. For the simulation where the learning rate itself is being varied, each replica is annealed different value of the learning rate, and exchanges start being proposed right after annealing.
The results for LeNet-like models are presented in Fig. Replicas differing by a "temperature" defined by Dropout and learning rate are generated, and the model performance (in terms of clas- sification error) of the best independent replica, corresponding to fixed hyperparameter value, is compared against parallel tempering solution, where replica swaps introduced. In all of the cases, for both datasets, the best parallel tempered path achieves a significantly lower error rate. We have also observed increased resilience to overfitting in small-scale simulations, though this aspect requires more careful study. It is worth noting, that for the EMNIST task an error rate com- parable the best untempered results is achieved in a much smaller number of training epochs. Overall, the training is at least as fast as for the independent simulations.
16
0.1
Restie:20 (0.003.... 0.020'"
Restier20 y - 0.0060
Restlttea y (0. 00..0............ 0.006)*
0.009
Rcsticte4 y - 000045
0.008
Error (%)
0.007
0.0D6
0.005
0.004
0.003
10K
20K
30K
40K
5DK
50K
30K
40K
50K
60K
Mini-Batch Steps
Figure 4: Left: Example of exchanges between replicas differing in learning rates, for the ResNet architecture Shaded in grey: the path in hyperparameter space taken by the hest simulation. Right: Test error curves for the ResNet architectures. Dashed lines correspond to the best annealed learning rate, while solid lines show the results with replica exchanges. The mean and standard deviations for multiple simulations are summarized in Table 3
In order to showcase the flexibility of the algorithm, and its potential for generating additional im- provements for already efficient and optimized models, we benchmark the approach on the residual architectures for 10, where follow He et al. (2016). Specifically, images are normal- ized by subtracting per- pexep mean and augmented via a number transformations: the image is left-righh-ffipped with probability 0.5 and 4 pixels are zero-padded
each side, with following random 32 x 32 crop. We use weight decay of 0.0001 and momentum of 0.9. We apply Batch Normalization, as in the original paper, and compare exchangeable learning rate fixed one after the initial learning rate annealing. The training begins with learning rate 0.1, annealed only once at step 32K; the total amount of training steps is 64K, as in He al. (2016).
In Fig. test error curves for ResNet20 and ResNet44 are presented, along with a visualisation of hyperparameter exchanges between replicas in a representative simulation. Introduction of the exchanges consistently reduces the overall minimal testing error for both architectures. With eight replicas we obtain an improvement of 1.04% for ResNet20 and 1.66% for ResNet44 (see table 3 for mean and standard deviation values). We expect larger improvements for larger number of replicas. Note that the hyperparameter value corresponding to the the best individually trained model (found by grid-search), is included in the parameter ladder for the PT simulation together with small number of "suboptimal" values. The PT solution improves on all of them, by leveraging the replicas to explore the parameter space in nonlocal fashion. This is seen in the non-trivial trajectories in hyperparameter space taken by all replicas as the training of their weights proceeds, in particular the trajectory of the ultimately best one.
7

---

Under review as a conference paper at ICLR 2020
Table 3: ResNet classification enror for best independent and parallel-tempered (PT) solutions, from five runs with randomized 90/10 train-validation splits.
Model
MEAN
STD
ResNet20
0.0791
0.0006
ResNet20 PT
0.0783
0.0011
ResNet44
0.0674
0.0007
ResNet44 PT
0.0663
0.0014
CONCLUSIONS AND DISCUSSION
We introduced a new training approach to improve model optimization, by coupling previously independent grid search simulations different hyperparameter values using the replica exchange technique. The method, which can be thought of as optimizing the model over non-monotonic and nonlocal paths in the joint hyperparameterrweighhe space, is very general: can be applied any hyperparameter which admits interpretation of facilitating weight diffusion during training. This diffusion test is, in fact, simple experiment which can be performed at little cost to establish, whether any given parameter is related to landscape smoothness. We show that this is the case for Dropout, learning rate, but also Batch Normalization. The method is easily parallelizable, and similar in cost to standard grid search.
temperature-like quantity, in the very weak sense
Experiments performed on LeNet (with CIFAR and EMNIST datasets) and ResNet (with CIFAR) architectures, showed consistently lower test error. In particular, obtain improvement over the benchmark results for ResNet20 and ResNet44 on the CIFAR dataset.
A number of further improvements possible. Practically, the most important concerns the tomation of hyperparameter range selection, and the subsequent replica temperature ladder choice to optimize acceptances, which can potentially also reduce the test error. Conceptually, the idea can also be generalized to the case of multiple parameters controlling the smoothness of the en- ergy landscape, which are not necessarily lemperature-like Fukunishi al.] (2002). In this setup, multidimensional exchanges between replicas with different values of distinct hyperparameters are permitted, using the hyper-paralle, tempering method Yan & de Pablo (1999); /Sugita et al. (2000), allowing for more complex paths. These topics are the subject of an ongoing work.
REFERENCES
Guozhong An. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643-66) 674, 1996.
James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In J. Shawe-Taylor, R. S. 7emel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp- 2546-2554 Curran Associates, Inc., 2011. URL http:: nips ccppeec/paper/ 4443 al algoritthmae hmss- for for-hyper- parameter catee-optmmmeee wwt.oppimtiiiiin..oom pdf
Chris M. Bishop. Training with noise is equivalent to tikhonov regularization. Neural Computation, 7(1):108-116 1995. doi: 2..116//ncco.1995...1 neco 995.
URL
108
Anna Choromanska, MIkael Henaff, Michael Mathieu, Gerard Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. In Guy Lebanon and N. Vishwanathan (eds.), Proceedings of the Eighteenth International Conference Artificial Intelligence and Statis- tics, volume 38 of Proceedings of Machine Learning Research, pp. 192-204, San Diego, Cali- http: httt::://proceedingg ml mtt.sssss.rrrr.e
fornia, USA, 09-12 May 2015. PMLR. URL TRRhttp.
choromanskal15 h
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters. arXiv preprint arXiv: 172.:3702.05373. 2017.
8

---

Under review as a conference paper at ICLR 2020
Guillaume Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delalleau. Tempered markov chain monte carlo for training of restricted boltzmann machines. In Yee Whye Tch and Mike Titterington (eds.), Proceedings
the Thirteenth International Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 152. PMLR, 13-15 May 2010. URL http::/prroceddingg/ ml m.l..pp www.rrrres/vvv jas html
David J. Earl and Michael W. Deem. Parallel tempering: Theory, applications, and new perspectives. Phys. Chem. Chem. Phys.. 7:3910-3916, 2005. doi: 10.1039/B509983H. URL http:://a/ //dx
doi wal..org/10.o 1039/15099933333
Hiroaki Fukunishi, Osamu Watanabe, and Shoji Takada. On the hamiltonian replica exchange method for efficient sampling of biomolecular systems: Application to protein structure predic- tion. The Journal of Chemical Physics, 116(20):9058 9060099697609800,,009 2002. doi: 10.1063/1.14725100 URL
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the de- scription length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT '93, pp. 5-13, New York, NY, USA, 1993. ACM. ISBN 0- 0-89791-611-55 URL http::
doi: 10. 00.1145/668044166888
11441/16011455 168366
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/ 0ass/120..553 200000 2012. URL httpp://aaxi...rrr///bssss/e//se
0580
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In G. Tesauro, D. S. Touretzky, and T. K. Leen (eds.), Advances in Neural Information Process- ing Systems 7, pp. 899-simpl Fy in neural neeeelll-eettteee by scovering flatt-fllatt---
MIT Press, 1995. URL hRt/:tt/://ppeer nips cc/paper/ pp..
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/1711.008846
2017.
orq/ abs 1711.11100 C9846
Helmut G Katzgraber, Simon Trebst, David A Huse, and Matthias Troyer. Feedback parallel tempering monte carlo. Journal of Statistical Mechanics: Theory and Experiment, 2006 (03::P03318-P0301 mar 2006. doi: 10.1088/174 000.0///174424566600000000000000
URL httpp://
doi doi.org/10. 108882F1 220
Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science, 220 doi: 22t20//science.22.455959 httg://oneeet/22200 (etttenn//2204558//7777
(4598):671-680, 1983. ISSN 0036-8075.
URL https:
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1:21)2:278222)) 1998.
Arvind Neelakantan, Luke Vilnis, Quoe V. Le, llya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding Gradient Noise Improves Learning for Very Deep Networks. arXiv e-prints, art. arXiv: 111155116666 156807, Nov 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2493.2493. Curran Associates, Inc., 2018. URL httt::t/ppppeeeaea nips cctppsccccccccccccc cips.cc/ppppr. zat on he p opt wwwimiiitiiooooi
7515-how does -ectoosss-ttttee baw

---

Under review as a conference paper at ICLR 2020
H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples.
Phys. Rev. A. 45:6056 6091, Apr 1992. doi: 10.1 10.1103/hhyseew4.45..0555 URL https:
link. aps wwtp//tgg/iooooooo 1133//yyseee.....mm 45. 6056
Jocelyn
Sietsma and Robert J.F. Dow. Creating artificial neural networks that general-
ize. Neural Networks, 4(1):67 79, 1991. ISSN 0893-6080. doi: https//doi.org)
101001600933-00009999000333----3 URL http: /ttp:://www... www.wcciieeddeerttccoom com/ /toom/scince/////
atticle/B110083300910000000
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 2, NIPS 12, PP. 2951-2959, USA, 2012. Curran Associates Inc.
URL http: acm. org/ citatiionnn wtw.icfmmm.........m..... htt.::oc......e22222229222399939
Yuji Sugita, Akio Kitao, and Yuko Okamoto. Multidimensional replica-exchange method for free- energy calculations. The Journal of Chemical Physics, 113(15)6642-6601, 2000. doi: 10.1063/
1.1308516. URL htLhttt:: http:://doooo.....o...oo. 000.0.00331111 130051
Robert H. Swendsen and -lin--heng Wang. Replica monte carlo simulation of spin-glasses. Phys. Rev. Lett. 57:2607-2609, Nov 1986. doi: 10. 10w./PhysRewLeet..7....0.. URL https://
link.ar aps wttg/////l/oooooooo 103/ www//oyaeeeeeettt.......aaaa.... .7226077
Simon Trebst, Matthias Troyer, and Ulrich H. E. Hansmann. Optimized parallel tempering simu-
lations of proteins. The Journal of Chemical Physics, 124(17):174903, 2006. doi: 10.1063/1.
2186639. URL httph::t///oolt 21t.:/........111111111.11..6200
10w3/1
Qiliang Yan and Juan de Pablo. Hyper-parallel tempering monte carlo: Application to the lennard- jones fluid and the restricted primitive model. The Journal of Chemical Physics, 111(21) 9)5509 9516, 1999. doi: 00.106//1.40255 URL http:::ttpssoooo wrt.1...11/01111/1111111 4802822
Yao Zhang, Andrew M. Saxe, Madhu S. Advani, and Alpha A. Lee. Energy-entropy competition and the effecctiveness of stochastic gradient descent in machine learning. Molecular Physics, 116 (21-22)3311-33223 2018, doi: 10. 1000000068996..2000 201..71333.8.. URL //dol.or/////////////////
10. 10.1000/0026976... 2018 1483535
10

---

Under review as a conference paper at ICLR 2020
T3
T2
T1
W
Figure The training dynamics the algorithm. The model replicas at different values of hyperparameters correspond to different fictitious temperatures T; At every temperature the algorithm performs standard SGD optimization (as opposed to Markov Chain Monte Carlo dynamics typically used in physical simulations), which is depicted with green trajectories. Periodically, swaps between neighbouring temperatures The proposals are accepted using MC Metropolis Hastiigs acceptance criteria, based on the current values of the loss functions L(W) and temperatures the replicas (see Eq. 7]. Swap moves are shown with blue arrows in the figure. At higher temperatures the landscape is effectively flttter and noisy SGD dynamies can access larger portions of (the energy scale for SGD moves is shown with red shading). By using non-local moves mediated by higher temperature replicas the SGD dynamics can probe previously inaccessible regions of the potential landscape.
proposed.
A APPENDIX
In Fig. 5 the dynamics of the training algorithm across multiple replicas is shown. We emphasize that the dynamics for any fixed value of a hyperparameter is performed using standard SGD dynamics, and not Monte Carlo. Moreover, SGD can be replaced by c.g. Adam optimization, as MC exchanges are only performed between replicas, within which dynamics can be arbitrary.
For conceptual simplicity we have assumed single parameter controlling the acceptance ratio of swap moves between all the replicas in our proof-of of-principle implementation. The performance of the algorithm is sensitive to the value of C. the table we provide comparison of average ac- ceptance rates for different values of C. In fact, for optimal performance acceptances between each pair of neighbouring replicas should adjusted, to allow for an efficient exploration the land- scape at all scales. It may hyperparameters, but this is not the case. The tuning of acceptances, or equivalently the distribution of hyperparameters for the replicas, graber et al. (2006); Trebst et al. (2006). In the current implementation we used a fixed acceptance parameter focus the discussion the main idea of replica exchange of hyperparameters.
that this requires excessive tuning, in effect introducing additional
be performed automatically in adaptive procedure Katz-
Table 4: Comparison study of dependance of average acceptance across all the replicas difference values of constant C.
C
Average acceptance of move
500
0.875
1000
0.785
5000
0.43
10000
0.304
50000
0,209
11


Reviews:
[{"review_id": "HygEfDYmqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an analogous optimization algorithm that is inspired by MCMC. The algorithm maintains a series of weight replicas and sweeps them according to loss functions, based on rules from statistical physics.\n\nI appreciate the idea of introducing temperature here, and the part that describes the idea is nicely written. I may have concerns, as in training neural network, most optimization algorithms use stochastic gradient which intrinsically contain noise inside, although certain level of gradient noise helps to escape saddle point on the landscape, there is no necessary analysis whether this additional noise would contribute to the optimization positively or negatively, after all the noise level has to be bounded for the optimizer to converge.\n\nThe experiments are a little insufficient, as the authors used very small datasets, and there is no comparison with other carefully tuned optimization algorithms, which, circle back to my previous concern, is that whether such an algorithm could actually outperform a carefully tuned SGD, ADAM.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lQe-5hFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presented an improvement of grid search algorithm for certain hyperparameters in deep neural nets training. These hyperparameters, such as learning rate and drop out, have \"temperature\" like meaning to control the noise injected in the training. With this analogy, the author proposed to use the idea of parallel tempering in statistical physics to allow exchange these hyperparameters during the training. Their empirical results showed this improves standard grid search.\n\nThe main contributions are two folds: 1. the connection between some hyperparameters to its physical perspective (temperature) and 2. the empirical validation of the proposed algorithm. The paper is clearly written and easy to follow. I would like to accept this paper if the authors can address following comments.\n\n1. In Line 11 of algorithm 1, what is \\alpha and how to update it?\n2. How do the authors come up with the values for learning rate and drop out in the experiments? How sensitive the results are to the choice of those values?\n3. What is the value of C used in the experiments. Do you have ablation study of this choice? How to pick this value?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkeRAd5lqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "REVIEW 2\n\nThe paper proposes a new paradigm to perform hyperparameter search by proposing a way to jointly optimize over the hyperparameter space and the parameter space as opposed to the traditional way to performing these in separation (with the hyperparameter search invoking the parameter optimization) The paper’s main claim is that this allows the seach to follow non-local paths in the joint space. The main methodology proposed in the paper is inspired by the idea of parallel tempering from physics. The paper proposes to view parameter learning under a certain hyperparameter as running a langevin chain at a particular temperature. This is motivated by considering common hyperparameters as batch size, dropout rate or learning rate as inducing a specific level of noise to the training process, the variance of which is analogous to the inverse temperature in Langevin diffusion. \n\nUnder the above analogy, the paper proposes to run an analogue of parallel tempering in the following way. Consider running parallel threads of parameter optimization for various fixed choices of hyperparameter values (each representing a temperature). After running certain number of steps of SGD in each configuration, the paper proposes to exchange hyperparameters within a pair of threads, with the chance of exchange depending upon a criteria motivated by metropolis hastings.  \n\nWhile the paper proposes an interesting idea I have many reservations about the paper and applicability of the idea in practice prompting my score which I outline below. \n\n1. The first and major shortcoming of the paper is an extreme lack of detail and rigour. The paper seems to be written in a very hand wavy fashion. While that is okay in working towards the proposal of the algorithm, which of course is merely claimed to be inspired by parallel tempering. I believe the paper needs to do an important job of specifying exactly how one selects the notion of temperature for each choice of the hyperparameter. The mapping of the choice to the temperature value is essential for the algorithm to succeed and as described in the paper, there is no hope of replicating the algorithm as the authors are extremely vague about how to figure out this mapping. A second instance of this is diffusion curves in the introduction. The paper does not define what a numerical value of diffusion is. It only relies on a hand-waved understanding of diffusion to even understand presented curves. Note that the paper does not have an appendix and there is no link to code either. \n\n2. Lack of intuition - for this being a good procedure in terms of selecting a good path eventually. Note that hyperparameter optimization is an optimization procedure and it is very unclear to me why sampling from the distribution which will look like the product distribution over the replicas is even a good idea. If it is the case this needs to be specified carefully.\n3. The experiments performed seem to follow a common trend. A single hyperparameter is selected (learning rate/dropout rate), a small grid of fixed values for the hyperparameter are selected and the procedure is performed. In light of this, what a “good” path through the algorithm represents is a good schedule for adjusting these hyperparameters. This is a well known intuition that for parameters like learning rate one needs to adjust them periodically to achieve good training dynamics. Since the comparison is to fixed learning rates, the improvement therefore looks predictable. \n\n4. Lack of applicability of the algorithm for hyperparameter search in practice - There are a couple of things which make the algorithm restrictive - \nThis seems to be useful for hyperparameters that can be in a sense equated to parameters that change the training dynamics. The examples considered in the paper - batch size, learning rate, dropout rate. I think the authors should make this distinction and state it upfront because there are many hyperparameters that we optimize over which change the target distribution and not just the noise. \nSecondly, large scale hyperparameter search takes the idea of separating hyper parameter and parameter search as then it can run in a parallel asynchronous fashion. The algorithm is highly synchronous way of doing this search. While it can be argued that we should move to this paradigm if the results are significant but unfortunately no comparisons with existing hyperparameter search algorithms are provided. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses a variant of parallel tempering to tune the subset of neural net hyperparameters which control the amount of noise and/or rate of diffusion (e.g. learning rate, batch size). It's certainly an appealing idea to run multiple chains in parallel and periodically propose swaps between them. However, I'm not persuaded about the details. The argumentation in the paper is fairly informal, and it uses ideas from optimization and MCMC somewhat interchangeably. Since the individual chains aren't sampling from any known stationary distribution, it's not clear to me what MH-based swaps will achieve. 

The authors are upset with one of the reviews and think it misrepresents their paper. However, I find myself agreeing with most of the reviewer's points. Furthermore, as a general principle, the availability of code doesn't by itself make a paper reproducible. One should be able to reproduce it without the code, and one shouldn't need to refer to the code for important details about the algorithm.

Another limitation (pointed out by various reviewers) is that there aren't any comparisons against prior work on hyperparameter optimization. Overall, I think there are some promising and appealing ideas in this submission, but it needs to be cleaned up before it's ready for publication at ICLR.

Decision: Reject