Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

1 INTRODUCTION . The remarkable improvement in performance of machine learning models has been paid for with an increased model complexity . Part of this complexity is directly related to architectural choices , such as model topology and depth or increased overall number of parameters ( weights ) , but an increasingly important aspect , from both theoretical and practical standpoint , is the growth of the number of hyperparameters . Though the classification may seem fluid , with structural features such as the number of layers being treated as a hyperparameters as well , the difference is laid bare by the distinct nature of training , which involves a double loop of optimizations . The inner loop finds the best model weights – with hyperparameters held fixed to some arbitrary values – using the performance on the training dataset , while the outer loop is responsible for finding the optimal value of hyperparameters to ensure a robust performance on new data , i.e . generalization . A separate dataset is used for this key task . Hyperparameters thus optimized may include structural features , regularization constants , and even dynamical characteristics of the training algorithm . The practical importance of hyperparameters can not be underestimated : the very same architecture may deliver state-of-the-art or mediocre results , depending on the choices made . Simultaneously as the number of hyperparameters grows , the search space becomes high-dimensional and thus necessitates proper hyperparameter optimization ( HO ) procedures , which go beyond rule-of-thumb choices . This is additionally complicated by the fact that computational resources and time are usually limited . A number of HO approaches have been developed ( Bergstra et al. , 2011 ) ; the simplest ones involve grid- or random searches , but their main weakness is poor scaling and failure to reuse information obtained from previous parameter choices . A broad family of methods addressing the latter issue arise in the Bayesian framework : here the the assignment of new hyperparameter values to be evaluated is informed by the performance of the model with the previous assignments ( Snoek et al. , 2012 ) . The high computational cost of evaluation inspires additional strategies , for instance bandit methods , to leverage cheaper but cruder results obtained with partial datasets . All of the above HO strategies , however , share a key characteristic : optimization of hyperparameters is decoupled from that of the weights , and the procedures amount to a more or less clever sampling of the points in hyperparameter space , using the validation dataset only . A family of genetic HO algorithms overcome this issue by optimization in both weights and hyperparameter space , however , in this case , in a greedy way ( see for example Jaderberg et al . ( 2017 ) ) . They can not be thus guaranteed to explore space in an unbiased way , and the results may explicitly depend on initial conditions . Here , we propose a radically different strategy : that of coupling the usually independent copies of the model at different hyperparameter values during the training . The instances of the model are allowed to exchange their hyperparametes as the optimization of the weights on the training dataset progresses , in effect tracing out a highly nonlocal trajectory in the product of hyperparameter and weight spaces . The final trained model can not thus be thought of as being characterized by a particular point value of the hyperparameters , but rather by a path or history . The exchange procedure is based on the physical technique of parallel tempering ( Swendsen & Wang , 1986 ) , which itself depends on a mapping of the hyperparameter values to an effective “ temperature ” of the model . We demonstrate empirically , that this HO approach – which leverages also the training dataset – has a number of desirable properties : it results in models more resilient to overfitting , achieving smaller overall errors , and achieving them faster . The method is naturally parallelizable and the computational cost overhead over usual grid searches is negligible . The goal of this work is then twofold : first , to provide ( or , in some instances , recapitulate ) a unifying and intuitive physical perspective on various types of hyperparameters , which can be interpreted as setting the level of “ noise ” during the training of the model . This , in turn , defines an effective parameter - similar in spirit to temperature - controlling the smoothness of the generalization function . Second , and much more important in applications , is to show that this perspective allows to move beyond the usual paradigm of two decoupled weight/hyperparameter optimizations in a well-motivated and controlled fashion , and ultimately to train more robust models . This paper is organized as follows : in section 2 we review the interpretation of hyperparameters as controls of noise , and the mapping to an effective ” temperature ” . In section 3 we introduce the main concepts of our path-based model optimization . In section 4 we show numerical tests of the approach on neural nets trained on EMNIST and CIFAR-10 datasets . The discussed examples of hyperparameters include the learning rate and drop-out rate , among others . Finally , in section 5 we discuss the implications , as well as potential generalizations of the method . 2 HYPERPARAMETERS AS CONTROLS OF SMOOTHNESS OF POTENTIAL LANDSCAPE . Training of modern machine learning models , in particular of deep neural networks ( DNNs ) , is a complex non-convex optimisation problem of minimizing the total objective function ( otherwise called loss or energy ) : L ( x , W ) = 1 N N∑ i=1 li ( xi , W ) , ( 1 ) where li ( xi , W ) captures deviations of the predictions of the model , parametrized by weights W , for the i-th data point xi in the training set of total size N . The standard choice of the optimisation method for this problem remains the Stochastic Gradient Descent ( SGD ) algorithm Wt+1 = Wt − γ |NB | ∑ i∈NB ∇Wli ( xi , Wt ) ( 2 ) with a step size γ and the gradient computed only on a batch NB of the training set . It is well established , that various forms of direct noise injection , whether to the inputs , weights or gradients , aid generalization performance ( Sietsma & Dow , 1991 ; Neelakantan et al. , 2015 ) . From the theory point of view it has been shown that the effect of such methods is analogous – though not identical – to introducing particular types of regularization ( Bishop , 1995 ; An , 1996 ) smoothing the cost function . Furthermore , an often invoked intuition is that noise allows the algorithm to escape narrow minima of the cost function defined by the training data , thereby preventing overfitting . This is also true for non-white noise introduced “ indirectly ” , such as the one generated by the minibatches in Eq.2 . Aided by the results suggesting poor generalization for “ deep ” ( Choromanska et al. , 2015 ) , and , conversely , good generalization for “ flat ” minima of the loss function ( Hinton & van Camp , 1993 ; Hochreiter & Schmidhuber , 1995 ) , it was postulated in Zhang et al . ( 2018 ) that strong performance of SGD is , in fact , due to the correlated nature of the induced batch noise . In what follows , we will greatly benefit from an intuitive – if not always rigorous – picture relating various kinds of noise to an effective parameter controlling smoothness of the potential landscape , similar in spirit to temperature . Common hyperparameters , such as the learning rate , dropout or batch size , can be treated on the same footing , as we now demonstrate . Let us begin with the prototypical case of added Langevin noise , for which the temperature analogy is exact ( Seung et al. , 1992 ) : the gradients are corrupted by white noise ξ with a variance 〈ξi ( t ) ξj ( t′ ) 〉 = 2Tδijδtt′ . The naive GD can then be viewed as a diffusion process of particle in a complex potential landscape , with the weight updates a discretized version of the continuum equation : ∂W ∂t = −∇WL+ ξ ( t ) . ( 3 ) At long-times Eq . 3 converges to a Gibbs probability distribution , from which weights W can be directly sampled : P ( W ) = 1 Z exp ( −βL ( W ) ) . ( 4 ) The inverse temperature β = 1/T is not arbitrary , but rather is defined by the variance of the noise ξ . Furthermore , the multiplicative prefactor β in Eq . 4 controls the overall scale of variation of the potential landscape . Strong noise , corresponding to large variance , and therefore high temperature T , results in small β which “ flattens ” the landscape of L ( W ) . Conversely , weak noise results in large β , amplifying potential differences . This is the central idea behind changing temperature in the familiar simulated annealing method ( Kirkpatrick et al. , 1983 ) . An alternative way of phrasing the above is that the increased variance of the noise does not allow the finer details the potential landscape to be explored by the dynamics , effectively smoothing it . The crucial observation is that the latter perspective extends also to cases where the noise is not white . Though , strictly speaking , the variance of the noise is not the temperature anymore , it still performs an analogous function : it effectively controls the roughness/smoothness of the landscape . An immediate consequence is that higher variance facilitates diffusion of the model in such high-dimensional space , and improves ergodicity ( see Fig . 1 ) . As an illustration , we now consider common examples of hyperparameters from this perspective , and examine their influence on diffusion curves . For the learning rate γ , its magnitude is directly related to the size of the time step in the discretization of the Langevin equation , and by dimensional analysis it is analogous to increasing the noise variance ( thus temperature ) by a factor of γ . In the case of SGD dynamics Eq . 2 the noise variance can be computed explicitly ( Zhang et al. , 2018 ) . Even though the noise is highly correlated , i.e . not thermal , the variance scales overall as inverse batch size : |Nb|−1 , and thus smaller batch size smoothens the potential . Similarly , dropout regularization has the effect of increasing the variance of neural outputs at training by a factor of inverse dropout retention rate p−1 ( Hinton et al. , 2012 ) , therefore amplifying noise and improving diffusion , as seen in Fig . 1 . The case of L2 regularization is different : an increased L2 naturally diminishes the magnitude of the potential landscape variations , and so the initial diffusion is faster ( see Fig . 1 ) , however at later times the diffusion is effectively suppressed ; the plateau value reached is , naturally , the lower the stronger the L2 regularization . Since , therefore , the models do not diffuse at different rates , L2 does not satisfy the basic requirements of our procedure , and we do not expect systematic improvements for this hyperparameter . On the other hand , for parameters relating to direct noise injection ( for instance to gradients or weights ) , the relationship of large magnitude to high “ temperature ” is natural , as described above , and their smoothing effect has been noted in the literature An ( 1996 ) . Since generically there are many hyperparameters , and results or intuitions such as above may not be readily available , it is imperative to be able to systematically identify whether they are related to landscape smoothness . This is , fortunately , possible . The functional significance of the notion of smoothness lies precisely in determining the diffusion properties of the random walk , therefore it is natural to test for smoothing properties by performing a diffusion experiment . To wit , sensitivity of the weight diffusion curves – obtained from SGD training runs – to the value of hyperparameter in question implies relation to smoothness . To exemplify this we consider Batch Normalization ( BN ) . By studying Lipschitz properties Santurkar et al . ( 2018 ) ultimately confirmed it is related to effective landscape smoothing . Alternatively , comparing training with and without BN , it can be immediately shown to improve weight diffusion ( see Fig.2 ) . Training the model with different hyperparameters can thus be intuitively thought of as minimizing the objective at different intensities of noise , or , equivalently , in effective landscapes of varying degree of smoothness . We will now use this insight to construct a new nonlocal hyperparameter optimization procedure .

Summary: The paper proposes a new paradigm to perform hyperparameter search by proposing a way to jointly optimize over the hyperparameter space and the parameter space as opposed to the traditional way to performing these in separation (with the hyperparameter search invoking the parameter optimization) The paper’s main claim is that this allows the seach to follow non-local paths in the joint space. The main methodology proposed in the paper is inspired by the idea of parallel tempering from physics. The paper proposes to view parameter learning under a certain hyperparameter as running a langevin chain at a particular temperature. This is motivated by considering common hyperparameters as batch size, dropout rate or learning rate as inducing a specific level of noise to the training process, the variance of which is analogous to the inverse temperature in Langevin diffusion.