Learning with Random Learning Rates.

Hyperparameter tuning is a bothersome step in the training of deep learning models . One of the most sensitive hyperparameters is the learning rate of the gradient descent . We present the All Learning Rates At Once ( Alrao ) optimization method for neural networks : each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude . This comes at practically no computational cost . Perhaps surprisingly , stochastic gradient descent ( SGD ) with Alrao performs close to SGD with an optimally tuned learning rate , for various architectures and problems . Notably , Alrao is robust : all runs were able to learn without any tuning . This makes Alrao a candidate for situations in which a neural network has to work from the first try . This text comes with a PyTorch implementation of the method , which can be plugged on an existing PyTorch model . 1 INTRODUCTION . Hyperparameter tuning is a notable source of computational cost with deep learning models ( Zoph and Le , 2016 ) . One of the most critical hyperparameters is the learning rate of the gradient descent ( Theodoridis , 2015 , p. 892 ) . With too large learning rates , the model does not learn ; with too small learning rates , optimization is slow and can lead to local minima and poor generalization ( Jastrzebski et al. , 2017 ; Kurita , 2018 ; Mack , 2016 ; Surmenok , 2017 ) . Although popular optimizers like Adam ( Kingma and Ba , 2015 ) come with default hyperparameters , fine-tuning and scheduling of the Adam learning rate is still frequent ( Denkowski and Neubig , 2017 ) , and we suspect the default setting might be somewhat specific to current problems and architecture sizes . Such hyperparameter tuning takes up a lot of engineering time . These and other issues largely prevent deep learning models from working out-of-the-box on new problems , or on a wide range of problems , without human intervention ( AutoML setup , ( Guyon et al. , 2016 ) ) . We propose All Learning Rates At Once ( Alrao ) , an alteration of standard stochastic gradient descent for deep learning models . Alrao uses multiple learning rates at the same time in the same network . By sampling one learning rate per feature , Alrao reaches performance close to the performance of the optimal learning rate , without having to try multiple learning rates . In our experiments , Alrao was found to be quite resilient : not a single run failed to learn , provided a large enough range of admissible learning rates are included . On the other hand , the admissible range of learning rates for SGD is quite narrow . Alrao could be useful in situations where a neural network has to be trained and work out of the box , or for quickly testing network architectures . Alrao increases the size of a model on the output layer , but not on the internal layers : this usually adds little computational cost unless most parameters occur on the output layer . This text comes along with a Pytorch implementation usable on a wide set of architectures . Related Work . Automatically using the “ right ” learning rate for each parameter was one motivation behind “ adaptive ” methods such as RMSProp ( Tieleman and Hinton , 2012 ) , AdaGrad ( Duchi et al. , 2011 ) or Adam ( Kingma and Ba , 2015 ) . Adam with its default setting is currently considered the default method in many works ( Wilson et al. , 2017 ) , and we use it as a baseline . However , further global adjustement of the Adam learning rate is common ( Liu et al. , 2017 ) . Many other heuristics for setting the learning rate have been proposed , e.g. , ( Schaul et al. , 2013 ) ; most start with the idea of approximating a second-order Newton step to define an optimal learning rate ( LeCun et al. , 1998 ) . Methods that directly set per-parameter learning rates are equivalent to preconditioning the gradient descent with a diagonal matrix . Asymptotically , an arguably optimal preconditioner is either the Hessian of the loss ( Newton method ) or the Fisher information matrix ( Amari , 1998 ) . These can be viewed as setting a per-direction learning rate after redefining directions in parameter space . From this viewpoint , Alrao just replaces these preconditioners with a random diagonal matrix whose entries span several orders of magnitude . Another approach to optimize the learning rate is to perform a gradient descent on the learning rate itself through the whole training procedure ( for instance ( Maclaurin et al. , 2015 ) ) . This can be applied online to avoid backpropagating through multiple training rounds ( Massé and Ollivier , 2015 ) . This idea has a long history , see , e.g. , ( Schraudolph , 1999 ; Mahmood et al. , 2012 ) . Some training algorithms depart from gradient descent altogether , and become learning rate-free , such as ( Orabona and Tommasi , 2017 ) using betting strategies to simulate gradient descent . The learning rate can also be optimized within the framework of architecture search , exploring both the architecture and learning rate at the same time ( e.g. , ( Real et al. , 2017 ) ) . The methods range from reinforcement learning ( Zoph and Le , 2016 ; Baker et al. , 2016 ; Li et al. , 2017 ) , evolutionary algorithms ( e.g. , ( Stanley and Miikkulainen , 2002 ; Jozefowicz et al. , 2015 ; Real et al. , 2017 ) ) , Bayesian optimization ( Bergstra et al. , 2013 ) or differentiable architecture search ( Liu et al. , 2018 ) . These methods are resource-intensive and do not allow for finding a good learning rate in a single run . Motivation . Alrao was inspired by the intuition that not all units in a neural network end up being useful . Hopefully , in a large enough network , a sub-network made of units with a good learning rate could learn well , and hopefully the units with a wrong learning rate will just be ignored . ( Units with a too large learning rate may produce large activation values , so this assumes the model has some form of protection against those , such as BatchNorm or sigmoid/tanh activations . ) Several lines of work support the idea that not all units of a network are useful or need to be trained . First , it is possible to prune a trained network without reducing the performance too much ( e.g. , ( LeCun et al. , 1990 ; Han et al. , 2015a ; b ; See et al. , 2016 ) ) . Li et al . ( 2018 ) even show that performance is reasonable if learning only within a very small-dimensional affine subspace of the parameters , chosen in advance at random rather than post-selected . Second , training only some of the weights in a neural network while leaving the others at their initial values performs reasonably well ( see experiments in Appendix H ) . So in Alrao , units with a very small learning rate should not hinder training . Alrao is consistent with the lottery ticket hypothesis , which posits that “ large networks that train successfully contain subnetworks that—when trained in isolation—converge in a comparable number of iterations to comparable accuracy ” ( Frankle and Carbin , 2018 ) . This subnetwork is the lottery ticket winner : the one which had the best initial values . Arguably , given the combinatorial number of subnetworks in a large network , with high probability one of them is able to learn alone , and will make the whole network converge . Viewing the per-feature learning rates of Alrao as part of the initialization , this hypothesis suggests there might be enough sub-networks whose initialization leads to good convergence . 2 ALL LEARNING RATES AT ONCE : DESCRIPTION . Principle . Alrao starts with a standard optimization method such as SGD , and a range of possible learning rates ( ηmin , ηmax ) . Instead of using a single learning rate , we sample once and for all one learning rate for each feature , randomly sampled log-uniformly in ( ηmin , ηmax ) . Then these learning rates are used in the usual optimization update : θl , i ← θl , i − ηl , i · ∇θl , i ` ( Φθ ( x ) , y ) ( 1 ) where θl , i is the set of parameters used to compute the feature i of layer l from the activations of layer l − 1 ( the incoming weights of feature i ) . Thus we build “ slow-learning ” and “ fast-learning ” features , in the hope to get enough features in the “ Goldilocks zone ” . What constitutes a feature depends on the type of layers in the model . For example , in a fully connected layer , each component of a layer is considered as a feature : all incoming weights of the same unit share the same learning rate . On the other hand , in a convolutional layer we consider each convolution filter as constituting a feature : there is one learning rate per filter ( or channel ) , thus keeping translation-invariance over the input image . In LSTMs , we apply the same learning rate to all components in each LSTM unit ( thus in the implementation , the vector of learning rates is the same for input gates , for forget gates , etc. ) . However , the update ( 1 ) can not be used directly in the last layer . For instance , for regression there may be only one output feature . For classification , each feature in the final classification layer represents a single category , and so using different learning rates for these features would favor some categories during learning . Instead , on the output layer we chose to duplicate the layer using several learning rate values , and use a ( Bayesian ) model averaging method to obtain the overall network output ( Fig . 1 ) . Appendix B contains a proof ( under convexity assumptions ) that this mechanism works , given the initial layers . We set a learning rate per feature , rather than per parameter . Otherwise , every feature would have some parameters with large learning rates , and we would expect even a few large incoming weights to be able to derail a feature . So having diverging parameters within a feature is hurtful , while having diverging features in a layer is not necessarily hurtful since the next layer can choose to disregard them . Still , we tested this option ; the results are compatible with this intuition ( Appendix G ) . Definitions and notation . We now describe Alrao more precisely for deep learning models with softmax output , on classification tasks ( the case of regression is similar ) . Let D = { ( x1 , y1 ) , ... , ( xN , yN ) } , with yi ∈ { 1 , ... , K } , be a classification dataset . The goal is to predict the yi given the xi , using a deep learning model Φθ . For each input x , Φθ ( x ) is a probability distribution over { 1 , ... , K } , and we want to minimize the categorical cross-entropy loss ` over the dataset : 1N ∑ i ` ( Φθ ( xi ) , yi ) . A deep learning model for classification Φθ is made of two parts : a pre-classifier φθpc which computes some quantities fed to a final classifier layer Cθcl , namely , Φθ ( x ) = Cθcl ( φθpc ( x ) ) . The classifier layer Cθcl with K categories is defined by Cθcl = softmax ◦ ( WTx+ b ) with θcl = ( W , b ) , and softmax ( x1 , ... , xK ) k = exk/ ( ∑ i e xi ) .The pre-classifier is a computational graph composed of any number of layers , and each layer is made of multiple features . We denote log-U ( · ; ηmin , ηmax ) the log-uniform probability distribution on an interval ( ηmin , ηmax ) : namely , if η ∼ log-U ( · ; ηmin , ηmax ) , then log η is uniformly distributed between log ηmin and log ηmax . Its density function is log-U ( η ; ηmin , ηmax ) = 1ηmin≤η≤ηmax log ( ηmax ) −log ( ηmin ) × 1 η . Alrao for the pre-classifier : A random learning rate for each feature . In the pre-classifier , for each feature i in each layer l , a learning rate ηl , i is sampled from the probability distribution log-U ( . ; ηmin , ηmax ) , once and for all at the beginning of training.1 Then the incoming parameters of each feature in the preclassifier are updated in the usual way with this learning rate ( Eq . 3 ) . Alrao for the classifier layer : Model averaging from classifiers with different learning rates . In the classifier layer , we build multiple clones of the original classifier layer , set a different learning rate for each , and then use a model averaging method from among them . The averaged classifier and the overall Alrao model are : CAlraoθcl ( z ) : = Ncl∑ j=1 aj Cθclj ( z ) , Φ Alrao θ ( x ) : = C Alrao θcl ( φθpc ( x ) ) ( 2 ) where the Cθclj are copies of the original classifier layer , with non-tied parameters , and θ cl : = ( θcl1 , ... , θ cl Ncl ) . The aj are the parameters of the model averaging , and are such that for all j , 0 ≤ aj ≤ 1 , and ∑ j aj = 1 . These are not updated by gradient descent , but via a model averaging method from the literature ( see below ) . For each classifier Cθclj , we set a learning rate ηj defined by log ηj = log ηmin + j−1 Ncl−1 log ( ηmax ηmin ) , so that the classifiers ’ learning rates are log-uniformly spread on the interval [ ηmin , ηmax ] . 1With learning rates resampled at each time , each step would be , in expectation , an ordinary SGD step with learning rate Eηl , i , thus just yielding an ordinary SGD trajectory with more noise . Thus , the original model Φθ ( x ) leads to the Alrao model ΦAlraoθ ( x ) . Only the classifier layer is modified , the pre-classifier architecture being unchanged . Update rule . Alg . 1 presents the full Alrao algorithm . The updates for the pre-classifier , classifier , and model averaging weights are as follows . • The update rule for the pre-classifier is the usual SGD one , with per-feature learning rates . For each feature i in each layer l , its incoming parameters are updated as : θl , i ← θl , i − ηl , i · ∇θl , i ` ( ΦAlraoθ ( x ) , y ) ( 3 ) • The parameters θclj of each classifier clone j on the classifier layer are updated as if this classifier alone was the only output of the model : θclj ← θclj − ηj · ∇θclj ` ( Cθclj ( φθpc ( x ) ) , y ) ( 4 ) ( still sharing the same pre-classifier φθpc ) . This ensures classifiers with low weights aj still learn , and is consistent with model averaging philosophy . Algorithmically this requires differentiating the loss Ncl times with respect to the last layer ( but no additional backpropagations through the preclassifier ) . • To set the weights aj , several model averaging techniques are available , such as Bayesian Model Averaging ( Wasserman , 2000 ) . We decided to use the Switch model averaging ( Van Erven et al. , 2012 ) , a Bayesian method which is both simple , principled and very responsive to changes in performance of the various models . After each sample or mini-batch , the switch computes a modified posterior distribution ( aj ) over the classifiers . This computation is directly taken from ( Van Erven et al. , 2012 ) and explained in Appendix A . The observed evolution of this posterior during training is commented in Appendix C. Implementation . We release along with this paper a Pytorch ( Paszke et al. , 2017 ) implementation of this method . It can be used on an existing model with little modification . A short tutorial is given in Appendix J . The features ( sets of weights which will share the same learning rate ) need to be defined for each layer type : for now we have done this for linear , convolutional , and LSTMs layers . Algorithm 1 Alrao for model Φθ = Cθcl ◦φθpc withNcl classifiers and learning rates in [ ηmin , ηmax ] 1 : aj ← 1/Ncl for each 1 ≤ j ≤ Ncl . Initialize the Ncl model averaging weights aj 2 : ΦAlraoθ ( x ) : = ∑Ncl j=1 aj Cθclj ( φθ pc ( x ) ) . Define the Alrao architecture 3 : for all layers l , for all feature i in layer l do 4 : Sample ηl , i ∼ log-U ( . ; ηmin , ηmax ) . . Sample a learning rate for each feature 5 : for all classifiers j , 1 ≤ j ≤ Ncl do 6 : Define log ηj = log ηmin + j−1Ncl−1 log ηmax ηmin . . Set a learning rate for each classifier j 7 : while stopping criterion is false do . The criterion we use is given in section 3 8 : zt ← φθpc ( xt ) . Store the pre-classifier output 9 : for all layers l , for all feature i in layer l do 10 : θl , i ← θl , i − ηl , i · ∇θl , i ` ( ΦAlraoθ ( xt ) , yt ) . Update the pre-classifier weights 11 : for all Classifier j do 12 : θclj ← θclj − ηj · ∇θclj ` ( Cθclj ( zt ) , yt ) . Update the classifiers ’ weights 13 : a← ModelAveraging ( a , ( Cθcli ( zt ) ) i , yt ) . Update the model averaging weights . 14 : t← t+ 1 mod N

Summary: In this paper the authors propose a method called “All learning rates at once” (Alrao) which aims to save the time needed to tune learning rate for DNN models testing. The method sets individual learning rate to each feature in each layer of a network using the values sampled from truncated log-uniform distribution. The only cost of the method is the creation of several branches of the classifier layer. Each of the branches is trained with a predefined learning rate value, and the final predictions are obtained by model averaging. In the presented experiments Alrao demonstrates performance comparable to SGD with optimal learning rate and more stable results compared to Adam. The authors indicate limitations of Alrao caused by the overhead in the final layer which complicates the application of the method for models with large classifier layer.