A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs

1 INTRODUCTION . Deep architectures are generally trained by minimizing a non-convex loss function via underlying optimization algorithm such as stochastic gradient descent or its variants . It takes a fairly large amount of time to find the best suited optimization algorithm and its optimal hyperparameters ( such as learning rate , batch size etc . ) for training a model to the desired accuracy , this being a major challenge for academicians and industry practitioners alike . Usually , such tuning is done by initial configuration optimization through grid search or random search ( Bergstra et al. , 2011 ; Snoek et al. , 2012 ; Thornton et al. ) . Recent works have also formulated it as a bandit problem ( Li et al. , 2017 ) . However , it has been widely demonstrated that hyperparameters , especially the learning rate often needs to be dynamically adjusted as the training progresses , irrespective of the initial choice of configuration . If not adjusted dynamically , the training might get stuck in a bad minima , and no amount of training time can recover it . In this work , we focus on learning rate which is the foremost hyperparameter that one seeks to tune when training a deep learning model to get favourable results . Certain auto-tuning and adaptive variants of SGD , such as AdaGrad ( Duchi et al. , 2011 ) , Adadelta ( Zeiler , 2012 ) , RMSProp ( Tieleman & Hinton , 2012 ) , Adam ( Kingma & Ba , 2015 ) among others have been proposed that automatically adjust the learning rate as the training progresses , using functions of gradient . Yet others have proposed fixed learning rate and/or batch size change regimes ( Goyal et al. , 2017 ) , ( Smith et al. , 2018 ) for certain data set and model combination . In addition to traditional natural learning tasks where a good LR regime might already be known from past experiments , adversarial training for generating robust models is gaining a lot of popularity off late . In these cases , tuning the LR would generally require time consuming multiple experiments , since the LR regime is unlikely to be known for every attack for every model and dataset of interest1 . Moreover , new models are surfacing every day courtesy the state-of-the-art model synthesis systems , and new datasets are also becoming available quite often in different domains such as healthcare , automobile industy etc . In each of these cases , no prior LR regime would be known , and would require considerable manual tuning in the absence of a universal method , with demonstrated effectiveness over a wide range of tasks , models and datasets . Wilson et al . ( 2017 ) observed that solutions found by existing adaptive methods often generalize worse than those found by non-adaptive methods . Even though initially adaptive methods might 1For example , one can see a piecewise LR schedule given by Madry et al . ( 2018 ) at https : //github . com/MadryLab/cifar10_challenge/blob/master/config.json for a particular model . display faster initial progress on the training set , their performance quickly plateaus on the test set , and learning rate tuning is required to improve the generalization performance of these methods . For the case of SGD with Momentum , learning rate ( LR ) step decay is very popular ( Goyal et al. , 2017 ) , ( Huang et al. , 2017 ) , ReduceLRonPlateau2 . However , in certain junctures of training , increasing the LR can potentially lead to a quick , further exploration of the loss landscape and help the training to escape a sharp minima ( having poor generalisation Keskar et al . ( 2016 ) ) . Further , recent works have shown that the distance traveled by the model in the parameter space determines how far the training is from convergence Hoffer et al . ( 2017 ) . This inspires the idea that increasing the LR to take bigger steps in the loss landscape , while maintaining numerical stability might help in better generalization . The idea of increasing and decreasing the LR periodically during training has been demonstrated by Smith ( 2017 ) ; Smith & Topin ( 2017 ) in their cyclical learning rate method ( CLR ) . This has also been shown by Loshchilov & Hutter ( 2016 ) , in Stochastic Gradient Descent with Warm Restarts ( SGDR , popularly referred to as Cosine Annealing with Warm Restarts ) . In CLR , the LR is varied periodically in a linear manner , between a maximum and a minimum value , and it is shown empirically that such increase of learning rate is overall beneficial to the training compared to fixed schedules . In SGDR , the training periodically restarts from an initial learning rate , and then decreases to a minimum learning rate through a cosine schedule of LR decay . The period typically increases in powers of 2 . The authors suggest optimizing the initial LR and minimum LR for good performance . Schaul et al . ( 2013 ) had suggested an adaptive learning rate schedule that allows the learning rate to increase when the signal is non-stationary and the underline distribution changes . This is a computationally heavy method , requiring computing the Hessian in an online manner . Recently , there has been some work that explore gradients in different forms for hyperaparameter optimization . Maclaurin et al . ( 2015 ) suggest an approach by which they exactly reverse SGD with momentum to compute gradients with respect to all continuous learning parameters ( referred to as hypergradients ) ; this is then propagated through an inner optimization . Baydin et al . ( 2018 ) suggest a dynamic LR-tuning approach , namely , hypergradient descent , that apply gradient-based updates to the learning rate at each iteration in an online fashion . We propose a new algorithm to automatically determine the learning rate for a deep learning job in an autonomous manner that simply compares the current training loss with the best observed thus far to adapt the LR . The proposed algorithm works across multiple datasets and models for different tasks such as natural as well as adversarial training . It is an ‘ optimistic ’ method , in the sense that it increases the LR to as high as possible by examining the training loss repeatedly . We show through rigorous experimentation that in spite of its simplicity , the proposed algorithm performs surprisingly well as compared to the state-of-the-art . Our contributions : • We propose a novel , and simple algorithmic approach for autonomous , adaptive learning rate determination that does not require any manual tuning , inspection , or pre-experimental discovery of the algorithmic parameters . • Our proposed algorithm works across data sets and models with no customization and reaches higher or comparable accuracy as standard baselines in literature in the same number of epochs on each of these datasets and models . It consistently performs well , finding stable minima with good generalization and converges smoothly . • Our algorithm works very well for adversarial learning scenario along with natural training as demonstrated across different models and datasets . • We provide extensive empirical validation of our algorithm and convergence discussion . 2 PROPOSED METHOD . We propose an autonomous , adaptive LR tuning algorithm 1 towards determining the LR trajectory during the course of training . It operates in two phases : Phase 1 : Initial LR exploration , that strives 2For eg . https : //www.tensorflow.org/api_docs/python/tf/keras/callbacks/ ReduceLROnPlateau . to find a good starting LR ; Phase 2 : Optimistic Binary Exploration . The pseudocode is provided at Algorithm 1 . For the rest of the paper , we refer to the Automated Adaptive Learning Rate tuning algorithm as AALR in short . Algorithm 1 Automated Adaptive Learning Rate Tuning Algorithm ( AALR ) for Training DNNs Require : Model θ , N Training Samples ( xi , yi ) Ni=1 , Optimizer SGD , Momentum=0.9 , Weight De- cay , Batch size , Number of epochs T , Loss Function J ( θ ) . Initial LR η0 = 0.1 . Ensure : Learning Rate ηt at every epoch t. 1 : Initialize : θ , SGD with LR= η0 , best loss L∗ ← J ( θ ) ( forward pass through initial model ) . 2 : PHASE 1 : Start Initial LR Exploration . 3 : Set patience p← 10 , patience counter i← 0 , epoch number t← 0 4 : while i < p do 5 : Evaluate new loss L← J ( θ ) after training for an epoch . Increment i and t by 1 each . 6 : if L > L∗ or L is NAN then 7 : Halve LR : η0 ← η0/2 . 8 : Reload θ , reset optimizer with LR= η0 , and reset counter i = 0 . 9 : else 10 : L∗ ← L ( Update best loss ) . 11 : end if 12 : end while 13 : Save checkpoint θ and L∗ . 14 : PHASE 2 : Start Optimistic Binary Exploration 15 : Double LR ηt ← 2η0 . Patience p← 1 . 16 : while t < T do 17 : Train for p epochs . Increment epoch number t. 18 : Evaluate new loss L← J ( θ ) . 19 : if L is NAN then 20 : Halve LR : ηt+1 ← ηt/2 . 21 : Load checkpoint θ and L∗ . Reset optimizer with LR= ηt+1 . 22 : Double patience pt+1 ← 2pt . Continue . 23 : end if 24 : if L < L∗ then 25 : Update L∗ ← L. Save checkpoint θ and L∗ . 26 : Double LR ηt+1 ← 2ηt . Set patience p = 1 . 27 : else 28 : Train for another p epochs . Increment epoch number t. 29 : Evaluate new loss L← J ( θ ) . 30 : if L < L∗ then 31 : Update L∗ ← L. Save checkpoint θ and L∗ . 32 : Double LR ηt+1 ← 2ηt . Set patience p = 1 . 33 : else 34 : Halve LR : ηt+1 ← ηt/2 . 35 : Double patience pt+1 ← 2pt . 36 : if L is NAN then 37 : Load checkpoint θ and L∗ . Reset optimizer with LR= ηt+1 . 38 : end if 39 : end if 40 : end if 41 : end while The notation used in the following description is as follows . Patience : p , Learning rate : η , best loss L∗ , current loss L. Model θ , Loss function J ( θ ) . L∗ is initialized as the J ( θ ) after initializing the model , before training starts . Phase 1 : Initial LR exploration Phase 1 starts from an initial learning rate η = 0.1 , and patience p = 10 . It trains for an epoch , evaluates the loss L , and compares to the best loss L∗ . If L < L∗ , the L∗ is updated , and it continues training for another epoch . Otherwise , the model θ is reloaded and re-initialized , LR is halved η = η/2 , and optimizer is reset with the new LR . The patience counter is reset . This continues till a stable LR is determined by the algorithm , in which it trains at this LR for p epochs . The loss L∗ , the model θ and the optimizer state after Phase 1 is saved in a checkpoint . Phase 2 : Optimistic Binary Exploration In this phase , AALR keeps the learning rate η as high as possible for as long as possible at any given state of the training . Phase 2 starts by doubling LR to 2η , and setting p = 1 . After training for p+ 1 epochs , firstly AALR checks if the loss is NAN . In this case , the checkpoint ( model θ and optimizer ) corresponding to the best loss along with the best loss value L∗ are reloaded . Then LR is halved η = η/2 , patience is doubled , and the training continues . If instead , the loss is observed to decrease compared to the best loss , L < L∗ , then L∗ is updated , and the corresponding model θ , optimizer and L∗ are updated in checkpoint . This is followed by doubling the LR η = 2η , resetting p to 1 and continuing training for the next p+ 1 epochs . On the other hand , if L ≥ L∗ , AALR trains for another p + 1 epochs and check the loss L. This is because as informally stated before , AALR is ‘ optimistic ’ and persists in the high LR for some more time . ( In case , the newly evaluated loss is NAN , the previous approach is followed . ) However , if the new lossL ≥ L∗ , then AALR finally lowers the LR . AALR halves the LR η = η/2 , doubles patience p = 2p , and continues training for p + 1 epochs . If however , the loss had decreased , L < L∗ , the previous approach is followed : i.e. , it doubles the LR η = 2η , resets the patience p = 1 , updates best loss and checkpoint , and repeats training for p+ 1 epochs . The above cycle repeats till the stopping criterion is met . For ease of exposition , the pseudocode is given in Algorithm 1

Summary: This paper proposes an algorithm for automatically tuning the learning rate of SGD while training deep neural networks. The proposed learning rate tuning algorithm is a finite state machine and consists of two phases: the first phase finds the largest learning rate that the network can begin training with for p = 10 epochs; the second phase is an optimistic binary exploration phase which increases or decreases the learning rate depending upon whether the loss is NaN, increasing or decreasing. Empirical results are shown on a few standard neural networks for image classification on CIFAR-10/100 datasets and for adversarial training on the CIFAR-10 dataset.