Title: A SIMPLE DYNAMIC LEARNING RATE TUNING ALGO-RITHM FOR AUTOMATED TRAINING OF DNNS Anonymous

Abstract: Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach consistently achieves top-level accuracy compared to SOTA baselines in the literature in natural as well as adversarial training.

1. INTRODUCTION:

Deep architectures are generally trained by minimizing a non-convex loss function via underlying optimization algorithm such as stochastic gradient descent or its variants. It takes a fairly large amount of time to find the best suited optimization algorithm and its optimal hyperparameters (such as learning rate, batch size etc.) for training a model to the desired accuracy, this being a major challenge for academicians and industry practitioners alike. Usually, such tuning is done by initial configuration optimization through grid search or random search (Bergstra et al., 2011; Snoek et al., 2012; Thornton et al.) . Recent works have also formulated it as a bandit problem (Li et al., 2017) .

However, it has been widely demonstrated that hyperparameters, especially the learning rate often needs to be dynamically adjusted as the training progresses, irrespective of the initial choice of configuration. If not adjusted dynamically, the training might get stuck in a bad minima, and no amount of training time can recover it. In this work, we focus on learning rate which is the foremost hyperparameter that one seeks to tune when training a deep learning model to get favourable results.

Certain auto-tuning and adaptive variants of SGD, such as AdaGrad (Duchi et al., 2011) , Adadelta (Zeiler, 2012) , RMSProp (Tieleman & Hinton, 2012) , Adam (Kingma & Ba, 2015) among others have been proposed that automatically adjust the learning rate as the training progresses, using functions of gradient. Yet others have proposed fixed learning rate and/or batch size change regimes (Goyal et al., 2017) , (Smith et al., 2018) for certain data set and model combination.

In addition to traditional natural learning tasks where a good LR regime might already be known from past experiments, adversarial training for generating robust models is gaining a lot of popularity off late. In these cases, tuning the LR would generally require time consuming multiple experiments, since the LR regime is unlikely to be known for every attack for every model and dataset of interest1 . Moreover, new models are surfacing every day courtesy the state-of-the-art model synthesis systems, and new datasets are also becoming available quite often in different domains such as healthcare, automobile industy etc. In each of these cases, no prior LR regime would be known, and would require considerable manual tuning in the absence of a universal method, with demonstrated effectiveness over a wide range of tasks, models and datasets. Wilson et al. (2017) observed that solutions found by existing adaptive methods often generalize worse than those found by non-adaptive methods. Even though initially adaptive methods might display faster initial progress on the training set, their performance quickly plateaus on the test set, and learning rate tuning is required to improve the generalization performance of these methods. For the case of SGD with Momentum, learning rate (LR) step decay is very popular (Goyal et al., 2017) , (Huang et al., 2017) , ReduceLRonPlateau2 . However, in certain junctures of training, increasing the LR can potentially lead to a quick, further exploration of the loss landscape and help the training to escape a sharp minima (having poor generalisation Keskar et al. (2016) ). Further, recent works have shown that the distance traveled by the model in the parameter space determines how far the training is from convergence Hoffer et al. (2017) . This inspires the idea that increasing the LR to take bigger steps in the loss landscape, while maintaining numerical stability might help in better generalization.

The idea of increasing and decreasing the LR periodically during training has been demonstrated by Smith (2017); Smith & Topin (2017) in their cyclical learning rate method (CLR). This has also been shown by Loshchilov & Hutter (2016) , in Stochastic Gradient Descent with Warm Restarts (SGDR, popularly referred to as Cosine Annealing with Warm Restarts). In CLR, the LR is varied periodically in a linear manner, between a maximum and a minimum value, and it is shown empirically that such increase of learning rate is overall beneficial to the training compared to fixed schedules. In SGDR, the training periodically restarts from an initial learning rate, and then decreases to a minimum learning rate through a cosine schedule of LR decay. The period typically increases in powers of 2. The authors suggest optimizing the initial LR and minimum LR for good performance. Schaul et al. (2013) had suggested an adaptive learning rate schedule that allows the learning rate to increase when the signal is non-stationary and the underline distribution changes. This is a computationally heavy method, requiring computing the Hessian in an online manner.

Recently, there has been some work that explore gradients in different forms for hyperaparameter optimization. Maclaurin et al. (2015) suggest an approach by which they exactly reverse SGD with momentum to compute gradients with respect to all continuous learning parameters (referred to as hypergradients); this is then propagated through an inner optimization. Baydin et al. (2018) suggest a dynamic LR-tuning approach, namely, hypergradient descent, that apply gradient-based updates to the learning rate at each iteration in an online fashion.

We propose a new algorithm to automatically determine the learning rate for a deep learning job in an autonomous manner that simply compares the current training loss with the best observed thus far to adapt the LR. The proposed algorithm works across multiple datasets and models for different tasks such as natural as well as adversarial training. It is an 'optimistic' method, in the sense that it increases the LR to as high as possible by examining the training loss repeatedly. We show through rigorous experimentation that in spite of its simplicity, the proposed algorithm performs surprisingly well as compared to the state-of-the-art.

Our contributions::

• We propose a novel, and simple algorithmic approach for autonomous, adaptive learning rate determination that does not require any manual tuning, inspection, or pre-experimental discovery of the algorithmic parameters. • Our proposed algorithm works across data sets and models with no customization and reaches higher or comparable accuracy as standard baselines in literature in the same number of epochs on each of these datasets and models. It consistently performs well, finding stable minima with good generalization and converges smoothly. • Our algorithm works very well for adversarial learning scenario along with natural training as demonstrated across different models and datasets. • We provide extensive empirical validation of our algorithm and convergence discussion.

2. PROPOSED METHOD:

We propose an autonomous, adaptive LR tuning algorithm 1 towards determining the LR trajectory during the course of training. It operates in two phases: Phase 1: Initial LR exploration, that strives to find a good starting LR; Phase 2: Optimistic Binary Exploration. The pseudocode is provided at Algorithm 1. For the rest of the paper, we refer to the Automated Adaptive Learning Rate tuning algorithm as AALR in short. Halve LR: η 0 ← η 0 /2.

8::

Reload θ, reset optimizer with LR= η 0 , and reset counter i = 0. The notation used in the following description is as follows. Patience: p, Learning rate: η, best loss L * , current loss L. Model θ, Loss function J(θ). L * is initialized as the J(θ) after initializing the model, before training starts.

Phase 1: Initial LR exploration Phase 1 starts from an initial learning rate η = 0.1, and patience p = 10. It trains for an epoch, evaluates the loss L, and compares to the best loss L * . If L < L * , the L * is updated, and it continues training for another epoch. Otherwise, the model θ is reloaded and re-initialized, LR is halved η = η/2, and optimizer is reset with the new LR. The patience counter is reset. This continues till a stable LR is determined by the algorithm, in which it trains at this LR for p epochs. The loss L * , the model θ and the optimizer state after Phase 1 is saved in a checkpoint.

Phase 2: Optimistic Binary Exploration:

In this phase, AALR keeps the learning rate η as high as possible for as long as possible at any given state of the training. Phase 2 starts by doubling LR to 2η, and setting p = 1. After training for p + 1 epochs, firstly AALR checks if the loss is NAN. In this case, the checkpoint (model θ and optimizer) corresponding to the best loss along with the best loss value L * are reloaded. Then LR is halved η = η/2, patience is doubled, and the training continues. If instead, the loss is observed to decrease compared to the best loss, L < L * , then L * is updated, and the corresponding model θ, optimizer and L * are updated in checkpoint. This is followed by doubling the LR η = 2η, resetting p to 1 and continuing training for the next p + 1 epochs.

On the other hand, if L ≥ L * , AALR trains for another p + 1 epochs and check the loss L. This is because as informally stated before, AALR is 'optimistic' and persists in the high LR for some more time. (In case, the newly evaluated loss is NAN, the previous approach is followed.) However, if the new loss L ≥ L * , then AALR finally lowers the LR. AALR halves the LR η = η/2, doubles patience p = 2p, and continues training for p + 1 epochs. If however, the loss had decreased, L < L * , the previous approach is followed: i.e., it doubles the LR η = 2η, resets the patience p = 1, updates best loss and checkpoint, and repeats training for p + 1 epochs. The above cycle repeats till the stopping criterion is met. For ease of exposition, the pseudocode is given in Algorithm 1

3. MOTIVATION AND RELATED WORK:

Increasing the LR optimistically can potentially help the training to escape saddle points that slow down the training, as well as find flatter minima with good generalization performance. This is inspired mainly from the following observations in the literature. Dauphin et al. (2014) suggest that saddle points slow down the training of deep networks. Xing et al. (2018) states that SGD moves in valley like regions of the loss surface in deep networks by jumping from one valley wall to another at a height above the valley floor which is determined by the LR. Large LR can help in generalization by helping SGD to quickly cross over the valley floor as well as its barriers, to travel far away from the initialization point in a short time. Similarly, Hoffer et al. (2017) describe the initial training phase as a high-dimensional "random walk on a random potential" process, with an "ultra-slow" logarithmic increase in the distance of the weights from their initialization.

From the above discussion, it seems that if one could increase the step size or LR continuously (as long as stability is maintained), it might considerably speed up the increase in distance of the weights from the initialization point, making the initial ultra-slow diffusion process faster. In this way, further exploration of the loss landscape might be possible, leading to better generalization.

The idea of increasing the LR has been explored by algorithms like SGDR and CLR. In SGDR, the LR is reset to a high value in a periodic manner; this is referred to as warm restart. After this, the LR decays to a low value following a cosine annealing schedule. In CLR, the LR is increased and decreased linearly in a periodic manner. While the regular increase in LR in most cases, probably helps generalization and helps in finding flatter minima, they follow a preset method, that does not depend on the training state or progress. Detection of convergence also becomes difficult due to heavy fluctuation in the training output (which happens due to the periodic nature of these methods). Moreover, the authors of each of these methods suggest tuning the parameters of the algorithm for better performance. Even though Schaul et al. (2013) suggest a method that uses information about the state and distribution, it is computationally heavy method. Similarly, hypergradient descent due to Baydin et al. (2018) requires additional computation of gradients. Moreover, it requires tuning of initial LR and introduces additional hyperparameters to be tuned, such as the learning rate for the LR itself.

We propose the simple idea of exploring LR in a binary fashion, without requiring any parameter tuning. This is an adaptive LR tuning algorithm that tries to follow the training state and set the LR accordingly. Increasing LR for better generalization through exploration (and also, potential acceleration of initial phase of SGD) are the main motivations for the optimistic doubling. At the same time, once SGD is in the vicinity of a good minimum, LR might need to be reduced to access the valley. Hence, if the algorithm observes that the loss is not reducing even after a few 'patience' iterations, it halves the LR. The reduction is kept conservative at 1/2 to encourage finding flatter minima.

The automated adaptive LR algorithm we propose achieves good generalization in all cases, including adversarial scenario, and converges smoothly in roughly the same time as LR-tuned SGD baselines available in the literature and community.

4. CONVERGENCE DISCUSSION:

Convergence analysis of SGD typically requires that the sequence of step sizes, or, learning rates used during training satisfy the following conditions:

∞ t=1 η t = ∞ and ∞ t=1 η 2 t < ∞.

Consider an optimal stochastic gradient approach OPT that any point in time has oracle access to (and applies) the highest value of learning rate, that would be amenable for good training (ensuring fast convergence and good generalization). The sequence of LRs chosen by OPT satisfy the above condition. The sequence ensure that OPT will converge (to a good generalization), at the same time, the convergence is the fastest since the step sizes or LRs are kept as high as possible. Let the LR of OPT at any epoch t be denoted as ρ t .

One can define OPT as the following:

Definition: An optimal oracle SGD, with LR ρ t at any epoch t, such that the following properties hold:

1. ∞ t=1 ρ t = ∞, 2. ∞ t=1 ρ 2 t < ∞, 3.

Any first order stochastic gradient-based algorithm that has the same location in parameter space as OPT at the start of an epoch t must have LR ≤ ρ t , otherwise training will diverge here onward (in other words, gradients will explode and the training of the network cannot be recovered from here).

4. Any first order stochastic gradient based algorithm starting from the same location in parameter space as OPT and achieving similar generalization for a given training task, will require at least as many epochs as OPT for convergence.

We will compare AALR with OPT, and show that the sequence of LRs chosen by AALR follow the sequence of LRs of OPT with a bounded delay under some assumptions. We also show that divergence will not happen.

A typical well-tuned SGD algorithm can be thought to be a proxy for OPT for a given scenario, and hence this analysis will bound the convergence time of AALR with respect to LR-tuned SGD for the same problem.

In a typical step decay LR-regime for SGD, the LR does not increase, but generally decreases at certain intervals by some factor γ ∈ {2, . . . , 10}. For standard LR schedules, one can see that the following rule-of-thumb holds: the number of epochs ∆ in between two consecutive LR changes is directly proportional to γ. In fact one can see that for standard regimes, ∆ ≥ cγ, where c ≥ 2. (For example, change by a factor of 2 happens at every 5 epochs or more, or, change by a factor of 10 happens every 30 epochs or more). Such typical LR regimes are often designed out of observing of loss plateauing. We assume that OPT has a similar behaviour in the following analysis.

4.1. BOUNDING THE DELAY IN CONVERGENCE DUE TO DOUBLING:

Let the LR of OPT at any epoch t be denoted as ρ t and that of AALR be denoted as η t . We assume that Phase 1 has estimated a stable initial LR η 0 ≤ ρ 0 , and that both AALR and OPT are roughly in the same space in the loss valley at the start of Phase 2, denoted as epoch 1 (for simplicity). In the following, we refer to decrease in loss compared to the best observed loss thus far as an improvement in state.

Assuming that the loss surface is smooth, the loss will continue to decrease for AALR, as long as η t ≤ ρ t , and it will start increasing otherwise.

We first argue that AALR will not diverge. From Algorithm 1, it can be seen that every time state improves, the checkpoint is updated. If and when, due to doubling (or due to initial LR), loss diverges and goes to NAN, the last checkpoint is reloaded, LR is halved and training continues. This will continue till a stable LR is reached, and loss is no longer NAN. In this way, AALR can avoid losing way due to exploding gradients, caused by undue increase in LR.

Now, let us consider the case, when OPT has increased its LR. AALR, by design is always optimistically doubling the LR whenever state improves. For an increase in ρ t by a factor γ, it can be seen that AALR will require 2 log 2 γ epochs. This is because, when state improves, patience p will be reset to 1, LR will be doubled, and the state will be checked again after training for p + 1 = 2 epochs.

We would next show that AALR reaches the same or lower LR as OPT (with some delay) every time OPT reduces LR.

AALR starts with η 1 = 2ρ 0 and trains for p + 1 = 2 epochs and checks the state. If OPT has maintained ρ(t) at ρ 0 , then state will not improve. AALR will train for another 2 epochs, and then reduce LR by half, and double the patience. It trains at this LR for 2p + 1 = 3 epochs. Therefore, effective training for AALR is for 3 epochs out of the 7 epochs it spent. Now if the state improves, AALR would double the LR and the above cycle would repeat till we come to the state where OPT needs to reduce the LR for making progress. Let there be k such cycles, such that OPT has trained for 3(k -1) < q ≤ 3k epochs and AALR has trained for 7k epochs to arrive at roughly the same location in parameter space (assuming bounded gradients) and both have the same LR. Now, let OPT reduce its LR by γ, i.e., ρ q+1 = ρ p /γ (For simplicity, let us assume that γ is a power of 2). AALR would be first doubling the LR to η 7k+1 = 2η 7k = 2ρ q (since its state was improving till 7k epochs), and patience p will be reset to 1. It will need to reduce its LR 1 + log 2 γ times before it observes an improves in state (by assumption on OPT maintaining the highest possible LR for training progress). It will train for 2(p + 1) = 2(1 + 1) = 4 epochs, then halve the LR, double p, train for 2(p + 1) = 2(2 + 1) = 6 epochs, and repeat this for n = 1 + log 2 γ times. One can see by induction that AALR will be spending a total of N AALR = n-1 i=0 2(2 i +1)=2n+2 n+1 -1 epochs. At this time, p = 2 n-1 . Now, AALR will train for 2(p + 1) = 2 n + 2 epochs at this LR, after which it will observe an improvement in state. Note that by the earlier observation regarding typical LR regimes and the behavior of OPT, OPT would train for at least ∼ 2γ = 2 n epochs at this new LR. Hence, AALR has trained for a total of roughly twice the number of epochs as OPT, and at the new LR for roughly the same number of epochs as OPT. Therefore, both are now at a similar location in parameter and loss space. After this AALR would again double the LR, and the earlier cycle would repeat for another k times, such that OPT would have trained for 3(k -1) < p ≤ 3k epochs and AALR would have trained for 7k epochs till the next LR change happens in OPT. Therefore, one can see that AALR would take 2 times the number of epochs as OPT to reach the same or lower LR, every time OPT lowers the LR.

Since the LR sequence of AALR follows the LR sequence of OPT with some finite delay, it can be argued that the following convergence requirements on the LR sequence hold for AALR: (1)

∞ t=1 η t = ∞, and, (2) ∞ t=1 η 2 t < ∞.

In practice, we observe that AALR converges in around the same time as LR-tuned SGD.

5. EXPERIMENTS:

We trained with AALR on several model-datasets combinations, in multiple scenarios such as natural training, as well as adversarial training. We observed that AALR achieve similar or better accuracy as the state-of-the-art baselines.

We have compared to standard SOTA (SGD or other) LR tuned values reported in the literature and with three other adaptive LR tuning algorithms, SGDR (Cosine Scheduling with Warm Restarts), CLR (Cyclic Learning Rates), ADAM, and Hypergrad (Baydin et al., 2018 ) (few runs were not complete). Since the principle claim of AALR is that it is a completely autonomous adaptive approach that does not require any tuning, for fair comparison, we have not tuned the parameters of any other adaptive approaches compared with. Since AALR does not have any tunable parameters by design, sensitivity analyses experiments were not performed for AALR.

Additionally, we have done some experiments examining the relationship of batchsize andor entropy with the learning rate. We find the AALR finds LR trajectory that roughly obey the square root rule as suggested by Hoffer et al. (2017) and 5.1 SETTINGS, DATASETS AND MODELS Experiments were done in PyTorch in x86 systems using 6 cores and 1 GPU. Where baselines for SGDR (Loshchilov & Hutter, 2016) and CLR are not available in literature, the PyTorch provided implementations of the corresponding LR schedulers with default settings were used (available here https://pytorch.org/docs/stable/_modules/torch/optim/lr_ scheduler.html. For Hypergrad (we have tried SGDHD optimizer), we have used the code available at https://github.com/gbaydin/hypergradient-descent.

We have tested on datasets CIFAR10 and CIFAR100 using standard data augmentation for both, on models Resnet-18, WideResnet-28-10 with dropout and cutout, WideResnet-34-10 (for adversarial scenario only), SimplenetV1, and Vgg16 with and without batch normalization. For Resnet-18, WideResNets and Vgg16 models, we ran all algorithms for 200 epochs at a batch size of 128 for both datasets. For SimplenetV1, we ran for 540 epochs and used batch size of 100 and 64 for CIFAR10 and CIFAR100 respectively for all algorithms. The code for SimpleNetV1 was obtained from https://github.com/Coderx7/SimpleNet_Pytorch. The code for Resnet-18 and WideResnets was obtained from https://github.com/uoguelph-mlrg/ Cutout. The code for Vgg16 was obtained from https://github.com/chengyangfu/ pytorch-vgg-cifar10/blob/master/main.py. We use cross entropy loss in all cases.

The batchsize and number of epochs used in each case was as per the baselines in literature, i.e., the reported SOTA results for LR-tuned SGD. The baseline (and in where available, adaptive algorithms') results were reported from the cited sources. For the relationship of batchsize with learning rate, we use FMNIST on ResNet-20.

5.2. OBSERVATIONS:

Our experiments comprehensively show that AALR is a state-of-the-art automated adaptive LR tuning algorithm that works universally across models-datasets for both natural and adversarial training. It is either better or comparable to LR-tuned baselines and other adaptive algorithms uniformly and consistently, with a smooth convergence behavior. It effective across batchsizes, without requiring any tuning. This makes the case that for new models andor datasets, AALR should be a reliable LR algorithm of choice, in the absence of any prior tuning or experimentation.

For the adaptive algorithms we compared with, SGDR though performs comparable with AALR in most cases of or natural training, it catastrophically failed for at least two natural training cases (which indicates it require tuning of either initial LR or some other parameters, and hence not a completely stand alone automated approach) and moreover, it generally did significantly worse than AALR for adversarial training. CLR achieved slightly lower accuracy compared to AALR in most cases of natural training, and in adversarial training its performance fluctuated on a case by case basis. ADAM generally converged to lower accuracy and significantly lower in adversarial scenario, and furthermore, it catastrophically failed in two cases of natural training, which shows it requires extensive tuning of parameters.

AALR was consistently top-level in every case, which makes the case for its universal and reliable applicability, especially when new models/datasets/training tasks surface for which prior tuning or information is not available.

5.3. NATURAL TRAINING:

The baseline values reported have the following sources:

• Resnet-18, CIFAR10 as reported by DeVries & Taylor (2017),

• WideResnet-28-10, CIFAR10 and CIFAR100 baseline as reported by Zagoruyko & Komodakis (2016) 

5.5. RELATIONSHIP OF BATCHSIZE AND ENTROPY WITH LEARNING RATE:

The relationship of learning rate with batchsize for training DNNs is a well-studied topic. In particular, a linear relationship is suggested by Goyal et al. (2017) ; Smith (2017) and also by Chaudhari & Soatto (2017) . The linear scaling rule suggests that the ratio of learning rate to batch size: η/B should remain constant for similar generalization across batch sizes for a given dataset and model). This has been exploited to achieve good generalization across batchsizes by Goyal et al. (2017) and Smith (2017). Chaudhari & Soatto (2017) suggest that this ratio should not be too small to maintain the implicit regularization of SGD, otherwise the entropy would go to zero and generalization would be poor. This observation is in line to that by Keskar et al. (2016) where small batchsizes were seen to be essential for good generalization. A square root scaling rule of learning rate and batchsize (η ∝ √ B) was suggested by Hoffer et al. (2017) .,

We performed a simple experiment (thanks to the suggestion by anonymous reviewers) to observer if AALR can work across batch sizes, adjusting the LR to achieve same generalization. (Note that in the earlier experiments the batch sizes chosen were as per the baseline settings as given in the literature, which already shows the effectiveness of AALR across batch sizes, however this is a more direct experiment to observe the same). We used a different dataset and model than the ones tested earlier: FMNIST on Resnet-20, and tried batch sizes 32, 128 and 512. We obtained similar generalization performance across all the batch sizes. In particular, the peak accuracy were as follows: 95.33 for BS 128, 95.3 for BS 512, and 94.85 for BS 32. (The baseline for the above is 95.63 given by Zhong et al. (2017) . On BS 128, we tried this with Random Erasing (p = 0.5), and again got similar generalization: 95.47 (reported baseline is 95.98). The corresponding LR trajectory is given in Figure 1 . Interestingly, AALR seems to find trajectories that follow the square root rule (with a lot of fluctuations), rather than linear. In particular, if one observes closely around epochs 120 and 200 (places where the LR changes across the batchsizes), this would be apparent. Around 120, BS 32 has LR oscillating between 0.0 and 0.025, BS 128 has LR oscillating between 0.1 and 0.5 and BS 512 has LR oscillating between 0.2 and 0.1. Around epoch 200, for BS 32, LR is between 0.025 and 0.0125, for BS 128 between 0.5 and 0.025, for BS 512, between 0.1 and 0.5. Furthermore, we compared the trajectory obtained on FMNIST, ResNet-20 on the same batch size 128, with and without random erasing (Figure 2 ). Random erasing is the process of randomly replacing the image pixels with random values for certain rectangular patches with certain probability, commonly used for data augmentation for generating more robust models and improving generalization. Clearly random erasing is a regularizing technique that increases entropy. Interestingly, the LR trajectory shows that AALR is much more conservative in the case of random erasing, in the sense that it maintains a much lower LR compared to the standard case. This fits very well with the observation by Chaudhari & Soatto (2017) that the entropy or implicit regularization in SGD depends on the ratio of η/B. Since in this case, the entropy or regularization is high due to random erasing, AALR lowers η and decreases the implicit entropy of SGD to maintain stability and converge to the similar generalization.

6. CONCLUSION:

We have presented an autonomous adaptive algorithm AALR that works without tuning across models, datasets, batchsizes and natural or adversarial training to achieve generalization performance comparable or better than SOTA LR-tuned values. Moreover, it is a stable algorithm and consistently converges smoothly, unlike other SOTA adaptive algorithms compared with. This is very promising as it possibly implies that AALR can be reliably applied for new datasets or models or adversarial attacks without requiring extensive manual experimentation. In the future, we would like to try this on tasks other than image classification, and develop a rigorous theoretical justification behind its consistent performance.

:

For example, one can see a piecewise LR schedule given byMadry et al. (2018) at https://github. com/MadryLab/cifar10_challenge/blob/master/config.json for a particular model.

For eg. https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ ReduceLROnPlateau.

For readers unfamiliar with adversarial training and FGSMPGD attacks, details can be found inMadry et al. (2018). We have given a short description here. Adversarial training aims at solving the min-max saddle point problem defined as: min θ ρ(θ) where ρ(θ) = E (x,y)∼D [max δ∈S L(θ, x + δ, y)] in contrast to standard training which simply aims at minimizing E (x,y)∼D[L(θ, x, y)]. FGSM simply perturbs the data as x + ε sgn(∇xL(θ, x, y)), whereas PGD does it for k steps iteratively as follows: Πx+S (x t + αεsgn(∇xL(θ, x, y))). White Box attack refers to the attacker having access to the model parameters.

Figures and Tables:

Figure 1: FMNIST, LR trajectory across batch sizes.

Figure 2: FMNIST, LR trajectory with and without random erasing on same batchsize.

Save checkpoint θ and L * . Reset optimizer with LR= η t+1 .

Table content: <table><tr><td>9:</td><td>else</td></tr><tr><td>10:</td><td>L  *  ← L (Update best loss).</td></tr><tr><td>11:</td><td>end if</td></tr><tr><td colspan="2">12: end while</td></tr><tr><td colspan="2">13: Save checkpoint θ and L * .</td></tr><tr><td colspan="2">14: PHASE 2: Start Optimistic Binary Exploration</td></tr><tr><td colspan="2">15: Double LR η t ← 2η 0 . Patience p ← 1.</td></tr><tr><td colspan="2">16: while t &lt; T do</td></tr><tr><td>17:</td><td>Train for p epochs. Increment epoch number t.</td></tr><tr><td>18:</td><td>Evaluate new loss L ← J(θ).</td></tr><tr><td>19:</td><td>if L is NAN then</td></tr><tr><td>20:</td><td>Halve LR: η t+1 ← η t /2.</td></tr><tr><td>21:</td><td>Load checkpoint θ and L * . Reset optimizer with LR= η t+1 .</td></tr><tr><td>22:</td><td>Double patience p t+1 ← 2p t . Continue.</td></tr><tr><td>23:</td><td>end if</td></tr><tr><td>24:</td><td>if L &lt; L  *  then</td></tr><tr><td>25:</td><td>Update L  *  ← L. Save checkpoint θ and L * .</td></tr><tr><td>26:</td><td>Double LR η t+1 ← 2η t . Set patience p = 1.</td></tr><tr><td>27:</td><td>else</td></tr><tr><td>28:</td><td>Train for another p epochs. Increment epoch number t.</td></tr><tr><td>29:</td><td>Evaluate new loss L ← J(θ).</td></tr><tr><td>30:</td><td>if L &lt; L  *  then</td></tr><tr><td colspan="2">31: Update L  32: Double LR η t+1 ← 2η t . Set patience p = 1.</td></tr><tr><td>33:</td><td>else</td></tr><tr><td>34:</td><td>Halve LR: η t+1 ← η t /2.</td></tr><tr><td>38:</td><td>end if</td></tr><tr><td>39:</td><td>end if</td></tr><tr><td>40:</td><td>end if</td></tr><tr><td colspan="2">41: end while</td></tr></table>

Natural Training on CIFAR10. Comparison of test accuracy of model trained with AALR with baselines and with those obtained by training using different adaptive learning rate techniques on various models.*:With AALR, ADAM drastically improves. Peak accuracy 78.46 from 10.Here we outline the results and observations from adversarial training. We observe that AALR is particularly effective in Adversarial Training and achieves (to the best of our knowledge) state-ofthe-art adversarial test accuracy for FGSM attack in a White Box model 3 .It generally does significantly better compared to the other adaptive algorithms compared with and convergence is easier to detect, unlike the other methods. It would be interesting to explore theoretical justification regarding the effectiveness of AALR in first-order adversarial training, and it might be related to the loss landscape of the min-max saddle point problem defined byMadry et al. (2018). The effectiveness of AALR in training models for different and new scenarios is clearly underlined by these experiments. All attacks are ∞ within [0, 1] ball. All models were trained on CIFAR10 for 200 epochs, using batch size of 128.The baseline we could find is as follows. For FGSM White Box atack on CIFAR10,Madry et al. (2018) report 56.1%. For other cases, we could not find baseline figures for these. Therefore, we consider 56.1% as the representative baseline for each of the cases in FGSM.For PGD,Madry et al. (2018) had reported 50.0% for k = 7 steps, and 45.8% for k = 20 steps. We performed PGD for k = 10 steps on AALR on CIFAR10 on WRN-28-10, and obtained 52.83%. On SimpleNetV1, PGD with 10 steps on CIFAR10 with AALR gives 45.32%.

Table content: <table><tr><td colspan="7">, and corresponding SGDR values as reported by Loshchilov &amp; Hutter</td></tr><tr><td>(2016)</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">• WideResnet-28-10 with Dropout and Cutout, CIFAR10 and CIFAR100 as reported by De-</td></tr><tr><td>Vries &amp; Taylor (2017)</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">• SimplenetV1, CIFAR10 and CIFAR100 as reported at HasanPour et al. (2016) and</td></tr><tr><td colspan="5">https://github.com/Coderx7/SimpleNet_Pytorch</td><td/><td/></tr><tr><td colspan="7">• Vgg16 with and without Batch Normalization for CIFAR10 as reported at https:</td></tr><tr><td colspan="7">//github.com/chengyangfu/pytorch-vgg-cifar10 and http://torch.</td></tr><tr><td colspan="7">ch/blog/2015/07/30/cifar.html (the former values are higher). (CIFAR100 for</td></tr><tr><td colspan="6">Vgg16 values were not reported at these places, hence not provided in the table.)</td><td/></tr><tr><td>Model</td><td>Baseline</td><td colspan="3">AALR SGDR CLR</td><td>ADAM</td><td>Hypergrad</td></tr><tr><td>Resnet-18</td><td>95.28</td><td>94.94</td><td colspan="2">94.81 93.85</td><td>93.07</td><td>84.48</td></tr><tr><td>SimpleNet-V1</td><td>95.51</td><td>95.17</td><td colspan="2">95.44 93.66</td><td>93.99</td><td>87.01</td></tr><tr><td>Vgg16</td><td>91.4, 92.63,</td><td>92.16</td><td colspan="3">10.00 91.95 10.00 (78.46*)</td><td>10.00</td></tr><tr><td>Vgg16-BN</td><td colspan="2">92.45, 93.86 93.23</td><td colspan="2">93.56 92.65</td><td>91.48</td><td>85.05</td></tr><tr><td>WRN-28-10 (Dropout)</td><td>96.00</td><td>95.75</td><td colspan="2">95.91 95.34</td><td>94.01</td><td>-</td></tr><tr><td>WRN-28-10 (Dropout + Cutout)</td><td>96.92</td><td>96.44</td><td>96.6</td><td>95.42</td><td>95.46</td><td>-</td></tr><tr><td>Model</td><td colspan="5">Baseline AALR SGDR CLR ADAM</td><td/></tr><tr><td>WRN-28-10 (Dropout)</td><td>79.96</td><td>80.45</td><td colspan="2">80.26 78.63</td><td>73.67</td><td/></tr><tr><td>SimpleNet-V1</td><td>78.51</td><td>78.21</td><td colspan="2">77.47 74.02</td><td>73.48</td><td/></tr><tr><td>Vgg16</td><td>-</td><td>65.03</td><td colspan="2">10.00 67.79</td><td>10.00</td><td/></tr><tr><td colspan="7">Table 2: Natural Training on CIFAR100: Comparison of test accuracy of model trained with AALR</td></tr><tr><td colspan="7">with baselines and with those obtained by training using different adaptive learning rate techniques</td></tr><tr><td>on various models.</td><td/><td/><td/><td/><td/><td/></tr><tr><td>5.4 ADVERSARIAL TRAINING</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">In the process of these experiments, we discover that SimpleNetV1 is a very effective adversarially</td></tr><tr><td colspan="7">strong model with respect to CIFAR10, when trained especially with AALR for FGSM attack ( =</td></tr><tr><td colspan="7">8/255 and α = 2/255) in White Box model. This is a light weight model. Adversarial training</td></tr><tr><td colspan="7">generally being a compute heavy and time consuming process, becomes much easier and faster</td></tr><tr><td>with it.</td><td/><td/><td/><td/><td/><td/></tr></table>

Adversarial Training on CIFAR10: Peak Adversarial Accuracy on FGSM White Box attack, obtained on different models by training with AALR and other adaptive algorithms.

Table content: <table><tr><td colspan="4">WRN-28-10(A) SimpleNetV1(A) ResNet-18(A) SimpleNetV1(B)</td></tr><tr><td>52.83</td><td>45.32</td><td>50.3</td><td>31.76</td></tr></table>

AALR for adversarial Training on CIFAR10 and 100: Peak Adversarial Accuracy on (A)PGD attack on CIFAR10, (B)FGSM on CIFAR100.

Table content: <table/>

Peer review (iclr2020):
The paper considers the problem of automated adaptation of learning rate during (deep) neural network training. The use cases described are standard and adversarial training for image classification. Given the wide use of DNNs in computer vision (and other areas), learning rate tuning is clearly an important problem and is being actively researched. The proposed learning rate adaptation procedure consists of a straightforward combination of learning rate halving/doubling and model checkpointing. Experimental results from implementing the adaptive learning rule for standard and adversarial training on CIFAR are provided. Multiple architectures are tested in each setting. The paper claims a primary advantage of the proposed learning rule to be that it requires no tuning as opposed to other rules such as SGD, Adam. My decision is to reject the paper due to methodological issues with the experiments and lack of evidence wrt/ dataset variety. The paper should be considered a work-in-progress that may have potential in a more focused setting, e.g., adversarial training as described in the paper. *** The major claim of the proposed algorithm not requiring any manual tuning is technically true but misleading. The algorithm does have parameters (SGD momentum, batch size, initial learning rate, patience) with values that were set somehow. In fact, a major methodological issue with the experiments is that the reader does not know if the datasets were used to both set these values and to assess performance, i.e., there are no obvious *held-out* datasets. Also, there is no rigorous or even informal justification of the settings. It could be that the paper is arguing that the specific values will result in competitive, if not better, performance than baselines across a variety of datasets - unfortunately, only two datasets are utilized in the experiments, and one, CIFAR10, is not considered challenging. This leads to the second issue with the paper: the experimental validation is not extensive wrt/ datasets which is significant given that the form of the evidence for the proposed method is almost entirely empirical. Additionally, I don*t agree that competitor algorithms should not be tuned b/c the proposed method does not require tuning. Even if the proposed method does not require tuning (as stated previously, I don*t believe this to be accurate), that does not imply a fair comparison precludes tuning competitors via, e.g., cross validation. The only relevant quantities are final test-set performance and total training time/resources required. The well-known interdependence between learning rate and batchsize as noted in e.g., Hoffer et al. (2018), is not addressed by the experiments. Batchsizes in the experiments vary, but no justification is provided for how these are selected. Finally, the paper is unfinished as some experimental runs were not complete at the time of submission. On the positive side, the general point about the necessity of learning rate tuning for adversarial training (described in the fourth paragraph of the introduction) is a very good one, and there may be an opportunity for a more focused application of the proposed algorithm perhaps among further datasets and considering additional, alternative attacks. *** Suggestions for improvement / questions (related to decision): * It should not be a challenge to find more image classification datasets to include in the experimental comparison: SVHN, Fashion MNIST, Imagenet, ... Using these, the paper can either follow the standard train/test methodology *across datasets*, i.e., split the meta-dataset into train/test, and/or provide a more compelling body of evidence for the proposed method. Also, the performance dependence on batchsize amongst the proposed algorithm and competitors should be investigated experimentally. * The Baydin et al. (2018) algorithm should be added to the set of competitors since it would provide a relatively easy* reference point wrt/ *hypergradient* approaches. I don*t agree with the statement in the related work section that this entails *additional computation of gradients.* *In the sense that the rule should be straightforward to implement. * The convergence analysis assumption that the optimal oracle SGD follows typical learning rate regimes motivated by loss plateauing seems to be in direct contradiction to the sentiment expressed in the cited Hoffer et al. (2018) paper that such *rules of thumb* may be misguided. Can the authors discuss the appropriateness of their assumption wrt/ this point? Also, in the convergence analysis, the phrase *in expectation* is used twice. This has a specific probabilistic meaning, but appears to be used heuristically in this section. Can the authors clarify whether this usage is informal or formal? If the latter is true, it would be better to provide a more formal convergence argument that explicitly takes the inherent randomness into account. *** Editorial comments (not related to decision): * Introduction: The first two sentences of the second paragraph, particularly the second, would do well to have an accompanying reference or references. * Proposed method: Even as an informal statement, the second sentence of the second paragraph under the Phase 2 sub-heading is problematic. The proposed method does not *resist* lowering the learning rate *for as long as possible* so much as it doesn*t lower the learning rate for a fixed number of epochs (algorithm parameter). * *Adversarial training* section (5.4): The paper assumes the reader is familiar with the terms *FGSM*, *white box*, and parameters epsilon and alpha since these are referred to w/o description. Perhaps a short (2-3 sentence) description of the adversarial scenario could be added? * Experiments: It would be good for the paper to include RMSProp and Adagrad results to the experimental tables as these rules are both readily available for use and widely used. * Experiments: Is the reporting of the peak accuracy standard in the literature? * Experiments: I want to give the paper credit for performance on CIFAR100, but this is difficult without explicit points of comparison. This can be easily remedied by including SOTA performance values (along with appropriate references) in the tables or text. * (Potential) Typos: Proposed method algorithm description: Requirements has a weight decay parameter which seems strange given that the algorithm is performing automated learning rate adaptation... The epoch counter is incremented in line 5, but not reset prior to Phase 2. Does this mean that Phase 1 training epochs are counted toward the total (T)? Line 7 should be eta_t <- eta_0 / 2. The patience counter in line 15 is not utilized below. Line 23 could/should be an else statement.