A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
Authors: Koyel Mukherjee
Alind Khare
Yogish Sabharwal
Ashish Verma
Venue:  (2020)
Keywords: adaptive LR tuning algorithm
generalization

Abstract: Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach \emph{consistently} achieves top-level accuracy compared to SOTA baselines in the literature in natural training, as well as in adversarial training.

Under review as a conference paper at ICLR 2020
A SIMPLE DYNAMIC LEARNING RATE TUNING ALGO- RITHM FOR AUTOMATED TRAINING OF DNNS
Anonymous authors Paper under double-blind review
ABSTRACT
Training neural networks on image datasets generally require extensive experi- mentation to find the optimal learning rate regime. Especially, for the cases of ad- versarial training or for training newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and mod- els for both natural and adversarial training, without requiring any dataset/model specific tuning. is a stand-alone, parameterless, adaptive approach with no com- putational overhead. We theoretically discuss the algorithm's 's convergence behav- ior. We empirically validate our algorithm extensively. Our results show that our proposed approach consistently achieves top-level accuracy compared to SOTA baselines in the literature in natural as well as adversarial training.
INTRODUCTION
Deep architectures are generally trained by minimizing non-convex loss function via underlying optimization algorithm such as stochastic gradient descent or its variants. It takes a fairly large amount of time to find the best suited optimization algorithm and its optimal hyperparameters (such as learning rate, batch size etc.) for training a model to the desired accuracy, this being a major challenge for academicians and industry practitioners alike. Usually, such tuning is done by initial configuration optimization through grid search or random search (Bergstra et al. 2011 Snoek et al.,
2012 Thornton al.) Recent works have also formulated
bandit problem (Li et. 2017).
However, it has been widely demonstrated that hyperparameters, especially the learning rate often needs to be dynamically adjusted as the training progresses. irrespective of the initial choice of configuration. If not adjusted dynamically, the training might get stuck in a bad minima, and no amount of training time can recover it. In this work, we focus on learning rate which the foremost hyperparameter that one seeks to tune when training a deep learning model to get favourable results.
Certain auto-tuning and adaptive variants of SGD, such as AdaGrad (Duchi et al. 2011 Adadelta (Zeiler 2012), RMSProp (Tieleman & Hinton 2012). Adam (Kingma & Ba 2015) among others have been proposed that automatically adjust the learning rate as the training progresses, us- ing functions of gradient. Yet others have proposed fixed learning rate and/or batch size change regimes (Goya et al. 2017]. (Smith et al. (0188) for certain data set and model combination.
In addition to traditional natural learning tasks where good LR regime might already be known from past experiments, adversarial training for generating robust models is gaining a lot of pop- ularity off late. In these cases, tuning the LR would generally require time consuming multiple experiments, since the LR regime is unlikely to be known for every attack for every model and dataset of interes{] Moreover, new models are surfacing every day courtesy the state-of-the-art model synthesis systems, and new datasets are also becoming available quite often in different do- mains such as healthcare, automobile industy etc. In each of these cases, prior LR regime would be known, and would require considerable manual tuning in the absence of universal method, with demonstrated effectiveness over a wide range of tasks, models and datasets.
Wilson et al. (2017) observed that solutions found by existing adaptive methods often generalize worse than those found by non-adaptive methods. Even though initially adaptive methods might
For example, http://vyyaaa.c//oooo/./mmm///oe htw.aaa1eeeg///oooaaa/ttee/////e
one one see piecewise I.R schedule given by Madry et al. (2018) at ahttpps: //gg /htb. json for particular model.
com comm/Madrrylab/cifaarllll

---

Under review as a conference paper at ICLR 2020
display faster initial progress the training set, their performance quickly plateaus the test set, and learning rate tuning is required to improve the generalization performance of these methods. For the case of SGD with Momentum, learning rate (LR) step decay is very popular (Goyal et al.) (Huagg et al. 2017). CeatuccelRoPPaatttatt creasing the LR can potentially lead to a quick, further exploration of the loss landscape and help the training to escape sharp minima (having poor generalisation Keskar et al.] (2016)) Further, recent works have shown that the distance traveled by the model in the parameter space determines how far the training is from convergence Hoffer et al. (2017). This inspires the idea that increasing the LR take bigger steps in the loss landscape, while maintaining numerical stability might help in better generalization.
However, in certain junctures of training, in-
The idea of increasing and decreasing the LR periodically during training has been demonstrated by Smith (2017): Smith & Topin (2017) in their cyclical learning rate method (CLR). This has also been shown by Loshchilov & Hutter (2016), Stochastic Gradient Descent with Warm Restarts (SGDR, popularly referred to as Cosine Annealing with Warm Restarts). In CLR, the LR is varied period- ically in a linear manner, between a maximum and a minimum value, and it is shown empirically that such increase of learning rate is overall beneficial to the training compared to fixed schedules. In SGDR, the training periodically restarts from an initial learning rate, and then decreases to a min- imum learning rate through a cosine schedule of LR decay. The period typically increases in powers 2f 2. The authors suggest optimizing the initial LR and minimum LR for good performance.
Schaul et al. (2013) had suggested adaptive learning rate schedule that allows the learning rate to increase when the signal is non-stationary and the underline distribution changes. This is a com- putationally heavy method, requiring computing the Hessian an online manner.
Recently, there has been some work that explore gradients in different forms for hyperaparameter optimization. Maclaurin et al. (2015) suggest an approach by which they exactly reverse SGD with momentum to compute gradients with respect to all continuous learning parameters (referred to as hypergradients); this is then propagated through an inner optimization. [aydiin al. (2018) suggest dynamic LR-ttnnig approach, namely, hypergradient descent, that apply gradient-based updates to the learning rate at each iteration in an online fashion.
We propose new algorithm to automatically determine the learning rate for a deep learning job in an autonomous manner that simply compares the current training loss with the best observed thus far to adapt the LR. The proposed algorithm works across multiple datasets and models for different tasks such as natural as well as adversarial training. It is an 'optimistic' method, in the sense that increases the LR to high possible by examining the training loss repeatedly. We show through rigorous experimentation that in spite of its simplicity, the proposed algorithm performs surprisingly well as compared to the -tate-
Our contributions:
We propose a novel, and simple algorithmic approach for autonomous, adaptive learning rate determination that does not require any manual funing, inspection, pre-experimental discovery of the algorithmic parameters.
Our proposed algorithm works across data sets and models with no customization and reaches higher or comparable accuracy as standard baselines in literature in the same num- ber of epochs on each of these datasets and models. consistently performs well, finding stable minima with good generalization and converges smoothly.
Our algorithm works very well for adversarial learning scenario along with natural training as demonstrated across different models and datasets.
We provide extensive empirical validation of our algorithm and convergence discussion.
PROPOSED METHOD
We propose an autonomous, adaptive LR tuning algorithm I towards determining the LR trajectory during the course of training. It operates in two phases: Phase 1: Initial LR exploration, that strives
For eg. Reduueel.ROnP ateau.
https: http:///www.coooooooo/////////// /bac///////eerrraaaaaaaa//a/
2

---

Under review as a conference paper at ICLR 2020
to find good starting LR; Phase 2: Optimistic Binary Exploration. The pseudocode is provided Algorithm [ For the rest of the paper, we refer to the Automated Adaptive Learning Rate tuning algorithm as AALR in short.
Algorithm 1 Automated Adaptive Learning Rate Tuning Algorithm (AALR) for Training DNNs
Require: Model 0, N Training Samples (æ;, Mo)i= Optimizer SGD, Momentum=0.9, Weight De- cay, Batch size, Number epochs T, Loss Function J()). Initial LR f/o = 0.1.
Ensure: Learning Rate π) every epoch t.
1: Initialize: 8, SGD with LR= 17u. best loss L + J(0) (forward pass through initial model).
2: PHASE 1: Start Initial LR Exploration.
3: Set patience p <- 10, patience counter + 0, epoch number t <-
4: while
do
5: Evaluate new loss L + (0) after training for an epoch. Increment i and by each.
6: if L > L* L NAN then
7: Halve LR: 1N + 7o/2.
8: Reload reset optimizer with LR= I7u, and reset counter
9: else
I.* <- (Update best loss).
11: end if
12: end while
13: Save checkpoint 0 and L*.
14: PHASE 2: Start Optimistic Binary Exploration
15: Double LR 17t 2%/. Patience p + 1.
16: while T do
Train for p epochs. Increment epoch number
18: Evaluate new loss L + (f)).
if L is NAN then
20:
Halve LR: 1/++1 + 7/2/2. Load checkpoint and L*. Reset optimizer with LR= 71+1-
22:
Double patience Pt &- 2p.- Continue.
23:
end if
24:
if L < L* then
25:
Update L +- L. Save checkpoint and L*. Double LR 7/+11 + 2n. Set patience p 1.
26:
27:
else
28:
Train for another p epochs. Increment epoch number t. Evaluate new loss < J(0).
29:
L L then
31:
Update L +- L. Save checkpoint 0 and L*.
Double LR M7+11 2n,. Set patience p
33:
else
34:
Halve LR: 7 + 1/:/2.
35:
Double patience Pt+1 +- 2pt-
36:
is NAN then
37:
Load checkpoint 0 and L*. Reset optimizer with LR= 7h+1 end
38:
39:
end if
40:
end if
41:
end while
The notation used in the following description is as follows. Patience: p. Learning rate: n, best loss L" current loss L. Model 0, Loss function J()). L* initialized as the (()) after initializing the model, before training starts.
Phase 1: Initial LR exploration
Phase 1 starts from initial learning rate 7 0.1, and patience p 10. It trains for epoch, evaluates the loss L, and compares to the best loss L". L < L", the L is updated, and continues training for another epoch. Otherwise, the model 0 is reloaded and re-initialized, LR

---

Under review as a conference paper at ICLR 2020
is halved 1 2/2, and optimizer is reset with the new LR. The patience counter is reset. This continues till a stable LR is determined by the algorithm, in which it trains at this LR for p epochs. The loss L*, the model 0 and the optimizer state after Phase I is saved in checkpoint.
Phase 2: Optimistic Binary Exploration
In this phase, AALR keeps the learning rate 7) as high as possible for as long as possible at any given state of the training. Phase 2 starts by doubling LR to 21,, and setting p = 1. After training for + epochs, firstly AALR checks if the loss is NAN, In this case, the checkpoint (model 0 and optimizer) corresponding to the best loss along with the best loss value L are reloaded. Then LR is halved 7 = n/2, patience doubled, and the training continues. If instead, the loss is observed to decrease compared to the best loss, L < L', then L' is updated, and the corresponding model 6, optimizer and I.* are updated in checkpoint. This is followed by doubling the I.R rj = 21y, resetting p to 1 and continuing training for the next p + 1 epochs.
On the other hand, Z L", AALR trains for another p + epochs and check the loss L. This is because as informally stated before, AALR is "optimistic" and persists in the high LR for some more time. (In case, the newly evaluated loss NAN, the previous approach is followed.) However, if the new loss L Z L", then AALR finally lowers the LR. AALR halves the LR
11/2, doubles patience p 2p, and continues training for p- + 1 epochs. If however, the loss had decreased, L < L', the previous approach is followed: i.c., it doubles the LR r 21,, resets the patience 1, updates best loss and checkpoint, and repeats training for p + 1 epochs. The above cycle repeats till the stopping criterion is met. For ease of exposition, the pseudocode is given in Algorithm
3 MOTIVATION AND RELATED WORK
Increasing the LR optimistically can potentially help the training to escape saddle points that slow down the training, as well as find flatter minima with good generalization performance. This is inspired mainly from the following observations in the literature.
Dauphin et al. 2 suggest that saddle points slow down the training of deep networks. Xing itt al. (2018) states that SGD moves in valley like regions of the loss surface in deep networks by jumping from one valley wall to another at a height above the valley floor which is determined by the LR. Large LR can help in generalization by helping SGD to quickly cross over the valley floor as well as its barriers, to travel far away from the initialization point short time. Similarly, Hoffer eet al. (2017) describe the initial training phase as a high-dimensional "random walk on random potential" process, with an "ultra-slow" logarithmic increase in the distance of the weights from their initialization.
From the above discussion, it seems that if one could increase the step size or LR continuously (as long stability is maintained), it might considerably speed up the increase in distance of the weights from the initialization point, making the initial ultra-slow diffusion process faster. In this way, further exploration of the loss landscape might be possible, leading better generalization.
The idea of increasing the LR has been explored by algorithms like SGDR and CLR. In SGDR, the LR is reset to high value in a periodic manner; this is referred to warm restart. After this, the LR decays to a low value following a cosine annealing schedule. In CLR,, the LR is increased and decreased linearly in a periodic manner. While the regular increase in LR in most cases, probably helps generalization and helps in finding flatter minima, they follow a preset method, that does not depend on the training state or progress. Detection of convergence also becomes difficult due to heavy fluctuation in the training output (which happens due to the periodic nature of these methods). Moreover, the authors of each of these methods suggest tuning the parameters of the algorithm for better performance. Even though [Schaul et al. (2013) suggest method that uses information about the state and distribution, it computationally heavy method. Similarly, hypergradient descent due to [aydiin et al. (2018) requires additional computation of gradients. Moreover, it requires tuning of initial LR and introduces additional hy perparameterss to be tuned, such as the learning rate for the LR itself
We propose the simple idea of exploring LR in a binary fashion, without requiring any parameter tuning. This is an adaptive LR tuning algorithm that tries to follow the training state and set the LR accordingly. Increasing LR for better generalization through exploration (and also, potential acceleration of initial phase of SGD) are the main motivations for the optimistic doubling. At the
4

---

Under review as a conference paper at ICLR 2020
same time, once SGD is in the vicinity of good minimum, LR might need to be reduced to access the valley. Hence, if the algorithm observes that the loss is not reducing even after a few 'patience' iterations, it halves the LR. The reduction is kept conservative at 1/2 to encourage finding flatter minima.
The automated adaptive LR algorithm propose achieves good generalization in all cases, in- cluding adversarial scenario, and converges smoothly in roughly the same time as LR-tuned SGD baselines available in the literature and community.
4 CONVERGENCE DISCUSSION
Convergence analysis of SGD typically requires that the sequence of step sizes, or, learning rates used during training satisfy the following conditions: Σt=1 7/ = oc and Ei= mi < co.
Consider optimal stochastic gradient approach OPT that any point in time has oracle access to (and applies) the highest value of learning rate, that would be amenable for good training (ensuring fast convergence and good generalization). The sequence of LRs chosen by OPT satisfy the above condition. The sequence ensure that OPT will converge (to a good generalization), at the same time, the convergence is the fastest since the step sizes LRs are kept high possible. Let the LR of OPT at any epoch t be denoted as
One can define OPT as the following:
Definition: An optimal oracle SGD, with LR Pr at any epoch I, such that the following properties hold:
1. -t=1 Pt 00,
2. Σ P < oc,
3. Any first order stochastic gradient-based algorithm that has the same location in parameter space as OPT at the start of an epoch / must have LR S Pt. otherwise training will diverge here onward (in other words, gradients will explode and the training of the network cannot be recovered from here).
4. Any first order stochastic gradient based algorithm starting from the same location in pa- rameter space as OPT and achieving similar generalization for given training task, will require at least as many epochs as OPT for convergence.
We will compare AALR with OPT, and show that the sequence of LRs chosen by AALR follow the sequence of LRs of OPT with a bounded delay under some assumptions. We also show that divergence will not happen.
A typical well-tuned SGD algorithm can be thought to be proxy for OPT for a given scenario, and hence this analysis will bound the convergence time of AALR with respect LR-tuned SGD for the same problem.
In a typical step decay LR-regime for SGD, the LR does not increase, but generally decreases at For standard LR schedules, one can see that the following rule-of-thumb holds: the number of epochs in between two consecutive LR changes is directly proportional to In fact one can see that for standard regimes, A 2 e7, where 2 2. (For example, change by a factor of 2 happens at every 5 epochs or more, or, change by factor of 10 happens every 30 epochs more). Such typical LR regimes often designed out of observing of loss plateauing. We assume that OPT has a similar behaviour in the following analysis.
certain intervals by some factor ( 10}.
4.1 BOUNDING THE DELAY IN CONVERGENCE DUE TO DOUBLING
Let the LR of OPT at any epoch be denoted as Pt and that of AALR be denoted as nh- We assume that Phase has estimated a stable initial LR 7/ < Po. and that both AALR and OPT are roughly in the same space in the loss valley at the start of Phase 2, denoted as epoch (for simplicity). In the following, we refer to decrease in loss compared to the best observed loss thus far as an improvement in state.
5

---

Under review as a conference paper at ICLR 2020
Assuming that the loss surface is smooth, the loss will continue to decrease for AALR, as long as " $ Pt. and it will start increasing otherwise.
We first argue that AALR will not diverge. From Algorithm 1 it can be seen that every time state improves, the checkpoint is updated. If and when, due to doubling (or due to initial LR), loss diverges and goes to NAN, the last checkpoint is reloaded, LR is halved and training continues. This will continue till a stable LR is reached, and loss is no longer NAN. In this way, AALR can avoid losing way due to exploding gradients, caused by undue increase in LR.
Now, let us consider the case, when OPT has increased its LR. AALR, by design is always opti- mistically doubling the LR whenever state improves. For increase in pr by factor 7, it can be seen that AALR will require 2 logg 7 epochs. This is because, when state improves, patience will be reset to I, LR will be doubled, and the state will be checked again after training for p + epochs.
We would next show that AALR reaches the same or lower LR as OPT (with some delay) every time OPT reduces LR.
AALR starts with 771 = 2po and trains for p + 1 epochs and checks the state. If OPT has maintained p(t) at Po. then state will not improve. AALR will train for another 2 epochs, and then reduce LR by half, and double the patience. trains at this LR for 2p + 1 3 epochs. Therefore, effective training for AALR is for 3 epochs out of the 7 epochs it spent. Now if the state improves, AALR would double the LR and the above cycle would repeat till we come to the state where OPT needs to reduce the LR for making progress. Let there be k such cycles, such that OPT has trained for 3(k - 1) < q S 3k epochs and AALR has trained for 7k epochs to arrive at roughly the same location in parameter space (assuming bounded gradients) and both have the same LR.
Now, let OPT reduce its LR by y. i.c., Py+1 = Pp/y (For simplicity, let us assume that y is a power of 2). AALR would be first doubling the LR to 7/A++1 2//7A 2pg (since its state was improving till 7k epochs), and patience p will be reset to 1. It will need to reduce its LR + log, " times before observes an improves in state (by assumption on OPT maintaining the highest possible LR for training progress). It will train for 2(p + 2(1 + 1) = 4 epochs, then halve the LR, double p, train for 2(p 1) = 2(2 + 1) = epochs, and repeat this for n
log2 y times. One can see i-a 2(2 22) +1 epochs. At this time, p = 2 2--1, Now, AALR will train for 2(p+11 + 1) = 2 + epochs at this LR, after which it will observe improvement in state. Note that by the earlier observation regarding typical LR regimes and the behavior of OPT, OPT would train for at least ~ 2y = 2" epochs at this new LR. Hence, AALR has trained for a total of roughly twice the number of epochs as OPT, and the new LR for roughly the same number of epochs as OPT. Therefore, both are now at similar location in parameter and loss space. After this AALR would again double the LR, and the earlier cycle would repeat for another k' times, such that OPT would have trained for 3(K' - 1) < p' < 3k' epochs and AALR would have trained for 7k' epochs till the next LR change happens in OPT. Therefore, one can see that AALR would take 2 times the number of epochs as OPT to reach the same or lower LR, every time OPT lowers the LR.
by induction that AALR will be spending a total of N NAALR = Σ
Since the LR sequence of AALR follows the LR sequence of OPT with some finite delay, it can be argued that the following convergence requirements on the LR sequence hold for AALR: (1) Ee 1//- oc, and, (2) >e= oz < co. In practice, we observe that AALR converges in around the same time as LR-tuned SGD.
EXPERIMENTS
We trained with AALR on several madel-datasets combinations, in multiple scenarios such as nat- ural training, as well as adversarial training. We observed that AALR achieve similar better accuracy as the state-of-the-art baselines.
We have compared to standard SOTA (SGD other) LR tuned values reported in the literature and with three other adaptive LR tuning algorithms, SGDR (Cosine Scheduling with Warm Restarts), CLR (Cyclic Learning Rates), ADAM, and Hypergrad (Baydin et al. 2018) (few runs were not complete). Since the principle claim of AALR is that it is a completely autonomous adaptive ap- proach that does not require any funing, for fair comparison, we have not tuned the parameters of

---

Under review as a conference paper at ICLR 2020
any other adaptive approaches compared with. Since AALR does not have any tunable parameters by design, sensitivity analyses experiments were not performed for AALR.
Additionally, we have done with the learning rate. We find the AALR finds LR trajectory that roughly obey the square root rule suggested by Hoffer et al. (2017) and
experiments examining the relationship of batchsize andor entropy
5.1 SETTINGS, DATASETS AND MODELS
Experiments done in PyTorch in x86 systems using 6 cores and 1 GPU. Where baselines for SGDR (Loshchilov & Hutter 2016) and CLR are not available in literature, the PyTorch provided implementations of the corresponding LR schedulers with default settings were used m_d__mmolll weelll/ttorcie htttrcc//ottm////mmm 1mrrm
(available here https:::./............tt...tt http:://yyt..cccc.coooom. org docs scheduler er html For Hypergrad (we have tried SGDHD optimizer), have used the code available at btteps://githubee com
wwtp:/www..ooo////...../occccaaa http//bayynn/n///e hatpaa/hhpgggraiiinn descent
We have tested on datasets CIFARIO and CIFARI00 using standard data augmentation for both, on models Resnet-18, WideResnet-28- 10 with dropout and cutout, WideResnet-33-10 (for adversar- ial scenario only). SimplenetV1,, and Vgg16 with and without batch normalization. For Resnet- 18, WideResNets and Vgg16 models, we ran all algorithms for 200 epochs at a batch size of 128 for both datasets. For SimplenetV1, we ran for 540 epochs and used batch size of 100 and 64 for CIFAR10 and CIFARION respectively for all algorithms. The code for SimpleNetV1 com ctwm:/ooeeee////iiniaaaaiiaa Perittt__yoorhhhh The code for Resnet-18 and WideResnets was obtained from https: wtm. com www.wogueeeeccoe................ 1ttp:/uuuuuullppppppppp Cutout The code for Vgg16 was obtained from http::::t//t/c //tttuub com/ chengyangfu/ We use entropy loss in all cases.
was obtained from htttss://gtthhi:
The batchsize and number of epochs used in each case was as per the baselines in literature, ie., the reported SOTA results for LR-tuned SGD. The baseline (and in where available, adaptive algorithms') results were reported from the cited sources. For the relationship of batchsize with learning rate, use FMNIST on ResNet-20.
5.2 OBSERVATIONS
Our experiments comprehensively show that AALR is state-of-the- automated adaptive LR tun- ing algorithm that works universally across moodels-dtassed for both natural and adversarial training. It is either better comparable to LR-tuned baselines and other adaptive algorithms uniformly and consistently, with a smooth convergence behavior. It effective across batchsizes, without requiring any tuning. This makes the case that for new models andor datasets, AALR should be reliable LR algorithm of choice, in the absence of any prior tuning or experimentation.
For the adaptive algorithms we compared with, SGDR though performs comparable with AALR in most cases of or natural training, it catastrophically failed for at least two natural training cases (which indicates it require tuning of either initial LR or some other parameters, and hence not completely stand alone automated approach) and moreover, it generally did significantly worse than AALR for adversarial training. CLR achieved slightly lower accuracy compared to AALR in most cases of natural training, and in adversarial training its performance fluctuated basis. ADAM generally converged to lower accuracy and significantly lower in adversarial scenario, and furthermore, it catastrophically failed in two cases of natural training, which shows it requires extensive tuning of parameters.
case by case
AALR consistently tep-level in every case, which makes the case for its universal and reliable applicability, especially when new models/dlattscts/rrainng tasks surface for which prior tuning or information not available.
5.3 NATURAL TRAINING
The baseline values reported have the following sources:
Resnet-18. CIFAR10 reported by Veiesss & Taylor 2017
7

---

Under review as a conference paper at ICLR 2020
WideResnet-25 10, CIFARIO and CIFARI00 baseline reported by Zagoruyko & Ko- modakis 2016), and corresponding SGDR values as reported by Loshchilov Hutter (2016)
WideResnet-28- 10 with Dropout and Cutout, CIFARIO and CIFARI00 reported by De- Vries & Taylor 2017)
SimplenetV1. CIFARIO and CIFARI00 as reported at HasanPour et al. 2016 and
attps://githus http:///ooeeeee///mmeeeeeeee__e/ Pytorchh
Vgg16 with and without Batch Normalization for CIFARIO as reported at https: //gg.hhb..ccm//hhecaaaaggtuuu/uu ch ct//blog/20155077/0//c1111
farlffarlpp (an hatpoaantdd/t/orrrrrrr html (the former values are higher). (CIFARI00 for Vgg 16 values were not reported at these places, hence not provided in the table.)
Model
Baseline
AALR
SGDR
CLR
ADAM
Hypergrad
Resnet-18 18 SimpleNet-VI
95.28
94.94
94.81
93.85
93.07
84.48
95.51
95.17
95.44
93.66
93.99
87.01
Vgg16
91.4, 92.63.
92.16
10.00
91.95
10.00 (78.46*)
10.00
Vgg16-BN
92.45, 93.86
93.23
93.56
92.65
91.48
85.05
WRN-28-10 (Dropout)
96.00
95.75
95.91
95.34
94.01
WRN-28-10 (Dropout Cutout)
96.92
96.44
96.6
95.42
95.46
Table I: Natural Training on CIFARIO. Comparison of test accuracy of model trained with AALR with baselines and with those obtained by training using different adaptive learning rate techniques on various models. With AALR, ADAM drastically improves. Peak accuracy 78.46 from 10.
Model
Baseline
AALR
SGDR
CLR
ADAM
WRN-28-10 10 (Dropout)
79.96
80.45
80.26
78.63
73.67
SimpleNet-Vi 6gg16
78.51
78.21
77.47
74.02
73.48
65.03
10.00
67.79
10.00
Table 2: Natural Training on CIFARIO0: Comparison of test accuracy of model trained with AALR with baselines and with those obtained by training using different adaptive learning rate techniques on various models.
5.4 ADVERSARIAL TRAINING
Here we outline the results and observations from adversarial training. We observe that AALR is particularly effective in Adversarial Training and achieves (to the best of our knowledge) state-of- the-art adversarial test accuracy for FGSM attack in White Box mode
It generally does significantly better compared to the other adaptive algorithms compared with and convergence is casier detect, unlike the other methods. would be interesting to explore theoreti- cal justification regarding the effectiveness of AALR in first-order adversarial training, and it might be related to the loss landscape of the min-max saddle point problem defined by Madry et al. (2018).
In the process of these experiments, we discover that SimpleNetVi 1 is very effective adversarially strong model with respect to CIFAR1O, 10, when trained especially with AALR for FGSM attack (e 8/ 8/255 and = 2/255) in White Box model. This is a light weight model. Adversarial training generally being a compute heavy and time consuming process, becomes much easier and faster with it. The effectiveness of AALR in training models for different and scenarios is clearly
Foor readers unfamiliar with adversarial training and FGSMPGD attacks, details can be found Madry al. 22018) We have given a short description here. Adversarial training aims at solving the max saddle point problem defined
v. in contrast standard training which simply aims minimizing E( L(0, y)]. FGSM simply perturbs data x. + agn(" V T.(0, v, v)), whereas PGD does it for k steps iteratively as follows: [ + assgn(V2 )) White Box attack refers the attacker having access the model parameters.
mins p(0) where p(0) E( [maxaes L(0,
8

---

Under review as a conference paper at ICLR 2020
underlined by these experiments. All attacks Eo within [0, 1) ball. All models were trained on CIFARIO for 200 epochs, using batch size of 128.
The baseline we could find is follows. For FGSM White Box atack CIFARIO, Madry et al. (2018) report 56.1%. For other cases, we could not find bascline figures for these. Therefore, we consider 56.1% the representative baseline for each of the cases in FGSM.
For PGD, Madry et al. (2018) had reported 50.0% for k = 7 steps, and 45.8% for k: = 20 steps. We performed PGD for k = 10 steps on AALR CIFAR 10 WRN-28- 10, and obtained 52.83%. On SimpleNetV1, PGD with 10 steps on CIFARIO with AALR gives 45.32%.
We cross entropy loss in all cases (next set of experiments would examine the performance with CW loss for adversarial attacks).
Model
AALR
SGDR
CLR
ADAM
Resnet-18 18 (c 8/255, 2/255)
66.91
59.89
68.19
33.26
WRN-34-100 (e = 4/255, a = -2/255)
65.86
61.55
55.63
16.13
SimpleNet-VI (c 8/255, -2/255)
65.02
55.4
62.43
17.88
WRN-28-10 (e = 8/255, a = 2/255), 200 epochs
68.25
62.32
63.06
17.69
Table Adversarial Training on CIFARIO: Peak Adversarial Accuracy on FGSM White Box attack, obtained on different models by training with AALR and other adaptive algorithms.
WRN-28-10(A)
SimpleNetVI(A)
ResNet-18(A)
SimpleNetV I(B)
52.83
45.32
50.3
31.76
Table 4: AALR for adversarial Training on CIFARIO and 100: Peak Adversarial Accuracy on (A)PGD attack CIFARIO, (B)FGSM on CIFAR100.
5.5 RELATIONSHIIP OF BATCHSIZE AND ENTROPY WITH LEARNING RATE
The relationship of learning rate with batchsize for training DNNs a well-studied topic. In partic- ular, a linear relationship is suggested by Goyal et al. (2017); Smith (017) and also by Chaudhari & Soatto (017). The linear scaling rule suggests that the ratio of learning rate to batch size: 7/ B should remain constant for similar generalization across batch sizes for a given dataset and model). This has been exploited to achieve good generalization across batchsizes by Goyal al. (2017) and Smith (017). Chaudhari & Soatto (017) suggest that this ratio should not be too small to maintain the implicit regularization of SGD, otherwise the entropy would go to zero and generalization would be poor. This observation is in line to that by Keskar et al. (2016) where small batchsizes were seen to be essential for good generalization. A square root scaling rule of learning rate and batchsize (1 x /B) was suggested by Hoffer et al. (0117)..
We performed simple experiment (thanks to the suggestion by anonymous reviewers) to observer AALR can work across batch sizes, adjusting the LR to achieve same generalization. (Note that in the earlier experiments the batch sizes chosen were per the baseline settings given in the literature, which already shows the effectiveness of AALR across batch sizes, however this is more direct experiment to observe the same). We used different dataset and model than the ones tested earlier: FMNIST on Resnet-20, and tried batch sizes 32, 128 and 512. We obtained similar generalization performance across all the batch sizes. In particular, the peak accuracy were as follows: 95.33 for BS 128, 95.3 for BS 512, and 94.85 for BS 32. (The bascline for the above 95.63 given by Zhong al. (2017). On BS 128, we tried this with Random Erasing (p = 0.5), and again got similar generalization lization:: 95.47 (reported baseline is 95.98). The corresponding LR trajectory is given in Figure Interestingly, AALR seems to find trajectories that follow the square root rule (with lot of fluctuations), rather than linear. In particular, if one observes closely around epochs 120 and 200 (places where the LR changes across the batchsizes), this would be apparent. Around 120, BS 32 has LR oscillating between 0.0 and 0.025, BS 128 has LR oscillating between 0.1 and 0.5 and BS 512 has LR oscillating between 0.2 and 0.1. Around epoch 200, for BS 32, LR is between 0.025 and 0.01 25, for BS 128 between 0.5 and 0.025, for BS 512, between 0.1 and 0.5.

---

Under review as a conference paper at ICLR 2020
Trajectory comperisonns 128, ResNet-20
0.00
Figure 1: FMNIST, LR trajectory across batch sizes.
Comparison Lf trajectory with and without random erasing on ResNet-20, BS 128
0.25
Um
MM
Figure 2: FMNIST, LR trajectory with and without random erasing same batchsize.
Furthermore, we compared the trajectory obtained on FMNIST, ResNet-20 on the same batch size 128, with and without random erasing (Figure 2 Random erasing is the process of randomly re- placing the image pixels with random values for certain rectangular patches with certain probability, commonly used for data augmentation for generating more robust models and improving generaliza- tion. Clearly random erasing is a regularizing technique that increases entropy. Interestingly, the LR trajectory shows that AALR is much that it maintains a much lower LR compared to the standard case. This fits very well with the ob- servation by Chaudhari Soatto (207)) that the entropy implicit regularization in SGD depends on the ratio of 1)/ B. Since in this case, the entropy or regularization is high due to random erasing, AALR lowers 7) and decreases the implicit entropy of SGD to maintain stability and converge the similar generalization.
conservative in the case of random erasing, in the sense
6 CONCLUSION
We have presented an autonomous adaptive algorithm AALR that works without tuning across mod- els, datasets, batchsizes and natural or adversarial training to achieve generalization performance comparable or better than SOTA LR-tuned values. Moreover, it is stable algorithm and consis- tently converges smoothly, unlike other SOTA adaptive algorithms compared with. This is very promising as it possibly implies that AALR can be reliably applied for new datasets or models or adversarial attacks without requiring extensive manual experimentation. In the future. would like to try this on tasks other than image classification, and develop a rigorous theoretical justification behind its consistent performance.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. ICLR, 2018.
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kegl. Algorithmss for hyperparameter optimization. NIPS, 2011.
P. Chaudhari and S. Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. htpp://arrivorraab///00011222 2017.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op-
timization. In Advances in neural information processing systems, pp. 2933-2941, 2014.
Terrence Deiesss and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. https://arxiv.org/bs//// http://bbs/700005555... 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Uu))2121-2159, 2011.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: training imagenet in hour. arXiv preprint arXiv: (rX.vv/706.0267, 2017.
Seyyed Hossein HasanPour, Mohammad Rouhani, Mohsen Fayyaz, and Mohammad Sabokrou. Lets keep simple, using simple architectures outperform deeper and more complex architectures. arXiv preprint arviv: 06037, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 17411, 2017.
Guo Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. CVPR, 2017.
Nitish Shirish Keskar, Dhcevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint Xiv: 1609::66090000 04866, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: Bandit-based configuration evaluation for hyperparameter optimization. Journal of Ma- chine Learning Research, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /60iv:/608. 1608:036,..,383,,, 2016.
D Maclaurin, D Duvenaud, and R.P Adams. Gradient-based hyperparameter optimization through reversible learning. ICMI. 2015, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. ICLR, 2018.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Con- ference on Machine Learning, pp. 343-351, 2013.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. 464-472. IEFE, 2017.
Leslie N Smith and Nicholay Topin. Exploring loss function topology with cyclical learning rates. arXiv preprint arXiv: 1702. 2017.
11

---

Under review as a conference paper at ICLR 2020
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. ICLR, 2018.
J. Snoek, H. Larochelle, and R. Adams. Practical bayesian optimization of machine learning algo- rithms. NIPS, 2012.
C. Thornton, F. Hutter, H. Hoos, and K. Leyton- Browwn Auto-weka: Combined selection and hy- perparameter optimization of classification algorithms.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
A.C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. NIPS 2017, 2017.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
16X1v:1605.07146, 2016.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint 12Xiv:1212.57000 2012.
Z. Zhong, Zheng, G. Kang. S. Li. and Yang. Random erasing data augmentation. http://awwktooos/p//////88822822 2017.
12


Reviews:
[{"review_id": "S1xV-3gVqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for automatically tuning the learning rate of SGD while training deep neural networks. The proposed learning rate tuning algorithm is a finite state machine and consists of two phases: the first phase finds the largest learning rate that the network can begin training with for p = 10 epochs; the second phase is an optimistic binary exploration phase which increases or decreases the learning rate depending upon whether the loss is NaN, increasing or decreasing. Empirical results are shown on a few standard neural networks for image classification on CIFAR-10/100 datasets and for adversarial training on the CIFAR-10 dataset.\n\nI recommend rejecting this paper for the following reasons: (i) the algorithm developed here is extremely heuristic, no insight, theoretical or empirical, is provided as to why this could be a general algorithm, (ii) a major claim in the paper is that the automatic learning rate tuning does not have any hyper-parameters but the actual algorithm does have parameters such as patience and successive doubling of the learning rate although they are tuned adaptively using ad-hoc heuristics, (iii) the convergence analysis is not at all rigorous, in particular the optimal oracle for SGD  may not exist, and (iv) the baseline algorithms are not tuned and the minor improvements of the proposed algorithm over them is therefore not significant.\n\nSome questions that I would like the authors to answer:\n\n1. While the first phase of the algorithm seems a reasonable thing to do, the second phase is full of heuristics which I am not sure will work well for all problems. For instance, I do not see why the algorithm performs trains for p epochs twice even if the loss increased after the first stage, or why the learning rate should be increased if the loss decreased after the second stage.\n2. Section 4, bullet 3/4 in the definition are problematic: the loss in SGD is not monotonically decreasing with respect to time. What does divergence of training mean here? What does “Any SGD algorithm” mean? Do you instead mean any first-order stochastic gradient-based algorithm?\n3. If you imagine a double well potential with one wide minimum and one sharp minimum, both at the same training loss, if OPT starts in the sharp valley, it will not be able to go to the wide valley without the training loss increasing.\n4. Have you tried this algorithm on other problems which are sensitive to the values of learning rate, e.g., training optical flow or segmentation networks?\n5.  The wordy and heuristic argument in Section 4.1 rests on statements like “AALR and OPT arrive at roughly the same location after so and so epochs and hence reaches similar generalization performance”. This cannot happen in a non-convex landscape, the trajectory of SGD starting from the same initial condition can be very different across two independent runs. Therefore, I also don’t see why the latter half of the statement about generalization should be true.\n6. Can you make the development in Section 4 rigorous?\n7. Why are some runs for SGDR stuck at 10% accuracy in Table 1-2?\n8.  FGSM is a very weak attack for measuring adversarial accuracy. Can you show results with a better attack, say a few steps of PGD?\n\n\nSome suggestions to improve the paper:\n\n1. A simple experiment to check the automatic tuning would be to increase the batch-size of the same network while maintaining the ratio of batch-size and learning constant (see https://arxiv.org/pdf/1706.02677.pdf, https://arxiv.org/abs/1710.11029, among others). It would be interesting to see whether the auto-tuner finds a learning rate that corresponds to stable learning without degradation in the generalization performance.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyx_lophKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of automated adaptation of learning rate during (deep) neural network training. The use cases described are standard and adversarial training for image classification. Given the wide use of DNNs in computer vision (and other areas), learning rate tuning is clearly an important problem and is being actively researched.\n\nThe proposed learning rate adaptation procedure consists of a straightforward combination of learning rate halving/doubling and model checkpointing. Experimental results from implementing the adaptive learning rule for standard and adversarial training on CIFAR are provided. Multiple architectures are tested in each setting. The paper claims a primary advantage of the proposed learning rule to be that it requires no tuning as opposed to other rules such as SGD, Adam.\n\nMy decision is to reject the paper due to methodological issues with the experiments and lack of evidence wrt/ dataset variety. The paper should be considered a work-in-progress that may have potential in a more focused setting, e.g., adversarial training as described in the paper.\n\n***\n\nThe major claim of the proposed algorithm not requiring any manual tuning is technically true but misleading. The algorithm does have parameters (SGD momentum, batch size, initial learning rate, patience) with values that were set somehow. In fact, a major methodological issue with the experiments is that the reader does not know if the datasets were used to both set these values and to assess performance, i.e., there are no obvious \"held-out\" datasets. Also, there is no rigorous or even informal justification of the settings. It could be that the paper is arguing that the specific values will result in competitive, if not better, performance than baselines across a variety of datasets - unfortunately, only two datasets are utilized in the experiments, and one, CIFAR10, is not considered challenging. This leads to the second issue with the paper: the experimental validation is not extensive wrt/ datasets which is significant given that the form of the evidence for the proposed method is almost entirely empirical.\n\nAdditionally, I don't agree that competitor algorithms should not be tuned b/c the proposed method does not require tuning. Even if the proposed method does not require tuning (as stated previously, I don't believe this to be accurate), that does not imply a fair comparison precludes tuning competitors via, e.g., cross validation. The only relevant quantities are final test-set performance and total training time/resources required. \n\nThe well-known interdependence between learning rate and batchsize as noted in e.g., Hoffer et al. (2018), is not addressed by the experiments. Batchsizes in the experiments vary, but no justification is provided for how these are selected.\n\nFinally, the paper is unfinished as some experimental runs were not complete at the time of submission.\n\nOn the positive side, the general point about the necessity of learning rate tuning for adversarial training (described in the fourth paragraph of the introduction) is a very good one, and there may be an opportunity for a more focused application of the proposed algorithm perhaps among further datasets and considering additional, alternative attacks.\n\n***\n\nSuggestions for improvement / questions (related to decision):\n\n* It should not be a challenge to find more image classification datasets to include in the experimental comparison: SVHN, Fashion MNIST, Imagenet, ... Using these, the paper can either follow the standard train/test methodology *across datasets*, i.e., split the meta-dataset into train/test, and/or provide a more compelling body of evidence for the proposed method. Also, the performance dependence on batchsize amongst the proposed algorithm and competitors should be investigated experimentally.\n\n* The Baydin et al. (2018) algorithm should be added to the set of competitors since it would provide a relatively easy* reference point wrt/ \"hypergradient\" approaches. I don't agree with the statement in the related work section that this entails \"additional computation of gradients.\" *In the sense that the rule should be straightforward to implement.\n\n* The convergence analysis assumption that the optimal oracle SGD follows typical learning rate regimes motivated by loss plateauing seems to be in direct contradiction to the sentiment expressed in the cited Hoffer et al. (2018) paper that such \"rules of thumb\" may be misguided. Can the authors discuss the appropriateness of their assumption wrt/ this point? Also, in the convergence analysis, the phrase \"in expectation\" is used twice. This has a specific probabilistic meaning, but appears to be used heuristically in this section. Can the authors clarify whether this usage is informal or formal? If the latter is true, it would be better to provide a more formal convergence argument that explicitly takes the inherent randomness into account.\n\n***\n\nEditorial comments (not related to decision):\n\n* Introduction: The first two sentences of the second paragraph, particularly the second, would do well to have an accompanying reference or references.\n\n* Proposed method: Even as an informal statement, the second sentence of the second paragraph under the Phase 2 sub-heading is problematic. The proposed method does not \"resist\" lowering the learning rate \"for as long as possible\" so much as it doesn't lower the learning rate for a fixed number of epochs (algorithm parameter).\n\n* \"Adversarial training\" section (5.4): The paper assumes the reader is familiar with the terms \"FGSM\", \"white box\", and parameters \\epsilon and \\alpha since these are referred to w/o description. Perhaps a short (2-3 sentence) description of the adversarial scenario could be added?\n\n* Experiments: It would be good for the paper to include RMSProp and Adagrad results to the experimental tables as these rules are both readily available for use and widely used.\n\n* Experiments: Is the reporting of the peak accuracy standard in the literature?\n\n* Experiments: I want to give the paper credit for performance on CIFAR100, but this is difficult without explicit points of comparison. This can be easily remedied by including SOTA performance values (along with appropriate references) in the tables or text.\n\n* (Potential) Typos:\n\tProposed method algorithm description:\n\t\tRequirements has a weight decay parameter which seems strange given that the algorithm is performing automated learning rate adaptation...\n\t\tThe epoch counter is incremented in line 5, but not reset prior to Phase 2. Does this mean that Phase 1 training epochs are counted toward the total (T)?\n\t\tLine 7 should be \\eta_t <- \\eta_0 / 2.\n\t\tThe patience counter in line 15 is not utilized below.\n\t\tLine 23 could/should be an else statement.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xlCzUqtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new way of scheduling the learning rate in optimization algorithms such as SGD. It is a stand-alone, parameter-free approach that optimistically doubles the learning rate at every loss improvement between two epochs, until the loss increases too much or diverges, in which case the learning rate is divided by two.\nThis approach is theoretically proven to converge and to follow an optimal scheduling strategy.\nIn addition, the authors experimentally tested their approach on two image classification tasks, showing that the proposed algorithm yields similar to baseline results.\n\nI am rejecting this paper because it seems to motivate things with non-related facts, experiments are not robust and thorough enough, and there is no conclusion (not even in the appendix).\n\n- The most important thing in this paper to me is the fact that \"adversarial training\" is used to motivate this approach a lot. it is mentioned 14 times across the paper: 3 times in the abstract alone. Yet there is no explanation of what it is, and how is it different from \"natural training\" as mentioned in the paper. I suggest the authors either to clearly explain the difference between the two and explain why their approach may help in one setting or the other; or to simply remove the mentions of \"adversarial training\" if it is not important to the approach.\n- to better motivate the approach, I would suggest the authors include different tasks, rather than different training settings. For instance by having one image classification task (keep one of the two current ones) and one text classification or even generation task. This would show that the proposed approach generalizes well to other network architectures.\n\n- The second concern I have is about the experiments. If increasing the learning rate like the proposed approach is making training to convergence faster, then why are the experiments only measuring test set accuracy and not also runtime to convergence?\n- Overall, the experiments are not complete and thorough enough: some table values are missing, the set of adversarial training experiments on CIFAR100 are not reported, and some experiments diverged with the ADAM optimizer. Less than 20% accuracy on a 10-class image classification task seems very far from optimal.\n\n- Eventually, I strongly suggest the authors submit a better closing statement than \"We use cross entropy loss in all cases.\" (especially after having read this same sentence earlier in section 5.1 of the paper). No conclusion is added to the paper, not even in the appendix.\n\nBelow are a few minor points not taken into account in the scoring but that could make the paper slightly better:\n- Section 1, paragraph #1, first sentence: a few citations here would be nice.\n- typo on the first line of page 3: \"5The pseudocode ...\"\n- typo in the 2sn paragraph of section 4: \"... the convergence is the fast*est* since the step sizes ...\"\n- typos in the first line of the second paragraph of section 4.1: \"Assuming that *the* loss surface is smooth, *the* loss will continue...\"\n- page 6: \"At this time, p=2^{n-1} at this time.\"\n- Section 5.1, paragraph 2, first sentence: \"... with dropout and with both dropout and cutout, ...\"\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an automatic tuning procedure for the learning rate of SGD. Reviewers were in agreement over several of the shortcomings of the paper, in particular its heuristic nature. They also took the time to provide several ways of improving the work which I suggest the authors follow should they decide to resubmit it to a later conference.
Decision: Reject