AutoLR: A Method for Automatic Tuning of Learning Rate
Authors: Nipun Kwatra
V Thejas
Nikhil Iyer
Ramachandran Ramjee
Muthian Sivathanu
Venue:  (2020)
Keywords: Automatic Learning Rate
Deep Learning
Generalization
Stochastic Optimization

Abstract: One very important hyperparameter for training deep neural networks is the
learning rate of the optimizer. The choice of learning rate schedule determines
the computational cost of getting close to a minima, how close you actually get
to the minima, and most importantly the kind of local minima (wide/narrow)
attained. The kind of minima attained has a significant impact on the
generalization accuracy of the network. Current systems employ hand tuned
learning rate schedules, which are painstakingly tuned for each network and
dataset. Given that the state space of schedules is huge, finding a
satisfactory learning rate schedule can be very time consuming. In this paper,
we present AutoLR, a method for auto-tuning the learning rate as training
proceeds. Our method works with any optimizer, and we demonstrate results on
SGD, Momentum, and Adam optimizers.

We extensively evaluate AutoLR on multiple datasets, models, and across
multiple optimizers. We compare favorably against state of the art learning
rate schedules for the given dataset and models, including for ImageNet on
Resnet-50, Cifar-10 on Resnet-18, and SQuAD fine-tuning on BERT. For example,
AutoLR achieves an EM score of 81.2 on SQuAD v1.1 with BERT_BASE compared to
80.8 reported in (Devlin et al. (2018)) by just auto-tuning the learning rate
schedule. To the best of our knowledge, this is the first automatic learning
rate tuning scheme to achieve state of the art generalization accuracy on these
datasets with the given models.


Under review as a conference paper at ICLR 2020
AUTOLR: A METHOD FOR AUTOMATIC TUNING OF LEARNING RATE
Anonymous authors Paper under double-blind review
ABSTRACT
One very important hyperparameter for training deep neural networks is the Iearn- ing rate of the optimizer. The choice of learning rate schedule determines the computational cost of getting close to minima, how close you actually get to the minima, and most importantly the kind of local minima (wide/narrow) attained. The kind of minima attained has a significant impact the generalization accu- racy of the network. Current systems employ hand tuned learning rate schedules, which are painstakingly tuned for each network and dataset. Given that the state space of schedules is huge, finding a satisfactory learning rate schedule can be very time consuming, In this paper. we present AutoLR, method for auto-tuning the learning rate as training proceeds. Our method works with any optimizer, and we demonstrate results on SGD, Momentum, and Adam optimizers.
We extensively evaluate AutoLR on multiple datasets, models, and across multiple optimizers. We compare favorably against state of the art learning rate schedules for the given dataset and models, including for ImageNet on Resnet-50, Cifar-10 10 on Resnet-18, and SQuAD fine-tuning BERT. For example, AutoLR achieves an EM score of 81 1.2 on SQuAD v1.1 with BERTBASE compared to 80.8 reported in (Devlin et al. (2018) by just auto-tuning the learning rate schedule. To the best of our knowledge, this is the first automatic learning rate tuning scheme to achieve state of the art generalization accuracy on these datasets with the given models.
INTRODUCTION
Learning rate one of the most important hyperparameters that impact deep neural network (DNN) training performance. It determines how fast get close to minima, which in turn determines the computational cost of optimization. It also determines how close we get to the minima, c.g. higher learning rates may get in the neighborhood of minima much faster, but then just bounce above the minima (Xing et al. (2018); Naa & Sastry (2018)). The learning rate also determines the kind of minima (e.g. wide vs narrow) attained (Keskar al. (2016)), which has a significant impact on the generalization accuracy.
height
Therefore, it is surprising that lot of effort has gone into automatically tuning the learning rate (Schraudolph) 1999); [Schaul et al. (2013b); Rolinck & Martius Martius(2018). However, till date, none of these techniques have been able deliver state of the art test accuracy on standard benchmarks. Instead, deep learning researchers today rely on a mixture of brute force search, augmented with simple heuristics such as using a staircase, polynomial, or exponential decay- based learning rate schedules. This problem is further exacerbated by the fact that different optimizers such as SGD or Adam work best on different datasets and require very different learning rate schedules.
For example, while training image datasets such as Cifar- 10 and ImageNet with SGD or Momentum, a staircase schedule with high starting learning rate typically performs best. On the other hand, with the Adam optimizer, smaller starting learning rate is generally used. Further, in NLP tasks such as machine translation with Adam, typically a linear warmup is followed by polynomial decay, e.g. the Transformer network (Vaswani et al. (2017)) uses linear warmup followed by inverse square root decay. Similar schedules are used for BERT (Devlin et al. (2018) pre-training, while BERT fine-tuning uses linear decay.
Our key idea to tackle this problem is driven by the following simple observation. Consider training Cifar- 10/ using the typical staircase schedule. One can observe that the initial high learning

---

Under review as a conference paper at ICLR 2020
rate results in training loss stagnating after a few epochs. However, if one were to decay the learning rate when loss stagnates, the achieved test accuracy is considerably lower as compared to letting the high learning rate continue for longer, despite apparent improvement in training loss.
Our insight is that this initial phase of training using a high learning rate, even with no improvement in loss, is crucial for generalization and is of the key missing pieces from prior attempts at automatic learning rate tuning. We hypothesize that for DNNs, the number of narrow minimas far outnumber the wide minimas. To generalize well, We want the optimizer to land in wide minimas. An interesting intuitive observation is that a large learning rate can escape narrow minimas easily (as the optimizer can jump out of them with large steps), however once reaches wide minima, is likely to get stuck in it (if the "width" of the wide minima is large compared to the step size).
The above hypothesis motivates our Sxplore--xplood scheme where we force the optimizer to first explore the landscape with high learning rate for sometime in order to land in a wide minima. We should give the explore phase enough time so that the probability of landing in a wide minima high. Once we are confident that the optimizer is stuck in the vicinity of a wide minima, we activate the Exploit phase of AutoLR.
The basic idea of the exploit phase is as follows. We first look at how local perturbations in the current learning rate impact the training loss. Since we only look at local perturbations, we can model the loss as function of these perturbations via Taylor series expansion. We then make a quadratic approximation, and solve for the optimal perturbation in the learning rate which will minimize the loss. This is similar to Newton's method, but applied only in the descent direction (section[)). 2 We do extensive validation that the quadratic is good approximation here. Although the basic idea is simple, it is complicated by the fact that deep learning relies on stochastic optimization methods, such as SGD, which causes the loss values to be noisy across mini-batchis potentially throwing off our estimates. We discuss how we handle stochasticity section 2.2
We demonstrate the efficacy of the explore-exploit approach by evaluating AutoLR across a wide range of models and datasets, ranging from NLP (SQuAD on BERT-base, Transformer on IWSLT) to CNNs (e.g. ImageNet on ResNet-50, Cifar-10 on ResNet18), and across multiple optimizers: SGD, Momentum and Adam. In all cases, AutoLR matches or beats test accuracy of state-of-the art hand-tuned learning rate schedules. For example, on SQuAD v1.1 fine-tuning with BERT BASE, AutoLR is able achieve an EM score of 81.2, compared to 80.8 reported in Devlin et al. (0118) by just auto-tuning the learning rate schedule (all other parameters were unchanged). To the best of our knowledge, AutoLR is the first automatic learning rate scheduler to achieve state of the art results on these datasets for the given models. We show extensive evaluation of our method in section
The main contributions of our work are:
1. The observation that an initial Explore phase with high learning rate is crucial for good generalization in automatic learning rate schemes.
2. Incorporating this observation via an Explore-Exple scheme for tuning learning rate using a local approximation of the optimization landscape.
approach with novel exploitation
3. The first automatic learning rate tuning scheme that beats achieves generalization of state of the art learning rate schedules in multiple models/datasets including ImageNet.
2 METHOD
Let L(O) the loss of the network as a function of its parameters 0. Note that practice, since work with stochastic optimizers such as SGD, the loss computed in each minibatch is only an approximation of the true loss. This distinction is important when we come to the estimation of the best learning rate, but can be ignored for the formulation below.
The loss of the network in the next time step is L(O - (())) where d's the step direction, and 1) is the learning rate. If we think of L(O - -nd) as a function of the learning rate 7,, our task is to find an which minimizes this function. Since is hard to directly estimate L as function of 1j, we instead
consider what happens if we perturb by a small amount e, i.e we look I.(0 (n e)d). Looking
2

---

Under review as a conference paper at ICLR 2020
at this function of e, and applying Taylor series expansion get,
[(c) = L(0 - (n c)d) = L(0 - pl ed)
L(0 nd) - cdd g 1 3 Hd Hd+
where 9 is the gradient of w.r.t 8, and H is the Hessian. Note that computing accurate value of the gradient ÿ is expensive, as it requires going over the entire dataset (not just minibatch), while computing the Hessian H typically intractable for deep networks which have millions of parameters. However, it turns out that we don't 't actually need the values of ğ and H. In fact, we can just look at L as quadratic function of e, i.e.
((e) ko + kie f k2t² =
To compute {kg, k1, k2). we simply evaluate the loss at a few values of and fit a quadratic poly- nomial. Note that we are able to get away with having to calculate the high dimensional T. and H, because the loss function only looks at the effect of these first and second order derivatives in one single direction d, as determined by the terms d'g g and d Hd. Once we know the values of kk. k2). we can simply minimize this quadratic to get the optimal value of 6min = minimize the loss L in the next time step. Also observe that our method only needs the search direction d for computing the various loss samples. As a result, our method works for any opti- mizer, long
ky to
2kg
access the search direction. We have implemented our method for multiple optimizers including SGD, Momentum, and Adam.
Note that it is very important use perturbation for the Taylor expansion in equation 1 rather than just expanding on 1) via L(0 nyd L(0) -ndd g dnd (17)). This because n may not be small, causing the (1)²) error to be quite large. See appendix A for empirical validation that second order approximation captures the loss perturbations accurately.
Epsilon Thresholding. The quadratic approximation in equation yields an optimal value of Cwii the minimum loss at the next time step. Note that the quadratic approximation is valid only up to (eea) error,
Tpelte
kj for
2hg
2.1
2/2
while the minimization can yield min with large absolute values, therehy making huge errors. That is, although our quadratic approximation valid only for small values of [c]. the quadratic's 's minima may be beyond this approximation range, as illustrated in Figure [ Thus, we need to thresh- old Gmin to a reasonable value. Since approximating the loss to some precision, we use relative threshold as follows. We approximate ((e3) le|3, and bound the error:
2.54
interested in
-0.00
-302
Figure 1: A large [emial can give bad approximation. Orange and green triangles show loss samples used in fitting and testing respectively. Red circle shows the minimum loss value of the quadratic, and purple triangle shows the true loss there.
1 <
(3)
where r the relative error threshold.
2.1 TACKLING GENERALIZATION
The choice of learning rate an important factor in determining generalization quality of the trained network. The above method for tuning the learning rate is a local method, and does not take into account the highly non-linear and non-convex global optimization landscape of deep networks. As result it can take globally suboptimal decisions which can impact generalization. In this section we discuss the problem more detail, and our solution.
Although understanding generalization of deep neural networks is an open problem, there have been interesting findings recently. Kawaguchi (2016) found that deep neural networks have many local minimas, but all local minimas also the global minima (also see Goodfellow et al. (2016)). Also, it is widely believed that wide minimas generalize much better than narrow minimas Hoccheiter & Schmidhuber (1997); Keskar at. (016) Jastrzebski et al. (217)) Wang et al. (018) even though
3

---

Under review as a conference paper at ICLR 2020
they have the same training loss. Keskar et al. (2016) found that small batch SGD generalizes better than large batch SGD and also lands in wider minimas, suggesting that noise in SGD acts as an implicit regularizer. Interestingly however, more recent work was been able to generalize quite well even with very large batch sizes (Gioyal et al. 2017);| McCandlish et al.] (2018) (Saallle et al. (2018), ), by scaling the learning rate linearly function of the batch size. This suggests that the lack of noise in large batch SGD can be compensated with high learning rates, and thus the learning rate plays a crucial role in generalization.
Many popular learning rate schedules start the training with high learning rates, and then reduce the learning rate after every few epochs or following some hand tuned fall curve. An interesting observation in many such examples is that even though a high learning rate stagnates after a few epochs, one still needs to run enough iterations at the higher learning rate order to get good generalization. For example, see Figure 2 which shows the training loss of a Resnet-18 18 model training on Cifar10 dataset at a fixed LR of 0.1 with SGD. The training loss stagnates after R 50 epochs, however as shown in Table generalization continues to improve if we increase the number of epochs trained at a higher learning rate.
Thiniin apocts (L3)).3)
14-
Explore Epochs
Test Accuracy
1.2
40
94.52
3.9
50
94.63
60
94.67
80
94.72
0.6
1.4
Table I: Cifar- 10 Resnet-18 is trained for 200 epochs with Momentum. A LR of 0.1 is used for the explore epochs. Half the remain- ing epochs are trained at 0.01 and the other half at 0.001. Reported results are average over 4 runs.
1.2
0.
40
100
Figure 2: SGD training loss for Cifar-10 at LR of 0.1. Loss stagnates after Re 50 epochs.
To understand the above phenomena better, we run the following experiment. We train Cifar-10 on R8snet-18 using a high learning rate of 0.1 for only 40 epochs and then use learning rates of 10-2 and 10-3 for 80 epochs each. We do this training many times with different random initializations. On an average this yields low generalization accuracy of 94.58 over 20 runs. However, we find that in 1 of the 20 runs our test accuracy reaches 94.98!
We compute the curvature of the loss surface at the end of training for the various runs and are able to validate that the high test accuracy indeed corresponds to a wider minima than the low test accuracy runs. Specifically, we use the highest eigenvalud of the Hessian at the minima as a measure of the minima width (Jastrzebski et al. (2019); Keskar et al. (2016)). We find that the high accuracy runs consistently have smaller eigenvalues compared to low accuracy runs. For example the run with highest test accuracy of 94.98 (training loss: 1.52e 3) had eigenvalue of 0.02, while the run
with median accuracy of 94.58 (training loss: 1.56e - 3) had an eigenvalue of 0.05, and the run with
minimum accuracy of 94.32 (training loss: 1.5e - 3) had an eigenvalue of 0.10.
low density of wide minima in the loss landscape of deep neural networks. To explain the observation that training high learning rate for long duration consistently achieves high test accuracy while training at high learning rate for shorter duration until train loss saturates rarely achieves high test accuracy, we hypothesize that for deep neural networks, narrow minima far outnumber wide minima. An intuitive explanation of the observed phenomena is that a large learning rate can escape narrow minima "valleys" easily (as the optimizer can jump out of them with large steps), however once it reaches a wide minima "valley", it is likely to get stuck in it (if
Hypothesis:
We found a bug in our previous Cifar-10 10 Resnet model, where while computing the test accuracy we used the test batch's mean/variance for batch normalization layers, rather than using the running average from training. We have fixed this bug and reran all Cifar experiments. The numbers here and elsewhere for Cifar experiments have been updated with this fix.
We used the opensource implementation
at https: //gt hhwb.com//oahhhl//
pytorch- heaschhhhesiaee an- ea genthings
4

---

Under review as a conference paper at ICLR 2020
the "width" of the wide valley is large compared the step size). See Jastrzebski al. (2019) and Wu et al. (2018) for a discussion on relationship between learning rate and minima shape. When the density of wide minima is low, using high learning rate for long enough duration should still be able to find the wide minima eventually. However, using a high learning for a short duration will only find the wide minima with some probability based the chance of landing in the wide minima early during training. Since we find empirically that the probability of landing in the wide minima early in training is low (about 1/20 based on our experiments), we hypothesize that wider minima are fewer in number than narrow minima. Please see Section for another experiment in the literature that adds more evidence to this hypothesis.
The above hypothesis motivates our Explore-Explos scheme where we force the optimizer to first explore the landscape with a high learning rate for sometime in order to land in the vicinity of a wide minima. During the explore phase, we only allow monotonic increase of LR and disallow any decrease even if suggested by the local quadratic fit. We should give the explore phase enough time so that the probability of landing in a wide minima is high. Since the ratio of number of narrow vs wide minimas can depend on the architecture, dataset, etc., predicting the right explore phase duration is hard, and is currently hyperparameter for AutoLR. Once wide minima, we start the exploit phase and
Explore-Exploit
confident that
the optimizer must be now stuck in the vicinity allow AutoLR R reduce the learning rate dictated by the local method make fast progress and get close to the minima of this hopefully wide minima.
Explore-Exploit improves the generalization of AutoLR significantly. Without explore phase, the AutoLR scheme will start reducing the learning rate as soon as it reaches close to a local minima, irrespective of whether it narrow or wide minima. As shown in Figure 3 locally at epoch 50, makes sense to reduce the LR as it increases the rate of descent. Without any explore phase, AutoLR Eq 2p.
will make such a decision as well, potentially hurting generalization.
Thaining epocts
1.2
0.68
1.0
0.6
0.4
0.02
0.2
0.00
5 10 1. 22
30
50 70 60 100
Figure 3: SGD training loss for 10 on Resnet-18. Plots are split across two to permit higher y-axis fidelity. In the orange plot, a fixed LR of 0. is used, while in the blue plot the LR is reduced from 0.1 to 0.01 at epoch 50. Clearly, epoch 50, it locally makes sense to reduce the LR as it increases the rate of descent significantly.
Explore Epochs
Test Accuracy
0
93.77
94.01
25
94.27
50
94.52
70
94.58
80
94.62
100
94.79
Table 2: Cifar-10 on Resnet-18 trained for 200 epochs with Momentum. An LR of 0.1 is used for the explore epochs, and is determined by AutoLR for rest the epochs. We report averages over runs.
To validate the effectiveness of the explore-exploit scheme, we first analyze a state-of-the learn- ing rate schedule for Cifar- 10ffr-10mm Resnet-18. This learning rate schedule trains with learning rates of 10 10-2, 2 10 - for 100, 50, 50 epochs each. The first 100 epochs at a high learning rate of 0.1, can be thought of explore phase here, as discussed in Figure 2 We experimented with reducing
5

---

Under review as a conference paper at ICLR 2020
this "explore" phase, and as shown in Table is clear that explore phase plays important role in the final generalization accuracy. We now evaluate explore-exploit on the same model and dataset but with learning rate determined by AutoLR in the exploit phase, and try different duration of the explore phase. Table 2 shows the test accuracy for different number of explore epochs. As shown, the explore phase helps with improving generalization accuracy significantly from 93.77 without explore to 94.79 79 with 100 explore epochs.
The above discussion suggests that overall, higher learning rates are better for generalization and thus one should prefer them even though a local analysis may suggest oth- erwise. To incorporate this idea into our method employ what we call saturation threshold. The idea is that we want to continue with the current learning rate and not lower it, unless it has saturated in terms of loss drop per iteration. That is, if the training loss drop rate has dropped below a threshold for the current learning rate, it suggests that the current learning rate has served its pur- pose, and we can move on lower learning rate if suggested so by AutoLR. We can either use an absolute threshold, or a relative threshold where we use the ratio of loss-drop rate when we started using the current learning rate to the current loss-drop rate. Finding the best absolute threshold for each model/dataset can be tricky, so we use relative threshold, which we found easy to tune. Also, we noticed that the loss drop rate doesn't 't change drastically towards the end of the training when the loss has anyways stabilized. This can cause high relative saturation threshold to be too striet in the later stages of training, while it would perform well early on. To handle this, we used a simple strategy where we choose relative saturation threshold initially, and first time the saturation thresh- old is crossed, we switch to an absolute threshold with the current loss drop date as the absolute saturation threshold. This essentially amounts to using the relative threshold to determine good absolute threshold value for the current model and dataset, which is then used subsequently.
Saturation Threshold:
Note that, although both explore phase and saturation threshold prefer higher learning rates, they serve different purposes. While the explore phase ensures that the optimization reaches neighbor- hood of a wide minima before reduces the learning rate (by forcing the optimizer use higher learning rate for some minimum time, possibly even after it has saturated); saturation threshold adds an overall preference for a higher learning rate until that learning rate has served its purpose and not optimizing the loss satisfactorily.
2.2 HANDLING STOCHASTICITY
For computation of the quadratic coefficients in equation 2 we need to compute the loss values L(c) at multiple values of c. Note that in a typical DNN setting we never compute the full loss, but only a stochastic loss based on given minibatch. This loss, however, can be noisy and throw off our estimate of émin- To handle this we follow two simple strategies. The first is to use a bigger minibatch (called superbatch) for computing the loss values, and the second is use the same superbatch for computing all the losses for a particular estimate. The same strategy is used when computing loss drop rates for the saturation threshold check. We use an integer multiple of minibatches to make superbatch. This allows us to compute the superbatch loss by simply averaging multiple single batch forward passes, and does not increase the peak GPU memory usage. We use a conservative superbatch size of 100 for all our examples. See section B for details on selection of superbatch size.
Finally, as safeguard we also added rollback policy. In case our system makes bad call on the learning rate change (most likely because of bad superbatch sample), which leads to a reduction in loss drop rate, rollback the decision and revert the state of the network to the time we made the learning rate change. Although, rollback triggers rarely, it is helpful in preventing the optimization from going astray because of one bad decision.
EXPERIMENTS
We extensively evaluate our method on multiple networks and datasets, well as multiple opti- mizers including SGD, Momentum and Adam. We have implemented AutoLR as an optimizer in PyTorch (Paszke al. (2017)), which wraps existing optimizer, such SGD, Adam, etc. For our experiments, we used an out of the box policy as in Rolinek & Martius (011)., where we only
change the optimizer to Autol.R, R, and don't modify anything else. We evaluate on multiple image

---

Under review as a conference paper at ICLR 2020
datasets Imagenet Resnet-50, Cifar-10 10 Resnet- 18, MNIST, FashionMNIST; well as NLP datasets Squad v1.1 for BERT finetuning and IWSLT with transformer networks. (Note: The source code for AutoLR will be open sourced, and link to it has been sent to Reviewers and ACs for this submission.)
3.1 IMAGENET IMAGE CLASSIFICATINN ON RESNET-50
In this experiment we trained the ImageNet dataset (Russakovsky et al. (2015)) on Resnet-50 net- work We evaluated method on SGD with momentum which performs best for this dataset. For baseline runs, we used the standard hand-tuned learning rate schedule of 10 10 2 10100 for 30 epochs each. With AutoLR, trained the network with 25 explore epochs, and used the same seed learning rate as baseline, i.e. 0.1. Table shows the training loss and test accuracies for the various runs. As shown,
comfortably beat the test accuracy of the baseline. Figure 4 shows the learning rate and test top-5 accuracy of AutoLR against that of the baseline. See figure for more detailed comparisons of training loss, test accuracy, and learning rate.
LR Schedule
Training Loss
Test Top Acc.
Test Top Acc.
Baseline
0.74 (0.001)
75.87 (0.035)
92.90 (0.015)
AutoLR
0.74 (0.041)
76.04 (0.098)
93.01 (0.024)
Table 3: Training loss and Test accuracy for ImageNet on Resnet-50. We report the mean and standard deviation over 3 runs.
0110
14-
132
0003
25
30
75
200
325
1.50
175
100
(a) ImageNet
(b) Cifar
4.300
03.2
77.5
241e:9
75.2
72.5
155e-03
03.2
1x
8/3
7:206
2.376/4
65.3
1x10
0.0 0.5
2.0
3.8te 05
20
22
2.22624
(c) SQuAD
(d) IWSLT
Figure 4: Generalization accuracies and learning rates. AutoIR plots are show in blue, and baseline in orange. Learning rates are plotted with dashed lines with y-scale on the right, while generalization accuracies plotted with solid lines with y-scale the left. The generalization metrics plotted are (a) test top-5 accuracy (b) test accuracy (c) test EM score, and (d) validation perplexity.
3.2 CIFAR-10 10 IMAGE CLASSIFICATION ON RESNET-18
In this experiment we trained the Cifar- dataset Krizhevsky al. (2009)) on Resnet-18 network (e et al. (2016) We evaluated our method on SGD, SGD with momentum and Adam optimiz- ers. For baseline runs SGD and Momentum, we used a hand-tuned learning rate schedule of
We used the implementation at: http://jhhha.ommmyyyyrraaamgggg old
We used the implementation hhtp://ggthbbcoookkunnngiinnnnnn

---

Under review as a conference paper at ICLR 2020
1011,10-2, 10-3 3 for 100, 50, 50 epochs, respectively; while for Adam used fixed learning rate of le-3. With AutoLR, we trained the network with 100 explore epochs, and used the same seed learning rate as baseline, ie. 0.1 for SGD, Momentum, and 10 for Adam. Table 4 shows the training loss and test accuracy for the various runs. We also compare against other learning rate schedules such one-cycle, linear decay and cosine decay for the momentum optimizer in Table the learning rate and test loss of AutoLR against that of the baseline for Momentum optimizer. See figures I0
11 12 for more detailed comparisons of training loss, test accuracy, and learning rate for SGD, Momentum and Adam optimizers, respectively.
Optimizer
LR Schedule
SGD
Momentum
Adam
Training Loss
Baseline
4.6c-4 (3.3c-5)
0.002 (6.5e-5)
0.005 (8c-4)
AutoLR
7e-4 (1.1e-4)
8e-4 (le-4)
0.004 (4e-4)
Test Accuracy
Baseline
92.63 (0.002)
94.81 (0.001)
92.46 (0.002)
AutoLR
92.70 (0.004)
94.79 (0.001)
92.75 (0.005)
Table Training loss and Test accuracy for Cifar- 10 on Resnet-18. We report the mean and standard deviation over 7 runs.
See sections G. and section G.2 for results on MNIST and FashionMNIST datasets with SGD, Momentum and Adam optimizers.
3.3 SQUAD FINE-TUNING ON BERT
We now evaluate AutoLR on a few NLP tasks. In the first task, we fine-tune the BERTBASE model (Devlin al. (2018) on SQuAD v1.1 (Rajpurkar al. (2016) [] BERT fine-tuning is prone to ovefitting because of a huge model size compared to the fine-tuning dataset, and is typically run for only a few epochs. We use the standard baseline which trains for epochs with a seed learning rate of 2e 5 5 with linear decay. With AutoLR we found that the model overfits much earlier, seemingly because AutoLR reduces the training loss much more quickly (see figure [4]. and we thus needed to fine-tune for only 2 epochs. The AutoLR runs were trained with 2500 explore steps (ss half epoch), and the same seed learning rate of 2e - as baseline. Table 5 shows our results over runs. We achieve best EM score of 81.2 (after epochs), compared to baseline's best of 80.7 (after 3 epochs). We found it interesting that AutoLR was able to tune the learning rate effectively even in the small budget of epochs beat the baseline number of BERT (see Table-2 of Devlin al. (2018) who reported an EM score of 80.8). Figure 4c shows the learning rate and test EM score of AutoLR against that of the baseline. See figure 14 for detailed comparisons. We also compare against other learning rate schedules such as one-cycle, linear decay and cosine decay in Table I1]
LR Schedule
Train Loss (av) 0.96 (0.075)
EM (best)
EM (av)
Fl (av)
Baseline
80.7
80.4 (0.18)
88.2 (0.02)
AutoLR
1.05 (0.008)
81.2
80.8 (0.52)
88.5 (0.09)
Table 5: SQuAD fine-tuning on BERT. We report the average training loss, best test EM score, and average test EM, Fl scores over 3 runs.
3.4 MACHINE TRANSLATION ON TRANSFORMER NETWORK WITH IWSLT
In the second NLP task, we train the Transformer network /Vaswani et al. 2017) on the IWSLT Herman-to-Enghhee (De-En) dataset (Cettolo et al. (2014)) with the Adam optimizer For baseline, we used the learning rate schedule mentioned in Vaswani et al. (017)). The bascline learning rate starts at 1.25e - 7, and is linearly increased for 4000 steps, followed by inverse square root decay till 50 epochs. With AutoLR, we trained the network with 8 explore epochs, and used a seed learning
We used the implementation at: http://ghh...ooomngggggggprooooh We used the implementation http:///gttubb.co//ytttrrhttiii
8

---

Under review as a conference paper at ICLR 2020
rate of 5e-5. In both cases we use the model checkpoint with least loss the validation set for computing BLEU scores on the test set. Table shows the training loss and test accuracy averaged over 3 runs. As shown, AutoLR achieves a mean test BLEU score of 34.88, compared to 34.70 for the baseline. Figure 4d shows the learning rate and validation perplexity of AutoIR against that of the baseline. See figure 13 for detailed comparisons of training/validation perplexity, learning rate, etc. We also compare against other learning rate schedules such as one-cycle, linear decay and cosine decay in Table [0]
It is interesting to note that both linear decay and cosine decay schedules outperform the baseline inverse square root decay schedule of Vaswani et al. (2017)
LR Schedule
Train ppl
Validation ppl
Test BLEU Score
Baseline
3.55 (0.029)
5.10 (0.033)
34.70 (0.001)
AutoLR
3.46 (0.16)
4.86 (0.014)
34.88 (0.005)
Table 6: Training, validation perplexity and test BLEU scores for IWSLT on Transformer networks. The test BLEU scores are computed on the checkpoint with the best validation perplexity. We report the mean and standard deviation over 3 runs.
Section E has more details of the one-cycle, cosine decay and linear decay baselines. Table shows the test accuracy different experiments on all learning rate schedules tried in this paper. As shown, AutoLR compares favorably against all other learning rate schedules, except slightly missing out in Cifar-10. For one-cycle we report results from the default policy of finding the maximum learning rate, well by hand tuning it (see section E for more details). The Trapezoid schedule follows the same seed learning rate and max learning rate as the tuned values of the corresponding one-cycle run. Similarly, the default cosine decay and linear decay schedules did work well for IWSLT, so we added a warmup phase in the beginming as is done in the original paper's 's baseline. We report both these results under the default and tuned columns, respectively. An interesting observation is that both cosine and linear decay perform better than the inverse square root decay learning rate schedule suggested the "Attention all you need" paper. A similar observation can made for the Bert fine-tuning for SQuAD. The fact that the original authors used a suboptimal learning rate points to the general difficulties learning rate tuning. More importantly, these results show that for fixed function classes of learning rate schedules, it is not just important to tune the hyperparameters of each function class, but also try out different function classes. Since the space of function classes is huge in itself, searching for the best schedule is typically intractable.
Experiment
AutoLR
Baseline
Trapezoid
One-Cycle tuned
Cosine Decay tuned
Linear Decay default
default
default
tuned
ImageNet
93.01
92.90
Cifar-10
94.79
94.81
94.69
93.38
94.59
94.52
94.24
IWSLT
34.88
34.70
34.85
32.35
34.57
0.33
34.85
0.28
34.82
SQuAD 81.2
80.7
80.1
79.8
80.9
80.8
80.7
Table 7: Test accuracy numbers for different experiments on all learning rate schedules tried in this paper. We report the average top-5 accuracy for ImageNet, average top-1 accuracy for Cifar- 10, average BLEU score for IWSLT and the best EM score for SQuAD. The ImageNet results for one-cycle, cosine-decay, linear-decay are still running and will be updated in an updated draft.
4 RELATED WORK
Generalization. There has been a lot of work recently on understanding the generalization char- acteristics of DNNs. [Kawaguchi (2016) found that DNNs have many local minima, but local minima were also the global minima. It has been observed by several authors that wide minima gen- cralize better than narrow minima (Arora et al. (2018): Hochreiter & Schmidhuber [197);; Keskar et al. 2016))Dastzzehh
et al. (2017) Wang et al. (2018) but there have been other work questioning this hypothesis as well (Dinh et al. (2laaa/ Golatkar et al. (2019): ; Guiroy et al. (2019): Jastrzebski
et al. (2019); Yoshida & Miyato| (2017h).
Keskar et al. (0066) found that small batch SGD generalizes better and lands in wider minima than large batch SGD. However, recent work has been able generalize quite well even with very large

---

Under review as a conference paper at ICLR 2020
batch sizes (Goyal et al. learning rate linearly as a function of the batch size. Jastrzebski et al. (2019) analyze how batch size and learning rate influence the curvature of not only the SGD endpoint but also the whole trajec- tory. They found that small batch or large step SGD have similar characteristics, and yield smaller and earlier peak of spectral norm well as smaller largest eigenvalue. DDinh al. (2017) show analytically using model reparamcterization that wide minima can be converted to sharp minima without hurting generalization. |Wang et al. (2018) analytically show that generalization of model is related to the Hessian and propose a new metric for the generalization capability of a model that unaffected by model reparameterization of Dinh et al.] (0077). Yoshida & Miyato| (0017) argue that regularizing the spectral norm of the weights of the neural network help them generalize better. On the other hand, AAroa et al. (2018) 8 derive generalization bounds by showing that networks with low stable rank (high spectral norm) generalize better. Guiroy et al. (2019) looks at generalization in gradient- based meta-lcarning and they show experimentally that generalization and wide minima are not always correlated. Finally, Golatkar et al. (2019) show that regularization results in higher test accuracy specifically when it is applied during initial phase of training, similar to the importance of AutoLRá explore phase during initial phase of training.
(2177: McCandlish al. (2018); Shallue al. (0118) by scaling the
Lower density of wide minima. Wu et al. (2018) compares the sharpness of minima obtained by batch gradient descent (GD) with different learning rates for small neural networks FashionM- NIST and Cifarl0 datasets. They find that GD with a given learning rate finds the theoretically sharpest feasible minima for that learning rate. Thus, in the presence of several flatter minimas, GD with lower learning rates does not find them, leading the authors to conjecture that density of sharper minima is perhaps larger than density of wider minima.
Adaptive learning. Many adaptive learning rate methods have been developed which adapt the leamning rate on a per parameter basis, including AdaGrad (Duchi al. (2011)). ADADELTA ZZeiler (0122), RMSProp (Tieleman & Hinton (2012), and Adam (Kingma & Ba (2014)). However, most of these methods still require specifying a global learning rate schedule which impacts the performance significantly. Many methods have been proposed to adaptively tune the global learning rate, such as Schaul et al. (2013a) which makes an idcalized quadratic loss function assumption near the minima, Baydin et al. (017)) which uses gradient w.r.t the learning rate to update the learning rate in a first order fashion, and Rolinek & Martius (2018) which again uses a first order method along with estimate of minimum loss achievable in step. While Schaul et al. (201 and Baydin et al. (017)) compare to only constant learning rate baselines, [Rolinek & Martius (2018) does compare state-of-the-ard learning rate schedule for 10, but is unable to match their test accuracy, although does better on train loss. Finally, approaches like one cycle learning (Smith) (201 8); Xing al. (2018) have been proposed that linearly increase and decrease the learning rate in a specific manner. Based on our evaluation, one cycle scheme works reasonably well with manual tuning of the learning rate but does not always achieve oate-of-the-ar art accuracy.
5 CONCLUSIONS
We have presented AutoLR, an Explore-Exploit method for auto learning rate tuning for training deep neural networks. Our explore phase is based on the hypothesis that narrow minimas far outnumber the wide minimas, and thus require some minimum exploration at a high learning rate to land in a wide minima region with high probability. We do multiple validations of this hypothesis, but also plan to further study it both theoretically and empirically in future work. Our exploit phase is based on local quadratic approximation of loss in the search direction, as a function of perturbations in the learning rate. We extensively validate AutoLR on both image (ImageNet, Cifar- 10, MNIST, FashionMNISTT and NLP (IWSLT, Squad) datasets on as well as multiple optimizers, and achieve or beat generalization accuracy of state of the art hand tuned learning rate schedules.
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296, 2018.
Atilim Gunes Baydin, Robert Comish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv: arXiv: 1703.04882
10

---

Under review as a conference paper at ICLR 2020
2017.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam, pp. 57, 2014.
Jacob Devlin, Ming- Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 18..v:88000048000 2018.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1019-1028. JMLR. org, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120ul):2121-2155 2011.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep net- works: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. arXiv preprint arXiv: 19X5::1905..27777 2019.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint VrXivv/706.02660 2017.
Simon Guiroy, Vikas Verma, and Christopher Pal. Towards understanding generalization in gradient- based meta-leamingg arXiv preprint arXiv:1907.0728% 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.
Stanistaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint 1rXiv:1711.04023, 2017.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amost Storkey. On the relation between the sharpest directions of DNN loss and the SGD step length. In International Conference on Learning Representations, 2019. URL httpt::ss///ppeeeviiii
-ine2iiddSSkaaae j05t
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 0et.l6669990636,.. 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 14Xi::1412.6995 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (((11)22278-2224 1998.
Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint Nkiv:1908.0326 2019.
11

---

Under review as a conference paper at ICLR 2020
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:18/2.06112,, 2018.
Kamil Nar and Shankar Sastry. Step size matters in deep learning. In Advances in Neural Informa- fion Processing Systems, pp. 3436-3444, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:/606.05200 2016.
Michal Rolinek and Georg Martius. L4: Practical loss-based stepsize adaptation for deep learning. In Advances in Neural Information Processing Systems, pp. 6433-6443, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)211-252 2015.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Con- ference on Machine Learning, pp. 343-351, 2013a.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Con- ference Machine Learning, pp. 343-351, 2013b.
Nicol Schraudolph. Local gain adaptation in stochastic gradient descent. 1999.
Christopher J Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism neural network training. arXiv preprint wrXiv:181..03600 2018.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Leslie N Smith. A disciplined approach to neural network hyper-parameters: Part 1-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:/803.09822 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26~ 31, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Huan Wang, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Identifying generalization properties in neural networks. arXiv preprint arXiv: 1809.07402, 2018.
Lei Wu, Chao Ma, and E Weinan. How sgd selects the global minima in wver-parameterized learning: A dynamical stability perspective. Advances Neural Information Processing Systems, pp- 8279-8288, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:1708.077477 2017.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770 2018
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 1705. 10941, 2017.
Matthew D Zeiler. Adadelta: adaptive learning rate method. arXiv preprint arXiv: 1212.522,,, 2012.
12

---

Under review as a conference paper at ICLR 2020
A VALIDATION OF QUADRATIC APPROXIMATION
We want to validate whether our quadratic approximation good approximation for modelling the loss as function of the perturbation Figure shows few examples demonstrating the effectiveness of second order approximation. As shown, the loss values at various samples overlap with the quadratic curve almost perfectly, including those used in estimation of the quadratic (orange triangles). Also, the estimated loss value at the quadratic minima matches the true loss value there quite well. Tigure6]showe quadratic plots for more datasets/models.
Optimal Epsilen 0.003681
Optimal Fpsilon -0.020481
2.314
22766
2,312
22764
2.300
2,762
2.38
22758
2304
-0.305
-2.005
-0.002
3,000
5,000
0.005
(a) Minimum within range
(b) Minimum outside range
Figure 5: Quadratic approximation of loss function of Shown two examples from Cifar- 10 on Resnet-18 18 runs where the minimum is (a) within and (b) outside the range of loss samples. The orange triangles show loss samples used in fitting the quadratic, the blue line shows approximation, and the green triangles show more loss samples which were not used for fitting. The red circle shows the minimum loss value as per the quadratic, while the purple triangle show the true value at that e.
quadratic
6.29
505/9
1336
1000
5.5
1.40
250
472
110
4.500
02000
-C2431
20000
64/04
400000
200000
0036000
3.00020
(a) ImageNet Resnet-50
(b) IWSLT Transformer
(c) SQuAD fine-tuning on BERT
Figure 6: Quadratic approximation curve samples from (a) ImageNet on Resnet-50, (b) IWSLT on Transformer and (c) SQuAD fine-tuning on BERT. The legend same as figure [ can be seen that the quadratic approximation works pretty well.
B SUPERBATCH SIZE SELECTION
To choose an appropriate superbatch size, we measure the standard deviation loss as a function of superbatch size. We compute this by evaluating the loss with 10 different randomly sampled superbatches and calculate the standard deviation. Figure shows the measurements. We used a conservative superbatch size of 100 in all our examples, that higher superbatch sizes add an increased computational overhead, but since we recompute our learning rate infrequently the total overhead is not very high.
corresponded to low variance. Note
13

---

Under review as a conference paper at ICLR 2020
over vs superbatch
over superbatch size
0.003
0.007
0.004
0.002
0.0015
0.0030
60
80
100
20
100
Figure 7: Standard deviation of loss as a function of superbatch size for (a) Cifarl0 on Resnet-18, and (b) MNIST.
C COMPUTATIONAL COST
The primary computational cost of our method comes from computing the samples for quadratic approximation. We use superbatch size of 100 and pick 5 samples for approximating our quadratic, thus incurring a cost of 500 forward passes each time we need to recompute the learning rate. Note that we recompute the learning rate only once in recompute _atdddwwwwdwwwwweeee steps. In most of our examples we keep the recompute_ windwwwwwwwwwwwwwoee such that our learning rate recomputation occurs 2 times each epoch, except in BERT fine-tuning where we only have 2-3 epochs to train, and thus want more frequent tuning of the learning rate. Cifar-10 10 on Resnet-18 18 for example, we used a recompute recompute_windoww wwiddww of 1000 minibatch steps. A backward pass is typically 2x more computationally expensive than forward pass, and 1000 minibatch steps thus cost 3000 forward pass worth of compute. Our recomputation costs 500 forward passes of compute, and thus the computational overhead is 500/3000 ss 16.6%. Note, however that in exploit phase, we do recomputation of learning rate only when saturation threshold crossed. So the computational cost comes down even further and 16.6% overhead for Cifar-10 is an upper bound.
Note that since the main focus of this work was to develop an automatic learning rate tuning scheme which generalizes well as state of the art hand tuned learning rate schedules, we have not invested much effort on reducing the computational cost. For example, a 100 superbatch size an overkill and a size of 50 should be suitable for most examples. Similarly, 3 samples for estimating the quadratic are enough most of the times instead of the used, as the quadratic fit very close mostly. Also, we can experiment with higher recompute cooompute_iiddow window sizes reduce the cost further.
D HYPERPARAMETERS
Table 8 shows the hyperparameters used for all examples. As shown, the saturation threshold and epsilon thresholds are pretty stable across all image examples, and the main parameter to tune the explore epochs, which we typically set to around 20 30% of the total budget.
The reason for different scale of epsilon thresholds in NLP datasets compared to image datasets, because they typically run on much lower learning rates (of the order of le 5) compared to image datasets (of the order of 1c - 1 to le 3), which suggests that the optimization landscapes are more sensitive to smaller perturbations and thus need more aggressive clipping. We have some initial ideas on computing this threshold dynamically by looking at change in loss magnitude we increase the e, and choosing threshold which limits this change magnitude to some amount. This can be computationally more expensive, only few times the each run.
need to search for the right epsilon, but can be done
We also noticed that the loss drop rate changes much more drastically in image datasets than NLP datasets, again most likely because of the very low learning rates used in NLP datasets. Thus we need to use a lower saturation threshold in NLP experiments compared to image ones. We found this hyperparameter very easy to set, by simply eyeballing at the loss drop rate changes of a trial run with fixed L.R.
14

---

Under review as a conference paper at ICLR 2020
We also looking ways to automate the explore epochs hyperparameter, by looking second order information about the loss landscape, etc. to determine if we have reached wider minima region. See Jastrzebski et al. (2019) for some interesting analysis this front.
Explore Epochs Total Epochs
Experiments
Saturation Threshold
Epsilon Threshold
MNIST Fashion-MNN -NNIST
10 / 50
100
le-3
10 / 50
100
le-3
Cifar-10
100 / 200
100
5e-3
Imagenet
25 / 90
500
1e-3
IWSLT" 14 (De-En)
50
5
le-9
SQuADv1.1 (Bert-Base)
0.45 /
2
le-12
Table 8: Hyperparameters used for all experiments.
E COMPARISONS WITH MORE BASELINE LEARNING RATE SCHEDULES
In this section compare RutoL. against more learning rate schedules one-cycle, linear decay and cosine decay.
One- One-Cycle:
The one-cycle learning rate schedule proposed in Smith (2018) (also see Smith (017)). This schedule first chooses a maximum learning rate based on an LR Range test. The LR range test starts from small learning rate and keeps increasing the learning rate until the loss starts exploding. |mmihh (2018) suggests that the maximum learning rate should be chosen to be bit before the minima. in region where the loss is still decreasing. There subjectivity in making this choice, although some blogs and libraries" suggest using a learning rate one order lower than the one at minima. However, in our experience, we found that this choice did not perform well. Instead, we found that values closer to the minima performed much better. In the experiments below we tried multiple values closer to the minima where the loss was still decreasing and report the best results as one-cycle (tuned). Results with the default policy of 1/10th the minima reported as one-cycle (default).
Once the maximum learning rate is chosen, the one-cycle schedule proceeds as follows. The learning rate starts at a specified fraction of the maximum learning rate and is increased linearly to the maximum learning rate for 45 percent of the training budget and then decreased linearly for the remaining 45. For the final 10 percent, the learning rate is reduced by a large factor (we chose a Siniiiieer.Sttth// factor of 10). For the Momentum optimizer, [Smith (2018) suggests reduce the momentum linearly
when Ir is increasing, and vice versa. We used an opensource implementation @ for our experiments.
The trapezoid learning rate schedule was proposed in Xing ation on the one-cycle schedule. The trapezoid schedule increases the learning rate as in the first phase of one-cycle, but then keeps it flat that value for sometime, before linearly decaying it again like the second phase of one-cycle. Following the Trapezoid (long) schedule of Xing et al. (2018) (see Figure 7,8 there), all our experiments we linearly warmup for the first 10% of iterations, stay flat for the next 80%, and then linearly decay for the final 10% iterations. For the maximum learning rate we use the values of the tuned one-cycle runs.
Trapezoid:
(2018), and is vari-
Linear Decay: The linear decay learning rate schedule simply decays the learning rate linearly to zero starting from a seed LR. the case of IWSLT we found that starting from a high seed caused instability issues (most likely because of Adam optimizer. See Liu et al. (2019)) In this case also ran experiments with a linear warmup followed by this linear decay.
"eee
Also see http::///aauggggrgg.ooo////o///r
-at.o---aannnng-atta......... and https: //ppss fast.
ai/cal waaaabaccs..oeecccleee.. hat.a/ooittaaaaaaaaaaaaaaaaaaaaa
chose the frac-
tion to be 0.1 in our experiments.
15

---

Under review as a conference paper at ICLR 2020
The cosine decay learning rate schedule decays the learning rate zero following cosine curve, starting from seed LR. Similar to Linear decay, in the case of IWSLT, we found that starting from high seed caused instability issues. In this case also ran experiments with linear warmup followed by this cosine decay.
Cosine Decay:
0141
012
1,3
83
64
13
18
628
0.01
01:00
2004
1.00
0.08
0.001
C000
(a) LR Range test for CIFAR-10 10
(b) LR Range test for IWSLT
(c) LR Range test for SQuAD
Figure 8: LR Range test for selecting the maximum learning rate. A good choice is the learning rate a bit before the minima a region where the loss is still decreasing.
E.
CIFAR-10
Figure Sal shows the LR range test for Cifar-10 with the Resnet-18 network. The minima occurs around learning rate of 0.09. For the maximum learning rate, we choose 9e-3 for the default one- cycle policy, but found 0.06 perform best our tuning. For linear, cosine decay schedules we start with seed learning rate of 0.1 as used in the standard baselines. Table 9 shows the training loss and test accuracy for the various schedules.
LR Schedule
Train Loss
Test Accuracy
Trapezoid
0.0009 (8)
94.69 (0.0007)
One-Cycle (default)
0.0003 (3e-5)
93.38 (0.001)
One-Cycle (tuned)
0.0004 (3.1e-5)
94.59 (0.0007)
Cosine Decay
0.0003 (2.7e-5)
94.52 (0.0016)
Linear Decay
0.0002 (2.5e-5)
94.24 (0.0009)
AutoLR
0.0008 (1.0c-4)
94.79 (0.001)
Table Training loss Test accuracy for Cifar- 10 on Resnet-18 more learning rate schedules. We report the mean and standard deviation over runs. For One-Cyele, we report experiments with maximum learning rate chosen via the default Ir-range finder policy and via manual tuning. The Trapezoid schedule uses maximum learning rate of the tuned setting.
E.2 IWSLT' 14 DE-EN
Figure 8b shows the LRR range test for IWSLT on the transformer networks. The minima occurs at 9e-4. For the maximum learning rate, choose 9e-5 for the default one-cycle policy and found 2e-4 to perform best in our tuning. For linear, cosine decay schedules we first used a seed learning rate of Se-4 (the maximum learning rate in the schedule in Vaswani
(2017) ). However, we found this to perform very poorly, most likely due to initial instabilities in Adam discussed in Liu et al. (2019). So also ran experiments with linear warmup as Vaswani et al. (2017) followed by linear/cosine decays which performed much better. Table 10 shows the training, validation perplexity and BLEU scores for the various schedules.
E.3 SQUAD-v1.1
Figure 8c show the LR range test for SQuAD fine-tuning BERT. The minima occurs le-4. For the maximum learning rate, we choose le-5 for the default one-cycle policy and found 4e-5 to perform best in tuning. For linear, cosine decays start with seed learning rate of 2e-5 as used in Devlin et al. (2018). Table the average training loss, best test EM score, and average test FM, F1 scores for the various schedules.
16

---

Under review as a conference paper at ICLR 2020
LR Schedule
Train ppl
Validation ppl
Test BLBU Score
Trapczoid
3.25 (0.016)
4,92 (0.014)
34.85 (0.008)
One- One-Cycle (dcfault)
5.51 (0.028)
5.87 (0.02)
32.35 (0.18)
One-Cycle (tuned)
3.82 (0.09)
(0.03)
34.57 (0.002)
Cosine Decay
9.54 (0.06)
3861.6 (883.4)
0.33 (le-S)
Cosine Decay with warmup
3.29 (0.02)
5.04 (0.019)
34.85 (0.008)
Linear Decay
9.50 (0.07)
3261.5 (849.7)
0.28 (2e-5)
Linear Decay with warmup
3.30(0.02)
5.03 (0.016)
34.82 (0.006)
AutoLR
3,46 (0.16)
4.86 (0.014)
34.88 (0.005)
Table 10: Training, validation perplexity and test BLEU scores for IWSLT on Transformer networks for more learning rate schedules. The test BLEU scores are computed on the checkpoint with the best validation perplexity. We report the mean and standard deviation over 3 runs.
LR Schedule
Train Loss (av)
EM (best)
EM (av)
Fl (av)
Trapezoid
0.95(0.002)
80.2
80.1 (0.05)
87.9 (0.04)
One Cycle (default)
1.364 (0.011)
79.8
79.4 (0.17)
87.3 (0.12)
One Cycle (tuned)
1.06 (0,004)
80.9
80.7 (0.12)
88.4 (0.09)
Cosine Decay
0.89 (0.003)
80.8
(0.06)
88.4 (0.13)
Linear decay
0.96 (0.075)
80.7
80.4 (0.18)
88.2 (0.02)
AutoLR
1.05 (0.008)
81.2
(0.52)
88.5 (0.09)
Table 11: SQuAD fine-tuning BERT for more learning rate schedules. We report the average training loss, best test EM score, and average test EM, Fl scores over runs.
F SEED SENSITIVITY
We performed sensitivity analysis on the importance of the choice of seed learning rate for AutoLR. We ran AutoLR for Cifar-10 on Resnet-18 with different seeds 10x range. Table 12 shows the average test accuracies over 3 runs for the different seeds. As shown, the seed learning rate impact the final accuracy, but AutoLR is not highly sensitive to it, especially in the lower ranges.
Seed LR
Test Accuracy
0.05
94.60
0.075
94.71
0.1
94.79
0.125
94.58
0.15
94.25
Table 12: Cifar-10 10 on Resnet-18 trained for 200 epochs with Momentum. We report averages over 3 runs.
G MORE RESULTS
G.1 MNIST
layer network with two convolution layers and two fully connected layers To We evaluated our method on SGD, SGD with momentum and Adam optimizers. For baseline runs, we used fixed learning rate of 0.1 for SGD, Momentum, and le-3 for Adam; and trained for 50 epochs. With AutoIR, we trained the network with 10 explore epochs, and used the same seed learning rate baseline, i.c. 0.1 for SGD, Momentum, and le-3 for Adam. Table [3 shows the training loss and test accuracy for the various runs. See figures 15 16 17 for more detailed comparisons of training loss, test accuracy, and learning rate for SGD, Momentum and Adam optimizers, respectively.
In this experiment trained the MNIST dataset LeCun al. (1998)
We used the implementation http://hthh.commyyaaoacoccccccee
17

---

Under review as a conference paper at ICLR 2020
Optimizer
LR Schedule
SGD
Momentum
Adam
Training Loss
Baseline
2.3e-5 (9.4e-7)
7.2e-6 (4.2e-7) 7.3e-6 (7.7c-7)
4.5e-3 (1.3e-3)
RutoLR
2.4e-5 (5.5e-6)
3.9e-4 (9.5c-4)
Test Accuracy
Baseline
99.32 (3.4e-4)
99.34 (6.0e-4)
99.21 (1.2e-3)
AutoL.
99.32 (3.7e-4)
99.38 (5. (5.1e-4)
99.35 (7.2e-4)
Table 13: Training loss and Test accuracy for MNIST. We report the mean and standard deviation over runs.
G.2 FASHION MNIST
In this experiment we trained the Fashion MNIST dataset Xiao et al. (2017) on the same 4 layer network as for the MNIST dataset. We evaluated our method on SGD, SGD with momentum and Adam optimizers. For baseline runs, we used fixed learning rate of 0. for SGD, Momentum, and 1e-3 for Adam; and trained for 50 epochs. With AutoLR, we trained the network with 10 explore epochs, and used the same seed learning rate as baseline, i.e. 0.1 for SGD, Momentum, and 1e-3 for Adam. Table [4 shows the training loss and test accuracy for the various runs. See figures 18 19 20 for more detailed comparisons of training loss, test accuracy, and learning rate for SGD, Momentum and Adam optimizers, respectively.
Optimizer
I.R Schedule
SGD
Momentum
Adam
Training Loss
Baseline AutoIR
0.005 (0.006)
0.073 (0.007)
0.018 (0.003)
0.001 (0.001)
4.3e-4 (7.2c-4)
0.002 (0.006)
Test Accuracy
Baseline
91.33 (0.005)
89.31 (0.003)
90.56 (0.003)
AutoI
91.88 (0.001)
91.66 (0.002)
91.65 (0.003)
Table 14: Training loss and Test accuracy for FashionMNIST. We report the mean and standard deviation over runs
18

---

Under review as a conference paper at ICLR 2020
H
DETAILED PLOTS
Trahiing Epusha
1.0
1.00
43
130
1.0
33
23
115
22
475
15
Accurazy
12.-
30-
165-
00-
20
20
25
25
85
10.0
93.5
mMn
93.93
64.25
06.00
30-
40
18.
Leamiing Rate Esemns
0.000
3.100
0.009
0.304
0.005
1.5
8.000
0.008
1.00
20
Figure 9: ImageNet on Resnet-50 trained with Momentum. Shown are the training loss, top- potop/top- test accuracy and learning rate function epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into parts to permit higher fidelity in the y-axis range.
19

---

Under review as a conference paper at ICLR 2020
Trakking
144
0.00
0.000
6.0015
M
13-
6.0004
3.5
42-
0.00
Tesk Aucurace vs goodl
0.900
0,000
0.903
0.025
2.579
45
106
120
3.0
150
380
0.5
2.34
0.4
2.55
0.32
0.00
0.00
380
330
100
340
Figure 10: Cifar-10 on Resnet-18 18 trained with SGD. Shown are the training loss, test accuracy and learning rate function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.
20

---

Under review as a conference paper at ICLR 2020
Traihing Eposho
14
1,0m
13
1,000
2.9-
4.
0,03
Test Accumncy Coachs
1954
3.80
0.7
0.8
2344
1910
115
130
300
Leurting Eoodh
1.124
0.00
0.52
0.58
0.2%
0.004
0.44
0.89
6.002
130
110
129
350
150
Figure 11: Cifar-10 on Resnet-18 trained with Momentum. Shown are the training loss, test accuracy and learning rate function of epochs, for the baseline scheme (orange) the AutoLR scheme (blue). The plot is split into parts to permit higher fidelity in the y-axis range.
21

---

Under review as a conference paper at ICLR 2020
1.4
12
537
M MMAAMM u
4.
Test Accurscyyyyy Coechs
8.224
1.8
0.112
My Wwlhann
0.75
6.13
8.00
0.40
29
44
136
100
114
300
339
149
150
350
170
139
Lowering Aata
90419
0.318
8,009
904102
904109
90099
040M4
8,000
90099
330
700
375
Figure 12: Cifar- 10 on Resnet-18 trained with Adam. Shown are the training loss, test accuracy and learning rate function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.
22

---

Under review as a conference paper at ICLR 2020
306
63
106
102
10-
Wullariinn Porshokky as tpocts
112
6.-
3.2
134
5.4
14
16
39
20
22
305
354
305
10
R5
45
425
Leanding Aata
2.007
6.00
1.001
90099
6,000
E.00
6.8324
E.0044
132222
C,002
C,003
00005
14
725
Figure 13: IWSLT on Transformer network trained with Adam. Shown are the training perplexity, validation perplexity and learning rate as function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.
23

---

Under review as a conference paper at ICLR 2020
Epochs
2.6
1.3
244
12-
11
1.8-
1.-
1.6-
1.4
0.9
0.0
0.4
0.6
0.8
1.00
1.25
1.50
1.75
2.00
2.25
2.50
2.75
Test EM Score Epochs
78-
81.0
76
80.0
70-
79.5
68
79.0
64
2.50
Learning Rate vs Epachs
0.9000300
0.9000275
0.000015
0.0000250
0.0000225
8.000010
0.0000200
0.000005
0.000000
0.0
0.2
0.4
0.6
0.8
1.00
1.25
1.50
1.75
2.00
2.25
2.50
2.75
Figure 14: SQuAD fine-tuning on BERT trained with Adam. Shown are the training loss, test EM score, and learning rate function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue), The plot is split into 2 parts to permit higher fidelity in the y-axis range. It is clear that with AutoLR the network starts overfit after the 2nd epoch, where the testing loss continues to go down, but generalization suffers. We saw similar behavior with different seeds, and thus need train with R for only epochs.
24

---

Under review as a conference paper at ICLR 2020
Training Loss vs Epochs
0.175
0.009200
0.000175
0.125
0.000150
0.075
0.025
0,000325
23
5.0
75
10.0
12.5
15.0
17.5
20.0
20
25
30
35
40
45
50
Test Accuracy Epochs
0.994
0.3938
0.992
0.9936
0.990
0.988
0.9932
0.986
0.9930
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
20
25
30
35
40
45
50
Learning Rate Epochs
0.098
6.102
0.096
6.100
0.094
0.096
25
5.0
7.5
10.0
12.5
15.0
17.5
20.0
20
25
30
35
45
50
Figure 15: MNIST on Resnet-18 trained with SGD. Shown are the training loss, test accuracy and learning rate function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range.
25

---

Under review as a conference paper at ICLR 2020
Training Loss vs Epochs
0.14
0.0000255
0.0099230
0.0000225
6.10
0.0000200
0.06
0,0000130
0.0000125
6.02
0.0001000
0.0000075
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
20
25
30
35
45
50
Test Accuracy
9.9946
0.994
0.992
0.9944
0.9942
6,988
0.9940
0.986
0.9938
0.984
0.9936
7.5
12.5
15.0
30
Learning Epochs
9.1025
0.1020
0.102
0.1015
0.100
0.888
0.096
0.1000
2.
5.0
7.5
10.0
12.5
15.0
17.5
20.0
30
45
Figure 16: MNIST on Resnet-18 trained with Momentum. Shown are the training loss, test accuracy and learning rate as function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into parts to permit higher fidelity in the y-axis range.
26

---

Under review as a conference paper at ICLR 2020
Training
0.008
w
0.007
0.006
6.10
0.08
0.04
0.002
0.001
0.00
0.000
Test Accuracy vs Epochs
0.992
0.900
0.986
Learning Rate Epochs
0.00100
0.00100
0.00095
0.00095
0.00090
0.00090
0.00085
0.00000
6.00075
0.00070
0.00855
2.5
5.0
10.0
12.5
15.0
17.5
20.0
35
40
50
Figure 17: MNIST on Resnet-1s 18 trained with Adam. Shown are the training loss, test accuracy and learning rate function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range.
27

---

Under review as a conference paper at ICLR 2020
Training Epochs
0.5
0.12
0.30
0.4
0.3
0.00
0.2
0.02
2
10
10
25
30
35
40
45
50
Acouracy Epochs
0.88
M
0.84
0.89
0.88
Learning Rate vs Epochs
0.10
0.104
0.100
0.04
0.098
0.02
Figure 18:
Fashion MNIST trained with SGD, Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity the y-axis range.
28

---

Under review as a conference paper at ICLR 2020
Training Epochs
0.07
0.4
0.06
0.05
0.3
0.04
0.03
0.02
0.1
0.01
0.00
25
5.0
7.5
10.0
12.5
15.0
17.5
20.0
25
35
45
50
Accuracy Epochs
0,920
0.91
0.915
0.90
0.910
0.905
0.89
0,900
0.88
0.87
0.890
0.685
0.86
Learning Rate vs Epochs
0.100
0.055
0.005
0.090
0.090
0.055
0.080
0.075
0.070
0.070
0.065
0.065
0.060
Figure 19: Fashion MNIST trained with Momentum. Shown the training loss, accuracy and learning rate as function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range.
29

---

Under review as a conference paper at ICLR 2020
Training Loss vs Epochs
0.049
0.4
0.025
0.020
0.2
0.015
0.1
0.000
Test Accuracy vs Epochs
0.915
0.920
0.905
0.915
0.895
6.890
0.885
0.880
0.900
Learring Rate vs Epochs
0.0000
0.0010
0.0008
0.0008
0.0006
3.0004
0.0005
3.0002
0.0003
2.5
10.0
12.5
15.0
17.5
20.0
25
Figure 20: Fashion MNIST trained with Adam. Shown are the training loss, test accuracy and learning rate function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range.
30


Reviews:
[{"review_id": "HyeIHzXhFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an automatic tuning scheme for learning rate while training neural networks. Since learning rate is the most sensitive and important hyperparameter during training, an automatic versatile method for choosing learning rate for various workloads would be of great importance. \n\nThe proposed method works for various optimizers (SGD/Momentum/Adam shown in the paper) and shown to perform as good or better than typical baseline learning rate schedule practitioners use to obtain competitive performance. There are a good number of empirical checks on both image and language tasks.\n\nWhile I’m not fully familiar with literature on automatic learning rates, the authors claim that this is the first auto learning rate tuning scheme to achieve SOTA performance. \n\nTwo main components of the scheme are based on observation that initial high learning rate seemingly without making any improvements is essential for obtaining good final performance. Therefore initially there is an `explore’ phase and then in exploit phase quadratic local approximation around high learning rate update is used. Novelty in the exploit phase is expansion respect to perturbation around some potentially large step size instead of assuming step size is small. \n\nOne main concern of the proposed method is the choice of seed learning rate and duration of explore phase is still somewhat arbitrary and requires tuning for specific model/dataset. For example, in CIFAR/ResNet there are experiments showing that duration of `explore’ phase is important and choose 50 epochs. In the case of BERT for SQUAD fine tuning 2500 explore steps(half epoch) are chosen. It seems choice needs to be tuned to get good performance and with this the proposed scheme is semi-auto tuning at best. Similar points for seed learning rate could be made. \n\nIn order for the proposed method to be fully successful, either authors need to show insensitivity to general choices of seed learning rate and explore phase, or an automatic method to choose good values. Without that  I do not see significant improvement beyond tuning learning rate schedule via linear or cosine decay schemes (which is also known to perform comparatively to multiple drop schemes). \n\nIn that spirit I think an interesting baseline to compare is tuning learning rate schedule. \n\nThe proposed explore-exploit method has great potential in terms of generality and strong performance on various tasks. I would happily raise my score if the main concern is addressed, however at this point I slightly lean toward rejection. \n\nUPDATE POST AUTHOR RESPONSE:\nI thank the authors for carefully responding to the concerns raised in the initial review. I do suggest that refraining calling current method as 'auto' since it could be misleading. I think full automatic learning rates, if it can be done, will have great impact in neural network optimization. I have raised my score as author's response cleared some of issues (sensitivity to seed / comparison with other common tuned schedules).  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlbJWlTFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors propose an automatic learning rate schedule based on an explore (always increase LR initially) and then exploit (more typical patience based decay) strategy. The strategy seems to factor in recent understanding of deep learning optimization, and I was very much convinced by the overall idea, even if not fully convinced by the motivation. The main issues with the paper I have is that (1) I does not compare to a strong enough baseline (one cycle), and (2) the schedule does not feel automatic enough to call it \"automatic\"; see detailed comments. Finally, I found the remarks and analysis regarding width of the minima, and the stipulation that there are more narrow minimas, not substantiated and even contradicatory to some of the results in the literature. Based on this, I am currently leaning towards rejecting the paper. I am willing to raise my score if issues with the experimental setting are addressed.\n\nDetailed comments:\n\n1. I am not convinced that the baseline is strong enough.\n\nWhile I appreciate the extensive range of architecture and datasets, I am not convinced that the baseline schedule tested against in each setting is actually \"state of the art\" as stated in the abstract. To the best of my knowledge, the state-of-the-art learning rate schedule, which uses a similar computational budget as the proposed method, is one cycle. In particular, one cycle uses LR range test to select the appropriate starting learning rate and then warmups to it gradually; see [1] for more details (see also [2]). \n\nNote that one cycle includes warming up the learning rate, which is another motivation for including it. As seen for instance in Fig. 4 the proposed AutoLR can increase learning rate initially, while the hand tuned schedules don't. This does not seem to be a fair comparison, given that the state of the art schedules in vision models very often do warm-up the learning rate.\n\n2. The learning rate schedule requires feeding in \"seed\" learning rate. Is it an automatic learning rate schedule then? I am a bit confused by this claim. Could you please clarify what is the claim about the method?\n\n3. To sum up point 1 and 2, while I like what the paper set out to do, I think that it is key to either (1) demonstrate that the automatic schedule is substantially better than the state of the art schedule, or (2) demonstrate that initial seed learning rate is not an important hyperparameter. If both are not true, then how would you convince the reader to use the method in practice?\n\n4. It is implicitely assumed in the analysis that wide minima are good for generalization. It seems to me that recent evidence points towards the direction that width of the minima is a epiphenomenon, see [3,4,5,6] experiments. I think at least a discussion of (some) of these papers is very important. Also it is worth noting that [5] studied how a high learning rate (or a small batch size) enters wide regions of the loss surface early.  \n\n5. \"In that respect, an interesting intuitive observation is that a large learning rate can escape narrow minima “valleys” easily (as the optimizer can jump out of them with large steps), however once it reaches a wide minima “valley”, it is likely to get stuck in it (if the “width” of the wide valley is large compared to the step size).\". I think [5,8] should be cited here. Both papers studied how learning rate selects minima shape.\n\n6. At the same time, I am not convinced by the experimental data for \"We hypothesize that for deep neural networks, narrow minimas far outnumber the wide minimas.\":\n\n6a) Taking on the face value that the experiment shows that a low learnring rate can find a wide minima, this doesn't prove the hypothesis. At best (i.e. assuming the indeed there is a strong correlation between width of the low lr minima, and generalization) it proves that low learning rate can find wide minima. It doesn't establish any bound/estimation on the relative number of narrow to wide minima. What if low learning rate by default is driven to narrow minima, and high learning rate is driven to wide minima, but the two are equal in number? I think a similar experimental data would be observed in such a null world.\n\n6b) More importantly, it is not demonstrated that the one experiment that worked better had lower curvature. It seems implicitely assumed that if it worked better then it for sure ended up in a wider minima. I do not think this is a valid reasoning given the weak data for a causal relationship between curvature and generalization (see 2). Could you please report curvature explicitely, rather than making this assumption?\n\nReferences:\n\n[1] Fast.ai documentation on one cycle method, https://docs.fast.ai/callbacks.one_cycle.html\n[2] Arpit et al, Walk with SGD, https://arxiv.org/abs/1802.08770\n[3] Golatkar et al, Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence, https://arxiv.org/abs/1905.13277\n[4] Yoshida et al, Spectral Norm Regularization for Improving the Generalizability of Deep Learning, https://arxiv.org/pdf/1705.10941.pdf\n[5] Jastrzebski et al, On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length, https://arxiv.org/abs/1807.05031\n[6] Guiroy et al, Towards Understanding Generalization in Gradient-Based Meta-Learning, https://arxiv.org/abs/1907.07287\n[7] Wang et al, Identifying generalization properties in neural networks, https://arxiv.org/pdf/1809.07402.pdf\n[8] Wu et al, How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective, https://papers.nips.cc/paper/8049-how-sgd-selects-the-global-minima-in-over-parameterized-learning-a-dynamical-stability-perspective.pdf\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method for automatic tuning of learning rates. The reviewers liked the idea but felt that there are much more extensive experiments to be done especially better baselines. Also, clarifying what aspect is automated is important, because no method can be truly automatic: they all have some hyperparameters. 
Decision: Reject