Title: AUTOLR: A METHOD FOR AUTOMATIC TUNING OF LEARNING RATE

Abstract: One very important hyperparameter for training deep neural networks is the learning rate of the optimizer. The choice of learning rate schedule determines the computational cost of getting close to a minima, how close you actually get to the minima, and most importantly the kind of local minima (wide/narrow) attained. The kind of minima attained has a significant impact on the generalization accuracy of the network. Current systems employ hand tuned learning rate schedules, which are painstakingly tuned for each network and dataset. Given that the state space of schedules is huge, finding a satisfactory learning rate schedule can be very time consuming. In this paper, we present AutoLR, a method for auto-tuning the learning rate as training proceeds. Our method works with any optimizer, and we demonstrate results on SGD, Momentum, and Adam optimizers. We extensively evaluate AutoLR on multiple datasets, models, and across multiple optimizers. We compare favorably against state of the art learning rate schedules for the given dataset and models, including for ImageNet on Resnet-50, Cifar-10 on Resnet-18, and SQuAD fine-tuning on BERT. For example, AutoLR achieves an EM score of 81.2 on SQuAD v1.1 with BERT BASE compared to 80.8 reported in (Devlin et al. (2018) ) by just auto-tuning the learning rate schedule. To the best of our knowledge, this is the first automatic learning rate tuning scheme to achieve state of the art generalization accuracy on these datasets with the given models.

1. INTRODUCTION:

Learning rate is one of the most important hyperparameters that impact deep neural network (DNN) training performance. It determines how fast we get close to a minima, which in turn determines the computational cost of optimization. It also determines how close we get to the minima, e.g. higher learning rates may get in the neighborhood of a minima much faster, but then just bounce at a height above the minima (Xing et al. (2018) ; Nar & Sastry (2018) ). The learning rate also determines the kind of minima (e.g. wide vs narrow) attained (Keskar et al. (2016) ), which has a significant impact on the generalization accuracy. Therefore, it is not surprising that a lot of effort has gone into automatically tuning the learning rate (Schraudolph (1999) ; Schaul et al. (2013b) ; Rolinek & Martius (2018) ). However, till date, none of these techniques have been able to deliver state of the art test accuracy on standard benchmarks. Instead, deep learning researchers today rely on a mixture of brute force search, augmented with simple heuristics such as using a staircase, polynomial, or exponential decay-based learning rate schedules. This problem is further exacerbated by the fact that different optimizers such as SGD or Adam work best on different datasets and require very different learning rate schedules.

For example, while training image datasets such as Cifar-10 and ImageNet with SGD or Momentum, a staircase schedule with high starting learning rate typically performs best. On the other hand, with the Adam optimizer, a smaller starting learning rate is generally used. Further, in NLP tasks such as machine translation with Adam, typically a linear warmup is followed by polynomial decay, e.g. the Transformer network (Vaswani et al. (2017) ) uses a linear warmup followed by an inverse square root decay. Similar schedules are used for BERT (Devlin et al. (2018) ) pre-training, while BERT fine-tuning uses a linear decay.

Our key idea to tackle this problem is driven by the following simple observation. Consider training Cifar-10/ImageNet using the typical staircase schedule. One can observe that the initial high learning rate results in training loss stagnating after a few epochs. However, if one were to decay the learning rate when loss stagnates, the achieved test accuracy is considerably lower as compared to letting the high learning rate continue for longer, despite no apparent improvement in training loss.

Our insight is that this initial phase of training using a high learning rate, even with no improvement in loss, is crucial for generalization and is one of the key missing pieces from prior attempts at automatic learning rate tuning. We hypothesize that for DNNs, the number of narrow minimas far outnumber the wide minimas. To generalize well, we want the optimizer to land in wide minimas. An interesting intuitive observation is that a large learning rate can escape narrow minimas easily (as the optimizer can jump out of them with large steps), however once it reaches a wide minima, it is likely to get stuck in it (if the "width" of the wide minima is large compared to the step size).

The above hypothesis motivates our Explore-Exploit scheme where we force the optimizer to first explore the landscape with a high learning rate for sometime in order to land in a wide minima. We should give the explore phase enough time so that the probability of landing in a wide minima is high. Once we are confident that the optimizer is stuck in the vicinity of a wide minima, we activate the Exploit phase of AutoLR.

The basic idea of the exploit phase is as follows. We first look at how local perturbations in the current learning rate impact the training loss. Since we only look at local perturbations, we can model the loss as a function of these perturbations via a Taylor series expansion. We then make a quadratic approximation, and solve for the optimal perturbation in the learning rate which will minimize the loss. This is similar to Newton's method, but applied only in the descent direction (section 2). We do extensive validation that the quadratic is a good approximation here. Although the basic idea is simple, it is complicated by the fact that deep learning relies on stochastic optimization methods, such as SGD, which causes the loss values to be noisy across mini-batches, potentially throwing off our estimates. We discuss how we handle stochasticity in section 2.2.

We demonstrate the efficacy of the explore-exploit approach by evaluating AutoLR across a wide range of models and datasets, ranging from NLP (SQuAD on BERT-base, Transformer on IWSLT) to CNNs (e.g. ImageNet on ResNet-50, Cifar-10 on ResNet18), and across multiple optimizers: SGD, Momentum and Adam. In all cases, AutoLR matches or beats test accuracy of state-of-theart hand-tuned learning rate schedules. For example, on SQuAD v1.1 fine-tuning with BERT BASE , AutoLR is able achieve an EM score of 81.2, compared to 80.8 reported in Devlin et al. (2018) by just auto-tuning the learning rate schedule (all other parameters were unchanged). To the best of our knowledge, AutoLR is the first automatic learning rate scheduler to achieve state of the art results on these datasets for the given models. We show extensive evaluation of our method in section 3.

The main contributions of our work are:

1. The observation that an initial Explore phase with high learning rate is crucial for good generalization in automatic learning rate schemes. 2. Incorporating this observation via an Explore-Exploit approach with a novel exploitation scheme for tuning learning rate using a local approximation of the optimization landscape. 3. The first automatic learning rate tuning scheme that beats or achieves generalization of state of the art learning rate schedules in multiple models/datasets including ImageNet.

2. METHOD:

Let L(θ) be the loss of the network as a function of its parameters θ. Note that in practice, since we work with stochastic optimizers such as SGD, the loss computed in each minibatch is only an approximation of the true loss. This distinction is important when we come to the estimation of the best learning rate, but can be ignored for the formulation below.

The loss of the network in the next time step is L(θ -η d), where d is the step direction, and η is the learning rate. If we think of L(θ -η d) as a function of the learning rate η, our task is to find an η which minimizes this function. Since it is hard to directly estimate L as a function of η, we instead consider what happens if we perturb η by a small amount , i.e we look at L(θ -(η + ) d). Looking at this as a function of , and applying Taylor series expansion we get,

EQUATION

)

where g is the gradient of L w.r.t θ, and H is the Hessian. Note that computing accurate value of the gradient g is expensive, as it requires going over the entire dataset (not just a minibatch), while computing the Hessian H is typically intractable for deep networks which have millions of parameters. However, it turns out that we don't actually need the values of g and H. In fact, we can just look at L as a quadratic function of , i.e.

EQUATION

To compute {k 0 , k 1 , k 2 }, we simply evaluate the loss at a few values of and fit a quadratic polynomial. Note that we are able to get away with having to calculate the high dimensional g, and H, because the loss function only looks at the effect of these first and second order derivatives in one single direction d, as determined by the terms d T g and d T H d. Once we know the values of {k 0 , k 1 , k 2 }, we can simply minimize this quadratic to get the optimal value of min = -k1 2k2 to minimize the loss L in the next time step. Also observe that our method only needs the search direction d for computing the various loss samples. As a result, our method works for any optimizer, as long as we can access the search direction. We have implemented our method for multiple optimizers including SGD, Momentum, and Adam.

Note that it is very important to use a perturbation for the Taylor expansion in equation 1, rather than just expanding on η via L(θ

-η d) = L(θ) -η d T g + 1 2 η 2 d T H d + O(η 3

). This is because η may not be small, causing the O(η 3 ) error to be quite large. See appendix A for empirical validation that a second order approximation captures the loss perturbations accurately. Optimal Epsilon = -0.065769 Epsilon Thresholding. The quadratic approximation in equation 2 yields an optimal value of min = -k1 2k2 for the minimum loss L at the next time step. Note that the quadratic approximation is valid only up to an O( 3 ) error, while the minimization can yield min with large absolute values, thereby making huge errors. That is, although our quadratic approximation is valid only for small values of | |, the quadratic's minima may be beyond this approximation range, as illustrated in Figure 1 . Thus, we need to threshold min to a reasonable value. Since we are interested in approximating the loss to some precision, we use a relative threshold as follows. We approximate O( 3 ) as | | 3 , and bound the error:

EQUATION

)

where r is the relative error threshold.

2.1. TACKLING GENERALIZATION:

The choice of learning rate is an important factor in determining generalization quality of the trained network. The above method for tuning the learning rate is a local method, and does not take into account the highly non-linear and non-convex global optimization landscape of deep networks. As a result it can take globally suboptimal decisions which can impact generalization. In this section we discuss the problem in more detail, and our solution.

Although understanding generalization of deep neural networks is an open problem, there have been interesting findings recently. Kawaguchi (2016) found that deep neural networks have many local minimas, but all local minimas are also the global minima (also see Goodfellow et al. (2016) ). Also, it is widely believed that wide minimas generalize much better than narrow minimas (Hochreiter & Schmidhuber (1997) To understand the above phenomena better, we run the following experiment. We train Cifar-10 on Resnet-18 using a high learning rate of 0.1 for only 40 epochs and then use learning rates of 10 -2 and 10 -3 for 80 epochs each. We do this training many times with different random initializations. On an average this yields a low generalization accuracy of 94.58 over 20 runs. However, we find that in 1 of the 20 runs our test accuracy reaches 94.98!

We compute the curvature of the loss surface at the end of training for the various runs and are able to validate that the high test accuracy indeed corresponds to a wider minima than the low test accuracy runs. Specifically, we use the highest eigenvalue 2 of the Hessian at the minima as a measure of the minima width (Jastrzebski et al. (2019); Keskar et al. (2016) ). We find that the high accuracy runs consistently have smaller eigenvalues compared to low accuracy runs. For example the run with highest test accuracy of 94.98 (training loss: 1.52e -3) had an eigenvalue of 0.02, while the run with median accuracy of 94.58 (training loss: 1.56e -3) had an eigenvalue of 0.05, and the run with minimum accuracy of 94.32 (training loss: 1.5e -3) had an eigenvalue of 0.10.

Hypothesis: low density of wide minima in the loss landscape of deep neural networks. To explain the observation that training at high learning rate for long duration consistently achieves high test accuracy while training at high learning rate for shorter duration until train loss saturates rarely achieves high test accuracy, we hypothesize that for deep neural networks, narrow minima far outnumber wide minima. An intuitive explanation of the observed phenomena is that a large learning rate can escape narrow minima "valleys" easily (as the optimizer can jump out of them with large steps), however once it reaches a wide minima "valley", it is likely to get stuck in it (if the "width" of the wide valley is large compared to the step size). See Jastrzebski et al. (2019) and Wu et al. (2018) for a discussion on relationship between learning rate and minima shape. When the density of wide minima is low, using a high learning rate for long enough duration should still be able to find the wide minima eventually. However, using a high learning for a short duration will only find the wide minima with some probability based on the chance of landing in the wide minima early during training. Since we find empirically that the probability of landing in the wide minima early in training is low (about 1/20 based on our experiments), we hypothesize that wider minima are fewer in number than narrow minima. Please see Section 4 for another experiment in the literature that adds more evidence to this hypothesis.

Explore-Exploit: The above hypothesis motivates our Explore-Exploit scheme where we force the optimizer to first explore the landscape with a high learning rate for sometime in order to land in the vicinity of a wide minima. During the explore phase, we only allow a monotonic increase of LR and disallow any decrease even if suggested by the local quadratic fit. We should give the explore phase enough time so that the probability of landing in a wide minima is high. Since the ratio of number of narrow vs wide minimas can depend on the architecture, dataset, etc., predicting the right explore phase duration is hard, and is currently a hyperparameter for AutoLR. Once we are confident that the optimizer must be now stuck in the vicinity of a wide minima, we start the exploit phase and allow AutoLR to reduce the learning rate as dictated by the local method to make fast progress and get close to the minima of this hopefully wide minima.

Explore-Exploit improves the generalization of AutoLR significantly. Without an explore phase, the AutoLR scheme will start reducing the learning rate as soon as it reaches close to a local minima, irrespective of whether it is a narrow or wide minima. As shown in Figure 3 , locally at epoch 50, it makes sense to reduce the LR as it increases the rate of descent. Without any explore phase, AutoLR ( Eq 2), will make such a decision as well, potentially hurting generalization. Table 2 : Cifar-10 on Resnet-18 trained for 200 epochs with Momentum. An LR of 0.1 is used for the explore epochs, and is determined by AutoLR for rest of the epochs. We report averages over 4 runs.

To validate the effectiveness of the explore-exploit scheme, we first analyze a state-of-the-art learning rate schedule for Cifar-10 on Resnet-18. This learning rate schedule trains with learning rates of 10 -1 , 10 -2 , 10 -3 for 100, 50, 50 epochs each. The first 100 epochs at a high learning rate of 0.1, can be thought of as an explore phase here, as discussed in Figure 2 . We experimented with reducing this "explore" phase, and as shown in Table 1 , it is clear that explore phase plays an important role in the final generalization accuracy. We now evaluate explore-exploit on the same model and dataset but with learning rate determined by AutoLR in the exploit phase, and try different duration of the explore phase. Table 2 shows the test accuracy for different number of explore epochs. As shown, the explore phase helps with improving generalization accuracy significantly from 93.77 without explore to 94.79 with 100 explore epochs.

Saturation Threshold: The above discussion suggests that overall, higher learning rates are better for generalization and thus one should prefer them even though a local analysis may suggest otherwise. To incorporate this idea into our method we employ what we call a saturation threshold.

The idea is that we want to continue with the current learning rate and not lower it, unless it has saturated in terms of loss drop per iteration. That is, if the training loss drop rate has dropped below a threshold for the current learning rate, it suggests that the current learning rate has served its purpose, and we can move on to a lower learning rate if suggested so by AutoLR. We can either use an absolute threshold, or a relative threshold where we use the ratio of loss-drop rate when we started using the current learning rate to the current loss-drop rate. Finding the best absolute threshold for each model/dataset can be tricky, so we use a relative threshold, which we found easy to tune. Also, we noticed that the loss drop rate doesn't change drastically towards the end of the training when the loss has anyways stabilized. This can cause a high relative saturation threshold to be too strict in the later stages of training, while it would perform well early on. To handle this, we used a simple strategy where we choose a relative saturation threshold initially, and first time the saturation threshold is crossed, we switch to an absolute threshold with the current loss drop date as the absolute saturation threshold. This essentially amounts to using the relative threshold to determine a good absolute threshold value for the current model and dataset, which is then used subsequently.

Note that, although both explore phase and saturation threshold prefer higher learning rates, they serve different purposes. While the explore phase ensures that the optimization reaches neighborhood of a wide minima before it reduces the learning rate (by forcing the optimizer to use a higher learning rate for some minimum time, possibly even after it has saturated); saturation threshold adds an overall preference for a higher learning rate until that learning rate has served its purpose and is not optimizing the loss satisfactorily.

2.2. HANDLING STOCHASTICITY:

For computation of the quadratic coefficients in equation 2, we need to compute the loss values L( ) at multiple values of . Note that in a typical DNN setting we never compute the full loss, but only a stochastic loss based on a given minibatch. This loss, however, can be noisy and throw off our estimate of min . To handle this we follow two simple strategies. The first is to use a bigger minibatch (called superbatch) for computing the loss values, and the second is to use the same superbatch for computing all the losses for a particular estimate. The same strategy is used when computing loss drop rates for the saturation threshold check. We use an integer multiple of minibatches to make a superbatch. This allows us to compute the superbatch loss by simply averaging multiple single batch forward passes, and does not increase the peak GPU memory usage. We use a conservative superbatch size of 100 for all our examples. See section B for details on selection of superbatch size.

Finally, as a safeguard we also added a rollback policy. In case our system makes a bad call on the learning rate change (most likely because of a bad superbatch sample), which leads to a reduction in loss drop rate, we rollback the decision and revert the state of the network to the time we made the learning rate change. Although, rollback triggers rarely, it is helpful in preventing the optimization from going astray because of one bad decision.

3. EXPERIMENTS:

We extensively evaluate our method on multiple networks and datasets, as well as multiple optimizers including SGD, Momentum and Adam. We have implemented AutoLR as an optimizer in PyTorch (Paszke et al. (2017) ), which wraps an existing optimizer, such as SGD, Adam, etc. For our experiments, we used an out of the box policy as in Rolinek & Martius (2018) , where we only change the optimizer to AutoLR, and don't modify anything else. We evaluate on multiple image datasets -Imagenet on Resnet-50, Cifar-10 on Resnet-18, MNIST, FashionMNIST; as well as NLP datasets -Squad v1.1 for BERT finetuning and IWSLT with transformer networks. (Note: The source code for AutoLR will be open sourced, and a link to it has been sent to Reviewers and ACs for this submission.)

3.1. IMAGENET IMAGE CLASSIFICATION ON RESNET-50:

In this experiment we trained the ImageNet dataset (Russakovsky et al. (2015) ) on Resnet-50 network3 . We evaluated our method on SGD with momentum which performs best for this dataset.

For baseline runs, we used the standard hand-tuned learning rate schedule of 10 -1 , 10 -2 , 10 -3 for 30 epochs each. With AutoLR, we trained the network with 25 explore epochs, and used the same seed learning rate as baseline, i.e. 0.1. 

3.2. CIFAR-10 IMAGE CLASSIFICATION ON RESNET-18:

In this experiment we trained the Cifar-10 dataset (Krizhevsky et al. (2009) ) on Resnet-18 network (He et al. (2016) ) 4 . We evaluated our method on SGD, SGD with momentum and Adam optimizers. For baseline runs of SGD and Momentum, we used a hand-tuned learning rate schedule of 10 -1 , 10 -2 , 10 -3 for 100, 50, 50 epochs, respectively; while for Adam we used a fixed learning rate of 1e-3. With AutoLR, we trained the network with 100 explore epochs, and used the same seed learning rate as baseline, i.e. 0.1 for SGD, Momentum, and 10 -3 for Adam. Table 4 shows the training loss and test accuracy for the various runs. We also compare against other learning rate schedules such as one-cycle, linear decay and cosine decay for the momentum optimizer in Table 9 .

Figure 4b 

3.3. SQUAD FINE-TUNING ON BERT:

We now evaluate AutoLR on a few NLP tasks. In the first task, we fine-tune the BERT BASE model (Devlin et al. (2018) ) on SQuAD v1.1 (Rajpurkar et al. ( 2016)) 5 . BERT fine-tuning is prone to ovefitting because of a huge model size compared to the fine-tuning dataset, and is typically run for only a few epochs. We use the standard baseline which trains for 3 epochs with a seed learning rate of 2e -5 with linear decay. With AutoLR we found that the model overfits much earlier, seemingly because AutoLR reduces the training loss much more quickly (see figure 14 ), and we thus needed to fine-tune for only 2 epochs. The AutoLR runs were trained with 2500 explore steps (≈ half epoch), and the same seed learning rate of 2e -5 as baseline. Table 5 shows our results over 3 runs. We achieve a best EM score of 81.2 (after 2 epochs), compared to baseline's best of 80.7 (after 3 epochs). We found it interesting that AutoLR was able to tune the learning rate effectively even in the small budget of 2 epochs to beat the baseline number of BERT BASE (see 

3.4. MACHINE TRANSLATION ON TRANSFORMER NETWORK WITH IWSLT:

In the second NLP task, we train the Transformer network (Vaswani et al. (2017) ) on the IWSLT German-to-English (De-En) dataset (Cettolo et al. (2014) ) with the Adam optimizer6 . For baseline, we used the learning rate schedule mentioned in Vaswani et al. (2017) . The baseline learning rate starts at 1.25e -7, and is linearly increased for 4000 steps, followed by an inverse square root decay till 50 epochs. With AutoLR, we trained the network with 8 explore epochs, and used a seed learning rate of 5e-5. In both cases we use the model checkpoint with least loss on the validation set for computing BLEU scores on the test set. Similarly, the default cosine decay and linear decay schedules did not work well for IWSLT, so we added a warmup phase in the beginning as is done in the original paper's baseline. We report both these results under the default and tuned columns, respectively. An interesting observation is that both cosine and linear decay perform better than the inverse square root decay learning rate schedule suggested in the "Attention is all you need" paper. A similar observation can be made for the Bert fine-tuning for SQuAD. The fact that the original authors used a suboptimal learning rate points to the general difficulties in learning rate tuning. More importantly, these results show that for fixed function classes of learning rate schedules, it is not just important to tune the hyperparameters of each function class, but also try out different function classes. Since the space of function classes is huge in itself, searching for the best schedule is typically intractable. 

4. RELATED WORK:

Generalization. There has been a lot of work recently on understanding the generalization characteristics of DNNs. Kawaguchi (2016) 2018)) have been proposed that linearly increase and decrease the learning rate in a specific manner. Based on our evaluation, one cycle scheme works reasonably well with manual tuning of the learning rate but does not always achieve state-of-the-art accuracy.

5. CONCLUSIONS:

We have presented AutoLR, an Explore-Exploit method for auto learning rate tuning for training deep neural networks. Our explore phase is based on the hypothesis that narrow minimas far outnumber the wide minimas, and thus require some minimum exploration at a high learning rate to land in a wide minima region with high probability. We do multiple validations of this hypothesis, but also plan to further study it both theoretically and empirically in future work. Our exploit phase is based on a local quadratic approximation of loss in the search direction, as a function of perturbations in the learning rate. We extensively validate AutoLR on both image (ImageNet, Cifar-10, MNIST, FashionMNIST) and NLP (IWSLT, Squad) datasets on as well as multiple optimizers, and achieve or beat generalization accuracy of state of the art hand tuned learning rate schedules.

A VALIDATION OF QUADRATIC APPROXIMATION:

We want to validate whether our quadratic approximation is a good approximation for modelling the loss as a function of the perturbation . Figure 5 shows a few examples demonstrating the effectiveness of second order approximation. As shown, the loss values at various samples overlap with the quadratic curve almost perfectly, including those not used in estimation of the quadratic (orange triangles). Also, the estimated loss value at the quadratic minima matches the true loss value there quite well. Figure 6 shows quadratic plots for more datasets/models. 

B SUPERBATCH SIZE SELECTION:

To choose an appropriate superbatch size, we measure the standard deviation of loss as a function of superbatch size. We compute this by evaluating the loss with 10 different randomly sampled superbatches and calculate the standard deviation. Figure 7 shows the measurements. We used a conservative superbatch size of 100 in all our examples, as it corresponded to low variance. Note that higher superbatch sizes add an increased computational overhead, but since we recompute our learning rate infrequently the total overhead is not very high.

Under review as a conference paper at ICLR 2020 

C COMPUTATIONAL COST:

The primary computational cost of our method comes from computing the samples for quadratic approximation. We use a superbatch size of 100 and pick 5 samples for approximating our quadratic, thus incurring a cost of 500 forward passes each time we need to recompute the learning rate. Note that we recompute the learning rate only once in recompute window steps. In most of our examples we keep the recompute window such that our learning rate recomputation occurs 2 times each epoch, except in BERT fine-tuning where we only have 2-3 epochs to train, and thus want more frequent tuning of the learning rate. In Cifar-10 on Resnet-18 for example, we used a recompute window of 1000 minibatch steps. A backward pass is typically 2x more computationally expensive than forward pass, and 1000 minibatch steps thus cost 3000 forward pass worth of compute. Our recomputation costs 500 forward passes of compute, and thus the computational overhead is 500/3000 ≈ 16.6%. Note, however that in exploit phase, we do a recomputation of learning rate only when saturation threshold is crossed. So the computational cost comes down even further and 16.6% overhead for Cifar-10 is an upper bound.

Note that since the main focus of this work was to develop an automatic learning rate tuning scheme which generalizes as well as state of the art hand tuned learning rate schedules, we have not invested much effort on reducing the computational cost. For example, a 100 superbatch size is an overkill and a size of 50 should be suitable for most examples. Similarly, 3 samples for estimating the quadratic are enough most of the times instead of the 5 used, as the quadratic fit is very close mostly. Also, we can experiment with higher recompute window sizes to reduce the cost further.

D HYPERPARAMETERS:

Table 8 shows the hyperparameters used for all our examples. As shown, the saturation threshold and epsilon thresholds are pretty stable across all image examples, and the main parameter to tune is the explore epochs, which we typically set to around 20 -30% of the total budget.

The reason for different scale of epsilon thresholds in NLP datasets compared to image datasets, is because they typically run on much lower learning rates (of the order of 1e -5) compared to image datasets (of the order of 1e -1 to 1e -3), which suggests that the optimization landscapes are more sensitive to smaller perturbations and thus need more aggressive clipping. We have some initial ideas on computing this threshold dynamically by looking at change in loss magnitude as we increase the , and choosing a threshold which limits this change magnitude to some amount. This can be computationally more expensive, as we need to search for the right epsilon, but can be done only few times in the each run.

We also noticed that the loss drop rate changes much more drastically in image datasets than NLP datasets, again most likely because of the very low learning rates used in NLP datasets. Thus we need to use a lower saturation threshold in NLP experiments compared to image ones. We found this hyperparameter very easy to set, by simply eyeballing at the loss drop rate changes of a trial run with fixed LR.

Cosine Decay: The cosine decay learning rate schedule decays the learning rate to zero following a cosine curve, starting from a seed LR. Similar to Linear decay, in the case of IWSLT, we found that starting from a high seed caused instability issues. In this case we also ran experiments with a linear warmup followed by this cosine decay. Figure 8 : LR Range test for selecting the maximum learning rate. A good choice is the learning rate is a bit before the minima in a region where the loss is still decreasing.

E.1 CIFAR-10:

Figure 8a shows the LR range test for Cifar-10 with the Resnet-18 network. The minima occurs around learning rate of 0.09. For the maximum learning rate, we choose 9e-3 for the default onecycle policy, but found 0.06 to perform best in our tuning. For linear, cosine decay schedules we start with a seed learning rate of 0.1 as used in the standard baselines. We report the mean and standard deviation over 7 runs. For One-Cycle, we report experiments with maximum learning rate chosen via the default lr-range finder policy and via manual tuning. The Trapezoid schedule uses maximum learning rate of the tuned setting.

E.2 IWSLT'14 DE-EN:

Figure 8b shows the LR range test for IWSLT on the transformer networks. The minima occurs at 9e-4. For the maximum learning rate, we choose 9e-5 for the default one-cycle policy and found 2e-4 to perform best in our tuning. For linear, cosine decay schedules we first used a seed learning rate of 5e-4 (the maximum learning rate in the schedule in Vaswani et al. (2017) ). However, we found this to perform very poorly, most likely due to initial instabilities in Adam as discussed in Liu et al. (2019) . So we also ran experiments with linear warmup as in Vaswani et al. (2017) followed by linear/cosine decays which performed much better. Table 10 shows the training, validation perplexity and BLEU scores for the various schedules.

E.3 SQUAD-V1.1:

Figure 8c show the LR range test for SQuAD fine-tuning on BERT. The minima occurs at 1e-4.

For the maximum learning rate, we choose 1e-5 for the default one-cycle policy and found 4e-5 to perform best in our tuning. For linear, cosine decays we start with a seed learning rate of 2e-5 as used in Devlin et al. (2018) . 

F SEED SENSITIVITY:

We performed a sensitivity analysis on the importance of the choice of seed learning rate for AutoLR. We ran AutoLR for Cifar-10 on Resnet-18 with different seeds in a 10x range. Table 12 shows the average test accuracies over 3 runs for the different seeds. As shown, the seed learning rate can impact the final accuracy, but AutoLR is not highly sensitive to it, especially in the lower ranges. 

G MORE RESULTS:



G.1 MNIST:

In this experiment we trained the MNIST dataset LeCun et al. (1998) on a 4 layer network with two convolution layers and two fully connected layers10 . We evaluated our method on SGD, SGD with momentum and Adam optimizers. For baseline runs, we used a fixed learning rate of 0.1 for SGD, Momentum, and 1e-3 for Adam; and trained for 50 epochs. With AutoLR, we trained the network with 10 explore epochs, and used the same seed learning rate as baseline, i.e. 0.1 for SGD, Momentum, and 1e-3 for Adam. The plot is split into 3 parts to permit higher fidelity in the y-axis range. The plot is split into 3 parts to permit higher fidelity in the y-axis range. score, and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range. It is clear that with AutoLR the network starts to overfit after the 2nd epoch, where the testing loss continues to go down, but generalization suffers. We saw similar behavior with different seeds, and thus need to train with AutoLR for only 2 epochs. The plot is split into 2 parts to permit higher fidelity in the y-axis range. and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range. The plot is split into 2 parts to permit higher fidelity in the y-axis range. The plot is split into 2 parts to permit higher fidelity in the y-axis range. The plot is split into 2 parts to permit higher fidelity in the y-axis range. The plot is split into 2 parts to permit higher fidelity in the y-axis range.

:

We found a bug in our previous Cifar-10 Resnet model, where while computing the test accuracy we used the test batch's mean/variance for batch normalization layers, rather than using the running average from training. We have fixed this bug and reran all Cifar experiments. The numbers here and elsewhere for Cifar experiments have been updated with this fix.

We used the opensource implementation at https://github.com/noahgolmant/ pytorch-hessian-eigenthings

We used the implementation at: https://github.com/cybertronai/imagenet18 old

We used the implementation at: https://github.com/kuangliu/pytorch-cifar

We used the implementation at: https://github.com/huggingface/pytorch-transformers

We used the implementation at: https://github.com/pytorch/fairseq

We used the implementation at: https://github.com/pytorch/examples/tree/master/mnist

Figures and Tables:

Figure 1: A large | min | can give bad approximation. Orange and green triangles show loss samples used in fitting and testing respectively. Red circle shows the minimum loss value of the quadratic, and purple triangle shows the true loss there.

Figure3: SGD training loss for Cifar-10 on Resnet-18. Plots are split across two to permit higher y-axis fidelity. In the orange plot, a fixed LR of 0.1 is used, while in the blue plot the LR is reduced from 0.1 to 0.01 at epoch 50. Clearly, at epoch 50, it locally makes sense to reduce the LR as it increases the rate of descent significantly.

Figure 4: Generalization accuracies and learning rates. AutoLR plots are show in blue, and baseline in orange. Learning rates are plotted with dashed lines with y-scale on the right, while generalization accuracies are plotted with solid lines with y-scale on the left. The generalization metrics plotted are (a) test top-5 accuracy (b) test accuracy (c) test EM score, and (d) validation perplexity.

Figure 5: Quadratic approximation of loss as a function of . Shown are two examples from Cifar-10 on Resnet-18 runs where the minimum is (a) within and (b) outside the range of loss samples. The orange triangles show loss samples used in fitting the quadratic, the blue line shows our quadratic approximation, and the green triangles show more loss samples which were not used for fitting. The red circle shows the minimum loss value as per the quadratic, while the purple triangle show the true value at that .

Figure 7: Standard deviation of loss as a function of superbatch size for (a) Cifar10 on Resnet-18, and (b) MNIST.

Figure9: ImageNet on Resnet-50 trained with Momentum. Shown are the training loss, top-1/top-5 test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.

Figure 10: Cifar-10 on Resnet-18 trained with SGD. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue).The plot is split into 3 parts to permit higher fidelity in the y-axis range.

Figure12: Cifar-10 on Resnet-18 trained with Adam. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.

Figure13: IWSLT on Transformer network trained with Adam. Shown are the training perplexity, validation perplexity and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.

Figure14: SQuAD fine-tuning on BERT trained with Adam. Shown are the training loss, test EM score, and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range. It is clear that with AutoLR the network starts to overfit after the 2nd epoch, where the testing loss continues to go down, but generalization suffers. We saw similar behavior with different seeds, and thus need to train with AutoLR for only 2 epochs.

Figure 15: MNIST on Resnet-18 trained with SGD. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue).The plot is split into 2 parts to permit higher fidelity in the y-axis range.

Figure16: MNIST on Resnet-18 trained with Momentum. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range.

Figure 17: MNIST on Resnet-18 trained with Adam. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue).The plot is split into 2 parts to permit higher fidelity in the y-axis range.

Figure18: Fashion MNIST trained with SGD. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 2 parts to permit higher fidelity in the y-axis range.

Figure 19: Fashion MNIST trained with Momentum. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue).The plot is split into 2 parts to permit higher fidelity in the y-axis range.

Figure 20: Fashion MNIST trained with Adam. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue).The plot is split into 2 parts to permit higher fidelity in the y-axis range.

Table3shows the training loss and test accuracies for the various runs. As shown, we comfortably beat the test accuracy of the baseline. Figure4ashows the learning rate and test top-5 accuracy of AutoLR against that of the baseline. See figure9, for more detailed comparisons of training loss, test accuracy, and learning rate.

Table content: <table><tr><td colspan="4">LR Schedule Training Loss Test Top 1 Acc. Test Top 5 Acc.</td></tr><tr><td>Baseline</td><td>0.74 (0.001)</td><td>75.87 (0.035)</td><td>92.90 (0.015)</td></tr><tr><td>AutoLR</td><td>0.74 (0.041)</td><td>76.04 (0.098)</td><td>93.01 (0.024)</td></tr></table>

Training loss and Test accuracy for ImageNet on Resnet-50. We report the mean and standard deviation over 3 runs.

Table content: <table/>

shows the learning rate and test loss of AutoLR against that of the baseline for Momentum optimizer. See figures 10, 11, 12 for more detailed comparisons of training loss, test accuracy, and learning rate for SGD, Momentum and Adam optimizers, respectively.

Table content: <table><tr><td>LR Schedule</td><td>SGD</td><td>Optimizer Momentum</td><td>Adam</td></tr><tr><td/><td colspan="2">Training Loss</td><td/></tr><tr><td>Baseline</td><td colspan="2">4.6e-4 (3.3e-5) 0.002 (6.5e-5)</td><td>0.005 (8e-4)</td></tr><tr><td>AutoLR</td><td>7e-4 (1.1e-4)</td><td>8e-4 (1e-4)</td><td>0.004 (4e-4)</td></tr><tr><td/><td colspan="2">Test Accuracy</td><td/></tr><tr><td>Baseline</td><td>92.63 (0.002)</td><td>94.81 (0.001)</td><td>92.46 (0.002)</td></tr><tr><td>AutoLR</td><td>92.70 (0.004)</td><td>94.79 (0.001)</td><td>92.75 (0.005)</td></tr></table>

Training loss and Test accuracy for Cifar-10 on Resnet-18. We report the mean and standard deviation over 7 runs.See sections G.1 and section G.2 for results on MNIST and FashionMNIST datasets with SGD, Momentum and Adam optimizers.

Table content: <table/>

Table-2 ofDevlin et al. (2018) who reported an EM score of 80.8). Figure4cshows the learning rate and test EM score of AutoLR against that of the baseline. See figure14for detailed comparisons. We also compare against other learning rate schedules such as one-cycle, linear decay and cosine decay in Table11.

Table content: <table><tr><td colspan="3">LR Schedule Train Loss (av) EM (best)</td><td>EM (av)</td><td>F1 (av)</td></tr><tr><td>Baseline</td><td>0.96 (0.075)</td><td>80.7</td><td colspan="2">80.4 (0.18) 88.2 (0.02)</td></tr><tr><td>AutoLR</td><td>1.05 (0.008)</td><td>81.2</td><td colspan="2">80.8 (0.52) 88.5 (0.09)</td></tr></table>

SQuAD fine-tuning on BERT. We report the average training loss, best test EM score, and average test EM, F1 scores over 3 runs.

Table content: <table/>

Table 6 shows the training loss and test accuracy averaged over 3 runs. As shown, AutoLR achieves a mean test BLEU score of 34.88, compared to 34.70 for the baseline. Figure4dshows the learning rate and validation perplexity of AutoLR against that of the baseline. See figure13for detailed comparisons of training/validation perplexity, learning rate, etc. We also compare against other learning rate schedules such as one-cycle, linear decay and cosine decay in Table10. It is interesting to note that both linear decay and cosine decay schedules outperform the baseline inverse square root decay schedule ofVaswani et al. (2017).

Table content: <table><tr><td>LR Schedule</td><td>Train ppl</td><td colspan="2">Validation ppl Test BLEU Score</td></tr><tr><td>Baseline</td><td>3.55 (0.029)</td><td>5.10 (0.033)</td><td>34.70 (0.001)</td></tr><tr><td>AutoLR</td><td>3.46 (0.16)</td><td>4.86 (0.014)</td><td>34.88 (0.005)</td></tr></table>

Training, validation perplexity and test BLEU scores for IWSLT on Transformer networks. The test BLEU scores are computed on the checkpoint with the best validation perplexity. We report the mean and standard deviation over 3 runs.Section E has more details of the one-cycle, cosine decay and linear decay baselines. Table7shows the test accuracy of different experiments on all learning rate schedules tried in this paper. As shown, AutoLR compares favorably against all other learning rate schedules, except slightly missing out in Cifar-10. For one-cycle we report results from the default policy of finding the maximum learning rate, as well as by hand tuning it (see section E for more details). The Trapezoid schedule follows the same seed learning rate and max learning rate as the tuned values of the corresponding one-cycle run.

Table content: <table/>

Test accuracy numbers for different experiments on all learning rate schedules tried in this paper. We report the average top-5 accuracy for ImageNet, average top-1 accuracy for Cifar-10, average BLEU score for IWSLT and the best EM score for SQuAD. The ImageNet results for one-cycle, cosine-decay, linear-decay are still running and will be updated in an updated draft.

Table content: <table><tr><td colspan="4">Experiment AutoLR Baseline Trapezoid</td><td colspan="2">One-Cycle</td><td colspan="2">Cosine Decay</td><td colspan="2">Linear Decay</td></tr><tr><td/><td/><td/><td/><td colspan="6">default tuned default tuned default tuned</td></tr><tr><td>ImageNet</td><td>93.01</td><td>92.90</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Cifar-10</td><td>94.79</td><td>94.81</td><td>94.69</td><td>93.38</td><td>94.59</td><td>94.52</td><td>-</td><td>94.24</td><td>-</td></tr><tr><td>IWSLT</td><td>34.88</td><td>34.70</td><td>34.85</td><td>32.35</td><td>34.57</td><td>0.33</td><td>34.85</td><td>0.28</td><td>34.82</td></tr><tr><td>SQuAD</td><td>81.2</td><td>80.7</td><td>80.1</td><td>79.8</td><td>80.9</td><td>80.8</td><td>-</td><td>80.7</td><td>-</td></tr></table>

found that DNNs have many local minima, but all local minima were also the global minima. It has been observed by several authors that wide minima gen- Goyal et al. (2017);McCandlish et al. (2018);Shallue et al. (2018)), by scaling the learning rate linearly as a function of the batch size.Jastrzebski et al. (2019) analyze how batch size and learning rate influence the curvature of not only the SGD endpoint but also the whole trajectory. They found that small batch or large step SGD have similar characteristics, and yield smaller and earlier peak of spectral norm as well as smaller largest eigenvalue.Dinh et al. (2017) show analytically using model reparameterization that wide minima can be converted to sharp minima without hurting generalization.Wang et al. (2018) analytically show that generalization of a model is related to the Hessian and propose a new metric for the generalization capability of a model that is unaffected by model reparameterization ofDinh et al. (2017).Yoshida & Miyato (2017) argue that regularizing the spectral norm of the weights of the neural network help them generalize better. On the other hand,Arora et al. (2018) derive generalization bounds by showing that networks with low stable rank (high spectral norm) generalize better.Guiroy et al. (2019) looks at generalization in gradient-based meta-learning and they show experimentally that generalization and wide minima are not always correlated. Finally,Golatkar et al. (2019) show that regularization results in higher test accuracy specifically when it is applied during initial phase of training, similar to the importance of AutoLRś explore phase during initial phase of training.Lower density of wide minima.Wu et al. (2018) compares the sharpness of minima obtained by batch gradient descent (GD) with different learning rates for small neural networks on FashionM-NIST and Cifar10 datasets. They find that GD with a given learning rate finds the theoretically sharpest feasible minima for that learning rate. Thus, in the presence of several flatter minimas, GD with lower learning rates does not find them, leading the authors to conjecture that density of sharper minima is perhaps larger than density of wider minima.

Table content: <table/>

Table 9 shows the training loss and test accuracy for the various schedules.

Table content: <table><tr><td>LR Schedule</td><td>Train Loss</td><td>Test Accuracy</td></tr><tr><td>Trapezoid</td><td colspan="2">0.0009 (8.7e-5) 94.69 (0.0007)</td></tr><tr><td>One-Cycle (default)</td><td>0.0003 (3e-5)</td><td>93.38 (0.001)</td></tr><tr><td>One-Cycle (tuned)</td><td colspan="2">0.0004 (3.1e-5) 94.59 (0.0007)</td></tr><tr><td>Cosine Decay</td><td colspan="2">0.0003 (2.7e-5) 94.52 (0.0016)</td></tr><tr><td>Linear Decay</td><td colspan="2">0.0002 (2.5e-5) 94.24 (0.0009)</td></tr><tr><td>AutoLR</td><td>0.0008 (1.0e-4)</td><td>94.79 (0.001)</td></tr></table>

Training loss and Test accuracy for Cifar-10 on Resnet-18 for more learning rate schedules.

Table content: <table/>

Table 11 show the average training loss, best test EM score, and average test EM, F1 scores for the various schedules.

Table content: <table><tr><td>LR Schedule</td><td>Train ppl</td><td>Validation ppl</td><td>Test BLEU Score</td></tr><tr><td>Trapezoid</td><td>3.25 (0.016)</td><td>4.92 (0.014)</td><td>34.85 (0.008)</td></tr><tr><td>One-Cycle (default)</td><td>5.51 (0.028)</td><td>5.87 (0.02)</td><td>32.35 (0.18)</td></tr><tr><td>One-Cycle (tuned)</td><td>3.82 (0.09)</td><td>5.01 (0.03)</td><td>34.57 (0.002)</td></tr><tr><td>Cosine Decay</td><td>9.54 (0.06)</td><td>3861.6 (883.4)</td><td>0.33 (1e-5)</td></tr><tr><td>Cosine Decay with warmup</td><td>3.29 (0.02)</td><td>5.04 (0.019)</td><td>34.85 (0.008)</td></tr><tr><td>Linear Decay</td><td>9.50 (0.07)</td><td>3261.5 (849.7)</td><td>0.28 (2e-5)</td></tr><tr><td>Linear Decay with warmup</td><td>3.30(0.02)</td><td>5.03 (0.016)</td><td>34.82 (0.006)</td></tr><tr><td>AutoLR</td><td>3.46 (0.16)</td><td>4.86 (0.014)</td><td>34.88 (0.005)</td></tr></table>

Training, validation perplexity and test BLEU scores for IWSLT on Transformer networks for more learning rate schedules. The test BLEU scores are computed on the checkpoint with the best validation perplexity. We report the mean and standard deviation over 3 runs.

Table content: <table><tr><td>LR Schedule</td><td colspan="2">Train Loss (av) EM (best)</td><td>EM (av)</td><td>F1 (av)</td></tr><tr><td>Trapezoid</td><td>0.95(0.002)</td><td>80.2</td><td colspan="2">80.1 (0.05) 87.9 (0.04)</td></tr><tr><td>One Cycle (default)</td><td>1.364 (0.011)</td><td>79.8</td><td colspan="2">79.4 (0.17) 87.3 (0.12)</td></tr><tr><td>One Cycle (tuned)</td><td>1.016 (0.004)</td><td>80.9</td><td colspan="2">80.7 (0.12) 88.4 (0.09)</td></tr><tr><td>Cosine Decay</td><td>0.89 (0.003)</td><td>80.8</td><td colspan="2">80.6 (0.06) 88.4 (0.13)</td></tr><tr><td>Linear decay</td><td>0.96 (0.075)</td><td>80.7</td><td colspan="2">80.4 (0.18) 88.2 (0.02)</td></tr><tr><td>AutoLR</td><td>1.05 (0.008)</td><td>81.2</td><td colspan="2">80.8 (0.52) 88.5 (0.09)</td></tr></table>

SQuAD fine-tuning on BERT for more learning rate schedules. We report the average training loss, best test EM score, and average test EM, F1 scores over 3 runs.

Table content: <table/>

Cifar-10 on Resnet-18 trained for 200 epochs with Momentum. We report averages over 3 runs.

Table content: <table><tr><td colspan="2">Seed LR Test Accuracy</td></tr><tr><td>0.05</td><td>94.60</td></tr><tr><td>0.075</td><td>94.71</td></tr><tr><td>0.1</td><td>94.79</td></tr><tr><td>0.125</td><td>94.58</td></tr><tr><td>0.15</td><td>94.25</td></tr></table>

Table 13 shows the training loss and test accuracy for the various runs. See figures 15, 16, 17 for more detailed comparisons of training loss, test accuracy, and learning rate for SGD, Momentum and Adam optimizers, respectively.

Table content: <table><tr><td>LR Schedule</td><td>SGD</td><td>Optimizer Momentum</td><td>Adam</td></tr><tr><td/><td colspan="2">Training Loss</td><td/></tr><tr><td>Baseline</td><td colspan="3">2.3e-5 (9.4e-7) 7.2e-6 (4.2e-7) 4.5e-3 (1.3e-3)</td></tr><tr><td>AutoLR</td><td colspan="3">2.4e-5 (5.5e-6) 7.3e-6 (7.7e-7) 3.9e-4 (9.5e-4)</td></tr><tr><td/><td colspan="2">Test Accuracy</td><td/></tr><tr><td>Baseline</td><td>99.32 (3.4e-4)</td><td>99.34 (6.0e-4)</td><td>99.21 (1.2e-3)</td></tr><tr><td>AutoLR</td><td>99.32 (3.7e-4)</td><td>99.38 (5.1e-4)</td><td>99.35 (7.2e-4)</td></tr></table>

Training loss and Test accuracy for MNIST. We report the mean and standard deviation over 7 runs.G.2 FASHION MNISTIn this experiment we trained the Fashion MNIST datasetXiao et al. (2017) on the same 4 layer network as for the MNIST dataset. We evaluated our method on SGD, SGD with momentum and Adam optimizers. For baseline runs, we used a fixed learning rate of 0.1 for SGD, Momentum, and 1e-3 for Adam; and trained for 50 epochs. With AutoLR, we trained the network with 10 explore epochs, and used the same seed learning rate as baseline, i.e. 0.1 for SGD, Momentum, and 1e-3 for Adam. Table14shows the training loss and test accuracy for the various runs. See figures 18, 19, 20 for more detailed comparisons of training loss, test accuracy, and learning rate for SGD, Momentum and Adam optimizers, respectively.

Table content: <table><tr><td>LR Schedule</td><td>SGD</td><td>Optimizer Momentum</td><td>Adam</td></tr><tr><td/><td colspan="2">Training Loss</td><td/></tr><tr><td>Baseline</td><td>0.005 (0.006)</td><td>0.073 (0.007)</td><td>0.018 (0.003)</td></tr><tr><td>AutoLR</td><td colspan="3">0.001 (0.001) 4.3e-4 (7.2e-4) 0.002 (0.006)</td></tr><tr><td/><td colspan="2">Test Accuracy</td><td/></tr><tr><td>Baseline</td><td>91.33 (0.005)</td><td>89.31 (0.003)</td><td>90.56 (0.003)</td></tr><tr><td>AutoLR</td><td>91.88 (0.001)</td><td>91.66 (0.002)</td><td>91.65 (0.003)</td></tr></table>

Training loss and Test accuracy for FashionMNIST. We report the mean and standard deviation over 7 runs.

Table content: <table/>

Figure11: Cifar-10 on Resnet-18 trained with Momentum. Shown are the training loss, test accuracy and learning rate as a function of epochs, for the baseline scheme (orange) vs the AutoLR scheme (blue). The plot is split into 3 parts to permit higher fidelity in the y-axis range.

Table content: <table><tr><td>0.4 0.6 0.8 0.0 0.2 0.4 0.6 1.0 1.2 1.4 1.6 0.8 1.0 1.2 1.4</td><td>0 0</td><td/><td>20 20</td><td>40 40</td><td>60 60</td><td>80 80</td><td>100 100</td><td>0.00 0.05 0.10 0.15 0.01 0.02 0.03 0.04 0.20 0.25 0.30 0.05 0.06 0.07 0.08</td><td colspan="2">100 100</td><td colspan="3">110 110 Training Loss vs Epochs 120 130 120 130 Training Loss vs Epochs</td><td>140 140</td><td>150 150</td><td>0.000 0.0025 0.0050 0.0075 0.0100 0.002 0.004 0.0125 0.006 0.0150 0.008 0.0175 0.0200 0.010</td><td>150 150</td><td>160 160</td><td>170 170</td><td>180 180</td><td>190 190</td><td>200 200</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">Test Accuracy vs Epochs Test Accuracy vs Epochs</td><td/><td/><td>0.950 0.934</td><td/><td/><td/><td/></tr><tr><td>0.80 0.85 0.85 0.90</td><td/><td/><td/><td/><td/><td/><td/><td>0.92 0.94 0.925</td><td/><td/><td/><td/><td/><td/><td/><td>0.930 0.948 0.932</td><td/><td/><td/><td/></tr><tr><td>0.60 0.65 0.70 0.75 0.70 0.75 0.80</td><td/><td/><td/><td/><td/><td/><td/><td>0.86 0.88 0.910 0.915 0.90 0.920</td><td/><td/><td/><td/><td/><td/><td/><td>0.944 0.922 0.924 0.926 0.946 0.928</td><td/><td/><td/><td/></tr><tr><td>0.50 0.55 0.60 0.65</td><td/><td/><td/><td/><td/><td/><td/><td>0.82 0.84 0.905</td><td/><td/><td/><td/><td/><td/><td/><td>0.940 0.942 0.918 0.920</td><td/><td/><td/><td/></tr><tr><td/><td>0 0</td><td/><td>20 20</td><td>40 40</td><td>60 60</td><td>80 80</td><td>100 100</td><td/><td/><td>100 100</td><td>110 110</td><td>120 120</td><td>130 130</td><td>140 140</td><td>150 150</td><td/><td>150 150</td><td>160 160</td><td>170 170</td><td>180 180</td><td>190 190</td><td>200 200</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">Learning Rate vs Epochs Learning Rate vs Epochs</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">0.104 0.00104</td><td/><td/><td/><td/><td/><td/><td colspan="3">0.08 0.10 0.0008 0.0010</td><td/><td/><td/><td/><td/><td colspan="2">0.0008 0.0022 0.0024 0.0010</td><td/><td/><td/></tr><tr><td colspan="2">0.102 0.00102</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.0020</td><td/><td/><td/><td/></tr><tr><td colspan="2">0.100 0.00100</td><td/><td/><td/><td/><td/><td/><td colspan="3">0.06 0.0006</td><td/><td/><td/><td/><td/><td colspan="2">0.0018 0.0006</td><td/><td/><td/></tr><tr><td colspan="2">0.098 0.00098</td><td/><td/><td/><td/><td/><td/><td colspan="3">0.04 0.0004</td><td/><td/><td/><td/><td/><td colspan="2">0.0014 0.0016 0.0004</td><td/><td/><td/></tr><tr><td colspan="2">0.096 0.00096</td><td/><td/><td/><td/><td/><td/><td colspan="3">0.02 0.0002</td><td/><td/><td/><td/><td/><td colspan="2">0.0012 0.0002</td><td/><td/><td/></tr><tr><td/><td>0</td><td>0</td><td>20 20</td><td>40 40</td><td>60 60</td><td>80 80</td><td>100 100</td><td colspan="2">0.00</td><td>100 100</td><td>110 110</td><td>120 120</td><td>130 130</td><td>140 140</td><td>150 150</td><td>0.0010</td><td>150 150</td><td>160 160</td><td>170 170</td><td>180 180</td><td>190 190</td><td>200 200</td></tr></table>

Peer review (iclr2020):
The paper proposes an automatic tuning scheme for learning rate while training neural networks. Since learning rate is the most sensitive and important hyperparameter during training, an automatic versatile method for choosing learning rate for various workloads would be of great importance. The proposed method works for various optimizers (SGD/Momentum/Adam shown in the paper) and shown to perform as good or better than typical baseline learning rate schedule practitioners use to obtain competitive performance. There are a good number of empirical checks on both image and language tasks. While I’m not fully familiar with literature on automatic learning rates, the authors claim that this is the first auto learning rate tuning scheme to achieve SOTA performance. Two main components of the scheme are based on observation that initial high learning rate seemingly without making any improvements is essential for obtaining good final performance. Therefore initially there is an `explore’ phase and then in exploit phase quadratic local approximation around high learning rate update is used. Novelty in the exploit phase is expansion respect to perturbation around some potentially large step size instead of assuming step size is small. One main concern of the proposed method is the choice of seed learning rate and duration of explore phase is still somewhat arbitrary and requires tuning for specific model/dataset. For example, in CIFAR/ResNet there are experiments showing that duration of `explore’ phase is important and choose 50 epochs. In the case of BERT for SQUAD fine tuning 2500 explore steps(half epoch) are chosen. It seems choice needs to be tuned to get good performance and with this the proposed scheme is semi-auto tuning at best. Similar points for seed learning rate could be made. In order for the proposed method to be fully successful, either authors need to show insensitivity to general choices of seed learning rate and explore phase, or an automatic method to choose good values. Without that I do not see significant improvement beyond tuning learning rate schedule via linear or cosine decay schemes (which is also known to perform comparatively to multiple drop schemes). In that spirit I think an interesting baseline to compare is tuning learning rate schedule. The proposed explore-exploit method has great potential in terms of generality and strong performance on various tasks. I would happily raise my score if the main concern is addressed, however at this point I slightly lean toward rejection. UPDATE POST AUTHOR RESPONSE: I thank the authors for carefully responding to the concerns raised in the initial review. I do suggest that refraining calling current method as *auto* since it could be misleading. I think full automatic learning rates, if it can be done, will have great impact in neural network optimization. I have raised my score as author*s response cleared some of issues (sensitivity to seed / comparison with other common tuned schedules).