AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
Authors: Yuchen Jin
Tianyi Zhou
Liangyu Zhao
Yibo Zhu
Chuanxiong Guo
Marco Canini
Arvind Krishnamurthy
Venue: ICLR 2021 Poster (2021)

Abstract: The learning rate (LR) schedule is one of the most important hyper-parameters needing careful tuning in training DNNs. However, it is also one of the least automated parts of machine learning systems and usually costs significant manual effort and computing. Though there are pre-defined LR schedules and optimizers with adaptive LR, they introduce new hyperparameters that need to be tuned separately for different tasks/datasets. In this paper, we consider the question: Can we automatically tune the LR over the course of training without human involvement? We propose an efficient method, AutoLRS, which automatically optimizes the LR for each training stage by modeling training dynamics. AutoLRS aims to find an LR that minimizes the validation loss, every $\tau$ steps. We formulate it as black-box optimization and solve it by Bayesian optimization (BO). However, collecting training instances for BO requires a system to evaluate each LR queried by BO's acquisition function for $\tau$ steps, which is prohibitively expensive in practice. Instead, we apply each candidate LR for only $\tau'\ll\tau$ steps and train an exponential model to predict the validation loss after $\tau$ steps. This mutual-training process between BO and the exponential model allows us to bound the number of training steps invested in the BO search. We demonstrate the advantages and the generality of AutoLRS through extensive experiments of training DNNs from diverse domains and using different optimizers. The LR schedules auto-generated by AutoLRS leads to a speedup of $1.22\times$, $1.43\times$, and $1.5\times$ when training ResNet-50, Transformer, and BERT, respectively, compared to the LR schedules in their original papers, and an average speedup of $1.31\times$ over state-of-the-art highly tuned LR schedules.

Published as a conference paper at ICLR 2021
AUTOLRS: AUTOMATIC LEARNING-RATE SCHEDULE BY BAYESIAN OPTIMIZATION ON THE FLY
Yuchen Jin, Tianyi Zhou, Liangyu Zhao University of Washington (yuchenj, tianyizh, liangyu) wwwsswawhington.c.
Yibo Zhu, Chuanxiong Guo
ByteDance Inc. (zhuyibe, guochuanxiong @ebytedance.cmm
Marco Canini
Arvind Krishnamurthy University of Washington
KAUST
ma maro@kaust.eduu
arvindêcs. washington.
ABSTRACT
The learning rate (LR) schedule one of the most important hyper-parameten needing careful tuning in training DNNs. However, it is also one of the least auto- mated parts of machine learning systems and usually costs significant manual effort and computing. Though there are pre-defined LR schedules and optimizers with adaptive I.R, they introduce new hyperparameters that need to be tuned separately for different tasks/datasets. In this paper, we consider the question: Can we auto- matically tune the over the course of training without human involvemen? We propose efficient method, AutoLRs which automatically optimizes the LR for each training stage by modeling training dynamics. Aut cLRsLRS aims to find an LR applied to every steps that minimizes the resulted validation loss. We solve this black-box optimization on the fly by Bayesian optimization (BO). However, col- lecting training instances for BO requires system to evaluate each LR queried by BO's acquisition function for 7 steps, which prohibitively expensive practice. Instead, we apply each candidate LR for only 7' < T steps and train an exponential model to predict the validation loss after steps. This mutual-trainin tween BO and the loss loss-prediction model allows us to limit the training steps invested in the BO search. We demonstrate the advantages and the generality of Aut o oLRS through extensive experiments of training DNNs for tasks from diverse domains using different optimizers. The LR schedules auto- auto-generated by Aut OLRS lead to speedup of 1.22x, 1.43x, and 1.5x x when training ResNet-50, Transformer, and BERT, respectively, compared to the LR schedules in their original papers, and average speedup of 1.31x over state-of-the of-the-art heavily- -tuneed LR schedules.
process be-
INTRODUCTION
In the regime of deep learning, the success training largely depends on the choice of the learning rate (LR) schedule, since most optimizers will have difficulty traversing non-smooth and non-convex loss landscape with multiple local minimums and possibly saddle points (Kawaguchi) 2016)] [Jin et al.] 2017, Goodfellow al. 2016 Li et al.) (018a). To achieve stable and fast convergence towards a solution with good generalization performance, one has to tune the LR schedules carefully for different tasks (Nar & Sastry| 2018 Jastrzębski et al.) 2017). This tuning is usually non-trivial and requires many tmal-and-erroi iterations that are computationally expensive. Moreover, the randomness of the widely-used mini-batch stochastic gradient descent (SGD) may introduce more uncertainty and difficulty in the tuning process. For the same reasons, it is also hard to directly formulate the search of the LR schedule well-posed optimization problem and address it through standard optimization.

---

Published as a conference paper at ICLR 2021
The broadly-adopter strategy is to either pick one from a family of pre-defined LR schedules or apply an optimizer that has a built-in mechanism changing the LR adaptively. However, we have a limited number of choices for pre-defined LR schedules, most of which are simple functions such as exponent or cosine and thus cannot perfectly align with the non-smooth loss landscape. The latter set of adaptive optimizers, e.g., Adam (Kingma & Ba 2015) and Adadelta (Zeiler 2012). are extended from convex optimization and rely on strong assumptions to make the convergence properties hold. Moreover, the methods in both categories introduce new syper-parameters that have to be tuned separately for different tasks or datasets, requiring significant human involvement.
In this paper, we study the question: can we automatically tune the LR over the course of training without human involvement? At the beginning of every T steps (i.e., a "stage" in our method), we seek to identify an LR that optimizes the validation loss (i.e.. an empirical estimate of the generalization error) at the end of the stage. To do so, we employ Bayesian optimization (BO) that treats the validation loss black-box function of LR. BO simultancously updates posterior estimation of the black-box function and searches for the best LR with respect to the posterior. This approach is, however, computationally expensive since estimating the posterior needs many (input, output) instances of the function, and acquiring each instance costs T steps of training. We, therefore, develop simple yet efficient approximation: for every LR that BO decides to evaluate, we train the model by using the LR for only 7' < T steps and use the validation loss over the r' steps to train a time-series forecasting model that provides prediction of the validation loss after 7 steps. As we will show later, an exponential model suffices to produce accurate predictions when using a small T' = T T/10. Then, Aut cLRS can allow BO to explore ten different LRs in each stage and still bound the total running time to approximately twice the training cost associated with the generated schedule, i.c., the time spent to find the stage-specific LRs is roughly equal to the time spent training the model with the identified LRs.
AutoLRS does not depend on a pre-defined LR schedule, dataset, or specified task and is compatible with almost all optimizers. Hence, it can be generally deployed across a broad range of ML tasks without much human involvement or expensive tuning over choices of LR schedules and their hyperparameters. Moreover, since it directly minimizes the validation loss, does not only accelerate the convergence but also improves the generalization compared to just minimizing the training loss. Furthermore, Aut oLRS only needs to update two extremely light-weigh models, i.e., the BO posterior and the exponential forecasting model, and it is efficient in exploring the loss landscape. Hence, it does not result in notable extra costs in either memory or computation. Note that Aut oLRS searches for better LRs based on the training dynamics, which can be seen as form of self- supervision. The interaction between BO and the forecasting model is an example of mutual learning, where one produces training data for the other.
In experiments, we apply AutoLRS to train three representative DNNs widely used in practice, i.e., ResNet-50 (He et al. (2) on ImageNet classification (Russakovsky et al. 2015); Trans- former (Vaswani et al. 2201 7) and BERT (Devlin et al. 2019) for NLP tasks. Though they have been extensively studied and have hand-tuned LR schedules, the LR schedules computed by Aut OLRS faster than the original, hand-tuned, LR schedules by 1.22 x, 1.43x, and 1.5x for training ResNet-50, Transformer, and BERT, respectively, in terms of the training steps used to update the DNN (i.c., excluding the costs of the LR/hyperparameter search). It meanwhile achieves test-set performance better on par with state thethhrrrt results. We also carefully hand-tuned two state-of-the-an art learning rate schedules, CLR (Smith 2017) and SGDR (Loshchilov & Hutter 2017). and conducted more than ten experiments with different CLR/SGDR hyperparameters an average speedup of 1. 1.29x and 1.34 across the three models, in terms of training steps, compared to the best CLR and SGDR LR schedules, respectively. The AutoLRS implementation is available at hattps://githut com/ wwwae//ucheennnnnnne wwtpannnnnnauuttrrrrro
each model. Aut OLRS still has
RELATED WORK
Learning rate scheduling: In contrast to traditional LR schedules with a monotone decreasing sequence of LRs and multi-step LR schedule, recent class of LR schedules propose to apply multiple cycles of LR decay. Cyclical Learning Rate (CLR) changes LR from a maximal LR (Imax) minimal LR (7/min) at a pre-defined frequency and achieves faster convergence for some DNNs (Smith 2017). The approach requires a "LR range test" to estimate the minimal and maximal LR. The L.R range test trains the model with linearlyincreeasing LR between a low LR
2

---

Published as a conference paper at ICLR 2021
and high LR, and finds the LR range (Mumin)7aax)) over which the training loss decreases. The authors proposed three variants of CLR: triangular2 that halves the maximum LR bound after each eycle; exp__taae tanae that exponentially reduces the maximum LR bound after each cycle; and Icycle containing only one triangular cycle (Smith) 2018). Similar to CLR, Stochastic Gradient Descent with Warm Restarts (SGDR) restarts the LR and then applies cosine annealing/decay at pre-defined frequency (Loshchilov & Hutter 2017). Neither CLR or SGDR is automatic, because they are quite sensitive to their hyperparameters, which require careful hand-tuning. CLR and SGDR may even cause undesirable divergence in loss during training with suboptimal hyperparameters (see $
Learning rate adaptation with hypergradient descent: Aiming for the same goal of automatically tuning the LR, the hypergradient based technique (Almeida et al. 1998 Franceschi et al. 2017] Baydin et al.) 2018: Donini et al. 2020) optimizes the LR schedule by applying gradient descent of the objective function w.r.t. the LR during training. In addition to the initial value of the regular additional hypergradient LR whose initial value is another hyperparameter to be
LR. it introduces specified. We experimentally show that this technique is subject to overfitting, it is quite sensitive to its two hyperparameters, and it is unable to match the state-of-the test-set performance on the models we test ( ($A.3.T) We also compare its performance against AutoLRS (SA.5.2).
DNN hyperparameter optimization: Automatic hyperparameter searching for DNNs has been broadly studied in recent years. When applied to learning rates, they can determine an optimized value for LR that is kept constant (or constrained to be a pre-defined shape) through the entire training process, as opposed to determining an LR schedule. They can be primarily categorized into Bayesian optimization based approaches (Hutter et al., [2011] [nnckk et al. 2012 Bergstra et al. 2013). sandit-based solutions (Li et al., 2017; 2018b), hybrid approaches that combine bandit-based and Bayesian optimization based approaches (Falkner al. 2018 Zela et 2018), and population-base methods (Jaderberg et al. 2017| Parker-Holder et al. 2020). It might be possible to extend these techniques to determine LR schedule with an optimized LR for each training stage, but it is not sample-efficient and time-efficient to do so since the LR schedule would correspond to hundreds or thousands of hyperparameters.
Optimization methods with adaptive LR: These optimizers can adaptively adjust LR for each training step by maintaining an estimate of better learning rate separately for each parameter in the DNN. Adagrad (Duchi et al. 201T) applies lower LRs to parameters with larger accumulated gradients and higher learning rates to the ones with smaller accumulated gradients. RMSprop (Tieleman & Hinton 2012). AdaDelta (Zeiler) 2012), and Adam (Kingma & Bal 2015) were later proposed to address the issue in Adagrad that the model stops learning due to the continual decay of LR. These optimizers with adaptive LR are orthogonal to our automatic LR scheduler, and they still require a global learning rate schedule, which can be obtained from our out. In particular, their default hyperparameters do not always work well and need careful tuning, e.g., Adam's is default LR 0.001 performs poorly in training BERT and Transformer, and better-tuned LR schedule can significantly reduce the training time (5.. Recent optimization methods (Schaul et al.] 2013) Mahsereci & Hennig 2015) proposed to remove the need for LR tuning in SGD altogether, but they are not widely used potentially due to their limited applicability and sub-optimal performance (Baydin et al. 2018
PROBLEM FORMULATION
Training of DNNs can be written in a general form of minimizing a loss function L(x; 8) over training samples x E Dirain. where 0 represents the model weights being optimized. The minimization is conducted by applying an optimizer that updates / iteratively. For example, at each step t, mini-batch SGD updates 0 using the gradient computed on a mini-batch of samples Brrain S Dirain:
0.+1 0,- [eeasin ExeBrrui. VoL(x;
(1)
where n. is the learning rate (LR) step t and VxL(x; 6.) denotes the gradient of the loss (xx;0) w.r.t. Og at step I. Given Barain and 0,, 0.+1 can be represented as a function of LR T)t i.e., 0c+ (mt).
Our ultimate goal is to search for an optimal schedule of LR, i.e., a sequence of LRs 77.1 A (m, (7,.72. ,.) applied to the total T training steps, such that the generalization error can be minimized. Ideally, we need to optimize the entire sequence of LRs. This, however, is intractable in practice given the large number of possible LR schedules and since evaluating each one of those possible LRR schedules requires a full training of T steps. Hence, we break down the LR schedule optimization into a dynamic optimization of constant LR for every 7 steps, which we refer to
3

---

Published as a conference paper at ICLR 2021
as "training stage". Since most tasks prefer relatively small LR due to the non-smoothness of DNNs' loss landscapes, when F is also small, the LR-resulted change on the validation loss might be too small and overwhelmed by the randomness of mini-batch SGD. Hence, in this case, need to increase т, so the effect of LR 7) on the validation loss can be accumulated for more steps to overcome noise. A large also reduces the frequency of applying LR search and saves computation. On the other hand, setting T to be too large might lose some optimality of the induced LR schedule. Therefore, need to trade-off the above two issues to find an appropriate т. In our final algorithm, we propose a curriculum for т, i.e., we start from a small т, in line with the greater volatility during early stages, and gradually increase focus on LR search within a Stage, for simplicity, we will use F instead of Tę for the exposition below.
training proceeds (as described in $4.4). Since we mainly
We study greedy approach and split the whole training process into multiple stages of steps each. We choose an LR at the beginning of each stage and apply r steps of optimization using this LR, i.c. at step-t 0, 2T, ... T T. we aim to find the LR ntt+++ that minimizes the validation loss Deal (i.c., an estimate of the generalization crror) after step- + T). This can be formulated as:
min, ExED... L(x; Oc+7 (T1),, 1 = 0, T, - ,,, -T-
(2)
We try to sequentially solve [T/r] sub-problems of the above form. However, we cannot apply stan- dard optimization to solve each sub-problem in practice because: (i) it is high-order optimization of 7 since we need to unroll 4 in Eq. (2 backward for T steps using Eq. ([.. which requires prohibitive memory and is unstable for DNNs; (ix) one step of optimizing 1) needs to apply steps of optimization on 8, which is costly and weakens the advantage of searching LR for better efficiency. To avoid these issues, treat the objective function in Eq. (2) for t t + a black-box function fc( and study how to optimize it based on the observed training dynamics through Bayesian optimization (BO).
4 AUTOMATIC LEARNING RATE SCHEDULE SEARCH
!T We first elaborate on the details of our BO algorithm 44.1 that identifies the LR for cach stage" However, collecting even one data point (1) (n)) for BO requires us to train the model for r steps, which is costly and impractical since the LR computed by the entire BO process is used for only steps. To reduce the cost of generating instances of (n. f(r).. in $4.2 $A.3 we propose to train a light-weight time-series forecasting model to predict f() based on the validation loss observed during the first r' (7' < T) steps of applying LR v). We find that a simple exponential model suffices to produce accurate predictions. Our LR search then reduces to a multi-training process between BO and the forecasting model, where one produces training instances for the other. The resulting algorithm can automatically find an LR schedule without introducing significant extra computation.
4.1 BAYESIAN OPTIMIZATION
BO (Shahriari et al. 2016) is one of the state-of-the-ard techniques for black-box optimization. It applies exploration and exploitation to the objective by sequentially and actively querying the function values of some input instances. Specifically, BO uses Gaussian process as a surrogate model (prior) to fit the black-box objective function fn) It sequentially updates a posterior of fn) by using its
likelihood on newly evaluated (7/. y: f(n)) + e) pairs where y is noisy observation of f()) and is the validation loss after T steps. Then, finds the next 7/.41 to evaluate based on an acquisition
function (n) defined by the posterior mean p4: (1,(1) and standard deviation (1))) uy (7) performs a
trade-off between exploration (i.e., large o; (7)) and exploitation (i.e., small (4())) In AutoLRS, we use Lower Confidence Bound (LCB) (Cox John 1992 Auer 2002) as (n). Given n/s and
their corresponding validation loss y1.i. we determine the next LR 7/+1 by minimizing LCB, i.e.,
4/41 = arg min,, us (1), (n) A p:(n)) - kn;(n),
where (7) and (n) are defined in Eq. 7 in $A.1 exploration and exploitation. In experiments, K. 1000 works consistently well. BO repeats the above process until it achieves a precise posterior distribution of f(). See $ for more details.
positive hyper-parameter to balance
'Sice this section mainly focuses to solve a aro-pprobem Eq. 2 within one stage 1+ we temporarily remove the subscript from fi) and other variahles/functionss that do not change within stage for simplicity. "Here i 1,... A for lt steps in BO: it indexes the exploration step of BO within training stage and differs from subscript indexing the training steps. We use superseript on r/' mark the LRs explored by BO.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: AutoLRS
Input (1) Number of steps in each training stage,
(2) Leamiin-raat search interval (Main; Mmmax)
(3) Number of LRs to evaluate by BO in cach training stage, k
(4) Number of training steps to evaluate each LR in BO, r'
(5) Trade-off weight in the acquisition function of BO, K.
while not converge do
initialize a GP prior: (o) = 0, on()) = (17, n) defined in Eq.
in A.1
c f checkpoint of model parameters and optimizer states; for + 1 to k do
f mutual mutual-trainigg loop between BO and loss forecasting model */
choose the next LR explore: T/. arg min, ( (1)):
M1.- train the DNN with LR r/. for r' steps and record the corresponding validation loss series;
yn train exponential forecasting model y1rr' and predict the validation loss after steps;
update the GP posterior by (1/, vi)) and update new pt and (T) using Eq. in A.1 restore the checkpoint of model parameters and optimizer states;
10
end
< the LR with the minimal predicted validation loss μt (71) among the k explored LRs T/L above; 11
12
train the DNN using LR 1)' for steps; 1" training model using BO- 30-searched best learning rate */
13
end
4.2 TIME-SERIES FORECASTING MODEL OF Loss
Typically, BO would requires T training steps to measure the validation loss associated with every LR y that it considers during a stage. This is computationally expensive. We now introduce a simple yet effective approach that substantially reduces the number of training steps required to evaluate each LR candidate: for each LR v that is evaluated, we only apply it for 7' < steps and the validation loss observed in the r' steps to train a short-term time-series forecasting model. We then use the resulting forecasting model predict the validation loss after 7 steps.
In numerous experiments, we observed that when a DNN trained with a reasonable LR, the validation loss typically decreases exponentially and converges to a small value. We show examples of practical loss time series and their exponential-model fitting results in Figure Moreover, recent al. 2019b) also proves the linear convergence of training DNNs. In addition, a simple model to fit the observed loss time-series can filter the noise and avoid possible overfitting. Hence, we propose train an exponential model in the form of L(() a exp(b() + c with parameters a, c and b < 0 and for / = I,..., T, as the forecasting model for the time series of the validation loss in a training stage of steps with given LR 17. $A.2 describes how estimate o,
deep learning theory (Allen-Zhu et
b, and e based on the validation loss observed in the first 7' steps, and A.3 describes how we filter noise and outliers.
4.3 MUTUAL TRAINING BETWEEN BO AND EXPONENTIAL PREDICTION
We present the complete procedure of AutoLsS in Algorithm T It sequentially optimizes LR for every training stage during the training of a DNN model, solely based on the observed training dynamics, and it can be seen as form of self-supervision. For each training stage, it searches for the LR that leads to the largest improvement in the validation loss via an efficient black-box function optimization conducted by mutual training loop between Bayesian optimization and short-term forecasting model for each loss series. It then applies the best LR among the explored ones for t steps and repeats the above process until convergence.
In line 5, the algorithm solves a constrained optimization problem over 17, in the range of [Mmin: Mmaa]] In practice, we prefer large learning-rate search interval (Amin: 7/max), across orders of magnitude, but also need fine-grained optimization over small LRs. Hence, we operate on 1 in its log-scale space, i.e., we replace by log 77 in Algorithm| [ except in lines 6 and 12 when we use the original LR (rather than log 1)) to train the DNN.
At the end of each iteration in the mutual training loop (line 9), we restore the checkpoint of model parameters and optimizer states the one saved the beginning of the training stage By doing so,
We save the checkpoint in the CPU memory, Autoo lutol.ss does not have GPU memory overhead.
5

---

Published as a conference paper at ICLR 2021
guarantee that the & different LRs all start from the same model and their losses can be compared. A.4 illusststs how BO learns the underlying function in practice for early and late stages of training.
Hyperparameters: AutoLRS substantially reduces the amount of hyperparameters that need to be hand-tuned in existing LR schedules or policies. However, as shown in Algorithm II we still have hyperparameters in out. First, we need to set a search interval (7min Tmax for LR. However, this interval can be reasonably wide by using an LR range test (Loshchilov & Hutter 2017) as we will show in $ Secondly, our default settings of k, T', T, and work well for a diverse set of DNN models from different domains and tasks, though it is possible to achieve further improvements by fine-tuning them.
4.4 PRACTICAL. IMPROVEMENTS
We found the following modifications can further improve the performance of Aut oLRS in practice.
Gradually increase over the course of training: Often, in DNN training, the loss and the model parameters experience rapid changes only during the first few epochs before they enter a phase of stable improvement. Our approach can adapt to this phenomenon. For the early stages, when the loss is less predictable for the time-series forecasting model, we use a small T (and r'). As training proceeds and the model becomes stable, we gradually increase (and r') and adjust the LR more lazily. This curriculum of increasing r places more exploration in earlier stages and more exploitation in later stages. In practice, we start with t 1000 and 7' 100. and double them after every stage until it reaches Tmax. Tmax is a hyperparameter that limits the maximum number of steps in stage. We will discuss more of Tmax in $5 This gradual increase of can provide stability to the LR schedule search. Similar strategies have been widely used in previous pre-defined LR schedules, e.g., the multi-stage schedule with increasing epochs within each stage, and some recent cyclical LR schedules (Loshchilov & Hutter 2017).
Computing the validation loss series for a candidate nf' requires considerable computation if we were to use the entire validation dataset at each step of mutual training. Recall, however, that the primary purpose of minimizing the validation loss instead of the training loss is to avoid overfitting on the training set when the training loss notoriously deviates from the generalization error. However, a variety of empirical evidence and recent theory ZAll ett al. (2019a) show that overfitting is unlikely while training aver-parameterized DNNs due to the inductive bias of random initialization and SGD, especially during the early phase of training. Hence, in practice, for the first several training stages, we can safely approximate the validation loss in our method by the corresponding training loss, which is by-product of forward propagation and free to obtain. In later stages (i.e., once T reaches Tiaa)), since the model is stable and the loss changes smoothly, we can evaluate the validation loss on small subset of the validation set without compromising robustness. In our experiments, this set is composed of merely 10 we evaluate the validation loss on them every 50 training steps (as opposed to every step). Therefore, the evaluation of validation loss in our approach does not introduce notable extra computation
Minimizing training loss in early stages:
and
5 EXPERIMENTS
We now evaluate Aut OLS by applying it to three widely- and representative DNNs: ResNet-50, Transformer, and BERT. Here are some highlights:
The LR schedules computed by AutoLRS are 1.22x, 1.43x, and 1.5x faster, in terms of train- ing steps, than the original, hand-tuned LR schedules for ResNet-50,, Transformer, and BERT, respectively. Meanwhile, it improves matches the test-set performance.
For each model, we carefully hand-tuned CLR and SGDR using more than ten experiments with different CLR/SGDR hyperparameters. Across the three models, the LR schedules computed by AutoLRS achieve an average speedup of 1.29x and 1.34x, in terms of training steps, over the best tuned LR schedules under CLR and SGDR, respectively. While CLR and SGDR had to be run
"The per-candidate-LLR total cost of evaluating validation loss during the BO search in a later stage is 10 A |'' r//50 where A is the time for computing the validation loss on one mini batch. Since the cost of back propagation is roughly twice that of forward propagation (Wen et al. 2018). the total cost of evaluating validation loss is approximately 1/15 of the training time spent on BO search.

---

Published as a conference paper at ICLR 2021
for at least 10 trials to find good LR schedule, AutoLRS only costs slightly over 2x the training
time associated with the computed LR schedule even after accounting for the BO search cost.
Auto LLRS is robust to the change of hyperparameters and consistently finds better LR schedules
than other baselines. In contrast, CLR and SGDR are sensitive to the choices of hyperparameters.
We perform ablation studies in $A.5.4|to demonstrate that both BO and the exponential forecasting model are essential for Aut oLRS to find good LR schedules.
Hypergradient descent is subject to overfitting, and it is unable to match the state-of-the-art test-set performance using all the guideline values of its two hyperparameters on VGG-16 (Simonyan & Zisserman 2015) and ResNet-50 P.. In contrast, AutoLRS can consistently improve or match the saate-of-the-and test-set performance with different Tmax values using fewer training
steps than the hand-tuned LR schedules ($A.5.2).
Using Hyperband (Li et al. 2017) for LR schedule search incurs a high computational overhead. Moreover, it cannot find an LR schedule that matches the state-of-the-arr accuracy ($A.5.3)
ML practitioners typically need to hand-tune the LR schedules carcfully for a long time to achieve satisfying performance, so the LR schedule adopted in each model's original paper is a presumably tough-to-beat baseline to compare with. For CLR and SGDR, we hand-tune their hyperparameters separately for each DNN. Hyperparameters in CLR include the high/low LR for the LR range test to sweep, the number of steps to perform the test, the number of steps in each triangular eycle, and the choice of variants (triangular2, exp_ p__aagee, Icycle) introduced in $2 Hyperparameters in SGDR include the number of steps/epochs in each cycle and the initial LR at the beginning of each eycle. We carefully tuned these hyperparameterss separately for each DNN and chose the LR schedule producing the best validation-sed set performance among 2 10 trials of different hyperparameters.
Baseline Setup:
Hy Hyperpaaameterss in AutoLRS: In our default setting, we set k 10 and T' 7/10 so that the training steps spent on BO equals the training steps spent on updating the DNN model. We start from 1000 and r' = 100 and double r and 7 after each stage until reaches Tmax- We use Tmax = 8000 for ResNet-50 and Transformer, Tmax = 32000 for BERT. We also tried Tmax = 8000, 16000, and 32000 for each DNN and found that the resulting LR schedules are not very sensitive to Tmax- (An analysis of the sensitivity to Tmax is in 5A.5.2 The LR search interval (Thin Mmax))
for R0,, Transformer, and BERT are (10-3 1). (I0 10 3), and (10 6, 10 3),, respectively. These are easily found by an LR range test (Loshchilov & Hutter 2017).
ResNet (He et al. 2016a b) is one of the most popular DNNs in computer vision tasks. We train ResNet-50 on ImageNet (Russakovsky et al.) 2015) using SGD with momentum on 32 NVIDIA Tesla V100 GPUs with data parallelism and a mini-batch size of 1024. The LR schedule in the original paper adopts a warmup phase of 5 epochs at the beginning and performs a 3-step decay as in (Goyal et al., 2017). Figure la presents different LR schedules for training ResNet-50 on ImageNet. We report how their top- accuracy on the validation set changes during training in Figure [b ALt achieves a speedup of 1.19x and 1.22x over SGDR and the original LR schedule respectively but is slightly (i.e., 5.4%) slower than CLR. Note that the best CLR result is achieved after 10 trials of heavy hand-tuning to hyperparameters. (In fact, 7 out of 10 CLR trials failed to achieve the best possible test-set accuracy, and the second best and the third best trials are 5.4% and 7.9% slower than Aut oLRS). oLRS achieves competitive speed even though it invests a significantly lower search cost that is comparable to the overall model update time associated with the identified LR schedule.
ResNet-50:
Transformer (Vaswani et al. 2017) is a neural machine translation (NMT) model that is built upon multi-head self-attention mechanism to capture the contextual dependencies and achieves promising translation performance. We train Transforme on a standard benchmark, i.e.. WMT 2014 English-German dataset, using 8 NVIDIA Tesla V100 GPUs. Following (Vaswani et al. 2017), we use Adam (Kingma & Ba 2015) with 33 = 0.9, 8g = 0.98, and E 10-9. The LR schedule in the original paper starts from linear warmup of 4,000 steps from 0 to 7e followed by 96,000 steps of decaying the LR proportionally to 1/v7 for step-t. In AuutooRS we also use the same linear warmup. The current Aut oLRS does not scarch LR for warmup steps since warmup
Transformer:
$ All the 1 accuracy on CITARENACCCTAARR-OOCTAAAAR the validation/test data excluding every sample used during training. We report top-1 validation accuracy on ImageNet and top-1 test accuracy on 10/CAR--//IFRR--I 100 becasue Imagenet does not disclose the labels of its test set, and CIFAR-1OICIFAR- 100 datasets only have the test sets (10,000 test images). 'We used the code htps://github.c wwwp//essooffo/mmmo/eeeeeoooo
we report in the paper is evaluated on all
/etp:////eeeeiii/////ffccccaaaaa 4e/offcia//rransformer, which achieves
27.3 BLEU (uncased) using the original LR schedule.
7

---

Published as a conference paper at ICLR 2021
Ausuuss 0-1g
Topa Recurecy
Leaering
78-
CLA
246A
AsteLi
Enigirl
C1A
5GDn
20 60
48330
20000 46030 00633 acdio 500033
Training Epech
Traninn Epoch
Training Step
Training Step
(a) LR on ResNet-50. (b) Val.Acc. on ResNet-50. (c) LR for Transformer.
(d) BLEU of Transformer.
Figure 1: Comparison of different LR schedules in training ResNet-50 ImageNet (a. b), and the Transformer base model (c. d). When training ResNet-50, AutoLRs, CLR, SGDR, and the original LR achieve 75.9% top-1 accuracy at epoch 74, 70, 88, and 90, respectively. When training Transformer base, Aut OLRS, SGDR, and original achieve 27.3 BLEU score (uncased) at step 69,000, 91,000, 98,000, respectively. CLR (the best we were able to find) achieves 27.2 BLEU score at step 99,000.
Learning Rate
AutoLRS
AuteLiS Original CLR
2ooaao Loss
AutiLiSS Oriiiaal
Original
CLR
50002 102000
150035
250002 250020
50300 130003
150020
200003
150000
175000
203000
225030
250000
Training Step
Training Step
Training Step
(a) LR schedules (Phase + 2). (b) Training loss Phase 1.
(c) Training loss in Phase 2.
Figure Comparison of different LR schedules and training loss in pre-training BERTRASF.
does not have explicit optimization objective, such as minimizing the validation loss. Warmup usually takes very few steps, and its main purpose is to prevent deeper layers in a DNN from creating training instability (Gotmare et al. 2019). Figure Ic| visualizes different LR schedules in training the Transformer model. Their BLEU scores on the test set during training are reported in Figure [d Overall, the LR schedule searched by AutoLRS yields a 1.32 1.43x speedup over the hand-tuned LR schedules. Auto LLS consistently achieves a similar amount of speedup over three trials- they achieve 27.3 BLEU score (uncased) at step 69,000, 69,000 and 70,000, respectively. Interestingly, if we continue the LR search of AutoLRS, we can get 27.4 7.4 BLEU score (uncased) at step 99,000.
BERT (Devlin et al. 2019 is recent model that achieved state-of-the-an results on 11 NLP tasks. It first pre-trains a language representation model on a large text corpus by unsupervised learning and then fine-tunes it for downstream NLP tasks. The BERT BASE model has 110M parameters, which makes the pre-training phase expensive. and hand-tuning the LR schedule might be impractical. We pre-train BERT with mixed precision (Micikevicius et al.) (018)) on the English Wikipedia and the BooksCorpus datase ? paper,
BERT Pre-training:
(huu et al. 2015a). Following the original Adam with 1.2 weight decay of 0.01 and 3 0.9, 82 0.999. The pre-training is divided into two phases: Phase includes 90% of the total training steps and uses a sequence length of 1 128, while Phase 2 uses sequence length of 512 for the rest 10% of training steps. We apply this two-phase training in the experiments of all LR schedules. We pre-train BERT BASE on 32 NVIDIA Tesla V 100 GPUs using mini-batch size of 1024 sequences, which is 4x the batch size in the original paper. To adapt the original LR schedule to our batch size, we tried both the linear scaling rule (Goyal et al. 2017) and the square root scaling rule (Krizhevsky/ 2014), and found that the square root scaling rule works better while the linear scaling rule made the loss diverge.
As shown in Figure 2 Phase 1 contains 150,000/225,000 steps and Phase 2 contains 16,000/25,000 steps respectively for AutoLRS and all baselines, since AutoLRS requires much less total steps. In both AUtoLRS and SGDR, we apply linear warmup in the first 2,500 steps make the deeper we report the training loss achieved by different schemes.
layers of BERT stable. in Figures 2b and 2c,
We fine-tune the pre- pre-trained models four downstream NLP tasks: Microsoft Research Paraphrase Corpus (MRPC) for identifying semantic textual similarity (Dolan & Brockett (005); Multi-Genn
Natural Language Inference (MNLI) for entailment classification (Williams al. 2018); Corpus of Linguistic Acceptability (CoLA) for predicting whether an English sentence is linuuistically ac- ceptable (Warstadt et al. 2019;; and Stanford Question Answering Dataset (SQuAD) v1.1 (Rajpurkar Table reports the after-fine fine----nnnnnee tuning performance on the four tasks. Since fine-tuning performance is unstable small datasets like MRPC, we fine-tuned on each task several times and report the best Dev-set performance. It shows that the model pre-trained by AutoLRS outperforms
ctt
We collected books from smashwords.com and built our own dataset with 980M words because the authors of BooksCorpus and longer have the dataset available for public download (7hu al. (0155).
8

---

Published as a conference paper at ICLR 2021
Table 1:
Pine-tuning BERTEASE that is pre-trained using different LR schedules on 4 downstream tasks. We report the accuracy the Dev of MRPC, MNII, and CoLA, and F1 scores the Dev set of SQuAD v1.1.
LR schedule (Phase I/Phase 2)
MRPC
MNLI
CoLA
SQuAD v1.1
Original 225,0000/250000
86.5
82.2
47.8
87.0
CLR (225,00 255.0000220000
86.0
80.7
44.4
86.5
SGDR 222,000025500000
84.8
81.6
38.7
86.2
Aut (150,000/16,000)
88.0
82.5
47.6
87.1
Table 2: Performance comparison with LR schedules searched by prior solutions on CIFAR-10 training with VGG-16 (batch size 128). Note that the hand-tuned LR schedule can achieve 93. top-1 test accuracy in 350 epochs. The Runtime column shows how long cach method takes on one NVIDIA Titan RTX GPU to find the LR schedule shown in the previous column. The runtime of HD and MARTHE include trying the guideline values of their hyperparameters to get decent LR schedule.
Method
Best top-1 accuracy achieved in 350 epochs
Runtime (seconds)
HD MARTHE Hyperband
91.31%
187,110
92.99%
67,578
93.24%
109,454
AutoLRS
94.13%
6,538
those using other LR schedules in most downstream tasks and meanwhile achieves a speedup of 1.5x. Note Aut oLRS consistently achieves this speedup over 3 trials (details in $A.5.5). We also tried pre-training using other LR schedules for fewer steps but the fine-tuning performances were worse. Notably, when we use CLR and SGDR for pre-training BERT the training loss diverged after 100,000 steps in several trials, even as we decreased the maximal LR and increased the number of steps per cycle. This illustrates how difficult and computationally intensive it is hand-tune the hyperparameters of existing LR schedules on complicated models and tasks. In contrast, AutoLRS significantly simplifies the process and saves human effort.
Hypergradient descent (HD) (Baydin et al. 2018) is hypergradient based method to adjust the learning rate in an online fashion by deriving the derivative of the training loss with respect to the learning rate, and performing gradient descent on the learning rate during training. MARTHE (Donini al. 2020) is generalization of two hypergradient based methods, HD and RTHO (Franceschi et al. (017). One distinction between MARTHE and HD is that MARTHE computes the gradient of the validation loss instead of training loss with respect to the learning rate. Hyperband is a multi-armed bandit approach for DNN hyperparameter optimization. We use HDD, MARTHE, and Hyperband to tune the LR schedules for CIFAR- training with VGG-16, and compare their performance with AutoLRS in Table 2 AutoLRS achieves higher best top-1 I test accuracy than the other methods as well as the hand-tuned LR schedule, with much less overhead. Detailed descriptions of these methods and the experimental results are in AA.5. and AA.53
Experimental comparison to prior methods:
6 CONCLUSION
We propose automatic learning-rate schedule method, AutoLRS. more efficient and versatile alternative to hand-tuning that can be broadly applied to train different DNNs for tasks in diverse application domains. We break down the sequence optimization learning rate search for minimizing validation loss in each training stage and then solve this sub-problem by Bayesian optimization (BO). To reduce the cost of BO exploration, stage training dynamics of BO exploration. AutoLRS achieves a speedup of 1.22x, 1.43x, and 1.5x training ResNet-50, Transformer, and BERT compared to their highly hand-tuned schedules.
train light-weight loss-forecasting model from the early-
ACKNOWLEDGMENTS
We would like to thank the anonymous ICLR reviewers for their valuable feedback. We would also like thank Damien Fay for his suggestions time series analysis. This work was partially supported by DARPA. For computer time, this research used the resources at ByteDance and the Supercomputing Laboratory at KAUST.

---

Published as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in neural information processing systems, pp. 6158-6169, 2019a.
Zeyuan Allen Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In Proceedings of the 36th International Conference on Machine Learning. volume 97, pp. 242-252, 2019b.
Luís B Almeida, Thibault Langlois, José D Amaral, and Alexander Plakhov. Parameter adaptation in stochastic optimization. On-Line Learning in Neural Networks, Publications of the Newton Institute, pp. 114-134, 1998.
Peter Auer. Using confidence bounds for exploitation-exploraiion trade-offs. Journal of Machine Learning Research, 3(Nov).397-422 2002.
Atilm Güney Baydin. Robert Cornish, David Martínez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representationss, 2018.
James Bergstra, Dan Yamins, and David D Cox. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python science conference, pp. 13-20. Citeseer, 2013.
Dennis D Cox and Susan John. A statistical method for global optimization. In Proceedings of the 1992 IEEE International Conference on Systems, Man, and Cybernetics, pp. 1241-1246. IEEE, 1992.
Zhongxiang Dai, Haibin Yu, Bryan Kian Hsiang Low, and Patrick Jaillet. Bayesian optimization meets Bayesian optimal stopping. In International Conference on Machine Learning, pp. 1496-1506, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers). pp. 2019. Association for Computational Linguistics.
Minneapolis, Minnesota, June
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing ( (/WP2005). 2005.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In IJCAI, pp. 3460-3468, 2015.
Michele Donini, Luca Franceschi, Orchid Majumder, Massimiliano Pontil, and Paolo Frasconi. MARTHE: Scheduling the learning rate via online hypergradients. In IJCAI-20, pp. 2119-2125, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):1111-2154 2011.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter opti-
mization at scale. arXiv preprint arXiv:1807.01774, 2018.
Luca Franceschi, Michele Donini, Paolo Frasconi. and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1165-1173. PMLR, 2017.
Marc G Genton. Classes of kernels for machine learning: a statistics perspective. Journal of machine learning research, 2993332 2001.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.
10

---

Published as a conference paper at ICLR 2021
Deepak Akhilesh Gotmare, Shirish Nitish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. international conference on learning representations. 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint vxXiv:/76.002677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016b.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown Sequential model-based optimization for general algorithm configuration. In International conference learning and intelligent optimization, pp. 507-523. Springer, 2011.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09845 2017.
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pp. 240-248, 2016.
Stanislaw Jastrzębski. Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint 1xXiv1711.006223 2017.
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In Proceedings of the 34th International Conference on Machine Learning- Volume 70. pp. 1732. JMLR. org, 2017.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp.
2016.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
A. Klein, Stefan Falkner, Jost Tobias Springenberg, and F. Hutter. Learning curve prediction with Bayesian neural networks. In International Conference on Learning Representations, 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv: 1404.5997, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-100 (Canadian Institute for Advanced cs. twronto......o... edu / -rri firiizz z //c ci far.farr.ttte
Research). htRp::/t:///e
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.
Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, pp. 6899, 2018a.
Liam Li, KG Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, MH Jonathan Ben-Tzur, Recht, and A Talwalkar. A system for massively parallel hyperparameter tuning. In Conference Machine Learning and Systems, 2020a, volume 1, 2018b.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband:
A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-663 2017.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In International Conference Learning Representations, 2017.
11

---

Published as a conference paper at ICLR 2021
Maren Mahsereci and Philipp Hennig. Probabilistic line searches for stochastic optimization. In Advances in Neural Information Processing Systems, volume 28, 2015.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference Learning Representations, 2018.
Kamil Nar and Shankar Sastry. Step size matters in deep learning. In Advances in Neural Information Processing Systems, pp. 3436-3444, 2018.
Jack Parker-Holder, Vu Nguyen, and Stephen Roberts. Provably efficient online hyperparameter optimization with population-based bandits. Advances in Neural Information Processing Systems, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics.
CE. Rasmussen and CKI. Williams. Gaussian Processes for Machine Learning. Adaptive Computa- tion and Machine Learning. MIT Press, Cambridge, MA, USA, January 2006.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. International Conference Machine Learning, pp. 343-351, 2013.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, January 2016. ISSN 0018-9219.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 464-472. IEEE, 2017.
Leslie N Smith. disciplined approach neural network hyper-parameters: Part 1-learning rate, batch size, momentum, and weight decay. arXiv preprint 1803.v:8003.000 2018.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 205222951999 2012.
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams, Freeze-thaw bayesian optimization. arXiv preprint arXiv: arXiv:1406..0 1406.3896, 2014.
T. Tieleman and G. Hinton. Lecture 6.5 6.s-Rmp Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 9998-6008, 2017.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo- independent weight perturbations on mini-batches. In International Conference on Learning Representations, 2018
12

---

Published as a conference paper at ICLR 2021
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, Volume (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguisties.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations, 2018.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. In ICML 2018 AutoML Workshop, July 2018
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings the 2015 IEEE International Conference Computer Vision (ICCV), ICCV '15 pp. 19-27, USA, 2015a. IEEE Computer Society. ISBN 9781467383912.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. BookCorpus website. https:: //ykzzha. wixsite. com mbweb 2015b.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 BAYESIAN OPTIMIZATION (MORE DETAILS)
BO (Shahriari et al. 2016) is one of the state-of-the-ard techniques for black-box optimization. It applies exploration and exploitation to the black-box objective by sequentially and actively querying the function values of some input instances. Specifically, BO uses Gaussian process surrogate model to fit the black-box objective function f(n). It updates a posterior distribution of (() by using its likelihood newly evaluated (n, y f(n) <) pairs, where y is noisy observation of f(n) and is the validation loss after steps in our case. Then, it determines the next LR 1) to evaluate as the one maximizing an acquisition function, which is computed from the updated posterior. The acquisition function performs a trade-off between exploration and exploitation in evaluating the candidates of LR. BO repeats the above process until achieving precise posterior predictive distribution of [())-
Surrogate model (prior): We apply a commonly used surrogate model - Gaussian process
(GP) (Rasmussen & Williims 2006) as the prior of the black-box objective function in Eq. (2)
A GP prior is specified by its mean function ju(.) and its covariance function (i.e., kernel function)
.. We adopt common choice uu-)) 0 and set ((:-)) to be the Matern kernel (Genton 2001
with smoothness factor v = 2.5 and length scale 1 = 1, which is defined as
1
<22 11,l22
V
K. <22ll m/4 11yl12
K (mR, n3) T(w)2--1
(4)
1
where (-) is a modified Bessel function and r(-) is the gamma function, and K (7%, n2)) performs convolution of the unit ball. Comparing the radial basis function (RBF) kernel which always generates infinitely differentiable functions that might be overly smooth, GP with Matern kernel can control the smoothness of generated functions to be [v] 1 times differentiable (Rasmussen & Williams, 2006). This helps to capture the less-smooth local changes. In our case, v = 225 leads to twice-differentiabee functions.
Posterior prediction: In the following, we will use simplified notations V/E. and f(%1.) for vectors composed of {n/)-= and (. respectivell, The GP prior indicates a Gaussian distribution over function values, i.e., f(T(x)) 1/1. ~ NN K) where K,, K (n/. m/)), Vi, E [A]]. After training steps using LR 1//- we evaluate the validation loss denoted by yı noisy observation of fvn)) i.e., V/ = f(%)) + where Gaussian white noise c ~ (N(0, o²). Given the noisy observations Yi:k, we can update the GP posterior of the black-box function f()) as
f(h ) Th Vick (N K + I21))..
(5)
Given new LR y,. we can now use the above GP posterior to predict the distribution of f(n) by the following reasoning based on Bayes' theorem, i.e.,
(()()) Tha vi.k) = / P(f(n)) (( tha Vr.k 31=x)dff (Tha),
(6)
which yields the posterior predictive distribution of f(1) as (N (1)n(1)..0 (n)))
f(r)l ThLs V1.k
Me, (1) 8 k(K + - yi:k:
(n) A K(n. kT + -
where k; = K(1, (.). The above result about single LR 1) can be trivially extended to multiple LRs.
Acquisition function: Given the posterior predictive distribution of (() in Eq 7. BO finds the next 7/4,1 to evaluate based on an acquisition function (1) defined by the posterior mean (4(1)) and standard deviation (n)) A promising acquisition function should balance the trade-off between exploration (i.e., large (z)) and exploitation (i.e., small (4(())). In AutoLRS,
Lower Confidence Bound (L.CB) (Cox & John| 1992 Auer (002) as our acquisition function. In particular, given 1/4 and their corresponding validation loss yi.k, we determine the next LR 1/441 by minimizing LCB, i.e.,
T/ arg min u4 (n). (((n) e (T( ko,(1)) 7 where (v)()) and (1)1) were defined in Eq. 7 k a positive hyper-parameter to balance the trade-off between exploration and exploitation. In experiments, we set 1000 and it works consistently well.
14

---

Published as a conference paper at ICLR 2021
mniiinn loss Expemintal
Valida 1555 Eeponartia
Tran mg loss Expenental
8.6
Vule
Locs
8.4
2.1
12
40 60
80
100
200
400
000
800
50
1000
150
200
Training Step
Traiing Stop
Training Stop
(a) Training loss during 100 train- ing steps and fitting by ex- ponential time-series forecasting model.
(b) Validation loss during 800 training steps and fitting by an exponential time-series forecast- ing model.
(c) corner case when exponen- tial model cannot fully capture the non-monotone change of the loss during the first 50 steps.
Figure 3: Fitting the time-series of loss by exponential model when training ResNet-50 on ImageNet.
A.2 EXPONENTIAL MODEL (MORE DETAILS)
We take two steps to estimate a, b, and in fitting the exponential model L(t) = a exp(bt) + c, based on the validation loss observed in the first r' steps, which is represented by y/, f = 1,. , First, we reduce the fitting problem to an optimization problem. Define function g(b) as the least squared error between predictions and observations w.r.t. and We can write the original fitting problem in the following two-stage form.
min g(),, g(b) A min 6<0
(a c y
a,e
It is a 1- I-dimensional optimization problem. Moreover, with b fixed, the minimization problem w.r.t. a, is linear regression problem that has a closed-form solution. Hence, we apply a simple gradient descent method that starts from an initial b, computes the linear least squares w.r.t. a, under b, search for the next b by the gradient descent method, and repeats these two steps. Thereby, practice we can achieve fast decrease on the regression error. In addition, to enforce the negative constraint be b -cxp(l/. The problem now reduces to
for b, we re-parameterize
min min a,c
(a (xexp(eex(()))) + =
ve)²
(10)
Although there might exist other possible strategies to optimize Eq. (2). we find the above method is stable and fast in reducing the regression error and thus keeps the fitting process highly efficient.
We empirically test whether the exponential model obtained by our method can ideally fit the loss time-series in different cases. Figure 3a and Figure 3b araare are two typical examples fitting the time series of training loss and validation loss by the proposed model. They show that the model can precisely predict the main trends of the time-varying loss, though ruling out some less informative noises.
In Figure 3d we also show a rare corner case when the model fails to fit the increasing loss in early steps. However, the loss- loss-increasing stage usually does not last long and thus the inaccuracy is not so harmful to the prediction of later-stage loss, which is our major goal since 7 is usually larger than the length of loss-increasing stage. To overcome such corner cases and outliers in the observed validation loss series, we present a pre-processing strategy to make stable exponential fitting in $A.3 Every time we predict the validation loss after steps, we first pre-process the loss observed in the T' steps, and then fit the pre-processed loss series with the exponential model.
In our empirical study, we also tried other more sophisticated time-series forecasting models including Holt- Winters, autoregressive integrated moving average (ARIMA) and singular spectrum analysis (SSA). We show two examples to compare their performance with our simple exponential prediction model in Figure
4 Some prior works also fit and predict learning curves (Swersky et al.) 2014 Domhan et al.) 2013 Klein al.] 2017| Dai et al. 2019) for early termination of evaluations of
hyperparameters when doing DNN hyperparameter optimization, but they need non-negligible time for training their models and performing inference. They are much more computationally intensive than our lightweight exponential prediction model, and this makes them less practical to be used in automatic L. schedule tuning.
15

---

Published as a conference paper at ICLR 2021
2.2
Groad
2,0
3.5
1.8
500
1000
1500
2000
1000
2000
3000
4000
Trairing Stop
Training Stop
(a) Predict the training loss after 2000 steps.
(b) Predict the training loss after 4000 steps.
Figure 4: Examples of forecasting the loss series by various time-series forecasting models when training ResNet-50 on ImageNet. Our simple exponential prediction model yields the least mean squared error (MSE) among all the models.
n 5
n 10
Loss curve Spline fit
curve Spline
Loss curve Spline fit
50
100
150
200
100
150
200
0
50
100
150
200
Training
Training Step
Training
Figure 5: The loss sequence in Figure spline smoothing.
3c
and its quadratic spline smoothing result after 1, 5, and 10 iterations of
A.3 PRE-PROCESS LOsS SERIES BY ITERATIVE SPLINE SMOOTHING
We show a corner case in Figure 3e where the loss decreases rapidly at first, then increases for a while, but finally decreases stably. It might be result of large LR happens when escaping from a possibly poor local minimum or a saddle point (Goodfellow et al, 2016). Our exponential model cannot fully capture the loss change in early steps of this case. But we also consistently observe in our experiments that the early instability of loss only lasts for at most hundreds of steps after we switch to new LR.
Nevertheless, we find that adding a pre-processing step to eliminate the noises, anomalies, and corner cases in the observed validation loss series makes the exponential fitting easier and more stable. Hence, we propose to apply an iterative spline smoothing to the validation loss observed in r' steps before training the forecasting model. In particular, when evaluating LR 17 for training stage, firstly run 7' training steps and fit the observed sequence of validation loss by a quadratic spline. We do such spline smoothing for multiple iterations. At the end of each iteration, we remove the loss values that are among the farthest 3% from the spline smoothing results they are collected in the first r1/2 steps (when the corner cases like the one in Figure 33 might happen). So the next iteration's 's spline smoothing only aims to fit the rest loss values. After certain number of iterations, we use the final spline smoothing values to train the exponential forecasting model.
Empirically, we find that 10 iterations of the above spline smoothing are necessary before training the exponential forecasting model. Figure shows the loss sequence from Figure Be before smoothing and after 1, 5 and 10 iterations of smoothing. As shown in the plots, the iterative spline smoothing can effectively remove the unnecessary noise and unstable changes during the early phase.
A.4 POSTERIOR LEARNED BY BO
6 shows how the BO posterior gradually learns an increasingly more accurate estimation of the underlying black-box objective. We also visualize the "learning progress" of BO in earlier stage and a later stage during training. It shows that in both early and late stages, by exploring more LRs, BO achieve more accurate posterior estimation of the objective function, and A 10 suffices to obtain a satisfying estimate. Moreover, the posteriors in the later stages have much smaller variance/uncertainty than in the earlier stages.
16

---

Published as a conference paper at ICLR 2021
Early Stage
1=5
i=10
15
15
15
fin)
10
10
10
10
10-1
10a
10-3
10"²
10
10a
10
10a²
10:1
10a
Late Stage
1=2
1=5
i=10
7.5
7.5
7.5
((())
5.0
Observations
5.0
5.0
2.5
2.5
2.5
10-1
10-2
10-1
10a
10-3
10
10
10a
10-1
10-2
10-1
10
Learning rate n
Figure 6: BO's posterior of the black-box objective function after exploring LRs (red dots determined by Eq. (3) at an early stage and late stage during training of ResNet-50 on ImageNet. The dashed lines show the mean function ft (1) (indicating the predicted validation loss applying LR n for 7 steps) and the shaded areas show the standard deviation (1)) (indicating the prediction uncertainty) in the form of (u(TT) o1()))
A.5 EXPERIMENTS (MORE DETAILS)
A.5.1 ANALYSIS OF ONLINE LEARNING RATE ADAPTATION WITH HYPERGRADIENT BASED METHODS
Hypergradient descent (HI) |aayiin al. 2018) is method adjust the learning rate in online fashion by performing gradient descent on the learning rate at the same time as the underlying DNN is optimized. For simplicity, rewrite Eq. [ which performs mini-batch SGD updates model weights 0 at each step f, as:
Oc+1 0, 7/ VL(O.).
(11)
where N is the learning rate (LR) step & and (L(),) denotes the gradient of the loss function L w.r.t. the model weights 0g at step t. By making the assumption that the optimal value of LR does not change much between two consecutive iterations, HD derives the partial derivative of the loss function L with respect to the learning rate ry:
aL(),) (e,)) -00-11-6L00000 VL(0)(--V(B.-..)
(12)
Dtj
An update rule for the learning rate constructed as:
1 ?e 30L(0.) 7) SVL(
(13)
which introduces hyperparameter the hypergradient learning rate. Updating the learning rate is a single vector multiplication between the gradient of the model weights at the previous step and the one at the current step. By updating both the learning rate 7)t and the model weights 0, using Eq, (13) and Eq. (T) in each step, the HD algorithm performs gradient descent on both learning rate and the model weights during training.
HD can be applied to optimizers including SGD, SGD with Nesterov momentum, and Adam. The original paper empirically shows that these optimizers equipped with HD are much less sensitive to the choice of the initial regular learning rate and the convergence rate is improved on set of tasks. However, the paper only compares HD with constant LR baselines on small models and small datasets. To study how HD compares to hand-tuned LR schedules on larger models and datasets, we train VGG-16 (Simonyan & Zisserman, 2015) and ResNet-50 neural networks on the CIFAR-10 image recognition dataset (Krizhevsky & Hinton 2009) with a mini-batch size of 128 using Py Torch implementation 8 A hand-funed L.R schedule consists of total of 350 epochs, starting with 0. and multiplying the learning rate by 0.1 at epoch 150 and 250. This hand-tuned LR schedule can achieve around 93. 73700% and 95.56% top-1 accuracy on the test set for VGG-16 and ResNet-50, respectively when we train the models on one NVIDIA Titan RTX GPU. We apply SGD with HD (SGD-HD) train the two models, sweep all the guideline values of the two hyperparameters (regular LR and
17

---

Published as a conference paper at ICLR 2021
hypergradient LR) in SGD-HD, and report the best top-1 accuracy that SGD-HD can achieve for VGG-16 and ResNet-50 within 500 epochs in Table Sable/aand and T We have three observations: (1) Hypergradient descent is very sensitive to the selection of the regular LR and the hypergradient LR. The top-1 accuracy ranges from 10.00% to 91.80% for VGG-16 16 and ranges from 10.00% to 92.52% for ResNet-50, with all suggested values of the two hyperparameters. (2) It cannot match the top-1 accuracy achieved with hand-tuned LR schedules: the best top-1 accuracy it can achieve among all the different hyperparameter settings are 1.90% and 3.04% behind the accuracy achieved with hand-tuned LR schedules for VGG- and ResNet-55,, even though we ran each of them 150 epochs more than the hand-tuned LR schedule. (3) It is prone to overfitting For example, when using regular LR = 10 3 and hypergradient LR- = 10 10-5 to train VGG-16, the top-1 accuracy is only 90.74% while the training accuracy is already 99.98%.
MARTHE (Donini et al.] 2020) adaptively interpolates between two hypergradient based methods, HD and RTHO (Franceschi et al. 2017). and it computes the gradient of the loss function the validation set instead of training set w.r.t. the learning rate. Besides the two hyperparameters in HD, MARTHE introduces another hyperparameter μ that controls how quickly past history is forgotten. We sample js between 0.9 and 0.999, sample the hypergradient LR in [10 10-3 10 6] log-uniformly, and set the initial LR to O.I, as how the MARTHE paper set its hyperparameters for training VGG-11 on CIFAR-10. We apply SGD with MARTHE to train VGG-16 on CIFAR-10 10. The best top-1 T0
accuracy MARTHE can achieve among all the hyperparameter settings in 350 epochs is 92.99%, which is 0.71 lower than the accuracy achieved with the hand-tuned LR schedule.
A.5.2 SENSITIVITY TEST OF Tmax IN Aut OLRS AND MEASURE OF VARIABILITY
Recall from $44.4 that AutoLRS starts with T = 1000 and 7' = 100, and doubles them after every stage until it reaches Tmax- We test the sensitivity of AutoIRS to this hyperparameter, Tmax, by comparing the generated L.R schedules with different Tmax values for the VGG- 16 neural network on CIFAR-10 10 as in $ The LR search interval (77mnn we use is (10-3 10- We report the training epochs to reach the target 93. 70% top-1 accuracy using the LR schedules generated among 5 trials for different Tmax values in Table 6 AutoLRs with different Tmax values can consistently achieve the target top-1 accuracy achieved with the hand-tuned LR schedule (i.e., 93.70% in fewer training steps. We also see that the best Aut outoos--geeeate LR schedule can achieve 94.13% top-1 accuracy within 350 training epochs (excluding the costs of the LR search).
In the last column of Table 6 we report the mean and standard deviation of the top-1 accuracy achieved by Ru over trials for each Tmax- To further measure the variability of outol.sS, we train VGG-16 on CIFAR-100 100 (Krizhevsky et al.) with a mini-batch size of 128. A carefully hand- tuned LR schedule consists of a total of 200 epochs, starting with 0.1 and dividing the learning rate by 5 at epoch 60, 120, and 160 This hand-tuned LR schedule can achieve 72.93% top-1 accuracy, We train VGG-16 on CIFAR- for 200 epochs with AutoLRS for 10 trials using different random seeds, and report the top-1 accuracy they achieve in Table The LR search interval (Tmmin 7mmax) we use is (10-3, 10-1), and Tmax is set to 8000. The top-1 accuracy achieved by MeroooRS-ggeeeteee LR schedules over 10 trials are distributed with a mean of 73.05% and a standard deviation of 0. 0.14% The best AutoLRS LRS-generaeed LR schedule can achieve 73.30% top-1 accuracy, which is 0.37% higher than the accuracy achieved using the hand-tuned LR schedule.
A.5.3 LEARNING RATE SCHEDULE SEARCH WITH HYPERBAND
Hyperband is a multi-armed bandit approach for DNN hyperparameter optimization. It dynamically allocates resources to randomly sampled configurations and uses successive halving (Jamieson & Talwalkar 2016) to early stop poorly-performing configurations. We attempt to use Hyperband to optimize the LR schedule on CIFAR- 10 training with VGG-16 by searching for an exponential decay LR schedule, which can be parameterized with an initial learning rate and a decay factor. The learning rate is decayed by the decay factor every epoch. Exponential decay a commonly used LR schedule and is also used in other DNN hyperparameter optimization methods (Falkner et al., 2018). We use the search space of (10-3, - 10-1) for the initial IR, and the search space of (0.9, 1) for the decay rate. The decay rate is uniformly random sampled, and the initial LR is uniformly random sampled in its log-scale space. We use the default setting of Hyperband that sets the maximum epochs that can be
18

---

Published as a conference paper at ICLR 2021
allocated to single configuration to 350 and discards two-thirds of the configurations in each round of successive halving. This results in evaluating 384 configurations with different numbers epochs with total of 12600 epochs. which has a computational overhead of 36x compared to a single run of training with the hand-tuned LR schedule. The best configuration found by Hyperband achieves 93. 33.24% top-1 accuracy, which is 0.46% lower than the accuracy achieved with the hand-tuned LR schedule.
A.5.4 ABLATION STUDY
To illustrate the effects of the exponential model and BO of AutoLRS, we perform ablation studies using the VGG-16 16 neural network on CIFAR- 10 as in $A.5.1
Exponential model: What if we remove the exponential forecasting model and simply use the validation loss at 7' step to update the BO posterior? Will the LR schedules generated by Aut oLRS be significantly worse? We apply AutoLRS without the exponential forecasting schedules for VGG-16 16 on CIFAR-10. 10. With Tmax being chosen from the set of { (4000, 8000, 16000), the best top-1 test accuracy that AutoLRS can achieve within 350 training epochs are 91.73%. 92.59%. 92. 92.24%. respectively. Therefore, it is unable to match the target accuracy in reasonable training steps without the exponential forecasting model. The reason for this is that the objective of BO has become to minimize the validation loss at the 7' step, which will lead to short-horizon issues (Wu et al. 2018). As a consequence, it tends to select conservative LR, which is often small LR around Mmain in the late stages. In contract, with the exponential forecasting model, the goal of the BO is to find the LR that minimizes the predicted validation loss in steps. This allows the LR selected in the current stage to be higher than that in the past stages, and the loss to even increase in short period of time, as long as the predicted loss in steps is low. This phenomenon can be seen in Figure I and Figure 2
find the LR
Bayesian Optimization: What if we replace BO in AutoLRS with random search or grid search? Will the LR schedules generated by AutoLRS get worse? We replace the BO part in Aut outoLRS with random search and grid search while keeping the exponential forecasting part of it, and apply the LR schedules for VGG-16 on CIFAR-IO. The LR search interval is (10-3 10-1), the same as in and Table & show the results of random search and grid search with different Tmax values, respectively. We observe that both random search and grid search have at least one trial that fails to match the hand-tuned LR schedule to achieve 93. 93.70% top-1 test accuracy within 350 epochs (denoted by N/A in the tables). The top-1 accuracy achieved on average across trials in 350 epochs by random search and grid search 0.09% and 0.24% behind Aut with BO, respectively, We also replace BO with grid search and apply it to find the LR schedules for VGG-16 on CIFAR- 100. The top-1 accuracy achieved over 10 trials are distributed with a mean of 72.63% and standard deviation of 0.56%. Compared to the outo LRg-gggrrred LR schedules in Table 9 the mean of the grid search accuracy is out of two standard deviations from the BO accuracy 73.05 % 0..1%.
find
$A.5.2 Table and
A.5.5 AutoLRS FINE-TUNING RESUL.TS OF BERT BASE
BASE ACROSS 3 TRIALS
We pre-trained BERT BASE with AutoLRS for 3 trials, and report their fine-tuning results in Table
Table 3:
Finc-tuning results of BERT models pre-trained with AutoLRs for trials. Accuracy scores on the Dev set are reported for MRPC, MNLI, and CoLA, Fl scores on the Dev set are reported for SQuAD v1.1.
MRPC
MNLI
CoLA
SQUAD vl.1
Trial
88.0
82.5
47.6
87.1
Trial
88.0
82.7
46.5
87.0
Trial
87.8
82.3
47.0
86.6
19

---

Published as a conference paper at ICLR 2021
Table 4: The accuracy information of tuning the regular LR and the hypergradient LR of SGD-HD for CIFAR- training with VGG-16 (batch size 128). We train the model for 500 epochs using SGD-HD with each suggested value regular LR and hypergradient LR, and report best top-1 test accuracy can achieve. its corresponding training accuracy, and the epoch number. Note that a hand-tuned LR schedule can achieve 93.701 70% top-1 test accuracy in 350 epochs.
regular LR
hypergradient LR
Top-1 Test Accuracy
Training Accuracy
Epoch
10
86.13%
99.77%
438
10
10
88.79%
99.98%
480
10
10
86.3 %
98. 10%
494
10 10-6
10 10-3
90.70%
99.95%
499
10
10-2
10.30%
9.90%
40
10
6
10-1
10.00%
10.00%
1
10
10
86. 14%
99.7 99.73%
394
10
10
88.49%
99.95%
448
10
10
87.67%
98.78%
483
10
10
88.70%
99. 99%4
469
10
2
10.22%
9.92%
170
10
10
10.00%
10.00%
1
10
10
86.09%
99. 8.%
481
10
10
88. 88.82%
99.94%
304
10
10 10-1
86.63%
95.37%
479
10
10-3 3
10.22%
10.13%
1
10
10-2
10.02%
10.00%
1
10
10
10.00%
10.00%
1
10
10
86.13%
99.73%
406
10
10-5
88.78%
99. 19.99%
346
10
10-4
90.74%
99.98%
484
10
10
44.12%
43.02%
500
10
10
88.48%
99.55%
467
10
10
10.00%
10.00%
1
10
10-6
91 1.9%%
99.97%
389
10
10-5
88.53%
99.89%
397
10
10
89.11%
99.92%
484
10
10 10-3
10.07%
9.90%
265
10
10
10.00%
10.02%
10
10-1
10.00%
9.99%
10
10-6 6
91.80
99.93
476
10
10-5
91.48%
99.85%
317
10
10
88.8 11% %
99.57%
499
10
10-3
90.42%
99.80%
393
10
10-2
11.24%
10.45%
I
10
10-1
10.00%
10.02%
1
20

---

Published as a conference paper at ICLR 2021
Table 5: The accuracy information of tuning the regular LR and the hypergradient LR of SGD-HD for CIFAR-10 training with ResNet-50 (batch size 128). We train the model for 500 epochs using SGD-HI with each suggested value for regular LR and the hypergradient LR, and report the best top-1 test accuracy can achieve, its corresponding training accuracy, and the epoch number. Note that achieve 95.56% top-1 test accuracy in 350 epochs.
LR schedule can
regular LR
hypergradient LR
Top-1 I Test Accuracy
Training Accuracy
Epoch
10
83.67%
99.71%
410
10
10
88.75%
99.4 99.44%
490
10
10
83.77%
99.68%
494
10 10-6
10-3
71.03%
72. 14%
491
10
10-2
10.11 %
10.03%
261
10
6
10-1
10.0%
10.0%
10
10
83.99%
99.64%
420
10
10
89.15%
99.97%
460
10
10
10.12%
9.95%
206
10
10
19.73%
18.53
13
10
2
10.03%
9.98%
137
10
10
10.0%
10.0%
10
10
84.98%
99.85%
488
10
10
89.27%
99.94%
482
10
10 10-1
84.36%
97.78%
424
10
10-3 3
88.72%
99.84%
484
10
10-2
10.00%
10.00%
10
10
10.00%
10.00%
10
10
83.22%
99.81% 11%
487
10
10-5
88.56%
99.98%
492
10
10-4
86. 86.00%
97.32%
440
10
10
10.10%
9.76%
367
10
10
42.80%
40.1
497
10
10
10.00%
10.00%
1
10
10-6
92.40%
99.99%
459
10
10-5
88.51% %
99.98%
440
10
10
90.72%
99.91%
452
10
10 10-3
10.19%
9.64%
315
10
10
10.05%
9.99%
8
10
10-1
10.00%
10.00%
10
10-6 6
92.18%
99.97%
487
10
10-5
92.52% %
99.97
494
10
10
87.74%
99.86%
492
10
10-3
84.32%
97.23%
477
10
10-2
10.00%
10.11%
I
10
10-1
10.00%
10.00%
1
21

---

Published as a conference paper at ICLR 2021
Table Performance of AutoLRS with different Tmax values for CIFAR- 10 training with VGG- (batch size 128). a hand-tuned LR schedule can achieve 93.70% top-1 test accuracy epochs. We report the top-1 accuracy achieved within 350 epochs for cach trial, and the mean and standard deviation of the top-1 accuracy achieved by Aut OLRS over 5 trials for each Tmax.
Tinax
Trial Number
Epoch to 93.70% Top-1 Accuracy
Top-1 Accuracy Achieved
Mean-tstd
Trial
94.13%
Trial
181
93.96%
4000
Trial
223
94.07%
94.01%++0.153
Trial
315
93.82%
Trial
287
94.09%
Trial
115
94.03%
Trial 2
265
93.92%
8000
Trial
203
93.94%
93.96%:10.079
Trial
194
94.02%
Trial
93.87%
Trial
229
93.77%
Trial
250
93.95%
16000
Trial
93.73%
93.80%
Trial
93.71%
Trial
330
93.82%
Table 7: Experimental results after replacing BO in Aut with random search for CIFAR-I6 10 training with VGG-16 (batch size 128). We also report the top-1 accuracy achieved within 350 epochs for each trial.
Tmax
Trial Number
Epoch 93. 70% Top-1 Accuracy
Top-1 Accuracy Achieved
Trial
199
93.80%
4000
Trial 2
209
93.97%
Trial
298
93.84%
Trial Trial 2
344
93.71
8000
225
93.98%
Trial 3
175
93.91%
Trial
N/A
93.64%
16000
Trial 2
316
93.96%
Trial 3
310
93.86%
Table 8: Experimental results after replacing BO in Aut outo with grid search for CIFAR-10 training with VGG-16 (batch size 128). We also report the top-1 accuracy achieved within 350 epochs for each trial.
Taxx
Trial Number
Epoch 93. 70% Top-1 Accuracy
Top-1 Accuracy Achieved
Trial
304
93.88%
4000
Trial 2
233
93.88%
Trial 3
180
93.91%
Trial
239
93.72%
8000
Trial 2
296
93.95%
Trial 3
N/A
93.32%
Trial
N/A
93.02%
16000
Trial 2
153
93.78%
Trial 3
288
93.70%
Table 9: Top-1 test accuracy achieved by RuS.e..Seeeeeeeteee schedules CIFAR-100 training with VGG-16 16 over trials.
73.12%
73.20%
72.90%
72.93%
73.03%
73.16%
73.30%
72.85%
73.00%
72.97%
22


Reviews:
[{"review_id": "pGfd_ukAgv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper uses Bayesian optimization (BO) to dynamically tune the learning rate during the course of training of DNNs.  In every stage of training, the algorithm firstly uses BO to explore different learning rates with the help of a parametric exponential model for learning rate extrapolation, and then applies the selected learning rate at the current stage. The algorithm is applied to the training of state-of-the-art DNN models, and is shown to outperform the original learning rate schedules, as well as other methods for learning rate scheduling.\n\nStrong points:\n- I think this is a very interesting and practical application of Bayesian optimization (BO).\n- The way BO is used for learning rate selection in every stage is very clever, especially the trick of reverting back to the checkpoint model parameter before the evaluation of every LR, which resolves the non-stationarity issue due to evolving model parameters.\n- The experiments are conducted using state-of-the-art DNN models such as transformers and BERT.\n- The paper is in general vey well written, although some places describing the algorithm could be improved as specified below.\n\nWeak points/questions:\n- I think the brief descriptions of the algorithm especially in the Abstract and Introduction are not very accurate and sometimes confusing (especially the 3rd paragraph of the Introduction). I couldn't really tell how the algorithm works until I've read Algorithm 1. I feel it would be easier to understand if these descriptions clearly say that: in every stage, BO is firstly used to select the learning rate, where each learning rate evaluation during BO starts at the same model parameter checkpoint and is early-stopped using an exponential model; after BO, the selected optimal learning rate is used for throughout the current stage.\n- Since your algorithm incurs additional computational cost due to BO, I think it would be a more fair comparison if you compare the runtime (instead of training epochs/steps as in Figures 1 and 2) of different algorithms. I wonder whether the proposed algorithm still has an advantage after taking into account the additional computational cost due to BO.\n- In the experiments, the number of BO iterations is selected as k=10. I think this is a small number of iterations for BO. Given that BO needs to perform exploration at the initial iterations, I wonder is how your algorithm will perform if you replace BO by pure exploration (i.e., simple random search)?\n- The proposed Algorithm 1 seems extremely similar to Refs [1] and [2] given below, and hence they should be referenced. Both [1] and [2] also use an exponentially decaying kernel to model the evolution of the learning curve, and hence early-stop some evaluations of hyperparameters (learning rate).\n\n[1] Bayesian optimization meets Bayesian optimal stopping, ICML 2019.\n[2] Freeze-thaw Bayesian optimization, Arxiv, 2014.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h3vleGUQGu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The auto method presented in the paper is based on Bayesian optimization, but with an modification to make it less expensive. The method should be interesting to machine learning community. \n\nThe is clearly written and relatively easy to understand. The idea proposed in new and the experimental results demonstrate its effectiveness. \n\nPros: \n1. The method is new and seems to be effective in speedup  the training several types of models. Using a forecast model to reduce running time is an interesting idea, thought the forecast model is an exponential time-series forecasting model. \n2. The paper demonstrates the effectiveness of this method by pretty solid experiments. The three models used are popular and important ones. \n3. The paper also describe two practical improvements( Gradually increase $\\tau$ over the course of training, and Minimizing training loss in early stage) \n\nCons:\n1. The method itself introduced a few hyperparameters including k, $\\tau$, etc. Tuning them seems a manual process. The paper can say a bit more how and why choose values used in the experiment\n2. The paper focuses on learning rate. In practice, there are many other hyperparameters that have impact on performance. How to make sure the results reported in paper is independent of the setting of other hyperparameters?  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6VJt9VnTApo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Overview\n---\nTraining deep neural networks is typically done using gradient-based methods with either pre-defined learning-rate schedules or off-the-shelf adaptive optimizers (such as Adam). The former can not reliably align with the non-linear loss landscape, while the latter add additional hyperparameters to tune. This paper proposes an algorithm for automatically tuning a learning-rate schedule. The method works by modelling the training dynamics and adapting the learning-rate to optimize performance on the validation set. The method does end up introducing additional hyperparameters,\nbut the empirical results suggest some degree of robustness to them. Empirical performance is investigated on a number of widely-used models which are expensive to tune.\n\n**Overall, I recommend this paper for acceptance.** I have some minor concerns which are detailed below, but generally the clarity was excellent throughout and the approach seems novel.\n\nIt seems that there are various \"hidden\" hyperparameters in addition to the ones stated. The algorithm seems to perform well in a variety of problems with them fixed, but there is a concern that the default settings suggested in the paper could just happen to fit well in these particular problems. For example,\n  - **Page 4**: Don't you need to know (or guess) the variance of the noise term $\\varepsilon$ to fit a GP? If so, how was this noise variance chosen? what's the sensitivity to it?\n  - **Page 14, Appendix A.2**: *\"Hence, we apply a simple gradient descent method that starts from an initial b, computes the linear least squares w.r.t. a, c under b, search for the next b by the gradient descent method, and repeats these two steps.\"*\n    The problem in a,c can be solved in closed form but how is the step-size for descent in b chosen? Is this a tuned hyper-parameter? How was it chosen?\n  - **Page 15, Appendix A.3**: *\"At the end of each iteration, we remove the loss values that are among the farthest 3%\"* How was 3\\% chosen? This one in particular seems like it would depend a lot on how noisy the particular data is\n\nMy other concern is that there's no indication of statistical spread in any of results (neither tables nor figures). It  would be nice to see at least standard error bars so we know the variance in performance of the proposed method, and how it compares to the baselines. I assume this is because the experiments presented are too expensive to perform enough trials to get statistically significant results, but it would be worth-while to add an appendix showing results on smaller tasks on which variance can be assessed. \n\n## Clarifications\n---\n- **Page 2**: *\"Moreover, since it directly minimizes the validation loss, it does not only accelerate the convergence but also improves the generalization.\"* This isn't true necessarily, one could just be overfitting to the validation set. This is why we tune on a validation set and test performance on a separate test set --- to ensure that the performance truly does generalize to unseen data\n\n## Minor Comments (which did not affect my score)\n---\n- **Page 4**: \"loss decreases exponentially\" vs. \"linear convergence\" (iteration complexity) is an unfortunate quirk in the optimization literature, as the equivalence is easy to miss. It would be useful for many readers if the equivalence was clarified --- in a footnote, perhaps --- or if the same convention was used in both phrases.\n- I think an interesting addition to the empirical results would be something like \"computation to performance\" for each of the methods. For example, how much compute is it going to cost me to reach a certain level of performance using each method when we account for time spent tuning hyperparameters. I am unsure what the best measure would be here, but tuning non-trivial models is often so expensive overall that it would really showcase the benefits of the proposed method. (To be clear, *I am not requesting such a result*)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0q5Le39isi6", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an algorithm (AutoLRS) to tune learning rate schedules based on Bayesian Optimization with Gaussian processes and loss curve forecasting with exponential functions. The authors conduct experiments on three large scales settings in computer vision and natural language processing.\n\nStrengths:\n- This is probably one of the first attempts to use BO to fit LR schedules on the fly.\n- The exponential forecast model seems a simple but effective idea. \n- Experiments are very large scale and show that particular instantiations of the proposed algorithm achieve noticeable speed-ups and can also improve final performances in some cases.\n\nWeaknesses:\n- The method has several hyperparameters, which potentially limits its significance and applicability.\n- Unclear contribution of the BO part of the method.\n- Comparisons with other methods and experimental protocol could be improved.\n\nComments:\n- AutoLRS has several hyperparameters.  The authors mention four in the relative paragraph, but considering also Sec 4.4 I personally count at least another three. They argue that some standard choices work well across the three experimented settings, but some of them (e.g. the number of iterations) are clearly task-dependent. Besides, there are also several choices related to the BO part of the method that feel somewhat arbitrary to me (choice of kernel function for the GP/acquisition function). Would they work also in another domain/learning setting, or should they also be considered as hyperparameters of AutoLRS? The presence of all these configuration parameters may render the application of the proposed algorithm impractical (lowering the significance of this work).\n- The paper reads overall fairly well, but I think it could have been organized better. Some information (e.g. description of training details and in-depth reports of the numerical results) could have been postponed to the appendix and, vice-versa, some content of the appendix e.g. comparison with other methods should have been included in the main text.\n- There is no ablation of the BO part of the method.  I find this quite surprising and I would like to see this in the rebuttal. Specifically, looking at Figure 6 (last column), it seems that after 10 samples the points are more or less evenly spaced (in log scale). I wonder if the BO part of the method adds that much. One could simply pick 10 values on a grid (or sample randomly) and only run the exponential forecasting part of the method. This reminds me of [1]; which could be related. \n- I have doubts about the comparisons with other methods. A) It is not clear to me how the authors searched for the parameters of the schedulers (CLR/SGDR), besides that they did >= 10 trials. I would argue that manual search is not a fair method in this setting  B) Within the hypergradient descent class, the authors only reported some results with the method of Baydin et al. but none with Donini et al. (which is a generalization of HD that also considers optimizing with a validation set). C) Runs with Hyperband use exponential decay. Depending on which functional form they use, sampling the decay rate uniformly in (0,1) may be unnecessarily punitive. E.g. if the schedule is $\\eta_0 \\delta^{t}$ then clearly I would expect that only values of $\\delta$ close to 1 would yield decent results. These points can be clarified in the rebuttal phase.\n- The authors should be very careful when using the word \"optimal\" and related. As they say, the method that they propose is a greedy one that performs a (very) partial stagewise optimization. Therefore it is not clear to me concerning what they consider the various quantities as `\"optimal\". I strongly suggest replacing the occurrences with words such as \"tuned\" or similar. \n\nMinor comments:\n- The experimental results do not report any kind of measure of uncertainty, for instance, related to the random seed. While this is understandable for the large scale experiments, I would have appreciated a set of smaller-scale experiments to consider this.\n- Algorithm 1 does not use explicitly $\\eta_{min}$ and $\\eta_{max}$. To improve clarity, authors should specify that line 5 solves a *constrained* optimization problem over $\\eta$, in the range $[\\eta_{min}, \\eta_{max}]$\n- A small explanation of the “LR range test” would be useful, due to the fact it is used to calibrate 2 of the 6 hyperparameters of the proposed method.\n\n\nReferences\n[1]Zhang, Michael, et al. \"Lookahead optimizer: k steps forward, 1 step back.\" Advances in Neural Information Processing Systems. 2019.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been actively discussed, both during and after the rebuttal phase. I am thankful for the active communication that took place between the authors and some of the reviewers.

The paper was, overall, found quite clear, with an interesting methodology (especially the introduction of a forecasting step) and a solid large-scale experimental evaluation. As a result, it is recommended for acceptance.

However, several concerns remained after the rebuttal phase and we strongly encourage the authors to try to improve the following aspects of their submission:
* Clarify as much as possible (notably in the light of the ablation studies further added in the paper) the importance & impact of the BO component (which cast some doubts among the some reviewers on its necessity to get good performance)
* Transparently discuss the choice of, _and the robustness with respect to_, the “hyper-hyperparameters” of the proposed method (e.g., k, tau, tau’, kappa, tau_max, mini-batch size of validation set,...). Such an in-depth discussion is essential to fully demonstrate the practical value of the method. 

Decision: Accept (Poster)