MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks
Authors: Jun Shu
Yanwen Zhu
Qian Zhao
Deyu Meng
Zongben Xu
Venue:  (2021)
Keywords: Meta Learning
Hyperparameters Learning
Generalization on Tasks
Optimization
LR Schedules Learning
DNNs Training

Abstract: The learning rate (LR) is one of the most important hyper-parameters in stochastic gradient descent (SGD) for deep neural networks (DNN) training and generalization. However, current hand-designed LR schedules need to manually pre-specify a fixed form, which limits their ability to adapt to non-convex optimization problems due to the significant variation of training dynamics. Meanwhile, it always needs to search a proper LR schedule from scratch for new tasks. To address these issues, we propose to parameterize LR schedules with an explicit mapping formulation, called MLR-SNet. The learnable structure brings more flexibility for MLR-SNet to learn a proper LR schedule to comply with the training dynamics of DNN.  Image and text classification benchmark experiments substantiate the capability of our method for achieving proper LR schedules. Moreover, the meta-learned MLR-SNet is tuning-free plug-and-play to generalize to new heterogeneous tasks. We transfer our meta-trained MLR-SNet to tasks like different training epochs, network architectures, datasets, especially large scale ImageNet dataset, and achieve comparable performance with hand-designed LR schedules. Finally, MLR-Net can achieve better robustness when training data is biased with corrupted noise. 

Under review as a conference paper at ICLR 2021
MLR-SNET: TRANSFERABLE LR SCHEDULES FOR HETEROGENEOUS TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
The learning rate (LR) is one of the most important hyper-parameters in stochastic gradient descent (SGD) for deep neural networks (DNN) training and generalization. However, current designe LR schedules need to manually pre-specify a fixed form, which limits their ability to adapt to non-convex optimization problems due to the significant variation of training dynamics. Meanwhile, it always needs search proper LR schedule from scratch for new tasks. To address these issues, we propose to parameterize LR schedules with an explicit mapping formulation, called MLR-SNet. The learnable structure brings more flexibility for MLR-SNet to learn a proper LR schedule to comply with the training dynamics of DNN. Image and text classification benchmark experiments substantiate the capability of our method for achieving proper LR schedules. Moreover, the meta-learned MLR-SNet plug- and-play to generalize to new heterogeneous tasks. We transfer our meta-trained MI.R-SNet to tasks like different training epochs, network architectures, datasets, especially large scale ImageNet dataset, and achieve comparable performance with hand-desie hand-designed LR schedules. Finally, MLR-SNet can achieve better robustness when training data are biased with corrupted noise.
INTRODUCTION
Stochastic gradient descent (SGD) and its many variants (Rbbinn & Monro 1951 Duchi et al. 2011 Zeiler 2012 Tieleman & Hinton 2012 Kingma Ba 2015), have been served as the cornerstone of modern machine learning with big data. It has been empirically shown that DNN achieves state- of-the-art generalization performance on a wide variety of tasks when trained with SGD ZZang et al. (017). Several recent researches observe that SGD tends to select the so-called flat minima (Hochreiter & Schmidhuber 1997a Keskar al. 2017), which seems to generalize better in practice.
Scheduling learning rate (LR) for SGD is one of the most widely studied aspects to help improve the SGD training for DNN. Specifically, it has been experimentally studied how the LR (Jastrzebski et al. 2017) influences mimima solutions found by SGD. Theoretically, Wu et al. (2018a) analyze that LR plays an important role in minima selection from a dynamical stability perspective. He et al. (2019) provide PAC-Bayes generalization bound for DNN trained by SGD, which is correlated with LR. In a word, finding a proper LR schedule highly influences the generalization performance of DNN,
which has been widely studied recently (Bengio 2012 Schaul et al. 2013 Nar & Sastry 2018).
There mainly exist three kinds of hand-designed LR schedules: (1) Pre-defined LR policy is mostly
used in current DNN training, like decaying or cyclic LR (Gower al. 2019
Loshchilov & Hutter
2017), and brings large improvements in training efficiency. Some theoretical works suggested that
the decaying schedule can yield faster convergence (Ge al. 2019 Davis et al. 2019) or avoid strict
saddles (Lee et al. 2019 Panageas et al. 2019) under some mild conditions. (2) LR search methods in tranditional convex optimization (Nocedal & Wright 2006) can be extended to DNN training
by searching LR adaptively in each step, such as Polyak's update rule (Rolinek & Martius 2018 8),
Frank-Wolfe algorithm (Berrada et al. 2009), and Armijo line-search (Vaswani et al. 2019). etc. (3)
Adaptive gradient methods like Adam (Duchi et al. [001 Ticleman & Hinton| 2012 Kingma & Ba 2015), adapt LR for each parameters separately according to some gradient information.
Although above LR schedules (as depicted in Fig. ((a) and ((b) can achieve competitive results on their learning tasks, they still have evident deficiencies in practice. On the one hand, these policies need to manually pre-specify the form of LR schedules, suffering from the limited flexibility to adapt to non-convex optimization problems due to the significant variation of training dynamics. On the other hand, when solving new heterogeneous tasks, it always needs to search proper LR

---

Under review as a conference paper at ICLR 2021
MLR-SNet
CIFAS11D & CIFAR-100
Penn Treebanc
Looming
Papanenttal SCOR
A(LG
W++1 Wt
at
W
Epoch
(a) Pre-set LR schedule image datasel
(b) Pre set LR schedule text dataset
(c) Diagram of MLR SNet principle
CIFAR-10 110
Treebonk
CFAR, 100
Ponn Tresbank
Hommnng Paro
Lawning
CIFAA-15 10
C1FA0-100
2- L5TH
34ayer L5TH
OursiTeati
Ousiiiss
Epach
Epoch
Eporh
Epoch
(d) LR schedule learned on image dutasse
(e) schedule
text dataset
(f)
Predicted schedules by MLR-SNG SNet
Figure I: Pre-set LR schedules for (a) image and (b) text classification. (c) Visualization of how we input current loss L, to MLR-SNet, which then outputs a proper LR a to help SGD find better minima. LR schedules learned by MLR-SNet on (d) image and (e) text classification. (f) We transfer LR schedules learned on CIFAR-10 10 to image (CIFAR- 100) and text (Penn Treebank) classification, and the subfigure shows the predicted LR during training.
schedule from scratch, as well as to tune their involving hyper-parameters This process is time and computation expensive, which tends to further raise their application difficulty in real problems.
To alleviate the aforementioned issues, this paper presents a model to learn a plug-and-play LR schedule. The main idea is to parameterize the LR schedule as a LSTM network (Hochreiter & Schmidhuber 1997b), which is capable of dealing with such long-term information dependent problem. As shown in Fig. [[c] the proposed Meta-LRRSchhedun
(MLR-SNet) learns an explicit loss-LR dependent relationship. In nutshell, this paper makes the following three-fold contributions.
(1) We propose a MLR-SNew to learn an adaptive LR schedule, which can adjust LR based on current training loss as well as the information delivered from past training histories stored in the MLR- Met- Due to the parameterized form of the MLR-SNet, it can be more flexible than hand-designed policies to find proper LR schedule for the specific learning task. Fig(Tc)) and [(e] show our learned LR schedules, which have similar tendency as pre-defined policies, but more variations at their locality. This validates the efficacy of our method for adaptively adjusting LR according to training dynamics.
(2) With an explicit parameterized structure, the meta-trained MLR-SNet can be transferred to new heterogeneous tasks (meta-test stage), including different training epochs, network architectures and datasets. Experimental results verify that our plug- plug-and-pl and-pla LR schedules can achieve comparable performance, while do not have any parameerrsssrss compared with tranditional LR schedules. This potentially saves large labor and computation cost in real world applications.
(3) The MLR-SNet is meta-learned to improve generalization performance on unseen data. We validate that with the guidance of clean data, our MLR-SNet can achieve better robustness when training data are biased with corrupted noise than hand-designed LR schedules.
2 RELATED WORK
Meta learning for optimization. Meta learning has a long history in psychology (Ward] 1937 Lake
al. (0017). Meta learning for optimization can date back to 1980s- 1880s-190ss (Schmidhuber 1992
Bengio et al.] 1991). aiming to meta-learn the optimization process of learning itself. Recently,
Andrychowicz et al. (2016) Ravi & Larochelle (2017) Chen et al. 2017);; Wichrowska al. (2017)
Li & Malik (2017) Lv et al. (2017) have attempted to scale this idea to larger DNN optimization problems. The main idea is to construct meta-learner as the optimizer, which takes the gradients as input and outputs the whole updating rules. These approaches tend to make selecting appropriate training algorithms, scheduling L.R and tuning other hyper-parameter in an automatic way. Except for solving continuous optimization problems, some works employ these ideas to other optimization problems, such as black-box functions (Chen et 2017), few-shot learning (Li et al. 2017)), model's curvature (Park& & Oliva 2019), evolution strategies (Houthooft et al. 2018). combinatorial functions (Rosenfeld et al. 2018). etc.
Though faster in decreasing training loss than the traditional optimizers in some cases, the learned optimizers may not always generalize well to diverse problems, especially longer horizons (Lv et al., 2017) and large scale optimization problems (Wichrowska al. 2017). Moreover, they can not be
2

---

Under review as a conference paper at ICLR 2021
guaranteed to output a proper descent direction in each iteration for DNN training, since they assume all parameters share one small net and ignore the relationship between each parameters. Our proposed method attempts to learn adaptive LR schedule rather than the whole update rules. This makes it easy to learn and the meta-learned LR schedule can be transferred to new heterogeneous tasks.
HPO and LR schedule adaptation. Hyper-paramest optimization (HPO) was historically inves- tigated by selecting proper values for algorithm nyper-parameters to obtain better performance on validation (see (Hutter al., 2019) for an overview). Typical methods include grid search, random search (Bergstra & Bengio 2012), Bayesian optimization (Snoek et al. 2012), gradient-based methods (Franceschi al., [017] Shu et al. 2020a b). etc. Recently, some works attempt to find a proper LR schedule under the framework of gradient- -asee HPO, which can be solved by bilevel optimization (Franceschi et al., 2017] Baaydin et al.) 2018). However, most HPO techniques tend to fall into short-horizon bias and easily find a bad minima [Wu et al.. (018b). Our MLR-SNet has an explicit function form, which makes the optimization of the LR schedules more robust and effective.
Transfer to heterogeneous tasks. Transfer learning (Pan & Yang 2009) aims to transfer knowledge obtained from source task to help the learning on the target task. Most transfer learning methods assume the source and target tasks consist of the same instance, feature or model spaces (Yang et al., 2020), which greatly limits their applications. Recently, meta learning (Finn et al. 2017) aims to learn common knowledge shared over distribution of tasks, such that the learned knowledge can transfer to unseen heterogeneous tasks. Most meta learning approaches focus on few shot learning framework, while we attempt to extend it into standard learning framwork. The hand-designed LR schedules and HPO methods just try to find proper LR schedule for given tasks, and need to be learned from scratch for new tasks. However, our meta-learned MLR-SNet is plug-and-play, which can directly transfer how to schedule LR for SGD to heterogeneous tasks without additional learning.
3
THE PROPOSED
(MLR-SNET) METHOD
The problem of training DNN can be formulated as the following non-convex optimization problem,
1 N (w),
min (Ty WER
where CTV is the training loss function for data samples i Dr, = 11.2, ,N). which characters the deviation of the model prediction from the data, and w R2 represents the parameters of the model (e.g., the weight matrices in DNN) to be optimized. SGD (Robbins & Monro 1951 Polyak
1964) and its variants, including Momentum [Teeng 1998), Adagrad (Duchi ct al. 2011), Adadelta
(Zeiler 2012). RMSprop (Tieleman & Hinton 2012). Adam (Kingma & Bal 2015). are often used
for training DNN. In general, these algorithms can be summarized as the following formulation,
W++1 w/ + Awl, Aw, O,(V (w.), He: Ot),
(2)
where w/ is 1-th updating model parameters, V VCTr(ws, denotes the gradient of L at wr. He represents the historical gradient information, and θ, the hyperparameter of the optimizer O, e.g.. LR. To present our method's efficiency, we focus on the following vanilla SGD formulation.
1
Σ
VC (wy)
W++1 Wi
[.
iEB,
where B, C Dry denotes the batch samples randomly sampled from the training dataset, [Be] denotes the number of the sampled batch samples, and V.TTv (CTT(uu (wx) denotes the gradient of sample computed at We and on is the LR at t-th iteration.
3.1 EXISTING LR SCHEDULE STRATEGIES
As Bengio 2012) demonstrated, the choice of LR remains central to effective DNN training with SGD. As mentioned in Section I
variety of hand-designed LR schedules have been proposed. Though they achieve competitive results on some learning tasks, they share several drawbacks: (1) The pre-defined LR schedules suffer from the limited flexibility adapt the significantly changed training dynamics for the non-convex optimization problems. (2) It needs to be learned from scratch to find proper I.R schedule for the new tasks, which raises their application difficulty in real problems.
3

---

Under review as a conference paper at ICLR 2021
Sil=Fiind
SLEMI Pall Stap
Re-s
Apua
Adam
Figure 3: (Above) Train loss and (Below) test loss function of point on a random ray starting at the solutions for different methods on CIFAR- 1-0 with ResNet-18.
time-step
time
time
Leai Potat wote
Crnatenat Cap
(a) Computational graph MLR SNet
(b) One step of MLR R.R.R-NNe
Figure 2: The structure of our proposed MLR-SNet.
2017 Shu et al., 2018 2019), some Inspired by current meta-learning developments (Finn et al. researches proposed to learn a generic optimizer from data (Andycchoiccz et al. 2016; Ravi & Larochelle| 2017 Chen al., 20177 Wichrowska et al., 2017; Li & Malik 2017; Lv al. (017). The main idea to learn meta- learerr as the optimizer to guide the learning of the whole updating rules. For example. /Andrychowicz et al. (2016) try replace Eq. (2 with the following formulation,
W++1 We + It. [9t, het1l" = (v. Ø),
where Se is the output of LSTM net m, parameterized by o. whose state is h4. This strategy can make selecting appropriate training algorithms, scheduling LR and tuning other cyyper-parametrr in a unified and automatic way. Though faster decreasing training loss than the traditional optimizers in some cases, the learned optimizer may not always generalize well more variant and diverse problems, like longer horizons (Lv et al. 2017) and large scale optimization problems (Wichrowskal al. 2017). Moreover, it can not guarantee output proper descent direction in each iteration for network training. This tends to further increase their application difficulty real problems.
Recently, some methods (Franceschi et al. 2017; Baydin et al. 2018) consider the following con- strained optimization problem to search the optimal LR schedule a* such that the produced models associated with small validation error,
min
av DVal, wr), s.t. W141 (e(w,, a1), t= 0,1, T - 1,
(5)
o= a=/apr" ar-1)
where L denotes the validation loss function, Dv = (1,2, ... MM denotes hold-out validation set, is to-be-solved hyper-parameter, di R" x R__ + Rn is stochastic weight update dynamics, like the updating rule (2) or the vanilla SGD in (3. and T is the maximum itcration step. Though achieving comparable results some tasks with hand-designed LR schedules, they can not directly transfer to new tasks, since they do not have an explict transferable structure form.
3.2 PROPOSED
SEE.LL---SIEEEEEEEEE (MLR-SNET) METHOD
To address aforementioned issues, the main idea to design meta-leamer with an explicit mapping formulation to parameterize LR schedules as shown in Fig [(c) called MLR-SNet. The parameterized structure can bring two benefits: 1) It gives more flexibility to learn a proper LR schedule to comply with the significantly changed training dynamics of DNN: 2) It makes the meta-learned LR schedules be transferable and plug-and and-play, which can be applied to new heterogeneous tasks.
Formulation of MLR-SNet. The computational graph of MLR-SNe is depicted in Fig Let A(:0) 0) denote the MLR-SNet, and then the updating equation of SGD in Eq. (3) can be rewritten as
Wh+11 we A((;)))
Σ iEB.
VCT
L.
Σ iEB.
C (wr)),
[B
BB
where 0g the parameter of MLR-SNet at f-th iteration 0, . T 1). At any iteration steps, A(:; () can learn explicit loss-LR dependent relationship, such that the net adaptively predict LR according to the current input loss L., as well as the historical information stored the net. For every iteration step, the whole forward computation process is (as shown in Fig. 2(b)
C fo 3 c--1 +4 4
ft =
a
W2
ReLU ReLU
)
Wi
h4-1
Li )
h = o O tanh(c))
(7)
Ot
Pt (Wsht)
9t tanh
Ote = Y
4

---

Under review as a conference paper at ICLR 2021
where it, or or denote the Input, Forget and Output gates, respectively. Different from vanilla LSTM, the input he-1 and the training loss L₁ are preprocessed by fully-connected layer W1 with ReLU activation function. Then it works LSTM and obtains the output h, After that, the predicted value Pt is obtained by a linear transform W3 on the h2 with Sigmoid activation function. Finally, we introduce a scale factor 7 -to guarantee the final predicted LR located in the interval of [0, 7)- Albeit simple, this net is known for dealing with such long-term information dependent problem, and thus capable of finding a proper LR schedule comply with the complex variations of training dynamics.
adapting to the training dynamics of DNN. The MLR-SNet can be meta-trained to improve the generalization performance on unseen validation data for DNN training as follows:
Meta- Train:
min LV (Dv at Wr), s.t. W/+11 = A 0)), t = 0, 1,.1,.... ,T 1
(8)
Now the important question is how to efficiently meta-learn the parameter 0 for the MLR-SNet. We employ the online approximation technique in (Shu et al.] 2019) to jointly update 9 and model parameter w explore proper LR schedule with better generalization for DNNs training. However, the step-wise optimization for is still expensive to handle large-scale datasets and DNN. Furthermore, we attempt to update 0 after updating several steps (Tval) summarized in Algorithm [
Updating 0. When it does not satisfy the updating conditions, is fixed; otherwise, is updated using the model parameter wA and MLR-SNet parameter O, obtained in the last step by minimizing the
validation loss defined in Eq. (8. Adam can be employed to optimize the validation loss, i.e.,
O2+1 02 Adamm(VoLy Dm, t 1 (0)); ne);
where Adam denotes the Adam algorithm, whose input is the gradient of validation loss with respect to MLR-MMNNN -SRt parameter 0 on m mini-batch samples Dm from Dv at- ?e denotes the LR of Adam.
i 1 (0) is formulated on a mini-batch training samples D,, from DT, as follows:
w (0) = wi A(Cr,(D,, wr); 0) VwLrr w)
(10)
Updating w. Then, the updated 0.+1 is employed to ameliorate the model parameter w, i...
W(+1= = wi- A(Cr (Dn, wt): 02+1) Wi w)
(11)
The whole meta-train learning al- gorithm can be summarized in Al-
Algorithm 1 The Meta-Train Algorithm of MLR-SNet
gorithm [ All computations of gradients can be efficiently im- plemented by automatic differ- entiation librarics, like PyTorch (Paszke et al.] 2019), and gener- alized to any DNN architectures. It can be seen that the MLR-SNG can be gradually optimized during the learning process and adjust the IR dynamically hased on the train- ing dynamics of DNNs.
Input: Training data Dr.. validation set Dv batch size n. m. max iterations T', updating period T Trat.
Output: Model parameter wr and MI.K MLRRSS M-SR-N parameter Or
1: Initialize model parameter wp and MI.R-NNet parameter fo.
2: for = 0
do Dn. <- SampleMiniBathh(Orry
3:
4:
if L % Ti.t O. then
5:
D., <- SSamleMMimBBacch (vauch(DDaiina m)..
6:
Update 6.+1 by Eq.
7:
end if
8: Update w241 by Eq.
9:
end for
Meta-Test. Test: transferring to heterogeneous tasks. When we obtain the meta-learned MLR-SNet can be easily applied to new tasks. Now the new model parameter u for the new task is updated by,
u1-1 = Ut- A(Crr u.) 0*) VaLrr w)
(12)
where 6* is the parameter of the meta-learned MLR-SNet, which is fixed in the meta-test stage.
4 EXPERIMENTAL RESULTS
To evaluate the proposed MLR-SNet, we firstly conduct experiments to show our method is capable of finding proper LR schedules compared with baseline methods. Then transfer the learned LR schedules to various tasks to show its superiority in generalization. Finally, we show our method behaves robust and stable when training data contain different data corruptions.
As we know that the performance of designedd LR schedules and HPO methods is very sensitive to the initial LR. To avoid carefully tuning the initial LR, we learn the I.R schedules from interval 7), and now the initial L.R determined by the output of the MI SNet. We set f = for image tasks, and 7 = 40 for text tasks in all our experiments to climinate the influence of loss magnitude between two different tasks.
Notice that w (0) here function of to guarantce the gradient Eq. o be able compute.
5

---

Under review as a conference paper at ICLR 2021
RodNet19 CIFAR, 10
Wid:ResNet 28 CIAAR 100
LSTM Tresbanic
Sopa Statogy
R710
Accurcyy %.
HD
Adderii Stattey
Oua(Tran)
NO
Tooch
Epoch
Epoch
(a) CIFAR-IO with SGD
(b) CIFAR- 100 with SGD
(c) Penn Treebank with SGD
Resher 18 CF
W guResse: 28 10 OFAR 100
3. ayay LSTH on Ponn Treebonk
1 RaLURuNncc
MICH-SSEK
KINO
5500 SESE-EN
Epach
Epoch
Spach
(d) CIFAR-10 with SGDM
(e) CIFAR- 100 with SGDM
(1) Penn Treebank with SGD
Figure+ 4: Test accuracy of our methods (train) and compared baselines on different datasets.
Roshet-18 on CIFAN 100 with 100 Epoch
Reshett18 on CIFAR 100 with 400 Epoch
BesNe: 18 on CFAR- 100 th 1200 Epoch
Tos A2E
T ALC
Tos ALC
SCOMESEER
503MeeFladd
50084
GEO and
SODN-SODA
Adam
Aram
Sco Sheo
Admm
SCDMI Exporiat
SCOMi Tust
SCOM Deparent of
OussiTcal
Epech
Epoch
Epoch
(a) Epoch 100
(b) Epoch 400
(c) Epoch 1200
Figure 5: Test accuracy on CIFAR- of ResNet-18 18 with varying epochs.
4.1 META-TRAIN: EVALUATION ON THE LR SCHEDULE LEARNED BY MLR-SNET
Datasets and models. To verify general effectiveness of our method, we respectively train different models on four benchmark data, including R (He et al., 2016) on CIFAR-10, WedeResNet- 28-10 Zagoruyko & Komodakis 2016) CIFAR- 100 (Krizhevsky 2009), 2-layer LSTM and 3-layer LSTM on Penn Treebank (Marcus & Marcinkiewicz).
Baselines. For image classification tasks, the compared methods include SGD with hand-designed LR schedules: 1) Fixed LR, 2) Exponential decay, 3) MultiStep decay, 4) SGD with restarts (SGDR) (Lohhhhllvv & Hutter (0077) Also, we compare with SGD with Momentum (SGDM) with above four LR schedules. The momentum is fixed as 0.9. Mcanwhile, we compare with adaptive gradient method: 5)Adam, L.R search method: 6) L44 (Rolinek & Martius, 2018), and current LR schedule adaptation methods: 7) hyper-gradient descent (HD) (Baydin et al. 2018). 8) real-time syper-parameter optimization (RTHO) (Franceschi et al., 2017). For text classification tasks, we compare with 1) SGD and 2) Adam with LR tuned using a validation set. They drop the LR by factor of when the validation loss stops decreasing. Also, we compared with 3) L4, 4) HD, 5) RTHO. We run all experiments with 3 different seeds reporting accuracy. The detailed illustrations of experimental setting, and more experimental results are presented in Appendix B.
Image tasks. Fig ((a) and 4(b) show the classification accuracy CIFAR-10 and CIFAR-100 test sets, respectively. can be observed that: 1) our algorithm outperforms all other competing methods, and the learned LR schedules by MLR-SNet
presented in Fig [(d] which have similar shape as the hand-designed policies, while with more elaborate variation details in locality for adapting training dynamics. 2) The Fixed LR has similar performance to other baselines at the early training, while falls into fluctuations at the later training. This implies that the Fixed LR can not finely adapt to such DNN training dynamics. 3) The MultiStep LR drops the LR at some epochs, and such elegant strategy overcomes the issue of Fixed LR and obtains higher and stabler performance at the later training. 3) The Exponential LR improves test performance faster at the early training than other baselines, while makes slow progress due to smaller LR at the later training. 4) SGDR uses the cyclic LR, which needs more epochs to obtain a stable result. 5) Though Adam has an adaptive coordinate-specific LR, it behaves worse than MultiStep and Exponential LR as demonstrated in Wilson et al. (2017). An extra tuning is necessary for better performance. 6) L4 greedily searches LR locally to decrease loss, while the complex DNN training dynamics not guarantee it to obtain a good minima. 7) HD and RTHO are able to achieve similar performance to hand- designdd LR schedules. The LR schedules learned by I.4, HD and RTHO can be found in supplementary material.

---

Under review as a conference paper at ICLR 2021
Rechet SVeN
Reshett Tinylmagehet
LSTM Porn Tresbonk
50D+ Strenazy Adaeeta Statteyy
40 1
Oua/Test
Tedk AC
Test ALC
50230 Frece
OusiTess
Tpacn
Eooch
(a) SVHN
(b) TinyImageNet
(c) Penn Treebank
Figure 6: Test accuracy of transferred LR schedules on different datasets.
GrAB-10
Te
Acc
TesAk SCON+
5:D Fland
SGCM+5GER
SCOM=FTane
SCDN- and Scdteniul
GSDN-G2DA
Alam
Acom
Tpach
Epsch
(a) ShuffleNeV2
(b) MobileNetV2
(c) NASNet
Figure 7: Test accuracy CIFAR-I0 of different network architectures
Since image tasks often use SGDM to train DNNs, Fig 4(d) and 4(e) show the results of baseline methods trained with SGDM, and they obtain a remarkable improvement than SGD. Though not using extra historical gradient information to help optimization, our method achieves comparable results with baselines by finding a proper LR schedule for SGD.
Text tasks. 4() and 400) show the test perplexity on the Penn Treebank with 2-layer and 3-layer LSTM, respectively, Adam and SGD heuristically drops LR when the validation loss stops decreasing. However, our MLR-SNet predicts LR according to training dynamics by minimizing the validation loss, which is a more intelligent way to employ the validation dataset. Thus our method achieves comparable or even better performance than Adam and SGD. The leamnd LR schedules of the MLR-SNet are presented in Fig ((b) which have similar shape as the hand-designed policies. L.4 often falls into bad minima since greedily searches L.R locally. HD and RTHO directly optimize LR to improve the performance on validation dataset, obtaining the similar results as Adam and SGD. With an explicit strcuture, our method behaves more robust and efficient than HD and RTHO.
Remark. Actually, the performance of the hand-design LR schedules can be regarded as the best/ upper performance bound. Since these strategies have been tested work well for the specific tasks, and they are written into the standard deep learning library. For different image and text tasks, our MLR-SNet can achieve the similar or even little better performance compared with the best baselines, demostrating the effectiveness and generality of our method.
4.2 META-TESS TRANSFERABILITY OF
LR SCHEDULES
The learned MLR-SNet is transferable and plug-and-play. Here we validate if MLR-SNet can transfer to new heterogeneous tasks. Since the methods HD,,RTTH in Section 4. are not able to generalize, we do not compare them here. Actually our results show superiority on image tasks beyond baseline methods when trained with SGD, here we present stronger baseline results in which compared methods are trained with SGDM. We use the MLR-SNet meta-learned on CIFAR-10 with ResNet- 18 in Section 4.1 as the plug-and-pla LR schedules for the following experiments.
Transfer to different epochs. The plug-and-play MLR-SNet is meta-trained with epoch 200, and we transfer it to other different training epochs, e.g., 100, 400, 1200. As shown in Fig our MLR-SNet has the ability to train for longer horizons and achieves almost same performance as MultiStep LR. The slight shakes for epoch 1200 may due to that our MLR-SNet can learn the LR similar to SGDR locally. The Exponential LR has a little performance decreased for the longer epochs.
Transfer to different datasets. We transfer the LR schedules meta-learned on CIFAR- to SVHN (Ntterr et al. 2011), TinyImageNer [ and Penn Treebank (Marcus & Marcinkiewicz). As shown in Fig 6 though datasets vary from image text, our method can still obtain a relatively stable and comparable generalization performance for different tasks with baseline method.
Transfer to different net architectures. We also transfer the LRR schedules meta-learned on ResNet- 18 to hight-weight nets ShuffleNetV2 aM. 2018). MobileNerV2 (Sandler et al.) (018) or NASNet
It can be downloaded at http://tnnyiimgeent.t.ooouappmpp
7

---

Under review as a conference paper at ICLR 2021
Imageke!
Rendel-18 on CIFR-10
BesNectn on OFAR-10
SipMefiee
SODMAN,
S2DM4 Experienca
SODM SEEA
OunsTran
Adam
OusiTeso
SCOM
SCON+SGER
Acom
LSIM Mat Roamn MLP MoD carner
OusiTent
Esech
Epoch
Epech
Figure 8:
Test accuracy on Ima-
Figure 9: Comparion of differ- ent meta-learners.
Figure 10: Time consuming of
geNet with ResNet-50.
different LR schedules methods.
Table 1: Test accuracy (%) on CIFAR-10 10 and CIFAR-100 100 training set of different methods trained on CIFAR-10-C and CIFAR- 100-C. Best and Last denote the results of the best and the last epoch.
MethootsMethods
Fixed
Muliitop
Exponential
SGDR
Adam
Ours(Train)
CIFAR-10-C
Best
79.78=3.95
85.5241.72
83.48=1.45
85,94:1.52
81.45= 1.42
86.04
Last
77.88=3.91
15.36++11.11
83.32=1.43
78.2142.001
80.29=11664
15,87+1.54
Best
46.74 3.004=3.03
52.26:22.58
49.72=1.99 1.97
52,5442.49
45.45:1.94
22.56112.26
CIFAR-100-C
34.79.3.91
52.16.12.59
49.5811.9
41.58.1..24
43.76.12.22
52.42.1..34
Zoph et al. 2018
As shown in Fig 7 our method achieves almost similar results SGDM with MultiStep or Exponential L.R.
Transfer to large scale optimization problem. To our best knowledge, only Wichrowska al. (2017) attempted to use the learned optimizers to train DNN on ImageNet dataset (Deng et al. 20099) among existing gearning-to-optinii
literatures. However, can only be executed for thousands of steps, and then its loss begins to increase dramatically, far from the optimization process in practice. We transfer the LR schedule meta- retanerained on CIFAR- 10 with ResNet- 18 to ImageNet dataset with ResNet-50
[ As shown in Fig the validation accuracy of our method is competitive with those hand-designed LR schedules methods. This implies our method is capable of dealing with such large scale optimization problem, making learning-to-opimnzz to-optimzze ideas towards more practical applications.
4.3 ROBUSTNESS ON DIFFERENT DATA CORRUPTIONS
In this section, we validate that whether our MLR-SNet behaves robust against corrupted training data. To this aim, we design experiments as follows: we take CIFAR-IO-C and CIFAR- 100-C (Hendrycks) & Dietterich 2019) as training set, consisting of 15 types of different generated corruptions test images data of CIFAR- CAAAR-O/ 100, and the original training set of CIFAR-10/100 as test set. Though the original images of CIFAR- 1010000011----0 are the same with the CIFAR- 10/100 test set, different corruptions have changed the data distributions. To guarantee the calculated models finely generalize test set, we choose the validation set 10 clean images for each class. Each corruption can be roughly regarded as task, and thus we obtain 15 models trained on CIFAR-10/100-C. Table showws the mean test accuracy of 15 models (#sttd). As can be seen, our proposed MLR-SNet is capable of achieving better generalization performance on clean test data than baseline methods, which implies that our method behaves more robust and stable than the pre-set LR schedules when the learning tasks are changed. This is due to that our MLR-SNet has more flexibility to adapt the variation of the data distribution than the pre-set LR schedules, and can find a proper LR schedule through minimizing the generalization error which is based on the knowledge specifically conveyed from the given validation data. The detailed illustrations of experimental setting, and the transferrablity experiment of meta-learned MLR-SNG are presented in supplementary material.
SOME ANALYSIS OF MLR-SNET CONVERGENCE ANALYSIS OF MLR-SNET
5.1
The preliminary experimental evaluations show that method gives good convergence performance on various tasks. We find that the meta-learned LR schedules in our experiments follow a consistent trajectory shown in Fig [ sharing similar tendency as the Exponential LR schedules. To provide a theoretical convergence analysis, we roughly assume that the LR predicted by MI M-R-SNet obey a Exponential LR form. The convergence analysis for DNN training can refer to (Li et al. 2020). Here, we provide a convergence amalysis of the MLR-SNet training. The proof is listed in the Appendix A.
Theorem Suppose the loss function is Lipschitz smooth with respect to the model parameter with constant L, and have p-bounded gradients with respect to trainin//vaiidatinn data. And the A(0) is differential with a 8-bounded gradient and twice differential with its Hessian bounded
*The pytorch code of all these networks can be found on 'The training code can be found on http:///illlfflyyccccccmeeeeeeee
8

---

Under review as a conference paper at ICLR 2021
by B. Let the learning rate ag A((O,) predicted by MLR-SNet obey the exponential LR, i.e., ar 0x,3", , (T/T))/T,T 2 1. Let Ttt for all t E [T). If we use Adam algorithm update MLR-SNet, we choose yn satisfied v S ZI and 1 82 S 16p where B2, e are the hyperparameter of the Adam algorithm. Then for Ot generated using Adam, we have the following bound:
LVll (0))) S O Cln(T)
min
T +
(13)
OSEST
where C is some constant independent of the convergence process, is the variance of drawing uniformly mini-batch sample at random.
5.2 THE STRUCTURE OF THE MLR-SNET
We regard the LR scheduling a long-term information dependent problem, and thus we parameterize the LR schedule as an LSTM network. As we known, MLP network can also learn an explicit mapping but ignore the temporal information. Here, we compare the performance of the two types of meta- learners. As shown Fig. D in the early training stage, both of them achieve the similar performance. While at the later training stage, the LSTM meta-leamer brings notable performance increase compared with MLP meta-learner. This may due to that the accumulated temporal information the LSTM meta-learner can help find a more proper LR for such DNNs training.
5.3 COMPUTATIONAL COMPLEXITY ANALYSIS
In the meta-training stage, our MLR-SNG learning algorithm can be roughly regarded as requiring two extra full forward and backward passes of the network (step in algorithm 1) in the presence of the normal network parameters update (step 8 in algorithm 1), together with the forward passes of MLR-SNe -Seet for every LR. Therefore compared normal training, our method needs about 3x computation time for one iteration. Since we periodically update MLR-SNe after several iterations, this will not substantially increase the computational complexity compared with normal network training. In the meta-test stage, our transferred I.R schedules predict LR for each iteration by small MLt- whose computational cost should be significantly less than the cost the normal network training. To empirically show the differences between hand-designed LR schedules and our method, we conduct experiments with ResNet-18 on CIFAR-10 and report the running time for all methods. All experiments are implemented on computer with Intel Xeon(R) CPU E5-2686 v4 and a NVIDIA GeForce RTX 2080 8GB GPU. We follow the corresponding settings in Section 4.1, and results are shown in Figure TO Except that RTHO costs significantly more time, other methods including MLR-SNet training and testing have similar time consuming. Our MLR-SNet takes barely longer time to complete the meta- meta-training and meta-testing phase compared to hand annd-designed LR schedules. Therefore our method is completely capable of practical application.
6 CONCLUSION AND DISCUSSION
In this paper, we have proposed to learn an adaptive and transferrable LR schedule in a meta learning manner. To this aim, design an LSTM-type meta-learner (MLR-SNett) parameterize LR schedules, which gives more flexibility to adaptively learn a proper LR schedule to comply with the significantly complex training dynamics of DNN. Meanwhile, the meta-learned LR schedules are plug-and-play and transferrable, which can be transferred how to schedule LR for SGD to new heterogeneous tasks. Comprehensive experiments substantiate the superiority of our method on various image and text benchmarks in its adaptability, transferrability and robustness, as compared with current LR schedules policies. The MLR-SNet is highly practical as it requires negligible increase in the parameter size and computation time, and no transferrable cost for new tasks. We believe our proposed method has a potential to become new tool to study how to design LR schedules to help improve current DNN training, well more practical applications.
Recently, Keskar et al. (0177); :): Dinh et al. (0117) suggested that the width of local optimum is related to generalization. Wider optima leads to better generalization. We use the visualization technique in (Lzmailov et al. 2018 to visualize the "width" of the solutions for different LR schedules on CIFAR- 100 with ResNet- 18. As shown in Fig our method lies a wide flat region of the train loss. This could explain the better generalization of our method compared with pre-set LR schedules. Deeper understandings this point will be further investigated.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, 2016.
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In ICLR, 2018.
Y Bengio, S Bengio, and J Cloutier. Learning synaptic learning rule. In IJCNN, volume 2, pp. 969-vol. IEEE, 1991.
Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade, pp. 437-478. Springer, 2012.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. JMLR, 2012.
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Deep frank-wolfe for neural network optimization. In ICLR, 2019.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando De Freitas. Learning learn without gradient descent by gradient descent. In ICML, 2017.
Damek Davis, Dmitriy Drusvyatskiy, and Vasileios Charisopoulos. Stochastic algorithms with geometric step decay converge linearly on sharp functions. arXiv:1907.09547 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In ICML, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(1u1:2121--119, 2011.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In JCMI., 2017.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In ICMI,, 2017.
Rong Ge. Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: near optimal, geometrically decaying learning rate procedure for least squares. In NeurIPS, 2019.
Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. Sgd: General analysis and improved rates. In ICML, 2019.
Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. NeurlPS, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1442, 1997a.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997b.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In NewrIPS, 2018.
10

---

Under review as a conference paper at ICLR 2021
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning. Springer, 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. In UAI, 2018.
Stanistaw Jastrzebski. Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv: 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR. 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Jason D Lee, loannis Panageas, Georgios Piliouras, Max Simehowitz, Michael Jordan, and Benjamin Recht. First-order methods almost always avoid saddle points. Mathematical Programming, 2019.
Ke Li and Jitendra Malik. Learning optimize neural nets. In ICLR, 2017.
Xiaoyu Li, Zhenxun Zhuang, and Francesco Orabona. Exponential step sizes for non-convex optimization. arXiv preprint arXiv:2002.05211 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09833 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In ICML, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018.
Mitchell P Marcus and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2).
Kamil Nar and Shankar Sastry. Step size matters deep learning. In NeurlPS, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 1359, 2009.
Ioannis Panageas, Georgios Piliouras, and Xiao Wang. First-order methods almost always avoid saddle points: The case of vanishing step-sizes. In NeurlPS, 2019.
Eunbyung Park and Junier B Oliva. Meta-curvature. In NeurlPS, 2019.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Ginelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.

---

Under review as a conference paper at ICLR 2021
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Michal Rolinek and Georg Martius. 14: Practical loss-based stepsize adaptation for deep learning. In NeurIPS, 2018.
Nir Rosenfeld, Eric Balkanski, Amir Globerson, and Yaron Singer. Learning to optimize combinato- rial functions. In ICML, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Laang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Tom Schaul, Sixin Zhang, and Yann LeCun. more pesky learning rates. ICML, 2013.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent
networks. Neural Compulation, 4(1::131-139, 1992.
Jun Shu, Zongben Xu, and Deyu Meng. Small sample learning in big data era. arXiv:1808.0455 2018.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. In NeurlPS, 2019.
Jun Shu, Qian Zhao, Keyu Chen, Zonghen Xu, and Deyu Meng. Learning adaptive loss for robust learning with noisy labels. arXiv:2002.06482 2020a.
Jun Shu, Qian Zhao, Zengben Xu, and Deyu Meng. Meta transition adaptation for robust deep learning with noisy labels. arXiv preprint arXiv:2006.06697, 2020b.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NeurlPS, 2012.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Neural networks for machine learning, 2012.
Paul Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. SIAM Journal on Optimization, 8(2):506-531, 1998.
Sharan Vaswani, Aaron Mishkin, Issam Laradji. Mark Schmidt, Gauthier Gidel, and Simon Lacoste- Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. In NeurIPS, 2019.
Lewis B Ward. Reminiscence and rote learning. Psychological Monographs, 49(4), 1937.
Olga Wichrowska, Niru Maheswaranathan. Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In ICML, 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stemn, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In NewrIPS, 2017.
Lei Wu, Chao Ma. and E Weinan. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In NeurIPS, 2018a
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In ICLR, 2018b.
Zhen Xu, Andrew M Dai, Jonas Kemp, and Luke Metz. Learning an adaptive learning rate schedule. 1xXiv:909.0097 2019.
12

---

Under review as a conference paper at ICLR 2021
Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Transfer learning. Cambridge University Press, 2020.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. In BMVC, 2016.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. NeurlPS, pp- 9793 9803, 2018.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv:/212.5701, 2012.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.
13

---

Under review as a conference paper at ICLR 2021
A CONVERGENCE ANALYSIS OF THE MLR-SNET
Lemma 1 Suppose the loss function & is Lipschitz smooth with respect the model parameter with constant L, and have p-bounded gradients with respect to training/validatinn data. And the A()) is differential with 6 6-bounded gradient and twice differential with its Hessian bounded by B. Then the gradient of MLR-SNer parameter with respect to loss is Lipschitz smooth.
Proof The gradient of MLR-SNet parameter 0 with respect loss
Vol (l)( (0) 0A((O) (1)
8A(O) 00
(1,((()() (l,(w,)
0A(())
(1)
Owl.
00
10,
Let Gij
0t, (.,(()) (F,(we), and then take gradient of 0 in both sides of above equallity, we have (0) Owc
-0t n
Σ
12
aG
0A(O)
+ Gij 0A72
(14)
00
00
002
For the first term in the right hand side, we have that
BG
0A((0)
06, (1,(
(l,()r)
<5
00
00
(
ow
(l,((0((0)
(l,(()))
0.((0)
Ols(we)
=
lo,
Tri (8)
n
i=1
(i)
Ow,
00
Swr
(15)
O ( (0))
als(wr))
0A(00)
(l:(w:)
Oid
Owe
09
0.
Ow.
LaxLp2
882
For the second term in the right hand side, we have that
(0)
Gig
< Bp²
(16)
002
Combining the above two inequalities Eq. 15 we have
(o s ap (a, + B).
(17)
Define LA (or + B), and based on the Lagrange mean value theorem, we have:
VLLa((W((0)) - (vv)( S LA |0: 02ll
(18)
Thus the conclusion holds.
Theorem Suppose the loss function f is Lipschitz smooth with respect the model parameter w with constant L, and have p-bounded gradients with respect to training/validation data. And the A(0) is differential with 6-bounded gradient and twice differential with its Hessian bounded by B. Let the learning rate = A(e,) predicted by MLR-SNe obey the exponential LR, i.e., a₂ = 003" , = //T r 1. Let π) = for all t E [T), If we use Adam algorithm to update ,
MLR-SNet, we choose 1) satisfied n) S z and 1 B2 S 1op where B23 are the hyperparameter of the Adam algorithm. Then for 0, generated using Adam, we have the following bound:
VV w (w.) < O Cln(T)
min OSSST
+02),
(19)
T
where is some constant independent of the convergence process, a the variance of drawing uniformly mini-bated sample random.
14

---

Under review as a conference paper at ICLR 2021
Proof Suppose we have small validation set with M samples {x1, T2. IM ). each associating with a validation loss function (,(w(f)), where w the parameter of the model, and 6 is the parameter of the MLR-SNet. The overall validation loss would be,
1 Lva((w) = M ((o(0)).
(20)
According the updating algorithm we have:
(au(@+++(00+1)) LV (w:
(0e - Cv (2)((4)))) + ( - (ora()
(21)
(a)
(6)
For term (a),
( al
L.
(22)
VVV (0c+1)), wh+1 (0c+1) (i((0++))) (0(42)-
According Eq 6 we have
@141(0141) (0+(1++) -04 V (1,
(23)
where ae = (1 0c), (wr) = [Bi]
=ioi ΣiEB, VLT This can be written as
(1+1(00+1) - 10 (00+1)) = -04 (W + +(0)
(24)
where ((t) = V ( (u, (0++1)) Since Be is the mini-batch samples drawn
uniformly from the entire data set, we have [[[(()] Furthermore, g(t) are i.id random variable with finite variance, since B, are drawn i.i.d with finie number of samples. Then Eq (22) can be written as
(V -0 V..TT + 5500)
+ L
2/4
- V=LCT((00((00+) + +
(vr 0 (w. 5(1))
Lai
2 (1 + || (V Lx (1+x()++)))
< (v -at Vwn (a (10
+ {0)) + 2 at p2² 11 (V
For term (b), according to Lemma 1, i.e., the validation loss is Lipschitz smooth with respect to the MLR-SNet parameter for briefly denote
Lvai(1 (0.((0+))) Lvaa((0x(0)))
(25)
S <(VoLva (0.)) 0.+1 - 0z) + 2 |10+11 - -21l
If we adopt Adam to update the parameter of MLR-SNet, 0:-1 O2 in Eq. 25 is updated by
gt.i
0.+1 = 02 7)
(26)
Wt.5
where gt. =
Now; we have
(v: (val((((0.))
< n/6
V ((
St.i
+
Laf
22
(27)
Vori +e
2
(Ver, e)2
15

---

Under review as a conference paper at ICLR 2021
Based the proof process in (Zaheer al. 2018
(Eq 4 in p. 13),
- V ( (0x))
VoLv (tx: 1hPvI-33
32
Lij
o²
(28)
S 2(vB2p +
+
62
26
M
Now Eq. 21 has become the following:
( (VuCva((o((0++)))) -a( V (10 + +<(00)
10:0(10+1111111 - (ova(((())) -
L 2 + ai
[MPVT (
32
Li²² o²
(V 2(vBap +
(0)
22
26
M
(29)
Taking expectations with respect to & on both side of Eq. 29 and rearranging the inequality, we can obtain:
T/t
Eg
12p +
[
Laf² o²
Sarp² 2 + (p² + +²²)
+ (v)((()(0)))+ +
+
6
2ca
M
Using telscoping sum, we obtain
T
f
E
2( 1=1
I
[T:PV/
B2
Tnp²
Q²T
<Cva((()(())) -
+ p²
Σ
ai +
(p²
+
t02)
af +
2
+
2
2e2
M
1=1
1=1
T
11PV1
32
Laf
2T
<Lv))((,((1)) + 0²
Σ
at + (p= + 2
a +
22
2e
M
(30)
Therefore,
T1
Eg
(10
min Eg [IVoCra((0,(0)))]
Lt=1 2(vPapte)
<
Σt
7/1
1 2(VBapte)
(val(( + p² P² a: + (p² + 0z2)
ΣT o
Lup²
2²T M
x2
20²
2(
B2p +
Σ
2( 2(vB2p+ +
T
L
T
1PPV 1- B2 [IPPVI-
o²T
(0))
p²
aq +
(p²
+ 0²)
2 + Σ
T'
2
2²
2
M
1=1
1=1
<0( In(
+ + T
16

---

Under review as a conference paper at ICLR 2021
class LSTMCe11 :odule))
def init _____ini________________ (self, num inputs, [idden_size) super (1S1MCel1, self self.hidden_size hidden_size
.inft__ii_i___
self.fc_i2h nn. Sn.squennial
nn. (num (m_iineartuu___nuutt hidden hidden_size), nn. nn.Retū()
nn. Ln.einerr thiiden_ssieee 4 hidden_size)
self.fc_h2h nn. Sequential (
nn In.tinear (hidden_size, hiddcn_size), nn.Relū() (), In.linear (hidden_size, 4 hidden_size)
def forward inputs, state):
hx, ex = state
i2h self.fc_i2h (inputs)
h2h self. (hx)
x i2h h2h
gates = x. (sef. hiize 1)
in_gate = torch.sigmoiee sigmoid (gates(00))
forget_gate torch. sigmodd (gates (1])
out_gate torch. (incchssigmi((gttss122))
in_transform torch, farch (gates
сx = forget_gate ox + in_gate in_transform
hx out_gate torsh.tanh return hx,
class MLRNet Module))
def init_. (self, layers, hidden hidden_size) super (MLRNet, self) inii .iii____________ fff.hidden_siza hidden_size
self self.layerl = LSTMCell (1, hidden_size) self. nn. Linear (hidden_size, 1)
def forward (self,, x, gamma)
self.hx, self.cx self.layeri (x, (self.hx, self.cx))
self.hx
self.
out torch. sigmoid
return gamma out
B EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS IN SECTION 4.
In this section, we attempt evaluate the capability MLR-SNet to learn LR schedules compared with baseline methods. Here, we provide implementation details of all experiments.
Datasets. We choose two datasets in image classification (CIFAR-10 and CIFAR-100), and one dataset in text classification (Penn Treebank) to present the efficiency of our method. CIFAR-10 and CIFAR- 1EARR-10/e5 Krizhevsky (2009), consisting of 32x32 color images arranged 10 and 100 classes. respectively. Both datasets contain 50,000 training and 10,000 test images. Penn Treebank Marcus & (reebank|Marcuss
Marcinkiewicz)s is composed of 929k training words, 73k validation words, and 82k test words, with 10k vocabulary in total. Our algorithm and RTHO Franceschi al. (2017) randomly select 1,000 clean images in the training set of CIFAR- 10/100 as validation data, and directly use the validation set in Penn Treebank validation data.
CIFAR-10 & CIFAR-100. We employ ResNet-18 on CIFAR-10 and WideResNNet-2-- Zagoruyko & Komodakis 2016) on CIFAR- 100. All compared methods and MLR-SNet trained for 200 epochs with batch size 128. For baselines involving SGD as base optimizer, we set the initial LR to 0.1, weight decay parameter 5e -1 and momentum to 0.9 if used. While for Adam, just follow
17

---

Under review as a conference paper at ICLR 2021
Table 2: Test accuracy (%) of CIFAR dataset with SGD baselines.
Optimizer
CIFAR-10 10 with ResNet18 CIFAR-100 100 with WRN-28-10 10
SGD++ixed
92.26 + 0.12
70.67 + 0.34
SGD+MulüStep
93.82 + 0.09
77.04 + 0.17
SGD+Exponential
90.93 0.11
72.52 0.34
SGD+SGDR
93.92 = 0.11
72.32 + 0.34
Adam
90.86 + 0.15
68,94 + 0.24
SGD+L4
89.15 = 0.14
63.61 + 0.65
SGD+HD
92.34 + 0.09
72.22 + 0.30
SGD+RTHO
92.60 0.18
72.32 0.47
MLR-SNet (Meta-train)
94.70 + 0.16
79.41
Table 3: Test accuracy (%) of CIFAR dataset with SGDM baselines.
Optimizer
CIFAR-10 10 with ResNet18
CIFAR-100 with WRN-28-10
SGDM+Fixed
87.69 + 0.14
70.88 + 0.12
SGDM+MultiStep
95.08 0.13
80.74 + 0.19
SGDMM+EXponential
94.64 + 0.05
78.87 + 0.04
SGDM+SGDR
95.06 + 0.17
80.93 + 0.05
Adam
90.86 0.15
68.94 + 0.24
SGDM+L4
91.03 + 0.14
66.31 + 2.83
SGDM+HD
93.99 0.12
76.80 1 t 0.19
SGDM+RTHO
93.17 1 0.49
76. 14.14 + 0.29
MLR-SNet (Meta-train)
94.70 0.16
79.41=:0114
the default parameter setting. The hyper- paremmerrseeerrs of hand-designed LR schedules are listed below: Exponential decay, multiplying LR with 0.95 every epoch; MultiStep decay, decaying LR by 10 every 60 epochs; SGDR, setting T r_O 10, T T_Mult to 2 and minimum LR le-5 1.4, HD and RTHO update LR every data batch, and we use the recommended setting in the original paper of LA (a 0.15) and search different hyper-Irs from {le le Le Ie Pe for HD and RTHO, reporting the best performing hyper-Ir.
Penn Treebank. We use 2-layer and 3-layer LSTM network which follows word-embedding layer and the output is fed into a linear layer to compute the probability of each word in the vocabulary. Hidden size of LSTM cell is set to 512 and so is the word-embedding layer and the final linear layer. Dropout is applied to the output of word-embedding layer together with both the first and second LSTM layers with a rate of 0.5. As for training, the LSTM net is trained for 155 epochs with batch size of 32 and sequence length of 35. We set the base optimizer SGD to have an initial LR of 20 without momentum, for Adam, the initial LR is set to 0.01 and weight for moving average of gradient is set to O. We apply weight decay of 5e to both base optimizers. All experiments involve a 0.25 clipping to the network gradient norm. For both SGD and Adam, we decrease LR by factor of 4 when performance progress. For L4, we try different or in 10.1, 0.05, 0.01, 0.005) and reporting the best test perplexity among them. For both HD and RTHO, search the hyper-Ir lying in (1, 0.5, 0.1, 0.05), and report the best results.
size. We tie weights of the
validation set shows no
Table 4: Test perplexity on the Penn Treebank dataset.
Optimizer
2-layer LSTM
3-layer LSTM
SGD+ Val Strategy
74.33 0.23
76.05 + 0.39
Adam+ Val Strategy
71.17 0.23
74.80 + 0.73
SGD+L4
82.58 + 1.32
92.27 + 0.92
SGD+HD
76.90 0.33
78.63 = 0.08
SGD+RTHO
76.69 = 0.11
78.52 + 0.16
MLR-SNer (Meta-train
72.09 0.72
72.71
18

---

Under review as a conference paper at ICLR 2021
ResNett1 on CIFAR-LO
ResNer18 on CFAR-10
Resheil8 CIAA- 10
110
Tes: Accurscy %.
Train 1055
R
502 E3se
Enoch
Epoch
Epoch
(a) Train loss on CIFAR-10 with SGD
(b) Test accuracy on CIFAR-IO with SGD
(c) LR schedules learned by basclines
HeaNet--22
SPAR-10
ResNer-18 on CIPA-100
Rashei-18 CRAN-10
auctellbad
292M-5601
RT10
40
KIHG
Tos: Accuory
Triin
Esoch
Epach
Epoch
(d) Train on CIFAR-IO with SGDM
(e)
Test accuracy on CIFAR- with SGDM
(I)
Learned schedules
2isietleddet-22-11000 on CFAR-100
Wideetee -ww. CPAR-100
W Wuneltessee22-112 20-10 on CPP48-100
HD
Tox Accursyy %
KTHG
OuasTrun)
Thai Loss
Eooch
Epach
Epach
(g)
Train loss on CIFAR 100 with SGD
(h) Test accuracy on CIFAR 100 with SGD
(i) Learned LR schedulees
WideRcsheet 28.10 on CFAR 100
WidshesNet 28 10 on CEAR 100
W WideReshet 28 10 on CEAA, 100
T:st Accurocy %
Tran LOBS
KI-0
502M Coneeena
Epoch
Epoch
Epoch
(i) Train loss on CIFAR-100 with SGDM (k) Test accuracy on CIFAR- with SGDM
Leamed LR schedules
2-layer
24ayer
sob-lal Streeayy
ST1O
503-1ol Stastegy
Rm10
Trait Porplaity
Adamm Val Salstoe
Acomi Val Struaayy
HD
Qusll raint
HD
ousiiiaa)
SGD /Val 5D Xi4ap
Hi +1
Adart (Va) 50 stgt
Thri/rairr)
Epoch
Epoch
Epoch
(m) 'Train perplexity on Penn Treebank
(n) Test perplexity Penn Treebank
(0) Learned LR schedules
3-layer 15 TM Pern Treetenic
3-ayer LSTM on Pern Treebark
34ayer LSTM en Penn Treebatic
Shanngy
Straegy
Ton Peploiiy
Stuuoss
siggy
Strllgy
HD
upp
Esoch
Epoch
Epsch
(p) Train perplexity on Penn Treebank
(q) Test perplexity on Penn Treebank
Learned LR schedules
Figure 11: Train loss (perplexity), test accuracy (perplexity) and learned LR schedules of our methods (train) and compared baselines different tasks.
MLR-SNet architecture and parameter setting. The architecture of MLR-SNet is illustrated in Section 3.2. In our experiment, the size of hidden nodes is set as 40. The initialization of MLR-SNar follows the default setting in Pytorch. The Pytorch implementation of MLR-SNet is listed above.
19

---

Under review as a conference paper at ICLR 2021
We employ Adam optimizer to train MLR-SNel and just set the parameters originally recom- mended with a LR of Le and a weight decay of le which avoids extra hyper- parameter tuning. For image classification tasks, the imput of MLR-SNet is the training loss of mini batch samples. Every data batch's LR is predicted by MI R-SR-Net and we update it twice per epoch according to the loss of the validation data. While for text classification tasks, we take Tng(vacabularyy sizej as imput of MLR-SNet to deal with the influence of large scale classes of text. MLR-SNet is updated every 100 batches due the large number of batches per epoch compared to that in image datasets.
hyper-parameter
Results. Due to the space limitation, we only present the test accuracy in the main paper. Here, we present the training loss and test accuracy of our method and all compared methods on image and text tasks, as shown in Fig I1 For image tasks, except for Adam and SGD with fixed LR, other methods can decrease the loss to 0 almostly. Though local minima can be reached by these methods, the generalization ability of the these mimimas has a huge difference, which can be summarized from test accuracy curves. As shown in Fig. [T(a] (1) [ [ when using SGD to train DNNs, the compared methods SGD with Exponential LR, 14, HD, RTHO fail to find such good solutions to generalize well. Especially, L4 greedily searches LR to decrease loss to O, making it fairly hard to adapt the complex DNNs training dynamic and obtain good mimima, while our method can adjust LR to comply with the significant variations of training dynamic, leading to a better generalization solution. As shown in Fig. [Td] I1 [1( when baseline methods are trained with SGDM, these methods make a great progress in escaping from the bad minimas. In spite of this, our method still shows superiority in finding a solution with better generalization compared with these competitive training strategies.
In the third column in Fig. 11 we plot learned L.R schedules of compared methods and method. As can be seen, our method can learn LR schedules approximating the hand-designed LR schedules while with more locally varying. HD and RTHO often have the same trajectory while producing lower or faster downward trend than ours. This tends to explain our final performances on test set better than HD and RTHO, since our method can adaptively adjust LR utilizing the past training histories explicitly. L4 greedily searches a LR to decrease the loss. This often leads to a large value causing fluctuations or even divergence (Fig. IIO),, or a small value causing slow progress (Fig. I both of them (Fig. II(e) If( Ii()) IIo) Such LR schedules often result in bad mimimas. Moreover, all compared methods regard LR as hyper-parameter to learn without a transferable formulation, and the learned LR schedules can not generalize to other learning tasks directly. Generally, they just try to find a proper LR schedule from scratch for new tasks. However, our meta-learned MLR-SNet is plug-and-play and transferrable, which can directly transfer how to schedule LR for SGD to heterogeneous tasks without additional learning.
Ablation study.
(1) The architecture of MLR-SNet. Fig 12(a) shows the test accuracy on CIFAR-10 with ResNet-18 of different architectures of MLR-SNet. As can be seen, our algorithm is not sensitive to the choose of the MLR-SNet's ' architectures. This implies that our algorithm is robust and stable for helping improve DNN training.
(2) The gobal LR of the meta optimizer. To further validate that whether our MLR-SNet behaves robust to the meta optimizer. We adapot Adam optimizer to search the proper LR schedules. Fig. achieves the similar performance even for different global LRs. This implies our MI R RNet needs not carefully tune the LR of the meta optimizer, which makes it easy to reproduce and apply to various problems.
ssobss shows that our MLR-SO
(3) The different y values of the MLR-SNet. One important hyperparameter of the MLR-SNet is 7, here we verify our method is not sensitive to the choose of value. We test y values from 0.1 to 10 for the DNNs training. As shown Fig 12(c) even with different learning scales, our method can still help DNNs achieve almost similar performance. This implies the MLR-SNet is robust to the choose of the 7, which makes it easy to be applied into parctice.
C EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS IN SECTION 4.2
We investigate the transferability of the learned LR schedule when applied to various tasks in Section
4.2 of the main paper. We use the -NR- meta-learned on CIFAR- 10 with ResNet-18 in Section 4.1 to directly predict the LR for SGD algorithm to new heterogeneous tasks. We save the learned MLR- M-R- at different epochs in the whole one meta-train run. As is shown in Fig 13(a) if we use
20

---

Under review as a conference paper at ICLR 2021
Ne on GFAR-10
Reckeb-il on CPAR-11
Res on CIAA-11
Tex: ACC
1.20
1-120
LR-
2.03
250
3.43
U1-764
LR-203
1011
Spach
Epoch
Spoch
(a) Different Architectures of MLR-(b) Different LRs Meta Optimizer (c) Different 7 value of MLR-SNet SNet
Figure 12: Ablation study. (a) Test accuracy on CIFAR-10 with ResNet-18 18 of different architectures of MLR-SNet. 'a-b' denotes the configurations of MLR-SNet where 'a' represents the number of layers, and 'b' represents the number of hidden nodes. (b)Test accuracy on CIFAR- R-10 with ResNet- 18 of different LRs of meta optimizer 'Adamm. (c) Test accuracy on CIFAR-10 with ResNet-18 of different gamma values of MLR-SNet.
18 Variation Curoe wingg beeations
Restiet-1 on OFAB-1 with Biffezent Strategtis
RedMel--1 18 on CIFA-110 with Different Grahegges
Eoech
Net
Epoch 40 Epoch
Esech 150
Two Nets Nets
Enech 200
Three Ners
Netx Each Eooch
TOc Ac2
5
Three Nats
Nats Each Epoch
Baraaions
Epoch
Epoch
(a)
(h)
(c)
Figure 13: (a) We plot the LR variation curves along iterations with the same input for learned MLR-SNet at different epochs. As is shown, when iteration increases, the LR is almost constant. This means the leamned MLR-SNet overfits the short trajectories, while fails for the long trajectories. (c).() show the recording train loss and test accuracy with R8 on CIFAR-100 of different test strategies.
the single learned MLR-SNe at certain epoch, it can be seen that the predicted LR by the learned LR schedules converges after several iterations. This is because that the training trajectories are long in our experiments, and the learned MLR-SNet can not memory all the information since we locally adjust our MLR-SNet according to the validation error. If we directly select one MLR-SNet learned at any epoch, that will raise overfitting issues as shown in Fig 13(a) Thus we should select more than two learned MLR-SNets for test. Here, we propose a heuristic strategy to select MLR-SNets for test. Generally, if we want to select k nets for test, the MLR-SNet learned at 220-m 1-hh epoch (1 = 0, 2, .......... k- 1) should be chosen, where [-] denotes ceiling operator. Fig 13(b) and 13(c) show the train loss and test accuracy with ResNet-18 on CIFAR- 100 of diffcrent test strategies, I.c., choosing different number of nets to transfer. It can be seen that almost choosing more than three nets have similar performance. Therefore, in the following experiments we choose three MIRR-SNets to show the transferability.
Transfer to different epochs. We transfer the LR schedules meta-trained with epoch 200 to other different epochs, e.g., 100, 400, 2200. All the methods trained with Re on CIFAR-100 with batch size 128 for different epochs. The hyper-pann parameter setting for compared hand hand-designed LR schedules is the same with Section 4.1 in the main paper as illustrated above, except for MultiStep LR. For epoch 100, MultiStep LR decays LR by 10 every 30 epochs; For epoch 400, MultiStep LR decays LR by 10 every 120 epochs; For epoch 1200, MultiStep LR decays LR by 10 every 360
Table 5: Test accuracy (%) of CIFAR- 10 dataset with different networks.
Optimizer
MobileNetV2 ShuffleNetV2
NASNet
SGD+Fixed
87.06 + 0.33
90.85 0.52
89.14 1.15
SGD+MultiStep
88.99 + 0.11
92.28 + 0.28
94.97 + 0.10
SGD+tixponetial
89.45 + 0.14
93.47 + 0.12
94.59 + 0.14
SGD+SGDR
89.30 + 0.83
92.00 + 1.21
94.75 1 0.82
Adam
87.95 + 0.31
90.64 + 0.54
90.60 + 0.47
MLR-SNet (Meta-test)
89.09 + 0.33
93.11 = 0.10
95.18
21

---

Under review as a conference paper at ICLR 2021
Table 6: Validation accuracies ImageNet dataset.
Optimizer
Top-1 Accuracy
Top-5 Accuracy
SGDM+Fixed
68.23
88.47
SGDM-MultiStep
75.90
92.90
SGIM++Xpooeen
75.68
92.69
SGDM+SGDR
75.82
92.87
Adam
63.62
85.43
MLR-SNet (Meta-test)
75.03
92.39
Dersette::45 (b=12) CHAB--D
Den (6-12) CHAI-DD
Densefeel-100 (c-24) on CIAAN-10
140
SaD+faed
SCCM45GER
SCD+Fhis
SCON+55ER
5GD-fleed
SCDN-52DA
SCD NJlSStep
Alam
Aram
SCON--iuu Stea
Adam
$2D Experinnt
OutsToat
Epoch
Epoch
Epoch
(a) 40 (k=12)
(b) Densenet-100 (k=12)
(c) Densenet- 100 (k=24)
Figure 14: Test accuracy on CIFAR-100 of different DenseNet architectures.
of MultiStep LR keep unchanged. For our method, we use the transferred strategy below: 1) For epoch 100, employ the 3 nets at 0-33, 33-67, 67-100 epoch, respectively; 2) For epoch 400, we employ the nets at 0-133, 133-267, 267-400 epoch, respectively; 3) For epoch 1200, employ the 3 nets at 0-400, 400-800, 800-1200 epoch, respectively.
epochs. Other
Transfer to different datasets. We transfer the LR schedules meta-learned on CIFAR-10 to SVHN (Netzer et al. 2011). TinylmageNer() and Penn Treebank (Mareus & Marcinkiewicz). For image classification, we train a ResNet-18 18 on SVHN and IinylmageNet, respectively. The hyper-parameterrs of all compared methods are set the same as those of CIFAR-10. For text classification, train 3-layer LSTM on Penn Treebank. The hyper-parameters of all compared methods are with the same setting as introduced in Section 4.1.
Transfer to different net architectures. We transfer the learned LR schedules for different net architectures training. All the methods are trained on CIFAR- 10 with different net architectures. The hyper-parametery of all methods are the same with the setting of CIFAR- 10 with ResNet- 18. We test the meta-learned IRR schedule to different configurations of NenseNet Huang et al. (2017). As DenseeNetFFuang
shown in Fig. [14] our method perform slightly stable than MultiStep strategy at about 75- 125 epochs. This tends to show the superiority of adaptive LR to train the DenseNets. Also, we transfer the LR schedules to several novel networks, the results are presented in Fig.8 in the main paper.
Transfer to large scale optimization. We transfer the learned LR schedules for the training of the large scale optimization problems. The predicted LR by MLR-SNet will not substantially in- crease the complexity compared with hand-designed LR schedules for DNNs training. This makes it feasible and reliable to transfer our meta-leamed LR schedules to such large scale optimization problems. We train a ResNet-50 on ImageNet with hand-designed LR schedules and our trans- ferred LR schedules. The training code be found on ht / thub. com/ pyttortoooo ch/ examples tteee//e/////e / master imagenet and the parameter setting keeps unchanged except the LR. All compared hand-designed LR schedules are trained by SGDM with momentum 0.9, weight decay 5c an initial learning rate 0.1 for 90 epochs. and batch size 256. Fixed LR uses 0.1 LR during the whole training: Exponential LR multiplies LR with 0.95 every epoch; MultiStep LR decays LR by 10 every 30 epochs; SGDR sets T_O to 10, T Mult to 2 and minimum LR to 1e-5; Adam just uses the default parameter setting. The results are presented in Fig. 9 in the main paper.
't can be downloaded http://hnyiimggeenttteeemupppmmm
22

---

Under review as a conference paper at ICLR 2021
Table 7: Test accuracy (%) CIFAR-10 10 and CIFAR-100 training set of different methods trained on CIFAR-10- 10-C and CIFAR-100-C. Best and Last denote the results of the best and the last epoch. The Bold and Underline Bold denote the first and second best results, respectively.
Mettocs
Fixed
MuhiSSep
Exponential
SGDR
Adam
Qurs(Train)
Best
79.96-14.09
85.6441.771
83,63:1336
86, 1,44
81.57:41.39
85.73=1.71
CIFAR-IO-C
Last
77,89+4.05
85.48++1.71
83.47=1.37
78.46=11.92
10,39:
85.62=1.76
CI
Best
43.91
52.3842243
49,90=11.33
22.80:22.39
45,58:11955
52.51=2.38
Last
4..91
52.28:12.44
49.75.1.94
13.68
43.94.12.18
52.35 1.4.6
D EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS IN SECTION 4.3
The datasets CI-CR-10-C and CIFAR- 100-C Hendryeks & Dietterich 2019) can be downloaded at https://zenodo: org/ record/ (tnoor//25399677 Xt 4mVi attps://zenonn org/ record/3555552+ Xt 4mdSgzZP Each dataset contains 15 types of algorithmically gen- erated corruptions from noise, blur, weather, and digital categories. These corruptions contain Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic, Pixelate and JPEG. All the corruptions are gererated on 10,000 test set images, and each corruption contains 50,000 images since each type of corruption has five levels of severity. We treat CIFAR-I0-CA or CIFAR-100-C dataset as training set, and train model with ResNet-18 18 for each corruption dataset. Finally, we can obtain 15 models for CIFAR- 0/100-C. Each corruption can be roughly regarded as task, and the average accuracy of 15 models on test data is used to evaluate the robust performance of different tasks for each LR schedules strategy.
For experimental setting in Section 4.3, all compared hand-designed LR schedules are trained with ResNet- 18 by SGDM with a momentum 0.9, a weight decay 5e -4, an initial learning rate 0.1 for 100 epochs, and batch size 128. Fixed LR uses 0.1 LR during the whole training; Exponential LR multiplies LR with 0.95 every epoch; MultiStep LR decays LR by 10 every 30 epochs; SGDR sets T_O to 10, T Mult to 2 and minimum LR to 1e 5; Adam just uses the default parameter setting. Our method trains the ResNet-18 by SGD with a weight decay 5e-A, and the MLR-SNet is learned under the guidance of small set of validation set without corruptions. We randomly choose 10 clean images for each class as validation set. The experimental result is listed in Table in the main paper.
Additional robustness results of transferrable LR schedules on different data corruptions. Fur- thermore, we want to explore the robust performance of different tasks for our transferrable LR schedules. Different from above experiments where all 15 models are trained under the guidance of a small set of validation set, we just train a ResNet- 18 on Gaussian Noise corruption to meta-learn the MLR-SNet, and then transfer the meta-learned LR schedules to other 14 corruptions. We report the average accuracy of 14 models on test data to show the robust performance of our transferred LR schedules. All the methods are meta-tested with ResNet-18 for 100 epochs with batch size 128. The hyper-parameter setting of hand-designed LR schedules keeps same with above. Table shows the mean test accuracy of models. As can be seen, our transferrable LR schedules obtain the final best performance compared with hand hand-designed LR schedules. This implies that our transferrable LR schedules can also perform robust and stable than the pre-set LR schedules when the learning tasks changed. However, our transferrable LR schedules plug-and-play and have additional hyper-parameters to tune when transferred to new heterogeneous tasks.
E THE PRELIMINARY EXPLORATION OF THE INFLUENCE ON META-TEST TASKS OF THE DIFFERENT META-TRAINING TASKS.
In this section, we study the performance influence on the target task of the different meta-training tasks. Here we fixed the target task as training ResNet-18 on TinyImageNet. We choose three different meta-training task: 1) training Re on CIFAR- 10; 2) training WideResNet-28-- 10 on CIFAR-100; 100; 3) training 3-layer LSTM on Penn Treebank. I5 the meta-test performance of three different meta-training tasks. It can be seen that the meta-training task more related to the target task would obtain better transferable performance on the target task.
We use the original 50,000 train images of CIFAR-10/100 as test data.
23

---

Under review as a conference paper at ICLR 2021
ResNet-18 TinylmageNet
ResNet-18 TinylmageNet
Resket-18 en CIFAR-10
WideheesWet228-11 CIFAR-100
3-aaer on Treebank
Test Accuracy
Train Less
ResNet-18 CIFAR-10
Widekesh et 28-10 on CIFAR-100
3-layer
Epach
Epech
(a) Train loss TinyImageNet
(b) Test accuracy TinyImageNet
Figure 15: The meta-test performance TinyImageNet (target task) of different meta-training tasks.
Adam++ Fixred
Adam: Durs(Trair)
Train Loss
Test Accunccy
Adamelixed
Adam+OurstTrah)
Epach
Epech
(a) Train loss CIFAR-10
(b) Test accuracy
CIFAR-10
Figure 16: Performance comparion on CIFAR-10 of the MLR-SNet and baselines.
F APPLYING MLR-SNET ON TOP OF ADAM.
To further demostrate the versatility of our method, apply the MLR-SNet top of the Adam algorithm. Fig [6 shows that our methods can substantially improve the performance the original Adam algorithm.
G EXPERIMENTAL RESULTS ADDITIONAL COMPARED METHOD LR CONTROLLER
In this section, we present the experimental results of LR Controller Xu et al. (2019), which is a related work of ours but under the reinforcement learning framework. Due their learning algorithm relatively computationally expensive and not very easy to optimize, we will show our method has superiority in finding such good LR schedule that scales and generalizes.
To start fair comparison, we follow all the training settings and structure of LR Controller proposed in Xu al. (2019) except that we modify the batch size 128 and increase training steps cover 200 epochs of data to match our setup in Section 4.1 Firstly, we train LR Controller on CIFAR- 10 with ResNet-18 and CIFAR-100 with WideResNet-28- as do in Section 4.1. As shown in Fig. 17 our method demonstrates evident superiority in finding a solution with better generalization compared with LR Controller strategies. LR Controller performs steadily in the early training phase, but soon fluctuates significantly and fails to progress. This tends to show that the LR Controller suffers from a severe stability issue when training step increases, especially being compared to our MNt-
Then we transfer the LR schedules learned on CIFAR-10 for our method and LR Controller to CIFAR- 100 to verify their transferability. Test settings the same with those related in Section 4.2. As shown in Fig. [8] LR Controller makes a comparatively slower progress in the whole training process. While method achieves competitive performance, which indicates the capability of transferring to other tasks for our method.
*Code for I.R Controller be found http:///wtccccoomooaaoaoooaaaaae
24

---

Under review as a conference paper at ICLR 2021
Cortieetttr CureTran
Than Loos
(a) Train loss CIFAR-I0
(b) Test accuracy CIFAR-I0
(c) Learned LR schedules
WidcRezNet 28 10 CIRAR 300
Waseeeseer on dFAR
Witckeshitt 28 10 OFAA 100
dundTaa
Sparh
Epoch
Spach
on CIFAR-100
Test accuracy on CIFAR-100
Learned LR schedules
Figure 17: Train loss, test accuracy and learned LR schedules of method(train) and LR Con- troller(train) on CIFAR-10 and CIFAR-100.
ResNet 18 on CIFAR-100
ResNet 18 on CIFAR-100
Controlle(Test) Ours(Tast)
50-
Train
60-
Test
LR Controlle((Test)
Ours(Test)
8.4
45
Epoch
Epach
(a) Train loss CIFAR-100
(b) Test accurncy CIFAR-100
Figure 18: Train loss, test accuracy of our method(test) and LR Controller(test) on CIFAR-100.
25


Reviews:
[{"review_id": "sxSEN765By1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis paper proposes a learning-to-learn type approach for step size schedules. An LSTM is used to predict step sizes based on observed values of the training loss. It is meta-trained with the goal of achieving maximal validation loss. The proposed method is evaluated empirically on image and text classification problems.\n\n\n## Rating\n\nThe basic idea of the paper is solid. Fully meta-learned optimization methods have shown promising results, but have been brittle when transferred to settings that deviate substantially from those they have been meta-trained on. Using a standard optimizer update direction and only meta-learning the step size schedule seems like a reasonable idea to tackle the major pain point (step size tuning) while potentially providing more robustness. But, in my opinion, the idea is not executed and evaluated up to the standard of this conference. I have several issues with the quality and transparency of the experimental evaluation and the proposed method shows only limited success in the presented experiments. I am detailing my concerns below. Moreover, the paper is not really well-written. **In its current state, I recommend rejecting this paper.**\n\n\n## Major Comments\n\n1) The quality of the empirical evaluation is questionable in my opinion.\n    a) The settings and competing methods are chosen somewhat erratically. For example, why is there no comparison to a simple multi-step schedule and/or SGDM for the text datasets?\n    b) There is no explanation whatsoever of the used protocol for tuning the hyperparameters of the methods involved in the experimental comparison.\n    c) Why is there no comparison to other automatic adaptation techniques (L4, HD) in the transfer experiments? The paper says that “Since the methods […] are not able to generalize, we do not compare them here”. I don’t understand what that is supposed to mean. Since they aren’t meta-learned, there is no notion of generalization for this method. But they could still be applied in the transfer setting as baselines.\n\n2) This is a purely empirical paper proposing a practical method for learning rate scheduling. The impact of such a paper comes down, to a large extent, to the success of the proposed method. Unfortunately, the results are really a mixed bag. The proposed method outperforms baselines only in a small subset of the experiments: mainly when meta-training the MLR-SNet on the identical dataset/architecture (Section 4.1), and even there it gets outperformed by a multi-step schedule when using SGD with momentum. In the transfer experiments (Section 4.2), the proposed method is matched or outperformed by one of the simple baselines in almost all cases. Of course, there might be an argument to be made in terms of the effort of manual tuning that goes into the different baselines. But that point is not really driven home in the paper and I am skeptical for two reasons: (i) the absence of a clear hyperparameter tuning protocol (see previous point) and (ii) the fact that the method still relies on a manually-chosen scaling factor $\\gamma$ for the step size. Overall, I don’t think the experimental results are convincing.\n\n3) I am missing a discussion of the computational and memory cost of the proposed method. The meta-updates of the LSTM weights (line 6 in Algorithm 1) requires backpropagating through an interval of $T_\\text{val}$ past iterations. Such back propagation through unrolled training trajectories are usually very memory-intensive (the entire state of the model has to be kept in memory for the unrolled iterates)  and adds considerable computational cost. In Section 4.1, this supposedly very costly method is compared to baselines that are essentially for free without any consideration given to that discrepancy in computational cost. Of course, this is a one-off cost when using a pre-trained MLR-SNet as a plug-and-play step size scheduler, but the results for this transfer setting are not really convincing (see above).\n\n4) The paper proposes to adapt the learning rate based on the observed loss values. Fully meta-learned optimizers use gradient observations in addition to the loss. Why did you choose not to use the gradient information. To keep the method light-weight? Because a good step size can be chosen exclusively based on loss values? I think this modelling choice should be discussed and justified in the paper.\n\n\n## Minor Comments\n\n5) The quality of the writing is subpar. I am trying not to get hung up on linguistic mistakes, but occasionally it is really hard to decipher the meaning of certain sentences. There are also plenty of typos and stylistic errors. The bib-file could need a thorough review: you are citing arXiv versions of several papers that have been published in peer-reviewed venues, conferences are cited only by their acronym, et cetera.\n\n6) It would have been nice to investigate the versatility of the proposed method by applying it to other base update directions than SGD(M). For example, could the MLR-SNet be applied on top of Adam as well?", "extra_scores": {}, "extra_text": {}}, {"review_id": "mCEKZspX1Zr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to learn an adaptive LR schedule, which can adjust LR based on current\ntraining loss and the information delivered from past training histories. It adopts a (parameterized) LSTM as the schedule, which is meta-learned following the existing work meta-weight-net. Extensive experiments are conducted to show its effectiveness on image classification, compared to pre-defined policies. \n\nIn general, this paper addresses an important problem of LR schedule and the suggested method is easy to follow and the experimental results are promising. However, there are several issues that need to be addressed to improve the work further.\n\n1.\tThe novelty seems quite limited – it appears to be a straightforward application of meta-weight-net to the problem of LR scheduling.\n2.\tAnalysis on the learned optimizers is lacking, such as the convergence and speed.\n3.\tIt is not clear why the learned optimizer can be transferred to new heterogeneous tasks. Some deeper insights might be helpful. For example, can a LR scheduler learned from an image classification task be transferrable to a task of object detection?\n4.\tOther forms of parameterized LR scheduler should be discussed and compared, such as MLP. It is not obviously clear why LSTM is a good choice -- the structure of LSTM seems quite complicated.\n5.\tMissing relevant work: Meta-SGD (https://arxiv.org/abs/1707.09835) which also meta-learns a LR scheduler.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YVMfa7Lwt0B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors use an LSTM to meta-learn learning rate schedules. This LSTM depends only on the validation loss at time t. They train this LSTM for some tasks and they show that it gives good performance when compared with baselines. Then they transfer one of these trained LSTMs to different tasks and gives good results. \n\nUnderstanding learning rate schedules is an interesting problem in machine learning and the authors seem to provide a useful blackbox learning rate scheduler.\n\n-In the SM, they discuss that they have to choose 3 MLR-SNET for transferability. This part is very confusing and should be written more clearly. This should also be mentioned in the main text because the impression there is that one just transfered the MLR-SNET as is. \n-How many runs are needed to meta train the LSTM? Is the transferable LSTM coming from only one run?\n\n-How robust is the performance of MLR-SNET with respect to different initial learning rates?\n\n-It would be nice if the authors could add tables reporting the accuracies, it is not clear from the plots how good/bad the performance of MLR-SNET is. The Adam baseline is pretty weak, it should be supplemented with learning rate schedules, one should be able to get acc > 73% with Adam+schedules on imagenet.", "extra_scores": {}, "extra_text": {}}, {"review_id": "63mBoQ2uMqM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes to parameterize learning rate (LR) schedule with an explicit mapping formulation. This learnable structure allowed the proposed meta-trained MLR-SNet to achieve good LR schedules. For validation, the proposed method is evaluated on both image and text classification benchmark with various network architectures and datasets, as well as transfer the learned network for new task or architectures. \n\nJustification of rating: The paper solve a practical problem that is not handled in the existing literature. Despite the straightforwardness of the proposed approach and the methodology, this work has the potential to bring high impact to the research community. \n\nStrengths:\n+ This work proposed to parameterize the LR schedule with a MLR-SNet.  The results shows it is more flexible and general than the hand-picked LR schedule.\n+ The meta-learned approach allow the learned model to be applied to unseen data.\n+ The paper provide comprehensive experiment to validate the efficacy of the proposed model. \n\nComments:\n- Experiment on Penn Treebank shows the convergence of the proposed MLR-SNet is slower than SGD and Adam. The paper argue that it predicts LR according to training dynamics by minimizing the validation loss. Please provide more details why is this a more intelligent way to employ validation set. Is this also observed in any other dataset?\n- This work transfer the learned LR schedules on CIFAR-10 with ResNet-18 to several other datasets. Has the author  try to transfer MLR-SNet learned with other source dataset? How will the model trained with different model/dataset behave when transferred to new datasets or networks. It might be interesting to explore if the certain type of network architecture (more complex or simple) would learned a more generic model.\n- In the \"Formulation of MLR-SNet\", it states that the input $h_{t-1}$ and the training loss are preprocessed by a fully-connected layer $W_1$ with ReLu activation function. Please describe the purpose of this layer. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a meta-learning method for tuning the learning rate. In the discussion, reviewers agreed that the key issue is that the empirical evaluation is not yet sufficient to demonstrate the efficacy of the method. In particular, this is an especially pressing issue given that there are now many meta-learning methods for tuning the learning rate (none popular in practice though), and the paper does not compare to any of them. Relatedly, most reviewers found that the novelty of the method is not clearly established and discussed in the paper. 

Based on the above, I have to recommend rejecting the paper. I would like to thank the authors for submitting the work for consideration to ICLR. I hope the feedback will be useful for improving the work.

Decision: Reject