On the Variance of the Adaptive Learning Rate and Beyond
Authors: Liyuan Liu
Haoming Jiang
Pengcheng He
Weizhu Chen
Xiaodong Liu
Jianfeng Gao
Jiawei Han
Venue:  (2020)
Keywords: warmup
adam
adaptive learning rate
variance

Abstract: The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate -- its variance is problematically large in the early stage, and presume warmup works as a variance reduction technique. We provide both empirical and theoretical evidence to verify our hypothesis. We further propose Rectified Adam (RAdam), a novel variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the efficacy and robustness of RAdam. 

Published as a conference paper at ICLR 2020
ON THE VARIANCE OF THE ADAPTIVE LEARNING RATE AND BEYOND
Liyuan Liu
Haoming Jiang
Pengcheng He, Weizhu Chen
University of Illinois, Urbann-Champaign 1128111- rof
Georgia Tech
Microsoft Dynamics 365 Al
11 anghm@gattch.edd
(penhe, wzchen] www.ott.occoooooooooooo.ooo com
Xiaodong Liu, Jianfeng Gao Microsoft Research (xiaodl, jagao) emicrosott com
Jiawei Han
University of Illinois, Urbana-Champann
hanj@illizoir
ABSTRACT
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Pursuing the theory behind warmup, we identify a problem
the adaptive learning rate its vari- ance is problematically large the early stage, and presume warmup works as a variance reduction technique. We provide both empirical and theoretical evidence verify our hypothesis. We further propose Rectified Adam (RAdam), a novel variant of Adam, by introducing term to rectify the variance of the adaptive learning rate. Experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the efficacy and robustness of Addmm
INTRODUCTION
Fast and stable optimization algorithms what generations of researchers have been pursuing (Gauss) 1823 Cauchy 1847). Remarkably, stochastic gradient-based optimization, such as stochastic gradient descent (SGD), has witnessed tremendous success in many fields of science and engineering despite its simplicity. Recently, many efforts have been made to accelerate optimization by applying adaptive learning rate. In particular, Adagrad (Duchi et al. (2010) and its variants, e.g., RMSprop (Hinton et al. 2012), Adam (Kingma Ba, 2014). Adadelta (Zeiler) (012) and Nadam (Dozat, (206)), stand out due to their fast convergence, and have been considered as the optimizer of choice in many applications.
Ovr s000
Adam-eps
Adam-2k
Adam-vanilla
RAdam
Adam-warmup
Figure Training loss v.s. # of iterations of Transformers the De-En IWSLT" 14 dataset.
However, it has been observed that these optimization methods may converge to bad/suspicious local optima, and have resort warmup heuristic using small learning rate in the first few epochs training to mitigate such problem (Vaswani et al.) 2017: [Popel & Bojar 2018). For example, when training typical Transformers based neural machine translation models on the De-Ei IWSLT' 14 dataset, removing the warmup stage increases the training loss from to around 10, as shown in Figure I Similar phenomena are observed in other scenarios like BERT (a bidirectional transformer language model) pre-traae pre-training (Devlin et al. 2019).
Duo to the lack of the theoretical underpinnings, there is neither guarantee that warmup would bring consistent improvements for various machine learning settings nor guidance on how we should
"Work done during internship Microsoft.
1 Work was done during an internship at Microsoft.
All implementations are available at: http:://ggghhhee.cm//aaaaaaaaaaa

---

Published as a conference paper at ICLR 2020
conduct warmup. Thus, researchers typically different settings in different applications and have to take a trial-and-eroor approach, which can be tedious and time-consuming.
In this paper, conduct both empirical and theoretical analysis of the convergence issue to identify its origin. We show that its root cause is: the adaptive learning rate has undesirably large variance in the early stage of model training, due the limited amount of training samples being used. Thus, to reduce such variance, it better to use smaller learning rates the first few epochs of training, which justifies the warmup heuristic.
Inspired by our analysis results, we propose a new variant of Adam, called Rectified Adam (RAdam), which explicitly rectifies the variance of the adaptive learning rate based on derivations. We conduct extensive experiments on language modeling, image classification, and neural machine translation. RAdam brings consistent improvement over the vanilla Adam, which verifies the variance issue generally exists on various tasks across different network architectures.
In summary, our main contributions are two-fold:
We identify the variance issue of the adaptive learning rate and present a theoretical justification for the warmup heuristic. We show that the convergence issue is due to the undesirably large variance of the adaptive learning rate in the early stage of model training.
We propose new variant of Adam (i.e., RAdam), which not only explicitly rectifies the variance and is theoretically sound, but also compares favorably with the heuristic warmup.
2 PRELIMINARIES AND MOTIVATIONS
Generic adaptive methods. Algorithm is generic framework (all operations are element-wise).
It describes various popular stochastic gradient descent algorithms (Reddi et al. (018)) Specifically, different optimization algorithms can be specified by different choices of ((.) and (.)., where o(.)
specifies how the momentum at time step is calculated, and (b()) how the adaptive learning rate at t calculated. For example, in the Adam algorithm, we have:
p0g1.-. (c) (1 31) E = 3 Oe and (0(g1. .9t) (1 32) Σi= 82 9
83
(1)
8
For numerical stability, the function (.) in Equation is usually calculated as ((gi, 9t)
VI-
e+v (1- 32) EL o where e a relatively small / negligible value (e.g., x 10-*).
Algorithm 1: Generic adaptive optimization method setup. All operations are element-wist
Input: {ar [ step size, {Φr, the H" function to calculate momentum and adaptive rate,
0g: initial parameter, f0): stochastic objective function.
Output: 0y: resulting parameters
while = 1 do
It (0 (Calculate gradients w.r.t. stochastic objective at timestep t)
3
m + de ................................
... 99)) (Calculate momentum)
4 + velgi. 9t) (Calculate adaptive learning rate)
8, t f - a (Update parameters)
return fr
Learning rate warmup. Instead of setting the learning rate a as a constant or in a decreasing order, a learning rate warmup strategy sets a; as smaller values in the first few steps. thus not satisfying Vl S or- For example, linear warmup sets a; = Lop when < Ty. Warmup has been demonstrated to be beneficial in many deep learning applications. For example, in the NMT experiments in Figure [] the training loss convergences around 10 when warmup is not applied (Adam-vanilla), and it surprisingly decreases below 3 after applying warmup (Adam-warmup
To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on log scale in Figure 2 We observe that, without applying warmup, the gradient distribution distorted to have a mass center in relatively small values within 10 updates. Such gradient dis- tortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few
2

---

Published as a conference paper at ICLR 2020
The distributionis distorted within 10 updates. 400uip"
Moratton
Beritton
Adam without warmup
Adam with warmup
Figure 2: The absolute gradient histogram of the Transformers on the De-En IWSLT' 14 dataset during the training (stacked along the y-axis). X-axis is absolute value in the log scale and the height is the frequency. Without warmup, the gradient distribution is distorted in the first 10 steps.
2.4
Adan-eps
Figure 3: The histogram of the absolute value of gradients (on a log scale) during the training of Transformers on the De-En IWSLT' 14 dataset. using Adam-2k, RAdam and Adam-eps.
updates. Warmup essentially reduces the impact of these problematic updates to avoid the conver- gence problem. In the following sections, we focus our analysis on learning rate warmup for the Adam algorithm, while can be applied to other algorithms that use similar adaptive learning rate (w))) designs, e.g., R.SSprop (Hinton et al. 2012) and Nadam Dozat 2016).
VARIANCE OF THE ADAPTIVE LEARNING RATE
In this section, we first introduce empirical evidence, then analyze the variance of the adaptive learning rate to support hypothesis Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.
To convey our intuition, we begin with a special case. When = 1, we have ((g)) = / We view (91,... . as i.i.d. Gaussian random variables following N 2 Therefore, 1/g is subject to the scaled inverse chi-squared distribution, Scale-inv-X 1/a²), and Varl divergent. It means that the adaptive ratio can be undesirably large in the first stage of learning. Meanwhile, setting small learning rate at the early stage can reduce the variance (Var[ox) a² Var(z]), thus alleviating this problem. Therefore, we suggest it the unbounded variance of the adaptive learning rate in the early stage that causes the problematic updates.
3.1 WARMUP AS VARIANCE REDUCTION
In this section, design a set of controlled experiments verify our hypothesis. Particularly, design two variants of Adam that reducing the variance of the adaptive learning rate: Adam-2k and Adam-eps. We compare them vanilla Adam with and without warmup on the IWSLT' 14 German to English translation dataset (Cettolo et al., 2014).
In order reduce the variance of the adaptive learning rate (v)())), Adam-2k only updates (b.) in the first two thousand iterations, while the momentum (o(-) and parameters (0) are fixed") other than this, it follows the original Adam algorithm. To make comparison with other methods, its iterations are indexed from -1999 instead of 1. In Figure [ we observe that, after getting these additional two thousand samples for estimating the adaptive learning rate, Adam-2k avoids the convergence problem of the vanilla-Adam, Also, comparing Figure 2 and Figure getting large enough samples prevents the gradient distribution from being distorted. These observations verify our hypothesis that the lack of sufficient data samples in the early stage is the root cause of the convergence issue.
The mean zero normal assumption is valid at the beginning of the training, since weights are sampled from normal distributions with mean zero (Balduzzi et al. 2017), further analysis is conducted in Section 5.3 Different from Gotmare al. 2019 all parameters and first moments are frozen in the first 2000 iterations.
3

---

Published as a conference paper at ICLR 2020
Another straightforward way to reduce the variance is to increase the value of e in (gg,.... .9t)
V1-8i ctv(1 82] 2i 1 Actually, if we assume (b)) is subject to the uniform distribution, its vari-
ance equals T2 Therefore. we design Adam-eps, which uses non-negligibly large c 10 while 10 for vanilla Adam. Its performance is summarized in Figure We observe that does not suffer from the serious convergence problem of vanilla-Adam. This further demonstrates that the convergence problem can be alleviated by reducing the variance of the adaptive learning rate, and also explains why tuning is important in practice (Liu et al.) (219)) Besides, similar to Adam-24 2k, it prevents the gradient distribution from being distorted (as shown in Figure However, as in Figure [ it produces a much worse performance comparing to Adam-2k and Adam- warmup.. We conjecture that this is because large induces a large bias into the adaptive learning rate and slows down the optimization process. Thus, need a more principled and rigorous way to con- trol the variance of the adaptive learning rate. In the next subsection, we will present a theoretical analysis of the variance of the adaptive learning rate.
3.2 ANALYSIS OF ADAPTIVE LEARNING RATE VARIANCE
As mentioned before, Adam uses the exponential moving average to calculate the adaptive learning rate. For gradients {91..... }. their exponential moving average has a larger variance than their simple average. Also, in the early stage (f is small), the difference of the exponential weights of {gi.... } } is relatively small (up the distribution of the exponential moving average as the distribution of the simple average (Nau
11 Therefore, for ease of analysis, we approximate
2014), ie, p(1(()) = P(V [1- (1-8))
R P(V VVEi
Since di ~ (O, o²), we have
also subjects to a scaled inverse chi-square distribution with p degrees freedom (further analysis on this approximation conducted in Section (.3). Based this assumption, we can calculate Var(e) (.)] and the PDF of
Scale-inv-2 ) Therefore, we assume (1-ffa bi A
1-2:
(.). Now, we proceed to the analysis of its square root variance, í.e., Var (.)]. and show how the variance changes with p (which corresponds number of used training samples).
Theorem 1. If x) (.) Scale-inv- (p. )²), Var[p(-)] monotonically decreases as p increases.
Proof. For Vp p 4, have:
Var El -))))
222p-5
B p p-
)2)
p p-2
π
2
where B()) is the beta function. By analyzing the derivative of Var[1b()), we know it monotonically decreases as p increases. The detailed derivation is elaborated in the Appendix AA]
Theorem D gives a qualitative analysis of the variance of the adaptive learning rate. It shows that, due to the lack of used training samples in the early stage, Var[u/()] is larger than the late stage (Figure g To rigorously constraint the variance, perform quantified analysis Var[y())] by estimating the degree of freedoms p.
RECTIFIED ADAPTIVE LEARNING RATE
In the previous section, Equation 2 gives the analytic form of Var[u>.)], where p is the degree of freedoms. Here, we first give estimation of based on t to conduct quantified analysis for Var Var[v/g1, ... 3. then we describe the design of the learning rate rectification, and compare it to the heuristic warmup strategies.
4.1 ESTIMATION OF
The exponential moving average (EMA) can be interpreted as an approximation to the simple mov- ing average (SMA) in real application (Nau 2014), i.e.
(1
32)
82
Σ
f(t.82)
p
= p
(3)
f(t,82)
4

---

Published as a conference paper at ICLR 2020
Algorithm 2: Rectified Adam. All operations are element-wise.
Input: (ar [o=) step size, {81. decay rate to calculate moving average and moving 2nd moment, 0p: initial parameter, (0): stochastic objective function.
Output: 8y: resulting parameters
me; d0 +- 0, (Initialize moving 1st and 2nd moment)
Pro <- 2/(1- (2) (Compute the maximum length of the approximated SMA)
while = {1.... .) do
9t - (Calculate gradients w.r.t stochastic objective timestep t)
Dt 1/82th-1 +1 (2)9g (Update exponential moving 2nd moment)
my - B1m4-1 + 81)g: (Update exponential moving 1st moment)
my + mi/(1 - } (Compute bias-corrected moving average)
Pt 5 Poo -2t (Comuute the length of the approximated SMA)
if the variance is tractable, i.e., Pt then
4 v(1 3/)/v, (Compute adaptive learning rate)
11
Ft
(poo -]60 2]ps
[p: 00--4Xpr- 21p
(Compute the variance rectification term)
12
0. 01-1 -arriilee (Update parameters with adaptive momentum)
else
14
0₂ t f1-- - a,m, (Update parameters with un-adapted momentum)
15
return Or
where ft, B2) is the length of the SMA which allows the SMA to have the same "center of mass" with the EMA. In other words, (t., 82) satisfies:
(1 82) Ei= S3-+.
1(1,0)) 11+)
(4)
1 8/
f(t,32)
By solving Equation
we have: f(t, = 1-
the previous section, Scale-inv- X2 ..). Here, since 9: NN(000),, we have
2 -
21 1-3
we assume: (1-8g) 9
Σ (() Firs Seale-inv-x? 32), 3/2 Thus, Equation 3 views Scale-inv-X (f((, B2), } ((.
as approximation to Scale-inv-2" # Therefore, we treat [t,, 32) as estimation of For ease of notation, mark f(t, B2) Pt. Also, refer I-32 as Poo (maximum length of the approximated SMA), due to the inequality 31, S lim 3(1 = I-3
2
4.2 VARIANCE ESTIMATION AND RECTIFICATION
have Var[w(()] ( 200 P--2 value of this function in the early stage is significantly larger than the late stage (as analyzed later, decays roughly at the speed of O (1) )). For example, the variance at Pt 5 is over 100 times larger than the variance Pt 500. Additionally, based Theorem
Based on previous estimations,
B3 p--1 p=)l) )2). The
know minp, Var[e>())
Var (ar(w) (-) p. and mark this minimal value as Cvar- order to ensure that the adaptive learning rate (x)) has consistent variance, rectify the variance at the t-th timestamp as below.
Varlra (' .9:)] = Ovar where r, /Cvar/ //Cva//Vrrjp /arlV/ar1. .9))].
Although we have the analytic form of Var[v(-)] (i.e., Equation 2). it is not numerically stable. Therefore, we use the first-order approximation to calculate the rectification term. Specifically, by approximating ())((.) to the first order (Wolter 2007).
(.) - (.I) and Varl& Vark (.)
2 [E))(.)
4E
Since +²()) Scale-inv- 2 ).), we have:
Var [ar[ ~ pt/1((pt () 4
(5)
In Section 5.3 conduct simulation experiments to examine Equation and find that it is a reliable
approximation. Based Equation 5 know that (b)) decreases approximately at the
5

---

Published as a conference paper at ICLR 2020
Table 1: Image Classification
Training PPL
Test PPL
Method
Acc.
170
100
190
CIFARIO
SGD
91.51
Adam
90.54
RAdam
91.38
13M
120
201
Gradient updates
Iterations training
ImageNet
SGD
69.86
RAdam
Adam
Adam
66.54
Figure 4: Language modeling (LSTMs) on the One Billion Word.
RAdam
67.62
RAdan
Adam
0.13
Duit orlin donneet
Herraton DUST dattunl
Harmatoo buar comtent
ImageNet
CIFAR10
Figure Training of ResNet-18 on the ImageNet and ResNet-20 on the CIFAR1O dataset.
speed of (.)) With this approximation, we can calculate the rectification term as:
(p: - 4)(pr p0
r1
(Poo - 4) 2)pt
Applying our rectification term to Adam, we come up with new variant of Adam, Rectified Adam (RAdam), as summarized in Algorithm Specifically, when the length the approximated SMA is less or equal than 4, the variance of the adaptive learning rate is intractable and the adaptive learning rate inactivated. Otherwise, we calculate the variance rectification term and update parameters with the adaptive learning rate. It is worth mentioning that, if (3 S 0.6, we have Pso S 4 and RAdam is degenerated to SGD with momentum.
4.3 IN COMPARISON WITH WARMUP AND OTHER STABILIZATION TECHNIQUES
Different from the analysis in this paper, warmup is originally proposed to handle training with very large batches for SGD (Goyal et al. 2017 Gotmare al. 2019 [Bensttii et al. 2018 Xiao et al.) 2017]. We notice that F; has a similar form the heuristic linear warmup, which can be viewed as setting the rectification term as (...). It verifies our intuition that warmup works as a variance Te
reduction technique. RAdam deactivates the adaptive learning rate when its variance divergent, thus avoiding undesired instability in the first few updates. Besides, method does require an additional hyperparameter (i.e., Tw)) and can automatically adapt to different moving average rules.
Here, we identify and address underlying issue of adaptive optimization methods independent of (neural) model architectures. Thus, the proposed rectification term is orthogonal to other train- ing stabilization techniques such as gradient clipping (Bengio al. 2013), smoothing the adaptive learning rate (i.e., increasing e, applying geometric mean filter (Chen & Gu 2018). adding range constraints (Luo al. 2019 9)), initialization (Balduzzi et al. 2017 Zhang al. (019)) and normal-
ization (Ba et al.] 2016 loffe & Szegedy, 2015). Indeed, these techniques can be combined with the
proposed variance rectification method.
EXPERIMENTS
We evaluate RAdam on several benchmarks: One Billion Word for language modeling: Cifar10 and ImageNet for image classification; IWSLT" 14 De-En/EN-DE and WMT" 16 EN-De for neural machine translation. Following Loshchilov Hutter (2018), we decouple weight decays in the vanilla Adam, Adam with warmup and RAdam in our experiments. Details are in Appendix B
5.1 COMPARING TO VANILLA ADAM
As analyzed before, the adaptive learning rate has undesirably large variance in the early stage of training and leads suspicious/had local optima on NMT. One question are interested in

---

Published as a conference paper at ICLR 2020
Sensitive to the choice the learning rate.
Ir- 0.1
Traning losi sccuracy
I= 0.03
Different lead similar
0.01
performance.
0.003
100
143
0150
0.6
0.65
3.00
0.30
0225
2.25
0.22
0.55
2.55
11115
X-axis is the epoch #.
80
140
150
185
100
590
Figure 6: Performance of RAdam, Adam and SGD with different learning rates on CIFARIO.
Comparing RAdam, warmup
warmup length get similar performance.
RAdam
Adam with
langth:
Test accurso)
Training loss
Ir 0.1
1 0.03
Ir 0.01
0.003
X-axis epoch
Figure 7: Performance of RAdam, Adam with warmup on CIFAR10 10 with different learning rates.
is: whether such an issue widely exits other similar tasks and applications. Thus, we conduct set of experiments with two classical tasks of NLP and CV, i.e., language modeling and image classification. RAdam not only results consistent improvements over the vanilla Adam, but also demonstrates its robustness to the change of learning rates. It verifies that the variance issue exists in various machine learning applications, and has big impact on the model behavior.
language modeling (i.e., One Billion Word CCelbba et al. (203)) and image classification (i.e., CIFARIO (Krizhevsky et al. 2009) and 2009)) are presented in Figure 4 The results show that RAdam out- performs Adam in all three datasets. As shown in Figure 4 although the rectification term makes RAdam slower than the vanilla Adam in the first few epochs, it allows RAdam converge faster after that. In other words, by reducing the variance of the adaptive learning rate in the early stage, it gets both faster convergence and better performance, which verifies the impact of the variance issue. We also observe that RAdam obtains consistent improvements over Adam on image classification. is worth noting that, on both ImageNet and CIFAR1O. although RAdam fails outperform SGD in terms of test accuracy, it results in a better training performance (e.g., the training accuracy of SGD, Adam, and RAdam on ImageNet are 69.57, 69.12 and 70.30 respectively).
Performance Comparison. The performances
ImageNet (Deng
Robustness to Learning Rate Change. Besides performance improvements, RAdam also improves the robustness of model training. We use different initial learning rates, conduct experiments with ResNet-20 on the CIFARI0 datasets, and summarize their performance in Figure 6 For learning rates within a broad range (i.e., 1, 0.03, 0.01, 0.003}), RAdam achieves consistent model perfor- mances (their test accuracy curves highly overlap with each other), while Adam and SGD shown to more sensitive to the learning rate. The observation can be interpreted that by rectifying the variance of the adaptive learning rate, RAdam improves the robustness of model training and can adapt to different learning rates of a broader range.
5.2 COMPARING TO HEURISTIC WARMUP
To examine the effectiveness of RAdam, first conduct comparisons neural machine transla- tion. on which the state-of-the-aas employs Adam with the linear warmup. Specifically, we conduct experiments three datasets, i.e., IWSLT" 14 De-En, IWSLT' 14 En-De, and WMT" 16 En-De. Due

---

Published as a conference paper at ICLR 2020
Table 2: BLEU score on Neural Machine Translation.
Method
IWSLT 14 DE-EN
IWSIT" 14 EN-DE
WMT 16 EN-DE
Adam with warmup
34.66 + 0.014
28.56 = 0.067
27.03
RAdam
34.76 0.003
28.48 + 0.054
27.27
to the limited size of the IWSLT" 14 dataset, we conduct experiments using 5 different random seeds and report their mean and standard derivation. As discussed before, the vanilla Adam algorithm leads to suspicious/bad local optima (i.e., converges to a training perplexity around 500), and needs a learning rate warmup stage to stabilize the training.
We summarize the performance obtained with the heuristic warmup and our proposed rectification term in Table 2 and visualize the training curve of IWSLT De-En Figure TT With a consistent adaptive learning rate variance, our proposed method achieves similar performance that of previ- ous state-of-the-art warmup heuristics. It verifies our intuition that the problematic updates Adam indeed caused by the undesirably large variance in the early stage.
Moreover, we applied Adam with warmup on the CIFAR1O dataset. Its best accuracy on the test set is 91.29, which is similar to RAdam (91.38) However, found that RAdam requires less hy- perparameter tuning. Specifically, we visualize their learning curves in Figure For some warmup steps, Adam with warmup is relatively more sensitive the choice of the learning rate. RAdam, at the same time, is not only more robust, but also can automatically control the warmup behav- ior (i.e., without requiring the length of warmup). For example, when setting the learning rate as 0.1, Adam with 100 steps of warmup fails to get satisfying performance and only results in an ac- curacy of 90.13; RAdam successfully gets an accuracy of 91.06, with the original setting of moving average calculation (i.e., 31 = 0.9, /32 = 0.999). We conjecture the reason is due to the fact that RAdam, which based on rigorous variance analysis, explicitly avoids the extreme situation where the variance is divergent, and rectifies the variance to be consistent in other situations.
5.3 SIMULATEED VERIFICATION
In Sections and 4 we approximate Varly E gi) to the first order, and assume 2l)
(1- 92) -i subjects to a sealed inverse chi-square distribution (this assumption covers the approximation from EMA to SMA). Here, we examine these two approximations using simulations.
First Order Approximation of Var Varl/ ///////// 9²). To compare Equations and we assume
= and plot their values and difference for v (5..... 500} in Figure g The curve of the analytic form and the first-order approximation highly overlap, and their difference is much smaller than their value. This result verifies that our first-order approximation is very accurate.
Scaled Inverse Chi-Square Distribution Assumption, this paper, we assume 9i accords to a Normal distribution with zero mean. We also assume (.) accords to the scaled inverse chi-square distribution to derive the variance of Var(e).))], based on the similarity between the exponential moving average and simple moving average. Here, empirically verify this assumption.
Specifically, since gi in the optimization problem may not be zero-mean, we assume its expectation μ and sample 9; from (j,,, 1). Then, based these samples, we calculate the variance of the original adaptive learning rate and the proposed rectified adaptive learning rate, i.e., Varl 1 and Varlg gt] respectively. We set 82 to 0.999, the number of sampled trajectories to 5000, the number of iterations to 6000, and summarize the simulation results in Figure Across all six settings with different μ. the adaptive learning rate has larger variance in the first stage and the rectified adaptive learning rate has relative consistent variance. This verifies the reliability of our assumption.
6 CONCLUSION
In this paper, we explore the underlying principle of the effectiveness of the warmup heuristic used for adaptive optimization algorithms. Specifically, we identify that, due to the limited amount of samples in the early stage of model training, the adaptive learning rate has an undesirably large variance and can cause the model to converge to suspicious/bad local optima. We provide both empirical and theoretical evidence support our hypothesis, and further propose new variant
8

---

Published as a conference paper at ICLR 2020
10a
10
10
10-1
Varig
Difference
10-1
Andlytic First Order Appros.
10"
10-2
0
11 0.001
/4
0.01
10-4
10"
10"
10-5
10
10.4
16
200
400
Figure 8: The value of Equation Equation[aland their difference (abso- lute difference). The x-axis is and the y-axis is the variance (log scale).
2
(540
5000
2500
2001
0.1
/1
10
Figure The simulation of Varl 1 and Var [ [ar[e:] The x-axis y. is iteration (from 5), the y-axis is the variance (log scale).
of Adam, whose adaptive learning rate is rectified so have consistent variance. Empirical results demonstrate the effectiveness of our proposed method. In future work, we plan to replace the rectification strategy by sharing the second moment estimation across similar parameters.
ACKNOWLEDGE
We thank Aeyuan Allen-Zhu for valuable discussions and comments, Microsoft Research Technol- ogy Engineering team for setting GPU machines. Research was sponsored in part by DARPA No. 1911NF-17-C-00E and FA8750-19 19-2-1--2---- 2-10004 National Science Foundation IIS 16-18481, IIS 17-04532, and IIS-17-41317, and DTRA HDTRA118 1810026.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In ICML, 2017.
Yoshua Bengio, Nicolas Boulanger-Lewanddoskk and Razvan Pascanu. Advances in optimizing recurrent networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8624-8628, IEEE, 2013.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Arizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems. In ICMI,, 2018.
Augustin Cauchy. Méthode générale pour la résolution des systemes déquations simultanées. Comp. Rend. Sci. Paris, 25(1847):536- 538, 1847.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation,, 2014.
Ciprian Chelba, Tomas Mikolov, Michael Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language mod- eling. In INTERSPEECH. 2013.
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.0677 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In ICML, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HIT,, 2019.

---

Published as a conference paper at ICLR 2020
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In COLT, 2010.
Carl-Friedrich Gauss. Theoria combinationis observationum erroribus minimis obnoxiae. Commen- tationes Societatis Regiae Scientiarum Gottingensis Recentiores, 1823.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In ICLR, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: (tdi:::706.02677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 2012.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Liyuan Liu, Xiang Ren, Jingho Shang, Jian Peng, and Jiawei Han. Efficient contextualized repre- sentation: Language model pruning for sequence labeling. EMNLP, 2018
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907 2019.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ICLR, 2018.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. ICLR, 2019.
Robert Nau. Forecasting with moving averages. 2014.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL, 2019.
Martin Popel and Ondłcj Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1)433770, 2018.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Kirk Wolter. Taylor series methods. In Introduction to variance estimation. 2007.
Lin Xiao, Adams Wei Yu, Qihang Lin, and Weizhu Chen. Dscovr: Randomized primal-dual block coordinate algorithms for asynchronous distributed optimization. Mach. Learn. Res., 2017.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1212.5701 2012.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2020
A PROOF OF THEOREM 1
For case of notation, we refer (.) as x and
p = pr2 expl ppr 0² as 7². Thus, x ~ Scale-inv-x2"ppp77 and:
2x
and E[x] (p- 2) (vp> 2)
(6)
T(p/2) rl+p/2
where [()) is the gamma function. Therefore, we have:
Vzp(r) dr TVDI((P- 1)/2)
B x
(Vp > 4).
v2T(p/2)
Based on Equation 6 and for V 4, we have:
p 2220-5
B p - 1 p -- )3)
Var[pb(-] Varlvar = R [x]] - E [ -2
(8)
p 2
2 2
where B() is the beta function. To prove the monotonic property of Var[{) we need to show:
Lemma 1. for t 2 4, 0 ( - B 2 )
t-1
<0
Proof. The target inequality can be re-wrote as
t
2
t
1
B
01
1 2
2
2
-2
2
t-1
t
t220 14
1-1 1 t t-1
1
B
B
(t - 2)
2
1
2
2 2
2t 222- 5
1 1
1
(w(s) ['xx)
B
)2(
(V(
) Vi(t (1))
2
2
2
T(x)
<0
This inequality is equivalent to:
64m
tln +21D( t-1 2 ورا +1 +1++th +1+
(t 2) (-1
2
> 2/7(( - (1) @ || - + v(5) c In 4),
where (i) is derived from Legendre duplication formula. Simplify the above inequality, we get:
64
2 t 6 > 0.
+ 1 + tv
We only need to show
64m
(111 - ) 1m1))
1-1 1
+ 1 + 1V
- 2)249((-1)
ا-ا
[-1 )2
2
2 ?
64T
+2 +(n((t/)2)) 1/(t/2 - +ln(t/2)
(1 - B)
1-1
[-1 )2
2
64
2
(t - 2
1-1
ا-با )2
t-1
2
64
2
t-2 2 0,
(1- 2 ر
2 ?
where the first inequality is from In(x) 1/(2x) > ((r) > In +05) - 1/2. Therefore, we only need to show
32 2 (1- 2)4'B 1 1-1 t -ر11
2 2
which is equivalent to
t t- )² = (1- 2)1² T(
(t (
2 2
M(] - 1)2
I( [((-1)2
2 x =16 - 2) [ [( )2
@ (t- (t-2)4
< 32T
T(t/2)2
r(t/2)2
11

---

Published as a conference paper at ICLR 2020
where (i) is from Legendre duplication formula.
So we only need to show
(t 2) T(1/2)3 2
(9)
[(++1) Using Gautschi's inequality [[x++)]
(xr + 1)²-²), we have
r 2
I (t - 2)( 1 -
2(
(t (t r(t/2)2
<2
(10)
2
t-1
n
B IMPLEMENTATION DETAILS
B.1 LANGUAGE MODELING
Our implementation is based on the previous work (Liu et al. 2018). Specifically, we use two-layer LSTMs with 2048 hidden states with adaptive softmax to conduct experiments the one billion words dataset. Word embedding (random initialized) of 300 dimensions is used as the input and the adaptive softmax incorporated with default setting (cut-off's are Additionally, as pre-processing. we replace all tokens occurring equal or less than 3 times with as UNK, which shrinks the dictionary from 7.9M to 6.4M. Dropout is applied to each layer with a ratio of 0.1, gradients are clipped at 5.0. We use the default hyper-parre parameters to update moving averages, i.e.3, = 0.9 and 32- = 0.999. The learning rate is set to start from 0.001, and decayed at the start of 10th epochs. LSTMs are unrolled for 20 steps without resetting the LSTM states and the batch size is set to 128. All models are trained on one NVIDIA Tesla V100 GPU.
[4000, 40000, 200000)).
B.2 IMAGEINE CLASSIFICATION
We use the default ResNet architectures (He et al. 2016 in public pytorch re-implementation Specifically, we use 20-layer ResNet (9 Basic Blocks) for CIFAR-10 and 18-layer ResNet (8 Basic Blocks) for ImageNet. Batch size is 128 for CIFAR- 10 and 256 for ImageNet. The model is trained for 186 epoches and the learning rate decays at the 81-th and the 122-th epoches by 0.1 on CIFAR- 10, while the model is trained for 90 epoches and the learning rate decays at the 31-th and the 61-th epoch by 0.1 on ImageNet. For Adam and RAdam, we set 3 = 0.9, $2 = 0.999. For SGD, we set the momentum factor as 0.9. The weight decay rate 10 4, Random cropping and random horizontal flipping are applied to training data.
B.3 NEURAL MACHINE TRRANSLATION
Our experiments are based on the default Transformers (Vaswani et al. 2017) implementation from the fairseq package (Ott et al. 2019). Specifically, we use word embedding with 512 dimensions and 6-layer encoder decoder with head and 1024 hidden dimensions on the IWSLT14' dataset; use word embedding with 512 dimension and 6-layer encoder / decoder with 8 heads and 2048 hidden dimensions. Label smoothed cross entropy used as the objective function with an uncertainty = 0.1 (Szegedy et al. 2016). We use linear learning rate decay starting from 3e 3e-4, and the checkpoints of the last 20 epoches are averaged before evaluation. As to the wamrup strategy, we use a linear warmup for Adam in the first 4000 updates, and set 32 to satisfy IWSLT' 14 dataset, we conduct training on one NVIDIA Tesla V100 GPU, set maximum batch size 4000, apply dropout with ratio 0.3, using weight decay of 0.0001 and clip the gradient norm at 25. In the WMT" 16 dataset, we conduct training on four NVIDIA Quadro R8000 GPUs and set maximum batch size as 8196.
4000 (32 0.9995). In the
C DOWNGRADING TO SGDM
As byproduct determined by math derivations, we degenerated RAdam to SGD with momentum in the first several updates. Although this stage only contains several gradient updates, these up-
htt.ar///g.tbb....weererrrrrrrrr assi ficaiion
12

---

Published as a conference paper at ICLR 2020
dates could be quite damaging (e.g., in Figure 2 the gradient distribution is distorted within 10 gradient updates). Intuitively, updates with divergent adaptive learning rate variance could be more damaging than the ones with converged variance, as divergent variance implies more instability. As a case study, we performed experiments on the CIFAR 10 dataset. Five-run average results are sum- marized in Table The optimizer fails get equally reliably model when changing the first 4 updates to Adam, yet the influence of switching is less deleterious when we change 5-8 updates instead. This result verifies our intuition and agreement with our theory the first few updates could be more damaging than later updates. By saying that, we still want to emphasize that this part (downgrading SGDM) is only a minor part of our algorithm design whereas our main focus is on the mechanism of warmup and the derivation of the rectification term.
Table 3: Performance CIFARIO (Ir 0.1).
1-4 steps
5-8 steps
8+ steps
test acc
train loss
train error
RAdam
RAdam
RAdam
91.08
0.021
0.74
Adam (w. divergent var.)
RAdam
RAdam
89.98
0.060
2.12
SGD
Adam (w. convergent var.)
RAdam
90.29
0.038
1.23
13


Reviews:
[{"review_id": "SyeHRMd1cS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I haven't worked in this area before, and my knowledge to the topic of designing optimizers for improving stochastic gradient descent is limited to the level of advanced ML courses at graduate school. Nevertheless, below I try my best to evaluate the technicality of this paper\n\n====================\nIn this work the authors studied the variance issue of the adaptive learning rate and aim to justify the warm-start heuristic. They also demonstrate that  the convergence issue of many of the stochastic gradient descent algorithms is due to large variance induced by the adaptive learning rate in the early stage of training. To tackle this issue, they proposed a variant of ADAM, which is known as rectified ADAM, whose learning rate not only takes the momentum into the account, but it also adapts to the variance of the previous gradient updates.  \n\nIn the first part of the paper, the authors analyzed the variance issue exists in the existing ADAM algorithm, such that with limited samples in the early stage of training, the variance of the adaptive learning rate becomes rather large and it induces high variance to the gradient update to ADAM. In general I found this theoretical justification on the observation of variance issue in ADAM sound, and quite intuitive. In the second part, they proposed the algorithm, namely rectified ADAM, where the difference here to take the second moment  of the gradient into account when updating the adaptive learning rate. They showed that the variance of the adaptive learning rate with such rectification is more numerically stable (especially when variance is intractable in vanilla ADAM), and under some regularity assumption it decreases in the order or O(1/\\rho_t).\n\nIn extensive numerical studies of supervised learning, the authors showed that RADAM achieves a better accuracy than ADAM (although in Table 1, I am a bit puzzled why the best accuracy is indeed from SGD, if so, what's the point of all adaptive learning rates, is that because SGD requires extensive lr tuning?) Because the superiority in accuracy they also showed that RADAM manages to have more stable training and achieves lower training loss, which is quite interesting. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkxsh6bZcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, authors show that the bad performance of Adam is from the large variance of adaptive learning rate at the beginning of the training.\nPros:\n1.\tAuthors demonstrate that the variance of the first few stages is large, which may interpret the degradation in the performance of Adam.\n2.\tThe empirical study supports the claim about the large variance.\n\nCons:\n1.\tTheoretically, authors didn’t illustrate why the large variance can result in the bad performance in terms of, e.g., convergence rate, generalization error, etc.\n2.\tThe performance of the proposed algorithm is still worse than SGD and it makes the analysis less attractive.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxzMg4AYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a way to rectify the variance of the adaptive learning rate (RAdam) and apply the optimizer to applications in image classification, language modeling and neural machine translation. The experiments demonstrate not only a strong results over baseline Adam with warmup learning rate but the robustness of the optimizer. The authors additionally demonstrate the theoretical justification behind their optimizer, however I am not very qualified to make the judgement on the theory. Overall judging from the authors description of approach and experimental results, I recommend acceptance.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers an important topic of the warmup in deep learning, and investigates the problem of the adaptive learning rate. While the paper is somewhat borderline, the reviewers agree that it might be useful to present it to the  ICLR community.
Decision: Accept (Poster)