AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
Authors: Wenjie Li
Zhaoyang Zhang
Xinjiang Wang
Ping Luo
Venue:  (2020)
Keywords: Optimization Algorithm
Machine Learning
Deep Learning
Adam

Abstract: Adaptive optimization algorithms such as RMSProp and Adam have fast convergence and smooth learning process. Despite their successes, they are proven to have non-convergence issue even in convex optimization problems as well as weak performance compared with the first order gradient methods such as stochastic gradient descent (SGD). Several other algorithms, for example AMSGrad and AdaShift, have been proposed to alleviate these issues but only minor effect has been observed. This paper further analyzes the performance of such algorithms in a non-convex setting by extending their non-convergence issue into a simple non-convex case and show that Adam's design of update steps would possibly lead the algorithm to local minimums. To address the above problems, we propose a novel adaptive gradient descent algorithm, named AdaX, which accumulates the long-term past gradient information exponentially. We prove the convergence of AdaX in both convex and non-convex settings. Extensive experiments show that AdaX outperforms Adam in various tasks of computer vision and natural language processing and can catch up with SGD.


Under review as a conference paper at ICLR 2020
ADAX: ADAPTIVE GRADIENT DESCENT WITH EXPO- NENTIAL LONG TERM MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Adaptive optimization algorithms such as RMSProp and Adam have fast con- vergence and smooth learning process. Despite their successes, they are proven to have non-convergence issue even in convex optimization problems as well as weak performance compared with the first order gradient methods such as stochas- tic gradient descent (SGD). Several other algorithms, for example AMSGrad and AdaShift, have been proposed to alleviate these issues but only minor effect has been observed. This paper further analyzes the performance of such algorithms in a non-convex setting by extending their non-convergence issue into a simple non- convex case and show that Adam's 's design of update steps would possibly lead the algorithm to local minimums. To address the above problems, we propose a novel adaptive gradient descent algorithm, named AdaX, which accumulates the long-term past gradient information exponentially. We prove the convergence of AdaX in both convex and non-convex settings. Extensive experiments show that AdaX outperforms Adam in various tasks of computer vision and natural language processing and can catch up with SGD.
INTRODUCTION
In the of deep learning, Stochastic Gradient Descent (SGD), though proposed in the last century, remains the most effective algorithm in training deep neural networks (Robbins & Monro 1951). Many methods have been created accelerate the training process and boost the performance of SGD, such as momentum (Polyak 1964) and Nesterov's acceleration (Nesterov 1983). Recently, adaptive optimization methods have become popular as they adjust parameters' learning rates in different scales, instead of directly controlling the overall step size. These algorithms schedule the learning rates using a weighted average of the past gradients. For example, AdaGrad (Duchi et al.) 2011) chooses the square root of the global average of the past gradients the adaptive learning rates. is shown that when the gradients are sparse or small, AdaGrad can converge faster than vanilla SGD. However, its performance is quite limited in non-sparse settings.
the denominator of
Other adaptive algorithms have been proposed to replace the global average in AdaGrad by us- ing the exponential moving average of past gradients, such RMSProp (Tieleman & Hinton, 2012], AdaDelta (Zeiler) 2012), and Adam (Kingma & Ba 2015). Among all these variants, Adam (Kingma & Ba 2015) is the most popular yet controversial optimization algorithm since has faster convergence rate than the others. However, Adam has worse performance (i.c. generalization ability in testing stage) compared with SGD. Recent theories (Wilson et al. 2017 Reddi al.) 2018) have also shown that Adam suffers from conceevveeeeece issue and weak generalization ability. For ex- ample, Reddi al. (2018) proposed that Adam's convergeeee problem originate from a mistake in their proof of convergence. They constructed counterexample for Adam and thoroughly proved that Adam did not guarantee convergence even in simple convex setting.
In the meantime, Zhou et al. (2019) theoretically proved that Adam could make large update when gradients are small, and a small update when gradients are large, which probably lead the optimization process to wrong directions. SSazzerr & Stern (2018) also empirically showed that Adam's 's parameter updates are not stable and its second moment could be out of date. Luo et al. (201)) examined the effective learning rate of Adam in training and found that Adam would produce too large or too small extreme learning rates. All the above analyses have suggested that Adam's unstable design of adaptive learning rate may impair the optimization process.

---

Under review as a conference paper at ICLR 2020
To address the above issues, propose new adaptive optimization method, termed AdaX, which guarantees convergence in both convex and non-convex settings. This is done by memorizing the long-term square of gradients exponentially as the second moment. To achieve this, we first extend the convex counterexample in Reddi et al. 2018) to non-convex setting. We then analyze how the second moment instability in Adam and R MMPProp could cause the optimization process to converge to local minimum even without noisy gradients, revealing that Adam produces update steps that too large in optimization. We also show how AMSGrad (Reddi et al., 2018) unable solve Adam's 's problem completely. because its effectiveness relies heavily on the magnitude of the maxi- mal second moment, for instance, too large second moment would lead insufficient training and thus sub-optimal solutions. To address the above problems, we introduce novel AdaX algorithm and theoretically prove that converges with similar speed to Adam, but gets rid of the second moment instability. Extensive experiments show that AdaX outperforms Adam and is comparable to SGD with momentum in many tasks of computer vision and natural language processing.
BACKGROUND AND NOTATION
Overview of Adaptive Methods. To compare AdaX with the previous optimization methods, we follow Reddi et al. 2018) to present a generic framework of adaptive algorithms as shown in Algo- rithm Let S be a set of positive definite matrices in Raxd and F be the parameter domain. In line take Φ1 : F Fd and 1i F 8 input, which unspecified moment functions that vary among different optimization algorithms. After obtaining the gradient at time in line 3, chosen
we can calculate the corresponding first and second moment my; V,, In line 5, a2 = a/v as the step size in each iteration for the convergence analysis of adaptive algorithms. The projection operation in line 6, 11 M (y) is defined as argmin 1lv M (z 3) ||. where M E SJ and y E IRd
Algorithm Generic Adaptive Optimization Algorithm
1: Input: F, step size a, sequence of functions (Øe, the ]
2: for to T do
3: Sr V (zz)
4: m, = de 0e(11992 .
9t) and V = (9..9,99.......
5: at = a/ Vt
6: T++1 IT FV aamm///
7:
end for
The main differences between the adaptive methods and the conventional SGD are in line and 6, where the matrix V, scales the overall step size on element-wisely by I/v known as the adaptive learning rate. Using the general framework Algorithm [ we are able to summarize many adaptive optimization algorithms proposed recently. For example, AdaGrad (Duchi et al. (011) designed its V, as the global average of past gradients, while Adam (Kingma & Ba, 2015) and RMSProp Tieleman & Hinton 2012) chose the exponential moving average as follows instead.
82 5 diag
V =
35 9²))
(Adam)
1
where B2 is the fixed second moment coefficient and o² denotes the element-wise square of the gradients. The diagonal operation diag( perform the dimension transformation from R St. In order to improve the performance of Adam, Reddi et al. (2018) proposed AMSGrad, which took max operation on the second moment. Zhou et al. (011 9 argued that we could replace g² in Vi with some past gradient squares oz_n to temporarily remove the correlation between the first and second moment. (Huang et al. (2019) constructed Aosa where sequence of 33x gave higher weights to past gradients. We provide summary of different designs of adaptive learning rate in Table I It's noticeable that these algorithms, due to their exponential moving average design, still assign relatively high weights recent gradients and past information is not emphasized. Besides, Loshchilov & Hutter (2019) noticed the difference between L2 regularization and weight decay in adaptive algorithms and improved Adam with standard weight decay by proposing AdamW.
Optimization Framework. A commonly used framework for analyzing convex optimization al- gorithms constructed by Zinkevich| (2003), named the online optimization problem. In this
2

---

Under review as a conference paper at ICLR 2020
Table I: Comparisons of different designs of the second moment
SGDM
AdaGrad
RMSProp
1/4
I
diag
(5-192/0)
(1 da)diag Ei= 11-110
Adam
AMSGrad
AdaShift
4/+
diag Ei= "
diag (max((+-1.v.))
diag 2/ + (1 B2)gk-n))
NosAdam
AdaX (ours)
4 diag(32:01-11 + (1 820 (20)9(nn)
diag El= (1 (2)3
framework setting, the optimization algorithm chooses parameter set 0, E JF and an unknown cost function fe(O) evaluates its performance at 0g in each iteration. Suppose that there exists a best
parameter f+() such that 6* argminge> (Ei= f((0)): then metric used to show the algo- rithm's 's performance is the regret function Ry = Ei= (+,) ) fe(0") and we want to ensure that Rr = (T) so that the algorithm will always converge to the optimal solution. (Zinkevich) 2003).
Non-convergence of Adam. Reddi et al. (2018) proposed that the matrix I, defined as follows, was mistakenly assumed be positive semi-definite in the original convergence proof of Adam.
Vi+1
VV aq
a+++
where Vú and a, defined in Algorithm [ Based such observation, they constructed the following online convex optimization problem, in which Adam failed to converge to the optimal solution. Let C be fixed constant and {fr} be the sequence of cost functions whose sum is to be minimized. Define fi as follows
f((x) Ca, for & mod 3 = 1 -xx, otherwise
(2)
where x E 1-1, this problem, Adam could not distinguish between the true large gradient direction (C) and the noisy gradient directions (-1) because its vV, scales the gradients be of similar sizes, which forces the algorithm to reach a highly suboptimal solution = 1 every three iterations. However, SGD and AdaGrad are both able counteract the noisy gradients and converge to the optimum, which reveals the fact that Adam's 's design of adaptive learning rate is very unstable.
3 THE NONCONVERGENCE OF ADAM IN A NON-CONVEX SETTING
In this section, we extend the non-convergence problem in (2) to the non-convex setting and explain why the fast convergence of Adam impairs its performance in the long term. Let E (1, +o0), (0, 1) be constants in IR, consider the following simple sequence of non-convex functions ft.
CX vr for x 2 0 C2 for <<
fi(x))
1
(3)
It can be easily observed that for the domain F 1-2, {)], the minimum of each Se is obtained at i 0. Suppose we start from 2i > 0. then this problem simulates situation where the gradient decreases exponentially as time increases, implying that the algorithm is approaching the global minimum and smaller step sizes are needed. Compared with the non-convergence problem proposed by |eddi et al. (2018), high-frequency noise exist in our gradients. However, next to the optimal solution, there a local minimum trap where no gradients exist and thus no algorithm could escape. Let a' a be the initial step size, we able show that SGD capable of avoiding the trap, even without the learning rate decrease a a/vi, and will converge to x 0 initialized well. However, Adam ignores the gradient decrease information and always enters the trap regardless of initialization. We summarize the above results in the following lemma
3

---

Under review as a conference paper at ICLR 2020
Lemma 3.1 In problem 3.. with 61 = 0, $2 (0, and as 2 a/tt, Adam will always reach the o² local minimum, i.e. 1 I-x 1o
We provide all the proofs in the Appendix. This parameter setting of Adam is the same RMSProp except for the bias correction term, and the condition 31 < vB2 mentioned by Kingma & Ba (2015) automatically satisfied. at 2 ax/l is weak requirement for the step sizes and can be ensured with constant step sizes or ar = a/vV as the convergence analysis. Intuitively, Adam would scale the decreasing gradient by 1/vV,, which approximately increases with the same speed. Therefore, its update steps would be larger than a fixed constant at any time step and would ultimately lead parameters the trap regardless initialization. People may wonder whether the first moment design helps Adam in such a situation, but we can show that as long the condition B1 < vB2 is satisfied, Adam would always reach the local minimum. Hence, Adam converges faster than SGD because of its large updates, but it cannot slow down when approaching the global minimum.
AMSGrad is constructed to address the problem Adam's 's large steps during optimization. How- it suffers from two major issues in non-convex situation. 1) The never decreasing Vi in AMSGrad could lead to early stops and therefore insufficient training during optimization, as re- vealed by Huang al. (2019); 2) The time for achieving the maximum of V, uncontrollable. We show that for certain cases in our problem, AMSGrad is unable to help Adam.
Lemma 3.2 In problem 3 with 31 0 and an 2 a/t, VB2 E (0,1), E (V32, 1), such that AMSGrad will always reach the local minimum, i.e. Σ S(1/T + I VI > 0.
The lemma essentially states for any fixed 82, we can find a such that AMSGrad cannot help Adam because V, keeps increasing before stepping into the trap. Therefore, we still need an algorithm that can generate stable learning rates and control the update steps effectively. We emphasize that al- though the functions in (3) are not smooth, the problem does successfully provide some intuition on why Adam variants trains much faster than SGD, but cannot have comparable testing performance.
ALGORITHM AND CONVERGENCE
Next, we introduce our novel optimization algorithm and present its special way of adjusting the adaptive learning rate. Based upon the above discussions that current gradients lead to unstable second moment and that long-term memory algorithms are preferred, we design our algorithm AdaX by weighting exponentially
on the history gradients and less the current gradients, as shown Algorithm? [ The most important differences between AdaX and Adam are in line and 7, where instead of an exponential moving average, we change 32 1 B2 and accumulate the past gradients. Such a design guarantees that noisy and extreme gradients cannot greatly influence the update steps, and ún would gradually become stable. Similar Kingma & Ba (015) derivation, in order to
Algorithm 2 AdaX Algorithm
1: Input: F, step size {o B1, Bg
2: Initialize mg 0, =
3: for = to do
4: 9: fz((
5: m = f1m4-1 + 83)9
6: U (1 B2) + B29t
7:
0; vt/[(1 + 32) - and
and V, = diag(Or) (0x)
8:
Te+1 = II (2t a vo))
9: end for
achieve an unbiased estimate of second moment, we obtain our bias correction term as follows. Let ge be the gradient at timestep i and further suppose gr 's drawn from stationary distribution
4

---

Under review as a conference paper at ICLR 2020
9t p(o)). Take expectation on both sides of line 6 in Algorithm
2 get
B(ee)= = [((1 + b2) + 329
= Σ (1+ + (2) --)
(4)
1=1
[1 + 32)² 1|
To maintain an accurate second moment, we would naturally divide Uy by (1+ 32) - 1 in line 7.
However, it's worth mentioning that we do not include first moment correction term (1 - {1) as
Kingma & Ba 2013) did for the following reason. Consider the Stochastic Gradient Descent with
momentum(SGDMM algorithm and Adam's 's first moment,
my ----1 9t SGDM:
1m1
Adam:
my 31m=11m-11 + (1 B1)gt = (1 (1-11)) B1)
8 9:
can be observed that they have the same form except for the scaling constant 1- B1, and therefore the first order bias correction term is counter intuitive. We change the scaling constant from B1 to 1 - 33 in line in our algorithm to obtain a more general form of the moment expression, and when B3 * $1, it helps scale the hyper-parameters (such step size, weight decay) of adaptive algorithms to the same size as SGD. Next we show that our algorithm ensures the positive semi-definiteness of T and hence does not have the non-convergence issue of Adam. Consider the following lemma which leads to the conclusion that I is positive semi-definite our algortihm,
Lemma 4.1 Algorithm ensures that the matrix
V/2 of
0
Finally, we provide the convergence analysis of our algorithm in both convex and non-convex set- tings. Using the analysis framework by Ziinevichh (2003), the following theorem states that we are able to obtain a regret bound of (vVT), which is the same as the results of Reddi et al. 2018). A domain F is said have bounded diameter if || yll < Doc, Yx, y F for some D E.
Theorem 4.1 Let {at} and {ve} be the sequences obtained from Algorithum 2 a4 = az B1,1 31, & for all I E [T] and 32 = -82/1, = -1//Il Assume that F has bounded diameter Doo and [+)|l Goe for all i [T] and x E F. Then for Ia generated using Algorithm 2 have the following bound on the regret.
D2
D2
aC log T
RT 2aT(1-
T
1/2
Ilu. (5)
B1)
2(1
31)
at
(1 31) 32
The following corollary follows naturally from the above theorem.
Corollary Suppose 31e = -1 in Theorem 4.1 then we have
D2 VT RT 20(1 31) TT 20(1
d3, GoD aCv1 + log (1- 31)3
Σ
Σ 1lg.T.ll2
(6)
Analyzing optimization algorithms in non-convex setting is slightly different from the convex case, where instead of the average regret, stationarity in gradient is utilized to show convergence in time. Following Chen al. (2019). suppose we are minimizing function that satisfies the following three assumptions
A1. f is differentiable and has L-Lipschitz gradient, i.c. Vz, y. [Vf(z) V T||l yll- and f(x") where x* is the optimal solution.
5

---

Under review as a conference paper at ICLR 2020
monde
1000 1000
2012
(a) Training Accuracy for Le Regularization
(b) Testing Accuracy for L Regularization
(c) Training Accuracy for Weight Decay
(d) Testing Accuracy for Weight Decay
Figure Training and Testing Accuracy on CIFAR-10
Method
Top 1 Acc
Method
Top Acc
Adam
90.96 = 0.03
AdamW, wd=le-4
91.86= 0.04
AdaGrad
86.56 = 0.07
AdamW, W. wd=5e-4
92.12 + 0.05
AMSGrad
90.98 + 0.04
SGDM
92.30 + 0.09
RMSProp
89.64 + 0.05
AdaX-W, wd=5e-4(ours)
92.32 + 0.04
AdaX(ours)
91.60 + 0.10
AdaX-W, wd=1e-3(ous)
92.51 0.07
Table 2: Validation accuracy CIFAR-10.
A2. Suppose that the true gradients and the noisy gradients are bounded i.e.
yor S for some G A3. The noisy gradient is unbiased and the noise is independent, i.e. 9e fz(z) + nr B/m] = 0 and is independent of 7) if (fit) j.
(=(()) Goe; || S Goo, VI 2 Also, ||a: my
then we would obtain the following theorem and corollary, which prove that AdaX converges with speed close to AMSGrad as mentioned by Chen et al. (2019).
Theorem 4,2 Let {xr} and {vi} be the sequences obtained from Algorithm 2 a4 a/VL,1111111 31. B1. I 31. for all E [T] and 82z = 32/t, 33r = 31. Assume that [ S Go for all
t [T] and x E F. Then for generated using Algorithm
2
we have the following bound.
Go min E [IV](())]]] a a +T) BEfT
G c²
C2da
Czda² o²
+
+
+ +CA)
where Ci, C2, Cs, C4 are constants independent T
Corollary 4.2 Suppose B2 = 32, with the other assumptions same as in Theorem 4.2 we have
Go avT c²
GG
C2da
Czda²
min E [[V](
<
C4)
(8)
2
EXPERIMENTS
In this section, we evaluate the performance of AdaX on various tasks in comparison with SGD with Nesterov momentum (SGDM), Adam(W), and many other common optimizers. The implementa- tion of AdaX consists two parts, AdaX and AdaX-W, representing using La regularization and standard weight decay in the algorithm respectively discussed in Loshchilov & Hutter (2019). We relegate the detailed implementation of AdaX to the Appendix. We show that AdaX, combined with proper weight decay, is capable of performing better than Adam and SGDM in many tasks.
5.1 CONVOLUTIONAL. NEURAL NETWORK ON CIFAR-10
Using ResNet-20 proposed by He et al. 2016), we verified the performance of AdaX on CIFAR-10 Krizhevsky al. 2009) image classification task. In experiments, utilized learning rate

---

Under review as a conference paper at ICLR 2020
AdaK
Adwit
500
500
13330
20000
203000
40000
300000
130030
2000CE
10300
4X00
300000
(a) Training Top-1 Accuracy on ImageNet
(b) Testing Top-1 Accuracy on ImageNet
Figure 2: Training and Testing Results ImageNet.
(a) Validation accuracy on ImageNct and IoU
(b) Validation perplexity One Billion
VOC2012 Segmentation
Word for language modeling.
Method
ImageNet Top
Method
VOC2012 IoU
Method
Validation PPL
SGDM
69.90
SGDM
76.28
Adam
36.90
AdamW
66.92
AdamW
74.62
AdaX(ours)
35.22
AdaX-W(ours)
69.87
RMSProp
58.70
AMSGrad
73.62
AdaX-W W
76.53
Table 3: Performance of AdaX on ImageNet, VOC2012 Segmentation and One Billion Word
schedule that the initial step size was scaled down by 0.1 and 0.01 at the 100-th and the 150-th epoch. The experimental results in Table actually taking steps that were "too large" and would potentially converge to local minimum at the end.
2 corresponded to our theoretical finding that Adam was
L2 Regularization. We used Le regularization of strength 5e-4 for all the optimizers. As shown in Figure Ta and Tb although AdaX relatively slow at the beginning compared with other optimizers, its testing accuracy quickly caught up with the others after the first learning rate decrease and became the highest (91.6) at last.
Weight Decay. The baseline was trained with SGDM with weight decay 5e-4. Adam with weight decay, named AdamW (Loshchilov & Hutter 2019) also trained for comparisons. Although AdamW W with le-4 weight decay converged much faster than SGDM and AdaX-W, its final accuracy could catch up with the other two (see Figure Ic Id.A A higher weight decay could potentially help AdamW V achieve a better performance (92.1), 1), but it was as slow as SGDM. On the other hand, AdaX-W W with step size rate 0.5 and 5e-4 weight decay converged fast and yielded the same perfor- mance as SGDM(99.322. The best result we obtained was AdaX-W W with step size 0.25 and le-3 weight decay, which resulted in 92.51 Top-1 accuracy, even slightly higher than SGDM.
5.2 CONVOLUUIINNAL NEURAL NETWORK ON IMAGENET
We also conducted experiments to examine the performance of AdaX-W on ImageNet (Deng al., 2009). SGDM, -, and Adam were used to train a ResNet-18 model on ImageNet, with a standard le-4 weight decay and 0.1, 0.5, le-3 step sizes as in CIFAR-I0 respectively. A warm up scheme was applied in the initial 25k iterations (Goyal et al. 2017), and then the step size was multiplied by 0.1 at the 150k, 300k and 450k-th iteration steps. As observed from Figure although AdamW was fast at the beginning of training, its test accuracy stagnated after the second learning rate decrease. AdaX- W. on the other hand, converged faster than SGDM without loss of testing accuracy (69.87), ),, as shown in Table 3a
7

---

Under review as a conference paper at ICLR 2020
Auce
480
3 100
14
(a) Training Dynamies on One Billion Word.
(b) Training Loss on VOC2012 Segmentation
(c) Testing IoU
Figure 3: (a) Traning loss curves for Adam and AdaX on One Billion Word. (b, c) Training Loss and Testing Results on VOC2012 Segmentation task. In (c), dashed lines mean accuracy values and solid lines are Intersection over Union (IoU) values
5.3 RECURRENT NEURAL NETWORK ON LANGUAGE MODELING
AdaX has also been validated on One Billion Word (Chelba et al. 2013) dataset of language mod- eling task. For the One Billion Word, we used a two-layer LSTMs with 2048 hidden states and sam- pled softmax. The global experiment settings in the released public code Rdspring] was adopted in this study. For both vanilla Adam and AdaX, the LSTMs were trained using for 5 epochs, with learning rate decaying to 1e-8 linearly. Similarly, the weight decay for AdaX was set to 0. Note that the regular SGD is not suitable in this task, so it is not included in the comparison.
The training loss and the validation perplexity is shown in Figure 3a and Table 3b We the AdaX outperforms the Adam baseline by a significant margin (35.22 vs. 36.90). Moreover, similar the effect on image classification tasks described above, AdaX starts little slower the early stage, but it soon surpasses Adam on both training and validation performance.
that
5.4 TRANSFER LEARNING
To further examine the robustness of AdaX in transfer learnings such as semantic segmentation, we evaluated its performance on the PASCAL VOC2012 augmented dataset (Everingham et al. 2014) (Hariharan et al. 2011). The classic Deeplab-ASPP model (Chen et al. 2016) was adopted with ResNet-101 backbone pretrained on MS-COCO dataset(Lin et al. 2014 Adaptive methods are seldom used semantic segmentation tasks because of their bad performances, but AdaX can surprisingly be applied to train these models as well. The initial step sizes for different algortithms was recorded section A.7 and all other parameters stuck to the default setting in Chen et al. (211)). We evaluated the algorithms' performances at the 5k, 10k, 15k and 20k iterations using intersection over union (IoU). As can be observed in Figure 3c AdaX-W trained faster than SGDM and obtained higher loU (76.5) at the same time. On the other hand, Adam W not capable of obtaining comparable results.
5.5 COMPARISON OF SECOND MOMENT DESIGN
Finally, we compared the second moment design of Adam and AdaX empirically and showed the instability of Adam's 's second moment could have large impact on its performance while our design was stable and robust. We first evaluated the performance of different algorithms in our synthetic
example (3) The problem parameters were set to be C = 1e 3, X 0.9999, = 1. To ensure fair comparisons, default hyperparameters were chosen for all the algorithms, specifically ag = 0.1, y 0.9 for SGDM, ag le 3, 31 0.9, 32 0.999 for Adam, and ag 0.15, 81 0.9, B2 = le - 4, 33 = 0.999 for AdaX. As shown Figure 4a and 4bb SGD and AdaX quickly converged under the strong gradient decrease information and could potentially reach the global minimum if initialized well. However, the update steps of Adam decreased with a much slower rate, which resulted in substantial changes in and ultimately lead the algorithm to the local minimum.
Zaheer et al. (2018) found that Adam's 's performance could be affected by different values of e, which was originally designed to avoid zeros in the denominator. Such finding revealed the instability
8

---

Under review as a conference paper at ICLR 2020
20000
(a) Updates over Itera- tions
(b) x- e--osition over Iter- ations
(c) Training Accuracy
(d) Testing Accuracy for Different values of
for Different values of e
Figure 4: Second momentum stability comparisons. (a).(b) Training updates and x- x-positions on
Problem G). (c).(d) Training and testing results of AdamW and AdaX-W W with different e's
of Adam's second moment. We verified their claim by testing Adam with different values of {le 8, 1e 5, 1e 3} the CIFAR-10 dataset. We found that larger values of c, since it helped to stabilize very small second moment, did improve Adam W's performance by around 0.35 percent accuracy
CIFAR-10. However, AdaX-W's performance was not affected by different choices of as shown in Figure 4d) which again proved our claim that a long term memory design was more stable than the exponential moving average in Adam.
The experiments shown above verify the effectiveness of AdaX, showing that the accumulated long- term past gradient information can enhance the model performance, by getting rid of the second moment instability in vanilla Adam. It is also worth noticing that the computational cost for each step of AdaX and Adam are approximately the same, as they both memorize the first and second momentum in the past. Therefore AdaX enables one to get higher performance than Adam in various tasks using the same training budget.
CONCLUSION
In this paper, present novel optimization algorithm named AdaX in order to improve the per- formance of traditional adaptive methods. We first extend the non-convergence issue of Adam to non-convex case, and show that Adam's fast convergence impairs its convergence. We then pro- pose our variant of Adam, analyze its convergence, and evaluate its performance on various learning tasks. Our theoretical analysis and experimental results both show that AdaX is more stable and performs better than Adam in various tasks. In the future, more experiments still need to be per- formed to evaluate the overall performance of AdaX and AdaX-W. Moreover, our paper is first step into designing adaptive learning rates in ways different from simple and exponential average methods. Other new and interesting designs should also examined. We believe that new adap- tive algorithms that outperform AdaX in convergence rate and performance still exist and remain to explore.
REFERENCES
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint 31X1v:131..3005, 2013.
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Muchine Intelligence, 40:834- 848. 2016.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithm for non-convex optimization. Proceedings of 7th International Conference on Learning Representatioos((ILRR) 2019.

---

Under review as a conference paper at ICLR 2020
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. in 2009 ieee conference on computer vision and pattern recognition. IEEE, 40:248255, 2009.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (JMLR), pp. 12:2121-2159, 2011.
Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision(IJCV), 2014.
Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: training imagenet in hour. arXiv preprint arXiv: 777600677777. 2017.
Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. International Conference of Computer Vision(ICCV), 2011.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CromnnCCO (CVPR), 2016.
Haiwen Huang, Chang Wang, and Bin Dong. Nostalgic adam: Weighting more of the past gradients when designing the adaptive learning rate. arXiv preprint arXiv: 1805.07557, 2019.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). 2009.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs//105500 0.15..03.22002 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Proceedings of 7th International Conference Learning Representations (ICLR), 2019.
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. Proceedings 7th International Conference on Learning Representations, 2019.
7th
Brendan Mcmahan and Matthew Streeter. Adaptive bound optimization for online convex opti- mization. Proceedings of the 23rd Annual Conference On Learning Theory (COLT), pp. 244-256, 2010.
Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence o(1/k²), Doklady AN USSR, pp. (269), 543-547, 1983.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. USSR Compu- lational Mathematics and Mathemutical Physics, pp. 4(5): 1964.
Rdspring1. Pytorch gbw Im.
CBW LM
Sashank J. Reddi, Stayen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. Pro- ceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, pp. 2203)-00--477 1951.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv: 1804.04235), 2018.
10

---

Under review as a conference paper at ICLR 2020
Tijmen Tieleman and Geoffrey Hinton. Rmsprop:
Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, pp. 3(2):26-31, 2012
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances in Neural Information Pro- cessing Systems 30, pp. 4148-4158, 2017.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive meth- ods for nonconvex optimization. Advances in Neural Information Processing Systems 31, 2018.
Matthew D. Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang, Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence of adaptive learning rate methods. Proceedings of 7th International Conference on Learning Representations (ICLR), 2019.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. lernational Conference on Machine Learning (ICML), 2003.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 PROOFS OF LEMMA 3.1 LEMMA 3.2
We consider a one dimensional non-convex case where {ft are sequence linear functions that have decreasing gradients in the long term. We want to show that because Adam trusts its current gradient as the second moment, its step sizes are too large and the algorithm would converge suboptimal solution. Let constant C be the initial gradient, define cost function ft as follows:
CX for x 2 0
f((x)
for æ < 0 v4 2
where À is the decreasing factor of gradient. Consider JF - 1-2, the minimum regret is obtained at x O. Let the initial step size be 01 a, we then consider the performances of different algorithms in this setting.
[-2, C/(1 >)], then it's obvious that
(SGD). We first show that without the momentum, vanilla SGD is able to converge to the optimum solution and avoid the trap. Take derivative with respect x, we obtain that
Vf((2) = Cxt-1 for I 2 0
80
C
(10)
Vf(()) = Cx - 1-X
1=1
Therefore, even if we set 0r = a, V1 > 1, as long the initial point 2 oC
SGD is able to
x
avoid the trap. If a/vi, then the condition can be even less strict: able converge the optimum if the equal signs are true.
oc aCA'=1 SGD
Vt.
(Adam). We consider the Adam algorithm with the following parameter setting:
31 0,0 /B2 X A 1, and az
(11)
VE
Note that this parameter setting of Adam is the same as RMSProp, but we can further show that even 31 # we still obtain similar results. Consider how 8y changes in time, before it reaches the negative region, the gradients are positive and
1t = 12 (1- - 12)
Σ 1=1
(1 3ox
(1 32
X2
(12)
X²
32
i=1
(1
C2 83)
X2 B2
Note that À
33, therefore the update rule is:
/x2 32A-11
/X2 - 32A-1
gt
T+41
F4
Ot
Ft
02
Ft 0t
0tt
(x
-B²)
B2) - -(2)))
(13)
X2 - B2
S x4 Vt
X²(1 (32)
12

---

Under review as a conference paper at ICLR 2020
Note that the series Lt=1 vi diverges, hence Adam would always reach the negative region. Same argument applies as long as a 2 a/t. We would emphasize here that the bias correction term in 32 2 B2 and therefore the update steps still
Adam does not change the final result bounded. We could further show that when B1 * 0, 0.31 < vBz, Adam will still go to the negative
region. Since m 81m4-1 + (1 - B1)g. = E (1 - 31)
Jt. therefore
mh V
(1 - 31)(A' B1)
X2 B2
- 31
V(1 822 - 132
(1 Bi) - 88
N 31
(A 81 -82)
V - 32
(14)
(1 81) 32
1 (3)
(A BivVI B2
VI (B2/X3)
(1 3:ly X 32
31
(1
(1 Bi)v1 B2
Since the update steps lower bounded, the algorithm would still go to the negative region. (AMSGrad).. We now evaluate the performance AMSGrad in our formulated problem. Note that Vy in AMSGrad would take the same form Adam, and di max(e:] We suppose that the maximum obtained at as an example, then
(1
B2) (A2 82)
=11 C2
X2
32
(15)
1'11
9t
9t
Voe
V
v1
B2
As see, AMSGrad partially solves the problem of Adam and restores the gradient decrease information as its 14 is lower bounded. the maximum of 1 is obtained before the parameters enter the trap, AMSGrad could possibly have better performance in this problem as prevents the update steps from being too large. However, one important determining factor is the time when the maximum value is obtained. If 0, in fact keeps increasing before very large number T, then AMSGrad would have the same performance Adam. We explain the above intuition as follows. Let h()) = 2 3r, then
dh(t) 2 2 In B2 -32
(16)
dh(t))
If
0, have
dt
In 2
In 32 B8 2 0
X2 ) - B2
In 32
In A
When the equal sign true, we have
In B3
In In da
log 2a
0g
In X
In X
(n(2
(17)
lim
lim
82
92 +>2
In(
lim {
X-+1
13

---

Under review as a conference paper at ICLR 2020
The first equal sign in the first limit is due to L'Hospital's rule. Therefore, the value of T where X = max depends on the difference between 32 and X2, and the value of X. If B2 is close to 2 or À is close to I, then e2 needs large number of steps to obtain the maximum. In such cases, AMSGrad may not able to help Adam. Specifically, for a fixed 92, since limx-11 t co. and
X2 32 g = 12 - B2) vI
32 X
(18)
we know that larger x will lead to both larger update steps and larger when the maximum is obtained, hence YE E (0. 1), such that AMSGrad cannot avoid the trap.
(AdaX). We provide the performance of AdaX in this problem for completeness. We only show for the case when B1 83 O, but the same results hold when the first order momentum is used.
Σ 82(1 +
0 =
(1 + 32)
(1 + 32
X
Σ
(1 + 82)² -1
1+ B2 1=1
(19)
(1 + B2
1- (1
(1 + 32) -1 1+ 1+ X
0₂0
(1+ (1+) -
1+ B2 X²
(1+ B2)' -1
+
32
X2
X-1 1(1 B2)
1 B2
X2
(20)
12
B2
((1 32) X
82
As we can see, AdaX successfully restores the gradient decrease information and controls the de- crease speed by almost fixed parameter instead of trusting the gradient a certain moment, and is therefore expected to perform better than AMSGrad since its step sizes are not affected by extreme gradients. With suitable initial step size and starting point, AdaX is able to avoid the designed trap.
A.2
PROOF OF LEMMA 4.1
Proof.
Vi
E (1 12)2))--100 (1) B29
2 =
a
a
- 22)=-1 (1+ B2)²
El
(1 + 32)
a
(1+ 32)'-(11 33'--1+10 B2)
(21)
Σ
(1+ 022 -320
a
(1 f2) (1 B2)
(1+ 83) -- B299
V2-1
a2
(1+ 2-
07_1
where in the first inequality we utilize the fact that (1 32) 1 + tB2 and hence (
[-1 (1+0)) Intuitively, it is easier this inequality if we simply let B2 to be small number such le-4 in our implementation, then the denominator doesn't change much while the numerator decreases.
A.3 AUXILLARY LEMMAS FOR CONVERGENCE ANALYSIS
Lemma A.1 Assume that 321 32, .2c Bolt, with B2 E (0, and 0, [(1 + (:20) + 32g9 +11++000000 Bz) - 1], V₁ = diag(Et), then we have V/ 2 a
V
14

---

Under review as a conference paper at ICLR 2020
Proof: Similar to Lemma 4.1 in the algorithm section, have
V2
EE
82il1 k=1 (1 + B2(1-R
1==
a
(1+ + $2x)
Σ
L
$1 (1 + 1-k+1 ggh
1
(1+ -1
Σ> 1
1-1
11 (1+
8
19
7
1- x+1
(22)
2
(1+ 22
Σ
i=1
t-|
-1 +
9 09
a
(1 4) 11-11 (1+ )
1 1-++
)g
V1-1
(1 + A y-1
07
The first inequality comes from deleting the last term fact:
82 and second one comes from the following
(1 + B2 B2ye-1 - (1+
) -
B2 g--=1
= (1+
t+ 82
1
32 2
11 1
82
B2
$ +
B2 +
...
+
1-
t B2
L 1)
32
('' 1
(8)
B2 r
)
82
B2
=1+ B2
2
1
+ B2
(-1)
32
1 1
B2 1-1 j-11-1
$1+ 32
2
f
B2
(1
|11 -1
(23)
Therefore the positive semi-definiteness is satisfied.
Lemma A.2 For the parameter settings and conditions assumed in Theorem 4.1
we have
Σ
11c0kt
1 1/4 m,
11
aCv1+ log T
Σ
1lg.
(24)
2(1 31e)
(1 31 /32
132
1=1
Proof: We first analyze with the following process directly from the update rules, note that
T
Σ 11 - Bay 31 B1er - k+1)9h
MT,
1=1
(25)
Σ
12 T (1 (1+
(1 + 32r
T
j=1
15

---

Under review as a conference paper at ICLR 2020
04 V 1/4
m 1||
2
ar(1 -,,)llV,"
1/4 m4-1 112 QT
M
UT,
d
( Ei=(( BS, MI Bie k+ 1196,)2
-1 Σ ar(1 - 1/4 -m- -a1 (12.27) -1 Σ
T T 1s 7 (1 + -
i=1
T-1
Σ 1=1
ar(1 - VV
1/4 M 112 + /a/1 +2T)
T
- (1(T- {- Bz,) 211 k= 31 x+1)9)))
j=1
ETEI= $23 1=110 (1 + 82(T k+1) 103
ET of - ) /111 - Bag 235) 93,
ar(1 - -io0ll, 1/4 m 112 + +ay
(1 aa
-- -
TE T Bog 1 (1 + 121T -x+1) )o3
i=1
Σ Σ (1 - Ba,) 188'11333 93,5
m 11 @v 0/(11 (1+ + B2T)T -1
ar(l 3, 1/4
(1- B12TT
2=== I T k=1 (1 + BB(TT- - 0
T-1
m-- 112 av /1 + BOT)" -1 (1 /12T
1 B $
Σ
ac(1 V, 1/4
v1
1 (1 + BBCT -2(T-RR1))9
aC
T Σ =1
ac(l - BiollV, 1/4 m1-11
(1 BI
B
(26)
where the first inequality is due to an application of Cauchy-Schwarz inequality. The second incqual-
ity is due to the fact that 31e S B1, Vt. The third inequality follows from Σ311 38--3 < 1/(1 - 31) and the fact that - Bag < 1. The fourth one comes from only keeping one of the positive terms in the denominator. The final one is from the fact that v(1+ BOr)² - < C' for constant C Ve - T. By using induction all the terms in the equation, we are able to further bound it.
j
T
T
aC
T aC
d
Σ VillV, 1/4
Ig.al
=
Σ
199551
v
BB2
BiVBB2
32
1=1
1=1
i=1
1=1
i=1
j=
aC
T
T
gi-t
aC
d
T
T 1
g
19t,i
(1 (1- B1
VJ
(1 -
Vt
j=t
i=1 t=1
j=t
aC
d
T
aC
T
T
<
Σ
19r,e12
11 B1)
2
Vt (1 - b1) VB
t=1
aC
d
T
< aC log
T
T
9t.4
gt,s
3
Vi (1- V3
(27)
The second equality follows from ar-arane of sum order. The third inequality comes from the fact
that Σ 8/ < 1/(1-31) < 1/(1-31). The fourth inequality is again from the Cauchy-Schwan inequality and the final one is from the harmonic sum Σt S (1 log T). By denoting I1at, be
16

---

Under review as a conference paper at ICLR 2020
the vector of the past graidents from time to f in the i-th dimension, i... 91.t,1 we complete the proof the lemma.
19 92.6
It.a]
Lemma A.3 For the parameter settings and conditions assumed in Theorem
we have
D2
d
IV 1/21/((0 (z. - x-) 11V, (5++1 - -x x0)1
T
1/2
(28)
200T
1/2
8//2
Proof: Using the definition of L2 norm, by Lemma A.1
since
a2
cx-1
T
Σ
VV |/ll (xz x²)1 I1 I/
(1++1 x 111
a
T /111 (21 - x²)11
[ |/2/4 (xx - x11)12
|1 (x++1 - 112
1/4
Σ
Q6-1
1=
1/2
1/2
1/2
1.1 (x1,1 -x,)²
(Feti - =11)²
02 2-1 (Ni,i - xx?)
(29)
01 1=1
A= i=1
1/2
1,11 (I11,
1/2 (21, 10,,,-1711 xx3)² +
f-1,
(It,1 xr,,²
a1
at-1
D2
d
Σ
T/
20T
1=1
where the first inequality is from separating the first term and getting rid of the last negative term in the summation. The last inequality from a telescopic summation and the diameter bound that
|| x"1l DDD. Doo
A.4 PROOF OF REGRET BOUND
A.4.1 PROOF THEOREM 4.J
Proof. Following the proof given by Reddi et al. (2018), we provide the proof of regret bound in Theorem 4.1. Beginning with the definition the projection operation II H.,, we have the observation
Te+1 = FI, F, ar V, m.) = min
FEF |V 1//1/((x (x (x: - a.VVV 1/2 1/-m
(30)
Using Lemma 4 in Reddi al. (2018) proved by Mcmahan Streeter (010)) with direct substitute of 21 = (x+ Vn 1/2 mc), Q 1/2 and 22 x= for x² E F, the following inequality holds:
[V; =2)ll²= ||V. 1/4 (xz41 =²)
(z. a.V. 1/2 m, )12
1/2m,
IIV." (xx ==) + of V2 1/4 mm - 20, (ate x="
= LIV./((xx x=) x²) + at VV 1/4
mlll (2 + (1 11--in Bre)g, (x+ - x²))
(31)
where the first equality is due to the fact that II, II [=(==(()) = x² Rearrange the last inequality, we obtain
17

---

Under review as a conference paper at ICLR 2020
(1 - Kgr (2, -x")) - 204
||, (Ie - x²) z²][1= ( + at
V 1/4
me
2
B11 (14-11 1, (I) - xx'))
-
|
||v 1/4 (xx - -xx) x21]|² [1V
(Ie+1 - x² 2²)]3]
+
//2
at
V 1/4 mell
20x+
2
[V 1/4 m +
Sie 11 2x))"
+ 2
2ay
(32)
The second inequality comes from applications of Cauchy-Schwarz and Young's inequality. We now make use of the approach of bounding the regret using convexify of ft in Kingma & Ba 2015). Following Lemma A.2 and Lemma A.3 we have
Σ [((x)) - fe(x)) S Σ Le, (xt x²))
T
11V 1/1 (xt- x²)]
xz))1²] -IVe//((x4411
11V 1//14 (1++1
+ 2(1 B1.)
at
11,
20r(1 - 31c)
1m]
+ 2(1
Biras
I1V,
1/4 1/111mmmmm mm- 112
B1e
11 11211/4(1 (xx x²²)1²
B1r)
200
-B1e)
Dx22
UT
1/2
+
T
B1c
1/4 (Ete - x11
aCVI+ log T
Σ
1lg.zll
20T (1 -B1)
20x
B1)
(1- B1) /32
iml
t=1
im
Doc2
T
d
aCVI+ log T B1)3 32
1/2
Bio(ze,1 - x;))
21 -B1)
UT
+
201
B1)
1=1
t=1
i=1
Doo2
D2
31001/2
1/22
aCVT+ log T
1/2
$ 220 (1 - B1)
UT. +
2(1 - 31)
04
(1 31)3/33
Dx22
D3
T
d
1/2
aCvI+ logT
Σ 1gr.,ll2
TT
+
20rT (1 B1)
2(1 31
(1- 31) 13)3..3. /
1=1
(33)
A.4.2 PROOF OF COROLLARY 4.1
Proof. We first take a look at the size of 11/2
11 note that (V)llo S Goo
60.5
B2g
IT 1
82(0-
(1+
B20
1=1
G2
B2
32
(1 +
8n j=l
j
1-k+1
(34)
< 2
+
t- k +1
1=1
G Σ
t+1
102
10 +1)
The first inequality is due to the fact that (1 + 322 (1 + 32) and the gradient bound. The second inequality follows from 32 < 1. The last inequality is from the telescopic sum. Then we have the following inequality,
18

---

Under review as a conference paper at ICLR 2020
1/2
S dG
B1
Σ
dG
(35)
04
a(1
212
The second inequality is due to the arithmetic geometric series sum Σt= X reason is as follows
(1-1)a the
SS= x0 + 2X + + AS x +222 EX*
(36)
(1 - A)S X + 1 ex S + + < 1-
(37)
Therefore we have the following regret bound
D
d3 13,D2.G Go
+ aCVI+ log T
RT S 2a(1 B1)
1/2 + T
(38)
2x(1 - Bi)1- X)2
(1- B1) 3133D V3
A.5 PROOF OF NON-CONVEX CONVERGENCE
A.5.1 PROOF THEOREM 4.2
Proof. We first directly refer to the original paper and obtain the following bound (Chen et al. 2019).
E
V/t.)/
E
laggt/ vel 2 + C2
01-1 0
C3
2
C
Ut
(39)
where C1 C2 C3 are constants independent of and T. C4 is a constant independent of T. For the first term, assume that minje(d (V01) 2 > 0, we have
C1G
E
C
/t /
Vel
SE
C1
lage// cll2
E
C1
c²
(40)
The second inequality is from the gradient bound | f(x))) I Goo. For the second term with Cz, similarly by the positive semi-
where the first inequality follows from Lemma A.1
as
f
of
definiteness in Lemma A.I we have
T
at-1
E
2
E
C2
(V0t-1
01-1
(voe)
1=2
1=1
2=2
(41)
On
OC
d
on
C2da
E
C2
S
C2
(Vo1);
(T
(v
Ui);
j=1
j=1
The second equality is from the telescope and for the third term
19

---

Under review as a conference paper at ICLR 2020
T
Cyda 22
C3
at11
|
<
R
C3
a1-1
(42)
where the first inequality is because previous inequality with second term. Hence in summary, we have
S e
and the last one is due to the
-
v
G a²
C2da +
Cada²
E
Σ on
S
C4
(43)
Now by the proof of Corollary 4.1 we have an upper bound (de) S tG therefore we can bound the effective step sizes,
Ott
(44)
(vor))
10
And thus we have
T
T 1
E
Σ (T)())).V/(()///))))))
2 E
Σ
a min (xx))) EE(T]
Σ
7
10
Go
min (E
V((x)) log(1 + T)
Go
(45)
The last inequality is due to the fact that E 1/1 2
H 1/td = log(1 + T), therefore we have
T+1
Gx log(1 + T)
C1G a
C2da
Csda² + 2
min
E [(+))
C2,)
(46)
LE(T]
We would emphasize that the assumption | V S G in the theorem is automatically satis- fied as
ar
<
e1
Hence /lalm//vlll & aGx
VeT
A.5.2 PROOF OF COROLLARY 4.2
Similar to the proof of Theorem 4.2 we still have
C2do
Cıda²
E ar (V)
+ ²
Ca
(47)
Note that (6t), has the following upper bound as [f)((l) S Goo.
01 (1+ 32)
Σ
82(1 + 32)-
j=l
(48)
G2
< (1 B2) -
Σ
82 + 32) - G2
1=1
And thus have
20

---

Under review as a conference paper at ICLR 2020
T
T a
T
Σ
a (VS(x))
V/(r://va))
2 E
[V)()))
2 G EE[
min E
BiV
Σ
ViG
Vi
IV
min tE/2
[[Vf(z;)]
VT
(49)
where the last inequality is by the fact that Σ
1
VT, therefore we have
V
Gx
G.
Czda² C2da + 2 + C4)
min B [[V[[[I <
(50)
[
ovT
A.6 IMPLEMENTATION ADAX AND ADAX-W
The detailed implementations of AdaX and AdaX-W robust with respect to the value 32, but we recommend smaller values such as le - 4, le reduce computational cost. The value of A3 be 0.9 match the first moment design of Adam, consequently we need 100 step sizes. The size of weight decay should also be adjusted respectively.
follows. The performance of AdaX is
Algorithm 3 AdaX Algorithm with Le Regularization and Weight Decay
Input: F, step size {ar)?= (31, (3) = (0.9, le 0.999), weight decay À 5c 4, le- 12
2: Initialize mi 0, v0 0
3: for t 1 to T do
4: 9: = fz(( + Ax
5: my 31m4-1 + (1 33
6:
Vt (1+ U2) B2gt
7:
0 0t/((1 + 82) - and V, = diag(0;) (i:)
8:
F++1 f. J.VV (xt (mme (vo + c) Are )
9: end for
A.7 SUPER-PARAMETER TUNING IN THE EXPERIMENTS
The hyperparameters in different algorithms have huge impact their performances in the ex- periments. To efficiently find the best step sizes, we follow Wilson et al. (2017) to perform logarithmically-speeed grid search of the optimal step sizes and list the step sizes we tried in the following tables, where the step sizes bold are the ones with best performances and used in the experiments section.
Step size: Image Classification
SGD(M) (10, 1, 1e-1, le-2, le-3)
Adam/Adam (1e-2, 3e-3, 1e-3, 3e-4, le-4)
AdaGrad [5e-2, 1e-2, 5e-3, le-3, 5e-4]
AMSGrad [le-2, 3e-3. 1e-3, 3e-4, lc-4)
RMSProp (1e-2, Se-3,le,, 5e-4, le-4)
AdaX(ours) (5e-1, 2.5e-1, 2e-1, 1.5e-1, le-1, 1e-2, Se-3, le-3)
AdaX-W(ours) (1,5e-1, 4c-1, 3e-1, 2.5e-1(CIEAR), 2.5. le-1, 1e-2, 5e-3, le-3)
21

---

Under review as a conference paper at ICLR 2020
Step size: VOC2012 Segmentation
SGD(M) (le-3, 5e-4, 2.5e-4, lc-4, 5e-5}
5e-4,
AdamW (5e-4, le-5, 5e-5, 1e-6, 5e-7]
AMSGrad (5e-4, 1e-5, 5e-5, 1e-6, 5e-7}
RMSProp (le-4, 5e-5,
5e-5, 1e-5, 5c-6, 1e-6}
AdaX-W(ours) (le-2, 5e-3, 1e-3, 5e-4, le-4)
Step size: Billionwords
SGD (30, 20, 10, 5, 1, 1e-1, le-2
Adam {5e-3, 2e-3, le-3, 5e-4, le-4)
AdaX(ours) (5e-3, 2e-3, le-3, 5e4, 1e-43 (0.5, 1, 15, 25, 50, 100)
Momentum parameters. For the momentum parameters of Adam, RMSProp and AMSGrad, we tuned over (31,82) = (0.99, 0.999), (0.99. 0.999), (0.99, 0.99 0.9999)} and found that the default values (0.9, 0.999) in Kingma & Ba (2015) yield the best result. For the momentum param- eters (81. $2, 33) in AdaX, we directly applied 81 = 0.9 as in Adam and we tuned B2 over {le 3,
le 5). We found that the value of B2 didn't affect the general performance of AdaX. Tuning 33 is the same as tunning the step sizes, as we will show in the next subsection.
For the other hyperparameters such as weight decay and batch size, we directly apply the same settings as in the baselines (He et al. [Chnn et al., (016))Rdsprin)) (Loshchilov & Hutter 2019).
A.8 THE EFFECT $3
As we mentioned in section 4 and 5, adding new hyper-parameter 83 does not influence the first momentum sum AdaX as is just a constant common factor that can be extracted out as follows.
m, 81m-- (1 (11-3a). Ba)g:
33)
B'' 9
(51)
Therefore, tuning the hyperparameter 33 will have exactly the same effect as tuning the step size a. As long as (1 - 3a)a remains the same, the performance of AdaX doesn't change. We also show this fact empirically using different combinations of (a, B3) on the CIFAR-10 dataset.
Milro wilh mo
10099
20000
50000
18930 20000
31030
50000
(a) Training Top-1 Accuracy
(b) Testing Top-1 Accuracy
Figure 5: Training and Testing Results on CIFAR-10 with Different combinations of (a, 2z).
As shown above, we trained the ResNet-20 model (He et al. 2016) on the CIFAR-10 dataset with different combinations of and Bg from {(0.5, 0.999), (0.05, 0.99), (0.005, 0.9)). However, the
22

---

Under review as a conference paper at ICLR 2020
value of (1- - Ba)a is kept to be 5e-4. It can be easily observed that they have approximately the same training and testing curves, as well as similar final performances, which proves our claim.
23


Reviews:
[{"review_id": "rkeCnYcy9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new adaptive gradient descent algorithm with exponential long term memory. The authors analyzed the non-convergence issue in Adam into a simple non-convex case. The authors also presented the convergence of the proposed AdaX in both convex and non-convex settings.\n\n- The proposed algorithm revisited the non-convergence issue in Adam and proposed a new algorithm design to try to address this issue. However, the new algorithm design is a bit strange to me, especially in Line 6 of Algorithm 2, the authors proposed to update v_t by (1+ \\beta_2) v_{t-1} + \\beta_2 g_t^2, where normally people would use (1-\\beta2) and \\beta2 as the coefficients. I am not quite get the intuition of using such as strange design. I wonder if the authors could further explain that.\n\n- The authors also add change \\beta_1 in Line 5 of Algorithm 2 into \\beta_3. And then in theory, the authors again choose \\beta_3 as \\beta_1. It seems that \\beta_3 is not contributing to any theoretical result. It seems to me to just have another parameter to tune in order to get better performances. Can the authors gives more justification on why introducing such a term here? And also show how the different choice of \\beta_3 affects the final result? \n\n- Missing some important references closely related to this paper:\n\nChen, Jinghui, and Quanquan Gu. \"Closing the generalization gap of adaptive gradient methods in training deep neural networks.\" arXiv preprint arXiv:1806.06763 (2018).\nZaheer, Manzil, et al. \"Adaptive methods for nonconvex optimization.\" Advances in Neural Information Processing Systems. 2018.\nZhou, Dongruo, et al. \"On the convergence of adaptive gradient methods for nonconvex optimization.\" arXiv preprint arXiv:1808.05671 (2018).\n\nI would suggest the authors to also compare with the above mentioned baselines to better demonstrate its performances and theoretical results.\n\n- The authors include theoretical analysis in both convex and non-convex settings, which is appreciated, however, the theoretical result seems to show similar convergence guarantees with AMSGrad. I wonder if the authors could provide theoretical justifications on why the proposed method is better than prior arts, probably some sharper convergences or some generalization guarantees?\n\n- In the experiments part, I wonder why the authors did not compare with AMSGrad, RMSProp in later parts such as ImageNet, IoU and RNN parts? I makes no sense to drop them for those experiments. Also, are the authors fully tuned the hyper-parameters for other baselines such as step size and weight decay on SGDM?\n\n================\nafter the rebuttal\n\nI thank the authors for their response but I still feel that the intuition of this paper is not clear enough and comparison with more baselines is needed. Therefore I decided to keep my score unchanged.  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylElFr-cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out that existing adaptive methods (especially for the methods designing second-order momentum estimates in a exponentially moving average fashion) do not consider gradient decrease information and this might lead to suboptimal convergences via simple non-cvx toy example. \nBased on this observation, the authors provide a novel optimization algorithm for long-term memory of past gradients by modifying second-order momentum design. Also, they provide aconvex regret analysis and convergence analysis for non-convex optimization. Finally, the authors evaluate their methods on various deep learning problems.\n\nSignificance/Novelty: While there have been many studies on non-convergence of Adam, raising an issue on ignoring the gradient decrease information seems novel. \n\nPros:\n1. The motivating toy example in Section 3 is useful for readers to get intuitions.\n\n2. By introducing long-term memory on past-gradients, the authors fix the Adam's issues and they can also improve the convergence rate in a non-convex optimization (Corollary 4.2).\n\n3. Empirical studies show superiority to original Adam (Section 5).\n\nCons:\nWhile they provide a significant study on Adam's failure and a novel optimization algorithm, I have several concerns:\n1. What is default hyperparameters for AdaX in Algorithm 1? Is it the same as AdaX-W (Algorithm 3) in Appendix? The bias correction term in the line 7 of Algorithm 1 will be very large even with small $\\beta_2$ since it is expoential (For example, (1 + 0.0001)^(100000) ~ 20000 for $\\beta_2$ = 10^(-4)). So, it is not clear that the second momentum estimate of Ada-X is really stable. For this, it would be interesting to see how the trajectories of second-order momentum estimates of Adam, AMSGrad, Ada-X are different. I think this will help to understand Ada-X better.\n\n2. In terms of theory, I think the Lemma 4.1 is inevitable for convergence guarantees in Theorem 4.1 and Theorem 4.2. Although the authors effectively remove log T in the numerator in Corollary 3.2 of Chen et al. (2019) using their lemma 4.1 (I think this is the key point), the assumption that $\\beta_{2t} = \\beta_2 / t$ seems quite strong, and original Adam paper has no such assumptions. For a real deep learning problems such as training ResNet on CIFAR-10, the $\\beta_{2t}$ is almost zero after even one or two epochs where Ada-X behaves like vanilla SGD. Is there no room for relaxing this assumption such as $\\beta_{2t} = \\beta_2 / \\sqrt{t}$? Also, it is not clear how the authors derive Corollary 4.2 from Theorem 4.2 since Theorem 4.2 assumes $\\beta_{2t} = \\beta_2 / t$ while Corollary 4.2 does not.\n\n3. In the experiment, it is not clear that the authors use the same strategy for constructing first-order momentum for Adam with a newly introduced parameter $\\beta_3$. In other words, the authors should use the same policy on constructing the first-order momentum estimate for both Adam and Ada-X. Also, as the authors add an additional hyperparameter $\\beta_3$, the effect of $\\beta_3$ on performance should be discussed at least empirically.\n\n4. There are many studies on fixing poor generalization of adaptive methods (such as AdaBound which the authors cited). In this context, Zaheer et al. (2018, Adaptive methods for non-convex optimization) propose a large epsilon value (numerical stability parameter) such as $\\epsilon = 10^{-3}$ for better generalization. It will be more interesting to see the comparisons in this regime.\n\n5. In my experience with Adam-W (Decoupled weight decay regularization), Adam-W requires a relatively large weight decay parameter $\\lambda$.\nAs an example, DenseNet-BC-100-12 shows a similar validation accuracy with Adam-W $\\lambda = 0.05$ under the learning rate scheduling in (Huang et al. 2016, DenseNet)  as vanilla SGD.\nTherefore, the authors should consider more broader range of weight decay parameters for at least image classification tasks.\n\nMinor:\n1. In eq (2), the domain of x should be mentioned: according to Reddie et al, it is [-1,1].\n2. In both theorem 4.1 and corollary 4.1, $D_{\\infty^2}$ should be $D_{\\infty}^2$?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkenmE9a_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new step-size adaptation algorithm called AdaX. AdaX \nbuilds on the ideas of the Adam algorithm to address instability and non-convergence issues. \nConvergence of AdaX is proven in both convex and non-convex settings. The paper \nalso provides an empirical comparison of AdaX against its predecessors \n(SGD, RMSProp, Adam, AMSGrad) on a variety of tasks.\n\nI recommend the paper be rejected. I believe the convergence results could be a significant contribution, but the \nquality of the paper is hampered by its experimental design. The paper felt generally unpolished, containing\nfrequent grammatical errors, imprecise language, and uncited statements.\n\nMy main issue with the paper is the experimental design. I am not convinced that we can \ndraw valid conclusions from the experimental results for the following reasons:\n  - The experiments are lacking important details. How many independent runs of the experiment \n    were the experimental results averaged over? All of the experiments have random initial conditions \n    (e.g. initialization of the network), and should be ran multiple times, not just once. \n    There's no error bars in any of the plots, so it's unclear whether AdaX really \n    does provide a statistically significant improvement over the baselines. \n    Similarly, the data in all the tables is quite similar, so without indicating the \n    spread of these estimates its impossible to tell whether these results are significant or not.\n\n  - How were the hyperparameters and step-size schedules chosen? The performance of Adam, AMSGrad, and \n    RMSProp are quite sensitive to their hyperparameters, and the optimal hyperparameters are problem-dependent. \n    Some of the experiments just use the default hyperparameters; this is insufficient when trying to directly \n    compare the performance of these methods, as their performance can vary greatly with different values of\n    these parameters. I'm not convinced that we should be drawing conclusions about the relative \n    performance of these algorithms from any of the experiments for this reason.\n\nOf course, meaningful empirical results are not necessarily characterized by statistically outperforming the baselines. \nWell designed experiments can highlight important ways in which the performances differ, providing the community \nwith a deeper understanding of the methods investigated. I would argue that the experiments in the paper do not \nachieve this either; the experiments do not provide any new intuition or understanding of the methods, showing \nonly the relative performances in terms of learning curves on a somewhat random collection of supervised learning problems. Why were these specific problems chosen? What makes these problems ideal for showcasing the performance of AdaX? If AdaX is an improvement over Adam, why? What exactly is happening with it's effective step-sizes that leads \nto the better performance? Can you show how their step-sizes differ over time? \n\nStatements that need citation or revision:\n  - \"Adaptive optimization algorithms such as RMSProp and Adam... as well as weak performance \n     compared to the first order gradient methods such as SGD\" (Abstract). This needs a citation. \n     Similarly, \"AdaX outperforms various tasks of computer vision and natural language processing and can catch \n     up with SGD\"; as above, I'm unaware of work (other than theoretical) that shows that SGD significantly \n     outperforms Adam in deep neural networks.\n  -  \"In the era of deep learning, SGD ... remains the most effective algorithm in training deep neural \n      networks\" (Introduction). What are you referring to here? Vanilla SGD? Or are you including Adam etc here? \n      As above, this should have a citation. Adam's popularity is largely due to its effectiveness in training \n      deep neural networks.\n  - \"However, Adam has worse performance (i.e. generalization ability in testing stage) compared with SGD\" \n     (Introduction). Citation needed.\n  - In the last paragraph of the Introduction, you introduced AdaX twice: \"To address the above issues, we propose a \n    new adaptive optimization method, termed AdaX, which guarantees convergence...\", and, \"To address \n    the above problems, we introduce a novel AdaX algorithm and theoreetically prove that it converges...\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJluZCr19H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new adaptive gradient algorithm AdaX, which the authors claim results in better convergence and generalization properties compared to previous adaptive gradient methods. The paper overall is fairly clear, although the writing can be improved in places.\n\nSection 3 is interesting, although calling the section the \"nonconvergence of Adam\" is a bit misleading, since the algorithm does converge to a local minimum.\n\nI have some concerns about the rest of the paper however. I am a bit confused about the changes proposed to Adam that gives rise to the AdaX algorithm. \n\n1. Won't replacing beta2 with 1+beta2 keep increasing the magnitude of the denominator of the algorithm just like AdaGrad does? In that case, is the algorithm expected to work better when having sparse gradients?\n\n2. I also do not quite understand how to interpret using a separate hyperparameter for the momentum term (ie the beta3 hyperparameter that is introduced). How is beta1 and beta3 related? The numerator loses the interpretation of momentum, i.e., averaged past gradients, when using a separate beta3 parameter, and this does not feel like a principled change.\n\nI have a number of questions about the experiments as well, which makes it hard for me to interpret the significance of the empirical results presented:\n\n1. What is the minibatch size used? Do any of the conclusions presented change if the minibatch size is changed?\n\n2. Is the learning rate tuned? The authors mention the initial learning rate used for the experiments, but it is not clear why those values are used? Was the same initial learning rate used for all algorithms?\n\n3. Were the beta1, beta2 and beta3 values tuned for AdaX? What about beta1 and beta2 for Adam? What are the optimal values of these parameters that were observed?\n\n4. How sensitive is performance to the values of these hyperparameters?\n\nOverall I think this work requires quite a bit of work before it is ready for publication, and would benefit from a much more thorough empirical evaluation of the algorithm.\n\n==================================\n\nEdit after rebutall:\nI thank the authors for their response. While the paper has definitely improved in the newer draft, after having read the other reviews and the updated draft, I believe the paper still requires a bit of work before being ready for publication. I am sticking to my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the non-convergence issue in Adam in a simple non-convex case. The authors propose a new adaptive gradient descent algorithm based on exponential long term memory, and analyze its convergence in both convex and non-convex settings. The major weakness of this paper pointed out by many reviewers is its experimental evaluation, ranging from experimental design to missing comparison with strong baseline algorithms. I agree with the reviewers’ evaluation and thus recommend reject.
Decision: Reject