Adapting Stepsizes by Momentumized Gradients Improves Optimization and Generalization
Authors: Yizhou Wang
Yue Kang
Can Qin
Huan Wang
Yi Xu
Yulun Zhang
Yun Fu
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning Optimizer
Neural Network Optimization
Neural Network Generalization

Abstract: Adaptive gradient methods, such as Adam, have achieved tremendous success in machine learning. Scaling gradients by square roots of the running averages of squared past gradients, such methods are able to attain rapid training of modern deep neural networks. Nevertheless, they are observed to generalize worse than stochastic gradient descent (SGD) and tend to be trapped in local minima at an early stage during training. Intriguingly, we discover that substituting the gradient in the second moment estimation term with the momentumized version in Adam can well solve the issues. The intuition is that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a better choice for scaling than that of the raw gradient. Thereby we propose AdaMomentum as a new optimizer reaching the goal of training fast while generalizing better. We further develop a theory to back up the improvement in optimization and generalization and provide convergence guarantees under both convex and nonconvex settings. Extensive experiments on a wide range of tasks and models demonstrate that AdaMomentum exhibits state-of-the-art performance consistently. The source code is available at https://anonymous.4open.science/r/AdaMomentum_experiments-6D9B.

Under review as a conference paper at ICLR 2022
ADAPTING STEPSIZES BY MOMENTUMIZED GRADI- ENTS IMPROVES OPTIMIZATION AND GENERALIZA- TION
Anonymous authors Paper under double-bling brind review
ABSTRACT
Adaptive gradient methods, such ADAM, have achieved tremendous success in machine learning. Scaling gradients by square roots the running averages squared past gradients, such methods are able to attain rapid training of modern deep neural networks. Nevertheless, they are observed to generalize worse than stochastic gradient descent (SGD) and tend to be trapped in local minima early stage during training. Intriguingly, we discover that substituting the gra- dient in the second moment estimation term with the momentumized version in ADAM can well solve the issues. The intuition is that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a better choice for scaling than that of the raw gradient. Thereby we propose ADAMOMENTUM as new optimizer reaching the goal of training fast while generalizing better. We further develop a theory to back up the improvement in optimization and generalization and provide convergence guarantees under both convex and nonconvex settings. Extensive experiments on wide range of tasks and models demonstrate that ADAMOMENTUM exhibits state-of-the-art perfor- mance consistently. The source code is available at hatpps:/ anonymous. 4open. sci wwweeeeceer/rraaaeeee wecpnttt_e___ee_eeeeeee expprrmentts. 6D9B
INTRODUCTION
Prevailing first-order optimization algorithms in modern machine learning can be classified into two categories. One stochastic gradient descent (SGD) (Robbins & Monro 195I), which widely adopted due to its low memory cost and outstanding performance. SGDM (Sutskever et al.) 2013) which incorporates the notion of momentum into SGD, has become the best choice for optimizer in computer vision. The drawback of SGD(M) is that it scales the gradient uniformly in all directions, making the training slow especially at the begining and fail to optimize complicated models well be- yond Convolutional Neural Networks (CNN). The other type is adaptive gradient methods. Unlike SGD, adaptive gradient optimizers adapt the stepsize (a.k.a. learning rate) elementwise according to the gradient values. Specifically, they scale the gradient by the square roots of some form of the ning average of the squared values of the past gradients. Popular examples include AdaGrad (Duchi al. 2011). RMSprop (Tijmen Tieleman 2012) and Adam (Kingma & Ba 2015)) etc. Adam, in particular, has become the default choice for many machine learning application areas owing to its rapid optimizing speed and outstanding ability handle sophisticated loss curvatures.
Despite their fast speed in the early training phase, adaptive gradient methods are found by stud- ies (Wilson et al.) 2017: Zhou et al.) (220)) to be more likely to exhibit poorer generalization ability than SGD. This is discouraging because the ultimate goal of training in many machine learning tasks exhibit high performance during testing phase. In recent years researchers have put many efforts to mitigate the deficiencies of adaptive gradient algorithms. AMSGrad (Reddi et al.) 2019) corrects the errors in the convergence analysis of Adam and proposes faster version. Yogi (Reddi et al.) 2018) takes the effect of batch size into consideration. M-SVAG (Balles & Hennig 2018) transfers the variance adaptation mechanism from Adam to SGD. AdamW (Loshchilov & Hutter (017b) first- time decouples weight decay from gradient descent for Adam-alike algorithms. SWATS (Keskar & Socher (017) switches from Adam to SGD throughout the training process via a hard schedule and AdaBound (Luo al. 2019) switches with smooth transation by imposing dynamic bounds

---

Under review as a conference paper at ICLR 2022
on stepsizes. RAdam (Luu et al. 2019) rectifies the variance of the adaptive learning rate through investigating the theory behind warmup heuristic (Vaswani et al. 2017 Popel & Bojar 2018). Ad- aBelief (Zhuang al. 2020) adapts stepsizes by the belief in the observed gradients. Nevertheless, most of the above variants can only surpass (as they claim) Adam or SGD in limited tasks or under specifically and carefully defined scenarios. Till today, SGDD and Adam are still the top options in machine learning, especially deep learning (Schmidt et al.. (021). Conventional rules for choos- ing optimizers are: from task perspective, choose SGDM for vision, and Adam (or AdamW) for language and speech; from model perspective, choose SGDM for Fully Connected Networks and CNNs, and Adam (or AdamW) for Recurrent Neural Networks (RNN) (Cho et al.. 2014 (Hochreiter & Schmidhuber, 1997b). Transformers (Vaswani et al. (017) and Generative Adversarual Networks (GAN) (Goodfellow et al. 2014). Based on the above observations, a natural question is:
Is there a computationally efficient adaptive gradient algorithm that can converge fast and mean- while generalize well?
In this work, we are delighted to discover that simply replacing the gradient term in the second moment estimation term of Adam with its momentumized version can achieve this goal. Our idea comes from the origin of Adam optimizer, which is combination of RMSprop and SGDM. RM- Sprop scales the current gradient by the square root of the exponential moving average (EMA) of the squared past gradients, and Adam replaces the raw gradient in the numerator of the update term of RMSprop with its EMA form, i.e., with momentum. Since the momentumized gradient is a more accurate estimation of the appropriate direction descent, consider putting it in the second mo- ment estimation term as well. We find such operation makes the optimizer more suitable for the general loss curvature and can theoretically converge minima that generalize better. Extensive experiments on a broad range of tasks and models indicate that: without bells and whistles, our pro- posed optimizer can be as good as SGDM on vision problems and outperforms all the competitors in other tasks, meanwhile maintaining fast convergence speed. Our algorithm is efficient with no additional memory cost, and applicable to a wide range of scenarios in machine learning, especially deep learning. More importantly, AdaMomentum requires little effort in hyperparameter tuning and the default parameter setting for adaptive gradient method works well consistently in our algorithm.
Notation We use t, T to symbolize the current and total iteration number in the optimization pro- cess. 0 E Rd denotes the model parameter and =() E IR denotes the loss function. We further use Og to denote the parameter step / and fi to denote the noisy realization of f at time / because of the mini-batch stochastic gradient mechanism. stepsize. my, Ut represent the EMA of the gradient and the second moment estimation term time t of adaptive gradient methods respectively. e is a small constant number added in adaptive gradient methods refrain the denominator from being too close to zero. B1, B2 are the decaying parame- ter in the EMA formulation of m, and i2 correspondingly. For any vectors a, b E Rd we employ vú, a², |al, a/b,u a 2 b, a S b for elementwise square root, square, absolute value, division, greater or equal to, less than or equal to respectively. For any 1 S i S d, or,; denotes the i-th element of 0, Given a vector x E de.d, we use ||ær||2 to denote its l2-norm and || to denote its I. -norm.
denotes the 1-th time gradient and a denotes
2 ALGORITHM
Preliminaries & Motivation Omitting the debiasing operation and the damping term c, the adaptive gradient methods can be generally written in the following form:
Table 1: Comparison of AdaMomentum and clas- sic adaptive gradient methods in my and i in
Optimizer
SGD
0++ 0
(1)
Rprop
Vo
RMSprop
(1 - B2)
Adam
(1 (1) EL
8
(1 - 32)
Here me, U/ are called the first and second mo- ment estimation terms. When m, = gt and
Ours
(1 (1) EI
B5 ou
32)
ey 1. degenerates to the vanilla SGD. Rprop DDuchi et al. 2011) is the pioneering work us- ing the notion of adaptive learning rate, in which m. = It and 2/2 = 9.. Actually it is equivalent to only using the sign of gradients for different weight parameters. R.MSprop (Tijmen Tieleman, 2012) forces the number divided to be similar for adjacent batches by incorporating momen- tum acceleration into et. Adam (Kingma & Ba 2015) is built upon RMSprop in which it turns
2

---

Under review as a conference paper at ICLR 2022
((0)
large, |m]]
global minimum
local minimum
|m] gradually becomes large as [g]
update
AdaMomentum
Adam
lst [m] is large
Zoom
0,
0
163
0
Figure 1: Illustration of the optimization process of Adam and AdaMomentum. A general loss curve can be composed to three areas: A) transition from a plateau to downgrade; B) steep downgrade; C) from downgrade to entering the basin containing the optimum. An ideal optimizer ought to sus- tain large stepsize before reaching the optimum and reduce its stepsize near the optimum. Compared Adam, AdaMomentum can adapt the true stepsize more appropriately along the loss curve and maintain smaller stepsize near convergence. Refer to Section 3.1 for more detailed analysis.
into momentumized version. Both RMSprop and Adam boost their performance thanks to the smoothing property of EMA using momentum. Due to the fact that momentumized gradient is more accurate estimation than raw gradient, we deem that there is no reason to use in lieu of me second moment estimation term vt. Therefore we propose replace the g&s v of Adam with their momentumized versions m,s, which further smooths the exponential moving average.
Algorithm AdaMomentum (ours). All mathematical operations are clement-wise.
1: Initialization Parameter initialization on step size a, damping term mg <- 0, Un + 0,
2: while 0g not converged do
3: t x 1
> Updating time step >> Acquiring stochastic gradient at time
4:
Jr - (efe(0
5:
me + B1m=-1 + (1 Bi)gt
EMA of gradients EMA of squared momentumized gradients Bias correction of first moment estimation Bias correction of second moment estimation > Updating parameters
6:
Ut - 322/-1 + +1 1-32)mm 32)mm +c
7:
m me/(1 (1- 39)
8:
Tt t w/(11 (1- B2)
9:
O₂ + 0 a mm/ /o
10: end while
Detailed Algorithm The detailed procedure of proposed optimizer is displayed in Algo- rithm [ There are two major modifications based on Adam, which are marked in red and blue respectively. One is that we replace the dt in U, of Adam with m, which is the momentumized gradient. Hence we name our proposed optimzier as AdaMomentum. The other is the location of e (in Adam e is added after vf line 10 Alg I]. We discover that moving the adding from outside the radical symbol to inside can consistently enhance performance. To the best of our knowledge, our method is the first attempt to put momentumized gradient in the second moment estimation term of adaptive gradient methods. Note that although the modifications seem simple to some degree, they can lead to siginificant changes in the performance of an adaptive gradient optimizer due to the iterative nature of optimization methods, which will also be elaborated in the following section.
3 WHY ADAMOMENTUM OVER ADAM?
3.1 ADAMOMENTUM IS MORE SUITABLE FOR GENERAL LOSS CURVATURE
In this section, we show that AdaMomentum can converge to (global) minima faster than Adam does via 1-D example. The left part of Figure is the process of optimization from plateau to basin area, where global optimum is assumed to exist. The right part is the zoomed-in version of the situation near the minimum, where have some peaks and valleys. This phenomenon frequently takes place optimization since there is only one global minimum with probably great number of local minima surrounding (Hochreiter & Schmidhuber 1997a Keskar et al. 2017).

---

Under review as a conference paper at ICLR 2022
Benefits of substituting It with my- We first explain how substituting my for 9t in the precon- ditioner 2/2 can improve training via decomposing the trajectory of parameter point along the loss curve. 1) In area A, the parameter point starts to slide down the curve and [9.1 begins to enlarge abruptly. So the actual stepsize 0x/ / is small for Adam. However the absolute value of the mo- mentumized gradient m, is small since it is the EMA of the past gradients, making a/ /u, still large for AdaMomentum. Hence AdaMomentum can maintain higher training speed than Adam in this changing corner of the loss curve, which is what an optimal optimizer should do. 2) In area B, since the exponential moving average decays the impact of past gradients exponentially w.r.t. t, the mag- nitude of the elements of m, will gradually becomes as large as g:- 3) In area C, when the parameter approaches the basin, the magnitude of y). decreases, making the stepsizes of Adam increase imme- diately. In contrast, the stepsize of AdaMomentum is still comparatively small as |ime] is still much larger than l9cl which is desired for an ideal optimizer. Small stepsize near optimum has benefits for convergence and stability. A more concrete illustration is given in the right part of Figure [I If the stepsize is too large (e.g. in Adam), the weight parameter O, may rush to 0 022) and miss the global optimum. In contrast, small stepsize can guarantee the parameter to be close to the global minimum
(see 0 (1) even if there may be tiny oscillations within the basin before the final convergence.
Benefits of changing the location cf Next we elaborate why putting under the v is beneficial. We denote the debiased second moment estimation in AdaMomentum as iG and the second moment estimation term without as Fyg By simple calculation, we have
& = ((1 (11 (1-32)/(11 - ()) Σ
f m + 1 1 B2"
T = 0((1 (2)(11)(11-)) -3))
Σ
B/ m
1m1
Hence we have 02 = -%-- c/(1 32). Then the actual stepsizes are a/(v (11+ 32) and a/(vT, (V,++e) +e) respectively. In the final stage of optimization, iy is very close 0 (because the values of gradients are near 0) and far less than hence the actual stepsizes can be approximately written as VI- VE and a e.AAs usually takes very tiny values ranging from 10 80 to 10 16 and 32 usu- ally take values that are extremely close to (usually 0.999), we have vI - 32a/ VE < a/e. There- fore we may reasonably come to the conclusion that after moving e term into the radical symbol, AdaMomentum further reduces the stepsizes when the training is near minima, which contributes to enhancing convergence and stability as we have discussed above.
3.2 ADAMOMENTUM CONVERGES TO MINIMA THAT GENERALIZE BETTER
The outline of Adam and our proposed AdaMomentum can be written in the following unified form:
m 8ime-1 + (1 - -1)gt, Vt - 112--1110 + (1 32)
0c+1 = 02 - a aam (11
/2/(1 (11-8)y 81) - (35
where ky = 9: in Adam and k m, in AdaMomentum. Inspired by line of work (Pavlyukevich) 2011 Simsskli et al. [2019] Zhou et al. 2020). we can consider ( as a discretization of continuous time process and reformulate it its corresponding Lévy-driven stochastic differential equation (SDE), Assuming that the gradient noise Ç = 0: - (0.,) )) is independent and centered symmetric ä-stahle (SãS) (Lévy Lévy 1954 distributed with covariance matrix F., possessing heavy-tailed signature (ā E (0, 2)), we are able to derive the driy-e SDE of (2) as:
d0,- -t midt + oR, "idde, dm, Bi (V)( mc), du, = Ba(k? - vv),
(3)
where R2- = diag( /v B9)), v 1-1// 4: = 1/(1 81) and L, is the ã-stable Lévy motion with independent components. We are interested in the local stability of the optimizers and therefore suppose process 3 is initialized
local basin 22 with a minimum 0- (w.l.o.g.. we assume 8" 0). To investigate the escaping behavior of 6., we first introduce two technical definitions.
Definition 1 (Radon Measure (Simon et al. 1983)). If measure m(-) defined on the o-algebra of Borel sets of Hausdorff topological space X is 1) inner regular open sets, 2) outer regular all Borel sets, and 3) finite on all compact sets, then the measure is called a Radon measure.
Definition 2 (Escaping Time & Escaping Set). We define escaping time I = inf(t 2 0 : 0. & Ω Q=""), }. where Q2-"" [y E Ω dis(0S2. w) 2 v²). Here 7 > is constant. We define escaping R & 52 ""3, where Eg- lime, F,, Rg- lime R.-
set r = {v E Rd
4

---

Under review as a conference paper at ICLR 2022
We study the relationship between I' and T and impose some standard assumptions before proceeding. Assumption 1. f is non non-negative with an upper bound, and locally p-strongly convex in 52. Assumption 2. There exists some constant 1. O, s.t. [Vx) Vf(iill S T. || 7اlا Vo. Assumption 3. assume that So (O.( + f(0)), q.Ht me) ) 2 a.e.. and 81 S 32 2 There exist v-, 14 > 0 s.t. each coordinate of /oa can be uniformly bounded in (v_, u..) and there exist Tm, 7 > 0 s.t. ||mm Mull2 S Tm Ji- (m, m.)) dx and || 2 + || where my and 0, are caleulated
by solving with e = 0.
Assumption I and 2 impose some standard assumptions of stochastic optimization Ghadimi & Lan (2013);
Johnson & Zhang 2013). Assumption 3 requires momentumized gradient m. and (f))) to have similar
directions for most the time, which have heen empirically justified to be true in Adam (Zhou et al. (020).
Based on the ahove assumptions, we can prove that for algorithm of form ( the expected escaping time is inversely proportional to the Radon measure of the escaping set:
Lemma 1. Under Assumptions|
3 let a
@())) and In (2. (wu -
where - f(6b) f(0"). Then given any 0g E S2 220 for 3 we have
E(T)= ((v/m(T)),
where m(-) a non-zero Radon measure satisfying that m(lf) mVV
Because larger set has larger volume, i.e. V(2)= I V(V) i C V, from Lemma we have the escaping time is negatively correlated with the volume of the set Y. Therefore, we can come to the conclusion that for both Adam and AdaMomentum, if the basin 52 sharp which is ubiquitous during the early stage of training. Y has a large Radon measure, which leads to smaller escaping time I. This means both Adam and AdaMomentum prefer relatively flat or asymmetric basin He et (2019) through the training process.
On the other hand, upon encountering a comparatively flat basin or asymmetric valley Z, we are able to prove that AdaMomentum will stay longer inside. Before we proceed, we need to impose two mild assumptions.
Assumption 4. There exists a constant H s.t. V (0:)l2 S H, 119- I H, VL E [Z].
Assumption 5. For AdaMomentum, there exists To s.t., diag(E.) B. 1) 1) 81) when t To-
Here Assumption 4 is a common assumption in stochastic optimization (Ghadimi & Lan 2013 Johnson & Zhang 2013). As 31 is always set positive number close to 1, Assumption 5 basically requires that the gradient noise variance to be smaller than the second moment of m when is very large. This assumption mild as 1) we can select bini-bbtch size to be large enough to satisfy as the noise variance is inversely proportional to batch size (Bubeck] 2014)). 2) The magnitudes of the variances of the stochastic gradients are usually much lower than that of the gradients (Faghri et al.] 2020). Then we can come to the following result.
Proposition Under Assumptions we have
5 upon encountering a comparatively flat basin asymmetric valley 52,
E
(ADAMOMINTUMM)
(I(AAAm))
other words, when falling into a flat/asymmetticc basin, AdaMomentum is more stable than Adam and will not easily escape from it. Combining the aforementioned results and the fact that minima at the flat or asymmetric basins tend to exhibit better generalization performance (as observed in Keskar et al. (2017); He et al. (2019): Hochreiter & Schmidhuber (1997a): [zmailov et al. (2018): Li al. (2018), we able conclude that AdaMomentum more likely to converge to minima that generalize better, which may buttress the improve- ment of Momeentu empirical performance. All the proofs in section 3.2 are provided Appendix [
CONVERGENCE ANALYSIS OF ADAMOMENTUM
In this section, we establish the convergence theory for AdaMomentum under both convex and non-convex object function conditions. We omit the two bias correction steps in the Algorithm foo simplicity and the following analysis can be casily adapted and applicd to the de- biadd version well.
4.1 CONVERGENCE ANALYSIS IN CONVEX OPTIMIZATION
We analyze the convergence of AdaMomentum in convex setting utilizing the online learning frame-
work (Zinkevichl 2003 Given sequence of convex cost functions fi(0),.- (f), the regret is
defined
R(T)- Et f.)).) fe f(()),, where 8* = argmin, Σ, (0) is the optimal parameter and fe can be interpreted as the loss function the t-th step. Then have:
5

---

Under review as a conference paper at ICLR 2022
Theorem Let {0:) and {ur) be the sequences yielded AdaMomentum. Let aavi,, $1
S1.c S B1 < 1,1, U2 Ve+1 for all [T] and y = 31/v32 1. Assume that the distance between any A. generated by AdaMomentum is bounded, ||8m - 8n me S Doo for any m, n E (1,-... ,T), Then we have the following bound on the regret:
R(T) S 20(1 B1)
D2 V
D 2(1 B1)
BL.V VE,
0V +ov++log' log
Σ 1Iu..lll
Ot
(1 31) - 7)11 B2
Theorem implies that the regret of AdaMomentum can be bounded by c VTT) especially when the data features are sparse as Section 1.3 in Duchi al. 2011) then we have Σi=1 VTT < vd and
En 119.., < vdT. When we impose additional assumptions that 81,s decays exponentially and that the gradicnts of ft are bounded (Kingma & Ba 2015 Liu ct al. 2019), we can obtain the following corollary:
Corollary 1. Further Suppose B1, = 31 X and the function fe has bounded gradients, [f(()l) Go for all 8 R" AdaMomentum achieves the guarantee R(T)/T Ö(1/V)) for all T 2 1:
R(T) T
dG aG=
dD
2x - dD2.G
< (1
B2YT
2a(1 B.V
Clearly observed from Corollary 1 the average regret of AdaMomentum converges to zero as I goes to infinity. The proofs of Theorem and Corollary are provided in Appendix C.I
4.2 CONVERGENCE ANALYSIS IN NON-CONVEX OPTIMIZATION
When f is non-convex and lower-bounded, we derive the non-asymptotic convergence rate of AdaMomentum.
Theorem We suppose that Assumptions 2 and 4 hold, and 91,, is chosen such that 0 < $1,441 $ B1, 1,0 < 92,1 < 1, E [T]. We further assume au/ /ox 0441/ 0041///e for any E [2]. Σ at S (T)) S Tar, (T) V. c and minee(T] Oo. I ax, then have:
((T)
C2 Vc + Cs da²
da
min E ( Tor
+Ca
Tor (Q) Qzn(T)),
for some positive constants QI, Q2. Here C1, Cz,, Cs positive constants independent of d or while C4 is a positive constant independent of
Note that the conditions in Theorem can satisfied most scenarios (Kinmma & Ba
2015 Chen et al
2019 Reddi al. 2019). For instance, we can simply employ the common setting a Vt, 82, 1/t. Corollary 2. When ae is further chosen to be a: a/Vi, AdaMomentum satisfies:
H
Ci a log(T)) + C2
da
da +Cs Vc +
min #E|T]
+ Ca
VTa
c
VT (Qi log())))
for some constants Qi, Qi. Ci, C2 Cs, C4 are similarly defined in Theorem
Corollary 2 manifests the (log (T)lo(TT///)) /T) convergence rate AdaMomentum the nonconvex case when we
commonly Vt. We refer readers the detailed proof in Appendix C.2
EXPERIMENTS
We empirically investigate the performance of AdaMomentum in both optimization and generalization.
5.1 FASTER & BETTER OPTIMIZATION
5.1.1
2-D TOY EXAMPLES
We compare the optimization process of AdaMomenturn with three prevalent optimziers SGDM, RMSprop. Adam and recently proposed AdaBelief. four classic and representative 2-variable objective functions in
(-) denotes O(-) with hidden logarithmic factors.

---

Under review as a conference paper at ICLR 2022
SCOM
560k
Adam
RMSprop
RMSprop
AdaBellet
AdaBolie!
AdaMomentum
AdaMomentum
SCIDM
SGDM
Adam
Adam
RNSprop
AMSprop
AdaBellef
AdeBelief
AdaMomentum
AdaMomentum
(a) Trajectory for Sphere (b) Trajectory for Three- (c) Trajectory for Beale (d) Trajectory for Ackley Function.
Hump Camel Function. Function.
Function.
Figure 2: 2D Trajectory visualization of SGDM, Adam, RMSprop, AdaBelief and AdaMomentum on classic functions. AdaMomentum reaches the optimal point (marked purple cross) the fastest in all the cases and converges stably to the optimum without big oscillations. Best viewed in color.
Table 2: FID score (4) of Spectral Normalized Generative Adversarial Network CIFAR-
10 Krizhevsky & Hinton 2009) dataset. i is reported in Zhuang et al. (2020).
SGDM Adam(W)
Yogi!
AdaBound
RAdam
AdaBelieft
Ours
49.70 13.05 1 0.19 14.25 0.15 55.65
12.70 0.12 12.52 0.16 12.06 1 0.21
numerical optimization literature: Sphere Function (bowl-shaped) (Dixon 1978), Three-Hump Camel Function (valley-shaped) Beale Function (multimodal)) and Ackley Function (with numerous local minima) (Adorio & Diliman 2013). To ensure fair comparison, we use the same hyperparameters in the four adaptive gradi- ent methods and finetune the learning rate of SGDM for the best performance. As illustrated in Figure 2 AdaMomentum achieves the most rapid convergence in all the cases and is stable once reaching the global mininum. Mcanwhile, SGDM is highly unstable and inaccurate the descending directions, and RMSprop. Adam, AdaBelief is much slower. Although these toy examples are simple, they give hints to the behavior of optimizers complex deep learning tasks as they can he viewed as the local dynamics which occur frequently in deep learning (Zhuang et al. 2020). The details of the loss functions and hyperparameter configurations of the experiment in Appendix A The GIFs and the 3D trajectory figures included in the supplementary.
5.1.2 GENERATIVE ADVERSARIAL NETWORK
Training of GANs is extremely unstable. To further study the optimization ability and stability of AdaMomen- tum, experiment GAN equipped with spectal normalization (Miyato et al. 2018). For the generator and the discriminator network, we adopt ResNets for adequate expression ability. We train model for 100000 iterations on CIFAR-10 10 with batch size 64, and the two learning rates are set both as 0.0002. For AdaMomen- tum all the other hyperparameters are set as default values. Experiments are run times independently and we report the mean and standard deviation of Frechet Inception Distance (FID, the lower the better) Heusel et al.) (2017) in Table 2 From Table 2 is reasonable draw the conclusion that AdaMomentum outperforms all the best tuned baseline optimizers hy a large margin, reaching mean FID score as low as 12.06 with its default hyperparameter values. Here Adam equals AdamW because the optimal weight decay parameter value is 0.
5.2 SUPERIOR GENERALIZATION
We conduct experiments various modern network architectures for different tasks covering both vision language processing area: 1) image Classification on CIFAR-10 (Krizeesky Hinton (009) and Ima- geNet (Russakovsky et al. (2015) with CNN: 2) language modeling on Penn Treebank (Marcus et al. [993]
dataset using Long Short- Term Memory (LSTM) (Hochreiter & Schmidhuber 1997b): 3) neural machine trans-
lation IWSTL' 14 DE-EN (Cettolo et al. (014) dataset employing Transformer. We compare AdaMomen- with seven state-of-the-art optimizers: SGDM (Sutskever al., 2013), Adam (Kingma
Ba 2015).
AdamW (Loshchilov & Hutter 7b).,), Yogi (Reddi et al. 2018), AdaBound (Luo et al. 2019), RAdam (Liu
at 2019) and AdaBelicf (Zhuang et al, 2020). We We perform careful and extensive hyperparamttee tuning for
https://en.wikippeet http:///ro.occcooeeoooooooooooon
http://www-oph amp. wtmpp..kyotoopp.. www.kyyotoo..oookk ac / /
fil 111es/eestc..hm

---

Under review as a conference paper at ICLR 2022
Train Accuracy (%6
Train Accuracy (%6)
Train Accuracy (%6)
KOON
(a) Train Accuracy for VGGNet.
(b) Train Accuracy for ResNet.
(c) Train Accuracy for DenseNet.
Accuracy
Test
225
139
175
30
15
2.00
140
125
Epoch
Epoch
Epoch
(d) Test Accuracy for VGGNet.
(e) Test Accuracy for ResNet.
(l) Test Accuracy for DenseNet.
Figure 3: Train and test accuracy of different optimizers CIFAR- 10 Krizhevsky & Hinton 2009
all the optimizers compared, and the detailed strategy and configuration are given in Appendix due to space limit. It worth mentioning that in experiments we discover that setting a 0.001, 81 0.9, 92 0.999 (the default setting for adaptive gradient methods in applied machine learing) works well most cases. This elucidates that our optimizer is tuning-frienn friendll, which reduces human labor and time cost and is crucial in practice. The mean results with standard deviations over 5 seeds are reported in all the following experiments.
5.2.
CNN FOR IMAGE CLASSIFICATION
Table Test accuracy (%) of CNNs on CIFAR-10 dataset. The best in Red and second best in blue.
Non-adaptive
Adaptive gradient methods
Architecture
SGDM
Adam
AdamW
Yogi
AdaBound
RAdam
AdaBelief
Ours
16 ResNet-34
94,73
93,29
93.33
93 43.44
93. 93.7970..17
93 03.90
94.57=0.00
94,80
96.4740.00
95.4844010 95.3910.11
95.28
95.51
95.6720.16
96.0440.07
96.3340.07
DenseNet-121
95.03
90.92
93.87*0014
93.72
93.99 93.99
94.00
94.74
95.08
Table 4: Top-1 test accuracy (%) on ImageNet (Russakovsky et al.
2013) dataset.
SGDM
Adam
AdamW
Yogi
AdaBound
RAdam
AdaBelief
Ours
00
65.36=0.25
68.77+0.11 68.93+0.08 69.32-1..19 69.2440.12 69.98+0.09
7..45++0.6
experimented with three prevailing deep CNN architectures: VGG-16 (Simonyan Zisser- man 2015). ResNet-34 (He et al. 2016) and DenseNet-12) 121 (Huang et al.] (207)). The growth rate of DenseNet- 121 set as 12 to match CIFAR- TO dataset. In each experiment we train the model for 200 epochs with batch size 128 and decay the learning rate by 0.2 at the 60-th, 120-th and 160-th epoch. We employ label smoothing technique (Szegedy al. 2016) and the smoothing factor choosen as 0.1. Figure displays the training and testing results of all the compared optimizers As indicated, both the training accuracy and the testing accuracy using AdaMomentum can be improved as fast as with other adaptive gradient methods, being much faster than SGDM, especially before the third learning rate annealing. In testing phase, AdaMomentum can exhibit performance good SGDM and far exceeds other baseline adaptive gradient methods, including the recently proposed AdaBelief (Zhuang et al. (2000) optimizer. This contradicts the result reported Zhuang let al. (2020), where they claim AdaBelief can be better than SGDM. This largely stems from the fact that Zhuang et al. 2020) did not take an appropriate stepsize annealing strategy or tune the hyperparameters well. Training 200 epochs with ResNet-34 on CIFAR-10, experiments show that AdaMomentum and SGDM can reach over 96% accuracy, while in Zhuang al. (020) the accuracy of SGDM is only around 94%.
CIFAR-10
8

---

Under review as a conference paper at ICLR 2022
Aaaaarrr Adam Morr Togi
50.4
Adom
Test Perplexity
Adamtii
Test Per Porploxtty
Vegt
Addddi
AdUdE No!
Test
100
135
110
175
380
20
1.0
135
15
159
125
150
1.5
Epoch
Epach
Epoc h
(a) L-Layer LSTM.
(b) 2-Layer LSTM.
(c) 3-Layer LSTM.
Figure 4: Test perplexity curve on Penn Treebank (Marcus et al.
1993) dataset.
Table 5: Test perplexity (4) results of LSTMs on Penn Treebank (Mareus et al.)
1993) dataset.
Tayer
SGDM
Adam
Adam
Yogi
AdaBound
RAdam
AdaBelief
Ours
85.31
84.55#00.0
88. 18.1840.14
86.87#0.14
85.10=00.2
88.60#0.22
84.30=0.25
80.82+0.15
2
67,1 67,11 67.25=0.20
73.61=0.15
71.5420.14
67.69=20.24
73. 73.80=0.25
66.66=#.11
64.85
3
#0.52
64.10#0022
69.91
67.58*0.s
63.52#0.11
70.10
61.33=0.19
60.08
ImageNet To further corroborate the effectiveness of algorithm on comprehensive dataset, we perform experiments on ImageNet utilizing ResNet-18 as backbone network. execute each optimizer for 90 epochs utilizing cosine annealing strategy, which can exhibit better performance results than step-based decay strategy on ImageNet (Loshchilov & Hutter] 22 Ma 2021). As indicated in Table AdaMomentum far exceeds Adam in Top-1 test accuracy and outperforms all the competitors including SGD with momentum.
5.2.2 LSTM FOR LANGUAGE MODELING
We implement LSTMs with layer number from methods are the main-stream choices (much better than SGD). In each experiment we train the model for 200 epochs with batch size of 20 and decay the learning rate by 0.1 at 100-th and 145-th epoch. Test perplexity (the lower the better) against training epochs is plotted in Figure[aand the best perplexity value is summarized in Table 5 Clealy observed from Figure 4 and Table 5 AdaMomentum achieves the lowest perplexity in all the settings and consistently outperform other competitors by considerable margin. The training curve given Figure Appendix Didue to space limit. Particularly on 2-layer and 3-layer LSTM, AdaMomentum maintains both the fastest convergence and the best performance, which substantiates its superiority.
Penn Treehank dataset, where adaptive gradient
5.2.3 TRANSFORMER FOR NEURAL MACHINE TRANSLATION
Transformers have been the dominating architecture in NLP and adaptive gradient methods are usually employed to train transformers due to their stronger ability to handle attention-models (Zhang et al.] 2019) To test AdaMomentum on transformer, experiment IWSTL'14 permman-to-Englise with the Transformer small model adapting the code from fairseq package We set the length penalty as 1.0, beam size as 5, warmup initial stepsize as 10 and the warmup updates iteration number to be 8000. We train the models for 55 epochs and the results are reported according to the average of the last checkpoints. As shown in Table 12 our optimizer achieves the highest average BLEU score with the lowest variance.
Table 6: BLEU score (t) on IWSTL'14 DE- EN (Cettolo et al., 2014) dataset.
Adam 28.2220.24 30.14:1.56 35.62=0.13
Adami
AdaBelief
Ours
35.6020.12
35.66:10.11
6 CONCLUSION
In this work, we proposed AdaMomentum as new optimizer for machine learning. We theoretically demon- strate why AdaMomentum outperforms Adam optimization generalization. We further validates the superiority of AdaMomentum through both toy examples and large-scale experiments on real-world datasets. Our algorithm is simple and effective with four key advantages: 1) maintaining fast convergence rate; 2) clos- ing the generalization gap between adaptive gradient methods and SGD; 3) applicable to various tasks and models; introducing no additional parameters and easy tune. Combination of AdMMentuu with other techniques such as Nesterov's accelerated gradient (Dozat 2016) may of independent interest in future.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
Our work follows all ethical standards and laws. All the experiments conducted on publically available datasets, with no new data concerning human or animal subjects generated.
REPRODUCIBILITY STATEMENT
We adhere to ICLR reprodueibility standards and provide all necessary information to reproduce our experi-
mental and theoretical results. We ensure the reproducibility of our work through several ways, namely
All the source code and presented figures are available anonymous link httta::/aaonnmonn oopen sci
http:///mmmnnnneeeepeeeeeeeeeeee ments- 6D98
The detailed descriptions of the classic loss functions used in Toy exmpales in Section are given in Appendix [A]
All the technical details and proofs in Section 3.2 are included in Appendix All the proofs in
Section provided in Appendix C
The detailed hyperparameter tuning rule and configurations the experiments Section are given
in Appendix D
REFERENCES
E.P. Adorio and U.P. Diliman. MVF: Multivariate test functions library in c for unconstrained global optimiza- tion. Retrieved June 2013, from http::/www.gcoci.i wttp:/ttee....ceeeeeeeeeeeeeeeee ws//aadorlo//mff.pa.. .pd 2013.
ws
Lukas Balles and Philipp Hennig. Dissecting adam: The sign. magnitude and variance of stochastic gradients. In ICML, 2018
Sébastien Bubeck. Convex optimization: Algorithms and complexity. arXiv preprint 1rXiv:1405.49880 2014.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop Spoken Language Trunslation, Hanoi, Vietnam, 2014.
Xiangyi Chen, Sijia Liu, Ruoyu Sun. and Mingyi Hong. On the convergence of class of adam-type algorithms for non-convex optimization. In ICLR, 2019.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoderr approaches. Proceedings of SSST-8, Eighth Workshop Syntax, Semantics and Structure in Statistical Translation, 2014.
Laurence Charles Ward Dixon. The global optimization problem. introduction. Toward global optimization, 1978.
Timothy Dozat. Incorporating nesterov momentum into adam. ICLR Workshop, 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011.
Fartash Faghri, David Duvenaud, David J Fleet, and Jimmy Ba. A study of gradient variance in deep learning. arXiv preprint arXiv:2007.04532. 2020.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic pro- gramming. SIAM Journal Optimization, 2013.
Ian J Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. NeurlPS, 2014.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. NearIPS,
2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bemhard Nessler, and Sepp Hochreiter. Gans trained by a time-scale update rule converge to local nash equilibrium. NewrIPS, 2017.
10

---

Under review as a conference paper at ICLR 2022
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation. 1997a.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997b.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. UAI, 2018.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. NewrlPS, 2013.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd, arXiv preprint arXiv: 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiay images. Master's thesis. Depart- ment of Computer Science, University of Toronto, 2009.
Paul Lévy and Paul Lévy, Théorie de I' l'addition des variables aléatoires. Gauthicr- Villars, 1954.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Goldstein. Visualizing the loss landscape of neural nets. NeurlPS, 2018.
Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. the variance of the adaptive learning rate and beyond. ICLR, 2019.
Πya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, ICLR, 2017a.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2017b.
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. ICLR, 2019.
Xuezhe Ma. Apollo: An adaptive parameter-wised diagonal quasi-newton method for nonconvex stochastic optimization, 2021. URL httpps///ppenrevvieeeeee/e//eees
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: The Penn Treebank. Computational Linguisties, 1993.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Ilya Pavlyukevich. First exit times of solutions of stochastic differential equations driven multiplicative lévy noise with heavy tails. Stochastics Dynamics, 2011.
Martin Popel and Ondřej Bojar. Training tips for the transformer model. PBML, 2018.
S Reddi, Manzil Zaheer, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In NeurlPS, 2018
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. ICLR, 2019.
Herhert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, 1951.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through crowded valley- bellyy-benchmrringg deep learning optimizers. ICML, 2021.
Leon Simon et al. Lectures on geometric measure theory. The Australian National University, Mathematical Sciences Institute, Centre for Mathematics & its Applications, 1983.

---

Under review as a conference paper at ICLR 2022
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. JCML, 2019.
Πlya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum deep learning. In ICML, 2013.
Christian Szegedy, Vincent Vanhoucke. Sergey loffe. Jon Shlens, and Zbigniew Wojna. Rethinking the incep- tion architecture for computer vision. In CVPR, 2016.
Geoffrey Hinton Tijmen Tieleman. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. Coursera: Neural networks for machine learning, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. NeurIPS, 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stem, Nathan Stebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. NeurIPS, 2017.
Jingzhao Zhang, Sai Pranceth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good attention models? arXiv preprint arXiv:1912.031194 2019.
Zhou, Jiashi Feng, Chao Ma, Caiming Xiong. Steven Hoi. and Weinan E. Towards theoretically under- standing why sgd generalizes better than adam in deep learning. In NeurlPS, 2020.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. NewrlPS, 2020.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In ICML, 2003.
12

---

Under review as a conference paper at ICLR 2022
A DETAILS OF TOY EXAMPLES
The detailed description of the four 2-parameter loss functions given as below:
The Sphere Function
f(x) = 2 + 22
whose global minimum f(x")) at
(0,-.-,0)
The Three Hump Camel Function
f(z) = 2nf 1.05xt +
x1 + 22.
whose glohal minimum f(x') 0 at (0....
The Beale Function is
f(z) - -1.5 x112)) (2.2 21 + x112)² + (2.625 21 + x122)²
whose global minimum is f Oat (3,0.5).
The Ackley Function is
exp (cos(2 + cos(( ) 20 exp(1),
f(x) -0 exp (0..2
(if =
2
whose global minimum f(x')="+=))))))))"))))) 0 at xt (0.... ... ,0)
Hyperparameter configuration For all the four toy experiments, we employ the stepsize for the three adaptive gradient methods: RMSProp, Adam and AdaMomentum. parameter as 0.9 and finetune the learning rate for each toy experiment in set (0.0001, 0.001, 0.01, 0.1, 1, 10} as the optimal learning rate varies for different loss functions. After careful parameter tuning, the configuration is: for Sphere Function, the learning rate for adaptive gradient methods and SGDM are both 0.1; for Three- Hump Camel Function, the learning
SGDM We set the momentum
for adaptive gradient methods is 0.1 and the learning rate for SGDM 0.001; for Beale Function, the leanring rate adaptive, gradient methods is 0.1 and the learning rate for SGDM is 0,1: for Ackley Function, the leanring rate for adaptive, gradient methods is 0.1 and the learning rate for SGDM is 0.1. Combining this with Figure we can see that AdaMomentum with the stepsize 0. manifests universally faster and more stable convergence the optimum than discreetly tuned SGDM and other adaptive gradient approaches.
B TECHNICAL DETAILS OF SUBSECTION 3.2
Here we provide more construction details and technical proofs the Lévy-driven SDE Adam-alike adap- tive gradient algorithm (2). In the beginning we introduce a detailed derivation of the process (3) as well as its corresponding escaping set Y in definition 2 Then we give some auxiliary theorems and lemmas. and summarize the proof of I Lemma] T Finally we prove the proposition and give more detailed analysis of the conclusion that the expected escaping time of AdaMomentum longer than that of Adam in comparatively flat basin.
B. DERIVATION THE LÉVY-DRIVEN SDE 3
To derive the SDE of Adam-alike algorithms m6 = O. Then by the definition it holds that
we firstly define mi = 3imi (1 (()))(()) with
m - m. = (81
3 'c.
Following Simsckli et al. 2019). the gradient noise Çe has heavy tails in reality and hence we assume that
(m) me) obeys SaS distribution with time-dependent covariance matrix Σ.. Since can formulate
as
0 = 0, a
mi
+ a
(m/. -ma)
where 21 = (1 - Biy
2x
(4)
zt
21
(1 3.)
and can replace the term (ml) m²) by a 1 (1
3 E,, where each coordinate of S is independent and identically distrihuted as SGS(1) based the property of centered symmetric a-stable distribution. Let
13

---

Under review as a conference paper at ICLR 2022
R. diag(
(1-P5) ). and further assume that the step size is small, then the continuous-time version of
the process becomes the following SDE:
d0: RT midt +a Ri E,dLa, dmn B1 ((:)) -m)). du B2(k2 vt).
(1 8[
After replacing my with my for brevity, we get the SDE
consequently.
B.2 PROOF LEMMA
To prove Lemma we first introduce Theorem
Theorem Suppose Assumptions hold. We define πi
and fut (Br (/)) with
a constant er. Let u O Po 10(1+02) and In (10 S K2U 1/3 where A = f(0v) - f(00) and
constant c2. Then any 0u E f2 2u? 2 1-11 and lim p = 0, the Adam-alike algorithm in obey
1. v E (0, va]. 7 (0, %al and E (0, po) satisfying v" Po
1+p
B (eeepp(aam())
1+ P
1
From Theorem 3 by setting v small, it holds that for any adaptive gradient algorithm the upper and lower
bounds of its expected escaping time I' is at the order of (m(T)) which directly implies Lemma conclusively. Therefore, it suffices to validate Theorem B
Theorem is adapted from Theorem in Zhou al. 2020) and the proof is given in Section B.2.3 Before we proceeed, we first provide some prerequisite nolations in Section [.2.1 and list some useful theorems and lemmas in Section B.2.2
B.2.1 PRELIMINARIES
For analyzing the uniform Lévy-driven SDEs in & and é;, namely
3) we first introduce the Lévy process L, into two components
L, Ets
whosc characteristic functions are respectively defined
E e (A..51)
=e N Nad (Ilbllas )u(ns)
B (((Ar)))
= Nad (1lvill2 )((d))
where e (((.)) -1 T [llvl22 S }} with defined in and constant 5 s.t. n 1. Accordingly, the Lévy measure of the stochastic processes & and are
" = (
(An (An [1vlla
}) ve = (An
|IMll 2 ) where A E B(R").
Besides, for analysis, we should consider affects of the Lévy motion Le to the Lévy-driven SDE of Adam variants. Here we define the Lévy-free SDE accordingly:
d0, -,:Q2'm mi.
difce ((.))(0)) me)),
(6)
dū, Bz(V) bt).
where @: = diag(v/)
B.2.2 AUXILIARY THEOREMS AND LEMMAS
Theorem 4 Zhou et al.
(020
Suppose Assumptions hold. Assume the sequence {(0,, me, Uz)) are
produced by (6). Let St
(Vwres) with ht, = 3. q (1 3) wi (1 -55--1 define
|| = E, m.2?. Then for Lévy-driven Adam SDEs in 6 its Lyapunov function L(t) = f(O.) f(0)) +
| with the optimum solution 8" in the current local basin Ω obeys
2μr
322
)))
C(t) < exp
3i8 + HT 31 -1-44
14

---

Under review as a conference paper at ICLR 2022
where A = - 1 -(0)) due to mo
0. The sequence (0.) produced by (6)
obeys
| 0-
2A
2pT
(3:
82
))
exp
(T BIu- + +uT
P
4
Lemma 2 (Zhou et al. (2020). (1) The process & in the Lévy process decomposition can be decomposed into two processes & and linear drift, namely,
& = + Mott
(7)
where E is a zero mean Lévymartingale with bounded jumps. (2) Let ā E (0, pm = R(gi) and T, = e for some 0 >> 0: po()= - > 0 and Op = (6) = I- Suppose v sufficiently small such that (11) v -02 and vT 2 + ((1)) 2
with constant C = 1oocusi u² (n)) E (0, +o0). Then for all 5 E (0, 60) A E 0o) there are po pu(6) = f and Vu vo(6, p) such that the estimates
||oT., 11 || 11 < u and P([VET. 2 v*) S exp(-u
hold for all p E (0, po]
and E (0, vo]
Lemma (Zhou et al. (2020). ). Let d' E (0. 1) and giz0 be a bounded adapted cádlág stochastic process with
values in R To
e > Suppose supezo Ilg'll is well bounded. Assume Po -u(()
1-5
16 > 0.
Op- (5( = 1-5
>
0. Pa = G. For & in (7) there is & = 5o(d) > such that for all p E (0, Po) and
3
0 E (0, 6o). it holds
d Σ i=1
L/' go de, 2 v0) S 2xxp P-u
sup
v
OseST
for all p E (0. po] and < v S uo with vo = v(p), where C represents the i-th entry Es.
Lemma Zhou al.
20200). Under Assumptions hold, assume ó E (0, 1), Po pu(0) 16 16(1+))ms)
1-8
>
0, 0. (6)
0. po min( Kl+ m1)
p).
24
S
w 8o where K1 = v IT -I and
og
C2 Fini +xo (Bi )) in Adam-alike adaptive gradient algorithms. For all e E (0, p (0.
Bpr
0 <<< to with vo (Ø). and do = 0p. we have
200) s 2exp(-u- F), 8.11 2 -
sup P sup
(8)
#g en
Nosteo
where the sequences 0g and 0, are respectively produced by 33 and 66 in adaptive gradient method
B.2.3 PROOF OF THEOREM
Proof. The idca of this proof comes from we showed Lemma 4 where the sequenec Oe and 0, start from the same initialization. Based Theorem
know that the sequence {0.) from (6) exponentially converges the minimum 8* of the local hasin Ω. To escape the local basin 52, we can either take small steps in the process ( or large jumps Jn in the process e. However, (8) suggests that these small jumps might not be helpful
for escaping the basin. And for big jumps, the escaping time I of the sequence {0.} most likely occurs at the
time if the big jump eJi in the process is large.
The verification of our desired results can be divided into two separate parts, namely establishing upper bound for any u > 1. Both of them can established based on
and lower bound of E [exp (-um(T)@(r" "}r]] the following facts:
- 8' @(v-1)
[e (Ro "ouJe 52="" [IJeelll S =R) P (Ht. Doruda & =5="" .IlJJell S $R) 112"",||uJell
A(u-o))
0(0-1)
IP (R JouJ & Q. [|:J s R) P (Rg. vig $ 22 [|JJell2 S R) s 0(u-5))
& 22 - PP Lg. & 22 |J 11 S R) < 8' O O (v-5)
TP (
99)
um ( (T) we consider both the big jumps the process e and small jumps in the process Ç which may escape the local minimum. Instead of estimating the escaping time I' from Ω. we first estimate the escaping time = from Ω P. Here we define the inner part of Ω S P == {u 22 dis(a v) 2 7). Then by setting ক + we can E for decent estimation of T.
Specifically, for the upper bound Texp (-um
15

---

Under review as a conference paper at ICLR 2022
We denote = where 7 constant such that the results of Lemma 2 hold. So for the upper bound mainly focus on E in the beginning and then transfer the results to r. In the beginning, we can show that for any it holds that.
+:00
exp (-um(T)@((
B
u (T))
M { = tu + Ress
where
E
E (ta-1, tis)
if (-1.0)
Resk
RR
1 { (tx-1: tx)
}
if E +00).
Then using the strong Markov property can bound the first term E
= tx
R1
+:00
Σ
B
(m 1)4h [T = tu
a. (1+ p/33)
- an((1 - p)
h-1 1 + 400
1400 u0w
k=
a. (1 + p/3)
1 av p)
h-1
uny
1 + p/3
1 1+ 1
On the other hand, for the lower bound of E Texp (-um (m we only consider the hig jumps in the process which could escape from the basin, and ignore the probability that the small jumps in the process find lower bound by
( which may also lead to escape from the local minimum 0" Specifically, discretization:
E Texp (-um(())()
() tu
Then we can lower bound each term by three equations do E 22-
we just listed here, which implies that for any
an
t00
ar. (1+
k-1
1 1-p
P
1 + mov
1+u+p
where p - Oas + O. The proof is completed.
B.3 PROOF OF PROPOSITION I
Proof. Since we assumed the minimizer 0" = 0 in the basin Ω which is usually small,we can employ second- order Taylor expansion to approximate Q as a quadratic basin whose center is 0". In other words, can write
1 2 2= { E Rd f(07) + デy H((")y by S h(0*)
where H000)
h((0) basin height. Then according to Hessian matrix at function
Detinition 2 we have
r r= f=
T Ie y Eg- Ry- H(" (1000)) RRg Ea-y 21}
Here Rg. = lime, - diag( Vieg(Vve//1- 155) is matrix depending the algorithm, h; = 2(h(0') f(0))) and Eg. is independent of the alogorithm, i.c. the same for Adam and AdaMomentum. Firstly, we will prove that uf (ADAMOMMNTUNN) 2 vy ((DDM) when + To clarify the notation, we 6,, my, Vt, St. to denote the symbols for Adam and 0r, 17,, Ve, for AdaMomentum, and & is the gradient noise. By using Lemma and above results, we have 0; = 0e so 8r before escaping when is large, and thus v = lime,->0 (Vf + Ge and Ue = limo,-to [81m,-11 + (1 f(O.) + 2. We will firstly show that E(Te) 2 E(ve) when t is large.
(B(..) B( lim (V)(0.) c: @ lim E( + +.)3
lim 00
(B(V)((A))) + 22 (0.))) + RE(T))
E lim Vf(0,)²) + lim E(2V (()) )) + lim B((?) $.-5
lim B(C)):
16

---

Under review as a conference paper at ICLR 2022
where (i) and (ii) are due the dominated convergence theorem (DCT) since have that we know both f(,,ll and f(:)) + Çalla could be bounded by H in Assumption And (iii) is due the fact that ff(0") ) = since function f attains its minimum point at 6", and Ç. has zero mean, i.c.
lim ((V)()))):) lim ((V)().))B(G)) O. A:-4"
And similarly we can prove that,
B(%.) E
lim [8mm- + (1 -1- 3.) +(())
[B( lim - 3:) f(.)) + + -1)
@ of
B(m--1)
(1 31)² lim B(())
02-
0:-0
where we can get the equality (i) simply by the same argument with dominated convergence theorem we just used:
lim B(V(f(0)) B( lim v(((O)))) 20. 0.
0.-0
lim E(V(f(()G)) = Bi lim V(((O.),.) @
lim E(m.-I(Vf()) + (.) = E( lim M vf(O.)) lim ( 0--00
0, ---0"
P--0"
where get the equality (i) and (ii) since the function ((,))² and ((,)), could absolutely bounded by H² And the first term equality (iii) since we have ||me-lll
1112 S H by its definition and (((0") = 0,
and the second term vanishes since the noise &e has zero mean. Based on the Assumption 5 we have
B(M-1) 2 81 B(??)
which implies that (B);)) > ((te) when large. further indicates that Rt We consider the volume the complementary set
(AAAMMMMNTUU
R R(ADAM)
r {or Rd
Eg- RT. (.0 Eg- < hi).
which can be viewed as dddimensional ellipsoid. We can further decompose the symmetric matrix M = Eg. Rgg H(O0) RR- Eg. by SVD decomposition
M AU,
where U is an orthogonal matrix and A is diagonal matrix with nonnegative elements. Hence the transforma- tion Uy is orthogonal transformation which means the volume of Y equals the volume of set
{ E E
Av'
< h2
Considering the fact that the volume of a d-dimensional ellipsoid centered at 0 Ea(r) = { 22, ,In)) d < is
v(Ea(r)) I(2 IR.,
and the fact we just proved that 12 (AAAMMMMMNNTUM) R2( Therefore deduce the volume of (AnAMOMNNT))) is smaller than that of (AAna which indicates that for Radon measure m(-) we have m (((AAMMMNnNN)))) 2 (A(AM)))) Based Lemma we consequently have ((T(AAMMOMISTII)) 2 (AD(A))))).
C PROOFS IN SECTION
C.1 PROOF OF THE CONVERGENCE RESULTS FOR THE CONVEX CASE
17

---

Under review as a conference paper at ICLR 2022
C.1.1
PROOF OF THEOREM I
Proof.
Firstly, according to the definition of AdaMomentum in Algorithm |
by algebraic shrinking we have
mis
T-1
m
(E - 7,
Vivi,
v TTE (1 - --)),3 3m3,
m
E3= 11/-1 B1, - ) ) 11 B - 14955.)
Viv,
VTE (1 - 02) 3m3,
T-1 mi
E 8f " (8.
__
Vior.
T(I 3₂) Σラ 85 3
Σn 87 9g3
mi.i
Vive,
31 T B2) 2 82 m3
m
-
Viv,,
(1- - (1-8.) T(1 32)
- 1/ -/ (Ein (1- - B
mi.i
+ (i-B))TT 33) 5=1
Vlve,
Di 85--1 ((1 -Bi))gaa)
87-2
Mid
Vive,
(1- T(1 - 832)
32 '11 )2.
mis
93.45
Viels
(1 32 / T(1 B2)) 1=1
where (i) arises from 31,t S 81, and (ii) comes from the definition that 7
2. v
Then by induction, have
r
mi
Σ
Vtu
(1 B1)2
111
32)
T
(1 8
32
9i.
j=1
(
(1- 8/ -11-- 32
D1.1
j=(
T
T
7
(1 81 V 82
g1,
Vi
1=l
<
9 v)v
(1 8,
B2 t=1
T
(1 (1 7) 32
Vt
1l
(1 3:) - 7) - B3
t
m
/oglogT T
11:.,,1112>
(1 51) 7) Br
where (i) exchangings the indices of summing, (ii) employs Cauchy-Schwarz Inequality and (iii) comes from the following bound on harmonic sum:
T
Σ
+ logT.
18

---

Under review as a conference paper at ICLR 2022
Due to convexity of ft, we get
f((0.) - f:(0" S [0; -0)
(0, 0,).
(10)
==
According the updating rule, we have
me
0++1
02
Vui
fI.t
1 31.t
0.
m
+
g1
(11)
Vei
Substracting 8", squaring both sides and considering only the element vectors, we obtain
(0t41,1- - 03)²² ( - -²,)² 20c
BB
Vursi
11
1 Br,t gr.i) (0,, 03: + +
Vor,
/ue,
mi,s
By rearranging the terms, we
20::
1 - BI,t (0.. 07)) = (Bc,, -0-3))=(00+11.003 -20
BIe Vor
mi 1,,1, -6,)
me,i Vers
Vor,
Further we have
(0,, (8;)
Vt,
[B.,- -0,)² (0e41.1-7))
/uL/VL. Vt.
mass
2
20 (1 )1,t)
2(1 BI,e)
B1.t
(0): 0e,.)m--1
31,
(0..; 0%))² -0ct1.1 (0+t1.1-:))+ +
m
20 (1
B1,t)
2(1 BI.e)
+ 1 31,t
B1
(0), 0c.)) .Va.
14-1.
va
1
Vur,
m
1(01,s - 071)² -00 03)) 2(1 B1)
(12)
20 (1 B1)
B1.
Bia
m
+
(0., Oc,)) Voi. Vot. +
(13)
20.(1 BI,e)
2(1
B1)
Vtvr,"
where 13 bounds the last term 112 by Cauchy- Schwarz Inequality and plugs in the value of ou. Plugging into 11I and summing from & to T, we obtain
R(T)
gr,(0e, - 07))
VeL.
m
20 - -B) [[0c,- -0,)² - -(0 -
2(1 - B1)
Vtur.
31,
T
d
Bia
m
(0'; 0..))² /u.. +
20 B1,1)
2(1 B1)
ytvr,i
1m km1
1=1
1=1
T
Vu,i
(01,1
0²)² 2(1 31)
(O., 0
(14)
201
(1
81)
at
ot-1
1
1m2 1ml
31
(0" 0c.)² Ver. +
a
m
(15)
20
31)
B1
/tv,s
19

---

Under review as a conference paper at ICLR 2022
where L3
rearranges the first term of (14) Finally utilizing the assumptions in Theorem
gel
R(T)& S
V1,
D
+
Do
Vu,i V
20x
[1
31
2(1
31)
a2
au
i=1
D
T
av1 + log'r'
1g1.T.ll2
2(1
31)
C1
(1
B1 (1
2)v
1=1
d
VT
D
T
D
2
81)
2(1 31)
a
1=1
t=
0y 1 log
1g1.7:l12
(16)
31
1)v B2
which is our desired result.
C.1.2 PROOF OF COROLLARY 1
Proof. Plugging a Vi and 31, = 31 into
16 we get
D2 VT
D
R(T)
UT, 2a(1 {11)
31 Vters
21 (1)
ay u log Tog'
11g.,,1ll2.
(17)
(1 7) - 82
Next, we employ Mathematical Induction to prove that vi, S Goo. for any 0 E DEITT,1SiS d. Vi, we have
mb,1 = 0 I G Suppose 11-1111 & Go., we have
m (31,me-1, + B1.))gr.))
531.m + (1-B1,t)gi.
$ 11, (1 (1- B3 G
where (i) comes from the convexity of function f x². Hence by induction, we have mi,e S G2 for all 0 v Ghe Suppose U1-1, S G we have 0 I < T. Furthermore, Vi, have Ua,,
Vt. 32 1.1 + (1 32)mi
- 3r + (1 B2) = C
Therefore, by induction, we have Vt,i S G2 Vi,t. Combining this with the fact that Ed. Ilu S dG VT and (17). obtain
dG D2.VT + R(T)
dC D2 81 2a(1 81)
dC dG Tog T
Σ V +
(18)
20(1 - 31)
(1 (1- (1))))(11 (1-3))
For Etn X2 V., we apply arithmetic geometric series upper bound:
EAVis
Σ
ix' (1 2)2
(19)
1=1
Plugeing 19 into (88) and dividing both sides by T. we obtain
R(T)
dG>av/ + T
dD Goo
dD Goo
(1 -
BAT 2a(1 - 3.T 2n(1 Bi)(1 XY2T
which concludes the proof.
C.2 PROOF OF THE CONVERGENCE RESULTS FOR THE NON-CONVEX CASE
20

---

Under review as a conference paper at ICLR 2022
C.2.1 USEFUL THEOREM
Theorem 5. (Chen et al 2019) Suppose Assumptions and 4 are satisfied, 91, is chosen such that 0 s
S1.441 S SI.1 < 1,0 82 V 0. There exists some constant such that |
my
G, Vt. Then
yot
Adam type algorithms yield
E
Vf(O)// ae (Vf(0.),
S
2
C2
C2
a1-1
C3
Ct
Qi-1
Ca.
(20)
Vt
V
V
where C1, C2 and C3 are constants independent of and T, C4 is constant independent of T, the expectation taken w.r.t all randomness corresponding {gr]. Furthermore, let r minse(4] min 1 /aa denotes the minimum possible value of effective stepsize
time I over all possible coordinate and past gradients {9)) The convergence rate of Adam-type algorithm is given
81 B =
min E
$2 (T)
where (T) is defined through the upper bound of RHS of 220 and Σ 71 ((x2(T)).
We present the proof of this Theorem in subsection C.2.4 and C.2.5
for completeness and reader's convenience.
C.2.2 PROOF THEOREM
Prouf. We will first bound each term RIIS of Equation have that
20
Given all conditions in Theorem hold,
Σ Jaig
2
a1
V
cg |l
1
2²2
(21)
where (i) arises from fact that 0 < c vc, VI e [T] and inequality (ii) based on || S H, e [T]. Then,
VI
T
Oz
0t-1
E
Ot-1
Oi
Vei
v
01
Ut-
1,1
Ut,
C1
OT
E
U1,
VTT,
S B
U1,4
da
(22)
Ve'
and here (i) holds since we assume that
2
Gr+1
VL E [T]. and (ii) comes from the fact that < c S
Vot
Ve. O < as S ax Vt [T]. Next.
12
E
Σ
at-1
=
E
Ut
1,
(
R
01-1
/U-1,
Ve
2
(23)
21

---

Under review as a conference paper at ICLR 2022
where we have (i) because
On the other hand, we can obtain
- 1.7
a lower bound of the LHS Equation
20
T Σ
V((O)) VoL
)
Tor H (E[2)
E
VS(O))
at
min E (V)())))
(24)
where the last equality comes from the fact that is weighted average of m² and ||mall2 I H since my is also an exponential moving average gr-
By combining the results in (1).
22
23
and
to
we obatain
ToT
min E
f(:))
H
T
Vs((.)
E
V.(O.):
Voi
Lt=l
T
2
SE
C1
U1
g+
C2
ax
+ C3² Σ
at
a=1
+ Cc
t=1
1=1
Vui
V
t=1
Vtr
2
C
C2da
Csda²
CiH
da
da²
ai
+
+C
+ C2
+ Cs
+Ca.
c
Ve
Ve
After rearrangement, we easily deduce that
H C.HlT((T)
+ da
+ +Cs da²
min BI
< Tor
+C
EE|T
vc
e
(Q1 + Q2n(T)),
(25)
Tar
where
C2 da
+ +Cs da²
+ci). Q2= = H
Q1 H
Ve
c
C.2.3 PROOF OF COROLLARY 2
By choosing a = aa//tt/ a, Vt [7', we have
Tor = avT, (T))
Σ a a
S a² (1 + log(T)).
Combining this with 25p and making some rearrangement, have:
H
Ci a IT (1=
log(T)) = C2 da
+ Cs do²
min EIVS
+ Ca
avT
1
QQi + (ooo(T)).
VT
where
Haa
+C2 -Ca da² +ci), +Ci), Vc
da
CiH
Qi H
Qi
C.2.4 TECHNICAL LEMMAS FOR PROOF OF THEOREM
In this section, introduce seven useful lemmas.
Lemma 5 (Chen et al. 2019 ), Let 0p A 0, in the Algorithm. consider the sequence
81,1
ze Or 1- 31,0
(0. 0--1), 2.
22

---

Under review as a conference paper at ICLR 2022
The following holds true:
$1,141
31,1
ammy
Z++1
2t
81,141
$1,t
V
$1.
01-1
M4-1
>
1,
S1,
Vii
V
and
B1,2
31,1
01
22 21
1
31.2
f1,1
V
01
Lemma 6 (Chen et al.
2019) ). Suppose that the conditions in Theorem
hold, then we have
B [[f(ze11
f(z))) -
Σ
T
where
Ti = -B
V f(z);
Bi
as-1
mi-1
))
(26)
B1,
Voz
| ves
gi a1 vz(2).
T2 =
(27)
31.41
B:
a.m;
T3 = E
(28)
- B1,441
T4 = E
Σ
3L
31,i41
B1,
(29)
2
B1,1+1
1 1
B1,
3L
f1
0:--1
T3 = E
(30)
2
$1.4
Vei
Vi-1
3L
To E
(31)
2
1=1
Lemma 7 (Chen et al. (019)) ). Suppose that the condition in Theorem 5 Thold, then for Ti in (26 it holds that
T -
V(())
31.5
0ti-1
m;-1
1-
B1,5
V0--11
2
8
E
d
a
0
1
Lemma (Chen et al. 2019)). Suppose the conditions in Theorem are satisficd, then T3 bounded as:
28 can bc
E
Σ
f((s))
31
Br
aam
I- B1s+1
1- B,
31
B1.t11
(H2 + G2).
Bi
31,141
Lemma (Chen et al. 2019)) ) Suppose assumptions Theorem are satisfied, then for Tç (29). holds that
3L
31.141
BI.1
CuiT V
E
2
B1,111
1
BI,5
1=1
3L
Bi
B1,441
G².
2
1-
81
1
B1.e11
Lemma 10 Chen et al. 2019 ). Suppose the assumptions in Theorom are satisfied, then for T3 in
30), we
have
To
3L
$1,1
ai
a:
M,-1
2
1 1
B1,
Vii
Vūi-
2
3L
31
H
0:-1
2 1 1 31
/
23

---

Under review as a conference paper at ICLR 2022
Lemma 11 (Chen al. 2019 ). Suppose the assumptions in Theorem S are satisfied, then T3 in 27 be bounded as:
T2 = - E
V
(=:);
Ji
Vor
<B
040
L
81
E
[-1
Oige vu;
27
2
V
-
31
31
1=2
J=1
i=2
+T2
81
1
E
0:-1
1- 31
81
Voi-1
+
ax
04-1
V
0i=1
i=2
+ +
E
Σ
1
C.2.5
PROOF OF THEOREM
Proof. We can prove Theorem alter combining Lemma Lemma it holds that
6 7
9
10 and
Specifically, firstly based on
E (((4))))(
T
B1
Br.1
2
-F
L
a
Ve:
+ E
L |la.gs/ vvall
31,411
1- 11,1,1/
+ E
3
L
B1,3
a2
a-1
Mi-1
2
B1,5
(ii
Vo-1
E
V(()),
B1, ( aj
a:-1
mm-1
- E
(V((x)).
1 1 Br,
(o.
vi-1
()()))) (1 B1,111
Br,
vinl )
E
asmis/
I
B1,541
1 81,5
Then can combine Lemma
11 and further merge similar terms. We have that
24

---

Under review as a conference paper at ICLR 2022
R [ff(+++)) - f(x)] s
<Σn Ti
<11
81 E
0: 1
1- B1
V
v
1=2 j=1
+
31
B1,t+1
+
+
+
3L
B1
11,t
2
a²
1
-
$1,041
2
1
$1
BI.
3L
31
H
Σ
1
d
0:-1
2
1
V1--1
(1=2 )=1
+ +E)
1
0:0i
L
B1
2
2
E
aigi
2
Vv
1 1-
B1
-
8g
Voi
j==
d
+
HH
f1
E
a
31
1
$1
Vii
v
Lizl
i=2
11
d
+2H²E
ΣΣ
04-1
2
01
Ve,
V
Vui
1=2
- EE Σ a: Vfzz
Finally, after rearrangement and some calculation, it can be verified that
04
R
C1
laigt/
V
C
Σ +
a1
a--1
1=1
T
C33
Σ
ax
a:-1
+ CCa
vi
/
t=1
where we have
3
1
B1
C
A
+
2
2
1- 81
1
31
B1
C2
A
22
2H²
1 81
C3
1+
1
B1
112
Bi
1 B1
1- 81
1
$1
C4 A
B1
(IT2 + G2)
+
31
G² + 21 /|n//oill BBf(2)) - -f(=")l
3r
3r
and z" is an optimal where f()) takes its minimum. This result directly implies Theorem 5
D ADDITIONAL EXPERIMENTAL DETAILS
Hyperparameter tuning rule For hyperparameter tuning, we perform extensive and careful grid search choose the best hyperparameters for all the baseline algorithms.
For SGDM, we the momentum parameter 0.9 and search the optimal learning rate among set (30.0, 1.0, 0.0015, 0.001, 0.0005, 0.0002) for all the experiments.
For adaptive gradient method baselines (Adam, AdamW, Yogi. Adabound, RAdam and AdaBeliet), we search for 31 among (0.5, 0.6, 0.8, 0.9). Bo among {0.9, 0.98, 0.99, 0.999, 0.9999). a among (0.1,0.01, 0.0015. 0.001, 0.0005), weight decay parameter among {1.2 x 100 10-1, 1,5 10-4] and e among {10 10 12 10 161 for all the experiments except ImageNet. All the additional method-specific parameters
25

---

Under review as a conference paper at ICLR 2022
carefully chosen according the original paper setting. Due extremely large computing workload on ImageNet dataset, we set 81 0.9, 32 0.999, a 0.001 and methods, and finetune weight decay parameter from set {0, 1x 10 5 10 5x100 10000003000 2,0 10-2) ?) for the values
10 for all the adaptive gradient
we reported as we run.
employ the default parameters as 31 0.9 and choose learning rale and weight decay parameter using same parameter searching scheme the adaptive gradient baseline methods. 82 is set as 0.999 for all the tasks. We Choose tasks. We find this setting is universally ample for satisfactory performance, which further demonstrates the superiority of AdaMomentum that little tuning effort is needed.
For our AdaMomentum,
10 image classification tasks and x 10016 for other
All the experiments reported are performed on NVIDIA GeForce RTX 2080Ti GPUs with Intel Core i7-8700K 3.70GHz CPUs. We provide some additional information concerning the empirical experiments for complete- ness.
D.1
IMAGE CLASSITICATION
Table 7:
Well tuned hyperparameter configuration of the adaptive gradient methods for CNNs on
CIFAR-10.
Algorithm
Adam
AdamW
Yogi
AdaBound
RAdam
AdaBelicf
AdaMomentum
Stepsize a
0.001
0.001
0.001
0.001
0.001
0.001
0.001
D1
0.9
0.9
0.9
0.9
0.9
0.9
0.9
B2
0.999
0.999
0.999
0.999
0.999
0.999
0.999
Weight decay
5 x 10 4
5 x 10 4
5 x 10-4
5 x 10
5 x 10-4
4
x 10
5 x 10
10 10-5
10-8
10-8
10-8
10-8
10
10 10-5
CIFAR datasets The values of the hyperparameters after careful tuning of the reported results of the adap- tive gradient methods on CIFAR-10 the main paper summarized in Table 7 For SGDM, the optimal hyperparameter setting is: the learning rate is 0.1, the momentum parameter is 0.9. the weight decay param- eter is 5 x 10- For Adabound, the final learning rate is set as 0.1 (matching SGDM) and the value of the hyperparameter gamma is 10
Table 8: Well tuned hyperparameter configuration of the adaptive gradient methods for CNNs on ImageNet.
Algorithm
Adam
AdamW
Yogi
AdaBound
RAdam
AdaBelief
AdaMomentum
Stepsize a
0.001
0.001
0.001
0.001
0.001
0.001
0.001
B1
0.9
0.9
0.9
0.9
0.9
0.9
0.9
82
0.999
0.999
0.999
0.999
0.999
0.999
0.999
Weight decay
I 10-1
x 10
x 10-1
I x 10
I x 10-1
x 10-2
5 x 10-2
10-8
10
10-8
10-8
10-8
10-8
10-b
The values of the hyperparameters after careful tuning reported results of the adaptive gradient methods on CIFAR-10 10 in the main paper is summarized in Table[8] For SGDM, the stepsize is 0.1, the momentum parameter is 0.9 and the weight decay is 5 x 10"4.
ImageNet
D.2 LSTM ON LANGUAGE MODELING
Table 9: Well tuned hyperparameter configuration of adaptive gradient methods for I-layer-LSTM on Penn Treebank dataset.
Algorithm Adam
AdamW
Yogi
AdaBound
RAdam
AdaBelief
AdaMomentum
Stepsize
0.001
0.001
0.01
0.01
0.001
0,001
0.001
Bi
0.9
0.9
0.9
0.9
0.9
0.9
0.9
B2
0.999
0.999
0.999
0.999
0.999
0.999
0.999
Weight decay
1.2
10 4
1.2 10-4
10-4
1.2
10
4
1.2 10
x 10-4
1.2 x 10
10-12
10-12
12
10-3
10-8
10-12 12
10 16
10-18
26

---

Under review as a conference paper at ICLR 2022
NaWremmun SCBV
SGOM
5COM
Adum
Adam
Train Perpleatty
Train Perplexity
Train Perplextty
Vegt
Addlbuud
Adblourd
R.
RAdai
Epoch
Epach
Epoc h
(a) L-Layer LSTM.
(b) L-Ler LSTM.
(c) 3-Layer LSTM.
Figure 5: Train perplexity curve on Penn Treebank dataset.
Table 10: Well tuned hyperparameter configuration of adaptive gradient methods for 2-layer-LSTM on Penn Treehank dataset.
Algorithm
Adam
AdamW
Yogi
AdaRound
RAdam
AdaBelier
AdaMomentum
Stepsize
0.001
0.01
0.001
0.01
0.001
B;
0.9
0.9
0.9
0.9
0.9
0.9
Bg
0.999
0.999
0.999
0.999
0.990
0.999
0.999
Weight decay 1.2 10-*
1.2 x 10
1.2 x 10
1.2 10
1.2 x 10
1.2 x 10
1,2 x 10
10
10
10-5
10 10-8
10-12
10
10 10-18 18
Table Well tuned hyperparameter configuration of adaptive gradient methods for 3-layer-L.STM on Penn Treebank dataset.
Algorithm
Adam
AdamW
Yogi
AdaBound
RAdam
AdaBelief
AdaMomentum
Stepsize
0.01
0.001
0.01
0.01
0.001
0.01
0.001
Bi
0.9
0.9
0.9
0.9
0.9
0.9
0.9
0.999
0.999
0.999
0.999
0.999
0.999
0.999
Weight decay
1.2 10
1.2 x
4
x 10
1.2
1.2
x 10
1.2 x 10
10-12
10-12
10
12
10
10
10 10-12
12
10
10 10-18
27

---

Under review as a conference paper at ICLR 2022
The training perplexity curve is illustrated in Figure 5 We can clearly that AdaMomentum is able make the perplexity descent faster than SGDM and most other adaptive gradient methods. In experimental settings, the size of the word embeddings is 400 and the number of hidden units per layer is 1150. We employ dropout in training and the dropout rate for RNN layers is 0.25 and the dropout rate for input embedding layers is 0.4.
The optimal hyperparameters of adaptive gradient methods for I-layer, 2-layer and 3-layer LSTM are listed in Tables and respectively. For SGDM, the Well tuned stepsize is 30.0 and the momentum parameter For Adabound, the final learning rate is as 30.0 (matching SGDM) and the value of the hyperparameter gamma is 10
D.3 TRANSFORMER ON NEURAL MACHINE TRANSLATION
Table 12: Well tuned hyperparameter configuration of adaptive gradient methods for transformer on IWSTL'14 DE-EN dataset.
Algorithm
Adam
AdamW
AdaBelief
AdaMomentum
Stepsize a
0.0015
0.0015
0.0015
0.0005
B
0.9
0.9
0.9
0.9
Bg
0.98
0.98
0.999
0.999
Weight decay
10
10
10
10
10-5
10-5
10-16
10-16
transformer on NMT task, the well tuned hyperparameter values are summarized Table 12 The stepsize of SGDM 0.1 and the momentum parameter of SGDM is 0.9. Initial learning rate 10 and the minimum learning rate threshold is set as 10 in the warm-up process for all the optimizers.
28


Reviews:
[{"review_id": "hSe5EZT3Qtb", "reviewer": "Reviewer_hk95", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: the empirical results are good for language modeling, and it is comparable with SGD with momentum on vision tasks.\n\nWeaknesses: my main concern of this manuscript is limited technical novelty and minimal insight. \n\n1. Lemma 1 leverages the tool of SDE to show closed-form formula of escaping time. However, the analysis is almost the same as the Theorem 1 in [Zhou et al. NeurIPS 2020]. One only need to change some definitions in the proof of [Zhou et al. NeurIPS 2020] and then can easily prove the Lemma 1.\n\n2. Assumption 5 is the key to show the improved escaping time in the proof of Proposition 1. It is an unrealistic assumption, it is possible that the RHS is smaller than LHS when $t$ gets large. Please give more explanations. \n\n3. In Theorem 1, the authors require the iterates to be bounded. One cannot simply assume it holds without proving it. \n\n4. In Theorem 2, the authors assume $\\alpha_t/\\sqrt{v_t}\\geq \\alpha_{t+1}/\\sqrt{v_{t+1}}$ to prove the convergence in non-convex case, which is problematic. For example, [Reddi et al., ICLR 2018] provided a non-convergence analysis of Adam and they showed that this inequality does not hold at all.\n\n\n\n\n", "summary_of_the_paper": "This manuscript proposes to substitute the gradient in the second moment estimation term with the momentumized version, and show that it improves both optimization and generalization. Some theory is presented, and there are also some promising deep learning experiments on different tasks.", "summary_of_the_review": "Overall, this manuscript presents an interesting modification of Adam to improve empirical performance. However, the technical novelty is very limited and there is no new theoretical insight. The generalization analysis using SDE comes from [Zhou et al. NeurIPS 2020], and the optimization analysis is based on a problematic assumption pointed out by [Reddi et al. ICLR 2018]. There are also some unrealistic assumptions (e.g., Assumption 5) which needs to be further justified. \n\nFor its current version, it clearly does not reach the bar of top venues such as ICLR.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gB59fFh1InA", "reviewer": "Reviewer_mUyQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nI appreciate the authors' efforts to provide both theoretical and experimental validations on the proposed method. The authors try to understand both the convergence and generalization of the algorithm, and the experiments also cover a wide range of neural network architectures and tasks.\n\nWeakness:\nDespite that I appreciate the authors' efforts, I found potential flaws in both theory and experiments, listed below.\n\nTheory:\n\n1. Fig1 as a demonstration of the idea does not make sense to me. It essentially states that AdaMomentum is easier than Adam to be trapped at the 1st local minimum it encounters. But the question is, why the local minimum you encountered early is the global optimal? Give me an example that behaves like in Fig.1, I can easily modify the loss function to the right of $\\theta_{t+1}^{(2)}$ as the global optima, and keep the function to its left the same as in the figure (so that the behavior of Adam and AdaMomentum don't change until they pass $\\theta_{t+1}^{(2)}$). In this case the right minimum is better than the left one, and Adam would jump in the global optima while AdaMomentum will be trapped in a bad local minimum.  \nIn short, which minima is the global minima can be easily changed, AdaMomentum sticks to the 1st one, but the 1st minima can either be a local or global minimum. You only pick the case that's good for AdaMomentum, but ignore the bad case.\n\n2. The generalization part mostly follows [1], however, I think your results contradict [1]. [1] stated that SGD has a **smaller** escaping time than Adam hence SGD can escape local minima and generalize **better**. Your paper stated that AdaMomentum has a **longer** escaping time than Adam but still generalizes better? There's a clear contradiction here.    \nI might miss something with the \"flat basin\" part, I think it's referring to assumption 4. After going over the proof, I think you will have $\\mathbb{E}\\tilde{v} \\geq \\mathbb{E} v$ for both sharp and flat minima, hence AdaMomentum always has a longer escaping time than Adam. At least this implies that AdaMomentum is more likely than Adam to stay at a bad local minimum if they fall into it, this would only hurt the generalization rather than improve it. \n\n3. The proposed method essentially slows down the change of denominator, how does this compare to using a larger $\\beta_2$?\n\n4. The current version of AdaMomentum scales lr up by a factor of $1/(1-\\beta_1^t)$ (see comment Experiment 1 below for an example), which contradicts findings in RAdam that warmup from a small lr to a large lr would stabilize the training. I wonder what's the authors' comment on this?  \n\n5. Discussion on $\\epsilon$. First I don't think the equation above Sec3.2 is correct, it should be $v_t^\\prime=v_t + \\epsilon/ (1-\\beta_2^t)$, as $t\\to \\infty$, $v^\\prime \\to v_t + \\epsilon$. Second the authors confuse $\\epsilon_{Adam}$ with the $\\epsilon$ here, in general $\\epsilon_{Adam}$ is outside $\\sqrt{.} $ but $\\epsilon_{AdaBelief}$ is inside $\\sqrt{.}$, that's why default is $\\epsilon_{Adam}=1e-8, \\epsilon_{AdaBelief}=1e-16$. Considering this, $\\sqrt{\\epsilon_{AdaBelief}} \\approx \\epsilon_{Adam}$, and AdaMomentum uses the same default as AdaBelief, I don't think the arguments by the authors are convincing. The effect of $\\epsilon$ is amplified because you move it outside the $\\sqrt{.}$, but this effect is corrected by using a squared value (1e-16 rather than 1e-8) as default.\n\nExperiments:\n\n1. Toy examples. It's very hard to notice until I checked the code that I found the authors only plot the behavior of the first 30 steps or so. Note that $\\mathbb{E} m_t = (1-\\beta_1^t) \\mathbb{E} g_t$ in general. This implies that, AdaMomentum scales the denominator by a factor of $1-\\beta_1^t$, and the effective stepsize would be larger than Adam by a factor of $1/(1-\\beta_1^t)$. When $t$ is small, this factor is very large (e.g. $t=1, \\beta_1=0.9$, this factor is 10). This means the fast convergence of AdaMentum with few steps is heavily influenced by the large effective learning rate, rather than the $m_t$ vs $g_t$ issue. The authors need to consider this in the statement and experiments.\n\n2. I was surprised to see the authors reported an FID of 12.06, so I ran the code twice, and got 12.68 and 12.80, which is way much worse than reported, even worse than baseline Adam. In case the authors uploaded the wrong file, I would run it again if the authors can re-upload the code (in a single zip file please, from the previous link I have to download files one by one).\n\n3. ImageNet experiments with ResNet18. This is again tricky that the authors switched to cosine learning rate, yet the results by AdaBelief is worse than reported in the original paper using a step lr schdule. I tried ResNet18 on ImageNet with a cosine learning rate, batchsize 4096, 90 epochs, initial learning rate 0.001*4096/512, weight decay 5e-2, and got 72.2 top-1 accuracy (70.45 for AdaMomentum). I'm using Jax with the AdaBelief in Flax, all default hyper-params. The settings are slightly different from this paper, but typically large-batch training is worse than small-batch training. So the ImageNet results look unconvincing to me.\n\n[1] Zhou, Pan, et al. \"Towards theoretically understanding why sgd generalizes better than adam in deep learning.\" arXiv preprint arXiv:2010.05627 (2020).", "summary_of_the_paper": "This paper proposed AdaMomentum, which uses the EMA (exponential moving average) of the square of EMA of gradient as sthe denominator, while Adam uses the EMA of the square of the gradient. The authors conducted experiments in CNN, Transformer, LSTM and GAN to show the superior performance of AdaMomentum. The authors also tried to provide theoretical analysis.", "summary_of_the_review": "The theoretical parts on generalization contradict the literature. The experimental results also mismatch literature, and I got a far worse result on GAN training using the authors' code (could be the wrong version, I can re-run if the authors can re-upload code into a single file). I tend to reject for now and would increase the rating if the authors can resolve my comments above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "evlOyS2vije", "reviewer": "Reviewer_5ZsD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\nThe paper is in general well-organized. The technical contribution seems to be reasonable. I do not carefully check the proofs in the appendix. The empirical evaluations show that the proposed method is robust.\n\n## Weaknesses\n\n### 1.  The presentation in Sec 3.1 can be further improved.\nIt is more clear if the authors can construct a 1D example like Figure 1 and plot the trajectory of Adam and the proposed method. Readers may not fully understand the statement made in the section Benefits of substiuting g with m.\n\n### 2. Sec 4\nThe convergence analysis looks quite standard. The authors should tell readers if the analysis gives a better bound compared to Adam or other variants.\n\n### 3. General non-convex optimization (related to Sec 5.1.1)\nAdam and many diagonal adaptive-gradient methods do not work well for general non-convex functions.\nI wonder whether the proposed method has the same issue. For example, Ackley function and  Rosenbrock function are multivariate (d>1) objective functions. In Sec 5.1.1., the authors only consider the simplest case when d=2. I wonder about the performance of the proposed method and Adam for higher-dimensional (e.g., d=100) Ackley functions and Rosenbrock functions.  \n\n\n  \n\n", "summary_of_the_paper": "In this work, the authors proposed an improved version of Adam by using a momentumized version of the second moment.\nThe authors justify the proposed modification and give some standard convergence analysis.\nThe authors test the proposed methods in several deep learning tasks and show the improvement over some existing methods. ", "summary_of_the_review": "The paper is in general well-organized.  However, as pointed out by other reviewers, the empirical evaluations are not fair and the technical statement could be incorrect.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Go5XmwiJUXS", "reviewer": "Reviewer_a87S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "ttStrengths \n1. The paper is well written.\n2.The intuition of estimating  second moment by momentum  is interesting.\n3. Theoretical results and experimental result is provided.\n\nWeaknesses \n1. The assumption in Theorem 1 seems wired, where it requires the   distance between any θt generated by AdaMomentum is bounded. I am not sure whether this assumption is a widely accepted assumption or it is a special assumption required by the proposed algorithm. If yes, it will be much better if the authors can clearly state this for clarification. \n2. The assumption in Theorem 2 seems has the same confusion, where it requires alpha_t/sqrt(v_t) > alpha_(t+1)/sqrt(v_(t+1)). Whether it can be satisfied by using the setting in Corollary 2 or it still a assumption is not clear. It will be much better if the authors can clarify this in their paper.", "summary_of_the_paper": "This paper study the  adaptive type gradient methods, such as ADAM which scaling gradients by square roots of the running averages of\nsquared past gradients. Nevertheless,  they discover that substituting the gradient in the second moment estimation term with the momentumized version in ADAM can well solve the weaknesses of ADAM which  generalize worse than stochastic gradient descent (SGD) and tend to be trapped in local minima at an early stage during training.Their intuition is that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a better choice for scaling than that of the raw gradient. Thereby they propose ADAMOMENTUM as a new optimizer reaching the goal of training fast while generalizing better. Convergence guarantee is provided   and   extensive experiments on a wide range of tasks and models is also conducted.", "summary_of_the_review": "This paper proposed a adaptive type gradient methods named ADAMOMENTUM  by the intuition that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a better choice for scaling than that of the raw gradient.  The intuition, theory, and experiments seems OK. Therefore, I tend to accept this paper at this time. I am not an expert at this area correct me if I am wrong, and I am willing to change my score to align with other expert's score if needed and reasonable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to substitute the gradient in the second moment estimation term with the "momentumized" version, arguing that it improves both optimization and generalization. Some theoretical results are shown as well as empirical results.

The paper has been widely discussed by the reviewers and several weak points have been raised. Let me list some of the most important ones.
- The theory appears to be incremental and overall very weak. The authors themselves acknowledged that this "is not a pure optimization theory paper". In details, the generalization analysis is a straightforward extension of Zhou et al. [NeurIPS 2020], while the optimization analysis inherits all the known weaknesses of previous similar analysis in deep learning optimization papers. In particular, *none* of the following is correct: the use of a regret analysis for a stochastic non-convex optimization algorithm, the assumption of bounded iterates, Assumption 5, the assumption in Theorem 2 on $\alpha_t/\sqrt{v_t}$. The fact that similar mistakes were done in previous papers does not make them correct: The community should aspire at doing better not at reiterating known mistakes.
- On $\epsilon$: the reviewers correctly pointed out that moving $\epsilon$ under the square root and not changing its value is not fair. The answers of the authors on this point were unconvincing.
- Doubts on empirical results: it seems that not all the possible hyperparameters of the baselines were properly tuned. For example, despite being common practice, epsilon should also be tuned, see for example the experiments in Agarwal et al. 2020.

I didn't consider the discussion on AdaBelief because only marginally relevant to this paper.

Overall, the paper does not seem interesting from a theoretical point of view and its empirical comparison cannot be fully trusted for the presence of some weaknesses.
Decision: Reject