ADOPT: Modified Adam Can Converge with Any $\beta_2$ with the Optimal Rate
Authors: Shohei Taniguchi
Keno Harada
Gouki Minegishi
Yuta Oshima
Seongcheol Jeong
Go Nagahara
Tomoshi Iiyama
Masahiro Suzuki
Yusuke Iwasawa
Yutaka Matsuo
Venue: NeurIPS 2024 poster (2024)
Keywords: Adam
convergence analysis
adaptive gradient method
Primary area: optimization_for_deep_networks

Abstract: Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., $\beta_2$, in a problem-dependent manner. There have been many attempts to fix the non-convergence (e.g., AMSGrad), but they require an impractical assumption that the gradient noise is uniformly bounded. In this paper, we propose a new adaptive gradient method named ADOPT, which achieves the optimal convergence rate of $\mathcal{O} ( 1 / \sqrt{T} )$ with any choice of $\beta_2$ without depending on the bounded noise assumption. ADOPT addresses the non-convergence issue of Adam by removing the current gradient from the second moment estimate and changing the order of the momentum update and the normalization by the second moment estimate. We also conduct intensive numerical experiments, and verify that our ADOPT achieves superior results compared to Adam and its variants across a wide range of tasks, including image classification, generative modeling, natural language processing, and deep reinforcement learning. The implementation is available at https://github.com/iShohei220/adopt.

ADOPT: Modified Adam Can Converge with Any B2 with the Optimal Rate
Shohei Taniguchi
Keno Harada
The University of Tokyo
The University of Tokyo
taniguchi@ueblatt u-u jp.
keno keno.haradee hww.hh@webbab.bb....m.. t. u-tookyo ac. jc.
Gouki Minegishi
Yuta Oshima
The University of Tokyo
The University of Tokyo
t.u-tokyo.ac ac.
yuta. www.ma@weblab. t. u-tokyo.ac..
jp.
Scong Cheol Jeong
Go Nagahara
The University of Tokyo
The University of Tokyo
jeong@weblab. t. w-t.kko..a..... ac.
nagaharago@weblab.. t. w.tookyo.ac..ooo ac.
Tomoshi liyama The University of Tokyo i jiyama@veblab.. t. u-tokyo jp.jp jp
Masahiro Suzuki The University of Tokyo masa@veblab.c t. t.u-tokyo.. ac. jp
Yusuke Iwasawa
Yutaka Matsuo The University of Tokyo
The University of Tokyo
wwssawa@weblab. t.t..t..tttoo..... u-uooo. ac. jp
matsuo@weblab.t t. wttp://bl..a....t.t.oooooooooo u-tokkoo.ac...
Abstract
Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparame- ter, i.c., 32, in a problem-dependent manner. There have been many attempts fix the non-convergence (e.g., AMSGrad), but they require an impractical assumption that the gradient noise uniformly bounded. In this paper, we propose new adaptive gradient method named ADOPT, which achieves the optimal convergence rate of ((I/VT) with any choice of 32 without depending on the bounded noise assumption. ADOPT addresses the non-convergence issue of Adam by removing the current gradient from the second moment estimate and changing the order of the momentum update and the normalization by the second moment estimate. We also conduct intensive numerical experiments, and verify that our ADOPT achieves superior results compared to Adam and its variants across a wide range of tasks, including image classification, generative modeling, natural language processing, and deep reinforcement learning. The implementation is available https: btttp:://github com/ www.mm//thheee////aaaaeeooe
Introduction
Stochastic optimization algorithms, such stochastic gradient descent (SGD), play a central role in deep learning. In particular, adaptive gradient methods based on exponential moving averages, such as Adam [Kingma and Ba. 2014], are widely used in practice. Despite the empirical success, it is
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

known that Adam does not converge in theory in general cases, For example, et al. [2018] show that Adam fails to converge correct solution in a simple example where the objective function at time is given as:
{ CB. for t mod 3 fa (0) 1 -0, otherwise,
where C >>> 2 and f E [-1,1]. In this online optimization setting, Adam converges to wrong solution (i.e., 0- = 1) instead of the true solution (i.c., = -1) especially when the hyperparameter 32 is set to small value. There have been several attempts to fix the non-convergent behavior of Adam Reddi et al. 2018 Zou et al. 2019]. For example, AMSGrad [Reddi et al., 2018 ensures the convergence for online convex optimization by making slight modifications to the Adam algorithm. Subsequent studies [Chen et al.) 2019 Zhou et al.) 2018) show that AMSGrad also converges to a stationary point for smooth nonconvex stochastic optimization problems. However, the convergence proofs rely on the assumption that the gradient noise is uniformly bounded. This assumption is stronger than the Tsitsiklis 2000 Khaled and Richtárik] 2023), J. where the gradient variance is assumed to be uniformly bounded. In fact, the bounded noise assumption is often violated in practice. For example, when Gaussian noise is used in the gradient estimation (e.g., variational autoencoders [Kingma and Welling
used for the analysis of vanilla SGD [Ghadimi and Lan 2013 Bertsekas and
2014] and diffusion models Ho al. 2020 Song et al. 202I)) ) the stochastic gradient is no longer bounded.
Concurrently, Zhou et al. 2019) analyze the non-convergence of Adam in the problem described in Eq. (T from the perspective of the correlation between the current gradient and the second moment estimate based the exponential moving average. Specifically, they show that the non-convergence problem can be resolved by excluding the gradient of some recent steps from the calculation of the second moment estimate. Based on the analysis, they propose AdaShift, another variant of Adam. However, their theoretical analysis is limited to single online convex problem described in Eq. and the convergence of AdaShift for general nonconvex problems is unclear.
More recently, some works have demonstrated that Adam can converge by choosing 3g in a problem-
dependent manner (Shi et al. 2020 Zhang et al. 2022 Wang et al. 2022 Li et al. 2023 Wang et al., 2023]. However, tuning 33 for each specific problem is troublesome; hence developing algorithms with the bobblemiindepeneee
convergence guarantee is still important to safely apply adaptive gradient methods to a wide range of machine learning problems.
In this paper, we propose an alternative approach to addressing the non-convergence problem of Adam without relying on the choice of 32 or strong assumptions such as the bounded noise assumption. To derive our algorithm, we first examine the case without momentum, analyzing the convergence bound of RMSprrp for general smooth nonconvex optimization problems. Through the analysis, we uncover the fundamental cause of non-convergence, which stems from the correlation between the second moment estimate and the current gradient. This finding aligns with the results demonstrated by Zhou al. [2019] for online convex optimization. This correlation can be easily eliminated by excluding the current gradient from the second moment estimate.
Subsequently, we extend our findings to the case where momentum is incorporated, as in Adam, and discover that the Adam-style momentum also contributes to non-convergence. To address it, we propose to change the order of the momentum update and the normalization by the second moment estimate. With this small adjustment, we successfully eliminate the non-convergence problem of Adam without relying specific hyperparameter choice and the bounded noise assumption. We provide theoretical evidence demonstrating that our derived algorithm, named ADOPT, can achieve convergence with the optimal rate of ((1/ (T) for smooth nonconvex optimization.
In our experiments, we begin by assessing the performance of ADOPT in toy example where Adam typically fails converge depending on the choice of 3.. This toy example an extension of the one presented in Eq. (T) by Reddi et al. [2018). but we consider a scenario where AMSGrad is also hard to converge due the dependence on the bounded noise assumption. Our results demonstrate that ADOPT rapidly converges to the solution, while Adam fails to converge, and AMSGrad exhibits extremely slow convergence. Next, we conduct an experiment using a simple multi-layer perceptron on the MNIST classification task to evaluate the performance of ADOPT in nonconvex optimization. Our findings indicate that ADOPT outperforms existing adaptive gradient methods, including Adam, AMSGrad, and AdaShift. Finally, evaluate the performance of ADOPT in various practical

---

applications, such as image classification of CIFAR-10 10 and ImageNet using ResNet [He et al. 2016] and Transforffrrmrr [Liu et al., 202T). training of deep generative models (NVAE), fine-tuming of language models (LLaMA)), and deep reinforcement learning for continuous control. Our empirical results demonstrate that ADOPT achieves superior results over existing algorithms (e.g., Adam) in these practical applications.
Preliminary
2.1 Problem Definition
We consider the minimization of the objective function f FR R with respect the parameter E IR.. In this context, we focus on first-order stochastic optimization methods, where only the stochastic gradient is accessible. As the objective / can be nonconvex, the goal find stationary point where V = 0 [Blair, 1985 Vavasis 1995]. In order to analyze the convergence behavior of stochastic optimization algorithms, the following assumptions are commonly employed in the literature:
Assumption 2.1. The functionn f(0) is lower-bounded, i.e., f(0) 2 fint > -00 for all O.
Assumption 2.2. The stochastic gradient is an unbiased estimator of the objective f(0,-1), i.e. Elg.] V (0t-1) for all t 1.
Assumption 2.3. The objective function is L-smooth on TR i.e., there exists constant L > 0 such that (x) - V f(y)l I L|| yll for all E RP.
Assumption 2.4. Variance of the stochastic gradient is uniformly bounded i.e., there exists constant σ > 0 such that [[[]g / (0.:))))
For the analysis of adaptive gradient methods (c.g.. Adam and AdaGrad), many of previous works DD
Tossez et al.] 2022 Li and Orabona [2019] Ward al. 2020 Zou et al. 2018 use little stronger assumption instead of Assumption for ease of proofs:
Assumption 2.5. The stochastic gradient has finite second moment, i.e., there exists constant G 0 such that B|llg(11] < G².
Assumption 2.5 requires that the true gradient / ʃ is also uniformly bounded in addition to the variance of the stochastic gradient g. Moreover, the convergence proof of AMSGrad tends rely on an even stronger assumption as follows [Chen et al. 2019 Zhou et al. 2018).
Assumption 2.6. The stochastic gradient is uniformly upper- bounded i.e., there exists constant G >>00 O such that ||g.]| S G.
2.6 the gradient noise & 9t V is assumed be bounded almost surely in addition to the true graidient Vf. Note that when Assumption 2.6 holds, Assumption 2.5) is automatically satisfied; hence, Assumption 2.6 is stronger assumption compared Assumption 2.5) In this paper, we adopt Assumptions 2.1 2.2 2.3 and 2.5 for analysis, because one of our derive the upper bound of min, (E[IIV / (O.)) 2/333/22 to investigate the convergence rate of the stochastic optimization algorithms, which is commonly performed the literature [ééossez et al. 2022 7ou et al. 2019
In Assumption
motivations is to address the omission of Assumption 2.6 In the analysis,
2.2 Review of Stochastic Optimization Algorithms for Nonconvex Objectives
The convergence of the vanilla SGD have been studied extensively in previous works. For smooth nonconvex functions, Ghadimi and Lan [2013] showed that SGD with constant learning rate converges with an O(IVTT) rate under Assumptions 2.1 2.4 by setting ae a O(1 /T). where a is a learning rate at the t-th step, and T is a total number of parameter updates. This convergence rate known be minimax optimal up to a constant (Drori and Shamir [020]. For the diminishing learning rate scheme, the convergence bound of (log T/VT) is well-known for a = =a/Vt [Ghadimi and Lan 2013). Recently, Wang et al. [2021] have proved that SGD with ar = a can also achieve the optimal rate (11/VT) by additionally assuming that the objective f is upper-bounded,
3

---

While the vanilla SGD is still one of the most popular choices for stochastic optimization, adaptive gradient methods are dominantly used especially for deep learning. adaptive gradient methods, the parameter 0 is updated additionally using the second moment estimate in the following form:
9t
0 = 0 a4 0z E
where e is small positive constant. The division between vectors is applied in an element-wise manner, and the addition between vector a and scalar b is defined (a b); == 04 + b. In AdaGrad Duuchi et al. 2011]. Ur is defined as Un = 0 and ve = 1 St ② gt- In RMSprop (Hinton al. 2012 an exponential moving average is substituted for the simple summation, i.c., U, - + (1- 52)g. @ g., where 0 S B2 < 1. Adam [Kingma and Ba, 2014) uses momentum in addition to the second moment estimate to accelcrate the convergence as follows:
m, 31 m
31)
(3)
m.
0: 0 at Vv, + c²
(4)
where mg O. Here, we omit the bias correction technique used in the original paper for clar- ity. Unfortunately, RMSprop and Adam are not guaranteed to converge even in a simple convex optimization problem demonstrated by Reddi et al. [2018], whereas AdaGrad with constant learning rate is known to converge with an O (log T/ T/vT)) rate under Assupmtions 2. 2.3 and 2.5 for smooth nonconvex cases [Li and Orabona 2019 Ward et al. 2020 Zou et al. 2018 Chen et al. 2019 Défossez et al. [022]. Although the convergence of Adam can be assured by choosing B3 in manner [Shi et al., 2020 Zhang et al. 2022 Wang et al. 2022 Li et al. 2023
Wang et al. 2023]. it is difficult to know the proper choice of B2 for each problem before training.
To fix the non-convergence of Adam without depending on 82, some researchers have proposed variants of Adam. Reddi et al. [2018) proposed AMSGrad, which substitute in for e in Eq. 3 where
G = O and é; = max ( J. The idea behind AMSGrad is that the scaling factor vē: e should be non-decreasing to ensure the convergence, After Reddi al. (2018) originally proved the convergence of AMSGrad for online convex optimization, Chen et al. [019 showed that AMSGrad with a = a/ vt converges with (logT//TT for nonconvex settings. Zhou et al. 2018 also analyzed the convergence of AMSGrad for nonconvex optimization, and derived the convergence rate of O(L/VT) for constant learning rate of a a = ((I/VT). However, their results depend on Assumption 2.6, which is often violated in practice. For example, variational autoencoders (Kingma and Welling 2014) and diffusion models [Ho al.] [020 Soogg et al. [202T] are typical examples in which Assumption 2.6|does not hold because they utilize unbounded Gaussian noise in the gradient estimation. The cause of requirement for Assumption 2.6 is the max operation in the definition of i,-
Since the max operation is convex, B[ūx) < max, E 0y does not hold; hence Assumption 2.6 is
required to upper-bound [[ in their proofs.
Zhou et al. [019) also tried to fix the non-convergent behavior of Adam. Their proposed AdaShift uses Ut-n instead of v. for the second moment estimate, and calculate the momentum using the latest gradients as follows:
m: k
(5)
ER=0
BK
m:
0, 0, at Vo--n
(6)
In the original paper, some additional techniques (e.g., the block-wise adaptive learning rate) are used, but we omit them for clarity here. Though they give theoretical analysis for single online convex
example, any convergence bounds are not provided for nonconvex cases. More detailed discussion on existing analyses is provided in Appendix
3 Analysis: Cause of Non-convergence of Adam and How to Fix It
In this section, to derive an algorithm that can converge with any /3 without Assumption 2.6 we analyze the cause of non-convergence of Adam, and discuss how it can be eliminated. To start from simple case, we first analyze the case without momentum. Subsequently, we extend it to the case with momentum and provide way to fix the convergence issue of Adam.

---

3.1 Case without Momentum
We first analyze the convergence of RMSprop, which corresponds to the no-mon no-mmmeentm case of Adam when we omit the bias correction. For R.Spprp,, we derive the following convergence bound. the following holds for the RMSprop with a
Theorem 3.1. Under Assumptions constant learning rate at a:
2.7
2.3
and
2.5
3/2
fo fo--im
fine
C2 log T
G² e²
E
SCi
1+
C2 C2 log 32) (7)
aT
where C₁ 22 C2
aDL 2[1
2DG v1-
and fo = f (0n)
B2'
Sketch of proof. By Assumption 2.3 the following holds:
E [ (0.) S / (0.-1) +
a²L
9t
2
(O
g: Vv, (vrt
(8)
2
Vo: +
+
Applying Lemmas G.4 and G.6 in the appendix to this, the following inequality is derived:
lf (0.)
a²L + 22GVV111 1 {/2 -12
2
< / (0,-1) +
gt
v (0,-1)
gt
(9)
Vu + 22
V8, + E2
3/2
< (0(-1)) +
aLL + 20G
-132] 8/) 82 2aG V 11
>>>
a
E |V] (0,-1)
(10)
Vv. +
2
V(1 32
G2 +2
where 8. terms, we have
82 (1 B2) @ g.). Telescoping this for t 1,...,T and rearranging the
Σ E (I
T
3/2
(1(0)) fin + C2 log
( G2 t2) 37 2
(2)+2)
S SC
(II)
where the last inequality holds due Assumption 2.1 and Lemma G.5 is derived using min,=1...,
Therefore, the bound in Eq
Σ
E f (0:11) 11/3/3/2/7 /T.
O
A detailed proof is provided in the appendix. When the learning rate is chosen so that = O(1/VT), the first and second terms on the right hand side of Eq. ( converge with O(1/ and O(1/T) rates, respectively. However, the last term includes a constant factor in terms of T, which represents the non-convergent behavior of RMSprop in the smooth nonconvex setting. More precisely, RMSprop is guaranteed to converge only to bounded region around a stationary point, and the size of the bounded region depends on the hyperparameter Bn and the problem-dependee and Therefore, we need to choose 32 dependently on each problem to make the bounded region adequately small. Since lim log /2/ B2 O, the size of the bounded region be made small by setting 32 to a value close to 1, which aligns with practical observations. However, how close to 1 it should be relies the
factors D, G,
factors, which cannot be observed in advance. This result is consistent with recent results of convergence analyses of Adam and RMSprop (Shi et al.
2020 Zhang al. 2022
As can be seen from Eqs. 8 and (9. the constant term in Eq. 7 derived from the last term of Eq.
(8). Because 9: and are not statistically independent, this term is first decomposed as in Eq. 9
After the decomposition, gr and Ü, now conditionally independent given go. 9 5o Eq.
is derived using the following fact:
9t
(0
E
= E
(12)
+
This indicates that, the second moment estimate v, designed to be conditionalyy independent to Its the constant term in the convergence bound will be removed, because the second term of Eq. (8)
5

---

Algorithm 1 ADOPT algorithm
Require: Learning rate {ar), initial parameter 0o
Require: Exponential decay rate O S 31 < 1,0 S B2 S 1, small constant > 0
Vo f go @ gu. m2 < 91 max co.c)
for t = T do
0, - 0 m,
< 82 1 + (1 (2) gt ④
m c B1 m. + (1 81) max c
end for
return {0
can be directly lower-bounded without the decomposition. A simple way achieve the conditional independence is to substitute U--1 for as a second moment estimate, because V--1 does not have information about gt- This solution similar to AdaShift, in which vt-n is substituted for va as described in Eq. ( In fact, the modified version of RMSprop is identical to AdaShift with n = 1 and 31 0 except for the additional techniques (e.g., the block-wise adaptive learning rate).
3.2 Case with Momentum
As we have described, RMSprop can be modified to be convergent by removing the current gradient Ji from the second moment estimate v,. However, when we combine adaptive gradient methods with momentum like Adam, the convergence analysis becomes more complicated. Unfortunately, when Adam-style momentum in Eq. (3) is applied, the algorithm does not converge in general even when using V,-1 as second moment estimate instead of U₁. This is because the momentum m, contains all history of the past gradients 9u.-.- 9t; hence the second moment estimate always correlates with m,, AdaShift prevents this problem by calculating the momentum m, only using the latest n gradients described in Eq. (5). In that case, the momentum m, and the second moment estimate Ut-n are conditionally independent, so the convergence can be retained. However, this approach has trade-off in the choice of When is small, m, has little information about the past gradients; when n is large, tt-n only has access to the gradient information in the distant past.
To remove this trade-off, instead of truncating the momentum to the latest steps, propose to use momentum of the following form:
gt
+ (1 (1- B1) m. = -3mm-1++
(13)
/
0, = 0,-1 am
(14)
The main difference to the Adam-style momentum in Eq. 3 is the order of update of m, and the normalization by 11;1 +2. In Eq. (3) the normalization is performed after the update of m whereas in Eq. (13), the normalization first applied to the current gradient 9: in advance to the update of m In this case, the second moment estimate u--1 1 is only used to normalize the current
gradient gr, so the convergence can be guaranteed. more detailed convergence analysis provided in Section
4 Method: Adaptive Gradient Method with the Optimal Convergence Rate
Based on the analysis in the previous section, we propose a new adaptive gradient method named ADOPT (ADaptive gradient method with the OPTimal convergence rate). The entire procedure is summarized in Algorithm 4 For a simple discription, we place the update of m after the parameter update in Algorithm 4 but it is equivalent to Eqs. (33) and (14 except that max {vØ,e) e is
substitued for / + e². The substitition is applied because we find that it contributes to slightly better performance in practice. We provide an equivalent expression of Algorithm in Algorithm C in the appendix, which is closer to a practical implementation. By this modification, ADOPT can converge with the optimal rate for smooth nonconvex optimization follows:

---

k: = 10
1,0
1.0
1.0
0.5
0.0
0.0
0.0
82 0.999
-0.5
-0.5
-0.5
-1.0
-1.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Step
1e6
Step
1e6
Step
1e6
k = 50
1.0
1.0
0.5
0.5
0.5
0.0
0.0
-0.5
-0.5
-0.5
0.0
0.5
1.0
2.0
0.0
1.5
2.0
0.5
1.0
Step
1e6
Step
1e6
Step
1e6
Adam
AMSGrad
ADOPT
Figure I: Performance comparison between Adam, AMSGrad and ADOPT a simple univariate convex optimization problem. The plots show transitions of the parameter value, which should converge to the solution -1.
Theorem 4.1. Under Assumptions 2.7 2.3 and 2.5
the following holds for the ADOPT algorithm
with a constant learning rate a₁ = = L/vTT)
1/3 3/2
s 0 (1/VT)
min
E
(15)
The detailed proof and related lemmas are provided in the appendix. We also provide the convergence bound for the case of diminishing learning rale (i.e., ar a/ Vt) in the appendix, which closer practical situations. In that case, ADOPT also converges with the optimal rate.
Experiments
In the experiments, we first validate our ADOPT algorithm using simple toy example in which Adam is known to fail to converge, and confirm our theoretical findings through numerical simulation. Secondly, we run an experiment training simple multi-layer perceptron (MLP) for the MNIST dataset to verify the effectiveness of our ADOPT for nonconvex optimization problems. Finally, we evaluate our ADOPT in a wide range of practical applications, including image classification, natural language processing (NLP) tasks, generative modeling, and deep reinforcement learning. Detailed experimental settings are described in the appendix.
Toy problem: We consider a convex optimization problem with an objective f(0) = 0 for E [-11, 1]. obvious that solution for the problem is 0 - -1. Through the optimization, we only have access to the stochastic objective fe as follows:
with probability 1/k 1-k0, k0,, with probability 1/k:
(0)
(16)
where k 2 1. Because Eff f() holds, the stochastic gradient 9r fc()) is an unbiased
estimator of the true gradient f regardless of the choice of k. satisfying Assumption 2.2 This problem is equivalent, except for scaling, to the stochastie optimization version of Eq. ① provided by Reddi et al. (2018) as case where Adam fails converge. In this setting, the constant k controls the magnitude gradient noise. When & 1, it corresponds to the noiseless case where f, = with probability 1. As l gets large, stochastic gradient becomes noisy, making G in Assumptions
7

---

Training Accuracy
Test Accuracy
Accuracy (%)
Accuracy (%)
ADOPT
ADOPT
Adam
Adam
AMSGrad
AMSGrad
AdaShift
AdaShift
Step
Step
Figure 2: Accuracy for training data (left) and test data(right) in MNIST classification. The error bars show the 95% confidence intervals of three trials.
2.5 and 2.6
large large. Therefore, the optimization will be more difficult when & becomes larger. In the experiment, we set & = 10 or 50, and compare the robustness Adam, AMSGrad, and ADOPT for various hyperparameter settings by changing B2 from 0.1 0.999. We set algorithms, which is common choice in practice. We set the learning rate to ar 0.01 /11/V112 + 0.011.
0.9 for all the
The result is shown in Figure I It can be seen that, when k = 10, Adam fails to converge except for 32 0.999 while AMSGrad and ADOPT rapidly converge to the correct solution, i.e., 0 - with any 32. In more extreme case where This aligns with Theorem 3.1, since, when the gradient noise is large (i.e., G is large), the bounded region of the convergence bound also gets large, leading divergence of Adam. Moreover, when AMSGrad also becomes much slower than ADOPT.
50, Adam fails to converge even with B2 0.999.
== 50, it is observed that the convergence In fact, this phenomenon is also consistent with theory. In this problem setting, the second moment E[g²] is OIE²), while the squared norm of the stochastic gradient 9² is O(k*). Since the convergence bound of AMSGrad depends the uniform bound of the stochastic gradient in Assumption [.6 instead of the second moment in Assumption 2.5 its convergence also deteriorates with the order of L. Compared AMSGrad, ADOPT only depends the second moment bound for its convergence, so it converges much faster than AMSGrad even in such an extreme setting.
We also perform ablation study how the two algorithmic changes from Adam to ADOPT affect the convergence. The differences between Adam and ADOPT are (1) decorrelation between the second moment estimate and the current gradient, and (2) change of order of momentum update and normalization by the second moment estimate. In this experiment, we remove each algorithmic change from ADOPT, and compare the result in the toy example. We set k = 50, and ($1, 32] (0.9, 0.999), since it is a common hyperparameter choice. The result is shown in Figure 3 It can be observed that ADOPT fails converge with the exception of either algorithmic change. Therefore, applying both changes is essential to overcome the non-convergence of Adam, which also aligns with theory. These results correspond to the theoretical findings, showing the superiority of ADOPT to Adam and AMSGrad in terms of the convergence speed and its robustness to hyperparameter choices.
MNIST classification: To investigate the effectiveness of ADOPT nonconvex optimization, we train nonlinear neural networks for MNIST classification tasks, and compare the performance between ADOPT and existing optimization algorithms, such as Adam, AMSGrad and AdaShift. In this experiment, we use a simple MLP with a single hidden layer, and the number of hidden units is set to 784. We set the learning rate to a, = a /t, and a is tuned in the range of {1, 1001, 10 - 10 10-33. We apply weight decay of 1 x 10 to prevent over-fitting, and 10K iterations of parameter updates. Figure 2 shows the learning curves of training and test accuracy. We observe our ADOPT performs slightly better than the others in terms of the convergence speed and the final performance.
Image classification: As more practical application, we conduct experiments of image classification using real-world image datasets. We first compare ADOPT and Adam in the classification task of the CIFAR-I0 dataset using R8 [H et al., 2016), a widely-used convolutional neural network. We conduct a similar hyperparameter search to the case of MNIST classification. A detailed experimental setting is provided in the appendix. The learning curves of test accuracy are visualized in Figure can be observed that ADOPT converges little faster than Adam.
8

---

1.0
0.5
Test Accuracy
90
ADOPT ADOPT DE
0.0
85
ADOPT CO
80
Optimizer
-0.5
Adam
75
-1.0
ADOPT
0.0
0.5
1.0 Step
1.5
2.0 1e6
70
0
50
100
150
200
Epoch
Figure 3: Ablation study of algorithmic changes between Adam and ADOPT. "DE" and CO denote "decorrelation" and "change of order", respectively.
Figure 4: Learning curves of test accuracy for CIFAR-10 classification by ResNet-18 trained with Adam and ADOPT.
Table Top-1 accuracy (%) for ImageNet clas- sification by SwinTraasformer.
Table 2:
Negative lig-liiel of NVAEs for MNIST density estimation. Lower is better.
Epoch
200
300
Epoch
200
300
AdamW
79.29 + 0.05
81.26
0.04
Adamax
80.19 + 0.08
79.41 0.07
AMSGrad
78.91 + 0.03
81.17 + 0.03
ADOPT
79.02 + 0.10
78.88 + 0.09
ADOPT
79.62 + 0.03
81.50 = 0.04
To confirm that ADOPT works well for modern neural network architectures based Transform- ers (Vaswani et al. 2017]. we perform an experiment of ImageNet classification using SwinTrans former [Liu al. 202II. We follow the official training recipe of Swin Transformer-ting provided by Torchvision Paszke et al., 2019a], and fix the training settings except for the optimizer choice. We use A Loshchilov and Hutter 2019] as a baseline because it optimizer. We also compare with AMSGGad as another way to fix the non-convergence issue of Adam. Since AdamW uses decoupled weight decay, we also apply to the other optimizers for fair comparison. We report the top-1 accuracy at 200 and 300 epochs in Tables []] We observe that ADOPT outperforms Adam W and AMSGrad throughout the training terms demonstrating the effectiveness of ADOPT for this setting.
the default official
test accuracy,
Generative modeling: We train NVAE Vahdat and Kautz, 2020 for MNIST using our ADOPT. In the official implementation of NVAE, Adamax [Kingma and Ba 2014]. an infinite-norm variant of Adam, used an optimizer, so we use Adamax as baseline method. We use the exactly the same setting of the official implementation except that the learning rate for ADOPT is set to 2 x 10 since the default value 0.01 is too large for ADOPT. We report the negative log likkliihoood for test data on It is observed that the model trained with ADOPT shows the better likelihood.
Table 2
Pretraining of large language models: We run pre-training of GPT-2 [Radford al.) 2019 using the nanoGPT (Karpathy| 2022] code base to compare Adam and ADOPT. We use Open Web- Text [Gokaslan and Cohen 2019 the training data. Experimental setup conforms the default settings of nanoGPT except for the selection of the optimizer. We also test a case in which the total setting where the gradient noise becomes larger. The results are summarized in Figure 5 The most notable finding is that in the small batch size case, Adam causes loss spikes the early stages of training and fails converge, while ADOPT is always able to train stably. This is consistent with Adam's 's theory of non-convergence. As the gradient noise increases, in Theorem 3.1 also increases, and the constant term Adam's 's convergence bounds becomes non-negligible especially when using large-scale dataset like Open WebText. As result, Adam more likely to fail to train in such cases. Our ADOPT, on the other hand, does not suffer from this problem because it can always guarantee convergence. We also observed that both Adam and ADOPT work well when the batch size is large, but even this case, ADOPT performs slightly better.
batch size was changed from 480 to 96,

---

Adam (batch sze: 480)
Adam (batch size: 480)
Adam Ibatch
Adam (batch sze: 96)
10
ADOPT (batch sit: 480)
ADOPT (batch sae: 480)
validation Loss
Training
7
20000
400C0
60300
80000
100000
20000
40000
60000
80000
100000
Steps
Steps
Figure 5: Learning curves of GPT-2 pretraining for training set (left) and validation (right).
Finetuning of large language models: We finetune the pretrained LLaMA-7B on 52K instruction following data provided by Stanford Alpaca and compare the performance between the default optimizer (Adam) and our ADOPT under the exactly same experimental setting. For evaluation, we use Multi-task Language Understanding (MMIU) Benchmark (Hendrycks et al.) 2021], which widely used to assess the performance of large language models. The MMLJU score for LLaMA-7B without finetuning 35.1. After fine-tuned tion with Adam, the score improves to 41.2. When substitute ADOPT for Adam, the score even improves to 42.13. The detailed score comparison for each task summarized in Figure 7 in the appendix. Other experimental results, including deep RL experiments, and detailed experimental settings are also provided in the appendix.
cntrructionfoollowwii using the baseline implementa-
Conclusion
In this paper, we demystified the fundamental cause of divergence of adaptive gradient methods based on the exponential moving average, such Adam and RMSprop, in general smooth nonconvex optimization problems, and demonstrate a way to fix the issue, proposing new optimizer named ADOPT. Not only does ADOPT converge with the optimal rate without depending on hyperpa- rameter choice in theory, but ADOPT demonstrates better performance in a wide range of pracital applications.
We expect that this work will serve as a bridge between theory and practice in the research of adaptive gradient methods. Since ADOPT can be safely applied to many machine learning problems without careful tuning of hyperparameters, it can be expected to improve the training stability and the model performance in practice by substituting it for the existing adaptive gradient methods (e.g., Adam).
One of the limitations of our analysis is that it still relies on the assumption that the second moment of stochastic gradient is uniformly bounded (i.e., Assumption 2.5). Although this assumption is weaker than the bounded stochastic gradient assumption (i.e., Assumption 2.6). it would be more desirable to relax it to the bounded variance assumption (i.e., Assumption 2.4). which is often adopted in the analysis of the vanilla SGD |Ghadimi and Lan 2013). For Adam, a recent work by Wang et al.) [2023] have derived problem-dependent convergence bound which achieves the (LlvVT) rate without Assumption 2.5 Their proof techniques may help relax our assumptions in the proof of Theorem which we leave as future work.
From a broader perspective, adaptive gradient methods like Adam have been widely used even for the training large-scale foundation models (e.g., large language models). Although such models can be useful for people, their negative aspects, such concerns about copyright infringement, are not negligible. Researchers needs to deeply recognize and understand such social impacts of machine learning algorithms.
10

---

References
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rX1v91412.69900 2014.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018. URL httpps://ppenrevieeee net/
Forun?id=ryQuff-
Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for conver- gences of adam and rmsprop. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11127-11135, 2019.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. In International Conference on Learning Representations, 2019.
net/forum?id-Hlxx>XSS
Dongruo Zhou, Jinghui Chen, Yuan Cao. Yiqi Tang. Ziyan Yang, and Quanquan Gu. On the conver- gence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.0567) 05671, 2018.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341 1-2688, 2013.
Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors.
SIAM Journal on Optimization, 10(3):627-644 2000.
Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world. Transactions on Machine Learning Research, 2023, ISSN 2835-8856. URL https://openrevieww wtw.:eewwwe//oormmmm Ad=AUNNNS Survey Certification.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840-688 6851, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-basee generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL http:/ttpp:///perevvveee. htwp:eewwwwwiweeeeeeeeeeeeeeeeee www./fouummeee
PxTIG12RRHS
Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang. Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence adaptive learning rate methods. In International Conference on Learning Representations, 2019. URL http:s//opppeerreiirr httt:s///pppmrriiiieeeeforrrrrtn Highouunkkidt
Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun. Rmsprop converges with proper hyper- parameter. In International Conference on Learning Representations, 2020.
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https::/openrevieee wtwp:arrwwiieeeeeooouuummiiie.aa wwwttfouumifieeeeeeeee htw.orrrmmmid-11500yaaaaaaaaa
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. Provable adaptivity in adam. arXiv preprint arXiv:2208.09900 2022.
Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of adam under relaxed assumptions. arXiv preprint arXiv:2304.13977 2023.
Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and lower bound of adam's iteration complexity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

---

hateee//ccc3///sldes////eee Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
Charles Blair. Problem complexity and method efficiency in optimization (as nemirovsky and db yudin). Siam Review, 27(2))264, 1985.
Stephen A Vavasis. Complexity issues in global optimization: a survey. Handbook of global optimization, pages 27-41, 1995.
Alexandre Défosse7, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
httpp:///ppenreviex. httpe/viwwwwwweeeeeeeeeee httpe//oouunid-2Phh2255000
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics, pages 983-992. PMLR, 2019.
Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research, 21(1):9047-9076, 2020.
Fangyu Zou, Li Shen, Zequn Jie, Ju Sun, and Wei Liu. Weighted adagrad with unified momentum. arXiv preprint arXiv: 0044000, 2018
Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In International Conference on Machine Learning, pages 2658-26611 PMLR, 2020.
Xiaoyu Wang, Sindri Magnússonn, and Mikael Johansson. On the convergence of step decay step-size for stochastic optimization. Advances in Neural Information Processing Systems, 34: 14:14226-14238, 2021.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):111--159, 2011. URL
http://jmlr org/ wwtggpapers/viie httprr//vv/dacthi1eeee duchiiia.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Lecture 6e rmsprop: Divide the gradient by running average of its recent magnitude, 2012. URL httpps://ww. cs. toronto edu/
lecies___e_p.pppp pddf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
www.iiinnnnnnnnnnneeeeeeeeeeeeee cc/ wtt.fffees/ppper/11///ie/// .df
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019a.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. UKLLhppps///gpenrreeeeee///oooop
Bxg6RiCqY7
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667-19679, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
12

---

Andrej Karpathy. NanoGPT.
http:///gggttubb..o.m./aaaaatat/ 2022.
2022.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.
http:://kyyioon077.guu git.hb................... io/
OpenwebTextCCoppus 2019.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob
Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021. URL httphtts//pppereviieee htt::toruurmmddd7788111110.....
Jeff Haochen and Suvrit Sra. Random shuffling beats sgd after finite epochs. International Conference on Machine Learning, pages 2624-2633, PMLR, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. 7achary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019b. URL http:///://pa htw.://pppprrr..eupppsppp -iirray.......
pdf
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870, PMLR, 10-15 Jul 2018. URL htttps://proceedings mlr wwwppprsssvee.
preass//00/haarooaaa.00 http://aaannnoaaaoo.oaa html
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. 2012. doi: 10. 200.9//ROS..02..38610000
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines Reliable reinforcement learning implementations. The Journal of Machine Learning Research,
12355, 2021.
13

---

A Detailed Relationships to Existing Analyses
In this section, we discuss the relationships between our analysis and existing ones on the convergence of Adam-like optimizers in smooth nonconvex optimization problems. Tables 3 and 4 are a summary of comparisons between them in terms of their problem settings and derived convergence rates.
Zhang et al. [2022] focus on convergence of Adam in the finite sum problem, where the objective has following form:
I (0) - Σ (0)
(17)
fi is, for example, loss function for i-th training sample. Although many deep learning problems can be formulated as finite sum problem, training of the variational autoencoders (VAEs) or diffusion infinite sum (i.e.,
models is out of the finite-sum problem, since their objective is formulated an expectation over continuous variables). Moreover, they assume the stochastic gradient is L-- Lipschitz, whereas we only assume true gradient V is L-Lipschitz. They also assume growth condition as follows:
E [1ad]
s c. |
Gf (0-1)))
(18)
This growth condition is weaker than our Assumption 2.5 Assumption 2.5 is a special case of the growth condition where G₁ = O. Their derived convergence rate has a constant factor of (Go); hence the strong growth condition (i.e., Go 0) required to assure convergence. Moreover, to assure convergence, one needs to choose sufficiently large 82, which has to be tuned in problem- dependent manner.
Wang et al. [2022] also focus on convergence of Adam in the finite sum problem, but they relax the L-Lipschitz condition on g to the (Lo, Li L-Lischitzz condition. They also assume the growth condition in Eq. (18), and their convergence rate has the same order with Zhang et al. [2022]. so it still requires the strong growth condition (i.e., Go 0) to assure convergence. The condition of (3 is also similar to Zhang et al. (2022).
Li et al. [2033] consider Adam's convergence on general smooth nonconvex problems. Similar to Wang et al. 2022]. they use (Lo, L-psshiischttz condition on the true gradient Vf. They also assume that the gradient noise is almost surely bounded:
lgg Vfll So
(19)
The relationship between this assumption and our Assumption 2.5 is a little complicated. Assumption 2.5 is equivalent to a combination of Assumption 2.4 and the following assumption: Assumption A.1. The true gradient is uniformly bounded, i.e., there exist constants G and such that [|V/ (0)1 < G o2 and < a SG.
The bounded noise assumption of Eq. (19) is strictly stronger than Assumption 2.4 but they do not assume the bounded true gradient (i.e., Assumption A. The bounded noise assumption is often violated in practice (e.g., training of VAEs), because the gradient is often estimated using unbounded noise (i.e., Gaussian noise). Their convergence rate ( /T) is better than Zhang et al. [2022) and Wang al. [2022, while it still requires constraints on the hyperparameters, which have to chosen depnlem manner.
Défossez et al. [2022) analyzes the convergence of Adam under exactly the same assumptions with ours, and they derive the (/vo) rate, which is worse than our ADOPT's convergence rate. Moereover, to assure the convergence, B2 has to be chosen dependently on the total number of iterations T..
Wang et al. 2023 analyzes the convergence of Adam under Assumptions 2.1 2.4 and they derive the (11/ VT) rate. However, to assure the convergence, 82 has to be chosen dependently on the total number of iterations T as in Défossez et al. 2022].
Chen et al. 2019 and Zhou et al. 2018 analyze the convergence of AMSGrad for general smooth nonconvex problems, and derive the convergence rate of Ollog T/VT) and ((1/vT) respectively. However, to guarantee the convergence, the stochastic gradient has to be bounded almost surely
14

---

Algorithm
Problem
Smoothness
Gratheen Growth
Zhang
L-Lipechiu g
Bllg
G
G
Wang 202
Adam
Finite
(Ta, La i-ppccic g
[
Go
Vill
Lreral KBA
Adam
General
(Lo, -iipscitt V
1
ffl 5
Defossez al
Adam
General
L-Lipschitz y
G
L-Lipschitu y
Blllg
V $ S
Lhen at
2119
AMSGrad
General
I. Lipschitz f
lg S
7hoo
T.-iipccitzz y
9 S
G² G Table 3: Comparison of the problem settings between our analysis and other existing works.
ADOPT
Goneral
L-Lipschitz V
Bllll
Constraints
Convergence
Zhang et al
2022
B1 VD.. 02 7(1)
Ollog T/VT) O
Wang et a
2022
8 v 8(82) O(1/Gi)
O(log TJvT) + ((G)
Li et al
2023
81 < VB,, CcL..
0(1vT))
Dérossez
A022
B1 < v 32 ((1TT)
Ollog T/vT)
81
$ v
81 8)) p. -.1- 0017))
O(1/vT)
Chen
2019
J32
Ollog T/vT)
//ou al
2018
V
O(1/VT)
Ours
0(/vT)
Table 4: Comparison of the convergence rate and imposed constraints on the hyperparameters between our analysis and other existing works. Please refer to the original papers for the definitions of y and c.
(Assumption 2.6 which is often violated in practice. In addition, the hyperparameter 31 and B3 should be chosen satisfying B1 < vB2. This constraint is relatively minor compared to the constraint manner.
imposed in the analyses of Adam, since it can be satisfied in
B With-Replacement vs. Without-Replacement
In the optimization of finite-sum problems, practitioners often use withoott-replacement sampling, which is also known as random shuffling. to obtain stochastic gradient. In this case, the stochastic gradient has small bias due to the lack of replacement, so Assumption 2.2 violated. However, the vanilla SGD is known to converge with the without-replacement strategy [Hoochnn and Sra 2019 and some of the analyses of Adam also adopt without-rrplacemmet sampling [Zhang et al. 2022 Wang et al. 2022]
Unfortunately, find that ADOPT has counter example, in which ADOPT fails to converge when using without-reelacemmen sampling. For example, when we consider minimizing (() Σ3 (f). where 0 E 1,, fi (0) = 1.90 and f2( = fz( = -f, it can be easily observed that ADOPT with 81 32 0 fails to converge to the correct solution, i.e., 0 1.
This non-convergence can be easily avoided by using the with-replacement strategy. Moreover, the difference between with and withhout-rellacement sampling becomes negligible when in the finite-sum Σ fa is large enough; hence it does not affect the practical performance very much. In fact, our experiments except for the toy example are performed using without-replacemmnt sampling, but divergent behaviors are not observed. If one applies ADOPT to problems where the difference seems severe (e.g., when training with small dataset), we recommend to use with-replacement sampling instead of random shuffling for stable training. When one uses PyTorch (Paszke et al. 2019b) for the implementation, for example, with-replacemmnt sampling can be easily applied by specifying replacemnet-Trie for torch. utils. data. RandomSampler, and feeding it to the sampler argument of torch. utils. data. DataLoader.
C Another Expression of ADOPT
15

---

Algorithm 2 Alternative representation of ADOPT algorithm
Require: Learning rate {ar), initial parameter Oo
Require: Exponential decay rate 0 S 31 < 1,0 S 32 s 1, small constant >> 0 Vo + 90 ④ gu
for t = to T do
if 1 then
m. <- 9t/ max { [v
else
m, t- 31 m + (1 - -11) 91) max {
end if'
0, 0:-1- 0,772
Vr + 82 1--1 + (1 (2) gt og:
end for return [0:]T-
D Recommendation of Hyperparameter Settings for ADOPT
We experimentally find that our ADOPT works similarly to Adam when the same hyperparameters are used, but should be set little larger value (e.g., 10 for ADOPT compared to Adam, in which is set to 1 x 10-$ by default. Our recommendation of the hyperparameter settings for ADOPT is provided in Table 5
81
0.9
B2
0.9999
x 10-6
Table 5: Recommended hyperparameters for the ADOPT algorithm
E Theorems
Theorem E.1. Under Assumptions 2.7
and2.3 if the objective J is upper-bounded by Ssup- the following holds for the ADOPT algorithm with a learning rate a₁ = at/ /t:
2.2 2.3
3/2
E |J (0:) (0.))
min
3 max [G" +
Ssup Siat (T+
V2a aG²L 21 B1)
(T)
2 (T (T+ 1//3/2 -1)
22
+ 3 3 /mx (G2, +
2V23I (VT+I-1)
0. T
(T + 1)3/2 -1)
<(1 (11-3)
e(1 -11 +1
+ 3 3/mas max (G2,1) +
2a
a23,GLL
log (T + +1).
(20)
(T 1)3/2 1)3
(c(1- - 31)3
V2e² (1 31)
F Proofs
Proof of Theorems
and E.I
We define Φ, for > 1 as follows:
B1 O.-1-
Φ.
0
(21)
1 81
16

---

We also define Φu- = 0,. By Assumption 2.3 the following holds for t 2 1:
f f Vf (Ø: x1-1) +
2 11.: 1111
(22)
= =
(Ø) - -
- T -bi-))
2 1l 111
L
(23)
- < ( (0: ( (Ø: - 11
+ L
+ - / (0.- )l| I1 -1--111
11 - Φ1 111
(24)
2
Sf ( Vf (0:-1) _ (Ø: - -
+ + 01-11 11 Φ2-1|1 + I. 11 211=11?
(25)
where the second inequality holds due to the Cauchy-Schward inequality, and the last inequality holds due to Assumption 2.3
By taking the expectation, the following holds:
[f (0x) s [f
V (0,-1) (Ø: -
|E 0--1l1 - ---111l1 + - E | - 11
(26)
E [V 1/37 3/2
< Ef (0)-1)] + (0-11-01) 81 (1- 11-1 G²
BI-
aq
v(1 - 13) GP +
(27)
+ 01 (0 1 aa) 88 (1 1 GPI apa 18 1 -1-
1
G2L G2L
(1 - 31)²
(1 11-31)e 2
(0h-1 - (B)(( (1 -1-- 1 G
2 (1-)))2ee² 2
L.
ai (a--1 az) B1 v 1 1 1-- GLL
262
2 32
When ar = a. the following holds:
E |IV (0.- 1/3 3/2
a² 1 8 G2
a²GLL
B < (6.-1)
(28)
V(1 (1 -82) G2 +
(1 3
2e
E | / (0.- 3/2
a² (1+18)
(29)
31)
88') +2
2(1-B1)
Telescoping it for t ,,...,T,, we have
T,
|V (0, (0.11))1) 3/3 3/2
Elf (Q+)) ((,) (f - Σt E
a² (1+
(30)
V(1 (1-B3) G2 +
2(1 ((1-B) e2
||V (0.-1) 1/1 3/2
ΣT1 E
a² (1 + B1) LT
<0(00) - a
(31)
V(1 (2) G² +
2(1 - 31)
17

---

By rearranging the terms, we have
V (0.-1) -1)1)11 3/2
E
[VS (0: 1)111/111/// 3/2
E
(32)
T
(f(0)) - fint
a1+ 31) GDL L
(1 - (3
(33)
aT
2 - 211-3:) 62
(34)
When ar =a/v the following holds for 2 2:
1-11
(35)
Vt Vi
(Vt- vt-1) a
(36)
Vt(t-1)
(37)
Vt(t-1) 1) (Vt+ Vt-1)
(38)
2 - 1)8/2
V2a
(39)
33/2
18

---

This also holds for t = by defining ag = Applying it to Eq. (27 we have
Ef < Bf (Øt-1)1+ (01-1 a4),91 (1- 1) G2
E
|V (0,-1))l;
1/33 3/2
4/3)
at
(1-
V(1 83) G2 62
0/-1 (0t-1-00 89 (1- 91-1)
G
a1a1-18 1
1 1 87 G2L
2 (1 (1)
(1- 6²
(0/ ax)² 37 B1-1) G a
2(1- 6
22
01 (at-1 - aa) 81 1 L
G
+
(40)
2 - 3,)) 2
a E (V (0.- 1)11 4//3///2 3/2
32 (1 - 1) G2
$ BBf
1/2 (1 (1-
Vi
1(1 (1- 38 +2
202BRGLI
3
LLL
+ L
81 d28,02L
(1-
(1- 31) 2 213 2
(41)
V2 - 21)
|V 10 1/3 3/2
= -B[ Vt
v(1 - 33) G2 G2+2
a² (1 + (2V2 - 1) B1) GLL
.1-+ V2nBi G2
.t
2(1 2(1-31)
(1 (1-3))c
a B² (1+ (2V2-1) 3) G
-2 Q2BRG22
(1 (1-3))² 33.
(42)
V2 - 31)² 22
Multiplying to the both sides and rearranging the terms, we have
Vi |V (0 4/3
V(1-83 G0+++ G22
- -[(()) .t a 21++2222 (2V2 1)33) G
V23 G2
(1- 7
(43)
2(1 - 2
03 (1+ + (2V2- 1) B1) G
L 1-2
1-
(44)
v2 (2(1-30) 31)222
(1 31) 6
19

---

8/2
VE |V (0.-1))1 (0--))))1
((1 (3) G2 +
(80)
EE ((:) a a1+ -2V2-1) 1)) GL
2(1 2
V2
T
ト + 0 (1+ (2V2 1) B1) G T
aBPGL
T
Σ
Σ r-2 (45)
+
-
V2 - 31)² 2
(1
B,) 22
2
1=1
1=1
1=1
fsup - fint T +
(1+ (2v2 - 1) 31) G
2(1 =[-
G2
1+ {"
T
aB1 (1+ + (2v2- G
1+ {" p
(1-B)e
11
aBPGL
T
1+ 1
dd
(46)
(L (1-3)) 1-) 2
Saup - funt T a (1+ (2V2 - 31) L
V2
(2VT-) -1)
2(1 - e
(1-
a (1 + (2V2 - 11) G
(1 + log + agPG2I. (2 1
2
(47)
(1 (1-
T
Therefore, the following bound is derived.
E [ 3/2)
min
v (0.-1) 4/37 3/2
E ViE
(48)
Σ Vi
ΣT ViE
V f (0.=1))1)
(49)
S6' Vidt
3CT (Jrup
Sint)
3a (1 (2 B1) CTGLL
(2
VT +
2a
4(1 BVV
V2 -1)
T T3/2
3031 (1+ (2v2- 1) B1) CTGLL
log
2
2/2(1 (1 - 3)²
T3/2
T3/2
2 - 31
T3/2
T5/2
(50)
where Cr (1- G2 +2.
G Lemmas
Lemma G.1. For all 0 e R and t 2 1, the following holds
V (0.11) G.
(51)
20

---

Proof.
| (0,
|E [ || 11 I2) - IE
(52)
(53)
S.
(54)
The first inequality holds because [[(g:) v ( and the second incquality holds due to Assumption 2.5
Lemma G.2. For all 0 E ERR and 1 2 1, the the following holds
E [1gg: []] S G
(55)
Proof.
E [1l S E |1 1/2
(56)
G
(57)
where the first inequality holds due the Hölder's inequality and the second one holds due to Assumption22 2.5
Lemma G.3. For the RMSprop algorithm, the following holds for t 2 1:
D
E
v1);
s (1- G²
(58)
Proof.
D
D t
E
v0);
=E E
(1 (2)
1 (g
(59)
4= Rm
S (1 - 82) 2
81 k
(60)
(1 81) G²
(61)
Lemma G.4. For the RMSprop algorithm, the following holds:
E
v (0.-))
(voz+e)
+
gt
1 E (0--1)
Vf
(voz +
9:
)
2G 2GV1 - 1-32L
(62)
2
Vor + +
Proof.
E
VJ (0-1)
gt Vo, +
)1
>>
(Vʃ (0.1))))), (g:);
(63)
(v+)) c²
We define 6, as follows:
8, = 53 + (1 11 og.
(64)
21

---

Using this, the following holds:
[ (0,-1)); (gt);
((t), +
(Vf (0,-1)), (g));
1
1
E
E
(Vf (f)())))), (g)),
(65)
(oe), +
((ui);+ +c V(oo); + +
V (01-1))
E
1E
(V (0-1))))) (9));
(66)
(or), +
((oo), V(oi), +
(Or(0)))))
(VS (0, (g:);
IB
(67)
((rr) +
((vz); + /(oe),
where the last inequality holds due A 2 -|A]. For the second term, the following holds:
(V. (0x-1))) (gt); (Vo)))+ + ((rr, +
(g)) (g.)
(1 (2) (0.,))))), (9)); ((v)); + 2 2 (00); (V(v)) 2 + (0,); +2²)
(68)
(g), E (g:)
(VV/ (g:) (g:);
$ (1 -2) ((x,, + ((i)); + (2)
(69)
((x)); + () (ot), +
where the last inequality holds due to the triangle inequality. For the first term, the following holds:
(V. (0. (9:), E (
E
V(v.), + 2 ((0:), + x2)
(gt)? (g.) 21
(Vf (0.1)))
5 (1 32)
E
+(1 - 82) E
(70)
4 (Be); +
((vt), + <2) ((o)) 22 3/2
(gh)²
(gt)
(V (0.11)) 1)) 4 4 (o., +
<
E
+
(71)
(1 32)
VI B2
((v)); +
(Vf (0-1)))
G E
(gi)
(72)
(1 32)
4 (8x), +
V1 B2
(v:); +
The first inequality is derived using the following fact:
+ 32
AA > 0, R, xy S 2
(73)
2
2X
22

---

For the second term of Eq. 69 the following holds:
(V. (0e-)), (gt) v (g:);
E
(74)
((w+); + 22) (0x); c
(g
(V. (0, 1)) 4V (v)); + + E (9
(g.);
(g:);
< (1 - B2)
E
+ (1 82)
(75)
Vūr); ; e ((t)) + 2))
(gt)) )
(g))
f (0 1))
(76)
(1 82)
4 (v)) t
VI 3e (0)); - +2)
f (0 1)); 4 (o)) +
G E
(gt)
< (1 - 32)
(77)
v1 B2
(vi), + 2
The first inequality is derived using Eq. Putting these inequalities together, the following is derived:
73
E
V (0+-1)
gt Vv +
D
f (0(-1))
(g))
E
-GG -2GV1- BOE
(78)
/0((1)) +
(we); + 62
1=1
)]
27
Vi (0c-1))
gt
2G 2GV/I 33LE
9t
(79)
Vū +2
V0, 22
Lemma G.5. For the RMSprop algorithmm, the following holds:
(1 32
G²
9+
s D
log 1+
T log 32
(80)
VU: +
22
Proof.
2
(g))
g:
(81)
Vt +
(U:); +
(g))
(1 32) (gr)
(82)
11)), +22
1 8g (ve); , E
1
1 (1 B2) (g))
log
(83)
32
(V:); 22
1
(ve), +
log
(84)
1 B2
Bo (V1-1); +
(Ui); +
(V--1); +
log
+ +log
(85)
32
(0/-1); +
Ba (Vr-1); + +(²
(v1), +
log
log 32
(86)
1 82
(v.-1); +
23

---

Σ
1
log
UUT)
+ 2
T log B2)
(87)
(0)) + 6
1
32
1
(1
35 G2
log
14
7 log 32
(88)
1
32
62
(g+)
gt
E
E
(89)
Vol + 2
(01); +
1=1
(UT); 2
]
DT log 32
log
1
(90)
1
82
1 82
D
18 [((T):
DT log 32
log
|
(91)
62
1
B2
1=1
1+ (1- 32
1--32)'²
G2
log
T log 32
(92)
1 B2
22
Lemma G.6. For the RMSprop algorithm, the following holds:
E |V (0.-1) 4/33) 3/2
)
(V
E
(0:-)))
2
(93)
+ 22
V(1 8b) G2 +
Proof.
E
V (0--))
9t
Vū. + e2
D
(Vf (0(-))) (gt);
Σ
E
V(oe); +
1=1
(Vf (0:-1))²
E
V B2 (we-1), 1), +
V f 101
2 E
D (öx), +
V (0, 11ll 1/11
3/2
E
D (ox)s
+
E
M (0,
(0.-1)11 4/3
3/2
(94)
35) G +
The second equality holds due to Assumption 2.2 The first inequality holds because (w:); > 0 for all i 1, The second inequality holds due to the Hölder's inequality. The last inequality holds
due to Lemma G.3
Lemma G.7. For the ADOPT algorithm, the following holds for 2 1:
(at-1 - 04) B1
gi
Φ: Φ1-1
1 1-31 31 m aq
(95)
max V 0
where we define oo a.
24

---

Proof. For t = 1, the following holds by definition:
1
81
1)
Φ
Φ0
01
11 31
00
(96)
1- 31
(0)
00)
(97)
1 B1
0191
(98)
max {vDD.E)
For t 2 2, the following holds:
81
Φ -
(0, 01-1)
(0:-1 01-2)
(99)
1- 31
1 31
00--18mm4- (x.7+))
(100)
1 - 31
01-1181 an fi max [v
gr.
(101)
1- 31
(at-1 a1) 81m,-1 0rg B1)
9t
(102)
1 31
max (v)
(ap-1 a1) 81
1 B1 M max cv
(103)
L
Lemma G.8. For the ADOPT algorithm, the following holds for 2
131
Φ1
0
B1 m.-1.
(104)
Proof. For 1, Eq.
104 holds obviously because Φ = = and my = 0. For 2 2, the following
holds:
1
1)
31 0.-2
Φ O2-1
0-11
(105)
1- B1
1- Bi
31
(0,-1 - -0--2)
(106)
- Bi
01-1 31
m,
(107)
1 B1
Lemma G.9. For the ADOPT algorithm, the following holds for 2
E
V (0t-1) ($:: -1-1)
(00-1 - 0r) 31 (1 -3 11- 62
E V (0, 1)1
3/2
4/3
3/2
(108)
(1 (1- V 82
2
+ +2
3))
@2
+
Proof.
V (0.-1) (Ø: $:-1)
(at-1 -ar) 3
(0--1)))mm_11a - ((...))))) max { g: (v
V (0.-1)) m -
(109)
1 Bi
(a--1 ar)
B1 |V (0.-)))
| a2 V Vf (0,1) max {1,
(110)
1 31
(a-11
3
|m,
0. (/
gt
(111)
31
max (VU,_1,
25

---

By taking the expectation for both sides, the following holds:
VJ (0. (0.-1)" (0: - -
(0.-1- -a,) B/G
9:
E m 11l a,E
Vf (0t-1)
(112)
1 -31
max { ((
(0--1
04)B1..
(V (0,-1)); (g));
E [|| 04
E1
(113)
1- 31
max { V 1//1011))) 1)
(at-1
-a+) B1G
(V.(()--)));
E
[1-
0x
II
(114)
1 1-
31
max (V(v--))
1=1
(at 1 a2) BiG
(Vf (O,-1))
EF [ 11
033 Σ TR.
(115)
1 $1
( 1)+ + e
(0t-1 1 (1) B
/V/ (0. 1)1
E m a
(116)
1 $1
Eis (U )) +
4/33 3/2
(at 1 (z) B
E
V f ( 11,
E
||-
(117)
$1
El (v.-1), 1/1
+
3/2
0t-1 - B,G
E
V (0, 11
1- 31 IE
[[[ [||1
(118)
83) G2 +2
(0 (4) 11 (1 1181-11)
G2
E
V (0, )|
11//33
4/3 3/2
(119)
(1 (1- (1) V 1/2 +22
((1 8)) +
Lemma G.10. For the ADOPT algorithm, the following holds for t 2 0:
D
E
Σ
(v));
< 311-5))6 G² G².
(120)
1=1
Proof.
D
D
E
Σ (p))
= (1 32) Σ
35
& (9x~1)²
(121)
1=1 k=1
S (1 32) G2 Σ 3/-
(122)
k=1
(1 8) G².
(123)
Lemma G.11. For the ADOPT algorithm, the following holds for 0 <tsT
E
m
(124)
2
26

---

Proof.
E
m+
E
81 + +1-
gt
(125)
max { (1
1
B7 m1-111 (1--1 -1)2 E
9t
+
+ (1 m)
gt
max 11ot-11 0t 1,
max ( 11
(126)
$ E
81 [m 112 + +(1- - -11)
(127)
max
01-1,
< E
3 || 2 +
-
31
|1
(128)
62
31
< E
1
(129)
2
k=1
(1
81) 2
(130)
2
(1 8)
(131)
2
$ 2
(132)
First inequality is derived using the following fact:
A
1
AA > 0, R x TTy
+
|/0l2
(133)
2
2x
By setting X (1 B1) /B,, = 31m--11 (1 B1) ge/ max (V0e-1,6) we obtain
gt
$ B1 (1 - B1) (Im--1P
gr
231 (1 (1) m
(134)
max e
max (v
Injecting it into Eq.
126 we obtain Eq. 127
Lemma G.12. For the ADOPT algorithm, the following holds for 0.
m, 11 < G
(135)
E
27

---

Proof.
E mill E
(1 31)
8
9k
(136)
max {
I (1 - B1)
B9-KE
9k
(137)
max ( 1.
k=1
gt-k
S (1 - (1)
E [1|g
(138)
1=1
1- 81
8 |1ga 12)
19]/2 1/2
(139)
km
31) G
81
(140)
(1
B1) G
(141)
6
G $
(142)
E
Lemma G.13. For the ADOPT algorithm, the the following holds for t. 1:
E 11 0,-1 11 Φ1-111]
04-1 (01-1 - (rt);37 B8 (1 - (8-11) G2 a1a1-181 1-80- G22 G2
(143)
(1)1=11)
Proof.
|| 1 - 01-1/1 || - -1-111
01-131
(a:-1 ax) 81
gt
mi-1
m:-1 aq
(144)
1 - B1
1 31
max {
04- 11
||m,-1|| ||
(0t-1 az) B1
|| 1| + Ctt
gt
(145)
31
1 - $1
max /ul, 1,
< 0t-1 (0/-1 an) 189
|m-_1l 112
+
0,00-111
|m
9t max
(146)
(1 (1-313 31)
1-
31
(VeL_1,c)
Taking the expectation yields:
E |1 01-11 || $--111
04-1 (01-1 -as) 187
E
2
0:04-181
E
|m,- ||
9t
(147)
(1- 21)
31
max
{v 01 1,
01-1 (a/-1 av) 32
0,:4-1 181
E
| [12]
E
@||mm=-11 [|9:|]]
(148)
- (1=
(1
81)e
0--1 (0t-1 - a3) (1 - )
G2
0:0 0,04-1811 1B1
1
1-- G2
(149)
(1- -1)²
(1- 32
28

---

Lemma G.14. For the ADOPT algorithm, the following holds for t 2 1:
|l:: - +111
(0t-1 - -a)
-1--1 11 11-111111111 G2
af
a a110111- (a1-1-ar)666 81 1 1/-1 6
(150)
(1- 6²
22 (1-31) c2
Proof.
|1: - $1-111
(a1-1 -Ot) $1
9t
m4-1 aq
(151)
1 1-31
max (v 11 1/-
(a--1 -xa)² 39
|| 11| +
gt
(1-B1 B11²
max (V V
0t (ar-1 -at) 31
9t
m.
(152)
1 1 11
B1
max
(a--1 az)² BP
|
1|
a2 2
|g
(1- - 31)²
(0t-1 - -at) 11
||m
9t
(153)
1 1-31
max (v
(at-1 az)) 37
ar (at-1 - (ke) B
|
1|
|9:11
|| [g
(154)
(1 31)²
2
(1 B1)
Taking the expectation yields:
E
10 - 111
(ac-1 2 32
E |lm
[|aul]]
au (ai-1 (0)-1-a() E [|]mm [Ig.
(155)
- (1-31)²
(1-3))e
(ai-1 2 (1 1 G²
a²6²
as (ai-1 ar) B3
EE'
M+ |1
(156)
(1 (1-31)²& 2
- (1-B))e
(ai-1 - an)2 87 (1 B
G
a²
04 (ah-1 arr B1
31 -22
(157)
(1- 2
(1- (1-31) 31 6
H
Additional Experiments
Deep reinforcement learning:
We train reinforcement learning (RL) agents using the soft actor crtitic algorithm [Harrnjja et al. 2018 with ADOPT for the optimizer. As a benchmark, we use a continuous control tasks of HallCheethh-ye on MuloCo simulator [Todorov et al.) [012]. For comparison to ADOPT, Adam used as a baseline optimizer. We follow the hyperparameter settings recommended by Stable- Baselines3 Raffin et al. 2021]. and just change the choice of an optimizer. The result is shown in Figure 6 The error bars indicate 95% confidence intervals of three trials. We observe slight performance improvement by using ADOPT instead of Adam.
29

---

6000
10000
4000
Return
7500
Return
5000
Optimizer ADOPT
2000
Optimizer
2500
ADOPT
Adam
Adam
0
0.00
0.25
0.50
0.75
1.00 1e6
0.00
0.25
0.50
0.75
1.00
Step
Step
le6
HalfCheetah-v4
Ant-v4
Figure 6: Performance comparison between Adam and ADOPT in reinforcement learning.
Figure 7: Comparison of MMLU scores for LLaMA-7B finetuned via instruction following using AdamW and ADOPT.
I Details of Experimental Setups
I.1
Code
Our implementation for the experiment is available at uttttts://ggthhui wtw.://w...commShheee22200000000
adoppt
1.2
Total amount of compute
We our experiments mainly cloud GPU instances with 8x A100. It took approximately 320 hours for our experiments in total.
1.3 License of Assets
The MNIST database is downloaded from http://yann. lecun. com ftom/exdb/m/e exxdbmmistt ist which license-free. The terms of access for the ImageNet database is provided at hittps:///// hopp:://wr //www
Datasets:
image-net. org/ download The dataset of Stanford Alpaca is CC BY NC 4.0 (allowing only non-commercial use).
Pretrained models: The pretrained model of LLaMA is provided under GNU General Public License v3.0.
Simulator: MuJoCo is provided under Apache License 2.0.
Code: Our implementation of ImageNet classification is based on the Torchvision's 's official training recipe provided stttps://gitee http///uaaaattttoooiiiiioioooooo http///eeeee/ttt/eeeeeeeeeeeeeee classification Torchvision is provided under BSD 3-Clause License. We use the official imple-
30

---

att mentation of NVAE arovided at httpg////ggghhbb.o////////////// whose license is described
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution is to demystify the cause of non-convergence of Adam, which clearly written in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are described in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
Assumptions and proofs are provided in Section 2 and the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental settings are provided in Section and the appendix. We also share the implementation of the experiment.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
Data and code are provided in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//guiii wwwticccguie//eeeeeeeeeii CoodeSubeiiieeeec ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed experimental settings are provided Section 5 and the appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported in all the figures and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
34

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational resources used in experiments reported in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:://eeuipp.eeeee httprriii..ccpuulleeeetcccccueee
Answer:
[Yes]
Justification:
We confirmed that our research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed them in the last section of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
35

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
They are provided both the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
36

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "ukD1Azz1fT", "reviewer": "Reviewer_f5HN", "summary": "The paper proposes a new adaptive gradient method called ADOPT, which addresses the non-convergence issue of popular methods like Adam and RMSprop.  The method modifies the calculation of second moment estimates and the order of momentum calculation and scaling operations. Extensive numerical experiments demonstrate that ADOPT achieves competitive or superior results compared to existing methods across various tasks.", "questions": "see weakness.", "limitations": "Yes, The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a new adaptive gradient method ADOPT that is as easy as the implementation of Adam, and enjoys easy convergence proofs.\n\nThe paper gives in-depth analysis for the convergence of ADOPT with toy examples, in comparison with the failure cases of Adam.\n\nThe paper conducts comprehensive numerical experiments on various tasks, demonstrating the competitive performance of ADOPT \n compared to the widely used Adam.", "weaknesses": "The convergence of a modified version of Adam is not significant from theoretical sense unless the ADOPT can beat the performance of Adam in practice given existing convergence proofs of Adam.\n\nFrom the empirical results, the performance of ADOPT is not superior over Adam very much. People may be reluctant to use ADOPT in practice.\n\nMore importantly, the algorithm 1 (ADOPT) seems to require storage of three parts: g_t, m_t, v_t, which is more than what the standard Adam requires. This is quite significant drawback of ADOPT if this cannot be optimized.\n\n------------------------------------------------------------------------------------------------- comments after rebuttal -------------------------------------------------------------The authors' response clearly resolved the memory cost concerns. I would like to increase the score to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We will answer your questions to address your concerns. Please also refer to “Author Rebuttal by Authors” and the attached PDF.\n\n> The convergence of a modified version of Adam is not significant from theoretical sense unless the ADOPT can beat the performance of Adam in practice given existing convergence proofs of Adam.\n\n> From the empirical results, the performance of ADOPT is not superior over Adam very much. People may be reluctant to use ADOPT in practice.\n\nYour point is that ADOPT does not seem to perform so well compared to Adam in the practical experiments in Section 5 except for the toy examples. In fact, what our theory shows is that Adam fails catastrophically when the constant term of Adam's convergence bound in Theorem 3.1 becomes large; hence, in a correctly tuned situation in practice (e.g., with a sufficiently large $\\beta_2$), it is natural that ADOPT will not improve significantly relative to Adam. \n\nHowever, to show that even in practice Adam can make catastrophic failures and ADOPT can avoid them, we have added an experiment in pre-training language models. In this experiment, we used the nanoGPT [1] code base to run the GPT-2 pre-training with OpenWebText. See the general response and the attached PDF for more details. We observed that Adam suffered from loss spikes and completely failed the training when the batch size was small, while ADOPT was able to train stably even with small batch sizes. We also observed that ADOPT performed slightly better than Adam even for large batch sizes. This result is consistent with theory, since the gradient noise is larger and the constant term of Adam's convergence bound is larger when the batch size is small. Thus, the experimental results confirm that ADOPT has an advantage over Adam even in practical cases such as pre-training language models. We hope these results will address your concerns.\n\n> More importantly, the algorithm 1 (ADOPT) seems to require storage of three parts: g_t, m_t, v_t, which is more than what the standard Adam requires. This is quite significant drawback of ADOPT if this cannot be optimized.\n\nI respectfully point out that this is a misunderstanding: the memory cost of ADOPT is equivalent to that of Adam. Only $m_t$ and $v_t$ need to be stored in ADOPT, and $g_t$ can be discarded once those updates are done. Presumably, this misunderstanding arises from the fact that $g_{t+1}$ is used to update $m_{t+1}$ in Algorithm 1, but this does not mean that $g_t$ needs to be stored. To clarify this, an equivalent alternative representation is given in Algorithm 2 in the attached PDF of the general response. In fact, experimental results confirm that the memory costs of ADOPT and Adam are equivalent. For example, the memory cost for pre-training GPT-2 is approximately 18 GB per GPU for both.\n\nWe would be glad to respond to any further questions and comments that you may have.\n\nThanks.\n\n### References\n\n[1] github.com/karpathy/nanoGPT", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LRPevPyAyR", "reviewer": "Reviewer_qABQ", "summary": "Motivated by the counterexample due to Reddi et al., this work designs an adaptive optimizer that can converge for the choice of beta_2 that is independent of the problem instance. Their analysis works for a more general condition than the previous works.", "questions": "- In Theorem 4.1, please specify the choice of hyperparameters (beta_1, beta_2, \\eps). I could not find this even in the appendix, even though the main text said it can be found there. In particular, does the choice found by your analysis similar to what people use in practice?\n\n- If the choice of beta's do not match with your theoretical prediction, I think the claim that the new optimizer AdaOPT alleviates the parameter-tuning is not an overclaim. At the end of the day, it seems that the authors have to tune those parameters depending on the settings. Hence, practically speaking, there's no reason for practitioners to use the proposed algorithm over the original Adam.\n\n- It seems that for your algorithm, $\\epsilon$ needs to be chosen much larger than that of Adam. How sensitive is this choice? Why is this the case? $\\epsilon$ really should be for numerical stability in the original Adam. Does your analysis suggest such a large value for $\\epsilon$?", "limitations": "Overall, the main contribution in this work seems quite marginal. For a submission of this type, I would support a clear acceptance if \n- (i) the theoretical guarantees have a noticeable innovation over the previous ones.\n- (ii) or the proposed method works much better than the previous ones.\n\nIt seems that the theoretical improvement is the removal of the uniformly upper bounded assumption of stochastic gradients.\nTo me, this looks like a minor improvement over the previous work.\nMoreover, the resulting algorithm does not seem to have major advantages over Adam. In particular, unlike the claim made in this paper, I don't think the resulting algorithm alleviates the hyper-parameter tuning. I don't see why practitioners should choose this algorithm over Adam. \nTherefore, I recommend \"borderline accept\" for this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It has overall good presentation. The main scope and results are presented well.", "weaknesses": "The theoretical results look sound. I have some questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We will answer your questions to address your concerns. Please also refer to the general response and the PDF attached with it.\n\n\n> In Theorem 4.1, please specify the choice of hyperparameters (beta_1, beta_2, \\eps). I could not find this even in the appendix, even though the main text said it can be found there. In particular, does the choice found by your analysis similar to what people use in practice?\n\n> If the choice of beta's do not match with your theoretical prediction, I think the claim that the new optimizer AdaOPT alleviates the parameter-tuning is not an overclaim. At the end of the day, it seems that the authors have to tune those parameters depending on the settings. Hence, practically speaking, there's no reason for practitioners to use the proposed algorithm over the original Adam.\n\nThe concrete convergence bound is provided in Eq. (33) in Appendix E, which shows that the convergence rate is $O ( 1 / \\sqrt{T} )$ with any $(\\beta_1, \\beta_2, \\epsilon)$. The bound gets tighter when $\\beta_2$ is chosen to be close to 1, which corresponds to practical choices. In fact, as shown in Figure 1, the training with ADOPT tends to be stable when $\\beta_2$ is close to 1, although the convergence is achieved even with small $\\beta_2$.\n\nIn terms of $\\beta_1$, there is a gap between theory and practice. In theory, the bound will be tighter when $\\beta_1$ is small, whereas $\\beta_1 = 0.9$ is used in practice. This gap is consistently observed in the literature on the convergence analysis of Adam (e.g., [2, 3]). To the best of our knowledge, the effectiveness of momentum in Adam-type optimizers is still an open question.\n\n> It seems that for your algorithm,  needs to be chosen much larger than that of Adam. How sensitive is this choice? Why is this the case?  really should be for numerical stability in the original Adam. Does your analysis suggest such a large value for ?\n\nYou are correct that ADOPT requires a larger $\\epsilon$ than Adam. This can be explained by theory: as can be seen from Theorem 3.1, the convergence bounds for Adam are of the order of $\\log(\\epsilon^{-2})$, while the convergence bounds for ADOPT in Eq. (33) are of the order of $\\epsilon^{-2}$. In other words, when $\\epsilon$ gets small, the convergence bounds of ADOPT become loose more rapidly than those of Adam. For this reason, it is safer for ADOPT to use a slightly larger value for $\\epsilon$ than Adam. Our default settings have been found to work robustly in all experiments, so tuning of the $\\epsilon$ is rarely needed, even in practice.\n\n> Overall, the main contribution in this work seems quite marginal. For a submission of this type, I would support a clear acceptance if \n\n> (i) the theoretical guarantees have a noticeable innovation over the previous ones.\n\n> (ii) or the proposed method works much better than the previous ones.\n\n> It seems that the theoretical improvement is the removal of the uniformly upper bounded assumption of stochastic gradients. To me, this looks like a minor improvement over the previous work. Moreover, the resulting algorithm does not seem to have major advantages over Adam. In particular, unlike the claim made in this paper, I don't think the resulting algorithm alleviates the hyper-parameter tuning. I don't see why practitioners should choose this algorithm over Adam. \n\nThank you for your suggestions. To demonstrate the practical effectiveness of our ADOPT more clearly, we have added an experiment in pre-training language models, in which Adam tends to suffer from optimization difficulties like loss spikes. In this experiment, we used the nanoGPT [1] code base to run the GPT-2 pre-training in OpenWebText. We observed that Adam suffered from loss spikes and completely failed the training when the batch size was small, while ADOPT was able to train stably even with small batch sizes. We also observed that ADOPT performed slightly better than Adam even for large batch sizes. This result is consistent with theory, since, when the batch size is small, the gradient noise is larger and the constant term of Adam's convergence bound in Theorem 3.1 also gets larger. Thus, the experimental results confirm that ADOPT has an advantage over Adam even in practical cases such as pre-training language models. We hope these results will address your concerns.\n\nWe would be glad to respond to any further questions and comments that you may have.\n\nThanks.\n\n\n### References\n\n[1] github.com/karpathy/nanoGPT\n\n[2] Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. \n\n[3] Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and lower bound of adam’s iteration complexity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FgCy3Gghj", "reviewer": "Reviewer_AhBk", "summary": "The paper titled \"ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate\" introduces a new adaptive gradient method named ADOPT. This method aims to address the non-convergence issues of the Adam optimization algorithm. Adam, despite its popularity in deep learning, does not theoretically converge unless the hyperparameter β2 is chosen in a problem-dependent manner. The paper proposes ADOPT, which achieves the optimal convergence rate with any choice of β2, without relying on the bounded noise assumption. ADOPT modifies Adam by removing the current gradient from the second moment estimate and changing the order of the momentum update and normalization. The paper also presents extensive numerical experiments showing ADOPT's superior performance across various tasks, including image classification, generative modeling, natural language processing, and deep reinforcement learning.", "questions": "1. How can the assumption on the second moment being uniformly bounded be relaxed? Are there any potential methods or future work suggested to address this?\n2. What is the computational cost of ADOPT compared to Adam and AMSGrad? Is there a significant overhead that might affect its practicality in large-scale applications?\n3. While ADOPT performs well on a variety of tasks, how does it perform on tasks not covered in the experiments? Are there specific types of problems where ADOPT might not be as effective?", "limitations": "see the weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides a robust theoretical foundation for the proposed ADOPT algorithm, demonstrating its convergence with an optimal rate\n2. The proposed method is practically significant as it eliminates the need for problem-dependent tuning of the β2 parameter, making it more user-friendly and broadly applicable.\n3. The paper includes comprehensive experiments across various tasks, showing that ADOPT consistently outperforms Adam and its variants.\n4. The paper clearly identifies the non-convergence problem of Adam and provides a well-justified solution in ADOPT.", "weaknesses": "1. The analysis still relies on the assumption that the second moment of the stochastic gradient is uniformly bounded, which might not always hold in practice.\n2. The paper could have included comparisons with more recent optimizers beyond Adam to strengthen its empirical claims, like the recent work CO2 (\"CO2: Efficient distributed training with full communication-computation overlap.\" ICLR (2024).).\n3. The paper does not thoroughly discuss the potential computational overhead introduced by ADOPT compared to Adam and other optimizers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We will answer your questions to address your concerns. Please also refer to the general response and the attached PDF.\n\n\n> The analysis still relies on the assumption that the second moment of the stochastic gradient is uniformly bounded, which might not always hold in practice.\n> How can the assumption on the second moment being uniformly bounded be relaxed? Are there any potential methods or future work suggested to address this?\n\nAs you mentioned, our analysis still relies on the bounded second moment assumption, which is sometimes violated in practice, although it is milder than the bounded stochastic gradient assumption used in the previous works. A promising direction to further relax this assumption is to use the bounded variance assumption, where $\\mathbb{E} [ \\|| g - \\nabla f \\||^2 ]$ is assumed to be bounded. We mentioned it in the third paragraph of Section 6 as a limitation and future work.\n\n> The paper could have included comparisons with more recent optimizers beyond Adam to strengthen its empirical claims, like the recent work CO2 (\"CO2: Efficient distributed training with full communication-computation overlap.\" ICLR (2024).).\n\nThank you for your suggestion. We think that CO2 is specifically designed for efficient distributed training, so their contributions are orthogonal to ours. In our experiment, we compare other optimizers which share the motivation of addressing the non-convergence issue of Adam (e.g., AMSGrad). Of course, combining the techniques of other works (e.g., CO2) with our ADOPT could be a promising direction to further improve the performance, but we leave it for future work.\n\n> The paper does not thoroughly discuss the potential computational overhead introduced by ADOPT compared to Adam and other optimizers.\n> What is the computational cost of ADOPT compared to Adam and AMSGrad? Is there a significant overhead that might affect its practicality in large-scale applications?\n\nThank you for your comment. Let us clarify the computational cost of ADOPT compared to Adam and AMSGrad. The computational cost of ADOPT is equal to that of Adam and less than that of AMSGrad. Since both Adam and ADOPT need to store momentum $m_t$ and second-order moments $v_t$, their memory costs are about the same. In addition, Adam computes bias corrections when updating parameters, while our implementation of ADOPT omits them, making ADOPT slightly less computationally expensive than Adam. On the other hand, AMSGrad requires storing $\\hat{v}_t$ in addition to $m_t$ and $v_t$, so its memory cost is larger than ADOPT (and Adam).\n\n> While ADOPT performs well on a variety of tasks, how does it perform on tasks not covered in the experiments? Are there specific types of problems where ADOPT might not be as effective?\n\nAs far as we have experimented, ADOPT has always shown better results than Adam. To further strengthen our experimental results, we also performed pre-training experiments on GPT-2 and found that ADOPT performed better than Adam in those experiments as well. In particular, we observed that Adam suffers from loss spikes when the batch size is small and the gradient noise is large, while ADOPT always performs well stably. See the general response for more details.\n\nWe would be glad to respond to any further questions and comments that you may have.\n\nThanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBLsUjMULQ", "reviewer": "Reviewer_bGLP", "summary": "The submitted work analyzes the divergence of Adam and RMSprop in smooth nonconvex settings. The authors propose a new optimizer ADOPT, whose convergence does not depend on the second-moment coefficient $\\beta_2$.\nThe proposed optimizer is evaluated in toy settings, image classification, language modeling (finetuning settings), and reinforcement learning.", "questions": "* In realistic settings, where is the marginal convergence speed in ADOPT coming from? My understanding is that non-convergence of Adam would matter near the minima only. Faster training way before reaching the minima is not expected. In particular, in language modeling pretraining, convergence is usually not achieved.\n* Does the difference in the recommendation of the epsilon parameter for ADOPT simply arise because it is added inside the square root instead of outside?", "limitations": "See Weaknesses above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper tackles the non-convergence problem of the Adam optimizer. As Adam is the go-to optimizer for deep learning, deeply understanding Adam optimizer in detail is valuable.\n* The paper resolves the non-convergence issue of Adam without requiring bounded noise assumptions or specific $\\beta_2$.\n* The experiments are performed in diverse settings, including image classification, generative modeling, and deep reinforcement learning.", "weaknesses": "* The non-convergence of Adam is not an issue in practice and I am unsure about the practical implications of ADOPT over Adam. The non-convergence issue arises near the minima and most modern deep learning models are not trained to convergence.\n* The paper does not thoroughly investigate the sensitivity of ADOPT optimizer parameters in practical scenarios. A more comprehensive study on the optimizer parameters and comparison with Adam is required to understand its robustness and practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We will answer your questions to address your concerns. Please also refer to the general response and the PDF attached with it.\n\n\n> The non-convergence of Adam is not an issue in practice and I am unsure about the practical implications of ADOPT over Adam. The non-convergence issue arises near the minima and most modern deep learning models are not trained to convergence.\n\n> In realistic settings, where is the marginal convergence speed in ADOPT coming from? My understanding is that non-convergence of Adam would matter near the minima only. Faster training way before reaching the minima is not expected. In particular, in language modeling pretraining, convergence is usually not achieved.\n\n> The paper does not thoroughly investigate the sensitivity of ADOPT optimizer parameters in practical scenarios. A more comprehensive study on the optimizer parameters and comparison with Adam is required to understand its robustness and practical utility.\n\nI agree with you that in modern deep learning models, such as large language models, training is rarely run until full convergence is achieved. However, even in such cases, Adam's non-convergence can be a problem. For example, when training with a large dataset or small batch size, the constant term in Theorem 3.1 can become non-negligible due to the large gradient noise. In such cases, Adam may become unstable even in the early stages of training. To confirm that this happens empirically, we have added a new experiment of pre-training of GPT-2. See the figure in the PDF attached to the general response for the results. We observed that ADOPT is always stable in this experiment, whereas Adam actually diverges in the early phase of training due to loss spikes when the batch size is small. We also observed that ADOPT performed slightly better even when the batch size was large. We believe that these results clearly demonstrate the practical effectiveness of ADOPT in modern deep learning.\n\n> Does the difference in the recommendation of the epsilon parameter for ADOPT simply arise because it is added inside the square root instead of outside?\n\nThis is a good question. In fact, the reason that ADOPT requires larger $\\epsilon$ than Adam is not simply due to the order of square root and addition. This can be explained by theory: as can be seen from Theorem 3.1, the convergence bounds for Adam are of the order of $\\log(\\epsilon^{-2})$, while the convergence bounds for ADOPT in Eq. (33) are of the order of $\\epsilon^{-2}$. In other words, when$\\epsilon$ is small, the convergence bound of ADOPT becomes loose more rapidly than those of Adam. For this reason, it is safer for ADOPT to use slightly larger $\\epsilon$ than Adam. However, our default settings have been found to work robustly in all experiments, so tuning of the eps is rarely needed, even in practice.\n\nWe would be glad to respond to any further questions and comments that you may have.\n\nThanks.\n\n### References\n\n[1] github.com/karpathy/nanoGPT", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their comments. They are insightful and help us to make our paper better. We have added new experiments and explanations to address the reviewer's concerns. Please also refer to the individual responses to each reviewer.

## Additional experiments of pre-training GPT-2
Since many reviewers seem to have concerns about the practical effectiveness of ADOPT, we have added a new experiment to reinforce it. In this experiment, we ran a pre-training of GPT-2 using the nanoGPT [1] code base to compare Adam and ADOPT. We used OpenWebText as the training data. Experimental setup conformed to the default settings of nanoGPT except for the selection of the optimizer. We also tested a case in which the total batch size was changed from 480 to 96, as a setting where the gradient noise becomes larger. The results are summarized in Figure 7 of the attached PDF file. The most notable finding is that in the small batch size case, Adam causes loss spikes in the early stages of training and fails to converge, while ADOPT is always able to train stably. This is consistent with Adam's theory of non-convergence. As the gradient noise increases, $G$ in Theorem 3.1 also increases, and the constant term in Adam's convergence bounds becomes non-negligible especially when using a large-scale dataset like OpenWebText. As a result, Adam is more likely to fail to train in such cases. Our ADOPT, on the other hand, does not suffer from this problem because it can always guarantee convergence. We also observed that both Adam and ADOPT work well when the batch size is large, but even in this case, ADOPT performs slightly better.


## Clarification of memory cost of ADOPT
Some reviewers have raised concerns about the memory cost of ADOPT, so we will address them as well.
The memory cost of ADOPT is exactly the same as that of Adam, with $m_t$ and $v_t$ as the two parameters to be stored during training; AMSGrad requires storage of $\hat{v}_t$ in addition to these, making its memory cost larger than that of ADOPT and Adam. We also experimentally confirmed that ADOPT and Adam have the same memory consumption during training in the GPT-2 experiment. We observed that both ADOPT and Adam require 18 GB per GPU during training.

We hope that it will address the reviewers' concerns. We would be glad to respond to any further questions and comments that you may have.

Thanks.

References

[1] github.com/karpathy/nanoGPT

Meta-review: The paper introduces a new adaptive gradient method named ADOPT, which is designed to address the non-convergence issues of Adam, particularly when the $\beta_2$ parameter is not chosen in a problem-dependent manner. ADOPT modifies Adam in the computation of the second moment estimates and the order of momentum updates. It is claimed that ADOPT achieves optimal convergence rates without relying on the assumption that the gradient noise is uniformly bounded.

The reviewers acknowledged the theoretical contribution of ADOPT, though the contribution is marginal; for example, the convergence still relies on the assumption that the second moment of the stochastic gradient is uniformly bounded. Besides, the reviewers also agreed that ADOPT could be beneficial in specific scenarios, as shown by experiments. Overall, the reviewers are mostly supportive of this paper, and I think this paper could be accepted.
Decision: Accept (poster)