On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
Authors: Huan Li
Yiming Dong
Zhouchen Lin
Venue: NeurIPS 2025 poster ()
Keywords: AdamW
Adam
convergence analysis
nonconvex optimization
Primary area: optimization
TL;DR: This paper establishes the convergence rate $\frac{1}{K}\sum_{k=1}^K E\left[\|\nabla f(x^k)\|_1\right]\leq O(\frac{\sqrt{d}C}{K^{1/4}})$ for AdamW.

Abstract: As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning. However, its convergence behavior is not theoretically well-understood. This paper establishes the convergence rate $\frac{1}{K}\sum_{k=1}^K E[||\nabla f(x^k)||_1]\leq O(\frac{\sqrt{d}C}{K^{1/4}})$ for AdamW measured by $\ell_1$ norm, where K represents the iteration number, d denotes the model dimension, and C matches the constant in the optimal convergence rate of SGD. Theoretically, we have $E[||\nabla f(x)||_1]\geq\sqrt{\frac{2d}{\pi}}E[||\nabla f(x)||_2]$ when each element of $\nabla f(x)$ is generated from Gaussian distribution $\mathcal N(0,1)$. Empirically, our experimental results on real-world deep learning tasks reveal $||\nabla f(x)||_1=\varTheta(\sqrt{d})||\nabla f(x)||_2$. Both support that our convergence rate can be considered to be analogous to the optimal convergence rate of SGD.

Vd Vd
On the O( Convergence Rate of AdamW Measured by l Norm
K1/41
Huan Li 1m
Yiming Dong
Zhouchen Lin 2,3,3
lihuanss@nanaai.o edu.
ymdongdstu pku.edu...
zlinêpku. cer.ppuu.ddu...om
1. College of Artificial Intelligence, Nankai University, Tiajjin, China
2. State Key Lab of General AI, School of Intelligence Science and Technology. Peking University
3. Institute for Artificial Intelligence, Peking University, Beijing, China
4. Pazhou Laboratory (Huangpu), Guangzhou, China
Abstract
As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning. However, its convergence behavior is not theoretically well-understood. This paper establishes the convergence rate & E [xV)((>))) [[a] O( váC for AdamW measured by & norm, where K represents the iteration number, denotes the model dimension, and C matches the constant in the optimal convergence rate of SGD. Theoretically, for any high-dimensional vector [V](x)]l & [xx))l1 (x)lf
K1/4
have
and E fx) 2 V 2d [II f when each element of f is gener- ated from Gaussian distribution (0, 1). Empirically, our experimental results real-world deep learning tasks reveal [x)f(x)]lı ((xd)] Both support that our convergence rate can be considered to be analogous to the optimal K
E E [x < O k174
convergence rate of SGD.
K
Introduction
AdamW, which modifies Adam by decoupling weight decay from gradient-based updates, has emerged as the dominant optimizer for training deep neural networks, particularly for large language models. Adam W represents the pinnacle of adaptive gradient algorithms, having developed through
the progression of AdaGrad 22, RMSProp 3 Adam 4 and finally AdamW (5) itself. Although the literature on the convergence analysis of adaptive gradient algorithms is quite extensive, there has been little research on convergence properties of AdamW.
Recently, Xie and Li (6) proved that if the iterates of AdamW converge to some X then X is KKT point of the constrained problem
min fxx,, s.t. ||x|| S X'
XERJ
where f(x) is the nonconvex objective function. 11 is the infinity norm, and À is the weight decay parameter. Moreover, x is KKT point of problem (I) iff (6]
| - and (x(x))(x)) +VVx))) 0.
Xie and Li characterized which solution does Adam W converge to, if it indeed converges. The next fundamental question to address whether and how fast AdamW converges. Zhou et al.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Hyper parameters: 17, 0, (, X, conducted preliminary exploration on this problem. However, their analysis requires the weight decay parameter to decrease exponentially, making AdamW reduce Adam finally. To the best of our knowledge, aside from ? we have not found any other literature addressing the convergence issue of AdamW.
In practical deep learning training, we often initialize the network weights small and employ modest weight decay, for example, À 0.01, which empirically confines the optimization trajectory within the foo norm constraint, as empirically demonstrated in Figure 3 That is, | S ā for some < 1, making (Ax, fx)) + fx))), 1 lower bounded by (1 V l1- This key property enables the use of ||V (x)]l1 as an effective yet significantly simpler convergence metric for Adam W in practical settings.
Building on the above observation, this paper focuses on the convergence rate of Adam within the constraint in problem [ Specifically, we prove the following convergence rate for AdamW
( vd K1/4 Vo3L(5(2)
[dL[((x)) S*)
[x>]((>>)]]+>>> so
(3)
K
by proper parameter settings such that |x*||x < for all iterates, where K is the total iteration number, d is the model dimension, da is the gradient noise variance, L is the Lipschtiz smooth constant, and f is a lower bound of fxx). Recall the classical convergence rate of SGD B
1 K Σ E [IV)(())
so (Vo2 VoLL([(x--- -
K1/4
which matches the lower bound of nonconvex stochastic optimization (9]. Comparing (3) with (4), we see that our convergence rate (3) also achieves the same lower bound with respect to K Jas L. and f(x²) f The only coefficient left unclear whether it is tight is the dimension Theoretically, we have f < [ S V fxx)l22 for any high-dimensional vector x
and E (x)V]( 2
[x) [22] when each element of V (x) is generated from Gaussian
distribution (O0, 1). Empirically. we have observed V [xx)]] ((v
real-world deep learning tasks, as shown in Figure 2 Thus, we could say that our convergence rate (3) can be considered to be analogous to of SGD in the ideal case.
As special case, we also establish the same convergence rate (3) for Adam under slightly relaxed parameter settings than Adam W. To the best of our knowledge, this convergence rate only appears for RMSProp firstly proved [10], and similar results for AdaGrad subsequently appeared in [2] and RMSProp in [13] under different assumptions. Notably, comparable convergence guarantees remain unproven for AdamW and Adam.
2 Convergence Rate of AdamW
This section presents our convergence rate analysis for AdamW. We first describe the assumptions used throughout this paper as follows, where denote Fx rg2,," ........ .g*) be the sigma field of the stochastic gradients up to k, denote Ex.H- [ as the expectation with respect to Fx and E, F+ the conditional expectation with respect to g* conditioned on FR
Algorithm 1 AdamW
Assumptions:
1. Smoothness:
Initialize x!, m" = 0. v = 0 for k = 1,2, K do
[Vf(y) f((x)) Dlly x/|.
2. Unbiased estimator:
g* GradOracle(x
Ex (g* [Fx=11] (x**),
m 0m k-1 + (1 0)g*
3. Coordinate-wiss bounded noise variance:
v = Bvk-1 + (1 3) (g 02
Ex [| - ( oof
b+1 (1 Xn)x Vi++c C m
end for
Denoting a [01, ".. Ga) as the noise variance vector and ||ا||| v Ed of we have the

---

following standard bounded noise variance assumption
Ex ||lg*
[[(x*)| S
Algorithm [T provides the complete AdamW implementation, where we denote @ for the Hadamard product. Setting the weight decay parameter À 0 recovers the standard Adam. For analytical simplicity, we omit the bias correction term in our analysis.
Based on Assumptions 1-3, we provide the convergence rate of AdamW W in the following theorem. Note that we do not assume the boundedness of the gradient fxx*x) or stochastic gradient g*.
Theorem Suppose that Assumptions 1-3 hold. Define 03 max { (oz [(f(x')--))
KT ] with any f(x1)-1'
constant
E (0, 1). Ler 1 0 [(f(x')-1)
Kož
f*)
0
SSB
V
JRdL
8 d. x
Va V/2K3/4
- and 1|x S V -(((x))-(-)
Then for AdamW, we have ||xt < 3t2 for
db
all k 1,2, K and
1 [IV
8vd
d
(f2 301
K
Ki/I
Specially, when o² --1)
we have 0 0 7,
S VO, 11 V
-(x))-f-
KT
4Kat
)-1 X 1 L(S(x')
1-1") || S
-(
||x* 10:0 < to and accordingly
dkg"
[2K
dI.
1 K
E [IV]((() Σ
V
Ky
Theorem demonstrates that AdamW minimizes the gradient norm directly while restricting As a comparison, f regularized Adam only minimizes [[V + Xx||. rather
||x*
than [x)]]
As special case, we also establish the same convergence rate for Adam in the following corollary under slightly relaxed parameter settings. The complete description of Corollary I is given in Appendix B
Corollary With the same assumptions and parameter settings of π7, and as Theorem [ but
only requiring S 3 S rather than both < 3 < v0 and ||x 1l S v
K [K(f(x)) f*) we have the
dl
same convergence rate for Adam as established in Theorem 7
2.1 Optimality of Our Convergence Rate
When comparing convergence rate (3) with the optimal rate (4) of SGD, which aligns with the lower bound nonconvex stochastic optimization, we observe that our rate is also optimal with respect K, a. L, and fxx') f*. The only remaining uncertainty concerns the tightness of the dimension d. Theoretically, |/ fx)]ll < =[V[(x))l S fx)]l2 holds for any high- dimensional vector x, and when each element of V fx()) is drawn from Gaussian distribution (0, 1),
we have E V 2
2d
[[)) from Lemma Empirically, experiments real deep
neural networks training confirm [xx)]] @(vā)
(xx) |2, demonstrated in Figure 2 Thus,
our convergence rate can be regarded to be analogous to SGD's optimal rate (4.
Lemma When each entry of Rd is generated from Gaussian distribution with zero mean and unit variance, we have E [||
2d E [|]]]]]
We gratefully thank the anonymous NeurIPS reviewer to derive this looser bound. Our original bound is
0 << 8 (149)²
3

---

ResNet50 CIFAR100
ResNet50 ImageNet
GPT2 Open WebText
Average Training
20%
Epoch
Epoch
Step
Figure 1:
Illustration of average training loss f(x*) over epochs/steps, and at the initialization,
f(x²) 8.
1000
Grudient Norm Ratio
1000
4000
800
800
600
400
mm
200
20
40
60
80
1.00
40
60
1.0k
20k
50k
40k
50k
Epoch
Epoch
Step
Figure 2: Illustration of [VV e [(V2) over epochs/steps. The gradient norm
/> and vd 4868, 5060, and 11136, respectively.
ratio shows
12
10
Model Eo Norm
2.50
2.00
1.75
40
60
100
20
40
60
10k
20k
30k
40%
50k
Epoch
Epoch
Step
coc A 0.01, 0.1, and 0.05, respectively.
Figure 3: Illustration ||x*
1/2 over epochs/steps. The model foo norm shows ||x** |loc and
x10-7
x10
x10
Magnitude ofdd
1.5
2
1.0
whh
/ WM
Epoch
Epoch
Step
Figure 4: Ilustration of small & over epochs/steps. The magnitude o3 is approximated by l|g
V fx without taking expectation, and d = 2.37 x 10?, 2.56 x 10?, and 1.24 x 10a, respectvell..

---

Recently, Jiang et al.
I established a fundamental lower bound for SGD when measuring gradients
by 4 norm, which is order (V (VLL(rix
1- |- K
under Assumptions 1-3.
K
When ||]|| R val| da this lower bound precisely aligns with our convergence rate in (3). We further conjecture that this lower bound applies more broadly to general first-order stochastic optimization algorithms under & norm gradient measurement. This would imply that our derived convergence rate is nearly tight.
2.2 Separating the Convergence Rate by the Noise Variance
In Theorem
! we separate the convergence rate by the magnitude of o, When o²
[(f(x*)- Ry
both the convergence rates of AdamW and Adam are O( vd
(T74).
When of becomes smaller than
Mf((")-F7 --), the convergence rates improve to O(y R), matching that of gradient descent measured Ka
by 2 norm.
2.3 Reasonable Weight Decay Parameter and Initialization Interval
13 In Theorem 1 we set the weight decay parameter À smaller than va -- In modern VT2K3/4 V
deep neural networks, the dimension d is typically extremely large, for example, 1.75 10 in GPT-3, making K374 almost certainly exceed 0.01, which the default setting of in PyTorch official CIFAR-100
vd
implementation. For example, in the experiments of our paper, we train ResNet-50 and ii) ImageNet dataset, and GPT-2 on iii) WebTex,,, and observe (K, = (39100, 2.37 x 10"), (28080, 2.56 107), and (50000, 1.24 x 10*), resulting in Vd R 1.75, 2.33, and 3.33, respectively. We empirically show in Appendix D that large À may cause AdamW not converge and thus a upper
bound is necessary. We also initialize ||x*|| S
[(((()))) Although K is typically smaller d
than in large language models training. it remains not too small. In practical configurations, we often initialize the network weights very small. On the other hand, although we always initialize the scale parameter BatchNorm//ayerNoom 1, we do not use weight decay for the scale parameter in practice.
2.4 Small Setting
In practice, e typically set to a very small value, for example, approximately 10-16 in PyToch implementation to prevent division by zero while maintaining the adaptive properties of AdamW and Adam. Larger E values would make AdamW and Adam behave similarly to SGD, losing its adaptive learning rate adjustment. In Theorem
8² max { a dKn
[(f(x))-f") which
set d
remains small due to extremely large d and modest o². We have empirically shown in Figure that o 10-7 10-9, and 10-10 in experiments of ResNet-50 CIFAR-100 and ImageNet and GPT-2 on Open WebText, respectively. Intuitively, should be smaller than the square stochastic gradient at each coordinate, otherwise, would dominate the magnitude of v in Vvł+c
Our setting
of e, the coordinate- wise average of gradient noise variance, approximatell resides this critical threshold. Notably, our convergence rates for both Adam W and Adam do not depend on explicitly. In comparison, existing convergence rates for Adam W and Adam in the literature either explicitly depend on e or exhibit a higher dependence on the dimension d.
2.5 Unpractical Settings of 1), 0, and
In Theorem I we set the learning rate 7) very small and the parameters 0 and 3 nearly equal to 1 to satisfy the proof requirements. This differs from standard implementations where (0, B) (0.9, 0.999) is typically used. Although investigating AdamW/Adam's property under realistic
In PoToch official implementation, appeared in different place in 2 and E 10-3, while we
5

---

configurations represents an important research direction, as it could yield valuable insights for deep learning hyperparameters tuning, the practical configurations may not guarantee the convergence. For instance, prior work [14 15 demonstrates through constructed examples that Adam with common hyperparameters
0.9, 3 (0.999, 0.997, 0.995, 0.993), and Tis = vF) fails to converge to stationary points (see 13 Figuire 2)). On the other hand, empirical evidence from recent studies [16] Figure 9] 17 Figure demonstrate that during the training of language models with practical parameter configurations, the gradient norm hardly decreases during the training run, although the objective function decreases sufficiently.
2.6 No Conflict with
For sufficiently large weight decay parameter where no critical points exist within the constrained domain of problem (T). the KKT conditions (2) serve as natural convergence metric. As À diminishes, problem [ asymptotically approaches an unconstrained optimization problem, and AdamW reduces to Adam the limit. There exists critical threshold beyond which V || also becomes viable metric for convergence. Consequently, our results do not conflict with [6].
Proof Sketch
In this section, we outline the proof sketch of Theorem 1 The detailed proofs are provided in Appendix A From the Lipschitz smoothness of fxx and the update ++1 in Algorithm we have
[f [[(x (xx++1) [[Fk]] f(x*)
(V((x*) s+1 -x--x x²) + - - +|x x" Fx-1
<Bx
(5)
vif(**,, m + Ax / v +e
Laf
2
m +
Ax
v +
=B
Fk
+
2
v + E
i=1
1=1
term (a)
term (b)
Decompose term (a) into
2
)= 2
m Ax
+
d
V Vif(x*)
m - Xx
7
Σ
17
Vv
Vv +
(6)
term (c)
term (d)
and relax term (b) as follows to absorb it within term (c)
2
Ln
m + Ax* V ns
d
m
+ Xx v /v
+
term (b)
(7)
VVF +
Vv +
1=1
Next, we consider term (d) and relax it follows
term (d) S
Vf(x*) mI|
1 +
Ax
v +e.
(8)
i=1
We see that the parameter & plays a pivotal rule in steps
7
and (8. Combing 5 51-88
we have
V V.f(x*)
2
m + Ax +E
Es [f(x [[- f(x*) SEn
Vv +e
Vvf +
i=1
d
+
v fx
m
+1
Ax
v/ + FR-1
lerm (e)

---

Considering term (c). we can use standard techniques in the analysis of momentum SGD to build
recursion (Lemma 4 as follows
Eg [|V - m [Fk-1]
Ax.-1 4-1 Vv v/ +E
1²
m
Ax
(10)
<0 30|mm= 8-1 VS(* ||
+ +11 (1- 0)
VE(1
1=1
+e
Multiplying both sides of
(10 by Ve(1-0)
adding it to (9, and letting n1² - =(1-0)
have
41,2
[s((*) m m1 m + Xx vv +
n0
Ex
1(x -
F
VE(1
VV. +
Vif(x*)
<f(x*) - -5 +
En
A 1
v + FR-1
(11)
v +
1=1
term (g)
torm (f)
+
Ax R-1
V +
VE(1 Vf(xx-11 >====
1/0
11 1
11 0)o² VE
+
The above analysis comes from the standard framework and contains nothing new. We can recur- sively eliminate certain terms in (1I) after telescoping, except for the troublesome term (g). The following outlines the key technical components of our proof address term (g) and achieve the tight convergence rate.
3.1
Bounding |Ax!'| in Term (g) by 0 (())))
The analysis for AdamW proves more challenging than for SignSGD-type methods with weight decay 18. because SignSGD maintains a fixed update size 1, whereas A's updates can be arbitrarily
large. Specifically, for AdamW and Adam, have Im. v
(1- (Lemma 2). where the latter is minimized to be by setting 0 = 8. However, when setting 0 = O(1) (for example, 0 0.9) (1-0)²g
(1-8)
and A K. we have (1-3)(1-22 = O(K). leading to unbounded updates in AdamW. This fundamental difficulty prevents direct extension of the proof framework in [10] to AdamW. We set
0 < < v in Theorem such that (1-3)(8PP < 4 (Lemma 2). Then for the update of x in
(1-8)²8
AdamW, we have
2
| 100 2 (1-
||x||| 110
2
When (1 - * decreases fast, we have (1 n/ (1|x 11:0
2) - 0 and ||x++||| Ioo is loosely bounded by 3. which is far from target A||x++1| loo
0 (1/aa) To address this issue,
control the decrease of (1 n/)** by setting parameter properly such that nd 2R5/4 and
ye
(1 -
v ( 2 E R.7 2 1 - K=74 for some and any h: < K. Equipped with proper initialization of
Vo |xx| I1oo I KI/AA. we finally have (Lemma 3
2 A
(1
VD
(2
Ve
3 VV
x 1
<
S
K1/4
K1/4X
X A
and
9rju
term (g)
<
v +e.
K1/2
Intuitively, when the initialization is far from the boundary of problem and (1 nd)* = 1, the iterates x" k+1 are guaranteed to be far from the boundary throughout the optimization process.
7

---

3.2 Absorbing Term (g) within Term (f)
To absorb term (g) within term (f), we first relax v - E to of V and
+ + as follows by the concavity
Eg
1
Vif [x*)
+ / vf + Fk-1
)x*( 2
Dy K1/2
V +
t
+
term (g)
(f)
where we define V
+ (1- 3) +
(xx)
Then, we can bound v V + E as
v.s(x*)
follows (Lemma and absorb term (h) within 2/2 ER=1 Ei=l
derived from term (f),
V +e
V
) +
E Ex | +
<K||ol1 KdvE +
> E B
(12)
term (h)
Summing over k and combing the above analysis, we have
m Ax Vv +E
Exx
[(x
VE(1 0) ||
0
m
K
V +E
K
V vxf(x*) 27
V v.f(x*)
1
Bxx
K| KdvE+2 +2
Bx,
2
/ +
K//2
V +E
k=1
1=
Kn(1 0)0 VE
+ f(x')
f
Vs(1 0) Er [ m [|
1
term (
(x*()*) 2
S 71
FJ
+ 18qudy KE term (j)
V +E
by letting
9v
and a² such that K||ol11 I KdvE. Letting
1 -2
11/2
72d
L(f(x Ko²
and 17 -(f(x))---) both term (i) and mudd Ke are of the order 4Ko2L
1 1 0
Ko²L(f(x))-F1 This accounts for why bounding |A=1 by O vi ( otherwise, term (i) would
7
slow the convergence rate established in Theorem
Intuitively, when v.f(x*) such that of. have Σk Σi V V HER
v vhv + (1- B) E 8h gk-r (vV)(('') + +o?
K| K| + KdvE, making term (i) neg-negligible in 12
3.3 Eliminating in the Final Convergence Rate
Based on the above analysis, get the following bound
K
V (x*()") 2
Ko f*)
EF
50
V +
k=1
f=1l
Using Holder's 's inequality and 12
again, we finally have
K
Σ
Ex
[|V [11a]
A=1
[.f(x+)))
1)
K
d
E
Σ
E E
V
vit VV
+E
/
8

---

K
d
[. 1)
K
)+
BF
Klloll++
+ Σ
EF
V
VV. +E
1= ==1
(o)L(1(x))
(
+ K 1 + KdvE) ))
00
E
and
1 K
EF
[IV]
K
k=
1 K
(o+L(f(x))
(ooL(f(x*) - ))
0
(K|lolli + K
term (k)
The above convergence rate is not optimal due to its explicit dependence on E, which is absent from the optimal rate (4 of SGD. By setting = z. we obtain K||o|11 s kdys and (K||o]] + KdvE)² < &f which allows to eliminate in the denominator of term (k). This yields the following final convergence rate
K Σ FF [IV]((*]]]] so K
(
[ K
Vd K1/4 V
-
Although smaller value e does not affect the convergence of Adam W and Adam, this term cannot be eliminated any more and consequently slows the convergence rate by introducing explicit dependence. On the other hand, while larger e does not impact the convergence rate, it makes AdamW closer SGD.
At last, in order to incorporate the scenario when of S -
we define o?
max { {o3, - with any constant E (0, 1] and replace of by 0% in the definitions of 8, v, 0, and n
Literature Comparisons
In this section, compare our theoretical results with representative ones in the literature.
substantial amount of literature exists regarding the convergence analysis of adaptive gradient
algorithms, such as 19 20, 21
[2
23 for AdaGrad-norm, 22
11
12
24
25
for AdaGrad, 26
27
28 10 13 for RMSProp. 29 for Adam-norm, 330 26,
27
31
33
15
35
36
37
38
39
for Adam, and 40 41
42
43 44
45 46
47
48
for other variants. We primarily compare
with the literature AdamW and Adam. For Adam, restrict our comparison to studies with the state-of-the-art convergence rates that do not require the bounded gradient assumption.
4.1 AdamW: Comparison with
To the best of our knowledge based on comprehensive literature review, appears to be the only existing paper addressing Adam convergence and convergence rate. We compare with in the following aspects. Firstly, the assumptions in 7 are stronger than ours. Denoting f(x) Ex <)], they assumed || [Vf(y;
Vf(x; O) S Llly - x|| (under which the lower bound O (E)),, rather than O(+)) ② and || S Coo, while we only assume []V v s Elyy x|| without the bounded gradient assumption. Secondly, they set the weight decay parameter
A = X(1 _ )(= which decreases exponentially, making AdamW reduce to standard Adam in the limit. Thirdly, they establish the complexity of Ommax{ 2 La (xx))xxx)) ect
to achieve 1 Σ Fx < 2 where F₁ is dynamic E regularized objective. Their complexity depends on e explicitly, which usually small practice, for example, e as 10-16 in PyTorch implementation. As comparison, our convergence rate does not depend explicitly.
21:26

---

4.2 Adam: Comparison with [10
Li al. [TO] studied RMSProp and its momentum extension, where RMSProp is a special case of Adam by letting 0 = 0 and À = 0 in Algorithm [ The convergence analysis of Adam presents substantially greater challenges than RMSProp and we cannot extend the proofs in [TO] to Adam. Alternatively, this paper uses a different proof framework to establish for Adam the same convergence rate achieved by [10 under identical assumptions. As a trade-off, one limitation of our proof is that
a²
relies larger value of parameter E, although E d' is very small in practice. Specifically, under
the parameter settings of [
R. vg À Aaxx(af गk J. and X K(J(x)-FF in 10
have
1 2 S 8r I 1 for any t f K and
v = 3h + (1 3)
3k
ft) +
K [(((
K
where v plays the role of E in Algorithm comparison, in this paper, we have
which is of the order of k or approximately ak As
62 v +c= 3) Σ 8h gil
k
2 + 3) k
+ (1- 3) Σ
d
When Vx(x') R such that lg'l R oi we have (1 - B) Σ (g of. Thus, & accounts
for nearly half of (vî + E)'s size, while in [0]. v² only makes up close to & of v total size. Other representative studies (34 35 33 have derived convergence guarantees for Adam built upon weak e-dependent analysis. However, these results all yield slower convergence rates than ours with higher dependence the dimension d.
4.3 Adam: Comparison with 37
Li al. 1 and proved K Ek f iterations. That is, K Ei lVf('>>) I (g)5/8 Kiza (f f*), where G 2
(37 studied Adam under assumption lgg |((*))|
as with probability
e² with high probability within O 0 23
max o. V /L([(x)) - l*) and appeared in different place in Vv++ē (hence we may consider & to be equal to VE). When [IV deep learning training, our convergence rate is ( 5//8 times faster than (37]. In Py PyTrch implemen-
as empirically observed in real-world
tation, the default value of is typically set to 10 10-8 To eliminate the dependence on e, Li et al. (37
requires 22 (<: E) = G² max(o?, I(f(x'( - f²)} 2 o² while we only need - which is times smaller.
Conclusion
This paper studies the popular AdamW optimizer in deep learning. We establish the convergence rate for AdamW measured by E norm. It can be considered to be analogous the optimal rate of SGD in the ideal case of [|V ((V))IV)(()) which is verified on real-world deep learning tasks.
K > E [[]]](* < O vdC
K174
An important direction for future research would be to investigate the optimal convergence rate using weak e-dependent analysis (for example, log for Adam and Adam. On the other hand, it is currently unclear whether convergence rate under a looser upper bound would be meaningful. This study is primarily concerned with theoretical analysis and it does not yield direct negative societal impacts.
upper bound on À is tight. Investigating how to prove the optimal
Acknowledgements
H. Li was supported by the NSF China (No. 62476142) and Z.. Lin was supported by the NSF China (No. 62276004). Li and Lin are the corresponding authors.
10

---

References
[1] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7)22211-159, 2011.
[2] H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. In Conference on Learning Theory (COLT), 2010.
[3] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 6.5.RMSProp: Divide the gradient by running average of its recent magnitude. In COURSERA: Neural Networks for Machine Learning, 2012.
[4] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[5] Ilya Loshchilov Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[6] Shuo Xic and Zhiyuan Li. Implicit bias of Adam W: E. norm constrained optimization. In International Conference on Machine Learning (ICML), 2024.
[7] Pan Zhou, Xingyu Xie, Zhouchen Lin, and Shuicheng Yan. Towards understanding convergence and generalization of AdamW, IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(9):6486 6493, 2024.
[8] Leon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 6012):223-311, 2018.
[9] Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Progranming, 199:165-214, 2023.
[10] Huan Li, Yiming Dong, and Zhouchen Lin. On the O( (( convergence rate of RMSProp and its momentum extension measured by & norm. Journal of Machine Learning Revearch, 26(131)::122 arXiv: 2402,00389, 2025.
[11] Ruichen Jiang, Devyani Maladkar, and Aryan Mokhtari. Convergence analysis of adaptive gradient methods under refined smoothness and noise assumptions. In Conference on Learning Theory (COLT), arXiv: 2406.0 2025.
[12] Yuxing Liu, Rui Pan, and Tong Zhang. AdaGrad under anisotropic smoothness. In International Conference on Learning Representations (ICLR), arXiv: 2406. 15244, 2025.
[13] Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits fu--geomerry of loss landscape via coordinate- wise adaptivity. In International Conference Learning Representations (ICLR), arXiv: 2410.08198, 2025.
[14] Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhiquan Luo. Adam can converge without any modification on update rules. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[15] Bohan Wang. Yushun Zhang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, Tie-Yan Liu, Zhi-Quan Luo, and Wei Chen. Provable adaptivity of Adam under non-uniform smoothness. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2024.
[16] Hoang Tran, Qinzi Zhang, and Ashok Cutkosky. Empirical tests of optimization assumptions in deep lcarning. arXiv: 2407.0182. 2024.
[17] Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them. arXiv: 2509.02200 020460 2025.
[18] Yiming Dong Huan Li. and Zhouchen Lin. Convergence rate analysis of LION. arXiv: 2411.07724, 2024.
[19] Rachel Ward, Xiaoxia Wu, and Leon Bottou. AdaGrad stepsizes: Sharp convergence over nonconvex landscapes. Journal of Machine Learning Research, 21(1):9047-900 -976, 2020.
[20] Ali Kavis, Kfrr Yehuda Levy, and Volkan Cevher. High probability bounds for a class of nonconvex algorithms with AdaGrad stepsize. In International Conference on Learning Representations (ICLR), 2022.

---

[21] Matthew Faw, Isidoros Tziotis, Constantine Caramanis, Aryan Mokhtari, Sanjay Shakkottai, and Rachel Ward. The power of adaptivity in SGD: self-tuning step sizes with unbounded gradients and afline variance. In Conference on Learning Theory (COLT), 2022.
[22] Bohan Wang. Huishuai 7hang, Zhiming Ma, and Wei Chen. Convergence of AdaGrad for non-convex objectives: simple proofs and relaxed assumptions. In Conference Learning Theory (COI.T), 2023.
[23] Amit Attia and Tomer Koren. SGD with AdaGrad stepsizes: full adaptivity with high probability to unknown parameters, unbounded gradients and aftine variance. In International Conference on Machine Learning (ICML), 2023.
[24] Zijian Liu, Ta Duy Nguyen, Thien Hang Nguyen, Alina Ene, and Huy Nguyen. High probability con- vergence of stochastic gradient methods. In International Conference Machine Learning (ICML), 2023.
[25] Yusu Hong and Junhong Lin. Revisiting convergence of AdaGrad with relaxed assumptions. In Uncertainty in Artificial Intelligence (UAI), 2024.
[26] Fangyu Zou, Li Shen, Zequn Jie. Weizhong Zhang, and Wei Liu. A sufficient condition for convergences of Adam and RMSProp. In JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2019.
[27] Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of Adam and AdaGrad. Transactions on Machine Learning Research, 2022.
[28] Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun. RMSProp converges with proper ayper-parameter In International Conference Learning Representations (ICLR), 2020.
[29] Bohan Wang. Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, and Wei Chen. On the convergence of Adam under non-uniform smoothness: Separability from SGDM and beyond. arXiv: 2403. 2024.
[30] Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In
International Conference on Learning Representations (ICLR), 2018.
[31] Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. novel convergence analysis for algorithms of the Adam family, arviv: 2112.03459, 2021.
[32] Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu. Towards practical Adam: Non-convexity, convergence theory, and mini-batch acceleration. Journal of Muchine Learning Research, 23(229): 1-47, 2022.
[33] Bohan Wang, Jingwen Fu, Huishuai Zhang. Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and the lower bound of Adam's iteration complexity. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[34] Yusu Hong and Junhong Lin. High probability convergence of Adam under unbounded gradients and affine variance noise. arXiv: 1.1002000 2023.
[35] Yusu Hong and Junhong Lin. On convergence of Adam for stochastic optimization under relaxed assump- tions. Conference on Neural Information Processing Systems (NeurlPS), 2024.
[36] Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence guarantees for RMSProp and Adam in generalized- smooth non-convex optimization with affine noise variance. Transactions Machine Learning Research. 2025.
[37] Haochuan Li, Alexander Rakhlin, and Ali Jadhahaie. Convergence of Adam under relaxed assumptions. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[38] Kwangjun Ahn and Ashok Cutkosky. Adam with model exponential moving average is effective for nonconvex optimization. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[39] Ruinan Jin, Xiao Li, Yaoliang Yu, and Baoxiang Wang. A comprehensive framework for analyzing the convergence of Adam: Bridging the gap with SGD. In International Conference on Machine Learning (ICML), 2025.
[40] Manzil Zaheer, Sashank J.Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In Conference on Neurol Information Processing Systems (NIPS), 2018
[41] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of Adam-type algorithms for non-convex optimization. In International Conference on Learning Representations (ICLR), 2019.
12

---

[42] Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In International Conference Learning Representations (ICLR), 2019.
[43] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song. James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. International Conference on Learning Representations (ICLR). 2019.
[44] Juntang Zhuang, Tommy Tang. Yifan Ding, Sekhar C' Tatiikonaa, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
[45] Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang. Yuan Cao. and Quanquan Gu. Closing generalization gap of adaptive gradient methods in training deep neural networks. In International Joint Conferences on Artificial Intelligence (ICAAI), 2021.
[46] Pedro Savarese, David McAllester, Sudarshan Babu, and Michael Maire. Domain independent dominance of adaptive methods. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
[47] Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signSGD. In Conference Neural Information Processing Systems (NeuriPS), 2022.
[48] Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9508 9520, 2024.
[49] Shohei Taniguchi, Keno Harada. Gouki Minegishi, Yuta Oshima. Seong Cheol Jeong, Go Nagahara. Tomoshi liyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. Adopt: Modified Adam can converge with any 82 with the optimal rate. In Conference on Neural Information Processing Systems (NeurlPS), 2024.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[51] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
[52] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathcesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bersstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, (1:(3)221122222 2015.
[53] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[54] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. OpenWebText corpus. Skylion007 github. wwwgggiittt.pheheeee ho/0penneebTextCorpp.. 2019.
http://
[55] Ross Wightman. Pytorch image models. https:://gthhii: http:///////iioooo//////aataaaaa hww.:///www...ggttaaa/ootoooochc 2019.
image models
[56] Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Mcgatron-LMs Training multi-billion parameter language models using model parallelism. arXiv:1909008053 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our main contribution is to establish convergence rate AdamW. clearly written in the abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are provided in Section and the proofs provided in the appendix.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
14

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental settings are provided in Appendix We also share the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: decision.
Code is provided in the supplementary material. It will be available on Github after the
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpa://nips cwtp:ccccccccccccc ic/
CodeSubmi guides/CodeSubmin asionPolicy for more details.
While we encourage release of code and data, we understand that this might not possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //nips//.../////// cc/public/ gwides//CoeeSueee muzides//oodeubbllsssioooo...... for more details.
15

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are provided in the Appendix E This work is solely concerned with training aspects, with consideration of testing performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Our figures show the convergence properties of AdamW for specific runs. People often do not plot the behaviors over multiple averaged runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yicld results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
Computational resources information are provided in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, with NeurlPS Code of Ethics
httpa//www.ppp.pppp.aaaaaa..c.aa [Yes]
Answer:
Justification:
The paper follows the NeurIPS Code Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
They are properly cited based on their licenses.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssi thcode. com/ ds has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
18

---

Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the rescarch, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http:///auurpp.../oo//o///eer//e LLM) for what should or should not be described.
19

---

A Proof of Theorem 1
Proof As the gradient is L- L-Lipschitz, we have
B2
f(x k+ | [ f(x*)
<Bg
Vx(x" x*)
b 51 1 k+1 x111
Fx-1
mf + dxt vv +e
111
mf
xx vv +
=Ra
7
vxf(x*)
F
V +E
2
v +
)x((x")
m
+ Xx
v
+
B
VvF +E
VV +e
1 4
(x*(x*)
m Ax +e
Lu
2
d
m
+ Ax
V V. 3
Fk-1
2 1=1
2
+
[=1
Vxf(x*))
xx v v.
+
<Bk
Vv +
2 Vvē +
(13)
V.f(x*) m + Xx, vv +
m + Xx /v +
Lnp
+1 Σ
Fa
VV
2 2
v +
1=1
v.f(x*)]
m + Ax
v. +
Bx
Σ
Σ
+ V8
[(((') - m²11 -mal
Vv +e
VV +
+1 Σ
|xc 1 V va + Fx
(2)
B&
)
1 Σ
Xx
V +
71 Vf((") - ml|
-
Vv +
v +
iml
1-1
9np
v v + 8 h-
K1/2
()) 2L in and use Lemma
(2)
where we ler S
J
3 in S. Denote
र = =Bv k- + (1-3) (17
From the concavity of Jo and + and Assumptions and 3, we have
Es v +
S v Ba [ [v + Ao + (1 3)BB [|g! Fu-1] +
S /B (1 B) (x+()))) 2 + +z?) + = v /V/+a. +..
Be J - [.,((*))² 2
v
Ex
Fx $
Vv +
V
20

---

Plugging into 13 and rearranging the terms, we have
+ 2
d
m
Axt
Be
- +
1
77
V[(x')
== mt|pppa
4
Vv." +E
(14)
<f(x') 8 --- 17 [x+ )
Onp
Σ
+
2 1=1
V
+
K1/2
Multiplying both sides of (18 in Lemma by ve(l-oj and adding it to
14 we have
Re 5
n0
V m
m
Ax
+
FR
VE(1
0)
4
V +
<f(x")
- 2
|x*
9mp
V. + +
n0
V k-
m
111
+e
K1/2
VE(1
12
1.21y
d
m
k-1 + xx
+
17(1 0)
+ s(1 0)3
(15)
VE
+e
)x ?
<f(x") f"
9mp
1 +
V +E
K1/2
1/0 VE(1- - 0) [
m
1|
7
m
R-1
+
Ax!
1-1
//
+e
71 0)o
/v
vee
+
where we let x² S
e01-0)² in the last inequality. For both [4
and over
15
taking expectation with rexpect to
412
Fk-1, rearranging the terms, summing 14) with k 1 and B3
K
2, 3, ... K, we have
m Xx
EFx
f(x [(xx+1)- -5-
n/0
K
K
71
VE(1-
0)
4
/Vk +
K
7 (+
Srp
<S(x') f"
E
2
- +E
K1/2
k=
+ Ve(1 - 8)
n1
EF, [I [
|1'] + Br [[V]((') - m'mm
(K 1)n(1 0)
Ve
V(((**
S fx(') )- - 8 2
E BT
V +
9np K1/2
Kl|ool1 Kdve=
Σ> Σ E
[xf((t) )1
V +e
(16)
#=1 1m1
(K-
1)m(1 0)02
Es, [[x(()
[|3] +
Ve(1 - 0)
Ve
vxf(x*) 11² v +e
fx() - 8" 4
E
+ 18mudy Ke
"
ET [[|]f(xt) m m211
(K- 1)m(1
0)o²
Va(1 0)
Ve
3
sf(x²) -
By
V +
(n((((x
)r) 111 - -0o²
(K - 1)n(1 -0)0
+ 18pudv Ke
+
VE(1 0)
V8
V8
21

---

(3)
and 2 o/ such that |||||| 2 vdllor12 Vdo, S ddve in 2
where we Lemma
in S, let xtyZ
and m 0,
I" S s x I [(() 5((x) _ (VS(x) + 1/ L | = -((x) 2L [[][(}))
and
Ex, [IV m
EA, -0 +1 [|V](x') g [1
- (1)
-
in S. So from
16). we have
V vxf(x*)) 1|²
B---1
Vi +
4(f(x) 1
(L( ) -)
4K - 0)o² -))
+ 72udV KE
(17)
7
Ve(l 0)
VE
4(((x²)) 1*)
8L([(x)) -x(()
+ 4K - 010²
72ndV KE
VE(1 - 00
VE
where we denote 6²- max (o2,
-( K
with any constant y (0. 1].
Recall that require the parameters satisfying the following relations in the above proof
V8 S 17² S
E(1 -
0)
9v
E a²
41.
K1/2
8
and
Vv
Ve
Ve
ad S
<1,
|x S
8 v0 <1
2K5/4'
K1/4
K1/4X¹
in Lemma 3
Recalling the definition of o. and letting d. 1 0
Lif(x -(f(x*)--")
=(f(x')
1(x')- akaL
Ro
4K0TT.
02L(5(01)- 27
[(f(x i-1) X
J
Va
L3
and ||x 11
2K3/""
V72K3/2
-2Kn S K(((2)) dP
the above requirements are satisfied. So we have from Ki/AX'
17 that
K
d
) 27
Ko]I(f(x))
ΣΣ
Br--1
2211
+E
Using Holder's 's inequality and Lemma 5
we have
K
| [1V/(x()]1]]
Exa-1
K d k=1
Exz-1
Exa-1
( v +E
V +e
V+f(x*)
Exz-1 --
K||a||| KdvE+2 2
Br
V +e
Vv +
(o)L(((x))
(x*)( - )r)
21
42
+ KKlool1 + Kdvé
8
and
K
1
| )11
Er
K
1
30
KoOL(f(x)) - ft)
+ +
KoOL(f(x!) -
(Kl|oT) + KdvE)²
K
8
8
dx([(xt) -t))
8Vd Vorl(f(x')
<30
R3/4
22

---

by letting 4 and using ||0112 vdllo112 Vdo, dy.. At last. from Lemma 3 and the settings of and os have
44
3/0 3 v/ [(((x)) -5)
Al|x*11 10 S
K2/4
V72
KOž
v/2
for all k = 1,2,- K, leading to ||x*||0c < 3.
B Proof of Corollary 1
We give the complete description of Corollary in the following corollary.
Corollary 2 Suppuse that Assumptions 1-3 hold. Define 0 max
oo. M((x"))-1 Kg
with constant
7 E (0, 1]. Let 1 - 0 V
1-1( KR
0 1331
1-x- and AdRL
62.
Then for Adam, we have
#.
K
6vd
- + dI.(f(x²) -M) r*)
Σ
-
K
K1
K
Specially, when of accordlingly
-
have 1 0 0- 3... 11
[(x))-1"
-
and
Kx
4KaL
dR
Σ [IV((()))) s 2
dI((fxx)) f³)
K
Ky
Proof 2 When À = 0, the
Squ K1/2
D vv - term disappears in
13)
in the proof of Theorem
and 16
becomes
EFx 1(x [(x -
77
V(((**)
m
2
+
m K
+
Ax
v
+
VE(1= 0)
4
+
7 <f(x') - 8r
K
V.f(x*) 13
2n f³)
Ve(1 -0 + Kn((1 - 0)0?
B
2
+E
Ve
where
the term 18nudy Ke disappears
because we do not need Lemma
5
bound
9nv K Similar to the proof of Theorem 7 we have
Ei EL Br.-, VV/+e any more.
(T(f(x)) -f) + 2K(1 - 0)o? 3(f(x')-F) + VE(1- 0) VV
Ex
Vvite
-
$10
Compuring with 17 we see that the term 72ndy Ke disappears. Following the proof of Theorem we have the conclusion. Note that do not use Lemmas 3and in the proof of Corollary so Corollary does not
require 0 < 8 < v0 and ||x 1lo0 S R1y4x any more.
C Supporting Lemmas
Lemma Suppose m = v = O, and 0 < S v0 < 1, then we have
m (1 (1-0
4.
(1 () 1 82)
Proof3 From the recursions of m and vtt we have
m = =m + (1 -1- 0) Σ
gi = (1 (1- (1- Σ gh gk-rgh
gi
V = vi + (1 3)
(1 - 8
1=1
23

---

Using Holder's inequality, we have
k
k 82
m
12 0)-
0
gi
5 (1-
Σ
gg-""1s 1s 13
n=1
(1 - 8) 2
82)
k-r
1 (1-0)²
(1) (1 0]=
1- 3
3
S.
1-B 1 1- 6r
S vi (1 3)3
8
(2)
(0) v0)
Svi(1 + v0)² S 4v
(1 - v02²
(1)
where we use S B in S and 3 S v0 in S.
Lemma 3 Suppose dn S
2k3/4. ||x' 1100 S
vv
x1/ax' k1/4 < 1, and 0 5 vo < 1, then we have
Vv ve
3/0
1|x* 10 S
4k= 1, ... K.
K3/4
Proof 4 From the update of we have
||x ++1
20
2 =
(1 - n))x
7
O
m
2
Vv +E
X
5(1 m/A)]|= 11:00
1
2 X
Vvv
+E
mI
(1)
2(1- [A)]|X
11000 + 2111
X
-(1- 171)
(1x* 11x 11:00 X
5(1 - nA)a
[|x ||x||1 11000 - 2
X
s (1 72)* 2 vv
K2/4
()) in S. Since lux S x - 1 and e² 2 x + for any x > 0 and 1/ S
where we use Lemma have for any S K that
Vv
S 2. we
2K574
1 121 2 - 111 -11
Knd
Vi
kln(1 m7a k In
1 npd
K1/4
Vv Ki/4
(1 12)* 2 e x114 21
and
2
vv
v/v
2 3 Vv
||x
11000
2
K1/4
K1/4
Lemma 4 Suppose that Assumptions 1-3 hold. Then we have
Es Imn
12n12 <0 k- - VE(1- 0)
m
Xx
V +e
(18)
+ (1 +11-0)002.
+
Proof 5 Denoting 5* gz - fx(xt) ), from the update of m²" we have
m" * Vs(x*) =0m k-1 + (1 - 0)8²* - V[(x")
=1 m h-1 - v + (1 00) (Vf((()*)+++))))))))())))))))() - --((+)+
+ +
= m h-1 v)(x(11) + (1 -0yc* 0 - -VF(*+1)
24

---

and
Es
|m ((((x))11
(m' - -V )) (VI(x)) vs(x* ))(x +1--))03
(1+ 1 - | ,/( + 1+ T-0) 0 / - -/(tt111²) + + 0)203
0
=0²
k- | + 1-0 |
I?²
x 1| + (1 - -0)²0²
<
m
2
m
+ Xx; -1 v +
d
-0
m
+ 1-0 0
+ (1 15-07 0)?03
+
Ax k-1 /v +
k-1
<0
m
v
+
1
m
+ (1 0)
VE(1
The following lemma is moditied from [10] We give the proof here only for the sake of completeness.
Lemma Suppose that Assumptions 1-3 hold. Let 1 and v = 0. Then we have
ΣΣ BFa-1 [ + kkllol1 KdVE++
V v+f(x') 2
Ex--1
VV +
k=1
Proof 6 From the definition of V. we have
E-
-E
V Av (11 (V.5((())))))))+++ +03)) +.
gv -
+ (1- 3)07 +E
(1- 3) [xf(x*)
=F-r=-1
Ov + 3) (IV: +o?) +e
Vort- + (1 8) (IV f(++)()))' 112 + of) te
|.
/ovt + -1- B)o7 + + (1 3)
VV +
Consider the first part in the general case. From the recursion vf have
Ex.- V B h-t + {1 +
/Btt k-11 + B0 (3(113)) 11 3t +e
Er.
= E.,
Ep //tta//--1 3²(1 B) (1 30)0 + --1
Er
git 11(1 3)Ea_ [| 12 + (1 - 30)o? +
(2)
E EFa-e-1
11
(1 - 30
Ex
11(1 A-r-1
8) / 1)2 + BL-1) +
g 1-1 g )o +
=Es,
g1t + (1 (e V 212 + (1 - gi+ oo tej
3 B) f D1
+ E.
/ + 01 f( (1- 3(+1)
25

---

<BFA-e- [VBrtiv a + (1 10 +
(01 (3) 1|
+ BFA-e- (ooting- (1( 3)1 v. /x 112 + (8' 10+ + AtE
[ /Bttl (1 B( + BB B)BFI.--1
V. v.f(x* 11
-Era-o-
(1)
where use the concavity of v in and Assumptions 2 and 3 in S. Applying the above inequality recursively for 1,2, k 1, we have
Exa Bvv + (1 3)o7 +e
sv 3/v + (1- -3 +
(1- B)
+e
and
k B40 - Σ
Vif(x*)
E
1 85- V7 - Sv
Sv g/v +
gr- (1-8)
1=1
s ov + Σ
8h (k-1((1- - -B
/ +
)x1(x')1
Sa. + VE+ +
V 31 (1 - B)BF.-
V +
where we use vi 0. Summing over i 1,2, d and k: 1,,, K, we have
K
)xf('')
v +
SK||ol11 + KdvE+ +
//--1(1 (1 3) B
/V.+
kml t=1
{x
=K[|o]1 + KdvE+
+
3K (1 118) 13)
E
+
(=1
1
K
Vif((*))
SK||o|| Kdvē+ +
BR
1
VV
1=1 i=1
K
V)f(x')²
=K|o + Kdve (1
B
ΣΣ
EF
VV!
+e
Lemma When each entry xf E Rd is generated from Gaussian distribution with zero mean and unit variance, we have E [[x[[[]] 2 / / [[x[]a].
Proof 7 When x. ~ (O, we have
2
[[
E [x 1,
R. [|xx]] =dy 2
E [[|]|]]]
[[|x]]]
E [x]] d,
E =
[||]||]]
|
E = Vd.
E
|X|||
'2d
X
2
(1)
where use the concavity of /2 in
S.
26

---

10
Averaged gradient nomn
6
Magnitude iterates
10-2
A=10'
4
A=10
A=10
A=10
10-3
A=10
A=10
A-10
A=10
A=10
A A-10
2=0
A=
10
10
10
10
10
Step
Step
Figure 5: Illustrations of Ei [f(2') (left) and x² (right) steps the toy example.
D A Toy Example with Large X
Consider following function:
f(x) = (x 1==x"> x²)²
with the stochastic gradient g(x) - -1,
with probability p 0.1,
200
x²
)
with probability 1 p.
We set K = 10th = 3
V0. - JR. 10 10 m
= 0, 00 = =0, and = =x + for
AdamW, where xx = 5 is the minimum solution of f(x). We test À {10 1000",100 10-3, 1,000, 1000 0}
such that x² < and thus the KKT conditions 2 reduce to [V[(x") = 0 the minimum solution. So can use the gradient norm [x) to measure the convergence. From Figure we see Adam fails to 10 101"). indicating that large values of X exceeding a certain threshold
converge to x² when À = (10-'. may AdamW neither converge the minimum solution nor KKT point satisfying [ In practical implementations, excessively large values X are typically avoided, they may drive the parameters toward zero and away from the minimum solution.
10 '
E
Experimental Details
In the main paper, conduct several representative deep learning experiments to empirically support our claims, covering classic image classification and language processing tasks. For the vision tasks, we independently train
ResNet50 (50) on CIFAR100 [51) and ImageNet [52) datasets; For the language task, adopt the GPT-2 (53)
architecture and pretrain on the WpenWebText (54 dataset. Code is released at https://gihub..om/aabbo -indddniisdyd// Convergence-Rane Rate- Adam W
Our experiments involve the computation of the full training loss f(x*) as well as the full gradient V fx*".. However. in typical stochastic training paradigm, one often updates the parameter x" on-the-fly fly immediately after obtaining the stochastic gradient g from the backward pass. To get an accurate measurement and avoid interfering with the normal training process, propose to split each epoch into two separate phases: training phase and logging phose. In the training phase, traverse the dataset once with stochastic updates, where the model parameters second traversal over the training dataset while keeping the model parameters frozen. Since the loss function typically defined to be the average over all training samples and the gradient computation is inherently linear, we accumulate the losses and stochastic gradients across mini-batches during this phase. This yields the exact values of the full training loss f(x") (xx)) and full gradient V Vx(x*) the current iteration.
updated upon processing each mini-batch.
the logging phase, we conduct
In the following, detail each experimental setup individually:
ResNet50- CIFAR100: CIFAR is simple benchmark dataset that widely used for quick and efficient evaluation of deep learning tasks. contains training split 50000 examples and split of 10000 examples, although we do not perform evaluation on the test set in this work. Following the official implementation, we use the torch. optim Adamw API to configure the optimizer. We initialize the learning rate to 3 x 1 train the ResNet50 model for 100 epochs, and apply a cosine learning rate decay schedule during the whole training process. Setting the batch size 128, each epnch consists of 550000/128] + 391 steps, where the additional step accounts for the final truncated hatch which contains the remaining samples. The total number of
This does conflict with 6 because 6 only considered deterministic AdamW.
27

---

steps is K = 391 x 100 39100. Without loss of generality, compute the noise vector o² = Vx((x) using the stochastic gradient gx ohtained from the first batch at the logging phase. We leave the weight decay as its default value 0.01, and complete the training task with single NVIDIA A100 GPU.
ii) ResNer50- ImageNer: To evaluate the scalability of our conclusions on larger-scalc dataset, we conduct experiments the ImageNet datasct using the same ResNet50 architecture. ImageNet consists of approximately 1.28 million training images and 50,000 validation images actoss 1,000 classes, which also come with official dataset split. employ the training script from Py Torch Image Models (timm) [55], making only the necessary modifications to suit our experimental setup. We adopt the same optimizer configuration as previously, but compute the noise vector using the last batch at the logging phase, the timm script discards incomplete batch and ensures uniform batch sizes. We follow the standard ImageNet training protocol for ResNet-50, which consists of epochs as commonly adopted the literature and official implementations [50] (55). The first 10 epochs are used for learning rate linear warmup from to 3 x 10 10"3, followed by cosine decay over the remaining 80 epochs. We apply standard data augmentation techniques including RandAugment, Mixup (0.1), and CutMix (1.0). Setting the batchsize to 4096, each epoch consists of 3 312 minibatches and the total number of steps is K 28080. We set À 0.1 and complete the training task using NVIDIA A100 GPUs.
iii) GPT2 WeenWeeTex:: To assess the generality of conclusions across different modalities, we further evaluate on a language modeling task using GPT-2. We pretrain this model on the OpenWeb WebText dataset under the NVIDIA Megatron-LM, LM codebase [56]. which is a widely adopted framework for large-scale language model training. Unlike the previous settings, where computing the full training loss and gradient over the entire datasct is tractable, the OpenWebText dataset is substantially larger, containing approximately 9 billion tokens. Consequently, an entire pass through the dataset to get the full training loss fxx") and gradient f computationally infeasible. Instead, we approximate these quantities by accumulating their values over 100 consecutive mini-batches at the logging phase. We follow the Megatron-L.M official GPT-2 training configuration with minimal modifications to suit our experimental needs. We train GPT-2 Small model with approximately 125M parameters. The model is optimized using the fused implementation of AdamW from NVIDIA Apex package, which the default setting Megatron-LMV We set the learning rate to x 0.05. Following the de facto standard in large-scale language model training. we use (0, B) = (0.9, 0.95) (that is, the commonly used (81, 32)) instead of the conventional (0.9, 0.999) setting. The total training process runs for 50,000 iterations, where the learning rate is linearly warmed up for the first 2,000 iterations and then decayed following a cosine schedule. We set the global hatch size to 640 and train the model for K 50000 steps, and complete training task using 8 NVIDIA A100 GPUs.
and weight decay to
28


Reviews:
[{"review_id": "fHdiLdxURL", "reviewer": "Reviewer_FJAz", "summary": "The authors proved a novel convergence rate of AdamW in terms of the minimal expected $L^1$-norm of the gradient, where the theoretical requirement for the hyperparameters almost fit the realities.", "questions": "1. Under a different set of conditions, [1] successfully derived a convergence rate of Adam, while $\\beta_2  = 1-\\tilde O(K^{-1})$ and is similar to yours. They claim that the proof can adapt to non-zero $\\beta_1$  but hurts by having a factor of $\\mathrm{poly}(1/(1-\\beta_1))$. While in your corollary on the convergence rate of Adam, you have no requirement on $\\beta_2$  and $\\beta_1$  is close to $1$. Can you briefly compare how you improved this result?  \n2. Does it directly implies that the sequence converge to the KKT point? I am wondering if the infty norm of the trajectory is well controlled so that the gradient $L^1$ norm convergence implies the convergence to KKT points. \n\n[1] Xie, S., Mohamadi, M. A., & Li, Z. (2024). Adam Exploits $\\ell_\\infty $-geometry of Loss Landscape via Coordinate-wise Adaptivity. arXiv preprint arXiv:2410.08198.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Pros**\n\n1. While being the most preferable optimizer in the era of large language models, AdamW is not well understood in terms of its theoretical properties like convergence and implicit biases. This work is the first one that proved the convergence rate of Adam with weight decay. \n2. The content is well-organized and the result and its consequences are clearly stated. A \n\n**Cons** \n\n1. Some conditions on the hyperparameters are not realistic, indicating that the bound might not be sufficiently explaining  real-world optimization trajectories. \n2. The theoretical results seem to provide little practical implications on tuning the hyperparameters and understandings on the dynamics, trejactories or implicit bias of AdamW. See questions. \n3. The authors claim that the result is optimal in the sense that the corresponding 2-norm bound matchs the lower bound of SGD. However, the translation between 1-norm and 2-norm are semi-empirical.", "was_revised": true, "final_justification": "The authors have properly clarified the limitation and the strength of analysis by showing some related papers, and these limitaitons are    rather inherited from the whole line of the analysis. The main contribution in the analysis of the weight decay version of adam is still significant. Therefore, I raise my score based on careful consideration.", "rebuttal": "We sincerely thank a lot for the reviewer's effort and valuable suggestions on our manuscript. Below are our responses to the major concern.\n\n$\\mathbf{Response\\enspace to\\enspace Weaknesses\\enspace (Cons)}$:\n\n$\\mathbf{Cons\\enspace 1\\enspace and\\enspace 2}$. \n\nWe agree with the reviewer's insightful comment regarding the theory-practice gap in hyperparameter configuration. Indeed, modern LLMs are not trained to full convergence, and practical configurations often deviate from theoretical settings. For instance, prior work [1,2] demonstrates through constructed examples that Adam with common hyperparameters ($\\beta_1=0.9$, $\\beta_2={0.999, 0.997, 0.995, 0.993\\}$, and step-size $\\eta_k=0.1/\\sqrt{k}$) fail to converge to stationary points (see [2, Figure 2]). Similarly, while nonconvex SGD analysis typically assumes step-sizes decaying to zero ($\\eta_k=O\\left(1/\\sqrt{k}\\right)$), practical LLMs training often employs cosine decay scheduler that plateau at small positive values.\n\nWe fully agree that investigating Adam's property under realistic configurations represents an important research direction, as it could yield valuable insights for LLMs hyperparameter tuning and training dynamics interpretation. However, the practical configurations may not guarantee the convergence. For example, empirical evidence from recent study [3, Figure 9] demonstrates that during the successful training of BERT and GPT-2, the $\\ell_2$ gradient norm typically remains within the range [0,10] and hardly decreases throughout the optimization process, even though the objective function decreases sufficiently. So we maintain that our current work, which aligns with standard convergence proof methodologies in the traditional optimization field, satisfies NeurIPS' rigorous theoretical standards.\n\n[1]. Y. Zhang, et al. Adam can converge without any modification on update rules. NeurIPS 2024.\n\n[2]. B. Wang, et al. Provable adaptivity of Adam under non-uniform smoothness. KDD 2024.\n\n[3]. H. Tran, et al. Empirical tests of optimization assumptions in deep learning. arXiv 2407.01825, 2024.\n\n$\\mathbf{Cons\\enspace 3}$. \n\nThe recent work [4,Theorem 4.1] established a fundamental lower bound of order $ \\Omega\\left( \\left( \\frac{ dL ( f(x^1)-f^* ) ||\\sigma||_1^2 }{T} \\right)^{1/4} \\right) $ for SGD when measuring gradients by $\\ell_1$-norm. Notably, when $||\\sigma||_1 \\approx \\sqrt{d} ||\\sigma||_2 = \\sqrt{d} \\sigma_s$ (as typically expected), this lower bound precisely aligns with our convergence rate in Eq. (3). We further conjecture that this lower bound applies more broadly to general first-order stochastic optimization algorithms under $\\ell_1$-norm gradient measurement. This would imply that our derived convergence rate is nearly tight.\n\n[4]. R. Jiang, et al. Convergence analysis of adaptive gradient methods under refined smoothness and noise assumptions. COLT 2025.\n\n$\\mathbf{Response\\enspace to\\enspace Questions}$:\n\n$\\mathbf{Question\\enspace 1}$. \n\nTo the best of our knowledge, existing convergence analyses for adaptive gradient methods can be broadly categorized into two frameworks. The first framework (e.g., [5-7] for Adam and [8] for RMSProp) requires $\\beta_2=1-O(1/T)$ but imposes no constraints on $\\beta_1$. While this approach achieves weak $\\varepsilon$-dependence (typically $\\log\\frac{1}{\\varepsilon}$), it often results in complex proofs with stronger dependence on terms $L$, $\\sigma_s$, $f(x^1)-f^*$, and especially the dimension $d$ (only for Adam, excluding the simpler AdaGrad and RMSProp). The second framework (e.g., [9]) instead requires $\\beta_1=1-O(1/\\sqrt{T})$ without restricting $\\beta_2$, which generally yields simpler proofs and tighter convergence rates but introduces explicit $\\varepsilon$-dependence (typically $1/\\varepsilon^a$ for some $a$). The second proof framework attains these advantages because the proof is highly dependent on $\\varepsilon$.\n\nOur work adopts the second framework while innovatively addressing its explicit $\\varepsilon$-dependence in the convergence rate through: (i) careful bounding of the second moment estimate via Eq. (12) (line 153), and (ii) strategic setting of $\\varepsilon=\\frac{\\sigma_s^2}{d}$ (such that $K ||\\sigma||_1 \\leq Kd \\sqrt{\\varepsilon}$) to eliminate the explicit dependence on $\\varepsilon$ in term (k) (line 163). This approach effectively achieves the tight convergence rate without explicit dependence on $\\varepsilon$.\n\n[5]. A. Defossez, et al. A simple convergence proof of Adam and AdaGrad. TMLR 2022.\n\n[6]. Y. Hong, et al. On convergence of Adam for stochastic optimization under relaxed assumptions. NeurIPS 2024.\n\n[7]. B. Wang, et al. Closing the gap between the upper bound and the lower bound of Adam’s iteration complexity. NeurIPS 2023.\n\n[8]. S. Xie, et al. Adam exploits $\\ell_{\\infty}$-geometry of loss landscape via coordinate-wise adaptivity. ICLR 2025.\n\n[9]. H. Li, et al. Convergence of Adam under relaxed assumptions. NeurIPS 2023.\n\n$\\mathbf{Question\\enspace 2}$. \n\nThe reviewer is correct. From the KKT conditions in Eq. (2), we can characterize the convergence by  \n\n$||x||_{\\infty}\\leq \\frac{1}{\\lambda}$ and $ < \\lambda x , \\nabla f(x) > + || \\nabla f(x) ||_1 \\leq \\epsilon$. \n\nWhen the trajectory's infinity norm is bounded as $||x||_{\\infty}\\leq \\frac{c}{\\lambda}$ for some $c<1$, we derive the inequality:\n\n$< \\lambda x , \\nabla f(x) > + || \\nabla f(x) ||_1 \\geq (1-c) || \\nabla f(x) ||_1$\n\nThis inequality establishes that convergence in the gradient's $\\ell_1$-norm directly implies convergence to KKT points."}, {"review_id": "g3sPCTeHPN", "reviewer": "Reviewer_pXdm", "summary": "This paper provides a comprehensive convergence analysis for AdamW under non-convex smooth optimization. The convergence rate is optimal with respect to the iteration number, noise level, the initial function value gap, and the smooth parameter.", "questions": "I do not have any further questions.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe major contribution of this paper is mainly in the theoretical part.\n\na. The convergence result for AdamW in this paper looks novel and sound. It seems that this is the first theoretical convergence result for AdamW with an explicit form of convergence rate. I check the parameter setups in detail, and most of the setups are reasonable. Although there are some gaps between the theoretical setups and the practical ones, it's acceptable for a theoretical paper.\n\nb. The proof sketch is clearly written. There are some novel techniques to handle the additional decay parameter brought by AdamW. \n\n**Weaknesses:**\n\na. For the detailed parameter setups, I have some concerns. First, as the authors also claim in Section 4.1, the setup for $\\lambda$ can exceed the default $0.01$. Specifically, if $d$ is sufficiently large, e.g., the dimension of GPT-3.5, the requirement for $\\lambda$ is not very satisfactory. I also find that $\\lambda$ for experiments in this paper still applies $0.01$ or $0.1$. It's better to provide some experiments to verify that a large $\\lambda$ is also acceptable.\n\nb. The dependency of $\\epsilon$ in the convergence rate seems to be $O(1/\\epsilon)$. The authors then use a sufficiently small $1-\\theta$ and $\\eta$, namely $O(\\epsilon)$ order to eliminate the factor $O(1/\\epsilon)$. This may not be so satisfactory. There are some convergence results for Adam with a $O(\\log(1/\\epsilon))$ dependency. Is it possible to achieve the same dependency for AdamW? Also, the setup for $\\epsilon$ is determined by $d$ instead of any constant like the setup in previous literature.\n\nc. I think that a key difference of AdamW to Adam is to hand to the term $\\lambda \\\\|x_i\\\\|_{\\infty}$. The key solution is Lemma 3. So I advise the author to provide the detailed version of Lemma 3 in the main body of the paper.", "was_revised": true, "final_justification": "Some of my concerns have been addressed by the additional experiments. However, the somewhat unsatisfactory setup of $\\lambda$ and the polynomial dependency of $\\epsilon$ have not been well addressed. I maintain a positive review.", "rebuttal": "We sincerely thank a lot for the reviewer's effort and valuable suggestions on our manuscript. Below are our responses to the major concern.\n\n$\\mathbf{Response\\enspace to\\enspace Weaknesses}$:\n\n$\\mathbf{Weakness\\enspace 1}$.\n\nIn Theorem 1, our convergence rate holds for all $\\lambda$ satisfying $\\lambda\\leq \\frac{ \\sqrt{d} }{ \\sqrt{72} K^{3/4} } \\sqrt[4]{ \\frac{L^3}{\\hat\\sigma_s^2 (f(x^1)-f^*)} }$. As the dimension $d$ increases, it is reasonable to expect that the upper bound on the right-hand side grows accordingly, making it more likely to exceed 0.01 or 0.1, showing that our theory covers commonly used practical configurations. \n\nFollowing the reviewer's valuable suggestion, we have conducted additional experiments to evaluate the training behavior across different $\\lambda$ values, especially large values. We perform the analysis on both ResNet-50 (CIFAR100) and GPT-2 (OpenWebText), study the training behavior, and estimate the value of the derived upper bound of $\\lambda$ to assess its compatibility with practical choices. For each experiment, we vary $\\lambda$ values as ${0.01, 0.05, 0.1, 0.5}$ and report the average training loss at the beginning, end, and several intermediate epochs/steps. To provide an accurate yet computationally feasible estimation of the upper bound, we set $\\hat{\\sigma}_s^2 = \\sigma_s^2$ and approximate the noise variance as \n\n$\\sigma_s^2\\approx\\max_{k=1}^K ||g^k - \\nabla f(x^k)||^2$. \n\nSimilarly, the Lipschitz smooth constant is approximated as \n\n$L\\approx\\max_{k=1}^K \\frac{||\\nabla f(x^{k+1}) - \\nabla f(x^k)||}{||x^{k+1} - x^k||}$. \n\nAdditionally, we set $f^\\star = 0$ to further ensure a conservative estimate. To obtain the best estimation of $\\sigma_s^2$ and $L$, we compute the maximum over all $\\lambda$ settings.  Given the knowledge of $d$, $K$, and $f(x^1)$, the required $\\lambda$ bound can be computed. Specifically, for ResNet-50 (CIFAR100), we have the estimated upper bound of $0.5729$. For GPT-2 (OpenWebText), we estimate as $0.3464$. Below are the detailed numerical results.\n\n- ResNet-50 (CIFAR100)\n| Epoch         | 1 | 20 | 40 | 60 | 80 | 100 |\n|--------------|----|-----|-----|-----|-----|------|\n| $\\lambda$=0.01 | 3.92237 | 0.55829 | 0.09655 | 0.02322 | 0.00325 | 0.00103 |\n| $\\lambda$=0.05 | 3.95005 | 0.63125 | 0.22489 | 0.09712 | 0.00893 | 0.00120 |\n| $\\lambda$=0.1  | 3.90191 | 0.93962 | 0.53329 | 0.21229 | 0.02505 | 0.00218 |\n| $\\lambda$=0.5  | 3.87442 | 1.98060 | 1.78694 | 1.46280 | 0.76590 | 0.17032 |\n\n- GPT-2 (OpenWebText)\n| Step         | 1 | 10000 | 20000 | 30000 | 40000 | 50000 |\n|--------------|----|--------|--------|--------|--------|--------|\n| $\\lambda$=0.01 | 7.05598 | 3.08517 | 3.02195 | 2.99443 | 2.98804 | 2.99229 |\n| $\\lambda$=0.05 | 7.05515 | 3.09249 | 3.02159 | 2.97827 | 2.95811 | 2.96719 |\n| $\\lambda$=0.1  | 7.06873 | 3.11096 | 3.03222 | 2.97760 | 2.94311 | 2.95332 |\n| $\\lambda$=0.5  | 7.05931 | 3.27434 | 3.16656 | 3.06161 | 2.96586 | 2.96249 |\n\nThese results indicate that AdamW exhibits stable convergence behavior even under relatively large $\\lambda$ values, though it converges more slowly when $\\lambda$ approaches the upper bound for ResNet. \n\n$\\mathbf{Weakness\\enspace 2}$. \n\n(1) Use sufficiently small $1-\\theta$ and $\\eta$ to eliminate $ O(1/\\varepsilon)$.\n\nFrom term (k) on line 163, our convergence rate depends on $\\frac{1}{K} \\sqrt[4]{ \\frac{K}{\\varepsilon} ( K ||\\sigma||_1 + Kd \\sqrt{\\varepsilon} )^2 }$. When $d \\sqrt{\\varepsilon} \\geq ||\\sigma||_1$ (satisfied by $\\varepsilon \\geq \\frac{\\sigma_s^2}{d}$), the convergence rate becomes $\\frac{1}{K} \\sqrt[4]{ \\frac{K}{\\varepsilon} 4K^2d^2 \\varepsilon }=O\\left(\\frac{\\sqrt{d}}{K^{1/4}}\\right)$ and we can eliminate the dependence on $\\varepsilon$. Since practical implementations often use very small $\\varepsilon$, we specifically choose $\\varepsilon = \\frac{\\sigma_s^2}{d}$ to make $\\varepsilon$ as small as possible while maintaining this condition. While larger values of $\\varepsilon$ remain theoretically permissible, selecting values smaller than $\\frac{\\sigma_s^2}{d}$ would reintroduce explicit $\\varepsilon$-dependence in the final convergence rate.\n\nSo the crucial technique for eliminating the dependence on $\\varepsilon$ in our convergence rate stems from the setting of $\\varepsilon = \\frac{\\sigma_s^2}{d}$, rather than the specific choices of $1-\\theta$ and $\\eta$. From lines 352-355, we set $1-\\theta$ to minimize the last two terms in Eq. (17), and determine $\\eta$ and $\\nu$ to balance the first two terms with the last two terms in Eq. (17), while strictly maintaining the constraint $\\eta^2\\leq \\frac{\\varepsilon(1-\\theta)^2}{4L^2}$. When $\\varepsilon$ increases, $\\eta$, $\\nu$, and $\\lambda$ adjust proportionally.\n\n(2) Achieve the $O(\\log(1/\\varepsilon))$ dependency?\n\nTo the best of our knowledge, existing convergence analyses for adaptive gradient methods can be broadly categorized into two frameworks. The first framework (e.g., [1-3] for Adam and [4] for RMSProp) achieves weak $\\varepsilon$-dependence (typically $\\log\\frac{1}{\\varepsilon}$) but often results in complex proofs with stronger dependence on terms $L$, $\\sigma_s$, $f(x^1)-f^*$, and especially the dimension $d$ (only for Adam, excluding the simpler AdaGrad and RMSProp). The second framework (e.g., [5]) instead generally yields simpler proofs and tighter convergence rates but introduces explicit $\\varepsilon$-dependence in the convergence rate (typically $1/\\varepsilon^a$ for some $a$). Our work adopts the second framework. Currently, we do not know how to combine the advantages of the two frameworks to achieve the tight convergence rate while maintaining the $\\log\\frac{1}{\\varepsilon}$ dependence on $\\varepsilon$. Thanks for this insightful suggestion and we will investigate it in future work.\n\n(3) $\\varepsilon$ is determined by $d$ instead of any constant.\n\nAs discussed above, our analysis only requires $d \\sqrt{\\varepsilon} \\geq ||\\sigma||_1$ for $\\varepsilon$, which is satisfied by $\\varepsilon = \\frac{\\sigma_s^2}{d}$, making $\\varepsilon$ only dependent on $\\sigma_s$ and $d$, rather than the other constants.\n\n[1]. A. Defossez, et al. A simple convergence proof of Adam and AdaGrad. TMLR 2022.\n\n[2]. Y. Hong, et al. On convergence of Adam for stochastic optimization under relaxed assumptions. NeurIPS 2024.\n\n[3]. B. Wang, et al. Closing the gap between the upper bound and the lower bound of Adam’s iteration complexity. NeurIPS 2023.\n\n[4]. S. Xie, et al. Adam exploits $\\ell_{\\infty}$-geometry of loss landscape via coordinate-wise adaptivity. ICLR 2025.\n\n[5]. H. Li, et al. Convergence of Adam under relaxed assumptions. NeurIPS 2023.\n\n$\\mathbf{Weakness\\enspace 3}$. \n\nWe are grateful to the reviewer for this constructive suggestion. We will move it in the main body of our final version."}, {"review_id": "VndqZc3tSd", "reviewer": "Reviewer_9v39", "summary": "The paper proves that AdamW with a constant weight-decay parameter λ achieves a non-convex convergence rate $\\frac1K\\sum_{k=1}^K \\mathbb E\\bigl[\\|\\nabla f(x_k)\\|_1\\bigr] = O \\Bigl(\\tfrac{\\sqrt d}{K^{1/4}}\\Bigr)$,\nmatching the best known $K^{-1/4}$ rate of SGD up to a $\\sqrt d$ factor. The analysis avoids the vanishing-$\\lambda$ assumption common in prior work on Adam-type methods and introduces a neat decoupling of the EMA constants $\\beta, \\theta$ from weight decay. A corollary shows that standard Adam ($λ = 0$) meets the same rate without the restrictive $\\beta–\\theta$ coupling used in earlier proofs. The authors also provide empirical evidence that $\\|\\nabla f\\|_1\\approx\\Theta(\\sqrt d)\\|\\nabla f\\|_2$, clarifying why an $\\ell_1$ metric is theoretically comparable to SGD’s $\\ell_2$ guarantee.", "questions": "- Although I am positive about the paper overall, I am still not sure why $\\ell_1$-norm bound is adopted for the convergence analysis. Is an analogous $\\ell_2$ or averaged-iterate result attainable with the same technique, or is $\\ell_1$ essential?\n\n- Could the author include a study varying $\\lambda$ to show where theory-suggested upper bounds start to matter?", "limitations": "- The paper itself acknowledges that the step-size/$\\lambda$ regime is mainly of theoretical interest; practical recipes (e.g., linear warm-up or cosine decay) are not analyzed. Suggest adding a short discussion of how to bridge this gap or empirical traces showing safe hyper-parameter inflation.\n\n- There are some typos (e.g., $1 \\leq \\beta \\leq 1$ in Corollary 2), but they are easily fixable.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. First constant-$\\lambda$ analysis for AdamW: Earlier theory required $\\lambda \\to 0$ or elaborate scheduling; this work closes that gap and aligns theory with common practice.\n\n2. Tight rate in an $\\ell_1$ norm: By linking $\\ell_1$ and $\\ell_2$ gradient norms in high-dimensional settings, the paper shows its bound is essentially on par with the optimal SGD rate.\n\n3. Clean proof techniques: The authors streamline variance-control lemmas and sidestep bounding tricks (e.g., biased-second-moment correction) that earlier analyses relied on, making the argument easier to follow or reuse.\n\n## Weaknesses\n\n1. Practical hyper-parameter ranges are narrow. The theorem fixes $\\(\\eta=\\tfrac{\\sqrt{f(x_1)-f^\\*}}{4dKL}\\)$, which can be orders of magnitude below learning-rates used in LLM training, and imposes conservative upper bounds on $\\lambda$. This limits immediate applicability.\n\n2. There are some minor presentation flaws: A few typographical lapses remain (e.g., $\\beta$ range in Corollary 2 printed as $1 \\leq \\beta \\leq 1$) and some constants are undeclared in the main text; these are easily fixed in revision.\n\n## Other comments\n\n- The authors should include the recent paper below [1], which is related to the convergence analysis of Adam-style optimizer.\n\n[1] ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate. https://arxiv.org/abs/2411.02853", "was_revised": true, "final_justification": "Since the authors have addressed most of my concerns during the rebuttal period, and I want to support this paper for the acceptance, I have increased my score from 5 to 6.", "rebuttal": "We sincerely thank a lot for the reviewer's effort and valuable suggestions on our manuscript. Below are our responses to the major concern.\n\n$\\mathbf{Response\\enspace to\\enspace Weaknesses}$:\n\n$\\mathbf{Weakness\\enspace 1}$. \n\nWe acknowledge that some hyper-parameters are unpractical, such as $\\eta=\\sqrt{ \\frac{ f(x^1)-f^* }{ 4KdL }}$. We would like to emphasize the inherent challenges in bridging theoretical analysis with practical implementations. Indeed, modern LLMs are not trained to full convergence, and practical configurations often deviate from theoretical settings. For instance, prior work [1,2] demonstrates through constructed examples that Adam with common hyperparameters ($\\beta_1=0.9$, $\\beta_2={0.999, 0.997, 0.995, 0.993\\}$, and step-size $\\eta_k=0.1/\\sqrt{k}$) fail to converge to stationary points (see [2, Figure 2]). Similarly, while nonconvex SGD analysis typically assumes small step-sizes decaying to zero ($\\eta_k=O(\\frac{1}{\\sqrt{k}})$), practical LLMs training often employs cosine decay scheduler that plateau at small positive values. We fully agree that investigating Adam's property under realistic configurations represents an important research direction, as it could yield valuable insights for LLMs hyperparameter tuning and training dynamics interpretation. However, the practical configurations may not guarantee the convergence. For example, empirical evidence from recent study [3, Figure 9] demonstrates that during the successful training of BERT and GPT-2, the $\\ell_2$ gradient norm typically remains within the range [0,10] and hardly decreases throughout the optimization process, even though the objective function decreases sufficiently. \n\nWe will follow the reviewer's valuable suggestion to conduct dedicated research to bridge this theory-practice gap in future work.\n\n[1]. Y. Zhang, et al. Adam can converge without any modification on update rules. NeurIPS 2024.\n\n[2]. B. Wang, et al. Provable adaptivity of Adam under non-uniform smoothness. KDD 2024.\n\n[3]. H. Tran, et al. Empirical tests of optimization assumptions in deep learning. arXiv 2407.01825, 2024\n\n$\\mathbf{Weakness\\enspace 2}$. \n\nWe are grateful to the reviewer for these presentation flaws and will carefully address them in our final version.\n\n$\\mathbf{Other\\enspace comments}$. \n\nWe also sincerely thank the reviewer for bringing this relevant citation to our attention. It will be properly cited in our revised manuscript.\n\n$\\mathbf{Response\\enspace to\\enspace Questions}$:\n\n$\\mathbf{Question\\enspace 1}$. \n\nWe appreciate this insightful question regarding our use of the $\\ell_1$ norm. Our justification is threefold:\n\nFirst, since Adam can be viewed as a smoothed variant of SignSGD/Signum [4,5], both of which employ $\\ell_1$ norm gradient measurements. Thus, adopting the $\\ell_1$ norm for Adam's convergence analysis represents a consistent choice.\n\nSecond, as discussed on lines 19-23, recent work [6] established that Adam's iterates converge to solutions of problem (1) under the infinity norm constraint. The resulting KKT conditions (Eq. 2) naturally involve the gradient's $\\ell_1$ norm. Then, we explained why we subsequently simplify the KKT conditions to the pure gradient $\\ell_1$ norm on lines 28-32.\n\nThird, when using the gradient $\\ell_2$ norm, it would yield slower convergence rate as demonstrated in [7] (see the comparisons on lines 204-211). While our techniques could theoretically improve [7]'s rate, this would require the impractical condition $\\varepsilon\\geq \\sigma_s^2$. In contrast, our $\\ell_1$-norm approach only needs $\\varepsilon= \\frac{ \\sigma_s^2 }{d}$, a significantly milder requirement that aligns with common practice.\n\n[4]. J. Bernstein, et al. SignSGD: compressed optimisation for non-convex problems. ICML 2018.\n\n[5]. L. Balles, et al. Dissecting Adam: The sign, magnitude and variance of stochastic gradients. ICML 2018.\n\n[6]. S. Xie, et al. Implicit bias of AdamW: $\\ell_{\\infty}$ norm constrained optimization. ICML 2024.\n\n[7]. H. Li, et al. Convergence of Adam under relaxed assumptions. NeurIPS 2023.\n\n$\\mathbf{Question\\enspace 2}$. \n\nFollowing the reviewer's valuable suggestion, we have conducted additional experiments to evaluate the training behavior across different $\\lambda$ values. We perform the analysis on both ResNet-50 (CIFAR100) and GPT-2 (OpenWebText), study the training behavior, and estimate the value of the derived upper bound of $\\lambda$ to assess its compatibility with practical choices. For each experiment, we vary $\\lambda$ values as ${0.01, 0.05, 0.1, 0.5}$ and report the average training loss at the beginning, end, and several intermediate epochs/steps. To provide an accurate yet computationally feasible estimation of the upper bound, we set $\\hat{\\sigma}_s^2 = \\sigma_s^2$ and approximate the noise variance as \n\n$\\sigma_s^2\\approx\\max_{k=1}^K ||g^k - \\nabla f(x^k)||^2$. \n\nSimilarly, the Lipschitz smooth constant is approximated as \n\n$L\\approx\\max_{k=1}^K \\frac{||\\nabla f(x^{k+1}) - \\nabla f(x^k)||}{||x^{k+1} - x^k||}$. \n\nAdditionally, we set $f^\\star = 0$ to further ensure a conservative estimate. To obtain the best estimation of $\\sigma_s^2$ and $L$, we compute the maximum over all $\\lambda$ settings.  Given the knowledge of $d$, $K$, and $f(x^1)$, the required $\\lambda$ bound can be computed. Specifically, for ResNet-50 (CIFAR100), we have the estimated upper bound of $0.5729$. For GPT-2 (OpenWebText), we estimate as $0.3464$. Below are the detailed numerical results.\n\n- ResNet-50 (CIFAR100)\n| Epoch         | 1 | 20 | 40 | 60 | 80 | 100 |\n|--------------|----|-----|-----|-----|-----|------|\n| $\\lambda$=0.01 | 3.92237 | 0.55829 | 0.09655 | 0.02322 | 0.00325 | 0.00103 |\n| $\\lambda$=0.05 | 3.95005 | 0.63125 | 0.22489 | 0.09712 | 0.00893 | 0.00120 |\n| $\\lambda$=0.1  | 3.90191 | 0.93962 | 0.53329 | 0.21229 | 0.02505 | 0.00218 |\n| $\\lambda$=0.5  | 3.87442 | 1.98060 | 1.78694 | 1.46280 | 0.76590 | 0.17032 |\n\n- GPT-2 (OpenWebText)\n| Step         | 1 | 10000 | 20000 | 30000 | 40000 | 50000 |\n|--------------|----|--------|--------|--------|--------|--------|\n| $\\lambda$=0.01 | 7.05598 | 3.08517 | 3.02195 | 2.99443 | 2.98804 | 2.99229 |\n| $\\lambda$=0.05 | 7.05515 | 3.09249 | 3.02159 | 2.97827 | 2.95811 | 2.96719 |\n| $\\lambda$=0.1  | 7.06873 | 3.11096 | 3.03222 | 2.97760 | 2.94311 | 2.95332 |\n| $\\lambda$=0.5  | 7.05931 | 3.27434 | 3.16656 | 3.06161 | 2.96586 | 2.96249 |\n\nThese results indicate that AdamW exhibits stable convergence behavior even under relatively large $\\lambda$ values, though it converges more slowly when $\\lambda$ approaches the upper bound for ResNet. \n\n$\\mathbf{Response\\enspace to\\enspace Limitations}$:\n\n$\\mathbf{Limitation\\enspace 1}$. \n\nWe sincerely appreciate this insightful suggestion regarding the theory-practice gap in step-size selection. We fully agree that bridging our theoretical analysis with practical step-size regimes represents valuable future work. We will add a short discussion in Section 2.5 of the final version.\n\n$\\mathbf{Limitation\\enspace 2}$. \n\nWe are grateful to the reviewer for catching the typos and we will correct it in our final version."}, {"review_id": "1C3ROHU2bv", "reviewer": "Reviewer_bJjL", "summary": "This paper investigates the convergence properties of AdamW under standard optimization settings. The convergence is analyzed with respect to the average of the L1-norm of the gradient across iterations. The assumptions made—smoothness of the objective function, unbiased stochastic gradients, and bounded gradient variance—are standard in the literature. Under these assumptions, the authors show that AdamW achieves convergence rates competitive with those of SGD. \n\nThe core contribution lies in the theoretical convergence analysis, along with a detailed comparison of assumptions and results relative to prior work. The authors claim theirs is the first convergence result for AdamW and Adam$^{1}$ established under these standard conditions.\n\n\n$^{1}$ [1] provides similar convergence results for Adam under standard assumptions, although their analysis requires a conversion over optimality conditions, targets a different set of converged parameters, and involves compromises in the range of hyperparameters.\n\n[1] Ahn, Kwangjun, and Ashok Cutkosky. \"Adam with model exponential moving average is effective for nonconvex optimization.\" arXiv preprint arXiv:2405.18199 (2024).", "questions": "$\\textbf{Technical quality:}$\n\n1. Line 88: The stated rate $\\mathcal{O}\\left(\\frac{\\sqrt{d}}{\\sqrt{K}}\\right)$ is reminiscent of convergence guarantees for stochastic problems. However, is this rate optimal in the deterministic setting? Could the authors elaborate on why guarantees for stochastic settings can be directly extended to deterministic ones?\n\n\n2. Line 136: You assume a relatively tight constraint on $\\beta_2$ (i.e., $\\beta_1 \\leq\\beta_2 \\leq \\frac{(1 + \\beta_1)^{2}}{4}$), but one can derive similar bounds under the looser condition $\\beta_1 \\leq \\beta_2 \\leq \\sqrt{\\beta_1}$:\n$$\\frac{(1-\\beta_{1})^{2}}{(1-\\beta_{2})}\\sum_{n=1}^{s}\\left(\\frac{\\beta_{1}^2}{\\beta_{2}}\\right)^{s-1-n} = \\frac{(1-\\beta_{1})^{2}}{(1-\\beta_{2})}\\frac{1 - \\left(\\frac{\\beta_{1}^2}{\\beta_{2}}\\right)^{s-1}}{1 - \\frac{\\beta_{1}^2}{\\beta_{2}}} \\leq \\frac{(1-\\beta_{1})^{2}}{(1-\\beta_{2})^{2}}\\leq \\frac{(1- \\sqrt{\\beta_{1}})^{2}(1+ \\sqrt{\\beta_{1}})^{2}}{(1-\\sqrt{\\beta_{1}})^{2}}\\leq 4.$$\nFor instance, supposing $\\beta_{1} = 0.9$, then the range for $\\beta_{2}$ changes from $(0.9, 0.9025)$ to $(0.9, 0.9486)$. This alternative formulation could potentially simplify the analysis and lead to similar bounds. \n\n3. One of the core distinctions between Adam and AdamW is the additional weight decay term in AdamW. I assume that one of the main technical challenges lies in handling this term. However, it appears that simply controlling $\\lambda$ allows the adaptation of Adam's existing analysis to AdamW with minimal changes, (intuitively). Could the authors clarify what specific technical hurdles they faced in extending the convergence analysis framework beyond tuning $\\lambda$ to a particular range?\n\n$\\textbf{Other questions:}$\n\n1. Regarding the AdamW update rule:, I believe the $\\epsilon$ term is typically added outside the square root in the denominator. (But I assume it will not affect the analysis or negate the conclusion.). Additionally, I recommend adding the line: $g \\leftarrow \\text{GradOracle}(\\cdot)$.\n\n2. Figure 3: Please specify the model used in the experiments. I assume each subfigure corresponds to the same model trained with different values of $\\lambda = 0.01, 0.05, 0.1$. However, it is unclear why the maximum norm is not monotonic with respect to $\\lambda$—for example, $\\lambda = 0.05$ yields the largest norm (~12.5). Can the authors provide insight into this behavior?\n\n3. The paper repeatedly states that “∇f(x) is drawn from Gaussian distribution N (0, 1)”. While this is convenient for analysis, it may oversimplify the problem. In practice, the gradient distribution often exhibits drifting means, I believe.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "$\\textbf{Strengths:}$\n\n1. The paper provides a complete and rigorous convergence proof of AdamW under standard assumptions—namely, smoothness of the objective, unbiased gradient estimates, and bounded variance. As claimed by the authors, this is the first such result under these widely accepted conditions.\n\n\n2. The exposition includes a reasonably clear summary of existing convergence results for both Adam and AdamW, helping position the current work within the broader literature.\n\n\n$\\textbf{Weakness:}$\n\n1. The technical contributions in Section 3.1 are presented as key innovations; however, they are relatively standard. In particular, Lemma 2 follows known analysis patterns and could be further improved (as discussed in point 2 under the \"Technical Quality\" of  \"Question\"). The second main result is similarly derived by fixing $\\lambda$ to a specific value, which limits its novelty.\n\n\n2. The paper lacks sufficient interpretability of its technical derivations. For instance, some important steps in the proofs are not explained. Line 152 appears to rely on Jensen’s inequality, and line 3 of Proof 3 uses the Cauchy–Schwarz inequality, yet neither is explicitly stated. This omission makes it harder for readers to follow the reasoning.\n\n\n3. The analysis omits the bias correction terms in AdamW. While this simplification may help theoretical tractability, such terms are critical for practical performance.\n\n\n4. The theoretical results do not offer meaningful insight into Adam/AdamW’s empirical superiority, nor do they provide guidance for practical algorithm design. The convergence guarantees require hyperparameter settings—such as $\\beta_1$, $\\beta_2$, $\\lambda$, and $\\epsilon$—that deviate from those typically used in practice, which reduces the relevance of the results.\n\n\n5. Overall, while the work fills a theoretical gap, its practical and conceptual significance remains unclear beyond establishing convergence under standard conditions (by setting extreme values for the hyperparameters, which potentially ``approximate'' SGD).", "was_revised": true, "final_justification": "I will keep my score due to the raised weaknesses 4 and 5.", "rebuttal": "We sincerely thank a lot for the reviewer's effort and valuable suggestions on our manuscript. Below are our responses to the major concern.\n\n$\\mathbf{Response\\enspace to\\enspace Weaknesses}$:\n\n$\\mathbf{Weakness\\enspace 1}$. \n\nWhile the technical derivation in Section 3.1 is relatively straightforward, we highlight its importance in establishing the bound of term (g) (line 147) to constrain term (i) (line 154). While the techniques in Section 3.1 represent one aspect of our innovations, the key theoretical innovation of our proof appears in Section 3.2, where we strategically relax terms (f) and (g) (line 151) and control the second moment estimate via Eq. (12) (line 153). Then we effectively absorb term (h) into the relaxed bound derived from term (f) (line 154). This sequence of operations enables us to manage the troublesome term (g) (line 129) introduced by weight decay and derive the bound on line 161. Finally, we eliminate the dependence on $\\varepsilon$ in term (k) (line 163) by the tight bound of the second moment estimate in Eq. (12) and careful configuration of $\\varepsilon$.\n\n$\\mathbf{Weakness\\enspace 2\\enspace and\\enspace 3}$. \n\nWe are grateful to the reviewer for these constructive suggestions and will carefully incorporate detailed explanations of the technical derivations in our final version. Specifically, we will try to add a new theorem establishing convergence guarantees for the complete AdamW with bias correction terms in the Appendix.\n\n$\\mathbf{Weakness\\enspace 4\\enspace and\\enspace 5}$. \n\n(1). Insight into Adam/AdamW’s empirical superiority.\n\nThe reviewer has raised several valuable research questions, for example, offering insight into Adam/AdamW’s empirical superiority, which indeed is a fundamentally important challenge worthy of dedicated investigation in a separate paper (e.g., [1-6]), and it falls beyond the scope of our current study. Our paper establishes the convergence rate of AdamW, which is analogous to that of SGD, through the lens in the traditional optimization field. \n\n(2). Provide guidance for practical algorithm design.\n\nDesigning practical algorithms for LLMs training is also a fundamentally important challenge worthy of dedicated investigation in a separate paper. While numerous optimizers are proposed annually, some marginally outperforming AdamW while others failing to deliver claimed improvements, AdamW remains the dominant optimizer for LMMs to date. Providing novel and effective algorithmic design principles presents significant challenges and it is also beyond the scope of this paper.\n\n(3). Hyperparameter settings that deviate from those typically used in practice.\n\nThere is a gap between theory and practice in hyperparameter configuration. Indeed, modern LLMs are not trained to full convergence, and practical configurations often deviate from theoretical settings. For instance, prior work [7,8] demonstrates through constructed examples that Adam with common hyperparameters ($\\beta_1=0.9$, $\\beta_2={0.999, 0.997, 0.995, 0.993}$, and step-size $\\eta_k=0.1/\\sqrt{k}$) fail to converge to stationary points (see [8, Figure 2]). Similarly, while nonconvex SGD analysis typically assumes step-sizes decaying to zero ($\\eta_k=1/\\sqrt{k}$), practical LLMs training often employs cosine decay scheduler that plateau at small positive values. We fully agree that investigating Adam's property under realistic configurations represents an important research direction, as it could yield valuable insights for Adam/AdamW’s empirical superiority and practical algorithm design. However, the practical configurations may not guarantee the convergence. For example, empirical evidence from recent study [9, Figure 9] demonstrates that during the successful training of BERT and GPT-2, the $\\ell_2$ gradient norm typically remains within the range [0,10] and hardly decreases throughout the optimization process. \n\n(4). Extreme values for the hyperparameters potentially approximating SGD.\n\nWhile our analysis employs standard convergence proof methodologies in the traditional optimization field, it exhibits fundamental difference from that of SGD, although certain hyperparameter choices were made primarily for theoretical tractability. See the key theoretical innovations in the response to Weakness 1. We sincerely thank the reviewer for recognizing that our work fills a theoretical gap and we maintain that this theoretical contribution meets NeurIPS' rigorous theoretical standards.\n\n[1]. Y. Zhang, et al. Why transformers need Adam: A Hessian perspective. NeurIPS 2024.\n\n[2]. F. Kunstner, et al. Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be. ICLR 2023.\n\n[3]. F. Kunstner, et al. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. NeurIPS 2024.\n\n[4]. T. Sreckovic, et al. Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. arXiv 2506.12543, 2025.\n\n[5]. A. Tomihari, et al. Understanding why Adam outperforms SGD: gradient heterogeneity in transformers. arXiv 2502.00213, 2025.\n\n[6]. Y. Pan, et al. Toward understanding why Adam converges faster than SGD for transformers. arXiv 2306.00204, 2023.\n\n[7]. Y. Zhang, et al. Adam can converge without any modification on update rules. NeurIPS 2024.\n\n[8]. B. Wang, et al. Provable adaptivity of Adam under non-uniform smoothness. KDD 2024.\n\n[9]. H. Tran, et al. Empirical tests of optimization assumptions in deep learning. arXiv 2407.01825, 2024\n\n$\\mathbf{Response\\enspace to\\enspace Questions}$:\n\n$\\mathbf{Technical\\enspace Quality\\enspace 1}$.\n\nRecent work [10, Theorem 4.1] established a fundamental lower bound $ \\Omega\\left( \\sqrt{ \\frac{dL ( f(x^1)-f^* )}{T} }+ \\left( \\frac{ dL ( f(x^1)-f^* ) ||\\sigma||_1^2 }{T} \\right)^{1/4} \\right) $ for SGD with $\\ell_1$-norm gradient measurement, which implies that the $O( \\sqrt{ \\frac{d}{K} } )$ rate is optimal for GD under identical $\\ell_1$-norm criteria in the determnistic setting.\n\nWhen the noise variance is small satisfying $\\sigma_s^2 \\leq \\frac{ L( f(x^1)-f^* ) }{ K }$, the first term in our convergence rate (line 65) becomes identical to the second term, thus making the convergence rate for stochastic settings directly extended to deterministic ones (or, more broadly speaking, low-noise regimes).\n\n[10]. R. Jiang, et al. Convergence analysis of adaptive gradient methods under refined smoothness and noise assumptions. COLT 2025.\n\n$\\mathbf{Technical\\enspace Quality\\enspace 2}$.\n\nWe gratefully thank the reviewer for this looser condition. We will add an acknowledgment footnote in our final version to clarify that this condition is derived by the NeurIPS reviewer.\n\n$\\mathbf{Technical\\enspace Quality\\enspace 3}$. \n\nAs explained in the response to Weakness 1, our key theoretical innovations are as follows:\n\nFirst, we strategically relax terms (f) and (g) (line 151) and control the second moment estimate via Eq. (12) (line 153), which enables the novel absorption of term (h) into the relaxed bound derived from term (f) (line 154). This sequence of operations enables us to manage the troublesome term (g) (line 129) introduced by the weight decay and derive the bound on line 161.\n\nSecond, we eliminate the dependence on $\\varepsilon$ in term (k) (line 163) by careful configuration of $\\varepsilon$ and tight bound of the second moment estimate in Eq. (12).\n\nThe first step is crucial to deal with the troublesome weight decay term in AdamW and this idea has no precedent in the literature. We respectfully disagree with the reviewer that they are minimal and intuitively changes.\n\nFor Adam, while the first step becomes unnecessary (because weight decay disappears in Adam), the second step remains essential for establishing tight $\\varepsilon$-independent convergence rate. Without our two key techniques in the second step, the closest related work [11] achieves only slower and $\\varepsilon$-dependent convergence rate. See the detailed comparisons to [11] on lines 204-211. This is the principal technical hurdle existing literature on Adam faced to achieve tight convergence rate comparable to SGD.\n\n[11]. H. Li, et al. Convergence of Adam under relaxed assumptions. NeurIPS 2023.\n\n$\\mathbf{Other\\enspace Question\\enspace 1}$.\n\nThe reviewer is absolutely correct to note that in the standard PyTorch implementation, $\\varepsilon$ is added outside the square root (see Lines 208-209 in our paper). We placed this term inside the square root to simplify the mathematical expressions and it does not affect our conclusion. This is also adopted in some theoretical literature [12,13].\n\n$g\\leftarrow \\mbox{GradOracle}(\\cdot)$: we gratefully thank the reviewer for this suggestion and we will add this line in the algorithm representation of our final version.\n\n[12]. A. Defossez, et al. A simple convergence proof of Adam and AdaGrad. TMLR 2022.\n\n[13]. S. Xie, et al. Adam exploits $\\ell_{\\infty}$-geometry of loss landscape via coordinate-wise adaptivity. ICLR 2025.\n\n$\\mathbf{Other\\enspace Question\\enspace 2}$.\n\nWe have specified the models on the top of Figure 1. In Figures 1-4, the first column presents the experimental results of ResNet50 on CIFAR100, the second column is ResNet50 on ImageNet, and the third column is GPT2 on OpenWebText. For each model-dataset combination, we adopted the standard $\\lambda$ values commonly used in the literature. Because each subfigure represents distinct model-dataset pairs, the maximum norm does not exhibit monotonic behavior with respect to $\\lambda$.\n\n$\\mathbf{Other\\enspace Question\\enspace 3}$.\n\nWe fully agree with the reviewer. The gradient distribution often exhibits drifting means across different layers/blocks in modern architectures like Transformers. We conducted experiments on real-world deep learning tasks (Figure 2), which reveal that the ratio $\\frac{ || \\nabla f(x) ||_1 }{ || \\nabla f(x) ||_2 } \\approx \\frac{ \\sqrt{d} }{10} $, smaller than the theoretical prediction of $\\sqrt{d}$."}]

Author rebuttal:

Meta-review: The paper establishes the first convergence rate, $O(\frac{\sqrt{d}}{T^{1/4}})$, for the AdamW optimizer under the practical condition of a constant, non-vanishing weight decay.

**Strengths**: This paper provides a novel theoretical result for a widely-used optimizer, filling a significant gap in the literature. The proofs are rigorous, and the techniques for handling the epsilon term in the denominator are innovative. The paper is well-written and clearly positions its contribution.

**Weaknesses**: The hyperparameter settings required by the theory are more restrictive than those used in practice. Also the dependency on epsilon is polynomial and sensitive. It requires knowing the gradient noise covariance to set the epsilon for the proved rate. The analysis also omits the bias-correction terms used in standard implementations.

**Reason for Acceptance**:
This paper provides a foundational non-convex convergence analysis for a key optimizer in modern machine learning, AdamW. The reviewers all agree on the paper's novelty and praise the clarity of presentation. It is a clear accept.

**Rebuttal Summary**:
Reviewers raised concerns about the practicality of convergence result due to its narrow range of weight decay coefficient and the choice of the $\ell_1$ norm. The authors effectively addressed these with new experiments on large models and strong theoretical justifications. This productive discussion resolved the main concerns and led two reviewers to raise their scores, solidifying the consensus for acceptance.
Decision: Accept (poster)