Title: On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
Venue: NeurIPS 2025
Accepted: yes
Keywords: AdamW

Adam

convergence analysis

nonconvex optimization
Abstract: As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning. However, its convergence behavior is not theoretically well-understood. This paper establishes the convergence rate $\frac{1}{K}\sum_{k=1}^K E[||\nabla f(x^k)||_1]\leq O(\frac{\sqrt{d}C}{K^{1/4}})$ for AdamW measured by $\ell_1$ norm, where K represents the iteration number, d denotes the model dimension, and C matches the constant in the optimal convergence rate of SGD. Theoretically, we have $E[||\nabla f(x)||_1]\geq\sqrt{\frac{2d}{\pi}}E[||\nabla f(x)||_2]$ when each element of $\nabla f(x)$ is generated from Gaussian distribution $\mathcal N(0,1)$. Empirically, our experimental results on real-world deep learning tasks reveal $||\nabla f(x)||_1=\varTheta(\sqrt{d})||\nabla f(x)||_2$. Both support that our convergence rate can be considered to be analogous to the optimal convergence rate of SGD.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the convergence properties of AdamW under standard optimization settings. The convergence is analyzed with respect to the average of the L1-norm of the gradient across iterations. The assumptions made—smoothness of the objective function, unbiased stochastic gradients, and bounded gradient variance—are standard in the literature. Under these assumptions, the authors show that AdamW achieves convergence rates competitive with those of SGD. \n\nThe core contribution lies in the theoretical convergence analysis, along with a detailed comparison of assumptions and results relative to prior work. The authors claim theirs is the first convergence result for AdamW and Adam$^{1}$ established under these standard conditions.\n\n\n$^{1}$ [1] provides similar convergence results for Adam under standard assumptions, although their analysis requires a conversion over optimality conditions, targets a different set of converged parameters, and involves compromises in the range of hyperparameters.\n\n[1] Ahn, Kwangjun, and Ashok Cutkosky. \"Adam with model exponential moving average is effective for nonconvex optimization.\" arXiv preprint arXiv:2405.18199 (2024).\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{Strengths:}$\n\n1. The paper provides a complete and rigorous convergence proof of AdamW under standard assumptions—namely, smoothness of the objective, unbiased gradient estimates, and bounded variance. As claimed by the authors, this is the first such result under these widely accepted conditions.\n\n\n2. The exposition includes a reasonably clear summary of existing convergence results for both Adam and AdamW, helping position the current work within the broader literature.\n\n\n$\\textbf{Weakness:}$\n\n1. The technical contributions in Section 3.1 are presented as key innovations; however, they are relatively standard. In particular, Lemma 2 follows known analysis patterns and could be further improved (as discussed in point 2 under the \"Technical Quality\" of  \"Question\"). The second main result is similarly derived by fixing $\\lambda$ to a specific value, which limits its novelty.\n\n\n2. The paper lacks sufficient interpretability of its technical derivations. For instance, some important steps in the proofs are not explained. Line 152 appears to rely on Jensen’s inequality, and line 3 of Proof 3 uses the Cauchy–Schwarz inequality, yet neither is explicitly stated. This omission makes it harder for readers to follow the reasoning.\n\n\n3. The analysis omits the bias correction terms in AdamW. While this simplification may help theoretical tractability, such terms are critical for practical performance.\n\n\n4. The theoretical results do not offer meaningful insight into Adam/AdamW’s empirical superiority, nor do they provide guidance for practical algorithm design. The convergence guarantees require hyperparameter settings—such as $\\beta_1$, $\\beta_2$, $\\lambda$, and $\\epsilon$—that deviate from those typically used in practice, which reduces the relevance of the results.\n\n\n5. Overall, while the work fills a theoretical gap, its practical and conceptual significance remains unclear beyond establishing convergence under standard conditions (by setting extreme values for the hyperparameters, which potentially ``approximate'' SGD).\n\nQUESTIONS:\n$\\textbf{Technical quality:}$\n\n1. Line 88: The stated rate $\\mathcal{O}\\left(\\frac{\\sqrt{d}}{\\sqrt{K}}\\right)$ is reminiscent of convergence guarantees for stochastic problems. However, is this rate optimal in the deterministic setting? Could the authors elaborate on why guarantees for stochastic settings can be directly extended to deterministic ones?\n\n\n2. Line 136: You assume a relatively tight constraint on $\\beta_2$ (i.e., $\\beta_1 \\leq\\beta_2 \\leq \\frac{(1 + \\beta_1)^{2}}{4}$), but one can derive similar bounds under the looser condition $\\beta_1 \\leq \\beta_2 \\leq \\sqrt{\\beta_1}$:\n$$\\frac{(1-\\beta_{1})^{2}}{(1-\\beta_{2})}\\sum_{n=1}^{s}\\left(\\frac{\\beta_{1}^2}{\\beta_{2}}\\right)^{s-1-n} = \\frac{(1-\\beta_{1})^{2}}{(1-\\beta_{2})}\\frac{1 - \\left(\\frac{\\beta_{1}^2}{\\beta_{2}}\\right)^{s-1}}{1 - \\frac{\\beta_{1}^2}{\\beta_{2}}} \\leq \\frac{(1-\\beta_{1})^{2}}{(1-\\beta_{2})^{2}}\\leq \\frac{(1- \\sqrt{\\beta_{1}})^{2}(1+ \\sqrt{\\beta_{1}})^{2}}{(1-\\sqrt{\\beta_{1}})^{2}}\\leq 4.$$\nFor instance, supposing $\\beta_{1} = 0.9$, then the range for $\\beta_{2}$ changes from $(0.9, 0.9025)$ to $(0.9, 0.9486)$. This alternative formulation could potentially simplify the analysis and lead to similar bounds. \n\n3. One of the core distinctions between Adam and AdamW is the additional weight decay term in AdamW. I assume that one of the main technical challenges lies in handling this term. However, it appears that simply controlling $\\lambda$ allows the adaptation of Adam's existing analysis to AdamW with minimal changes, (intuitively). Could the authors clarify what specific technical hurdles they faced in extending the convergence analysis framework beyond tuning $\\lambda$ to a particular range?\n\n$\\textbf{Other questions:}$\n\n1. Regarding the AdamW update rule:, I believe the $\\epsilon$ term is typically added outside the square root in the denominator. (But I assume it will not affect the analysis or negate the conclusion.). Additionally, I recommend adding the line: $g \\leftarrow \\text{GradOracle}(\\cdot)$.\n\n2. Figure 3: Please specify the model used in the experiments. I assume each subfigure corresponds to the same model trained with different values of $\\lambda = 0.01, 0.05, 0.1$. However, it is unclear why the maximum norm is not monotonic with respect to $\\lambda$—for example, $\\lambda = 0.05$ yields the largest norm (~12.5). Can the authors provide insight into this behavior?\n\n3. The paper repeatedly states that “∇f(x) is drawn from Gaussian distribution N (0, 1)”. While this is convenient for analysis, it may oversimplify the problem. In practice, the gradient distribution often exhibits drifting means, I believe.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves that AdamW with a constant weight-decay parameter λ achieves a non-convex convergence rate $\\frac1K\\sum_{k=1}^K \\mathbb E\\bigl[\\|\\nabla f(x_k)\\|_1\\bigr] = O \\Bigl(\\tfrac{\\sqrt d}{K^{1/4}}\\Bigr)$,\nmatching the best known $K^{-1/4}$ rate of SGD up to a $\\sqrt d$ factor. The analysis avoids the vanishing-$\\lambda$ assumption common in prior work on Adam-type methods and introduces a neat decoupling of the EMA constants $\\beta, \\theta$ from weight decay. A corollary shows that standard Adam ($λ = 0$) meets the same rate without the restrictive $\\beta–\\theta$ coupling used in earlier proofs. The authors also provide empirical evidence that $\\|\\nabla f\\|_1\\approx\\Theta(\\sqrt d)\\|\\nabla f\\|_2$, clarifying why an $\\ell_1$ metric is theoretically comparable to SGD’s $\\ell_2$ guarantee.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. First constant-$\\lambda$ analysis for AdamW: Earlier theory required $\\lambda \\to 0$ or elaborate scheduling; this work closes that gap and aligns theory with common practice.\n\n2. Tight rate in an $\\ell_1$ norm: By linking $\\ell_1$ and $\\ell_2$ gradient norms in high-dimensional settings, the paper shows its bound is essentially on par with the optimal SGD rate.\n\n3. Clean proof techniques: The authors streamline variance-control lemmas and sidestep bounding tricks (e.g., biased-second-moment correction) that earlier analyses relied on, making the argument easier to follow or reuse.\n\n## Weaknesses\n\n1. Practical hyper-parameter ranges are narrow. The theorem fixes $\\(\\eta=\\tfrac{\\sqrt{f(x_1)-f^\\*}}{4dKL}\\)$, which can be orders of magnitude below learning-rates used in LLM training, and imposes conservative upper bounds on $\\lambda$. This limits immediate applicability.\n\n2. There are some minor presentation flaws: A few typographical lapses remain (e.g., $\\beta$ range in Corollary 2 printed as $1 \\leq \\beta \\leq 1$) and some constants are undeclared in the main text; these are easily fixed in revision.\n\n## Other comments\n\n- The authors should include the recent paper below [1], which is related to the convergence analysis of Adam-style optimizer.\n\n[1] ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate. https://arxiv.org/abs/2411.02853\n\nQUESTIONS:\n- Although I am positive about the paper overall, I am still not sure why $\\ell_1$-norm bound is adopted for the convergence analysis. Is an analogous $\\ell_2$ or averaged-iterate result attainable with the same technique, or is $\\ell_1$ essential?\n\n- Could the author include a study varying $\\lambda$ to show where theory-suggested upper bounds start to matter?\n\nLIMITATIONS:\n- The paper itself acknowledges that the step-size/$\\lambda$ regime is mainly of theoretical interest; practical recipes (e.g., linear warm-up or cosine decay) are not analyzed. Suggest adding a short discussion of how to bridge this gap or empirical traces showing safe hyper-parameter inflation.\n\n- There are some typos (e.g., $1 \\leq \\beta \\leq 1$ in Corollary 2), but they are easily fixable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a comprehensive convergence analysis for AdamW under non-convex smooth optimization. The convergence rate is optimal with respect to the iteration number, noise level, the initial function value gap, and the smooth parameter.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe major contribution of this paper is mainly in the theoretical part.\n\na. The convergence result for AdamW in this paper looks novel and sound. It seems that this is the first theoretical convergence result for AdamW with an explicit form of convergence rate. I check the parameter setups in detail, and most of the setups are reasonable. Although there are some gaps between the theoretical setups and the practical ones, it's acceptable for a theoretical paper.\n\nb. The proof sketch is clearly written. There are some novel techniques to handle the additional decay parameter brought by AdamW. \n\n**Weaknesses:**\n\na. For the detailed parameter setups, I have some concerns. First, as the authors also claim in Section 4.1, the setup for $\\lambda$ can exceed the default $0.01$. Specifically, if $d$ is sufficiently large, e.g., the dimension of GPT-3.5, the requirement for $\\lambda$ is not very satisfactory. I also find that $\\lambda$ for experiments in this paper still applies $0.01$ or $0.1$. It's better to provide some experiments to verify that a large $\\lambda$ is also acceptable.\n\nb. The dependency of $\\epsilon$ in the convergence rate seems to be $O(1/\\epsilon)$. The authors then use a sufficiently small $1-\\theta$ and $\\eta$, namely $O(\\epsilon)$ order to eliminate the factor $O(1/\\epsilon)$. This may not be so satisfactory. There are some convergence results for Adam with a $O(\\log(1/\\epsilon))$ dependency. Is it possible to achieve the same dependency for AdamW? Also, the setup for $\\epsilon$ is determined by $d$ instead of any constant like the setup in previous literature.\n\nc. I think that a key difference of AdamW to Adam is to hand to the term $\\lambda \\\\|x_i\\\\|_{\\infty}$. The key solution is Lemma 3. So I advise the author to provide the detailed version of Lemma 3 in the main body of the paper.\n\nQUESTIONS:\nI do not have any further questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proved a novel convergence rate of AdamW in terms of the minimal expected $L^1$-norm of the gradient, where the theoretical requirement for the hyperparameters almost fit the realities.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros**\n\n1. While being the most preferable optimizer in the era of large language models, AdamW is not well understood in terms of its theoretical properties like convergence and implicit biases. This work is the first one that proved the convergence rate of Adam with weight decay. \n2. The content is well-organized and the result and its consequences are clearly stated. A \n\n**Cons** \n\n1. Some conditions on the hyperparameters are not realistic, indicating that the bound might not be sufficiently explaining  real-world optimization trajectories. \n2. The theoretical results seem to provide little practical implications on tuning the hyperparameters and understandings on the dynamics, trejactories or implicit bias of AdamW. See questions. \n3. The authors claim that the result is optimal in the sense that the corresponding 2-norm bound matchs the lower bound of SGD. However, the translation between 1-norm and 2-norm are semi-empirical.\n\nQUESTIONS:\n1. Under a different set of conditions, [1] successfully derived a convergence rate of Adam, while $\\beta_2  = 1-\\tilde O(K^{-1})$ and is similar to yours. They claim that the proof can adapt to non-zero $\\beta_1$  but hurts by having a factor of $\\mathrm{poly}(1/(1-\\beta_1))$. While in your corollary on the convergence rate of Adam, you have no requirement on $\\beta_2$  and $\\beta_1$  is close to $1$. Can you briefly compare how you improved this result?  \n2. Does it directly implies that the sequence converge to the KKT point? I am wondering if the infty norm of the trajectory is well controlled so that the gradient $L^1$ norm convergence implies the convergence to KKT points. \n\n[1] Xie, S., Mohamadi, M. A., & Li, Z. (2024). Adam Exploits $\\ell_\\infty $-geometry of Loss Landscape via Coordinate-wise Adaptivity. arXiv preprint arXiv:2410.08198.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}