Simple Convergence Proof of Adam From a Sign-like Descent Perspective

## Abstract

Abstract Adam is widely recognized as one of the most effective optimizers for training deep neural networks (DNNs). Despite its remarkable empirical success, its theoretical convergence analysis remains unsatisfactory. Existing works predominantly interpret Adam as a preconditioned stochastic gradient descent with momentum (SGDM), formulated as 𝒙 t + 1 = 𝒙 t − γ t 𝒗 t + ϵ ∘ 𝒎 t subscript 𝒙 𝑡 1 subscript 𝒙 𝑡 subscript 𝛾 𝑡 subscript 𝒗 𝑡 italic-ϵ subscript 𝒎 𝑡 \bm{x}_{t+1}=\bm{x}_{t}-\frac{\gamma_{t}}{{\sqrt{\bm{v}_{t}}+\epsilon}}\circ%
\bm{m}_{t} bold_italic_x start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT = bold_italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT - divide start_ARG italic_γ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_ARG start_ARG square-root start_ARG bold_italic_v start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_ARG + italic_ϵ end_ARG ∘ bold_italic_m start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT . This perspective necessitates strong assumptions and intricate techniques, resulting in lengthy and opaque convergence proofs that are difficult to verify and extend.
In contrast, we propose a novel interpretation by treating Adam as a sign-like optimizer, expressed as 𝒙 t + 1 = 𝒙 t − γ t ⁢ | 𝒎 t | 𝒗 t + ϵ ∘ Sign ⁢ ( 𝒎 t ) subscript 𝒙 𝑡 1 subscript 𝒙 𝑡 subscript 𝛾 𝑡 subscript 𝒎 𝑡 subscript 𝒗 𝑡 italic-ϵ Sign subscript 𝒎 𝑡 \bm{x}_{t+1}=\bm{x}_{t}-\gamma_{t}\frac{|\bm{m}_{t}|}{{\sqrt{\bm{v}_{t}}+%
\epsilon}}\circ{\rm Sign}(\bm{m}_{t}) bold_italic_x start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT = bold_italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT - italic_γ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT divide start_ARG | bold_italic_m start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | end_ARG start_ARG square-root start_ARG bold_italic_v start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_ARG + italic_ϵ end_ARG ∘ roman_Sign ( bold_italic_m start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) . This reformulation significantly simplifies the convergence analysis. For the first time, with some mild conditions, we prove that Adam achieves the optimal rate of 𝒪 ⁢ ( 1 T 1 / 4 ) 𝒪 1 superscript 𝑇 1 4 {\cal O}(\frac{1}{T^{\nicefrac{{1}}{{4}}}}) caligraphic_O ( divide start_ARG 1 end_ARG start_ARG italic_T start_POSTSUPERSCRIPT / start_ARG 1 end_ARG start_ARG 4 end_ARG end_POSTSUPERSCRIPT end_ARG ) rather than the previous 𝒪 ⁢ ( ln ⁡ T T 1 / 4 ) 𝒪 𝑇 superscript 𝑇 1 4 {\cal O}\left(\frac{\ln T}{T^{\nicefrac{{1}}{{4}}}}\right) caligraphic_O ( divide start_ARG roman_ln italic_T end_ARG start_ARG italic_T start_POSTSUPERSCRIPT / start_ARG 1 end_ARG start_ARG 4 end_ARG end_POSTSUPERSCRIPT end_ARG ) under weak assumptions of the generalized p 𝑝 p italic_p -affine variance and ( L 0 , L 1 , q ) subscript 𝐿 0 subscript 𝐿 1 𝑞 (L_{0},L_{1},q) ( italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_q ) -smoothness, without dependence on the model dimensionality or the numerical stability parameter ϵ italic-ϵ \epsilon italic_ϵ .
Additionally, our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence and offers practical guidelines for tuning learning rates in Adam, further bridging the gap between theory and practice.

## 1 Introduction

Adam is typically formulated as follows:

$$ $\displaystyle{\bm{m}}_{t}=\beta_{1}{\bm{m}}_{t-1}+(1-\beta_{1})\bm{g}_{t},$ (1) $\displaystyle{\bm{v}}_{t}=\beta_{2}{\bm{v}}_{t-1}+(1-\beta_{2})\bm{g}_{t}^{2},$ $\displaystyle\bm{x}_{t+1}=\bm{x}_{t}-\gamma_{t}\frac{\bm{m}_{t}}{\sqrt{\bm{v}_ {t}}+\epsilon},$ $$

where $\bm{g}_{t}=\nabla f(\bm{x}_{t};\zeta_{t})$ denotes the stochastic gradient, $\gamma_{t}$ represents the learning rate, $\epsilon$ symbolizes the numerical stability parameter and $\beta_{1},\beta_{2}$ are the exponential moving average coefficients. For simplicity, bias corrections are omitted.

Currently, Adam (Adam2015,) has emerged as the predominant optimizer for training Transformers (Transformer_2017,), particularly for state-of-the-art large language models (LLMs) (GPT32020,; Palm2023,; Llama2023,) and large vision models (CLIP_2021,; SAM_2023,). Notably, Adam’s influence extends beyond Transformers to modern convolutional neural networks (CNNs), such as ConvNeXt (ConvNext_2022,; ConvNext2_2023,), where it has become the de facto choice for optimization. This is despite the traditional preference for stochastic gradient descent (SGD) (AlexNet2013,; ResNet2016,), which was historically considered more suitable for CNN training.

However, the theoretical convergence analysis of Adam lags behind its significant practical success. The original proof in (Adam2015,) was based on the convexity of the objective function but was later found to be flawed (AMSGrad2018,). To address this, AMSGrad, a fixed variant of Adam, was proposed, but its theoretical analysis still relied on the convexity assumption. Adam_type_proof_2018 were the first to theoretically demonstrate that a class of Adam-type optimizers, including AMSGrad and AdaFom, converge to stationary solutions for non-convex problems. Subsequently, Simple_Adam_proof_2020 provided a simplified proof analyzing the convergence rates of vanilla Adam and Adagrad. However, their analysis required $\beta_{1}<\beta_{2}$ and depended on the model dimensionality $d$. Prac_Adam_proof_2022 introduced practical, easy-to-check conditions to ensure the global convergence of Adam, but their proved convergence rate also heavily relied on the model dimensionality $d$. Notably, the analyses in (Adam_type_proof_2018,; Simple_Adam_proof_2020,; Prac_Adam_proof_2022,) all assumed bounded stochastic gradients. Later, Adam_proof_2022 provided a theoretical proof for random-reshuffling Adam under the weaker affine variance assumption. However, this proof achieved a slower convergence rate with an epoch-complexity bound and relies on the total number of samples. Additionally, these works assumed the conventional uniformly bounded smoothness condition, *i.e.*, the $L$-smoothness condition. Recent studies, however, have shown that the $L$-smoothness assumption is inadequate for optimizing complex DNNs such as LSTMs and Transformers (Clipped_SGD_Proof_2019,; Generalized_signSGD_2022,). Instead, it should be relaxed to the non-uniform $(L_{0},L_{1})$-smoothness condition (see Section 2 for details). Recently, Adaptivity_Adam_proof_2024 analyzed random-reshuffling Adam under the $(L_{0},L_{1})$-smoothness assumption, but their theoretical convergence rate was still based on epoch complexity and depended on the total number of samples. Relaxed_Adam_proof_2024 demonstrated that Adam provably converges to stationary points with the optimal rate under generalized $(L_{0},L_{1},q)$-smoothness. However, this bound heavily relied on a large $\epsilon$, making Adam behave similarly to SGD and losing its adaptive properties. Most recently, Adam_weak_assum_2025 established the convergence rate of a simplified Adam under both the affine variance and the generalized $(L_{0},L_{1},q)$-smoothness assumptions. However, their results still heavily depended on the model dimensionality. A detailed comparison of these convergence analyses for Adam is provided in Table 1.

**Table 1: Comparison of different convergence proofs for Adam. “Conv. Rate" denotes the convergence rate for approaching stationary points (*i.e.*, $\|\nabla F(\bm{x}_{T})\|_{2}\rightarrow 0$). $T$ represents the number of iterations, $E$ the number of epochs, $d$ the model dimensionality, $n$ the total number of samples, and $\epsilon$ the numerical stability parameter.**
$$ $\min_{\bm{x}\in\mathbb{R}^{d}}F(\bm{x})={\mathbb{E}}_{\zeta\sim\mathcal{D}}[f( \bm{x};\zeta)]=\frac{1}{n}\sum_{i=1}^{n}f(\bm{x};\omega_{i}),$ (2) $$

## 2 Preliminary

### 2.1 Notation

Throughout this paper, we use boldface letters to denote vectors, *i.e.*, $\bm{x}\in\mathbb{R}^{d}$, with $\bm{x}^{(j)}$ representing the $j$-th coordinate of the vector $\bm{x}$. $\|\cdot\|_{1}$ and $\|\cdot\|_{2}$ denote the $\ell_{1}$-norm and $\ell_{2}$-norm, respectively. The notation ${\mathbb{E}}[\cdot]$ represents the expectation operator, and $\circ$ denotes the Hadamard product (element-wise product). All divisions and square roots are computed element-wise.

In this paper, the optimizer aims to minimize the empirical risk loss of a model on a dataset, *i.e.*,

$$ $\min_{\bm{x}\in\mathbb{R}^{d}}F(\bm{x})={\mathbb{E}}_{\zeta\sim\mathcal{D}}[f( \bm{x};\zeta)]=\frac{1}{n}\sum_{i=1}^{n}f(\bm{x};\omega_{i}),$ (2) $$

where $\bm{x}\in\mathbb{R}^{d}$ and $\zeta$ are independently and identically sampled from the dataset $\{\omega_{i}:\omega_{i}\in\mathcal{D},1\leq i\leq n\}$. For simplicity, we sometimes use $\bm{g}=\nabla f(\bm{x};\zeta)$.

| \tabucline[1pt]-<br>Algorithm 1. Adam |
| --- |
| \tabucline[0.6pt]-<br>1: Input: the momentum $\bm{m}_{0}=0$ , $\bm{v}_{0}=0$, the numerical stable constant $\epsilon$, the exponential moving average coefficient $\beta_{1}$ and $\beta_{2}$, and the learning rate $\gamma$. |
| 2: for $t=0,...,T-1$ do |
| 3:     Randomly sample data and compute the gradient: $\bm{g}_{t}\leftarrow\nabla f(\bm{x}_{t};\zeta_{t})$ |
| 4:     Update the momentum $\bm{m}_{t}$: $\bm{m}_{t}\leftarrow\beta_{1}\bm{m}_{t-1}+(1-\beta_{1})\bm{g}_{t}$ |
| 5:     Update the momentum $\bm{v}$: $\bm{v}_{t}\leftarrow\beta_{2}\bm{v}_{t-1}+(1-\beta_{2})\bm{g}_{t}^{2}$ |
| 6:     Compute the bias corrected $\bm{\hat{m}}_{t}$: $\bm{\hat{m}}_{t}\leftarrow\frac{\bm{m}_{t}}{1-\beta_{1}^{t}}$ |
| 7:     Compute the bias corrected $\bm{\hat{v}}_{t}$: $\bm{\hat{v}}_{t}\leftarrow\frac{\bm{v}_{t}}{1-\beta_{2}^{t}}$ |
| 8:     Update the model parameter: $\bm{x}_{t+1}\leftarrow\bm{x}_{t}-\gamma\frac{\bm{\hat{m}}_{t}}{\sqrt{\bm{\bm{v<br>}}_{t}}+\epsilon}$ |
| 9: end for |
| \tabucline[1pt]– |

### 2.2 Details of Adam

To facilitate the analysis of Adam, we provide the details of Adam in Algorithm 1.

Following previous studies (Adam_type_proof_2018,; Simple_Adam_proof_2020,; Adam_proof_2022,; Adaptivity_Adam_proof_2024,), we omit the bias correction in Line 6-7 of Algorithm 1 for simplicity when analyzing the convergence rate.

### 2.3 Assumptions and Conditions

To analyze the convergence rate of Adam, we list the main assumption as follows.

Assumption A [Bounded Infimum]. * There exists a constant $F^{*}>-\infty$, and the objective function follows $F(\bm{x})\geq F^{*}$ for any $\bm{x}\in{\mathbb{R}}^{d}$. *

Assumption B.1 [$L$-Smoothness] *There exists a constants $L\geq 0$, and then for any $\bm{x},\bm{y}\in{\mathbb{R}}^{d}$, the gradient of the objective function follows*

$$ $\small\|\nabla F(\bm{y})-\nabla F(\bm{x})\|_{2}\leq L\|\bm{x}-\bm{y}\|_{2}.$ (3) $$

Assumption B.2 [$(L_{0},L_{1})$-Smoothness] *There exist constants $L_{0},L_{1}\geq 0$, and then for any $\bm{x},\bm{y}\in{\mathbb{R}}^{d}$ , the gradient of the objective function follows*

$$ $\small\|\nabla F(\bm{y})-\nabla F(\bm{x})\|_{2}\leq(L_{0}+L_{1}\|\nabla F(\bm{ x})\|_{2})\|\bm{x}-\bm{y}\|_{2}.$ (4) $$

Assumption B.3 [ $(L_{0},L_{1},q)$-Smoothness] *There exist constants $L_{0},L_{1}>0$ and $0\leq q\leq 1$, and then for any $\bm{x},\bm{y}\in{\mathbb{R}}^{d}$ , the gradient of the objective function follows*

$$ $\small\|\nabla F(\bm{y})-\nabla F(\bm{x})\|_{2}\leq(L_{0}+L_{1}\|\nabla F(\bm{ x})\|_{2}^{q})\|\bm{x}-\bm{y}\|_{2}.$ (5) $$

When $q=1$, the generalized $(L_{0},L_{1},q)$-smoothness (Assumption B.3) is reduced to the $(L_{0},L_{1})$-smoothness (Assumption B.2). When $L_{1}=0$ or $q=0$, the generalized $(L_{0},L_{1},q)$-smoothness (Assumption B.3) is reduced to the standard $L$-smoothness (Assumption B.1). $(L_{0},L_{1})$-smoothness was originally defined in (Clipped_SGD_Proof_2019,) as a bound on the second-order Hessian function. Following Imprv_clipping_SGD_2020, we reformulate the $(L_{0},L_{1})$-smoothness as an affine form of the gradient norm for first-order differentiable functions. Subsequently, Relaxed_Adam_proof_2024 first introduced the generalized $(L_{0},L_{1},q)$-smoothness to analyze the convergence of Adam, followed by Separa_Adam_proof_2024 and Adam_weak_assum_2025.

Assumption C.1 [ Bounded Variance]. *There exists a positive constant $\sigma_{0}>0$, and then for any $\bm{x}_{t}\in{\mathbb{R}}^{d}$, the noisy gradient of the objective function obeys*

$$ $\small{\mathbb{E}}[\nabla f(\bm{x};\zeta)]=\nabla F(\bm{x}),~{}~{}~{}~{}~{}~{} {\mathbb{E}}[\|\nabla f(\bm{x};\zeta)-\nabla F(\bm{x})\|_{2}^{2}]\leq\sigma_{0 }^{2}.$ (6) $$

Assumption C.2 [ Affine Variance]. *There exist constants $\sigma_{0},\sigma_{1}\geq 0$, and then for $\bm{x}\in{\mathbb{R}}^{d}$, the noisy gradient of the objective function obeys*

$$ $\small{\mathbb{E}}[\nabla f(\bm{x};\zeta_{t})]=\nabla F(\bm{x}),~{}~{}~{}~{}~{ }~{}{\mathbb{E}}[\|\nabla f(\bm{x};\zeta_{t})-\nabla F(\bm{x})\|_{2}^{2}]\leq \sigma_{0}^{2}+\sigma_{1}^{2}\|\nabla F(\bm{x})\|_{2}^{2}.$ (7) $$

Assumption C.3 [ $p$-Affine Variance]. *There exist constants $\sigma_{0},\sigma_{1}\geq 0$ and $0\leq p\leq 2$, and then $\bm{x}\in{\mathbb{R}}^{d}$ at any time, the noisy gradient of the objective function obeys*

$$ $\small{\mathbb{E}}[\nabla f(\bm{x};\zeta_{t})]=\nabla F(\bm{x}),~{}~{}~{}~{}~{ }~{}{\mathbb{E}}[\|\nabla f(\bm{x};\zeta_{t})-\nabla F(\bm{x})\|_{2}^{2}]\leq \sigma_{0}^{2}+\sigma_{1}^{2}\|\nabla F(\bm{x})\|_{2}^{p}.$ (8) $$

When $p=2$, the $p$-affine variance (Assumption C.3) is reduced to the affine variance (Assumption C.2). When $\sigma_{1}=0$ or $p=0$, the $p$-affine variance (Assumption C.3) is reduced to the bounded variance (Assumption C.1). The affine variance (Assumption C.2) was originally studied in (SGD_affine_var_2000,) to analyze the convergence behavior of SGD. It was later applied to analyze AdaGrad (AdaGrad-Norm_affine_var_2022,; AdaGrad_affine_var_2023,) and a simplified Adam (Separa_Adam_proof_2024,).

To the best of our knowledge, Assumption B.3 and Assumption C.3 are the weakest assumptions for analyzing the convergence of Adam among the existing literatures.

In addition, we also list the following required conditions.

Condition 1 *At any $t$-th iteration in Algorithm 1, the gradients satisfy $\sqrt{\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}^{2}}\leq\frac{C_
{0}}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}$ with $1\leq C_{0}\ll\sqrt{T}$.*

Condition 2 *At any $t$-th iteration in Algorithm 1, the coordinates of the update in Adam , i.e., $u_{t}=\frac{|\bm{m}_{t}^{(j)}|}{\sqrt{\bm{v}_{t}^{(j)}}+\epsilon}(j\in[d])$ are independently and identically distributed (i.i.d). *

Condition 3 * At any $t$-th iteration in Algorithm 1, the gradient satisfies $\|\nabla F(\bm{x}_{t})\|_{1}=\frac{\sqrt{d}}{{C_{1}}}\|\nabla F(\bm{x})\|_{2}$ with $1\leq C_{1}\ll\sqrt{d}$.*

Condition 1 is easily satisfied when the gradients $\|\nabla F(\bm{x}_{t})\|_{2}$ decrease at a rate of ${\cal O}\left(\frac{1}{t^{\alpha}}\right)$ for all $0<\alpha<\frac{1}{2}$, and the number of iterations is sufficiently large. We summarize this in the following proposition.

###### Proposition 2.1

If $\|\nabla F(\bm{x}_{t})\|_{2}$ decreases at the rate of ${\cal O}\left(\frac{1}{t^{\alpha}}\right)$ for $0\leq\alpha<\frac{1}{2}$ and $T\geq 8$, then it holds that

$$ $\small\frac{\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}^{2}}{\left (\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}\right)^{2}}\leq{\cal O }\left(\frac{2(1-\alpha)^{2}}{1-2\alpha}\right).$ (9) $$

SGD_lower_bound_2023 has demonstrated that the optimal convergence rate of $\|\nabla F(\bm{x}_{t})\|_{2}$ in non-convex stochastic optimization is $O\left(\frac{1}{T^{\nicefrac{{1}}{{4}}}}\right)$, which lies in the range $[0,\frac{1}{2})$. We choose a sufficiently large $T$ in practice, meaning $T$ is greatly larger than $\frac{2(1-\alpha)^{2}}{1-2\alpha}$. Therefore, Condition 1 commonly holds in practice.

Condition 2 commonly holds in practice, and we empirically validated it in our experiments, as shown in Figure [1](https://arxiv.org/html/2507.05966v1#S2.F1). Specifically, we employed Adam to train ResNet-50 on ImageNet and GPT-2 (350M) on OpenWebText. During training, we recorded $\nicefrac{{\bm{m}_{t}^{(j)}}}{{\sqrt{\bm{v}_{t}^{(j)}}}}$ for each coordinate in certain layers. To verify whether $\nicefrac{{|\bm{m}_{t}^{(j)}|}}{{\sqrt{\bm{v}_{t}^{(j)}}}}$ for each coordinate is drawn from the same distribution, we used the two-sample Kolmogorov-Smirnov (KS) test.
In this test, two groups of 10,000 samples were uniformly drawn from all coordinates of the layer, and these groups were used to run the two-sample KS test. We repeated this procedure 1,000 times and reported the mean $p$-value. As illustrated in Figure [1](https://arxiv.org/html/2507.05966v1#S2.F1), the mean $p$-value is significantly larger than the significance level of 0.05, strongly suggesting that $\nicefrac{{\bm{m}_{t}^{(j)}}}{{\sqrt{\bm{v}_{t}^{(j)}}}}$ for each coordinate in the layers is independently drawn from an identical distribution.

Figure: (a) ResNet-50, Layer#32
Refer to caption: x1.png

Condition 3 commonly holds in practice, and we also empirically validated it in our experiments, as shown in Figure [2](https://arxiv.org/html/2507.05966v1#S2.F2). Specifically, we employed Adam to train ResNet-50 on ImageNet and GPT-2 (350M) on OpenWebText. During training, we recorded the gradient $\nabla f(\bm{x}_{t})$ for each coordinate in selected layers. Subsequently, we computed $C_{1}=\frac{\sqrt{d}\|\nabla f(\bm{x}_{t})\|_{2}}{\|\nabla f(\bm{x}_{t})\|_{1}}$. As shown in Figure [2](https://arxiv.org/html/2507.05966v1#S2.F2), $C_{1}$ consistently remains below 3 throughout training, which is significantly smaller than $\sqrt{d}$, where $d$ represents the number of coordinates in the layers. This observation can be attributed to the fact that the coordinates of $\nabla f(\bm{x}_{t})$ tend to be densely clustered during training, as also depicted in Figure [2](https://arxiv.org/html/2507.05966v1#S2.F2).

Figure: (a) ResNet-50, Layer#32
Refer to caption: x5.png

###### Proposition 2.1

## 3 Main Result

In this section, we present our results on the convergence of Adam under two scenarios: 1). Assumption B.3 ($(L_{0},L_{1},q)$-smoothness) without reliance on Conditions 1-3. 2). Assumptions B.3 ($(L_{0},L_{1},q)$-smoothness) and C.3 ($p$-affine variance) with the inclusion of Conditions 1-3.

We first state the preliminary result in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10).

###### Theorem 3.1

Let $\{x_{t}\}_{t=0}^{T-1}$ be generated by Algorithm 1. Suppose that Assumptions A, B.3, and C.3, along with Condition 1, hold. Define $u_{t}^{(j)}:=\nicefrac{{|\bm{m}_{t}^{(j)}|}}{{(\sqrt{\bm{v}_{t}^{(j)}}+
\epsilon)}}$, $R:=\nicefrac{{(1-\beta_{1})}}{{\sqrt{(1-\beta_{2})(1-\frac{\beta_{1}^{2}}{
\beta_{2}}})}}$ , $\hat{L}:=L_{0}+(1-q)L_{1}$, and $\hat{\sigma}:=\sigma_{0}+\sqrt{\frac{2-p}{2}}$. Choose $\beta_{1}<\sqrt{\beta_{2}}$. Then, it holds for any $T\in{\mathbb{N}^{+}}$,

$$ $\displaystyle\frac{1}{T}\left(\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ \nabla F(\bm{x}_{t})\|_{1}]-\left(\frac{\gamma R^{2}dqL_{1}}{2}+{{2}C_{0}R \sqrt{d}\sigma_{1}\sqrt{p(1-\beta_{1})}}+\frac{2\gamma R^{2}{d}qL_{1}}{1-\beta _{1}}\right)\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2}]\right)$ (10) $\displaystyle\leq\frac{F(\bm{x}_{0})-F^{*}}{\gamma T}+\frac{2R\sqrt{d}\left\| \nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}+{2\sqrt{1-\beta_{1}}R\sqrt{d }}\hat{\sigma}+\frac{2\gamma R^{2}d\hat{L}}{1-\beta_{1}}+\frac{\gamma R^{2}{d} \hat{L}}{2T}.$ $$

When Assumption C.3 is replaced with Assumption C.1, Condition 1 becomes unnecessary, as detailed in the proof of Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10). Building on the general result established in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), we derive specific convergence bounds for the Adam algorithm under Assumptions A, B.3, and C.1, without requiring Conditions 1, 2, or 3.

###### Corollary 3.2 (Convergence of Adam without Condition 1-3)

Let $\{x_{t}\}_{t=0}^{T-1}$ be generated by Algorithm 1. Suppose that Assumptions A, B.3 and C.1 hold. Define $u_{t}^{(j)}:=\nicefrac{{|\bm{m}_{t}^{(j)}|}}{{(\sqrt{\bm{v}_{t}^{(j)}}+
\epsilon)}}$, $R:=\nicefrac{{1-\beta_{1}}}{{\sqrt{(1-\beta_{2})(1-\nicefrac{{\beta_{1}^{2}}}{
{\beta_{2}}}})}}$ and $\hat{L}:=L_{0}+(1-q)L_{1}$. Choose $\gamma=\frac{C_{2}}{T^{3/4}d^{\nicefrac{{1}}{{2}}}}$, $\beta_{1}<\sqrt{\beta_{2}}$, $1-\beta_{1}=\frac{C_{3}}{T^{\nicefrac{{1}}{{2}}}}$ and $0<v\leq\min_{t,j}u_{t}^{(j)}$. Then, it holds for any $T\in{\mathbb{N}^{+}}$ and $T\geq(\frac{4C_{2}R^{2}d^{\nicefrac{{1}}{{2}}}qL_{1}}{C_{3}v}+(\frac{C_{2}R^{2
}d^{\nicefrac{{1}}{{2}}}qL_{1}}{v})^{\nicefrac{{1}}{{3}}})^{4}$,

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{1 }]\leq$ $\displaystyle\frac{1}{v}\left(\frac{2(F(\bm{x}_{0})-F(\bm{x}^{*}))}{C_{2}T^{ \nicefrac{{1}}{{4}}}d^{\nicefrac{{1}}{{2}}}}+\frac{4Rd^{\nicefrac{{1}}{{2}}} \left\|\nabla F(\bm{x}_{0})\right\|_{2}}{C_{3}T^{\nicefrac{{1}}{{2}}}}\right.$ (11) $\displaystyle+$ $\displaystyle\left.\frac{4C_{3}Rd^{\nicefrac{{1}}{{2}}}\sigma_{0}}{T^{ \nicefrac{{1}}{{4}}}}+\frac{4C_{2}R^{2}d^{\nicefrac{{1}}{{2}}}\hat{L}}{C_{3}T^ {\nicefrac{{1}}{{4}}}}+\frac{C_{2}R^{2}d^{\nicefrac{{1}}{{2}}}\hat{L}}{T^{ \nicefrac{{7}}{{4}}}}\right).$ $$

Given the generic result in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), we derive the more comprehensive convergence bound of Adam under assumptions of Assumptions A, B.3 and C.3 with Condition 1, 2 and 3.

###### Corollary 3.3 (Convergence of Adam with Condition 1-3)

Let $\{x_{t}\}_{t=0}^{T-1}$ be generated by Algorithm 1. Suppose Assumptions A, B.3 and C.3, along Conditions 1,2 and 3, hold. Define $u_{t}:=\nicefrac{{|\bm{m}_{t}^{(j)}|}}{{(\sqrt{\bm{v}_{t}^{(j)}}+\epsilon)}}$, $R:=\nicefrac{{1-\beta_{1}}}{{\sqrt{(1-\beta_{2})(1-\nicefrac{{\beta_{1}^{2}}}{
{\beta_{2}}}})}}$, $\hat{L}:=L_{0}+(1-q)L_{1}$, and $\hat{\sigma}:=\sigma_{0}+\sqrt{\frac{2-p}{2}}$.

Case 1: General Setting with no Access to Oracles

Choose $\gamma=\frac{{C}_{2}}{T^{3/4}d^{\nicefrac{{1}}{{2}}}}$, $\beta_{1}<\sqrt{\beta_{2}}$, $1-\beta_{1}=\frac{C_{3}}{T^{\nicefrac{{1}}{{2}}}}$, and $0<\bar{v}\leq\min_{t}\mathbb{E}[\bm{u}_{t}^{(j)}]$. Then, it holds for any $T\in{\mathbb{N}^{+}}$ and $T\geq(\frac{4C_{0}C_{1}{C}_{3}{\nicefrac{{1}}{{2}}}p^{\nicefrac{{1}}{{2}}}R
\sigma_{1}}{\bar{v}}+\frac{4C_{1}{C_{2}}R^{2}qL_{1}}{{C}_{3}\bar{v}}+(\frac{C_
{1}{C}_{2}R^{2}qL_{1}}{\bar{v}})^{\nicefrac{{1}}{{3}}})^{4}$,

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2 }]\leq$ $\displaystyle\frac{C_{1}}{\bar{v}}\left(\frac{2(F(\bm{x}_{0})-F(\bm{x}^{*}))}{ C_{2}T^{\nicefrac{{1}}{{4}}}d^{\nicefrac{{1}}{{2}}}}+\frac{4R\left\|\nabla F( \bm{x}_{0})\right\|_{2}}{C_{3}T^{\nicefrac{{1}}{{2}}}}+\frac{4C_{3}R\hat{ \sigma}}{T^{\nicefrac{{1}}{{4}}}}+\frac{4C_{2}R^{2}\hat{L}}{C_{3}T^{\nicefrac{ {1}}{{4}}}}+\frac{C_{2}R^{2}\hat{L}}{T^{\nicefrac{{7}}{{4}}}}\right).$ (12) $$

Case 2: Lowest-Bound Setting with Access to Oracles

Choose $\hat{C}_{2}=\frac{(F(\bm{x}_{0})-F^{*})^{\nicefrac{{3}}{{4}}}}{2^{\nicefrac{{1
}}{{4}}}R\hat{\sigma}^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}}$, $\hat{C}_{3}=\frac{2^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{2}}}(F(\bm{x
}_{0})-F^{*})^{\nicefrac{{1}}{{2}}}}{\hat{\sigma}}$, $\gamma=\frac{\hat{C}_{2}}{T^{3/4}d^{\nicefrac{{1}}{{2}}}}$, $\beta_{1}<\sqrt{\beta_{2}}$, $1-\beta_{1}=\frac{\hat{C}_{3}}{T^{\nicefrac{{1}}{{2}}}}$, and $0<\bar{v}\leq\min_{t}\mathbb{E}[\bm{u}_{t}^{(j)}]$. Then, for any $T\in{\mathbb{N}^{+}}$ and $T\geq(\frac{4C_{0}C_{1}\hat{C}_{3}{\nicefrac{{1}}{{2}}}p^{\nicefrac{{1}}{{2}}}
R\sigma_{1}}{\bar{v}}+\frac{4C_{1}\hat{C_{2}}R^{2}qL_{1}}{\hat{C}_{3}\bar{v}}+
(\frac{C_{1}\hat{C}_{2}R^{2}qL_{1}}{\bar{v}})^{\nicefrac{{1}}{{3}}})^{4}$ the right-hand side of Eq. ([10](https://arxiv.org/html/2507.05966v1#S3.E10)) reaches the lowest bound, *i.e.*,

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2 }]\leq$ $\displaystyle\frac{C_{1}}{\bar{v}}\left(\frac{512^{\nicefrac{{1}}{{4}}}R\hat{ \sigma}^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}(F(\bm{x}_{0})-F^{*} )}{T^{\nicefrac{{1}}{{4}}}}+\frac{2C_{1}\hat{C}_{3}R\left\|\nabla F(\bm{x}_{0} )\right\|_{2}}{T^{\nicefrac{{3}}{{4}}}}+\frac{\hat{C_{2}}R^{2}\hat{L}}{T^{ \nicefrac{{7}}{{4}}}}\right).$ (13) $$

We have some observations from Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), Corollary [11](https://arxiv.org/html/2507.05966v1#S3.E11) and Corollary [13](https://arxiv.org/html/2507.05966v1#S3.E13) below.

Finding 1. To the best of our knowledge, we are the first to formally analyze Adam under the weak assumptions of generalized non-uniform $(L_{0},L_{1},q)$-smoothness (Assumption B.3) and $p$-affine variance (Assumption C.3). We also prove that $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2}]$ for Adam achieves a tighter bound of ${\cal O}\left(\frac{1}{T^{\nicefrac{{1}}{{4}}}}\right)$, compared to the previous ${\cal O}\left(\frac{\ln T}{T^{\nicefrac{{1}}{{4}}}}\right)$ (Adam_type_proof_2018,; Simple_Adam_proof_2020,; Prac_Adam_proof_2022,; Relaxed_Adam_proof_2024,). Furthermore, earlier works demonstrated that Adam’s convergence rates were dependent on the model dimensionality $d$ and the numerical-stability $\epsilon$ (Adam_type_proof_2018,; Simple_Adam_proof_2020,; Prac_Adam_proof_2022,; Relaxed_Adam_proof_2024,), which makes them unsuitable to analyze large-scale LLM training. However, as shown in Corollary [13](https://arxiv.org/html/2507.05966v1#S3.E13), we prove that Adam achieves dimension-free and $\epsilon$-free convergence, similar to SGD (Opt_Review_2018,). Notably, previous studies required a learning rate of $\gamma=1-O(\frac{1}{T})$ to reach the optimal convergence rate of ${\cal O}\left(\frac{\ln T}{T^{\nicefrac{{1}}{{4}}}}\right)$. This causes $\bm{v}_{t}$ to closely resemble a plain average of the past $T$ squared gradients, reducing Adam almost to AdaGrad (Simple_Adam_proof_2020,). By contrast, we only require the learning rate to satisfy $\gamma=1-O(\frac{1}{T^{\nicefrac{{3}}{{4}}}})$, which better aligns with Adam’s original design.

Finding 2. In the proof of Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), Condition 1 is no longer necessary when Assumption C.1 is used. As demonstrated in Corollary [11](https://arxiv.org/html/2507.05966v1#S3.E11), even in the absence of Conditions 1, 2, and 3, and assuming no access to the oracle values $L_{0}$, $L_{1}$, $\sigma_{0}$, and $\sigma_{1}$, we can still theoretically establish that $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2}]$ for Adam converges at the rate of $O\left(\frac{1}{T^{\nicefrac{{1}}{{4}}}}\right)$, under the assumptions of $(L_{0},L_{1},q)$-smoothness and affine variance.

Finding 3. The momentum coefficients $\beta_{1}$ and $\beta_{2}$ play a crucial role in Adam’s convergence. As shown in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), when $\beta_{1}=\beta_{2}=0$, Adam reduces to signSGD, which converges at best to a bounded region where $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2}]\leq O(
\frac{1}{T^{\nicefrac{{1}}{{4}}}}+\sigma_{0})$. This result aligns with prior work (signSGD_2018,). In contrast, sufficiently large values of $\beta_{1}$ and $\beta_{2}$ ensure Adam achieves the convergence rate of $O(\frac{1}{T^{\nicefrac{{1}}{{4}}}})$. For SGD, however, momentum has minimal impact on the optimal theoretical convergence rate, as both momentum-SGD and vanilla SGD converge at $O(\frac{1}{T^{\nicefrac{{1}}{{4}}}})$ (Opt_Review_2018,; SGDM_convergence_2020,).

Finding 4. Case 2 in Corollary [13](https://arxiv.org/html/2507.05966v1#S3.E13) states that Adam’s learning rate must satisfy $\gamma=O(\frac{1}{\sqrt{d}})$ to achieve the optimal convergence rate. This implies that, with fixed other hyperparameters, larger model sizes require smaller optimal learning rates. This observation has been empirically validated by practitioners training the Llama family of models across different sizes (Llama2023,; llama2_2023,; llama3_2024,). Interestingly, a similar theoretical conclusion appears in the work on Maximal Update Parametrization ($\mu P$) (MuP_2022,).

###### Theorem 3.1

###### Corollary 3.2 (Convergence of Adam without Condition 1-3)

###### Corollary 3.3 (Convergence of Adam with Condition 1-3)

## 4 Proof Sketch

In this section, we present the core ideas underlying the convergence proofs for Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10) and Corollary [13](https://arxiv.org/html/2507.05966v1#S3.E13). The proof ideas for Corollary [11](https://arxiv.org/html/2507.05966v1#S3.E11) are similar, and thus, we omit the details for simplicity.

Our main contribution lies in opening up a new approach to proving the convergence of Adam. All existing theoretical convergence proofs follow a path-dependent approach, treating Adam as a preconditioned SGD with momentum, as originally presented in the Adam paper (Adam2015,). Specifically, the update rule is defined as: *i.e.*,$\bm{x}_{t}+1=\bm{x}_{t}-\frac{\gamma_{t}}{{\sqrt{\bm{v}_{t}}+\epsilon}}\circ
\bm{m}_{t}$, where ${\sqrt{\bm{v}_{t}}+\epsilon}$ is used to precondition $\bm{m}_{t}$, and the effective learning rate is $\frac{\gamma_{t}}{\sqrt{\bm{v}_{t}}+\epsilon}$. This approach, however, encounters two intractable issues: ($i$) the effective learning rate $\frac{\gamma_{t}}{\sqrt{\bm{v}_{t}}+\epsilon}$ is not necessarily monotone-decreasing, and ($ii$) the random variable $\bm{v}_{t}$ is not independent of $\bm{g}_{t}$ or $\bm{m}_{t}$. To address these challenges, the proofs in previous works became complicated, lengthy, and opaque, making them difficult to verify and extend. In contrast, we treat Adam as a whole stochastic sign-like descent algorithm, *i.e.*, $\bm{x}_{t+1}=\bm{x}_{t}-\gamma_{t}\frac{|\bm{m}_{t}|}{{\sqrt{\bm{v}_{t}}+
\epsilon}}\circ{\rm Sign}(\bm{m}_{t})$ where we consider the term $\frac{|\bm{m}_{t}|}{\sqrt{\bm{v}_{t}}+\epsilon}$ as a single random variable. This transformation not only circumvents the problems mentioned above but also simplifies the proof process. We now provide a sketch of the proof.

Under Assumption B.3, we obtain (details please refer to Lamma 1 in the Appendix):

$$ $\small{\mathbb{E}}[F(\bm{x}_{t+1})]\leq F(\bm{x}_{t})+{\mathbb{E}}[\langle \nabla F(\bm{x}_{t}),\bm{x}_{t+1}-\bm{x}_{t}\rangle]+\frac{L_{0}+L_{1}\|\nabla F (\bm{x}_{t})\|_{2}^{q}}{2}{\mathbb{E}}[\|\bm{x}_{t+1}-\bm{x}_{t}\|_{2}^{2}].$ (14) $$

Defining $\bm{u}_{t}:=\frac{|\bm{m}_{t}|}{\sqrt{\bm{v}_{t}}+\epsilon}$, the update rule becomes $\bm{x}_{t+1}=\bm{x}_{t}-\gamma\frac{\bm{m}_{t}}{\sqrt{\bm{v}_{t}}+\epsilon}=
\bm{x}_{t}-\gamma\frac{|\bm{m}_{t}|}{\sqrt{\bm{v}_{t}}+\epsilon}\circ\frac{\bm
{m}_{t}}{|\bm{m}_{t}|}=\bm{x}_{t}-\gamma\bm{u}_{t}\circ{\rm Sign}(\bm{m}_{t})$. We further have

$$ $\displaystyle{\mathbb{E}}[F(\bm{x}_{t+1})]\leq$ $\displaystyle F(\bm{x}_{t})-\gamma{\mathbb{E}}[\|\bm{u}_{t}\nabla F(\bm{x}_{t} )\|_{1}]+\underbrace{\gamma{\mathbb{E}}[\left\langle\nabla F(\bm{x}_{t}),\bm{u }_{t}\circ({\rm Sign}(\nabla F(\bm{x}_{t}))-{\rm Sign}(\bm{m}_{t}))\right \rangle]}_{{\cal T}_{1}}$ (15) $\displaystyle+\underbrace{\frac{\gamma^{2}(L_{0}+L_{q}\|\nabla F(\bm{x}_{t})\| _{2}^{q})}{2}{\mathbb{E}}[\|\bm{u}_{t}\|_{2}^{2}]}_{{\cal T}_{2}}.$ $$

Next, we define $R:=\frac{1-\beta_{1}}{\sqrt{(1-\beta_{2})(1-\nicefrac{{\beta_{1}^{2}}}{{\beta_
{2}}}})}$, and by applying Lemma 2 in the appendix, we obtain that $\bm{u}_{t}^{(j)}\leq R$. Furthermore, Lemma 3 in the appendix indicates ${\mathbb{E}}[|{\rm Sign}(\nabla F(\bm{x}_{t}^{(j)}))-{\rm Sign}(\bm{m}_{t}^{(j
)})|]\leq 2\frac{{\mathbb{E}}[|\nabla F(\bm{x}_{t}^{(j)})-\bm{m}_{t}^{(j)}|]}{
|\nabla F(\bm{x}_{t}^{(j)})|}$, which leads to ${\cal T}_{1}\leq 2\gamma R\sqrt{d}{\mathbb{E}}[\|\nabla F(\bm{x}_{t})-\bm{m}_{
t}\|_{2}]$.

Employing the bound $\bm{u}_{t}^{(j)}\leq R$ above and applying Young’s inequality, we obtain ${\cal T}_{2}\leq\frac{\gamma^{2}R^{2}d(L_{0}+L_{1}((1-q)+q{\mathbb{E}}[\|
\nabla F(\bm{x}_{t})\|_{2}]))}{2}$.

By taking the expectation over the first to the $(T-1)$-th iteration, and then summing and rearranging the terms, we obtain:

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ\nabla F (\bm{x}_{t})\|_{1}]-\frac{\gamma R^{2}qL_{1}{d}}{2T}\sum_{t=0}^{T-1}\mathbb{E} [\|\nabla F(\bm{x}_{t})\|_{2}]$ (16) $\displaystyle\leq$ $\displaystyle\frac{F(\bm{x}_{0})-F^{*}}{\gamma T}+\frac{2R\sqrt{d}}{T}\sum_{t= 0}^{T-1}{\mathbb{E}}[\|\bm{m}_{t}-\nabla F(\bm{x}_{t})\|_{2}]+\frac{\gamma R^{ 2}{d}(L_{0}+(1-q)L_{1})}{2T}.$ $$

We now divide-and-conquer prove that

$$ $\displaystyle\frac{1}{T}\sum_{t=1}^{T}{\mathbb{E}}\left[\|\bm{m}_{t}-\nabla F( \bm{x}_{t})\|_{2}\right]\leq$ $\displaystyle\frac{\left\|\nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}+{ \sqrt{1-\beta_{1}}}\left(\sigma_{0}+\sqrt{\frac{2-p}{2}}\sigma_{1}\right)$ (17) $\displaystyle+$ $\displaystyle{C_{0}\sigma_{1}\sqrt{p(1-\beta_{1})}}\cdot\frac{1}{T}\sum_{t=0}^ {T-1}\|\nabla F(x_{t})\|_{2}$ $\displaystyle+$ $\displaystyle\frac{\gamma R\sqrt{d}(L_{0}+(1-q)L_{1})}{1-\beta_{1}}+\frac{ \gamma R\sqrt{d}qL_{1}}{1-\beta_{1}}\cdot\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E }}[\|\nabla F(\bm{x}_{t})\|_{2}]$ $$

Then, this leads to the conclusion in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), *i.e.*,

$$ $\displaystyle\frac{1}{T}\left(\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ \nabla F(\bm{x}_{t})\|_{1}]-\left(\frac{\gamma R^{2}dqL_{1}}{2}+{{2}C_{0}R \sqrt{d}\sigma_{1}\sqrt{p(1-\beta_{1})}}+\frac{2\gamma R^{2}{d}qL_{1}}{1-\beta _{1}}\right)\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2}]\right)$ (18) $\displaystyle\leq\frac{F(\bm{x}_{0})-F^{*}}{\gamma T}+\frac{2R\sqrt{d}\left\| \nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}+{2\sqrt{1-\beta_{1}}R\sqrt{d }}\hat{\sigma}+\frac{2\gamma R^{2}d\hat{L}}{1-\beta_{1}}+\frac{\gamma R^{2}{d} \hat{L}}{2T}.$ $$

where $\hat{L}:=L_{0}+(1-q)L_{1}$, and $\hat{\sigma}:=\sigma_{0}+\sqrt{\frac{2-p}{2}}$.

By choosing $\bar{v}=\min_{t}\mathbb{E}[\bm{u}_{t}^{(j)}]$ and applying Condition 2 and Condition 3, we obtain

$$ $\small\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ\nabla F(\bm{x}_{t})\|_{1} ]=\sum_{t=0}^{T-1}{\mathbb{E}}[\bm{u}_{t}^{(j)}]{\mathbb{E}}[\|\nabla F(\bm{x} _{t})\|_{1}]\geq\bar{v}\sum_{t=0}^{T-1}{\mathbb{E}}[\|(\nabla F(\bm{x}_{t}))\| _{1}]=\frac{\bar{v}\sqrt{d}}{C_{1}}\sum_{t=0}^{T-1}{\mathbb{E}}[\|(\nabla F( \bm{x}_{t}))\|_{2}].$ (19) $$

Next, by choosing $\gamma=\frac{C_{2}}{T^{\nicefrac{{3}}{{4}}}d^{\nicefrac{{1}}{{2}}}}$, $1-\beta_{1}=\frac{C_{3}}{T^{\nicefrac{{1}}{{2}}}}$ and setting $T\geq(\frac{4C_{2}R^{2}qL_{1}}{C_{3}\bar{v}}+{4}C_{0}C_{1}\sqrt{C_{3}}R\sigma_
{1}\sqrt{p}+(\frac{C_{2}R^{2}qL_{1}}{\bar{v}})^{\nicefrac{{1}}{{3}}})^{4}$, applying following Lemma 4 in the appendix, we have

$$ $\small\frac{\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{2}+{{2}C_{0}C_{1}R\sigma_{1}\sqrt {p(1-\beta_{1})}}+\frac{2\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{1-\beta_{1}}\leq \frac{\bar{v}}{2}.$ (20) $$

Then, we arrive Case 1 of Corollary [13](https://arxiv.org/html/2507.05966v1#S3.E13), *i.e.*,

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2 }]\leq$ $\displaystyle\frac{C_{1}}{\bar{v}}\left(\frac{2L_{1}(F(\bm{x}_{0})-F(\bm{x}^{* }))}{C_{2}T^{\nicefrac{{1}}{{4}}}d^{\nicefrac{{1}}{{2}}}}+\frac{4R\left\| \nabla F(\bm{x}_{0})\right\|_{2}}{C_{3}T^{\nicefrac{{1}}{{2}}}}+\frac{4C_{3}R \hat{\sigma}}{T^{\nicefrac{{1}}{{4}}}}+\frac{4C_{2}R^{2}\hat{L}}{C_{3}T^{ \nicefrac{{1}}{{4}}}}+\frac{C_{2}R^{2}\hat{L}}{T^{\nicefrac{{7}}{{4}}}}\right).$ (21) $$

Using generalized Young’s inequality, we minimize the bottleneck terms to achieve the lowest bound on the right-hand side of Eq. ([10](https://arxiv.org/html/2507.05966v1#S3.E10)) in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10), *i.e.*,

$$ $\displaystyle\frac{C_{1}(F(\bm{x}_{0})-F^{*})}{\gamma T\sqrt{d}}+{2C_{1}\sqrt{ 1-\beta_{1}}R}\hat{\sigma}+\frac{2C_{1}\gamma R^{2}\sqrt{d}\hat{L}}{1-\beta_{1 }}\geq\frac{512^{\nicefrac{{1}}{{4}}}C_{1}R\hat{\sigma}^{\nicefrac{{1}}{{2}}} \hat{L}^{\nicefrac{{1}}{{4}}}(F(\bm{x}_{0})-F^{*})}{T^{\nicefrac{{1}}{{4}}}},$ (22) $$

where the lowest bound achieved if and only if $\gamma=\frac{(F(\bm{x}_{0})-F^{*})^{\nicefrac{{3}}{{4}}}}{2^{\nicefrac{{1}}{{4
}}}T^{\nicefrac{{3}}{{4}}}d^{\nicefrac{{1}}{{2}}}R\hat{\sigma}^{\nicefrac{{1}}
{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}}$ and $1-\beta_{1}=\frac{2^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{2}}}(F(\bm{x
}_{0})-F^{*})^{\nicefrac{{1}}{{2}}}}{T^{\nicefrac{{1}}{{2}}}\hat{\sigma}}$.

Now, choosing $\hat{C}_{2}=\frac{(F(\bm{x}_{0})-F^{*})^{\nicefrac{{3}}{{4}}}}{2^{\nicefrac{{1
}}{{4}}}R\hat{\sigma}^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}}$, $\hat{C}_{3}=\frac{2^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{2}}}(F(\bm{x
}_{0})-F^{*})^{\nicefrac{{1}}{{2}}}}{\hat{\sigma}}$, $\gamma=\frac{\hat{C}_{2}}{T^{3/4}d^{\nicefrac{{1}}{{2}}}}$, $1-\beta_{1}=\frac{\hat{C}_{3}}{T^{\nicefrac{{1}}{{2}}}}$ and setting $T\geq(\frac{4C_{0}C_{1}\hat{C}_{3}{\nicefrac{{1}}{{2}}}p^{\nicefrac{{1}}{{2}}}
R\sigma_{1}}{\bar{v}}+\frac{4C_{1}\hat{C_{2}}R^{2}qL_{1}}{\hat{C}_{3}\bar{v}}+
(\frac{C_{1}\hat{C}_{2}R^{2}qL_{1}}{\bar{v}})^{\nicefrac{{1}}{{3}}})^{4}$, we arrive Case 2 of Corollary [13](https://arxiv.org/html/2507.05966v1#S3.E13), *i.e.*,

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2 }]\leq$ $\displaystyle\frac{C_{1}}{\bar{v}}\left(\frac{512^{\nicefrac{{1}}{{4}}}R\hat{ \sigma}^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}(F(\bm{x}_{0})-F^{*} )}{T^{\nicefrac{{1}}{{4}}}}+\frac{2C_{1}\hat{C}_{3}R\left\|\nabla F(\bm{x}_{0} )\right\|_{2}}{T^{\nicefrac{{3}}{{4}}}}+\frac{\hat{C_{2}}R^{2}\hat{L}}{T^{ \nicefrac{{7}}{{4}}}}\right).$ (23) $$

## 5 Conclusion

This work breaks with convention and provides a pioneering reinterpretation of Adam as a sign-like descent algorithm to analyze the convergence, simplifying its theoretical analysis and addressing limitations of the traditional preconditioned perspective. By treating $\frac{|\bm{m}_{t}|}{{\sqrt{\bm{v}_{t}}+\epsilon}}$ as a unified random variable, It is the first time to proved that Adam dimension-freely and $\epsilon$-freely achieves the optimal convergence rate of $\mathcal{O}(\frac{1}{T^{\nicefrac{{1}}{{4}}}})$ under the weak assumptions of generalized $(L_{0},L_{1},q)$-smoothness and affine variance. Additionally, we provide actionable insights for hyperparameter tuning, bridging theory and practice. This work advances the theoretical convergence analysis of adaptive optimizers and offers a foundation for further research and development.

Appendix

Limitations. While the proposed novel route for the convergence proof of Adam simplifies the theoretical analysis, extending the proof to weaker assumptions (Assumptions C.3) compared to prior work needs the introduction of additional constraints (Conditions 1–3), although these conditions are commonly satisfied in practical settings that are demonstrated in Section 3.

Broader Impacts. This paper advances the theoretical understanding of Adam’s convergence properties, which may contribute to more efficient and reliable training of deep neural networks. The main contributions of this paper are theoretical, and thus it do not inherently raise potential ethic and societal concerns.

## References

- (1)
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and
Blake Woodworth.
Lower bounds for non-convex stochastic optimization.
Mathematical Programming, 199(1-2):165–214, 2023.
- (2)
Amit Attia and Tomer Koren.
SGD with AdaGrad stepsizes: Full adaptivity with high probability
to unknown parameters, unbounded gradients and affine variance.
In International Conference on Machine Learning, pages
1147–1171. PMLR, 2023.
- (3)
Lukas Balles and Philipp Hennig.
Dissecting Adam: The sign, magnitude and variance of stochastic
gradients.
In International Conference on Machine Learning, pages
404–413, 2018.
- (4)
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree
Anandkumar.
signSGD: Compressed optimisation for non-convex problems.
In International Conference on Machine Learning, pages
560–569, 2018.
- (5)
Dimitri P Bertsekas and John N Tsitsiklis.
Gradient convergence in gradient methods with errors.
SIAM Journal on Optimization, 10(3):627–642, 2000.
- (6)
Léon Bottou, Frank E Curtis, and Jorge Nocedal.
Optimization methods for large-scale machine learning.
SIAM review, 60(2):223–311, 2018.
- (7)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:1877–1901, 2020.
- (8)
Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu.
Towards practical Adam: Non-convexity, convergence theory, and
mini-batch acceleration.
Journal of Machine Learning Research, 23(229):1–47, 2022.
- (9)
Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu.
Lion secretly solves constrained optimization: As lyapunov predicts.
arXiv preprint arXiv:2310.05898, 2023.
- (10)
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu
Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al.
Symbolic discovery of optimization algorithms.
arXiv preprint arXiv:2302.06675, 2023.
- (11)
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong.
On the convergence of a class of Adam-type algorithms for
non-convex optimization.
arXiv preprint arXiv:1808.02941, 2018.
- (12)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al.
PaLM: Scaling language modeling with pathways.
Journal of Machine Learning Research, 24(240):1–113, 2023.
- (13)
Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun
Zhuang.
Robustness to unbounded smoothness of generalized signSGD.
In Advances in neural information processing systems, pages
9955–9968, 2022.
- (14)
Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier.
A simple convergence proof of Adam and AdaGrad.
arXiv preprint arXiv:2003.02395, 2020.
- (15)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan,
et al.
The Llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.
- (16)
Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai.
Beyond uniform smoothness: A stopped analysis of adaptive SGD.
In The Thirty Sixth Annual Conference on Learning Theory, pages
89–160. PMLR, 2023.
- (17)
Matthew Faw, Isidoros Tziotis, Constantine Caramanis, Aryan Mokhtari, Sanjay
Shakkottai, and Rachel Ward.
The power of adaptivity in SGD: Self-tuning step sizes with
unbounded gradients and affine variance.
In Conference on Learning Theory, pages 313–355. PMLR, 2022.
- (18)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In IEEE Conference on Computer Vision and Pattern Recognition,
pages 770–778, 2016.
- (19)
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.
Neural networks for machine learning lecture 6a overview of
mini-batch gradient descent.
Cited on, 14(8):2, 2012.
- (20)
Yusu Hong and Junhong Lin.
High probability convergence of Adam under unbounded gradients and
affine variance noise.
arXiv preprint arXiv:2311.02000, 2023.
- (21)
Yusu Hong and Junhong Lin.
On convergence of Adam for stochastic optimization under relaxed
assumptions.
arXiv preprint arXiv:2402.03982v2, 2025.
- (22)
Florian Hübler, Junchi Yang, Xiang Li, and Niao He.
Parameter-agnostic optimization under relaxed smoothness.
In International Conference on Artificial Intelligence and
Statistics, pages 4861–4869. PMLR, 2024.
- (23)
Jikai Jin, Bohang Zhang, Haiyang Wang, and Liwei Wang.
Non-convex distributionally robust optimization: Non-asymptotic
analysis.
In Advances in Neural Information Processing Systems,
volume 34, pages 2771–2782, 2021.
- (24)
Hamed Karimi, Julie Nutini, and Mark Schmidt.
Linear convergence of gradient and proximal-gradient methods under
the polyak-łojasiewicz condition.
In Machine Learning and Knowledge Discovery in Databases:
European Conference, ECML-PKDD-2016, pages 795–811, 2016.
- (25)
Diederik Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In International Conference on Learning Representations (ICLR),
2015.
- (26)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura
Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.
Segment anything.
arXiv preprint arXiv:2304.02643, 2023.
- (27)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
Communications of the ACM, 60(6):84–90, 2017.
- (28)
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt.
Noise is not the main factor behind the gap between SGD and Adam
on transformers, but sign descent might be.
arXiv preprint arXiv:2304.13960, 2023.
- (29)
Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto
Bietti.
Heavy-tailed class imbalance and why Adam outperforms gradient
descent on language models.
arXiv preprint arXiv:2402.19449, 2024.
- (30)
Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie.
Convergence of Adam under relaxed assumptions.
Advances in Neural Information Processing Systems, 36, 2023.
- (31)
Yanli Liu, Yuan Gao, and Wotao Yin.
An improved analysis of stochastic gradient descent with momentum.
Advances in Neural Information Processing Systems,
33:18261–18271, 2020.
- (32)
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell,
and Saining Xie.
A convnet for the 2020s.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 11976–11986, 2022.
- (33)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
supervision.
In International conference on machine learning, pages
8748–8763, 2021.
- (34)
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar.
On the convergence of Adam and beyond.
In International Conference on Learning Representations, 2018.
- (35)
Martin Riedmiller and Heinrich Braun.
A direct adaptive method for faster backpropagation learning: The
rprop algorithm.
In IEEE international conference on neural networks, pages
586–591. IEEE, 1993.
- (36)
Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu.
1-bit stochastic gradient descent and its application to
data-parallel distributed training of speech DNNs.
In Conference of the International Speech Communication
Association, volume 2014, pages 1058–1062. Singapore, 2014.
- (37)
Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun.
RMSProp converges with proper hyperparameter.
In International Conference on Learning Representation, 2021.
- (38)
Nikko Ström.
Scalable distributed DNN training using commodity GPU cloud
computing.
In Conference of the International Speech Communication
Association, 2015.
- (39)
Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang.
Momentum ensures convergence of signSGD under weaker assumptions.
In International Conference on Machine Learning, pages
33077–33099, 2023.
- (40)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.
- (41)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023.
- (42)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.
- (43)
Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen.
Convergence of AdaGrad for non-convex objectives: Simple proofs and
relaxed assumptions.
In Conference on Learning Theory, pages 161–190. PMLR, 2023.
- (44)
Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen.
Convergence of AdaGrad for non-convex objectives: Simple proofs and
relaxed assumptions.
In The Thirty Sixth Annual Conference on Learning Theory, pages
161–190. PMLR, 2023.
- (45)
Bohan Wang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, and Wei Chen.
On the convergence of Adam under non-uniform smoothness:
Separability from sgdm and beyond.
arXiv preprint arXiv:2403.15146, 2024.
- (46)
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma,
Tie-Yan Liu, Zhi-Quan Luo, and Wei Chen.
Provable adaptivity of Adam under non-uniform smoothness.
In Proceedings of the 30th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining, pages 2960–2969, 2023.
- (47)
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So
Kweon, and Saining Xie.
Convnext v2: Co-designing and scaling convnets with masked
autoencoders.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 16133–16142, 2023.
- (48)
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David
Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao.
Tensor programs V: Tuning large neural networks via zero-shot
hyperparameter transfer.
arXiv preprint arXiv:2203.03466, 2022.
- (49)
Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang.
Improved analysis of clipping algorithms for non-convex optimization.
In Advances in Neural Information Processing Systems,
volume 33, pages 15511–15521, 2020.
- (50)
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie.
Why gradient clipping accelerates training: A theoretical
justification for adaptivity.
In International Conference on Learning Representations, 2019.
- (51)
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo.
Adam can converge without any modification on update rules.
Advances in neural information processing systems,
35:28386–28399, 2022.

## Appendix A Relate Work

There is a large amount of works on the theoretical analysis of stochastic descents algorithms. In this section, we list the most related references and make comparison with our work.

Convergence with Weak Assumptions. [5] first theoretically analyze SGD under the assumption of affine variances, obtaining an asymptotic convergence result. Until 2018, [6] proved the non-asymptotic convergence rate of $\|\nabla F(x)\|_{2}$ for SGD up to ${\cal O}\left(\frac{{\rm poly(ln}T)}{T^{\nicefrac{{1}}{{4}}}}\right)$, which matched its provable rate with the bounded variance condition. In terms of adaptive optimizers, [17] investigated the convergence rate of AdaGrad-Norm with the affine variance, and proved the rate could achieve ${\cal O}\left(\frac{{\rm poly(ln}T)}{T^{\nicefrac{{1}}{{4}}}}\right)$ as well when$\sigma_{1}={\cal O}\left(\frac{1}{\sqrt{T}}\right)$. [43] proved the AdaGrad-Norm obtained a similar convergence rate with no restriction over $\sigma_{1}$, and it further demonstrate vanilla AdaGrad could also achieve the same convergence rate under a stronger assumption of coordinate-wise affine variances. Meanwhile, [2] provided a probabilistic convergence rate for AdaGrad-Norm. Noted that [37] and [51] respectively proved random-shuffled AMSProp and Adam will converge to the neighbourhood of stationary points with the rate ${\cal O}\left(\frac{{\rm poly(ln}E)}{E^{\nicefrac{{1}}{{4}}}}+\sigma_{0}\right)$ where $E$ is the number of epoches rather than iterations under the affine growth condition that is equivalent to the affine variance condition, which is much slower than the optimal rate. Recently, [20] provably demonstrate the convergence rate of vanilla Adam in high probability perspective, but it only works with the stronger coordinate-wise affine variance.

[50] first introduced the unciform $(L_{0},L_{1})$-smooth condition to theoretically explain why Clipped-SGD converges faster than vanilla SGD, and they also empirically verified that local smoothness indeed varies with the norm of gradients during DNN training. [49] posits that it is also equivalent to an affine form of the gradient norm for the first-order differentiable function. Then, the this relaxed assumption is extended to analyzing clipping-SGD with momentum [49], distributionally robust optimization [23], normalized SGD with momentum [22], generalized signSGD [13]. Recently, [46] theoretically analyzing random-shuffled Adam under this condition, but its convergence rate is provable ${\cal O}\left(\frac{{\rm poly(ln}E)}{E^{\nicefrac{{1}}{{4}}}}\right)$ where $E$ is the number of epoch, just like [51].
[30] further extended the linear $(L_{0},L_{1})$-smooth to the generalized polynomial version, and proved that Adam will converged to ${\cal O}\left(\frac{{\rm poly(ln}T)}{T^{\nicefrac{{1}}{{4}}}}\right)$ with the weaker assumption. However, the bound heavily relies on a large $\epsilon$ , but Adam with a large $\epsilon$ is essentially similar to SGD and loses the nature of adaptivity.

Furthermore, [16, 44] theoretically analyze AdaGrad under the weak assumptions of both affine variance and unciform $(L_{0},L_{1})$-smooth, also obtaining the rate of ${\cal O}\left(\frac{{\rm poly(ln}T)}{T^{\nicefrac{{1}}{{4}}}}\right)$. [45] also provable provide a high probability convergence of a simplified Adam-Norm with the both weak assumptions. More recently, [21] prove the convergence rate of vanilla Adam with both the affine variance condition and the $(L_{0},L_{1})$-smooth condition, but its proof have a fatal error that may vacuum the validity (The constant and $T$-independent $G$ is the premise of the theoretical analysis, but $1-\beta_{2}={\cal O}(\frac{1}{T})$, $1-\beta_{2}=\frac{c}{T}$, Eq. (41), Eq. (56) and Eq. (58) in the proof suggest that $\|\bar{\bm{g}}_{t+1}\|^{2}$ should be larger than ${\cal O}(T)$. ).

In comparison to the convergence analyses above, we are the first to formally analyze vanilla under the weak assumptions of the generalized affine variance and the generalized unciform $(L_{0},L_{1})$-smooth, achieving a tighter bound of ${\cal O}\left(\frac{1}{T^{\nicefrac{{1}}{{4}}}}\right)$ rather than${\cal O}\left(\frac{{\rm poly(ln}T)}{T^{\nicefrac{{1}}{{4}}}}\right)$.

Convergence of Sign Descent. Sign-based algorithms that simply exploits the signs of gradients could date back to RPROP ([35]). [36, 38] proposed 1-bit SGD and empirically demonstrate it achieve good performance while dramatically reducing the communication costs in distributed system. The non-stochastic convergence proof of signSGD was first analyzed in ([24]) under the Polyak-Łojasiewicz condition. Then, [4] systematically establish the convergence rate of signSGD in stochastic and non-convex scenario, but it required an increasing batch size up to $O(\sqrt{n})$ where $n$ is the number of samples to guarantee convergence. Then, [39] first proved that momentum can ensures the convergence of signSGD without increasing batch size. [10] employs an AutoML method to discover an effective optimizer, Lion, resembling signSGD with momentum, and demonstrate superior performance to Adam across diverse DNN models. [9] theoretically analyzed the efficacy of Lion but did not provide the convergence proof. Meanwhile, the original version of Adam, RMSProp [19], were developed from the sign-based Rprop. [3] also found that sign descent algorithms has a deep connection with Adam. Recently, [28, 29] empirically showcase that the sign-like property of Adam is just the primary reason behind its superior performance for training DNNs. In short, signSGD has a close connection with Adam, but the existing convergence proofs of Adam were not built upon this connection.

To the best of our knowledge, our work is the first to prove the convergence rate of Adam from the perspective of sign descent, and the proof, thereby, becomes considerably simple, compared to the previous theoretical proofs of Adam.

## Appendix B Theoretical Analysis

### B.1 Proof of Proposition 2.1

Proof. It is known that

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}^{2}=$ $\displaystyle{\cal O}\left(\frac{1}{T}\sum_{t=0}^{T-1}\frac{1}{t^{2\alpha}}\right)$ (24) $\displaystyle\leq$ $\displaystyle{\cal O}\left(\frac{1}{T}\int_{t=0}^{T-1}\frac{1}{t^{2\alpha}}dt\right)$ $\displaystyle=$ $\displaystyle{\cal O}\left(\frac{T^{1-2\alpha}-1}{T(1-2\alpha)}\right)$ $\displaystyle\leq$ $\displaystyle{\cal O}\left(\frac{c^{2}}{1-2\alpha}\cdot\frac{1}{T^{2\alpha}} \right).$ $$

and

$$ $\displaystyle\left(\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2} \right)^{2}=$ $\displaystyle{\cal O}\left(\left(\frac{1}{T}\sum_{t=0}^{T-1}\frac{1}{t^{\alpha }}\right)^{2}\right)$ (25) $\displaystyle\geq$ $\displaystyle{\cal O}\left(\left(\frac{1}{T}\int_{t=2}^{T+1}\frac{1}{t^{\alpha }}dt\right)^{2}\right)$ $\displaystyle\geq$ $\displaystyle{\cal O}\left(\frac{1}{(1-\alpha)^{2}}\cdot\left(\frac{1}{T^{2 \alpha}}-\frac{2^{1-\alpha}}{T^{1+\alpha}}\right)\right).$ $$

It is easy to verify that when $T\geq{2^{\frac{2-\alpha}{1-\alpha}}}$, $\frac{2^{1-\alpha}}{T^{1+\alpha}}\leq\frac{1}{2}\cdot\frac{1}{T^{2\alpha}}$. Moreover, we can obtain ${2^{\frac{2-\alpha}{1-\alpha}}}\leq 8$ due to $0\leq\alpha<\frac{1}{2}$. Hence, when $T\geq 8$, we have

$$ $\frac{\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}^{2}}{\left(\frac {1}{T}\sum_{t=0}^{T-1}\|\nabla F(\bm{x}_{t})\|_{2}\right)^{2}}\leq{\cal O} \left(\frac{2(1-\alpha)^{2}}{1-2\alpha}\right).$ (26) $$

### B.2 Useful Lemmas

###### Lemma B.1

Under Assumption B.3, for any $\bm{x},\bm{y}\in{\mathbb{R}}^{d}$, the function obeys

$$ $F(\bm{y})\leq F(\bm{x})+\langle\nabla F(\bm{x}),\bm{y}-\bm{x}\rangle+\frac{L_{ 0}+L_{1}\|\nabla F(\bm{x})\|_{2}^{q}}{2}\|\bm{y}-\bm{x}\|_{2}^{2}.$ (27) $$

Proof. For any $\bm{x},\bm{y}\in{\mathbb{R}}^{d}$, we have

$$ $\displaystyle F(\bm{y})=$ $\displaystyle F(\bm{x})+\int_{0}^{1}\langle\nabla F(\bm{x}+t(\bm{y}-\bm{x})), \bm{y}-\bm{x}\rangle dt$ (28) $\displaystyle=$ $\displaystyle F(\bm{x})+\langle\nabla F(\bm{x}),\bm{y}-\bm{x}\rangle+\int_{0}^ {1}\langle\nabla F(\bm{x}+t(\bm{y}-\bm{x}))-\nabla F(\bm{x}),\bm{y}-\bm{x} \rangle dt$ $\displaystyle\overset{(i)}{\leq}$ $\displaystyle F(\bm{x})+\langle\nabla F(\bm{x}),\bm{y}-\bm{x}\rangle+\int_{0}^ {1}\|\nabla F(\bm{x}+t(\bm{y}-\bm{x}))-\nabla F(\bm{x})\|_{2}\|\bm{y}-\bm{x}\| _{2}dt$ $\displaystyle\overset{(ii)}{\leq}$ $\displaystyle F(\bm{x})+\langle\nabla F(\bm{x}),\bm{y}-\bm{x}\rangle+(L_{0}+L_ {1}\|\nabla F(\bm{x})\|_{2}^{q})\|\bm{y}-\bm{x}\|_{2}^{2}\int_{0}^{1}td_{t}$ $\displaystyle=$ $\displaystyle F(\bm{x})+\langle\nabla F(\bm{x}),\bm{y}-\bm{x}\rangle+\frac{L_{ 0}+L_{1}\|\nabla F(\bm{x})\|_{2}^{q}}{2}\|\bm{y}-\bm{x}\|_{2}^{2},$ $$

where$(i)$ holds due to Cauchy-Schwarz inequality, and $(ii)$ holds due to Assumption 2.

###### Lemma B.2

Let the sequences $\{\bm{m}_{t}\}$ and $\{\bm{v}_{t}\}$ be generated by Adam in Algorithm 1 with omitted bias correction. If the moving average coefficients $\beta_{1},\beta_{2}$ are constant and satisfy $\beta_{1}^{2}<\beta_{2}$, then

(1) For any $j\in[d]$, it holds that

$$ $\frac{|\bm{m}_{t}^{(j)}|}{\sqrt{\bm{v}_{t}^{(j)}}+\epsilon}\leq\frac{1-\beta_{ 1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{1}^{2}}{\beta_{2}}}}.$ (29) $$

(2) The maximal value of $\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{1}^{2}}{\beta_{2}}}}$ is lower bounded by $1$, and upper bounded by $\sqrt{\frac{1-\beta_{1}}{1-\beta_{2}}}$ when $\beta_{1}\leq\beta_{2}$.

Proof. (1) Recalling Adam, we know

$$ $\displaystyle\bm{m}_{t}^{(j)}=(1-\beta_{1})\sum_{k=1}^{t}\beta_{1}^{t-k}\bm{g} _{k}^{(j)}$ (30) $\displaystyle\bm{v}_{t}^{(j)}=(1-\beta_{2})\sum_{k=1}^{t}\beta_{2}^{t-k}(\bm{g }_{k}^{(j)})^{2}.$ $$

Then,

$$ $\displaystyle\frac{|\bm{m}_{t}^{(j)}|}{\sqrt{\bm{v}_{t}^{(j)}}+\epsilon}\leq$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}}\cdot\frac{|\sum_{k=1}^{t} \beta_{1}^{t-k}\bm{g}_{k}^{(j)}|}{\sqrt{\sum_{k=1}^{t}\beta_{2}^{t-k}(\bm{g}_{ k}^{(j)})^{2}}}$ (31) $\displaystyle\overset{(i)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}}\cdot\frac{\sum_{k=1}^{t} \beta_{1}^{t-k}|\bm{g}_{k}^{(j)}|}{\sqrt{\sum_{k=1}^{t}\beta_{2}^{t-k}(\bm{g}_ {k}^{(j)})^{2}}}$ $\displaystyle\overset{(ii)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}}\cdot\frac{\sqrt{\sum_{k=1} ^{t}\beta_{2}^{t-k}(\bm{g}_{k}^{(j)})^{2}}\sqrt{\sum_{k=1}^{t}\frac{\beta_{1}^ {2(t-k)}}{\beta_{2}^{t-k}}}}{\sqrt{\sum_{k=1}^{t}\beta_{2}^{t-k}(\bm{g}_{k}^{( j)})^{2}}}$ $\displaystyle=$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}}\cdot\sqrt{\sum_{k=1}^{t} \frac{\beta_{1}^{2(t-k)}}{\beta_{2}^{t-k}}}$ $\displaystyle\overset{(iii)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{1}^{2}} {\beta_{2}}}},$ $$

where $(i)$ holds due to the fact $|\bm{a}^{(j)}+\bm{b}^{(j)}|\leq|\bm{a}^{(j)}|+|\bm{b}^{(j)}|$; $(ii)$ holds due to Cauchy-Schiwaz inequality; $(iii)$ holds due to $\beta_{1}^{2}\leq\beta_{2}$.

(2) The maximal value of $\frac{|\bm{m}_{t}^{(j)}|}{\sqrt{\bm{v}_{t}^{(j)}}}$ is lower bounded by

$$ $\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{1}^{2}}{\beta_{2}}}} \geq\frac{1-\beta_{1}}{{1-\frac{1}{2}(\beta_{2}+\frac{\beta_{1}^{2}}{\beta_{2} })}}\geq{\frac{1-\beta_{1}}{1-\beta_{1}}}=1,$ (32) $$

where the first and second inequality reaches the lower bound if and only $\beta_{1}=\beta_{2}$.

When $\beta_{1}\leq\beta_{2}$, the maximal value of $\frac{|\bm{m}_{t}^{(j)}|}{\sqrt{\bm{v}_{t}^{(j)}}}$ is upper bounded by

$$ $\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{1}^{2}}{\beta_{2}}}} \leq\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{1}\beta_{2}}{ \beta_{2}}}}=\frac{\sqrt{1-\beta_{1}}}{\sqrt{1-\beta_{2}}}.$ (33) $$

###### Lemma B.3

For any random variable ${\mathcal{Z}}$ and a constant $C$, there exists

$$ ${\mathbb{E}}[|{\rm Sign}({\mathcal{Z}})-{\rm Sign}(C)|]\leq\frac{2{\mathbb{E}} [|{\mathcal{Z}}-C|]}{|C|}.$ (34) $$

Proof. Using Markov’equality, we direct obtain

$$ $\displaystyle{\mathbb{E}}[|{\rm Sign}({\mathcal{Z}})-{\rm Sign}(C)|]=$ $\displaystyle 2{\mathbb{P}}[{\mathbb{I}}({\rm Sign}({\mathcal{Z}})\neq{\rm Sign }(C))]$ (35) $\displaystyle\leq$ $\displaystyle 2{\mathbb{P}}[|{\mathcal{Z}}-C|\geq|C|]$ $\displaystyle\leq$ $\displaystyle\frac{2{\mathbb{E}}[|{\mathcal{Z}}-C|]}{|C|}.$ $$

###### Lemma B.4

Let $a,b,c>0$ and $0<\alpha<\beta$. If $x\geq({a+b^{\nicefrac{{\alpha}}{{\beta}}}})^{\nicefrac{{1}}{{\alpha}}}$, then $\frac{a}{x^{\alpha}}+\frac{b}{x^{\beta}}\leq 1$. The bound is tight up to the factor of 2 since $\frac{({a+b^{\nicefrac{{\alpha}}{{\beta}}}})^{\nicefrac{{1}}{{\alpha}}}}{2}
\leq\max({a},{b^{\nicefrac{{1}}{{\gamma}}}})\leq({a+b^{\nicefrac{{\alpha}}{{
\beta}}}})^{\nicefrac{{1}}{{\alpha}}}$.

Proof. Let $s=x^{\alpha}$ and $\gamma=\frac{\beta}{\alpha}\geq 1$, then the inequality becomes

$$ $\frac{a}{s}+\frac{b}{s^{\gamma}}\leq 1$ (36) $$

If $s^{*}$ is a solution of Eq. ([36](https://arxiv.org/html/2507.05966v1#A2.E36)), it should satisfy

$$ $s^{*}\geq\max({a},{b^{\nicefrac{{1}}{{\gamma}}}}).$ (37) $$

When we set $s_{+}={a+b^{\nicefrac{{1}}{{\gamma}}}}$, it is easy to verify that

$$ $\frac{a}{s_{+}}+\frac{b}{s_{+}^{\gamma}}=\frac{a}{a+b^{\nicefrac{{1}}{{\gamma} }}}+\frac{b}{(a+b^{\nicefrac{{1}}{{\gamma}}})^{\gamma}}\leq\frac{a}{a+b^{ \nicefrac{{1}}{{\gamma}}}}+\frac{b^{\nicefrac{{1}}{{\gamma}}}}{a+b^{\nicefrac{ {1}}{{\gamma}}}}=1.$ (38) $$

On the other hand, it is also easy to verify that $s_{-}=\frac{a+b^{\nicefrac{{1}}{{\gamma}}}}{2}$ does not satisfy Eq. ([37](https://arxiv.org/html/2507.05966v1#A2.E37)), which means that $s_{+}$ is at most a factor of $2$ worse than the smallest solution of Eq. ([36](https://arxiv.org/html/2507.05966v1#A2.E36)), so $x_{+}=({a+b^{\nicefrac{{\alpha}}{{\beta}}}})^{\nicefrac{{1}}{{\alpha}}}$ is at most a factor of $2$ worse than the smallest solution of $\frac{a}{x^{\alpha}}+\frac{b}{x^{\beta}}\leq 1$.

###### Lemma B.1

###### Lemma B.2

###### Lemma B.3

###### Lemma B.4

### B.3 Proof of Theorem 3.1

Proof. Following Lemma [27](https://arxiv.org/html/2507.05966v1#A2.E27) with $\bm{x}_{t+1}\rightarrow\bm{y}$ and $\bm{x}_{t}\rightarrow\bm{x}$, we have

$$ $F(\bm{x}_{t+1})\leq F(\bm{x}_{t})+\langle\nabla F(\bm{x}_{t}),\bm{x}_{t+1}-\bm {x}_{t}\rangle+\frac{L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{q}}{2}\|\bm{x}_{ t+1}-\bm{x}_{t}\|_{2}^{2}.$ (39) $$

Recalling the update rule $\bm{x}_{t+1}=\bm{x}_{t}-\gamma\frac{\bm{m}_{t}}{\sqrt{\bm{v}_{t}}+\epsilon}=
\bm{x}_{t}-\gamma\frac{|\bm{m}_{t}|}{\sqrt{\bm{v}_{t}}+\epsilon}\circ\frac{\bm
{m}_{t}}{|\bm{m}_{t}|}=\bm{x}_{t}-\gamma\bm{u}_{t}\circ{\rm Sign}(\bm{m}_{t})$, we further obtain

$$ $\displaystyle F(\bm{x}_{t+1})\leq$ $\displaystyle F(\bm{x}_{t})-\langle\nabla F(\bm{x}_{t}),\gamma\bm{u}_{t}\circ{ \rm Sign}(\bm{m}_{t})\rangle+\frac{(L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{q })\gamma^{2}}{2}\|\bm{u}_{t}\|_{2}^{2}$ (40) $\displaystyle=$ $\displaystyle F(\bm{x}_{t})-\langle\nabla F(\bm{x}_{t}),\gamma\bm{u}_{t}\circ{ \rm Sign}(\nabla F(\bm{x}_{t}))\rangle+{\left\langle\nabla F(\bm{x}_{t}), \gamma\bm{u}_{t}\circ({\rm Sign}(\nabla F(\bm{x}_{t}))-{\rm Sign}(\bm{m}_{t})) \right\rangle}$ $\displaystyle+\frac{\gamma^{2}(L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{q})}{2 }\|\bm{u}_{t}\|_{2}^{2}$ $\displaystyle\leq$ $\displaystyle F(\bm{x}_{t})-\langle\nabla F(\bm{x}_{t}),\gamma\bm{u}_{t}\circ{ \rm Sign}(\nabla F(\bm{x}_{t}))\rangle+\gamma{R}{\left\langle\nabla F(\bm{x}_{ t}),{\rm Sign}(\nabla F(\bm{x}_{t}))-{\rm Sign}(\bm{m}_{t})\right\rangle}$ $\displaystyle+\frac{\gamma^{2}R^{2}d(L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{ q})}{2},$ $$

where the last inequality holds due to $\bm{u}_{t}^{(j)}\leq\frac{1-\beta_{1}}{\sqrt{1-\beta_{2}}\sqrt{1-\frac{\beta_{
1}^{2}}{\beta_{2}}}}={R},\forall j\in[d]$ according to Lemma [B.2](https://arxiv.org/html/2507.05966v1#A2.Thmtheorem2).

Taking the expectation at the $t$ iteration, we obtain

$$ $\displaystyle{\mathbb{E}}[{F(\bm{x}_{t+1})}]\leq$ $\displaystyle F(\bm{x}_{t})-\gamma\langle\nabla F(\bm{x}_{t}),{\mathbb{E}}[\bm {u}_{t}]\circ{\rm Sign}(\nabla F(\bm{x}_{t}))\rangle+\gamma{R}{\left\langle \nabla F(\bm{x}_{t}),{\mathbb{E}}[{\rm Sign}(\nabla F(\bm{x}_{t}))-{\rm Sign}( \bm{m}_{t})]\right\rangle}$ (41) $\displaystyle+\frac{\gamma^{2}R^{2}d(L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{ q})}{2}$ $\displaystyle\overset{(i)}{\leq}$ $\displaystyle F(\bm{x}_{t})-\gamma\langle\nabla F(\bm{x}_{t}),{\mathbb{E}}[\bm {u}_{t}]\circ{\rm Sign}(\nabla F(\bm{x}_{t}))\rangle+2\gamma{R}\;{\mathbb{E}}[ \|\bm{m}_{t}-\nabla F(\bm{x}_{t})\|_{1}]$ $\displaystyle+\frac{\gamma^{2}R^{2}d(L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{ q})}{2}$ $\displaystyle\overset{(ii)}{\leq}$ $\displaystyle F(\bm{x}_{t})-\gamma\langle\nabla F(\bm{x}_{t}),{\mathbb{E}}[\bm {u}_{t}]\circ{\rm Sign}(\nabla F(\bm{x}_{t}))\rangle+2\gamma R\sqrt{d}\;{ \mathbb{E}}[\|\bm{m}_{t}-\nabla F(\bm{x}_{t})\|_{2}]$ $\displaystyle+\frac{\gamma^{2}R^{2}d(L_{0}+L_{1}\|\nabla F(\bm{x}_{t})\|_{2}^{ q})}{2}$ $\displaystyle\overset{(iii)}{\leq}$ $\displaystyle F(\bm{x}_{t})-\gamma\langle\nabla F(\bm{x}_{t}),{\mathbb{E}}[\bm {u}_{t}]\circ{\rm Sign}(\nabla F(\bm{x}_{t}))\rangle+2\gamma R\sqrt{d}\;{ \mathbb{E}}[\|\bm{m}_{t}-\nabla F(\bm{x}_{t})\|_{2}]$ $\displaystyle+\frac{\gamma^{2}R^{2}d(L_{0}+L_{1}((1-q)+q{\mathbb{E}}[\|\nabla F (\bm{x}_{t})\|_{2}]))}{2},$ $$

where $(i)$ holds due to ${\mathbb{E}}[|{\rm Sign}(\nabla F(\bm{x}_{t}^{(j)}))-{\rm Sign}(\bm{m}_{t}^{(j
)})|]\leq 2\frac{{\mathbb{E}}[|\nabla F(\bm{x}_{t}^{(j)})-\bm{m}_{t}^{(j)}|]}{
|\nabla F(\bm{x}_{t}^{(j)})|}(\forall j\in[d])$ according to Lemma [34](https://arxiv.org/html/2507.05966v1#A2.E34); $(ii)$holds owing to the fact $\|\bm{a}\|_{1}\leq\sqrt{d}\|\bm{a}\|_{2}$ for any $\bm{a}\in{\mathbb{R}}^{d}$; $(iii)$ holds due to the fact $a^{q}\leq(1-q)+qa$ according to Young’s inequality.

Taking expectation from the $1$-st iteration to the $T$-th iteration and then summing them, we have

$$ $\displaystyle{\mathbb{E}}[{F(\bm{x}_{t+1})}]\leq$ $\displaystyle F(\bm{x}_{1})-\gamma\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t} \circ\nabla F(\bm{x}_{t})\|_{1}]+2\gamma R\sqrt{d}\;\sum_{t=0}^{T-1}{\mathbb{E }}[\|\bm{m}_{t}-\nabla F(\bm{x}_{t})\|_{2}]$ (42) $\displaystyle+\sum_{t=0}^{T-1}\frac{\gamma^{2}Rd(L_{0}+L_{1}((1-q)+q\|\nabla F (\bm{x}_{t})\|_{2}))}{2}.$ $$

Rearranging the both sides and applying the facts that ${F(\bm{x}_{t+1})}\geq F^{*}$, we obtain

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ\nabla F (\bm{x}_{t})\|_{1}]-\frac{\gamma R^{2}qL_{1}{d}}{2T}\sum_{t=0}^{T-1}\mathbb{E} [\|\nabla F(\bm{x}_{t})\|_{2}]$ (43) $\displaystyle\leq$ $\displaystyle\frac{F(\bm{x}_{0})-F^{*}}{\gamma T}+\frac{2R\sqrt{d}}{T}\sum_{t= 0}^{T-1}{\mathbb{E}}[\|\bm{m}_{t}-\nabla F(\bm{x}_{t})\|_{2}]+\frac{\gamma R^{ 2}{d}(L_{0}+(1-q)L_{1})}{2T}.$ $$

Recalling $\bm{m}_{t}=\beta_{1}\bm{m}_{t-1}+(1-\beta_{1})\bm{g}_{t}$, we obtain

$$ $\displaystyle\bm{m}_{t}-\nabla F(\bm{x}_{t})=$ $\displaystyle\left(\beta_{1}\bm{m}_{t-1}+(1-\beta_{1})\bm{g}_{t}\right)-\nabla F (\bm{x}_{t})$ (44) $\displaystyle=$ $\displaystyle\beta_{1}(\bm{m}_{t-1}-\nabla F(\bm{x}_{t-1}))+(1-\beta_{1})(\bm{ g}_{t}-\nabla F(\bm{x}_{t}))-\beta_{1}(\nabla F(\bm{x}_{t})-\nabla F(\bm{x}_{t -1})).$ $$

Utilizing recursion, we further have

$$ $\displaystyle\bm{m}_{t}-\nabla F(\bm{x}_{t})=$ $\displaystyle-\beta_{1}^{t}\nabla F(\bm{x}_{0})+(1-\beta)\sum_{k=1}^{t}\beta_{ 1}^{t-k}(\bm{g}_{k}-\nabla F(\bm{x}_{k}))-\sum_{k=1}^{t}\beta_{1}^{t-k+1}( \nabla F(\bm{x}_{k})-\nabla F(\bm{x}_{k-1})),$ (45) $$

where $\bm{m}_{1}-\nabla F(\bm{x}_{0})=-\beta_{1}\nabla F(\bm{x}_{0})+(1-\beta_{1})(
\bm{g}_{1}-\nabla F(\bm{x}_{0}))$ due to $\bm{m}_{0}=0$.

Hence,

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}}\left[\|\bm{m}_{t}-\nabla F (\bm{x}_{t})\|_{2}\right]\leq$ $\displaystyle\underbrace{\frac{1}{T}\sum_{t=0}^{T-1}\beta_{1}^{t}\left\|\nabla F (\bm{x}_{0})\right\|_{2}}_{{\cal T}_{1}}+\underbrace{\frac{1-\beta_{1}}{T}\sum _{t=0}^{T-1}{\mathbb{E}}\left[\left\|\sum_{k=1}^{t}\beta_{1}^{t-k}\left(\bm{g} _{k}-\nabla F(\bm{x}_{k})\right)\right\|_{2}\right]}_{{\cal T}_{2}}$ (46) $\displaystyle+\underbrace{\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}}\left[\left\| \sum_{k=1}^{t}\beta_{1}^{t-k+1}(\nabla F(\bm{x}_{k})-\nabla F(\bm{x}_{k-1})) \right\|_{2}\right]}_{{\cal T}_{3}}$ $$

In terms of ${\cal T}_{1}$, we obtain

$$ ${\cal T}_{1}=\frac{1}{T}\sum_{t=0}^{T-1}\beta_{1}^{t}\left\|\nabla F(\bm{x}_{0 })\right\|_{2}\leq\frac{\left\|\nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1} )}.$ (47) $$

As for ${\cal T}_{2}$, we have

$$ $\displaystyle{\cal T}_{2}=$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}{\mathbb{E}}\left[\left\| \sum_{k=1}^{t}\beta_{1}^{t-k}(\bm{g}_{k}-\nabla F(\bm{x}_{k}))\right\|_{2}\right]$ (48) $\displaystyle\overset{(i)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{{\mathbb{E}}\left[ \left\|\sum_{k=1}^{t}\beta_{1}^{t-k}(\bm{g}_{k}-\nabla F(\bm{x}_{k}))\right\|_ {2}^{2}\right]}$ $\displaystyle\overset{(ii)}{=}$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1}^{t}\beta_{1 }^{2(t-k)}{\mathbb{E}}\left[\left\|\bm{g}_{k}-\nabla F(\bm{x}_{k})\right\|_{2} ^{2}\right]}$ $\displaystyle\overset{(iii)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1}^{t}\beta_{1 }^{2(t-k)}(\sigma_{0}^{2}+\sigma_{1}^{2}\|\nabla F(x_{k})\|_{2}^{p})}$ $\displaystyle\overset{(iv)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1}^{t}\beta_{1 }^{2(t-k)}\sigma_{0}^{2}}+\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1 }^{t}\beta_{1}^{2(t-k)}\sigma_{1}^{2}\|\nabla F(x_{k})\|_{2}^{p}}$ $\displaystyle\overset{(v)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1}^{t}\beta_{1 }^{2(t-k)}\sigma_{0}^{2}}+\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1 }^{t}\beta_{1}^{2(t-k)}\sigma_{1}^{2}\left(\frac{2-p}{2}+\frac{p}{2}\|\nabla F (x_{k})\|_{2}^{2}\right)}$ $\displaystyle\overset{(vi)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1}^{t}\beta_{1 }^{2(t-k)}\sigma_{0}^{2}}+\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1 }^{t}\frac{(2-p)\sigma_{1}^{2}}{2}\beta_{1}^{2(t-k)}}$ $\displaystyle+\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{\sum_{k=1}^{t}\frac{p }{2}\beta_{1}^{2(t-k)}\sigma_{1}^{2}\|\nabla F(x_{k})\|_{2}^{2}}$ $\displaystyle\overset{(vii)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{1}^{2}}}\left(\sigma_{0}+\sqrt{ \frac{2-p}{2}}\sigma_{1}\right)+\frac{1-\beta_{1}}{T}\sum_{t=0}^{T-1}\sqrt{ \sum_{k=1}^{t}\frac{p\sigma_{1}^{2}}{2}\beta_{1}^{2(t-k)}\|\nabla F(x_{k})\|_{ 2}^{2}}$ $\displaystyle\overset{(viii)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{1}^{2}}}\left(\sigma_{0}+\sqrt{ \frac{2-p}{2}}\sigma_{1}\right)+(1-\beta_{1})\sqrt{\frac{1}{T}\sum_{t=0}^{T-1} \sum_{k=1}^{t}\frac{p\sigma_{1}^{2}}{2}\beta_{1}^{2(t-k)}\|\nabla F(x_{k})\|_{ 2}^{2}}$ $\displaystyle\overset{(ix)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{1}^{2}}}\left(\sigma_{0}+\sqrt{ \frac{2-p}{2}}\sigma_{1}\right)+(1-\beta_{1})\sqrt{\frac{p\sigma_{1}^{2}}{2} \cdot\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(x_{t})\|_{2}^{2}\sum_{k=t}^{T}\beta _{1}^{2(t-k)}}$ $\displaystyle{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{1}^{2}}}\left(\sigma_{0}+\sqrt{ \frac{2-p}{2}}\sigma_{1}\right)+\frac{\sqrt{p}\sigma_{1}(1-\beta_{1})}{\sqrt{2 (1-\beta_{1}^{2})}}\sqrt{\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(x_{t})\|_{2}^{2}}$ $\displaystyle\overset{(x)}{\leq}$ $\displaystyle\frac{1-\beta_{1}}{\sqrt{1-\beta_{1}^{2}}}\left(\sigma_{0}+\sqrt{ \frac{2-p}{2}}\sigma_{1}\right)+\frac{C_{0}\sqrt{p}\sigma_{1}(1-\beta_{1})}{ \sqrt{2(1-\beta_{1}^{2})}}\cdot\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(x_{t})\|_ {2}$ $\displaystyle\leq$ $\displaystyle\sqrt{1-\beta_{1}}\left(\sigma_{0}+\sqrt{\frac{2-p}{2}}\sigma_{1} \right)+{C_{0}\sigma_{1}\sqrt{p(1-\beta_{1})}}\cdot\frac{1}{T}\sum_{t=0}^{T-1} \|\nabla F(x_{t})\|_{2}$ $$

where $(i)$ holds due to the fact $({\mathbb{E}}[Z])^{2}\leq{\mathbb{E}}[Z^{2}]$; $(ii)$ holds owing to ${\mathbb{E}}[\bm{g}_{k}-\nabla F(\bm{x}_{k})]=\bm{0}$ according to Assumption A; $(iii)$ holds resulting from ${\mathbb{E}}\left[\left\|\bm{g}_{k}-\nabla F(\bm{x}_{k})\right\|_{2}^{2}\right
]\leq\sigma^{2}$ according to Assumption C.3; $(iv)$ holds due to the fact $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$; $(v)$ holds resulting from the fact that $a^{p}\leq\frac{2-p}{2}+\frac{p}{2}a^{2},0\leq p\leq 2$ according to Young’s inequality; $(vi)$ holds due to using the fact $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ again;$(vii)$ holds due to the fact $\sum_{i}^{T}{a_{i}}\leq\sqrt{T}\sqrt{\sum_{i}^{T}a_{i}^{2}}$ according to Cauchy-Schwaz inequality; $(viii)$ holds owing to $\sum_{k=1}^{t}\beta_{1}^{2(t-k)}\leq\frac{1}{1-\beta_{1}^{2}}$; $(ix)$ holds thanks to the fact $\sum_{i=1}^{n}\sum_{j=1}^{i}f(i,j)g(j)=\sum_{j=1}^{n}g(j)\sum_{i=j}^{n}f(i,j)$; $(x)$ holds because of Condition 1.

Now we turn attention to ${\cal T}_{3}$, *i.e.*,

$$ $\displaystyle{\cal T}_{3}=$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}}\left[\left\|\sum_{k=1}^{t }\beta_{1}^{t-k+1}(\nabla F(\bm{x}_{k})-\nabla F(\bm{x}_{k-1}))\right\|_{2}\right]$ (49) $\displaystyle\overset{(i)}{\leq}$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\sum_{k=1}^{t}\beta_{1}^{t-k+1}{ \mathbb{E}}\left[\left\|\nabla F(\bm{x}_{k})-\nabla F(\bm{x}_{k-1})\right\|_{2 }\right]$ $\displaystyle\overset{(ii)}{\leq}$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\sum_{k=1}^{t}\beta_{1}^{t-k+1}{ \mathbb{E}}\left[(L_{0}+L_{1}\|\nabla F(\bm{x}_{k})\|_{2})\|\bm{x}_{k}-\bm{x}_ {k-1}\|_{2}\right]$ $\displaystyle\overset{(iii)}{=}$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\sum_{k=1}^{t}\beta_{1}^{t-k+1}{ \mathbb{E}}\left[\gamma(L_{0}+L_{1}\|\nabla F(\bm{x}_{k})\|_{2})\|\bm{u}_{t-1} \|_{2}\right])$ $\displaystyle\overset{(iv)}{\leq}$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}L_{1}\gamma R\sqrt{d}\sum_{k=1}^{t} \beta_{1}^{t-k+1}+\frac{\beta^{2}L_{1}\gamma R\sqrt{d}}{T}\sum_{k=1}^{t}\beta_ {1}^{t-k+1}{\mathbb{E}}[\|\nabla F(\bm{x}_{k}\|_{2}^{q})]$ $\displaystyle\leq$ $\displaystyle\frac{\gamma L_{0}R\sqrt{d}}{1-\beta}+\frac{\gamma L_{1}R\sqrt{d} }{T}\sum_{t=0}^{T-1}\sum_{k=1}^{t}\beta_{1}^{t-k+1}((1-q)+q{\mathbb{E}}[\| \nabla F(\bm{x}_{k})\|_{2}])$ $\displaystyle\overset{(v)}{\leq}$ $\displaystyle\frac{\gamma(L_{0}+(1-q)L_{1})R\sqrt{d}}{1-\beta}+\frac{\gamma qL _{1}R\sqrt{d}}{T}\sum_{t=0}^{T-1}\sum_{k=1}^{t}\beta_{1}^{t-k+1}{\mathbb{E}}[ \|\nabla F(\bm{x}_{k})\|_{2}]$ $\displaystyle\overset{(vi)}{=}$ $\displaystyle\frac{\gamma(L_{0}+(1-q)L_{1})R\sqrt{d}}{1-\beta_{1}}+\frac{ \gamma qL_{1}R\sqrt{d}}{T}\sum_{k=1}^{T}{\mathbb{E}}[\|\nabla F(\bm{x}_{k})\|_ {2}]\sum_{t=k}^{T}\beta_{1}^{t-k+1}$ $\displaystyle\leq$ $\displaystyle\frac{\gamma(L_{0}+(1-q)L_{1})R\sqrt{d}}{1-\beta_{1}}+\frac{ \gamma qL_{1}R\sqrt{d}}{(1-\beta_{1})T}\sum_{t=0}^{T-1}{\mathbb{E}}[\|\nabla F (\bm{x}_{t})\|_{2}]$ $$

where $(i)$ holds due to the fact $\|\bm{a}+\bm{b}\|_{2}\leq\|\bm{a}\|_{2}+\|\bm{b}\|_{2}$; $(ii)$ holds owing to Assumption B.3; $(iii)$ holds due to the update rule; $(iv)$ holds depending on $\bm{u}^{(j)}\leq\nicefrac{{1-\beta_{1}}}{{\sqrt{1-\beta_{2}}\sqrt{1-\frac{
\beta_{1}^{2}}{\beta_{2}}}}}=R$ according to Lemma [B.2](https://arxiv.org/html/2507.05966v1#A2.Thmtheorem2); $(v)$ holds thanks to the fact that $a^{q}\leq(1-q)+qa$ according to Young’s inequality; $(vi)$ holds resulting from the fact that $\sum_{i=1}^{n}\sum_{j=1}^{i}\bm{a}_{i,j}=\sum_{j=1}^{n}\sum_{i=j}^{n}\bm{a}_{i
,j}$.

Combining Eq.([46](https://arxiv.org/html/2507.05966v1#A2.E46)) - Eq.([49](https://arxiv.org/html/2507.05966v1#A2.E49)), we have

$$ $\displaystyle\frac{1}{T}\sum_{t=1}^{T}{\mathbb{E}}\left[\|\bm{m}_{t}-\nabla F( \bm{x}_{t})\|_{2}\right]\leq$ $\displaystyle\frac{\left\|\nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}+{ \sqrt{1-\beta_{1}}}\left(\sigma_{0}+\sqrt{\frac{2-p}{2}}\sigma_{1}\right)$ (50) $\displaystyle+$ $\displaystyle{C_{0}\sigma_{1}\sqrt{p(1-\beta_{1})}}\cdot\frac{1}{T}\sum_{t=0}^ {T-1}\|\nabla F(x_{t})\|_{2}$ $\displaystyle+$ $\displaystyle\frac{\gamma R\sqrt{d}(L_{0}+(1-q)L_{1})}{1-\beta_{1}}+\frac{ \gamma R\sqrt{d}qL_{1}}{1-\beta_{1}}\cdot\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E }}[\|\nabla F(\bm{x}_{t})\|_{2}]$ $$

Combining Eq.([43](https://arxiv.org/html/2507.05966v1#A2.E43)) and Eq.([50](https://arxiv.org/html/2507.05966v1#A2.E50)), we obtain

$$ $\displaystyle\frac{1}{T}\left(\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ \nabla F(\bm{x}_{t})\|_{1}]-\left(\frac{\gamma R^{2}dqL_{1}}{2}+{{2}C_{0}R \sqrt{d}\sigma_{1}\sqrt{p(1-\beta_{1})}}+\frac{2\gamma R^{2}{d}qL_{1}}{1-\beta _{1}}\right)\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2}]\right)$ (51) $\displaystyle\leq$ $\displaystyle\frac{F(\bm{x}_{0})-F^{*}}{\gamma T}+\frac{2R\sqrt{d}\left\| \nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}+{2R\sqrt{d}\sqrt{1-\beta_{1} }}\left(\sigma_{0}+\sqrt{\frac{2-p}{2}}\sigma_{1}\right)$ $\displaystyle+\frac{2\gamma R^{2}d(L_{0}+(1-q)L_{1})}{1-\beta_{1}}+\frac{ \gamma R^{2}{d}(L_{0}+(1-q)L_{1})}{2T}.$ $$

### B.4 Proof of Corollary 3.2

Proof. When C.1 holds, it implies $\sigma_{1}=0$. Also, it is known that $\|\nabla F(\bm{x}_{t})\|_{2}\leq\|\nabla F(\bm{x}_{t})\|_{1}$. Hence, the conclusion in Theorem 2 can be simplified as

$$ $\displaystyle\left(v-\frac{\gamma R^{2}dqL_{1}}{2}-\frac{2\gamma R^{2}{d}qL_{1 }}{1-\beta_{1}}\right)\cdot\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}}[\|\nabla F( \bm{x}_{t})\|_{1}]\leq$ $\displaystyle\frac{F(\bm{x}_{0})-F^{*}}{\gamma T}+\frac{2R\sqrt{d}\left\| \nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}$ (52) $\displaystyle+{2\sqrt{1-\beta_{1}}R\sqrt{d}}\sigma_{0}+\frac{2\gamma R^{2}d \hat{L}}{1-\beta_{1}}+\frac{\gamma R^{2}{d}\hat{L}}{2T},$ $$

where $0<v\leq\min_{t,j}\bm{u}_{t}^{(j)}$ and $\hat{L}=L_{0}+(1-q)L_{1}$.

Choosing $\gamma=\frac{C_{2}}{T^{\nicefrac{{3}}{{4}}}d^{\nicefrac{{1}}{{2}}}}$ and $1-\beta_{1}=\frac{C_{3}}{T^{\nicefrac{{1}}{{2}}}}$ and $T\geq(\frac{4C_{2}R^{2}d^{\nicefrac{{1}}{{2}}}qL_{1}}{C_{3}v}+(\frac{C_{2}R^{2
}d^{\nicefrac{{1}}{{2}}}qL_{1}}{v})^{\nicefrac{{1}}{{3}}})^{4}$, following Lemma [B.4](https://arxiv.org/html/2507.05966v1#A2.Thmtheorem4), it holds that

$$ $\frac{\gamma R^{2}dqL_{1}}{2}+\frac{2\gamma R^{2}{d}qL_{1}}{1-\beta_{1}}\leq \frac{v}{2}.$ (53) $$

Then, we arrive the conclusion

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{1 }]\leq$ $\displaystyle\frac{1}{v}\left(\frac{2(F(\bm{x}_{0})-F(\bm{x}^{*}))}{C_{2}T^{ \nicefrac{{1}}{{4}}}}+\frac{4Rd^{\nicefrac{{1}}{{2}}}\left\|\nabla F(\bm{x}_{0 })\right\|_{2}}{C_{3}T^{\nicefrac{{1}}{{2}}}}\right.$ (54) $\displaystyle+$ $\displaystyle\left.\frac{4C_{3}Rd^{\nicefrac{{1}}{{2}}}\sigma_{0}}{T^{ \nicefrac{{1}}{{4}}}}+\frac{4C_{2}R^{2}{d}^{\nicefrac{{1}}{{2}}}\hat{L}}{C_{3} T^{\nicefrac{{1}}{{4}}}}+\frac{C_{2}R^{2}d^{\nicefrac{{1}}{{2}}}\hat{L}}{T^{ \nicefrac{{7}}{{4}}}}\right).$ $$

## Appendix C Proof of Corollary 3.3

Proof. (1) Choosing $\bar{v}=\min_{t}\mathbb{E}[\bm{u}_{t}^{(j)}]$, we have

$$ $\displaystyle\sum_{t=0}^{T-1}{\mathbb{E}}[\|\bm{u}_{t}\circ\nabla F(\bm{x}_{t} )\|_{1}]=$ $\displaystyle\sum_{t=0}^{T-1}\sum_{j=1}^{d}{\mathbb{E}}[|\bm{u}_{t}^{(j)} \nabla F(\bm{x}_{t}^{(j)})|]$ (55) $\displaystyle\overset{(i)}{=}$ $\displaystyle\sum_{t=0}^{T-1}\sum_{j=1}^{d}{\mathbb{E}}[\bm{u}_{t}^{(j)}]{ \mathbb{E}}[|\nabla F(\bm{x}_{t}^{(j)})|]$ $\displaystyle\overset{(ii)}{=}$ $\displaystyle\sum_{t=0}^{T-1}{\mathbb{E}}[\bm{u}_{t}^{(j)}]\sum_{j=1}^{d}{ \mathbb{E}}[|\nabla F(\bm{x}_{t}^{(j)})|]$ $\displaystyle=$ $\displaystyle\sum_{t=0}^{T-1}{\mathbb{E}}[\bm{u}_{t}^{(j)}]{\mathbb{E}}[\|( \nabla F(\bm{x}_{t}))\|_{1}]$ $\displaystyle\overset{(iii)}{\geq}$ $\displaystyle\bar{v}\sum_{t=0}^{T-1}{\mathbb{E}}[\|(\nabla F(\bm{x}_{t}))\|_{1}]$ $\displaystyle\overset{(iv)}{=}$ $\displaystyle\frac{\bar{v}\sqrt{d}}{C_{1}}\sum_{t=0}^{T-1}{\mathbb{E}}[\|( \nabla F(\bm{x}_{t}))\|_{2}],$ $$

where $(i)$ holds due to $\bm{u}_{t}^{(j)}$ and $|\nabla F(\bm{x}_{t}^{j})|$ are mutually independent; $(ii)$ holds owing to applying Condition 2; $(iii)$ holds due to the condition $\bar{v}\leq\min_{t}{\mathbb{E}}[\bm{u}_{t}^{(j)}]$ ; $(iv)$ holds depending on Condition 3.

Then, we simplify the conclusion in Theorem [10](https://arxiv.org/html/2507.05966v1#S3.E10) as

$$ $\displaystyle\left(\bar{v}-\left(\frac{\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{2}+{{2 }C_{0}C_{1}R\sigma_{1}\sqrt{p(1-\beta_{1})}}+\frac{2\gamma C_{1}R^{2}\sqrt{d} qL_{1}}{1-\beta_{1}}\right)\right)\cdot\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E}} [\|\nabla F(\bm{x}_{t})\|_{2}]$ (56) $\displaystyle\leq$ $\displaystyle\frac{C_{1}(F(\bm{x}_{0})-F^{*})}{\gamma T\sqrt{d}}+\frac{2C_{1}R \left\|\nabla F(\bm{x}_{0})\right\|_{2}}{T(1-\beta_{1})}+{2C_{1}\sqrt{1-\beta_ {1}}R}\hat{\sigma}+\frac{2\gamma C_{1}R^{2}\sqrt{d}\hat{L}}{1-\beta_{1}}+\frac {\gamma C_{1}R^{2}\sqrt{d}\hat{L}}{2T}.$ $$

Choosing $\gamma=\frac{C_{2}}{T^{\nicefrac{{3}}{{4}}}d^{\nicefrac{{1}}{{2}}}}$, $1-\beta_{1}=\frac{C_{3}}{T^{\nicefrac{{1}}{{2}}}}$ and $T\geq(\frac{4C_{2}R^{2}qL_{1}}{C_{3}\bar{v}}+{4}C_{0}C_{1}\sqrt{C_{3}}R\sigma_
{1}\sqrt{p}+(\frac{C_{2}R^{2}qL_{1}}{\bar{v}})^{\nicefrac{{1}}{{3}}})^{4}$, following Lemma [B.4](https://arxiv.org/html/2507.05966v1#A2.Thmtheorem4), it holds that

$$ $\frac{\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{2}+{{2}C_{0}C_{1}R\sigma_{1}\sqrt{p(1- \beta_{1})}}+\frac{2\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{1-\beta_{1}}\leq\frac{ \bar{v}}{2}.$ (57) $$

Then, we reformulate Eq. ([56](https://arxiv.org/html/2507.05966v1#A3.E56)) as

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2 }]\leq$ $\displaystyle\frac{C_{1}}{\bar{v}}\left(\frac{2(F(\bm{x}_{0})-F(\bm{x}^{*}))}{ C_{2}T^{\nicefrac{{1}}{{4}}}d^{\nicefrac{{1}}{{2}}}}+\frac{4R\left\|\nabla F( \bm{x}_{0})\right\|_{2}}{C_{3}T^{\nicefrac{{1}}{{2}}}}+\frac{4C_{3}R\hat{ \sigma}}{T^{\nicefrac{{1}}{{4}}}}+\frac{4C_{2}R^{2}\hat{L}}{C_{3}T^{\nicefrac{ {1}}{{4}}}}+\frac{C_{2}R^{2}\hat{L}}{T^{\nicefrac{{7}}{{4}}}}\right).$ (58) $$

(2) Using generalized Young’s inequality, we minimize the bottle-neck terms to obtain the lowerest bound on the right hand of Eq. ([56](https://arxiv.org/html/2507.05966v1#A3.E56)), *i.e.*,

$$ $\displaystyle\frac{C_{1}(F(\bm{x}_{0})-F^{*})}{\gamma T\sqrt{d}}+{2C_{1}\sqrt{ 1-\beta_{1}}R}\hat{\sigma}+\frac{2C_{1}\gamma R^{2}\sqrt{d}\hat{L}}{1-\beta_{1}}$ (59) $\displaystyle\geq$ $\displaystyle\left(\frac{4C_{1}(F(\bm{x}_{0})-F^{*})}{\gamma T\sqrt{d}}\right) ^{\nicefrac{{1}}{{4}}}\cdot\left({4C_{1}\sqrt{1-\beta_{1}}R}\hat{\sigma}\right )^{\nicefrac{{1}}{{2}}}\cdot\left(\frac{8C_{1}\gamma R^{2}\sqrt{d}\hat{L}}{1- \beta_{1}}\right)^{\nicefrac{{1}}{{4}}}$ $\displaystyle=$ $\displaystyle\frac{512^{\nicefrac{{1}}{{4}}}C_{1}R\hat{\sigma}^{\nicefrac{{1}} {{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}(F(\bm{x}_{0})-F^{*})}{T^{\nicefrac{{1}}{{4 }}}},$ $$

where the lowest bound achieved if and only if $\frac{4C_{1}(F(\bm{x}_{0})-F^{*})}{\gamma T\sqrt{d}}={4C_{1}\sqrt{1-\beta_{1}}
R}\hat{\sigma}$ and ${4C_{1}\sqrt{1-\beta_{1}}R}\hat{\sigma}=\frac{8C_{1}\gamma R^{2}\sqrt{d}\hat{L
}}{1-\beta_{1}}$, and we further obtain

$$ $\displaystyle\gamma=\frac{(F(\bm{x}_{0})-F^{*})^{\nicefrac{{3}}{{4}}}}{2^{ \nicefrac{{1}}{{4}}}T^{\nicefrac{{3}}{{4}}}d^{\nicefrac{{1}}{{2}}}R\hat{\sigma }^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}},$ (60) $\displaystyle\beta_{1}=1-\frac{2^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{ {2}}}(F(\bm{x}_{0})-F^{*})^{\nicefrac{{1}}{{2}}}}{T^{\nicefrac{{1}}{{2}}}\hat{ \sigma}}.$ $$

When it is chosen $T\geq(\frac{4C_{0}C_{1}\hat{C}_{3}^{\nicefrac{{1}}{{2}}}p^{\nicefrac{{1}}{{2}}
}R\sigma_{1}}{v}+\frac{4C_{1}\hat{C_{2}}R^{2}qL_{1}}{\hat{C}_{3}v}+(\frac{C_{1
}\hat{C}_{2}R^{2}qL_{1}}{v})^{\nicefrac{{1}}{{3}}})^{4}$ where $\hat{C}_{2}=\frac{(F(\bm{x}_{0})-F^{*})^{\nicefrac{{3}}{{4}}}}{2^{\nicefrac{{1
}}{{4}}}R\hat{\sigma}^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}}$ and $C_{3}=\frac{2^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{2}}}(F(\bm{x}_{0})
-F^{*})^{\nicefrac{{1}}{{2}}}}{\hat{\sigma}}$, following Lemma [B.4](https://arxiv.org/html/2507.05966v1#A2.Thmtheorem4), it holds that

$$ $\frac{\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{2}+{{2}C_{0}C_{1}R\sigma_{1}\sqrt{p(1- \beta_{1})}}+\frac{2\gamma C_{1}R^{2}\sqrt{d}qL_{1}}{1-\beta_{1}}\leq\frac{v}{ 2}.$ (61) $$

Then, we reformulate Eq. ([56](https://arxiv.org/html/2507.05966v1#A3.E56)) as

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\bm{x}_{t})\|_{2 }]\leq$ $\displaystyle\frac{C_{1}}{\bar{v}}\left(\frac{512^{\nicefrac{{1}}{{4}}}R\hat{ \sigma}^{\nicefrac{{1}}{{2}}}\hat{L}^{\nicefrac{{1}}{{4}}}(F(\bm{x}_{0})-F^{*} )}{T^{\nicefrac{{1}}{{4}}}}+\frac{2C_{1}\hat{C}_{3}R\left\|\nabla F(\bm{x}_{0} )\right\|_{2}}{T^{\nicefrac{{3}}{{4}}}}+\frac{\hat{C_{2}}R^{2}\hat{L}}{T^{ \nicefrac{{7}}{{4}}}}\right).$ (62) $$

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 2
presentation: 1
questions: - Could Conditions 1-3 and the lower bound condition be further validated through more comprehensive experiments?

 - See the weakness part.
soundness: 2
strengths: - Studies Adam convergence from **sign-gradient-based algorithms** perspective
- Builds on motivation from Balles and Hennig (2018) and related works, as noted on Page 3.
summary: The paper recasts Adam as a sign-like descent algorithm by grouping the update magnitude $u_t = m_t / (\sqrt{v_t} + \epsilon)$ and proving an $ O(1/\sqrt{T})$ non-convex convergence rate that, according to the authors, is (i) dimension-free, (ii) independent of the stabilising constant $\epsilon$, and (iii) obtained under generalized smoothness assumptions.

The derived results also need some extra new introduced conditions. 

The main results are Theorem 2 and Corollary 3.

The key assumptions include:

- Generalized ( p )-affine variance of stochastic gradients,
- ((L_0, L_1, q))-smoothness of the objective function,
- Conditions 1–3, and
- A lower bound condition on the normalized momentum:
$ \mathbb{E}\frac{|m_t^j|}{\sqrt{v_t^j + \epsilon}} \geq \bar{\nu} > 0. $

Under these assumptions, the derived results establish upper bounds on different convergence measures, including:

- Theorem 3 (without requiring Conditions 2–3): $ {1\over T} \left( \sum_{t=1}^{T-1} E |u_t \nabla F(x_t)|_1-c\sum_{t=1}^{T-1} E |\nabla F(x_t)|_2 \right).$
- Corollary 3: $ \frac{1}{T} \sum_{t=1}^{T-1} \mathbb{E} |\nabla F(x_t)|_2. $
weaknesses: - The measure ${1\over T} \sum_{t=1}^{T-1} \mathbb{E}|u_t \nabla F(x_t)|_1 $ in Theorem 2 could be meaningless, as $u_t^j$ in some cases could be very small.

- Conditions 1-3 and the lower bound condition $E |m_t^j| /\sqrt{v_t^j + \epsilon} \geq \bar{v} >0$ are unnatural and strong. All existing convergence analysis for Adam in the nonconvex smooth optimization do not involve these assumptions.

- The comparisons with state-of-the-art results in Table 1 are unfair because the authors do not account for the new additional assumptions (Conditions 1-3 and the lower bound condition $E |m_t^j| /\sqrt{v_t^j + \epsilon} \geq \bar{v} >0$) introduced in this paper.

- The upper bound for Adam in Corollary 3 depends on ${1 \over \bar{v}}$, where $E |m_t^j| /\sqrt{v_t^j + \epsilon} \geq \bar{v} >0$, which could be very large in some cases.

- The numerical results for supporting Conditions 1-3 and the lower bound condition are limited. And the constant setup on $\beta_1$ and $\beta_2$ in numerical results seems not to be consistent with the theoretical results on $\beta_1$ (or $\beta_2$) which are depending on the total number of iterations.

- "$|\nabla F(x_t)|_2$ decreases at the rate $1/t^{\alpha}$" from  Proposition 1 is wrongly stated. According to the proof in Appendix, the authors essentially mean that $c/t^{\alpha} \leq |\nabla F(x_t)|_2  \leq C /t^{\alpha}$.

- Proposition 1 is insufficient to support the theoretical validity of Condition 1. The authors' logic appears circular: they use Condition 1 to prove that $|\nabla F(x_t)|_2  \leq C /t^{\alpha}$, and then justify the reasonableness of Condition 1 based on the resulting bound $c/t^{\alpha} \leq |\nabla F(x_t)|_2  \leq C /t^{\alpha}$.

- Some of the statement are wrongly stated or over-claimed. Take a few as example as follows.
  - Abstract, the mild conditions in the statement `For the first time, with some mild conditions, we prove that Adam achieves' are not truly mild.
  - Abstract, `Additionally, our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence'. `The role of momentum as a key factor ensuring convergence' is now a relatively  well known result. See e.g. (Xie S, Mohamadi M A, Li Z. Adam Exploits $\ell_\infty $-geometry of Loss Landscape via Coordinate-wise Adaptivity[J]. arXiv preprint arXiv:2410.08198, 2024; Li H, Dong Y, Lin Z. On the $ O (\frac {\sqrt {d}}{T^{1/4}}) $ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1 $ Norm[J]. arXiv preprint arXiv:2402.00389, 2024; ) and the earlier related papers.  In fact, (15) is similar to equations found in some of the literature, but the authors do not mention this."
  - Line 101, `Recently, Muon Jordan et al. (2024), an extended matrix-sign optimizer'. The statement matrix-sign is not precise.
  - Lines 633-634,  'Shi et al. (2021) and Zhang et al. (2022) respectively proved random-shuffled AMSProp and Adam will
converge to the neighbourhood of stationary points with the rate'. The latter  can also derive similar convergence rates if one chooses $\beta_1$ and $\beta_2$ that are depending on $T$.
  - Lines 658-659, `Eq. (41), Eq. (56) and Eq. (58) in the proof suggest that...'.  The authors fail to note that $\eta = \sqrt{1-\beta_2}= 1/\sqrt{T}$ in that paper. Thus, the bounds (41), (56), and (58) (which are also related to $\eta$) are $\mathcal{O}(\log T)$.
  - Please revise Page 20 to use space more efficiently. We also identified similar issues with excessive spacing in other sections.
  - The other places...

The manuscript appears to be underprepared; it requires thorough proofreading, revision, and a further round of review.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: There are some confusing notations in this paper, which deserve a more careful check.

- The notation $\mathbb{E}$ is confusing. For example, in Eq. (6) and (7), the expectation should be the conditional expectation given $x$, and in Eq. (12) and (13), it's the conditional expectation given $x_t$. However, $\mathbb{E}$ in convergence results refers to the total expectation.

- the notations $u_{t}^{(j)}$ and ${\bf u}_t^{(j)}$ are confused. Usually, the scalar terms use the light notation while the vector or matrix terms use the bold notation.
soundness: 2
strengths: (a). The major contribution in this paper lies in the theoretical part, where the convergence rate for Adam is established under the general weak assumptions, which are more realistic to reflect the practical training.

(b). The convergence rate is tight, and the result is novel to my best knowledge, in the context of general weak assumptions.

(c). The treatment of Adam as a sign-like gradient descent, although not proposed at the first time, is still novel in the convergence analysis of Adam in non-convex stochastic optimization.
summary: This paper provides a convergence proof for Adam from the perspective of sign-like descent instead of the commonly used preconditioned stochastic gradient descent with momentum. Based on this reformulation, they are able to prove the optimal rate of $O(T^{-1/4})$ for Adam under
the generalized $p$-affine variance and $(L\_0, L\_1, q)$-smoothness, without dependence on the model dimensionality or the numerical stability parameter.
weaknesses: My major concerns also lie in the theoretical part.

It's clear to see that the convergence rate is established based not only on the weak assumptions such as generalized smoothness and affine noise, but also heavily on Conditions 1-3. These conditions, however, do not sound so natural and reasonable. Specifically,
- Condition 1: Although the authors propose Proposition 1 to illustrate this condition, the assumption $\\|\nabla F(x\_t) \\|\_2$ decreases with $O(1/t^{\alpha})$ is not so reasonable. Existing results, such as Arjevani et al. (2023), only provide the guarantee of $O(1/T^{1/4})$ for $\min_{t \in [T]} \\|\nabla F(x\_t) \\|\_2$ or $\sum\_{t=1}^T \\|\nabla F(x\_t) \\|\_2/T$ instead of the precise description on the decreasing rate. Moreover, in the practical training, $\\|\nabla F(x\_t) \\|\_2$ usually does not follow such a decreasing formula.
- Condition 2 (I): The i.i.d. assumption on each coordinate of the update is not convincing. The authors provide a two-sample K-S test to illustrate this assumption. However, to my best knowledge, the K-S test is used to verify whether two distributions are the same or not. Also, in practical training, the i.i.d. is usually satisfied. 

- Condition 2 (II): the lower bound assumption of update is one of the central points in the convergence result, showing in the term $\bar{v} = \min_{t \in [T]} E[u\_t^{(j)}]$. First, the lower bound is not commonly used in literature, perhaps only in (Guo et al., 2020). This approach sidesteps the key difficulty in analyzing Adam, such as the correlation of adaptive stepsize and stochastic gradient. Second, even though the lower bound exists, the specific value of $\bar{v}$ remains unknown. It's not convincing to treat it as a constant without any further theoretical or empirical evidence.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Do you have any numerical evidence to support Condition 1?
2. Numerically, do you find that $\bar{v}$ from Condition 2 depends on $d$ or $T$?
3. Am I correct in understanding that Theorem 4 still depends on Condition 2?
soundness: 1
strengths: 1. The problem considered is important. The convergence of Adam is a long-standing area of research that is very relevant to deep learning in general.
2. The paper is well-written and clear about its stated contributions.
summary: This paper presents a convergence analysis of Adam for (generalized) smooth, non-convex functions with (generalized) affine variance, assuming some additional conditions. The analysis works by considering Adam as a kind of adaptive sign descent, rather than as a preconditioned gradient descent. Some experimental support for the additional conditions is also provided.
weaknesses: 1. The reliance on the additional conditions (Conditions 1-3) is a significant weakness for several reasons. At best these conditions obscure the results, and at worst the conditions are simply not valid.

    a. I did not find any convincing evidence for Condition 1, as numerical validation is only provided for Conditions 2 and 3. This is odd considering that Condition 1 seems like it would be quite simple to verify/disprove numerically. Proposition 1 is intended to provide theoretical support for Condition 1, but Proposition 1 depends on the assumption that $\|\nabla F(x_t)\|_2 = \Theta(t^{-\alpha})$. Not only does this assumption assume convergence to an approximate stationary point (which the paper aims to prove), but it also assumes a lower bound on this rate of convergence, for which I see no evidence at all. The discussion on Lines 221-224 is not convincing to me.

    b. From Condition 2, I agree that the variable $\bar{v}$ (defined as the lower bound of $\bar{u}_t$) should be larger than $0$, but a subtle issue is that $\bar{v}$ can depend on $d$ or $T$. For example, what if for some particular $F$ it holds that $\bar{v} = 1/d$ or $\bar{v} = 1/T$? Since the convergence rates (e.g. Equations 10 and 11) have a coefficient of $1/\bar{v}$ in front, this means that the rate would be $d/T^{1/4}$ when $\bar{v} = 1/d$ or $T^{3/4}$ when $\bar{v} = 1/T$! So it is not even clear to me that the convergence rates are non-vacuous. This is why I previously said that the conditions obscure the results.

    c. As stated, Condition 3 simply cannot be true. Condition 3 assumes that the ratio $\|\nabla F(x_t)\|_1 / \|\nabla F(x_t\|_2$ is constant across iterations $t$, which is definitely not true. Perhaps this condition can be weakened to assume that this ratio is upper and lower bounded by two different constants, but this would require significant modifications to the proof. To be clear: I am not requesting that the authors modify their proof to address this during the rebuttal window, as this seems like a tedious task that may not be worth the effort.

    d. Theorem 4 is supposed to show convergence without assuming Conditions 1-3, but it seems that Theorem 4 still uses Condition 2 through the variable $v$. In any case, the rate from Theorem 4 depends on $d^{1/2}$, which is a significantly weaker result than the main result.

2. There are several instances of unclear or exaggerated language which I think should be clarified. First, the paper is framed as relying on weaker assumptions than previous work, but as I pointed out in my Weakness #1, this paper relies on several dubious conditions not used in prior work. Second, it is claimed several times that the "sign descent" perspective simplifies the proof process, but it isn't clear to me that this is the case: the proof in this paper looks comparably long and complicated as previous analyses of Adam. Third, it is claimed several times throughout the paper that the interpretation of Adam as an adaptive sign descent is a novel perspective, e.g. in the conclusion: "This work breaks with convention and provides a pioneering reinterpretation of Adam as a sign-like descent algorithm". I think this is an inappropriate exaggeration, given that Balles & Hennig (2018) (and probably several other works since) have already established this perspective, even if they didn't use this perspective to develop new convergence proofs. Calibrating the tone of these claims will strengthen the paper.

Typos/small suggestions:
- As stated, the second part of Condition 2 ($\bar{u}_t > 0$) is trivially true. It seems like you may instead want $\bar{u}_t \geq \bar{v} > 0$.
- Caption of Table 1 refers to Theorem B.6. I think you mean Theorem 4.

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Can you justify the extra conditions?
soundness: 2
strengths: 1. Although it is widely believed that Adam is close to sign descent, this paper provides a set of conditions to ensure when such a perspective is valid (Condition 1, 2, 3, decreasing rate of gradient norm over time in Proposition 1).

2. The presentation of this paper is good, with a comprehensive comparison with prior work.
summary: This paper provides a convergence proof for Adam from the perspective of sign-like descent, which simplifies the convergence analysis under generalized-smooth nonconvex optimization.
weaknesses: 1. Some of the these extra conditions the authors assume are not reasonable. For example, the authors assume that the gradient norm is decreasing over time with certain rate in Proposition 1 to argue that Condition 1 commonly holds in practice. However, the gradient norm may not decrease over time in practice and in theory. For example, even the averaged gradient decrease over time, it does not indicate that the gradient itself is monotonically decreasing.

2. Despite the numerical experiments of hypothesis testing, I do not think Condition 2 is reasonable. Adam’s coordinate updates within each layer are neither independent nor identically distributed. In practice, all weight gradients in a layer share common input activations and backpropagated errors, which induces strong correlations across coordinates. In addition, Condition 2 assumes i.i.d. condition for **every coordinate**, not just within a single layer. This is unrealistic, as coordinates from adjacent layers are inherently statistically dependent due to the recursive structure of backpropagation.