SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bohan Wang $^{1*}$ , Jingwen Fu $^{2*}$ , Huishuai Zhang $^{3\dagger}$ , Nanning Zheng $^{2\dagger}$ , Wei Chen $^{4\dagger}$
bhwangfy@gmail.com
fu1371252069@stu.xjtu.edu.cn
huzhang@microsoft.com
nnzheng@mail.xjtu.edu.cn
chenwei2022@ict.ac.cn
<sup>1</sup>University of Science and Technology of China, <sup>2</sup>Xi'an Jiaotong University,
<sup>3</sup>Microsoft Research, <sup>4</sup>Institute of Computing Technology, Chinese Academy of Sciences
Recently, Arjevani et al. [1] establish a lower bound of iteration complexity for the first-order optimization under an  $L$ -smooth condition and a bounded noise variance assumption. However, a thorough review of existing literature on Adam's convergence reveals a noticeable gap: none of them meet the above lower bound. In this paper, we close the gap by deriving a new convergence guarantee of Adam, with only an  $L$ -smooth condition and a bounded noise variance assumption. Our results remain valid across a broad spectrum of hyperparameters. Especially with properly chosen hyperparameters, we derive an upper bound of iteration complexity of Adam and show that it meets the lower bound for first-order optimizers. To the best of our knowledge, this is the first to establish such a tight upper bound for Adam's convergence. Our proof utilizes novel techniques to handle the entanglement between momentum and adaptive learning rate and to convert the first-order term in the Descent Lemma to the gradient norm, which may be of independent interest.
First-order optimizers, also known as gradient-based methods, make use of gradient (first-order derivative) information to find the minimum of a function. They have become a cornerstone of many machine learning algorithms due to the efficiency as only gradient informaiton is required, and the flexibility as gradients can be easily computed for any function represented as directed acyclic computational graph via auto-differentiation [2, 25].
Therefore, it is fundamental to theoretically understand the properties of these first-order methods. Recently, Arjevani et al. [1] establish a lower bound on the iteration complexity of stochastic first-order methods. Formally, for a well-studied setting where the objective is  $L$ -smooth and a stochastic oracle can query the gradient unbiasedly with bounded variance (see Assumption 1 and 2), any stochastic first-order algorithm requires at least  $\varepsilon^{-4}$  queries (in the worst case) to find an  $\varepsilon$ -stationary point, i.e., a point with gradient norm at most  $\varepsilon$ . Arjevani et al. [1] further show that the above lower bound is tight as it matches the existing upper bound of iteration complexity of SGD [15].
On the other hand, among first-order optimizers, Adam [20] becomes dominant in training state-of-the-art machine learning models [3, 18, 4, 11]. Compared to vanilla stochastic gradient descent (SGD), Adam consists of two more key components: (i) momentum to accumulate historical gradient
† Corresponding Authors
information and (ii) adaptive learning rate to rectify coordinate-wise step sizes. The pseudo-code of Adam is given as Algorithm 1. While the sophisticated design of Adam enables its empirical superiority, it brings great challenges for the theoretical analysis. After examining a series of theoretical works on the upper bound of iteration complexity of Adam [33, 9, 10, 36, 16, 27, 34], we find that none of them match the lower bound for first-order optimizers: they not only consume more queries than the lower bound to reach  $\varepsilon$ -stationary iterations but also requires additional assumptions (see Section 3 for a detailed discussion).
This theoretical mismatch becomes even more unnatural given the great empirical advantage of Adam over SGD, which incites us to think:
Is the gap between the upper and lower bounds for Adam a result of the inherent complexity induced by Adam's design, or could it be attributed to the proof techniques not being sharp enough?
This paper answers the above question, validating the latter hypothesis, by establishing a new upper bound on iteration complexity of Adam for a wide range of hyperparameters that cover typical choices. Specifically, our contribution can be summarized as follows:
- We examine existing works that analyze the iteration complexity of Adam, and find that none of them meets the lower bound of first-order optimization algorithms;
- We derive a new convergence guarantee of Adam with only assuming  $L$ -smooth condition and bounded variance assumption (Theorem 1), which holds for a wide range of hyperparameters covering typical choices;
- With chosen hyperparameters, we further tighten Theorem 1 and show that the upper bound on the iteration complexity of Adam meets the lower bound, closing the gap (Theorem 2). Our upper bound is tighter than existing results by a logarithmic factor, in spite of weaker assumption.
To the best of our knowledge, this work provides the first upper bound on the iteration complexity of Adam without additional assumptions other than  $L$ -smooth condition and bounded variance assumption. It is also the first upper bound matching the lower bound of first-order optimizers.
Organization of this paper. The rest of the paper is organized as follows: in Section 2, we first present the notations and setup of analysis in this paper; in Section 3, we revisit the existing works on the iteration complexity of Adam; in Section 4, we present a convergence analysis of Adam with general hyperparameters (Theorem 1); in Section 5, we tighten Theorem 1 with a chosen hyperparameter, and derive an upper bound of Adam's iteration complexity which meets the lower bound; in Section 6, we discuss the limitation of our results.
The Adam algorithm is restated in Algorithm 1 for convenient reference. Note that compared to the original version of Adam in Kingma and Ba [20], the bias-correction terms are omitted to simplify the analysis, and our analysis can be immediately extended to the original version of Adam because the effect of bias-correction term decays exponentially. Also, in the original version of Adam, the adaptive learning rate is  $\frac{\eta}{\sqrt{\nu_t} + \lambda\mathbb{I}_d}$  instead of  $\frac{\eta}{\sqrt{\nu_t}}$ . However, our setting is more challenging and our result can be easily extend to the original version of Adam, since the  $\lambda$  term makes the adaptive learning rate upper bounded and eases the analysis.
Algorithm 1 Adam
Input: Stochastic oracle  $\mathcal{O}$ , learning rate  $\eta > 0$ , initial point  $\boldsymbol{w}_1 \in \mathbb{R}^d$ , initial conditioner  $\nu_0 \in \mathbb{R}^+$ , initial momentum  $m_0$ , momentum parameter  $\beta_1$ , conditioner parameter  $\beta_2$ , number of epoch  $T$
1: Sample  $r \sim \mathrm{Unif}\{1, \dots, T\}$
2: For  $t = 1 \rightarrow T$ :
3: Generate a random  $z_{t}$ , and query stochastic oracle  $g_{t} = O_{f}(w_{t},z_{t})$
4: Calculate  $\pmb{\nu}_{t} = \beta_{2}\pmb{\nu}_{t - 1} + (1 - \beta_{2})\pmb{g}_{t}^{\odot 2}$
5: Calculate  $\pmb{m}_t = \beta_1\pmb{m}_{t - 1} + (1 - \beta_1)\pmb{g}_t$
6: Update  $\pmb{w}_{t + 1} = \pmb{w}_t - \eta \frac{1}{\sqrt{\pmb{\nu}_t}}\odot \pmb{m}_t$
7: EndFor
Output:  $w_{r}$
Notations. For  $a, b \in \mathbb{Z}^{\geq 0}$  and  $a \leq b$ , denote  $[a, b] = \{a, a + 1, \dots, b - 1, b\}$ . For any two vectors  $\boldsymbol{w}, \boldsymbol{v} \in \mathbb{R}^{d}$ , denote  $\boldsymbol{w} \odot \boldsymbol{v}$  as the Hadamard product (i.e., coordinate-wise multiplication) between  $\boldsymbol{w}$  and  $\boldsymbol{v}$ . When analyzing Adam, we denote the true gradient at iteration  $t$  as  $\pmb{G}_{t} = \nabla f(\pmb{w}_{t})$ , and the sigma algebra before iteration  $t$  as  $\mathcal{F}_{t} = \sigma(\pmb{g}_{1}, \dots, \pmb{g}_{t-1})$ . We denote conditional expectation as  $\mathbb{E}^{|F_{t}|}[*] = \mathbb{E}[*|\mathcal{F}_{t}]$ . We also use asymptotic notations  $\boldsymbol{o}$ ,  $\mathcal{O}$ ,  $\Omega$ , and  $\Theta$ , where  $h_{2}(x) = o_{x \to x_{0}}(h_{1}(x))$  means that  $\lim_{x \to x_{0}} \frac{h_{2}(x)}{h_{1}(x)} = 0$  (when the context is clear, we abbreviate  $x \to x_{0}$  and only use  $\boldsymbol{o}(h_{1}(x))$ ;  $h_{2}(x) = \mathcal{O}(h_{1}(x))$  means that there exists constant  $\gamma$  independent of  $x$  such that  $h_{2}(x) \leq \gamma h_{1}(x)$ ;  $h_{2}(x) = \Omega(h_{1}(x))$  means that  $h_{1}(x) = \mathcal{O}(h_{2}(x))$ ; and  $h_{2}(x) = \Theta(h_{1}(x))$  means that  $h_{2}(x) = \mathcal{O}(h_{1}(x))$  and  $h_{2}(x) = \Omega(h_{1}(x))$ ).
Objective function. In this paper, we consider solving the following optimization problem:  $\min_{\boldsymbol{w} \in \mathbb{R}^d} f(\boldsymbol{w})$ . We make the following assumption on the objective function  $f$ .
Assumption 1 (On objective function). We assume  $f$  to be non-negative. We further assume that  $f$  satisfies  $L$ -smooth condition, i.e.,  $f$  is differentiable, and the gradient of  $f$  is  $L$ -Lipschitz.
We denote the set of all objective functions satisfying Assumption 1 as  $\mathcal{F}(L)$ .
Stochastic oracle. As  $f$  is differentiable, we can utilize the gradient of  $f$  (i.e.,  $\nabla f$ ) to solve the above optimization problem. However, the  $\nabla f$  is usually expensive to compute. Instead, we query a stochastic estimation of  $\nabla f$  through a stochastic oracle  $O$ . Specifically, the stochastic oracle  $O$  consists of a distribution  $\mathcal{P}$  over a measurable space  $\mathcal{Z}$  and a mapping  $O_f: \mathbb{R}^d \times \mathcal{Z} \to \mathbb{R}^d$ . We make the following assumption on  $O$ .
Assumption 2 (On stochastic oracle). We assume that  $\pmb{O}$  is unbiased, i.e.,  $\forall \pmb{w} \in \mathbb{R}^d$ ,  $\mathbb{E}_{z \sim \mathcal{P}} \pmb{O}_f(\pmb{w}, z) = \nabla f(\pmb{w})$ . We further assume  $\pmb{O}$  has bounded variance, i.e.,  $\forall \pmb{w} \in \mathbb{R}^d$ ,  $\mathbb{E}_{z \sim \mathcal{P}}[\| \pmb{O}_f(\pmb{w}, z) - \nabla f(\pmb{w})\|^2] \leq \sigma^2$ .
We denote the set of all stochastic oracles satisfying Assumption 2 with variance bound  $\sigma^2$  as  $\mathfrak{D}(\sigma^2)$ .
Algorithm. Adam belongs to first-order optimization algorithms, which is defined as follows:
Definition 1 (First-order optimization algorithm). An algorithm  $\mathbf{A}$  is called a first-order optimization algorithm, if it takes an input  $\mathbf{w}_1$  and hyperparameter  $\theta$ , and produces a sequence of parameters as follows: first sample a random seed  $r$  from some distribution  $\mathcal{P}_r^*$ , set  $\mathbf{w}_1^{\mathbf{A}(\theta)} = \mathbf{w}_1$  and then update the parameters as
$$
\boldsymbol {w} _ {t + 1} ^ {\boldsymbol {A} (\theta)} = \boldsymbol {A} _ {\theta} ^ {t} (r, \boldsymbol {w} _ {1} ^ {\boldsymbol {A} (\theta)}, \boldsymbol {O} _ {f} (\boldsymbol {w} _ {1} ^ {\boldsymbol {A} (\theta)}, z _ {1}), \dots , \boldsymbol {O} _ {f} (\boldsymbol {w} _ {t} ^ {\boldsymbol {A} (\theta)}, z _ {t})),
$$
where  $z_{1},z_{2},\dots ,z_{t}$  are sampled i.i.d. from  $\mathcal{P}$
Iteration complexity. Denote the set of all first-order optimization algorithms as  $\mathcal{A}_{\mathrm{first}}$ . We next introduce iteration complexity to measure the convergence rate of optimization algorithms.
Definition 2 (Iteration complexity). The iteration complexity of first-order optimization algorithm  $A$  is defined as
$$
\mathcal{C}_{\varepsilon}(\boldsymbol {A},\Delta ,L,\sigma^{2}) = \sup_{\boldsymbol {O}\in \mathfrak{D}(\sigma^{2})}\sup_{f\in \mathcal{F}(L)}\sup_{\boldsymbol{w}_{1}:f(\boldsymbol{w}_{1}) = \Delta}\inf_{\theta}\{T:\mathbb{E}\| \nabla f(\boldsymbol{w}_{T}^{\boldsymbol{A}(\theta)})\| \leq \varepsilon \} .
$$
*Such a random seed allows sampling from all iterations to generate the final output of the optimization algorithm. As an example, Algorithm 1 sets  $\mathcal{P}_r$  as a uniform distribution over  $[T]$ .
Furthermore, the iteration complexity of the family of first-order optimization algorithms  $\mathcal{A}_{\mathrm{first}}$  is
$$
\mathcal {C} _ {\varepsilon} (\Delta , L, \sigma^ {2}) = \sup  _ {\boldsymbol {O} \in \mathfrak {D} (\sigma^ {2})} \sup  _ {f \in \mathcal {F} (L)} \sup  _ {\boldsymbol {w} _ {1}: f (\boldsymbol {w} _ {1}) = \Delta} \inf  _ {\boldsymbol {A} \in \mathcal {A} _ {\mathrm {f i r s t}}} \inf  _ {\theta} \{T: \mathbb {E} \| \nabla f (\boldsymbol {w} _ {T} ^ {\boldsymbol {A} (\theta)}) \| \leq \varepsilon \}.
$$
It should be noticed that the iteration complexity of the family of first-order optimization algorithms is a lower bound of the iteration complexity of a specific first-order optimization algorithm, i.e.,  $\forall A\in \mathcal{A}_{\mathrm{first}},\mathcal{C}_{\varepsilon}(A,\Delta ,L,\sigma^{2})\geq \mathcal{C}_{\varepsilon}(\Delta ,L,\sigma^{2})$
3 Related works: none of existing upper bounds match the lower bound
In this section, we examine existing works that study the iteration complexity of Adam, and defer a discussion of other related works to Appendix A. Specifically, we find that none of them match the lower bound for first-order algorithms provided in [1] (restated as follows).
Proposition 1 (Theorem 3, [1]).  $\forall L,\Delta ,\sigma^2 >0$  we have  $\mathcal{C}_{\varepsilon}(\Delta ,L,\sigma^{2}) = \Omega (\frac{1}{\varepsilon^{4}})$
Note that in the above bound, we omit the dependence of the lower bound over  $\Delta$ ,  $L$ , and  $\sigma^2$ , which is a standard practice in existing works (see Cutkosky and Mehta [8], Xie et al. [32], Faw et al. [13] as examples) because the dependence over the accuracy  $\varepsilon$  can be used to derive how much additional iterations is required for a smaller target accuracy and is thus of more interest. In this paper, when we say "match the lower bound", we always mean that the upper bound has the same order of  $\varepsilon$  as the lower bound.
Generally speaking, existing works on the iteration complexity of Adam can be divided into two categories: they either (i) assume that gradient is universally bounded or (ii) make stronger assumptions on smoothness. Below we respectively explain how these two categories of works do not match the lower bound in [1].
The first line of works, including Zaheer et al. [33], De et al. [9], Defossez et al. [10], Zou et al. [36], Guo et al. [16], assume that the gradient norm of  $f$  is universally bounded, i.e.,  $\| \nabla f(\boldsymbol{w}) \| \leq G$ ,  $\forall \boldsymbol{w} \in \mathbb{R}^d$ . In other words, what they consider is another iteration complexity defined as follows:
$$
\mathcal{C}_{\varepsilon}(\boldsymbol {A},\Delta ,L,\sigma^{2},G)\triangleq \sup_{\boldsymbol {O}\in \mathfrak{D}(\sigma^{2})}\sup_{f\in \mathcal{F}(L),\| \nabla f\| \leq G}\sup_{\boldsymbol{w}_{1}:f(\boldsymbol{w}_{1}) = \Delta}\inf_{\theta}\{T:\mathbb{E}\| \nabla f(\boldsymbol{w}_{T}^{\boldsymbol{A} (\theta)})\| \leq \varepsilon \} .
$$
This line of works do not match the lower bound due to the following two reasons: First of all, the upper bound they derive is  $O\left(\frac{\log 1 / \varepsilon}{\varepsilon^4}\right)$ , which has an additional  $\log 1 / \varepsilon$  factor more than the lower bound; secondly, the bound they derive is for  $\mathcal{C}_{\varepsilon}(\pmb {A},\Delta ,L,\sigma^{2},G)$ . Note that  $\mathcal{F}(L)\cap \{f:\| \nabla f\| \leq G\}$  is a proper subset of  $\mathcal{F}(L)$  for any  $G$ , where a simple example in  $\mathcal{F}(L)$  but without bounded gradient is the quadratic function  $f(x) = \| x\| ^2$ . Therefore, we have that
$$
\mathcal {C} _ {\varepsilon} (\boldsymbol {A}, \Delta , L, \sigma^ {2}) \geq \mathcal {C} _ {\varepsilon} (\boldsymbol {A}, \Delta , L, \sigma^ {2}, G), \quad \forall G \geq 0, \tag {1}
$$
and thus the upper bound on  $\mathcal{C}_{\varepsilon}(\pmb {A},\Delta ,L,\sigma^2,G)$  does not apply to  $\mathcal{C}_{\varepsilon}(\pmb {A},\Delta ,L,\sigma^{2})$  . Moreover, their upper bound of  $\mathcal{C}_{\varepsilon}(\pmb {A},\Delta ,L,\sigma^{2},G)$  tends to  $\infty$  as  $G\to \infty$  , which indicates that if following their analysis, the upper bound of  $\mathcal{C}_{\varepsilon}(\pmb {A},\Delta ,L,\sigma^{2})$  would be infinity based on Eq. (1).
The second line of works [27, 34, 30] additionally assume a mean-squared smoothness property besides Assumption 1 and 2, i.e.,  $\mathbb{E}_{z\sim \mathcal{P}}\| O_f(\boldsymbol {w},z) - O_f(\boldsymbol {v},z)\| ^2\leq L\| \boldsymbol {w} - \boldsymbol {v}\| ^2$  . Denote  $\tilde{\mathfrak{D}} (\sigma^2,L)\triangleq$ $\{O:\mathbb{E}_{z\sim \mathcal{P}}\| O_f(\boldsymbol {w},z) - O_f(\boldsymbol {v},z)\| ^2\leq L\| \boldsymbol {w} - \boldsymbol {v}\| ^2,\forall \boldsymbol {w},\boldsymbol {v}\in \mathbb{R}^d\} \cap \mathfrak{D}(\sigma^2)$  . The iteration complexity that they consider is defined as follows:
$$
\tilde {\mathcal {C}} _ {\varepsilon} (\boldsymbol {A}, \Delta , L, \sigma^ {2}) = \sup  _ {\boldsymbol {O} \in \tilde {\mathfrak {D}} (\sigma^ {2}, L)} \sup  _ {f \in \mathcal {F} (L)} \sup  _ {\boldsymbol {w} _ {1}: f (\boldsymbol {w} _ {1}) = \Delta} \inf  _ {\theta} \{T: \mathbb {E} \| \nabla f (\boldsymbol {w} _ {T} ^ {\boldsymbol {A} (\theta)}) \| \leq \varepsilon \}.
$$
The rate derived in [27, 34, 30] is  $O\left(\frac{\log 1 / \varepsilon}{\varepsilon^6}\right)$ , which is derived by minimizing the upper bounds in [27, 34, 30] with respect to the hyperparameter of adaptive learning rate  $\beta_{2}$ . According to Arjevani et al. [1], the lower bound of iteration complexity of  $\hat{C}_{\varepsilon}(A,\Delta,L,\sigma^{2})$  is  $\Omega \left(\frac{1}{\varepsilon^3}\right)$  and smaller than the original lower bound  $\Omega \left(\frac{1}{\varepsilon^4}\right)$ , resulting in an even larger gap between the upper and lower bounds.
Recently, there is a concurrent work [21] which does not require bounded gradient assumption and mean-squared smoothness property but poses a stronger assumption on the stochastic oracle: the set of stochastic oracles they consider is  $\tilde{\mathfrak{D}} = \{O:\forall \pmb {w}\in \mathbb{R}^d,\mathbb{E}_{z\sim \mathcal{P}}\pmb {O}_f(\pmb {w},z) = \nabla f(\pmb {w}),\mathbb{P}\left(\| \pmb {O}_f(\pmb {w},z) - \nabla f(\pmb {w})\| ^2\leq \sigma^2\right) = 1\}$ .  $\tilde{\mathfrak{D}}$  is a proper subset of  $\mathfrak{D}$  because a simple example is that  $\pmb {O}_f(\pmb {w},z) = \nabla f(\pmb {w}) + z$  where  $z$  is a standard gaussian variable. Therefore, their result does not provide a valid upper bound of  $\mathcal{C}_{\varepsilon}(A,\Delta ,L,\sigma^{2})$ .
4 Convergence analysis of Adam with only Assumptions 1 and 2
As discussed in Section 3, existing works on analyzing Adam require additional assumptions besides Assumption 1 and 2. In this section, we provide the first convergence analysis of Adam with only Assumption 1 and 2, which naturally gives an upper bound on the iteration complexity  $\mathcal{C}_{\varepsilon}(A,\Delta ,L,\sigma^2)$ . In fact, our analysis even holds when the stochastic oracle satisfies the following more general assumption.
Assumption 3 (Coordinate-wise affine noise variance). We assume that  $\pmb{O}$  is unbiased, i.e.,  $\forall \pmb{w} \in \mathbb{R}^d$ ,  $\mathbb{E}_{z \sim \mathcal{P}} \pmb{O}_f(\pmb{w}, z) = \nabla f(\pmb{w})$ . We further assume  $\pmb{O}$  has coordinate-wise affine variance, i.e.,  $\forall \pmb{w} \in \mathbb{R}^d$  and  $\forall i \in [d]$ ,  $\mathbb{E}_{z \sim \mathcal{P}}[|(\pmb{O}_f(\pmb{w}, z))_i|^2] \leq \sigma_0^2 + \sigma_1^2 \partial_i f(\pmb{w})^2$ .
One can easily observe that Assumption 3 is more general than Assumption 2 since Assumption 2 immediately indicates Assumption 3 with  $\sigma_0 = \sigma$  and  $\sigma_{1} = 1$ . We consider Assumption 3 not only because it is more general but also because it allows the noise to grow with the norm of the true gradient, which is usually the case in machine learning practice [14, 19].
Our analysis under Assumption 1 and Assumption 3 is then given as follows.
Theorem 1. Let  $\mathbf{A}$  be by Adam (Algorithm 1) and  $\theta = (\eta, \beta_1, \beta_2)$  are the hyperparameters of  $\mathbf{A}$ . Let Assumption 1 and 2 hold. Then, if  $0 \leq \beta_1 \leq \sqrt{\beta_2} - 8\sigma_1^2(1 - \beta_2)\beta_2^{-2}$  and  $\beta_2 < 1$ , we have
$$
\begin{array}{l} \mathbb {E} \sum_ {t = 1} ^ {T} \| \nabla f (\boldsymbol {w} _ {t}) \| \leq \sqrt {C _ {2} + 2 C _ {1} \sum_ {i = 1} ^ {d} \left(\ln \left(2 (T + 1) \sum_ {i = 1} ^ {d} \sqrt {\nu_ {0 , i} + \sigma_ {0} ^ {2}} + 2 4 d \frac {\sigma_ {1} ^ {2} C _ {1}}{\sqrt {\beta_ {2}}} \ln d \frac {\sigma_ {1} ^ {2} C _ {1}}{\sqrt {\beta_ {2}}} + \frac {1 2 \sigma_ {1} ^ {2}}{\sqrt {\beta_ {2}}} C _ {2}\right)\right)} \\ \times \sqrt {2 (T + 1) \sum_ {i = 1} ^ {d} \sqrt {\nu_ {0 , i} + \sigma_ {0} ^ {2}} + 2 4 d \frac {\sigma_ {1} ^ {2} C _ {1}}{\sqrt {\beta_ {2}}} \ln d \frac {\sigma_ {1} ^ {2} C _ {1}}{\sqrt {\beta_ {2}}} + \frac {1 2 \sigma_ {1} ^ {2}}{\sqrt {\beta_ {2}}} C _ {2}}. \tag {2} \\ \end{array}
$$
where  $\pmb{\nu}_{0,i}$  is the  $i$ -th coordinate of  $\pmb{\nu}_0$
$$
C _ {1} = \frac {3 2 L \eta \left(1 + \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {3}}{\left(1 - \beta_ {2}\right) \left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {3}} + \frac {1 6 \beta_ {1} ^ {2} \sigma_ {0} (1 - \beta_ {1})}{\beta_ {2} \sqrt {1 - \beta_ {2}} \left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {3}} + \frac {6 4 (1 + \sigma_ {1} ^ {2}) \sigma_ {1} ^ {2} L ^ {2} \eta^ {2} d}{\beta_ {2} ^ {2} \left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {4} \sigma_ {0} (1 - \beta_ {2}) ^ {\frac {3}{2}}},
$$
$$
C _ {2} = \frac {1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}}{1 - \beta_ {1}} \frac {8}{\eta} f (\pmb {u} _ {1}) + \frac {3 2}{\beta_ {2} \left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\pmb {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\nu} _ {1 , i}}} + 2 C _ {1} \sum_ {i = 1} ^ {d} \left(\ln \left(\frac {1}{\sqrt {\beta_ {2} \pmb {\nu} _ {0 , i}}}\right) - T \ln \beta_ {2}\right).
$$
A proof sketch is given in Section 4.2 and the full proof is deferred to Appendix.
The right-hand side in Eq. (2) looks messy at the first glance. We next explain Theorem 1 in detail and make the upper bound's dependence over hyperparameters crystally clear.
4.1 Discussion on Theorem 1
Required assumptions and conditions. As mentioned previously, Theorem 1 only requires Assumption 1 and 2, which aligns with the setting of the lower bound (Proposition 1). To our best knowledge, this is the first analysis of Adam without additional assumptions.
As for the range of  $\beta_{1}$  and  $\beta_{2}$ , one can immediately see that the condition  $\beta_{1} \leq \sqrt{\beta_{2}} - 8\sigma_{1}^{2}(1 - \beta_{2})\beta_{2}^{-2}$  degenerates to  $\beta_{1} \leq \sqrt{\beta_{2}}$  in the bounded gradient case (i.e.,  $\sigma_{1} = 0$ ), the weakest condition required in existing literature [36]. When  $\sigma_{1} \neq 0$ , such a condition is stronger than  $\beta_{1} \leq \sqrt{\beta_{2}}$ . We point out that this is not due to technical limitations but instead agrees with existing counterexamples for Adam: Reddi et al. [26], Zhang et al. [34] show that when  $\sigma_{1} \neq 0$ , there exists a counterexample
satisfying Assumption 1 and Assumption 3 and a pair of  $(\beta_{1},\beta_{2})$  with  $\beta_{1} < \sqrt{\beta_{2}}$  and Adam with  $(\beta_{1},\beta_{2})$  diverges over such a counterexample.
Dependence over  $\beta_{2}$ ,  $\eta$ , and  $T$ . Here we consider the influence of  $\beta_{2}$ ,  $\eta$ , and  $T$  while fixing  $\beta_{1}$  constant (we will discuss the effect of  $\beta_{1}$  in Section 6). With logarithmic factors ignored and coefficients hidden,  $C_1$ ,  $C_2$  and the right-hand-side of Eq. (2) can be rewritten with asymptotic notations as
$$
\begin{array}{l} C _ {1} = \tilde {\mathcal {O}} \left(\frac {1}{\sqrt {1 - \beta_ {2}}} + \frac {\eta^ {2}}{\sqrt {(1 - \beta_ {2}) ^ {3}}}\right), C _ {2} = \tilde {\mathcal {O}} \left(\frac {1}{\sqrt {1 - \beta_ {2}}} + \frac {\eta^ {2}}{\sqrt {(1 - \beta_ {2}) ^ {3}}} + \frac {1}{\eta} + T \sqrt {1 - \beta_ {2}} + \frac {\eta^ {2}}{\sqrt {1 - \beta_ {2}}} T\right), \\ \mathbb {E} \sum_ {t = 1} ^ {T} \| \nabla f (\boldsymbol {w} _ {t}) \| = \tilde {\mathcal {O}} \left(C _ {1} + C _ {2} + \sqrt {T C _ {1}} + \sqrt {T C _ {2}}\right), \\ \end{array}
$$