SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We want to bound the gradients along the optimization trajectory mainly for two reasons. First, as discussed in Section 2, many existing analyses of Adam rely on the assumption of bounded gradients, because unbounded gradient norm leads to unbounded second order momentum  $\hat{v}_t$  which implies very small stepsize, and slow convergence. On the other hand, once the gradients are bounded, it is straightforward to control  $\hat{v}_t$  as well as the stepsize, and therefore the analysis essentially reduces to the easier one for AdaBound. Second, informally speaking<sup>1</sup>, under Assumption 2, bounded gradients also imply bounded Hessians, which essentially reduces the  $(\rho, L_0, L_\rho)$  smoothness to the standard smoothness. See Section 5.2 for more formal discussions.
In this paper, instead of imposing the strong assumption of globally bounded gradients, we develop a new analysis to show that with high probability, the gradients are always bounded along the trajectory of Adam until convergence. The essential idea can be informally illustrated by the following "circular" reasoning that we will make precise later. On the one hand, if  $\| \nabla f(x_{t})\| \leq G$  for every  $t\geq 1$ , it is not hard to show the gradient converges to zero based on our discussions above. On the other hand, we know that a converging sequence must be upper bounded. Therefore there exists some  $G^{\prime}$  such that  $\| \nabla f(x_{t})\| \leq G^{\prime}$  for every  $t\geq 1$ . In other words, the bounded gradient condition implies the convergence result and the convergence result also implies the boundedness condition, forming a circular argument.
This circular argument is of course flawed. However, we can break the circularity of reasoning and rigorously prove both the bounded gradient condition and the convergence result using a contradiction
<sup>1</sup>The statement is informal because here we can only show bounded gradients and Hessians at the iterate points, which only implies local smoothness near the neighborhood of each iterate point (see Section 5.2). However, the standard smoothness condition is a stronger global condition which assumes bounded Hessian at every point within a convex set.
argument. Before introducing the contradiction argument, we first need to provide the following useful lemma, which is the reverse direction of a generalized Polyak-Lojasiewicz (PL) inequality, whose proof is deferred in Appendix D.3.
Lemma 5.1. Under Assumptions 1 and 2, we have  $\| \nabla f(x) \|^2 \leq 3(3L_0 + 4L_\rho \| \nabla f(x) \|^\rho)(f(x) - f^*)$ .
Define the function  $\zeta(u) \coloneqq \frac{u^2}{3(3L_0 + 4L_\rho u^\rho)}$  over  $u \geq 0$ . It is easy to verify that if  $\rho < 2$ ,  $\zeta$  is increasing and its range is  $[0, \infty)$ . Therefore,  $\zeta$  is invertible and  $\zeta^{-1}$  is also increasing. Then, for any constant  $G > 0$ , denoting  $F = \zeta(G)$ , Lemma 5.1 suggests that if  $f(x) - f^* \leq F$ , we have
$$
\left\| \nabla f (x) \right\| \leq \zeta^ {- 1} \left(f (x) - f ^ {*}\right) \leq \zeta^ {- 1} (F) = G.
$$
In other words, if  $\rho < 2$ , the gradient is bounded within any sub-level set, even though the sub-level set could be unbounded. Then, let  $\tau$  be the first time the sub-optimality gap is strictly greater than  $F$ , truncated at  $T + 1$ , or formally,
$$
\tau := \min  \left\{t \mid f \left(x _ {t}\right) - f ^ {*} > F \right\} \wedge (T + 1). \tag {3}
$$
Then at least when  $t < \tau$ , we have  $f(x_{t}) - f^{*} \leq F$  and thus  $\| \nabla f(x_{t}) \| \leq G$ . Based on our discussions above, it is not hard to analyze the updates before time  $\tau$ , and one can contract some Lyapunov function to obtain an upper bound on  $f(x_{\tau}) - f^{*}$ . On the other hand, if  $\tau \leq T$ , we immediately obtain a lower bound on  $f(x_{\tau})$ , that is  $f(x_{\tau}) - f^{*} > F$ , by the definition of  $\tau$  in (3). If the lower bound is greater than the upper bound, it leads to a contradiction, which shows  $\tau = T + 1$ , i.e., the sub-optimality gap and the gradient norm are always bounded by  $F$  and  $G$  respectively before the algorithm terminates. We will illustrate the technique in more details in the simple deterministic setting in Section 5.3, but first, in Section 5.2, we introduce several prerequisite lemmas on the  $(\rho, L_0, L_\rho)$  smoothness.
5.2 Local smoothness
In Section 5.1, we informally mentioned that  $(\rho, L_0, L_\rho)$  smoothness essentially reduces to the standard smoothness if the gradient is bounded. In this section, we will make the statement more precise. First, note that Lemma 3.4 implies the following useful corollary.
Corollary 5.2. Under Assumptions 1 and 2, for any  $G > 0$  and two points  $x \in \mathrm{dom}(f), y \in \mathbb{R}^d$  such that  $\| \nabla f(x) \| \leq G$  and  $\| y - x \| \leq r \coloneqq \min \left\{\frac{1}{5L_{\rho}G^{\rho - 1}}, \frac{1}{5(L_0^{\rho - 1}L_\rho)^{1 / \rho}}\right\}$ , denoting  $L := 3L_0 + 4L_\rho G^\rho$ , we have  $y \in \mathrm{dom}(f)$  and
$$
\| \nabla f (y) - \nabla f (x) \| \leq L \| y - x \|, \quad f (y) \leq f (x) + \left\langle \nabla f (x), y - x \right\rangle + \frac {L}{2} \| y - x \| ^ {2}.
$$
The proof of Corollary 5.2 is deferred in Appendix D.4. Although the inequalities in Corollary 5.2 look very similar to the standard global smoothness condition with constant  $L$ , it is still a local condition as it requires  $\| x - y\| \leq r$ . Fortunately, at least before  $\tau$ , such a requirement is easy to satisfy for small enough  $\eta$ , according to the following lemma whose proof is deferred in Appendix E.5.
Lemma 5.3. Under Assumption 3, if  $t < \tau$  and choosing  $G \geq \sigma$ , we have  $\| x_{t + 1} - x_t\| \leq \eta D$  where  $D \coloneqq 2G / \lambda$ .
Then as long as  $\eta \leq r / D$ , we have  $\| x_{t + 1} - x_t\| \leq r$  which satisfies the requirement in Corollary 5.2. Then we can apply the inequalities in it in the same way as the standard smoothness condition. In other words, most classical inequalities derived for standard smooth functions also apply to  $(\rho ,L_0,L_\rho)$  smooth functions.
5.3 Warm-up: analysis in the deterministic setting
In this section, we consider the simpler deterministic setting where the stochastic gradient  $\nabla f(x_{t},\xi_{t})$  in Algorithm 1 or (18) is replaced with the exact gradient  $\nabla f(x_{t})$ . As discussed in Section 5.1, the key in our contradiction argument is to obtain both upper and lower bounds on  $f(x_{\tau}) - f^{*}$ . In the following derivations, we focus on illustrating the main idea of our analysis technique and ignore minor proof details. In addition, all of them are under Assumptions 1, 2, and 3.
In order to obtain the upper bound, we need the following two lemmas. First, denoting  $\epsilon_t \coloneqq \hat{m}_t - \nabla f(x_t)$ , we can obtain the following informal descent lemma for deterministic Adam.
Lemma 5.4 (Descent lemma, informal). For any  $t < \tau$ , choosing  $G \geq \lambda$  and a small enough  $\eta$ ,
$$
f \left(x _ {t + 1}\right) - f \left(x _ {t}\right) \lesssim - \frac {\eta}{4 G} \left\| \nabla f \left(x _ {t}\right) \right\| ^ {2} + \frac {\eta}{2 \lambda} \left\| \epsilon_ {t} \right\| ^ {2}, \tag {4}
$$
where  $\leqslant$  omits less important terms.
Compared with the standard descent lemma for gradient descent, there is an additional term of  $\left\| \epsilon_t \right\|^2$  in Lemma 5.4. In the next lemma, we bound this term recursively.
Lemma 5.5 (Informal). Choosing  $\beta = \Theta (\eta G^{\rho +1 / 2})$ , if  $t < \tau$ , we have
$$
\left\| \epsilon_ {t + 1} \right\| ^ {2} \leq (1 - \beta / 4) \| \epsilon_ {t} \| ^ {2} + \frac {\lambda \beta}{1 6 G} \| \nabla f (x _ {t}) \| ^ {2}. \tag {5}
$$
The proof sketches of the above two lemmas are deferred in Appendix B. Now we combine them to get the upper bound on  $f(x_{\tau}) - f^{*}$ . Define the function  $\Phi_t \coloneqq f(x_t) - f^* + \frac{2\eta}{\lambda\beta} \| \epsilon_t \|^2$ . Note that for any  $t < \tau$ , (4) and  $\frac{2\eta}{\lambda\beta} \times (5)$  gives
$$
\Phi_ {t + 1} - \Phi_ {t} \leq - \frac {\eta}{8 G} \| \nabla f (x _ {t}) \| ^ {2}. \tag {6}
$$
The above inequality shows  $\Phi_t$  is non-increasing and thus a Lyapunov function. Therefore, we have
$$
f (x _ {\tau}) - f ^ {*} \leq \Phi_ {\tau} \leq \Phi_ {1} = \Delta_ {1},
$$
where in the last inequality we use  $\Phi_1 = f(x_1) - f^* = \Delta_1$  since  $\epsilon_{1} = \hat{m}_{1} - \nabla f(x_{1}) = 0$  in the deterministic setting.
As discussed in Section 5.1, if  $\tau \leq T$ , we have  $F < f(x_{\tau}) - f^{*} \leq \Delta_1$ . Note that we are able to choose a large enough constant  $G$  so that  $F = \frac{G^2}{3(3L_0 + 4L_\rho G^\rho)}$  is greater than  $\Delta_1$ , which leads to a contradiction and shows  $\tau = T + 1$ . Therefore, (6) holds for all  $1 \leq t \leq T$ . Taking a summation over  $1 \leq t \leq T$  and re-arranging terms, we get
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {8 G (\Phi_ {1} - \Phi_ {T + 1})}{\eta T} \leq \frac {8 G \Delta_ {1}}{\eta T} \leq \epsilon^ {2},
$$
if choosing  $T \geq \frac{8G\Delta_1}{\eta\epsilon^2}$ , i.e., it shows convergence with a gradient complexity of  $\mathcal{O}(\epsilon^{-2})$  since both  $G$  and  $\eta$  are constants independent of  $\epsilon$  in the deterministic setting.
5.4 Extension to the stochastic setting
In this part, we briefly introduce how to extend the analysis to the more challenging stochastic setting. It becomes harder to obtain an upper bound on  $f(x_{\tau}) - f^{*}$  because  $\Phi_t$  is no longer non-increasing due to the existence of noise. In addition,  $\tau$  defined in (3) is now a random variable. Note that all the derivations, such as Lemmas 5.4 and 5.5, are conditioned on the random event  $t < \tau$ . Therefore, one can not simply take a total expectation of them to show  $\mathbb{E}[\Phi_t]$  is non-increasing.
Fortunately,  $\tau$  is in fact a stopping time with nice properties. If the noise is almost surely bounded as in Assumption 3, by a more careful analysis, we can obtain a high probability upper bound on  $f(x_{\tau}) - f^{*}$  using concentration inequalities. Then we can still obtain a contradiction and convergence under this high probability event. If the noise has sub-Gaussian norm as in Assumption 4, one can change the definition of  $\tau$  to
$$
\tau := \min  \left\{t \mid f \left(x _ {t}\right) - f ^ {*} > F \right\} \wedge \min  \left\{t \mid \| \nabla f \left(x _ {t}\right) - \nabla f \left(x _ {t}, \xi_ {t}\right) \| > \sigma \right\} \wedge (T + 1)
$$
for appropriately chosen  $F$  and  $\sigma$ . Then at least when  $t < \tau$ , the noise is bounded by  $\sigma$ . Hence we can get the same upper bound on  $f(x_{\tau}) - f^{*}$  as if Assumption 3 still holds. However, when  $t \leq T$ , the lower bound  $f(x_{\tau}) - f^{*} > F$  does not necessarily hold, which requires some more careful analyses. The details of the proofs are involved and we defer them in Appendix E.
6 Variance-reduced Adam
In this section, we propose a variance-reduced version of Adam (VRAdam). This new algorithm is depicted in Algorithm 2. Its main difference from the original Adam is that in the momentum update
rule (Line 6), an additional term of  $(1 - \beta)\left(\nabla f(x_{t},\xi_{t}) - \nabla f(x_{t - 1},\xi_{t})\right)$  is added, inspired by the STORM algorithm [9]. This term corrects the bias of  $m_{t}$  so that it is an unbiased estimate of  $\nabla f(x_{t})$  in the sense of total expectation, i.e.,  $\mathbb{E}[m_t] = \nabla f(x_t)$ . We will also show that it reduces the variance and accelerates the convergence.
Aside from the adaptive stepsize, one major difference between Algorithm 2 and STORM is that our hyper-parameters  $\eta$  and  $\beta$  are fixed constants whereas theirs are decreasing as a function of  $t$ . Choosing constant hyper-parameters requires a more accurate estimate at the initialization. That is why we use a mega-batch  $S_{1}$  to evaluate the gradient at the initial point to initialize  $m_{1}$  and  $v_{1}$  (Lines 2-3). In practice, one can also do a full-batch gradient evaluation at initialization. Note that there is no initialization bias for the momentum, so we do not re-scale  $m_{t}$  and only re-scale  $v_{t}$ . We also want to point out that although the initial mega-batch gradient evaluation makes the algorithm a bit harder to implement, constant hyper-parameters are usually easier to tune and more common in training deep neural networks. It should be not hard to extend our analysis to time-decreasing  $\eta$  and  $\beta$  and we leave it as an interesting future work.
Algorithm 2 VARIANCE-REDUCED ADAM (VRADAM)
1: Input:  $\beta, \beta_{\mathrm{sq}}, \eta, \lambda, T, S_1, x_{\mathrm{init}}$
2: Draw a batch of samples  $S_{1}$  with size  $S_{1}$  and use them to evaluate the gradient  $\nabla f(x_{\mathrm{init}},S_1)$ .
3: Initialize  $m_{1} = \nabla f(x_{\mathrm{init}},\mathcal{S}_{1})$ $v_{1} = \beta_{\mathrm{sq}}m_{1}^{2}$  , and  $x_{2} = x_{\mathrm{init}} - \frac{\eta m_{1}}{|m_{1}| + \lambda}$
4: for  $t = 2, \dots, T$  do
5: Draw a new sample  $\xi_{t}$  and perform the following updates:
6:  $m_{t} = (1 - \beta)m_{t - 1} + \beta \nabla f(x_{t},\xi_{t}) + (1 - \beta)\left(\nabla f(x_{t},\xi_{t}) - \nabla f(x_{t - 1},\xi_{t})\right)$
7:  $v_{t} = (1 - \beta_{\mathrm{sq}})v_{t - 1} + \beta_{\mathrm{sq}}(\nabla f(x_{t},\xi_{t}))^{2}$
8:  $\hat{v}_t = \frac{v_t}{1 - (1 - \beta_{\mathrm{sq}})^t}$
9:  $x_{t + 1} = x_t - \frac{\eta}{\sqrt{\hat{v}_t + \lambda}}\odot m_t$
In addition to Assumption 1, we need to impose the following assumptions which can be viewed as stronger versions of Assumptions 2 and 3, respectively.
Assumption 5. The objective function  $f$  and the component function  $f(\cdot, \xi)$  for each fixed  $\xi$  are  $(\rho, L_0, L_\rho)$  smooth with  $0 \leq \rho < 2$ .
Assumption 6. The random variables  $\{\xi_t\}_{1\leq t\leq T}$  are sampled i.i.d. from some distribution  $\mathcal{P}$  such that for any  $x\in \mathrm{dom}(f)$ ,
$$
\mathbb {E} _ {\xi \sim \mathcal {P}} [ \nabla f (x, \xi) ] = \nabla f (x), \quad \| \nabla f (x, \xi) - \nabla f (x) \| \leq \sigma , a. s.
$$
Remark 6.1. Assumption 6 is stronger than Assumption 3. Assumption 3 applies only to the iterates generated by the algorithm, while Assumption 6 is a pointwise assumption over all  $x \in \operatorname{dom}(f)$  and further assumes an i.i.d. nature of the random variables  $\{\xi_t\}_{1 \leq t \leq T}$ . Also note that, similar to Adam, it is straightforward to generalize the assumption to noise with sub-Gaussian norm as in Assumption 4.
6.1 Analysis
In this part, we briefly discuss challenges in the analysis of VRAdam. The detailed analysis is deferred in Appendix F. Note that Corollary 5.2 requires bounded update  $\| x_{t + 1} - x_t\| \leq r$  at each step. For Adam, it is easy to satisfy for a small enough  $\eta$  according to Lemma 5.3. However, for VRAdam, obtaining a good enough almost sure bound on the update is challenging even though the gradient noise is bounded. To bypass this difficulty, we directly impose a bound on  $\| \nabla f(x_t) - m_t\|$  by changing the definition of the stopping time  $\tau$ , similar to how we deal with the sub-Gaussian noise condition for Adam. In particular, we define
$$
\tau := \min  \left\{t \mid \| \nabla f (x _ {t}) \| > G \right\} \wedge \min  \left\{t \mid \| \nabla f (x _ {t}) - m _ {t} \| > G \right\} \wedge (T + 1).
$$
Then by definition, both  $\| \nabla f(x_{t}) \|$  and  $\| \nabla f(x_{t}) - m_{t} \|$  are bounded by  $G$  before time  $\tau$ , which directly implies bounded update  $\| x_{t + 1} - x_{t} \|$ . Of course, the new definition brings new challenges to lower bounding  $f(x_{\tau}) - f^{*}$ , which requires more careful analyses specific to the VRAdam algorithm. Please see Appendix F for the details.
6.2 Convergence guarantees for VRAdam
In the section, we provide our main results for convergence of VRAdam under Assumptions 1, 5, and 6. We consider the same definitions of problem-dependent constants  $\Delta_1, r, L$  as those in Section 4 to make the statements of theorems concise. Let  $c$  be a small enough numerical constant and  $C$  be a large enough numerical constant. The formal convergence result is shown in the following theorem.
Theorem 6.2. Suppose Assumptions 1, 5, and 6 hold. For any  $0 < \delta < 1$ , let  $G > 0$  be a constant satisfying  $G \geq \max \left\{2\lambda, 2\sigma, \sqrt{C\Delta_1L_0 / \delta}, (C\Delta_1L_\rho / \delta)^{\frac{1}{2 - \rho}}\right\}$ . Choose  $0 \leq \beta_{\mathrm{sq}} \leq 1$  and  $\beta = a^2\eta^2$ , where  $a = 40L\sqrt{G}\lambda^{-3/2}$ . Choose
$$
\eta \leq c \cdot \min \left\{\frac {r \lambda}{G}, \quad \frac {\lambda}{L}, \quad \frac {\lambda^ {2} \delta}{\Delta_ {1} L ^ {2}}, \quad \frac {\lambda^ {2} \sqrt {\delta} \epsilon}{\sigma G L} \right\}, \quad T = \frac {6 4 G \Delta_ {1}}{\eta \delta \epsilon^ {2}}, \quad S _ {1} \geq \frac {1}{2 \beta^ {2} T}.
$$
Then with probability at least  $1 - \delta$ , we have  $\| \nabla f(x_{t}) \| \leq G$  for every  $1 \leq t \leq T$ , and  $\frac{1}{T} \sum_{t=1}^{T} \| \nabla f(x_{t}) \|^{2} \leq \epsilon^{2}$ .
Note that the choice of  $G$ , the upper bound of gradients along the trajectory of VRAdam, is very similar to that in Theorem 4.1 for Adam. The only difference is that now it also depends on the failure probability  $\delta$ . Similar to Theorem 4.1, there is no requirement on  $\beta_{\mathrm{sq}}$  and we choose a very small  $\beta = \mathcal{O}(\epsilon^2)$ . However, the variance reduction technique allows us to take a larger stepsize  $\eta = \mathcal{O}(\epsilon)$  (compared with  $\mathcal{O}(\epsilon^2)$  for Adam) and obtain an accelerated gradient complexity of  $\mathcal{O}(\epsilon^{-3})$ , where we only consider the leading term. We are not sure whether it is optimal as the  $\Omega (\epsilon^{-3})$  lower bound in [3] assumes the weaker bounded variance condition. However, our result significantly improves upon [47], which considers a variance-reduced version of Adam by combining Adam and SVRG [22] and only obtains asymptotic convergence in the non-convex setting. Similar to Adam, our gradient complexity for VRAdam is dimension free but its dependence on  $\lambda$  is  $\mathcal{O}(\lambda^{-2})$ . Another limitation is that, the dependence on the failure probability  $\delta$  is polynomial, worse than the poly-log dependence in Theorem 4.1 for Adam.
7 Conclusion and future works
In this paper, we proved the convergence of Adam and its variance-reduced version under less restrictive assumptions compared to those in the existing literature. We considered a generalized non-uniform smoothness condition, according to which the Hessian norm is bounded by a sub-quadratic function of the gradient norm almost everywhere. Instead of assuming the Lipschitzness of the objective function as in existing analyses of Adam, we use a new contradiction argument to prove that gradients are bounded by a constant along the optimization trajectory. There are several interesting future directions that one could pursue following this work.
Relaxation of the bounded noise assumption. Our analysis relies on the assumption of bounded noise or noise with sub-Gaussian norm. However, the existing lower bounds in [3] consider the weaker bounded variance assumption. Hence, it is not clear whether the  $\mathcal{O}(\epsilon^{-4})$  complexity we obtain for Adam is tight in this setting. It will be interesting to see whether one can relax the assumption to the bounded variance setting. One may gain some insights from recent papers such as [16; 46] that analyze AdaGrad under weak noise conditions. An alternative way to show the tightness of the  $\mathcal{O}(\epsilon^{-4})$  complexity is to prove a lower bound under the bounded noise assumption.
Potential applications of our technique. Another interesting future direction is to see if the techniques developed in this work for bounding gradients (including those in the the concurrent work [26]) can be generalized to improve the convergence results for other optimization problems and algorithms. We believe it is possible so long as the function class is well behaved and the algorithm is efficient enough so that  $f(x_{\tau}) - f^{*}$  can be well bounded for some appropriately defined stopping time  $\tau$ .
Understanding why Adam is better than SGD. We want to note that our results can not explain why Adam is better than SGD for training transformers, because [26] shows that non-adaptive SGD converges with the same  $\mathcal{O}(\epsilon^{-4})$  gradient complexity under even weaker conditions. It would be interesting and impactful if one can find a reasonable setting (function class, gradient oracle, etc) under which Adam or other adaptive methods provably outperform SGD.
This work was supported, in part, by the MIT-IBM Watson AI Lab and ONR Grants N00014-20-1-2394 and N00014-23-1-2299. We also acknowledge support from DOE under grant DE-SC0022199, and NSF through awards DMS-2031883 and DMS-1953181.
[1] Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jababaie, and Suvrit Sra. Linear attention is (maybe) all you need (to understand transformer optimization). arXiv preprint arXiv:2310.01082, 2023.
[2] Zeyuan Allen-Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. In International conference on machine learning, pages 699–707. PMLR, 2016.
[3] Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 (1-2):165-214, 2023.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. *ArXiv*, abs/2005.14165, 2020.
[5] Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu. Towards practical adam: Non-convexity, convergence theory, and mini-batch acceleration. The Journal of Machine Learning Research, 23(1):10411–10457, 2022.
[6] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class ofadam-type algorithms for non-convex optimization. arXiv preprint arXiv:1808.02941, 2018.
[7] Ziyi Chen, Yi Zhou, Yingbin Liang, and Zhaosong Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. arXiv preprint arXiv:2303.02854, 2023.
[8] Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in Neural Information Processing Systems, 35:9955-9968, 2022.
[9] Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. ArXiv, abs/1905.10018, 2019.
[10] Soham De, Anirbit Mukherjee, and Enayat Ullah. Convergence guarantees for rmsprop and adam in non-convex optimization and an empirical comparison to nesterov acceleration. arXiv: Learning, 2018.
[11] Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014.
[12] Alexandre D'efossez, Léon Bottou, Francis R. Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. arXiv: Machine Learning, 2020.
[13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.