SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is worth pointing out that the existing lower bounds for first-order methods [3] and SGD [17] do not contain any exponential terms. On the other hand, Vaswani et al. [63] established a lower bound in the strongly convex setting for GD with exponentially decreasing step sizes  $(1 / T)^{t / T}\eta$ , where  $T$  denotes the total number of iterations and  $\eta > 0$ . With a quadratic function, they showed that if  $\eta > 3 / \ell$ , the distance to the optimal solution  $\| x_t - x^*\|$  exhibits exponential growth in  $t$  during the initial  $\widetilde{\Theta}(T / \log T)$  iterations. However, this exponential growth is rapidly mitigated afterward, resulting in no exponential term at the final iterate  $T$ . This behavior starkly contrasts with our nonconvex setting.
We illustrate our hard instance for Theorem 2 in Figure 2, which is one-dimensional. The algorithm starts from a valley of the function  $f(x) = \ell x^2 / 2$ , i.e., Segment 1. Because of the large initial stepsize and steep slope, in the first  $t_0$  iterations, Gradient Descent increases the function value as large as  $\tilde{\Delta} = \Omega \left( (8e)^{\eta^2 \ell^2 / 16} \Delta \right)$ . Then the iterate  $x_{t_0 + 1}$  jumps to the top of a very flat valley, i.e., Segment 4, so that Gradient Descent decreases the gradient as slowly as  $\Omega (T^{-1/4})$ .
Why do not we assume gradients to be bounded? The assumption on bounded gradients is not satisfied even for the simple function  $f(x) = \ell x^{2} / 2$ . When training neural networks, gradient explosion is often observed [55, 59], which directly suggests that this assumption is not satisfied or only satisfied with a numerically large constant. In Proposition 4 in the appendix, we also provide a simple proof for the convergence under the additional assumption of bounded gradient, i.e.,  $\| \nabla f(x)\| \leq G$  for all  $x$ , attaining a sample complexity of  $\tilde{\mathcal{O}} (\eta^2\ell^2 G^4\sigma^2\epsilon^{-4})$  without any information about problem parameters. However, compared with Theorem 1 and 2, constant  $G$  hides the exponential term. In Figure 1, we observe that the gradient bound along the trajectory of non-adaptive stepsize can be much larger than that of adaptive stepsize even if starting from the same initial point, so assuming bounded gradient will obscure the difference between them.
4 Power of Adaptive Methods
In this section, we focus on the convergence behaviors of adaptive methods, which adjust their stepsizes based on the observed gradients. In particular, when arriving at a point with a large gradient, adaptive methods automatically decrease their step sizes to counter the effect of possible gradient increase; to list a few, Normalized SGD [26], AdaGrad [18], Adam [35]. Since the analysis for adaptive methods is usually on a case-by-case basis, we will examine three examples – Normalized SGD, AMSGrad-norm, and AdaGrad-norm – to establish a universal observation that they avoid exponential dependency in  $\ell$  without tuning. Although many existing analyses rely on bounded gradients (and function values) or information on problem parameters, we will abandon such assumptions as noted in the previous section. We focus on the norm instead of the coordinate-wise version of adaptive methods, which means each coordinate adopts the same stepsize, because the norm version is usually dimension-independent in the complexity, and is also widely used in both theory and practice [73, 45, 43, 38, 54, 32].
4.1 Family of Normalized SGD
Normalized (Stochastic) Gradient Descent [53, 26], referred to as NGD and NSGD, is one of the simplest adaptive methods. It takes the stepsize in (1) to be normalized by the norm of the current (stochastic) gradient:
$$
\eta_ {t} = \frac {\gamma_ {t}}{\| g (x _ {t} ; \xi_ {t}) \|},
$$
where  $\{\gamma_t\}_{t\geq 0}$  is a sequence of positive learning rate. Cutkosky and Mehta [14] and Zhao et al. [74] show that NSGD with  $\gamma_{t} = \gamma /\sqrt{T}$  can find an  $\mathcal{O}(1 / \sqrt{T} +\sigma)$ -stationary point. In order to compare fairly with untuned SGD with decaying stepsize, we present a modification with decaying  $\gamma_{t} = \gamma /\sqrt{t + 1}$  in NSGD.
Proposition 1. Under Assumption 1 and 2, if we run NSGD with  $\gamma_t = \frac{\gamma}{\sqrt{t + 1}}$ , then for any  $\gamma > 0$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| \leq 3 \left(\frac {\Delta}{\gamma} + \ell \gamma \log (T)\right) T ^ {- 1 / 2} + 2 4 \sigma .
$$
NGD. In the deterministic setting, by Proposition 1, NGD converges to an  $\epsilon$ -stationary point with a complexity of  $\widetilde{\mathcal{O}}((\gamma^{-2} + \gamma^{2}\ell^{2})\epsilon^{-2})$  for any  $\gamma > 0$ , which importantly does not include any exponential term. Thus, even if the initial stepsize is not small enough, it does not result in a catastrophic gradient explosion.
NSGD. In the stochastic setting, Proposition 1 implies that NSGD can find an  $\epsilon$ -stationary point only when the noise variance is small enough, i.e.,  $\sigma \leq \mathcal{O}(\epsilon)$ . This is not the consequence of a loose analysis. Hazan et al. [26] show that NSGD with constant  $\gamma_t \equiv \gamma$  does not converge when the mini-batch size is smaller than  $\Theta(\epsilon^{-1})$  for a non-smooth convex function. Here we provide a non-convergence result in the gradient norm with a smooth objective for all uniformly bounded stepsizes. The intuition behind this is illustrated in Figure 3 in the appendix, where  $\mathbb{E}_{\xi} g(x; \xi) / \|g(x; \xi)\|$  can easily vanish or be in the opposite direction of  $\nabla f(x)$  under certain noises.
Theorem 3. Fixing  $\ell >0$ ,  $\sigma >0$ ,  $\epsilon >0$ ,  $\Delta >0$  and stepsize sequence  $\{\gamma_t\}_{t = 0}^{\infty}$  with  $\gamma_{t}\leq \gamma_{\mathrm{max}}$  that  $\epsilon^2 < \min \{\sigma^2,2\ell \Delta ,2\Delta (\sigma -\epsilon) / \gamma_{\mathrm{max}}\}$ , there exists an  $\ell$ -smooth convex function  $f$ , initial point  $x_0$  with  $f(x_0) - \min_x f(x)\leq \Delta$  and zero-mean noises with  $\sigma^2$  variance such that the output from NSGD satisfies  $\mathbb{E}\| \nabla f(x_t)\| \geq \epsilon$  for all  $t$ .
This theorem implies that, given a fixed function class  $(\ell, \Delta, \sigma)$  and any sequence  $\{\gamma_t\}_t$  uniformly upper bounded by  $\gamma_{\mathrm{max}}$ , NSGD cannot converge to an arbitrarily small  $\epsilon$ . Specifically, the expected gradient norm will always stay larger than  $\min \{\sigma, \sqrt{2\ell\Delta}, \gamma_{\mathrm{max}}^{-1}(-\Delta + \sqrt{\Delta^2 + 2\Delta\gamma_{\mathrm{max}}\sigma})\}$ . Most  $\{\gamma_t\}_t$  used in practice is upper bounded, e.g., constant or decreasing sequences. The condition  $\epsilon^2 < 2\ell\Delta$  is necessary by noting that  $\|\nabla f(x_0)\|^2 \leq 2\ell[f(x_0) - \min_x f(x)] \leq 2\ell\Delta$ . Considering  $\gamma_t = 1/\sqrt{t+1}$ , when  $\Delta \geq \sigma$  and  $\sqrt{2\ell\Delta} \geq \sigma$ , it matches with Proposition 1, where NSGD can only converge to a  $\Theta(\sigma)$ -stationary point. Since Sign-SGD and NSGD coincide for one-dimensional objectives, our non-convergent example also applies to Sign-SGD. This sheds light on why increasing batch size improves Normalized and Sign-SGD [74, 37]. However, these methods are generally different in higher dimensions, as Karimireddy et al. [29] show that sign-SGD may not converge even with a full batch.
NSGD with momentum. While NSGD may not always converge, Cutkosky and Mehta [14] introduced NSGD with momentum (NSGD-M) presented in Algorithm 1 with constant  $\gamma_t \equiv \gamma$ . We provide the following modification with diminishing  $\gamma_t$  that eliminates the need to specify the total number of runs beforehand.
Proposition 2. Under Assumptions 1 and 2, if we run NSGD-M with  $\alpha_{t} = \frac{\sqrt{2}}{\sqrt{t + 2}}$  and  $\gamma_{t} = \frac{\gamma}{(t + 1)^{3 / 4}}$ , then for any  $\gamma >0$ ,
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| \leq C \left(\frac {\Delta}{\gamma} + (\sigma + \ell \gamma) \log (T)\right) T ^ {- \frac {1}{4}},
$$
where  $C > 0$  is a numerical constant.
It implies that NSGD-M attains a complexity of  $\widetilde{\mathcal{O}} ((\gamma^{-4} + \gamma^4\ell^4)\epsilon^{-4})$  for any  $\gamma >0$ . Compared with Theorem 1 and 2, NSGD-M not only achieves near-optimal dependency in the target accuracy  $\epsilon$ , but also shreds the exponential term when the hyper-parameter is agnostic to smoothness constant.
Algorithm 1 NSGD-M
1: Input: initial point  $x_0$ , stepsize sequence  $\{\gamma_t\}$ , momentum sequence  $\{\alpha_t\}$ , and initial momentum  $g_0$ .
2: for  $t = 0,1,2,\ldots$  do
3:  $x_{t + 1} = x_t - \frac{\gamma_t}{\|g_t\|} g_t$
4: sample  $\xi_{t + 1}$
5:  $g_{t + 1} = (1 - \alpha_t)g_t + \alpha_t g(x_{t + 1};\xi_{t + 1})$
6: end for
Algorithm 2 AMSGrad-norm
1: Input: initial point  $x_0$ , momentum parameters  $0 \leq \beta_1 < 1$  and  $0 \leq \beta_2 \leq 1$ , stepsize sequence  $\{\gamma_t\}$  and initial momentum  $m_0$  and  $v_0 > 0$ .
2:  $\hat{v}_0 = v_0$
3: for  $t = 0,1,2,\ldots$  do
4: sample  $\xi_{t}$
5:  $m_{t + 1} = \beta_1 m_t + (1 - \beta_1) g(x_t; \xi_t)$
6:  $v_{t + 1}^2 = \beta_2v_t^2 +(1 - \beta_2)\| g(x_t;\xi_t)\| ^2$
7:  $\hat{v}_{t + 1}^2 = \max \{\hat{v}_t^2,v_{t + 1}^2\}$
8:  $x_{t + 1} = x_t - \frac{\gamma_t}{\sqrt{v_{t + 1}^2}} m_{t + 1}$
9: end for
4.2 AMSGrad-norm
AMSGrad was introduced by Reddi et al. [57] to fix the possible non-convergence issue of Adam. Notably, current analyses of AMSGrad in the stochastic setting show a convergence rate of  $\widetilde{\mathcal{O}}(1/T^{1/4})$ , but they rely on the assumption of bounded stochastic gradients [12, 75], which is much stronger than assumptions used for SGD analysis. Here, we examine the simpler norm version of AMSGrad, presented in Algorithm 2. We prove that without assuming bounded stochastic gradients, AMSGrad-norm with default  $\gamma_t = \gamma/\sqrt{t + 1}$  may converge at an arbitrarily slow polynomial rate. In fact, this holds even if the true gradients are bounded. We believe this result is of independent interest.
Theorem 4. For any  $\ell >0$ ,  $\Delta >0$ ,  $\sigma >0$  and  $T > 1$ , there exists a  $\ell$ -smooth function  $f:\mathbb{R}^2\to \mathbb{R}^2$ ,  $x_0$  with  $f(x_0) - \inf_x f(x)\leq \Delta$  and noise distribution  $P$  with variance upper bounded by  $\sigma^2$ , such that if we run AMSGrad-norm with  $0\leq \beta_{1}\leq 1$ ,  $0\leq \beta_{2} < 1$  and  $\gamma_t = \frac{\gamma}{\sqrt{t + 1}}$ , we have with probability  $\frac{1}{2}$ , it holds that
$$
\min  _ {t \in \{0, 1, \dots , T - 1 \}} \| \nabla f (x _ {t}) \| \geq \sqrt {\frac {\Delta}{1 6 \max  \left\{1 / \ell , \frac {\gamma \sqrt {2 \Gamma \left(1 - \frac {\zeta}{2}\right)}}{\sigma \left(e \left(\frac {1}{\zeta} - 1\right)\right) ^ {\frac {\zeta}{2}} (1 - \zeta) \sqrt {1 - \beta_ {2}}} (T ^ {1 - \zeta} - \zeta) \right\}}}
$$
for any  $\frac{1}{2} <  \zeta <  1$  where  $\Gamma (\cdot)$  denotes the Gamma function.
The intuition behind this thermoem is that since AMSGrad utilizes the maximum norm of past stochastic gradients with momentum in the denominator of stepsizes, some noise distributions enable this maximum norm to increase polynomially, making the stepsizes too small. However, we can still explore its benefit in the deterministic setting. Whether it converges without assuming bounded gradients, to the best of our knowledge, is unknown. Here, for simplicity, we consider AMSGrad-norm without momentum, i.e.,  $\beta_{1} = \beta_{2} = 0$ .
Theorem 5. Under Assumption 1, if we run AMSGrad-norm with  $\gamma_t = \frac{\gamma}{\sqrt{t + 1}}$ ,  $v_0 > 0$  and  $\beta_1 = \beta_2 = 0$  in the deterministic setting, then for any  $\gamma > 0$  and  $0 < \alpha < 1$ ,
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| \leq \left\{ \begin{array}{l l} T ^ {- \frac {1}{4}} \sqrt {2 \Delta \max  \{v _ {0} , \sqrt {2 \ell \Delta} \} \gamma^ {- 1}}, & \text {w h e n v o r m a l p h a <   \gamma \ell}, \\ T ^ {- \frac {1}{2}} \gamma^ {2} \ell^ {2} v _ {0} ^ {- 2} + T ^ {- \frac {1}{4}} \sqrt {2 \gamma (M + \Delta) \max  \{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \}}, & \text {w h e n v o r m a l p h a \geq \gamma \ell}, \end{array} \right.
$$
where  $M = \ell \gamma^2\left(1 + \log \left(\frac{\ell\gamma}{v_0}\right)\right)$
The theorem implies that AMSGrad-norm achieves a complexity of  $\widetilde{\mathcal{O}}((\ell^4\gamma^4 + \ell^2 + \ell^3\gamma^2 + \ell\gamma^{-2})\epsilon^{-4})$  with the default  $\gamma_t = \Theta(t^{-1/2})$  [57, 12, 25]. Compared with untuned Gradient Descent, it gets rid of the exponential dependency. In the proof, we show that before the first iteration  $\tau$  when stepsize  $\eta_t$
reduces to  $1 / \ell$ , the accumulated gradient norms  $\sum_{t = 0}^{\tau -1}\| \nabla f(x_t)\|^2$  are upper bounded polynomially, which is in striking contrast with SGD in Theorem 2. We further provide theoretical guarantees for more general schemes  $\frac{\gamma}{(t + 1)^{\alpha}}$  with  $0 < \alpha < 1$  in Theorem 7 in the appendix. We also derive matching lower bounds in Theorem 8 for any  $0 < \alpha < 1$ , and justify that AMSGrad may fail to converge with constant  $\gamma_t\equiv \gamma$  (i.e.,  $\alpha = 0$ ) if the problem parameter is unknown.
4.3 AdaGrad-norm
AdaGrad chooses its stepsize to be inversely proportional to the element-wise accumulated past gradients [18, 48]. Its norm-version, AdaGrad-norm [61, 66], picks stepsize in (1) to be
$$
\eta_ {t} = \frac {\eta}{\sqrt {v _ {0} ^ {2} + \sum_ {k = 0} ^ {t} \| g (x _ {k} ; \xi_ {k}) \| ^ {2}}},
$$
where  $v_{0} > 0$ . Very recently, AdaGrad is proven to converge in nonconvex optimization without the assumption on bounded gradients or tuning  $\eta$  [20, 68]. Although the result in [68] are presented for minimax optimization problems, a similar result follows immediately for minimization problems. We present the following result for the completeness of the paper and to further illustrate the benefits of adaptive methods over SGD.
Proposition 3. Under Assumptions 1 and 2, if we run AdaGrad-norm, then for any  $\eta >0$  and  $v_{0} > 0$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| \leq \frac {2 A}{\sqrt {T}} + \frac {\sqrt {v _ {0} A}}{\sqrt {T}} + \frac {2 \sqrt {A \sigma}}{T ^ {\frac {1}{4}}},
$$
where  $A = \widetilde{\mathcal{O}}\left(\frac{\Delta}{\eta} + \sigma + \ell \eta\right)$ .
The above result implies a complexity of  $\widetilde{\mathcal{O}}\left((\eta^{-2} + \sigma^2 + \eta^2\ell^2)(\epsilon^{-2} + \sigma^2\epsilon^{-4})\right)$ . Notably, if  $\eta$  can be chosen to be  $1 / \sqrt{\ell}$ , it achieves the optimal complexity in both  $\ell$  and  $\epsilon$  up to logarithmic terms like well-tuned SGD [3]. Even if  $\eta$  is agnostic to  $\ell$ , AdaGrad-norm does not suffer from the exponential term present in untuned SGD. One of the intuitions in the deterministic setting, similar to the AMSGrad-norm, is that the accumulated squared gradient norm before the first iteration with stepsize smaller than  $1 / \ell$  will be upper bounded by a polynomial term (see Theorem 3.2 in [42]). Another benefit of AdaGrad over other methods is to achieve optimal convergence rates simultaneously in deterministic and stochastic settings with the same hyper-parameters. This is sometimes referred to as "noise adaptivity", which is out of the scope of this paper.
We convey a crucial message: SGD with a polynomially decaying stepsize can converge at the order-optimal rate, while remaining agnostic to the problem-specific parameter - a notion that may challenge common belief. However, it is subject to an exponential term, which can be avoided by utilizing an appropriate adaptive scheme. We further reveal the similarity and differences between several adaptive schemes; a detailed exposition of these results can be found in Appendix A. This work opens up several avenues for future research. Firstly, it would be valuable to investigate whether the benefits of adaptive methods extend to high-probability convergence guarantees and apply to a broader range of adaptive optimizers. Secondly, removing the assumption of bounded gradients is crucial for a comprehensive analysis of adaptive algorithms, as it can reveal the true dependence on the smoothness parameter  $\ell$  and highlight their advantage over SGD. Lastly, examining the impact of adaptive algorithms on optimizing non-smooth nonconvex objectives, which are prevalent in training modern machine learning models, presents an interesting research direction.
The work is supported by ETH research grant and Swiss National Science Foundation (SNSF) Project Funding No. 200021-207343. Ilyas Fatkhullin is partly funded by ETH AI Center.
[51] A. S. Nemirovski and D. B. Yudin. Problem Complexity and Method Efficiency in Optimization. John Wiley UK/USA, 1983. (Cited on page 4.)
[52] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003. (Cited on page 6.)
[53] Yurii E Nesterov. Minimization methods for nonsmooth convex and quasiconvex functions. Matekon, 29(3):519-531, 1984. (Cited on pages 2, 4, and 7.)
[54] Werner Palfinger. Continuous remeshing for inverse rendering. Computer Animation and Virtual Worlds, 33(5):e2101, 2022. (Cited on page 7.)
[55] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. PMLR, 2013. (Cited on pages 2, 4, and 7.)
[56] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855, 1992. (Cited on page 4.)
[57] Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR, 2018. (Cited on pages 2, 4, 5, and 9.)
[58] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951. (Cited on pages 1 and 4.)
[59] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61: 85-117, 2015. (Cited on page 7.)
[60] Naichen Shi and Dawei Li. Rmsprop converges with proper hyperparameter. In International conference on learning representation, 2021. (Cited on page 4.)
[61] Matthew Streeter and H Brendan McMahan. Less regret via online conditioning. arXiv preprint arXiv:1002.4862, 2010. (Cited on pages 2, 4, and 10.)
[62] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. (Cited on page 1.)
[63] Sharan Vaswani, Benjamin Dubois-Taine, and Reza Babanezhad. Towards noise-adaptive, problem-adaptive (accelerated) stochastic gradient descent. In International Conference on Machine Learning, pages 22015-22059. PMLR, 2022. (Cited on page 7.)
[64] Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. Provable adaptivity in adam. arXiv preprint arXiv:2208.09900, 2022. (Cited on page 5.)
[65] Xiaoyu Wang, Sindri Magnússon, and Mikael Johansson. On the convergence of step decay step-size for stochastic optimization. Advances in Neural Information Processing Systems, 34: 14226-14238, 2021. (Cited on page 4.)
[66] Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research, 21(1):9047-9076, 2020. (Cited on pages 4 and 10.)
[67] Zeke Xie, Xinrui Wang, Huishuai Zhang, Issei Sato, and Masashi Sugiyama. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In International Conference on Machine Learning, pages 24430–24459. PMLR, 2022. (Cited on page 5.)
[68] Junchi Yang, Xiang Li, and Niao He. Nest your adaptive algorithm for parameter-agnostic nonconvex minimax optimization. arXiv preprint arXiv:2206.00743, 2022. (Cited on pages 2, 3, 4, 10, 16, and 32.)
[69] D. Matthew Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701v1, 2012. (Cited on page 2.)
[70] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jababaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2019. (Cited on page 5.)
[71] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383-15393, 2020. (Cited on page 5.)
[72] Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems, 2022. (Cited on page 4.)
[73] Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), pages 1-2. IEEE, 2018. (Cited on page 7.)
[74] Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 64(3):1-13, 2021. (Cited on pages 4 and 8.)
[75] Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyuan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671, 2018. (Cited on pages 4 and 9.)
[76] Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for convergences of adam and rmsprop. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11127-11135, 2019. (Cited on page 4.)
A Results Summary
Table 2: Comparisons of complexities to find an  $\epsilon$ -stationary point, i.e.,  $\mathbb{E}\|\nabla f(x)\| \leq \epsilon$ , between SGD, NSGD, NSGD-M, AMSGrad-norm and AdaGrad-norm. We only assume  $f$  is  $\ell$ -smooth, and unbiased stochastic gradients have bounded variance  $\sigma^2$ . Hyper-parameters (e.g.,  $\gamma$  and  $\eta$ ) are arbitrary and untuned. In this table,  $\widetilde{\mathcal{O}}$  and  $\Omega$  hide polynomial terms in problem parameters and hyper-parameters, and  $\widetilde{\mathcal{O}}$  also hides all logarithmic terms. We use  $\eta_t$  to denote the effective stepsize at iteration  $t$ .
Algorithms	Upper bound; deterministic	Lower bound; deterministic	Upper bound; stochastic	Lower bound; stochastic
SGD (Eq. 1)ηt = η(t+1)α	O((4e2α)(ηt)1/α(1-α)∧αε(1-α)∧α)α ∈ (0,1) [Thm. 1,6]	Ω((8e)η2ℓ2/8ε-4)α = 1/2 [Thm. 2]	O((4e2α)(ηt)1/α(1-α)∧αε(1-α)∧α)α ∈ (0,1) [Thm. 1,6]	Ω((8e)η2ℓ2/8ε-4)α = 1/2 [Thm. 2]
NSGD (Alg. 3)ηt = γt||g(xt;ξt)||	O(ε-2), γt = γ√t+1[14] & [Prop. 1]	Ω(ε-2)[11]	N/A due to lower bound	Nonconvergent ∀ bounded {γt} [Thm. 3]
NSGD-M (Alg. 1)ηt = γ(t+1)α||gt|	O(ε-2), α = 1/2[14] & [Prop. 1]	Ω(ε-2)[11]	O(ε-4), α = 3/4[14] & [Prop. 1]	Ω(ε-4)[3]
AMSGrad-norm (Alg. 2)ηt = γ(t+1)α√v2t+1	O(ε-2/(1-α)), α ∈ (0,1) [Thm. 5,7]	Ω(ε-2/(1-α)), α ∈ (0,1) Nonconvergent, α = 0 [Thm. 8]	N/A due to lower bound	Ω(ε-2/(1-ζ)), α = 1/2∀ζ ∈ (0.5,1) [Thm. 4]
AdaGrad-norm (Alg. 5)ηt = η/√v02+Σk=0||g(xk;ξk)||2	O(ε-2)[68] & [Prop. 3]	Ω(ε-2)[11]	O(ε-4)[68] & [Prop. 3]	Ω(ε-4)[3]