SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haochuan Li
haochuan@mit.edu
Alexander Rakhlin
rakhlin@mit.edu
Ali Jadbabaie
jadbabai@mit.edu
In this paper, we provide a rigorous proof of convergence of the Adaptive Moment Estimate (Adam) algorithm for a wide class of optimization objectives. Despite the popularity and efficiency of the Adam algorithm in training deep neural networks, its theoretical properties are not yet fully understood, and existing convergence proofs require unrealistically strong assumptions, such as globally bounded gradients, to show the convergence to stationary points. In this paper, we show that Adam provably converges to  $\epsilon$ -stationary points with  $\mathcal{O}(\epsilon^{-4})$  gradient complexity under far more realistic conditions. The key to our analysis is a new proof of boundedness of gradients along the optimization trajectory of Adam, under a generalized smoothness assumption according to which the local smoothness (i.e., Hessian norm when it exists) is bounded by a sub-quadratic function of the gradient norm. Moreover, we propose a variance-reduced version of Adam with an accelerated gradient complexity of  $\mathcal{O}(\epsilon^{-3})$ .
In this paper, we study the non-convex unconstrained stochastic optimization problem
$$
\min  _ {x} \left\{f (x) = \mathbb {E} _ {\xi} [ f (x, \xi) ] \right\}. \tag {1}
$$
The Adaptive Moment Estimation (Adam) algorithm [23] has become one of the most popular optimizers for solving (1) when  $f$  is the loss for training deep neural networks. Owing to its efficiency and robustness to hyper-parameters, it is widely applied or even sometimes the default choice in many machine learning application domains such as natural language processing [44; 4; 13], generative adversarial networks [35; 21; 55], computer vision [14], and reinforcement learning [28; 33; 40]. It is also well known that Adam significantly outperforms stochastic gradient descent (SGD) for certain models like transformer [50; 24; 1].
Despite its success in practice, theoretical analyses of Adam are still limited. The original proof of convergence in [23] was later shown by [37] to contain gaps. The authors in [37] also showed that for a range of momentum parameters chosen independently with the problem instance, Adam does not necessarily converge even for convex objectives. However, in deep learning practice, the hyper-parameters are in fact problem-dependent as they are usually tuned after given the problem and weight initialization. Recently, there have been many works proving the convergence of Adam for non-convex functions with various assumptions and problem-dependent hyper-parameter choices. However, these results leave significant room for improvement. For example, [12; 19] prove the convergence to stationary points assuming the gradients are bounded by a constant, either explicitly or implicitly. On the other hand, [51; 45] consider weak assumptions, but their convergence results are still limited. See Section 2 for more detailed discussions of related works.
To address the above-mentioned gap between theory and practice, we provide a new convergence analysis of Adam without assuming bounded gradients, or equivalently, Lipschitzness of the objective function. In addition, we also relax the standard global smoothness assumption, i.e., the Lipschitzness of the gradient function, as it is far from being satisfied in deep neural network training. Instead, we
consider a more general, relaxed, and non-uniform smoothness condition according to which the local smoothness (i.e., Hessian norm when it exists) around  $x$  is bounded by a sub-quadratic function of the gradient norm  $\| \nabla f(x) \|$  (see Definition 3.2 and Assumption 2 for the details). This generalizes the  $(L_0, L_1)$  smoothness condition proposed by [49] based on language model experiments. Even though our assumptions are much weaker and more realistic, we can still obtain the same  $\mathcal{O}(\epsilon^{-4})$  gradient complexity for convergence to an  $\epsilon$ -stationary point.
The key to our analysis is a new technique to obtain a high probability, constant upper bound on the gradients along the optimization trajectory of Adam, without assuming Lipschitzness of the objective function. In other words, it essentially turns the bounded gradient assumption into a result that can be directly proven. Bounded gradients imply bounded stepsize at each step, with which the analysis of Adam essentially reduces to the simpler analysis of AdaBound [31]. Furthermore, once the gradient boundedness is achieved, the analysis under the generalized non-uniform smoothness assumption is not much harder than that under the standard smoothness condition. We will introduce the technique in more details in Section 5. We note that the idea of bounding gradient norm along the trajectory of the optimization algorithm can be use in other problems as well. For more details, we refer the reader to our concurrent work [26] in which we present a set of new techniques and methods for bounding gradient norm for other optimization algorithms under a generalized smoothness condition.
Another contribution of this paper is to show that the gradient complexity of Adam can be further improved with variance reduction methods. To this end, we propose a variance-reduced version of Adam by modifying its momentum update rule, inspired by the idea of the STORM algorithm [9]. Under additional generalized smoothness assumption of the component function  $f(\cdot ,\xi)$  for each  $\xi$ , we show that this provably accelerates the convergence with a gradient complexity of  $\mathcal{O}(\epsilon^{-3})$ . This rate improves upon the existing result of [47] where the authors obtain an asymptotic convergence of their approach to variance reduction for Adam in the non-convex setting, under the bounded gradient assumption.
In light of the above background, we summarize our main contributions as follows.
- We develop a new analysis to show that Adam converges to stationary points under relaxed assumptions. In particular, we do not assume bounded gradients or Lipschitzness of the objective function. Furthermore, we also consider a generalized non-uniform smoothness condition where the local smoothness or Hessian norm is bounded by a sub-quadratic function of the gradient norm. Under these more realistic assumptions, we obtain a dimension free gradient complexity of  $\mathcal{O}(\epsilon^{-4})$  if the gradient noise is centered and bounded.
- We generalize our analysis to the setting where the gradient noise is centered and has sub-Gaussian norm, and show the convergence of Adam with a gradient complexity of  $\mathcal{O}(\epsilon^{-4}\log^{3.25}(1 / \epsilon))$ .
- We propose a variance-reduced version of Adam (VRAdam) with provable convergence guarantees. In particular, we obtain the accelerated  $\mathcal{O}(\epsilon^{-3})$  gradient complexity.
In this section, we discuss the relevant literature related to convergence of Adam and the generalized smoothness condition, and defer additional related work on variants of Adam and variance reduction methods to Appendix A.
Convergence of Adam. Adam was first proposed by Kingma and Ba [23] with a theoretical convergence guarantee for convex functions. However, Reddi et al. [37] found a gap in the proof of this convergence analysis, and also constructed counter-examples for a range of hyper-parameters on which Adam does not converge. That being said, the counter-examples depend on the hyperparameters of Adam, i.e., they are constructed after picking the hyper-parameters. Therefore, it does not rule out the possibility of obtaining convergence guarantees for problem-dependent hyperparameters, as also pointed out by [42; 51].
Many recent works have developed convergence analyses of Adam with various assumptions and hyper-parameter choices. Zhou et al. [54] show Adam with certain hyper-parameters can work on the counter-examples of [37]. De et al. [10] prove convergence for general non-convex functions
assuming gradients are bounded and the signs of stochastic gradients are the same along the trajectory. The analysis in [12] also relies on the bounded gradient assumption. Guo et al. [19] assume the adaptive stepsize is upper and lower bounded by two constants, which is not necessarily satisfied unless assuming bounded gradients or considering the AdaBound variant [31]. [51; 45] consider very weak assumptions. However, they show either 1) "convergence" only to some neighborhood of stationary points with a constant radius, unless assuming the strong growth condition; or 2) convergence to stationary points but with a slower rate.
Generalized smoothness condition. Generalizing the standard smoothness condition in a variety of settings has been a focus of many recent papers. Recently, [49] proposed a generalized smoothness condition called  $(L_0,L_1)$  smoothness, which assumes the local smoothness or Hessian norm is bounded by an affine function of the gradient norm. The assumption was well-validated by extensive experiments conducted on language models. Various analyses of different algorithms under this condition were later developed [48; 34; 52; 17; 38; 8]. One recent closely-related work is [45] which studies converges of Adam under the  $(L_0,L_1)$  smoothness condition. However, their results are still limited, as we have mentioned above. In this paper, we consider an even more general smoothness condition where the local smoothness is bounded by a sub-quadratic function of the gradient norm, and prove the convergence of Adam under this condition. In our concurrent work [26], we further analyze various other algorithms in both convex and non-convex settings under similar generalized smoothness conditions following the same key idea of bounding gradients along the trajectory.
Notation. Let  $\|\cdot\|$  denote the Euclidean norm of a vector or spectral norm of a matrix. For any given vector  $x$ , we use  $(x)_i$  to denote its  $i$ -th coordinate and  $x^2$ ,  $\sqrt{x}$ ,  $|x|$  to denote its coordinate-wise square, square root, and absolute value respectively. For any two vectors  $x$  and  $y$ , we use  $x \odot y$  and  $x/y$  to denote their coordinate-wise product and quotient respectively. We also write  $x \preceq y$  or  $x \succeq y$  to denote the coordinate-wise inequality between  $x$  and  $y$ , which means  $(x)_i \leq (y)_i$  or  $(x)_i \geq (y)_i$  for each coordinate index  $i$ . For two symmetric real matrices  $A$  and  $B$ , we say  $A \preceq B$  or  $A \succeq B$  if  $B - A$  or  $A - B$  is positive semi-definite (PSD). Given two real numbers  $a, b \in \mathbb{R}$ , we denote  $a \wedge b := \min\{a, b\}$  for simplicity. Finally, we use  $\mathcal{O}(\cdot)$ ,  $\Theta(\cdot)$ , and  $\Omega(\cdot)$  for the standard big-O, big-Theta, and big-Omega notation.
3.1 Description of the Adam algorithm
Algorithm 1 ADAM
1: Input:  $\beta, \beta_{\mathrm{sq}}, \eta, \lambda, T, x_{\mathrm{init}}$
2: Initialize  $m_0 = v_0 = 0$  and  $x_{1} = x_{\mathrm{init}}$
3: for  $t = 1, \dots, T$  do
4: Draw a new sample  $\xi_{t}$  and perform the following updates
5:  $m_{t} = (1 - \beta)m_{t - 1} + \beta \nabla f(x_{t},\xi_{t})$
6:  $v_{t} = (1 - \beta_{\mathrm{sq}})v_{t - 1} + \beta_{\mathrm{sq}}(\nabla f(x_{t},\xi_{t}))^{2}$
7:  $\hat{m}_t = \frac{m_t}{1 - (1 - \beta)^t}$
8:  $\hat{v}_t = \frac{v_t}{1 - (1 - \beta_{\mathrm{sq}})^t}$
9:  $x_{t + 1} = x_t - \frac{\eta}{\sqrt{v_t + \lambda}}\odot \hat{m}_t$
The formal definition of Adam proposed in [23] is shown in Algorithm 1, where Lines 5-9 describe the update rule of iterates  $\{x_{t}\}_{1\leq t\leq T}$ . Lines 5-6 are the updates for the first and second order momentum,  $m_{t}$  and  $v_{t}$ , respectively. In Lines 7-8, they are re-scaled to  $\hat{m}_t$  and  $\hat{v}_t$  in order to correct the initialization bias due to setting  $m_0 = v_0 = 0$ . Then the iterate is updated by  $x_{t + 1} = x_t - h_t\odot \hat{m}_t$  where  $h_t = \eta /(\sqrt{\hat{v}_t} +\lambda)$  is the adaptive stepsize vector for some parameters  $\eta$  and  $\lambda$ .
3.2 Assumptions
In what follows below, we will state our main assumptions for analysis of Adam.
3.2.1 Function class
We start with a standard assumption in optimization on the objective function  $f$  whose domain lies in a Euclidean space with dimension  $d$ .
Assumption 1. The objective function  $f$  is differentiable and closed within its open domain  $\operatorname{dom}(f) \subseteq \mathbb{R}^d$  and is bounded from below, i.e.,  $f^{*} \coloneqq \inf_{x} f(x) > -\infty$ .
Remark 3.1. A function  $f$  is said to be closed if its sub-level set  $\{x \in \operatorname{dom}(f) \mid f(x) \leq a\}$  is closed for each  $a \in \mathbb{R}$ . In addition, a continuous function  $f$  over an open domain is closed if and only if  $f(x)$  tends to infinity whenever  $x$  approaches to the boundary of  $\operatorname{dom}(f)$ , which is an important condition to ensure the iterates of Adam with a small enough stepsize  $\eta$  stay within the domain with high probability. Note that this condition is mild since any continuous function defined over the entire space  $\mathbb{R}^d$  is closed.
Besides Assumption 1, the only additional assumption we make regarding  $f$  is that its local smoothness is bounded by a sub-quadratic function of the gradient norm. More formally, we consider the following  $(\rho, L_0, L_\rho)$  smoothness condition with  $0 \leq \rho < 2$ .
Definition 3.2. A differentiable real-valued function  $f$  is  $(\rho, L_0, L_\rho)$  smooth for some constants  $\rho, L_0, L_\rho \geq 0$  if the following inequality holds almost everywhere in  $\operatorname{dom}(f)$
$$
\left| \left| \nabla^ {2} f (x) \right| \right| \leq L _ {0} + L _ {\rho} \| \nabla f (x) \| ^ {\rho}.
$$
Remark 3.3. When  $\rho = 1$ , Definition 3.2 reduces to the  $(L_0, L_1)$  smoothness condition in [49]. When  $\rho = 0$  or  $L_{\rho} = 0$ , it reduces to the standard smoothness condition.
Assumption 2. The objective function  $f$  is  $(\rho, L_0, L_\rho)$  smooth with  $0 \leq \rho < 2$ .
The standard smooth function class is very restrictive as it only contains functions that are upper and lower bounded by quadratic functions. The  $(L_0, L_1)$  smooth function class is more general since it also contains, e.g., univariate polynomials and exponential functions. Assumption 2 is even more general and contains univariate rational functions, double exponential functions, etc. See Appendix D.1 for the formal propositions and proofs. We also refer the reader to our concurrent work [26] for more detailed discussions of examples of  $(\rho, L_0, L_\rho)$  smooth functions for different  $\rho$ s.
It turns out that bounded Hessian norm at a point  $x$  implies local Lipschitzness of the gradient in the neighborhood around  $x$ . In particular, we have the following lemma.
Lemma 3.4. Under Assumptions 1 and 2, for any  $a > 0$  and two points  $x \in \mathrm{dom}(f), y \in \mathbb{R}^d$  such that  $\|y - x\| \leq \frac{a}{L_0 + L_\rho(\|\nabla f(x)\| + a)^\rho}$ , we have  $y \in \mathrm{dom}(f)$  and
$$
\left\| \nabla f (y) - \nabla f (x) \right\| \leq \left(L _ {0} + L _ {\rho} \left(\| \nabla f (x) \| + a\right) ^ {\rho}\right) \cdot \| y - x \|.
$$
Remark 3.5. Lemma 3.4 can be actually used as the definition of  $(\rho, L_0, L_\rho)$  smooth functions in place of Assumption 2. Besides the local gradient Lipschitz condition, it also suggests that, as long as the update at each step is small enough, the iterates will not go outside of the domain.
For the special case of  $\rho = 1$ , choosing  $a = \max \{\| \nabla f(x)\|, L_0 / L_1\}$ , one can verify that the required locality size in Lemma 3.4 satisfies  $\frac{a}{L_0 + L_1(\|\nabla f(x)\| + a)} \geq \frac{1}{3L_1}$ . In this case, Lemma 3.4 states that  $\| x - y\| \leq 1 / (3L_1)$  implies  $\| \nabla f(y) - \nabla f(x)\| \leq 2(L_0 + L_1\|\nabla f(x)\|)\| y - x\|$ . Therefore, it reduces to the local gradient Lipschitz condition for  $(L_0, L_1)$  smooth functions in [49; 48] up to numerical constant factors. For  $\rho \neq 1$ , the proof is more involved because Gronwall's inequality used in [49; 48] no longer applies. Therefore we defer the detailed proof of Lemma 3.4 to Appendix D.2.
3.2.2 Stochastic gradient
We consider one of the following two assumptions on the stochastic gradient  $\nabla f(x_{t},\xi_{t})$  in our analysis of Adam.
Assumption 3. The gradient noise is centered and almost surely bounded. In particular, for some  $\sigma \geq 0$  and all  $t \geq 1$ ,
$$
\mathbb {E} _ {t - 1} \left[ \nabla f \left(x _ {t}, \xi_ {t}\right) \right] = \nabla f (x _ {t}), \quad \| \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \| \leq \sigma , a. s.,
$$
where  $\mathbb{E}_{t - 1}[\cdot ]\coloneqq \mathbb{E}[\cdot |\xi_1,\ldots ,\xi_{t - 1}]$  is the conditional expectation given  $\xi_{1},\dots ,\xi_{t - 1}$
Assumption 4. The gradient noise is centered with sub-Gaussian norm. In particular, for some  $R \geq 0$  and all  $t \geq 1$ ,
$$
\mathbb {E} _ {t - 1} [ \nabla f (x _ {t}, \xi_ {t}) ] = \nabla f (x _ {t}), \quad \mathbb {P} _ {t - 1} \left(\| \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \| \geq s\right) \leq 2 e ^ {- \frac {s ^ {2}}{2 R ^ {2}}}, \forall s \in \mathbb {R},
$$
where  $\mathbb{E}_{t - 1}[\cdot ]\coloneqq \mathbb{E}[\cdot |\xi_1,\ldots ,\xi_{t - 1}]$  and  $\mathbb{P}_{t - 1}[\cdot ]\coloneqq \mathbb{P}[\cdot |\xi_1,\ldots ,\xi_{t - 1}]$  are the conditional expectation and probability given  $\xi_1,\dots ,\xi_{t - 1}$
Assumption 4 is strictly weaker than Assumption 3 since an almost surely bounded random variable clearly has sub-Gaussian norm, but it results in a slightly worse convergence rate up to poly-log factors (see Theorems 4.1 and 4.2). Both of them are stronger than the most standard bounded variance assumption  $\mathbb{E}[\| \nabla f(x_t,\xi_t) - \nabla f(x_t)\|^2 ]\leq \sigma^2$  for some  $\sigma \geq 0$ , although Assumption 3 is actually a common assumption in existing analyses under the  $(L_0,L_1)$  smoothness condition (see e.g. [49; 48]). The extension to the bounded variance assumption is challenging and a very interesting future work as it is also the assumption considered in the lower bound [3]. We suspect that such an extension would be straightforward if we consider a mini-batch version of Algorithm 1 with a batch size of  $S = \Omega (\epsilon^{-2})$ , since this results in a very small variance of  $\mathcal{O}(\epsilon^2)$  and thus essentially reduces the analysis to the deterministic setting. However, for practical Adam with an  $\mathcal{O}(1)$  batch size, the extension is challenging and we leave it as a future work.
In the section, we provide our convergence results for Adam under Assumptions 1, 2, and 3 or 4. To keep the statements of the theorems concise, we first define several problem-dependent constants. First, we let  $\Delta_1 \coloneqq f(x_1) - f^* < \infty$  be the initial sub-optimality gap. Next, given a large enough constant  $G > 0$ , we define
$$
r := \min  \left\{\frac {1}{5 L _ {\rho} G ^ {\rho - 1}}, \frac {1}{5 \left(L _ {0} ^ {\rho - 1} L _ {\rho}\right) ^ {1 / \rho}} \right\}, \quad L := 3 L _ {0} + 4 L _ {\rho} G ^ {\rho}, \tag {2}
$$
where  $L$  can be viewed as the effective smoothness constant along the trajectory if one can show  $\| \nabla f(x_{t})\| \leq G$  and  $\| x_{t + 1} - x_t\| \leq r$  at each step (see Section 5 for more detailed discussions). We will also use  $c_{1},c_{2}$  to denote some small enough numerical constants and  $C_1,C_2$  to denote some large enough ones. The formal convergence results under Assumptions 1, 2, and 3 are presented in the following theorem, whose proof is deferred in Appendix E.
Theorem 4.1. Suppose Assumptions 1, 2, and 3 hold. Denote  $\iota \coloneqq \log (1 / \delta)$  for any  $0 < \delta < 1$ . Let  $G$  be a constant satisfying  $G \geq \max \left\{2\lambda, 2\sigma, \sqrt{C_1\Delta_1L_0}, (C_1\Delta_1L_\rho)^{\frac{1}{2 - \rho}}\right\}$ . Choose
$$
0 \leq \beta_ {\mathrm {s q}} \leq 1, \quad \beta \leq \min  \left\{1, \frac {c _ {1} \lambda \epsilon^ {2}}{\sigma^ {2} G \sqrt {\imath}} \right\}, \quad \eta \leq c _ {2} \min  \left\{\frac {r \lambda}{G}, \frac {\sigma \lambda \beta}{L G \sqrt {\imath}}, \frac {\lambda^ {3 / 2} \beta}{L \sqrt {G}} \right\}.
$$
Let  $T = \max \left\{\frac{1}{\beta^2},\frac{C_2\Delta_1G}{\eta\epsilon^2}\right\}$ . Then with probability at least  $1 - \delta$ , we have  $\| \nabla f(x_{t})\| \leq G$  for every  $1\leq t\leq T$ , and  $\frac{1}{T}\sum_{t = 1}^{T}\| \nabla f(x_{t})\|^{2}\leq \epsilon^{2}$ .
Note that  $G$ , the upper bound of gradients along the trajectory, is a constant that depends on  $\lambda, \sigma, L_0, L_\rho$ , and the initial sub-optimality gap  $\Delta_1$ , but not on  $\epsilon$ . There is no requirement on the second order momentum parameter  $\beta_{\mathrm{sq}}$ , although many existing works like [12; 51; 45] need certain restrictions on it. We choose very small  $\beta$  and  $\eta$ , both of which are  $\mathcal{O}(\epsilon^2)$ . Therefore, from the choice of  $T$ , it is clear that we obtain a gradient complexity of  $\mathcal{O}(\epsilon^{-4})$ , where we only consider the leading term. We are not clear whether the dependence on  $\epsilon$  is optimal or not, as the  $\Omega(\epsilon^{-4})$  lower bound in [3] assumes the weaker bounded variance assumption than our Assumption 3. However, it matches the state-of-the-art complexity among existing analyses of Adam.
One limitation of the dependence of our complexity on  $\lambda$  is  $\mathcal{O}(\lambda^{-2})$ , which might be large since  $\lambda$  is usually small in practice, e.g., the default choice is  $\lambda = 10^{-8}$  in the PyTorch implementation. There are some existing analyses on Adam [12; 51; 45] whose rates do not depend explicitly on  $\lambda$  or only depend on  $\log(1 / \lambda)$ . However, all of them depend on  $\mathrm{poly}(d)$ , whereas our rate is dimension free. The dimension  $d$  is also very large, especially when training transformers, for which Adam is widely used. We believe that independence on  $d$  is better than that on  $\lambda$ , because  $d$  is fixed given the architecture of the neural network but  $\lambda$  is a hyper-parameter which we have the freedom to tune. In fact, based on our preliminary experimental results on CIFAR-10 shown in Figure 1, the performance of Adam is not very sensitive to the choice of  $\lambda$ . Although the default choice of  $\lambda$  is  $10^{-8}$ , increasing it up to 0.01 only makes minor differences.
As discussed in Section 3.2.2, we can generalize the bounded gradient noise condition in Assumption 3 to the weaker sub-Gaussian noise condition in Assumption 4. The following theorem formally shows the convergence result under Assumptions 1, 2, and 4, whose proof is deferred in Appendix E.6.
(a) CNN
(b) ResNet-Small
Figure 1: Test errors of different models trained on CIFAR-10 using the Adam optimizer with  $\beta = 0.9$ ,  $\beta_{\mathrm{sq}} = 0.999$ ,  $\eta = 0.001$  and different  $\lambda$ s. From left to right: (a) a shallow CNN with 6 layers; (b) ResNet-Small with 20 layers; and (c) ResNet110 with 110 layers.
(c) ResNet110
Theorem 4.2. Suppose Assumptions 1, 2, and 4 hold. Denote  $\iota := \log(2/\delta)$  and  $\sigma := R\sqrt{2\log(4T/\delta)}$  for any  $0 < \delta < 1$ . Let  $G$  be a constant satisfying  $G \geq \max \left\{2\lambda, 2\sigma, \sqrt{C_1\Delta_1L_0}, (C_1\Delta_1L_\rho)^{\frac{1}{2-\rho}}\right\}$ . Choose
Let  $T = \max \left\{\frac{1}{\beta^2}, \frac{C_2 \Delta_1 G}{\eta \epsilon^2}\right\}$ . Then with probability at least  $1 - \delta$ , we have  $\| \nabla f(x_t) \| \leq G$  for every  $1 \leq t \leq T$ , and  $\frac{1}{T} \sum_{t=1}^{T} \| \nabla f(x_t) \|^2 \leq \epsilon^2$ .
Note that the main difference of Theorem 4.2 from Theorem 4.1 is that  $\sigma$  is now  $\mathcal{O}(\sqrt{\log T})$  instead of a constant. With some standard calculations, one can show that the gradient complexity in Theorem 4.2 is bounded by  $\mathcal{O}(\epsilon^{-4}\log^p (1 / \epsilon))$ , where  $p = \max \left\{3,\frac{9 + 2\rho}{4}\right\} < 3.25$ .
5.1 Bounding the gradients along the optimization trajectory