SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ilyas Fatkhullin
ilyas.fatkhullin@ai.ethz.ch
Xiang Li*
xiang.li@inf.ethz.ch
The classical analysis of Stochastic Gradient Descent (SGD) with polynomially decaying stepsize  $\eta_t = \eta / \sqrt{t}$  relies on well-tuned  $\eta$  depending on problem parameters such as Lipschitz smoothness constant, which is often unknown in practice. In this work, we prove that SGD with arbitrary  $\eta > 0$ , referred to as untuned SGD, still attains an order-optimal convergence rate  $\widetilde{\mathcal{O}}(T^{-1/4})$  in terms of gradient norm for minimizing smooth objectives. Unfortunately, it comes at the expense of a catastrophic exponential dependence on the smoothness constant, which we show is unavoidable for this scheme even in the noiseless setting. We then examine three families of adaptive methods — Normalized SGD (NSGD), AMSGrad, and AdaGrad — unveiling their power in preventing such exponential dependency in the absence of information about the smoothness parameter and boundedness of stochastic gradients. Our results provide theoretical justification for the advantage of adaptive methods over untuned SGD in alleviating the issue with large gradients.
In this work, we study the stochastic optimization problem of the form:
$$
\min  _ {x \in \mathbb {R} ^ {d}} f (x) = \mathbb {E} _ {\xi \sim P} [ F (x; \xi) ],
$$
where  $P$  is an unknown probability distribution, and  $f: \mathbb{R}^d \to \mathbb{R}$  is an  $\ell$ -Lipschitz smooth function and can be non-convex. In the context of machine learning,  $\xi$  typically represent an individual training sample drawn from the data distribution  $P$ , and  $x$  denotes the weights of the model.
Stochastic Gradient Descent (SGD), originating from the seminal work [58], performs the following update iteratively:
$$
x _ {t + 1} = x _ {t} - \eta_ {t} \nabla F (x _ {t}; \xi_ {t}),
$$
where  $\eta_t > 0$  presents a positive stepsize, and  $\nabla F(x_t; \xi_t)$  is an unbiased stochastic gradient. SGD has shown remarkable empirical success in many modern machine learning applications, e.g., [7, 62]. Its efficiency is usually attributed to its cheap per iteration cost and the ability to operate in an
Table 1: Complexities of finding an  $\epsilon$ -stationary point for SGD, NSGD [53], NSGD-M [14], AMSGrad-norm (norm version of AMSGrad [57]), and AdaGrad-norm [61]. We only assume  $f$  is  $\ell$ -smooth, and unbiased stochastic gradients have bounded variance  $\sigma^2$ . Hyper-parameters (e.g.,  $\gamma$  and  $\eta$ ) are untuned. Here,  $\tilde{\mathcal{O}}$  and  $\Omega$  hide polynomial terms in problem parameters and hyperparameters. The bounds are with respect to specific algorithms and step sizes, and lower bounds for general first-order methods still hold [11, 3]. We list each algorithm's effective stepsize at iteration  $t$  directly below its name.
Algorithms	Upper bound; deterministic	Lower bound; deterministic	Upper bound; stochastic	Lower bound; stochastic
SGD (Eq. 1) η/√t+1	O((4e)2(ηℓ)2ε-4)[Thm. 1, 6]	Ω((8e)η2ℓ2/8ε-4)[Thm. 2]	O((4e)2(ηℓ)2ε-4)[Thm. 1, 6]	Ω((8e)η2ℓ2/8ε-4)[Thm. 2]
NSGD (Alg. 3) γ/√t+1||g(xt;ξt)||	O(ε-2)[14] & [Prop. 1]	Ω(ε-2)[11]	N/A due to lower bound	Nonconvergent [Thm. 3]
NSGD-M (Alg. 1) γ(t+1)α||gt||	O(ε-2), α=1/2[14] & [Prop. 1]	Ω(ε-2)[11]	O(ε-4), α=3/4[14] & [Prop. 1]	Ω(ε-4)[3]
AMSGrad-norm (Alg. 2) γ/√(t+1)vt+1	O(ε-4)[Thm. 5, 7]	Ω(ε-4)[Thm. 8]	N/A due to lower bound	Ω(ε-2/1-ζ) ∀ζ∈(1/2,1)[Thm. 4]
AdaGrad-norm (Alg. 5) η/√v02+Σk=0||g(xk;ξk)||2	O(ε-2)[68] & [Prop. 3]	Ω(ε-2)[11]	O(ε-4)[68] & [Prop. 3]	Ω(ε-4)[3]
online fashion, making it suitable for large-scale problems. However, empirical evidence also reveals undesirable behaviors of SGD, often linked to challenges in selecting appropriate step sizes. In particular, a number of works report the gradient explosion effect [6, 55, 24] during the initial phase of training, which may eventually lead to divergence or prohibitively slow convergence. The phenomenon is also observed in our experiments (see Figure 1(f)) when the stepsize is poorly chosen. Unfortunately, this phenomenon has not been well understood from a theoretical point of view. The classical analysis of SGD in the smooth non-convex case [22], prescribes to select a non-increasing sequence of step sizes  $\{\eta_t\}_{t\geq 1}$  with  $\eta_1 < 2 / \ell$ . In particular, the choice  $\eta_t = 1 / (\ell \sqrt{t})$ , guarantees² to find a point  $x$  with  $\mathbb{E}[\| \nabla f(x)\| ]\leq \epsilon$  after  $\mathcal{O}\left(\epsilon^{-4}\right)$  stochastic gradient calls, which is also known to be unimprovable in the smooth non-convex setting unless additional assumptions are made [3, 17].
However, the bound on the smoothness parameter  $\ell$  is usually not readily available for practitioners, and the limited computing power usually prevents them from exhaustive tuning to find the best stepsize. It is therefore important to provide a theoretical understanding for SGD with an arbitrary stepsize (which we refer to as untuned SGD) that is agnostic to the problem parameter. The following intriguing question remains elusive in the stochastic optimization literature:
How does untuned SGD with a decaying stepsize  $\eta_t = \eta / \sqrt{t}$  perform when  $\eta$  is independent of the smoothness parameter? How can we explain the undesirably large gradients encountered in training with SGD?
Recently, there has been a surge of interest in adaptive gradient methods such as Adam [35], RMSProp [28], AdaDelta [69], AMSGrad [57], AdaGrad [18], Normalized SGD [26], and many others. These methods automatically adjust their step sizes based on past stochastic gradients, rather than using pre-defined iteration-based schedules. Empirically, they are observed to converge faster than SGD and mitigate the issue of gradient explosion across a range of problems, even without explicit knowledge of problem-specific parameters [35, 46, 55]. Figure 1 provides a basic illustration of performance differences between SGD with  $\eta_t = 1/\sqrt{t}$  stepsizes and adaptive schemes such as AdaGrad and Normalized SGD with momentum (NSGD-M) [14]. Notably, when the initial step size is too large (relative to  $1/\ell$  value), SGD reaches the region with large gradients, while adaptive methods do not
2 Given access to unbiased stochastic gradient oracle with bounded variance.
(a) quadratic function  $(\ell = 1)$
(b) quadratic function  $(\ell = 5)$
(c) quadratic function  $(\ell = 10)$
(d) ResNet  $(\eta = 3)$
(e) ResNet  $(\eta = 5)$
(f) ResNet  $(\eta = 7)$
Figure 1: Comparison of SGD, AdaGrad, and NSGD-M on a quadratic function  $f(x) = \ell x^{2} / 2$  and a deep neural network. SGD employs a diminishing stepsize of  $\eta / \sqrt{t}$ , while the step sizes for AdaGrad and NSGD-M are specified in Propositions 2 and 3, respectively. For experiments on the quadratic function, we set  $\eta = 1$  for all methods and test with 3 different values of  $\ell$ . For the figures in the second row, We train a 50-layer ResNet [27] on the CIFAR-10 dataset [36] using cross-entropy loss. Due to the large randomness observed for this setting, we run each combination of optimizer and stepsize 10 times and plot both the average and variance. Note that the scale of the y-axis varies for each sub-figure and SGD diverges to NaN when  $\eta = 7$ .
suffer from such effect. However, the theoretical benefits of adaptive methods over SGD remain unclear. A large number of existing analyses of adaptive methods assume bounded gradients, or even stochastic gradients, precluding not only a fair comparison with SGD whose convergence does not need bounded gradient but also the possibility to explain their benefit when facing gradient explosions. While recent developments show that AdaGrad-type methods [20, 68] can attain a  $\tilde{\mathcal{O}}\left(\epsilon^{-4}\right)$  sample complexity under the same standard assumptions as for SGD analysis, there is still a lack a good explanation for the substantial performance gap observed in practice despite SGD with well-tuned stepsizes theoretically achieving the lower complexity bound. We will address the following open question:
Can we justify the theoretical benefits of adaptive methods over untuned SGD for smooth non-convex problems without assuming bounded gradients?
Consequently, this work is grounded on the premise that gradients may be not bounded and that hyper-parameters are independent of the problem parameters. The main contributions are as follows:
- We show that untuned SGD with diminishing stepsizes  $\eta_t = \eta / \sqrt{t}$  finds an  $\epsilon$ -stationary point of an  $\ell$ -smooth function within  $\widetilde{\mathcal{O}}((\ell^2 + \sigma^4 \eta^4 \ell^4)(4e)^{2\eta^2 \ell^2} \epsilon^{-4})$  iterations for any  $\eta > 0$ . Here,  $\sigma^2$  corresponds to the variance of the stochastic gradient. Although this classical algorithm converges and has the optimal dependence on  $\epsilon$ , we further show that the disastrous exponential term in  $\eta^2 \ell^2$  is unavoidable even when the algorithm has access to exact gradients. This explains its proneness to gradient explosion when the problem parameter is unknown. Previous analyses have failed to capture this exponential term, since they assume  $\eta$  is well-tuned to be  $\Theta(1/\ell)$ .
- AMSGrad, which was proposed to fix the nonconvergence of Adam, is not yet fully understood, with previous analyses depending on bounded stochastic gradients. We show that AMSGrad (norm version) is free from exponential constants in the deterministic setting without tuning, in stark contrast with SGD. Surprisingly, in the stochastic setting when the stochastic gradients are
unbounded, we show that AMSGrad may converge at an arbitrarily slow polynomial rate. To the best of our knowledge, these are the first results for AMSGrad without assuming bounded gradients.
- To further illuminate the advantages of adaptive methods, we re-examine the results for Normalized Gradient Descent (NGD), Normalized SGD with momentum (NSGD-M) from [14] and AdaGrad-norm from [68], considering stepsize independent of the problem parameters similar to untuned SGD. They all achieve near-optimal complexities while shredding off the exponential factor. As a side result, we provide a strong non-convergence result of NSGD without momentum under any bounded stepsizes, which might be of independent interest.
Our findings contribute a fresh understanding of the performance gap between SGD and adaptive methods. Albeit with a near-optimal rate, untuned SGD is vulnerable to gradient explosion and slow convergence due to a large exponential constant in its complexity, which can be circumvented by several adaptive methods. To the best of our knowledge, this substantial difference is unaddressed in the previous literature, because the majority of analyses for SGD and adaptive methods turn to either well-tuned stepsize based on problem parameters or the assumption of bounded gradients. Part of our results are summarized in Table 1, and full results for a broader range of step sizes can be found in Table 2 in the appendix.
SGD in nonconvex optimization. Stochastic approximation methods and SGD in particular have a long history of development [58, 34, 8, 13, 51, 56]. When the objective is  $\ell$ -smooth and the gradient noise has bounded variance  $\sigma^2$ , Ghadimi and Lan [22] and Bottou et al. [9] prove that if stepsize  $\eta_t = \eta / \sqrt{T}$ , where  $\eta = \eta(\ell, \sigma^2)$  and  $T$  is the total iteration budget, then SGD can find an  $\epsilon$ -stationary point within  $\mathcal{O}(\ell \sigma^2 \epsilon^{-4})$  iterations. Similar complexity (up to a logarithmic term) can also be achieved by decaying stepsizes  $\eta / \sqrt{t}$  [22, 17, 65]. This result was later shown to be optimal for first-order methods under these assumptions [3]. Several works consider various relaxations of the stochastic oracle model with bounded variance, for instance, biased oracle [2] and relaxed growth condition [9]. However, these results also heavily rely on sufficiently small  $\eta$ , e.g.,  $\eta \leq 1/\ell$ , and the convergence behavior in the large  $\eta$  regime is rarely discussed. Remarkably, Lei et al. [39] characterize the convergence of SGD under individual smoothness and unbiased function values. They consider Robbins-Monro stepsize schemes, which includes  $\eta / t^\alpha$  when  $\alpha > 1/2$ , and derive  $\mathcal{O}(\epsilon^{\frac{2}{\alpha - 1}})$  sample complexity including an exponential dependence on the smoothness parameter. Unlike [39], we focus on the standard assumptions and derive better dependency in smoothness constant when  $\alpha > 1/2$ . Importantly, we further justify that the exponential constants are unavoidable with a lower bound.
Adaptive methods. We focus on methods directly using gradients to adjust stepsize, rather than other strategies like backtracking line search [4]. Normalized Gradient Descent (NGD) was introduced by [53] for quasi-convex functions. Hazan et al. [26] apply NGD and NSGD with minibatch to the class of locally-quasi-convex functions. Later, Cutkosky and Mehta [14] and Zhao et al. [74] prove NSGD with momentum or minibatch can find an  $\epsilon$ -stationary point in smooth nonconvex optimization with sample complexity  $\mathcal{O}(\epsilon^{-4})$ . AdaGrad was introduced in the online convex optimization [18, 48]. In nonconvex optimization, AdaGrad and its scalar version, AdaGrad-norm [61], achieve competitive convergence rates with SGD [66, 43, 31, 44]. RMSProp [28] and Adam [35] use the moving average of past gradients, but may suffer from divergence without hyper-parameter tuning [57]. Recently, it was shown in the finite-sum setting that they converge to a neighborhood, whose size shrinks to 0 by tuning hyper-parameters [60, 72]. However, most of the results on AdaGrad and Adam-type algorithms assume both Lipschitz and bounded gradients [75, 12, 16, 66, 76]. Very recently, Faw et al. [20] and Yang et al. [68] independently show that AdaGrad-norm converges without assuming bounded gradients and without the need for tuning, attaining a sample complexity of  $\widetilde{\mathcal{O}} (\epsilon^{-4})$ .
SGD v.s. adaptive methods. Despite similar complexities, adaptive methods typically converge faster than SGD in practice [10, 47] and are widely used to prevent large gradients [55, 23]. Various attempts have been made to theoretically explain these differences. Some suggest that the advantage of adaptive algorithms is their ability to achieve order-optimal rates without knowledge of problem parameters such as smoothness and noise variance [66, 40, 30]. Other studies investigate the faster
escape from saddle points by adaptive methods [41, 50, 67]. The importance of coordinate-wise normalization in Adam has also been highlighted [5, 37]. Furthermore, the influence of heavy-tail noise on the performance of adaptive methods is studied [71]. However, most previous works do not provide an explanation for the faster convergence of adaptive methods in terms of sample complexity. Notably, Zhang et al. [70] and Wang et al. [64] explain the benefits of gradient clipping and Adam under a relaxed smoothness assumption, a setting where SGD with non-adaptive stepsizes may not converge. In contrast, we analyze SGD and several adaptive methods under standard smoothness and noise assumptions, distinguishing it from the recent work of Wang et al. [64] that focuses on one variant of Adam for finite-sum problems with individual relaxed smoothness and random shuffling.
Throughout this work, we focus on minimizing an  $\ell$ -smooth function  $f: \mathbb{R}^d \to \mathbb{R}$ . We have access to a stochastic gradient oracle that returns  $g(x; \xi)$  at any point  $x$ , and we make the following standard assumptions in nonconvex optimization.
Assumption 1 (smoothness). Function  $f(x)$  is  $\ell$ -smooth with  $\ell > 0$ , that is,  $\| \nabla f(x_1) - \nabla f(x_2) \| \leq \ell \| x_1 - x_2 \|$  for any  $x_1$  and  $x_2 \in \mathbb{R}^d$ .
Assumption 2 (stochastic gradients). The stochastic gradient  $g(x;\xi)$  is unbiased and has a bounded variance, that is,  $\mathbb{E}_{\xi}\left[g(x;\xi)\right] = \nabla f(x)$  and  $\mathbb{E}_{\xi}\left[\| g(x;\xi) - \nabla f(x)\| ^2\right]\leq \sigma^2$  for any  $x\in \mathbb{R}^d$ .
We present the general scheme of SGD with initial point  $x_0$  and a stepsize sequence  $\{\eta_t\}_{t=0}^{\infty}$ :
$$
x _ {t + 1} = x _ {t} - \eta_ {t} g \left(x _ {t}; \xi_ {t}\right). \tag {1}
$$
Some commonly used stepsizes include polynomially and geometrically decaying stepsize, constant stepsize, cosine stepsize, etc. When the stepsize depends on the instantaneous or past gradients, i.e.,  $\{g(x;\xi_k)\}_{k\leq t}$ , we call it adaptive stepsize, namely Normalized SGD [26], AdaGrad [18], Adam [35], AMSGrad [57], etc. In some adaptive methods, momentum is also considered, replacing  $g(x_{t};\xi_{t})$  in (1) with a moving average  $m_{t + 1}$  of the past stochastic gradients (see Section 4 for more details). To set the stage for our analysis, we assume that  $f(x_0) - \min_{x\in \mathbb{R}^d}f(x)\leq \Delta$ , where  $\Delta$  represents the initial gap. Given that the function class of interest is nonconvex, we aim to find an  $\epsilon$ -stationary point  $x$  with  $\mathbb{E}[\| \nabla f(x)\| ]\leq \epsilon$ .
3 Convergence of Untuned SGD
In this section, we focus on SGD with the decaying stepsize:
$$
\eta_ {t} = \frac {\eta}{\sqrt {t + 1}},
$$
where  $\eta > 0$  is the initial stepsize. Most convergent analysis requires  $\eta < 2/\ell$  [22, 9] so that there is "sufficient decrease" in function value after each update, and if  $\eta$  is carefully chosen, it can achieve the near-optimal complexity of  $\widetilde{\mathcal{O}}(\ell\epsilon^{-4}\sigma^2)$  [3]. Nevertheless, as the smoothness parameter is usually unknown, providing guarantees with optimal  $\eta$  or assuming  $\eta$  to be problem-dependent does not give enough insights into practical training with SGD. Hence we are interested in its convergence behavior in both small and large initial stepsize regimes, i.e.,  $\eta \leq 1/\ell$  and  $\eta > 1/\ell$ .
Theorem 1. Under Assumptions 1 and 2, if we run SGD with stepsize  $\eta_t = \frac{\eta}{\sqrt{t + 1}}$ , where  $\eta > 0$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \left\{ \begin{array}{l l} 2 A \eta^ {- 1} T ^ {- \frac {1}{2}}, & \text {w h e n} \eta \leq 1 / \ell , \\ 4 \sqrt {2} \ell A (4 e) ^ {\tau} (\pi T) ^ {- \frac {1}{2}}, & \text {w h e n} \eta > 1 / \ell , \end{array} \right.
$$
where  $\tau = \lceil \eta^2\ell^2 -1\rceil$  and  $A = \left(\Delta +\frac{\ell\sigma^2\eta^2}{2} (1 + \log T)\right)$
This theorem implies that when the initial stepsize  $\eta > 1 / \ell$ , SGD still converges with a sample complexity of  $\widetilde{\mathcal{O}}((\ell^2 + \sigma^4 \eta^4 \ell^4)(4e)^{2\eta^2 \ell^2} \epsilon^{-4})$ . Although the dependency in the target accuracy  $\epsilon$  is
Figure 2: Demonstration of the constructed function used to prove the lower bound. The definitions of the segments comprising the function can be found in Appendix B.2.
near-optimal, it includes a disastrous exponential term in  $\eta^2\ell^2$ . This is due to polynomially decaying stepsizes: in the first stage before  $\tau = \lceil (\eta \ell)^2 - 1 \rceil$  iterations, the function value and gradients may keep increasing in expectation until reaching an exponential term in  $\eta^2\ell^2$ , which is in stark contrast with adaptive methods that we will see in Section 4; in the second stage after  $t \geq \tau$ , the stepsize is small enough to decrease the function value in expectation at a rate of  $1 / \sqrt{T}$  up to a small term in  $\sigma$ .
If we arbitrarily select  $\eta = \Theta(1)$ , untuned SGD may induce large gradients growing exponentially in  $\ell$  in the first stage, as observed in Figure 1. Moreover, deriving the dependence in hyper-parameter  $\eta$  is crucial for assessing the effort required in its tuning: SGD with an  $\eta$  that is  $c > 1$  times larger than the optimally tuned value can yield a gradient norm that is  $\exp(\mathrm{poly}(c))$  times larger in the convergence guarantee.
To the best of our knowledge, there is limited study for non-asymptotic analysis of untuned SGD under the same assumptions. Moulines and Bach [49] study untuned SGD under individual smoothness and convexity assumptions, i.e.,  $g(x;\xi)$  is Lipschitz continuous and  $F(x;\xi)$  is convex almost surely. They showed a convergence rate of  $\mathcal{O}(1 / T^{1 / 3})$  for the last iterate and a rate arbitrarily close to  $\mathcal{O}(1 / T^{1 / 2})$  for the averaged iterate. Later, Fontaine et al. [21] provide  $\widetilde{\mathcal{O}} (1 / T^{1 / 2})$  convergence rate for untuned SGD in the convex setting, albeit without an explicit dependency in  $\ell$  and  $\eta$ . In the nonconvex setting, Khaled and Richtárik [33] showed, using SGD with constant stepsize  $\eta >0$ , the squared gradient  $\min_{t < T}\mathbb{E}\| \nabla f(x_t)\| ^2$  converges at a rate of  $\mathcal{O}\left(\ell \sigma \eta +(1 + \ell^2\eta^2)^T\Delta /( \eta T)\right)$ . When the total number of iterations  $T$  is predetermined, selecting a constant stepsize  $\eta = \eta_0 / \sqrt{T}$  results in the rate  $\mathcal{O}\left(\frac{\ell\sigma\eta_0}{\sqrt{T}} +\frac{(1 + \ell^2\eta_0^2 / T)^T\Delta}{\eta_0\sqrt{T}}\right)\leq \mathcal{O}\left(\frac{\ell\sigma\eta_0}{\sqrt{T}} +\frac{e^{\ell^2\eta_0^2}\Delta}{\eta_0\sqrt{T}}\right)$ , which exhibits the same dependence on  $T$  as Theorem 1 and also includes an exponential constant. However, our focus is on settings where  $T$  is not known in advance.
Remark 1. We consider stepsize in the order of  $1 / \sqrt{t}$ , because it is known for SGD to achieve the best dependency in  $\epsilon$  for nonconvex optimization [17] and easier to compare with adaptive stepsizes. We also present the convergence results for more general polynomialsally decaying stepsizes, i.e.,  $\eta_t = \frac{\eta}{(t + 1)^\alpha}$  with  $0 < \alpha < 1$ , in Theorem 6 of the appendix. There exists a trade-off between convergence speed  $\mathcal{O}(1 / T^{\frac{1 - \alpha}{2}})$  and the exponential term in  $(\eta \ell)^{1 / \alpha}$  for  $\alpha \in [1 / 2, 1)$ . Intuitively, larger  $\alpha$  leads to a shorter time in adapting to  $1 / \ell$  stepsize but a slower convergence rate. We do not consider constant stepsize, i.e.,  $\alpha = 0$ , because it is well known to diverge even in the deterministic setting if the stepsize is agnostic to the problem parameter [52, 1].
The question arises as to whether the exponential term is necessary. In the following, we provide a lower bound for SGD under this choice of stepsize.
Theorem 2. Fixing  $T \geq 1, \eta > 0, \ell > 0$  and  $\Delta > 0$  that  $\eta \ell \geq 5$ , there exists a  $\ell$ -smooth function  $f: \mathbb{R} \to \mathbb{R}$  and an initial point  $x_0$  with  $f(x_0) - f^* \leq \Delta$  such that if we run Gradient Descent with stepsize  $\eta_t = \frac{\eta}{\sqrt{t + 1}}$ , then for  $t \leq t_0 = \lfloor \eta^2 \ell^2 / 16 - 1 \rfloor$ ,
$$
| \nabla f (x _ {t}) | \geq \sqrt {\frac {2 \ell \Delta}{3 \sqrt {t}}} (8 e) ^ {t / 2} a n d | \nabla f (x _ {t _ {0}}) | \geq \sqrt {\frac {8 \Delta}{3 \eta}} (8 e) ^ {\eta^ {2} \ell^ {2} / 3 2 - 4};
$$
if  $T > t_0$ , then for  $t_0 < t \leq T$
$$
| \nabla f (x _ {t}) | \geq \frac {1}{4} \sqrt {\tilde {\Delta}} \min  \left\{\ell^ {1 / 2}, (2 \eta) ^ {- 1 / 2} T ^ {- 1 / 4} \right\}, w h e r e \tilde {\Delta} \geq \frac {4}{3 \eta \ell} (8 e) ^ {\eta^ {2} \ell^ {2} / 1 6 - 2} \Delta .
$$
This theorem suggests that Gradient Descent (GD) with decaying stepsize  $\eta / \sqrt{t + 1}$  needs at least  $\Omega (\eta^{-4}\ell^{-2}(8e)^{\eta^2\ell^2 /8}\epsilon^{-4})$  iterations to find an  $\epsilon$ -stationary point in the large initial stepsize regime. Therefore, it validates that an exponential term in  $\eta^2\ell^2$  multiplied by  $\epsilon^{-4}$  is not avoidable even in the deterministic setting. It is crucial to note that our result is limited to untuned (S)GD with this particular stepsize scheme.