SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we study stochastic gradient methods for minimizing smooth functions in the parameter-agnostic regime. Firstly, we show SGD with polynomially decaying stepsize  $1 / \sqrt{t}$  is able to converge with the order-optimal rate, with and without bounded gradients (Proposition 4 and Theorem 1). Its limitation lies in an unavoidable exponential term in  $\ell^2$  when we do not assume bounded gradients (Theorem 2). We demonstrate that several existing adaptive methods do not suffer from the exponential dependency, such as NGD, AdaGrad, AMSGrad-norm in the deterministic setting (Proposition 1 and Theorem 5), and NSGD-M, AdaGrad in the stochastic setting (Proposition 2 and Proposition 3). However, it does not mean adaptive methods are always better than SGD. We provide a non-convergence result for NSGD (Theorem 3) and a slow convergence result for AMSGrad-norm (Thoerem 4) in the stochastic case. We believe our results shed light on explaining commonly observed large gradients during training and provide a better theoretical understanding of the convergence behaviors of adaptive methods in the regime with unbounded stochastic gradients.
B Proofs for SGD in Section 3
B.1 Upper Bounds for SGD
We provide an extended theorem of Theorem 1 and include more general decaying step sizes  $\eta_t = \eta/(t+1)^\alpha$  with  $0 < \alpha < 1$ .
Theorem 6. Under Assumptions 1 and 2, if we run SGD with stepsize  $\eta_t = \eta / (t + 1)^\alpha$  where  $\eta > 0$  and  $1/2 \leq \alpha < 1$ , then with  $\eta \leq 1/\ell$ ,
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq \left\{ \begin{array}{l} \frac {2}{\eta \sqrt {T}} \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2} (1 + \log T)\right), w h e n \alpha = 1 / 2, \\ \frac {2}{\eta T ^ {1 - \alpha}} \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2 (1 - 2 ^ {1 - 2 \alpha})}\right), w h e n 1 / 2 <   \alpha <   1, \\ \frac {2}{\eta T ^ {\alpha}} \left(\frac {\Delta}{T ^ {1 - 2 \alpha}} + \frac {\ell \sigma^ {2} \eta^ {2}}{2 (1 - 2 \alpha)}\right), w h e n 0 <   \alpha <   1 / 2; \end{array} \right.
$$
with  $\eta > 1 / \ell$ ,
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq \left\{ \begin{array}{l} \frac {\sqrt {2} (4 e) ^ {\tau}}{\eta \sqrt {\pi \tau T}} \left[ 1 + \ell \eta (1 + 2 \sqrt {\tau}) \right] \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2} (1 + \log T)\right), \\ \text {w h e n} \alpha = 1 / 2, \\ \frac {2 (4 e ^ {2 \alpha}) ^ {\tau}}{\eta (2 \pi \tau) ^ {\alpha} T ^ {1 - \alpha}} \left[ 1 + \ell \eta \left(1 + \frac {\tau^ {1 - \alpha}}{1 - \alpha}\right) \right] \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2 (1 - 2 ^ {1 - 2 \alpha})}\right), \\ \text {w h e n} 1 / 2 <   \alpha <   1, \\ \frac {2 (4 e ^ {2 \alpha}) ^ {\tau}}{\eta (2 \pi \tau) ^ {\alpha} T ^ {\alpha}} \left[ 1 + \ell \eta \left(1 + \frac {\tau^ {1 - \alpha}}{1 - \alpha}\right) \right] \left(\frac {\Delta}{T ^ {1 - 2 \alpha}} + \frac {\ell \sigma^ {2} \eta^ {2}}{2 (1 - 2 \alpha)}\right), \\ \text {w h e n} 0 <   \alpha <   1 / 2, \end{array} \right.
$$
where  $\tau = \lceil (\eta \ell)^{1 / \alpha} - 1\rceil$
Proof. By  $\ell$ -smoothness of  $f(\cdot)$ ,
$$
\begin{array}{l} f (x _ {t + 1}) \leq f (x _ {t}) + \langle \nabla f (x _ {t}), x _ {t + 1} - x _ {t} \rangle + \frac {\ell}{2} \| x _ {t + 1} - x _ {t} \| ^ {2} \\ = f (x _ {t}) - \eta_ {t} \langle \nabla f (x _ {t}), g (x _ {t}; \xi_ {t}) \rangle + \frac {\ell \eta_ {t} ^ {2}}{2} \| g (x _ {t}; \xi_ {t}) \| ^ {2} \\ \end{array}
$$
Taking expectation,
$$
\begin{array}{l} \mathbb {E} f (x _ {t + 1}) \leq \mathbb {E} f (x _ {t}) - \eta_ {t} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2} \\ \leq \mathbb {E} f (x _ {t}) - \left(\eta_ {t} - \frac {\ell \eta_ {t} ^ {2}}{2}\right) \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2} \tag {2} \\ \end{array}
$$
We note that  $\eta_t - \frac{\ell\eta_t^2}{2} \geq \frac{\eta_t}{2}$  when  $\eta_t \leq \frac{1}{\ell}$ , i.e.,  $t \geq (\eta \ell)^{1/\alpha} - 1$ . Define  $\tau = \lceil (\eta \ell)^{1/\alpha} - 1 \rceil$ . Therefore, for all  $t < \tau$ ,
$$
\mathbb {E} f \left(x _ {t + 1}\right) \leq \mathbb {E} f \left(x _ {t}\right) + \frac {\ell \eta_ {t} ^ {2}}{2} \mathbb {E} \| \nabla f \left(x _ {t}\right) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}. \tag {3}
$$
For all  $t \geq \tau$ , we have
$$
\mathbb {E} f \left(x _ {t + 1}\right) \leq \mathbb {E} f \left(x _ {t}\right) - \frac {\eta_ {t}}{2} \mathbb {E} \| \nabla f \left(x _ {t}\right) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}. \tag {4}
$$
Summing from  $t = \tau$  to  $T - 1$ , we have
$$
\sum_ {t = \tau} ^ {T - 1} \frac {\eta_ {t}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \mathbb {E} f (x _ {\tau}) - \mathbb {E} f (x _ {T}) + \sum_ {t = \tau} ^ {T - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2} \tag {5}
$$
Now we want to bound  $\mathbb{E}f(x_{\tau}) - f(x_{T})\leq \mathbb{E}f(x_{\tau}) - f^{*}$ , where  $f^{*}\triangleq \min_{x\in \mathbb{R}^{d}}f(x)$ . From (3),
$$
\begin{array}{l} \mathbb {E} f (x _ {t + 1}) - f ^ {*} \leq \mathbb {E} f (x _ {t}) - f ^ {*} + \frac {\ell \eta_ {t} ^ {2}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2} \\ \leq (1 + \ell^ {2} \eta_ {t} ^ {2}) [ \mathbb {E} f (x _ {t}) - f ^ {*} ] + \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}, \\ \end{array}
$$
where in the second inequality we use  $\| \nabla f(x)\| ^2\leq 2\ell [f(x) - f^* ]$  . When  $\tau = 0,f(x_{\tau}) - f(x_{T})\leq \Delta$  when  $\tau \geq 1$  , recursing the inequality above, for  $j\leq \tau$
$$
\mathbb {E} f (x _ {j}) - f ^ {*} \leq \Delta \left(\prod_ {t = 0} ^ {j - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) + \sum_ {k = 0} ^ {j - 2} \left(\prod_ {t = k + 1} ^ {j - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) \frac {\ell \eta_ {k} ^ {2}}{2} \sigma^ {2} + \frac {\ell \eta_ {j - 1} ^ {2}}{2} \sigma^ {2}
$$
$$
\begin{array}{l} \leq \left(\prod_ {t = 0} ^ {j - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) \left(\Delta + \sum_ {t = 0} ^ {j - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}\right) \\ \leq \left(\prod_ {t = 0} ^ {\tau - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) \left(\Delta + \sum_ {t = 0} ^ {\tau - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}\right). \tag {6} \\ \end{array}
$$
Also, with  $\| \nabla f(x)\| ^2\leq 2\ell [f(x) - f^* ]$  , if  $\tau \geq 1$
$$
\begin{array}{l} \sum_ {t = 0} ^ {\tau - 1} \frac {\eta_ {t}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \sum_ {t = 0} ^ {\tau - 1} \eta_ {t} \ell \mathbb {E} \left(f (x _ {t}) - f ^ {*}\right) \\ \leq \ell \left(\sum_ {t = 0} ^ {\tau - 1} \eta_ {t}\right) \left(\prod_ {t = 0} ^ {\tau - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) \left(\Delta + \sum_ {t = 0} ^ {\tau - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}\right), \\ \end{array}
$$
where in the second inequality we use (6). Combining with (5) and (6), if  $\tau \geq 1$
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \left(\prod_ {t = 0} ^ {\tau - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) \left(\Delta + \sum_ {t = 0} ^ {\tau - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}\right) + \sum_ {t = 0} ^ {T - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2} \\ + \ell \left(\sum_ {t = 0} ^ {\tau - 1} \eta_ {t}\right) \left(\prod_ {t = 0} ^ {\tau - 1} 1 + \ell^ {2} \eta_ {t} ^ {2}\right) \left(\Delta + \sum_ {t = 0} ^ {\tau - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}\right). \\ \end{array}
$$
$$
\begin{array}{l} \prod_ {t = 0} ^ {\tau - 1} \left(1 + \ell^ {2} \eta_ {t} ^ {2}\right) = \prod_ {t = 0} ^ {\tau - 1} \left(1 + \frac {\ell^ {2} \eta^ {2}}{(t + 1) ^ {2 \alpha}}\right) = \frac {\prod_ {t = 0} ^ {\tau - 1} \left(\ell^ {2} \eta^ {2} + (t + 1) ^ {2 \alpha}\right)}{(\tau !) ^ {2 \alpha}} \leq \frac {\left(\ell^ {2} \eta^ {2} + \tau^ {2 \alpha}\right) ^ {\tau}}{(\tau !) ^ {2 \alpha}} \\ \leq \frac {\left(2 \ell^ {2} \eta^ {2}\right) ^ {\tau}}{(\tau !) ^ {2 \alpha}} \leq \frac {(2 \ell^ {2} \eta^ {2}) ^ {\tau}}{\left[ \sqrt {2 \pi \tau} \left(\frac {\tau}{e}\right) ^ {\tau} \exp \left(\frac {1}{1 2 \tau + 1}\right) \right] ^ {2 \alpha}} \\ \leq \frac {1}{(2 \pi \tau) ^ {\alpha}} \left(\frac {2 \ell^ {2} \eta^ {2} e ^ {2 \alpha}}{\tau^ {2 \alpha}}\right) ^ {\tau} \leq \frac {1}{(2 \pi \tau) ^ {\alpha}} \left(4 e ^ {2 \alpha}\right) ^ {\tau}, \\ \end{array}
$$
where in the third inequality we use Stirling's approximation. Therefore,
$$
\sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {1}{(2 \pi \tau) ^ {\alpha}} \left(4 e ^ {2 \alpha}\right) ^ {\tau} \left[ 1 + \ell \left(\sum_ {t = 0} ^ {\tau - 1} \eta_ {t}\right) \right] \left(\Delta + \sum_ {t = 0} ^ {T - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2}\right).
$$
Plugging in  $\eta_t = \eta /(t + 1)^\alpha$ , when  $\alpha = 1/2$
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {\sqrt {2 T}}{\eta \sqrt {\pi \tau}} (4 e) ^ {\tau} [ 1 + \ell \eta (1 + 2 \sqrt {\tau}) ] (\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2} (1 + \log T));
$$
when  $1 / 2 < \alpha < 1$
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 T ^ {\alpha}}{\eta (2 \pi \tau) ^ {\alpha}} (4 e ^ {2 \alpha}) ^ {\tau} \left[ 1 + \ell \eta \left(1 + \frac {\tau^ {1 - \alpha}}{1 - \alpha}\right) \right] \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2 (1 - 2 ^ {1 - 2 \alpha})}\right).
$$
when  $0 < \alpha < 1/2$ ,
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 T ^ {\alpha}}{\eta (2 \pi \tau) ^ {\alpha}} (4 e ^ {2 \alpha}) ^ {\tau} \left[ 1 + \ell \eta \left(1 + \frac {\tau^ {1 - \alpha}}{1 - \alpha}\right) \right] \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2} T ^ {1 - 2 \alpha}}{2 (1 - 2 \alpha)}\right);
$$
If  $\tau = 0$ , from (5),
$$
\sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \Delta + \sum_ {t = 0} ^ {T - 1} \frac {\ell \eta_ {t} ^ {2}}{2} \sigma^ {2},
$$
Plugging in  $\eta_t$ , when  $\alpha = 1/2$ ,
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 \sqrt {T}}{\eta} \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2} (1 + \log T)\right);
$$
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 T ^ {\alpha}}{\eta} \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2}}{2 (1 - 2 ^ {1 - 2 \alpha})}\right).
$$
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 T ^ {\alpha}}{\eta} \left(\Delta + \frac {\ell \sigma^ {2} \eta^ {2} T ^ {1 - 2 \alpha}}{2 (1 - 2 \alpha)}\right).
$$
Remark 2. When we run SGD with stepsize  $\eta_t = \eta / (t + 1)^\alpha$ , where  $1/2 < \alpha < 1$ , Theorem 6 implies a complexity of  $\mathcal{O}\left((4e^{2\alpha})^{\frac{(\eta\ell)^{1/\alpha}}{1-\alpha}} (\eta\ell)^{\frac{1}{\alpha(1-\alpha)}} \cdot \epsilon^{\frac{-2}{1-\alpha}}\right)$  in the large initial stepsize regime  $\eta > 1/\ell$ . Compared with the case  $\alpha = 1/2$ , when  $\alpha$  is larger, the convergence rate in  $T$  is slower, but it also comes with a smaller exponent, i.e.,  $(\eta\ell)^{1/\alpha}$ . This is because  $\alpha = 1/2$  leads to the best convergence rate in  $T$  [17], while the faster decaying stepsize  $\alpha > 1/2$  will reach the desirable stepsize  $1/\ell$  earlier so that it accumulates less gradient norms before  $\tau$ . For  $0 < \alpha < 1/2$ , however, it comes with both a larger exponent and a slower convergence rate.
Proposition 4 (with bounded gradient). Under Assumption 1, 2 and additionally assuming that the gradient norm is upper bounded by  $G$ , i.e.,  $\| \nabla f(x) \| \leq G$  for all  $x \in \mathbb{R}^d$ , if we run SGD with stepsize  $\eta_t = \eta / \sqrt{t + 1}$  with  $\eta > 0$ , then
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq \frac {1}{\sqrt {T}} \left(\frac {\Delta}{\eta} + \frac {\ell \eta (G ^ {2} + \sigma^ {2})}{2} \log T\right).
$$
Proof. By the smoothness of  $f(\cdot)$ , we have
$$
f (x _ {t + 1}) \leq f (x _ {t}) - \eta_ {t} \langle \nabla F (x _ {t}; \xi_ {t}), \nabla f (x _ {t}) \rangle + \frac {\ell \eta_ {t} ^ {2}}{2} \| \nabla F (x _ {t}; \xi_ {t}) \| ^ {2}.
$$
Taking expectation and summing from  $t = 0$  to  $T - 1$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq f (x _ {0}) - f (x _ {T}) + \frac {\ell}{2} \sum_ {t = 0} ^ {T - 1} \eta_ {t} ^ {2} \mathbb {E} \left[ \| \nabla F (x _ {t}; \xi_ {t}) \| ^ {2} \right] \\ \leq \Delta + \frac {\ell}{2} \sum_ {t = 0} ^ {T - 1} \eta_ {t} ^ {2} \mathbb {E} \left[ \| \nabla F (x _ {t}; \xi_ {t}) \| ^ {2} \right] \\ \leq \Delta + \frac {\ell}{2} \sum_ {t = 0} ^ {T - 1} \eta_ {t} ^ {2} \left(\| \nabla f (x _ {t}) \| ^ {2} + \mathbb {E} \left[ \| \nabla F (x _ {t}; \xi_ {t}) - f (x _ {t}) \| ^ {2} \right]\right) \\ \leq \Delta + \frac {\ell}{2} \sum_ {t = 0} ^ {T - 1} \eta_ {t} ^ {2} (G ^ {2} + \sigma^ {2}). \\ \end{array}
$$
Let  $\eta_t = \eta / \sqrt{t + 1}$ ,
$$
\begin{array}{l} \frac {\eta}{\sqrt {T}} \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \frac {\eta}{\sqrt {t + 1}} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq \Delta + \frac {\ell (G ^ {2} + \sigma^ {2})}{2} \sum_ {t = 0} ^ {T - 1} \frac {\eta^ {2}}{t + 1}, \\ \leq \Delta + \frac {\ell \eta^ {2} (G ^ {2} + \sigma^ {2})}{2} \log T. \\ \end{array}
$$
B.2 Lower Bound for SGD
Proof for Theorem 2
Proof. We construct the hard instance with 4 segments of quadratic functions. The function is symmetric about  $x = 0$ , and we will define it on  $x \leq 0$  as below. We illustrate it in Figure 2.
Segment 1. We define  $f(x) = \frac{\ell}{2} x^2$ . We pick  $x_0$  such that  $f(x_0) - f^* = \Delta$ , i.e.,  $x_0 = \sqrt{\frac{2\Delta}{\ell}}$ . We define  $t_0$  to be the first iteration that  $\eta_{t_0} = \frac{\eta}{\sqrt{t_0 + 1}} \geq \frac{4}{\ell}$ , i.e.,  $t_0 = \left\lfloor \frac{\eta^2\ell^2}{16} - 1 \right\rfloor$ . With the update rule  $x_{t+1} = x_t - \frac{\eta\ell}{\sqrt{t+1}} x_t = \left(1 - \frac{\eta\ell}{\sqrt{t+1}}\right) x_t$ , we have for  $t \leq t_0$
$$
\begin{array}{l} | x _ {t} | ^ {2} = \left[ \prod_ {k = 1} ^ {t} \left(\frac {\eta \ell}{\sqrt {k}} - 1\right) \right] ^ {2} | x _ {0} | ^ {2} \geq \prod_ {k = 1} ^ {t} \frac {\eta^ {2} \ell^ {2}}{2 k} | x _ {0} | ^ {2} \\ = \frac {(\eta^ {2} \ell^ {2} / 2) ^ {t}}{(t) !} | x _ {0} | ^ {2} > \frac {(\eta^ {2} \ell^ {2} / 2) ^ {t}}{\sqrt {2 \pi t} (t / e) ^ {t} e ^ {1 / 1 2 t}} | x _ {0} | ^ {2} \geq \frac {1}{3 \sqrt {t}} (8 e) ^ {t} | x _ {0} | ^ {2}, \\ \end{array}
$$
where in the inequality we use  $\left(\frac{\eta\ell}{\sqrt{k}} - 1\right)^2 \geq \frac{\eta^2\ell^2}{2k}$  with  $k \leq t_0$ , in the second inequality we use Stirling's approximation, and in the last inequality we use  $t \leq \eta^2\ell^2 / 16$ . We note that
$$
| x _ {t _ {0}} | ^ {2} \geq \frac {1}{3 \sqrt {t _ {0}}} (8 e) ^ {t _ {0}} | x _ {0} | ^ {2} \geq \frac {4}{3 \eta \ell} (8 e) ^ {\eta^ {2} \ell^ {2} / 1 6 - 2} | x _ {0} | ^ {2}.
$$
Without loss of generality, we assume  $x_{t_0} > 0$ . Segment 1 is defined on the domain  $\{x:|x|\leq x_{t_0}\}$
Segment 2. This segment is the mirror of Segment 1. On domain  $\{x : x_{t_0} \leq x \leq 2x_{t_0}\}$ , we define  $f(x) = -\frac{\ell}{2}(x - 2x_{t_0})^2 + \ell x_{t_0}^2$ .
Segment 3. We note that
$$
x _ {t _ {0} + 1} = x _ {t _ {0}} - \eta_ {t _ {0}} \ell x _ {t _ {0}} = \left(1 - \frac {\eta \ell}{\sqrt {t _ {0} + 1}}\right) x _ {t _ {0}} \leq - 3 x _ {t _ {0}},
$$
where the inequality is from the definition of  $t_0$ , and
$$
\tilde {\Delta} \triangleq \frac {\ell x _ {t _ {0}} ^ {2}}{2} \geq \frac {2}{3 \eta} (8 e) ^ {\eta^ {2} \ell^ {2} / 1 6 - 2} | x _ {0} | ^ {2} = \frac {4}{3 \eta \ell} (8 e) ^ {\eta^ {2} \ell^ {2} / 1 6 - 2} \Delta .
$$
We construct a quadratic function such that: it passes  $(-2x_{t_0},\ell x_{t_0}^2)$  with gradient 0; the gradient at  $x = x_{t_0 + 1}$  is  $\frac{\sqrt{\tilde{\Delta}}}{2\sqrt{\max\{1 / \ell,\sum_{t = t_0 + 1}^{T - 1}\eta_t\}}}$ . This quadratic function is uniquely defined to be
$$
f (x) = - \frac {\sqrt {\widetilde {\Delta}} (x + 2 x _ {t _ {0}}) ^ {2}}{4 (- 2 x _ {t _ {0}} - x _ {t _ {0} + 1}) \sqrt {\max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}} + \ell x _ {t _ {0}} ^ {2}.
$$
It can be verified that this function is  $\ell$ -smooth: as  $x_{t_0 + 1} \leq -3x_{t_0}$ ,
$$
\begin{array}{l} \frac {\sqrt {\frac {1}{2} \ell x _ {t _ {0}} ^ {2}}}{2 (- 2 x _ {t _ {0}} - x _ {t _ {0} + 1})} \leq \sqrt {\ell} \Longleftrightarrow \frac {\sqrt {\tilde {\Delta}}}{2 (- 2 x _ {t _ {0}} - x _ {t _ {0} + 1}) \sqrt {\ell}} \leq \ell \\ \Longrightarrow \frac {\sqrt {\widetilde {\Delta}}}{2 (- 2 x _ {t _ {0}} - x _ {t _ {0} + 1}) \sqrt {\max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}} \leq \ell . \\ \end{array}
$$
The function is defined on the domain  $\{x : x_{t_0 + 1} \leq x \leq -2x_{t_0}\}$ .
Segment 4. For convenience, we define  $w = f(x_{t_0 + 1})$ . We can verify that  $w \geq \tilde{\Delta}$ : as  $\frac{1}{\sqrt{t_0}} < \frac{4}{\eta \ell}$ ,
$$
\begin{array}{l} - 2 x _ {t _ {0}} - \left(1 - \frac {\eta \ell}{\sqrt {t _ {0} + 1}}\right) x _ {t _ {0}} \leq 4 \sqrt {\frac {x _ {t _ {0}}}{2}} \Longleftrightarrow \frac {- 2 x _ {t _ {0}} - x _ {t _ {0} + 1}}{4} \leq \sqrt {\frac {x _ {t _ {0}}}{2}} \\ \Longrightarrow \frac {- 2 x _ {t _ {0}} - x _ {t _ {0} + 1}}{4 \sqrt {\max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}} \leq \sqrt {\frac {1}{2} \ell x _ {t _ {0}} ^ {2}} \Longleftrightarrow \frac {\sqrt {\tilde {\Delta}} (x _ {t _ {0} + 1} + 2 x _ {t _ {0}}) ^ {2}}{4 (- 2 x _ {t _ {0}} - x _ {t _ {0} + 1}) \sqrt {\max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}} \leq \tilde {\Delta}. \\ \end{array}
$$
So we conclude  $w \geq \tilde{\Delta}$ . Now we construct a quadratic function similar to that in Proposition 1 of [17]: it passes  $(x_{t_0 + 1},w)$  with gradient  $\frac{\sqrt{\tilde{\Delta}}}{2\sqrt{\max\{1 / \ell,\sum_{t = t_0 + 1}^{T - 1}\eta_t\}}}$ ; the minimum is at  $x = x_{t_0 + 1} - \sqrt{\tilde{\Delta}\max\{1 / \ell,\sum_{t = t_0 + 1}^{T - 1}\eta_t\}}$ . This quadratic function is defined to be
$$
f (x) = \frac {\left(x - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}\right) ^ {2}}{4 \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}} + w - \frac {\tilde {\Delta}}{4}
$$
on the domain  $\{x : x \leq x_{t_0 + 1}\}$ . It is obvious that  $f(x) \geq 0$  and is  $\ell$ -smooth. Following the same reasoning of Proposition 1 in [17], also presented as Lemma 1 in the appendix for completeness, we can conclude for all  $t : t_0 + 1 \leq t \leq T$ ,
$$
| \nabla f (x _ {t}) | \geq \frac {\sqrt {\tilde {\Delta}}}{4 \sqrt {\max  \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}} \geq \frac {1}{4} \sqrt {\tilde {\Delta}} \min  \left\{\sqrt {\ell}, (2 \eta) ^ {- 1 / 2} T ^ {- 1 / 4} \right\},
$$
where in the second inequality we use  $\sum_{t = t_0 + 1}^{T - 1}\eta_t = \sum_{t = t_0 + 1}^{T - 1}\frac{\eta}{\sqrt{t + 1}}\leq 2\eta T^{1 / 2}$ .
The following lemma is used in the proof of Theorem 2. It is a straightforward modification of Proposition 1 in [17]. We present it here for completeness.
Lemma 1. Under the same setting and notations as the proof of Theorem 2, if we run gradient descent with stepsize  $\{\eta_t\}_{t = t_0 + 1}^{T - 1}$  starting from point  $x_{t_0 + 1}$  on function
$$
f (x) = \frac {\left(x - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}\right) ^ {2}}{4 \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}} + w - \frac {\tilde {\Delta}}{4},
$$
then for all  $t:t_0 + 1\leq t\leq T$
$$
| \nabla f (x _ {t}) | \geq \frac {\sqrt {\tilde {\Delta}}}{4 \sqrt {\max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}}.
$$
Proof. From the update of gradient descent, we have
$$
x _ {t + 1} = x _ {t} - \eta_ {t} \cdot \frac {x _ {t} - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}}{2 \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}},
$$
$$
\begin{array}{l} x _ {t + 1} - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max  \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}} \\ = \left(1 - \frac {\eta_ {t}}{2 \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}\right) \left(x _ {t} - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}\right). \\ \end{array}
$$
Recursing this, for  $j \leq T$
$$
\begin{array}{l} x _ {j} - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max  \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}} \\ = \prod_ {k = t _ {0} + 1} ^ {j - 1} \left(1 - \frac {\eta_ {k}}{2 \max  \left\{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \right\}}\right) \left(x _ {t _ {0} + 1} - x _ {t _ {0} + 1} + \sqrt {\tilde {\Delta} \max  \left\{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \right\}}\right) \\ \geq \exp \left(\log \frac {1}{2} \cdot \sum_ {k = t _ {0} + 1} ^ {j - 1} \frac {\eta_ {k}}{2 \max  \left\{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \right\}}\right) \sqrt {\tilde {\Delta} \max  \left\{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \right\}} \\ \geq \frac {1}{2} \sqrt {\tilde {\Delta} \max  \{1 / \ell , \sum_ {t = t _ {0} + 1} ^ {T - 1} \eta_ {t} \}}, \\ \end{array}
$$
where in the second inequality, we use  $1 - z / 2 \geq \exp \left(\log \frac{1}{2} \cdot z\right)$  for  $0 \leq z \leq 1$ . This directly implies what we want to prove by computing  $\nabla f(x_j)$ .
C Proofs for NSGD Family in Section 4
Algorithm 3 Normalized Stochastic Gradient Descent (NSGD)
1: Input: initial point  $x_0$
2: for  $t = 0,1,2,\ldots$  do
3: sample  $\xi_{t}$  and set learning rate  $\gamma_{t}$
4:  $x_{t + 1} = x_t - \frac{\gamma_t}{\|g(x_t;\xi_t)\|} g(x_t;\xi_t)$
5: end for
Proof for Theorem 3
Proof. Let us pick  $f(x) = \frac{L}{2} x^2$  with  $\frac{\epsilon^2}{2\Delta} < L \leq \ell$  and  $L < \frac{\sigma - \epsilon}{\gamma_{\max}}$ . Then we pick  $x_0$  such that  $\frac{\epsilon}{L} < x_0 < \sqrt{\frac{2\Delta}{L}}$ , which implies that  $\| \nabla f(x_0) \| > \epsilon$  and  $f(x_0) - \min_x f \leq \Delta$ . Now we define  $D = \{x : -w \leq x \leq w\}$  with  $\frac{\epsilon}{L} + \gamma_{\max} < w < \frac{\sigma}{L}$ . For  $x \in D$ , we have  $\| \nabla f(x) \| \leq \sigma$  and we
(b) example 2
Figure 3: The expected update of NSGD can vanish (example 1) or be in the opposite direction (example 2) of the true gradient. The solid black arrow represents the true gradient and the dashed arrows are the possible stochastic gradients (with equal possibilities). The solid blue arrow is the expected direction of NSGD update.
construct the noisy gradients: with  $\delta > 1$
$$
g (x; \xi) = (1 + \delta) \nabla f (x) \text {w . p .} \frac {1}{2}, \text {a n d} g (x; \xi) = (1 - \delta) \nabla f (x) \text {w . p .} \frac {1}{2}.
$$
It is obvious that  $\nabla f(x) = \mathbb{E}[g(x;\xi)]$  and the variance at this point  $\mathbb{E}\| \nabla f(x) - g(x;\xi)\| ^2 = \delta^2\| \nabla f(x)\| ^2\leq \sigma^2$  with  $\delta$  sufficiently close to 1. With the update rule, we note that  $x_{t + 1} = x_t - \gamma_t$  w.p.  $1 / 2$  and  $x_{t + 1} = x_t + \gamma_t$  w.p.  $1 / 2$ , and therefore
$$
\mathbb {E} _ {\xi_ {t}} \left[ \| \nabla f (x _ {t + 1}) | x _ {t} \in D \| \right] = \frac {1}{2} \left[ L \| x _ {t} - \gamma_ {t} \| + L \| x _ {t} + \gamma_ {t} \| \right] \geq L \| x _ {t} \| = \| \nabla f (x _ {t}) \|.
$$