SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For  $x \notin D$ , we have  $\| x \| > \epsilon / L + \gamma_{\max}$ , and we assume there is no noise in the gradients. Therefore, if  $x_t \notin D$ , we know that after one step of update  $\| x_{t+1} \| > \epsilon / L$ , which implies  $\|\nabla f(x_{t+1})\| > \epsilon$ . Combining two cases that  $x_t \in D$  and  $x_t \notin D$ , we know that  $\mathbb{E} \| \nabla f(x_t) \| > \epsilon$  for all  $t$ .
Proof. Denote  $e_t = g(x_t; \xi_t) - \nabla f(x_t)$ . By Lemma 2 in [14],
$$
f (x _ {t + 1}) - f (x _ {t}) \leq - \frac {\gamma_ {t}}{3} \| \nabla f (x _ {t}) \| + \frac {8 \gamma_ {t}}{3} \| e _ {t} \| + \frac {\ell \gamma_ {t} ^ {2}}{2}.
$$
Telescoping from  $t = 0$  to  $T - 1$ ,
$$
\frac {\gamma}{3 T ^ {1 / 2}} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| \leq \frac {1}{3} \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \| \nabla f (x _ {t}) \| \leq \Delta + \frac {8}{3} \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \| e _ {t} \| + \sum_ {t = 0} ^ {T - 1} \frac {\ell \gamma_ {t} ^ {2}}{2},
$$
Taking expectation, rearranging and using  $\mathbb{E}\left[\| e_t\|\right] \leq \left(\mathbb{E}\left[\| e_t\|^2\right]\right)^{1/2} \leq \sigma$ , we derive
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla f (x _ {t}) \| ] \leq 3 T ^ {1 / 2} \left[ \frac {\Delta}{\gamma} + \frac {8 \sigma}{3} \sum_ {t = 0} ^ {T - 1} \frac {1}{(t + 1) ^ {1 / 2}} + \frac {\ell \gamma}{2} \sum_ {t = 0} ^ {T - 1} \frac {1}{t + 1} \right] \\ \leq 3 T ^ {1 / 2} \left[ \frac {\Delta}{\gamma} + 8 \sigma T ^ {1 / 2} + \ell \gamma \log (T) \right]. \\ \end{array}
$$
Proof for Proposition 2
Proof. We define  $\hat{e}_t = g_t - \nabla f(x_t)$ . By Lemma 2 in [14], for any  $\gamma_t > 0$
$$
f \left(x _ {t + 1}\right) - f \left(x _ {t}\right) \leq - \frac {\gamma_ {t}}{3} \| \nabla f \left(x _ {t}\right) \| + \frac {8 \gamma_ {t}}{3} \| \hat {e} _ {t} \| + \frac {\ell \gamma_ {t} ^ {2}}{2}. \tag {7}
$$
$$
\frac {\gamma}{3 T ^ {3 / 4}} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| \leq \frac {1}{3} \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \| \nabla f (x _ {t}) \| \leq \Delta + \frac {8}{3} \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \| \hat {e} _ {t} \| + \sum_ {t = 0} ^ {T - 1} \frac {\ell \gamma_ {t} ^ {2}}{2},
$$
By taking expectation on both sides, rearranging and controlling the variance term using Lemma 2, we derive
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla f (x _ {t}) \| ] \leq 3 T ^ {3 / 4} \left[ \frac {\Delta}{\gamma} + \frac {8}{3 \gamma} \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \mathbb {E} [ \| \hat {e} _ {t} \| ] + \frac {\ell \gamma}{2} \sum_ {t = 0} ^ {T - 1} (t + 1) ^ {- 3 / 2} \right] \\ \leq 3 T ^ {3 / 4} \left[ \frac {\Delta}{\gamma} + \frac {8}{3} \left(C _ {1} \sigma + C _ {2} \ell \gamma\right) \log (T) + \frac {2 \ell \gamma}{T ^ {1 / 2}} \right] \\ \leq C T ^ {3 / 4} \left[ \frac {\Delta}{\gamma} + (\sigma + \ell \gamma) \log (T) \right]. \\ \end{array}
$$
Lemma 2. Under the setting of Theorem 2, there exist numerical constants  $C_1, C_2 > 0$  such that for all  $t \geq 1$ ,
$$
\mathbb {E} \left[ \| \hat {e} _ {t} \| \right] \leq C _ {1} \sigma \alpha_ {t} ^ {1 / 2} + C _ {2} \ell \gamma_ {t} \alpha_ {t} ^ {- 1},
$$
$$
\sum_ {t = 0} ^ {T - 1} \gamma_ {t} \mathbb {E} \left[ \| \hat {e} _ {t} \| \right] \leq \left(C _ {1} \sigma \gamma + C _ {2} \ell \gamma^ {2}\right) \log (T),
$$
where  $\hat{e}_t = g_t - \nabla f(x_t)$
Proof. Define  $e_t = g(x_t; \xi_t) - \nabla f(x_t)$ ,  $S_t = \nabla f(x_t) - \nabla f(x_{t+1})$ . Then
$$
\begin{array}{l} \hat {e} _ {t + 1} = g _ {t + 1} - \nabla f (x _ {t + 1}) \\ = (1 - \alpha_ {t}) g _ {t} + \alpha_ {t} g \left(x _ {t + 1}; \xi_ {t + 1}\right) - \nabla f \left(x _ {t + 1}\right) \\ = (1 - \alpha_ {t}) \hat {e} _ {t} + \alpha_ {t} \epsilon_ {t + 1} + (1 - \alpha_ {t}) S _ {t}. \\ \end{array}
$$
Unrolling the recursion from  $t = T - 1$  to  $t = 0$ , we have
$$
\hat {e} _ {T} = \left(\prod_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right)\right) \hat {e} _ {0} + \sum_ {t = 0} ^ {T - 1} \alpha_ {t} e _ {t + 1} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right) + \sum_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right) S _ {t} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right). (8)
$$
Define the  $\sigma$ -field  $\mathcal{F}_t \coloneqq \sigma(\{x_0, \xi_0, \ldots, \xi_{t-1}\})$ . Notice that for any  $t_2 > t_1 \geq 0$  we have
$$
\mathbb {E} \left[ \langle e _ {t _ {1}}, e _ {t _ {2}} \rangle \right] = \mathbb {E} \left[ \mathbb {E} \left[ \langle e _ {t _ {1}}, e _ {t _ {2}} \rangle | \mathcal {F} _ {t _ {2}} \right] \right] = \mathbb {E} \left[ \langle e _ {t _ {1}}, \mathbb {E} \left[ e _ {t _ {2}} | \mathcal {F} _ {t _ {2}} \right] \rangle \right] = 0. \tag {9}
$$
Then taking norm, applying expectation on both sides of (8) and using  $\mathbb{E}\left[\| \hat{e}_0\|\right] \leq \sigma$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \hat {e} _ {T} \| \right] \leq \left(\prod_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right)\right) \sigma + \mathbb {E} \left[ \left\| \sum_ {t = 0} ^ {T - 1} \alpha_ {t} e _ {t + 1} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right) \right\| \right] \\ + \mathbb {E} \left[ \left\| \sum_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right) S _ {t} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right) \right\| \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \left(\prod_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right)\right) \sigma + \left(\mathbb {E} \left[ \left\| \sum_ {t = 0} ^ {T - 1} \alpha_ {t} e _ {t + 1} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right) \right\| ^ {2} \right]\right) ^ {1 / 2} \\ + \sum_ {t = 0} ^ {T - 1} (1 - \alpha_ {t}) \mathbb {E} [ \| S _ {t} \| ] \prod_ {\tau = t + 1} ^ {T - 1} (1 - \alpha_ {\tau}) \\ \leq \left(\prod_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right)\right) \sigma + \left(\sum_ {t = 0} ^ {T - 1} \alpha_ {t} ^ {2} \mathbb {E} \left[ \| e _ {t + 1} \| ^ {2} \right] \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right) ^ {2}\right) ^ {1 / 2} \\ + \ell \sum_ {t = 0} ^ {T - 1} (1 - \alpha_ {t}) \gamma_ {t} \prod_ {\tau = t + 1} ^ {T - 1} (1 - \alpha_ {\tau}) \\ \leq \left(\prod_ {t = 0} ^ {T - 1} \left(1 - \alpha_ {t}\right)\right) \sigma + \left(\sum_ {t = 0} ^ {T - 1} \alpha_ {t} ^ {2} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right)\right) ^ {1 / 2} \sigma + \left(\sum_ {t = 0} ^ {T - 1} \gamma_ {t} \prod_ {\tau = t + 1} ^ {T - 1} \left(1 - \alpha_ {\tau}\right)\right) \ell , \\ \end{array}
$$
where the first inequality holds by Jensen's inequality applied to  $x \mapsto x^2$ , the second inequality follows by (9) and the bound  $\| S_t \| \leq \ell \| x_{t+1} - x_t \| = \ell \gamma_t$ . The last step is due to bounded variance  $\mathbb{E}\left[\|\hat{\epsilon}_0\|\right] \leq \sigma$  and  $\alpha_t \leq 1$ .
By the choice of momentum sequence, we have  $\alpha_0 = 1$  and the first term is zero. By Lemma 3, there exist numerical constants  $C_1, C_2 > 0$  such that
$$
\left(\sum_ {t = 0} ^ {T - 1} \alpha_ {t} ^ {2} \prod_ {\tau = t + 1} ^ {T - 1} (1 - \alpha_ {\tau})\right) ^ {1 / 2} \leq C _ {1} \alpha_ {T} ^ {1 / 2}, \quad \left(\sum_ {t = 0} ^ {T - 1} \gamma_ {t} \prod_ {\tau = t + 1} ^ {T - 1} (1 - \alpha_ {\tau})\right) \leq C _ {2} \gamma_ {T} \alpha_ {T} ^ {- 1}.
$$
Therefore, for all  $T \geq 1$ , we have
$$
\mathbb {E} \left[ \| \hat {e} _ {T} \| \right] \leq C _ {1} \sigma \alpha_ {T} ^ {1 / 2} + C _ {2} \ell \gamma_ {T} \alpha_ {T} ^ {- 1}.
$$
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \mathbb {E} \left[ \| \hat {e} _ {t} \| \right] \leq C _ {1} \sigma \sum_ {t = 0} ^ {T - 1} \gamma_ {t} \alpha_ {t} ^ {1 / 2} + C _ {2} \ell \sum_ {t = 0} ^ {T - 1} \gamma_ {t} ^ {2} \alpha_ {t} ^ {- 1} \\ \leq C _ {1} \sigma \gamma \sum_ {t = 0} ^ {T - 1} (t + 1) ^ {- 3 / 4} (t + 1) ^ {- 1 / 4} + C _ {2} \ell \gamma^ {2} \sum_ {t = 0} ^ {T - 1} (t + 1) ^ {- 3 / 2} (t + 1) ^ {1 / 2} \\ \leq \left(C _ {1} \sigma \gamma + C _ {2} \ell \gamma^ {2}\right) \log (T). \\ \end{array}
$$
Lemma 3 (Lemma 15 in [19]). Let  $q \in [0,1)$ ,  $p \geq 0$ ,  $\gamma_0 > 0$  and let  $\eta_t = \left(\frac{2}{t + 2}\right)^q$ ,  $\gamma_t = \gamma_0\left(\frac{1}{t + 1}\right)^p$  for every integer  $t$ . Then for any integers  $t$  and  $T \geq 1$ , it holds
$$
\sum_ {t = 0} ^ {T - 1} \gamma_ {t} \prod_ {\tau = t + 1} ^ {T - 1} (1 - \eta_ {\tau}) \leq C \gamma_ {t} \eta_ {T} ^ {- 1},
$$
where  $C \coloneqq 2^{p - q}(1 - q)^{-1}t_0\exp \left(2^q (1 - q)t_0^{1 - q}\right) + 2^{2p + 1 - q}(1 - q)^{-2}$  and  $t_0 \coloneqq$
$$
\max \left\{\left(\frac {p}{(1 - q) 2 ^ {q}}\right) ^ {\frac {1}{1 - q}}, 2 \left(\frac {p - q}{(1 - q) ^ {2}}\right) \right\} ^ {\frac {1}{1 - q}}.
$$
D Proofs for Scalar AMSGrad and AdaGrad in Section 4
The following is an extended version of Theorem 5 including  $\gamma_{t} = \frac{\gamma}{(t + 1)^{\alpha}}$  with  $0 < \alpha < 1$ .
Theorem 7. Under Assumption 1, if we run AMSGrad-norm with  $\gamma_t = \frac{\gamma}{(t + 1)^\alpha}$ ,  $v_0 > 0$  and  $\beta_1 = \beta_2 = 0$  in the deterministic setting, then for any  $\gamma > 0$  and  $0 < \alpha < 1$ , if  $v_0 < \gamma \ell$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 \Delta}{\gamma T ^ {1 - \alpha}} \max  \{v _ {0}, \sqrt {2 \ell \Delta} \},
$$
if  $v_{0}\geq \gamma \ell$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {\left(\frac {\ell \gamma}{v _ {0}}\right) ^ {\frac {1}{\alpha}} \gamma^ {2} \ell^ {2}}{T} + \frac {2 (M + \Delta)}{\gamma T ^ {1 - \alpha}} \max \{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \},
$$
$$
M = \left\{ \begin{array}{l} \ell \gamma^ {2} \left(1 + \log \left(\frac {\ell \gamma}{v _ {0}}\right)\right), w h e n \alpha = 1 / 2, \\ \frac {\ell \gamma^ {2}}{2 (1 - 2 ^ {1 - 2 \alpha})}, w h e n 1 / 2 <   \alpha <   1, \\ \frac {\gamma (\ell \gamma) ^ {\frac {1}{\alpha} - 1}}{2 (1 - 2 \alpha) v _ {0} ^ {\frac {1}{\alpha} - 2}}, w h e n 0 <   \alpha <   1 / 2. \end{array} \right.
$$
Proof. The effective stepsize of AMSGrad-norm contains a maximum over all gradient norms in the denominator. As it is desirable to find a lower bound for the effective stepsize, we begin by bounding the gradient norms.
Let  $\tau$  be the first iteration where the effective stepsize is less or equal to  $1 / \ell$ , i.e.,  $\eta_{\tau - 1} > 1 / \ell$  and  $\eta_{\tau} \leq 1 / \ell$ . First, we assume  $\tau \geq 1$ , i.e.,  $v_0 < \gamma \ell$ . The time stamp  $\tau$  itself is naturally bounded by
$$
\eta_ {\tau - 1} = \frac {\gamma}{\tau^ {\alpha} v _ {\tau}} > \frac {1}{\ell} \quad \Longrightarrow \quad \tau <   \left(\frac {\ell \gamma}{v _ {\tau}}\right) ^ {\frac {1}{\alpha}} \leq \left(\frac {\ell \gamma}{v _ {0}}\right) ^ {\frac {1}{\alpha}}.
$$
$$
\sum_ {t = 0} ^ {\tau - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \tau \gamma^ {2} \ell^ {2} \leq \left(\frac {\ell \gamma}{v _ {0}}\right) ^ {\frac {1}{\alpha}} \gamma^ {2} \ell^ {2}. \tag {10}
$$
By  $\ell$ -smoothness of  $f(\cdot)$ ,
$$
\begin{array}{l} f (x _ {t + 1}) \leq f (x _ {t}) + \langle \nabla f (x _ {t}), x _ {t + 1} - x _ {t} \rangle + \frac {\ell}{2} \| x _ {t + 1} - x _ {t} \| ^ {2} \\ = f \left(x _ {t}\right) - \eta_ {t} \| \nabla f \left(x _ {t}\right) \| ^ {2} + \frac {\ell \eta_ {t} ^ {2}}{2} \| \nabla f \left(x _ {t}\right) \| ^ {2} \tag {11} \\ \leq f (x _ {t}) + \frac {\ell \eta_ {t} ^ {2}}{2} \| \nabla f (x _ {t}) \| ^ {2}. \\ \end{array}
$$
$$
f (x _ {\tau}) - f (x _ {0}) \leq \frac {\ell}{2} \sum_ {t = 0} ^ {\tau - 1} \eta_ {t} ^ {2} \| \nabla f (x _ {t}) \| ^ {2} = \frac {\ell}{2} \sum_ {t = 0} ^ {\tau - 1} \frac {\gamma_ {t} ^ {2}}{v _ {t + 1} ^ {2}} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {\ell}{2} \sum_ {t = 0} ^ {\tau - 1} \gamma_ {t} ^ {2}
$$
$$
\leq \left\{ \begin{array}{l} \frac {\ell \gamma^ {2}}{2} (1 + \log \tau), \text {w h e n} \alpha = 1 / 2, \\ \frac {\ell \gamma^ {2}}{2 \left(1 - 2 ^ {1 - 2 \alpha}\right)}, \text {w h e n} 1 / 2 <   \alpha <   1, \\ \frac {\ell \gamma^ {2} \tau^ {1 - 2 \alpha}}{2 (1 - 2 \alpha)}, \text {w h e n} 0 <   \alpha <   1 / 2. \end{array} \right.
$$
We denote the right hand side as  $M$ . Also from (11) and definition of  $\tau$ , we know that  $f(x_{t}) \leq f(x_{\tau})$  for  $t \geq \tau$  and therefore, for all  $t \geq \tau$ ,
$$
f (x _ {t}) - f ^ {*} = f (x _ {\tau}) - f (x _ {0}) + f (x _ {0}) - f ^ {*} \leq M + \Delta ,
$$
$$
\| \nabla f (x _ {t}) \| ^ {2} \leq 2 \ell (f (x _ {t}) - f ^ {*}) \leq 2 \ell (M + \Delta).
$$
Therefore, we can bound for all  $t \geq 0$ ,
$$
v _ {t} \leq \max  \left\{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \right\}.
$$
For  $t\geq \tau$  ,by (11)
$$
f (x _ {t + 1}) - f (x _ {t}) \leq - \frac {\eta_ {t}}{2} \| \nabla f (x _ {t}) \| ^ {2}.
$$
By telescoping from  $t = \tau$  to  $T - 1$ , we get
$$
\begin{array}{l} 2 \left(f (x _ {\tau}) - f (x _ {T})\right) \geq \sum_ {t = \tau} ^ {T - 1} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} \\ = \sum_ {t = \tau} ^ {T - 1} \frac {\gamma}{(t + 1) ^ {\alpha} v _ {t + 1}} \| \nabla f (x _ {t}) \| ^ {2} \\ \geq \sum_ {t = \tau} ^ {T - 1} \frac {\gamma}{T ^ {\alpha} v _ {t + 1}} \| \nabla f (x _ {t}) \| ^ {2} \\ \geq \sum_ {t = \tau} ^ {T - 1} \frac {\gamma}{T ^ {\alpha} \max \{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \}} \| \nabla f (x _ {t}) \| ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = \tau} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2}{\gamma} \left(f (x _ {\tau}) - f (x _ {T})\right) T ^ {\alpha} \max  \{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \} \\ \leq \frac {2}{\gamma} \left(f \left(x _ {\tau}\right) - f \left(x ^ {*}\right)\right) T ^ {\alpha} \max  \left\{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \right\} \\ \leq \frac {2 (M + \Delta)}{\gamma} T ^ {\alpha} \max  \{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \}. \\ \end{array}
$$
Combining with (10), we obtain
$$
\sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \left(\frac {\ell \gamma}{v _ {0}}\right) ^ {\frac {1}{\alpha}} \gamma^ {2} \ell^ {2} + \frac {2 (M + \Delta) T ^ {\alpha}}{\gamma} \max  \{\gamma \ell , \sqrt {2 \ell (M + \Delta)} \}.
$$
When  $\tau = 0$ , we have
$$
2 \left(f (x _ {0}) - f (x _ {T})\right) \geq \sum_ {t = 0} ^ {T - 1} \frac {\gamma}{T ^ {\alpha} v _ {t + 1}} \| \nabla f (x _ {t}) \| ^ {2} \geq \sum_ {t = \tau} ^ {T - 1} \frac {\gamma}{T ^ {\alpha} \max \{v _ {0} , \sqrt {2 \ell \Delta} \}} \| \nabla f (x _ {t}) \| ^ {2},
$$
$$
\sum_ {t = 0} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 \Delta T ^ {\alpha}}{\gamma} \max  \{v _ {0}, \sqrt {2 \ell \Delta} \}.
$$
Remark 3. For any  $0 < \alpha < 1$ , if we compare simplified AMSGrad with  $\gamma_t = \frac{\gamma}{(t + 1)^\alpha}$  to SGD with  $\eta_t = \frac{\eta}{(t + 1)^\alpha}$  in the deterministic case (setting  $\sigma = 0$  in Theorem 6), we observe that they achieve the same convergence rate. However, the complexity of simplified AMSGrad only includes polynomial term in  $\gamma$  and  $\ell$ , while that of SGD includes an exponential term in  $(\eta \ell)^{1 / \alpha}$ .
In the following, we will first provide the lower bounds for scalar version of AMSGrad (referred to as AMSGrad-norm) with each  $\alpha \in (0,1)$  and discuss why it may fail with  $\alpha = 0$  when problem parameters are unknown, which means that it can not achieve the optimal complexity  $\mathcal{O}(\epsilon^{-2})$  in the deterministic setting. Second, we show that it also fails to achieve the optimal convergence rate in the stochastic setting when stochastic gradients are unbounded. To make the results more general, we consider the standard scalar AMSGrad with momentum hyper-parameters  $\beta_{1}$  and  $\beta_{2}$ , which is presented in Algorithm 2.
Before proceeding to our results, we present a lemma which is handy for conducting lower bounds for SGD-like algorithms with momentum (see Algorithm 4). As long as an upper bound is known for stepsize  $\eta_t$ , we can derive a lower bound similar to Proposition 1 in [17].
Algorithm 4 General SGD with Momentum
1: Input: initial point  $x_0$ , momentum parameters  $0 \leq \beta_{1} < 1$  and initial moment  $m_0$ .
2: for  $t = 0,1,2,\ldots$  do
3: sample  $\xi_{t}$
4:  $m_{t + 1} = \beta_1 m_t + (1 - \beta_1) g(x_t; \xi_t)$
5: obtain stepsize  $\eta_t > 0$
6:  $x_{t + 1} = x_t - \eta_t m_{t + 1}$
7: end for
Lemma 4. For any  $\ell >0$ ,  $\Delta >0$  and  $T > 1$ , there exists a  $\ell$ -smooth function  $f:\mathbb{R}\to \mathbb{R}$ , and  $x_0$  with  $f(x_0) - \inf_x f(x)\leq \Delta$ , such that if we run Algorithm 4 with deterministic gradients and  $\eta_t\leq \tilde{\eta}_t$  for  $t = 0,1,2,\ldots,T - 1$ , then we have
$$
\min  _ {t \in \{0, 1, \dots , T - 1 \}} | \nabla f (x _ {t}) | \geq \sqrt {\frac {\Delta}{1 6 \max  \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}}}.
$$
Proof. We construct a quadratic function similar to Proposition 1 in [17]. The following function is considered:
$$
f (x) = \frac {x ^ {2}}{4 \max  \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}}.
$$
Without loss of generality, we assume the initial moment  $m_0$  is non-positive, and we set the initial point  $x_0$  as
$$
x _ {0} = \sqrt {\Delta \max  \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}}.
$$
Otherwise if the initial moment is set to be positive, then we let  $x_0$  be negative and follow the same reasoning.
Since  $x_0$  is positive, the first gradient direction would be positive, i.e.,  $\nabla f(x_0) > 0$ . Let  $\tau$  be the first iteration such that  $m_{\tau} > 0$ . By the update rule and definition of  $\tau$ , it is obvious that  $x_t \geq x_0$  for  $t \leq \tau - 1$ . If  $T \leq \tau$ , it trivially holds that  $\nabla f(x_t) \geq \nabla f(x_0)$  for all  $0 \leq t \leq T - 1$ . Otherwise,
we have  $m_{\tau} = \beta_{1}m_{\tau -1} + (1 - \beta_{1})\nabla f(x_{\tau -1})\leq (1 - \beta_{1})\nabla f(x_{\tau -1})$ . That is to say, the gradient estimation  $m_{\tau}$  used in the  $\tau$ -th step has the correct direction but its magnitude is no larger than the actual gradient. Starting from the  $\tau$ -th iteration,  $x_{t}$  will monotonically move left towards the solution. Note that since our stepsize is small enough, i.e.,
$$
\eta_ {t} \leq \tilde {\eta} _ {t} <   2 \max  \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\},
$$
the updates will remain positive, i.e.,  $x_{t} > 0$  for  $t \geq \tau$ . By the update rule, we note that  $x_{t + 1} \leq x_{t}$  for  $t \geq \tau$ , and therefore  $\nabla f(x_{t + 1}) < \nabla f(x_{t})$ . We can conclude that for any  $t \geq \tau$ , we have  $m_{t} \leq \nabla f(x_{\tau - 1})$ . Then for  $t \geq \tau - 1$  we have
$$
\begin{array}{l} x _ {t} = x _ {\tau - 1} - \sum_ {k = \tau - 1} ^ {t - 1} \eta_ {t} m _ {t + 1} \\ \geq x _ {\tau - 1} - \sum_ {k = \tau - 1} ^ {t - 1} \tilde {\eta} _ {t} \nabla f (x _ {\tau - 1}) \\ = x _ {\tau - 1} - \sum_ {k = \tau - 1} ^ {t - 1} \frac {\tilde {\eta} _ {t}}{2 \max  \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}} x _ {\tau - 1} \\ \geq \frac {1}{2} x _ {\tau - 1} \\ \geq \frac {1}{2} x _ {0}. \\ \end{array}
$$
Then we conclude by
$$
| \nabla f (x _ {t}) | = \frac {x _ {t}}{2 \max \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}} \geq \frac {x _ {0}}{4 \max \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}} = \sqrt {\frac {\Delta}{1 6 \max \left\{1 / \ell , \sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \right\}}}.
$$
Now we proceed to provide the lower bound for deterministic case.
Theorem 8. For any  $\ell >0$ ,  $\Delta >0$  and  $T > 1$ , there exists a  $\ell$ -smooth function  $f:\mathbb{R}\to \mathbb{R}$  and  $x_0$  with  $f(x_0) - \inf_x f(x)\leq \Delta$ , such that if we run Algorithm 2 with deterministic gradients,  $0 < v_{0}\leq \frac{\ell\gamma}{2}$ , and  $\gamma_{t} = \frac{\gamma}{(t + 1)^{\alpha}}$  with  $\gamma \leq \frac{4\Delta}{v_0}$ , we have (1) if  $0 < \alpha < 1$ , for any  $0\leq \beta_{1} < 1$  and  $0\leq \beta_{2}\leq 1$ , we have
$$
\min _ {t \in \{0, 1, \dots , T - 1 \}} | \nabla f (x _ {t}) | \geq \sqrt {\frac {\Delta}{1 6 \max \{1 / \ell , \frac {\gamma}{(1 - \alpha) v _ {0}} T ^ {1 - \alpha} \}}},
$$
and (2) if  $\alpha = 0$ , for  $\beta_{1} = 0$  and any  $0 \leq \beta_{2} \leq 1$ , we have
$$
\min_{t\in \{0,1,\ldots ,T - 1\}}| \nabla f(x_{t})|\geq v_{0}.
$$
Remark 4. From the theorem, we can conclude that the optimal convergence rate  $\frac{1}{\sqrt{T}}$  for  $\| \nabla f(x_{t})\|$  is infeasible for AMSGrad with polynomially decreasing stepsize. When  $\alpha = 0$ , a similar result can be obtained for the case  $\beta_{1}\geq 0$ ,  $\beta_{2} = 0$  and small enough  $v_{0}$ .
Proof. For  $\alpha > 0$ , we have
$$
\eta_ {t} = \frac {\gamma}{(t + 1) ^ {\alpha} \sqrt {\hat {v} _ {t + 1} ^ {2}}} \leq \frac {\gamma}{(t + 1) ^ {\alpha} v _ {0}}.
$$
Let  $\tilde{\eta}_t = \frac{\gamma}{(t + 1)^\alpha v_0}$  and then we have
$$
\sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} = \sum_ {t = 0} ^ {T - 1} \frac {\gamma}{(t + 1) ^ {\alpha} v _ {0}} \leq \frac {\gamma}{(1 - \alpha) v _ {0}} T ^ {1 - \alpha}.
$$
Applying Lemma 4 directly gives us the desired result.
For  $\alpha = 0$ , we consider function
$$
f (x) = \frac {v _ {0}}{\gamma} x ^ {2}.
$$
Note that since  $v_0 \leq \frac{\ell \gamma}{2}$ , the function is  $\ell$ -smooth. Let
$$
x _ {0} = \frac {\gamma}{2},
$$
which satisfies the condition that  $f(x_0) \leq \Delta$ . Then after one update
$$
v _ {1} ^ {2} = \beta_ {2} v _ {0} ^ {2} + (1 - \beta_ {2}) \| \nabla f (x _ {0}) \| ^ {2} = v _ {0} ^ {2}
$$
$$
x _ {1} = x _ {0} - \frac {\gamma}{\sqrt {v _ {1} ^ {2}}} \nabla f (x _ {0}) = - \frac {\gamma}{2} = - x _ {0}.
$$
If we continue this calculation, we find that the iterates will oscillate between  $\frac{\gamma}{2}$  and  $-\frac{\gamma}{2}$  forever, which finishes the proof.
Proof for Theorem 4
Proof. We consider a two-dimensional function  $f: \mathbb{R}^2 \to \mathbb{R}^2$ , for  $x = (x^{1}, x^{2})^{\top} \in \mathbb{R}^{2}$
$$
f (x) = F \left(x ^ {1}\right),
$$
where its function value only depends on the first dimension and we will define  $F: \mathbb{R} \to \mathbb{R}$  later. The gradient at  $x$  is  $\nabla f(x) = (\nabla F(x^1), 0)^\top$ . We add the noise only to the second dimension, i.e.,  $g(x; \xi) = (\nabla F(x^1), \xi)$ . For any  $t \geq 0$ , the probability density function of the noise as
$$
p _ {\xi_ {t}} (x) = \left\{ \begin{array}{l l} \frac {1}{s \zeta} \left(\frac {x}{s}\right) ^ {- 1 - \frac {2}{\zeta}} e ^ {- \left(\frac {x}{s}\right) ^ {- \frac {2}{\zeta}}}, & x \geq 0; \\ \frac {1}{s \zeta} \left(\frac {- x}{s}\right) ^ {- 1 - \frac {2}{\zeta}} e ^ {- \left(\frac {- x}{s}\right) ^ {- \frac {2}{\zeta}}}, & x <   0, \end{array} \right.
$$
where  $s = \frac{\sigma}{\sqrt{\Gamma(1 - \frac{\zeta}{2})}}$ . Note that the distribution is symmetric and  $\mathbb{E}[\xi_t] = 0$ . Also, we note that  $|\xi_t|$  follows the Fréchet distribution [15] with cumulative distribution function
$$
\operatorname * {P r} (| \xi_ {t} | \leq x) = e ^ {- \left(\frac {x}{s}\right) ^ {- \frac {2}{\xi}}},
$$
$$
\begin{array}{l} \operatorname {V a r} \left[ \xi_ {t} \right] = \mathbb {E} \left[ | \xi_ {t} | ^ {2} \right] - (\mathbb {E} \left[ \xi_ {t} \right]) ^ {2} \\ = s ^ {2} \Gamma \left(1 - \frac {\zeta}{2}\right) - (\mathbb {E} [ \xi_ {t} ]) ^ {2} \\ \leq s ^ {2} \Gamma \left(1 - \frac {\zeta}{2}\right) \\ \leq \sigma^ {2}, \\ \end{array}
$$
where we used the exact second moment for Fréchet distribution.
Next, we will show that  $\tilde{\xi}_t\coloneqq \max_{0\leq k\leq t}\{| \xi_k|\} \geq \Omega \left(\frac{1}{(t + 1)^{\zeta - 1 / 2}}\right)$  with probability  $\frac{1}{2}$ . We know that  $\tilde{\xi}_t$  also follows Fréchet distribution with CDF
$$
\Pr (\tilde {\xi} _ {t} \leq x) = \exp \left(- \left(\frac {x}{s \cdot (t + 1) ^ {\frac {\zeta}{2}}}\right) ^ {- \frac {2}{\zeta}}\right).
$$
Then for constant  $C > 0$
$$
\begin{array}{l} \Pr (\tilde {\xi} _ {t} \leq C \cdot (t + 1) ^ {\zeta - \frac {1}{2}}) = \exp \left(- \left(\frac {C \cdot (t + 1) ^ {\zeta - \frac {1}{2}}}{s \cdot (t + 1) ^ {\frac {\zeta}{2}}}\right) ^ {- \frac {2}{\zeta}}\right) \\ = \exp \left(- \left(\frac {C}{s}\right) ^ {\frac {2}{\zeta}} (t + 1) ^ {\frac {1}{\zeta} - 1}\right) \\ \leq \frac {1}{4 (t + 1) ^ {2}}, \\ \end{array}
$$