SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the last inequality is by selecting  $C = \frac{s\left(e\left(\frac{1}{\zeta} - 1\right)\right)^{\frac{\zeta}{2}}}{\sqrt{2}}$  and using  $\exp \left(-\frac{2^{m + 1}}{em}\cdot t^m\right)\leq \frac{1}{4t^2}$  for any  $t > 0$  and  $0 < m < 1$ . Then using union bound, we have
$$
\Pr \left(\tilde {\xi} _ {t} > C \cdot (t + 1) ^ {\zeta - \frac {1}{2}} \quad \text {f o r} \quad 0 \leq t \leq T - 1\right) \geq 1 - \sum_ {t = 0} ^ {T - 1} \frac {1}{4 (t + 1) ^ {2}} \geq \frac {1}{2}.
$$
Now we have shown that with some probability, the noise is large enough. We can use this property to provide an upper bound  $\tilde{\eta}_t$  for the stepsize as follows
$$
\begin{array}{l} \eta_ {t} = \frac {\gamma}{\sqrt {t + 1} \sqrt {\tilde {v} _ {t + 1}}} \\ = \frac {\gamma}{\sqrt {t + 1} \sqrt {\max  _ {0 \leq k \leq t} \left\{\beta_ {2} v _ {k} + (1 - \beta_ {2}) \| g \left(x _ {k} ; \xi_ {k}\right) \| ^ {2} \right\}}} \\ \leq \frac {\gamma}{\sqrt {t + 1} \sqrt {\max _ {0 \leq k \leq t} \left\{(1 - \beta_ {2}) \| g (x _ {k} ; \xi_ {k}) \| ^ {2} \right\}}} \\ \leq \frac {\gamma}{\sqrt {t + 1} \sqrt {\max _ {0 \leq k \leq t} \left\{\left(1 - \beta_ {2}\right) \| \xi_ {k} \| ^ {2} \right\}}} \\ = \frac {\gamma}{\sqrt {t + 1} \sqrt {(1 - \beta_ {2})} \tilde {\xi} _ {t}} \\ \leq \frac {\gamma}{C (t + 1) ^ {\zeta} \sqrt {(1 - \beta_ {2})}} \triangleq \tilde {\eta} _ {t}, \\ \end{array}
$$
$$
\sum_ {t = 0} ^ {T - 1} \tilde {\eta} _ {t} \leq \frac {\gamma}{(1 - \zeta) C \sqrt {(1 - \beta_ {2})}} \left(T ^ {1 - \zeta} - \zeta\right).
$$
We observe that the update with AMSGrad-norm in function  $f$  corresponds to applying general SGD with momentum (Algorithm 4) to function  $F$  with stepsize  $\eta_t$ . Therefore, we can pick a hard instance  $F$  according to Lemma 4, and by noting that  $\| \nabla f(x) \| = |\nabla F(x^1)|$  we reach our conclusion.
Remark 5. As we see above, the function  $F$  in the proof is constructed by Lemma 4. We note that even assuming the gradients of  $f$  to be bounded, i.e.,  $\| \nabla f(x) \| \leq K$  for all  $x$ , will not prevent the slow convergence in Theorem 4. This is because in the proof of Lemma 4 all iterates stay between  $[0, x_{\tau - 1}]$  (e.g.,  $\tau = 1$  if  $m_0 = 0$ ), so we can construct any Lipschitz function outside of this segment.
Algorithm 5 AdaGrad-norm
1: Input: initial point  $x_0$ ,  $v_0 > 0$  and  $\eta > 0$
2: for  $t = 0,1,2,\ldots$  do
3: sample  $\xi_{t}$
4:  $v_{t + 1}^2 = v_t^2 + \| g(x_k; \xi_k) \|^2$
5:  $x_{t + 1} = x_t - \frac{\eta}{\sqrt{v_{t + 1}^2}} g(x_t;\xi_t)$
6: end for
Proof for Proposition 3
Proof. Define a function  $\widetilde{f}:\mathbb{R}^d\times \mathbb{R}\to \mathbb{R}$  such that  $\widetilde{f} (x,y) = f(x) - \frac{\ell}{2} y^{2}$ . Since the  $\widetilde{f}$  is  $\ell$ -smooth and  $\ell$ -strongly concave about  $y$ , the condition number is defined to be  $\kappa = 1$ . Applying AdaGrad-norm to  $f$  is equivalent to applying NeAda-AdaGrad (Algorithm 3 in [68]) to  $\widetilde{f}$  with  $y_{t}\equiv 0$ . For every  $x$ , we know  $y^{*}(x)\triangleq \arg \max_{y}\widetilde{f} (x,y) = 0$ . Then  $\mathcal{E}\triangleq \sum_{t = 0}^{T - 1}\frac{\ell^2\|y_t - y^*(x_t)\|^2}{2v_0} = 0$ . Plugging in  $\kappa = 1$ ,  $\mathcal{E} = 0$  and batchsize  $M = 1$  to Theorem 3.1 in [68], we reach the conclusion.