SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat {\alpha} _ {t} \| \boldsymbol {m} _ {t} \| ^ {2} = \hat {\alpha} _ {t} \| \sum_ {i = 1} ^ {t} (1 - \beta_ {1, t - i + 1}) \boldsymbol {g} _ {t - i + 1} \prod_ {j = 1} ^ {i - 1} \beta_ {1, t - j + 1} \| ^ {2} \leq \hat {\alpha} _ {t} \| \sum_ {i = 1} ^ {t} \boldsymbol {g} _ {t - i + 1} \beta_ {1} ^ {i - 1} \| ^ {2} \\ = \hat {\alpha} _ {t} \sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {t} g _ {t - i + 1, j} \beta_ {1} ^ {i - 1}\right) ^ {2} \leq \hat {\alpha} _ {t} \sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {t} g _ {t - i + 1, j} ^ {2} \beta_ {1} ^ {i - 1}\right) \left(\sum_ {i = 1} ^ {t} \beta_ {1} ^ {i - 1}\right) \\ <   \frac {\alpha}{\sqrt {t}} \frac {1}{1 - \beta_ {1 , t}} \frac {n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {2}} \leq \frac {n \alpha G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3}} \frac {1}{\sqrt {t}}, \\ \end{array}
$$
where the second inequality follows from Cauchy-Schwartz inequality. Therefore,
$$
\sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \| \boldsymbol {m} _ {t} \| ^ {2} <   \frac {n \alpha G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3}} \sum_ {t = 1} ^ {T} \frac {1}{\sqrt {t}} <   \frac {2 n \alpha G _ {\infty} ^ {2} \sqrt {T}}{(1 - \beta_ {1}) ^ {3}},
$$
where the last inequality follows from
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \frac {1}{\sqrt {t}} = 1 + \int_ {2} ^ {3} \frac {1}{\sqrt {2}} d s + \dots + \int_ {T - 1} ^ {T} \frac {1}{\sqrt {T}} d s \\ <   1 + \int_ {2} ^ {3} \frac {1}{\sqrt {s - 1}} d s + \dots + \int_ {T - 1} ^ {T} \frac {1}{\sqrt {s - 1}} d s \\ = 1 + \int_ {2} ^ {T} \frac {1}{\sqrt {s - 1}} d s = 2 \sqrt {T - 1} - 1 <   2 \sqrt {T}. \\ \end{array}
$$
Now we return to the proof of Theorem 2. Let  $\hat{\alpha}_0\coloneqq \hat{\alpha}_1$ . By Lemma 1, Lemma 2, Lemma 4, Equation (16) and the third inequality of Equation (6), we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(f _ {t} (\boldsymbol {w} _ {t}) - f _ {t} (\boldsymbol {w} ^ {*})\right) \leq \frac {1}{1 - \beta_ {1}} \left[ \frac {1}{2 \hat {\alpha} _ {1}} \| \sqrt {\boldsymbol {v} _ {1}} \left(\boldsymbol {w} _ {1} - \boldsymbol {w} ^ {*}\right) \| ^ {2} + \sum_ {t = 2} ^ {T} \left(\frac {1}{2 \hat {\alpha} _ {t}} \| \sqrt {\boldsymbol {v} _ {t}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}\right) \| ^ {2} - \frac {1}{2 \hat {\alpha} _ {t - 1}} \| \sqrt {\boldsymbol {v} _ {t - 1}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}\right) \| ^ {2}\right) \right. \\ \left. + \sum_ {t = 1} ^ {T} \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} \| \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*} \| ^ {2} + \sum_ {t = 1} ^ {T} \left(\frac {\hat {\alpha} _ {t - 1}}{2} \| \boldsymbol {m} _ {t - 1} \| ^ {2} + \frac {\hat {\alpha} _ {t}}{2} \| \frac {\boldsymbol {m} _ {t}}{\sqrt {\boldsymbol {v} _ {t}}} \| ^ {2}\right) \right] \\ \leq \frac {1}{1 - \beta_ {1}} \left[ \frac {D _ {\infty} ^ {2}}{2 \hat {\alpha} _ {1}} \| \sqrt {\boldsymbol {v} _ {1}} \| ^ {2} + \sum_ {t = 2} ^ {T} D _ {\infty} ^ {2} \left(\frac {\| \sqrt {\boldsymbol {v} _ {t}} \| ^ {2}}{2 \hat {\alpha} _ {t}} - \frac {\| \sqrt {\boldsymbol {v} _ {t - 1}} \| ^ {2}}{2 \hat {\alpha} _ {t - 1}}\right) + \sum_ {t = 1} ^ {T} \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} n D _ {\infty} ^ {2} \right. \\ \left. + \sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \left(\frac {1}{2} + \frac {1}{2 \delta \sqrt {1 - \beta_ {2}}}\right) \| \boldsymbol {m} _ {t} \| ^ {2} \right] \\ = \frac {1}{1 - \beta_ {1}} \left[ D _ {\infty} ^ {2} \frac {\| \sqrt {\boldsymbol {v} _ {T}} \| ^ {2}}{2 \alpha_ {T}} + \sum_ {t = 1} ^ {T} \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} n D _ {\infty} ^ {2} + \sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \left(\frac {1}{2} + \frac {1}{2 \delta \sqrt {1 - \beta_ {2}}}\right) \| \boldsymbol {m} _ {t} \| ^ {2} \right] \\ <   \frac {1}{1 - \beta_ {1}} \left[ \frac {n (2 G _ {\infty} + \delta) D _ {\infty} ^ {2}}{2 \alpha \sqrt {1 - \beta_ {2}} (1 - \beta_ {1}) ^ {2}} \sqrt {T} + \sum_ {t = 1} ^ {T} \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} n D _ {\infty} ^ {2} + \frac {n \alpha G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3}} \left(1 + \frac {1}{\delta \sqrt {1 - \beta_ {2}}}\right) \sqrt {T} \right]. \\ \end{array}
$$
E Proof of Corollary 3
Proof. Since  $\beta_{1,t} = \beta_1 / t$ , we have
$$
\sum_ {t = 1} ^ {T} \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} = \sum_ {t = 1} ^ {T} \frac {(1 - \beta_ {1 , t} ^ {t}) \sqrt {t} \beta_ {1 , t}}{2 \alpha \sqrt {1 - \beta_ {2} ^ {t}}} <   \sum_ {t = 1} ^ {T} \frac {\sqrt {t} \beta_ {1 , t}}{2 \alpha \sqrt {1 - \beta_ {2}}} = \frac {\beta_ {1}}{2 \alpha \sqrt {1 - \beta_ {2}}} \sum_ {t = 1} ^ {T} \frac {1}{\sqrt {t}} <   \frac {\beta_ {1}}{\alpha \sqrt {1 - \beta_ {2}}} \sqrt {T}.
$$