SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Since  $\beta_{1,t}$  is non-increasing with respect to  $t$ , we have  $\beta_{1,t+1}^{t+1} \leq \beta_{1,t+1}^t \leq \beta_{1,t}^t$ . Hence,  $\beta_{1,t}^t$  is non-increasing and  $\frac{1}{1 - \beta_{1,t}^t}$  is non-increasing. Thus, we only need to prove  $\phi(t) = \alpha_t \sqrt{1 - \beta_2^t}$  is decreasing. Since the proof is trivial when  $\beta_2 = 0$ , we only need to consider the case where  $\beta_2 \in (0,1)$ . Taking the derivative of  $\phi(t)$ , we have
$$
\phi^ {\prime} (t) = - \frac {\alpha}{2} t ^ {- \frac {3}{2}} (1 - \beta_ {2} ^ {t}) ^ {\frac {1}{2}} - \frac {\alpha}{2} t ^ {- \frac {1}{2}} (1 - \beta_ {2} ^ {t}) ^ {- \frac {1}{2}} \beta_ {2} ^ {t} \log_ {e} \beta_ {2} = - \frac {\alpha}{2} t ^ {- \frac {3}{2}} (1 - \beta_ {2} ^ {t}) ^ {- \frac {1}{2}} (\underbrace {1 - \beta_ {2} ^ {t} + t \beta_ {2} ^ {t} \log_ {e} \beta_ {2}} _ {\psi (t)}).
$$
$$
\psi^ {\prime} (t) = - \beta_ {2} ^ {t} \log_ {e} \beta_ {2} + \beta_ {2} ^ {t} \log_ {e} \beta_ {2} + t \beta_ {2} ^ {t} (\log_ {e} \beta_ {2}) ^ {2} = t \beta_ {2} ^ {t} (\log_ {e} \beta_ {2}) ^ {2},
$$
we have  $\psi'(t) > 0$  when  $t > 0$  and  $\psi'(t) = 0$  when  $t = 0$ . Combining  $\psi(0) = 0$ , we get  $\psi(t) > 0$  when  $t > 0$ . Thus, we have  $\phi'(t) < 0$ , which completes the proof.
Lemma 2. For the parameter settings and assumptions in Theorem 1 or Theorem 2, we have
$$
\left\| \sqrt {\boldsymbol {v} _ {t}} \right\| ^ {2} <   \frac {n (2 G _ {\infty} + \delta)}{(1 - \beta_ {1}) ^ {2}}, t \in [ T ],
$$
where  $\pmb{v}_t = \max (\sqrt{\pmb{b}_t},\delta \sqrt{1 - \beta_2^t})$
$$
\begin{array}{l} \| \boldsymbol {m} _ {t} \| _ {\infty} = \| \sum_ {i = 1} ^ {t} (1 - \beta_ {1, t - i + 1}) \boldsymbol {g} _ {t - i + 1} \prod_ {j = 1} ^ {i - 1} \beta_ {1, t - j + 1} \| _ {\infty} \leq \| \sum_ {i = 1} ^ {t} \boldsymbol {g} _ {t - i + 1} \beta_ {1} ^ {i - 1} \| _ {\infty} \leq \frac {G _ {\infty}}{1 - \beta_ {1}}, \\ \| \boldsymbol {s} _ {t} \| _ {\infty} \leq \left\{ \begin{array}{l l} \frac {\| \boldsymbol {m} _ {1} \| _ {\infty}}{1 - \beta_ {1 , t}} \leq \frac {G _ {\infty}}{(1 - \beta_ {1}) ^ {2}} <   \frac {2 G _ {\infty}}{(1 - \beta_ {1}) ^ {2}} & t = 1, \\ \frac {\| \boldsymbol {m} _ {t} \| _ {\infty}}{1 - \beta_ {1 , t} ^ {t}} + \frac {\| \boldsymbol {m} _ {t - 1} \| _ {\infty}}{1 - \beta_ {1 , t} ^ {t - 1}} \leq \frac {2 G _ {\infty}}{(1 - \beta_ {1}) ^ {2}} & t > 1, \end{array} \right. \\ \| \pmb {b} _ {t} \| _ {\infty} = \| (1 - \beta_ {2}) \sum_ {i = 1} ^ {t} \pmb {s} _ {t - i + 1} ^ {2} \beta_ {2} ^ {i - 1} \| _ {\infty} \leq \frac {4 G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {4}}, \\ \left\| \sqrt {\boldsymbol {v} _ {t}} \right\| ^ {2} = \sum_ {i = 1} ^ {n} v _ {t, i} <   n \left(\left\| \sqrt {\boldsymbol {b} _ {t}} \right\| _ {\infty} + \delta\right) \leq \frac {n \left(2 G _ {\infty} + \delta\right)}{\left(1 - \beta_ {1}\right) ^ {2}}. \\ \end{array}
$$
By assumptions 2, 4, Lemma 1 and Lemma 2,  $\forall t\in [T]$ , we have
$$
\left\| \boldsymbol {m} _ {t} \right\| _ {\infty} \leq \frac {G _ {\infty}}{1 - \beta_ {1}}, \quad \left\| \boldsymbol {v} _ {t} \right\| _ {\infty} \leq \frac {2 G _ {\infty}}{(1 - \beta_ {1}) ^ {2}}, \quad \frac {\hat {\alpha} _ {t}}{v _ {t , i}} > \frac {\hat {\alpha} _ {t + 1}}{v _ {t + 1 , i}}, \forall i \in [ n ]. \tag {6}
$$
Following Yang et al. [33], Chen et al. [9], Zhou et al. [37], we define an auxiliary sequence  $\{\pmb{u}_t\} : \forall t \geq 2$
$$
\boldsymbol {u} _ {t} = \boldsymbol {w} _ {t} + \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} _ {t - 1}\right) = \frac {1}{1 - \beta_ {1 , t}} \boldsymbol {w} _ {t} - \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \boldsymbol {w} _ {t - 1}. \tag {7}
$$
$$
\begin{array}{l} \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} = \left(\frac {1}{1 - \beta_ {1 , t + 1}} - \frac {1}{1 - \beta_ {1 , t}}\right) \boldsymbol {w} _ {t + 1} - \left(\frac {\beta_ {1 , t + 1}}{1 - \beta_ {1 , t + 1}} - \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}}\right) \boldsymbol {w} _ {t} \\ + \frac {1}{1 - \beta_ {1 , t}} \left(\boldsymbol {w} _ {t + 1} - \boldsymbol {w} _ {t}\right) - \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} _ {t - 1}\right) \\ = \left(\frac {1}{1 - \beta_ {1 , t + 1}} - \frac {1}{1 - \beta_ {1 , t}}\right) \left(\boldsymbol {w} _ {t} - \hat {\alpha} _ {t} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}}\right) - \left(\frac {\beta_ {1 , t + 1}}{1 - \beta_ {1 , t + 1}} - \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}}\right) \boldsymbol {w} _ {t} \\ - \frac {\hat {\alpha} _ {t}}{1 - \beta_ {1 , t}} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} + \frac {\beta_ {1 , t} \hat {\alpha} _ {t - 1}}{1 - \beta_ {1 , t}} \frac {\boldsymbol {m} _ {t - 1}}{\boldsymbol {v} _ {t - 1}} \\ = \left(\frac {1}{1 - \beta_ {1 , t}} - \frac {1}{1 - \beta_ {1 , t + 1}}\right) \hat {\alpha} _ {t} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} - \frac {\hat {\alpha} _ {t}}{1 - \beta_ {1 , t}} \left(\beta_ {1, t} \frac {\boldsymbol {m} _ {t - 1}}{\boldsymbol {v} _ {t}} + (1 - \beta_ {1, t}) \frac {\boldsymbol {g} _ {t}}{\boldsymbol {v} _ {t}}\right) + \frac {\beta_ {1 , t} \hat {\alpha} _ {t - 1}}{1 - \beta_ {1 , t}} \frac {\boldsymbol {m} _ {t - 1}}{\boldsymbol {v} _ {t - 1}} \\ = \left(\frac {1}{1 - \beta_ {1 , t}} - \frac {1}{1 - \beta_ {1 , t + 1}}\right) \hat {\alpha} _ {t} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} + \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \left(\frac {\hat {\alpha} _ {t - 1}}{\boldsymbol {v} _ {t - 1}} - \frac {\hat {\alpha} _ {t}}{\boldsymbol {v} _ {t}}\right) \boldsymbol {m} _ {t - 1} - \hat {\alpha} _ {t} \frac {\boldsymbol {g} _ {t}}{\boldsymbol {v} _ {t}}. \tag {8} \\ \end{array}
$$
By assumption 1 and Equation (8), we have
$$
\begin{array}{l} f \left(\boldsymbol {u} _ {t + 1}\right) \leq f \left(\boldsymbol {u} _ {t}\right) + \left\langle \nabla f \left(\boldsymbol {u} _ {t}\right), \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\rangle + \frac {L}{2} \left\| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\| ^ {2} \\ = f \left(\boldsymbol {u} _ {t}\right) + \left\langle \nabla f \left(\boldsymbol {w} _ {t}\right), \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\rangle + \left\langle \nabla f \left(\boldsymbol {u} _ {t}\right) - \nabla f \left(\boldsymbol {w} _ {t}\right), \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\rangle + \frac {L}{2} \| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \| ^ {2} \\ = f \left(\boldsymbol {u} _ {t}\right) + \left\langle \nabla f \left(\boldsymbol {w} _ {t}\right), \left(\frac {1}{1 - \beta_ {1 , t}} - \frac {1}{1 - \beta_ {1 , t + 1}}\right) \hat {\alpha} _ {t} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} \right\rangle + \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \left\langle \nabla f \left(\boldsymbol {w} _ {t}\right), \left(\frac {\hat {\alpha} _ {t - 1}}{\boldsymbol {v} _ {t - 1}} - \frac {\hat {\alpha} _ {t}}{\boldsymbol {v} _ {t}}\right) \boldsymbol {m} _ {t - 1} \right\rangle \\ - \hat {\alpha} _ {t} \left\langle \nabla f (\boldsymbol {w} _ {t}), \frac {\boldsymbol {g} _ {t}}{\boldsymbol {v} _ {t}} \right\rangle + \left\langle \nabla f (\boldsymbol {u} _ {t}) - \nabla f (\boldsymbol {w} _ {t}), \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\rangle + \frac {L}{2} \| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \| ^ {2}. \tag {9} \\ \end{array}
$$
Rearranging Equation (9) and taking expectation both sides, by assumption 3 and Equation (6), we get
$$
\begin{array}{l} \frac {\left(1 - \beta_ {1}\right) ^ {2} \hat {\alpha} _ {t}}{2 G _ {\infty}} \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {t}) \| ^ {2} \right] \leq \hat {\alpha} _ {t} \mathbf {E} \left[ \left\langle \nabla f (\boldsymbol {w} _ {t}), \frac {\nabla f (\boldsymbol {w} _ {t})}{\boldsymbol {v} _ {t}} \right\rangle \right] \\ \leq \mathbf {E} [ f (\boldsymbol {u} _ {t}) - f (\boldsymbol {u} _ {t + 1}) ] + \underbrace {\mathbf {E} \left[ \left\langle \nabla f (\boldsymbol {w} _ {t}) , \left(\frac {1}{1 - \beta_ {1 , t}} - \frac {1}{1 - \beta_ {1 , t + 1}}\right) \hat {\alpha} _ {t} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} \right\rangle \right]} _ {P _ {1}} \\ + \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \underbrace {\mathbf {E} \left[ \left\langle \nabla f (\boldsymbol {w} _ {t}), \left(\frac {\hat {\alpha} _ {t - 1}}{\boldsymbol {v} _ {t - 1}} - \frac {\hat {\alpha} _ {t}}{\boldsymbol {v} _ {t}}\right) \boldsymbol {m} _ {t - 1} \right\rangle \right]} _ {P _ {2}} \\ + \underbrace {\mathbf {E} \left[ \left\langle \nabla f \left(\boldsymbol {u} _ {t}\right) - \nabla f \left(\boldsymbol {w} _ {t}\right) , \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\rangle \right]} _ {P _ {3}} + \frac {L}{2} \underbrace {\mathbf {E} \left[ \left\| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\| ^ {2} \right]} _ {P _ {4}}. \tag {10} \\ \end{array}
$$
To further bound Equation (10), we need the following lemma.
Lemma 3. For the sequence  $\{\pmb{u}_t\}$  defined as Equation (7),  $\forall t \geq 2$ , we have
$$
\left\| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\| \leq \frac {\sqrt {n} G _ {\infty}}{\delta} \left(\frac {\hat {\alpha} _ {t} \beta_ {1 , t}}{(1 - \beta_ {1}) ^ {3}} + \frac {\hat {\alpha} _ {t - 1} \beta_ {1 , t}}{(1 - \beta_ {1}) ^ {2}} + \hat {\alpha} _ {t}\right),
$$
$$
\left\| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\| ^ {2} \leq \frac {3 n G _ {\infty} ^ {2}}{\delta^ {2}} \left(\frac {\hat {\alpha} _ {t} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {6}} + \frac {\hat {\alpha} _ {t - 1} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {4}} + \hat {\alpha} _ {t} ^ {2}\right).
$$
Proof. Since  $\forall t\in [T],\forall i\in [n],1 / (1 - \beta_{1,t})\geq 1 / (1 - \beta_{1,t + 1}),\pmb{v}_t\geq \delta \sqrt{1 - \beta_2},\hat{\alpha}_{t - 1} / v_{t - 1,i} > \hat{\alpha}_t / v_{t,i}$
By Equation (8), we have
$$
\begin{array}{l} \| \boldsymbol{u}_{t + 1} - \boldsymbol{u}_{t}\| \leq \hat{\alpha}_{t}\frac{\sqrt{n}G_{\infty}}{(1 - \beta_{1})\delta\sqrt{1 - \beta_{2}}}\left(\frac{\beta_{1,t} - \beta_{1,t + 1}}{(1 - \beta_{1,t})(1 - \beta_{1,t + 1})}\right) + \frac{\beta_{1,t}}{1 - \beta_{1,t}}\hat{\alpha}_{t - 1}\frac{\sqrt{n}G_{\infty}}{(1 - \beta_{1})\delta\sqrt{1 - \beta_{2}}} +\hat{\alpha}_{t}\frac{\sqrt{n}G_{\infty}}{\delta\sqrt{1 - \beta_{2}}} \\ \leq \frac {\sqrt {n} G _ {\infty}}{\delta \sqrt {1 - \beta_ {2}}} \left(\frac {\hat {\alpha} _ {t} \beta_ {1 , t}}{(1 - \beta_ {1}) ^ {3}} + \frac {\hat {\alpha} _ {t - 1} \beta_ {1 , t}}{(1 - \beta_ {1}) ^ {2}} + \hat {\alpha} _ {t}\right), \\ \end{array}
$$
$$
\left\| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \right\| ^ {2} \leq \frac {3 n G _ {\infty} ^ {2}}{\delta^ {2} (1 - \beta_ {2})} \left(\frac {\hat {\alpha} _ {t} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {6}} + \frac {\hat {\alpha} _ {t - 1} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {4}} + \hat {\alpha} _ {t} ^ {2}\right),
$$
where the last inequality follows from Cauchy-Schwartz inequality. This completes the proof.
Now we bound  $P_{1}, P_{2}, P_{3}$  and  $P_{4}$  of Equation (10), separately. By assumptions 1, 2, Equation (6) and Lemma 3, we have
$$
\begin{array}{l} P _ {1} \leq \hat {\alpha} _ {t} \left(\frac {1}{1 - \beta_ {1 , t}} - \frac {1}{1 - \beta_ {1 , t + 1}}\right) \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {t}) \| \| \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} \| \right] \\ \leq \hat {\alpha} _ {t} \left(\frac {\beta_ {1 , t} - \beta_ {1 , t + 1}}{(1 - \beta_ {1 , t}) (1 - \beta_ {1 , t + 1})}\right) \frac {n G _ {\infty} ^ {2}}{\delta \sqrt {1 - \beta_ {2}} (1 - \beta_ {1})} \leq \frac {n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3} \delta \sqrt {1 - \beta_ {2}}} \hat {\alpha} _ {t} (\beta_ {1, t} - \beta_ {1, t + 1}), \\ \end{array}
$$
$$
P _ {2} = \mathbf {E} \left[ \sum_ {i = 1} ^ {n} \nabla_ {i} f (\boldsymbol {w} _ {t}) m _ {t - 1, i} \left(\frac {\hat {\alpha} _ {t - 1}}{v _ {t - 1 , i}} - \frac {\hat {\alpha} _ {t}}{v _ {t , i}}\right) \right] \leq \frac {G _ {\infty} ^ {2}}{1 - \beta_ {1}} \sum_ {i = 1} ^ {n} \left(\frac {\hat {\alpha} _ {t - 1}}{v _ {t - 1 , i}} - \frac {\hat {\alpha} _ {t}}{v _ {t , i}}\right),
$$
$$
\begin{array}{l} P _ {3} \leq \mathbf {E} \left[ \| \nabla f (\boldsymbol {u} _ {t}) - \nabla f (\boldsymbol {w} _ {t}) \| \| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \| \right] \leq L \mathbf {E} \left[ \| \boldsymbol {u} _ {t} - \boldsymbol {w} _ {t} \| \| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \| \right] \\ = L \hat {\alpha} _ {t - 1} \frac {\beta_ {1 , t}}{1 - \beta_ {1 , t}} \mathbf {E} \left[ \| \frac {\boldsymbol {m} _ {t - 1}}{\boldsymbol {v} _ {t - 1}} \| \| \boldsymbol {u} _ {t + 1} - \boldsymbol {u} _ {t} \| \right] \leq \frac {L n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {2} \delta^ {2} (1 - \beta_ {2})} \left(\frac {\hat {\alpha} _ {t - 1} \hat {\alpha} _ {t} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {3}} + \frac {\hat {\alpha} _ {t - 1} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {2}} + \hat {\alpha} _ {t - 1} \hat {\alpha} _ {t} \beta_ {1, t}\right) \\ <   \frac {L n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {2} \delta^ {2} (1 - \beta_ {2})} \left(\frac {\hat {\alpha} _ {t - 1} ^ {2}}{(1 - \beta_ {1}) ^ {3}} + \frac {\hat {\alpha} _ {t - 1} ^ {2}}{(1 - \beta_ {1}) ^ {2}} + \hat {\alpha} _ {t - 1} ^ {2}\right) <   \frac {3 L n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {5} \delta^ {2} (1 - \beta_ {2})} \hat {\alpha} _ {t - 1} ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} P _ {4} \leq \frac {3 n G _ {\infty} ^ {2}}{\delta^ {2} (1 - \beta_ {2})} \left(\frac {\hat {\alpha} _ {t} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {6}} + \frac {\hat {\alpha} _ {t - 1} ^ {2} \beta_ {1 , t} ^ {2}}{(1 - \beta_ {1}) ^ {4}} + \hat {\alpha} _ {t} ^ {2}\right) <   \frac {3 n G _ {\infty} ^ {2}}{\delta^ {2} (1 - \beta_ {2})} \left(\frac {\hat {\alpha} _ {t} ^ {2}}{(1 - \beta_ {1}) ^ {6}} + \frac {\hat {\alpha} _ {t - 1} ^ {2}}{(1 - \beta_ {1}) ^ {4}} + \hat {\alpha} _ {t} ^ {2}\right) \\ <   \frac {9 n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {6} \delta^ {2} (1 - \beta_ {2})} \hat {\alpha} _ {t - 1} ^ {2}. \tag {11} \\ \end{array}
$$
Replacing  $P_{1}, P_{2}, P_{3}$  and  $P_{4}$  of Equation (10) with Equation (11) and telescoping Equation (10) for  $t = 2$  to  $T$ , we have
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} \frac {\left(1 - \beta_ {1}\right) ^ {2} \hat {\alpha} _ {t}}{2 G _ {\infty}} \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {t}) \| ^ {2} \right] <   \mathbf {E} \left[ f (\boldsymbol {u} _ {2}) - f (\boldsymbol {u} _ {T + 1}) \right] + \frac {n G _ {\infty} ^ {2}}{\left(1 - \beta_ {1}\right) ^ {3} \delta \sqrt {1 - \beta_ {2}}} \sum_ {t = 2} ^ {T} \hat {\alpha} _ {t} \left(\beta_ {1, t} - \beta_ {1, t + 1}\right) \\ + \frac {\beta_ {1} G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {2}} \sum_ {i = 1} ^ {n} \left(\frac {\hat {\alpha} _ {1}}{v _ {1 , i}} - \frac {\hat {\alpha} _ {T}}{v _ {T , i}}\right) + \frac {3 L n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {5} \delta^ {2} (1 - \beta_ {2})} \sum_ {t = 2} ^ {T} \hat {\alpha} _ {t - 1} ^ {2} + \frac {9 L n G _ {\infty} ^ {2}}{2 (1 - \beta_ {1}) ^ {6} \delta^ {2} (1 - \beta_ {2})} \sum_ {t = 2} ^ {T} \hat {\alpha} _ {t - 1} ^ {2} \\ <   \mathbf {E} [ f (\boldsymbol {u} _ {2}) ] - f (\boldsymbol {w} ^ {*}) + \frac {n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3} \delta \sqrt {1 - \beta_ {2}}} \sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \left(\beta_ {1, t} - \beta_ {1, t + 1}\right) + \frac {\alpha \beta_ {1} n G _ {\infty} ^ {2}}{\left(1 - \beta_ {1}\right) ^ {3} \delta} \\ + \frac {1 5 L n G _ {\infty} ^ {2}}{2 (1 - \beta_ {1}) ^ {6} \delta^ {2} (1 - \beta_ {2})} \sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} ^ {2}. \tag {12} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} \hat {\alpha} _ {t} = \sum_ {t = 2} ^ {T} \frac {\alpha}{\sqrt {t}} \frac {\sqrt {1 - \beta_ {2} ^ {t}}}{1 - \beta_ {1 , t} ^ {t}} \geq \alpha \sqrt {1 - \beta_ {2}} \sum_ {t = 2} ^ {T} \frac {1}{\sqrt {t}} \\ = \alpha \sqrt {1 - \beta_ {2}} \left(\int_ {2} ^ {3} \frac {1}{\sqrt {2}} d s + \dots + \int_ {T - 1} ^ {T} \frac {1}{\sqrt {T}} d s\right) > \alpha \sqrt {1 - \beta_ {2}} \int_ {2} ^ {T} \frac {1}{\sqrt {s}} d s \\ = 2 \alpha \sqrt {1 - \beta_ {2}} \left(\sqrt {T} - \sqrt {2}\right). \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} ^ {2} = \sum_ {t = 1} ^ {T} \frac {\alpha^ {2}}{t} \frac {1 - \beta_ {2} ^ {t}}{(1 - \beta_ {1 , t}) ^ {2}} \leq \frac {\alpha^ {2}}{(1 - \beta_ {1}) ^ {2}} \sum_ {t = 1} ^ {T} \frac {1}{t} \\ = \frac {\alpha^ {2}}{(1 - \beta_ {1}) ^ {2}} \left(1 + \int_ {2} ^ {3} \frac {1}{2} d s + \dots + \int_ {T - 1} ^ {T} \frac {1}{T} d s\right) <   \frac {\alpha^ {2}}{(1 - \beta_ {1}) ^ {2}} \left(1 + \int_ {2} ^ {T} \frac {1}{s - 1} d s\right) \tag {13} \\ = \frac {\alpha^ {2}}{(1 - \beta_ {1}) ^ {2}} (\log (T - 1) + 1) <   \frac {\alpha^ {2}}{(1 - \beta_ {1}) ^ {2}} (\log T + 1), \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {E} \left[ f (\boldsymbol {u} _ {2}) \right] \leq f (\boldsymbol {w} _ {1}) + \mathbf {E} \left[ \left\langle \nabla f (\boldsymbol {w} _ {1}), \boldsymbol {u} _ {2} - \boldsymbol {w} _ {1} \right\rangle \right] + \frac {L}{2} \mathbf {E} \left[ \| \boldsymbol {u} _ {2} - \boldsymbol {w} _ {1} \| ^ {2} \right] \\ = f \left(\boldsymbol {w} _ {1}\right) - \frac {\hat {\alpha} _ {1}}{1 - \beta_ {1 , 2}} \mathbf {E} \left[ \left\langle \nabla f \left(\boldsymbol {w} _ {1}\right), \frac {\boldsymbol {m} _ {1}}{\boldsymbol {v} _ {1}} \right\rangle \right] + \frac {L \hat {\alpha} _ {1} ^ {2}}{2 (1 - \beta_ {1 , 2}) ^ {2}} \mathbf {E} \left[ \| \frac {\boldsymbol {m} _ {1}}{\boldsymbol {v} _ {1}} \| ^ {2} \right] \\ \leq f \left(\boldsymbol {w} _ {1}\right) + \frac {\alpha \sqrt {1 - \beta_ {2}}}{(1 - \beta_ {1}) ^ {2}} \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {1}) \| \| \frac {\boldsymbol {m} _ {1}}{\boldsymbol {v} _ {1}} \| \right] + \frac {L \alpha^ {2} (1 - \beta_ {2})}{2 (1 - \beta_ {1}) ^ {4}} \mathbf {E} \left[ \| \frac {\boldsymbol {m} _ {1}}{\boldsymbol {v} _ {1}} \| ^ {2} \right] \\ \leq f (\boldsymbol {w} _ {1}) + \frac {\alpha n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {2} \delta} + \frac {L \alpha^ {2} n G _ {\infty} ^ {2}}{2 (1 - \beta_ {1}) ^ {4} \delta^ {2}} \leq f (\boldsymbol {w} _ {1}) + \frac {n G _ {\infty} ^ {2} \alpha}{2 (1 - \beta_ {1}) ^ {4} \delta^ {2}} (2 \delta + L \alpha), \\ \end{array}
$$
C Proof of Corollary 1
Proof. Since  $\beta_{1,t} = \beta_1 / \sqrt{t}$ , we have
$$
\sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \left(\beta_ {1, t} - \beta_ {1, t + 1}\right) \leq \sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \beta_ {1, t} = \sum_ {t = 1} ^ {T} \frac {\alpha}{\sqrt {t}} \frac {\sqrt {1 - \beta_ {2} ^ {t}}}{1 - \beta_ {1 , t} ^ {t}} \beta_ {1, t} <   \frac {\alpha}{1 - \beta_ {1}} \sum_ {t = 1} ^ {T} \frac {1}{t} <   \frac {\alpha}{1 - \beta_ {1}} (\log T + 1). \tag {14}
$$
Substituting Equation (14) into Equation (5), we have
$$
\begin{array}{l} \min  _ {t \in [ T ]} \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {t}) \| ^ {2} \right] <   \frac {G _ {\infty}}{\alpha (1 - \beta_ {1}) ^ {2} (1 - \beta_ {2}) ^ {2}} \left(f (\boldsymbol {w} _ {1}) - f (\boldsymbol {w} ^ {*}) + \frac {n G _ {\infty} ^ {2} \alpha}{(1 - \beta_ {1}) ^ {8} \delta^ {2}} (2 \delta + 8 L \alpha) \right. \\ + \frac {\alpha \beta_ {1} n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3} \delta} \bigg) \frac {1}{\sqrt {T} - \sqrt {2}} + \frac {n G _ {\infty} ^ {3}}{(1 - \beta_ {2}) ^ {2} (1 - \beta_ {1}) ^ {1 0} \delta^ {2}} \left(\frac {1 5}{2} L \alpha + \delta\right) \frac {\log T}{\sqrt {T} - \sqrt {2}}. \\ \end{array}
$$
D Proof of Theorem 2
Proof. Denote  $\hat{\alpha}_{t} = \alpha_{t}\frac{\sqrt{1 - \beta_{2}^{t}}}{1 - \beta_{1,t}^{t}}$  and  $\pmb{v}_{t} = \max (\sqrt{\pmb{b}_{t}},\delta \sqrt{1 - \beta_{2}^{t}})$ , then
$$
\begin{array}{l} \| \sqrt {\boldsymbol {v} _ {t}} (\boldsymbol {w} _ {t + 1} - \boldsymbol {w} ^ {*}) \| ^ {2} = \| \sqrt {\boldsymbol {v} _ {t}} (\boldsymbol {w} _ {t} - \hat {\alpha} _ {t} \frac {\boldsymbol {m} _ {t}}{\boldsymbol {v} _ {t}} - \boldsymbol {w} ^ {*}) \| ^ {2} = \| \sqrt {\boldsymbol {v} _ {t}} (\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}) \| ^ {2} - 2 \hat {\alpha} _ {t} \left\langle \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}, \boldsymbol {m} _ {t} \right\rangle + \hat {\alpha} _ {t} ^ {2} \| \frac {\boldsymbol {m} _ {t}}{\sqrt {\boldsymbol {v} _ {t}}} \| ^ {2} \\ = \left\| \sqrt {\boldsymbol {v} _ {t}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}\right) \right\| ^ {2} + \hat {\alpha} _ {t} ^ {2} \left\| \frac {\boldsymbol {m} _ {t}}{\sqrt {\boldsymbol {v} _ {t}}} \right\| ^ {2} - 2 \hat {\alpha} _ {t} \beta_ {1, t} \left\langle \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}, \boldsymbol {m} _ {t - 1} \right\rangle - 2 \hat {\alpha} _ {t} \left(1 - \beta_ {1, t}\right) \left\langle \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}, \boldsymbol {g} _ {t} \right\rangle . \tag {15} \\ \end{array}
$$
Rearranging Equation (15), we have
$$
\begin{array}{l} \langle \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}, \boldsymbol {g} _ {t} \rangle = \frac {1}{1 - \beta_ {1 , t}} \left[ \frac {1}{2 \hat {\alpha} _ {t}} \left(\| \sqrt {\boldsymbol {v} _ {t}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}\right) \| ^ {2} - \| \sqrt {\boldsymbol {v} _ {t}} \left(\boldsymbol {w} _ {t + 1} - \boldsymbol {w} ^ {*}\right) \| ^ {2}\right) - \beta_ {1, t} \langle \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}, \boldsymbol {m} _ {t - 1} \rangle + \frac {\hat {\alpha} _ {t}}{2} \| \frac {\boldsymbol {m} _ {t}}{\sqrt {\boldsymbol {v} _ {t}}} \| ^ {2} \right] \\ \leq \frac {1}{1 - \beta_ {1}} \left[ \frac {1}{2 \hat {\alpha} _ {t}} \left(\| \sqrt {\boldsymbol {v} _ {t}} \left(\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}\right) \| ^ {2} - \| \sqrt {\boldsymbol {v} _ {t}} \left(\boldsymbol {w} _ {t + 1} - \boldsymbol {w} ^ {*}\right) \| ^ {2}\right) + \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} \| \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*} \| ^ {2} \right. \\ \left. + \frac {\beta_ {1 , t} \hat {\alpha} _ {t}}{2} \| \boldsymbol {m} _ {t - 1} \| ^ {2} + \frac {\hat {\alpha} _ {t}}{2} \| \frac {\boldsymbol {m} _ {t}}{\sqrt {\boldsymbol {v} _ {t}}} \| ^ {2} \right], \\ \end{array}
$$
where the first inequality follows from Cauchy-Schwartz inequality and  $ab \leq \frac{1}{2} (a^2 + b^2)$ . Hence, the regret
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(f _ {t} \left(\boldsymbol {w} _ {t}\right) - f _ {t} \left(\boldsymbol {w} ^ {*}\right)\right) \leq \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}, \boldsymbol {g} _ {t} \right\rangle \\ \leq \frac {1}{1 - \beta_ {1}} \sum_ {t = 1} ^ {T} \left[ \frac {1}{2 \hat {\alpha} _ {t}} \left(\| \sqrt {\boldsymbol {v} _ {t}} (\boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*}) \| ^ {2} - \| \sqrt {\boldsymbol {v} _ {t}} (\boldsymbol {w} _ {t + 1} - \boldsymbol {w} ^ {*}) \| ^ {2}\right) + \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} \| \boldsymbol {w} _ {t} - \boldsymbol {w} ^ {*} \| ^ {2} \right. \\ \left. + \frac {\beta_ {1 , t} \hat {\alpha} _ {t}}{2} \| \boldsymbol {m} _ {t - 1} \| ^ {2} + \frac {\hat {\alpha} _ {t}}{2} \| \frac {\boldsymbol {m} _ {t}}{\sqrt {\boldsymbol {v} _ {t}}} \| ^ {2} \right], \tag {16} \\ \end{array}
$$
where the first inequality follows from the convexity of  $f_{t}(\pmb{w})$ . For further bounding Equation (16), we need the following lemma.
Lemma 4. For the parameter settings and conditions assumed in Theorem 2, we have
$$
\sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \| \boldsymbol {m} _ {t} \| ^ {2} <   \frac {2 n \alpha G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3}} \sqrt {T}.
$$
substituting Equation (13) into Equation (12), we finish the proof.
Proof. From Equation (4), we have