SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition E.1. Denote  $\alpha_{t} = \frac{\beta}{1 - (1 - \beta)^{t}}$  and  $\alpha_{t}^{\mathrm{sq}} = \frac{\beta_{\mathrm{sq}}}{1 - (1 - \beta_{\mathrm{sq}})^{t}}$ . Then the update rule in Algorithm 1 is equivalent to
$$
\hat {m} _ {t} = (1 - \alpha_ {t}) \hat {m} _ {t - 1} + \alpha_ {t} \nabla f (x _ {t}, \xi_ {t}),
$$
$$
\hat {v} _ {t} = \left(1 - \alpha_ {t} ^ {\mathrm {s q}}\right) \hat {v} _ {t - 1} + \alpha_ {t} ^ {\mathrm {s q}} \left(\nabla f \left(x _ {t}, \xi_ {t}\right)\right) ^ {2}, \tag {18}
$$
$$
x _ {t + 1} = x _ {t} - \frac {\eta}{\sqrt {\hat {v} _ {t}} + \lambda} \odot \hat {m} _ {t},
$$
where initially we set  $\hat{m}_1 = \nabla f(x_1, \xi_1)$  and  $\hat{v}_1 = (\nabla f(x_1, \xi_1))^2$ . Note that since  $1 - \alpha_1 = 1 - \alpha_1^{\mathrm{sq}} = 0$ , there is no need to define  $\hat{m}_0$  and  $\hat{v}_0$ .
Proof of Proposition E.1. Denote  $Z_{t} = 1 - (1 - \beta)^{t}$ . Then we know  $\alpha_{t} = \beta / Z_{t}$  and  $m_{t} = Z_{t} \hat{m}_{t}$ . By the momentum update rule in Algorithm 1, we have
$$
Z _ {t} \hat {m} _ {t} = (1 - \beta) Z _ {t - 1} \hat {m} _ {t - 1} + \beta \nabla f \left(x _ {t}, \xi_ {t}\right).
$$
Note that  $Z_{t}$  satisfies the following property
$$
(1 - \beta) Z _ {t - 1} = 1 - \beta - (1 - \beta) ^ {t} = Z _ {t} - \beta .
$$
$$
\begin{array}{l} \hat {m} _ {t} = \frac {Z _ {t} - \beta}{Z _ {t}} \cdot \hat {m} _ {t - 1} + \frac {\beta}{Z _ {t}} \cdot \nabla f (x _ {t}, \xi_ {t}) \\ = (1 - \alpha_ {t}) \hat {m} _ {t - 1} + \alpha_ {t} \nabla f \left(x _ {t}, \xi_ {t}\right). \\ \end{array}
$$
Next, we verify the initial condition. By Algorithm 1, since we set  $m_0 = 0$ , we have  $m_{1} = \beta \nabla f(x_{1},\xi_{1})$ . Therefore we have  $\hat{m}_1 = m_1 / Z_1 = \nabla f(x_1,\xi_1)$  since  $Z_{1} = \beta$ . Then the proof is completed by applying the same analysis on  $v_{t}$  and  $\hat{v}_{t}$ .
E.2 Useful lemmas for Adam
In this section, we list several useful lemmas for the convergence analysis. Their proofs are all deferred in Appendix E.5.
First note that when  $t < \tau$ , all the quantities in the algorithm are well bounded. In particular, we have the following lemma.
Lemma E.2. If  $t < \tau$ , we have
$$
\| \nabla f (x _ {t}) \| \leq G, \quad \| \nabla f (x _ {t}, \xi_ {t}) \| \leq G + \sigma , \quad \| \hat {m} _ {t} \| \leq G + \sigma ,
$$
$$
\hat {v} _ {t} \preceq (G + \sigma) ^ {2}, \quad \frac {\eta}{G + \sigma + \lambda} \preceq h _ {t} \preceq \frac {\eta}{\lambda}.
$$
Next, we provide a useful lemma regarding the time-dependent re-scaled momentum parameters in (18).
Lemma E.3. Let  $\alpha_{t} = \frac{\beta}{1 - (1 - \beta)^{t}}$ , then for all  $T \geq 2$ , we have  $\sum_{t=2}^{T} \alpha_{t}^{2} \leq 3(1 + \beta^{2}T)$ .
In the next lemma, we provide an almost sure bound on  $\epsilon_t$  in order to apply Azuma-Hoeffding inequality (Lemma C.1).
Lemma E.4. Denote  $\gamma_{t - 1} = (1 - \alpha_t)(\epsilon_{t - 1} + \nabla f(x_{t - 1}) - \nabla f(x_t))$ . Choosing  $\eta \leq \min \left\{\frac{r}{D},\frac{\sigma\beta}{DL}\right\}$ , if  $t\leq \tau$ , we have  $\| \epsilon_t\| \leq 2\sigma$  and  $\| \gamma_{t - 1}\| \leq 2\sigma$ .
Finally, the following lemma hides messy calculations and will be useful in the contradiction argument.
Lemma E.5. Denote
$$
I _ {1} := \frac {8 G}{\eta \lambda} \left(\Delta_ {1} \lambda + 8 \sigma^ {2} \left(\frac {\eta}{\beta} + \eta \beta T\right) + 2 0 \sigma^ {2} \eta \sqrt {(1 / \beta^ {2} + T) \iota}\right),
$$
$$
I _ {2} := \frac {8 G F}{\eta} = \frac {8 G ^ {3}}{3 \eta L}.
$$
Under the parameter choices in either Theorem 4.1 or Theorem 4.2, we have  $I_1 \leq I_2$  and  $I_1 / T \leq \epsilon^2$ .
E.3 Proof of Theorem 4.1
Before proving the main theorems, several important lemmas are needed. First, we provide a descent lemma for Adam.
Lemma E.6. If  $t < \tau$ , choosing  $G \geq \sigma + \lambda$  and  $\eta \leq \min \left\{\frac{r}{D}, \frac{\lambda}{6L}\right\}$ , we have
$$
f (x _ {t + 1}) - f (x _ {t}) \leq - \frac {\eta}{4 G} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\eta}{\lambda} \| \epsilon_ {t} \| ^ {2}.
$$
Proof of Lemma E.6. By Lemma E.2, we have if  $t < \tau$
$$
\frac {\eta I}{2 G} \leq \frac {\eta I}{G + \sigma + \lambda} \preceq H _ {t} \preceq \frac {\eta I}{\lambda}. \tag {19}
$$
Since we choose  $\eta \leq \frac{r}{D}$ , by Lemma 5.3, we have  $\| x_{t + 1} - x_t\| \leq r$  if  $t < \tau$ . Then we can apply Corollary 5.2 to show that for any  $t < \tau$ ,
$$
\begin{array}{l} f (x _ {t + 1}) - f (x _ {t}) \leq \left\langle \nabla f (x _ {t}), x _ {t + 1} - x _ {t} \right\rangle + \frac {L}{2} \left\| x _ {t + 1} - x _ {t} \right\| ^ {2} \\ = - \left(\nabla f (x _ {t})\right) ^ {\top} H _ {t} \hat {m} _ {t} + \frac {L}{2} \hat {m} _ {t} ^ {\top} H _ {t} ^ {2} \hat {m} _ {t} \\ \leq - \left\| \nabla f (x _ {t}) \right\| _ {H _ {t}} ^ {2} - \left(\nabla f (x _ {t})\right) ^ {\top} H _ {t} \epsilon_ {t} + \frac {\eta L}{2 \lambda} \left\| \hat {m} _ {t} \right\| _ {H _ {t}} ^ {2} \\ \leq - \frac {2}{3} \| \nabla f (x _ {t}) \| _ {H _ {t}} ^ {2} + \frac {3}{4} \| \epsilon_ {t} \| _ {H _ {t}} ^ {2} + \frac {\eta L}{\lambda} \left(\| \nabla f (x _ {t}) \| _ {H _ {t}} ^ {2} + \| \epsilon_ {t} \| _ {H _ {t}} ^ {2}\right) \\ \leq - \frac {1}{2} \| \nabla f (x _ {t}) \| _ {H _ {t}} ^ {2} + \| \epsilon_ {t} \| _ {H _ {t}} ^ {2} \\ \leq - \frac {\eta}{4 G} \left\| \nabla f (x _ {t}) \right\| ^ {2} + \frac {\eta}{\lambda} \left\| \epsilon_ {t} \right\| ^ {2}, \\ \end{array}
$$
where the second inequality uses (17) and (19); the third inequality is due to Young's inequality  $a^\top Ab \leq \frac{1}{3} \|a\|_A^2 + \frac{3}{4} \|b\|_A^2$  and  $\|a + b\|_A^2 \leq 2 \|a\|_A^2 + 2 \|b\|_A$  for any PSD matrix  $A$ ; the second last inequality uses  $\eta \leq \frac{\lambda}{6L}$ ; and the last inequality is due to (19).
The following lemma bounds the sum of the error term  $\| \epsilon_t \|^2$  before the stopping time  $\tau$ . Since its proof is complicated, we defer it in Appendix E.4.
Lemma E.7. If  $G \geq 2\sigma$  and  $\eta \leq \min \left\{\frac{r}{D}, \frac{\lambda^{3/2}\beta}{6L\sqrt{G}}, \frac{\sigma\beta}{DL}\right\}$ , with probability  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {\tau - 1} \| \epsilon_ {t} \| ^ {2} - \frac {\lambda}{8 G} \| \nabla f (x _ {t}) \| ^ {2} \leq 8 \sigma^ {2} (1 / \beta + \beta T) + 2 0 \sigma^ {2} \sqrt {(1 / \beta^ {2} + T) \log (1 / \delta)}.
$$
Combining Lemma E.6 and Lemma E.7, we obtain the following useful lemma, which simultaneously bounds  $f(x_{t}) - f^{*}$  and  $\sum_{t = 1}^{\tau -1}\| \nabla f(x_{t})\|^{2}$ .
Lemma E.8. If  $G \geq 2\max\{\lambda, \sigma\}$  and  $\eta \leq \min\left\{\frac{r}{D}, \frac{\lambda^{3/2}\beta}{6L\sqrt{G}}, \frac{\sigma\beta}{DL}\right\}$ , then with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {\tau - 1} \left\| \nabla f (x _ {t}) \right\| ^ {2} + \frac {8 G}{\eta} \left(f (x _ {\tau}) - f ^ {*}\right) \\ \leq \frac {8 G}{\eta \lambda} \left(\Delta_ {1} \lambda + 8 \sigma^ {2} \left(\frac {\eta}{\beta} + \eta \beta T\right) + 2 0 \sigma^ {2} \eta \sqrt {(1 / \beta^ {2} + T) \log (1 / \delta)}\right). \\ \end{array}
$$
Proof of Lemma E.8. By telescoping, Lemma E.6 implies
$$
\sum_ {t = 1} ^ {\tau - 1} 2 \left\| \nabla f \left(x _ {t}\right) \right\| ^ {2} - \frac {8 G}{\lambda} \left\| \epsilon_ {t} \right\| ^ {2} \leq \frac {8 G}{\eta} \left(f \left(x _ {1}\right) - f \left(x _ {\tau}\right)\right) \leq \frac {8 \Delta_ {1} G}{\eta}. \tag {20}
$$
Lemma E.7 could be written as
$$
\sum_ {t = 1} ^ {\tau - 1} \frac {8 G}{\lambda} \| \epsilon_ {t} \| ^ {2} - \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {8 G}{\lambda} \left(8 \sigma^ {2} (1 / \beta + \beta T) + 2 0 \sigma^ {2} \sqrt {(1 / \beta^ {2} + T) \log (1 / \delta)}\right). \tag {21}
$$
$(20) + (21)$  gives the desired result.
With Lemma E.8, we are ready to complete the contradiction argument and the convergence analysis. Below we provide the proof of Theorem 4.1.
Proof of Theorem 4.1. According to Lemma E.8, there exists some event  $\mathcal{E}$  with  $\mathbb{P}(\mathcal{E})\geq 1 - \delta$ , such that conditioned on  $\mathcal{E}$ , we have
$$
\frac {8 G}{\eta} \left(f \left(x _ {\tau}\right) - f ^ {*}\right) \leq \frac {8 G}{\eta \lambda} \left(\Delta_ {1} \lambda + 8 \sigma^ {2} \left(\frac {\eta}{\beta} + \eta \beta T\right) + 2 0 \sigma^ {2} \eta \sqrt {\left(1 / \beta^ {2} + T\right) \log (1 / \delta)}\right) =: I _ {1}. \tag {22}
$$
By the definition of  $\tau$ , if  $\tau \leq T$ , we have
$$
\frac {8 G}{\eta} \left(f \left(x _ {\tau}\right) - f ^ {*}\right) > \frac {8 G F}{\eta} = \frac {8 G ^ {3}}{3 \eta L} =: I _ {2}.
$$
Based on Lemma E.5, we have  $I_1 \leq I_2$ , which leads to a contradiction. Therefore, we must have  $\tau = T + 1$  conditioned on  $\mathcal{E}$ . Then, Lemma E.8 also implies that under  $\mathcal{E}$ ,
$$
\frac {1}{T} \sum_ {t = 1} ^ {T - 1} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {I _ {1}}{T} \leq \epsilon^ {2},
$$
where the last inequality is due to Lemma E.5.
E.4 Proof of Lemma E.7
In order to prove Lemma E.7, we need the following several lemmas.
Lemma E.9. Denote  $\gamma_{t - 1} = (1 - \alpha_t)(\epsilon_{t - 1} + \nabla f(x_{t - 1}) - \nabla f(x_t))$ . If  $G \geq 2\sigma$  and  $\eta \leq \min \left\{\frac{r}{D}, \frac{\lambda^{3/2}\beta}{6L\sqrt{G}}\right\}$ , we have for every  $2 \leq t \leq \tau$ ,
$$
\| \epsilon_ {t} \| ^ {2} \leq \left(1 - \frac {\alpha_ {t}}{2}\right) \| \epsilon_ {t - 1} \| ^ {2} + \frac {\lambda \beta}{1 6 G} \| \nabla f (x _ {t - 1}) \| ^ {2} + \alpha_ {t} ^ {2} \sigma^ {2} + 2 \alpha_ {t} \Bigg \langle \gamma_ {t - 1}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle .
$$
Proof of Lemma E.9. According to the update rule (18), we have
$$
\begin{array}{l} \epsilon_ {t} = (1 - \alpha_ {t}) \left(\epsilon_ {t - 1} + \nabla f \left(x _ {t - 1}\right) - \nabla f \left(x _ {t}\right)\right) + \alpha_ {t} \left(\nabla f \left(x _ {t}, \xi_ {t}\right) - \nabla f \left(x _ {t}\right)\right) \\ = \gamma_ {t - 1} + \alpha_ {t} (\nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t})). \tag {23} \\ \end{array}
$$
Since we choose  $\eta \leq \frac{r}{D}$ , by Lemma 5.3, we have  $\| x_{t} - x_{t - 1}\| \leq r$  if  $t\leq \tau$ . Therefore by Corollary 5.2, for any  $2\leq t\leq \tau$ ,
$$
\left\| \nabla f \left(x _ {t - 1}\right) - \nabla f \left(x _ {t}\right) \right\| \leq L \left\| x _ {t} - x _ {t - 1} \right\| \leq \frac {\eta L}{\lambda} \| \hat {m} _ {t - 1} \| \leq \frac {\eta L}{\lambda} \left(\| \nabla f \left(x _ {t - 1}\right) \| + \| \epsilon_ {t - 1} \|\right), \tag {24}
$$
$$
\begin{array}{l} \left\| \gamma_ {t - 1} \right\| ^ {2} = \left\| \left(1 - \alpha_ {t}\right) \epsilon_ {t - 1} + \left(1 - \alpha_ {t}\right) \left(\nabla f \left(x _ {t - 1}\right) - \nabla f \left(x _ {t}\right)\right) \right\| ^ {2} \\ \leq (1 - \alpha_ {t}) ^ {2} \left(1 + \alpha_ {t}\right) \left\| \epsilon_ {t - 1} \right\| ^ {2} + (1 - \alpha_ {t}) ^ {2} \left(1 + \frac {1}{\alpha_ {t}}\right) \left\| \nabla f (x _ {t - 1}) - \nabla f (x _ {t}) \right\| ^ {2} \\ \leq \left(1 - \alpha_ {t}\right) \| \epsilon_ {t - 1} \| ^ {2} + \frac {1}{\alpha_ {t}} \| \nabla f (x _ {t - 1}) - \nabla f (x _ {t}) \| ^ {2} \\ \leq \left(1 - \alpha_ {t}\right) \| \epsilon_ {t - 1} \| ^ {2} + \frac {2 \eta^ {2} L ^ {2}}{\lambda^ {2} \beta} \left(\| \nabla f (x _ {t - 1}) \| ^ {2} + \| \epsilon_ {t - 1} \| ^ {2}\right) \\ \leq \left(1 - \frac {\alpha_ {t}}{2}\right) \| \epsilon_ {t - 1} \| ^ {2} + \frac {\lambda \beta}{1 6 G} \| \nabla f (x _ {t - 1}) \| ^ {2}, \\ \end{array}
$$
where the first inequality uses Young's inequality  $\| a + b\|^2 \leq (1 + u)\| a\|^2 + (1 + 1 / u)\| b\|^2$  for any  $u > 0$ ; the second inequality is due to
$$
\begin{array}{l} (1 - \alpha_ {t}) ^ {2} (1 + \alpha_ {t}) = (1 - \alpha_ {t}) (1 - \alpha_ {t} ^ {2}) \leq (1 - \alpha_ {t}), \\ (1 - \alpha_ {t}) ^ {2} \left(1 + \frac {1}{\alpha_ {t}}\right) = \frac {1}{\alpha_ {t}} (1 - \alpha_ {t}) ^ {2} (1 + \alpha_ {t}) \leq \frac {1}{\alpha_ {t}} (1 - \alpha_ {t}) \leq \frac {1}{\alpha_ {t}}; \\ \end{array}
$$
the third inequality uses (24) and Young's inequality; and in the last inequality we choose  $\eta \leq \frac{\lambda^{3/2}\beta}{6L\sqrt{G}}$ , which implies  $\frac{2\eta^2L^2}{\lambda^2\beta} \leq \frac{\lambda\beta}{16G} \leq \frac{\beta}{2} \leq \frac{\alpha_t}{2}$ . Then by (23), we have
$$
\begin{array}{l} \| \epsilon_ {t} \| ^ {2} = \| \gamma_ {t - 1} \| ^ {2} + 2 \alpha_ {t} \Bigg \langle \gamma_ {t - 1}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle + \alpha_ {t} ^ {2} \| \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \| ^ {2} \\ \leq \left(1 - \frac {\alpha_ {t}}{2}\right) \| \epsilon_ {t - 1} \| ^ {2} + \frac {\lambda \beta}{1 6 G} \| \nabla f (x _ {t - 1}) \| ^ {2} + \alpha_ {t} ^ {2} \sigma^ {2} + 2 \alpha_ {t} \Bigg \langle \gamma_ {t - 1}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle . \\ \end{array}
$$
Lemma E.10. Denote  $\gamma_{t - 1} = (1 - \alpha_t)(\epsilon_{t - 1} + \nabla f(x_{t - 1}) - \nabla f(x_t))$ . If  $G \geq 2\sigma$  and  $\eta \leq \min \left\{\frac{r}{D}, \frac{\sigma\beta}{DL}\right\}$ , with probability  $1 - \delta$ ,
$$
\sum_ {t = 2} ^ {\tau} \alpha_ {t} \Bigg \langle \gamma_ {t - 1}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle \leq 5 \sigma^ {2} \sqrt {(1 + \beta^ {2} T) \log (1 / \delta)}.
$$
Proof of Lemma E.10. First note that
$$
\sum_ {t = 2} ^ {\tau} \alpha_ {t} \Bigg \langle \gamma_ {t - 1}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle = \sum_ {t = 2} ^ {T} \alpha_ {t} \Bigg \langle \gamma_ {t - 1} 1 _ {\tau \geq t}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle .
$$
Since  $\tau$  is a stopping time, we know that  $1_{\tau \geq t}$  is a function of  $\{\xi_s\}_{s < t}$ . Also, by definition, we know  $\gamma_{t - 1}$  is a function of  $\{\xi_s\}_{s < t}$ . Then, denoting
$$
X _ {t} = \alpha_ {t} \Bigg \langle \gamma_ {t - 1} 1 _ {\tau \geq t}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle ,
$$
we know that  $\mathbb{E}_{t - 1}[X_t] = 0$ , which implies  $\{X_{t}\}_{t\leq T}$  is a martingale difference sequence. Also, by Assumption 3 and Lemma E.4, we can show that for all  $2\leq t\leq T$
$$
\left| X _ {t} \right| \leq \alpha_ {t} \sigma \left\| \gamma_ {t - 1} 1 _ {\tau \geq t} \right\| \leq 2 \alpha_ {t} \sigma^ {2}.
$$
Then by the Azuma-Hoeffding inequality (Lemma C.1), we have with probability at least  $1 - \delta$
$$
\left| \sum_ {t = 2} ^ {T} X _ {t} \right| \leq 2 \sigma^ {2} \sqrt {2 \sum_ {t = 2} ^ {T} \alpha_ {t} ^ {2} \log (1 / \delta)} \leq 5 \sigma^ {2} \sqrt {(1 + \beta^ {2} T) \log (1 / \delta)},
$$
where in the last inequality we use Lemma E.3.
Then we are ready to prove Lemma E.7.
Proof of Lemma E.7. By Lemma E.9, we have for every  $2 \leq t \leq \tau$ ,
$$
\begin{array}{l} \frac {\beta}{2} \left\| \epsilon_ {t - 1} \right\| ^ {2} \leq \frac {\alpha_ {t}}{2} \left\| \epsilon_ {t - 1} \right\| ^ {2} \leq \left\| \epsilon_ {t - 1} \right\| ^ {2} - \left\| \epsilon_ {t} \right\| ^ {2} + \frac {\lambda \beta}{1 6 G} \left\| \nabla f (x _ {t - 1}) \right\| ^ {2} + \alpha_ {t} ^ {2} \sigma^ {2} \\ + 2 \alpha_ {t} \Bigg \langle \gamma_ {t - 1}, \nabla f (x _ {t}, \xi_ {t}) - \nabla f (x _ {t}) \Bigg \rangle . \\ \end{array}
$$
Taking a summation over  $t$  from 2 to  $\tau$ , we have
$$
\begin{array}{l} \sum_ {t = 2} ^ {\tau} \frac {\beta}{2} \| \epsilon_ {t - 1} \| ^ {2} - \frac {\lambda \beta}{1 6 G} \| \nabla f (x _ {t - 1}) \| ^ {2} \leq \| \epsilon_ {1} \| ^ {2} - \| \epsilon_ {\tau} \| ^ {2} + \sigma^ {2} \sum_ {t = 2} ^ {\tau} \alpha_ {t} ^ {2} + 1 0 \sigma^ {2} \sqrt {(1 + \beta^ {2} T) \log (1 / \delta)} \\ \leq 4 \sigma^ {2} (1 + \beta^ {2} T) + 1 0 \sigma^ {2} \sqrt {(1 + \beta^ {2} T) \log (1 / \delta)}, \\ \end{array}
$$
where the first inequality uses Lemma E.10; and the second inequality uses Lemma E.3 and  $\| \epsilon_1\|^2 = \|\nabla f(x_1, \xi_1) - \nabla f(x_1)\|^2 \leq \sigma^2$ . Then we complete the proof by multiplying both sides by  $2/\beta$ .
E.5 Omitted proofs for Adam
In this section, we provide all the omitted proofs for Adam including those of Lemma 5.3 and all the lemmas in Appendix E.2.
Proof of Lemma 5.3. According to Lemma E.2, if  $t < \tau$
$$
\left\| x _ {t + 1} - x _ {t} \right\| \leq \frac {\eta}{\lambda} \left\| \hat {m} _ {t} \right\| \leq \frac {\eta (G + \sigma)}{\lambda} \leq \frac {2 \eta G}{\lambda}.
$$
Proof of Lemma E.2. By definition of  $\tau$ , we have  $\| \nabla f(x_{t}) \| \leq G$  if  $t < \tau$ . Then Assumption 3 directly implies  $\| \nabla f(x_{t}, \xi_{t}) \| \leq G + \sigma$ .  $\| \hat{m}_{t} \|$  can be bounded by a standard induction argument as follows. First note that  $\| \hat{m}_{1} \| = \| \nabla f(x_{1}, \xi_{1}) \| \leq G + \sigma$ . Supposing  $\| \hat{m}_{k - 1} \| \leq G + \sigma$  for some  $k < \tau$ , then we have
$$
\left\| \hat {m} _ {k} \right\| \leq \left(1 - \alpha_ {k}\right) \left\| \hat {m} _ {k - 1} \right\| + \alpha_ {k} \left\| \nabla f (x _ {k}, \xi_ {k}) \right\| \leq G + \sigma .
$$
Then we can show  $\hat{v}_t \preceq (G + \sigma)^2$  in a similar way noting that  $(\nabla f(x_t, \xi_t))^2 \preceq \| \nabla f(x_t, \xi_t) \|^2 \leq (G + \sigma)^2$ . Given the bound on  $\hat{v}_t$ , it is straightforward to bound the stepsize  $h_t$ .
Proof of Lemma E.3. First, when  $t \geq 1 / \beta$ , we have  $(1 - \beta)^{t} \leq 1 / e$ . Therefore,
$$
\sum_ {1 / \beta \leq t \leq T} (1 - (1 - \beta) ^ {t}) ^ {- 2} \leq (1 - 1 / e) ^ {- 2} T \leq 3 T.
$$
Next, note that when  $t < 1 / \beta$ , we have  $(1 - \beta)^{t} \leq 1 - \frac{1}{2}\beta t$ . Then we have
$$
\sum_ {2 \leq t <   1 / \beta} (1 - (1 - \beta) ^ {t}) ^ {- 2} \leq \frac {4}{\beta^ {2}} \sum_ {t \geq 2} t ^ {- m} \leq \frac {3}{\beta^ {2}}.
$$
Therefore we have  $\sum_{t=2}^{T} \alpha_t^2 \leq 3(1 + \beta^2 T)$ .
Proof of Lemma E.4. We prove  $\| \epsilon_t\| \leq 2\sigma$  for all  $t\leq \tau$  by induction. First, note that for  $t = 1$ , we have
$$
\left\| \epsilon_ {1} \right\| = \left\| \nabla f (x _ {1}, \xi_ {1}) - \nabla f (x _ {1}) \right\| \leq \sigma \leq 2 \sigma .
$$
Now suppose  $\| \epsilon_{t - 1} \| \leq 2\sigma$  for some  $2 \leq t \leq \tau$ . According to the update rule (18), we have
$$
\epsilon_ {t} = (1 - \alpha_ {t}) \left(\epsilon_ {t - 1} + \nabla f \left(x _ {t - 1}\right) - \nabla f \left(x _ {t}\right)\right) + \alpha_ {t} \left(\nabla f \left(x _ {t}, \xi_ {t}\right) - \nabla f \left(x _ {t}\right)\right),
$$
$$
\left\| \epsilon_ {t} \right\| \leq (2 - \alpha_ {t}) \sigma + \left\| \nabla f \left(x _ {t - 1}\right) - \nabla f \left(x _ {t}\right) \right\|.
$$
Since we choose  $\eta \leq \frac{r}{D}$ , by Lemma 5.3, we have  $\| x_{t} - x_{t - 1}\| \leq \eta D\leq r$  if  $t\leq \tau$ . Therefore by Corollary 5.2, we have for any  $2\leq t\leq \tau$
$$
\| \nabla f (x _ {t}) - \nabla f (x _ {t - 1}) \| \leq L \| x _ {t} - x _ {t - 1} \| \leq \eta D L \leq \sigma \alpha_ {t},
$$
where the last inequality uses the choice of  $\eta$  and  $\beta \leq \alpha_{t}$ . Therefore we have  $\| \epsilon_t\| \leq 2\sigma$  which completes the induction. Then it is straight forward to show
$$
\left\| \gamma_ {t - 1} \right\| \leq \left(1 - \alpha_ {t}\right) \left(2 \sigma + \alpha_ {t} \sigma\right) \leq 2 \sigma .
$$
Proof of Lemma E.5. We first list all the related parameter choices below for convenience.
$$
G \geq \max  \left\{2 \lambda , 2 \sigma , \sqrt {C _ {1} \Delta_ {1} L _ {0}}, \left(C _ {1} \Delta_ {1} L _ {\rho}\right) ^ {\frac {1}{2 - \rho}} \right\}, \quad \beta \leq \min  \left\{1, \frac {c _ {1} \lambda \epsilon^ {2}}{\sigma^ {2} G \sqrt {\iota}} \right\},
$$
$$
\eta \leq c _ {2} \min  \left\{\frac {r \lambda}{G}, \frac {\sigma \lambda \beta}{L G \sqrt {\imath}}, \frac {\lambda^ {3 / 2} \beta}{L \sqrt {G}} \right\}, \quad T = \max  \left\{\frac {1}{\beta^ {2}}, \frac {C _ {2} \Delta_ {1} G}{\eta \epsilon^ {2}} \right\}.
$$
We will show  $I_1 / I_2 \leq 1$  first. Note that if denoting  $W = \frac{3L}{\lambda G^2}$ , we have
$$
I _ {1} / I _ {2} = W \Delta_ {1} \lambda + 8 W \sigma^ {2} \left(\frac {\eta}{\beta} + \eta \beta T\right) + 2 0 W \sigma^ {2} \sqrt {\left(\eta^ {2} / \beta^ {2} + \eta^ {2} T\right) \iota},
$$
Below are some facts that can be easily verified given the parameter choices.
(a) By the choice of  $G$ , we have  $G^2 \geq 6\Delta_1(3L_0 + 4L_\rho G^\rho) = 6\Delta_1L$  for large enough  $C_1$ , which implies  $W \leq \frac{1}{2\Delta_1\lambda}$ .
(b) By the choice of  $T$ , we have  $\eta \beta T \leq \frac{\eta}{\beta} + \frac{C_2 \Delta_1 G \beta}{\epsilon^2}$ .
(c) By the choice of  $T$ , we have  $\eta^2 T = \max \left\{\left(\frac{\eta}{\beta}\right)^2, \frac{C_2 \eta \Delta_1 G}{\epsilon^2} \right\} \leq \left(\frac{\eta}{\beta}\right)^2 + \frac{C_2 \Delta_1 \sigma \beta}{\epsilon^2} \cdot \frac{\eta}{\beta} \leq \frac{3}{2} \left(\frac{\eta}{\beta}\right)^2 + \frac{1}{2} \left(\frac{C_2 \Delta_1 \sigma \beta}{\epsilon^2}\right)^2$ .
(d) By the choice of  $\eta$ , we have  $\eta / \beta \leq \frac{c_2 \sigma \lambda}{LG \sqrt{\iota}}$ , which implies  $W \sigma^2 \sqrt{\iota} \cdot \frac{\eta}{\beta} \leq \frac{3c_2 \sigma^3}{G^3} \leq \frac{1}{200}$  for small enough  $c_2$ .
(e) By the choice of  $\beta$  and (a), we have  $\frac{W\sigma^2\Delta_1G\sqrt{\iota}\beta}{\epsilon^2} \leq \frac{\sigma^2G\sqrt{\iota}\beta}{2\lambda\epsilon^2} \leq \frac{1}{100C_2}$  for small enough  $c_{1}$ .
$$
\begin{array}{l} I _ {1} / I _ {2} \leq \frac {1}{2} + 8 W \sigma^ {2} \left(\frac {2 \eta}{\beta} + \frac {C _ {2} \Delta_ {1} G \beta}{\epsilon^ {2}}\right) + 2 0 W \sigma^ {2} \sqrt {\imath} \left(\sqrt {\frac {5 \eta^ {2}}{2 \beta^ {2}} + \frac {1}{2} \left(\frac {C _ {2} \Delta_ {1} \sigma \beta}{\epsilon^ {2}}\right) ^ {2}}\right) \\ \leq \frac {1}{2} + 4 8 W \sigma^ {2} \sqrt {\imath} \cdot \frac {\eta}{\beta} + \frac {2 4 C _ {2} W \sigma^ {2} \Delta_ {1} G \sqrt {\imath} \beta}{\epsilon^ {2}} \\ \leq 1, \\ \end{array}
$$
where the first inequality is due to Facts (a-c); the second inequality uses  $\sigma \leq G$ ,  $\iota \geq 1$ , and  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$  for  $a, b \geq 0$ ; and the last inequality is due to Facts (d-e).
Next, we will show  $I_1 / T \leq \epsilon^2$ . We have
$$
\begin{array}{l} I _ {1} / T = \frac {8 G \Delta_ {1}}{\eta T} + \frac {6 4 \sigma^ {2} G}{\lambda \beta T} + \frac {6 4 \sigma^ {2} G \beta}{\lambda} + \frac {1 6 0 \sigma^ {2} G \sqrt {\iota}}{\lambda} \sqrt {\frac {1}{\beta^ {2} T ^ {2}} + \frac {1}{T}} \\ \leq \frac {8 \epsilon^ {2}}{C _ {2}} + \frac {2 2 4 \sigma^ {2} G \sqrt {\imath}}{\lambda \beta T} + \frac {6 4 \sigma^ {2} G \beta}{\lambda} + \frac {1 6 0 \sigma^ {2} G \sqrt {\imath}}{\lambda \sqrt {T}} \\ \leq \frac {8 \epsilon^ {2}}{C _ {2}} + \frac {4 5 0 \sigma^ {2} G \sqrt {\imath} \beta}{\lambda} \\ = \left(\frac {8}{C _ {2}} + 4 5 0 c _ {1}\right) \epsilon^ {2} \\ \leq \epsilon^ {2}, \\ \end{array}
$$
where in the first inequality we use  $T \geq \frac{C_2\Delta_1G}{\eta\epsilon^2}$  and  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$  for  $a, b \geq 0$ ; the second inequality uses  $T \geq \frac{1}{\beta^2}$ ; the second equality uses the parameter choice of  $\beta$ ; and in the last inequality we choose a large enough  $C_2$  and small enough  $c_1$ .
E.6 Proof of Theorem 4.2
Proof of Theorem 4.2. We define stopping time  $\tau$  as follows
$$
\begin{array}{l} \tau_ {1} := \min  \left\{t \mid f \left(x _ {t}\right) - f ^ {*} > F \right\} \wedge (T + 1), \\ \tau_ {2} := \min  \left\{t \mid \| \nabla f (x _ {t}) - \nabla f (x _ {t}, \xi_ {t}) \| > \sigma \right\} \wedge (T + 1), \\ \tau := \min  \left\{\tau_ {1}, \tau_ {2} \right\}. \\ \end{array}
$$
Then it is straightforward to verify that  $\tau_{1},\tau_{2},\tau$  are all stopping times.
Since we want to show  $\mathbb{P}(\tau \leq T)$  is small, noting that  $\{\tau \leq T\} = \{\tau = \tau_1 \leq T\} \cup \{\tau = \tau_2 \leq T\}$ , it suffices to bound both  $\mathbb{P}(\tau = \tau_1 \leq T)$  and  $\mathbb{P}(\tau = \tau_2 \leq T)$ .
First, we know that