SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Pr \left[ \left| \sum_ {t = 1} ^ {T} X _ {t} \right| > b a n d \sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2} \leq F \right] \leq 2 \exp \left(- \frac {b ^ {2}}{2 F + 2 c b / 3}\right).
$$
B Missing Proofs from Section 3
Proof of Lemma 3.3. By the smoothness of  $f$  and the update  $x_{t + 1} = x_t - \frac{1}{\eta_t}\widetilde{\nabla} f(x_t)$  we have
$$
\begin{array}{l} f \left(x _ {t + 1}\right) - f \left(x _ {t}\right) \\ \leq \left\langle \nabla f (x _ {t}), x _ {t + 1} - x _ {t} \right\rangle + \frac {L}{2} \| x _ {t + 1} - x _ {t} \| ^ {2} \\ = - \eta_ {t} \left\langle \nabla f (x _ {t}), \widetilde {\nabla} f (x _ {t}) \right\rangle + \frac {L \eta_ {t} ^ {2}}{2} \left\| \widetilde {\nabla} f (x _ {t}) \right\| ^ {2} \\ = - \eta_ {t} \left\langle \nabla f (x _ {t}), \theta_ {t} + \nabla f (x _ {t}) \right\rangle + \frac {L \eta_ {t} ^ {2}}{2} \| \theta_ {t} + \nabla f (x _ {t}) \| ^ {2} \\ = - \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} - \eta_ {t} \langle \nabla f (x _ {t}), \theta_ {t} \rangle + \frac {L \eta_ {t} ^ {2}}{2} \| \theta_ {t} \| ^ {2} + \frac {L \eta_ {t} ^ {2}}{2} \| \nabla f (x _ {t}) \| ^ {2} + L \eta_ {t} ^ {2} \langle \nabla f (x _ {t}), \theta_ {t} \rangle \\ = - \left(\eta_ {t} - \frac {L \eta_ {t} ^ {2}}{2}\right) \| \nabla f (x _ {t}) \| ^ {2} + \frac {L \eta_ {t} ^ {2}}{2} \| \theta_ {t} \| ^ {2} + \left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} \rangle \\ = - \left(\eta_ {t} - \frac {L \eta_ {t} ^ {2}}{2}\right) \| \nabla f (x _ {t}) \| ^ {2} + \frac {L \eta_ {t} ^ {2}}{2} \| \theta_ {t} \| ^ {2} + \underbrace {\left(L \eta_ {t} ^ {2} - \eta_ {t}\right)} _ {\leq 0} \bigl \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} + \theta_ {t} ^ {b} \bigr \rangle . \\ \end{array}
$$
Using Cauchy-Schwarz, we have  $\left\langle \nabla f(x_{t}),\theta_{t}^{b}\right\rangle \leq \frac{1}{2}\left\| \nabla f(x_{t})\right\|^{2} + \frac{1}{2}\left\| \theta_{t}^{b}\right\|^{2}$ . Thus, we derive
$$
\begin{array}{l} \Delta_ {t + 1} - \Delta_ {t} \leq - \left(\frac {2 \eta_ {t} - L \eta_ {t} ^ {2}}{2}\right) \| \nabla f (x _ {t}) \| ^ {2} + \frac {L \eta_ {t} ^ {2}}{2} \| \theta_ {t} \| ^ {2} + \left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle \\ + \frac {\eta_ {t} - L \eta_ {t} ^ {2}}{2} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\eta_ {t} - L \eta_ {t} ^ {2}}{2} \| \theta_ {t} ^ {b} \| ^ {2} \\ \leq - \frac {\eta_ {t}}{2} \| \nabla f (x _ {t}) \| ^ {2} + \frac {L \eta_ {t} ^ {2}}{2} \| \theta_ {t} \| ^ {2} + \left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle + \frac {\eta_ {t}}{2} \| \theta_ {t} ^ {b} \| ^ {2} \\ \leq - \frac {\eta_ {t}}{2} \left\| \nabla f (x _ {t}) \right\| ^ {2} + L \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {u} \right\| ^ {2} + \left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \left\langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \right\rangle + \left(L \eta_ {t} ^ {2} + \frac {\eta_ {t}}{2}\right) \left\| \theta_ {t} ^ {b} \right\| ^ {2} \\ \leq - \frac {\eta_ {t}}{2} \| \nabla f (x _ {t}) \| ^ {2} + L \eta_ {t} ^ {2} \| \theta_ {t} ^ {u} \| ^ {2} + \left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle + \frac {3 \eta_ {t}}{2} \| \theta_ {t} ^ {b} \| ^ {2}, \\ \end{array}
$$
where the third inequality is due to  $\| \theta_t\| ^2\leq 2\| \theta_t^u\| ^2 +2\left\| \theta_t^b\right\| ^2$  , and the last inequality is due to  $\eta_t\leq \frac{1}{L}$  . Rearranging, adding, and subtracting  $\mathbb{E}\left[\| \theta_t^u\| ^2\mid \mathcal{F}_{t - 1}\right]$  , we obtain the lemma.
Detailed proof of Lemma 3.5. We state the following simple properties of the choice of  $\eta$  and  $\lambda$  in Theorem 3.1. We have
$$
\frac {1}{L} \left(\frac {\sigma}{\lambda}\right) ^ {p} \leq \eta \tag {7}
$$
$$
\eta \leq \frac {1}{L} \tag {8}
$$
$$
\left(\frac {\sigma}{\lambda}\right) ^ {p} T ^ {\frac {p}{3 p - 2}} \leq \frac {1}{3 2} \tag {9}
$$
$$
T L \left(\frac {\sigma}{\lambda}\right) ^ {p} \lambda^ {2} \eta^ {2} \leq \frac {\Delta_ {1}}{2 0 4 8}. \tag {10}
$$
We will now prove by induction on  $N$  that  $E_N$  happens with probability at least  $1 - \frac{(N - 1)\delta}{T}$ . For  $N = 1$ , the event happens with probability 1. Suppose that for some  $N \leq T$ ,  $\operatorname{Pr}[E_N] \geq 1 - \frac{(N - 1)\delta}{T}$ . We will prove that  $\operatorname{Pr}[E_{N + 1}] \geq 1 - \frac{N\delta}{T}$ .
Since the LHS of (5) is non-negative, for  $k \leq N$ , we have, under the event  $E_N$
$$
\begin{array}{l} \Delta_ {k} \leq \Delta_ {1} + \left(L \eta^ {2} - \eta\right) \sum_ {t = 1} ^ {k - 1} \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle + L \eta^ {2} \sum_ {t = 1} ^ {k - 1} \left(\| \theta_ {t} ^ {u} \| ^ {2} - \mathbb {E} _ {t} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \right]\right) \\ + \frac {3 \eta}{2} \sum_ {t = 1} ^ {k - 1} \left\| \theta_ {t} ^ {b} \right\| ^ {2} + L \eta^ {2} \sum_ {t = 1} ^ {k - 1} \mathbb {E} _ {t} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \right] \leq 2 \Delta_ {1}. \\ \end{array}
$$
From the induction hypothesis and Lemma 3.3, we have that for all  $k \leq N$ ,  $\Delta_k \leq 2\Delta_1$ . Since the LHS of (5) is non-negative, by summing over  $t$  from 1 to  $N$  we have,
$$
\begin{array}{l} \Delta_ {N + 1} \leq \underbrace {\left(\eta - L \eta^ {2}\right) \sum_ {t = 1} ^ {N} \langle - \nabla f (x _ {t}) , \theta_ {t} ^ {u} \rangle} _ {A} + \underbrace {\frac {3 \eta}{2} \sum_ {t = 1} ^ {N} \left\| \theta_ {t} ^ {b} \right\| ^ {2}} _ {B} \\ + \underbrace {L \eta^ {2} \sum_ {t = 1} ^ {N} \left(\left\| \theta_ {t} ^ {u} \right\| ^ {2} - \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right]\right)} _ {C} + \underbrace {L \eta^ {2} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right]} _ {D}. \\ \end{array}
$$
The bounds for  $B$  and  $D$  are straightforward from Lemma 2.1. First, with probability 1, we have  $\| \theta_t^u\| \leq 2\lambda$ . By the smoothness of  $f$  and the fact that  $f$  is bounded below, we have
$$
\left\| \nabla f (x _ {t}) \right\| \leq \sqrt {2 L \Delta_ {t}}.
$$
Furthermore, when the event  $E_N$  happens, we have
$$
\| \nabla f (x _ {t}) \| \leq \sqrt {2 L \Delta_ {t}} \leq \sqrt {4 L \Delta_ {1}} \leq \frac {\lambda}{2}.
$$
Thus, we can apply Lemma 2.1 and obtain  $\left\| \theta_t^b\right\| \leq 4\sigma^p\lambda^{1 - p}$  and  $\mathbb{E}_t\left[\| \theta_t^u\| ^2\right]\leq 40\sigma^p\lambda^{2 - p}$ .
Upperbound for  $B$ . By (3), when the event  $E_N$  happens,
$$
\begin{array}{l} B = \frac {3 \eta}{2} \left\| \theta_ {t} ^ {b} \right\| ^ {2} \leq \frac {3 \eta}{2} \sum_ {t = 1} ^ {N} 1 6 \sigma^ {2 p} \lambda^ {2 - 2 p} = 2 4 \sigma^ {2 p} \lambda^ {2 - 2 p} \eta N \\ \leq 2 4 T \left(\frac {\sigma}{\lambda}\right) ^ {2 p} \lambda^ {2} \eta \leq 2 4 T L \left(\frac {\sigma}{\lambda}\right) ^ {p} \lambda^ {2} \eta^ {2} \leq \frac {3 \Delta_ {1}}{2 5 6}. \\ \end{array}
$$
Upperbound for  $D$ . By 4, when the event  $E_N$  happens,
$$
\begin{array}{l} D = L \eta^ {2} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right] \leq L \eta^ {2} \sum_ {t = 1} ^ {N} 4 0 \sigma^ {p} \lambda^ {2 - p} \\ \leq 4 0 \sigma^ {p} \lambda^ {2 - p} L \eta^ {2} N \leq 4 0 L T \left(\frac {\sigma}{\lambda}\right) ^ {p} (\lambda \eta) ^ {2} \leq \frac {5 \Delta_ {1}}{2 5 6}. \\ \end{array}
$$
To bound  $A$  and  $C$ , we use Freedman's inequality (Theorem A.1). We define, for  $t \geq 1$ , the following random variables
$$
Z _ {t} = \left\{ \begin{array}{l l} - \nabla f (x _ {t}) & \text {i f} \Delta_ {t} \leq 2 \Delta_ {1} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
Thus  $\| Z_t\| \leq \| \nabla f(x_t)\| \leq 2\sqrt{L\Delta_1}$  for all  $t$
Upperbound for  $A$ . Instead of bounding  $A = (\eta - L\eta^2) \sum_{t=1}^{N} \langle -\nabla f(x_t), \theta_t^u \rangle$ , we will bound  $A' = (\eta - L\eta^2) \sum_{t=1}^{N} \langle Z_t, \theta_t^u \rangle$ . We check the conditions to apply Freedman's inequality. First  $\mathbb{E}_t \left[ (\eta - L\eta^2) \langle Z_t, \theta_t^u \rangle \right] = 0$ . Further, with probability 1,  $\| \theta_t^u \|^2 \leq 2\lambda$ , and  $Z_t \leq 2\sqrt{L\Delta_1}$ , thus  $\left| (\eta - L\eta^2) \langle Z_t, \theta_t^u \rangle \right| \leq (\eta - L\eta^2) \| Z_t \| \| \theta_t^u \| \leq 4\sqrt{L\Delta_1} (\eta - L\eta^2) \lambda \leq 4\sqrt{L\Delta_1} \eta \lambda$ . Hence,  $\left\{ (\eta - L\eta^2) \langle Z_t, \theta_t^u \rangle \right\}$  is a bounded martingale difference sequence. Therefore, for constant  $a$  and  $F$  to be chosen we have
$$
\begin{array}{l} \Pr \left[ \left| \sum_ {t = 1} ^ {N} \left(\eta - L \eta^ {2}\right) \langle Z _ {t}, \theta_ {t} ^ {u} \rangle \right| > a \text {a n d} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left(\left(\eta - L \eta^ {2}\right) \langle Z _ {t}, \theta_ {t} ^ {u} \rangle\right) ^ {2} \right] \leq F \ln \frac {4 T}{\delta} \right] \\ \leq 2 \exp \left(- \frac {a ^ {2}}{2 F \ln \frac {4 T}{\delta} + \frac {8}{3} \sqrt {L \Delta_ {1}} \eta \lambda a}\right) \\ \end{array}
$$
We choose  $a$  such that
$$
2 \exp \left(- \frac {a ^ {2}}{2 F \ln \frac {4 T}{\delta} + \frac {8}{3} \sqrt {L \Delta_ {1}} \eta \lambda a}\right) = \frac {\delta}{2 T}
$$
$$
a = \left(\frac {4}{3} \sqrt {L \Delta_ {1}} \eta \lambda + \sqrt {\frac {1 6 L \Delta_ {1} \eta^ {2} \lambda^ {2}}{9} + 2 F}\right) \ln \frac {4 T}{\delta}.
$$
If we choose  $F = 64L\Delta_{1}\sigma^{p}\lambda^{2 - p}\eta^{2}T$ , we can easily show that  $a \leq \frac{7\Delta_1}{12}$ . Therefore, with probability at least  $1 - \frac{\delta}{2T}$  we have
$$
E _ {A} = \left\{\text {e i t h e r} A ^ {\prime} \leq \left| \sum_ {t = 1} ^ {N} \left(\eta - L \eta^ {2}\right) \langle Z _ {t}, \theta_ {t} ^ {u} \rangle \right| \leq \frac {7 \Delta_ {1}}{1 2} \right.
$$
$$
\left. \text {o r} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left(\left(\eta - L \eta^ {2}\right) \langle Z _ {t}, \theta_ {t} ^ {u} \rangle\right) ^ {2} \right] > F \ln \frac {4 T}{\delta} \right\}.
$$
Also notice that under the event  $E_N$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left(\left(\eta - L \eta^ {2}\right) \langle Z _ {t}, \theta_ {t} ^ {u} \rangle\right) ^ {2} \right] \\ \leq \eta^ {2} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \| Z _ {t} \| ^ {2} \| \theta_ {t} ^ {u} \| ^ {2} \right] \leq 4 \eta^ {2} L \Delta_ {1} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \right] \\ \leq 6 4 L \Delta_ {1} \sigma^ {p} \lambda^ {2 - p} \eta^ {2} N \leq 6 4 \Delta_ {1} L T \left(\frac {\sigma}{\lambda}\right) ^ {p} \lambda^ {2} \eta^ {2} \leq F \leq F \ln \frac {4 T}{\delta}. \tag {11} \\ \end{array}
$$
Under  $E_N$ , we have that  $Z_t = -\nabla f(x_t)$  for all  $t \leq N$ . Therefore, when  $E_N \cap E_A$  happens, we have  $A = A' \leq a$ .
Upperbound for  $C$ . We check the conditions to apply Freedman's inequality. First,  $\mathbb{E}_t\left[L\eta^2\left(\|\theta_t^u\|^2 - \mathbb{E}_t\left[\|\theta_t^u\|^2\right]\right)\right] = 0$ . Further, with probability 1,  $\|\theta_t^u\|^2 \leq 2\lambda$ , thus  $\left|L\eta^2\left(\|\theta_t^u\|^2 - \mathbb{E}_t\left[\|\theta_t^u\|^2\right]\right)\right| \leq L\eta^2(4\lambda^2 + 4\lambda^2) = 8L\lambda^2\eta^2$ . Hence  $\left\{L\eta^2\left(\|\theta_t^u\|^2 - \mathbb{E}_t\left[\|\theta_t^u\|^2\right]\right)\right\}$  is a bounded martingale difference sequence. Applying Freedman's inequality for constants  $c$  and  $G$  to be chosen, we have
$$
\begin{array}{l} \Pr \left[ \left| L \eta^ {2} \sum_ {t = 1} ^ {N} \left(\left\| \theta_ {t} ^ {u} \right\| ^ {2} - \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right]\right) \right| > c \text {a n d} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left(L \eta^ {2} \left(\left\| \theta_ {t} ^ {u} \right\| ^ {2} - \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right]\right)\right) ^ {2} \right] \leq G \ln \frac {4 T}{\delta} \right] \\ \leq 2 \exp \left(- \frac {c ^ {2}}{2 G \ln \frac {4 T}{\delta} + \frac {1 6}{3} L \lambda^ {2} \eta^ {2} c}\right) \\ \end{array}
$$
We choose  $c$  such that
$$
2 \exp \left(- \frac {c ^ {2}}{2 G \ln \frac {4 T}{\delta} + \frac {1 6}{3} L \lambda^ {2} \eta^ {2} c}\right) = \frac {\delta}{2 T}
$$
$$
c = \left(\frac {8}{3} L \lambda^ {2} \eta^ {2} + \sqrt {\frac {6 4 L ^ {2} \lambda^ {4} \eta^ {4}}{9} + 2 G}\right) \ln \frac {4 T}{\delta}
$$
If we choose  $G = 256L^{2}\sigma^{p}\lambda^{4 - p}\eta^{4}T$ , a simple calculation shows that  $c \leq \frac{7\Delta_1}{48}$ . We can show that with probability at least  $1 - \frac{\delta}{2T}$ , the following event happens
$$
\begin{array}{l} E _ {C} = \left\{\text {e i t h e r} C \leq \left| L \eta^ {2} \sum_ {t = 1} ^ {N} \left(\left\| \theta_ {t} ^ {u} \right\| ^ {2} - \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right]\right) \right| \leq \frac {7 \Delta_ {1}}{4 8} \right. \\ \mathrm {o r} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left(L \eta^ {2} \left(\| \theta_ {t} ^ {u} \| ^ {2} - \mathbb {E} _ {t} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \right]\right)\right) ^ {2} \right] \geq G \ln \frac {4 T}{\delta} \Bigg \}. \\ \end{array}
$$
Notice that when  $G = 256L^2\sigma^p\lambda^{4 - p}\eta^4 T$ , under  $E_N$  we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left(L \eta^ {2} \left(\| \theta_ {t} ^ {u} \| ^ {2} - \mathbb {E} _ {t} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \right]\right)\right) ^ {2} \right] \\ \leq 8 L \lambda^ {2} \eta^ {2} \sum_ {t = 1} ^ {N} \mathbb {E} _ {t} \left[ \left| L \eta^ {2} \left(\left\| \theta_ {t} ^ {u} \right\| ^ {2} - \mathbb {E} _ {t} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right]\right) \right| \right] \leq 1 6 L ^ {2} \lambda^ {2} \eta^ {4} \sum_ {t = 1} ^ {N} \mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \right] \\ \leq 2 5 6 L ^ {2} \sigma^ {p} \lambda^ {4 - p} \eta^ {4} N \leq G <   G \ln \frac {4 T}{\delta}. \tag {12} \\ \end{array}
$$
Therefore, when  $E_N \cap E_C$  happens, we have  $C \leq c$ .
Finally, combining all the bounds for  $A, B, C, D$  using union bound and selecting  $\lambda$  and  $\eta$  appropriately to simplify the constants, we obtain the lemma.
Proof of Lemma 3.6. We have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \exp \left(\left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \stackrel {(a)} {\leq} \mathbb {E} \left[ \exp \left(z _ {t} \left(\left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle + L \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right)\right)\right) \mid \mathcal {F} _ {t - 1} \right] \\ \stackrel {(b)} {\leq} \exp \left(\mathbb {E} \left[ \frac {3}{4} \left(z _ {t} \left(\left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle + L \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right)\right)\right) ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \stackrel {(c)} {\leq} \exp \left(\mathbb {E} \left[ \frac {3}{2} z _ {t} ^ {2} \eta_ {t} ^ {2} \| \nabla f (x _ {t}) \| ^ {2} \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right] + \mathbb {E} \left[ \frac {3}{2} L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \| \theta_ {t} ^ {u} \| ^ {4} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \stackrel {(d)} {\leq} \exp \left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right] + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ = \exp \left(\left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \end{array}
$$
For  $(a)$  we use Lemma 3.3. For  $(b)$  we use Lemma 2.2. Notice that
$$
\mathbb {E} \left[ \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle \right] = \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] \right] = 0,
$$
and since  $\| \theta_t^u\| \leq 2\lambda_t$  and  $\| \nabla f(x_{t})\| \leq \sqrt{2L\Delta_{t}}$  for an  $L$ -smooth function, we have
$$
\left| \left(L \eta_ {t} ^ {2} - \eta_ {t}\right) \langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \rangle + L \eta_ {t} ^ {2} \left(\left\| \theta_ {t} ^ {u} \right\| ^ {2} - \mathbb {E} \left[ \left\| \theta^ {u} \right\| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \right|
$$
$$
\begin{array}{l} \leq 2 \eta_ {t} \lambda_ {t} \| \nabla f (x _ {t}) \| + L \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| ^ {2} + \mathbb {E} \left[ \| \theta^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \leq 2 \eta_ {t} \lambda_ {t} \| \nabla f (x _ {t}) \| + 8 L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} \\ \leq 2 \eta_ {t} \lambda_ {t} \sqrt {2 L \Delta_ {t}} + 8 L \eta_ {t} ^ {2} \lambda_ {t} ^ {2}. \\ \end{array}
$$
Thus  $z_{t} \leq \frac{1}{2\eta_{t}\lambda_{t}\sqrt{2L\Delta_{t}} + 8L\eta_{t}^{2}\lambda_{t}^{2}}$ . For (c) we use  $(a + b)^{2} \leq 2a^{2} + 2b^{2}$  and  $\mathbb{E}\left[(X - \mathbb{E}[X])^2\right] \leq \mathbb{E}[X^2]$ . For (d), we use  $\| \nabla f(x_{t})\|^{2} \leq 2L\Delta_{t}$  and  $\| \theta_t^u\| \leq 2\lambda_t$ . We obtain
$$
\mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \leq 1.
$$
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(S _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] = \exp \left(S _ {t - 1}\right) \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \\ \leq \exp \left(S _ {t - 1}\right) \\ \end{array}
$$
which means  $(\exp (S_t))_{t\geq 1}$  is a supermartingale. By Ville's inequality, we have, for all  $k\geq 1$
$$
\Pr \left[ S _ {k} \geq \log \frac {1}{\delta} \right] \leq \delta \mathbb {E} [ \exp (S _ {1}) ] \leq \delta .
$$
In other words, with probability at least  $1 - \delta$ , for all  $k \geq 1$
$$
\sum_ {t = 1} ^ {k} Z _ {t} \leq \log \frac {1}{\delta}.
$$
Plugging in the definition of  $Z_{t}$  we have
$$
\begin{array}{l} \frac {1}{2} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} + \sum_ {t = 1} ^ {k} \left(z _ {t} \Delta_ {t + 1} - z _ {t} \Delta_ {t}\right) \\ \leq \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} \frac {3 z _ {t} \eta_ {t}}{2} \left\| \theta_ {t} ^ {b} \right\| ^ {2} \\ + \sum_ {t = 1} ^ {k} \left(\left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2} + z _ {t} L \eta_ {t} ^ {2}\right) \mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right). \\ \end{array}
$$
Note that we have  $z_{t}$  is a decreasing sequence by construction (see the proof of Proposition 3.7 below). Hence, the LHS of the above inequality can be bounded by
$$
\begin{array}{l} \mathrm {L H S} = \frac {1}{2} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} + z _ {k} \Delta_ {k + 1} - z _ {1} \Delta_ {1} + \sum_ {t = 2} ^ {k} \left(z _ {k - 1} - z _ {k}\right) \Delta_ {k} \\ \geq \frac {1}{2} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} + z _ {k} \Delta_ {k + 1} - z _ {1} \Delta_ {1}. \\ \end{array}
$$
We obtain the desired inequality.
Proof of Proposition 3.7. We will prove by induction on  $k$  that
$$
\frac {1}{2} \sum_ {i = 1} ^ {k} \eta_ {i} \| \nabla f (x _ {i}) \| ^ {2} + \Delta_ {k + 1} \leq \left(\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}\right) ^ {2}.
$$
The base case  $k = 0$  is trivial. Suppose the statement is true for all  $t \leq k \leq \ell$ . Now we show for  $k + 1$ . Recall that
$$
z _ {t} = \frac {1}{2 P _ {t} \eta_ {t} \lambda_ {t} \max _ {i \leq t} \sqrt {2 L \Delta_ {i}} + 8 Q _ {t} L \eta_ {t} ^ {2} \lambda_ {t} ^ {2}}.
$$
Let us choose
$$
P _ {t} = \frac {C _ {1}}{\lambda_ {t} \eta_ {t} \sqrt {2 L}} \geq 1
$$
$$
Q _ {t} = \frac {C _ {1} ^ {2} \sqrt {A}}{2 L \eta_ {t} ^ {2} \lambda_ {t} ^ {2}} \geq 1.
$$
$$
z _ {t} = \frac {1}{2 C _ {1} \max _ {i \leq t} \sqrt {\Delta_ {i}} + 4 C _ {1} ^ {2} \sqrt {A}}.
$$
Now, note that  $(z_{t})_{t\geq 1}$  is a decreasing sequence. By the induction hypothesis  $\max_{i\leq k}\sqrt{\Delta_i}\leq$ $\sqrt{\Delta_1} +2\sqrt{A} C_1$ . Hence:
$$
\begin{array}{l} \frac {z _ {t}}{z _ {k}} = \frac {2 C _ {1} \max _ {i \leq k} \sqrt {\Delta_ {i}} + 4 C _ {1} ^ {2} \sqrt {A}}{2 C _ {1} \max _ {i \leq t} \sqrt {\Delta_ {i}} + 4 C _ {1} ^ {2} \sqrt {A}} \\ \leq \frac {2 C _ {1} \left(\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}\right) + 4 C _ {1} ^ {2} \sqrt {A}}{2 C _ {1} \sqrt {\Delta_ {1}} + 4 C _ {1} ^ {2} \sqrt {A}} \\ = \frac {\sqrt {\Delta_ {1}} + 4 \sqrt {A} C _ {1}}{\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}} \leq 2. \\ \end{array}
$$
By the choice of  $\lambda_{t}$ , for all  $t\leq k$ ,  $\| \nabla f(x_{t})\| \leq \frac{\lambda_{t}}{2}$ , we can apply the second part of Lemma 2.1 to obtain
$$
\left\| \theta_ {t} ^ {b} \right\| \leq 4 \sigma^ {p} \lambda_ {t} ^ {1 - p};
$$
$$
\mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \mid \mathcal {F} _ {t - 1} \right] \leq 4 0 \sigma^ {p} \lambda_ {t} ^ {2 - p}.
$$
$$
\begin{array}{l} \frac {1}{2} z _ {k} \sum_ {t = 1} ^ {k} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} + z _ {k} \Delta_ {k + 1} \\ \leq z _ {1} \Delta_ {1} + \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} \frac {3 z _ {t} \eta_ {t}}{2} \left\| \theta_ {t} ^ {b} \right\| ^ {2} \\ + \sum_ {t = 1} ^ {k} \left(\left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2} + z _ {t} L \eta_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \leq z _ {1} \Delta_ {1} + \log \frac {1}{\delta} + 2 4 \sigma^ {2 p} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \lambda_ {t} ^ {2} \left(\frac {1}{\lambda_ {t}}\right) ^ {2 p} \\ + 4 0 \sigma^ {p} \sum_ {t = 1} ^ {k} \left(\left(3 z _ {t} ^ {2} \Delta_ {t} + 6 z _ {t} ^ {2} L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} + z _ {t}\right) L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} \left(\frac {1}{\lambda_ {t}}\right) ^ {p}\right). \\ \end{array}
$$
Since  $\frac{z_t}{z_k} \leq 2$ , we have
$$
\begin{array}{l} \frac {1}{2} \sum_ {t = 1} ^ {k} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} + \Delta_ {k + 1} \\ \leq \frac {z _ {1} \Delta_ {1}}{z _ {k}} + \frac {1}{z _ {k}} \log \frac {1}{\delta} + 4 8 \sigma^ {2 p} \sum_ {t = 1} ^ {k} \eta_ {t} \lambda_ {t} ^ {2} \left(\frac {1}{\lambda_ {t}}\right) ^ {2 p} \\ + 8 0 \sigma^ {p} \sum_ {t = 1} ^ {k} \left(\left(3 z _ {t} \Delta_ {t} + 6 z _ {t} L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} + 1\right) L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} \left(\frac {1}{\lambda_ {t}}\right) ^ {p}\right) \\ \stackrel {(a)} {\leq} \frac {\sqrt {\Delta_ {1}} + 4 \sqrt {A} C _ {1}}{\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}} \Delta_ {1} + 2 C _ {1} \left(\sqrt {\Delta_ {1}} + 4 \sqrt {A} C _ {1}\right) \log \frac {1}{\delta} + 4 8 \sigma^ {2 p} C _ {2} \sum_ {t = 1} ^ {k} L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \\ + 8 0 \sigma^ {p} \sum_ {t = 1} ^ {k} \left(\left(\frac {3 (\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}) ^ {2}}{2 C _ {1} (\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1})} + \frac {6}{8 Q _ {t}} + 1\right) L \eta_ {t} ^ {2} \lambda_ {t} ^ {2} \left(\frac {1}{\lambda_ {t}}\right) ^ {p}\right) \\ \end{array}
$$