SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} + \frac {1 6 \eta^ {2} \sigma^ {2}}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} + 4 0 \eta^ {2} \sigma^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \\ + 4 \eta^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} (\| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} - \mathbb {E} _ {Z _ {t - s + 1}} [ \| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} | \mathcal {F} _ {t - s + 1} ]) \\ + 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} - \theta_ {t - s} ^ {*} \| \| \psi_ {t - s + 1} ^ {(b)} \| + 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \| \psi_ {t - s + 1} ^ {(b)} \| \\ - 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} - \theta_ {t - s} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle - 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle \\ + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} + 4 \eta \lambda \mathcal {D} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0}. \tag {52} \\ \end{array}
$$
Further applying the bound that  $\| \psi_t^{(b)}\| \leq \frac{4\sigma^2}{\lambda}$
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) ^ {t - 1} \left\| \theta_ {1} - \theta_ {1} ^ {*} \right\| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s} \left\| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \right\| _ {2} ^ {2} \\ + \frac {1 6 \eta^ {2} \sigma^ {2}}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} + 4 0 \eta^ {2} \sigma^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \\ + 4 \eta^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} (\| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} - \mathbb {E} _ {Z _ {t - s + 1}} [ \| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} | \mathcal {F} _ {t - s} ]) \\ + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} - \theta_ {t - s} ^ {*} \| + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \\ - 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} - \theta_ {t - s} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle \underbrace {- 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle} _ {\mathrm {a n d}} \\ + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} + 4 \eta \lambda \mathcal {D} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0}. \tag {53} \\ \end{array}
$$
We further simplify the underscored term above as
$$
\begin{array}{l} - 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle \leq 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \| \psi_ {t} ^ {(v)} \|, \\ \stackrel {4 9} {\leq} 4 \eta \lambda \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \|. \\ \end{array}
$$
Using the inequality above into Equation (53) along with the bound that  $4\eta^{2}\lambda^{2}\sum_{s = 1}^{t - 1}(1 - \eta m)^{s - 1}\mathbf{1}_{C_{t - s + 1}\neq 0}\leq \frac{4\eta\lambda^{2}\Lambda_{T}}{m}$ , where  $\Lambda_T\coloneqq \sum_{t = 1}^T\mathbf{1}_{C_t\neq 0}$ , we get
$$
\begin{array}{l} \| \theta_ {t} ^ {*} - \theta_ {t} \| _ {2} ^ {2} \leq (1 - \eta m) ^ {t - 1} \| \theta_ {1} - \theta_ {1} ^ {*} \| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| _ {2} ^ {2} \\ + \frac {1 6 \eta \sigma^ {2}}{\lambda m} + \frac {4 0 \eta \sigma^ {2}}{m} + \underbrace {4 \eta^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} (\| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} - \mathbb {E} _ {Z _ {t - s + 1}} [ \| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} | \mathcal {F} _ {t - s} ])} _ {\text {T e r m I}} \\ + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} - \theta_ {t - s} ^ {*} \| + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \\ \end{array}
$$
$$
\begin{array}{l} \underbrace {- 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} - \theta_ {t - s} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle} _ {\text {T e r m I I}} + 4 \eta \lambda \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \\ + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} + 4 \eta \lambda \mathcal {D} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0}. \tag {54} \\ \end{array}
$$
Denote by the event  $\mathcal{E}^{(1)}$  as
$$
\begin{array}{l} \mathcal {E} ^ {(1)} = \bigcap_ {t = 1} ^ {T} \left\{4 \eta^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \left(\| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} - \mathbb {E} _ {Z _ {t - s + 1}} [ \| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} | \mathcal {F} _ {t - s} ]\right) \leq \right. \\ \left. 3 2 \eta^ {2} \lambda^ {2} \ln \left(\frac {2 T}{\delta}\right) + \frac {3 2 \eta^ {3 / 2} \lambda \sigma \sqrt {\ln \left(\frac {2 T}{\delta}\right)}}{\sqrt {2 m}} \right\} \\ \end{array}
$$
Lemma 20.6.
$$
\mathbb {P} [ \mathcal {E} ^ {(1)} ] \geq 1 - \delta .
$$
For every  $t\in \{1,\dots ,T\}$  , denote by the constant
$$
\begin{array}{l} U _ {t} = \left[ (1 - \eta m) ^ {t - 1} \| \theta_ {1} - \theta_ {1} ^ {*} \| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| _ {2} ^ {2} \right. \\ + \frac {1 6 \eta \sigma^ {2}}{\lambda m} + \frac {4 0 \eta \sigma^ {2}}{m} + 3 2 \eta^ {2} \lambda^ {2} \ln \left(\frac {2 T}{\delta}\right) + \frac {3 2 \eta^ {3 / 2} \lambda \sigma \sqrt {\ln \left(\frac {2 T}{\delta}\right)}}{\sqrt {2 m}} + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \\ + 4 \eta \lambda \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \left\| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \right\| + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} \\ \left. + 4 \eta \lambda \mathcal {D} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} \right]. \tag {55} \\ \end{array}
$$
Proposition 20.7. For every  $1 \leq s < t \leq T$ ,
$$
(1 - \eta m) ^ {s - 1} U _ {t - s} \leq U _ {t}. \tag {56}
$$
For each  $t\in \{1,\dots ,T\}$  , denote by the event  $\mathcal{E}_t^{(2)}$  as
$$
\mathcal {E} _ {t} ^ {(2)} = \left\{- 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} - \theta_ {t - s} ^ {*}, \psi_ {t - s + 1} ^ {(v)} \rangle \leq 4 \eta \lambda \sqrt {Q U _ {t}} \left(1 + \sqrt {1 + \frac {2}{\eta m}}\right) \right\}
$$
Observe from definition,  $\mathbb{P}[\mathcal{E}_1^{(2)}] = 1$ . In order to bound the other events, the following lemma holds.
Lemma 20.8. For each  $t\in \{2,\dots ,T\}$
$$
\mathbb {P} [ \mathcal {E} _ {t} ^ {(2)} | \mathcal {E} _ {t - 1} ^ {(2)}, \dots , \mathcal {E} _ {1} ^ {(2)} ] \geq 1 - \frac {\delta}{T}.
$$
$$
\mathcal {E} ^ {(2)} := \bigcap_ {t = 1} ^ {T} \mathcal {E} _ {t} ^ {(2)}. \tag {57}
$$
Corollary 20.9.
$$
\mathbb {P} [ \mathcal {E} ^ {(2)} ] \geq 1 - \delta ,
$$
where event  $\mathcal{E}^{(2)}$  is given in Equation (57).
Lemma 20.10. Under events  $\mathcal{E}^{(1)}$  and  $\mathcal{E}^{(2)}$ , for all times  $t \in \{1, \dots, T\}$
$$
l _ {t} ^ {2} \leq Q U _ {t},
$$
where  $Q = \frac{206\sigma^2}{\lambda^2m\eta} +\frac{4\lambda^2}{\sigma^2} +2.$
We prove this lemma by induction. The base case of  $t = 1$  holds trivially by definition.
By the induction hypothesis, assume that for some  $t > 1$ ,  $l_s^2 \leq QU_s$  holds for all  $s < t$ . Then, from Equation (54) we have that
$$
\begin{array}{l} l _ {t} ^ {2} \leq U _ {t} + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \theta_ {t - s} - \theta_ {t - s} ^ {*} \| + 4 \eta \lambda \sqrt {Q U _ {t}} \left(1 + \sqrt {1 + \frac {2}{\eta m}}\right), \\ \stackrel {(a)} {\leq} U _ {t} + \frac {8 \sigma^ {2} \eta}{\lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s - 1}{2}} \sqrt {(1 - \eta m) ^ {s} Q U _ {t - s}} + 4 \eta \lambda \sqrt {Q U _ {t}} \left(1 + \sqrt {1 + \frac {2}{\eta m}}\right), \\ \stackrel {(5 6)} {\leq} U _ {t} + \frac {8 \sigma^ {2} \eta}{\lambda} \sqrt {Q U _ {t}} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s - 1}{2}} + 4 \eta \lambda \sqrt {Q U _ {t}} \left(1 + \sqrt {1 + \frac {2}{\eta m}}\right), \\ \leq U _ {t} + \frac {6 4 \sigma^ {2}}{m \lambda} \sqrt {Q U _ {t}} + 4 \eta \lambda \sqrt {Q U _ {t}} \left(1 + \sqrt {1 + \frac {2}{\eta m}}\right), \\ \stackrel {(b)} {\leq} U _ {t} + \frac {6 4 \sigma^ {2}}{m \lambda} \sqrt {Q U _ {t}} + 1 6 \lambda \sqrt {\frac {\eta Q U _ {t}}{m}} \\ \end{array}
$$
Step  $(a)$  follows from the induction hypothesis. Step  $(b)$  follows since  $\frac{2}{nm} > 1$ .
Claim If  $Q \geq \left(\frac{206\sigma^2}{\lambda^2m\eta} + \frac{4\lambda^2}{\sigma^2}\right) + 2$ , then  $U_t + \frac{64\sigma^2}{m\lambda}\sqrt{QU_t} + 16\lambda \sqrt{\frac{\eta QU_t}{m}} \leq QU_t$ .
We prove this claim by contradiction. Assume that  $Q \geq \left( \frac{206\sigma^2}{\lambda^2m\eta} + \frac{4\lambda^2}{\sigma^2} \right) + 2$  and that  $U_t + \frac{64\sigma^2}{m\lambda} \sqrt{QU_t} + 16\lambda \sqrt{\frac{\eta QU_t}{m}} > QU_t$ . Re-arranging, we see that
$$
\begin{array}{l} \sqrt {Q U _ {t}} \left(\frac {6 4 \sigma^ {2}}{m \lambda} + 1 6 \lambda \sqrt {\frac {\eta}{m}}\right) > (Q - 1) U _ {t}, \\ \sqrt {Q} \left(\frac {6 4 \sigma^ {2}}{m \lambda} + 1 6 \lambda \sqrt {\frac {\eta}{m}}\right) > (Q - 1) \sqrt {U _ {t}}, \\ \sqrt {Q} \left(\frac {6 4 \sigma^ {2}}{m \lambda} + 1 6 \lambda \sqrt {\frac {\eta}{m}}\right) > (Q - 1) \sqrt {\frac {4 0 \eta \sigma^ {2}}{m}}.. \\ \end{array}
$$
The last inequality follows from Equation (55), where  $U_{t} \geq \frac{40\eta\sigma^{2}}{m}$ . Re-aranging the last inequality, we see that
$$
\left(\frac {6 4 \sigma}{\lambda \sqrt {4 0 m \eta}} + \frac {1 6 \lambda}{\sqrt {4 0} \sigma}\right) \sqrt {Q} > (Q - 1).
$$
It is easy to verify that the above inequality and the fact that  $Q = \left(\frac{206\sigma^2}{\lambda^2m\eta} + \frac{4\lambda^2}{\sigma^2}\right) + 2$  cannot hold simultaneously. This can be checked by squaring both sides and solving for the root of the quadratic equation in  $Q$ .
20.2 Proof of Theorem 20.1
We know from Lemma 20.6 and Corollary 20.9, that with probability at-least  $1 - 2\delta$  , events  $\mathcal{E}^{(1)}\cap \mathcal{E}^{(2)}$  holds. Lemma 20.10 gives that for all  $t\in \{1,\dots ,T\}$ $\| \theta_t - \theta_t^*\| ^2\leq QU_t$  , where  $Q$  is given in Lemma 20.10 and  $U_{t}$  is given in Equation (55). Now applying the inequality that  $\sqrt{x + y}\leq \sqrt{x} +\sqrt{y}$  we get that
$$
\| \theta_ {t} - \theta_ {t} ^ {*} \| \leq \left(\frac {1 5 \sigma}{\lambda \sqrt {m \eta}} + \frac {2 \lambda}{\sigma} + 2\right) \left[ (1 - \eta m) ^ {\frac {t - 1}{2}} \| \theta_ {1} - \theta_ {1} ^ {*} \| + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s}{2}} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| \right].
$$
$$
\begin{array}{l} + \frac {4 \sqrt {\eta} \sigma}{\sqrt {\lambda m}} + \frac {7 \sigma \sqrt {\eta}}{\sqrt {m}} + 6 \eta \lambda \sqrt {\ln \left(\frac {2 T}{\delta}\right)} + \frac {6 \eta^ {3 / 4} \sqrt {\lambda \sigma \sqrt {\ln \left(\frac {2 T}{\delta}\right)}}}{2 m ^ {1 / 4}} + \frac {3 \sigma \sqrt {\eta}}{\sqrt {\lambda}} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s - 1}{2}} \sqrt {\| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \|} \\ + 2 \sqrt {\eta \lambda} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s - 1}{2}} \sqrt {\| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \|} + 2 \eta \lambda \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s - 1}{2}} \mathbf {1} _ {C _ {t - s + 1} \neq 0} \\ \left. + 2 \sqrt {\eta \lambda \mathcal {D}} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {\frac {s - 1}{2}} \mathbf {1} _ {C _ {t - s + 1} \neq 0} \right] \\ \end{array}
$$
Recalling that  $\mathrm{Reg}_T\coloneqq \sum_{t = 1}^T\| \theta_t - \theta_t^*\|$ , we have by simplifying using the series expansions in Equations (34, 35, 36, 37), we get that
$$
\begin{array}{l} \operatorname {R e g} _ {T} \leq \left(\frac {1 5 \sigma}{\lambda \sqrt {m \eta}} + \frac {2 \lambda}{\sigma} + 2\right) \left[ \frac {2 \left(l _ {1} + \Phi_ {T}\right)}{\eta m} + 4 \sigma T \sqrt {\frac {\eta}{\lambda m}} + 7 \sigma T \frac {\eta}{m} + 6 \eta \lambda T \sqrt {\ln \left(\frac {2 T}{\delta}\right)} \right. \\ + \frac {6 T \eta^ {3 / 4} \sqrt {\lambda \sigma \sqrt {\ln \left(\frac {2 T}{\delta}\right)}}}{2 m ^ {1 / 4}} + 3 \sigma \sqrt {\frac {T \Phi_ {T}}{\eta \lambda m ^ {2}}} + 2 \sqrt {\frac {\lambda T \Phi_ {T}}{\eta}} + \frac {4 \lambda \Lambda_ {T}}{\eta m} + 4 \Lambda_ {T} \sqrt {\frac {\lambda \mathcal {D}}{\eta m ^ {2}}} ]. \\ \end{array}
$$
20.3 Proof of Lemma 20.8
Proof. Fix a time  $t \in \{1, \dots, T\}$ . We wish to bound the sum  $\sum_{s=1}^{t-1} 2\eta(1 - \eta m)^{s-1} \langle \theta_{t-s}^* - \theta_{t-s}, \psi_{t-s+1}^{(v)} \rangle$ . In order to do so, we define the following sequence of random vectors
$$
\xi_ {s} = \left\{ \begin{array}{l l} \theta_ {s} ^ {*} - \theta_ {s} & \text {i f} \| \theta_ {s} ^ {*} - \theta_ {s} \| ^ {2} \leq Q U _ {s}, \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {58}
$$
where  $Q$  is defined in Lemma 20.10. From the induction hypothesis, on the event  $\mathcal{E}_{t-1}^{(2)}, \dots, \mathcal{E}_1^{(2)}$ , we have
$$
\sum_ {s = 1} ^ {t - 1} 2 \eta (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} ^ {*} - \theta_ {t - s}, \psi_ {t - s + 1} ^ {(v)} \rangle = \sum_ {s = 1} ^ {t - 1} 2 \eta (1 - \eta m) ^ {s - 1} \langle \xi_ {t - s}, \psi_ {t - s + 1} ^ {(v)} \rangle ,
$$
holding almost-surely. We will now bound the sum  $\sum_{s=1}^{t-1} 2\eta(1 - \eta m)^{s-1} \langle \xi_{t-s}, \psi_{t-s+1}^{(v)} \rangle$  using Freedman's martinglae inequality. We know that,
$$
\begin{array}{l} | 2 \eta (1 - \eta m) ^ {s - 1} \langle \xi_ {t - s}, \psi_ {t - s + 1} ^ {(v)} \rangle | \leq 2 \eta (1 - \eta m) ^ {s - 1} \| x i _ {t - s} \| \| \psi_ {t - s + 1} ^ {(v)} \|, \\ \leq 4 \eta \lambda \sqrt {Q U _ {t}}. \\ \end{array}
$$
Further, the sum of conditional variances can be bounded as
$$
\sum_ {s = 1} ^ {t - 1} \left| 2 \eta (1 - \eta m) ^ {s - 1} \langle \xi_ {t - s}, \psi_ {t - s + 1} ^ {(v)} \rangle \right| ^ {2} \leq \sum_ {s = 1} ^ {t - 1} \left| 1 6 \lambda^ {2} \eta^ {2} (1 - \eta m) ^ {s - 1} Q U _ {t}, \right.
$$
$$
\leq \frac {1 6 \lambda^ {2} \eta Q U _ {t}}{m}.
$$
Applying Freedman's inequality, we see that
$$
\begin{array}{l} \sum_ {s = 1} ^ {t - 1} 2 \eta (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} ^ {*} - \theta_ {t - s}, \psi_ {t - s + 1} ^ {(v)} \rangle \leq 4 \eta \lambda \sqrt {Q U _ {t}} + \sqrt {1 6 \eta^ {2} \lambda^ {2} Q U _ {t} \ln^ {2} (T / \delta) + \frac {3 2 \lambda^ {2} \eta Q U _ {t}}{m} \ln (T / \delta)}, \\ \leq 4 \eta \lambda \sqrt {Q U _ {t}} \left(1 + \sqrt {1 + \frac {2}{\eta m}}\right). \\ \end{array}
$$
20.4 Useful Martingale concentration inequality
Lemma 20.11 (Freedman's inequality[59]). Suppose  $Y_{1}, \dots, Y_{T}$  is a bounded martingale with respect to a filtration  $(\mathcal{F}_t)_{t=0}^T$  with  $\mathbb{E}[Y_t | \mathcal{F}_{t-1}] = 0$  and  $\mathbb{P}[|Y_t| \leq B] = 1$  for all  $t \in \{1, \dots, T\}$ . Denote by  $V_s := \sum_{n=1}^{s} \text{Var}(Y_n | \mathcal{F}_{n-1})$  be the sum of conditional variances. Then, for every  $a, v > 0$ ,
$$
\mathbb {P} \left(\exists n \in [ 1, T ] \text {s u c h t h a t} \sum_ {t = 1} ^ {n} Y _ {t} \geq a \text {a n d} V _ {n} \leq v\right) \leq \exp \left(\frac {- a ^ {2}}{2 (v + B a)}\right). \tag {59}
$$
Re-arranging the above inequality, we see that if
$$
a \geq B \ln \left(\frac {2 T}{\delta}\right) + \sqrt {\left(B \ln \left(\frac {2 T}{\delta}\right)\right) ^ {2} + 2 v \ln \left(\frac {2 T}{\delta}\right)}, \tag {60}
$$
then the RHS of Equation (59) is bounded above by  $\frac{\delta}{2}$ .
20.5 Proof of Lemma 20.6
Proof of Lemma 20.6. Fix a  $t \in \{1, \dots, T\}$ . For  $s \in \{1, \dots, t - 1\}$ , denote by the random variable  $Y_{s}^{(t)} \coloneqq 4\eta^{2}(1 - \eta m)^{s - 1}(\| \psi_{t - s + 1}^{(v)} \|_{2}^{2} - \mathbb{E}_{Z_{t - s + 1}}[ \| \psi_{t - s + 1}^{(v)} \|_{2}^{2} | \mathcal{F}_{t - s}])$ . Observe that the sequence  $(Y_{s}^{(t)})_{s = 1}^{t - 1}$  is a martingale difference sequence with respect to the filtration  $(\mathcal{G}_s)_{s = 1}^{t - 1}$ , where  $\mathcal{G}_s \coloneqq \mathcal{F}_{t - s}$ . Furthermore, Lemma 20.5 gives that with probability  $1$ ,  $|Y_{s}^{(t)}| \leq 4\eta^{2}(4\lambda^{2} + 4\lambda^{2}) \leq 32\eta^{2}\lambda^{2}$ . We can bound the conditional variance as
$$
\begin{array}{l} \sum_ {s = 1} ^ {t - 1} \mathrm {V a r} (Y _ {s} ^ {(t)} | \mathcal {G} _ {s}) \leq 1 6 \eta^ {4} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {2 (s - 1)} \mathbb {E} _ {Z _ {t - s}} [ (\| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} - \mathbb {E} _ {Z _ {t - s + 1}} [ \| \psi_ {t - s + 1} ^ {(v)} \| _ {2} ^ {2} | \mathcal {F} _ {t - s} ]) ^ {2} | \mathcal {F} _ {t - s} ], \\ \stackrel {4 9} {\leq} 1 6 \eta^ {4} 8 \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {2 (s - 1)} \mathbb {E} _ {Z _ {t - s}} [ \| \lVert \psi_ {t - s + 1} ^ {(v)} \rVert_ {2} ^ {2} - \mathbb {E} _ {Z _ {t - s + 1}} [ \| \lVert \psi_ {t - s + 1} ^ {(v)} \rVert_ {2} ^ {2} | \mathcal {F} _ {t - s} ] | | \mathcal {F} _ {t - s} ], \\ \leq 1 2 8 \eta^ {4} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {2 (s - 1)} (2 \mathbb {E} _ {Z _ {t - s}} [ | | \psi_ {t - s + 1} ^ {(v)} | | _ {2} ^ {2} | \mathcal {F} _ {t - s} ]), \\ \stackrel {5 1} {\leq} 2 5 6 0 \eta^ {4} \lambda^ {2} \sigma^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {2 (s - 1)}, \\ \leq \frac {2 5 6 0 \eta^ {3} \lambda^ {2} \sigma^ {2}}{m}. \\ \end{array}
$$
Now, putting  $B \coloneqq 32\eta^2\lambda^2$  and  $v = \frac{2560\eta^3\lambda^2\sigma^2}{m}$ , we get from Equation (60) that with probability at-least  $1 - \delta / 2$ ,
$$
\text {T e r m} \mathrm {I} \leq 3 2 \eta^ {2} \lambda^ {2} \ln \left(\frac {2 T}{\delta}\right) + \sqrt {\left(3 2 \eta^ {2} \lambda^ {2} \ln \left(\frac {2 T}{\delta}\right)\right) ^ {2} + \frac {5 1 2 0 \eta^ {3} \lambda^ {2} \sigma^ {2}}{m} \ln \left(\frac {2 T}{\delta}\right)},
$$
$$
\leq 3 2 \eta^ {2} \lambda^ {2} \ln \left(\frac {2 T}{\delta}\right) + \frac {3 2 \eta^ {3 / 2} \lambda \sigma \sqrt {\ln \left(\frac {2 T}{\delta}\right)}}{\sqrt {2 m}}.
$$
20.6 Proof of Lemma 20.5 reproduced from [25]
This follows from the analysis of [25] and reproduced here for completeness.
Proof of Equation (49)
$$
\begin{array}{l} \left\| \psi_ {t} ^ {(v)} \right\| = \left\| \psi_ {t} - \mathbb {E} _ {Z _ {t}} \left[ \psi_ {t} \mid \mathcal {F} _ {t - 1} \right] \right\|, \\ \stackrel {(a)} {=} \left\| \operatorname {c l i p} \left(\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right), \lambda\right) - \mathbb {E} _ {Z _ {t}} \left[ \operatorname {c l i p} \left(\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right), \lambda\right) \mid \mathcal {F} _ {t - 1} \right], \right. \\ \leq \left\| \operatorname {c l i p} \left(\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right), \lambda\right) \right\| + \left\| \mathbb {E} _ {Z _ {t}} \left[ \operatorname {c l i p} \left(\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right), \lambda\right) \mid \mathcal {F} _ {t - 1} \right] \right\|, \\ \leq \lambda + \lambda \\ \end{array}
$$
Equality  $(a)$  follows from the fact that  $\theta_{t - 1}\in \mathcal{F}_{t - 1}$
Proof of Equation (50)
Denote by the event  $\xi_t \coloneqq \mathbf{1}_{\|\nabla \mathcal{L}(Z_t, \theta_{t-1})\| > \lambda}$  and by  $\chi_t \coloneqq \mathbf{1}_{\|\nabla \mathcal{L}(Z_t, \theta_{t-1}) - \nabla \mathcal{R}_t(\theta_{t-1})\| > \frac{\lambda}{2}}$ . Since we have  $\lambda > 2G$ , we have the inequality that
$$
\xi_ {t} \leq \chi_ {t} a. s. \tag {61}
$$
$$
\operatorname {c l i p} \nabla \mathcal {L} (t, \theta_ {t - 1})), \lambda) = \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) (1 - \xi_ {t}) + \frac {\lambda}{\| \nabla \mathcal {L} (Z _ {t} , \theta_ {t - 1}) \|} \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \xi_ {t}.
$$
Now, expanding out  $\psi_t^{(b)}$ , we get
$$
\begin{array}{l} \| \psi_ {t} ^ {(b)} \| _ {2} = \| \mathbb {E} _ {Z _ {t}} [ \psi_ {t} | \mathcal {F} _ {t - 1} ] \| _ {2}, \\ = \| \mathbb {E} _ {Z _ {t}} [ \operatorname {c l i p} (\nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}), \lambda) - \mathcal {R} _ {t} (\theta_ {t - 1}) | \mathcal {F} _ {t - 1} ] \| _ {2}, \\ = \left\| \mathbb {E} _ {Z _ {t}} \left[ \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) + \left(\frac {\lambda}{\| \nabla \mathcal {L} (Z _ {t} , \theta_ {t - 1}) \|} - 1\right) \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \xi_ {t} - \mathcal {R} _ {t} (\theta_ {t - 1}) \middle | \mathcal {F} _ {t - 1} \right] \right\| _ {2}, \\ \stackrel {(a)} {=} \left\| \mathbb {E} _ {Z _ {t}} \left[ \left(\frac {\lambda}{\| \nabla \mathcal {L} (Z _ {t} , \theta_ {t - 1}) \|} - 1\right) \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \xi_ {t} \middle | \mathcal {F} _ {t - 1} \right] \right\| _ {2}, \\ \leq \mathbb {E} _ {Z _ {t}} \left[ \left| \frac {\lambda}{\| \nabla \mathcal {L} (Z _ {t} , \theta_ {t - 1}) \|} - 1 \right| \| \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \| \xi_ {t} \right\rvert   \mathcal {F} _ {t - 1} \Bigg ], \\ \leq \mathbb {E} _ {Z _ {t}} \left[ 1 - \left(\frac {\lambda}{\| \nabla \mathcal {L} (Z _ {t} , \theta_ {t - 1}) \|}\right) \| \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \| \xi_ {t} \Big | \mathcal {F} _ {t - 1} \right], \\ \leq \mathbb {E} _ {Z _ {t}} [ \| \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \| \xi_ {t} | \mathcal {F} _ {t - 1} ], \\ \stackrel {6 1} {\leq} \mathbb {E} _ {Z _ {t}} [ \| \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) \| _ {\chi_ {t}} | \mathcal {F} _ {t - 1} ], \\ \leq \mathbb {E} _ {Z _ {t}} [ \| \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) - \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| \chi_ {t} | \mathcal {F} _ {t - 1} ] + \| \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| \mathbb {E} _ {Z _ {t}} [ \chi_ {t} | \mathcal {F} _ {t - 1} ], \\ \leq \sqrt {\mathbb {E} _ {Z _ {t}} [ \| \nabla \mathcal {L} (Z _ {t} , \theta_ {t - 1}) - \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| _ {2} ^ {2} | \mathcal {F} _ {t - 1} ] \mathbb {E} _ {Z _ {t}} [ \chi_ {t} ^ {2} | \mathcal {F} _ {t - 1} ]} + \| \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| \mathbb {E} _ {Z _ {t}} [ \chi_ {t} | \mathcal {F} _ {t - 1} ], \\ \leq \sigma \sqrt {\mathbb {E} _ {Z _ {t}} \left[ \chi_ {t} \mid \mathcal {F} _ {t - 1} \right]} + \frac {\lambda}{2} \mathbb {E} _ {Z _ {t}} \left[ \chi_ {t} \mid \mathcal {F} _ {t - 1} \right]. \tag {62} \\ \end{array}
$$
Now computing the probability
$$
\mathbb {E} _ {Z _ {t}} \left[ \chi_ {t} \mid \mathcal {F} _ {t - 1} \right] = \mathbb {P} [ \| \nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right) - \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \mid \mathcal {F} _ {t - 1} \| \geq \frac {\lambda}{2} ],
$$
$$
\begin{array}{l} = \mathbb {P} [ \| \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) - \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| ^ {2} \geq \frac {\lambda^ {2}}{4} | \mathcal {F} _ {t - 1} ], \\ \leq \frac {4 \sigma^ {2}}{\lambda^ {2}}. \tag {63} \\ \end{array}
$$
The last inequality is from Chebychev's inequality. Substituting Equation (63) into Equation (62) yields the result.
Proof of Equation (51)