SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] = \mathbb {E} \left[ P _ {t} - U _ {t} \mid \mathcal {F} _ {t} \right] \leq \mathbb {E} \left[ | P _ {t} - U _ {t} | \mid \mathcal {F} _ {t} \right] \\ = \mathbb {E} \left[ | P _ {t} - U _ {t} | \left(\mathbf {1} [ | U _ {t} | \leq (A + 2 \mu_ {1}) R C _ {4} ] \right. \right. \\ \left. + \sum_ {k = 2} ^ {\infty} \mathbf {1} \left[ (k - 1) A R C _ {4} + 2 \mu_ {1} R C _ {4} \leq | U _ {t} | \leq k A R C _ {4} + 2 \mu_ {1} R C _ {4} \right]\right) \Bigg ] \\ = \mathbb {E} \left[ \sum_ {k = 2} ^ {\infty} \left| P _ {t} - U _ {t} \right| \mathbf {1} \left[ (k - 1) A R C _ {4} + 2 \mu_ {1} R C _ {4} \leq \left| U _ {t} \right| \leq k A R C _ {4} + 2 \mu_ {1} R C _ {4} \right] \right] \\ \leq \sum_ {k = 2} ^ {\infty} \left(k A R C _ {4} + 2 \mu_ {1} R C _ {4} - (A + 2 \mu_ {1}) R C _ {4}\right) R C _ {4} \mathbb {E} \left[ \mathbf {1} \left[ | U _ {t} | \geq (k - 1) A R C _ {4} + 2 \mu_ {1} R C _ {4} \right] \right] \\ \leq \sum_ {k = 1} ^ {\infty} k A R C _ {4} \mathbb {E} \left[ \mathbf {1} \left[ \left(Q _ {t} + \mu_ {1}\right) R C _ {4} \geq k A R C _ {4} + 2 \mu_ {1} R C _ {4} \right] \right] \quad (\text {d u e t o 3}) \\ = \sum_ {k = 1} ^ {\infty} k A R C _ {4} \Pr \left[ Q _ {t} \geq k A + \mu_ {1} \right] \leq A R C _ {4} \sum_ {k = 1} ^ {\infty} k \Pr \left[ \left| Q _ {t} - \mu_ {1} \right| ^ {2 p} \geq (k A) ^ {2 p} \right] \\ \leq A R C _ {4} \sum_ {k = 1} ^ {\infty} \frac {M k}{k ^ {2 p} A ^ {2 p}} = A ^ {1 - 2 p} R C _ {4} M \sum_ {k = 1} ^ {\infty} k ^ {1 - 2 p} \leq 2 A ^ {1 - 2 p} R C _ {4} M \\ \end{array}
$$
where the last inequality is because  $p \geq 2$ . We obtain
$$
\sum_ {t = 0} ^ {k} \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] \leq 2 A ^ {1 - 2 p} R C _ {4} M T
$$
The term  $\sum_{t=0}^{k} B_{t} \leq \sum_{t=0}^{k} |B_{t}| \leq \sum_{t=0}^{T-1} |B_{t}|$  will be bounded by Markov inequality. From the above deduction,
$$
\mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} | B _ {t} | \right] = \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ | B _ {t} | \right] = \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \mathbb {E} \left[ | U _ {t} - P _ {t} | \mid \mathcal {F} _ {t} \right] \right] \leq 2 A ^ {1 - 2 p} R C _ {4} M T
$$
With probability at least  $1 - \delta$
$$
\sum_ {t = 0} ^ {T - 1} \left| B _ {t} \right| \leq 2 T A ^ {1 - 2 p} R C _ {4} M \frac {1}{\delta} = 2 R C _ {4} A ^ {1 - 2 p} \left(\frac {M T}{\delta}\right)
$$
Finally, we will use Freedman's inequality to bound the remaining term  $\sum_{t=0}^{k}\left(P_{t}-\mathbb{E}\left[P_{t} \mid \mathcal{F}_{t}\right]\right)$ . First, notice that
$$
\begin{array}{l} \mathbb {E} \left[ \left| P _ {t} - \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] \right| ^ {2} \mid \mathcal {F} _ {t} \right] \leq \mathbb {E} \left[ P _ {t} ^ {2} \mid \mathcal {F} _ {t} \right] \\ \leq \mathbb {E} \left[ U _ {t} ^ {2} \mid \mathcal {F} _ {t} \right] \leq \mathbb {E} \left[ \left(\ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(v _ {t}\right)\right) ^ {2} \mid \mathcal {F} _ {t} \right] \\ \leq R ^ {2} C _ {4} ^ {2} \mathbb {E} \left[ Q _ {t} ^ {2} \right] \leq R ^ {2} C _ {4} ^ {2} \mu_ {2}. \\ \end{array}
$$
We have  $(P_{t} - \mathbb{E}[P_{t}\mid \mathcal{F}_{t}])$  is a martingale difference sequence with  $|P_t - \mathbb{E}[P_t\mid \mathcal{F}_t]|\leq 2(A + 2\mu_1)RC_4$ . We can apply Freedman's inequality,
$$
\begin{array}{l} \Pr \left[ \exists k \geq 0: \left| \sum_ {t = 0} ^ {k} P _ {t} - \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] \right| > a \text {a n d} \sum_ {t = 0} ^ {k} \mathbb {E} \left[ \left| P _ {t} - \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] \right| ^ {2} \mid \mathcal {F} _ {t} \right] \leq F \right] \\ \leq 2 \exp \left(\frac {- 2 a ^ {2}}{2 F + 4 (A + 2 \mu_ {1}) R C _ {4} a / 3}\right) \\ \end{array}
$$
If we select
$$
F = T \mu_ {2} R ^ {2} C _ {4} ^ {2}
$$
$$
\text {a n d} a = \frac {2}{3} \log \frac {2}{\delta} (A + 2 \mu_ {1}) R C _ {4} + R C _ {4} \sqrt {\log \frac {2}{\delta} T \mu_ {2}}
$$
we obtain with probability at least  $1 - \delta$ , for all  $k \geq 0$
$$
\sum_ {t = 0} ^ {k} P _ {t} - \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] \leq \frac {2}{3} \log \frac {2}{\delta} (A + 2 \mu_ {1}) R C _ {4} + R C _ {4} \sqrt {\log \frac {2}{\delta} T \mu_ {2}}
$$
Therefore with probability at least  $1 - 3\delta$  we have the following event  $E:$  for all  $k\geq 0$
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} U _ {t} \leq \frac {2}{3} \log \frac {2}{\delta} (A + 2 \mu_ {1}) R C _ {4} + R C _ {4} \sqrt {\log \frac {2}{\delta} T \mu_ {2}} + 4 R C _ {4} A ^ {1 - 2 p} \left(\frac {M T}{\delta}\right) \\ \leq \frac {2}{3} \gamma \left(7 \left(\frac {M T}{\delta}\right) ^ {1 / 2 p} + 2 \mu_ {1}\right) R C _ {4} + R C _ {4} \sqrt {\log \frac {2}{\delta} T \mu_ {2}} \\ \end{array}
$$
$$
\text {a n d} \sum_ {t = 1} ^ {k + 1} Q _ {t} ^ {2} \leq T \mu_ {2} + 2 M \sqrt {T} \left(\frac {2}{\delta}\right) ^ {\frac {1}{p}}.
$$
where we denote  $\gamma = \max \left\{1, \log \frac{2}{\delta}\right\}$ . Furthermore
$$
\begin{array}{l} \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} (v _ {t}) \| _ {*} ^ {2} \leq \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} Q _ {t + 1} ^ {2} \left(C _ {1} + C _ {2} \left(1 + \| v _ {t} \|\right)\right) ^ {2} \\ \leq \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} Q _ {t + 1} ^ {2} \left(C _ {4} + C _ {2} \| v _ {t} - w _ {\mathrm {r e f}} \|\right) ^ {2} \\ \leq \eta^ {2} C _ {4} ^ {2} \sum_ {t = 1} ^ {k + 1} Q _ {t} ^ {2} + \eta^ {2} C _ {2} ^ {2} \sum_ {t = 0} ^ {k} Q _ {t + 1} ^ {2} \| v _ {t} - w _ {\mathrm {r e f}} \| ^ {2} \\ \leq \eta^ {2} \left(C _ {4} ^ {2} + C _ {2} ^ {2} R ^ {2}\right) \left(T \mu_ {2} + 2 M \sqrt {T} \left(\frac {2}{\delta}\right) ^ {\frac {1}{p}}\right) \\ \end{array}
$$
Now we will proceed by induction to show that conditioned on the event  $E$ ,  $w_{t} = v_{t}$ . For the base case, we have  $w_{0} = v_{0}$ . Suppose that we have  $w_{t} = v_{t}$  for all  $t \leq k$ . We will show that  $w_{k + 1} = v_{k + 1}$ . From Lemma 2, we have
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq D _ {0} + \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {\text {r e f}}\right) - \mathcal {R} ^ {*}\right) \\ + \eta \sum_ {t = 0} ^ {k} \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2} \\ \leq D _ {0} + \eta \sqrt {T} D _ {0} + \eta \sum_ {t = 0} ^ {k} U _ {t} + \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} (v _ {t}) \| _ {*} ^ {2} \\ \leq D _ {0} \left(1 + \eta \sqrt {T}\right) \\ + \left(\frac {2}{3} \gamma \left(7 \left(\frac {M T}{\delta}\right) ^ {1 / 2 p} + 2 \mu_ {1}\right) + \sqrt {\log \frac {2}{\delta} T \mu_ {2}}\right) \eta R C _ {4} \\ + \eta^ {2} \left(C _ {4} ^ {2} + C _ {2} ^ {2} R ^ {2}\right) \left(T \mu_ {2} + 2 M \sqrt {T} \left(\frac {2}{\delta}\right) ^ {\frac {1}{p}}\right) \\ \leq \frac {R ^ {2}}{2} \\ \end{array}
$$
Thus  $\| w_{k + 1} - w_{\mathrm{ref}} \| \leq R$ . And thus  $w_{k + 1} = v_{k + 1}$ . Finally, we can conclude that with probability at least  $1 - 3\delta$ , for all  $0 \leq k \leq T - 1$
$$
\frac {1}{k + 1} \sum_ {t = 0} ^ {k} (\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}})) + \frac {\mathbf {D} _ {\psi} (w _ {\mathrm {r e f}} ; w _ {k + 1})}{\eta (k + 1)} \leq \frac {R ^ {2}}{2 \eta (k + 1)}.
$$
Proof of Theorem 13. For  $0 \leq j \leq \tau - 1$ , we define
$$
\begin{array}{l} Z _ {i} ^ {j} = z _ {\tau i + j} \eta \left(\mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\text {r e f}}\right) \mid \mathcal {F} _ {\tau i + j} \right] - \mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right) \mid \mathcal {F} _ {\tau i + j} \right]\right) \\ + z _ {\tau i + j} \eta \left(\ell_ {\tau (i + 1) + j} \left(w _ {\mathrm {r e f}}\right) - \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right)\right) - 8 \left(z _ {\tau i + j} \eta\right) ^ {2} C _ {4} ^ {2} \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {\tau i + j}\right), \\ \forall 0 \leq i \leq \frac {T - 1 - j}{\tau} \\ \end{array}
$$
$$
S _ {k} ^ {j} = \sum_ {i = 0} ^ {k} Z _ {i} ^ {j}, \forall 0 \leq k \leq \frac {T - 1 - j}{\tau}
$$
$$
z _ {t} = \frac {1}{8 \eta^ {2} C _ {4} ^ {2} (T + 1 + t)} \quad \forall - 1 \leq t \leq T - 1
$$
We bound
$$
\begin{array}{l} \left| \mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\text {r e f}}\right) \mid \mathcal {F} _ {\tau i + j} \right] - \mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right) \mid \mathcal {F} _ {\tau i + j} \right] \right. \\ + \ell_ {\tau (i + 1) + j} \left(w _ {\text {r e f}}\right) - \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right) \\ \leq 2 C _ {4} \left\| w _ {\text {r e f}} - w _ {\tau i + j} \right\| \\ \end{array}
$$
By Lemma 15
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {i} ^ {j}\right) \mid \mathcal {F} _ {\tau i + j} \right] \\ = \exp \left(- 8 \left(z _ {\tau i + j} \eta\right) ^ {2} C _ {4} ^ {2} \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {\tau i + j}\right)\right) \\ \times \mathbb {E} \left[ \right. \exp \left(z _ {\tau i + j} \eta \left(\mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\text {r e f}}\right) \mid \mathcal {F} _ {\tau i + j} \right] - \mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right) \mid \mathcal {F} _ {\tau i + j} \right]\right.\right. \\ \left. \left. + \ell_ {\tau (i + 1) + j} \left(w _ {\text {r e f}}\right) - \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right)\right)\right) \mid \mathcal {F} _ {\tau i + j} \Bigg ] \\ \leq \exp \left(- 8 (z _ {\tau i + j} \eta) ^ {2} C _ {4} ^ {2} \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {\tau i + j})\right) \exp \left(4 (z _ {\tau i + j} \eta) ^ {2} C _ {4} ^ {2} \| w _ {\mathrm {r e f}} - w _ {\tau i + j} \| ^ {2}\right) \leq 1 \\ \end{array}
$$
Therefore  $\mathbb{E}\left[\exp \left(Z_i^j\right)\mid \mathcal{F}_{\tau i + j}\right]\leq 1$  and hence  $\left(\exp \left(S_k^j\right)\right)_{k\geq 0}$  is a supermartingale. By Ville's inequality, we have with probability at least  $1 - \delta$ , for all  $0\le k\le \kappa$
$$
\sum_ {i = 0} ^ {k} Z _ {i} ^ {j} \leq \log \frac {1}{\delta}
$$
By union bound over  $j = 0, \dots, \tau - 1$ , and with probability at least  $1 - \tau \delta$  we have for all  $0 \leq k \leq T - \tau - 1$
$$
\sum_ {t = 0} ^ {k} z _ {t} \eta \left(\mathbb {E} \left[ \ell_ {t + \tau} \left(w _ {\text {r e f}}\right) \mid \mathcal {F} _ {t} \right] - \mathbb {E} \left[ \ell_ {t + \tau} \left(w _ {t}\right) \mid \mathcal {F} _ {t} \right] + \ell_ {t + \tau} \left(w _ {\text {r e f}}\right) - \ell_ {t + \tau} \left(w _ {t}\right)\right)
$$
$$
\leq \sum_ {t = 0} ^ {k} 8 (z _ {t} \eta) ^ {2} C _ {4} ^ {2} \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t}) + \tau \log \frac {1}{\delta}
$$
We will proceed to prove by induction that  $\mathbf{D}_{\psi}(w_{\mathrm{ref}};w_t)\leq \frac{1}{2} R^2$
For the base case  $t = 0$ , this holds trivially. Suppose that this is true for all  $0 \leq t \leq k$ , we now show for  $t = k + 1$ .
If  $k\leq \tau -1$
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq D _ {0} + \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {\text {r e f}}\right) - \mathcal {R} ^ {*}\right) \\ + \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \eta | \mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) + \ell_ {t + 1} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {t}) | \\ \leq \sum_ {t = 0} ^ {k} 2 \eta C _ {4} \| w _ {\text {r e f}} - w _ {t} \| \leq 2 \eta C _ {4} R (k + 1) \leq 2 \eta C _ {4} R \tau \\ \end{array}
$$
$$
\begin{array}{l} \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2} \leq \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \left(C _ {1} + C _ {2} \left(1 + \| w _ {t} \|\right)\right) ^ {2} \\ \leq \frac {\eta^ {2}}{2} \sum_ {t = 0} ^ {k} \left(C _ {4} + C _ {2} \| w _ {t} - w _ {\text {r e f}} \|\right) ^ {2} \\ \leq \eta^ {2} C _ {4} ^ {2} (k + 1) + \eta^ {2} C _ {2} ^ {2} R ^ {2} (k + 1) \\ \leq \tau \eta^ {2} \left(C _ {4} ^ {2} + C _ {2} ^ {2} R ^ {2}\right) \\ \end{array}
$$
$$
\mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \leq D _ {0} + \eta D _ {0} \sqrt {T} + 2 \eta C _ {4} R \tau + \tau \eta^ {2} \left(C _ {4} ^ {2} + C _ {2} ^ {2} R ^ {2}\right) \leq \frac {R ^ {2}}{2}.
$$
If  $k\geq \tau$
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} ^ {*}\right) + z _ {k} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) - z _ {- 1} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {0}\right) \\ \leq \sum_ {t = 0} ^ {k} z _ {t} \eta \left(\mathcal {R} \left(w _ {\mathrm {r e f}}\right) - \mathcal {R} ^ {*}\right) + \sum_ {t = 0} ^ {k} z _ {t} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\mathrm {r e f}}\right)\right) \\ + \sum_ {t = 0} ^ {k} z _ {t} \eta \left(\ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \sum_ {t = 0} ^ {k} \frac {z _ {t} \eta^ {2}}{2} \| g _ {t + 1} \| _ {*} ^ {2} + \sum_ {t = 0} ^ {k} \left(z _ {t} - z _ {t - 1}\right) \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) \\ \leq \sum_ {t = 0} ^ {k} z _ {t} \eta \left(\mathcal {R} \left(w _ {\text {r e f}}\right) - \mathcal {R} ^ {*}\right) + \sum_ {t = k - \tau + 1} ^ {k} z _ {t} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right) \\ + \sum_ {t = 0} ^ {k - \tau} z _ {t} \eta (\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) - \mathbb {E} [ \ell_ {t + \tau} (w _ {\text {r e f}}) | \mathcal {F} _ {t} ] + \mathbb {E} [ \ell_ {t + \tau} (w _ {t}) | \mathcal {F} _ {t} ]) \\ \end{array}
$$
where in the last inequality we use  $z_{t} = \frac{1}{8\eta^{2}C_{4}^{2}(T + 1 + t)}$  to see that  $z_{t} + 8(z_{t}\eta)^{2}C_{4}^{2}\leq z_{t - 1}$ . Notice that,  $\frac{z_t}{z_k}\leq 2$ , and  $(\mathcal{R}(w_{\mathrm{ref}}) - \mathcal{R}^*)\leq \frac{D_0}{\sqrt{T}}$  we have
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq 2D_{0} + 2\eta D_{0}\sqrt{T} +16\eta^{2}C_{4}^{2}T\tau \log \frac{1}{\delta} +\underbrace{2\eta\sum_{t = k - \tau + 1}^{k}|\mathcal{R}(w_{t}) - \mathcal{R}(w_{\mathrm{ref}})|}_{A} \\ + \underbrace {2 \eta \sum_ {t = 0} ^ {k - \tau} | \mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) - \mathbb {E} [ \ell_ {t + \tau} (w _ {\text {r e f}}) \mid \mathcal {F} _ {t} ] + \mathbb {E} [ \ell_ {t + \tau} (w _ {t}) \mid \mathcal {F} _ {t} ] |} _ {B} \\ \end{array}
$$
$$
\begin{array}{l} + \underbrace {\eta^ {2} \sum_ {t = 0} ^ {k} \left\| g _ {t + 1} \right\| _ {*} ^ {2}} _ {C} + \underbrace {2 \eta \sum_ {t = 0} ^ {k - \tau} \left| \ell_ {t + \tau} \left(w _ {t}\right) - \ell_ {t + \tau} \left(w _ {t + \tau - 1}\right) \right|} _ {D} \\ + \underbrace {\frac {2 (\tau - 1) \eta}{T} \sum_ {t = \tau - 1} ^ {k - 1} | \ell_ {t + 1} (w _ {t}) - \ell_ {t + 1} (w _ {\mathrm {r e f}}) | + 2 \eta \sum_ {t = 0} ^ {\tau - 2} | \ell_ {t + 1} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {t}) | + 2 \eta | \ell_ {k + 1} (w _ {\mathrm {r e f}}) - \ell_ {k + 1} (w _ {k}) |} _ {E} \\ \end{array}
$$
Now we bound each term. For  $A$
$$
A = 2 \eta \sum_ {t = k - \tau + 1} ^ {k} | \mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) | \leq 2 \eta \sum_ {t = k - \tau + 1} ^ {k} C _ {4} \| w _ {\mathrm {r e f}} - w _ {t} \| \leq 2 \eta \tau C _ {4} R
$$
For  $B$ , by Assumption 3,  $\sup_{t\in \mathbb{Z}_{\geq 0}}\sup_{\mathcal{F}_t}\mathrm{TV}\left(P_t^{t + \tau},\pi\right)\leq \epsilon$
$$
2 \eta | \mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) - \mathbb {E} \left[ \ell_ {t + \tau} (w _ {\text {r e f}}) \mid \mathcal {F} _ {t} \right] + \mathbb {E} \left[ \ell_ {t + \tau} (w _ {t}) \mid \mathcal {F} _ {t} \right] | \leq 2 \eta C _ {4} R \epsilon
$$
$$
B = 2 \eta \sum_ {t = 0} ^ {k - \tau} | \mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) - \mathbb {E} [ \ell_ {t + \tau} (w _ {\mathrm {r e f}}) \mid \mathcal {F} _ {t} ] + \mathbb {E} [ \ell_ {t + \tau} (w _ {t}) \mid \mathcal {F} _ {t} ] | \leq 2 \eta C _ {4} R \epsilon T
$$
For  $C$ , similarly to before
$$
C = \eta^ {2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2} \leq 2 T \eta^ {2} \left(C _ {4} ^ {2} + C _ {2} ^ {2} R ^ {2}\right)
$$
For  $D$ , we have
$$
\begin{array}{l} \left| \ell_ {t + \tau} (w _ {t}) - \ell_ {t + \tau} (w _ {t + \tau - 1}) \right| \\ \leq \sum_ {i = t + 1} ^ {t + \tau - 1} \left| \ell_ {t + \tau} (w _ {i}) - \ell_ {t + \tau} (w _ {i - 1}) \right| \\ \leq \sum_ {i = t + 1} ^ {t + \tau - 1} \| w _ {i} - w _ {i - 1} \| \left(C _ {1} + C _ {2} \left(1 + \| w _ {i} \|\right)\right) \\ \leq \sum_ {i = t + 1} ^ {t + \tau - 1} \eta \left\| \nabla \ell_ {i} \left(w _ {i - 1}\right) \right\| \left(C _ {4} + C _ {2} \left\| w _ {i} - w _ {\mathrm {r e f}} \right\|\right) \\ \leq \eta \left(C _ {4} + C _ {2} R\right) \sum_ {i = t + 1} ^ {t + \tau - 1} \left(C _ {4} + C _ {2} \left\| w _ {i - 1} - w _ {\mathrm {r e f}} \right\|\right) \\ \leq \eta \left(C _ {4} + C _ {2} R\right) ^ {2} \tau \leq \eta \tau \left(2 C _ {4} ^ {2} + 2 C _ {2} ^ {2} R ^ {2}\right) \\ \end{array}
$$
$$
D = 2 \eta \sum_ {t = 0} ^ {k - \tau} \left| \ell_ {t + \tau} \left(w _ {t}\right) - \ell_ {t + \tau} \left(w _ {t + \tau - 1}\right) \right| \leq 2 \eta^ {2} \tau T \left(2 C _ {4} ^ {2} + 2 C _ {2} ^ {2} R ^ {2}\right)
$$
For  $E$ , since
$$
\left| \ell_ {t + 1} \left(w _ {t}\right) - \ell_ {t + 1} \left(w _ {\text {r e f}}\right) \right| \leq C _ {4} R
$$
$$
\begin{array}{l} E = \frac {2 (\tau - 1) \eta}{T} \sum_ {t = \tau - 1} ^ {k - 1} | \ell_ {t + 1} (w _ {t}) - \ell_ {t + 1} (w _ {\mathrm {r e f}}) | \\ + 2 \eta \sum_ {t = 0} ^ {\tau - 2} \left| \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right) \right| + 2 \eta \left| \ell_ {k + 1} \left(w _ {\text {r e f}}\right) - \ell_ {k + 1} \left(w _ {k}\right) \right| \\ \end{array}
$$
$$
\leq 2 \eta C _ {4} R \left(\frac {(\tau - 1) (k - \tau + 1)}{T} + \tau\right) \leq 4 \eta \tau C _ {4} R
$$
Sum up we have
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \eta \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq 2 D _ {0} + 2 \eta D _ {0} \sqrt {T} + 1 6 \eta^ {2} C _ {4} ^ {2} T \tau \log \frac {1}{\delta} \\ + 2 \eta \tau C _ {4} R + 2 \eta C _ {4} R e T + 2 T \eta^ {2} \left(C _ {4} ^ {2} + C _ {2} ^ {2} R ^ {2}\right) \\ + 2 \eta^ {2} \tau T \left(2 C _ {4} ^ {2} + 2 C _ {2} ^ {2} R ^ {2}\right) + 4 \eta \tau C _ {4} R \\ \leq \frac {R ^ {2}}{2} \\ \end{array}
$$
as needed. Finally we have