SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t} \right] \times \exp \left(\frac {3}{1 6} z _ {t} \eta \left(\mathcal {R} \left(w _ {\mathrm {r e f}}\right) + \mathcal {R} \left(w _ {t}\right)\right)\right) \\ \leq \exp \left(\frac {3}{4} \frac {1}{4} z _ {t} ^ {2} \eta^ {2} \mathbb {E} \left[ \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) ^ {2} \mid \mathcal {F} _ {t} \right]\right) \\ \leq \exp \left(\frac {3}{1 6} z _ {t} ^ {2} \eta^ {2} \mathbb {E} \left[ \left(\ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) ^ {2} \mid \mathcal {F} _ {t} \right]\right) \\ \leq \exp \left(\frac {3}{1 6} z _ {t} ^ {2} \eta^ {2} C _ {4} \| w _ {\text {r e f}} - w _ {t} \| \mathbb {E} \left[ \ell_ {t + 1} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {t}\right) \mid \mathcal {F} _ {t} \right]\right) \\ \leq \exp \left(\frac {3}{1 6} z _ {t} \eta \left(\mathcal {R} (w _ {\mathrm {r e f}}) + \mathcal {R} (w _ {t})\right)\right) \\ \end{array}
$$
Therefore  $\mathbb{E}\left[\exp (Z_t)\mid \mathcal{F}_t\right]\leq 1$  and hence  $(\exp (S_{t}))_{t\geq 0}$  is a supermartingale. By Ville's inequality, we have with probability at least  $1 - \delta$ , for all  $0\le k\le T - 1$
$$
\sum_ {t = 0} ^ {k} Z _ {t} \leq \log \frac {1}{\delta}
$$
Expanding this inequality, in combination with Lemma 5, we obtain the conclusion.
Remark 6. The new analysis does not change the conclusion observed in [35]—that is, with high probability, the iterate sequence  $(w_{t})_{t\geq 0}$  behaves as if the domain of the problem is bounded. We improve the probability that this event happens.
3.2 Non-realizable case
In the non-realizable case, we do not aim for  $1 / T$  but only  $1 / \sqrt{T}$  rates. Hence we do not assume that the comparator  $w_{\mathrm{ref}}$  satisfies  $\mathcal{R}(w_{\mathrm{ref}})\leq \rho \mathbf{D}_{\psi}(w_{\mathrm{ref}};w_0) / T$  but rather the following assumption on the excess risk:
Assumption 4. Let  $\mathcal{R}^* = \inf_{v\in \mathcal{X}}\mathcal{R}(v)$ , assume that  $\mathcal{R}(w_{\mathrm{ref}}) - \mathcal{R}^*\leq \frac{\mathbf{D}_{\psi}(w_{\mathrm{ref}};w_0)}{\sqrt{T}}$
We also relax the assumption on the data samples. In the previous case, the data are bounded, i.e  $\{\| x\|_{*},|y|\} \leq 1$  a.s. We will consider in this section two settings, one when the data come from a sub-Gaussian distribution and one when the data distribution has a polynomial tail.
Remark on Theorem 10 in [35]. There seems to be an issue with the proof of Theorem 10 in [35]. The proof uses a variant of Azuma's inequality ([37], Problem 3.11) which allows the ranges of the random variables to not be specified up front. However, when bounding the range of relevant random variables, the proof uses  $Z_{t + 1}$ , which is correlated with the data  $\| x_{t + 1}\| _*$  and  $y_{t + 1}$ . Thus the condition of Azuma's inequality is not satisfied. We do not see an immediate way to resolve this problem. In the following, we consider separately the two cases of sub-Gaussian and polynomial
tailed data for which we use different proof techniques to show the error bounds. In the first case of IID data with sub-Gaussian tails, we proceed by bounding the moment generating function of a well-chosen martingale sequence. In the second case of IID data with polynomial tails, we introduce a truncation technique. In both cases, we are able to obtain better bounds compared with [35].
3.2.1 IID data with sub-Gaussian tails
We will show the following guarantee:
Theorem 7. Suppose  $\ell$  is convex,  $(C_1,C_2)$ -quadratically-bounded. Given  $T$ ,  $((x_{t},y_{t}))_{t\leq T}$  are IID samples with  $Q_{t} = \max \left\{1,\| x_{t}\|_{*}^{2},|y_{t}|^{2}\right\}$  and there exists  $\sigma \geq 0$  such that for all  $\lambda$
$$
\left. \right. \max  \left\{\mathbb {E} \left[ \exp \left(\lambda \left(Q _ {t} ^ {2} - \mathbb {E} \left[ Q _ {t} ^ {2} \right]\right)\right)\right], \mathbb {E} [ \exp (\lambda (Q _ {t} - \mathbb {E} [ Q _ {t} ])) ] \right\} \leq \exp \left(\lambda^ {2} \sigma^ {2}\right)
$$
Let  $\mu_{1} = \mathbb{E}\left[Q_{t}\right]$  and  $\mu_{2} = \mathbb{E}\left[Q_{t}^{2}\right]$ . Suppose that  $w_{\mathrm{ref}}$  satisfies Assumption 4. Then for  $\eta \leq \frac{1}{4C_2\sqrt{T\mu_2 + 2\sigma\sqrt{T\log\frac{1}{\delta}}}}$ , with probability at least  $1 - 2\delta$ , for every  $0 \leq k \leq T - 1$
$$
\frac {1}{k + 1} \sum_ {t = 0} ^ {k} (\mathcal {R} (w _ {t}) - \mathcal {R} ^ {*}) + \frac {\mathbf {D} _ {\psi} (w _ {\mathrm {r e f}} ; w _ {k + 1})}{\eta (k + 1)} \leq \frac {R ^ {2}}{\eta (k + 1)}
$$
where  $R^2 = 16C_4^2\left(\sigma^2 +4\mu_1^2\right)\log \frac{1}{\delta}\eta^2 T + 4D_0(1 + \eta \sqrt{T}) + 4\eta^2 C_4^2\left(T\mu_2 + 2\sigma \sqrt{T\log\frac{1}{\delta}}\right) = O(1).$
Remark 8. For zero-mean sub-Gaussian variable  $X$ , the definition  $\mathbb{E}\left[\exp (\lambda X)\right]\leq \exp \left(\lambda^2\sigma^2\right)$  for all  $\lambda$  is equivalent to  $\mathbb{E}\left[\exp \left(\lambda^2 X^2\right)\right]\leq \exp \left(\lambda^2\sigma^2\right)$  for all  $0\leq \lambda \leq \frac{1}{\sigma}$  (see [38]). The lemma below shows a property of sub-Gaussian variables under scaling and translating. First let us consider  $\sum_{t = 1}^{T}Q_{t}^{2}$ . Similar to Lemma 5, by bounding the moment generating function of this term, we have the following (see also Section B4 in [35]).
Lemma 9. With probability at least  $1 - \delta$ ,  $\sum_{t=1}^{T} Q_t^2 \leq T \mu_2 + 2\sigma \sqrt{T \log \frac{1}{\delta}}$ .
Proof of Theorem 7. The proof of this Theorem uses the technique developed in [21]. We will also analyze the moment generating function of a suitable martingale sequence. However, the choice of the coefficients will differ significantly from the previous proof. In this case the structure of the problem is deeply integrated into the analysis of the martingale. We define
$$
\begin{array}{l} Z _ {t} = z _ {t} \eta (\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}})) + z _ {t} (\mathbf {D} _ {\psi} (w _ {\text {r e f}}; w _ {t + 1}) - \mathbf {D} _ {\psi} (w _ {\text {r e f}}; w _ {t})) \\ - \frac {1}{2} z _ {t} \eta^ {2} \| g _ {t + 1} \| _ {*} ^ {2} - 4 z _ {t} ^ {2} \eta^ {2} C _ {4} ^ {2} (\sigma^ {2} + 4 \mu_ {1} ^ {2}) \mathbf {D} _ {\psi} (w _ {\text {r e f}}; w _ {t}) \quad \forall 0 \leq t \leq T - 1 \\ \end{array}
$$
where  $z_{t} = \frac{1}{4\eta^{2}C_{4}^{2}\left(\sigma^{2} + 4\mu_{1}^{2}\right)(T + t + 1)}$
and let  $S_{t} = \sum_{i=0}^{t} Z_{i}$ ;  $\forall 0 \leq t \leq T - 1$ . By Lemma 2, we have
$$
Z _ {t} + 4 z _ {t} ^ {2} \eta^ {2} C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {t}\right) \leq z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} \left(w _ {\mathrm {r e f}}\right) + \ell_ {t + 1} \left(w _ {\mathrm {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right)
$$
where we have  $\mathbb{E}\left[(\mathcal{R}(w_t) - \mathcal{R}(w_{\mathrm{ref}}) + \ell_{t + 1}(w_{\mathrm{ref}}) - \ell_{t + 1}(w_t))\right] = 0$  and using the same notation  $C_4 = C_1 + C_2(1 + \| w_{\mathrm{ref}}\|)$ , by Lemma 1,
$$
\begin{array}{l} \left. \left| \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) \right| \right. \\ \leq \left| \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right) \right| + \left| \mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) \right| \\ \leq \left| \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right) \right| + \mathbb {E} \left[ \left| \ell_ {x, y} \left(w _ {t}\right) - \ell_ {x, y} \left(w _ {\text {r e f}}\right) \right| \right] \\ \leq \left(Q _ {t} + \mu_ {1}\right) \| w _ {\text {r e f}} - w _ {t} \| C _ {4} = \left(\left(Q _ {t} - \mu_ {1}\right) + 2 \mu_ {1}\right) \| w _ {\text {r e f}} - w _ {t} \| C _ {4} \\ \end{array}
$$
Hence applying Lemma 15, we have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t} \right] \exp \left(4 z _ {t} ^ {2} \eta^ {2} C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right)\right) \\ = \mathbb {E} \left[ \exp \left(z _ {t} \eta_ {t} (\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) + \ell_ {t + 1} (w _ {\text {r e f}}) - \ell_ {t + 1} (w _ {t}))\right) \mid \mathcal {F} _ {t} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \exp \left(2 z _ {t} ^ {2} \eta^ {2} C _ {4} ^ {2} \| w _ {\text {r e f}} - w _ {t} \| ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right)\right) \\ \leq \exp \left(4 z _ {t} ^ {2} \eta^ {2} C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right)\right) \\ \end{array}
$$
Expanding this inequality we have
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} z _ {t} \eta \mathcal {R} (w _ {t}) + z _ {k} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq \log \frac {1}{\delta} + z _ {- 1} D _ {0} + \eta \mathcal {R} (w _ {\mathrm {r e f}}) \sum_ {t = 0} ^ {k} z _ {t} + \frac {1}{2} \sum_ {t = 0} ^ {k} z _ {t} \eta^ {2} \| g _ {t + 1} \| _ {*} ^ {2} \\ + \sum_ {t = 0} ^ {k} \underbrace {\left(z _ {t} + 4 z _ {t} ^ {2} \eta^ {2} C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) - z _ {t - 1}\right)} _ {\leq 0} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) \\ \stackrel {(a)} {\leq} \log \frac {1}{\delta} + z _ {- 1} D _ {0} + \eta \mathcal {R} (w _ {\mathrm {r e f}}) \sum_ {t = 0} ^ {k} z _ {t} + \frac {1}{2} \sum_ {t = 0} ^ {k} z _ {t} \eta^ {2} \| g _ {t + 1} \| _ {*} ^ {2} \\ \end{array}
$$
where for  $(a)$ , by the choice of  $z_{t} = \frac{1}{4\eta^{2}C_{4}^{2}\left(\sigma^{2} + 4\mu_{1}^{2}\right)(T + 1 + t)}$  we have  $z_{t - 1} - z_{t} \geq 4z_{t}^{2}\eta^{2}C_{4}^{2}\left(\sigma^{2} + 4\mu_{1}^{2}\right)$ . We highlight that this is where the structure of the problem comes into play. That is, by setting appropriate coefficients, we can leverage gain in the distance in the martingale difference sequence  $((z_{t} - z_{t - 1})\mathbf{D}_{\psi}(w_{\mathrm{ref}};w_{t}))$  to cancel out the loss from bounding the moment generating function  $(4z_{t}^{2}\eta^{2}C_{4}^{2}\left(\sigma^{2} + 4\mu_{1}^{2}\right)\mathbf{D}_{\psi}(w_{\mathrm{ref}};w_{t}))$ . Another important property of the sequence  $(z_{t})$  is that it is a decreasing sequence and  $\frac{z_t}{z_k} \leq 2$  for all  $t,k$ . Hence we have
$$
\begin{array}{l} \eta \sum_ {t = 0} ^ {k} \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq 4 C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \log \frac {1}{\delta} \eta^ {2} (T + 1 + k) + 2 D _ {0} + 2 \left(\mathcal {R} \left(w _ {\mathrm {r e f}}\right) - \mathcal {R} ^ {*}\right) \eta (k + 1) + \eta^ {2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2}. \\ \end{array}
$$
Combined with Lemma 9, with probability at least  $1 - 2\delta$ , for all  $0 \leq k \leq T - 1$
$$
\begin{array}{l} \eta \sum_ {t = 0} ^ {k} \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq 4 C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \log \frac {1}{\delta} \eta^ {2} (T + 1 + k) + 2 D _ {0} + 2 \left(\mathcal {R} \left(w _ {\mathrm {r e f}}\right) - \mathcal {R} ^ {*}\right) \eta (k + 1) + \eta^ {2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2}; \\ \end{array}
$$
$$
\text {a n d} \sum_ {t = 1} ^ {k + 1} Q _ {t} ^ {2} \leq T \mu_ {2} + 2 \sigma \sqrt {T \log \frac {1}{\delta}}
$$
Conditioned on this event, we will prove by induction that
$$
\begin{array}{l} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k}\right) \\ \leq R ^ {2} := 1 6 C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \log \frac {1}{\delta} \eta^ {2} T + 4 D _ {0} + 4 D _ {0} \eta \sqrt {T} + 4 \eta^ {2} C _ {4} ^ {2} \left(T \mu_ {2} + 2 \sigma \sqrt {T \log \frac {1}{\delta}}\right) \\ \end{array}
$$
For the base case  $k = 0$ , it is trivial. Suppose for all  $t \leq k$  we have  $\mathbf{D}_{\psi}(w_{\mathrm{ref}}; w_t) \leq R^2$ , now we prove for  $t = k + 1$ . By Lemma 1,
$$
\eta^ {2} \sum_ {t = 0} ^ {k} \| g _ {t + 1} \| _ {*} ^ {2} \leq \eta^ {2} \sum_ {t = 0} ^ {k} Q _ {t + 1} ^ {2} \left(C _ {1} + C _ {2} \left(1 + \| w _ {t} \|\right)\right) ^ {2} \leq \eta^ {2} \sum_ {t = 0} ^ {k} Q _ {t + 1} ^ {2} \left(C _ {4} + C _ {2} \| w _ {t} - w _ {\mathrm {r e f}} \|\right) ^ {2}
$$
$$
\begin{array}{l} \leq 2 \eta^ {2} C _ {4} ^ {2} \sum_ {t = 1} ^ {k + 1} Q _ {t} ^ {2} + 2 \eta^ {2} C _ {2} ^ {2} \sum_ {t = 0} ^ {k} Q _ {t + 1} ^ {2} \| w _ {t} - w _ {\mathrm {r e f}} \| ^ {2} \\ \leq \eta^ {2} \left(2 C _ {4} ^ {2} + 4 C _ {2} ^ {2} R ^ {2}\right) \left(T \mu_ {2} + 2 \sigma \sqrt {T \log \frac {1}{\delta}}\right) \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \leq 8 C _ {4} ^ {2} \left(\sigma^ {2} + 4 \mu_ {1} ^ {2}\right) \log \frac {1}{\delta} \eta^ {2} T + 2 D _ {0} + 2 \left(\mathcal {R} \left(w _ {\text {r e f}}\right) - \mathcal {R} ^ {*}\right) \eta (k + 1) \\ + \eta^ {2} \left(2 C _ {4} ^ {2} + 4 C _ {2} ^ {2} R ^ {2}\right) \left(T \mu_ {2} + 2 \sigma \sqrt {T \log \frac {1}{\delta}}\right) \\ \leq \frac {R ^ {2}}{2} + 4 \eta^ {2} C _ {2} ^ {2} \left(T \mu_ {2} + 2 \sigma \sqrt {T \log \frac {1}{\delta}}\right) R ^ {2} \leq R ^ {2}. \\ \end{array}
$$
Finally we obtain,  $\eta \sum_{t=0}^{k} (\mathcal{R}(w_t) - \mathcal{R}^*) + \mathbf{D}_{\psi}(w_{\mathrm{ref}}; w_{k+1}) \leq R^2$ , as needed.
3.2.2 IID data with polynomial tails
Theorem 10. Suppose  $\ell$  is convex,  $(C_1,C_2)$ -quadratically bounded. Given  $T$ ,  $((x_{t},y_{t}))_{t\leq T}$  are IID samples with  $Q_{t} = \max \left\{1,\| x_{t}\|_{*}^{2},|y_{t}|^{2}\right\}$  and for some  $p\geq 2$  there exists  $M\geq \frac{p}{e}$  such that for all  $\lambda$
$$
\left. \right. \max  \left\{\sup  _ {2 \leq r \leq 2 p} \left\{\mathbb {E} \left[\left| Q _ {t} - \mathbb {E} \left[ Q _ {t} \right]\right| ^ {r} \right]\right\}, \sup  _ {2 \leq r \leq p} \left\{\mathbb {E} \left[\left| Q _ {t} ^ {2} - \mathbb {E} \left[ Q _ {t} ^ {2} \right]\right| ^ {r} \right]\right\}\right\} \leq M
$$
Let  $\mu_{1} = \mathbb{E}\left[Q_{t}\right]$  and  $\mu_{2} = \mathbb{E}\left[Q_{t}^{2}\right]$ . Suppose that  $w_{\mathrm{ref}}$  satisfies Assumption 4. Then for  $\eta \leq C_2\sqrt{6\left(T\mu_2 + 2M\sqrt{T}\left(\frac{2}{\delta}\right)^{\frac{1}{P}}\right)}$ , with probability at least  $1 - 3\delta$ , for every  $0 \leq k \leq T - 1$
$$
\frac {1}{k + 1} \sum_ {t = 0} ^ {k} \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \frac {\mathbf {D} _ {\psi} \left(w _ {\text {r e f}} ; w _ {k + 1}\right)}{\eta (k + 1)} \leq \frac {R ^ {2}}{2 \eta (k + 1)}
$$
where  $R = \max \left\{\sqrt{6\left(D_0\left(1 + \eta\sqrt{T}\right) + \eta^2C_4^2\left(T\mu_2 + 2M\sqrt{T}\left(\frac{2}{\delta}\right)^{\frac{1}{p}}\right)}\right), 6\left(\frac{2}{3}\gamma \left(7\left(\frac{MT}{\delta}\right)^{1 / 2p} + 2\mu_1\right) + \sqrt{\log\frac{2}{\delta}T\mu_2}\eta C_4\right)\right\} = O(1), \gamma = \max \left\{1, \log \frac{2}{\delta}\right\}$ .
Remark 11. Since  $p \geq 2$ , the rate is  $O\left(\frac{1}{T^{1/2}} \log \frac{1}{\delta} + \frac{1}{T^{3/4}} \left( \frac{1}{\delta} \right)^{\frac{1}{2p}}\right)$ . This rate improves over the  $O\left(\left( \frac{1}{T^{1/2}} + \frac{1}{T^{3/4}} \left( \frac{T}{\delta} \right)^{\frac{1}{2p}} \right) \log \frac{T}{\delta}\right)$  rate by [35] by a polynomial factor  $T^{\frac{1}{2p}} \log \frac{T}{\delta}$  in the high probability regime where  $\delta = \frac{1}{\mathrm{poly}(T)}$ .
We will give a proof sketch for this theorem. The full proof is deferred to the appendix.
Proof Sketch. The heavy tailed distribution of the data does not allow us to analyze the moment generating function. In this case, we rely on the coupling technique as in [35]. Since it is not possible to apply Azuma's inequality due to the bounds on the variables being not measurable, and the variables are heavy tailed, we use truncation technique. We define,
$$
v _ {t} = \arg \min  _ {\| w - w _ {\text {r e f}} \| \leq R} \left\{\langle \eta_ {t} g _ {t} (v _ {t - 1}), w \rangle + \mathbf {D} _ {\psi} (w; v _ {t - 1}) \right\}
$$
where we use  $g_{t}(v_{t - 1})$  to denote the gradient at  $v_{t - 1}$  using the same data point  $(x_{t},y_{t})$  when computing  $w_{t}$  and we define
$$
U _ {t} = \left(\mathcal {R} (v _ {t}) - \mathcal {R} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(v _ {t}\right)\right)
$$
$$
P _ {t} = \left\{ \begin{array}{l l} U _ {t} & \text {i f} | U _ {t} | \leq (A + 2 \mu_ {1}) R C _ {4} \\ (A + 2 \mu_ {1}) R C _ {4} \text {s i g n} (U _ {t}) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\text {w h e r e} A = \left(\frac {M T}{\delta}\right) ^ {1 / 2 p} \text {a n d} B _ {t} = U _ {t} - P _ {t}.
$$
$$
\sum_ {t = 0} ^ {k} U _ {t} = \sum_ {t = 0} ^ {k} \left(P _ {t} - \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right]\right) + \sum_ {t = 0} ^ {k} \mathbb {E} \left[ P _ {t} \mid \mathcal {F} _ {t} \right] + \sum_ {t = 0} ^ {k} B _ {t}
$$
We bound  $\sum_{t=0}^{k}\left(P_{t}-\mathbb{E}\left[P_{t} \mid \mathcal{F}_{t}\right]\right)$  by applying Freedman's inequality. The terms  $\sum_{t=0}^{k}\mathbb{E}\left[P_{t} \mid \mathcal{F}_{t}\right]$  and  $\sum_{t=0}^{k}B_{t}$  are both the bias terms can be bounded by analyzing the tail of the distribution and Markov's inequality. We also use Lemma 12 to bound  $\sum_{t=0}^{k}\|g_{t+1}(v_{t})\|_{*}^{2}$ . Finally, using the induction technique, we can prove that  $w_{t}=v_{t}$  with high probability and obtain the desired result.
Lemma 12 (Lemma A.5 from [35]). With probability  $\geq 1 - \delta$ ,  $\sum_{t=1}^{T} Q_t^2 \leq T \mu_2 + 2M \sqrt{T} \left( \frac{2}{\delta} \right)^{\frac{1}{p}}$ .
4 Generalization bounds of SMD for Markovian data
The final result we present in this paper is the following theorem for the case when the data are sampled from a Markov chain.
Theorem 13. Suppose  $\ell$  is convex,  $(C_1,C_2)$ -quadratically bounded. Given  $T$ ,  $((x_{t},y_{t}))_{t\leq T}$  are sampled from a Markov chain with  $\max \left\{\| x_t\|_*^2,|y_t|^2\right\} \leq 1$  and  $\left(\pi ,\tau ,\epsilon = \frac{1}{\sqrt{T}}\right)$  is an approximate stationarity witness. Suppose that  $w_{\mathrm{ref}}$  satisfies Assumption 4. Then for  $\eta \leq \frac{1}{2C_2\sqrt{T(1 + 2\tau)}}$ , with probability at least  $1 - \tau \delta$ , for every  $0\le k\le T - 1$
$$
\frac {1}{k + 1} \sum_ {t = 0} ^ {k} \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} ^ {*}\right) + \frac {\mathbf {D} _ {\psi} \left(w _ {\text {r e f}} ; w _ {k + 1}\right)}{\eta (k + 1)} \leq \frac {R ^ {2}}{2 \eta (k + 1)}.
$$
where  $R = \max \left\{\sqrt{6\left(2D_0 + 2\eta D_0\sqrt{T} + 16\eta^2C_4^2T\tau\log\frac{1}{\delta} + 2T\eta^2C_4^2 + 4\eta^2\tau TC_4^2\right)}\right.$ ,  $6(2\eta \tau C_4 + 2\eta C_4\epsilon T + 4\eta \tau C_4)\} = O(1)$  and  $C_4 = C_1 + C_2(1 + \| w_{\mathrm{ref}}\|)$ .
We will give a proof sketch for this theorem.
Proof Sketch. The proof of this Theorem follows similarly to that of Theorem 7. The difference here is not needed to bound  $\tau$  different martingale difference sequences in the form of
$$
\mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\mathrm {r e f}}\right) \mid \mathcal {F} _ {\tau i + j} \right] - \mathbb {E} \left[ \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right) \mid \mathcal {F} _ {\tau i + j} \right] + \ell_ {\tau (i + 1) + j} \left(w _ {\mathrm {r e f}}\right) - \ell_ {\tau (i + 1) + j} \left(w _ {\tau i + j}\right)
$$
for  $0 \leq j \leq \tau - 1$ ,  $0 \leq i \leq \frac{T - 1 - j}{\tau}$ . We also need the assumption on the approximate stationarity witness to see that
$$
\left| \mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) - \mathbb {E} \left[ \ell_ {t + \tau} \left(w _ {\text {r e f}}\right) \mid \mathcal {F} _ {t} \right] + \mathbb {E} \left[ \ell_ {t + \tau} \left(w _ {t}\right) \mid \mathcal {F} _ {t} \right] \right| \leq C _ {4} R \epsilon .
$$
Now we only need the union bound over  $\tau$  sequences, instead of all iterations. The success probability will decrease from  $1 - \delta$  to  $1 - \tau \delta$ .
In this paper, we show a new approach to analyze the generalization error of SMD for quadratically bounded losses. Our approach improves a logarithmic factor for the realizable setting and nonrealizable setting with light tailed data and a poly  $T$  factor for the non-realizable setting with polynomial tailed data from the prior work by [35]. An inherent limitation of the current approach is the assumption that we can obtain a fresh sample in each iteration, whereas the setting with finite training data is still not well understood. In the realizable setting, we require that the data is bounded, as opposed to more relaxed assumptions in the non-realizable settings. We leave the question of resolving these issues for future works.
TDN and AE were supported in part by NSF CAREER grant CCF-1750333, NSF grant III-1908510, and an Alfred P. Sloan Research Fellowship. HN was supported by NSF CAREER grant CCF-1750716 and NSF grant 2311649.