SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33:4381-4391, 2020.
[2] Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovsky. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory, pages 610-626. PMLR, 2020.
[3] Radu Alexandru Dragomir, Mathieu Even, and Hadrien Hendrikx. Fast stochastic bregman gradient methods: Sharp analysis and variance reduction. In International Conference on Machine Learning, pages 2815-2825. PMLR, 2021.
[4] John C Duchi, Alekh Agarwal, Mikael Johansson, and Michael I Jordan. Ergodic mirror descent. SIAM Journal on Optimization, 22(4):1549-1578, 2012.
[5] Vitaly Feldman and Jan Vondrak. Generalization bounds for uniformly stable algorithms. Advances in Neural Information Processing Systems, 31, 2018.
[6] Vitaly Feldman and Jan Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory, pages 1270-1279. PMLR, 2019.
[7] David A Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100-118, 1975.
[8] Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization: A generic algorithmic framework. SIAM Journal on Optimization, 22(4):1469-1492, 2012.
[9] Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechenskii, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. Advances in Neural Information Processing Systems, 35:31319-31332, 2022.
[10] Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems, 33:15042-15053, 2020.
[11] Eduard Gorbunov, Marina Danilova, Innokentiy Shibaev, Pavel Dvurechensky, and Alexander Gasnikov. Near-optimal high probability complexity bounds for non-smooth stochastic optimization with heavy-tailed noise. arXiv preprint arXiv:2106.05958, 2021.
[12] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225–1234. PMLR, 2016.
[13] Nicholas JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory, pages 1579-1613. PMLR, 2019.
[14] Elad Hazan and Satyen Kale. Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. The Journal of Machine Learning Research, 15(1):2489-2512, 2014.
[15] Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018.
[34] Matus Telgarsky. Margins, shrinkage, and boosting. In International Conference on Machine Learning, pages 307-315. PMLR, 2013.
[35] Matus Telgarsky. Stochastic linear optimization never overfits with quadratically-bounded losses on general data. In Conference on Learning Theory, pages 5453-5488. PMLR, 2022.
[36] Joel Tropp. Freedman's inequality for matrix martingales. 2011.
[37] R van Handel. Apc 550: Probability in high dimension. Lecture Notes. Princeton University. Retrieved from https://web.math.princeton.edu/rvan/APC550.pdf on December, 21:2016, 2016.
[38] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
A Concentration Inequalities
Lemma 14. Let  $X$  be a random variable such that  $\mathbb{E}[X] = 0$  and  $|X| \leq R$  almost surely. Then for  $0 \leq \lambda \leq \frac{1}{R}$
The following lemma is similar to Lemma 2.2 in [21].
Lemma 15. Suppose that  $Q$  satisfies for all  $0 \leq \lambda \leq \frac{1}{\sigma}$ ,  $\mathbb{E}\left[\exp \left(\lambda^2 Q^2\right)\right] \leq \exp \left(\lambda^2 \sigma^2\right)$ . Then for variable  $X$  such that  $\mathbb{E}[X] = 0$  and  $|X| \leq a (Q + b)$  for some  $a \geq 0$  then for all  $\lambda \geq 0$
$$
\mathbb {E} \left[ \exp (\lambda X) \right] \leq \exp \left(2 \lambda^ {2} a ^ {2} (\sigma^ {2} + b ^ {2})\right).
$$
In particular, if  $b = 0$  we can have a tighter constant:  $\mathbb{E}\left[\exp \left(\lambda X\right)\right] \leq \exp \left(\lambda^2 a^2\sigma^2\right)$ .
Proof. We consider  $\mathbb{E}$  [exp  $(\lambda X)]$
If  $0 \leq \lambda \leq \frac{1}{\sqrt{2a}\sigma}$  then using  $\exp(x) \leq x + \exp(x^2)$
$$
\begin{array}{l} \mathbb {E} \left[ \exp (\lambda X) \right] \leq \mathbb {E} \left[ \exp (\lambda^ {2} X ^ {2}) \right] \\ \leq \mathbb {E} \left[ \exp \left(\lambda^ {2} a ^ {2} (Q + b) ^ {2}\right) \right] \\ \leq \mathbb {E} \left[ \exp \left(2 \lambda^ {2} a ^ {2} Q ^ {2} + 2 \lambda^ {2} a ^ {2} b ^ {2}\right) \right] \\ \leq \exp \left(2 \lambda^ {2} a ^ {2} b ^ {2}\right) \mathbb {E} \left[ \exp \left(2 \lambda^ {2} a ^ {2} Q ^ {2}\right) \right] \\ \leq \exp \left(2 \lambda^ {2} a ^ {2} \left(\sigma^ {2} + b ^ {2}\right)\right) \\ \end{array}
$$
Otherwise  $\frac{1}{\sigma} \leq \lambda \sqrt{2a}$
$$
\begin{array}{l} \mathbb {E} \left[ \exp (\lambda X) \right] \leq \mathbb {E} \left[ \exp \left(\lambda^ {2} a ^ {2} \sigma^ {2} + \frac {X ^ {2}}{4 a ^ {2} \sigma^ {2}}\right) \right] \\ \leq \exp \left(\lambda^ {2} a ^ {2} \sigma^ {2}\right) \mathbb {E} \left[ \exp \left(\frac {(Q + b) ^ {2}}{4 \sigma^ {2}}\right) \right] \\ \leq \exp \left(\lambda^ {2} a ^ {2} \sigma^ {2}\right) \mathbb {E} \left[ \exp \left(\frac {Q ^ {2} + b ^ {2}}{2 \sigma^ {2}}\right) \right] \\ \leq \exp \left(\lambda^ {2} a ^ {2} \sigma^ {2}\right) \exp \left(\frac {b ^ {2}}{2 \sigma^ {2}}\right) \exp \left(\frac {1}{2}\right) \\ \leq \exp \left(\lambda^ {2} a ^ {2} \sigma^ {2}\right) \exp \left(\lambda^ {2} a ^ {2} b ^ {2}\right) \exp \left(\lambda^ {2} a ^ {2} \sigma^ {2}\right) \\ \leq \exp \left(2 \lambda^ {2} a ^ {2} (\sigma^ {2} + b ^ {2})\right). \\ \end{array}
$$
Theorem 16 (Freedman's inequality [7, 36]). Let  $(X_{t})_{t\geq 1}$  be a martingale difference sequence. Assume that there exists a constant  $c$  such that  $|X_{t}|\leq c$  almost surely for all  $t\geq 1$  and define  $\sigma_t^2 = \mathbb{E}\left[X_t^2\mid X_{t - 1},\ldots ,X_1\right]$ . Then for all  $b > 0$ ,  $F > 0$  and  $T\geq 1$
$$
\Pr \left[ \exists T \geq 1: \left| \sum_ {t = 1} ^ {T} X _ {t} \right| > b a n d \sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2} \leq F \right] \leq 2 \exp \left(- \frac {b ^ {2}}{2 F + 2 c b / 3}\right).
$$
Proof of Lemma 2. Using the optimality condition
$$
\langle \eta g _ {t + 1} + \nabla \psi (w _ {t + 1}) - \nabla \psi (w _ {t}), w _ {\mathrm {r e f}} - w _ {t + 1} \rangle \geq 0
$$
$$
\begin{array}{l} \langle \eta g _ {t + 1}, w _ {t} - w _ {\text {r e f}} \rangle = \langle \eta g _ {t + 1}, w _ {t + 1} - w _ {\text {r e f}} \rangle + \langle \eta g _ {t + 1}, w _ {t} - w _ {t + 1} \rangle \\ \leq \left\langle \nabla \psi \left(w _ {t + 1}\right) - \nabla \psi \left(w _ {t}\right), w _ {\text {r e f}} - w _ {t + 1} \right\rangle + \left\langle \eta g _ {t + 1}, w _ {t} - w _ {t + 1} \right\rangle \\ = \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) - \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(w _ {t + 1}; w _ {t}\right) + \langle \eta g _ {t + 1}, w _ {t} - w _ {t + 1} \rangle \\ \leq \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t}) - \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t + 1}) - \frac {1}{2} \| w _ {t} - w _ {t + 1} \| ^ {2} + \langle \eta g _ {t + 1}, w _ {t} - w _ {t + 1} \rangle \\ \leq \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t}) - \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t + 1}) + \frac {\eta^ {2}}{2} \| g _ {t + 1} \| _ {*} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {t}\right) \leq \langle \eta g _ {t + 1}, w _ {\mathrm {r e f}} - w _ {t} \rangle + \frac {\eta^ {2}}{2} \| g _ {t + 1} \| _ {*} ^ {2} \\ \leq \eta \left(\ell_ {t + 1} \left(w _ {\mathrm {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \frac {\eta^ {2}}{2} \left\| g _ {t + 1} \right\| _ {*} ^ {2} \\ \end{array}
$$
Proof of Lemma 4. We have
$$
\begin{array}{l} \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(w _ {\mathrm {r e f}}; w _ {t}\right) \leq \eta \left(\ell_ {t + 1} \left(w _ {\mathrm {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \frac {\eta^ {2}}{2} \left\| g _ {t + 1} \right\| _ {*} ^ {2} \\ \leq \eta \left(\ell_ {t + 1} \left(w _ {\mathrm {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \frac {\eta^ {2}}{2} \ell_ {t + 1} ^ {\prime} \left(w _ {t}\right) ^ {2} \\ \leq \eta \left(\ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \eta^ {2} \rho \ell_ {t + 1} \left(w _ {t}\right) \\ = \eta \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \frac {\eta}{2} \ell_ {t + 1} \left(w _ {t}\right) \leq \eta \ell_ {t + 1} \left(w _ {\text {r e f}}\right). \\ \end{array}
$$
Summing up, we have, for any  $0 \leq t \leq T$
$$
\mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) \leq \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {0}\right) + \eta \sum_ {i = 1} ^ {t} \ell_ {i} \left(w _ {\text {r e f}}\right) = D _ {0} + \eta \sum_ {i = 1} ^ {t} \ell_ {i} \left(w _ {\text {r e f}}\right).
$$
Proof of Lemma 5. We have  $|\ell_i(w_{\mathrm{ref}}) - \mathcal{R}(w_{\mathrm{ref}})| \leq \max \{\ell_i(w_{\mathrm{ref}}), \mathcal{R}(w_{\mathrm{ref}})\} \leq C_3$  thus by lemma 14, for  $\lambda \leq \frac{1}{C_3}$
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(\lambda \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right)\right) \right] \\ \leq \exp \left(\frac {3}{4} \lambda^ {2} \mathbb {E} \left[ \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right) ^ {2} \right]\right) \\ \stackrel {(a)} {\leq} \exp \left(\frac {3}{4} \lambda^ {2} \mathbb {E} \left[ \ell_ {i} \left(w _ {\text {r e f}}\right) ^ {2} \right]\right) \\ \stackrel {(b)} {\leq} \exp \left(\frac {3}{4} \lambda^ {2} C _ {3} \mathcal {R} (w _ {\mathrm {r e f}})\right) \leq \exp \left(\frac {3}{4} \lambda \mathcal {R} (w _ {\mathrm {r e f}})\right), \\ \end{array}
$$
where for  $(a)$  we use  $\mathbb{E}\left[(X - \mathbb{E}[X])^2\right] \leq \mathbb{E}[X^2]$  and for  $(b)$  we use  $\ell_i(w_{\mathrm{ref}}) \leq C_3$ . Since  $\ell_i(w_{\mathrm{ref}})$  are independent random variables, we have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(\lambda \sum_ {i = 1} ^ {T} \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right)\right) \right] = \mathbb {E} \left[ \prod_ {i = 1} ^ {T} \exp \left(\lambda \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right)\right) \right] \\ = \prod_ {i = 1} ^ {T} \mathbb {E} \left[ \exp \left(\lambda \left(\ell_ {i} \left(w _ {\mathrm {r e f}}\right) - \mathcal {R} \left(w _ {\mathrm {r e f}}\right)\right)\right) \right] \leq \prod_ {i = 1} ^ {T} \exp \left(\frac {3}{4} \lambda \mathcal {R} \left(w _ {\mathrm {r e f}}\right)\right) = \exp \left(\frac {3}{4} \lambda T \mathcal {R} \left(w _ {\mathrm {r e f}}\right)\right). \\ \end{array}
$$
Hence by Markov's inequality
$$
\begin{array}{l} \Pr \left[ \lambda \sum_ {i = 1} ^ {T} \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right) \geq \frac {3}{4} \lambda T \mathcal {R} \left(w _ {\text {r e f}}\right) + \log \frac {1}{\delta} \right] \\ = \Pr \left[ \exp \left(\lambda \sum_ {i = 1} ^ {T} \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right)\right) \geq \frac {1}{\delta} \exp \left(\frac {3}{4} \lambda T \mathcal {R} \left(w _ {\text {r e f}}\right)\right) \right] \\ \leq \frac {\mathbb {E} \left[ \exp \left(\lambda \sum_ {i = 1} ^ {T} \left(\ell_ {i} \left(w _ {\text {r e f}}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right)\right)\right) \right]}{\frac {1}{\delta} \exp \left(\frac {3}{4} \lambda T \mathcal {R} \left(w _ {\text {r e f}}\right)\right)} \leq \delta \\ \end{array}
$$
Choose  $\lambda = \frac{1}{C_3}$  we have with probability at least  $1 - \delta$
$$
\sum_ {i = 1} ^ {T} \left(\ell_ {i} \left(w _ {\mathrm {r e f}}\right) - \mathcal {R} \left(w _ {\mathrm {r e f}}\right)\right) \leq \frac {3}{4} T \mathcal {R} \left(w _ {\mathrm {r e f}}\right) + C _ {3} \log \frac {1}{\delta}.
$$
Proof of Theorem 3. Towards bounding the risk  $\sum_{t=0}^{k}\mathcal{R}(w_t)$ , we define random variables
$$
\begin{array}{l} Z _ {t} = \frac {1}{2} z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {\mathrm {r e f}})\right) + z _ {t} \left(\mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t + 1}) - \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t})\right) \\ - \frac {3}{1 6} z _ {t} \eta (\mathcal {R} (w _ {\mathrm {r e f}}) + \mathcal {R} (w _ {t})); \quad \forall 0 \leq t \leq T - 1 \\ \end{array}
$$
$$
\text {w h e r e} z _ {t} = \frac {1}{\eta C _ {4} \sqrt {2 \eta \gamma C _ {3} + 2 D _ {0} + 2 \eta \sum_ {i = 1} ^ {t} \ell_ {i} (w _ {\mathrm {r e f}})}}; \quad \gamma = \max  \left\{1, \log \frac {1}{\delta} \right\}
$$
$$
\text {a n d} S _ {t} = \sum_ {i = 0} ^ {t} Z _ {i}; \quad \forall 0 \leq t \leq T - 1
$$
The reason to define these variables is because from Lemma 4, we can bound
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t} \right] \times \exp \left(\frac {3}{1 6} z _ {t} \eta (\mathcal {R} (w _ {\text {r e f}}) + \mathcal {R} (w _ {t}))\right) \\ \leq \mathbb {E} \left[ \exp \left(\frac {1}{2} z _ {t} \eta (\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) - \ell_ {t + 1} (w _ {\text {r e f}})) + z _ {t} (\eta \ell_ {t + 1} (w _ {\text {r e f}}) - \frac {\eta}{2} \ell_ {t + 1} (w _ {t}))\right) \mid \mathcal {F} _ {t} \right] \\ = \mathbb {E} \left[ \exp \left(\frac {1}{2} z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) + \ell_ {t + 1} (w _ {\text {r e f}}) - \ell_ {t + 1} (w _ {t})\right)\right) \mid \mathcal {F} _ {t} \right] \\ \end{array}
$$
where now inside the expectation, we have the term  $\mathcal{R}(w_t) - \mathcal{R}(w_{\mathrm{ref}}) + \ell_{t+1}(w_{\mathrm{ref}}) - \ell_{t+1}(w_t)$  which has expectation 0. This reminds us of Lemma 14. To use this lemma, we notice that, by the assumption that the samples are IID with  $\max \{\|x\|_*, |y|\} \leq 1$  and Lemma 1,
$$
| \ell_ {x, y} (w _ {\mathrm {r e f}}) - \ell_ {x, y} (w _ {t}) | \leq \| w _ {\mathrm {r e f}} - w _ {t} \| \underbrace {(C _ {1} + C _ {2} (1 + \| w _ {\mathrm {r e f}} \|))} _ {C _ {4}}
$$
$$
\left| \mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) \right| = \left| \mathbb {E} \left[ \ell_ {x, y} \left(w _ {\text {r e f}}\right) - \ell_ {x, y} \left(w _ {t}\right) \right] \right| \leq C _ {4} \| w _ {\text {r e f}} - w _ {t} \|
$$
$$
\left| \frac {\eta}{2} \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) + \ell_ {t + 1} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {t})\right) \right| \leq \eta C _ {4} \| w _ {\mathrm {r e f}} - w _ {t} \|
$$
By the choice of  $z_{t}$  we have
$$
z _ {t} \leq \frac {1}{\eta C _ {4} \sqrt {2 \eta C _ {3} + 2 D _ {0} + 2 \eta \sum_ {i = 1} ^ {t} \ell_ {i} (w _ {\mathrm {r e f}})}} \leq \frac {1}{\eta C _ {4} \sqrt {2 \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}} ; w _ {t})}} \leq \frac {1}{\eta C _ {4} \| w _ {\mathrm {r e f}} - w _ {t} \|}
$$
Now we can apply Lemma 14 to bound
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t} \right] \times \exp \left(\frac {3}{1 6} z _ {t} \eta \left(\mathcal {R} \left(w _ {\text {r e f}}\right) + \mathcal {R} \left(w _ {t}\right)\right)\right) \\ \leq \exp \left(\frac {3}{4} \frac {1}{4} z _ {t} ^ {2} \eta^ {2} \mathbb {E} \left[ \left(\mathcal {R} \left(w _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) ^ {2} \mid \mathcal {F} _ {t} \right]\right) \\ \leq \exp \left(\frac {3}{1 6} z _ {t} ^ {2} \eta^ {2} \mathbb {E} \left[ \left(\ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) ^ {2} \mid \mathcal {F} _ {t} \right]\right) \\ \leq \exp \left(\frac {3}{1 6} z _ {t} ^ {2} \eta^ {2} C _ {4} \| w _ {\text {r e f}} - w _ {t} \| \mathbb {E} \left[ \ell_ {t + 1} \left(w _ {\text {r e f}}\right) + \ell_ {t + 1} \left(w _ {t}\right) \mid \mathcal {F} _ {t} \right]\right) \\ \leq \exp \left(\frac {3}{1 6} z _ {t} \eta \left(\mathcal {R} \left(w _ {\text {r e f}}\right) + \mathcal {R} \left(w _ {t}\right)\right)\right) \\ \end{array}
$$
Expanding this inequality, we obtain
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \frac {5}{1 6} z _ {t} \eta \mathcal {R} (w _ {t}) + z _ {k} \mathbf {D} _ {\psi} (w _ {\text {r e f}}; w _ {k + 1}) \\ \leq \log \frac {1}{\delta} + z _ {0} \mathbf {D} _ {\psi} (w _ {\text {r e f}}; w _ {0}) + \frac {1 1}{1 6} \eta \mathcal {R} (w _ {\text {r e f}}) \sum_ {t = 0} ^ {k} z _ {t} + \frac {1}{2} \sum_ {t = 0} ^ {k} z _ {t} \eta \ell_ {t + 1} (w _ {\text {r e f}}) \\ + \sum_ {t = 1} ^ {k} \underbrace {\left(z _ {t} - z _ {t - 1}\right)} _ {\leq 0} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) \tag {1} \\ \end{array}
$$
$$
\stackrel {(a)} {\leq} \log \frac {1}{\delta} + z _ {0} D _ {0} + \frac {1 1}{1 6} \eta \mathcal {R} (w _ {\mathrm {r e f}}) (k + 1) z _ {0} + \frac {1}{2} \sum_ {t = 0} ^ {k} \frac {\eta \ell_ {t + 1} (w _ {\mathrm {r e f}})}{\eta C _ {4} \sqrt {2 \eta C _ {3} + 2 D _ {0} + 2 \eta \sum_ {i = 1} ^ {t} \ell_ {i} (w _ {\mathrm {r e f}})}}
$$
$$
\stackrel {(b)} {\leq} \log \frac {1}{\delta} + z _ {0} D _ {0} + \frac {1 1}{1 6} \eta \mathcal {R} \left(w _ {\text {r e f}}\right) (k + 1) z _ {0} + \frac {1}{2 \eta C _ {4}} \sum_ {t = 0} ^ {k} \frac {\eta \ell_ {t + 1} \left(w _ {\text {r e f}}\right)}{\sqrt {2 D _ {0} + 2 \eta \sum_ {i = 1} ^ {t + 1} \ell_ {i} \left(w _ {\text {r e f}}\right)}} \tag {2}
$$
For (a) we use the fact that  $(z_{t})$  is a decreasing sequence and  $\mathcal{R}(w_{\mathrm{ref}})\leq \frac{\rho D_0}{T}$ . For (b) we use the assumption  $\ell_{t + 1}(w_{\mathrm{ref}})\leq C_3$ . Now notice that we can write  $\frac{\eta\ell_{t + 1}(w_{\mathrm{ref}})}{\sqrt{2D_0 + 2\eta\sum_{i = 1}^{t + 1}\ell_i(w_{\mathrm{ref}})}}\leq \sqrt{2D_0 + 2\eta\sum_{i = 1}^{t + 1}\ell_i(w_{\mathrm{ref}})} -\sqrt{2D_0 + 2\eta\sum_{i = 1}^{t}\ell_i(w_{\mathrm{ref}})}$  and sum over  $t$  we obtain
$$
\begin{array}{l} \frac {5}{1 6} z _ {k} \eta \sum_ {t = 0} ^ {k} \mathcal {R} \left(w _ {t}\right) + z _ {k} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq \log \frac {1}{\delta} + \frac {1 1 (k + 1) \mathcal {R} \left(w _ {\mathrm {r e f}}\right)}{1 6 C _ {4} \sqrt {2 \eta \gamma C _ {3} + 2 D _ {0}}} + \frac {1}{\sqrt {2} \eta C _ {4}} \sqrt {D _ {0} + \eta \sum_ {i = 1} ^ {k + 1} \ell_ {i} \left(w _ {\mathrm {r e f}}\right)} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \mathcal {R} (w _ {t}) + \frac {1 6}{5 \eta} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq \frac {1 6 C _ {4}}{5} \left(\log \frac {1}{\delta} + \frac {1 1 (k + 1) \mathcal {R} \left(w _ {\text {r e f}}\right)}{1 6 C _ {4} \sqrt {2 \eta \gamma C _ {3} + 2 D _ {0}}} + \frac {1}{\sqrt {2} \eta C _ {4}} \sqrt {D _ {0} + \eta \sum_ {i = 1} ^ {T} \ell_ {i} \left(w _ {\text {r e f}}\right)}\right) \\ \end{array}
$$
$$
\times \sqrt {2 \eta \gamma C _ {3} + 2 D _ {0} + 2 \eta \sum_ {i = 1} ^ {T} \ell_ {i} (w _ {\mathrm {r e f}})}
$$
By Lemma 5, with probability at least  $1 - \delta$  we have
$$
\sum_ {i = 1} ^ {T} \ell_ {i} (w _ {\mathrm {r e f}}) \leq \frac {7}{4} T \mathcal {R} (w _ {\mathrm {r e f}}) + C _ {3} \log \frac {1}{\delta} \leq \frac {7}{4} \rho D _ {0} + C _ {3} \gamma
$$
Therefore with probability at least  $1 - 2\delta$
$$
\begin{array}{l} \sum_ {t = 0} ^ {k} \mathcal {R} (w _ {t}) + \frac {1 6}{5 \eta} \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {k + 1}\right) \\ \leq \left(\frac {1 6 C _ {4}}{5} \log \frac {1}{\delta} + \frac {1 1 (k + 1)}{5 \sqrt {2 \eta \gamma C _ {3} + 2 D _ {0}}} \mathcal {R} (w _ {\mathrm {r e f}}) + \frac {8}{5 \eta} \sqrt {\frac {1 5}{4} D _ {0} + 2 \eta \gamma C _ {3}}\right) \sqrt {\frac {1 5}{4} D _ {0} + 4 \eta \gamma C _ {3}} \\ \leq \frac {1 6 C _ {4}}{5} \log \frac {1}{\delta} \sqrt {\frac {1 5}{4} D _ {0} + 4 \eta \gamma C _ {3}} + \left(\frac {6}{\eta} D _ {0} + \frac {3 2}{5} \gamma C _ {3}\right) + 3 (k + 1) \mathcal {R} \left(w _ {\mathrm {r e f}}\right). \\ \end{array}
$$
which gives us the conclusion.
Proof of Theorem 10. First we consider the bounded domain case. Let
$$
v _ {t} = \arg \min  _ {\| w - w _ {\text {r e f}} \| \leq R} \left\{\left\langle \eta_ {t} g _ {t} \left(v _ {t - 1}\right), w \right\rangle + \mathbf {D} _ {\psi} (w; v _ {t - 1}) \right\}
$$
where we use  $g_{t}(v_{t - 1})$  to denote the gradient at  $v_{t - 1}$  using the same data point  $(x_{t},y_{t})$  when computing  $w_{t}$  and we choose
$$
\begin{array}{l} R = \max  \left\{\sqrt {6 \left(D _ {0} + \eta^ {2} C _ {4} ^ {2} \left(T \mu_ {2} + 2 M \sqrt {T} \left(\frac {2}{\delta}\right) ^ {\frac {1}{p}}\right)\right)}, \right. \\ \left. 6 \left(\frac {2}{3} \gamma \left(7 \left(\frac {M T}{\delta}\right) ^ {1 / 2 p} + 2 \mu_ {1}\right) + \sqrt {\log \frac {2}{\delta} T \mu_ {2}}\right) \eta C _ {4} \right\} \\ \end{array}
$$
$$
\begin{array}{l} \left| \left(\mathcal {R} (v _ {t}) - \mathcal {R} (w _ {\text {r e f}}) + \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} (v _ {t})\right) \right| \\ \leq \left| \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(v _ {t}\right) \right| + \left| \mathcal {R} \left(v _ {t}\right) - \mathcal {R} \left(w _ {\text {r e f}}\right) \right| \\ \leq \left(Q _ {t} + \mu_ {1}\right) \| w _ {\text {r e f}} - v _ {t} \| C _ {4} \leq \left(Q _ {t} + \mu_ {1}\right) R C _ {4} \tag {3} \\ \end{array}
$$
Let us define the following variables
In words,  $U_{t}$  is the variable of our interest and  $P_{t}$  is the truncated version of  $U_{t}$  and  $B_{t}$  is the bias. We would want to control these terms in order to bound  $\sum_{t=0}^{k} U_{t}$ . We start with the following decomposition
First, we consider the term  $\sum_{t=0}^{k} \mathbb{E}[P_t \mid \mathcal{F}_t]$ .