SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we revisit the generalization error of stochastic mirror descent for quadratically bounded losses studied in Telgarsky (2022). Quadratically bounded losses is a broad class of loss functions, capturing both Lipschitz and smooth functions, for both regression and classification problems. We study the high probability generalization for this class of losses on linear predictors in both realizable and non-realizable cases when the data are sampled IID or from a Markov chain. The prior work relies on an intricate coupling argument between the iterates of the original problem and those projected onto a bounded domain. This approach enables blackbox application of concentration inequalities, but also leads to suboptimal guarantees due in part to the use of a union bound across all iterations. In this work, we depart significantly from the prior work of Telgarsky (2022), and introduce a novel approach for establishing high probability generalization guarantees. In contrast to the prior work, our work directly analyzes the moment generating function of a novel supermartingale sequence and leverages the structure of stochastic mirror descent. As a result, we obtain improved bounds in all aforementioned settings. Specifically, in the realizable case and non-realizable case with light-tailed sub-Gaussian data, we improve the bounds by a log  $T$  factor, matching the correct rates of  $1 / T$  and  $1 / \sqrt{T}$ , respectively. In the more challenging case of heavy-tailed polynomial data, we improve the existing bound by a poly  $T$  factor.
Along with convergence analysis of optimization methods, understanding the generalization of models trained by these methods on unseen data is an important question in machine learning. However, despite the number of works attempting to answer it, the problem has not been fully understood, even in the simplest setting of linear predictors constructed with the standard stochastic gradient/mirror descent. A great part of prior works [33, 15, 30, 31, 32] focus only on the generalization on linearly separable data and/or of models trained with specific losses with exponentially decaying tails such as logistic loss. The question of what we can guarantee beyond these settings remains open.
Recently, [35] proposes a new approach to analyze the generalization error with high probability of stochastic mirror descent for a broad class of quadratically bounded losses, beyond the realizable setting. This class of losses encapsulates both Lipschitz and smooth functions, for both regression and classification problems. The obtained bounds complement existing in-expectation bounds [12] and nearly match the counterpart of convergence rates in optimization. While this result pushes forward the state of the art, the obtained guarantees do not completely resolve the problem. The central piece of the proposed approach is a "coupling" technique between the iterates of the original problem and those projected onto a bounded domain. In this technique, one first constrains the problem in a bounded domain with a well chosen diameter. The bounded domain diameter allows to apply concentration inequalities as a blackbox and obtain bounds in high probability. Then using an inductive argument and a union bound across all iterations, one can show that the iterates in the original problem coincide with the ones in the constrained problem. Due to the union bound, the success probability decreases from  $1 - \delta$  to  $1 - T\delta$ , where  $T$  is the number of iterations in the algorithm. This loss translates to a milder log  $T$  factor loss in the guarantee in the case of realizable data, and a more significant poly  $T$  factor loss in the non-realizable setting when the data has polynomial tails. Thus a natural question arises of whether we can obtain a stronger analysis that closes these remaining gaps.
In this paper, we revisit these generalization bounds for quadratically bounded losses by [35]. We introduce a novel approach to analyze the generalization errors of stochastic mirror descent in both realizable and non-realizable cases when the data are sampled IID or from a Markov chain. In all these cases, we remove the need to use the union bound argument, thus preventing the loss in the success probability. This translates to the following improvements:
- In the realizable, and the non-realizable cases with sub-gaussian tailed data and Markovian data, we improve the bounds by a  $\log T$  factor. This improvement comes from analyzing the moment generating function of a martingale difference sequence with well-chosen coefficients. In these cases, we also remove the necessity of using the coupling-based argument used in the same work by [35]. Instead, by solely making use of the problem structure, we arrive at the same conclusion that with high probability, the iterates of stochastic mirror descent for quadratically bounded losses behave as if the problem domain is bounded.
- In the non-realizable case with polynomial tailed data, we improve the existing bound by a poly  $T$  factor. Due to the polynomial dependency on  $\frac{1}{\delta}$ , being able to maintain the same success probability through all iterations is crucial in this case. Unlike the previous work, we rely on a truncation technique. Using a more refined analysis of the truncated random variables, in combination with suitable concentration inequalities and the coupling technique, we improve the existing bounds significantly.
Broadly speaking, there is a rich body of works in optimization and generalization that provide convergence guarantees and generalization bounds for stochastic methods. Earlier works often focus on in-expectation bounds [8, 24, 26, 18, 12, 3], and bounds in high probability [16, 28, 14, 13] for problems with bounded domains or under various additional assumptions such as strong convexity, noise with light tails. Recent developments for optimization [25, 10, 20, 23, 11, 17, 9, 19, 29, 22, 21] are able to handle unconstrained problems and relax these assumptions, but also require changes to the algorithm such as gradient clipping.
In generalization error analysis, specifically, a number of prior works, including [33, 15, 30, 31, 32], focus only on linearly separable data. Among these, [33, 15, 32] only deal with exponentially tailed losses while [30, 31] show generalization bounds for general smooth convex losses. Our work, similarly to [35], goes beyond the realizable setting and specific losses. We show high probability generalization bounds in both realizable and non-realizable settings for the broad class of quadratically bounded losses, for both regression and classification problems.
Other related works include the line of works that examine generalization errors via algorithmic stability. The works by [12, 5, 6, 2, 1] show the generalization error of an arbitrary algorithm via a quantity called uniform stability. By bounding this quantity for specific algorithms on a fixed training dataset, they derive generalization bounds. Our work focuses on a different setting where we assume the algorithm has access to a fresh data sample in each iteration. In this regard, the setting in our work,
as well as the prior work by [35], stays closer to the world of optimization. However, in contrast to the optimization world in which we commonly impose assumptions on the stochastic gradients (such as having bounded variance or sub-gaussian noise), we make assumptions on the data (such as sub-gaussian or polynomial tailed data). This difference introduces several challenges which we overcome in our work.
The main point of reference for this paper is the work by [35]. This work develops a "coupling" technique to bound the generalization error of stochastic mirror descent for quadratically bounded losses. This technique has been employed in prior works [10, 11, 9, 29, 27, 22] to obtain high probability convergence bounds of stochastic methods in optimization. Our work improves their results by using a different approach that takes a closer look at the mechanism of the concentration inequalities and leverages the problem structure. When the data are bounded or have sub-gaussian tails, analyzing the moment generating function of a novel martingale difference sequence allows us to maintain the same success probability, without using either the coupling technique or the union bound. This new analysis, however, does not change the observation by [35] that the iterates of the unconstrained and the constrained problems coincide with high probability. When the data have a polynomial tail, we rely on a truncation technique. In this case, the coupling technique is necessary but not the union bound, and we are still able to significantly improve the success rate.
In terms of techniques, the work by [21] for optimization is the closest to ours. In this work, the authors develop the whitebox approach to analyzing stochastic methods for optimization with light-tailed noise. In this work, we study generalization errors. Moreover, in all settings, our choice of martingale difference sequences and coefficients are a significant departure from the prior work. In particular, in [21] the choice of coefficients only depends on the problem parameters whereas in the realizable case, our coefficients depend also on the historical data. Our approach also allows for a flexible use of an induction argument without decreasing the success probability, while in [21] the bounds are simpler and can be easily achieved in a single step.
In this section, we provide the general set up and necessary notations before analyzing stochastic mirror descent in the subsequent sections. Overall, we closely follow notations used in [35].
Domain and norms. In this work, we consider  $\mathcal{X}$  the domain of the problem to be a closed convex set or  $\mathbb{R}^d$ . We will use  $\| \cdot \|$  to denote an arbitrary norm on  $\mathcal{X}$  and let  $\| \cdot \|_*$  be its dual norm. We define the Bregman divergence as  $\mathbf{D}_{\psi}(w;v) = \psi (w) - \psi (v) - \langle \nabla \psi (v),w - v\rangle$  where  $\psi :\mathbb{R}^d\to \mathbb{R}$  is a differentiable function that is 1-strongly convex with respect to the norm  $\| \cdot \|$ .
Loss functions. Each loss function  $\ell : \mathbb{R} \times \mathbb{R} \to \mathbb{R}_{\geq 0}$  in our consideration can be written using a convex scalar function  $\widetilde{\ell}$  in one of the two following forms: 1)  $\ell(y, \widehat{y}) = \widetilde{\ell}(\mathrm{sign}(y) \widehat{y})$  where  $\mathrm{sign}(y) = 1$  if  $y \geq 0$  and  $= -1$  otherwise; and 2)  $\ell(y, \widehat{y}) = \widetilde{\ell}(y - \widehat{y})$ . The first form captures classification losses and the second regression losses. We will assume that subgradients  $\partial \ell$  of  $\ell$  in the second argument always exist, and let  $\ell'$  denote a subgradient in  $\partial \ell$ . For a function  $f$ , we also use  $\| \partial f(w) \| := \sup \{ \| g \| : g \in \partial f(w) \}$ . We further make the following assumptions, introduced in [35] as quadratic boundedness and self-boundedness.
Assumption 1. We assume that  $\ell$  is  $(C_1, C_2)$ -quadratically-bounded, for some constants  $C_1, C_2 \geq 0$ , i.e., for all  $y, \widehat{y}$
$$
\left| \ell^ {\prime} (y, \widehat {y}) \right| \leq C _ {1} + C _ {2} \left(\left| y \right| + \left| \widehat {y} \right|\right).
$$
This condition captures both classes of Lipschitz and smooth functions. Indeed, Lemma 1.2 from [35] shows that  $\alpha$ -Lipschitz functions are  $(\alpha, 0)$ -quadratically-bounded while  $\beta$ -smooth functions are  $\left(\left|\partial \widetilde{\ell}(0)\right|, \beta\right)$ -quadratically-bounded.
Assumption 2. In the realizable setting, we assume that  $\ell$  is  $\rho$ -self-bounding, i.e.,  $\widetilde{\ell}$  satisfies  $\widetilde{\ell'}(z)^2 \leq 2\rho \widetilde{\ell}(z)$  for all  $z \in \mathbb{R}$ .
The second assumption is a generalization of smoothness. This assumption is satisfied by smooth losses but also certain non-smooth losses such as the exponential loss. This condition is necessary in the current analysis to prove  $1 / T$  rates in the realizable setting. The readers can refer to [34, 35] for more detailed discussion on this assumption.
Algorithm 1 Stochastic Mirror Descent
Input  $w_{0}$  step size  $\eta$    
For  $t$  in 1...T  $g_{t}\in \partial \ell_{t}(w_{t - 1})$ $w_{t} = \arg \min_{w\in \mathcal{X}}\{\langle \eta g_{t},w\rangle +\mathbf{D}_{\psi}(w;w_{t - 1})\}$
Assumptions 1 and 2 are satisfied by commonly used loss functions in machine learning. These include the logistic loss  $\ell(y, \widehat{y}) = \ln(1 + \exp(-y\widehat{y}))$  and the squared loss  $\ell(y, \widehat{y}) = \frac{1}{2}(y - \widehat{y})^2$  (see Lemma 1.4 in [35]).
For the loss function  $\ell$  and the configuration  $w$ , and sample  $(x,y)$  where  $x$  denotes the attribute and  $y$  the label, we will write  $\ell_{x,y} = \ell(y, w^T x)$ . We state the following crucial lemma which is the same as Lemma A.1 in [35], whose proof will be omitted.
Lemma 1 (Lemma A.1 in ([35])). Suppose  $\ell$  is  $(C_1, C_2)$ -quadratically-bounded and  $B_x \geq 0$  is given. Given  $(x, y)$  such that  $\max \{\| x \|_* |y|\} \leq B_x$  and any  $u, v$ ,
$$
\begin{array}{l} \left\| \partial \ell_ {x, y} (u) \right\| _ {*} \leq B _ {x} \left(C _ {1} + C _ {2} B _ {x} \left(1 + \| u \|\right)\right) \\ \left| \ell_ {x, y} (u) - \ell_ {x, y} (v) \right| \leq B _ {x} \| u - v \| \left(C _ {1} + C _ {2} B _ {x} \left(1 + \| u \|\right)\right). \\ \end{array}
$$
Risk, IID and Markovian data. When sample  $(x_{i},y_{i})$  arrives in iteration  $i$  of an algorithm, we will use the notation  $\ell_i(w) = \ell (y_i,w^T x_i)$ . For an algorithm of  $T$  iterations, we use  $\mathcal{F}_t = \sigma ((x_1,y_1),\ldots ,(x_t,y_t))$  to denote the natural filtration up to and including time  $t$ . When the data are IID and generated from a distribution  $\pi$ , we define the risk
$$
\mathcal {R} (w) = \mathbb {E} _ {(x, y) \sim \pi} \left[ \ell \left(y, w ^ {T} x\right) \right];
$$
In contrast to IID data, Markovian data come from a stochastic process. This setting has also been considered in [4]. We let  $P_{s}^{t}$  be the distribution of  $(x_{t},y_{t})$  at iteration  $t$  conditioned on  $\mathcal{F}_s$ . We make the following assumption regarding the uniform mixing time of the stochastic process. Note that similar assumptions have also appeared in [35, 4].
Assumption 3. We assume that for some  $\epsilon, \tau \geq 0$  of our choice, there is a distribution  $\pi$  such that
$$
\sup_{t\in \mathbb{Z}_{\geq 0}}\sup_{\mathcal{F}_{t}}\mathrm{TV}\left(P_{t}^{t + \tau},\pi\right)\leq \epsilon .
$$
We refer to the triple  $(\pi, \tau, \epsilon)$  as an approximate stationarity witness. We then define the risk according to the approximate stationary distribution  $\pi \colon \mathcal{R}(w) = \mathbb{E}_{(x,y) \sim \pi}[\ell(y, w^T x)]$ .
Algorithm. Stochastic Mirror Descent is given in Algorithm 1. In this algorithm, for the simplicity of the analysis, we consider a fixed step size  $\eta$ . In each iteration, we pick a subgradient  $g_{t} \in \partial \ell_{t}(w_{t-1})$  and perform the update step.
We finally introduce a standard lemma used in the analysis of Stochastic Mirror Descent.
Lemma 2. For  $t\geq 0$  and  $w_{\mathrm{ref}}\in \mathcal{X}$  we have
$$
\mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) \leq \eta \left(\ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \ell_ {t + 1} \left(w _ {t}\right)\right) + \frac {\eta^ {2}}{2} \left\| g _ {t + 1} \right\| _ {*} ^ {2}.
$$
Other notations. We will use  $w_{\mathrm{ref}}$  to refer to a comparator of interest. For the simplicity of the exposition, we let  $D_0 = \mathbf{D}_{\psi}(w_{\mathrm{ref}};w_0)$ , and  $\mathcal{R}^* = \inf_{v\in \mathcal{X}}\mathcal{R}(v)$ . For a loss function  $\ell$  that is  $(C_1,C_2)$ -quadratically-bounded, we let  $C_4 = C_1 + C_2(1 + \| w_{\mathrm{ref}}\|)$ .
3 Generalization bounds of SMD for IID data
In this section, we distinguish between two cases: the realizable case and the non-realizable case. In the realizable case, there exists an optimal solution  $w^{*} \in \mathcal{X}$  such that  $\mathcal{R}(w^{*}) = 0$ . We will show that under mild assumptions, the risks of the solutions output by Algorithm 1 are bounded by  $O(1 / T)$ . In the non-realizable case, we will show, on the other hand, a weaker statement that the excess risks of the solutions are bounded by  $O(1 / \sqrt{T})$ .
3.1 Realizable case
In the realizable case, the comparator  $w_{\mathrm{ref}}$  is not necessarily the global minimizer. To show the  $1 / T$  rate, we will assume  $w_{\mathrm{ref}}$  satisfies  $\mathcal{R}(w_{\mathrm{ref}}) \leq \rho \mathbf{D}_{\psi}(w_{\mathrm{ref}}; w_0) / T$  and that the loss at  $w_{\mathrm{ref}}$  is bounded. The guarantee for the iterates of Algorithm 1 is provided in Theorem 3.
Theorem 3. Suppose  $\ell$  is convex,  $(C_1,C_2)$ -quadratically-bounded, and  $\rho$ -self-bounding. Given  $T$ ,  $((x_{t},y_{t}))_{t\leq T}$  are IID samples with  $\max \{\| x_t\|_*,|y_t|\} \leq 1$  almost surely,  $w_{\mathrm{ref}}$  satisfies  $\mathcal{R}(w_{\mathrm{ref}}) \leq \rho \mathbf{D}_{\psi}(w_{\mathrm{ref}};w_0) / T$ , and  $\max_{t < T}\ell_{t + 1}(w_{\mathrm{ref}}) \leq C_3$  almost surely. Then for  $\eta \leq \frac{1}{2\rho}$ , with probability at least  $1 - 2\delta$ , for every  $0 \leq k \leq T - 1$
$$
\frac {1}{k + 1} \sum_ {t = 0} ^ {k} \mathcal {R} (w _ {t}) + \frac {1 6 \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}} ; w _ {k + 1})}{5 (k + 1) \eta} \leq \frac {C}{k + 1} + 3 \mathcal {R} (w _ {\mathrm {r e f}}).
$$
where  $C = \frac{16C_4}{5}\log \frac{1}{\delta}\sqrt{\frac{15}{4}D_0 + 4\eta\gamma C_3} +\left(\frac{6}{\eta} D_0 + \frac{32}{5}\gamma C_3\right)$  with  $\gamma = \max \left\{1,\log \frac{1}{\delta}\right\}$ .
The analysis of Theorem 3 relies on the use of concentration inequalities. In contrast to existing works that utilize concentration inequalities as a blackbox, we will make use of the mechanism for proving concentration inequalities in order to obtain stronger guarantees. The type of concentration inequalities we consider are shown by analyzing the moment generating function of suitably chosen martingale sequences. We will use Lemma 14 (Appendix) which gives a basic inequality that bounds the moment generating function of a bounded random variable. To start the analysis, we use Lemma 2 and Assumption 2 to obtain
Lemma 4. For all  $t \geq 0$ , we have
$$
\mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(w _ {\text {r e f}}; w _ {t}\right) \leq \eta \ell_ {t + 1} \left(w _ {\text {r e f}}\right) - \frac {\eta}{2} \ell_ {t + 1} \left(w _ {t}\right),
$$
and hence,  $\mathbf{D}_{\psi}\left(w_{\mathrm{ref}};w_t\right)\leq \mathbf{D}_{\psi}\left(w_{\mathrm{ref}};w_0\right) + \eta \sum_{i = 1}^{t}\ell_i\left(w_{\mathrm{ref}}\right) = D_0 + \eta \sum_{i = 1}^{t}\ell_i\left(w_{\mathrm{ref}}\right).$
First, let us pay attention to the term  $\sum_{i=1}^{t} \ell_i(w_{\mathrm{ref}})$ . Recall that the terms  $\ell_i(w_{\mathrm{ref}})$  are non-negative and bounded by a constant  $C_3$  almost surely. We can analyze the term  $\sum_{i=1}^{T} \ell_i(w_{\mathrm{ref}})$  which upper bounds all sums  $\sum_{i=1}^{t} \ell_i(w_{\mathrm{ref}})$  by studying its moment generating function (or via a concentration inequality). We state this bound in the next lemma and defer the proof to the appendix.
Lemma 5. With probability at least  $1 - \delta$ ,  $\sum_{i=1}^{T} \ell_i(w_{\mathrm{ref}}) \leq \frac{7}{4} T \mathcal{R}(w_{\mathrm{ref}}) + C_3 \log \frac{1}{\delta}$ .
Lemma 4 and lemma 5 and the assumption that  $\mathcal{R}(w_{\mathrm{ref}}) = O(1 / T)$  imply that with probability at least  $1 - \delta$ ,  $\mathbf{D}_{\psi}\left(w_{\mathrm{ref}};w_t\right)$  is bounded. In other words, with probability at least  $1 - \delta$ , the iterates  $w_{t}$  all lie in a bounded region. One could therefore proceed to assume that the problem domain is simply this bounded ball around  $w_{\mathrm{ref}}$ . This is the basic idea behind the "coupling" technique demonstrated in [35]. However, the important question is how to obtain a bound for the risk of all iterates even when we are working with a problem with unbounded domain. Here, not paying close attention to the structure of the problem and the blackbox use of concentration inequalities lead to suboptimal bounds. On the other hand, as discussed above, a crucial novelty in our analysis is the choice of a supermartingale difference sequence, defined in the proof below. By working from first principles using moment generating function of this sequence, we derive two conclusions: 1) an improved risk bound can be obtained, and 2) the coupling technique is not necessary.
Proof Sketch. Towards bounding the risk  $\sum_{t=0}^{k} \mathcal{R}(w_t)$ , we define random variables
$$
\begin{array}{l} Z _ {t} = \frac {1}{2} z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {\mathrm {r e f}})\right) + z _ {t} \left(\mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t + 1}) - \mathbf {D} _ {\psi} (w _ {\mathrm {r e f}}; w _ {t})\right) \\ - \frac {3}{1 6} z _ {t} \eta (\mathcal {R} (w _ {\mathrm {r e f}}) + \mathcal {R} (w _ {t})), \quad \forall 0 \leq t \leq T - 1 \\ \end{array}
$$
where  $z_{t} = \frac{1}{\eta C_{4}\sqrt{2\eta\gamma C_{3} + 2D_{0} + 2\eta\sum_{i = 1}^{t}\ell_{i}(w_{\mathrm{ref}})}};$ $\gamma = \max \left\{1,\log \frac{1}{\delta}\right\}$
and we let  $S_{t} = \sum_{i=0}^{t} Z_{i}$ ;  $\forall 0 \leq t \leq T - 1$ . The reason to define these variables is because from Lemma 4, we can bound
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t} \right] \times \exp \left(\frac {3}{1 6} z _ {t} \eta \left(\mathcal {R} \left(w _ {\text {r e f}}\right) + \mathcal {R} \left(w _ {t}\right)\right)\right) \\ \leq \mathbb {E} \left[ \exp \left(\frac {1}{2} z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {\mathrm {r e f}})\right) + z _ {t} \left(\eta \ell_ {t + 1} (w _ {\mathrm {r e f}}) - \frac {\eta}{2} \ell_ {t + 1} (w _ {t})\right)\right) \mid \mathcal {F} _ {t} \right] \\ = \mathbb {E} \left[ \exp \left(\frac {1}{2} z _ {t} \eta \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\mathrm {r e f}}) + \ell_ {t + 1} (w _ {\mathrm {r e f}}) - \ell_ {t + 1} (w _ {t})\right)\right) \mid \mathcal {F} _ {t} \right] \\ \end{array}
$$
where now inside the expectation, we have the term  $\mathcal{R}(w_t) - \mathcal{R}(w_{\mathrm{ref}}) + \ell_{t + 1}(w_{\mathrm{ref}}) - \ell_{t + 1}(w_t)$  which has expectation 0. Let  $C_4 = C_1 + C_2(1 + \| w_{\mathrm{ref}}\|)$ , we can bound
$$
\left| \frac {\eta}{2} \left(\mathcal {R} (w _ {t}) - \mathcal {R} (w _ {\text {r e f}}) + \ell_ {t + 1} (w _ {\text {r e f}}) - \ell_ {t + 1} (w _ {t})\right) \right| \leq \eta C _ {4} \| w _ {\text {r e f}} - w _ {t} \|
$$
and  $z_{t} \leq \frac{1}{\eta C_{4}\|w_{\mathrm{ref}} - w_{t}\|}$ . Now we can apply Lemma 14 (Appendix) to bound