SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Considering a Wasserstein distance in Equation (2) makes our algorithm more flexible than in Equation (3), the KL divergence implies absolute continuity w.r.t. the prior  $\pi$ . Such an assumption is not required to use Equation (2) and covers the case of prior Dirac distributions. Finally, Equation (2) relies on a fixed value  $K$  whose value is discussed below.
Role of  $K$ . We study the cases  $K = 1$ ,  $\sqrt{m}$ , and  $m$  in Theorem 2. We refer to Appendix A for a detailed treatment. First of all, when  $K = 1$ , we recover a classical batch learning setting where all data are collected at once. In this case, we have a single Wasserstein with no convergence rate coupled with a statistical ersatz of  $\sqrt{\ln(1 / \delta) / m}$ . However, similarly to [AEMM22, Theorem 12], in the case of a finite predictor class, we are able to recover an explicit convergence rate. The case  $K = \sqrt{m}$  provides a tradeoff between the number of points required to have good data-dependent priors (which may lead to a small  $\sum_{i=1}^{\sqrt{m}} \mathrm{W}(\rho, \pi_i)$ ) and the number of sets required to have an explicit convergence rate. Finally, the case  $K = m$  leads to a vacuous bound as we have the incompressible term  $\sqrt{\ln(m / \delta)}$ , which makes the bound vacuous for large values of  $m$ . This means that the batch setting is not fitted to deal with a data stream arriving sequentially. To mitigate that weakness, we propose in Section 3.2 the first online PAC-Bayes bounds with Wasserstein distances.
3.2 Wasserstein-based generalisation bounds for online learning
Here, we use the online setting described in Section 2 and derive the first online PAC-Bayes bounds involving Wasserstein distances in Theorems 3 and 4. Online PAC-Bayes bounds are meant to derive online counterparts of classical PAC-Bayesian algorithms [HG22], where the KL-divergence acts as a regulariser. We show in Theorems 3 and 4 that it is possible to consider online PAC-Bayesian algorithms where the regulariser is a Wasserstein distance, which allows us to optimise on measure spaces without a restriction of absolute continuity.
Theorem 3. We assume our loss  $\ell$  to be  $L$ -Lipschitz. Then, for any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$  over the sample  $\mathcal{S}$ , the following holds for the distributions  $\pi_{i,\mathcal{S}} \coloneqq \pi_i(\mathcal{S},.)$  and for any sequence  $(\rho_i)_{i=1\dots m} \in \mathcal{M}(\mathcal{H})^m$ :
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \mathbb {E} _ {h _ {i} \sim \rho_ {i}} \left[ \mathbb {E} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \mid \mathcal {F} _ {i - 1} \right] - \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] \leq 2 L \sum_ {i = 1} ^ {m} \mathrm {W} \left(\rho_ {i}, \pi_ {i, S}\right) \\ + \frac {\lambda}{2} \sum_ {i = 1} ^ {m} \underset {h _ {i} \sim \pi_ {i, S}} {\mathbb {E}} \left[ \hat {V} _ {i} (h _ {i}, \mathbf {z} _ {i}) + V _ {i} (h _ {i}) \right] + \frac {\ln (1 / \delta)}{\lambda}, \\ \end{array}
$$
where for all  $i$ ,  $\hat{V}_i(h_i, \mathbf{z}_i) = (\ell(h_i, \mathbf{z}_i) - \mathbb{E}_{i-1}[\ell(h_i, \mathbf{z}_i)])^2$  is the conditional empirical variance at time  $i$  and  $V_i(h_i) = \mathbb{E}_{i-1}[\hat{V}(h_i, \mathbf{z}_i)]$  is the true conditional variance.
The proof is deferred to Appendix B.4. We also provide the following bound, being an online analogous of Theorem 2, valid for non-negative heavy-tailed losses.
Theorem 4. We assume our loss  $\ell$  to be non-negative and  $L$ -Lipschitz. We also assume that, for any  $i, S, \mathbb{E}_{h \sim \pi_i(.S)}[\mathbb{E}_{i-1}[\ell(h, \mathbf{z}_i)^2]] \leq 1$  (bounded conditional order 2 moments for priors). Then, for any  $\delta \in (0, 1]$ , with probability at least  $1 - \delta$  over the sample  $S$ , any online predictive sequence (used as priors)  $(\pi_i)_{i \geq 1}$ , we have with probability at least  $1 - \delta$  over the sample  $S \sim \mu$ , the following, holding for the data-dependent measures  $\pi_{i,S} \coloneqq \pi_i(S,.)$  and any posterior sequence  $(\rho_i)_{i \geq 1}$ :
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \underset {h _ {i} \sim \rho_ {i}} {\mathbb {E}} \left[ \mathbb {E} [ \ell (h _ {i}, \mathbf {z} _ {i}) \mid \mathcal {F} _ {i - 1} ] - \ell (h _ {i}, \mathbf {z} _ {i}) \right] \leq \frac {2 L}{m} \sum_ {i = 1} ^ {m} \mathrm {W} (\rho_ {i}, \pi_ {i, \mathcal {S}}) + \sqrt {\frac {2 \ln \left(\frac {1}{\delta}\right)}{m}}.
$$
The proof is deferred to Appendix B.5.
Analysis of our bounds. Theorems 3 and 4 are, to our knowledge, the first results involving Wasserstein distances for online PAC-Bayes learning. They are the online counterpart of Theorems 1 and 2, and the discussion of Section 3.1 about the involved assumptions also apply here. The sum of
Wasserstein distances involved here is a consequence of the online setting and must grow sublinearly for the bound to be tight. For instance, when  $(\rho_{i} = \delta_{h_{i}})_{i\geq 1}$  is the output of an online algorithm, outputting Dirac measures and  $\pi_{i,S} = \rho_{i - 1}$ , the sum of Wasserstein is exactly  $\sum_{i = 1}^{m}d(h_{i},h_{i - 1})$ . This sum has to be sublinear for the bound to be non-vacuous, and the tightness depends on the considered learning problem. An analogous of this sum can be found in dynamic online learning [Zin03] where similar sums appear as path lengths to evaluate the complexity of the problem.
Comparison with literature. We compare our results to existing PAC-Bayes bounds for martingales of  $\left[\mathrm{SLCB}^{+}12\right]$ .  $\left[\mathrm{SLCB}^{+}12, \text{Theorem 4}\right]$  is a PAC-Bayes bound for martingales, which controls an average of martingales, similar to our Theorem 1. Under a boundedness assumption, they recover a McAllester-typed bound, while Theorem 1 is more of a Catoni-typed result. Also,  $\left[\mathrm{SLCB}^{+}12, \text{Theorem 7}\right]$  is a Catoni-typed bound involving a conditional variance, similar to our Theorem 4. They require to bound uniformly the variance on all the predictor sets, while we only assume averaged variance with respect to priors, which is what we required to perform Theorem 4.
A new online algorithm. [HG22] derived from their main theorem, an online counterpart of Equation (3), proving it comes with guarantees. Similarly, we exploit Theorem 4 to derive the online counterpart of Equation (2), from the data-free initialisation  $\rho_{1}$
$$
\forall i \geq 1, \rho_ {i} \in \underset {\rho \in \mathcal {M} (\mathcal {H})} {\operatorname {a r g m i n}} \underset {h \sim \rho} {\mathbb {E}} [ \ell (h _ {i}, \mathbf {z} _ {i}) ] + 2 L W (\rho , \pi_ {i, S}). \tag {4}
$$
We highlight the merits of the algorithm defined by Equation (4), alongside with the one from Equation (2), in Section 4.
4 Learning via Wasserstein regularisation
Theorems 2 and 4 are designed to be informative on the generalisation ability of a single hypothesis even when Dirac distributions are considered. In particular, our results involve Wasserstein distances acting as regularisers on  $\mathcal{H}$ . In this section, we show that a Wasserstein regularisation of the learning objective, which comes from our theoretical bounds, helps to better generalise in practice. Inspired by Equations (2) and (4), we derive new PAC-Bayesian algorithms for both batch and online learning involving a Wasserstein distance (see Section 4.1), we describe our experimental framework in Section 4.2 and we present some of the results in Section 4.3. Additional details, experiments, and discussions are gathered in Appendix C due to space constraints. All the experiments are reproducible with the source code provided on GitHub at https://github.com/paulviallard/NeurIPS23-PB-Wasserstein.
4.1 Learning algorithms
Classification. In the classification setting, we assume that the data space  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}$  is composed of a  $d$ -dimensional input space  $\mathcal{X} = \{\mathbf{x}\in \mathbb{R}^d\mid \| \mathbf{x}\| _2\leq 1\}$  and a finite label space  $\mathcal{V} = \{1,\dots ,|\mathcal{V}|\}$  with  $|\mathcal{V}|$  labels. We aim to learn models  $h_{\mathbf{w}}:\mathbb{R}^{d}\to \mathbb{R}^{|\mathcal{V}|}$  parameterised by a weight vector  $\mathbf{w}$  that outputs, given an input  $\mathbf{x}\in \mathcal{X}$ , a score  $h_\mathrm{w}(\mathbf{x})[y']\in \mathbb{R}$  for each label  $y^\prime$ . This score allows us to assign a label to  $\mathbf{x}\in \mathcal{X}$ ; to check if  $h_\mathrm{w}$  classifies correctly the example  $(\mathbf{x},y)$ , we use the classification loss defined by  $\ell^c (h_{\mathbf{w}},(\mathbf{x},y))\coloneqq \mathbb{1}[h_{\mathbf{w}}(\mathbf{x})[y] - \max_{y'\neq y}h_{\mathbf{w}}(\mathbf{x})[y']\leq 0]$ , where  $\mathbb{1}$  denotes the indicator function.
Batch algorithm. In the batch setting, we aim to learn a parametrised hypothesis  $h_{\mathbf{w}} \in \mathcal{H}$  that minimises the population classification risk  $\Re_{\mu}(h_{\mathbf{w}}) = \mathbb{E}_{(\mathbf{x},y)\sim \mu}\ell^{c}(h_{\mathbf{w}},(\mathbf{x},y))$  that we can only estimate through the empirical classification risk  $\Re_{\mathcal{S}}(h_{\mathbf{w}}) = \frac{1}{m}\sum_{i = 1}^{m}\ell^{c}(h_{\mathbf{w}},(\mathbf{x}_{i},y_{i}))$ . To learn the hypothesis, we start from Equation (2), when the distributions  $\rho$  and  $\pi_1,\ldots ,\pi_K$  are Dirac masses, localised at  $h_{\mathbf{w}},h_{\mathbf{w}_1},\dots h_{\mathbf{w}_K} \in \mathcal{H}$  respectively. Indeed, in this case,  $\mathrm{W}(\rho ,\pi_{i,S}) = d(h_{\mathbf{w}},h_{\mathbf{w}_i})$  for any  $i$ . However, the loss  $\ell^c (.,\mathbf{z})$  is not Lipschitz and the derivatives are zero for all examples  $\mathbf{z} \in \mathcal{X} \times \mathcal{Y}$ , which prevents its use in practice to obtain such a hypothesis  $h_{\mathbf{w}}$ . Instead, for the population risk  $R_{\mu}(h)$  and the empirical risk  $\hat{R}_{\mathcal{S}}(h)$  (in Theorem 2 and Equation (2)), we consider the loss  $\ell (h,(\mathbf{x},y)) = \frac{1}{|\mathcal{V}|}\sum_{y'\neq y}\max (0,1 - \eta (h[y] - h[y]))$ , which is  $\eta$ -Lipschitz w.r.t. the outputs  $h[1],\ldots ,h[|\mathcal{V}|]$ . This loss has subgradients everywhere, which is convenient in practice. We go a
step further by (a) setting  $L = \frac{1}{2}$  and (b) adding a parameter  $\varepsilon > 0$  to obtain the objective
$$
\underset {h _ {\mathbf {w}} \in \mathcal {H}} {\operatorname {a r g m i n}} \left\{\hat {\mathrm {R}} _ {\mathcal {S}} \left(h _ {\mathbf {w}}\right) + \varepsilon \left[ \sum_ {i = 1} ^ {K} \frac {\left| \mathcal {S} _ {i} \right|}{m} d \left(h _ {\mathbf {w}}, h _ {\mathbf {w} _ {i}}\right) \right] \right\}. \tag {5}
$$
To (approximately) solve Equation (5), we propose a two-step algorithm. First, PRIORS LEARNING learns  $K$  hypotheses  $h_{\mathbf{w}_1}, \ldots, h_{\mathbf{w}_K} \in \mathcal{H}$  by minimising the empirical risk via stochastic gradient descent. Second, POSTERIOR LEARNING learns the hypothesis  $h_{\mathbf{w}} \in \mathcal{H}$  by minimising the objective associated with Equation (5). More precisely, PRIORS LEARNING outputs the hypotheses  $h_{\mathbf{w}_1}, \ldots, h_{\mathbf{w}_K}$ , obtained by minimising the empirical risk through mini-batches. Those batches are designed such that for any  $i$ , the hypothesis  $h_{\mathbf{w}_i}$  does not depend on  $S_i$ . Then, given  $h_{\mathbf{w}_1}, \ldots, h_{\mathbf{w}_K} \in \mathcal{H}$ , POSTERIOR LEARNING minimises the objective in Equation (5) with minibatches. Those algorithms are presented in Algorithm 1 of Appendix C. While  $\varepsilon$  is not suggested by Equation (2), it helps to control the impact of the regularisation in practice. Equation (5) then optimises a tradeoff between the empirical risk and the regularisation term  $\varepsilon \sum_{i=1}^{K} \frac{|S_i|}{m} d(h_{\mathbf{w}}, h_{\mathbf{w}_i})$ .
Online algorithm. Online algorithms output, at each time step  $i \in \{1, \dots, m\}$ , a new hypothesis  $h_{\mathbf{w}_i}$ . From Equation (4), particularised to a sequence of Dirac distributions (localised in  $h_{\mathbf{w}_1}, \dots, h_{\mathbf{w}_K}$ ), we design a novel online PAC-Bayesian algorithm with a Wasserstein regulariser:
$$
\forall i \geq 1, h _ {i} \in \underset {h _ {\mathbf {w}} \in \mathcal {H}} {\operatorname {a r g m i n}} \ell \left(h _ {\mathbf {w}}, \mathbf {z} _ {i}\right) + d \left(h _ {\mathbf {w}}, h _ {\mathbf {w} _ {i - 1}}\right) \quad s. t. d \left(h _ {\mathbf {w}}, h _ {\mathbf {w} _ {i - 1}}\right) \leq 1. \tag {6}
$$
According to Theorem 4, such an algorithm aims to bound the population cumulative classification loss  $\mathfrak{C}_{\mu} = \sum_{i=1}^{m} \mathbb{E}[\ell^{c}(h_{\mathbf{w}_{i}}, \mathbf{z}_{i}) \mid \mathcal{F}_{i-1}]$ . Note that we added the constraint  $d(h_{\mathbf{w}}, h_{\mathbf{w}_{i-1}}) \leq 1$  compared to Equation (4). This constraint ensures that the new hypothesis  $h_{\mathbf{w}_{i}}$  is not too far from  $h_{\mathbf{w}_{i-1}}$  (in the sense of the distance  $\|\cdot\|_{2}$ ). Note that the constrained optimisation problem in Equation (6) can be rewritten in an unconstrained form (see [BV04]) thanks to a barrier  $B(\cdot)$  defined by  $B(a) = 0$  if  $a \leq 0$  and  $B(a) = +\infty$  otherwise; we have
$$
\forall i \geq 1, h _ {i} \in \underset {h _ {\mathbf {w}} \in \mathcal {H}} {\operatorname {a r g m i n}} \ell \left(h _ {\mathbf {w}}, \mathbf {z} _ {i}\right) + d \left(h _ {\mathbf {w}}, h _ {\mathbf {w} _ {i - 1}}\right) + B \left(d \left(h _ {\mathbf {w}}, h _ {\mathbf {w} _ {i - 1}}\right) - 1\right). \tag {7}
$$
When solving the problem in Equation (7) is not feasible, we approximate it with a log barrier of  $\left[\mathrm{KDY}^{+}22\right]$  (suitable in a stochastic gradient setting); given a parameter  $t > 0$ , the log barrier extension is defined by  $\hat{B}(a) = -\frac{1}{t}\ln(-a)$  if  $a \leq -\frac{1}{t^2}$  and  $\hat{B}(a) = ta - \frac{1}{t}\ln(\frac{1}{t^2}) + \frac{1}{t}$  otherwise. We present in Appendix C Algorithm 2 that aims to (approximately) solve Equation (7). To do so, for each new example  $(\mathbf{x}_i, y_i)$ , the algorithm runs several gradient descent steps to optimise Equation (7).
4.2 Experimental framework
In this part, we assimilate the predictor space  $\mathcal{H}$  to the parameter space  $\mathbb{R}^d$ . Thus, the distance  $d$  is the Euclidean distance between two parameters:  $d(h_{\mathbf{w}}, h_{\mathbf{w}'}) = \| \mathbf{w} - \mathbf{w}'\|_2$ . This implies that the Lipschitzness of  $\ell$  has to be taken w.r.t.  $\mathbf{w}$  instead of  $h_{\mathbf{w}}$ .
Models. We consider that the models are either linear or neural networks (NN). Linear models are defined by  $h_{\mathbf{w}}(\mathbf{x}) = W\mathbf{x} + b$ , where  $W \in \mathbb{R}^{|\mathcal{Y}| \times d}$  is the weight matrix,  $b \in \mathbb{R}^{|\mathcal{Y}|}$  is the bias, and  $\mathbf{w} = \mathrm{vec}(\{W, b\})$  its vectorisation; the vector  $\mathbf{w}$  with the zero vector. Thanks to the definition of  $\mathcal{X}$ , we know from Lemma 8 (and the composition of Lipschitz functions) that the loss is  $\sqrt{2}\eta$ -Lipschitz w.r.t.  $\mathbf{w}$ . For neural networks, we consider fully connected ReLU neural networks with  $L$  hidden layers and  $D$  nodes, where the leaky ReLU activation function  $\mathrm{ReLU}: \mathbb{R}^D \to \mathbb{R}^D$  applies elementwise  $x \mapsto \max(x, 0.01x)$ . More precisely, the network is defined by  $h_{\mathbf{w}}(\mathbf{x}) = Wh^{L}(\cdots h^{1}(\mathbf{x})) + b$  where  $W \in \mathbb{R}^{|\mathcal{Y}| \times D}$ ,  $b \in \mathbb{R}^{|\mathcal{Y}|}$ . Each layer  $h^{i}(\mathbf{x}) = \mathrm{ReLU}(W_{i}\mathbf{x} + b_{i})$  has a weight matrix  $W_{i} \in \mathbb{R}^{D \times D}$  and bias  $b_{i} \in \mathbb{R}^{D}$  except for  $i = 1$  where we have  $W_{1} \in \mathbb{R}^{D \times d}$ . The weights  $\mathbf{w}$  are also the vectorisation  $\mathbf{w} = \mathrm{vec}(\{W, W_{L}, \ldots, W_{1}, b, b_{L}, \ldots, b_{1}\})$ . We have precised in Lemma 9 that our loss is Lipschitz w.r.t. the weights  $\mathbf{w}$ . We initialise the network similarly to [DR17] by sampling the weights from a Gaussian distribution with zero mean and a standard deviation of  $\sigma = 0.04$ ; the weights are further clipped between  $-2\sigma$  and  $+2\sigma$ . Moreover, the values in the biases  $b_{1}, \ldots, b_{L}$  are set to 0.1, while the values for  $b$  are set to 0. In the following, we consider  $D = 600$  and  $L = 2$ ; more experiments are considered in the appendix.
Optimisation. To perform the gradient steps, we use the COCOB-Backprop optimiser [OT17] (with parameter  $\alpha = 10000$ ). This optimiser is flexible as the learning rate is adaptative and, thus, does
2The parameter  $\alpha$  in COCOB-Backprop can be seen as an initial learning rate; see [OT17].
not require hyperparameter tuning. For Algorithm 1, which solves Equation (5), we fix a batch size of 100, i.e.,  $|\mathcal{U}| = 100$ , and the number of epochs  $T$  and  $T'$  are fixed to perform at least 20000 iterations. Regarding Algorithm 2, which solves Equation (7), we set  $t = 100$  for the log barrier, which is enough to constrain the weights and the number of iterations to  $T = 10$ .
Datasets. We study the performance of Algorithms 1 and 2 on UCI datasets [DG17] along with MNIST [LeC98] and FashionMNIST [XRV17]. We also split all the data (from the original training/test set) in two halves; the first part of the data serves in the algorithm (and is considered as a training set), while the second part is used to approximate the population risks  $\Re_{\mu}(h)$  and  $\mathfrak{C}_{\mu}$  (and considered as a testing set).
We present in Table 1 the performance of Algorithms 1 and 2 compared to the Empirical Risk Minimisation (ERM) and the Online Gradient Descent (OGD) with the COCOB-Backprop optimiser. Tables 1a and 1c present the results of Algorithm 1 for the i.i.d. setting on linear and neural networks respectively, while Tables 1b and 1d present the results of Algorithm 2 for the online case.
Analysis of the results. In batch learning, we note that the regularisation term brings generalisation improvements compared to the empirical risk minimisation. Indeed, our batch algorithm (Algorithm 1) has a lower population risk  $\Re_{\mu}(h)$  on 11 datasets for the linear models and 9 datasets for the neural networks. In particular, notice that NNs obtained from Algorithm 1 are more efficient than the ones obtained from ERM on MNIST and FASHIONMNIST, which are the more challenging datasets. This suggests that the regularisation term helps to generalise well. For the online case, the performance of the linear models obtained from our algorithm (Algorithm 2) and by OGD are comparable: we have a tighter population classification risk  $\Re_{\mu}(h)$  on 5 datasets over 13. However, notice that the risk difference is less than 0.05 on 6 datasets. The advantage of Algorithm 2 is more pronounced for neural networks: we improve the performance in all datasets except ADULT and SENSORLESS. Hence, this confirms that optimising the regularised loss  $\ell(h_{\mathbf{w}}, \mathbf{z}_i) + \|\mathbf{w} - \mathbf{w}_{i-1}\|$  brings a good advantage compared to the loss  $\ell(h_{\mathbf{w}}, \mathbf{z}_i)$  only. A possible explanation would be that OGD suffers from underfitting (with a high empirical risk  $\mathfrak{C}_{\mu}$ ) while we are able to control overfitting through a regularisation term. Indeed, only one gradient descent step is done for each new datum  $(\mathbf{x}_i, y_i)$ , which might not be sufficient to decrease the loss. Instead, our method solves the problem associated with Equation (7) and constrains the descent with the norm  $\|\mathbf{w} - \mathbf{w}_{i-1}\|$ .
5 Conclusion and Perspectives
We derived novel generalisation bounds based on the Wasserstein distance, both for batch and online learning, allowing for the use of deterministic hypotheses through PAC-Bayes. We derived new learning algorithms which are inspired by the bounds, with remarkable empirical performance on a number of datasets: we hope our work can pave the way to promising future developments (both theoretical and practical) of generalisation bounds based on the Wasserstein distance. Given the mostly theoretical nature of our work, we do not foresee an immediate negative societal impact, although we hope a better theoretical understanding of generalisation will ultimately benefit practitioners of machine learning algorithms and encourage virtuous initiatives.
We warmly thank the reviewers who provided insightful comments and suggestions which greatly helped us improve our manuscript. P.V. and U.S. are partially supported by the French government under the management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). U.S. is also partially supported by the European Research Council Starting Grant DYNAMY - 101039676. B.G. acknowledges partial support by the U.S. Army Research Laboratory and the U.S. Army Research Office, and by the U.K. Ministry of Defence and the U.K. Engineering and Physical Sciences Research Council (EPSRC) under grant number EP/R013616/1. B.G. acknowledges partial support from the French National Agency for Research, grants ANR-18-CE40-0016-01 and ANR-18-CE23-0015-02.
Table 1: Performance of Algorithms 1 and 2 compared respectively to ERM and OGD on different datasets on linear and neural network models. For the i.i.d. setting, we consider  $\varepsilon = 1 / m$  and  $\varepsilon = 1 / \sqrt{m}$  and with  $K = 0.2\sqrt{m}$ . For each method, we plot the empirical risk  $\Re_S(h)$  or  $\mathfrak{C}_S$  with its associated test risk  $\Re_{\mu}(h)$  or  $\mathfrak{C}_{\mu}$ . The risk in bold corresponds to the lowest one among the ones considered. For the online case, the two population risks are underlined when the absolute difference is lower than 0.05.
(a) Linear model - batch learning
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)	ERM
RS(h)	Rμ(h)	RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	.165	.166	.165	.167	.166	.167
FASHIONMNIST	.128	.151	.126	.148	.139	.153
LETTER	.285	.297	.287	.296	.287	.297
MNIST	.200	.216	.066	.092	.065	.091
MUSHROOMS	.001	.001	.001	.001	.001	.001
NURSERY	.766	.773	.760	.773	.794	.807
PENDIGITS	.049	.059	.050	.061	.052	.064
PHISHING	.063	.067	.065	.069	.064	.067
SATIMAGE	.144	.200	.138	.201	.148	.209
SEGMENTATION	.057	.216	.164	.386	.087	.232
SENSORLESS	.129	.129	.131	.131	.134	.136
TICTACTOE	.388	.299	.013	.021	.228	.238
YEAST	.527	.497	.524	.504	.470	.427
(b) Linear model - online learning
Alg. 2	OGD
εS	εμ	εS	εμ
.230	.236	.248	.248
.223	.282	.540	.548
.919	.935	.916	.926
.284	.310	.378	.397
.218	.222	.082	.087
.794	.807	.789	.805
.342	.484	.589	.600
.226	.242	.226	.220
.669	.938	.635	.888
.749	.803	.738	.893
.906	.910	.825	.830
.443	.468	.390	.303
.699	.713	.667	.708
(c) NN model - batch learning
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)	ERM
RS(h)	Rμ(h)	RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	.164	.164	.166	.165	.165	.163
FASHIONMNIST	.159	.163	.156	.160	.163	.167
LETTER	.259	.272	.250	.260	.258	.270
MNIST	.112	.120	.084	.094	.119	.127
MUSHROOMS	.000	.000	.000	.000	.000	.000
NURSERY	.706	.719	.706	.719	.706	.719
PENDIGITS	.009	.023	.021	.032	.009	.022
PHISHING	.042	.050	.039	.054	.046	.055
SATIMAGE	.132	.184	.149	.172	.141	.189
SEGMENTATION	.145	.250	.189	.373	.174	.389
SENSORLESS	.076	.079	.077	.079	.075	.078
TICTACTOE	.392	.301	.000	.038	.000	.023
YEAST	.679	.666	.487	.478	.644	.682
(d) NN model - online learning
Alg. 2	OGD
εS	εμ	εS	εμ
.241	.254	.248	.248
.096	.327	.397	.446
.829	.945	.958	.963
.092	.265	.470	.521
.082	.122	.202	.217
.800	.805	.793	.806
.323	.537	.871	.879
.164	.222	.331	.318
.401	.763	.626	.857
.619	.857	.739	.913
.899	.910	.622	.633
.388	.309	.397	.309
.662	.720	.702	.720