SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the inequality comes from the Kantorovich-Rubinstein duality theorem. Let  $f(S_{i}) = \mathbb{E}_{h\sim \pi_{i}}\frac{1}{m}\left[|\mathcal{S}_{i}|\mathrm{R}_{\mu}(h) - \sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell (h,\mathbf{z}_{i})\right]$ , the function has the bounded difference inequality, i.e., for two datasets  $S_{i}$  and  $S_{i}^{\prime}$  that differs from one example (the  $k$ -th example, without loss of generality), we have
$$
\begin{array}{l} |f(\mathcal{S}_{i}) - f(\mathcal{S}_{i}^{\prime})| = \left|\underset {h\sim \pi_{i}}{\mathbb{E}}\frac{1}{m}\left[|\mathcal{S}_{i}|\mathrm{R}_{\mu}(h) - \sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell (h,\mathbf{z})\right] - \underset {h\sim \pi_{i}}{\mathbb{E}}\frac{1}{m}\left[|\mathcal{S}_{i}|\mathrm{R}_{\mu}(h) - \sum_{\mathbf{z}^{\prime}\in \mathcal{S}_{i}^{\prime}}\ell (h,\mathbf{z}^{\prime})\right]\right| \\ = \left| \underset {h \sim \pi_ {i}} {\mathbb {E}} \left[ \frac {1}{m} | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) - \frac {1}{m} \sum_ {\mathbf {z} \in \mathcal {S} _ {i}} \ell (h, \mathbf {z}) - \frac {1}{m} | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) + \frac {1}{m} \sum_ {\mathbf {z} ^ {\prime} \in \mathcal {S} _ {i} ^ {\prime}} \ell (h, \mathbf {z} ^ {\prime}) \right] \right| \\ = \left| \mathbb {E} _ {h \sim \pi_ {i}} \left[ \frac {1}{m} \sum_ {\mathbf {z} ^ {\prime} \in S _ {i} ^ {\prime}} \ell (h, \mathbf {z} ^ {\prime}) - \frac {1}{m} \sum_ {\mathbf {z} \in S _ {i}} \ell (h, \mathbf {z}) \right] \right| \\ = \left| \underset {h \sim \pi_ {i}} {\mathbb {E}} \left[ \frac {1}{m} \ell (h, \mathbf {z} _ {k} ^ {\prime}) - \frac {1}{m} \ell (h, \mathbf {z} _ {k}) \right] \right| \\ \leq \frac {1}{m}. \\ \end{array}
$$
Hence, from Mcdiarmid's inequality, we have with probability at least  $1 - \frac{\delta}{K}$  over  $S \sim \mu^m$
$$
\begin{array}{l} \underset {h \sim \pi_ {i}} {\mathbb {E}} \frac {1}{m} \left[ | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) - \sum_ {\mathbf {z} \in \mathcal {S} _ {i}} \ell (h, \mathbf {z}) \right] \\ \leq \underset {\mathcal {S} \sim \mu^ {m}} {\mathbb {E}} \underset {h \sim \pi_ {i}} {\mathbb {E}} \frac {1}{m} \left[ | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) - \sum_ {\mathbf {z} \in \mathcal {S} _ {i}} \ell (h, \mathbf {z}) \right] + \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}} \\ = \underset {\mathcal {S} _ {i} ^ {c} \sim \mu^ {m - | \mathcal {S} _ {i} |}} {\mathbb {E}} \underset {\mathcal {S} _ {i} \sim \mu^ {| \mathcal {S} _ {i} |}} {\mathbb {E}} \underset {h \sim \pi_ {i}} {\mathbb {E}} \frac {1}{m} \left[ | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) - \sum_ {\mathbf {z} \in \mathcal {S} _ {i}} \ell (h, \mathbf {z}) \right] + \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}} \\ = \underset {\mathcal {S} _ {i} ^ {c} \sim \mu^ {m - | \mathcal {S} _ {i} |} h \sim \pi_ {i}} {\mathbb {E}} \underset {h \sim \pi_ {i}} {\mathbb {E}} \frac {1}{m} \left[ | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) - \underset {\mathcal {S} _ {i} \sim \mu^ {| \mathcal {S} _ {i} |}} {\mathbb {E}} \sum_ {\mathbf {z} \in \mathcal {S} _ {i}} \ell (h, \mathbf {z}) \right] + \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}} \\ = \underset {\mathcal {S} _ {i} ^ {c} \sim \mu^ {m - | \mathcal {S} _ {i} |} h \sim \pi_ {i}} {\mathbb {E}} \underset {h \sim \pi_ {i}} {\mathbb {E}} \frac {1}{m} \left[ | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) - | \mathcal {S} _ {i} | \mathrm {R} _ {\mu} (h) \right] + \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}} \\ = \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}}. \\ \end{array}
$$
From the union bound, we have with probability at least  $1 - \delta$  over  $\mathcal{S} \sim \mu^m$ , for any  $\rho \in \mathcal{M}(\mathcal{H})$ ,
$$
\underset {h \sim \rho} {\mathbb {E}} \left[ \mathbf {R} _ {\mu} (h) - \hat {\mathbf {R}} _ {\mathcal {S}} (h) \right] \leq \sum_ {i = 1} ^ {K} \frac {2 | \mathcal {S} _ {i} | L}{m} \mathrm {W} (\rho , \pi_ {i}) + \sum_ {i = 1} ^ {K} \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}},
$$
which is the claimed result.
We are now able to give a corollary of Theorem 6.
Corollary 7. We assume our loss  $\ell$  to be in  $[0,1]$  and  $L$ -Lipschitz. Then, for any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$  over the sample  $\mathcal{S}$ , the following holds for the hypotheses  $h_{i,\mathcal{S}} \in \mathcal{H}$  associated with the Dirac distributions  $\pi_{i,\mathcal{S}}$  and for any  $h \in \mathcal{H}$ :
$$
R _ {\mu} (h) \leq \hat {R} _ {\mathcal {S}} (h) + \sum_ {i = 1} ^ {K} \frac {2 | \mathcal {S} _ {i} | L}{m} d (h, h _ {i, \mathcal {S}}) + \sum_ {i = 1} ^ {K} \sqrt {\frac {| \mathcal {S} _ {i} | \ln \frac {K}{\delta}}{2 m ^ {2}}}.
$$
Such a bound was impossible to obtain from the PAC-Bayesian bounds based on a KL divergence. Indeed, the KL divergence is infinite for two distributions with disjoint supports. Hence, the PAC-Bayesian framework based on the Wasserstein distance allows us to provide uniform-convergence bounds from a proof technique different from the ones based on the Rademacher complexity [KP00, BM01, BM02] or the VC-dimension [VC68, VC74]. In Section 4, we provide an algorithm minimising such a bound.
B.4 Proof of Theorem 3
Theorem 3. We assume our loss  $\ell$  to be  $L$ -Lipschitz. Then, for any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$  over the sample  $\mathcal{S}$ , the following holds for the distributions  $\pi_{i,\mathcal{S}} \coloneqq \pi_i(\mathcal{S},\cdot)$  and for any sequence  $(\rho_i)_{i = 1\dots m} \in \mathcal{M}(\mathcal{H})^m$ :
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \mathbb {E} _ {h _ {i} \sim \rho_ {i}} \left[ \mathbb {E} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \mid \mathcal {F} _ {i - 1} \right] - \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] \leq 2 L \sum_ {i = 1} ^ {m} \mathrm {W} \left(\rho_ {i}, \pi_ {i, \mathcal {S}}\right) \\ + \frac {\lambda}{2} \sum_ {i = 1} ^ {m} \underset {h _ {i} \sim \pi_ {i, S}} {\mathbb {E}} \left[ \hat {V} _ {i} (h _ {i}, \mathbf {z} _ {i}) + V _ {i} (h _ {i}) \right] + \frac {\ln (1 / \delta)}{\lambda}, \\ \end{array}
$$
Proof. First of all, note that for a  $L$ -Lipschitz loss function  $\ell : \mathcal{H} \times \mathcal{Z} \to \mathbb{R}$ , we have
$$
\left| \left(\underset {i - 1} {\mathbb {E}} [ \ell (h _ {i}, \mathbf {z} _ {i}) ] - \ell (h _ {i}, \mathbf {z} _ {i})\right) - \left(\underset {i - 1} {\mathbb {E}} [ \ell (h _ {i} ^ {\prime}, \mathbf {z} _ {i}) ] - \ell (h _ {i} ^ {\prime}, \mathbf {z} _ {i})\right) \right| \leq 2 L d \left(h _ {i}, h _ {i} ^ {\prime}\right). \tag {15}
$$
Indeed, we can deduce Equation (15) from Jensen inequality, the triangle inequality, and by definition that we have
$$
\begin{array}{l} \left| \left(\underset {i - 1} {\mathbb {E}} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] - \ell \left(h _ {i}, \mathbf {z} _ {i}\right)\right) - \left(\underset {i - 1} {\mathbb {E}} \left[ \ell \left(h _ {i} ^ {\prime}, \mathbf {z} _ {i}\right) \right] - \ell \left(h _ {i} ^ {\prime}, \mathbf {z} _ {i}\right)\right) \right| \\ \leq \mathbb {E} _ {i - 1} \left[ | \ell \left(h _ {i}, \mathbf {z} _ {i} ^ {\prime}\right) - \ell \left(h _ {i} ^ {\prime}, \mathbf {z} _ {i} ^ {\prime}\right) | + | \ell \left(h _ {i}, \mathbf {z} _ {i}\right) - \ell \left(h _ {i} ^ {\prime}, \mathbf {z} _ {i}\right) | \right] \\ \leq \underset {i - 1} {\mathbb {E}} 2 L d (h _ {i}, h _ {i} ^ {\prime}) = 2 L d (h _ {i}, h _ {i} ^ {\prime}). \\ \end{array}
$$
From the Kantorovich-Rubinstein duality theorem [Vil09, Remark 6.5], we have
$$
\sum_ {i = 1} ^ {m} \mathbb {E} _ {h _ {i} \sim \rho_ {i}} \left[ \mathbb {E} _ {i - 1} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] - \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] \leq 2 L \sum_ {i = 1} ^ {m} W _ {1} \left(\rho_ {i}, \pi_ {i, \mathcal {S}}\right) + \sum_ {i = 1} ^ {m} \mathbb {E} _ {h \sim \pi_ {i, \mathcal {S}}} \left[ \mathrm {R} _ {\mu} \left(h _ {i}\right) - \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right].
$$
Now, we define  $X_{i}(h_{i},\mathbf{z}_{i}) \coloneqq \mathbb{E}_{i - 1}[\ell (h_{i},\mathbf{z}_{i})] - \ell (h_{i},\mathbf{z}_{i})$ . We also recall that for any  $i$ , we have  $\hat{V}_i(h_i,\mathbf{z}_i) = (\ell (h_i,\mathbf{z}_i) - \mathbb{E}_{i - 1}[\ell (h_i,\mathbf{z}_i)])^2$  and  $V_{i}(h_{i}) = \mathbb{E}_{i - 1}[\hat{V} (h_{i},\mathbf{z}_{i})]$ . To apply the supermartingales techniques of [HG23a], we define the following function:
$$
f _ {m} (S, h _ {1}, \dots , h _ {m}) := \sum_ {i = 1} ^ {m} \lambda X _ {i} \left(h _ {i}, \mathbf {z} _ {i}\right) - \frac {\lambda^ {2}}{2} \sum_ {i = 1} ^ {m} \left(\hat {V} _ {i} \left(h _ {i}, \mathbf {z} _ {i}\right) + V _ {i} \left(h _ {i}\right)\right).
$$
Now, Lemma 3.2 of [HG23a] state that the sequence  $(SM_{m})_{m\geq 1}$  defined for any  $m$  as:
$$
S M _ {m} := \underset {(h _ {1}, \dots , h _ {m}) \sim \pi_ {1, \mathcal {S}} \otimes \dots \otimes \pi_ {m, \mathcal {S}}} {\mathbb {E}} \left[ \exp \left(f _ {m} (\mathcal {S}, h _ {1}, \dots , h _ {m})\right) \right],
$$
is a supermartingale. We exploit this fact as follows:
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \mathbb {E} _ {h \sim \rho_ {i - 1}} \left[ \mathbb {E} _ {i - 1} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] - \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] = \mathbb {E} _ {\left(h _ {1}, \dots , h _ {m}\right) \sim \pi_ {1, S} \otimes \dots \otimes \pi_ {m, S}} \left[ \sum_ {i = 1} ^ {m} X _ {i} \left(h _ {i}, \mathbf {z} _ {i}\right) \right] \\ = \frac {1}{\lambda} _ {(h _ {1}, \dots , h _ {m}) \sim \pi_ {1, S} \otimes \dots \otimes \pi_ {m, S}} [ f _ {m} (S, h _ {1}, \dots , h _ {m}) ] \\ + \frac {\lambda}{2} \sum_ {i = 1} ^ {m} \mathbb {E} _ {h _ {i} \sim \pi_ {i, S}} \left[ \hat {V} _ {i} \left(h _ {i}, \mathbf {z} _ {i}\right) + V _ {i} \left(h _ {i}\right) \right] \\ \leq \frac {\ln (S M _ {m})}{\lambda} + \frac {\lambda}{2} \sum_ {i = 1} ^ {m} \mathbb {E} _ {h _ {i} \sim \pi_ {i, S}} \left[ \hat {V} _ {i} \left(h _ {i}, \mathbf {z} _ {i}\right) + V _ {i} \left(h _ {i}\right) \right] \\ \end{array}
$$
The last line holds thanks to Jensen's inequality. Now using Ville's inequality ensures us that:
$$
\underset {S} {\mathbb {P}} \left(\forall m, S M _ {m} \leq \frac {1}{\delta}\right) \geq \frac {1}{\delta}.
$$
Thus, with probability  $1 - \delta$ , for any  $m$  we have  $\ln (SM_m) \leq \ln \left(\frac{1}{\delta}\right)$ . This concludes the proof.
B.5 Proof of Theorem 4
Proof. The proof starts similarly to the one of Theorem 3. Indeed, note that for a  $L$ -Lipschitz loss function  $\ell: \mathcal{H} \times \mathcal{Z} \to \mathbb{R}$ , we have
$$
\left| \left(\underset {i - 1} {\mathbb {E}} [ \ell (h _ {i}, \mathbf {z} _ {i}) ] - \ell (h _ {i}, \mathbf {z} _ {i})\right) - \left(\underset {i - 1} {\mathbb {E}} [ \ell (h _ {i} ^ {\prime}, \mathbf {z} _ {i}) ] - \ell (h _ {i} ^ {\prime}, \mathbf {z} _ {i})\right) \right| \leq 2 L d \left(h _ {i}, h _ {i} ^ {\prime}\right). \tag {16}
$$
Indeed, we can deduce Equation (16) from Jensen inequality, the triangle inequality, and by definition that we have
$$
\sum_ {i = 1} ^ {m} \underset {h _ {i} \sim \rho_ {i}} {\mathbb {E}} \left[ \underset {i - 1} {\mathbb {E}} \left[ \ell (h _ {i}, \mathbf {z} _ {i}) \right] - \ell (h _ {i}, \mathbf {z} _ {i}) \right] \leq 2 L \sum_ {i = 1} ^ {m} W _ {1} (\rho_ {i}, \pi_ {i, \mathcal {S}}) + \sum_ {i = 1} ^ {m} \underset {h \sim \pi_ {i, \mathcal {S}}} {\mathbb {E}} \left[ \mathrm {R} _ {\mu} (h _ {i}) - \ell (h _ {i}, \mathbf {z} _ {i}) \right].
$$
Now, we define  $X_{i}(h_{i},\mathbf{z}_{i}) \coloneqq \mathbb{E}_{i - 1}[\ell (h_{i},\mathbf{z}_{i})] - \ell (h_{i},\mathbf{z}_{i})$ . To apply the supermartingales techniques of [CWR23], we define the following function:
$$
f _ {m} (S, h _ {1}, \dots , h _ {m}) := \sum_ {i = 1} ^ {m} \lambda X _ {i} (h _ {i}, \mathbf {z} _ {i}) - \frac {\lambda^ {2}}{2} \sum_ {i = 1} ^ {m} \underset {i - 1} {\mathbb {E}} [ \ell (h _ {i}, \mathbf {z} _ {i}) ^ {2} ].
$$
Now, because our loss is nonnegative, [CWR23, Lemma A.2 and Lemma B.1] state that the sequence  $(SM_{m})_{m\geq 1}$  defined for any  $m$  as:
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \underset {h \sim \rho_ {i - 1}} {\mathbb {E}} \left[ \underset {i - 1} {\mathbb {E}} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] - \ell \left(h _ {i}, \mathbf {z} _ {i}\right) \right] = \underset {\left(h _ {1}, \dots , h _ {m}\right) \sim \pi_ {1, S} \otimes \dots \otimes \pi_ {m, S}} {\mathbb {E}} \left[ \sum_ {i = 1} ^ {m} X _ {i} \left(h _ {i}, \mathbf {z} _ {i}\right) \right] \\ = \frac {1}{\lambda} \underset {(h _ {1}, \dots , h _ {m}) \sim \pi_ {1, \mathcal {S}} \otimes \dots \otimes \pi_ {m, \mathcal {S}}} {\mathbb {E}} [ f _ {m} (\mathcal {S}, h _ {1}, \dots , h _ {m}) ] \\ + \frac {\lambda}{2} \sum_ {i = 1} ^ {m} \underset {h _ {i} \sim \pi_ {i, \mathcal {S}}} {\mathbb {E}} \left[ \underset {i - 1} {\mathbb {E}} \left[ \ell (h _ {i}, \mathbf {z} _ {i}) ^ {2} \right] \right] \\ \leq \frac {\ln \left(S M _ {m}\right)}{\lambda} + \frac {\lambda}{2} \sum_ {i = 1} ^ {m} \mathbb {E} _ {h _ {i} \sim \pi_ {i, S}} \left[ \mathbb {E} _ {i - 1} \left[ \ell \left(h _ {i}, \mathbf {z} _ {i}\right) ^ {2} \right] \right] \\ \end{array}
$$
$$
\mathbb {P} _ {\mathcal {S}} \left(\forall m, S M _ {m} \leq \frac {1}{\delta}\right) \geq \frac {1}{\delta}
$$
Thus, with probability  $1 - \delta$ , for any  $m$  we have  $\ln (SM_m) \leq \ln \frac{1}{\delta}$ . We conclude the proof by exploiting the boundedness assumption on conditional order 2 moments and optimising the bound in  $\lambda$ .
C Supplementary insights on experiments
In this section, Appendix C.1 presents the learning algorithm for the i.i.d. setting. We also introduce the online algorithm in Appendix C.2. We prove the Lipschitz constant of the loss for the linear models in Appendix C.3. Finally, we provide more experiments in Appendix C.5.
C.1 Batch algorithm for the i.i.d. setting
The pseudocode of our batch algorithm is presented in Algorithm 1.
Algorithm 1 (Mini-)Batch Learning Algorithm with Wasserstein distances
1: procedure PRIORS LEARNING  
2:  $h_1, \ldots, h_K \gets$  initialize the hypotheses  
3: for  $t \gets 1, \ldots, T$  do  
4: for each mini-batch  $\mathcal{U} \subseteq S$  do  
5: for  $i \gets 1, \ldots, K$  do  
6:  $\mathcal{U}_i \gets \mathcal{U} \setminus S_i$   
7:  $h_i \gets$  perform a gradient descent step with  $\nabla \mathbb{R}_{\mathcal{U}_i}(h_i)$   
8: return hypotheses  $h_1, \ldots, h_K$   
9:  
10: procedure POSTERIOR LEARNING  
11:  $h \gets$  initialize the hypothesis  
12: for  $t \gets 1, \ldots, T'$  do  
13: for each mini-batch  $\mathcal{U} \subseteq S$  do  
14:  $h \gets$  perform a gradient descent step with  $\nabla [\mathbb{R}_{\mathcal{U}}(h) + \varepsilon \sum_{i=1}^{K} \frac{|S_i|}{m} d(h, h_i)]$   
15: return hypothesis  $h$
PRIORS LEARNING minimises the empirical risk through mini-batches  $\mathcal{U} \subseteq S$  for  $T$  epochs. More precisely, for each epoch, we (a) sample a mini-batch  $\mathcal{U}$  (line 4) by excluding the set  $S_{i}$  from  $\mathcal{U}$  for each  $h_{i} \in \mathcal{H}$  (line 5-6), then (b) the hypotheses  $h_{1}, \ldots, h_{K} \in \mathcal{H}$  are updated (line 7). In POSTERIOR LEARNING, we perform a gradient descent step (line 14) on the objective function associated with Equation (5) for  $T'$  epochs in a mini-batch fashion.
C.2 Learning algorithm for the online setting
Algorithm 2 presents the pseudocode of our online algorithm.
Algorithm 2 Online Learning Algorithm with Wasserstein distances
1: Initialize the hypothesis  $h_0\in \mathcal{H}$    
2: for  $i\gets 1,\ldots ,m$  do   
3: for  $t\gets 1,\dots ,T$  do   
4:  $h_i\gets$  perform a gradient step with  $\nabla [\ell (h_i,\mathbf{z}_i) + \hat{B} (d(h_i,h_{i - 1}) - 1)]$  (Eq. (7) with  $\hat{B}$ )   
5: return hypotheses  $h_1,\ldots ,h_m$
For each time step  $i$ , we perform  $T$  gradient descent steps on the objective associated with Equation (6) (line 4). Note that we can retrieve OGD from Algorithm 2 by  $(a)$  setting  $T = 1$  and  $(b)$  removing the regularisation term  $\hat{B}(d(h_i, h_{i-1}) - 1)$ .
C.3 Lipschitzness for the linear model
Recall that we use, in our experiments, the multi-margin loss function from the Pytorch module defined for any linear model with weights  $W \in \mathbb{R}^{|\mathcal{V}| \times d}$  and biases  $b \in \mathbb{R}^{|\mathcal{V}|}$ , any data point  $\mathbf{z} \in \mathcal{X} \times \mathcal{Y}$ .
$$
\ell (W, b, \mathbf {z}) = \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} \max  \left(0, f (W, b, \mathbf {z}, y ^ {\prime})\right),
$$
where  $f(W, b, \mathbf{z}, y') = 1 + \langle W[y'] - W[y], \mathbf{x} \rangle + b[y'] - b[y]$ , and  $W[y] \in \mathbb{R}^d$  and  $b[y] \in \mathbb{R}$  are respectively the vector and the scalar for the  $y$ -th output.
To apply our theorems, we must ensure that our loss function is Lipschitz with respect to the linear model, hence the following lemma.
Lemma 8. For any  $\mathbf{z} = (\mathbf{x},y)\in \mathcal{X}\times \mathcal{Y}$  with the norm of  $\mathbf{x}$  bounded by 1, the function  $W,b\mapsto \ell (W,b,\mathbf{z})$  is 2-Lipschitz.
Proof. Let  $(W,b),(W^{\prime},b^{\prime})$  both in  $\mathbb{R}^{|\mathcal{V}|\times d}\times \mathbb{R}^{|\mathcal{V}|}$ , we have
$$
| \ell (W, b, \mathbf {z}) - \ell (W ^ {\prime}, b ^ {\prime}, \mathbf {z}) | \leq \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | \max {(0, f (W, b, \mathbf {z}, y ^ {\prime}))} - \max {(0, f (W ^ {\prime}, b ^ {\prime}, \mathbf {z}, y ^ {\prime}))} |.
$$
Note that because  $\alpha \mapsto \max (0,\alpha)$  is 1-Lipschitz, we have:
$$
| \ell (W, b, \mathbf {z}) - \ell (W ^ {\prime}, b ^ {\prime}, \mathbf {z}) | \leq \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | f (W, b, \mathbf {z}, y ^ {\prime}) - f (W ^ {\prime}, b ^ {\prime}, \mathbf {z}, y ^ {\prime}) |.
$$
Finally, notice that:
$$
\begin{array}{l} \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | f (W, b, \mathbf {z}, y ^ {\prime}) - f (W ^ {\prime}, b ^ {\prime}, \mathbf {z}, y ^ {\prime}) | \leq \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | \langle (W - W ^ {\prime}) [ y ^ {\prime} ] - (W - W ^ {\prime}) [ y ], \mathbf {x} \rangle | \\ + \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | (b - b ^ {\prime}) [ y ^ {\prime} ] - (b - b ^ {\prime}) [ y ] | \\ \leq \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} \| (W - W ^ {\prime}) [ y ^ {\prime} ] - (W - W ^ {\prime}) [ y ] \| \| \mathbf {x} \| \\ + \frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | (b - b ^ {\prime}) [ y ^ {\prime} ] - (b - b ^ {\prime}) [ y ] |. \\ \end{array}
$$
Because we consider the Euclidean norm, we have for any  $y' \in \mathcal{V}$ :
$$
\begin{array}{l} \| (W - W ^ {\prime}) [ y ^ {\prime} ] - (W - W ^ {\prime}) [ y ] \| = \sqrt {\| (W - W ^ {\prime}) [ y ^ {\prime} ] - (W - W ^ {\prime}) [ y ] \| ^ {2}} \\ \leq \sqrt {2 \left(\| (W - W ^ {\prime}) [ y ^ {\prime} ] \| ^ {2} + \| (W - W ^ {\prime}) [ y ] \| ^ {2}\right)} \\ \leq \sqrt {2} \| W - W ^ {\prime} \|. \\ \end{array}
$$
The second line holding because for any scalars  $a, b$ , we have  $(a - b)^2 \leq 2(a^2 + b^2)$  and the last line holding because  $\| W - W' \|^2 = \sum_{y \in \mathcal{Y}} \| (W - W')[y] \|^2$ . A similar argument gives
$$
\frac {1}{| \mathcal {Y} | - 1} \sum_ {y ^ {\prime} \neq y} | (b - b ^ {\prime}) [ y ^ {\prime} ] - (b - b ^ {\prime}) [ y ] | \leq \sqrt {2} \| b - b ^ {\prime} \|.
$$
Then, using that  $\| x \| \leq 1$  and summing on all  $y'$  gives:
$$
\left| \ell (W, b, \mathbf {z}) - \ell \left(W ^ {\prime}, b ^ {\prime}, \mathbf {z}\right) \right| \leq \sqrt {2} \left(\| W - W ^ {\prime} \| + \| b - b ^ {\prime} \|\right).
$$
Finally, notice that  $(\| W - W' \| + \| b - b' \|)^2 \leq 2 (\| W - W' \|^2 + \| b - b' \|^2) = 2 \| (W, b) - (W', b') \|^2$ .
Thus  $\| W - W' \| + \| b - b' \| \leq \sqrt{2} \| (W, b) - (W', b') \|$ . This concludes the proof.
C.4 Lipschitzness for neural networks
Recall that we use, in our experiments, the multi-margin loss function from the Pytorch module defined we consider the loss  $\ell(h, (\mathbf{x}, y)) = \frac{1}{|\mathcal{Y}|} \sum_{y' \neq y} \max(0, 1 - \eta(h[y] - h[y']))$ , which is  $\eta$ -Lipschitz w.r.t. the outputs  $h[1], \ldots, h[|\mathcal{Y}|]$ . For neural networks,  $h$  is the output of the neural network with input  $\mathbf{x}$ . Note that this loss is  $\eta$ -lipschitz with respect to the outputs. To apply our theorems, we must ensure that our loss function is Lipschitz with respect to the weights of the neural networks, hence the following lemma with associated background.
We define a FCN recursively as follows: for a vector  $\mathbf{W}_1 = \mathrm{vec}(\{W_1, b\})$ , (i.e., the vectorisation of a weight matrix  $W_1$  and a bias  $b$ ) and an input datum  $\mathbf{x}$ ,  $\mathrm{FCN}_1(\mathbf{W}_1, \mathbf{x}) = \sigma_1(W_1\mathbf{x} + b_1)$ , where  $\sigma_1$  is the activation function. Also, for any  $i \geq 2$  we define for a vector  $\mathbf{W}_i = (W_i, b_i, \mathbf{W}_{i-1})$  (defined recursively as well),  $\mathrm{FCN}_i(\mathbf{W}_i, \mathbf{x}) = \sigma_i(W_i\mathrm{FCN}_{i-1}(\mathbf{W}_{i-1}, \mathbf{x}) + b_i)$ . Then, setting  $\mathbf{z} = (\mathbf{x}, y)$  a datum and  $h_i(\mathbf{x}) := \mathrm{FCN}_i(\mathbf{W}_i, \mathbf{x})$  we can rewrite our loss as a function of  $(\mathbf{W}_i, \mathbf{z})$ .
Lemma 9. Assume that all the weight matrices of  $\mathbf{W}_i$  are bounded and that the activation functions are Lipschitz continuous with constant bounded by  $K_{\sigma}$ . Then for any datum  $\mathbf{z} = (\mathbf{x},y)$ , any  $i$ ,  $\mathbf{W}_i \to \ell(\mathbf{W}_i, \mathbf{z})$  is Lipschitz continuous.
Proof. We consider the Frobenius norm on matrices as  $\mathbf{W}_2$  is a vector as we consider the L2-norm on the vector. We prove the result for  $i = 2$ , assuming it is true for  $i = 1$ . We then explain how this proof generalises the case  $i = 1$  and works recursively. Let  $\mathbf{z}, \mathbf{W}_2, \mathbf{W}_2'$  for clarity we write  $\mathrm{FCN}_2(\mathbf{x}) \coloneqq \mathrm{FCN}(\mathbf{W}_2, \mathbf{x})$  and  $\mathrm{FCN}_2'(\mathbf{x}) \coloneqq \mathrm{FCN}(\mathbf{W}_2', \mathbf{x})$ . As  $\ell$  is Lipschitz on the outputs  $\mathrm{FCN}_2(\mathbf{x}), \mathrm{FCN}_2'(\mathbf{x})$ . We have
$$
\begin{array}{l} \left| \ell \left(\mathbf {W} _ {2}, \mathbf {z}\right) - \ell \left(\mathbf {W} _ {2} ^ {\prime}, \mathbf {z}\right) \right| \leq \eta \left\| \mathrm {F C N} _ {2} (\mathbf {x}) - \mathrm {F C N} _ {2} ^ {\prime} (\mathbf {x}) \right\| \\ \leq \eta \left\| \sigma_ {2} \left(W _ {2} \mathrm {F C N} _ {1} (\mathbf {x}) + b _ {2}\right) - \sigma_ {2} \left(W _ {2} ^ {\prime} \mathrm {F C N} _ {1} ^ {\prime} (\mathbf {x}) + b _ {2} ^ {\prime}\right) \right\| \\ \leq \eta K _ {\sigma} \| W _ {2} \mathrm {F C N} _ {1} (\mathbf {x}) + b _ {2} - W _ {2} ^ {\prime} \mathrm {F C N} _ {1} ^ {\prime} (\mathbf {x}) - b _ {2} ^ {\prime} \| \\ \leq \eta K _ {\sigma} \left(\left| \left| \left(W _ {2} - W _ {2} ^ {\prime}\right) \mathrm {F C N} _ {1} (\mathbf {x}) \right| \right| + \left| \left| W _ {2} ^ {\prime} \left(\mathrm {F C N} _ {1} (\mathbf {x}) - \mathrm {F C N} _ {1} ^ {\prime} (\mathbf {x})\right) \right| \right| + \left| b _ {2} - b _ {2} ^ {\prime} \right| \right|. \\ \end{array}
$$
Then, we have  $||(W_2 - W_2')\mathrm{FCN}_1(\mathbf{x})|| \leq ||(W_2 - W_2')||_F||\mathrm{FCN}_1(\mathbf{x})|| \leq K_{\mathbf{x}}||(W_2 - W_2')||_F$ . The second inequality holding as  $\mathrm{FCN}_1(\mathbf{x})$  is a continuous function of the weights. Indeed, as on a compact space, a continuous function reaches its maximum, then its norm is bounded by a certain  $K_{\mathbf{x}}$ . Also, as the weights are bounded, any weight matrix has its norm bounded by a certain  $K_W$  thus  $\| W_2'(\mathrm{FCN}_1(\mathbf{x}) - \mathrm{FCN}_1'(\mathbf{x})) \| \leq \| W_2' \|_F \| (\mathrm{FCN}_1(\mathbf{x}) - \mathrm{FCN}_1'(\mathbf{x})) \| \leq K_W \| \mathrm{FCN}_1(\mathbf{x}) - \mathrm{FCN}_1'(\mathbf{x}) \|$ . Finally, taking  $K_{\mathrm{temp}} = \eta K_{\sigma} \max(K_{\mathbf{x}}, K_W, 1)$  gives: