SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This suggests the idea of introducing, for each  $s \in \mathcal{Z}^n$ , a coupling of  $P_{W|S = s}$  and  $Q_W$ , i.e., a probability law  $P_{UV|S = s}$  for a random element  $(U,V)$  of  $\mathcal{W} \times \mathcal{W}$  with marginals  $P_U = P_{W|S = s}$  and  $P_V = Q_W$ . We then have the following:
Theorem 3. For  $u, v \in \mathcal{W}$  and  $\tilde{s} = (s, s') \in \mathcal{Z}^n \times \mathcal{Z}^n$ , define
$$
\sigma^ {2} (u, v, \tilde {s}) := \sum_ {i = 1} ^ {n} \left(\left(\ell \left(u, z _ {i} ^ {\prime}\right) - \ell \left(v, z _ {i} ^ {\prime}\right)\right) - \left(\ell \left(u, z _ {i}\right) - \ell \left(v, z _ {i}\right)\right)\right) ^ {2}. \tag {11}
$$
Then, for any  $Q_W \in \mathcal{P}(\mathcal{W})$ , any family of couplings  $P_{UV|S=s} \in \Pi(P_{W|S=s}, Q_W)$  depending measurably on  $s \in \mathcal{Z}^n$ , and any  $\mu_{UV} \in \mathcal{P}(\mathcal{W} \times \mathcal{W})$ ,
$$
\mathbf {E} [ \operatorname {g e n} (W, S) ] \leq \frac {\sqrt {2 4}}{n} \mathbf {E} \left[ \sigma (U, V, \tilde {S}) \psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} P _ {U V | S}}{\mathrm {d} \mu_ {U V}}\right) + \sqrt {\mathbf {E} [ \sigma^ {2} (\bar {U} , \bar {V} , \tilde {S}) | \tilde {S} ]} \right], \tag {12}
$$
where the expectation on the right-hand side is w.r.t. the joint law of  $(U,V,\bar{U},\bar{V},\tilde{S})$ , under which  $(S,U,V)$  are distributed according to  $P_{S}\otimes P_{UV|S}$ ,  $(\bar{U},\bar{V})$  are distributed according to  $\mu_{UV}$  independently of  $(U,V,S)$ , and  $S^{\prime}$  is distributed according to  $P_{S}$  independently of everything else.
The proof makes essential use of symmetrization using an auxiliary  $n$ -tuple  $\varepsilon$  of i.i.d. Rademacher random variables, which allows us to apply Lemma 1 conditionally on  $\tilde{S}$ .
The coupling-based formulation looks rather complicated compared to the setting of Section 4. However, being able to choose not just the "prior"  $Q_W$ , but also the couplings  $P_{UV|S}$  of  $P_{W|S}$  and  $Q_W$  and the reference measure  $\mu_{UV}$ , allows us to overcome some of the shortcomings of the set-up of Section 4. Consider, for example, the case when the learning algorithm ignores the data, i.e.,  $P_{W|S} = P_W$ . Then we can choose  $Q_W = P_W$ ,  $P_{UV|S}(\mathrm{d}u,\mathrm{d}v) = P_W(\mathrm{d}u)\otimes \delta_u(\mathrm{d}v)$ , where  $\delta_{u}$  is the Dirac measure concentrated on the point  $u$ , and  $\mu_{UV} = P_{UV}$  (since the latter does not depend on  $S$ ). With these choices,  $U = V$  and  $\bar{U} = \bar{V}$  almost surely, so the right-hand side of (12) is identically zero. By contrast, the bounds of Theorems 1 and 2 always include an additional  $O(1 / \sqrt{n})$  term even when  $W$  and  $\tilde{S}$  are independent.
Moreover, Theorem 3 can be used to recover the bounds of Theorems 1 and 2 up to multiplicative constants. For example, to recover Theorem 1, we apply Theorem 3 with  $P_{UV|S} = P_{W|S} \otimes Q_W$ ,  $\mu_{UV} = Q_W \otimes Q_W$ , and with an estimate on  $\sigma(U,V,\tilde{S})$  based on the subgaussianity of  $\ell(w,Z)$ .
For a more manageable bound that will be useful later, let us define the following for  $u, v \in \mathcal{W}$ :
$$
d _ {S, \ell} (u, v) := \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\ell (u, Z _ {i}) - \ell (v, Z _ {i})\right) ^ {2}\right) ^ {1 / 2} \equiv \| \ell (u, \cdot) - \ell (v, \cdot) \| _ {L ^ {2} \left(P _ {n}\right)}
$$
$$
d _ {\ell} (u, v) := \left(\mathbf {E} \left[ \left(\ell (u, Z) - \ell (v, Z)\right) ^ {2} \right]\right) ^ {1 / 2} \equiv \| \ell (u, \cdot) - \ell (v, \cdot) \| _ {L ^ {2} (P _ {Z})},
$$
Corollary 3. Under the assumptions of Theorem 3,
$$
\mathbf {E} [ \mathrm {g e n} (W, S) ] \leq \sqrt {\frac {4 8}{n}} \mathbf {E} \bigg [ \big (d _ {\ell} (U, V) + d _ {S, \ell} (U, V) \big) \psi_ {2} ^ {- 1} \bigg (\frac {\mathrm {d} P _ {U V | S}}{\mathrm {d} \mu_ {U V}} \bigg) + d _ {\ell} (\bar {U}, \bar {V}) \bigg ].
$$
6 Refined estimates via chaining in the space of measures
We now combine the use of couplings as in Section 5 with a chaining argument. The basic idea is as follows: Instead of coupling  $P_{W|S}$  with  $Q_W$  directly, we interpolate between them using a (possibly infinite) sequence of Markov kernels  $P_{W|S}^0, P_{W|S}^1, \ldots, P_{W|S}^K$ , such that  $P_{W|S}^0 = Q_W$  and  $P_{W|S}^K = P_{W|S}$  (or  $\lim_{k \to \infty} P_{W|S}^k = P_{W|S}$  in an appropriate sense, e.g., weakly for each  $S$ , if the sequence is infinite). Given any such sequence, we telescope the terms in (10) as follows:
$$
\mathbf {E} [ \mathrm {g e n} (W, S) ] = \int_ {\mathcal {Z} \times \mathcal {Z}} P _ {\hat {S}} (\mathrm {d} \tilde {s}) \sum_ {k = 1} ^ {K} \Big (\langle P _ {W | S = s} ^ {k}, \tilde {L} _ {n} \rangle - \langle P _ {W | S = s} ^ {k - 1}, \tilde {L} _ {n} \rangle \Big).
$$
For each  $k$ , we can now choose a family of random couplings  $P_{W_k W_{k-1}|S} \in \Pi(P_{W|S}^k, P_{W|S}^{k-1})$  and a deterministic probability measure  $\rho_{W_k W_{k-1}} \in \mathcal{P}(\mathcal{W} \times \mathcal{W})$ . The following is an immediate consequence of applying Corollary 3 to each summand:
Theorem 4. Let  $P_{W|S}$ ,  $Q_W$ ,  $P_{W_k W_{k-1}|S}$ , and  $\rho_{W_k W_{k-1}}$  be given as above. Then
$\mathbf{E}[\mathrm{gen}(W,S)]$
$$
\leq \sqrt {\frac {4 8}{n}} \sum_ {k = 1} ^ {K} \mathbf {E} \left[ \left(d _ {\ell} \left(W _ {k}, W _ {k - 1}\right) + d _ {S, \ell} \left(W _ {k}, W _ {k - 1}\right)\right) \psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} P _ {W _ {k} W _ {k - 1} | S}}{\mathrm {d} \rho_ {W _ {k} W _ {k - 1}}}\right) + d _ {\ell} \left(\bar {W} _ {k}, \bar {W} _ {k - 1}\right) \right],
$$
where in the  $k$ th term on the right-hand side  $(S, W_k, W_{k-1})$  are jointly distributed according to  $P_S \otimes P_{W_k W_{k-1}|S}$  and  $(\bar{W}_k, \bar{W}_{k-1})$  are jointly distributed according to  $\rho_{W_k W_{k-1}}$ .
Apart from Theorem 1, we have been imposing only minimal assumptions on  $\ell$  and then using symmetrization to construct various subgaussian random variables conditionally on  $W$  and  $\tilde{S}$ . For the next series of results, we will assume something more, namely that  $(\mathcal{W},d)$  is a metric space and that the following holds for the centered loss  $\bar{\ell} (w,z)\coloneqq \ell (w,z) - \mathbf{E}[\ell (w,Z)]$ :
$$
\left\| \sum_ {i = 1} ^ {n} \left(\bar {\ell} \left(u, Z _ {i}\right) - \bar {\ell} \left(v, Z _ {i}\right)\right) \right\| _ {\psi_ {2}} \leq \sqrt {n} d (u, v), \quad \forall u, v \in \mathcal {W}. \tag {13}
$$
In other words, the centered empirical process  $\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\bar{\ell}(w,Z_i)$  indexed by the elements of  $(\mathcal{W},d)$  is a subgaussian process [1-3].
Theorem 5. Assume (13). Then
$$
\mathbf {E} [ \operatorname {g e n} (W, S) ] \leq \sqrt {\frac {2}{n}} \sum_ {k = 1} ^ {K} \mathbf {E} \left[ d \left(W _ {k}, W _ {k - 1}\right) \psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} P _ {W _ {k} W _ {k - 1} | S}}{\mathrm {d} \rho_ {W _ {k} W _ {k - 1}}}\right) + d \left(\bar {W} _ {k}, \bar {W} _ {k - 1}\right) \right] \tag {14}
$$
As a byproduct, we recover the stochastic chaining bounds of Zhou et al. [20] (which, in turn, subsume the bounds of Asadi et al. [18]):
Corollary 4. Let  $P_Z$  and  $P_{W|S}$  be given, and let  $P_W$  be the marginal law of  $W$ . Let  $\left(P_{W_k|S}\right)_{k\geq 0}$  be a sequence of Markov kernels satisfying the following conditions: (i)  $P_{W_0|S} = P_W$ ; (ii)  $P_{W_k|S} \xrightarrow{k\to\infty} P_{W|S}$ ; (iii) for every  $k\geq 1$ ,  $S - W - W_k - W_{k-1}$  is a Markov chain. Then
$$
\begin{array}{l} \mathbf {E} [ \operatorname {g e n} (W, S) ] \leq \sqrt {\frac {2}{n}} \sum_ {k = 1} ^ {\infty} \mathbf {E} \left[ d \left(W _ {k}, W _ {k - 1}\right) \left(\sqrt {D \left(P _ {S \mid W _ {k}} \| P _ {S}\right)} + 1\right) \right] (15) \\ \leq \sqrt {\frac {2}{n}} \sum_ {k = 1} ^ {\infty} \sqrt {\mathbf {E} \left[ d ^ {2} \left(W _ {k} , W _ {k - 1}\right) \right]} (\sqrt {I \left(W _ {k} ; S\right)} + 2). (16) \\ \end{array}
$$
Finally, we give an estimate based on 2-Wasserstein distances (cf. Section 2 for definitions and notation). Let  $\mathsf{W}_2(\cdot ,\cdot)$  be the 2-Wasserstein distance on  $\mathcal{P}_2(\mathcal{W})$  induced by the metric  $d$  on  $\mathcal{W}$ . A (constant-speed) geodesic connecting two probability measures  $P,Q\in \mathcal{P}_2(\mathcal{W})$  is a continuous path  $[0,1]\ni t\mapsto \rho_t\in \mathcal{P}_2(\mathcal{W})$ , such that  $\rho_0 = P$ ,  $\rho_{1} = Q$ , and  $\mathsf{W}_2(\rho_s,\rho_t) = (t - s)\mathsf{W}_2(P,Q)$  for all  $0\leq s\leq t\leq 1$  [22, 23]. Then we have the following corollary of Theorem 5:
Corollary 5. Let  $P_Z$  and  $P_{W|S}$  be given, and let  $P_W$  be the marginal law of  $W$ . With respect to 2-Wasserstein distance, let  $(P_{W_k|S})_{0 \leq k \leq K}$  be some points on the constant-speed geodesic  $(\rho_t)_{t \in [0,1]}$  with endpoints  $\rho_0 = P_{W_0|S} = P_{W|S}$  and  $\rho_1 = P_{W_K|S} = P_W$  (where  $K$  may be infinite), i.e., there exist some  $t_0 = 0 < t_1 < \dots < t_k < \dots < t_K = 1$ , such that  $P_{W_k|S} = \rho_{t_k}$  for  $k = 0, 1, \ldots$ . For each  $k$  let  $P_{W_kW_{k-1}|S}$  be the optimal coupling between the neighboring points  $P_{W_{k-1}|S}$  and  $P_{W_k|S}$ , i.e., the one that achieves  $\mathsf{W}_2(P_{W_{k-1}|S}, P_{W_k|S})$ . Then
$$
\leq \sqrt {\frac {2}{n}} \left(2 \mathbf {E} \left[ \mathbb {W} _ {2} \left(P _ {W | S}, P _ {W}\right) \right] + \sum_ {k = 1} ^ {K} \mathbf {E} \left[ \mathbb {W} _ {2} \left(P _ {W _ {k} | S}, P _ {W _ {k - 1} | S}\right) \sqrt {D \left(P _ {W _ {k} W _ {k - 1} | S} \mid \mid P _ {W _ {k} W _ {k - 1}}\right)} \right]\right). \tag {17}
$$
Observe that the first term on the right-hand side of (17) is the expected 2-Wasserstein distance between the posterior  $P_{W|S}$  and the prior  $P_W$ , while the second term is a sum of "divergence weighted" Wasserstein distances. Also note that the form of the second term is in the spirit of the Dudley entropy integral [1-3], where the Wasserstein distance corresponds to the radius of the covering ball and the square root of the divergence corresponds to square root of the metric entropy. We should also point out that the result in Corollary 5 does not require Lipschitz continuity of the loss function  $\ell (w,z)$  w.r.t. the hypothesis  $w\in \mathcal{W}$ , except in a weaker stochastic sense as in (13), in contrast to some existing works that obtain generalization bounds using Wasserstein distances [27,28].
7 Tail estimates
Next, we turn to high-probability tail estimates on  $\mathrm{gen}(W,S)$ . We start with the following simple observation: Assume  $\ell (w,Z)$  is  $\sigma$ -subgaussian for all  $w\in \mathcal{W}$  when  $Z\sim P_Z$ . Then, for any  $Q_{W}\in \mathcal{P}(\mathcal{W})$  such that  $P_{W|S = s}\ll Q_W$  for all  $s\in \mathcal{Z}^n$ , we have
$$
\mathbf {E} \left[ \exp \left(\frac {\mathrm {g e n} ^ {2} (W , S)}{6 \sigma^ {2} / n} - \log \left(1 + \frac {\mathrm {d} P _ {W | S}}{\mathrm {d} Q _ {W}} (W)\right)\right) \right] \leq \mathbf {E} \left[ \exp \left(\frac {\mathrm {g e n} ^ {2} (\bar {W} , S)}{6 \sigma^ {2} / n}\right) \right] \leq 1
$$
with  $\bar{W} \sim Q_W$  independent of  $(S, W)$ . Thus, by Markov's inequality, for any  $0 < \delta < 1$ ,
$$
\mathbf {P} \left[ | \mathrm {g e n} (W, S) | > \sigma \sqrt {\frac {6}{n}} \left(\psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} P _ {W | S}}{\mathrm {d} Q _ {W}} (W)\right) + \sqrt {\log \frac {1}{\delta}}\right) \right] \leq \delta .
$$
In other words,  $|\mathrm{gen}(W,S)| \lesssim \frac{\sigma}{\sqrt{n}} \psi_2^{-1} \left( \frac{\mathrm{d}P_{W|S}}{\mathrm{d}Q_W} \right)$  with high  $P_{SW}$ -probability. Similar observations are made by Hellström and Durisi [9] with  $Q_W = P_W$ , giving high-probability bounds of the form  $|\mathrm{gen}(W,S)| \lesssim \sqrt{\frac{\sigma^2 D(P_{W|S} \| P_W)}{n}}$ . Generalization bounds in terms of the divergence  $D(P_{W|S} \| P_W)$  are also common in the PAC-Bayes literature [4, 5]. Moreover, using the inequality (5) in Lemma 1, we can give high  $P_S$ -probability bounds on the conditional expectation
$$
\langle P _ {W | S}, | \operatorname {g e n} (W, S) | \rangle = \langle P _ {W | S}, | L (W) - L _ {n} (W) | \rangle .
$$
Theorem 6. Assume  $\ell(w, Z)$  is  $\sigma$ -subgaussian for all  $w$  when  $Z \sim P_Z$ . Then, for any  $Q_W \in \mathcal{P}(\mathcal{W})$ , the following holds with  $P_S$ -probability of at least  $1 - \delta$ :
$$
\langle P _ {W | S}, | \mathrm {g e n} (W, S) | \rangle \leq \sqrt {\frac {2 4 \sigma^ {2}}{n}} \Big (\Big \langle P _ {W | S}, \psi_ {2} ^ {- 1} \Big (\frac {\mathrm {d} P _ {W | S}}{\mathrm {d} Q _ {W}} \Big) \Big \rangle + 1 + \sqrt {\log \frac {2}{\delta}} \Big).
$$
Another type of result that appears frequently in the literature on PAC-Bayes methods pertains to so-called transductive bounds, i.e., inequalities for the difference between
$$
\langle P _ {n} ^ {\prime} \otimes P _ {W | S}, \ell \rangle - \langle P _ {n} ^ {\prime} \otimes Q _ {W}, \ell \rangle \equiv \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {E} [ \ell (W, Z _ {i} ^ {\prime}) - \ell (\bar {W}, Z _ {i} ^ {\prime}) | \tilde {S} ],
$$
$$
\langle P _ {n} \otimes P _ {W | S}, \ell \rangle - \langle P _ {n} \otimes Q _ {W}, \ell \rangle \equiv \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {E} [ \ell (W, Z _ {i}) - \ell (\bar {W}, Z _ {i}) | \tilde {S} ],
$$
where  $Q_W$  is some fixed "prior" and where  $\bar{W} \sim Q_W$  is independent of  $(S', S, W)$ . Using our techniques, we can give the following general transductive bound:
Theorem 7. Let  $P_{W|S}$  and  $Q_W$  be given and take any  $(P_{W_k W_{k-1}|S})_{k=1}^K$  and  $(\rho_{W_k W_{k-1}})_k^K$  as in Theorem 4. Also, let  $\pmb{p} = (p_1, p_2, \ldots)$  be a strictly positive probability distribution on  $\mathbb{N}$ . Then the following holds with  $P_{\tilde{S}}$ -probability at least  $1 - \delta$ :
$$
\begin{array}{l} \left(\langle P _ {n} ^ {\prime} \otimes P _ {W | S}, \ell \rangle - \langle P _ {n} ^ {\prime} \otimes Q _ {W}, \ell \rangle\right) - \left(\langle P _ {n} \otimes P _ {W | S}, \ell \rangle - \langle P _ {n} \otimes Q _ {W}, \ell \rangle\right) \\ \leq \sqrt {\frac {9 6}{n}} \sum_ {k = 1} ^ {K} \left(\sqrt {\langle \rho_ {W _ {k} W _ {k - 1}} , d _ {\tilde {S} , \ell} ^ {2} \rangle} + \left\langle P _ {W _ {k} W _ {k - 1} | S}, d _ {\tilde {S}, \ell} \psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} P _ {W _ {k} W _ {k - 1} | S}}{\mathrm {d} \rho_ {W _ {k} W _ {k - 1}}}\right) \right\rangle \right. \\ \left. + \left\langle P _ {W _ {k} W _ {k - 1} | S}, d _ {\tilde {S}, \ell} \right\rangle \sqrt {\log \frac {2}{p _ {k} \delta}}\right), \\ \end{array}
$$
$$
d _ {\tilde {S}, \ell} ^ {2} (u, v) := \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(\left(\ell (u, Z _ {i}) - \ell (v, Z _ {i})\right) ^ {2} + \left(\ell (u, Z _ {i} ^ {\prime}) - \ell (v, Z _ {i} ^ {\prime})\right) ^ {2}\right).
$$
This result subsumes some existing transductive PAC-Bayes estimates, such as Theorem 2 of Audibert and Bousquet [17]. Let us briefly explain how we can recover this result from Theorem 7. Assume that  $\mathcal{W}$  is countable and let  $(\mathcal{A}_k)$  be an increasing sequence of finite partitions of  $\mathcal{W}$  with  $\mathcal{A}_0 = \{\mathcal{W}\}$ . For each  $k$  and each  $w \in \mathcal{W}$ , let  $A_k(w)$  be the unique set in  $\mathcal{A}_k$  containing  $w$ . Choose a representative point in each  $A \in \mathcal{A}_k$  and let  $\mathcal{W}_k$  denote the set of all such representatives, with  $\mathcal{W}_0 = \{w_0\}$ . Take  $P_{W_\infty|S} = P_{W|S}$  and  $P_{W_0} = Q_W = \delta_{w_0}$ . Now, for each  $k \geq 0$ , we take  $P_{W_k|S}$  as the projection of  $P_{W|S}$  onto  $\mathcal{W}_k$ , i.e., the finite mixture
$$
P _ {W _ {k} | S} := \sum_ {w \in \mathcal {W} _ {k}} P _ {W | S} (A _ {k} (w)) \delta_ {w}.
$$
Moreover, given some "prior"  $\pi \in \mathcal{P}(\mathcal{W})$ , we can construct a sequence  $(\pi_k)_{k=0}^{\infty}$  of probability measures with  $\pi_{\infty} = \pi$  and  $\pi_0 = \delta_{w_0}$ , such that  $\pi_k$  is a projection of  $\pi$  onto  $\mathcal{W}_k$ . Now observe that, for each  $k$ ,  $S - W_{\infty} - W_k - W_{k-1}$  is a Markov chain. Indeed, if we know  $P_{W_k|S}$ , then we can construct  $P_{W_{\ell}|S}$  for any  $\ell < k$  without knowledge of  $S$ . With these ingredients in place, let us choose  $P_{W_kW_{k-1}|S} = P_{W_{k-1}|W_k} \otimes P_{W_k|S}$  and  $\rho_{W_kW_{k-1}} = \pi_k \otimes P_{W_{k-1}|W_k}$ . Then, using Cauchy-Schwarz and Jensen, we conclude that the following holds with  $P_{\tilde{S}}$ -probability at least  $1 - \delta$ :
$$
\begin{array}{l} \left(\langle P _ {n} ^ {\prime} \otimes P _ {W | S}, \ell \rangle - \langle P _ {n} ^ {\prime} \otimes \delta_ {w _ {0}}, \ell \rangle\right) - \left(\langle P _ {n} \otimes P _ {W | S}, \ell \rangle - \langle P _ {n} \otimes \delta_ {w _ {0}}, \ell \rangle\right) \\ \leq \sqrt {\frac {9 6}{n}} \sum_ {k = 1} ^ {\infty} \left(\sqrt {\langle \pi_ {k} \otimes P _ {W _ {k - 1} | W _ {k}} , d _ {\tilde {S} , \ell} ^ {2} \rangle} \right. \\ \left. + \sqrt {2 \langle P _ {W _ {k} | S} \otimes P _ {W _ {k - 1} | W _ {k}} , d _ {\tilde {S} , \ell} ^ {2} \rangle \left(D \left(P _ {W _ {k} | S} \| \pi_ {k}\right) + \log \frac {2 e}{p _ {k} \delta}\right)}\right). \\ \end{array}
$$
This recovers [17, Thm. 2] up to an extra term that scales like  $\frac{1}{\sqrt{n}} \sum_{k} \sqrt{\langle \pi_k \otimes P_{W_{k-1}|W_k}, d_{\tilde{S},\ell}^2 \rangle}$ .
8 The Fernique-Talagrand bound
As a bonus, we show that a combination of decorrelation and chaining in the space of measures can be used to recover the upper bounds of Fernique [29] and Talagrand [30] on the expected supremum of a stochastic process in terms of majorizing measures (see Eq. (19) below and also [31, 32]).
For simplicity, let  $(T,d)$  be a finite metric space with  $\mathrm{diam}(T) = \sup \{d(u,v):u,v\in T\} < \infty$ . Let  $B(t,r)$  denote the ball of radius  $r\geq 0$  centered at  $t\in T$ , i.e.,  $B(t,r)\coloneqq \{u\in T:d(u,t)\leq r\}$ . Let  $(X_{t})_{t\in T}$  be a centered stochastic process defined on some probability space  $(\Omega ,\mathcal{A},\mathbb{P})$  and satisfying
$$
\mathbf {E} \left[ \psi_ {p} \left(\frac {\left| X _ {u} - X _ {v} \right|}{d (u , v)}\right) \right] \leq 1, \quad \forall u, v \in T \tag {18}
$$
for some  $p \geq 1$ . Then we can obtain the following result using chaining in the space of measures and decorrelation estimates:
Theorem 8. Let  $\tau$  be a random element of  $T$ , i.e., a measurable map  $\tau : \Omega \to T$  with marginal probability law  $\nu$ . Then for any  $\mu \in \mathcal{P}(T)$  we have
$$
\mathbf {E} \left[ X _ {\tau} \right] \lesssim \operatorname {d i a m} (T) + \int_ {T} \int_ {0} ^ {\operatorname {d i a m} (T)} \left(\log \frac {1}{\mu (B (t , \varepsilon))}\right) ^ {1 / p} \mathrm {d} \varepsilon \nu (\mathrm {d} t).
$$
Applying Theorem 8 to  $\tau^{*}$  defined in (1) and then minimizing over  $\mu$ , we recover a Fernique-Talagrand type bound on the expected supremum of  $X_{t}$ :
$$
\mathbf {E} \left[ \sup  _ {t \in T} X _ {t} \right] = \mathbf {E} \left[ X _ {\tau^ {*}} \right] \lesssim \operatorname {d i a m} (T) + \inf  _ {\mu \in \mathcal {P} (T)} \sup  _ {t \in T} \int_ {0} ^ {\operatorname {d i a m} (T)} \left(\log \frac {1}{\mu (B (t , \varepsilon))}\right) ^ {1 / p} \mathrm {d} \varepsilon . \tag {19}
$$
9 Conclusion and future work
In this paper, we have presented a unified framework for information-theoretic generalization bounds based on a combination of two key ideas (decorrelation and chaining in the space of measures). However, our method has certain limitations, which we plan to address in future work. For example, it would be desirable to cover the case of processes satisfying Bernstein-type (mixed  $\psi_{1}$  and  $\psi_{2}$ ) increment conditions. It would also be of interest to see whether there are any connections to the convex-analytic approach of Lugosi and Neu [33]. Finally, since our method seamlessly interpolates between Fernique-Talagrand type bounds and information-theoretic bounds, we plan to use it to further develop the ideas of Hodgkinson et al. [21], who were the first to combine these two complementary approaches to analyze the generalization capabilities of iterative learning algorithms.
This work was supported by the Illinois Institute for Data Science and Dynamical Systems (iDS²), an NSF HDR TRIPODS institute, under award CCF-1934986. The authors would like to thank Matus Telgarsky for some valuable suggestions and the anonymous reviewers for pointing out some relevant work that was not cited in the original submission.
[21] Liam Hodgkinson, Umut Şimşekli, Rajiv Khanna, and Michael W. Mahoney. Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers. In International Conference on Machine Learning, July 2022.
[22] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. Birkhäuser, 2008.
[23] Cédric Villani. Topics in Optimal Transportation. American Mathematics Society, Providence, RI, 2003.
[24] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
[25] Roman Vershynin. High-Dimensional Probability. Cambridge University Press, 2018.
[26] Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, forthcoming. URL: https://people.lids.mit.edu/yp/homepage/data/itbook-export.pdf.
[27] Hao Wang, Mario Diaz, José Candido S. Santos Filho, and Flavio P. Calmon. An information-theoretic view of generalization via Wasserstein distance'. In IEEE International Symposium on Information Theory, 2019.
[28] Borja Rodrígues-Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. Tighter expected generalization error bounds via Wasserstein distance. In Advances in Neural Information Processing Systems, 2021.
[29] Xavier Fernique. Regularité des trajectoires des fonctions aléatoires Gaussiennes. In X. M. Fernique, J. P. Conze, J. Gani, and P. L. Hennequin, editors, *Ecole d'Eté de Probabilités de Saint-Flour IV—1974*, Lecture Notes in Mathematics, pages 1–96. Springer, 1975.
[30] Michel Talagrand. Regularity of gaussian processes. Acta Mathematica, 159:99-149, 1987.
[31] Witold Bednorz. A theorem on majorizing measures. The Annals of Probability, 34(5), September 2006.
[32] Witold Bednorz. The majorizing measure approach to sample boundedness. Colloquium Mathematicae, 139(2):205-227, 2015.
[33] Gábor Lugosi and Gergely Neu. Generalization bounds via convex analysis. In Conference on Learning Theory, 2022.
A Some elementary facts
We first list some useful inequalities for  $\psi_p$  and  $\psi_p^{-1}$ . Note that the estimates may not be the sharpest, but they suffice for our needs.
Proposition A.2. For  $p \geq 1$  and  $x \geq 0$ , let  $\psi_p(x) = \exp (x^p) - 1$  and let  $\psi_p^{-1}(x) = (\log (x + 1))^{1 / p}$  be its inverse. Then we have the following:
(i)  $\psi_p^2 (x / 2^{1 / p})\leq \psi_p(x)$
(ii)  $x\psi_{p}(x / 4^{1 / p})\leq 2^{1 / p}\psi_{p}(x / 2^{1 / p}).$
(iii) for  $x \geq 0$  and  $q \geq 1$ ,  $\psi_p^{-1}(x^q) \leq q^{1/p} \psi_p^{-1}(x)$ .
(iv) For  $x \geq 1$ ,  $\psi_p^{-1}(x) \leq (\log(x))^{1/p} + 1$ .
(i) For any  $x\geq 0$
$$
\begin{array}{l} \psi_ {p} (x) = \exp (x ^ {p}) - 1 = (\exp (x ^ {p} / 2) - 1) (\exp (x ^ {p} / 2) + 1) \geq (\exp (x ^ {p} / 2) - 1) ^ {2} \\ = \psi_ {p} ^ {2} (x / 2 ^ {1 / p}). \\ \end{array}
$$
(ii) We only need to consider the case  $x \geq 1$  since otherwise the inequality is obvious. Since  $y \leq 2(\exp(y/4) + 1)$  for all  $y \geq 1$ , we have
$$
x \leq 2 ^ {1 / p} (\exp (x ^ {p} / 4) + 1) ^ {1 / p} \leq 2 ^ {1 / p} (\exp (x ^ {p} / 4 p) + 1) \leq 2 ^ {1 / p} (\exp (x ^ {p} / 4) + 1).
$$
$$
\begin{array}{l} x \psi_ {p} (x / 4 ^ {1 / p}) = x (\exp (x ^ {p} / 4) - 1) \\ \leq 2 ^ {1 / p} \left(\exp \left(x ^ {p} / 4\right) + 1\right) \left(\exp \left(x ^ {p} / 4\right) - 1\right) \\ = 2 ^ {1 / p} \left(\exp \left(x ^ {p} / 2\right) - 1\right) \\ = 2 ^ {1 / p} \psi_ {p} (x / 2 ^ {1 / p}). \\ \end{array}
$$
(iii) Since  $x \geq 0$  and  $q \geq 1$ ,