SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A Unified Framework for Information-Theoretic Generalization Bounds
Yifeng Chu Maxim Raginsky
{ychu26,maxim}@illinois.edu*
This paper presents a general methodology for deriving information-theoretic generalization bounds for learning algorithms. The main technical tool is a probabilistic decorrelation lemma based on a change of measure and a relaxation of Young's inequality in  $L_{\psi_p}$  Orlicz spaces. Using the decorrelation lemma in combination with other techniques, such as symmetrization, couplings, and chaining in the space of probability measures, we obtain new upper bounds on the generalization error, both in expectation and in high probability, and recover as special cases many of the existing generalization bounds, including the ones based on mutual information, conditional mutual information, stochastic chaining, and PAC-Bayes inequalities. In addition, the Fernique-Talagrand upper bound on the expected supremum of a subgaussian process emerges as a special case.
The generalization error of a learning algorithm is a useful proxy for evaluating the performance of the learned model on previously unseen data. Formally, it is defined as the expected (absolute) difference between the population risk and the empirical risk of the hypothesis returned by the algorithm. One of the classical methods for estimating the generalization error is via uniform convergence of various empirical processes indexed by the hypothesis class [1, 2]. For example, in the analysis of Empirical Risk Minimization, one can estimate the expected generalization error via Rademacher averages, which can be bounded from above using chaining techniques [3].
However, the bounds based on uniform convergence are often too pessimistic and may even become vacuous when the hypothesis space is extremely large, a typical situation with deep neural net models. For this reason, it is preferable to obtain algorithm-dependent generalization bounds that take into account the joint distribution of the training samples and of the output hypothesis. In this context, one capitalizes on the intuition that the generalization ability of a learning algorithm should be related to the amount of information the output hypothesis reveals about the training data. This idea, which has origins in the work on PAC-Bayes methods [4, 5], is the basis of the growing literature on information-theoretic generalization bounds, first proposed in [6] and further developed in [7-16] and many other works.
In fact, it is possible to effectively combine the information-theoretic approach with the classical framework based on various measures of complexity of the hypothesis class: One can use chaining techniques to successively approximate the hypothesis class by simpler model classes, which can then be analyzed using information-theoretic tools. This methodology, again originating in the PAC-Bayes literature [17], has been developed recently in [18-21]. Our goal in this work is to develop these ideas further by giving a unified framework for information-theoretic generalization bounds, from which many of the existing results emerge as special cases.
*Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois, Urbana, IL 61801, USA.
1.1 The main idea, informally
The main idea behind our framework is surprisingly simple. We first give an abstract description and then show how it can be particularized to various settings of interest. Let  $(X_{t})_{t\in T}$  be a centered (zero-mean) stochastic process defined on a probability space  $(\Omega ,\mathcal{A},\mathbb{P})$  and indexed by the elements of some set  $T$ . Let  $Q$  be a Markov kernel from  $\Omega$  to  $T$ , i.e., a measurable mapping taking each  $\omega \in \Omega$  to a probability measure  $Q(\cdot |\omega)$  on  $T$ . Together,  $\mathbb{P}$  and  $Q$  define a probability measure  $\mathbb{P}\otimes Q$  on the product space  $\Omega \times T$ . The mathematical object we would like to study is the expected value
$$
\langle \mathbb {P} \otimes Q, X \rangle := \int_ {\Omega \times T} X _ {t} (\omega) Q (\mathrm {d} t | \omega). \mathbb {P} (\mathrm {d} \omega).
$$
For example, assuming that there exists a measurable map  $\tau^{*}:\Omega \to T$ , such that
$$
X _ {\tau^ {*} (\omega)} (\omega) = \sup  _ {t \in T} X _ {t} (\omega), \quad \mathbb {P} - \text {a . s .} \tag {1}
$$
we can take  $Q(A|\omega)\coloneqq \mathbf{1}_{\{\tau^{*}(\omega)\in A\}}$  for all measurable subsets  $A$  of  $T$ . Then
$$
\langle \mathbb {P} \otimes Q, X \rangle = \mathbf {E} \Big [ \sup _ {t \in T} X _ {t} \Big ]
$$
is the expected supremum of  $X_{t}$ , the central object of study in the theory of generic chaining, where  $(T,d)$  is a metric space and increments  $X_{u} - X_{v}$  are "stochastically small" relative to  $d(u,v)$ . Alternatively, consider a statistical learning problem with instance space  $\mathcal{Z}$ , hypothesis space  $\mathcal{W}$ , and loss function  $\ell : \mathcal{W} \times \mathcal{Z} \to \mathbb{R}_{+}$ . Let  $P_Z$  be the (unknown) probability law of the problem instances in  $\mathcal{Z}$ . Then we could take  $\Omega = \mathcal{Z}^n$ ,  $\mathbb{P} = P_Z^{\otimes n}$ ,  $T = \mathcal{W}$ , and
$$
X _ {w} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(L (w) - \ell \left(w, Z _ {i}\right)\right),
$$
where  $L(w) \coloneqq \mathbf{E}_{Z \sim P_Z}[\ell(w, Z)]$  is the population risk of  $w$ . Let  $Q$  be a (randomized) learning algorithm that associates to each sample  $S = (Z_1, \ldots, Z_n) \sim \mathbb{P}$  a probability measure  $Q(\cdot | S)$  on the hypothesis space  $\mathcal{W}$ . Then
$$
\langle \mathbb {P} \otimes Q, X \rangle = \mathbf {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left(L (W) - \ell \left(W, Z _ {i}\right)\right) \right]
$$
is the expected generalization error of  $Q$ . In either case, we can proceed to analyze  $\langle \mathbb{P} \otimes Q, X \rangle$  via a combination of the following two steps:
- **Decorrelation** — We can remove the correlations encoded in  $\mathbb{P} \otimes Q$  by choosing a convenient product measure  $\mathbb{P} \otimes \mu$  on  $\Omega \times T$ , so that (roughly)
$$
\langle \mathbb {P} \otimes Q, X \rangle \lesssim \sqrt {D (\mathbb {P} \otimes Q \| \mathbb {P} \otimes \mu)} + \text {E r r o r}
$$
provided the process  $(X_{t})_{t\in T}$  is regular enough for the error term to be small. Here, we use the relative entropy (or information divergence)  $D(\cdot \| \cdot)$  to illustrate the key idea with a minimum of detail; the precise description is given in Section 3.
- Chaining in the space of measures — Since the process  $(X_{t})_{t\in T}$  is centered and  $\mathbb{P}\otimes \mu$  is a product measure, we automatically have  $\langle \mathbb{P}\otimes \mu ,X\rangle = 0$  even though  $\langle \mathbb{P}\otimes Q,X\rangle \neq 0$ . We can therefore interpolate between  $\mathbb{P}\otimes Q$  and  $\mathbb{P}\otimes \mu$  along a (possibly infinite) sequence  $Q_0,Q_1,\ldots ,Q_K$  of Markov kernels, such that  $\mathbb{P}\otimes Q_K = \mathbb{P}\otimes Q,\mathbb{P}\otimes Q_0 = \mathbb{P}\otimes \mu$ , and the differences  $\langle \mathbb{P}\otimes Q_k,X\rangle -\langle \mathbb{P}\otimes Q_{k - 1},X\rangle$  are suitably small. Telescoping, we get
$$
\langle \mathbb {P} \otimes Q, X \rangle = \sum_ {k = 1} ^ {K} \left(\langle \mathbb {P} \otimes Q _ {k}, X \rangle - \langle \mathbb {P} \otimes Q _ {k - 1}, X \rangle\right).
$$
For each  $k$ , we then apply the decorrelation procedure to the increment process  $(X_{u} - X_{v})_{u,v\in T}$ , with  $\mathbb{P}$  as before and with a suitably chosen family of couplings of  $Q_{k}(\cdot |\omega)$  and  $Q_{k - 1}(\cdot |\omega)$ . This step can be combined effectively with other techniques, such as symmetrization.
Basic definitions. All measurable spaces in this paper are assumed to be standard Borel spaces. The set of all Borel probability measures on a space  $\mathcal{X}$  will be denoted by  $\mathcal{P}(\mathcal{X})$ . A Markov kernel from  $(\mathcal{X},\mathcal{A})$  to  $(\mathcal{Y},\mathcal{B})$  is a mapping  $P_{Y|X}:\mathcal{B}\times \mathcal{X}\to [0,1]$ , such that  $P_{Y|X = x}(\cdot)\coloneqq P_{Y|X}(\cdot |x)$  is an element of  $\mathcal{P}(\mathcal{Y})$  for every  $x\in \mathcal{X}$  and the map  $x\mapsto P_{Y|X = x}(B)$  is measurable for every  $B\in \mathcal{B}$ . The set of all such Markov kernels will be denoted by  $\mathcal{M}(\mathcal{Y}|\mathcal{X})$ .
The product of  $P_{X} \in \mathcal{P}(\mathcal{X})$  and  $P_{Y|X} \in \mathcal{M}(\mathcal{V}|\mathcal{X})$  is the probability measure  $P_{X} \otimes P_{Y|X} \in \mathcal{P}(\mathcal{X} \times \mathcal{V})$  defined on product sets  $A \times B \subseteq \mathcal{X} \times \mathcal{V}$  by  $(P_{X} \otimes P_{Y|X})(A \times B) := \int_{A} P_{Y|X = x}(B) P_{X}(\mathrm{d}x)$  and then extended to all Borel subsets of  $\mathcal{X} \times \mathcal{V}$  by countable additivity. This defines a joint probability law for a random element  $(X, Y)$  of  $\mathcal{X} \times \mathcal{V}$ , so that  $P_{X}$  is the marginal law of  $X$ ,  $P_{Y|X}$  is the conditional law of  $Y$  given  $X$ , and  $P_{Y}(\cdot) = \int_{\mathcal{X}} P_{Y|X = x}(\cdot) P_{X}(\mathrm{d}x)$  is the marginal law of  $Y$ . The product measure  $P_{X} \otimes P_{Y}$ , under which  $X$  and  $Y$  are independent, is a special case of this if we interpret  $P_{Y}$  as a trivial Markov kernel with  $P_{Y|X = x} = P_{Y}$  for all  $x$ .
A coupling of  $\mu \in \mathcal{P}(\mathcal{X})$  and  $\nu \in \mathcal{P}(\mathcal{Y})$  is a probability measure  $P \in \mathcal{P}(\mathcal{X} \times \mathcal{Y})$ , such that  $P(\cdot \times \mathcal{Y}) = \mu(\cdot)$  and  $P(\mathcal{X} \times \cdot) = \nu(\cdot)$ . We will denote the set of all couplings of  $\mu$  and  $\nu$  by  $\Pi(\mu, \nu)$ . Let the space  $\mathcal{X} \cup \mathcal{Y}$  be equipped with a metric  $d$ , and let  $\mathcal{P}_p$ , for  $p \geq 1$ , denote the space of all probability measures  $\rho$  on  $\mathcal{X} \cup \mathcal{Y}$ , for which there exists some  $z_0 \in \mathcal{X} \cup \mathcal{Y}$  such that  $\int_{\mathcal{X} \cup \mathcal{Y}} d(z, z_0)^p \rho(\mathrm{d}z) < \infty$ . Then the  $p$ -Wasserstein distance between  $\mu \in \mathcal{P}(\mathcal{X}) \cap \mathcal{P}_p$  and  $\nu \in \mathcal{P}(\mathcal{Y}) \cap \mathcal{P}_p$  is given by
$$
\mathsf {W} _ {p} (\mu , \nu) := \inf  _ {\pi \in \Pi (\mu , \nu)} \left(\int d (x, y) ^ {p} \pi (\mathrm {d} x, \mathrm {d} y)\right) ^ {1 / p}
$$
(see [22, 23] for details).
$L^p$  and  $L_{\psi_p}$  spaces. The  $L^p(\mu)$  norms of  $f: \mathcal{X} \to \mathbb{R}$ , for  $\mu \in \mathcal{P}(\mathcal{X})$  and  $p \geq 1$ , are defined as
$$
\| f \| _ {L ^ {p} (\mu)} := \left(\int_ {\mathcal {X}} | f | ^ {p} d \mu\right) ^ {1 / p}
$$
whenever the expectation on the right-hand side exists. We will often use the linear functional notation for expectations, i.e.,  $\langle \mu ,f\rangle = \int_{\mathcal{X}}f\mathrm{d}\mu$
For  $p \geq 1$ , define the function  $\psi_p : \mathbb{R}_+ \to \mathbb{R}_+$  by  $\psi_p(x) := \exp(x^p) - 1$ . Its inverse is given by  $\psi_p^{-1}(x) = (\log(x + 1))^{1/p}$ , where  $\log$  will always denote natural logarithms. Some useful properties of these two functions are collected in Appendix A of Supplementary Material. The function  $\psi_p$  arises in the context of controlling the tail behavior of random variables (see [1,24,25] for details). The Orlicz  $\psi_p$ -norm of a real-valued random variable  $X$  is defined as
$$
\| X \| _ {\psi_ {p}} := \inf  \left\{c > 0: \mathbf {E} \left[ \psi_ {p} \left(\frac {| X |}{c}\right) \right] \leq 1 \right\},
$$
and the tails of  $X$  satisfy  $\mathbf{P}[|X|\geq u]\leq Ke^{-Cu^p}$  for all  $u\geq 0$  and some  $K,C > 0$  if and only if  $\| X\|_{\psi_p} < \infty$ . The Orlicz space  $L_{\psi_p}$  is the space of all random variables  $X$  with  $\| X\|_{\psi_p} < \infty$ . In particular, if  $X$  is  $\sigma$ -subgaussian, i.e.,  $\mathbf{P}[|X|\geq u]\leq 2e^{-u^2 /2\sigma^2}$  for all  $u\geq 0$ , then  $\| X\|_{\psi_2}\leq \sqrt{6}\sigma$ ; conversely, every  $X\in L_{\psi_2}$  is  $\sigma$ -subgaussian with  $\sigma \leq c\| X\|_{\psi_2}$  for some absolute constant  $c > 0$ .
Information-theoretic quantities. The relative entropy (or information divergence)  $D(\mu \| \nu)$  between two probability measures  $\mu, \nu$  on the same space  $\mathcal{X}$  is defined as
$$
D (\mu \| \nu) := \left\langle \mu , \log \frac {\mathrm {d} \mu}{\mathrm {d} \nu} \right\rangle
$$
if  $\mu \ll \nu$  (i.e.,  $\mu$  is absolutely continuous w.r.t.  $\nu$ ), and  $D(\mu \| \nu) \coloneqq +\infty$  otherwise. The following inequality will be useful (proofs of all results are in Appendix B of the Supplementary Material):
Proposition 1. If  $\mu \ll \nu$ , then for any  $p \geq 1$
$$
\left\langle \mu , \psi_ {p} ^ {- 1} \left(\frac {\mathrm {d} \mu}{\mathrm {d} \nu}\right) \right\rangle \leq \left(D (\mu \| \nu) + 1\right) ^ {1 / p}.
$$
The conditional divergence between  $P_{V|U}, Q_{V|U} \in \mathcal{M}(\mathcal{V}|\mathcal{U})$  given  $P_U \in \mathcal{P}(\mathcal{U})$  is defined by
$$
D \left(P _ {V | U} \| Q _ {V | U} P _ {U}\right) := D \left(P _ {U} \otimes P _ {V | U} \| P _ {U} \otimes Q _ {V | U}\right).
$$
The mutual information  $I(X;Y) \coloneqq D(P_{Y|X} \| P_Y | P_X)$  and conditional mutual information  $I(X;Y|Z) \coloneqq D(P_{Y|XZ} \| P_{Y|Z} | P_{XZ})$  are special cases of the above definition, and the identities
$$
D \left(P _ {Y \mid X} \| Q _ {Y} \mid P _ {X}\right) = I (X; Y) + D \left(P _ {Y} \| Q _ {Y}\right) \tag {2}
$$
$$
D \left(P _ {Y \mid X Z} \| Q _ {Y \mid Z} \mid P _ {X Z}\right) = I (X; Y \mid Z) + D \left(P _ {Y \mid Z} \| Q _ {Y \mid Z} \mid P _ {Z}\right). \tag {3}
$$
hold whenever all the quantities are finite. See, e.g., [26] for details.
3 The decorrelation lemma
All of our subsequent developments make use of the following decorrelation lemma:
Lemma 1. Let  $\mu, \nu$  be two probability measures on a space  $\mathcal{X}$  such that  $\mu \ll \nu$ , and let  $f, g: \mathcal{X} \to \mathbb{R}_+$  be two nonnegative measurable functions. Then the following inequalities hold:
$$
\langle \mu , f g \rangle \leq 2 ^ {1 / p} \left\langle \mu , f \psi_ {p} ^ {- 1} \left(\frac {\mathrm {d} \mu}{\mathrm {d} \nu}\right) \right\rangle + \langle \nu , f \psi_ {p} (g) \rangle \tag {4}
$$
$$
\langle \mu , f g \rangle \leq 2 ^ {1 / p} \| f \| _ {L ^ {2} (\nu)} + 4 ^ {1 / p} \left\langle \mu , f \psi_ {p} ^ {- 1} \left(\frac {\mathrm {d} \mu}{\mathrm {d} \nu}\right) \right\rangle + 4 ^ {1 / p} \| f \| _ {L ^ {1} (\mu)} \left(\log \langle \nu , \exp (g ^ {p}) \rangle\right) ^ {1 / p}. \tag {5}
$$
The proof makes extensive use of various properties of  $\psi_p$  and  $\psi_p^{-1}$ . In particular, Eq. (4) is a relaxation of the Young-type inequality  $xy \leq \psi_p^*(x) + \psi_p(y)$ , where  $\psi_p^*(x) := \sup_{y \geq 0} (xy - \psi_p(y))$  is the (one-sided) Lengendre-Fenchel conjugate of  $\psi_p$ . (We refer the reader to [13] for another use of duality in Orlicz spaces in the context of generalization bounds.)
Every use of Lemma 1 in the sequel will be an instance of the following scheme: Let  $P_X \in \mathcal{P}(\mathcal{X})$ ,  $Q_Y \in \mathcal{P}(\mathcal{Y})$ , and  $P_{Y|X} \in \mathcal{M}(\mathcal{Y}|\mathcal{X})$  be given, such that  $P_{Y|X = x} \ll Q_Y$  for all  $x \in \mathcal{X}$ . Let  $(X,Y,\bar{Y})$  be a random element of  $\mathcal{X} \times \mathcal{Y} \times \mathcal{Y}$  with joint law  $P_X \otimes P_{Y|X} \otimes Q_Y$ ; in particular,  $\bar{Y}$  is independent of  $(X,Y)$ . Furthermore, let  $f: \mathcal{Y} \to \mathbb{R}_+$  and  $g: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}_+$  be given, such that  $\mathbf{E}[\psi_p(g(X,y))] \leq 1$  for all  $y \in \mathcal{Y}$ . Then, applying Lemma 1 conditionally on  $X = x$  with  $\mu = P_{Y|X = x}, \nu = Q_Y, f$ , and  $g(x,\cdot)$ , and then taking expectations w.r.t.  $P_X$ , we obtain
$$
\mathbf {E} [ f (Y) g (X, Y) ] \leq 2 ^ {1 / p} \mathbf {E} \left[ f (Y) \psi_ {p} ^ {- 1} \left(\frac {\mathrm {d} P _ {Y | X}}{\mathrm {d} Q _ {Y}} (Y)\right) \right] + \mathbf {E} [ f (\bar {Y}) ].
$$
In specific cases, the quantity on the right-hand side can be further upper-bounded in terms of the information divergences  $D(P_{Y|X} \| Q_Y)$  using Proposition 1.
4 Some estimates for the absolute generalization error
We adopt the usual set-up for the analysis of (possibly randomized) learning algorithms and their generalization error. Let an instance space  $\mathcal{Z}$ , a hypothesis space  $\mathcal{W}$ , and a nonnegative loss function  $\ell: \mathcal{W} \times \mathcal{Z} \to \mathbb{R}_+$  be given. A learning algorithm is a Markov kernel  $P_{W|S}$  from the product space  $\mathcal{Z}^n$  into  $\mathcal{W}$ , which takes as input an  $n$ -tuple  $S = (Z_1, \ldots, Z_n)$  of i.i.d. random elements of  $\mathcal{Z}$  with unknown marginal probability law  $P_Z$  and generates a random element  $W$  of  $\mathcal{W}$ . We define the empirical risk and the expected (or population) risk of each  $w \in \mathcal{W}$  by
$$
L _ {n} (w) := \left\langle P _ {n}, \ell (w, \cdot) \right\rangle = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell (w, Z _ {i}), \qquad L (w) := \left\langle P _ {Z}, \ell (w, \cdot) \right\rangle = \mathbf {E} [ \ell (w, Z) ]
$$
where  $P_{n}$  is the empirical distribution of  $S$ , and the pointwise generalization error by
$$
\operatorname {g e n} (w, S) := L (w) - L _ {n} (w).
$$
It will also be convenient to introduce an auxiliary  $n$ -tuple  $S' = (Z_1', \ldots, Z_n') \sim P_Z^{\otimes n}$ , which is independent of  $(S, W) \sim P_Z^{\otimes n} \otimes P_{W|S}$ . We will use  $\tilde{S}$  to denote the pair  $(S', S)$  and write  $L_n'(w)$  for the empirical risk of  $w$  on  $S'$ .
As a first illustration of our general approach, we show that it can be used to recover some existing results on the generalization error, including the bounds of Xu and Raginsky [7] involving the mutual information and the bounds of Steinke and Zakynthinou [10] involving the conditional mutual information. We start with the following estimate on the expected value of  $|\mathrm{gen}(W,S)|$ :
Theorem 1. Assume the random variables  $\ell(w, Z)$ ,  $w \in \mathcal{W}$ , are  $\sigma$ -subgaussian when  $Z \sim P_Z$ . Let a learning algorithm  $P_{W|S}$  be given. Then, for any  $Q_W \in \mathcal{P}(\mathcal{W})$ ,
$$
\mathbf {E} [ | \operatorname {g e n} (W, S) | ] \leq \sqrt {\frac {1 2 \sigma^ {2}}{n}} \left(\mathbf {E} \left[ \psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} P _ {W \mid S}}{\mathrm {d} Q _ {W}}\right) \right] + 1\right), \tag {6}
$$
where the expectation on both sides is w.r.t.  $P_{S}\otimes P_{W|S} = P_{Z}^{\otimes n}\otimes P_{W|S}$
The key step in the proof is to apply the decorrelation lemma, conditionally on  $S$ , to  $\mu = P_{W|S}$ ,  $\nu = Q_W$ ,  $f(w) = \sigma \sqrt{6 / n}$ , and  $g(w) = \frac{\left|\mathrm{gen}(w,S)\right|}{\sigma\sqrt{6 / n}}$ . The same subgaussianity assumption was also made by Xu and Raginsky [7]. Minimizing the right-hand side of (6) over  $Q_W$ , we recover their generalization bound up to a multiplicative constant and an extra  $O(1 / \sqrt{n})$  term (which is unavoidable since we are bounding the expected absolute generalization error):
Corollary 1. Under the assumptions of Theorem 1,
$$
\mathbf {E} [ | \operatorname {g e n} (W, S) | ] \leq \sqrt {\frac {2 4 \sigma^ {2}}{n} \left(I (W ; S) + 4\right)}. \tag {7}
$$
A notable shortcoming of Theorem 1 and Corollary 1 is that they yield vacuous bounds whenever the mutual information  $I(W; S)$  is infinite, which will be the case, e.g., when the marginal probability laws  $P_Z$  and  $P_W$  are nonatomic (i.e., assign zero mass to singletons) and the learning algorithm is deterministic. To remove this drawback, we will use an elegant auxiliary randomization device introduced by Steinke and Zakynthinou [10].
Let  $\varepsilon = (\varepsilon_{1},\ldots ,\varepsilon_{n})$  be an  $n$ -tuple of i.i.d. Rademacher random variables, i.e.,  $\mathbf{P}[\varepsilon_i = \pm 1] = 1 / 2$ , independent of  $\tilde{S}$ . For each  $i$  let  $\tilde{Z}_i^1 \coloneqq Z_i$  and  $\tilde{Z}_i^{-1} \coloneqq Z_i'$  and let  $\bar{P} = \bar{P}_{\tilde{S}\varepsilon W}$  be the joint probability law of  $(\tilde{S},\varepsilon ,W)$ , such that  $\bar{P}_{\tilde{S}\varepsilon} = P_{\tilde{S}}\otimes P_{\varepsilon}$  and  $\bar{P}_{W|\tilde{S}\varepsilon} \coloneqq P_{W|\tilde{S}^\varepsilon}$  where  $S^{\varepsilon} \coloneqq (\tilde{Z}_1^{\varepsilon_1},\dots,\tilde{Z}_n^{\varepsilon_n})$ . In other words, under  $\bar{P}$ ,  $\tilde{S}$  and  $\varepsilon$  are independent and have their respective marginal distributions, while  $W$  is generated by feeding the learning algorithm  $P_{W|S}$  with the tuple  $\tilde{S}^{\varepsilon}$ . Consequently,  $W$  is independent of  $\tilde{S}^{-\varepsilon} = (\tilde{Z}_1^{-\varepsilon_1},\dots,\tilde{Z}_n^{-\varepsilon_n})$ . Then, letting  $P$  be the joint law of  $(\tilde{S},W)$ , we have
$$
\begin{array}{l} \mathbf {E} _ {P} [ | \mathrm {g e n} (W, S) | ] = \mathbf {E} _ {P} \left| \mathbf {E} _ {P} \left[ L _ {n} ^ {\prime} (W) - L _ {n} (W) | S, W \right] \right| \\ \leq \mathbf {E} _ {P} \left| L _ {n} ^ {\prime} (W) - L _ {n} (W) \right| \\ = \mathbf {E} _ {\bar {P}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\ell \left(W, \tilde {Z} _ {i} ^ {- \varepsilon_ {i}}\right) - \ell \left(W, \tilde {Z} _ {i} ^ {\varepsilon_ {i}}\right)\right) \right| \\ = \mathbf {E} _ {\bar {P}} \Big | \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \Big (\ell (W, Z _ {i} ^ {\prime}) - \ell (W, Z _ {i}) \Big) \Big |. \\ \end{array}
$$
Thus, all the analysis can be carried out w.r.t.  $\bar{P}$ , as in the following:
Theorem 2. Assume there exists a function  $\Delta: \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}_+$ , such that  $|\ell(w, z) - \ell(w, z')| \leq \Delta(z, z')$  for all  $w \in \mathcal{W}$  and  $z, z' \in \mathcal{Z}$ . Then for any Markov kernel  $Q_{W|\tilde{S}}$  with access to  $\tilde{S}$  but not to  $\varepsilon$  we have
$$
\mathbf {E} _ {P} [ | \operatorname {g e n} (W, S) | ] \leq \frac {\sqrt {1 2}}{n} \mathbf {E} _ {\bar {P}} \left[ \| \Delta (\tilde {S}) \| _ {\ell^ {2}} \left(\psi_ {2} ^ {- 1} \left(\frac {\mathrm {d} \bar {P} _ {W | \tilde {S} _ {\varepsilon}}}{\mathrm {d} Q _ {W | \tilde {S}}}\right) + 1\right) \right], \tag {8}
$$
where  $\| \Delta (\tilde{s})\|_{\ell^2}\coloneqq \left(\sum_{i = 1}^{n}\Delta (z_i,z_i')^2\right)^{1 / 2}$
The same assumption on  $\ell$  was also made in [10]. Optimizing over  $Q_{W|\tilde{S}}$ , we can recover their Theorem 5.1 (again, up to a multiplicative constant and a  $O(1 / \sqrt{n})$  fluctuation term):
Corollary 2. Under the assumptions of Theorem 2,
$$
\mathbf {E} _ {P} [ | \operatorname {g e n} (W, S) | ] \leq \sqrt {\frac {2 4}{n}} \mathbf {E} [ \Delta^ {2} (Z, Z ^ {\prime}) ] \left(I (W; \varepsilon | \tilde {S}) + 4\right), \tag {9}
$$
where  $Z$  and  $Z'$  are independent samples from  $P_Z$  and where the conditional mutual information is computed w.r.t.  $\bar{P}$ .
The main advantage of using conditional mutual information is that it never exceeds  $n \log 2$  (of course, the bound is only useful if  $I(W; \varepsilon | \tilde{S}) = o(n)$ ).
5 Estimates using couplings
We now turn to the analysis of  $\mathbf{E}[\mathrm{gen}(W,S)]$  using couplings. The starting point is the following observation: With  $(S^{\prime},S,W)$  be constructed as before, consider the quantities
$$
\tilde {L} _ {n} (w) := L _ {n} ^ {\prime} (w) - L _ {n} (w) \equiv \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\ell (w, Z _ {i} ^ {\prime}) - \ell (w, Z _ {i})\right).
$$
Then, using the fact that  $\langle P_{\tilde{S}}\otimes Q_W,\tilde{L}_n\rangle = 0$  for any  $Q_{W}\in \mathcal{P}(\mathcal{W})$ , we have
$$
\begin{array}{l} \mathbf {E} [ \mathrm {g e n} (W, S) ] = \langle P _ {\tilde {S}} \otimes P _ {W | S}, \tilde {L} _ {n} \rangle - \langle P _ {\tilde {S}} \otimes Q _ {W}, \tilde {L} _ {n} \rangle \\ = \int_ {\mathcal {Z} \times \mathcal {Z}} P _ {\tilde {S}} (\mathrm {d} \tilde {s}) \left(\left\langle P _ {W | S = s}, \tilde {L} _ {n} \right\rangle - \left\langle Q _ {W}, \tilde {L} _ {n} \right\rangle\right). \tag {10} \\ \end{array}
$$