SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sokhna Diarra Mbacke
Université Laval sokhna-diarra.mbacke.1@ulaval.ca
Florence Clerc  
McGill University  
florence.clerc@mail.mcgill.ca
Pascal Germain
Université Laval  
pascal.germain@ift.ulaval.ca
Since their inception, Variational Autoencoders (VAEs) have become central in machine learning. Despite their widespread use, numerous questions regarding their theoretical properties remain open. Using PAC-Bayesian theory, this work develops statistical guarantees for VAEs. First, we derive the first PAC-Bayesian bound for posterior distributions conditioned on individual samples from the data-generating distribution. Then, we utilize this result to develop generalization guarantees for the VAE's reconstruction loss, as well as upper bounds on the distance between the input and the regenerated distributions. More importantly, we provide upper bounds on the Wasserstein distance between the input distribution and the distribution defined by the VAE's generative model.
In recent years, deep generative models have exhibited tremendous empirical success. Two of the most important families of generative models are Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and Variational Autoencoders (Kingma and Welling, 2014; Rezende et al., 2014). GANs take an adversarial approach, whereas VAEs are based on maximum likelihood estimation and variational inference. VAEs comprise two main components: an encoder which parameterizes an approximation of the posterior distribution over the latent variables, and a decoder which parameterizes the likelihood. In addition to generative modelling tasks such as image generation (Vahdat and Kautz, 2020) and text generation (Bowman et al., 2016), VAEs have been successfully applied to other topics such as semi-supervised learning (Kingma et al., 2014), anomaly detection (An and Cho, 2015), and dimensionality reduction (Kaur et al., 2021). However, despite their empirical success, the question of statistical guarantees for the performance of VAEs remains largely open. Namely, how can one certify that VAEs generalize well, both in terms of reconstruction and generation?
PAC-Bayesian theory (McAllester, 1999; Catoni, 2003) is an influential tool of statistical learning theory dedicated to providing generalization bounds for machine learning models. PAC-Bayes has been applied to a wide variety of problems such as classification (Germain et al., 2009; Parrado-Hernandez et al., 2012), meta-learning (Amit and Meir, 2018), co-clustering (Seldin and Tishby, 2010), domain adaptation (Germain et al., 2020), and online learning (Haddouche and Guedj, 2022). In recent years, PAC-Bayes has been used to derive non-vacuous generalization bounds for supervised learning algorithms based on neural networks (Dziugaite and Roy, 2018; Pérez-Ortiz et al., 2021). See Guedj (2019) and Alquier (2021) for excellent surveys.
The objective of this work is to utilize PAC-Bayesian theory to derive statistical guarantees for VAEs. Our generalization bounds investigate the reconstruction, regeneration, as well as the generation properties of VAEs.
In order to explain the empirical success of deep generative models, a lot of attention has been put into deriving theoretical guarantees for these models. Most of the results, however, have been dedicated to GANs and their variants (Arora et al., 2017; Zhang et al., 2018; Liang, 2021; Singh et al., 2018; Schreuder et al., 2021; Biau et al., 2021; Mbacke et al., 2023). A possible explanation for this plethora of theoretical results is the adversarial loss function, which directly offers an estimation of the discrepancy between the input distribution and the generator's distribution. Despite being central tools in modern machine learning, VAEs have not benefited from such a thorough theoretical analysis (Chakrabarty and Das, 2021).
The work of Chakrabarty and Das (2021) studies the regeneration properties of Wasserstein autoencoders (WAEs) (Tolstikhin et al., 2018), which come from the same family as VAEs. Using VC theory, Chakrabarty and Das (2021) derive rates of convergence for the Wasserstein distance between the input distribution and the distribution regenerated by the WAE, as well as the total variation distance between the empirical latent distribution and the latent prior. Taking a more empirical approach, Cherief-Abdellatif et al. (2022) use PAC-Bayes to study the generalization properties of stochastic reconstruction models. They define a  $[0,1]$ -bounded reconstruction loss function, then utilize McAllester's bound (McAllester, 2003) to formulate a generalization bound for models with probabilistic neural networks (Langford and Caruana, 2001). Then, they re-scale their loss and compare the empirical results to the reconstruction of standard VAEs on benchmark datasets.
We also mention the work of Mbacke et al. (2023), who developed PAC-Bayesian bounds for the analysis of adversarial generative models. Using McDiarmid's inequality, they proved upper bounds on the distance between the input distribution and the generator's distribution, for WGANs (Arjovsky et al., 2017) and EBGANs (Zhao et al., 2017).
In this work, we derive theoretical guarantees for variational autoencoders using PAC-Bayesian theory. We provide three types of guarantees: reconstruction guarantees showing that VAEs can successfully reconstruct unseen samples from the input distribution; regeneration guarantees proving upper bounds on the Wasserstein distance between the input distribution and the distribution regenerated by the VAE, given the training set as input; and finally, generation guarantees showing upper bounds on the Wasserstein distance between the data-generating distribution and the VAE's generated distribution defined by the latent prior and the decoder. To the best of our knowledge, these are the first generalization bounds for the standard VAE's reconstruction and regeneration properties, as well as the first statistical guarantees for the VAE's generative model.
In our analysis, the PAC-Bayesian posterior coincides with the variational posterior, which requires the PAC-Bayesian posterior to be conditional. Since, to the best of our knowledge, such PAC-Bayes bounds do not exist in the literature, we start by developing the first PAC-Bayesian bound for conditional posterior distributions. Then, we provide upper bounds for the VAE's performance under two main assumptions: we start by assuming the instance space is bounded, then we take advantage of the manifold hypothesis. Our bounds are functions of the optimization objective of the VAE, namely, the empirical reconstruction loss, and the empirical KL-loss.
The remainder of this paper is organized as follows. In Section 2, we define some preliminary concepts, then briefly introduce VAEs and PAC-Bayesian theory. Section 3 presents our general PAC-Bayesian theorem for conditional posteriors. Then, in Sections 4 and 5, we present our generalization bounds for the reconstruction loss, and the regeneration and generation guarantees.
2.1 Definitions and Notations
Given metric spaces  $(\mathcal{X},d)$  and  $(\mathcal{Y},d^{\prime})$  , and a real number  $K > 0$  , a function  $f:\mathcal{X}\to \mathcal{Y}$  is  $K$  -Lipschitz continuous if for any  $\mathbf{x},\mathbf{y}\in \mathcal{X}$  , we have
$$
d ^ {\prime} (f (\mathbf {x}), f (\mathbf {y})) \leq K d (\mathbf {x}, \mathbf {y}).
$$
The smallest  $K$  such that this condition is satisfied is called the Lipschitz norm or Lipschitz constant of  $f$  and is denoted  $\| f\|_{\mathrm{Lip}}$ . Moreover, the set of  $K$ -Lipschitz continuous functions  $f:\mathcal{X}\to \mathcal{Y}$  is denoted  $\operatorname {Lip}_K(\mathcal{X},\mathcal{Y})$  (the underlying metrics will be clear from the context).
Throughout the paper, we use lower case letters  $p, q$  to denote both probability distributions and their densities w.r.t. the Lebesgue measure. We may add variables between parentheses to improve readability (e.g.  $p(\mathbf{z})$  to emphasize that  $p$  is a distribution on the space of variables  $\mathbf{z}$ , and  $q(\mathbf{z}|\mathbf{x})$  to indicate that  $q$  is a conditional distribution). The set of probability measures on a space  $\mathcal{X}$  is denoted  $\mathcal{M}_{+}^{1}(\mathcal{X})$ . The Kullback-Leibler (KL) divergence between  $p, q \in \mathcal{M}_{+}^{1}(\mathcal{X})$  is denoted  $\mathrm{KL}(p||q)$ . We omit the absolute continuity condition  $p \ll q$  in the statements of the results below, since if it is not satisfied, then one may assume the KL divergence is infinite and the bounds hold trivially.
Integral Probability Metrics (IPMs, see Müller (1997)) are a class of pseudo-metrics defined on the space of probability measures. Given a family  $\mathcal{F}$  of real-valued functions defined on  $\mathcal{X}$ , the IPM defined by  $\mathcal{F}$  is denoted  $d_{\mathcal{F}}$  and defined as
$$
d _ {\mathcal {F}} (p, q) = \sup  _ {f \in \mathcal {F}} \left| \int f d p - \int f d q \right|, \quad \forall p, q \in \mathcal {M} _ {+} ^ {1} (\mathcal {X}). \tag {1}
$$
Stemming from the theory of optimal transportation (Villani, 2009), the Wasserstein distances (see Definition A.2) are a class of metrics between probability measures. The Wasserstein distance of order 1, also referred to simply as the Wasserstein distance, is the IPM defined by the set  $\mathcal{F} = \{f:\mathcal{X}\to \mathbb{R}\text{s.t.}\| f\|_{\mathrm{Lip}}\leq 1\}$ .
Finally, we recall the definition of a pushforward measure. Let  $p$  be a probability distribution on a space  $\mathcal{Z}$  and  $g: \mathcal{Z} \to \mathcal{X}$  be a measurable function. The pushforward measure defined by  $g$  and  $p$  and denoted  $g\sharp p$  is a probability distribution on  $\mathcal{X}$  defined as  $g\sharp p(A) = p(g^{-1}(A))$ , for any measurable set  $A \subseteq \mathcal{X}$ . In other words, sampling  $\mathbf{x} \sim g\sharp p$  means sampling  $\mathbf{z} \sim p$  first, then setting  $\mathbf{x} = g(\mathbf{z})$ .
2.2 Variational Autoencoders
We consider a Euclidean observation space  $\mathcal{X}$ , a data-generating distribution  $\mu \in \mathcal{M}_{+}^{1}(\mathcal{X})$ , and a latent space  $\mathcal{Z} = \mathbb{R}^{d_{\mathcal{Z}}}$ . VAEs comprise two main components: the encoder network whose parameters are denoted  $\phi$ , and the decoder network whose parameters are denoted  $\theta$ . For simplicity, we may refer to  $\phi$  and  $\theta$  as the encoder and decoder respectively. The encoder parameterizes a distribution  $q_{\phi}(\mathbf{z}|\mathbf{x})$  over the latent space  $\mathcal{Z}$ , which is a variational approximation of the Bayesian posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$ . The likelihood  $p_{\theta}(\mathbf{x}|\mathbf{z})$  is parameterized by the decoder network. In this work, we consider the standard VAE, with a standard Gaussian prior  $p(\mathbf{z}) = \mathcal{N}(\mathbf{0},\mathbf{I})$  on  $\mathcal{Z}$  and Gaussian latent distributions  $q_{\phi}(\mathbf{z}|\mathbf{x})$ . More precisely, for any  $\mathbf{x} \in \mathcal{X}$ , the distribution  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is a Gaussian distribution with a diagonal covariance matrix  $\mathcal{N}(\mu_{\phi}(\mathbf{x}), \mathrm{diag}(\sigma_{\phi}^2 (\mathbf{x})))$ , where
$$
\mu_ {\phi}: \mathcal {X} \rightarrow \mathcal {Z} = \mathbb {R} ^ {d _ {Z}} \quad \text {a n d} \quad \sigma_ {\phi}: \mathcal {X} \rightarrow \mathbb {R} _ {> 0} ^ {d _ {Z}}.
$$
Note that  $\mathrm{diag}(\sigma)$  denotes the diagonal matrix whose main diagonal is the vector  $\sigma$ . In order to simplify some of the expressions below, it may be useful to express the encoder network as a function
$$
Q _ {\phi}: \mathcal {X} \rightarrow \mathbb {R} ^ {2 d _ {\mathcal {Z}}}, \quad \text {w h e r e} Q _ {\phi} (\mathbf {x}) = \left[\begin{array}{l}\mu_ {\phi} (\mathbf {x})\\\sigma_ {\phi} (\mathbf {x})\end{array}\right]. \tag {2}
$$
We express the decoder as a parametric function  $g_{\theta}:\mathcal{Z}\to \mathcal{X}$ . For any  $\mathbf{x}\in \mathcal{X}$ , upon receiving  $\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})$ , the decoder's output  $g_{\theta}(\mathbf{z})$  is a reconstruction of  $\mathbf{x}$ . Given a training set  $S = \{\mathbf{x}_1,\dots ,\mathbf{x}_n\}$ , the encoder and decoder networks are jointly trained by minimizing the following objective:
$$
\mathcal {L} _ {\mathrm {V A E}} (\phi , \theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \underset {\mathbf {z} \sim q _ {\phi} (\mathbf {z} | \mathbf {x} _ {i})} {\mathbb {E}} \left[ - \log p _ {\theta} \left(\mathbf {x} _ {i} \mid \mathbf {z}\right) \right] + \beta \mathrm {K L} \left(q _ {\phi} \left(\mathbf {z} \mid \mathbf {x} _ {i}\right) \| p (\mathbf {z})\right) \right], \tag {3}
$$
where the first part of (3) is the reconstruction loss and the second part is the KL-divergence between the latent distributions (associated to the training samples) and the prior over the latent space, weighted by a hyperparameter  $\beta > 0$  (Higgins et al., 2017). The reconstruction loss measures the similarity between  $\mathbf{x}$  and its reconstruction  $g_{\theta}(\mathbf{z})$ , and can be defined in many ways. With a Gaussian likelihood, the reconstruction loss is the squared  $L_{2}$  norm  $\| \mathbf{x} - g_{\theta}(\mathbf{z})\|^{2}$ .
After training, the VAE defines a generative model using the prior  $p(\mathbf{z})$  and the decoder  $g_{\theta}$  (Kingma and Welling, 2014). The distribution  $g_{\theta}\sharp p(\mathbf{z})\in \mathcal{M}_{+}^{1}(\mathcal{X})$  allows one to generate new samples by first sampling a latent vector from the prior, then passing it through the decoder. We refer to  $g_{\theta}\sharp p(\mathbf{z})$  as the VAE's generated distribution.
2.3 A Brief Introduction to PAC-Bayesian Theory
Dating back to McAllester (1999), PAC-Bayesian theory develops high-probability generalization bounds for machine learning algorithms. In essence, PAC-Bayes frames the output of such an algorithm as a posterior distribution over a class of hypotheses, and provides an upper bound on the discrepancy between a model's empirical risk and its population risk.
PAC-Bayes considers the following concepts: a hypothesis class  $\mathcal{H}$ , a training set  $S = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$  iid sampled from an unknown distribution  $\mu$  over an instance space  $\mathcal{X}^1$ , and a real-valued loss function  $\ell : \mathcal{H} \times \mathcal{X} \to [0, \infty)$ . Moreover, the primary goal of PAC-Bayes is to provide generalization bounds uniformly valid for any posterior  $q \in \mathcal{M}_{+}^{1}(\mathcal{H})$ . These bounds are dependent on the empirical performance of  $q$  and its closeness to a chosen prior distribution  $p \in \mathcal{M}_{+}^{1}(\mathcal{H})$ , as measured by the KL-divergence. The empirical and true risks of a posterior distribution  $q \in \mathcal{M}_{+}^{1}(\mathcal{H})$  are defined as
$$
\hat {\mathcal {R}} _ {S} (q) = \underset {h \sim q (h)} {\mathbb {E}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \ell (h, \mathbf {x} _ {i}) \right] \quad \text {a n d} \quad \mathcal {R} (q) = \underset {h \sim q (h)} {\mathbb {E}} \left[ \underset {\mathbf {x} \sim \mu} {\mathbb {E}} \ell (h, \mathbf {x}) \right].
$$
As an illustration, consider the following PAC-Bayesian bound for bounded loss functions developed by Catoni (2003).
Theorem 2.1. Given a probability measure  $\mu$  on  $\mathcal{X}$ , a hypothesis class  $\mathcal{H}$ , a prior distribution  $p$  on  $\mathcal{H}$ , a loss function  $\ell: \mathcal{H} \times \mathcal{X} \to [0,1]$ , real numbers  $\delta \in (0,1)$  and  $\lambda > 0$ , with probability at least  $1 - \delta$  over the random draw of  $S \sim \mu^{\otimes n}$ , the following holds for any posterior  $q \in \mathcal{M}_{+}^{1}(\mathcal{H})$ :
$$
\mathcal {R} (q) \leq \hat {\mathcal {R}} _ {S} (q) + \frac {\lambda}{8 n} + \frac {\mathrm {K L} (q | | p) + \log \frac {1}{\delta}}{\lambda}.
$$
The connection between PAC-Bayesian theory and Bayesian inference was highlighted by Grünwald (2012) and Germain et al. (2016), who showed that with a proper choice of  $\lambda$  and the negative log-likelihood as the loss function  $\ell$ , the optimal posterior minimizing the right-hand side of Catoni's bound is the Bayesian posterior. Note that although the Bayesian posterior is unique (for a given prior and likelihood), a "PAC-Bayesian posterior" could be, in principle, any distribution over  $\mathcal{H}$ .
In our PAC-Bayesian analysis of VAEs, we will use the latent space  $\mathcal{Z}$  as our hypothesis class, so that the VAE's prior will coincide with the PAC-Bayesian prior and the variational posterior  $q_{\phi}(\mathbf{z}|\mathbf{x})$  will stand for our PAC-Bayesian posterior. An immediate concern with this approach is that the encoder's distributions are conditioned on individual samples  $\mathbf{x} \sim \mu$ , whereas the usual PAC-Bayesian bounds hold for unconditional posteriors  $q(h)$ . We address this issue in the next section, by developing a novel PAC-Bayesian bound for posterior distributions  $q(\cdot|\mathbf{x})$ . This general result will be later utilized to analyze VAEs.
3 A General PAC-Bayesian Bound with a Conditional Posterior
In this section, we present our general PAC-Bayesian bound with a conditional posterior distribution. Note that the novelty of this result is not the conditioning on observations, since this can be achieved by exploiting the existing PAC-Bayesian bounds. Indeed, Haddouche and Guedj (2022) utilized the general theorem of Rivasplata et al. (2020) to derive bounds for the online learning framework.
<sup>1</sup>In supervised learning, the instance space has the form  $\mathcal{X} \times \mathcal{Y}$  where  $\mathcal{X}$  is a set of features, and  $\mathcal{Y}$  a set of labels. We use a more general formulation to encompass the unsupervised learning setting.
Instead, the contribution of Theorem 3.1 is to predict the behavior of  $q(h|\mathbf{x})$ , for any (previously unseen)  $\mathbf{x} \sim \mu$ , when the posterior  $q$  was only learned using the training samples  $\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ . To the best of our knowledge, this is the first PAC-Bayesian bound where the posterior distribution is a conditional distribution conditioned on individual elements from the instance space. This bound will require the posterior  $q$  and the loss function  $\ell$  to satisfy the following technical assumption.
Assumption 1. We say that a distribution  $q(\cdot |\mathbf{x})$  and a loss function  $\ell$  satisfy Assumption 1 with a constant  $K > 0$  if there exists a family  $\mathcal{E}$  of functions  $\mathcal{H}\to \mathbb{R}$  such that the following properties hold.
1. The function  $\mathbf{x} \mapsto q(\cdot|\mathbf{x})$  is continuous in the following sense: for any  $\mathbf{x}_1, \mathbf{x}_2 \in \mathcal{X}$ ,
$$
d _ {\mathcal {E}} \left(q (h | \mathbf {x} _ {1}), q (h | \mathbf {x} _ {2})\right) \leq K d (\mathbf {x} _ {1}, \mathbf {x} _ {2}).
$$
2. For any  $\mathbf{x} \in \mathcal{X}$ , the function  $\ell(\cdot, \mathbf{x}): \mathcal{H} \to \mathbb{R}$  is in  $\mathcal{E}$ :
$$
\ell (\cdot , \mathbf {x}) \in \mathcal {E}, \quad \text {f o r a n y} \mathbf {x} \in \mathcal {X}.
$$
Before stating the general result, let us pause and discuss this assumption. Intuitively, the goal of a generalization bound is to predict the behavior of the posterior distribution  $q(h|\mathbf{x})$  on previously unseen examples  $\mathbf{x} \sim \mu$ . Since the posterior  $q(h|\mathbf{x})$  is learned by minimizing the loss function  $\ell$  on the training samples  $S = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ , one may need two things to be true.
First, the mapping  $\mathbf{x} \mapsto q(h|\mathbf{x})$  has to be somewhat continuous. This is ensured by the first part of Assumption 1, which states that the posterior  $q$  is Lipschitz-continuous with respect to the IPM  $d_{\varepsilon}$  and the underlying metric  $d$  on  $\mathcal{X}$ . Indeed, this tells us that if  $\mathbf{x}_1$  and  $\mathbf{x}_2$  are close w.r.t. the underlying metric on  $\mathcal{X}$ , then  $q(h|\mathbf{x}_1)$  and  $q(h|\mathbf{x}_2)$  are close, w.r.t. the IPM  $d_{\varepsilon}$ .
Second, that continuity has to be "understood" by the loss function  $\ell$ , which corresponds to the second part of the assumption. It states that the loss function's discriminative power is weaker than the one defined by the IPM  $d_{\mathcal{E}}$ . In other words, the discrepancy measure used to measure the similarity between the distributions  $q(h|\mathbf{x}_1)$  and  $q(h|\mathbf{x}_2)$  needs to be just strong enough to fool the loss function into thinking that the distributions are close to each other. An alternate formulation of Assumption 1 is provided in the supplementary material (Remark F.1).
Finally, we emphasize that Assumption 1 is not as restrictive as it may seem at first. For instance, it is satisfied by a VAE's variational posterior, when the encoder and decoder networks have finite Lipschitz norms and the reconstruction loss is defined with the  $L_{2}$  norm (see Proposition 4.1). We are ready to state our first result.
Theorem 3.1. Let  $(\mathcal{X},d)$  be a metric space. Consider a probability measure  $\mu$  on  $\mathcal{X}$ , a hypothesis class  $\mathcal{H}$ , a prior distribution  $p(h)$  on  $\mathcal{H}$ , a loss function  $\ell: \mathcal{H} \times \mathcal{X} \to \mathbb{R}$ , real numbers  $\delta \in (0,1)$  and  $\lambda > 0$ . With probability at least  $1 - \delta$  over the random draw of  $S \sim \mu^{\otimes n}$ , the following holds for any conditional posterior  $q(h|\mathbf{x})$  such that Assumption 1 is satisfied by  $q(h|\mathbf{x})$  and  $\ell$  with a constant  $K > 0$ :
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mu} {\mathbb {E}} \underset {h \sim q (h | \mathbf {x})} {\mathbb {E}} \ell (h, \mathbf {x}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \underset {h \sim q (h | \mathbf {x} _ {i})} {\mathbb {E}} \ell (h, \mathbf {x} _ {i}) \leq \frac {1}{\lambda} \left[ \sum_ {i = 1} ^ {n} \mathrm {K L} (q (h | \mathbf {x} _ {i}) \mid | p (h)) + \frac {\lambda K}{n} \sum_ {i = 1} ^ {n} \underset {\mathbf {x} \sim \mu} {\mathbb {E}} d (\mathbf {x}, \mathbf {x} _ {i}) + \right. \\ \left. \log \frac {1}{\delta} + n \log \underset {h \sim p (h)} {\mathbb {E}} \underset {\mathbf {x} \sim \mu} {\mathbb {E}} e ^ {\frac {\lambda}{n} \left(\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim \mu} \ell (h, \mathbf {x} ^ {\prime}) - \ell (h, \mathbf {x})\right)} \right]. \\ \end{array}
$$
In order to prove Theorem 3.1, we start by deriving a bound where the expected loss for samples  $\mathbf{x} \sim \mu$  is computed w.r.t. distributions  $q(h|\mathbf{x}_i)$  associated to the training samples (see Lemma B.1). This result uses standard PAC-Bayesian techniques, with a key difference: we start with  $n$  iid hypotheses from the prior  $p(h)$ , then we perform the change of measure with  $n$  posteriors  $q_{\phi}(\mathbf{z}|\mathbf{x}_1), \ldots, q_{\phi}(\mathbf{z}|\mathbf{x}_n)$ , and show that the resulting exponential moment is equal to the one in Theorem 3.1. Moreover, one of the original aspects of this work comes from Assumption 1, which enables us to obtain a bound where the expected loss for  $\mathbf{x} \sim \mu$  is computed w.r.t. the posterior  $q(h|\mathbf{x})$ , associated to  $\mathbf{x}$  itself instead of all the training samples. However, the price to pay for having a posterior  $q(h|\mathbf{x})$  for each  $\mathbf{x} \in \mathcal{X}$  is that the bound depends on  $\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{\mathbf{x} \sim \mu}d(\mathbf{x},\mathbf{x}_i)$ , which we refer to as the average distance.
${}^{2}{d}_{\varepsilon }$  is a pseudo-metric in the general case,so we abuse the definition by calling this Lipschitz continuity, since the latter concept is only defined for metric spaces.