SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $S, S', \hat{W}_e \sim P_{S, \hat{W}_e} P_{S'}$  and the infimum is over all Markov kernels  $P_{\hat{W}_e|S}$  such that for  $\hat{W} = (\hat{W}_e, W_d)$ ,  $\mathbb{E}_{P_{S,W}} P_{\hat{W}_e|S}[\mathrm{gen}(S, W) - \mathrm{gen}(S, \hat{W})] \leqslant \epsilon$ .
The bound of Theorem 4 on the generalization error of representation learning, and the related IB-encoder, is to the best of our knowledge the first of its kind and significance. In fact, while few works have already investigated the generalization error of IB [SST10, VPV18, KDJH23], for the special case of discrete variables, their bounds, which in part are expressed in terms of the empirical mutual information  $\hat{I}(U;X)$ , appear to be vacuous for most settings as already observed in [RG19, GK19, AG19, DKSV20] (see introduction and Appendix B.2 for more details). Furthermore, as also extensively discussed in [AG19, GK19, DKSV20], mutual information may not be a good indicator of the generalization error. Such aspects, including how mutual information fails to reflect "simplicity" or "structure" of the encoder, are discussed in more detail in the previous section. An alternate bound on the generalization error of representation learning appeared in [DKSV20]. However, their bound only holds for encoders that find the optimal representation in a sense defined therein.
Now, few remarks on the result of Theorem 4 are in order. First, note that the generalization bound of this theorem depends only on the encoder and more precisely on the complexity of the latent space  $U$ ; and, so, this bound is valid for any choice decoder. Next, similar to the bounds of Theorems 1-5, the KL-divergence term of Theorem 4 explicitly takes into account the "structure" and "simplicity" of the encoder, and, therefore, it resolves one of the major issues of the IB method [AG19, GK19, DKSV20].
Moreover, the result of Theorem 4 suggests that the considered prior could depend on the data and model. This enables a larger class of choices for the prior. Examples include (i) Symmetric jointly Gaussian priors, (ii) priors that depend on the category (label), i.e., the prior for a category  $k \in [K]$  at each optimization iteration could possibly depend on some statistics of the latent variables of all training and test samples having label  $k$ , and (iii) priors that steer latent variables toward some pre-defined "constellations" in the latent space, depending on their label. On this aspect, note that allowing the prior to depend on labels enables a connection with the "conditional information-bottleneck" of [Fis20].
Another important property of our bound of Theorem 4 is that, unlike the empirical mutual information term of [SST10, VPV18, KDJH23], it does not become infinite for deterministic encoders with continuous input-output. Moreover, our approach which is based on lossy compression provides an interpretation of the geometric compression of [GK19, GVDBG $^{+}$ 19] where latent variables are concentrated around some
2In some cases, the bound may be loose, however, e.g., with a decoder  $w_{d}$  that produces an estimate  $\hat{Y}$  independently of the obtained representation  $U$ .
3The lossy compression trick, however, can also be applied for results of [SST10, VPV18, KDJH23].
constellation points. $^4$  We hasten to mention that "lossy compression" here should not be confused with approaches that add noise after the encoder. The main difference is that in those approaches the noisy representations are passed to the decoder; while here, the "noisy" representations are used only to estimate lossy compressibility. These "noisy" representations can be achieved by either adding small "noise" to the model parameters or the latent variable. Note that by increasing the noise level the  $\epsilon$  term in Theorem 4 increases while the KL-divergence term potentially decreases. In practice, a suitable trade-off between the two effects can be found by treating the amount of added noise as a hyper-parameter to optimize.
Finally, a similar tail bound on the generalization error has been established in Appendix A.
In this section, we illustrate our results via some experiments. For more detail and other experiments the reader is referred to Appendix D.
Our main Theorems 4 and 7 suggest that for the representation learning setup of Fig. 1b the generalization error is controlled essentially by the divergence term  $D_{KL}\Big(P_{U|X,W_e}^{\otimes 2n}(\mathbf{U},\mathbf{U}'|\mathbf{X},\mathbf{X}',\mathbf{Y},\mathbf{Y}',W_e)\| \mathbf{Q}\Big)$  where  $\mathbf{Q}$  is a type-III symmetric prior. In a sense, this also means that, with a proper data-dependent choice of the prior, the usage of the aforementioned divergence term as a regularizer possibly offers better generalization guarantees (relative, e.g., to the conventional data-independent prior of VIB). In what follows, we propose a new family of data-dependent priors which appear to better capture the "simplicity" or "structure" of the encoder. We also compare the associated accuracy with that offered by the fixed prior of VIB.
More precisely, in VIB the prior  $\mathbf{Q}$  factorizes as a product of  $2n$  scalar standard Gaussian priors, i.e.,  $\mathbf{Q} = Q^{\otimes 2n}$ , where  $Q = \mathcal{N}(\mathbf{0}_m,\mathrm{I}_m)$ ,  $\mathbf{0}_m\in \mathbb{R}^m$  is the zero vector and  $\mathrm{Im}$  is the  $m\times m$  identity matrix. In our lossless approach, which we here coin as Lossless Category-Dependent VIB (CDVIB), the prior  $\mathbf{Q}$  still factorizes as a product of  $2n$  scalar Gaussian priors, i.e.,  $\mathbf{Q} = \prod_{i\in [2n]}Q_i$ , but with three major differences: i. Each  $Q_{i}$  can be chosen from a set of  $M\times K$  priors -  $M$  priors for each label. ii. Unlike VIB, each of  $M\times K$  priors can depend on some statistics of  $(\mathbf{S},\mathbf{S}^{\prime})$  and also on the label of each sample, iii. Unlike VIB, where the prior is fixed, here  $\mathbf{Q}$  is 'learned' during the training phase. To this end, the mean and variance of the scalar priors are updated after each training iteration using a moving average with some small coefficient, allowing the latent space to better adapt to the structure of the encoder and the data. Taking the moving average also has another role, which is to "partially" reproduce the effect of the "ghost data" (which comes from the test dataset that is usually unavailable during training). In lossy CDVIB, similar priors are considered but over "noisy" versions of the latent variables, i.e.,  $(\hat{\mathbf{U}},\hat{\mathbf{U}}^{\prime})$ . Note that while the noisy versions are considered for the regularizer, the decoder receives as input  $(\mathbf{U},\mathbf{U}^{\prime})$ .
We consider CIFAR10  $\left[\mathrm{KH}^{+}09\right]$  image classification using a small CNN-based encoder and a linear decoder. The results shown in Fig. 2 indicate that the model trained using our priors achieves better  $(\sim 2.5\%)$  performance in terms of both generalization error and population risk. This suggests that our priors help to find a better representation than the standard VIB prior.
(a) Accuracy.
(b) Log-likelihood.
Figure 2: Accuracy during test phase of our two-step prediction model trained using the standard VIB prior and our "lossless" CDVIB and "lossy" CDVIB priors computed for  $M = 5$ . The values are averaged over 5 runs. The graphs are displayed together with  $95\%$  bootstrap confidence intervals.
4The reader is referred to Appendix C.3 for a simple example of geometric compression.
The authors would like to thank the anonymous reviewers for their many insightful comments and suggestions. In particular, by pointing out the connection between our results and the f-CMI literature.
[AFDM17] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
[AG19] Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, 42(9):2225-2239, 2019.
[AGNZ18] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254–263. PMLR, 2018.
[Aud04] Jean-Yves Audibert. PAC-Bayesian statistical learning theory. PhD thesis, Université Paris VI., 2004.
[AZ19] Inaki Estella Aguerri and Abdellatif Zaidi. Distributed variational representation learning. IEEE transactions on pattern analysis and machine intelligence, 43(1):120-138, 2019.
$\left[\mathrm{BBR}^{+}18\right]$  Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062, 2018.
[BEHW87] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam's razor. Information processing letters, 24(6):377-380, 1987.
[BHMZ20] Olivier Bousquet, Steve Hanneke, Shay Moran, and Nikita Zhivotovskiy. Proper learning, helly number, and an optimal svm bound. In Conference on Learning Theory, pages 582-609. PMLR, 2020.
[BL03] Avrim Blum and John Langford. Pac-mdl bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pages 344-357. Springer, 2003.
[BM06] Nader H Bshouty and Hanna Mazzawi. Exact learning composed classes with a small number of mistakes. In Learning Theory: 19th Annual Conference on Learning Theory, COLT 2006, Pittsburgh, PA, USA, June 22-25, 2006. Proceedings 19, pages 199-213. Springer, 2006.
[BO18] Léonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018.
$\left[\mathrm{BSE}^{+}21\right]$  Melih Barsbey, Milad Sefidgaran, Murat A Erdogdu, Gael Richard, and Umut Şimsekli. Heavy tails in SGD and compressibility of overparametrized neural networks. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[BZV20] Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening mutual information-based bounds on generalization error. IEEE Journal on Selected Areas in Information Theory, 1(1):121-130, May 2020.
[Cat07] Olivier Catoni. Pac-bayesian supervised classification. Lecture Notes-Monograph Series. IMS, 1277, 2007.
[CK11] Imre Csiszár and János Körner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Cambridge University Press, 2 edition, 2011.
[CK22] Dan Tsir Cohen and Aryeh Kontorovich. Learning with metric losses. In Conference on Learning Theory, pages 662-700. PMLR, 2022.
The appendices are organized as follows.
- Appendix A contains further theoretical results. Specifically,
- in Appendix A.1 two more tail bounds on the generalization error in terms of the predicted label complexity are presented.
- in Appendix A.2 a tail bound on the generalization error in terms of the latent variable complexity is presented.
- We discuss some of the related works in more detail in Appendix B. In particular,
- in Appendix B.1, we recall the compressibility framework of Blum and Langford [BL03].
- in Appendix B.2, we discuss the results of [KDJH23].
- In Appendix C, we discuss various topics that could not be sufficiently addressed in the paper, due to lack of space. More precisely,
- we define various notions of compressibility, including lossless, lossy, and variable-size compressibility in Appendix C.1,
- we present an intuition on the function  $h_D(\cdot, \cdot)$ , used in Section 2, in Appendix C.2,
- we give a simple example of geometrical compression, and its relation to lossy compressibility, in Appendix C.3,
- and finally we present the conclusion of our work together with some interesting future directions in Appendix C.4.
- Appendix D contains the details of the experiments presented in Section 4.
- Appendix D.1 recalls the standard VIB objective function,
- Appendix D.2 defines a family of lossless objective functions using data-dependent priors,
- Appendix D.3 defines a family of lossy objective functions using data-dependent priors,
- Appendix D.4 details the training and testing datasets used in experiments,
- Appendix D.5 details the model architecture used in experiments,
- Appendix D.6 provides training details,
- Appendix D.7 presents and discusses the numerical results.
- Appendix E contains the deferred proofs of all our theoretical results. More precisely,
- the intuition behind our proof techniques is presented in Appendix E.1,
- Appendix E.2 contains the proof of Theorem 1,
- Appendix E.3 contains the proof of Lemma 1,
- Appendix E.4 contains the proof of Theorem 3,
- Appendix E.5 contains the proof of Theorem 4,
- Appendix E.6 contains the proof of Theorem 5,
- Appendix E.7 contains the proof of Theorem 6,
- Appendix E.8 contains the proof of Theorem 7,
- Appendix E.9 contains the proof of Lemma 3.
In this section, we present several tail bounds on the generalization error in terms of the complexity of the predicted labels or the latent variables.
A.1 Tail bounds on the generalization error in terms of predicted label complexity
First, we start by stating a lossy tail bound on the generalization error in terms of the complexity of the tail bound.
Theorem 5. Consider the learning framework defined above. Let  $\mathbf{Q}$  be any fixed type-I symmetric prior on  $(\hat{\mathbf{Y}},\hat{\mathbf{Y}}^{\prime})$  that could depend on  $(\mathbf{X},\mathbf{Y},\mathbf{X}^{\prime},\mathbf{Y}^{\prime})$ . Then for any  $\epsilon \in \mathbb{R}$  and  $\delta \in \mathbb{R}^{+}$  with probability at least  $(1 - \delta)$  over choice of  $S$  and  $S^{\prime}$ , we have that  $\mathbb{E}_{W\sim P_{W|S}}[\mathrm{gen}(S,W)]$  is upper bounded by
$$
\sqrt {\frac {\log (2 / \delta)}{2 n}} + \inf  \sqrt {\frac {4}{2 n - 1} \left(\mathbb {E} _ {\hat {W} \sim P _ {\hat {W} | S}} \left[ D _ {K L} \left(P _ {\hat {Y} | X , \hat {W}} ^ {\otimes 2 n} (\hat {\mathbf {Y}}, \hat {\mathbf {Y}} ^ {\prime} | \mathbf {X} , \mathbf {X} ^ {\prime} , \hat {W}) \| \mathbf {Q}\right) \right] + \log (\sqrt {8 n} / \delta)\right) + \epsilon},
$$
where the infimum is over all  $P_{\hat{W} |S}$  that satisfy
$$
\left. \mathbb {E} _ {P _ {W | S} P _ {\hat {W} | S}} \left[ \left| \left(\hat {\mathcal {L}} (S ^ {\prime}, W) - \hat {\mathcal {L}} (S, W)\right) - \left(\hat {\mathcal {L}} (S ^ {\prime}, \hat {W}) - \hat {\mathcal {L}} (S, \hat {W})\right) \right| \right] \leqslant \epsilon / 2. \right. \tag {6}
$$
The theorem is proved in Appendix E.6. In the above result, it is easy to replace the expectation with respect to  $P_{W|S}$  with any arbitrary expectations, as it is common in PAC-Bayes bounds and used for example in [NBS18].
Next, we state a tail bound in terms of the function  $h_D(\cdot ,\cdot)$ , defined in (5).
Theorem 6. Consider the learning framework of Section 2. Let  $\mathbf{Q}$  be any fixed type-II symmetric prior on  $(\hat{\mathbf{Y}},\hat{\mathbf{Y}}^{\prime})$  that could depend on  $(\mathbf{X},\mathbf{Y},\mathbf{X}^{\prime},\mathbf{Y}^{\prime})$ . Then, for any  $\delta \in \mathbb{R}^{+}$ , with probability at least  $(1 - \delta)$  over choices of  $(S,S^{\prime},W)\sim P_{S^{\prime}}P_{S,W}$ , it holds that
$$
n h _ {D} \left(\hat {\mathcal {L}} (S ^ {\prime}, W), \hat {\mathcal {L}} (S, W)\right) \leqslant D _ {K L} \left(P _ {\hat {Y} | X, W} ^ {\otimes 2 n} (\hat {\mathbf {Y}}, \hat {\mathbf {Y}} ^ {\prime} | \mathbf {X}, \mathbf {X} ^ {\prime}, W) \| \mathbf {Q}\right) + \log (n / \delta).
$$
This theorem is proved in Appendix E.7. In particular, when  $\hat{\mathcal{L}}(S,W) = 0$ , using Lemma 1 conclude that, this result yields that with probability at least  $(1 - \delta)$ ,
$$
\hat {\mathcal {L}} (S ^ {\prime}, W) \leqslant \frac {D _ {K L} \left( \right.P _ {\hat {Y} | X , W} ^ {\otimes 2 n} (\hat {\mathbf {Y}} , \hat {\mathbf {Y}} ^ {\prime} | \mathbf {X} , \mathbf {X} ^ {\prime} , W) \left\| \right. \mathbf {Q}\left. \right) + \log (n / \delta)}{n}.
$$
A.2 Tail bounds on the generalization error in terms of latent variable complexity
In this section, we present a tail generalization bound for the two-step learning setup of Section 3 that highlights the relevance of latent variable compressibility to generalization error.
Theorem 7. Consider a  $K$ -classification learning task with the above defined framework. Let  $\mathbf{Q}$  be a type-III symmetric conditional prior over  $U^{2n}$  given  $X^{2n}, Y^{2n}$  and  $W_{e} \in \mathcal{W}_{e}$ , namely,  $\mathbf{Q}\left((u_{\pi(1)}, \ldots, u_{\pi(2n)})|(x_1, \ldots, x_{2n}), (y_1, \ldots, y_{2n}), w\right)$  remains the same for all permutations  $\pi \colon [2n] \mapsto [2n]$  that preserves the label, i.e.,  $y_{\pi(i)} = y_i$  for  $i \in [n]$ . Then, for any  $\lambda \in \mathbb{R}^+$ ,
i. with probability at least  $(1 - \delta)$  over choices of  $(S, S', W) \sim P_{S'}P_{S,W}$ ,
$$
\hat {\mathcal {L}} \left(S ^ {\prime}, W\right) - \hat {\mathcal {L}} (S, W) \leqslant \frac {D _ {K L} \left(P _ {U | X , W _ {e}} ^ {\otimes 2 n} \left(\mathbf {U} , \mathbf {U} ^ {\prime} \mid \mathbf {X} , \mathbf {X} ^ {\prime} , W _ {e}\right) \| \mathbf {Q}\right) + (K + 2) / 2 + \log (1 / \delta)}{\lambda} + \frac {2 \lambda}{n},
$$
ii. with probability at least  $(1 - \delta)$  over choices of  $(S, S', W) \sim P_{S'}P_{S,W}$ ,
$$
\operatorname {g e n} (S, W) \leqslant \frac {D _ {K L} \left(P _ {U | X , W _ {e}} ^ {\otimes 2 n} (\mathbf {U} , \mathbf {U} ^ {\prime} | \mathbf {X} , \mathbf {X} ^ {\prime} , W _ {e}) \| \mathbf {Q}\right) + (K + 2) / 2 + \log (2 / \delta)}{\lambda} + \frac {2 \lambda}{n} + \sqrt {\frac {\log (2 / \delta)}{n}}.
$$
Note that the second part of the theorem can be easily derived from the first part using Hoeffding's inequality. We provide the proof of the first part in Appendix E.8.
In this section, we discuss some of the related works in more detail.
B.1 PAC-MDL framework of Blum and Langford
Here, we recall the PAC-MDL framework of [BL03] which, therein, was introduced in the form of a (compression) game between two agents, Alice and Bob. Alice has access to both a labeled training set  $S = (X^n, Y^n)$ , consisting of  $n$  labeled samples, and a test set  $S' = X'^n$ , consisting of  $n$  unlabeled samples, all drawn independently from  $\mu$ . Bob has available just the test set and the unlabeled version of the training set, i.e.,  $(X^n, X'^n)$ , ordered in some predefined (e.g., lexicographic) order known to both agents. Hereafter we denote the ordered vector of labeled and unlabeled samples as  $\pi(X^n, X'^n)$ . It is assumed that by observing the vector  $\pi(X^n, X'^n)$  Bob cannot know which samples of it are from the training set and which are from the test set. The goal of Alice is to communicate the labels  $\pi(Y^n, Y'^n)$  to Bob using as few bits as possible. To this end, let  $\mathcal{E} \colon \mathcal{Z}^n \times \mathcal{X}^n \to \{0, 1\}^*$  be a mapping (encoder) used by Alice and  $\sigma \coloneqq \mathcal{E}(S, X'^n)$  the string transmitted to Bob. The goal of Bob is to guess the labels of both sets  $X^n$  and  $X'^n$ ; and does so by running a decoder mapping  $\mathcal{D} \colon \mathcal{X}^{2n} \times \{0, 1\}^* \to \mathcal{Y}^{2n}$ . That is, Bob forms an estimate of the true labels as  $\mathcal{D}(\pi(X^n, X'^n), \sigma)$ . In this context, the empirical risk is measured as  $\hat{\mathcal{L}}(\sigma, s, s') \coloneqq \frac{1}{n} \sum_{i \in [n]} \mathbb{1}_{\{y_i \neq \hat{y}_i\}}$  and a proxy test risk is measured as  $\mathcal{L}(\sigma, s, s') \coloneqq \frac{1}{n} \sum_{i \in [n]} \mathbb{1}_{\{y_i' \neq \hat{y}_i'\}}$ . Essentially, the PAC-MDL bound of [BL03] can be seen as a generalized version of Occam's-Razor theorem [LW86, BEHW87] which states that if one can explain (or encode) the labels of a set of  $n$  training samples by a hypothesis that can be described using only  $|\sigma| \ll n$  bits then this guarantees that this hypothesis generalizes well for unseen samples.
Theorem 8 ([BL03, Theorem 6]). For any prior distribution  $\mathbf{Q}(\sigma)$  of  $\sigma$  and any  $\delta > 0$ , with probability at least  $1 - \delta$  over the choices of  $S, S' \sim P_S P_{S'}$ , we have:
$$
\forall \sigma : n \mathcal {L} (\sigma , S, S ^ {\prime}) \leqslant b _ {\max } \left(n, \hat {\mathcal {L}} (\sigma , S, S ^ {\prime}), \mathbf {Q} (\sigma) \delta\right), \tag {7}
$$
$$
b _ {\max } \left(n, \frac {a}{n}, \delta\right) := \max  \{b: B u c k e t (n, a, b) \geqslant \delta \}, \tag {8}
$$
$$
\operatorname {B u c k e t} (n, a, b) := \sum_ {c \in [ b, a + b ]} \frac {\binom {n} {c} \binom {n} {a + b - c}}{\binom {2 n} {a + b}}. \tag {9}
$$
The result is made more explicit for the so-called realizable case, i.e., when  $\hat{\mathcal{L}} (\sigma ,S,S^{\prime}) = 0$ . For this case, we have [BL03, Corollary 3]
$$
\mathbb {P} \left(\forall \sigma : \hat {\mathcal {L}} (\sigma , S, S ^ {\prime}) > 0 \text {o r} \mathcal {L} (\sigma , S, S ^ {\prime}) \leqslant \left(| \sigma | + \log_ {2} (1 / \delta)\right) / n\right) > 1 - \delta ,
$$
where  $|\sigma|$  denotes the size in bits of the string  $\sigma$  using some a priori fixed codewords. The result clearly shows that if the string  $\sigma$  can be sent using few bits then the algorithm generalizes well. Also, the approach can be used in order to establish similar results for the VC-dimension and PAC-Bayes bounds.
We emphasize that our construction is in sharp contrast with an adaptation of the above-described approach of Blum and Langford [BL03] in which one would reveal both labeled and unlabeled samples  $(S^{m},X^{\prime mn})$  to Alice and only unlabeled samples  $(X^{mn},X^{\prime mn})$  to Bob (both in a predefined order). We, however, in our proposed approach in Section 2.1, reveal the vector  $\mathfrak{Z}^{2mn}$ , containing  $(S^{m},S^{\prime m})$  in a predefined order to both. Furthermore, we emphasize that the goal in our approach is not to recover the labels  $\mathfrak{Y}^{2mn}$ , but the predictions  $\hat{\mathfrak{Y}}^{2mn}$  as produced by the picked models or hypotheses  $W^{m}$ .
On another note, we remark that the interpretation of  $R$ , which appeared in the size of the codebook in our approach, i.e.,  $|\hat{\mathcal{Y}}_m| \leqslant e^{mR}$ , corresponds to the average number of bits (per dataset  $S$ ) that is needed to send a compressed version of the string  $\sigma$  of [BL03]. As such, for the fixed-size codebook explained in Section 2.1 and when  $|\sigma|$  is constant,  $R \leqslant |\sigma|$ . Similar relations hold in general, by considering the variable-size codebook.
For simplicity, we assume here that the training and test sets  $S$  and  $S'$  have identical sizes, i.e.,  $|S| = |S'|$ ; but all the results that will follow extend easily to the case of  $|S| \neq |S'|$ .
Note that Alice does not know the true labels  $Y'^n$  of the test samples  $X'^m$  and can only estimate them as  $\hat{Y}'^n$ .
B.2 On the generalization bounds of [KDJH23]
After the initial submission of our work, another work [KDJH23] appeared on arXiv, accepted at ICML 2023, that provides an upper bound on the generalization error of representation learning algorithms. As claimed, this result justifies the benefits of the information bottleneck principle, by relating IB to the generalization error. The results are provided for the multi-layer neural networks and for different choices of latent variables corresponding to the output of different layers. Here, to adapt the results to the setup of this paper, we only consider the output of the encoder layer as the latent variable and adapt correspondingly the notations of the results in [KDJH23] to our notations.
With the adapted notations, [KDJH23] claims to bound the generalization error of a representation learning algorithm "roughly by
$$
\tilde {\mathcal {O}} \left(\sqrt {\frac {I (X ; U | Y) + 1}{n}}\right).
$$