SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, firstly, since this bound is in terms of the mutual information function, the critics on the relation of mutual information and generalization error, discussed in [KTVK18, RG19, AG19, DKSV20] and provided also in the "Critics to IB" section of the introduction (Section 1), are valid for the results of [KDJH23], as well. More importantly, we could not conclude the reported order-wise behavior from [KDJH23, Theorem 2]. Using the notations of our work, their result states that with probability at least  $1 - \delta$  over training data  $S$ , the generalization error is bounded by
$$
G _ {3} \sqrt {\frac {(I (X ; U | Y) + I (W _ {e} ; S)) \ln (2) + \hat {\mathcal {G}} _ {2}}{n}} + \frac {G _ {1}}{\sqrt {n}},
$$
where the constants  $G_{1}$ ,  $\hat{\mathcal{G}}_2$ , and  $G_{3}$  are defined in [KDJH23, Appendix E.1] and claimed that  $\hat{\mathcal{G}}_2 \sim \tilde{\mathcal{O}}(1)$  and  $G_{3} \sim \tilde{\mathcal{O}}(1)$ , as  $n \to \infty$ . However, we were unable to resolve the following concerns regarding this bound.
i. Firstly, it is not clear how  $I(W_{e};S)$  is considered to behave as  $\tilde{\mathcal{O}}(1)$ , when  $n \to \infty$ . Indeed, the size of dataset  $S$  and the learning algorithm  $P_{W_{e}|S}$  change as  $n \to \infty$ .
ii. Secondly, by referring to the definitions of the constants in [KDJH23, Appendix E.1], it can be easily verified that  $\hat{\mathcal{G}}_2 = C_1 + (H(U|X,Y) + H(W_e|S))\ln(2)$ , for some non-negative constant  $C_1$ . Hence, the bound can be re-written as
$$
G _ {3} \sqrt {\frac {(H (U | Y) + H (W _ {e})) \ln (2) + C _ {1}}{n}} + \frac {G _ {1}}{\sqrt {n}}.
$$
Thus, the bound is in terms of  $H(U|Y) + H(W_e)$ , and not  $I(X;U|Y) + I(W_e;S)$ .
iii. Lastly, the term  $G_{3}$ , defined in [KDJH23, Appendix E.1], is composed of  $T_{Y} \in \mathbb{N}$  elements. By [KDJH23, Lemma 2],  $T_{Y}$  behaves roughly as  $\tilde{\mathcal{O}}(2^{H(U|Y)})$ . Using [KDJH23, Lemma 2], it is shown that  $G_{3}$  is bounded. However, it seems that this is only shown by assuming that the  $T_{Y}$  terms of  $G_{3}$  satisfy certain conditions related to the decreasing rate of their ordered values. This assumption however, may not hold in general, and hence  $G_{3}$  may behave as  $\tilde{\mathcal{O}}(T_{y}) \approx \tilde{\mathcal{O}}(2^{H(U|Y)})$ , which then becomes the dominant term in the bound.
C Further clarifications and discussions
In this section, we explain our compressibility framework in more detail. Moreover, we present some intuitions on the function  $h_D(\cdot, \cdot)$  and give an example of how lossy compression is related to geometrical compression. Finally, we discuss some potential future works.
C.1 Compressibility framework
In this section, we propose various notions of the compressibility used in Section 2.
C.1.1 Lossless fixed-size compressibility
We start by recalling the needed elements for joint compression of a block of the predicted labels. Consider  $m$  i.i.d. pairs of train and test datasets  $S_{j} \coloneqq (Z_{j,1},\ldots ,Z_{j,n})$  and  $S_{j}^{\prime} \coloneqq (Z_{j,1}^{\prime},\ldots ,Z_{j,n}^{\prime})$ ,
where  $Z_{j,i} = (X_{j,i},Y_{j,i})$  and  $Z_{j,i}' = (X_{j,i}',Y_{j,i}')$ . Let  $S^m = (S_1,\dots,S_m)$ ,  $S'^m = (S_1',\dots,S_m')$  and  $W^m := (W_1,\dots,W_m)$ , where  $W_j \sim P_{W_j|S_j}$ . It is important to note that the model or hypothesis  $W_j$  is chosen based on the dataset  $S_j$  only and the introduction of the (ghost) dataset  $S_j'$  is here only for the sake of the analysis, similarly as it was done in the derivation of the PAC-MDL bound of [BL03] or in Rademacher sample complexity or the conditional mutual information of [SZ20]. Denote the predicted labels using model  $W_i$  for inputs  $X_{i,j}$  and  $X_{j,i}'$  as  $\hat{Y}_{j,i}$  and  $\hat{Y}_{j,i}'$ , for  $j \in [m]$  and  $i \in [n]$ .
Moreover, let  $\mathfrak{Z}^{2mn} \in \mathcal{Z}^{2mn}$  denote a rearrangement of the elements of  $(S^m, S'^m)$  such that while the entire sets  $S^m$  and  $S'^m$  are known by having the vector  $\mathfrak{Z}^{2mn}$ , but they are arranged in an order that one cannot distinguish whether a given sample  $z$  is from  $S^m$  or  $S'^m$ . Accordingly, denote the rearranged versions of  $(Y^{mn}, Y'^{mn})$  and  $(\hat{Y}^{mn}, \hat{Y}'^{mn})$  respectively by  $\mathfrak{V}^{2mn}$  and  $\hat{\mathfrak{V}}^{2mn}$ . In Sections 2.2 and 2.3, we presented two methods for such rearrangement. As mentioned before, in both methods, the rearrangement of  $(S^m, S'^m)$  as  $\mathfrak{Z}^{2mn} = \{\mathfrak{Z}_{j,i}\}_{j \in [m], i \in [n]}$  is done in two steps: First, for each  $j \in [m]$ , we rearrange indistinguishably  $(S_j, S_j')$  as  $\mathfrak{Z})j^{2n}$ , and then we concatenated the  $m$  resulting  $\{\mathfrak{Z}_j^{2n}\}_{j \in [m]}$  to achieve  $\mathfrak{Z}^{2mn}$ . Now, we explain how to rearrange indistinguishably a given  $(S, S')$  as  $\mathfrak{Z}^{2n}$  using two methods:
- Section 2.2: Let  $\mathbf{J} = (J_1, \ldots, J_n)$  be a vector of  $n$  i.i.d. Bernoulli  $(\frac{1}{2})$  random variables  $J_i \in \{i, i + n\}$ ,  $i \in [n]$ . Denote by  $\mathbf{J} = (J_1^c, \ldots, J_n^c)$  the vector of the complementary choices in  $\mathbf{J}$ , i.e.,  $J_i \cup J_i^c = \{i, i + n\}$ . Now, for each  $i \in [n]$ , let  $(\mathfrak{Z}_{J_i}, \mathfrak{Z}_{J_i^c})$  to be equal to  $(Z_i, Z_i')$ .
- Section 2.3: Let  $\mathbf{T} = \{T_1, \dots, T_n\}$ , be a random set obtained by picking uniformly  $n$  indices from  $\{1, \dots, 2n\}$ , without replacement. Note that in contrast to  $\mathbf{J}$  which had i.i.d. components, here the components of  $\mathbf{T}$  are dependent. Let  $\mathbf{T}^c = \{1, \dots, 2n\} \backslash \mathbf{T}$  be the complement of  $\mathbf{T}$ , having the elements  $\mathbf{T}^c = \{T_1^c, \dots, T_n^c\}$ . Now, for each  $i \in [n]$ , let  $(\mathfrak{Z}_{T_i}, \mathfrak{Z}_{T_i}^c) = (Z_i, Z_i')$ .
Based on the above, hereafter we study the compressibility of the sorted model-predicted labels  $\hat{\mathfrak{Y}}^{2mn}$ , from an information-theoretic point of view. The rationale is that, in accordance with "Occam's Razor" theorem [LW86, BEHW87], since the (rearranged) predicted-labels vector  $\hat{\mathfrak{Y}}^{2mn}$  agrees mostly with the true labels  $\mathfrak{Y}^{2mn}$  on the dataset  $S^m$ , if  $\hat{\mathfrak{Y}}^{2mn}$  can be described using only a few bits, this guarantees that the model  $W$  generalizes well. As it will be shown from the result that will follow, instead of the size of the message needed to be sent in the Blum-Langford approach (see Appendix B.1), a new quantity emerges in our work as a measure of the compressibility of  $\hat{\mathfrak{Y}}^{2mn}$ : the relative entropy of the joint conditional  $P(\hat{Y}^n, \hat{Y}'^n | Y^n, Y'^n)$  and a (symmetric) conditional prior  $\mathbf{Q}$  over  $\hat{\mathcal{Y}}^{2n}$  given  $Y^{2n}$ . Depending on the way we rearrange  $(S^m, S'^m)$ , the type-I or type-II symmetric priors are needed to be applied.
Let  $R \in \mathbb{R}^{+}$ . In our block-coding rate-distortion theoretic framework,  $R$  is said to be achievable if there exists a compression codebook of size  $\approx e^{mR}$ , fixed a priori, which covers the space spanned by the model-predicted labels  $\hat{\mathfrak{Y}}^{2mn}$  with high probability. Specifically, if there exists a sequence of label books  $\{\hat{y}_m\}_{m \in \mathbb{N}}$ , where  $\hat{y}_m \coloneqq \{\hat{\mathbf{y}}[r], r \in [l_m]\} \subseteq \mathcal{Y}^{2mn}$ ,  $l_m \in \mathbb{N}$ ,  $\hat{\mathbf{y}}[r] = (\hat{\mathbf{y}}_1[r], \dots, \hat{\mathbf{y}}_m[r])$  and  $\hat{\mathbf{y}}_j[r] = (\hat{y}_{j,1}[r], \dots, \hat{y}_{j,2n}[r]) \in \mathcal{Y}^{2n}$  such that:
(i)  $l_{m}\leqslant e^{mR},$
(ii) there exist a sequence  $\{\delta_m\}_{m\in \mathbb{N}}$  for which  $\lim_{m\to \infty}\delta_m = 0$  and with probability at least  $(1 - \delta_{m})$  over the choices of  $S^m$ ,  $S'^m$ , and  $\mathbf{J}$  or  $\mathbf{T}$ , one can find at least one index  $r\in [l_m]$  whose associated  $\hat{\mathbf{y}}[r]$  exactly equals  $\hat{\mathfrak{Y}}^{2mn}$ .
The above framework, explained in Section 2.1, is called fixed-size compressibility, as the size of the codebook does not depend on a given dataset. This type of compressibility is useful for establishing "data-independent" bounds, $^{7}$  e.g., bounds on the expectation of the generalization error. This is also called lossless, since we look for a codeword that is exactly equal to  $\hat{\mathfrak{Y}}^{2mn}$ .
C.1.2 Lossy fixed-size compressibility
As already stated, the above approach and results can be extended to any bounded loss function and continuous variables  $Y$  and  $\hat{Y}$  (see for example Section 3 where continuous latent variables are studied). In this case, a standard result of rate-distortion theory states that to cover losslessly and reliably  $\hat{\mathfrak{V}}^{2mn}$ ,  $R$  should be infinity, which makes the framework and resulting bounds vacuous. However, the approach can
Here, we emphasize that by data-dependent bounds we refer to bounds that depend on the particular sample of the input data at hand, rather than for example just on the distribution of the data which is unknown.
be easily extended to include the lossy compression of the labels; in the same spirit as done in [SGRS22] for the hypothesis compression. More precisely, for a given distortion threshold  $\epsilon \in \mathbb{R}$ , one can consider the same way of codebook generation, but with the following conditions:
(i)  $l_{m}\leqslant e^{mR},$
(ii) there exist a sequence  $\{\delta_m\}_{m\in \mathbb{N}}$  for which  $\lim_{m\to \infty}\delta_m = 0$  and with probability at least  $(1 - \delta_{m})$  over the choices of  $S^m$ ,  $S'^m$ , and  $\mathbf{J}$  or  $\mathbf{T}$ , one can find at least one index  $r\in [l_m]$  whose associated  $\hat{\mathbf{y}}[r]$  satisfies:
$$
\frac {1}{m n} \sum_ {j \in [ m ], i \in [ n ]} \left(\left(\mathbb {1} _ {\{\mathfrak {Y} _ {j, t _ {i} ^ {c}} \neq \hat {\mathfrak {Y}} _ {j, t _ {i} ^ {c}} \}} - \mathbb {1} _ {\{\mathfrak {Y} _ {j, t _ {i}} \neq \hat {\mathfrak {Y}} _ {j, t _ {i}} \}}\right) - \left(\mathbb {1} _ {\{\mathfrak {Y} _ {j, t _ {i} ^ {c}} \neq \hat {y} _ {j, t _ {i} ^ {c}} [ r ] \}} - \mathbb {1} _ {\{\mathfrak {Y} _ {j, t _ {i}} \neq \hat {y} _ {j, t _ {i}} [ r ] \}}\right)\right) <   \epsilon ,
$$
where it is assumed that the set of indices  $\{(j, t_i)\}_{j \in [m], i \in [n]}$  and  $\{(j, t_i^c)\}_{j \in [m], i \in [n]}$  are the sets of indices in the rearranged sequence that belong to the training and ghost datasets, respectively. Here,  $\bigcup_{i \in [n]} (t_i \cup t_i^c) = \{1, \ldots, 2n\}$ .
In the case of lossy compressibility, the block-coding technique brings another advantage in addition to the advantages discussed for the lossless case: it allows to consider the average distortion criterion, instead of worst-case distortion criterion. Please refer to [SGRS22] for further discussion on this.
C.1.3 Variable-size compressibility
In the frameworks, explained in previous sections, for a given  $m$ , the size of the codebook  $e^{mR}$  is fixed. Hence, the bounds derived using this framework, cannot depend on a particular training dataset at hand. In particular, while above mentioned frameworks are useful to establish bounds on the expectation of the generalization error, they are not appropriate for establishing data-dependent tail bounds. To overcome this issue, in [SZ23], a "variable-size" compressibility is proposed, in which, one is allowed to search for a suitable  $\hat{\mathbf{y}}[r]$  only in the first  $e^{mR_{(S,S',W)}}$  elements of this codebook, where  $R_{(S,S',W)}$  is the data-dependent term that will consequently appear in the resulting bound using this approach. For instance, the term  $R_{(S,S',W)}$  in Part.ii of Theorem 1 is exactly the KL-divergence term appearing in the bound.
C.2 Intuition about the function  $h_{D}$
In Section 2.3, we have provided the bound on the generalization error in terms of the function  $h_D(x; x') \colon [0,1] \times [0,1] \to [0,2]$ , defined as:
$$
h _ {D} (x, x ^ {\prime}) := 2 h _ {b} \left(\frac {x + x ^ {\prime}}{2}\right) - h _ {b} (x) - h _ {b} (x ^ {\prime}),
$$
where  $h_b(x) \coloneqq -x\log_2(x) - (1 - x)\log_2(1 - x)$ . As mentioned,  $\frac{1}{2} h_D(x,x')$  is equal to the Jensen-Shannon divergence between two binary Bernoulli distributions with parameters  $x$  and  $x'$ .
In this section, we provide an intuition about this function, by showing its relation with the combinatorial term that appeared in [BL03]. Recall that the main result of [BL03], i.e., Theorem 6 therein (re-stated in the Appendix B.1), is established in terms of  $b_{\max}(n,a / n,\delta)$  defined as follows:
$$
b _ {\max } \left(n, \frac {a}{n}, \delta\right) := \max  \{b \colon \operatorname {B u c k e t} (n, a, b) \geqslant \delta \},
$$
$$
\operatorname {B u c k e t} (n, a, b) := \sum_ {c \in [ b, a + b ]} \frac {\binom {n} {c} \binom {n} {a + b - c}}{\binom {2 n} {a + b}},
$$
where  $[b, a + b] \subset \mathbb{N}$  denotes the integer interval and  $c \in \mathbb{N}$ .
Now, the intuition about the function  $h_D(x,x')$  is as follows: by Stirling's formula, we have that
$$
\frac {\binom {m n} {m t} \binom {m n} {m a + m b - m t}}{\binom {2 m n} {m a + m b}} \longrightarrow e ^ {- m n h _ {D} \left(\frac {t}{n}, \frac {a + b - t}{n}\right)} \tag {10}
$$
as  $m\to \infty$ . Hence, for large (infinite) values of  $m$  we have that the function  $\mathrm{Bucket}(mn,ma,mb)$  of [BL03] is dominated by
$$
\max  _ {t \in [ [ b, a + b ] ]} e ^ {- m n h _ {D} \left(\frac {t}{n}, \frac {a + b - t}{n}\right)}, \tag {11}
$$
where  $\llbracket b, a + b\rrbracket \subset \mathbb{R}$  denotes the real interval and  $t \in \mathbb{R}$ . Furthermore and as a consequence
$$
\frac {1}{m} b _ {\max } \left(m n, \frac {a}{n}, \delta^ {m}\right) \longrightarrow \max  \left\{b: \min  _ {t \in [ b, a + b ]} n h _ {D} \left(\frac {t}{n}, \frac {a + b - t}{n}\right) \leqslant \log (1 / \delta) \right\}, \tag {12}
$$
as  $m\to \infty$
It can be observed that by considering block-coding and letting  $m\to \infty$  , the intractable combinatorial terms appeared in [BL03] can be expressed in terms of the function  $h_D(\cdot ,\cdot)$
C.3 On relation between lossy compressibility and geometric compression
The experimental studies suggest the existence of a relation between generalization performance and geometrical compression [GK19]. Geometrical compression occurs when the latent variables are concentrated around a limited number of clusters. Please refer to [GK19, Fig. 2] for a visual representation. As mentioned both in Section 3, lossy compression provides an interpretation of the geometric compression of [GK19,  $\mathrm{GVDBG}^{+}19]$  where latent variables are concentrated around some constellation points. In this section, we provide a simple example of that.
In geometrical compression, the latent variables  $U \sim P_{U|X}$  of  $X$  are distinct but concentrated around a few "centers". Hence, while in this case, the "lossless compression" captured by  $I(U;X)$  becomes very large or infinite, the "lossy compression" captured by rate-distortion may be very small, as these scattered latent variables around the centers can be seen as a lossy version of the mappings from  $X$  to one of these centers, with some small distortion. To give a simple example, suppose  $X \in [0,1]$ , and if  $X < 0.5$ , then  $U = -1 + X/5$ , and if  $X > 0.5$ , then  $U = 1 + X/5$ . In this case while  $I(U;X) = \infty$ , a simple lossy mapping  $P_{\hat{U}|X}$  with average distortion less than 0.05 can be found such that  $I(\hat{U};X) = 1$ . This explains the geometrical compression observation.
C.4 Conclusion and future directions
In this paper, inspired by [BL03], we developed a compressibility framework that we used to establish various bounds on the generalization error of the stochastic learning algorithms. The bounds are expressed in terms of the newly defined notion of "minimum description length"(MDL) of the predicted labels or the latent variables. The new notion is expressed in terms of a "symmetric" prior, where the "symmetry" is defined and used in three different ways. The type-I and type-II symmetries are useful to establish the bounds in terms of MDL of the predicted labels. The former one, in particular, shows a clear connection between the seemingly different approaches of [BL03] and CMI [SZ20, HRVSG21]. The type-III symmetry is used to derive the main result of this paper which is a bound in terms of MDL of latent variables. The results of the last section suggest that the generalization error in representation learning is related to the newly defined MDL of latent variables. Unlike mutual information, which captures the "information leakage", the new notion of MDL captures the simplicity and structure of the encoder. These insights are then partly exploited to propose new regularizers in terms of new "data-dependent" priors. The performed simulations show the advantage of these priors over the classical ones used in VIB.
Our proposed framework and obtained results also open up several future research directions. In the following, we discuss some of those directions.
- In Part i. of Theorem 1, we proposed a tail bound, which unlike the classical PAC-Bayes bound with a single draw from the posterior [Cat07], does not contain a "disintegrated" term. As explained, this is due to the choice of the loss function, which inherently contains an expectation term. It would be interesting to consider a loss function that captures the "one-shot" prediction of the performance and to develop a disintegrated bound for such a loss function.
- One of the contributions of this work has been to establish "rate-distortion theoretic" bounds by using a lossy compressibility framework. For example, the established bound in Theorem 4, contains an infimum over all "compressed algorithms"  $P_{\hat{W}_e|S}$  that satisfy some distortion criterion. Note that this implies that this bound holds for "any" choice of eligible  $P_{\hat{W}_e|S}$ , and thus, taking the infimum to obtain a "valid bound" is not necessarily required. Thus, any simple technique that adds noise or applies parameter quantization can be considered. However, besides these general approaches, an interesting direction is how to find an "optimal"  $P_{\hat{W}_e|S}$ . Perhaps, the approach taken in [THP23], which uses a combination of the MINE estimator  $[\mathrm{BBR}^{+}18]$  and the "SFRL" [LEG18] could be adapted to our setup.
- Another potentially valuable direction would be to compute the bounds analytically in simple setups, such as two-layer neural networks with Gaussian data, and compare the results with the existing results such as [VPV18]. Also, it is instructive to study numerically the geometry and MDL of the latent variables for different network architectures, e.g., FCN versus CNN. This may lead to a new understanding of how and why some architectures produce "better" representations.
- Inspired by our results, we have proposed some simple "data-dependent" priors. While the proposed priors show improvements over the classical priors, they are limited to priors that can be factorized as products of some Gaussian distributions. Thus, they only "partially" capture the joint compressibility of the latent variables (and hence the encoder structure). We imagine that more "appropriate" priors can be proposed that capture better the structure of the encoder.
- In this paper, we partially discuss the intuition behind the importance of the "structure" of the encoder. However, the explicit effect of the "structure" on the geometry of latent variables and predictions needs to be investigated, as partially shown in [BM06].
- Finally, we emphasize that in representation learning, one is interested in extracting good representations that are suitable in terms of generalization error for multiple learning tasks, simultaneously. It would be interesting to extend our results to such more realistic settings.
D Details of the experiments
In this section, we present the details of the experiments presented in Section 4.
D.1 VIB objective function
The traditional information bottleneck (IB) approach [TPB00, SST10] for training representation learning models, and particularly its variational implementation (VIB) by [AFDM17], considers a fixed data-independent prior  $\mathbf{Q} = Q^{\otimes 2n}$ . Then, for some Lagrange multiplier  $\beta > 0$ , the VIB approach minimizes
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \Bigl \{\beta D _ {K L} \Big [ P _ {U | X, W _ {e}} (U _ {i} | x _ {i}, W _ {e}) \| Q \B ] - \mathbb {E} _ {U _ {i} \sim P _ {U | X, W _ {e}} (U _ {i} | x _ {i}, W _ {e})} \Big [ \log P _ {\hat {Y} | U, W _ {d}} (y _ {i} | U _ {i}, W _ {d}) \Big ] \Bigr \},
$$
using the reparametrization trick of [KW14]. As can be noticed, the first term, which acts as a regularizer, only takes into account the training dataset part of  $D_{KL}\Big(P_{U|X,W_e}^{\otimes 2n}(\mathbf{U},\mathbf{U}'|\mathbf{X},\mathbf{X}',W_e)\big\| \mathbf{Q}\Big)$ , as the test set is not available. The second term attempts to maximize the relevance of  $U$  for prediction. Intuitively, it seeks to find the best decoder among the possible choices, i.e., the one that minimizes the empirical risk.
A popular choice for  $Q$  is the multi-dimensional standard Gaussian distribution  $Q = \mathcal{N}(\mathbf{0}_m,\mathrm{I}_m)$ , where  $\mathrm{I}_m$  is the identity matrix,  $m$  is the dimension of the latent variable  $U$ , and  $\mathbf{0}_m\in \mathbb{R}^m$  is the all zero vector. In the original implementation of [AFDM17], for each sample  $x$ , the encoder generates the mean  $\mu_{x} = (\mu_{x,1},\dots ,\mu_{x,m})\in \mathbb{R}^{m}$  and variance  $\sigma_x^2 = (\sigma_{x,1}^2,\ldots ,\sigma_{x,m}^2)\in \mathbb{R}^m$  of the latent variable  $U$ . Then, we let  $P_{U|X,W_e}(U|x,W_e) = \mathcal{N}(\mu_x,\mathrm{diag}(\sigma_x^2))$ , where  $\mathrm{diag}(\sigma_x^2)\in \mathbb{R}^{m\times m}$  denotes a diagonal matrix whose diagonal elements are denoted by the vector  $\sigma_x^2$ . This means that the latent variable for the input  $x$  is generated according to  $U\sim \mathcal{N}(\mu_x,\mathrm{diag}(\sigma_x^2))$ . Hence, the objective function to minimize becomes
$$
\frac {1}{b} \sum_ {i = 1} ^ {b} \left\{\beta D _ {K L} \left[ \mathcal {N} \left(\mu_ {x _ {i}}, \operatorname {d i a g} \left(\sigma_ {x _ {i}} ^ {2}\right)\right) \| \mathcal {N} (\mathbf {0} _ {m}, \mathrm {I} _ {m}) \right] - \mathbb {E} _ {U _ {i} \sim P _ {U | X, W _ {e}} \left(U _ {i} \mid x _ {i}, W _ {e}\right)} \left[ \log P _ {\hat {Y} | U, W _ {d}} \left(y _ {i} \mid U _ {i}, W _ {d}\right) \right] \right\}, \tag {13}
$$
where  $b$  is the size of a mini-batch of training samples  $z_{1},\ldots ,z_{b},z_{i} = (x_{i},y_{i})$ . Moreover, (13) is repeated iteratively over multiple mini-batches until the convergence of the representation learning model.
D.2 Lossless CDVIB objective function
We describe here another learning approach which, unlike the VIB, uses a data-dependent prior. This prior, coined Category-Dependent VIB (CDVIB) is again factorized as a product of  $2n$  scalar Gaussian priors, i.e.,  $\mathbf{Q} = \prod_{i\in [2n]}Q_i$ . Each of these scalar priors  $Q_{i}$  is chosen among one of the  $K\times M$  Gaussian priors (centers)  $-M$  priors per each label. More precisely, for each label  $k\in [K]$ , we consider  $M\in \mathbb{N}$  priors
$$
Q _ {k, r} ^ {(t)} = \mathcal {N} \bigg (\bar {\mu} _ {k, r} ^ {(t)}, \mathrm {d i a g} \bigg (\bar {\sigma} _ {k, r} ^ {(t)} ^ {2} \bigg) \bigg), \quad r \in [ M ], k \in [ K ],
$$
defined over  $\mathbb{R}^m$ , where the superscript  $t \in \mathbb{N}$  represents the optimization iteration.
Unlike in the VIB approach, the mean and variance of the scalar priors are updated during the training process. Firstly, the vectors  $\bar{\mu}_{k,r}^{(t)}$  and  $\bar{\sigma}_{k,r}^{(t)^2}$  are initialized respectively as  $m$ -dimensional vectors of zeros and ones. Next, these initialized vectors are updated at each iteration using a procedure defined below.