Title: Minimum Description Length and Generalization Guarantees for Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Information Bottleneck

Representation Learning

Generalization Error

Minimum Description Length
Abstract: A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has spurred much interest, most existing such approaches are heuristic; and very little is known about theoretical generalization guarantees. For example, the information bottleneck method seeks a good generalization by finding a minimal description of the input that is maximally informative about the label variable, where minimality and informativeness are both measured by Shannon’s mutual information. 

In this paper, we establish a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm in terms of the ``Minimum Description Length'' (MDL) of the labels or the latent variables (representations). Rather than the mutual information between the encoder’s input and the representation, which is often believed to reflect the algorithm’s generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the "multi-letter" relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior. In particular, these new bounds reflect the structure of the encoder and are not vacuous for deterministic algorithms. Our compressibility approach, which is information-theoretic in nature, builds upon that of Blum-Langford for PAC-MDL bounds and introduces two essential ingredients: block-coding and lossy-compression. The latter allows our approach to subsume the so-called geometrical compressibility as a special case. To the best knowledge of the authors, the established generalization bounds are the first of their kind for Information Bottleneck type encoders and representation learning. Finally, we partly exploit the theoretical results by introducing a new data-dependent prior. Numerical simulations illustrate the advantages of well-chosen such priors over classical priors used in IB.

Minimum Description Length and Generalization Guarantees for Representation Learning

Abstract

A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has spurred much interest, most existing such approaches are heuristic; and very little is known about theoretical generalization guarantees. For example, the information bottleneck method seeks a good generalization by finding a minimal description of the input that is maximally informative about the label variable, where minimality and informativeness are both measured by Shannon's mutual information.

In this paper, we establish a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm in terms of the 'Minimum Description Length' (MDL) of the labels or the latent variables (representations). Rather than the mutual information between the encoder's input and the representation, which is often believed to reflect the algorithm's generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the 'multi-letter' relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior. In particular, these new bounds reflect the structure of the encoder and are not vacuous for deterministic algorithms. Our compressibility approach, which is information-theoretic in nature, builds upon that of Blum-Langford for PAC-MDL bounds and introduces two essential ingredients: block-coding and lossy-compression. The latter allows our approach to subsume the so-called geometrical compressibility as a special case. To the best knowledge of the authors, the established generalization bounds are the first of their kind for Information Bottleneck type encoders and representation learning. Finally, we partly exploit the theoretical results by introducing a new data-dependent prior. Numerical simulations illustrate the advantages of well-chosen such priors over classical priors used in IB.

1 Introduction

A key performance indicator of stochastic learning algorithms is their capability to generalize, i.e., perform equally well on training and unseen data. However, designing learning algorithms with good generalization guarantees remains a major challenge. A popular approach involves learning an encoder part and a decoder part. The encoder aims at generating a suitable representation of the input (referred to as 'latent variable') by extracting relevant features from the input data. The decoder aims at optimizing the performance of the learning task for the given training dataset, known as empirical risk minimization (ERM), based only on the learned latent variables. This approach is grounded on the idea that performing ERM on the latent variable (instead of the input itself) prevents overfitting.

Information Bottleneck. Several approaches have attempted to formalize the concept of a 'good representation' [SST10, AFDM17, VDOV ` 17, DKSV20]. Perhaps, the most prominent is the information bottleneck (IB) method which was first introduced in [TPB00] and then extended in several directions [SST10, AFDM17, AZ19, KTW19, Fis20, RGTS20, KASK22]. The IB approach was deemed useful to analyze deep neural networks [SZT17, ZEAS20, GP20, Gei21] and guide their training process. For

instance, in supervised learning tasks the IB seeks representations that capture 'minimum' information about the input data (shoots for good generalization power) while providing 'maximum' information about the label (shoots for small empirical risk), where the amounts of captured and provided information are measured using Shannon mutual information. More precisely, denoting by Y the label variable and by X the input data, the IB latent variable U is chosen so as to maximize the mutual information I p U ; Y q while minimizing the mutual information I p U ; X q . Equivalently, this can be formulated as maximizing the Lagrange cost

where β ě 0 designates a Lagrange multiplier. Let p X m , U m q denote a vector, or block, of m independent and identically distributed (i.i.d.) realizations p X i , U i q , i ' 1 , . . . , m , of discrete variables p X,U q ' P X,U . Fix a distribution P ˆ U such P X, ˆ U ' P X,U . Essentially by means of the rate-distortion theoretic covering lemma [CT06], it is easy to see that one can get a suitable description ˆ U of X using only I p U ; X q bits per symbol. Precisely, generating a codebook of roughly l m « 2 mI p U ; X q vectors ˆ u m r j s P U m , j ' 1 , . . . , m , all drawn i.i.d. according to P U , the covering lemma states that for large m there exists at least one index j for which the empirical distribution of p X m , ˆ u m r j sq is arbitrary close to P X,U in some suitable sense. That is, the empirical distributions of p X m , U m q and p X m , ˆ u m r j sq are close. Hence, an 'equivalent' version ˆ U m of U m can be described with roughly mI p U ; X q bits. Intuitively, this makes a connection between the mutual information I p U ; X q and the concept of Minimal Description Length (MDL) [Ris78, GMP05] when one considers MDL of the latent variables, not that of model parameters [VPV18, ZEAS20]. Moreover, it is now relatively well known that there exists a connection between the generalization error of a learning model and the MDL of the parameters of that model, see, e.g., [BEHW87, BO18, GR19]. A notable work [BL03] has considered MDL of the predicted labels of a super-sample of training and test data.

Critics to IB. The aforementioned connections perhaps formed a belief that I p U ; X q is closely related to the generalization performance of representation learning algorithms. This belief, however, is controversial and conflicting evidence has been reported [GK19]. For instance, using the term I p U ; X q as a regularizer has been criticized for four main reasons [KTVK18, RG19, AG19, DKSV20]: i. The few existing upper bounds on the generalization error which involve (among other terms) the mutual information I p U ; X q reported in [VPV18] and the very recent and concurrent work [KDJH23] are not convincing. For example, the bound of [VPV18] holds only when the alphabets of the input and latent spaces are finite; and, in that case, it states that for any δ ą 0 , with probability 1 ´ δ it holds that: gen p S, W q ď O ´ log p n q n ¯ a I p U ; X q ` C δ , where gen p S, W q is the generalization error of model W , C δ : ' O ` | U | L ? n ˘ , | U | is the size of the latent space and n is the size of the training dataset. For reasonable setups, however, the term C δ dominates and their bound becomes vacuous [RG19, LLS ` 23]. The bound reported in [KDJH23] suffers similar shortcomings - For further details on this, see Appendix B.2. ii. Experimental evidence shows dependence of the generalization error on the so-called geometrical compression rather than on I p U ; X q [GK19]. Geometrical compression occurs when the latent variables are concentrated around a limited number of clusters. See [GK19, Fig. 2] for a visual representation. iii. I p U ; X q is invariant to bijection; and, as such, it does not favor learning algorithms/representations with simple decision boundaries and does not reflect the 'structure' or 'simplicity' of the encoder/decoder. Please refer to [AG19, Section 4.3] for a detailed discussion and examples. iv. Finally, for deterministic algorithms I p U ; X q can take large or even infinite values, especially for continuous variables or high-dimensional data, hence limiting the usefulness of IB [KTVK18, AG19].

MDL/Compressibility. Several works have studied MDL/compressibility to establish generalization bounds. In this context, key is the 'Occam's razor' principle [LW86, BEHW87, BL03] which, e.g., for binary classification tasks, states that if the labels of the training data S ' t Z 1 , . . . , Z n u , Z i ' p X i , Y i q predicted by a model W can be described by k ! n number of bits, then the learned model W has a good generalization performance. There exist three closely related lines of work that use different means to describe the dataset labels: i. The first describes the predicted labels via the hypothesis (parameter models). It includes the works of [BEHW87, AGNZ18, SAM ` 20, HJTW21, BSE ` 21, SGRS22]. Recently, it was shown in [SGRS22, SZ23] that information-theoretic bounds [RZ16, XR17, SZ20], PAC-Bayes bounds [McA98] and intrinsic dimension-based approaches [S ¸ SDE20] also fall into this category. The proof uses so-called 'fixed-size' [SGRS22] and 'variable-size' [SZ23] compressibility frameworks introduced therein. ii. The second formulates the minimal description of labels as follows: if a learning algorithm can predict all labels of the training data S by using only samples from a small subset of S , then the learning algorithm is guaranteed to generalize well. This second approach was initiated by [LW86] and followed by others including [HK19, HKS19, BHMZ20, HK21, HKSW20, CK22]. iii. The third, initiated by [BL03], deals directly with the compression of the predicted labels. This approach is shown to be related to the previous two lines, and also to PAC-Bayes and VC-dimension-based results. Moreover, it is the closest to the problem

training

Y

X

Ŷ

W

S

Y

U

X

Ŷ

W

=(

W ,W

)

e

d

S

(a) Single-step prediction model.

training

(b) Two-step prediction model.

of establishing bounds on the generalization error of representation learning algorithms in terms of the compressibility of the latent variables. This approach is detailed in Appendix B.1.

In this work we aim at understanding the theory of representation learning through a rigorous investigation of the connection between the MDL (or compressibility) of the latent variable and the generalization performance of representation learning algorithms. In doing so, we first study the single-step prediction model of Fig. 1a; and, then, we leverage the developed tools to study the two-step (encoder-decoder) prediction model of Fig. 1b. We also examine the claimed relationship between MDL and I p U ; X q .

Contributions. Specifically, the main contributions of this work are as follows.

For the prediction model of Fig. 1a, inspired by [BL03] we establish an information-theoretic framework that allows us to measure the compressibility (MDL) of the predicted labels in terms of a new object which is the KL-divergence of a vector of labels and any arbitrary symmetric prior. For a proper choice of prior, this new measure reduces to the sample-wise mutual information and hence inherits the associated properties. However, unlike the sample-wise mutual information, it can also reflect the structure/simplicity of the learning algorithm. Furthermore, by extending the framework to 'lossy compressibility', this new measure does not become vacuous when one considers continuous variables instead of discrete labels. Moreover, it subsumes geometrical compressibility as a special case.

We also establish both in-expectation and tail generalization bounds in terms of this compressibility measure of the predicted labels. In a simple case, the in-expectation bound can be cast into the form

The tail bound involves a similar expression. This generalization bound easily recovers the VC-dimension bound as a special case; and, hence, it also shows how the introduced compressibility measure depends on the complexity of the hypothesis class.

Our results make a connection between the compressibility framework of [BL03] and the functional conditional mutual information (f-CMI) of [HRVSG21, HD22], which itself is an extension of the CMIframework developed by [SZ20]. In particular, this connection shows how the f-CMI framework can be leveraged to study the compressibility of the predicted labels.

For the encoder-decoder prediction model of Fig. 1b, the framework is extended non-trivially to the case in which instead of the compressibility of the predicted labels it is the compressibility of the latent variable which is considered. The results for this prediction model, which are our main results in this paper as given in Section 3, are generalization bounds for the representation learning algorithms in terms of the complexity of the latent variable. The established in-expectation and tail bounds hold for any decoder; and, for the K -class classification task for example, take the form

These bounds appear to be the first of their kind for the studied representation learning setup. In part, their utility lies in that: i) they reflect the structure of the encoder class, ii) they do not become vacuous for deterministic encoders with continuous latent space and iii) they can explain the geometrical compressibility phenomenon. Thus, our framework reveals that the 'joint' MDL of the latent space is more related to the encoder structure, rather than the mutual information I p U ; X q .

Finally, our results suggest that data-dependent priors can be used in place of the data-independent prior of the popular variational IB (VIB) method. We conduct experiments that illustrate the advantage brought up by proper choices of data-dependent priors over VIB.

Our results also open up several future directions as discussed in Appendix C.4.

Notation. Random variables, their realizations, and their alphabets are denoted respectively by upper-case letters, lower-case letters, and Calligraphy fonts, e.g., X , x , and X . Their distributions and expectations are denoted by P X and E r X s . For ease of presentation, when X is a discrete set, P X is a probability mass function . Otherwise, P X is a probability density function . Acollection of n random variables p X 1 , . . . , X n q is denoted by X n or X . We use the notation t x i u i m ' 1 to denote a sequence of m real or natural numbers. The set t 1 , . . . , K u , K P N , is denoted by r K s . Finally, R ` denotes the set of non-negative real numbers. Our results are expressed in terms of information-theoretic functions. For two distributions P and Q , the Kullback-Leibler (KL) divergence is defined as D KL p P } Q q : ' E P r log p P { Q qs if P ! Q , and 8 otherwise. The mutual information between two random variables X,Y ' P X,Y with marginals P X and P Y is defined as I p X ; Y q : ' D KL p P X,Y } P X P Y q . The reader is referred to [CT06, CK11] for further information.

Our results in this paper will be expressed in terms of symmetric conditional priors. The following definition formalizes three types of symmetry.

Definition 1 (Symmetric Priors) . For U, V ' P U,V , let p U 2 n , V 2 n q be vectors composed of 2 n i.i.d. instances p U i , V i q ' P U,V , i P r 2 n s . For any permutation π : r 2 n s Ñ r 2 n s , denote U 2 n π : ' p U π p 1 q , . . . , U π p 2 n q q and V 2 n π : ' p V π p 1 q , . . . , V π p 2 n q q .

Type-I symmetry: Define type-I permutations as the set of permutations π : r 2 n s Ñ r 2 n s having the property that for any i P r n s , the sets t π p i q , π p i ` n qu and t i, i ` n u are equal. The conditional prior Q p U 2 n | V 2 n q has type-I symmetry if Q p U 2 n π | V 2 n π q is invariant to arbitrary type-I permutations. This definition first appeared in [Aud04] (called 'almost exchangeable prior' therein) and was used for the CMI framework in [GSZ21].

Type-II symmetry: The conditional prior Q p U 2 n | V 2 n q has type-II symmetry if Q p U 2 n π | V 2 n π q is invariant to any arbitrary permutations π : r 2 n s Ñ r 2 n s .

Type-III symmetry: For a random variable Z ' P Z | U 2 n ,V 2 n , the conditional prior Q p U 2 n | V 2 n , Z q has type-III symmetry if Q p U 2 n π | V 2 n , Z q is invariant to any permutation π : r 2 n s Ñ r 2 n s having the property that V i ' V π p i q for every i P r 2 n s .

Throughout, if the underlying p U 2 n , V 2 n q is clear from the context, for ease of the notation the corresponding sets of Type-I and Type-II priors will be denoted simply as Q i and Q ii respectively.

Problem setup. Unless indicated otherwise, we consider the K -class classification setup. Let Z ' p X,Y q be some input data taking value over the input space Z ' X ˆ Y according to an unknown distribution µ . We call X P X the features of the data, and Y P Y its label , where Y ' r K s . We assume a training dataset S ' t Z 1 , . . . , Z n u ' µ b n ' : P S , composed of n i.i.d. samples Z i ' p X i , Y i q of the input data, is available. We denote the features and labels of S by X : ' X n ' µ b n X and Y : ' Y n ' µ b n Y , respectively. We often use also a ghost or test dataset S 1 ' t Z 1 1 , . . . , Z 1 n u ' µ b n ' : P S 1 , where Z 1 i ' p X 1 i , Y 1 i q . Similarly, we denote the features and labels of S 1 by X 1 : ' X 1 n ' µ b n X and Y 1 : ' Y 1 n ' µ b n Y , respectively.

2 Generalization bounds in terms of predicted label complexity

In this section, we formulate a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm. Our proposed framework can be seen as a suitable generalization of the framework of Blum and Langford [BL03], in which the generalization encompasses lossy compression of the predicted labels and which exploits block-coding . However, unlike in the original framework based on a (compression) game between two agents Alice and Bob (see Appendix B.1 for details), this work adopts a rate-distortion theoretic perspective.

For ease of exposition, the results are presented for classification problems and the 0-1 loss function, but they can be extended trivially to any continuous Y and bounded loss function. Consider the setup in Fig. 1a. Let A : Z n Ñ W be a possibly stochastic learning algorithm. That is, for a given S ' p Z 1 , . . . , Z n q P Z n the algorithm picks a hypothesis or model W ' A p S q P W . Also, let the induced joint distribution over S ˆ W be denoted as P S,W ; and the induced conditional distribution over W given S be denoted as P W | S . For every input data z ' p x, y q P Z , every choice of hypothesis w P W induces a conditional distribution P ˆ Y | X,W p ˆ Y | x, w q on ˆ Y ' Y . For convenience, we use the following handy notations:

The quality of the prediction is measured by the loss function ℓ : Z ˆ W Ñr 0 , 1 s given by

ℓ p z, w q : ' E ˆ Y ' P ˆ Y | X,W p ˆ Y | x,w q r ✶ t y ‰ ˆ Y u s , (1) where ✶ stands for the indicator function. The associated empirical and population risks for this loss are defined as ˆ L p s, w q : ' 1 n ř i Pr n s ℓ p z i , w q and L p w q : ' E Z ' µ r ℓ p Z, w qs , respectively. Finally, the generalization error is defined as gen p s, w q : ' L p w q ´ ˆ L p s, w q .

2.1 Compressibility framework

Now, we introduce briefly the joint compression of a block of the predicted labels. Further details can be found in Appendix C.1. Consider m i.i.d. pairs of train and test datasets S j : ' p Z j, 1 , . . . , Z j,n q and S 1 j : ' p Z 1 j, 1 , . . . , Z 1 j,n q , where Z j,i ' p X j,i , Y j,i q and Z 1 j,i ' p X 1 j,i , Y 1 j,i q . Let S m ' p S 1 , . . . , S m q , S 1 m ' p S 1 1 , . . . , S 1 m q and W m : ' p W 1 , . . . , W m q , where W j ' P W j | S j . Denote the predicted labels using model W j for inputs X j,i and X 1 j,i as ˆ Y j,i and ˆ Y 1 j,i , for j P r m s and i P r n s . Moreover, let Z 2 n j P Z 2 n denote a rearrangement of the elements of p S j , S 1 j q in a way that makes it indistinguishable whether a given sample z is from S j or S 1 j . In the following two subsections, we investigate two such rearrangements of a given p S, S 1 q as Z 2 n . Then, for the collection p S m , S 1 m q , each pair p S j , S 1 j q for j P r m s is rearranged independently and the matrix of all rearrangements is denoted by Z 2 mn . Finally, given some Z 2 mn , let denote the rearranged versions of p Y mn , Y 1 mn q and p ˆ Y mn , ˆ Y 1 mn q respectively by Y 2 mn and ˆ Y 2 mn .

Our approach is based on studying the compressibility of the rearranged model-predicted labels ˆ Y 2 mn , from an information-theoretic point of view. The rationale is as follows: since the (rearranged) predicted-labels vector ˆ Y 2 mn agrees mostly with the true labels Y 2 mn on the dataset S m , then in accordance with 'Occam's Razor' theorem [LW86, BEHW87] the model W is guaranteed to generalize well if ˆ Y 2 mn can be described using only a few bits (or nats). We leverage source coding arguments to measure the compressibility of ˆ Y 2 mn . In our block-coding rate-distortion theoretic framework, a compression rate R P R ` is said to be achievable if there exists a compression codebook of size « e mR (fixed a priori) which covers the space spanned by the model-predicted labels ˆ Y 2 mn with high probability. That is, if R is achievable then ˆ Y 2 mn can be described using R P R ` nats. Formally, R is achievable if there exists a sequence of label books t ˆ Y m u m P N , with ˆ Y m : ' t ˆ y r r s , r P r l m su Ď Y 2 mn , l m P N , ˆ y r r s ' p ˆ y 1 r r s , . . . , ˆ y m r r sq and ˆ y j r r s ' p ˆ y j, 1 r r s , . . . , ˆ y j, 2 n r r sq P Y 2 n , such that: i. l m ď e mR and ii. there exist a sequence t δ m u m P N for which lim m Ñ8 δ m ' 0 such that with probability at least p 1 ´ δ m q over the choices of S m , S 1 m one can find at least one index r P r l m s whose associated ˆ y r r s equals ˆ Y 2 mn .

As explained further in Appendix C.1.3, this fixed-size compressibility framework, which is suitable to upper bound the expectation of the generalization error, can be extended to variable-size compressibility in a way that is similar to [SZ23] in order to upper bound the generalization error with high probability. We notice that, in essence, the core idea of the compressibility framework of Blum-Langford, which we recall in Appendix B.1, can be seen as a one-shot counterpart of our approach. As discussed in [SGRS22, SZ23], in the one-shot case one deals with the 'worst case' scenario; and this results in bounds that involve combinatorial terms [BL03]. In contrast, our information-theoretic framework allows us to bound the compression rate R in terms of a simpler new quantity: the relative entropy of the joint conditional P p ˆ Y n , ˆ Y 1 n | Y n , Y 1 n q and a (symmetric) conditional prior Q over ˆ Y 2 n given Y 2 n .

Our approach to establishing a bound on the compressibility of ˆ Y 2 mn in terms of information-theoretic measures starting from the combinatorial approach of [BL03] and by using block-coding essentially consists in a suitable combination of the following key proof steps: (i) introduce and use the symmetries of Definition 1 to rearrange p S m , S 1 m q , (ii) generate the codewords using symmetric priors and (iii) analyze the minimum codebook size needed to 'cover' reliably ˆ Y 2 mn , essentially by use of the covering lemma . As shown in [SGRS22, SZ23] and also our proofs, the last two ingredients can be merged via the Donsker-Varadhan's variational representation lemma. The application of this lemma is reminiscent of information-theoretic works on the generalization error such as [RZ16, XR17, SZ20].

2.2 Generalization bounds using type-I symmetric priors

One way to rearrange indistinguishably p S, S 1 q as Z 2 n is as follows: Let J ' p J 1 , . . . , J n q be a vector of n i.i.d. Bernoulli p 1 2 q random variables J i P t i, i ` n u , i P r n s . For every i P r n s , let the random variable J c i P t i, i ` n u be defined such that J c i ' i ` n if J i ' i and J c i ' i if J i ' i ` n . Also, let the vector J c ' p J c 1 , . . . , J c n q . For i P r n s , we let the random variables Z J i and Z J c i de defined as Z J i ' Z i and

Z J c i ' Z 1 i . Observe that the vector Z 2 n ' p Z 1 , . . . , Z 2 n q is a J -dependent random re-arrangement of the samples of the training and ghost datasets S and S 1 . Without knowledge of J every element of the vector Z 2 n has equal likelihood to be from S or S 1 . A similar construction was used in the context of the analysis of Rademacher complexity and the CMI of [SZ20]. We use Type-I symmetric priors, block coding and information-theoretic covering arguments in order to analyse the space spanned by the random vector Z 2 n . This yields the generalization bound stated in the next theorem whose proof is deferred to Appendix E.2.

Theorem 1. i. Let Q i be the set of type-I symmetric conditional priors on p ˆ Y , ˆ Y 1 q given p Y , Y 1 q . Then, E S,W r gen p S, W qs ď a 2 R { n , with

where Y , Y 1 ' µ b 2 n Y and X 1 , X , W ' P X 1 | Y 1 P X ,W | Y . Also the mutual information is calculated with respect to the joint distribution P J , ˆ Y 2 n ,Y 2 n ' Bern p 1 { 2 q b n µ b 2 n Y P ˆ Y 2 n J , ˆ Y 2 n J c ˇ ˇ Y 2 n J Y 2 n J c and the latter term ' ı

For any δ P R ` and any conditional type-I symmetric prior Q on p ˆ Y , ˆ Y 1 q given p X , Y , X 1 , Y 1 q , with probability at least p 1 ´ δ q over choices of S, S 1 , W ' P S 1 P S,W , it holds that 1

A couple of remarks are in order. The part i of the result of Theorem 1 can be understood as being some form of the f-CMI of [HRVSG21] in which the function f represents the predicted labels - in fact our result is slightly stronger comparatively, since instead of conditioning on Z 2 n as in the f-CMI of [HRVSG21] one here conditions only on Y 2 n . Incidentally, the result also unveils an appealing connection between an extension of the compressibility approach of [BL03] (in this extension one needs to consider Type-I symmettric priors) and CMI and f-CMI [SZ20, HRVSG21, HD22]. However, for type-II and type-III symmetries, used in the coming sections, our framework goes beyond the CMI framework. The type-I priors have been also used in [GSZ21] to establish (fast-rate) tail bounds for the CMI framework. Due to the difference in the considered setups, their results are not directly comparable with ours. We also note that the above result (and some of the results in the rest of the paper) is kept for clarity and can be trivially improved, by i) moving E Y , Y 1 outside the square root, ii) by rewriting E r gen p s, w qs ' 1 n ř i Pr n s E r gen p z i , w qs , and applying the bound for each E r gen p z i , w qs , similar to [BZV20, HRVSG21, RGBTS21], iii) by extending the results in accordance with e-CMI framework [SZ20, HD22], and iv) establishing tail bound on gen p S, W q , by noting that with probability at least p 1 ´ δ q , ˆ L p S 1 , W q ě L p W q´ a log p 1 { δ q{p 2 n q (similar to Theorem 5).

Next, observe that for any Q : ' E X , X 1 | Y , Y 1 ' Q 1 p ˆ Y , ˆ Y 1 | X , X 1 , Y , Y 1 q ‰ , where Q 1 p ˆ Y , ˆ Y 1 | X , X 1 , Y , Y 1 q is an arbitrary type-I symmetric prior, and by using the Jensen's inequality, we have

Relation to Mutual Information. A particular choice of the conditional prior Q 1 in the RHS of (3) is Q b 2 n for some prior Q defined over Y . For this special choice, the RHS of (3) is given by

where ˆ µ X | S is the empirical distribution of X in the dataset S . Moreover, by choosing Q as the marginal distribution of ˆ Y under P ˆ Y | X,W P S,W , it is easy to see that the first term of the sum of the RHS of (4) coincides with n ˆ I p X ; ˆ Y q for that choice, where ˆ I p¨ ; ¨q stands for the 'empirical mutual information' as computed from the available samples. Thus, the contribution of this term to the bound on generalization error does not necessarily vanish as n Ñ8 (unless the empirical mutual information itself is small). In fact, as already observed in [GK19], there exist models which generalize well but have non-small mutualinformation ˆ I p X ; ˆ Y q . This instantiates that mutual-information type bounds may fall short of explaining true generalization capability, an observation which was already made in [GK19]. The bound on the generalization error of our Theorem 1, which is provably tighter (see (3)), then possibly remedies this issue.

In what follows we further investigate the relationship of the result of our Theorem 1, which is based on KL-divergence, to VC-dimension and mutual information type bounds on the generalization error.

Relation to VC-dimension Suppose that the VC-dimension of the hypothesis class is d . Then, using the Sauer-Shelah lemma [Sau72, She72] we get that given any X and X 1 one can have at most p 2 en { d q d distinct labels. By letting the conditional prior Q 1 be a uniform distribution over all such possible predictions, it is easily seen that the KL divergence term of our Theorem 1 is upper bounded by d log p 2 en { d q . This means that the result of our Theorem 1 recovers and possibly improves over the VC-dimension bound.

Structure and 'simplicity' of the learning algorithm. Let us consider a simple example. Suppose that X ' r 0 , 1 s Ă R and let τ P p 0 , 1 q be a parameter. Then, consider the set of deterministic classifiers w as follows: P ˆ Y | X,W p ˆ y | x, w q ' 1 if ( ˆ y ' 1 and x ě τ ) or ( ˆ y ' 0 and x ă τ ); and P ˆ Y | X,W p ˆ y | x, w q ' 0 otherwise. It is well known that the VC-dimension of this learning class is d ' 1 . Then, by recalling the aforementioned relation to the VC dimension, we obtain that our Theorem 1 yields a bound on the generalization error which is O p a log p n q{ n q . In particular, it is easy to see that this bound vanishes as n Ñ8 . This is in sharp contrast with the mutual information term ˆ I p X ; ˆ Y q which does not necessarily vanish for large n . For example, if the pair p X,Y q is such that Y ' 1 iff X ě τ ˚ for some τ ˚ P p 0 , 1 q , then the ratio between RHS of (4) and n converges (for large n ) to a value which is at least H p Y q (because ˆ I p X ; ˆ Y q Ñ I p X ; Y q as n Ñ8 and, in this example, I p X ; Y q ' H p Y q ).

The above example indicates that using mutual information as a regularizer for learning algorithms (as is the case in IB) may fail to find models that generalize well and have a comparatively simple structure. In fact, using a different approach, it was already observed in [AG19] and [DKSV20] that using the mutual information term as a complexity measure does not reflect the 'structure' of the learning algorithm and considering it as a regularizer might not favor 'simple' learning algorithms. This suggests that mutual information regularizer in the IB approach may be replaced by the KL divergence term of the RHS of (3) (when the latent variables are considered instead of predictions) which does reflect the complexity of the encoder's structure. As investigated and shown in Section 3, in addition to favoring encoders of simpler structure, our approach provides theoretical guarantees of the generalization error.

2.2.1 Lossy compressiblity

The above approach and results can be extended to any bounded loss function and continuous variables Y and ˆ Y (see the next section where continuous latent variables are studied). In the regime of continuous variables, a standard result of rate-distortion theory stipulates that any lossless encoding of ˆ Y n and ˆ Y 1 n may require an infinite number of bits, making the generalization bounds vacuous. This is precisely why Shannon's mutual information fails as a regularizer in deterministic learning algorithms with continuous alphabet variables. On the other hand, our approach can be easily extended to include the lossy compression of the labels (see Appendix C.1.2; in the same spirit as done in [SGRS22] for the hypothesis compression. The following result states a lossy in-expectation bound and a lossy tail bound is reported in Appendix A.

Theorem 2. Let Q i be the set of type-I symmetric conditional priors on p ˆ Y , ˆ Y 1 q given p Y , Y 1 q . Then, for any ϵ P R , E S,W r gen p S, W qs is upper bounded by

where Y , Y 1 ' µ b 2 n Y , X 1 , X , ˆ W ' P X 1 | Y 1 P X , ˆ W | Y , and the first infimum is over all P ˆ W | S satisfying E P S,W P ˆ W | S ' gen p S, W q ´ gen p S, ˆ W q ‰ ď ϵ .

A proof of this theorem follows by an easy combination of the distortion criterion with the bound on E S, ˆ W ' gen p S, ˆ W q ‰ obtained by application of Theorem 1 to the compressed model ˆ W (not W ). This simple trick, which is related conceptually to lossy source coding , prevents the KL-divergence term from taking very large (infinite) values for continuous alphabet variables. Moreover, the lossy compressibility also offers an interpretation of the geometrical compressibility concept that was observed to be related to the generalization performance in [GK19].

2.3 Generalization bounds using type-II symmetric priors

In this section, we consider another way to rearrange indistinguishably p S, S 1 q as Z 2 n , which is inspired by [BL03]. Let T ' t T 1 , . . . , T n u , be a random set obtained by picking uniformly n indices from t 1 , . . . , 2 n u , without replacement. Note that in contrast to J which had i.i.d. components, here the components of T

are dependent. let T c ' t 1 , . . . , 2 n uz T be the complement of T , having the elements T c ' t T c 1 , . . . , T c n u . Now, for each i P r n s , let p Z T i , Z c T i q ' p Z i , Z 1 i q . We use type-II symmetric prior to cover such rearranged vectors. To state the result, first we need to define the function h D p x ; x 1 q : r 0 , 1 s ˆ r 0 , 1 s Ñ r 0 , 2 s as follows:

where h b p x q : ' ´ x log 2 p x q ´ p 1 ´ x q log 2 p 1 ´ x q . Note that h D p x, x 1 q{ 2 is equal to the Jensen-Shannon divergence between two binary Bernoulli distributions with parameters x and x 1 . The reader is referred to Appendix A for the relation of this function with the combinatorial term appeared in [BL03]. The function h D p¨ , ¨q has the following interesting properties, proved in Appendix E.3.

Lemma 1. @ p x, x 1 q P r 0 , 1 sˆr 0 , 1 s , we have: (i) h D p x, x 1 q ě p x ´ x 1 q 2 , (ii) h D p x, 0 q ě x , (iii) h D p x, x 1 q is increasing with respect to x in the range r x 1 , 1 s , and (iv) h D p x, x 1 q is convex with respect to both inputs.

Now, we state the main result of this section.

Theorem 3. Let Q ii be the set of type-II symmetric priors on p ˆ Y , ˆ Y 1 q given p Y , Y 1 q . Then, for n ě 10 ,

where Y , Y 1 ' µ b 2 n Y and X 1 , X , W ' P X 1 | Y 1 P X ,W | Y . Also the mutual information is calculated with respect to the joint distribution P T , ˆ Y 2 n ,Y 2 n ' P T µ b 2 n Y P ˆ Y 2 n T , ˆ Y 2 n T c ˇ ˇ Y 2 n T Y 2 n T c and the latter term is defined as E X 1 , X ,W ' P b 2 n ˆ Y | X,W p ˆ Y 2 n T , ˆ Y 2 n T c | X , X 1 , W q ı in which X 1 , X , W ' P X 1 | Y 2 n T c P X ,W | Y 2 n T .

The proof of Theorem 3 is deferred to Appendix E.4. Also, a similar tail bound is provided in Appendix A.

Using the part (i) of Lemma 1, it can be seen that if the value of the KL divergence term is larger than log p n q then the bound of Theorem 3 is tighter than that Theorem 1. Also, using part (ii) of Lemma 1 it is seen that if the error on the training set is zero (a setting referred to as 'realizable case' in [BL03]), our Theorem 3 yields a bound on the generalization error which is O p 1 { n q .

3 Generalization bounds in terms of latent variable complexity

The generalization bounds of the previous section are particularly useful in the following sense: if the learning algorithm is 'simple' enough to produce a low 'relative entropy' sequence of labels for training and test sets, then the algorithm generalizes well. However, they have a downside that they cannot be used directly as they are in the optimization, since minimizing those bounds may result in solutions with large empirical risk. In this section, we extend the results of the previous section to settings in which the processing is split into two parts: an encoder part that produces a family of representations that have the property to generalize well (developed according to the guidelines of the previous section) and a decoder part that selects among that family one representation that minimizes the empirical risk. As such the goal of the encoder is to guarantee a small generalization error and that of the decoder is to guarantee a small empirical risk. This procedure, which is similar to the Information Bottleneck method, aims at finding a good balance between generalizing well to unseen data and minimizing the risk of the training data.

Let W ' p W e , W d q , where W e and W d are the hypotheses (or models) used by the encoder and the decoder, respectively, as illustrated in Fig. 1b. Also, let U denote the output of the encoder, which will be referred to hereafter interchangeably as 'representation' or 'latent variable'. The encoder produces the representation U according to the conditional P U | X,W e . The decoder produces an estimate ˆ Y of the true label Y according to the conditional P ˆ Y | U,W d . We consider a stochastic learning algorithm A : Z n Ñ W which picks a model W ' p W e , W d q P W : ' W e ˆ W d according to P W | S . Let a loss function ℓ : Z ˆ W Ñr 0 , 1 s be given. For a model w ' p w e , w d q the quality of the prediction of ˆ Y is evaluated as

✶ ✶ Empirical risk, population risk, and generalization error are defined similarly to the previous section. According to the above motivation, we are interested in establishing a bound on the generalization error of the algorithm A that depends only on the part W e of the model W . In accordance with our compressibility

framework of Section 2 such bound would then depend on the complexity of the latent variable U . However, here, the encoder part W e and the decoder part W d are both trained using the dataset S . Thus, they may be statistically dependent in general and therefore the 'rearrangement' ideas using type-I and type-II symmetries do not work here. To elaborate on this a bit more, recall that for example for type-I symmetry, to rearrange p Y , ˆ Y q and p Y 1 , ˆ Y 1 q in an indistinguishable manner, we randomly shuffle the positions of the pairs p Y i , ˆ Y i q and p Y 1 i , ˆ Y 1 i q . If we now consider using the type-I symmetry for the new setup, then shuffling of the pairs p Y i , U i , ˆ Y i q and p Y 1 i , U 1 i , ˆ Y 1 i q , changes the dataset S and thus the distributions P W | S and P ˆ Y | W d ,U . Hence, covering the resulting rearranged sequence would unavoidably depend on both encoder and decoder parts. To overcome this issue, we use type-III symmetry, where we leave the train and ghost datasets untouched and randomly 'swap' only those latent variables and their corresponding predictions, i.e., p U i , ˆ Y i q and p U 1 j , ˆ Y 1 j q , that are associated with the train and ghost samples with the same label, i.e., if Y i ' Y 1 j . The following result is the main theorem of this section and the paper derived using the type-III symmetric priors. A formal proof of this result can be found in Appendix E.5.

Theorem 4 (Generalization Bound for Representation Learning Algorithms) . Consider a K -classification learning task. Let Q be a type-III symmetric conditional prior over U 2 n given X 2 n , Y 2 n and ˆ W e P W e , namely, Q ´ p u π p 1 q , . . . , u π p 2 n q q ˇ ˇ p x 1 , . . . , x 2 n q , p y 1 , . . . , y 2 n q , ˆ w e ¯ remains the same for all permutations π : r 2 n s ÞÑ r 2 n s that preserves the label, i.e., y π p i q ' y i for i P r 2 n s . Then,

where S, S 1 , ˆ W e ' P S, ˆ W e P S 1 and the infimum is over all Markov kernels P ˆ W e | S such that for ˆ W ' p ˆ W e , W d q , E P S,W P ˆ We | S ' gen p S, W q ´ gen p S, ˆ W q ‰ ď ϵ .

The bound of Theorem 4 on the generalization error of representation learning, and the related IB-encoder, is to the best of our knowledge the first of its kind and significance. In fact, while few works have already investigated the generalization error of IB [SST10, VPV18, KDJH23], for the special case of discrete variables, their bounds, which in part are expressed in terms of the empirical mutual information ˆ I p U ; X q , appear to be vacuous for most settings as already observed in [RG19, GK19, AG19, DKSV20] (see introduction and Appendix B.2 for more details). Furthermore, as also extensively discussed in [AG19, GK19, DKSV20], mutual information may not be a good indicator of the generalization error. Such aspects, including how mutual information fails to reflect 'simplicity' or 'structure' of the encoder, are discussed in more detail in the previous section. An alternate bound on the generalization error of representation learning appeared in [DKSV20]. However, their bound only holds for encoders that find the optimal representation in a sense defined therein.

Now, few remarks on the result of Theorem 4 are in order. First, note that the generalization bound of this theorem depends only on the encoder and more precisely on the complexity of the latent space U ; and, so, this bound is valid for any choice decoder. 2 Next, similar to the bounds of the Theorems 1-5, the KL-divergence term of Theorem 4 explicitly takes into account the 'structure' and 'simplicity' of the encoder, and, therefore, it resolves one of the major issues of the IB method [AG19, GK19, DKSV20].

Moreover, the result of Theorem 4 suggests that the considered prior could depend on the data and model. This enables a larger class of choices for the prior. Examples include (i) Symmetric jointly Gaussian priors, (ii) priors that depend on the category (label), i.e., the prior for a category k P r K s at each optimization iteration could possibly depend on some statistics of the latent variables of all training and test samples having label k , and (iii) priors that steer latent variables toward some pre-defined 'constellations' in the latent space, depending on their label. On this aspect, note that allowing the prior to depend on labels enables a connection with the 'conditional information-bottleneck' of [Fis20].

Another important property of our bound of Theorem 4 is that, unlike the empirical mutual information term of [SST10, VPV18, KDJH23], 3 it does not become infinite for deterministic encoders with continuous input-output. Moreover, our approach which is based on lossy compression provides an interpretation of the geometric compression of [GK19, GVDBG ` 19] where latent variables are concentrated around some

constellation points. 4 We hasten to mention that 'lossy compression' here should not be confused with approaches that add noise after the encoder. The main difference is that in those approaches the noisy representations are passed to the decoder; while here, the 'noisy' representations are used only to estimate lossy compressibility. These 'noisy' representations can be achieved by either adding small 'noise' to the model parameters or the latent variable. Note that by increasing the noise level the ϵ term in Theorem 4 increases while the KL-divergence term potentially decreases. In practice, a suitable trade-off between the two effects can be found by treating the amount of added noise as a hyper-parameter to optimize.

Finally, a similar tail bound on the generalization error has been established in Appendix A.

4 Experiments

In this section, we illustrate our results via some experiments. For more detail and other experiments the reader is referred to Appendix D.

Our main Theorems 4 and 7 suggest that for the representation learning setup of Fig. 1b the generalization error is controlled essentially by the divergence term D KL ´ P b 2 n U | X,W e p U , U 1 | X , X 1 , Y , Y 1 , W e q › › Q ¯ where Q is a type-III symmetric prior. In a sense, this also means that, with a proper data-dependent choice of the prior, the usage of the aforementioned divergence term as a regularizer possibly offers better generalization guarantees (relative, e.g., to the conventional data-independent prior of VIB). In what follows, we propose a new family of data-dependent priors which appear to better capture the 'simplicity' or 'structure' of the encoder. We also compare the associated accuracy with that offered by the fixed prior of VIB.

More precisely, in VIB the prior Q factorizes as a product of 2 n scalar standard Gaussian priors, i.e., Q ' Q b 2 n , where Q ' N p 0 m , I m q , 0 m P R m is the zero vector and I m is the m ˆ m identity matrix. In our lossless approach, which we here coin as Lossless Category-Dependent VIB (CDVIB), the prior Q still factorizes as a product of 2 n scalar Gaussian priors, i.e., Q ' ś i Pr 2 n s Q i , but with three major differences: i. Each Q i can be chosen from a set of M ˆ K priors M priors for each label. ii. Unlike VIB, each of M ˆ K priors can depend on some statistics of p S , S 1 q and also on the label of each sample, iii. Unlike VIB, where the prior is fixed, here Q is 'learned' during the training phase. To this end, the mean and variance of the scalar priors are updated after each training iteration using a moving average with some small coefficient, allowing the latent space to better adapt to the structure of the encoder and the data. Taking the moving average also has another role, which is to 'partially' reproduce the effect of the 'ghost data' (which comes from the test dataset that is usually unavailable during training). In lossy CDVIB, similar priors are considered but over 'noisy' versions of the latent variables, i.e., p ˆ U , ˆ U 1 q . Note that while the noisy versions are considered for the regularizer, the decoder receives as input p U , U 1 q .

We consider CIFAR10 [KH ` 09] image classification using a small CNN-based encoder and a linear decoder. The results shown in Fig. 2 indicate that the model trained using our priors achieves better ( ' 2 . 5% ) performance in terms of both generalization error and population risk. This suggests that our priors help to find a better representation than the standard VIB prior.

(a) Accuracy.

(b) Log-likelihood.

0.66

-0.8

standard VIB

standard VIB

0.65

lossyCDVIB

-0.9

lossyCDVIB

lossless CDVIB

1.0

lossless CDVIB

0.64

poou

-1.1

0.63

1.2

0.62

6

1.3

1.4

0.61

1.5

0.60

1.6

0

1e-7

1e-6

1e-5

1e-4

1e-3

1e-2

le-1

1

0

1e-7

1e-6

1e-5

1e-4

1e-3

1e-2

le-1

1

Beta

Beta

References

[AFDM17] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations , 2017.

[AG19] Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence , 42(9):2225-2239, 2019.

[AGNZ18] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning , pages 254-263. PMLR, 2018.

[Aud04] Jean-Yves Audibert. PAC-Bayesian statistical learning theory. PhD thesis, Universit´ e Paris VI, , 2004.

[AZ19] Inaki Estella Aguerri and Abdellatif Zaidi. Distributed variational representation learning. IEEE transactions on pattern analysis and machine intelligence , 43(1):120-138, 2019.

[BBR ` 18] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 , 2018.

[BEHW87] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam's razor. Information processing letters , 24(6):377-380, 1987.

[BHMZ20] Olivier Bousquet, Steve Hanneke, Shay Moran, and Nikita Zhivotovskiy. Proper learning, helly number, and an optimal svm bound. In Conference on Learning Theory , pages 582-609. PMLR, 2020.

[BL03] Avrim Blum and John Langford. Pac-mdl bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings , pages 344-357. Springer, 2003.

[BM06] Nader H Bshouty and Hanna Mazzawi. Exact learning composed classes with a small number of mistakes. In Learning Theory: 19th Annual Conference on Learning Theory, COLT 2006, Pittsburgh, PA, USA, June 22-25, 2006. Proceedings 19 , pages 199-213. Springer, 2006.

[BO18] L´ eonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems , 31, 2018.

[BSE ` 21] Melih Barsbey, Milad Sefidgaran, Murat A Erdogdu, Ga¨ el Richard, and Umut S ¸ ims ¸ekli. Heavy tails in SGD and compressibility of overparametrized neural networks. In ThirtyFifth Conference on Neural Information Processing Systems , 2021.

[BZV20] Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening mutual informationbased bounds on generalization error. IEEE Journal on Selected Areas in Information Theory , 1(1):121-130, May 2020.

[Cat07] Olivier Catoni. Pac-bayesian supervised classification. Lecture Notes-Monograph Series. IMS , 1277, 2007.

[CK11] Imre Csisz´ ar and J´ anos K¨ orner. Information Theory: Coding Theorems for Discrete Memoryless Systems . Cambridge University Press, 2 edition, 2011.

[CK22] Dan Tsir Cohen and Aryeh Kontorovich. Learning with metric losses. In Conference on Learning Theory , pages 662-700. PMLR, 2022.

[CT06] Thomas M. Cover and Joy A. Thomas. Elements of information theory (2. ed.) . Wiley, 2006.

[Dev83] Luc Devroye. The equivalence of weak, strong and complete convergence in l1 for kernel density estimates. The Annals of Statistics , pages 896-904, 1983.

[DKSV20] Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. Advances in Neural Information Processing Systems , 33:18674-18690, 2020.

[Fis20] Ian Fischer. The conditional entropy bottleneck. Entropy , 22(9):999, 2020.

[Gal68] Robert G Gallager. Information theory and reliable communication , volume 588. Springer, 1968.

[GB10] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

[Gei21] Bernhard C Geiger. On information plane analyses of neural network classifiers-a review. IEEE Transactions on Neural Networks and Learning Systems , 2021.

[GK19] Bernhard C Geiger and Tobias Koch. On the information dimension of stochastic processes. IEEE transactions on information theory , 65(10):6496-6518, 2019.

[GMP05] Peter D Gr¨ unwald, In Jae Myung, and Mark A Pitt. Advances in minimum description length: Theory and applications . MIT press, 2005.

[GP20] Ziv Goldfeld and Yury Polyanskiy. The information bottleneck problem and its applications in machine learning. IEEE Journal on Selected Areas in Information Theory , 1(1):19-38, 2020.

[GR19] Peter Gr¨ unwald and Teemu Roos. Minimum description length revisited. International journal of mathematics for industry , 11(01):1930001, 2019.

[GSZ21] Peter Grunwald, Thomas Steinke, and Lydia Zakynthinou. Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vc classes. In Conference on Learning Theory , pages 2217-2247. PMLR, 2021.

[GVDBG ` 19] Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2299-2308. PMLR, 09-15 Jun 2019.

[HD22] Fredrik Hellstr¨ om and Giuseppe Durisi. A new family of generalization bounds using samplewise evaluated cmi. Advances in Neural Information Processing Systems , 35:1010810121, 2022.

[HJTW21] Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. In International Conference on Learning Representations , 2021.

[HK19] Steve Hanneke and Aryeh Kontorovich. A sharp lower bound for agnostic learning with sample compression schemes. In Algorithmic Learning Theory , pages 489-505. PMLR, 2019.

[HK21] Steve Hanneke and Aryeh Kontorovich. Stable sample compression schemes: New applications and an optimal svm margin bound. In Algorithmic Learning Theory , pages 697-721. PMLR, 2021.

[HKS19] Steve Hanneke, Aryeh Kontorovich, and Menachem Sadigurschi. Sample compression for real-valued learners. In Algorithmic Learning Theory , pages 466-488. PMLR, 2019.

[HKSW20] Steve Hanneke, Aryeh Kontorovich, Sivan Sabato, and Roi Weiss. Universal bayes consistency in metric spaces. In 2020 Information Theory and Applications Workshop (ITA) , pages 1-33. IEEE, 2020.

[HRVSG21] Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Informationtheoretic generalization bounds for black-box learning algorithms. Advances in Neural Information Processing Systems , 34, 2021.

[KASK22] Michael Kleinman, Alessandro Achille, Stefano Soatto, and Jonathan Kao. Gacs-korner common information variational autoencoder. arXiv preprint arXiv:2205.12239 , 2022.

[KB15] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

[KDJH23] Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 16049-16096. PMLR, 23-29 Jul 2023.

[KH ` 09] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada , 2009.

[KTVK18] Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. arXiv preprint arXiv:1808.07593 , 2018.

[KTW19] Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. Entropy , 21(12):1181, 2019.

[KW14] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR , 2014.

[LEG18] Cheuk Ting Li and Abbas El Gamal. Strong functional representation lemma and applications to coding theorems. IEEE Transactions on Information Theory , 64(11):6967-6978, 2018.

[LLS ` 23] Yilin Lyu, Xin Liu, Mingyang Song, Xinyue Wang, Yaxin Peng, Tieyong Zeng, and Liping Jing. Recognizable information bottleneck. arXiv preprint arXiv:2304.14618 , 2023.

[LW86] Nick Littlestone and Manfred Warmuth. Relating data compression and learnability. Citeseer , 1986.

[McA98] David A McAllester. Some pac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory , pages 230-234, 1998.

[NBS18] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks, 2018.

[PGM ` 19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

[RG19] Borja Rodriguez Galvez. The information bottleneck: Connections to other problems, learning and exploration of the ib curve, 2019.

[RGBTS21] Borja Rodr´ ıguez-G´ alvez, Germ´ an Bassi, Ragnar Thobaben, and Mikael Skoglund. On random subset generalization error bounds and the stochastic gradient langevin dynamics algorithm. In 2020 IEEE Information Theory Workshop (ITW) , pages 1-5. IEEE, 2021.

[RGTS20] Borja Rodr´ ıguez G´ alvez, Ragnar Thobaben, and Mikael Skoglund. The convex information bottleneck lagrangian. Entropy , 22(1):98, 2020.

[Ris78] Jorma Rissanen. Modeling by shortest data description. Automatica , 14(5):465-471, 1978.

[RZ16] Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51 of Proceedings of Machine Learning Research , pages 1232-1240, Cadiz, Spain, 09-11 May 2016. PMLR.

[SAM ` 20] Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error. In International Joint Conference on Artificial Intelligence , pages 2839-2846, 2020.

[Sau72] Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A , 13(1):145-147, 1972.

[SGRS22] Milad Sefidgaran, Amin Gohari, Gael Richard, and Umut Simsekli. Rate-distortion theoretic generalization bounds for stochastic learning algorithms. In Conference on Learning Theory , pages 4416-4463. PMLR, 2022.

[She72] Saharon Shelah. A combinatorial problem; stability and order for models and theories in infinitary languages. Pacific Journal of Mathematics , 41(1):247 - 261, 1972.

[S ¸ SDE20] Umut S ¸ims ¸ekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 5138-5151. Curran Associates, Inc., 2020.

[SST10] Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science , 411(29-30):2696-2711, 2010.

[SZ20] Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 3437-3452. PMLR, 09-12 Jul 2020.

[SZ23] Milad Sefidgaran and Abdellatif Zaidi. Data-dependent generalization bounds via variablesize compressibility. arXiv preprint arXiv:2303.05369 , 2023.

[SZT17] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017.

[THP23] Dor Tsur, Bashar Huleihel, and Haim Permuter. Rate distortion via constrained estimated mutual information minimization. In 2023 IEEE International Symposium on Information Theory (ISIT) , pages 695-700, 2023.

[TPB00] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

[VDOV ` 17] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

[VPV18] Mat´ ı Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck in representation learning. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 1580-1584, 2018.

[Wai19] Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.

[XR17] Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems , 30, 2017.

[ZEAS20] Abdellatif Zaidi, I˜ naki Estella-Aguerri, and Shlomo Shamai. On the information bottleneck problems: Models, connections, applications and information theoretic views. Entropy , 22(2):151, 2020.

The appendices are organized as follows.

Appendix A contains further theoretical results. Specifically,

-in Appendix A.1 two more tail bounds on the generalization error in terms of the predicted label complexity are presented.

-in Appendix A.2 a tail bound on the generalization error in terms of the latent variable complexity is presented.

We discuss some of the related works in more detail in Appendix B. In particular,

-in Appendix B.1, we recall the compressibility framework of Blum and Langford [BL03].

-in Appendix B.2, we discuss the results of [KDJH23].

In Appendix C, we discuss various topics that could not be sufficiently addressed in the paper, due to lack of space. More precisely,

-we define various notions of compressibility, including lossless, lossy, and variable-size compressibility in Appendix C.1,

-we present an intuition on the function h D p¨ , ¨q , used in Section 2, in Appendix C.2,

-we give a simple example of geometrical compression, and its relation to lossy compressibility, in Appendix C.3,

-and finally we present the conclusion of our work together with some interesting future directions in Appendix C.4.

Appendix D contains the details of the experiments presented in Section 4.

-Appendix D.1 recalls the standard VIB objective function,

-Appendix D.2 defines a family of lossless objective functions using data-dependent priors,

-Appendix D.3 defines a family of lossy objective functions using data-dependent priors,

-Appendix D.4 details the training and testing datasets used in experiments,

-Appendix D.5 details the model architecture used in experiments,

-Appendix D.6 provides training details,

-Appendix D.7 presents and discusses the numerical results.

Appendix E contains the deferred proofs of all our theoretical results. More precisely,

-the intuition behind our proof techniques is presented in Appendix E.1,

-Appendix E.2 contains the proof of Theorem 1,

-Appendix E.3 contains the proof of Lemma 1,

-Appendix E.4 contains the proof of Theorem 3,

-Appendix E.5 contains the proof of Theorem 4,

-Appendix E.6 contains the proof of Theorem 5,

-Appendix E.7 contains the proof of Theorem 6,

-Appendix E.8 contains the proof of Theorem 7,

-Appendix E.9 contains the proof of Lemma 3.

A Additional theoretical results

In this section, we present several tail bounds on the generalization error in terms of the complexity of the predicted labels or the latent variables.

A.1 Tail bounds on the generalization error in terms of predicted label complexity

First, we start by stating a lossy tail bound on the generalization error in terms of the complexity of the tail bound.

Appendices

Theorem 5. Consider the learning framework defined above. Let Q be any fixed type-I symmetric prior on p ˆ Y , ˆ Y 1 q that could depend on p X , Y , X 1 , Y 1 q . Then for any ϵ P R and δ P R ` with probability at least p 1 ´ δ q over choice of S and S 1 , we have that E W ' P W | S r gen p S, W qs is upper bounded by

where the infimum is over all P ˆ W | S that satisfy

The theorem is proved in Appendix E.6. In the above result, it is easy to replace the expectation with respect to P W | S with any arbitrary expectations, as it is common in PAC-Bayes bounds and used for example in [NBS18].

Next, we state a tail bound in terms of the function h D p¨ , ¨q , defined in (5).

Theorem 6. Consider the learning framework of Section 2. Let Q be any fixed type-II symmetric prior on p ˆ Y , ˆ Y 1 q that could depend on p X , Y , X 1 , Y 1 q . Then, for any δ P R ` , with probability at least p 1 ´ δ q over choices of p S, S 1 , W q ' P S 1 P S,W , it holds that

This theorem is proved in Appendix E.7. In particular, when ˆ L p S, W q ' 0 , using Lemma 1 conclude that, this result yields that with probability at least p 1 ´ δ q ,

A.2 Tail bounds on the generalization error in terms of latent variable complexity

In this section, we present a tail generalization bound for the two-step learning setup of Section 3 that highlights the relevance of latent variable compressibility to generalization error.

Theorem 7. Consider a K -classification learning task with the above defined framework. Let Q be a type-III symmetric conditional prior over U 2 n given X 2 n , Y 2 n and W e P W e , namely, Q ´ p u π p 1 q , . . . , u π p 2 n q q ˇ ˇ p x 1 , . . . , x 2 n q , p y 1 , . . . , y 2 n q , w ¯ remains the same for all permutations π : r 2 n s ÞÑ r 2 n s that preserves the label, i.e., y π p i q ' y i for i P r n s . Then, for any λ P R ` , i. with probability at least p 1 ´ δ q over choices of p S, S 1 , W q ' P S 1 P S,W ,

ii. with probability at least p 1 ´ δ q over choices of p S, S 1 , W q ' P S 1 P S,W ,

Note that the second part of the theorem can be easily derived from the first part using Hoeffding's inequality. We provide the proof of the first part in Appendix E.8.

B Related works

In this section, we discuss some of the related works in more detail.

B.1 PAC-MDL framework of Blum and Langford

Here, we recall the PAC-MDL framework of [BL03] which, therein, was introduced in the form of a (compression) game between two agents, Alice and Bob. Alice has access to both a labeled training set S ' p X n , Y n q , consisting of n labeled samples, and a test set S 1 ' X 1 n , consisting of n unlabeled samples, all drawn independently from µ . 5 Bob has available just the test set and the unlabeled version of the training set, i.e., p X n , X 1 n q , ordered in some predefined (e.g., lexicographic) order known to both agents. Hereafter we denote the ordered vector of labeled and unlabeled samples as π p X n , X 1 n q . It is assumed that by observing the vector π p X n , X 1 n q Bob cannot know which samples of it are from the training set and which are from the test set. The goal of Alice is to communicate the labels π p Y n , Y 1 n q to Bob using as few bits as possible. 6 To this end, let E : Z n ˆ X n Ñt 0 , 1 u ˚ be a mapping (encoder) used by Alice and σ : ' E p S, X 1 n q the string transmitted to Bob. The goal of Bob is to guess the labels of both sets X n and X 1 n ; and does so by running a decoder mapping D : X 2 n ˆt 0 , 1 u ˚ Ñ Y 2 n . That is, Bob forms an estimate of the true labels as D p π p X n , X 1 n q , σ q . In this context, the empirical risk is measured as ˆ L p σ, s, s 1 q : ' 1 n ř i Pr n s ✶ t y i ‰ ˆ y i u and a proxy test risk is measured as L p σ, s, s 1 q : ' 1 n ř i Pr n s ✶ t y 1 i ‰ ˆ y 1 i u . Essentially, the PAC-MDL bound of [BL03] can be seen as a generalized version of Occam's-Razor theorem [LW86, BEHW87] which states that if one can explain (or encode) the labels of a set of n training samples by a hypothesis that can be described using only | σ | ! n bits then this guarantees that this hypothesis generalizes well for unseen samples.

Theorem 8 ([BL03, Theorem 6]) . For any prior distribution Q p σ q of σ and any δ ą 0 , with probability at least 1 ´ δ over the choices of S, S 1 ' P S P S 1 , we have:

The result is made more explicit for the so-called realizable case, i.e., when ˆ L p σ, S, S 1 q ' 0 . For this case, we have [BL03, Corollary 3]

where | σ | denotes the size in bits of the string σ using some a priori fixed codewords. The result clearly shows that if the string σ can be sent using few bits then the algorithm generalizes well. Also, the approach can be used in order to establish similar results for the VC-dimension and PAC-Bayes bounds.

We emphasize that our construction is in sharp contrast with an adaptation of the above-described approach of Blum and Langford [BL03] in which one would reveal both labeled and unlabeled samples p S m , X 1 mn q to Alice and only unlabeled samples p X mn , X 1 mn q to Bob (both in a predefined order). We, however, in our proposed approach in Section 2.1, reveal the vector Z 2 mn , containing p S m , S 1 m q in a predefined order to both. Furthermore, we emphasize that the goal in our approach is not to recover the labels Y 2 mn , but the predictions ˆ Y 2 mn as produced by the picked models or hypotheses W m .

On another note, we remark that the interpretation of R , which appeared in the size of the codebook in our approach, i.e., | ˆ Y m | ď e mR , corresponds to the average number of bits (per dataset S ) that is needed to send a compressed version of the string σ of [BL03]. As such, for the fixed-size codebook explained in Section 2.1 and when | σ | is constant, R ď | σ | . Similar relations hold in general, by considering the variable-size codebook.

where and

B.2 On the generalization bounds of [KDJH23]

After the initial submission of our work, another work [KDJH23] appeared on arXiv, accepted at ICML 2023, that provides an upper bound on the generalization error of representation learning algorithms. As claimed, this result justifies the benefits of the information bottleneck principle, by relating IB to the generalization error. The results are provided for the multi-layer neural networks and for different choices of latent variables corresponding to the output of different layers. Here, to adapt the results to the setup of this paper, we only consider the output of the encoder layer as the latent variable and adapt correspondingly the notations of the results in [KDJH23] to our notations.

With the adapted notations, [KDJH23] claims to bound the generalization error of a representation learning algorithm 'roughly by

However, firstly, since this bound is in terms of the mutual information function, the critics on the relation of mutual information and generalization error, discussed in [KTVK18, RG19, AG19, DKSV20] and provided also in the 'Critics to IB' section of the introduction (Section 1), are valid for the results of [KDJH23], as well. More importantly, we could not conclude the reported order-wise behavior from [KDJH23, Theorem 2]. Using the notations of our work, their result states that with probability at least 1 ´ δ over training data S , the generalization error is bounded by

where the constants G 1 , ˆ G 2 , and G 3 are defined in [KDJH23, Appendix E.1] and claimed that ˆ G 2 ' ˜ O p 1 q and G 3 ' ˜ O p 1 q , as n Ñ8 . However, we were unable to resolve the following concerns regarding this bound.

Firstly, it is not clear how I p W e ; S q is considered to behave as ˜ O p 1 q , when n Ñ8 . Indeed, the size of dataset S and the learning algorithm P W e | S change as n Ñ8 .

Secondly, by referring to the definitions of the constants in [KDJH23, Appendix E.1], it can be easily verified that ˆ G 2 ' C 1 `p H p U | X,Y q ` H p W e | S qq ln p 2 q , for some non-negative constant C 1 . Hence, the bound can be re-written as

Thus, the bound is in terms of H p U | Y q ` H p W e q , and not I p X ; U | Y q ` I p W e ; S q .

Lastly, the term G 3 , defined in [KDJH23, Appendix E.1], is composed of T Y P N elements. By [KDJH23, Lemma 2], T Y behaves roughly as ˜ O p 2 H p U | Y q q . Using [KDJH23, Lemma 2], it is shown that G 3 is bounded. However, it seems that this is only shown by assuming that the T Y terms of G 3 satisfy certain conditions related to the decreasing rate of their ordered values. This assumption however, may not hold in general, and hence G 3 may behave as ˜ O p T y q « ˜ O p 2 H p U | Y q q , which then becomes the dominant term in the bound.

C Further clarifications and discussions

In this section, we explain our compressibility framework in more detail. Moreover, we present some intuitions on the function h D p¨ , ¨q and give an example of how lossy compression is related to geometrical compression. Finally, we discuss some potential future works.

C.1 Compressibility framework

In this section, we propose various notions of the compressibility used in Section 2.

C.1.1 Lossless fixed-size compressibility

We start by recalling the needed elements for joint compression of a block of the predicted labels. Consider m i.i.d. pairs of train and test datasets S j : ' p Z j, 1 , . . . , Z j,n q and S 1 j : ' p Z 1 j, 1 , . . . , Z 1 j,n q ,

where Z j,i ' p X j,i , Y j,i q and Z 1 j,i ' p X 1 j,i , Y 1 j,i q . Let S m ' p S 1 , . . . , S m q , S 1 m ' p S 1 1 , . . . , S 1 m q and W m : ' p W 1 , . . . , W m q , where W j ' P W j | S j . It is important to note that the model or hypothesis W j is chosen based on the dataset S j only and the introduction of the ( ghost ) dataset S 1 j is here only for the sake of the analysis, similarly as it was done in the derivation of the PAC-MDL bound of [BL03] or in Rademacher sample complexity or the conditional mutual information of [SZ20]. Denote the predicted labels using model W i for inputs X i,j and X 1 j,i as ˆ Y j,i and ˆ Y 1 j,i , for j P r m s and i P r n s .

Moreover, let Z 2 mn P Z 2 mn denote a rearrangement of the elements of p S m , S 1 m q such that while the entire sets S m and S 1 m are known by having the vector Z 2 mn , but they are arranged in an order that one cannot distinguish whether a given sample z is from S m or S 1 m . Accordingly, denote the rearranged versions of p Y mn , Y 1 mn q and p ˆ Y mn , ˆ Y 1 mn q respectively by Y 2 mn and ˆ Y 2 mn . In Sections 2.2 and 2.3, we presented two methods for such rearrangement. As mentioned before, in both methods, the rearrangement of p S m , S 1 m q as Z 2 mn ' t Z j,i u j Pr m s ,i Pr n s is done in two steps: First, for each j P r m s , we rearrange indistinguishably p S j , S 1 j q as Z q j 2 n , and then we concatenated the m resulting t Z 2 n j u j Pr m s to achieve Z 2 mn . Now, we explain how to rearrange indistinguishably a given p S, S 1 q as Z 2 n using two methods:

Section 2.2: Let J ' p J 1 , . . . , J n q be a vector of n i.i.d. Bernoulli p 1 2 q random variables J i P t i, i ` n u , i P r n s . Denote by J ' p J c 1 , . . . , J c n q the vector of the complementary choices in J , i.e., J i Y J c i ' t i, i ` n u . Now, for each i P r n s , let p Z J i , Z J c i q to be equal to p Z i , Z 1 i q .

Section 2.3: Let T ' t T 1 , . . . , T n u , be a random set obtained by picking uniformly n indices from t 1 , . . . , 2 n u , without replacement. Note that in contrast to J which had i.i.d. components, here the components of T are dependent. let T c ' t 1 , . . . , 2 n uz T be the complement of T , having the elements T c ' t T c 1 , . . . , T c n u . Now, for each i P r n s , let p Z T i , Z c T i q ' p Z i , Z 1 i q .

Based on the above, hereafter we study the compressibility of the sorted model-predicted labels ˆ Y 2 mn , from an information-theoretic point of view. The rationale is that, in accordance with 'Occam's Razor' theorem [LW86, BEHW87], since the (rearranged) predicted-labels vector ˆ Y 2 mn agrees mostly with the true labels Y 2 mn on the dataset S m , if ˆ Y 2 mn can be described using only a few bits, this guarantees that the model W generalizes well. As it will be shown from the result that will follow, instead of the size of the message needed to be sent in the Blum-Langford approach (see Appendix B.1), a new quantity emerges in our work as a measure of the compressibility of ˆ Y 2 mn : the relative entropy of the joint conditional P p ˆ Y n , ˆ Y 1 n | Y n , Y 1 n q and a (symmetric) conditional prior Q over ˆ Y 2 n given Y 2 n . Depending on the way we rearrange p S m , S 1 m q , the type-I or type-II symmetric priors are needed to be applied.

Let R P R ` . In our block-coding rate-distortion theoretic framework, R is said to be achievable if there exists a compression codebook of size « e mR , fixed a priori, which covers the space spanned by the model-predicted labels ˆ Y 2 mn with high probability. Specifically, if there exists a sequence of label books t ˆ Y m u m P N , where ˆ Y m : ' t ˆ y r r s , r P r l m su Ď Y 2 mn , l m P N , ˆ y r r s ' p ˆ y 1 r r s , . . . , ˆ y m r r sq and ˆ y j r r s ' p ˆ y j, 1 r r s , . . . , ˆ y j, 2 n r r sq P Y 2 n such that:

(i) l m ď e mR ,

(ii) there exist a sequence t δ m u m P N for which lim m Ñ8 δ m ' 0 and with probability at least p 1 ´ δ m q over the choices of S m , S 1 m , and J or T , one can find at least one index r P r l m s whose associated ˆ y r r s exactly equals ˆ Y 2 mn .

The above framework, explained in Section 2.1, is called fixed-size compressibility , as the size of the codebook does not depend on a given dataset. This type of compressibility is useful for establishing 'data-independent' bounds, 7 e.g., bounds on the expectation of the generalization error. This is also called lossless , since we look for a codeword that is exactly equal to ˆ Y 2 mn .

C.1.2 Lossy fixed-size compressibility

As already stated, the above approach and results can be extended to any bounded loss function and continuous variables Y and ˆ Y (see for example Section 3 where continuous latent variables are studied). In this case, a standard result of rate-distortion theory states that to cover losslessly and reliably ˆ Y 2 mn , R should be infinity, which makes the framework and resulting bounds vacuous. However, the approach can

be easily extended to include the lossy compression of the labels; in the same spirit as done in [SGRS22] for the hypothesis compression. More precisely, for a given distortion threshold ϵ P R , one can consider the same way of codebook generation, but with the following conditions:

(i) l m ď e mR ,

(ii) there exist a sequence t δ m u m P N for which lim m Ñ8 δ m ' 0 and with probability at least p 1 ´ δ m q over the choices of S m , S 1 m , and J or T , one can find at least one index r P r l m s whose associated ˆ y r r s satisfies:

where it is assumed that the set of indices tp j, t i qu j Pr m s ,i Pr n s and tp j, t c i qu j Pr m s ,i Pr n s are the sets of indices in the rearranged sequence that belong to the training and ghost datasets, respectively. Here, Ť i Pr n s p t i Y t c i q ' t 1 , . . . , 2 n u .

In the case of lossy compressibility, the block-coding technique brings another advantage in addition to the advantages discussed for the lossless case: it allows to consider the average distortion criterion, instead of worst-case distortion criterion. Please refer to [SGRS22] for further discussion on this.

C.1.3 Variable-size compressibility

In the frameworks, explained in previous sections, for a given m , the size of the codebook e mR is fixed. Hence, the bounds derived using this framework, cannot depend on a particular training dataset at hand. In particular, while above mentioned frameworks are useful to establish bounds on the expectation of the generalization error, they are not appropriate for establishing data-dependent tail bounds. To overcome this issue, in [SZ23], a 'variable-size' compressibility is proposed, in which, one is allowed to search for a suitable ˆ y r r s only in the first e mR p S,S 1 ,W q elements of this codebook, where R p S,S 1 ,W q is the datadependent term that will consequently appear in the resulting bound using this approach. For instance, the term R p S,S 1 ,W q in Part.ii of Theorem 1 is exactly the KL-divergence term appearing in the bound.

C.2 Intuition about the function h D

In Section 2.3, we have provided the bound on the generalization error in terms of the function h D p x ; x 1 q : r 0 , 1 s ˆ r 0 , 1 s Ñ r 0 , 2 s , defined as:

where h b p x q : ' ´ x log 2 p x q´p 1 ´ x q log 2 p 1 ´ x q . As mentioned, 1 2 h D p x, x 1 q is equal to the Jensen-Shannon divergence between two binary Bernoulli distributions with parameters x and x 1 .

In this section, we provide an intuition about this function, by showing its relation with the combinatorial term that appeared in [BL03]. Recall that the main result of [BL03], i.e., Theorem 6 therein (re-stated in the Appendix B.1), is established in terms of b max p n, a { n, δ q defined as follows:

where r b, a ` b s Ă N denotes the integer interval and c P N .

Now, the intuition about the function h D p x, x 1 q is as follows: by Stirling's formula, we have that

as m Ñ8 . Hence, for large (infinite) values of m we have that the function Bucket p mn,ma,mb q of [BL03] is dominated by

where

where J b, a ` b K Ă R denotes the real interval and t P R . Furthermore and as a consequence

as

m

Ñ8

.

It can be observed that by considering block-coding and letting m Ñ8 , the intractable combinatorial terms appeared in [BL03] can be expressed in terms of the function h D p¨ , ¨q .

C.3 On relation between lossy compressibility and geometric compression

The experimental studies suggest the existence of a relation between generalization performance and geometrical compression [GK19]. Geometrical compression occurs when the latent variables are concentrated around a limited number of clusters. Please refer to [GK19, Fig. 2] for a visual representation. As mentioned both in Section 3, lossy compression provides an interpretation of the geometric compression of [GK19, GVDBG ` 19] where latent variables are concentrated around some constellation points. In this section, we provide a simple example of that.

In geometrical compression, the latent variables U ' P U | X of X are distinct but concentrated around a few 'centers'. Hence, while in this case, the 'lossless compression' captured by I p U ; X q becomes very large or infinite, the 'lossy compression' captured by rate-distortion may be very small, as these scattered latent variables around the centers can be seen as a lossy version of the mappings from X to one of these centers, with some small distortion. To give a simple example, suppose X P r 0 , 1 s , and if X ă 0 . 5 , then U ' ´ 1 ` X { 5 , and if X ą 0 . 5 , then U ' 1 ` X { 5 . In this case while I p U ; X q ' 8 , a simple lossy mapping P ˆ U | X with average distortion less than 0 . 05 can be found such that I p ˆ U ; X q ' 1 . This explains the geometrical compression observation.

C.4 Conclusion and future directions

In this paper, inspired by [BL03], we developed a compressibility framework that we used to establish various bounds on the generalization error of the stochastic learning algorithms. The bounds are expressed in terms of the newly defined notion of 'minimum description length'(MDL) of the predicted labels or the latent variables. The new notion is expressed in terms of a 'symmetric' prior, where the 'symmetry' is defined and used in three different ways. The type-I and type-II symmetries are useful to establish the bounds in terms of MDL of the predicted labels. The former one, in particular, shows a clear connection between the seemingly different approaches of [BL03] and CMI [SZ20, HRVSG21]. The type-III symmetry is used to derive the main result of this paper which is a bound in terms of MDL of latent variables. The results of the last section suggest that the generalization error in representation learning is related to the newly defined MDL of latent variables. Unlike mutual information, which captures the 'information leakage', the new notion of MDL captures the simplicity and structure of the encoder. These insights are then partly exploited to propose new regularizers in terms of new 'data-dependent' priors. The performed simulations show the advantage of these priors over the classical ones used in VIB.

Our proposed framework and obtained results also open up several future research directions. In the following, we discuss some of those directions.

In Part i. of Theorem 1, we proposed a tail bound, which unlike the classical PAC-Bayes bound with a single draw from the posterior [Cat07], does not contain a 'disintegrated' term. As explained, this is due to the choice of the loss function, which inherently contains an expectation term. It would be interesting to consider a loss function that captures the 'one-shot' prediction of the performance and to develop a disintegrated bound for such a loss function.

One of the contributions of this work has been to establish 'rate-distortion theoretic' bounds by using a lossy compressibility framework. For example, the established bound in Theorem 4, contains an infimum over all 'compressed algorithms' P ˆ W e | S that satisfy some distortion criterion. Note that this implies that this bound holds for 'any' choice of eligible P ˆ W e | S , and thus, taking the infimum to obtain a 'valid bound' is not necessarily required. Thus, any simple technique that adds noise or applies parameter quantization can be considered. However, besides these general approaches, an interesting direction is how to find an 'optimal' P ˆ W e | S . Perhaps, the approach taken in [THP23], which uses a combination of the MINE estimator [BBR ` 18] and the 'SFRL' [LEG18] could be adapted to our setup.

Another potentially valuable direction would be to compute the bounds analytically in simple setups, such as two-layer neural networks with Gaussian data, and compare the results with the existing results such as [VPV18]. Also, it is instructive to study numerically the geometry and MDL of the latent variables for different network architectures, e.g., FCN versus CNN. This may lead to a new understanding of how and why some architectures produce 'better' representations.

Inspired by our results, we have proposed some simple 'data-dependent' priors. While the proposed priors show improvements over the classical priors, they are limited to priors that can be factorized as products of some Gaussian distributions. Thus, they only 'partially' capture the joint compressibility of the latent variables (and hence the encoder structure). We imagine that more 'appropriate' priors can be proposed that capture better the structure of the encoder.

In this paper, we partially discuss the intuition behind the importance of the 'structure' of the encoder. However, the explicit effect of the 'structure' on the geometry of latent variables and predictions needs to be investigated, as partially shown in [BM06].

Finally, we emphasize that in representation learning, one is interested in extracting good representations that are suitable in terms of generalization error for multiple learning tasks, simultaneously. It would be interesting to extend our results to such more realistic settings.

D Details of the experiments

In this section, we present the details of the experiments presented in Section 4.

D.1 VIB objective function

The traditional information bottleneck (IB) approach [TPB00, SST10] for training representation learning models, and particularly its variational implementation (VIB) by [AFDM17], considers a fixed dataindependent prior Q ' Q b 2 n . Then, for some Lagrange multiplier β ą 0 , the VIB approach minimizes

using the reparametrization trick of [KW14]. As can be noticed, the first term, which acts as a regularizer , only takes into account the training dataset part of D KL ´ P b 2 n U | X,W e p U , U 1 | X , X 1 , W e q › › › Q ¯ , as the test set is not available. The second term attempts to maximize the relevance of U for prediction. Intuitively, it seeks to find the best decoder among the possible choices, i.e., the one that minimizes the empirical risk.

A popular choice for Q is the multi-dimensional standard Gaussian distribution Q ' N p 0 m , I m q , where I m is the identity matrix, m is the dimension of the latent variable U , and 0 m P R m is the all zero vector. In the original implementation of [AFDM17], for each sample x , the encoder generates the mean µ x ' p µ x, 1 , . . . , µ x,m q P R m and variance σ 2 x ' p σ 2 x, 1 , . . . , σ 2 x,m q P R m of the latent variable U . Then, we let P U | X,W e p U | x, W e q ' N p µ x , diag p σ 2 x qq , where diag p σ 2 x q P R m ˆ m denotes a diagonal matrix whose diagonal elements are denoted by the vector σ 2 x . This means that the latent variable for the input x is generated according to U ' N p µ x , diag p σ 2 x qq . Hence, the objective function to minimize becomes

where b is the size of a mini-batch of training samples z 1 , . . . , z b , z i ' p x i , y i q . Moreover, (13) is repeated iteratively over multiple mini-batches until the convergence of the representation learning model.

D.2 Lossless CDVIB objective function

We describe here another learning approach which, unlike the VIB, uses a data-dependent prior. This prior, coined Category-Dependent VIB (CDVIB) is again factorized as a product of 2 n scalar Gaussian priors, i.e., Q ' ś i Pr 2 n s Q i . Each of these scalar priors Q i is chosen among one of the K ˆ M Gaussian priors (centers) M priors per each label. More precisely, for each label k P r K s , we consider M P N priors

defined over R m , where the superscript t P N represents the optimization iteration.

Unlike in the VIB approach, the mean and variance of the scalar priors are updated during the training process. Firstly, the vectors µ p t q k,r and σ p t q k,r 2 are initialized respectively as m -dimensional vectors of zeros and ones. Next, these initialized vectors are updated at each iteration using a procedure defined below.

For any sample z ' p x, y q with label y ' k and for any iteration t ě 1 , let r P r M s denote the index of the category-dependent prior (center) which is the closest to N p µ x , diag p σ 2 x qq in terms of the KL-divergence

Suppose that the picked mini-batch at iteration t is B t ' t z 1 , . . . , z b u . For each k P r K s and r P r M s , let

and let b k,r ' | I k,r | . Now, if b k,r ‰ 0 , update µ p t q k,r and σ p t q k,r as

where α P r 0 , 1 s is a coefficient that smoothens the evolution of the mean and variance, and also implicitly takes into account the effect of the ghost dataset S 1 appearing in the bounds of Theorems 4 and 7. The optimal value of α , among others, depends on the mini-batch size b and the number of centers M . 8

Finally, the considered objective function at iteration t in the Lossless CDVIB approach is

D.3 Lossy CDVIB objective function

Inspired by the lossy compression and bounds introduced in our work, we consider the MDL of the 'perturbed' latent variable, while passing the un-perturbed latent variable to the decoder. More precisely, as before, we consider the log loss for evaluation of the relevance of U in the decoder, i.e.,

For the regularizer, we first consider the perturbed U as

where Z 1 and Z 2 are independently drawn from the same distribution N p 0 m , I m q . Note that we chose

Hence, given p X,W e q , ˆ U 1 ' N p µ X , I m q is independent from ˆ U 2 ' N p 0 m , diag p σ 2 X qq . Let us define two sets of priors Q 1 : ' t Q 1 ,k,r u k Pr K s ,r Pr M s over ˆ U 1 and Q 2 : ' t Q 2 ,k,r u k Pr K s ,r Pr M s over ˆ U 2 . Next, for each i P r b s , we select two priors Q 1 ,i P Q 1 and Q 2 ,i P Q 2 , in a manner that will become clear in the following. Denote the induced prior for ˆ U ' ˆ U 1 ` ˆ U 2 , where ˆ U 1 ' Q 1 ,i and ˆ U 2 ' Q 2 ,i by Q i . Then, the KL divergence of P ˆ U | X,W e p ˆ U | X,W e q and Q i , can be upper bounded by

We use this upper bound for our regularizer. To make things more formal, for each label k P r K s , we consider 2 M priors, M P N ,

defined over R m , where the superscript t P N represents the optimization iteration.

Similar to lossless CDVIB, firstly, the vectors µ p t q k,r and σ p t q k,r 2 are initialized respectively as m -dimensional vectors of zeros and ones. Next, these initialized vectors are updated at each iteration using a procedure defined below.

For any sample z ' p x, y q with label y ' k and for any iteration t ě 1 , let r P r M s denote the index of the category-dependent prior (center) which has the smallest distance to p Q p t q 1 ,k,r , Q p t q 2 ,k,r q in a sense that minimizes the RHS of (17). In other words, r p t q z is equal to

Then, the mean and variances of the priors are updated exactly similarly to Lossless CDVIB. For completeness, we repeat this procedure here. Suppose that the picked mini-batch at iteration t is B t ' t z 1 , . . . , z b u . For each k P r K s and r P r M s , let

and let b k,r ' | I k,r | . Now, if b k,r ‰ 0 , update µ p t q k,r and σ p t q k,r as

where α P r 0 , 1 s is a coefficient that smoothens the evolution of the mean and variance, and also implicitly takes into account the effect of the ghost dataset S 1 appearing in the bounds of Theorems 4 and 7. The optimal value of α , among others, depends on the mini-batch size b and the number of centers M .

Finally, the considered objective function at iteration t in the Lossy CDVIB approach is

D.4 Datasets

In the experiments, we used CIFAR10 [KH ` 09]. The full dataset was split into a training set with 50,000 labeled images and a validation set with 10,000 labeled images, all of them of size 32 ˆ 32 ˆ 3 . The input images were scaled to have most values between 0 and 1 before being fed to the network.

D.5 Architecture details

The model architecture considered in our experiments is detailed in Table 1. The encoder part of our twostep prediction model is a convolutional network consisting of four convolutional layers followed by two linear layers. We use max-pooling and a LeakyReLU activation function with a negative slope coefficient equal to 0 . 1 . The encoder takes as input re-scaled images and produces parameters µ x and variance σ 2 x of the latent variable of dimension m ' 64 . The latent samples are generated using the reparameterization trick of [KW14]. Next, the produced latent samples are processed by a decoder consisting of one linear layer with a softmax activation function. The decoder outputs a soft class prediction.

Similarly, as in [DKSV20], our evaluated encoder is complex enough in order to make it close to 'a universal function approximator'. On the other hand, we use a simple decoder as in [AFDM17] in order to reduce spurious regularization introduced by the high decoder's complexity and hence to highlight the benefits of our regularizer in terms of generalization performance.

D.6 Implementation and training details

Our prediction model was trained using PyTorch [PGM ` 19] and a GPU Tesla P100 with CUDA 11.0. All weights were initialized using the default PyTorch Xavier initialization scheme [GB10] with all biases initialized to zero. The Adam optimizer [KB15] ( β 1 ' 0 . 5 , β 2 ' 0 . 999 ) was used with an initial learning rate of 10 ´ 4 and an exponential decay of 0.97. The batch size was equal to 128 throughout the whole experiment. The code used in the experiments is available at https://github.com/PiotrKrasnowski/ MDL_and_Generalization_Guarantees_for_Representation_Learning .

During the training phase, we jointly trained the encoder and the decoder parts for 200 epochs using either the standard Gaussian prior of the traditional VIB objective function or our CDVIB objective functions. As in [AFDM17], we generated one latent sample per image during training and 12 samples during testing.

D.7 Numerical findings

Figure 3 displays the training and test performance of our model for a wide range of parameters β . It could be noticed that the best test accuracy with our Lossy CDVIB objective function is 65% (for β ' 0 . 01 ), which is about 2.5% better than the best test accuracy for VIB (62.5% for β ' 0 . 005 ). In the case of Lossless CDVIB, the best achieved test accuracy is 64% for β ' 1 e ´ 5 .

(a) Accuracy.

(b) Log-likelihood.

1.0

0.0

0.8

-0.5

Log likelihood

0.6

-1.0

0.4

standard VIB,train

-1.5

standard VIB,train

standardVIB,test

standardVIB,test

0.2

-2.0

losSyCDVIB,test

lossless CDVIB,train

lossless CDVIB,train

lossless CDVIB,test

lossless CDVIB,test

0.0

2.5

0

1

10

0

le-71e-6le-5

le-4le-3le-2le-l

1

10

Beta

Beta

E Proofs

E.1 General proof techniques

Most of our proofs contain two main steps.

Technically speaking, the first step uses Donsker-Varadhan's variational representation lemma, to change the measure. As an example, in proof of Theorem 1, the first step gives

that changes the measure from

This has a particular meaning and intuition in our fixed-size compressibility approach. Indeed, one can show that there exists a proper sequence of compression books , with size | ˆ Y m | ď e mR , where

This means that by this step, we fix a suitable sequence of compression books such that with high probability (that goes to 1 as m Ñ8 ), one can find the sequence of predicted labels in this codebook. Using DonskerVaradhan's change of measure is a shortcut to this, as previously explained in [SGRS22, Appendix B.1.] in the context of hypothesis compression. Then, we consider the union bound over all elements of this codebook in the next step. For tail bounds, similar interpretations hold, but this time with variable-size compressibility notion, as shown in [SZ23], in the context of hypothesis compression.

The second step can be seen as bounding the generalization error for every element of such codebook. This step is achieved in quite different manners for the proofs of different results of the paper, as follows in the coming sections. Note that in this step, the predicted labels have distributions according to the prior Q p ˆ Y n , ˆ Y 1 n i | Y n , Y 1 n q , rather than the one induced by the learning algorithm, i.e., P b 2 n ˆ Y | X,W p ˆ Y n , ˆ Y 1 n | X n , X 1 n , W q .

E.2 Proof of Theorem 1

E.2.1 Part i.

Proof. We first show that

Let Q 1 i , be the set of conditional priors Q 1 that can be written as

ȷ

for some arbitrary distribution Q 1 1 . Here Y 2 n and ˆ Y 2 n are the concatenations of the vectors p Y , Y 1 q and p ˆ Y , ˆ Y 1 q , respectively. It is easy to verify that Q i ' Q 1 i . Hence, by denoting P p ˆ Y , ˆ Y 1 | Y , Y 1 q : '

'

to

This completes the proof of showing the equality in (2).

Now, we proceed to prove the upper bound. Let Q be any fixed type-I symmetric conditional prior on p ˆ Y n , ˆ Y 1 n q given p Y n , Y 1 n q . We show that

where again X 1 n , X n , W ' P X 1 n | Y 1 n P X n ,W | Y n . We start the proof by applying the change of measure using Donsker-Varadhan's variational representation (step p a q below):

λ E S,W r gen p S, W qs

Now, we bound the second term. Consider the notation Y i,j P Y 2 n and ˆ Y i,j P Y 2 n for i P r n s and j P t 1 , 2 u . Denote j c : ' ✶ t j ' 1 u ` 1 . Furthermore, for every i P r n s , let K i be a random variable that takes values uniformly over t 1 , 2 u -The variables t K i u are assumed to be mutually independent. Let the complement variable K c i be equal to 2 if K i,j ' 1 and 1 otherwise.

where p a q is concluded by symmetry of Q and p b q by inequality e x ` e ´ x 2 ď e x 2 { 2 . Combining this with (22) yield

Letting

completes the proof.

E.2.2 Part ii.

Proof. The proof of this proposition follows similarly as proof of Theorem 5 (with ϵ ' 0 ), and avoided for brevity. Note that similar to Theorem 5, by noting that with probability at least p 1 ´ δ q , ˆ L p S 1 , W q ě L p W q ´ a log p 1 { δ q{p 2 n q , one can also establish a tail bound on gen p S, W q .

E.3 Proof of Lemma 1

Proof. Parts i. and ii. can be easily verified numerically. To show Part iii., take the derivative with respect to x . This derivative, i.e., log ´ 2 ´ x ´ x 1 x ` x 1 ¯ ´ log ` 1 ´ x x ˘ , is always non-negative for 1 ą x ą x 1 . For Part iv. the second partial derivative of h D p x, x 1 q with respect to x is equal to 1 x p 1 ´ x q ´ 2 p x ` x 1 qp 2 ´ x ´ x 1 q which is always positive for 0 ď x, x 1 ď 1 . Finally, we show the convexity with respect to both variables x and x 1 simultaneously. The Hessian of the function h D p x, x 1 q equals

We show the eigenvalues of this symmetric matrix is always non-negative, and hence H is positive semidefinite. This completes the proof.

Solving

reduces to solving

where a : ' 2 p x ` x 1 qp 2 ´ x ´ x 1 q . The roots of this equation are

It is straightforward to verify that both roots are always non-negative, which completes the proof.

E.4 Proof of Theorem 3

Proof. We first show that

Let Q 1 ii , be the set of conditional priors Q 1 that can be written as

for some arbitrary (and not necessarily symmetric distribution Q 1 1 . Here Y 2 n and ˆ Y 2 n are the concatenations of the vectors p Y , Y 1 q and p ˆ Y , ˆ Y 1 q , respectively. It is easy to verify that Q ii ' Q 1 ii . Hence, by denoting

This completes the proof of showing the equality in (28).

Now we proceed to show the upper bound on nh D ´ E W r L p W qs , E S,W ' ˆ L p S, W q ı¯ . Let Q be any fixed type-II symmetric conditional prior on p ˆ Y n , ˆ Y 1 n q given p Y n , Y 1 n q . We show that for n ě 10 ,

where Y n , Y 1 n ' µ b 2 n Y and S 1 , S, W ' P S 1 | Y 1 n P S,W | Y n . For ease of notation, denote

1 n

Now, we compute the last term, which does not depend on W anymore. Suppose that Unif p 2 n q is a distribution that picks uniformly n indices among indices 2 n indices, i.e., the probability of each one is 1 p 2 n n q . We denote such indices by T ' p T 1 , . . . , T n q , and the corresponding distribution by Unif p 2 n q . For a vector Y 2 n of length 2 n , we denote the elements corresponding to n indices picked by T as Y 2 n T ' p Y 2 n T 1 , . . . , Y 2 n T n q . We denote by T c ' p T c 1 , . . . , T c n q the other remaining n elements in 1 , . . . , 2 n that are not picked by T . We denote Y T n,c ' p Y T c , . . . , Y T c q . Then,

Let V be a random variable indicating V : ' ř i Pr 2 n s ✶ t Y i ‰ ˆ Y i u in the sequence Y 2 n . Then, we consider different cases for V and show that for n ě 10 . This completes the proof.

We use the following lemma repeatedly in the rest of the proof.

Lemma 2 ([Gal68, Exercise 5.8.a]) . For j ě 1 and n ´ j ě 1 , where j, n P N ,

Now, i. If V P r 1 , n s :

where p a q is deduced using Lemma 2, p b q is due to inequality 1 ? xy ď 1 2 x ` 1 2 y for any x, y ą 0 , p c q by the upper bound on the Harmonic series, and p d q holds for n ě 2 .

ii. If V P r n ` 1 , 2 n ´ 1 s :

where

and the infimum is over all P ˆ W e | S such that for ˆ W ' p ˆ W e , W d q ,

Proof. We prove the theorem for ˆ W ' W and ϵ ' 0 . The general result follows by the distortion criterion and applying the theorem on gen p S, ˆ W q . We start the proof similar to the proof of Theorem 1; but with a different expansion of the probability distribution.

λ

where

Now, we bound the last term. Let

where p a q is deduced using Lemma 2, p b q is due to inequality 1 ? xy ď 1 2 x ` 1 2 y for any x, y ą 0 and by verifying the first term for V ' n ` 1 numerically, p c q by the upper bound on the Harmonic series, and p d q holds for n ě 10 .

E.5 Proof of Theorem 4

Let Q be a conditional prior over U 2 n given X 2 n , Y 2 n and ˆ W e P W e , which is symmetric in the following sense: Q ´ p u π p 1 q , . . . , u π p 2 n q q ˇ ˇ p x 1 , . . . , x 2 n q , p y 1 , . . . , y 2 n q , ˆ w e ¯ remains the same for all permutations π : r 2 n s ÞÑ r 2 n s that preserves the label, i.e., y π p i q ' y i . Then, we show that for the K -classification learning task,

Then, the last term in (34) can be written as

Note that since we have P ˆ Y | U,W d ,X,Y ' P ˆ Y | U,W d , it can be easily verified that the distribution P Q ´ ˆ Y n , ˆ Y 1 n | S, S 1 , W ¯ is symmetric with respect to all permutations π that preserve the labels of Y .

For each s, s 1 , let f : r n s Ñ r n s , i P r n s and k : r n s Ñ r n s , i P r n s be permutations of indices of s and s 1 , respectively, where Y f i ' Y 1 k i for i ď T and Y f i ‰ Y 1 k i for i ą T , and where T equals n ´ n 2 } ˆ p s ´ ˆ p s 1 } 1 . Here, ˆ p s and ˆ p s are empirical distributions of Y in s and s 1 , respectively. We denote p ˆ y n , ˆ y 1 n q by ˆ y 2 ˆ n , where @ i P r n s , ˆ y 1 ,i ' ˆ y 1 i and ˆ y 2 ,i ' ˆ y i .

Consider the binary random variable K i taking value as p 2 , f i q or p 1 , k i q with probability 1 { 2 (independent of other j ‰ i ). Denote the complementary choice as K c i , i.e., K c i ' p 2 , f i q iff K i ' p 1 , k i q .

Then, due to the particular symmetry of Q , and by using shorthand notation

we have

where p a q is deduced from Lemma 3, conditioned that λ { n ă 1 . 36 .

Now, combining this with (34), and letting λ ' λ ˚ ' a n p 2 B ` K ` 2 q{ 4 , the expectation of generalization error is upper bounded by

if λ ˚ { n ă 1 . 36 . Note that if 1 . 36 ă λ ˚ { n ' b p 2 B ` K ` 2 q 4 n , then

c

2

`

K

n

'

4

p

λ

{

n

q ą

1

.

Since generalization error is always bounded by 1, hence, this bound always holds.

2

p

B

`

2

q

˚

Lemma 3. Suppose that Y n and Y 1 n are 2 n i.i.d. instances of Y ' p P r K s . Then, if λ { n ă 0 . 68 , then,

The lemma is proved in Appendix E.9 using results of [Dev83].

E.6 Proof of Theorem 5

Let Q be any fixed symmetric prior on p ˆ Y n , ˆ Y 1 n q that could depend on p X n , Y n , X 1 n , Y 1 n q . We show that for any ϵ P R and δ P R ` with probability at least p 1 ´ δ q over choices of S and S 1 , we have that E W ' Q r gen p S, W qs is upper bounded by

where the infimum is over all P ˆ W | S that satisfy

Proof. Consider a distribution P ˆ W | S that satisfies (37). Denote λ ˚ ' 2 n ´ 1 4 and

Furthermore, we use the shorthand notations ˆ p : ' P b 2 n ˆ ˆ p ˆ Y n , ˆ Y 1 n | X n , X 1 n , ˆ W q . Then,

where p a q holds by Hoeffding inequality and using the fact that for arbitrary random variables U, V and constants a, b P R , P p U ` V ą a ` b q ď P p U ą a q ` P p V ą b q , p b q by applying the Jensen inequality on the convex function f p x q ' x 2 , p c q by using the distortion function (37) and since the loss is bounded by one, p d q by using the Donsker-Varadhan's variational representation lemma, p e q is shown in the following, and p f q by Markov inequality.

Hence, it remains to show the step p e q . To this end, it is sufficient upper bound

by log p ? 2 n q . Note that this terms equals

where p a q is concluded by symmetry of Q 1 and p b q is deduced since

is 1 { ? n -subgaussian process and hence

due to [Wai19, Theorem 2.6.IV.]. This completes the proof.

E.7 Proof of Theorem 6

Let Q be any fixed symmetric prior on p ˆ Y n , ˆ Y 1 n q that could depend on p X n , Y n , X 1 n , Y 1 n q . Then, for any δ P R ` ,we show that with probability at least p 1 ´ δ q over choice of p S, S 1 , W q ' P S 1 P S,W ,

Proof. The proof is a combination of proofs of Theorems 3 and 5. Denote the RHS of the bound in the theorem as ∆ p S, S 1 , W q .

First, note that

where p a q follows by (41), p b q holds since

E.8 Proof of Theorem 7

Let Q be a type-III symmetric conditional prior over U 2 n given X 2 n , Y 2 n and W e P W e , namely, Q ´ p u π p 1 q , . . . , u π p 2 n q q ˇ ˇ p x 1 , . . . , x 2 n q , p y 1 , . . . , y 2 n q , w ¯ remains the same for all permutations π : r 2 n s ÞÑ r 2 n s that preserves the label, i.e., y π p i q ' y i for i P r n s . Then, for any λ P R ` , we show that for K -classification learning task, with probability at least p 1 ´ δ q over p S, S 1 , W q ' P S 1 P S,W ,

The poof of the second part follows trivially using Hoeffding's inequality.

Proof. Note that whenever λ { n ě 1 . 36 , the bound trivially holds. Hence, assume that λ { n ă 1 . 36 . Denote the RHS of the bound of Part i. as ∆ p S, S 1 , W q . We use the shorthand notation p 1 : ' P b 2 n U | X,W e p U n , U 1 n | X n , X 1 n , W e q and p 2 : ' P b 2 n ˆ Y | U,W d p ˆ Y n , ˆ Y 1 n | U n , U 1 n , W d q .

Note that p S, S 1 , W q ' P S 1 P S,W . Now,

where p a q by using the Donsker-Varadhan's variational representation lemma, p b q since due to proof of Theorem 4 (in Appendix E.5),

when λ { n ă 1 . 36 , and p d q by Markov inequality. This completes the proof.

E.9 Proof of Lemma 3

Proof. The proof takes its main elements from [Dev83]. Let Y 1 , Y 2 , . . . and Y 1 1 , Y 1 2 , . . . be i.i.d. realizations of Y ' p P r K s . Denote P Y p k q ' p k for k P r K s . Let N and N 1 be two independent Poisson random variables with mean n . Let U i and U 1 i , i P r K s , be number of occurrences of value k among Y r n s and Y r 1 n s , respectively. Moreover, let V i and V 1 i , i P r K s , be number of occurrences of value k among Y r N s and Y r 1 N 1 s , respectively. As shown in [Dev83], V 1 , . . . , V K and V 1 1 , . . . , V 1 K are independent Poisson random variables with means E r V k s ' E ' V 1 k ‰ ' np k , for k P r K s . In addition, p U 1 , . . . , U K q and p U 1 1 , . . . , U 1 K q are two independent multinomial p n, p 1 , . . . , p K q random vectors. Then,

where p a q is due to Cauchy-Schwarz inequality.

Recall that for random variable X having the Poisson distribution with mean η , we have E ' e tX ı ' e η p e t ´ 1 q . Now,

and

Combining (44),(45), and (45) yield

where p a q holds due to two inequalities that can be verified numerically: i) for x ă 1 . 36 , e x ă 1 ` x ` x 2 1 . 2 and ii) for x ă 1 . 3 , e x ` e ´ x 2 ă 1 ` 7 12 x 2 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper obtains new generalization bounds of the conditional mutual information-flavor (Steinke and Zakynthinou, 2020), where the bounds depend only on the predictions that the algorithm induces on the supersample. These bounds are phrased more generally in terms of a KL divergence with symmetric priors, analogous to the almost exchangeable priors of Audibert (2004), and are placed in a rate-distortion theoretic perspective. This is further extended to lossy compression and tail bounds. Tighter bounds in terms of a function of the binary entropy are derived, which appear similar to classical PAC-Bayes bounds with the binary KL divergence in the style of Maurer (2004). Throughout, this is related to the representation in the style of the information bottleneck, leading to the main result of the paper: a generalization bound for representation learning algorithms which depends only on the encoder part, allowing for the decoder part to be chosen freely to minimize the training loss without affecting the bound. This allows for priors that are in some senses data-dependent, and the potential benefit of this is demonstrated numerically.\n\n**Post-Rebuttal edit**: I read the author rebuttal, which addressed my questions, in particular regarding related literature. I consequently updated the rating (as stated below).\n\nSTRENGTHS:\n— The new results and their relevance for representation learning are well-established.\n\n— The theoretical analysis is thorough, covering several possible variants and tightenings of the results.\n\n— The presentation is overall very clear and pedagogical, including a very thorough discussion of some related prior work.\n\n— The numerical evaluation nicely illustrates the potential benefit of the proposed methods.\n\nWEAKNESSES:\n— There are some missing connections to previous work (see questions).\n\n— A few of the results appear to be stated incorrectly, and some notation seems undefined (see questions).\n\nQUESTIONS:\n— There appears to be a clear connection to the frameworks of functional and evaluated CMI (f-CMI, e-CMI) of, e.g., Harutyunyan et al (2021) and Hellström and Durisi (2022). Is there a clear distinction between these bounds and your Theorem 1, other than the use of an auxiliary prior (the possibility of which is also noted in the mentioned works)?\n\n— The concept of symmetric priors seems identical to the almost exchangeable priors discussed by Audibert (2004). Is there any difference with these?\n\n— The bounds in terms of the function $h_D$ appear closely related to bounds with the binary KL divergence on the left-hand side, e.g. Maurer (2004) and Seeger (2002) — the properties of the function appear to be very similar. Is there a difference or are the two approaches equivalent?\n\n— Proposition 1: The left-hand side here is a single-draw result, which typically does not lead to bounds in terms of the KL divergence, but instead some form of information density or log-likelihood ratio. This is also possible to derive from your results, but the derivations have to be slightly altered. In the proof of Theorem 4, you instead consider a “PAC-Bayesian” bound which is averaged over the posterior, which naturally leads to bounds in terms of the KL divergence. I assume this is what you meant to write in Proposition 1 as well. As noted by Grunwald et al (2023), such PAC-Bayesian bounds follow immediately from the same derivation as average generalization bounds if they are based on “exponential inequalities” (as yours are). This can be extended to single-draw bounds as well, under more stringent absolute continuity assumptions.\n\nMinor: \n\nAbstract: IB undefined\n\nIntro: $\\text{gen}(W,S)$ and the corresponding notation for population and training losses (from e.g. Proposition 1) appear to be undefined\n\nQuotation mark lines 34, 234, 33\n\nLine 50: A sentence with only citations.\n\nLine 131: KL divergence notation is flipped in LHS of definition\n\nFig. 1 is not on the same page as it is discussed\n\nMargin overflow in Eq. (3), lines 579-581, more in appendix\n\nLine 242: $\\tau*$ instead of $\\tau^*$\n\nLines 266-272: inconsistent conventions for fractions. Can some expressions be moved to equation environments? Maybe this intuition can be expressed more informally and the math can be moved to the appendix where there is space\n\nI assume $[a,b]=\\{a,a+1,\\dots,b\\}$ for integers in some expressions, but I do not think this is defined and it conflicts with the standard use for closed intervals\n\nTheorem 3: The prior is called $P$ in the theorem statement but $Q$ in the equation.\n\nLine 329, 686: Incomplete sentence\n\nReferences:\n\nAudibert, “A better variance control for PAC-Bayesian classification”, 2004\n\nMaurer, “A Note on the PAC Bayesian Theorem”, 2004\n\nHarutyunyan et al, \"Information-theoretic generalization bounds for black-box learning algorithms”, 2021\n\nHellstrom and Durisi, “A New Family of Generalization Bounds Using Samplewise Evaluated CMI”, 2022\n\nSeeger, “PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification”, 2002\n\nGrunwald et al, “Exponential Stochastic Inequality”, 2023\n\nLIMITATIONS:\nPerhaps the discussion of limitations can be expanded, but it is not crucial.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper advances state-of-the-art on information theoretic generalization guarantees for a class of representational learning problems. The contribution is positioned in the framework of information bottleneck approach and builds on the classical work of Blum and Langford for PAC-MDL bounds. The main result is on non-trivial upper bounds on generalization error of a representational learning algorithm in terms of so-called minimum description length of the label. The result is expressed in terms of KL-divergence of a vector of labels and any arbitrary symmetric prior, and resolves the issues related to use of mutual information measure. The aspects related to block coding and lossy compression are new to this type of results.\n\nSTRENGTHS:\nThe writing is adequately clear both in terms of technical precision and intuitive interpretations, where appropriate. The problem is important and the proposed results advance state-of-the-art.\n\nWEAKNESSES:\nPerhaps a weakness of the paper is that it relies on a solid understanding of information theoretic background; for e.g., [GK19]. More importantly, Theorem 5 which is main result on generalization bound for representation learning needs a bit more explanation, both in terms of intuitive/conceptual ideas and technical contribution based on work in the previous section. The proposed result – as the authors claim – resolves issues with the information bottleneck method, but the reader gets limited understanding of how the complexity of latent space and validity of any choice of decoder plays out in practical settings. On the other hand, the discussion in Section 2 could be more pointed – with focus on key novelties (which might help save space and leave room for more extensive discussion in Section 3). It also appears that a tabular comparison of generalization bounds proposed in this paper, relative to the ones based on mutual information would clarify some of the issues regarding positioning of the results. \nUnless I am mistaken, it seems the authors do not formally introduce the key object $\\mathbb{E}[gen(S,W)]$.\n\nQUESTIONS:\n1. Better explain the implications of Theorem 5 relative to generalization based on mutual information. Why *should* the encoder structure matter more than information in latent variable. \n\n2. The connection/interpretation to/of geometric compressibility and how the lossy compressibility plays a role here is not very clear. \n\n3. Is possible to have a concrete example where proposed bounds are shown to be superior to classical bounds based on mutual information?\n\nLIMITATIONS:\nnot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Post-rebuttal**\n\nI thank the authors for their response and have raised my score to an accept.\n\n\nUnderstanding the generalization properties of modern machine learning algorithms is one of the key challenges in statistical learning theory. Focusing on the classification setup, this paper first studies the generalization error in terms of the compressibility of the predicted labels using tools from rate distortion theory. The proposed bounds feature a new KL divergence term that takes into account the structure and \"simplicity\" of the algorithm. The authors then extend the framework to give a generalization bound for information bottleneck-type representation learning in terms of the compressibility of the latent variable. The authors also empirically demonstrate a practical benefit of their theoretical results for the information bottleneck.\n\nSTRENGTHS:\nIn terms of originality, the generalization guarantee in terms of the compressibility of the representation that takes into account the complexity and structure of the encoder class appear novel. Also unlike existing mutual information-based bounds, the new bounds are not vacuous for deterministic algorithms. The work builds on prior work on PAC-MDL bounds from Blum and Langford (2003). The latter used a bits-back coding argument, while the tools developed in this paper rely on rate distortion theory and block coding arguments building on prior work [SGRS22] and [SZ23].\n\nThe authors also demonstrate a practical benefit of their theoretical results by comparing the performance of the vanilla VIB and a `\"category-dependent\" version, with the latter performing better in terms of generalization error and population risk.\n\nOverall, the writing is clear though the presentation can be improved in certain parts.\n\nWEAKNESSES:\nThe presentation in the main text can be improved in certain parts, e.g., giving more intuition for the proof techniques; see below. Also, in its current form, the paper seems to end abruptly. A separate Conclusions section summarizing the main contributions along with scope for future work / citing open problems can be very useful.\n\nQUESTIONS:\n* What is a high-level intuition for variable-size compressibility for the tail bounds? Likewise, for block covering does compressing multiple instances simultaneously allow for smaller distortion levels? It would definitely help the reader if some of the high-level proof ideas are laid out more clearly either in the main text or supplementary.\n\nLIMITATIONS:\nI do not foresee any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission considers supervised learning problems and derives two types of generalization gap bounds. The first type states that generalization gap can be upper bounded by approximately $\\sqrt{\\mathrm{KL}(P || Q) / n}$, where $P$ is the average (wrt data and training stochasticity) of predicted label distribution on $n$ training examples and $n$ test examples conditioned on the labels of the $2n$ examples, while $Q$ is a fixed **symmetric** prior distribution for predicted labels on $2n$ examples given $2n$ labels. $Q$ is symmetric when $Q(\\hat{y_1},\\ldots,\\hat{y_{2n}} | y_1,\\ldots,y_{2n}) = Q(\\hat{y_{\\pi(1)}},\\ldots,\\hat{y_{\\pi(2n)}} | y_{\\pi(1)},\\ldots,y_{\\pi(2n)})$ for any permutation $\\pi$. In a sense, the symmetricity means that $Q$ puts a distribution on $\\hat{\\mathcal{Y}}^{2n}$ given $y_1,\\ldots,y_{2n}$ without \"knowing\" which $n$ of the $2n$ examples belong to the training set.\n\nThe second type of generalization bounds are designed for classifiers which have a representation part (an encoder) and a classification head. The main term in these type of bounds is also of the form $\\sqrt{\\mathrm{KL}(P || Q) / n}$, but where representations play the role of predicted labels. Again there are $n$ training, $n$ testing examples, and a symmetric prior $Q$ but now over representations. Notably, this type of bounds do not depend on the classification head, they only depend on the encoder.\n\nFor both type of generalization bounds, both in-expectation and in-probability bounds are derived. Furthermore, the authors derive tighter bounds where the mismatch between empirical and population risks is measured by a convex function $h:[0,1] \\times [0,1] \\rightarrow [0,2]$ instead of the $|x-y|$ function. Finally, to avoid KL divergence becoming infinite when working with continuous labels or continuous representations, the authors derive bounds where one seeks a \"quantized\" algorithm that has a similar generalization gap but non-infinite KL divergence.\n\nSTRENGTHS:\n**Strength #1: Relevance.** Understanding how some properties of learned representations affect generalization is an important and highly relevant direction for the NeurIPS community. The authors present a good summary of the criticism of information-bottleneck compression ideas in explaining generalization and propose to consider instead the joint compressibility of training and test representations.\n\n**Strength #2: Originality.** To my best knowledge, the bounds that depend on representation complexity (results of Section 3) are novel. In some sense, these results suggest a new type of information bottleneck for representation learning.\n\n**Weaker strength: Soundness.** The results of this submission are sound as far as I checked. However, in some places I couldn't follow the proofs very carefully as some steps were not detailed enough or the notation was confusing (please see my comments below).\n\nWEAKNESSES:\n**Weakness #1: Hard to judge the significance.** While the overall direction of obtaining generalization guarantees via joint compressibility is interesting and this submission makes a good step towards exploring that direction, it is hard to judge how significant are the main results of Section 3.\n- [Estimation] I don't see a good way of estimating the bound of Theorem 5. In the case of continuous representations, how can one take an infimum over learning algorithms that have similar generalization gap?\n- [Experiments] The experiments do not help to judge whether the KL term in the bound can be successfully used as a regularization.\n\t* [E1] Line 608, \"As in the VIB approach, inevitably, we only restrict the optimization to the training set.\": This seems to go against the spirit of the approach employed throughout the paper. Namely, as I understand the symmetricity of $\\boldsymbol{Q}$ can be interpreted as $\\boldsymbol{Q}$ not being aware of the train-test set split. Forcing the encoder to be close to such a distribution $\\boldsymbol{Q}$ forces the encoder to behave similarly for seen and unseen examples. From this one gets a generalization guarantee.\n\t- [E2] Even if one splits an original training set into a new training set and a ghost set to follow the logic of the main text, the KL regularization term will probably be fitted by \"cheating\", in the sense that representations of the new training set and ghost set examples will be different from those of unseen examples. The nature of ghost examples (to be not used during the training) makes it impossible to use the KL divergence term of Theorem 5 as a regularization.\n\t* [E3] Lines 612-617: $\\boldsymbol{Q}$ will not be symmetric in this construction, as it is constructed using statistics of *training data*. Consequently, the KL divergence can become optimistically small (when training examples are \"memorized\").\n\t- [E4] Lines 667-669, \"It can be observed that our scheme outperforms VIB in terms of both generalization error and population risk, for all values of the Lagrange multiplier $\\beta$\": Since the two objectives are different one should not compare then for a fixed $\\beta$. Instead, we can consider tuning $\\beta$ for each of them independently to minimize test error. We see that in that case there is no significant difference between the two approaches. Similarly, there is no significant difference if we compare generalization gaps for a fixed level of small test error.\n- [Applications] The submission would benefit significantly from an analytical evaluation/bounding of the bound of Theorem 5 in some tractable settings (maybe a two-layer neural network on Gaussian data). This would allow to see if the bound is powerful enough to produce meaningful results in some settings.\n\n**Weakness #2: Clarity & Presentation.** This submission would benefit from improvements in terms of clarity and presentation. It was hard for me to read and understand the details. There are many small mistakes, the notation is cumbersome, and some parts of derivations are not detailed enough. Please see more detailed comments and suggestions below.\n\n**Weakness #3: Missing important related work.** This submission does not cite well the literature of information-theoretic generalization bounds initiated by the works Russo and Zou [1] and Xu and Raginsky [2], and developed further by many others. This line of work is relevant for multiple reasons.\n1. The general technique of deriving bounds in this work and in the mentioned line of work is the same (using Donsker-Varadhan inequality and bounding a cumulant-generating function).\n2. The general idea of measuring how different a learned function acts on training and testing examples has been used in many follow-ups on [3].\n3. The idea of symmetric priors appears in [4] under the name \"almost exchangeable data-dependent priors\". In their bound there is also a term of the form $\\mathrm{KL}(P||Q)$, where $P$ is the output distribution (over hypotheses) after training on $n$ training examples, while $Q$ is a symmetric prior over hypotheses given $2n$ examples. **Important:** when considering in-expectation bounds, the optimal $Q$ is the average of output distributions over all possible $2^n$ training/test splits. The KL divergence for this optimal prior becomes the conditional mutual information (CMI) $I(W; J | \\tilde{Z}_0, \\tilde{Z}_1)$, where $J \\sim \\mathrm{Uniform}\\\\{0,1\\\\}^n$ is the training-test split, $W$ is the output of the learning algorithm when splitting with $J$, and $\\tilde{Z}_0$ and $\\tilde{Z}_1$ are the $n$ training and $n$ testing examples respectively.\n4. While in works [2], [3], and [4] the bounds depend on some kind of information captured by the output of a learning algorithm (i.e., the output hypothesis), in [5] there are bounds that measure information in *predicted labels* in the random subsample setting of Steinke and Zakynthinou [3]. Given the important part of the last point, the result of Theorem 1 can be seen as an f-CMI bound written differently. The only minor difference is that in f-CMI bounds there is conditioning on $\\tilde{Z}_0$ and $\\tilde{Z}_1$ instead of $\\tilde{Y}_0$ and $\\tilde{Y}_1$ (i.e., only the label parts), but it is easy to see that the proof does not use the input parts ($\\tilde{X}_0$ and $\\tilde{X}_1$).\n5. It is also worth to mention that CMI (or f-CMI or e-CMI) bounds are usually improved by measuring information with individual pairs separately (variations of the technique of Bu et al. [6]), and by conditioning on an individual pair of train/test examples (e.g., [7]). These two techniques together produce f-CMI bounds of form $\\sum_{i=1}^n \\sqrt{I(\\hat{Y_{0,i}}, \\hat{Y_{1,i}}; J_i | Y_{0,i}, Y_{1,i})}$ that can be much tighter than the bound of Theorem 1 (the equivalent of the f-CMI bound when measuring information in all predictions at once).\n\n\n**References**\n\n[1] D. Russo and J. Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory, 2019.\n\n[2] A. Xu and M. Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. NeurIPS 2017.\n\n[3] T. Steinke and L. Zakynthinou. Reasoning About Generalization via Conditional Mutual Information. COLT 2020.\n\n[4] P. Grunwald, T. Steinke, L. Zakynthinou. Pac-bayes, Mac-bayes and conditional mutual information: Fast rate bounds that handle general VC classes. COLT 2021.\n\n[5] H. Harutyunyan, M. Raginsky, GV. Steeg, and A. Galstyan. Information-theoretic generalization bounds for black-box learning algorithms. NeurIPS 2021.\n\n[6] Y. Bu, S. Zou, and V. V. Veeravalli. Tightening mutual information-based bounds on generalization error. IEEE Journal on Selected Areas in Information Theory, 2020.\n\n[7] B. Rodríguez-Gálvez, G. Bassi, R. Thobaben, and M. Skoglund. On random subset generalization error bounds and the stochastic gradient Langevin dynamics algorithm. IEEE ITW 2021.\n\nQUESTIONS:\n1. Lines 45-48: What exact result is being referred here? The mutual information between two variables cannot be seen directly as some sort of description length (just consider two variables with 0 mutual information). \n2. Section 2.1 Compressibility framework: The concepts introduced in this section are not used in the main text (specifically the block quantities). I recommend moving this subsection to Appendix C. This way it becomes easier to read the main text and the already complicated notation does not become more complicated. The only idea that I recommend keeping from this section is that if one can encode $\\hat{Y}^{2n}$ with short code on average, then one should expect good generalization. Similarly, the sentence of lines 209-211 should be moved too.\n3. Theorem 1: $\\text{gen}(S,W)$ should be defined.\n4. Equation (3): The training algorithm can act differently on training examples with different indices. It is not necessarily true that the RHS of eq. (3) can be written as $n$ times the quantity for a single example.\n5. Theorem 2: $\\mathcal{L}(W)$ and $\\hat{\\mathcal{L}}(S,W)$ should be defined.\n6. Line 295: $\\boldsymbol{Q}$ instead of $P$ for the prior.\n7. Theorem 4: Why is the idea of a posterior $Q$ introduced here, can't it be just $P_{W|S}$ and $P_{\\hat{W}|S}$? As it has been done before this point, the letter $Q$ would be good to use only for priors. In the equation after line 311, it should be $\\hat{W} \\sim P_{\\hat{W}|S}$ instead of $\\hat{W} \\sim Q$. Also, in eq. (6) it would be good to write it as a difference of two expectations. In that case it will be clear what distributions $W$ and $\\hat{W}$ follow.\n8. Line 802: It should be $\\boldsymbol{Q}$ (the symmetric prior) instead of $Q$ (the posterior over $W$). Similarly in all $\\Delta(S,Q)$ terms. Also $\\Delta(S,Q)$ should be $\\Delta(S,S',Q)$ as it depends on $S'$ too.\n9. Line 350: Why is the property of preserving labeling important for the permutation $\\pi$ in the definition of a symmetric prior? I recommend adding more details of the transition at line 839 where the symmetricity of $\\boldsymbol{Q}$ is used. \n10. Line 831-832, \"it can be easily verified that this distribution is symmetric\": Which distribution is referred here?\n11. Line 833: It should be $k : [n] \\rightarrow [n]$.\n12. Line 834: $||\\hat{p_s} - \\hat{p_{s'}}||_1 \\in [0, 2]$, so $T$ can become negative. It should be divided by 2.\n13. Lines 833-834: The construction of the permutation $k$ is not always possible. It might not be possible to map the *first* $T$ samples in $Y$ to another $T$ equal samples in $Y'$.\n14. The technique of taking an infimum over close enough \"quantized\" algorithms is a step that can be applied no matter whether the bound depends on joint compressibility or $\\hat{I}(X; U)$ or something else. Therefore, I feel that one should not claim advantage over other results in terms of being applicable also to continuous variables when the advantage is a result of applying this general technique (not due to something inherent to the approach).\n15. It would be good to have a small discussion on the fact that representation learning has many goals (e.g., learning representations that are \"useful\" for many tasks). Learning representations that guarantee small generalization gap for a *given* task is not of high importance in common representation learning settings. I don't think that one would be happy to find out that after a heavy pretraining round with a pretraining task $Y$ the network learned a representation $R=\\phi(X)$ such that $R=g(Y)$ for some function $g$.\n\nLIMITATIONS:\nThe limitations are addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}