Title: A unified framework for information-theoretic generalization bounds
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalization bounds

information theory

chaining

PAC-Bayes

couplings
Abstract: This paper presents a general methodology for deriving information-theoretic generalization bounds for learning algorithms. The main technical tool is a probabilistic decorrelation lemma based on a change of measure and a relaxation of Young's inequality in $L_{\psi_p}$ Orlicz spaces. Using the decorrelation lemma in combination with other techniques, such as symmetrization, couplings, and chaining in the space of probability measures, we obtain new upper bounds on the generalization error, both in expectation and in high probability, and recover as special cases many of the existing generalization bounds, including the ones based on mutual information, conditional mutual information, stochastic chaining, and PAC-Bayes inequalities. In addition, the Fernique--Talagrand upper bound on the expected supremum of a subgaussian process emerges as a special case.

A Unified Framework for Information-Theoretic Generalization Bounds

Abstract

This paper presents a general methodology for deriving information-theoretic generalization bounds for learning algorithms. The main technical tool is a probabilistic decorrelation lemma based on a change of measure and a relaxation of Young's inequality in L ψ p Orlicz spaces. Using the decorrelation lemma in combination with other techniques, such as symmetrization, couplings, and chaining in the space of probability measures, we obtain new upper bounds on the generalization error, both in expectation and in high probability, and recover as special cases many of the existing generalization bounds, including the ones based on mutual information, conditional mutual information, stochastic chaining, and PAC-Bayes inequalities. In addition, the Fernique-Talagrand upper bound on the expected supremum of a subgaussian process emerges as a special case.

1 Introduction

The generalization error of a learning algorithm is a useful proxy for evaluating the performance of the learned model on previously unseen data. Formally, it is defined as the expected (absolute) difference between the population risk and the empirical risk of the hypothesis returned by the algorithm. One of the classical methods for estimating the generalization error is via uniform convergence of various empirical processes indexed by the hypothesis class [1,2]. For example, in the analysis of Empirical Risk Minimization, one can estimate the expected generalization error via Rademacher averages, which can be bounded from above using chaining techniques [3].

However, the bounds based on uniform convergence are often too pessimistic and may even become vacuous when the hypothesis space is extremely large, a typical situation with deep neural net models. For this reason, it is preferable to obtain algorithm-dependent generalization bounds that take into account the joint distribution of the training samples and of the output hypothesis. In this context, one capitalizes on the intuition that the generalization ability of a learning algorithm should be related to the amount of information the output hypothesis reveals about the training data. This idea, which has origins in the work on PAC-Bayes methods [4,5], is the basis of the growing literature on information-theoretic generalization bounds, first proposed in [6] and further devoloped in [7-16] and many other works.

In fact, it is possible to effectively combine the information-theoretic approach with the classical framework based on various measures of complexity of the hypothesis class: One can use chaining techniques to successively approximate the hypothesis class by simpler model classes, which can then be analyzed using information-theoretic tools. This methodology, again originating in the PAC-Bayes literature [17], has been developed recently in [18-21]. Our goal in this work is to develop these ideas further by giving a unified framework for information-theoretic generalization bounds, from which many of the existing results emerge as special cases.

1.1 The main idea, informally

The main idea behind our framework is surprisingly simple. We first give an abstract description and then show how it can be particularized to various settings of interest. Let ( X t ) t ∈ T be a centered (zero-mean) stochastic process defined on a probability space (Ω , A , P ) and indexed by the elements of some set T . Let Q be a Markov kernel from Ω to T , i.e., a measurable mapping taking each ω ∈ Ω to a probability measure Q ( ·| ω ) on T . Together, P and Q define a probability measure P ⊗ Q on the product space Ω × T . The mathematical object we would like to study is the expected value

For example, assuming that there exists a measurable map τ ∗ : Ω → T , such that

we can take Q ( A | ω ) := 1 { τ ∗ ( ω ) ∈ A } for all measurable subsets A of T . Then

is the expected supremum of X t , the central object of study in the theory of generic chaining, where ( T, d ) is a metric space and increments X u -X v are 'stochastically small' relative to d ( u, v ) . Alternatively, consider a statistical learning problem with instance space Z , hypothesis space W , and loss function glyph[lscript] : W×Z → R + . Let P Z be the (unknown) probability law of the problem instances in Z . Then we could take Ω = Z n , P = P ⊗ n Z , T = W , and

where L ( w ) := E Z ∼ P Z [ glyph[lscript] ( w,Z )] is the population risk of w . Let Q be a (randomized) learning algorithm that associates to each sample S = ( Z 1 , . . . , Z n ) ∼ P a probability measure Q ( ·| S ) on the hypothesis space W . Then

[

1

n

n

∑

i

=1

is the expected generalization error of Q . In either case, we can proceed to analyze 〈 P ⊗ Q,X 〉 via a combination of the following two steps:

Decorrelation -Wecanremovethe correlations encoded in P ⊗ Q by choosing a convenient product measure P ⊗ µ on Ω × T , so that (roughly)

provided the process ( X t ) t ∈ T is regular enough for the error term to be small. Here, we use the relative entropy (or information divergence) D ( ·‖· ) to illustrate the key idea with a minimum of detail; the precise description is given in Section 3.

Chaining in the space of measures -Since the process ( X t ) t ∈ T is centered and P ⊗ µ is a product measure, we automatically have 〈 P ⊗ µ, X 〉 = 0 even though 〈 P ⊗ Q,X 〉 glyph[negationslash] = 0 . We can therefore interpolate between P ⊗ Q and P ⊗ µ along a (possibly infinite) sequence Q 0 , Q 1 , . . . , Q K of Markov kernels, such that P ⊗ Q K = P ⊗ Q , P ⊗ Q 0 = P ⊗ µ , and the differences 〈 P ⊗ Q k , X 〉 - 〈 P ⊗ Q k -1 , X 〉 are suitably small. Telescoping, we get

For each k , we then apply the decorrelation procedure to the increment process ( X u -X v ) u,v ∈ T , with P as before and with a suitably chosen family of couplings of Q k ( ·| ω ) and Q k -1 ( ·| ω ) . This step can be combined effectively with other techniques, such as symmetrization.

) ]

〈

P

⊗

Q,X

〉

=

E

(

L

(

W

)

-

glyph[lscript]

(

W,Z

i

)

2 Preliminaries

Basic definitions. All measurable spaces in this paper are assumed to be standard Borel spaces. The set of all Borel probability measures on a space X will be denoted by P ( X ) . A Markov kernel from ( X , A ) to ( Y , B ) is a mapping P Y | X : B × X → [0 , 1] , such that P Y | X = x ( · ) := P Y | X ( ·| x ) is an element of P ( Y ) for every x ∈ X and the map x ↦→ P Y | X = x ( B ) is measurable for every B ∈ B . The set of all such Markov kernels will be denoted by M ( Y|X ) .

The product of P X ∈ P ( X ) and P Y | X ∈ M ( Y|X ) is the probability measure P X ⊗ P Y | X ∈ P ( X × Y ) defined on product sets A × B ⊆ X ×Y by ( P X ⊗ P Y | X )( A × B ) := ∫ A P Y | X = x ( B ) P X (d x ) and then extended to all Borel subsets of X × Y by countable additivity. This defines a joint probability law for a random element ( X,Y ) of X × Y , so that P X is the marginal law of X , P Y | X is the conditional law of Y given X , and P Y ( · ) = ∫ X P Y | X = x ( · ) P X (d x ) is the marginal law of Y . The product measure P X ⊗ P Y , under which X and Y are independent, is a special case of this if we interpret P Y as a trivial Markov kernel with P Y | X = x = P Y for all x .

A coupling of µ ∈ P ( X ) and ν ∈ P ( Y ) is a probability measure P ∈ P ( X × Y ) , such that P ( · × Y ) = µ ( · ) and P ( X × · ) = ν ( · ) . We will denote the set of all couplings of µ and ν by Π( µ, ν ) . Let the space X ∪ Y be equipped with a metric d , and let P p , for p ≥ 1 , denote the space of all probability measures ρ on X ∪ Y , for which there exists some z 0 ∈ X ∪ Y such that ∫ X∪Y d ( z, z 0 ) p ρ (d z ) < ∞ . Then the p -Wasserstein distance between µ ∈ P ( X ) ∩ P p and ν ∈ P ( Y ) ∩ P p is given by

(see [22, 23] for details).

L p and L ψ p spaces. The L p ( µ ) norms of f : X → R , for µ ∈ P ( X ) and p ≥ 1 , are defined as

whenever the expectation on the right-hand side exists. We will often use the linear functional notation for expectations, i.e., 〈 µ, f 〉 = ∫ X f d µ .

For p ≥ 1 , define the function ψ p : R + → R + by ψ p ( x ) := exp( x p ) -1 . Its inverse is given by ψ -1 p ( x ) = ( log( x +1) ) 1 /p , where log will always denote natural logarithms. Some useful properties of these two functions are collected in Appendix A of Supplementary Material. The function ψ p arises in the context of controlling the tail behavior of random variables (see [1, 24, 25] for details). The Orlicz ψ p -norm of a real-valued random variable X is defined as

and the tails of X satisfy P [ | X | ≥ u ] ≤ Ke -Cu p for all u ≥ 0 and some K,C > 0 if and only if ‖ X ‖ ψ p < ∞ . The Orlicz space L ψ p is the space of all random variables X with ‖ X ‖ ψ p < ∞ . In particular, if X is σ -subgaussian, i.e., P [ | X | ≥ u ] ≤ 2 e -u 2 / 2 σ 2 for all u ≥ 0 , then ‖ X ‖ ψ 2 ≤ √ 6 σ ; conversely, every X ∈ L ψ 2 is σ -subgaussian with σ ≤ c ‖ X ‖ ψ 2 for some absolute constant c > 0 .

Information-theoretic quantities. The relative entropy (or information divergence) D ( µ ‖ ν ) between two probability measures µ, ν on the same space X is defined as

if µ glyph[lessmuch] ν (i.e., µ is absolutely continuous w.r.t. ν ), and D ( µ ‖ ν ) := + ∞ otherwise. The following inequality will be useful (proofs of all results are in Appendix B of the Supplementary Material):

Proposition 1. If µ glyph[lessmuch] ν , then for any p ≥ 1

The conditional divergence between P V | U , Q V | U ∈ M ( V|U ) given P U ∈ P ( U ) is defined by

The mutual information I ( X ; Y ) := D ( P Y | X ‖ P Y | P X ) and conditional mutual information I ( X ; Y | Z ) := D ( P Y | XZ ‖ P Y | Z | P XZ ) are special cases of the above definition, and the identities

hold whenever all the quantities are finite. See, e.g., [26] for details.

3 The decorrelation lemma

All of our subsequent developments make use of the following decorrelation lemma :

Lemma 1. Let µ, ν be two probability measures on a space X such that µ glyph[lessmuch] ν , and let f, g : X → R + be two nonnegative measurable functions. Then the following inequalities hold:

and

The proof makes extensive use of various properties of ψ p and ψ -1 p . In particular, Eq. (4) is a relaxation of the Young-type inequality xy ≤ ψ ∗ p ( x ) + ψ p ( y ) , where ψ ∗ p ( x ) := sup y ≥ 0 ( xy -ψ p ( y )) is the (one-sided) Lengendre-Fenchel conjugate of ψ p . (We refer the reader to [13] for another use of duality in Orlicz spaces in the context of generalization bounds.)

Every use of Lemma 1 in the sequel will be an instance of the following scheme: Let P X ∈ P ( X ) , Q Y ∈ P ( Y ) , and P Y | X ∈ M ( Y|X ) be given, such that P Y | X = x glyph[lessmuch] Q Y for all x ∈ X . Let ( X,Y, ¯ Y ) be a random element of X × Y × Y with joint law P X ⊗ P Y | X ⊗ Q Y ; in particular, ¯ Y is independent of ( X,Y ) . Furthermore, let f : Y → R + and g : X × Y → R + be given, such that E [ ψ p ( g ( X,y ))] ≤ 1 for all y ∈ Y . Then, applying Lemma 1 conditionally on X = x with µ = P Y | X = x , ν = Q Y , f , and g ( x, · ) , and then taking expectations w.r.t. P X , we obtain

In specific cases, the quantity on the right-hand side can be further upper-bounded in terms of the information divergences D ( P Y | X ‖ Q Y ) using Proposition 1.

4 Some estimates for the absolute generalization error

We adopt the usual set-up for the analysis of (possibly randomized) learning algorithms and their generalization error. Let an instance space Z , a hypothesis space W , and a nonnegative loss function glyph[lscript] : W×Z → R + be given. A learning algorithm is a Markov kernel P W | S from the product space Z n into W , which takes as input an n -tuple S = ( Z 1 , . . . , Z n ) of i.i.d. random elements of Z with unknown marginal probability law P Z and generates a random element W of W . We define the empirical risk and the expected (or population ) risk of each w ∈ W by

where P n is the empirical distribution of S , and the pointwise generalization error by

It will also be convenient to introduce an auxiliary n -tuple S ′ = ( Z ′ 1 , . . . , Z ′ n ) ∼ P ⊗ n Z , which is independent of ( S, W ) ∼ P ⊗ n Z ⊗ P W | S . We will use ˜ S to denote the pair ( S ′ , S ) and write L ′ n ( w ) for the empirical risk of w on S ′ .

As a first illustration of our general approach, we show that it can be used to recover some existing results on the generalization error, including the bounds of Xu and Raginsky [7] involving the mutual information and the bounds of Steinke and Zakynthinou [10] involving the conditional mutual infornation. We start with the following estimate on the expected value of | gen( W,S ) | :

Theorem 1. Assume the random variables glyph[lscript] ( w,Z ) , w ∈ W , are σ -subgaussian when Z ∼ P Z . Let a learning algorithm P W | S be given. Then, for any Q W ∈ P ( W ) ,

where the expectation on both sides is w.r.t. P S ⊗ P W | S = P ⊗ n Z ⊗ P W | S .

The key step in the proof is to apply the decorrelation lemma, conditionally on S , to µ = P W | S , ν = Q W , f ( w ) = σ √ 6 /n , and g ( w ) = | gen( w,S ) | σ √ 6 /n . The same subgaussianity assumption was also made by Xu and Raginsky [7]. Minimizing the right-hand side of (6) over Q W , we recover their generalization bound up to a multiplicative constant and an extra O (1 / √ n ) term (which is unavoidable since we are bounding the expected absolute generalization error):

Corollary 1. Under the assumptions of Theorem 1,

A notable shortcoming of Theorem 1 and Corollary 1 is that they yield vacuous bounds whenever the mutual information I ( W ; S ) is infinite, which will be the case, e.g., when the marginal probability laws P Z and P W are nonatomic (i.e., assign zero mass to singletons) and the learning algorithm is deterministic. To remove this drawback, we will use an elegant auxiliary randomization device introduced by Steinke and Zakynthinou [10].

Let ε = ( ε 1 , . . . , ε n ) be an n -tuple of i.i.d. Rademacher random variables, i.e., P [ ε i = ± 1] = 1 / 2 , independent of ˜ S . For each i let ˜ Z 1 i := Z i and ˜ Z -1 i := Z ′ i and let ¯ P = ¯ P ˜ SεW be the joint probability law of ( ˜ S, ε, W ) , such that ¯ P ˜ Sε = P ˜ S ⊗ P ε and ¯ P W | ˜ Sε := P W | ˜ S ε where S ε := ( ˜ Z ε 1 1 , . . . , ˜ Z ε n n ) . In other words, under ¯ P , ˜ S and ε are independent and have their respective marginal distributions, while W is generated by feeding the learning algorithm P W | S with the tuple ˜ S ε . Consequently, W is independent of ˜ S -ε = ( ˜ Z -ε 1 1 , . . . , ˜ Z -ε n n ) . Then, letting P be the joint law of ( ˜ S, W ) , we have

Thus, all the analysis can be carried out w.r.t. ¯ P , as in the following:

Theorem 2. Assume there exists a function ∆ : Z × Z → R + , such that | glyph[lscript] ( w,z ) -glyph[lscript] ( w,z ′ ) | ≤ ∆( z, z ′ ) for all w ∈ W and z, z ′ ∈ Z . Then for any Markov kernel Q W | ˜ S with access to ˜ S but not to ε we have

where ‖ ∆(˜ s ) ‖ glyph[lscript] 2 := (∑ n i =1 ∆( z i , z ′ i ) 2 ) 1 / 2 .

The same assumption on glyph[lscript] was also made in [10]. Optimizing over Q W | ˜ S , we can recover their Theorem 5.1 (again, up to a multiplicative constant and a O (1 / √ n ) fluctuation term):

Corollary 2. Under the assumptions of Theorem 2,

where Z and Z ′ are independent samples from P Z and where the conditional mutual information is computed w.r.t. ¯ P .

The main advantage of using conditional mutual information is that it never exceeds n log 2 (of course, the bound is only useful if I ( W ; ε | ˜ S ) = o ( n ) ).

5 Estimates using couplings

We now turn to the analysis of E [gen( W,S )] using couplings. The starting point is the following observation: With ( S ′ , S, W ) be constructed as before, consider the quantities

Then, using the fact that 〈 P ˜ S ⊗ Q W , ˜ L n 〉 = 0 for any Q W ∈ P ( W ) , we have

This suggests the idea of introducing, for each s ∈ Z n , a coupling of P W | S = s and Q W , i.e., a probability law P UV | S = s for a random element ( U, V ) of W×W with marginals P U = P W | S = s and P V = Q W . We then have the following:

Theorem 3. For u, v ∈ W and ˜ s = ( s, s ′ ) ∈ Z n ×Z n , define

Then, for any Q W ∈ P ( W ) , any family of couplings P UV | S = s ∈ Π( P W | S = s , Q W ) depending measurably on s ∈ Z n , and any µ UV ∈ P ( W×W ) ,

where the expectation on the right-hand side is w.r.t. the joint law of ( U, V, ¯ U, ¯ V , ˜ S ) , under which ( S, U, V ) are distributed according to P S ⊗ P UV | S , ( ¯ U, ¯ V ) are distributed according to µ UV independently of ( U, V, S ) , and S ′ is distributed according to P S independently of everything else.

The proof makes essential use of symmetrization using an auxiliary n -tuple ε of i.i.d. Rademacher random variables, which allows us to apply Lemma 1 conditionally on ˜ S .

The coupling-based formulation looks rather complicated compared to the setting of Section 4. However, being able to choose not just the 'prior' Q W , but also the couplings P UV | S of P W | S and Q W and the reference measure µ UV , allows us to overcome some of the shortcomings of the set-up of Section 4. Consider, for example, the case when the learning algorithm ignores the data, i.e., P W | S = P W . Then we can choose Q W = P W , P UV | S (d u, d v ) = P W (d u ) ⊗ δ u (d v ) , where δ u is the Dirac measure concentrated on the point u , and µ UV = P UV (since the latter does not depend on S ). With these choices, U = V and ¯ U = ¯ V almost surely, so the right-hand side of (12) is identically zero. By contrast, the bounds of Theorems 1 and 2 always include an additional O (1 / √ n ) term even when W and ˜ S are independent.

Moreover, Theorem 3 can be used to recover the bounds of Theorems 1 and 2 up to multiplicative constants. For example, to recover Theorem 1, we apply Theorem 3 with P UV | S = P W | S ⊗ Q W , µ UV = Q W ⊗ Q W , and with an estimate on σ ( U, V, ˜ S ) based on the subgaussianity of glyph[lscript] ( w,Z ) .

For a more manageable bound that will be useful later, let us define the following for u, v ∈ W :

Corollary 3. Under the assumptions of Theorem 3,

6 Refined estimates via chaining in the space of measures

We now combine the use of couplings as in Section 5 with a chaining argument. The basic idea is as follows: Instead of coupling P W | S with Q W directly, we interpolate between them using a (possibly infinite) sequence of Markov kernels P 0 W | S , P 1 W | S , . . . , P K W | S , such that P 0 W | S = Q W and P K W | S = P W | S (or lim k →∞ P k W | S = P W | S in an appropriate sense, e.g., weakly for each S , if the sequence is infinite). Given any such sequence, we telescope the terms in (10) as follows:

For each k , we can now choose a family of random couplings P W k W k -1 | S ∈ Π( P k W | S , P k -1 W | S ) and a deterministic probability measure ρ W k W k -1 ∈ P ( W×W ) . The following is an immediate consequence of applying Corollary 3 to each summand:

Theorem 4. Let P W | S , Q W , P W k W k -1 | S , and ρ W k W k -1 be given as above. Then

where in the k th term on the right-hand side ( S, W k , W k -1 ) are jointly distributed according to P S ⊗ P W k W k -1 | S and ( ¯ W k , ¯ W k -1 ) are jointly distributed according to ρ W k W k -1 .

Apart from Theorem 1, we have been imposing only minimal assumptions on glyph[lscript] and then using symmetrization to construct various subgaussian random variables conditionally on W and ˜ S . For the next series of results, we will assume something more, namely that ( W , d ) is a metric space and that the following holds for the centered loss ¯ glyph[lscript] ( w,z ) := glyph[lscript] ( w,z ) -E [ glyph[lscript] ( w,Z )] :

In other words, the centered empirical process 1 √ n ∑ n i =1 ¯ glyph[lscript] ( w,Z i ) indexed by the elements of ( W , d ) is a subgaussian process [1-3].

Theorem 5. Assume (13) . Then

As a byproduct, we recover the stochastic chaining bounds of Zhou et al. [20] (which, in turn, subsume the bounds of Asadi et al. [18]):

Corollary 4. Let P Z and P W | S be given, and let P W be the marginal law of W . Let ( P W k | S ) k ≥ 0 be a sequence of Markov kernels satisfying the following conditions: (i) P W 0 | S = P W ; (ii) P W k | S k →∞ - - - - → P W | S ; (iii) for every k ≥ 1 , S -W -W k -W k -1 is a Markov chain. Then

Finally, we give an estimate based on 2 -Wasserstein distances (cf. Section 2 for definitions and notation). Let W 2 ( · , · ) be the 2 -Wasserstein distance on P 2 ( W ) induced by the metric d on W . A (constant-speed) geodesic connecting two probability measures P, Q ∈ P 2 ( W ) is a continuous path [0 , 1] glyph[owner] t ↦→ ρ t ∈ P 2 ( W ) , such that ρ 0 = P , ρ 1 = Q , and W 2 ( ρ s , ρ t ) = ( t -s ) W 2 ( P, Q ) for all 0 ≤ s ≤ t ≤ 1 [22,23]. Then we have the following corollary of Theorem 5:

Corollary 5. Let P Z and P W | S be given, and let P W be the marginal law of W . With respect to 2-Wasserstein distance, let ( P W k | S ) 0 ≤ k ≤ K be some points on the constant-speed geodesic ( ρ t ) t ∈ [0 , 1] with endpoints ρ 0 = P W 0 | S = P W | S and ρ 1 = P W K | S = P W (where K may be infinite), i.e., there exist some t 0 = 0 < t 1 < · · · < t k < · · · < t K = 1 , such that P W k | S = ρ t k for k = 0 , 1 , . . . . For each k let P W k W k -1 | S be the optimal coupling between the neighboring points P W k -1 | S and P W k | S , i.e., the one that achieves W 2 ( P W k -1 | S , P W k | S ) . Then

Observe that the first term on the right-hand side of (17) is the expected 2 -Wasserstein distance between the posterior P W | S and the prior P W , while the second term is a sum of 'divergence weighted' Wasserstein distances. Also note that the form of the second term is in the spirit of the Dudley entropy integral [1-3], where the Wasserstein distance corresponds to the radius of the covering ball and the square root of the divergence corresponds to square root of the metric entropy. We should also point out that the result in Corollary 5 does not require Lipschitz continuity of the loss function glyph[lscript] ( w,z ) w.r.t. the hypothesis w ∈ W , except in a weaker stochastic sense as in (13), in contrast to some existing works that obtain generalization bounds using Wasserstein distances [27,28].

7 Tail estimates

Next, we turn to high-probability tail estimates on gen( W,S ) . We start with the followoing simple observation: Assume glyph[lscript] ( w,Z ) is σ -subgaussian for all w ∈ W when Z ∼ P Z . Then, for any Q W ∈ P ( W ) such that P W | S = s glyph[lessmuch] Q W for all s ∈ Z n , we have

with W ∼ Q W independent of ( S, W ) . Thus, by Markov's inequality, for any 0 < δ < 1 ,

In other words, | gen( W,S ) | glyph[lessorsimilar] σ √ n ψ -1 2 ( d P W | S d Q W ) with high P SW -probability. Similar observations are made by Hellström and Durisi [9] with Q W = P W , giving high-probability bounds of the form | gen( W,S ) | glyph[lessorsimilar] √ σ 2 D ( P W | S ‖ P W ) n . Generalization bounds in terms of the divergence D ( P W | S ‖ P W ) are also common in the PAC-Bayes literature [4,5]. Moreover, using the inequality (5) in Lemma 1, we can give high P S -probability bounds on the conditional expectation

Theorem 6. Assume glyph[lscript] ( w,Z ) is σ -subgaussian for all w when Z ∼ P Z . Then, for any Q W ∈ P ( W ) , the following holds with P S -probability of at least 1 -δ :

Another type of result that appears frequently in the literature on PAC-Bayes methods pertains to so-called transductive bounds , i.e., inequalities for the difference between

and

where Q W is some fixed 'prior' and where ¯ W ∼ Q W is independent of ( S ′ , S, W ) . Using our techniques, we can give the following general transductive bound:

Theorem 7. Let P W | S and Q W be given and take any ( P W k W k -1 | S ) K k =1 and ( ρ W k W k -1 ) K k =1 as in Theorem 4. Also, let p = ( p 1 , p 2 , . . . ) be a strictly positive probability distribution on N . Then the following holds with P ˜ S -probability at least 1 -δ :

where

This result subsumes some existing transductive PAC-Bayes estimates, such as Theorem 2 of Audibert and Bousquet [17]. Let us briefly explain how we can recover this result from Theorem 7. Assume that W is countable and let ( A k ) be an increasing sequence of finite partitions of W with A 0 = {W} . For each k and each w ∈ W , let A k ( w ) be the unique set in A k containing w . Choose a representative point in each A ∈ A k and let W k denote the set of all such representatives, with W 0 = { w 0 } . Take P W ∞ | S = P W | S and P W 0 = Q W = δ w 0 . Now, for each k ≥ 0 , we take P W k | S as the projection of P W | S onto W k , i.e., the finite mixture

Moreover, given some 'prior' π ∈ P ( W ) , we can construct a sequence ( π k ) ∞ k =0 of probability measures with π ∞ = π and π 0 = δ w 0 , such that π k is a projection of π onto W k . Now observe that, for each k , S -W ∞ -W k -W k -1 is a Markov chain. Indeed, if we know P W k | S , then we can construct P W glyph[lscript] | S for any glyph[lscript] < k without knowledge of S . With these ingredients in place, let us choose P W k W k -1 | S = P W k -1 | W k ⊗ P W k | S and ρ W k W k -1 = π k ⊗ P W k -1 | W k . Then, using Cauchy-Schwarz and Jensen, we conclude that the following holds with P ˜ S -probability at least 1 -δ :

.

This recovers [17, Thm. 2] up to an extra term that scales like 1 √ n ∑ k √ 〈 π k ⊗ P W k -1 | W k , d 2 ˜ S,glyph[lscript] 〉 .

8 The Fernique-Talagrand bound

As a bonus, we show that a combination of decorrelation and chaining in the space of measures can be used to recover the upper bounds of Fernique [29] and Talagrand [30] on the expected supremum of a stochastic process in terms of majorizing measures (see Eq. (19) below and also [31,32]).

For simplicity, let ( T, d ) be a finite metric space with diam( T ) = sup { d ( u, v ) : u, v ∈ T } < ∞ . Let B ( t, r ) denote the ball of radius r ≥ 0 centered at t ∈ T , i.e., B ( t, r ) := { u ∈ T : d ( u, t ) ≤ r } . Let ( X t ) t ∈ T be a centered stochastic process defined on some probability space (Ω , A , P ) and satisfying

for some p ≥ 1 . Then we can obtain the following result using chaining in the space of measures and decorrelation estimates:

Theorem 8. Let τ be a random element of T , i.e., a measurable map τ : Ω → T with marginal probability law ν . Then for any µ ∈ P ( T ) we have

Applying Theorem 8 to τ ∗ defined in (1) and then minimizing over µ , we recover a FerniqueTalagrand type bound on the expected supremum of X t :

9 Conclusion and future work

In this paper, we have presented a unified framework for information-theoretic generalization bounds based on a combination of two key ideas (decorrelation and chaining in the space of measures). However, our method has certain limitations, which we plan to address in future work. For example, it would be desirable to cover the case of processes satisfying Bernstein-type (mixed ψ 1 and ψ 2 ) increment conditions. It would also be of interest to see whether there are any connections to the convex-analytic approach of Lugosi and Neu [33]. Finally, since our method seamlessly interpolates between Fernique-Talagrand type bounds and information-theoretic bounds, we plan to use it to further develop the ideas of Hodgkinson et al. [21], who were the first to combine these two complementary approaches to analyze the generalization capabilities of iterative learning algorithms.

Acknowledgments

This work was supported by the Illinois Institute for Data Science and Dynamical Systems (iDS 2 ), an NSF HDR TRIPODS institute, under award CCF-1934986. The authors would like to thank Matus Telgarsky for some valuable suggestions and the anonymous reviewers for pointing out some relevant work that was not cited in the original submission.

References

Aad W. van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes . Springer, 1996.

Evarist Giné and Richard Nickl. Mathematical Foundations of Infinite-Dimensional Statistical Models . Cambridge University Press, 2016.

Michel Talagrand. Upper and Lower Bounds for Stochastic Processes: Modern Methods and Classical Problems . Springer, 2014.

David A. McAllester. PAC-Bayesian model averaging. In Conference on Learning Theory , 1999.

Tong Zhang. Information-theoretic upper and lower bounds for statistical estimation. IEEE Transactions on Information Theory , 52(4):1307-1321, April 2006.

Daniel Russo and James Zou. Controlling Bias in Adaptive Data Analysis Using Information Theory. In Conference on Artificial Intelligence and Statistics , 2016.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems , 2017.

Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening Mutual Information Based Bounds on Generalization Error. IEEE Journal on Selected Areas in Information Theory , 1(1):121-130, May 2020.

Fredrik Hellström and Giuseppe Durisi. Generalization Bounds via Information Density and Conditional Information Density. IEEE Journal on Selected Areas in Information Theory , 1(3):824-839, November 2020.

Thomas Steinke and Lydia Zakynthinou. Reasoning About Generalization via Conditional Mutual Information. arXiv:2001.09122, June 2020.

Hassan Hafez-Kolahi, Zeinab Golgooni, Shohreh Kasaei, and Mahdieh Soleymani. Conditioning and processing: Techniques to improve information-theoretic generalization bounds. In Advances in Neural Information Processing Systems , 2020.

Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. In Advances in Neural Information Processing Systems , 2020.

Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa. Generalization error bounds via Rényi-, f -divergences and maximal leakage. IEEE Transactions on Information Theory , 67(8):4986-5004, August 2021.

Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. Towards a unified information-theoretic framework for generalization. In Advances in Neural Information Processing Systems , 2021.

Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Informationtheoretic generalization bounds for black-box learning algorithms. In Advances in Neural Information Processing Systems , 2021.

Ruida Zhou, Chao Tian, and Tie Liu. Individually conditional individual mutual information bound on generalization error. IEEE Transactions on Information Theory , 68(5):3304-3316, May 2022.

Jean-Yves Audibert and Olivier Bousquet. Combining PAC-Bayesian and Generic Chaining Bounds. Journal of Machine Learning Research , 8(32):863-889, 2007.

Amir Asadi, Emmanuel Abbe, and Sergio Verdú. Chaining Mutual Information and Tightening Generalization Bounds. In Advances in Neural Information Processing Systems , 2018.

Eugenio Clerico, Amitis Shidani, George Deligiannidis, and Arnaud Doucet. Chained generalisation bounds. In Conference on Learning Theory , 2022.

Ruida Zhou, Chao Tian, and Tie Liu. Stochastic Chaining and Strengthened InformationTheoretic Generalization Bounds. In IEEE International Symposium on Information Theory , January 2022.

Liam Hodgkinson, Umut ¸ Sim¸ sekli, Rajiv Khanna, and Michael W. Mahoney. Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers. In International Conference on Machine Learning , July 2022.

Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures . Birkhäuser, 2008.

Cédric Villani. Topics in Optimal Transportation . American Mathematics Society, Providence, RI, 2003.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

Roman Vershynin. High-Dimensional Probability . Cambridge University Press, 2018.

Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning . Cambridge University Press, forthcoming. URL: https://people.lids.mit.edu/yp/homepage/data/itbook-export.pdf .

Hao Wang, Mario Diaz, José Cândido S. Santos Filho, and Flavio P. Calmon. An informationtheoretic view of generalization via Wasserstein distance'. In IEEE International Symposium on Information Theory , 2019.

Borja Rodrígues-Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. Tighter expected generalization error bounds via Wasserstein distance. In Advances in Neural Information Processing Systems , 2021.

Xavier Fernique. Regularité des trajectoires des fonctions aléatoires Gaussiennes. In X. M. Fernique, J. P. Conze, J. Gani, and P. L. Hennequin, editors, Ecole d'Eté de Probabilités de Saint-Flour IV-1974 , Lecture Notes in Mathematics, pages 1-96. Springer, 1975.

Michel Talagrand. Regularity of gaussian processes. Acta Mathematica , 159:99-149, 1987.

Witold Bednorz. A theorem on majorizing measures. The Annals of Probability , 34(5), September 2006.

Witold Bednorz. The majorizing measure approach to sample boundedness. Colloquium Mathematicae , 139(2):205-227, 2015.

Gábor Lugosi and Gergely Neu. Generalization bounds via convex analysis. In Conference on Learning Theory , 2022.

A Some elementary facts

We first list some useful inequalities for ψ p and ψ -1 p . Note that the estimates may not be the sharpest, but they suffice for our needs.

Proposition A.2. For p ≥ 1 and x ≥ 0 , let ψ p ( x ) = exp( x p ) -1 and let ψ -1 p ( x ) = (log( x +1)) 1 /p be its inverse. Then we have the following:

Proof.

(i) For any x ≥ 0 ,

(ii) We only need to consider the case x ≥ 1 since otherwise the inequality is obvious. Since y ≤ 2(exp( y/ 4) + 1) for all y ≥ 1 , we have

x

2

1

/p

(exp(

x

p

/

4) + 1)

1

/p

2

1

/p

(exp(

x

p

/

4

p

) + 1)

2

1

/p

(exp(

x

p

/

4) + 1)

.

Then

(iii) Since x ≥ 0 and q ≥ 1 ,

(iv) When x ≥ 1 ,

The following simple result is for converting between sums and integrals:

Proposition A.3. For any r ≥ 2 , K ∈ N , and a continuous nonincreasing f : (0 , + ∞ ) → (0 , + ∞ ) , we have

Proof. Using the monotonicity of f , we have

B Omitted proofs

B.1 Proofs for Section 2

Proof of Proposition 1. It follows from the inequality x log( x +1) ≤ x log x +1 , x ≥ 0 , that

Using this and Jensen's inequality, we get

B.2 Proofs for Section 3

Proof of Lemma 1. To prove (4), we start with the Young-type inequality

is the (one-sided) Legendre-Fenchel conjugate of ψ p . While a closed-form expression for ψ ∗ p is not available, we claim that we can bound it from above as ψ ∗ p ( x ) ≤ 2 1 /p xψ -1 p ( x ) , resulting in

where

To establish the claim, we write

and consider two cases:

if y ≤ 2 1 /p ψ -1 p ( x ) , then

Applying (B.1) with x = d µ d ν and y = g gives

so that

To prove (5), define the event

Then, since 〈 ν, exp( g p ) 〉 ≥ 1 ,

On the other hand,

where the first inequality is by the definition of E , the second inequality follows from Proposition A.2(ii), and the third inequality is by Cauchy-Schwarz. Putting everything together, we get (5).

B.3 Proofs for Section 4

Proof of Theorem 1. It follows from the independence of Z 1 , . . . , Z n that gen( w,S ) is ( σ/ √ n ) -subgaussian, so

Using Lemma 1 with µ = P W | S , ν = Q W , f ( w ) = σ √ 6 /n , and g ( w ) = | gen( w,S ) | σ √ 6 /n , we have

Taking expectations of both sides w.r.t. P S and using Fubini's theorem and (B.2), we get (6).

Proof of Corollary 1. Applying Proposition 1 conditionally on S gives

where the divergence D ( P W | S ‖ Q W ) , being a function of S , is a random variable. Substituting this into (6) and using Jensen's inequality, the definition of conditional divergence, and √ a + b ≤ √ a + √ b ≤ √ 2( a + b ) , we get

Taking the infimum of both sides w.r.t. Q W and using (2), we get (7).

Proof of Theorem 2. For each fixed ( w, ˜ s ) , the random variable δ ( w, ˜ s, ε ) := | ∑ n i =1 ε i ( glyph[lscript] ( w,z ′ i ) -glyph[lscript] ( w,z i )) | is σ ( w, ˜ s ) -subgaussian, where σ ( w, ˜ s ) := ( ∑ n i =1 ( glyph[lscript] ( w,z ′ i ) -glyph[lscript] ( w,z i )) 2 ) 1 / 2 . Thus,

Applying Lemma 1 conditionally on ( ˜ S, ε ) with µ = ¯ P W | ˜ Sε , ν = Q W | ˜ S , f ( w ) = σ ( w, ˜ S ) , g ( w ) = ζ ( w, ˜ S, ε ) , we obtain

Taking expectations of both sides w.r.t. ˜ S and ε , then using Fubini's theorem, (B.3), and the inequality E P [ | gen( W,S ) | ] ≤ 1 n E ¯ P [ δ ( W, ˜ S, ε )] , we obtain (8).

Proof of Corollary 2. For any Q W | ˜ S , using Proposition 1, Cauchy-Schwarz, and the independence of ( Z ′ i , Z i ) , we have

Substituting this estimate into (8), taking the infimum of both sides w.r.t. Q W | ˜ S , and using (3), we get (9).

B.4 Proofs for Section 5

Proof of Theorem 3. Let

For each fixed ( u, v ) ∈ W 2 , δ ( u, v, Z i , Z ′ i ) , 1 ≤ i ≤ n , are i.i.d. symmetric random variables. Therefore, introducing a tuple ε = ( ε 1 , . . . , ε n ) of i.i.d. Rademacher random variables independent of everything else and using the fact that the joint distributions of ( δ ( u, v, Z i , Z ′ i ) ) n i =1 and ( ε i δ ( u, v, Z i , Z ′ i ) ) n i =1 are the same, we see that

where the inequality follows from the fact that, conditionally on S and S ′ , the random variables ∑ n i =1 ε i δ ( u, v, Z i , Z ′ i ) are σ ( u, v, ˜ S ) -subgaussian.

Now, given Q W ∈ P ( W ) and a family of couplings P UV | S = s ∈ Π( P W | S = s , Q W ) , it follows from the above definitions and from (10) that

Picking any ρ UV ∈ P ( W×W ) such that P UV | S = s glyph[lessmuch] ρ UV for all s ∈ Z n and applying Lemma 1, we get

Using the inequality ψ 2 2 ( x/ 2) ≤ ψ 2 ( x ) (see Proposition A.2(i)), Cauchy-Schwarz, and (B.4), we have

Putting everything together and taking expectations w.r.t. S and S ′ , we obtain (12).

Proof of Corollary 3. For σ defined in Theorem 3, we have

Taking conditional expectations given U, V, S and using Jensen's inequality gives

An analogous argument gives

Substituting these estimates into (12) gives the desired result.

B.5 Proofs for Section 6

Proof of Theorem 5. Using the definition of ¯ glyph[lscript] , we have

Applyng Lemma 1 conditionally on S with f ( u, v ) = d ( u, v ) , g ( u, v ) = | ∑ n i =1 ( ¯ glyph[lscript] ( u,,Z i ) -¯ glyph[lscript] ( v,Z i )) | √ nd ( u,v ) , µ = P W k W k -1 | S and ν = ρ W k W k -1 , taking expectations w.r.t. P S , and using (13) gives the desired result.

Proof of Corollary 4. For each k ≥ 1 , let ρ W k W k -1 = P W k W k -1 . Then

where we have made use of Bayes' rule and the fact that S ⊥ ⊥ W k -1 | W k . Using this in (14) together with Proposition 1 gives (15). An application of Cauchy-Schwarz and Jensen gives (16).

Proof of Corollary 5. For each k ≥ 1 , let ρ W k W k -1 = P W k W k -1 . Notice that, by disintegration and the choice of couplings,

where we have used the fact that W 2 ( · , · ) dominates W 1 ( · , · ) [23]. Since P W k | S are points on the geodesic connecting P W | S and P W , we have

Using this together with Cauchy-Schwarz and Proposition 1, we obtain (17).

B.6 Proofs for Section 7

Proof of Theorem 6. Applying (5) conditionally on S with f ( w ) = σ √ 6 /n , g ( w ) = | gen( w,S ) | σ √ 6 /n ,

Since glyph[lscript] ( w,S ) is ( σ/ n ) -subgaussian for all w , Markov's inequality gives, for any 0 < δ < 1 ,

which concludes the proof.

Proof of Theorem 7. The argument is almost identical to the proof of Theorem 3, with the difference that (5) is used for decorrelation.

To lighten the notation, let π S k := P W k W k -1 | S and ρ k := ρ W k W k -1 . Use the same definitions of δ, σ, ζ as in the proof of Theorem 3. Then, applying (5) with f ( · ) = σ ( · , ˜ S ) , g ( · ) = ζ ( · , ˜ S ) , µ = π S k , ν = ρ k , we have

By Markov's inequality and the union bound, for any 0 < δ < 1 ,

Using this together with the estimate σ ( · , ˜ S ) ≤ 2 √ nd ˜ S,glyph[lscript] ( · ) yields the result in the statement.

B.7 Proofs for Section 8

Proof of Theorem 8. Without loss of generality, we assume diam( T ) = 1 . Let Q be the Markov kernel from Ω to T defined by Q ( ·| ω ) = δ τ ( ω ) ( · ) ; in particular, ν ( · ) = ∫ Ω P (d ω ) Q ( ·| ω ) .

Fix some r ≥ 2 . For each k ≥ 0 and each t ∈ T , let B k ( t ) := B ( t, r -k ) . Since T is finite, there exists some K ∈ N , such that B K ( t ) = { t } for all t ∈ T . Let µ ∈ P ( T ) be given. Define the following sequence of Markov kernels from Ω to T :

Observe that Q 0 = µ and Q K = Q . Then, since

we can write

Applying (4) conditionally on ω with

and using the fact that Q k ( ·| ω ) is supported on B k ( τ ( ω )) and B k ( τ ( ω )) ⊆ B k -1 ( τ ( ω )) , we have

where the first term in the last step is due to Proposition A.2(iii). Then, using the increment condition and Proposition A.3, we have

Since 1 /µ ( B ( t, ε )) ≥ 1 , we can apply Proposition A.2(iv) to obtain the inequality

We can now take r = 2 to get the desired result when diam( T ) = 1 ; the general finite-diameter case follows by straightforward rescaling.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the generalization error of the statistical learning algorithms from the information-theoretic point of view. In particular, by leveraging a “decorrelation lemma”, the authors show how various previously established upper bounds on the generalization error, including the common information-theoretic bounds, PAC-Bayes bounds, and chaining versions of them, can be recovered using this decorrelation lemma; and therefore, present a unifying methodology for establishing these bounds.\n\nSTRENGTHS:\nThe authors using the introduced decorrelation lemma managed to compactly recover various existing upper bounds on the generalization error. In particular, the method naturally can be combined with the chaining methods to recover a number of results such as the generalization bounds of Zhou et al., Asadi et al., and the bound of Fernique and Talagrand on the expected supremum of a stochastic process. Moreover, the work presents high probability bounds; which is rarely common in the information-theoretic bounds on the generalization.\n\nI believe the decorrelation lemma is a powerful technical lemma using which hopefully, new understandings and bounds can be achieved in future works. For this reason, I think the paper could be of interest to the part of the Neurips audiences that are interested in the theoretical understating of statistical learning algorithms.\n\nWEAKNESSES:\n1. The main weakness of the work is that it remains only at the technical level. \n\n-  1.a. The form of the general bounds, e.g. Theorem 3, does not seem to be easy to compute. It is not clear whether the general results can be leveraged to derive new bounds or to give a new understanding of the generalization error of the learning algorithms (in general or for a particular algorithm). Can authors provide cases where the results of this work give “new” and “simple” or \"practical\" results? I think the authors would agree that the justification of how coupling in Theorem 3 can be used to show that the expected generalization error is zero when $P_{W|S}=P_W$ is not convincing enough to show the utility of that.\n\n- 1.b. The only cases where these general results give concrete results are when they recover previous results (e.g. Information-theoretic bounds or PAC-Bayes ones) but only with the worst coefficients.\n\n- 1.c. Moreover, even at the intuition level, it is not clear if this work, and more specifically, this decorrelation lemma, gives any new “understanding\" of the generalizability of the learning algorithms. Does this work, and this decorrelation lemma, can provide any new intuition/understanding?\nFor these reasons, unfortunately, the paper remains only on the “technical level” and we can only hope that this interesting tool can be used for establishing new concrete results/understandings.\n\n2. The paper lacks a proper literature review and comparison with other works. Indeed, only 25 works have been cited; most of which are purely listed in the very short introduction. The authors are needed to widen the comparison of the current work with previous literature on generalization error (including different information-theoretic approaches, PAC-Bayes ones, compression-based ones, etc.) to make a clear picture of where this work/tools stand.\n\n3. As an example of the previous point, perhaps, as claimed by the authors, one of the main advantages of this work is the unifying methodology for deriving various existing bounds. This is indeed interesting, as has been done in some other works before. Non-exhaustive examples of some of these works, sorted by year, are: “PAC-MDL Bounds” (Blum and Langford 2003), arXiv:1912.01439, arXiv:2006.13057, arXiv:2105.01747, arXiv:2110.11216, arXiv:2111.05275, arXiv:2203.02474, arXiv:2303.05369. Although some of these works are less directly related as they consider different approaches (and some maybe are more directly related), I believe it is helpful to put this work in this context as well: how does the unifying approach of this work differ from previous attempts? What are the similarities? What are the “limitations” of those works and this one?\n\nQUESTIONS:\nThe main ones are mentioned above. Some other minor typos/comments:\n\n-\tAlthough trivial but K and C are not introduced in line 93.\n-\tIn lines 120-121, it is claimed that “the right-hand side can be further upper-bounded in terms of the information divergence $D(P_{Y|X}\\| Q_Y)$ using Proposition 1.” Can it be done for any function $f(y)$?\n-\tTypo in line 141: $\\frac{|gen(w,S)|}{\\sigma \\sqrt{6/n}}$\n-\tThe proof of Theorem 5, or a comment on how it is derived seems to be missing.\n-\tThere are typos in the proof of Proposition 1;  after line 367, the LHS should be $\\frac{d\\mu}{d\\nu}\\log(\\frac{d\\mu}{d\\nu}+1)$. A similar typo exists on the first line after 368.\n-\tThe term “f” seems to be missing in the last line of equations after line 379.\n\nLIMITATIONS:\nMentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes some steps of the standard formula to obtain generalization error bounds: (i) decoupling of the joint distribution + (ii) chaining. Then, they use this formula contributing mainly on the first front by deriving new decoupling results based on Orlicz $\\psi\\_p$-norms that can be cast into the standard decoupling results based on the relative entropy.\n\nAdditionally, they also include a section describing results based on couplings between the hypothesis' distribution obtained with the algorithm's kernel evaluated on a training set and the hypothesis' marginal distribution. These are used to develop their bounds on chaining, but they can stand alone independently.\n\nFinally, the authors note that their results can be extended to high-probability PAC-Bayes bounds and that these can be used to also recover bounds of the type of Fernique's and Talagrand's bounds on the expected supremum of a stochastic process.\n\nSTRENGTHS:\nThe paper is generally well-written and easy to follow. It does a good work summarizing how to derive generalization error bounds using Orlicz norms and, therefore, using the relative entropy of the posterior hypothesis distribution $P\\_{W|S}$ with respect to a prior $Q\\_W$.\n\nTheir decoupling result (or \"decorrelation lemma\") is interesting and they gain clarity and interoperability by relaxing the tighter standard result they could have obtained with Young's inequality. I believe that Lemma 1 is the main result of the paper, which is later interpreted and contextualized into the generalization error set-up.\n\nIt is good that the presented bounds can be brought down (up to constants) to known bounds in the community. This is usually a necessary \"goodness test\" to know if the results obtained are valuable. Particularly, it was good to see that Fernique's and Talagrand's bounds were recoverable this way.\n\n[*Even though the strengths section will be shorter than the weaknesses, I believe this is a good paper. The authors should note that this will be the case only to help them improve the manuscript and to expand on each of the weaknesses so it can be easily addressed.*]\n\nWEAKNESSES:\nWhile their \"decorrelation lemma\" is novel and interesting, the applications of this lemma afterwards for the estimates using couplings, the usage of chaining in the space of measures, and the tail estimates are standard and shown otherwise with different decoupling lemmas (see e.g. [7,9,11,15,16,17,18] of the references of their own paper). While probably not on purpose, the paper seems to position itself as if it is introducing this set of steps to obtain this kind of bounds. This is not accurate, as these procedures have been used and done previously, just with a different (although maybe less general in some situations) decoupling lemma. I believe it would benefit the paper to make this clear both in the introduction and at the start of each of these sections.\n\nThe paper obtains results based on the inverse of an Orlicz norm of the Radon-Nikodym derivative of the posterior with respect to the prior. While this seems to be more general, the paper only gives examples of this instantiating into relative entropy-based bounds. In the end, it does not provide any example of anything new to gain by using their framework with respect to what was already known. This is not super problematic, but when the constants of the bounds are deteriorating under this framework, it is good to show that it can produce new results that lead to interesting conclusions.\n\nSimilarly to the comment above, the paper does not give examples or motivation to many of their bounds. For instance:\n- Why are the bounds using couplings useful? The paper mentions the fact that when the algorithm ignores the data, i.e. $P\\_{W|S} = P\\_W$ a.s. and the prior $Q\\_W$ is chosen to be the real marginal of the hypothesis $P\\_W$, then the resulting bound is exactly zero, which is not the case for the bounds presented in Section 4. However, the bounds in Section for are for the expected *absolute* generalization error, where the extra term $O(1/\\sqrt{n})$ is hard to avoid (or unavoidable in many cases). In Section 5, the bounds are for the expected generalization error instead, where the standard bounds (e.g. from [7]) already achieve a zero generalization error with this set-up.\n- Why are the bound using chaining useful? The paper basically uses the same techniques as [17] with their \"decorrelation lemma\". In [16,17] they come up with a slightly artificial example to showcase why their results are useful when compared to previous bounds. It is unclear if this technique provides any practical advantage to previous results, either in terms of obtaining bounds with a better rate in certain settings, in terms of interpreting the resulting bounds, or even in terms of simplicity to obtain actually computable bounds.\n\nThe paper does not discuss some relevant literature in certain parts of the text:\n-  In line 102 they state that they \"define the conditional divergence [...]\". This is a standard notation for the conditional divergence, sometimes credited to Verdú, although I am unsure which is the origin. Probably this is just a writing thing, but it contrasts with the previous introduction of the relative entropy where they introduce it with \"[...] is defined as\".\n- Even though the techniques employed are different, some mention to [A] would be interesting in the introduction and/or in the preliminaries. In this paper, they obtain PAC-Bayes bounds considering Orlicz norms with general Orlicz function, not necessarily $\\exp(x^p) - 1$.\n- In Section 5, it would be interesting to compare and/or mention other works that deal with the couplings of the posterior and the prior to bound the generalization error, e.g. [B,C,D]. In [C], for instance, they also mention the relationship of these bounds with chaining in Appendix A and with the relative entropy and subgaussian conditions in Appendix B.\n- In Section 7, it would be good to compare and discuss the work in [E] regarding chained generalization error bounds.\n\n**Additional References**\n\n[A] Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa. \"Generalization Error Bounds Via Rényi, f-Divergences and Maximal Leakage\". IEEE Transactions on Information Theory. 2021.\n\n[B] Hao Wang, Mario Diaz, José Cândido S. Santos Filho, and Flavio P. Calmon. \"An Information-Theoretic View of Generalization via Wasserstein Distance\". IEEE ISIT. 2019.\n\n[C] Borja Rodríguez-Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. \"Tighter Expected Generalization Error Bounds via\nWasserstein Distance\". NeurIPS. 2021.\n\n[D] Ron Amit, Baruch Epstein, Shay Moran, and Ron Meir. \"Integral Probability Metrics PAC-Bayes Bounds\". NeurIPS. 2022.\n\n[E] Eugenio Clerico, Amitis Shidani, George Deligiannidis, and Arnaud Doucet. \"Chained Generalisation Bounds\". COLT. 2022.\n\nQUESTIONS:\n- I believe in the equations after line 379 there is an $f$ missing in the last equality.\n\n- I don't really follow the set of inequalities after line 381. Could you please clarify it?\n\n- Can (how does) this framework incorporate other advances in generalization error bounds based on mutual information like the single sample bounds from [8], [F], [G] or the data-dependent bounds from [H], [12], [F]?\n\n- How can we interpret the provided bounds? Can new results be obtained from them other than recovering the standard relative entropy-based bounds? Could you give some examples where the abstraction to Orlicz norms is beneficial to the analysis?\n\n**Additional References**\n\n[F] Borja Rodríguez-Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. \"On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithm\". ITW. 2020.\n\n[G] Ruida Zhou, Chao Tian, and Tie Liu. \"Individually Conditional Individual Mutual Information Bound on Generalization Error\". IEEE Transactions on Information Theory. 2022.\n\n[H] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. \"Information-theoretic generalization bounds for SGLD via data-dependent estimates\". NeurIPS. 2019.\n\nLIMITATIONS:\nThe limitations of the paper are not generally pointed out in the paper (e.g. some of the questions and weaknesses above are not discussed such as how the extension to Orlicz spaces gives any advantage to the standard relative entropy and the decoupling lemma using Donsker-Vardhan).\n\nRegarding potential negative societal impact, there is no mention but there is no need to be as this is theoretical, fundamental research.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper presents a unified perspective of information-theoretic generalization bounds through decorrelation and coupling/chaining. Various existing generalization bounds in the literature are recovered or generalized via this perspective. The Fernique-Talagrand upper bound on the expected supremum of subgaussian processes also emerge as a special case from this framework.

STRENGTHS:
I did not check carefully the proofs in the supplementary materials, but the presented results look sound to this reviewer. The paper offers a good service to the learning theory community for understanding information-theoretic generalization bounds and key techniques therein for their development. The development in this paper is original, to this reviewer. 

The paper is also clearly written. I appreciate the authors' informal summary in section 1.1, making the paper easy to read.

WEAKNESSES:
On the minor side, the work appears to be limited only to unification, without aiming at developing novel and/or tighter bounds using the developed tool, despite that some bounds are stated in more general forms (e.g., Theorem 7) and some are shown to have certain improvement (e.g. Theorem 3). It would be much better appreciated if the authors can further demonstrate the power of their framework by presenting novel bounds of greater significance.

QUESTIONS:
In the discussion after Theorem 3, the authors demonstrated the advantage of the theorem in the trivial case where W and S are independent. Are there more interesting (i.e. less trivial) cases where such an advantage can be demonstrated?

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper proposes a unified framework for deriving information-theoretic generalization bounds for learning algorithms. The main technical result relies on a probabilistic decorrelation lemma based on a change of measure and Young’s inequality in $L_{\psi_p}$ Orlicz spaces. Combining it with other techniques, such as summarization, couplings, and changing the space of probability measures, new upper bounds on the generalization error can be obtained both in expectation and in high probability. The proposed framework also recovers many of the existing generalization bounds as special cases, including the ones based on mutual information, conditional mutual information, stochastic chaining, and PAC-Bayes inequalities. Strength: Lemma 1 provides a very general way to derive generalization error bounds in multiple setting, and it recovers many existing results as special case.

STRENGTHS:
The decorrelation Lemma 1 provides a very general way to derive generalization error bounds in multiple settings, and it recovers many existing results as a special case.

WEAKNESSES:
As the proposed bounds are not based on standard information measures, is there a way to evaluate these quantities even in some illustrative examples, like the mean estimation problem considered in [8] and [17]?

QUESTIONS:
Could the proposed framework recover the ICIMI bounds obtained in the following paper? In general, is it compatible with the individual sample technique used in [8] and [12]?

Zhou, Ruida, Chao Tian, and Tie Liu. "Individually conditional individual mutual information bound on generalization error." IEEE Transactions on Information Theory 68, no. 5 (2022): 3304-3316.

LIMITATIONS:
The limitation has been addressed well in the confusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07