Title: Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Bounds
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalization

information-theoretic bounds

stability
Abstract: We present new information-theoretic generalization guarantees through the a novel construction of the "neighboring-hypothesis" matrix and a new family of stability notions termed sample-conditioned hypothesis (SCH) stability.  Our approach yields sharper bounds that improve upon previous information-theoretic bounds in various learning scenarios. Notably, these bounds address the limitations of existing information-theoretic bounds in the context of stochastic convex optimization (SCO) problems, as explored in the recent work by Haghifam et al. (2023).

Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Bounds

Abstract

We present new information-theoretic generalization guarantees through the a novel construction of the "neighboring-hypothesis" matrix and a new family of stability notions termed sample-conditioned hypothesis (SCH) stability. Our approach yields sharper bounds that improve upon previous information-theoretic bounds in various learning scenarios. Notably, these bounds address the limitations of existing information-theoretic bounds in the context of stochastic convex optimization (SCO) problems, as explored in the recent work by Haghifam et al. (2023).

1 Introduction

Information-theoretic upper bounds for generalization error have recently been developed since the seminal works of [52, 67]. On the one hand, these bounds have attracted increasing interest due to their distribution dependence and algorithm dependence, making them highly suitable for reasoning the generalization behavior of modern deep learning. In fact, subsequent studies have demonstrated that information-theoretic bounds can effectively track the dynamics of generalization error in deep neural networks [40, 60, 23, 61, 62, 25, 64]. On the other hand, recent studies [58, 19, 23, 20, 25] have revealed the expressive nature of information-theoretic bounds in the distribution-free setting. Notably, the conditional mutual information (CMI) framework proposed by [58] has shown great promise by establishing connections to VC theory [59] and matching minimax rates for the binary classification [19, 20]. Additionally, in the case of the 0 -1 loss and the realizable setting, where an interpolating algorithm attains zero empirical risk, information-theoretic bounds give an exact characterization of the generalization error [20, 64], thereby providing the tightest possible generalization bound.

Nonetheless, information-theoretic bounds have been extensively discussed due to their two main deficiencies. The first deficiency concerns the unbounded nature of the original input-output mutual information (IOMI) bounds [67, 4, 32, 31]. To address this issue, several techniques have been developed, including the chaining method [2], the individual technique [9] or random subset technique [40], and the Gaussian noise perturbation technique [41]. Notably, the CMI bound [58] stands out as it has a finite upper bound for any learning scenario due to its supersample construction. These techniques are now often applied jointly for analyzing generalization [18, 49, 70, 63, 23, 25, 64]. The second deficiency concerns the sub-optimal convergence rate of information-theoretic bounds. Specifically, when the information-theoretic quantities, whether IOMI or CMI, are bounded by constants, the bounds exhibit a decaying rate in the order of O (1 / √ n ) , where n is the sample size. In contrast, generalization errors in practice may decay at a faster rate, e.g., O (1 /n ) . To address this limitation, several works, inspired by some PAC-Bayesian literature, have proposed fast-rate information-theoretic bounds [24, 25, 64], demonstrating a good characterization in some instances of non-convex settings such as deep learning. More recently, [65, 66, 71] present IOMI bounds for the Gaussian mean estimation problem, that achieve optimal convergence rates, contrasting previous bounds [9, 70].

The issue of slow convergence in information-theoretic bounds has recently been amplified by the observation that these bounds may not even vanish. [21] highlight this limitation in the context of stochastic convex optimization (SCO) problems [55]. Specifically, [21] shows that all existing information-theoretic bounds, including the CMI bound [58], the Gaussian noise perturbed IOMI bound [41], the individual IOMI bound [9], the evaluated CMI (e-CMI) bound [58] and the functional CMI ( f -CMI) bound [23], fail to vanish in at least one of the counterexamples they constructed. These failures stem from the dimension-dependent nature of current information-theoretic quantities [31], appearing an intrinsic barrier for overcoming these limitations. However, if we dissect the process by which these bounds are derived, opportunities do exist. Specifically, recall that all these information-theoretic bounds are built upon the Donsker-Varadhan (DV) variational representation of the KL divergence [44, Theorem 3.5] (see Lemma A.2 in the Appendix). Using this representation, a generalization upper bound is derived in terms of information-theoretic quantity and a cumulantgenerating function (CGF), as illustrated below.

Particularly note that the CGF depends on certain choice of the auxiliary function in the DV formula. Then, except for the IOMI or CMI itself, the tightness of the generalization bound hinges on two key factors: the selection of the DV auxiliary function and the approach used to bound the CGF, the latter of which often requires additional assumptions specific to the chosen DV auxiliary function. The most common choices for the DV auxiliary function involve making assumptions such as subGaussian loss or bounded loss. For instance, in the case of IOMI bounds, the DV auxiliary function is typically chosen as the single loss or average loss, and the sub-Gaussian assumption (or boundedness assumption) is utilized. In CMI bounds, the DV auxiliary function is defined as the difference in loss between a training sample and a 'ghost' sample, and the boundedness property is employed. We now note that exploiting bounded loss is the fundamental reason behind the failures in SCO problems. Although [21] does not explicitly rely on boundedness, they do make use of the product of the Lipschitz constant and the diameter of the hypothesis domain, which essentially serves as an upper bound for the loss. The product does not vanish with n , thereby neutralizing the potential to include another decaying factor in the bound. Arguably, at least for these SCO problems, the selection of the DV auxiliary function may not be optimal, for example, the resulting CGF = O (1 /n ) , giving vacuous generalization guarantees. This analysis inspires us to explore alternative DV auxiliary function and to adopt different assumptions for bounding the CGF. For instance, we may devise appropriate stability assumptions and create opportunities to upper bound the CGF using a term in O ( β 2 /n ) , where β is the stability parameter that decays as n . This is promising in light of the capability of the stability-based framework in explaining the generalization of SCO problems [6, 56, 22, 5].

In this paper, we combine information-theoretic analysis with stability notions to develop IOMI bounds and CMI bounds that improve upon previous information-theoretic bounds for stable learning algorithms, achieving faster convergence rates. The main contributions of our paper are summarized below.

We introduce our new notions of algorithmic stability, referred to as sample-conditioned hypothesis (SCH) stability. We also present a novel construction of a sample-dependent hypothesis matrix, where each column is a neighborhood pair of hypotheses obtained from two training samples that differ in only one element, inspired by the supersample setting of CMI [58].

We present new IOMI bounds, which explicitly include the SCH stability parameters and are shown superior to previous bounds for stable learning algorithms.

We show that the sample-dependent hypothesis matrix, similar to the supersample matrix, enjoys a symmetry property. Exploiting this symmetry, we establish novel CMI bounds. Specifically, we present hypotheses-conditioned CMI bounds that are analogous to the previous supersampleconditioned CMI bounds. Additionally, we derive sample-conditioned CMI bounds exploiting other assumptions. Notably, these bounds introduce novel CMI quantities and include SCH stability parameters. In particular, the new CMI quantities remain boundedness as the original CMI. Consequently, these CMI bounds vanish no slower than their stability parameters. In addition, we also obtain a second-moment generalization bound that matches the tightest known bound in the literature [14] under the same condition.

We apply our new bounds to a convex-Lipschitz-bounded (CLB) example in which previous information-theoretic bounds fail to explain generalization [21], and show that the new IOMI

and CMI bounds vanish, benefiting from SCH stability. Additionally, we discuss another CLB example where the uniform stability parameter is non-vanishing or has a slow convergence rate, but our information-theoretic bounds remain tight up to a constant.

We extend our analysis to derive information-theoretic generalization bounds based on Bernstein condition, which leverages a connection between stability and Bernstein condition. We also show that stability can be incorporated into generalization bounds to obtain stronger results using alternative information-theoretic quantities such as the loss difference based CMI, e-CMI and f -CMI. Furthermore, we illustrate the expressiveness of our new CMI notions under the distribution-free setting.

2 Preliminaries

Probability and Information Theory Notation Unless otherwise noted, a random variable will be denoted by a capitalized letter, and its realization by the corresponding lower-case letter. The distribution of a random variable X is denoted by P X , and the conditional distribution of X given Y is denoted by P X | Y . When conditioning on a specific realization y , we use the shorthand P X | Y = y or simply P X | y . Denote by E X expectation over X ∼ P X , and by E X | Y = y (or E X | y ) expectation over X ∼ P X | Y = y . The entropy of a random variable X is denoted by H ( X ) , and the KL divergence of probability distribution P with respect to Q is denoted by D KL ( P || Q ) . The mutual information (MI) between random variables X and Y is denoted by I ( X ; Y ) , and the conditional mutual information between X and Y given Z is denoted by I ( X ; Y | Z ) . We also define the disintegrated mutual information as I z ( X ; Y ) ≜ D KL ( P X,Y | Z = z || P X | Z = z P Y | Z = z ) , following the notation in [40]. Note that I ( X ; Y | Z ) = E Z [ I Z ( X ; Y )] .

Generalization Error and Uniform Stability We consider the supervised learning setting, where we have a domain of instances Z = X × Y , with input and label spaces denoted by X and Y respectively. The distribution of an instance is given by µ , and we have a training sample S = { Z i } n i =1 ∼ µ n . Let R ∈ R be a source of randomness (a random variable independent of S over an appropriate space R ), from which a learning algorithm A : Z n × R → W takes the training sample S and R as input, and outputs a hypothesis W = A ( S, R ) ∈ W . To evaluate the quality of the output hypothesis W , we use a loss function ℓ : W×Z → R + 0 . Given a fixed w , we define the population risk L µ ( w ) ≜ E Z ′ [ ℓ ( w,Z ′ )] , where Z ′ ∼ µ is a testing instance. The quantity L µ = E W [ L µ ( W )] is then the expected population risk. For a fixed w ∈ W , the empirical risk on S is defined as L S ( w ) ≜ 1 n ∑ n i =1 ℓ ( w,Z i ) . Similarly, we define the expected empirical risk as ˆ L n = E W,S [ L S ( W )] . Thus, the expected generalization error is given by E µ ( A ) ≜ L µ -ˆ L n .

We now give two notions of uniform stability [6, 13], where s ≃ s i denotes two training sets s and s i that differ only at the i th element. We say a learning algorithm A is β 1 -weakly uniformly stable if

and β 2 -strongly uniformly stable if

Remark 2.1. Notably the weak uniform stability above is the standard in the literature. It is evident that if A is β -strongly uniformly stable, it must also be β -weakly uniformly stable. It is also worth noting that for deterministic algorithms (e.g., GD or fixed permutation SGD), the two notions are identical. We note that in this paper, when speaking of uniform stability, we refer to the strong notion.

The following well-known result (e.g., [6, Lemma 7], [54, Thm. 13.2]) is frequently used in this paper.

Lemma 2.1. For any algorithm A , we have L µ = E S,S ′ ,R [ 1 n ∑ n i =1 ℓ ( A ( S i , R ) , Z i ) ] , and

Supersample and Sample-Conditioned Hypothesis Stability Following [58], let ˜ Z ∈ Z n × 2 be a supersample matrix with n rows and 2 columns, each entry drawn independently from µ . We index the columns of ˜ Z by 0 , 1 and denote the i th row of ˜ Z by ˜ Z i , with entries ( ˜ Z i, 0 , ˜ Z i, 1 ) . We often use the superscripts + and -to respectively replace the subscripts 0 and 1 , i.e., writing ˜ Z + i for ˜ Z i, 0 and ˜ Z -i for ˜ Z i, 1 . Correspondingly, ˜ Z + [ n ] and ˜ Z -[ n ] denote the first and second columns, respectively. Additionally, we will use ˜ Z + [ n ] ∼ i to denote ˜ Z + [ n ] in which the i th element ˜ Z + i is replaced with the corresponding element ˜ Z -i in the second column.

Let ˜ W + = A ( ˜ Z + [ n ] , R ) , and ˜ W -i = A ( ˜ Z + [ n ] ∼ i , R ) for each i ∈ [ n ] . That is, ˜ W + and each ˜ W -i are obtained by two 'neighboring' training samples, namely those differ only on one instance (the i th instance). We then construct matrix ˜ W ∈ W n × 2 , where the i th row ˜ W i = ( ˜ W + i , ˜ W -i ) = ( ˜ W i, 0 , ˜ W i, 1 ) and ˜ W + i = ˜ W + for all i ∈ [ n ] , as shown in Table 1. Unlike the supersample matrix, elements in ˜ W are identically distributed but not independent. In this case, Eq. (1) can be rewritten as

˜

Z

+

1

˜

Z

-

1

˜

Z

+

2

˜

Z

-

2

.

.

.

.

.

.

˜

Z

+

n

˜

Z

-

n

A

=

˜

W

+

˜

W

-

1

˜

W

+

˜

W

-

2

.

.

.

.

.

.

˜

W

+

˜

W

-

n

The summand in Eq (2) exhibits an interesting 'symmetry': for any i ,

That is, the + / -superscripts in the summand of Eq (2) can be flipped for any i . We may then use n binary ( { 0 , 1 } -valued) variables ( U 1 , U 2 , . . . , U n ) := U to govern whether we choose to flip the signs for each of the n terms, where U i = 0 indicates 'no flipping'. Let U i ≜ 1 -U i , we have

where in Eq. (4) we have chosen U to be an i.i.d. Bernoulli-( 1 2 ) sequence, and in Eq. (5) we have renamed ˜ Z i,U i as ̂ Z i and denoted E ≜ ( ̂ Z 1 , ̂ Z 2 , . . . , ̂ Z n ) . Note that E , induced by U from ˜ Z , contains n instances, each serving to evaluate the loss difference between a pair of hypotheses ( ˜ W -i , ˜ W + ) . The polarities of these evaluations are governed by U .

We now define some new notions of stability.

Definition 2.1 (Sample-Conditioned Hypothesis (SCH) Stability) . Let S ∼ µ n , W and W i generated via W = A ( S, R ) and W i = A ( S i , R ) . Let Z w,w i denote the support of the conditional distribution P Z i | w,w i , and W s denote the support of the conditional distribution P W | s . We introduce four types of SCH stability, referred to types A, B, C, D. Specifically, a learning algorithm A is

where Z ′ in Eq. (7) is an independent instance drawn from µ .

Remark 2.2. By definition, we can see γ 2 ≤ γ 1 , and it is expected that γ 4 ≤ γ 3 . Note that all of them are smaller than β 2 . In addition, it is expected that γ 2 , γ 3 and γ 4 are smaller than β 1 , although the relationship between γ 1 and β 1 is uncertain. Moreover, it is also expected that γ 4 is larger than γ 2 due to the independence of Z ′ in Eq. (7). We emphasize that supreme sup w in Eq. (6) is taken over the sample-dependent hypothesis space ∪ s ∈Z n W s , not the whole hypothesis space W . Notably, the notion of 'hypothesis set stability' in [16] is closely related to our SCH stability. In fact, γ 1 -SCH-A stable implies γ 1 -hypothesis set stable. Due to space constraints, we provide further elaboration on the definition of SCH stability in Appendix B.

3 IOMI Bounds for Stable Algorithms

We are now in a position to give the IOMI bounds for stable learning algorithms.

Theorem 3.1. If the learning algorithm A is γ 1 -SCH-A stable, then

We note that I ( ˜ W + ; ˜ Z + i ) = I ( ˜ W -i ; ˜ Z -i ) = I ( W ; Z i ) , the only difference between the first bound in Theorem 3.1 and the previous individual IOMI bound in [9] is that the sub-Gaussian variance proxy is replaced by the stability parameter γ 1 in our bound. In fact, while we use ˜ W to better understand the appearance of γ 1 in the IOMI bounds, the first bound in Theorem 3.1 itself does not necessarily rely on the supersample and the construction of ˜ W .

In addition, if A is a deterministic algorithm, the term I ( ˜ W + ; ˜ Z + i | ˜ W -i ) in the second bound is tighter than the mutual information stability or erasure information studied in [46, 23], that is, I ( ˜ W + ; ˜ Z + i | ˜ W -i ) ≤ I ( ˜ W + ; ˜ Z + i | ˜ Z + [ n ] \ i ) (see Remark C.1 in the Appendix for an explanation).

Proof Sketch of Theorem 3.1. Motivated by Lemma 2.1 and Eq. (2), the main innovation in this proof is to let the auxiliary function in DV be the 'relative loss' instead of the single loss, namely we let g ( ˜ w + i , ˜ z + i ) = E ˜ W -i | ˜ w + [ ℓ ( ˜ W -i , ˜ z + i ) ] -ℓ ( ˜ w + , ˜ z + i ) and let the auxiliary function f = t · g for t > 0 in Lemma A.2. This enables us to utilize Eq. (6) to bound the CGF. The remaining steps are routine. The complete proof can be found in Appendix C.1.

The following corollary, immediately following from γ 1 ≤ β 2 , suggests a clear improvement over the previous IOMI bound for the uniformly stable deterministic algorithm with vanishing β 2 .

A key message conveyed in this paper, as shown in the proof of Theorem 3.1, is the importance of carefully selecting the DV auxiliary function and appropriate assumptions for various learning scenarios. Corollary 3.1 also suggests a potential enhancement for uniform stability in the certain deterministic setting, provided that I ( W ; Z i ) also vanishes appropriately with n .

Similar to [40, 49, 23, 25], we also give an R -conditioned IOMI bound below.

This disintegrated IOMI bound is not directly comparable to the bounds in Theorem 3.1, but it is equivalent to Corollary 3.1 for deterministic algorithms. With additional assumptions, we may remove the square-root in Theorem 3.1, accelerating the decay of IOMI as shown in Theorem 3.3.

Theorem 3.3. Under the same conditions in Theorem 3.1, if A is further γ 2 -SCH-B stable, then

Notice that γ 2 2 /γ 1 ≤ γ 2 1 /γ 1 = γ 1 . If γ 2 2 /γ 2 1 decays faster than 1 n ∑ n i =1 √ I ( ˜ W + ; ˜ Z + i ) , then Theorem 3.3 is qualitatively strictly stronger than Theorem 3.1.

4 CMI Bounds for Stable Algorithms

Hypotheses-Conditioned CMI Bounds In this section, we give a handful of CMI bounds based on a new information-theoretic quantity.

Theorem 4.1. Suppose that there exists ∆ 1 : W 2 → R such that for every ˜ w i = ( ˜ w + i , ˜ w -i ) , sup z i ∈Z ˜ w + i , ˜ w -i ∣ ∣ ℓ ( ˜ w + i , z i ) -ℓ ( ˜ w -i , z i ) ∣ ∣ ≤ ∆ 1 ( ˜ w i ) . Then,

Furthermore, if A is γ 3 -SCH-C stable, then such a ∆ 1 exists and

Compared to the standard supersample-conditioned CMI bound [58] in terms of I ( W ; U | ˜ Z ) , which assesses how well one can infer the 'training-set membership' from the output hypothesis, our new CMI quantity, namely I ( ̂ Z i ; U i | ˜ W i ) , measures our ability to decide if an instance ̂ Z i contributes to the training of ˜ W + i or to the training of ˜ W -i when we know it contributes to only one of them. When ˜ W + i and ˜ W -i are similar, this decision (i.e., determining U i ) is difficult, giving rise to small I ( ̂ Z i ; U i | ˜ W i ) . Additionally, for uniformly stable algorithms, sup w i ∆ 1 ( ˜ w i ) ≤ β 2 and it is also expected that E ˜ W i [ ∆ 1 ( ˜ W i ) 2 ] ≤ β 2 1 . Moreover, if we simply replace γ 3 by an upper bound of ℓ , the second bound of Theorem 4.1 becomes a uniform convergence bound if I ˜ w ( ̂ Z i ; U i ) vanishes with n .

Proof Sketch of Theorem 4.1. Again we find motivation in Lemma 2.1. Additionally, the symmetry exhibited in Eq. (5), analogous to the symmetry between ˜ Z + i and ˜ Z -i used in deriving the standard CMI bounds, allows a similar development. Specifically, letting g ( ˜ w i , ˆ z i , u i ) = ( -1) u i ( ℓ ( ˜ w -i , ˆ z i ) -ℓ ( ˜ w + i , ˆ z i ) ) and f = t · g for t > 0 in Lemma A.2 enables the bounding of the CGF via invoking the assumptions in the theorem. The complete proof is given in Appendix D.1.

Notably, our new CMI quantity in the bound preserves the boundedness of the original CMI in [58], that is, I ( ̂ Z i ; U i | ˜ W ) ≤ H ( U i ) = log 2 . Furthermore, parallel to supersample-conditioned CMI being smaller than IOMI [18], a similar result for hypotheses-conditioned CMI is given below.

Theorem 4.2. For any A and µ , we have I ( ̂ Z i ; U i | ˜ W i ) ≤ I ( W ; Z i ) .

Similar to Theorem 3.3, we present a CMI bound without square-root, which could be much stronger in certain regimes.

Theorem 4.3. Let ∆ be defined in the same way as in Theorem 4.1, and we let Λ( ˜ w ) =

1 i E ̂ Z i | ˜ w i [ ( ℓ ( ˜ w -i , ̂ Z i ) -ℓ ( ˜ w + i , ̂ Z i ) ) 2 ] / ∆ 1 ( ˜ w i ) 2 , then

If A is further β 2 -uniform stable and γ 4 -SCH-D stable, then

Note that it is valid to set γ 3 = E ˜ W i [ ∆ 1 ( ˜ W i ) ] , which can be viewed as a generalization bound in its own right. Additionally, it can be verified that Λ( ˜ w i ) ≤ 1 for any ˜ w i . As I ˜ W i ( ̂ Z i ; U i ) ≤ log 2 ≈ 0 . 69 , Eq. (10) can be further upper bounded by 1 . 41 γ 3 . This ensures that Eq. (10) will decay no slower than O ( γ 3 ) .

We also present a second moment generalization bound.

Theorem 4.4. Assume that A is β 2 -uniform stable and symmetric with respect to S , i.e. it does not depend on the order of the elements in S . Let ℓ ( · , · ) ∈ [0 , 1] , then

Since I ( E ; U | ˜ W ) ≤ O ( n ) , the bound can be further upper bounded by O ( β 2 2 +1 /n ) , which matches the previous tight bound for the second moment generalization error in [14, Thm. 1.2]. Notice that [14, Thm. 1.2] only holds for the deterministic setting, while our bound also holds for randomized algorithms, and we also give a stronger result in Appendix D.4 based on our SCH stability notions.

Supersample-Conditioned CMI Bounds It is also possible to give ˜ Z -conditioned CMI bounds that explicitly contain the stability parameters. Let W i = ˜ W i,U i and W i = ˜ W i,U i , we have the following results.

Theorem 4.5. Suppose there exists ∆ 2 : Z → R such that sup w,w i ∈W 2 z i ∣ ∣ ℓ ( w,z i ) -ℓ ( w i , z i ) ∣ ∣ ≤ ∆ 2 ( z i ) for every z i , where W 2 z i is the support of the conditional distribution P W,W i | z i . Then,

The proof is deferred to Appendix D.5. The interpretation of the CMI quantity I ( W i , W i ; U i | ˜ Z + i ) appears identical to our earlier CMI quantity I ( ̂ Z i ; U i | ˜ W i ) . A closer look in fact reveals that the two quantities are mathematically equal. To see this, first note I ( ̂ Z i ; U i | ˜ W i ) = H ( U i ) -H ( U i | ̂ Z i , ˜ W i ) and I ( W i , W i ; U i | ˜ Z + i ) = H ( U i ) -H ( U i | ˜ Z + i , W i , W i ) . Let random variable A 1 = ( ̂ Z i , ˜ W + i , ˜ W -i ) and let A 2 = ( ˜ Z + i , W i , W i ) , these two random variables are identically distribution (since P ˜ W + i = P ˜ W -i , P ˜ Z + i = P ˜ Z -i and U i ∼ Bernoulli -( 1 2 ) and also we have P A 1 | U i = P A 2 | U i so P A 1 ,U i = P A 2 ,U i , which gives us H ( U i | A 1 ) = H ( U i | A 2 ) . This indicates that I ( ̂ Z i ; U i | ˜ W i ) = I ( W i , W i ; U i | ˜ Z + i ) .

In Theorem 4.5, a data-dependent hypothesis space W 2 z i is defined. A similar concept has been utilized in the hypothesis set cross-validation (CV) stability studied in [16]. Furthermore, [16] derives some bounds based on either their transductive Rademacher complexity or their hypothesis set CV stability. They show that these two notions dominate in different learning scenarios. Given the close relationship between the supersample construction and the Rademacher complexity [58, 64], and the inspiration behind our ˜ W construction, our framework is likely to have a fundamental connection to [16]. Additionally, obtaining the similar results of ˜ Z -conditioned CMI as in Theorem 4.3-4.4 is warranted, which may require some stability notions analogous to the average CV-stability in [16].

5 Convex-Lipschitz-Bounded (CLB) Problems

We now discuss two examples of the convex-Lipschitz-bounded (CLB) problem, a subclass of SCO problems.

The first example is previously given in [21, Thm. 17], in which nearly all previous informationtheoretic bounds are non-vanishing. We will demonstrate that our CMI bounds are non-vacuous in this example.

Example 1. Let d ∈ N and Z = { e ( i ) : i ∈ [ d ] } where e ( i ) is a one-hot vector with 1 at the i -th coordinate. Let µ = Unif( Z ) . Given a sample S = { Z i } n i =1 drawn i.i.d. from µ , we choose the 1 -Lipschitz convex loss function ℓ ( w,z ) = -⟨ w,z ⟩ and use GD to select a hypothesis w from W = { w ∈ R d : || w || ≤ 1 } . Let the number of GD iterations be T = n 2 and let the learning rate be η = 1 n √ n .

Let ˆ µ = 1 n ∑ n i =1 z i be the sample mean. In this deterministic setting, it's easy to see that

Let ˆ µ i be the sample mean of s i and let w i t be its corresponding hypothesis at time t . Notice that Euclidean projection does not increase the distance between projected points, namely non-expansive [22, Lemma 4.6]. Hence, whether w t = ηt ˆ µ or its truncated version ηt ˆ µ/ || ηt ˆ µ || limited within the unit ball, we have || w t -w i t || ≤ || ηt ˆ µ -ηt ˆ µ i || ≤ O ( ηt/n ) . Recall that the loss function is 1 -Lipschitz, we have |E µ ( A ) | ≤ β 2 ≤ O ( ηt/n ) . In this example, ηT/n = 1 / √ n so β 2 ∈ O (1 / √ n ) . One can also directly obtain this rate from [22, 5].

Now following the same setting in [21], if we let d = 2 n 2 , we can find that I ( W T ; Z i ) ∈ Ω(1) (see [21, Thm. 17] or Appendix F). Thus, IOMI itself could not explain the generalization of GD in this problem. Furthermore, all our CMI quantities including those in Section 6 also have the order of Ω(1) , that is, they fail to vanish as n →∞ (see Appendix F for more elaboration).

Therefore, the stability parameter β 2 should not be replaced by some constant (e.g., the upper bound of the loss function) in the IOMI or CMI bound. In fact, for our CMI bounds, due to their boundedness property, we have the following corollary.

Corollary 5.1 provides a solution to the the non-vanishing limitation of the previous informationtheoretic bounds in Example 1 (and also the counterexample in [21, Thm. 4]), as it can explain generalization as long as the stability-based bound is sufficient. Thus, the shortfalls of informationtheoretic bounds in analyzing deterministic algorithms for CLB problems are tempered by the stability-based framework.

We now show another CLB Example from [21, Thm. 3] where A is not uniformly stable, and we will see information-theoretic bounds in this paper are tight up to a constant. This example is also studied in [42, Sec. 5].

Example 2. Let W ∈ R d be a ball with radius R 0 , and let the input space be Z = { z 0 /R 0 , -z 0 /R 0 } where z 0 ∈ W such that || z 0 || = R 0 . Let µ = Unif( Z ) . Consider a convex and L-Lipschiz loss function ℓ ( w,z ) = -L ⟨ w,z ⟩ . In addition, A is any empirical risk minimization (ERM) algorithm.

In this example, β 2 = 2 LR 0 is a constant. [21] has shown that |E µ ( A ) | ≥ LR 0 √ 2 n and I ( W ; S ) ≤ 1 (see [21, Thm. 3] or Appendix F for an explanation). This gives us 2 LR 0 n ∑ i =1 √ I ( W ; Z i ) ≤ 2 LR 0 √ I ( W ; S ) n ≤ 2 LR 0 √ n . Thus, the distribution-dependent property of IOMI can improve the stability-based bound in this case. In addition, we know that I ( ̂ Z i ; U i | ˜ W i ) ≤ I ( W ; Z i ) from Theorem 4.2, our new CMI bounds are also tight (up to a constant) in this example.

Notably, we can construct an additional example building upon Example 2, where A is uniformly stable but the uniform stability itself results in a slow convergence rate for generalization error. Specifically, let R 0 = 1 d and let d = √ n , then β 2 ∈ O (1 / √ n ) while |E µ ( A ) | ≥ L √ 2 n . Note that the information-theoretic bounds in this paper can still provide a tight rate, namely O (1 /n ) .

These examples demonstrate that our bounds can improve both the stability-based bound and information-theoretic bounds in some learning scenarios.

Additional applications of our bounds are discussed in Appendix G.

6 Extensions

Connection with Bernstein Condition The Bernstein condition is commonly used to derive fastrate generalization bound for both PAC-Bayes bounds [69, 10, 36, 17] and stability-based bounds [28], then it is natural to explore the relationship between our fast-rate bounds and the Bernstein condition, formally defined below.

Definition 6.1 (Bernstein Condition) . Assume that w ∗ = arg min w ∈W L µ ( w ) is a risk minimizer. We say that the Bernstein assumption is satisfied with some B > 0 and κ ∈ [1 , + ∞ ) if for any w ∈ W ,

This condition can be easily satisfied in many common situations [1, 28]. In the following proposition, we can see that the Bernstein condition implies the γ 2 -SCH-B stability.

Proposition 1. If the Bernstein condition is satisfied with some B and κ , then A is γ 2 -SCH-B stable, where γ 2 2 = 4 B E W [ ( L µ ( W ) -L µ ( w ∗ )) 1 κ ] .

Therefore, invoking the Bernstein condition, we can obtain the fast-rate bound as presented in Theorem 3.3, where we need to assume the loss is bounded, as shown below as a by-product.

Corollary 6.1. If the Bernstein condition is satisfied with κ = 1 and ℓ ∈ [0 , C ] , then

Recently, [65, 66] use the unexpected excess risk as the DV auxiliary function and invoke the ( η, c ) -central condition to establish some optimal-rate bounds for specific learning problems, e.g., Gaussian mean estimation. This again highlights the significance of selecting appropriate DV auxiliary functions and corresponding assumptions tailored to different learning problems. It is worth mentioning that the Bernstein condition also implies their ( η, c ) -central condition. Therefore, unifying these conditions can be considered as a potential avenue for future research.

Loss Difference, Evaluated, and Functional CMI for Stable Algorithms Similar to [64], we derive some tighter bounds based on the loss difference.

Theorem 6.1. Let ∆ L i = ℓ ( W i , ˜ Z + i ) -ℓ ( W i , ˜ Z + i ) . If A is β 2 -uniform stable, then

We note that while it is feasible to replace β 2 in the (disintegrated) CMI bounds above with certain sample-conditioned hypothesis stability, it is not possible to apply the same substitution for the unconditional MI bound in Theorem 6.1.

Furthermore, notice that ∆ L i -( L i , ¯ L i ) -( F i , F i ) -( W i , W i ) forms a Markov chain given ˜ Z + i , wherein ( L i , ¯ L i ) are the loss pair evaluated at ˜ Z + i using ( W i , W i ) , and ( F i , F i ) are label predictions of ˜ Z + i using ( W i , W i ) . By the data-processing inequality, one can obtain e-CMI bound [58, 25], f -CMI bound [23] and recover I ( W i , W i ; U i | ˜ Z + i ) based bound from Theorem 6.1: I (∆ L i ; U i | ˜ Z + i ) ≤ I ( L i , ¯ L i ; U i | ˜ Z + i ) ≤ I ( F i , F i ; U i | ˜ Z + i ) ≤ I ( W i , W i ; U i | ˜ Z + i ) . Additionally, notice that we can also apply the similar technique for the hypotheses-conditioned CMI, which should give the same results.

Expressiveness of New CMI Notions Under Distribution-Free Setting Previous works [58, 19, 20, 23, 25] have demonstrated that the CMI framework is expressive enough to establish connections with VC theory in the distribution-free learning setting. Here, we further illustrate the expressiveness of the sample-conditioned CMI discussed in this work.

Theorem 6.2. Let Z = X × { 0 , 1 } , and let F = { f w : X → { 0 , 1 }| w ∈ W} be a functional hypothesis class with finite VC dimension d . Let n > d + 1 , for any algorithm A , we have 1 n ∑ n i =1 √ I ( F i , ¯ F i ; U i | ˜ Z + i ) ≤ O (√ d n log ( n d ) ) .

Like the previous works, this bound matches the classic result of the uniform convergence bound [59]. Notice that the result could be extended to multi-class classification with finite Natarajan dimension [39] by proceeding similarly to [25, Thm. 8].

We invoke Theorem 6.2 to demonstrate that our new information-theoretic quantities have the same expressive power as standard CMI quantities. The expressiveness result for the (functional) hypotheses-conditioned CMI is expected to align with Theorem 6.2. This alignment is due to the equivalence between hypotheses-conditioned CMI and supersample-conditioned CMI, as discussed in Theorem 4.5.

7 Related Works and Additional Discussions

Stability-Based Framework vs. Information-Theoretic Framework Using stability methods to analyze generalization errors can be traced back to several seminal works, such as [51, 11, 12,

34, 27]. It is worth noting that stability arguments have proven particularly effective in analyzing the learnability of SCO problems [56], where traditional uniform convergence bounds may not be sufficient to explain the generalization behavior. The application of stability approaches has gained popularity for providing high-probability guarantees since the work of [6]. Recent advancements have further sharpened the convergence rates of high-probability generalization upper bounds for uniformly stable algorithms in a series of works [14, 15, 8, 28]. While information-theoretic bounds are commonly used to analyze the in-expectation generalization, it is expected that combining them with the stability framework will yield sharper high-probability bounds.

Additionally, we note that in the realizable setting, where A is an interpolating algorithm, informationtheoretic bounds exhibit greater power than stability-based bounds. For instance, in the case of the 0 -1 loss, information-theoretic bounds can achieve the known optimal minimax rates [19] and even provide exact characterizations of the generalization error [20, 64]. However, it should be noted that due to the inherent fitting-stability tradeoff property [54, Sec. 13.4], interpolating algorithms tend to be unstable, rendering stability arguments inapplicable. Given the prevalence of zero empirical risk in modern deep learning [68], it is natural to question whether information-theoretic bounds require the stability-based approach for analyzing non-convex (and potentially non-smooth and non-Lipschitz continuous) learning scenarios.

Connection Between Two Frameworks in Previous Works The connection between informationtheoretic bounds, including some PAC-Bayes bounds, and algorithmic stability has been explored in previous literature [46, 33, 48, 29, 58, 23, 3, 47]. These works primarily focus on either regarding the information-theoretic quantities as notions of distributional stability [46, 67, 29, 58, 3] and/or converting information-theoretic quantities to some other algorithmic stability notions [58, 23, 47]. The later often relies on the addition of Gaussian noise to the hypotheses (or assuming that the prior and posterior distributions are Gaussian in PAC-Bayes). In [33], the authors also combine the DV formula with stability assumptions, where they derive some PAC-Bayes bounds. However, comparing these bounds with others is challenging in general due to the presence of their hyperparameter stability.

Comparison with Standard CMI While our IOMI quantity aligns with the previous work [9], the new CMI quantity I ( ̂ Z i ; U i | ˜ W i ) (or equivalently I ( W i , W i ; U i | ˜ Z + i ) ) may not be directly comparable to the standard individual CMI I ( W ; U i | ˜ Z i ) in [49, 70]. Specifically, we have I ( ̂ Z i ; U i | ˜ W i ) = H ( U i ) -H ( U i | ̂ Z i , ˜ W i ) and I ( W ; U i | ˜ Z i ) = H ( U i ) -H ( U i | W, ˜ Z i ) . The relationship between H ( U i | ̂ Z i , ˜ W i ) and H ( U i | W, ˜ Z i ) is not trivial. Exploring and quantitatively comparing these CMI measures would be an intriguing research direction.

Leave-One-Out CMI Our construction of ˜ W bears resemblance to the leave-one-out (LOO) setting, where there are also n +1 hypotheses. It is worth noting that LOO-CMI has been recently proposed in concurrent works [20, 47]. In LOO-CMI, the supersample ˜ Z = Z [ n +1] consists of n +1 instances, and U is an index uniformly drawn from [ n +1] to select one hold-out instance. Consequently, Z U represents the testing data, while Z [ n +1] \ U serves as the training sample. In this context, LOO-CMI can be defined as I ( W ; U | ˜ Z ) . Notice that this quantity still fails to explain the generalization in Example 1. The LOO setting is often associated with the stability-based framework [6], and it is expected that the n + 1 -supersample induced ˜ W could yield new CMI bounds that also contain the SCH stability notions. Nevertheless, in this paper, we do not adopt the LOO setting because in that case, H ( U ) = log ( n +1) , the LOO-CMI bound is no longer upper bounded by a constant independent of n (note that the LOO-CMI bound for general setting in [20, Thm. 2.5] does not contain the 1 / √ n factor).

8 Concluding Remarks

We propose a novel construction of the hypothesis matrix and a new family of stability notions called sample-conditioned hypothesis stability. Leveraging these concepts, we derive sharper informationtheoretic bounds for stable learning algorithms. Several promising avenues for future research include comparing our new CMI quantities with the standard CMI in a quantitative manner, analyzing the generalization of gradient-based optimization algorithms like SGD using our bounds, and establishing new high-probability generalization guarantees. Further discussions can be found in Appendix H.

References

Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv preprint arXiv:2110.11216 , 2021.

Amir Asadi, Emmanuel Abbe, and Sergio Verdú. Chaining mutual information and tightening generalization bounds. Advances in Neural Information Processing Systems , 31, 2018.

Arindam Banerjee, Tiancong Chen, Xinyan Li, and Yingxue Zhou. Stability based generalization bounds for exponential family langevin dynamics. In International Conference on Machine Learning , pages 1412-1449. PMLR, 2022.

Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory . PMLR, 2018.

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33, 2020.

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

Olivier Bousquet, Steve Hanneke, Shay Moran, and Nikita Zhivotovskiy. Proper learning, helly number, and an optimal svm bound. In Conference on Learning Theory , pages 582-609. PMLR, 2020.

Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory , pages 610-626. PMLR, 2020.

Yuheng Bu, Shaofeng Zou, and Venugopal V Veeravalli. Tightening mutual information based bounds on generalization error. In 2019 IEEE International Symposium on Information Theory (ISIT) , pages 587-591. IEEE, 2019.

O Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning. Vol. 56. Lecture Notes - Monograph Series. Institute of Mathematical Statistics , 2007.

Luc Devroye and Terry Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Transactions on Information Theory , 25(2):202-207, 1979.

Luc Devroye and Terry Wagner. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory , 25(5):601-604, 1979.

Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil, and Leslie Pack Kaelbing. Stability of randomized learning algorithms. Journal of Machine Learning Research , 6(1), 2005.

Vitaly Feldman and Jan Vondrak. Generalization bounds for uniformly stable algorithms. Advances in Neural Information Processing Systems , 31, 2018.

Vitaly Feldman and Jan Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory , pages 1270-1279. PMLR, 2019.

Dylan J Foster, Spencer Greenberg, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Hypothesis set stability and generalization. Advances in Neural Information Processing Systems , 32, 2019.

Peter Grunwald, Thomas Steinke, and Lydia Zakynthinou. Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vc classes. In Conference on Learning Theory . PMLR, 2021.

Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. Advances in Neural Information Processing Systems , 2020.

Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. Towards a unified information-theoretic framework for generalization. Advances in Neural Information Processing Systems , 34:26370-26381, 2021.

Mahdi Haghifam, Shay Moran, Daniel M Roy, and Gintare Karolina Dziugiate. Understanding generalization via leave-one-out conditional mutual information. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 2487-2492. IEEE, 2022.

Mahdi Haghifam, Borja Rodríguez-Gálvez, Ragnar Thobaben, Mikael Skoglund, Daniel M Roy, and Gintare Karolina Dziugaite. Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization. In International Conference on Algorithmic Learning Theory , pages 663-706. PMLR, 2023.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning , pages 12251234. PMLR, 2016.

Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Informationtheoretic generalization bounds for black-box learning algorithms. In Advances in Neural Information Processing Systems , 2021.

Fredrik Hellström and Giuseppe Durisi. Fast-rate loss bounds via conditional information measures with applications to neural networks. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 952-957. IEEE, 2021.

Fredrik Hellström and Giuseppe Durisi. A new family of generalization bounds using samplewise evaluated CMI. In Advances in Neural Information Processing Systems , 2022.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. In Proceedings of the tenth annual conference on Computational learning theory , pages 152-162, 1997.

Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate o (1 /n ) . Advances in Neural Information Processing Systems , 34:5065-5076, 2021.

Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. In International Conference on Learning Representations , 2020.

Nick Littlestone. Relating data compression and learnability. Technical report , 1986.

Roi Livni. Information theoretic lower bounds for information theoretic upper bounds. arXiv preprint arXiv:2302.04925 , 2023.

Roi Livni and Shay Moran. A limitation of the pac-bayes framework. Advances in Neural Information Processing Systems , 33:20543-20553, 2020.

Ben London. A pac-bayesian analysis of randomized learning with application to stochastic gradient descent. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2935-2944, 2017.

Gábor Lugosi and Miroslaw Pawlak. On the posterior-probability estimate of the error rate of nonparametric classification rules. IEEE Transactions on Information Theory , 40(2):475-481, 1994.

Colin McDiarmid. Concentration , pages 195-248. Springer Berlin Heidelberg, 1998.

Zakaria Mhammedi, Peter Grünwald, and Benjamin Guedj. Pac-bayes un-expected bernstein inequality. Advances in Neural Information Processing Systems , 32, 2019.

Michael Mitzenmacher and Eli Upfal. Probability and Computing: Randomization and Probabilistic Techniques in Algorithms and Data Analysis . Cambridge University Press, 2017.

Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints. In Conference on Learning Theory , pages 605-638. PMLR, 2018.

Balas K Natarajan. On learning sets and functions. Machine Learning , 4:67-97, 1989.

Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems , 2019.

Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. Informationtheoretic generalization bounds for stochastic gradient descent. In Conference on Learning Theory . PMLR, 2021.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy, iterative algorithms. In 2018 IEEE International Symposium on Information Theory (ISIT) . IEEE, 2018.

Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. Lecture Notes for 6.441 (MIT), ECE 563 (UIUC), STAT 364 (Yale), 2019. , 2019.

Tiberiu Popoviciu. Sur les équations algébriques ayant toutes leurs racines réelles. Mathematica , 9(129-145):20, 1935.

Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu. Informationtheoretic analysis of stability and bias of learning algorithms. In 2016 IEEE Information Theory Workshop (ITW) , pages 26-30. IEEE, 2016.

Mohamad Rida Rammal, Alessandro Achille, Aditya Golatkar, Suhas Diggavi, and Stefano Soatto. On leave-one-out conditional mutual information for generalization. In Advances in Neural Information Processing Systems , 2022.

Omar Rivasplata, Emilio Parrado-Hernández, John S Shawe-Taylor, Shiliang Sun, and Csaba Szepesvári. Pac-bayes bounds for stable algorithms with instance-dependent priors. Advances in Neural Information Processing Systems , 31, 2018.

Borja Rodríguez-Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. On random subset generalization error bounds and the stochastic gradient langevin dynamics algorithm. In 2020 IEEE Information Theory Workshop (ITW) , pages 1-5. IEEE, 2021.

Borja Rodríguez Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. Tighter expected generalization error bounds via wasserstein distance. Advances in Neural Information Processing Systems , 34, 2021.

William H Rogers and Terry J Wagner. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics , pages 506-514, 1978.

Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics . PMLR, 2016.

Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A , 13(1):145-147, 1972.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In COLT 2009 - The 22nd Conference on Learning Theory , 2009.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research , 11:2635-2670, 2010.

Saharon Shelah. A combinatorial problem; stability and order for models and theories in infinitary languages. Pacific Journal of Mathematics , 41(1):247-261, 1972.

Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory . PMLR, 2020.

Vladimir Vapnik. Statistical learning theory . Wiley, 1998. ISBN 978-0-471-03003-4.

Hao Wang, Yizhe Huang, Rui Gao, and Flavio Calmon. Analyzing the generalization capability of sgld using properties of gaussian channels. Advances in Neural Information Processing Systems , 34:24222-24234, 2021.

Ziqiao Wang and Yongyi Mao. On the generalization of models trained with SGD: Informationtheoretic bounds and implications. In International Conference on Learning Representations , 2022.

Ziqiao Wang and Yongyi Mao. Two facets of sde under an information-theoretic lens: Generalization of sgd via training trajectories and via terminal states. arXiv preprint arXiv:2211.10691 , 2022.

Ziqiao Wang and Yongyi Mao. Information-theoretic analysis of unsupervised domain adaptation. In International Conference on Learning Representations , 2023.

Ziqiao Wang and Yongyi Mao. Tighter information-theoretic generalization bounds from supersamples. In International Conference on Machine Learning . PMLR, 2023.

Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu. Fast rate generalization error bounds: Variations on a theme. In 2022 IEEE Information Theory Workshop (ITW) , pages 43-48. IEEE, 2022.

Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu. On the tightness of information-theoretic bounds on generalization error of learning algorithms. arXiv preprint arXiv:2303.14658 , 2023.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems , 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017.

Tong Zhang. Information-theoretic upper and lower bounds for statistical estimation. IEEE Transactions on Information Theory , 52(4):1307-1321, 2006.

Ruida Zhou, Chao Tian, and Tie Liu. Individually conditional individual mutual information bound on generalization error. IEEE Transactions on Information Theory , 68(5):3304-3316, 2022.

Ruida Zhou, Chao Tian, and Tie Liu. Exactly tight information-theoretic generalization error bound for the quadratic gaussian problem. arXiv preprint arXiv:2305.00876 , 2023.

Appendices

A Some Useful Lemmas

In this paper, there are some equivalent forms of the generalization error we will study, e.g., Eq. (2) and Eq. (5) in the main text, which are presented in the following lemma.

Lemma A.1. Let W i = ˜ W i,U i and W i = ˜ W i,U i . For any learning algorithm A , the following equations hold

Proof. This lemma is a consequence of Lemma 2.1, with further utilizing some symmetric properties. Recall Eq. (1) in Lemma 2.1,

Note that Eq. (2) in the main text is from the second equation above, which is used to derive individual IOMI bounds in Section 3.

Similar to the standard setting for CMI bounds, where the role of each ˜ Z + i and ˜ Z -i can be exchanged, a key observation here is that for each i , ˜ W + i and ˜ W -i can also be exchanged arbitrarily. That is to say,

also holds true. Notice that we do not change the definitions of any the random variable, e.g., ˜ W + = A ( ˜ Z + [ n ] , R ) and ˜ W -i = A ( ˜ Z + [ n ] ∼ i , R ) .

What differs from the standard CMI is that the roles of the whole sequences ˜ Z + [ n ] and ˜ Z -[ n ] are not exchangeable with each other. Here, when we exchange each ˜ Z + i and ˜ Z -i , we need to keep the other positions in S unchanged.

By introducing U i ∼ Unif( { 0 , 1 } ) , we have

To obtain Eq. (13), notice that ̂ Z i = ˜ Z i,U i , we have

This, as we have already seen in Eq. (5) in the main text, is used to derive hypotheses-conditioned CMI bounds in Section 4. It's easy to see that when U i = 0 , Eq. (16) becomes Eq. (12), and when U i = 1 , we obtain Eq. (15) via Eq. (16).

To obtain Eq. (14), we let W i = ˜ W i,U i , W i = ˜ W i,U i , and fix ̂ Z i = ˜ Z + i . Similarly,

This is used to derive supersample-conditioned CMI bounds in Section 4. It's easy to see that both U i = 0 and U i = 1 will give us Eq. (12).

Like all the previous information-theoretic bounds, the following lemma is widely used in our paper. LemmaA.2 (Donsker-Varadhan (DV) variational representation of KL divergence [44, Theorem 3.5]) . Let Q , P be probability measures on Θ , for any bounded measurable function f : Θ → R , we have D KL ( Q || P ) = sup f E θ ∼ Q [ f ( θ )] -ln E θ ∼ P [exp f ( θ )] .

We also invoke some other lemmas as given below.

Lemma A.3 (Hoeffding's Lemma [26]) . Let X ∈ [ a, b ] be a bounded random variable with mean µ . Then, for all t ∈ R , we have E [ e tX ] ≤ e tµ + t 2 ( b -a ) 2 8 .

Lemma A.4 (Popoviciu's inequality [45]) . Let M and m be upper and lower bounds on the values of any random variable X , then Var( X ) ≤ ( M -m ) 2 4 .

The following lemma is from [35, Lemma 2.8], we provide a self-contained proof.

Lemma A.5. Let h ( x ) = e x -x -1 x 2 be the Bernstein function. If a random variable X satisfies E [ X ] = 0 and X ≤ b , then E [ e X ] ≤ e h ( b ) E [ X 2 ] .

Proof. It's easy to verify that h ( x ) is an increasing function for x > 0 . Thus, h ( x ) ≤ h ( b ) for x ≤ b . Then,

For the bounded random variable X with zero mean, we have

x

The last inequality is by e ≥ x +1 . This completes the proof.

B Further Elaborations on SCH Stability

We note that the reason we introduce four types of SCH stability in Definition 2.1 is that solely using β 2 in our bounds might be too loose, as it considers the supremum over all sources of randomness. By incorporating SCH stabilities, we aim to demonstrate that theoretically, we can achieve significantly tighter stability parameters.

The basic set up is as follows. Assume a random sample S gives rise to W . For each Z i ∈ S , we construct S i by replacing Z i with another independently drawn instance; call training result W i , the neighbor of W .

In a), γ 1 -SCH-A stability measures the difference between the loss of w and the expected loss of its neighbor W i at a worst z and the worst possible w . While in (b), γ 2 -SCH-B stability measures the

square of this difference, not in the worst case, but in an average case, where the average is over an independently Z ′ for the loss evaluation, the training sample, and the algorithm randomness. Since 'average is smaller than worst', γ 2 ≤ γ 1 .

In c), we consider the difference between the loss of W and the loss of its neighbor when evaluated at the worst possible Z i that when included in S gives rise to W . The expected value of this difference is γ 3 -SCH-C stability.

In d), γ 4 -SCH-D stability measures the expected squared difference between the loss of W and the loss of its neighbor when evaluated at Z i (a member of S ). For a similar 'average smaller worst' reason, one expects that γ 4 ≤ γ 3 .

We expect that γ 2 , γ 3 , and γ 4 are all smaller than β 1 . This is because in β 1 , we consider the worst evaluated instance, whereas in the other cases, we take the expectation over all instances. Additionally, in Theorem 4.1, we expect that E ˜ W i ∆ 1 ( ˜ W i ) 2 ≤ β 2 1 , this is because β 1 -stability holds for all the possible s and s i , namely it holds for all the ( w,w i ) pair (that shares the same randomness) while in E ˜ W i ∆ 1 ( ˜ W i ) 2 , we take the expectation of these pairs.

We expect γ 2 ≤ γ 4 due to the following reason: first by Jensen's inequality, we have E S,R,Z ′ [ ℓ ( W,Z ′ ) -E W i | W ℓ ( W i , Z ′ ) ] 2 ≤ E W,W i ,Z ′ [ ℓ ( W,Z ′ ) -ℓ ( W i , Z ′ ) ] 2 , then since Z ′ is an independent of both W and W ′ , Z ′ can be regarded as a testing point for both W and W ′ , we could expect that the expectation of ℓ ( W,Z ′ ) -ℓ ( W i , Z ′ ) is small. While in E S,Z ′ i ,R [ ℓ ( W,Z i ) -ℓ ( W i , Z i ) ] 2 , Z i is a training point for obtaining W , so ℓ ( W,Z i ) could be small in general, and Z i is a testing point for W i . Therefore, it is reasonable to expect E W,W i ,Z ′ [ ℓ ( W,Z ′ ) -ℓ ( W i , Z ′ ) ] 2 ≤ E S,Z ′ i ,R [ ℓ ( W,Z i ) -ℓ ( W i , Z i ) ] 2 , namely γ 2 ≤ γ 4 .

As a concrete example, let ℓ be zero-one loss and assume A is an interpolating algorithm and and randomly makes predictions for unseen data. By Jensen's inequality, γ 2 2 ≤ E W,W i ,Z ′ [ ℓ ( W,Z ′ ) -ℓ ( W i , Z ′ ) ] 2 = E W,Z ′ [ ℓ ( W,Z ′ )] -2 E W,W i ,Z ′ [ ( ℓ ( W,Z ′ ) ℓ ( W i , Z ′ )) ] + E W i ,Z ′ [ ℓ ( W i , Z ′ ) ] 2 , where we use ℓ 2 = ℓ for zero-one loss. Since Z ′ is an unseen data for both W and W i , we have γ 2 2 ≤ E W i ,Z ′ [ ℓ ( W i , Z ′ ) ] 2 + 1 2 -1 2 = E W i ,Z ′ [ ℓ ( W i , Z ′ ) ] 2 . While in this case γ 2 4 = E W i ,Z i [ ℓ ( W i , Z i ) ] 2 so γ 2 ≤ γ 4 .

C Omitted Proofs and Additional Discussions in Section 3

C.1 Proof of Theorem 3.1

Proof. Let g ( ˜ w + , ˜ z + i ) = E ˜ W -i | ˜ w + [ ℓ ( ˜ W -i , ˜ z + i ) ] -ℓ ( ˜ w + , ˜ z + i ) be the average loss difference between ˜ w + and its neighboring hypothesis, and let f = t · g for t > 0 in Lemma A.2. Let ˜ Z + ′ i be an independent copy of ˜ Z + i , then

Since ˜ Z + ′ i is independent of both ˜ W -i and ˜ W + , and ˜ W -i and ˜ W + are identically distributed, we have

By the definition of γ 1 -SCH-A stability,

so g ( ˜ W + , ˜ Z + ′ i ) is a zero-mean random variable bounded in [ -γ 1 , γ 1 ] . By Lemma A.3, we have

where the last equality is obtained by optimizing the bound over t , i.e. letting t = √ I ( ˜ W + ; ˜ Z + i ) 2 γ 2 1 .

Recall Eq. (12) in Lemma A.1 and applying Jensen's inequality to the absolute function, the first bound is then obtained by

Furthermore, by the chain rule of mutual information,

Notice that I ( ˜ W -i ; ˜ Z + i ) = 0 in the RHS, we have

which will give us the second bound. This concludes the proof.

Remark C.1 (Comparison with Mutual Information Stability [46, 23]) . To compare with the mutual information stability I ( ˜ W + ; ˜ Z + i | ˜ Z + [ n ] \ i ) , recall Eq.(18): I ( ˜ W + ; ˜ Z + i | ˜ W -) = I ( ˜ W -i ; ˜ Z + i | ˜ W + ) + I ( ˜ W + ; ˜ Z + i ) , and similarly we also have I ( ˜ W + ; ˜ Z + i | ˜ Z + [ n ] \ i ) = I ( ˜ Z + [ n ] \ i ; ˜ Z + i | ˜ W + ) + I ( ˜ W + ; ˜ Z + i ) .

Thus, we only need to compare I ( ˜ W -i ; ˜ Z + i | ˜ W + ) and I ( ˜ Z + [ n ] \ i ; ˜ Z + i | ˜ W + ) . Notice that for a deterministic A , we have I ( ˜ W -i ; ˜ Z + i | ˜ W + ) ≤ I ( ˜ Z + [ n ] \ i , ˜ Z -i ; ˜ Z + i | ˜ W + ) . Since ˜ Z -i ⊥ ⊥ ( ˜ W + , ˜ Z + [ n ] ) , we further have I ( ˜ Z + [ n ] \ i , ˜ Z -i ; ˜ Z + i | ˜ W + ) = I ( ˜ Z + [ n ] \ i ; ˜ Z + i | ˜ W + ) , which gives us the desired result:

C.2 Proof of Theorem 3.2

Proof. The proof is nearly the same to the proof of Theorem 3.1, except that now the randomness of the algorithm is given for each DV auxiliary function, so the randomness of ˜ W i is completely controlled by ˜ Z .

Let g ( ˜ w + , ˜ z + i , r ) = E ˜ W -i | ˜ w + ,r [ ℓ ( ˜ W -i , ˜ z + i ) ] -ℓ ( ˜ w + , ˜ z + i ) and let f = t · g for t > 0 in Lemma A.2. Let ˜ Z + ′ i be an independent copy of ˜ Z + i , then

Notice that

still holds since ˜ Z + i and ˜ Z -i are i.i.d. drawn.

Thus, g ( ˜ W + , ˜ Z + ′ i , r ) is a zero-mean random variable bounded in [ -β 2 , β 2 ] . By Lemma A.3, the remaining part is routine:

Thus,

This completes the proof.

C.3 Proof of Theorem 3.3

Proof. Let h ( x ) = e x -x -1 x 2 be the Bernstein function. Similar to the proof of Theorem 3.1, we let g ( ˜ w + , ˜ z + i ) = E ˜ W -i | ˜ w + [ ℓ ( ˜ W -i , ˜ z + i ) ] -ℓ ( ˜ w + , ˜ z + i ) . We have already known that E ˜ W + , ˜ Z + ′ i [ g ( ˜ W + , ˜ Z + ′ i ) ] = 0 and ∣ ∣ ∣ g ( ˜ W + , ˜ Z + ′ i ) ∣ ∣ ∣ ≤ γ 1 . By Lemma A.5,

where the second inequality is by the definition of γ 2 -SCH-B stability. Plugging the above into Eq. (17),

Usually we have γ 2 2 ≤ γ 2 1 ≤ γ 1 , we let t = 1 /γ 1 , then

Thus,

This concludes the proof.

D Omitted Proofs in Section 4

D.1 Proof of Theorem 4.1

Proof. We now prove the first bound. Let g ( ˜ w i , ˆ z i , u i ) = ( -1) u i ( ℓ ( ˜ w -i , ˆ z i ) -ℓ ( ˜ w + i , ˆ z i ) ) . By Lemma A.2, we have

Since U ′ i ⊥ ⊥ ̂ Z i , we have E U ′ i [ g ( ˜ w i , ˆ z i , U ′ i )] = E U ′ i [ ( -1) U ′ i ( ℓ ( ˜ w -i , ˆ z i ) -ℓ ( ˜ w + i , ˆ z i ) ) ] = 0 for any ˜ w i and ˆ z i . Ergo,

By the definition of ∆ 1 ( ˜ w i ) ,

Thus, g ( ˜ w i , ̂ Z i , U ′ i ) is a zero-mean random variable bounded in [ -∆ 1 ( ˜ w i ) , ∆ 1 ( ˜ w i )] for a fixed ˜ w i . By Lemma A.3, we have

Plugging the above into Eq. (19),

Recall Eq. (14) in Lemma A.1 and by Jensen's inequality for the absolute function, the first bound is obtained:

To prove the second bound, we return to Eq. (20), and take expectation over ˜ W i first. By Jensen's inequality,

Therefore, we have the second bound as below

For the second part of Theorem 4.1, notice that it's valid to let γ 3 = E ˜ W i [ ∆( ˜ W i ) ] , then recall Eq. (21),

This completes the proof.

D.2 Proof of Theorem 4.2

Proof. The proof is similar to [18, Theorem 2.1]. By the chain rule,

Since H ( ̂ Z i | U i , ˜ W i ) = H ( ̂ Z i | W i , U i , ˜ W i ) = H ( ̂ Z i | W i ) , we have I ( ̂ Z i ; U i , ˜ W i ) = H ( ̂ Z i ) -H ( ̂ Z i | U i , ˜ W i ) = H ( ̂ Z i ) -H ( ̂ Z i | W i ) = I ( ̂ Z i ; W i ) . Thus, I ( ̂ Z i ; U i , ˜ W i ) = I ( ̂ Z i ; W i ) . Recall Eq. (24) and by the non-negativity of mutual information, we have I ( ̂ Z i ; U i | ˜ W i ) ≤ I ( W i ; ̂ Z i ) . Note that I ( W i ; ̂ Z i ) = I ( ˜ W + i ; ˜ Z + i ) = I ( W ; Z i ) . This completes the proof.

D.3 Proof of Theorem 4.3

Proof. We first return to Eq. (19) in the previous proof, and we have already known that g ( ˜ w i , ̂ Z i , U ′ i ) is a zero-mean random variable bounded in [ -∆ 1 ( ˜ w i ) , ∆ 1 ( ˜ w i )] for a fixed ˜ w i .

Plugging the above into Eq. (19),

Let t = 1 ∆ 1 ( ˜ w i ) , we have

For the second part, if A is further β 2 -uniform stable, recall Eq. (25) and by the non-decreasing property of h , we have

Let t = 1 β 2 and taking expectation over ˜ W i , we have

where the equality is by the definition of γ 4 -SCH-D stability. Thus,

This concludes the proof.

D.4 Proof of Theorem 4.4

We present a stronger version of Theorem 4.4.

Theorem D.1. Under the same conditions in Theorem 4.1, and we further assume that A is γ 2 -SCH-B stable and symmetric with respect to S , i.e. it does not depend on the order of the elements in the training sample. Let ¯ ∆ 1 ( ˜ W ) = 1 n ∑ n i =1 ∆ 1 ( ˜ W i ) 2 , we have

Then Theorem 4.4 is a corollary of Theorem D.1.

Proof of Theorem 4.4. For β 2 -uniform stable algorithm, by ¯ ∆ 1 ( ˜ W ) ≤ β 2 2 and γ 2 2 ≤ β 2 2 , we have

This completes the proof.

Before we prove Theorem D.1, we need to first obtain the following lemma.

Lemma D.1. Under the same conditions in Theorem 4.1, let ¯ ∆ 1 ( ˜ W ) = 1 n ∑ n i =1 ∆ 1 ( ˜ W i ) 2 , we have

Proof of Lemma D.1. Here we borrow some proof techniques used in [58, Thm. 2].

u ( -+ )

Let g ( ˜ w, ˆ z i , u i ) = ( -1) i ℓ ( ˜ w i , ˆ z i ) -ℓ ( ˜ w i , ˆ z i ) and let G ∼ N (0 , 1) be an independent standard Gaussian random variable. Let f = t · ( 1 n ∑ n i =1 g ) 2 in Lemma A.2, then

where Eq. (26) is by the moment generating function of Gaussian distribution: E G [ e λG ] = e λ 2 2 for all λ ∈ R , Eq. (27) is by Lemma A.3 and Eq. (28) is by the moment generating function of chi-squared distribution: E G [ e λG 2 ] ≤ 1 √ 1 -2 λ for λ < 1 2 .

Let t = n 2 3 ∑ n i =1 ∆ 1 ( ˜ w i ) 2 be substituted to the last equation above, we have

Let ¯ ∆ 1 ( ˜ w ) = 1 n ∑ n i =1 ∆ 1 ( ˜ w i ) 2 , and taking expectation over ˜ W for both sides,

Applying Jensen's inequality to the square function, we have

Combining Eq. (30) with the inequality above will concludes the proof.

We are now in a position to prove Theorem D.1.

Proof of Theorem D.1.

where the last inequality is by ( x + y ) 2 ≤ 2 x 2 + 2 y 2 . Notice that B 1 can be bounded by using Lemma D.1. We now focus on B 2 . Since E Z ′ [ ℓ ( W,Z ′ )] = 1 n ∑ n i =1 E Z ′ [ ℓ ( W,Z ′ )] , we have

For B 3 , we apply Jensen's inequality to move the expectation over W i outside of the square function,

Notice that S ′ , S and R are all independent with each other (so W i , Z i and Z ′ i are also independent with each other). If we further let A be symmetric, namely W i does not dependent on i , then the inequality above is equivalent to

Hence, the inner expectation in the RHS above is just the variance of the sample mean of n i.i.d bounded random variables. Recall that ℓ ( · , · ) ∈ [0 , 1] , thereby

where the second inequality is by Lemma A.4.

Then, for B 4 , we also apply Jensen's inequality to the square function, and by the definition of γ 2 -SCH-B stability, we have

Putting everthing together, we have

where the last inequality is by Lemma D.1. This completes the proof.

D.5 Proof of Theorem 4.5

Proof. Let g (˜ z + i , w i , ¯ w i , u i ) = ( -1) u i ( ℓ ( ¯ w i , ˜ z + i ) -ℓ ( w i , ˜ z + i ) ) . Again, by Lemma A.2, we have

Similar to the previous proofs, it's easy to see that g (˜ z + i , W i , W i , U ′ i ) is a zero-mean random variable bounded in [ -∆ 2 (˜ z + i ) , ∆ 2 (˜ z + i )] . Thus,

To prove the first bound, we let t = √ I ( W i ,W i ; U i | ˜ Z + i =˜ z + i ) 2∆ 2 (˜ z + i ) 2 , then

Recall Eq. (14) in Lemma A.1, hence,

To prove the second bound, we take expectation over ˜ Z + i for Eq. (32),

This completes the proof.

E Omitted Proof in Section 6

E.1 Proof of Proposition 1

Proof. By Jensen's inequality and triangle inequality, for any i ∈ [ n ] , we have

where the last inequality is by the definition of the Bernstein condition. This completes the proof.

E.2 Proof of Theorem 6.1

Proof. Let g (∆ ℓ i , u i ) = ( -1) u i ∆ ℓ i . Notice that | g (∆ L i , U ′ i ) | ≤ β 2 and g (∆ L i , U ′ i ) is agian zero-mean, then

Thus,

Then, using the chain rule,

By the non-negativity of mutual information, we have

Furthermore, by the independence of each U i (i.e. I ( U i ; U [ n ] \ i | ˜ Z + [ n ] ) = 0 ), we have

Combining Eq. (35-37) will conclude the proof.

We now prove Theorem 6.2.

For a given ˜ Z [ n ] , the number of distinct values of their predictions, denoted by k , are upper bounded by the growth function of F evaluated at n ,

where the second inequality is by Sauer-Shelah lemma [53, 57] for n > d +1 . Thus,

To prove the disintegrated CMI bound, we let g be defined in the same way, and the remaining development is the same with the proof in Theorem 4.5.

For the second inequality, notice that I (∆ L i ; U i ) ≤ I (∆ L i ; U i | ˜ Z + i ) by using the chain rule of mutual information and the independence between ˜ Z + i and U i . In addition, moving the expectation over ˜ Z + i inside the square-root function by Jensen's inequality, we have E ˜ Z + i √ I ˜ Z + i (∆ L i ; U i ) ≤ √ I (∆ L i ; U i | ˜ Z + i ) .

E.3 Proof of Theorem 6.2

Proof. Before we prove Theorem 6.2, we first show the following lemma.

Proof of Lemma E.1. First, by I ( F i , ¯ F i ; U i | ˜ Z + i ) = H ( U i ) -H ( U i | F i , ¯ F i , ˜ Z + i ) and I ( F i , ¯ F i ; U i | ˜ Z + [ n ] ) = H ( U i ) -H ( U i | F i , ¯ F i , ˜ Z + [ n ] ) , and notice that H ( U i | F i , ¯ F i , ˜ Z + [ n ] ) ≤ H ( U i | F i , ¯ F i , ˜ Z + i ) , we have

By Jensen's inequality and Lemma E.1, we have

Plugging Eq. (38) into the inequality above,

which completes the proof.

F CLB Exampls

In Example 1, [21, Thm. 17] demonstrates the non-vanishing behavior of individual IOMI and e-CMI. This is primarily attributed to the dimension-dependent nature of IOMI and CMI. Specifically, there are certain dimensional settings where IOMI can grow faster than O ( n ) , as shown in [21, Thm.4], and CMI approaches a certain fraction of its upper bound, as illustrated in Example 1, resulting in non-vanishing behavior. Specifically, in Example 1, [21] employs the birthday paradox [37, Sec. 5.1] problem to demonstrate that for a large value of d , the probability that no pair of instances in ˜ Z sharing the same non-zero coordinate (referred to as event E 0 ) is smaller than a constant probability (that could be independent of n ). Particularly, it is shown that if d ≥ 2 n -1 1 -c 1 / (2 n -1) , then P ( E 0 ) ≥ c ≥ ( 1 -2 n -1 d ) 2 n -1 . As an example, [21] chooses d = 2 n 2 , resulting in c ≥ 0 . 1 .

Failure of I ( W ; Z i ) Consider the case where d = 2 n 2 . For the individual CMI [49, 70], I ( W ; U i | ˜ Z i ) , we have the following:

This inequality holds because when event E 0 does not occur, one can determine the value of U i completely, as the returned hypothesis is a weighted sum of the sample. In other words, examining the non-zero coordinates of W is sufficient to determine U i . For an in-depth derivation of this inequality, readers are referred to the updated version of [21], where their corrected proof involves Fano's inequality. Furthermore, using the relation I ( W ; Z i ) ≥ I ( W ; U i | ˜ Z i ) [70], we conclude that I ( W ; Z i ) ∈ Ω(1) .

Failure of I ( ̂ Z i ; U i | ˜ W i ) Notably, our hypotheses-conditioned CMI also does not vanish for the same reason. More precisely, when ˜ W i and ̂ Z i are given, there exists a constant probability (independent of n ) that allows us to fully determine the returned hypothesis based on ̂ Z i , thereby determining the value of U i .

Failure of I (∆ L i ; U i ) Furthermore, even the loss difference based CMI (e.g., as shown in Theorem 6.1), which provides the tightest CMI measure, still does not vanish. This is attributed to the fact that if the hypothesis W is independent of certain Z , there exists a constant probability where the loss becomes zero (recall that the loss is the negative inner product of W and Z ). Consequently, one can determine the value of U i by observing the sign of the random variable ∆ L i . This also indicates the limitations of e-CMI and f -CMI in capturing the generalization behavior for Example 1.

In Example 2, following the approach in [21], the training sample S = { Z i } n i =1 ∼ µ n can be represented as S = z 0 R 0 ( ε 1 , ε 2 , . . . , ε n ) , where { ε i } n i =1 is a sequence of independent Rademacher random variables, i.e., ε i ∼ Unif( {-1 , 1 } ) . The empirical risk is given by L S ( W ) = --L nR 0 ⟨ W, ∑ n i =1 ε i ⟩ . In this case, the ERM solution is W ERM = z 0 if sign( ∑ n i =1 ε i ) = 1 , and W ERM = -z 0 if sign( ∑ n i =1 ε i ) = -1 . It is clear that

Hence, we observe that β 2 is now a constant, whereas IOMI has an upper bound: ∑ n i =1 I ( W ; Z i ) ≤ I ( W ; S ) ≤ I ( W ; sign( ∑ n i =1 ε i )) ≤ H (sign( ∑ n i =1 ε i )) ≤ 1 , where the second inequality follows from the Markov chain S -sign( ∑ n i =1 ε i ) -W . This provides us with a generalization bound of 2 LR 0 √ n . Meanwhile, the actual generalization error satisfies E µ ( A ) ≥ LR 0 √ 2 n (see [21, AppendixB] for a derivation). Thus, the IOMI bound is tight up to a constant, and the stability bound β 2 itself is vacuous. It is worth noting that I ( ̂ Z i ; U i | ˜ W i ) ≤ I ( W ; Z i ) ≤ 1 by Theorem4.2, indicating that the CMI bound is also tight.

We would like to note that the failures of chained mutual information bounds [2] are not demonstrated in the counterexamples presented in [21]. Notably, when the hypothesis is quantized, it becomes more challenging to guess U i or Z i . Therefore, exploring the potential of chained information-theoretic bounds, which do not necessarily rely on stability notions, could be another avenue to explain the generalization behavior observed in these counterexamples.

G Additional Applications

G.1 Compression Schemes

We now consider the algorithm that has a compression scheme [30]. Formally, a sample compression scheme of size k ∈ N is a pair of maps ( A 1 , A 2 ) . Specifically, for all samples s with n > k , A 1 : Z n → Z k compresses the sample into a lengthk subsequence A 1 ( s ) ⊆ s . Then A 2 : Z k → W could be some arbitrary mapping. Hence, A ( · ) = A 2 ( A 1 ( · )) . Let K be the index set for S selected by A 1 , and let K be the selected index set for S i . In this case, our supersampleconditioned CMI has an upper bound: I ( W i , W i ; U i | ˜ Z + i ) ≤ I ( K,K ; U i | ˜ Z + i ) ≤ H ( K,K | ˜ Z + i ) ≤ 2 log ( n k ) ≤ 2 k log n . Then, if A is further β 2 -uniform stable, then we have the generalization bound E µ ( A ) ≤ O ( β 2 √ k log n ) . If β 2 < O (1 / √ n ) , this bound improves the bound in [58]. It is unclear if we can obtain any improved bound for stable compression schemes [7], in which case [19] provides an optimal bound that removing the log n factor for the realizable setting. A main difficulty is that an interpolating algorithm is usually unstable due to the fitting-stability tradeoff [54, Sec. 13.4].

G.2 Distillation Algorithm

The high-probability generalization property of distillation algorithm is studied in [16]. In the first training stage of distillation, we obtain a w ∗ s from a highly complex hypothesis space W 1 based on a training sample s . Same to [16], we assume that the first learning stage is α -sensitive, namely || w ∗ s -w ∗ s i || ≤ α = O (1 /n ) . In the second stage, the algorithm A will select a hypothesis that is λ -close to w ∗ s from a less complex hypothesis space W 2 = { w ∈ W : || w -w ∗ s || ∞ ≤ λ } . Let the loss function ℓ be L -Lipschitz with respect to the first argument. Consequently, γ 3 ≤ L || w ∗ s -w ∗ s i || ≤ Lα . Then, by Theorem 4.1, we have E µ ( A ) ≤ Lα 1 n ∑ n i =1 √ 2 H ( U i ) = √ 2 log 2 Lα. Notice that the loss here may not necessarily be bounded or sub-Gaussian, rendering previous bounds inapplicable.

G.3 Regularized Empirical Risk Minimization

Regularized Empirical Risk Minimization (ERM) learning rules involve minimizing the empirical risk and a regularization function jointly: arg min w ∈W L S ( w ) + f reg ( w ) , where f reg : W → R . Here we specifically consider Tikhonov regularization [54], namely f reg ( w ) = λ || w || 2 , where λ > 0 is a tradeoff coefficient. The regularized ERM algorithm A aims to find

This regularization term ensures strong convexity of the training objective. Based on Theorem 4.3, we can derive the following results.

Corollary G.1. Assume that the loss function ℓ is convex and L -Lipschitz. Then, for the regularized ERM algorithm with Tikhonov regularization, we have

Proof of Corollary G.1. By invoking [54, Corollary 13.6], we know that γ 4 ≤ β 2 = 2 L 2 λn . Plugging the value of β 2 will give us the desired result.

Corollary G.2. Assume that the loss function is ρ -smooth and nonnegative. Let λ ≥ 2 ρ n . Then, for the regularized ERM algorithm with Tikhonov regularization, we have

Proof of Corollary G.2. By invoking [54, Corollary 13.7], we know that γ 4 ≤ β 2 = 48 ρ ˆ L n λn . Plugging the value of β 2 will give us the desired result.

Although these bounds do not enhance the convergence rate of O (1 /n ) in these settings, they consistently offer tighter results compared to uniform stability-based bounds if 1 n ∑ n i =1 I ( ̂ Z ; U i | ˜ W i ) ≤ 0 . 28 . In addition, the expected empirical risk ˆ L n appears in the bound of Corollary G.2. While λ has a lower bound, ˆ L n could not be arbitrarily small for the regularized ERM.

Notice that previous information-theoretic bounds could not obtain the convergence rate of O (1 /n ) as in our results unless ICMI or CMI itself decays with O (1 /n ) .

H Additional Discussions and Open Problems

Stochastic Gradient Descent (SGD) Since the influential work of [22], stability approaches have been widely employed to provide generalization guarantees for (sub)gradient-based optimization algorithms, such as SGD, under certain conditions like the convex-smooth-Lipschitz loss. More recently, [5] extended the results of [22] to the non-smooth loss function in the SCO setting.

In contrast, information-theoretic (weight/hypothesis-based) bounds are typically used to analyze the noisy version of SGD, known as SGLD [43, 40, 18, 49, 60]. Directly analyzing SGD poses challenges because the returned hypothesis W contains a significant amount of information about S or Z i , resulting in potentially large (even infinite) mutual information. The prevalent approach to applying information-theoretic bounds to SGD is by introducing noise [41, 61], but this has been shown to yield non-vanishing bounds in [21, Thm. 4].

The combination of information-theoretic bounds with stability for analyzing the generalization of SGD presents a promising future direction. However, some potential difficulties may arise. For instance, if we continue to use the Gaussian noise perturbation method for the weight-based information-theoretic bounds, we would need to characterize the stability property for the perturbed SGD, which might require techniques employed in [38]. Additionally, when combining stability notions with loss difference based CMI (or e-CMI/ f -CMI) bounds, as they cannot be unrolled using the chain rule and data processing inequality as in the case of weight-based IOMI/CMI bounds, it may not be possible to bound such CMI terms using trajectory-based quantities. This raises doubts about the potential for obtaining more informative generalization bounds compared to the stability-based bounds themselves.

Generalization Bounds beyond Mutual Information In the information-theoretic literature, it is common to replace mutual information with alternative distributional measures, such as Wasserstein distance based bounds and total variation based bounds [50]. A promising future direction is to incorporate the stability property of algorithms into these bounds, as demonstrated in this work. It is worth noting that obtaining KL divergence-based bounds should be straightforward since they rely on the same foundational Lemma A.2 as discussed in this paper.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInformation-theoretic generalization bounds offer a new approach in generalization theory by providing complexity measures that depend on the data distribution and learning algorithm itself. In the past years, it has been observed that information-theoretic generalization bounds are not compatible with the classical approaches to prove generalization such as vc theory or uniform stability. \n\nIn this paper the authors propose several new information theoretic quantities to study generalization. The proposed bounds are interesting as they are compatible with several notions of uniform stability.\n\nSTRENGTHS:\nI think the paper addresses an important limitation of information-theoretic generalization that is incompatibility with the uniform stability framework. They also provide various examples to show the expressiveness of their bounds.\n\nWEAKNESSES:\n1– The presentation of the paper needs significant improvement. Specifically the part on defining the additional structure is extremely vague. For instance, there are many variables that are not clearly defined. For instance, what is the tilde{W}_{i,0} and tilde{W}_{i,1}.\n\n2– Compared to CMI bound, the intuition behind the results is not clear. For instance CMI is closely related to membership inference. \n\n3– Some of the results in the paper seem redundant. For instance, theorem 3.1 has been proved by work by Bu et al under the name of individual sample bounds.\n\n\n4– The major drawback of the bounds in the paper is the following: Uniform stability bounds for randomized algorithms are very important. For instance, in the seminal work by Hardt et al, the fact that the algorithm is randomized is important. The other example is the algorithm by Feldman and Dagan for leaning VC classes with stable algorithms. However, to me all the bounds in the paper rely on a very strong notion of stability which only holds for “deterministic algorithms. It is not clear what is the roadblock for extending the results for randomized algorithms.\n\n\nFeldman, Dagan. PAC learning with stable and private predictions\n\n\n5– The main limitation of uniform stability bounds is that it considers the “worst-case” data distribution. However, a learning algorithm may be more stable for “easier” distributions. However, all the bounds in the paper already depend on the worst-case stability parameter. I think this leads to suboptimal bounds.\n\nQUESTIONS:\n1- weakly uniform stability versus strong uniform stability: Consider the seminal work of Hardt et al paper in which they derive stability bound for SGD under the notion of weakly uniform stability. Why in the paper do authors only consider the strong notion of stability?\n\n2- text after corollary 3.1 seems unclear to me. If the algorithm is deterministic then the mutual information term will blow up. It is not clear to me how this may result in improvement.\n\n3- Theorem 3.3: Let's assume that the algorithm is deterministic and also permutation invariant in the sense that the order of the training points does not affect the output. Then, this results states that  gen.gap <= gamma_1  (  1  + mutual information term). It shows that even if the mutual information term goes to zero, the convergence rate is determined by gamma_1.\n\n\n4- For Example 1, what is the exact generalization error? It should be of order O(1/n), however, the obtained bound using the proposed approach has an order of 1/sqrt(n).\n\nTypos:\n\nLemma A.5. Bernstein.\n\nLIMITATIONS:\nThe main limitations are explained in Items 4 and 5 of weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes several new stability assumptions named the sample-conditioned hypothesis (SCH) stability. Based on these notions, the authors present new IOMI and CMI bounds to address the limitations of existing information-theoretic bounds in the context of stochastic convex optimization (SCO) problems.\n\nSTRENGTHS:\n- The paper adopts different assumptions for bounding the CGF, which can improve existing information-theoretic bounds, especially in the context of SCO.\n- The connection between Bernstein and the proposed SCH-B stability is interesting.\n- The paper is generally well-written.\n\nWEAKNESSES:\n- It seems to me the most significant contribution is to introduce uniform stability for bounding the CGF, which can solve the counter-example proposed in [1]. However, it's hard to estimate the order of the stability parameter without any knowledge of the distribution. Especially in practice, such parameters do not affect the algorithm design. \n\n- The effectiveness of the proposed SCH stabilities is unclear. It's mentioned that all the stability parameters $\\gamma_{1:4}$ are smaller than $\\beta_2$. From Theorem 4.3, we have $\\|\\mathcal{E}\\_{\\mu}(A)\\| \\leq \\frac{\\beta_2}{n} \\sum_{i=1}^n I(\\hat{Z_i};U_i|\\tilde{W_i}) + 0.72 \\beta_2$. Similar to the discussion in Example 2, the first term can vanish. However, the second term still exists since $\\beta_2$ is a constant. To prove the effectiveness of the SCH stability bounds, it needs to identify the order of $\\gamma_4^2$.  \n\n- Example 1 requires the data and parameter dimension $d=2n^2$ to ***grow with $n$*** for proving the $\\Omega(1)$ lower bound of the information quantities. Does this really make sense?  Even though the uniform stability is much tighter in this case, I do not think this counter-example is representative enough. Could you conceive another counter-example that is more convincing?\n\n- The SCH-B and SCH-D stability are distribution dependent, the corresponding stability parameter cannot be estimated, and these notions can be confused with other distribution assumptions.\n- The SCH-A and SCH-C stability are distribution-free but related to the distribution (or set) of the output model parameters. It’s unclear to me how to identify the stability parameter effectively for different algorithms.\n\n>[1] Mahdi Haghifam, Borja Rodríguez-Gálvez, Ragnar Thobaben, Mikael Skoglund, Daniel M Roy, and Gintare Karolina Dziugaite. Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization. In International Conference on Algorithmic Learning Theory, pages 663–706. PMLR, 2023.\n\nQUESTIONS:\nPlease see the above section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to develop information-theoretic generalization error bounds under the assumption that the algorithm is uniformly stable under a certain loss. Typically, these kinds of bounds are based on properties of the loss with respect to the data (e.g. subgaussian or bounded), while in this case they look at a property of both the loss and the algorithm, namely that it is uniformly stable. Considering these two together allows the bounds to decrease at faster rates as the stability parameter can decrease with the number of samples while the boundedness or subgaussianty parameter does not.\n\nIn particular, they extend the individual mutual information bounds from [9] to this assumption. They also consider a matrix of $n \\times 2$ hypotheses, where the first column is the output hypotheses, and each row in the second column is the hypothesis outputted by the algorithm when the $i$-th sample is swapped by another sample that is i.i.d. In a similar fashion to [58], they manage to develop conditional mutual information bounds, where they condition on this set of hypotheses. Intuitively, these bounds state the ability to determine if a sample was used to generate the real hypothesis or the hypothesis where the $i$-th sample was swapped. Then, they develop individual conditional mutual information bounds similar to [49] under this setup.\n\nAs an application, they showcase that while most of the current information-theoretic bounds fail in the example from [21], the presented bounds can succeed as they are stable with a stability parameter that decreases with the number of samples. Also, they make the case that for very stable algorithms (with constant in $\\Omega(1/\\sqrt{n})$) their bound improves upon [58]'s. Finally, they connect their results with the Berstein condition and further adapt the bounds from [63] to their setup.\n\nSTRENGTHS:\nThe main strength of the paper is the link between stability and the current individual mutual information bounds, which in turn develop into mutual information bounds for stable algorithms with an explicit appearance of the stability parameter. This is good because we know the stability parameter in certain algorithms and this mutual information can be bounded in a good amount of settings.\n\nSimilarly, the connection with the Berstein condition is also important as we also know that a good amount of algorithms under certain losses respect this condition, and therefore these bounds can be applied.\n\nFinally, as in [63], the paper showcases how a simple trick like the Donsker-Varadhan together with a careful choice of the function in the supremum can lead to good bounds.\n\nWEAKNESSES:\nWhile the results are interesting, I fail to understand exactly how results other than those in Section 3 are useful. Yes, they can achieve the rates desired in Example 1, but this comes basically because the individual CMI is <= 1 and one can directly have the bound expected generalization error <= $\\beta$, which is already known from the uniform stability literature. \n\n* A suggestion for improvement, is to state that directly as a stronger result. Since the individual CMI <= 1, then one may recover the bounds from the uniform stability literature. Then, every result that we know from there is also subsumed into this setup. This way, one can see more clearly that Example 1 works immediately from uniform stability and many other results as well. I believe that this way you may be able to comment on larger sets of problems.\n\nI think that they definitely have potential, but this potential is not explained in the paper. The main reason I mention this is that the previous bounds were dealing with a KL divergence between terms that were \"easily tractable\" like the posterior $P\\_{W|S}$ (or $P\\_{W|Z\\_i}$) and a prior $Q\\_{W}$. However, the terms appearing in these bounds seem very difficult to treat and characterize. \n\n* Another suggestion would be to find an example where this mutual information can be calculated so the reader can have an idea of why this is interesting other than the fact that it can be reduced to the known bound from uniform stability as I mentioned above. An interesting example would include a setup where neither uniform stability nor \"classical\" information-theoretic bounds can achieve the desired expected generalization error rate, but combined with the presented bounds can.\n\nSome results and statements are included but are loosely explained.\n\n* Theorems 3.3 and 4.3 are examples of this. When can this be useful? Could you provide us with a particular example where we can see how are we benefitting from including $\\gamma\\_2^2 / \\gamma_1$ or $\\Lambda(\\Tilde{W}\\_i)$?\n* In Remark 2.2, what do you mean with \"it is expected that $\\gamma\\_4$ is larger than $\\gamma\\_2$ due to the independence of $Z'$ in Eq. (7)\"? A similar claim is done in lines 217-218. When doing this kind of remarks and statements, it would be important to justify more the reasoning behind them.\n* The whole set of definitions in Definiton 2.1 is not very clear to me. It would be nice to have a larger, more comprehensive commentary on what are these definitions and why are they included / why are they useful.\n\nSmall corrections in the literature, not very important:\n\n* In line 31, it should be \"decay at a faster rate, e.g. $\\mathcal{O}(1/n)$\".\n* In line 33, \"demonstrating tightness in non-convex learning cases such as deep learning\" may be misleading. Some people may interpret that this means that they achieve the correct rate and therefore are tight. An alternative writing could be \"demonstrating a good characterization in some instances of non-convex settings (e.g. deep learning)\".\n* In line 198, $R$-conditioned information-theoretic bounds also appeared in [49].\n\nQUESTIONS:\n* Could you clarify further the definitions in Definition 2.1? Also, could you expand in the reasoning of why some parameters are larger/smaller than the others?\n\n* Could you give some examples where Theorems 3.3 and 4.3 improve upon the preceding theorems?\n\n* Could you try to find examples where neither uniform stability nor \"classical\" information-theoretic bounds can achieve the desired expected generalization error rate, but combined with the presented bounds can?\n\n* Could you find examples where the bounds with the conditioning can be employed to gain an understanding of some problem beyond the fact that it is stable?\n\nI am happy to increase my score if these questions are satisfactorily answered. I like the paper, but it seems a little immature at the moment. Once these things are addressed, it could become a better paper that ticks many boxes and deals with important questions.\n\nLIMITATIONS:\nSome limitations are included throughout the text and in the Appendices. Some others, like the ones included in the weaknesses, are not. Regarding potential negative societal impact, as the work is of a theoretical, fundamental nature, I do not forsee any issue with not addressing them.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work improves information-theoretic generalization gap bounds by doing more careful derivations. As a result, the derived bounds get a multiplicative factors that capture some notions of hypothesis stability, while the existing bounds usually have a multiplicative constant factors that depend on the loss function and the data distribution. For example, the bound of Bu et al. [9] that states that $\\text{expected gen. gap} \\le \\frac{1}{n}\\sum_i \\sqrt{2 I(W;Z_i)}$, with $W$ being the output of the learning algorithm on the dataset $(Z_1,\\ldots,Z_n)$, gets improved by a multiplicative stability term $\\gamma_1 \\le 1$ that captures how much (in the worst-case) the loss on an example $z'$ can change when one replaces a single training example. Similar improvements are derived for conditional mutual information (CMI) bounds, both with standard CMI terms and a novel CMI term. In the random-subsample setting of Steinke and Zakynthinou [58], this novel CMI term measures the conditional mutual information between the $i$-th example that contributed to the training and the selection variable of the $i$-th pair, given two hypotheses: one corresponding to training with the first example of the $i$-th pair, the other corresponding to training with the second example, keeping everything else the same.\n\nThe authors show that in some known cases when existing information-theoretic bounds fail to vanish with $n$, adding the stability factor result in a vanishing bound. They also show that there are cases when the stability term alone does not vanish, but when the mutual information part is also considered, the resulting bound vanishes.\n\nSTRENGTHS:\n**Strength #1: Significance & Originality.** Information-theoretic generalization bounds have gained a lot of attention recently, partly because they are algorithm and distribution-dependent and some variants of them are nonvacuous in practical setting for deep learning. Recently, Haghifam et al [21] uncovered some limitations of information-theoretic bounds, constructing cases when the existing bounds do not even vanish with $n$. This submission addresses these limitations, and is therefore a significant contribution. While the derivations mostly follow common techniques, the resulting unification of stability-based and information-theoretic bounds is a novel (to my best knowledge) and significant contribution.\n\n**Strength #2: Quality.**  The submission is technically sound.  I have checked the proofs. The related work is cited adequately.\n\n**Strength #3:  Clarity & Presentation.** This work is well-written and presented.\n\nWEAKNESSES:\n**Weakness #1: Limitations [minor].** As the authors acknowledge, the proposed stability terms are hard to evaluate or estimate in practical deep learning settings. Therefore, the extent of improvements outside of the considered synthetic settings is unclear.\n\nQUESTIONS:\n- Line 153: \"are identically but not independent distributed\" --> \"are identically distributed but not independent\".\n- Line 169: It should be $\\gamma_2 \\le \\gamma_1$ and $\\gamma_4 \\le \\gamma_3$.\n- Line 219: there might be no uniform convergence if the conditional mutual information terms vanish with different rates for different $\\tilde{w}_i$ such that their supremum does not vanish.\n- Equation (14): $W_i$ and $\\bar{W}_i$ need to be defined in the Lemma statement.\n- Theorem 4.4: Is the assumption of $\\mathcal{A}$ being symmetric with respect to $S$ necessary?\n- Line 280: Are $\\hat{\\mu}$ and $\\hat{\\mu}^i$ treated like constants here?\n- Lines 713-720: If $E_0$ denotes the event of two elements of $\\tilde{Z}$ sharing the same non-zero coordinate, then we are interested in upper bounding $p(E_0)$ (in contrast to what is written in line 717). On line 720, \"the first inequality holds because when event $E_0$ occurs, one can determine the value of $U_i$ completely\", it should be when $E_0$ *does not* occur.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}