Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Bounds
Authors: Ziqiao Wang
Yongyi Mao
Venue: NeurIPS 2023 poster (2023)
Keywords: generalization
information-theoretic bounds
stability
TL;DR: We derive some sharper information-theoretic bounds by combing with the stability-based framework.

Abstract: We present new information-theoretic generalization guarantees through the a novel construction of the "neighboring-hypothesis" matrix and a new family of stability notions termed sample-conditioned hypothesis (SCH) stability.  Our approach yields sharper bounds that improve upon previous information-theoretic bounds in various learning scenarios. Notably, these bounds address the limitations of existing information-theoretic bounds in the context of stochastic convex optimization (SCO) problems, as explored in the recent work by Haghifam et al. (2023).

Sample-Conditioned Hypothesis Stability Sharpens
Information-Theomee Generalization Bounds
Ziqiao Wang University of Ottawa zwang266buottwwa.co
Yongyi Mao University of Ottawa ymao@uottawa. ca
Abstract
We present new cnoommttiontthoorete generalization guarantees through the novel construction of the hicighboringgyypoteesiss matrix and new family of stability notions termed sample-condition hypothesis" hypothesis (SCH) stability. Our approach yields sharper bounds that improve upon previous information-theoredr various learning scenarios. Notably, these bounds address the limitations of existing information-theorett bounds in the context of stochastic convex optimization (SCO) problems, as explored the recent work by Haghifam et al. (2023).
bounds in
Introduction
information-theoretii upper bounds for generalization error have recently been developed since the seminal works of [52, 67]. On the one hand, these bounds have attracted increasing interest due to their distribution dependence and algorithm dependence, making them highly suitable for reasoning the generalization behavior of modern deep learning. In fact, subsequent studies have demonstrated that anformation-theorette bounds can effectively track the dynamics of generalization error in deep neural networks [40, 60, 23, 61, 62, 25, 64]. On the other hand, recent studies [58, 19, 23, 20, 25] have revealed the expressive nature of information -hhooretic bounds in the distribution-fre free setting. Notably, the conditional mutual information (CMI) framework proposed by [58] has shown great promise by establishing connections to VC theory [59] and matching minimax rates for the binary classification [19, 20). Additionally, in the case of the 0 - 1 loss and the realizable setting, where an interpolating algorithm attains zero empirical risk, Eeformation-thooree
bounds give an exact characterization of the generalization error [20, 64], thereby providing the tightest possible generalization bound.
bounds have been extensively discussed due to their main deficiencies. The first deficiency concerns the unbounded nature of the original input-output mutual information (IOMI) bounds [67, 4, 32, 31]. To address this issue, several techniques have been developed, including the chaining method [2]. the individual technique [9] or random subset technique [40], and the Gaussian noise perturbation technique [41]. Notably, the CMI bound [58] stands out as it has finite upper bound for any learning scenario due to its supersample construction. These techniques are now often applied jointly for analyzing generalization [18, 49, 70, 63, 23, 25, 64]. The second deficiency concerns the sub-optimal convergence rate of information-theoretic bounds. Specifically, when the information theoretii quantities, whether IOMI or CMI, are bounded by constants, the bounds exhibit a decaying rate in the order of ((1/VR), where is the sample size. In contrast, generalization errors in practice may decay at faster rate, c.g., (LL/n). To address this limitation, several works, inspired by some PAC-Bayesian literature, have proposed fast-rate information-theoredt bounds [24, 25, 64], demonstrating good characterization in some instances of non-conves settings such as deep learning. More recently, [65, 66, 71] present IOMI bounds for the Gaussian mean estimation problem, that achieve optimal convergence rates, contrasting previous bounds [9, 70].
Nonetheless, information-theonee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The issue of slow convergence in information theoretic bounds has recently been amplified by the observation that these bounds may not even vanish. [21] highlight this limitation in the context of stochastic convex optimization (SCO) problems [55]. Specifically, [21] shows that all existing information-thenn -heeorrettt bounds, including the CMI bound [58], the Gaussian noise perturbed IOMI bound [41], the individual IOMI bound [9], the evaluated CMI (e-CMI) bound [58] and the functional CMI (f-CMD bound [23]. fail to vanish in at least one of the counterexamples they constructed. These failures stem from the dimension-dependen nature of current information-theoretii quantities [31], appearing an intrinsic barrier for overcoming these limitations. However, we dissect the process by which these bounds are derived, opportunities do exist. Specifically, recall that all these informationntheoreete bounds are built upon the Donsker- Varadhan (DV) variational representation of the KL divergence [44, Theorem 3.5] (see Lemma A.2 in the Appendix). Using this representation, a generalization upper bound is derived in terms of information-theerrtie quantity and a cumulant- generating function (CGF), as illustrated below.
IOMI or CMI + CGF
Generalization Error S inf
Para.
Para.
Particularly note that the CGF depends on certain choice of the auxiliary function in the DV formula. Then, except for the IOMI or CMI itself, the tightness of the generalization bound hinges on two key factors: the selection of the DV auxiliary function and the approach used to bound the CGF, the latter of which often requires additional assumptions specific the chosen DV auxiliary function. The most common choices for the DV auxiliary function involve making assumptions such as sub- Gaussian loss or bounded loss. For instance, in the case of IOMI bounds, the DV auxiliary function is typically chosen as the single loss or average loss, and the sub-Gaussian assumption (or boundedness assumption) utilized. In CMI bounds, the DV auxiliary function defined as the difference in loss between a training sample and a "ghost" sample, and the boundedness property is employed. We now note that exploiting bounded loss is the fundamental reason behind the failures in SCO problems. Although [21] does not explicitly rely on boundedness, they do make use of the product of the Lipschitz constant and the diameter of the hypothesis domain, which essentially serves as an upper bound for the loss. The product does not vanish with n, thereby neutralizing the potential to include another decaying factor the bound. Arguably, at least for these SCO problems, the selection of the DV auxiliary function may not be optimal, for example, the resulting CGF= (1/nn, giving vacuous generalization guarantees. This analysis inspires us to explore alternative DV auxiliary function and to adopt different assumptions for bounding the CGF. For instance, we may devise appropriate stability assumptions and create opportunities to upper bound the CGF using a term in O(8² /r)), where 3 is the stability parameter that decays as n. This is promising in light of the capability of the stability-baan -ased framework in explaining the generalization of SCO problems [6. 56, 22, 5].
In this paper, we combine information -eermation-thhortiic analysis with stability notions to develop IOMI bounds and CMI bounds that improve upon previous information -teeorettonntheoreetie bounds for stable learning algorithms, achieving faster convergence rates. The main contributions of our paper are summarized below.
We introduce our new notions of algorithmic stability, referred to as sample-conditionel hypoth- esis (SCH) stability. We also present a novel construction of a sample-dependent hypothesis matrix, where each column is neighborhood pair of hypotheses obtained from two training samples that differ in only one element, inspired by the supersample setting of CMI [58].
We present new IOMI bounds, which explicitly include the SCH stability parameters and are shown superior to previous bounds for stable learning algorithms.
We show that the sample-dependent hypothesis matrix, similar to the supersample matrix, enjoys a symmetry property. Exploiting this symmetry, we establish novel CMI bounds. Specifically, we present hypotheses-conditione conditioned CMI bounds. Additionally, we derive cample-conditioned CMI bounds exploiting other assumptions. Notably, these bounds introduce novel CMI quantities and include SCH stability parameters. In particular, the new CMI quantities remain boundedness the original CMI. Consequently, these CMI bounds vanish no slower than their stability parameters. In addition, we also obtain second-moment generalization bound that matches the tightest known bound in the literature [14] under the same condition.
CMI bounds that are analogous to the previous supersample-
We apply our new bounds to a convex-Lipschitz-oonee bounded (CI.B) example in which previous information thhooretic bounds fail to explain generalization [21], and show that the new IOMI

---

and CMI bounds vanish, benefiting from SCH stability. Additionally, we discuss another CLB example where the uniform stability parameter non-vanishing or has slow convergence rate, but our information-theoree
bounds remain tight up to constant.
We extend our analysis to derive information -heeorti generalization bounds based on Bernstein condition, which leverages connection between stability and Bernstein condition. We also show that stability can be incorporated into generalization bounds to obtain stronger results using alternative information-theorett quantities such as the loss difference based CMI, e-CMI and f-CMI. Furthermore, we illustrate the expressiveness of our new CMI notions under the distribution-fiee setting.
2 Preliminaries
Probability and Information Theory Notation Unless otherwise noted, a random variable will be denoted by a capitalized letter, and its realization by the corresponding lower-case letter. The distribution of a random variable X is denoted by Px, and the conditional distribution of X given Y denoted by Px|y. When conditioning on a specific realization y. we use the shorthand Yxyy=y or simply px Denote by Ex expectation over X ~ Px. and by Exjy-y (or Exly) expectation over X ~ =x The entropy of a random variable X is denoted by ((X). ), and the KI. divergence of probability distribution P with respect to Q is denoted by DKL(PI
The mutual information (MI) between random variables X and Y is denoted by YYX;Y), and the conditional mutual information between X and Y given Z is denoted by (|). We also define the disintegrated mutual information as (X) A that (I) = Ez
(Px.((Px11 Pxiz=s Py) following the notation in [40]. Note
Generalization Error and Uniform Stability We consider the supervised learning setting. where we have domain of instances Z = X' x y. with input and label spaces denoted by X' and Y respectively. The distribution of an instance is given by μl, and we have a training sample S = (Z) ~ n². Let R E R be a source of randomness (a random variable independent of S over an appropriate space R), from which a learning algorithm A
2n R - W takes the training sample S and R as input, and outputs a hypothesis W = A(S, R) E W. To evaluate the quality of the output hypothesis W. we use a loss function f W x Z + R Given a fixed w, we define the population risk (w) _ Ez [e(w, Z')], where 7' ~ fr is testing instance. The quantity L = IEw (W)]l is then the expected population risk. For a fixed w E W. the empirical risk on S is defined as (s(w)= A Σ f(w, 2.). Similarly, we define the expected empirical risk as i.,. Ew.s [W)]. Thus, the expected generalization error is given by & & L L.
We now give two notions of uniform stability [6. 13], where s ~ sf denotes two training sets s and s that differ only at the ith element. We say learning algorithm AA is f1-weakly uniformly stable if sup ER ((A(s, R), =) - E(A(s', , R). 2) s $1.
$2s',
and 32-strongly uniformly stable if
sup sup suple((As, E(A(s .),, B2. smaat
Remark 2.1. Notably the weak uniform stability above is the standard in the literature. It is evident that if A is 3-strongly uniformly stable, it must also be B-weakly uniformly stable. It is also worth noting that for deterministic algorithms (e.g., GD or fixed permutation SGD), the two notions are identical. We note that in this paper, when speaking of uniform stability, we refer to the strong notion.
Let S'
(4D1 ~ 1²" be an independent copy of S, and let S S (2,) so S = S U (22).
The following well-known result (e.g., [6. Lemma 7], [54, Thm. 13.2l) is frequently used in this paper.
Lemma 2.1. For any algorithm A, we have Ly = Es,s R 1 En ((A(S", ZJ)) and
EA (A(SS R), 2.) EAss,n)s((A(SS, M), 2.) (1)
(A) = Es,s'
3

---

Supersample and Sample-Conditioned Hypothesis Stability Following [58], let Ž E gnx2 be a supersample matrix with n rows and 2 columns, each entry drawn independently from μ. We index the columns of Z by 0,1 I and denote the ith row of 2 by Zi with entries (Z.o; Z,,1). We often use the superscripts + and - to respectively replace the subscripts 0 and 1, L.e, writing Z for Z,, and Zí for Z,,1 Correspondingly, Zhi and Z denote the first and second columns, respectively. Additionally, which the ith element 2 is replaced with the corresponding element Z in the second column. Let W+ = A(Z R), and W, = [[ R) for each i [n]. That is, W+ and each W, are obtained by two "neighboring" training samples, namely those differ only on
will use 7 |n|- denote Tni in
Zi
Z
W+
Wi I
Z
W+
W
one instance (the ith instance). We then construct matrix W E Wnx2, where the ith row W, = (W,+, W = (W.,.. W.,)
W+
and W W for all i [ni]. as shown in Table 1. Unlike the supersample matrix, elements in W are identically distributed but not independent. In this case, Eq. (1) can be rewritten as
Table Supersample (Left) and the induced hypotheses (Right).
(A)
Σ Z 1=1
(WW 7,) - eWW, .2,)
The summand in Eq (2) exhibits an interesting "symmetry": for any
-, W. E Ezz w. <W; ž,) e -
[(W, - e(W. 25)
That is, the +/ superscripts in the summand of Eqq (2) can be flipped for any i. We may then n binary (10, 1) )-valeed) variables (U1, U,...., Un, == U to govern whether we choose to flip the
signs for each of the n terms, where U, = 0 indicates "no flipping". Let U, 2 1 - U,, we have
n
(A)
E W.W.
(WW., Z(u,) -e(We- Z.visl
(-1) (nWW, Ze...) ( -e(Wit 3,.ui))]
E Z, W.
(4)
Σ (-100) (RWV, (
n
-((wt 2.)]
EE E.U
(5)
where in Eq. (4) we have chosen to be an i.i.d. Bernoulli-(t) sequence, and in Eq. (5) we have renamed Z,,, as 2, and denoted E' 2 (2,, 2z. ... 2n.). Note that E, induced by U from Z, contains n instances, each serving to evaluate the loss difference between pair of hypotheses (W,, W+). The polarities of these evaluations are governed by U.
We now define some new notions of stability.
Definition 2.1 Cample-Conditioons
Hypothesis (SCH) Stability). Let p.,,, W and W generated via W = A(S. R) and W = A(S*, R). Let Zw we denote the support of the conditional distribution Pz ww and W. denote the support of the conditional distribution Pw]s- We introduce four types of SCH stability, referred to types A, B, C, D. Specifically, a learning algorithm A is
a) SCHCH- stable if E [m],
sup
sup [((w,)) -Ew+|uc [((W', [x]] S m,
(6)
WEU,EZn W, 2EZ
b) SCHHSC stable if [m].
EsSR2
((WW, 7') Wwijw [[(W*, ;Z)])²
S 7
(7)
c) SC-SCC-C stable Vi [n].
Ew
sup 2.EZ
|e(W, e(Wi, z:)
S 73:
(8)
d) 77--SCH 7CH-DD stable ii E [n].
Es.z.
((W",2))) (e(W, E(Wi, =z))² 5
(9)

---

where Z' in Eq. (7) is an independent instance drawn from μ., Remark 2.2. By definition,
see " ",, and it is expected that 74 S. 73. Note that all of them are smaller than 32. In addition, is expected that "2- Y and Y are smaller than 31, although the relationship between "/1 and 81 is uncertain. Moreover, it is also expected that a is larger than 222 due to the independence of Z' in Eq. (7). We emphasize that supreme sup,, n: in Eq. (6) is taken over hypothesis space Uaez" Ws. not the whole hypothesis space W. Notably, the notion of "hypothesis set stability" in [16] is closely related to our SCH stability. In fact, 71-CCH-A stable implies "M -hypothesis stable. Due to space constraints, we provide further elaboration on the definition of SCH stability in Appendix B.
the
IOMI Bounds for Stable Algorithms
We are now in a position to give the IOMI bounds for stable learning algorithms. Theorem 3.1. If the learning algorithu A is -S-CC-A stable, then
V
22 Σ 1=1
(p(AA)
z
T(W+ T W;;
We note that 1 1(W ( I(W Z.), the only difference between the first bound in Theorem 3.1 and the previous individual IOMI bound [9] is that the sub-Gaussian variance proxy is replaced by the stability parameter "1 in our bound. In fact, while we use W to better understand the appearance of 71 in the IOMI bounds, the first bound in Theorem 3.1 itself does not necessarily rely on the supersample and the construction of W.
In addition, if A is a deterministic algorithm, the term LWW ; (W;) in the second bound is tighter than the mutual information stability erasure information studied in [46, 23], that is, I(W+ 2,+WW S 1 W 2 (M
(see Remark C.1 in the Appendix for an explanation).
Proof Sketch of Theorem 3.1. Motivated by Lemma 2.1 and Eq. (2), the main innovation in this proof is to let the auxiliary function in DV be the "relative loss" instead of the single loss, namely we let
giū 2,") Bw p ((W, .,,) f(w+,2 ,,,) and let the auxiliary function f = t for t > in Lemma A.2. This enables utilize Eq. (6) bound the CGE The remaining steps routine. The complete proof can be found in Appendix C.I.
The following corollary, immediately following from 71 S B2. suggests a clear improvement over the previous 1OMI bound for the uniformly stable deterministic algorithm with vanishing 32.
Corollary 3.1. If A is 32-uniform stable, then (Aa(( < 22 E
L( +; 2,
A key message conveyed in this paper, shown in the proof of Theorem 3.1, is the importance of carefully selecting the DV auxiliary function and appropriate assumptions for various learning scenarios. Corollary 3.1 also suggests potential enhancement for uniform stability in the certain deterministic setting, provided that I(W; Zz) also vanishes appropriately with n.
Similar to [40, 49, 23, 25], we also give an R-conditioned IOMI bound below.
Theorem 3.2. If A is 82-uniform stable, we have E (A)) < V2d2 Σ BRY (R( 7t).
This disintegrated IOMI bound is not directly comparable to the bounds in Theorem 3.1, but is equivalent to Corollary 3.1 for deterministic algorithms. With additional assumptions, we may remove the square-root in Theorem 3.1, accelerating the decay of IOMI as shown in Theorem 3.3. Theorem 3.3. Under the same conditions in Theorem 3.1, if A is further SCHCCBB stable, then
T(+ +. 7 + 0.72 7
(A))
&
1=1
Notice that 7/2/1 S 17/m 71 If 7// decays faster than 1 E V I(W+; 2,,,, then Theo- rem 3.3 is qualitatively strictly stronger than Theorem 3.1.
5

---

4 CMI Bounds for Stable Algorithms
CMI Bounds In this section, give handful of CMI bounds based on a new information-theoretie quantity.
Hypotheses-Conde
Theorem 4.1. Suppose that there exists A1 W + R such that for every i; (w, ui
[(w0 24) f(w; 24) = A (w.). Then,
suP,,zz at
.t.
v2 (x)
min
E Wr
(W I (Z. V V.w. ((W.)))/1(2
Furthermore, if A is 73-SCH-C stable, then such a An exists and
sup I UZ) V213 [x,(A))
Compared to the standard supersamplee-condtiooeed CMI bound [58] in terms of I(W;U) U)). which assesses how well one can infer the "training-set membership" from the output hypothesis, our new CMI quantity, namely 1 (Z,; U, (W.), measures our ability to decide if an instance Z, contributes to the training of W or to the training of W,- when we know it contributes to only one of them. When W and W,T are similar, this decision (i.e., determining U₂) difficult, giving rise to small [(2;; U: (W). Additionally, for uniformly stable algorithms, sup.. A (w,) S f2 and it is also expected that Ew. A1 S .f. Moreover, if we simply replace "/s by an upper bound of ł, the second bound of Theorem becomes a uniform convergence bound if T (Z; U:) vanishes with n.
Proof Sketch of Theorem 4.1. Again we find motivation in Lemma 2.1. Additionally, the sym- metry exhibited in Eq. (5), analogous to the symmetry between 2, and Z used in deriving the standard CMI bounds, allows a similar development. Specifically, letting g(W;, &, u,) = (-1)" (l(w; 2) - E(ū 2)) and f t.-. for > 0 in Lemma A.2 enables the bounding of the CGF via invoking the assumptions in the theorem. The complete proof is given in Appendix D.I. O
Notably, our new CMI quantity in the bound preserves the boundedness of the original CMI [58], that is, T(2; ()) < HU,)) = log 2. Furthermore, parallel to supersample-condition CMI being smaller than IOMI [18], a similar result for hypotheses-conditioned CMI is given below.
Theorem 4.2. For any A and μ., we have [2;;; (W, S I(W; Zz).
Similar to Theorem 3.3, we present a CMI bound without square-rood root, which could be much stronger in certain regimes.
Theorem 4.3. Let Ai be defined in the same way as Theorem 4.1, and we let A(ws) 27
Ez, si (e(T) 2.) f(w; 2.)) / (w: then
(A) 1 n
E E.w.
A (W.) (TH
(10)
If A is further 32-uniform stable and SCHHDD stable, then
B2
7 82
[xa(A))
12Z + 0.72
(11)
n
A (W)))) which can be viewed as generalization bound in its own right. Additionally, it can be verified that A(wx) < 1 for any w. As W. (2; < log 2 ss 0.69, Eq. (10) can be further upper bounded by 1.41 This ensures that Eq. (10) will decay slower than O7ra).
Note that it is valid to set Ew
We also present second moment generalization bound.

---

Theorem 4,4. Assume that A is 32-uniform stable and symmetric with respect to S, i.e. it does not depend on the order of the elements S. Let e(--)) [0, 1), then
Ew.s [W) (ss)) 2 482 (.5((E 0.82 + +
1
n
n
Since [E) < O()), the bound can be further upper bounded by 0 + +//n), which matches the previous tight bound for the second moment generalization error in [14, Thm. 1.2]. Notice that [14, Thm. 1.2] only holds for the deterministic setting, while our bound also holds for randomized algorithms, and we also give stronger result in Appendix D.4 based our SCH stability notions.
gnpersaaple-Condittonn CMI Bounds is also possible to give Z-conditioned CMI bounds that explicitly contain the stability parameters. Let W, Wi,., and W, W., we have the following results.
Theorem 4.5. Suppose there exists 22 2 + such that sup EWW |e(w, 21) l(w', 2:)l S ew2 (2 for every 24, where W is the support of the conditional distribution Pw,Wfla. Then,
(A)
min Ez!
A2 A2 12 (W,, (W,,W&U.) U.)
Z [2( I(W, W U,ittt
The proof is deferred to Appendix D.5. The interpretation of the CMI quantity I(W, W ;: Up Ž appears identical to our carlier CMI quantity I I(Z (W)))
closer look in fact reveals that the two quantities are mathematically equal. To see this, first note [(2;; U.IW.) = H(U,) (U,) Wa) and I IWW,, W 4 U,12,t) H(U,)) H (U+)2, W., W.). Let random variable A (2,, W,+, ) and let A2 = (Z,+, W,, W.). .). these two random variables are identically distribution (since Pw: = W.: Zzz = Zz. and U, ~ Bernoulli- (2) and also we have PAL = PALIU.
so ,A U. U,.,U, which gives H(U,)A:) (()))A2). This indicates that I [(;; W.) [W,, W.: ;; ).
In Theorem 4.5, a data-dependent hypothesis space W is defined. A similar concept has been utilized in the hypothesis set cross- vols-vvaliotion (CV) stability studied in [16]. Furthermore, [16] derives some bounds based on either their transductive Rademacher complexity or their hypothesis set CV stability. They show that these two notions dominate in different learning scenarios. Given the close relationship between the supersample construction and the Rademacher complexity [58, 64]. and the inspiration behind our W construction, our framework is likely have fundamental connection to [16]. Additionally, obtaining the similar results of Z-conditioned CMI as in Theorem 4.3-4.4 is warranted, which may require some stability notions analogous to the average CV-stability in [16].
5 Conver-Lipchitz-Bountee (CLB) Problems
We now discuss two examples of the comvesLLipschitz--ouneedd (CLB) problem, subclass of SCO problems.
The first example is previously given in [21, Thm. 17]. in which nearly all previous information- theoretic bounds are non-vanishing. We will demonstrate that our CMI bounds are non-vacuous in this example.
[d] where e(i) is a one-hot vector with 1 at the i-th coordinate. Let U2) Given a sample S = {Z,)r_ drawn i.i.d. from we choose the 1-Lipschitz convex loss function f(w, 2) -ww, 2) and use GD to select a hypothesis w from (w E Rdd, || < 1). Let the number of GD iterations be T n² and let the learning rate be 7 a/n"
Example 1. Let d N and Z {e(i)
W
Let = 1 Ei= 24 be the sample mean. In this deterministic setting, it's easy to see that
ntt if nt $ 1,
(ntp/1ltt)))
otherwise.
7

---

Let ji' be the sample mean of s' and let wi be its corresponding hypothesis at time t. Notice that Euclidean projection does not increase the distance between projected points, namely non-expansive [22, Lemma 4.6]. Hence, whether Wt = or its truncated version n/tp/ p limited within the unit ball, we have ||us; will S ||nt - =pffl S (ntt/n). Recall that the loss function is 1-Lipschitz, we have (e, S B2 S (nt(/n). In this example, ny 1/ /n n so B2 E O(1/vn).
One can also directly obtain this rate from [22, 5].
Now following the same setting in [21]. if we let == -d=220 2n we can find that (WW;; Z.) E 2(1) (see [21. Thm. 17] or Appendix F). Thus, IOMI itself could not explain the generalization of GD this problem. Furthermore, all our CMI quantities including those in Section 6 also have the order of ((1), that is, they fail vanish as n + oo (see Appendix for more elaboration).
Therefore, the stability parameter 33 should not be replaced by some constant (e.g., the upper bound of the loss function) in the IOMI or CMI bound. In fact, for our CMI bounds, due to their boundedness property, we have the following corollary.
Corollary 5.1. A is 3y-uniform stable, we have B2 E VT(2; S ((3)).
Corollary 5.1 provides a solution to the the non-vanishing limitation of the previous information- theoretic bounds in Example I (and also the counterexample in [21, Thm. 4J), as it can explain generalization as long as the stability- bound is sufficient. Thus, the shortfalls of information- theoretic bounds in analyzing deterministic algorithms for CLB problems are tempered by the cubbilttybaseed framework.
We now show another CLB Example from [21, Thm. 3] where A is not uniformly stable, and we will see cntemmtiiontthhoreeee bounds in this paper are tight up to a constant. This example is also studied in [42, Sec. 5J.
Example 2. Let W E Rd be a ball with radius Ry, and let the input space be Z = / Ro. - 20/ Rob where 20 W such that || = Ro. Let μ. Unif(2), Consider a convex and L-Lipschiz loss function E(w, 2) = - L(w, 2). In addition, A is any empirical risk minimization (ERM) algorithm.
In this example, B2 2LR, is a constant. [21] has shown that [E,(A)) Z
LRa and I (W: S 1 22n
(see [21, Thm. 3] or Appendix F for an explanation). This gives us 2LRe Σi=1 VI(W: Z,) s
2LRoy I(W:S)
2LRa. Thus, the distributionndeeeedentt property of IOMI can improve the
72
Vn
Sabilittybasee bound in this case. In addition, we know that T(2,; W.) S I(W; Z,) from
Theorem 4.2, our new CMI bounds are also tight (up to a constant) in this example.
Notably, we can construct an additional example building upon Example 2, where A is uniformly stable but the uniform stability itself results in a slow convergence rate for generalization error. Specifically, let Ro and let d Vn, then .3 E O(1 while [A)] Z LL_ Note that the .22n
ifoomatiion-thoretti bounds in this paper can still provide a tight rate, namely ((1/n).
These examples demonstrate that our bounds can improve both the stability-based bound and information-theoree bounds in some learning scenarios.
Additional applications of our bounds are discussed in Appendix G.
Extensions
Connection with Bernstein Condition The Bernstein condition is commonly used derive fast- rate generalization bound for both PAC-Bayes bounds [69, 10, 36, 17] and stability based bounds [28], then it is natural explore the relationship between our fast-rate bounds and the Bernstein condition, formally defined below.
Definition 6.1 (Bernstein Condition). Assume that w" arg min (w) is risk mininizer. We say that the Bernstein assumption is satisfied with some B > and 6. E +o0) for any w E W,
Ez [C(w,, -) - l(w", 2))} < B (L,( L (x)
This condition can be easily satisfied many common situations [1, 28). In the following proposition, we can see that the Bernstein condition implies the SCHSCH stability.
8

---

Proposition 1. If the Bernstein condition is satisfied with some B and R. then i 7/- SC--BBBB stable, where 2 4BE (L) - [,,(w)))
Therefore, invoking the Bernstein condition, we can obtain the fast-rate bound as presented in Theorem 3.3, where we need to assume the loss is bounded, shown below by-product. Corollary 6.1. If the Bernstein condition is satisfied with it = 1 and & E [o, C), then
C I(W;Z.)
2.88B C (Ln ---((w)))
(A)) &
Recently, [65, 66] use the unexpected excess risk as the DV auxiliary function and invoke the (n, c)eentrll condition establish some optimal-rate bounds for specific learning problems, e.g.. Gaussian mean estimation. This again highlights the significance of selecting appropriate DV auxiliary functions and corresponding assumptions tailored to different learning problems. It is worth mentioning that the Bernstein condition also implies their (1) c)-central condition. Therefore, unifying these conditions can be considered as potential avenue for future research.
Loss Difference, Evaluated, and Functional CMI for Stable Algorithms Similar to [64], we derive some tighter bounds based on the loss difference.
Theorem 6.1. Let AL; = e(W, ž E(W,, 4, Zt). A 22-uniform stable, then
(x)) & V233 n
min { VIIALL U.). Ez 2 (ALBU.))
V232
We note that while is feasible to replace 32 in the (disintegrated) CMI bounds above with certain sample-conditioned hypothesis stability, it is not possible to apply the same substitution for the unconditional MI bound in Theorem 6.1.
Furthermore, notice that AL, - (L,, [.) (F) (W, W.) forms a Markov chain given Z,+ wherein (Le, L.) are the loss pair evaluated at Z,+ using (W,, W.). and (Pi, F3) are label predictions obtain e-CMI bound [58, 25], CMI bound [23] and recover 1WW,, W,; ( based bound from Theorem 6.1: T(A U, S F;;
of Z,₁ using (W, W.). By the data-processin piateprocessingg inequality,
< I(W, W U,, Additionally, notice that we can also apply the similar technique for the aypotheses-conddiione CMI, which should give the same results.
[[L,, L; U, U <
Expressiveness of New CMI Notions Under Distribution-FFre Setting Previous works [58, 19, 20, 23, 25] have demonstrated that the CMI framework is expressive enough establish connections with VC theory in the distribution-fre learning setting. Here, we further illustrate the expressiveness of the sample-conditioned CMI discussed in this work.
Theorem 6.2. Let = X' x {0, 1}, and les F = {fr X + {0,1)/ E W} be a functional hypothesis class with finite VC dimension d. Let d + for any algorithm A, we have
A EE / F.U S (V log (4)
Like the previous works, this bound matches the classic result of the uniform convergence bound [59]. Notice that the result could extended multi-class classification with finite Natarajan dimension [39] by proceeding similarly to [25, Thm. 8).
We invoke Theorem 6.2
demonstrate that new informatiin--heerriiic quantities have the same expressive power as standard CMI quantities. The expressiveness result for the (functional) repotheses-conndiione CMI is expected align with Theorem 6.2. This alignment is due to the equivalence between hypotheses-conditione CMI and supersample-conditionee CMI, as discussed in Theorem 4.5.
7 Related Works and Additional Discussions
Stability-Based Framework vs. Ceermaation-Thhoreee Framework Using stability methods
to analyze generalization errors can be traced back to several seminal works, such [51, 11, 12,

---

34, 27]. is worth noting that stability arguments have proven particularly effective in analyzing the learnability of SCO problems [56]. where traditional uniform convergence bounds may not be sufficient to explain the generalization behavior. The application of stability approaches has gained popularity for providing high-probability guarantees since the work of [6]. Recent advancements have further sharpened the convergence rates of high-probability generalization upper bounds for uniformly stable algorithms in a series of works [14, 15, 8, 28]. While information-thert thheorri bounds commonly used to analyze the in-expectation generalization, it is expected that combining them with the stability framework will yield sharper high-probability bounds.
Additionally, we note that in the realizable setting, where A is an interpolating algorithm, information- theoretic bounds exhibit greater power than stability- based bounds. For instance, the case of the
0 - 1 loss, information thhooretic bounds can achieve the known optimal minimax rates [19] and even provide exact characterizations of the generalization error [20, 64). However, it should be noted that due to the inherent fitting- srting-stablltt tradeoff property [54, Sec. 13.4), interpolating algorithms tend to be unstable, rendering stability arguments inapplicable. Given the prevalence of zero empirical risk in modern deep learning [68], it is natural to question whether cntormation-thhoreee bounds require the stability-based approach for analyzing non-convex (and potentially non-smooth and non-Lipschitz continuous) learning scenarios.
Connection Between Two Frameworks in Previous Works The connection between information- theoretic bounds, including some PAC- Bayes bounds, and algorithmic stability has been explored in previous literature [46, 33, 48, 29, 58, 23, 3, 47]. These works primarily focus on either regarding the information -hhoorti quantities notions of distributional stability [46, 67, 29, 58, 3] and/or converting information theorretcc quantities to some other algorithmic stability notions [58, 23, 47]. The later often relies the addition of Gaussian noise to the hypotheses (or assuming that the prior and posterior distributions are Gaussian in PAC-Bayes). In [33], the authors also combine the DV formula with stability assumptions, where they derive some PAC-Bayes bounds. However, comparing these bounds with others is challenging in general due to the presence of their hyperparameter stability.
Comparison with Standard CMI While our IOMI quantity aligns with the previous work [9], the new CMI quantity I I(24 U,) (or equivalently [W,, W U,[2,") )) may not be directly comparable to the standard individual CMI I(W in [49, 70]. Specifically, we have 1(2;; U,W,) = H(U,) HU,, W:) and I(W;U.1Z H(U, W.) and H(U,WW Z,) is not trivial. Exploring and quantitatively comparing these CMI measures would be intriguing research direction.
HU;)) H.U, W, Z,). The relationship between
Leave-One-Out CMI Our construction of W bears resemblance to the leave-one-out (LOO) setting, where there are also n + hypotheses. It is worth noting that LOO-CMI has been recently proposed in concurrent works [20, 47]. In LOO-CMI, the supersample Ž = Z[n+1] consists of n + 1 instances, and U is an index uniformly drawn from [n + 1] to select one hold-out instance. Consequently, Zu represents the testing data, while 1]/e serves as the training sample. In this context, LOO-CMI can be defined as I IWW U Z). Notice that this quantity still fails to explain the generalization in Example 1. The LOO setting is often associated with the stability-based framework [6], and is expected that the n + 1-supersample induced W could yield new CMI bounds that also contain the SCH stability notions. Nevertheless, in this paper, we do not adopt the LOO setting because in that case, (Ú) = log (n + 1), the LOO-CMI bound is no longer upper bounded by a constant independent of (note that the LOO-CMI bound for general setting in [20, Thm. 2.5] does not contain the 1/ V factor).
8 Concluding Remarks
We propose a novel construction of the hypothesis matrix and a new family of stability notions called cample-conditioned hypothesis stability. Leveraging these concepts, we derive sharper information- theoretic bounds for stable learning algorithms. Several promising avenues for future research include comparing our new CMI quantities with the standard CMI in a quantitative manner, analyzing the generalization of gradient- basedd optimization algorithms like SGD using our bounds, and establishing new high--robability generalization guarantees. Further discussions can be found in Appendix H.
10

---

Acknowledgements
This work is supported partly by an NSERC Discovery grant. Ziqiao Wang also supported in part by the NSERC CREATE program through the Interdisciplinary Math and Artificial Intelligence (INTER-MATH-AJ project. The authors would like to thank the anonymous AC and reviewers for their careful reading and valuable suggestions.
References
[1] Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv:2110.11216, 2021.
arXiv preprint
[2] Amir Asadi, Emmanuel Abbe, and Sergio Verdú. Chaining mutual information and tightening generalization bounds. Advances in Neural Information Processing Systems, 31, 2018.
[3] Arindam Banerjee, Tiancong Chen, Xinyan Li, and Yingxue Zhou, Stability based generalization bounds for exponential family langevin dynamics. In International Conference on Machine Learning, pages 1412-1444 1449. PMLR, 2022.
[4] Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory. PMLR, 2018.
[5] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33, 2020.
[6] Olivier Bousquet and André Elisseeff. Stability and generalization, The Journal of Machine Learning Research, 2:499-526, 2002.
[7] Olivier Bousquet, Steve Hanneke, Shay Moran, and Nikita Zhivotovskiy. Proper learning, helly number, and an optimal svm bound. In Conference on Learning Theory, pages 582-609. PMLR, 2020.
[8] Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory, pages 1110-626. PMLR, 2020.
[9] Yuheng Bu, Shaofeng Zou, and Venugopal V Veeravalli. Tightening mutual information based bounds on generalization error. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 587-591. IEEE, 2019.
[10] O Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning. Vol. 56. Lecture Notes - -Monogrphh Series. Institute of Mathematical Statistics, 2007.
[11] Luc Devroye and Terry Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Transactions on Information Theory, 25(2):202-207, 1979.
[12] Luc Devroye and Terry Wagner. Distribution-fres performance bounds for potential function rules. IEEE Transactions on Information Theory, 25(5):601-604 1979.
[13] Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil, and Leslie Pack Kaelbing. Stability of randomized learning algorithms. Journal of Machine Learning Research, 6(1). 2005.
[14] Vitaly Feldman and Jan Vondrak. Generalization bounds for uniformly stable algorithms. Advances in Neural Information Processing Systems, 31, 2018.
[15] Vitaly Feldman and Jan Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory, pages 1270-1279. PMLR, 2019.
[16] Dylan J Foster, Spencer Greenberg, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Hypothesis set stability and generalization. Advances Neural Information Processing Systems, 32, 2019.

---

[17] Peter Grunwald, Thomas Steinke, and Lydia Zakynthinou. Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vo classes. Conference on Learning Theory. PMLR, 2021.
[18] Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. Advances Neural Information Processing Systems, 2020.
[19] Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. Towards a unified information-thewee thhoretticcc framework for generalization. Advances Neural Information Processing Systems, 34:26370-22388 2021.
[20] Mahdi Haghifam, Shay Moran, Daniel M Roy, and Gintare Karolina Dziugiate. Understanding generalization via leave-one-out conditional mutual information. In 2022 IEEE International Symposium on Information Theory (ISIT), pages 2487-2492. IEEE, 2022.
[21] Mahdi Haghifam, Borja Rodviguez-Gálvez, Ragnar Thobaben, Mikael Skoglund, Daniel M Roy. and Gintare Karolina Dziugaite. Limitations of information-theoree generalization bounds for gradient descent methods in stochastic convex optimization. In International Conference Algorithmic Learning Theory, pages 663-706. PMLR, 2023.
[22] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225- 1234. PMLR, 2016.
[23] Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Information- theoretic generalization bounds for black-box learning algorithms. In Advances in Neural Information P Processing Systems, 2021.
[24] Fredrik Hellström and Giuseppe Durisi. Fast-rate loss bounds via conditional information measures with applications to neural networks. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 952-957. IEEE, 2021.
[25] Fredrik Hellström and Giuseppe Durisi. A new family of generalization bounds using sample- wise evaluated CMI. In Advances in Neural Information Processing Systems, 2022.
[26] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13-38 1963.
[27] Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. In Proceedings of the tenth annual conference on Computational learning theory, pages 152-162, 1997.
[28] Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate (1/n)). Advances in Neural Information Processing Systems, 34:5065- 5076, 2021.
[29] Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. In International Conference on Learning Representations, 2020.
[30] Nick Littlestone, Relating data compression and learnability. Technical report, 1986.
[31] Roi Livni. Information theoretic lower bounds for information theoretic upper bounds. arXiv preprint 23Xi::2302.04925, 2023.
[32] Roi Livni and Shay Moran. A limitation of the pac-bayes framework. Advances in Neural Information Processing Systems, 33:20543-20553, 2020.
[33] Ben London. A pac-bayesian analysis of randomized learning with application to stochastic gradient descent. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 2935-2944, 2017.
12

---

[34] Gábor Lugosi and Miroslaw Pawlak. On the posterior-probabilie probability estimate of the error rate of nonparametric classification rules. IEEE Transactions on Information Theory, 40(2):475-481, 1994.
[35] Colin McDiarmid. Concentration, pages 195-248. Springer Berlin Heidelberg. 1998.
[36] Zakaria Mhammedi, Peter Grünwald, and Benjamin Guedj. Pac-bayes un-expected bernstein inequality. Advances in Neural Information Processing Systems, 32, 2019.
[37] Michael Mitzemmacher and Eli Upfal. Probability and Computing: Randomization and Proba- bilistic Techniques in Algorithms and Data Analysis. Cambridge University Press, 2017.
[38] Wenlong Mou, Liwei Wang. Xiyu Zhai, and Kai Zheng. Generalization bounds of sgld for non non-convex learning: Two theoretical viewpoints. In Conference on Learning Theory, pages 605-638, PMLR, 2018.
[39] Balas K Natarajan. On learning sets and functions. Machine Learning, 4:67-97, 1989.
[40] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theorett generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems, 2019.
[41] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. Information- theoretic generalization bounds for stochastic gradient descent. In Conference Learning Theory. PMLR, 2021.
[42] Francesco Orabona. A modern introduction to online learning, arXiv preprint arXiv:1912.13211 2019.
[43] Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy. iterative algorithms. In 2018 IEEE International Symposiun on Information Theory (ISIT). IEEE, 2018.
[44] Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. Lecture Notes for 6.441 (MIT), ECE 563 (UIUC), STAT 364 (Yale), 2019., 2019.
[45] Tiberiu Popoviciu. Sur les équations algébriques ayant toutes leurs racines réelles. Mathematica, 1(129-145):20, 1935.
[46] Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu. Information- theoretic analysis of stability and bias of learning algorithms. In 2016 IEEE Information Theory Workshop (ITW), pages 26-30. IEEE, 2016.
[47] Mohamad Rida Rammal, Alessandro Achille, Aditya Golatkar, Suhas Diggavi, and Stefano Soatto. On leave-one-out conditional mutual information for generalization. In Advances in Neural Information Processing Systems, 2022.
[48] Omar Rivasplata, Emilio -errandee John S Shawe-Taylor, Shiliang Sun, and Csaba Szepesvári. Pac-bayes bounds for stable algorithms with instance-dependent priors. Advances Neural Information Processing Systems, 31, 2018
[49] Borja Modriguez-Gállezz Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. On random subset generalization error bounds and the stochastic gradient langevin dynamics algorithm. In 2020 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2021.
[50] Borja Rodríguez Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund. Tighter expected generalization error bounds via wasserstein distance. Advances in Neural Information Processing Systems, 34, 2021.
[51] William Rogers and Terry Wagner. A finite sample distribution frree performance bound for local discrimination nules. The Annals of Statisties, pages 506-514, 1978.
[52] Daniel Russo and James Zou. Controlling bias adaptive data analysis using information theory. In Artificial Intelligence and Statistics. PMLR, 2016.
13

---

[53] Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A, 13(1):145-147, 1972.
[54] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[55] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In COLT 2009 The 22nd Conference on Learning Theory, 2009.
[56] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11:2635-2670, 2010.
[57] Saharon Shelah. A combinatorial problem; stability and order for models and theories in infinitary languages. Pucific Journal of Mathematics, 41(1):247-261, 1972.
[58] Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory. PMLR, 2020.
[59] Vladimir Vapnik. Statistical learning theory. Wiley, 1998. ISBN 978-0-471-03003-44
[60] Hao Wang, Yizhe Huang, Rui Gao, and Flavio Calmon. Analyzing the generalization capability of sgld using properties of gaussian channels. Advances in Neural Information Processing Systems, 34:24222-24234, 2021.
[61] Ziqiao Wang and Yongyi Mao. On the generalization of models trained with SGD: Information- theoretic bounds and implications. In International Conference on Learning Representations, 2022.
[62] Ziqiao Wang and Yongyi Mao. Two facets of sde under an information-theoretie lens: General- ization of sgd via training trajectories and via terminal states. arXiv preprint arXiv:2211.10691. 2022.
[63] Ziqiao Wang and Yongyi Mao. Information-thherrtiic analysis of unsupervised domain adapta- tion. In International Conference on Learning Representations, 2023.
[64] Ziqiao Wang and Yongyi Mao. Tighter information-theorede generalization bounds from supersamples. In International Conference on Machine Learning. PMLR, 2023.
[65] Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu. Fast rate generalization error bounds: Variations on theme. In 2022 IEEE Information Theory Workshop (ITW), pages 43-48 IEEE, 2022.
[66] Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu. On the tightness of cnnmmatioonthhoretee bounds on generalization error of learning algorithms. arXiv preprint arXiv:2303. /4658, 2023.
[67] Aolin Xu and Maxim Raginsky. Information-theoreti analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 2017.
[68] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
[69] Tong Zhang. Information-theone thhoorttcc upper and lower bounds for statistical estimation. IEEE Trunsactions on Information Theory, 1307)1130 2006.
[70] Ruida Zhou, Chao Tian, and Tie Liu. Individually conditional individual mutual information bound on generalization error. IEEE Transactions Information Theory, (8(5):3304-333 2022.
[71] Ruida Zhou, Chao Tian, and Tie Liu. Exactly tight iiformation-theoreti generalization error bound for the quadratic gaussian problem. arXiv preprint arXiv:2305.0088 2023,
14

---

Appendices
A Some Useful Lemmas
In this paper, there are some equivalent forms of the generalization error we will study, e.g., Eq. (2) and Eq. (5) in the main text, which are presented in the following lemma.
Lemma A.1. Let W W.,, and W; W., ",. For any learning algorithm A, the following equations hold
(A)
1
Σ
n2
E
[(W, 2,, ((Wit22+)
(12)
W.
71
1
Bi: E2.00.1 [[-1]0c (e(i)= (((W,,2,--0 - ].
(13)
71
i=1
Bw..W.o..2 ((-1)² - e(w, 2, 21) }]]
Ez
(14)
Proof. This lemma is a consequence of Lemma 2.1, with further utilizing some symmetric properties. Recall Eq. (1) in Lemma 2.1,
]
(A) - 22t 2mm
ww: 13 3, eW, 22,)- - -E 1 e(W+, 23,)
1
(WW, -,, - e(ii, (i,,t.:.]]
= BB x W
E Ezt .. [e(W: ž - -(W+,,+) )
Note that Eq. (2) in the main text is from the second equation above, which is used to derive individual IOMI bounds in Section 3.
Similar to the standard setting for CMI bounds, where the role of each ž,- and 2, can be exchanged, a key observation here that for each i, W;+ and W,- can also be exchanged arbitrarily. That is to say,
(A) Ez W.
]
(15)
also holds true. Notice that we do not change the definitions of any the random variable, e.g.,
W+ = A( R) and W;- = =A( [n|=a
Rt).
What differs from the standard CMI is that the roles of the whole sequences [tt and [.j are not
exchangeable with each other. Here, when we exchange each 7,' and ž, we need to keep the other positions in S unchanged.
By introducing U, ~ Unif(10, 1J), we have
1
[(WT, (7,,7)))(((1) - l(W,t, }
(A)
E Zz .W.
n
1
[e(W,, Z..u.) - e(Wer Z.w.).
Ez. W.
15

---

To obtain Eq. (13), notice that 2, = Zo,0, we have
12
(A)
E 2, W.UUU
E(W,,, -.V,,,,))
7
(-1) (-I)).(((V, - (...))))
E 2, W.
(16)
This, as we have already seen in Eq. (5) in the main text, is used derive typotheses-conditee CMI bounds in Section 4. It's easy to see that when Uj = 0, Eq. (16) becomes Eq. (12), and when U, 1, we obtain Eq. (15) via Eq. (16).
To obtain Eq. (14), we let W, WL,,, W. W. and fix Z, = Z Similarlyy
(A)
2
<(W, E Bw.,w.u (-1) (v..
72
This is used to derive supersamplee-condttioeed CMI bounds in Section 4. It's easy to see that both U, = and U, will give us Eq. (12).
Like all the previous information -theoretcccc bounds, the following lemma is widely used in our paper. Lemma A.2 (Donsker- Varadhan (DV) variational representation of KI. divergence [44, Theorem 3.5]). Let Q P be probability measures on O for any bounded measurable function f O R, have PQ] = sup, Es-Q [f()] Ea~p jexp [()]
We also invoke some other lemmas given below.
Lemma A.3 (Hoeffding's Lemma [26]). Let X E [a, b] be a bounded random variable with mean Then, for all & E R. we have E. [e I e/u+ (3a--)22 Lemma A.4 (Popoviciu's 's incquality [45]). Let M and m be upper and lower bounds on the values of any random variable X, then Var(X) < (M-m)?
The following lemma is from [35, Lemma 2.8), we provide a self-contained proof. Lemma A.S. Let h(x) = ---0-1 be the Bernstein function. If random variable X satisfies
E [[ = 0 and X < 6, then E [ex) < [6bB[[x))
Proof. It's easy to verify that h(x) is an increasing function for x > 0. Thus, h(x) s h(b)) for æ s b. Then,
= 2²h(x) x x 2b)
For the bounded random variable X with zero mean, we have
E [e x < E [ 1+ E [xb) s (())[(2))
The last incquality is by ex 2 r + 1. This completes the proof.
B Further Elaborations on SCH Stability
We note that the reason we introduce four types of SCH stability in Definition 2.1 is that solely using 32 in our bounds might be too loose, as it considers the supremum over all sources of randomness. By incorporating SCH stabilities, we aim to demonstrate that theoretically, we can achieve significantly tighter stability parameters.
The basic set up is as follows. Assume random sample 5 gives rise to W. For each Z, S, construct St by replacing Z, with another independently drawn instance; call training result W the neighbor of W.
In a), SCH-A stability measures the difference between the loss of w and the expected loss of its neighbor W worst and the worst possible While in (b), 72 SCH-B stability measures the
16

---

square of this difference, not in the worst case, but in an average case, where the average is over an independently Z' for the loss evaluation, the training sample, and the algorithm randomness. Since "average is smaller than worst", S 71-
In c), we consider the difference between the loss of W and the loss of its neighbor when evaluated at the worst possible Z; that when included in S gives rise to W. The expected value of this difference 73-SCH-C stability.
In d), SC-HCH stability measures the expected squared difference between the loss and the loss of its neighbor when evaluated at Z, (a member of 5). For similar "average smaller worst" reason, one expects that 7/1 S 7/3-
We expect that 72, 73, and 7/4 are all smaller than 31. This is because in 31, we consider the worst evaluated instance, whereas in the other cases, we take the expectation over all instances. Additionally, in Theorem 4.1, we expect that Ew. (W,,) & B8. this is because B1 -sttabilyyy holds for all the possible and s², namely it holds for all the (w, w²) pair (that shares the same randomness) while in Bw. (W.), we take the expectation these pairs.
We expect 72 I 74 due the following reason: first by Jensen's inequality, we have Es [ Z") - -wriww(W' .Z'] S Ew,w [e(W, Z") - E(W .Z')] 2 then since Z' an independent of both W and W", Zz can be regarded as a testing point for both and W', we could ex- pect that the expectation of É(W; (') Z²) is small. While in Es,. [e(W, Z.) E(W', 2:) Z; is a training point for obtaining W. E(W, Z,) could be small in general, and Z, is test- ing point for W Therefore, is reasonable to expect Ew,ws, 22 [&(,, (') - E(W z²) s Es [e(W, Z.) E(Wi, ] namely 7/2 S41
As a concrete example, let & be zero-one loss and assume A is an interpolating algo- rithm and and randomly makes predictions for unseen data. By Jensen's inequality, 7/ S Ew,w5,2' [<(W, Z') E(W", I²)]
Ew,z' [e(W, Z')] 2Ew,w ,2 Z')(W' 2Z))]
E [e(W', ? where we use f = é for zero-one loss. Since Z' is an unseen data for both W and W we have 73 < EEys, 2' [[(Ws, 2²) + 2 2 Ew [((W', z²) While in
this case 7 = E 2, [e(W's 2+)] so 72 S 74-
2
C Omitted Proofs and Additional Discussions in Section 3
C.1 Proof of Theorem 3.1
Proof. Let g(w ((w)(,)))))))))
.,/) = -. jat [eW,, , - -(ww ,2,) be the average loss difference between
w and its neighboring hypothesis, and let f = 1 g for > 0 in Lemma A.2. Let " be an independent copy of 2,+, then
I( + log E Ew 27" [elo(T- (Z'')
Ew++ 2. [o(W+ 2 S inf 1>0
(17)
Since Z,"' is independent of both W,- and W+, and W.- and W are identically distributed, we have
E W 2 ((( _ = Bw: 2," (eW,, [ E W 2t [((+, = 0.
By the definition of -SH--A stability,
(eW, +2] l(ww, .2) S Sm.
sup
E w. -
6
17

---

so g( W++ .//
is zero-mean random variable bounded in [-n, ni By Lemma A.3, we have
T(W+; Z log B .2/4
[elo(i- (2,'
W+ 37
((W+Z,+)
- inf
1>0
t
(W++Z?* + 1
inf 1>0
-V 27
1(W+,2
where the last equality is obtained by optimizing the bound over 6, i.e. letting t
2x
Recall Eq. (12) in Lemma A.I and applying Jensen's 's inequality to the absolute function, the first bound is then obtained by
(A S
E Ew+ 2t [o (1
71
2 ;2,),
1=1
Furthermore, by the chain rule of mutual information,
1(W;;; 7 ( + I(W+; 2, = [ (W,) 1(W,;;
(18)
Notice that ;WWT22;) 0 in the RHS, we have
(W Σ S 1( I1,,)
which will give the second bound. This concludes the proof.
Remark C.I (Comparison with Mutual Information Stability [46, 23]). To compare with the mutual information stability T (W+; 7 1
m ))
recall Eq.(18): I(W+; 7 (W-) = I(W, :W, (
I[W+; ), and similarly we also have I(W+; ; 2, 2 (m)))) = I mn) 2 (W+) + I( 2
Thus, we only need to compare I(W; 2 IW+ and I ( 2,' W+) Notice that for a determin-
istic A, have T(W, 7 W+) < T T(7
(1 7 ; Z IW
Since 7i
11. (W+ (irt,7tt)) we further
have [(Z ;;: ( = 1[ 1( (n) ; 2,7" (W+) which gives us the desired result:
r( 2 W Σ I
W 2, k
C.2 Proof of Theorem 3.2
Proof. The proof is nearly the same the proof of Theorem 3.1, except that now the randomness of the algorithm is given for each DV auxiliary function, so the randomness W; is completely controlled by Z.
Let g r)) Ew- atttr [e(W, . 1(o ; xxt) and let f f t. g for 0 Lemma A.2.
Let 7 be an independent copy of 2,, then
T( 7, R r) log Ew 2,' [e m))
E 1 g(W+ 2, ,r)
inf t2
Notice that
Bw+ty' Ir (( 2,,1" r) E w 2/1
((W; ((W,,2,t)) -,,1 E Bwt,zt'1 17 (e(W. _z,i
still holds since 2 and 2, are i..d drawn.
18

---

Thus, g(W+ - .r) is a zero-mean random variable bounded in 1-32, 82]. By Lemma A.3, the remaining part is routine:
@(W+,Zt+ rr] S 3
E W 2
Thus,
B 27. n [ .1,,, 11]
82 E 21 (+;
(A)) S
This completes the proof.
C.3 Proof of Theorem 3.3
Proof. Let h(z) = e--z-1 be the Bernstein function. Similar to the proof of Theorem 3.1, we let g(wt,2t) = Bw, 1 ,(W,,
x²
2 {(u
We have already known that
W 2 (o(W+,2'')
0 and g(W+, - S sn By Lemma A.5,
log B z [eto (21') <h( ( WW
[(BV-I NWWT,2t3 ((W+,t)) )
<h(rat)t²2
where the second inequality is by the definition of S2-SCH-B stability. Plugging the above into Eq. (17),
(doOW .2,))
(W++2,² + log Ew 2,
W 27 (g(W+,27'
$ inf
1>0
I(W+; + 1(m)))122
< inf 120
t
Usually we have 2/ 3 7I, we let 1/71,
then
h(1)2 xe 0.72
12
h(nt)e
71
71
Thus,
(A)) Σ
71 n
I(W+; 2+) +
0.7272
7/1
This concludes the proof.
D
Omitted Proofs in Section 4
D.1 Proof of Theorem 4.1
Proof. We now prove the first bound. Let (1,0,,2,,4) (-1)" (f(w, 2.) - e(wüt 24)). By
Lemma A.2, we have
U.W:U,WW wx) + log Ez,
(e00(1022...))
E E, (g(ū,, 2, U.)
in t>
(19)
t
Since U. 11. 2, we have Byy lg 21, U(}] Euy [-1)03 (0(0,, - Elat 2.)) for any
w and 2, Ergo,
E2, nz
Bug ( 2, U7)] =0.
19

---

By the definition of (20)),
g Zr, U/} = ((w;; Z.) -((w;t 2.)
sup =
[( 24) E(w;, 24) s (w.)
Thus, g(0; 2g, U?) is a zero-mean random variable bounded in ( A (w:)) for a fixed i;
By Lemma A.3, we have
E 2,
Plugging the above into Eq. (19),
1(2; U;|,; =;;) + (a,)
E E (g(ūo,
Z,, U.)
inf
(20)
- 21 Zz;
U.W: = w.))
Recall Eq. (14) in Lemma A.I and by Jensen's 's inequality for the absolute function, the first bound is obtained:
n
(A)) -
21
(21)
W;
To prove the second bound, we return to Eq. (20), and take expectation over W, first. By Jensen's inequality,
FEW, [A(W.)]]
I 1(2,: U,( +
E R. [o(W, 2i,
(1 U²)
- inf 120
(22)
t
- A(W.)2 (2::
Therefore, have the second bound below
(A)) S
A(W.)]][(22:.111))
(23)
For the second part of Theorem 4.1, notice that it's valid to let 73 Ew, A(.) then recall Eq. (21),
Bw A(Wa)
273
I UZ)
sup (.E(W.
This completes the proof.
D.2 Proof of Theorem 4.2
Proof. The proof is similar to [18, Theorem 2.1]. By the chain rule,
1,Z W. = U2; W. + 1(2,;
(24)
Since H W.) H2,,W, U, W:) = W,), we have T(2;; U., W,) = H(2))
H (2, W:) ((.)) -2(2,),, I(2;; W.). Thus, 1(2, U. W.) I(2;; W.). Recall
Eq. (24) and by the non-negativity of mutual information, we have I [2; U, W.) & I(W;; 2,). Note
that [W;; 2,) :(W;; I(W; Zz). This completes the proof.
20

---

D.3 Proof of Theorem 4.3
Proof. We first return to Eq. (19) in the previous proof, and we have already known that g(ws, 2,, U!) is a zero-mean random variable bounded in [ A1 (w;) for fixed iz;
By Lemma A.5, we have
[o(Te, Z., 01)9 <h <h ((.))) 6 2 U/0..
log E 1,10
[ 2,) e(10t, 2.))
=h (t)) E
Plugging the above into Eq. (19),
Eg 2, [0 2,, Ui)
- inf 120
[Z; U, W = w..)
+ (O:))) 2,, (e(ir 2.) -((i0 2:i)
t
(25)
Lett= Ar(01;3 have
(e(iT,, e(iit 7n1) 27
Eg ii
12,00 [g( 2., S (A((T)) = +0.77
(
Let A(w:s) = E2iiie (1(07 22) - - 2))) ///>///))))) tthnn
then
1 (A)) v
E w
(W.)))
For the second part, if A is further B2-uniform stable, recall Eq. (25) and by the non-decreasing property of h, we have
< inf [(2;: U, W. wi)) th (B2t) 2, [1(0 2.) El1o, 2,))
2, [(
2,, U.)
Lett = 1 and taking expectation over W,, we have
[eTT; ,2,) e(W: 2,))²
E Eg.w.
0.72 E R.u...W [W,, 2, U.) 582T(2 0.72
Ba
U, 0.72 0.72
where the equality is by the definition of 7--SCHD stability. Thus,
2
(A))
< n
SiI(Z; + 0.72
13
This concludes the proof.
D.4 Proof of Theorem 4.4
We present a stronger version of Theorem 4.4.
Theorem D.1. Under the same conditions Theorem 4.1, and we further assume that A SCH-B stable and symmetric with respect to S, i.e. it does not depend on the order of the elements in the training sample. Let (W) = ΣT A,(W.)² we have
6
AI(W)) ( (1 (E:U)) + log 3 }]
1
Ew.s [LS(W)) (W))) < n
+ + 472.
w
2
n
21

---

Then Theorem 4.4 is corollary of Theorem D.I.
Proof of Theorem 4.4. For 32- uniforf stable algorithm, by (W( - 32 and 7/ S 32 we have
Ew,s (LS(W)- (W)) 632
I(EÜUW) + log 3
+
482
n
L551(E + 0.82
1
433
+
11
This completes the proof.
Before we prove Theorem D.I, we need to first obtain the following lemma.
Lemma D.I. Under the same conditions in Theorem 4.1, let AI
- - A (W:)) we have
Ew,s 1
n Ew ww [e(W*, Z,]] - Is(mm)
(I (E:U)+ log
)
E W
(,(W)
Proof of Lemma D.I. Here we borrow some proof techniques used in [58, Thm. 2].
Let g(ū, 2s, u:) (-1)a: (e(w), 2.) A(ū, 2:)) and let G ~ _/(0, 1) be an independent standard Gaussian random variable. Let f = (2 Em= g)² in Lemma A.2, then
I(EUW = w) + log Eg [e 07))
1 Σ
EF it
ب 2, U.)
< inf
120
I(EBUW = w) + log E E
EG e
Gv2I 201 ( (,,0%)
inf
8>0
(26)
I(E = w) + log Ba
I Byy e
Gvm s( 22,0%)
= inf
ES0
0 O2IE 01(101)
I(EBU w log Eg
inf
(27)
120
t
I(E; U w) + log 1/v
28 F.
(1
inf
16 T> A1(wy)²
(28)
I(EUUW -w) - log (1 2t Σ"" (10.)
inf
16
2 (1
where Eq. (26) is by the moment generating function of Gaussian distribution: E& -2a = +
for all A E R, Eq. (27) is by Lemma A.3 and Eq. (28) is by the moment generating function of chi chi-squared distribution: Eg AAc < - for for }
n² Lett= 3E A1()) be substituted the last equation above, have
( [
E EggUJ
g(ū, 2, 2..1.)
<
Σ A (wx)) 1=1
(IE:UI] =w)+=
w)+ log 3 2
(29)
n²
i=l
22

---

Let A1(0)= = 1/2 Ex= ( and taking expectation over W for both sides,
E (( 2, -o
)
3
(((M))
(I((E 2
+ log }
E
F.U.W
<
(30)
Applying Jensen's 's inequality to the square function, we have
(
27
2
ト n1
ww [&(W*, Z:]] (ss(W))
n [ s(W Ui)
Ew,s
E B.U.W
Combining Eq. (30) with the inequality above will concludes the proof.
We are now position to prove Theorem D.I.
Proof of Theorem D.I.
Ew.s
(Ls(W) - (W)))
27
n I(W, 74) Ez (e(W, 21)1
=Ew.s
27
n
n
n Ewiw [e(Wi, 2.)] Ez (e(w, Za11
=Ew.s
e(W, Z+)
Σ
Ewiww [((W' [i]]
n
n
1
e(W, 21)
Σ
Wwiww [e(W*) zii]
Bi
+ 2 Ew.s
Σ
E Bwiju W [((W', 2a]] Ezy [e(W, Z1)
where the last inequality is by (x + y) S 2x + 2y²- Notice that B₁ can be bounded by using Lemma D.I. We now focus Bz. Since Ez (e(W, Z')]
1 >. Ez [e(W, Z')], we have
27
B2 -Ews
Wwiw [((W', [()
Ez [e(W; 3') Bwij (((W:,)) + Ewiiw ((W:,))) Σ ==1
1 Eniju w le(Wi, (2) - Ez ((WWi, 22))]
Ezi [e(W,2') - -Ewi (e(Wi, 23)1|
-Ewss
1=1
1=1
27 Ez [((Wi, z')]]]
<2Ew.s
Ewiww [((Wi, Zz)
Ba
1
+ 2 2EW
Σ
Ez [e(W,
Ew w [e(W*, [(W]
i=1
B4
23

---

For B3, we apply Jensen's 's inequality to move the expectation over W outside of the square function,
{
H Σ &(W*, Zz) Ex [e(W*, : 21)]
B3 w3,.,
27
(A(SS, R), 2.)
2.) 2
-E) Ezy (E(A(S", 21)
Notice that S", and R are all independent with each other (so W Zi and Z are also independent with each other). If we further let A be symmetric, namely W does not dependent i, then the inequality above equivalent to
27
n E(W, 2
B3 EEws
Eg 1& 2')1
=Ew
Egg
e(W, 2!) Ez (e(W, Zz)]
n
Hence, the inner expectation in the RHS above is just the variance of the sample mean of i.i.d bounded random variables. Recall that .) E [0, 1]. thereby
Varz (E(W, Z3))
1
B3 S Ew
4m
where the second inequality is by Lemma A.4.
Then, for B4, we also apply Jensen's inequality to the square function, and by the definition of -2-SCH stability, we have
e(W Ewiw [e(Wi, 23)]
B4 <Ew
1
Ew.z' [((W, - EEw [e(Wi, z2)])² S S
Putting everthing together, we have
Ew.s [Ls(W]- (.
<2B1 + 2B2
<2B 4B3 4B4
1
SBB + + 4
6 (W)
W (E:UE) + log 3
+ + +12
where the last inequality is by Lemma D.1. This completes the proof.
D.5 Proof of Theorem 4.5
Proof. Let g( Wa, W,, u&) (-1)ut (e(w,, - E(w,, ()) Again, by Lemma A.2, we have
I(W,, W W; U.IZ at) + log E W. W. Wuli
E www.WUR.E [o( W., W. U.) < inf
E50
t
(31)
24

---

Similar to the previous proofs, it's easy to see that 9(2 W,, W, U,) is zero-mean random variable
bounded in 1~ (2 )]. Thus,
I(W, W = x 1² )²)
E Ew..w.e.
(g( Wt, W W., U.] S inf 130
(32)
To prove the first bound, we let t (W..W.U.l..
then
2A²(E?)²
B. W.U.F [Ø(2, W,, U.U) $ $A2(2)) / 21 W W.U.ZZ. = =,,) Recall Eq. (14) in Lemma A.I, hence,
E (A))
V !
A2(2,t) 21 (W Wi;
(33)
To prove the second bound, we take expectation over 2 for Eq. (32),
e 2 (2)
I(W, W. U, 2
+
W.W.W.U.... g( W,, W, U.) S inf
120
t
1(W, )
Lett
2B2z A2(2))3 then
B... 19 W.
W.,,W, W..
s S 2E
ba(zt)/((ws. w.; W.: U.l
Ergo,
n [Ea(A)] -
Σ
2E
A2(2 I(W,, U.
(34)
This completes the proof.
E Omitted Proof in Section 6
F.1 Proof of Proposition
Proof.
By Jensen's inequality and triangle inequality, for any i E [m]]. we have
E..R.
[C(W, - Ww [<(W",2')]) 2))1)²
[e(W, -e(wi, =² ?(Wi,z'))?
<Bwwwi
[((W,22) - -l(w", Zz' + E(w", 2') - e(W',
-Bw.w 2'
<2EW.2'
(E(W,Z') -(ww, z²))²
+ 2E , [e(ww), Zz) E(w", zz')²
"('",,z')²"
<4BEw
(W) - La (w")) #
where the last inequality is by the definition of the Bernstein condition. This completes the proof.
E.2
Proof of Theorem 6.1
Proof. Let ( u.) = zero-mean, then
Notice that (g(A,,, U\) s B2 and g(A U.) is agian I(L U,) + log E FALLYY (colssnn
EAL.U, (g(ALi, <
<82 2
25

---

Thus,
[A)) S TT
Σ 21(L;;; i=1
To prove the disintegrated CMI bound, we let be defined in the same way, and the remaining development is the same with the proof in Theorem 4.5.
For the second inequality, notice that (AL; U,) I U, by using the chain rule of mutual information and the independence between 7 and U,. In addition, moving the expectation over 2 inside the square-root function by Jensen's inequality, we have EE z+ / (ALi U.) S
V
E.3 Proof of Theorem 6.2
Proof. Before we prove Theorem 6.2, we first show the following lemma.
Lemma E.1. For any i [n], we have Ei= I(F, s (FFi)) FImj: Ul)
Proof of Lemma E.I. First, by I[F,, P.; U. I(F,, F.; ( = H(U,) - Fi Fú, [nj) and notice that HUU;F, F,, (mi) 5
P.;UU Z,
H(U,) HUU F,, 2,4) and
FR Fú, 2t), we have
I(F, F. U,IZ)) S F:
(35)
Then, using the chain rule,
[(Fr, Fr; (() [(Ppn) PIojE U/ U1] F,, Ft) I(Fin), Pin5 11)
By the non-negativity of mutual information, we have
T(F; R(U S [(F))) Pn] Ug (z1))))))
(36)
Furthermore, by the independence of each U; (i.e. I(U; Um]yy (2m1)) = 0), we have
Σ (LR,, U.lZ - [(Fin), Fin] U1 U12
(37)
Combining Eq. (35-37) will conclude the proof.
We now prove Theorem 6.2.
For given 7 the number of distinct values of their predictions, denoted by are upper bounded by the growth function of F evaluated at n,
where the second inequality is by Sauer-Shelad lemma [53, 57] for n > + 1.
Thus,
[(F).), FInj: U| & (F] 1m1) H(F [m)) Tサ + (F 7 - 2d log
enn
(38)
26

---

By Jensen's 's inequality and Lemma E. we have
n
[[Fin)
Fimls U
(
T(F F;
T(R, P:U.1 sv
7
Plugging Eq. (38) into the inequality above,
1
(v/ log (7)
((F,PR so
which completes the proof.
F
CLB Exampls
In Example I, [21, Thm. 17] demonstrates the non-vanishing behavior of individual IOMI and e-CMI. This is primarily attributed to the dimension-deeendern nature of TOMI and CMI. Specifically,
there are certain dimensional settings where IOMI can grow faster than ((n), as shown in [21, Thm.4], and CMI approaches a certain fraction of its upper bound, as illustrated in Example resulting in non-vanishing behavior. Specifically, in Example 1, [21] employs the birthday paradox [37, Sec. 5.1] problem demonstrate that for large value of d, the probability that no pair of instances in sharing the same non-zero coordinate (referred to as event Eσ) is smaller than a constant probability (that could be independent of n). Particularly, it is shown that if d 2 T-ol/(2n 17+
2n
then P(Fo)
(1 2n 2n-1
As an example, [21] chooses d = 2n resulting in 0.1.
Failure of I(W; Z,) Consider the case where d 2n². For the individual CMI [49, 70]. ([WU,))) 22). we have the following:
I(W: U. = log H(U. W. Z,) 2 0.1 log
This inequality holds because when event Eg does not occur, completely, as the returned hypothesis is weighted sum of the sample. other words, examining the non-zero coordinates of W is sufficient to determine U4. For an in-depth derivation of this inequality, readers are referred to the updated version of [21], where their corrected proof involves Fano's inequality. Furthermore, using the relation (W; Za) 2 (W;U,12,) [70]. we conclude that I(W; Z+) E ((1).
can determine the value of U,
Failure of I(2,; U,WW) Notably, our hypotheses-conddtionee CMI also does not vanish for the same reason. More precisely, when W; and Z, are given, there exists constant probability (indepen- dent of n) that allows us fully determine the returned hypothesis based 2,, thereby determining the value of U,
Failure of I(AL; Furthermore, even the loss difference based CMI (e.g., as shown in Theo- rem 6.1), which provides the tightest CMI measure, still does not vanish. This is attributed to the fact that if the hypothesis W is independent of certain Z, there exists a constant probability where the loss becomes zero (recall that the loss is the negative inner product of W and Z). Consequently, one can determine the value of U, by observing the sign of the random variable Al.. This also indicates the limitations of e-CMI and f-CMI in capturing the generalization behavior for Hxample 1.
In Example 2, following the approach in [21], the training sample S (2 1¹" can be repre- sented as S = Rg ($1, variables, i.c., E
én), where {er) is sequence of independent Rademacher random ((-1,((1,,))) The empirical risk is given by Ls(W) -L (W, Ei= E1). In this case, the ERM solution WERM = 20 if sign(> Es) = 1, and WERM = 220
20.
sign(En Ex) = -1. is clear that
sup [(( z) E(w", 2) < sup Ellw - w
will S 2LRo-
w.wi.z
27

---

Hence, we observe that B2 is now a constant, whereas IOMI has an upper bound: Σ I,W S S) < (ign e.)) < H (sign(Z i=1 E4) S 1, where the second inequality follows from the Markov chain S sign Ex= E4) W. This provides us with a generalization bound of 21.Rg Vn Meanwhile, the actual generalization error satisfies (A( 2 LBg (see [21, AppendixB] for derivation). Thus, the IOMI bound is tight up to constant, and the stability bound [3 itself is vacuous. It is worth noting that I [(2 (W) S (W; Z.) s by Theorem4.2, indicating that the CMI bound is also tight.
V2n
We would like note that the failures of chained mutual information bounds [2] are not demonstrated in the counterexamples presented in [21]. Notably, when the hypothesis is quantized, it becomes more challenging to guess U; Z, Therefore, exploring the potential of chained information -hhootii bounds, which do not necessarily rely on stability notions, could be another avenue to explain the generalization behavior observed in these counterexamples.
G
Additional Applications
G.1 Compression Schemes
We now consider the algorithm that has a compression scheme [30]. Formally, a sample compression scheme of size k E N is a pair of maps A2). Specifically, for all samples s with n > k,
A 2 - 2* compresses the sample into a length-k subsequence A (s) S 8. Then A 2 + W could be some arbitrary mapping. Hence, A(-) = A-)) Let K be the index set for S selected by A1, and let K be the selected index set for S In this case, our supersample- conditioned CMI has an upper bound: I(W,, W ( S T(K, K; U, 2 S HK,, K(Z,)) S 2 log (l) < 2k log n. Then, if A is further 32-uniform stable, then we have the generalization bound (A) < ( (1322/logg n.. If Bg < (l/ this bound improves the bound in [58]. It is unclear if we can obtain any improved bound for stable compression schemes [7], in which case [19] provides an optimal bound that removing the log n factor for the realizable setting. A main difficulty is that an interpolating algorithm is usually unstable due to the fitting-stabin stabilitt tradeoff [54, Sec. 13.4].
G.2 Distillation Algorithm
The high-probabillyg generalization property of distillation algorithm is studied in [16]. In the first training stage of distillation, we obtain a wt from a highly complex hypothesis space Wi based on a training sample s. Same to [16]. we assume that the first learning stage is a-sensitive, namely ||w w || S a (L/nn). In the second stage, the algorithm AA will select a hypothesis that is X-close to " from a less complex hypothesis space W22 = {w E W ||w-- will> I A). Let the loss function be I.-Lipschitz with respect to the first argument. Consequently, 7 S L w w |l S La.
Then, by Theorem 4.1, we have (A) < - (2H(U,) = v log La. Notice that the loss here may not necessarily be bounded or sub-Gaussian, rendering previous bounds inapplicable.
G.3 Regularized Empirical Risk Minimization
Regularized Empirical Risk Minimization (ERM) learning rules involve minimizing the empirical
risk and a regularization function jointly: arg minwew (s(w) + freg(1)) (w), where Frog : W - IR. Here we specifically consider Tikhonov regularization [54], namely fre((w) = Al| w 12, where À >>> 0
a tradeoff coefficient. The regularized ERM algorithm AA aims to find
arg min Ls(w) + A/|w|/2
w-
wEW
This regularization term ensures strong convexity of the training objective. Based on Theorem 4.3, we can derive the following results.
Corollary G.1. Assume that the loss function E is convex and L-Lipschitz. Then, for the regularized ERM algorithm with Tikhonov regularization, we have
2L2
(A)) 5
1( U. W.) + 0.72
An
n
2=1
28

---

Proof of Corollary G.I. By invoking [54, Corollary 13.6), we know that 744 S 32 the value of 32 will give us the desired result.
21." Plugging
An
Corollary G.2. Assume that the loss function is p-smooth and nonnegative. Let X 2 2p Then, for the regularized ERM algorithm with Tikhonov regularization, we have
48pLn An
(x)) S
T(2; U, W. + 0.72
1=1
Proof of Corollary G.2. By invoking [54, Corollary 13.7], we know that 74 B2 48pLe. Plugging the value of 32 will give us the desired result.
Although these bounds do not enhance the convergence rate of ((1/n) in these settings, they con- sistently offer tighter results compared to uniform stability-based bounds if 4 Ei=1 I(Z; U,JW) < 0.28. In addition, the expected empirical risk I.n appears in the bound of Corollary G.2. While X has lower bound, L, could not be arbitrarily small for the regularized ERM.
Notice that previous information thhooretic bounds could not obtain the convergence rate of ((/nn) as in our results unless ICMI or CMI itself decays with (1/n))
H Additional Discussions and Open Problems
Stochastic Gradient Descent (SGD) Since the influential work of [22], stability approaches have been widely employed to provide generalization guarantees for (sub)gradient-based optimization algorithms, such as SGD, under certain conditions like the convex- sewrecx-mmoth-Liipe Lippshhtti loss. More recently, [5] extended the results of [22] to the non-smooth loss function in the SCO setting.
In contrast, information-thhoredee (weigh/hyyoothesis-ben based) bounds are typically used to analyze the noisy version of SGD, known as SGLD [43, 40, 18, 49, 60]. Directly analyzing SGD poses challenges because the returned hypothesis W contains a significant amount of information about or Z,, resulting in potentially large (even infinite) mutual information. The prevalent approach to applying cooommtiion-thorretee bounds to SGD is by introducing noise [41, 61], but this has been shown to yield non-vanishen non-vanishing bounds in [21, Thm. 4].
The combination of information-theorettii bounds with stability for analyzing the generalization of SGD presents a promising future direction. However, some potential difficulties may arise. For instance, if continue to use the Gaussian noise perturbation method for the weight-based information-theoretic bounds, we would need to characterize the stability property for the perturbed SGD, which might require techniques employed in [38]. Additionally, when combining stability notions with loss difference based CMI (or e- 1-CMI/f-CMM bounds, they cannot be unrolled using the chain rule and data processing inequality as the case of weight- based IOMI/CMI bounds, may not be possible to bound such CMI terms using trajectory-bases quantities. This raises doubts about the potential for obtaining more informative generalization bounds compared to the stability-based bounds themselves.
Generalization Bounds beyond Mutual Information the information-theone theoretic literature, is common to replace mutual information with alternative distributional measures, such as Wasserstein distance based bounds and total variation based bounds [50]. promising future direction is to incorporate the stability property of algorithms into these bounds, as demonstrated in this work. It is worth noting that obtaining KL divergence- based bounds should be straightforward since they rely on the same foundational Lemma 2.2 as discussed in this paper.
29


Reviews:
[{"review_id": "3ZIHEV3d71", "reviewer": "Reviewer_LJ4V", "summary": "This work improves information-theoretic generalization gap bounds by doing more careful derivations. As a result, the derived bounds get a multiplicative factors that capture some notions of hypothesis stability, while the existing bounds usually have a multiplicative constant factors that depend on the loss function and the data distribution. For example, the bound of Bu et al. [9] that states that $\\text{expected gen. gap} \\le \\frac{1}{n}\\sum_i \\sqrt{2 I(W;Z_i)}$, with $W$ being the output of the learning algorithm on the dataset $(Z_1,\\ldots,Z_n)$, gets improved by a multiplicative stability term $\\gamma_1 \\le 1$ that captures how much (in the worst-case) the loss on an example $z'$ can change when one replaces a single training example. Similar improvements are derived for conditional mutual information (CMI) bounds, both with standard CMI terms and a novel CMI term. In the random-subsample setting of Steinke and Zakynthinou [58], this novel CMI term measures the conditional mutual information between the $i$-th example that contributed to the training and the selection variable of the $i$-th pair, given two hypotheses: one corresponding to training with the first example of the $i$-th pair, the other corresponding to training with the second example, keeping everything else the same.\n\nThe authors show that in some known cases when existing information-theoretic bounds fail to vanish with $n$, adding the stability factor result in a vanishing bound. They also show that there are cases when the stability term alone does not vanish, but when the mutual information part is also considered, the resulting bound vanishes.", "questions": "- Line 153: \"are identically but not independent distributed\" --> \"are identically distributed but not independent\".\n- Line 169: It should be $\\gamma_2 \\le \\gamma_1$ and $\\gamma_4 \\le \\gamma_3$.\n- Line 219: there might be no uniform convergence if the conditional mutual information terms vanish with different rates for different $\\tilde{w}_i$ such that their supremum does not vanish.\n- Equation (14): $W_i$ and $\\bar{W}_i$ need to be defined in the Lemma statement.\n- Theorem 4.4: Is the assumption of $\\mathcal{A}$ being symmetric with respect to $S$ necessary?\n- Line 280: Are $\\hat{\\mu}$ and $\\hat{\\mu}^i$ treated like constants here?\n- Lines 713-720: If $E_0$ denotes the event of two elements of $\\tilde{Z}$ sharing the same non-zero coordinate, then we are interested in upper bounding $p(E_0)$ (in contrast to what is written in line 717). On line 720, \"the first inequality holds because when event $E_0$ occurs, one can determine the value of $U_i$ completely\", it should be when $E_0$ *does not* occur.", "limitations": "The limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Strength #1: Significance & Originality.** Information-theoretic generalization bounds have gained a lot of attention recently, partly because they are algorithm and distribution-dependent and some variants of them are nonvacuous in practical setting for deep learning. Recently, Haghifam et al [21] uncovered some limitations of information-theoretic bounds, constructing cases when the existing bounds do not even vanish with $n$. This submission addresses these limitations, and is therefore a significant contribution. While the derivations mostly follow common techniques, the resulting unification of stability-based and information-theoretic bounds is a novel (to my best knowledge) and significant contribution.\n\n**Strength #2: Quality.**  The submission is technically sound.  I have checked the proofs. The related work is cited adequately.\n\n**Strength #3:  Clarity & Presentation.** This work is well-written and presented. ", "weaknesses": "**Weakness #1: Limitations [minor].** As the authors acknowledge, the proposed stability terms are hard to evaluate or estimate in practical deep learning settings. Therefore, the extent of improvements outside of the considered synthetic settings is unclear. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive comments. Our responses follow.\n\n>- Line 219: there might be no uniform convergence if the conditional mutual information terms vanish with different rates for different $\\tilde{w}_i$ such that their supremum does not vanish.\n \n**Response.** We agree that if the CMI with the worst $\\tilde{w}_i$ does not vanish, the current bound cannot be considered a uniform convergence bound. We will provide clarification in the revised version.\n\n>- Theorem 4.4: Is the assumption of $\\mathcal{A}$ being symmetric with respect to $S$ necessary?\n\n**Response.** It is necessary in our proof of Theorem 4.4, otherwise we may not be able to obtain the inequalities in Line 678. \n\n\n>- Line 280: Are $\\hat{\\mu}$ and $\\hat{\\mu}^i$ treated like constants here?\n\n**Response.** In our main text, yes, we let $\\hat{\\mu}$ and $\\hat{\\mu}^i$ be fixed. However, it is worth noting that all these developments still apply when $\\hat{\\mu}$ and $\\hat{\\mu}^i$ are treated as random variables, namely $||\\eta t\\hat{\\mu}-\\eta t\\hat{\\mu}^i||=\\eta t||\\frac{1}{n}\\sum_{j=1}^n Z_j-\\frac{1}{n}(\\sum_{j\\neq i} Z_j+Z'_i)||=\\frac{\\eta t}{n}||Z_i-Z'_i||\\leq\\mathcal{O}({\\eta t}/{n})$.\n\n>- Lines 713-720: If $E_0$ denotes the event of two elements of $\\widetilde{Z}$ sharing the same non-zero coordinate, then we are interested in upper bounding $p(E_0)$ (in contrast to what is written in line 717). On line 720, \"the first inequality holds because when event $E_0$ occurs, one can determine the value of $U_i$ completely\", it should be when $E_0$ does not occur.\n\n**Response.** Thank you very much for your careful reading and pointing out these. There indeed exists a typo, $E_0$ should be the event that **no** pair of instances in $\\widetilde{Z}$ share the same non-zero coordinate. We apologize for this oversight, and we have corrected this in the revision.\n\n>- Line 153:...; Line 169:...;Equation (14):.... \n\n**Response.** Thanks for the suggestions, we have revised our paper according to your comments."}, {"review_id": "RJBKehnAcQ", "reviewer": "Reviewer_s7qd", "summary": "This paper studies how to develop information-theoretic generalization error bounds under the assumption that the algorithm is uniformly stable under a certain loss. Typically, these kinds of bounds are based on properties of the loss with respect to the data (e.g. subgaussian or bounded), while in this case they look at a property of both the loss and the algorithm, namely that it is uniformly stable. Considering these two together allows the bounds to decrease at faster rates as the stability parameter can decrease with the number of samples while the boundedness or subgaussianty parameter does not.\n\nIn particular, they extend the individual mutual information bounds from [9] to this assumption. They also consider a matrix of $n \\times 2$ hypotheses, where the first column is the output hypotheses, and each row in the second column is the hypothesis outputted by the algorithm when the $i$-th sample is swapped by another sample that is i.i.d. In a similar fashion to [58], they manage to develop conditional mutual information bounds, where they condition on this set of hypotheses. Intuitively, these bounds state the ability to determine if a sample was used to generate the real hypothesis or the hypothesis where the $i$-th sample was swapped. Then, they develop individual conditional mutual information bounds similar to [49] under this setup.\n\nAs an application, they showcase that while most of the current information-theoretic bounds fail in the example from [21], the presented bounds can succeed as they are stable with a stability parameter that decreases with the number of samples. Also, they make the case that for very stable algorithms (with constant in $\\Omega(1/\\sqrt{n})$) their bound improves upon [58]'s. Finally, they connect their results with the Berstein condition and further adapt the bounds from [63] to their setup.", "questions": "* Could you clarify further the definitions in Definition 2.1? Also, could you expand in the reasoning of why some parameters are larger/smaller than the others?\n\n* Could you give some examples where Theorems 3.3 and 4.3 improve upon the preceding theorems?\n\n* Could you try to find examples where neither uniform stability nor \"classical\" information-theoretic bounds can achieve the desired expected generalization error rate, but combined with the presented bounds can?\n\n* Could you find examples where the bounds with the conditioning can be employed to gain an understanding of some problem beyond the fact that it is stable?\n\nI am happy to increase my score if these questions are satisfactorily answered. I like the paper, but it seems a little immature at the moment. Once these things are addressed, it could become a better paper that ticks many boxes and deals with important questions.", "limitations": "Some limitations are included throughout the text and in the Appendices. Some others, like the ones included in the weaknesses, are not. Regarding potential negative societal impact, as the work is of a theoretical, fundamental nature, I do not forsee any issue with not addressing them.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strength of the paper is the link between stability and the current individual mutual information bounds, which in turn develop into mutual information bounds for stable algorithms with an explicit appearance of the stability parameter. This is good because we know the stability parameter in certain algorithms and this mutual information can be bounded in a good amount of settings.\n\nSimilarly, the connection with the Berstein condition is also important as we also know that a good amount of algorithms under certain losses respect this condition, and therefore these bounds can be applied.\n\nFinally, as in [63], the paper showcases how a simple trick like the Donsker-Varadhan together with a careful choice of the function in the supremum can lead to good bounds. ", "weaknesses": "While the results are interesting, I fail to understand exactly how results other than those in Section 3 are useful. Yes, they can achieve the rates desired in Example 1, but this comes basically because the individual CMI is <= 1 and one can directly have the bound expected generalization error <= $\\beta$, which is already known from the uniform stability literature. \n\n* A suggestion for improvement, is to state that directly as a stronger result. Since the individual CMI <= 1, then one may recover the bounds from the uniform stability literature. Then, every result that we know from there is also subsumed into this setup. This way, one can see more clearly that Example 1 works immediately from uniform stability and many other results as well. I believe that this way you may be able to comment on larger sets of problems.\n\nI think that they definitely have potential, but this potential is not explained in the paper. The main reason I mention this is that the previous bounds were dealing with a KL divergence between terms that were \"easily tractable\" like the posterior $P\\_{W|S}$ (or $P\\_{W|Z\\_i}$) and a prior $Q\\_{W}$. However, the terms appearing in these bounds seem very difficult to treat and characterize. \n\n* Another suggestion would be to find an example where this mutual information can be calculated so the reader can have an idea of why this is interesting other than the fact that it can be reduced to the known bound from uniform stability as I mentioned above. An interesting example would include a setup where neither uniform stability nor \"classical\" information-theoretic bounds can achieve the desired expected generalization error rate, but combined with the presented bounds can.\n\nSome results and statements are included but are loosely explained.\n\n* Theorems 3.3 and 4.3 are examples of this. When can this be useful? Could you provide us with a particular example where we can see how are we benefitting from including $\\gamma\\_2^2 / \\gamma_1$ or $\\Lambda(\\Tilde{W}\\_i)$?\n* In Remark 2.2, what do you mean with \"it is expected that $\\gamma\\_4$ is larger than $\\gamma\\_2$ due to the independence of $Z'$ in Eq. (7)\"? A similar claim is done in lines 217-218. When doing this kind of remarks and statements, it would be important to justify more the reasoning behind them.\n* The whole set of definitions in Definiton 2.1 is not very clear to me. It would be nice to have a larger, more comprehensive commentary on what are these definitions and why are they included / why are they useful.\n\nSmall corrections in the literature, not very important:\n\n* In line 31, it should be \"decay at a faster rate, e.g. $\\mathcal{O}(1/n)$\".\n* In line 33, \"demonstrating tightness in non-convex learning cases such as deep learning\" may be misleading. Some people may interpret that this means that they achieve the correct rate and therefore are tight. An alternative writing could be \"demonstrating a good characterization in some instances of non-convex settings (e.g. deep learning)\".\n* In line 198, $R$-conditioned information-theoretic bounds also appeared in [49].\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments, and we appreciate your positive feedback on our paper. Our responses follow.\n\n>- Could you clarify further the definitions in Definition 2.1? Also, could you expand in the reasoning of why some parameters are larger/smaller than the others?\n\n**Response.** Due to the character constraints in the separate rebuttal, we have made the decision to relocate the response to this question to the global response. Kindly refer to the global response provided above; we sincerely apologize for any inconvenience caused.\n\n>- Could you give some examples where Theorems 3.3 and 4.3 improve upon the preceding theorems?\n\n**Response.** Consider the loss function is bounded by $C$, if an algorithm $\\mathcal{A}$ is $\\gamma_2$-SCH-B stable but not $\\gamma_1$-SCH-A stable (note that if $\\mathcal{A}$ is $\\gamma_1$-SCH-A stable, it is also $\\gamma_1$-SCH-B stable since $\\gamma_2\\leq\\gamma_1$),  we replace $\\gamma_1$ by $C$ in Theorem 3.1 and Theorem 3.3, since  $I(\\widetilde{W}^+;\\widetilde{Z}^+\\_{i})$ decays faster than $\\sqrt{I(\\widetilde{W}^+;\\widetilde{Z}^+\\_{i})}$, as long as $\\gamma_2^2$ also decays faster than $\\sqrt{I(\\widetilde{W}^+;\\widetilde{Z}^+\\_{i})}$, Theorem 3.3 is tighter than Theorem 3.1. Similar arguments also apply to comparing Theorem 4.1 and Theorem 4.3 when the $\\mathcal{A}$ is $\\gamma_4$-SCH-D stable but not $\\gamma_3$-SCH-C stable.\n\nIn addition, the $\\gamma_2$-SCH-B stablility condition in Theorem 3.3 gives us a chance to connect to the Bernstein condition as shown in Corollary 6.1. Thus, if the Bernstein condition is satisfied but $\\mathcal{A}$ is not $\\gamma_1$-SCH-A stable, Theorem 3.3 will be tighter. \nAs a simple concrete example: Let $\\mathcal{W}$ be finite, i.e. $|\\mathcal{W}|=K$. To simplify the setting, assume $L_\\mu(w^*)=0$ and let $\\mathcal{A}$ be an interpolating algorithm. In this case, for Theorem 3.1, we have $\\mathcal{E}\\_\\mu(\\mathcal{A})\\leq\\frac{\\sqrt{2}C}{n}\\sum_{i=1}^n\\sqrt{I(W;Z_i)}\\leq\\sqrt{2}C\\sqrt{\\frac{I(W;S)}{n}}\\leq\\mathcal{O}(\\log{K}/\\sqrt{n})$. For Theorem 3.3 or Corollary 6.1, we have $\\mathcal{E}\\_\\mu(\\mathcal{A})\\leq\\mathcal{O}(\\sum_{i=1}^n\\frac{I(W;Z_i)}{n})\\leq\\mathcal{O}(\\frac{I(W;S)}{n})\\leq\\mathcal{O}(\\frac{\\log{K}}{n})$. Then clearly, Theorem 3.3 is tighter than Theorem 3.1. We will try to construct more examples in the revision.\n\n>- Could you try to find examples where neither uniform stability nor \"classical\" information-theoretic bounds can achieve the desired expected generalization error rate, but combined with the presented bounds can?\n\n**Response.** This is a good question and we did try to find such examples, but this appears difficult.\n\nThe reason is that if the algorithm satisfies certain stability assumptions, stability-based bounds often achieve the optimal rate, particularly after the work of [1,2]. Specifically, in the context of SCO with SGD/GD, where the loss function is either smooth or nonsmooth, stability-based bounds can attain tight upper and lower bounds [3,4]. However, in the context of nonconvex learning, the challenge lies in obtaining the exact generalization error rate and the decaying rate of both the stability parameter and the MI/CMI term.\n\nWe note that we have demonstrated that information-theoretic quantities can enhance the stability analysis, as exemplified in Example 2 and Section F.3. Thus, the advantages of our bounds are beyond mere reduction to uniform stability bounds.\nMoreover, when our interest extends beyond the decaying rate w.r.t. $n$, there is a potential case that our bounds improve both stability-based bounds and the MI/CMI bounds before. For instance, in Figure 3(d) in [5], previous MI/CMI bounds are loose at the early phase of training. As the stability parameter usually grows with the iteration number, it remains small at the beginning, and its large value at the end can be mitigated by the MI/CMI term. Consequently, the product of the stability parameter and the MI/CMI term could potentially offer a more accurate reflection of the dynamics of the true generalization error. The  remaining challenge is to rigorously characterze SCH parameters for SGD (or SGLD), as previous analyses were mainly limited to characterizing $\\beta_1$.\n\n>- Could you find examples where the bounds with the conditioning can be employed to gain an understanding of some problem beyond the fact that it is stable?\n\n**Response.** We note that CMI bounds have already been widely discussed in the existing literature where the algorithm is not necessarily stable. In Section F.3, we establish a connection between our newly introduced CMI terms and the classical VC theory, building upon previous research. Consequently, for learning problems with finite VC dimension, the CMI terms themselves can serve as a means to demonstrate the learnability of such problems.\n\n>- A suggestion for improvement, ...\n>- Another suggestion would be ...\n>- Small corrections in the literature, ...\n\nThank you again for providing these valuable suggestions for improving our paper, we will revise our paper according to your comments.\n\nPlease do let us know if you have further questions.\n\n[1] Yegor Klochkov, et. al. Stability and Deviation Optimal Risk Bounds with Convergence Rate O(1/n). NeurIPS 2021.\n\n[2] Olivier Bousquet, et. al. Sharper bounds for uniformly stable algorithms. COLT 2020.\n\n[3] Moritz Hardt, et. al. Train faster, generalize better: Stability of stochastic gradient descent. ICML 2016.\n\n[4] Raef Bassily, et al. Stability of stochastic gradient descent on nonsmooth convex losses. NeurIPS 2020.\n\n[5] Ziqiao Wang, et al. Tighter information-theoretic generalization bounds from supersamples. ICML 2023."}, {"review_id": "SCLart0iXC", "reviewer": "Reviewer_Ptk3", "summary": "The paper proposes several new stability assumptions named the sample-conditioned hypothesis (SCH) stability. Based on these notions, the authors present new IOMI and CMI bounds to address the limitations of existing information-theoretic bounds in the context of stochastic convex optimization (SCO) problems.", "questions": "Please see the above section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper adopts different assumptions for bounding the CGF, which can improve existing information-theoretic bounds, especially in the context of SCO.\n- The connection between Bernstein and the proposed SCH-B stability is interesting.\n- The paper is generally well-written.", "weaknesses": "- It seems to me the most significant contribution is to introduce uniform stability for bounding the CGF, which can solve the counter-example proposed in [1]. However, it's hard to estimate the order of the stability parameter without any knowledge of the distribution. Especially in practice, such parameters do not affect the algorithm design. \n\n- The effectiveness of the proposed SCH stabilities is unclear. It's mentioned that all the stability parameters $\\gamma_{1:4}$ are smaller than $\\beta_2$. From Theorem 4.3, we have $\\|\\mathcal{E}\\_{\\mu}(A)\\| \\leq \\frac{\\beta_2}{n} \\sum_{i=1}^n I(\\hat{Z_i};U_i|\\tilde{W_i}) + 0.72 \\beta_2$. Similar to the discussion in Example 2, the first term can vanish. However, the second term still exists since $\\beta_2$ is a constant. To prove the effectiveness of the SCH stability bounds, it needs to identify the order of $\\gamma_4^2$.  \n\n- Example 1 requires the data and parameter dimension $d=2n^2$ to ***grow with $n$*** for proving the $\\Omega(1)$ lower bound of the information quantities. Does this really make sense?  Even though the uniform stability is much tighter in this case, I do not think this counter-example is representative enough. Could you conceive another counter-example that is more convincing?\n\n- The SCH-B and SCH-D stability are distribution dependent, the corresponding stability parameter cannot be estimated, and these notions can be confused with other distribution assumptions.\n- The SCH-A and SCH-C stability are distribution-free but related to the distribution (or set) of the output model parameters. It’s unclear to me how to identify the stability parameter effectively for different algorithms.\n\n>[1] Mahdi Haghifam, Borja Rodríguez-Gálvez, Ragnar Thobaben, Mikael Skoglund, Daniel M Roy, and Gintare Karolina Dziugaite. Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization. In International Conference on Algorithmic Learning Theory, pages 663–706. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you sincerely for your comments to our paper. Our responses follow.\n\n>- It seems to me the most significant contribution ...\n\n**Response.** Accurately estimating the order of the stability parameter is challenging in all stability-based bounds. However it is still possible to bound the parameter, without access to the underlying distribution, and obtain insights to guide the design of learning algorithms. As shown in Section F.2, without the knowledge of the data distribution, it is possible to explicitly bound the stability parameter, and the bound inspires Tikhonov regularization, which finds smoother solutions for better generalization.\n\nIn addition to bounding the stability parameter, one may also have the option of estimating the parameter via influence function [1]. By employing such estimation, the appearance of the Hessian matrix of the solution can also offer valuable insights for algorithm design. \n\nWe would like to re-emphasize that our primary focus is to demonstrate the existence of information-theoretic bounds capable of explaining the learnability of problems with certain stability properties, such as the SCO problems discussed in Section 5, and resolving the limitations shown in [2]. While it is not our primary goal, we agree that providing new algorithmic insights is indeed an important dimension for studying generalization. Notably novel generalization bounds may lead to the discovery of new algorithmic properties that improve generalization for the problem or algorithm is specialized to certain forms; for example, the bounds of [3] specialized to SGD leads to new insights in understanding and improving SGD in [4]. It is curious to explore this aspect for the bounds developed in this paper. \n\n>- The effectiveness of ....\n\n**Response.** If both $\\beta_2$ and $\\gamma_4$ are constants, then the algorithm is neither $\\beta_2$-uniform stable nor $\\gamma_4$-SCH-D stable. In this case, Theorem 4.3 is not of any interest for algorithms that are not stable in these senses. \n\nIf $\\beta_2$ is a constant, i.e., the algorithm is not $\\beta_2$-stable. In this case, there is still an opportunity that the algorithm is $\\gamma_4$-SCH-D stable and the second term vanishes at least at the speed of $\\gamma_4$. Then we agree that identifying the order of $\\gamma_4$ is needed for the bound to be useful. Similar to other distribution-dependent stability notions, determining the order of $\\gamma^2_4$ is challenging. Consequently, uniform stability remains the most extensively studied and well-understood concept in the learning theory community. However, we would like to note that our SCH parameters offer an extension of the bounds' applicability, such as enabling connections with the Bernstein condition.\n\n>- Example 1 requires the data ...\n\n**Response.**  Example 1 was constructed in [2, Theorem 17]. The objective of this exercise is to create a \"bad\" learning problem for which information-theoretic bounds demonstrate certain limitations. Such an exercise does not require the created \"bad problem\" to be representative of the reality. In this paper, we show that even for such \"bad problems\", the presented bounds in this paper overcome these limitations. \n\nRegarding the choice of $d=2n^2$, we remark that the dimension $d$ is usually related to the model capacity, and many generalization bounds have the rate of $\\mathcal{O}((\\frac{d}{n})^{\\alpha})$ (for some $\\alpha>0$). Thus, we do care about how $d$ changes affect the generalization.  An important feature of gradient descent for SCO problems is that the sample complexity is dimension-independent. Thus, for any generalization bounds that explicitly depend on the model dimension, there may always exist a setting where they don't diminish, e.g., $d=2n^2$ or $d=3T2^n/4$ for MI/CMI bounds in [2].  In fact, this setting does make sense because we hope the generalization bounds to explain the learnability of a class of CLB problems here instead of only a specific setting with a fixed $d$. If $d$ is treated as a constant, then there would be no such limitations since the dimension-dependent property of MI/CMI bounds would not be considered. Thus, our key message here is  that when $d=2n^2$ in this CLB problem, the exact generalization error of gradient descent does vanish with $n$ increases, but  the previous information-theoretic bounds are unable to establish the learnability of this learning problem. In this sense, this example stands as a representative case.\n\nAdditionally, here we focus on the regime of overparameterization where $d>n$, which holds true for the most deep neural networks. Also allowing the model dimension grows with $n$ aligns with practical scenarios. For example, achieving good performance on CIFAR-10 and ImageNet often necessitates employing a larger model for the latter.\n\n>- The SCH-B and ....\n>- The SCH-A and ....\n\n**Response.** We acknowledge that these SCH stability parameters are hard to estimate in general, which we also discussed in Section G. But this limitation, which also exists in many other bounds, does not negate the potential usefulness of these bounds, as we already discussed above. In addition, as we emphasize before, uniform stability is still the only one that has been widely studied and well understood in the learning theory community. Further development of the practical usage of stability itself may eventually also help to overcome the estimation difficulty of our bounds.\n\n[1] Pang Wei Koh, et. al. Understanding black-box predictions via influence functions. ICML 2017.\n\n[2] Mahdi Haghifam, et. al. Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization. ALT 2023.\n\n[3] Aolin Xu, et. al. Information-theoretic analysis of generalization capability of learning algorithms. NeurIPS 2017.\n\n[4] Ziqiao Wang, et. al. On the generalization of models trained with SGD: Information-theoretic bounds and implications. ICLR 2022."}, {"review_id": "7UNsFY6zQB", "reviewer": "Reviewer_uPAq", "summary": "Information-theoretic generalization bounds offer a new approach in generalization theory by providing complexity measures that depend on the data distribution and learning algorithm itself. In the past years, it has been observed that information-theoretic generalization bounds are not compatible with the classical approaches to prove generalization such as vc theory or uniform stability. \n\nIn this paper the authors propose several new information theoretic quantities to study generalization. The proposed bounds are interesting as they are compatible with several notions of uniform stability. \n", "questions": "1- weakly uniform stability versus strong uniform stability: Consider the seminal work of Hardt et al paper in which they derive stability bound for SGD under the notion of weakly uniform stability. Why in the paper do authors only consider the strong notion of stability?\n\n2- text after corollary 3.1 seems unclear to me. If the algorithm is deterministic then the mutual information term will blow up. It is not clear to me how this may result in improvement.\n\n3- Theorem 3.3: Let's assume that the algorithm is deterministic and also permutation invariant in the sense that the order of the training points does not affect the output. Then, this results states that  gen.gap <= gamma_1  (  1  + mutual information term). It shows that even if the mutual information term goes to zero, the convergence rate is determined by gamma_1.\n\n\n4- For Example 1, what is the exact generalization error? It should be of order O(1/n), however, the obtained bound using the proposed approach has an order of 1/sqrt(n).\n\nTypos:\n\nLemma A.5. Bernstein.\n", "limitations": "The main limitations are explained in Items 4 and 5 of weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I think the paper addresses an important limitation of information-theoretic generalization that is incompatibility with the uniform stability framework. They also provide various examples to show the expressiveness of their bounds.\n\n", "weaknesses": "1– The presentation of the paper needs significant improvement. Specifically the part on defining the additional structure is extremely vague. For instance, there are many variables that are not clearly defined. For instance, what is the tilde{W}_{i,0} and tilde{W}_{i,1}.\n\n2– Compared to CMI bound, the intuition behind the results is not clear. For instance CMI is closely related to membership inference. \n\n3– Some of the results in the paper seem redundant. For instance, theorem 3.1 has been proved by work by Bu et al under the name of individual sample bounds.\n\n\n4– The major drawback of the bounds in the paper is the following: Uniform stability bounds for randomized algorithms are very important. For instance, in the seminal work by Hardt et al, the fact that the algorithm is randomized is important. The other example is the algorithm by Feldman and Dagan for leaning VC classes with stable algorithms. However, to me all the bounds in the paper rely on a very strong notion of stability which only holds for “deterministic algorithms. It is not clear what is the roadblock for extending the results for randomized algorithms.\n\n\nFeldman, Dagan. PAC learning with stable and private predictions\n\n\n5– The main limitation of uniform stability bounds is that it considers the “worst-case” data distribution. However, a learning algorithm may be more stable for “easier” distributions. However, all the bounds in the paper already depend on the worst-case stability parameter. I think this leads to suboptimal bounds.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. Our responses follow.\n\n>- The presentation of the paper needs significant improvement. Specifically the part on defining the additional structure is extremely vague. For instance, there are many variables that are not clearly defined. For instance, what is the tilde{W}{i,0} and tilde{W}{i,1}.\n\n**Response.** We apologize for lacking clarity at places and will make our best effort in the revision. Specifically, $\\widetilde{W}\\_{i,0}$ and $\\widetilde{W}\\_{i,1}$ represents the $i$-th row, the first column and the second column of the matrix $\\widetilde{W}$, respectively. \n\n>- Compared to CMI bound, the intuition behind the results is not clear. For instance CMI is closely related to membership inference.\n\n**Response.** We have tried to explain the intuition of $I(\\widehat{Z}_i;U_i|\\widetilde{W}_i)$ in Line 212-217, and here we restate: the new CMI terms in our paper are also closely related to the membership inference. Specifically, $I(\\widehat{Z}_i;U_i|\\widetilde{W}_i)$ measures the ability to decide if an instance $\\widehat{Z}_i$ contributes to the training of $\\widetilde{W}^+$ or  to the training of $\\widetilde{W}^+_i$ when we know it contributes to only one of them. \n\n>- Some of the results in the paper seem redundant. For instance, theorem 3.1 has been proved by work by Bu et al ....\n\n**Response.** Agreeably our Theorem 3.1 has a very similar form to the bound of Bu et al. However, the two bounds differ significantly due to the replacement of sub-gaussian variance proxy with stability parameter. This difference is critical since   the bound of Bu et al. has been shown inadequate in explaining the learnability of certain SCO problems [1], as illustrated by Example 1 (please refer to Section E for an explanation) in our paper. However, our Theorem 3.1 overcomes this limitation. For further comparisons between the two bounds, please see Line 191-197 in our paper.\n\n>- The major drawback of the bounds in the paper is the following: Uniform stability bounds for randomized algorithms are very important. .... However, to me all the bounds in the paper rely on a very strong notion of stability which only holds for “deterministic algorithms....\n>- weakly uniform stability versus strong uniform stability: .... Why in the paper do authors only consider the strong notion of stability?\n\n**Response.** On the one hand, our four SCH stability notions in different bounds relax the strong notion of uniform stability, taking into account the randomness of the algorithm, particularly for $\\gamma_2$, $\\gamma_3$, and $\\gamma_4$.\n\nOn the other hand, even for the strong uniform stability notion, such as the $\\beta_2$-based bounds, they remain applicable to randomized algorithms. This is because the MI and CMI terms in these bounds are algorithm-dependent, capturing the randomness in the algorithm. Notably, $\\beta_2$-uniform stability is still a weaker assumption than the bounded loss assumption.\n\nOne important reason that we apply strong uniform stability notion here is because information-theoretic bounds are sometimes vacuous in the deterministic setting, so incorporating $\\beta_2$ can significantly sharpens the information-theoretic bounds in this setting, as shown in the examples in this paper. Moreover, it's worth noting that replacing $\\beta_2$ with the weak notion $\\beta_1$ in our bounds may not be feasible.\n\n>- The main limitation of uniform stability bounds is that it considers the “worst-case” data distribution ....\n\n**Response.** We note that the $\\gamma_2$-SCH-B stability and the $\\gamma_4$-SCH-D stability are distribution-dependent notions. They are not defined from the worst-case data distribution. Additionally, if the loss function is upper-bounded by $C$, as commonly employed in proving the optimal rate of the stability-based bound, all the worst-case stability parameters can be replaced by $C$, allowing us to focus solely on the distribution-dependent notions. Remarkably, we achieve the known fastest rate of the second moment generalization error bound, as demonstrated in Theorem 4.4.\n \n>- text after corollary 3.1 seems unclear to me....\n\n**Response.** For deterministic algorithms, the individual mutual information term $I(W;Z_i)$ may not necessarily blow up. Simple examples illustrating this can be found in Example A and B in Bu et al.'s work. In fact, the primary motivation behind Bu et al.'s proposal for the individual bound is to mitigate the problem of blowing up in the sample-wise mutual information ($I(W;S)$) under the deterministic setting.\n\n>- Theorem 3.3: Let's assume that the algorithm is deterministic and also permutation invariant ....\n\n**Response.** We completely agree. While Theorem 3.3 exhibits a faster decay rate compared to Theorem 3.1 due to the removal of the square-root function, it still does not surpass $\\gamma_1$ in terms of rate. We must note that our work also demonstrates the enhancement of the stability-based framework through information-theoretic analysis, as highlighted in Example 2.\n\n>- For Example 1, what is the exact generalization error? It should be of order O(1/n)...\n\n**Response.** The exact generalization error in Example 1 decays with the rate of $\\mathcal{O}(1/\\sqrt{n})$, as given in [1, Theorem 17].\n\nIn fact, for GD with nonsmooth convex loss, such as in Example 1, [2, Theorem 3.2] gives a tight generalization lower bound: $\\mathcal{O}(\\eta \\sqrt{T}+\\frac{\\eta T}{n})$. By substituting $\\eta=\\frac{1}{n\\sqrt{n}}$ and $T=n^2$ from Example 1, we can deduce the lower bound of $\\mathcal{O}(1/\\sqrt{n})$ as well. Combining the upper bound obtained in our paper, we conclude that the exact generalization error has the order of $\\mathcal{O}(1/\\sqrt{n})$.\n\n[1] Mahdi Haghifam, et. al. Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization. ALT 2023.\n\n[2] Raef Bassily, et al. Stability of stochastic gradient descent on nonsmooth convex losses. NeurIPS 2020."}]

Author rebuttal:
# To all reviewers, particularly to Reviewer s7qd:

>- Reviewer s7qd has pointed out that our Definition 2.1 lacks clarity. We would like to address this concern by offering the following commentary:

We first note that the reason we introduce SCH stabilities in Definition 2.1 is that solely using $\beta_2$ (as mentioned in Line 132) in our bounds might be too loose for the randomized setting (but it is still weaker than the bounded loss assumption), as it considers the supremum over all sources of randomness. By incorporating SCH stabilities, we aim to demonstrate that theoretically, we can achieve significantly tighter stability parameters.

The basic set up is as follows. Assume a random sample $S$ gives rise to $W$. For each $Z_i\in S$, we construct $S^i$ by replacing $Z_i$ with another independently drawn instance; call the training result $W^i$, the neighbor of $W$.  

In (a), $\gamma_1$-SCH-A stability measures the difference between the loss of $w$ and the expected loss of its neighbor $W^i$ at a worst $z$ and the worst possible $w$. While in (b), $\gamma_2$-SCH-B stability measures the square of this difference, not in the worst case, but in an average case, where the average is over an independently $Z'$ for the loss evaluation, the training sample, and the algorithm randomness. Since "average is smaller than worst", $\gamma_2\leq\gamma_1$ can be rigorously proved.

In (c),  we consider the difference between the loss of $W$ and the loss of its neighbor when evaluated at the worst possible $z_i$ that when included in $S$ gives rise to $W$. The expected value of this difference is $\gamma_3$-SCH-C stability. While in (d), $\gamma_4$-SCH-D stability measures the expected squared difference between the loss of $W$ and the loss of its neighbor when evaluated at $Z_i$ (a member of $S$). For a similar "average smaller worst" reason, one expects that $\gamma_4\leq\gamma_3$. However, this result can not be rigourously proved. We will revise this in Remark 2.2. Note that this relationship has never been used in this work.

We expect that $\gamma_2$, $\gamma_3$, and $\gamma_4$ are all smaller than $\beta_1$. This is because in $\beta_1$, we consider the worst evaluated instance, whereas in the other cases, we take the expectation over all instances. In Line 217-218, we also expect that ${\mathbb{E}\_{\widetilde{W}_i}{\Delta_1(\widetilde{W}_i)^2}}\leq\beta^2_1$, similarly, this is because $\beta_1$-stability holds for all the possible $s$ and $s^i$, namely it holds for all the $(w,w^i)$ pair (that shares the same randomness) while in ${\mathbb{E}\_{\widetilde{W}_i}{\Delta_1(\widetilde{W}_i)^2}}$, we take the expectation of these pairs. 

We expect $\gamma_2\leq\gamma_4$ due to the following reason: first by Jensen's inequality, we have $\mathbb{E}\_{S,R,Z'}{\left[\ell(W,Z')-\mathbb{E}\_{W^i|W}{\ell(W^i,Z')}\right]^2}\leq\mathbb{E}\_{W,W^i,Z'}{\left[\ell(W,Z')-{\ell(W^i,Z')}\right]^2}$, then since $Z'$ is independent of both $W$ and $W'$, $Z'$ can be regarded as a testing data point for both $W$ and $W'$, we could expect that the expectation of $\ell(W,Z')-{\ell(W^i,Z')}$ is small. While in $\mathbb{E}\_{S,Z'_i,R}{\left[\ell(W,Z_i)-\ell(W^i,Z_i)\right]^2}$, $Z_i$ is a training data point for obtaining $W$, so $\ell(W,Z_i)$ could be small in general, and $Z_i$ is a testing point for $W^i$. Therefore, it is reasonable to expect that $\mathbb{E}\_{W,W^i,Z'}{\left[\ell(W,Z')-{\ell(W^i,Z')}\right]^2}\leq\mathbb{E}\_{S,Z'_i,R}{\left[\ell(W,Z_i)-\ell(W^i,Z_i)\right]^2}$, namely $\gamma_2\leq\gamma_4$.

We will include these explainations in the revision.

Meta-review: The paper introduced a novel generalization bound of the form $\textrm{stability} \cdot \textrm{information}$ which can potentially improve over both stability and information-theoretic generalization bounds.

- The motivation of the paper, to remedy the lower bound in  Haghifam et al. was not found to be convincing. In particular, it doesn't seem that the above bound can be dimension dependent if any of the bounds (stability or MI) are dimension dependent.

- Nevertheless the authors could demonstrate a proof of concept where the above bound can be superior to both the stability bound and the information bound. It is true that this can happen only under certain distributional assumptions (as opposed to pure algorithmic assumptions) but this is the best that can be done given L'22 lower bound.

- The AC as well as reviewer strongly recommend to rewrite the introduction and reframe their result in the right light. It seems the authors acknowledged this need.
Decision: Accept (poster)