SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziqiao Wang
zwang286@uottawa.ca
Yongyi Mao
ymao@uottawa.ca
We present new information-theoretic generalization guarantees through the a novel construction of the "neighboring-hypothesis" matrix and a new family of stability notions termed sample-conditioned hypothesis (SCH) stability. Our approach yields sharper bounds that improve upon previous information-theoretic bounds in various learning scenarios. Notably, these bounds address the limitations of existing information-theoretic bounds in the context of stochastic convex optimization (SCO) problems, as explored in the recent work by Haghifam et al. (2023).
Information-theoretic upper bounds for generalization error have recently been developed since the seminal works of [52, 67]. On the one hand, these bounds have attracted increasing interest due to their distribution dependence and algorithm dependence, making them highly suitable for reasoning the generalization behavior of modern deep learning. In fact, subsequent studies have demonstrated that information-theoretic bounds can effectively track the dynamics of generalization error in deep neural networks [40, 60, 23, 61, 62, 25, 64]. On the other hand, recent studies [58, 19, 23, 20, 25] have revealed the expressive nature of information-theoretic bounds in the distribution-free setting. Notably, the conditional mutual information (CMI) framework proposed by [58] has shown great promise by establishing connections to VC theory [59] and matching minimax rates for the binary classification [19, 20]. Additionally, in the case of the  $0 - 1$  loss and the realizable setting, where an interpolating algorithm attains zero empirical risk, information-theoretic bounds give an exact characterization of the generalization error [20, 64], thereby providing the tightest possible generalization bound.
Nonetheless, information-theoretic bounds have been extensively discussed due to their two main deficiencies. The first deficiency concerns the unbounded nature of the original input-output mutual information (IOMI) bounds [67, 4, 32, 31]. To address this issue, several techniques have been developed, including the chaining method [2], the individual technique [9] or random subset technique [40], and the Gaussian noise perturbation technique [41]. Notably, the CMI bound [58] stands out as it has a finite upper bound for any learning scenario due to its supersample construction. These techniques are now often applied jointly for analyzing generalization [18, 49, 70, 63, 23, 25, 64]. The second deficiency concerns the sub-optimal convergence rate of information-theoretic bounds. Specifically, when the information-theoretic quantities, whether IOMI or CMI, are bounded by constants, the bounds exhibit a decaying rate in the order of  $\mathcal{O}(1 / \sqrt{n})$ , where  $n$  is the sample size. In contrast, generalization errors in practice may decay at a faster rate, e.g.,  $\mathcal{O}(1 / n)$ . To address this limitation, several works, inspired by some PAC-Bayesian literature, have proposed fast-rate information-theoretic bounds [24, 25, 64], demonstrating a good characterization in some instances of non-convex settings such as deep learning. More recently, [65, 66, 71] present IOMI bounds for the Gaussian mean estimation problem, that achieve optimal convergence rates, contrasting previous bounds [9, 70].
The issue of slow convergence in information-theoretic bounds has recently been amplified by the observation that these bounds may not even vanish. [21] highlight this limitation in the context of stochastic convex optimization (SCO) problems [55]. Specifically, [21] shows that all existing information-theoretic bounds, including the CMI bound [58], the Gaussian noise perturbed IOMI bound [41], the individual IOMI bound [9], the evaluated CMI (e-CMI) bound [58] and the functional CMI ( $f$ -CMI) bound [23], fail to vanish in at least one of the counterexamples they constructed. These failures stem from the dimension-dependent nature of current information-theoretic quantities [31], appearing an intrinsic barrier for overcoming these limitations. However, if we dissect the process by which these bounds are derived, opportunities do exist. Specifically, recall that all these information-theoretic bounds are built upon the Donsker-Varadhan (DV) variational representation of the KL divergence [44, Theorem 3.5] (see Lemma A.2 in the Appendix). Using this representation, a generalization upper bound is derived in terms of information-theoretic quantity and a cumulant-generating function (CGF), as illustrated below.
$$
\text {G e n e r a l i z a t i o n E r r o r} \leq \inf  _ {\text {P a r a .} > 0} \frac {\text {I O M I o r C M I} + \text {C G F}}{\text {P a r a .}}.
$$
Particularly note that the CGF depends on certain choice of the auxiliary function in the DV formula. Then, except for the IOMI or CMI itself, the tightness of the generalization bound hinges on two key factors: the selection of the DV auxiliary function and the approach used to bound the CGF, the latter of which often requires additional assumptions specific to the chosen DV auxiliary function. The most common choices for the DV auxiliary function involve making assumptions such as sub-Gaussian loss or bounded loss. For instance, in the case of IOMI bounds, the DV auxiliary function is typically chosen as the single loss or average loss, and the sub-Gaussian assumption (or boundedness assumption) is utilized. In CMI bounds, the DV auxiliary function is defined as the difference in loss between a training sample and a "ghost" sample, and the boundedness property is employed. We now note that exploiting bounded loss is the fundamental reason behind the failures in SCO problems. Although [21] does not explicitly rely on boundedness, they do make use of the product of the Lipschitz constant and the diameter of the hypothesis domain, which essentially serves as an upper bound for the loss. The product does not vanish with  $n$ , thereby neutralizing the potential to include another decaying factor in the bound. Arguably, at least for these SCO problems, the selection of the DV auxiliary function may not be optimal, for example, the resulting  $\mathrm{CGF} = \mathcal{O}(1 / n)$ , giving vacuous generalization guarantees. This analysis inspires us to explore alternative DV auxiliary function and to adopt different assumptions for bounding the CGF. For instance, we may devise appropriate stability assumptions and create opportunities to upper bound the CGF using a term in  $\mathcal{O}(\beta^2 /n)$  where  $\beta$  is the stability parameter that decays as  $n$ . This is promising in light of the capability of the stability-based framework in explaining the generalization of SCO problems [6, 56, 22, 5].
In this paper, we combine information-theoretic analysis with stability notions to develop IOMI bounds and CMI bounds that improve upon previous information-theoretic bounds for stable learning algorithms, achieving faster convergence rates. The main contributions of our paper are summarized below.
- We introduce our new notions of algorithmic stability, referred to as sample-conditioned hypothesis (SCH) stability. We also present a novel construction of a sample-dependent hypothesis matrix, where each column is a neighborhood pair of hypotheses obtained from two training samples that differ in only one element, inspired by the supersample setting of CMI [58].
- We present new IOMI bounds, which explicitly include the SCH stability parameters and are shown superior to previous bounds for stable learning algorithms.
- We show that the sample-dependent hypothesis matrix, similar to the supersample matrix, enjoys a symmetry property. Exploiting this symmetry, we establish novel CMI bounds. Specifically, we present hypotheses-conditioned CMI bounds that are analogous to the previous supersample-conditioned CMI bounds. Additionally, we derive sample-conditioned CMI bounds exploiting other assumptions. Notably, these bounds introduce novel CMI quantities and include SCH stability parameters. In particular, the new CMI quantities remain boundedness as the original CMI. Consequently, these CMI bounds vanish no slower than their stability parameters. In addition, we also obtain a second-moment generalization bound that matches the tightest known bound in the literature [14] under the same condition.
- We apply our new bounds to a convex-Lipschitz-bounded (CLB) example in which previous information-theoretic bounds fail to explain generalization [21], and show that the new IOMI
and CMI bounds vanish, benefiting from SCH stability. Additionally, we discuss another CLB example where the uniform stability parameter is non-vanishing or has a slow convergence rate, but our information-theoretic bounds remain tight up to a constant.
- We extend our analysis to derive information-theoretic generalization bounds based on Bernstein condition, which leverages a connection between stability and Bernstein condition. We also show that stability can be incorporated into generalization bounds to obtain stronger results using alternative information-theoretic quantities such as the loss difference based CMI, e-CMI and  $f$ -CMI. Furthermore, we illustrate the expressiveness of our new CMI notions under the distribution-free setting.
Probability and Information Theory Notation Unless otherwise noted, a random variable will be denoted by a capitalized letter, and its realization by the corresponding lower-case letter. The distribution of a random variable  $X$  is denoted by  $P_{X}$ , and the conditional distribution of  $X$  given  $Y$  is denoted by  $P_{X|Y}$ . When conditioning on a specific realization  $y$ , we use the shorthand  $P_{X|Y=y}$  or simply  $P_{X|y}$ . Denote by  $\mathbb{E}_X$  expectation over  $X \sim P_X$ , and by  $\mathbb{E}_{X|Y=y}$  (or  $\mathbb{E}_{X|y}$ ) expectation over  $X \sim P_{X|Y=y}$ . The entropy of a random variable  $X$  is denoted by  $H(X)$ , and the KL divergence of probability distribution  $P$  with respect to  $Q$  is denoted by  $D_{\mathrm{KL}}(P||Q)$ . The mutual information (MI) between random variables  $X$  and  $Y$  is denoted by  $I(X;Y)$ , and the conditional mutual information between  $X$  and  $Y$  given  $Z$  is denoted by  $I(X;Y|Z)$ . We also define the disintegrated mutual information as  $I^z(X;Y) \triangleq D_{\mathrm{KL}}(P_{X,Y|Z=z}||P_{X|Z=z}P_{Y|Z=z})$ , following the notation in [40]. Note that  $I(X;Y|Z) = \mathbb{E}_Z[I^Z(X;Y)]$ .
Generalization Error and Uniform Stability We consider the supervised learning setting, where we have a domain of instances  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}$ , with input and label spaces denoted by  $\mathcal{X}$  and  $\mathcal{Y}$  respectively. The distribution of an instance is given by  $\mu$ , and we have a training sample  $S = \{Z_i\}_{i = 1}^n\sim \mu^n$ . Let  $R\in \mathcal{R}$  be a source of randomness (a random variable independent of  $S$  over an appropriate space  $\mathcal{R}$ ), from which a learning algorithm  $\mathcal{A}:\mathcal{Z}^n\times \mathcal{R}\to \mathcal{W}$  takes the training sample  $S$  and  $R$  as input, and outputs a hypothesis  $W = \mathcal{A}(S,R)\in \mathcal{W}$ . To evaluate the quality of the output hypothesis  $W$ , we use a loss function  $\ell :\mathcal{W}\times \mathcal{Z}\rightarrow \mathbb{R}_0^+$ . Given a fixed  $w$ , we define the population risk  $L_{\mu}(w)\triangleq \mathbb{E}_{Z'}[\ell (w,Z')]$ , where  $Z^{\prime}\sim \mu$  is a testing instance. The quantity  $L_{\mu} = \mathbb{E}_W[L_\mu (W)]$  is then the expected population risk. For a fixed  $w\in \mathcal{W}$ , the empirical risk on  $S$  is defined as  $L_{S}(w)\triangleq \frac{1}{n}\sum_{i = 1}^{n}\ell (w,Z_{i})$ . Similarly, we define the expected empirical risk as  $\hat{L}_n = \mathbb{E}_{W,S}[L_S(W)]$ . Thus, the expected generalization error is given by  $\mathcal{E}_{\mu}(\mathcal{A})\triangleq L_{\mu} - \hat{L}_n$ .
We now give two notions of uniform stability [6, 13], where  $s \simeq s^i$  denotes two training sets  $s$  and  $s^i$  that differ only at the  $i$ th element. We say a learning algorithm  $\mathcal{A}$  is  $\beta_{1}$ -weakly uniformly stable if
$$
\sup  _ {s \simeq s ^ {i}, z} \mathbb {E} _ {R} \left| \ell (\mathcal {A} (s, R), z) - \ell (\mathcal {A} (s ^ {i}, R), z) \right| \leq \beta_ {1},
$$
and  $\beta_{2}$ -strongly uniformly stable if
$$
\sup_{s\simeq s^{i},z}\sup_{r}\left|\ell (\mathcal{A}(s,r),z) - \ell (\mathcal{A}(s^{i},r),z)\right|\leq \beta_{2}.
$$
Remark 2.1. Notably the weak uniform stability above is the standard in the literature. It is evident that if  $\mathcal{A}$  is  $\beta$ -strongly uniformly stable, it must also be  $\beta$ -weakly uniformly stable. It is also worth noting that for deterministic algorithms (e.g.,  $GD$  or fixed permutation  $SGD$ ), the two notions are identical. We note that in this paper, when speaking of uniform stability, we refer to the strong notion.
Let  $S' = \{Z_{i}'\}_{i=1}^{n} \sim \mu^n$  be an independent copy of  $S$ , and let  $S^{\backslash i} = S \setminus \{Z_i\}$  so  $S^i = S^{\backslash i} \cup \{Z_i'\}$ .
The following well-known result (e.g., [6, Lemma 7], [54, Thm. 13.2]) is frequently used in this paper.
Lemma 2.1. For any algorithm  $\mathcal{A}$ , we have  $L_{\mu} = \mathbb{E}_{S,S',R}\left[\frac{1}{n}\sum_{i=1}^{n}\ell(\mathcal{A}(S^i,R),Z_i)\right]$ , and
$$
\mathcal {E} _ {\mu} (\mathcal {A}) = \mathbb {E} _ {S, S ^ {\prime}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \mathbb {E} _ {\mathcal {A} (S ^ {i}, R) | S, Z _ {i} ^ {\prime}} \ell (\mathcal {A} (S ^ {i}, R), Z _ {i}) - \mathbb {E} _ {\mathcal {A} (S, R) | S} \ell (\mathcal {A} (S, R), Z _ {i}) \right] \right]. \tag {1}
$$
Supersample and Sample-Conditioned Hypothesis Stability Following [58], let  $\widetilde{Z} \in \mathcal{Z}^{n \times 2}$  be a supersample matrix with  $n$  rows and 2 columns, each entry drawn independently from  $\mu$ . We index the columns of  $\widetilde{Z}$  by 0,1 and denote the  $i$ th row of  $\widetilde{Z}$  by  $\widetilde{Z}_i$ , with entries  $(\widetilde{Z}_{i,0}, \widetilde{Z}_{i,1})$ . We often use the superscripts  $+$  and  $-$  to respectively replace the subscripts 0 and 1, i.e., writing  $\widetilde{Z}_i^+$  for  $\widetilde{Z}_{i,0}$  and  $\widetilde{Z}_i^-$  for  $\widetilde{Z}_{i,1}$ . Correspondingly,  $\widetilde{Z}_{[n]}^+$  and  $\widetilde{Z}_{[n]}^-$  denote the first and second columns, respectively. Additionally, we will use  $\widetilde{Z}_{[n] \sim i}^+$  to denote  $\widetilde{Z}_{[n]}^+$  in which the  $i$ th element  $\widetilde{Z}_i^+$  is replaced with the corresponding element  $\widetilde{Z}_i^-$  in the second column.
Let  $\widetilde{W}^{+} = \mathcal{A}(\widetilde{Z}_{[n]}^{+},R)$ , and  $\widetilde{W}_i^{-} = \mathcal{A}(\widetilde{Z}_{[n]\sim i}^{+},R)$  for each  $i\in [n]$ . That is,  $\widetilde{W}^{+}$  and each  $\widetilde{W}_i^{-}$  are obtained by two "neighboring" training samples, namely those differ only on one instance (the  $i$ th instance). We then construct matrix  $\widetilde{W}\in \mathcal{W}^{n\times 2}$ , where the  $i$ th row  $\widetilde{W}_i = (\widetilde{W}_i^+, \widetilde{W}_i^-) = (\widetilde{W}_{i,0},\widetilde{W}_{i,1})$  and  $\widetilde{W}_i^+ = \widetilde{W}^+$  for all  $i\in [n]$ , as shown in Table 1. Unlike the supersample matrix, elements in  $\widetilde{W}$  are identically distributed but not independent. In this case, Eq. (1) can be rewritten as
Z1+	Z1-
Z2+	Z2-
:	:
Zn+	Zn-
Table 1: Supersample (Left) and the induced hypotheses (Right).
$$
\mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}, \widetilde {W} _ {i}} \left[ \ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {+}\right) \right]. \tag {2}
$$
The summand in Eq (2) exhibits an interesting "symmetry": for any  $i$ ,
$$
\mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}, \widetilde {W} _ {i}} \left[ \ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {+}\right) \right] = \mathbb {E} _ {\widetilde {Z} _ {i} ^ {-}, \widetilde {W} _ {i}} \left[ \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {-}\right) - \ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {-}\right) \right]. \tag {3}
$$
That is, the  $+ / -$  superscripts in the summand of Eq (2) can be flipped for any  $i$ . We may then use  $n$  binary ( $\{0,1\}$ -valued) variables  $(U_1,U_2,\ldots ,U_n)\coloneqq U$  to govern whether we choose to flip the signs for each of the  $n$  terms, where  $U_{i} = 0$  indicates "no flipping". Let  $\overline{U}_i\triangleq 1 - U_i$ , we have
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i}, \widetilde {W} _ {i}, U _ {i}} \left[ \ell \left(\widetilde {W} _ {i, \overline {{U}} _ {i}}, \widetilde {Z} _ {i, U _ {i}}\right) - \ell \left(\widetilde {W} _ {i, U _ {i}}, \widetilde {Z} _ {i, U _ {i}}\right) \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i}, \widetilde {W} _ {i}, U _ {i}} \left[ (- 1) ^ {U _ {i}} \left(\ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i, U _ {i}}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i, U _ {i}}\right)\right) \right] (4) \\ = \mathbb {E} _ {\widetilde {W}, E, U} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} (- 1) ^ {U _ {i}} \left(\ell \left(\widetilde {W} _ {i} ^ {-}, \widehat {Z} _ {i}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widehat {Z} _ {i}\right)\right) \right], (5) \\ \end{array}
$$
where in Eq. (4) we have chosen  $U$  to be an i.i.d. Bernoulli-  $(\frac{1}{2})$  sequence, and in Eq. (5) we have renamed  $\widetilde{Z}_{i,U_i}$  as  $\widehat{Z}_i$  and denoted  $E\triangleq (\widehat{Z}_1,\widehat{Z}_2,\dots ,\widehat{Z}_n)$ . Note that  $E$ , induced by  $U$  from  $\widetilde{Z}$ , contains  $n$  instances, each serving to evaluate the loss difference between a pair of hypotheses  $(\widetilde{W}_i^{-},\widetilde{W}^{+})$ . The polarities of these evaluations are governed by  $U$ .
We now define some new notions of stability.
Definition 2.1 (Sample-Conditioned Hypothesis (SCH) Stability). Let  $S \sim \mu^n$ ,  $W$  and  $W^i$  generated via  $W = \mathcal{A}(S, R)$  and  $W^i = \mathcal{A}(S^i, R)$ . Let  $\mathcal{Z}_{w,w^i}$  denote the support of the conditional distribution  $P_{Z_i|w,w^i}$ , and  $\mathcal{W}_s$  denote the support of the conditional distribution  $P_{W|s}$ . We introduce four types of SCH stability, referred to types A, B, C, D. Specifically, a learning algorithm  $\mathcal{A}$  is
a)  $\gamma_{1}$ -SCH-A stable if  $\forall i\in [n]$ ,  $\sup_{w\in \cup_{s\in \mathbb{Z}^n}\mathcal{W}_s}\sup_{z\in \mathcal{Z}}\left|\ell (w,z) - \mathbb{E}_{W^i |w}\left[\ell (W^i,z)\right]\right|\leq \gamma_1,$  (6)
b)  $\gamma_{2}$ -SCH-B stable if  $\forall i\in [n]$ ,  $\mathbb{E}_{S,R,Z^{\prime}}\left[\left(\ell (W,Z^{\prime}) - \mathbb{E}_{W^{i}|W}\left[\ell (W^{i},Z^{\prime})\right]\right)^{2}\right]\leq \gamma_{2}^{2},$  (7)
c)  $\gamma_{3}$ -SCH-C stable if  $\forall i\in [n]$ ,  $\mathbb{E}_{W,W^i}\left[\sup_{z_i\in \mathcal{Z}_{W,W^i}}\big|\ell (W,z_i) - \ell (W^i,z_i)\big|\right]\leq \gamma_3,$  (8)
$d)\gamma_{4}$  -SCH-  $D$  stable if  $\forall i\in [n]$ $\mathbb{E}_{S,Z_i',R}\left[\left(\ell (W,Z_i) - \ell (W^i,Z_i)\right)^2\right]\leq \gamma_4^2,$  (9)
where  $Z^{\prime}$  in Eq. (7) is an independent instance drawn from  $\mu$
Remark 2.2. By definition, we can see  $\gamma_{2} \leq \gamma_{1}$ , and it is expected that  $\gamma_{4} \leq \gamma_{3}$ . Note that all of them are smaller than  $\beta_{2}$ . In addition, it is expected that  $\gamma_{2}, \gamma_{3}$  and  $\gamma_{4}$  are smaller than  $\beta_{1}$ , although the relationship between  $\gamma_{1}$  and  $\beta_{1}$  is uncertain. Moreover, it is also expected that  $\gamma_{4}$  is larger than  $\gamma_{2}$  due to the independence of  $Z'$  in Eq. (7). We emphasize that  $\sup_{w} \mathcal{W}_{s}$  in Eq. (6) is taken over the sample-dependent hypothesis space  $\cup_{s \in \mathbb{Z}^{n}} \mathcal{W}_{s}$ , not the whole hypothesis space  $\mathcal{W}$ . Notably, the notion of "hypothesis set stability" in [16] is closely related to our SCH stability. In fact,  $\gamma_{1}$ -SCH-A stable implies  $\gamma_{1}$ -hypothesis set stable. Due to space constraints, we provide further elaboration on the definition of SCH stability in Appendix B.
3 IOMI Bounds for Stable Algorithms
We are now in a position to give the IOMI bounds for stable learning algorithms.
Theorem 3.1. If the learning algorithm  $\mathcal{A}$  is  $\gamma_{1}$ -SCH-A stable, then
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\sqrt {2} \gamma_ {1}}{n} \sum_ {i = 1} ^ {n} \sqrt {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+})} \leq \frac {\sqrt {2} \gamma_ {1}}{n} \sum_ {i = 1} ^ {n} \sqrt {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+} | \widetilde {W} _ {i} ^ {-})}.
$$
We note that  $I(\widetilde{W}^{+};\widetilde{Z}_{i}^{+}) = I(\widetilde{W}_{i}^{-};\widetilde{Z}_{i}^{-}) = I(W;Z_{i})$ , the only difference between the first bound in Theorem 3.1 and the previous individual IOMI bound in [9] is that the sub-Gaussian variance proxy is replaced by the stability parameter  $\gamma_{1}$  in our bound. In fact, while we use  $\widetilde{W}$  to better understand the appearance of  $\gamma_{1}$  in the IOMI bounds, the first bound in Theorem 3.1 itself does not necessarily rely on the supersample and the construction of  $\widetilde{W}$ .
In addition, if  $\mathcal{A}$  is a deterministic algorithm, the term  $I(\widetilde{W}^{+};\widetilde{Z}_{i}^{+}|\widetilde{W}_{i}^{-})$  in the second bound is tighter than the mutual information stability or erasure information studied in [46, 23], that is,  $I(\widetilde{W}^{+};\widetilde{Z}_{i}^{+}|\widetilde{W}_{i}^{-})\leq I\left(\widetilde{W}^{+};\widetilde{Z}_{i}^{+}|\widetilde{Z}_{[n]\setminus i}^{+}\right)$  (see Remark C.1 in the Appendix for an explanation).
Proof Sketch of Theorem 3.1. Motivated by Lemma 2.1 and Eq. (2), the main innovation in this proof is to let the auxiliary function in DV be the "relative loss" instead of the single loss, namely we let  $g(\tilde{w}_i^+, \tilde{z}_i^+) = \mathbb{E}_{\widetilde{W}_i^-| \tilde{w}^+} \left[ \ell(\widetilde{W}_i^-, \tilde{z}_i^+) \right] - \ell(\tilde{w}^+, \tilde{z}_i^+)$  and let the auxiliary function  $f = t \cdot g$  for  $t > 0$  in Lemma A.2. This enables us to utilize Eq. (6) to bound the CGF. The remaining steps are routine. The complete proof can be found in Appendix C.1.
The following corollary, immediately following from  $\gamma_1 \leq \beta_2$ , suggests a clear improvement over the previous IOMI bound for the uniformly stable deterministic algorithm with vanishing  $\beta_2$ .
Corollary 3.1. If  $\mathcal{A}$  is  $\beta_{2}$ -uniform stable, then  $|\mathcal{E}_{\mu}(\mathcal{A})| \leq \frac{\sqrt{2}\beta_{2}}{n}\sum_{i=1}^{n}\sqrt{I(\widetilde{W}^{+};\widetilde{Z}_{i}^{+})}$ .
A key message conveyed in this paper, as shown in the proof of Theorem 3.1, is the importance of carefully selecting the DV auxiliary function and appropriate assumptions for various learning scenarios. Corollary 3.1 also suggests a potential enhancement for uniform stability in the certain deterministic setting, provided that  $I(W;Z_{i})$  also vanishes appropriately with  $n$ .
Similar to [40, 49, 23, 25], we also give an  $R$ -conditioned IOMI bound below.
Theorem 3.2. If  $\mathcal{A}$  is  $\beta_{2}$ -uniform stable, we have  $|\mathcal{E}_{\mu}(\mathcal{A})| \leq \frac{\sqrt{2}\beta_{2}}{n}\sum_{i=1}^{n}\mathbb{E}_{R}\sqrt{I^{R}(\widetilde{W}^{+};\widetilde{Z}_{i}^{+})}$ .
This disintegrated IOMI bound is not directly comparable to the bounds in Theorem 3.1, but it is equivalent to Corollary 3.1 for deterministic algorithms. With additional assumptions, we may remove the square-root in Theorem 3.1, accelerating the decay of IOMI as shown in Theorem 3.3.
Theorem 3.3. Under the same conditions in Theorem 3.1, if  $\mathcal{A}$  is further  $\gamma_{2}$ -SCH-B stable, then