SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\gamma_ {1}}{n} \sum_ {i = 1} ^ {n} I (\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+}) + 0. 7 2 \frac {\gamma_ {2} ^ {2}}{\gamma_ {1}}.
$$
Notice that  $\gamma_2^2 /\gamma_1\leq \gamma_1^2 /\gamma_1 = \gamma_1$ . If  $\gamma_2^2 /\gamma_1^2$  decays faster than  $\frac{1}{n}\sum_{i = 1}^{n}\sqrt{I(\widetilde{W}^{+};\widetilde{Z}_{i}^{+})}$ , then Theorem 3.3 is qualitatively strictly stronger than Theorem 3.1.
4 CMI Bounds for Stable Algorithms
Hypotheses-Conditioned CMI Bounds In this section, we give a handful of CMI bounds based on a new information-theoretic quantity.
Theorem 4.1. Suppose that there exists  $\Delta_1: \mathcal{W}^2 \to \mathbb{R}$  such that for every  $\tilde{w}_i = (\tilde{w}_i^+, \tilde{w}_i^-)$ ,  $\sup_{z_i \in \mathcal{Z}_{\tilde{w}_i^+, \tilde{w}_i^-}} |\ell(\tilde{w}_i^+, z_i) - \ell(\tilde{w}_i^-, z_i)| \leq \Delta_1(\tilde{w}_i)$ . Then,
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\sqrt {2}}{n} \sum_ {i = 1} ^ {n} \min \left\{\mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta_ {1} (\widetilde {W} _ {i}) \sqrt {I ^ {\widetilde {W} _ {i}} (\widehat {Z} _ {i} ; U _ {i})} \right], \sqrt {\mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta_ {1} (\widetilde {W} _ {i}) ^ {2} \right] I (\widehat {Z} _ {i} ; U _ {i} | \widetilde {W} _ {i})} \right\}.
$$
Furthermore, if  $\mathcal{A}$  is  $\gamma_3$ -SCH- $C$  stable, then such a  $\Delta_1$  exists and
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\sqrt {2} \gamma_ {3}}{n} \sum_ {i = 1} ^ {n} \sqrt {\sup  _ {\tilde {w} _ {i}} I ^ {\tilde {w} _ {i}} \left(\widehat {Z} _ {i} ; U _ {i}\right)}.
$$
Compared to the standard supersample-conditioned CMI bound [58] in terms of  $I(W;U|\widetilde{Z})$ , which assesses how well one can infer the "training-set membership" from the output hypothesis, our new CMI quantity, namely  $I(\widehat{Z}_i;U_i|\widetilde{W}_i)$ , measures our ability to decide if an instance  $\widehat{Z}_i$  contributes to the training of  $\widetilde{W}_i^+$  or to the training of  $\widetilde{W}_i^-$  when we know it contributes to only one of them. When  $\widetilde{W}_i^+$  and  $\widetilde{W}_i^-$  are similar, this decision (i.e., determining  $U_i$ ) is difficult, giving rise to small  $I(\widehat{Z}_i;U_i|\widetilde{W}_i)$ . Additionally, for uniformly stable algorithms,  $\sup_{w_i}\Delta_1(\widetilde{w}_i)\leq \beta_2$  and it is also expected that  $\mathbb{E}_{\widetilde{W}_i}\left[\Delta_1(\widetilde{W}_i)^2\right]\leq \beta_1^2$ . Moreover, if we simply replace  $\gamma_3$  by an upper bound of  $\ell$ , the second bound of Theorem 4.1 becomes a uniform convergence bound if  $I^{\tilde{w}}(\widehat{Z}_i;U_i)$  vanishes with  $n$ .
Proof Sketch of Theorem 4.1. Again we find motivation in Lemma 2.1. Additionally, the symmetry exhibited in Eq. (5), analogous to the symmetry between  $\widetilde{Z}_i^+$  and  $\widetilde{Z}_i^-$  used in deriving the standard CMI bounds, allows a similar development. Specifically, letting  $g(\tilde{w}_i,\hat{z}_i,u_i) = (-1)^{u_i}\left(\ell (\tilde{w}_i^-, \hat{z}_i) - \ell (\tilde{w}_i^+, \hat{z}_i)\right)$  and  $f = t\cdot g$  for  $t > 0$  in Lemma A.2 enables the bounding of the CGF via invoking the assumptions in the theorem. The complete proof is given in Appendix D.1.
Notably, our new CMI quantity in the bound preserves the boundedness of the original CMI in [58], that is,  $I(\widehat{Z}_i; U_i | \widetilde{W}) \leq H(U_i) = \log 2$ . Furthermore, parallel to supersample-conditioned CMI being smaller than IOMI [18], a similar result for hypotheses-conditioned CMI is given below.
Theorem 4.2. For any  $\mathcal{A}$  and  $\mu$ , we have  $I(\widehat{Z}_i;U_i|\widetilde{W_i})\leq I(W;Z_i)$ .
Similar to Theorem 3.3, we present a CMI bound without square-root, which could be much stronger in certain regimes.
Theorem 4.3. Let  $\Delta_{1}$  be defined in the same way as in Theorem 4.1, and we let  $\Lambda (\tilde{w}_i) = \mathbb{E}_{\widehat{Z}_i|\tilde{w}_i}\left[\left(\ell (\tilde{w}_i^{-},\widehat{Z}_i) - \ell (\tilde{w}_i^{+},\widehat{Z}_i)\right)^2\right]\bigg{/}\Delta_1(\tilde{w}_i)^2$  , then
$$
\left| \mathcal {E} _ {\mu} (\mathcal {A}) \right| \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta_ {1} \left(\widetilde {W} _ {i}\right) \left(I ^ {\widetilde {W} _ {i}} \left(\widehat {Z} _ {i}; U _ {i}\right) + 0. 7 2 \Lambda \left(\widetilde {W} _ {i}\right)\right) \right]. \tag {10}
$$
If  $\mathcal{A}$  is further  $\beta_{2}$ -uniform stable and  $\gamma_{4}$ -SCH-D stable, then
$$
\left| \mathcal {E} _ {\mu} (\mathcal {A}) \right| \leq \frac {\beta_ {2}}{n} \sum_ {i = 1} ^ {n} I \left(\widehat {Z} _ {i}; U _ {i} \mid \widetilde {W _ {i}}\right) + 0. 7 2 \frac {\gamma_ {4} ^ {2}}{\beta_ {2}}. \tag {11}
$$
Note that it is valid to set  $\gamma_3 = \mathbb{E}_{\widetilde{W}_i}\left[\Delta_1(\widetilde{W}_i)\right]$ , which can be viewed as a generalization bound in its own right. Additionally, it can be verified that  $\Lambda (\tilde{w}_i)\leq 1$  for any  $\tilde{w}_i$ . As  $I^{\widetilde{W}_i}(\widehat{Z}_i;U_i)\leq \log 2\approx 0.69$ , Eq. (10) can be further upper bounded by  $1.41\gamma_{3}$ . This ensures that Eq. (10) will decay no slower than  $\mathcal{O}(\gamma_3)$ .
We also present a second moment generalization bound.
Theorem 4.4. Assume that  $\mathcal{A}$  is  $\beta_{2}$ -uniform stable and symmetric with respect to  $S$ , i.e. it does not depend on the order of the elements in  $S$ . Let  $\ell(\cdot, \cdot) \in [0,1]$ , then
$$
\mathbb {E} _ {W, S} \left[ \left(L _ {\mu} (W) - L _ {S} (W)\right) ^ {2} \right] \leq 4 \beta_ {2} ^ {2} \left(\frac {1 . 5 I (E ; U | \widetilde {W}) + 0 . 8 2}{n} + 1\right) + \frac {1}{n}.
$$
Since  $I(E;U|\widetilde{W}) \leq \mathcal{O}(n)$ , the bound can be further upper bounded by  $\mathcal{O}(\beta_2^2 + 1/n)$ , which matches the previous tight bound for the second moment generalization error in [14, Thm. 1.2]. Notice that [14, Thm. 1.2] only holds for the deterministic setting, while our bound also holds for randomized algorithms, and we also give a stronger result in Appendix D.4 based on our SCH stability notions.
Supersample-Conditioned CMI Bounds It is also possible to give  $\widetilde{Z}$ -conditioned CMI bounds that explicitly contain the stability parameters. Let  $W_{i} = \widetilde{W}_{i,U_{i}}$  and  $\overline{W}_i = \widetilde{W}_{i,\overline{U}_i}$ , we have the following results.
Theorem 4.5. Suppose there exists  $\Delta_2: \mathcal{Z} \to \mathbb{R}$  such that  $\sup_{w, w^i \in \mathcal{W}_{z_i}^2} |\ell(w, z_i) - \ell(w^i, z_i)| \leq \Delta_2(z_i)$  for every  $z_i$ , where  $\mathcal{W}_{z_i}^2$  is the support of the conditional distribution  $P_{W, W^i|z_i}$ . Then,
$$
\left| \mathcal {E} _ {\mu} (\mathcal {A}) \right| \leq \frac {\sqrt {2}}{n} \sum_ {i = 1} ^ {n} \min  \left\{\mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \Delta_ {2} \left(\widetilde {Z} _ {i} ^ {+}\right) \sqrt {I ^ {\widetilde {Z} _ {i} ^ {+}} \left(W _ {i} , \overline {{W}} _ {i} ; U _ {i}\right)} \right], \sqrt {\mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \Delta_ {2} \left(\widetilde {Z} _ {i} ^ {+}\right) ^ {2} \right] I \left(W _ {i} , \overline {{W}} _ {i} ; U _ {i} \mid \widetilde {Z} _ {i} ^ {+}\right)} \right\}.
$$
The proof is deferred to Appendix D.5. The interpretation of the CMI quantity  $I(W_{i}, \overline{W}_{i}; U_{i}|\widetilde{Z}_{i}^{+})$  appears identical to our earlier CMI quantity  $I(\widehat{Z}_i;U_i|\widetilde{W}_i)$ . A closer look in fact reveals that the two quantities are mathematically equal. To see this, first note  $I(\widehat{Z}_i;U_i|\widetilde{W}_i) = H(U_i) - H(U_i|\widehat{Z}_i,\widetilde{W}_i)$  and  $I(W_{i},\overline{W}_{i};U_{i}|\widetilde{Z}_{i}^{+}) = H(U_{i}) - H(U_{i}|\widetilde{Z}_{i}^{+},W_{i},\overline{W}_{i})$ . Let random variable  $A_{1} = (\widehat{Z}_{i},\widetilde{W}_{i}^{+},\widetilde{W}_{i}^{-})$  and let  $A_{2} = (\widetilde{Z}_{i}^{+},W_{i},\overline{W}_{i})$ , these two random variables are identically distribution (since  $P_{\widetilde{W}_i^+} = P_{\widetilde{W}_i^-}$ ,  $P_{\widetilde{Z}_i^+} = P_{\widetilde{Z}_i^-}$  and  $U_{i} \sim \mathrm{Bernoulli} - (\frac{1}{2})$  and also we have  $P_{A_1|U_i} = P_{A_2|U_i}$  so  $P_{A_1,U_i} = P_{A_2,U_i}$ , which gives us  $H(U_i|A_1) = H(U_i|A_2)$ . This indicates that  $I(\widehat{Z}_i;U_i|\widetilde{W}_i) = I(W_i,\overline{W}_i;U_i|\widetilde{Z}_i^+)$ .
In Theorem 4.5, a data-dependent hypothesis space  $\mathcal{W}_{z_i}^2$  is defined. A similar concept has been utilized in the hypothesis set cross-validation (CV) stability studied in [16]. Furthermore, [16] derives some bounds based on either their transductive Rademacher complexity or their hypothesis set CV stability. They show that these two notions dominate in different learning scenarios. Given the close relationship between the supersample construction and the Rademacher complexity [58, 64], and the inspiration behind our  $\widetilde{W}$  construction, our framework is likely to have a fundamental connection to [16]. Additionally, obtaining the similar results of  $\widetilde{Z}$ -conditioned CMI as in Theorem 4.3-4.4 is warranted, which may require some stability notions analogous to the average CV-stability in [16].
5 Convex-Lipschitz-Bounded (CLB) Problems
We now discuss two examples of the convex-Lipschitz-bounded (CLB) problem, a subclass of SCO problems.
The first example is previously given in [21, Thm. 17], in which nearly all previous information-theoretic bounds are non-vanishing. We will demonstrate that our CMI bounds are non-vacuous in this example.
Example 1. Let  $d \in \mathbb{N}$  and  $\mathcal{Z} = \{e(i) : i \in [d]\}$  where  $e(i)$  is a one-hot vector with 1 at the  $i$ -th coordinate. Let  $\mu = \mathrm{Unif}(\mathcal{Z})$ . Given a sample  $S = \{Z_i\}_{i=1}^n$  drawn i.i.d. from  $\mu$ , we choose the 1-Lipschitz convex loss function  $\ell(w, z) = -\langle w, z \rangle$  and use  $GD$  to select a hypothesis  $w$  from  $\mathcal{W} = \{w \in \mathbb{R}^d : \|w\| \leq 1\}$ . Let the number of  $GD$  iterations be  $T = n^2$  and let the learning rate be  $\eta = \frac{1}{n\sqrt{n}}$ .
Let  $\hat{\mu} = \frac{1}{n}\sum_{i=1}^{n}z_{i}$  be the sample mean. In this deterministic setting, it's easy to see that
$$
w _ {t} = \left\{ \begin{array}{l l} \eta t \hat {\mu} & \text {i f} \eta t | | \hat {\mu} | | \leq 1, \\ \eta t \hat {\mu} / | | \eta t \hat {\mu} | | & \text {o t h e r w i s e}. \end{array} \right.
$$
Let  $\hat{\mu}^i$  be the sample mean of  $s^i$  and let  $w_t^i$  be its corresponding hypothesis at time  $t$ . Notice that Euclidean projection does not increase the distance between projected points, namely non-expansive [22, Lemma 4.6]. Hence, whether  $w_t = \eta t\hat{\mu}$  or its truncated version  $\eta t\hat{\mu} / ||\eta t\hat{\mu}||$  limited within the unit ball, we have  $||w_t - w_t^i|| \leq ||\eta t\hat{\mu} - \eta t\hat{\mu}^i|| \leq \mathcal{O}(\eta t / n)$ . Recall that the loss function is 1-Lipschitz, we have  $|\mathcal{E}_{\mu}(\mathcal{A})| \leq \beta_2 \leq \mathcal{O}(\eta t / n)$ . In this example,  $\eta T / n = 1 / \sqrt{n}$  so  $\beta_2 \in \mathcal{O}(1 / \sqrt{n})$ . One can also directly obtain this rate from [22, 5].
Now following the same setting in [21], if we let  $d = 2n^2$ , we can find that  $I(W_T; Z_i) \in \Omega(1)$  (see [21, Thm. 17] or Appendix F). Thus, IOMI itself could not explain the generalization of GD in this problem. Furthermore, all our CMI quantities including those in Section 6 also have the order of  $\Omega(1)$ , that is, they fail to vanish as  $n \to \infty$  (see Appendix F for more elaboration).
Therefore, the stability parameter  $\beta_{2}$  should not be replaced by some constant (e.g., the upper bound of the loss function) in the IOMI or CMI bound. In fact, for our CMI bounds, due to their boundedness property, we have the following corollary.
Corollary 5.1. If  $\mathcal{A}$  is  $\beta_{2}$ -uniform stable, we have  $\frac{\beta_2}{n} \sum_{i=1}^{n} \sqrt{I(\widehat{Z}_i; U_i | \widetilde{W}_i)} \leq \mathcal{O}(\beta_2)$ .
Corollary 5.1 provides a solution to the non-vanishing limitation of the previous information-theoretic bounds in Example 1 (and also the counterexample in [21, Thm. 4]), as it can explain generalization as long as the stability-based bound is sufficient. Thus, the shortfalls of information-theoretic bounds in analyzing deterministic algorithms for CLB problems are tempered by the stability-based framework.
We now show another CLB Example from [21, Thm. 3] where  $\mathcal{A}$  is not uniformly stable, and we will see information-theoretic bounds in this paper are tight up to a constant. This example is also studied in [42, Sec. 5].
Example 2. Let  $\mathcal{W} \in \mathbb{R}^d$  be a ball with radius  $R_0$ , and let the input space be  $\mathcal{Z} = \{z_0 / R_0, -z_0 / R_0\}$  where  $z_0 \in \mathcal{W}$  such that  $||z_0|| = R_0$ . Let  $\mu = \mathrm{Unif}(\mathcal{Z})$ . Consider a convex and L-Lipschitz loss function  $\ell(w,z) = -L\langle w,z\rangle$ . In addition,  $\mathcal{A}$  is any empirical risk minimization (ERM) algorithm.
In this example,  $\beta_{2} = 2LR_{0}$  is a constant. [21] has shown that  $|\mathcal{E}_{\mu}(\mathcal{A})| \geq \frac{LR_{0}}{\sqrt{2n}}$  and  $I(W;S) \leq 1$  (see [21, Thm. 3] or Appendix F for an explanation). This gives us  $\frac{2LR_{0}}{n}\sum_{i=1}\sqrt{I(W;Z_{i})} \leq 2LR_{0}\sqrt{\frac{I(W;S)}{n}} \leq \frac{2LR_{0}}{\sqrt{n}}$ . Thus, the distribution-dependent property of IOMI can improve the stability-based bound in this case. In addition, we know that  $I(\widehat{Z}_{i};U_{i}|\widetilde{W}_{i}) \leq I(W;Z_{i})$  from Theorem 4.2, our new CMI bounds are also tight (up to a constant) in this example.
Notably, we can construct an additional example building upon Example 2, where  $\mathcal{A}$  is uniformly stable but the uniform stability itself results in a slow convergence rate for generalization error. Specifically, let  $R_0 = \frac{1}{d}$  and let  $d = \sqrt{n}$ , then  $\beta_{2} \in \mathcal{O}(1 / \sqrt{n})$  while  $|\mathcal{E}_{\mu}(\mathcal{A})| \geq \frac{L}{\sqrt{2n}}$ . Note that the information-theoretic bounds in this paper can still provide a tight rate, namely  $\mathcal{O}(1 / n)$ .
These examples demonstrate that our bounds can improve both the stability-based bound and information-theoretic bounds in some learning scenarios.
Additional applications of our bounds are discussed in Appendix G.
Connection with Bernstein Condition The Bernstein condition is commonly used to derive fast-rate generalization bound for both PAC-Bayes bounds [69, 10, 36, 17] and stability-based bounds [28], then it is natural to explore the relationship between our fast-rate bounds and the Bernstein condition, formally defined below.
Definition 6.1 (Bernstein Condition). Assume that  $w^{*} = \arg \min_{w\in \mathcal{W}}L_{\mu}(w)$  is a risk minimizer. We say that the Bernstein assumption is satisfied with some  $B > 0$  and  $\kappa \in [1, + \infty)$  if for any  $w\in \mathcal{W}$ ,
$$
\mathbb {E} _ {Z} \left[ (\ell (w, Z) - \ell (w ^ {*}, Z)) ^ {2} \right] \leq B \left(L _ {\mu} (w) - L _ {\mu} (w ^ {*})\right) ^ {\frac {1}{\kappa}}.
$$
This condition can be easily satisfied in many common situations [1, 28]. In the following proposition, we can see that the Bernstein condition implies the  $\gamma_{2}$ -SCH-B stability.
Proposition 1. If the Bernstein condition is satisfied with some  $B$  and  $\kappa$ , then  $\mathcal{A}$  is  $\gamma_{2}$ -SCH-B stable, where  $\gamma_{2}^{2} = 4B\mathbb{E}_{W}\left[(L_{\mu}(W) - L_{\mu}(w^{*}))^{\frac{1}{\kappa}}\right]$ .
Therefore, invoking the Bernstein condition, we can obtain the fast-rate bound as presented in Theorem 3.3, where we need to assume the loss is bounded, as shown below as a by-product.
Corollary 6.1. If the Bernstein condition is satisfied with  $\kappa = 1$  and  $\ell \in [0,C]$ , then
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {C}{n} \sum_ {i = 1} ^ {n} I (W; Z _ {i}) + \frac {2 . 8 8 B}{C} \left(L _ {\mu} - L _ {\mu} (w ^ {*})\right).
$$
Recently, [65, 66] use the unexpected excess risk as the DV auxiliary function and invoke the  $(\eta, c)$ -central condition to establish some optimal-rate bounds for specific learning problems, e.g., Gaussian mean estimation. This again highlights the significance of selecting appropriate DV auxiliary functions and corresponding assumptions tailored to different learning problems. It is worth mentioning that the Bernstein condition also implies their  $(\eta, c)$ -central condition. Therefore, unifying these conditions can be considered as a potential avenue for future research.
Loss Difference, Evaluated, and Functional CMI for Stable Algorithms Similar to [64], we derive some tighter bounds based on the loss difference.
Theorem 6.1. Let  $\Delta L_{i} = \ell (W_{i},\widetilde{Z}_{i}^{+}) - \ell (\overline{W}_{i},\widetilde{Z}_{i}^{+})$  . If  $\mathcal{A}$  is  $\beta_{2}$  -uniform stable, then
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\sqrt {2} \beta_ {2}}{n} \sum_ {i = 1} ^ {n} \min \left\{\sqrt {I (\Delta L _ {i} ; U _ {i})}, \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \sqrt {I ^ {\widetilde {Z} _ {i} ^ {+}} (\Delta L _ {i} ; U _ {i})} \right\} \leq \frac {\sqrt {2} \beta_ {2}}{n} \sum_ {i = 1} ^ {n} \sqrt {I (\Delta L _ {i} ; U _ {i} | \widetilde {Z} _ {i} ^ {+})}.
$$
We note that while it is feasible to replace  $\beta_{2}$  in the (disintegrated) CMI bounds above with certain sample-conditioned hypothesis stability, it is not possible to apply the same substitution for the unconditional MI bound in Theorem 6.1.
Furthermore, notice that  $\Delta L_i - (L_i,\bar{L}_i) - (F_i,\overline{F}_i) - (W_i,\overline{W}_i)$  forms a Markov chain given  $\widetilde{Z}_i^+$ , wherein  $(L_{i},\bar{L}_{i})$  are the loss pair evaluated at  $\widetilde{Z}_i^+$  using  $(W_{i},\overline{W}_{i})$ , and  $(F_{i},\overline{F}_{i})$  are label predictions of  $\widetilde{Z}_i^+$  using  $(W_{i},\overline{W}_{i})$ . By the data-processing inequality, one can obtain e-CMI bound [58, 25],  $f$ -CMI bound [23] and recover  $I(W_{i},\overline{W}_{i};U_{i}|\widetilde{Z}_{i}^{+})$  based on Theorem 6.1:  $I(\Delta L_i;U_i|\widetilde{Z}_i^+) \leq I(L_i,\bar{L}_i;U_i|\widetilde{Z}_i^+) \leq I(F_i,\overline{F}_i;U_i|\widetilde{Z}_i^+) \leq I(W_i,\overline{W}_i;U_i|\widetilde{Z}_i^+)$ . Additionally, notice that we can also apply the similar technique for the hypotheses-conditioned CMI, which should give the same results.
Expressiveness of New CMI Notions Under Distribution-Free Setting Previous works [58, 19, 20, 23, 25] have demonstrated that the CMI framework is expressive enough to establish connections with VC theory in the distribution-free learning setting. Here, we further illustrate the expressiveness of the sample-conditioned CMI discussed in this work.
Theorem 6.2. Let  $\mathcal{Z} = \mathcal{X}\times \{0,1\}$ , and let  $\mathcal{F} = \{f_w:\mathcal{X}\to \{0,1\} |w\in \mathcal{W}\}$  be a functional hypothesis class with finite VC dimension  $d$ . Let  $n > d + 1$ , for any algorithm  $\mathcal{A}$ , we have  $\frac{1}{n}\sum_{i = 1}^{n}\sqrt{I(F_i,\bar{F}_i;U_i|\tilde{Z}_i^+)}\leq \mathcal{O}\left(\sqrt{\frac{d}{n}\log\left(\frac{n}{d}\right)}\right)$ .
Like the previous works, this bound matches the classic result of the uniform convergence bound [59]. Notice that the result could be extended to multi-class classification with finite Natarajan dimension [39] by proceeding similarly to [25, Thm. 8].
We invoke Theorem 6.2 to demonstrate that our new information-theoretic quantities have the same expressive power as standard CMI quantities. The expressiveness result for the (functional) hypotheses-conditioned CMI is expected to align with Theorem 6.2. This alignment is due to the equivalence between hypotheses-conditioned CMI and supersample-conditioned CMI, as discussed in Theorem 4.5.
7 Related Works and Additional Discussions
Stability-Based Framework vs. Information-Theoretic Framework Using stability methods to analyze generalization errors can be traced back to several seminal works, such as [51, 11, 12,
34, 27]. It is worth noting that stability arguments have proven particularly effective in analyzing the learnability of SCO problems [56], where traditional uniform convergence bounds may not be sufficient to explain the generalization behavior. The application of stability approaches has gained popularity for providing high-probability guarantees since the work of [6]. Recent advancements have further sharpened the convergence rates of high-probability generalization upper bounds for uniformly stable algorithms in a series of works [14, 15, 8, 28]. While information-theoretic bounds are commonly used to analyze the in expectation generalization, it is expected that combining them with the stability framework will yield sharper high-probability bounds.
Additionally, we note that in the realizable setting, where  $\mathcal{A}$  is an interpolating algorithm, information-theoretic bounds exhibit greater power than stability-based bounds. For instance, in the case of the  $0 - 1$  loss, information-theoretic bounds can achieve the known optimal minimax rates [19] and even provide exact characterizations of the generalization error [20, 64]. However, it should be noted that due to the inherent fitting-stability tradeoff property [54, Sec. 13.4], interpolating algorithms tend to be unstable, rendering stability arguments inapplicable. Given the prevalence of zero empirical risk in modern deep learning [68], it is natural to question whether information-theoretic bounds require the stability-based approach for analyzing non-convex (and potentially non-smooth and non-Lipschitz continuous) learning scenarios.
Connection Between Two Frameworks in Previous Works The connection between information-theoretic bounds, including some PAC-Bayes bounds, and algorithmic stability has been explored in previous literature [46, 33, 48, 29, 58, 23, 3, 47]. These works primarily focus on either regarding the information-theoretic quantities as notions of distributional stability [46, 67, 29, 58, 3] and/or converting information-theoretic quantities to some other algorithmic stability notions [58, 23, 47]. The later often relies on the addition of Gaussian noise to the hypotheses (or assuming that the prior and posterior distributions are Gaussian in PAC-Bayes). In [33], the authors also combine the DV formula with stability assumptions, where they derive some PAC-Bayes bounds. However, comparing these bounds with others is challenging in general due to the presence of their hyperparameter stability.
Comparison with Standard CMI While our IOMI quantity aligns with the previous work [9], the new CMI quantity  $I(\widehat{Z}_i; U_i | \widetilde{W}_i)$  (or equivalently  $I(W_i, \overline{W}_i; U_i | \widetilde{Z}_i^+)$ ) may not be directly comparable to the standard individual CMI  $I(W; U_i | \widetilde{Z}_i)$  in [49, 70]. Specifically, we have  $I(\widehat{Z}_i; U_i | \widetilde{W}_i) = H(U_i) - H(U_i | \widehat{Z}_i, \widetilde{W}_i)$  and  $I(W; U_i | \widetilde{Z}_i) = H(U_i) - H(U_i | W, \widetilde{Z}_i)$ . The relationship between  $H(U_i | \widehat{Z}_i, \widetilde{W}_i)$  and  $H(U_i | W, \widetilde{Z}_i)$  is not trivial. Exploring and quantitatively comparing these CMI measures would be an intriguing research direction.