SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {W _ {i}, \bar {W} _ {i}, U _ {i} | \tilde {z} _ {i} ^ {+}} \left[ g \left(\tilde {z} _ {i} ^ {+}, W _ {i}, \bar {W} _ {i}, U _ {i}\right) \right] \leq \inf  _ {t > 0} \frac {I \left(W _ {i} , \bar {W} _ {i} ; U _ {i} \mid \widetilde {Z} _ {i} ^ {+} = \tilde {z} _ {i} ^ {+}\right) + \log \mathbb {E} _ {W _ {i} , \bar {W} _ {i} , U _ {i} ^ {\prime} | \tilde {z} _ {i} ^ {+}} \left[ e ^ {t g \left(\tilde {z} _ {i} ^ {+}, W _ {i} , \bar {W} _ {i} , U _ {i} ^ {\prime}\right)} \right]}{t}. \tag {31}
$$
Similar to the previous proofs, it's easy to see that  $g(\tilde{z}_i^+, W_i, \overline{W}_i, U_i')$  is a zero-mean random variable bounded in  $[-\Delta_2(\tilde{z}_i^+), \Delta_2(\tilde{z}_i^+)]$ . Thus,
$$
\mathbb {E} _ {W _ {i}, \bar {W} _ {i}, U _ {i} | \tilde {z} _ {i} ^ {+}} \left[ g (\tilde {z} _ {i} ^ {+}, W _ {i}, \bar {W} _ {i}, U _ {i}) \right] \leq \inf  _ {t > 0} \frac {I \left(W _ {i} , \bar {W} _ {i} ; U _ {i} \mid \widetilde {Z} _ {i} ^ {+} = \tilde {z} _ {i} ^ {+}\right) + \frac {t ^ {2} \Delta_ {2} \left(\tilde {z} _ {i} ^ {+}\right) ^ {2}}{2}}{t}. \tag {32}
$$
To prove the first bound, we let  $t = \sqrt{\frac{I(W_i,\overline{W}_i;U_i|\widetilde{Z}_i^+ = \widetilde{z}_i^+)}{2\Delta_2(\widetilde{z}_i^+)^2}}$ , then
$$
\mathbb {E} _ {W _ {i}, \overline {{W}} _ {i}, U _ {i} | \tilde {z} _ {i} ^ {+}} \left[ g (\tilde {z} _ {i} ^ {+}, W _ {i}, \overline {{W}} _ {i}, U _ {i}) \right] \leq \Delta_ {2} (\tilde {z} _ {i} ^ {+}) \sqrt {2 I (W _ {i} , \overline {{W}} _ {i} ; U _ {i} | \widetilde {Z} _ {i} ^ {+} = \tilde {z} _ {i} ^ {+})}.
$$
Recall Eq. (14) in Lemma A.1, hence,
$$
\left| \mathcal {E} _ {\mu} (\mathcal {A}) \right| \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \Delta_ {2} \left(\widetilde {Z} _ {i} ^ {+}\right) \sqrt {2 I ^ {\widetilde {Z} _ {i} ^ {+}} \left(W _ {i} , \bar {W} _ {i} ; U _ {i}\right)} \right]. \tag {33}
$$
To prove the second bound, we take expectation over  $\widetilde{Z}_i^+$  for Eq. (32),
$$
\mathbb {E} _ {W _ {i}, \bar {W} _ {i}, U _ {i}, \tilde {Z} _ {i} ^ {+}} \left[ g (\tilde {Z} _ {i} ^ {+}, W _ {i}, \bar {W} _ {i}, U _ {i}) \right] \leq \inf  _ {t > 0} \frac {I (W _ {i} , \bar {W} _ {i} ; U _ {i} | \tilde {Z} _ {i} ^ {+}) + \frac {t ^ {2} \mathbb {E} _ {\tilde {Z} _ {i} ^ {+}} \left[ \Delta_ {2} (\tilde {Z} _ {i} ^ {+}) ^ {2} \right]}{2}}{t}.
$$
Let  $t = \sqrt{\frac{I(W_i,\overline{W}_i;U_i[\widetilde{Z}_i^+})}{2\mathbb{E}_{\widetilde{Z}_i^+}\left[\Delta_2(\widetilde{Z}_i^+)^2\right]}},$  then
$$
\mathbb {E} _ {W _ {i}, \overline {{W}} _ {i}, U _ {i}, \widetilde {Z} _ {i} ^ {+}} \left[ g (\widetilde {Z} _ {i} ^ {+}, W _ {i}, \overline {{W}} _ {i}, U _ {i}) \right] \leq \sqrt {2 \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \Delta_ {2} (\widetilde {Z} _ {i} ^ {+}) ^ {2} \right] I (W _ {i} , \overline {{W}} _ {i} ; U _ {i} | \widetilde {Z} _ {i} ^ {+})}.
$$
Ergo,
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \sqrt {2 \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \Delta_ {2} \left(\widetilde {Z} _ {i} ^ {+}\right) ^ {2} \right] I \left(W _ {i} , \bar {W} _ {i} ; U _ {i} \mid \widetilde {Z} _ {i} ^ {+}\right)}. \tag {34}
$$
E Omitted Proof in Section 6
Proof. By Jensen's inequality and triangle inequality, for any  $i \in [n]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {S, R, Z ^ {\prime}} \left[ \left(\ell (W, Z ^ {\prime}) - \mathbb {E} _ {W ^ {i} | W} \left[ \ell \left(W ^ {i}, Z ^ {\prime}\right) \right]\right) ^ {2} \right] \\ \leq \mathbb {E} _ {W, W ^ {i}, Z ^ {\prime}} \left[ \left(\ell (W, Z ^ {\prime}) - \ell (W ^ {i}, Z ^ {\prime})\right) ^ {2} \right] \\ = \mathbb {E} _ {W, W ^ {i}, Z ^ {\prime}} \left[ \left(\ell (W, Z ^ {\prime}) - \ell \left(w ^ {*}, Z ^ {\prime}\right) + \ell \left(w ^ {*}, Z ^ {\prime}\right) - \ell \left(W ^ {i}, Z ^ {\prime}\right)\right) ^ {2} \right] \\ \leq 2 \mathbb {E} _ {W, Z ^ {\prime}} \left[ \left(\ell \left(W, Z ^ {\prime}\right) - \ell \left(w ^ {*}, Z ^ {\prime}\right)\right) ^ {2} \right] + 2 \mathbb {E} _ {W ^ {i}, Z ^ {\prime}} \left[ \left(\ell \left(W ^ {i}, Z ^ {\prime}\right) - \ell \left(w ^ {*}, Z ^ {\prime}\right)\right) ^ {2} \right] \\ \leq 4 B \mathbb {E} _ {W} \left[ \left(L _ {\mu} (W) - L _ {\mu} \left(w ^ {*}\right)\right) ^ {\frac {1}{\kappa}} \right], \\ \end{array}
$$
where the last inequality is by the definition of the Bernstein condition. This completes the proof.
E.2 Proof of Theorem 6.1
Proof. Let  $g(\Delta \ell_i, u_i) = (-1)^{u_i} \Delta \ell_i$ . Notice that  $|g(\Delta L_i, U_i')| \leq \beta_2$  and  $g(\Delta L_i, U_i')$  is again zero-mean, then
$$
\begin{array}{l} \mathbb {E} _ {\Delta L _ {i}, U _ {i}} \left[ g \left(\Delta L _ {i}, U _ {i}\right) \right] \leq \frac {I \left(\Delta L _ {i} ; U _ {i}\right) + \log \mathbb {E} _ {\Delta L _ {i} , U _ {i} ^ {\prime}} \left[ e ^ {t g \left(\Delta L _ {i} , U _ {i} ^ {\prime}\right)} \right]}{t} \\ \leq \beta_ {2} \sqrt {2 I (\Delta L _ {i} ; U _ {i})}. \\ \end{array}
$$
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\beta_ {2}}{n} \sum_ {i = 1} ^ {n} \sqrt {2 I (\Delta L _ {i} ; U _ {i})}.
$$
To prove the disintegrated CMI bound, we let  $g$  be defined in the same way, and the remaining development is the same with the proof in Theorem 4.5.
For the second inequality, notice that  $I(\Delta L_i; U_i) \leq I(\Delta L_i; U_i | \widetilde{Z}_i^+)$  by using the chain rule of mutual information and the independence between  $\widetilde{Z}_i^+$  and  $U_i$ . In addition, moving the expectation over  $\widetilde{Z}_i^+$  inside the square-root function by Jensen's inequality, we have  $\mathbb{E}_{\widetilde{Z}_i^+} \sqrt{I^{\widetilde{Z}_i^+}(\Delta L_i; U_i)} \leq \sqrt{I(\Delta L_i; U_i | \widetilde{Z}_i^+)}$ .
Proof. Before we prove Theorem 6.2, we first show the following lemma.
Lemma E.1. For any  $i\in [n]$ , we have  $\sum_{i = 1}^{n}I(F_i,\bar{F}_i;U_i|\widetilde{Z}_i^+) \leq I(F_{[n]},\bar{F}_{[n]};U|\widetilde{Z}_{[n]}^+)$ .
Proof of Lemma E.1. First, by  $I(F_i, \bar{F}_i; U_i | \widetilde{Z}_i^+) = H(U_i) - H(U_i | F_i, \bar{F}_i, \widetilde{Z}_i^+)$  and  $I(F_i, \bar{F}_i; U_i | \widetilde{Z}_{[n]}^+) = H(U_i) - H(U_i | F_i, \bar{F}_i, \widetilde{Z}_{[n]}^+)$ , we have
$$
I \left(F _ {i}, \bar {F} _ {i}; U _ {i} \mid \widetilde {Z} _ {i} ^ {+}\right) \leq I \left(F _ {i}, \bar {F} _ {i}; U _ {i} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right). \tag {35}
$$
Then, using the chain rule,
$$
I (F _ {i}, \bar {F} _ {i}; U _ {i} | \widetilde {Z} _ {[ n ]} ^ {+}) + I (F _ {[ n ] \backslash i}, \bar {F} _ {[ n ] \backslash i}; U _ {i} | \widetilde {Z} _ {[ n ]} ^ {+}, F _ {i}, \bar {F} _ {i}) = I (F _ {[ n ]}, \bar {F} _ {[ n ]}; U _ {i} | \widetilde {Z} _ {[ n ]} ^ {+}).
$$
By the non-negativity of mutual information, we have
$$
I \left(F _ {i}, \bar {F} _ {i}; U _ {i} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right) \leq I \left(F _ {[ n ]}, \bar {F} _ {[ n ]}; U _ {i} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right). \tag {36}
$$
Furthermore, by the independence of each  $U_{i}$  (i.e.  $I(U_{i};U_{[n]\backslash i}|\widetilde{Z}_{[n]}^{+}) = 0$ ), we have
$$
\sum_ {i = 1} ^ {n} I \left(F _ {[ n ]}, \bar {F} _ {[ n ]}; U _ {i} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right) \leq I \left(F _ {[ n ]}, \bar {F} _ {[ n ]}; U \mid \widetilde {Z} _ {[ n ]} ^ {+}\right). \tag {37}
$$
Combining Eq. (35-37) will conclude the proof.
We now prove Theorem 6.2.
For a given  $\widetilde{Z}_{[n]}$ , the number of distinct values of their predictions, denoted by  $k$ , are upper bounded by the growth function of  $\mathcal{F}$  evaluated at  $n$ ,
$$
k \leq \sum_ {i = 1} ^ {d} \left( \begin{array}{c} n \\ i \end{array} \right) \leq (\frac {e n}{d}) ^ {d},
$$
where the second inequality is by Sauer-Shelah lemma [53, 57] for  $n > d + 1$ .
$$
I \left(F _ {[ n ]}, \bar {F} _ {[ n ]}; U \mid \widetilde {Z} _ {[ n ]} ^ {+}\right) \leq H \left(F _ {[ n ]}, \bar {F} _ {[ n ]} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right) \leq H \left(F _ {[ n ]} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right) + H \left(\bar {F} _ {[ n ]} \mid \widetilde {Z} _ {[ n ]} ^ {+}\right) \leq 2 d \log \left(\frac {e n}{d}\right). \tag {38}
$$
By Jensen's inequality and Lemma E.1, we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \sqrt {I (F _ {i} , \bar {F} _ {i} ; U _ {i} | \widetilde {Z} _ {i} ^ {+})} \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} I (F _ {i} , \bar {F} _ {i} ; U _ {i} | \widetilde {Z} _ {i} ^ {+})} \leq \sqrt {\frac {I (F _ {[ n ]} , \bar {F} _ {[ n ]} ; U | \widetilde {Z} _ {[ n ]} ^ {+})}{n}}.
$$
Plugging Eq. (38) into the inequality above,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \sqrt {I (F _ {i} , \bar {F} _ {i} ; U _ {i} | \widetilde {Z} _ {i} ^ {+})} \leq \mathcal {O} \left(\sqrt {\frac {d}{n} \log \left(\frac {n}{d}\right)}\right),
$$
F CLB Examples
In Example 1, [21, Thm. 17] demonstrates the non-vanishing behavior of individual IOMI and e-CMI. This is primarily attributed to the dimension-dependent nature of IOMI and CMI. Specifically, there are certain dimensional settings where IOMI can grow faster than  $\mathcal{O}(n)$ , as shown in [21, Thm.4], and CMI approaches a certain fraction of its upper bound, as illustrated in Example 1, resulting in non-vanishing behavior. Specifically, in Example 1, [21] employs the birthday paradox [37, Sec. 5.1] problem to demonstrate that for a large value of  $d$ , the probability that no pair of instances in  $\widetilde{Z}$  sharing the same non-zero coordinate (referred to as event  $E_0$ ) is smaller than a constant probability (that could be independent of  $n$ ). Particularly, it is shown that if  $d \geq \frac{2n - 1}{1 - c^{1 / (2n - 1)}}$ , then  $P(E_0) \geq c \geq \left(1 - \frac{2n - 1}{d}\right)^{2n - 1}$ . As an example, [21] chooses  $d = 2n^2$ , resulting in  $c \geq 0.1$ .
Failure of  $I(W;Z_i)$  Consider the case where  $d = 2n^2$ . For the individual CMI [49, 70],  $I(W;U_i|\widetilde{Z}_i)$ , we have the following:
$$
I (W; U _ {i} | \widetilde {Z} _ {i}) = \log 2 - H (U _ {i} | W, \widetilde {Z} _ {i}) \geq 0. 1 \cdot \log 2.
$$
This inequality holds because when event  $E_0$  does not occur, one can determine the value of  $U_i$  completely, as the returned hypothesis is a weighted sum of the sample. In other words, examining the non-zero coordinates of  $W$  is sufficient to determine  $U_i$ . For an in-depth derivation of this inequality, readers are referred to the updated version of [21], where their corrected proof involves Fano's inequality. Furthermore, using the relation  $I(W; Z_i) \geq I(W; U_i | \widetilde{Z}_i)$  [70], we conclude that  $I(W; Z_i) \in \Omega(1)$ .
Failure of  $I(\widehat{Z}_i; U_i | \widetilde{W}_i)$ . Notably, our hypotheses-conditioned CMI also does not vanish for the same reason. More precisely, when  $\widetilde{W}_i$  and  $\widehat{Z}_i$  are given, there exists a constant probability (independent of  $n$ ) that allows us to fully determine the returned hypothesis based on  $\widehat{Z}_i$ , thereby determining the value of  $U_i$ .
Failure of  $I(\Delta L_i; U_i)$  Furthermore, even the loss difference based CMI (e.g., as shown in Theorem 6.1), which provides the tightest CMI measure, still does not vanish. This is attributed to the fact that if the hypothesis  $W$  is independent of certain  $Z$ , there exists a constant probability where the loss becomes zero (recall that the loss is the negative inner product of  $W$  and  $Z$ ). Consequently, one can determine the value of  $U_i$  by observing the sign of the random variable  $\Delta L_i$ . This also indicates the limitations of e-CMI and  $f$ -CMI in capturing the generalization behavior for Example 1.
In Example 2, following the approach in [21], the training sample  $S = \{Z_i\}_{i=1}^n \sim \mu^n$  can be represented as  $S = \frac{z_0}{R_0} (\varepsilon_1, \varepsilon_2, \ldots, \varepsilon_n)$ , where  $\{\varepsilon_i\}_{i=1}^n$  is a sequence of independent Rademacher random variables, i.e.,  $\varepsilon_i \sim \mathrm{Unif}(\{-1,1\})$ . The empirical risk is given by  $L_S(W) = -\frac{-L}{nR_0}\langle W, \sum_{i=1}^n \varepsilon_i \rangle$ . In this case, the ERM solution is  $W_{ERM} = z_0$  if  $\mathrm{sign}(\sum_{i=1}^n \varepsilon_i) = 1$ , and  $W_{ERM} = -z_0$  if  $\mathrm{sign}(\sum_{i=1}^n \varepsilon_i) = -1$ . It is clear that
$$
\sup  _ {w, w ^ {i}, z} \left| \ell (w, z) - \ell \left(w ^ {i}, z\right) \right| \leq \sup  _ {w, w ^ {i}, z} L | | w - w ^ {i} | | \leq 2 L R _ {0}.
$$
Hence, we observe that  $\beta_{2}$  is now a constant, whereas IOMI has an upper bound:  $\sum_{i=1}^{n} I(W; Z_{i}) \leq I(W; S) \leq I(W; \mathrm{sign}(\sum_{i=1}^{n} \varepsilon_{i})) \leq H(\mathrm{sign}(\sum_{i=1}^{n} \varepsilon_{i})) \leq 1$ , where the second inequality follows from the Markov chain  $S - \mathrm{sign}(\sum_{i=1}^{n} \varepsilon_{i}) - W$ . This provides us with a generalization bound of  $\frac{2LR_{0}}{\sqrt{n}}$ . Meanwhile, the actual generalization error satisfies  $\mathcal{E}_{\mu}(\mathcal{A}) \geq \frac{LR_{0}}{\sqrt{2n}}$  (see [21, AppendixB] for a derivation). Thus, the IOMI bound is tight up to a constant, and the stability bound  $\beta_{2}$  itself is vacuous. It is worth noting that  $I(\widehat{Z}_{i}; U_{i}|\widetilde{W}_{i}) \leq I(W; Z_{i}) \leq 1$  by Theorem4.2, indicating that the CMI bound is also tight.
We would like to note that the failures of chained mutual information bounds [2] are not demonstrated in the counterexamples presented in [21]. Notably, when the hypothesis is quantized, it becomes more challenging to guess  $U_{i}$  or  $Z_{i}$ . Therefore, exploring the potential of chained information-theoretic bounds, which do not necessarily rely on stability notions, could be another avenue to explain the generalization behavior observed in these counterexamples.
G Additional Applications
G.1 Compression Schemes
We now consider the algorithm that has a compression scheme [30]. Formally, a sample compression scheme of size  $k \in \mathbb{N}$  is a pair of maps  $(\mathcal{A}_1, \mathcal{A}_2)$ . Specifically, for all samples  $s$  with  $n > k$ ,  $\mathcal{A}_1: \mathcal{Z}^n \to \mathcal{Z}^k$  compresses the sample into a length-  $k$  subsequence  $\mathcal{A}_1(s) \subseteq s$ . Then  $\mathcal{A}_2: \mathcal{Z}^k \to \mathcal{W}$  could be some arbitrary mapping. Hence,  $\mathcal{A}(\cdot) = \mathcal{A}_2(\mathcal{A}_1(\cdot))$ . Let  $K$  be the index set for  $S$  selected by  $\mathcal{A}_1$ , and let  $\overline{K}$  be the selected index set for  $S^i$ . In this case, our supersample-conditioned CMI has an upper bound:  $I(W_i, \overline{W}_i; U_i | \widetilde{Z}_i^+) \leq I(K, \overline{K}; U_i | \widetilde{Z}_i^+) \leq H(K, \overline{K} | \widetilde{Z}_i^+) \leq 2\log \binom{n}{k} \leq 2k\log n$ . Then, if  $\mathcal{A}$  is further  $\beta_2$ -uniform stable, then we have the generalization bound  $\mathcal{E}_{\mu}(\mathcal{A}) \leq O(\beta_2\sqrt{k\log n})$ . If  $\beta_2 < O(1/\sqrt{n})$ , this bound improves the bound in [58]. It is unclear if we can obtain any improved bound for stable compression schemes [7], in which case [19] provides an optimal bound that removing the log  $n$  factor for the realizable setting. A main difficulty is that an interpolating algorithm is usually unstable due to the fitting-stability tradeoff [54, Sec. 13.4].
G.2 Distillation Algorithm
The high-probability generalization property of distillation algorithm is studied in [16]. In the first training stage of distillation, we obtain a  $w_{s}^{*}$  from a highly complex hypothesis space  $\mathcal{W}_1$  based on a training sample  $s$ . Same to [16], we assume that the first learning stage is  $\alpha$ -sensitive, namely  $||w_s^* -w_{s^i}^* ||\leq \alpha = \mathcal{O}(1 / n)$ . In the second stage, the algorithm  $\mathcal{A}$  will select a hypothesis that is  $\lambda$ -close to  $w_{s}^{*}$  from a less complex hypothesis space  $\mathcal{W}_2 = \{w\in \mathcal{W}:||w - w_s^* ||_\infty \leq \lambda \}$ . Let the loss function  $\ell$  be  $L$ -Lipschitz with respect to the first argument. Consequently,  $\gamma_3\leq L||w_s^* -w_{s^i}^* ||\leq L\alpha$ . Then, by Theorem 4.1, we have  $\mathcal{E}_{\mu}(\mathcal{A})\leq L\alpha \frac{1}{n}\sum_{i = 1}^{n}\sqrt{2H(U_i)} = \sqrt{2\log 2} L\alpha$ . Notice that the loss here may not necessarily be bounded or sub-Gaussian, rendering previous bounds inapplicable.
G.3 Regularized Empirical Risk Minimization
Regularized Empirical Risk Minimization (ERM) learning rules involve minimizing the empirical risk and a regularization function jointly:  $\arg \min_{w\in \mathcal{W}}L_S(w) + f_{\mathrm{reg}}(w)$ , where  $f_{\mathrm{reg}}:\mathcal{W}\to \mathbb{R}$ . Here we specifically consider Tikhonov regularization [54], namely  $f_{\mathrm{reg}}(w) = \lambda ||w||^2$ , where  $\lambda >0$  is a tradeoff coefficient. The regularized ERM algorithm  $\mathcal{A}$  aims to find
$$
w = \arg \min  _ {w \in \mathcal {W}} L _ {S} (w) + \lambda | | w | | ^ {2}.
$$
This regularization term ensures strong convexity of the training objective. Based on Theorem 4.3, we can derive the following results.
Corollary G.1. Assume that the loss function  $\ell$  is convex and  $L$ -Lipschitz. Then, for the regularized ERM algorithm with Tikhonov regularization, we have
$$
\left| \mathcal {E} _ {\mu} (\mathcal {A}) \right| \leq \frac {2 L ^ {2}}{\lambda n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} I (\widehat {Z}; U _ {i} | \widetilde {W _ {i}}) + 0. 7 2\right).
$$
Proof of Corollary G.1. By invoking [54, Corollary 13.6], we know that  $\gamma_4 \leq \beta_2 = \frac{2L^2}{\lambda n}$ . Plugging the value of  $\beta_{2}$  will give us the desired result.
Corollary G.2. Assume that the loss function is  $\rho$ -smooth and nonnegative. Let  $\lambda \geq \frac{2\rho}{n}$ . Then, for the regularized ERM algorithm with Tikhonov regularization, we have
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {4 8 \rho \hat {L} _ {n}}{\lambda n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} I (\widehat {Z}; U _ {i} | \widetilde {W _ {i}}) + 0. 7 2\right).
$$
Proof of Corollary G.2. By invoking [54, Corollary 13.7], we know that  $\gamma_4 \leq \beta_2 = \frac{48\rho\tilde{L}_n}{\lambda n}$ . Plugging the value of  $\beta_{2}$  will give us the desired result.
Although these bounds do not enhance the convergence rate of  $\mathcal{O}(1 / n)$  in these settings, they consistently offer tighter results compared to uniform stability-based bounds if  $\frac{1}{n}\sum_{i = 1}^{n}I(\widehat{\widetilde{Z}};U_i|\widetilde{W}_i)\leq$  0.28. In addition, the expected empirical risk  $\hat{L}_n$  appears in the bound of Corollary G.2. While  $\lambda$  has a lower bound,  $\hat{L}_n$  could not be arbitrarily small for the regularized ERM.
Notice that previous information-theoretic bounds could not obtain the convergence rate of  $\mathcal{O}(1 / n)$  as in our results unless ICMI or CMI itself decays with  $\mathcal{O}(1 / n)$ .
H Additional Discussions and Open Problems
Stochastic Gradient Descent (SGD) Since the influential work of [22], stability approaches have been widely employed to provide generalization guarantees for (sub)gradient-based optimization algorithms, such as SGD, under certain conditions like the convex-smooth-Lipschitz loss. More recently, [5] extended the results of [22] to the non-smooth loss function in the SCO setting.
In contrast, information-theoretic (weight/hypothesis-based) bounds are typically used to analyze the noisy version of SGD, known as SGLD [43, 40, 18, 49, 60]. Directly analyzing SGD poses challenges because the returned hypothesis  $W$  contains a significant amount of information about  $S$  or  $Z_{i}$ , resulting in potentially large (even infinite) mutual information. The prevalent approach to applying information-theoretic bounds to SGD is by introducing noise [41, 61], but this has been shown to yield non-vanishing bounds in [21, Thm. 4].
The combination of information-theoretic bounds with stability for analyzing the generalization of SGD presents a promising future direction. However, some potential difficulties may arise. For instance, if we continue to use the Gaussian noise perturbation method for the weight-based information-theoretic bounds, we would need to characterize the stability property for the perturbed SGD, which might require techniques employed in [38]. Additionally, when combining stability notions with loss difference based CMI (or e-CMI/f-CMI) bounds, as they cannot be unrolled using the chain rule and data processing inequality as in the case of weight-based IOMI/CMI bounds, it may not be possible to bound such CMI terms using trajectory-based quantities. This raises doubts about the potential for obtaining more informative generalization bounds compared to the stability-based bounds themselves.
Generalization Bounds beyond Mutual Information In the information-theoretic literature, it is common to replace mutual information with alternative distributional measures, such as Wasserstein distance based bounds and total variation based bounds [50]. A promising future direction is to incorporate the stability property of algorithms into these bounds, as demonstrated in this work. It is worth noting that obtaining KL divergence-based bounds should be straightforward since they rely on the same foundational Lemma A.2 as discussed in this paper.