SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Leave-One-Out CMI Our construction of  $\widetilde{W}$  bears resemblance to the leave-one-out (LOO) setting, where there are also  $n + 1$  hypotheses. It is worth noting that LOO-CMI has been recently proposed in concurrent works [20, 47]. In LOO-CMI, the supersample  $\widetilde{Z} = Z_{[n + 1]}$  consists of  $n + 1$  instances, and  $U$  is an index uniformly drawn from  $[n + 1]$  to select one hold-out instance. Consequently,  $Z_U$  represents the testing data, while  $Z_{[n + 1] \setminus U}$  serves as the training sample. In this context, LOO-CMI can be defined as  $I(W; U | \widetilde{Z})$ . Notice that this quantity still fails to explain the generalization in Example 1. The LOO setting is often associated with the stability-based framework [6], and it is expected that the  $n + 1$ -supersample induced  $\widetilde{W}$  could yield new CMI bounds that also contain the SCH stability notions. Nevertheless, in this paper, we do not adopt the LOO setting because in that case,  $H(U) = \log (n + 1)$ , the LOO-CMI bound is no longer upper bounded by a constant independent of  $n$  (note that the LOO-CMI bound for general setting in [20, Thm. 2.5] does not contain the  $1 / \sqrt{n}$  factor).
We propose a novel construction of the hypothesis matrix and a new family of stability notions called sample-conditioned hypothesis stability. Leveraging these concepts, we derive sharper information-theoretic bounds for stable learning algorithms. Several promising avenues for future research include comparing our new CMI quantities with the standard CMI in a quantitative manner, analyzing the generalization of gradient-based optimization algorithms like SGD using our bounds, and establishing new high-probability generalization guarantees. Further discussions can be found in Appendix H.
This work is supported partly by an NSERC Discovery grant. Ziqiao Wang is also supported in part by the NSERC CREATE program through the Interdisciplinary Math and Artificial Intelligence (INTER-MATH-AI) project. The authors would like to thank the anonymous AC and reviewers for their careful reading and valuable suggestions.
[1] Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv preprint arXiv:2110.11216, 2021.
[2] Amir Asadi, Emmanuel Abbe, and Sergio Verdu. Chaining mutual information and tightening generalization bounds. Advances in Neural Information Processing Systems, 31, 2018.
[3] Arindam Banerjee, Tiancong Chen, Xinyan Li, and Yingxue Zhou. Stability based generalization bounds for exponential family Langevin dynamics. In International Conference on Machine Learning, pages 1412-1449. PMLR, 2022.
[4] Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory. PMLR, 2018.
[5] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33, 2020.
[6] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[7] Olivier Bousquet, Steve Hanneke, Shay Moran, and Nikita Zhivotovsky. Proper learning, helly number, and an optimal svm bound. In Conference on Learning Theory, pages 582-609. PMLR, 2020.
[8] Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovsky. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory, pages 610-626. PMLR, 2020.
[9] Yuheng Bu, Shaofeng Zou, and Venugopal V Veeravalli. Tightening mutual information based bounds on generalization error. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 587-591. IEEE, 2019.
[10] O Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning. Vol. 56. Lecture Notes - Monograph Series. Institute of Mathematical Statistics, 2007.
[11] Luc Devroye and Terry Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Transactions on Information Theory, 25(2):202-207, 1979.
[12] Luc Devroye and Terry Wagner. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory, 25(5):601-604, 1979.
[13] Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil, and Leslie Pack Kaelbing. Stability of randomized learning algorithms. Journal of Machine Learning Research, 6(1), 2005.
[14] Vitaly Feldman and Jan Vondrak. Generalization bounds for uniformly stable algorithms. Advances in Neural Information Processing Systems, 31, 2018.
[15] Vitaly Feldman and Jan Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory, pages 1270-1279. PMLR, 2019.
[16] Dylan J Foster, Spencer Greenberg, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Hypothesis set stability and generalization. Advances in Neural Information Processing Systems, 32, 2019.
A Some Useful Lemmas
In this paper, there are some equivalent forms of the generalization error we will study, e.g., Eq. (2) and Eq. (5) in the main text, which are presented in the following lemma.
Lemma A.1. Let  $W_{i} = \widetilde{W}_{i,U_{i}}$  and  $\overline{W}_i = \widetilde{W}_{i,\overline{U}_i}$ . For any learning algorithm  $\mathcal{A}$ , the following equations hold
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}, \widetilde {W} _ {i}} \left[ \ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {+}\right) \right], (12) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} | \widetilde {W} _ {i}} \left[ (- 1) ^ {U _ {i}} \left(\ell \left(\widetilde {W} _ {i} ^ {-}, \widehat {Z} _ {i}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widehat {Z} _ {i}\right)\right) \right] \right], (13) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \mathbb {E} _ {W _ {i}, \overline {{W}} _ {i}, U _ {i} | \widetilde {Z} _ {i} ^ {+}} \left[ (- 1) ^ {U _ {i}} \left(\ell (\overline {{W}} _ {i}, \widetilde {Z} _ {i} ^ {+}) - \ell (W _ {i}, \widetilde {Z} _ {i} ^ {+})\right) \right] \right]. (14) \\ \end{array}
$$
Proof. This lemma is a consequence of Lemma 2.1, with further utilizing some symmetric properties. Recall Eq. (1) in Lemma 2.1,
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\mathcal {A}) = \mathbb {E} _ {\widetilde {Z} _ {[ n ]} ^ {+}, \widetilde {Z} _ {[ n ]} ^ {-}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \mathbb {E} _ {\widetilde {W} _ {i} ^ {-} | \widetilde {Z} _ {[ n ]} ^ {+}, \widetilde {Z} _ {i} ^ {-}} \ell (\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}) - \mathbb {E} _ {\widetilde {W} ^ {+} | \widetilde {Z} _ {[ n ]} ^ {+}} \ell (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right] \right], \\ = \mathbb {E} _ {\widetilde {Z} _ {[ n ]} ^ {+}, \widetilde {W}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \ell (\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}) - \ell (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right] \right], \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}, \widetilde {W} _ {i}} \left[ \ell (\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}) - \ell (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right]. \\ \end{array}
$$
Note that Eq. (2) in the main text is from the second equation above, which is used to derive individual IOMI bounds in Section 3.
Similar to the standard setting for CMI bounds, where the role of each  $\widetilde{Z}_i^+$  and  $\widetilde{Z}_i^-$  can be exchanged, a key observation here is that for each  $i$ ,  $\widetilde{W}_i^+$  and  $\widetilde{W}_i^-$  can also be exchanged arbitrarily. That is to say,
$$
\mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {-}, \widetilde {W} _ {i}} \left[ \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {-}\right) - \ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {-}\right) \right] \tag {15}
$$
also holds true. Notice that we do not change the definitions of any the random variable, e.g.,  $\widetilde{W}^{+} = \mathcal{A}(\widetilde{Z}_{[n]}^{+},R)$  and  $\widetilde{W}_i^{-} = \mathcal{A}(\widetilde{Z}_{[n]\sim i}^{+},R)$ .
What differs from the standard CMI is that the roles of the whole sequences  $\widetilde{Z}_{[n]}^{+}$  and  $\widetilde{Z}_{[n]}^{-}$  are not exchangeable with each other. Here, when we exchange each  $\widetilde{Z}_i^+$  and  $\widetilde{Z}_i^-$ , we need to keep the other positions in  $S$  unchanged.
By introducing  $U_{i} \sim \mathrm{Unif}(\{0,1\})$ , we have
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}, \widetilde {W} _ {i}} \left[ \ell (\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+}) - \ell (\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right], \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i}, \widetilde {W} _ {i}, U _ {i}} \left[ \ell (\widetilde {W} _ {i, \overline {{U}} _ {i}}, \widetilde {Z} _ {i, U _ {i}}) - \ell (\widetilde {W} _ {i, U _ {i}}, \widetilde {Z} _ {i, U _ {i}}) \right]. \\ \end{array}
$$
To obtain Eq. (13), notice that  $\widehat{Z}_i = \widetilde{Z}_{i,U_i}$  , we have
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widehat {Z} _ {i}, \widetilde {W} _ {i}, U _ {i}} \left[ \ell (\widetilde {W} _ {i, \overline {{U}} _ {i}}, \widehat {Z} _ {i}) - \ell (\widetilde {W} _ {i, U _ {i}}, \widehat {Z} _ {i}) \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widehat {Z} _ {i}, \widetilde {W} _ {i}, U _ {i}} \left[ (- 1) ^ {U _ {i}} \left(\ell \left(\widetilde {W} _ {i} ^ {-}, \widehat {Z} _ {i}\right) - \ell \left(\widetilde {W} _ {i} ^ {+}, \widehat {Z} _ {i}\right)\right) \right]. \tag {16} \\ \end{array}
$$
This, as we have already seen in Eq. (5) in the main text, is used to derive hypotheses-conditioned CMI bounds in Section 4. It's easy to see that when  $U_{i} = 0$ , Eq. (16) becomes Eq. (12), and when  $U_{i} = 1$ , we obtain Eq. (15) via Eq. (16).
To obtain Eq. (14), we let  $W_{i} = \widetilde{W}_{i,U_{i}}$ ,  $\overline{W}_i = \widetilde{W}_{i,\overline{U}_i}$ , and fix  $\widehat{Z}_i = \widetilde{Z}_i^+$ . Similarly,
$$
\mathcal {E} _ {\mu} (\mathcal {A}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {Z} _ {i} ^ {+}} \left[ \mathbb {E} _ {W _ {i}, \overline {{W}} _ {i}, U _ {i} | \widetilde {Z} _ {i} ^ {+}} \left[ (- 1) ^ {U _ {i}} \left(\ell (\overline {{W}} _ {i}, \widetilde {Z} _ {i} ^ {+}) - \ell (W _ {i}, \widetilde {Z} _ {i} ^ {+})\right) \right] \right].
$$
This is used to derive supersample-conditioned CMI bounds in Section 4. It's easy to see that both  $U_{i} = 0$  and  $U_{i} = 1$  will give us Eq. (12).
Like all the previous information-theoretic bounds, the following lemma is widely used in our paper.
Lemma A.2 (Donsker-Varadhan (DV) variational representation of KL divergence [44, Theorem 3.5]). Let  $Q$ ,  $P$  be probability measures on  $\Theta$ , for any bounded measurable function  $f: \Theta \to \mathbb{R}$ , we have  $\mathrm{D}_{\mathrm{KL}}(Q||P) = \sup_f \mathbb{E}_{\theta \sim Q}[f(\theta)] - \ln \mathbb{E}_{\theta \sim P}[\exp f(\theta)]$ .
We also invoke some other lemmas as given below.
Lemma A.3 (Hoeffding's Lemma [26]). Let  $X \in [a, b]$  be a bounded random variable with mean  $\mu$ . Then, for all  $t \in \mathbb{R}$ , we have  $\mathbb{E}\left[e^{tX}\right] \leq e^{t\mu + \frac{t^2(b - a)^2}{8}}$ .
Lemma A.4 (Popoviciu's inequality [45]). Let  $M$  and  $m$  be upper and lower bounds on the values of any random variable  $X$ , then  $\operatorname{Var}(X) \leq \frac{(M - m)^2}{4}$ .
The following lemma is from [35, Lemma 2.8], we provide a self-contained proof.
Lemma A.5. Let  $h(x) = \frac{e^x - x - 1}{x^2}$  be the Bernstein function. If a random variable  $X$  satisfies  $\mathbb{E}[X] = 0$  and  $X \leq b$ , then  $\mathbb{E}\left[e^{X}\right] \leq e^{h(b)\mathbb{E}\left[X^2\right]}$ .
Proof. It's easy to verify that  $h(x)$  is an increasing function for  $x > 0$ . Thus,  $h(x) \leq h(b)$  for  $x \leq b$ . Then,
$$
e ^ {x} = x + 1 + x ^ {2} h (x) \leq x + 1 + x ^ {2} h (b).
$$
For the bounded random variable  $X$  with zero mean, we have
$$
\mathbb {E} \left[ e ^ {X} \right] \leq \mathbb {E} [ X ] + 1 + \mathbb {E} \left[ X ^ {2} h (b) \right] \leq e ^ {h (b) \mathbb {E} \left[ X ^ {2} \right]}.
$$
The last inequality is by  $e^x \geq x + 1$ . This completes the proof.
B Further Elaborations on SCH Stability
We note that the reason we introduce four types of SCH stability in Definition 2.1 is that solely using  $\beta_{2}$  in our bounds might be too loose, as it considers the supremum over all sources of randomness. By incorporating SCH stabilities, we aim to demonstrate that theoretically, we can achieve significantly tighter stability parameters.
The basic set up is as follows. Assume a random sample  $S$  gives rise to  $W$ . For each  $Z_{i} \in S$ , we construct  $S^{i}$  by replacing  $Z_{i}$  with another independently drawn instance; call training result  $W^{i}$ , the neighbor of  $W$ .
In a),  $\gamma_{1}$ -SCH-A stability measures the difference between the loss of  $w$  and the expected loss of its neighbor  $W^{i}$  at a worst  $z$  and the worst possible  $w$ . While in (b),  $\gamma_{2}$ -SCH-B stability measures the
square of this difference, not in the worst case, but in an average case, where the average is over an independently  $Z'$  for the loss evaluation, the training sample, and the algorithm randomness. Since "average is smaller than worst",  $\gamma_2 \leq \gamma_1$ .
In c), we consider the difference between the loss of  $W$  and the loss of its neighbor when evaluated at the worst possible  $Z_{i}$  that when included in  $S$  gives rise to  $W$ . The expected value of this difference is  $\gamma_{3}$ -SCH-C stability.
In d),  $\gamma_4$ -SCH-D stability measures the expected squared difference between the loss of  $W$  and the loss of its neighbor when evaluated at  $Z_{i}$  (a member of  $S$ ). For a similar "average smaller worst" reason, one expects that  $\gamma_4 \leq \gamma_3$ .
We expect that  $\gamma_{2},\gamma_{3}$ , and  $\gamma_{4}$  are all smaller than  $\beta_{1}$ . This is because in  $\beta_{1}$ , we consider the worst evaluated instance, whereas in the other cases, we take the expectation over all instances. Additionally, in Theorem 4.1, we expect that  $\mathbb{E}_{\widetilde{W}_i}\Delta_1(\widetilde{W}_i)^2\leq \beta_1^2$ , this is because  $\beta_{1}$ -stability holds for all the possible  $s$  and  $s^i$ , namely it holds for all the  $(w,w^{i})$  pair (that shares the same randomness) while in  $\mathbb{E}_{\widetilde{W}_i}\Delta_1(\widetilde{W}_i)^2$ , we take the expectation of these pairs.
We expect  $\gamma_{2} \leq \gamma_{4}$  due to the following reason: first by Jensen's inequality, we have  $\mathbb{E}_{S,R,Z'}\left[\ell(W,Z') - \mathbb{E}_{W^i|W}\ell(W^i,Z')\right]^2 \leq \mathbb{E}_{W,W^i,Z'}\left[\ell(W,Z') - \ell(W^i,Z')\right]^2$ , then since  $Z'$  is an independent of both  $W$  and  $W'$ ,  $Z'$  can be regarded as a testing point for both  $W$  and  $W'$ , we could expect that the expectation of  $\ell(W,Z') - \ell(W^i,Z')$  is small. While in  $\mathbb{E}_{S,Z_i',R}\left[\ell(W,Z_i) - \ell(W^i,Z_i)\right]^2$ ,  $Z_i$  is a training point for obtaining  $W$ , so  $\ell(W,Z_i)$  could be small in general, and  $Z_i$  is a testing point for  $W^i$ . Therefore, it is reasonable to expect  $\mathbb{E}_{W,W^i,Z'}\left[\ell(W,Z') - \ell(W^i,Z')\right]^2 \leq \mathbb{E}_{S,Z_i',R}\left[\ell(W,Z_i) - \ell(W^i,Z_i)\right]^2$ , namely  $\gamma_{2} \leq \gamma_{4}$ .
As a concrete example, let  $\ell$  be zero-one loss and assume  $\mathcal{A}$  is an interpolating algorithm and randomly makes predictions for unseen data. By Jensen's inequality,  $\gamma_2^2 \leq \mathbb{E}_{W,W^i,Z'}\left[\ell(W,Z') - \ell(W^i,Z')\right]^2 = \mathbb{E}_{W,Z'}\left[\ell(W,Z')\right] - 2\mathbb{E}_{W,W^i,Z'}\left[(\ell(W,Z')\ell(W^i,Z'))\right] + \mathbb{E}_{W^i,Z'}\left[\ell(W^i,Z')\right]^2$ , where we use  $\ell^2 = \ell$  for zero-one loss. Since  $Z'$  is an unseen data for both  $W$  and  $W^i$ , we have  $\gamma_2^2 \leq \mathbb{E}_{W^i,Z'}\left[\ell(W^i,Z')\right]^2 + \frac{1}{2} - \frac{1}{2} = \mathbb{E}_{W^i,Z'}\left[\ell(W^i,Z')\right]^2$ . While in this case  $\gamma_4^2 = \mathbb{E}_{W^i,Z_i}\left[\ell(W^i,Z_i)\right]^2$  so  $\gamma_2 \leq \gamma_4$ .
C Omitted Proofs and Additional Discussions in Section 3
Proof. Let  $g(\tilde{w}^{+},\tilde{z}_{i}^{+}) = \mathbb{E}_{\widetilde{W}_{i}^{-}|\tilde{w}^{+}}\left[\ell (\widetilde{W}_{i}^{-},\tilde{z}_{i}^{+})\right] - \ell (\tilde{w}^{+},\tilde{z}_{i}^{+})$  be the average loss difference between  $\tilde{w}^{+}$  and its neighboring hypothesis, and let  $f = t\cdot g$  for  $t > 0$  in Lemma A.2. Let  $\widetilde{Z}_i^{+'}$  be an independent copy of  $\widetilde{Z}_i^+$ , then
$$
\mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}} \left[ g \left(\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}\right) \right] \leq \inf  _ {t > 0} \frac {I \left(\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+}\right) + \log \mathbb {E} _ {\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime}} \left[ e ^ {t g \left(\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime}\right)} \right]}{t}. \tag {17}
$$
Since  $\widetilde{Z}_i^{+'}$  is independent of both  $\widetilde{W}_i^-$  and  $\widetilde{W}^+$ , and  $\widetilde{W}_i^-$  and  $\widetilde{W}^+$  are identically distributed, we have
$$
\mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ ^ {\prime}}} \left[ g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ ^ {\prime}}) \right] = \mathbb {E} _ {\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+ ^ {\prime}}} \left[ \ell (\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+ ^ {\prime}}) \right] - \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ ^ {\prime}}} \left[ \ell (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ ^ {\prime}}) \right] = 0.
$$
By the definition of  $\gamma_{1}$ -SCH-A stability,
$$
\sup _ {\tilde {w} ^ {+}, z} \left| \mathbb {E} _ {\widetilde {W} _ {i} ^ {-}} | _ {\tilde {w} ^ {+}} \left[ \ell (\widetilde {W} _ {i} ^ {-}, z) \right] - \ell (\tilde {w} ^ {+}, z) \right| \leq \gamma_ {1},
$$
so  $g(\widetilde{W}^{+}, \widetilde{Z}_{i}^{+^{\prime}})$  is a zero-mean random variable bounded in  $[-\gamma_1, \gamma_1]$ . By Lemma A.3, we have
$$
\begin{array}{l} \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}} \left[ g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right] \leq \inf  _ {t > 0} \frac {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+}) + \log \mathbb {E} _ {\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime}} \left[ e ^ {t g (\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime})} \right]}{t} \\ \leq \inf  _ {t > 0} \frac {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+}) + \frac {t ^ {2} \gamma_ {1} ^ {2}}{2}}{t} \\ = \sqrt {2 \gamma_ {1} ^ {2} I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+})}, \\ \end{array}
$$
where the last equality is obtained by optimizing the bound over  $t$ , i.e. letting  $t = \sqrt{\frac{I(\widetilde{W}^{+};\widetilde{Z}_{i}^{+})}{2\gamma_{1}^{2}}}$ .
Recall Eq. (12) in Lemma A.1 and applying Jensen's inequality to the absolute function, the first bound is then obtained by
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}} \left[ g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right] \right| \leq \frac {\gamma_ {1}}{n} \sum_ {i = 1} ^ {n} \sqrt {2 I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+})},
$$
Furthermore, by the chain rule of mutual information,
$$
I \left(\widetilde {W} _ {i} ^ {-}; \widetilde {Z} _ {i} ^ {+} \mid \widetilde {W} ^ {+}\right) + I \left(\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+}\right) = I \left(\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+} \mid \widetilde {W} _ {i} ^ {-}\right) + I \left(\widetilde {W} _ {i} ^ {-}; \widetilde {Z} _ {i} ^ {+}\right). \tag {18}
$$
Notice that  $I(\widetilde{W}_i^-;\widetilde{Z}_i^+) = 0$  in the RHS, we have
$$
I \left(\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+}\right) \leq I \left(\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+} \mid \widetilde {W} _ {i} ^ {-}\right),
$$
which will give us the second bound. This concludes the proof.
Remark C.1 (Comparison with Mutual Information Stability [46, 23]). To compare with the mutual information stability  $I\left(\widetilde{W}^{+}; \widetilde{Z}_{i}^{+} | \widetilde{Z}_{[n] \setminus i}^{+}\right)$ , recall Eq.(18):  $I(\widetilde{W}^{+}; \widetilde{Z}_{i}^{+} | \widetilde{W}^{-}) = I(\widetilde{W}_{i}^{-}; \widetilde{Z}_{i}^{+} | \widetilde{W}^{+}) + I(\widetilde{W}^{+}; \widetilde{Z}_{i}^{+})$ , and similarly we also have  $I(\widetilde{W}^{+}; \widetilde{Z}_{i}^{+} | \widetilde{Z}_{[n] \setminus i}^{+}) = I(\widetilde{Z}_{[n] \setminus i}^{+}; \widetilde{Z}_{i}^{+} | \widetilde{W}^{+}) + I(\widetilde{W}^{+}; \widetilde{Z}_{i}^{+})$ .
Thus, we only need to compare  $I(\widetilde{W}_i^-; \widetilde{Z}_i^+ | \widetilde{W}^+)$  and  $I(\widetilde{Z}_{[n] \setminus i}^+; \widetilde{Z}_i^+ | \widetilde{W}^+)$ . Notice that for a deterministic  $\mathcal{A}$ , we have  $I(\widetilde{W}_i^-; \widetilde{Z}_i^+ | \widetilde{W}^+) \leq I(\widetilde{Z}_{[n] \setminus i}^+; \widetilde{Z}_i^-; \widetilde{Z}_i^+ | \widetilde{W}^+)$ . Since  $\widetilde{Z}_i^- \perp \downarrow (\widetilde{W}^+, \widetilde{Z}_{[n]}^+)$ , we further have  $I(\widetilde{Z}_{[n] \setminus i}^+, \widetilde{Z}_i^-; \widetilde{Z}_i^+ | \widetilde{W}^+) = I(\widetilde{Z}_{[n] \setminus i}^+; \widetilde{Z}_i^+ | \widetilde{W}^+)$ , which gives us the desired result:
$$
I \left(\widetilde {W} _ {i} ^ {+}; \widetilde {Z} _ {i} ^ {+} | \widetilde {W} ^ {-}\right) \leq I \left(\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+} | \widetilde {Z} _ {[ n ] \backslash i} ^ {+}\right).
$$
Proof. The proof is nearly the same to the proof of Theorem 3.1, except that now the randomness of the algorithm is given for each DV auxiliary function, so the randomness of  $\widetilde{W}_i$  is completely controlled by  $\widetilde{Z}$ .
Let  $g(\tilde{w}^{+},\tilde{z}_{i}^{+},r) = \mathbb{E}_{\widetilde{W}_{i}^{-}|\tilde{w}^{+},r}\left[\ell (\widetilde{W}_{i}^{-},\tilde{z}_{i}^{+})\right] - \ell (\tilde{w}^{+},\tilde{z}_{i}^{+})$  and let  $f = t\cdot g$  for  $t > 0$  in Lemma A.2. Let  $\widetilde{Z}_i^{+'}$  be an independent copy of  $\widetilde{Z}_i^+$ , then
$$
\mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+} | r} \left[ g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}, r) \right] \leq \inf _ {t > 0} \frac {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+} | R = r) + \log \mathbb {E} _ {\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime} | r} \left[ e ^ {t g (\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime} , r)} \right]}{t}.
$$
$$
\mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime} | r} \left[ g \left(\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime}, r\right) \right] = \mathbb {E} _ {\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+ \prime} | r} \left[ \ell \left(\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+ \prime}\right) \right] - \mathbb {E} _ {\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime} | r} \left[ \ell \left(\widetilde {W} _ {i} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime}\right) \right] = 0
$$
still holds since  $\widetilde{Z}_i^+$  and  $\widetilde{Z}_i^-$  are i.i.d. drawn.
Thus,  $g(\widetilde{W}^{+}, \widetilde{Z}_{i}^{+^{\prime}}, r)$  is a zero-mean random variable bounded in  $[-\beta_{2}, \beta_{2}]$ . By Lemma A.3, the remaining part is routine: