SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem 2.2. Let  $\mathcal{D} \in \operatorname{Realizable}(\mathcal{H})$  and  $m \in \mathbb{N}$ . Learning rule  $A^{\star}$  operates as follows. Given a sample  $S = \{(x_1, y_1), \ldots, (x_m, y_m)\}$ ,  $A^{\star}$  simulates an online learning game, in which  $S$  is the set of 'experts',  $\mathcal{F} = \{0, 1\}^{\mathcal{X}}$  is the set of 'instances', and the learner's utility for playing expert  $(x, y)$  on instance  $f \in \mathcal{F}$  is  $\mathbb{1}(f(x) \neq y)$ . Namely, in this game the learner is attempting to select an  $(x, y)$  pair that disagrees with the instance  $f$ .
In this simulation, the learner executes an instance of the online learning algorithm of Appendix A.2 with expert set  $S$ . Denote this instance  $\mathcal{O}_S$ .
The adversary's strategy is as follows. Recall that at each round  $t$ ,  $\mathcal{O}_S$  chooses a distribution  $w_t$  over  $S$ . Note that if  $S$  is realizable then so is  $w_t$ . At each round  $t$ , the adversary selects an instance  $f \in \mathcal{F}$  by executing  $A$  on a training set sampled from  $w_t$ , as in Algorithm 1.
We prove the following:
1.  $A^{\star}$  interpolates, namely  $\mathbb{P}[\mathrm{L}_S(A^\star (S)) = 0] = 1$
2.  $A^{\star}$  has logarithmic KL-stability, as in Eq. (8).
3.  $A^{\star}$  PAC learns  $\mathcal{H}$  as in Eq. (9).
For Item 1, assume for contradiction that  $A^{\star}$  does not interpolate. Seeing as  $A^{\star}$  outputs  $\mathsf{Maj}(f_1,\ldots ,f_T)$ , there exists an index  $i\in [m]$  such that
$$
\frac {T}{2} \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left(f _ {t} \left(x _ {i}\right) \neq y _ {i}\right) = U (i, T), \tag {10}
$$
where  $U(i, T)$  is the utility of always playing expert  $i$  throughout the game.
Let  $\mathcal{E}_t$  denote the event that  $S_{t}$  was resampled (i.e., there were multiple iterations of the do-while loop in round  $t$ ). Eq. (7) and Markov's inequality imply
$$
\mathbb {P} \left[ \mathcal {E} _ {t} \right] = \mathbb {P} \left[ \mathrm {K L} \left(A \left(S _ {t}\right) \| \mathcal {P}\right) \geq 2 b / \gamma \right] \leq \gamma / 2. \tag {11}
$$
The utility of  $\mathcal{O}_S$  at time  $t$  is
$$
\begin{array}{l} u _ {t} ^ {\mathcal {O} _ {S}} = \underset {S _ {t} \sim (w _ {t}) ^ {k}} {\mathbb {E}} [ \mathbb {1} (f _ {t} (x) \neq y) ] \\ \begin{array}{l} f _ {t} \sim A (S _ {t}) \\ (x, y) \sim w _ {t} \end{array} \\ \leq \underset {S _ {t} \sim (w _ {t}) ^ {k}} {\mathbb {E}} \left[ \mathrm {L} _ {w _ {t}} \left(A (S _ {t})\right) \mid \neg \mathcal {E} _ {t} \right] + \mathbb {P} [ \mathcal {E} _ {t} ] \leq \left(\frac {1}{2} - \gamma\right) + \frac {\gamma}{2}, \\ \end{array}
$$
where the last inequality follows from Eqs. (6) and (11). Hence, the utility of  $\mathcal{O}_S$  throughout the game is
$$
U \left(\mathcal {O} _ {S}, T\right) = \sum_ {t = 1} ^ {T} u _ {t} ^ {\mathcal {O} _ {S}} \leq \left(\frac {1}{2} - \frac {\gamma}{2}\right) \cdot T. \tag {12}
$$
Combining Eqs. (10) and (12) and Theorem A.6 yields
$$
\frac {\gamma}{2} \cdot T \leq U (i, T) - U (\mathcal {O} _ {S}, T) \leq \operatorname {R e g r e t} (\mathcal {O} _ {S}, T) \leq \sqrt {2 T \log (m)},
$$
which is a contradiction for our choice of  $T$ . This establishes Item 1.
For Item 2, for every  $\ell \in \mathbb{N}$  let  $\mathcal{P}_{\ell}^{\star} \in \Delta\big(\{0,1\}^{\mathcal{X}}\big)$  be the distribution of  $\mathrm{Maj}(g_1,\ldots,g_\ell)$ , where  $(g_1,\ldots,g_\ell) \sim \mathcal{P}^\ell$ . Let  $\mathcal{P}^{\star} = \frac{1}{z}\sum_{\ell=1}^{\infty}\mathcal{P}_{\ell}^{\star}/\ell^{2}$  where  $z = \sum_{\ell=1}^{\infty}1/\ell^{2} = \pi^{2}/6$  is a normalization factor.
For any  $S\in (\mathcal{X}\times \{0,1\})^m$
$$
\begin{array}{l} \operatorname {K L} \left(A ^ {\star} (S) \| \mathcal {P} _ {T} ^ {\star}\right) = \operatorname {K L} (\operatorname {M a j} \left(f _ {1}, \dots , f _ {T}\right) \| \operatorname {M a j} \left(g _ {1}, \dots , g _ {T}\right)) \\ \leq \mathrm {K L} \left(\left(f _ {1}, \dots , f _ {T}\right) \| \left(g _ {1}, \dots , g _ {T}\right)\right) \quad (\text {B y}) \\ = \sum_ {t = 1} ^ {T} \mathrm {K L} \left(\left(f _ {t} \mid f _ {<   t}\right) \| \left(g _ {t} \mid g _ {<   t}\right)\right) \quad (\text {B y}) \\ = \sum_ {t = 1} ^ {T} K L \left(\left(f _ {t} \mid f _ {<   t}\right) \| g _ {t}\right). \quad \left(g _ {i} \text {’ s a r e i n d e p e n d e n t}\right) \\ = \sum_ {t = 1} ^ {T} \mathrm {K L} \left(A \left(S _ {t}\right) \| \mathcal {P}\right) \leq T \cdot 2 b / \gamma = O (\log (m)), \tag {13} \\ \end{array}
$$
where the last inequality is due to the do-while loop in Algorithm 1. For any  $S \in (\mathcal{X} \times \{0,1\})^m$ ,
$$
\begin{array}{l} \mathsf {K L} (A ^ {\star} (S) \parallel \mathcal {P} ^ {\star}) = \mathbb {E} _ {h \sim P _ {A ^ {\star} (S)}} \left[ \log \left(\frac {P _ {A ^ {\star} (S)} (h)}{\mathcal {P} ^ {\star} (h)}\right) \right] \\ \leq \mathbb {E} _ {h \sim P _ {A ^ {\star} (S)}} \left[ \log \left(\frac {P _ {A ^ {\star} (S)} (h)}{\mathcal {P} _ {T} ^ {\star} (h) / (z T ^ {2})}\right) \right] \\ = \operatorname {K L} \left(A ^ {\star} (S) \| \mathcal {P} _ {T} ^ {\star}\right) + O (\log (T)) = O (\log (m)). \quad \text {(B y E q . (1 3))} \\ \end{array}
$$
This establishes Item 2.
Item 3 follows by plugging  $\beta = \frac{1}{m}$  and Items 1 and 2 in the PAC-Bayes theorem (Theorem 3.2), yielding
$$
\underset {S \sim \mathcal {D} ^ {m}} {\mathbb {P}} \left[ \mathrm {L} _ {\mathcal {D}} (A ^ {\star} (S)) \leq O \left(\sqrt {\frac {\log (m)}{m}}\right) \right] \geq 1 - \frac {1}{m}.
$$
This implies Item 3 because the 0-1 loss is at most 1.
Remark A.7. Our definition of the learning rule  $A^{\star}$  depends on  $A$  and  $\mathcal{P}$ . The mapping  $S_{t} \mapsto \mathrm{KL}(A(S_{t}) \parallel \mathcal{P})$  is well-defined, so  $A^{\star}$  is a well-defined learning rule.
B Proof of Theorem 2.1 (DI Equivalences)
In this section, we prove Theorem 2.1.
Theorem (Theorem 2.1, Restatement). Let  $\mathcal{H}$  be a hypothesis class. The following is equivalent.
1. There exists a learning rule that PAC learns  $\mathcal{H}$  and satisfied pure differential privacy (Definition 3.5).
2.  $\mathcal{H}$  has finite fractional clique dimension.
3. For every  $\alpha \in [1,\infty ]$ , there exists a learning rule that PAC learns  $\mathcal{H}$  and satisfied distribution-independent  $\mathsf{D}_{\alpha}$ -stability (Definition 3.6).
4. For every  $\alpha \in [1,\infty ]$ , there exists a distribution-independent  $\mathsf{D}_{\alpha}$ -stable PAC learner  $A$  for  $\mathcal{H}$ , that satisfies the following:
The next subsections contain Theorem B.1, which is a useful result from [AMSY23], followed by the statements and proofs of Lemmas B.2 and B.4, which rely on Theorem B.1 and our boosting result (Theorem 2.2). The proof of Theorem 2.1 is a consequence of these results, as follows.
2We remark that if  $A$  is a randomized Turing machine, then  $\mathsf{KL}(A(S_t)\parallel \mathcal{P})$  can be estimated to arbitrary precision by a Turing machine with oracle access to the function  $\mathcal{P}$ . Namely, consider a Turing machine that can query an oracle for the value of  $\mathcal{P}(h)$  up to precision  $2^{-q}$  for any  $h$  and  $q\in \mathbb{N}$  of its choosing. To see that such a machine can estimate  $\mathsf{KL}(A(S_t)\parallel \mathcal{P})$ , observe that if  $A$  uses some finite number of random coins, then  $A(S_{t})$  has a finite support, and so computing  $\mathsf{KL}(A(S_t)\parallel \mathcal{P})$  involves querying  $\mathcal{P}$  at a finite number of locations. Moreover, if  $A$  uses a number  $R$  of random coins, which is itself a random variable that may be unbounded but satisfies  $\mathbb{E}[R] < \infty$ , then by Markov's inequality there exists an explicit algorithm  $A^{\prime}$  that uses at most  $\mathbb{E}[R] / \alpha$  random coins, such that  $\mathsf{TV}(A(S_t),A'(S_t)) < \alpha$ . Hence,  $\mathsf{KL}(A'(S_t)\parallel \mathcal{P})$  can be estimated to arbitrary precision as before. Taking small enough values of  $\alpha$  yields a modified version of  $A^{\star}$  that can be shown to satisfy the requirements of Theorem 2.2.
Proof of Theorem 2.1. The proof follows from:
$$
\text {I t e m} 1 \xleftarrow {\text {T h e o r e m B . 1}} \text {I t e m} 2 \xrightarrow {\text {L e m m a B . 2}} \text {I t e m} 4 \xrightarrow {(*)} \text {I t e m} 3 \xrightarrow {\text {L e m m a B . 4}} \text {I t e m} 2,
$$
where  $(*)$  is immediate.
B.1 Characterization of Pure DP Learnability via the Fractional Clique Dimension
For every hypothesis class  $\mathcal{H}$ , they define a quantity  $\omega_{m}^{\star} = \omega_{m}^{\star}(\mathcal{H})$ , called the fractional clique number of  $\mathcal{H}$ . The definition of  $\omega_{m}^{\star}$  involves an LP relaxation of clique numbers on a certain graph corresponding to  $\mathcal{H}$ , but for our purposes it will be more convenient to use the following alternative characterization (Eq. 6 and Theorem 2.8 in [AMSY23]):
$$
\forall m \in \mathbb {N}: \frac {1}{\omega_ {m} ^ {\star}} = \sup  _ {\mathcal {P}} \inf  _ {\substack {S \sim \mathcal {S} \\ h \sim \mathcal {P}}} \mathbb {P} [ \mathrm {L} _ {S} (h) = 0 ], \tag{14}
$$
where the supremum is taken over distributions over  $\mathcal{H}$ , and the infimum is taken over distributions over samples of size  $m$  that are realizable by  $\mathcal{H}$ . In words,  $1 / \omega_{m}^{\star}$  is the value of a game in which player 1 selects a distribution of hypotheses over  $\mathcal{H}$ , player 2 selects a distribution over realizable samples of size  $m$ , and player 1 wins if and only if the hypothesis correctly labels all the points in the sample.
The fractional clique number characterizes pure DP learnability, as follows:
Theorem B.1 (Restatement of Theorems 2.3 and 2.6 in [AMSY23]). For any hypothesis class  $\mathcal{H}$ , exactly one of the following statements holds:
1.  $\mathcal{H}$  is pure DP learnable (as in Definition 3.5), and there exists a polynomial  $p$  such that  $\omega_{m}^{\star}(\mathcal{H}) \leq p(m)$  for all  $m \in \mathbb{N}$ .
2.  $\mathcal{H}$  is not pure DP learnable, and  $\omega_{m}^{\star}(\mathcal{H}) = 2^{m}$  for all  $m\in \mathbb{N}$ .
The fractional clique dimension of  $\mathcal{H}$  is defined by  $\mathsf{CD}^{\star}(\mathcal{H}) = \sup \left\{m\in \mathbb{N}:\omega_{m}^{\star}(\mathcal{H}) = 2^{m}\right\}$ . So in other words, Theorem B.1 states that  $\mathcal{H}$  is pure DP learnable if and only if  $\mathsf{CD}^{\star}(\mathcal{H})$  is finite.
B.2 Finite Fractional Clique Dimension  $\Rightarrow$  DI Renyi-Stability
Lemma B.2. In the context of Theorem 2.1: Item 2  $\Longrightarrow$  Item 4.
Proof of Lemma B.2. Given that  $\mathcal{H}$  is DP learnable, we define a learning rule  $A$  and a prior  $\mathcal{P}$ , and show that  $A$  PAC learns  $\mathcal{H}$  subject to distribution-independent KL-stability with respect to  $\mathcal{P}$ .
By Theorem B.1 there exists a polynomial  $p$  such that  $\omega_{m}^{\star}(\mathcal{H}) \leq p(m)$  for all  $m \in \mathbb{N}$ . By Eq. (14), for every  $m \in \mathbb{N}$ , there exists a prior  $\mathcal{P}_m \in \Delta\big(\{0,1\}^\mathcal{X}\big)$  such that for any  $\mathcal{H}$ -realizable sample  $S \in (\mathcal{X} \times \{0,1\})^m$ ,
$$
\mathop{\mathbb{P}}_{h\sim \mathcal{P}_{m}}[\mathrm{L}_{S}(h) = 0]\geq \frac{1}{\omega_{m}^{\star}}\geq \frac{1}{p(m)}.
$$
$$
\mathcal {P} = \frac {1}{z} \sum_ {m = 1} ^ {\infty} \frac {\mathcal {P} _ {m}}{m ^ {2}}
$$
be a mixture, where  $z = \sum_{m=1}^{\infty} 1/m^{2} = \pi^{2}/6$  is a normalization factor.  $\mathcal{P}$  is a valid distribution over  $\{0,1\}^{\mathcal{X}}$ .
For every  $m\in \mathbb{N}$  and for any  $\mathcal{H}$ -realizable sample  $S\in (\mathcal{X}\times \{0,1\})^m$
$$
\underset {h \sim \mathcal {P}} {\mathbb {P}} [ \mathrm {L} _ {S} (h) = 0 ] \geq \frac {1}{z m ^ {2}} \cdot \underset {h \sim \mathcal {P} _ {m}} {\mathbb {P}} [ \mathrm {L} _ {S} (h) = 0 ] \geq \frac {1}{z m ^ {2} p (m)} = \frac {1}{q (m)}, \tag {15}
$$
where  $q(m) = zm^{2}p(m)$ .
For any sample  $S$ , let  $C_S = \{h \in \{0,1\}^\mathcal{X} : \mathrm{L}_S(h) = 0\}$  be the set of hypotheses consistent with  $S$ . Let  $A$  be a randomized learning rule given by  $S \mapsto \mathcal{Q}_S \in \Delta(\{0,1\}^\mathcal{X})$  such that  $\mathcal{Q}_S(h) = \mathcal{P}(h \mid C_S)$  if  $h \in C_S$ , and  $\mathcal{Q}_S(h) = 0$  otherwise.  $A$  can be written explicitly as a rejection sampling algorithm:
$A(S)$  :   
do: sample  $h\gets \mathcal{P}$    
while  $\mathrm{L}_S(h) > 0$    
return  $h$
Algorithm  $A$  terminates with probability 1, because for any realizable sample  $S$  of size  $m \in \mathbb{N}$  and any  $t \in \mathbb{N}$ ,
$$
\mathbb {P} [ A \text {d i d n o t t e r m i n a t e a f t e r i t a t i o n s} ] = \left(\mathbb {P} _ {h \sim \mathcal {P}} \left[ \mathrm {L} _ {S} (h) > 0 \right]\right) ^ {t} \leq \left(1 - \frac {1}{q (m)}\right) ^ {t} \xrightarrow {t \to \infty} 0,
$$
where the inequality follows by Eq. (15).
To complete the proof, we show that  $A$  satisfies (i), (ii) and (iii) in Item 4.
Item (i) is immediate from the construction of  $A$ . For Item (ii), let  $m \in \mathbb{N}$ . For any sample  $S$  of size  $m$  and hypothesis  $h \in C_S$ ,
$$
\mathcal {Q} _ {S} (h) = \mathcal {P} (h \mid C _ {S}) = \frac {\mathcal {P} (\{h \} \cap C _ {S})}{\mathcal {P} (C _ {S})} \leq q (m) \cdot \mathcal {P} (h), \tag {16}
$$
where the inequality follows from Eq. (15). Hence,
$$
\begin{array}{l} \mathsf {D} _ {\infty} (\mathcal {Q} _ {S} \| \mathcal {P}) = \log \left(\operatorname * {e s s} _ {\mathcal {Q} _ {S}} \frac {\mathcal {Q} _ {S} (h)}{\mathcal {P} (h)}\right) \\ \leq \log \left(\operatorname {e s s} _ {\mathcal {Q} _ {S}} \sup  \frac {q (m) \cdot \mathcal {P} (h)}{\mathcal {P} (h)}\right) \quad (\text {f r o m E q .} (1 6) \text {a n d} \mathcal {Q} _ {S} \left(C _ {S}\right) = 1) \\ \leq \log (q (m)) = O (\log (m)). \\ \end{array}
$$
Item (ii) follows from monotonicity of  $\mathsf{D}_{\alpha}$  with respect to  $\alpha$  (Lemma A.1). In particular,  $\mathsf{KL}(\mathcal{Q}_S\parallel \mathcal{P}) = O(\log (m))$ .
Item (iii) follows from the PAC-Bayes theorem (Theorem 3.2). Indeed, take  $\beta = \frac{1}{m}$  and note that  $\mathrm{L}_S(\mathcal{Q}_S) = 0$  for all realizable  $S$ . Then for any  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ ,
$$
\underset {S \sim \mathcal {D} ^ {m}} {\mathbb {P}} \left[ \mathrm {L} _ {\mathcal {D}} (A (S)) \leq \sqrt {\frac {\operatorname {K L} (\mathcal {Q} _ {S} \| \mathcal {P}) + 2 \ln m}{2 (m - 1)}} \right] \geq 1 - \frac {1}{m}.
$$
This implies that for any  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ ,
$$
\underset {S \sim \mathcal {D} ^ {m}} {\mathbb {E}} \left[ \mathrm {L} _ {\mathcal {D}} (A (S)) \right] \leq \frac {1}{m} + \sqrt {\frac {\operatorname {K L} \left(\mathcal {Q} _ {S} \| \mathcal {P}\right) + 2 \ln m}{2 (m - 1)}} = O \left(\sqrt {\frac {\log m}{m}}\right),
$$
Remark B.3. The 'furthermore' section of Lemma A.1 implies that in the foregoing proof,  $\mathsf{D}_{\alpha}(\mathcal{Q}_S\parallel \mathcal{P}) = \mathsf{D}_{\beta}(\mathcal{Q}_S\parallel \mathcal{P})$  for any  $\alpha ,\beta \in [0,\infty ]$
B.3 DI Rényi-Stability  $\Longrightarrow$  Finite Fractional Clique Dimension
Lemma B.4. In the context of Theorem 2.1: Item  $3 \Longrightarrow$  Item 2.
Proof of Lemma B.4. By Theorem B.1 and Eq. (14) it suffices to show that there exist  $m \in \mathbb{N}$  and a prior  $\mathcal{P}$  such that for every  $\mathcal{H}$ -realizable sample  $S \in (\mathcal{X} \times \{0,1\})^m$ ,
$$
\underset {h \sim \mathcal {P}} {\mathbb {P}} [ \mathrm {L} _ {S} (h) = 0 ] > \frac {1}{2 ^ {m}}. \tag {17}
$$
By the assumption (Item 3) and Theorem 2.2, there exists an interpolating learning rule  $A^{\star}$ , a prior  $\mathcal{P}^{\star}$ , and a constant  $C > 0$  such that for every  $\mathcal{D} \in \mathrm{Realizable}(\mathcal{H})$ , the equality
$$
\mathbb {P} _ {S \sim \mathcal {D} ^ {m}} [ \mathrm {K L} (A ^ {\star} (S) \| \mathcal {P} ^ {\star}) \leq C \log (m) ] = 1 \tag {18}
$$
holds for all  $m \in \mathbb{N}$  large enough. Fix such an  $m$ . We show that taking  $\mathcal{P} = \mathcal{P}^{\star}$  satisfies Eq. (17) for this  $m$ .
Let  $\mathcal{Q}$  denote the distribution of  $A^{\star}(S')$  where  $S' \sim (\mathrm{U}(S))^m' = P_{S'}$ ,  $\mathrm{U}(S)$  is the uniform distribution over  $S$ , and  $m' = m \ln(4m)$ . The proof follows by noting that if  $\mathrm{KL}(\mathcal{Q} \parallel \mathcal{P}^{\star})$  is small then one can lower bounding the probability of an event according to  $\mathcal{P}^{\star}$  by its probability according to  $\mathcal{Q}$ .
To see that the KL is indeed small, let  $P_{A^{\star}(S')}, S'$  and  $P_{H^{\star}, S'}$  be two joint distributions. The variable  $S'$  has marginal  $P_{S'}$  in both distributions,  $A^{\star}(S') \sim \mathcal{Q}$  depends on  $S'$ , but  $H^{\star} \sim \mathcal{P}^{\star}$  is independent of  $S'$ . Then,
$$
\begin{array}{l} \mathsf {K L} (\mathcal {Q} \| \mathcal {P} ^ {\star}) = \mathsf {K L} \left(P _ {A ^ {\star} (S ^ {\prime})} \| P _ {H ^ {\star}}\right) \\ \leq \operatorname {K L} \left(P _ {A ^ {\star} \left(S ^ {\prime}\right) \mid S ^ {\prime}} \| P _ {H ^ {\star} \mid S ^ {\prime}} \mid P _ {S ^ {\prime}}\right) \quad (\text {L e m m a A . 5}) \\ = \mathsf {K L} \left(P _ {A ^ {\star} \left(S ^ {\prime}\right) \mid S ^ {\prime}} \| P _ {H ^ {\star}} \mid P _ {S ^ {\prime}}\right) \quad (H ^ {\star} \bot S ^ {\prime}) \\ = \mathbb {E} _ {S ^ {\prime}} [ \mathrm {K L} (A ^ {\star} (S ^ {\prime}) \| \mathcal {P} ^ {\star}) ] \quad (\text {D e f i n i t i o n o f c o n d i t i o n a l K L}) \\ \leq C \log (m). \quad (\text {B y E q .} (1 8) \text {a n d c h o i c e o f} m) \tag {19} \\ \end{array}
$$
Taking  $k = 2C\log (m)$ ,
$$
\mathbb {P} _ {h \sim \mathcal {Q}} \left[ \log \left(\frac {\mathcal {Q} (h)}{\mathcal {P} ^ {\star} (h)}\right) \geq k \right] \leq \frac {\mathrm {K L} (\mathcal {Q} \| \mathcal {P} ^ {\star})}{k} \leq \frac {1}{2} \tag {20}
$$
holds by Markov's inequality and the definition of the KL divergence. We are interested in the probability of the event  $\mathcal{E} = \left\{h\in \{0,1\}^{\mathcal{X}}:\mathrm{L}_S(h) = 0\right\}$ . Because  $A^{\star}$  is interpolating,
$$
\mathcal {Q} (\mathcal {E}) \geq \mathbb {P} _ {\substack {S ^ {\prime} \sim (\mathrm {U} (S)) ^ {m ^ {\prime}} \\ h \sim A ^ {*} (S ^ {\prime})}} [ S \subseteq S ^ {\prime} ] \geq 1 - m \left(1 - \frac {1}{m}\right) ^ {m ^ {\prime}} \geq \frac {3}{4}. \tag{21}
$$
Finally, we lower bound  $\mathcal{P}^{\star}(\mathcal{E})$  as follows.
$$
\begin{array}{l} \mathcal {P} ^ {\star} (\mathcal {E}) \geq \underset {h \sim \mathcal {P} ^ {\star}} {\mathbb {P}} \left[ \mathcal {E} \wedge \log \left(\frac {\mathcal {Q} (h)}{\mathcal {P} ^ {\star} (h)}\right) \leq k \right] \\ = \mathbb {P} _ {h \sim \mathcal {P} ^ {\star}} \left[ \mathcal {E} \wedge \mathcal {P} ^ {\star} (h) \geq 2 ^ {- k} \cdot \mathcal {Q} (h) \right] \\ \geq \mathbb {P} _ {h \sim \mathcal {Q}} \left[ \mathcal {E} \wedge \mathcal {P} ^ {\star} (h) \geq 2 ^ {- k} \cdot \mathcal {Q} (h) \right] \cdot 2 ^ {- k} \\ = \mathbb {P} _ {h \sim \mathcal {Q}} \left[ \mathcal {E} \wedge \log \left(\frac {\mathcal {Q} (h)}{\mathcal {P} ^ {\star} (h)}\right) \leq k \right] \cdot 2 ^ {- k}. \\ \geq \left(\mathcal {Q} (\mathcal {E}) - \underset {h \sim \mathcal {Q}} {\mathbb {P}} \left[ \log \left(\frac {\mathcal {Q} (h)}{\mathcal {P} ^ {\star} (h)}\right) \leq k \right]\right) \cdot 2 ^ {- k}. \quad (\text {D e M o r g a n ' s + u n i o n b o u n d}) \\ \geq \frac {1}{4} \cdot 2 ^ {- k} = \frac {1}{4 m ^ {2 C}} = \frac {1}{\operatorname {p o l y} (m)}. \quad (\text {B y E q s . (2 0) a n d (2 1) a n d c h o i c e o f} k) \\ \end{array}
$$
This establishes Eq. (17), as desired.
C Proof of Theorem 1.4 (DD Equivalences)
C.1.1 Littlestone Dimension
The Littlestone dimension is a combinatorial parameter which captures mistake and regret bounds in online learning [Lit87, BPS09].
Definition C.1 (Mistake Tree). A mistake tree is a binary decision tree whose nodes are labeled with instances from  $\mathcal{X}$  and edges are labeled by 0 or 1 such that each internal node has one outgoing edge labeled 0 and one outgoing edge labeled 1. A root-to-leaf path in a mistake tree can be described as a sequence of labeled examples  $(x_{1},y_{1}),\ldots ,(x_{d},y_{d})$ . The point  $x_{i}$  is the label of the  $i$ -th internal node in the path, and  $y_{i}$  is the label of its outgoing edge to the next node in the path.
Definition C.2 (Shattering). Let  $\mathcal{H}$  be a hypothesis class and let  $T$  be a mistake tree.  $\mathcal{H}$  shatters  $T$  if every root-to-leaf path in  $T$  is realizable by  $\mathcal{H}$ .
Definition C.3 (Littlestone Dimension). Let  $\mathcal{H}$  be a hypothesis class. The Littlestone dimension of  $\mathcal{H}$ , denoted  $\mathsf{LD}(\mathcal{H})$ , is the largest number  $d$  such that there exists a complete mistake tree of depth  $d$  shattered by  $\mathcal{H}$ . If  $\mathcal{H}$  shatters arbitrarily deep mistake trees then  $\mathsf{LD}(\mathcal{H}) = \infty$ .
C.1.2 Clique Dimension
Definition C.4 (Clique; [AMSY23]). Let  $\mathcal{H}$  be a hypothesis class and let  $m\in \mathbb{N}$ . A clique in  $\mathcal{H}$  of order  $m$  is a family  $\mathcal{S}$  of realizable samples of size  $m$  such that (i)  $|\mathcal{S}| = 2^m$ ; (ii) every two distinct samples  $S^{\prime}, S^{\prime \prime}\in \mathcal{S}$  contradicts, i.e., there exists a common example  $x\in \mathcal{X}$  such that  $(x,0)\in S^{\prime}$  and  $(x,1)\in S^{\prime \prime}$ .
Definition C.5 (Clique Dimension; [AMSY23]). Let  $\mathcal{H}$  be a hypothesis. The clique dimension of  $\mathcal{H}$ , denoted  $\mathrm{CD}(\mathcal{H})$ , is the largest number  $m$  such that  $\mathcal{H}$  contains a clique of order  $m$ . If  $\mathcal{H}$  contains cliques of arbitrary large order then we write  $\mathrm{CD}(\mathcal{H}) = \infty$ .
C.2 Global Stability  $\Rightarrow$  Replicability
Lemma C.6. Let  $\mathcal{H}$  be a hypothesis class and let  $A$  be a  $(m,\eta)$ -globally stable learner for  $\mathcal{H}$ . Then,  $A$  is an  $\eta$ -replicable learner for  $\mathcal{H}$ .
This follows immediately by noting that global stability is equivalent to 2-parameters replicability, which is qualitatively equivalent to 1-parameter replicability [ILPS22].
Lemma C.7 ([ILPS22]). For every  $\rho, \eta, \nu \in [0,1]$ ,
1. Every  $\rho$ -replicable algorithm is also  $\left(\frac{\rho - \nu}{1 - \nu}, \nu\right)$ -replicable.
2. Every  $(\eta, \nu)$ -replicable algorithm is also  $(\eta + 2\nu - 2)$ -replicable.
Proof of Lemma C.6. By the assumption, there exists an hypothesis  $h$  such that for every population  $\mathcal{D}$ , we have  $\mathbb{P}_{R\sim \mathcal{R}}[\mathbb{P}_{S\sim \mathcal{D}^m}[A(S;r) = h]\geq \eta ] = 1$ . Hence  $A$  is  $(\eta ,1)$ -replicable, and by Lemma C.7 it is also  $\eta$ -replicable.
C.3 DDKL-Stability  $\Longrightarrow$  Finite Littlestone Dimension
Lemma C.8. Let  $\mathcal{H}$  be a hypothesis class that is distribution-dependent KL-stable. Then  $\mathcal{H}$  has finite Littlestone dimension.
This lemma is an immediate result of the relation between thresholds and the Littlestone dimension, and the fact that the class of thresholds on the natural numbers does not admit any learning rule that satisfies a non-vacuous PAC-Bayes bound [LM20]. The next lemma is a corollary of Theorem 2 in [LM20].
Theorem C.9 (Corollary of Theorem 2 [LM20]). Let  $m \in \mathbb{N}$  and let  $N \in \mathbb{N}$ . Then, there exists  $n \in \mathbb{N}$  large enough such that the following holds. For every learning rule  $A$  of the class of thresholds over  $[n]$ ,  $\mathcal{H}_n = \{\mathbb{1}_{[x > k]} : [n] \to \{0,1\} \mid k \in [n]\}$ , there exists a realizable population distribution  $\mathcal{D} = \mathcal{D}_A$  such that for any prior distribution  $\mathcal{P}$ ,
$$
\underset {S \sim \mathcal {D} ^ {m}} {\mathbb {P}} \left[ \mathrm {K L} (A (S) \| \mathcal {P}) > N o r, \quad \mathrm {L} _ {\mathcal {D}} (A (S)) > \frac {1}{4} \right] \geq \frac {1}{1 6}
$$