SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem C.10 (Littlestone dimension and thresholds [She90]). Let  $\mathcal{H}$  be a hypothesis class. Then,
1. If  $\mathsf{LD}(\mathcal{H})\geq d$  then  $\mathcal{H}$  contains  $\lfloor \log d\rfloor$  thresholds.
2. If  $\mathcal{H}$  contains  $d$  thresholds then  $\mathsf{LD}(\mathcal{H})\geq \lfloor \log d\rfloor$
Proof of Lemma C.8. If by contradiction the Littlestone dimension of  $\mathcal{H}$  is unbounded, then by Theorem C.10,  $\mathcal{H}$  contains a copy of  $\mathcal{H}_n$ , the class of thresholds over  $[n]$ , for arbitrary large  $n$ 's. Hence, by Theorem C.9  $\mathcal{H}$  does not admit a PAC learner that is KL-stable.
C.4 Ml-Stability  $\Rightarrow$  DDKL-Stability
Lemma C.11. Let  $\mathcal{H}$  be a hypothesis class and let  $A$  be a mutual information stable learner with information bound  $f(m) = o(1)$ . (I.e. for every population distribution  $\mathcal{D}$ ,  $I(A(S);S) \leq f(m)$  where  $S \sim \mathcal{D}^m$ .) Then,  $A$  is a distribution-dependent KL-stable learner with KL bound  $g(m) = \sqrt{f(m) \cdot m}$  and confidence  $\beta(m) = \sqrt{f(m)/m}$ .
The following statement is an immediate corollary.
Corollary C.12. Let  $\mathcal{H}$  be a hypothesis class that is mutual information stable. Then  $\mathcal{H}$  is distribution-dependent KL-stable.
Proof of Lemma C.11. Let  $\mathcal{D}$  be a population distribution. Define a prior distribution  $\mathcal{P}_{\mathcal{D}} = \mathbb{E}_S[A(S)]$ , i.e.  $\mathcal{P}_{\mathcal{D}}(h) = \mathbb{P}_{S\sim \mathcal{D}^m}[A(S) = h]$ . We will show that  $A$  is KL stable with respect to the prior  $\mathcal{P}_{\mathcal{D}}$ . We use the identity  $I(X;Y) = \mathsf{KL}(P_{X,Y},P_XP_Y)$ . Let  $P_{A(S),S}$  be the joint distribution of the training sample  $S$  and the hypothesis selected by  $A$  when given  $S$  as an input, and let  $P_{A(S)}P_S$  be the product of the marginals. Note that  $P_{A(S)}P_S$  is equal in distribution to  $P_{A(S')}P_S$ , where  $S'$  is an independent copy of  $S$ . Hence,
$$
\begin{array}{l} I (A (S); S) = \mathsf {K L} \left(P _ {A (S), S}, P _ {A (S)} P _ {S}\right) \\ = \mathsf {K L} \left(P _ {A (S) | S} P _ {S}, P _ {A (S ^ {\prime})} P _ {S}\right), \\ = \mathsf {K L} \left(P _ {S}, P _ {S}\right) + \mathbb {E} _ {s \sim P _ {S}} \left[ \mathsf {K L} \left(P _ {A (S) | S = s}, P _ {A \left(S ^ {\prime}\right) | S = s}\right) \right] \quad (\text {C h a i n r u l e}) \\ = \mathbb {E} _ {s \sim P _ {S}} \left[ \mathsf {K L} \left(P _ {A (S) | S = s}, P _ {A (S ^ {\prime}) | S = s}\right) \right] \\ = \mathbb {E} _ {s \sim P _ {S}} \left[ \mathsf {K L} \big (P _ {A (S) | S = s}, P _ {A (S ^ {\prime})} \big) \right]. \\ \end{array}
$$
Note that  $P_{A(S')}$  and the prior  $\mathcal{P}_{\mathcal{D}}$  are identically distributed, and  $P_{A(S)|S=s}$  is exactly the posterior produced by  $A$  given the input sample  $s$ . By Markov's inequality,
$$
\begin{array}{l} \underset {S \sim D ^ {m}} {\mathbb {P}} \left[ \mathrm {K L} (A (S) \| P _ {\mathcal {D}}) \geq \sqrt {m \cdot I (A (S) ; S)} \right] \leq \frac {I (A (S) ; S)}{\sqrt {m I (A (S) ; S)}} \\ = \sqrt {\frac {I (A (S) ; S)}{m}}. \tag {22} \\ \end{array}
$$
Since  $I(A(S);S)\leq f(m)$  , by Eq. (22)
$$
\underset {S \sim D ^ {m}} {\mathbb {P}} \left[ \mathsf {K L} (A (S) \| P _ {\mathcal {D}}) \geq \sqrt {f (m) \cdot m} \right] \leq \sqrt {\frac {f (m)}{m}}.
$$
Note that since  $f(m) = o(m)$ , indeed  $\sqrt{f(m) / m} \xrightarrow{m \to \infty} 0$  and  $\sqrt{f(m) \cdot m} = o(m)$ .
C.5 Finite Littlestone Dimension  $\Rightarrow$  Ml-Stability
Lemma C.13. Let  $\mathcal{H}$  be a hypothesis class with finite Littlestone dimension. Then  $\mathcal{H}$  admits an information stable learner.
This lemma is a direct result of Theorem 2 in [PNG22].
Definition C.14. The information complexity of a hypothesis class  $\mathcal{H}$  is
$$
\mathsf{IC}(\mathcal{H}) = \sup_{|S|}\inf_{A}\sup_{\mathcal{D}}I(A(S);S)
$$
where the supremum is over all sample sizes  $|S|\in \mathbb{N}$  and the infimum is over all learning rules that PAC learn  $\mathcal{H}$ .
Theorem C.15 (Theorem 2 [PNG22]). Let  $\mathcal{H}$  be a hypothesis class of with Littlestone dimension  $d$ . Then the information complexity of  $\mathcal{H}$  is bounded by
$$
\mathbb {C} (\mathcal {H}) \leq 2 ^ {d} + \log (d + 1) + 3 + \frac {3}{e \ln 2}.
$$
Proof of Lemma C.13. Since finite information complexity implies that  $\mathcal{H}$  admits an information stable learner, the proof follows from Theorem C.15