SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, by von Neumann's minimax theorem [27] there exists a distribution  $Q$  over all possible subsets  $U \subseteq S$  of size at most  $4pd$  such that for all examples  $(x,y) \in S$  it holds that:
$$
\operatorname * {P r} _ {U \sim Q} [ f _ {U} (x) = y ] <   \frac {1}{4 p}.
$$
Let  $U_{1}, \ldots, U_{\ell}$  be random samples from  $Q$  where  $\ell = \lceil 8p^{2}\ln (2m)\rceil$ . Denote  $\bar{U}_{\ell} = (U_{1},\dots ,U_{\ell})$ , and define  $F_{\bar{U}_{\ell}}$  to be a corresponding averaged-vote, such that for any  $x \in \mathcal{X}$  and  $y \in [p]$ :
$$
F _ {\bar {U} _ {\ell}} (x, y) = \frac {1}{\ell} \sum_ {j = 1} ^ {\ell} \mathbb {1} \left[ f _ {U _ {j}} (x) = y \right].
$$
Observe that for a fixed example pair  $(x,y)\in S$  then by a Chernoff bound we have,
$$
\begin{array}{l} \Pr_ {\bar {U} _ {\ell} \sim Q ^ {\ell}} \left[ F _ {\bar {U} _ {\ell}} (x, y) \geq \frac {1}{2 p} \right] \leq \Pr_ {\bar {U} _ {\ell} \sim Q ^ {\ell}} \left[ F _ {\bar {U} _ {\ell}} (x, y) \geq \mathbb {E} [ F _ {\bar {U} _ {\ell}} (x, y) ] + \frac {1}{4 p} \right] (31) \\ \leq e ^ {- 2 \left(\ell / 4 p\right) ^ {2} / \ell} = e ^ {- \ell / \left(8 p ^ {2}\right)} \leq \frac {1}{2 m}. (32) \\ \end{array}
$$
Then, by a union bound over all  $m$  examples in  $S$  we have that with positive probability over the random choice of  $\bar{U}_{\ell}$  it holds that  $F_{\bar{U}_{\ell}}(x,y) < \frac{1}{2p}$  over all  $(x,y)\in S$  simultaneously. This implies that there exist a particular choice of  $\bar{U}_{\ell}$  for which we have with probability 1 for all  $(x,y)\in S$  that:
$$
y \notin \operatorname * {a r g   m a x} _ {\hat {y} \in [ p ]} F _ {\bar {U} _ {\ell}} (x, \hat {y}).
$$
In words, this means that taking the plurality-vote as induced by  $\bar{U}_{\ell}$ , we will yield an incorrect label with probability 1 over all examples in  $S$  simultaneously.
Lastly, we are ready to define the final list function  $\mu : \mathcal{X} \to [p]^{p-1}$  by:
$$
\mu (x) = [ p ] \setminus \operatorname * {a r g   m a x} _ {\hat {y} \in [ p ]} F _ {\bar {U} _ {\ell}} (x, \hat {y}).
$$
To conclude, we have constructed an algorithm based on  $m \to 4pd \cdot \ell$  compression scheme for  $(p - 1)$ -lists, that is consistent with the training sample  $S$ , as claimed.
Algorithm 6  $k$ -List PAC Learning for a class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  with  $d_{DS}^{k} = d < \infty$
Given: Training data  $S\in (\mathcal{X}\times \mathcal{Y})^m$
Output: A list  $\mu : \mathcal{X} \mapsto \mathcal{Y}^k$ .
1: Set  $p = k \cdot (d + 1) \cdot \ln(2m)$  (as chosen in Lemma9).
2: Initialize: applying pre-processing as given in Lemma 9 to get  $\mu_1:\mathcal{X}\mapsto \mathcal{Y}^p$
3: for  $j = 1, \dots, p - k$  do
4: Let  $S^j$  and  $\mathcal{H}^j$  denote  $S$  and  $\mathcal{H}$  with all labels converted from  $\mathcal{V}$  to  $[p - j + 1]$  via  $\mu_j$ .
5: Call the list learner given in Theorem 8 over  $S^j$  and  $\mathcal{H}^j$ , to obtain  $\tilde{\mu}_j:\mathcal{X}\to [p - j + 1]^{p - j}$ .
6: Define  $\mu_{j + 1}:\mathcal{X}\to \mathcal{Y}^{p - j}$  by :
$$
\mu_ {j + 1} (x) = \left\{y: \exists \ell , \mu_ {j} (x) _ {\ell} = y \wedge \ell \in \tilde {\mu} _ {j} (x) \right\}.
$$
8: Output the final list  $\mu \coloneqq \mu_{p - k + 1}$
Proof of Theorem 5. The proof is given via a sample compression scheme argument, demonstrating that for a class with a finite  $k$ -DS dimension  $d$ , Algorithm 6 is a  $k$ -list PAC learner. That is, we prove that it returns a list  $\mu : \mathcal{X} \mapsto \mathcal{Y}^k$  which can be represented using a small number of training examples, and that it is consistent with the entire training set.
We will then show for each  $j = 1 \dots p - k$  that  $\mu_{j + 1}$  satisfies the following 2 properties: (a) the list  $\mu_{j + 1}$  is consistent with the entire sample  $S$ , and (b) it can be represented using only a small number  $r_j$  of training examples.
First, notice that by Lemma 9 it is guaranteed  $\mu_{1}$  is consistent with  $S$ , and that it can be represented by  $r_1 = d(d + 1)\ln (2m)$  examples, as it is constructed by an  $m\to r_1$  compression scheme.
Next, we will show that the 2 properties (a) and (b) above hold for all  $j \geq 2$ . Notice that in each round  $j$  of Algorithm 6, the class  $\mathcal{H}^j$  has only  $p - j + 1$  labels, and its  $k$ -DS dimension remains  $d$  as in  $\mathcal{H}$ . Furthermore, by the properties of the  $k$ -DS dimension it holds that for any  $k' > k$  we have  $d_{DS}^{k'} \leq d_{DS}^k$ . Thus, for all  $j \leq p - k$ ,
$$
d _ {D S} ^ {p - j} (\mathcal {H} ^ {j}) \leq d _ {D S} ^ {k} (\mathcal {H} ^ {j}) = d.
$$
Therefore, we can apply Theorem 8 and get that both properties (a) and (b) hold with  $r_j = 4(p - j + 1)d \cdot \lceil 8(p - j + 1)^2\ln (2m)\rceil$ , for all  $j = 2,\dots,p - k$ .
Overall, we have shown that the final list  $\mu \coloneqq \mu_{p - k + 1}$  is both consistent with the sample  $S$ , and can be represented using only  $r$  examples, where,
$$
\begin{array}{l} r = r _ {1} + \sum_ {j = 2} ^ {p - k + 1} r _ {j} (33) \\ = d (d + 1) \ln (2 m) + \sum_ {j = 2} ^ {p - k + 1} \left(4 (p - j + 1) d \cdot \lceil 8 (p - j + 1) ^ {2} \ln (2 m) \rceil\right) (34) \\ \leq d (d + 1) \ln (2 m) + 3 2 d \ln (2 m) p ^ {4} + p (35) \\ = O \left(d ^ {5} \cdot k ^ {4} \cdot \ln^ {5} (m)\right), (36) \\ \end{array}
$$
where the bound follows by plugging in the value of  $p$  from Algorithm 6 (and Lemma 9).
We can now apply a sample compression scheme bound to obtain the final result. Specifically, we apply Theorem 7, for a  $m \to r$  sample compression scheme algorithm  $\mathcal{A}$  equipped with a reconstruction function  $\rho$  (see Definition 8). We denote  $\mathrm{err}_{\mathcal{D}}(\mu) = \operatorname*{Pr}_{(x,y) \sim \mathcal{D}}[\mu(x) \neq y]$ . Then, by Theorem 7 we get that for any  $\delta > 0$ ,
$$
\Pr_ {S \sim \mathcal {D} ^ {m}} \left[ \operatorname {e r r} _ {\mathcal {D}} (\mu) > \frac {r \ln (m) + \ln (1 / \delta)}{m - r} \right] \leq \delta ,
$$
Plugging in  $r$  from Equation (36) yields the desired bound.