SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lastly, note that the confidence parameter of  $\frac{1}{2k}$  can easily be made arbitrarily large using standard confidence-boosting techniques. That is, for any  $\delta > 0$ , we can set  $q = 2k\log(2/\delta)$  and draw  $q$  sample sets  $S_1, \ldots, S_q$  from  $\mathcal{D}^{m_\ell}$  and call the algorithm we described so far to generate hypotheses  $h_1, \ldots, h_q$  which satisfy Equation (22) with respect to the different samples  $S$ , and different random draws of  $j \sim \mathrm{Unif}(k)$ . By the choice of  $q$ , we have that the probability that no  $h$  has accuracy at least  $\frac{1 - \epsilon}{k}$ , is at most  $(1 - \frac{1}{2k})^q \leq e^{-\frac{q}{2k}} = \delta/2$ . Thus, we get that with probability at least  $1 - \delta/2$  at least one of these hypothesis is good, in that it has accuracy of at least  $\frac{1 - \epsilon}{k}$ . We then pick the best hypothesis as tested over another independent set  $S_{q+1}$ , which we sample i.i.d. from  $\mathcal{D}^r$  where  $r = \frac{10\log(2q/\delta)}{\epsilon'^2}$  and  $\epsilon' = \epsilon/k$ . Then, by Chernoff's inequality we have for each of the  $h_1, \ldots, h_q$  that with probability  $1 - \delta/(2q)$  that the gap between its accuracy over  $S_{q+1}$  and over  $\mathcal{D}$  is at most  $\epsilon'/2$ . By taking the union bound, we get that this holds with probability  $1 - \delta/2$  for all  $h_1, \ldots, h_q$  simultaneously.
Thus when we choose the  $h$  with the most empirical accuracy over  $S_{q + 1}$ , then with probability  $1 - \delta$  it will have an empirical accuracy of at least  $\frac{1 - \epsilon}{k} -\epsilon^{\prime} / 2$ , and true accuracy of at least  $\frac{1 - \epsilon}{k} -\epsilon^{\prime}$ . Therefore, with probability at least  $1 - \delta$  we get that the hypothesis chosen by our final algorithm has accuracy of at least  $\gamma = \frac{1 - 2\epsilon}{k}$  over  $\mathcal{D}$ .
Overall, we get that  $\mathcal{W}$  is a  $\gamma$ -weak PAC learner for  $\mathcal{H}$ , where  $\gamma = \frac{1 - 2\epsilon}{k}$ , and with sample complexity  $m = O\left(k \cdot \log(1/\delta) \cdot m_{\ell}(\epsilon, 1/2) + k^2 \cdot \frac{\log(k/\delta)}{\epsilon^2}\right)$ .
B.2 Proof of Theorem 3
Proof of Theorem 3. We start with showing that  $\gamma (\mathcal{H})\geq 1 / k(\mathcal{H})$ . By assumption, we have that for any fixed  $\epsilon_1,\delta_1 > 0$ , when given  $m_{1}\coloneqq m_{1}(\epsilon_{1},\delta_{1})$  examples  $S$  drawn i.i.d. from a  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , the list learner returns a function  $\mu_S:\mathcal{X}\to \mathcal{Y}^{k(\mathcal{H})}$  such that,
$$
\Pr_ {S \sim \mathcal {D} ^ {m _ {1}}} \left[ \Pr_ {(x, y) \sim \mathcal {D}} [ y \in \mu_ {S} (x) ] \geq 1 - \epsilon_ {1} \right] \geq 1 - \delta_ {1}. \tag {23}
$$
By Lemma 3 we get that for any  $\epsilon >0$ , there is a  $\gamma$ -weak learner for  $\mathcal{H}$ , where  $\gamma := \frac{1 - 2\epsilon}{k(\mathcal{H})}$ .
Moreover, by definition of  $\gamma (\mathcal{H})$  as the supremum over all such  $\gamma$  values, we have  $\gamma (\mathcal{H})\geq \frac{(1 - \epsilon)}{k(\mathcal{H})}$  Lastly, since this holds for any choice of  $\epsilon$ , we can take  $\epsilon \to 0$  and get  $\gamma (\mathcal{H})\geq \frac{1}{k(\mathcal{H})}$ .
We now show that  $\gamma (\mathcal{H})\leq 1 / k(\mathcal{H})$ . Let  $k$  denote the smallest integer for which  $\frac{1}{k} < \gamma (\mathcal{H})$ , and let  $\sigma_0\coloneqq \gamma (\mathcal{H}) - 1 / k > 0$ . We will now show how to construct a list learner via the given weak learner, for a list size of  $k - 1$ . It suffices to show that, since by the choice of  $k$  we have  $\gamma (\mathcal{H})\leq 1 / (k - 1)$  and so it holds that  $k(\mathcal{H})\leq k - 1\leq 1 / \gamma (\mathcal{H})$  and we get the desired bound of  $\gamma (\mathcal{H})\leq 1 / k(\mathcal{H})$ . Thus, all that is left is to show the existence of the  $(k - 1)$ -list learner.
Let  $\epsilon_1, \delta_1 > 0$ , and let  $m(\epsilon_1, \delta_1)$  be determined as  $m$  in lemma 5 for  $\epsilon_1, \delta_1$ . Set  $T = \frac{8 \log(m)}{\sigma_0^2}$ . Let  $m_0 := m_0(\delta_0 := \delta_1 / (2T))$  denote the sample size of a  $\gamma$ -weak PAC learner with  $\gamma := \frac{\sigma_0}{2} + \frac{1}{k}$ . Denote this weak learner by  $\mathcal{W}$ . Then, by applying Lemma 5 with the  $\gamma$ -weak learner  $\mathcal{W}$  and with  $\sigma := \sigma_0$ , we obtain the  $(k-1)$ -list learner, which concludes the proof.
Proof of Theorem 4. The proof follows by constructing a weak learner from the given oracle, and then applying a weak-to-list boosting procedure.
First, by Lemma 3 we get that there is a  $\gamma$ -weak learning algorithm  $\mathcal{W}$  for  $\mathcal{H}$  where  $\gamma := \frac{1 - 2\epsilon_0}{k_0}$ .<sup>5</sup>
Then, by applying lemma 5 with this  $\gamma$ -weak learner  $\mathcal{W}$ , we obtain a  $(t - 1)$ -List PAC learner, where  $\frac{1}{t} < \gamma \leq \frac{1}{t - 1}$ . Then, we also get that  $\frac{1 - 2\epsilon_0}{k_0} \leq \frac{1}{t - 1}$  and so  $(t - 1) \leq \left\lfloor \frac{k_0}{1 - 2\epsilon_0} \right\rfloor$ .
Therefore, there is a  $k$ -List PAC learning algorithm for  $\mathcal{H}$  with a fixed list size  $k = \left\lfloor \frac{k_0}{1 - 2\epsilon_0} \right\rfloor$ .
C Proof of Theorem 5
At a high level, the proof describes a construction of a simple weak learner which is shown to satisfy our BRG condition (Definition 4). This implies it is also amenable to our boosting method, which yields the final result given in Theorem 5.
The construction of the weak learner is based on an object called the One-inclusion Graph (OIG) [13, 20] of a hypothesis class, which is often useful in devising learning algorithms. Typically, one is interested in orienting the edges of this graph in a way that results in accurate learning algorithms. As in the analysis used to characterize PAC, and List PAC learning [5, 8], as well as in most applications of the OIG [9, 13, 20], a good learner corresponds to an orientation with a small maximal out-degree. However, here we show that a simpler task of minimizing the in-degree is sufficient to obtain a reasonable weak learner, and therefore a strong learner as well.
We start with introducing relevant definitions.
The DS dimension was originally introduced by [9]. Here we follow the formulation given in [5], and so we first introduce the notion of pseudo-cubes.
Definition 9 (Pseudo-cube). A class  $\mathcal{H} \subseteq \mathcal{Y}^d$  is called a pseudo-cube of dimension  $d$  if it is non-empty, finite and for every  $h \in \mathcal{H}$  and  $i \in [d]$ , there is an  $i$ -neighbor  $g \in \mathcal{H}$  of  $h$  (i.e.,  $g(i) \neq h(i)$ ) and  $g(j) = h(j)$  for all  $j \neq i$ .
When  $\mathcal{V} = \{0,1\}$ , the two notions "Boolean cube" and "pseudo-cube" coincide: The Boolean cube  $\{0,1\}^d$  is of course a pseudo-cube. Conversely, every pseudo-cube  $\mathcal{H} \subseteq \{0,1\}^d$  is the entire Boolean cube  $\mathcal{H} = \{0,1\}^d$ . When  $|\mathcal{V}| > 2$ , the two notions do not longer coincide. Every copy of the Boolean cube is a pseudo-cube, but there are pseudo-cubes that are not Boolean cubes (see [5] for further details). We are now ready to define the DS dimension.
Definition 10 (DS dimension). A set  $S \in \mathcal{X}^n$  is DS-shattered by  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  if  $\mathcal{H}|_S$  contains an  $n$ -dimensional pseudo-cube. The DS dimension  $d_{DS}(\mathcal{H})$  is the maximum size of a DS-shattered sequence.
A natural analogue of this dimension in the context of predicting lists of size  $k$  is as follows:
Definition 11 ( $k$ -DS dimension [8]). Let  $\mathcal{H} \subseteq \mathcal{V}^{\mathcal{X}}$  be a hypothesis class and let  $S \in \mathcal{X}^d$  be a sequence. We say that  $\mathcal{H}$ $k$ -DS shatters  $S$  if there exists  $\mathcal{F} \subseteq \mathcal{H}$ ,  $|\mathcal{F}| < \infty$  such that  $\forall f \in \mathcal{F}|_S$ ,  $\forall i \in [d]$ ,  $f$  has at least  $k$ $i$ -neighbors. The  $k$ -DS dimension of  $\mathcal{H}$ , denoted as  $d_{DS}^k = d_{DS}^k(\mathcal{H})$ , is the largest integer  $d$  such that  $\mathcal{H}$ $k$ -DS shatters some sequence  $S \in \mathcal{X}^d$ .
Observe that the two definitions above differ only in the number of  $i$ -neighbors they require each hypothesis to have, and that  $d_{DS} = d_{DS}^k$  for  $k = 1$ . Moreover, there are simple hypothesis classes with infinite DS dimension, yet finite  $k$ -DS dimension for  $k \geq 2$ , see Example 3 in [8].
Next, we define the object called the One-inclusion Graph (OIG) [13, 20] of a hypothesis class, and related definitions which will be used for the construction of the weak learner.
Definition 12 (One-inclusion Graph [13, 20]). The one-inclusion graph of  $\mathcal{H} \subseteq \mathcal{Y}^n$  is a hypergraph  $\mathcal{G}(\mathcal{H}) = (V, E)$  that is defined as follows. The vertex-set is  $V = \mathcal{H}$ . For each  $i \in [n]$  and
<sup>5</sup>We note that although Lemma 3 assumes access to a  $k$ -List PAC learner, the same argument holds for the weaker version of a list learner with list size  $k_{0}$  which depends on  $\epsilon_0$ , and the proofs of both cases is identical.
<sup>6</sup>We use the term "one-inclusion graph" although it is actually a hypergraph.
$f:[n]\setminus \{i\} \to \mathcal{Y}$  , let  $e_{i,f}$  be the set of all  $h\in \mathcal{H}$  that agree with  $f$  on  $[n]\setminus \{i\}$  . The edge-set is
$$
E = \left\{e _ {i, f}: i \in [ n ], f: [ n ] \backslash \{i \} \rightarrow \mathcal {Y}, e _ {i, f} \neq \emptyset \right\}. \tag {24}
$$
We say that the edge  $e_{i,f} \in E$  is in the direction  $i$ , and is adjacent to/contains the vertex  $v$  if  $v \in e_{i,f}$ . Every vertex  $h \in V$  is adjacent to exactly  $m$  edges. The size of the edge  $e_{i,f}$  is the size of the set  $|e_{i,f}|$ .
We remark that similarly to [5, 8], edges could be of size one, and each vertex  $v$  is contained in exactly  $n$  edges. This is not the standard structure of edges in hypergraphs, but we use this notation because it provides a better model for learning problems.
With respect to the one-inclusion graph, one can think about the degrees of its vertices, all of which originally introduced in [8], and are natural generalizations of the degrees defined in [9, 5]
Definition 13 ( $k$ -degree). Let  $\mathcal{G}(\mathcal{H}) = (V, E)$  be the one-inclusion graph of  $\mathcal{H} \subseteq \mathcal{Y}^m$ . The  $k$ -degree of a vertex  $v \in V$  is
$$
\left. \deg^ {k} (v) = \left| \left\{e \in E: v \in e, | e | > k \right\} \right|. \right.
$$
We now define the notion of orienting edges of a one-inclusion graph to lists of vertices they are adjacent to. As alluded to earlier, an orientation corresponds to the behavior of a (deterministic) learning algorithm while making predictions on an unlabeled test point, given a set of labeled points as input.
Definition 14 (List orientation). A list orientation  $\sigma^k$  of the one-inclusion graph  $\mathcal{G}(\mathcal{H}) = (V,E)$  having list size  $k$  is a mapping  $\sigma^k: E \to \{V' \subseteq V : |V'| \leq k\}$  such that for each edge  $e \in E$ ,  $\sigma^k(e) \subseteq e$ .
The  $k$ -out-degree of a list orientation  $\sigma^k$  is defined as:
Definition 15 ( $k$ -out-degree of a list orientation). Let  $\mathcal{G}(\mathcal{H}) = (V, E)$  be the one-inclusion graph of a hypothesis class  $\mathcal{H}$ , and let  $\sigma^k$  be a  $k$ -list orientation of it. The  $k$ -out-degree of  $v \in V$  in  $\sigma^k$  is
$$
\operatorname {o u t d e g} ^ {k} (v; \sigma^ {k}) = | \{e: v \in e, v \notin \sigma^ {k} (e) \} |. \tag {25}
$$
The maximum  $k$ -out-degree of  $\sigma^k$  is
$$
\operatorname {o u t d e g} ^ {k} \left(\sigma^ {k}\right) = \sup  _ {v \in V} \operatorname {o u t d e g} ^ {k} (v; \sigma^ {k}). \tag {26}
$$
The following lemmas demonstrate how a bound on the dimensions helps one greedily construct a list orientation of small maximum  $k$ -out-degree.
Lemma 6 (Lemma 3.1, [8]). If  $\mathcal{H} \subseteq \mathcal{Y}^{d+1}$  has  $k$ -DS dimension at most  $d$ , then there exists a  $k$ -list orientation  $\sigma^k$  of  $\mathcal{G}(\mathcal{H})$  with  $\text{outdeg}^k(\sigma^k) \leq d$ .
We now describe a list version of the one-inclusion algorithm below, originally introduced by [8].
Algorithm 5 The one-inclusion list algorithm for  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$
Input: An  $\mathcal{H}$ -realizable sample  $U = ((x_{1},y_{1}),\ldots ,(x_{n},y_{n}))$
Output: A  $k$ -list hypothesis  $\mu_U^k:\mathcal{X}\to \{Y\subseteq \mathcal{Y}:|Y|\leq k\}$ .
For each  $x\in \mathcal{X}$  , the  $k$  -list  $\mu_S^k (x)$  is computed as follows:
1: Consider the class of all patterns over the unlabeled data  $\mathcal{H}|_{(x_1,\ldots ,x_n,x)}\subseteq \mathcal{Y}^{n + 1}$
2: Find a  $k$ -list orientation  $\sigma^k$  of  $\mathcal{G}(\mathcal{H}|_{(x_1,\ldots,x_n,x)})$  that minimizes the maximum  $k$ -out-degree.
3: Consider the following edge defined by revealing all labels in  $U$ :
$$
e = \left\{h \in \mathcal {H} \mid_ {\left(x _ {1}, \dots , x _ {n}, x\right)}: \forall i \in [ n ] h (i) = y _ {i} \right\}.
$$
4: Set  $\mu_U^k (x) = \{h(x):h\in \sigma^k (e)\}$ .
Lemma 7. Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be a hypothesis class, let  $\mathcal{D}$  be an  $\mathcal{H}$ -realizable distribution over  $\mathcal{X} \times \mathcal{Y}$ , and let  $k, n > 0$  be integers. Let  $M$  be an upper bound on the maximum  $k$ -out-degree of all orientations  $\sigma^{k}$  chosen by Algorithm 5, and let  $\mu_{U}^{k}$  denote its output for an input sample  $U \sim \mathcal{D}^{n}$ . Then,
$$
\Pr_ {(U, (x, y)) \sim \mathcal {D} ^ {n + 1}} \left[ \mu_ {U} ^ {k} (x) \not \ni y \right] \leq \frac {M}{n + 1}.
$$
Proof. By the leave-one-out symmetrization argument (Fact 14, [7]) we have,
$$
\operatorname * {P r} _ {(U, (x, y)) \sim \mathcal {D} ^ {n + 1}} \left[ \mu_ {U} ^ {k} (x) \not \ni y \right] = \operatorname * {P r} _ {(U ^ {\prime}, i) \sim \mathcal {D} ^ {n + 1} \times \operatorname {U n i f} (n + 1)} \left[ \mu_ {U _ {- i} ^ {\prime}} ^ {k} \left(x _ {i} ^ {\prime}\right) \not \ni y _ {i} ^ {\prime} \right].
$$
It therefore suffices to show that for every sample  $U^{\prime}$  that is realizable by  $\mathcal{H}$
$$
\Pr_ {i \sim \operatorname {U n i f} (n + 1)} \left[ \mu_ {U _ {- i} ^ {\prime}} ^ {k} \left(x _ {i} ^ {\prime}\right) \not \exists y _ {i} ^ {\prime} \right] \leq \frac {M}{n + 1}. \tag {27}
$$
Fix  $U'$  that is realizable by  $\mathcal{H}$  for the rest of the proof. Denote by  $\sigma^k$  the orientation of  $\mathcal{G}(\mathcal{H}')$  that the algorithm chooses.
Let  $y^\prime$  denote the vertex in  $\mathcal{G}(\mathcal{H}^{\prime})$  defined by  $y^\prime = (y_1^\prime ,\dots ,y_{n + 1}^\prime)$ , and let  $e_i$  denote the edge in the  $i^{\mathrm{th}}$  direction adjacent to  $y^\prime$ . Then, we have that
$$
\begin{array}{l} \operatorname * {P r} _ {i \sim \operatorname {U n i f} (n + 1)} \left[ \mu_ {U _ {- i} ^ {\prime}} ^ {k} \left(x _ {i} ^ {\prime}\right) \not \ni y _ {i} ^ {\prime} \right] = \frac {1}{n + 1} \sum_ {i = 1} ^ {n + 1} \mathbb {1} \left[ \mu_ {U _ {- i} ^ {\prime}} ^ {k} \left(x _ {i} ^ {\prime}\right) \not \ni y _ {i} ^ {\prime} \right] \\ = \frac {1}{n + 1} \sum_ {i = 1} ^ {n + 1} \mathbb {1} \left[ \sigma^ {k} \left(e _ {i}\right) \not \ni y ^ {\prime} \right] \\ = \frac {\operatorname {o u t d e g} ^ {k} \left(y ^ {\prime} ; \sigma^ {k}\right)}{n + 1} \\ \leq \frac {M}{n + 1}. \\ \end{array}
$$
C.1 Initial phase: pre-processing
Lemma 8. Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be a hypothesis class with  $k$ -DS dimension  $d < \infty$ . Then, for every  $\mathcal{H}$ -realizable set  $S \in (\mathcal{X} \times \mathcal{Y})^m$ , any  $\mathcal{D}$  distribution over  $S$ , for a sample  $U \sim \mathcal{D}^n$  for  $n := d$ , when given  $U \in (\mathcal{X} \times \mathcal{Y})^n$ , Algorithm 5 returns  $h_U: \mathcal{X} \mapsto \mathcal{Y}$  such that,
$$
\mathbb {E} _ {U \sim \mathcal {D} ^ {d}} \left[ \Pr_ {(x, y) \sim \mathcal {D}} [ y \in \mu_ {U} ^ {k} (x) ] \right] \geq \frac {1}{d + 1}. \tag {28}
$$
Proof. This follows directly by combining Lemma 6 and Lemma 7.  $\square$
Lemma 9. Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  with  $k$ -DS dimension  $d < \infty$ , and let  $S \in (\mathcal{X} \times \mathcal{Y})^m$  be an  $\mathcal{H}$ -realizable set. Denote by  $\mathcal{L}$  a  $k$ -list learning algorithm as given in Algorithm 5. Denote  $p = k \cdot q$  and  $q = (d + 1)\ln(2m)$ , and let  $r = qd = O(d^2\ln(m))$ . Then, there exists an algorithm  $\mathcal{A}$  based on an  $m \to r$  compression scheme for  $p$ -lists as described next. When provided with  $S$ , and given oracle access to  $\mathcal{L}$ , the algorithm yields a list function  $\mu: \mathcal{X} \to \mathcal{Y}^p$  such that  $\mu$  is consistent with  $S$ .
Proof. We describe an algorithm that sequentially finds  $q$  subsets of examples  $S_1', \ldots, S_q' \subseteq S$  each of size  $d$ , and returns a list  $\mu : \mathcal{X} \mapsto \mathcal{Y}^k$ , using  $r$  examples. First, let  $\mathcal{U}$  denote the uniform distribution over the  $m$  examples in  $S$  and let  $\alpha = \frac{1}{d + 1}$ . By Lemma 8 applied to the distribution  $\mathcal{U}$ , for a random sample  $S_1' \sim \mathcal{U}^d$ , in expectation at least  $\alpha$  of the examples  $(x, y) \in S$  satisfy that  $y \in \mu_1(x)$ , where  $\mu_1$  is the output of  $\mathcal{L}$  over  $S_1'$ . In particular, there exists  $S_1'$  for which the corresponding  $\mu_1$  covers at least  $\alpha m$  examples. Algorithmically, it can be found via a brute-force search over all possible  $m^d$  subsets  $S_1' \subseteq S$ . Next, remove from  $S$  all examples  $(x, y)$  for which  $y \in \mu_1(x)$  and repeat the same reasoning on the remaining sample. This way at each step  $j$  we find a sample  $S_j'$  and a list
$\mu_{j} = \mathcal{L}(S_{j}^{\prime})$  that covers at least an  $\alpha$ -fraction of the remaining examples. After  $q$  steps, all examples in S are covered since  $|S_q|\leq (1 - \frac{1}{d + 1})^q m < e^{-\frac{q}{d + 1}}m < 1$ .
The final list returned by our method is  $\mu$  defined by the concatenation:
$$
\mu (x) = \bigcup_ {j = 1} ^ {q} \mu_ {j} (x),
$$
and is based only on the examples in  $S_{q}^{\prime}, \ldots, S_{q}^{\prime}$  which is a total of  $r$  examples as claimed.
C.2 Constructing a weak learner for wrong labels
The following lemmas demonstrate how a bound on the dimensions helps one greedily construct a list orientation of small maximum  $(p - 1)$ -out-degree, for a class with  $p$  labels.
Lemma 10. Let  $p, n \in \mathbb{N}$  and assume  $\mathcal{H} \subseteq [p]^n$  is a hypothesis class with a  $(p - 1)$ -DS dimension at most  $d$ , where  $d < n$ . Then, there exists an orientation  $\sigma^{p - 1}$  of  $\mathcal{G}(\mathcal{H})$  with  $\text{outdeg}^{\bar{p} - 1}(\sigma^{p - 1}) \leq 4(p - 1)^2 d$ .
Proof. Notice that  $\mathcal{H}$  is finite, and so the number of vertices in  $\mathcal{G}(\mathcal{H})$  is finite. We consider another notion of dimension called the  $k$ -Exponential dimension. We say that  $S \subseteq \mathcal{X}$  is  $k$ -exponential shattered by  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  if  $|\mathcal{H}|_S| \geq (k + 1)^{|S|}$ . The  $k$ -exponential dimension  $d_E^k(\mathcal{H})$  is the maximum size of an  $k$ -exponential shattered sequence.
Notice that for the case that  $\mathcal{V} = [p]$  and  $k = p - 1$ , we have that a set  $S$  is  $(p - 1)$ -exponential shattered if  $|\mathcal{H}|_S| \geq p^{|S|}$ . Observe that this definition coincides with that of the  $(p - 1)$ -DS dimension shattering. That is, a set  $S$  is  $(p - 1)$ -DS shattered if for every  $h \in \mathcal{H}|_S$  and every  $i \in S$  it holds that  $h$  has exactly  $p - 1$  neighbors that agree with it on  $[n] \setminus \{i\}$ . This definition implies that  $|\mathcal{H}|_S| \geq p^{|S|}$ .
Therefore, for  $\mathcal{H} \subseteq [p]^n$  we have that
$$
d _ {D S} ^ {p - 1} (\mathcal {H}) = d _ {E} ^ {p - 1} (\mathcal {H}).
$$
Then, by Corollary 6.5 of [8] we get that there is an orientation of  $\mathcal{G}(\mathcal{H})$  with maximum  $(p - 1)$ -out-degree at most  $4(p - 1)^2 d_E^{p - 1}(\mathcal{H})$ . Replacing  $d_E^{p - 1}(\mathcal{H})$  by  $d_{DS}^{p - 1}(\mathcal{H})$  yields the desired bound.
The following lemma is analogous to Lemma 8.
Lemma 11. Let  $p \in \mathbb{N}$  and assume  $\mathcal{H} \subseteq [p]^{\mathcal{X}}$  is a hypothesis class with a  $(p - 1)$ -DS dimension at most  $d$ . For every  $\mathcal{H}$ -realizable set  $S \in (\mathcal{X} \times [p])^m$ , any  $\mathcal{D}$  distribution over  $S$ , denote by  $\mu_U^{p - 1}: \mathcal{X} \mapsto [p]^{p - 1}$  the  $p - 1$ -list that is the output of Algorithm 5 when given a sample  $U \sim \mathcal{D}^{4pd}$ . Then,
$$
\mathbb {E} _ {U \sim \mathcal {D} ^ {4 p d}} \left[ \Pr_ {(x, y) \sim \mathcal {D}} [ y \in \mu_ {U} ^ {p - 1} (x) ] \right] \geq 1 - \frac {1}{4 p}. \tag {29}
$$
Proof. For any  $U = \{(x_1, y_1), \ldots, (x_n, y_n)\}$  where  $n = 4pd$ , and any text point  $(x, y) \sim \mathcal{D}$ , let  $H \subseteq [p]^{n+1}$  be the class of all patterns over the unlabeled data  $\mathcal{H}|_{(x_1, \ldots, x_n, x)}$ , as in Algorithm 5. Notice that by Lemma 10, we have that for any such class  $H$ , there is an orientation with a maximal  $(p-1)$ -out-degree of at most  $d$ . Therefore, by Lemma 7 we get that,
$$
\begin{array}{l} \mathbb {E} _ {U \sim \mathcal {D} ^ {4 p d}} \left[ \Pr_ {(x, y) \sim \mathcal {D}} [ y \in \mu_ {U} ^ {p - 1} (x) ] \right] = \Pr_ {(U, (x, y)) \sim \mathcal {D} ^ {4 p d + 1}} \left[ y \in \mu_ {U} ^ {p - 1} (x) \right] \\ \geq 1 - \frac {d}{4 p d + 1} = \frac {d (4 p - 1) + 1}{4 p d + 1} > (4 p - 1) / 4 p. \\ \end{array}
$$
Theorem 8. Let  $p \in \mathbb{N}$  and assume  $\mathcal{H} \subseteq [p]^{\mathcal{X}}$  is a hypothesis class with a  $(p - 1)$ -DS dimension at most  $d$ , and let  $S \in (\mathcal{X} \times [p])^m$  be an  $\mathcal{H}$ -realizable set of examples. Denote by  $\mathcal{L}$  a  $(p - 1)$ -list learning algorithm as given in Algorithm 5, and let  $r = 4pd \cdot \lceil 8p^2 \ln(2m) \rceil$ .
Then, there exists an algorithm  $\mathcal{A}$  based on an  $m\to r$  compression scheme for  $(p - 1)$ -lists such that, when provided with  $S$  and given oracle access to  $\mathcal{L}$ , the algorithm  $\mathcal{A}$  yields a list function  $\mu :\mathcal{X}\rightarrow [p]^{p - 1}$  that is consistent with  $S$ .
Proof. Let  $D$  denote any distribution over the  $m$  examples in  $S$ . For a random sample  $U \sim \mathcal{U}^{4pd}$  denote  $\mu_U = \mathcal{L}(U)$ . By Lemma 11 we have that in expectation over the random choice of  $U$ , it holds that  $\operatorname*{Pr}_{(x,y) \sim D}[y \in \mu_U(x)] \geq 1 - \frac{1}{4p}$ . Therefore, there exists a particular set  $U$  of size  $4pd$  for which this holds as well. Algorithmically, this particular set can be found via a brute-force search over all possible  $m^{4pd}$  subsets  $U \subseteq S$  of size  $4pd$ . We now have a  $(p-1)$ -list function  $\mu_U = \mathcal{L}(U)$  such that with probability 1 satisfies  $\operatorname*{Pr}_{(x,y) \sim D}[y \in \mu_U(x)] \geq 1 - \frac{1}{4p}$ .
Notice that while  $\mu_U$  covers a large fraction of the examples, it is not entirely consistent with  $S$ . The remainder of the proof will be concerned with boosting the list learner we have described above, into a new list  $\mu : \mathcal{X} \to \mathcal{Y}^{p-1}$  that is indeed consistent with the entire sample.
Towards that end, we first use  $\mu_U$  to define a classifier  $f_U: \mathcal{X} \mapsto [p]$  that predicts the incorrect labels. Specifically, we set  $f_U(x) = \hat{y}$  for  $\hat{y} \in [p] \setminus \mu_U(x)$ . Thus, we have that with probability 1,
$$
\Pr_ {(x, y) \sim D} [ f _ {U} (x) = y ] <   \frac {1}{4 p}. \tag {30}
$$
So far we have shown that for any distribution  $D$  over  $S$  there is a set  $U \in S^{4pd}$  and a corresponding function  $f_U: \mathcal{X} \mapsto [p]$  (constructed from  $\mu_U = \mathcal{L}(U)$  as above), that satisfies Equation (30).