SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Erin L Allwein, Robert E Schapire, and Yoram Singer. Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of machine learning research, 1(Dec):113-141, 2000.
[2] Anastasios N, Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
[3] Ron Appel and Pietro Perona. A simple multi-class boosting framework with theoretical guarantees and empirical proficiency. In International Conference on Machine Learning, pages 186-194. PMLR, 2017.
[4] Oscar Beijbom, Mohammad Saberian, David Kriegman, and Nuno Vasconcelos. Guess-averse loss functions for cost-sensitive multiclass boosting. In International Conference on Machine Learning, pages 586–594. PMLR, 2014.
[5] Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff. A characterization of multiclass learnability. arXiv preprint arXiv:2203.01550, 2022.
[6] Nataly Brukhim and Elad Hazan. Online boosting with bandit feedback. In Algorithmic Learning Theory, pages 397-420. PMLR, 2021.
[7] Nataly Brukhim, Elad Hazan, Shay Moran, and Robert E. Schapire. Multiclass boosting and the cost of weak learning. In NIPS, 2021.
[8] Moses Charikar and Chirag Pabbaraju. A characterization of list learnability. arXiv preprint arXiv:2211.04956, 2022.
[9] Amit Daniely and Shai Shalev-Shwartz. Optimal learners for multiclass problems. In  $COLT$ , pages 287-316, 2014.
[10] Sally Floyd and Manfred Warmuth. Sample compression, learnability, and the vapnikchervonenkis dimension. Machine learning, 21(3):269-304, 1995.
[11] Yoav Freund. Boosting a weak learning algorithm by majority. In Mark A. Fulk and John Case, editors, Proceedings of the Third Annual Workshop on Computational Learning Theory, COLT 1990, University of Rochester, Rochester, NY, USA, August 6-8, 1990, pages 202-216. Morgan Kaufmann, 1990.
[12] Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci., 55(1):119-139, 1997.
[13] David Haussler, Nick Littlestone, and Manfred K Warmuth. Predicting  $\{0, 1\}$ -functions on randomly drawn points. Information and Computation, 115(2):248-292, 1994.
[14] M. Kearns. Thoughts on hypothesis boosting. Unpublished, December 1988.
[15] Balázs Kégl. The return of adaboost. mh: Multi-class hamming trees. ICLR, 2014.
[16] Vitaly Kuznetsov, Mehryar Mohri, and Umar Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems, pages 2501-2509, 2014.
[17] Nick Littlestone and Manfred Warmuth. Relating data compression and learnability. Unpublished manuscript, 1986.
[18] Shay Moran, Ohad Sharon, and Iska Tsubari. List online classification. arXiv preprint arXiv:2303.15383, 2023.
[19] Indraneel Mukherjee and Robert E Schapire. A theory of multiclass boosting. Journal of Machine Learning Research, 14:437-497, 2011.
[20] Benjamin Rubinstein, Peter Bartlett, and J Hyam Rubinstein. Shifting, one-inclusion mistake bounds and tight multiclass expected risk bounds. In NIPS, pages 1193-1200, 2006.
[21] Mohammad J Saberian and Nuno Vasconcelos. Multiclass boosting: Theory and algorithms. In Advances in Neural Information Processing Systems, pages 2124-2132, 2011.
[22] Robert E. Schapire. The strength of weak learnability. Machine Learning, 5(2):197-227, 1990.
[23] Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. Cambridge university press, 2012.
[24] Robert E Schapire and Yoram Singer. Improved boosting algorithms using confidence-rated predictions. Machine learning, 37(3):297-336, 1999.
[25] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
[26] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
[27] John von Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100(1):295-320, 1928.
[28] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
[29] Shaodan Zhai, Tian Xia, and Shaojun Wang. A multi-class boosting method with direct optimization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 273-282, 2014.
A Missing proofs of Section 2
Proof of Lemma 1. First, standard analysis of the Hedge algorithm [12] implies that,
$$
\sum_ {t = 1} ^ {T} \Pr_ {i \sim \mathcal {D} _ {t}} \left[ h _ {t} \left(x _ {i}\right) = y _ {i} \right] \leq \frac {\ln (m)}{\eta} + \eta T + H \left(x _ {i}, y _ {i}\right). \tag {6}
$$
For completeness we re-prove it here. Denote  $\phi_t = \sum_{i=1}^m w_t(i)$  and  $\alpha_t = \operatorname{Pr}_{i \sim \mathcal{D}_t}[h_t(x_i) = y_i]$ , and observe that,
$$
\begin{array}{l} \phi_ {t + 1} = \sum_ {i = 1} ^ {m} w _ {t + 1} (i) = \sum_ {i = 1} ^ {m} w _ {t} (i) e ^ {- \eta \mathbb {1} \left[ h _ {t} \left(x _ {i}\right) = y _ {i} \right]} = \phi_ {t} \sum_ {i = 1} ^ {m} \mathcal {D} _ {t} (i) e ^ {- \eta \mathbb {1} \left[ h _ {t} \left(x _ {i}\right) = y _ {i} \right]} \\ \leq \phi_ {t} \sum_ {i = 1} ^ {m} \mathcal {D} _ {t} (i) \left(1 - \eta \mathbb {1} \left[ h _ {t} \left(x _ {i}\right) = y _ {i} \right] + \eta^ {2}\right) = \phi_ {t} \left(1 - \eta \alpha_ {t} + \eta^ {2}\right) \leq \phi_ {t} e ^ {- \eta \alpha_ {t} + \eta^ {2}}, \\ \end{array}
$$
where the two inequalities follow from  $e^{-x} \leq 1 - x + x^2$  for  $x \geq 0$ , and  $1 + x \leq e^x$ , respectively. Therefore, after  $T$  rounds we get  $\phi_T \leq me^{-\eta} \sum_{t=1}^{T} \alpha_t + T \eta^2$ . Furthermore, for every  $i \in [m]$  we have,
$$
e ^ {- \eta H (x _ {i}, y _ {i})} = w _ {T} (i) \leq \phi_ {T} \leq m e ^ {- \eta \sum_ {t = 1} ^ {T} \alpha_ {t} + T \eta^ {2}}.
$$
Taking logarithms and re-arranging we get Equation (6), as needed.
Next, we lower bound the left-hand side with  $\frac{1}{k} + \gamma$  by the weak learning guarantee. By also substituting  $\eta = \sqrt{\frac{\ln(m)}{2T}}$ , we get,
$$
\frac {1}{k} + \gamma - \sqrt {\frac {2 \ln (m)}{T}} \leq \frac {H \left(x _ {i} , y _ {i}\right)}{T}.
$$
Plugging in  $T$  yields the desired bound.
To prove the second part of the lemma, observe that by its first statement it holds that for each  $(x,y)\in S$  there must be a label  $\ell \neq y$  for which  $H(x,\ell) < T / k$ .
The following lemma proves that Algorithm 2 constructs a good initial hint to the boosting algorithm (Algorithm 3).
Lemma 4 (Initial hint). Let  $m_0, \gamma > 0$ , and let  $S \in (\mathcal{X} \times \mathcal{Y})^m$ . Given oracle access to an empirical  $\gamma$ -BRG learner for  $S$  (Definition 4), when run with  $p = \lceil \ln(m) / \gamma \rceil$ , Algorithm 2 outputs  $\mu$  of size  $p$ , such that for all  $(x, y) \in S$  it holds that  $y \in \mu(x)$ .
Proof. Observe that for all  $j \in [p]$  the output of the weak learner satisfies,
$$
\Pr_ {(x, y) \sim \mathcal {U} _ {j}} [ h _ {j} (x) = y ] \geq \gamma , \tag {7}
$$
where this follows from Definition 4. This implies that  $|S_p| \leq (1 - \gamma)^p m < e^{-\gamma p}m < 1$ .
We say that a list function  $\mu : \mathcal{X} \to \mathcal{Y}^k$  is consistent with a sample  $S \in (\mathcal{X} \times \mathcal{Y})^m$  if for every  $(x, y) \in S$ , it holds that  $y \in \mu(x)$ .
We now define a compression scheme for lists as follows. Specifically, we say that an algorithm  $\mathcal{A}$  is based on an  $m\to r$  compression scheme for  $k$ -lists if there exists a reconstruction function  $\rho :(\mathcal{X}\times \mathcal{Y})^r\to (\mathcal{Y}^k)^\mathcal{X}$ , such that when  $\mathcal{A}$  is provided with a set  $S\in (\mathcal{X}\times \mathcal{Y})^m$  of training examples, it somehow chooses (possibly in a randomized fashion) a subset  $S^{\prime}\subsetneq S$  of size  $r$ , and outputs a  $k$ -list function  $\mu = \rho (S^{\prime})$ . Denote its error with respect to a distribution  $\mathcal{D}$  over  $\mathcal{X}\times \mathcal{Y}$  as,
$$
\operatorname {e r r} _ {\mathcal {D}} (\mu) = \Pr_ {(x, y) \sim \mathcal {D}} \left[ y \notin \mu (x) \right].
$$
Next, we prove a generalization bound for a compression scheme for  $k$ -lists. The proof follows similarly to [17, 10], extending these classical generalization bounds to the notion of lists.
Theorem 6. Let  $r \leq m$  be integers and  $k < |\mathcal{Y}|$ . Let  $\mathcal{A}$  denote a (possibly randomized) algorithm based on an  $m \to r$  compression scheme for  $k$ -lists. For any  $S \in (\mathcal{X} \times \mathcal{Y})^m$  denote by  $f_S$  the output of  $\mathcal{A}$  over  $S$ . Let  $\delta > 0$  and set  $\epsilon = \frac{r\ln(m) + \ln(1 / \delta)}{m - r}$ . Then, for any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , with probability at least  $1 - \delta$  it holds that if  $f_S$  is consistent with  $S$  then,  $\mathrm{err}_{\mathcal{D}}(f_S) \leq \epsilon$ .
Proof. We want to show that,
$$
\Pr_ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ f _ {S} \text {i s c o n s i s t e n t w i t h} S \Rightarrow \operatorname {e r r} _ {\mathcal {D}} \left(f _ {S}\right) \leq \epsilon \right] \geq 1 - \delta . \tag {8}
$$
In other words, we want to show that,
$$
\Pr_ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ f _ {S} \text {i s c o n s i s t e n t w i t h} S \wedge \operatorname {e r r} _ {\mathcal {D}} \left(f _ {S}\right) > \epsilon \right] \leq \delta . \tag {9}
$$
The equivalence of Equations (8) and (9) is because if  $A$  and  $B$  are events, then  $A \Rightarrow B$  is exactly equivalent to  $(\neg A) \lor B$  whose negation is  $A \land (\neg B)$ .
We overload notation and for any  $S' \in (\mathcal{X} \times \mathcal{Y})^r$  we denote  $f_{S'} := \rho(S')$ , where  $\rho$  is the reconstruction function associated with  $\mathcal{A}$ .
First, we fix indices  $i_1,\dots,i_r\in [m]$ . Given the random training set  $S$ . Observe that for any list function  $f:\mathcal{X}\to \mathcal{Y}^k$  that is  $\epsilon$ -bad, then the probability of also having  $f$  be consistent with  $S\setminus S'$  is at most  $(1 - \epsilon)^{m - r}$ . This holds for any such  $f$ , and in particular  $f_{S'}$  if it happens to be  $\epsilon$ -bad. Since this also holds for any other fixed choice of indices  $i_1,\dots,i_r\in [m]$ , we can bound the above for all possible sequences of indices. Therefore, by the union bound, since there are  $m^r$  choices for these indices, we have that for any  $f_S$  chosen by  $\mathcal{A}$  it holds that,
$$
\begin{array}{l} \Pr_ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ f _ {S} \text {c o n s i s t e n t w i t h} S \wedge f _ {S} \text {i s} \epsilon - \text {b a d} \right] (10) \\ \leq \Pr_ {S \sim \mathcal {D} ^ {m}} \left[ \exists S ^ {\prime} \in S ^ {r}: f _ {S ^ {\prime}} \text {c o n s i s t e n t w i t h} S \backslash S ^ {\prime} \wedge f _ {S ^ {\prime}} \text {i s} \epsilon - \text {b a d} \right] (11) \\ \leq \sum_ {i _ {1} \dots i _ {r} \in [ m ]} \Pr_ {S ^ {\sim} \mathcal {D} ^ {m}} \left[ f _ {S ^ {\prime}} \text {c o n s i s t e n t w i t h} S \backslash S ^ {\prime} \wedge f _ {S ^ {\prime}} \text {i s} \epsilon - \text {b a d} \right] (12) \\ = m ^ {r} \cdot \Pr_ {\substack {S ^ {\prime \prime} \sim \mathcal {D} ^ {m - r} \\ S ^ {\prime} \sim \mathcal {D} ^ {r}}} \left[ f _ {S ^ {\prime}} \text {consistent with } S ^ {\prime \prime} \wedge f _ {S ^ {\prime}} \text {is} \epsilon \text {-bad} \right] (13) \\ = m ^ {r} \cdot \Pr_ {\substack {S ^ {\prime \prime} \sim \mathcal {D} ^ {m - r} \\ S ^ {\prime} \sim \mathcal {D} ^ {r}}} \left[ f _ {S ^ {\prime}} \text {consistent with } S ^ {\prime \prime} \mid f _ {S ^ {\prime}} \text {is} \epsilon \text {-bad} \right] \cdot \Pr_ {S ^ {\prime} \sim \mathcal {D} ^ {r}} \left[ f _ {S ^ {\prime}} \text {is} \epsilon \text {-bad} \right] (14) \\ \leq m ^ {r} \cdot \Pr_ {\substack {S ^ {\prime \prime} \sim \mathcal {D} ^ {m - r} \\ S ^ {\prime} \sim \mathcal {D} ^ {r}}} \left[ f _ {S ^ {\prime}} \text {consistent with } S ^ {\prime \prime} \mid f _ {S ^ {\prime}} \text {is} \epsilon \text {-bad} \right] (15) \\ = m ^ {r} \cdot \mathbb {E} _ {S ^ {\prime} \sim \mathcal {D} ^ {r}} \left[ \Pr_ {S ^ {\prime \prime} \sim \mathcal {D} ^ {m - r}} \left[ f _ {S ^ {\prime}} \text {i s c o n s i s t e n t w i t h} S ^ {\prime \prime} \right] \mid f _ {S ^ {\prime}} \text {i s} \epsilon - \text {b a d} \right] (16) \\ \leq m ^ {r} (1 - \epsilon) ^ {m - r}, (17) \\ \end{array}
$$
where in Equation (12) we use the notation  $S' = \{(x_{i_1}, y_{i_1}), \dots, (x_{i_r}, y_{i_r})\}$ , and where the last inequality holds since for any particular selection of the examples  $S' \in (\mathcal{X} \times \mathcal{Y})^r$  for which  $f_{S'}$  is  $\epsilon$ -bad, we have that  $\operatorname{Pr}_{S'' \sim \mathcal{D}^{m-r}}[f_{S'}$  is consistent with  $S''] \leq (1 - \epsilon)^{m-r}$ . This means that it also holds true if these examples are selected at random. Then, we get the desired bound since  $m^r (1 - \epsilon)^{m-r} \leq m^r e^{-\epsilon (m-r)} = \delta$ , which follows by solving for  $\delta$  via our definition of  $\epsilon$  above.
Theorem 7. Let  $r \leq m$  be integers and  $k < |\mathcal{Y}|$ . Let  $\mathcal{A}$  denote a (possibly randomized) algorithm based on an  $m \to r$  compression scheme for  $k$ -lists. For any  $S \in (\mathcal{X} \times \mathcal{Y})^m$  denote by  $f_S$  the output of  $\mathcal{A}$  over  $S$ . Let  $\delta > 0$  and set  $\epsilon = \frac{r\ln(m) + \ln(1 / \delta)}{m - r}$ . Then, for any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ ,
$$
\Pr_ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ \operatorname {e r r} _ {\mathcal {D}} \left(f _ {S}\right) > \epsilon \right] \leq \Pr_ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ f _ {S} \text {i s n o t c o n s i s t e n t w i t h} S \right] + \delta . \tag {18}
$$
Proof. Let  $C_A(S) = \mathbf{1} \left[ f_S \text{ is consistent with } S \right]$ . First, using the simple fact that for any two events  $A$  and  $B$  it holds that  $\operatorname{Pr}[A] = \operatorname{Pr}[A \wedge B] + \operatorname{Pr}[A \wedge \neg B] \leq \operatorname{Pr}[A \wedge B] + \operatorname{Pr}[\neg B]$ , we get,
$$
\operatorname * {P r} _ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ f _ {S} \text {i s} \epsilon \text {- b a d} \right] \leq \operatorname * {P r} _ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ C _ {\mathcal {A}} (S) \wedge f _ {S} \text {i s} \epsilon \text {- b a d} \right] + \operatorname * {P r} _ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ \neg C _ {\mathcal {A}} (S) \right].
$$
Thus, it remains to prove that,
$$
\operatorname * {P r} _ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ C _ {\mathcal {A}} (S) \wedge f _ {S} \text {i s} \epsilon - \text {b a d} \right] \leq \delta , \tag {19}
$$
which holds true by Theorem 6 above.
Algorithm 4 Boosting Weak-to-List Learning
Given: training data  $S\in (\mathcal{X}\times \mathcal{Y})^m$  , edge  $\gamma >0$  parameters  $T,\eta >0$
Output: A list  $\mu : \mathcal{X} \mapsto \mathcal{Y}^{k-1}$ .
1: Let  $k$  be the smallest integer such that  $\frac{1}{k} < \gamma$ .
2: Call Hedge (Algorithm 1) with  $S$  and parameters  $\eta, T$  to get  $H: \mathcal{X} \times \mathcal{Y} \mapsto [0, T]$ .
3: Construct  $\mu : \mathcal{X}|_{S} \mapsto \mathcal{Y}^{k-1}$  such that for all  $x$ ,
$$
\mu (x) = \left\{\ell \in \mathcal {Y}: H (x, \ell) > \frac {T}{k} \right\}.
$$
4: Output  $\mu : \mathcal{X} \to \mathcal{Y}^{k-1}$ .  $\backslash$  For  $x \notin S$ , truncate  $\mu(x)$  to first  $k-1$  labels, if contains more.
The following lemma describes the list-learning guarantees that can be obtained when given access to a  $\gamma$ -weak learner (Definition 6). It is a re-phrasing of Lemma 2 from the main paper, using exact constants. The next paragraph highlights the assumptions that were made, followed by the proof of the lemma.
Specifically, we assume for simplicity that the learning algorithm does not employ internal randomization. Thus, it can be regarded as a fixed, deterministic mapping from a sequence of  $m_0$  unweighted examples, to a hypothesis  $h: \mathcal{X} \to \mathcal{Y}$ . We note that our results remain valid for a randomized learner as well, yet we assume the above for ease of exposition.
Lemma 5 (Weak-to-List Boosting). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be a hypothesis class. Let  $\gamma, \epsilon, \delta > 0$ , let  $k$  be the smallest integer such that  $\frac{1}{k} < \gamma$ , and denote  $\sigma = \gamma - \frac{1}{k}$ . Let  $\mathcal{D}$  be an  $\mathcal{H}$ -realizable distribution over  $\mathcal{X} \times \mathcal{Y}$ . Then, when given a sample  $S \sim \mathcal{D}^{m}$  for  $m \geq \frac{16 m_{0} \ln^{2}(m)}{\sigma^{2} \epsilon} + \frac{\ln(2 / \delta)}{\epsilon}$ , oracle access to a  $\gamma$ -weak learner (Definition 6) for  $\mathcal{H}$  with  $m_{0} \geq m_{0}(\frac{\delta}{2T})$  and  $T \geq \frac{8 \ln(m)}{\sigma^{2}}$ , and  $\eta = \sqrt{\frac{\ln(m)}{2T}}$ , Algorithm 4 outputs a list function  $\mu: \mathcal{X} \to \mathcal{Y}^{k-1}$  such that with probability at least  $1 - \delta$ ,
$$
\operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \left[ y \notin \mu (x) \right] \leq \epsilon .
$$
Proof. By the learning guarantee in Definition 6, we know that for every distribution  $D$  over the examples in  $S$ , when given  $m_0$  examples from  $D$ , with probability at least  $1 - \delta/(2T)$  the learner returns  $h: \mathcal{X} \to \mathcal{Y}$  such that
$$
\sum_ {i = 1} ^ {m} D (i) \mathbb {1} [ h (x _ {i}) = y _ {i} ] \geq \gamma = \frac {1}{k} + \sigma .
$$
In line 2 of Algorithm 4, we call Algorithm 1 with  $S$  and the weak learner, with  $T$ ,  $\eta$  and  $m_0$  as defined above. Then, we follow the same proof as in Lemma 1, applied to  $S$  such that the guarantees of the weak learner holds with probability at least  $1 - \delta / (2T)$  as above. Then, taking the union bound over all  $T$  calls to the weak learner, by the same argument of Lemma 1 we get that with probability at least  $1 - \delta / 2$  it holds that for all  $(x, y) \in S$ ,
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {1} [ h _ {t} (x) = y ] \geq \frac {1}{k} + \frac {\sigma}{2}, \tag {20}
$$
where  $h_1, \dots, h_T$  are the weak hypotheses obtained via Algorithm 1 and that comprise of  $H$ , the output of Algorithm 1. Moreover, notice that since the weak learning algorithm can be regarded as a fixed, deterministic mapping from a sequence of  $m_0$  unweighted examples to a hypothesis, hypothesis  $h_t$  can be expressed via these very  $m_0$  examples. Notice that  $H$  can be represented by  $T \cdot m_0$  examples. Thus, the list function  $\mu$  defined in Line 4 of Algorithm 4, can also be represented by  $T \cdot m_0$  examples.
Furthermore, we show that for every  $(x,y)\in S$  the list  $\mu (x)$  is indeed of size at most  $k - 1$ . This is true since for every  $x\in \mathcal{X}$  by the definition of  $H$  it holds that  $\sum_{\ell \in \mathcal{Y}}H(x,\ell)\leq T$ , and so we have that for all  $(x,y)\in S$  it holds that at most  $k - 1$  labels  $\ell \in \mathcal{V}$  can satisfy  $H(x,\ell) > T / k$ , and thus  $|\mu (x)|\leq k - 1$ .
Lastly, we observe that  $\mu$  is consistent with  $S$ . That is, by Equation (20) and the definition of  $\mu$  we know that for every  $(x,y)\in S$ , we have  $y\in \mu (x)$ .
Overall, we get that with probability at least  $1 - T\cdot \delta /(2T) = 1 - \delta /2$  over the random choice of  $Tm_0$  examples, it holds that the list function  $\mu$  is consistent with the sample  $S$ . Moreover, it can be represented using only  $r$  examples, where  $r = Tm_{0}$ .
We can now apply a sample compression scheme bound to obtain the final result. Specifically, we consider the bound given in Theorem 7 for a  $m \to r$  sample compression scheme for  $k$ -lists. Then, if  $\mu$  is consistent with  $S$  then with probability of at least  $1 - \delta / 2$  over the choice of  $S$  it also holds that,
$$
\Pr_ {(x, y) \sim \mathcal {D}} \left[ y \notin \mu (x) \right] \leq \frac {r \ln (m) + \ln (2 / \delta)}{m - r}.
$$
Plugging in  $r = Tm_0$  from above, and setting
$$
m = \frac {8 m _ {0} \ln^ {2} (m)}{\sigma^ {2} \epsilon} + \frac {\ln (2 / \delta)}{\epsilon} + \frac {8 \ln (m) m _ {0}}{\sigma^ {2}},
$$
yields the desired bound. That is, with probability at least  $1 - \delta$ , we have  $\operatorname{Pr}_{\mathcal{D}}[y \notin \mu(x)] \leq \epsilon$ .
Proof of Lemma 3. First, recall that the given oracle is such that for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , when given  $S \sim \mathcal{D}^{m_{\ell}}$  for  $m_{\ell} \geq m_{\ell}(\epsilon, \delta)$ , it returns  $\mu_S: \mathcal{X} \to \mathcal{Y}^k$  such that with probability  $1 - \delta$ ,
$$
\Pr_ {(x, y) \sim \mathcal {D}} [ y \in \mu_ {S} (x) ] \geq 1 - \epsilon . \tag {21}
$$
Then, we show that there is a  $\gamma$ -weak learning algorithm  $\mathcal{W}$  for  $\mathcal{H}$  where  $\gamma := \frac{1 - 2\epsilon}{k}$ , for any  $\epsilon > 0$ . The construction of the algorithm  $\mathcal{W}$  is as follows. First, let  $m_{\ell} := m_{\ell}(\epsilon, \delta_0)$  for  $\delta_0 = 1/2$ . The  $\gamma$ -weak learner  $\mathcal{W}$  has a sample size  $m$  to be determined later such that  $m \geq m_{\ell}$ , and is given a training set  $S \sim \mathcal{D}^m$ . Then, the learner  $\mathcal{W}$  first calls the  $k$ -list learner with parameters  $\epsilon, \delta_0, m_{\ell}$  over  $S$  and obtains the list function  $\mu_S$  that satisfies Equation (21) with probability at least  $1 - \delta_0 = 1/2$ .
Henceforth, we only consider the probability-half event in which we have that Equation (21) holds. Furthermore, let  $Z$  denote the set of all elements  $(x,y) \sim \mathcal{D}$  for which  $y \in \mu_S(x)$ . Then, notice that by randomly picking  $j \leq k$  and denoting the hypothesis  $h_S^j: \mathcal{X} \mapsto \mathcal{Y}$  defined by the  $j$ -the entry of  $\mu_S$ , i.e.,  $h_S^j(x) := \mu_S(x)_j$ , we get for any  $(x,y) \in Z$ ,
$$
\operatorname * {P r} _ {j \sim \mathrm {U n i f} (k)} \big [ y = h _ {S} ^ {j} (x) \big ] \geq \frac {1}{k}.
$$
Then, by von Neumann's minimax theorem [27] it holds that for any distribution  $D'$  over  $Z$ , there exists some particular  $j_0 \leq k$  for which
$$
\operatorname * {P r} _ {(x, y) \sim D ^ {\prime}} \left[ y = h _ {S} ^ {j _ {0}} (x) \right] \geq \frac {1}{k}.
$$
Since this is true for the distribution  $\mathcal{D}$  conditioned on  $Z$  as well, the algorithm can pick  $j \leq k$  uniformly at random and so it will pick  $j_0$  with probability  $\frac{1}{k}$ . Overall we get that when given  $S$  the
algorithm can call the list learner to obtain  $\mu_S$  and sample  $j$  uniformly to obtain  $h_S^j: \mathcal{X} \mapsto \mathcal{Y}$  such that,
$$
\operatorname * {P r} _ {S \sim \mathcal {D} ^ {m _ {\ell}}} \left[ \operatorname * {P r} _ {j \sim \operatorname {U n i f} (k)} \left[ \operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \left[ y = h _ {S} ^ {j} (x) \right] \geq \frac {1 - \epsilon}{k} \right] \geq \frac {1}{k} \right] \geq 1 - \delta_ {0}.
$$
Since we set  $\delta_0 = 1 / 2$ , we get that,
$$
\operatorname * {P r} _ {S \sim \mathcal {D} ^ {m _ {\ell}, j} \sim \operatorname {U n i f} (k)} \left[ \operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \left[ y = h _ {S} ^ {j} (x) \right] \geq \frac {1 - \epsilon}{k} \right] \geq \frac {1}{2 k}. \tag {22}
$$