SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\left[\mathrm{AAC}^{+}23\right]$  Anders Aamand, Alexandr Andoni, Justin Y. Chen, Piotr Indyk, Shyam Narayanan, and Sandeep Silwal. Data structures for density estimation. In Proceedings of the 40th International Conference on Machine Learning. PMLR, 2023.
$\left[\mathrm{ABH}^{+}20\right]$  Hassan Ashtiani, Shai Ben-David, Nicholas J. A. Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. J. ACM, 67(6):32:1-32:42, 2020.
[ABM18] Hassan Ashtiani, Shai Ben-David, and Abbas Mehrabian. Sample-efficient learning of mixtures. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 2679–2686. AAAI Press, 2018.
[ADLS17] Jayadev Acharya, Ilias Diakonikolas, Jerry Li, and Ludwig Schmidt. Sample-optimal density estimation in nearly-linear time. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA, pages 1278-1289. SIAM, 2017.
[AF23] Tomer Adar and Eldar Fischer. Refining the adaptivity notion in the huge object model. CoRR, abs/2306.16129, 2023.
[AFJ+18] Jayadev Acharya, Moein Falahatgar, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Maximum selection and sorting with adversarial comparators. The Journal of Machine Learning Research, 19:59:1-59:31, 2018.
[AJOS14] Jayadev Acharya, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Sorting with adversarial comparators and application to density estimation. In 2014 IEEE International Symposium on Information Theory, pages 1682-1686, 2014.
[AMNW22] Maryam Aliakbarpour, Andrew McGregor, Jelani Nelson, and Erik Waingarten. Estimation of entropy in constant space with improved sample complexity. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32474–32486. Curran Associates, Inc., 2022.
$\left[\mathrm{BBK}^{+}21\right]$  Olivier Bousquet, Mark Braverman, Gillat Kol, Klim Efremenko, and Shay Moran. Statistically near-optimal hypothesis selection. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS 2021, Denver, CO, USA, February 7-10, 2022, pages 909-919. IEEE, 2021.
[BBS22] Gavin Brown, Mark Bun, and Adam Smith. Strong memory lower bounds for learning natural models. In Proceedings of Thirty Fifth Conference on Learning Theory COLT, 2022.
[BKM19] Olivier Bousquet, Daniel Kane, and Shay Moran. The optimal approximation factor in density estimation. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 318–341. PMLR, 25–28 Jun 2019.
[BKSW19] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing
$\left[\mathrm{GKK}^{+}20\right]$  Sivakanth Gopi, Gautam Kamath, Janardhan Kulkarni, Aleksandar Nikolov, Zhiwei Steven Wu, and Huanyu Zhang. Locally private hypothesis selection. In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria], volume 125 of Proceedings of Machine Learning Research, pages 1785-1816. PMLR, 2020.
[GR22] Oded Goldreich and Dana Ron. Testing Distributions of Huge Objects. In Mark Braverman, editor, 13th Innovations in Theoretical Computer Science Conference (ITCS 2022), volume 215 of Leibniz International Proceedings in Informatics (LIPics), pages 78:1-78:19, Dagstuhl, Germany, 2022. Schloss Dagstuhl – Leibniz-Zentrum für Informatik.
[KMV12] Adam Tauman Kalai, Ankur Moitra, and Gregory Valiant. Disentangling gaussians. Commun. ACM, 55(2):113-120, 2012.
[KSS18] Pravesh K. Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via sum of squares. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018, pages 1035-1046. ACM, 2018.
[MS08] Satyaki Mahalanabis and Daniel Stefankovic. Density estimation in linear time. In 21st Annual Conference on Learning Theory - COLT, pages 503-512, 2008.
[Pea95] Karl Pearson. Mathematical contributions to the theory of evolution. ii. skew variation in homogeneous material. [abstract]. Proceedings of the Royal Society of London, 57:257-260, 1895.
[Raz18] Ran Raz. Fast learning requires good memory: A time-space lower bound for parity learning. Journal of the ACM (JACM), 66(1):1-18, 2018.
[SD15] Jacob Steinhardt and John Duchi. Minimax rates for memory-bounded sparse linear regression. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 1564–1587. PMLR, 03–06 Jul 2015.
[Sha14] Ohad Shamir. Fundamental limits of online and distributed algorithms for statistical learning and estimation. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 163-171, 2014.
[SOAJ14] Ananda Theertha Suresh, Alon Orlitsky, Jayadev Acharya, and Ashkan Jafarpour. Near-optimal-sample estimators for spherical gaussian mixtures. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, 2014.
[SSV19] Vatsal Sharan, Aaron Sidford, and Gregory Valiant. Memory-sample tradeoffs for linear regression with small error. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC, pages 890–901. ACM, 2019.
[Vad12] Salil P. Vadhan. Pseudorandomness. Foundations and Trends® in Theoretical Computer Science, 7(1-3):1-336, 2012.
[Yat85] Yannis G. Yatracos. Rates of convergence of minimum distance estimators and kolmogorov's entropy. The Annals of Statistics, 13(2):768 - 774, 1985.
Figure 1: Random ladder tournament
A Random ladder tournament (cont.)
Figure 1 illustrates a visual representation of the algorithm.
A.1 Amplifying the confidence parameter
In this section, we describe how one can amplify the confidence parameter of the random ladder tournament. Note that to keep  $\alpha = 3$ , we should avoid any two-step process that comes up with a collection of hypotheses to narrow down the possible choices and then pick an acceptable hypothesis among them. Such approaches generally lead to  $\alpha = 9$  or higher. Here, we argue that if we run the random ladder tournament for a longer time, a random hypothesis in  $Q$  will be acceptable with very high probability. In particular, if we set  $k$ , the number of steps equal to  $\Theta(1 / (\delta p_0))$  (instead of setting  $k = \Theta(1 / p_0)$ ), we find an acceptable hypothesis with probability at least  $1 - \delta$ .
Corollary A.1. Suppose that we can draw i.i.d. random hypotheses from an arbitrary meta distribution  $\mathcal{D}$  over  $\mathcal{H}$ . And, we have a hypothesis  $P$  that we aim to learn properly in  $\mathcal{H}$ . Assume we are given parameters  $p_0$  and  $\Gamma$  such that the probability that a random hypothesis drawn from the meta distribution is  $\Gamma$ -close to  $P$  is at least  $p_0$ . For any  $\epsilon, \delta \in (0,1)$ , Algorithm 2 can be modified to an  $(\alpha = 3,\epsilon,\delta)$ -proper learner (with promise) for class  $\mathcal{H}$  by setting  $k := \lceil 2\lceil 2 / \delta\rceil /p_0\rceil = \Theta (1 / (\delta p_0))$ .
Proof. Note that  $\delta$  in Theorem 4 is 0.1. Hence, it suffices to focus on  $\delta < 0.1$ . As we have shown in the proof of Theorem 4, the expected number of unacceptable hypotheses in  $Q$  is at most one. Let  $\hat{H}$  be a random hypothesis in  $Q$ . Let  $t \coloneqq \lceil 2 / \delta \rceil$ , and set  $k$  to  $\lceil 2t / p_0 \rceil$ . It is not hard to see that the probability of  $\hat{H}$  being unacceptable is bounded by  $\delta$ :
$$
\begin{array}{l} \Pr \left[ \hat {H} \text {i s} (3 \Gamma + \epsilon) - \text {f a r} \right] = \Pr \left[ \hat {H} \text {i s u n a c c e p t a b l e} (3 \Gamma + \epsilon) - \text {f a r} \right] \\ = \Pr [ \text {a r a n d o m h y p o t h e s i s i n} Q \text {i s u n a c c e p t a b l e} ] \\ = \sum_ {\ell = 1} ^ {k + 1} \frac {\mathbf {E} [ \# \text {u n a c c e p t a b l e h y p o t h e s e s i n} Q \mid | Q | = \ell ]}{\ell} \cdot \Pr [ | Q | = \ell ]. \\ \end{array}
$$
Now, we separate the first  $t$  terms in the summation, and bound them from above by the probability of  $|Q|$  being at most  $t$ . Moreover, we use the fact that the size of  $|Q|$  is essentially a binomial random variable drawn from  $\mathbf{Bin}(k,p_0)$  plus one. Hence, we obtain:
$$
\begin{array}{l} \mathbf {P r} \left[ \hat {H} \text {i s} (3 \Gamma + \epsilon) \text {- f a r} \right] \leq \mathbf {P r} [ | Q | \leq t ] + \sum_ {\ell = t + 1} ^ {k + 1} \frac {\mathbf {E} [ \# \text {u n a c c e p t a b l e h y p o t h e s e s i n} Q \mid | Q | = \ell ]}{t} \cdot \mathbf {P r} [ | Q | = \ell ] \\ \leq \Pr [ \mathbf {B i n} (k, p _ {0}) <   t ] + \frac {\mathbf {E} [ \# \text {u n a c c e p t a b l e h y p o t h e s e s i n} Q ]}{t} \\ \leq \Pr \left[ \operatorname {B i n} (k, p _ {0}) <   \frac {k \cdot p _ {0}}{2} \right] + \frac {1}{t} \\ \leq \exp \left(- \frac {k \cdot p _ {0}}{8}\right) + \frac {\delta}{2} \leq \exp \left(- \frac {1}{2 \delta}\right) + \frac {\delta}{2} \leq \frac {3 \delta}{4}, \\ \end{array}
$$
(By the Hoeffding inequality)
where the last inequality holds due to the fact that  $e^{-1 / (2\delta)} < \delta / 4$  for  $\delta \leq 0.1$ . Hence, the proof is complete.
A.2 Proofs of Memory-data tradeoffs of random ladder tournament
Proof. Recall that it suffices to draw  $m \coloneqq O(\log p_0^{-1} / \epsilon^2)$  samples to make sure all the comparisons we make in Line 7 are accurate with probability at least 0.99. Since we obtain this accuracy bound via the union bound, we may reuse the samples. The idea here is to process every  $t$  hypotheses together with the same set of  $m$  samples. To implement this algorithm in  $b$  bits of memory, we first draw  $t$  random hypotheses and store their indices using  $O(t \log n)$  bits. Then, we draw  $m$  sample and store the Scheffé counts of every pair of hypotheses that are drawn. Counting the number of samples in each Scheffé sets takes  $O(\log m) = O(\log (\log n / \epsilon))$  bits. And, there are  $O(t^2)$  many of them. Now, we can make all the comparisons between the  $t$  hypotheses using  $O(t^2 \log m) = O(t^2 \log (\log p^{-1} / \epsilon))$  bits. Now, we run the first  $t$  steps of Algorithm 2 and compute the current winner up to round  $t$ ,  $w_t$ . Then, repeat this process for  $w_t$  and the next  $t - 1$  random hypotheses with fresh samples until we reach the end of  $k$  hypotheses. Since at each round we process  $t - 1$  hypotheses (with the exception of the first round), the number of rounds is  $\lceil (k - 1) / (t - 1) \rceil = O(1 / (tp_0))$ , so we use  $O(m / (tp_0))$  samples in total. Hence, Algorithm 2 uses  $O(\log n / (t\epsilon^2 p_0))$  samples and  $b = O(t^2 \log (\log n / \epsilon) + t \log n)$  bits of memory overall.
Figure 2: The sketch of our approach for the main result
Proof. The proof is very similar to the proof of Lemma 3.1. However, instead of using the Scheffé counts we use the sorted lists to store the samples. The only difference is that to process  $t$  hypotheses, we will need  $O(mt\log n) = O(t\log p_0^{-1}\log n / \epsilon^2)$  bits instead of  $O(t^{2}\log m)$  bits. This observation concludes the proof.
B Main result
In this section, we present an algorithm that achieves the tradeoff described in our main theorem. As we have described in Section 1.2, we obtain our main result by running the random ladder tournament on the hypotheses filtered by a filtering algorithm.
Recall that in the random ladder tournament, the algorithm can draw a random hypothesis such that with a probability of at least  $p_0$ , it is a  $\Gamma$ -close hypothesis. The number of comparisons made in the random ladder tournament is approximately  $k \coloneqq \Theta(1/p_0)$ . To reduce  $k$ , we design a randomized filtering algorithm that produces an acceptable hypothesis (( $3\Gamma + \epsilon'$ )-close to  $P$ ) with a certain descent probability. Roughly speaking, this probability is  $\min(1/\sqrt{n}, b/n)$ . Although this probability is not very high, it is still greater than  $1/n$ , which allows us to execute the random ladder in fewer steps. The only caveat is that we must now run the random ladder tournament with a new parameter  $\Gamma' \coloneqq 3\Gamma + \epsilon'$ . Hence, the accuracy guarantee of the resulting hypothesis will also be based on  $\Gamma'$ , and we will obtain a  $(9\Gamma + \Theta(\epsilon'))$ -close hypothesis to  $P$ . Figure 2 illustrates a visual representation of this approach.
The filtering algorithm operates on the basis of a simple observation: the excellent hypothesis  $H^{*}$  will not lose to any unacceptable hypotheses (those that are  $(3\Gamma + \epsilon')$ -far). Therefore, if there is no decent hypothesis, one can identify the excellent hypothesis  $H^{*}$  through an elimination tournament, where the algorithm pairs all hypotheses, compares them, and repeats this process while only the winners advance to the next round. Conversely, if numerous decent hypotheses exist, it suggests that a random hypothesis will have a reasonable chance of being decent (and hence acceptable). The
filtering algorithm combines these two strategies and does the following for each with probability a half:
1. It outputs a random hypothesis.
2. It runs an elimination tournament under the assumption that there are no decent hypotheses, then it outputs the result.
This approach leads to finding an excellent or decent hypothesis with an acceptable chance regardless of how scarce the decent hypotheses are.
Note that to run an elimination algorithm, we need to memorize the winners of each round. Hence, implementing the second part above with limited memory is not trivial. Therefore, we focus on a randomly sampled subset of elements and provide an algorithm that is extremely memory efficient. Roughly speaking, the algorithm processes  $n$  hypotheses using only  $O(n)$  bits of memory. This is one of the key components of our result that allows us to avoid extra  $\log n$  factors and achieve the tradeoff  $s \cdot b \approx O(n \log n)$ .
The algorithm operates in rounds, employing a tree-like structure. In each round of this algorithm, we group a carefully chosen number of hypotheses and run an all-against-all tournament among them. Next, the winner of each group proceeds to the next round. We continue this process until we find a single winner.
In Section B.1, we describe the group elimination tournament. In Section B.2, we detail how to conduct the group elimination tournament on a random set of  $n'$  hypotheses while utilizing only  $n'$  bits of memory. In Section B.3, we outline the filtering algorithm. Finally, we conclude with Section B.4, which combines all the pieces together and presents the proof of our main result.
B.1 Group elimination tournament
In this section, we focus on the case that there is no decent hypothesis in  $\mathcal{H}$ . That is, every hypothesis that is OPT-far from  $P$  will lose to a hypothesis  $H^{*}$  with  $\| H^{*} - P\|_{\mathrm{TV}} = \mathrm{OPT}$ . The main advantage of this assumption is that if we have a knockout style tournament: only an OPT-close hypothesis will survive to the top. Here, we propose an algorithm that implements a knockout style tournament, which we call group elimination tournament. However, instead of comparing two hypotheses at every step, we run the all-go-against-all tournament among a small group of hypotheses and send the winner to the next round. For simplicity, in this section, we describe our approach in Algorithm 3 where we assume no memory restriction. We prove the performance of this algorithm in Theorem 5. For a discussion on how to implement this algorithm using roughly  $O(n)$  bits of memory, see Section B.2.
Algorithm 3 Hypothesis selection with no decent hypothesis
1: procedure HYPOTHESIS-SELECTION-NO-DECENT  $(\mathcal{H},n,\epsilon ,\delta$  sample access to  $P$    
2:  $\mathcal{H}_1\gets \mathcal{H}$    
3: for  $\ell = 1,2,\ldots ,L$  do   
4:  $\mathcal{G}_1,\mathcal{G}_2,\dots ,\mathcal{G}_{s_{\ell +1}}\gets$  Partition  $\mathcal{H}_{\ell}$  into groups of size  $g_{\ell}$    
5:  $\mathcal{H}_{\ell +1}\gets \emptyset$    
6:  $S\coloneqq \{(i,j)|\exists \mathcal{G}\in \{\mathcal{G}_1,\mathcal{G}_2,\dots ,\mathcal{G}_{s_{\ell +1}}\}$  such that  $H_{i}\in \mathcal{G}$  and  $H_{j}\in \mathcal{G}\}$    
7: Draw  $m_{\ell}$  samples. For each sample, update the Scheffé counts of  $S(H_i,H_j)$  if  $(i,j)\in S$    
8: for  $\mathcal{G} = \mathcal{G}_1,\mathcal{G}_2,\dots ,\mathcal{G}_{s_{\ell +1}}$  do   
9: Run a ALL-GO-AGAINST-ALL TOURNAMENT  $(P,\mathcal{G},\epsilon ,\delta /(g_{\ell}))$  over the hypotheses in group  $\mathcal{G}$    
10: Add the winner to the set  $\mathcal{H}_{\ell +1}$    
11: Return the hypothesis in  $\mathcal{H}_{L + 1}$
Lemma B.1. Suppose we have a set of  $n$  hypotheses  $\mathcal{H} = \{H_1, H_2, \dots, H_n\}$  and an unknown hypothesis  $P$  which we wish to properly learn in  $\mathcal{H}$ . Suppose there are no decent hypothesis in  $\mathcal{H}$ , that is all hypotheses in  $\mathcal{H}$  are either OPT-close, or they will lose to any OPT-close hypothesis. For every  $2 \leq r \leq n$ , there exists an algorithm that uses  $O((\log (\log_r n) \cdot \log (1 / \delta) + \log n) / \epsilon^2)$  samples from  $P$ , and it outputs  $\hat{H}$  that is OPT-close to  $P$  with probability  $1 - \delta$ .
Round #	# hypotheses	group size	# groups or # winners	space
1	s1=n	g1=⌊√r⌋	s1/g1≤n/√r	O(n·r·log(log(1/δ)/ε))
2	s2=⌊n/g1⌋	g2=⌊(√r)1.5⌋	s2/g2≤n/r1.25	O(n·r·log(log(1/δ)/ε))
3	s3=⌊n/(g1g2)⌋	g3=⌊(√r)2.25⌋	s3/g3≤n/r2.375	O(n·r·log(log(1/δ)/ε))
ℓ=L	sℓ=⌊n/∑L-1gℓ⌋	gℓ=⌊(√r)1.5ℓ-1⌋	sℓ/gℓ≤n/r(1.5)ℓ-1	O(sℓ/gℓ·gℓ2·logmℓ)=O(n·r·log(⌊log(1/δ)/ε⌋))
Table 1: Branching factors in the tree structure
Proof. The algorithm runs in  $L$  rounds. At round  $\ell \in [L]$ , we start with  $s_\ell$  hypotheses. We partition them into groups of size  $g_\ell$ . For each round, we draw  $m_\ell \coloneqq \Theta(\log(g_\ell^3 / \delta) / \epsilon^2)$  fresh samples. We keep track of the number of samples in the Scheffé sets of every pair of hypotheses that are in the same group. Using the Scheffé counts, we run an all-go-against-all tournament in each group  $G_i$  with overall confidence probability  $1 - \delta / (6g_i)$  and find the winner. Then, we repeat this process among the winners in the next round until we end up with only one hypothesis. See Algorithm 3 for the description.
Parameters: We set the group sizes as follows:
$$
g _ {1} = \left\lceil \sqrt {r} \right\rceil , \quad g _ {\ell} := \left[ (\sqrt {r}) ^ {1. 5 ^ {\ell - 1}} \right] \approx g _ {\ell - 1} ^ {1. 5} \quad \forall \ell \in \{2, \dots , L \}. \tag {2}
$$
We start with  $s_1 \coloneqq n$ . The number of hypotheses in the  $\ell$ -th round is:
$$
s _ {\ell} := \left\lceil \frac {s _ {\ell - 1}}{g _ {\ell - 1}} \right\rceil = \left\lceil \frac {n}{\prod_ {i = 1} ^ {\ell - 1} g _ {i}} \right\rceil \leq \left\lceil \frac {n}{r ^ {1 . 5 ^ {\ell - 1} - 1}} \right\rceil \quad \forall \ell \in \{2, \dots , L \}. \tag {3}
$$
The equation above is due to Fact D.2 and Fact D.3.
Observe that we stop our process when only one hypothesis is left: the ultimate winner. Thus,  $L$  is the smallest integer such that  $s_{L + 1}$  is one. That is,  $\prod_{\ell = 1}^{L}g_{\ell}\geq n$ . It is not hard to show that  $L$  is  $O(\log (\log_rn))$  due to the following argument. For any  $L\geq \log_{1.5}(\log_rn + 1)$ , one can show  $\prod_{\ell = 1}^{L}g_{\ell}$  is at least  $n$  via Fact D.3:
$$
\prod_ {\ell = 1} ^ {L} g _ {\ell} \geq r ^ {1. 5 ^ {L} - 1} \geq n.
$$
Therefore,  $L$  cannot be larger than  $\lceil \log_{1.5}(\log_r n + 1)\rceil$
We describe how we set our parameters in Table 1.
Sample complexity: For round  $\ell$ , we draw  $m_{\ell}$  fresh samples. Thus, the total number of samples is:
$$
\begin{array}{l} \# \text {s a m p l e s} = \sum_ {\ell = 1} ^ {L} m _ {\ell} = \sum_ {\ell = 1} ^ {L} O \left(\frac {1}{\epsilon^ {2}} \cdot \log \frac {g _ {\ell} ^ {3}}{\delta}\right) \\ = O \left(\frac {L \cdot \log (1 / \delta)}{\epsilon^ {2}}\right) + O \left(\frac {1}{\epsilon^ {2}}\right) \cdot \sum_ {\ell = 1} ^ {L} \log g _ {\ell} \\ = O \left(\frac {L \cdot \log (1 / \delta)}{\epsilon^ {2}}\right) + O \left(\frac {1}{\epsilon^ {2}}\right) \cdot \log \left(\prod_ {\ell = 1} ^ {L} g _ {\ell}\right) \\ = O \left(\frac {L \cdot \log (1 / \delta) + \log n}{\epsilon^ {2}}\right) = O \left(\frac {\log (\log_ {r} n) \cdot \log (1 / \delta) + \log n}{\epsilon^ {2}}\right). \\ \end{array}
$$
Note that in the last line we use the fact that  $\prod_{\ell = 1}^{L}g_{\ell}$  is poly  $(n)$  which is not difficult to prove using the following argument: We define  $L$  to be the smallest integer such that  $\prod_{\ell = 1}^{L}g_{\ell}$  is at least  $n$ . Thus,  $\prod_{\ell = 1}^{L - 1}g_{\ell}$  is at most  $n$ . Hence, we have:
$$
\prod_ {\ell = 1} ^ {L} g _ {\ell} = g _ {L} \cdot \prod_ {\ell = 1} ^ {L - 1} g _ {\ell} \leq O \left(g _ {L - 1} ^ {1. 5}\right) \cdot \prod_ {\ell = 1} ^ {L - 1} g _ {\ell} \leq O \left(\prod_ {\ell = 1} ^ {L - 1} g _ {\ell}\right) ^ {2. 5} \leq O \left(n ^ {2. 5}\right).
$$
Correctness: Suppose  $\mathcal{G}$  is a group in round  $\ell$  that contains an OPT-close hypothesis. For the all-go-against-all tournament in round  $\ell$  we use  $m_{\ell} = O(\log (g_{\ell}^{3} / \delta) / \epsilon^{2})$  samples. That implies we estimate the probability of all the Scheffé sets in  $\mathcal{G}$  with accuracy  $\epsilon$  with probability at least  $1 - \delta /(6g_{\ell})$ . Given that we do not have any decent hypothesis, using Fact D.1, the all-go-against-all tournament has to output an OPT-close hypothesis with probability  $1 - \delta /(6g_{\ell})$ .
Now, let  $\mathcal{G}^{(1)}$  be the group in round one that  $H^{*}$  belongs to. Let  $\mathcal{G}^{(2)}$  be the group that the winner of  $\mathcal{G}^{(1)}$  belongs to in round two, and similarly we define:  $\mathcal{G}^{(3)},\ldots ,\mathcal{G}^{(L)}$ . Clearly, the all-go-against-all tournament over  $\mathcal{G}^{(i)}$  will return an OPT-close hypothesis with probability  $1 - \delta /(6g_{\ell})$  if  $\mathcal{G}^{(i)}$  contains an OPT-close hypothesis. Thus, using the union bound, the probability that the final group  $\mathcal{G}^{(L)}$  does not output an OPT-close hypothesis is bounded by the following:
$\mathbf{Pr}[\text{Output hypothesis is not OPT - close}] \leq \sum_{\ell=1}^{L} \frac{\delta}{6g_{\ell}} \leq \frac{\delta}{6} \cdot \sum_{\ell=1}^{L} (\sqrt{r})^{-1.5^{\ell-1}} \leq \frac{\delta}{6} \cdot \sum_{\ell=1}^{L} (\sqrt{r})^{-\ell/2}$
$$
\leq \frac {\delta}{6} \cdot \frac {r ^ {- 1 / 4}}{1 - r ^ {- 1 / 4}} \leq \delta .
$$
B.2 Memory usage of Algorithm 3
In this section, we provide an approach to implement Algorithm 3 using roughly  $O(n)$  bits of memory and prove our main theorem.
Theorem 5. Suppose we have a set of  $n$  hypotheses  $\mathcal{H} = \{H_1, H_2, \dots, H_n\}$  and an unknown hypothesis  $P$  which we wish to properly learn in  $\mathcal{H}$ . Suppose there are no decent hypothesis in  $\mathcal{H}$ , that is all hypotheses in  $\mathcal{H}$  are either OPT-close, or they will lose to any OPT-close hypothesis. For every  $2 \leq r \leq n$ , there exists an algorithm that uses  $O(n \cdot r \cdot \log (\log (1 / \delta) / \epsilon))$  bits of memory and  $O((\log (\log_r n) \cdot \log (1 / \delta) + \log n) / \epsilon^2)$  samples from  $P$ , and it outputs  $\hat{H}$  that is OPT-close to  $P$  with probability  $1 - \delta$ .