SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maryam Aliakbarpour
Houston, TX 77005
maryama@rice.edu
Mark Bun
Boston, MA 02215
mbun@bu.edu
Hypothesis selection is a fundamental problem in learning theory and statistics. Given a dataset and a finite set of candidate distributions, the goal is to select a distribution that matches the data as well as possible. More specifically, suppose that we have sample access to an unknown distribution  $P$  over a domain  $\mathcal{X}$  that we know is well-approximated by one of a class of  $n$  distributions (a.k.a. hypotheses),  $\mathcal{H} := \{H_1, H_2, \ldots, H_n\}$ . The goal is to design an algorithm that outputs a distribution  $\hat{H} \in \mathcal{H}$  whose total variation distance from  $P$  is nearly minimal. In this work, we study the hypothesis selection problem under memory constraints. We consider a model where samples from  $P$  are presented in a stream and we access each sample  $x$  via "PDF-comparison" queries that allow us to compare the probability densities of any pair of hypotheses at the domain point  $x$  (i.e., is  $H_i(x) < H_j(x)?$ ). This model allows us to study how much needs to be stored, at any point in time, about the portion of the stream seen so far. Our main result is an algorithm that achieves a nearly optimal tradeoff between memory usage and sample complexity. In particular, given  $b$  bits of memory (for  $b$  roughly between  $\log n$  and  $n$ ), our algorithm solves the hypothesis selection problem with  $s$  samples, where  $b \cdot s = O(n \log n)$ . This result is optimal up to an  $O(\log n)$  factor, for all  $b$ .
Learning the probability density function of observed data is a fundamental question in statistics with numerous applications in machine learning. Variants of this problem have been studied for nearly a century. Hypothesis selection is a classic version of this problem where the goal is to learn a distribution within a pre-specified class. Let  $\mathcal{H} \coloneqq \{H_1, H_2, \ldots, H_n\}$  be a class of  $n$  known distributions over the same domain  $\mathcal{X}$ . Suppose that we have sample access to an unknown distribution  $P$  over  $\mathcal{X}$  which is in, or a very close to, a member of  $\mathcal{H}$ . The goal is to design an algorithm that outputs a distribution  $\hat{H} \in \mathcal{H}$  whose total variation distance to  $P$  is close to that of the closest distribution in  $\mathcal{H}$ . A great deal of effort has been dedicated to solving this problem using a number of samples that does not depend on the domain size  $|\mathcal{X}|$ . Perhaps surprisingly, one can learn an unknown distribution  $P \in \mathcal{H}$  with  $O(\log n)$  samples, independent of the domain size. In contrast, learning an arbitrary distribution  $P$  over  $\mathcal{X}$  requires  $\Omega(|\mathcal{X}|)$  samples. Refined guarantees for this problem have been studied extensively, building an understanding of the accuracy, sample
complexity, and computational efficiency achievable, as well as the compatibility of hypothesis selection with robustness and privacy [Yat85, DL96, DL97, DL01, MS08, DDS12, DK14, DKS17, ADLS17, BKM19, CKM+19, BKSW19, GKK+20, ABH+20, BBK+21].
In this paper, we expand upon the emerging theory of learning with limited memory [SD15, Raz18, Sha14, DS18, DGKR19, SSV19, AMNW22, BBS22] by studying hypothesis selection from this new perspective. We strive to answer the following questions: Given a working memory of  $b$  bits, how many data points (samples) do we need to solve the hypothesis selection problem? Prior work assumes that we can store all the samples in the memory, which can be quite large when we aim to learn a distribution over extremely large objects, such as, DNA sequences, gene expression data, text data, and brain image data.
We consider a model where samples are processed one at a time in a stream. Similar to models in [GR22, CFG+23, AF23], we access each sample  $x$  via queries that allow us to compare the PDF of  $H_{i}$ 's at point  $x$ , and we measure the size of the memory retained between processing samples. Our main result is an algorithm that achieves a nearly optimal tradeoff between the memory size  $b$  and the number of samples  $s$ . Given  $b$  bits of memory (for  $b$  roughly between  $\log n$  and  $n$ ), our algorithm solves the hypothesis selection problem with  $s$  samples where  $b \cdot s = O(n\log n)$ . A result of Shamir [Sha14, Theorem 2] gives a class of  $n$  hypotheses  $\mathcal{H}$  such that any algorithm that learns unknown distributions in  $\mathcal{H}$  requires  $s \cdot b = \Omega(n)$ . Our tradeoff is thus optimal up to a  $O(\log n)$  factor.
1.1 Main result
Suppose we have a class of  $n$  known distributions  $\mathcal{H} \coloneqq \{H_1, H_2, \ldots, H_n\}$  and an unknown distribution  $P$ . We aim to design a proper learning algorithm that outputs a distribution in  $\mathcal{H}$  whose total variation distance to  $P$  is comparable to that of the closest distribution in  $\mathcal{H}$ . The algorithm has access to a stream of i.i.d. samples drawn from  $P$ . At every given point, the algorithm can look at one sample, namely  $x$ , in the stream and query a PDF-comparator oracle by sending  $i, j \in [n]$ , and receiving a bit indicating if  $H_i(x) < H_j(x)$ . This is equivalent to asking whether a sample  $x$  is in the Scheffé set of two distributions  $H_i$  and  $H_j$ , which is defined as the set of elements to which  $H_j$  assigns higher probability than  $H_i$ . The algorithm can also discard the current sample and move on to the next one. In addition to having access to the samples of  $P$ , we have more direct access to the known distributions in  $\mathcal{H}$ . The algorithm can query the probability masses of the Scheffé sets according to each  $H_i$  as is customary in the literature [DL01, MS08]. This assumption can be relaxed; only estimates of such probabilities are needed, which can be obtained if we have sample access to  $H_i$ 's. Further details are available in Remark 3. To summarize our access model, our algorithm can make one of the following types of queries: 1) Request a new sample. 2) Query the PDF-comparator on the current sample  $x$  and ask if  $H_i(x) < H_j(x)$  for every pair of indices  $i, j \in [n]$ . 3) Asks for the probability mass of Scheffé set between  $H_i$  and  $H_j$  according to  $H_i$  for every pair of indices  $i, j \in [n]$ .
The accuracy of our output distribution is measured with respect to how far the best distribution in  $\mathcal{H}$  is from  $P$ . We define  $\mathrm{OPT}(\mathcal{H}, P) \coloneqq \min_{H \in \mathcal{H}} \|P - H\|_{\mathrm{TV}}$ . When  $\mathcal{H}$  and  $P$  are clear from context, we denote this simply by  $\mathrm{OPT}$ . With this setup in mind, we define a proper learning algorithm (with promise) for hypothesis selection. The term "promise" refers to the extra parameter  $\Gamma$  that the algorithm receives as input: the learner may assume that  $\mathrm{OPT} \leq \Gamma$ . (See Remark 2 for the case where  $\Gamma$  is not provided.)
Definition 1.1. Suppose  $\mathcal{A}$  has sample access to an unknown distribution  $P$ . And, it can query the probabilities of the Scheffé sets according to each  $H_{i}$  and a PDF-comparator for every pair of hypotheses in  $\mathcal{H}$ . Assume  $\mathcal{A}$  is given an additional input parameter  $\Gamma > 0$ . We say algorithm  $\mathcal{A}$  is an  $(\alpha, \epsilon, \delta)$ -proper learner with a promise for  $\mathcal{H}$  if the following holds: for every distribution  $P$ , if  $\Gamma \geq \mathrm{OPT}(\mathcal{H}, P)$  then, with probability at least  $1 - \delta$  over its input sample (drawn i.i.d. from  $P$ ) and internal coin tosses,  $\mathcal{A}$  outputs a distribution  $\hat{H} \in \mathcal{H}$  such that:
$$
\| P - \hat {H} \| _ {T V} \leq \alpha \cdot \Gamma + \epsilon . \tag {1}
$$
One can generally reduce  $\epsilon$  and  $\delta$  by increasing the number of samples taken or the time spent, whereas the value of OPT, and hence  $\Gamma$ , is inherent to the problem instance. Therefore, it is more important and challenging to design algorithms that minimize the multiplicative parameter  $\alpha$ .
Main theorem: Our main result is a proper learner (with promise) that obtains a nearly optimal tradeoff between memory and data. Formally, we have the following theorem:
Theorem 1. There exists a constant  $c$  for which the following holds. Let  $\mathcal{H}$  be an arbitrary class consisting of  $n$  distributions, and let  $\epsilon > 0$ . For every natural number  $b$  where  $c \cdot (\log n + \log ((\log n) / \epsilon)) \leq b \leq n$ , there exists an  $(\alpha = 9, \epsilon, \delta = 0.1)$ -proper learner (with promise) for  $\mathcal{H}$  using  $b$  bits of memory and the following number of samples:
$$
s = \left\{ \begin{array}{l l} O \left(\frac {n \log n}{b} \cdot \frac {1}{\epsilon^ {2}} \log \left(\frac {1}{\epsilon}\right)\right) & \text {w h e n} b \leq \frac {n}{\log \log n}, \\ O \left(\frac {n \log n}{b} \cdot \frac {1}{\epsilon^ {2}} \log \left(\frac {\log n}{\epsilon}\right)\right) & \text {w h e n} \frac {n}{\log \log n} \leq b \leq n. \end{array} \right.
$$
Roughly speaking, this theorem says that given  $b$  bits of memory that suffice to perform basic operations such as keeping track of indices and counting samples (i.e.,  $b = \Omega (\log n + \log ((\log n) / \epsilon))$ ), then  $s\cdot b\approx O(n\log (n) / \epsilon^2)$  samples suffice to solve the hypothesis selection problem. A result of Shamir [Sha14, Theorem 2] gives a class of  $n$  hypotheses  $\mathcal{H}$  such that every algorithm that learns a random  $P$  in  $\mathcal{H}$  requires  $s\cdot b = \Omega (n)$ . Our tradeoff is thus optimal up to an  $O(\log n)$  factor. We speculate that our result is tight even for the class considered in [Sha14], but we leave the proof of a tighter lower bound to future work.
Remark 2. Both versions of the hypothesis selection problem (with and without a promise that  $\mathsf{OPT} \leq \Gamma$ ) have been extensively studied (e.g. [DL01, DK14]). For simplicity, we present our result when  $\Gamma$  is given a priori. Section C describes a reduction that yields a similar result when  $\Gamma$  is not provided; the accuracy guarantee is analogous to that of Equation 1, but for slightly larger  $\alpha$ .
1.2 Overview of our key ideas and other results
Background on comparing two hypotheses: Like many previous hypothesis selection algorithms, our algorithm relies on the ability to compare two hypotheses  $H_{1}$  and  $H_{2}$  based on the probabilities of their Scheffé set  $S$ . We estimate  $P(S)$ , and declare  $H_{i}$  the winner for which  $H_{i}(S)$  is closer to  $P(S)$ . This natural approach works especially well when  $P \in \{H_1, H_2\}$  or when  $P$  is much closer to one of the two hypotheses. Intuitively, a near-optimal hypothesis  $H$  should win many comparisons against other hypotheses. A typical hypothesis selection algorithm will run a tournament to identify such an  $H$ . The primary advantage of using Scheffé based comparisons is that they are inexpensive to compute. Only  $O(\log n / \epsilon^2)$  samples are sufficient to estimate the probability masses of all Scheffé sets within an error of  $\epsilon$ . This exceptional sample efficiency enables the hypothesis selection algorithms circumvent the lower bound of  $\Omega(|\mathcal{X}| / \epsilon^2)$  for distribution learning in cases where we have prior knowledge that  $P$  is close to a particular class of distributions.
That being said, the Scheffé based comparisons are not straightforward. One challenge is that these comparisons are not ideal: even if our estimates of the probabilities of the Scheffé sets were perfect, we might pick the hypothesis that is further from  $P$  in total variation distance (because  $S$  is just one event and might not ideally distinguish  $P$  from either  $H_{1}$  or  $H_{2}$ ). Furthermore, comparisons are not transitive (again, because we measure only Scheffé set probabilities). Essentially the only useful property they have is this: if  $H_{1}$  is  $\Gamma$ -close to  $P$ , it will win; if  $H_{1}$  is much further from  $P$  (by a constant factor) than both  $\Gamma$  and  $\| H_{2} - P\|_{\mathrm{TV}}$ , then it will lose. These issues make the outputs of comparisons hard to interpret and complicate their use for selection.
Our terminology: To elaborate on our approach, we start by defining a acceptable hypotheses, and our general terminology for referring to the quality of the hypothesis. We partition the hypotheses into three groups: 1) Excellent hypotheses, whose distance to  $P$  is  $\Gamma$ . 2) Decent hypotheses, whose distance to  $P$  is larger than  $\Gamma$ , but smaller than  $3\Gamma + \epsilon$ . These distances are set in a way that decent hypothesis may win against an excellent hypothesis. Decent hypotheses are challenging to deal with because they may fool us into believing that an excellent hypothesis is not a good output, yet later they may lose against a very far hypothesis. 3) Unacceptable hypotheses, whose distance to  $P$  is larger than  $3\Gamma + \epsilon$ , which will lose to every excellent hypotheses. We say a hypothesis is acceptable if it is either excellent or decent. Generally, the goal is to find an acceptable hypothesis.
1.2.1 Random ladder tournament
The main technical tool we introduce is a simple algorithm we call the random ladder tournament, which is inspired by ladder tournaments, a type of tournaments used in games and sports. Our algorithm finds an acceptable hypothesis using a linear number of comparisons. The algorithm proceeds in rounds as follows. Holding on to a single hypothesis in each round, we sample a uniformly random hypothesis from  $\mathcal{H}$  and compare it to the current hypothesis. The winner proceeds to the next round. Our novel analysis of this tournament shows that after  $O(|\mathcal{H}|)$  rounds, either the final winner produced at the end of the tournament or a winner selected randomly along its trajectory will be acceptable with high probability.
Proof overview: As previously mentioned, comparisons are often far from ideal; Even when the excellent hypothesis wins at a certain step, there is no guarantee that it would remain the winner till the end. To evade this issue, we utilize our knowledge of the upper bound of OPT, denoted by  $\Gamma$ , and modify the comparisons in a way that ensures the excellent hypothesis never loses once it wins. In our comparisons, we favor the current winner; If the distance of the current winner to  $P$  on the Scheffé set is at most  $\Gamma$ , we declare the current winner as the winner (even when the opponent hypothesis seems closer). Clearly, the excellent hypothesis will never be more than  $\Gamma$  away for  $P$  on any set. Hence, it will never lose again.
However, what happens when the excellent hypothesis never wins? At each step, we pick each hypothesis in  $\mathcal{H}$  uniformly at random, implying the excellent hypothesis is expected to be chosen in at least  $O(1)$  steps in our algorithms. The fact that the random hypothesis loses to the current winners of those steps indicates that those winners must be acceptable hypotheses; otherwise, the excellent hypothesis would have won them. Additionally, the fact that the excellent hypothesis appears as the random hypothesis in  $O(1)$  steps and loses to all of them, confirms that a constant fraction of the current winners are indeed acceptable. By combining these observations, we prove that the random ladder tournament will either find the excellent hypothesis at the end or encounter many acceptable hypotheses along the way.
Implementing the tournament with memory constraints: This algorithm is a highly effective technical tool for solving our problem. It can be implemented in a very memory-efficient manner. The algorithm processes the hypotheses in an "online" fashion, and it does not need to memorize the result of any past comparisons. At any given moment, it only remembers two hypotheses: the current hypothesis and the next randomly drawn hypothesis. And, if we draw fresh samples for every comparison, we can implement this algorithm by storing essentially  $O(1)$  numbers but sampling  $O(n\log (n) / \epsilon^2)$  data points. On the other extreme, we can implement this algorithm with very little data, i.e.,  $O(\log n / \epsilon^2)$  samples, but a large amount of memory by storing an appropriate "summary" of samples (e.g., the probabilities of all the Scheffé sets in  $\tilde{O}(n^2)$  bits).
We leverage this flexibility in memory usage of the random ladder tournament and provide an algorithm that only uses  $b$  bits of memory. We perform the comparisons in the random tournament in blocks of size  $t$ . For each block of comparisons, we draw new samples and store a summary of them. We pick the parameter  $t$  in way that the summary fits in  $b$  bits, and it is sufficient for us to perform the next the  $t$  comparisons. For example, if  $t \approx \sqrt{b}$ , we can store the number of samples in all the Scheffé sets of  $t$  hypotheses in  $b$  bits. This summary is enough for us to perform any comparisons between those hypotheses. Combined with a novel summary—sorted lists, described in Section 2.2.2—we get a tradeoff of the form  $s \cdot b \approx O(n\log^3 (n) / \epsilon^4)$ . In Section 3.2, we presented this tradeoff alongside another tradeoff (with better dependence to  $\epsilon$ , but worse dependence to  $\log n$ ). Both of these tradeoffs are worse than our main result, in a sense that  $s \cdot b$  is larger by a factor of  $O(\mathrm{poly}(\log (n),1 / \epsilon))$ . However, the these results are yielding more accurate proper learners; For the random ladder tournament,  $\alpha = 3$ , while in our main result  $\alpha = 9$ .
Simpler linear-time selection: The random ladder tournament leads to new results for the hypothesis selection problem outside of the scope of our original motivation for this paper. If we ignore memory constraints and store every sample, the random ladder tournament algorithm is simultaneously near-optimal in terms of number of samples, time, and accuracy — to our knowledge, it is the first such algorithm. Specifically, it makes a linear number of comparisons and uses  $O(\log n / \epsilon^2)$  samples from  $P$ , which are both optimal for worst-case choices of  $\mathcal{H}$ . Moreover, it finds a hypothesis with optimal accuracy under a promise. More precisely, it finds a hypothesis in  $\mathcal{H}$  that is roughly
$3\Gamma$ -close to  $P$  given a parameter  $\Gamma$  that is guaranteed to be at least OPT. This is essentially optimal, as no proper learning algorithm can achieve accuracy better than 3 OPT [BKM19] even when  $\Gamma = \mathrm{OPT}$  is given to the algorithm. Furthermore, with a simple adjustment to our algorithm, one can solve the hypothesis selection problem without any knowledge of  $\Gamma$  in nearly linear time and with roughly the same number of samples and obtain a 5 OPT-close hypothesis in  $\mathcal{H}$ . See Section C.
1.2.2 Improved tradeoff by hypothesis filtration
The flexibility of the random ladder tournament already results in memory-data tradeoffs, but these are suboptimal by a factor of poly  $(\log n, 1 / \epsilon)$ . To improve the result, we design an algorithm called filter, that picks an acceptable hypothesis with a modest chance. While this chance is not high, we can use the output of the filtration and run the random ladder tournament on these filtered hypotheses. Since the quality of the input hypotheses in this second round is better, the random ladder tournament can be implemented effectively on a smaller set  $\mathcal{H}$  and results in a better tradeoff. Hence, we obtain our main result. See Section B.4 for the proof of our main results. Below, we give a short description of the filter algorithm. We defer the full description and the proofs to Section B.3.
hypothesis filtration: The best approach to finding an acceptable hypothesis drastically depends on the quality of the hypotheses in  $\mathcal{H}$ . For example, if there is no decent hypothesis in  $\mathcal{H}$ , any single elimination tournament will easily find an excellent hypothesis due to the fact that any comparison involving at least one excellent hypothesis will declare the excellent hypothesis the winner. On the other hand, if there are a lot of decent hypotheses, then there is a decent chance that a randomly selected hypothesis is decent, hence an acceptable one. The filtration algorithm works by combining these two observations. The algorithm randomly performs one of the following, each with probability  $1/2$ : 1) It draws a set of random hypotheses. Assuming there is no decent hypothesis in the set, it runs a group elimination tournament (see below), and outputs the result. 2) It outputs a random hypothesis in  $\mathcal{H}$ . One can show that if the decent hypotheses are scarce in  $\mathcal{H}$ , we have a decent chance of getting an excellent hypothesis and no decent hypothesis in the random set; hence, we find an excellent hypothesis via our group elimination algorithm. On the other hand, if the decent hypotheses are abundant, then we have a decent chance of picking one in the second step. By selecting each strategy with a probability  $1/2$ , we preserve half of the success probability in whichever case holds. While this probability is not a lot, it is larger than  $1/n$  (the probability of picking the excellent hypothesis from  $\mathcal{H}$ ). We take advantage of this fact, and show one can run the random ladder tournament in fewer rounds and obtain the desired result. See Section B.3 for more details.
Group elimination tournament: Our second technical tool is an algorithm that finds an excellent hypothesis if no decent hypothesis exists in  $\mathcal{H}$ . This algorithm is a combination of a single-elimination tournament and an all-go-against-all tournament. In this algorithm, at every step, we partition the hypotheses into multiple groups, run a previously known all-go-against-all tournament within each group (say minimum distance estimate [DL01, MS08]), and then send the winners of each group to the next round. If a group contains an excellent hypothesis, then in an all-go-against-all tournament, we will pick an excellent hypothesis as the winner. Hence, an excellent hypothesis will "bubble up" in each round and be declared the final winner. With a careful choice of group sizes and the number of rounds, this algorithm can be implemented using  $O(n)$  bits of memory and  $O(\log n)$  samples under the assumption that the hypotheses are indexed from 1 to  $n$ . A technical challenge that arises, however, is that we need to run this algorithm on a subset of roughly  $O(b)$  random hypotheses (so that we can run it within a memory bound of  $b$ ). However, storing  $b$  random indices already requires  $O(b\log n)$  space. We show that we can instead draw indices pseudorandomly using a pairwise independent generator that can be implemented in small space. See Section B.1 for more details.
Prior to our work, numerous studies have investigated the problem of hypothesis selection in various settings. In their seminal work, Yatracos [Yat85], and Devroye and Lugosi [DL96, DL97, DL01] present algorithms to find a close distribution in  $\mathcal{H}$  based on only estimation of the probabilities of the Scheffé sets of pairs of hypothesis in  $\mathcal{H}$ . These approaches suggest that we only need accurate estimation of the probability that  $P$  assigns to  $O(|\mathcal{H}|^2) = O(n^2)$  sets, resulting in sample complexity proportional to  $O(\log n)$  instead of  $\Omega(|\mathcal{X}|)$ . For a comprehensive overview, see Chapter 6 in [DL01].
Mahalanabis and Stefankovic in [MS08] provide an algorithm with linearly many probes to  $P$  and  $\alpha = 3$  but they require an exponential time pre-processing of the class  $H$ .<sup>1</sup> Daskalakis and Kamath
It is worth noting that we do not have any pre-processing step in this paper.