Title: Hypothesis Selection with Memory Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: Hypothesis selection

memory constrained algorithms

density estimation

limited space
Abstract: Hypothesis selection is a fundamental problem in learning theory and statistics. 
Given a dataset and a finite set of candidate distributions, the goal is to select a distribution that matches the data as well as possible. 
More specifically, suppose we have sample access to an unknown distribution $P$ over a domain $\mathcal{X}$ that we know is well-approximated by one of a 
a class of $n$  distributions (a.k.a. hypotheses), $\mathcal{H} \coloneqq \{H_1, H_2, \ldots, H_n\}$. The goal is to design an algorithm that outputs a distribution $\hat{H} \in \mathcal{H}$ whose total variation distance from $P$ is nearly minimal.

In this work, we study the hypothesis selection problem under memory constraints. We consider a model where samples from $P$ are presented in a stream and we access each sample $x$ via ``PDF-comparison'' queries that allow us to compare the probability densities of any pair of hypotheses
at the domain point $x$ (i.e., is $H_i(x) < H_j(x)$?). This model allows us to study how much memory is needed at any point in time to store information about the portion of the stream seen so far.

Our main result is an algorithm that achieves a nearly optimal tradeoff between memory usage and the number of samples required. In particular, given $b$ bits of memory (for $b$ roughly between $\log n$ and $n$), our algorithm solves the hypothesis selection problem with $s$ samples, where $b \cdot s = O(n \log n)$. This result is optimal up to an $O(\log n)$ factor, for all $b$.

Hypothesis Selection with Memory Constraints

Abstract

Hypothesis selection is a fundamental problem in learning theory and statistics. Given a dataset and a finite set of candidate distributions, the goal is to select a distribution that matches the data as well as possible. More specifically, suppose that we have sample access to an unknown distribution P over a domain X that we know is well-approximated by one of a class of n distributions (a.k.a. hypotheses), H := { H 1 , H 2 , . . . , H n } . The goal is to design an algorithm that outputs a distribution ˆ H ∈ H whose total variation distance from P is nearly minimal. In this work, we study the hypothesis selection problem under memory constraints. We consider a model where samples from P are presented in a stream and we access each sample x via 'PDF-comparison' queries that allow us to compare the probability densities of any pair of hypotheses at the domain point x (i.e., is H i ( x ) < H j ( x ) ?). This model allows us to study how much needs to be stored, at any point in time, about the portion of the stream seen so far. Our main result is an algorithm that achieves a nearly optimal tradeoff between memory usage and sample complexity. In particular, given b bits of memory (for b roughly between log n and n ), our algorithm solves the hypothesis selection problem with s samples, where b · s = O ( n log n ) . This result is optimal up to an O (log n ) factor, for all b .

1 Introduction

Learning the probability density function of observed data is a fundamental question in statistics with numerous applications in machine learning.Variants of this problem have been studied for nearly a century. Hypothesis selection is a classic version of this problem where the goal is to learn a distribution within a pre-specified class. Let H := { H 1 , H 2 , . . . , H n } be a class of n known distributions over the same domain X . Suppose that we have sample access to an unknown distribution P over X which is in, or a very close to, a member of H . The goal is to design an algorithm that outputs a distribution ˆ H ∈ H whose total variation distance to P is close to that of the closest distribution in H . A great deal of effort has been dedicated to solving this problem using a number of samples that does not depend on the domain size |X| . Perhaps surprisingly, one can learn an unknown distribution P ∈ H with O (log n ) samples, independent of the domain size. In contrast, learning an arbitrary distribution P over X requires Ω( |X| ) samples. Refined guarantees for this problem have been studied extensively, building an understanding of the accuracy, sample

complexity, and computational efficiency achievable, as well as the compatibility of hypothesis selection with robustness and privacy [Yat85, DL96, DL97, DL01, MS08, DDS12, DK14, DKS17, ADLS17, BKM19, CKM + 19, BKSW19, GKK + 20, ABH + 20, BBK + 21].

In this paper, we expand upon the emerging theory of learning with limited memory [SD15, Raz18, Sha14, DS18, DGKR19, SSV19, AMNW22, BBS22] by studying hypothesis selection from this new perspective. We strive to answer the following questions: Given a working memory of b bits, how many data points (samples) do we need to solve the hypothesis selection problem? Prior work assumes that we can store all the samples in the memory, which can be quite large when we aim to learn a distribution over extremely large objects, such as, DNA sequences, gene expression data, text data, and brain image data.

We consider a model where samples are processed one at a time in a stream. Similar to models in [GR22, CFG + 23, AF23], we access each sample x via queries that allow us to compare the PDF of H i 's at point x , and we measure the size of the memory retained between processing samples. Our main result is an algorithm that achieves a nearly optimal tradeoff between the memory size b and the number of samples s . Given b bits of memory (for b roughly between log n and n ), our algorithm solves the hypothesis selection problem with s samples where b · s = O ( n log n ) . A result of Shamir [Sha14, Theorem 2] gives a class of n hypotheses H such that any algorithm that learns unknown distributions in H requires s · b = Ω( n ) . Our tradeoff is thus optimal up to a O (log n ) factor.

1.1 Main result

Suppose we have a class of n known distributions H := { H 1 , H 2 , . . . , H n } and an unknown distribution P . We aim to design a proper learning algorithm that outputs a distribution in H whose total variation distance to P is comparable to that of the closest distribution in H . The algorithm has access to a stream of i.i.d. samples drawn from P . At every given point, the algorithm can look at one sample, namely x , in the stream and query a PDF-comparator oracle by sending i, j ∈ [ n ] , and receiving a bit indicating if H i ( x ) < H j ( x ) . This is equivalent to asking whether a sample x is in the Scheffé set of two distributions H i and H j , which is defined as the set of elements to which H j assigns higher probability than H i . The algorithm can also discard the current sample and move on to the next one. In addition to having access to the samples of P , we have more direct access to the known distributions in H . The algorithm can query the probability masses of the Scheffé sets according to each H i as is customary in the literature [DL01, MS08]. This assumption can be relaxed; only estimates of such probabilities are needed, which can be obtained if we have sample access to H i 's. Further details are available in Remark 3. To summarize our access model, our algorithm can make one of the following types of queries: 1) Request a new sample. 2) Query the PDF-comparator on the current sample x and ask if H i ( x ) < H j ( x ) for every pair of indices i, j ∈ [ n ] . 3) Asks for the probability mass of Scheffé set between H i and H j according to H i for every pair of indices i, j ∈ [ n ] .

The accuracy of our output distribution is measured with respect to how far the best distribution in H is from P . We define OPT ( H , P ) := min H ∈H ∥ P -H ∥ TV . When H and P are clear from context, we denote this simply by OPT . With this setup in mind, we define a proper learning algorithm (with promise) for hypothesis selection. The term 'promise' refers to the extra parameter Γ that the algorithm receives as input: the learner may assume that OPT ≤ Γ . (See Remark 2 for the case where Γ is not provided.)

Definition 1.1. Suppose A has sample access to an unknown distribution P . And, it can query the probabilities of the Scheffé sets according to each H i and a PDF-comparator for every pair of hypotheses in H . Assume A is given an additional input parameter Γ > 0 . We say algorithm A is an ( α, ϵ, δ ) -proper learner with a promise for H if the following holds: for every distribution P , if Γ ≥ OPT ( H , P ) then, with probability at least 1 -δ over its input sample (drawn i.i.d. from P ) and internal coin tosses, A outputs a distribution ˆ H ∈ H such that:

One can generally reduce ϵ and δ by increasing the number of samples taken or the time spent, whereas the value of OPT , and hence Γ , is inherent to the problem instance. Therefore, it is more important and challenging to design algorithms that minimize the multiplicative parameter α .

Main theorem: Our main result is a proper learner (with promise) that obtains a nearly optimal tradeoff between memory and data. Formally, we have the following theorem:

Theorem 1. There exists a constant c for which the following holds. Let H be an arbitrary class consisting of n distributions, and let ϵ > 0 . For every natural number b where c · (log n +log((log n ) /ϵ )) ≤ b ≤ n , there exists an ( α = 9 , ϵ, δ = 0 . 1) -proper learner (with promise) for H using b bits of memory and the following number of samples:

Roughly speaking, this theorem says that given b bits of memory that suffice to perform basic operations such as keeping track of indices and counting samples (i.e., b = Ω(log n +log((log n ) /ϵ )) ), then s · b ≈ O ( n log( n ) /ϵ 2 ) samples suffice to solve the hypothesis selection problem. A result of Shamir [Sha14, Theorem 2] gives a class of n hypotheses H such that every algorithm that learns a random P in H requires s · b = Ω( n ) . Our tradeoff is thus optimal up to an O (log n ) factor. We speculate that our result is tight even for the class considered in [Sha14], but we leave the proof of a tighter lower bound to future work.

Remark 2. Both versions of the hypothesis selection problem (with and without a promise that OPT ≤ Γ ) have been extensively studied (e.g. [DL01, DK14]). For simplicity, we present our result when Γ is given a priori. Section C describes a reduction that yields a similar result when Γ is not provided; the accuracy guarantee is analogous to that of Equation 1, but for slightly larger α .

1.2 Overview of our key ideas and other results

Background on comparing two hypotheses: Like many previous hypothesis selection algorithms, our algorithm relies on the ability to compare two hypotheses H 1 and H 2 based on the probabilities of their Scheffé set S . We estimate P ( S ) , and declare H i the winner for which H i ( S ) is closer to P ( S ) . This natural approach works especially well when P ∈ { H 1 , H 2 } or when P is much closer to one of the two hypotheses. Intuitively, a near-optimal hypothesis H should win many comparisons against other hypotheses. A typical hypothesis selection algorithm will run a tournament to identify such an H . The primary advantage of using Scheffé based comparisons is that they are inexpensive to compute. Only O (log n/ϵ 2 ) samples are sufficient to estimate the probability masses of all Scheffé sets within an error of ϵ . This exceptional sample efficiency enables the hypothesis selection algorithms circumvent the lower bound of Ω( |X| /ϵ 2 ) for distribution learning in cases where we have prior knowledge that P is close to a particular class of distributions.

That being said, the Scheffé based comparisons are not straightforward. One challenge is that these comparisons are not ideal: even if our estimates of the probabilities of the Scheffé sets were perfect, we might pick the hypothesis that is further from P in total variation distance (because S is just one event and might not ideally distinguish P from either H 1 or H 2 ). Furthermore, comparisons are not transitive (again, because we measure only Scheffé set probabilities). Essentially the only useful property they have is this: if H 1 is Γ -close to P , it will win; if H 1 is much further from P (by a constant factor) than both Γ and ∥ H 2 -P ∥ TV, then it will lose. These issues make the outputs of comparisons hard to interpret and complicate their use for selection.

Our terminology: To elaborate on our approach, we start by defining a acceptable hypotheses, and our general terminology for referring to the quality of the hypothesis. We partition the hypotheses into three groups: 1) Excellent hypotheses, whose distance to P is Γ . 2) Decent hypotheses, whose distance to P is larger than Γ , but smaller than 3 Γ + ϵ . These distances are set in a way that decent hypothesis may win against an excellent hypothesis. Decent hypotheses are challenging to deal with because they may fool us into believing that an excellent hypothesis is not a good output, yet later they may lose against a very far hypothesis. 3) Unacceptable hypotheses, whose distance to P is larger than 3Γ + ϵ , which will lose to every excellent hypotheses. We say a hypothesis is acceptable if it is either excellent or decent. Generally, the goal is to find an acceptable hypothesis.

1.2.1 Random ladder tournament

The main technical tool we introduce is a simple algorithm we call the random ladder tournament , which is inspired by ladder tournaments , a type of tournaments used in games and sports. Our algorithm finds an acceptable hypothesis using a linear number of comparisons. The algorithm proceeds in rounds as follows. Holding on to a single hypothesis in each round, we sample a uniformly random hypothesis from H and compare it to the current hypothesis. The winner proceeds to the next round. Our novel analysis of this tournament shows that after O ( |H| ) rounds, either the final winner produced at the end of the tournament or a winner selected randomly along its trajectory will be acceptable with high probability.

Proof overview: As previously mentioned, comparisons are often far from ideal; Even when the excellent hypothesis wins at a certain step, there is no guarantee that it would remain the winner till the end. To evade this issue, we utilize our knowledge of the upper bound of OPT , denoted by Γ , and modify the comparisons in a way that ensures the excellent hypothesis never loses once it wins. In our comparisons, we favor the current winner; If the distance of the current winner to P on the Scheffé set is at most Γ , we declare the current winner as the winner (even when the opponent hypothesis seems closer). Clearly, the excellent hypothesis will never be more than Γ away for P on any set. Hence, it will never lose again.

However, what happens when the excellent hypothesis never wins? At each step, we pick each hypothesis in H uniformly at random, implying the excellent hypothesis is expected to be chosen in at least O (1) steps in our algorithms. The fact that the random hypothesis loses to the current winners of those steps indicates that those winners must be acceptable hypotheses; otherwise, the excellent hypothesis would have won them. Additionally, the fact that the excellent hypothesis appears as the random hypothesis in O (1) steps and loses to all of them, confirms that a constant fraction of the current winners are indeed acceptable. By combining these observations, we prove that the random ladder tournament will either find the excellent hypothesis at the end or encounter many acceptable hypotheses along the way.

Implementing the tournament with memory constraints: This algorithm is a highly effective technical tool for solving our problem. It can be implemented in a very memory-efficient manner. The algorithm processes the hypotheses in an 'online' fashion, and it does not need to memorize the result of any past comparisons. At any given moment, it only remembers two hypotheses: the current hypothesis and the next randomly drawn hypothesis. And, if we draw fresh samples for every comparison, we can implement this algorithm by storing essentially O (1) numbers but sampling O ( n log( n ) /ϵ 2 ) data points. On the other extreme, we can implement this algorithm with very little data, i.e., O (log n/ϵ 2 ) samples, but a large amount of memory by storing an appropriate 'summary' of samples (e.g., the probabilities of all the Scheffé sets in ˜ O ( n 2 ) bits).

We leverage this flexibility in memory usage of the random ladder tournament and provide an algorithm that only uses b bits of memory. We perform the comparisons in the random tournament in blocks of size t . For each block of comparisons, we draw new samples and store a summary of them. We pick the parameter t in way that the summary fits in b bits, and it is sufficient for us to perform the next the t comparisons. For example, if t ≈ √ b , we can store the number of samples in all the Scheffé sets of t hypotheses in b bits. This summary is enough for us to perform any comparisons between those hypotheses. Combined with a novel summary-sorted lists, described in Section 2.2.2-we get a tradeoff of the form s · b ≈ O ( n log 3 ( n ) /ϵ 4 ) . In Section 3.2, we presented this tradeoff alongside another tradeoff (with better dependence to ϵ , but worse dependence to log n ). Both of these tradeoff are worse than our main result, in a sense that s · b is larger by a factor of O (poly(log( n ) , 1 /ϵ )) . However, the these results are yielding more accurate proper learners; For the random ladder tournament, α = 3 , while in our main result α = 9 .

Simpler linear-time selection: The random ladder tournament leads to new results for the hypothesis selection problem outside of the scope of our original motivation for this paper. If we ignore memory constraints and store every sample, the random ladder tournament algorithm is simultaneously near-optimal in terms of number of samples, time, and accuracy - to our knowledge, it is the first such algorithm. Specifically, it makes a linear number of comparisons and uses O (log n/ϵ 2 ) samples from P , which are both optimal for worst-case choices of H . Moreover, it finds a hypothesis with optimal accuracy under a promise. More precisely, it finds a hypothesis in H that is roughly

3 Γ -close to P given a parameter Γ that is guaranteed to be at least OPT . This is essentially optimal, as no proper learning algorithm can achieve accuracy better than 3 OPT [BKM19] even when Γ = OPT is given to the algorithm. Furthermore, with a simple adjustment to our algorithm, one can solve the hypothesis selection problem without any knowledge of Γ in nearly linear time and with roughly the same number of samples and obtain a 5 OPT -close hypothesis in H . See Section C.

1.2.2 Improved tradeoff by hypothesis filtration

The flexibility of the random ladder tournament already results in memory-data tradeoffs, but these are suboptimal by a factor of poly(log n, 1 /ϵ ) . To improve the result, we design an algorithm called filter , that picks an acceptable hypothesis with a modest chance. While this chance is not high, we can use the output of the filtration and run the random ladder tournament on these filtered hypotheses. Since the quality of the input hypotheses in this second round is better, the random ladder tournament can be implemented effectively on a smaller set H and results in a better tradeoff. Hence, we obtain our main result. See Section B.4 for the proof of our main results. Below, we give a short description of the filter algorithm. We defer the full description and the proofs to Section B.3.

hypothesis filtration: The best approach to finding an acceptable hypothesis drastically depends on the quality of the hypotheses in H . For example, if there is no decent hypothesis in H , any single elimination tournament will easily find an excellent hypothesis due to the fact that any comparison involving at least one excellent hypothesis will declare the excellent hypothesis the winner. On the other hand, if there are a lot of decent hypotheses, then there is a decent chance that a randomly selected hypothesis is decent, hence an acceptable one. The filtration algorithm works by combining these two observations. The algorithm randomly performs one of the following, each with probability 1 / 2 : 1) It draws a set of random hypotheses. Assuming there is no decent hypothesis in the set, it runs a group elimination tournament (see below), and outputs the result. 2) It outputs a random hypothesis in H . One can show that if the decent hypotheses are scarce in H , we have a decent chance of getting an excellent hypothesis and no decent hypothesis in the random set; hence, we find an excellent hypothesis via our group elimination algorithm. On the other hand, if the decent hypotheses are abundant, then we have a decent chance of picking one in the second step. By selecting each strategy with a probability 1 / 2 , we preserve half of the success probability in whichever case holds. While this probability is not a lot, it is larger than 1 /n (the probability of picking the excellent hypothesis from H ). We take advantage of this fact, and show one can run the random ladder tournament in fewer rounds and obtain the desired result. See Section B.3 for more details.

Group elimination tournament: Our second technical tool is an algorithm that finds an excellent hypothesis if no decent hypothesis exists in H . This algorithm is a combination of a single-elimination tournament and an all-go-against-all tournament. In this algorithm, at every step, we partition the hypotheses into multiple groups, run a previously known all-go-against-all tournament within each group (say minimum distance estimate [DL01, MS08]), and then send the winners of each group to the next round. If a group contains an excellent hypothesis, then in an all-go-against-all tournament, we will pick an excellent hypothesis as the winner. Hence, an excellent hypothesis will 'bubble up' in each round and be declared the final winner. With a careful choice of group sizes and the number of rounds, this algorithm can be implemented using O ( n ) bits of memory and O (log n ) samples under the assumption that the hypotheses are indexed from 1 to n . A technical challenge that arises, however, is that we need to run this algorithm on a subset of roughly O ( b ) random hypotheses (so that we can run it within a memory bound of b ). However, storing b random indices already requires O ( b log n ) space. We show that we can instead draw indices pseudorandomly using a pairwise independent generator that can be implemented in small space. See Section B.1 for more details.

1.3 Related work

Prior to our work, numerous studies have investigated the problem of hypothesis selection in various settings. In their seminal work, Yatracos [Yat85], and Devroye and Lugosi [DL96, DL97, DL01] present algorithms to find a close distribution in H based on only estimation of the probabilities of the Scheffé sets of pairs of hypothesis in H . These approaches suggest that we only need accurate estimation of the probability that P assigns to O ( |H| 2 ) = O ( n 2 ) sets, resulting in sample complexity proportional to O (log n ) instead of Ω( |X| ) . For a comprehensive overview, see Chapter 6 in [DL01].

Mahalanabis and Stefankovic in [MS08] provide an algorithm with linearly many probes to P and α = 3 but they require an exponential time pre-processing of the class H . 1 Daskalakis and Kamath

in [DK14] give a nearly-linear time algorithm that given an upper bound on OPT ≤ Γ , returns ˆ H such that ∥ ˆ H -P ∥ TV ≤ α · Γ+ ϵ for some large constant α = 512 . Acharya et al. in [AJOS14, AFJ + 18] provide a O ( n log n ) time algorithm that finds a hypothesis with accuracy guarantee of α = 9 . For proper learners, Bousquet et al. in [BKM19] showed that the best α in Equation 1 that one can hope for is α = 3 as long as the sample complexity does not grow with the domain size |X| . Aamand et al. in [AAC + 23] also study statistical computational tradeoffs (time-data) for the hypotheses over discrete domains . Unlike our work, their setting allows pre-processing of the class H in polynomial time.

There is a long history of research focusing on a special case of this problem where H is a specific structured class of distributions such as mixtures of Gaussians [KMV12, DK14, SOAJ14, KSS18, ABM18, ABH + 20], histograms [Pea95, CDSS14, CDKL22], and polynomials [ADLS17]. The abstract hypothesis selection we study here is commonly used as a subroutine in solving these problems (usually in conjunction with some sort of a cover method). For a survey of results, see [Dia16].

Lastly, there is another field of study that tackles a question similar to ours: the problem of sorting items with noisy comparisons. One can view hypothesis selection as the task of finding the minimum item. With the exception of [AFJ + 18] that we have discussed above, these probabilistic noise models do not capture the geometric structure presenting in our problem. Therefore, we did not find any of these result yielding an immediate solution to our problem.

2 Preliminaries

2.1 Notation

Let [ n ] denote the set { 1 , 2 , . . . , n } . Suppose that we have a probability distribution P over a domain X . For element x in X , P ( x ) denotes the probability of x . We assume X is the domain of all the probability distributions in this article. For any subset of the domain S , P ( S ) denotes the sum of the probabilities of all the elements in S . We denote the total variation distance between two distributions P 1 and P 2 by ∥ P 1 -P 2 ∥ TV, and it is defined as max S ⊆X | P 1 ( S ) -P 2 ( S ) | where S is any measurable subset of the domain. We say P is ϵ -close to Q iff ∥ P -Q ∥ TV is at most ϵ . Also, we say P is ϵ -far from Q iff ∥ P -Q ∥ TV is greater than ϵ . We denote the Scheffé set of two distributions as follows: S ( H 1 , H 2 ) := { x ∈ X | H 1 ( x ) < H 2 ( x ) } .

2.2 Basic tools

In this section, we focus on our primary tools we have used throughout this article. First, we start with a comparisons algorithm that allows us to compare two hypotheses. Next, we use the fact that one can estimate the probabilities of the O ( n 2 ) many Scheffé sets up to error c · ϵ (for some small constant c < 1 ) with probability at least 1 -δ using O (log( n/δ ) /ϵ 2 ) samples from P .

2.2.1 Comparing two hypotheses

Our algorithms works based on a basic operation that allows us to compare two hypotheses H 1 and H 2 based on the probabilities of their Scheffé sets. We pick the hypothesis that appears to be closer to P and declare it the winner (and the other hypothesis the loser ). The challenging part is that these comparisons are not ideal. As in, even when our estimations of the probabilities of the Scheffé sets are accurate enough, we may pick a hypothesis that is further. However, one can guarantee that if the distance of the further hypothesis among H 1 and H 2 is much worse than the distance of the closer one, the comparison procedure would never select it as the winner. The algorithm is given three parameters: 1) Γ : an upper bound for OPT ; 2) ϵ : indicating the estimation error for the Scheffé set probabilities is Θ( ϵ ) ; and 3) the confidence parameter δ . Upon invoking this algorithm, it determines the result of the comparison with the guarantees formalized in the following lemma:

Lemma 2.1. Upon receiving three parameters: Γ , ϵ , and δ , Algorithm 1 uses O (log(1 /δ ) /ϵ 2 ) samples and satisfies the following guarantees with probability at least 1 -δ :

If H 1 is Γ -close to P , then the algorithm returns H 1 .

If H 1 is (max(Γ , ∥ H 2 -P ∥ TV ) + 2 ∥ H 2 -P ∥ TV + ϵ ) -far from P , then the algorithm returns H 2 .

The proof of this lemma is provided in Section D.

Our terminology based on comparisons: We say a distribution H is excellent iff ∥ H -P ∥ TV is Γ . Weusually use H ∗ to denote an excellent hypothesis. We say a distribution H is decent iff ∥ H -P ∥ TV is greater than Γ , but smaller than 3 Γ + ϵ . It implies that a decent hypothesis may win an excellent H ∗ ∈ H . We say a distribution H is acceptable iff it is excellent or decent. And, we say a distribution H is unacceptable iff it is not acceptable. Note that it is impossible for an unacceptable hypothesis to win an excellent hypothesis with probability greater than δ . Throughout this article, when we say valid comparisons , we refer to an event in which the Scheffé sets are estimated accurately, so the conditions in Lemma 2.1 hold. Since we have only O ( n 2 ) many Scheffé sets, using O (log( n/δ ) /ϵ 2 ) samples is enough to assume all the comparisons are valid with a probability of at least 1 -δ .

2.2.2 Space efficient sample summaries for comparisons within a batch of hypotheses

In order to obtain sample efficiency, we need sufficient information about samples that enables us to reuse the same set of samples for multiple comparisons. Below, we describe two approaches to summarize the sample set.

Scheffé counts. Observe that to compare two hypotheses H 1 and H 2 , we only use the count of the number of samples in their Scheffé set S ( H 1 , H 2 ) . Hence, instead of storing samples directly, for every pair of hypotheses in H , we store the number of samples in their Scheffé set. We refer to this number as the Scheffé counts . If we have m samples, we can store all the O ( n 2 ) Scheffé counts using O ( n 2 log m ) bits.

Storing each sample via a sorted list. In this approach, we store a succinct summary of each sample that allows us to infer its membership in each Scheffé set. This information suffices to perform the comparisons we need later. To store a sample x , we save an ordered list of hypotheses that is sorted according to the probability of x (the PDF at point x ). That is, we store a list of indices i 1 , i 2 , . . . , i n such that H i 1 ( x ) ≤ H i 2 ( x ) ≤ · · · ≤ H i n ( x ) (use hypothesis indices to break the tie). The summary of a single sample can be computed in O ( n log n ) time and takes O ( n log n ) bits of space to store. Now, to compare the PDF of two hypotheses H i and H j , we simply can find i and j in the list and check whether i precedes j in the list. 2 Alternatively if we want to compare PDF's faster, we can store the indices in another list call d x that maps indices to list positions. We set d x [ i ℓ ] = ℓ for every ℓ ∈ [ n ] . In this case, to compare the PDF's, we can simple check if d x [ i ] < d x [ j ] .

2.2.3 All-go-against-all tournament

There are standard techniques in the literature to solve hypothesis selection problem when the estimates of all the Scheffé sets are available. See Chapter 6 in [DL01] and [MS08]. These approaches

Algorithm 2 Random ladder tournament

suggest that we only need accurate estimation of the probability that P assigns to O ( |H| 2 ) = O ( n 2 ) sets, therefore, the sample complexity can be proportionate to O (log n ) instead of O ( |X| ) . For a formal statement, see Fact D.1.

3 Random ladder tournament

In this section, we focus on the Random ladder tournament which is a proper learner (with promise) of P in a finite class of n hypotheses, H . The algorithm is given a parameter Γ as an upper bound for OPT ( H , X ) . It outputs ˆ H ∈ H such that, with high probability we have: ∥ ˆ H -P ∥ TV ≤ 3 Γ + ϵ . Bousquet et al. in [BKM19] have shown that no proper learner can achieve an error better than 3 OPT + ϵ unless the sample complexity grows with the domain size |X| . Their lower bound holds even when the algorithm is provided with Γ = OPT . Hence, the factor α = 3 above is optimal.

For clarity in our presentation, in Section 3.1, we present the random ladder tournament assuming that: 1) We set the confidence parameter to be a small constant ( δ = 0 . 1 ). 2) the algorithm can call the COMPARE subroutine without specifying how this subroutine is implemented. In Section A.1, we show one can modify this algorithm to work for arbitrary small confidence parameter δ . Also, in Section 3.2, we discuss how one can implement this algorithm with memory constraints. We use the sample summary approaches described in Section 2.2.2, and we obtain two (sub-optimal) memory-data tradeoffs for this algorithm.

3.1 Random ladder tournament with no memory constraints

In this section, we present the random ladder tournament that solves the hypothesis selection problem while a parameter Γ is given to the algorithm as an upper bound for OPT . For this result, we assume there is a meta distribution over H , denoted by D , with a non-negligible probability p 0 to draw a hypothesis that is OPT -close. The algorithm considers hypotheses one at a time where at step i , we draw R i from the meta distribution. While later we use this algorithm with other meta distributions, it may help the reader to view the meta distribution as a uniform distribution over a set of n hypotheses, H . In this case, if we draw a random hypothesis from H , it is guaranteed to be Γ -close to P with probability p 0 ≥ 1 /n . Our algorithm finds a sufficiently close hypothesis using Θ(1 /p 0 ) comparisons.

At a high-level, our algorithm goes through a list of randomly drawn hypotheses from D , compares them, and keeps track of the current winner hypothesis. We denote the current winner hypothesis at step i by W i . Initially, we start with W 0 being equal to a fake hypothesis that loses to any other hypothesis. Then at every step, we take a randomly drawn hypothesis R i and compare it with W i -1 . We set the current winner of step i , W i , be the winner of a comparison between W i -1 and R i . We show that after k = Θ(1 /p 0 ) steps either the final winner, W k , is an Γ -close hypotheses, or many of W i 's that we have encountered are close to P . That is, either W k is an excellent hypothesis or a random W i is an acceptable choice. To exploit this fact at every step, we add each W i to a list, namely Q , with some small probability. At the end, we also add W k to Q . We prove a random hypothesis in Q will be close to P with high probability. Algorithm 2 shows the formal description of our approach, and we prove its performance in Theorem 4. Figure 1 illustrates a visual representation of the algorithm.

Theorem 4. Suppose we can draw i.i.d. random hypotheses from an arbitrary meta distribution D over H . And, we have a hypothesis P that we aim to learn properly in H . Assume we are given parameters p 0 and Γ such that the probability that a random hypothesis is Γ -close to P is at least p 0 . For any ϵ ∈ (0 , 1) , Algorithm 2 is ( α = 3 , ϵ, δ = 0 . 1) -proper learner (with promise) for the class H .

Proof. First, note that each comparison, invoked in Line 7, will satisfies the properties of Lemma 2.1 with probability at least 1 -1 / (100 k ) . Hence, using Lemma 2.1 and the union bound, one can assume with probability 0 . 99 for all the comparisons we have:

If W i is Γ -close to P , it will not lose; it remains as the current winner for the rest of the steps: W i = W i +1 = · · · = W k .

If W i -1 is (3Γ + ϵ ) -far from P , and R i is Γ -close to P , then W i is equal to R i .

For the rest of this proof, we fix the response of comparisons for which the above conditions hold. The randomness used in the rest of this proof only depends on the randomness in the meta distribution over hypotheses and the internal coin tosses of the algorithm.

Our goal is to show that most of the hypotheses in the list, Q , are acceptable, so a randomly selected hypothesis in Q will be acceptable as well. Our first step is to show that the expected number of acceptable hypotheses in Q is high compared to the size of Q . For each i in [ k ] , we define an indicator random variable I i that is one if we add an acceptable hypothesis to the Q at step i in Line 8, and zero otherwise. Also, we define another indicator variable I k +1 corresponding to the event that the final W k , which we added to Q in Line 9, is an acceptable hypothesis. More formally, we have:

I i := ✶ W i is acceptable, and we add W i to Q in Line 8. ∀ i ∈ [ k ] , I k +1 := ✶ W k is acceptable. . Clearly, the sum of I i 's indicates the number of acceptable hypotheses in Q , so we focus on finding a lower bound for the expected value of this quantity. Let α i be the probability of W i being acceptable. Without loss of generality, we set α 0 equal to zero. It is not hard to see that: E [ I i ] = p 0 · α i for i ∈ [ k ] . Moreover for the last indicator variable, we have:

Assuming the pairwise comparisons are done perfectly, R i being Γ -close to P , automatically implies that R i does not lose to any of the hypotheses R i +1 , . . . , R k . Thus, we have: k

Note that the probability of ∥ R i -P ∥ TV ≤ Γ is at least p 0 . Now given that R i is Γ -close to P , it will certainly win any hypotheses that is (3 Γ+ ϵ ) -far from P . Thus, the event that W i is (3Γ+ ϵ ) -far must have a lower probability than the event that W i -1 loses to R i . Therefore, we obtain the following lower bound:

Recall that we pick R i independently from all the previous hypothesis, R 1 , R 2 , . . . , R i -1 , so one can say W i -1 is independent of R i . This implies:

Putting it all together, the expected value of the sum of I i 's is:

The last inequality above is due to the fact that we set α 0 to zero. Note that the above equation states that the expected number of acceptable hypotheses in Q is at least k · p 0 . On the other hand, the expected number of hypotheses in Q is k · p 0 +1 . Thus, the expected number of unacceptable hypotheses in Q is at most one. Now, by Markov's inequality, the probability that we have more than 50 unacceptable hypotheses in Q is at most 0.02. And, by the Chernoff bound, we know that the probability of having less than 1000 many hypothesis in Q is at most:

It is not hard to see that the ratio of unacceptable hypotheses to the total number of hypotheses in Q is at most 50/1000=0.05. Therefore, a random hypothesis in Q is acceptable with probability at least 0 . 95 . Hence, by the union bound the total error probability is bounded by:

Pr [ Outputting an unacceptable hypothesis ] ≤ Pr [ Outputting an unacceptable hypothesis | valid comparisons + Pr [ at least one invalid comparisons ] ≤ 0 . 05 + 0 . 01 < 0 . 1

Therefore, the proof is complete.

3.2 Memory-data tradeoffs of random ladder tournament

As we have discussed, one of the advantages of Algorithm 2 is its flexibility in the usage of memory and data. In the following, we describe a tradeoff that we can obtain for this algorithm using the sample summaries presented in Section 2.2.2. At a high-level, the following is how we achieve the tradeoff: suppose we have b bits of memory. We choose the largest integer t so that we can store the sample summary needed to compare t hypotheses. Then, we run the random ladder tournament while we draw new samples and refresh the sample summary at every t step. Our two described sample summaries Scheff'e counts and the sorted list lead to the following memory-data tradeoffs. Although these tradeoffs are not as tight as our main result (by factors of log n and 1 /ϵ ), they have better accuracy guarantees ( α = 3 instead of α = 9 in our main result).

Lemma 3.1. Suppose we have n hypotheses in H . For every p 0 ≥ 1 /n , ϵ , and an integer t between two and k = Θ(1 /p 0 ) , one can implement Algorithm 2 in such away that it uses:

Lemma 3.2. Suppose we have n hypotheses in H . For every p 0 ≥ 1 /n , ϵ , and an integer t between two and k = Θ(1 /p 0 ) , one can implement Algorithm 2 in such away that it uses:

For the proofs of the above lemmas, see Section A.2.

]

References

[AAC + 23] Anders Aamand, Alexandr Andoni, Justin Y. Chen, Piotr Indyk, Shyam Narayanan, and Sandeep Silwal. Data structures for density estimation. In Proceedings of the 40th International Conference on Machine Learning . PMLR, 2023.

[ABH + 20] Hassan Ashtiani, Shai Ben-David, Nicholas J. A. Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. J. ACM , 67(6):32:1-32:42, 2020.

[ABM18] Hassan Ashtiani, Shai Ben-David, and Abbas Mehrabian. Sample-efficient learning of mixtures. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , pages 2679-2686. AAAI Press, 2018.

[ADLS17] Jayadev Acharya, Ilias Diakonikolas, Jerry Li, and Ludwig Schmidt. Sample-optimal density estimation in nearly-linear time. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 1278-1289. SIAM, 2017.

[AF23] Tomer Adar and Eldar Fischer. Refining the adaptivity notion in the huge object model. CoRR , abs/2306.16129, 2023.

[AFJ + 18] Jayadev Acharya, Moein Falahatgar, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Maximum selection and sorting with adversarial comparators. The Journal of Machine Learning Research , 19:59:1-59:31, 2018.

[AJOS14] Jayadev Acharya, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Sorting with adversarial comparators and application to density estimation. In 2014 IEEE International Symposium on Information Theory , pages 1682-1686, 2014.

[AMNW22] Maryam Aliakbarpour, Andrew McGregor, Jelani Nelson, and Erik Waingarten. Estimation of entropy in constant space with improved sample complexity. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 32474-32486. Curran Associates, Inc., 2022.

[BBK + 21] Olivier Bousquet, Mark Braverman, Gillat Kol, Klim Efremenko, and Shay Moran. Statistically near-optimal hypothesis selection. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS 2021, Denver, CO, USA, February 7-10, 2022 , pages 909-919. IEEE, 2021.

[BBS22] Gavin Brown, Mark Bun, and Adam Smith. Strong memory lower bounds for learning natural models. In Proceedings of Thirty Fifth Conference on Learning Theory COLT , 2022.

[BKM19] Olivier Bousquet, Daniel Kane, and Shay Moran. The optimal approximation factor in density estimation. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 318-341. PMLR, 25-28 Jun 2019.

[BKSW19] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing

Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 156-167, 2019.

[CDKL22] Clément L. Canonne, Ilias Diakonikolas, Daniel Kane, and Sihan Liu. Nearly-tight bounds for testing histogram distributions. In Advances in Neural Information Processing Systems 35 (NeurIPS) , 2022. To appear.

[CDSS14] Siu-on Chan, Ilias Diakonikolas, Rocco A. Servedio, and Xiaorui Sun. Near-optimal density estimation in near-linear time using variable-width histograms. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada , pages 1844-1852, 2014.

[CFG + 23] Sourav Chakraborty, Eldar Fischer, Arijit Ghosh, Gopinath Mishra, and Sayantan Sen. Testing of index-invariant properties in the huge object model. In 36th Annual Conference on Learning Theory - COLT , 2023.

[CKM + 19] Clément L. Canonne, Gautam Kamath, Audra McMillan, Adam D. Smith, and Jonathan R. Ullman. The structure of optimal private tests for simple hypotheses. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 310-321. ACM, 2019.

[DDS12] Constantinos Daskalakis, Ilias Diakonikolas, and Rocco A. Servedio. Learning poisson binomial distributions. In Howard J. Karloff and Toniann Pitassi, editors, Proceedings of the 44th Symposium on Theory of Computing Conference, STOC 2012, New York, NY, USA, May 19 - 22, 2012 , pages 709-728. ACM, 2012.

[DGKR19] Ilias Diakonikolas, Themis Gouleakis, Daniel M. Kane, and Sankeerth Rao. Communication and memory efficient testing of discrete distributions. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 1070-1106. PMLR, 25-28 Jun 2019.

[Dia16] Ilias Diakonikolas. Learning structured distributions. In CRC Handbook of Big Data , pages 267-283. 2016.

[DK14] Constantinos Daskalakis and Gautam Kamath. Faster and sample near-optimal algorithms for proper learning mixtures of gaussians. In Conference on Learning Theory , pages 1183-1213. PMLR, 2014.

[DKS17] Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In Chris Umans, editor, 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, Berkeley, CA, USA, October 15-17, 2017 , pages 73-84. IEEE Computer Society, 2017.

[DL96] Luc Devroye and Gábor Lugosi. A universally acceptable smoothing factor for kernel density estimates. The Annals of Statistics , 24(6):2499 - 2512, 1996.

[DL97] Luc Devroye and Gábor Lugosi. Nonasymptotic universal smoothing factors, kernel complexity and Yatracos classes. The Annals of Statistics , 25(6):2626 - 2637, 1997.

[DL01] Luc Devroye and Gábor Lugosi. Combinatorial methods in density estimation . Springer, 2001.

[DS18] Yuval Dagan and Ohad Shamir. Detecting correlations with little memory and communication. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory , volume 75 of Proceedings of Machine Learning Research , pages 1145-1198. PMLR, 06-09 Jul 2018.

[GKK + 20] Sivakanth Gopi, Gautam Kamath, Janardhan Kulkarni, Aleksandar Nikolov, Zhiwei Steven Wu, and Huanyu Zhang. Locally private hypothesis selection. In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 1785-1816. PMLR, 2020.

[GR22] Oded Goldreich and Dana Ron. Testing Distributions of Huge Objects. In Mark Braverman, editor, 13th Innovations in Theoretical Computer Science Conference (ITCS 2022) , volume 215 of Leibniz International Proceedings in Informatics (LIPIcs) , pages 78:1-78:19, Dagstuhl, Germany, 2022. Schloss Dagstuhl - Leibniz-Zentrum für Informatik.

[KMV12] Adam Tauman Kalai, Ankur Moitra, and Gregory Valiant. Disentangling gaussians. Commun. ACM , 55(2):113-120, 2012.

[KSS18] Pravesh K. Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via sum of squares. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018 , pages 1035-1046. ACM, 2018.

[MS08] Satyaki Mahalanabis and Daniel Stefankovic. Density estimation in linear time. In 21st Annual Conference on Learning Theory - COLT , pages 503-512, 2008.

[Pea95] Karl Pearson. Mathematical contributions to the theory of evolution. ii. skew variation in homogeneous material. [abstract]. Proceedings of the Royal Society of London , 57:257-260, 1895.

[Raz18] Ran Raz. Fast learning requires good memory: A time-space lower bound for parity learning. Journal of the ACM (JACM) , 66(1):1-18, 2018.

[SD15] Jacob Steinhardt and John Duchi. Minimax rates for memory-bounded sparse linear regression. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 1564-1587. PMLR, 03-06 Jul 2015.

[Sha14] Ohad Shamir. Fundamental limits of online and distributed algorithms for statistical learning and estimation. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada , pages 163-171, 2014.

[SOAJ14] Ananda Theertha Suresh, Alon Orlitsky, Jayadev Acharya, and Ashkan Jafarpour. Near-optimal-sample estimators for spherical gaussian mixtures. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27, 2014.

[SSV19] Vatsal Sharan, Aaron Sidford, and Gregory Valiant. Memory-sample tradeoffs for linear regression with small error. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC , pages 890-901. ACM, 2019.

[Vad12] Salil P. Vadhan. Pseudorandomness. Foundations and Trends® in Theoretical Computer Science , 7(1-3):1-336, 2012.

[Yat85] Yannis G. Yatracos. Rates of convergence of minimum distance estimators and kolmogorov's entropy. The Annals of Statistics , 13(2):768 - 774, 1985.

winner

W

0

Compare

W

1

Compare

...

W

2

Compare

W

k

R

1

R

2

R

k

Q

ˆ

H

R

i

's are drawn from

H

according to the meta distribution.

W

i

's are added to

Q

with

some small probability.

ˆ

H

is selected from members

of

Q

uniformly at random.

W

k

is always added to

Q.

A Random ladder tournament (cont.)

Figure 1 illustrates a visual representation of the algorithm.

A.1 Amplifying the confidence parameter

In this section, we describe how one can amplify the confidence parameter of the random ladder tournament. Note that to keep α = 3 , we should avoid any two-step process that comes up with a collection of hypotheses to narrow down the possible choices and then pick an acceptable hypothesis among them. Such approaches generally lead to α = 9 or higher. Here, we argue that if we run the random ladder tournament for a longer time, a random hypothesis in Q will be acceptable with very high probability. In particular, if we set k , the number of steps equal to Θ(1 / ( δ p 0 )) (instead of setting k = Θ(1 /p 0 ) ), we find an acceptable hypothesis with probability at least 1 -δ .

Corollary A.1. Suppose that we can draw i.i.d. random hypotheses from an arbitrary meta distribution D over H . And, we have a hypothesis P that we aim to learn properly in H . Assume we are given parameters p 0 and Γ such that the probability that a random hypothesis drawn from the meta distribution is Γ -close to P is at least p 0 . For any ϵ, δ ∈ (0 , 1) , Algorithm 2 can be modified to an ( α = 3 , ϵ, δ ) -proper learner (with promise) for class H by setting k := ⌈ 2 ⌈ 2 /δ ⌉ /p 0 ⌉ = Θ(1 / ( δ p 0 )) .

Proof. Note that δ in Theorem 4 is 0.1. Hence, it suffices to focus on δ < 0 . 1 . As we have shown in the proof of Theorem 4, the expected number of unacceptable hypotheses in Q is at most one. Let ˆ H be a random hypothesis in Q . Let t := ⌈ 2 /δ ⌉ , and set k to ⌈ 2 t/p 0 ⌉ . It is not hard to see that the probability of ˆ H being unacceptable is bounded by δ :

Now, we separate the first t terms in the summation, and bound them from above by the probability of | Q | being at most t . Moreover, we use the fact that the size of | Q | is essentially a binomial random variable drawn from Bin ( k, p 0 ) plus one. Hence, we obtain:

(By the Hoeffding inequality)

where the last inequality holds due to the fact that e -1 / (2 δ ) < δ/ 4 for δ ≤ 0 . 1 . Hence, the proof is complete.

A.2 Proofs of Memory-data tradeoffs of random ladder tournament

Lemma 3.1. Suppose we have n hypotheses in H . For every p 0 ≥ 1 /n , ϵ , and an integer t between two and k = Θ(1 /p 0 ) , one can implement Algorithm 2 in such away that it uses:

Proof. Recall that it suffices to draw m := O (log p -1 0 /ϵ 2 ) samples to make sure all the comparisons we make in Line 7 are accurate with probability at least 0.99. Since we obtain this accuracy bound via the union bound, we may reuse the samples. The idea here is to process every t hypotheses together with the same set of m samples. To implement this algorithm in b bits of memory, we first draw t random hypotheses and store their indices using O ( t log n ) bits. Then, we draw m sample and store the Scheffé counts of every pair of hypotheses that are drawn. Counting the number of samples in each Scheffé sets takes O (log m ) = O (log(log n/ϵ )) bits. And, there are O ( t 2 ) many of them. Now, we can make all the comparisons between the t hypotheses using O ( t 2 log m ) = O ( t 2 log(log p -1 /ϵ )) bits. Now, we run the first t steps of Algorithm 2 and compute the current winner up to round t , w t . Then, repeat this process for w t and the next t -1 random hypotheses with fresh samples until we reach the end of k hypotheses. Since at each round we process t -1 hypotheses (with the exception of the first round), the number of rounds is ⌈ ( k -1) / ( t -1) ⌉ = O (1 / ( t p 0 )) , so we use O ( m/ ( t p 0 )) samples in total. Hence, Algorithm 2 uses O (log n/ ( tϵ 2 p 0 )) samples and b = O ( t 2 log(log n/ϵ ) + t log n ) bits of memory overall.

Lemma 3.2. Suppose we have n hypotheses in H . For every p 0 ≥ 1 /n , ϵ , and an integer t between two and k = Θ(1 /p 0 ) , one can implement Algorithm 2 in such away that it uses:

winner

W

0

Compare

W

1

Compare

...

Compare

W

k

R

1

R

2

R

k

ˆ

H

Filtering algorithm

Filtering algorithm

Filtering algorithm

...

R

i

's are the output of the filtering algorithm.

Random Ladder Tournament

Proof. The proof is very similar to the proof of Lemma 3.1. However, instead of using the Scheffé counts we use the sorted lists to store the samples. The only difference is that to process t hypotheses, we will need O ( mtlogn ) = O ( t log p -1 0 log n/ϵ 2 ) bits instead of O ( t 2 log m ) bits. This observation concludes the proof.

B Main result

In this section, we present an algorithm that achieves the tradeoff described in our main theorem. As we have described in Section 1.2, we obtain our main result by running the random ladder tournament on the hypotheses filtered by a filtering algorithm.

Recall that in the random ladder tournament, the algorithm can draw a random hypothesis such that with a probability of at least p 0 , it is a Γ -close hypothesis. The number of comparisons made in the random ladder tournament is approximately k := Θ(1 /p 0 ) . To reduce k , we design a randomized filtering algorithm that produces an acceptable hypothesis ( (3 Γ + ϵ ′ ) -close to P ) with a certain descent probability. Roughly speaking, this probability is min(1 / √ n, b/n ) . Although this probability is not very high, it is still greater than 1 /n , which allows us to execute the random ladder in fewer steps. The only caveat is that we must now run the random ladder tournament with a new parameter Γ ′ := 3 Γ + ϵ ′ . Hence, the accuracy guarantee of the resulting hypothesis will also be based on Γ ′ , and we will obtain a (9 Γ + Θ( ϵ ′ )) -close hypothesis to P . Figure 2 illustrates a visual representation of this approach.

The filtering algorithm operates on the basis of a simple observation: the excellent hypothesis H ∗ will not lose to any unacceptable hypotheses (those that are (3 Γ + ϵ ′ ) -far). Therefore, if there is no decent hypothesis, one can identify the excellent hypothesis H ∗ through an elimination tournament, where the algorithm pairs all hypotheses, compares them, and repeats this process while only the winners advance to the next round. Conversely, if numerous decent hypotheses exist, it suggests that a random hypothesis will have a reasonable chance of being decent (and hence acceptable). The

filtering algorithm combines these two strategies and does the following for each with probability a half:

It outputs a random hypothesis.

It runs an elimination tournament under the assumption that there are no decent hypotheses, then it outputs the result.

This approach leads to finding an excellent or decent hypothesis with an acceptable chance regardless of how scarce the decent hypotheses are.

Note that to run an elimination algorithm, we need to memorize the winners of each round. Hence, implementing the second part above with limited memory is not trivial. Therefore, we focus on a randomly sampled subset of elements and provide an algorithm that is extremely memory efficient. Roughly speaking, the algorithm processes n hypotheses using only O ( n ) bits of memory. This is one of the key components of our result that allows us to avoid extra log n factors and achieve the tradeoff s · b ≈ O ( n log n ) .

The algorithm operates in rounds, employing a tree-like structure. In each round of this algorithm, we group a carefully chosen number of hypotheses and run an all-against-all tournament among them. Next, the winner of each group proceeds to the next round. We continue this process until we find a single winner.

In Section B.1, we describe the group elimination tournament. In Section B.2, we detail how to conduct the group elimination tournament on a random set of n ′ hypotheses while utilizing only n ′ bits of memory. In Section B.3, we outline the filtering algorithm. Finally, we conclude with Section B.4, which combines all the pieces together and presents the proof of our main result.

B.1 Group elimination tournament

In this section, we focus on the case that there is no decent hypothesis in H . That is, every hypothesis that is OPT -far from P will lose to a hypothesis H ∗ with ∥ H ∗ -P ∥ TV = OPT . The main advantage of this assumption is that if we have a knockout style tournament: only an OPT -close hypothesis will survive to the top. Here, we propose an algorithm that implements a knockout style tournament, which we call group elimination tournament. However, instead of comparing two hypotheses at every step, we run the all-go-against-all tournament among a small group of hypotheses and send the winner to the next round. For simplicity, in this section, we describe our approach in Algorithm 3 where we assume no memory restriction. We prove the performance of this algorithm in Theorem 5. For a discussion on how to implement this algorithm using roughly O ( n ) bits of memory, see Section B.2.

Proof. The algorithm runs in L rounds. At round ℓ ∈ [ L ] , we start with s ℓ hypotheses. We partition them into groups of size g ℓ . For each round, we draw m ℓ := Θ(log( g 3 ℓ /δ ) /ϵ 2 ) fresh samples. We keep track of the number of samples in the Scheffé sets of every pair of hypotheses that are in the same group. Using the Scheffé counts, we run an all-go-against-all tournament in each group G i with overall confidence probability 1 -δ/ (6 g i ) and find the winner. Then, we repeat this process among the winners in the next round until we end up with only one hypothesis. See Algorithm 3 for the description.

Parameters: We set the group sizes as follows:

We start with s 1 := n . The number of hypotheses in the ℓ -th round is:

The equation above is due to Fact D.2 and Fact D.3.

Observe that we stop our process when only one hypothesis is left: the ultimate winner. Thus, L is the smallest integer such that s L +1 is one. That is, ∏ L ℓ =1 g ℓ ≥ n . It is not hard to show that L is O (log (log r n )) due to the following argument. For any L ≥ log 1 . 5 (log r n +1) , one can show ∏ L ℓ =1 g ℓ is at least n via Fact D.3:

Therefore, L cannot be larger than ⌈ log 1 . 5 (log r n +1) ⌉ .

We describe how we set our parameters in Table 1.

Sample complexity: For round ℓ , we draw m ℓ fresh samples. Thus, the total number of samples is:

Note that in the last line we use the fact that ∏ L ℓ =1 g ℓ is poly( n ) which is not difficult to prove using the following argument: We define L to be the smallest integer such that ∏ L ℓ =1 g ℓ is at least n . Thus, ∏ L -1 ℓ =1 g ℓ is at most n . Hence, we have:

Correctness: Suppose G is a group in round ℓ that contains an OPT -close hypothesis. For the all-go-against-all tournament in round ℓ we use m ℓ = O (log( g 3 ℓ /δ ) /ϵ 2 ) samples. That implies we estimate the probability of all the Scheffé sets in G with accuracy ϵ with probability at least 1 -δ/ (6 g ℓ ) . Given that we do not have any decent hypothesis, using Fact D.1, the all-go-against-all tournament has to output an OPT -close hypothesis with probability 1 -δ/ (6 g ℓ ) .

Now, let G (1) be the group in round one that H ∗ belongs to. Let G (2) be the group that the winner of G (1) belongs to in round two, and similarly we define: G (3) , . . . , G ( L ) . Clearly, the all-go-against-all tournament over G ( i ) will return an OPT -close hypothesis with probability 1 -δ/ (6 g ℓ ) if G ( i ) contains an OPT -close hypothesis. Thus, using the union bound, the probability that the final group G ( L ) does not output an OPT -close hypothesis is bounded by the following:

B.2 Memory usage of Algorithm 3

In this section, we provide an approach to implement Algorithm 3 using roughly O ( n ) bits of memory and prove our main theorem.

Theorem 5. Suppose we have a set of n hypotheses H = { H 1 , H 2 , . . . , H n } and an unknown hypothesis P which we wish to properly learn in H . Suppose there are no decent hypothesis in H , that is all hypotheses in H are either OPT -close, or they will lose to any OPT -close hypothesis. For every 2 ≤ r ≤ n , there exists an algorithm that uses O ( n · r · log(log(1 /δ ) /ϵ )) bits of memory and O ((log(log r n ) · log(1 /δ ) + log n ) /ϵ 2 ) samples from P , and it outputs ˆ H that is OPT -close to P with probability 1 -δ .

Proof. Note that there are two main sets of values that we need to store. First, the values of the Scheffé counts at every round that allow us to perform the all-go-against-all tournaments. Second, the

indices of the winner hypotheses that go to the next round. Below, we discuss bounding the memory usage for each part.

Memory usage of Scheffé counts: Here, we show to store the Scheffé counts, we need O ( n · r · log(log(1 /δ ) /ϵ )) bits at every round. For a single group of size g ℓ , there are O ( g 2 ℓ ) pairs of hypotheses, therefore Scheffé sets, so we require O ( g 2 ℓ log m ℓ ) bits to count the number of samples in all the sets. Thus, in round ℓ , we need the following number of bits:

Next, we show the last term in the last line is O ( r ) . For round 1 , the memory bound holds since √ r · log( √ r ) ≤ r . Now for ℓ ≥ 2 using Equation (3), we have:

Hence, the bit complexity at every step is:

Storing the indices: Note that to run the comparisons, one needs the true indices of the hypotheses to query the PDF-comparator or draw a sample. Thus, in the trivial implementation of our algorithm, we potentially need O ( n log n ) bits to keep track of the indices of the winners at every round. In this section, we explain how one can implement our algorithm in a memory-efficient manner, so keeping track of the winners' indices does not take more than O ( n ) at every round.

We start by the following assumption that there is natural ordering among the hypotheses: H 1 , H 2 , . . . , H n . In this way, for the first round we do not need to remember the set of indices, we only need to remember n . At every level, we group 'adjacent" hypotheses together and preserve this natural ordering. More precisely, at round one with group size g 1 , the groups are:

For the next round, we keep a list of ⌈ n/g 1 ⌉ winners. However, instead of fully writing down the index of the winners, we use O (log( g 1 )) bits per each, and save the index of the winner within the group. For the next round, we partition the hypotheses into groups of size g 2 keeping the 'adjacent" hypotheses in the same group. We repeat the same process to store the winners. However now, each winner are 'representing" O ( g 1 · g 2 ) hypotheses, so we need O (log( g 1 · g 2 )) bits to store its index. We continue this approach. At round ℓ > 1 , we have a list of s ℓ = O ( n/ ∏ ℓ -1 i g i ) hypotheses, and we require O (log ∏ ℓ -1 i g i ) many bits to do so. Thus, at every step we need O ( n ) bits of memory.

Note that at every point in the process, it is easy to compute the actual index from the stored index. If right before starting round i , the j -th index in the list is k , then the true index is:

B.3 Filtering acceptable hypotheses

In this section, we describe a filtering algorithm that allow us to pick an acceptable hypothesis with a modest chance. Our algorithm is simple we draw roughly n ′ ≪ n hypothesis at random. If one of these hypothesis is OPT -close, and there is no decent hypothesis among them, Algorithm 3 returns an excellent hypothesis. On the other hand, if there are a lot of decent hypotheses, then there is a modest chance that a randomly selected hypothesis is decent, hence an acceptable one.

In the description of Algorithm 3, we assume we have a class of hypotheses of size n , and roughly O ( n ) bits of memory. However, we aim to use this algorithm in a setting that we can process n ′ random hypotheses in roughly O ( n ′ ) bits of memory. The primary challenge here is that we need roughly O ( n ′ log n ) many bits to memorize which hypotheses are participating, and we cannot use the 'natural ordering" of the hypotheses that we use in the proof of Theorem 5. Thus, we are looking for a mapping that maps indices in [ n ′ ] to a random set of indices in [ n ] in a memory-efficient manner. To do so, we relax our requirement regarding that we need to draw n ′ random hypotheses uniformly from H . Instead, we are looking for a set of random pairwise independent indices while finding the index of the selected elements requires small amount of memory. This relaxation gives us a process that filters acceptable hypotheses:

Theorem 6. Suppose we have a set of n hypotheses H = { H 1 , H 2 , . . . , H n } and an unknown hypothesis P which we wish to properly learn in H . For every positive integer n ′ < n , there exists an algorithm that uses O ( n ′ · log(1 /ϵ )) bits of memory and O ((log n ) /ϵ 2 ) samples from P , and it outputs ˆ H that is excellent or decent with the probability:

Proof. First, we start off by describing how we randomly pick n ′ hypotheses from the set of n hypotheses. We use a standard technique for generating pairwise independent random indices described in [Vad12, Chapter 3.5]. The following is the description of a randomized mapping that for any given index x ∈ [ n ′ ] it gives an index i in [ n ] that can be computed in a memory efficient manner.

Let q be a prime number between n and 2 n ( n < q < 2 n ) . Such q always exists via the BertrandChebyshev theorem for any n ≥ 2 . Without loss of generality, we assume we have q hypotheses by adding fake hypotheses to H that lose to any other hypothesis. Now, consider the finite field of Z q . Proposition 3.24 in [Vad12] implies that if we use two random numbers a and b in Z q , then the following mapping generates a set of pairwise independent indices:

More precisely, by iterating x from 1 to n ′ , this randomized mapping selects n ′ hypotheses with the following indices: f a,b (1) , f a,b (2) , . . . , f a,b ( n ′ ) . 3 Note that to compute this mapping, we only need to memorize a and b , which requires O (log n ) bits, significantly smaller compared to O ( n ′ log n ) which we would need to memorize n ′ random indices. Using the pairwise independence of the indices generated by this mapping, it is not too difficult to show that with some modest chance we pick exactly one excellent hypothesis and no decent hypothesis with this mapping.

̸

Lemma B.2. For a prime q , assume we have a class of q hypotheses that contains t e ≥ 1 excellent hypotheses and t d decent hypotheses. Let a = 0 and b be two random numbers in Z q selected uniformly at random. Let H ′ be the set of following hypotheses: { H f a,b ( x ) : x ∈ [ n ′ ] } for which f a,b ( x ) := a · x + b ( mod q ) . If ( t e + t d ) /q ≤ 1 / (2 n ′ ) , then the probability that H ′ contains exactly one excellent hypothesis and no decent hypothesis is at least n ′ · t e / (2 q ) .

See Section D for the proof of this lemma.

Algorithm 4 Filtering Algorithm

R

x

's

are

H

a

·

x

+

b

( mod

q

)

Group Elimination

Algorithm

R

1

R

2

R

n

′

. . .

ˆ

H

H

i

Random Hypothesis in

H

with probability 1/2

with probability 1/2

The algorithm: Assume H contains t e ≥ 1 prefect hypotheses and t d decent hypotheses. The algorithm is simple: with probability a half we set ˆ H to be a random hypothesis from H . And, with probability a half we use the randomize mapping to pick n ′ hypotheses and run Algorithm 3 with parameters δ = 0 . 5 and r = 2 on them. And, we output the output of that algorithm. Figure 3 shows a visual representation of the filtering algorithm.

The probability of getting an acceptable hypothesis by picking a random one is ( t e + t d ) /n . Note that if this quantity is at least 1 / 2 n ′ , the statement of the lemma is true. Thus, assume ( t e + t d ) /n is less than 1 / 2 n ′ which implies that ( t e + t d ) /q is less than 1 / 2 n ′ . Given this condition, by the above lemma, our randomized mapping selects a set of hypotheses with only one excellent hypothesis and no other acceptable hypotheses with probability at least n ′ · t e / (2 q ) . And, since we set δ to a half, with probability at least a half, Algorithm 3 will choose the excellent hypothesis as its output. Using that q < 2 n and Pr [ C = tail ] = 1 / 2 , with probability at least n ′ · t e / (8 q ) ≤ n ′ / (16 n ) , ˆ H is the excellent hypothesis as desired in the statement. Hence, the proof is complete.

B.4 Putting everything together

Theorem 1. There exists a constant c for which the following holds. Let H be an arbitrary class consisting of n distributions, and let ϵ > 0 . For every natural number b where c · (log n +log((log n ) /ϵ )) ≤ b ≤ n , there exists an ( α = 9 , ϵ, δ = 0 . 1) -proper learner (with promise) for H using b bits of memory and the following number of samples:

Proof. Here, we combine the filtering algorithm and the random ladder tournament to prove the upper bound. We first use the filtering algorithm, Algorithm 4, to increase the probability of getting an acceptable hypothesis ( 3 Γ+ ϵ ′ -close P ) with parameter ϵ ′ = ϵ/ 4 . Then, we use the random ladder tournament, Algorithm 2, to select an acceptable hypothesis among the hypotheses that have passed the filtering. Note that in the random ladder tournament, we need to use a new Γ ′ parameter, that is, 3 Γ + ϵ ′ .

Here, we use the memory bounds provided in Theorem 6 and Lemma 3.1 for these two algorithms. Fix two small constants c 0 , c 1 ≤ 1 that we determine later. Let t be the largest integer such that:

Note that given our lower bound for b , one can find a t that is at least two. Set n ′ as follows:

We consider two cases in the following:

Case 1: n ′ < 2 √ n n ′ < 2 √ n n ′ < 2 √ n . In this case, n ′ is equal to c 1 · b/ log(1 /ϵ ) . One can run Algorithm 4 and find an acceptable hypothesis with probability p 0 := min( n ′ / (16 n ) , 1 / (4 n ′ )) = n ′ / (16 n ) . Now in this case, we invoke Algorithm 4 O (1 /p 0 ) times, and in each round we use O (log n/ϵ 2 ) samples. Furthermore, the random ladder tournament uses O ((log p -1 0 ) / ( t p 0 ϵ 2 )) samples. Therefore overall, our sample complexity is:

Case 2: n ′ = 2 √ n n ′ = 2 √ n n ′ = 2 √ n . Similar to the previous case, we would like to run Algorithm 4 and find an acceptable hypothesis with probability p 0 := min( n ′ / (16 n ) , 1 / (4 n ′ )) = 1 / (4 n ′ ) . Now, since n ′ ≤ c 1 · b/ log(1 /ϵ ) , we may be able to run multiple instances of Algorithm 4 in parallel while using the same number of samples. Let r = c 1 · b/ ( n ′ · log(1 /ϵ )) . Now, the number of samples we use for this stage is:

Now, we focus on the number of samples we need for the random ladder tournament. Note that t should be:

Thus, we use the following amount of samples:

Putting these two cases together, we obtain the desire sample-memory tradeoffs:

Regarding the accuracy guarantee, it is not hard to see that in the filtration algorithm, with a modest probability, we pick a distribution that is (3 Γ + ϵ ′ ) -close to P . Now, we run the random ladder tournament with a new Γ ′ = 3Γ + ϵ ′ . Then, we get a hypothesis with accuracy:

Hence, the proof is complete.

C Proper learner without promise

Throughout this paper, we mainly focus on proper learners with promise, for which the algorithm is given a parameter Γ , and we are promised that OPT is at most Γ . In this section, we focus on the case where no such information is available to the algorithm. First, we formally define a proper learner without promise.

Definition C.1. Suppose A has sample access to an unknown distribution P . And, it can query the probabilities of the Scheffé sets according to each H i ∈ H and a PDF-comparator for every pair of hypotheses in H . We say algorithm A is an ( α, ϵ, δ ) -proper learner for H if the following holds: for every distribution P , and for every class of n distributions H , with probability at least 1 -δ over its input sample (drawn i.i.d. from P ) and internal coin tosses, A outputs a distribution ˆ H ∈ H such that:

Here, we provide a reduction from a proper learner without promise to a proper learner with promise. At a high level, we perform a binary search over possible values of Γ in (0 , 1) . At every step, we try a value of Γ m and run A with Γ = Γ m . The main challenge is that regardless of Γ m being at least OPT or not, A may return a hypothesis that seems close to P , and it is hard to refute that the suggested value of Γ is at least OPT . For this reason, we look into the output of A , say ˜ H , and see if we find evidence that ˜ H is far from P . We check the distance between ˜ H and P on every Scheffé set of ˜ H . More precisely, we check if W ( ˜ H ) is larger than ( α · OPT + ϵ ) where W ( H ) is defined as follows for every hypothesis H i :

Now, if W ( ˜ H ) is larger than ( α · OPT + ϵ ) , then one can imply that ˜ H is not ( α · OPT + ϵ ) -close to P , therefore, Γ m must have been less than OPT . On the other hand, if W ( ˜ H ) is at most ( α · OPT + ϵ ) , one can show ˜ H is not too far away from P (even when Γ m is less than OPT ). Putting these observations together and accounting for errors in estimation of W ( ˜ H ) , we obtain a reduction that is described in Algorithm 5, and we prove its accuracy in Theorem 7.

Proof. Our goal here is to show that for the output hypothesis ˆ H with probability at least 1 -t · δ + δ ′ , we have:

Observe that at every iteration of the while loop in the algorithm, Γ h -Γ ℓ is divided by two. Thus, the while loop takes ⌈ log 2 (1 /ϵ ) ⌉ iterations. Thus, t is the number of times that we invoke A in Algorithm 5. Each time A works as expected with probability 1 -δ . Hence, using the union bound, with probability 1 -t · δ , we can assume all invocations of A result in correct answers. In addition, with probability 1 -δ ′ , all the estimates we use for P ( S ( ˆ H,H i ) ) in Line 8 are accurate up to an additive error ϵ ′ . Hence, ∣ ∣ ∣ ˜ W ( ˜ H ) -W ( ˜ H ) ∣ ∣ ∣ is at most ϵ ′ with probability at least 1 -δ ′ . Hence, by the union bound, the following event happens with probability at lest 1 -( t · δ + δ ′ ) : In every invocation of A , if OPT is below the given parameter Γ , then A returns a hypothesis, ˜ H , that is ( α Γ+ ϵ ) -close to P . And, ˜ W ( ˜ H ) is at most α Γ+ ϵ + ϵ ′ . Hence, for the rest of this proof, we assume this event holds, and we prove Equation 4.

Consider Γ h and Γ ℓ in the final iteration of while loop in Algorithm 5. Observe that the output ˆ H is what A has found when it was invoked with input Γ = Γ h . Now, we consider two cases as follows:

Case 1: OPT ≤ Γ h OPT ≤ Γ h OPT ≤ Γ h : In this case, ˆ H is ( α · Γ h + ϵ ) -close to P by our assumption that A works as expected. Thus, ∥ ˆ H -P ∥ TV is at most α · Γ h + ϵ . This upper bound is written based on Γ h , while we aim to find an upper bound on the distance that only contains OPT . For this purpose, we need to use the fact that why the binary search algorithm stoped at this particular Γ h . For Γ ℓ , we have two possibilities: either Γ ℓ is zero; Or, we have run A with Γ = Γ ℓ , and we have not found a desired ˜ H .

In both cases, Γ ℓ is a lower bound on OPT . Thus, we have:

Now, using this relationship between Γ h and OPT , we obtain:

implying Equation 4.

Case 2: OPT > Γ h OPT > Γ h OPT > Γ h : In this case, clearly, Γ h cannot be one (since OPT is at most one). Thus, Γ h was set in Line 10 implying that ˜ W ( ˆ H ) is at most α Γ h + ϵ + ϵ ′ . Since we assumed the estimation of W ( ˆ H ) was accurate up to error ϵ ′ , we have:

Given this bound on W ( ˆ H ) , we show that ˆ H cannot be far from P . Let H ∗ be a hypothesis in H that is OPT -close to P . Then, we use the definition of the Scheffé sets and the triangle inequality to bound the distance between ˆ H and P :

Now, we use the definition of W ( ˆ H ) and conclude:

Therefore, the proof is complete.

Corollary C.2. For every parameters ϵ ′′ and δ ′′ in (0 , 1) , there exists an ( α = 5 , ϵ = ϵ ′′ , δ = δ ′′ ) -proper learner (without promise) that uses O ( log( n/δ ′′ ) / ( ϵ ′′ ) 2 ) samples and runs ˜ O ( n/ ( δ ′′ ( ϵ ′′ ) 2 )) time.

Proof. First, we start off with the parameters: We set ϵ := ϵ ′′ / 6 , ϵ ′ := ϵ ′′ / 6 , δ ′ = δ ′′ / 2 , and δ = δ ′′ / (2 t ) where t := ⌈ log 2 (1 /ϵ ) ⌉ + 1 is the number of iterations in Algorithm 5. Using the Chernoff bound and reusing all samples, we can estimate all the Scheffé sets with additive error at most ϵ ′ with probability at least 1 -δ ′ .

Note that based on Corollary A.1, the random ladder tournament (with an arbitrarily small confidence parameter) is a ( α = 3 , ϵ, δ ) -proper learner (with promise) that uses O ( log( n/δ ) /ϵ 2 ) samples and runs in O ( n log( n/δ ′′ ) / ( δ ′′ ( ϵ ′′ ) 2 )) time. We use this learner as our A and run the reduction algorithm B with the parameters we specified earlier. Now, using Theorem 7, we obtain a proper learner without promise that satisfies the following guarantee with probability at least 1 -δ ′′ = 1 -( δ ′ + t · δ ) :

Regarding the time complexity, the while loop in Algorithm 5 is repeated t times. In each iteration, we compute W ( ˜ H ) which takes O ( n log ( n/δ ′′ ) / ( ϵ ′′ ) 2 ) time, and we call the random ladder tournament (i.e, A ). Hence, the overall time complexity is:

Hence, the proof is complete.

Memory-data consumption of reduction algorithm: Besides the sample and memory usage of A , B needs to keep track of O (1) numbers and also computes ˜ W ( ˜ H ) . Now, given b bits of memory, we can partition H into consecutive blocks of size r = b/ (log(log( n ) /ϵ )) . At every round, we process one block of size r , we keep track of the Scheffé counts of ˜ H and the hypotheses in the block. For each round we use O (log( n ) /ϵ 2 ) samples, ane we have O ( n/r ) rounds. Thus, the total number of samples is s := O ( n/r · (log( n ) /ϵ 2 )) . Thus, for this part we will have:

Remark 8. Using the reduction algorithm with the memory-data tradeoff described above, one can translate our main theorem, Theorem 1, to a ( α = 11 , ϵ, 0 . 1) -proper learner (without promise) incurring additional factors of poly (log log( n ) , log(1 /ϵ )) .

D Auxiliary lemmas and facts

Fact D.1. [adapted from Theorem 4 in [MS08]] For every pair i, j ∈ [ n ] , let q ij denote an estimate of the probabilities of the Scheffé set of H i and H j according to P . Let ˆ H be the hypothesis that minimizes of the following:

For every ϵ and δ , if we estimate q ij using O (log( n/δ ) /ϵ 2 ) samples from P , then ˆ H satisfies the following guarantee:

Lemma 2.1. Upon receiving three parameters: Γ , ϵ , and δ , Algorithm 1 uses O (log(1 /δ ) /ϵ 2 ) samples and satisfies the following guarantees with probability at least 1 -δ :

If H 1 is Γ -close to P , then the algorithm returns H 1 .

If H 1 is (max(Γ , ∥ H 2 -P ∥ TV ) + 2 ∥ H 2 -P ∥ TV + ϵ ) -far from P , then the algorithm returns H 2 .

Proof. Let q , p 1 , and p 2 denote the true probability of the Scheffé set of H 1 and H 2 according to P , H 1 , and H 2 :

And, let ˆ q be the estimate of q from the samples. the Using the Hoeffding bound, one can show that | q -ˆ q | is at most ϵ ′ := ϵ/ 2 with probability 1 -δ , since we use O (log(1 /δ ) /ϵ 2 ) samples. Now, if ∥ H 1 -P ∥ TV is at most Γ , then | p 1 -q | must be at most Γ implying:

Therefore, the algorithm outputs H 1 as the winner.

Now, we show the second guarantee. If the total variation distance between H 1 and P , ∥ H 1 -P ∥ TV, is greater than (max(Γ , ∥ H 2 -X ∥ TV ) + 2 ∥ H 2 -X ∥ TV + ϵ ) , then by the triangle inequality and the definition of total variation distance, we will have:

Thus, the algorithm outputs H 2 as the winner.

Lemma B.2. For a prime q , assume we have a class of q hypotheses that contains t e ≥ 1 excellent hypotheses and t d decent hypotheses. Let a = 0 and b be two random numbers in Z q selected uniformly at random. Let H ′ be the set of following hypotheses: { H f a,b ( x ) : x ∈ [ n ′ ] } for which f a,b ( x ) := a · x + b ( mod q ) . If ( t e + t d ) /q ≤ 1 / (2 n ′ ) , then the probability that H ′ contains exactly one excellent hypothesis and no decent hypothesis is at least n ′ · t e / (2 q ) .

̸

Proof. Using Proposition 3.24 in [Vad12] the mapping is in fact a class of pairwise independent hash functions. That is, the following holds:

Fix two indices i ∈ Z p and x ∈ [ n ′ ] . The probability of mapping x to i is:

̸

Fix four indices i, j ∈ Z p and x, y ∈ [ n ′ ] where x = y . The probability of mapping x to i and y to j is:

Suppose H e and H d indicate the set of excellent and decent hypotheses respectively. Fix an index i for which H i is an excellent hypothesis. Suppose one of the indices in [ n ′ ] , say x , is mapped to i . Now, the probability of another hypothesis in H e ∪ H d being chosen for a fixed index y = x in [ n ′ ] is:

̸

Hence, the expected number of hypotheses (other than H i ) that are selected is:

The last inequality is due to the assumption we have: ( t e + t d ) /q ≤ 1 / (2 n ′ ) . Next, by Moarkov's inequality, the probability that the number of y 's are at least one (that is two times the expected value) is at most 1 / 2 . Hence, if x is mapped to H i , with probability at least 1 / 2 , there is no other excellent or decent hypothesis is selected.

Now, we compute the probability that an excellent hypothesis is selected and no other excellent or decent hypothesis is selected as follows:

Pr [ One excellent hypothesis and no decent one ]

Hence, the proof is complete.

Fact D.2. Let n, a, b be three positive integers. Then, we have:

Proof. Let r denote the remainder of n divided by a . If r = 0 , then the claim is trivial. Therefore, assume r is in { 1 , 2 , . . . , a -1 } . Let s be the remainder of n divided by ab . Clearly, s has to be of the form: t · a + r for t ∈ { 0 , 1 , . . . , b -1 } . Hence, we have

Fact D.3. Given the definition of the g ℓ 's in Equation (2) , for any positive integer ℓ , we have:

Proof. By the properties of the geometric series, we obtain:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies hypothesis selection in a streaming setting, with samples arriving online. The main result gives a near-optimal memory-sample trade-off for hypothesis selection.  Along the way, the paper invents several new techniques to avoid expensive memory usage of prior work.\n\nSTRENGTHS:\nThis is one of the most classic questions in nonparametric statistics. Prior work [MS08, DK14] have tackled the run-time problem. I believe that its memory efficiency is also an important aspect. This paper offers exactly such a study.\n\nAt a technical level, the paper nearly resolves the question, up to some minor logarithmic factors. The high-level approach is simple and may be practical. The techniques, as far as I know, are novel and could potentially be used by later works. I have not checked the proof details in the appendix, but based on the main body, the main arguments are sound.\n\nThe paper is generally well-written.\n\nWEAKNESSES:\nI do not find any major weakness of this paper. \n\nOne thing I should point out is that the imperfect comparison issue (discussed in section 2.2.1) has been studied in the literature of noisy sorting. In fact, I believe the main result from the random ladder tournament, *at least* in the offline setting (section 3.1), can be derived simply in a blackbox way from Theorem 3.8 of Sorting and Selection with Imprecise Comparisons by Miklós Ajtai, Vitaly Feldman, Avinatan Hassidim and Jelani Nelson (TALG 2015) (https://dl.acm.org/doi/10.1145/2701427). In particular, their $\\delta$ is your $3\\Gamma$, the error gap between good and bad hypotheses in the comparison procedure (Algorithm 1). Their number of items there is your number of hypotheses, both denoted by $n$. So up to another factor of $k=3$, applying their randomized max-finding algorithm would yield a hypothesis that’s $9\\Gamma$ close to the optimal.  (Another related work is https://arxiv.org/abs/1606.02786 where an improved algorithm is given and an application to hypothesis selection is explicitly derived. I have not checked the details, though.)\n\nI would ask the author(s) to confirm this and see if an alternative argument using the paper above can be made and whether it’s interesting. \n\nI should note that this, if true, does not trivialize the results in this paper. The algorithm of Ajtai et al cannot be implemented in the memory-constrained setting, below a memory size of $n^{1/3}$. Also in the offline setting, the resulting constant is not optimal for the hypothesis selection problem. \n\n\n \n\nMinor comments\n---\nLine 62–67: list these two query access assumptions as \\begin{itemize}\n\nLine 65: “ the probabilities of the Scheffé sets”  — I think it’s more clear to refer to this as “the probability masses of the Scheffe sets”.\n\nLine 117: “we sample a uniformly random new hypothesis from H” — Please clarify: is this sampling with replacement, if I view the meta-distribution as uniform over the n input hypotheses? That is, a hypothesis can be selected multiple times against the same single hypothesis in memory. \n\nLine 202: Regarding [MS08], mention that the simple linear-time selection from Appendix A doesn’t do such expensive preprocessing.\n\nLine 265: Can you clarify in which scenario or parameter regime would you use Scheffé counts? (I didn’t read the appendix.) Note that the memory usage of this technique is quadratic in $n$. Hence, are you tracking these counts on a small subset of hypotheses?\n\nQUESTIONS:\nTo avoid the assumption of having access to the probability of Scheffe sets, have you considered approaches that don’t require Scheffe sets at all? I understand that the minimum distance estimate [DL01, MS08] is used in the all-go-against-all tournament step. Can ideas like that be used throughout, so we don’t need query Scheffe sets?\n\nLIMITATIONS:\nThe author(s) have discussed some future directions, though I think a conclusion section summarizing these would be great.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of hypothesis selection under a memory constraint. Here one is given $n$ distributions $H_1, \\dots, H_n$ with access to an oracle that can output 1) $H_i(H_j > H_k)$ for any $i,j,k$ and 2) $1(H_i(x) > H_j(x))$ for any $i,j$ and any point $x$ in the underlying space. Given a stream of observations $x_1,x_2...$ from some unknown $P$ the task is to select an estimator $\\hat{H} \\in (H_i)_{i=1}^n$ that is as close as possible to $P$ in TV distance up to a multiplicative and additive constant $\\alpha,\\epsilon$ respectively. Moreover, one wishes to do so using at most $b$ bits of memory at any point during the algorithm. \n\nThe author's main result Theorem 1 shows that $s$ samples from $P$ are sufficient to output a suitable estimator with constant error probability, provided $bs \\gtrsim \\tilde{\\mathcal \\Omega}(n\\log(n)/\\epsilon^2)$ and note that $bs \\gtrsim n$ is necessary by previous work. They achieve this result by introducing multiple technical ideas, the main one being the 'random ladder tournament'.\n\nSTRENGTHS:\n- The paper is mostly clearly written \n- The paper appears technically sound\n- The paper introduces simple but novel technical ideas that not only near-optimally tackle their proposed problem, but also recover other known results.\n\nWEAKNESSES:\n- My main concern is that due to the technical nature of the paper the short format of neurips is simply not enough for a meaningful presentation of the results. The motivating problem is under a memory constraint, yet just as one would learn how one can trade off memory for samples in the random ladder tournament (Lemma 3.1) the paper ends. More generally, from the main text, I don't feel like I have learnt how the algorithm actually works. In sections 1.2, 2 and 3 we are given glimpses of the key technical ideas necessary but it doesn't feel cohesive. \n\n\nTypos:\nPage 4 \"Results in a better...\"\n\nQUESTIONS:\nI would suggest to shorten (or remove) the proof of theorem 4 and to replace it with a (sketch) proof of the memory-sample trade-off.\n\nLIMITATIONS:\nThe authors address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The authors study the problem of hypothesis testing for pdfs in a streaming model. The problem is to find a hypothesis H* (corresponding to a pdf) from a family {H1,...,Hn} that is closest to an unknown pdf P. The input is a stream of points drawn i.i.d. from P. At any time step the algorithm can ask for a new point or query a scheffe set of two distributions Hi and Hj in {H1,..,Hn}. 

The main theorem in the paper shows that the algorithm can "learn" P properly by using O(n\log n/b\cdot\frac{\log(1/\eps)}{\eps^2}) queries, where b is the memory used. This is close to optimal - within log n factor. 

Note: 
- proper learning a pdf means: |P-H*|_TV\le\alpha\cdot OPT({H1,...,Hn}, P)+\eps.
- scheffe set of two distributions H1, H2 is {x : H1(x)<H2(x)}

STRENGTHS:
- Clean presentation and summary at the beginning of the paper.
- Almost tight bounds (upto log factors)

WEAKNESSES:
- Perhaps the model can be motivated better? Why the restriction to scheffe sets?
- Worth comparing the algorithm to vast statistical literature on Neyman-Pearson tests using log-likelihood computation.
- Need to compare and cite some of the vast literature on orienteering tournaments in directed graphs - I suspect some of the underlying results may already be known.

QUESTIONS:
Why not design an online Neyman-Pearson test - compute approximate log likelihoods based on the query results? The log-likelihood can then be used to upper bound the TV distance.

LIMITATIONS:
NA. The paper is theoretical in nature and is fairly upfront about it.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of agnostic distribution learning where i.i.d. samples are generated from an unknown distribution $X$. The goal is to find the best distribution from a given set of finite distributions $\\\\{H_1,\\cdots,H_n\\\\}$ that is closest to $X$ under total variation distance. The authors specifically study the sample complexity, under which the memory that is needed to store the information of the samples is bounded. The main contribution is a trade-off between the number of bits of the memory and samples needed, which is tight up to a $\\log n$ factor in the minimax sense.\n\nThe main proof technique is based on the so called \"random ladder tournament\". This differs from conventional approach that first estimating the probability of Scheffé sets of all pairs in $\\\\{H_1,\\cdots,H_n\\\\}$ and then selecting a distribution closest to the estimates. In contrast, the authors propose an ingenious streaming algorithm that tracks the \"best\" distribution in an online fashion, which is suited for achieving their desired trade-off between memory usage and sample complexity.\n\nSTRENGTHS:\nThe paper's primary strength lies in its novel \"random ladder tournament\" algorithm. This algorithm effectively achieves a memory-efficient sample complexity for agnostic distribution learning. The approach incorporates some interesting technical ideas that may be of independent  interest. As far as I'm aware, this particular scenario has not being studied in the literature.\n\nWEAKNESSES:\nMy main concern regarding this paper is its overall significance to the machine learning community. The paper has a strong TCS flavor, and the introduced model appears somewhat contrived. Specifically, the derived trade-offs, though mathematically intriguing, do not offer any surprising insights. Thus, its practical implications and utility within a broader machine learning context may be limited.\n\nAdditionally, the paper's presentation could certainly benefit from some improvements. The authors dedicate a significant portion of the introduction to outlining their techniques, which, unfortunately, are quite challenging to grasp without delving into the substantive technical details found mainly in the appendix.\n\nI would recommend eliminating most of this preliminary discussion and focusing primarily on the \"random ladder tournament\" argument, as outlined in Section 3. Providing a more detailed and accessible explanation of this central argument could make the paper more digestible for readers. Following this, a more straightforward discussion of the techniques used to improve the logarithmic terms would be better placed and easier to understand. This reordering should contribute to a more coherent and engaging narrative throughout the paper.\n\nQUESTIONS:\nSee comments above.\n\nLIMITATIONS:\nNo issue with negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}