Title: Optimal Hypothesis Selection in (Almost) Linear Time
Venue: NeurIPS 2024
Accepted: yes
Keywords: hypothesis selection

distribution learning

density estimation

time efficient algorithms

computational constraints
Abstract: Hypothesis selection, also known as density estimation, is a fundamental problem in statistics and learning theory. Suppose we are given a sample set from an unknown distribution $P$ and a finite class of candidate distributions (called hypotheses) $\mathcal{H} \coloneqq \{H_1, H_2, \ldots, H_n\}$. The aim is to design an algorithm that selects a distribution $\hat H$ in  $\mathcal{H}$ that best fits the data. The algorithm's accuracy is measured based on the distance between $\hat{H}$ and $P$ compared to the distance of the closest distribution in $\mathcal{H}$ to $P$ (denoted by $OPT$). Concretely, we aim for $\|\hat{H} - P\|_{TV}$ to be at most $ \alpha \cdot OPT + \epsilon$ for some small $\epsilon$ and $\alpha$. 
While it is possible to decrease the value of $\epsilon$ as the number of samples increases, $\alpha$ is an inherent characteristic of the algorithm. In fact, one cannot hope to achieve $\alpha < 3$ even when there are only two candidate hypotheses, unless the number of samples is proportional to the domain size of $P$ [Bousquet, Kane, Moran '19]. Finding the best $\alpha$ has been one of the main focuses of studies of the problem since early work of [Devroye, Lugosi '01]. Prior to our work, no algorithm was known that achieves $\alpha = 3$ in near-linear time. We provide the first algorithm that operates in almost linear time ($\tilde{O}(n/\epsilon^3)$ time) and achieves $\alpha = 3$.  This result improves upon a long list of results in hypothesis selection. Previously known algorithms either had worse time complexity, a larger factor  $\alpha$, or extra assumptions about the problem setting.
In addition to this algorithm, we provide another (almost) linear-time algorithm with better dependency on the additive accuracy parameter $\epsilon$, albeit with a slightly worse accuracy parameter, $\alpha = 4$.

Optimal Hypothesis Selection in (Almost) Linear Time

Abstract

Hypothesis selection, also known as density estimation, is a fundamental problem in statistics and learning theory. Given a sample set from an unknown distribution P and a finite class of candidate distributions (hypotheses) H := H 1 , H 2 , . . . , H n , the goal is to design an algorithm that selects a distribution ˆ H from H that best describes P . The accuracy of the algorithm is measured by the distance between ˆ H and P , compared to the distance between the closest distribution in H and P (denoted by OPT ). Specifically, we aim for ∥ ˆ H -P ∥ TV to be at most α · OPT + ϵ for some small ϵ and α .

While the value of ϵ can be reduced with an increasing number of samples, α is an inherent characteristic of the algorithm. Achieving α < 3 is impossible, even with only two candidate hypotheses, unless the number of samples is proportional to the domain size of P [Bousquet, Kane, Moran '19]. Finding a computationally efficient algorithm that achieves the optimal α has been a primary focus of research since the early work of [Devroye, Lugosi '01]. Before our work, the algorithms achieving α < 5 required time Ω( n 2 ) . We present the first algorithm that operates in almost linear time ( ˜ O ( n/ϵ 3 ) ) and achieves α = 3 . This result improves upon a long line of hypothesis selection research. Previously known algorithms had either worse time complexity, a larger α factor, or additional assumptions about the problem setting. Additionally, we provide another (almost) linear-time algorithm with better dependency on the additive accuracy parameter ϵ , albeit with a slightly worse accuracy parameter of α = 4 .

1 Introduction

Hypothesis selection, also known as density estimation, is a fundamental problem in statistics and learning theory. This problem involves identifying a density function that accurately represents the distribution of a given dataset. Suppose we are given a dataset of samples drawn from an unknown distribution P and a finite class of known distributions, representing different hypotheses: H := { H 1 , H 2 , . . . , H n } . The goal is to select a distribution in H that is close to P in total variation (TV) distance.

Typically, learning a distribution over a domain X with an ϵ error in TV distance requires Ω( |X| /ϵ 2 ) samples, which presents a substantial lower bound in sample complexity for distributions over large

domains. Surprisingly, the findings of Yatracos, Devroye, and Lugosi revealed that for hypothesis selection, this sample complexity can be independent of the domain size and only logarithmic in the number of hypotheses [Yat85, DL96, DL97]. With just s := Θ(log n/ϵ 2 ) samples from P , it is possible to learn P within error α · OPT + ϵ , where OPT denotes the distance of the nearest distribution in H to P . Specifically, Devroye and Lugosi introduced two algorithms for this problem: the Scheffé tournament, which operates in O ( n 2 · s ) time 1 and achieves α = 9 ; and the minimum distance estimate, which runs in O ( n 3 · s ) time and achieves α = 3 [DL01, Chapter 6]. In these results, although ϵ can be decreased as the number of samples increases, α remains an inherent parameter of the algorithm.

Significant effort has been directed towards finding computationally efficient algorithms for this problem while maintaining sample efficiency. The trade-off between the accuracy parameter α and computational efficiency has been a focal point. Mahalanabis and Stefankovic in [MS08] enhanced the minimum distance estimate, improving the time complexity to O ( n 2 · s ) . They also introduced a nearly linear-time algorithm that achieves α = 3 , but requires exponential time in n for preprocessing the class H . Other nearly linear-time algorithms were developed achieving α = 9 in [AJOS14, AFJ + 18, AAC + 23] 2 and α = 5 in [ABS23]. Furthermore, linear-time algorithms have been presented under the assumption that the algorithm receives the value of OPT (or its upper bound) as input [DK14, ABS23]. However, achieving α < 3 is not possible unless the number of samples becomes poly( |X| ) , as indicated in [BKM19].

Despite the long-standing history of this problem, the following question remained open:

Is there an algorithm for hypothesis selection with the optimal number of samples s = O (log( n ) /ϵ 2 ) and optimal accuracy parameter α = 3 that runs in O ( n · s ) time?

We present the first almost linear-time algorithm that uses the optimal number of samples and achieves the optimal accuracy parameter α = 3 . Our algorithm runs in ˜ O ( n · s / ϵ ) time (see Theorem 2). Additionally, we introduce another algorithm with improved dependency on ϵ , running in ˜ O ( n · s ) time while obtaining a slightly higher accuracy parameter, α = 4 (see Theorem 3). Our results represent a significant step forward, as they are the first in decades to achieve time complexity linear in n for any α < 5 . See Table 1.

Applications of Hypothesis Selection The primary application of hypothesis selection is to identify the best distribution from a set of known models that represent potential underlying data distributions we can effectively manage. For example, this set might include Poisson, gamma, and binomial distributions with various parameters used to model the number of arrivals per time unit. This makes hypothesis selection applicable to tasks like interpretable distribution selection and strategy optimization, where the objective is to choose the most suitable model from available options.

Another key strength of hypothesis selection is its agnostic nature, allowing it to adapt even when the true distribution lies outside the considered class. This robustness makes it effective in noisy data settings, with applications in denoising and anomaly detection. From a theoretical standpoint, hypothesis selection is fundamental in learning structured distributions, particularly when combined with the cover method, as seen in learning mixtures of Gaussians [DK14, SOAJ14]. For additional references, see Section 1.3.

Importance of improving α by a constant factor In most learning algorithms, the error guarantee decreases polynomially as the number of samples increases, so constant factors may not be as crucial. However, this is not the case in hypothesis selection. The output hypothesis is guaranteed to be ( α · OPT + ϵ ) -close to P in total variation distance. While increasing the number of samples can reduce ϵ to negligible levels, it does not improve the term α · OPT . Hence, α is an inherent property of the algorithm and directly impacts the best achievable error guarantee. Therefore, even a constant improvement in α is significant.

One might argue that, alternatively, OPT could be reduced by carefully curating the class H . However, beyond the practical challenges of finding a better H , it is unclear whether this can be achieved

without significantly increasing the size of the hypothesis class H . For example, in the cover method, when aiming to learn a distribution within a class C , we set H to be a γ -net that serves as a cover for C , ensuring that OPT < γ . While using a finer γ -net can reduce OPT , it may also drastically increase the size of H , thereby increasing the algorithm's running time, since the size of the net can grow super-polynomially with respect to γ . For instance, in the case of mixtures of k Gaussians, the size of the net depends on γ as roughly O ( γ -3 · k ) (see [SOAJ14]). Thus, reducing γ by a factor of three could exponentially increase the size of H in terms of k , thereby increasing both the running time and sample complexity, ultimately resulting in an inefficient algorithm.

1.1 Problem Setup

Suppose we have an unknown distribution P over a domain X and a set of n known distributions H := { H 1 , H 2 , . . . , H n } . Let OPT denote the distance between P and the closest distribution to it in H 3 : OPT ( H , P ) := min H ∈H ∥ H -P ∥ TV . We use the standard access model for this problem as introduced in [DL01]. The algorithm accesses the distributions through the following types of queries:

The algorithm can request a sample from the unknown distribution P .

The algorithm can compare the PDF of two known distributions: For every domain element x ∈ X and every pair of indices i and j , it can ask if H j ( x ) < H i ( x ) . This is equivalent to asking if x is in the Scheffé set of H i and H j (defined in Equation (2)).

The algorithm can query the probability mass of the Scheffé sets according to all the known distributions.

Remark 1. The last requirement of our model can be relaxed. Only estimates of the probability masses of the Scheffé sets are needed for our algorithms. Thus, one can alternatively assume sample access to H i 's, and estimate these values via samples.

Definition 1.1 (Proper learner for hypothesis selection) . Suppose algorithm A is given parameters ϵ, δ ∈ (0 , 1) , α ∈ R > 0 and has access to an unknown distribution P and a class of n known distributions H (as described above). We say A is an ( α, ϵ, δ ) -proper learner for the hypothesis selection problem if for every P and H , A outputs ˆ H ∈ H for which, with probability at least 1 -δ ,

We refer to α as the accuracy parameter, ϵ as the error (or proximity) parameter, and δ as the confidence parameter of the algorithm.

1.2 Main theorems

Below, we provide informal versions of our theorems.

Theorem 2. For every ϵ, δ ∈ (0 , 1) , Algorithm 1 is an ( α = 3 , ϵ, δ ) -proper learner for hypothesis selection that uses s = O (log( n/δ ) /ϵ 2 ) samples and time ˜ O ( n · s / ( δ 3 ϵ )) = ˜ O ( n/ ( δ 3 ϵ 3 )) .

Theorem 3. For every ϵ, δ ∈ (0 , 1) , Algorithm 4 is an ( α = 4 , ϵ, δ ) -proper learner for hypothesis selection that uses s = O (log( n/δ ) /ϵ 2 ) samples and time ˜ O ( n · s · log(1 /δ )) = ˜ O ( n · log 2 (1 /δ ) /ϵ 2 )) .

For formal statements, see Theorem 5 (Appendix A) and Corollary B.1 (Appendix B).

Our results are the first to achieve time complexity linear in n for any α < 5 . To achieve this, we introduce novel algorithmic techniques that will hopefully be broadly useful-see Section 3 for an overview. Both algorithms use the optimal number of samples. The first algorithm obtains optimal accuracy parameter α = 3 with a time complexity of ˜ O ( n/ ( δ 3 ϵ ) · s ) , which is off by an O (1 / ( δ 3 ϵ )) factor. Our second algorithm achieves the optimal time complexity up to logarithmic factors while achieving a slightly higher α = 4 . Our results leave a fascinating open question: Can one combine the best aspects of both algorithms, maintaining α = 3 , achieving O ( n · s ) time (or even lower), and sample complexity s = O (log( n/δ ) /ϵ 2 ) ?

Remark 4. Readers may be surprised by the polynomial dependence on 1 /δ in Theorem 2. In many settings, the success probability of a learning algorithm can be amplified from a constant, say 2/3, to at least 1 -δ at a cost of at most log(1 /δ ) in running time and sample complexity. However, in hypothesis selection, there is no (known) general technique for boosting the confidence parameter while keeping α the same. The issue is that choosing the best output from several runs of a given algorithm requires executing a second hypothesis selection algorithm, which introduces another factor of α in the approximation-leading to a total factor of at least 9 . As a result, these kinds of two-phase algorithms are not sufficient in the low α regime. Some previous results, such as [ABS23], also suffer from a polynomial dependency on δ .

1.3 Other related work

Hypothesis selection has been studied in various settings including improper setting. In [BKM19], the authors consider the improper version of the problem where the output hypothesis ˆ H may not necessarily be in H . They presented an improper learner with accuracy guarantee α = 2 . The sample complexity of their algorithm was improved by [BBK + 21], who gave an algorithm with nearly optimal sample complexity and the same accuracy parameter α = 2 . It is worth noting that our algorithms are proper learners and solve this problem with a slightly better sample complexity. In addition, like other proper learners, our algorithms select their output from a predefined set, which can facilitate choosing a distribution with specific structural property (e.g., mixture of Gaussians). In certain applications, this selection ensures consistency with the problem's underlying assumptions, which enhances interpretability and robustness.

The problem of hypothesis selection in sublinear time was studied for distributions on discrete domains [AAC + 23]. Among other results, the authors developed a data structure that upon receiving samples from a unknown distribution P returns a hypothesis ˆ H in o ( n ) time. While their algorithm runs in sublinear time, their sample complexity depends on the domain size of the distribution, and their setting allows pre-processing of the class H in polynomial time. Another interesting variation for hypothesis selection is designing differentially private learners for the problem which has received attention over the past few years [BKSW19, CKM + 19, GKK + 20].

An important application of hypothesis selection arises when there is a structural assumption on the underlying distributions. One approach for learning these classes is to selectively choose a cover for the class. One can then use the learners for the standard hypothesis selection problem (which we study in the paper) and use the cover as the class H . Examples of such structural assumptions include Poisson binomial distributions [DDS12], mixtures of Gaussians [KMV12, DK14, SOAJ14, DKS17, KSS18, ABM18, ABH + 20], distributions with piecewise polynomial PDFs [ADLS17], and histograms [Pea95, CDSS14, CDKL22]. See Diakonikolas [Dia16] for a survey of results.

2 Preliminaries

Notation: We use the following notation throughout this article. We use [ n ] to indicate the set { 1 , 2 , . . . , n } . For a distribution Q over X , Q ( x ) denotes the PDF of Q at the domain element x ∈ X . For any measurable subset of the domain S ⊆ X , Q ( S ) indicates the probability mass of the set S according to Q . We use ∥ Q 1 -Q 2 ∥ TV := sup S ⊆X | Q 1 ( S ) -Q 2 ( S ) | to denote the total variation distance between Q 1 and Q 2 . We say Q 1 is ϵ -close to Q 2 if ∥ Q 1 -Q 2 ∥ TV is at most ϵ . Conversely, we say Q 1 is ϵ -far from Q 2 if ∥ Q 1 -Q 2 ∥ TV is greater than ϵ . We use the standard O, Ω , Θ notation for asymptotic functions. Additionally, we use ˜ O , ˜ Ω , and ˜ Θ to hide polylog factors.

Scheffé sets: For every pair of hypotheses H i and H j in H , we define the Scheffé set of H i and H j as follows:

It is known that the Scheffé set maximizes the probability discrepancy between H i and H j , thus fully characterizing the total variation distance between the two distributions:

The optimal hypothesis: Recall that we assume the algorithm is given samples drawn from an unknown distribution P . Let H i ∗ denote the closest hypothesis in H to P . That is, H i ∗ is the hypothesis for which ∥ H i ∗ -P ∥ TV = OPT . When there is more than one hypothesis with this property, we pick one arbitrarily as H i ∗ .

Semi-distances: For every pair i, j in [ n ] , we define w j ( H i ) to be | H i ( S i,j ) -P ( S i,j ) | . In words, w j ( H i ) is the distance of H i to P observed on the Scheffé set of H i and H j . For every pair i, j ∈ [ n ] , we use ˆ w j ( H i ) to denote an estimate of w j ( H i ) based on the observed sample. For the sake of consistency, we define w i ( H i ) to be zero. In addition, we define the score function W ( H i ) := max j ∈ [ n ] w j ( H i ) . Similarly, ˆ W ( H i ) is defined to be max j ∈ [ n ] ˆ w j ( H i ) .

Refined access model: Similar to previous work [DL01, MS08], we use estimates of the semidistances. One can easily estimate these quantities, denoted by ˆ w j ( H i ) , via the access model we have described earlier by letting ˆ w j ( H i ) be the empirical ratio of the samples that are in S i,j . Throughout this paper, we assume that there are two universal parameters δ ′ = Θ( δ ) and ϵ ′ = Θ( ϵ ) , for which with probability 1 -δ ′ every ˆ w j ( H i ) is within ϵ ′ of w j ( H i ) :

A simple application of the Hoeffding bound and the union bound shows that one can compute all of the estimates via s = O (log( n/δ ′ ) /ϵ ′ 2 ) samples, and each estimate can be computed in O ( s ) time.

Our algorithms access the distributions in H∪{ P } only via querying ˆ w j ( H i ) . This fact brings the sample complexity of our algorithms to s = O (log( n/δ ′ ) /ϵ ′ 2 ) samples. The time complexity of our algorithms is determined by the number of queries they make to the ˆ w j ( H i ) 's, multiplied by time that we spend on each query, O ( s ) . Moreover, in the proofs of our theorems, we assume without loss of generality that all the ˆ w j ( H i ) 's are accurate. Conditioning on the accuracy will not decrease the probability of correctness of any of our algorithms by more than δ ′ due to Fact C.1.

3 Overview of our techniques

In this section, we provide a high-level discussion of our algorithm. The important notations and definitions used here are provided in Section 2. For the high-level discussions in this section, we assume we have access to the exact values of the semi-distances w j ( H i ) . In the formal proofs presented in subsequent sections, we will substitute each w j ( H i ) with an estimated value ˆ w j ( H i ) .

If the error of these estimates is below ϵ ′ = Θ( ϵ ) , it can be shown that the overall impact of this substitution on our final distance guarantee (Equation 1) is at most Θ( ϵ ) . See the 'Refined access model' in Section 2 for further details.

3.1 Background: Semi-distances and the minimum distance estimate

To solve the hypothesis selection problem, we seek a certificate that ensures we output a hypothesis ˆ H such that ∥ ˆ H -P ∥ TV is at most 3 OPT . Similar to previous work, our algorithms operate based on the probability masses of the Scheffé sets (Equation (2)) of pairs of hypotheses in H . The semi-distance w j ( H i ) , defined as | H i ( S i,j ) -P ( S i,j ) | , captures the 'distance' between H i and P as measured on this particular set S i,j . One suggestion for readers to internalize the semi-distances is to view them as a distance between H i and P that is measured from the perspective of H j . By definition, w j ( H i ) is always a lower bound for ∥ H i -P ∥ TV:

However, it is possible for w j ( H i ) to be much lower, making it difficult for an algorithm to estimate the TV distance between H i and P just using semi-distances.

Nevertheless, for each hypothesis H i , a specific semi-distance w i ∗ ( H i ) , associated with the optimal hypothesis H i ∗ , determines its quality. As shown in the following proof, if w i ∗ ( H i ) ≤ OPT , then H i is 3 , OPT -close to P , with the total variation distance between H i and P bounded by w i ∗ ( H i ) and OPT via the triangle inequality:

This observation implies that if we assert that w i ∗ ( H i ) is bounded by OPT , we can output H i as our final solution to the problem and be done. The challenge is that we neither know i ∗ nor OPT .

This issue is addressed by the minimum distance estimate presented in [DL01, MS08] using a score function W ( H i ) , as defined earlier: W ( H i ) := max j ∈ [ n ] w j ( H i ) . The minimum distance estimate outputs a hypothesis ˆ H that minimizes W ( H i ) . We can assert that, for the output of this algorithm, ˆ H , w i ∗ ( ˆ H ) is at most OPT . This approach simultaneously bypasses the issues of not knowing i ∗ and OPT .

Note that W ( H i ) serves as a proxy for the quality of H i since W ( H i ) is an upper bound for w i ∗ ( H i ) . Using W ( H i ) , we can address the issue of not knowing i ∗ . On the other hand, although OPT is not known, we have a good lower bound for OPT , which is W ( H i ∗ ) . Putting these observations together, we obtain:

These inequalities are derived using the triangle inequality and the fact that ˆ H was chosen to be the arg min i ∈ [ n ] W ( H i ) . See Figure 1 for an illustration of the above inequality.

The primary hurdle with the minimum distance estimate is that it is costly to compute. Computing each W ( H i ) takes O ( n · s ) time, bringing the total time complexity of the algorithm to O ( n 2 · s ) . One might naturally conjecture that sampling may help to compute an estimate of W ( H i ) . Instead of using W ( H i ) := max j ∈ [ n ] w j ( H i ) , we can use ˜ W ( H i ) := max j ∈ R w j ( H i ) where R is a set of random indices in [ n ] . The issue with this approach is that there is no guarantee of i ∗ being selected in R , making ˜ W ( H i ) too low while H i may be far from P . Hence sampling, if used in a trivial manner, is not beneficial.

3.2 The algorithm with α = 3

In this section, we present an overview of Algorithm 1 that attains α = 3 . The details of this algorithm and its related theorems are provided in Section A. At a high level, similar to the minimum distance estimate, we work towards finding a hypothesis that minimizes W ( H i ) . To increase efficiency, we

P

(

S

i,j

)

H

i

∗

(

S

i,j

)

H

i

(

S

i,j

)

OPT

w

i

∗

(

H

i

)

W

(

H

i

)

w

i

(

H

i

∗

)

∥

H

i

-

H

i

∗

∥

TV

H

i

OPT

H

i

∗

P

∥

H

i

-

P

∥

TV

TV distance

=

absolute difference of probabilities on Scheffé set

Probability of

S

i,j

according

to different distributions:

work with estimates of W ( H i ) 's, denoted by ˜ W ( H i ) . The general structure of our algorithm is as follows: Initially, all ˜ W ( H i ) are set to zero. At every step, we come up with several pairs of hypotheses H i and H j and update our estimates by setting ˜ W ( H j ) to max( ˜ W ( H j ) , w i ( H j )) . Our approach ensures that at every step, ˜ W ( H i ) is equal to max j ∈ R w j ( H i ) for a small, carefully chosen set R . Eventually, we output a hypothesis with roughly the smallest ˜ W ( H i ) .

Focusing on small ˜ W ˜ W ˜ W : Our first idea is to focus on updating the ˜ W for hypotheses whose ˜ W ( H i ) values are around the current minimum ˜ W . The rationale for this action comes from a simple fact: ˜ W ( H j ) always underestimates the value of W ( H j ) . Hence, if we observe that ˜ W ( H i ) is substantially larger than the current minimum, then W ( H i ) is also substantially larger than the current minimum. This implies that H i is not a suitable candidate for the minimum at this stage of the algorithm, and it can be ignored for now.

̸

Bucketing hypotheses based on ˜ W ˜ W ˜ W : To implement this idea, we partition the hypotheses into k = Θ(1 /ϵ ′ ) buckets B := { B 1 , B 2 , . . . , B k } based on ˜ W ( H i ) . The bucket ℓ contains all the hypotheses H i such that ˜ W ( H i ) ∈ [( ℓ -1) ϵ ′ , ℓ ϵ ′ ) . At every step, we focus on the smallest nonempty bucket B ℓ (the smallest ℓ for which | B ℓ | = 0 ). B ℓ contains all the hypotheses whose ˜ W ( H j ) is around the minimum ˜ W . We pick pairs of hypotheses, H i ∈ H and H j ∈ B ℓ , and update ˜ W ( H j ) . Note that our updates may increase ˜ W ( H j ) , and we may remove H j from B ℓ and put it into a larger bucket (a bucket with a larger index ℓ ). Also, observe that we never move a hypothesis into a smaller bucket since ˜ W ( H j ) never decreases. We continue our updates to reach one of the following outcomes:

B ℓ becomes empty. That is, our updates made all ˜ W ( H i ) fall out of the range of these buckets [0 , ℓ ϵ ′ ) . Thus, every ˜ W ( H i ) (and consequently every W ( H i ) ) is at least ℓ · ϵ ′ . Every time that we empty out a bucket, we have increased our threshold for minimum W ( H i ) by ϵ ′ . Hence, we are getting closer to a bucket with the true minimum, which we hope to reach in O (1 /ϵ ′ ) steps.

B ℓ is not empty, but we can confidently confirm most of the hypotheses in B ℓ are an acceptable output for the algorithm. Although we cannot ensure that H i ∗ is indeed in B ℓ , we can find an acceptable final answer by selecting a random hypothesis in B ℓ .

Which pairs to update? Next, we outline our update scheme to implement the above ideas in linear time. To enhance time efficiency, our aim is to optimize the updating process to ensure both quality and quantity in the chosen updates. Quality, in this context, relates to the extent of change in ˜ W ( H j ) following an update: We consider H i to have made a substantial update to H j if ˜ W ( H j ) + ϵ ′ < ˆ w j ( H i ) . Such updates cause a significant shift, increasing the value of ˜ W ( H j ) by

more than ϵ ′ and subsequently moving H j to a different bucket with a higher index ℓ . We refer to this event as H i removing H j from its bucket. The quantity, on the other hand, relates to the number of H j instances that a given H i can remove from B ℓ . An ideal H i eliminates a substantial portion of hypotheses from B ℓ (say a constant fraction). We call such an H i a prompting hypothesis . Now, if for O (log | B ℓ | ) rounds we find a prompting hypothesis and update all the ˜ W ( H j ) for H j ∈ B ℓ , we will empty out the bucket B ℓ , and we can move forward.

Finding a prompting hypothesis: To find a prompting hypothesis quickly, we iterate over all H i in H , sample a few H j , and check if H i substantially updates ˜ W ( H j ) . If H i substantially improves a large fraction of the sampled hypotheses, then we declare that H i is a prompting hypothesis. See Section A.1.1 for further details. In addition to that, we provide a more advanced version of this procedure in Section A.1.2 that allows us to shave off an O (log n ) factor.

Getting stuck? Here is your way out: What happens when B ℓ is not empty, and we cannot find a prompting hypothesis? We show that if we do not find a prompting hypothesis, then a random hypothesis in B ℓ is an acceptable answer.

The surprising part about this statement is that it holds regardless of the size of the bucket B ℓ due to hypothesis sampling procedure we have to find a prompting hypothesis. In search for a prompting hypothesis, we iterate over all H i and sample roughly O (log n/δ ) many hypotheses H j 's in B ℓ . We check, if H i can remove them from the bucket. Note that if H i was not found to be prompting, it implies that H i ∗ cannot substantially update most of the hypotheses in B ℓ . Thus, We have that with high probability for 1 -δ fraction of the hypotheses in B ℓ , w i ( H j ) ≤ ϵ ′ · ℓ .

The clever hack here is an observation about H i ∗ . Earlier, we discussed that we are looking for a hypothesis H i with w i ∗ ( H i ) ≤ OPT . We claim that a random hypothesis in the last B ℓ (almost) has this property. On one hand, given that H i ∗ was not found to be a prompting hypothesis, for 1 -δ fraction of H i in B ℓ , w i ∗ ( H i ) must be at most ℓ · ϵ ′ . On the other hand, the fact that all the previous buckets, B 1 , . . . , B ℓ -1 , are empty indicates H i ∗ has shown a semi-distance of at least ( ℓ -1) ϵ ′ . Hence, OPT , which is at least as large as all H i ∗ semi-distances, is at least ( ℓ -1) ϵ ′ . Therefore, w i ∗ ( H i ) ≤ ϵ ′ + OPT . Therefore, for 1 -δ fraction of the hypotheses in B ℓ , they are 3 OPT +Θ( ϵ ) close to P . For a formal argument, see Lemma A.1.

Now, assume the search for the prompting hypothesis fails. Recall that during the search, we have checked every single hypothesis in H . During the search, at some point, we must have stumbled upon H i ∗ and did not declare it as a prompting hypothesis. In this case, either our sampling for substantial updates failed (which happens with small probability), or we can infer that there are not too many far hypotheses in B ℓ . Either way, we can output a random hypothesis from B ℓ as the final sample without increasing the error probability by too much.

Dependency on δ : It is worth noting that this last step results in a polynomial dependency on δ (as opposed to a more desirable dependency of log(1 /δ ) ). This is mainly due to the fact that to ensure that a random hypothesis in B ℓ is not too far, with a probability of 1 -δ in a single round, we have to try O (log( n ) /δ ) hypotheses in B ℓ and check if H i ∗ is prompting them. Hence, relying on this structural property of H i ∗ makes a polynomial dependency on δ inevitable. Improving the dependency on δ for this algorithm would require new algorithmic ideas.

3.3 The algorithm with α = 4

We introduce another (almost) linear-time algorithm for the hypothesis selection problem, where the time complexity shows improved dependency on ϵ . However, this algorithm has a slightly worse accuracy parameter compared to our previous algorithm: α = 4 . This algorithm and the subsequent theorems are provided in Section B.

Algorithm with a guessed upper bound of OPT : As mentioned earlier, unlike previous work, our algorithm does not receive any prior information about the value of OPT . It might be speculated that there exists an easy reduction between our problem and another version of hypothesis selection, where an auxiliary parameter σ is provided to the algorithm such that OPT ≤ σ . A learner without the knowledge of σ can make a guess about σ and run one of the existing algorithms that works with the knowledge of an upper bound of OPT (e.g., [ABS23]) and check if it finds a good hypothesis or

not. With this procedure in mind, we can run a binary search to find the smallest σ for which we find a hypothesis. 4 However, a challenge with this approach is that the algorithm might yield a hypothesis even when σ < OPT , making it difficult for us to refute the hypothesis that is found. Even if the output ˆ H satisfies W ( ˆ H ) > σ , it does not necessarily invalidate our guessed value σ .

To overcome this challenge, we design an algorithm, namely A , with enhanced accuracy guarantees for the output hypothesis. We provide A with the value σ . We treat σ as a "guess" for which we hope that OPT ≤ σ . We require the algorithm to either refute our guess and declare OPT > σ , or output a hypothesis ˆ H with a reasonable distance to P , irrespective of the relationship between OPT and σ . More precisely, the distance of the output hypothesis should be bounded by: ∥ ˆ H -P ∥ TV ≤ α · max( OPT , σ ) + ϵ . With these revised guarantees, it is permissible to run a binary search over different values of σ in { ϵ, 2 ϵ, . . . , ⌈ 1 /ϵ ⌉ ϵ } , and output the hypothesis that A returns on the smallest σ . For the rest of this discussion, we focus on designing A and a provided parameter σ .

Seeds and finding an acceptable hypothesis via seeds: In this algorithm, we introduce a new concept called a seed for hypotheses. A seed provides us with a structural property that enables us to identify an acceptable hypothesis, regardless of the relationship between H i ∗ and the seed. More formally, for a hypothesis H i , we define M a,b ( H i ) as the set of hypotheses H j such that w j ( H i ) is between a and b :

We may omit the subscript of M when it is clear from the context. We say a hypothesis H i is a seed if almost all of its ˆ w j ( H i ) are low and its W ( H i ) is not too large. The quality of a seed is determined by three parameters: a , b , and m .

Definition 3.1 (Seed) . Given parameters a, b ∈ R ≥ 0 , and a non-negative integer m , we say a hypothesis is an ( a, b, m ) -seed if the following hold: 1) ˆ W ( H i ) ≤ b , and 2) |M a,b ( H i ) | ≤ m .

A seed with a small-sized set M and a small b exhibits an interesting property: Suppose we have a seed H i with a constant-sized set M . In this case, in O ( n ) time, we can compute W ( H j ) of every hypothesis H j in M . Now, let H ℓ ∈ M be one of the hypotheses that minimize W ( H j ) : H ℓ := arg min H j ∈M W ( H j ) . We claim either H i or H ℓ is acceptable output in this case.

When H i ∗ is in H\M , we can infer that w i ∗ ( H i ) is at most b (or, more precisely, b + ϵ ′ ). From our earlier discussion, if b is sufficiently small (compared to σ ), we can show that H i is close to P . In cases where H i ∗ is in M , although we do not have strong guarantees for H i itself, we have a very good answer already: H ℓ is essentially a minimum distance estimate for the set M which includes H i ∗ . Thus, we can conclude: First, H ℓ is an acceptable output because it is 3 OPT -close; second, W ( H ℓ ) is a lower bound for OPT .

Now, as we described, we have two acceptable outputs for two cases of the problem. For H i ∗ ∈ H\M , H i is an acceptable answer. For H i ∗ ∈ M , H ℓ is an acceptable answer. The only problem is that we do not know which case we are in. Our strategy is to pick a hypothesis that is still reasonably close to P even if we are in the other case. Let us fix a threshold value T . We compare W ( H ℓ ) with Tσ : 1) If W ( H ℓ ) ≤ T · σ , as we have discussed earlier, H ℓ is ( T · σ +2 OPT ) -close to P . This bound holds regardless of where H i ∗ is. 2) Next, if W ( H ℓ ) > T · σ , we cannot say H ℓ is a good choice for us. However, we can conclude that H i is a close hypothesis to P even when H i ∗ happens to be in M . In this case, we know that OPT ≥ W ( H ℓ ) > T · σ . We take advantage of this knowledge and obtain the following bound:

Thus, regardless of where H i ∗ is, H i is (max( a, b/T ) + 2) · max( σ, OPT ) . Now, the algorithm is fairly straightforward. If W ( H ℓ ) is small, output H ℓ ; otherwise, output H i . We summarize these four cases in the following table. Depending on a and b , we set T to minimize the maximum distance we

endure in these cases. It is worth noting that in this argument, we did not rely on any prior knowledge concerning the relationship between σ and OPT .

How to find the first seed? Here, we provide an overview of our approach for identifying an initial seed. For a detailed explanation of the algorithm and its performance, refer to Section B.1.

To start, fix two parameters, a = σ and b = 3 σ . To identify a seed hypothesis, we iterate over all hypotheses H 1 , . . . , H n , checking whether each hypothesis H i is a strong seed (i.e., a seed with a small |M| ) by sampling several H j 's and verifying if w j ( H i ) is at most a . Roughly speaking, for an integer m ∈ [ n ] , if we sample approximately ˜ O ( n/m ) H j 's and find that no w j ( H i ) > a , we can, with high probability, confirm that the size of M does not exceed m . This approach requires O ( n 2 /m ) time.

There are, however, a few caveats with this method. First, if a seed is identified using the above approach, it may have a large W ( H i ) > b . In this case, we can infer that H i is likely far from P . Given the time invested in identifying H i , we aim to leverage this information. Broadly, observing that w j ( H i ) ≤ a implies that many hypotheses are close to H i (assuming that w i ( H j ) values are also small). Knowing that H i is far from P , we can mark all hypotheses close to H i as also far, allowing us to proceed with the search. While this may seem counterproductive, marking a significant number of hypotheses as 'far' constitutes progress for our algorithm. Specifically, if, at some point, all hypotheses are marked, we can declare that σ < OPT .

The second caveat is more challenging. Ideally, we seek a seed with a constant m , but finding such a seed would require ˜ O ( n 2 /m ) = ˜ O ( n 2 ) time. Consequently, for a linear-time algorithm, we can only afford to find seeds where m = Θ( n ) . In other words, the quality of the seed we can initially identify is much lower than the quality required for a producing solution. This leads to our next idea: boosting a seed, which is an approach to incrementally improve the seed's quality in roughly O (log n ) steps.

Boosting a Seed: In this process, we use an initial seed to iteratively find a stronger seed with a reduced value of |M| . For a formal argument, see Section B.2. Assume a rate parameter η ∈ (0 , 1) . As discussed earlier, by spending O ( n/η ) time, we can identify a seed with m ≈ η · n . Initially, m = |M| might be O ( n ) . Hence, rather than calculating W ( H j ) exactly for all H j ∈ M , we compute an approximate maximum semi-distance ˜ W ( H j ) by sampling t hypotheses. Specifically, for each H j ∈ M , we set ˜ W ( H j ) := max H k ˆ w k ( H j ) , where H k 's are sample hypotheses. Using these approximate values ˜ W , we process the seed as follows. Let H ℓ denote the hypothesis minimizing ˜ W ( H j ) , with the following possible outcomes:

High ˜ W ( H ℓ ) : If ˜ W ( H ℓ ) is high, all ˜ W ( H j ) (and thus W ( H j ) ) values are likely large, making H i an acceptable final solution.

Low ˜ W ( H ℓ ) : While this does not guarantee a low W ( H ℓ ) , it suggests that most w i ( H ℓ ) values are small. Based on the value of W ( H ℓ ) , we consider two cases:

2.1 W ( H ℓ ) ≫ ˜ W ( H ℓ ) : H ℓ is likely far from P but has many close hypotheses, allowing us to mark these nearby hypotheses as far.

2.2 Moderate W ( H ℓ ) : In this case, H ℓ can serve as our new seed, as it yields a smaller set M ( H ℓ ) than H i .

Note that when we select H ℓ as our new seed, M ( H ℓ ) is roughly O ( n/t ) . However, we compute ˜ W ( H j ) 's for only |M ( H i ) | ≈ η · n many hypotheses. Hence, with an increased sample size t = O (1 /η 2 ) , this approach yields a smaller |M ( H ℓ ) | ≈ η 2 · n . The step of the process requires O ( |M| · t ) = m/η 2 = O ( n/η ) , paralleling the initial search. By repeating, we progressively reduce m = |M| , and for a constant η , m decreases by a fixed factor until reaching a constant m seed, as desired.

Acknowledgments

M.A. was supported by NSF awards CNS-2120667, CNS-2120603, CCF-1934846, and BU's Hariri Institute for Computing. This work was initiated while M.A. was affiliated with Boston University and Northeastern University and was done in part while M.A. was a research fellow at the Simons Institute for the Theory of Computing. M.B. was supported in part by NSF award CNS-2046425 and a Sloan Research Fellowship. A.S. was supported in part by NSF awards CCF-1763786 and CNS-2120667 as well as Faculty Awards from Google and Apple.

References

[AAC + 23] Anders Aamand, Alexandr Andoni, Justin Y. Chen, Piotr Indyk, Shyam Narayanan, and Sandeep Silwal. Data structures for density estimation. In Proceedings of the 40th International Conference on Machine Learning . PMLR, 2023.

[ABH + 20] Hassan Ashtiani, Shai Ben-David, Nicholas J. A. Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. J. ACM , 67(6):32:1-32:42, 2020.

[ABM18] Hassan Ashtiani, Shai Ben-David, and Abbas Mehrabian. Sample-efficient learning of mixtures. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , pages 2679-2686. AAAI Press, 2018.

[ABS23] Maryam Aliakbarpour, Mark Bun, and Adam Smith. Hypothesis selection with memory constraints. To appear in NeurIPS 2023 , 2023.

[ADLS17] Jayadev Acharya, Ilias Diakonikolas, Jerry Li, and Ludwig Schmidt. Sample-optimal density estimation in nearly-linear time. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 1278-1289. SIAM, 2017.

[AFJ + 18] Jayadev Acharya, Moein Falahatgar, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Maximum selection and sorting with adversarial comparators. The Journal of Machine Learning Research , 19:59:1-59:31, 2018.

[AJOS14] Jayadev Acharya, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Sorting with adversarial comparators and application to density estimation. In 2014 IEEE International Symposium on Information Theory , pages 1682-1686, 2014.

[BBK + 21] Olivier Bousquet, Mark Braverman, Gillat Kol, Klim Efremenko, and Shay Moran. Statistically near-optimal hypothesis selection. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS 2021, Denver, CO, USA, February 7-10, 2022 , pages 909-919. IEEE, 2021.

[BKM19] Olivier Bousquet, Daniel Kane, and Shay Moran. The optimal approximation factor in density estimation. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 318-341. PMLR, 25-28 Jun 2019.

[BKSW19] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'AlchéBuc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 156-167, 2019.

[CDKL22] Clément L. Canonne, Ilias Diakonikolas, Daniel Kane, and Sihan Liu. Nearly-tight bounds for testing histogram distributions. In Advances in Neural Information Processing Systems 35 (NeurIPS) , 2022. To appear.

[CDSS14] Siu-on Chan, Ilias Diakonikolas, Rocco A. Servedio, and Xiaorui Sun. Near-optimal density estimation in near-linear time using variable-width histograms. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada , pages 1844-1852, 2014.

[CKM + 19] Clément L. Canonne, Gautam Kamath, Audra McMillan, Adam D. Smith, and Jonathan R. Ullman. The structure of optimal private tests for simple hypotheses. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 310-321. ACM, 2019.

[DDS12] Constantinos Daskalakis, Ilias Diakonikolas, and Rocco A. Servedio. Learning poisson binomial distributions. In Howard J. Karloff and Toniann Pitassi, editors, Proceedings of the 44th Symposium on Theory of Computing Conference, STOC 2012, New York, NY, USA, May 19 - 22, 2012 , pages 709-728. ACM, 2012.

[Dia16] Ilias Diakonikolas. Learning structured distributions. In CRC Handbook of Big Data , pages 267-283. 2016.

[DK14] Constantinos Daskalakis and Gautam Kamath. Faster and sample near-optimal algorithms for proper learning mixtures of gaussians. In Conference on Learning Theory , pages 1183-1213. PMLR, 2014.

[DKS17] Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In Chris Umans, editor, 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, Berkeley, CA, USA, October 15-17, 2017 , pages 73-84. IEEE Computer Society, 2017.

[DL96] Luc Devroye and Gábor Lugosi. A universally acceptable smoothing factor for kernel density estimates. The Annals of Statistics , 24(6):2499 - 2512, 1996.

[DL97] Luc Devroye and Gábor Lugosi. Nonasymptotic universal smoothing factors, kernel complexity and Yatracos classes. The Annals of Statistics , 25(6):2626 - 2637, 1997.

[DL01] Luc Devroye and Gábor Lugosi. Combinatorial methods in density estimation . Springer, 2001.

[GKK + 20] Sivakanth Gopi, Gautam Kamath, Janardhan Kulkarni, Aleksandar Nikolov, Zhiwei Steven Wu, and Huanyu Zhang. Locally private hypothesis selection. In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 1785-1816. PMLR, 2020.

[KMV12] Adam Tauman Kalai, Ankur Moitra, and Gregory Valiant. Disentangling gaussians. Commun. ACM , 55(2):113-120, 2012.

[KSS18] Pravesh K. Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via sum of squares. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018 , pages 1035-1046. ACM, 2018.

[MS08] Satyaki Mahalanabis and Daniel Stefankovic. Density estimation in linear time. In 21st Annual Conference on Learning Theory - COLT , pages 503-512, 2008.

[Pea95] Karl Pearson. Mathematical contributions to the theory of evolution. ii. skew variation in homogeneous material. [abstract]. Proceedings of the Royal Society of London , 57:257-260, 1895.

[SOAJ14] Ananda Theertha Suresh, Alon Orlitsky, Jayadev Acharya, and Ashkan Jafarpour. Near-optimal-sample estimators for spherical gaussian mixtures. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27, 2014.

[Yat85] Yannis G. Yatracos. Rates of convergence of minimum distance estimators and kolmogorov's entropy. The Annals of Statistics , 13(2):768 - 774, 1985.

Algorithm 1 An algorithm for hypothesis selection with α = 3 .

̸

A Almost linear time algorithm with α = 3 α = 3

In this section, we focus on our first result, the algorithm with α = 3 . The pseudocode of our approach is presented in Algorithm 1. We prove the performance of this algorithm in Theorem 5. An overview of our approach is described in Section 3.2.

At a high level, our algorithm begins with all ˜ W ( H i ) set to zero, and all hypotheses placed in B 1 . We then iterate over all buckets. In round ℓ , we check if there is a prompting hypothesis H i ∈ H that can move a significant fraction of hypotheses from B ℓ . Specifically, we look for an H i that moves a γ -fraction of hypotheses from B ℓ by updating them, where γ = Θ( δ ) . If we find such a hypothesis H i , we update all the hypotheses in B ℓ and move as many as possible out of B ℓ . During this process, we count the number of hypotheses that H i removes from B ℓ , which is tracked by C ℓ i in the algorithm. If the actual fraction of hypotheses removed from B ℓ is less than γ/ 2 , we infer that IS-PROMPTING falsely declared H i as a prompting hypothesis (false positive). Thus, the algorithm returns ⊥ . Otherwise, we restart the search by setting i = 1 and iterate over all H i to find the next prompting hypothesis. We continue this process until no prompting hypothesis is found or the bucket is emptied. If we completely empty a bucket, we move on to the next one. If not, we select a random hypothesis in B ℓ as the final output and halt.

Theorem 5. Suppose H is a class of n known hypotheses and P is an unknown hypothesis. Assume the algorithm has access to accurate estimates of ˆ w j ( H i ) 's that have an error of at most ϵ ′ . Let ϵ and δ be two parameters in (0 , 1) . If 3 ϵ ′ ≤ ϵ , then Algorithm 1 is an ( α = 3 , ϵ, δ ) -proper learner for P in H . The running time of our algorithm is ˜ O (( n/ ( δ 3 ϵ )) · T q ) , where T q is the time to obtain each ˆ w j ( H i ) .

If the algorithm returns ˆ H , with high probability, ˆ H is not too far from P .

While the algorithm may not produce a hypothesis all the time, the overall failure probability of the algorithm is low.

The running time of the algorithm meets the desired bounds in the theorem.

The output ˆ H ˆ H ˆ H is close with high probability. Suppose we output ˆ H ∈ B ℓ at step ℓ . Our goal is to show that ˆ H is (3 OPT + ϵ ) -close to P with probability 1 -δ ′ . We define two events E ℓ and E ℓ for each ℓ ∈ [ k ] based on the fraction of far hypotheses in B ℓ in Line 25: 5

E ℓ := the event that a non-empty B ℓ in Line 25 contains at least ⌈ γ · | B ℓ |⌉ hypotheses that are (3 OPT + ϵ ) -far from P .

E ℓ := the event that a non-empty B ℓ in Line 25 contains fewer than ⌈ γ · | B ℓ |⌉ hypotheses that are (3 OPT + ϵ ) -far from P .

Conditioning on the event E ℓ , fewer than a γ fraction of the hypotheses in B ℓ are far. Since ˆ H is a randomly selected hypothesis in B ℓ , the probability of ˆ H being far is less than γ :

Next, we focus on the case where E ℓ holds: at least a γ fraction of the hypotheses in B ℓ are (3 OPT + ϵ ) -far from P . In the following lemma, we show that H i ∗ is a prompting hypothesis for such a B ℓ . The proof of this lemma is available in Section A.2.

Lemma A.1. Suppose B ℓ ⊆ H is the smallest non-empty bucket for which at least a γ fraction of its hypotheses are (3 OPT + ϵ ) -far from P . Then H i ∗ is a prompting hypothesis that can remove a γ fraction of the hypotheses in B ℓ .

If we output a hypothesis in B ℓ , clearly, the prompting hypotheses had failed to empty out the bucket. Therefore, the last time the algorithm was iterating over all i 's in [ n ] , the IS-PROMPTING procedure returned false for every single H i , and no 'restart' happened. However, during those iterations, at some point, i reached i ∗ , implying that the IS-PROMPTING procedure did not catch that H i ∗ is a prompting hypothesis. This event happens with probability at most δ fn .

Given Equation (4) and Equation (5), we bound the probability of outputting a far ˆ H as follows:

The last line is due to the fact that we set δ fn = δ ′ /k and γ = δ ′ .

Bounding the overall error probability: We have shown that if we output ˆ H , it is an acceptable answer with high probability. Now, we need to show that the overall error probability of the algorithm is not too high as well.

Our first claim is that the algorithm always ends, meaning that we do not restart the 'for' loop infinitely many times in Line 24. Note that the restart happens only after we check that at least γ/ 2 of the hypotheses are removed from B ℓ in Line 22. Therefore, the total number of restarts for any given bucket is at most log 1 / (1 -γ/ 2) ( | B ℓ | ) + 1 ≤ 4 log( n ) /γ times.

In addition, we claim the algorithm does not end without outputting ˆ H or ⊥ . Every H i belongs to some bucket at every step of the algorithm. This is due to the assumption that ˆ w j ( H i ) is in [0 , 1] . Thus, ˜ W ( H i ) is always between 0 ≤ ˜ W ( H i ) ≤ 1 < kϵ ′ . Thus, it is not possible to reach | B ℓ | = 0 for all ℓ ∈ [ k ] in Line 25. For the bucket that cannot be emptied out, the algorithm outputs either ⊥ or ˆ H .

Next, we show that the probability of outputting ⊥ is at most δ ′ . Note that the algorithm outputs ⊥ when H i was not able to remove at least γ/ 2 of the hypotheses in B ℓ . As we have established earlier, the restart happens only 4 log( n ) /γ times for every bucket. Therefore, the IS-PROMPTING procedure is invoked at most 4 kn log( n ) /γ times overall. Each time we have a chance of δ fp of declaring a hypothesis prompting erroneously. The algorithm in Line 22 checks if the error has happened, and it outputs ⊥ when it does. Hence, we have:

The final inequality follows from setting δ fp = γ · δ ′ / (4 k n log( n )) . Thus, we can bound the overall error probability as follows:

The last inequality holds since δ ′ = δ/ 3 .

Running time: The initialization takes O ( n + k ) time. The algorithm runs over k buckets. For each bucket, we may restart the 'for' loop O (log( n ) /γ ) times. Within each 'for' loop before restarting, we invoke the IS-PROMPTING procedure O ( n ) times where, with the exception of one of them, all return false . Let T ( i ) false indicate the time that the procedure spends on H i in the false case. Also, set this quantity be T true if the procedure returns true (which happens at most once before restarting). If we find a prompting hypothesis, we query the ˆ w i ( H j ) 's O ( n ) times, and spend O ( n · T q ) for this step. Hence, the total time complexity is:

Advanced version of IS-PROMPTING: We described the guarantees of the advanced version of the IS-PROMPTING procedure in Lemma A.3. In that lemma, we have shown that the running time of the false case is proportionate to the number of hypotheses we remove from B ℓ during the procedure of IS-PROMPTING. We denote this quantity in the r -th search for the prompting hypothesis H i in bucket b ℓ via n ( ℓ,i ) r . Since the total number of hypotheses that we can remove is at most O ( n ) , we have:

Using the setting of our parameters, δ fp = γ · δ ′ / (4 k n log( n )) = O ( n log n/ ( δ ϵ )) and δ fn = δ ′ /k = O (1 / ( δ ϵ )) .

Moreover, for the case that a prompting hypothesis is found:

Hence, the total time complexity is:

Simple version of IS-PROMPTING: In this version, we only have one possible confidence parameter, which is min( δ fn , δ fp ) . Hence, using Lemma A.2, we have:

Thus, the proof is complete.

A.1 Identifying prompting hypothesis

In this section, we provide two algorithms for identifying a prompting hypothesis. The simple algorithm, presented in Section A.1.1, involves sampling the hypotheses in a bucket to estimate the fraction of the hypotheses that H i can move, using a standard application of the Hoeffding bound. The advanced algorithm, presented in Section A.1.2, is based on the fact that we require different confidence parameters for false negative and false positive errors, given our analysis for Algorithm 1. We also show that the time complexity of this algorithm, in the case it returns false , is proportional to the number of hypotheses we can remove from the bucket during this procedure, which later helps us to use an amortized argument for the overall time spent on false returns.

A.1.1 Simple version of IS-PROMPTING

Algorithm 2 An algorithm to identify a prompting hypothesis (simple version).

Lemma A.2. Suppose we are given two parameters δ, ϵ ′ ∈ (0 , 1) , and three positive integers n , k , and ℓ ∈ [ k ] . Assume we are given a class of hypotheses H , its partition into k buckets B = { B 1 , B 2 , . . . , B k } , and a hypothesis H i ∈ H . The IS-PROMPTING-SIMPLE procedure in Algorithm 2 receives H , B , ℓ, H i , δ fp , δ fn , and ϵ ′ as its input and returns true or false with the following guarantees:

If H i substantially updates at least a γ -fraction of the hypotheses in B ℓ , then IS-PROMPTINGSIMPLE returns true with probability at least 1 -δ .

If H i substantially updates less than a γ/ 2 -fraction of the hypotheses in B ℓ , then ISPROMPTING-SIMPLE returns false with probability at least 1 -δ .

We have the following guarantees for the algorithm:

where T q is the time to obtain each ˆ w j ( H i ) .

Proof. This is a simple application of the Hoeffding bound:

A.1.2 Advanced version of IS-PROMPTING

We present an overview of the algorithm for the advanced version of IS-PROMPTING. Suppose we are given a parameter γ , and our goal is to design an algorithm that does the following with high probability: it returns true when H i can substantially update a γ -fraction of the hypotheses in B ℓ , and it returns false when H i can substantially update less than a γ/ 2 fraction of the hypotheses in B ℓ .

For this new procedure, we take advantage of an asymmetry that exists between returning false and true . This asymmetry arises mainly from the analysis of Algorithm 1. Roughly speaking, for every bucket ℓ , to find a single prompting hypothesis, we check O ( n ) (potentially non-prompting) hypotheses. On the other hand, we only need log 1 /γ ( B ℓ ) prompting hypotheses to empty out the whole bucket. Hence, we expect that the procedure returns mostly false rather than true .

Another difference here is the cost we pay for mistakenly declaring true or false . When we declare a hypothesis prompting, we iterate over all hypotheses in B ℓ and check if we can remove them from the bucket. If the hypothesis was not actually prompting, we have paid a substantial cost (in time) without making progress. Thus, we cannot have too many hypotheses that the algorithm declares prompting while they are not (false positives). However, the probability of mistakenly missing a prompting hypothesis does not have to be this small (false negatives).

Below is a more precise definition of these errors:

δ fp := Pr [ Returning true | H i substantially updates less than a γ/ 2 fraction ]

δ fn := Pr [ Returning false | H i substantially updates at least a γ fraction ]

Given the analysis of our algorithm, we can show that the probability of false positives must be around ˜ O (1 / | B ℓ | ) = ˜ O (1 /n ) . On the other hand, our analysis is robust as long as the probability of false negatives is roughly O ( δ ) , where δ is the overall confidence of our algorithm. (The reason for this choice of parameter may not be obvious from this high-level discussion; however, it is a direct artifact of our analysis.) This gap is particularly large when δ is a small constant, as is common in the literature.

The IS-PROMPTING procedure (Algorithm 3) is designed in a way that has different time complexities depending on its output being true or false . The procedure uses more than O (log n ) (more like O (log n · log(1 /δ ) /γ 2 ) ) time if it returns true . However, the main advantage is that it uses much less time when the output is false . In fact, instead of paying O (log n ) , we only spend O (log(1 /δ )) time plus an amortized cost of O (1 /γ ) for the O ( B ℓ ) calls to the procedure.

The algorithm runs in roughly O (log n ) rounds. In each round, it draws a few random hypotheses from B ℓ (roughly O (log(1 /δ ) /γ 2 ) ). In each round, we check if the fraction of hypotheses H i can substantially update is close to γ . At any round, if we see that the fraction is not close to γ , we return false . The hope is that for a non-prompting hypothesis, we either stop very quickly or the hypothesis passes too many rounds without us noticing that it is not prompting. Therefore, we must have seen an inflated number of substantial updates in these rounds. Before returning false , we perform those substantial updates among the sampled hypotheses that we have observed to remove those movable hypotheses from B ℓ .

We can capitalize on this fact in our cost analysis. In fact, we can show that in the false case, if the procedure takes roughly O ( t/γ ) time, it must have removed t hypotheses from the bucket. Thus, one can show that the amortized cost of these elongated rounds is only O (1 /γ ) . 6

Lemma A.3. Suppose we are given three parameters δ fp , δ fn , ϵ ′ ∈ (0 , 1) , and three positive integers n , k , and ℓ ∈ [ k ] . Assume we are given a classes of hypotheses H , its partition into k buckets B = { B 1 , B 2 , . . . , B k } and a hypothesis H i ∈ H . The IS-PROMPTING procedure in Algorithm 3 receives H , B , ℓ, H i , δ fp , δ fn , and ϵ ′ as its input and returns true or false with the following guarantees:

If H i substantially updates at least γ fraction of the hypotheses in B ℓ , then IS-PROMPTING returns true with probability at least 1 -δ fn .

If H i substantially updates less than γ/ 2 fraction of the hypotheses in B ℓ , then ISPROMPTING returns false with probability at least 1 -δ fp .

If the algorithm returns true

, we have the following guarantees for the algorithm: ,

Suppose the algorithm returns false . Assume n ( ℓ,i ) r indicates the number of hypotheses that the algorithm removes from B ℓ . Then, we have the following guarantees for the algorithm:

In above bounds, where T q denotes the time complexity to obtain each ˆ w j ( H i ) .

Proof. The probabilities in this proof are taken over the random choices of G ( r ) j 's. Note that S contains at the hypotheses for which we can change.

Probability of false positive: First, we show that probability of outputting true , while H i can substantially update less γ/ 2 fraction of hypotheses in B ℓ , is at most δ fp . The algorithm return true in two places. First, at the beginning of each round, we check whether S r -1 contains at least γ/ 2 -fraction of hypotheses. This answer is always correct with probability one. Since the algorithm

has an evidence that H i can certainly update at least γ/ 2 of hypotheses in B ℓ . Thus, this case does not affect our false positive rate δ fp .

Second, the algorithm returns true after all rounds end. This case only happens when u r /t is at least 3 γ/ 4 in every round in Line 20. It is not hard to see that u r is a binomial random variable with t trials. The success probability of each trial is the ratio of the hypotheses in B ℓ that H i can update substantially. Thus, in the case where H i substantially updates less than γ fraction of hypotheses, by Markov's inequality, we have:

Pr [ Returning true | H i substantially updates less than γ/ 2 -fraction ]

Probability of false negative: Suppose that H i substantially updates at least γ -fraction of the hypotheses in B ℓ . That is, H i can update each G ( r ) j with probability at least γ . Fix a round r . We bound the probability of returning false in Line 19 and Line 24.

In Line 19, we return false when | S r | - | S r -1 | < γ · t 8 . That is, the number of new hypotheses we added to S r is not too large. Consider the random hypotheses we draw at round r . Our claim is that most of these random hypotheses are not in S r -1 . The probability that one G ( r ) j be selected from S r -1 is | S r -1 | / | B ℓ | . However, this expectation is less than γ/ 2 . Otherwise, the algorithm would have returned true earlier in Line 5. With this information in mind, we can use the Hoeffding bound and get:

In Line 24, we return false if u r /t is less than 3 γ/ 4 . By Chernoff bound, we have:

Taking the union bound over all rounds, we obtain:

Pr [ Returning false | H i substantially updates at least γ -fraction ] ≤ δ fn .

Running time: In the case the algorithm return true , the algorithm makes O ( R · t ) queries to ˆ w i ( G ( r ) j ) 's. And, it runs in the O ( R · t · T q ) . If algorithm returns false in round r , similarly it makes O ( r · t ) queries to ˆ w i ( G ( r ) j ) 's. And, it runs in the O ( r · t · T q ) . Note that with the exception of the last round in every round r ′ ∈ [ r -1] . At least ( γ · t ) / 8 new hypotheses are added to S r ′ . Therefore, the size of S r is at least ( γ · t · ( r -1)) / 8 . We remove all of these hypotheses before we output false which cause the size of B ℓ to drop by n ( ℓ,i ) r := | S r | . In this case, the algorithm makes O ( t + n ( ℓ,i ) r /γ ) queries to ˆ w i ( G ( r ) j ) 's. And, it runs in the O (( t + n ( ℓ,i ) r /γ ) · T q ) . Thus, the proof is complete by setting R := ⌈ log 3 / 2 (1 /δ fp ) ⌉ and t := ⌈ 48 log(2 R/δ fn ) /γ 2 ⌉ .

A.2 Proof of Lemma A.1

Lemma A.1. Suppose B ℓ ⊆ H is the smallest non-empty bucket for which at least a γ fraction of its hypotheses are (3 OPT + ϵ ) -far from P . Then H i ∗ is a prompting hypothesis that can remove a γ fraction of the hypotheses in B ℓ .

Proof. We show that H i ∗ can substantially update ˜ W of any far hypothesis in B ℓ . Let H f denote a (3 OPT + ϵ ) -far hypothesis. First, observe that ˆ w i ∗ ( H f ) must be large compared to OPT :

In addition, OPT cannot be much smaller than ℓ ϵ ′ . The main reason is that all the buckets before B ℓ are empty. Hence, H i ∗ belongs to a bucket B ℓ ′ where ℓ ′ is at least ℓ . By our definition of bucketing: ˜ W ( H i ∗ ) must be at least ( ℓ -1) ϵ ′ . Hence we have:

Note that we assume H f belongs to B ℓ , so ˜ W f is less than ℓϵ ′ . Putting all these observation together, we achieve:

Therefore, if we update ˜ W f via ˆ w i ∗ ( H f ) , then H i ∗ can significantly improve ˜ W f . This fact implies that H i ∗ can removes all the bad hypothesis from B ℓ which concludes the proof of the lemma.

B Linear time algorithm with α = 4

In this section, we present our second algorithm whose sample complexity has a better dependency on the accuracy parameter ϵ . An overview of our approach is presented in Section 3.3. The pseudocode of our approach is presented in Algorithm 4. We prove the performance of this algorithm in Theorem 6.

Apart from improving the accuracy parameters, we also provide modifications to our algorithm that can achieve low rounds of adaptivity (roughly speaking, it means the number of times the algorithm needs to look at the sample set). More formally, in our access model, we define one round of adaptivity as follows: The algorithm selects a set of t = O ( n 2 ) pairs of indices { ( i ℓ , j ℓ ) } ℓ ∈ [ t ] and query all ˆ w j ℓ ( H i ℓ ) at once. The number of adaptivity rounds represents the total count of times the algorithm needs to repeat this process in order to produce its final output. This is particularly interesting when considering a federated learning setting in which the rounds of interactivity are important. The modification of our algorithm runs in time O ( n 1+ λ ) runs in O (1 /λ ) rounds of adaptivity.

Theorem 6. Suppose Algorithm 4 receives these parameters as input: σ ∈ [0 , 1] , η ∈ (0 , 1 / 4) , ϵ, δ ∈ (0 , 1) . Also, assume the algorithm has access to a class of n hypotheses H , and a set of unmarked hypotheses in Q ⊆ H where initially Q = H . Suppose the algorithm can query ˆ w j ( H i ) 's with error at most ϵ ′ ≤ ϵ/ 6 . The algorithm has one of the two possible outcomes: either it declares that σ < OPT ; or it outputs a hypothesis H i such that: with probability at least 1 -δ we have:

The algorithm runs in O ( n/η · T q ) time and O ( log 2 1 /η ( n ) ) rounds of adaptivity. Here T q is the time required to obtain each ˆ w j ( H i ) .

Proof. For the sake of argument, assume each of the sub-routines in the algorithm works as guaranteed with probability one (as oppose to the case where they work as expected with probability 1 -δ ′′ ). Later on, we discuss the overall confidence of the algorithm to remove this assumption.

Accuracy of seeds: the procedure to find the initial seeds provides us with a ( σ + ϵ ′ , 3 σ +3 ϵ ′ , m ) -seed with high probability. Using Fact C.4, this seed is also a (2 σ + ϵ ′ , 4 σ +5 ϵ ′ , m ) . In boosting seeds, we start with (2 σ + ϵ ′ , 4 σ +5 ϵ ′ , m ) and we get (2 σ + ϵ ′ , 4 σ +5 ϵ ′ , m ′ ) where m ′ = ⌊ η · m ⌋ . This statement is justified by Theorem 9, setting κ = κ ′ = 2 , and using the same values for a = 2 , σ + ϵ ′ and b = 4 , σ +5 ϵ ′ for our seeds. It is worth noting that the only parameter that changes while we boost is m .

Accuracy of the output Consider the case where we produce output when |Q| is zero. That implies that our algorithm marked all the hypotheses. Throughout the course of algorithms, we never mark a hypothesis H i unless we have found an evidence that ∥ H i -P ∥ TV > σ . This was established by observing ˆ w j ( H i ) > σ + ϵ ′ , which implies ∥ H i -P ∥ TV must be greater than σ ; Or, showing a close-by hypotheses to H i is far from P and apply triangle inequality. See our argument for Case 3.2 in the proof of Theorem 9 for example. Thus, if all hypotheses are marked, it must be the case that for every H i , ∥ H i -P ∥ TV is greater than σ . Hence, when the size of Q is zero we can truly assert that ' σ < OPT '.

For the case that |Q| = 1 , the algorithm focuses on H i , the only unmarked hypothesis left. If we found ˆ W ( H i ) is greater than σ + ϵ ′ , we can simply conclude that for every H i , ∥ H i -P ∥ TV is greater than σ , and ' σ < OPT '. Otherwise, using Fact C.3, the H i is a valid answer:

In addition, the accuracy of the H i that is returned in Line 22 is guaranteed by the Theorem 9, and setting κ = κ ′ = 2 . Next, lets focus on the hypotheses we output after the 'if' condition in Line 29. When ˆ W ( H ℓ ) is small, we have:

When ˆ W ( H ℓ ) is large, the analysis is very similar to Case 1 in the proof of Theorem 9. Again, by setting κ = κ ′ = 2 , and using the facts that ˆ W ( H ℓ ) -ϵ ′ ≥ κ ′ σ and ˆ W ( H i ) ≤ ( κ +2) σ +5 ϵ ′ , we obtain:

Number of recursions: Given Theorem 8 and Theorem 9, we only see 'start over' when (1 -2 η ) fraction of unmarked hypotheses in Q have been marked. Since we have n hypotheses, we do not start over more than O (log 1 /η ( n )) times.

Running time: In the while loop, m decreases with a factor of η every time we find a new seed. Hence, the total number of iterations is bounded by O (log 1 /η ( n )) . Using Theorem 8 and Theorem 9, invoking the procedures for finding a seed and boosting a seed each takes O ( n log( n/δ ′′ ) /η · T q ) time. Thus, the total time complexity is:

Overall confidence parameter: The total number of subroutines we call here is at most O (log 2 1 /η ( n )) . Thus, by setting δ ′′ to O ( δ/ log 2 1 /η ( n )) and using the union bound, we can show the overall error probability is bounded by δ .

Rounds of adaptivity: Using Theorem 8 and Theorem 9, invoking the procedures for finding a seed and boosting a seed each takes O (1) rounds of adaptivity. Hence, the overall rounds of adaptivity is O ( log 2 1 /η ( n ) ) rounds.

Remark 7. One could argue that we have demonstrated the algorithm operates in ˜ O ( n/η · T q ) time with a probability of 1 -δ . However, this fact does not inherently guarantee that the algorithm consistently maintains the desired time complexity. Fortunately, a straightforward solution exists to address this concern.

Our algorithm may fail for two primary reasons: either too few hypotheses were marked, or the identified seed had a larger m value than anticipated. In either scenario, the algorithm can verify the occurrence of this undesirable, but improbable event. In these situations, we can output ⊥ to indicate the algorithm's failure to produce a valid answer. With this adjustment, the time complexity remains low as desired.

Corollary B.1. Suppose H is a class of n known hypotheses and P is an unknown hypothesis. Let ϵ and δ be two parameters in (0 , 1) . Assume the algorithm has access to accurate estimates of ˆ w j ( H i ) 's that have error at most ϵ ′ . If 3 ϵ ′ ≤ ϵ , then for every η ∈ (1 /n, 1 / 4) there exists an ( α = 4 , ϵ, δ ) -proper learner for P in H . The running time of our algorithm is ˜ O (( n/η ) · log(1 /δ ) · log(1 /ϵ ) · T q ) , and it can be implemented in O ( log 2 1 /η ( n ) · log(1 /ϵ ) ) rounds of adaptivity.

Proof. This is a direct corollary of Theorem 6 combined with a binary search over values of σ ∈ { ϵ ′ , ϵ ′ } where ϵ ′ := ϵ/ 100 . Note that for every σ ≥ OPT , the algorithm has to produce a hypothesis, since it cannot declare OPT < σ with high probability. Outputting the hypothesis associated with the smallest σ would give us the desired guaranteed.

B.1 Finding initial seed

In this section, we describe an algorithm that receives parameter σ as input and finds a ( σ + ϵ ′ , 3 σ + 2 ϵ ′ , ⌊ η · n ⌋ ) -seed in roughly ˜ O ( n/η · T q ) time with high probability or declares ' σ < OPT ".

Algorithm 5 finds a ( σ + ϵ ′ , 3 σ +3 ϵ ′ , m ) -seed

Theorem 8. Suppose that we are given a class of n hypotheses, H , a rate parameter η ∈ (0 , 1 / 4) , two parameters σ, δ ∈ (0 , 1) , and we have access to ˆ w j ( H i ) with error at most ϵ ′ for every i, j ∈ [ n ] . Algorithm 5 queries O ( n log( n/δ ) /η )) many ˆ w i ( H j ) and runs in O (( n log( n/δ ) /η ) · T q ) time. Also, it can be implemented in O (1) rounds of adaptivity. This algorithm returns a hypothesis H ℓ , declares σ < OPT , or returns 'start over' for which the following guarantees hold with probability 1 -δ :

If σ ≥ OPT , then the algorithm does not declare that σ is less than OPT .

If the algorithm returns a hypotheses H ℓ , then H ℓ is a ( σ + ϵ ′ , 3 σ +3 ϵ ′ , m ) -seed where m := ⌊ η · n ⌋ .

If the algorithm returns 'start over. ' with probability 1 -δ , it marks (1 -2 η ) fraction of the hypotheses in Q .

Proof. Our first claim is that if σ is at least OPT , then we do not declare otherwise. This is due to the fact that the discrepancy between H i ∗ and P on any subset of the domain, including the Scheffé sets, will not exceed OPT . Hence, ˆ w j ( H i ∗ ) is at most OPT + ϵ ′ ≤ σ + ϵ ′ , and we will not mark H i ∗ in Line 7, Line 10, nor Line 15. That is, for at least one hypothesis, i.e., H i ∗ , the 'if' statement in Line 16 holds. And, we return a seed or 'start over.' (And, we will never reach Line 25.)

Next, we show that if the algorithm outputs H i , then it is a ( σ + ϵ ′ , 3 σ +3 ϵ ′ , m ) -seed with high probability. First, observe that ˆ W ( H i ) must be at most 3 σ +3 ϵ ′ due to the 'if' condition in Line 18, satisfying one of the two conditions we need for our desired seed in Definition 3.1.

Now, we show the next required condition for H i to be a good seed holds as well: |M σ + ϵ ′ , 3 σ +3 ϵ ′ ( H i ) | ≤ m . This condition requires that the number of H j 's in H for which ˆ w j ( H i ) is in ( σ + ϵ ′ , 3 σ +3 ϵ ′ ] is bounded by m . We have already established that ˆ w j ( H i ) for every j ∈ [ n ] is bounded by 3 σ +3 ϵ ′ since ˆ W ( H i ) is at most 3 σ +3 ϵ ′ . Hence, we only need to show that there are at most m hypotheses H j such that w j ( H i ) is larger than σ + ϵ ′ .

Now, if there are more than m hypothesis H j in H such that ˆ w j ( H i ) > σ + ϵ ′ . Now, if we sample t := ⌈ 8 n log (2 n/δ ) /m ⌉ hypotheses, we should observe at least one H j for which ˆ w j ( H i ) is larger than σ + ϵ ′ , with probability at least 1 -δ/ (2 n ) . However, we know that such a hypothesis was never observed because we did not mark H i earlier. By the union bound, with a probability of at least 1 -δ/ 2 , no such H i exists. Hence, with probability at least 1 -δ/ 2 , H i is an ( σ + ϵ ′ , 3 σ +3 ϵ ′ , m ′ ) -seed as promised in the statement of the lemma.

Next, assume we output 'start over.' after finding an unmarked hypothesis H i . In this case, we know that H i is not marked and ˆ W ( H i ) > 2 σ +3 ϵ ′ . Note that we have sampled t edges of H i , and we never see w j ( H i ) > σ + ϵ ′ . Using a very similar argument as we have above: one can show there cannot be more than m ′ = ⌈ η · | Q |⌉ for which w j ( H i ) > σ + ϵ ′ with probability 1 -δ/ 2 (for every H i ). That means, we will mark |Q| m ′ many hypothesis in Q . If |Q| > t ≥ 1 /η , this implies that (1 -2 η ) fraction of the hypothesis in Q are removed. If |Q| ≤ t , then we know all the hypothesis in Q have w j ( H i ) ≤ σ + ϵ ′ and all of them will be marked.

Furthermore, we show that we did not wrongfully mark a hypothesis that was σ -close to P . Observe that the condition in Line 22 holds in two cases:

Case 1: ˆ w i ( H j ) > σ + ϵ ′ ˆ w i ( H j ) > σ + ϵ ′ ˆ w i ( H j ) > σ + ϵ ′ . It is straightforward to show that H j is not σ -close to P since we have:

Case 2: ˆ w i ( H j ) ≤ σ + ϵ ′ ˆ w i ( H j ) ≤ σ + ϵ ′ ˆ w i ( H j ) ≤ σ + ϵ ′ and ˆ w j ( H i ) ≤ σ + ϵ ′ ˆ w j ( H i ) ≤ σ + ϵ ′ ˆ w j ( H i ) ≤ σ + ϵ ′ . Even though ˆ w ℓ ( H j ) is small in this case, we can indirectly deduce that H j is not σ -close to P . By the definition of ˆ w j ( H i ) and ˆ w i ( H j ) , we have:

On the other hand, by the triangle inequality, we have:

Therefore, in both of the cases above, we do not mark an σ -close distribution to P .

By the union bound over all the steps, our guarantees hold with probability 1 -δ . In addition, it is not hard to see that the running time of the algorithm is O (( n · t + n ) · T q ) = O ( n · (log( n/δ )) · T q /η ) time.

Rounds of adaptivity: It is straight forward to show that this algorithm can be implemented with a constant round of adaptivity. Let Q 0 denote the initial state of unmarked hypotheses at the algorithm's outset. Since hypotheses are marked in sequence, in round i , the set of unmarked hypotheses is Q i := Q 0 ∩ H i , H i +1 , . . . , H n . As a result, we can agree on the set of random H j 's in Q and H and request ˆ w j ( H i ) within a single round. If the size of |Q i | is at most t , then we include all ˆ w j ( H i ) for every i ∈ n and H j ∈ Q . Next, upon discovering an unmarked H i , we initiate another round of adaptivity to request all ˆ w j ( H i ) and ˆ w j ( H i ) for all j ∈ [ n ] . This enables the implementation of the rest of the algorithm.

B.2 Boosting a seed

In this section, we provide an algorithm that receives a ( κσ + ϵ ′ , ( κ +2) σ +5 ϵ ′ , m ) -seed and aims to find a ( κ ′ σ + ϵ ′ , ( κ ′ +2) σ +5 ϵ ′ , m ′ ) -seed with smaller m ′ . That means, the size of M for this new seed is smaller which brings us closer to have enough time to fully investigate the set M for a future seed. Our algorithm runs in ˜ O ( n · T q /η ) time where η ≈ m ′ /m is our rate parameter. This running time is adaptive to our budget. If we are aiming for ˜ O ( n ) algorithm, we find a seed such that m ′ is only smaller than m by a constant factor (constant η ) ; Alternatively, if we have more time, we can decrease m ′ with a faster rate (smaller η ).

While our main goal is to find a better seed, our algorithm may not always be able to do so; However, it makes progress towards the end goal of the algorithm one way or the other. There are three possible outcomes for our algorithm:

The algorithm finds an accurate enough ˆ H as the final output of the algorithm.

The algorithm finds a better seed as we aimed for.

The algorithm marks (1 -2 η ) fraction of the unmarked hypotheses; And after that, we start over the procedure.

Although we may have regressed in last case above, it does not happen too often. Since we have only n unmarked hypotheses to begin with, and we marked a large fraction of them every time, we do not start more than O (log 1 / 2 η ( n )) times over the course of the algorithm. Our approach is presented in Algorithm 6, and we prove its performance in Theorem 9.

Algorithm 6 aims to find a better seed.

Theorem 9. Suppose Algorithm 6 receives these parameters as input: σ ∈ R ≥ 0 , κ, κ ′ ≥ 1 , η ∈ (0 , 1 / 4) , ϵ ′ , δ ∈ (0 , 1) , and two non-negative integers n, m . Also, assume the algorithm has access to a class of n hypotheses H , and a set of unmarked hypotheses in Q ⊆ H . It also receives a hypothesis H i ∈ H . The algorithm has one of the three possible outcomes listed below. Now, for any setting of such input parameters, if H i is a ( κσ + ϵ ′ , ( κ +2) σ +5 ϵ ′ , m ) seed, then the following guarantees hold for the outcome of Algorithm 6 with a probability of at least 1 -δ :

If the algorithm outputs a hypothesis ˆ H as the final answer, then we have for every ϵ ≥ 6 ϵ ′ :

If the algorithm outputs a hypothesis H ℓ as a new seed, then H ℓ is a ( κ ′ σ + ϵ ′ , ( κ ′ +2) σ + 5 ϵ ′ , m ′ ) -seed where m ′ := ⌊ η m ⌋ .

Algorithm 7 computes ˜ W ( H )

If the algorithm requires us to start over, then we have marked at least (1 -2 η ) of the unmarked hypotheses.

The algorithm runs in O ( n · (log( n/δ )) · T q /η ) time and can be implemented in O (1) rounds of adaptivity.

Proof. Let M denote the set M κσ + ϵ ′ , ( κ +2) σ +5 ϵ ′ ( H i ) . We consider the three possible outcomes of the algorithm, and prove the algorithm satisfied the desired guarantee in each of the three cases.

Case 1: ˜ W ( H ℓ ) > κ ′ σ + ϵ ′ ˜ W ( H ℓ ) > κ ′ σ + ϵ ′ ˜ W ( H ℓ ) > κ ′ σ + ϵ ′ . In this case, the algorithm will return H i as the final answer. We show that the total variation distance between H i and P is as desired. There are two possibilities depending on whether H i ∗ is in M or not.

Case 1.1: H i ∗ ∈ M H i ∗ ∈ M H i ∗ ∈ M . Using that H ℓ has the minimum ˜ W ( H ℓ ) among all the hypotheses in M , we can bound OPT as follows:

We prove that H i is not too far from P . Note that since H i was an ( κσ + ϵ ′ , ( κ +2) σ + 5 ϵ ′ , m ) -seed, we are guaranteed that ˆ W ( H i ) is bounded by ( κ +2) σ +5 ϵ ′ . Using the above bound for OPT and Fact C.3, we get:

Case 1.2: H i ∗ ̸∈ M H i ∗ ̸∈ M H i ∗ ̸∈ M . Since H i is a ( κσ + ϵ ′ , ( κ + 2) σ + 5 ϵ ′ , m ) -seed, ˆ w i ∗ ( H i ) is bounded by ˆ W ( H i ) ≤ ( κ + 2) σ + 5 ϵ ′ . Now that H i ∗ is not in M , it must be the case that ˆ w i ∗ ( H i ) is bounded by κσ + ϵ ′ . In this case, we have the following bound for the total variation distance between H i and P via Fact C.3:

Case 2: ˜ W ( H ℓ ) ≤ κ ′ σ + ϵ ′ ˜ W ( H ℓ ) ≤ κ ′ σ + ϵ ′ ˜ W ( H ℓ ) ≤ κ ′ σ + ϵ ′ and ˆ W ( H ℓ ) ≤ ( κ ′ +2) σ +5 ϵ ′ ˆ W ( H ℓ ) ≤ ( κ ′ +2) σ +5 ϵ ′ ˆ W ( H ℓ ) ≤ ( κ ′ +2) σ +5 ϵ ′ . In this case, we output H ℓ as a new seed. Given the assumptions of this case, we show that H ℓ is a ( κ ′ σ + ϵ ′ , ( κ ′ + 2) σ + 5 ϵ ′ , m ′ ) -seed. First, ˆ W ( H ℓ ) is at most ( κ ′ + 2) σ + 5 ϵ ′ implying the first desired property for being an ( κ ′ σ + ϵ ′ , ( κ ′ +2) σ +5 ϵ ′ , m ′ ) -seed holds for H ℓ . Next, we show the second desired property for H ℓ : the size of M κ ′ σ + ϵ ′ , ( κ ′ +2) σ +5 ϵ ′ ( H ℓ ) is bounded by m ′ . At a high level, we expect the size of

this set to be small; Otherwise, we would have observed one of these large ˆ w ℓ ( H j ) when we have sampled t hypotheses to compute ˜ W ( H ℓ ) .

Suppose H ℓ has more than m ′ edges with ˆ w j ( H ℓ ) > κ ′ σ + ϵ ′ . Now, if we sample t := ⌈ 8 n log (2 n/δ ) /m ′ ⌉ hypotheses, we should observe at least one H j for which ˆ w j ( H ℓ ) is larger than κ ′ σ + ϵ ′ , with probability at least 1 -δ/ (2 n ) . However, we know that such a hypothesis was never observed because ˜ W ( H ℓ ) , which is the maximum of ˆ w j ( H ℓ ) 's, is bounded from above by κ ′ σ + ϵ ′ . By the union bound, with a probability of at least 1 -δ/ 2 , no such H ℓ exists. Hence, with probability at least 1 -δ/ 2 , H ℓ is an ( κ ′ σ + ϵ ′ , ( κ ′ +2) σ +5 ϵ ′ , m ′ ) -seed as promised in the statement of the lemma.

Case 3: ˜ W ( H ℓ ) ≤ κ ′ σ + ϵ ′ ˜ W ( H ℓ ) ≤ κ ′ σ + ϵ ′ ˜ W ( H ℓ ) ≤ κ ′ σ + ϵ ′ and ˆ W ( H ℓ ) > ( κ ′ +2) σ +5 ϵ ′ ˆ W ( H ℓ ) > ( κ ′ +2) σ +5 ϵ ′ ˆ W ( H ℓ ) > ( κ ′ +2) σ +5 ϵ ′ . In this case the algorithm marks any unmarked hypothesis H j for which ˆ w ℓ ( H j ) > σ + ϵ ′ or ˆ w j ( H ℓ ) ≤ κ ′ σ + ϵ ′ , and we start over. First, we show that we did not wrongfully mark a hypothesis that was σ -close to P . Observe that the condition in Line 15 holds in two cases:

Case 3.1: ˆ w ℓ ( H j ) > σ + ϵ ′ ˆ w ℓ ( H j ) > σ + ϵ ′ ˆ w ℓ ( H j ) > σ + ϵ ′ . It is straightforward to show that H j is not σ -close to P since we have:

Case 3.2: ˆ w ℓ ( H j ) ≤ σ + ϵ ′ ˆ w ℓ ( H j ) ≤ σ + ϵ ′ ˆ w ℓ ( H j ) ≤ σ + ϵ ′ and ˆ w j ( H ℓ ) ≤ κ ′ σ + ϵ ′ ˆ w j ( H ℓ ) ≤ κ ′ σ + ϵ ′ ˆ w j ( H ℓ ) ≤ κ ′ σ + ϵ ′ . Even though ˆ w ℓ ( H j ) is small in this case, we can indirectly deduce that H j is not σ -close to P . By the definition of ˆ w j ( H ℓ ) and ˆ w ℓ ( H j ) , we have:

On the other hand, by the triangle inequality, we have:

Therefore, in both of the cases above, we do not mark an σ -close distribution to P .

Second, we claim that we mark at least (1 -2 η ) fraction of the unmarked hypotheses in Q . Recall that when we compute ˜ W ( H ℓ ) we also sample t hypotheses from the set of unmarked hypotheses, and we did not observe any edge ˆ w j ( H ℓ ) that is larger than κ ′ σ + ϵ ′ . With a very similar argument we had above, with probability 1 -δ/ (2 n ) , we do not have more than m ′′ := ⌈ η · |Q| · m/n ⌉ ≤ ⌈ η · | Q |⌉ hypotheses in Q such that ˆ w j ( H ℓ ) > κ ′ σ + ϵ ′ . Otherwise, we would have seen one of these edges, and ˜ W ( H ℓ ) would have been larger. By the union bound, this fact holds for every H ℓ . Hence, the 'if' condition in Line 15 holds for over |Q| - ⌈ η · |Q|⌉ of the unmarked hypotheses in Q , and the algorithm mark them. If |Q| ≥ 1 /η , it is easy to show that |Q| - ⌈ η · |Q|⌉ is at least (1 -2 η ) · |Q| . Thus, we mark at least (1 -2 η ) fraction of the hypotheses in Q as we have claimed. Now, if | Q | ≤ 1 /η , it is easy to show that | Q | < t . Hence, the algorithm involves all the H j 's in Q to compute ˆ W ( H ℓ ) . Therefore, every single ˆ w j ( H ℓ ) is at most κ ′ σ + ϵ ′ and we mark all the hypotheses in Q .

By the union bound over all the steps, our guarantees hold with probability 1 -δ . In addition, it is not hard to see that the running time of the algorithm is O (( m · t + n ) · T q ) = O ( n · (log( n/δ )) · T q /η ) time.

Rounds of adaptivity: Our algorithm can be implemented in constant rounds of adaptivity. To compute ˜ W 's, we can preselect the random H j 's. If Q contains fewer than t +1 hypotheses, we include every ˆ w j ( H i ) for each H j ∈ Q and i ∈ [ n ] . Upon discovering H ℓ , another round of adaptivity allows us to query every ˆ w j ( H ℓ ) and ˆ w ℓ ( H j ) to calculate ˆ W ( H ℓ ) and mark the hypotheses in Q .

C Preliminary facts and lemmas

Fact C.1. For all distribution P and for all probability event E defined under P , we have:

where P |E denotes the probability distribution P when conditioned on the event E .

Fact C.2. For every pair of distributions H i and H j , we have:

The following fact is adapted from [DL01, MS08, ABS23].

Fact C.3. Suppose H i ∗ is the closest hypothesis to P in H (i.e., ∥ H i ∗ -P ∥ TV). For every pair of hypotheses H i and H j , the following holds:

Proof. The proof is based on triangle inequality, and the definitions of w j ( H i ) 's and OPT . For every H j in H , we have:

Hence, Item 1 is proved. Now, if we set H j to be H i ∗ , then ∥ H i ∗ -P ∥ TV is equal to OPT implying Item 2. Item 3 is concluded from Item 2 and the fact that w i ∗ ( H i ) is upper bounded by W ( H i ) .

Fact C.4. Suppose we are given six parameters a, b, a ′ , b ′ ∈ R ≥ 0 , and m,m ′ ∈ Z ≥ 0 . If a ≤ a ′ , b ≤ b ′ , and m ≤ m ′ , every ( a, b, m ) -seed is also an ( a ′ , b ′ , m ′ ) -seed.

Fact C.5. Suppose we have a set of n hypotheses H , and a predicate, R ( H ) : H → { 0 , 1 } , that maps the hypotheses in H to zero or one. Assume H contains more than m hypotheses with R ( H ) = 1 for an arbitrary integer parameter 0 ≤ m < n . If we draw s ≥ 8 n log( δ -1 ) /m hypotheses from H uniformly at random, we will observe at least one hypothesis with R ( H ) = 1 with probability at least 1 -δ .

Proof. Let p denote the fraction of such hypothesis in H with R ( H ) = 1 . Observe that p > 0 since there are at least m +1 such hypothesis in H . Then, using the Chernoff bound, we have:

D Data structure for marked and unmarked hypotheses

Here, we describe a simple data structure that allows us to efficiently keep track of marked and unmarked hypotheses. Our data structure consists of two integers, n and t , and two arrays of size n : index[] and list[] . Here, n represents the number of hypotheses that the data structure supports. The list array is an arbitrary ordering of integers from 1 to n . Throughout the operations of this data structure, we preserve the following guarantee: the value of index[i] indicates where to find element i in the list . In other words, list[index[i]] is always i . The integer t serves as a threshold quantity. The first t numbers in the list correspond to unmarked hypotheses, while the remaining numbers represent marked hypotheses. Initially, list is a list of integers from 1 to n in

ascending order. And, for every hypothesis H i , index[i] is set to i . This data structure supports the following operations:

Initialization ds(size) : To initialize the data structure, we set n and t to size . Next, we set list to be a list of integers from 1 to n in ascending order. To ensure consistency with the list , each index[i] is set to i .

mark(i) : To mark hypothesis H i , we swap the element at index i with the element at index t in the list array, and then decrement t by 1 .

is_marked(i) : Given our definition, if index[i] is less than or equal to t , then H i is unmarked; otherwise, it is marked.

is_all_marked() : If t is equal to zero, then it means all the hypotheses are marked.

random_unmarked() : To select a random unmarked hypothesis, we generate a random integer r between 1 and t , and output the hypothesis at list[r] .

random_marked() : To select a random marked hypothesis, we follow the same process as for an unmarked hypothesis, except that r is generated between t+1 and n .

It is easy to see that the initialization takes O ( n ) time, and the rest of the operations only take O (1) time.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our results are a theoretical contribution to the field of learning theory. The paper, including the appendix, contains all the algorithms, theorem statements, and their proofs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see the discussion in Section 1.2 regarding potential improvements and future directions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have clearly stated our model and our assumptions. All the proofs are included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: We do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

4.1 If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

4.2 If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

4.3 If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

4.4 We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We do not have any experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: We do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our results are a theoretical formulation of a fundamental problem in statistics and learning theory. They can be viewed as a computationally efficient version of existing results. Given the theoretical nature of our findings, we do not anticipate any harm to society.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Given the theoretical nature of our findings, we do not anticipate broader societal impact for this paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not have any experimental results.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We did not use any existing assets, other than the publicly available articles that we have cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The contributions of this paper are theoretical. As is standard in the field, we will make our results publicly accessible and do not anticipate generating revenue from them.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not have any crowdsourcing experiments nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have any experiments involving crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the hypothesis selection problem: Given $n$ distributions $H_1, H_2, \\ldots, H_n$ and samples from an unknown distribution $P$, the goal is to output $\\hat H$ such that $\\mathrm{TV}(P, \\hat H) \\le \\alpha \\cdot \\mathsf{OPT} + \\epsilon$, where $\\mathsf{OPT} = \\min_{i \\in [n]}\\mathrm{TV}(P, H_i)$ is the TV-distance of the best hypothesis.\n\nIt is known that achieving a guarantee with $\\alpha < 3$ is impossible, if we want the sample complexity to be bounded by a function of $n$ and $\\epsilon$ (and not the domain size). Prior work gave various algorithms that achieve $\\alpha = O(1)$ (including some with $\\alpha = 3$), but none of them achieves $\\alpha = 3$ in near-linear time without making additional assumptions.\n\nThis work gives two improved algorithms. The first algorithm (Algorithm 1) achieves $\\alpha = 3$ in $\\tilde O(n\\cdot s / \\epsilon)$ time, where $s = \\Theta((\\log n) / \\epsilon^2)$ is the sample complexity. The second algorithm (Algorithm 4) achieves $\\alpha = 4$ in $\\tilde O(n\\cdot s)$ time.\n\nHere is an overview of Algorithm 1, outlined by the authors in Section 3:\n- Let $S_{i j}$ denote the subset of the domain that witnesses $\\mathrm{TV}(H_i, H_j)$. A simple argument shows that finding $i \\in [n]$ that minimizes the quantity $W(H_i) \\coloneqq \\max_{j \\in [n]}|H(S_{i j}) - P(S_{i j})|$ gives the $\\alpha = 3$ guarantee. However, the straightforward implementation takes $\\Omega(n^2)$ time to compute $W(H_i)$ for all $i \\in [n]$.\n- The actual algorithm maintains estimates on $W(H_i)$s (that always underestimate). These are grouped into $\\approx 1/\\epsilon$ \"buckets\", where bucket $l$ contains indices with estimated $W(H_i) \\approx l\\cdot\\epsilon$.\n- In each iteration, we find the lowest non-empty bucket $l$, and tries to update the estimates. Concretely, we loop over all $j \\in [n]$ and only sample a few indices $i$ from bucket $l$. For each $(i, j)$ pair, we try to update the estimate of $W(H_i)$ using $w_j(H_i)$. This might bump some $i$ into larger buckets.\n- If there is a single $j \\in [n]$ that bumps a substantial fraction of the sampled $i$s, we call such $j$ a \"prompting hypothesis\", and we use such $j$ to update all estimates in the current bucket. Note that if we keep finding prompting hypotheses, we will empty a bucket in $O(\\log n)$ repetitions, and the whole process must end in $O((\\log n)/\\epsilon)$ iterations. Moreover, each iteration is almost linear-time.\n- The harder part is to argue that, whenever we cannot find a prompting hypothesis, we are done. At a high level, this boils down to showing that the optimal hypothesis $i^*$ must be prompting, unless the majority of hypothesis in the lowest bucket are already good enough.\n\nSTRENGTHS:\n- This work makes progress on improving the runtime for a fundamental problem in statistics / machine learning.\n- The new algorithms are based on clever ideas and several new concepts, which might find applications in future work.\n- The writing is very clear and I found the main paper relatively easy to follow. I especially liked that the authors spent most of the main paper on the proof overview, which is enjoyable to read and should be able to convince the readers why the proposed methods work.\n\nOverall, this paper contains fairly significant results that are very well presented and should be of interest to most learning theorists. I vote for acceptance.\n\nWEAKNESSES:\nThe main weakness is that this work is unlikely to have said the last words on the problem---it remains open whether one can get $\\alpha = 3$ in $\\tilde O(n/\\epsilon^2)$ time.\n\nMinor writing suggestions:\n- Equation (2): Missing extra space in $S_{ij}$ on the left-hand side? (The notational convention in the rest of the paper seems to be $S_{i j}$, though personally writing $S_{i,j}$ might be clearer.)\n- Line 90: \"a $O(\\cdots)$ factor\" --> \"an $O(\\cdots)$ factor\"\n- Line 270: \"dependency\" not capitalized.\n\nQUESTIONS:\nIs there a reason why the \"right\" runtime should be $n \\cdot s = \\tilde O(n/\\epsilon^2)$? For example, given that the algorithm needs $\\Omega(n + s) = \\tilde\\Omega(n + 1/\\epsilon^2)$ time to read the inputs---can we hope to get an $\\tilde O(n + 1/\\epsilon^2)$ runtime?\n\nLIMITATIONS:\nLimitations are adequately addressed in Section 1.2.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the hypothesis selection problem, where given a set of candidate distributions $\\mathcal{H}=\\{H_1,\\dots,H_n\\}$ and samples from a distribution $P$, the learner wants to approximately select a distribution $H_i \\in \\mathcal{H}$ such that $||H_i-P||_{TV} \\le \\alpha opt+\\epsilon$ for some constant $\\alpha$, where $opt$ is the TV distance of the closest distribution in $\\mathcal{H}$ to $P$. Information theoretically, $\\alpha=3$ is the optimal constant one could achieve with optimal sample complexity $s=\\log n/\\epsilon^2$. Previous efficient algorithms either have a running time that has a bad dependence on $n$ or run in nearly linear time but achieve a suboptimal approximate factor $\\alpha \\ge 5$. In this paper, the authors design an algorithm that achieves the optimal approximate factor $\\alpha=3$, with running time $\\tilde{O}(ns/\\epsilon)$. This is the first algorithm that achieves the optimal approximate factor with a running time nearly linearly dependent on $n$. This paper also presents another algorithm that achieves an approximate factor $\\alpha = 4$ but has a better running time $\\tilde{O}(ns)$\n\nSTRENGTHS:\nThis paper makes progress on a fundamental problem, hypothesis selection. The two algorithms designed by the authors are highly non-trivial and involve many novel techniques. The paper is in general written well and gives a good intuition of why the algorithms work. Due to the time limit, I was not able to check the proofs in the appendix but given the discussions in the main body, I believe the results given by the paper are correct.\n\nWEAKNESSES:\nI want to give some minor suggestions. \n\n1. It would be nice to give some more applications or motivations at the beginning of the introduction. In the current version of the paper, the problem sounds mathematically interesting but lacks motivation.\n\n2. Since there is a long line of works that design nearly linear time algorithms for the problem. It would be helpful to give a brief review of the techniques used in prior works in the introduction.\n\n3. In line 327, I believe there is a typo in the equation $||H_{i^*}||_{TV}=...$\n\nQUESTIONS:\nI only have one question about the model. It seems that the algorithms in the paper crucially rely on estimating the semi-distance defined based on the Scheffé set. But I believe in many applications it would be very hard to check whether an example falls into the Scheffé set of two hypothesises. For example, many learning algorithms might first improperly learn a list of candidate hypothesises and use hypothesis selection to select a good one. Computing pdf of these candidate hypothesises may not be realistic in general. Would it be possible to relax the dependence on the knowledge of the pdf of the candidate hypothesises in the model?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces proper approximation algorithms for optimal hypothesis selection in the context of distribution learning.\nThe first algorithm achieves an optimal approximation factor of $\\alpha=3$ in time approximately linear in the number of hypotheses. The second achieves the slightly looser $\\alpha = 4$ approximation factor in exchange for significant reduced dependence \non $\\epsilon$ and $\\delta$, the standard learning parameters governing the error and confidence of the learning algorithms. The paper focuses on explanation of the algorithmic techniques involved.\n\nSTRENGTHS:\nThe paper introduces a novel learning algorithm for optimal approximation (Algorithm 1) which achieves a significantly reduced dependence (quadratic to near-linear) on the number of hypotheses in the hypothesis class in the computational complexity. This is notable given that the problem has been studied for a while, and practical, given that distribution learning over a discrete space requires a number of samples at least proportional to the cardinality of the domain on which the hypotheses are supported.\n\nWEAKNESSES:\nI found the writing to be unpolished, and not publication-ready. Much of the body attempts to furnish intuition to the algorithmic techniques introduced, but some portions seem underdeveloped, and occasionally read like direct translations from math to spoken language (e.g. line 358). For example, an extremely important concept in this paper (introduced in important prior works) is that of ``semi-distances'' $w_i(H_j)$. Seemingly the best intuition the reader is provided with regarding this concept comes at line 176: \"One suggestion for readers to internalize the semi-distances is to view them as a distance between $H_i$ and $P$ that is measured from the perspective of $H_j$\". I think this deserves more illuminating wording, and personally got a much better understanding by staring longer at the definition. Overall, I think the lack of attention to writing is somewhat a shame, as there seems to be a lot of nice algorithmic thought here which deserves a better exposition in my opinion.\n\nOn a more technical level, the dependence in the computational complexity of Algorithm 1 on the learning parameters $\\epsilon$ and $\\delta$ is heavy -- these enter as $1/\\delta^3\\epsilon^3$. One would really hope for something like $\\log(1/\\delta)polylog(1/\\epsilon)/\\epsilon^2$.  It seems there will be many regimes in which the guarantees of Algorithm 1 -- despite the linear dependence on the number of hypotheses -- will be looser than the quadratic algorithm of MS08.\n\nQUESTIONS:\n-Table 1: Is the dependence on $1/\\delta$ in $s$ just the standard $\\log(1/\\delta)$?  \n\n-line 180: The order of logical quanitifiers can be reversed to get a slightly stronger and more illustrative statement here, correct?  \n \n-line 240: It seems that to do this random sampling to decide if you have a prompting hypothesis, you need to check a number of hypotheses which is proportional to $1/\\delta^2$. I guess there is a naiver version of this algorithm which just checks all of the hypotheses at this step, incurring some quadratic dependence in $n$. Am I wrong in feeling like any sort of search for something like a promping hypotheses will always incur some sort of undesirable multiplicative interaction with $n$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper looks at proper distribution learning: given samples from
some distribution p, and a set of n candidate distributions H_i,
output one H_i that is close in TV; in particular, alpha * OPT + eps.
Surprisingly, this is possible with a sample complexity independent of
the domain size (as would be needed to actually estimate the TV).
There's been a long line of work aiming to improve the approximation
factor alpha.  alpha = 2 is possible for *improper* learners, but
proper learners can only hope for alpha = 3.  Getting alpha=3 in n^2 s
time is known; this paper gets that down to O~(ns) time (although an
extra eps factor in time, which they can avoid for alpha=4, and worse
delta dependence).

STRENGTHS:
It gives much better running time for a natural problem.

The approach is fairly clean.

WEAKNESSES:
The algorithm overview is a bit vague, and could be written more
clearly.

The failure probability dependence isn't good.

The alpha=3 approach is a fairly simple extension of prior work, and
alpha=4 isn't so exciting.

I'm not sure that the constant here matters for the applications of
hypothesis selection.  Like, in the application where we choose a
cover for the class, presumably we can just make a finer cover?

QUESTIONS:
Are there applications of hypothesis selection where the constant alpha matters?

LIMITATIONS:
Fine.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12