Title: Active Learning of General Halfspaces: Label Queries vs Membership Queries
Venue: NeurIPS 2024
Accepted: yes
Keywords: Active Learning

Membership Query

Linear Separator
Abstract: We study the problem of learning general (i.e., not necessarily homogeneous) 
halfspaces under the Gaussian distribution on $\mathbb{R}^d$ 
in the presence of some form of query access. 
In the classical pool-based active learning model, where the algorithm is
allowed to make adaptive label queries to previously sampled points, 
we establish a strong information-theoretic lower bound ruling out non-trivial
improvements over the passive setting. Specifically, we show that
any active learner requires label complexity of 
$\tilde{\Omega}(d/(\log(m)\epsilon))$, where $m$ is the number of unlabeled examples. 
Specifically, to beat the passive label complexity of $\tilde{O}(d/\epsilon)$, 
an active learner requires a pool of $2^{\mathrm{poly}(d)}$ unlabeled samples.
On the positive side, we show that this lower bound 
can be circumvented with membership query access, 
even in the agnostic model. Specifically, we give a computationally efficient 
learner with query complexity of $\tilde{O}(\min(1/p, 1/\epsilon) + d\mathrm{polylog}(1/\epsilon))$
achieving error guarantee of $O(\mathrm{opt}+\epsilon)$. Here $p \in [0, 1/2]$ 
is the bias and $\mathrm{opt}$ is the 0-1 loss of the optimal halfspace. 
As a corollary, we obtain a strong separation 
between the active and membership query models. 
Taken together, our results characterize the complexity of learning 
general halfspaces under Gaussian marginals in these models.

Active Learning of General Halfspaces: Label Queries vs Membership Queries

Abstract

We study the problem of learning general (i.e., not necessarily homogeneous) halfspaces under the Gaussian distribution on R d in the presence of some form of query access. In the classical pool-based active learning model, where the algorithm is allowed to make adaptive label queries to previously sampled points, we establish a strong information-theoretic lower bound ruling out non-trivial improvements over the passive setting. Specifically, we show that any active learner requires label complexity of ˜ Ω( d/ (log( m ) ϵ )) , where m is the number of unlabeled examples. Specifically, to beat the passive label complexity of ˜ O ( d/ϵ ) , an active learner requires a pool of 2 poly( d ) unlabeled samples. On the positive side, we show that this lower bound can be circumvented with membership query access, even in the agnostic model. Specifically, we give a computationally efficient learner with query complexity of ˜ O (min { 1 /p, 1 /ϵ } + d polylog(1 /ϵ )) achieving error guarantee of O (opt + ϵ ) . Here p ∈ [0 , 1 / 2] is the bias and opt is the 0-1 loss of the optimal halfspace. As a corollary, we obtain a strong separation between the active and membership query models. Taken together, our results characterize the complexity of learning general halfspaces under Gaussian marginals in these models.

1 Introduction

In Valiant's PAC learning model [Val84a, Val84b], the learner is given access to random labeled examples and aims to find an accurate approximation to the function that generated the labels. The standard PAC model is 'passive' in the sense that the learner has no control over the selection of the training set. Here we focus on interactive learning between a learner and a domain expert that can potentially lead to significantly more efficient learning procedures. A standard such paradigm is (pool-based) active learning [MN + 98], where the learner has access to a large pool of unlabeled examples S and has the ability to (adaptively) select a subset of S and obtain their labels. We will henceforth refer to this type of data access as label query access. An even stronger interactive model is that of PAC learning with membership queries [Ang88, Fel09]. A membership query (MQ) allows the learner to obtain the value of the target function on any desired point in the support of the marginal distribution. This model captures the ability to perform experiments or the availability of expert advice. While in active learning, the learner is allowed to query the labels of previously sampled points from S , in MQ learning the learner has black-box access to the target function. We refer the reader to Appendix A for formal definitions of these two learning models. Roughly speaking, when

the size of S becomes exponentially large (so that it is a good cover of the space), the model of active learning 'converges' to the model of learning with MQs. This intuitive connection will be useful in the proceeding discussion.

Active learning is motivated by the availability of large amounts of unlabeled data at low cost. As such, the typical goal in this model is to develop algorithms with qualitatively improved label complexity (compared to passive learning) at the expense of a larger - but, ideally, still reasonably bounded set of unlabeled data. Over the past two decades, a large body of work in theoretical machine learning has studied the possibilities and limitations of active learning in a variety of natural and important settings; see, e.g. [FSST97, Das04, Das05, DKM05, BBZ07, BHV10, H + 14, HY15, DMRT24 ? ].

A prototypical setting where active learning leads to substantial savings is for the task of learning homogeneous Linear Threshold Functions (LTFs) or halfspaces. An LTF is any function h : R d → {± 1 } of the form h ( x ) = sign( w · x + t ) , where w ∈ S d -1 is called the weight vector and t is called the threshold. If t = 0 , the halfspace is called homogeneous. The problem of learning halfspaces is one the classical problems in machine learning, going back to the Perceptron algorithm [Ros58] and has had a great impact on many other influential techniques, including SVMs [Vap97] and AdaBoost [FS97].

For the class of homogeneous halfspaces under well-behaved distributions (including the Gaussian and isotropic log-concave distributions), prior work has established that O ( d log(1 /ϵ )) label queries suffice, where d is the dimension and ϵ is the desired accuracy [BBZ07, DKM05, BL13]. Moreover, there are computationally efficient algorithms with near-optimal label complexity for this task [ABL17, YZ17, She21], even in the agnostic model that achieve error O (opt+ ϵ ) . Unfortunately, this logarithmic dependence on 1 /ϵ breaks down for general (potentially biased) halfspaces. Intuitively, this holds because if the bias of a halfspace (the probability mass of the small class) is p , then we need to obtain at least 1 /p labeled examples before we see the first point in the small class. This implies an information-theoretic label complexity lower bound of Ω(min { 1 /p, 1 /ϵ } + d log(1 /ϵ )) [Das05], even for realizable PAC learning under the uniform distribution on the sphere. Hanneke et al. [BHV10] showed an information-theoretic label complexity upper bound of ˜ O ((1 /p ) d 3 / 2 log(1 /ϵ )) for general halfspaces under the uniform distribution on the sphere (via an exponential-time algorithm).

In summary, prior to this work, the possibility that there is an active learner with label complexity O ( d log(1 /ϵ ) + min { 1 /p, 1 /ϵ } ) and unlabeled sample complexity poly( d/ϵ ) remained open. Our first main result is an information-theoretic lower bound ruling out this possibility.

Theorem 1.1 (Main Lower Bound) . For any active learning algorithm A , there is a halfspace h ∗ that labels S with bias p such that if A makes less than ˜ O ( d/ ( p log( m ))) label queries over S , a set of m i.i.d. points drawn from N (0 , I ) , then with probability at least 2 / 3 the halfspace ˆ h output by A has error more than p/ 2 with respect to h ∗ .

In particular, if p is chosen as Θ( ϵ log(1 /ϵ )) , learning a p -bias halfspace with error Cϵ (for any fixed constant C ) would require a learning algorithm to either make ˜ Ω( d/ϵ ) label queries or have a pool of 2 d unlabeled examples. Our information-theoretic lower bound essentially shows that the active setting does not provide non-trivial advantages for the class of general halfspaces, unless the learner is allowed to obtain exponentially many unlabeled examples. (As already mentioned, in this extreme setting, the active learning model approximates PAC learning with MQs.) This motivates the study of learning halfspaces in the stronger model with MQs, where better upper bounds may be attainable.

To circumvent the aforementioned lower bound, we consider the stronger model of PAC learning with MQs. We are interested in understanding the query complexity of learning general halfspaces under the Gaussian distribution. We study this question in the agnostic learning model and establish the following positive result, the proof of which can be found in Appendix G:

Theorem 1.2 (Main Algorithmic Result) . Consider the problem of agnostic PAC learning halfspaces with membership queries under the Gaussian distribution. There is an algorithm such that for every labeling function y ( x ) and for every ϵ, δ ∈ (0 , 1) , it makes M = ˜ O δ (min { 1 /p, 1 /ϵ } + d polylog(1 /ϵ )) * memberships queries, runs in poly( d, M ) time, where p is the bias of the optimal halfspace h ∗ , and outputs an ˆ h ∈ H such that with probability at least 1 -δ , err( ˆ h ) ≤ O (opt + ϵ ) .

In other words, we provide a computationally efficient constant factor agnostic query learner with query complexity ˜ O (min { 1 /p, 1 /ϵ } + d polylog(1 /ϵ )) . Due to known d poly(1 /ϵ ) complexity lower bounds for achieving optimal error of opt + ϵ [DKZ20, DKPZ21, DKR23], the majority of work [DKS18, DKTZ22] in the passive PAC model has focused on designing efficient learners achieving a constant factor approximation of O (opt+ ϵ ) These passive learning algorithms have sample complexity poly( d, 1 /ϵ ) . Note that, by Theorem 1.1, it is impossible to modify these algorithms (for general halfspaces) to achieve an active learner with low label complexity.

In the realizable setting under the Gaussian distribution, a learner may query many points that are extremely far from the origin to find examples from the small class with few queries. However, such an algorithm is quite fragile to even a tiny amount of noise. In particular, the query complexity achieved by our algorithm establishing Theorem 1.2 is nearly optimal in the agnostic setting.

On the one hand, Ω( d log(1 /ϵ )) queries are required because describing a halfspace up to error ϵ requires d log(1 /ϵ ) bits of information [KMT93]. On the other hand, we argue that the overhead term of Ω(min { 1 /p, 1 /ϵ } ) cannot be avoided in the agnostic setting. Such a statement can be deduced from a lower bound of [HKL20]: they showed that in the realizable setting, any algorithm requires at least Ω((1 /p ) 1 -o (1) ) MQs to see the first example from the small class (where p is the bias of the target halfspace with respect to the uniform distribution on the unit ball); they also showed a similar lower bound of Ω(1 /p ) if the underlying distribution is the uniform distribution over the unit sphere. As the dimension d increases, the standard Gaussian distribution is very well approximated by the uniform distribution over a d -dimensional sphere with radius ∼ √ d . Thus, an exponentially small level of noise would make every query far from this sphere contain no useful information. This allows us to show that, under the Gaussian distribution with a tiny amount of label noise, Ω((1 /p ) 1 -o (1) ) queries are needed to see a single example from the small class. The proof of this statement is essentially identical to the argument in [HKL20] for unit ball. The reader is referred to that work for the details.

1.1 Preliminaries

For a halfspace h ( x ) = sign( w · x + t ) , w ∈ S d -1 , t > 0 , we use p ( t ) = Pr x ∼ N (0 ,I ) ( h ( x ) = -1) to denote its bias. For a halfspace h ( x ) , we define its Chow-Parameter under the standard Gaussian distribution to be E x ∼ N (0 ,I ) xh ( x ) . Let y ( x ) : R d →{± 1 } be a (randomized) labeling function for examples in R d . We denote by err( h ) = Pr x ∼ N (0 ,I ) ( h ( x ) = y ( x )) to be the error of the hypothesis h and opt = min h ∈ H err( h ) , where H is the class of halfspaces over R d . We will use h ∗ to denote the halfspace with an error equal to opt . When there is no confusion, we will use p to denote the bias of the optimal halfspace h ∗ .

̸

Let D x be a distribution over R d , y ( x ) be a labeling function over R d and S = { ( x i , y ( x i )) } i m =1 be a set of i.i.d. examples drawn from the distribution D over R d ×{± 1 } such that the marginal distribution of D is D x . A membership query takes an x in the support of D x as input and outputs y ( x ) . A label query takes an x i , where ( x i , y ( x i )) ∈ S as input and outputs y ( x i ) . A learning algorithm A is allowed to use membership queries/label queries and aims to output a halfspace hypothesis ˆ h such that err( ˆ h ) ≤ O (opt + ϵ ) by making as few queries as possible.

2 Nearly-Tight Lower Bound on Label Complexity: Proof of Theorem 1.1

In this section, we prove our information-theoretic lower bound on the label complexity of active learning general halfspaces under the Gaussian distribution.

Before presenting our proof, we provide high-level intuition behind Theorem 1.1 and the strategy of our proof. Previous work, see, e.g. [Das04, DKM05, HKL20], showed that if S is a set of examples drawn uniformly from the unit sphere, and if h ∗ is a halfspace with bias p that is chosen uniformly, the following holds: no matter which query strategy a learning algorithm A uses, for the first r queries, in expectation only pr of them fall into the small cap on the sphere cut by h ∗ . Thus, if A makes less than 1 / (2 p ) queries, it will with constant probability not see any negative examples; and it is therefore impossible to learn the target halfspace.

In the Gaussian case, we will use a similar but stronger idea. If we are able to learn h ∗ up to error p/ 2 with few queries, then we can randomly partition S into two sets, use the first set to learn the halfspace and use the second part to find d negative examples by paying another O ( d ) queries in expectation. Formally, we have the following statement, the proof of which can be found in Appendix B.1.

Lemma 2.1. Suppose there is an active learning algorithm that can make r label queries over a pool S of m ≥ poly( d/p ) examples drawn from N (0 , I ) and learn any halfspace h ∗ ( x ) = sign( w ∗ · x + t ∗ ) with bias p up to error p/ 2 with probability at least 2 / 3 . Then there is an algorithm such that given a pool of 2 m random examples S drawn from the standard Gaussian distribution with hidden labels by some halfspace h ∗ ( x ) = sign( w ∗ · x + t ∗ ) with bias p , it makes r + O ( d ) queries and finds d negative examples from S with probability 1 / 2 .

We will show that finding d negative examples from S requires many queries. The idea is that since S is sampled from a standard Gaussian in high dimensions, every pair of examples is almost orthogonal unless m is as large as 2 d . If we have made 1 /ϵ queries over S and found our first negative example, then this negative example will only provide us with very little knowledge to find the next negative example - as no example in the pool has a large correlation with it. Therefore, it will still take us another approximately 1 /ϵ queries to find the next negative example. Such an issue only disappears after we have already found roughly d negative examples; at which time, the average of the d examples has a good correlation with w ∗ . Therefore, it would take us roughly d/ϵ queries in total. We remark that such an argument is hard to formalize, because, besides negative examples, the algorithm has also seen many positive examples in the process. It is thus challenging to argue that the algorithm cannot make good use of the information obtained from these positive examples.

To overcome this difficulty, our proof strategy works as follows. Each algorithm A can be described as a decision tree. Each tree node represents the example queried in a given round. Every time the algorithm sees a negative example, it moves to the left; otherwise, it moves to the right. Suppose that A wants to find k negative examples with r queries. Then there are at most ( r k ) ≤ ( er/k ) k paths of the tree, where A successfully finds k negative examples, and for each of the paths there are exactly k examples that are negative upon queried. For a k -tuple of examples, we will derive a deterministic condition such that if the k examples satisfy the condition, a random halfspace with bias p will have only roughly p k probability to label all of the k examples negative.

Formally, we establish the following technical lemma (see Appendix B.2 for the proof).

Lemma 2.2. Let A ∈ R k × d be a matrix with row vectors x 1 , . . . , x k . Let t ∗ > C > 0 for some sufficiently large constant C . Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a random halfspace with bias p with w ∗ ∼ S d -1 chosen uniformly from S d -1 . If ∥ ∥ AA ⊤ -dI ∥ ∥ 2 ≤ O ( d/ ( t ∗ ) 2 ) , then with probability at most O ( p log(1 /p )) k , where p is the bias of h ∗ under the Gaussian distribution, h ∗ ( x i ) = -1 for i = 1 , . . . , k .

Thus, if the ( r k ) tuples all satisfy such a condition, then A will succeed with a fairly tiny probability unless r is larger than k/p . So, in the last step of the proof, we will show in Lemma 2.3 that by taking k ≈ d/ (log( m )polylog(1 /p )) , with high probability every k -tuple of examples in S will satisfy the deterministic condition. Thus, no algorithm can succeed with a constant probability, unless it makes ˜ Ω( d/ ( p log( m ))) queries. The proof of Lemma 2.3 can be found in Appendix B.3.

Lemma 2.3. Let S ⊆ R d be a set of m examples drawn i.i.d. from N (0 , I ) . Let t ∗ > C > 0 for a sufficiently large constant C and k = O ( d/ log( m )( t ∗ ) 4 ) . Then, with probability at least 2 / 3 , for every k -tuple of examples { x 1 , . . . , x k } ⊆ S , ∥ ∥ AA ⊤ -dI ∥ ∥ 2 ≤ d/ ( t ∗ ) 2 , where A ∈ R k × d be a matrix with row vectors x 1 , . . . , x k .

Proof of Theorem 1.1. We will start by showing that, given a set S of m points drawn i.i.d. from a Gaussian distribution, the following holds. With probability at least 2 / 3 , for every algorithm A there exists a halfspace h ∗ = sign( w ∗ · x + t ∗ ) with bias p such that if A makes only r = ˜ O ( d/p log( m )) label queries over S , then with probability at least 2 / 3 it will not be able to find k negative examples in S for some k ≤ d . By Yao's minimax principle, it is sufficient to show that there is a distribution over halfspaces h ∗ such that for any deterministic active learning algorithm, the following holds: given m random Gaussian examples, if the learning algorithm makes r queries, with probability 2 / 3 it cannot find k negative examples. We will fix the threshold t ∗ of h ∗ and draw w ∗ uniformly from the unit sphere.

By Lemma 2.3, we know that by choosing k = O ( d/ log( m )( t ∗ ) 4 ) , with probability at least 2 / 3 , for every k -tuple of examples x 1 , . . . , x k ∈ S , ∥ ∥ AA ⊤ -dI ∥ ∥ 2 ≤ d/ ( t ∗ ) 2 , where A ∈ R k × d is a matrix with row vectors x 1 , . . . , x k . By Lemma 2.2, we know that every k -tuple of examples x 1 , . . . , x k ∈ S has a probability α k , which is at most O ( p log p ) k to be labeled all negative by the random halfspace h ∗ . Notice that every query algorithm can be expressed as a binary tree T . Each node of the tree represents an example where the algorithm makes queries at a time. If the example at node v is negative, then the algorithm will query the left child of v , and otherwise it will query the right child of v . The algorithm stops making queries when either it has queried r examples or it has queried k negative examples. In particular, for a given search algorithm, there are at most ( r k ) different possible outcomes where it successfully finds k negative examples. Furthermore, for each of the possible outcomes, there is a set of k examples in S that correspond to the k negative examples the algorithm finds. Thus, the probability that the algorithm successfully finds k negative examples is bounded above by the probability that there exists one of the ( r k ) k -tuples of examples in S that are all labeled negative by h ∗ . Such a probability can be bounded above by

if r ≤ O ( k/p log(1 /p )) = O ( d/ ( p log( m )polylog(1 /p )) . By Lemma 2.1, we know that if we can make O ( d/ ( p log( m )polylog(1 /p )) label queries to learn a p -biased halfspace up to error p/ 2 over a set S of m/ 2 Gaussian examples, then we can use O ( d/ ( p log( m )polylog(1 /p )) queries to find d negative examples among m Gaussian points. This leads to a contradiction. Thus, the label complexity of the learning problem is ˜ Ω( d/ ( p log( m ))) , as desired.

3 Robust Learning of General Halfspaces with MQs: Proof of Theorem 1.2

In this section, we present our main algorithmic result, Theorem 1.2. We refer the readers to Appendix G for the full proof of Theorem 1.2. Throughout the paper, we will assume for convenience that the noise level opt ≤ ϵ . Such an assumption can be made without loss of generality, as discussed in Appendix C.1. We first present our main algorithm, Algorithm 1. Algorithm 1 will maintain a list of polylog(1 /ϵ ) candidate hypotheses at least one of which has error O (opt + ϵ ) . We will then use a standard tournament approach to find an accurate hypothesis among them.

Algorithm 1 QUERY LEARNING HALFSPACE(Efficient Agnostic Learning Halfspaces with Queries)

At the beginning of Algorithm 1, we will use random queries to approximately estimate the bias p of the optimal halfspace up to a constant factor. As we will discuss in Appendix C.2, such an estimation can be done with only ˜ O (min { 1 /p, 1 /ϵ } ) queries by applying a doubling trick to the coin estimation problem. In particular, if we find p < Cϵ , we can directly output a constant hypothesis as it has

error only O ( ϵ ) . Since t ∗ is unknown to us, such an approach can prevent us from using some t ′ which is much larger than t ∗ in the rest of the learning procedure, which will potentially lead to a larger query complexity. With such a ˆ p , t ∗ will fall into a reasonable range [ t a , t b ] . We next partition [ t a , t b ] into a grid of size O (1 / log(1 /ϵ )) and use each of the grid points as an initial guess of t ∗ . In particular, at least one of these grid points t j is O (1 / log(1 /ϵ )) close to t ∗ . Although such a t j is not accurate enough to be used in the final output hypothesis, as t ∗ ≤ √ log(1 /ϵ ) , we will show later that such a t j is enough for us to use it to learn w ∗ , t ∗ accurately. Suppose now we have such a good t j . We will design two subroutines that make use of t j to produce a good hypothesis sign( w T · x + ˆ t ) . The first algorithm will take t j and the noise level ϵ as its input and produce a unit vector w 0 as an initialization. We will show in Section 3.2 that as long as | t j -t ∗ | ≤ 1 / log(1 /ϵ ) , we can with probability at least 1 / log(1 /ϵ ) produce some w 0 such that θ ( w 0 , w ∗ ) ≤ O (1 /t j ) . By repeating such an initialization algorithm polylog(1 /ϵ ) times, with high probability one of these runs will succeed. In particular, such an algorithm has a query complexity of ˜ O (1 /p + d polylog(1 /ϵ )) . Now assume we have such a w 0 as a warm-start. Our second subroutine is to refine the direction w 0 and the threshold t j . More specifically, we will maintain a unit vector w i such that θ i = θ ( w i , w ∗ ) and an upper bound σ i for sin( θ i / 2) . In each round of the refining algorithm, we will use ˜ O ( d ) queries to update w i . In particular, in each round σ i will decrease by a constant factor and thus after at most T = ˜ O (log(1 /ϵ )) rounds, we will have sin( θ T / 2) ≤ σ T = Cϵ exp( t 2 j / 2) . As we will show in Section 3.1, provided the correct t ∗ , sign( w T · x + t ∗ ) is at most O ( ϵ ) far from h ∗ . However, to output a good hypothesis, we still need to learn t ∗ up to a high accuracy. When t ∗ is small, we even have to estimate t ∗ up to error O ( ϵ ) , which typically needs many queries. However, as we will show in Section 3.1, given w T close enough to w ∗ , we are able to combine the localization technique used in [DKS18] with this fact to learn t ∗ using only O (log(1 /ϵ )) queries. This gives an overview of Algorithm 1 and its query complexity.

3.1 Refining A Warm-Start

We will start by discussing how to refine a warm start w 0 by proving the following theorem. The proof of the theorem and the main algorithm, Algorithm 3 can be found in Appendix D.5.

Theorem 3.1. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace such that err( h ∗ ) = opt ≤ ϵ . Let t ′ ≤ √ log(1 /ϵ ) , w 0 ∈ S d -1 be inputs of Algorithm 3. If t ′ -1 / log(1 /ϵ ) ≤ t ∗ ≤ t ′ , t ′ exp(( t ′ ) 2 / 2) ≤ 1 / ( Cϵ ) and sin( θ ( w 0 , w ∗ ) / 2) ≤ σ 0 := min { 1 /t ′ , 1 / 2 } , then Algorithm 3 makes M = ˜ O δ ( d polylog(1 /ϵ )) membership queries, runs in poly( d, M ) time, and outputs ( w T , ˆ t ) such that with probability at least 1 -O ( δ ) , err(sign( w T · x + ˆ t )) ≤ O ( ϵ ) .

As we discussed in Section 3, we will assume we have some t ′ such that t ′ -1 / log(1 /ϵ ) ≤ t ∗ ≤ t ′ and some w 0 such that sin( θ 0 / 2) ≤ σ 0 = min { 1 /t ′ , 1 / 2 } , i.e., some initial knowledge of t ∗ , w ∗ . Our algorithm runs in iterations and will maintain some w i in round i . We will maintain some unit vector w i and use ∥ w i -w ∗ ∥ = 2sin( θ i / 2) to measure the progress made by Algorithm 3. The method we use to update w i is a simple projected gradient descent algorithm. Specifically, we will construct a random vector G i over R d such that G i ⊥ w i and in expectation g i = E G i has bounded length and a good correlation with respect to w ∗ . We will show in the following lemma that by estimating E G i up to constant error with ˆ g i and using the update rule w i +1 = proj S d -1 ( w i + µ i ˆ g i ) , we are able to significantly decrease θ i . The proof of Lemma 3.2 can be found in Appendix D.1.

Lemma 3.2. Let w ∗ , w i ∈ S d -1 such that w ∗ = a i w i + b i u , where u ∈ S d -1 , u ⊥ w i , a i , b i > 0 , a 2 i + b 2 i = 1 . Let θ i = θ ( w i , w ∗ ) . Let G i be a random vector drawn from some distribution D such that with probability 1 , G i ⊥ w i . Let g i be the mean of G i . Let ˆ g i be the empirical mean of G i and µ i > 0 . The update rule w i +1 = proj S d -1 ( w i + µ i ˆ g i ) satisfies the following property,

Furthermore, if sin( θ i / 2) ≤ σ i ∈ (0 , 1) and there exist constant c 1 , c 2 such that g i · u ≥ c 1 / 10 , ∥ ˆ g i ∥ ≤ c 1 and ∥ g i -ˆ g i ∥ ≤ c 2 ≤ c 1 / 40 , then there exist constant C 1 , C 2 > 8 such that by taking µ i = σ i /C 1 and σ i +1 = (1 -1 /C 2 ) σ i , it holds that sin ( θ i +1 / 2) ≤ σ i +1 . In particular, if sin( θ i / 2) ≤ 3 σ i / 4 and ∥ ˆ g i ∥ ≤ c 1 then sin ( θ i +1 / 2) ≤ σ i +1 always holds.

In the rest of the section, we will show that as long as w i is not good enough, we can always efficiently construct a random vector G i whose expectation points to the correct direction and we can use very

few queries to estimate its expectation up to a desired accuracy. We adapt the localization technique used in [DKS18] to achieve this goal.

3.1.1 Finding A Good Gradient via Localization

In the i -th round of Algorithm 3, we write w ∗ = a i w i + b i u i , where u i ∈ S d -1 , u i ⊥ w i , a i , b i > 0 , a 2 i + b 2 i = 1 . Recall that σ i is an upper bound we maintain for sin( θ i / 2) . We will construct the random gradient as follows

where z ∼ N (0 , I ) , A i = I -(1 -σ 2 i ) w i w t i and ˜ t ∈ (0 , t ′ ) is a scalar. To see why G i is a good choice, we will start by analyzing G i assuming the noise rate opt = 0 . To simplify the notation, denote by ℓ i ( z ) = sign(( a i w i + b i u i /σ i ) z +( t ∗ -a ˜ t ) /σ i ) and ¯ g i = E z ∈ N (0 ,I ) proj w ⊥ i zℓ i ( z ) . A simple calculation gives us the following result.

Fact 3.3. Let h ( x ) = sign( w · x + t ) be a halfspace. Let v ∈ S d -1 such that w = av + bu , where a, b > 0 , a 2 + b 2 = 1 , u ∈ S d -1 , u ⊥ v . Let s, σ > 0 be real numbers and define A = I -(1 -σ 2 ) vv t . For each z ∈ R d , define ˜ z := A 1 / 2 z -sv . Then h (˜ z ) = ℓ ( z ) , where ℓ is the following halfspace

Fact 3.3 implies that if opt = 0 , then it always holds that f i ( z ) := y ( A 1 / 2 i z -˜ tw i ) = ℓ i ( z ) , ∀ z ∈ R d and we can view z as examples labeled by a halfspace ℓ i ( z ) . In particular, E z ∼ N (0 ,I ) zf i ( z ) is the Chow-Parameter of the halfspace ℓ i ( z ) under the standard Gaussian distribution.

Fact 3.4 (Lemma C.3 in [DKS18]) . Let h ( x ) = sign( w · x + t ) , where w ∈ S d -1 be a halfspace. Then E z ∼ N (0 ,I ) zh ( z ) = √ 2 π exp( -t 2 / 2) w .

By Fact 3.4, in the noiseless case, E z ∼ N (0 ,I ) zf i ( z ) is parallel to ( a i w i + b i u i /σ i ) with length Θ(exp( -T 2 i )) , where T i = t ∗ -a i ˜ t σ i √ a 2 i + b 2 i /σ 2 i and g i = ¯ g i is exactly the u i component of the ChowParameter. In particular, if T i is constant, then by estimating g i using ˆ g i up to a small constant error using ˜ O ( d ) queries, we are able to use Lemma 3.2 to improve w i . Assuming we set ˜ t = t ∗ , as σ i t ′ ≤ 1 and b i ≤ O ( σ i ) , it is easy to check T i can be bounded by some universal constant. However, as we mentioned before, we only know | t ′ -t ∗ | ≤ 1 log(1 /ϵ ) , when w i getting close to w ∗ , σ i could become very small and an error of 1 / log(1 /ϵ ) could potentially blow up T i , making the signal we want quite small. Such an issue is problematic for the algorithm, especially when f i ( z ) is a noisy version of ℓ i ( z ) . To overcome such an issue, we prove the following structural lemma in Appendix D.2 showing that we can always check whether the choice of ˜ t is good or not, by looking at the bias of ℓ ( z ) , using ˜ O (1) queries. Using this method, we can perform a binary search for ˜ t to find a correct choice in at most log(1 /ϵ ) rounds. Furthermore, as long as we select the correct ˜ t , it must hold that ∣ ∣ ˜ t -t ∗ ∣ ∣ ≤ O ( σ i ) . In particular, as σ T = Cϵ exp(( t ′ ) 2 / 2) , such a ˜ t is a good enough estimate for t ∗ to be used in the final hypothesis.

Lemma 3.5. Let w ∗ , w i ∈ S d -1 such that w ∗ = a i w i + b i u i , where u i ∈ S d -1 , u ⊥ w i , a i , b i > 0 , a 2 i + b 2 i = 1 . Let t ∗ , t ′ , σ i , ϵ be positive real numbers such that 0 ≤ t ∗ ≤ t ′ , sin( θ i / 2) ≤ σ i , and σ i t ′ ≤ 1 . Define T i := t ∗ -a i ˜ t σ i √ a 2 i + b 2 i /σ 2 i , ℓ i ( z ) = sign(( a i w i + b i u i /σ i ) z + ( t ∗ -a ˜ t ) /σ i ) and ¯ g i = E z ∈ N (0 ,I ) proj w ⊥ i zℓ i ( z ) for some ˜ t ∈ [0 , t ′ ] . Then the following three properties hold.

There exists an interval I t ′ ⊆ [0 , t ′ ] of length at least σ i such that for every ˜ t ∈ I t ′ , | T i | ≤ 5 .

When | T i | ≤ 6 , it holds that ¯ g i · u i = ∥ ¯ g i ∥ and e -19 b i /σ i ≤ ∥ ¯ g i ∥ ≤ 2 e -19 .

For every ∣ ∣ ˜ t -t ∗ ∣ ∣ > 40 σ i and ˜ t < t ′ , | T i | > 10 .

3.1.2 Robustness Analysis

So far, we have only considered the case when opt = 0 . Due to the presence of noise, it is impossible for us to estimate ¯ g i = E z ∈ N (0 ,I ) proj w ⊥ i zℓ i ( z ) because we only have a noisy version f i ( z ) of

̸

ℓ i ( z ) . In this section, we will show that as long as w i is close to w ∗ and | t ′ -t ∗ | ≤ 1 / log(1 /ϵ ) , the probability that for a Gaussian point z , ℓ i ( z ) = f i ( z ) is at most a tiny constant. This is incomparable with the bias of ℓ z ( z ) if ˜ t is chosen correctly, and does not affect the algorithm too much. We start with the following lemma which bounds the probability of ℓ i ( z ) = f i ( z ) .

̸

̸

Lemma 3.6. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace such that err( h ∗ ) = opt ≤ ϵ . Let ˜ t, σ i , t ′ be real numbers such that ˜ t ≤ t ′ and σ i t ′ ≤ 1 , σ i ≤ 1 / 2 . Let w ∗ = a i w i + b i u i , where u i ∈ S d -1 , u ⊥ w i , a i , b i > 0 , a 2 i + b 2 i = 1 . Define ℓ i ( z ) = sign(( a i w i + b i u i /σ i ) z +( t ∗ -a ˜ t ) /σ i ) and f i ( z ) = y ( A 1 / 2 i z -˜ tw i ) . Then Pr z ∼ N (0 ,I ) ( ℓ i ( z ) = f i ( z )) ≤ ϵ exp( ˜ t 2 / 2+4) /σ i . In particular, if σ i ≥ C exp(( t ′ ) 2 / 2) ϵ , for some sufficient large constant C , then there is a sufficiently small constant c such that Pr z ∼ N (0 ,I ) ( ℓ i ( z ) = f i ( z )) ≤ c ≤ e -40 .

The proof of Lemma 3.6 leverages the ( v, s, σ ) - rejection procedure introduced in [DKS18] (see Appendix D.3). We will use Lemma 3.6 to analyze the gradient descent approach we described in the presence of noise. Formally, we establish the following lemma (see Appendix D.4 for the proof).

Lemma 3.7. Let w ∗ , w i ∈ S d -1 such that w ∗ = a i w i + b i u i , where u i ∈ S d -1 , u ⊥ w i , a i , b i > 0 , a 2 i + b 2 i = 1 . Let t ∗ , t ′ , σ i , ϵ be positive real numbers such that 0 ≤ t ∗ ≤ t ′ , sin( θ i / 2) ≤ σ i , σ i ≥ C exp(( t ′ ) 2 / 2) ϵ , and σ i t ′ ≤ 1 . Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace such that err( h ∗ ) = opt ≤ ϵ . Define T i := t ∗ -a i ˜ t σ i √ a 2 i + b 2 i /σ 2 i , ℓ i ( z ) = sign(( a i w i + b i u i /σ i ) z +( t ∗ -a ˜ t ) /σ i ) ,

̸

¯ g i = E z ∈ N (0 ,I ) proj w ⊥ i zℓ i ( z ) and g i = E z ∈ N (0 ,I ) proj w ⊥ i zf i ( z ) , where f i ( z ) = y ( A 1 / 2 i z -˜ tw i ) for some ˜ t ∈ [0 , t ′ ] . Let η i := Pr z ∼ N (0 ,I ) ( ℓ i ( z ) = f i ( z )) and p i be the probability that f i ( z ) = -1 . Then the following two properties hold.

Lemma 3.7 says as the noise level is small, it will not affect the structure lemma we established in Lemma 3.5 too much, and thus we are able to find the correct threshold ˜ t by checking the probability of f i ( z ) = -1 . Furthermore, as long as we choose the correct threshold ˜ t , g i , the noisy version of ¯ g i still satisfies the conditions in the statement of Lemma 3.2 and thus can be used to improve w i .

3.2 Finding A Good Initialization

In Section 3.1, we have shown that given some w 0 non-trivially close to w ∗ and some t ′ such that t ′ -1 log(1 /ϵ ) ≤ t ∗ ≤ t ′ , we can use Algorithm 3 to learn a good hypothesis with high probability. In this section, we show how to find such a good initialization w 0 using a few membership queries. The most common way to get such a warm-start is by robustly estimating the Chow-Parameter (see for example [She21, YZ17]) using Fact 3.4. Such an approach does not work for general halfspaces because the length of the length of the Chow-Parameter can be as small as ˜ O ( p ) , and thus needs roughly d/p random queries to estimate. In this section, we show how to overcome such an issue using a label smoothing technique, which has been useful in related problems [DKK + 23]. The main results in this step can be summarized as follows. The proof of Theorem 3.8 is deferred to Appendix E.2

Theorem 3.8. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) and y ( x ) be any labeling function such that err( h ∗ ) = opt ≤ ϵ ≤ 1 /C for some large enough constant C . If | t -t ∗ | ≤ 1 / log(1 /ϵ ) , then with probability at least 1 / 3 , Algorithm 2 makes M = ˜ O (1 /p + d log(1 /ϵ )) , runs in poly( d, M ) time, and outputs some w 0 such that sin( θ ( w 0 , w ∗ ) / 2) ≤ max { min { 1 /t, 1 / 2 } , O ( η √ log(1 /η ) } , where η = ϵ/p .

Due to the space limitations, here we only consider the case when t ∗ is not extremely large, which roughly covers the regime when η √ log(1 /η ) ≤ 1 /t . This suffices to capture some of the ideas and illustrate the power of the smoothed labeling. For the case when η √ log(1 /η ) > 1 /t , we are still able to find such a warm start by leveraging the smoothed label method in combination with the technique used in Section 3.1 in a more complicated way. We postpone this analysis to Appendix F. Our algorithm, Algorithm 2, to find a warm start is presented as follows.

To analyze Algorithm 2, we introduce the following definitions and notations.

̸

Definition 3.9 (Smoothed Label) . Let x ∈ R d be a point and y ( x ) be any labeling function. For ρ ∈ [0 , 1] , define the random variable ˜ x = √ 1 -ρ 2 x + ρz , where z ∼ N (0 , I ) . The smoothed label of x with parameter ρ is defined as ˜ y ( x ) := y (˜ x ) .

We will require the following fact (whose proof follows via a direct calculation):

Fact 3.10. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace. Let x, z ∈ R d and define ˜ x := √ 1 -ρ 2 x + ρz . Then ˜ h ( z ) := h ∗ (˜ x ) = sign( w ∗ · z +( t ∗ + √ 1 -ρ 2 w ∗ · x ) /ρ ) is another halfspace for z with threshold ( t ∗ + √ 1 -ρ 2 w ∗ · x ) /ρ .

̸

Let h ∗ = sign( w ∗ · x + t ∗ ) be an optimal halfspace and let y ( x ) be any labeling function such that err( h ∗ ) = opt ≤ ϵ . For x ∈ R d , we denote by η ( x ) := Pr ( h ∗ (˜ x ) = ˜ y ( x )) , the noise level of the smoothed label. Assuming that we are given a random negative example x 0 , then with constant probability, it is close to the decision boundary, i.e., w ∗ · x 0 ∈ ( -t ∗ -1 t ∗ , -t ∗ ) . This implies that the threshold of ¯ h , the halfspace corresponding to the smoothed label at x 0 , is between ( -1 , 1) . Moreover, the Chow-Parameter of ¯ h under the standard Gaussian distribution is parallel to w ∗ with a constant length, by Fact 3.4. If opt = 0 , then for every t ≤ √ log(1 /ϵ ) , we only need another ˜ O ( d log(1 /ϵ )) queries to estimate the Chow-Parameter of ¯ h up to error O (1 /t ) ; thus, we get a warm start w 0 such that sin( θ 0 / 2) ≤ 1 /t , given | t -t ∗ | is small. Therefore, the total number of queries we use to run Algorithm 2 is ˜ O (1 /p + d log(1 /p )) . However, in general, it is impossible to estimate w ∗ up to arbitrary accuracy - even using an infinite number of queries - because of the presence of noise. In fact, using a random x 0 is important for Algorithm 2 to succeed. If we are given some adversarially selected x 0 , even if it is close to the decision boundary, the above method can easily fail. This is because almost all the queries we made are in a small neighborhood of x 0 and could be corrupted by noise arbitrarily. However, we show in Appendix E.1 that, with a probability at least 2 / 3 , the noise level η ( x 0 ) of the smoothed label around x 0 is at most O ( ϵ/p ) , if x 0 is a random example given y ( x 0 ) = -1 ; and thus we can still estimate w ∗ to a desired accuracy provided ϵ/p is not too large.

Lemma 3.11. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace and y ( x ) be any labeling function such that err( h ∗ ) = opt ≤ ϵ . Let x ∼ N (0 , I ) conditioned on y ( x ) = -1 be a Gaussian example with a negative label. If p > Cϵ for some large enough constant C , then with probability at least 1 / 2 we have η ( x ) ≤ 5 ϵ/p and w ∗ · x ∈ ( -t ∗ -1 /t ∗ , -t ∗ ) .

Finally, we briefly discuss how to obtain a warm start when the threshold t ∗ is very large. The details of this method can be found in Appendix F. By Theorem 3.8, when p is small, we are only able to get some w 0 such that sin( θ ( w 0 , w ∗ )) ≤ O ( η √ log(1 /η )) for η = ϵ/p . One possible approach is to use the localization technique we use in Section 3.1 to refine such w 0 . However, such an approach fails because after localization the noise rate would be possibly larger than the length of the Chow-Parameter that we want to estimate. This makes it impossible for us to learn the useful signal. On the other hand, [DKS18] gave a randomized localization method that can make the expected noise level sufficiently smaller than the length of the Chow-Parameter we want to estimate; and thus will succeed with constant probability in each round of refinement. Unfortinately, such an approach cannot be used in a query-efficient manner, because to implement such a method we need to know θ ( w i , w ∗ ) up to an error 1 / log(1 /ϵ ) , in each round of refinement. This implies that if we make a random guess of θ ( w i , w ∗ ) , the probability of success in each round drops to only 1 / log(1 /ϵ ) , which requires to rerun the whole algorithm too many times in order to succeed once.

Such an issue could be addressed in a similar but more complicated way to the method we use in Lemma 3.5, by looking at the bias of the halfspace after localization. The second issue is that even the noise level is smaller than the length of the Chow-Parameter we want to estimate, the length of the Chow-Parameter is only 1 /p c , for some small constant c , as we can only make θ 0 smaller than some small constant. This still requires us to use d/p c queries to estimate it. Such an issue can again be addressed using the smoothed label method, where we use only 1 /p c queries to search a small class example and use another ˜ O ( d ) queries to estimate the Chow-Parameter. Importantly, even such a method only succeeds with constant probability overall. As the refinement stage only runs for O (log log(1 /ϵ )) rounds, we only need to rerun the entire algorithm O (log(1 /ϵ )) times to succeed once.

References

[ABL17] Pranjal Awasthi, Maria Florina Balcan, and Philip M Long. The power of localization for efficiently learning linear separators with noise. Journal of the ACM (JACM) , 63(6):1-27, 2017.

[Ang88] Dana Angluin. Queries and concept learning. Machine learning , 2:319-342, 1988.

[BBZ07] Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory , pages 35-50. Springer, 2007.

[BHV10] Maria-Florina Balcan, Steve Hanneke, and Jennifer Wortman Vaughan. The true sample complexity of active learning. Machine learning , 80:111-139, 2010.

[BL13] Maria-Florina Balcan and Phil Long. Active and passive learning of linear separators under log-concave distributions. In Conference on Learning Theory , pages 288-316. PMLR, 2013.

[Das04] Sanjoy Dasgupta. Analysis of a greedy active learning strategy. Advances in neural information processing systems , 17, 2004.

[Das05] Sanjoy Dasgupta. Coarse sample complexity bounds for active learning. Advances in neural information processing systems , 18, 2005.

[DG03] Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms , 22(1):60-65, 2003.

[DKK + 23] Ilias Diakonikolas, Daniel M Kane, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Agnostically learning multi-index models with queries. arXiv preprint arXiv:2312.16616 , 2023. Conference version in FOCS'24.

[DKK + 24] Ilias Diakonikolas, Daniel Kane, Vasilis Kontonis, Sihan Liu, and Nikos Zarifis. Efficient testable learning of halfspaces with adversarial label noise. Advances in Neural Information Processing Systems , 36, 2024.

[DKM05] Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptronbased active learning. In Learning Theory: 18th Annual Conference on Learning Theory, COLT 2005, Bertinoro, Italy, June 27-30, 2005. Proceedings 18 , pages 249-263. Springer, 2005.

[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Proceedings of The 34 th Conference on Learning Theory, COLT , 2021.

[DKR23] I. Diakonikolas, D. M. Kane, and L. Ren. Near-optimal cryptographic hardness of agnostically learning halfspaces and relu regression under gaussian marginals. CoRR , abs/2302.06512, 2023.

[DKS18] Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart. Learning geometric concepts with nasty noise. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1061-1073, 2018.

[DKTZ22] Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning general halfspaces with adversarial label noise via online gradient descent. In International Conference on Machine Learning , pages 5118-5141. PMLR, 2022.

[DKZ20] Ilias Diakonikolas, Daniel Kane, and Nikos Zarifis. Near-optimal sq lower bounds for agnostically learning halfspaces and relus under gaussian marginals. Advances in Neural Information Processing Systems , 33:13586-13596, 2020.

[DMRT24] Ilias Diakonikolas, Mingchen Ma, Lisheng Ren, and Christos Tzamos. Fast co-training under weak dependence via stream-based active learning. In Forty-first International Conference on Machine Learning , 2024.

[Fel09] Vitaly Feldman. On the power of membership queries in agnostic learning. The Journal of Machine Learning Research , 10:163-182, 2009.

[FS97] Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

[FSST97] Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine learning , 28:133-168, 1997.

[H + 14] Steve Hanneke et al. Theory of disagreement-based active learning. Foundations and Trends® in Machine Learning , 7(2-3):131-309, 2014.

[HKL20] Max Hopkins, Daniel Kane, and Shachar Lovett. The power of comparisons for actively learning linear classifiers. Advances in Neural Information Processing Systems , 33:63426353, 2020.

[HY15] Steve Hanneke and Liu Yang. Minimax analysis of active learning. J. Mach. Learn. Res. , 16(1):3487-3602, 2015.

[KMT93] Sanjeev R Kulkarni, Sanjoy K Mitter, and John N Tsitsiklis. Active learning using arbitrary binary valued queries. Machine Learning , 11:23-35, 1993.

[KMT24] Vasilis Kontonis, Mingchen Ma, and Christos Tzamos. The gain from ordering in online learning. Advances in Neural Information Processing Systems , 36, 2024.

[MN + 98] Andrew McCallum, Kamal Nigam, et al. Employing em and pool-based active learning for text classification. In ICML , volume 98, pages 350-358. Citeseer, 1998.

[Ros58] Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review , 65(6):386, 1958.

[She21] Jie Shen. On the power of localized perceptron for label-optimal learning of halfspaces with adversarial noise. In International Conference on Machine Learning , pages 95039514. PMLR, 2021.

[Val84a] L. Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, 1984.

[Val84b] L. G. Valiant. A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing (STOC) , pages 436-445. ACM Press, 1984.

[Vap97] Vladimir N Vapnik. The support vector method. In International conference on artificial neural networks , pages 261-271. Springer, 1997.

[Ver18] R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

[YZ17] Songbai Yan and Chicheng Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. Advances in Neural Information Processing Systems , 30, 2017.

Supplementary Material

Here we give an organization of the supplementary material. In Appendix A, we present the formal definition of agnostic learning with membership queries and label queries. In Appendix B, we present the omitted proofs in Section 2 about the information-theoretical lower bound. In Appendix C, we discuss why we can without loss of generality assume the noise level opt ≤ ϵ and how to learn p up to a constant factor with ˜ O (1 /p ) queries. In Appendix D, we present the omitted proofs in Section 3.1 about how to learn a good hypothesis given a good initialization. In Appendix E.2, we present the omitted proofs in Section 3.2 about how to find a good initialization when the threshold is not extremely large. In Appendix F, we design an algorithm that finds a good initialization when the threshold is very large. In Appendix G, we prove Theorem 1.2.

A Active Learning with Membership Queries and Label Queries

̸

Definition A.1 (Active Learning Halfspace with Membership Queries) . Let H = { h ( x ) = sign( w · x + t ) : R d → {± 1 } | w ∈ S d -1 , t ≥ 0 } be the class of halfspaces over X = R d . The labeling function y ( x ) : X →{± 1 } is a random function that maps each x ∈ X to an unknown binary random variable. For each h ∈ H , denote by err( h ) = Pr x ∼ N (0 ,I ) ( h ( x ) = y ( x )) , opt := min h ∈ H err( h ) and h ∗ ( x ) = sign( w ∗ · x + t ∗ ) any halfspace with error opt . A membership query takes x ∈ X as an input and returns a label y ∼ y ( x ) . We say a learning algorithm A is a constant-approximate learning algorithm if for every labeling function y ( x ) , and for every ϵ, δ ∈ (0 , 1) , it outputs some ˆ h ∈ H by adaptively making memberships queries, such that with probability at least 1 -δ , err( ˆ h ) ≤ O (opt + ϵ ) . The query complexity of A is the total number of membership queries it uses during the learning process.

̸

Definition A.2 (Active Learning Halfspace with Label Queries) . Let H = { h ( x ) = sign( w · x + t ) : R d →{± 1 } | w ∈ S d -1 , t ≥ 0 } be the class of halfspaces over X = R d . Let D be a distribution over R d ×{± 1 } such that D x , the marginal distribution over x is the standard Gaussian distribution N (0 , I ) . For each h ∈ H , denote by err( h ) = Pr ( x,y ) ∼ N (0 ,I ) ( h ( x ) = y ) , opt := min h ∈ H err( h ) and h ∗ ( x ) = sign( w ∗ · x + t ∗ ) any halfspace with error opt . Let S be a set of m i.i.d. labeled examples drawn from D . An active learning algorithm with label query is given S but with hidden labels and is allowed to make a label query for each x ∈ S and see its label y . We say a learning algorithm A is a constant-approximate learning algorithm if for every distribution D and for every ϵ, δ ∈ (0 , 1) , it outputs some ˆ h ∈ H by adaptively making label queries over a set of m examples drawn i.i.d. from D , such that with probability at least 1 -δ , err( ˆ h ) ≤ O (opt + ϵ ) . The label complexity of A is the total number label queries made over S during the learning process

B Omitted Proofs in Section 2

B.1 Proof of Lemma 2.1

Proof of Lemma 2.1. Let A be such a learning algorithm. We select a random set of m examples S 1 and give it to A . With probability at least 2 / 3 , A makes r queries and learns a halfspace ˆ h with error p/ 2 with respect to h ∗ . This implies that given a Gaussian example, with probability at least p/ 2 it will predict negative, and given it predicts negative, with probability at least 1 / 2 it is actually negative. Since m is at least poly( d, 1 /p ) , we know that with enough high probability, at least Ω( d ) examples will be predicted by negative by ˆ h and at least a constant fraction of these examples are actually negative. Thus, given such a ˆ h with probability at least 3 / 4 , we can find d negative examples in S by randomly querying O ( d ) examples that are predicted as negative by ˆ h .

B.2 Proof of Lemma 2.2

Let v = Aw ∗ = ( w ∗ · x 1 , . . . , w ∗ · x k ) ⊤ . Consider the projection of w ∗ over the subspace spanned by the row vectors of A , A ⊤ ( AA ⊤ ) -1 Aw ∗ . Assuming that x 1 , . . . , x k are all negative, then ∥ v ∥ 2 ≥

k ( t ∗ ) 2 . This implies that the square of the norm of the projection of w ∗ onto the subspace is

Since w ∗ is uniformly chosen from the unit sphere, by Lemma B.1 in [KMT24], the square norm of w ∗ projected onto a fixed k -dimensional subspace is a random variable drawn from a beta distribution B ( k 2 , d -k 2 ) . By Lemma 2.2 in [DG03], if ∥ ∥ AA ⊤ ∥ ∥ 2 ≤ d (1 + O (1 / ( t ∗ ) 2 )) .

The last inequality follows by Fact D.4.

B.3 Proof of Lemma 2.3

We will first show that for a given A ∈ R k × d , ∥ ∥ AA ⊤ -dI ∥ ∥ 2 is small with high probability if the rows of A are drawn i.i.d. from d -dimensional standard Gaussian. Let N be an 1 / 4 -net of S k -1 . According to [Ver18], we know that |N| ≤ e 3 k and ∥ ∥ AA ⊤ -dI ∥ ∥ 2 ≤ 2 sup u ∈N ∣ ∣ u t ( AA ⊤ -dI ) u ∣ ∣ . Thus, to show that ∥ ∥ AA ⊤ -dI ∥ ∥ 2 is small with high probability, it is equivalent to show with high probability for every u ∈ N , ∣ ∣ u ⊤ ( AA ⊤ -dI ) u ∣ ∣ is small.

Fix u ∈ N to be a unit vector. We have

Notice that each u ⊤ A j is a standard Gaussian variable and thus ∑ d j =1 ( u ⊤ A j ) 2 is a chi-squared distribution with freedom d . By [Ver18], we have

Since |N| ≤ e 3 k , we know that

Since there are at most ( m k ) such k -tuples of examples, the probability that there exists a k -tuple such that ∥ ∥ AA ⊤ -dI ∥ ∥ 2 is larger than ξd = O ( d/ ( t ∗ ) 2 ) is at most

by choosing k = d/ (log( r )( t ∗ ) 4 ) and ξ = O (1 / ( t ∗ ) 2 ) .

C Omitted Details in Section 3

C.1 Discussion on the Noise Level opt

Assuming we know some α such that ϵ ≤ α/ 2 ≤ opt ≤ α , then learning ˆ h upto error O (opt + ϵ ) is equivalent to learning it up to error O ( α ) . By guessing α = ϵ 2 i for i = 0 , . . . , O (log(1 /ϵ )) , we

can always obtain a desired α and use it to run the learning algorithm and get a good hypothesis. Such an approach will generate a list of O (log(1 /ϵ )) different hypotheses, finding a good enough hypothesis among them only costs polylog(1 /ϵ ) queries using a standard tournament approach, such as the following lemma.

Lemma C.1 (Lemma 3.6 in [DKK + 24]) . Let ϵ, δ ∈ (0 , 1) and D a distribution over R d ×{ 0 , 1 } . Given a list of hypothesises { h ( i ) } k i =1 , there is an algorithm that draws O ( k 2 log( k/δ ) /ϵ ) unlabeled examples from D x and performs O ( k 2 log( d/δ )) label queries runs in poly( d, ϵ, δ ) times and output a hypothesis ˆ h such that

̸

C.2 Approximate Bias Estimation Using Queries

In this part, we describe a simple approach to estimate the bias p up to a constant factor using ˜ O (1 /p ) queries. To do this we will estimate ¯ p = Pr x ∼ N (0 ,I ) ( y ( x ) = -1) , the noise version of p as | ¯ p -p | ≤ ϵ . If we can estimate ˆ p such that ˆ p/ 2 ≤ ¯ p ≤ ˆ p or verify that ¯ p ≤ ( C -1) ϵ , then ˆ p satisfies our purpose.

By Chebyshev's inequality, if ¯ p ≤ 3ˆ p/ 4 , then taking O (1 / ˆ p ) random queries at x and computing the empirical probability of y ( x ) = -1 , with probability 2 / 3 , we are able to verify this fact by checking whether the empirical probability is less than 5ˆ p/ 6 . On the other hand, if 4ˆ p/ 5 ≤ ¯ p ≤ ˆ p , with probability 2 / 3 we are able to verify this fact by checking whether the empirical probability is greater than 5ˆ p/ 6 . Furthermore, by repeating this approach O (log(1 /δ )) times and using a majority voting trick, we can boost the probability of success up to 1 -δ . We will run the above approach for ˆ p = (4 / 5) i / 2 for i = 0 , 1 , . . . until we find ¯ p ≥ (4 / 5)ˆ p or ˆ p = C ′ ϵ for some constant C ′ . In the first case (4 / 5)ˆ p ≤ ¯ p ≤ (25 / 24)ˆ p and we find a good approximation for ¯ p and thus for p . In the second case, we can conclude that p is smaller than O ( ϵ )

D Omitted Proofs from Section 3.1

Algorithm 3 REFINE(Learn the correct direction w ∗ based on a warm start w 0 )

Input: Intial direction w 0 ∈ S d -1 , t ′ > 0 , an approximate threshold, error parameter ϵ ∈ (0 , 1) , confidence parameter δ ∈ (0 , 1)

Output: w T ∈ S d -1 , an approximation of w ∗ , ˆ t ∈ R , an approximation of t ∗

Let ϵ ′ = Cϵ exp( t ′ 2 / 2) m = ˜ O ( d ) , T = O (log(1 /ϵ ′ )) σ 0 ← min { 1 /t ′ , 1 / 2 } .

Let C 1 , C 2 be large enough constants

Find ˜ t ∈ { 0 , ϵ, 2 ϵ, . . . , t ′ } using the following binary search method, if no such ˜ t is found, then stop the algorithm and return w T = 0 . ▷ Find the correct threshold to construct the gradient.

Draw O (log(1 /δ )) Gaussian samples z ∼ N (0 , I ) , query A 1 / 2 i z -˜ tw i and compute p ( ˜ t ) , the empirical probability that a query returns -1 . If p ( ˜ t ) < e -17 , properly decrease ˜ t , if p ( ˜ t ) > e -17 , properly increase ˜ t . Otherwise, declare that ˜ t is found.

D.1 Proof of Lemma 3.2

Proof of Lemma 3.2. We first observe that

̸

It remains to upper bound ∥ w i + µ i ˆ g i -w ∗ ∥ 2 . We have

Here, in the second equality, we use the fact that ˆ g i ⊥ w i and in the fourth equality, we use the fact that ( g i -ˆ g i ) · w ∗ = ( g i -ˆ g i ) · a i w i +( g i -ˆ g i ) · b i u = ( g i -ˆ g i ) · b i u .

Next, we assume that sin( θ i / 2) ≤ σ i and show that we can carefully choose parameter µ i , σ i +1 to make sin( θ i +1 / 2) ≤ σ i +1 . We consider two cases. In the first case, we assume 3 σ i / 4 sin( θ i / 2) ≤ σ i . Since ∥ w i -w ∗ ∥ = 2sin θ i 2 , by Lemma 3.2, we have

where use the fact that b i ≤ 2 sin( θ i / 2) ≤ 3 σ i / 2 and the fact that C 1 can be made large enough. In the second case, we assume sin( θ i / 2) < 3 σ i / 4 . In this case, using the fact that

We have

where the last inequality holds because the parameter C 1 , C 2 can be chosen larger than 8 .

Lemma 3.2 implies that if g i has enough correlation with respect to w ∗ but is also not too long, then by estimating g i up to some error, we can ensure ∥ w i -w ∗ ∥ drops significantly each round. Formally, we have the following corollary.

Corollary D.1. In Algorithm 3, denote by θ i = θ ( w ∗ , w i ) . Assume that sin θ i 2 ≤ σ i . If there exist a suitable constant c 1 and a small enough constant c 2 such that g i · w ∗ ≥ c 1 σ i / 10 , ∥ ˆ g i ∥ ≤ c 1 and ∥ g i -ˆ g i ∥ ≤ c 2 . Then there exists large enough constant C 1 , C 2 such that by taking µ i = σ i /C 1 , it holds that sin θ i +1 2 ≤ (1 -1 /C 2 ) σ i .

Proof of Corollary D.1. Since ∥ w i -w ∗ ∥ = 2sin θ i 2 , by Lemma 3.2, we have

where the third and the fourth inequalities hold when C

1 is large enough.

D.2 Proof of Lemma 3.5

Proof of Lemma 3.5. We first prove Item 1. Since T i is a monotone decreasing function on ˜ t , and t ′ > t ∗ , it remains to show that for every ˜ t such that ∣ ∣ ˜ t -t ∗ ∣ ∣ ≤ σ i , | T i | ≤ 5 . Notice that

By Fact 3.3, we know that

Since √ a 2 i + b 2 i /σ 2 i ≤ √ 5 and | T i | ≤ 5 , we immediately obatin Item 2. Finally, we prove Item 3. Using the monotone property of T i , we prove the case where ˜ t < t ∗ -40 σ i and the case ˜ t > t ∗ +40 σ i can proved symmetrically. We have

where the first inequality holds because of Equation (1).

D.3 Proof of Lemma 3.6

To prove Lemma 3.6, we first introduce the following definition called ( v, s, σ ) - rejection procedure.

Definition D.2 ( ( v, s, σ ) - rejection procedure) . Let v ∈ R d be a unit vector and s, σ be real numbers such that σ < 1 . Given a point x ∈ R d , ( v, s, σ ) - rejection procedure accepts it with probability

and rejects it otherwise.

( v, s, σ ) - rejection procedure satisfies the following property.

Lemma D.3 (Lemma C.7, Lemma C.8 in [DKS18]) . If x ∼ N (0 , I ) is fed into the ( v, s, σ ) - rejection procedure, then it is accepted with probability σ exp( -s 2 / (2(1 -σ 2 ))) . In particular, when σs ≤ 2 and σ ≤ 1 / 2 , the accepted probability is at least σ exp( -s 2 / 2 -4) . Moreover, the distribution on x conditioned on acceptance is that of N ( -sv, A v,σ ) , where A v,σ = I -(1 -σ 2 ) vv t .

Proof of Lemma 3.6. Let ˜ z := A 1 / 2 i z -˜ tw i . By Fact 3.3, we know that ℓ i ( z ) = h ∗ (˜ z ) , ∀ z ∈ R d . By Lemma 3.6, we know that if z ∼ N (0 , I ) , then ˜ z ∼ N ( -˜ tw i , A i ) , which can be seen by feeding a Gaussian random vector into the ( w i , ˜ t, σ i ) -rejection procedure conditioned on acceptance. Since err( h ∗ ) = opt ≤ ϵ and the accepted rate is at least σ exp( -s 2 / 2 -4) , we know from Lemma 3.6 that

̸

In particular, if σ i ≥ C exp(( t ′ ) 2 / 2) ϵ , we have

D.4 Proof of Lemma 3.7

Before presenting the proof, we state the following two facts that will be used in our proof.

Fact D.4 (Komatsu's Inequality) . For any t ∈ R the bias p of a halfspace h ( x ) = sign( w ∗ · x + t ) can be bounded as

̸

Fact D.5 (Lemma B.4 in [DKTZ22]) . Let D be a distribution on R d ×{± 1 } with standard normal x -margin and let w,u be two orthogonal unit vectors. Let B be any interval over R and let S ( x, y ) be any event over R d ×{± 1 } , such that S ( x, y ) ⊆ { w · x ∈ B } then it holds

̸

Proof of Lemma 3.7. We start by proving the first part of Lemma 3.7. By Lemma 3.6, we know that η i := Pr z ∼ N (0 ,I ) ( ℓ i ( z ) = f i ( z )) ≤ e -40 . This implies that ∣ ∣ Pr z ∼ N (0 ,I ) ( ℓ i ( z ) = -1) -p i ∣ ∣ ≤ e -40 . We first show that when p i is in a reasonable range, | T i | < 6 . Assuming by contradiction that | T i | ≥ 6 , then by Fact D.4, the bias of ℓ i ( z ) must be at most exp( -T 2 i / 2) / (2 T i ) ≤ e -20 , which implies that it cannot be the case where p i ∈ ( e -18 , 1 -e -18 ) . Similarly, if | T i | < 5 , then by Fact D.4, the bias of ℓ i ( z ) must be at least exp( -T 2 i / 2) / 20 ≥ e -15 . 5 . As the noise level η i ≤ e -40 , we have p i ∈ ( e -18 , 1 -e -18 ) .

Next, we prove the second part of Lemma 3.7. We start by bounding the correlation between g i and u i . We have

̸

where the third and the last inequalities hold because u i ⊥ w i and Fact D.5.

We next bound the norm of g i . Since both ¯ g i and g i are orthogonal to w i . It is sufficient to show that for every unit vector u ⊥ w i , | g i · u | ≤ | ¯ g i · u | +4 √ eη i √ log(1 /η i ) . We have

̸

D.5 Proof of Theorem 3.1

In this section, we present the proof of Theorem 3.1. Before presenting the proof, we present the following fact that will be a crucial part of our proof.

Fact D.6 (Lemma 4.2 in [DKS18]) . Under the standard normal distribution for every pair of unit vector w,w ∗ and real number t ,

̸

Proof of Theorem 3.1. Denote by θ i := θ ( w i , w ∗ ) . We will first show by induction that with high probability in the i -th round of Algorithm 3, sin( θ i / 2) ≤ σ i . Assuming this is correct, since σ T = Cϵ exp( t ′ ) 2 / 2 for some large constant C . We will have

̸

Since opt ≤ ϵ , this implies that by providing a good enough estimation of t ∗ , we found a hypothesis with error at most O ( ϵ ) . Now we show that this is actually true. For i = 0 , sin( θ 0 / 2) ≤ σ 0 holds by our assumption.

Now, we assume this is correct for the i -round and we show this holds with high probability for the i + 1 -th round. We will show that with high probability the gradient ˆ g i we use in the update w i +1 = proj w ⊥ i ( w i + µ i ˆ g i ) satisfies the condition of Lemma 3.2.

̸

Recall that we have the following notations. w ∗ = a i w i + b i u i . T i := t ∗ -a i ˜ t σ i √ a 2 i + b 2 i /σ 2 i , ℓ i ( z ) = sign(( a i w i + b i u i /σ i ) z + ( t ∗ -a ˜ t ) /σ i ) , ¯ g i = E z ∈ N (0 ,I ) proj w ⊥ i zℓ i ( z ) and g i = E z ∈ N (0 ,I ) proj w ⊥ i zf i ( z ) , where f i ( z ) = y ( A 1 / 2 i z -˜ tw i ) . And η i := Pr z ∼ N (0 ,I ) ( ℓ i ( z ) = f i ( z )) < e -40 by Lemma 3.6.

We first show that with high probability, Algorithm 3 must be able to select a correct threshold ˜ t ≤ t ′ such that | T i | ≤ 6 . Denote by p i the probability that f i ( z ) = -1 . We notice that for each fixed ˜ t by randomly querying O (log(1 /δ )) f i ( z ) , we can with high probability check if p i ∈ ( e -17 , 1 -e -17 ) or not. This can be done using the same method we used in Appendix C.2.

Since b i / 2 = sin θ i / 2 ≤ sin( θ i / 2) ≤ σ i , we know from Lemma 3.7 that as long as we find some ˜ t such that p i ∈ ( e -17 , 1 -e -17 ) , we have have | T i | ≤ 6 . By Lemma 3.5 and Lemma 3.7, we know that there exists an interval I i ⊆ [0 , t ′ ] of length at least σ i > ϵ such that for every ˜ t ∈ I i , | T i | < 5 and thus p i ∈ ( e -16 , 1 -e -16 ) . Thus, by performing a binary search at most O (log(1 /ϵ )) times, with high probability, we are able to find such a ˜ t such that | T i | < 6 . Given that we find such a correct ˜ t , we will consider two cases.

First, we assume that 3 σ i / 4 ≤ sin( θ i / 2) ≤ σ i . We will show that with high probability g i and its empirical estimation ˆ g i satisfy the condition in the statement of Corollary D.1 and thus prove sin( θ i +1 / 2) ≤ σ i +1 . Since proj w ⊥ i zf i ( z ) is 1 -subgaussian random vector, by Hoeffding's inequality, we know that with ˜ O ( d ) samples of z , with high probability we have ∥ g i -ˆ g i ∥ ≤ c 2 ≤ e -40 .

By Lemma 3.7 and Lemma 3.5, we have

Thus, by Corollary D.1, we can conclude that sin( θ i +1 / 2) ≤ (1 -1 /C 2 ) σ i = σ i +1 , for a large constant C 2 .

Next, we consider the case where sin( θ i / 2) < 3 σ i / 4 . In this case, as we have shown that ∥ ˆ g i ∥ is bounded by some universal constant, the condition of Lemma 3.2 is fulfilled automatically and thus sin( θ i +1 / 2) ≤ (1 -1 /C 2 ) σ i = σ i +1 , for a large constant C 2 .

By induction, with a high probability for each i , we have sin( θ i / 2) ≤ σ i and thus w T is a good approximation of w ∗ . It remains to show that ˆ t is also a good approximation of t ∗ . Recall that ˆ t = ˜ t < t ′ such that | T T | < 6 . Lemma 3.5 implies that ∣ ∣ ˆ t -t ∗ ∣ ∣ ≤ 40 σ T = 40 Cϵ exp( t ′ ) 2 / 2 . Thus,

̸

̸

Since σ T t ′ = O ( ϵt ′ exp(( t ′ ) 2 / 2)) and t ′ exp(( t ′ ) 2 / 2) ≤ 1 / ( Cϵ ) , we can conclude that Pr x ∼ N (0 ,I ) ( sign( w T · x + t ∗ ) = sign( w T · x + ˆ t ) ) ≤ O ( ϵ ) . Thus, with high probability err(sign( w T · x + ˆ t )) ≤ O ( ϵ ) .

Finally, we count the number of queries used by Algorithm 3. In each round of the algorithm, we perform O (log(1 /ϵ )) binary searches to find the correct parameter ˜ t , each of which takes us only ˜ O (1) queries. We also make ˜ O ( d ) queries to construct ˆ g i in each round of the algorithm. Thus, each round of Algorithm 3 takes ˜ O ( d +log(1 /ϵ )) queries. Since there are at most O (log(1 /ϵ )) rounds, the query complexity of Algorithm 3 is ˜ O ( d polylog(1 /ϵ )) .

E Omitted Proofs from Section 3.2

E.1 Proof of Lemma 3.11

Proof of Lemma 3.11. Denote by D -the conditional distribution of x ∼ N (0 , I ) given y ( x ) = -1 . Recall that

We will first show that E x ∼ D -η ( x ) ≤ O ( ϵ/p ) . Denote by Z := √ 1 -ρ 2 x + ρz , where x ∼ D -, z ∼ N (0 , I ) , then

̸

̸

Since z and x are independent, we notice that Z can be simulated via the following reject sampling process. We draw x ∼ N (0 , I ) and Z ∼ N (0 , I ) to construct Z = √ 1 -ρ 2 x + ρz and accepted Z when y ( x ) = -1 . Since √ 1 -ρ 2 N (0 , I ) + ρN (0 , I ) = N (0 , I ) , Z can be seen as a reject sampling process with an accepted rate at least (1 -1 /C ) p . Since the noise rate opt ≤ ϵ , we know that

̸

By Markov's inequality, we know that with probability at least 3 / 4 , η ( x ) ≤ 5 ϵ/p , with x ∼ D -. Next, we show that with a constant probability a negative example x must be close to the decision boundary of h ∗ . We have

where in the third inequality, we use Fact D.4.

Thus, by union bound, with probability at least 1 / 2 , it simultaneously holds that η ( x ) ≤ 5 ϵ/p and w ∗ · x ≤ -t ∗ -1 /t ∗ .

E.2 Proof of Theorem 3.8

Proof of Theorem 3.8. We consider two cases. First, if t ′ < 1 , then each x ( i ) 0 = z i is drawn from the standard Gaussian. We have

̸

where the last inequality holds because of Fact D.5. Since each z i y ( z i ) is a standard Gaussian, by Hoeffding's inequality, we have

when m ≥ ˜ Ω( d/r 2 ) . By taking r = (20log(1 /ϵ )) -1 , we obtain that ∥ ∥ u 0 -E z ∼ N (0 ,I ) zy ( z ) ∥ ∥ ≤ (20 log(1 /ϵ )) -1 with high probability. Thus

By Fact 3.4, we know that E z ∼ N (0 ,I ) zh ∗ ( z ) = ξw ∗ for some ξ ≥ e -1 , which also implies that ∥ u 0 ∥ ≥ e -1 / 2 , because u 0 sufficiently close to E z ∼ N (0 ,I ) zh ∗ ( z ) . Since

we get sin( θ ( w 0 , w ∗ ) / 2) ≤ √ ∥ ∥ u 0 -E z ∼ N (0 ,I ) zh ∗ ( z ) ∥ ∥ 2 / 4 ∥ ∥ E z ∼ N (0 ,I ) zh ∗ ( z ) ∥ ∥ ∥ u 0 ∥ ≤ O (1 / log(1 /ϵ )) . In particular as ϵ < 1 /C for some large enough C , we conclude that sin( θ ( w 0 , w ∗ ) / 2) ≤ max { min { 1 /t, 1 / 2 } , O ( η √ log(1 /η ) } .

We next address the case when t > 1 . By Lemma 3.11, we know that with probability at least 1 / 2 , we have η ( x ) ≤ 5 ϵ/p and w ∗ · x ∈ ( -t ∗ -1 /t ∗ , -t ∗ ) . We will assume these two events happen in the rest of the proof. Let z ∼ N (0 , I ) and by Fact 3.10, define

̸

By Lemma 3.11, we know that Pr z ∼ N (0 ,I ) ˜ h ( z ) = ˜ y ( x 0 ) = η ( x 0 ) ≤ 5 ϵ/p . Similar to the first case, we have

̸

where in the second last inequality we use the fact that ∥ ∥ u 0 -E z ∼ N (0 ,I ) z ˜ y ( x 0 ) ∥ ∥ ≤ 1 / (100 √ log(1 /ϵ )) ≤ 1 / (100 t ) with high probability. Since ρ = 1 /t , | t -t ∗ | ≤ 1 / log(1 /ϵ ) and t ∗ ≤ √ log(1 /ϵ ) ≪ log(1 /ϵ ) , the threshold T ρ = t ∗ + √ 1 -ρ 2 w ∗ · x 0 ρ can be bounded as follows.

Fact 3.4 implies that E z ∼ N (0 ,I ) z ˜ h ( z ) = ξw ∗ for some ξ ≥ e -1 . Since u 0 is close to E z ∼ N (0 ,I ) z , ∥ u 0 ∥ ≥ e -1 / 2 . Thus, we obtain that

F Finding a Good Initialization with an Extreme Threshold

By Theorem 3.8, we know that Algorithm 2 can only find some w 0 such that sin( θ 0 / 2) ≤ O ( η √ log(1 /η )) , where η = ϵ/p when p is small such that η √ log(1 /η ) > O (1 /t ) . In this section, we design an algorithm that finds a warm start with a non-negligible probability of success when the threshold t ∗ falls in this range. Formally, we prove the following theorem.

Theorem F.1. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace with bias p and y ( x ) be any labeling function such that err( h ∗ ) = opt ≤ ϵ ≤ 1 /C for some large enough constant C . Let t be a scalar such that t -1 / log(1 /ϵ ) ≤ t ∗ ≤ t and 1 / (400 t ) ≤ η √ log(1 /η ) ≤ 1 /C for some large enough constant C , where η = ϵ/p , Algorithm 5 makes M = ˜ O (1 /p + d log(1 /ϵ )) membership queries, runs in poly( d, M ) time and with probability at least 1 / polylog(1 /ϵ ) , outputs some w 0 such that sin( θ ( w 0 , w ∗ ) / 2) ≤ 1 /t .

The high-level idea of our algorithm is as follows. Although Algorithm 2 will not provide us a w 0 such that θ 0 ≤ O (1 /t ) , θ 0 is still smaller than a sufficiently small constant. We want to use the localization technique to refine w 0 so that after T rounds of refinement, sin( θ T / 2) ≤ σ T = 1 /t . Recall in Appendix D, we introduce Definition D.2, ( v, s, σ ) -rejection procedure, which can be simulated using membership query. Passing a Gaussian random point to the ( v, s, σ ) -rejection procedure, according to Lemma D.3, we will get a another distribution over R d ×{± 1 } that behaves the same as another halfspace h ′ .

In this section, we want to design a ( v, s, σ ) -rejection procedure such that the direction of the halfspace h ′ has a constant correlation with respect to w ∗ and the noise level after the rejection procedure is much smaller than the length of the Chow-Parameter of h ′ . Write w ∗ = a i w i + b i u i . We want to set up v = w i , σ = 1 /t and s ∼ ( a i t, a i t + b i ) uniformly. Such a method is called the randomized threshold method in [DKS18]. This method has the following property.

Lemma F.2 (Proposition C.11 in [DKS18]) . Let a, b, t > 0 such that a 2 + b 2 = 1 and t larger than some constant C . Let w ∈ S d -1 . Let s ∼ [ at, at + b ] uniformly. For each x ∈ R d , the expected probability that x is accepted by the ( w,s, σ ) -rejection procedure is at most σ/b , where σ = 1 /t .

Lemma F.2 implies that in expectation over the randomness of s , only σ/b i -fraction of the noisy points will pass the ( w i , s, σ ) -rejection procedure. If we use query to simulate such a rejection procedure, by Lemma D.3, with a constant probability, the noise rate among our queries would be O ( ϵ exp( s 2 / 2) /b i ) . However, as we do not know b i , using some b that is slightly far from b i would make the noise level too high for us to learn the signal we want. To overcome this, we design the following test approach to show that given a b , we can with high probability check if it can be used to construct the rejection procedure or not and in particular, when b -1 / log(1 /ϵ ) < b i < b , such a b is guaranteed to pass our test.

Lemma F.3. Let h ∗ ( x ) = sign( w ∗ · x + t ∗ ) be a halfspace and y ( x ) be any labeling function such that err( h ∗ ) = opt ≤ ϵ . Let w ∈ S d -1 be unit vector such that w ∗ = a ∗ w + b ∗ u , a ∗ , b ∗ > 0 and ( a ∗ ) 2 + ( b ∗ ) 2 = 1 , b < 1 / 4 . Let t > 0 such that t exp( t 2 / 2) ≤ 1 / ( Cϵ ) for a sufficiently large constant C . Let a, b ∈ (0 , 1) such that a 2 + b 2 = 1 . Let b, t, w, δ be input of Algorithm 4. Let s ∼ ( at, at + b ) uniformly. Denote by p ( b, s ) be bias of a halfspace with threshold T bs := ( t -as ) /b . Let ℓ ( z ) = sign(( a ∗ σw + b ∗ u ) · z + t ∗ -a ∗ s ) be a halfspace with bias p s , where If the probability that p s > p ( b, s ) / 4 is at most 1 / 2 , then with probability at least 1 -δ , Algorithm 4 output 'No'. If the probability that p s > p ( b, s ) / 2 is at least 29 / 30 , then with probability at least 1 -δ , Algorithm 4 output 'Yes'. Furthermore, the query complexity of Algorithm 4 is ˜ O δ (1 /p 2 ( b, at )) = ˜ O δ (1 / √ p )

In particular, when b ∗ ≥ 1 . 5 /t ≥ 1 . 5 / √ log(1 /ϵ ) , | b -b ∗ | ≤ 1 / log(1 /ϵ ) and | t -t ∗ | ≤ 1 / log(1 /ϵ ) , with probability at least 1 -δ , Algorithm 4 will output 'Yes'.

Algorithm 4 ANGLE TEST(Check if b is a good approximation for sin θ ( w ∗ , w ) )

Input: A direction w , confidence parameter δ ∈ (0 , 1) , threshold t > 0 , parameter b

̸

Proof of Lemma F.3. By Lemma D.3 and Lemma F.2, we know that over the randomness of s , with probability at least 5 / 6 , η := Pr z ∼ N (0 ,I ) ( h ∗ ( Az -sw ) = y ( Az -sw )) ≤ 6 ϵ exp( s 2 / 2) /b . We assume, for now, such an event happens. We first show that such a noise rate is much smaller than p ( b, s ) . Write s = at + ξ , where ξ ∈ [0 , b ] , then we have

where, in the first inequality, we use the fact that T bs ≤ bt , in the second inequality, we use the fact that t exp( t 2 / 2) ≤ 1 / ( Cϵ ) for a sufficiently large constant C , and in the last inequality, we use the fact that a 2 + b 2 = 1 , ξ 2 < b 2 . By Fact D.4, we know that exp( -T 2 bs / 2) /T bs is at most 3 times p ( b, s ) , and thus η ≤ p ( b, s ) /C ′ for a large enough constant C ′ .

By Fact 3.3, we know that the ground truth label ℓ ( z ) = h ∗ ( Az -sw ) = sign(( a ∗ σw + b ∗ u ) · z + t ∗ -a ∗ s ) . By Hoeffding's inequality, with high probability, we are able to estimate the probability of y ( Az -sw ) = -1 up to error p ( b, s ) / 20 using ˜ O (1 /p 2 ( b, s )) queries. In particular, since T bs ≤ tb < 1 / 4 , by Fact D.4, we know that p ( b, s ) > p 1 / 4 and will cost us only ˜ O (1 / √ p ) queries.

If the probability that p s > p ( b, s ) / 4 is at most 1 / 2 , then in each round i of Algorithm 4, with probability at least 1 / 3 it holds simultaneously that p s < p ( b, s ) / 4 and η ≤ p ( b, s ) /C ′ . In this case, with high probability ˆ p s < p ( b, s ) / 3 and Count does not increase. Thus, with probability at least 1 -δ , after T = O (log(1 /δ )) rounds, Count < 3 T/ 4 by Hoeffding's inequality.

Similarly, if the probability that p s > p ( b, s ) / 2 is at least 29 / 30 , then in each round i of Algorithm 4, with probability at least 4 / 5 it holds simultaneously that p s > p ( b, s ) / 2 and η ≤ p ( b, s ) /C ′ . In this case, with high probability ˆ p s > p ( b, s ) / 3 and Count increases. Thus, with probability at least 1 -δ , after T = O (log(1 /δ )) rounds, Count > 3 T/ 4 by Hoeffding's inequality.

Finally, we show that when | b ∗ -b | ≤ 1 / log(1 /ϵ ) and when | t -t ∗ | ≤ 1 / log(1 /ϵ ) , Algorithm 4 with high probability outputs 'Yes'. To do this, we will show the true threshold of ℓ ( z ) is close to T bs . We have

By Fact D.4, it holds with probability 1 that p s > p ( b, s ) / 2 .

Now assume that we have sin( θ i / 2) ≤ σ i , then b i ≤ 2 σ i . This implies that by testing b = 2 σ i -j log(1 /ϵ ) for j = 0 , 1 , . . . , we only need O (log(1 /ϵ )) rounds to find the correct b . With this fact, we have the following Algorithm 5.

Proof of Theorem F.1. Let θ i = θ ( w i , w ∗ ) and write w ∗ = a i w i + b i u i , where a i , b i > 0 , a 2 i + b 2 i = 1 . By Algorithm 2, we know that with probability at least 1 / 3 , sin( θ 0 / 2) ≤ O ( ϵ/p √ log( p/ϵ )) . We will assume sin( θ 0 / 2) ≤ ϵ/p √ log( p/ϵ ) holds throughout the proof, since the constant before ϵ/p √ log( p/ϵ ) can always be assumed to be normalized as 1 /C is large enough. In round i of the algorithm, we write w ∗ = a i w i + b i u i where a i , b i > 0 , a 2 i + b 2 i = 1 . Similar to the analysis of Algorithm 3, we will show that if sin( θ i / 2) ≤ σ i then with probability 1 / 3 it also holds that sin( θ i +1 / 2) ≤ σ i +1 . If this is true then since 1 /t > 1 / √ log(1 /ϵ ) after O (log log(1 /ϵ )) rounds, we have sin( θ T / 2) ≤ 1 /t with probability at least 1 / polylog(1 /ϵ ) .

Recall the notation in the proof of Lemma F.3. Given ˆ b , we define p ( ˆ b, s ) to be the bias of a halfspace with a threshold T ˆ b,s = ( t -ˆ as ) / ˆ b . By Fact 3.3, we define ℓ ( z ) = h ∗ ( Az -sw i ) = sign(( a i σw i + b i u ) · z + t ∗ -a i s ) the ground truth label of y ( Az -sw i ) , t s to be its threshold and p s to be the bias of ℓ ( z ) .

̸

By Lemma F.3, we know that as long as b i > 1 . 5 /t , with high probability Algorithm 4 will output 'Yes' for some ˆ b such that with probability at least 1 / 2 , p s > p ( ˆ b, s ) / 2 > p 1 / 4 . On the other hand, by Equation (3), we know that with probability at least 5 / 6 , η := Pr z ∼ N (0 ,I ) ( ℓ ( z ) = y ( Az -sw i )) ≤ p ( b, s ) /C ′ for a sufficiently large constant C ′ . Thus, with a probability at least 1 / 3 , p s > p ( ˆ b, s ) / 2 and η ≤ p ( ˆ b, s ) /C ′ hold simultaneously. For now, we assume this happens and we will analyze the smoothed label around some z 0 such that y ( Az 0 -sw i ) = -1 . By Fact 3.10, the smoothed label

Algorithm 5 INITIALIZATION 2(Finding a good initialization under extreme threshold)

Input: error parameter ϵ ∈ (0 , 1) , confidence parameter δ ∈ (0 , 1) , threshold t > 0 Output: w 0 ∈ S d -1

i

= 0

, . . . , T

=

O

w

0

∈

S

(log log(1

d

/ϵ

-

1

))

. Let do

Run Algorithm 4 with input w i and ˆ b = 2 σ i -j log(1 /ϵ ) , j = 0 , . . . , σ i log(1 /ϵ )

Let ˆ b be the first parameter such that Algorithm 4 outputs 'Yes'

If Algorithm 4 outputs 'No' for all ˆ b or the ˆ b we use less than 1 /t , then return w i .

Estimate the probability ˆ p s of p s = y ( Az -sw i ) = -1 for z ∼ N (0 , I ) up to error p 1 / 4 / 100 using ˜ O ( √ 1 /p ) queries. Let ˆ t s be the threshold of a halfspace with bias ˆ p s

A

I

-

(1

-

σ

2

)

w

w

t

,

σ

= 1

/t

Draw z i ∼ N (0 , I ) , for i ∈ [ m ] , m = ˜ O ( d ) and query f i ( z i ) := y ( A ( 1 -ρz 0 + ρz i ) -sw i ) , where ρ = 1 / ˆ t s i i Draw z ∼ N (0 , I ) and query y ( Az -sw i ) until some z 0 such that y ( Az 0 -sw i ) = -1 is drawn √

return w T

around z 0 with respect to halfspace ℓ ( z ) can be seen as a halfspace

where W i := ( a i σw i + b i u i ) / √ ( a i σ ) 2 + b 2 i and T ρ,s = t s + √ 1 -ρ 2 W i · z 0 ρ

By Fact 3.10 and Lemma 3.11, we know that with probability at least 1 / 2 , such a z 0 satisfies

the noise level of the smoothed label is at most 5 η/p s ≤ 1 /C ′′ for some large enough constant C ′′

Since p s > p ( ˆ b, s ) / 2 , we can bound the threshold T ρ,s by

because ˆ t s is at least close to t s up to a small constant factor, otherwise ˆ p s would be far from p s . Combine Equation (4) and Fact 3.4, we know that E z ′ ∼ N (0 ,I ) proj w ⊥ i z ′ ℓ z 0 ( z ′ ) = ϕ u i b i √ ( a i σ ) 2 + b 2 i , for some ϕ ∈ ( e -2 , 1) . Since the noise level of the smoothed label around z 0 is as small as 1 /C ′′ for some large enough constant C ′′ , by Hoeffding's inequality, we know that ∥ ∥ ∥ g i -E z ′ ∼ N (0 ,I ) proj w ⊥ i z ′ ℓ z 0 ( z ′ ) ∥ ∥ ∥ can be smaller than some tiny constant with high probability.

As it always holds that σ i ≥ 1 /t for each i , we will consider two cases. In the first case, sin( θ i / 2) ≤ 3 σ i / 4 and ∥ g i ∥ is bounded by some universal constant.

In the second case, we have 3 σ i / 4 ≤ sin( θ i / 2) ≤ σ i . In this case we know that E z ′ ∼ N (0 ,I ) proj w ⊥ i z ′ ℓ z 0 ( z ′ ) = ϕ u i b i √ ( a i σ ) 2 + b 2 i = ψu i for some ψ ≥ e -4 , which implies that E z ′ ∼ N (0 ,I ) proj w ⊥ i z ′ ℓ z 0 ( z ′ ) · u i ≥ ψ b i √ ( a i σ ) 2 = b 2 i ≥ e -5 . Using Lemma 3.2, we know that sin( θ i +1 / 2) ≤ (1 -1 /C 1 ) σ i = σ i +1 .

Finally, we prove the query complexity of Algorithm 5. By Theorem 3.8, it takes us ˜ O (1 /p + d log(1 /ϵ )) queries to get some w 0 by running Algorithm 2. After obtaining w 0 , in each round of

σ

0

=

ϵ/p

√

log(

p/ϵ

)

be a parameter

Algorithm 4, we will run Algorithm 4 O (log(1 /ϵ )) times to find a desired ˆ b and each round takes us ˜ O (1 /p 2 ( ˆ b )) ≤ 1 /p 2 c ≤ 1 / √ p queries, because p ( ˆ b ) is the bias of a halfspace with threshold T ˆ b = ˆ bt , which is smaller than t by a tiny constant factor. Furthermore, after obtaining ˆ b it takes us ˜ O (1 /p ( ˆ b ) + d log(1 /ϵ )) queries to perform the gradient descent update. So, in total Algorithm 5 has query complexity at most ˜ O (1 /p + d log(1 /ϵ )) .

G Proof of Theorem 1.2

Proof of Theorem 1.2. We first show the correctness of Algorithm 1. When we run Algorithm 1, we will start with some interval [ t a , t b ] such that any halfspace with a threshold t ∈ [ t a , t b ] must have bias Θ( p ) . Next, Algorithm 1 partition [ t a , t b ] into grid such that | t j +1 -t j | ≤ 1 / log(1 /ϵ ) . This implies that there must be some t j ∈ [ t a , t b ] such that t j -1 / log(log(1 /ϵ )) ≤ t ∗ ≤ t j . By Theorem 3.8 and Algorithm 5, as long as p > Cϵ , with probability at least 1 / polylog(1 /ϵ ) , we can find some w 0 such that sin( θ 0 / 2) ≤ min { 1 /t j , 1 / 2 } . In particular, by running Algorithm 2 or Algorithm 5 polylog(1 /ϵ ) times, at least one of these w 0 satisfies the condition. Furthermore, with such a w 0 , we know from Theorem 3.1 that we can with high probability get some ˆ h such that err( ˆ h ) ≤ O (opt + ϵ ) . Thus within the list C of the candidate hypotheses maintained by Algorithm 1 at least one of them has error O (opt + ϵ ) . By Lemma C.1, we can with high probability find a hypothesis among C , whose error is at most 10 times the error of the best hypothesis in C and thus has error O (opt + ϵ ) .

Next, we prove the query complexity of Algorithm 1. According to Appendix C.2, we know that finding an interval [ t a , t b ] costs us ˜ O (min { 1 /p, 1 /ϵ } ) queries. If we find p < Cϵ then we are done. Otherwise, we will run the initialization algorithm and the refinement algorithm. By Theorem 3.8 and Algorithm 5, each time we run an initialization algorithm, it takes us ˜ O (1 /p + d polylog(1 /ϵ )) queries. By Algorithm 3, each time we run Algorithm 3, it takes us ˜ O ( d log(1 /ϵ ) queries. Since we will run these algorithms at most polylog(1 /ϵ ) times. We will in total make ˜ O (1 /p + d polylog(1 /ϵ )) queries. Finally, by Lemma C.1, finding a good hypothesis from the list of candidate hypotheses will only take us polylog(1 /ϵ ) queries. Thus, we conclude the query complexity of Algorithm 1 is ˜ O (min { 1 /p, 1 /ϵ } + d polylog(1 /ϵ )) .

H Implementing the Learning Algorithm via A Small-Class Oracle

In this section, we discuss how to implement Algorithm 1 to get an even smaller query complexity ˜ O δ ( d · polylog(1 /ϵ )) , assuming there is an oracle that can return a random small-class example. Before presenting the definition of the small-class oracle, we remind the reader that the notation ˜ O hides the dependence on polylog(1 /ϵ ) , and the notation O δ hides the dependence on polylog(1 /δ ) . A small class oracle is defined as follows.

Definition H.1 (Small-Class Oracle) . Let D be a distribution over R d ×{± 1 } and h ∗ = sign( w ∗ · x + t ∗ ) , w ∗ ∈ S d -1 , t ∗ > 0 be an optimal halfspace such that err( h ∗ ) = opt = min h ∈ H err( h ) . A small-class oracle EX ( -) ( D ) draws ( x, y ) ∼ D | y =1 and returns x .

In other words, a small-class oracle simulates the following rejection sampling procedure, where a learner keeps drawing x ∼ N (0 , I ) , querying its label and stops when it sees some x 0 with y ( x 0 ) = -1 . Such a procedure requires Ω(1 /p ) queries to implement, which is costly when p is small.

By Theorem 3.1, even without the small-class oracle, the query complexity of Algorithm 3 is always ˜ O δ ( d · polylog(1 /ϵ )) . Thus, a small-class oracle would only help reduce the query complexity of Algorithm 2 and Algorithm 5. In the rest of the section, we show that by calling the small-class oracle ˜ O δ (1) = O δ (polylog(1 /ϵ )) times, we can reduce the query complexity of Algorithm 2 and Algorithm 5 to ˜ O δ ( d · polylog(1 /ϵ )) .

We first consider Algorithm 2. By Theorem 3.8, the query complexity of Algorithm 2 is ˜ O (1 /p + d log(1 /ϵ )) , where Line 3 in Algorithm 2 takes 1 ˜ /p queries to find a random small-class example and Line 4-Line 5 in Algorithm 2 takes ˜ O ( d log(1 /ϵ )) queries. As a small-class oracle simulates the same rejection sampling procedure as Line 3 in Algorithm 2, we can implement Line 3 in Algorithm 2 with a single small-class oracle. Thus, by a single call of the small-class oracle, we are able to implement Algorithm 2 with ˜ O δ ( d · polylog(1 /ϵ )) query complexity.

Next, we consider Algorithm 5. Each implementation of Algorithm 5 runs in O (log log(1 /ϵ )) iterations. In each iteration, we call Algorithm 4 polylog(1 /ϵ ) times in Line 5, use queries to estimate ˆ p s in Line 9, find a single-small class example in Line 11 and improve the current hypothesis with ˜ O ( d ) queries in Line 12. Furthermore, only operations in Line 5, Line 9, and Line 11 have query complexity much larger than polylog(1 /ϵ ) . Thus, we only need to show with a small-class oracle, we can significantly reduce the query complexity of these steps.

We start with Algorithm 4. In Line 9 in Algorithm 4, we use query to estimate the probability of y ( Az -sw ) = -1 with an error up to error p ( b, s ) . By Lemma D.3, we know that if we pass a random sample x ∼ N (0 , I ) to the ( w,s, σ ) -rejection procedure, then the resulting distribution is N ( -sw, A ) . Thus, the probability of y ( Az -sw ) = -1 is exactly equal to the fraction of negative examples among examples that pass the ( w,s, σ ) -rejection procedure. Specifically, for the ( w,s, σ ) -rejection procedure, we denote by q the probability that a random example passes the rejection procedure and denote by q -the probability that a random negative example passes the rejection procedure and p the fraction of the negative example. Then we have Pr z ∼ N (0 ,I ) ( y ( Az -sw ) = -1) = pq -/q . This implies that if we know p and q -, then estimating Pr z ∼ N (0 ,I ) ( y ( Az -sw ) = -1) is equivalent to estimating q -, which can be done by calling the small-class oracle several times and estimate the probability that these examples pass the ( w,s, σ ) -rejection procedure. By Lemma D.3, we know that σ exp( -s 2 / (2(1 -σ 2 ))) can be computed precisely using the parameter s, σ . However, we do not know p precisely, as this requires us to know t ∗ up to a high accuracy. To overcome this difficulty, we use ˆ p , the bias of a halfspace with threshold t , because we only need to ensure the correctness of the algorithm when our guess t is close to t ∗ . In fact, when | t -t ∗ | ≤ 1 / log(1 /ϵ ) , ˆ p ∈ [(1 -1 /C ) p, (1 + 1 /C ) p ] for some large enough constant C , which is enough for ensuring the correctness of Algorithm 4. So, to estimate Pr z ∼ N (0 ,I ) ( y ( Az -sw ) = -1) up to error p ( b, s ) , we only need to estimate q -up to error qp ( b, s ) / ˆ p . We have

where we use Fact D.4 and s = at + ξ, ξ ∈ [0 , b ] . This implies that we only need to call a small class oracle ˜ O δ (1) = O δ (polylog(1 /ϵ )) times to estimate q -and thus can compute Pr z ∼ N (0 ,I ) ( y ( Az -sw ) = -1) up to error p ( b, s ) . In particular, in this implementation, we only need to call the small-class oracle and do not need to make membership queries.

Similarly, to implement Line 9 in Algorithm 5, we also only need to call the small-class oracle ˜ O δ (1) times and do not need to make membership queries.

Finally, we show that by calling the small-class oracle ˜ O δ (1) times, we are able to implement Line 11 in Algorithm 5. By Lemma D.3, Line 11 in Algorithm 5 draws a random negative example that passes the ( w i , s, σ ) -rejection procedure. By Lemma F.3, we know that p s = pq -/q ≥ Ω( p ( b, s )) . This implies that q -≥ Ω( p ( b, s ) q/p ) ≥ Ω(1 / log(1 /ϵ )) , by Equation (5). Thus, with high probability, we only need to pass O δ (log(1 /ϵ )) examples from the small class oracle to the ( w i , s, σ ) -rejection procedure to get one negative example that passes this rejection procedure. Thus, in each iteration of Algorithm 5, we will call ˜ O δ (1) times the small-class oracle and make ˜ O δ ( d ) membership queries.

In summary, we count the number of queries in Algorithm 1 using the new implementation with a small class oracle. Notice that with a small-class oracle, we do not need to worry about using some guess t much larger than t ∗ because the query complexity in the initialization step now has no dependence on the bias of the target halfspace. So we do not need to implement line 4 in Algorithm 1

but only need to guess t ′ = i/ log(1 /ϵ ) for i = 0 , . . . , ⌈ polylog(1 /ϵ ) ⌉ . This means in Algorithm 1, we will call Algorithm 2 and Algorithm 5 in total at most polylog(1 /ϵ ) times, so we will make ˜ O δ (1) small-class oracles and make ˜ O δ ( d · polylog(1 /ϵ )) membership queries.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract summarizes the results provided in Theorem 1.1 and Theorem 1.2. The introduction summarizes the motivations of this paper and describes prior work's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of this paper are discussed in the introduction of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The statement of each theorem provides all the assumptions and we provide complete proofs for all statements that are either in the main body of the paper or in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper is theoretical and does not contain experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper is theoretical and does not contain experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper is theoretical and does not contain experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper is theoretical and does not contain experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper is theoretical and does not contain experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is theoretical and we do not see any immediate implications on society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is theoretical.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This work does not use any assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not use any assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of actively learning non-homogeneous half-spaces under Gaussian distribution. It shows that in the pool based model, any active learner cannot do better than the passive learner in terms of label complexity, unless exponentially many unlabelled examples are drawn. On the other hand, it shows that in the membership query model, there exists active learner that can outperform the passive learner, thus demonstrating a separation between these two models.\n\nSTRENGTHS:\n1. This paper settled down the open problem whether it is possible to learn non-homogeneous half-spaces using $O(d\\log(1/\\epsilon)+\\min(1/p,\\epsilon))$ labelled data and $O(d/\\epsilon)$ unlabelled data under the Gaussian distribution in the pool-based active learning setting. It then shows that this label complexity is achievable under the membership query model, thus there's a separation of the two models. This result and technique used is complete and novel.\n\n2. This paper is well-organized. It gives enough motivation and background information to the problem and discussed related work.\n\n3. The writing is very clearly for most of the part. Notations are well-defined and the math is rigorous.\n\nWEAKNESSES:\n1. (I would adjust my rating based on the response of this point). The only real issue I have here is that I'm not fully convinced by your conclusion that \"in the pool-based model, active learning has no improvements over passive unless the pool size is exponential\". The passive bound has an explicit dependence on $\\epsilon$ while your Theorem 1.1 doesn't. It only has a dependence on $1/p$. If $p$ is constant, your conclusion is not correct. So your conclusion only holds for some regime of $p$. This also makes me wonder how tight this lower bound is, is it possible to have a lower bound with $\\epsilon$ appearing explicitly here?\n\n2. The lower bound holds for the realizable case while the upper bound holds for the upper bound is for the agnostic setting. This leaves a gap in the realizable upper bound and potentially easier algorithms.\n\n3. The paper is getting a bit too technical at the end. All the writings are extremely well, especially the lower bound part, but it gets more technical in section 3 and it's hard to follow for me.\n\n     i) I think it's better to give some examples in section 3, explaining the intuition more and moving some of the lemmas to appendix.\n\n     ii) It feels like your MQ algorithm isn't completely new but used the framework/idea/techniques (like the localization) from previous works. It's better to start with a simpler algorithm and explain what's new in your paper. \n\n     iii) It might be better to have a conclusion section to discuss some future directions.\n\n4. There're some very tiny typos.\n\n   i) In the abstract, your lower bound is $\\Omega(d/\\log(m)\\epsilon)$ while in your theorem it is $\\Omega(d/\\log(m)p)$.\n\n   ii) In algorithm 1, what's the subroutine TEST C? In the appendix you have a subroutine named ANGLE TEST.\n\n   iii) Line 337, you typed \"the length of\" twice.\n\nQUESTIONS:\n1. As I mentioned, is there a better MQ bound or simpler algorithm in the realizable case? In the realizable case is it possible to get $\\text{OPT}+\\epsilon$ exactly?\n\n2. Is there some distribution that is natural and actively can have an advantage over passive learning? I still believe active learning would have some advantages in many instances over passive for learning non-homogenous half-space in the pool-based setting.\n\n3. On the other hand, is it easy to extend your result to more general distributions? Like log-concave distributions?\n\nLIMITATIONS:\nIt's better to have a conclusion section to talk about future directions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-13"}

comments: SUMMARY:
This paper first provides a lower bound on active learning using label queries. This lowerbound is nearly tight compared with upperbound for label queries. To get around the lowerbound, the authors study active learning using membership query, which means the algorithm can directly access the random function that gives the label y to x. The membership query is clearly stronger than label queries and hence, the authors manage to prove an upperbound. The authors also mentioned lowerbounds in membership query from other work.

STRENGTHS:
The settings of active learning using label query and membership query are very natural and well-studied. The authors manage to provide a nearly-tight lowerbound and an interesting upperbound for general halfspace. The strategy they use in both proofs like considering negative examples in samples for lowerbound and finding a subclass using good initialization and fine-tuning seems to be intuitive and interesting.

WEAKNESSES:
I think the results are very interesting but the paper writing can be improved.

QUESTIONS:
How do you know whether to use initialization for an extremely large threshold(section F) or not (section E)?
I think it would make the paper easier to read if the authors could describe what 3.1 is doing at high level. 
I think there is a small notation error in A.2(sample (x,y)\sim D instead of N(0, I)).

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the question of active learning halfspaces over $\\mathbb{R}^d$, under the Gaussian distribution in two different models: Label queries and membership queries. In the label queries model, the authors prove a lower bound implying that the learner must have a pool of size $exp(d)$ in order to do better (in terms of samples/queries) than a passive learner. In the membership queries model, the authors prove that the above lower bound can be circumvented by using the strength of membership queries, and provide an upper bound which is better than the optimal sample complexity bound of passive learners. The upper bound is given by an efficient semi agnostic learner (a learner with risk guarantee of the form $O(opt + \\epsilon)$).\n\nSTRENGTHS:\n1. The studied problem is of a fundamental nature.\n2.  The lower bound and its circumvention via membership queries are interesting.\n3. The upper bound algorithm is efficient.\n\nWEAKNESSES:\n1. I find the introduction not clear enough. For example, I do not understand how the upper bound in line 56 settles with the lower bound in line 54. The upper bound has no dependence on $1/\\epsilon$. Also, how does Theorem 1.1 relate to those bounds? The bounds relate to the uniform distribution while the theorem relates to the Gaussian distribution.\n2. Line 93 says: \"The overhead term cannot be avoided in the agnostic setting...\". It is not discussed whether it can be avoided in the realizable setting.\n3. While the results are potentially interesting, I think that the presentation of the paper needs to be improved. High level ideas and technical details are mixed, which makes the paper hard to follow.\n4. No discussion on possible future research.\n5. The results are applicable to a relatively limited setting of halfspaces under the Gaussian distribution.\n6.  While the upper bound algorithm is efficient, it only gives a \"semi-agnostic\" guarantee in the form of $O(opt + \\epsilon)$. However, the constant hidden in the $O$ notation is not specified (at least not in the theorem), and it can sometimes be important. In some scenarios it might be perfectly reasonable that $opt = 1/1000$. So, if the hidden constant is at least $500$, the guarantee is no better than random guessing in those cases. It could be interesting to design an inefficient learner with better risk guarantees.\n\nQUESTIONS:\nQuestions:\n\n1. In line 157, I don't understand what does it mean that an argument is \"hard to formalize\". Do you mean that it is counter-intuitive?\n3. In line 205, do you mean that an upper bound $\\epsilon$ on the noise level is *given* to the learner beforehand?\n\n\nSuggestions:\n1.  I would recommend writing a \"technical overview\" section that separates the high-level novel ideas from known techniques and technical details.\n2. Add a discussion on future research. For example, in Theorem 1.2: can we do better in the realizable setting? Can we do better by using an inefficient learner?\n\n\nTypos:\n1. Line 337: \"the length of the length of\"\n2. Line 389: \"unfortinately:\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This paper considers active learning of general (non-homogeneous) halfspaces under Gaussian distribution. Define p=P(Y=-1). On the one hand, it proves that, roughly speaking, with standard label queries, one cannot learn a classifier with O(opt+epsilon) error that requires polynomially many unlabeled samples and $O(d\log\frac{1}{\epsilon}+\text{min}(1/p, 1/\epsilon))$ labels. On the other hand, with membership queries, it gives a computationally efficient algorithm that achieves $O(d\text{poly}\log(\frac{1}{\epsilon})+\text{min}(1/p, 1/\epsilon))$.

STRENGTHS:
- This paper considers an interesting open problem learning theory: a lot of research has considered learning homogeneous halfspaces, but what is the limit for non-homogeneous halfspaces? It gives an interesting result that it can't be very efficiently learned under the standard label query paradigm, even assuming gaussian distribution and noise-free, while membership query could help.

- The proposed method is sound and technically non-trivial, though I have not checked proofs in Appendix.

- The paper is written clearly in general, and explains high-level intuitions well.

WEAKNESSES:
- For the upper bound / algorithmic result, it looks to me it is highly depended on the Gaussian (or "nearly" symmetric) assumption. Does that mean that membership queries may not even be enough if the data distribution is more general?

- Also for the upper bound / algorithmic result, it aims at achieving O(opt+epsilon) instead of opt+epsilon error. I understand that in general the latter is proven to be hard, but it would be nice if there is a discussion about if the latter can be achieved with some assumptions about noise.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the classical problem of distribution-dependent---here standard normal disitrbution---active learning of non-homogeneous halfspaces using label and membership queries in the realizable and agnostic setting. There are two main contributions.\n\n1. A strong lower bound in the label query setting essentially showing that no significant improvement against (passive) supervised learning is possible (as long as no exponential number of unlabeled samples are used).\n2. An upper bound in the membership query setting achieved by an efficient (poly-time) algorithm.\n\nThe achieved bounds depend on the bias $p$ (of the target halfspace $w^Tx+p\\geq 0$), which make these results target/hypothesis dependent.\n\nThese results extends previous ones on the simpler problem of learning homogeneous halfspaces, where exact rates have been known and fit into similar line sof work, e.g., results under the uniform distribution on the sphere, instead of Gaussian.\n\nSTRENGTHS:\n* Strong near-optimal bounds for a well studied setting of actively learning halfspaces\n* Devise a near-optimal poly-time algorithm in the membership query setting, while most previous work ignore computational aspects or provide only exponential time algorithms.\n\nWEAKNESSES:\nWhile interesting, technically deep and novel, the results are somewhat limited, due to the quite strong standard normal assumption, see question below.\n\nComments:\n* Maybe consider changing adding something like \"under the Gaussian distribution\" to the title. Otherwise it might seem a bit too general.\n\n\n---rebuttal---\nraised score from 5 to 6.\n\nQUESTIONS:\nWhile interesting, the standard normal assumption is rather strong and limiting. Are there any possibilities to extend your results to broader families of distributions, e.g., sub-Gaussian, log-concave, ... ? Even just more general Gaussians ($\\mathcal{N}(\\mu,\\Sigma)$)?\n\nLIMITATIONS:\nSee question.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}