Title: Near-Optimal Bounds for Learning Gaussian Halfspaces with Random Classification Noise
Venue: NeurIPS 2023
Accepted: yes
Keywords: PAC Learning

Random Classification Noise
Abstract: We study the problem of learning general (i.e., not necessarily homogeneous) 
halfspaces with Random Classification Noise under the Gaussian distribution. 
We establish nearly-matching algorithmic and Statistical Query (SQ) lower bound results 
revealing a surprising information-computation gap for this basic problem. 
Specifically, the sample complexity of this learning problem is 
$\widetilde{\Theta}(d/\epsilon)$, where $d$ is the dimension and $\epsilon$ is the excess error. 
Our positive result is a computationally efficient learning algorithm with sample complexity
$\tilde{O}(d/\epsilon + d/\max(p, \epsilon))^2)$, where $p$ quantifies the bias of the target halfspace. 
On the lower bound side, we show that any efficient SQ algorithm (or low-degree test)
for the problem requires sample complexity at least 
$\Omega(d^{1/2}/(\max(p, \epsilon))^2)$. 
Our lower bound suggests that this quadratic dependence on $1/\epsilon$ is inherent for efficient algorithms.

Near-Optimal Bounds for Learning Gaussian Halfspaces with Random Classification Noise

Abstract

We study the problem of learning general (i.e., not necessarily homogeneous) halfspaces with Random Classification Noise under the Gaussian distribution. We establish nearly-matching algorithmic and Statistical Query (SQ) lower bound results revealing a surprising information-computation gap for this basic problem. Specifically, the sample complexity of this learning problem is ˜ Θ( d/ϵ ) , where d is the dimension and ϵ is the excess error. Our positive result is a computationally efficient learning algorithm with sample complexity ˜ O ( d/ϵ + d/ (max { p, ϵ } ) 2 ) , where p quantifies the bias of the target halfspace. On the lower bound side, we show that any efficient SQ algorithm (or low-degree test) for the problem requires sample complexity at least Ω( d 1 / 2 / (max { p, ϵ } ) 2 ) . Our lower bound suggests that this quadratic dependence on 1 /ϵ is inherent for efficient algorithms.

1 Introduction

A halfspace or Linear Threshold Function (LTF) is any Boolean function h : R d → {± 1 } of the form h ( x ) = sign ( w · x + t ) , where w ∈ R d is the weight vector and t ∈ R is the threshold. The function sign : R → {± 1 } is defined as sign( u ) = 1 if u ≥ 0 and sign( u ) = -1 otherwise. The problem of learning halfspaces is a classical problem in machine learning, going back to the Perceptron algorithm [Ros58] and has had a big impact in both the theory and the practice of the field [Vap98, FS97]. Here we study the problem of PAC learning halfspaces in the distributionspecific setting in the presence of Random Classification Noise (RCN) [AL88]. Specifically, we focus on the basic case in which the marginal distribution on examples is the standard Gaussian one of the simplest and most extensively studied distributional assumptions.

In the realizable PAC model [Val84b] (i.e., when the labels are consistent with a concept in the class), the class of halfspaces on R d is efficiently learnable to 0-1 error ϵ using ˜ O ( d/ϵ ) samples via linear programming (even in the distribution-free setting). This sample complexity upper bound is information-theoretically optimal, even if we know a priori that the distribution on examples is well-behaved (e.g., Gaussian or uniform). That is, in the realizable setting, there is an efficient algorithm for halfspaces achieving the optimal sample complexity (within logarithmic factors).

Learning Gaussian Halfspaces with RCN. The RCN model [AL88] is the most basic model of random noise. In this model, the label of each example is independently flipped with probability exactly η , for some noise parameter 0 < η < 1 / 2 . One of the classical results on PAC learning with RCN [Kea98] states that any Statistical Query (SQ) algorithm can be transformed into an RCN noise-tolerant PAC learner with at most a polynomial complexity blowup. Halfspaces are known to be efficiently PAC learnable in the presence of RCN, even in the distribution-free setting [BFKV97, Coh97, DKT21, DTK22]. Alas, all these efficient algorithms require sample complexity that is suboptimal within polynomial factors in d and 1 /ϵ .

The sample complexity of PAC learning Gaussian halfspaces with RCN is ˜ Θ( d/ ((1 -2 η ) ϵ )) . This bound can be derived, e.g., from [MN06], and the lower bound essentially matches the realizable case, up to a necessary scaling of (1 -2 η ) . 1 Given the fundamental nature of this learning problem, it is natural to ask whether a computationally efficient algorithm with (near-) optimal sample complexity (i.e., within logarithmic factors of the optimal) exists. That is, we are interested in a fine-grained sample size versus computational complexity analysis of the problem. This leads us to the following question:

Is there a sample near-optimal and polynomial-time algorithm for learning Gaussian halfspaces with RCN?

In this paper, we explore the above question and provide two main contributions - essentially resolving the question within logarithmic factors. On the positive side, we give an efficient algorithm with sample complexity ˜ O η ( d/ϵ + d/ (max { p, ϵ } ) 2 ) for the problem. Here the parameter p ∈ [0 , 1 / 2] (Definition 1.2) quantifies the bias of the target function; a 'balanced' function has p = 1 / 2 and a constant function has p = 0 . The worst-case upper bound arises when p = Θ( ϵ ) , in which case our algorithm has sample complexity of ˜ O η ( d/ϵ 2 ) . Perhaps surprisingly, we provide formal evidence that the quadratic dependence on the quantity 1 / max { p, ϵ } in the sample complexity cannot be improved for computationally efficient algorithms . Our lower bounds apply for two restricted yet powerful models of computation, namely Statistical Query algorithms and low-degree polynomial tests. Our lower bounds suggest an inherent statistical-computational tradeoff for this problem.

1.1 Our Results

We study the complexity of learning halfspaces with RCN under the Gaussian distribution. Let C = { f : R d → {± 1 } | f ( x ) = sign( w · x + t ) } be the class of general (i.e., not necessarily homogeneous) halfspaces in R d . The following definition summarizes our learning problem.

̸

Definition 1.1 (Learning Gaussian Halfspaces with RCN) . Let D be a distribution on ( x , y ) ∈ R d ×{± 1 } whose x -marginal D x is the standard Gaussian. Moreover, there exists η ∈ (0 , 1 / 2) and a target f ∈ C such that the label y of example x satisfies y = f ( x ) with probability 1 -η and y = -f ( x ) otherwise. Given ϵ > 0 and sample access to D , the goal is to output a hypothesis h that with high probability satisfies err D 0 -1 ( h ) := Pr D [ h ( x ) = y ] ≤ η + ϵ .

Our main contribution is a sample near-optimal efficient algorithm for this problem coupled with a matching statistical-computational tradeoff for SQ algorithms and low-degree polynomial tests. It turns out that the sample complexity of our algorithm depends on the bias of the target halfspace, defined below.

Definition 1.2 ( p -biased function) . For p ∈ [0 , 1 / 2] , we say that a Boolean function f : R d →{± 1 } is p -biased with respect to the distribution D x , if min { Pr x ∼D x [ f ( x ) = 1] , Pr x ∼D x [ f ( x ) = -1] } = p .

For example, a homogeneous halfspace f ( x ) = sign( w · x ) under the standard Gaussian distribution D x = N ( 0 , I ) satisfies E x ∼D x [ f ( x )] = 0 , and therefore has bias p = 1 / 2 . For a general halfspace f ( x ) = sign( w · x + t ) with ∥ w ∥ 2 = 1 , it is not difficult to see that its bias under the standard Gaussian is approximately p ∼ (1 /t ) exp( -t 2 / 2) (see Fact B.1).

We can now state our algorithmic contribution.

Theorem 1.3. (Main Algorithmic Result) There exists an algorithm that, given ϵ, δ ∈ (0 , 1 / 2) and N samples from a distribution D satisfying Definition 1.1, runs in time O ( dN/ϵ 2 ) and returns a

hypothesis h ∈ C such that with probability at least 1 -δ , it holds err D 0 -1 ( h ) ≤ η + ϵ . The sample complexity of the algorithm is N = ˜ O ( d (1 -2 η ) ϵ + d max( p (1 -2 η ) ,ϵ ) 2 ) log(1 /δ ) .

Some comments are in order. We note that the first term in the sample complexity matches the information-theoretic lower bound (within a logarithmic factor), even for homogeneous halfspaces ( p = 1 / 2 ); see, e.g., [MN06, HY15]. The second term - scaling quadratically with 1 / max { (1 -2 η ) p, ϵ } -is not information-theoretically necessary and dominates the sample complexity when p = O η ( √ ϵ ) . In the worst-case, i.e., when p = O η ( ϵ ) , our algorithm has sample complexity ˜ Θ η ( d/ϵ 2 ) . Perhaps surprisingly, we show in Theorem 1.5 that this quadratic dependence is required for any computationally efficient SQ algorithm; and, via [BBH + 20], for any low-degree polynomial test.

Basics on SQ Model. SQ algorithms are a broad class of algorithms that, instead of having direct access to samples, are allowed to query expectations of bounded functions of the distribution.

Definition 1.4 (SQ algorithms) . Let D be a distribution on R d . A statistical query is a bounded function q : R d → [ -1 , 1] . For u > 0 , the VSTAT( u ) oracle responds to the query q with a value v such that | v -E x ∼ D [ q ( x )] | ≤ τ , where τ = max(1 /u, √ Var x ∼ D [ q ( x )] /u ) . We call τ the tolerance of the statistical query. A Statistical Query algorithm is an algorithm whose objective is to learn some information about an unknown distribution D by making adaptive calls to the corresponding oracle.

The SQ model was introduced in [Kea98] as a natural restriction of the PAC model [Val84a]. Subsequently, the model has been extensively studied in a range of contexts, see, e.g., [Fel16]. The class of SQ algorithms is broad and captures a range of known supervised learning algorithms. More broadly, several known algorithmic techniques in machine learning are known to be implementable using SQs (see, e.g., [FGR + 17, FGV17]).

We can now state our SQ lower bound result.

Theorem 1.5 (SQ Lower Bound) . Fix any constant c ∈ (0 , 1 / 2) and let d be sufficiently large. For any p ≥ 2 -O ( d c ) , any SQ algorithm that learns the class of p -biased halfspaces on R d with Gaussian marginals in the presence of RCN with η = 1 / 3 to error less than η + p/ 3 either requires queries of accuracy better than ˜ O ( pd c/ 2 -1 / 4 ) , i.e., queries to VSTAT( ˜ O ( d 1 / 2 -c /p 2 )) , or needs to make at least 2 Ω( d c ) statistical queries.

Informally speaking, Theorem 1.5 shows that no SQ algorithm can learn p -biased halfspaces in the presence of RCN (with η = 1 / 3 ) to accuracy η + O ( ϵ ) (considering p > ϵ/ 2 ) with a sub-exponential in d Ω(1) many queries, unless using queries of small tolerance - that would require at least Ω( √ d/p 2 ) samples to simulate. This result can be viewed as a near-optimal information-computation tradeoff for the problem, within the class of SQ algorithms. When p = 2 ϵ , the computational sample complexity lower bound we obtain is Ω( √ d/ϵ 2 ) . That is, for sufficiently small ϵ , the computational sample complexity of the problem (in the SQ model) is polynomially higher than its information-theoretic sample complexity.

Via [BBH + 20], we obtain a qualitatively similar lower bound in the low-degree polynomial testing model; see Appendix D.

1.2 Our Techniques

Upper Bound. At a high level, our main algorithm consists of three main subroutines. We start with a simple Initialization (warm-start) subroutine which ensures that we can choose a weight vector w 0 with sufficiently small angle to the target vector w ∗ . This subroutine essentially amounts to estimating the degree-one Chow parameters of the target function and incurs sample complexity ˜ O ( d/ (max( p (1 -2 η ) , ϵ ) 2 ) . We emphasize that our procedure does not require knowing the bias p of the target halfspace; instead, it estimates this parameter to a constant factor.

Our next (and main) subroutine is an optimization procedure that is run for ˜ O (1 /ϵ 2 ) different guesses of the threshold t . At a high level, our optimization subroutine can be seen as a variant of Riemannian (sub)gradient descent on the unit sphere, applied to the empirical LeakyReLU loss - defined as LeakyReLU λ ( u ) = (1 -λ ) u 1 { u ≥ 0 } + λu 1 { u < 0 } -with parameter λ set to η , u = w · x ,

and with samples restricted to a band , namely a < | w · x | < b -with a and b chosen as functions of the guess for the threshold t . The band restriction is key in avoiding Ω( d/ϵ 2 ) dependence in the sample complexity; instead, we only require order-( d/ϵ ) samples to be drawn for the empirical LeakyReLU loss subgradient estimate. Using the band, the objective is restricted to a region where the current hypothesis incorrectly classifies a constant fraction of the mass from which we can perform 'denoising' with constantly many samples.

For a sufficiently accurate estimate ˆ t of t (which is satisfied by at least one of the guesses for which our optimization procedure is run), we argue that there is a sufficiently negative correlation between the empirical subgradient and the target weight vector w ∗ . This result, combined with our initialization, enables us to inductively argue that the distance between the weight vector constructed by the optimization procedure and the target vector w ∗ contracts and becomes smaller than ϵ within orderlog(1 /ϵ ) iterations. This result is quite surprising, since the LeakyReLU loss is nonsmooth (it is, in fact, piecewise linear) and we do not explicitly bound its growth outside the set of its minima (i.e., we do not prove a local error bound, which would typically be used to prove linear convergence). Thus, the result we establish is impossible to obtain using black-box results for nonsmooth optimization. Additionally, we never explicitly use the LeakyReLU loss function or prove that it is minimized by w ∗ ; instead, we directly prove that the vectors w constructed by our procedure converge to the target vector w ∗ . At a technical level, our result is enabled by a novel inductive argument, which we believe may be of independent interest (see Lemma 2.8 for more details).

Since each run of our optimization subroutine returns a different hypothesis, at least one of which is accurate (the one using the 'correct' guess of the threshold t ), we need an efficient way to select a hypothesis with the desired error guarantee. This is achieved via our third subroutine - a simple hypothesis testing procedure, which draws a fresh sample and selects a hypothesis with the lowest test error. By standard results [MN06], such a hypothesis satisfies our target error guarantee.

SQ Lower Bound. To prove our SQ lower bound, it suffices to establish the existence of a large set of distributions whose pairwise correlations are small [FGR + 17]. Inspired by the methodology of [DKS17], we achieve this by selecting our distributions on labeled examples ( x , y ) to be random rotations of a single one-dimensional distribution that nearly matches low-order Gaussian moments, and embedding this in a hidden random direction. Our hard distributions are as follows: We define the halfspaces f v ( x ) = sign( v · x -t ) , where v is a randomly chosen unit vector and the threshold t is chosen such that Pr x ∼N [ f v ( x ) = 1] = p . We then let y = f v ( x ) with probability 2 / 3 , and -f v ( x ) otherwise. By picking a packing of nearly orthogonal vectors v on the unit sphere (i.e., set of vectors with pairwise small inner product), we show that each pair of these f v 's corresponding to distinct vectors in the packing have very small pairwise correlations (with respect to the distribution where x is a standard Gaussian and y is independent of x ). While the results of [DKS17] cannot be directly applied to give our desired corerlation bounds, the Hermite analytic ideas behind them are useful in this context. In particular, the correlation between two such distributions can be computed in terms of their angle and the Hermite spectrum. A careful analysis (Lemma 3.3) gives an inner product that is ˜ O (cos( θ ) p ) , where θ is the angle between the corresponding vectors. Combined with our packing bound, this is sufficient to obtain our final SQ lower bound result.

1.3 Related and Prior Work

A long line of work in theoretical machine learning has focused on developing computationally efficient algorithms for learning halfspaces under natural distributional assumptions in the presence of RCN and related semi-random noise models; see, e.g., [ABHU15, ABHZ16, YZ17, ZLC17, DKTZ20a, DKTZ20b, DKK + 20, DKK + 21, DKK + 22]. Interestingly, the majority of these works focused on the special case of homogeneous halfspaces. We next describe in detail the most relevant prior work.

Prior work [YZ17, ZSA20, ZL21] gave sample near-optimal and computationally efficient learners for homogeneous halfspaces with RCN (and, more generally, bounded noise). Specifically, these works developed algorithms using near-optimal sample complexity of ˜ O η ( d/ϵ ) . However, their algorithms and analyses are customized to the homogeneous case, and it is not clear how to extend them for general halfspaces. In fact, since all of these algorithms are easily implementable in the SQ model, our SQ lower bound (Theorem 1.5) implies that these prior algorithms cannot be adapted to handle the general case without an increase in sample complexity. Finally, [DKTZ22] gave an algorithm with sample complexity ˜ O ( d/ϵ 2 ) to learn general Gaussian halfspaces with adversarial

label noise to error O (OPT) + ϵ , where OPT is the optimal misclassification error. Unfortunately, this algorithm does not suffice for our RCN setting (where OPT = η ), since its error guarantee is significantly weaker than ours.

Very recent work [DDK + 23] gave an SQ lower bound for γ -margin halfspaces with RCN, which has some similarities to ours. Specifically, [DDK + 23] showed that any efficient SQ algorithm for that problem requires sample complexity Ω(1 / ( γ 1 / 2 ϵ 2 )) . Intuitively, the margin assumption allows for a much more general family of distributions compared to our Gaussian assumption here. In particular, the SQ construction of that work does not have any implications in our setting. Even though the Gaussian distribution does not have a margin, it is easy to see that it satisfies an approximate margin property for γ ∼ 1 / √ d . In fact, using an adaptation of our construction, we believe we can quantitatively strengthen the lower bound of [DDK + 23] to Ω(1 / ( γϵ 2 )) . For more details, see Appendix A.

1.4 Preliminaries

For n ∈ Z + , we define [ n ] := { 1 , . . . , n } . We use lowercase bold characters for vectors and uppercase bold characters for matrices. For x ∈ R d and i ∈ [ d ] , x i denotes the i -th coordinate of x , and ∥ x ∥ 2 := ( ∑ d i =1 x i 2 ) 1 / 2 denotes the ℓ 2 -norm of x . We use x · y for the inner product of x , y ∈ R d and θ ( x , y ) for the angle between x and y . We slightly abuse notation and denote by e i the i th standard basis vector in R d . We further use 1 A to denote the characteristic function of the set A , i.e., 1 A ( x ) = 1 if x ∈ A and 1 A ( x ) = 0 if x / ∈ A . We use the standard O ( · ) , Θ( · ) , Ω( · ) asymptotic notation. We also use ˜ O ( · ) to omit poly-logarithmic factors in the argument. We use E x ∼D [ x ] for the expectation of the random variable x according to the distribution D and Pr [ E ] for the probability of event E . For simplicity of notation, we omit the distribution when it is clear from the context. For ( x , y ) distributed according to D , we denote by D x the distribution of x . As is standard, we use N to denote the standard normal distribution in d dimensions; i.e., with its mean being the zero vector and its covariance being the identity matrix.

2 Efficiently Learning Gaussian Halfspaces

In this section, we prove Theorem 1.3 by analyzing Algorithm 1. As discussed in the introduction and shown in Algorithm 1, there are three main procedures in our algorithm. The guarantees of our Initialization (warm start) procedure, which ensures sufficient correlation between the initial weight vector w 0 and the target vector w ∗ , are stated in Section 2.1, while the proofs and pseudocode are in Appendix B.1. Our main results for this section, including the Optimization procedure and associated analysis, are in Section 2.2. The Testing procedure is standard and deferred to Appendix B.2, together with most of the technical details from this section.

Throughout this section, we assume that the parameter η (RCN parameter) is known. As will become clear from our analysis, a constant factor approximation to the value of 1 -2 η is sufficient to obtain our results. For completeness, we show how to obtain such an approximation in Appendix B.3. For simplicity, we present the results for t ≥ 0 and t ≤ √ 2 log((1 -2 η ) /ϵ ) . This is without loss of

Algorithm 1 Main Algorithm

1: Input: δ, η, ϵ, sample access to distribution D

10:

2: [ w 0 , ˆ p ] = Initialization( δ, η, ϵ ) ; ϵ ′ = ϵ/ (1 -2 η ) 3: t 0 = √ 2 log(1 / ˆ p ) , M = 8 ⌈ √ 2(log(4 / ˆ p )) -√ 2 log(1 / ˆ p ) ( ϵ ′ ) 2 ⌉ +1 4: Draw N 2 = O ( d log(1 /δ ) log(1 /ϵ ′ ) (1 -2 η ) 2 ϵ ′ ) samples { ( x ( i ) , y ( i ) ) } N 2 i =1 from D 5: for m = 1 : M do 6: t m = t 0 +( m -1) ( ϵ ′ ) 2 8 , γ m = ϵ ′ 2 exp( t 2 m / 2) 7: ̂ w m = Optimization( w 0 , t m , γ m , η, { ( x ( i ) , y ( i ) ) } N 2 i =1 ) 8: end for 9: [ ̂ w out , t out ] = Testing(( ̂ w 1 , t 1 ) , ( ̂ w 2 , t 2 ) , . . . , ( ̂ w M , t M )) return ̂ w out , t out

Algorithm 2 Optimization

generality. For the former, it is by the simple symmetry of the standard normal distribution that the entire argument translates into the case t < 0 , possibly by exchanging the meaning of '+1' and '-1' labels. For the latter, we note that when the bias is small, i.e., for p ≤ ϵ/ (2(1 -2 η )) , a constant hypothesis suffices.

2.1 Initialization Procedure

Webegin this section with Lemma 2.1, which shows that given N 1 = ˜ O ( d/ ( κ 4 p 2 (1 -2 η ) 2 ) log(1 /δ )) i.i.d. samples from D , we can construct a good initial point w 0 that forms an angle at most κ with the target weight vector w ∗ . For our purposes, κ should be of the order 1 /t. For t ≤ √ 2 log(1 /ϵ ′ ) , where ϵ ′ = ϵ/ (1 -2 η ) , we can ensure that N 1 = ˜ O ( d/ ( p 2 (1 -2 η ) 2 ) log(1 /δ )) . The downside of the lemma, however, is that the number of samples N 1 requires at least approximate knowledge of the bias parameter p (or, more accurately, of e -t 2 / 2 ). We address this challenge by arguing (in Lemma 2.2) that we can estimate p using the procedure described in Algorithm 4, without increasing the total number of drawn samples by a factor larger than orderlog(1 /ϵ ′ ) .

Lemma 2.1 (Initialization via Chow Parameters) . Given κ > 0 , define p t = e -t 2 / 2 , N 1 = O ( d/ ( κ 4 p t 2 (1 -2 η ) 2 ) log(1 /δ )) and let ( x ( i ) , y ( i ) ) for i ∈ [ N 1 ] be i.i.d. samples drawn from D . Let u = 1 N 1 ∑ N 1 i =1 x ( i ) y ( i ) and w 0 = u / ∥ u ∥ 2 . Then, with probability 1 -δ, we have θ ( w 0 , w ∗ ) ≤ κ .

We now leverage Lemma 2.1 to argue about the correctness of implementable Initialization procedure, stated as Algorithm 4 in Appendix B.1, where the proofs for this subsection can be found.

Lemma 2.2. Consider the Initialization procedure described by Algorithm 4 in Appendix B.1. If 0 ≤ t ≤ √ 2 log((1 -2 η ) /ϵ ) , then with probability at least 1 -δ, exp( -t 2 / 2) ≤ ˆ p ≤ 4 exp( -t 2 / 2) . The algorithm draws a total of ˜ O ( d log(1 /δ ) max { (1 -2 η ) p, ϵ } 2 ) samples and ensures that θ ( w 0 , w ∗ ) ≤ min { 1 5 t , π 2 } .

2.2 Optimization

As discussed before, our Optimization procedure (Algorithm 2) can be seen as Riemannian subgradient descent on the unit sphere. Crucial to our analysis is the use of subgradient estimates from Line 5 and Line 6, where we condition on the event that the samples come from a thin band, defined in Line 4. Without this conditioning, the algorithm would correspond to projected subgradient descent of the LeakyReLU loss on the unit sphere. The conditioning effectively changes the landscape of the loss function being optimized, which cannot be argued anymore to even be convex, as the definition of the band depends on the weight vector w at which the vector ̂ g ( w ) is evaluated. Nevertheless, as we argue in this section, the optimization procedure can be carried out very efficiently, even exhibiting a linear convergence rate. To simplify the notation, in this section we denote the conditioned distribution D| E ( w , ˆ t ) by D ( w , ˆ t ) . We carry out the analysis assuming the estimate ˆ t is within additive ϵ 2 of the true threshold value t ; as argued before, this has to be true for at least one estimate ˆ t for which the Optimization procedure is invoked.

In the following lemma, we show that if the angle between a weight vector w and the target vector w ∗ is from a certain range, we can guarantee that g ( w ) is sufficiently negatively correlated with w ∗ . This condition is then used to argue about progress of our algorithm. The upper bound on θ will hold initially, by our initialization procedure, and we will inductively argue that it holds for all iterations. The lower bound, when violated, will imply that the distance between w and w ∗ is small, in which case we would have converged to a sufficiently good solution w .

Lemma 2.3. Fix any ϵ ′ ∈ (0 , 1) . Suppose that 0 ≤ t ≤ √ 2 log(1 /ϵ ′ ) and w ∈ R d is such that ∥ w ∥ 2 = 1 , and θ = θ ( w , w ∗ ) satisfies the inequality ϵ ′ exp( t 2 / 2) ≤ θ ≤ 1 / (5 t ) . If | ˆ t -t | ≤ ϵ ′ 2 / 8 and ˆ γ = (1 / 2) ϵ ′ exp( ˆ t 2 / 2) , then E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w ; x , y ) · w ∗ ] ≤ -(1 -2 η ) sin θ/ (2 √ 2 π ) .

Since, by construction, g ( w ) is orthogonal to w (see Line 5 in Algorithm 2), we can bound the norm of the expected gradient vector by bounding g ( w ) · u for some unit vectors u that are orthogonal to w using similar techniques as in Lemma 2.3. To be specific, we have the following lemma.

Lemma 2.4. Under the assumptions of Lemma 2.3, ∥ ∥ E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w ; x , y )] ∥ ∥ 2 ≤ (1 -2 η ) √ 2 π .

The last technical ingredient that we need is the following lemma which shows a uniform bound on the difference between the empirical gradient ̂ g ( w ) and its expectation (for more details, see Lemma B.6 and Corollary B.8 in Appendix B).

Lemma 2.5. Consider the learning problem from Definition 1.1. Let ϵ ′ , ˆ t, ˆ γ be parameters satisfying the conditions of Lemma 2.3. Let δ ∈ (0 , 1) . Then using ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 ϵ ′ )) samples to construct ̂ g , for any unit vector w such that ϵ ′ exp( t 2 / 2) ≤ θ ( w , w ∗ ) ≤ 1 / (5 t ) , it holds with probability at least 1 -δ : ∥ ̂ g ( w ) -E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w )] ∥ 2 ≤ (1 / 4) ∥ E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w )] ∥ 2 .

̸

We are now ready to present and prove our main algorithm-related result. A short roadmap for our proof is as follows. Since Algorithm 1 constructs a grid with grid-width ϵ ′ 2 / 8 that covers all possible values of the true threshold t , there exists at least one guess ˆ t that is ϵ ′ 2 -close to the true threshold t . We first show that to get a halfspace with error at most ϵ ′ , it suffices to use this ˆ t as the threshold and find a weight vector w such that the angle θ ( w , w ∗ ) is of the order ϵ ′ , which is exactly what Algorithm 2 does. The connection between θ ( w , w ∗ ) and the error is conveyed by the inequality Pr [sign( w · x + t ) = sign( w ∗ · x + t )] ≤ ( θ ( w , w ∗ ) /π ) exp( -t 2 / 2) ; see Appendix B. Let w k be the parameter generated by Algorithm 2 at iteration k for threshold ˆ t . We show that θ ( w k , w ∗ ) converges to zero at a linear rate. To this end, we prove that under our carefully devised step size µ k , there exists an upper bound on ∥ w k -w ∗ ∥ 2 , which contracts at each iteration. Note that since both w k and w ∗ are on the unit sphere, we have ∥ w k -w ∗ ∥ 2 = 2sin( θ ( w k , w ∗ ) / 2) . Essentially, this implies that Algorithm 2 produces a sequence of parameters w k such that θ ( w k , w ∗ ) converges to 0 linearly, under this threshold ˆ t . Thus, we can conclude that there exists a halfspace among all halfspaces generated by Algorithm 1 that achieves ϵ ′ error with high probability.

̸

Theorem 2.6. Consider the learning problem from Definition 1.1. Fix any unit vector w 0 ∈ R d such that θ ( w 0 , w ∗ ) ≤ min(1 / (5 t ) , π/ 2) . Fix any ϵ, δ > 0 . Let ˆ t > 0 be a threshold such that | ˆ t -t | ≤ ϵ 2 / (8(1 -2 η ) 2 ) , and let ˆ γ = ϵ/ (2(1 -2 η )) exp( ˆ t 2 / 2) . Then Algorithm 2 uses N 2 = ˜ O ( d/ ((1 -2 η ) ϵ ) log(1 /δ ) ) samples from D , has runtime ˜ O ( N 2 d ) , and outputs a weight vector w such that h ( x ) = sign( w · x + ˆ t ) satisfies Pr [ h ( x ) = y ] ≤ η + ϵ with probability at least 1 -δ .

̸

Proof. Let ϵ ′ = ( ϵ/ 1 -2 η ) , and denote by w k the vector produced by the algorithm at k th iteration for threshold ˆ t . For any unit vector w and | ˆ t -t | ≤ ϵ ′ 2 / 8 , it holds Pr [sign( w · x + ˆ t ) = sign( w ∗ · x + t )] ≤ ϵ ′ 2 / (4 √ 2 π ) + θ ( w , w ∗ ) /π exp( -t 2 / 2) (see Appendix B.2 for more details). Therefore, it suffices to find a parameter w such that θ ( w , w ∗ ) ≤ πϵ ′ exp( t 2 / 2) . Note that since both w and w ∗ are unit vectors, we have ∥ w -w ∗ ∥ 2 = 2sin( θ/ 2) , indicating that it suffices to minimize ∥ w -w ∗ ∥ 2 efficiently. As proved in Section 2.1, we can start with an initial vector w 0 such that θ ( w 0 , w ∗ ) ≤ 1 / (5 t ) by calling Algorithm 4 (in Appendix B.1). Denote θ k = θ ( w k , w ∗ ) and consider the case when θ k ≥ ϵ ′ exp( t 2 / 2) . We establish the following claim:

Claim 2.7. Let C 1 := (1 -2 η ) / √ 2 π . Drawing N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 ϵ ′ )) samples from distribution D , we have that if θ k ≥ ϵ ′ exp( t 2 / 2) then with probability at least 1 -δ : ∥ w k +1 -w ∗ ∥ 2 2 ≤ ∥ w -w ∗ ∥ 2 2 -( C 1 / 2) µ k sin θ k +4 C 2 1 µ 2 k .

It remains to choose the step size µ k properly to get linear convergence. By carefully designing a shrinking step size, we are able to construct an upper bound ϕ k on the distance of ∥ w k +1 -w k ∥ 2 using Claim 2.7. Importantly, by exploiting the property that both w and w ∗ are on the unit sphere, we show that the upper bound is contracting at each step, even though the distance ∥ w k +1 -w k ∥ 2 could be increasing. Concretely, we have the following lemma.

Lemma 2.8. Let ρ = 0 . 00098 and ϕ k = (1 -ρ ) k . Then, setting µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) it holds sin( θ k / 2) ≤ ϕ k for k = 1 , · · · , K .

Proof. Let ϕ k = (1 -ρ ) k where ρ = 0 . 00098 . This choice of ρ ensures that 32 ρ 2 +1020 ρ -1 ≤ 0 . We show by induction that choosing µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) = (1 -ρ ) k (1 -4 ρ ) / (16 C 1 ) , it holds sin( θ k / 2) ≤ ϕ k . The condition certainly holds for k = 1 since θ 1 ∈ [0 , π/ 2] . Now suppose that sin( θ k / 2) ≤ ϕ k for some k ≥ 1 . We discuss the following 2 cases: ϕ k ≥ sin( θ k / 2) ≥ 3 4 ϕ k and sin( θ k / 2) ≤ 3 4 ϕ k . First, suppose ϕ k ≥ sin( θ k / 2) ≥ 3 4 ϕ k . Since sin( θ k / 2) ≤ sin θ k , it also holds sin θ k ≥ 3 4 ϕ k . Bringing in the fact that ∥ w k +1 -w ∗ ∥ 2 = 2sin( θ k +1 / 2) and ∥ w k -w ∗ ∥ 2 = 2 sin( θ k / 2) , as well as the definition of µ k , the conclusion of Claim 2.7 becomes:

where in the second inequality we used sin θ k ≥ 3 4 ϕ k and in the last equality we used the definition of µ k by which µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) . Since ρ is chosen so that 32 ρ 2 +1020 ρ -1 ≤ 0 , we have:

as desired. Next, consider sin( θ k / 2) ≤ (3 / 4) ϕ k . Recall that w k +1 = proj B ( w k -µ k ̂ g ( w k )) and w k ∈ B , where B is the unit ball 2 ; therefore, ∥ w k +1 -w k ∥ 2 ≤ ∥ w k -µ k ̂ g ( w ) -w k ∥ 2 = µ k ∥ ̂ g ( w k ) ∥ 2 by the non-expansiveness of the projection operator. Furthermore, applying Lemma 2.5 and Lemma 2.4, it holds that ∥ ̂ g ( w k ) ∥ 2 ≤ (5 / 4) ∥ E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w )] ∥ 2 ≤ 2(1 -2 η ) / √ 2 π, i.e., we have ∥ ̂ g ( w k ) ∥ 2 ≤ 2 C 1 ; therefore, ∥ w k +1 -w k ∥ 2 ≤ 2 µ k C 1 , which indicates that:

2(sin( θ k +1 / 2) -sin( θ k / 2)) = ∥ w k +1 -w ∗ ∥ 2 -∥ w k -w ∗ ∥ 2 ≤ ∥ w k +1 -w k ∥ 2 ≤ 2 µ k C 1 . Since we have assumed sin( θ k / 2) ≤ (3 / 4) ϕ k , then it holds:

ϕ k +1 -sin( θ k +1 / 2) ≥ (1 -ρ ) ϕ k -ϕ k + ϕ k -sin( θ k / 2) -(1 -4 ρ ) ϕ k / 16 ≥ 3(1 -4 ρ ) ϕ k / 16 > 0 , since we have chosen µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) . Hence, it also holds that sin( θ k +1 / 2) ≤ ϕ k +1 .

̸

Lemma 2.8 shows that sin( θ k / 2) converges to 0 linearly. Therefore, using N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 ϵ ′ )) samples, after K = O ((1 /ρ ) log(1 / (exp( t 2 / 2) ϵ ′ )) = O (log(1 /ϵ ′ ) iterations, we get a w K such that θ K ≤ 2 sin( θ K / 2) ≤ ϵ ′ exp( t 2 / 2) . Let h ( x ) := sign( w K · x + ˆ t ) . Then it holds that the disagreement of h ( x ) and f ( x ) is bounded by Pr [ h ( x ) = f ( x )] ≤ ϵ ′ (see Appendix B for more details). Finally, since err D 0 -1 ( h ) = Pr ( x ,y ) ∼D [ h ( x ) = y ] = η +(1 -2 η ) Pr x ∼D x [ h ( x ) = sign( w ∗ · x + t )] , for any h : R d ↦→{± 1 } , to get misclassification error at most η + ϵ (with respect to the y ), it suffices to use ϵ ′ = ϵ/ (1 -2 η ) . Therefore, we get Pr ( x ,y ) ∼D [sign( w K · x +ˆ t ) = y ] ≤ η + ϵ , using N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) ϵ )) samples. Since the algorithm runs for O (log(1 /ϵ )) iterations, the overall runtime is ˜ O ( N 2 d ) . This completes the proof of Theorem 2.6.

̸

̸

Proof Sketch of Theorem 1.3. From Lemma 2.2, we get that with ˜ O ( d/ ((1 -2 η ) 2 p 2 )) samples our Initialization procedure (Algorithm 4) produces a unit vector w 0 so that θ ( w 0 , w ∗ ) ≤ min(1 / (5 t ) , π/ 2) with high probability. We construct a grid of ( ϵ 2 / (8(1 -2 η ) 2 ) -separated values, containing all the possible values of the threshold t of size roughly ∼ 1 /ϵ 2 . We run Algorithm 2 for each possible choice of the threshold t . Conditioned on the choice of ˆ t and w 0 that satisfies the assumptions of Theorem 2.6, Algorithm 2 outputs a weight vector ̂ w so that Pr ( x ,y ) ∼D [sign( ̂ w · x +ˆ t ) = y ] ≤ η + ϵ . Using standard concentration facts, we have that with a sample size of order ˜ O ( d/ ((1 -2 η ) ϵ )) from D , we can output the hypothesis with the minimum empirical error with high probability.

̸

̸

3 SQ Lower Bound for Learning Gaussian Halfspaces with RCN

To state our SQ lower bound theorem, we require the following standard definition.

Definition 3.1 (Decision/Testing Problem over Distributions) . Let D be a distribution and D be a family of distributions over R d . We denote by B ( D , D ) the decision (or hypothesis testing) problem in which the input distribution D ′ is promised to satisfy either (a) D ′ = D or (b) D ′ ∈ D , and the goal of the algorithm is to distinguish between these two cases.

Theorem 3.2 (SQ Lower Bound for Testing RCN Halfspaces) . Fix c ∈ (0 , 1 / 2) and let d ∈ N be sufficiently large. For any p ≥ 2 -O ( d c ) , any SQ algorithm that learns the class of (at most) p -biased Gaussian halfspaces on R d in the presence of RCN with η = 1 / 3 to error less than η + p/ 3 either requires queries to VSTAT( ˜ O ( d 1 / 2 -c /p 2 )) , or needs to make at least 2 Ω( d c ) statistical queries.

We note that our SQ lower bound applies to a natural testing version of our learning problem. By a standard reduction (see Lemma C.9), it follows that any learning algorithm for the problem requires either 2 Ω( d c ) many queries or at least one query to VSTAT( ˜ O ( d 2 c -1 / 2 /p 2 )) . We also note that the established bound is tight for the corresponding testing problem (see Appendix C.6).

Proof of Theorem 3.2. For any unit vector v ∈ R d , we define the LTF f v ( x ) = sign( v · x -t ) , where t > 0 and denote p = Pr x ∼N [ f v ( x ) = 1] . Let D v be the distribution on ( x , y ) with respect to f v with the random variable y supported on {± 1 } as follows: Pr [ y = f v ( x ) | x ] = 1 -η and x is distributed as standard normal. Denote by A v the distribution D v conditioned on y = 1 and by B v the distribution D v conditioned on y = -1 . It is easy to see that

and

Fix unit vectors v , u ∈ R d and let θ be the angle between them. We bound from above the correlation between f v ( x ) and f u ( x ) . Our main technical lemma is the following:

Lemma 3.3. Let f v ( x ) and f u ( x ) defined as above. Then it holds

Proof of Lemma 3.3. We start by calculating the Hermite coefficients of the univariate function sign( z -t ) . We will use the fact that E z ∼N [sign( z -t )He i ( z )] = 2 i -1 / 2 He i -1 ( t ) exp( -t 2 / 2) (see Claim C.7). Let c i be the Hermite coefficient of degree i . Without loss of generality (due to the rotational invariance of the Gaussian distribution), we can assume that v = e 1 and u = cos θ e 1 + sin θ e 2 . Using standard algebraic manipulations and orthogonality arguments (see Claim C.5 for more details), we have that

Note that He 0 ( x ) = 1 , therefore c 0 = E x ∼N [ f v ( x )] . Therefore, we have that

Let J = ∑ i ≥ 1 cos i θc 2 i . To complete the proof, it remains to bound the | J | . We show the following:

Claim 3.4. It holds that | J | ≤ 4 | cot( θ ) | exp( -t 2 ) exp ( | cos( θ ) | t 2 ) .

Proof of Claim 3.4. Note that from Claim C.7, we have that c i = 2He i -1 ( t ) exp( -t 2 / 2) / √ i , hence, it holds that J = 4cos( θ ) exp( -t 2 ) ∑ ∞ i =1 i -1 He 2 i -1 ( t ) cos i -1 θ . We use the following fact.

Fact 3.5 (Mehler Formula, see, e.g. [Foa78]) . For | ρ | < 1 and x, y ∈ R , it holds that

Applying Fact 3.5 for ρ = cos( θ ) and x = y = t , we get that

This completes the proof of Claim 3.4.

completing the proof of Lemma 3.3.

We associate each v and u to a distribution D v and D u , constructed as above. The following lemma provides explicit bounds on the correlation between the distributions D v and D u . Recall that the pairwise correlation of two distributions with cdfs D 1 , D 2 with respect to a distribution with cdf D is defined as χ D ( D 1 , D 2 ) + 1 := ∫ x ∈X D 1 ( x ) D 2 ( x ) /D ( x ) (see Definition C.1). We have the following lemma (see Appendix C.4 for its proof):

̸

For any c ∈ (0 , 1 / 2) , there exists a set S of 2 Ω( d c ) unit vectors in R d such that for any pair v = u ∈ S satisfies | v · u | < d -1 / 2+ c (Fact C.4). We associate each v ∈ S with f v and a distribution D v and denote D = { D v , v ∈ S} . By the definition of S and Lemma 3.3, for any v , u ∈ S , we have that E x ∼N [ f v ( x ) f u ( x )] ≤ 4 d -1 / 2+ c exp( -t 2 ) exp ( ( t/d 1 / 4 -c/ 2 ) 2 ) + E x ∼N [ f v ( x )] E x ∼N [ f u ( x )] . Since t/d 1 / 4 -c/ 2 ≤ 1 / 2 by assumption, we get that | E x ∼N [ f v ( x ) f u ( x )] -E x ∼N [ f v ( x )] E x ∼N [ f u ( x )] | ≤ d -1 / 2+ c exp( -t 2 ) . By Lemma 3.6, it follows that χ D 0 ( D v , D u ) ≤ C (1 -2 η ) exp( -t 2 ) d -1 / 2+ c and χ 2 ( D v , D 0 ) ≤ C (1 -2 η ) exp( -t 2 / 2) , where C > 0 is an absolute constant. From standard SQ machinery (see, e.g., Lemma C.3), we have that any SQ algorithm that solves the decision problem B ( D , D 0 ) , requires either 2 Ω( d c ) queries, or at least one query to VSTAT(exp( t 2 ) d 1 / 2+ c ) . Noting that p = O (exp( -t 2 / 2) /t ) (by Fact B.1) completes the proof of Theorem 3.2.

References

[ABHU15] P. Awasthi, M. F. Balcan, N. Haghtalab, and R. Urner. Efficient learning of linear separators under bounded noise. In Proceedings of The 28th Conference on Learning Theory, COLT 2015 , pages 167-190, 2015.

[ABHZ16] P. Awasthi, M. F. Balcan, N. Haghtalab, and H. Zhang. Learning and 1-bit compressed sensing under asymmetric noise. In Proceedings of the 29th Conference on Learning Theory, COLT 2016 , pages 152-192, 2016.

[AL88] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning , 2(4):343370, 1988.

[BBH + 20] M. Brennan, G. Bresler, S. Hopkins, J. Li, and T. Schramm. Statistical query algorithms and low-degree tests are almost equivalent. arXiv preprint arXiv:2009.06107 , 2020.

[BFKV97] A. Blum, A. Frieze, R. Kannan, and S. Vempala. A polynomial time algorithm for learning noisy linear threshold functions. Algorithmica , 22(1/2):35-52, 1997.

[Coh97] E. Cohen. Learning noisy perceptrons by a perceptron in polynomial time. In Proceedings of the Thirty-Eighth Symposium on Foundations of Computer Science , pages 514-521, 1997.

[DDK + 23] I. Diakonikolas, J. Diakonikolas, D. M. Kane, P. Wang, and N. Zarifis. Informationcomputation tradeoffs for learning margin halfspaces with random classification noise. CoRR , abs/2306.16352, 2023. Conference version in COLT'23.

[DKK + 20] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. A polynomial time algorithm for learning halfspaces with Tsybakov noise. arXiv , 2020.

[DKK + 21] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Efficiently learning halfspaces with Tsybakov noise. STOC , 2021.

[DKK + 22] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Learning general halfspaces with general massart noise under the gaussian distribution. In STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 24, 2022 , pages 874-885. ACM, 2022.

[DKS17] I. Diakonikolas, D. M. Kane, and A. Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 73-84, 2017.

[DKS18] I. Diakonikolas, D. M. Kane, and A. Stewart. Learning geometric concepts with nasty noise. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 1061-1073, 2018.

[DKT21] I. Diakonikolas, D. Kane, and C. Tzamos. Forster decomposition and learning halfspaces with noise. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 7732-7744, 2021.

[DKTZ20a] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with massart noise under structured distributions. In Conference on Learning Theory, COLT , 2020.

[DKTZ20b] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with Tsybakov noise. arXiv , 2020.

[DKTZ22] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning General Halfspaces with Adversarial Label Noise via Online Gradient Descent. In Proceedings of the 39th International Conference on Machine Learning , 2022.

[DTK22] I. Diakonikolas, C. Tzamos, and D. M. Kane. A strongly polynomial algorithm for approximate forster transforms and its application to halfspace learning. CoRR , abs/2212.03008, 2022. To appear in STOC'23.

[Fel16] V. Feldman. Statistical query learning. In Encyclopedia of Algorithms , pages 2090-2095. 2016.

[FGR + 17] V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. Statistical algorithms and a lower bound for detecting planted cliques. J. ACM , 64(2):8:1-8:37, 2017.

[FGV17] V. Feldman, C. Guzman, and S. S. Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Philip N. Klein, editor, Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017 , pages 1265-1277. SIAM, 2017.

[Foa78] D. Foata. A combinatorial proof of the mehler formula. Journal of Combinatorial Theory, Series A , 1978.

[FS97] Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

[God55] H. J. Godwin. On generalizations of tchebychef's inequality. Journal of the American Statistical Association , 50(271):923-945, 1955.

[HY15] S. Hanneke and L. Yang. Minimax analysis of active learning. J. Mach. Learn. Res. , 16:3487-3602, 2015.

[Kea98] M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM , 45(6):983-1006, 1998.

[MN06] P. Massart and E. Nedelec. Risk bounds for statistical learning. Ann. Statist. , 34(5):23262366, 10 2006.

[Ros58] F. Rosenblatt. The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review , 65:386-407, 1958.

[Val84a] L. Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, 1984.

[Val84b] L. G. Valiant. A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing (STOC) , pages 436-445. ACM Press, 1984.

[Vap98] V. Vapnik. Statistical Learning Theory . Wiley-Interscience, New York, 1998.

[YZ17] S. Yan and C. Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 , pages 1056-1066, 2017.

[ZL21] C. Zhang and Y. Li. Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise. In Proceedings of The 34th Conference on Learning Theory, COLT , 2021.

[ZLC17] Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, COLT 2017 , pages 1980-2022, 2017.

[ZSA20] C. Zhang, J. Shen, and P. Awasthi. Efficient active learning of sparse halfspaces with arbitrary bounded noise. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

Appendix

Organization The appendix is organized as follows: In Appendix A, we provide additional comparison to prior work. In Appendix B, we present the full version of Section 2, completing proofs and providing supplementary lemmas omitted in the main body. In Appendix C, we start with background on Hermite polynomials and the SQ model, followed by omitted proofs from Section 3. Finally, in Appendix D, we prove our lower bound for low-degree polynomial testing.

A Comparison with Previous Work

Here we provide a more detailed comparison with the very recent work of [DDK + 23], which gives an algorithm and an SQ lower bound for learning γ -margin halfspaces in the presence of RCN. We start by noting that the SQ-hard instance of [DDK + 23] is based on a discrete distribution on the hypercube. Consequently, neither the construction nor its analysis have any implications on the Gaussian setting studied here. More generally, the margin assumption intuitively captures a much more general family of distributions than the Gaussian distribution (though, formally speaking, the two assumptions are incomparable, as the Gaussian only exhibits an approximate margin property). On the positive side, [DDK + 23] gives an efficient algorithm for learning margin halfspaces in the presence of RCN with sample complexity ˜ O (1 / ( γ 2 ϵ 2 )) . It is important to note that the homogeneity (i.e., origin-centered) assumption provably does not help for the margin case - i.e., the case of homogeneous halfspaces with a margin is as hard as the case of general halfspaces with a margin. In sharp contrast, our algorithm in this work has sample complexity that crucially depends on the unknown bias p of the target halfspace - interpolating between ˜ O ( d/ϵ ) and ˜ O ( d/ϵ 2 ) .

B Full Version of Section 2

Throughout this paper, we frequently use the following fact. For x drawn from the standard normal distribution, the threshold t in the definition of the halfspace can be related to the bias of f ( x ) = sign( w ∗ · x + t ) , as follows.

Fact B.1 (Komatsu's Inequality) . For any t ∈ R , the bias p of a halfspace described by f ( x ) = sign( w ∗ · x + t ) can be bounded as:

We begin by stating our main result, in Theorem 1.3 below, and providing a high-level summary of the main algorithm, in Algorithm 3. We note that both the assumption that t ≥ 0 and that t ≤ √ 2 log((1 -2 η ) /ϵ ) are without loss of generality. For the former, it is by the simple symmetry of the standard normal distribution that the entire argument translates into the case t < 0 , possibly by exchanging the meaning of '+1' and '-1' labels. For the latter, we note that when the bias is small, i.e., for p ≤ ϵ/ (2(1 -2 η )) , a constant hypothesis suffices. Thus, only the cases covered by Theorem 1.3 are of interest to us. For the rest of the section, we assume that η is known a priori; we show in Appendix B.3 an efficient way to estimate it without increasing the sample complexity.

Theorem 1.3. (Main Algorithmic Result) There exists an algorithm that, given ϵ, δ ∈ (0 , 1 / 2) and N samples from a distribution D satisfying Definition 1.1, runs in time O ( dN/ϵ 2 ) and returns a hypothesis h ∈ C such that with probability at least 1 -δ , it holds err D 0 -1 ( h ) ≤ η + ϵ . The sample complexity of the algorithm is N = ˜ O ( d (1 -2 η ) ϵ + d max( p (1 -2 η ) ,ϵ ) 2 ) log(1 /δ ) .

Remark B.2. The sample complexity stated in Theorem 1.3 has two components: the first depends on ϵ , and the second depends on the bias p . As we demonstrate in Section 3, the 1 /p 2 term is required for computationally efficient SQ algorithms and low-degree tests. Moreover, the term ˜ O ( d log(1 /δ ) / ((1 -2 η ) ϵ )) is information-theoretically optimal, even when p = 1 / 2 , as shown, e.g., in [HY15].

At a high level, our main algorithm consists of three main subroutines, as summarized in Algorithm 3. The subroutines are specified in the rest of the section, where we analyze the sample complexity and the runtime of our algorithm, and, as a consequence, prove Theorem 1.3. In more detail, the

Algorithm 3 Main Algorithm

10:

Initialization subroutine (specified in Algorithm 4) ensures that we can choose a vector w 0 from the unit sphere that forms a sufficiently small angle with the target vector w ∗ , using ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 p 2 )) samples. This can be seen as warm-start for the main optimization procedure (specified in Algorithm 5). Crucially, the Initialization procedure does not require knowing the bias p of the target halfspace; instead, it estimates this parameter to a constant factor.

The Optimization procedure is run for different guesses of the threshold t ( O (log(1 /ϵ ) /ϵ 2 ) of them). At a high level, it can be seen as a variant of Riemannian (sub)gradient descent on the unit sphere, applied to the empirical LeakyReLU loss - defined as LeakyReLU λ ( u ) = (1 -λ ) u 1 { u ≥ 0 } + λu 1 { u < 0 } -with parameter λ set to η , u = w · x , and with samples restricted to a band , namely a < | w · x | < b -with a and b chosen as functions of the guess for the threshold t . The band restriction is key in avoiding d/ϵ 2 dependence in the sample complexity, and instead only requiring order-( d/ϵ ) samples to be drawn for the empirical LeakyReLU loss subgradient estimate. For a sufficiently accurate estimate ˆ t of t (which is satisfied by at least one of the guesses for which the Optimization procedure is run), we argue that there is a sufficiently negative correlation between the empirical subgradient and the target weight vector w ∗ . This result, combined with the Initialization result, then enables us to inductively argue that the distance between the weight vector constructed by the Optimization procedure and the target vector w ∗ contracts and becomes smaller than ϵ within orderlog(1 /ϵ ) iterations. This result is quite surprising, since the LeakyReLU loss is nonsmooth (it is, in fact, piecewise linear) and we do not explicitly bound its growth outside the set of its minima (i.e., we do not prove a local error bound, which would typically be used to prove linear convergence). Thus, the result as ours is impossible using black-box results for nonsmooth optimization. Additionally, we never even explicitly use the LeakyReLU loss function or prove that it is minimized by w ∗ ; instead, we directly prove that the vectors w constructed by our procedure converge to the target vector w ∗ . On a technical level, our result is enabled by a novel inductive argument, which we believe may be of independent interest.

Finally, since each run of the Optimization subroutine returns a different hypothesis, with only the one(s) with the guess of t being sufficiently accurate satisfying our theoretical guarantee, we need a principled approach to selecting a hypothesis with the target error guarantee. This is achieved in the Testing procedure, which simply draws a fresh sample and selects a hypothesis with the lowest test error. By a standard result due to [MN06], such a hypothesis satisfies our target error guarantee.

B.1 Initialization Procedure

We begin this section with Lemma 2.1, which shows that for any κ > 0 , given N 1 = ˜ O ( d/ ( κ 4 p 2 (1 -2 η ) 2 ) log(1 /δ )) i.i.d. samples from D , we can construct a good initial point w 0 that forms an angle at most κ with the target weight vector w ∗ . For our purposes, κ should be of the order 1 t . For t ≤ 2 √ log(1 /ϵ ′ ) , where ϵ ′ = ϵ/ (1 -2 η ) , we can ensure that N 1 = ˜ O ( d/ ( p 2 (1 -2 η ) 2 ) log(1 /δ )) . The downside of the lemma, however, is that the number of samples N 1 requires at least approximate knowledge of the bias parameter p (or, more accurately, of e -t 2 / 2 ). We address this challenge by arguing (in Lemma 2.2) that we can estimate p using the procedure described in Algorithm 4, without increasing the total number of drawn samples by a factor larger than orderlog(1 /ϵ ′ ) .

Algorithm 4 Initialization

1: Input: δ, η, ϵ > 0

2: Let j = 0

3: repeat

4: j ← j +1

5: p j ← 1 / 2 j

6: Draw n j = ⌈ 32 πd (log(1 /δ )+log log((1 -2 η ) /ϵ )) (1 -2 η ) 2 p 2 j ⌉ new samples from D

7: u j ← 1 n j ∑ n j i =1 y ( i ) x ( i )

8: until ∥ u j ∥ 2 ≥ 3 4 √ 2 π (1 -2 η ) p j or (1 -2 η ) p j ≤ ϵ

9: Let ˆ p = 2 p j , κ = 1 / (5 √ 2(log(4) + log(1 / ˆ p )))

10: Draw N 1 = ⌈ 64 πd log(2 /δ ) κ 4 (1 -2 η ) 2 ˆ p 2 ⌉ new samples from D

11: u ← 1 N 1 ∑ N 1 i =1 x ( i ) y ( i )

12: w 0 ← u / ∥ u ∥ 2

13: return w 0 , ˆ p

Lemma 2.1 (Initialization via Chow Parameters) . Given κ > 0 , define p t = e -t 2 / 2 , N 1 = O ( d/ ( κ 4 p t 2 (1 -2 η ) 2 ) log(1 /δ )) and let ( x ( i ) , y ( i ) ) for i ∈ [ N 1 ] be i.i.d. samples drawn from D . Let u = 1 N 1 ∑ N 1 i =1 x ( i ) y ( i ) and w 0 = u / ∥ u ∥ 2 . Then, with probability 1 -δ, we have θ ( w 0 , w ∗ ) ≤ κ .

Proof. We start by showing that E ( x ,y ) ∼D [ y x ] is parallel to w ∗ and has nontrivial magnitude, and then draw our conclusions from there. In particular, we prove that

To do so, observe first that for any vector v in the orthogonal complement of w ∗ (i.e., such that v · w ∗ = 0 ), x · v (projection of x onto v ) is independent of x · w ∗ , as x is drawn from N . Thus, E ( x ,y ) ∼D [ y x ] · v = 0 , which means that E ( x ,y ) ∼D [ y x ] is either a zero vector or parallel to w ∗ . To determine the magnitude of this vector, we next look at the projection of E ( x ,y ) ∼D [ y x ] onto w ∗ . Using that w ∗ · x is a one-dimensional standard normal random variable, we have

Recalling the definition of y , we now obtain Equation (1). We then conclude from Equation (1) that ∥ E ( x ,y ) ∼D [ y x ] ∥ 2 = √ 2 π (1 -2 η ) p t , as ∥ w ∗ ∥ 2 = 1 , by assumption.

The next step is to show that u = 1 N 1 ∑ N 1 i =1 y ( i ) x ( i ) concentrates near its expectation. Note that y x is √ 2 -sub-Gaussian as x is standard normal, therefore, by the multiplicative Hoeffding bound,

Thus, choosing N 1 = ⌈ 64 πd log(2 /δ ) κ 4 (1 -2 η ) 2 p t 2 ⌉ suffices to guarantee that ∥ u -E ( x ,y ) ∼D [ y x ] ∥ 2 ≤ κ 2 8 ∥ E ( x ,y ) ∼D [ y x ] ∥ 2 , with probability at least 1 -δ .

It remains to bound θ ( w 0 , w ∗ ) , where w 0 = u / ∥ u ∥ 2 . Since w ∗ is a unit vector, we have that with probability at least 1 -δ,

where we have used that E ( x ,y ) ∼D [ y x ] · w ∗ = ∥ E ( x ,y ) ∼D [ y x ] ∥ 2 (by Equation (1)) and ( u -E ( x ,y ) ∼D [ y x ]) · w ∗ ≥ -sup w : ∥ w ∥ 2 =1 ( u -E ( x ,y ) ∼D [ y x ]) · w , which is greater than or equal to -κ 2 8 ∥ E ( x ,y ) ∼D [ y x ] ∥ 2 , by the concentration argument used above.

As cos θ ( w 0 , w ∗ ) ≤ 1 -θ ( w 0 , w ∗ ) 2 / 4 , we further have

yielding the desired result that θ ( w 0 , w ) ≤ κ .

We now leverage Lemma 2.1 to argue about correctness of Algorithm 4, which provides an implementable initialization procedure.

Lemma 2.2. Consider the Initialization procedure described by Algorithm 4 in Appendix B.1. If 0 ≤ t ≤ √ 2 log((1 -2 η ) /ϵ ) , then with probability at least 1 -δ, exp( -t 2 / 2) ≤ ˆ p ≤ 4 exp( -t 2 / 2) . The algorithm draws a total of ˜ O ( d log(1 /δ ) max { (1 -2 η ) p, ϵ } 2 ) samples and ensures that θ ( w 0 , w ∗ ) ≤ min { 1 5 t , π 2 } .

Proof. Let ϵ ′ = ϵ/ (1 -2 η ) and J = ⌈ log 2 (1 /ϵ ′ ) ⌉ . As we have shown in the proof of Lemma 2.1, vector u j satisfies

Since the algorithm runs for at most J = ⌈ log 2 (1 /ϵ ′ ) ⌉ iterations, applying the union bound yields:

where we plugged in n j = ⌈ 32 πd (log(1 /δ )+log log(1 /ϵ ′ )) (1 -2 η ) 2 p 2 j ⌉ . Furthermore, recall that we have proved in Lemma 2.1 that ∥ E [ y x ] ∥ 2 = √ 2 /π (1 -2 η ) exp( -t 2 / 2) . Hence, with probability at least 1 -δ , we have that for all j = 1 , · · · , J ,

Since we have assumed 0 ≤ t ≤ √ 2 log(1 /ϵ ′ ) , it must be exp( -t 2 / 2) ≥ ϵ ′ , thus the claimed bound on ˆ p holds if the algorithm loop ends because j = ⌈ log 2 (1 /ϵ ′ ) ⌉ . Consider now the case that the loop ends before reaching the upper bound on the number of iterations. Observe that when p j is still far away from exp( -t 2 / 2) , i.e., when p j > 2 exp( -t 2 / 2) , Equation (2) shows that with probability at least 1 -δ , ∥ u j ∥ 2 < (3 / 4) √ 2 /π (1 -2 η ) p j ; thus, the algorithm will continue to decrease our guess p j . On the other hand, if p j is already small, i.e., if p j ≤ exp( -t 2 / 2) , Equation (2) implies that ∥ u j ∥ 2 ≥ (3 / 4) √ 2 /π (1 -2 η ) p j , reaching the repeat-until loop termination condition. As any iteration of the algorithm reduces the value of p j by a factor of 2, we conclude that the loop ends with p j that satisfies 1 2 exp( -t 2 / 2) ≤ p j ≤ 2 exp( -t 2 / 2) , hence the bound on ˆ p follows as ˆ p = 2 p j .

The bound on the total number of samples drawn by the algorithm follows by observing that for each iteration j of the repeat-until loop, n j = O ( N 1 ) , while there are O (log(1 /ϵ ′ )) total loop iterations.

To complete the proof, it remains to note that the bound on ˆ p implies

Hence κ selected in the algorithm satisfies κ ≤ 1 5 t . Furthermore, since the algorithm runs for at least one iteration, it holds that ˆ p ≤ 2 p 1 = 1 . Thus, our choice of κ also guarantees that κ ≤ 1 / (5 √ 2 log(4)) ≤ π 2 . It remains to apply Lemma 2.1.

Algorithm 5 Optimization

Input: w 0 , ˆ t, ˆ γ , η , N 2 i.i.d. samples ( x ( i ) , y ( i ) ) from D

10:

B.2 Optimization

As discussed before, our Optimization procedure (Algorithm 5) can be seen as Riemannian subgradient descent on the unit sphere. Crucial to our analysis is the use of subgradient estimates from Line 5 and Line 6, where we condition on the event that the samples come from a thin band, defined in Line 4. Without this conditioning, the algorithm would correspond to projected subgradient descent of the LeakyReLU loss on the unit sphere. The conditioning effectively changes the landscape of the loss function being optimized, which cannot be argued anymore to even be convex, as the definition of the band depends on the weight vector w at which the vector ̂ g ( w ) is evaluated. Nevertheless, as we argue in this section, the optimization procedure can be carried out very efficiently, even exhibiting a linear convergence rate. To simplify the notation, in this section we denote the conditioned distribution D| E ( w , ˆ t ) by D ( w , ˆ t ) . We carry out the analysis assuming the estimate ˆ t is within additive ϵ 2 of the true threshold value t ; as argued before, this has to be true for at least one estimate ˆ t for which the Optimization procedure is invoked.

In the following lemma, we show that if the angle between a weight vector w and the target vector w ∗ is from a certain range, we can guarantee that g ( w ) is sufficiently negatively correlated with w ∗ . This condition is then used to argue about progress of our algorithm. The upper bound on θ will hold initially, by our initialization procedure, and we will inductively argue that it holds for most iterations. The lower bound, when violated, will imply that the distance between w and w ∗ is small, in which case we would have converged to a sufficiently good solution w .

Lemma 2.3. Fix any ϵ ′ ∈ (0 , 1) . Suppose that 0 ≤ t ≤ √ 2 log(1 /ϵ ′ ) and w ∈ R d is such that ∥ w ∥ 2 = 1 , and θ = θ ( w , w ∗ ) satisfies the inequality ϵ ′ exp( t 2 / 2) ≤ θ ≤ 1 / (5 t ) . If | ˆ t -t | ≤ ϵ ′ 2 / 8 and ˆ γ = (1 / 2) ϵ ′ exp( ˆ t 2 / 2) , then E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w ; x , y ) · w ∗ ] ≤ -(1 -2 η ) sin θ/ (2 √ 2 π ) .

Proof. To simplify the notation, in the following we write g ( w ) = g ( w ; x , y ) , as ( x , y ) is clear from the context. Using the definition of conditional expectations as well as the definition of D ( w , ˆ t ) , we have

We carry out the proof by bounding the numerator E ( x ,y ) ∼D [ g ( w ) · w ∗ 1 {E ( w , ˆ t ) } ] . Recall that E ( x ,y ) ∼D [ y | x ] = (1 -2 η )sign( w ∗ · x + t ) . Furthermore, since the Gaussian distribution is rotationally invariant, we can assume without loss of generality that w = e 1 and w ∗ = cos θ e 1 + sin θ e 2 .

Therefore,

where in the final equality, we used the fact that under the event E ( w , ˆ t ) , x 1 = w · x is greater than or equal to -ˆ t and consequently, the expression sign( x 1 + ˆ t ) -sign(cos θ x 1 +sin θ x 2 + t ) equals 2 when cos θ x 1 +sin θ x 2 + t is less than or equal to zero, and it is zero in all other cases.

Recall that under the assumptions of the lemma, | t -ˆ t | ≤ ϵ ′ 2 / 8 . To bound the expectation in Equation (4), we consider two possible cases: t ≥ 1 and t ≤ 1 . The reason that we discuss these two cases is that when t is large, under the condition that -ˆ t ≤ x 1 ≤ -ˆ t +ˆ γ and x 2 sin θ ≤ -t -x 1 cos θ , it is guaranteed that x 2 ≤ 0 . On the other hand, when t is small, it is possible that x 2 ≥ 0 . However, we can show that even though x 2 ≥ 0 in some area of the band, the expectation E [ x 2 1 {E ( w , ˆ t ) , 0 ≤ x 2 sin θ ≤ -t -x 1 cos θ } ] is small since t is very small. This is handled in the following two claims, under the same assumptions as in the statement of the lemma.

Claim B.3. If t ≥ 1 , then

Proof. When t ≥ 1 , under event E ( w , ˆ t ) , we have

Since we have assumed t ≥ 1 and θ ≥ ϵ ′ exp( t 2 / 2) , we further have

where we used that sin( θ/ 2) ≥ θ/ 4 , which holds for any θ ≤ π . Therefore, x 2 ≤ 0 when 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -t -x 1 cos θ } = 1 .

Note that conditioning on E ( w , ˆ t ) we have x 1 ≤ -ˆ t +ˆ γ , hence it holds 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -t -x 1 cos θ } ≥ 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -t -( -ˆ t + ˆ γ ) cos θ } . In addition, since ˆ t ≥ t -ϵ ′ 2 / 8 , we have 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -t -x 1 cos θ } ≥ 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -(1 -cos θ ) t -( ϵ ′ 2 / 8 + ˆ γ ) cos θ } . Therefore, we have the following upper bound:

where we used the trigonometric identity (1 -cos θ ) / sin θ = tan( θ/ 2) . Since x 1 and x 2 are independent standard normal random variables, the expectation on the right-hand side of Equation (5) has the following closed form expression:

Let us now bound t tan ( θ 2 ) + ϵ ′ 2 8 cot θ +ˆ γ cot θ. Using the trigonometric inequalities tan( θ/ 2) ≤ θ/ 2 and cot θ ≤ 1 /θ , which hold for θ ∈ [0 , π/ 2] , and recalling that ϵ ′ exp( t 2 / 2) ≤ θ ≤ 1 / (5 t ) , we have

where the second inequality is by the definition of ˆ γ and the bounds on θ and the third inequality is by ˆ t -t ≤ ϵ ′ 2 8 and ˆ t, t ≤ √ 2 log(1 /ϵ ′ ) . Hence, combining Equation (5)-Equation (7), we get

as claimed.

We now proceed to the case where t < 1 .

Claim B.4. If t < 1 , then

Proof. In this case, it is possible that x 2 ≥ 0 when 1 {E ( w ) , x 2 sin θ ≤ -t -x 1 cos θ } = 1 . However, since ˆ γ = ϵ ′ exp( ˆ t 2 / 2) / 2 ≥ ϵ ′ 2 / 8 , it must be -ˆ t + ˆ γ ≥ -t -ϵ ′ 2 / 8 + ˆ γ ≥ -t , indicating that -t -( -ˆ t + ˆ γ ) cos θ ≤ -t + t cos θ ≤ 0 , hence x 2 ≤ 0 when 1 {E ( w , ˆ t ) , x 2 ≤ -t/ sin θ -( -ˆ t +ˆ γ ) cot θ } = 1 . Therefore, we split the indicator 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -t -x 1 cos θ } into the indicators of three sub-events:

where

For E x ∼D x [ x 2 1 {E 1 } ] , observe first that

Since x 2 ≤ 0 under E 1 , it then holds

following similar steps as in Claim B.3, Equation (5)-Equation (6). Again, note that we have assumed ϵ ′ exp( t 2 / 2) ≤ θ ≤ 1 / (5 t ) and have chosen ˆ γ = ϵ ′ exp( t 2 / 2) / 2 , thus, using the fact that tan( θ/ 2) ≤ θ/ 2 , cot θ ≤ 1 /θ , we further get:

using the same arguments as in Equation (7)-Equation (8). For E x ∼D x [ x 2 1 {E 3 } ] , note that x 2 ≤ 0 under E 3 , hence E x ∼D x [ x 2 1 {E 3 } ] ≤ 0 .

We now study E x ∼D x [ x 2 1 {E 2 } ] . Observe that

where the first inequality results from the condition that x 1 ≥ -ˆ t . Hence, the expectation E x ∼D x [ x 2 1 {E 2 } ] can be upper-bounded by

We again use the fact that tan( θ/ 2) ≤ θ and cot θ ≤ 1 /θ , then recall that ϵ ′ exp( t 2 / 2) ≤ θ ≤ 1 / (5 t ) , ϵ ′ /θ ≤ 1 , thus, we get

Combining Claim B.4 and Claim B.3, we have E x ∼D x [ x 2 1 {E ( w , ˆ t ) , x 2 sin θ ≤ -t -x 1 cos θ } ] ≤ -Pr [ E ( w , ˆ t )] 2 √ 2 π . Thus, plugging this result back to Equation (4) and then combining with Equation (3), we complete the proof of the lemma.

Since, by construction, g ( w ) is orthogonal to w (see Line 5 in Algorithm 5), we can bound the norm of the expected gradient vector by bounding g ( w ) · u for some unit vectors u that are orthogonal to w using similar techniques as in Lemma 2.3. To be specific, we have the following lemma.

Proof. First, we show that for any vector that is orthogonal to both w and w ∗ , the expected gradient of g ( w ) = g ( w ; x , y ) is zero. As a consequence, E [ g ( w )] must lie in the 2-dimensional space spanned by w and w ∗ . To see that, observe first that

where we used proj w ⊥ ( x ) = x -( x · w ) w and v · w = 0 , the fact that v · x is independent of w · x and w ∗ · x , and that E x ∼D x [ v · x ] = 0 . Furthermore, by construction, g ( w ) is orthogonal to w . This indicates that E ( x ,y ) ∼D [ g ( w )] is parallel to w ∗ , as both w and w ∗ are unit vectors. Since x ∼ D x is rotation invariant, we can assume w = e 1 and w ∗ = cos θ e 1 +sin θ e 2 . We thus only need to bound | E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w )] · e 2 | . Recall that D ( w , ˆ t ) is the distribution conditioned on the band E ( w , ˆ t ) = { x : -ˆ t ≤ w · x ≤ -ˆ t + ˆ γ } ; hence, by the definition of g ( w ) , we have

To proceed, we discuss the cases where t ≤ 1 and t ≥ 1 , following similar steps as in Lemma 2.3.

Claim B.5. Under the assumptions of Lemma 2.3, if t ≥ 1 , then

Proof. As shown in the proof of Claim B.3, when t ≥ 1 the condition x 2 ≤ -t/ sin θ -x 1 cot θ , -ˆ t ≤ x 1 ≤ -ˆ t + ˆ γ implies that x 2 ≤ 0 . Hence, in this case we have

and we can further conclude that

Plugging this back into Equation (9) yields the claimed result.

When t ≤ 1 , we use a slightly different decomposition

where

By the definitions of these two events, we have

Since in the proof of Lemma 2.3 we have shown that

it must hold that

Since 1 {E ′ 1 } ≤ 1 {E ( w , ˆ t ) , x 2 ≤ 0 } , we thus have

Plugging this back into Equation (9) completes the proof.

The following lemma establishes a uniform convergence result for the empirical subgradient.

Lemma B.6. Consider the learning problem defined in Definition 1.1. Fix ϵ ′ , δ ∈ (0 , 1 / 2) , and let α be any absolute constant in (0 , 1) . Consider the following class of functions for ( x , y ) ∼ D :

and where ˆ t satisfies | ˆ t -t | ≤ ϵ ′ 2 / 8 and ˆ γ = (1 / 2) ϵ ′ exp( ˆ t 2 / 2) . Then, using N = ˜ O ( d log(1 / ( δ )) (1 -2 η ) 2 ϵ ′ ) samples from D with probability at least 1 -δ , for any g ′ ( w , ˆ t ) ∈ F it holds

Proof. For simplicity, we will use g ′ ( w ) to denote g ′ ( w ; x , y ) and we further define

By definition, g ′ ( w ) is orthogonal to w , hence so is ̂ g ′ ( w ) . As already argued in the proof of Lemma 2.4, E ( x ,y ) ∼D [ g ′ ( w )] is also orthogonal to w . Thus, ∥ ̂ g ′ ( w ) -E ( x ,y ) ∼D [ g ′ ( w )] ∥ 2 is determined by ( ̂ g ′ ( w ) -E ( x ,y ) ∼D [ g ′ ( w )]) · w ′ , where w ′ is a unit vector that is orthogonal to w .

Fix unit vectors w , w ′ ∈ R d with w · w ′ = 0 . We are going to make use of the following variant of Bernstein's inequality (see, e.g., [God55]).

Fact B.7. Let X 1 , . . . , X N be zero mean i.i.d. random variables. Assume that for some positive reals L, σ > 0 , it holds that E [ | X i | k ] ≤ (1 / 2) σ 2 L k -2 k ! . Then, for any x ∈ (0 , √ Nσ 2 / (2 L )) ,

We show that the random variable g ′ ( w ) · w ′ satisfies the assumptions of Fact B.7. For any k ≥ 2 , using the definition of g ′ which enforces g ′ ( w ; x , y ) = 0 whenever 1 { w · x ∈ [ -ˆ t, -ˆ t + ˆ γ ] } = 0 , we have that

√

where the last inequality comes from the fact that for a 2 -sub-Gaussian variable z (e.g., a Gaussian random variable), it holds E [ | z | k ] ≤ ( √ 2 e √ k ) k ≤ C 1 C k -2 2 k ! for some absolute constants C 1 and C 2 , and we choose C to be a large enough multiple of C 1 , C 2 . Let σ 2 = Pr [ w · x ∈ [ -ˆ t, -ˆ t + ˆ γ ]] . Note that since ˆ γ = 1 2 ϵ ′ exp( ˆ t 2 / 2) , we have σ 2 ≥ ˆ γ exp( -ˆ t 2 / 2) = 1 2 ϵ ′ . Then, the condition of Fact B.7 is satisfied with σ 2 ≥ 1 2 ϵ ′ .

Next, we show that we can bound ∥ E ( x ,y ) ∼D [ g ′ ( w )] ∥ 2 from below. In Lemma 2.3 we showed that when ϵ ′ exp( t 2 / 2) ≤ θ ( w , w ∗ ) ≤ 1 / (5 t ) , | ˆ t -t | ≤ ϵ ′ 2 / 8 and ˆ γ = 1 2 ϵ ′ exp( ˆ t 2 / 2) , it holds

Let ˜ w be a unit vector orthogonal to w such that w ∗ = w cos( θ ( w , w ∗ )) + ˜ w sin( θ ( w , w ∗ )) . Then, recalling the fact that E ( x ,y ) ∼D [ g ′ ( w )] is also orthogonal to w , we have:

where in the last line we used Equation (10). Thus, as σ 2 ≤ 1 (it is defined as a probability), we conclude that

Applying Fact B.7, we now get that for any w ′ ∈ R d , ∥ w ′ ∥ 2 = 1 ,

Choosing x = α (1 -2 η ) 2 √ 2 π √ σ 2 N , where α ∈ (0 , 1) is an absolute constant, yields

In particular, since the above inequality holds for any unit w ′ , using Equation (11), it follows that

It remains to show that Equation (12) holds for all functions in F . To do that, we apply the union bound along all the directions w ′ , all the hypothesis w . A cover for these parameters will be of order (1 /ϵ ′ ) O ( d ) . Hence, we have that for any unit vector w ∈ R d with ϵ ′ exp( t 2 / 2) ≤ θ ( w , w ∗ ) ≤ 1 / (5 t ) ,

where in the last inequality, we used that σ 2 ≥ 1 2 ϵ ′ , and N ≥ ˜ O ( d log(1 /δ ) / ( ϵ ′ (1 -2 η ) 2 )) .

Recall that for some fixed w and ˆ t , we have defined the empirical gradient vector in Line 6 as:

where P ( ˆ t, ˆ γ ) = Pr z ∼N [ z ∈ [ -ˆ t, -ˆ t +ˆ γ ]] = Pr [ E ( w , ˆ t )] , since w is a unit vector and w · x follows standard Gaussian. Thus, ̂ g ( w ) = ̂ g ′ ( w ) / Pr [ E ( w , ˆ t )] . In addition, by definition we know that

and so Lemma B.6 immediately implies the following corollary.

Corollary B.8. Consider the learning problem from Definition 1.1. Let ϵ ′ , δ, ˆ t, ˆ γ be parameters satisfying the condition of Lemma B.6 and choose α = 1 / 4 . Then using ˜ O ( d log(1 / ( δ )) / ((1 -2 η ) 2 ϵ ′ )) samples to construct ̂ g , for any unit vector w such that ϵ ′ exp( t 2 / 2) ≤ θ ( w , w ∗ ) ≤ 1 / (5 t ) , it holds with probability at least 1 -δ : ∥ ̂ g ( w ) -E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w )] ∥ 2 ≤ (1 / 4) ∥ E ( x ,y ) ∼D ( w , ˆ t ) [ g ( w )] ∥ 2 .

We are now ready to present and prove our main algorithm-related result. A short roadmap for our proof is as follows. Since Algorithm 3 constructs a grid with grid-width ϵ ′ 2 / 8 that covers all possible values of the true threshold t , there exists at least one guess ˆ t that is ϵ ′ 2 -close to the true threshold t . We first show that to get a halfspace with error at most ϵ ′ , it suffices to use this ˆ t as the threshold and find a weight vector w such that the angle θ ( w , w ∗ ) is of the order ϵ ′ , which is exactly what Algorithm 5 does. The connection between θ ( w , w ∗ ) and the error is conveyed by the following fact:

Fact B.9 (see, e.g., Lemma 4.2 of [DKS18]) . Under the standard normal distribution, it holds:

̸

Let w k be the parameter generated by Algorithm 5 at iteration k for threshold ˆ t . We show that θ ( w k , w ∗ ) converges to zero at a linear rate. To this end, we prove that under our carefully devised step size µ k , there exists an upper bound on ∥ w k -w ∗ ∥ 2 , which contracts at each iteration. Note that since both w k and w ∗ are on the unit sphere, we have ∥ w k -w ∗ ∥ 2 = 2sin( θ ( w k , w ∗ ) / 2) . Essentially, this implies that Algorithm 5 produces a sequence of parameters w k such that θ ( w k , w ∗ ) converges to 0 linearly, under this threshold ˆ t . Thus, we can conclude that there exists a halfspace among all halfspaces generated by Algorithm 3 that achieves ϵ ′ error with high probability.

̸

̸

Theorem 2.6. Consider the learning problem from Definition 1.1. Fix any unit vector w 0 ∈ R d such that θ ( w 0 , w ∗ ) ≤ min(1 / (5 t ) , π/ 2) . Fix any ϵ, δ > 0 . Let ˆ t > 0 be a threshold such that | ˆ t -t | ≤ ϵ 2 / (8(1 -2 η ) 2 ) , and let ˆ γ = ϵ/ (2(1 -2 η )) exp( ˆ t 2 / 2) . Then Algorithm 2 uses N 2 = ˜ O ( d/ ((1 -2 η ) ϵ ) log(1 /δ ) ) samples from D , has runtime ˜ O ( N 2 d ) , and outputs a weight vector w such that h ( x ) = sign( w · x + ˆ t ) satisfies Pr [ h ( x ) = y ] ≤ η + ϵ with probability at least 1 -δ .

Proof. Let ϵ ′ = ϵ 1 -2 η , and denote w k as the parameter produced by the algorithm at k th iteration under threshold ˆ t . Observe that for any unit vector w :

̸

Since | ˆ t -t | ≤ ϵ ′ 2 / 8 , it holds

̸

̸

In addition, as shown in Fact B.9, Pr [sign( w · x + t ) = sign( w ∗ · x + t )] ≤ θ ( w , w ∗ ) π exp( -t 2 / 2) ; thus,

̸

Therefore, it suffices to find a parameter w such that θ ( w , w ∗ ) ≤ πϵ ′ exp( t 2 / 2) . Note that since both w and w ∗ are unit vectors, we have ∥ w -w ∗ ∥ 2 = 2sin( θ/ 2) , indicating that it suffices to minimize ∥ w -w ∗ ∥ 2 efficiently. As proved in Lemma 2.1 and Lemma 2.2, we can start with an initial vector w 0 such that θ ( w 0 , w ∗ ) ≤ 1 / (5 t ) by calling Algorithm 4. Starting from this w 0 , we show that ∥ w k -w ∗ ∥ 2 contracts linearly whenever the angle between w k and w ∗ is larger than ϵ ′ exp( t 2 / 2) , thus we reach the required upper bound for this angle within a logarithmic number of steps. Denote θ k = θ ( w k , w ∗ ) and consider the case when θ k ≥ ϵ ′ exp( t 2 / 2) .

Claim 2.7. Let C 1 := (1 -2 η ) / √ 2 π . Drawing N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 ϵ ′ )) samples from distribution D , we have that if θ k ≥ ϵ ′ exp( t 2 / 2) then with probability at least 1 -δ : ∥ w k +1 -w ∗ ∥ 2 2 ≤ ∥ w -w ∗ ∥ 2 2 -( C 1 / 2) µ k sin θ k +4 C 2 1 µ 2 k .

Proof. Observe first that since ̂ g ( w k ) is orthogonal to w k , we have ∥ w k -µ k ̂ g ( w k ) ∥ 2 2 = ∥ w k ∥ 2 2 + µ 2 k ∥ ̂ g ( w k ) ∥ 2 2 ≥ 1 , thus normalizing w k -µ ̂ g ( w k ) is equivalent to projecting w k -µ k ̂ g ( w k ) to the unit ball B . Since we have assumed w ∗ ∈ B , by the non-expansiveness of the projection operator and proj B ( w ∗ ) = w ∗ , we have:

Thus, expanding the squared norm on the right-hand side yields:

where in the first equality we used the fact that ̂ g ( w k ) and E ( x ,y ) ∼D ( w k , ˆ t ) [ g ( w k )] are both orthogonal to w k . Without loss of generality (because of the rotational invariance), assume w k = e 1 and

̸

̸

̸

w ∗ = cos θ e 1 +sin θ e 2 . Then, again by the fact that both ̂ g ( w k ) and E ( x ,y ) ∼D ( w k , ˆ t ) [ g ( w k )] are orthogonal to w k , we have

Thus, further invoking Lemma 2.3, we have:

Corollary B.8 (or Lemma 2.5) implies that with N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 ϵ ′ )) samples in total, for any unit vector w k satisfying ϵ ′ exp( t 2 / 2) ≤ θ k ≤ 1 / (5 t ) with probability at least 1 -δ , it holds:

Recall that we have shown in the proof of Lemma 2.4 that ∥ E ( x ,y ) ∼D ( w k , ˆ t ) [ g ( w k )] ∥ 2 ≤ 1 -2 η √ 2 π ; therefore, Equation (16) further gives that with probability at least 1 -δ :

Thus, plugging Equation (16) and Equation (17) back into Equation (15), we get that with probability at least 1 -δ ,

Let C 1 := 1 -2 η √ 2 π . Then Equation (18) is simplified to:

completing the proof of this claim.

It remains to choose the step size µ k properly to get linear convergence. By carefully designing a shrinking step size, we are able to construct an upper-bound ϕ k on the distance of ∥ w k +1 -w k ∥ 2 using Claim 2.7. Importantly, by exploiting the property that both w and w ∗ are on the unit sphere, we show that the upper bound is contracting at each step, even though the distance ∥ w k +1 -w k ∥ 2 could be increasing. Concretely, we have the following claim.

Lemma 2.8. Let ρ = 0 . 00098 and ϕ k = (1 -ρ ) k . Then, setting µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) it holds sin( θ k / 2) ≤ ϕ k for k = 1 , · · · , K .

Proof. Let ϕ k = (1 -ρ ) k where ρ = 0 . 00098 . This choice of ρ ensures that 32 ρ 2 +1020 ρ -1 ≤ 0 . We show by induction that choosing µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) = (1 -ρ ) k (1 -4 ρ ) / (16 C 1 ) , it holds sin( θ k / 2) ≤ ϕ k . The condition certainly holds for k = 1 since θ 1 ∈ [0 , π/ 2] . Now suppose that sin( θ k / 2) ≤ ϕ k for some k ≥ 1 . We discuss the following 2 cases: ϕ k ≥ sin( θ k / 2) ≥ 3 4 ϕ k and sin( θ k / 2) ≤ 3 4 ϕ k .

First, suppose ϕ k ≥ sin( θ k / 2) ≥ 3 4 ϕ k . Since sin( θ k / 2) ≤ sin θ k , it also holds sin θ k ≥ 3 4 ϕ k . Bringing in the fact that ∥ w k +1 -w ∗ ∥ 2 = 2sin( θ k +1 / 2) and ∥ w k -w ∗ ∥ 2 = 2sin( θ k / 2) , as well as the definition of µ k , Equation (19) becomes:

where in the second line we used sin θ k ≥ 3 4 ϕ k and in the third line we used the definition of µ k by which µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) . Since ρ is chosen so that 32 ρ 2 +1020 ρ -1 ≤ 0 , we have:

as desired.

Next, consider sin( θ k / 2) ≤ 3 4 ϕ k . Recall that w k +1 = proj B ( w k -µ k ̂ g ( w k )) and w k ∈ B ; therefore, ∥ w k +1 -w k ∥ 2 ≤ ∥ w k -µ k ̂ g ( w ) -w k ∥ 2 = µ k ∥ ̂ g ( w k ) ∥ 2 by the non-expansiveness of the projection operator. Furthermore, applying Equation (17), we have ∥ ̂ g ( w k ) ∥ 2 ≤ 2 C 1 ; therefore, ∥ w k +1 -w k ∥ 2 ≤ 2 µ k C 1 , which indicates that:

Since we have assumed sin( θ k / 2) ≤ 3 4 ϕ k , then it holds:

since we have chosen µ k = (1 -4 ρ ) ϕ k / (16 C 1 ) . Hence, it also holds that sin( θ k +1 / 2) ≤ ϕ k +1 .

Lemma 2.8 shows that sin( θ k / 2) converges to 0 linearly. Therefore, using N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 ϵ ′ )) samples, after K = O ( 1 ρ log(1 / (exp( t 2 / 2) ϵ ′ )) = O (log(1 / ( pϵ ′ )) iterations, we get a w K such that θ K ≤ 2 sin( θ K / 2) ≤ ϵ ′ exp( t 2 / 2) . Let h ( x ) := sign( w K · x + ˆ t ) . Equation (13) then implies that the disagreement of h ( x ) and f ( x ) is bounded by:

̸

Furthermore, for any boolean function h : R d ↦→{± 1 } it holds

̸

̸

̸

Thus, to get misclassification error at most η + ϵ (with respect to the y ), we only need to use ϵ ′ = ϵ/ (1 -2 η ) , and we finally get that Pr ( x ,y ) ∼D [sign( w K · x + ˆ t ) = y ] ≤ η + ϵ , using N 2 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) ϵ )) samples. Since the algorithm runs for O (log(1 /ϵ )) iterations, the overall runtime is ˜ O ( N 2 d ) . This completes the proof of Theorem 2.6.

Proof of Theorem 1.3. From Lemma 2.2, we get that with ˜ O ( d log(1 /δ ) / ((1 -2 η ) 2 p 2 )) samples Algorithm 4 produces a unit vector w 0 so that θ ( w 0 , w ∗ ) ≤ min(1 / (5 t ) , π/ 2) .

Since our guesses of the threshold t m , m ∈ [ M ] form a grid of ( ϵ 2 / (8(1 -2 η ) 2 ) -separated values on the interval [ √ 2 log(1 / ˆ p ) , √ 2 log(4 / ˆ p )] ∋ t , which covers all possible values of the true threshold t , there exists a ¯ m ∈ [ M ] such that | t ¯ m -t | ≤ ϵ 2 / (8(1 -2 η ) 2 ) . Thus, the condition of Theorem 2.6 is satisfied by at least one input threshold. Given w 0 , let ̂ w m , m ∈ [ M ] be the weight vector produced by Algorithm 5 at call m = 1 , · · · , M . From Theorem 2.6, we know that with ˜ O ( d log(1 /δ ) / ((1 -2 η ) ϵ )) samples, with probability at least 1 -δ we get a list of halfspaces { h m ( x ) : h m ( x ) = sign( ̂ w m + t m ) , m = 1 , · · · , M } so that

̸

Finally, to pick the optimal hypothesis from the list, we utilize the following fact.

Fact B.10 (Equation (7) in [MN06]) . Let D be a distribution over R d ×{± 1 } . Let F be a concept set of boolean functions with VC dimension at most d . Let ̂ D be the empirical distribution obtained by drawing ˜ O ( d log(1 /δ ) / ((1 -2 η ) ϵ )) samples from D . Then it holds that

̸

̸

Using the above fact and a sample size of N 3 = ˜ O ( d log(1 /δ ) / ((1 -2 η ) ϵ )) from D , we output the hypothesis with the minimum empirical error. By Fact B.10, we have that this will introduce an error at most ϵ with probability at least 1 -δ . Since M = O (1 /ϵ 2 ) , the total number of calls of Algorithm 5 in Algorithm 3 is O (1 /ϵ 2 ) , and the runtime is ˜ O ( Nd/ϵ 2 ) .

Algorithm 6 Testing Procedure

Input : Hypothesis weight vectors w 1 , w 2 , . . . , w m and thresholds t 1 , t 2 , . . . , t

Draw N 3 samples { ( x , y ) } 3 from D

Calculate the test error (the fraction of misclassified points) for h m ( x ) = sign( ̂ w m · x + t m ) , m ∈ [ M ] , using { ( x ( i ) , y ( i ) ) } N 3 i =1

̂ ̂ ̂ m ( i ) ( i ) N i =1

Let h ¯ m ( x ) = sign( ̂ w ¯ m · x + t ¯ m ) , ¯ m ∈ [ M ] be the halfspace with smallest empirical error. return ̂ w ¯ m , t ¯ m

B.3 The Case Where Both η and p are Unknown

Throughout this section, we carried out the analysis assuming knowledge of the noise parameter η. We now show how to relax this requirement, without changing the sample complexity (up to constant factors) and only affecting the algorithm runtime by a factor 1 /ϵ. In the following lemma, we show that with ˜ O ( d log(1 /δ ) / ( p 2 (1 -2 η ) 2 )) samples we can compute constant factor estimates of the values of p and 1 -2 η, which suffice for determining the correct number of samples to draw in all three subprocedures of our main algorithm (i.e., we can correctly determine N 1 , N 2 , and N 3 ).

Lemma B.11. There is an algorithm that uses ˜ O ( d log(1 /δ ) / ( p 2 (1 -2 η ) 2 )) samples, and with probability at least 1 -δ outputs estimates ˆ p, ˆ η , so that C ˆ p ≥ p ≥ ˆ p and C (1 -2ˆ η ) ≥ (1 -2 η ) ≥ (1 -2ˆ η ) , where C > 0 is a sufficient large absolute constant.

Proof Sketch. We note that Algorithm 4, can, in fact, be used to get an estimate of (1 -2 η ) p instead of only p . Therefore, Algorithm 4 outputs ˆ z so that 2ˆ z ≥ (1 -2 η ) p ≥ ˆ z .

We assume for simplicity that f ( x ) is positively biased, i.e., E x ∼N [ f ( x )] ≥ 0 . Note that E ( x ,y ) ∼D [ 1 { y = b } ] = (1 -2 η ) Pr x ∼N [ f ( x ) = b ] + η , where b ∈ {± 1 } . Because f ( x ) is positively biased, we have that Pr x ∼N [ f ( x ) = 1] ≥ 1 / 2 . Denote the random variable Z as Z = 1 { y = 1 } -1 / 2 . Note that E [ Z ] ≥ (1 / 2)(1 -2 η )( Pr [ f ( x ) = 1] -1 / 2) . Note that if p is less than a sufficiently small constant, then E [ Z ] ≥ (1 / 4)(1 -2 η ) , whereas if p = 1 / 2 , this expectation does not give any useful information. Note that by standard Chernoff bounds, using O ( N log(1 /δ )) samples, where N is a parameter, we can get estimates ̂ Z , so that E [ Z ] ≥ ̂ Z -√ 1 /N . By letting N = O (1 / ˆ z ) , we can distinguish between the cases that (1 -2 η )( Pr [ f ( x ) = 1] -1 / 2) ≤ ˆ z , in

which case we have that p is close to 1 / 2 , therefore ˆ z is an estimate of 1 -2 η that satisfies our requirements. Otherwise, we are in the case where E [ Z ] ≥ (1 / 4)(1 -2 η ) . Therefore, we run the following algorithm: In each round s , we draw N s = O (2 s log(log(1 /ϵ ) /δ )) samples, and we check whether ̂ Z s -√ 1 /N s ≥ 1 / 2 ̂ Z s . If ̂ Z s -√ 1 /N s < 1 / 2 ̂ Z s , we continue, otherwise we stop and return 1 / 2 ̂ Z s , which is an effective lower bound of (1 -2 η ) . The number of rounds is at most log(1 /ϵ ) , so by union bound, the probability of success is at least 1 -δ . After we have estimated an effective lower bound for (1 -2 η ) , we can get an estimate for the value of p , using the estimator ˆ z (recalling that 2ˆ z ≥ (1 -2 η ) p ≥ ˆ z ).

While Lemma B.11 is sufficient for ensuring that the number of samples our algorithm draws is not higher than when assuming the knowledge of η, it is not sufficient for correctly translating the 0-1 error and guaranteeing that it is bounded by η + O ( ϵ ) . However, it is not hard to verify that if we run the Optimization procedure for an estimate ˆ η that is within ± ϵ of η, then the correct 0-1 error bound of η + O ( ϵ ) would follow. This is resolved by simply running the entire optimization component of the algorithm (including all calls to Algorithm 2) for a grid of ϵ -separated values of η in the range (0 , 1 / 2) , which must contain the true value of η . It is immediate that this increases the runtime (and the number of hypothesis halfspaces) by a factor O (1 /ϵ ) . Yet the same number of samples suffices for the optimization and testing, as all that we require is that at least one hypothesis is constructed using estimates of η and t that are sufficiently close to their true values (by orderϵ and orderϵ 2 , respectively, as discussed before).

C Omitted Content from Section 3

C.1 Background on Hermite Polynomials

We define the standard L p norms with respect to the Gaussian measure, i.e., ∥ g ∥ L p = ( E x ∼N [ | g ( x ) | p ) 1 /p . We denote by L 2 ( N ) the vector space of all functions f : R d → R such that E x ∼N [ f 2 ( x )] < ∞ . The usual inner product for this space is E x ∼N [ f ( x ) g ( x )] . While, usually one considers the probabilist's or physicist's Hermite polynomials, in this work we define the normalized Hermite polynomial of degree i to be He 0 ( x ) = 1 , He 1 ( x ) = x, He 2 ( x ) = x 2 -1 √ 2 , . . . , He i ( x ) = ̂ He i ( x ) √ i ! , . . . where by ̂ He i ( x ) we denote the probabilist's Hermite polynomial of degree i . The unnormalized Hermite polynomials are defined as ̂ He i ( z ) exp( -z 2 / 2) = ( -1) i d i exp( -z 2 / 2) d z i . The normalized Hermite polynomials He 1 , He 2 , . . . , He i , . . . form a complete orthonormal basis for the single dimensional version of the inner product space defined above. To get an orthonormal basis for L 2 ( N ) , we use a multi-index V ∈ N d to define the d -variate normalized Hermite polynomial as He V ( x ) = ∏ d i =1 He v i ( x i ) . The total degree of He V is | V | = ∑ v i ∈ V v i . Given a function f ∈ L 2 , we compute its Hermite coefficients as ˆ f ( V ) = E x ∼N [ f ( x )He V ( x )] and express it uniquely as ∑ V ∈ N d ˆ f ( V )He V ( x ) .

C.2 Additional Background on the SQ Model

To define the SQ dimension, we need the following definition.

Definition C.1 (Pairwise Correlation) . The pairwise correlation of two distributions with probability density functions (pdfs) D 1 , D 2 : X → R + with respect to a distribution with pdf D : X → R + , where the support of D contains the supports of D 1 and D 2 , is defined as χ D ( D 1 , D 2 ) + 1 := ∫ x ∈X D 1 ( x ) D 2 ( x ) /D ( x )d x . Wesay that a collection of s distributions D = { D 1 , . . . , D s } over X is ( γ, β ) -correlated relative to a distribution D if | χ D ( D i , D j ) | ≤ γ for all i = j , and | χ D ( D i , D j ) | ≤ β for i = j .

̸

The following notion of dimension effectively characterizes the difficulty of the decision problem.

Definition C.2 (SQ Dimension) . For γ, β > 0 , a decision problem B ( D , D ) , where D is fixed and D is a family of distributions over X , let s be the maximum integer such that there exists D D ⊆ D such that D D is ( γ, β ) -correlated relative to D and | D D | ≥ s . We define the Statistical Query dimension with pairwise correlations ( γ, β ) of B to be s and denote it by SD( B , γ, β ) .

The connection between SQ dimension and lower bounds is captured by the following lemma.

Lemma C.3 ([FGR + 17]) . Let B ( D , D ) be a decision problem, where D is the reference distribution and D is a class of distributions over X . For γ, β > 0 , let s = SD( B , γ, β ) . Any SQ algorithm that solves B with probability at least 2 / 3 requires at least s · γ/β queries to the VSTAT(1 /γ ) oracle.

In order to construct a large set of nearly uncorrelated hypotheses, we need the following fact:

̸

Fact C.4 (see, e.g., [DKS17]) . Let d ∈ Z + . Let 0 < c < 1 / 2 . There exists a collection S of 2 Ω( d c ) unit vectors in R d , such that any pair v , u ∈ S , with v = u , satisfies | v · u | < d -1 / 2+ c .

C.3 Omitted Details from the Proof of Theorem 3.2

Claim C.5. Let f v ( x ) = sign( v · x -t ) and f u ( x ) = sign( u · x -t ) and let c i be the Hermite coefficient of He i . Then, it holds E x ∼N [ f v ( x ) f u ( x )] = ∑ ∞ i =0 cos i θ c 2 i , where θ is the angle between v and u .

Proof. We first need the following standard fact about the rotations of Hermite polynomials (see, e.g., Fact D.1 in [DKS17]):

Fact C.6. For θ ∈ R , it holds He i ( x cos( θ ) + y sin θ ) = ∑ i j =0 ( i j ) cos j θ sin i -j θ He j ( x )He i -j ( y ) .

We have that

where in the third equality, we used Fact C.6 and the orthogonality of the Hermite polynomials with respect to the Gaussian.

We prove the following.

Proof. Denote as ̂ He i ( z ) the non-normalized Hermite polynomial of order d . The Hermite polynomials are defined as follows:

By taking the derivative over z (which exists as He i is a polynomial and exp( -z 2 / 2) is differentiable), we have that ∫ ( ̂ He i ( z ) exp( -z 2 / 2)) = -̂ He i -1 ( z ) . Therefore, we have that

where we used that ∫ z ∈ R ̂ He i ( z ) G ( z )d z = 0 by the orthogonality of the Hermite Polynomials with respect to the Gaussian measure. Furthermore, using that ∫ ( ̂ He i ( z ) exp( -z 2 / 2)) = -̂ He i -1 ( z ) we get that

By normalizing the Hermite polynomial, we complete the proof of Claim C.7.

C.4 Proof of Lemma 3.6

We restate and prove the following lemma.

Lemma C.8. Let D 0 be a product distribution over N × {± 1 } , where Pr ( x ,y ) ∼ D 0 [ y = 1] = Pr ( x ,y ) ∼ D v [ y = 1] = p . We have χ D 0 ( D v , D u ) ≤ 2(1 -2 η )( E [ f v ( x ) f u ( x )] -E [ f v ( x )] E [ f u ( x )]) and χ 2 ( D v , D 0 ) ≤ (1 -2 η )( E [ f v ( x )] -E [ f v ( x )] 2 ) .

Proof. We have that

We bound each term. Note that by construction A v ( x ) = G ( x )( η +(1 -2 η ) 1 { f v ( x ) > 0 } ) / ( η + (1 -2 η ) E x ∼N [ 1 { f v ( x ) > 0 } ]) . Note that 1 { f v ( x ) > 0 } = ( f ( x ) + 1) / 2 , therefore A v ( x ) = G ( x )(1 + (1 -2 η ) f v ( x )) / (1 + (1 -2 η ) E x ∼N [ f v ( x )]) . Therefore,

Using the above, we get that χ N ( A v , A u ) = (1 -2 η ) /p ( E [ f v ( x ) f u ( x )] -E [ f v ( x )] E [ f u ( x )]) . Similarly, we get that χ N ( B v , B u ) = (1 -2 η ) / (1 -p )( E [ f v ( x ) f u ( x )] -E [ f v ( x )] E [ f u ( x )]) . It remains to bound χ 2 ( D v , D 0 ) . Note that χ 2 ( D v , D 0 ) = χ D 0 ( D v , D v ) , hence, χ 2 ( D v , D 0 ) ≤ (1 -2 η ) E x ∼N ([ f v ( x )] -E x ∼N [ f v ( x )] 2 ) .

C.5 Reduction of Testing to Learning

Lemma C.9 (Reduction of Testing to Learning) . Any algorithm that learns halfspaces with η = 1 / 3 RCN noise can be used to solve the decision problem of Theorem 3.2.

̸

Proof. Assume that there is an algorithm A which given ϵ > 0 and distribution D with Gaussian x - marginals and corrupted with η = 1 / 3 random classification noise, outputs a hypothesis h with Pr ( x ,y ) ∼ D [ h ( x ) = y ] ≤ η + ϵ . We can use A to solve the decision problem B ( D 0 , D ) . Note that if the distribution were D 0 , then any hypothesis would get error at least η + (1 -2 η ) p (as y is independent of x ). If the distribution were one in the set D , then the algorithm for ϵ = (1 -2 η ) p/ 2 would give a hypothesis such that Pr ( x ,y ) ∼ D [ h ( x ) = y ] ≤ η + (1 -2 η ) p/ 2 . So making one additional query of tolerance (1 -2 η ) p would be able to solve the decision problem. This completes the proof.

C.6 Solving the Decision Problem Efficiently

In this section, we show that our SQ lower bound (Theorem 3.2) for the testing problem is, in fact, tight. We prove the following:

Theorem C.10 (Efficient Algorithm for Testing) . Let d ∈ N and ϵ ∈ (0 , 1) and let D be a distribution supported on R d ×{± 1 } such that D x is the standard Gaussian on R d . There exists an algorithm that, given N = C √ d/ ( ϵ 2 log(1 /ϵ )) samples from D , where C > 0 is a sufficiently large absolute constant, distinguishes between the following cases with probability of error at most 1/3:

x is independent of y , where ( x , y ) ∼ D .

y is f ( x ) corrupted with RCN for η = 1 / 3 , where f is an LTF with Pr [ f ( x ) = 1] = ϵ .

Proof. Let Z = y x be the random variable where ( x , y ) ∼ D and let Z N = (1 /N ) ∑ N i =1 x ( i ) y ( i ) be the random variables where ( x ( i ) , y ( i ) ) , for i ∈ [ N ] are samples drawn from D . The tester works as follows: If ∥ Z N ∥ 2 2 > d/N + cϵ 2 log(1 /ϵ ) , where c > 0 is a sufficiently small universal constant, we answer that we are in Case 2, otherwise that we are in the Case 1. We prove the correctness of the algorithm below.

̸

We note that if we are in the Case 1, then Z follows the standard Gaussian distribution. To see that, we show the following simple claim:

Claim C.11. Let x be distributed as standard normal and let y supported in {± 1 } be a random variable independent of x , then y x is distributed as standard normal.

Proof. Weassume that y = 1 with probability 1 -η for some η ∈ (0 , 1) . Let ϕ A ( t ) = E [exp( itA )] be the characteristic function of A . Then, we have that ϕ y x ( t ) = E [exp( ity x )] = (1 -η ) E [exp( it x )] + η E [exp( -it x )] = (1 -η ) ϕ x ( t ) + ηϕ -x ( t ) = ϕ x ( t ) , where in the last equality we used that ϕ -x ( t ) = ϕ x ( t ) as the standard normal distribution is symmetric. Therefore, ϕ y x ( t ) = ϕ x ( t ) , hence the distribution of y x and x is the same.

From Claim C.11, we have that Z follows standard normal distribution if we are in Case 1 and therefore Z N follows N ( 0 , I /N ) . Hence, ∥ Z N ∥ 2 2 has mean d/N and standard deviation O ( √ d/N ) . By Chebyshev's inequality, we answer correctly in this case with a probability of at least 2 / 3 .

We next analyze the case where Z is in Case 2. Let f ( x ) = sign( v · x + t ) be the defining halfspace. Then for any u orthogonal to v , we have that the random variables y and ( x · u ) are independent as y only depends on x · v . Therefore, by Claim C.11, we have that y ( x · u ) is standard normal, therefore y x ⊥ v is standard ( d -1) -dimensional normal. Furthermore, note that from Equation (1) that E [ y x · v ] = Θ( ϵ √ log(1 /ϵ )) and Var[ y x · v ] = Θ(1) . Therefore, we can write ∥ Z N ∥ 2 2 = (1 /N )( ∑ d -1 i =1 g 2 i + E 2 ) , where g i are distributed as standard normal and E 2 is the contribution due to the noise. We show that with probability at least 2 / 3 , it holds that ∥ Z N ∥ 2 2 ≥ d/N + cϵ 2 log(1 /ϵ ) . Note that with probability at least 9 / 10 , we have that | E | = Θ( ϵ √ log(1 /ϵ )) by Chebyshev's inequality. Due to the independence of the directions, the random variables g i for i = 1 , . . . , d -1 are independent of E . Hence, conditioned on the event that | E | = Θ( ϵ √ log(1 /ϵ )) , we have that ∥ Z n ∥ 2 2 has mean ( d -1) /N +Θ( ϵ 2 log(1 /ϵ )) and standard deviation O ( √ d/N ) . Hence, again the tester would succeed with a probability of at least 2 / 3 .

C.7 SQ Algorithm for Learning Halfspaces with RCN with Exponential Number of Queries

Here we show that there exists a query-inefficient SQ algorithm that can be simulated with nearoptimal sample complexity of ˜ O ( d/ϵ ) .

Lemma C.12 (Inefficient SQ Algorithm) . There is an SQ algorithm that makes 2 O ( d )polylog(1 /ϵ ) queries to VSTAT(1 /ϵ ) , and learns the class of halfspaces on R d in the presence of RCN with η = 1 / 3 with error at most η + ϵ .

̸

Proof Sketch. We note that it is always possible to design an exponential query SQ algorithm that achieves the optimal sample complexity of ˜ O ( d/ϵ ) (if we were to simulate it with samples). One approach is to generate an ϵ -cover G that encompasses all hypotheses in R d (with size roughly (1 /ϵ ) d ), and then utilize the query function f ( x , y ) = 1 { h 1 ( x ) = y } -1 { h 2 ( x ) = y } for any h 1 , h 2 ∈ G . It can be readily observed that the variance of f is at most the probability that h 1 ( x ) = h 2 ( x ) , which means that VSTAT(1 /ϵ ) can distinguish which hypothesis, h 1 or h 2 , yields a smaller error. Note that if we were to simulate this SQ algorithm using samples, we would also need to do a union bound over the set of all hypotheses.

D Lower Bound for Low-Degree Polynomial Testing

D.1 Preliminaries: Low-Degree Method

We begin by recording the necessary notation, definitions, and facts. This section mostly follows [BBH + 20].

Low-Degree Polynomials A function f : R a → R b is a polynomial of degree at most k if it can be written in the form

̸

̸

where each f i : R a → R is a polynomial of degree at most k . We allow polynomials to have random coefficients as long as they are independent of the input x . When considering list-decodable estimation problems, an algorithm in this model of computation is a polynomial f : R d 1 × n → R d 2 × ℓ , where d 1 is the dimension of each sample, n is the number of samples, d 2 is the dimension of the output hypotheses, and ℓ is the number of hypotheses returned. On the other hand, [BBH + 20] focuses on binary hypothesis testing problems defined in Definition D.2.

A degreek polynomial test for Definition D.2 is a degreek polynomial f : R d × n → R and a threshold t ∈ R . The corresponding algorithm consists of evaluating f on the input x 1 , . . . , x n and returning H 0 if and only if f ( x 1 , . . . , x n ) > t .

Definition D.1 ( n -sample ϵ -good distinguisher) . We say that the polynomial p : R d × n ↦→ R is an n -sample ϵ -distinguisher for the hypothesis testing problem in Definition D.2 if

We call ϵ the advantage of the distinguisher.

Let C be the linear space of polynomials with a degree at most k . The best possible advantage is given by the low-degree likelihood ratio

where we denote ¯ D u = D u /D 0 and the notation f ≤ k denotes the orthogonal projection of f to C .

Another notation we will use regarding a finer notion of degrees is the following: We say that the polynomial f ( x 1 , . . . , x n ) : R d × n → R has samplewise degree ( r, k ) if it is a polynomial, where each monomial uses at most k different samples from x 1 , . . . , x n and uses degree at most r for each of them. In analogy to what was stated for the best degreek distinguisher, the best distinguisher of samplewise degree ( r, k ) -achieves advantage ∥ ∥ E u ∼ µ [( ¯ D ⊗ n u ) ≤ r,k ] -1 ∥ ∥ D ⊗ n 0 the notation f ≤ r,k now means the orthogonal projection of f to the space of all samplewise degree-( r, k ) polynomials with unit norm.

We begin by formally defining a hypothesis problem.

Definition D.2 (Hypothesis testing) . Let D 0 be a distribution and S = { D u } u ∈ S be a set of distributions on X . Let µ be a prior distribution on the indices S of that family. We are given access (via i.i.d. samples or oracle) to an underlying distribution where one of the two is true:

H 0 : The underlying distribution is D 0 .

H 1 : First u is drawn from µ and then the underlying distribution is set to be D u .

We say that a (randomized) algorithm solves the hypothesis testing problem if it succeeds with non-trivial probability (i.e., greater than 0 . 9 ).

Definition D.3. Let D 0 be the joint distribution over the pairs ( x , y ) ∈ R d ×{± 1 } where x ∼ N and y ∼ D 0 ( y ) independently of x . Let D v be the joint distribution over pairs ( x , y ) ∈ R d ×{± 1 } where the marginal on y is again D 0 ( y ) but the conditional distribution E v ( x | 1) is of the form A v (as in Theorem 3.2) and the conditional distribution E v ( x | -1) is of the form B v . Define S = { E v } v ∈ S for S being the set of d -dimensional nearly orthogonal vectors from Fact C.4 and let the hypothesis testing problem be distinguishing between D 0 vs. S with prior µ being the uniform distribution on S .

In this section, we prove the following:

Theorem D.4. Let 0 < c < 1 / 2 . Consider the hypothesis testing problem of Definition D.3. For d ∈ Z + with d larger than an absolute constant, any n ≤ Ω( d ) 1 / 2 -c /p 2 and any even integer k < d c/ 4 , we have that

We need the following variant of the statistical dimension from [BBH + 20], which is closely related to the hypothesis testing problems considered in this section. Since this is a slightly different definition from the statistical dimension ( SD ) used so far, we will assign the distinct notation ( SDA ) for it.

Notation For f : R → R , g : R → R and a distribution D , we define the inner product ⟨ f, g ⟩ D = E X ∼ D [ f ( X ) g ( X )] and the norm ∥ f ∥ D = √ ⟨ f, f ⟩ D .

Definition D.5 (Statistical Dimension) . For the hypothesis testing problem of Definition D.2, we define the statistical dimension SDA( S , µ, n ) as follows:

We will omit writing µ when it is clear from the context.

D.2 Proof of Theorem D.4

To prove Theorem D.4, we first need to bound the SDA of our setting. The following lemma translates the ( γ, β ) -correlation of S to a lower bound for the statistical dimension of the hypothesis testing problem. The proof is very similar to that of Corollary 8.28 of [BBH + 20] but it is given below for completeness.

Lemma D.6. Let 0 < c < 1 / 2 and d, m ∈ Z + . Consider the hypothesis testing problem of Definition D.3. Then, for any q ≥ 1 ,

Proof. The first part is to calculate the correlation of the set S . By Theorem 3.2, we know that the set S is ( γ, β ) -correlated with γ = p 2 Ω( d ) c -1 / 2 and β = 4 p .

We next calculate the SDA according to Definition D.5. We denote by ¯ E v the ratios of the density of E v to the density of R . Note that the quantity ⟨ ¯ E u , ¯ E v ⟩ -1 used there is equal to ⟨ ¯ E u -1 , ¯ E v -1 ⟩ . Let E be an event that has Pr u , v ∼ µ [ E ] ≥ 1 /q 2 . For d sufficiently large we have that

where the first inequality uses that Pr [ u = v | E ] = Pr [ u = v , E ] / Pr [ E ] and bounds the numerator in two different ways: Pr [ u = v , E ] / Pr [ E ] ≤ Pr [ u = v ] / Pr [ E ] = 1 / ( |S| Pr [ E ]) and Pr [ u = v , E ] / Pr [ E ] ≤ Pr [ E ] / Pr [ E ] = 1 .

In [BBH + 20], the following relation between SDA and low-degree likelihood ratio is established.

Fact D.7 (Theorem 4.1 of [BBH + 20]) . Let D be a hypothesis testing problem on R d with respect to null hypothesis D 0 . Let n, k ∈ N with k even. Suppose that for all 0 ≤ n ′ ≤ n , SDA( S , n ′ ) ≥ 100 k ( n/n ′ ) k . Then, for all r , ∥ ∥ E u ∼ µ [ ( ¯ D ⊗ n u ) ≤ r, Ω( k ) ] -1 ∥ ∥ 2 D ⊗ n 0 ≤ 1 .

In Lemma D.6 we set n = Ω( d ) 1 / 2 -c /p 2 and q = √ 2 Ω( d c/ 2 ) ( n/n ′ ) . Then, SDA( S , n ′ ) ≥ √ 2 Ω( d c/ 2 ) ( n/n ′ ) ≥ (100 n/n ′ ) k for k < d c/ 4 and then we apply the theorem above.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of learning $d$-dimensional halfspaces $h(x) = \\mathrm{sign}(w \\cdot x + t)$ over Gaussian marginals under random classification noise (where with probability $\\eta$, a sample is given the incorrect label). While the homogeneous case (where $t = 0$ and the bias $p = \\frac12$) by prior works, the paper focuses on the in-homogenous case where $t$ is unknown. The results give a nearly tight characterization of this regime:\n* Theorem 1.3 shows that there exists a polynomial-time algorithm for learning in-homogeneous halfspaces that uses $N = \\tilde{O}(d / ((1 - \\eta)\\epsilon) + d / \\max(p(1 - 2\\eta), \\epsilon)^2) \\log \\frac1\\delta$ samples. The algorithm (given and analyzed in Section 2 and Appendix B) works as follows:\n  * The Initialization procedure chooses a weight vector $w_0$ and predicts the bias $\\hat{p}$ by iteratively averaging together signed samples until the norm of their average is sufficiently large.\n  * The Optimization procedure locally updates $w_0$ by running Riemannian subgradient descent on a band of inputs on unit sphere of width $\\hat{t}$, for each $\\hat{t}$ belonging in an $\\epsilon$-net over all thresholds $t$.\n  * The Testing procedure identifies which threshold gives the lowest error on a new sample and outputs such a classifier.\n* Theorem 1.5 shows the near-optimality of the upper bound by showing that SQ-learning $p$-biased halfspaces under constant RCN requires either an exponential number of queries or at queries to an oracle of accuracy $d^{1/2 - c} / p^2$, whose dependence on $p$ matches the upper bounds.\n\nSTRENGTHS:\nThe paper is well-written, the bounds are mathematically interesting and have clear proofs, and the matching dependence on $1 / p^2$ in the upper and lower bounds is a strong argument that these bounds are nearly optimal. While I did not review all of the details of the proofs in the appendices, I didn't find any technical red flags. Theorems and lemmas are well-stated and easy to follow.\n\nWEAKNESSES:\nA minor note: The pseudocode for Algorithm 1 could be written in way that makes it more easily digestible for readers. Some helpful details (like the sample complexity for Initialization and Testing) could be included, while terms like $M$ could probably be given asymptotically for simplicity.\n\nQUESTIONS:\nThe upper and lower bounds have a substantial gap in $d$. Do the authors believe that this gap could be closed by a future work? And does a similar gap exist in the literature for learning homogeneous halfspaces? \n\nLikewise, do you expect it to be possible to prove a lower bound with a more sensitive dependence on the classification noise $\\eta$? It would be interesting to see if the problem gets much harder as the noise level approaches $\\frac12$.\n\nI don't exactly see why the Optimization procedure is subgradient descent on a leaky ReLU rather than just a scaled ReLU; if it were leaky, shouldn't there be another additive term in the gradient computation multiplied by $\\eta$, rather than just one term in the band multiplied by $(1 - \\eta)$?\n\nLIMITATIONS:\nAll limitations are well-documented in the assumptions underlying the theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper studies the problem of learning Gaussian half-spaces with random classification noise. Given labeled data $(x,y)\sim D$ where $x$ follows the standard Gaussian distribution,  a halfspace function $f$  such that $y=f(x)$ with probability $1-\eta$ and $y=-f(x)$ with probability $\eta$ and a precision parameter $\epsilon$. The goal is to construct a halfspace function $h$ such that for  $(x,y)\sim D$, $h(x)=y$ with probability at least $1-\eta-\epsilon$. 

This paper provides lower and upper bounds on the sample/time complexity in the general case where $f$ can be biased: $p=\min(Pr[f(x)=1],  Pr[f(x)=-1])\neq 1/2 $. An efficient algorithm is proposed with a sample complexity $\tilde{O}(d/\epsilon + d/\max^2(p,\epsilon) )$. A lower bound on the sample complexity of $\Omega(\sqrt{d}/ \max^2(p,\epsilon) )$  is proved for SQ efficient algorithms.

STRENGTHS:
Generalizing the problem of learning Gaussian halfspaces with RCN to not necessarily homogeneous halfspaces is a good contribution. Moreover, the bad term of the sample complexity $\tilde{O}( d/\max^2(p,\epsilon) )$ could not be improved in terms of the dependency in  $\max(p,\epsilon)$ thanks to the lower bound proved.

The algorithm is not a trivial generalization of the one for homogeneous halfspaces with RCN.

WEAKNESSES:
One weakness is that the lower bound has does not match the upper bound in terms of the dependency in $d$.

QUESTIONS:
Do you think your results could be generalized to (some) distributions $D$ with non Gaussian marginal $D_x$?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The authors provide an algorithm to learn d-dimensional halfspaces under random classification noise upto error \eps. The algorithm has time complexity O(dN/\eps^2), where N is the number of samples; and sample complexity ~ \tilde{O}(d/\eps + d/\eps^2). They also prove a lower bound, in the statistical query model, that requires \Omega(\sqrt{d}/\eps^2) samples, unless the model makes high accuracy queries.

STRENGTHS:
- The problem of learning half spaces under noise is an important and fundamental problem in machine learning.
- The results are tight in the statistical query model up to a factor \sqrt{d}.
- Clear and lucid technical overview..

WEAKNESSES:
No significant weaknesses.

QUESTIONS:
Some of the writing in the proofs can be simplified, constants like 0.000098 can be jarring.

LIMITATIONS:
Yes, adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the PAC learning complexity of half spaces (or linear threshold functions), when the labels are flipped randomly with some probability $\\eta$.\n\nFor realizable hypothesis classes, it is known that $O(d/\\epsilon)$ samples are enough to learn a halfspace with $\\epsilon$ 0-1 error. Under random classification error, the problem becomes much more challenging. This paper makes two contributions to this problem:\n- First, it presents an efficient algorithm with comparable scaling to $\\tilde O_{\\eta}(d/\\epsilon + d / (\\max(p, \\epsilon))^2 )$.\n- Second, it proves statistical query lower bounds that match the above upper bound under certain regimes of $\\eta$.\n\nThe upper bound involves three subroutines, including a warm start initialization that returns a vector close enough to the target. Second, it issues queries near the threshold, akin to learning a leaky ReLU loss. Lastly, it uses a simple hypothesis testing procedure, which draws a fresh sample and selects a hypothesis with the lowest test error.\n\nTher SQ lowe bound establishes the existence of a large set of distributions whose pariwise correlations are small. The construction builds on prior work of Diakonikolas, Kane and Steward (FOCS'17).\n\nSTRENGTHS:\nS1) The paper makes two contributions to learning halfspaces with random classification noise--this would be a nice contribution to this literature. The results seem solid and technically challenging.\n\nS2) The authors make an effort to explain their results at a high level, which is helpful.\n\nWEAKNESSES:\nW1) My main concern with this paper is the presentation of their technical results; in particular, there is a large number of notations, many of which are not clearly explained or are difficult to follow. This would limit the potential audience of their work within the broader NeurIPS community.\n\nW2) There are some inconsistencies between the statements, in particular, line 10 and line 46, which should be fixed or better explained.\n\nQUESTIONS:\n- Is the assumption of $p$-biasedness necessary for showing the sample complexity guarantee?\n\n- The comparison with a concurrent work of Diakonikolas et al. (2023) needs to be stated more clearly.\n\n- How critical are the results inherent to Gaussian inputs?\n\nLIMITATIONS:\nThe authors have not discussed the limitations of their work in the main text.\n\nThe paper is of a highly technical nature, so potential negative societal impacts of their work would be limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives new positive and negative results for the problem of learning general (nonhomogenous) Gaussian halfspaces under the random classification noise (RCN) model. The motivating question is whether there exists a polynomial time algorithm nearly achieving the (known) minimal sample complexity required for solving the problem.\n\nPositive result -- The authors give an efficient algorithm that solves this problem; this algorithm works with a sample complexity that is only off of the known optimal sample complexity by logarithmic factors. Specifically, the runtime of the algorithm is roughly $O(dN/\\varepsilon^2)$, where $N$ (sample complexity) is roughly $O(d/\\varepsilon^2 \\cdot \\log(1/\\delta))$ and $\\varepsilon$ is the target suboptimality. (note that I have omitted the dependences on the noise rate $\\eta$ and the bias $p$ of the target function -- both are probably optimal)\n\nTo obtain the positive result, there are three main algorithmic contributions. The first is an initialization step, which returns a vector whose correlation with the true weight vector is pretty high. The second is an optimization procedure that takes as input a guess for the threshold of the target halfspace and returns a halfspace of low error with that fixed threshold. The third is a procedure that tests several hypotheses and selects the one with the lowest error. \n\nNegative result -- The authors give a Statistical Query (SQ) model lower bound on the sample complexity of any algorithm solving the above problem -- either the statistical queries have to be very accurate, or there need to be an exponential number of queries made in order to recover a linear separator with nontrivial suboptimality. The authors also use a recent result almost equating SQ algorithms and low-degree tests to give a lower bound in the low-degree testing model. The exact statement is a bit technical, so I'll omit it here. It can easily be found in Theorem 1.5.\n\nEDIT 2023-09-01 -- As mentioned in my response, my review stands. Thank you for answering the questions!\n\nSTRENGTHS:\nThe paper almost (up to pesky logarithmic factors) resolves the motivating question of whether the optimal sample complexity can be achieved by an efficient algorithm for this problem. The algorithmic primitives are natural and easy to understand. The problem is pretty fundamental.\n\nFinally, the paper is clearly presented.\n\nWEAKNESSES:\nThere are minor typographic issues that can be cleaned up (e.g. spelling mistakes, long math strings, etc.) -- of course this is very minor.\n\nAlthough the optimality of the sample complexity is discussed, there appears to not be as much emphasis on the optimality of the runtime (in particular, the worst case dependence on $\\varepsilon$ in the runtime is $\\varepsilon^{-4}$ -- is this unavoidable?)\n\nQUESTIONS:\nWhat do you think are interesting future directions? e.g. do you think it is possible to obtain an even faster algorithm for this problem? If so, do you think extensions of the ideas you presented are likely to work, or do you think a totally different algorithmic approach is necessary?\n\nI probably missed this in the paper, but what is known about this problem when the input distribution is uniform over the vertices of the hypercube (i.e., $\\mathsf{Unif}(\\{\\pm 1\\})^d$? Do the techniques you give in this work readily transfer to this setting? It feels to me as though the Boolean setting is a more basic variant of the problem you study, but I would believe that the two are morally very similar.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning non-homogeneous halfspaces in the presence of Random Classification Noise, where the marginal distribution is standard Gaussian in $d$ dimensions. \n\nOn the upper bound side, this work provides an efficient algorithm that achieves learning an $\\epsilon$-optimal halfspace with sample complexity $\\tilde{O} (\\frac{d}{\\epsilon} +  \\frac{d}{\\max (p,\\epsilon)^2})$ where $p$ is the bias of the target halfspace. The algorithmic idea involves Initialization, Optimization, and final Testing subroutines. The algorithm runs for $O(1/\\epsilon^2)$ guesses of the threshold $t$ and selects the best output halfspace. \n\nOn the lower bound side, this work establishes a nearly matching lower bound that no efficient SQ algorithm can learn a Gaussian halfspace with $\\eta = 1/3$ with less than $\\Omega( \\frac{\\sqrt d}{\\max (p,\\epsilon)^2} )$ samples.\n\nSTRENGTHS:\n1. This work studies learning halfspace with RCN, which is a fundamental problem in machine learning theory. Although it has been known that halfspaces are efficiently PAC learnable with RCN, this work refines the sample complexity of the previous works in the Gaussian marginal distribution setting. Moreover, it provides formal evidence that the quadratic dependence on $ \\frac{1}{\\max (p,\\epsilon)}$ is the best one can hope for an efficient algorithm. \n\n2. This paper is technically strong.\n\n3. Along with the formal technical statements and proofs, this work exhibits some valuable informal explanations and implications of the results, which makes it easier for the readers to understand and appreciate the significance of the results in this paper. \n\n4. The main algorithm consists of several subroutines, each coming with a nice explanation and theoretical performance guarantees.\n\nWEAKNESSES:\n1. It could be nice to have a more comprehensive review of the related works. E.g., how is the sample complexity in this work related to the other papers on learning halfspaces with RCN with comparable settings?\n\n2. It could be good to include in the main algorithm the case where the threshold $t$ is significant.\n\nQUESTIONS:\n1. Could the main algorithm be modified a bit so that it could directly tell if the true threshold $t$ is significant enough and a constant hypothesis suffices?\n\n2. In Algorithm 1, the Optimization procedure is invoked for $M = O(1/\\epsilon^2)$ times, and $N_2 = \\tilde{O} (\\frac{d}{\\epsilon} )$ samples are drawn in each iteration. Could you please explain a bit more about why the total sample complexity (the dependence on $\\epsilon$) is less than $ \\tilde{O} (\\frac{1}{\\epsilon^3} )$?\n\nLIMITATIONS:\nThere are no obvious limitations to be addressed in my opinion. \n\nOne question that might be interesting to study is if it is possible to attain matching bounds in $d$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}