Title: A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise
Venue: NeurIPS 2024
Accepted: yes
Keywords: Massart Noise

margin halfspaces
Abstract: We study the problem of PAC learning $\gamma$-margin halfspaces in the presence of Massart noise. 
Without computational considerations, the sample complexity of this learning problem is known to be 
$\widetilde{\Theta}(1/(\gamma^2 \epsilon))$. 
Prior computationally efficient algorithms for the problem incur sample complexity 
$\tilde{O}(1/(\gamma^4 \epsilon^3))$ and achieve 0-1 error of $\eta+\epsilon$, 
where $\eta<1/2$ is the upper bound on the noise rate.
Recent work gave evidence of an information-computation tradeoff, 
suggesting that a quadratic dependence on $1/\epsilon$ is required 
for computationally efficient algorithms. 
Our main result is a computationally efficient learner with sample complexity 
$\widetilde{\Theta}(1/(\gamma^2 \epsilon^2))$, nearly matching this lower bound. 
In addition, our algorithm is simple and practical, 
relying on online SGD on a carefully selected sequence of convex losses.

A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise

Abstract

We study the problem of PAC learning γ -margin halfspaces in the presence of Massart noise. Without computational considerations, the sample complexity of this learning problem is known to be ˜ Θ(1 / ( γ 2 ϵ )) . Prior computationally efficient algorithms for the problem incur sample complexity ˜ O (1 / ( γ 4 ϵ 3 )) and achieve 0-1 error of η + ϵ , where η < 1 / 2 is the upper bound on the noise rate. Recent work gave evidence of an information-computation tradeoff, suggesting that a quadratic dependence on 1 /ϵ is required for computationally efficient algorithms. Our main result is a computationally efficient learner with sample complexity ˜ Θ(1 / ( γ 2 ϵ 2 )) , nearly matching this lower bound. In addition, our algorithm is simple and practical, relying on online SGD on a carefully selected sequence of convex losses.

1 Introduction

This work studies the algorithmic task of learning margin halfspaces in the presence of Massart noise (aka bounded label noise) [MN06] with a focus on fine-grained complexity analysis. A halfspace or Linear Threshold Function (LTF) is any Boolean-valued function h : R d → {± 1 } of the form h ( x ) = sign ( w · x -θ ) , where w ∈ R d is the weight vector and θ ∈ R is the threshold. The function sign : R → {± 1 } is defined as sign( t ) = 1 if t ≥ 0 and sign( t ) = -1 otherwise. The problem of learning halfspaces with a margin - i.e., under the assumption that no example lies too close to the separating hyperplane - is one of the earliest algorithmic problems studied in machine learning, going back to the Perceptron algorithm [Ros58].

In the realizable PAC model [Val84] (i.e., with clean labels), the sample complexity of learning γ -margin halfspaces on the unit ball in R d is Θ(1 / ( γ 2 ϵ )) , where ϵ > 0 is the desired 0-1 error; see, e.g., [SSBD14] 1 . Moreover, the Perceptron algorithm is a computationally efficient learner achieving this sample complexity. That is, without label noise, there is a sample-optimal and computationally efficient learner for margin halfspaces.

In this paper, we study the same problem in the Massart noise model that we now define.

Definition 1.1 (PAC Learning with Massart Noise) . Let D be a distribution over X × {± 1 } , and let C be a class of Boolean-valued functions over X . We say that D satisfies the η -Massart noise condition with respect to C , for some η < 1 / 2 , if there exists a concept f ∈ C and an unknown noise function η ( x ) : X ↦→ [0 , η ] such that for ( x , y ) ∼ D , the label y satisfies: with probability 1 -η ( x ) , y = f ( x ) ; and y = -f ( x ) otherwise. Given i.i.d. samples from D , the goal of the

̸

learner is to output a hypothesis h : X → {± 1 } such that with high probability the 0-1 error err D ( h ) def = Pr ( x ,y ) ∼ D [ h ( x ) = y ] is small.

The concept class of halfspaces with a margin is defined as follows.

Definition 1.2 ( γ -Margin Halfspaces) . Let D be a distribution over S d -1 ×{± 1 } , where S d -1 is the unit sphere in R d . Let w ∗ ∈ S d -1 and γ ∈ (0 , 1) . We say that the distribution D satisfies the γ -margin condition with respect the halfspace sign( w ∗ · x ) 2 , if (i) for ( x , y ) ∼ D , we have that y = sign( w ∗ · x ) , and (ii) Pr ( x ,y ) ∼ D [ | w ∗ · x | < γ ] = 0 . The parameter γ is called the margin of the halfspace sign( w ∗ · x ) .

Information-theoretically, the best possible 0-1 error attainable for learning a concept class with Massart noise is opt := E x ∼ D x [ η ( x )] . Since η ( x ) is uniformly bounded above by η , it follows that opt ≤ η ; also note that it may well be the case that opt ≪ η . Focusing on the class of γ -margin halfspaces, it follows from [MN06] that there exists a (computationally inefficient) estimator achieving error opt + ϵ with sample complexity ˜ O (1 / ((1 -2 η ) γ 2 ϵ )) ; and moreover that this sample upper bound is nearly best possible (within a logarithmic factor) for any estimator. (That is, the sample complexity of the Massart learning problem is essentially the same as in the realizable case, as long as η is bounded from 1 / 2 .)

Taking computational considerations into account, the feasibility landscape of the problem changes. Prior work [DK22, NT22, DKMR22] has provided strong evidence that achieving error better than η + ϵ is not possible in polynomial time. Consequently, algorithmic research has been focusing on achieving the qualitatively weaker error guarantee of η + ϵ . We note that efficiently obtaining any non-trivial guarantee had remained open since the 80s; see Appendix A.1 for a discussion. The first algorithmic progress for this problem is due to [DGT19], who gave a polynomial-time algorithm achieving error of η + ϵ with sample complexity poly(1 /γ, 1 /ϵ ) . Subsequent work [CKMY20] gave an efficient algorithm with improved sample complexity of ˜ O (1 / ( γ 4 ϵ 3 )) . Prior to the current work, this remained the best known sample upper bound for efficient algorithms.

In summary, known computationally efficient algorithms for learning margin halfspaces with Massart noise require significantly more samples-namely, ˜ Ω(1 / ( γ 4 ϵ 3 )) -than the information-theoretic minimum of ˜ Θ η (1 / ( γ 2 ϵ )) . It is thus natural to ask whether a polynomial-time algorithm with optimal (or near-optimal, i.e., within logarithmic factors) sample complexity exists. Recall that the answer to this question is affirmative in the realizable setting, where the Perceptron algorithm is optimal. Perhaps surprisingly, recent work [DDK + 23a] (see also [DDK + 23b]) gave evidence for the existence of inherent information-computation tradeoffs in the Massart noise model-in fact, even in the simpler model of Random Classification Noise (RCN) [AL88] 3 . Specifically, they showed that any efficient Statistical Query (SQ) algorithm or low-degree polynomial tasks requires Ω(1 /ϵ 2 ) samples-a near quadratic blow-up compared to the ˜ O (1 /ϵ ) information-theoretic upper bound. This discussion serves as the motivation for the following question:

What is the optimal computational sample complexity of the problem of learning γ -margin halfspaces with Massart noise?

By the term 'computational sample complexity' above, we mean the sample complexity of polynomial-time algorithms for the problem. Given the fundamental nature of this learning problem, we believe that a fine-grained sample complexity versus computational complexity analysis is interesting on its own merits. In this work, we develop a computationally efficient algorithm with sample complexity of ˜ O (1 / ( γ 2 ϵ 2 )) . Given the aforementioned information-computation tradeoffs, there is evidence that this upper bound is close to best possible. As a bonus, our algorithm is also simple and practical, relying on online SGD. (In fact, our algorithm runs in sample linear time, excluding a final testing step that slightly increases the runtime.)

1.1 Our Result and Techniques

Our main result is the following:

Theorem 1.3 (Main Result, Informal) . Let D be a distribution on S d -1 ×{± 1 } that satisfies the η -Massart noise condition with respect to an unknown γ -margin halfspace f ( x ) = sign( w ∗ · x ) . There is algorithm that draws n = ˜ O (1 / ( ϵ 2 γ 2 )) samples from D , runs in time ˜ O ( dn/ϵ ) , and with probability at least 9 / 10 returns a vector ˆ w such that err D ( ˆ w ) ≤ η + ϵ .

The sample upper bound of Theorem 1.3 nearly matches the computational sample complexity of the problem (for SQ algorithms and low-degree polynomial tests), which was shown to be Ω(1 / ( ϵ 2 γ ) + 1 / ( ϵγ 2 )) [MN06, DDK + 23a, DDK + 23b]. That is, Theorem 1.3 comes close to resolving the finegrained complexity of this basic task. Moreover, it matches known algorithmic guarantees for the easier case of Random Classification Noise [DDK + 23a, KIT + 23].

Independent Work Independent work [CKST24] obtained a learning algorithm for γ -margin halfspaces with essentially the same sample and computational complexity as ours.

Brief Overview of Techniques Here we provide a brief summary of our approach in tandem with a comparison to prior work. The algorithm of [DGT19] adaptively partitions the space into polyhedral regions and uses a different linear classifier in each region, each achieving error η + ϵ within the corresponding region. Their approach leverages the LeakyReLU loss (see (1)) as a convex proxy to the 0-1 loss. At a high-level, their approach reweights the samples in order to accurately classify a non-trivial fraction of points. [CKMY20] uses the LeakyReLU loss to efficiently identify a region where the value of the loss conditioned on this region is sub-optimal; they then use this procedure as a separation oracle along with online convex optimization (see also [DKTZ20b, DKK + 21]) to output a linear classifier with 0-1 error at most η + ϵ . Both of these approaches inherently require Ω(1 /ϵ 3 ) samples for the following reason: they both need to condition on a region where the probability mass of the distribution can be as small as Θ( ϵ ) . Thus, even estimating the error of the loss would require at least Ω(1 /ϵ 2 ) conditional samples. Beyond the dependence on 1 /ϵ , the sample complexity achieved in these prior works is also suboptimal in the margin parameter γ ; namely, Ω(1 /γ 4 ) . This dependence follows from the facts that both of these works require estimating the loss in each iteration within error of at most γϵ , and that their algorithmic approaches require Ω(1 /γ 2 ) iterations.

To circumvent these issues, novel ideas are required. At a high-level, we design a uniform approach to decrease the 'global' error, as opposed to the local error (as was done in prior work). Specifically, we construct a different sequence of convex loss functions, each of which attempts to accurately simulate the 0-1 objective. We note that a similar sequence of loss functions was used in the recent work [DKTZ24] in a related, but significantly different, adversarial online setting. Interestingly, a similar reweighting scheme was used in [CKMY20] for learning general Massart halfspaces. Beyond this similarity, these works have no implications for the sample complexity of our problem. (See Appendix A.2 for a detailed comparison.) Via this approach, we obtain an iterative algorithm which uses only O γ (1 /ϵ 2 ) samples in order to estimate the loss in each iterative step.

̸

In more detail, note that the 0-1 loss can be written in the form -E [ y w · x | w · x | ] . We convexify this objective by considering, in each step, the loss ℓ ( w , u ) = -E [ y w · x | u · x | ] , where u is independent of w ; this loss is convex with respect to w . Observe that ℓ ( w , w ) is proportional to the zero-one loss of w . Unfortunately, it is possible that no optimal vector w ∗ (under 0-1 loss) minimizes ℓ ( w ∗ , w ) . For this reason, we consider the objective ℓ η ( w , u ) = E [( 1 { y = sign( w · x ) } -η -ϵ ) | w · x | / | u · x | ] . This new objective satisfies the following: ℓ η ( w ∗ , u ) < -ϵγ for any vector u and any w ∗ that minimizes the 0-1 objective; and ℓ η ( w , w ) ≥ ϵ as long as w incurs 0-1 error at least η + ϵ . By the convexity of ℓ η ( w , u ) , this allows us to construct a separation oracle. Namely, we draw enough samples so that ̂ ℓ η ( w , w ) -̂ ℓ η ( w ∗ , w ) ≥ ϵ/ 2 , where ̂ ℓ is the emprical version of the loss. Due to the nature of these objectives, O γ (1 /ϵ 2 ) samples per iteration suffice for this purpose. This in turn implies that the cutting planes method efficiently finds a near-optimal weight vector after O (log(1 /ϵ ) /γ 2 ) iterations. Overall, this approach leads to an efficient algorithm with sample complexity ˜ O γ (1 /ϵ 2 ) . To get the desired sample complexity of ˜ O (1 / ( ϵ 2 γ 2 )) , more ideas are needed.

̸

In the previous paragraph, we hid an obstacle that makes the above approach fail. Specifically, it may be possible that, for many points x , the value of | u · x | is arbitrarily small. To fix this issue, we consider a clipped reweighting as follows: ℓ ′ η ( w , u ) = E [( 1 { y = sign( w · x ) }-η -ϵ ) | w · x | max( | u · x | ,γ ) ] . This clipping step is not a problem for us, because the target halfspace sign( w ∗ · x ) was assumed to have margin γ . This guarantees that the difference between the expected (over y ) pointwise losses at ( w , w ) and ( w ∗ , w ) is at least ϵ on the points x where | u · x | ≤ γ . Indeed, when this is the case, then | w ∗ · x | / | u · x | ≥ 1 . Overall, this suffices to guarantee that ℓ ′ η ( w , w ) -ℓ ′ η ( w ∗ , w ) ≥ ϵ .

1.2 Notation

For n ∈ Z + , let [ n ] def = { 1 , . . . , n } . We use small boldface characters for vectors. For x ∈ R d and i ∈ [ d ] , x i denotes the i -th coordinate of x , and ∥ x ∥ 2 def = ( ∑ d i =1 x 2 i ) 1 / 2 denotes the ℓ 2 -norm of x . We will use x · y for the inner product of x , y ∈ R d . For a subset S ⊆ R d , we define the proj S operator that maps a point x ∈ R d to the closest point in the set S . For a, b ∈ R , we denote W ( a, b ) def = 1 / max( a, b ) . We will use 1 A to denote the characteristic function of the set A , i.e., 1 { x ∈ A } = 1 if x ∈ A , and 1 { x ∈ A } = 0 if x / ∈ A . For A,B ∈ R , we write A ≳ B (resp. A ≲ B ) to denote that there exists a universal constant C > 0 , such that A ≥ CB (resp. A ≤ CB ).

̸

We use E x ∼ D [ x ] for the expectation of the random variable x with respect to the distribution D and Pr [ E ] for the probability of event E . For simplicity, we may omit the distribution when it is clear from the context. For ( x , y ) ∼ D , we use D x for the marginal distribution of x and D y ( x ) for the distribution of y conditioned on x . We use ̂ D N to denote the empirical distribution obtained by drawing N i.i.d. samples from D . We use err D ( w ) to denote the 0-1 error of the halfspace defined by the weight vector w with respect to the distribution D , i.e., err D ( w ) def = Pr ( x ,y ) ∼ D [sign( w · x ) = y ] . We will use err( w , x ) for the 0-1 error of sign( w · x ) conditioned on x , i.e., err( w , x ) := Pr y ∼ D y ( x ) [sign( w · x ) = y ] . Note that err D ( w ) = E x ∼ D x [err( w , x )] . If D satisfies the η -Massart noise condition with respect to the halfspace sign( w · x ) , then err( w , x ) = η ( x ) 1 { sign( w · x ) = sign( w ∗ · x ) } +(1 -η ( x )) 1 { sign( w · x ) = sign( w ∗ · x ) } .

̸

2 Our Algorithm and its Analysis: Proof of Theorem 1.3

In this section, we prove our main result. Algorithm 1 efficiently learns the class of margin halfspaces on the unit ball, in the presence of Massart noise, with sample complexity nearly matching the information-computation limit. Additionally, its runtime is linear in the sample size, excluding a final testing step to select the best hypothesis.

At a high-level, our algorithm leverages a carefully selected convex loss (or, more precisely, a sequence of convex losses) - serving as a proxy to the 0-1 error. A common loss function, introduced in this context by [DGT19] and leveraged in [DGT19, CKMY20], is the LeakyReLU function. This is the univariate function LeakyReLU λ ( t ) = (1 -λ ) 1 { t ≥ 0 } t + λ 1 { t < 0 } t , where λ ∈ (0 , 1) is the leakage parameter (that needs to be selected carefully). Roughly speaking, the convex function ℓ λ ( w , x , y ) = LeakyReLU λ ( -y ( w · x )) can be viewed as a reasonable proxy to the 0-1 loss of the halfspace sign( w · x ) on the point ( x , y ) . To see this, note that (see, e.g., Claim C.1)

̸

Observe that a point x that is classified correctly by the halfspace sign( w · x ) will satisfy

̸

which is non-positive for λ ≥ η ( x ) . Since the only guarantee we have is that η ( x ) ≤ η , this suggests that we need to select λ ≥ η . It turns out that λ := η is the optimal choice. We fix the choice of λ := η throughout. On the other hand, if (the halfspace defined by) w misclassifies the point x , this term becomes non-negative.

The factor | w · x | in Equation (1) reweights the 0-1 error so that points x for which | w · x | is sufficiently large (i.e., close to 1 ) have to be classified correctly by a minimizer of E ( x ,y ) ∼ D [ ℓ λ ( w , x , y )] . On the other hand, points closer to the separating hyperplane defined by w , or points where η ( x ) is close to λ = η , are not guaranteed to be classified correctly by the minimizer of this loss. We leverage this insight to construct a sequence of loss functions that reweight the points so that, to minimize the regret, we need to classify a large fraction of points; this leads to the desired error of η + ϵ with near-optimal sample complexity.

We now provide some intuition justifying our choice of surrogate loss functions. Observe that if we instead could minimize the function

̸

with respect to w , we would obtain a halfspace with minimum 0-1 error; unfortunately, this reweighted loss is just a shift of the 0-1 loss, hence non-convex. To fix this issue, instead of reweighting by

̸

1 / | w · x | , we will reweight by W ( v · x , γ ) def = 1 / max( | v · x | , γ ) , where γ is the margin parameter and v is an appropriately chosen vector that is independent of w . The new loss is defined as follows:

where for technical reasons we use γ/ 2 instead of γ in the maximum.

Since the parameter v is independent of w , the loss L λ, v ( w ) remains convex in w . At the same time, by carefully choosing v , we can accurately simulate the non-convex 0-1 loss. Note that our reweighting term is a maximum over two terms. The reason for this choice is that, for some points x , the quantity | v · x | can be arbitrarily small; taking the maximum avoids the loss becoming very large. In particular, the loss L λ, v ( w ) will be guaranteed to remain in a bounded length interval.

Our algorithm proceeds in a sequence of iterations. In the ( t +1) -st iteration, it sets v to be w t , where w t is the weight vector of step t . This choice attempts to simulate the 0-1 error at w t , as is suggested by Equation (2). Assume for simplicity that our current hypothesis is the halfspace defined by w and is such that E x ∼ D x [ 1 {| w · x | ≤ γ/ 2 } ] = 0 . Note this implies that W ( w · x , γ/ 2) = 1 / | w · x | . By combining Equations (2) and (3), we get that L λ, w ( w ) = err D ( w ) -λ ; note that as long as err D ( w ) ≥ λ + ϵ , we have that L λ, w ( w ) ≥ ϵ . On the other hand, the optimal halfspace w ∗ achieves a non-positive loss; from Equations (1) and (2), we have that

̸

where the inequality follows from the fact that η ( x ) ≤ η . Recalling that L λ, v ( w ) is convex, if we run an Online Convex Optimization (OCO) algorithm, after T steps we are guaranteed to find a vector w such that L λ, w ( w ) -L λ, w ( w ∗ ) ≤ O (1 / √ T ) . For T = O (1 /ϵ 2 ) , this gives that L λ, w ( w ) < ϵ/ 2 ; and therefore we would have err D ( w ) < λ + ϵ . We provide an approach using this idea and the cutting planes algorithm in Appendix B that achieves sample complexity ˜ O (1 / ( ϵ 2 γ 4 )) .

Our algorithm and its analysis work only with the gradient of L λ, v ( w ) . The key novelty is the analysis of the sample complexity. The gradient of ℓ λ ( w , x , y ) W ( v · x , γ ) with respect to w has the following explicit form:

Furthermore, we denote by G D ( w , v , η, γ ) = E ( x ,y ) ∼ D [ g η,γ ( w , v , x , y )] .

Before describing our algorithm and proving Theorem 2.1, we simplify our notation. We will omit the parameters η, γ from the function input (as they are fixed throughout). Therefore, we use G ̂ D t N ( w , v ) ≡ G ̂ D t N ( w , v , η, γ ) and g ( w , v , x , y ) ≡ g η,γ/ 2 ( w , v , x , y ) .

Our algorithm is described in pseudocode below.

Algorithm 1 employs online SGD applied to a sequence of convex loss functions. We show that, after a certain number of iterations, the algorithm will find a weight vector achieving 0-1 error at most η + ϵ . Since the desired vector may not be the last iterate, in the end, our algorithm returns the halfspace that achieves the smallest empirical 0-1 error.

We establish the following result, which implies Theorem 1.3.

Theorem 2.1 (Main Result) . Let D be a distribution on S d -1 × {± 1 } satisfying the η -Massart noise condition with respect to the γ -margin halfspace f ( x ) = sign( w ∗ · x ) . Given N = Θ(log(1 / ( γδ )) /ϵ (1 -2 η )) and T = Θ(log(1 /δ ) / ( ϵ 2 γ 2 )) , Algorithm 1 returns a vector ˆ w such that err D ( ˆ w ) ≤ η + ϵ with probability at least 1 -δ . The algorithm draws n = O ( N + T ) samples from D and runs in O ( dNT ) time.

The rest of this section is devoted to the proof of Theorem 2.1.

Our algorithm sets v = w t in each round, therefore for the rest of the section we proceed by setting v = w as arguments of g and G .

Input: Sample access to a distribution D supported in S d -1 ×{± 1 } corrupted with η -Massart noise with respect to a halfspace sign( w ∗ · x ) that satisfies the γ -margin condition; parameters ϵ, δ ∈ (0 , 1) , and N,T ∈ Z + .

Output: Weight vector ˆ w such that err D ( ˆ w ) ≤ η + ϵ with probability at least 1 -δ .

Let c > 0 be a sufficiently small universal constant.

t ← 0 , w 0 ← e 1 = (1 , 0 , . . . , 0) , and T = (1 /c ) log(1 /δ ) / ( ϵ 2 γ 2 ) .

While t ≤ T do

(a) Draw ( x ( t ) , y ( t ) ) sample from D .

(b) Set λ t ← cγ 2 ϵ .

(c) Update w t as follows:

▷ Update and project in the unit ball

(d) t ← t +1 .

Draw N samples from D and construct the empirical distribution ̂ D N .

Return ̂ w = argmin t ∈ [ T +1] err ̂ D N ( w t ) .

We decompose the stochastic gradient g ( w , w , x , y ) into two parts: g ( w , w , x , y ) = g 1 ( w , x ) + g 2 ( w , x , y ) , where

and

We also use G 1 ̂ D N ( w ) and G 2 ̂ D N ( w ) for the same decomposition after taking the empirical expectation, i.e., G 1 ̂ D N ( w ) = E x ∼ ( ̂ D x ) N [ g 1 ( w , x )] and G 2 ̂ D N ( w ) = E ( x ,y ) ∼ ̂ D N [ g 2 ( w , x , y )] .

This serves to decompose the gradient into two parts: one containing the population expectation over the random variable y , and the other containing the error between the empirical estimation of y and the population version of y . The vector G 1 ̂ D N ( w ) contains the direction that will decrease the distance between w and w ∗ , while G 2 ̂ D N ( w ) contains the estimation error. To see this, observe that if we take the population expectation of g 2 ( w , x , y ) , we will have:

where we used that E y ∼ D y ( x ) [ y ] = (1 -2 η ( x ))sign( w ∗ · x ) .

We start by bounding the contribution of G 1 ̂ D N ( w ) in the direction w -w ∗ . We show that if instead of the corrupted label y at the point x , we had access to E y ∼ D y ( x ) [ y ] = (1 -2 η ( x ))sign( w ∗ · x ) , then the gradient has a large component in the direction of w -w ∗ . This effectively implies that G 1 ̂ D N ( w ) can be used as a separation oracle, separating all the halfspaces with 0-1 error more than η + ϵ from the ones with smaller error.

Lemma 2.2 (Structural Lemma) . Let N ∈ Z + and let D be a distribution on S d -1 ×{± 1 } satisfying the η -Massart condition with respect to the optimal classifier f ( x ) = sign( w ∗ · x ) . Let w ∈ R d be such that ∥ w ∥ 2 ≤ 1 and let { x ( i ) } N i =1 be a multiset of N i.i.d. samples from D x . Then, it holds G 1 ̂ D N ( w ) · ( w -w ∗ ) ≥ 2(err ̂ D N ( w ) -η ) , where ̂ D N is the corresponding empirical distribution.

Proof. We partition R d into two subsets R 1 , R 2 as follows: R 1 contains the points that lie sufficiently far away from the separating hyperplane w · x = 0 , i.e., R 1 def = { x ∈ R d : | w · x | ≥ γ/ 2 } . R 2 contains the remaining points, i.e., R 2 def = { x ∈ R d : | w · x | < γ/ 2 } .

We first show that for any x ∈ R 1 , the vector g 1 ( w , x ) has a large component parallel to the direction w -w ∗ . The proof of the claim below can be found in Appendix C.

It remains to show that the same holds for all the points in R 2 . The proof of the claim below can be found in Appendix C.

Applying Claim 2.3 and Claim 2.4 for each sample in the set { x ( i ) } N i =1 , we get that

This completes the proof of Lemma 2.2.

By Lemma 2.2, the gradient points towards the direction w t -w ∗ , in the t -th iteration. This means that, in fact, the gradient is a subgradient of the potential loss Φ( w ) = ∥ w -w ∗ ∥ 2 2 . This allows us to show convergence, even though it is generally not possible in a sequence of loss functions in the stochastic setting. We are now ready to prove our main result.

Proof of Theorem 2.1. Let T be the maximum number of iterations of Algorithm 1. Denote by Z t := { ( x ( t ) , y ( t ) ) } the i.i.d. sample drawn from D in the t -th iteration, t ∈ [ T ] . Furthermore, let F 1 , . . . , F T be the filtration with respect to the σ -algebra generated by Z 1 , . . . , Z T . We denote by H t the event that err D ( w t ) ≥ η + ϵ .

Recall that Algorithm 1 uses the following update rule (see Step (3c)):

with λ t = cγ 2 ϵ , for some sufficiently small absolute constant c > 0 .

We begin by bounding from above the distance between w t +1 and w ∗ from the previous distance between w t and w ∗ . We have that

where in the first inequality we used the projection inequality, i.e., ∥ proj B ( v ) -proj B ( u ) ∥ 2 ≤ ∥ v -u ∥ 2 for any set B . We will decouple the mean of the random variable g ( w t , w t , x , y ) and make it zero-mean.

To simplify the notation, we denote by ξ t := ( g ( w t , w t , x ( t ) , y ( t ) ) -G 1 D ( w t ) ) · ( w t -w ∗ ) and note that ξ t is a zero-mean random variable over the sample ( x ( t ) , y ( t ) ) . Adding and subtracting G 1 D ( w t ) onto Inequality (4) a we get that

We now outline the main steps of our analysis. Instead of accurately estimating the gradients in each round, we denote by ̂ V t the estimation error from which we bound above their sum. We first add and

subtract the population gradient to obtain the I term, which is the decreasing direction. In this way, we decouple the expected decrease and the error of the approximation (see Claim 2.5). After that, we bound the contribution of the estimation error in Lemma 2.8. Observe that ̂ V t is a random variable that corresponds to the estimation error of the gradient. We will argue that with high probability the contribution of ∑ T t =1 ̂ V t is bounded; therefore, our algorithm will converge to an accurate solution.

Lemma 2.2 shows that G 1 ̂ D t N ( w t ) (and therefore the same holds for G 1 D ( w t ) ) contains substantial contribution towards to the direction w t -w ∗ , depending of the current error. We show that our choice of step size guarantees a decreasing direction. To this end, we prove the following:

Claim 2.5. Assume that the event H t happens, i.e., err D ( w t ) ≥ η + ϵ . If λ t ≤ γ 2 ϵ/ 8 , then I ≤ -λ t (err D ( w t ) -η ) .

Proof of Claim 2.5. Recall that I = -2 λ t G 1 D ( w t ) · ( w t -w ∗ ) + λ 2 t ∥ g ( w t , w t , x ( t ) , y ( t ) ) ∥ 2 2 . By Lemma 2.2, we get that G 1 ̂ D N ( w t ) · ( w t -w ∗ ) ≥ 2(err ̂ D N ( w t ) -η ) ; hence, by taking expectations over the samples, we also have G 1 D ( w t ) · ( w t -w ∗ ) ≥ 2(err D ( w t ) -η ) . Furthermore, we have that ∥ g ( w t , w t , x ( t ) , y ( t ) ) ∥ 2 2 ≤ 8 /γ 2 . Hence, I ≤ -2 λ t (err D ( w t ) -η ) + 8( λ 2 t /γ 2 ) . The claim follows by noting that if λ t ≤ γ 2 ϵ/ 8 , then -λ t (err D ( w t ) -η ) + 8( λ 2 t /γ 2 ) ≤ 0 . Therefore, we obtain

This completes the proof of Claim 2.5.

Therefore, our choice of parameters guarantees that λ t ≤ γ 2 ϵ/ 8 . Using Claim 2.5 onto Inequality (5), we have that

Using Claim 2.5 and Inequality (6), we have that

To complete the proof of Theorem 2.1, we need to bound the estimation error that corresponds to the random variable ̂ V t . We show that ̂ V t does not increase the error by a lot. Recall that ̂ V t = -2 λ t ξ t .

Before proceeding, we provide some basic background on subgaussian random variables.

Definition 2.6 (Subgaussian Random Variable) . For σ > 0 , a zero-mean random variable X ∈ R is called σ -subgaussian, if for any λ ∈ R it holds log( E [exp( λX )]) ≤ λ 2 σ 2 .

Note that any zero-mean bounded random variable is subgaussian. Specifically, we have the following:

Fact 2.7 (Hoeffding's lemma, see, e.g., [Ver18]) . Let X ∈ R be a zero-mean random variable such that | X | ≤ σ for some σ > 0 . Then X is Cσ -subgaussian, where C > 0 is a universal constant.

Equipped with the above context, we show the following:

Lemma 2.8. With probability at least 1 -δ over the random samples, it holds that ∑ T t =0 ̂ V t ≤ Cγ 2 ϵ 2 T +log(1 /δ ) , where C > 0 is an absolute constant.

Proof. We first show that ξ t is a subgaussian random variable.

Claim 2.9. The random vector ξ t is (16 /γ ) -subgaussian.

Proof of Claim 2.9. Note that ξ t = ( g ( w t , w t , x ( t ) , y ( t ) ) -E ( x ,y ) ∼ D [ g ( w t , w t , x , y )]) · ( w t -w ∗ ) and that by construction ∥ g ( w t , w t , x , y ) ∥ 2 ≤ 4 /γ . Therefore, it holds that | g ( w t , w t , x ( t ) , y ( t ) ) · ( w t -w ∗ ) | ≤ 8 /γ , where we used that ∥ w t -w ∗ ∥ 2 ≤ 2 as both of these vectors lie in the unit ball. Hence, by Fact 2.7, we have that ξ t is (16 /γ ) -subgaussian.

Using Claim 2.9 and Definition 2.6 with parameter λ = -2 λ t and X = ξ t , we have that

where C > 0 is a universal constant. To bound the contribution of ∑ T t =0 ̂ V t , we use Markov's inequality with respect to the filtration F 1 , . . . , F T . We have that for any Z ∈ R , it holds that

where in the second inequality we use the independence of ̂ V t with { ̂ V k } t -1 k =1 with respect to the filtration F t . Recalling that λ t = cγ 2 ϵ , where c > 0 is a sufficiently small universal constant, we have that

Setting Z = Cc 2 γ 2 ϵ 2 T +log(1 /δ ) and taking c to be a sufficiently small absolute constant (as is done in our algorithm), we get that Pr Z 1 ,..., Z T ∼ D [ ∑ T t =0 ̂ V t ≥ Z ] ≤ δ . This completes the proof of Lemma 2.8.

Assume that until the round T the event H T holds, i.e., for all i ∈ [ T ] we have that err D ( w i ) ≥ η + ϵ . Using Lemma 2.8 onto Inequality (7), with probability at least 1 -δ , we have that:

Running the algorithm for T = Θ(log(1 /δ ) / ( ϵ 2 γ 2 )) iterations guarantees that with probability at least 1 -δ , we will have that ∥ w T +1 -w ∗ ∥ 2 2 ≤ 0 , which means w T +1 = w ∗ . In that case, i.e., in the case where all the events H i for i ∈ [ T ] hold, w T +1 achieves the same error as the optimal halfspace, thus it has 0-1 error of at most η + ϵ . Therefore, at least one vector w t ′ with t ′ ∈ [ T + 1] achieves 0-1 error of at most η + ϵ . The algorithm, in Step (5), returns a vector ̂ w that has 0-1 error at most err D ( ̂ w ) ≤ min t ∈ [ T +1] err D ( w t ) + ϵ ≤ η + 2 ϵ . The algorithm requires N = O (log( T/δ ) / ( ϵ (1 -2 η ))) samples for Step (5), due to [MN06]. The algorithm draws a sample in each round and runs for at most T rounds. Therefore, Algorithm 1 draws n = N + T = ˜ O (log(1 /δ ) / ( ϵ 2 γ 2 )) samples. The algorithm needs to test each of the T hypotheses with N samples to find the closest one. Therefore, the total runtime is O ( dTN ) (as in the other subroutines the algorithm uses the samples only to estimate the gradients g , which requires O (1) additions of d -dimenional vectors). This completes the proof of Theorem 2.1.

3 Conclusions and Open Problems

In this paper, we give the first sample near-optimal and computationally efficient algorithm for learning margin halfspaces in the presence of Massart noise. Specifically, the sample complexity of our algorithm nearly matches the computational sample complexity of the problem and its computational complexity is polynomial in the sample size. An interesting direction for future work is to develop a sample near-optimal and computationally efficient learner for general halfspaces (i.e., without the margin assumption). While our approach can likely be leveraged to obtain an efficient algorithm with sample complexity poly( d ) /ϵ 2 , the sample dependence on the dimension d would be suboptimal. Obtaining the right dependence on the dimension seems to require novel ideas, as prior works rely on fairly sophisticated methods [DV04, DKT21, DTK23] to effectively reduce to the large margin case.

Acknowledgments

ID was supported in part by NSF Medium Award CCF-2107079 and an H.I. Romnes Faculty Fellowship. NZ was supported in part by NSF Medium Award CCF-2107079.

References

[ABHU15] P. Awasthi, M. F. Balcan, N. Haghtalab, and R. Urner. Efficient learning of linear separators under bounded noise. In Proceedings of The 28th Conference on Learning Theory, COLT 2015 , pages 167-190, 2015.

[ABHZ16] P. Awasthi, M. F. Balcan, N. Haghtalab, and H. Zhang. Learning and 1-bit compressed sensing under asymmetric noise. In Proceedings of the 29th Conference on Learning Theory, COLT 2016 , pages 152-192, 2016.

[AL88] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning , 2(4):343370, 1988.

[Blu03] A. Blum. Machine learning: My favorite results, directions, and open problems. In 44th Symposium on Foundations of Computer Science (FOCS 2003) , pages 11-14, 2003.

[CKMY20] S. Chen, F. Koehler, A. Moitra, and M. Yau. Classification under misspecification: Halfspaces, generalized linear models, and connections to evolvability. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[CKMY21] S. Chen, F. Koehler, A. Moitra, and M. Yau. Online and distribution-free robustness: Regression and contextual bandits with huber contamination. In FOCS , 2021.

[CKST24] G. Chandrasekaran, V. Kontonis, K. Stavropoulos, and K. Tian. Learning noisy halfspaces with a margin: Massart is no harder than random. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

[DDK + 23a] I. Diakonikolas, J. Diakonikolas, D. M. Kane, P. Wang, and N. Zarifis. Informationcomputation tradeoffs for learning margin halfspaces with random classification noise. In COLT , 2023.

[DDK + 23b] I. Diakonikolas, J. Diakonikolas, D. M. Kane, P. Wang, and N. Zarifis. Near-optimal bounds for learning gaussian halfspaces with random classification noise. In NeurIPS , 2023.

[DGT19] I. Diakonikolas, T. Gouleakis, and C. Tzamos. Distribution-independent PAC learning of halfspaces with Massart noise. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 4751-4762. Curran Associates, Inc., 2019.

[DIK + 21] I. Diakonikolas, R. Impagliazzo, D. M. Kane, R. Lei, J. Sorrell, and C. Tzamos. Boosting in the presence of Massart noise. In Proceedings of The 34th Conference on Learning Theory, COLT , 2021.

[DK22] I. Diakonikolas and D. Kane. Near-optimal Statistical Query hardness of learning halfspaces with Massart noise. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4258-4282. PMLR, 2022. Preliminary Version 2021: Arxiv eprint: 2012.09720.

[DKK + 20] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. A polynomial time algorithm for learning halfspaces with Tsybakov noise. arXiv , 2020.

[DKK + 21] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Efficiently learning halfspaces with Tsybakov noise. STOC , 2021.

[DKK + 22] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Learning general halfspaces with general Massart noise under the gaussian distribution. In STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, 2022 , pages 874-885. ACM, 2022.

[DKMR22] I. Diakonikolas, D. Kane, P. Manurangsi, and L. Ren. Cryptographic hardness of learning halfspaces with Massart noise. In Advances in Neural Information Processing Systems , 2022.

[DKRS22] I. Diakonikolas, D. Kane, L. Ren, and Y. Sun. SQ lower bounds for learning single neurons with Massart noise. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 , 2022.

[DKT21] I. Diakonikolas, D. Kane, and C. Tzamos. Forster decomposition and learning halfspaces with noise. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 7732-7744, 2021.

[DKTZ20a] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with Massart noise under structured distributions. In Conference on Learning Theory, COLT , 2020.

[DKTZ20b] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with Tsybakov noise. arXiv , 2020.

[DKTZ24] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Online Linear Classification with Massart Noise, 2024. Arxiv eprint: 2405.12958.

[DPT21] I. Diakonikolas, J. Park, and C. Tzamos. Relu regression with Massart noise. In Advances in Neural Information Processing Systems , 2021.

[DTK23] I. Diakonikolas, C. Tzamos, and D. M. Kane. A strongly polynomial algorithm for approximate forster transforms and its application to halfspace learning. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, STOC 2023 , pages 1741-1754. ACM, 2023.

[DV04] J. Dunagan and S. Vempala. Optimal outlier removal in high-dimensional spaces. J. Computer & System Sciences , 68(2):335-373, 2004.

[JL84] W. Johnson and J. Lindenstrauss. Extensions of Lipshitz mapping into Hilbert space. Contemporary Mathematics , 26:189-206, 1984.

[KIT + 23] V. Kontonis, F. Iliopoulos, K. Trinh, C. Baykal, G. Menghani, and E. Vee. Slam: Student-label mixing for distillation with unlabeled examples. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023 , 2023.

[MN06] P. Massart and E. Nedelec. Risk bounds for statistical learning. Ann. Statist. , 34(5):23262366, 10 2006.

[NT22] R. Nasser and S. Tiegel. Optimal SQ lower bounds for learning halfspaces with Massart noise. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1047-1074. PMLR, 2022.

[Ros58] F. Rosenblatt. The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review , 65:386-407, 1958.

[Slo88] R. H. Sloan. Types of noise in data for concept learning. In Proceedings of the First Annual Workshop on Computational Learning Theory , COLT '88, pages 91-96, San Francisco, CA, USA, 1988. Morgan Kaufmann Publishers Inc.

[Slo92] R. H. Sloan. Corrigendum to types of noise in data for concept learning. In Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992 , page 450, 1992.

[SSBD14] S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

[SZ07] S. Smale and D. Zhou. Learning theory estimates via integral operators and their approximations. Constructive approximation , 26(2):153-172, 2007.

[Val84] L. G. Valiant. A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing (STOC) , pages 436-445. ACM Press, 1984.

[Ver18] R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

[YZ17] S. Yan and C. Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 , pages 1056-1066, 2017.

[ZLC17] Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, COLT 2017 , pages 1980-2022, 2017.

Supplementary Material

Organization The structure of this appendix is as follows: In Appendix A, we provide additional summary and comparison with related and prior work. In Appendix B, we provide a polynomial time cutting-planes based algorithm with sample complexity ˜ O (1 / ( ϵ 2 γ 4 )) . Finally, in Appendix C, we provide the proofs omitted from Section 2.

A Related and Prior Work

A.1 Additional Related Work

The computational problem of learning halfspaces with Massart noise has been extensively studied, both in the distribution-specific and the distribution-free settings.

In the distribution-specific setting, the first efficient algorithm for homogeneous Massart halfspaces was given in [ABHU15]. Subsequent work generalized this result in various directions [ABHZ16, ZLC17, YZ17, DKTZ20a, DKTZ20b, DKK + 20, DKK + 21, DKK + 22].

The first algorithmic progress in the distribution-free setting was made by [DGT19], answering a longstanding open problem [Slo88, Slo92, Blu03]. Subsequent work gave an algorithm with improved sample complexity [CKMY20] and provided strong evidence that an error of η + ϵ is the best to hope for in polynomial time [DK22, NT22, DKMR22] (in both the Statistical Query model and under plausible cryptographic assumptions). In a related direction, [DIK + 21] gave the first efficient boosting algorithm in the presence of Massart noise, which can boost a weak learner to one with error η + ϵ . Finally, we note that natural generalizations of the Massart model to learning real-valued functions (in an essentially distribution-free setting) have also been studied [CKMY21, DPT21, DKRS22].

Very recent work [DDK + 23a] gave SQ (and low-degree polynomial testing) lower bounds for learning γ -margin halfspaces with RCN [AL88], which is a special case of Massart noise. Specifically, [DDK + 23a] showed that any efficient SQ algorithm for the problem requires sample complexity Ω(1 / ( γ 1 / 2 ϵ 2 )) . Subsequently, [DDK + 23b] showed a related SQ lower bound under the Gaussian distribution, which can be adapted to obtain a lower bound of Ω(1 / ( γϵ 2 )) for the margin setting.

A.2 Comparison with [DKTZ24]

The work [DKTZ24] uses a similar sequence of loss functions for the problem of 'online learning' Massart margin halfspaces. Intuitively, their goal is to minimize regret in an adversarial online setting. In their online setting, the adversary in each round commits to covariates x 1 , x 2 ∈ R d and distribution D t over R + × R + . Then the algorithm observes the covariates, chooses an action a ∈ { 1 , 2 } , and observes a reward r a ∈ R + . It is only guaranteed that there exists a unit vector w ∗ so that E ( r 1 ,r 2 ) ∼ D t [sign( w ∗ · x 1 -w ∗ · x 2 )( r a -r b )] ≥ ∆ for some ∆ > 0 .

Despite this superficial similarity, the work of [DKTZ24] has no new implications on the sample complexity of PAC learning Massart halfspaces with a margin. Specifically, they achieve a regret bound of O ( T 3 / 4 /γ ) . If one translates this bound to a sample complexity upper bound for PAC learning, one would obtain a bound of Ω(1 / ( ϵ 4 γ 8 )) -which is quantitatively worse than prior work of [DGT19, CKMY20].

At a technical level, our work leverages this sequence of loss functions as subgradients of the potential function Φ( w ) = ∥ w -w ∗ ∥ 2 2 . Via a novel analysis, we show that these subgradients Ω( ϵ ) -correlate with the direction of w -w ∗ . This in turn means that we can expect a decrease of order Ω( λϵ ) in each iteration, where λ is the corresponding step-size, as long as we get 0-1 error more than η + ϵ . This structural understanding suffices for obtaining an algorithm, based on a separation oracle, that achieves a sample complexity of ˜ O (1 / ( γ 4 ϵ 2 )) . In order to obtain an algorithm with near-optimal sample complexity (and runtime), we required additional new ideas as elaborated in the body of the paper.

B Learning Margin Massart Halfspaces via Cutting Planes

In this section, we show how to use the cutting-planes method along with Lemma 2.2 to efficiently learning margin Massart Halfspaces using ˜ O (1 / ( γ 4 ϵ 2 )) samples.

Specifically, we establish the following result:

Theorem B.1 (Learning Margin Massart Halfspaces with Cutting Planes) . Let D be a distribution on S d -1 ×{± 1 } which satisfies the η -Massart noise condition with respect to the γ -margin halfspace f ( x ) = sign( w ∗ · x ) . Given N = Θ(log(1 / ( γδ ) / ( γ 4 ϵ 2 )) i.i.d. samples from D , there is a poly( d, N ) time algorithm that returns a vector ˆ w such that err D ( ˆ w ) ≤ η + ϵ with probability at least 1 -δ .

Remark B.2. We can always assume that d = ˜ O (1 /γ 2 ) . This holds since we can efficiently preprocess the data, using the Johnson-Lindenstrauss transform [JL84]. Similar dimension-reduction steps have been use in prior work, e.g., [CKMY20, DDK + 23a].

Given the above remark, it suffices to establish the following:

Theorem B.3. Let D be a distribution on S d -1 ×{± 1 } which satisfies the η -Massart noise condition with respect to the γ -margin halfspace f ( x ) = sign( w ∗ · x ) . Given N = Θ( d log(1 / ( γδ ) / ( γ 2 ϵ 2 )) i.i.d. samples from D , there is a poly( d, N ) time algorithm that returns a vector ˆ w such that err D ( ˆ w ) ≤ η + ϵ with probability at least 1 -δ .

The idea of using the cutting plane method is slightly adapted from [CKMY20]. Given access to a separation oracle for a convex set K , we can find a point inside the set K by querying the separation oracle O ( d log d ) times. The difference with [CKMY20] is that we are using a more sophisticated (and sample efficient) separation oracle. This allows us to use O (1 /ϵ 2 ) samples, instead of O (1 /ϵ 3 ) samples, and leads to the optimal sample complexity as a function of ϵ (but not γ ).

Fact B.4. Suppose that K is an (unknown) convex body in R d which contains a Euclidean ball of radius r > 0 and contained in a Euclidean ball centered at the origin of radius R > 0 . There exists an algorithm which, given access to a separation oracle for K , finds a point x ∗ ∈ K , runs in time poly(log( R/r ) , d ) , and makes O ( d log( Rd/r )) calls to the separation oracle.

We first show that if we get enough samples, we can efficiently approximate the gradients G ( w , w ) . Formally, we have:

Proposition B.5 (Separation Oracle) . Let ϵ, δ ∈ (0 , 1) and let D be a distribution on S d -1 ×{± 1 } satisfying the η -Massart noise condition with respect to the halfspace f ( x ) = sign( w ∗ · x ) . Fix w ∈ R d with ∥ w ∥ 2 ≤ 1 . Let N ≳ log(1 / ( γδ )) / ( ϵ 2 γ 2 )) and ̂ D N be the corresponding empirical distribution. Then, with probability at least 1 -δ , it holds that

Proof. By construction, G ̂ D N ( w , w ) = G 1 ̂ D N ( w ) + G 2 ̂ D N ( w ) and by Lemma 2.2 we have that G 1 ̂ D N ( w ) · ( w -w ∗ ) ≥ 2(err ̂ D N ( w ) -η ) . By definition, we have E ( x (1) ,y (1) ) ,..., ( x ( N ) ,y ( N ) ) ∼ D [ G 2 ̂ D N ( w )] = 0 , where the expectation is taken with respect to the sample set. Note that the norm of g 1 ( w , x ) , g 2 ( w , x , y ) , i.e., ∥ g 1 ( w , x ) ∥ 2 , ∥ g 2 ( w , x , y ) ∥ 2 , is bounded pointwise from above by 4 /γ for all w ∈ R d . This can be seen as ∥ x ∥ 2 ≤ 1 , W ( · , γ/ 2) ≤ 2 /γ , and (1 -2 η ) , (1 -2 η ( x )) ≤ 1 .

We use the following concentration inequality to show that our sample size is enough to guarantee that the estimated gradient is close to its population version.

Fact B.6 ([SZ07], Lemma 1) . Let Z 1 , . . . , Z n ∈ R d be random vectors such that for each i ∈ [ n ] it holds ∥ Z i ∥ 2 ≤ M < ∞ almost surely and let σ 2 = ∑ n i =1 E [ ∥ Z i ∥ 2 2 ] . Then, we have that for any ϵ > 0 ,

Using Fact B.6, along with the inequality log(1 + z ) ≥ z/ 2 , for z ∈ (0 , 1) , we get that if N ≥ Θ( log(1 /δ ) ( ϵγ ) 2 ) , with probability at least 1 -δ , we have

and

To complete the proof, recall that by Lemma 2.2 it holds G 1 ̂ D N ( w ) · ( w -w ∗ ) ≥ 2(err ̂ D N ( w ) -η ) -ϵ . Therefore, by taking the expectation over D x , we get that

The proof is completed by recalling that ∥ G 1 ̂ D N ( w ) -E ( x ,y ) ∼ D [ g 1 ( w , x )] ∥ 2 ≤ ϵ from Inequality (8) and that E ( x ,y ) ∼ D [ g 2 ( w , x , y )] = 0 .

Equipped with Proposition B.5, we are ready to prove a weaker version of Theorem 2.1 using separation oracles and the cutting plane algorithm. Formally, we show that

Proof of Theorem B.3. Our convex set K is a Euclidean ball of radius γ/ 2 centered at w ∗ . To see that, note that for any v such that ∥ w ∗ -v ∥ 2 ≤ γ/ 2 , we have that | ( w ∗ -v ) · x | ≤ γ/ 2 for any x with ∥ x ∥ 2 = 1 . This implies that γ/ 2 + w ∗ · x ≥ v · x ≥ w ∗ · x -γ/ 2 . Moreover, by definition we have that w ∗ · x ≥ γ . Hence, if w ∗ · x ≥ 0 , we have that v · x ≥ γ/ 2 ; and if w ∗ · x ≤ 0 , we have that v · x ≤ -γ/ 2 . Therefore, this ball contains all the vectors w with margin γ/ 2 and separates the points in the same way as w ∗ .

Therefore, as long as we are not in the set K or the 0-1 error is more than η + ϵ , we can use Proposition B.5 to construct a new separation oracle. By Fact B.4, the maximum number of calls to the separation oracle is T = O ( d log( d/γ )) . By Proposition B.5, in each round we need n = O (log( T/δ )) / ( ϵ 2 γ 2 ) samples from D to construct a separation oracle. Therefore, the maximum number of samples is O ( nT ) = O ( d log( T/δ )) / ( ϵ 2 γ 2 ) . This completes the proof.

C Omitted Proofs from Section 2

C.1 Proof of Claim C.1

Claim C.1 (Claim 2.1 [DGT19]) . For any w , x , we have that

Proof. Recall that

Therefore, we have that

where we used that y ∈ {± 1 }

.

C.2 Proof of Claim 2.3

We restate and prove the following claim:

Claim 2.3. For any x ( i ) ∈ R 1 , we have that g 1 ( w , x ( i ) ) · ( w -w ∗ ) ≥ 2(err( w , x ( i ) ) -η ) .

Proof of Claim 2.3. For any x ( i ) ∈ R 1 , we have that

where we used that for any x ( i ) ∈ R 1 , W ( w · x ( i ) ) = 1 / | w · x ( i ) | , and hence W ( w · x ( i ) , γ/ 2) w · x ( i ) = sign( w · x ( i ) ) ; and that err( w , x ( i ) ) = η ( x ( i ) ) if sign( w · x ( i ) ) = sign( w ∗ · x ( i ) ) and 1 -η ( x ( i ) ) otherwise.

We now bound the contribution of w ∗ . Since η ( x ) ≤ η , we have

Therefore, we have that

which gives that -g ( w , x ) · w ≥ 0

1 ( i ) ∗ . This completes the proof of Claim 2.3.

C.3 Proof of Claim 2.4

We restate and prove the following:

Proof of Claim 2.4. We have that

where we used that max( γ/ 2 , | w · x ( i ) | ) = γ/ 2 for any x ( i ) ∈ R 2 . Since sign( w ∗ · x ) has γ -margin, we have that | w ∗ · x ( i ) | ≥ γ . Since x ( i ) ∈ R 2 , it holds | w · x ( i ) | < γ/ 2 . Therefore, -sign( w ∗ · x ( i ) )( w · x ( i ) -w ∗ · x ( i ) ) = ( | w ∗ · x ( i ) | -sign( w ∗ · x ( i ) ) w · x ( i ) ) ≥ γ/ 2 . This in turn implies that

completing the proof of Claim 2.4.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract summarizes the result provided in Theorem 1.3 (and Theorem 2.1). The introduction describes how this contribution resolves an open problem in the literature by summarizing the motivation for the model and describing prior work's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The limitations are clearly stated in the statements of each theorem and are discussed in the introduction of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Each theorem statement provides all the assumptions and we provide a complete proof for all statements that is either in the main body of the paper or in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work is theoretical and we do not see any major or immediate implications on society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work is theoretical.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This work does not use any assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not use any assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper considers the problem of PAC learning halfspaces with margin in the presence of Massart noise. The paper provides an algorithm that well-balances sample and computational efficiency. Specifically, the dependence of the algorithm on both $\epsilon$ and $\gamma$ is near-optimal.

STRENGTHS:
1. The studied problem is fundamental in the area of PAC learning, and the paper provides a significant progress on it.
2. The paper is well written, and the main techniques are well explained.

WEAKNESSES:
1. Results might be somewhat weaker than presented (see questions), and some phrasings in this context are too vague (for example "there is evidence that...").
2. The natural agnostic extension of the problem is not discussed.

QUESTIONS:
Questions:
1. I'm not sure about the statement "...we essentially settle this question..."  in line 71. As far as I understand, the optimal computational sample complexity might be $\tilde{O}(1/\epsilon^2 + 1/\epsilon \gamma^2)$ and not $\tilde{\Omega}(1/\epsilon^2 \gamma^2)$. Either way, the provided upper bound is impressive enough to recommend for acceptance.
2. Is there a specific reason not to discuss the agnostic case? Is it usualy considered in Massart noise problems?


Suggestions:

Consider writing what is $\eta$ in the abstract.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of PAC-learning $\\gamma$-margin halfspaces under $\\eta$-Massart noise. The paper provides an efficient algorithm achieving error $\\eta+\\epsilon$ with sample complexity $\\tilde{O}(1/(\\epsilon^2\\gamma^2))$. Since previous work provided evidence that an inverse-quadratic dependence on $\\epsilon$ is necessary for efficient algorithms, the algorithm appears to be near-optimal in terms of sample-complexity up to logarithmic factors.\n\nThe algorithm relies on an iterative stochastic-gradient descent approach, where at each iteration a new loss function is defined which determines the gradient descent of the next iteration. The paper proves that if $T$ iterations are performed (where $T$ is a sufficiently large multiple of $\\log(1/\\delta)/(\\epsilon^2\\gamma^2)$), then with probability at least $1-\\delta$, one of the halfspaces obtained in $T$ iterations have error at most $\\eta+\\epsilon$. By drawing some extra independent samples and computing the empirical error for each of these halfspaces, one can find a good halfspace.\n\nThe paper is generally well written.\n\nSTRENGTHS:\nThe paper provides a near-optimal algorithm (in terms of sample complexity) for learning $\\gamma$-margin halfspaces under Massart noise.\n\nWEAKNESSES:\nI haven't found significant weaknesses in the paper.\n\nTypos:\n- Page 2, line 34: “We say that that the distribution” ->  “We say that the distribution” \n- Page 5: Line 195: If I am not mistaken, the function g is twice the gradient.\n\nQUESTIONS:\n- The authors claim that the computational complexity of the algorithm is linear in the samples. However, it seems to me that the complexity of step (5) is $O(T\\cdot N\\cdot d)$. It does not seem to be trivial to implement step (5) in $O((T + N)\\cdot d)$ time.\n\n- While previous work show that $\\Omega(1/(\\gamma^2\\epsilon))$ and $\\Omega(1/\\epsilon^2)$ are lower bounds, it doesn't necessarily follow that  $\\Omega(1/(\\epsilon^2\\gamma^2))$ is a lower bound as well.\n\nLIMITATIONS:\nSince this is a theoretical paper, I can't see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
This paper studies the problem of PAC learning $\gamma$-margin halfspaces under Massart noise: to PAC learn any distribution $D$ such that there exists $w^*$ with the bounded norm of $1$ that has a margin of at least $\gamma$, i.e. $\mathbb{P}_{(x,y)\sim D}( |\langle w^*,x \rangle| \geq \gamma)=1$ and for every $x$, we have $\mathbb{P}(\text{sign}(\langle w^*,x \rangle) \neq y )=\eta(x)$, where a function $\eta$ is bounded in $[0,\eta]$ (with overload of notation).

This is a classical problem in learning theory. Information theoretically, one needs $\tilde{\Theta}(1/\gamma^2 \epsilon)$ samples, while for computationally efficient algorithms, it is widely believed that inverse quadratic dependence in $\epsilon$ (e.g. $1/\epsilon^2$) is necessary. 

The main result of this paper is to present an efficient algorithm with nearly optimal sample complexity $\tilde{O}(1/\gamma^2 \epsilon^2)$, essentially closing the question.

STRENGTHS:
1. The paper studies a problem that is important to the learning theory community and provides a state-of-the-art result. The previous best algorithm required $O(1/\gamma^4 \epsilon^3)$ samples.
2. The paper is well-written.

WEAKNESSES:
I do not see any major weaknesses.

QUESTIONS:
None.

LIMITATIONS:
This is a theoretical work with no societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

comments: SUMMARY:
The paper studies the problem of learning a $\\gamma$-margin halfspace with $\\eta$ Massart noise and provides the first computationally efficient algorithm having 0-1 error $<= \\eta + \\varepsilon$ with sample complexity $O(1/\\gamma^2\\varepsilon^2)$ which nearly matches the information theoretic bound of $O(1/\\gamma^2\\varepsilon^)$ improving upon the previous $O(1/\\gamma^4\\varepsilon^3)$ sample complexity by [Chen et al. ‘20].
The main contributions of the paper are a novel “global” (as opposed to previous conditioning based) optimization to solve this problem directly using gradient descent. Towards this, a novel convex loss is defined using an independent vector as a parameter, and using the margin as a bound on the scaling parameter. The gradient w.r.t. the solution vector is used in an SGD loop, while the independent vector is updated by the gradient step, and its projection on the unit ball is the updated solution. This clever formulation leads to the gradient being composed of the “correct” direction of optimization along with an estimation error. The former minimizes the distance to the true solution, while the latter is bounded using standard concentration arguments.

STRENGTHS:
1. Novel loss formulation and gradient update step which can be directly used via SGD.
2. Simple algorithm and analysis.
3. Near optimal computational bound on an important problem.

WEAKNESSES:
Result is specific to a particular noise model and it is not clear if the techniques are more broadly applicable.

QUESTIONS:
In Algorithm 1, $\\lambda_t$ seems to be a constant independent of $t$, so it can be fixed outside the loop.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper essentially resolves the *computational sample complexity* of learning $\\gamma$-margin halfspaces under the Massart noise model. Here, computational sample complexity refers to the number of samples required for polynomial-time algorithms, as opposed to general statistical estimators which may be computationally intractable. Previous works have given rigorous evidence, in the form of lower bounds against large classes of algorithms such as SQ algorithms, of an *information-computation* gap for learning $\\gamma$-margin halfspaces. Without restrictions on computation, the sample complexity is $\\Theta(1/(\\gamma^2 \\epsilon))$ for achieving zero-one loss of $\\eta + \\epsilon$, where $\\eta$ is the uniform bound on the Massart noise. For a large class of efficient algorithms, however, the required sample complexity is at least $\\Omega(1/(\\gamma^2 \\epsilon^2))$.\n\nThey show that the $1/(\\gamma^2 \\epsilon^2)$ sample complexity is essentially tight by designing a polynomial-time algorithm for learning $\\gamma$-margin halfspaces. Their algorithm is based on a novel and technically insightful choice of the loss sequence for online SGD, which results in a surprisingly simple and efficient algorithm with near-optimal sample complexity.\n\nSTRENGTHS:\nThis is a well-written paper that stands out for being both technically insightful and simple. A simple and efficient algorithm (online SGD with a specific choice of loss sequence) achieves optimal sample complexity and lends itself to a clean analysis. It's hard to ask for more than this. \n\nThe core idea behind the algorithm is a clever choice of the loss sequence $(\\ell_t)$ with respect to which one runs online SGD. Previous works have already employed the LeakyReLU loss as a convex surrogate for the zero-one loss and achieved suboptimal upper bounds on the sample complexity. LeakyReLU with parameter $\\lambda > 0$ is defined by $\\ell_\\lambda(a) = (1-\\lambda) \\mathbb{1}[a \\ge 0] + \\lambda \\mathbb{1}[a < 0]$. Applying this to the halfspace learning setting, a straightforward calculation shows that $\\ell_\\lambda(-y (w \\cdot x)) = (\\mathbb{1}[\\mathrm{sign}(w\\cdot x) \\neq y] - \\lambda)|w \\cdot x|$, where $(x, y) \\in \\mathbb{S}^{d-1} \\times \\\\{\\pm 1\\\\}$ is a sample from the $\\gamma$-margin halfspace distribution and $w \\in \\mathbb{R}^d$ is a candidate halfspace. Note the resemblance to the *shifted* zero-one loss $L(w) = \\mathbb{E}_{(x,y)} \\mathbb{1}[\\mathrm{sign}(w\\cdot x) \\neq y] - \\eta$ (when $\\lambda = \\eta$ in LeakyReLU). By the $\\eta$-Massart noise assumption, $L(w^{\\*}) \\le 0$ for the optimal halfspace $w^{\\*} \\in \\mathbb{R}^d$ and $L(w) \\ge \\epsilon$ for any halfspace $w$ with zero-one loss at least $\\eta + \\epsilon$. \n\nThe key difference between this shifted zero-one loss and the LeakyReLU is the $|w \\cdot x|$ term. In particular, if we reweight each sample LeakyReLU loss by $1/|w \\cdot x|$, we recover the shifted zero-one loss which is unfortunately non-convex. The authors overcome this issue by considering a *family* of bounded and convex loss functions $(\\ell_u)$ indexed by $u \\in \\mathbb{R}^{d}$. Each $\\ell_u$ is simply the LeakyReLU loss reweighted by $1/|u \\cdot x|$. It is precisely this decoupling of the halfspace parameter $w$ and the reweighting parameter $u$ that leads to the guarantees of the algorithm. The *sequence* of reweighting parameters $(u_t)$, each of which leads to a different loss $\\ell_t$, is chosen adaptively by online SGD.\n\nWEAKNESSES:\nI did not find any significant weaknesses, only minor comments regarding the presentation.\n\n- **The expression \"(vector) v is independent of w\" (line 174, 176) is confusing.** I think it's easy to confuse \"independence\" with statistical independence. It would be helpful to clarify this by adding that the reweighting term is *constant* with respect to the parameter $w \\in \\mathbb{R}^d$, and the reweighting term $W(v \\cdot x, \\gamma)$ remains the same when taking the gradient of $L_{v}(w)$. This is already implicit in the mathematical expressions, but providing additional explanation would benefit readers.\n\nQUESTIONS:\n- Does the near-optimal online SGD algorithm fall within the class of SQ algorithms? If not, could there still exist non-SQ algorithms that achieve sample complexity with subquadratic dependence on $1/\\epsilon$?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}