Title: Learning Noisy Halfspaces with a Margin: Massart is No Harder than Random
Venue: NeurIPS 2024
Accepted: yes
Keywords: pac learning

learning halfspaces

massart noise

sgd

robust learning
Abstract: We study the problem of PAC learning $\gamma$-margin halfspaces with Massart noise. We propose a simple  proper learning algorithm, the Perspectron, that has sample complexity $\widetilde{O}((\epsilon\gamma)^{-2})$ and achieves classification error at most $\eta+\epsilon$ where $\eta$ is the Massart noise rate. 
 Prior works (DGT19, CKMY20)  came with worse sample complexity
 guarantees (in both $\epsilon$ and $\gamma$) or could only
 handle random classification noise (DDKWZ23,KITBMV23)--- a much milder noise assumption. 
We also show that our results extend to the more challenging setting of learning generalized linear models with a known link function under Massart noise, achieving a similar sample complexity to the halfspace case. This significantly improves upon the prior state-of-the-art in this setting due to CKMY20, who introduced this model.

Learning Noisy Halfspaces with a Margin: Massart is No Harder than Random

Abstract

We study the problem of PAC learning γ -margin halfspaces with Massart noise. We propose a simple proper learning algorithm, the Perspectron, that has sample complexity ˜ O (( ϵγ ) -2 ) and achieves classification error at most η + ϵ where η is the Massart noise rate. Prior works [DGT19b, CKMY20] came with worse sample complexity guarantees (in both ϵ and γ ) or could only handle random classification noise [DDK + 23, KIT + 23] - a much milder noise assumption. We also show that our results extend to the more challenging setting of learning generalized linear models with a known link function under Massart noise, achieving a similar sample complexity to the halfspace case. This significantly improves upon the prior state-of-the-art in this setting due to [CKMY20], who introduced this model.

1 Introduction

We study the problem of learning halfspaces with a margin, one of the oldest problems in the field of machine learning dating to work of Rosenblatt [Ros58]. Specifically, we consider the following formulation of this problem, where the label distribution is corrupted by Massart noise [MN06], where we use the following notation for halfspace hypotheses h w : R d →{± 1 } :

Definition 1 (Massart halfspace model) . Let η ∈ [0 , 1 2 ] and γ ∈ (0 , 1) . We say that a distribution D on B d ×{± 1 } is an instance of the η -Massart halfspace model with margin γ if the following hold.

There exists w ⋆ ∈ R d such that ∥ w ⋆ ∥ = 1 , 4 and D x has margin γ with respect to w ⋆ , 5 i.e., Pr x ∼ D x [ | w ∗ · x | < γ ] = 0 .

̸

For all x ∈ supp ( D x ) , there is an η ( x ) ∈ [0 , η ] such that Pr [ y = h w ⋆ ( x ) | x ] = η ( x ) .

We note that Definition 1 extends straightforwardly to general halfspaces up to rescaling (i.e., with larger domain size bounds and constant shift terms), as discussed in Remark 2.

The Massart noise model of Definition 1 for halfspaces (and more generally, binary classification problems) has garnered interest from the statistics, machine learning, and algorithms communities for a variety of reasons. This noise model was originally introduced as an intermediate noise model, between the simpler (from an algorithmic design standpoint) random classification noise (RCN) model [AL88], and the more challenging agnostic model [Hau92b, KSS94]. In the RCN model, η ( x ) in Definition 1 is restricted to be η pointwise, i.e., the noise level is uniform; polynomial-time PAC learning has long since known to be tractable under RCN [Byl94, BFKV98]. On the other hand, in the agnostic model (where learning is computationally intractable under well-studied conjectures [GR06, FGKP06, Dan16]), an adversary is allowed to arbitrarily modify an η fraction of labels. As observed by [Slo88], the Massart noise model of Definition 1 is equivalent to allowing an oblivious adversary control an η fraction of labels, where the η fraction is crucially sampled independently at random. It was stated as a longstanding open question [Coh97, Blu03] whether this obliviousness of the adversary impacts the polynomial-time tractability of learning halfspaces, even with a margin.

For additional motivation, it is reasonable to consider Massart noise to be a more realistic model of real-life noise (even when benign) when compared to the RCN model, as it allows for some amount of non-uniformity. This made Definition 1 a possibly tractable way to relax the noise assumption, without running into the aforementioned computational barriers for agnostic learning. In a series of recent exciting developments, in large part spurred by the breakthrough work of [DGT19b] who gave an (improper) polynomial-time PAC learning algorithm in the Massart halfspace model, significant algorithmic advances have been made towards understanding the polynomialtime tractability of learning under Massart noise [ABHU15, DGT19b, DKTZ20a, CKMY20, ZL21, DKK + 22]. However, less is understood about the fine-grained sample and computational complexity of these problems, which is potentially of greater interest from a practical perspective.

We investigate this question of fine-grained complexity for the Massart halfspace model, inspired by a line of recent work on semi-random models [BS95], a popular framework for understanding the overfitting of algorithms to their modeling assumptions. To motivate semi-random models, observe that from a purely information-theoretic standpoint, one might suspect that learning under Massart noise is actually easier than RCN; the noise level η ( x ) is only allowed to decrease, giving more 'signal' with respect to w ⋆ . However, this modification poses challenges when designing algorithms, e.g., because it breaks independence between y and x beyond the value sign( w ⋆ · x ) . Indeed, this is reflected in our current knowledge of halfspace learning algorithms. While it is known that one can learn halfspaces with margin γ under the RCN model to ϵ error (in the zero-one loss) using ˜ O (( ϵγ ) -2 ) samples [DDK + 23, KIT + 23], state-of-the-art learners under Massart noise use ˜ O ( γ -4 ϵ -3 ) samples (if required to be proper) [CKMY20] or ˜ O (min( γ -4 ϵ -3 , γ -3 ϵ -5 )) samples (otherwise) [DGT19b]. The semi-random model framework posits that this discrepancy reflects a lack of robustness in the current algorithmic theory for learning halfspaces, due to their overfitting to the RCN assumption.

For many statistical learning problems, new algorithms have been developed under semi-random modeling assumptions, with guarantees matching, or nearly-matching, classical algorithms under the corresponding fully random models [CG18, KLL + 23, JLM + 23, GC23, BGL + 24]. This leads us to our motivating problem, which aims to accomplish this goal for learning halfspaces.

Is it possible to design algorithms for learning in the Massart halfspace model with sample complexities matching the state-of-the-art for learning in the RCN model? (2)

1.1 Our results

As our main contribution, we resolve (2) in the affirmative in the setting of Definition 1. We also extend our results a substantial generalization of this model in Definition 2.

Massart halfspace model. We begin with our basic result in the setting of the Massart halfspace model, Definition 1. Our goal in this setting is to find a proper hypothesis halfspace h w ( x ) = sign( w · x ) for w ∈ B d , achieving good zero-one loss ℓ 0-1 (see Section 2 for a definition) over examples ( x , y ) drawn from the distribution D . Our main result to this end is the following.

Theorem 1 (Informal, see Theorem 3) . Let D be an instance of the η -Massart halfspace model with margin γ , and let ϵ ∈ (0 , 1) . Then, Perspectron (Algorithm 1) returns w ∈ B d such that ℓ 0-1 ( w ) ≤ η + ϵ with probability 0 . 99 , 6 using ˜ O ( γ -2 ϵ -2 ) samples and ˜ O ( dγ -2 ϵ -4 ) time.

We pause to comment on Theorem 1. First, our error guarantee is of the form η + ϵ rather than the more stringent goal of ℓ 0-1 ( w ⋆ ) + ϵ = E x ∼ D x [ η ( x )] + ϵ . There is strong evidence that this distinction is necessary for polynomial-time algorithms in the statistical query (SQ) framework of [Kea98], which our algorithm is an instance of, due to [CKMY20, DK20, NT22]. Next, the sample complexity bound of Theorem 1 matches the results of [DDK + 23, KIT + 23], the state-of-the-art under the milder RCN model. There is evidence that the dependences in Theorem 1 on both ϵ -1 and γ -1 are individually tight. In particular, [MN06] shows the sample complexity of the problem is ˜ Ω( γ -2 ϵ -1 ) , and [DDK + 23] shows any efficient algorithm in the SQ framework must use ˜ Ω( γ -1 / 2 ϵ -2 ) samples. We also remark that we can assume without loss of generality that η is known (see Appendix B.1).

Finally, as mentioned, prior to our work, the best-known polynomial-time learners under Definition 1 had sample complexities ˜ O (min( γ -4 ϵ -3 , γ -3 ϵ -5 )) [CKMY20, DGT19b]. In Table 1, we summarize relevant sample complexity bounds for learning variants of halfspace models with noise.

Massart generalized linear models. Our second result is an extension of Theorem 1 to the more challenging setting of learning generalized linear models (GLMs) with a known link function σ under Massart noise. As before, we only consider distributions that have a margin with respect to the optimal halfspace. We now formally define the setting we study.

Definition 2 (Massart GLM) . Let σ : [ -1 , 1] → [ -1 , 1] be an odd, non-decreasing function. We say that a distribution D on B d ×{± 1 } is an instance of the σ -Massart generalized linear model (GLM) with margin γ if the following conditions hold.

There exists w ⋆ ∈ R d such that ∥ w ⋆ ∥ = 1 and Pr [ | w ∗ · x | < γ ] = 0 .

̸

For all x ∈ supp ( D x ) , it holds that η ( x ) := Pr [ y = h w ⋆ ( x ) | x ] ≤ 1 -| σ ( w ⋆ · x ) | 2 .

Remark 1. We remark that the assumption that σ is odd is also commonly used in prior works (see, e.g., [CN08, DKTZ20a, CKMY20]). We further show that our result extends to σ with bounded asymmetry, albeit with a weaker error guarantee (see Definition 3 and Theorem 4).

To provide intuition for Definition 2, observe that that if η ( x ) = 1 -| σ ( w ⋆ · x ) | 2 for some x ∈ supp ( D x ) , then E [ y | x ] = | σ ( w ⋆ · x ) | sign( w ⋆ · x ) = σ ( w ⋆ · x ) , i.e., Definition 2 corresponds to the standard GLM definition. In Definition 2 (compared to Definition 1), we replace the fixed noise rate upper bound η with a data-dependent upper bound which is monotone (i.e., decreases as | w ⋆ · x | grows more confident). That is, Definition 2 generalizes the problem of learning Massart halfspaces, which follows by taking σ ( t ) = (1 -2 η )sign( t ) for all t ∈ [ -1 , 1] .

When working with a Massart GLM D , we define opt RCN := E ( x ,y ) ∼ D [ 1 -| σ ( w ⋆ · x ) | 2 ] . Note that in the special case of a Massart halfspace model, we simply have opt RCN = η . As in the case of Massart half-

̸

spaces, known SQ lower bounds make competing with opt := ℓ 0-1 ( w ⋆ ) := Pr ( x ,y ) ∼ D [ y = h w ⋆ ( x )] an intractable target, so our focus is again on attaining ℓ 0-1 ( w ) ≈ opt RCN .

̸

To our knowledge, the model in Definition 2 was first studied in [CKMY20], though we note that similar models have been considered in prior works [ZLC17, HKLM20, DKTZ20a], which we describe and compare to Definition 2 in Section 1.3. In [CKMY20], the parameterization of this model is slightly different; they assume σ is L -Lipschitz and that Pr x ∼ D x [ | σ ( w ∗ · x ) | ≥ γ ] = 1 , i.e., they impose a margin on σ ( w ∗ · x ) rather than w ∗ · x . This implies our margin assumption (with γ ← γ L in Definition 2), but not vice versa. Under their slightly more restrictive assumptions, [CKMY20] claims a runtime which is an unspecified polynomial in Lγ -1 ϵ -1 , that is at least ˜ Ω( L 4 γ -4 ϵ -6 ) when specialized to the halfspace case (see their Theorems 5.2 and 6.14). On the other hand, we achieve improved rates extending our simple algorithmic approach for the Massart halfspace case in this more challenging setting.

Theorem 2 (Informal, see Theorem 4) . Let D be an instance of the σ -Massart GLM with margin γ , and let ϵ ∈ (0 , 1) . There is an algorithm returning w ∈ B d so that ℓ 0-1 ( w ) ≤ opt RCN + ϵ with probability 0 . 99 , using ˜ O ( γ -2 ϵ -4 ) samples and ˜ O ( dγ -2 ϵ -6 ) time.

In particular, parameterizing our problem using the margin and Lipschitz assumptions in [CKMY20] (with γ ← γ L ), we obtain an improved sample complexity of ˜ O ( L 2 γ -2 ϵ -4 ) .

1.2 Technical overview

Learning Massart halfspaces. Our learning algorithms are inspired by the certificate framework for learning with semi-random noise developed in [DKTZ20d, CKMY20]. In that framework, given a sub-optimal hypothesis w , i.e., with error Pr ( x ,y ) ∼ D [sign( w · x ) = y ] ≥ η + ϵ , the goal is to construct a certificate of sub-optimality in the form of a separating hyperplane between w and the target w ∗ , i.e., a vector g such that g · w ≥ g · w ⋆ ⇐⇒ g · ( w -w ⋆ ) ≥ 0 . Given such a separating hyperplane, prior works rely on cutting-plane methods (e.g., [Vai96]) or on first-order regret minimization methods to learn a hypothesis achieving the target error.

We first describe our algorithm in the halfspace setting, by motivating our choice of a certificate. Prior work [CKMY20] achieving a proper Massart halfspace learner uses the gradient of the Leaky-ReLU objective ℓ η ( t ) := (1 -η ) max(0 , t ) -η max(0 , -t ) conditioned on a band around the current hypothesis w as a separating hyperplane. That is, they argue that for some appropriate interval I , it holds that E [ ∇ ℓ η ( -y w · x ) | x · w ∈ I ] · ( w -w ∗ ) ≥ 0 . This yields a sample complexity scaling as ˜ O ( ϵ -3 ) because one has to sample condionally from the band I and estimate the gradient of the Leaky-ReLU on these samples up to error ϵ , as well as additional overhead in γ -1 due to use of expensive outer loops taking advantage of these certificates, such as cutting-plane methods.

To avoid the sample complexity overhead of this conditioning (implemented via rejection sampling), we use a simple reweighting scheme pointwise, which puts a larger weight of | w · x | -1 (an inverse margin) on points closer to the boundary of our current hypothesis h w . Intuitively, this reweighting is a soft implementation of the hard conditioning done in [CKMY20]. This is motivated by our first important observation: any significantly sub-optimal hypothesis w with ℓ 0-1 ( w ) ≥ η + ϵ satisfies

proven in Lemma 1. This suggests using g = E [ ∇ ℓ η ( -y w · x ) | w · x | -1 ] (rather than E [ ∇ ℓ η ( -y w · x ) | x · w ∈ I ] as in [CKMY20]) as our certificate, which we can estimate via a single sample.

While reweighting by the inverse margin | w · x | -1 gives a separating hyperplane certificate, it may be impossible to estimate this certificate from few samples, e.g., if the weight | w · x | -1 is often very large, which introduces significant variance. For instance, even if D x has margin with respect to a target w ⋆ , this is not necessarily true with respect to the current hypothesis w (without further distributional assumptions on D x ). To overcome this, we change our pointwise reweighting to be less aggressive and instead use ( | w · x | + γ ) -1 . In Lemma 2, we exploit the margin assumption about D x to show that when ℓ 0-1 ( w ) ≥ η + ϵ , it is still the case that E [ ∇ ℓ η ( -y w · x )( | w · x | + γ ) -1 ] · ( w -w ∗ ) ≥ ϵ . Moreover, we still have an unbiased estimator for this separating hyperplane E [ ∇ ℓ η ( -y w · x )( | w · x | + γ ) -1 ] , and the estimator is bounded in Euclidean norm by γ -1 with probability 1 by our margin assumption.

Standard concentration inequalities now show ˜ O ( γ -2 ϵ -2 ) samples suffice to obtain a separation oracle with high probability. Plugging this certificate into oracle-efficient cutting-plane methods (e.g., [Vai96]) implies an algorithm with sample complexity ˜ O ( γ -4 ϵ -2 ) (after a random projection process [AV99] to reduce to ˜ O ( γ -2 ) dimensions). This already improves upon the prior best-known ˜ O ( γ -4 ϵ -3 ) sample complexity. We further improve our dependence on γ by using it in a perceptronlike regret minimization scheme, where at every step we update the current hypothesis w using the aforementioned bounded unbiased estimator of our certificate, see Lemma 3 and Algorithm 1. Overall, our algorithm iterates the following very simple update for a step-size λ > 0 and β := 1 -2 η :

Since at every step we perform an (approximate) perspective projection ( | w · x | + γ ) -1 on our sample, we call Algorithm 1 which iterates the update in Equation (3) the Perspectron .

Learning Massart GLMs. For learning Massart GLMs (Definition 2), we use a similar certificatebased approach. While it is simple to show reweighting with the inverse margin | w · x | -1 still works in this case (see Lemma 5), using the bounded reweighting ( | w · x | + γ ) -1 does not. Instead we use a new reweighting of of the form ( | w · x | + αγ ) where α = O ( ϵ ) (see Lemma 6). Using a similar iterative method as the Perspectron defined in (3), we obtain our sample complexity of ˜ O ( γ -2 ϵ -4 ) for learning Massart GLMs.

1.3 Related work

We briefly survey some additional related works here. First, a common worst-case assumption used in statistical learning is that the label noise is adversarial (a.k.a. agnostic) [Hau92a]. In that setting, a lot of progress has been made for learning halfspaces when the underlying distribution satisfies structural assumptions (e.g., it is Gaussian or log-concave) [KKMS05, KOS08, ABHU15, DKS18, DKTZ20c, DKTZ22]. For learning halfspaces with a margin, the best-known agnostic results have runtime and sample complexity that depend exponentially on the margin γ and/or the accuracy parameter ϵ [SSS09, LS11, DKM19]. Another important line of work [DKTZ20d, DKK + 20, ZL21] has focused on learning halfspaces under Tsybakov noise: a semi-random noise model that extends Massart noise, but is still easier than the agnostic setting. We also note that variants of Definition 2 have appeared before: the generalized Tsybakov low noise condition of [HKLM20] is a close relative which imposes different noise rates within and outside a margin, and the strong Massart noise of [ZLC17, DKTZ20a] is an instance of Definition 2 without the margin restriction.

Our algorithms rely on the certificate framework developed in [DKTZ20b, CKMY20] and the LeakyReLU loss that has been extensively used in prior works on learning with random classification and Massart label noise [Byl98, DGT19a, CKMY20, DKT21]. Our main technical contribution is a new certificate that relies on an inverse-margin reweighting scheme and can be estimated using a single sample at every iteration. Similar, 'inverse-margin' reweighting schemes have been used for learning general halfspaces [CKMY20] and online linear classification [DKTZ24]. Those results have no implications for the sample complexity of the problem studied here. Finally, we mention that a local reweighting scheme that is somewhat similar in spirit to ours (but very different in its implementation) was previously employed by [KLL + 23], for a different semi-random statistical learning problem.

1.4 Limitations and open problems

One interesting open direction is providing improved sample complexity guarantees for more general noise models. For example, the misspecified GLM framework (Definition 3.2, [CKMY20]) generalizes Definition 2 to include an additional misspecification parameter ζ such that η ( x ) ≤ 1 -| σ ( w ∗ · x ) | 2 + ζ . Our approach does not directly apply in this setting, since ζ = 0 is important for our separation oracle result of Lemma 6. A different interesting generalization of the noise model corresponds to the case where the link function σ is unknown, which the [CKMY20] algorithm can handle (at a much higher sample complexity). While our algorithms require knowledge of σ , it would be interesting to explore whether techniques from learning single-index models (e.g., [KS09, GGKS23, ZWDD24]) can be used to extend our algorithms in this setting.

Aclear next step is to design efficient algorithms with sample complexities independent of the margin but still linear in the dimension d . 7 , e.g., with an ≈ dϵ -2 sample complexity. In prior work [DKT21] such an algorithm is given, albeit with a significantly worse poly( dϵ -1 ) sample complexity.

2 Preliminaries

We denote vectors in lower-case boldface, and ∥ x ∥ is the Euclidean norm of x ∈ R d . We use B d to denote the unit ball in R d , i.e. B d := { x ∈ R d | ∥ x ∥ ≤ 1 } . We use Π B d ( w ) := min { 1 , 1 ∥ w ∥ } w to denote the Euclidean projection of a vector w ∈ R d onto B d . We let 0 d be the all-zeroes vector in dimension d . We reserve the overline notation ¯ x to denote the unit vector in the direction of x , i.e. ¯ x := x ∥ x ∥ . We let sign : R →{± 1 } be defined so sign( t ) = 1 iff t ≥ 0 . We use ✶ {E} to denote the 0 -1 indicator of a random event E , Pr [ E ] to denote its probability, and E to denote the expectation operator. The support of a distribution D is denoted supp ( D ) , and [ N ] := { i ∈ N | i ≤ N } .

̸

Throughout the paper we study the problem of learning a binary classifier, given labeled examples from a distribution D over labeled examples ( x , y ) ∈ R d × { 0 , 1 } , under various models on the distribution to be discussed. We refer to the x -marginal of D by D x , and the conditional distribution of the label y | x by D y ( x ) . We will primarily be interested in learning halfspace hypotheses, which for w ∈ R d are the corresponding functions h w : R d →{± 1 } defined by h w ( x ) := sign ( w · x ) . We also denote the zero-one loss of a halfspace hypothesis h w corresponding to w ∈ R d as follows, when the distribution D over labeled examples is clear from context: ℓ 0-1 ( w ) := Pr ( x ,y ) ∼ D [ h w ( x ) = y ] . We define the Leaky-ReLU function with parameter λ > 0 as ℓ λ ( t ) := (1 -λ ) max(0 , t ) -λ max(0 , -t ) . Given vectors w , x and y ∈ {± 1 } , it holds that the (sub) gradient of ℓ λ ( -y w · x ) with respect to w is ∇ w ℓ λ ( -y w · x ) = 1 2 ((1 -2 λ )sign( w · x ) -y ) · x . We also provide some brief remarks on how to extend Definition 1 to more general settings here.

̸

Remark 2. Definition 1 extends straightforwardly to the case where D x has margin γ and is supported on a subset of R · B d for R = 1 , by rescaling so R ← 1 and γ ← γ R , as halfspace hypotheses and our label noise assumptions only depend on signs. Other than these margin and support assumptions, we make no additional distributional assumptions about the x -marginal D x . Further, due to working in the distributional assumption-free setting, we can assume with up to constant factor loss(in margin) that the halfspace is homogeneous, i.e., has no constant shift term. That is, given a halfspace sign( w · x + b ) with ∥ w ∥ , | b | ≤ 1 , after a feature expansion ( e : x ↦→ 1 √ 2 ( x , 1) ) the halfspace h w ′ ( z ) = sign(( w , b ) · z ) is homogeneous while still having a margin ≥ γ 2 with respect to w ′ = 1 √ 1+ b 2 ( w , b ) . Finally, the Massart noise model of [MN06] is defined for any hypothesis class and is not tied to halfspaces. Since we focus on learning halfspaces with a margin, we combined the hypothesis class { h w } w ∈ R d with the noise model in Definition 1 for simplicity.

3 Massart halfspaces

In this section, we give our result on learning Massart halfspaces with margin. Our proof is surprisingly short, and we separate it into its two main components: a structural lemma in Section 3.1 which shows how to estimate a separating hyperplane given a sub-optimal w , and a perceptron-like analysis of a stochastic iterative method in Section 3.2.

3.1 Separating hyperplanes for Massart halfspaces

We prove our main structural lemma here, used to argue the progress of our iterative method. As highlighted in Section 1.2, we show that when the current ℓ 0-1 ( w ) ≥ η + ϵ , we can construct an unbiased estimator for a separating hyperplane between w and the target vector w ⋆ .

Warmup: an 'unbounded' separating hyperplane. Before presenting the full proof, we first give a separating hyperplane that works for any feature distribution - even without margin assumptions. The proposed separating hyperplane works due to the fact that we can express the zero-one in terms of the Leaky-ReLU which is a convex function, as was previously observed by [DGT19b].

Lemma 1 (Separating hyperplane for Massart halfspaces) . Let D be an instance of the η -Massart halfspace model, and w ∈ R d has classification error ℓ 0-1 ( w ) ≥ η + ϵ . It holds that E ( x ,y ) ∼ D [ ∇ w ℓ η ( -y w · x ) | w · x | ] · ( w -w ∗ ) ≥ ϵ .

̸

Proof. We recall Claim 2.1 from [DGT19b]: for all w , x , it holds that E y ∼ D y ( x ) [ ℓ λ ( -y w · x )] = ( Pr y ∼ D y ( x ) [sign( w · x ) = y ] -λ ) ·| w · x | . In particular, we have E ( x ,y ) ∼ D [ ℓ η ( -y w · x ) | w · x | ] = ℓ 0-1 ( w ) -η . From the convexity of ℓ η ( -y w · x ) , we obtain that ∇ w ℓ η ( -y w · x ) · ( w -w ∗ ) ≥ ℓ η ( -y w · x ) -ℓ η ( -y w ∗ · x ) . By dividing both sides by | w · x | and taking expectation over x and y , we obtain

̸

where the last inequality follows from the following facts: (1) for all x , it holds that E y ∼ D y ( x ) [ ℓ η ( -y w ∗ · x )] = ( Pr y ∼ D y ( x ) [sign( w ∗ · x ) = y ] -η ) ·| w ∗ · x | = ( η ( x ) -η ) ·| w ∗ · x | ≤ 0 , and (2) E x ,y [ ℓ η ( -y w · x ) | w · x | ] = ℓ 0-1 ( w ) -η ≥ ϵ . This completes the proof.

A'bounded' separating hyperplane for γ -margin Massart halfspaces. Our claim in the previous lemma was very general: it works for any marginal distribution. However, as discussed in Section 1.2, the unbounded nature of this separating hyperplane may make it impossible to estimate from samples. To overcome this, we propose a new candidate hyperplane: E x ,y [ ∇ w ℓ η ( -y w · x ) | w · x | + γ ] . We prove that this candidate is indeed a separating hyperplane by leveraging the fact that we have margin γ with respect to the optimal halfspace w ∗ . Recall from Section 2 that ∇ w ℓ η ( -y w · x ) = 1 2 ((1 -2 η )sign( w · x ) -y ) .

Lemma 2 (Bounded separating hyperplane for Massart halfspaces) . Let D be an instance of the η -Massart halfspace model with margin γ (with respect to w ⋆ ) and define β = 1 -2 η . If w ∈ R d has ℓ 0-1 ( w ) ≥ η + ϵ , it holds that

Proof. We first observe that by the definition of the Massart halfspace model, E y ∼ D y ( x ) [ y ] = (1 -2 η ( x ))sign( w ∗ · x ) = β ( x )sign( w ⋆ · x ) , where β ( x ) := 1 -2 η ( x ) . Therefore, we have that

We denote by g ( x ) := ( β sign( w · x ) -β ( x )sign( w ∗ · x )) ( w · x -w ∗ · x ) | w · x | + γ , which we bound differently based on whether x falls in the agreement region A := { x ∈ B d | h w ⋆ ( x ) = h w ( x ) } . For x ∈ A ,

̸

The second equality follows from the fact that sign( w ∗ · x ) = sign( w · x ) . The final inequality holds since β -β ( x ) ≤ 0 and | w · x |-| w ∗ · x | | w · x | + γ ≤ 1 . Similarly, for x / ∈ A , an analogous calculation yields g ( x ) = ( β + β ( x ) ) | w · x | + | w ∗ · x | | w · x | + γ ≥ β + β ( x ) . The first equality holds because sign( w ∗ · x ) = sign( w · x ) and the final inequality follows since | w ∗ · x | ≥ γ from the margin assumption. Thus

We will now use our lower bound on ℓ 0-1 ( w ) , which we relate to Equation (4). We have ℓ 0-1 ( w ) = E x ∼ D x [ ✶ { x ∈ A } η ( x )] + E x ∼ D x [ ✶ { x ̸∈ A } (1 -η ( x ))] = E x ∼ D x [ ✶ { x ̸∈ A } β ( x )] + E x ∼ D x [ η ( x )] . Next, by our definition β ( x ) = 1 -2 η ( x ) , rearranging and expanding we have:

We finish the proof by combining Equation (4), Equation (5), and ℓ 0-1 ( w ) -η ≥ ϵ .

3.2 Perspectron

We now present and analyze Perspectron , our algorithm for learning Massart halfspaces.

̸

We begin by giving a self-contained analysis of a single loop j ∈ [ N ] of Line 6 to Line 9, showing that for sufficiently large T , at least one iterate achieves small ℓ 0-1 with constant probability.

Lemma 3. Let { x i , y i } i ∈ [ T ] ∼ i.i.d. D , where D is an instance of the η -Massart halfspace model with margin γ with respect to w ⋆ . Consider iterating, from w 1 := 0 d ,

Proof. Throughout the proof, say w ∈ R d is bad iff ℓ 0-1 ( w ) ≥ η + ϵ 2 , and let E t denote the event that all of the iterates { w s } s ∈ [ t ] updated according to (6) are bad. Define the potential function Φ t := E [ ✶ {E t } · ∥ w ⋆ -w t ∥ 2 ] for all t ∈ [ T ] . On expanding the expression for the squared norm

and using the fact that {E t +1 } ≤ {E t } ,

Here, the third inequality used x t ∈ B d and | β sign( w t · x t ) -y t | ≤ 2 , and the fourth applied Lemma 2. Now using that Φ 1 ≤ ∥ w ⋆ ∥ 2 = 1 , Φ T +1 ≥ 0 , and Pr [ E t ] ≥ Pr [ E T ] for t ∈ [ T ] , we have 2 λϵT Pr [ E T ] ≤ 1 + 4 λ 2 T γ 2 . The conclusion Pr [ E T ] ≤ 1 2 follows from our choices of λ , T .

We next analyze the hypothesis selection step in Line 13.

Lemma 4 (Hypothesis selection) . Suppose there exists ˆ w ∈ H with ℓ 0-1 ( w ) ≤ η + ϵ 2 . Then if T 2 ≥ 8 ϵ 2 log( 2 | H | δ ) , with probability ≥ 1 -δ the w returned by Line 13 satisfies ℓ 0-1 ( w ) ≤ η + ϵ .

̸

Proof. Because S is independent of H , Hoeffding's inequality and a union bound implies that ∣ ∣ Pr ( x ,y ) ∼ unif. S [ h w ( x ) = y ] -ℓ 0-1 ( w ) ∣ ∣ ≤ ϵ 4 with probability ≥ 1 -δ , for all w ∈ H . Conditioning on this event, ℓ 0-1 ( w ) > η + ϵ yields a contradiction:

̸

We are now ready to state and prove our main theorem.

Theorem 3 (Learning γ -margin Massart halfspaces) . Let D be an instance of the η -Massart halfspace model with margin γ , and let ϵ, δ ∈ (0 , 1) . Algorithm 1 with T 1 ≥ 16 ϵ 2 γ 2 ⌈ log 2 ( 2 δ ) ⌉ , T 2 ≥ 8 ϵ 2 log( 4 | T 1 | δ ) returns w such that ℓ 0-1 ( w ) ≤ η + ϵ with probability ≥ 1 -δ , using O (( ϵγ ) -2 log( δ -1 ) + ϵ -2 log(( ϵγδ ) -1 )) samples and O ( dϵ -4 γ -2 log( δ -1 ) log(( ϵγδ ) -1 ) time.

Proof. First, applying Lemma 3 to each of the N independent runs of Line 6 to Line 9 shows that the premise of Lemma 4 is met except with probability δ 2 . The correctness claim then follows from Lemma 4. The sample complexity is immediate, and the runtime bound follows because the bottleneck operation is computing the value of h w ( x ) for all ( x , y ) ∈ S and w ∈ H .

4 Massart generalized linear models

In this section, we present a key piece of intuition motivating our extension to the Massart GLM noise model (see Definition 2), deferring a full proof to Appendix A. In this setting, instead of being upper bounded by a fixed constant η , the noise rate is data-dependent and upper bounded by 1 -σ ( w ⋆ · x ) 2 where σ is odd non-decreasing and w ⋆ is the optimal halfspace. Inspired by our approach in Section 3, we propose a novel separating hyperplane based on the previously described reweighting scheme. We argue in this section that E x ,y [ ( σ ( w · x ) -y ) | w · x | x ] is a valid separating hyperplane, generalizing Lemma 1.

Lemma 5 (Separating hyperplane for Massart GLMs) . Let D be an instance of the σ -Massart GLM model with margin γ (with respect to halfspace w ⋆ ), and w ∈ R d has ℓ 0-1 ( w ) ≥ opt RCN + ϵ . We have that E ( x ,y ) ∼ D [ ( σ ( w · x ) -y ) | w · x | x ] · ( w -w ∗ ) ≥ 2 ϵ .

̸

Proof Sketch. We use the definition of sets A , β ( x ) from Lemma 2. By expanding out the expression for ℓ 0-1 ( w ) similarly to Lemma 2, we obtain that 1 2 · E ( x ,y ) ∼ D [( | σ ( w ∗ · x ) | -β ( x )) ✶ { x ∈ A } ] + 1 2 · E ( x ,y ) ∼ D [( | σ ( w ∗ · x ) | + β ( x )) ✶ { x / ∈ A } ] ≥ ϵ . Let g ( x ) = ( σ ( w · x ) -y ) | w · x | · ( w -w ∗ ) . Now, we argue that E x ∼ D x [ g ( x )] is greater than the left hand side of the previous inequality. We do a case analysis. For x ∈ A , we observe that g ( x ) ≥ ( | σ ( w ∗ · x ) | -β ( x )) | w · x |-| w ∗ · x | | w · x | ≥ ( | σ ( w ∗ · x ) | -β ( x )) . Here, we obtained the first inequality by adding and subtracting the corresponding term with σ ( w ∗ · x ) and then using monotonicity. The final inequality follows from the fact that β ( x ) ≥ | σ ( w ∗ · x ) | . For x / ∈ A , we obtain that g ( x ) = ( | σ ( w · x ) | + β ( x )) | w · x | + | w ∗ · x | | w · x | ≥ ( | σ ( w ∗ · x ) | + β ( x )) where we obtain the inequality by doing a case analysis: (1) | w · x | ≤ | w ∗ · x | , in this case | w · x | + | w ∗ · x | | w · x | ≥ 2 and 2 β ( x ) ≥ ( | σ ( w ∗ · x ) | + β ( x )) and (2), | w · x | ≥ | w ∗ · x | , in this case | σ ( w · x ) | ≥ | σ ( w ∗ · x ) | and hence we are done. Now, taking the expectation of g ( x ) completes the proof of the claim.

However, we are met with the same obstacle as before: | w · x | can be arbitrarily small. The previous approach of adding γ to the denominator does not work immediately. Instead, we add the rescaled term ϵ 2 -ϵ · γ . Adding a smaller term in the denominator increases the bound on the norm of the increments in each step, resulting to a larger bound on the number of iterations (and, therefore, sample complexity) by a factor of ϵ -2 . However, this rescaling is useful to obtain an analogue of Lemma 2 for the case of Massart GLMs (Lemma 6). The rescaling essentially accounts for the part of the distribution where | w · x | is smaller than | w ∗ · x | and the signs disagree. This is important because the size of | w · x | is quantitatively more significant in the Massart GLM case.

Combining this with a modified version of the Perspectron algorithm and analysis (see Algorithm 2), we obtain our final result with sample complexity ˜ O ( γ -2 ϵ -4 ) (see Theorem 4).

References

[ABHU15] Pranjal Awasthi, Maria-Florina Balcan, Nika Haghtalab, and Ruth Urner. Efficient learning of linear separators under bounded noise. In Proceedings of The 28th Conference on Learning Theory, COLT 2015 , volume 40 of JMLR Workshop and Conference Proceedings , pages 167-190. JMLR.org, 2015.

[AL88] D. Angluin and P. Laird. Learning from noisy examples. Mach. Learn. , 2(4):343-370, 1988.

[AV99] R. Arriaga and S. Vempala. An algorithmic theory of learning: Robust concepts and random projection. In Proceedings of the 40 th Annual Symposium on Foundations of Computer Science (FOCS) , pages 616-623, New York, NY, 1999.

[BFKV98] Avrim Blum, Alan M. Frieze, Ravi Kannan, and Santosh S. Vempala. A polynomial-time algorithm for learning noisy linear threshold functions. Algorithmica , 22(1/2):35-52, 1998.

[BGL + 24] Avrim Blum, Meghal Gupta, Gene Li, Naren Sarayu Manoj, Aadirupa Saha, and Yuanyuan Yang. Dueling optimization with a monotone adversary. In International Conference on Algorithmic Learning Theory , volume 237 of Proceedings of Machine Learning Research , pages 221-243. PMLR, 2024.

[Blu03] A. Blum. Machine learning: My favorite results, directions, and open problems. In 44 th Symposium on Foundations of Computer Science (FOCS 2003) , pages 11-14, 2003.

[BS95] Avrim Blum and Joel Spencer. Coloring random and semi-random k-colorable graphs. J. Algorithms , 19(2):204-234, 1995.

[Byl94] Tom Bylander. Learning linear threshold functions in the presence of classification noise. In Proceedings of the Seventh Annual ACM Conference on Computational Learning Theory, COLT 1994 , pages 340-347. ACM, 1994.

[Byl98] T. Bylander. Worst-case analysis of the Perceptron and exponentiated update algorithms. Artificial Intelligence , 106, 1998.

[CG18] Yu Cheng and Rong Ge. Non-convex matrix completion against a semi-random adversary. In Conference On Learning Theory, COLT 2018 , volume 75 of Proceedings of Machine Learning Research , pages 1362-1394. PMLR, 2018.

[CKMY20] Sitan Chen, Frederic Koehler, Ankur Moitra, and Morris Yau. Classification under misspecification: Halfspaces, generalized linear models, and evolvability. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 , 2020.

[CN08] R. M. Castro and R. D. Nowak. Minimax bounds for active learning. IEEE Transactions on Information Theory , 54(5):2339-2353, 2008.

[Coh97] Edith Cohen. Learning noisy perceptrons by a perceptron in polynomial time. In 38th Annual Symposium on Foundations of Computer Science, FOCS '97 , pages 514-523. IEEE Computer Society, 1997.

[Dan16] Amit Daniely. Complexity theoretic limitations on learning halfspaces. In Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2016 , pages 105-117. ACM, 2016.

[DDK + 23] Ilias Diakonikolas, Jelena Diakonikolas, Daniel M. Kane, Puqian Wang, and Nikos Zarifis. Information-computation tradeoffs for learning margin halfspaces with random classification noise. In The Thirty Sixth Annual Conference on Learning Theory, COLT 2023 , volume 195 of Proceedings of Machine Learning Research , pages 2211-2239. PMLR, 2023.

[DGT19a] I. Diakonikolas, T. Gouleakis, and C. Tzamos. Distribution-independent pac learning of halfspaces with massart noise. In Advances in Neural Information Processing Systems 32, NeurIPS . 2019.

[DGT19b] Ilias Diakonikolas, Themis Gouleakis, and Christos Tzamos. Distribution-independent PAC learning of halfspaces with massart noise. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 , pages 4751-4762, 2019.

[DK20] I. Diakonikolas and D. M. Kane. Hardness of learning halfspaces with massart noise. CoRR , abs/2012.09720, 2020.

[DKK + 20] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. A polynomial time algorithm for learning halfspaces with tsybakov noise. arXiv , 2020.

[DKK + 22] Ilias Diakonikolas, Daniel M Kane, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning general halfspaces with general massart noise under the gaussian distribution. In Symposium on Theory of Computation , volume 54, 2022.

[DKM19] Ilias Diakonikolas, Daniel Kane, and Pasin Manurangsi. Nearly tight bounds for robust proper learning of halfspaces with a margin. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

[DKS18] I. Diakonikolas, D. M. Kane, and A. Stewart. Learning geometric concepts with nasty noise. In Proceedings of the 50 th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 1061-1073, 2018.

[DKT21] Ilias Diakonikolas, Daniel Kane, and Christos Tzamos. Forster decomposition and learning halfspaces with noise. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 7732-7744. Curran Associates, Inc., 2021.

[DKTZ20a] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with massart noise under structured distributions. In Conference on Learning Theory, COLT , 2020.

[DKTZ20b] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with tsybakov noise. arXiv , 2020.

[DKTZ20c] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Non-convex SGD learns halfspaces with adversarial label noise. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[DKTZ20d] Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning halfspaces with tsybakov noise. arXiv e-prints , pages arXiv-2006, 2020.

[DKTZ22] Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning general halfspaces with adversarial label noise via online gradient descent. In International Conference on Machine Learning , pages 5118-5141. PMLR, 2022.

[DKTZ24] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Online Linear Classification with Massart Noise, 2024. Arxiv eprint: 2405.12958.

[FGKP06] V. Feldman, P. Gopalan, S. Khot, and A. K. Ponnuswami. New results for learning noisy parities and halfspaces. In 47 th Annual IEEE Symposium on Foundations of Computer Science (FOCS) , pages 563-574. IEEE Computer Society, 2006.

[GC23] Xing Gao and Yu Cheng. Robust matrix sensing in the semi-random model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 , 2023.

[GGKS23] A. Gollakota, P. Gopalan, A. R. Klivans, and K. Stavropoulos. Agnostically learning single-index models using omnipredictors. arXiv preprint arXiv:2306.10615 , 2023.

[GR06] Venkatesan Guruswami and Prasad Raghavendra. Hardness of learning halfspaces with noise. In 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2006) , pages 543-552. IEEE Computer Society, 2006.

[Hau92a] D. Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation , 100:78-150, 1992.

[Hau92b] David Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Inf. Comput. , 100(1):78-150, 1992.

[HKLM20] M. Hopkins, D. M. Kane, S. Lovett, and G. Mahajan. Noise-tolerant, reliable active classification with comparison queries. In COLT , 2020.

[JLM + 23] Arun Jambulapati, Jerry Li, Christopher Musco, Kirankumar Shiragur, Aaron Sidford, and Kevin Tian. Structured semidefinite programming for recovering structured preconditioners. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 , 2023.

[Kea98] M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM , 45(6):983-1006, 1998.

[KIT + 23] Vasilis Kontonis, Fotis Iliopoulos, Khoa Trinh, Cenk Baykal, Gaurav Menghani, and Erik Vee. Slam: Student-label mixing for distillation with unlabeled examples. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 , 2023.

[KKMS05] A. Kalai, A. Klivans, Y. Mansour, and R. Servedio. Agnostically learning halfspaces. In Proceedings of the 46 th IEEE Symposium on Foundations of Computer Science (FOCS) , pages 11-20, 2005.

[KLL + 23] Jonathan A. Kelner, Jerry Li, Allen Liu, Aaron Sidford, and Kevin Tian. Semi-random sparse recovery in nearly-linear time. In The Thirty Sixth Annual Conference on Learning Theory, COLT 2023 , volume 195 of Proceedings of Machine Learning Research , pages 2352-2398. PMLR, 2023.

[KOS08] A. Klivans, R. O'Donnell, and R. Servedio. Learning geometric concepts via Gaussian surface area. In Proc. 49th IEEE Symposium on Foundations of Computer Science (FOCS) , pages 541-550, Philadelphia, Pennsylvania, 2008.

[KS09] A. T. Kalai and R. Sastry. The isotron algorithm: High-dimensional isotonic regression. In COLT . Citeseer, 2009.

[KSS94] Michael J. Kearns, Robert E. Schapire, and Linda Sellie. Toward efficient agnostic learning. Mach. Learn. , 17(2-3):115-141, 1994.

[LS11] P. Long and R. Servedio. Learning large-margin halfspaces with more malicious noise. NIPS , 2011.

[MN06] P. Massart and E. Nedelec. Risk bounds for statistical learning. Ann. Statist. , 34(5):23262366, October 2006.

[NT22] R. Nasser and S. Tiegel. Optimal SQ lower bounds for learning halfspaces with massart noise. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1047-1074. PMLR, 2022.

[Ros58] F. Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review , 65:386-407, 1958.

[Slo88] R. H. Sloan. Types of noise in data for concept learning. In Proceedings of the First Annual Workshop on Computational Learning Theory , COLT '88, pages 91-96, San Francisco, CA, USA, 1988. Morgan Kaufmann Publishers Inc.

[SSS09] S. Shalev Shwartz, O. Shamir, and K. Sridharan. Agnostically learning halfspaces with margin errors. TTI Technical Report, 2009.

[Vai96] P. M. Vaidya. A new algorithm for minimizing convex functions over convex sets. Math. Prog. , 73(3):291-341, 1996.

[ZL21] Chicheng Zhang and Yinan Li. Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise. In Conference on Learning Theory , pages 4526-4527. PMLR, 2021.

[ZLC17] Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 30 th Conference on Learning Theory, COLT 2017 , pages 1980-2022, 2017.

[ZWDD24] Nikos Zarifis, Puqian Wang, Ilias Diakonikolas, and Jelena Diakonikolas. Robustly learning single-index models via alignment sharpness. CoRR , abs/2402.17756, 2024.

A Massart GLMs

In this section, we prove our result regarding learning in the Massart generalized linear model with a margin. Our analysis is similar to that of Section 3 but requires a modified version of Lemma 2, which gives a 'bounded" separating hyperplane in the case of Massart GLM (Lemma 6). We first state the definition of our model again (slightly generalized to relax the assumption that σ is odd).

Definition 3 (Massart GLM) . Let σ : [ -1 , 1] → [ -1 , 1] be a non-decreasing function. We say that a distribution D on B d ×{± 1 } is an instance of the σ -Massart generalized linear model (GLM) with constant shift τ and margin γ with respect to w ⋆ if the following conditions hold.

∣ ∣ | σ ( t ) | - | σ ( -t ) | ∣ ∣ ≤ τ for all t ∈ [0 , 1] .

There exists w ⋆ ∈ R d such that ∥ w ⋆ ∥ = 1 and Pr [ | w ⋆ · x | < γ ] = 0 .

For all x ∈ supp ( D x ) , there is an η ( x ) ∈ [0 , 1 -| σ ( w ⋆ · x ) | 2 ] such that

̸

We state and prove the following lemma which provides a separating hyperplane in this setting.

̸

Lemma6 (Separating Hyperplane) . Let D be a distribution on R d ×{± 1 } such satisfying Definition 3. Let w ∈ R d be such that Pr ( x ,y ) ∼ D [sign( w · x ) = y ] ≥ E x ∼ D x [ 1 -| σ ( w ∗ · x ) | + τ 2 ] + ϵ . Then, we have

̸

Proof. Let A = { x ∈ R d | sign( w · x ) = sign( w ∗ · x ) } and B = { x ∈ R d | sign( w · x = sign( w ∗ · x ) } . We have that

Define g ( x ) = ( σ ( w · x ) -β ( x )sign( w ∗ · x )) · ( w · x -w ∗ · x ) | w · x | + αγ . We analyze g ( x ) separately for x ∈ A and x ∈ B .

First consider points x in A . For any x ∈ A , we have that

The second inequality follows from the fact that ( σ ( w · x ) -σ ( w ∗ · x )) · ( w · x -w ∗ · x ) ≥ 0 since σ is monotonically increasing. The third inequality holds because sign( w · x ) = sign( w ∗ · x ) . The final inequality is true because | w · x |-| w ∗ · x | | w · x | + αγ ≤ 1 and β ( x ) ≥ | σ ( w ∗ · x ) | .

̸

We now consider the case where x ∈ B . Since sign( w · x ) = sign( w ∗ · x ) , we have that g ( x ) = ( | σ ( w · x ) | + β ( x ) ) · | w · x | + | w ∗ · x | | w · x | + αγ . We split B into two finer regions. Define B 1 = { x ∈ B | | w · x | ≥ | w ∗ · x |} and B 2 = { x ∈ B | | w · x | < | w ∗ · x |} . First, consider x ∈ B 1 . We have that | σ ( w · x ) | ≥ max(0 , | σ ( -w · x ) | -τ ) . We also have that | σ ( -w · x | ) ≥ | σ ( w ∗ · x ) | since | w · x | ≥ | w ∗ · x | and σ is monotone non-decreasing. Also, observe that | w · x | + | w ∗ · x | | w · x | + αγ ≥ 1 since | w ∗ · x | ≥ γ . Thus, we obtain that g ( x ) ≥ max(0 , | σ ( w ∗ · x ) | -τ ) + β ( x ) . Finally, we consider

x ∈ B 2 . Let c ( x ) = | w ∗ · x | | w · x | . We have that

The third inequality follows from the fact that | w ∗ · x | ≥ γ and the last inequality is true because 1+ c 1+ αc ≥ 2 -ϵ for any c ≥ 1 when α = ϵ 2 -ϵ . Since 1 ≥ β ( x ) ≥ | σ ( w ∗ · x ) | , we have that g ( x ) ≥ β ( x ) + | σ ( w ∗ · x ) | -ϵ .

Thus, we obtain that

We now use our assumption on the error of w . We have that

̸

We can now prove our main theorem about Massart GLMs. The algorithm we use, Algorithm 2, is a modified version of the Perspectron algorithm (Algorithm 1), where we subsitute the value of the parameter γ with γ · ϵ 2 -ϵ and the updates involve the function σ .

Theorem 4. Let D be an instance of the σ -Massart GLM with constant shift τ and margin γ with respect to w ⋆ , and let ϵ, δ ∈ (0 , 1) . Algorithm 2 with , T 1 ≥ ( 32 ϵ 4 γ 2 ) ⌈ log 2 ( 2 δ ) ⌉ , T 2 ≥ 8 ϵ 2 log( 4 | T 1 | δ

) returns w such that ℓ 0-1 ( w ) ≤ E x ∼ D x [ 1 -| σ ( w ∗ · x | + τ 2 ] + ϵ with probability ≥ 1 -δ , using

Algorithm 2: GLMPerspectron

1 Input: { x i , y i } i ∈ [ T 1 + T 2 ] ⊂ R d ×{± 1 } drawn i.i.d. from D in the σ -Massart GLM model with margin γ , parameter α ∈ (0 , 1) , step size λ > 0 , failure probability δ ∈ (0 , 1 2 )

̸

Proof. Given Lemma 6, the first step of the proof is exactly analogous to the proof of Lemma 3, i.e., we can show the following claim.

Claim 1. Let { x i , y i } i ∈ [ T ] ∼ i.i.d. D , where D satisfies Definition 3. Consider iterating, from w 1 := 0 d ,

The proof is analogous to the proof of Lemma 3, but we use γϵ/ (2 -ϵ ) in the place of γ . To amplify the success probability and finish the proof, we once more use Lemma 4.

B Omitted proofs

B.1 Learning with unknown noise rate

In this section, we can obtain the same sample complexity (upto logarithmic factors) as Theorem 3 even when the noise rate η is unknown to the learner. The argument is the following: we argue that our separating hyperplane (Lemma 2) is tolerant to O ( ϵ ) noise in the parameter η . We can then discretize the interval [0 , 1 / 2] into intervals of size ϵ and run the training algorithm multiple times for these different choices. Then, we can output the hypothesis with lowest validation error among the classifiers output by these different runs of the algorithm. We now argue that this idea indeed works.

Lemma 7. If D is an instance of the η -Massart halfspace model with margin γ with respect to w ⋆ , and w ∈ R d has ℓ 0-1 ( w ) ≥ η + ϵ ,

Proof. The proof is almost identical to the proof of Lemma 2 except for a few steps. We highlight the differences. We reuse the notation from the previous proof.

First consider x / ∈ A , we observe that using the same argument as before, we now obtain

In the case of x ∈ A , since β ( x ) ≥ β ≥ ˜ β , we obtain

Using this, we can can complete the proof by repeating the steps of the previous proof.

Having proved this, our algorithm is simple, run over the (1 / (2 ϵ )) choices of ˜ β in [0 , 1 / 2] and run the algorithm from Theorem 3 for each choice, reusing the same samples in the different run's of the algorithm. The correctness follows Lemma 7 and the proof of Theorem 3 since one of the the choices of parameters must lie in the interval ((1 -2 η ) -ϵ, (1 -2 η )] .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide proofs and/or references for all claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We thoroughly discuss the assumptions under which our results hold, as well as their scope and comparison with related work, in an explicit limitations and future work section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide full proofs for all of our formal statements.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We read the code of ethics and our work conforms with the stated code.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work is of theoretical nature, focusing on improving the reliability of a basic learning algorithm. We do not foresee any direct path to any negative applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies learning half-spaces in the Massart noise setting. That is for $\\gamma\\eta >0$ and $\\eta(x)\\leq \\eta $ for all $x\\in supp(D_{x})$,  half space $w^*$ and such that $P( | w^* x  | \\geq \\gamma)$ and $P ( sign( w^* x ) \\not= y)=\\eta(x)$ for all $x\\in supp(D_{x})$ - given such an instance and $\\varepsilon>0$ the paper now states the problem of finding a half spaces $w$ such that $P[sign(wx)\\not=y]\\leq \\eta+\\varepsilon]$ using as few samples and computations as possible. The paper achieves a new bound on the number of samples needed to solve this problem of $\\tilde{O}(\\gamma\\varepsilon)^{-2}$ (suppressing factors of  $\\log(1/\\delta)$). This is an improvement over previous results which uses $\\gamma^{-3}\\varepsilon^{-5}$, $\\gamma^{-4}\\varepsilon^{-3}$. They also show it in a more general setting for $\\sigma$ odd and non decreasing where $P[sign(w*x)\\not= y]=\\frac{1-\\sigma(wx)}{2}$ for all $x\\in supp(D_{x})$. They obtain a sample complexity bound of $(\\tilde{O})((\\gamma\\varepsilon)^{-2})$ here.\n\nSTRENGTHS:\nOriginality:\nThe paper seems to combine many ideas from different papers and in this way get a novel result and improvement over previous results. Related work is well cited. \n\nQuality: \nThe main of the paper includes a full proof of the Massart noise and a proof sketch of the general setting. The proof of the Massart noise case on a first quick read seems ok. The proof sketch is ok - I have not read the appendix with the full proof so I will not comment on the technically soundness of the proof. The authors also address the limits of their work and state interesting future work.\n\nClarity:\nThe paper is very well written and explains very well the ideas behind the proof.\n\nSignificance: \nThe improvement in sample complexity is a polynomial improvement in $(\\gamma\\varepsilon)^{-1}$ so I would say significant.\n\nWEAKNESSES:\nWeaknesses:\nNot anything significant - related to the question below do the other papers also use $\\eta$ in the error bound instead of $E[\\eta(x)]\\leq \\eta$? If yes why not use $E[\\eta(x)]$?\n\nQUESTIONS:\n\\section*{Questions}\n\nLine 18-20: I really like how it was $Pr_{x\\sim D_{x}}$ was used as it made the randomness explicit - I didn't at first get that the randomness in the $Pr$ (line 20) was over $y\\sim D_{y}(x)$ I don't know if other readers would have the same experience.\n\nLine 72 : what error does previous work in massart noise setting use? If other- why not use the same?\n \n\nLine 102 $\\eta(x)$?\n\nNot consistent whether it is $Pr_{\\cdot}$ or just $Pr$\n\nTable: Good table. Is the running time not interesting?\n\nThe real numbers $R$ and natural numbers $N$ didn't look like I'm used to seeing them - I don't know if this is on purpose if it is don't mind changing it - I just wanted to let you know in case that something has complied weirdly.\n\nLine 221: should it be $B^d\\times \\{-1,1\\}$?\n\nLine 229: why isn't it a vector? Can you give the derivation of this expression?\n\nLine 238: $h_{w'}=sign((w..$ shouldnt it be $h_{w'}=sign((w'..$ \n\nLine 261: what does the $l_\\eta(w*,x,h)$ mean? Happens more places in this proof.\n\nLine 263: Please show fact (1), or should it be $l_\\eta(-yw*x)$. If facts 1) and 2) follow from the fact stated at the beginning of the proof from [DGT19b] maybe consider moving these together.\n\nLine 276: it says $w^{\\star}$ instead of $w^*$\n\nLine 298-299: The third line in the equation missing $x^t$\n\nLine 289 algorithm line 7: this makes $w^t$ and $x^t$ independent right? Which is used in the proof of lemma 3  last inequality of the large equation combined with lemma 2?\n\nLine 307: why say anything about contradiction - doesn't it just follow from the inequality afterward?\n\nLine 313-316: I guess It doesn't matter that the last badge isn't necessarily of size $T$? Or is it necessarily of size $T$?\n\nLine 329: what is $B$?\n\nWould it be interesting to make the computational cost independent of d with the cost of polynomial factors in $1/\\gamma\\varepsilon$?\n\nLIMITATIONS:\nThey address the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

comments: SUMMARY:
The paper considers the problem of PAC-learning $\gamma$-margin halfspaces under $\eta$-Massart noise. The paper provides an efficient algorithm achieving error $\eta+\epsilon$ with sample complexity $\tilde{O}(1/(\epsilon^2\gamma^2))$. The individual dependence  of the sample complexity on $\epsilon$ and $\gamma$ appears to be optimal for efficient algorithms in the light of lower bounds provided in previous work.

The algorithm is a form of stochastic-gradient descent where the loss function is the LeakyReLu loss. The gradient is carefully weighted with an appropriate weight in order to yield the desired results.

The paper also provides an algorithm for a more general model, the generalized linear model (GLM), under Massart-noise.

The paper is generally very-well written.

STRENGTHS:
The problem of learning half-spaces under Massart noise is a fundamental problem that has received a lot of attention recently. This paper provides a simple algorithm with a low sample-complexity that is probably optimal.

The paper also provides generalizations to the GLM models.

WEAKNESSES:
I haven't found any significant weaknesses.

Typos:
- Page 3, line 97: “observe that that if” ->  “observe that if”

QUESTIONS:
- In Lemma 3, the parameters $\lambda$ and $T$ were chosen so that $\mathbb{P}[\mathcal{E}_T]\leq 1/2$, which necessitated the outer loop $j\in[N]=[\log_2(2/\delta)]$ to get success probability at least $1-\delta$. Wouldn't it be possible to choose the constant factors in $\lambda$ and $T$ differently to directly get  $\mathbb{P}[\mathcal{E}_T]\leq \delta$?

LIMITATIONS:
Since this is a theoretical paper, I do not see any potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the problem of learning halfspaces and generalized linear model in the Massart model under a margin assumption. In particular, for the case of halfspaces, the submission gives an efficient algorithm achieving (conjecturally optimal) error $\\eta + \\varepsilon$ using only $\\tilde{O}(\\gamma^{-2} \\varepsilon^{-2})$ samples, where $\\gamma$ is the margin parameter, and $\\eta$ the upper bound on the noise rate. This matches what is known in the more benign RCN model. All previously known efficient algorithms in the Massart model require a number of samples that is polynomially worse in $\\varepsilon$ and $\\gamma$.\n\nSTRENGTHS:\nThe Massart model is a semi-random noise model motivated by the question whether known (efficient) algorithmic approaches for the fully random model, in this case random classification noise (RCN), are overfitting to the specific aspects of the model. The work of [DGT19] designed the first efficient algorithm to non-trivially learn halfspaces in this harsher noise model (without margin assumption). The current work shows that under a margin assumption, this is possible with as few samples as known efficient algorithms for the more benign RCN model needs. All prior works were loose by polynomial factors in $\\gamma$ and $\\varepsilon$.\n\nOn a technical level the submission combines previous algorithmic approaches with a regret-minimization scheme leading to an elegant analysis and ultimately a better sample complexity.\n\n\n\n\n[DGT19]: I. Diakonikolas, T. Gouleakis, and C. Tzamos. Distribution-independent PAC learning of halfspaces with massart noise\n\nWEAKNESSES:\nThe presentation of the submission is generally solid but can be improved in several aspects. Below are comments aiming at this. Especially the comments related to the technical overview are important in order to understand why the submission is able to improve over previous work. I'm willing to increase my score to 6 or 7 if the authors agree to address the comments below and in particular, explain in their rebuttal how they would address the comments for the technical overview.\n\n\n## Technical Overview\n\nI like the structure of the technical overview and that it tries to highlight differences with previous works (this also shows expertise of the authors). Unfortunately however, in several places the writing is unclear (see below). I find it nice that the proof for the halfspace result is so short that it fits in the main body. Nevertheless, I would find a more extensive technical overview with commentary much more illustrative then including the full proof (the many formulas make this hard to read and understand what is going on).\n\nHere are some specific comments about the technical overview:\n\n- lines 129-136: It is not clear from the discussion why this approach incurs a too high sample complexity on a quantitative level.\n- lines 153-162: It is not at all clear from the discussion why the proposed update rule improves the dependence on $\\gamma$\n- lines 172-174: You say that this approach also works for the case of massart halfsapces. Why did you not follow this approach? From the discussion it seems it would significantly simplify a part of the analysis. Is there anything else that breaks?\n\n\n## Introduction\n\n- lines 46-58: Before diving into the details of the fine-grained aspects and to set the stage, it would be helpful to briefly explicitly recall what is known in the general Massart model (without margin assumption) -- e.g., error $\\eta$ is possible and this is likely optimal\n- when you say \"learn up to error $\\varepsilon$\" in the above paragraph, do you mean error $\\eta + \\varepsilon$?\n- I believe the work [DKMR22] is not mentioned at all, this should be added\n- Similarly, when talking about impossibility results in the agnostic model, the work [Tie23] (see also [DKMR22]) should be mentioned\n- for some of the citations, the arxiv version is cited. Why not cite the conference version?\n\n\n[DKMR22]: I. Diakonikolas, D. M. Kane, P. Manurangsi, and L. Ren, Cryptographic Hardness of Learning Halfspaces with Massart Noise\n\n[Tie23]: S. Tiegel, Hardness of Agnostically Learning Halfspaces from Worst-Case Lattice Problems\n\nQUESTIONS:\nSee last part of above section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focused on the fine-grained analysis of learning $\\gamma$-margin halfspace with massart noise. The authors designed a new certificate vector $g$ by dividing the gradient vector of leaky ReLU by $|w^\\top x| + \\gamma$, and showed that when the hyperplane $h_w(x)$ has large 0-1 loss $\\ell_{0,1}(w)\\geq \\eta + \\epsilon$, the certificate vector aligns well with the direction of $w - w^*$, i.e., $g^\\top(w. - w^*)\\geq \\epsilon$, hence a perceptron-like algorithm enjoys fast convergence to the optimal hyperplane $h_{w^*}(x)$. In the end, the authors showed that the proposed Perspectron algorithm requires only $O((\\gamma\\epsilon)^{-2})$ samples, matching with the sample complexity of learning margin halfspaces with RCN. The authors also extended the technique to massart GLM problem and achieved a similar $O((\\gamma\\epsilon)^{-2})$ sample complexity.\n\nSTRENGTHS:\nI enjoyed reading this paper due to its clarity and fluency in presentation. The authors did a good job explaining the intuitions and ideas. The result of this paper is very interesting to me, as I have always thought learning halfspaces with massart noises is much harder than with RCN, and this paper provided a surprising result that shows learning with massart noise can be achieved with similar sample complexity. The method (new certificate vector) that the authors proposed is also interesting, and can be informative for future research.\n\nWEAKNESSES:\nNo serious weakness of this paper, but there are several typos here and there. \nFor example line 263 should be $w^*$, there should be an $x$ in the end of line 270, etc.\n\nQUESTIONS:\nI am not very familiar with massart GLMs. If I understand correctly, it seems the goal of massart GLM is still trying to find a classification hyperplane, but under this massart GLM model, the massart noise $\\eta(x)$ is bounded by a function of $w^*\\cdot x$. If so, isn't massart GLM a sub-class of massart model, as we have further constraints on $\\eta(x)$ (in addition to simply restricting $\\eta(x)\\leq \\eta$), hence perhaps implying that massart GLM is simpler than massart model?\n\nLIMITATIONS:\nThe authors have addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper considers the problem of learning halfspaces with a margin, under the Massart noise model. The Massart noise model generalizes the Random Classification Noise (RCN) Model: while in the RCN model, each label is flipped with fixed probability \eta, in the Massart noise model the the probability of flipping the label can be a function of the covariates bounded above by \eta.  The paper asks the question of whether one can design learning algorithms under the Massart noise model, matching the sample complexity under the RCN model, and answers it positively. Specifically, the paper proposes a proper learning algorithm, with sample complexity 1/(\epsilon^2 \gamma^2), matching the state of the art under the RCN model  (\epsilon is the error of the algorithm, \gamma is the margin). The results are also extended to the case of generalized linear models.

STRENGTHS:
- The paper makes a concrete improvement to the sample complexity of learning halfspaces under Massart Noise, a classic problem in learning theory.

- The proposed algorithm is natural and simple, and the main ideas of the paper are explained well.

WEAKNESSES:
The set of people interested in the fine-grained complexity of learning under Massart Noise might not be very broad. So the significance of the results seems moderate.

QUESTIONS:
NA

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15