Title: Efficient Testable Learning of Halfspaces with Adversarial Label Noise
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine Learning
Abstract: We give the first polynomial-time algorithm for the testable learning 
of halfspaces in the presence of adversarial label noise under the Gaussian distribution. In the recently introduced testable learning 
model, one is required to produce a tester-learner such that if the data passes the tester, then one can trust the output of the robust learner on the data. Our tester-learner runs in time $\text{poly}(d/\epsilon)$ and outputs a halfspace with misclassification error $O(\text{opt})+\epsilon$, where $\text{opt}$ is the 0-1 error of the best fitting halfspace. At a technical level, our algorithm employs an iterative soft localization technique enhanced with appropriate testers to ensure that the data distribution is sufficiently similar to a Gaussian. Finally, our algorithm can be readily adapted to yield an efficient and testable active learner requiring only $d ~ \text{polylog}(1/\epsilon)$ labeled examples.

Efficient Testable Learning of Halfspaces with Adversarial Label Noise

Abstract

We give the first polynomial-time algorithm for the testable learning of halfspaces in the presence of adversarial label noise under the Gaussian distribution. In the recently introduced testable learning model, one is required to produce a testerlearner such that if the data passes the tester, then one can trust the output of the robust learner on the data. Our tester-learner runs in time poly( d/ϵ ) and outputs a halfspace with misclassification error O (opt) + ϵ , where opt is the 0-1 error of the best fitting halfspace. At a technical level, our algorithm employs an iterative soft localization technique enhanced with appropriate testers to ensure that the data distribution is sufficiently similar to a Gaussian. Finally, our algorithm can be readily adapted to yield an efficient and testable active learner requiring only d polylog(1 /ϵ ) labeled examples.

1 Introduction

A(homogeneous) halfspace is a Boolean function h : R d →{± 1 } of the form h w ( x ) = sign ( w · x ) , where w ∈ R d is the corresponding weight vector and the function sign : R →{± 1 } is defined as sign( t ) = 1 if t ≥ 0 and sign( t ) = -1 otherwise. Learning halfspaces from random labeled examples is a classical task in machine learning, with history going back to the Perceptron algorithm [Ros58]. In the realizable PAC model [Val84] (i.e., with consistent labels), the class of halfspaces is known to be efficiently learnable without distributional assumptions. On the other hand, in the agnostic (or adversarial label noise) model [Hau92, KSS94] even weak learning is computationally intractable in the distribution-free setting [Dan16, DKMR22, Tie22].

These intractability results have served as a motivation for the study of agnostic learning in the distribution-specific setting, i.e., when the marginal distribution on examples is assumed to be wellbehaved. In this context, a number of algorithmic results are known. The L 1 -regression algorithm of [KKMS08] agnostically learns halfspaces within near-optimal 0-1 error of opt + ϵ , where opt is the 0-1 error of the best-fitting halfspace. The running time of the L 1 -regression algorithm is d ˜ O (1 /ϵ 2 ) under the assumption that the marginal distribution on examples is the standard Gaussian (and for a few other structured distributions) [DGJ + 10, DKN10]. While the L 1 regression method leads to improper learners, a proper agnostic learner with qualitatively similar complexity was recently given

in [DKK + 21]. The exponential dependence on 1 /ϵ in the running time of these algorithms is known to be inherent, in both the Statistical Query model [DKZ20, GGK20, DKPZ21] and under standard cryptographic assumptions [DKR23].

Interestingly, it is possible to circumvent the super-polynomial dependence on 1 /ϵ by relaxing the final error guarantee - namely, by obtaining a hypothesis with 0-1 error f (opt)+ ϵ , for some function f ( t ) that goes to 0 when t → 0 . (Vanilla agnostic learning corresponds to the case that f ( t ) = t .) A number of algorithmic works, starting with [KLS09], developed efficient algorithms with relaxed error guarantees; see, e.g., [ABL17, Dan15, DKS18, DKTZ20b]. The most relevant results in the context of the current paper are the works [ABL17, DKS18] which gave poly( d/ϵ ) time algorithms with error C opt + ϵ , for some universal constant C > 1 , for learning halfspaces with adversarial label noise under the Gaussian distribution. Given the aforementioned computational hardness results, these constant-factor approximations are best possible within the class of polynomial-time algorithms.

Adrawback of distribution-specific agnostic learning is that it provides no guarantees if the assumption on the marginal distribution on examples is not satisfied. Ideally, one would additionally like an efficient method to test these distributional assumptions, so that: (1) if our tester accepts, then we can trust the output of the learner, and (2) it is unlikely that the tester rejects if the data satisfies the distributional assumptions. This state-of-affairs motivated the definition of a new model - introduced in [RV23] and termed testable learning -formally defined below:

Definition 1.1 (Testable Learning with Adversarial Label Noise [RV23]) . Fix ϵ, τ ∈ (0 , 1] and let f : [0 , 1] ↦→ R + . A tester-learner A (approximately) testably learns a concept class C with respect to the distribution D x on R d with N samples, and failure probability τ if the following holds. For any distribution D on R d ×{± 1 } , the tester-learner A draws a set S of N i.i.d. samples from D . In the end, it either rejects S or accepts S and produces a hypothesis h : R d ↦→{± 1 } . Moreover, the following conditions must be met:

(Completeness) If D truly has marginal D x , A accepts with probability at least 1 -τ .

̸

(Soundness) The probability that A accepts and outputs a hypothesis h for which Pr ( x ,y ) ∼ D [ h ( x ) = y ] > f (opt) + ϵ , where opt := min g ∈C Pr ( x ,y ) ∼ D [ g ( x ) = y ] is at most τ .

̸

The probability in the above statements is over the randomness of the sample S and the internal randomness of the tester-learner A .

The initial work [RV23] and the followup paper [GKK22] focused on the setting where f ( t ) = t (i.e., achieving optimal error of opt + ϵ ). These works developed general moment-matching based algorithms that yield testable learners for a range of concept classes, including halfspaces. For the class of halfspaces in particular, they gave a testable agnostic learner under the Gaussian distribution with sample complexity and runtime d ˜ O (1 /ϵ 2 ) -essentially matching the complexity of the problem in the standard agnostic PAC setting (without the testable requirement). Since the testable learning setting is at least as hard as the standard PAC setting, the aforementioned hardness results imply that the exponential complexity dependence in 1 /ϵ cannot be improved.

In this work, we continue this line of investigation. We ask whether we can obtain fully polynomial time testable learning algorithms with relaxed error guarantees - ideally matching the standard (non-testable) learning setting. Concretely, we study the following question:

Is there a poly( d/ϵ ) time tester-learner for halfspaces with error f (opt) + ϵ ? Specifically, is there a constant-factor approximation?

As our main result, we provide an affirmative answer to this question in the strongest possible sense -by providing an efficient constant-factor approximate tester-learner.

Labeling examples often requires hiring expert annotators, paying for query access to powerful language models, etc. On the other hand unlabeled examples are usually easy to obtain in practice. This has motivated the design of active learning algorithms that, given a large dataset of unlabeled examples, carefully choose a small subset to ask for their labels. A long line of research has studied active learning of halfspaces under structured distributions either with clean labels or in the presence of noise [BBL06, BBZ07, BL13, ABL17, She21].

In principle, testable learners could only rely on unlabeled examples during their 'testing phase' in order to verify that the x -marginal satisfies the required properties; and, assuming that it does so,

proceed to run an active learning algorithm. However, as the testing process is often coupled with the downstream learning task - that depends on labeled examples - it is an interesting question to investigate the design of testable learners in the active learning setting.

Definition 1.2 (Testable Active Learning) . A tester-learner has sample complexity N and label complexity q in the active learning model if it draws N unlabeled samples from D x , i.e., the x -marginal of D over R d , and queries the labels of at most q samples throughout its computation.

Main Result Our main result is the first polynomial-time tester-learner for homogeneous halfspaces with respect to the Gaussian distribution in the presence of adversarial label noise. Formally, we establish the following theorem:

Theorem 1.3 (Testable Active Learning Halfspaces under Gaussian Marginals) . Let ϵ, τ ∈ (0 , 1) and C be the class of homogeneous halfspaces on R d . There exists a tester-learner with sample complexity N = poly( d, 1 /ϵ ) log(1 /τ ) and runtime poly( d N ) for C with respect to N ( 0 , I ) up to 0-1 error O (opt) + ϵ , where opt is the 0-1 error of the best fitting function in C , that fails with probability at most τ . In addition, in the active learning model, the algorithm has sample complexity N and label complexity q = ˜ O (( d log(1 /ϵ ) + log 2 (1 /ϵ )) log(1 /τ )) .

Before we provide an overview of our technical approach, some remarks are in order. Theorem 1.3 gives the first algorithm for testable learning of halfspaces that runs in poly( d/ϵ ) time and achieves dimension-independent error (i.e., error of the form f (opt) + ϵ , where f satisfies lim t → 0 f ( t ) = 0 .) Moreover, the constant-factor approximation achieved is best possible, matching the known guarantees without the testable requirement and complexity lower bounds. Prior to our work, the only known result in the testable setting, due to [RV23, GKK22], achieves error opt + ϵ with complexity d poly(1 /ϵ ) . A novel (and seemingly necessary) feature of our approach is that the testing components of our algorithm depend on the labels (as opposed to the label-oblivious testers of [RV23, GKK22]). As will be explained in the proceeding discussion, to prove Theorem 1.3 we develop a testable version of the well-known localization technique that may be of broader interest.

Independent Work In concurrent and independent work, [GKSV23] gave an efficient testerlearner for homogeneous halfspaces under the Gaussian distribution (and strongly log-concave distributions) achieving dimension-independent error guarantees. Specifically, for strongly logconcave distributions, their algorithm achieves error O ( k 1 / 2 opt 1 -1 /k ) with sample complexity and running time of poly( d ˜ O ( k ) , (1 /ϵ ) ˜ O ( k ) ) . That is, they obtain error O (opt c ) , where c < 1 is a universal constant, in poly c ( d/ϵ ) time; and error ˜ O (opt) in quasi-polynomial ( d/ϵ ) polylog( d ) time. For the Gaussian distribution, they achieve error O (opt) with complexity poly( d, 1 /ϵ ) . We remark that since the learner in [GKSV23] corresponds to the stochastic gradient descent algorithm from [DKTZ20b], which requires labeled sample in every iteration, it is not clear whether their approach can be made label-efficient.

1.1 Overview of Techniques

Our tester-learner is based on the well-known localization technique that has been used in the context of learning halfspaces with noise; see, e.g., [ABL17, DKS18]. At a high-level, the idea of localization hinges on updating a given hypothesis by using 'the most informative' examples, specifically examples that have very small margin with respect to the current hypothesis. Naturally, the correctness of this geometric technique leverages structural properties of the underlying distribution over examples, namely concentration, anti-concentration, and anti-anti-concentration properties (see, e.g., [DKTZ20a]). While the Gaussian distribution satisfies these properties, they are unfortunately hard to test. In this work, we show that localization can be effectively combined with appropriate efficient testing routines to provide an efficient tester-learner.

̸

Localization and a (Weak) Proper Testable Learner Assume that we are given a halfspace defined by the unit vector w with small 0-1 error, namely Pr x ∼ D x [sign( v ∗ · x ) = sign( w · x )] ≤ δ , for some small δ > 0 , where v ∗ is the unit vector defining an optimal halfspace. The localization approach improves the current hypothesis, defined by w , by considering the conditional distribution D ′ on the points that fall in a thin slice around w , i.e., the set of points x satisfying | w · x | ≤ O ( δ ) . The goal is to compute a new (unit) weight vector w ′ that is close to an optimal halfspace, defined by v ∗ , with

Θ(

i

)

v

∗

u

δ

δ

δ

w

θ

= Θ(

δ

)

̸

respect to D ′ , i.e., Pr x ′ ∼ D ′ x [sign( v ∗ · x ′ ) = sign( w ′ · x ′ )] ≤ α , for an appropriate α > 0 . We can then show that the halfspace defined by w ′ will be closer to the target halfspace (defined by v ∗ ) with respect to the original distribution, i.e., we have that Pr x ∼ D x [sign( v ∗ · x ) = sign( w ′ · x )] ≤ O ( δα ) . By repeating the above step, we iteratively reduce the disagreement with v ∗ until we reach our target error of O (opt) . Similarly to [DKS18], instead of 'hard' conditioning on a thin slice, we perform a 'soft' localization step where (by rejection sampling) we transform the x -marginal to a Gaussian whose covariance is O ( δ 2 ) in the direction of w and identity in the orthogonal directions, i.e., Σ = I -(1 -δ 2 ) ww ⊤ ; see Fact 3.2.

̸

A crucial ingredient of our approach is a proper testable, weak agnostic learner with respect to the Gaussian distribution. More precisely, our tester-learner runs in polynomial time and either reports that the x -marginal is not N ( 0 , I ) or outputs a unit vector w with small constant distance to the target v ∗ , i.e., ∥ w -v ∗ ∥ 2 ≤ 1 / 100 ; see Proposition 2.1. Our weak proper tester-learner first verifies that the given x -marginal approximately matches constantly many low-degree moments with the standard Gaussian; and if it does, it returns the vector defined by the degree1 Chow parameters, i.e., c = E ( x ,y ) ∼ D [ y x ] . Our main structural result in this context shows that if D x approximately matches its low-degree moments with the standard Gaussian, then the Chow parameters of any homogeneous LTF with respect to D x are close to its Chow parameters with respect to N ( 0 , I ) , i.e., for any homogeneous LTF f ( x ) , we have that E x ∼ D x [ f ( x ) x ] ≈ E x ∗ ∼N ( 0 , I ) [ f ( x ∗ ) x ∗ ] ; see Lemma 2.3. Since the Chow vector of a homogeneous LTF with respect to the Gaussian distribution is parallel to its normal vector v ∗ (see Fact 2.2), it is not hard to show that the Chow vector of the LTF with respect to D x will not be very far from v ∗ and will satisfy the (weak) learning guarantee of ∥ c -v ∗ ∥ 2 ≤ 1 / 100 . Finally, to deal with label noise, we show that if x ′ has bounded second moments (a condition that we can efficiently test), we can robustly estimate E x ∼ D x [ f ( x ) x ] with samples from D up to error O ( √ opt) (see Lemma 2.4), which suffices for our purpose of weak learning. The detailed description of our weak, proper tester-learner can be found in Section 2.

̸

From Parameter Distance to Zero-One Error Having a (weak) testable proper learner, we can now use it on the localized (conditional) distribution D ′ and obtain a vector w ′ that is closer to v ∗ in ℓ 2 distance; see Lemma 3.3. However, our goal is to obtain a vector that has small zero-one disagreement with the target halfspace v ∗ . Assuming that the underlying x -marginal is a standard normal distribution, and that ∥ w -v ∗ ∥ 2 = δ , it holds that Pr x ∼ D x [sign( w · x ) = sign( v ∗ · x )] = O ( δ ) , which implies that achieving ℓ 2 -distance O (opt) + ϵ suffices. We give an algorithm that can efficiently either certify that small ℓ 2 -distance implies small zero-one disagreement with respect to the given marginal D x or declare that D x is not the standard normal.

The disagreement region of v ∗ and w is a union of two 'wedges' (intersection of two halfspaces); see Figure 1. In order for our algorithm to work, we need to verify that these wedges do not contain too much probability mass. Similarly to our approach for the weak tester-leaner, one could try a moment-matching approach and argue that if D x matches its 'low'-degree moments with N ( 0 , I ) , then small ℓ 2 -distance translates to small zero-one disagreement. However, we will need to use this result for vectors that are very close to the target (but still not close enough), namely ∥ w -v ∗ ∥ 2 = δ ,

where δ = Θ( ϵ ) ; this would require matching poly(1 /δ ) many moments (as we essentially need to approximate the wedge of Figure 1 with a polynomial) and would thus lead to an exponential runtime of d poly(1 /δ ) .

Instead of trying to approximate the disagreement region with a polynomial, we will make use of the fact that our algorithm knows w (but not v ∗ ) and approximate the disagreement region by a union of cylindrical slabs. We consider slabs of the form S i = { x : iδ ≤ | w · x | ≤ ( i + 1) δ } . If the target distribution is Gaussian, we know that the set | w · x | ≫ √ log(1 /ϵ ) has mass O ( δ ) and we can essentially ignore it. Therefore, we can cover the whole space by considering roughly M = O ( √ log(1 /δ ) /δ ) slabs of width δ and split the disagreement region into the disagreement region inside each slab S i . We have that

̸

where u is the unit direction parallel to the projection of the target v ∗ onto the orthogonal complement of w , see Figure 1. By the anti-concentration of the Gaussian distribution we know that each slab should have mass at most O ( δ ) . Note that this is easy to test by sampling and computing empirical estimates of the mass of each slab. Moreover, assuming that underlying distribution is N ( 0 , I ) , we have that, conditional on S i the orthogonal direction u · x ∼ N (0 , 1) (see Figure 1) and in particular u · x has bounded second moment. We do not know the orthogonal direction u as it depends on the unknown v ∗ but we can check that, conditional on the slab S i , the projection of D x onto the orthogonal complement of w is (approximately) mean-zero and has bounded covariance (i.e., bounded above by 2 I ). Note that both these conditions hold when x ∼ N ( 0 , I ) and can be efficiently tested with samples in time poly( d, 1 /δ ) . Under those conditions we have that that Pr [ S i ] = O ( δ ) for all i . Moreover, when the conditional distribution on S i (projected on the orthogonal complement of w ) has bounded second moment, we have that Pr [ | u · x | ≥ i | x ∈ S i ] ≤ O (1 /i 2 ) . Combining the above, we obtain that under those assumptions the total probability of disagreement is at most O ( δ ) . The detailed analysis is given in Section 3.1.

Obtaining an Active Learner. Our localization-based algorithm can be readily adapted to yield an efficient active learner-tester. We first observe that the tester that verifies that the parameter distance is proportional to 0/1 error does not require any labels, as it relies on verifying moments of the x -marginal of the underlying distribution. During each iteration of localization, since the learner is only trying to find a constant approximation of the defining vector of the unknown halfspace, the learner only needs roughly O ( d ) samples that fall in the slice we localize to. Since we can selectively query only the labels of samples that fall in the right region and the total number of iterations is at most O (log(1 /ϵ )) , the labels needed to learn the defining vectors is roughly O ( d log(1 /ϵ )) . In the end, the learner needs to find the best halfspace among the ones from O (log(1 /ϵ )) localization iterations. We remark this can also be done with polylog(1 /ϵ ) many label queries (see Lemma 3.6).

1.2 Preliminaries

We use small boldface characters for vectors and capital bold characters for matrices. We use [ d ] to denote the set { 1 , 2 , . . . , d } . For a vector x ∈ R d and i ∈ [ d ] , x i denotes the i -th coordinate of x , and ∥ x ∥ 2 := √ ∑ d i =1 x 2 i the ℓ 2 norm of x . We use x · y := ∑ n i =1 x i y i as the inner product between them. We use { E } to denote the indicator function of some event E .

✶ We use E x ∼ D [ x ] for the expectation of the random variable x according to the distribution D and Pr [ E ] for the probability of event E . For simplicity of notation, we may omit the distribution when it is clear from the context. For µ ∈ R d , Σ ∈ R d × d , we denote by N ( µ, Σ ) the d -dimensional Gaussian distribution with mean µ and covariance Σ . For ( x , y ) ∈ X distributed according to D , we denote D x to be the marginal distribution of x . Let f : R d ↦→{± 1 } be a boolean function and D a distribution over R d . The degree1 Chow parameter vector of f with respect to D is defined as E x ∼ D [ f ( x ) x ] . For a halfspace h ( x ) = sign( v · x ) , we say that v is the defining vector of h .

Moment-Matching In what follows, we use the phrase 'A distribution D on R d matches k moments with a distribution Q up to error ∆ ' . Similarly to [GKK22], we formally define approximate momentmatching as follows.

Definition 1.4 (Approximate Moment-Matching) . Let k ∈ N be a degree parameter and let M ( k, d ) be the set of d -variate monomials of degree up to k . Moreover, let ∆ ∈ R |M ( k,d ) | + be a slack parameter (indexed by the monomials of M ( k, d ) ), satisfying ∆ 0 = 0 . We say that two distributions D,Q match k moments up to error ∆ if | E x ∼ D [ m ( x )] -E x ∼ Q [ m ( x )] | ≤ ∆ m for every monomial m ( x ) ∈ M ( k, d ) . When the error bound ∆ is the same for all monomials we overload notation and simply use ∆ instead of the parameter ∆ .

2 Weak Testable Proper Agnostic Learning

As our starting point, we give an algorithm that performs testable proper learning of homogeneous halfspaces in the presence of adversarial label noise with respect to the Gaussian distribution. The main result of this section is given below and its proof can be found in Appendix A.

̸

Proposition 2.1 (Proper Testable Learner with Adversarial Label Noise) . Let D be a distribution on labeled examples ( x , y ) ∈ R d ×{± 1 } with x -marginal D x . Suppose that there exists a unit vector v ∗ ∈ R d such that Pr ( x ,y ) ∼ D [sign( v ∗ · x ) = y ] ≤ opt . There exists an algorithm (Algorithm 2) that given τ, η ∈ (0 , 1) , N = d ˜ O (1 /η 2 ) log(1 /τ ) i.i.d. unlabeled samples from D x , queries the labels of O ( d/η 2 ) log( d/τ ) of them, and runs in time poly( d, N ) and does one of the following:

The algorithm reports that the x -marginal of D is not N ( 0 , I ) .

The algorithm outputs a unit vector w ∈ R d .

With probability at least 1 -τ the following holds: (1) if the algorithm reports anything, the report is correct, and (2) if the algorithm returns a vector w , it holds ∥ v ∗ -w ∥ 2 ≤ C A √ opt + η , where C A > 0 is an absolute constant.

A couple of remarks are in order. First, notice that if the algorithm outputs a vector w , we only have the guarantee that ∥ v ∗ -w ∥ 2 is small - instead of that the hypothesis halfspace h w ( x ) = sign( w · x ) achieves small 0-1 error. Nonetheless, as we will show in the next section, conditioned on D passing some test, the error of the halfspace h w will be at most opt plus a constant multiple of ∥ v ∗ -w ∥ 2 (see Lemma 3.1). Second, unlike the testable improper learners in [RV23, GKK22] - which achieve error of opt + η with similar running time and sample complexity - our testable proper learner achieves the weaker error guarantee of O ( √ opt + η ) . This suffices for our purposes for the following reason: in the context of our localization-based approach, we only need an efficient proper weak learner that achieves sufficiently small constant error. This holds for our proper testable learner, as long as both opt and η are bounded above by some other sufficiently small constant.

To obtain a proper learner, we proceed to directly estimate the defining vector v ∗ of the target halfspace h ∗ ( x ) = sign( v ∗ · x ) , where we assume without loss of generality that v ∗ is a unit vector. The following simple fact relating the degree1 Chow-parameters of a homogeneous halfspace and its defining vector will be useful for us.

Fact 2.2 (see, e.g., Lemma 4.3 of [DKS18]) . Let v be a unit vector and h ( x ) = sign( v · x ) be the corresponding halfspace. If x is from N ( 0 , I ) , then we have that E x ∼N ( 0 , I ) [ h ( x ) x ] = √ 2 /π v .

To apply Fact 2.2 in our context, we need to overcome two hurdles: (i) the x marginal of D is not necessarily the standard Gaussian, and (ii) the labels are not always consistent with h ∗ ( x ) . The second issue can be circumvented by following the approach of [DKS18]. In particular, if the x marginal of D is indeed Gaussian, we can just treat D as a corrupted version of ( x , h ∗ ( x )) , where x ∼ N ( 0 , I ) and estimate the Chow parameters robustly.

To deal with the first issue, we borrow tools from [GKK22]. At a high level, we certify that the low-degree moments of D x -the x marginal of D -approximately match the corresponding moments of N ( 0 , I ) before estimating the Chow parameters. Importantly, this testing procedure uses only unlabeled samples. To establish the correctness of our algorithm, we show that, for any distribution B that passes the moment test, the Chow parameters of a halfspace under B will still be close to its defining vector. We state the lemma formally below. Its proof can be found in Appendix A.

Lemma 2.3 (From Moment-Matching to Chow Distance) . Fix η > 0 . Let k = C log(1 /η ) /η 2 and ∆ = 1 kd k ( 1 C √ k ) k +1 , where C > 0 is a sufficiently large absolute constant. Let B be a

distribution whose moments up to degree k match with those of N ( 0 , I ) up to additive error ∆ . Let h ( x ) = sign( v · x ) be a halfspace. Then we have that ∥ ∥ ∥ E x ∼ B [ h ( x ) x ] -√ 2 π v ∥ ∥ ∥ 2 ≤ O ( √ η ) .

With Lemma 2.3 in hand, we know it suffices to estimate the Chow parameters of h ∗ with respect to D x . This would then give us a good approximation to v ∗ conditioned on D x indeed having its low-degree moments approximately match those of N ( 0 , I ) . We use the following lemma, which estimates the Chow parameters robustly under adversarial label noise.

̸

Lemma 2.4. Let η, τ ∈ (0 , 1) and G be a distribution over R d ×{± 1 } such that E x ∼ G x [ xx ⊤ ] ≼ 2 I . Let v ∈ R d be a unit vector such that Pr ( x ,y ) ∼ G [sign( v · x ) = y ] ≤ η. Then there exists an algorithm that takes N = O ( d/η 2 ) log( d/τ ) labeled samples from G , runs in time poly( N ) , and outputs a vector w such that ∥ E x ∼ G x [sign( v · x ) x ] -w ∥ 2 ≤ O ( √ η ) with probability at least 1 -τ .

The proof follows from standard arguments in robust mean estimation and the details can be found in Appendix C. We remark that the number of labeled samples required by this procedure is (nearly) linear with respect to d for constant η , which is important for us to obtain (nearly) optimal label complexity overall.

We are now ready to describe our testable proper learner. In particular, it first certifies that the moments of the underlying distribution match with the standard Gaussian and then draws i.i.d. samples to estimate the Chow parameter. It then follows from Fact 2.2, Lemmas 2.3 and 2.4 that the resulting Chow vector, conditioned on that the moment test pass, is closed to the defining vector of the optimal halfspace. We given the pseudo-code of the algorith and its analysis, which also serves as the proof of Propostion 2.1, in the end of Appendix A.

3 Efficient Testable Learning of Halfspaces

In this section, we give our tester-learner for homogeneous halfspaces under the Gaussian distribution, thereby proving Theorem 1.3. Throughout this section, we will fix an optimal halfspace h ∗ ( x ) = sign( v ∗ · x ) , i.e., a halfspace with optimal 0-1 error.

The structure of this section is as follows: In Section 3.1, we present a tester which certifies that the probability of the disagreement region between two halfspaces whose defining vectors are close to each other is small under D x . In Section 3.2, we present and analyze our localization step and combine it with the tester from Section 3.1 to obtain our final algorithm.

3.1 From Parameter Distance to 0-1 Error

For two homogeneous halfspaces h u ( x ) = sign( u · x ) and h v ( x ) = sign( v · x ) , where u , v are unit vectors, if D x is the standard Gaussian, N ( 0 , I ) , we can express the probability mass of their disagreement region as follows (see, e.g., Lemma 4.2 of [DKS18]):

̸

Hence, learning homogeneous halfspaces under Gaussian marginals can often be reduced to approximately learning the defining vector of some optimal halfspace h ∗ . This is no longer the case if D x is an arbitrary distribution, which may well happen in our regime. We show in this section that it is still possible to 'certify' whether some relationship similar to the one in Equation (1) holds.

In particular, given a known vector v , we want to make sure that for any other vector w that is close to v , the mass of the disagreement region between the halfspaces defined by by v , w respectively is small. To do so, we will decompose the space into many thin 'slabs' that are stacked on top of each other in the direction of v . Then, we will certify the mass of disagreement restricted to each of the slab is not too large. For slabs that are close to the halfspace sign( v · x ) , we can check these slabs must not themselves be too heavy. For slabs that are far away from the halfspace, we use the observation that the points in the disagreement region must then have large components in the subspace perpendicular to v . Hence, as long as D has its second moment bounded, we can bound the mass of the disagreement region in these far-away slabs using standard concentration inequality. Formally, we have the following lemma, whose proof can be found in Appendix B.

Lemma 3.1 (Wedge Bound) . Let D x be a distribution over R d . Given a unit vector v and parameters η, τ ∈ (0 , 1 / 2) , there exists an algorithm (Algorithm 1) that draws i.i.d. samples from D x , runs in

Input: Sample access to a distribution D x over R d ; tolerance parameter η > 0 ; unit vector v ∈ R d ; failure probability τ ∈ (0 , 1) .

̸

Output: Certifies that for all unit vectors w such that ∥ w -v ∥ 2 ≤ η it holds that Pr x ∼ D x [sign( v · x ) = sign( w · x )] ≤ Cη , for some absolute constant C > 1 , or reports that D x is not N ( 0 , I ) .

Set B = ⌈ √ log(1 /η ) /η ⌉ .

Let ˜ D be the empirical distribution obtained by drawing poly( d, 1 /η ) log(1 /τ ) samples from D x .

For integers -B -1 ≤ i ≤ B , define E i to be the event that { v · x ∈ [ iη, ( i +1) η ] } and E B +1 to be the event that {| v · x | ≥ √ log(1 /η ) } .

Verify that ∑ B +1 i = -B -1 ∣ ∣ Pr N ( 0 , I ) [ E i ] -Pr ˜ D [ E i ] ∣ ∣ ≤ η.

Let S i the distribution of ˜ D conditioned on E i and S ⊥ i be S i projected on the subspace orthogonal to v .

For each i , verify that S ⊥ i has bounded covariance, i.e., check that E x ∼ S ⊥ i [ xx ⊤ ] ≼ 2 I .

̸

time poly( d, 1 /η ) log(1 /τ ) , and reports either one of the following: (i) For all unit vectors w such that ∥ w -v ∥ 2 ≤ η it holds Pr x ∼ D x [sign( v · x ) = sign( w · x )] ≤ Cη , for some absolute constant C > 1 . (ii) D x is not the standard Gaussian N ( 0 , I ) . Moreover, with probability at least 1 -τ , the report is accurate.

3.2 Algorithm and Analysis: Proof of Theorem 1.3

We employ the idea of 'soft' localization used in [DKS18]. In particular, given a vector v and a parameter σ , we use rejection sampling to define a new distribution D v ,σ that 'focuses' on the region near the halfspace sign( v · x ) .

Fact 3.2 (Rejection Sampling, Lemma 4.7 of [DKS18]) . Let D be a distribution on labeled examples ( x , y ) ∈ R d ×{± 1 } . Let v ∈ R d be a unit vector and σ ∈ (0 , 1) . We define the distribution D v ,σ as follows: draw a sample ( x , y ) from D and accept it with probability e -( v · x ) 2 · ( σ -2 -1) / 2 . Then, D v ,σ is the distribution of ( x , y ) conditional on acceptance. If the x -marginal of D is N ( 0 , I ) , then the x -marginals of D v ,σ is N ( 0 , Σ ) , where Σ = I -(1 -σ 2 ) vv T . Moreover, the acceptance probability of a point is σ .

We now briefly discuss the sample and label complexities of this rejection sampling procedure. Since the acceptance probability is σ , we need roughly 1 /σ samples from D to simulate one sample from D v ,σ . On the other hand, to simulate one labeled sample from D , one only need 1 label query in addition to the unlabeled samples as one can query only the points that are accepted by the procedure. This makes sure that rejection sampling is efficient in terms of its label complexity.

The main idea of localization is the following. Let v be a vector such that ∥ v -v ∗ ∥ 2 ≤ δ . Suppose that we use localization to the distribution D v ,δ . If we can learn a halfspace with defining vector w that achieves sufficiently small constant error with respect to the new distribution D v ,δ , we can then combine our knowledge of w and v to produce a new halfspace with significantly improved error guarantees under D . The following lemma formalizes this geometric intuition.

Lemma 3.3. Let v ∗ , v be two unit vectors in R d such that ∥ v -v ∗ ∥ 2 ≤ δ ≤ 1 / 100 . Let Σ = I -(1 -δ 2 ) vv T and w be a unit vector such that ∥ ∥ ∥ ∥ w -Σ 1 / 2 v ∗ ∥ Σ 1 / 2 v ∗ ∥ 2 ∥ ∥ ∥ ∥ 2 ≤ ζ ≤ 1 / 100 . Then it holds

We defer the proof to Appendix C. Below, we provide a few useful remarks regarding the relevant parameters.

Remark 3.4. Observe that in Lemma 3.3 we require that the distance of v and v ∗ is smaller than 1 / 100 . While this constant is not the best possible, we remark that some non-trivial error is indeed necessary so that the localization step works. For example, assume that v and v ∗ are orthogonal, i.e., v · v ∗ = 0 , and that Σ = I -(1 -ξ 2 ) vv T for some ξ ∈ [0 , 1] . Observe that Σ 1 / 2 scales vectors by a factor of ξ in the direction of v and leaves orthogonal directions unchanged. Similarly, its inverse Σ -1 / 2 scales vectors by a factor of 1 /ξ in the direction of v and leaves orthogonal directions unchanged. Without loss of generality, assume that v = e 1 , v ∗ = e 2 where e 1 , e 2 are the two standard basis vectors. Then Σ 1 / 2 v ∗ = v ∗ . Moreover, assume that w = a e 1 + b e 2 (with a 2 + b 2 = 1 ). We observe that ∥ w -Σ 1 / 2 v ∗ / ∥ Σ 1 / 2 v ∗ ∥ 2 ∥ 2 2 = ∥ w -v ∗ ∥ 2 2 = 2 -2 b . However, we have that s = Σ -1 / 2 w = ( a/ξ ) e 1 + b e 2 . Therefore, ∥ s / ∥ s ∥ 2 -v ∗ ∥ 2 2 = 2 -2 b/ √ ( a/ξ ) 2 + b 2 . Notice that for all ξ ∈ [0 , 1] it holds that s / ∥ s ∥ 2 is further away from v ∗ than w , i.e., rescaling by Σ -1 / 2 worsens the error.

We are now ready to present an iterative testing/learning procedure that will serve as the main component of our algorithm. At a high level, we first use Algorithm 2 from Proposition 2.1 to learn a vector v that is close to v ∗ in ℓ 2 -distance up to some sufficiently small constant. Then we localize to the learned halfspace and re-apply Algorithm 2 to iteratively improve v . In particular, we will argue that, whenever the learned halfspace is still significantly suboptimal, the algorithm either detects that the underlying distribution is not Gaussian or keeps making improvements such that v gets closer to v ∗ conditioned on v still being sub-optimal. The pseudocode and analysis of the algorithm can be found in Appendix C.

Lemma 3.5. Suppose ∥ ∥ v ∗ -v ( t ) ∥ ∥ 2 ≤ δ ≤ 1 / 100 . There is an algorithm (Algorithm 3) that with probability at least 1 -τ , either (i) correctly reports that the x -marginal of D is not N ( 0 , I ) or (ii) computes a unit vector v ( t +1) so that either ∥ ∥ v ∗ -v ( t +1) ∥ ∥ 2 ≤ δ/ 2 or ∥ ∥ v ∗ -v ( t ) ∥ ∥ 2 ≤ C opt , where C > 0 is an absolute constant. Furthermore, the algorithm draws N = poly( d, 1 /δ ) log(1 /τ ) many unlabeled samples and uses O ( d log( d/τ )) label queries.

After running Algorithm 3 for at most a logarithmic number of iterations, we know v must be close to v ∗ . Then, we can use Algorithm 1 from Lemma 3.1 to certify that the disagreement between the learned halfspace and h ∗ ( x ) is small. One technical difficulty is that we do not know precisely when to terminate the update and each update does not necessarily monotonically bring v closer to v ∗ . As a result, the process only yields a list of vectors v (1) , · · · v ( t ) with the guarantee that at least one of them is sufficiently closed to v ∗ . A natural idea is to estimate the 0-1 errors of the halfspaces defined by the vectors in the list and simply pick the one with the smallest empirical error. Naively, we need to estimate the errors up to an additive ϵ and this may take up to Ω(1 /ϵ 2 ) labeled samples, which is exponentially worse than our goal in terms of the dependency on ϵ . Nonetheless, we remark that comparing the errors of two halfspaces which differ by some multiplicative factors can be done much more efficiently in terms of its label complexity. In particular, it suffices for us to compare the errors of the two halfspaces restricted to the area in which the predictions made by them differ. Under this conditioning, the difference between the errors then gets magnified to some constant, making comparisons much easier. Hence, we can run a tournament among the the halfspaces, which reduces to perform pairwise comparisons among the lists, to obtain our final winner hypothesis. The result is summarized in the following lemma.

̸

Lemma 3.6 (Tournament) . Let ϵ, τ ∈ (0 , 1) and D a distribution over R d × {± 1 } . Given a list of halfspaces { h ( i ) } k i =1 , there is an algorithm that draws N = Θ ( k 2 log( k/τ ) /ϵ ) i.i.d. unlabeled samples from D x , uses Θ ( k 2 log( k/τ ) ) label queries, runs in time poly( d, N ) and outputs a halfspace ˜ h satisfying that Pr ( x ,y ) ∼ D [ ˜ h ( x ) = y ] ≤ 10 min i Pr ( x ,y ) ∼ D [ h ( i ) ( x ) = y ] + ϵ.

̸

We provide the pseudocode of the algorithm and its analysis in Appendix C. The proof of Theorem 1.3 then follows.

Our final algorithm (Algorithm 5) simply iteratively applies the localized update routine (Algorithm 3) and then runs a tournament (Algorithm 4) among the O (log(1 /ϵ )) candidate halfspaces obtained from the iterations. The pseudocode and its analysis, which serves as proof of our main theorem, can be found at the end of Appendix C.

References

[ABL17] P. Awasthi, M. F. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. J. ACM , 63(6):50:1-50:27, 2017.

[BBL06] M. Balcan, A. Beygelzimer, and J. Langford. Agnostic active learning. In Proceedings of the 23rd international conference on Machine learning , pages 65-72, 2006.

[BBZ07] M. Balcan, A. Broder, and T. Zhang. Margin based active learning. In Learning Theory: 20th Annual Conference on Learning Theory, COLT 2007, San Diego, CA, USA; June 13-15, 2007. Proceedings 20 , pages 35-50. Springer, 2007.

[BL13] M. Balcan and P. Long. Active and passive learning of linear separators under logconcave distributions. In Conference on Learning Theory , pages 288-316. PMLR, 2013.

[Dan15] A. Daniely. A PTAS for agnostically learning halfspaces. In Proceedings of The 28 th Conference on Learning Theory, COLT 2015 , pages 484-502, 2015.

[Dan16] A. Daniely. Complexity theoretic limitations on learning halfspaces. In Proceedings of the 48 th Annual Symposium on Theory of Computing, STOC 2016 , pages 105-117, 2016.

[DGJ + 10] I. Diakonikolas, P. Gopalan, R. Jaiswal, R. Servedio, and E. Viola. Bounded independence fools halfspaces. SIAM Journal on Computing , 39(8):3441-3462, 2010.

[DKK + 21] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Agnostic proper learning of halfspaces under gaussian marginals. In Proceedings of The 34 th Conference on Learning Theory, COLT , 2021.

[DKMR22] I. Diakonikolas, D. M. Kane, P. Manurangsi, and L. Ren. Cryptographic hardness of learning halfspaces with massart noise. CoRR , abs/2207.14266, 2022.

[DKN10] I. Diakonikolas, D. M. Kane, and J. Nelson. Bounded independence fools degree2 threshold functions. In FOCS , pages 11-20, 2010.

[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Proceedings of The 34 th Conference on Learning Theory, COLT , 2021.

[DKR23] I. Diakonikolas, D. M. Kane, and L. Ren. Near-optimal cryptographic hardness of agnostically learning halfspaces and relu regression under gaussian marginals. CoRR , abs/2302.06512, 2023.

[DKS18] I. Diakonikolas, D. M. Kane, and A. Stewart. Learning geometric concepts with nasty noise. In Proceedings of the 50 th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 1061-1073, 2018.

[DKTZ20a] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with massart noise under structured distributions. In Conference on Learning Theory, COLT , 2020.

[DKTZ20b] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Non-convex SGD learns halfspaces with adversarial label noise. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[DKZ20] I. Diakonikolas, D. M. Kane, and N. Zarifis. Near-optimal SQ lower bounds for agnostically learning halfspaces and ReLUs under Gaussian marginals. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[GGK20] S. Goel, A. Gollakota, and A. R. Klivans. Statistical-query lower bounds via functional gradients. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[GKK22] A. Gollakota, A. Klivans, and P. Kothari. A moment-matching approach to testable learning and a new characterization of rademacher complexity. arXiv preprint arXiv:2211.13312 , 2022. To appear in STOC'23.

[GKSV23] A. Gollakota, A. R. Klivans, K. Stavropoulos, and A. Vasilyan. An efficient tester-learner for halfspaces. CoRR , abs/2302.14853, 2023.

[Hau92] D. Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation , 100:78-150, 1992.

[KKMS08] A. Kalai, A. Klivans, Y. Mansour, and R. Servedio. Agnostically learning halfspaces. SIAM Journal on Computing , 37(6):1777-1805, 2008. Special issue for FOCS 2005.

[KLS09] A. Klivans, P. Long, and R. Servedio. Learning Halfspaces with Malicious Noise. Journal of Machine Learning Research , 10:2715-2740, 2009.

[KSS94] M. Kearns, R. Schapire, and L. Sellie. Toward Efficient Agnostic Learning. Machine Learning , 17(2/3):115-141, 1994.

[Ros58] F. Rosenblatt. The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review , 65:386-407, 1958.

[RV23] Ronitt Rubinfeld and Arsen Vasilyan. Testing distributional assumptions of learning algorithms. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 1643-1656, 2023.

[She21] J. Shen. On the power of localized perceptron for label-optimal learning of halfspaces with adversarial noise. In International Conference on Machine Learning , pages 95039514. PMLR, 2021.

[Tie22] S. Tiegel. Hardness of agnostically learning halfspaces from worst-case lattice problems. CoRR , abs/2207.14030, 2022.

[Val84] L. G. Valiant. A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing (STOC) , pages 436-445. ACM Press, 1984.

Supplementary Material

A Omitted Proofs for Weak Testable Proper Agnostic Learning

Proof of Lemma 2.3. It suffices to show that for any unit vector u ∈ R d , the following holds:

The following fact expresses a real number a as an integral of the sign function.

We apply Fact A.1 to the term u · x , which gives

where in the last line we switch the order of the integral of t and x by Fubini's theorem. We then split the above integral over t into two parts based on the magnitude of t ( t > 1 / √ η versus 0 ≤ t ≤ 1 / √ η ) and apply the triangle inequality:

We start by bounding the integral for t ≥ 1 / √ η .

Lemma A.2 (Chow-Distance Tail) . Let Q be distribution over R d with E x ∼ Q [ xx ⊤ ] ≼ 2 I . Moreover, let g ( x ) : R d ↦→ R be a bounded function, i.e., | g ( x ) | ≤ 1 for all x ∈ R d . It holds

Proof. We split the expectation into two parts based on the relative sizes of | u · x | and t . Specifically, we can write:

For the second term in Equation (3), we rely on the following observation: when | u · x | ≤ t , the quantities u · x -t and u · x + t have opposite signs. Hence, we conclude the integrand is 0 everywhere and therefore the second term is also 0 . For the first term, we have

where the first inequality follows from the triangle inequality, the second inequality uses the fact that the sign( · ) function is at most 1 and the third inequality follows from Chebyshev's inequality using the fact that the E [ xx ⊤ ] ≼ 2 I . Combining our analysis for the two terms in Equation (3), we can then conclude the proof of Lemma A.2.

Using the triangle inequality and applying Lemma A.2 on the distributions B and N ( 0 , I ) , we have that

We then turn our attention to the terms

To bound Equations (5) and (6), we need the following fact from [GKK22].

Fact A.3 (Theorem 5.6 of [GKK22]) . Let h : R d ↦→ {± 1 } be a function of p halfspaces, i.e., h ( x ) = g ( h 1 ( x ) , · · · , h p ( x )) where h i are halfspaces and g : {± 1 } p ↦→ {± 1 } . For any k ∈ N , let ∆ = √ p 2 k 1 d k ( 1 C ′ √ k ) k +1 for some sufficiently large absolute constant C ′ > 0 . Then, for any distribution B whose moments up to order k match those of N ( 0 , I ) up to ∆ , we have

for some constant C > 0 .

For a fixed t , note that h ( x )sign( u · x -t ) is a function of two halfspaces. Moreover, from the assumptions of Lemma 2.3, the distributions B and N ( 0 , I ) match k = C log(1 /η ) /η 2 moments up to error ∆ = 1 kd k ( 1 C √ k ) k +1 , where C > 0 is a sufficiently large absolute constant. Therefore, applying Fact A.3 and the triangle inequality gives

Similarly, we can show that Equation (6) is bounded by O ( √ η ) . Substituting the bounds from Equations (4) and (7) into Equation (2) then gives

Since u is chosen as an arbitrary unit vector, this implies that

Combining this with Fact 2.2 concludes the proof of Lemma 2.3.

Proof of Lemma 2.4. We first show that ∥ ∥ E x ∼ G x [sign( v · x ) x ] -E ( x ,y ) ∼ G [ y x ] ∥ ∥ 2 ≤ O ( √ η ) . For any unit vector u , we have that

̸

where we used the Cauchy-Schwarz inequality and the fact that Pr ( x ,y ) ∼ G [sign( v · x ) = y ] ≤ η . Therefore, we have that ∥ ∥ E x ∼ G x [sign( v · x ) x ] -E ( x ,y ) ∼ G [ y x ] ∥ ∥ 2 ≤ 4 √ η . Let ( x (1) , y (1) ) , . . . , ( x ( N 1 ) , y ( N 1 ) ) be samples drawn from G , where N 1 = O ( d/η 2 ) . Then, let ˜ w = (1 /N 1 ) ∑ N 1 j =1 y ( j ) x ( j ) . From Chebyshev's inequality, we know the i -th coordinate of ˜ w should satisfy Pr [ | ˜ w i -E ( x ,y ) ∼ D [ y x ] | ≥ η/ √ d ] ≤ 4 d/ ( N 1 η 2 ) ≤ 1 / 4 . Therefore, using the standard median technique, we can find a w median i , so that Pr [ | w median i -E ( x ,y ) ∼ D [ y x ] | ≥ η/ √ d ] ≤ τ/d , using N 2 = O ( N 1 log( d/τ )) samples. Let w = ( w median 1 , . . . , w median d ) , then we have that ∥ ∥ E ( x ,y ) ∼ G [ y x ] -w ∥ ∥ 2 ≤ O ( η ) with probability at least 1 -τ . Then, using the triangle inequality, we have that ∥ E x ∼ G x [sign( v · x ) x ] -w ∥ 2 ≤ O ( √ η ) , which concludes the proof of Lemma 2.4.

Proof of Proposition 2.1. Let k, ∆ , N be defined as in Algorithm 2. If D x is N ( 0 , I ) , the moments up to degree k of the x -marginal of the empirical distribution ̂ D N (obtained after drawing N i.i.d. samples from D x ) are close to those of N ( 0 , I ) up to additive error ∆ with probability at least 1 -τ/ 10 .

If Algorithm 2 did not terminate on Line 3, we then have that the moments up to degree k of ̂ D N are close to those of N ( 0 , I ) up to additive error ∆ with probability at least 1 -τ/ 10 . Then, applying Lemma 2.3 with B = ̂ D N and h ( x ) = sign( v ∗ · x ) , we get that

Now, suppose we were to query the labels of all points in ̂ D N . We could then construct another empirical distribution ̂ L N over labeled samples. By our assumption, the error of sign( v ∗ · x ) under

Input: Sample access to a distribution D x over unlabeled samples and query access to the labels of the drawn samples according to D ; certification range η ; failure probability τ .

d

Output: Either reports that the D x is not N ( 0 , I ) ; or returns a unit vector w ∈ R such that ∥ v ∗ -w ∥ 2 ≤ C A √ opt + η .

Set k = C log(1 /η ) /η 2 and ∆ = 1 kd k ( 1 C √ k ) k +1 , where C > 0 is a sufficiently large absolute constant.

Draw N = d Ck log k log(1 /τ ) samples from D x and construct the empirical distribution ̂ D N .

Certify that the moments of ̂ D N up to degree k match with those of N ( 0 , I ) up to error ∆ .

If the above does not hold; report that D x is not N ( 0 , I ) and terminate.

Draw N ′ = O ( d/η 2 ) log( d/η ) samples from ̂ D N and query their labels. Denote the resulting labeled samples as S .

Use algorithm from Lemma 2.4 on S and obtain w . Return w / ∥ w ∥ 2 .

Algorithm 2: Proper Testable Learner

D is at most opt . Hence, the error of sign( v ∗ · x ) under ̂ L N is at most opt + η with probability at least 1 -τ/ 10 . Next, notice that the set of samples S in Line 5 are exactly i.i.d. samples from ̂ L N . Hence, applying Lemma 2.4 on ̂ L N then gives that, with probability at least 1 -τ/ 10 , the vector w , computed on Line 6 of Algorithm 2 satisfies

Notice that the x -marginal of ̂ L N is exactly ̂ D N . Hence, combining Equations (8) and (9), we get that ∥ ∥ ∥ w -√ 2 /π v ∗ ∥ ∥ ∥ 2 ≤ O ( √ opt + η ) , as desired.

B Proof of Wedge Bound Lemma 3.1

Proof of Lemma 3.1. Recall that ˜ D is the empirical distribution made up of N i.i.d. samples from D x where N = poly( d, 1 /η ) log(1 /τ ) . We consider ˜ D restricted to a set of thin 'slabs' stacked on each other in the direction of v . More formally, we define S i to be the distribution of ˜ D conditioned on v · x ∈ [( i -1) η, iη ] , for i ∈ [ -√ log(1 /η ) /η, √ log(1 /η ) /η ] , and S ⊥ i to be the distribution S i projected into the subspace orthogonal to v .

Suppose that D x is indeed N ( 0 , I ) . Then the distribution ˜ D is the empirical distribution formed by samples taken from N ( 0 , I ) . In this case, it is easy to see that both Line 4 and 6 of Algorithm 1 pass with high probability.

Claim B.1. Assume that D x = N ( 0 , I ) . Then the tests at Line 4 and 6 of Algorithm 1 pass with probability at least 1 -τ/ 10 .

Proof. If x ∼ N ( 0 , I ) , then v · x ∼ N (0 , 1) . If we concatenate the values of Pr N ( 0 , I ) [ E i ] into a vector, it can be viewed as the discretization of N (0 , 1) into 2 B + 3 many buckets. On the other hand, Pr ˜ D [ E i ] is an empirical version of this discrete distribution composed of N i.i.d. samples where N = poly( d, 1 /η ) log(1 /τ ) . Since we can learn any discrete distribution with support n up to error η in total variation distance with Θ( n/η 2 ) log(1 /τ ) samples with probability at least 1 -τ , it follows that Line 4 will pass with high probability as long as we take more than Θ( B/η 2 ) log(1 /τ ) ≤ poly( d, 1 /η ) log(1 /τ ) many samples.

For Line 6, we remark that S ⊥ i is the empirical version of a ( d -1) -dimensional standard Gaussian. Since the empirical mean and the empirical covariance concentrates around the true mean and

covariance with probability at least 1 -τ if one takes more than Θ( d 2 /η 2 ) log(1 /τ ) many samples, it follows that Line 4 will pass with high probability as long as we take more than Θ( d 2 /η 2 ) log(1 /τ ) ≤ poly( d, 1 /η ) log(1 /τ ) many samples.

Suppose that both lines pass. We claim that this implies the following: for all unit vectors w such that ∥ w -v ∥ 2 ≤ η it holds

̸

for some absolute constant C > 1 . Given this, we can deduce that the same equation must also hold for D x with high probability - albeit with a larger constant C ′ . To see this, we remark that the left hand side of the equation can be treated as the error of the halfspace sign( w · x ) if the true labels are generated by sign( v · x ) . Since the VC-dimension of the class of homogeneous halfspaces is d , we have that the error for all w under D x is well-approximated by that under ˜ D up to an additive η with probability at least 1 -τ given N = poly( d, 1 /η ) log(1 /τ ) many samples. Hence, conditioned on Equation (10), it holds with probability at least 1 -τ that

̸

for all w .

We now proceed to show Equation (10) holds if the Algorithm 1 did not terminate on Lines 4 and 6. Conditioned on Line 6, for any unit vector u ∈ R d that is orthogonal to v , we have | E [ u · x ] | ≤ O (1) and Var[ u · x ] ≤ 2 . Using Chebyshev's inequality, for any α > 0 , it holds

̸

We can now bound Pr x ∼ ˜ D [sign( v · x ) = sign( w · x )] for an arbitrary unit vector w satisfying ∥ w -v ∥ 2 ≤ η . We proceed to rewrite w as (1 -γ 2 ) 1 / 2 v + γ u for some unit vector u ∈ R d that is orthogonal to v and γ ∈ (0 , η ) . Denote γ ′ = γ/ (1 -γ 2 ) 1 / 2 , then the event that sign( v · x ) = sign( w · x ) implies that γ ′ | u · x | ≥ | v · x | . Therefore, we have that

̸

̸

where in the second inequality we use the definition of S i , and in the third inequality we use that γ ≤ η and Equation (11). We now bound from above the total disagreement probability between w and v under ˜ D . We have that

̸

̸

where we used that Pr x ∼ ˜ D [ | x · v | > √ log(1 /η )] ≤ 3 η and Pr x ∼ ˜ D [ | x · v | < η ] ≤ 3 η , since in Line 4 we verified that the probabilities Pr x ∼ ˜ D [ v · x ∈ [( i -1) η, iη ]] are close to the probabilities under N ( 0 , I ) and hence bounded by O ( η ) , Equation (12), and the fact that the series ∑ i 1 i 2 is convergent and less than π 2 / 6 .

C Omitted Proofs of localization and main theorem

Proof of Lemma 3.3. Since ∥ v -v ∗ ∥ 2 ≤ δ , we can write

for some κ ∈ [0 , δ ] and some unit vector u perpendicular to v . By definition, Σ 1 / 2 shrinks in the direction of v by a factor of δ and leaves other orthogonal directions unchanged. Hence, it holds

Then, using the triangle inequality, we obtain

since κ is upper bounded by δ . Since ∥ ∥ ∥ ∥ w -Σ 1 / 2 v ∗ ∥ Σ 1 / 2 v ∗ ∥ 2 ∥ ∥ ∥ ∥ 2 ≤ ζ , we can write

for some | a | , b ∈ [0 , ζ ] and u ′ perpendicular to v . We can multiply both sides by γ and get

Using Equation (13), we then have

Let λ := 1 √ 1+ κ 2 + aγ/δ be the coefficient before v . We next identify the range of λ .

Claim C.1. It holds that λ ∈ [1 -δ -2 ζ, 1 + 2 ζ ] .

Proof. Recall that κ ∈ [0 , δ ] , | a | ∈ [0 , ζ ] and γ ∈ [0 , 2 δ ] . If we view λ as a function of κ, a, γ , it is minimized when κ = δ , a = -ζ , γ = 2 δ , which then gives

where in the second inequality we use the fact that 1 √ 1+ x 2 ≥ 1 -x for x ≥ 0 . On the other hand, λ is maximized when κ = 0 , a = ζ, γ = 2 δ , which gives λ ≤ 1 + 2 ζ. Hence, we can conclude that λ ∈ [1 -δ -2 ζ, 1 + 2 ζ ] .

We multiply both sides of Equation (14) by 1 λ √ 1+ κ 2 , which gives

where in the second equality we use Equation (13). We then bound from above and below the norm, and we get that

where we used triangle inequality. Note that

and that κ ≤ δ . Therefore, we obtain that

which implies that

Input: Sample access to a distribution D over labeled examples; unit vector v ∈ R d ; parameters τ, η, δ > 0 .

Output: Either reports that D x is not N ( 0 , I ) or computes a unit vector v ′ such that: either ∥ v ∗ -v ′ ∥ 2 ≤ δ/ 2 or ∥ v ∗ -v ′ ∥ 2 ≤ O (opt)

Let D v ,δ be the distribution obtained by running Rejection Sampling with parameters v and δ

.

Check the acceptance probability is within [ δ/ 2 , 3 δ/ 2] . Otherwise, report x -marginals of D is not N ( 0 , I ) .

Let G be the distribution obtained by applying the transformation Σ -1 / 2 on the x marginals of D v ,δ where Σ = I -(1 -δ 2 ) vv ⊤ .

Run Algorithm 2 on G with accuracy η = 1 / (20000 C 2 A ) to obtain a unit vector w .

If the algorithm reports the marginal of G is not N ( 0 , I ) , terminate and report it.

Set v ′ = Σ 1 / 2 w / ∥ ∥ Σ 1 / 2 w ∥ ∥ 2 and return v ′ .

Algorithm 3: Testable Localized-Update

This concludes the proof.

Proof of Lemma 3.5. For simplicity, we denote v ( t ) as v . Weapply the Rejection Sampling procedure from Fact 3.2 in the direction of v with σ = δ . If the x -marginal of D is N ( 0 , I ) , the acceptance probability of D v ,δ is exactly δ . We then estimate the acceptance probability with accuracy ϵ ; if it is not lying inside the interval [ δ/ 2 , 3 δ/ 2] (see Line 2 of Algorithm 3), we report that the x -marginal of D is not standard normal and terminate.

Conditioned on the event that the algorithm did not terminate, we can get an (unlabeled) sample from the x -marginal of D v ,δ , using O (1 /δ ) unlabeled samples from D x . To get a labeled sample from D v ,δ , we only need 1 additional label query. Note that, under the distribution D v ,δ , the error of the v ∗ is

̸

̸

̸

where we used that the probability of the acceptance is at least δ/ 2 . Denote by G the distribution of ( Σ -1 / 2 x , y ) , where ( x , y ) ∼ D v ,δ and Σ = I -(1 -δ 2 ) vv ⊤ . We note that if the x -marginal of D were the standard normal, then x -marginal of G is the standard normal. Hence, we can apply the algorithm Algorithm 2 from Proposition 2.1. Under the transformed distribution, we have that the new optimal vector ( v ∗ ) ′ := Σ 1 / 2 v ∗ / ∥ Σ 1 / 2 v ∗ ∥ 2 .

From Proposition 2.1, running Algorithm 2 on the normalized distribution G with error parameter η ≤ 1 / (2000 C 2 A ) and failure probability τ consumes poly( d ) log(1 /τ ) unlabeled samples and O ( d log( d/τ )) labeled samples from G . Since the rejection sampling procedure for G accepts points with probability at least Ω( ϵ ) Therefore, it consumes poly( d, 1 /ϵ ) log(1 /τ ) samples from D and uses O ( d log( d/τ )) additional label queries. Conditioned on the event that it succeeds (which happens with probability at least 1 -τ ), it either (i) reports that the x -marginal of G is not standard Gaussian (ii) returns a unit vector w such that

̸

In case (i), we can directly report that x -marginal of D is not N ( 0 , I ) since the x -marginal of G ought to be N ( 0 , I ) .

̸

In case (ii), we claim that at least one of the following hold (a) v before the localized update is already good enough, i.e. ∥ v ∗ -v ∥ 2 ≤ 40000 C A 2 opt ; (b) it holds ∥ ( v ∗ ) ′ -w ∥ 2 ≤ 1 / 100 . Suppose that (a) does not hold; we will show that it then must hold ∥ ( v ∗ ) ′ -w ∥ 2 ≤ 1 / 100 . Since ∥ v ∗ -v ∥ 2 ≤ δ and ∥ v ∗ -v ∥ 2 > 40000 C A 2 opt , we have that opt ≤ δ/ (40000 C 2 A ) . Furthermore, from Equation (15) we have that Pr ( x ,y ) ∼ D v ,δ [sign( v ∗ · x ) = y ] ≤ 2opt /δ , it then follows that Pr ( x ,y ) ∼ G [sign(( v ∗ ) ′ · x ) = y ] ≤ 2opt /δ ≤ 1 / (20000 C 2 A ) . Substituting this into Equation (16) then gives

Using our assumption that ∥ v -v ∗ ∥ 2 ≤ δ < 1 / 100 , we can apply Lemma 3.3, which gives that

Hence, we set v ( t +1) = ∥ ∥ ∥ ∥ Σ -1 / 2 w ∥ Σ -1 / 2 w ∥ 2 ∥ ∥ ∥ ∥ 2 and this completes the proof.

Input: Sample access to D x over unlabeled examples and query access to the labels of the samples drawn; a list of k halfspaces { h ( i ) } k i =1 ; parameters τ, ϵ > 0 .

Output: finds a halfspace ˜ h whose error is nearly optimal among the list.

Let C be a sufficiently large constant.

For i, j = 1 · · · k

(a) Take N 1 := C log( k/τ ) /ϵ samples D x .

̸

(b) Skip the iteration if there are fewer than N 2 := 0 . 1 C log( k/τ ) samples such that h ( i ) ( x ) = h ( j ) ( x ) . Denote the samples satisfying the condition as S 1 .

(c) Query the labels of a random subset of samples from S 1 of size N 2 . Denote the set of labeled samples as S 2

(d) Compute the empirical errors of h ( i ) , h ( j ) using S 2 .

(e) If the empirical errors differ by more than 1 / 4 , mark the one with larger error as sub-optimal.

Return any halfspace among the list that has not been marked as sub-optimal or declare failure if there is not any.

Algorithm 4: Tournament

Proof of Lemma 3.6. Fix a halfspace h ( i ) . Suppose there is another halfspace h ( j ) such that

̸

̸

̸

̸

̸

We argue the halfspace will be marked as sub-optimal with probability at least τ/ log(1 /ϵ ) . For convenience, denote a := Pr ( x ,y ) ∼ D [ h ( i ) ( x ) = y ∧ h ( i ) ( x ) = h ( j ) ( x )] , b := Pr ( x ,y ) ∼ D [ h ( j ) ( x ) = y ∧ h ( i ) ( x ) = h ( j ) ( x )] and m := Pr x ∼ D x [ h ( i ) ( x ) = h ( j ) ] . Notice that a, b are the errors of the two halfspaces from the area in which their predictions differ and m is the mass of the area. Then, it holds

̸

̸

̸

̸

̸

̸

̸

̸

̸

This further implies that a/m -b/m > 4 / 5 . Notice that a/m , b/m are precisely the errors of h ( i ) , h ( j ) under the conditional distribution of D restricted to the area in which h ( i ) ( x ) = h ( j ) ( x ) . Hence,

̸

with Θ(log( k/τ )) many samples, the corresponding empirical errors will differ by at least 1 / 4 with probability at least τ/k .

On the other hand, suppose

̸

̸

This implies that a/m < b/m . As a result, the algorithm either skips the iteration if there are fewer than Θ(log( k/τ )) many labeled samples from the conditional distribution or the empirical error of h ( i ) will not be less than that of h ( j ) by more than 1 / 4 with probability at least τ/k 2 if there are enough samples.

The correctness of the lemma then follows from union bound. Finally, we remark m , the mass of the area we condition to, should be at least ϵ if the errors of the two halfspaces differ by at least ϵ . In that case, it takes on average Θ(1 /ϵ ) unlabeled samples and 1 additional label query to simulate one labeled sample from the conditional distribution. Therefore, it suffices if the algorithm takes O ( k 2 log( k/τ ) /ϵ ) many unlabeled samples from D x and use Θ ( k 2 log( k/τ ) ) many additional label queries. This concludes the proof of Lemma 3.6.

Input: Sample access to a distribution D x over unlabeled examples and query access to the labels of the samples drawn; ϵ > 0 ; unit vector v ∈ R d .

̸

Output: Either reports that D x is not N ( 0 , I ) or computes hypothesis h such that Pr ( x ,y ) ∼ D [ h ( x ) = y ] = O (opt) .

Set τ = ( ϵ/ ( C log(1 /ϵ ))) for a sufficiently large constant C > 0 . Set η = 1 / (20000 C A ) where C A is the constant from Proposition 2.1.

Run Algorithm 2 on D with accuracy η to obtain unit vector v (0) .

If Algorithm 2 reports that D x is not N ( 0 , I ) , then report it and terminate.

For t = 0 . . . log(1 /ϵ ) : set δ = (1 / 100)2 -t and run Algorithm 3 with parameters v ( t ) , δ, η to obtain v ( t +1) .

For i = 0 . . . log(1 /ϵ ) :

(a) Run Algorithm 1 with v ( i ) and η = jϵ for j ∈ [1 /ϵ ] on the x -marginals of D .

(b) Set h ( i ) ( x ) = sign( v ( i ) · x ) .

Return the hypothesis obtained from running Algorithm 4 on the list of hypothesis { h ( i ) } log(1 /ϵ ) i =1 on D with accuracy ϵ and failure probability τ/ 10 .

Proof of Theorem 1.3. Denote by v ∗ , a unit vector with error at most opt , i.e., Pr ( x ,y ) ∼ D [sign( v ∗ · x ) = y ] ≤ opt . We start by analyzing Algorithm 5. In Line 2, Algorithm 5 uses Algorithm 2 with parameter η = 1 / (20000 C 2 A ) to get a hypothesis v (0) with small distance with v ∗ . From Proposition 2.1, Algorithm 2 either reports that x -marginal of D is not N ( 0 , I ) or outputs a vector v (0) small distance with v ∗ . If Algorithm 2 reports that the x -marginal of D is not N ( 0 , I ) , we can terminate the algorithm. Conditioned on the event that the algorithm did not terminate, then we have

We consider two cases depending on how large the value of opt is. If opt > 1 / (20000 C 2 A ) , then any unit vector achieves constant error; therefore, conditioned that the algorithm did not terminate on any proceeding test, any vector we output will satisfy the guarantees of Theorem 1.3. For the rest of the proof, we consider the case where opt ≤ 1 / (20000 C 2 A ) . In this case, ∥ ∥ v (0) -v ∗ ∥ ∥ 2 ≤ 1 / 100 , this means that Algorithm 5 on Lines 4-4 will decrease the distance between the current hypothesis and v ∗ .

Conditioned on the event that the algorithm did not terminate at Lines 4-4 of Algorithm 5, we claim that there must have some 0 ≤ t ∗ ≤ log(1 /ϵ ) such that ∥ ∥ v ( t ∗ ) -v ∗ ∥ ∥ 2 ≤ O (opt + ϵ ) . Let t ′ ∈ N be

̸

the maximum value so that 2 -t ′ / 100 ≥ 40000opt , then, for all t ≤ min( t ′ , log(1 /ϵ )) it holds that 2 -t / 100 ≥ 40000opt . From Lemma 3.5, we have that for all t ≤ min( t ′ , log(1 /ϵ )) it holds that

̸

From the above, note that if t ′ > log(1 /ϵ ) then ∥ ∥ v (log(1 /ϵ )) -v ∗ ∥ ∥ 2 ≤ ϵ/ 100 . If t ′ ≤ log(1 /ϵ ) , we have that ∥ ∥ ∥ v ( t ′ ) -v ∗ ∥ ∥ ∥ 2 ≤ O (opt) , which proves our claim. It remains to show that Algorithm 5 will return a vector v ′ so that Pr ( x ,y ) ∼ D [sign( v ′ · x ) = y ] = O (opt + ϵ ) . From Lemma 3.1, conditioned that Algorithm 5 did not terminate on Lines 5-5b, we have that for all vectors v (0) , . . . , v (log(1 /ϵ )) generated on Lines 4-4 of Algorithm 5, we have that

̸

̸

Hence, we can conclude that Pr ( x ,y ) ∼ D [sign( v ( t ′ ) · x ) = y ] ≤ O (opt + ϵ ) . Then, applying Lemma 3.6 gives that the returned halfspace will have error at most 10 · Pr ( x ,y ) ∼ D [sign( v ( t ′ ) · x ) = y ] + ϵ = O (opt + ϵ ) .

̸

To conclude the proof, note that each invocation of Algorithms 1, 2 and 3 consumes poly( d, 1 /ϵ ) log(1 /τ ) unlabeled samples and O ( d (log( d/τ ) + log log(1 /ϵ ))) label queries. Moreover, the total number of invocations is upper bounded by Θ(log(1 /ϵ )) . For algorithm Algorithm 4, it draws poly(1 /ϵ ) many unlabeled samples and use Θ ( log 2 (1 /ϵ ) (log log(1 /ϵ ) + log(1 /τ )) ) label queries. Therefore the total unlabeled samples consumed by Algorithm 5 is N = poly( d, 1 /ϵ ) and the total label quereis used is at most

All the procedures run in time that scales polynomially with the number of samples drawn. This concludes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning halfspaces under Gaussian distribution and adversarial label noise in the testable learning framework of Rubinfeld and Vasilyan (STOC'23). In this setup, the learning algorithm, given a few training examples drawn i.i.d. from some unknown distribution $\\mathcal{D}$, may either *accept* and returns a hypothesis or *reject*. The goal is to satisfy the following two natural constraints:\n- Completeness: If the marginal of $\\mathcal{D}$ is indeed Gaussian, the algorithm should accept w.h.p.\n- Soundness: The probability that the learning algorithm accepts and returns an inaccurate hypothesis is small.\n\nThe main result is a poly-time algorithm that works for the Gaussian distribution against adversarial label noise. In particular, if $\\mathsf{opt}$ is the smallest possible testing error achieved by the best halfspace on $\\mathcal{D}$, the algorithm guarantees $O(\\mathsf{opt}) + \\epsilon$ error in time $\\mathrm{poly}(d/\\epsilon)$. Prior work achieves $\\mathsf{opt} + \\epsilon$ error but requires $d^{\\mathrm{poly}(1/\\epsilon)}$ time.\n\nA first building block is a weak learner (given in Proposition 2.1) that outputs a weight vector $v$ that is $O(\\sqrt{\\mathsf{opt} + \\eta})$-close in time $d^{\\tilde O(1/\\eta^2)}$. (This is not sufficient due to the extra square root as well as the exponential dependence on $1/\\eta$.) This baseline learner is then boosted by localizing to the decision boundary of $\\mathrm{sgn}(v\\cdot x)$. In particular, each iteration either makes progress in improving the accuracy of $v$, or obtains a proof of non-Gaussianity in terms of moments.\n\nSTRENGTHS:\nThe paper studies a well-motivated and fundamental learning theory problem in the recently proposed model of testable learning. The results are strong, and the solution involves several novel and non-trivial ideas. Despite this complexity, the main paper is nicely written and achieves a great balance between intuition and technical details. I vote for accepting this paper.\n\nWEAKNESSES:\nThe algorithm only handles the homogeneous halfspaces without a bias term, and cannot achieve the $\\mathsf{opt} + \\epsilon$ error guarantee as in the prior work of [RV22, GKK22].\n\nQUESTIONS:\nFollowing up on the weakness part, some discussion on whether/how the current technique could handle non-homogeneous halfspaces and/or the stronger agnostic learning guarantee would be helpful.\n\nLIMITATIONS:\nThis work is purely theoretical and its limitations are in the assumptions (e.g., the concept class is the class of homogeneous halfspaces, and the \"reference distribution\" is a spherical Gaussian). This has been properly addressed in the paper (clearly stated in the abstract and the main paper).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearning halfspaces is a very well studied problem in machine learning. In the agnostic (adversarial label noise) and distribution free setting, this problem has been known to be computationally intractable. As a result, there have been several works of agnostic learning in distribution specific settings (where the marginal distribution belongs to a particular family of distributions, say Gaussian or log-concave). In this scenario, the learner has an error of the form $OPT + \\epsilon$, where OPT denotes the optimal 0-1 error. This however has complexity $d^{1/\\epsilon^2}$, and the exponential dependency on $1/\\eps$ is tight. This motivates the designing of learning algorithms that have better sample complexity with respect to $1/\\epsilon$, whereas the error becomes $f(OPT) + \\eps$ for some function f.\n\nIn this work, the authors studied this problem in the newly introduced Testable learning framework  by Rubinfeld and Vasilyan (STOC 23). Here the goal is if the tester accepts, then with high probability the output of the learning is close to some function of OPT, and if the data satisfies the distributional assumptions, the algorithm accepts.\n\nIn this paper, the authors study the class of homogeneous halfspaces over $R^d$ with Gaussian marginals. This work designs a tester-learner with sample complexity $N=poly(d, 1/\\epsilon)$ and runtime $poly(dN)$. Additionally, their algorithm also works in the active learning framework where the algorithm only queries the labels of a subset of the samples.\n\nThe authors first design a weak agnostic learner with respect to Gaussian distribution which checks if the low degree moments of the marginal distribution approximately matches with that of Gaussian distribution (Section 2). The idea is if the moments are close, then the vector corresponding to the degree-1 chow parameters will not be far from v^* (the true vector of the half-space), see Lemma 2.3. They also design an algorithm that either reports if the marginal distribution is not Gaussian or it outputs a unit vector w (Algorithm 2 corresponding to Proposition 2.1)  Once they have the weak learner, the authors will try to find a vector that has small 0-1 error with v^* by calling Algorithm 2 (Lemma 2.1) in an iterative manner. After logarithmic many iterations, they show that the probability mass of the disagreement region is bounded (Lemma 3.1). However, this process gives a collection of vectors such that one of them is close to the optimal vector v^*.The authors finally run a tournament among these candidate halfspaces corresponding to these vectors and outputs the winner hypothesis (Lemma 3.6).\n\nSTRENGTHS:\nThe paper gives the first algorithm for testable learning of halfspaces that runs in poly(d, epsilon). The algorithm is very nice.\nWith the complexity pulled down drastically, a proper implementation and experimental results for this algorithm would be possible and it would be nice to see the relevance of the concept of testable learning in various applications.\n\nWEAKNESSES:\nThe usefulness of the testable learning model in real life applications is yet to be understood.  The paper can only handle adversarial noise.\n\nQUESTIONS:\nCan this approach be used to design tester-learners for function classes other than halfspaces.\n\nLIMITATIONS:\nIt is a pure theoretical work in the paradigm of testable learning - a relatively new concept whose importance is not yet fully confirmed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers computationally efficient learning of halfspaces with adversarial noise in a recently proposed \"testable\" task: with high probability either reports the example distribution is not standard Guaissian, or outputs a halfspace with error of at most O(opt)+epsilon. There are existing works in a similar setting but targets the error of opt+epsilon, so their time and sample complexity is unavoidably $d^{O(1/\\epsilon^2)}$. This paper relaxed the error target to O(opt)+epsilon as in previous non-testable works, and shows that the testable task can be achieved with time and sample complexity of around poly(d, 1/epsilon) and label complexity of around d log(1/epsilon).\n\nTechnically, the results are obtained by (non-trivially) combining known ideas, like moment matching to assist testing, the Chow parameter (roughly speaking, E[yx]), robust mean estimation, and soft localization for learning in a label-efficient way.\n\n--\n\nI've read the author's response and I remain positive for this paper.\n\nSTRENGTHS:\n- This paper considers a recently proposed task of efficient testable learning of halfspaces with Gaussian distribution, and shows that, similar to the results in classical non-testable setting, it can be done with poly(d/epsilon) when considering constant-factor approximation. In my opinion, this result is not very surprising, but it is still interesting to know.\n\n- Most of the individual ideas used for the proof and algorithm are known to the community. On the other hand, the careful analysis to combine them in this new testable learning setting is still novel and nontrivial.\n\n- I briefly checked the proofs and they looked sound to me.\n\n- The paper is written very clearly: it clearly explains the problem and how it relates to existing work, and it organizes its technical part well so that one could follow both the high level steps and details.\n\nWEAKNESSES:\nThere are some weakensses (and I mentioned some of them above already), but I don't think they're really major:\n\n- The results are not very surprising to me, and the individual ideas used are mostly known.\n\n- It only considers standard Gaussian distributions and adversarial noise.\n\nQUESTIONS:\nI don't really have any questions, but one minor comment is that since this is a relatively new setting, it might help readers understand the difficulty of the problem by explaining that one cannot simply first test if the distribution is Gaussian and then proceed with standard learning algorithms.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of efficient testable learning of half spaces under Gaussian marginals and adversarial noise. The setting of testable learning addresses the issue of restrictive assumptions on the marginal distributions (such as Gaussianity) under learning algorithms are designed. Traditionally, a learning algorithm has no guarantee when the marginal distribution is not the prescribed one. In tester-learner setting, a learning algorithm consists of a tester and a learner. The tester \"certifies\" that the marginal distribution satisfies certain conditions (ideally this test would be a test of whether the marginal distribution is \"close\" to the original distribution). Conditioned on the tester passing, the learning algorithm has the desired correctness guarantee. The main idea is that tester does not need to test for distributional closeness but rather just closeness sufficient to \"fool\" the learning algorithm.\n\nA recent line of work has considered the learning of half spaces in this setting. Previous work show that the halfspaces can be agnostically learnt under Gaussian marginals in time $d^{1/\\eps^2}$ which is optimal if one insists on algorithms that return hypotheses that have error $OPT + \\eps$. The present paper relaxes this notion to require that the hypothesis returned only has error $O(OPT) + \\eps$. The paper presents a polynomial (in $d$ and $1/\\eps$) time tester-learner algorithm under Gaussian marginals.\n\nSTRENGTHS:\nThe paper presents a natural and interesting adaptation of the localization technique for learning half spaces in the testable learning setting. The main idea is that learning algorithms designed to work under Gaussian marginals require significantly less information (for example only rely on moments and anti concentration). The paper analyses a robust version of Chow parameter recovery under the condition that the marginals match Gaussian marginals (this can be tested efficiently). Using this as a weak learner, the paper uses a localization technique to convert a weak learner (from Chow parameter recovery) to a strong learner. Again the paper uses the fact that closeness of parameter and loss for halfspaces can be efficiently tested. In summary, the paper presents a nice analysis of learning algorithm\n\nWEAKNESSES:\nAs presented the paper is too specific to Gaussian marginals.\n\nQUESTIONS:\n- It would be nice to address which parts of the paper can be generalized to more general distributions? For example, what notions of testable concentration and anti concentrations would suffice for these algorithms to work. For example, would having sufficient moment decay suffice for the Chow parameter recovery?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}