Title: Tester-Learners for Halfspaces: Universal Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: testable learning

pac learning

agnostic learning

Massart label noise

adversarial label noise

distribution testing
Abstract: We give the first tester-learner for halfspaces that succeeds universally over a wide class of structured distributions. Our universal tester-learner runs in fully polynomial time and has the following guarantee: the learner achieves error $O(\mathrm{opt}) + \epsilon$ on any labeled distribution that the tester accepts, and moreover, the tester accepts whenever the marginal is any distribution that satisfies a Poincare inequality. In contrast to prior work on testable learning, our tester is not tailored to any single target distribution but rather succeeds for an entire target class of distributions. The class of Poincare distributions includes all strongly log-concave distributions, and, assuming the Kannan--Lovasz--Simonovits (KLS) conjecture, includes all log-concave distributions. In the special case where the label noise is known to be Massart, our tester-learner achieves error $\mathrm{opt} + \epsilon$ while accepting all log-concave distributions unconditionally (without assuming KLS).
Our tests rely on checking hypercontractivity of the unknown distribution using a sum-of-squares (SOS) program, and crucially make use of the fact that Poincare distributions are certifiably hypercontractive in the SOS framework.

Tester-Learners for Halfspaces: Universal Algorithms

Abstract

We give the first tester-learner for halfspaces that succeeds universally over a wide class of structured distributions. Our universal tester-learner runs in fully polynomial time and has the following guarantee: the learner achieves error O ( opt ) + ϵ on any labeled distribution that the tester accepts, and moreover, the tester accepts whenever the marginal is any distribution that satisfies a Poincaré inequality. In contrast to prior work on testable learning, our tester is not tailored to any single target distribution but rather succeeds for an entire target class of distributions. The class of Poincaré distributions includes all strongly log-concave distributions, and, assuming the Kannan-Lóvasz-Simonovits (KLS) conjecture, includes all log-concave distributions. In the special case where the label noise is known to be Massart, our tester-learner achieves error opt + ϵ while accepting all log-concave distributions unconditionally (without assuming KLS). Our tests rely on checking hypercontractivity of the unknown distribution using a sum-of-squares (SOS) program, and crucially make use of the fact that Poincaré distributions are certifiably hypercontractive in the SOS framework.

1 Introduction

In this paper we study the recent model of testable learning, due to Rubinfeld and Vasilyan [RV23]. Testable learning addresses a key issue with essentially all known algorithms for the basic problem of agnostic learning, in which a learner is required to produce a hypothesis competitive with the best-fitting hypothesis in a concept class C . The issue is that these algorithms make distributional assumptions (such as Gaussianity or log-concavity) that are in general hard to verify. This means that in the absence of any prior information about the distribution or the optimal achievable error, it can be hard to check that the learner has even succeeded at meeting its guarantee.

In the testable learning model, the learning algorithm, or tester-learner, is given access to labeled examples from an unknown distribution and may either reject or accept the unknown distribution. If it accepts, it must successfully produce a near-optimal hypothesis. Moreover, it is also required to accept whenever the unknown distribution truly has a certain target marginal D ∗ . Work of [RV23, GKK23, GKSV23, DKK + 23] provided tester-learners for a range of basic classes (including halfspaces, intersections of halfspaces, and more) with respect to particular target marginals D ∗ (such as the standard Gaussian). All of these algorithms, however, have the shortcoming that they are closely tailored to the particular target marginal D ∗ that is chosen. Indeed, their tests would reject many well-behaved distributions that are appreciably far from D ∗ . A highly natural question from both a theoretical and a practical perspective is: can we design tester-learners that accept a wide class of distributions simultaneously, without being tailored to any particular one?

Konstantinos Stavropoulos

In this work we answer this question in the affirmative by introducing and studying universally testable learning . We formally define this model as follows.

Definition 1.1 (Universally Testable Learning) . Let C be a concept class mapping R d to {± 1 } . Let D be a family of distributions over R d . Let ϵ, δ > 0 be parameters, and let ψ : [0 , 1] → [0 , 1] be some function. We say C can be universally testably learned w.r.t. D up to error ψ ( opt ) + ϵ with failure probability δ if there exists a tester-learner A meeting the following specification. For any distribution D XY on R d ×{± 1 } , A takes in a large sample S drawn from D XY , and either rejects S or accepts and produces a hypothesis h : R d →{± 1 } such that:

(a) (Soundness.) With probability at least 1 -δ over the sample S the following is true:

̸

If A accepts, then the output h satisfies P ( x ,y ) ∼ D XY [ h ( x ) = y ] ≤ ψ ( opt ( C , D XY )) + ϵ , where opt ( C , D XY ) = inf f ∈C P ( x ,y ) ∼ D XY [ h ( x ) = y ] .

̸

(b) (Completeness.) Whenever the marginal of D XY lies within D , A accepts with probability at least 1 -δ over the sample S .

In this terminology, the original definition of testable learning reduces to the special case where D = { D ∗ } . We stress that while the prior work of [GKK23] allowed D ∗ to be, say, any fixed strongly log-concave distribution, their tester-learners are still tailored to the particular D ∗ that is selected. This is because their tests rely on checking that the unknown distribution closely matches moments with D ∗ . By contrast, a universal tester-learner must accept all marginals in a family D .

Our main contribution in this paper is the first universal tester-learner for the class of halfspaces with respect to a broad family of structured continuous distributions. This family is the set of all distributions with bounded Poincaré constant (see Definition 2.4) and some mild concentration and anti-concentration properties (see Definition 2.1). It captures all strongly log-concave distributions, and in fact, under the well-known Kannan-Lóvasz-Simonovits (KLS) conjecture (see Conjecture 2.6), it captures all log-concave distributions as well. Our universal tester-learner significantly generalizes the main result of [GKSV23], who showed comparable guarantees only for the case where the target marginal is the standard Gaussian.

Theorem 1.2 (Universal Tester-Learner for Halfspaces; formally stated as Theorem 4.1) . Let C be the class of origin-centered halfspaces over R d . Let D be the class of Θ(1) -nice and Θ(1) -Poincaré distributions (see Definitions 2.1 and 2.4), which includes all isotropic strongly log-concave and, under KLS, all isotropic log-concave distributions. Then C can be universally testably learned w.r.t. D up to error O ( opt ) + ϵ in poly( d, 1 ϵ ) time and sample complexity.

A special and well-studied case of interest is when the label noise follows the Massart model, i.e. the label of every example is flipped by an adversary with probability at most η . In this case we are able to handle a considerably larger class D while also providing a stronger guarantee.

Theorem 1.3 (Universal Tester-Learner for Massart Halfspaces; formally stated as Theorem 4.1) . Let C be the class of origin-centered halfspaces over R d . Let D be the class of poly( d ) -nice and poly( d ) -Poincaré distributions, which includes all isotropic log-concave distributions (unconditionally). Suppose the label noise follows the Massart model with noise rate at most η < 1 2 . Then C can be universally testably learned w.r.t. D up to error opt + ϵ in poly( d, 1 ϵ , 1 1 -2 η ) time and sample complexity.

Technical Overview. We first describe the key reasons why prior tester-learners were tailored to a specific target D ∗ . All known polynomial-time algorithms for agnostically learning halfspaces up to error O ( opt )+ ϵ require some concentration and anti-concentration properties from the input marginal distribution (encapsulated e.g. in Definition 2.1). While concentration is relatively straightforward to check (e.g. by checking that the moments do not grow at too fast a rate), the key challenge in designing tester-learners for halfspaces is to check anti-concentration. All prior tester-learners [RV23, GKK23, GKSV23, DKK + 23] use the heavy machinery of moment-matching to achieve this. This approach relies on establishing structural properties of the following type: if D ∗ is a wellbehaved distribution (e.g. a strongly log-concave distribution), and D approximately matches D ∗ in its low-degree moments, then D is also well-behaved (in particular, anti-concentrated). A canonical statement of such a property is the main pseudorandomness result of [GKK23] (see Theorem 5.6 therein), which establishes that approximate moment-matching fools functions of a constant number of halfspaces. Applying this property inherently requires comparing the low-degree moments of

D with those of D ∗ . Such tests do (implicitly) succeed universally for the class of all distributions that match low-degree moments with D ∗ (e.g., if D ∗ is the uniform distribution over the hypercube, moment matching would accept all k -wise independent distributions). Definition 1.1, however, seeks a far broader kind of universality. Our tests are not tailored to a single target in any way, and are intended to succeed over practical classes of distributions that are commonly considered in learning theory (e.g., log-concave distributions). 1

We overcome this hurdle and design a conceptually simple way of checking anti-concentration without requiring the hammer of moment-matching. Our approach follows and improves on the roadmap used by [GKSV23] to design efficient tester-learners for halfspaces using non-convex SGD (building on [DKTZ20a, DKTZ20b]). Let us outline this approach at a high level (a more detailed technical overview may be found in [GKSV23, Sec 1.2]). The tester-learner first computes a stationary point w of a certain smooth version of the ramp loss, a surrogate for the 0-1 loss. Let w ∗ be any solution achieving 0-1 error opt . The tester-learner now checks distributional properties of the unknown marginal D that ensure that w is close in angular distance to w ∗ (specifically, they ensure the contrapositive, namely that any w that has large gradient norm must have large angle with w ∗ ). By a more careful analysis of the gradient norm than in [GKSV23] (see Proposition 4.2), we are able to reduce to showing the following weak anti-concentration property. Let v denote any unit vector orthogonal to w , and let D T denote D restricted to the band T = { x | |⟨ w , x ⟩| ≤ σ } (where the width σ is carefully selected according to certain constraints). Then the property we need is that

Our key observation is that the classical Paley-Zygmund inequality applied to the random variable Z = ⟨ v , x ⟩ 2 , where x ∼ D T , already gives us the following type of anti-concentration:

This turns out to suffice for our purposes-provided we can show a hypercontractivity property for Z , namely that E [ Z 2 ] ≤ Θ(1) E [ Z ] 2 (as well as that E [ Z ] = Θ(1) , which is just a second moment constraint).

Our main algorithmic idea is to use a sum-of-squares (SOS) program to check hypercontractivity of the random variable Z . To do so, we crucially leverage a result due to [KS17] stating that any D that has bounded Poincaré constant is certifiably hypercontractive in the SOS framework (and it turns out this extends to D T as well). This means that we can run a certain polynomial-time semidefinite program that checks hypercontractivity of Z over the sample, and whenever D is in fact Poincaré, we are guaranteed that the test will pass with high probability (see Proposition 3.5). This is sufficient to ensure that the stationary point w we have computed is indeed close in angular distance to w ∗ .

In order to finally arrive at our main results, we need to run further tests which ensure that the disagreement between our computed w and any (unknown) optimum w ∗ is bounded by the angle between them, i.e., P x ∼ D [sign( ⟨ w , x ⟩ ̸ = sign( ⟨ w ∗ , x ⟩ )] ≤ O ( ∡ ( w , w ∗ )) (see Lemma 3.1). This in turn guarantees that w has error O ( opt ) + ϵ . We stress that while [GKSV23] introduced similar testers for the special case of Gaussian marginals, our tests succeed universally with respect to a broad family of distributions including some heavy-tailed distributions (see Definition 2.1). From a technical perspective, prior to our work, such tests either produced a suboptimal bound, or required estimating the operator norms of a polynomial number of random matrices formed using rejection sampling. We significantly simplify this approach by showing that it is sufficient to estimate the operator norm of a single random matrix. Finally, to obtain our improved results for the Massart setting, it turns out that the proof admits certain simplifications that guarantee final error opt + ϵ while also allowing a wider range of Poincaré distributions.

Related Work. There is a large body of work on agnostic learning algorithms for halfspaces that run in fully polynomial time. We briefly mention only those that are most closely relevant to our work; please see [BH21] for a survey as well as [GKSV23, Sec 1.1] for further related work. Following a

long line of work on distribution-specific agnostic learners for halfspaces [KLS09, ABL17, Dan15, BZ17, YZ17, Zha18, ZSA20, ZL21], the work of [DKTZ20a] introduced a particularly simple approach for the Massart setting, based solely on non-convex SGD. This work, which sets the template that our approach also follows, achieved the information-theoretically optimal error of opt + ϵ for origin-centered Massart halfspaces over a wide range of structured distributions (and was later extended to general halfspaces by [DKK + 22]). The non-convex SGD approach was then generalized by [DKTZ20b] to show an O ( opt ) + ϵ guarantee for the fully agnostic setting.

The testable learning model was introduced by the work of [RV23], who showed a tester-learner for halfspaces achieving error opt + ϵ in time d ˜ O (1 /ϵ 4 ) for the case where the target marginal is Gaussian. Subsequently, [GKK23] provided a general algorithmic framework based on moment-matching for this problem, and showed a tester-learner for halfspaces only requiring time d ˜ O (1 /ϵ 2 ) with respect to any fixed strongly log-concave marginal (matching known lower bounds for ordinary agnostic learning over Gaussian marginals [GGK20, DKZ20, DKPZ21, DKR23]).

The most closely relevant work to the present one is that of [GKSV23] (see also [DKK + 23]), who showed fully polynomial-time tester-learners for halfspaces achieving error O ( opt )+ ϵ in the agnostic setting and opt + ϵ in the Massart setting for the case where the target marginal is the Gaussian. As detailed in the technical overview, their tests rely crucially on moment-matching and are tailored to a specific target marginal. By contrast, our tests check hypercontractivity using an SOS program and succeed universally for a wide class of certifiably hypercontractive distributions.

Certifying distributional properties such as hypercontractivity is an important aspect of a large body of work on robust algorithmic statistics using the SOS framework. We will not attempt to summarize this literature here and direct the reader to [KS17, BK21] for overviews of related work, as well as to [FKP + 19] for a textbook treatment. The notion of certifiable anti-concentration has also been studied (see e.g. [KKK19, RY20, BK21]), but it turns out not to be directly useful for our purposes as it is only known to hold for distributions satisfying very strong conditions such as rotational symmetry.

Limitations and Further Work. Open directions in testable learning (and universally testable learning) include the design of (efficient) tester-learners for concept classes other than the class of halfspaces, e.g., functions of halfspaces or neurons with other activations (like ReLU or sigmoid).

2 Preliminaries

̸

Notation and Terminology. For what follows, we consider D XY to be an unknown joint distribution over X × Y from which we receive independent samples, and its marginal on X will be denoted by D X . In particular X = R d , and labels will lie in Y = {± 1 } . We will use C to denote a concept class mapping R d to {± 1 } , which throughout this paper will be the class of halfspaces or functions of halfspaces over R d . We use opt ( C , D XY ) to denote the optimal error inf f ∈C P ( x ,y ) ∼ D XY [ f ( x ) = y ] , or just opt when C and D XY are clear from context. We recall that in Massart noise model, the labels satisfy P y ∼ D XY | x [ y = sign( ⟨ w ∗ , x ⟩ ) | x ] = η ( x ) , with η ( x ) ≤ η < 1 2 for all x . When we have adversarial noise (i.e., when we are in the agnostic model), the labels can be completely arbitrary. In both cases, the goal is to produce a hypothesis whose error is competitive with opt . We use E to denote the expectation of a random variable in brackets (or, correspondingly, P for the probability of an event), either over the unknown joint distribution or over the empirical distribution with respect to a sample S (e.g., E Z ∈ S [ f ( Z )] = 1 | S | ∑ Z ∈ S f ( Z ) ).

Definitions and Distributional Assumptions. For the problem of learning halfspaces in the agnostic and in Massart noise models, any of the known polynomial algorithms that achieve computationally optimal guarantees require that the marginal distribution has at least the following nice properties previously defined by, e.g., [DKTZ20b].

Definition 2.1 (Nice Distributions) . For a given constant λ ≥ 1 , we consider the class of λ -nice distributions over R d to be the distributions that satisfy the following properties:

For any unit vector v in R d the distribution satisfies E [ ⟨ v , x ⟩ 2 ] ∈ [ 1 λ , λ ] .(bounded spectrum)

For any two dimensional subspace V , the corresponding marginal density q V ( x ) satisfies q V ( x ) ≥ 1 /λ for any ∥ x ∥ 2 ≤ 1 /λ . (anti-anti-concentration)

̸

For any two dimensional subspace V , the corresponding marginal density q V ( x ) satisfies q V ( x ) ≤ Q ( ∥ x ∥ 2 ) for some function Q : R + → R + such that sup r ≥ 0 Q ( r ) ≤ λ and also ∫ ∞ r =0 r k Q ( r ) dr ≤ λ , for any k = 1 , 3 , 5 . (anti-concentration and concentration)

In the testable learning framework, however, corresponding results provide testable guarantees with respect to target marginals that are isotropic strongly log-concave [GKSV23], which is a strictly stronger condition than the one of Definition 2.1 (see Proposition 2.3 below). We now provide the standard definition of (strongly) log-concave distributions.

Definition 2.2 ((Strongly) Log-Concave Distributions [SW14]) . We say that a distribution over R d is ( β -strongly) log-concave, if its density can be written as e -φ , where φ is a ( β -strongly) convex function on R d (for some β > 0 ).

Proposition 2.3 (Log-Concave Distributions are Nice [LV07]) . There exists a universal constant λ ≥ 1 such that any isotropic log-concave distribution is λ -nice.

In this work, we provide universally testable guarantees with respect to the class of nice distributions with bounded Poincaré constant (see Definition 2.4 below).

Definition 2.4 (Poincaré Distributions) . For a given value γ > 0 , we say that a distribution over R d is γ -Poincaré, if Var( f ( x )) ≤ γ · E [ ∥∇ f ( x ) ∥ 2 2 ] for any differentiable function f : R d → R .

Although it is not clear whether one can efficiently obtain testable guarantees for the problem of learning noisy halfspaces under nice marginals (which is known to be an efficiently solvable problem in the non-testable setting [DKTZ20a, DKTZ20b]), by restricting our attention to nice distributions that, additionally, have bounded Poincaré constant, we obtain efficient learning results, even in the universally testable setting. Our results are strictly stronger than the ones in [GKSV23], since we capture isotropic strongly log-concave distributions universally, due to Proposition 2.3 and the fact that strongly log-concave distributions are also Poincaré, as per Proposition 2.5 below.

Proposition 2.5 (Strongly Log-Concave Distributions are Poincaré, [SW14, Proposition 10.1]) . Any 1 γ -strongly log-concave distribution is γ -Poincaré.

Furthermore, under a long-standing conjecture about the geometry of convex bodies [KLS95], our results capture the family of all isotropic log-concave distributions.

Conjecture 2.6 (Kannan-Lovász-Simonovits Conjecture [KLS95] reformulation from [LV18]) . There is a universal constant γ > 0 for which any isotropic log-concave distribution is γ -Poincaré.

3 Universal Testers

In this section, we present two basic testers that constitute the basic building blocks of the universal tester-learners we provide in the next section. The testers in this section might be of independent interest and their appeal is that they succeed even when the distribution in their input is unspecified up to certain bounds on a number of its statistics. In fact, the family of distributions for which each such tester succeeds is of infinite size, even non-parametric.

3.1 Universal Tester for Bounding Local Halfspace Disagreement

First, we present a universal tester that checks, given a parameter vector w , whether a set of samples S is such that bounding the angular distance of w from an optimum parameter vector, implies that the corresponding halfspace disagrees with the (unknown) optimum halfspace only on a bounded fraction of points in S . This property ensures that if w is close to the optimum parameter vector, then it is also an approximate empirical risk minimizer. The tester universally accepts samples from nice distributions with high probability (Definition 2.1).

Lemma 3.1 (Universally Testable Bound for Local Halfspace Disagreement) . Let D XY be a distribution over R d ×{± 1 } , w ∈ S d -1 , θ ∈ (0 , π/ 4] , λ ≥ 1 and δ ∈ (0 , 1) . Then, for a sufficiently large constant C , there is a tester that given δ , θ , w and a set S of samples from D X with size at least C · ( d 4 θ 2 δ ) , runs in time poly ( d, 1 θ , 1 δ ) and satisfies the following specifications:

( a ) If the tester accepts S , then for every unit vector w ′ ∈ R n satisfying ∡ ( w , w ′ ) ≤ θ we have P [sign( ⟨ w ′ , x ⟩ ) = sign( ⟨ w , x ⟩ )] ≤ C · θ · λ C

̸

( b ) If the distribution D X is λ -nice, the tester accepts S with probability 1 -δ .

The proof of Lemma 3.1 simplifies and improves the proof of a similar but weaker result in [GKSV23] (see their Proposition 4.5). The initial tester exploited the observation that the probability of disagreement between two halfspaces can be upper bounded by a sum of products, where each product has two terms: one corresponding to the probability of falling in a (known) strip orthogonal to w and one corresponding to the probability of having large enough inner product with some unknown vector orthogonal to w , conditioned in the (known) strip. The first term can be controlled by estimating the probability of falling in a (known) strip, while the second follows by Chebyshev's inequality, after estimating the largest eigenvalue of the covariance matrix conditioned in the known strip. This approach introduces a number of complications, including the fact that conditioning requires rejection sampling, which, in turn requires a lower bound on the probability of falling inside each strip. We propose a simpler tester that controls all of the terms of the sum simultaneously by estimating the largest eigenvalue of a single covariance matrix (without conditioning). Upper and lower bounds on the eigenvalues of random symmetric matrices can be universally tested with testers that are guaranteed to accept when the elements of the matrix have bounded second moments (spectral tester of Proposition A.2). We present our full proof in Appendix B.1.

3.2 Universally Testable Weak Anti-Concentration

We now provide an important universal tester, which ensures that for a given vector w , a sample set S and any unknown unit vector v orthogonal to w , among the samples falling within a (known) strip orthogonal to w , at least a constant fraction is absolutely correlated with v by a constant. In other words, the tester ensures that the conditional empirical distribution is weakly anti-concentrated in every direction. The tester universally accepts nice distributions that have bounded Poincaré constant.

Lemma 3.2 (Universally Testable Weak Anti-Concentration) . Let D be a distribution over R d . Then, there is a universal constant C > 0 and a tester that given a unit vector w ∈ R d , δ ∈ (0 , 1) , γ > 0 , λ ≥ 1 , σ ≤ 1 2 λ and a set S of i.i.d. samples from D with size at least C · d 4 σ 2 δ log( d ) λ C , runs in time poly( d, λ, 1 σ , 1 δ , log( 1 γ )) and satisfies the following specifications

( a ) If the tester accepts S , then for any unit vector v ∈ R d with ⟨ v , w ⟩ = 0 we have

( b ) If D is γ -Poincaré and λ -nice, then the tester accepts S with probability at least 1 -δ .

The proof of Lemma 3.2 is based on a simple fact from probability that is true for any non-negative random variable and ensures that the mass assigned to the tails is lower bounded by the ratio of the square of its expectation to the second moment.

Proposition 3.3 (Paley-Zygmund Inequality) . For any non-negative random variable Z , we have

In the special case where Z follows the distribution of ⟨ v , x ⟩ 2 conditioned on |⟨ w , x ⟩| ≤ σ for some unitary orthogonal vectors v , w , some σ > 0 and some random variable x whose distribution is, say, 1 -nice (see Definition 2.1), one can show that E [ Z ] is lower bounded by a constant and E [ Z 2 ] is upper bounded by another constant, so Z assigns a non-trivial mass to a set that is bounded away from zero. This property is useful in the context of learning noisy halfspaces, as we show in the following section (see Proposition 4.2 and Lemma 4.3). However, testing algorithms that check whether such a property holds for given w and σ , are guaranteed to succeed when the marginal distribution has, additionally, bounded Poincaré constant. The main part of the proof that requires a bounded Poincaré constant, is testing whether E [ Z 2 ] is bounded uniformly over the set of unit vectors v orthogonal to w , since Z 2 = ⟨ v , x ⟩ 4 , where v is unknown. We use the following result from [KS17].

Proposition 3.4 (Certifiable Hypercontractivity of Poincaré Distributions, Theorem 4.1 in [KS17]) . Let δ ∈ (0 , 1) , γ > 0 and let D be a γ -Poincaré distribution over R d . Let S be a set of independent samples from D with size at least (2 d log(4 d/δ )) 4 . Consider the constrained maximization problem

Then, the optimum solution of the degree-4 sum-of-squares relaxation of the problem (3.1) has value at most Cγ 4 for some universal constant C , with probability at least 1 -δ over the sample S .

Using Proposition 3.4, we are able to provide a universal tester for bounding the empirical fourth moments. The tester solves an appropriate SDP relaxation of the (hard) problem [HL13] of finding the direction with maximum fourth moment and is guaranteed to succeed if x has Poincaré parameter bounded by a known value.

Proposition 3.5 (Hypercontractivity Tester) . Let D be a distribution over R d . Then, there is a tester that given δ ∈ (0 , 1) , γ > 0 and a set S of i.i.d. samples from D with size at least (2 d log(4 d/δ )) 4 , runs in time poly( d, log 1 δ , log 1 γ ) and satisfies the following specifications

( a ) If the tester accepts S , then for any unit vector v ∈ R d we have

( b ) If the distribution D is γ -Poincaré, then the tester accepts S with probability at least 1 -δ .

Proof. The tester does the following:

1. Solves a degree4 sum-of-squares relaxation of problem (3.1) up to accuracy γ 4 . (For a formal definition of the relaxed problem, see Problem (2.3) in [KS17].)

If the solution has value larger than ( C -1) γ 4 , then reject . Otherwise accept .

The computational complexity of the tester is poly( | S | , d, log 1 γ ) , since the problem it solves can be written as a semidefinite program [Sho87, Par00, Nes00, Las01].

If the tester accepts S , then we know that the optimal solution of the relaxed problem is at most Cγ 4 and we also know that any solution of the initial problem (3.1) has value at most equal to the value of the relaxation. Therefore E [ ⟨ v , x ⟩ 4 ] ≤ Cγ 4 , for any v ∈ S d -1 .

On the other hand, if the true distribution D is γ -Poincaré, then, with probability at least 1 -δ , we have that the solution found in step 3.2 has, with probability at least 1 -δ , value at most C ′ γ 4 for some universal constant C ′ , due to Proposition 3.4. In order to ensure that the tester will accept with probability at least 1 -δ , it suffices to pick C = C ′ +1 .

We provide the full proof of Lemma 3.2, in Appendix B.2. The tests we perform include a spectral tester that accepts with high probability when the distribution of x is nice (similar to the spectral tester used for Lemma 3.1), a tester of the probability that |⟨ w , x ⟩| ≤ σ and the hypercontractivity tester of Proposition 3.5.

4 Universal Tester-Learners for Halfspaces

In this section, we present our main result on universally testable learning of halfspaces.

̸

Theorem 4.1 (Efficient Universal Tester-Learner for Halfspaces) . Let D XY be any distribution over R d ×{± 1 } . Let C be the class of origin centered halfspaces in R d . Then, for any λ ≥ 1 , γ > 0 , ϵ > 0 and δ ∈ (0 , 1) , there exists an universal tester-learner for C w.r.t. the class of λ -nice and γ -Poincaré marginals up to error poly( λ ) · (1+ γ 4 ) · opt + ϵ , where opt = min w ∈ S d -1 P D XY [ y = sign( ⟨ w , x ⟩ )] , and error probability at most δ , using a number of samples and running time poly( d, λ, γ, 1 ϵ , log 1 δ ) .

Moreover, if the noise is Massart with given rate η < 1 / 2 , then the algorithm achieves error opt + ϵ with time and sample complexity poly( d, λ, γ, 1 ϵ , 1 1 -2 η , log 1 δ ) .

Our proof follows a surrogate loss minimization approach that has been used for classical learning of noisy halfspaces [DKTZ20a, DKTZ20b] as well as classical (non-universal) testable learning [GKSV23]. In particular, the algorithm runs Projected Stochastic Gradient Descent (see A.5) on a surrogate loss whose stationary points are shown to be close to optimum parameter vectors under certain distributional assumptions. In the regular testable learning setting, given a stationary point, the above property can be tested with respect to any (fixed and known) target strongly log-concave marginal as shown by [GKSV23]. For such a stationary point, more tests are used in order to ensure

bounds on local halfspace disagreement. We provide some delicate refinements of the proofs in [GKSV23] that enable us to substitute their testers with the universal testers of Section 3.

We use the following surrogate loss function which was also used in [GKSV23].

In Equation (4.1), the function ℓ σ is a smoothed version of the step function as in Proposition A.4.

In order to analyze the properties of the stationary points of the surrogate loss, we provide the following refinement of results implicit in [GKSV23, DKTZ20a, DKTZ20b]. We show that the gradient of the surrogate loss is lower bounded by the difference between certain quantities that are controlled by the marginal distribution (see Figure 1). We stress that we do not use any assumptions for the marginal distribution in this step. Prior work included similar bounds, but the corresponding quantities were different. We need to be more precise and provide the following result, whose proof is based on two dimensional geometry and can be found in Appendix C.1.

Proposition 4.2 (Modification from [GKSV23, DKTZ20a, DKTZ20b]) . For a distribution D XY over R d ×{± 1 } let opt be the minimum error achieved by some origin-centered halfspace and w ∗ ∈ S d -1 a corresponding vector. Consider L σ as in Equation (4.1) for σ > 0 and let η < 1 / 2 . Let w ∈ S d -1 with ∡ ( w , w ∗ ) = θ < π 2 and v ∈ span( w , w ∗ ) such that ⟨ v , w ⟩ = 0 and ⟨ v , w ∗ ⟩ < 0 . Then, for some universal constant C > 0 and any α ≥ σ 2 tan θ we have ∥∇ w L σ ( w ; D XY ) ∥ 2 ≥ A 1 -A 2 -A 3 , where

A 1 = α C · σ · P [ |⟨ v , x ⟩| ≥ α and |⟨ w , x ⟩| ≤ σ 6 ] A 2 = C tan θ · P [ |⟨ w , x ⟩| ≤ σ 2 ] and A 3 = C σ · √ opt · √ E [ ⟨ v , x ⟩ 2 · ✶ {|⟨ w , x ⟩|≤ σ 2 } ] Moreover, if the noise is Massart with rate η , then ∥∇ w L σ ( w ; D XY ) ∥ 2 ≥ (1 -2 η ) A 1 -A 2 .

If the marginal distribution is nice, then the quantities A 1 , A 2 and A 3 are such that σ can be chosen accordingly so that stationary points of the surrogate loss (or their inverses) are close to some optimum vector (see Proposition A.3 for properties of nice distributions). We use some simple tests (e.g., estimate the probability of falling in a strip, P [ |⟨ w , x ⟩| ≤ σ/ 2] and appropriate spectral testers) as well as our universal tester for weak anti-concentration (see 3.2) to establish bounds on quantities A 1 , A 2 and A 3 which ensure that the desired property holds for a given vector w , under no distributional assumptions. The tester in the following result universally accepts nice distributions with bounded Poincaré parameter. The formal proof can be found in Appendix C.2.

Lemma 4.3 (Universally Testable Structure of Surrogate Loss) . Let D XY be any distribution over R d ×{± 1 } . Consider L σ as in Equation (4.1) . Then, there is a universal constant C > 0 and a tester that given a unit vector w ∈ R d , δ ∈ (0 , 1) , η < 1 / 2 , γ > 0 , λ ≥ 1 , σ ≤ 1 Cλ C and a set S of i.i.d. samples from D XY with size at least C · d 4 σ 2 δ log( d ) λ C , runs in time poly( d, λ, 1 σ , 1 δ , log( 1 γ )) and satisfies the following specifications

( a ) If the tester accepts S , then, the following statements are true for the minimum error opt S achieved by some origin-centered halfspace on S and the optimum vector w ∗ S ∈ S d -1

If the noise is Massart with associated rate η and ∥∇ w L σ ( w ; S ) ∥ 2 ≤ 1 -2 η Cλ C γ 4 then either ∡ ( w , w ∗ S ) ≤ Cλ C (1+ γ 4 ) 1 -2 η · σ or ∡ ( -w , w ∗ S ) ≤ Cλ C (1+ γ 4 ) 1 -2 η · σ .

If the noise is adversarial with opt S ≤ σ Cλ C and ∥∇ w L σ ( w ; S ) ∥ 2 < 1 Cλ C γ 4 then either ∡ ( w , w ∗ S ) ≤ Cλ C (1 + γ 4 ) · σ or ∡ ( -w , w ∗ S ) ≤ Cλ C (1 + γ 4 ) · σ .

( b ) If the marginal D X is λ -nice and γ -Poincaré, then the tester accepts S with probability at least 1 -δ .

We now give the algorithm for δ ← 1 / 3 since we can reduce the probability of failure with repetition (repeat O (log 1 δ ) times, accept if the rate of acceptance is Ω(1) and output the halfspace achieving the minimum test error among the halfspaces returned).

The algorithm receives λ ≥ 1 , γ > 0 , ϵ > 0 and η ∈ (0 , 1 / 2) ∪ { 1 } (say η = 1 when we are in the agnostic case) and does the following for some appropriately large universal constants C 1 , C 2 > 0 .

First, initialize E = ϵ C 1 λ C 1 , and let Σ be a list of real numbers and A be a positive real number, where Σ and A are defined as follows. If η = 1 , then Σ is an E C 1 λ C 1 -cover of the interval [ 0 , 1 C 1 λ C 1 ] and A = 1 C 1 λ C 1 γ 4 . Otherwise, let Σ = { E · (1 -2 η ) C 1 λ C 1 (1+ γ 4 ) } and A = 1 -2 η C 1 λ C 1 γ 4 .

Draw a set S 1 of C 2 ( λd γϵ ) C 2 i.i.d. samples from D XY and run PSGD, as specified in Proposition A.5 with ϵ ← A , δ ← δ C 1 on the loss L σ for each σ ∈ Σ .

Form a list L with all the pairs of the form ( w , σ ) where w ∈ S d -1 is some iterate of the PSGD subroutine performed on L σ .

Draw a fresh set S 2 of C 2 ( λd γϵ ) C 2 i.i.d. samples from D XY and compute for each ( w , σ ) ∈ L the value ∥∇ w L σ ( w ; S 2 ) ∥ 2 . If, for some σ ∈ Σ , ∥∇ w L σ ( w ; S 2 ) ∥ 2 > A for all ( w , σ ) ∈ L , then reject .

Update L by keeping for each σ ∈ Σ only one pair of the form ( w , σ ) for which we have ∥∇ w L σ ( w ; S 2 ) ∥ 2 ≤ A .

Run the following tests for each ( w , σ ) ∈ L . (This will ensure that part (a) of Lemma 4.3 holds for each of the elements of L , i.e., that any stationary point of the loss L σ that lies in L is angularly close to the empirical risk minimizer 2 .).

If P ( x ,y ) ∈ S 2 [ |⟨ w , x ⟩| ≤ σ 6 ] ≤ σ C 1 λ C 1 or P ( x ,y ) ∈ S 2 [ |⟨ w , x ⟩| ≤ σ 2 ] > σ · C 1 λ C 1 , then reject .

Compute the ( d -1) × ( d -1) matrices M + S 2 and M -S 2 as follows: 3

Reject if the maximum singular value of M + S 2 is greater than σ · C 1 λ C 1 .

Reject if the minimum singular value of M -S 2 is less than σ C 1 λ C 1 .

Run the hypercontractivity tester on S ′ = { proj ⊥ w x : ( x , y ) ∈ S 2 and |⟨ w , x ⟩| ≤ σ } , i.e., solve an appropriate SDP (see Prop. 3.5 with γ ← γ , δ ← δ/C 1 ) and reject if the solution is larger than a specified threshold.

Set θ = (1+ γ 4 ) σ Aγ 4 , and run the following tests for each pair of the form ( w , σ ) and ( -w , σ ) where ( w , σ ) ∈ L . (This will ensure that part (a) of Lemma 3.1 is activated, i.e., that the distance of a vector from the empirical risk minimizer is an accurate proxy for the error of the corresponding halfspace.)

If P ( x ,y ) ∈ S 2 [ |⟨ w , x ⟩| ≤ θ ] > C 1 λ C 1 θ then reject .

Compute the ( d -1) × ( d -1) matrix M S 2 as follows: 4

If ∥ M S ∥ op > C 1 θλ C 1 , then reject .

Otherwise, accept and output the vector w that achieves the smallest empirical error on S 2 among the vectors in the list L .

This concludes the algorithm. The full proof of Theorem 4.1 may be found in Appendix C.3.

Acknowledgments and Disclosure of Funding

We wish to thank the anonymous reviewers of NeurIPS 2023 for their constructive feedback. Aravind Gollakota was at UT Austin while this work was done, supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Adam R. Klivans was supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Konstantinos Stavropoulos was supported by NSF award AF-1909204, the NSF AI Institute for Foundations of Machine Learning (IFML), and by scholarships from Bodossaki Foundation and Leventis Foundation. Arsen Vasilyan was supported in part by NSF awards CCF-2006664, DMS2022448, CCF-1565235, CCF-1955217, CCF-2310818, Big George Fellowship and Fintech@CSAIL. Part of this work was done while Arsen Vasilyan was visiting UT Austin.

References

[ABL17] Pranjal Awasthi, Maria Florina Balcan, and Philip M Long. The power of localization for efficiently learning linear separators with noise. Journal of the ACM (JACM) , 63(6):1-27, 2017. 1

[BH21] Maria-Florina Balcan and Nika Haghtalab. Noise in classification. Beyond the WorstCase Analysis of Algorithms , page 361, 2021. 1

[BK21] Ainesh Bakshi and Pravesh K Kothari. List-decodable subspace recovery: Dimension independent error in polynomial time. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 1279-1297. SIAM, 2021. 1

[BZ17] Maria-Florina F Balcan and Hongyang Zhang. Sample and computationally efficient learning algorithms under s-concave distributions. Advances in Neural Information Processing Systems , 30, 2017. 1

[Dan15] Amit Daniely. A ptas for agnostically learning halfspaces. In Conference on Learning Theory , pages 484-502. PMLR, 2015. 1

[DKK + 22] Ilias Diakonikolas, Daniel M Kane, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning general halfspaces with general massart noise under the gaussian distribution. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 874-885, 2022. 1

[DKK + 23] Ilias Diakonikolas, Daniel M Kane, Vasilis Kontonis, Sihan Liu, and Nikos Zarifis. Efficient testable learning of halfspaces with adversarial label noise. arXiv preprint arXiv:2303.05485 , 2023. 1, 1

[DKPZ21] Ilias Diakonikolas, Daniel M Kane, Thanasis Pittas, and Nikos Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the sq model. In Conference on Learning Theory , pages 1552-1584. PMLR, 2021. 1

[DKR23] Ilias Diakonikolas, Daniel M Kane, and Lisheng Ren. Near-optimal cryptographic hardness of agnostically learning halfspaces and relu regression under gaussian marginals. arXiv preprint arXiv:2302.06512 , 2023. 1

[DKTZ20a] Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning halfspaces with massart noise under structured distributions. In Conference on Learning Theory , pages 1486-1513. PMLR, 2020. 1, 2, 4, 4, 4.2, A.5, C.1

[DKTZ20b] Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Non-convex sgd learns halfspaces with adversarial label noise. Advances in Neural Information Processing Systems , 33:18540-18549, 2020. 1, 2, 2, 4, 4, 4.2, C.1

[DKZ20] Ilias Diakonikolas, Daniel Kane, and Nikos Zarifis. Near-optimal sq lower bounds for agnostically learning halfspaces and relus under gaussian marginals. Advances in Neural Information Processing Systems , 33:13586-13596, 2020. 1

[FKP + 19] Noah Fleming, Pravesh Kothari, Toniann Pitassi, et al. Semialgebraic proofs and efficient algorithm design. Foundations and Trends® in Theoretical Computer Science , 14(1-2):1-221, 2019. 1

[GGK20] Surbhi Goel, Aravind Gollakota, and Adam Klivans. Statistical-query lower bounds via functional gradients. Advances in Neural Information Processing Systems , 33:21472158, 2020. 1

[GKK23] Aravind Gollakota, Adam R Klivans, and Pravesh K Kothari. A moment-matching approach to testable learning and a new characterization of rademacher complexity. Proceedings of the fifty-fifth annual ACM Symposium on Theory of Computing , 2023. To appear. 1, 1, 1

[GKSV23] Aravind Gollakota, Adam R Klivans, Konstantinos Stavropoulos, and Arsen Vasilyan. An efficient tester-learner for halfspaces. arXiv preprint arXiv:2302.14853 , 2023. 1, 1, 1, 2, 2, 3.1, 4, 4, 4.2, A.4, A.5, C.1

[HL13] Christopher J Hillar and Lek-Heng Lim. Most tensor problems are np-hard. Journal of the ACM (JACM) , 60(6):1-39, 2013. 3.2

[KKK19] Sushrut Karmalkar, Adam Klivans, and Pravesh Kothari. List-decodable linear regression. Advances in neural information processing systems , 32, 2019. 1

[KLS95] Ravi Kannan, László Lovász, and Miklós Simonovits. Isoperimetric problems for convex bodies and a localization lemma. Discrete & Computational Geometry , 13:541559, 1995. 2, 2.6

[KLS09] Adam R Klivans, Philip M Long, and Rocco A Servedio. Learning halfspaces with malicious noise. Journal of Machine Learning Research , 10(12), 2009. 1

[KS17] Pravesh K Kothari and Jacob Steinhardt. Better agnostic clustering via relaxed tensor norms. arXiv preprint arXiv:1711.07465 , 2017. 1, 3.2, 3.4, 3.2

[Las01] Jean B Lasserre. New positive semidefinite relaxations for nonconvex quadratic programs. Advances in Convex Analysis and Global Optimization: Honoring the Memory of C. Caratheodory (1873-1950) , pages 319-331, 2001. 3.2

[LV07] László Lovász and Santosh Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms , 30(3):307-358, 2007. 2.3

[LV18] Yin Tat Lee and Santosh S Vempala. The Kannan-Lovász-Simonovits conjecture. arXiv preprint arXiv:1807.03465 , 2018. 2.6

[Nes00] Yurii Nesterov. Squared functional systems and optimization problems. High performance optimization , pages 405-440, 2000. 3.2

[Par00] Pablo A Parrilo. Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization . California Institute of Technology, 2000. 3.2

[RV23] Ronitt Rubinfeld and Arsen Vasilyan. Testing distributional assumptions of learning algorithms. Proceedings of the fifty-fifth annual ACM Symposium on Theory of Computing , 2023. To appear. 1, 1

[RY20] Prasad Raghavendra and Morris Yau. List decodable learning via sum of squares. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 161-180. SIAM, 2020. 1

[Sho87] N.Z. Shor. Quadratic optimization problems. Izv. Akad. Nauk SSSR Tekhn. Kibernet. , 1987(1):128-139, 222, 1987. 3.2

[SW14] Adrien Saumard and Jon A Wellner. Log-concavity and strong log-concavity: a review. Statistics surveys , 8:45, 2014. 2.2, 2.5

[YZ17] Songbai Yan and Chicheng Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. Advances in Neural Information Processing Systems , 30, 2017. 1

[Zha18] Chicheng Zhang. Efficient active learning of sparse halfspaces. In Conference on Learning Theory , pages 1856-1880. PMLR, 2018. 1

[ZL21] Chicheng Zhang and Yinan Li. Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise. In Conference on Learning Theory , pages 4526-4527. PMLR, 2021. 1

[ZSA20] Chicheng Zhang, Jie Shen, and Pranjal Awasthi. Efficient active learning of sparse halfspaces with arbitrary bounded noise. Advances in Neural Information Processing Systems , 33:7184-7197, 2020. 1

A Technical Lemmas

In this section, we provide a list of technical results that we use in our proofs.

Lemma A.1 (Preservation of Poincaré constant) . Let I be an open interval in R and q : R d → R + the density of a γ -Poincaré distribution. Let v ∈ S d -1 and q ′ v : R d -1 → R + be the density of the distribution resulting from conditioning q to x · v ∈ I and projecting on the subspace perpendicular to v . Then, the distribution corresponding to q ′ v is γ -Poincaré.

Proof. Assume, without loss of generality, that v = e d . We have that

Let f : R d -1 → R be any differentiable function. In order to show that q ′ v is γ -Poincaré, it is sufficient to show that under no further assumptions on f , the quantity Var q ′ v ( f ( x <d )) is upper bounded by the product of γ and E q ′ v [ ∥∇ f ( x <d ) ∥ 2 2 ] . We expand the quantity Var q ′ v ( f ( x <d )) as follows

which concludes the proof.

Proposition A.2 (Spectral Tester) . Let D be a distribution over R d . Then, there is a tester that given δ ∈ (0 , 1) , λ ≥ 1 , θ > 0 and a set S of i.i.d. samples from D with size at least 2 λd 4 θ 2 δ , runs in time poly( d, 1 θ , | S | ) and satisfies the following specifications

( a ) If the tester accepts, then, for z ∼ S , E S [ zz T ] ⪰ θ 2 I d (resp. E S [ zz T ] ⪯ 2 θI d ).

( b ) If, for z ∼ D , E D [( z i z j ) 2 ] ≤ λ and E D [ zz T ] ⪰ θI d (resp. E D [ zz T ] ⪯ θI d ), then the tester accepts with probability at least 1 -δ .

Proof. The tester receives λ , a set S and δ ∈ (0 , 1) and does the following:

Compute the matrix M S = E S [ zz T ] .

If the minimum (resp. maximum) eigenvalue of M S is larger than θ 2 (resp. smaller than 2 θ ), then accept . Otherwise reject .

Clearly, if the tester accepts, then the desired property is satisfied by construction. If the distribution D satisfies the conditions of part (b), we can show that for M D = E z ∼ D [ zz T ] we have

which implies that M S ⪰ θ 2 I d (and M S ⪯ ( θ + θ 2 ) I d ⪯ 2 θI d ). In particular, we have that ( M S ) ij = E S [ z i z j ] , and by Chebyshev's inequality we have

By a union bound, we get that ∥ M S -M D ∥ max ≤ θ 2 d with probability at least 1 -δ and hence ∥ M S -M D ∥ op ≤ d ∥ M S -M D ∥ max ≤ θ 2 , which concludes the proof.

Proposition A.3. Let c ≥ 0 , λ ≥ 1 , σ ≤ 1 2 λ and D be a λ -nice distribution over R d . Then, for any unit vectors w , v , v ′ , u , u ′ ∈ R d with ⟨ w , v ⟩ = ⟨ w , v ′ ⟩ = 0 and for some universal constant C > 0 we have

(iv) E [ ⟨ v , x ⟩ 2 · ✶ {|⟨ w , x ⟩| ∈ [ c, c + σ ] } ] ≤ 2 σ · α C , for some α ≤ Cλ . Proof. We start by deriving property (i). Recall the function Q from the definition of a λ -nice distribution, which upper-bounds the density of any two-dimensional projection of a λ -nice distribution we see that:

Now, note that the region { ( x 1 , x 2 ) : | x 1 | ≤ σ } is a subset of the set

Therefore:

Note that in the last line above, we bounded the first term via the bound ∫ ∞ r =0 rQ ( r ) dr ≤ λ from the definition of λ -nice distributions. Likewise, we bounded the second term via the inequality Q ( r ) ≤ λ from the definition of λ -nice distributions. Overall, we get

Now, we shall lower-bound the same quantity. We have

Now, since σ ≤ 1 2 λ via the premise of the lemma, we see that the whole region of integration on the right side of the set { ( x 1 , x 2 ) : √ x 2 1 + x 2 2 ≤ 1 λ } . From the definition of λ -nice distributions, the density q span ( v , w ) is lower-bounded by 1 /λ in this region. Therefore, we have

which finishes the proof of property (i).

Now, we derive property (ii). Recall the function Q from the definition of a λ -nice distribution, which upper-bounds the density of any two-dimensional projection of a λ -nice distribution we see that:

Now, note that the region { ( x 1 , x 2 ) : | x 1 | ≤ σ } is a subset of the set

Therefore:

Note that in the last line above, we bounded the first term via the bound on ∫ ∞ r =0 r 3 Q ( r ) dr from the definition of λ -nice distributions. Likewise, we bounded the second term via the inequality Q ( r ) ≤ λ from the definition of λ -nice distributions. Therefore, we get

Now, we shall lower-bound the same quantity. We have

Now, since σ ≤ 1 2 λ via the premise of the lemma, we see that the whole region of integration on the right side of the set { ( x 1 , x 2 ) : √ x 2 1 + x 2 2 ≤ 1 λ } . From the definition of λ -nice distributions, the density q span ( v , w ) is lower-bounded by 1 /λ in this region. Therefore, we have which finishes the proof of property (ii).

We proceed to property (iii). We will denote the angle between v and v ′ as β , which allows us to write

which finishes the proof of property (iii).

Finally, we prove property (iv). For β ≥ 0 we have

Applying the above inequality to the quantity of property (iv), we get the desired result.

This concludes the proof of Proposition A.3.

Proposition A.4 (Proposition 4.2 of [GKSV23]) . There is a universal constant C > 0 , such that for any σ > 0 , there exists a continuously differentiable function ℓ σ : R → [0 , 1] with the following properties.

For any t ∈ [ -σ/ 6 , σ/ 6] , ℓ σ ( t ) = 1 2 + t σ .

For any t > σ/ 2 , ℓ σ ( t ) = 1 and for any t < -σ/ 2 , ℓ σ ( t ) = 0 .

For any t ∈ R , ℓ ′ σ ( t ) ∈ [0 , C/σ ] , ℓ ′ σ ( t ) = ℓ ′ σ ( -t ) and | ℓ ′′ σ ( t ) | ≤ C/σ 2 .

Proposition A.5 (PSGD Convergence [DKTZ20a], restated in [GKSV23]) . Let L σ be as in Equation (4.1) with σ ∈ (0 , 1] , ℓ σ as described in Proposition A.4, λ ≥ 1 and D XY such that the marginal D X on R d is λ -nice. Then for some universal constant C > 0 and for any ϵ > 0 and δ ∈ (0 , 1) , there is an algorithm whose time and sample complexity is O ( λ C d σ 4 + λ C log(1 /δ ) ϵ 4 σ 4 ) , which, having access to samples from D XY , outputs a list L of vectors w ∈ S d -1 with | L | = O ( λ C d σ 4 + λ C log(1 /δ ) ϵ 4 σ 4 ) so that there exists w ∈ L with

In particular, the algorithm performs Stochastic Gradient Descent on L σ Projected on S d -1 (PSGD).

B Proofs from Section 3

B.1 Proof of Lemma 3.1

We restate Lemma 3.1 here for convenience.

Lemma B.1 (Lemma 3.1) . Let D XY be a distribution over R d ×{± 1 } , w ∈ S d -1 , θ ∈ (0 , π/ 4] , λ ≥ 1 and δ ∈ (0 , 1) . Then, for a sufficiently large constant C , there is a tester that given δ , θ , w and a set S of samples from D X with size at least C · ( d 4 θ 2 δ ) , runs in time poly ( d, 1 θ , 1 δ ) and satisfies the following specifications:

( a ) If the tester accepts S , then for every unit vector w ′ ∈ R n satisfying ∡ ( w , w ′ ) ≤ θ we have P [sign( ⟨ w ′ , x ⟩ ) = sign( ⟨ w , x ⟩ )] ≤ C · θ · λ C

̸

( b ) If the distribution D X is λ -nice, the tester accepts S with probability 1 -δ .

Proof. The testing algorithm receives integer d , set S ⊂ R d , w ∈ S d -1 , θ ∈ (0 , π/ 4] , λ ≥ 1 and δ ∈ (0 , 1) and does the following for some sufficiently large universal constant C 1 > 0 :

If P x ∈ S [ |⟨ w , x ⟩| ∈ [0 , θ ]] > C 1 θλ C 1 , then reject .

Let proj ⊥ w : R d → R d -1 denote the operator that given any vector in R d , it outputs its projection into the ( d -1) -dimensional subspace of R d that is orthogonal to w .

Compute the ( d -1) × ( d -1) matrix M S as follows 5 :

Run the spectral tester of Proposition A.2 on M S given δ ← δ , λ ← C 1 λ C 1 and θ ← C 1 2 θλ C 1 , i.e., compute ∥ M S ∥ op and if ∥ M S ∥ op > C 1 θλ C 1 , then reject . Otherwise, accept .

First, suppose the test accepts. For the following, consider the vector w ′ ∈ R d to be an arbitrary unit vector and v ∈ R d to be the unit vector that is perpendicular to w , lies within the plane defined by w and w ′ and ⟨ v , w ′ ⟩ ≤ 0 . Then we have:

̸

For part (b), we suppose that the distribution D X is indeed λ -nice. We will show that with probability at least 1 -δ , the tester will accept, i.e., that

We first observe that the corresponding quantities under distribution D X due to Proposition A.3. In particular, we have that for some universal constant C ′ > 0

✶ If we let M D X = E D X [ M S ] , we get that ∥ M D X ∥ op = sup u ∈ S d -2 u T M D X u = sup v ′ ∈ S d -1 : ⟨ v ′ , w ⟩ =0 (proj ⊥ w v ′ ) T M D X (proj ⊥ w v ′ ) ≤ ∞ ∑ i =2 1 ( i -1) 2 sup v ′ ∈ S d -1 E x ∈ D X [ ⟨ v ′ , x ⟩ 2 · ✶ {|⟨ w , x ⟩| ∈ [ c, c + θ ] } ] ≤ ∞ ∑ i =2 1 ( i -1) 2 sup v ′ ∈ S d -1 C ′ θλ C ′ ≤ C ′ π 2 6 θλ C ′

By Proposition A.2, in order to satisfy expression (B.2), it remains to show that E z ∼ D [( z ℓ z j ) 2 ] ≤ C 1 λ C 1 for any ℓ, j ∈ [ d ] , where z is defined as follows

Since E z ∼ D [( z ℓ z j ) 2 ] ≤ E z ∼ D [ ⟨ u , x ⟩ 2 ⟨ u ′ , x ⟩ 2 ] , for some unit vectors u , u ′ ∈ S d -1 (orthogonal to w ), the desired bound follows from Proposition A.3.

It remains to bound the absolute distance between the quantities of the left hand side of expressions (B.1) and (B.3). This can be achieved by an application of the Hoeffding bound, since the empirical version of the quantity is the average of independent Bernoulli random variables.

B.2 Proof of Lemma 3.2

We restate Lemma 3.2 here for convenience.

Lemma B.2 (Universally Testable Weak Anti-Concentration) . Let D be a distribution over R d . Then, there is a universal constant C > 0 and a tester that given a unit vector w ∈ R d , δ ∈ (0 , 1) , γ > 0 , λ ≥ 1 , σ ≤ 1 2 λ and a set S of i.i.d. samples from D with size at least C · d 4 σ 2 δ log( d ) λ C , runs in time poly( d, λ, 1 σ , 1 δ , log( 1 γ )) and satisfies the following specifications

( a ) If the tester accepts S , then for any unit vector v ∈ R d with ⟨ v , w ⟩ = 0 we have

( b ) If D is γ -Poincaré and λ -nice, then the tester accepts S with probability at least 1 -δ .

Proof. The testing algorithm receives a set S ⊂ R d , w ∈ S d -1 , δ ∈ (0 , 1) , γ > 0 , λ ≥ 1 and σ ≤ 1 2 λ and does the following for some sufficiently large C 1 > 0 :

If P x ∈ S [ |⟨ w , x ⟩| ≤ σ ] > 2 σ · C 1 λ C 1 , then reject .

Compute the ( d -1) × ( d -1) matrix M S as follows:

Run the spectral tester of Proposition A.2 on M S given δ ← δ , λ ← C 1 λ C 1 and θ ← 2 σ C 1 λ C 1 , i.e., reject if the minimum singular value of M S is less than 2 σ C 1 λ C 1 .

Run the hypercontractivity tester (Prop. 3.5) on S ′ = { proj ⊥ w x : x ∈ S and |⟨ w , x ⟩| ≤ σ } , i.e., solve an appropriate SDP and reject if the solution is larger than a specified threshold. Otherwise, accept .

For part (a), we apply the Paley-Zygmund inequality to the random variable Z = ⟨ v , x ⟩ 2 condtitioned on |⟨ w , x ⟩| ≤ σ and get

Note that since ⟨ v , w ⟩ = 0 , we have ⟨ v , x ⟩ = ⟨ proj ⊥ w v , proj ⊥ w x ⟩ (where ∥ v ∥ 2 = ∥ proj ⊥ w v ∥ 2 ). Therefore, since S has passed the spectral tester as well as the tester for the probability of lying within the strip |⟨ w , x ⟩| ≤ σ , we have that

Moreover, { x ∈ S : |⟨ w , x ⟩| ≤ σ } has passed the hypercontractivity tester, and therefore, according to Proposition 3.5 we have

Combining the above inequalities we conclude the proof of part (a).

For part (b), we assume that D is indeed λ -nice and γ -Poincaré. We first use Proposition A.3 as well as a Hoeffding bound, to get that P x ∈ S [ |⟨ w , x ⟩| ≤ σ ] ∈ [ 2 σ C ′ λ C ′ , 2 σ · C ′ λ C ′ ] with probability at least 1 -δ/ 3 over S (since | S | is large enough), for some universal constant C ′ > 0 . Then, we use part (ii) of Proposition A.3 to lower bound the minimum eigenvalue of M D = E D [ M S ] by 4 σ C ′ λ C ′ . Using part (iii) of Proposition A.3 to bound the second moment of each of the elements of M D , we may use Proposition A.2 to get that M S ⪰ 2 σ C ′ λ C ′ I d -1 (and our spectral test passes) with probability at least 1 -δ/ 3 . It remains to show that the hypercontractivity tester will accept with probability at least 1 -δ/ 3 (since, then, the result follows from a union bound).

We acquire samples from the hypercontractivity tester through rejection sampling (we keep only the samples within the strip). Since the probability of falling inside the strip is at least 2 σ C ′ λ C ′ , the number of samples we will keep is at least | S ′ | ≥ | S | σ C ′′ λ C ′ , for some large enough constant C ′′ > 0 (due to Chernoff bound) and with probability at least 1 -δ/ 6 . We now apply Lemma A.1 to get that the distribution of proj ⊥ w x conditioned on the strip |⟨ w , x ⟩| ≤ σ is γ -Poincaré, since D is also γ -Poincaré. Hence, the hypercontractivity tester accepts with probability at least 1 -δ/ 6 due to Proposition 3.5.

C Proofs from Section 4

C.1 Proof of Proposition 4.2

We restate Proposition 4.2 here for completeness.

Proposition C.1 (Modification from [GKSV23, DKTZ20a, DKTZ20b]) . For a distribution D XY over R d × {± 1 } let opt be the minimum error achieved by some origin-centered halfspace and w ∗ ∈ S d -1 a corresponding vector. Consider L σ as in Equation (4.1) for σ > 0 and let η < 1 / 2 . Let w ∈ S d -1 with ∡ ( w , w ∗ ) = θ < π 2 and v ∈ span( w , w ∗ ) such that ⟨ v , w ⟩ = 0 and ⟨ v , w ∗ ⟩ < 0 . Then, for some universal constant C > 0 and any α ≥ σ 2 tan θ we have ∥∇ w L σ ( w ; D XY ) ∥ 2 ≥ A 1 -A 2 -A 3 , where

Proof. The proof is a slight modification of a part of the proof of Lemma 4.4 in [GKSV23], but we present it here for completeness.

For any vector x ∈ R d , let: x w = ⟨ w , x ⟩ and x v = ⟨ v , x ⟩ . It follows that proj V ( x ) = x v e 1 + x w e 2 , where proj V is the operator that orthogonally projects vectors on V . Using the fact that ∇ w ( ⟨ w , x ⟩ / ∥ w ∥ 2 ) = x -⟨ w , x ⟩ w = x -x w w for any w ∈ S d -1 , the interchangeability of the gradient and expectation operators and the fact that ℓ ′ σ is an even function we get that

Since the projection operator proj V is a contraction, we have ∥∇ w L σ ( w ) ∥ 2 ≥ ∥ proj V ∇ w L σ ( w ) ∥ 2 , and we can therefore restrict our attention to a simpler, two dimensional problem. In particular, since proj V ( x ) = x v e 1 + x w e 2 , we get

̸

̸

✶ Let F ( y, x ) denote 1 -2 ✶ { y = sign( ⟨ w ∗ , x ⟩ ) } . We may write x v as | x v | · sign( x v ) and let G ⊆ R 2 such that sign( x v ) · sign( ⟨ w ∗ , x ⟩ ) = -1 iff x ∈ G .

W

A1

A1

Then, sign( x v ) · sign( ⟨ w ∗ , x ⟩ ) = ✶ { x ̸∈ G} -✶ { x ∈ G} . We get

̸

Let A ′ 1 = E [ ℓ ′ σ ( | x w | ) · ✶ { x ∈ G}· F ( y, x ) · | x v | ] and A ′ 2 = E [ ℓ ′ σ ( | x w | ) · ✶ { x ̸∈ G}· F ( y, x ) · | x v | ] . In the Massart noise case E y | x [ F ( y, x )] = 1 -2 η ( x ) ∈ [1 -2 η, 1] , where 1 -2 η > 0 . Therefore, we have that A ′ 1 ≥ (1 -2 η ) · E [ ℓ ′ σ ( | x w | ) · ✶ { x ∈ G} · | x v | ] . When the noise is adversarial, we have A ′ 1 ≥ E [ ℓ ′ σ ( | x w | ) · ✶ { x ∈ G} · | x v | ] -2 E [ ℓ ′ σ ( | x w | ) · ✶ { x ∈ G} · ✶ { y = sign( ⟨ w ∗ , x ⟩ ) } · | x v | ] . For any α ≥ σ 2 tan θ , we have that

Moreover, for some universal constant C ′ > 0 , we similarly have

Hence, we have shown that, in the Massart noise case, we have ∥∇ w L σ ( w ) ∥ 2 ≥ (1 -2 η ) A 1 -A 2 as desired. For the adversarial noise case, it remains to bound the following quantity

̸

C.2 Proof of Lemma 4.3

We restate Lemma 4.3 here for convenience.

Lemma C.2 (Universally Testable Structure of Surrogate Loss) . Let D XY be any distribution over R d ×{± 1 } . Consider L σ as in Equation (4.1) . Then, there is a universal constant C > 0 and a tester that given a unit vector w ∈ R d , δ ∈ (0 , 1) , η < 1 / 2 , γ > 0 , λ ≥ 1 , σ ≤ 1 Cλ C and a set S of i.i.d. samples from D XY with size at least C · d 4 σ 2 δ log( d ) λ C , runs in time poly( d, λ, 1 σ , 1 δ , log( 1 γ )) and satisfies the following specifications

( a ) If the tester accepts S , then, the following statements are true for the minimum error opt S achieved by some origin-centered halfspace on S and the optimum vector w ∗ S ∈ S d -1

If the noise is Massart with associated rate η and ∥∇ w L σ ( w ; S ) ∥ 2 ≤ 1 -2 η Cλ C γ 4 then either ∡ ( w , w ∗ S ) ≤ Cλ C (1+ γ 4 ) 1 -2 η · σ or ∡ ( -w , w ∗ S ) ≤ Cλ C (1+ γ 4 ) 1 -2 η · σ .

If the noise is adversarial with opt S ≤ σ Cλ C and ∥∇ w L σ ( w ; S ) ∥ 2 < 1 Cλ C γ 4 then either ∡ ( w , w ∗ S ) ≤ Cλ C (1 + γ 4 ) · σ or ∡ ( -w , w ∗ S ) ≤ Cλ C (1 + γ 4 ) · σ .

( b ) If the marginal D X is λ -nice and γ -Poincaré, then the tester accepts S with probability at least 1 -δ .

Proof of Lemma 4.3. The testing algorithm receives w ∈ S d -1 , δ ∈ (0 , 1) , η < 1 / 2 , γ > 0 , λ ≥ 1 , σ ≤ 1 2 λ and a set S ⊂ R d ×{± 1 } and does the following for some sufficiently large C 1 > 0

̸

̸

Compute the ( d -1) × ( d -1) matrices M + S and M -S as follows:

Run the (maximum singular value) spectral tester of Proposition A.2 on M + S given δ ← δ 4 , λ ← C 1 λ C 1 and θ ← C 1 σλ C 1 2 , i.e., reject if the maximum singular value of M + S is greater than σ · C 1 λ C 1 .

Run the (minimum singular value) spectral tester of Proposition A.2 on M -S given δ ← δ 4 , λ ← C 1 λ C 1 and θ ← 2 σ C 1 λ C 1 , i.e., reject if the minimum singular value of M -S is less than σ C 1 λ C 1 .

Run the hypercontractivity tester on S ′ = { proj ⊥ w x : ( x , y ) ∈ S and |⟨ w , x ⟩| ≤ σ } , i.e., solve an appropriate SDP (see Prop. 3.5 with γ ← γ , δ ← δ/ 4 ) and reject if the solution is larger than a specified threshold. Otherwise, accept .

For part (a), we suppose that the testing algorithm has accepted S . Therefore, S has passed all the tests required for part (a) of Lemma 3.2 and there exists a universal constant C ′ > 0 such that

Since Proposition 4.2 holds for any distribution, it will also hold for the empirical distribution (uniform on S ). We apply Proposition 4.2 with α = 1 C ′ λ C ′ to lower bound ∥∇ w L σ ( w ; S ) ∥ 2 (or ∥∇ w L σ ( -w ; S ) ∥ 2 ) as follows

Combining the above inequalities with the bounds implied by the fact that S has passed the tests, concludes the proof of part (a), since (after observing that tan θ ≥ θ ) we get

For part (b), we follow a similar recipe as the one used to prove part (b) of Lemma 3.2, i.e., we use the following reasoning to show that the tests will pass with probability at least 1 -δ

We assume that the marginal distribution D X is λ -nice and γ -Poincaré.

Weuse Proposition A.3 to bound the values of the tested quantities under the true distribution.

We use appropriate concentration results (Hoeffding/Chernoff Bounds and Proposition A.2) to show that, since | S | is large enough, each of the empirical quantities at hand does not deviate a lot from its mean.

This concludes the proof of Lemma 4.3.

C.3 Proof of Main Theorem

We restate the main Theorem here for convenience.

Theorem C.3 (Efficient Universal Tester-Learner for Halfspaces) . Let D XY be any distribution over R d ×{± 1 } . Let C be the class of origin centered halfspaces in R d . Then, for any λ ≥ 1 , γ > 0 , ϵ > 0 and δ ∈ (0 , 1) , there exists an universal tester-learner for C w.r.t. the class of λ -nice and γ -Poincaré marginals up to error poly( λ ) · (1+ γ 4 ) · opt + ϵ , where opt = min w ∈ S d -1 P D XY [ y = sign( ⟨ w , x ⟩ )] , and error probability at most δ , using a number of samples and running time poly( d, λ, γ, 1 ϵ , log 1 δ ) .

̸

Moreover, if the noise is Massart with given rate η < 1 / 2 , then the algorithm achieves error opt + ϵ with time and sample complexity poly( d, λ, γ, 1 ϵ , 1 1 -2 η , log 1 δ ) .

Proof of Theorem 4.1. Note that we will give the algorithm for δ ← δ ′ = 1 / 3 since we can reduce the probability of failure with repetition (repeat O (log 1 δ ) times, accept if the rate of acceptance is Ω(1) and output the halfspace achieving the minimum test error among the halfspaces returned).

For reader's convenience, we now restate the algorithm on page 9 (note that together with the algorithm we include additional detail relevant to the analysis). The algorithm receives λ ≥ 1 , γ > 0 , ϵ > 0 and η ∈ (0 , 1 / 2) ∪ { 1 } (say η = 1 when we are in the agnostic case) and does the following for some appropriately large universal constant C 1 , C 2 > 0 .

First, create a set of parameters Σ and parameters E = ϵ C 1 λ C 1 and A > 0 as follows. If η = 1 , then Σ is an E C 1 λ C 1 -cover of the interval [ 0 , 1 C 1 λ C 1 ] and A = 1 C 1 λ C 1 γ 4 . Otherwise, let Σ = { E · (1 -2 η ) C 1 λ C 1 (1+ γ 4 ) } and A = 1 -2 η C 1 λ C 1 γ 4 .

Then, draw a set S 1 of C 2 ( λd γϵ ) C 2 i.i.d. samples from D XY and run PSGD, as specified in Proposition A.5 with ϵ ← A , δ ← δ ′ C 1 on the loss L σ for each σ ∈ Σ .

Form a list L with all the pairs of the form ( w , σ ) where w ∈ S d -1 is some iterate of the PSGD subroutine performed on L σ .

Draw a fresh set S 2 of C 2 ( λd γϵ ) C 2 i.i.d. samples from D XY and compute for each ( w , σ ) ∈ L the value ∥∇ w L σ ( w ; S 2 ) ∥ 2 . If, for some σ ∈ Σ , ∥∇ w L σ ( w ; S 2 ) ∥ 2 > A for all ( w , σ ) ∈ L , then reject .

Update L by keeping for each σ ∈ Σ only one pair of the form ( w , σ ) for which we have ∥∇ w L σ ( w ; S 2 ) ∥ 2 ≤ A .

Run the following tests for each ( w , σ ) ∈ L to ensure that part (a) of Lemma 4.3 holds for each of the elements of L , i.e., that any stationary point of the surrogate loss that lies in L is angularly close to the empirical risk minimizer (or the same holds for the inverse vector).

If P ( x ,y ) ∈ S 2 [ |⟨ w , x ⟩| ≤ σ 6 ] ≤ σ C 1 λ C 1 or P ( x ,y ) ∈ S 2 [ |⟨ w , x ⟩| ≤ σ 2 ] > σ · C 1 λ C 1 , then reject .

Compute the ( d -1) × ( d -1) matrices M + S 2 and M -S 2 as follows:

Run the (maximum singular value) spectral tester of Proposition A.2 on M + S 2 given δ ← δ ′ C 1 , λ ← C 1 λ C 1 and θ ← C 1 σλ C 1 2 , i.e., reject if the maximum singular value of M + S 2 is greater than σ · C 1 λ C 1 .

Run the (minimum singular value) spectral tester of Proposition A.2 on M -S 2 given δ ← δ ′ C 1 , λ ← C 1 λ C 1 and θ ← 2 σ C 1 λ C 1 , i.e., reject if the minimum singular value of M -S 2 is less than σ C 1 λ C 1 .

Run the hypercontractivity tester on S ′ = { proj ⊥ w x : ( x , y ) ∈ S 2 and |⟨ w , x ⟩| ≤ σ } , i.e., solve an appropriate SDP (see Prop. 3.5 with γ ← γ , δ ← δ ′ /C 1 ) and reject if the solution is larger than a specified threshold.

Run the following tests for each pair of the form ( w , σ ) and ( -w , σ ) where ( w , σ ) ∈ L to ensure that part (a) of Lemma 3.1 is activated, i.e., that the distance of a vector from the empirical risk minimizer is an accurate proxy for the error of the corresponding halfspace. Set θ ( σ ) = (1+ γ 4 ) σ 4 .

If P ( x ,y ) ∈ S 2 [ |⟨ w , x ⟩| ≤ θ ] > C 1 λ C 1 θ then reject .

Compute the ( d -1) × ( d -1) matrix M S 2 as follows 6 :

Run the spectral tester of Proposition A.2 on M S given δ ← δ ′ C 1 , λ ← C 1 λ C 1 and θ ← C 1 2 θλ C 1 , i.e., compute ∥ M S ∥ op and if ∥ M S ∥ op > C 1 θλ C 1 , then reject .

Otherwise, accept and output the vector w that achieves the smallest empirical error on S 2 among the vectors in the list L .

For the following, let α = 1 in the Massart noise case and α = C 1 λ C 1 γ 4 in the adversarial noise case. Consider also opt S 2 to be the error of the origin-centered halfspace with the minimum empirical error on S 2 and w ∗ S 2 the corresponding optimum vector.

Soundness. We first prove the soundness condition, i.e., that the following implication holds with probability at least 1 -δ ′ over the samples:

̸

The tester accepts only if for every σ ∈ Σ , we have some w ∈ L with ∥∇ w L σ ( w ; S 2 ) ∥ 2 ≤ A (step 4) and for which part (a) of each of Lemmas 4.3 (step 6) and 3.1 (step 7) is activated. Therefore, in the Massart noise case, for any σ ∈ Σ , there is some w such that either ( w , σ ) ∈ L or ( -w , σ ) ∈ L and also

̸

In the adversarial noise case, the above are true conditional on σ being such that opt S 2 ≤ σ C ′ λ C ′ .

Therefore, in the Massart noise case, the above are true for σ = E (1 -2 η ) C 1 λ C 1 (1+ γ 4 ) which gives

̸

In the agnostic case, condition C.2 is true for some σ ∈ [0 , 1 C 1 λ C 1 ] such that

unless opt > 1 C 1 C ′ λ C 1 + C ′ , in which case any halfspace has error at most 1 = opt · ( C 1 C ′ λ C 1 + C ′ ) . Hence we get

̸

Soundness follows from the fact that if | S 2 | is sufficiently large (but still polynomial in every parameter, since the VC dimension of the class of halfspaces in R d is d +1 ), then | opt S 2 -opt | ≤ ϵ C 1 λ C 1 (1+ γ ) 4 with probability at least 1 -δ ′ .

Completeness. Suppose now that the marginal is indeed λ -nice and γ -Poincaré. Then, for sufficiently large S 1 , after step 3, L will contain a stationary point of L σ ( · ; D XY ) for each σ ∈ Σ , due to Proposition A.5. If S 2 is large enough, then steps 4, 6 and 7 will each accept with probability at least 1 -δ ′ /C 1 , due to part (b) of Lemmas 4.3 and 3.1, as well as the fact that each coordinate of ∇ w L σ ( w ; S 2 ) has bounded second moment (Proposition A.3) and therefore ∇ w L σ ( w ; S 2 ) is concentrated around ∇ w L σ ( w ; D XY ) for any fixed w such that ( w , σ ) ∈ L (we also need a union bound over L ). Hence, in total, the tester will accept with probability at least 1 -δ ′ .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of designing tester-learners for halfspaces in the agnostic setting and under Massart noise. In the setting of tester-learners, we do not make distributional assumptions on our samples, but rather require that our algorithm produces an accurate output whenever the input satisfies a sequence of efficiently computable tests. Further, if the samples actually come from a specific target distribution, it is required that the tests pass with high probability.\n\nPrevious tester-learners for noisy halfspaces used tests tailored towards a *specific* distribution, e.g., a specific strongly log-concave distribution. As a consequence, these testers might reject samples from a different strongly log-concave distribution although algorithmically it would have been possible to learn the underlying halfspace. The authors propose a tester which simultaneously accepts for all Poincaré distributions that satisfy some natural niceness conditions. This class includes all isotropic strongly log-concave distributions. When their tester accepts, the tester-learner outputs a halfspace achieving error $O(\\mathrm{opt}) + \\varepsilon$, where $\\mathrm{opt}$ is the error of the optimal halfspace on the distribution of the input. For the Massart setting they can handle all isotropic log-concave distributions and achieve error $\\mathrm{opt} + \\varepsilon$.\n\nThe authors argue that previous works were inherently limited to work with only a single distribution since they were matching moments with this specific distribution. In their work, they overcome this issue by only checking the actual properties they need in the proof of accuracy which are not tied to any specific distribution.\n\nSTRENGTHS:\nIn my eyes, designing tester learners which simultaneously accept for a large class of distribution is an important step for bringing the framework of testable learning closer to practice. The motivation for testable learning was that designing algorithms that only work under specific distributional assumptions has little value in practice since likely these assumptions will not be met. Thus, we would at least like to know if we can trust the output of our algorithm when we run it on actual data or not. While previous tester-learners did have this benefit, it seems that the proposed testers \"overfit\" to specific distributions. However, the chances that real world data comes from any specific distribution are rather small. Thus, providing testers which work for larger classes of distributions is an important step.\n\nOn a technical level, their tester has a natural explanation that I like: It simply checks the properties that are need in the proof in the non-testable setting. Further, the paper is very well-written (except the point below).\n\nWEAKNESSES:\nI feel that the technical overview that the authors provide might be hard to understand if the reader is not familiar with the arguments in previous work or at least the general area. It would be nice if it could be adjusted slightly to make it more accessible to a wider audience.\n\nAlso, specifically for the agnostic setting, it would be useful to compare your approximation factor to (a) testable learners that only work for one specific (or a more restricted class of distributions) and (b) non-testable learners for the same distribution class you consider.\n\nQUESTIONS:\nRelated to the above: In the agnostic setting, your error bound scales as $\\mathrm{poly}(\\lambda) (1+\\gamma^4) \\mathrm{opt} + \\varepsilon$, where $\\lambda$ is the niceness parameter of the marginal distribution and $\\gamma$ the Poincaré constant. How does this compare to the non-testable learning error?\n\nLIMITATIONS:\nLimitations were addressed appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a tester-learner for halfspaces in the recently proposed testable learning framework of [RV23]. A recent work [GKSV23] had studied the same problem and proposed tester-learners for halfspaces with Massart noise and in the agnostic setting, for any particular strongly-log concave distribution. This work qualitatively improves upon [GKSV23] by developing a tester-learner for a class of strongly log-concave distributions rather than a fixed strongly log-concave distribution, and also provides better algorithmic complexity for the agnostic case. In fact, this work captures $\\gamma$-Poincare distributions which contain strongly log-concave distributions and under the [KLS95] conjecture also all log-concave distributions. In the Massart noise case, their algorithm works for log-concave distributions unconditionally.\n While following the recently developed approach (and used by [GKSV23]) of non-convex SGD for learning halfspaces over certain good distributions, this paper does a more careful analysis of the gradient lower bound leading to the requirement of a certain hypercontractivity property (which holds for Poincare distributions). The paper uses techniques for previous work [KS17] to show that sum-of-squares (SoS) SDP can be used to certify this hypercontractivity which is a main technical novelty.\n\nSTRENGTHS:\nThe paper proves better and qualitatively more general and stronger bounds for testable learning of halfspaces over a general class of distributions. The technical contributions - the use of properties of Poincare distributions and using SoS to certify hypercontractivity -  are interesting and novel in the context of this problem, and provide for universal tester-learners. The paper is well written and effectively conveys its main contributions and the techniques used. Overall, the paper is technically solid.\n\nWEAKNESSES:\nThe paper follows the same roadmap of [GKSV23] with better analysis and a new tester (based on previous work [KS17]) for hypercontractivity, and in that sense feels somewhat incremental. The results are also moderate improvements and generalizations of previous results and not entirely unexpected.\n\nQUESTIONS:\n1. It may be helpful to clarify why the main Theorem 4.1 also captures log-concave distributions unconditionally for the case with Massart noise, while only capturing strongly log-concave distributions in the agnostic setting. \n\n2. Is it clear that the previous work of [GKSV23] cannot be tweaked (e.g. by discretizing the space of the vector of moments to be matched) to work for the class of strongly log-concave distributions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the first tester-learner for learning the class of halfspaces universally over a class of strongly log-concave distributions (or the class of all log-concave distributions under the KLS assumption). Unlike prior works that crucially rely on testing a specific given marginal distribution while rejecting other well-behaved distributions, the proposed algorithm can accept the marginal distribution as long as it is in the family $\\mathcal{D}$. The proposed algorithm is motivated by the tester-learner using non-convex SGD [GKSV23] but with a more careful analysis on the anti-concentration property. The algorithm is powered by the SOS program to check the hypercontractivity of the desired random variable $Z$ that exhibits certain anti-concentration properties, which is known suffices for identifying the Poincare distributions ([KS17]).\n\nSTRENGTHS:\nDesigning provable tester-learners addresses the issue that traditional learning algorithms crucially rely on distribution assumptions which could be stringent in practice, which is of importance both from the practical and theoretical sides. This paper proposes the first tester-learner that universally accepts any distribution from a broad family of distributions that satisfy a Poincare inequality, largely generalizing those algorithms that only accepts a certain distribution. The technical contribution lies in identifying the Poincare distributions by applying SoS framework to certify a hypercontractive property of the constructed random variable, which generalizes the known results for standard Gaussian to distributions with weak anti-concentration properties. The technique could be of independent interest for testable weak anti-concentration.\n\nWEAKNESSES:\nFrom the technical side, the SoS program for checking the hypercontractive already exists in [KS17]. The contrition of this paper is more on the application side of such a framework on the problem of testing-learning halfspaces with additional assumptions of certain concentration and anti-concentration. That being said, it is a novel application of such a framework given that it solves a very interesting and important problem.\n\nQUESTIONS:\nWhile Massart noise is considered in this paper, is it possible to directly apply the techniques to other more challenging types of noise?\n\nLIMITATIONS:\nNo concerns.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearning halfspaces is a very well studied problem in machine learning. In the agnostic (adversarial label noise) and distribution free setting, this problem has been known to be computationally intractable. As a result, there have been several works of agnostic learning in distribution specific settings (where the marginal distribution belongs to a particular family of distributions, say Gaussian or log-concave). In this scenario, the learner has an error of the form $OPT + \\epsilon$, where OPT denotes the optimal 0-1 error. This however has complexity $d^{1/\\epsilon^2}$, and the exponential dependency on $1/\\epsilon$ is tight. This motivates the designing of learning algorithms that have better sample complexity with respect to $1/\\epsilon$, whereas the error becomes $f(OPT) + \\epsilon$ for some function f.\n\nOften these works use a single distribution as the target marginal distribution. In this work, the authors studied this problem with respect to a set of marginal distributions (Universal testable learning, Definition 1.1). The authors studied this problem in the newly introduced Testable learning framework  by Rubinfeld and Vasilyan. Here the goal is if the tester accepts, then with high probability the output of the learning is close to some function of OPT, and if the data satisfies the distributional assumptions, the algorithm accepts with high probability.\n\nHere the authors design a universal tester learner for Halfspaces with respect to distributions with bounded Poincare constant (Definition 2.4) and concentration and anti-concentration properties (Definition 2.1) in Theorem 1.2. Moreover, this class of distributions contains strongly log-concave distributions, and assuming Kannan–Lovasz–Simonovits (KLS) conjecture, contains all log-concave distributions (Definition 2.2-2.6). Later in theorem 1.3, they design a universal tester learner for Halfspaces with Massart noise (the labels are flipped by an adversary with probability $\\eta <1/2$).\n\nSome previous and concurrent works for single distribution D^* assumption use approximate moment matching techniques, where the algorithm tests if the low moments of the input distribution D approximately matches with that of D^*. However it is not clear if this same approach can be applied for a collection of distributions. This has also been discussed in the introduction.\n\nThe authors first design testers for testing bounded disagreement (Lemma 3.1) and of testing anti-concentration properties (Lemma 3.2) which uses a tester for testing hypercontrctivity. It is known that any distribution $D$ with bounded Poincare constant is hypercontractive in Sum-of-Squares (SOS) framework. Thus the authors run a polynomial time semidefinite program for the later purpose. Next in Section 4, the authors design the final tester (Theorem 4.1). They use a surrogate loss minimization technique for the surrogate loss function defined in Equation 4.1. The main idea is that stationary points of surrogate loss are close to some optimal vector corresponding to the halfspace (the related lemma is Lemma 4.3). Lemma 4.3 contains bounds for both Massart and adversarial noise settings.\n\nIt would be interesting if tester-learners can be designed for function classes other than halfspaces. The authors also discusses this at the end of Section 1.\n\nSTRENGTHS:\nOverall this is a nice result which generalizes several previous works designed for a single marginal distribution assumption and studies for the setting of a collection of marginal distributions.\n\nWEAKNESSES:\nWhat is the usefulness of the tester-learner model in real life applications.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nIt would be interesting if tester-learners can be designed for function classes other than halfspaces. The authors also discusses this at the end of Section 1.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}