SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Konstantinos Stavropoulos
kstavrop@cs.utexas.edu
Arsen Vasilyan
vasilyan@mit.edu
We give the first tester-learner for halfspaces that succeeds universally over a wide class of structured distributions. Our universal tester-learner runs in fully polynomial time and has the following guarantee: the learner achieves error  $O(\mathrm{opt}) + \epsilon$  on any labeled distribution that the tester accepts, and moreover, the tester accepts whenever the marginal is any distribution that satisfies a Poincaré inequality. In contrast to prior work on testable learning, our tester is not tailored to any single target distribution but rather succeeds for an entire target class of distributions. The class of Poincaré distributions includes all strongly log-concave distributions, and, assuming the Kannan-Lóvasz-Simonovits (KLS) conjecture, includes all log-concave distributions. In the special case where the label noise is known to be Massart, our tester-learner achieves error  $\mathrm{opt} + \epsilon$  while accepting all log-concave distributions unconditionally (without assuming KLS). Our tests rely on checking hypercontractivity of the unknown distribution using a sum-of-squares (SOS) program, and crucially make use of the fact that Poincaré distributions are certifiably hypercontractive in the SOS framework.
In this paper we study the recent model of testable learning, due to Rubinfeld and Vasilyan [RV23]. Testable learning addresses a key issue with essentially all known algorithms for the basic problem of agnostic learning, in which a learner is required to produce a hypothesis competitive with the best-fitting hypothesis in a concept class  $\mathcal{C}$ . The issue is that these algorithms make distributional assumptions (such as Gaussianity or log-concavity) that are in general hard to verify. This means that in the absence of any prior information about the distribution or the optimal achievable error, it can be hard to check that the learner has even succeeded at meeting its guarantee.
In the testable learning model, the learning algorithm, or tester-learner, is given access to labeled examples from an unknown distribution and may either reject or accept the unknown distribution. If it accepts, it must successfully produce a near-optimal hypothesis. Moreover, it is also required to accept whenever the unknown distribution truly has a certain target marginal  $D^{*}$ . Work of [RV23, GKK23, GKSV23, DKK+23] provided tester-learners for a range of basic classes (including halfspaces, intersections of halfspaces, and more) with respect to particular target margins  $D^{*}$  (such as the standard Gaussian). All of these algorithms, however, have the shortcoming that they are closely tailored to the particular target marginal  $D^{*}$  that is chosen. Indeed, their tests would reject many well-behaved distributions that are appreciably far from  $D^{*}$ . A highly natural question from both a theoretical and a practical perspective is: can we design tester-learners that accept a wide class of distributions simultaneously, without being tailored to any particular one?
In this work we answer this question in the affirmative by introducing and studying universally testable learning. We formally define this model as follows.
Definition 1.1 (Universally Testable Learning). Let  $\mathcal{C}$  be a concept class mapping  $\mathbb{R}^d$  to  $\{\pm 1\}$ . Let  $\mathcal{D}$  be a family of distributions over  $\mathbb{R}^d$ . Let  $\epsilon, \delta > 0$  be parameters, and let  $\psi : [0,1] \to [0,1]$  be some function. We say  $\mathcal{C}$  can be universally testably learned w.r.t.  $\mathcal{D}$  up to error  $\psi(\mathrm{opt}) + \epsilon$  with failure probability  $\delta$  if there exists a tester-learner  $A$  meeting the following specification. For any distribution  $D_{\mathcal{X}\mathcal{Y}}$  on  $\mathbb{R}^d \times \{\pm 1\}$ ,  $A$  takes in a large sample  $S$  drawn from  $D_{\mathcal{X}\mathcal{Y}}$ , and either rejects  $S$  or accepts and produces a hypothesis  $h: \mathbb{R}^d \to \{\pm 1\}$  such that:
(a) (Soundness.) With probability at least  $1 - \delta$  over the sample  $S$  the following is true:
If  $A$  accepts, then the output  $h$  satisfies  $\mathbb{P}_{(\mathbf{x},y)\sim D_{\mathcal{X}\mathcal{Y}}}[h(\mathbf{x})\neq y]\leq \psi (\mathrm{opt}(\mathcal{C},D_{\mathcal{X}\mathcal{Y}})) + \epsilon$  where  $\mathrm{opt}(\mathcal{C},D_{\mathcal{X}\mathcal{Y}}) = \inf_{f\in \mathcal{C}}\mathbb{P}_{(\mathbf{x},y)\sim D_{\mathcal{X}\mathcal{Y}}}[h(\mathbf{x})\neq y]$ .
(b) (Completeness.) Whenever the marginal of  $D_{\mathcal{X}\mathcal{Y}}$  lies within  $\mathcal{D}$ ,  $A$  accepts with probability at least  $1 - \delta$  over the sample  $S$ .
In this terminology, the original definition of testable learning reduces to the special case where  $\mathcal{D} = \{D^{*}\}$ . We stress that while the prior work of [GKK23] allowed  $D^{*}$  to be, say, any fixed strongly log-concave distribution, their tester-learners are still tailored to the particular  $D^{*}$  that is selected. This is because their tests rely on checking that the unknown distribution closely matches moments with  $D^{*}$ . By contrast, a universal tester-learner must accept all marginals in a family  $\mathcal{D}$ .
Our main contribution in this paper is the first universal tester-learner for the class of halfspaces with respect to a broad family of structured continuous distributions. This family is the set of all distributions with bounded Poincaré constant (see Definition 2.4) and some mild concentration and anti-concentration properties (see Definition 2.1). It captures all strongly log-concave distributions, and in fact, under the well-known Kannan-Lóvasz-Simonovits (KLS) conjecture (see Conjecture 2.6), it captures all log-concave distributions as well. Our universal tester-learner significantly generalizes the main result of [GKSV23], who showed comparable guarantees only for the case where the target marginal is the standard Gaussian.
Theorem 1.2 (Universal Tester-Learner for Halfspaces; formally stated as Theorem 4.1). Let  $\mathcal{C}$  be the class of origin-centered halfspaces over  $\mathbb{R}^d$ . Let  $\mathcal{D}$  be the class of  $\Theta(1)$ -nice and  $\Theta(1)$ -Poincaré distributions (see Definitions 2.1 and 2.4), which includes all isotropic strongly log-concave and, under KLS, all isotropic log-concave distributions. Then  $\mathcal{C}$  can be universally testably learned w.r.t.  $\mathcal{D}$  up to error  $O(\mathrm{opt}) + \epsilon$  in  $\mathrm{poly}(d, \frac{1}{\epsilon})$  time and sample complexity.
A special and well-studied case of interest is when the label noise follows the Massart model, i.e. the label of every example is flipped by an adversary with probability at most  $\eta$ . In this case we are able to handle a considerably larger class  $\mathcal{D}$  while also providing a stronger guarantee.
Theorem 1.3 (Universal Tester-Learner for Massart Halfspaces; formally stated as Theorem 4.1). Let  $\mathcal{C}$  be the class of origin-centered halfspaces over  $\mathbb{R}^d$ . Let  $\mathcal{D}$  be the class of poly  $(d)$ -nice and poly  $(d)$ -Poincaré distributions, which includes all isotropic log-concave distributions (unconditionally). Suppose the label noise follows the Massart model with noise rate at most  $\eta < \frac{1}{2}$ . Then  $\mathcal{C}$  can be universally testably learned w.r.t.  $\mathcal{D}$  up to error opt +  $\epsilon$  in  $\mathrm{poly}(d, \frac{1}{\epsilon}, \frac{1}{1 - 2\eta})$  time and sample complexity.
Technical Overview. We first describe the key reasons why prior tester-learners were tailored to a specific target  $D^{*}$ . All known polynomial-time algorithms for agnostically learning halfspaces up to error  $O(\mathrm{opt}) + \epsilon$  require some concentration and anti-concentration properties from the input marginal distribution (encapsulated e.g. in Definition 2.1). While concentration is relatively straightforward to check (e.g. by checking that the moments do not grow at too fast a rate), the key challenge in designing tester-learners for halfspaces is to check anti-concentration. All prior tester-learners [RV23, GKK23, GKSV23, DKK+23] use the heavy machinery of moment-matching to achieve this. This approach relies on establishing structural properties of the following type: if  $D^{*}$  is a well-behaved distribution (e.g. a strongly log-concave distribution), and  $D$  approximately matches  $D^{*}$  in its low-degree moments, then  $D$  is also well-behaved (in particular, anti-concentrated). A canonical statement of such a property is the main pseudorandomness result of [GKK23] (see Theorem 5.6 therein), which establishes that approximate moment-matching fools functions of a constant number of halfspaces. Applying this property inherently requires comparing the low-degree moments of
$D$  with those of  $D^{*}$ . Such tests do (implicitly) succeed universally for the class of all distributions that match low-degree moments with  $D^{*}$  (e.g., if  $D^{*}$  is the uniform distribution over the hypercube, moment matching would accept all  $k$ -wise independent distributions). Definition 1.1, however, seeks a far broader kind of universality. Our tests are not tailored to a single target in any way, and are intended to succeed over practical classes of distributions that are commonly considered in learning theory (e.g., log-concave distributions).<sup>1</sup>
We overcome this hurdle and design a conceptually simple way of checking anti-concentration without requiring the hammer of moment-matching. Our approach follows and improves on the roadmap used by [GKSV23] to design efficient tester-learners for halfspaces using non-convex SGD (building on [DKTZ20a, DKTZ20b]). Let us outline this approach at a high level (a more detailed technical overview may be found in [GKSV23, Sec 1.2]). The tester-learner first computes a stationary point  $\mathbf{w}$  of a certain smooth version of the ramp loss, a surrogate for the 0-1 loss. Let  $\mathbf{w}^*$  be any solution achieving 0-1 error opt. The tester-learner now checks distributional properties of the unknown marginal  $D$  that ensure that  $\mathbf{w}$  is close in angular distance to  $\mathbf{w}^*$  (specifically, they ensure the contrapositive, namely that any  $\mathbf{w}$  that has large gradient norm must have large angle with  $\mathbf{w}^*$ ). By a more careful analysis of the gradient norm than in [GKSV23] (see Proposition 4.2), we are able to reduce to showing the following weak anti-concentration property. Let  $\mathbf{v}$  denote any unit vector orthogonal to  $\mathbf{w}$ , and let  $D_T$  denote  $D$  restricted to the band  $T = \{\mathbf{x} \mid |\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma\}$  (where the width  $\sigma$  is carefully selected according to certain constraints). Then the property we need is that
$$
\underset {\mathbf {x} \sim D _ {T}} {\mathbb {P}} [ | \langle \mathbf {v}, \mathbf {x} \rangle | \geq \Theta (1) ] \geq \Theta (1).
$$
Our key observation is that the classical Paley-Zygmund inequality applied to the random variable  $Z = \langle \mathbf{v},\mathbf{x}\rangle^{2}$ , where  $\mathbf{x}\sim D_T$ , already gives us the following type of anti-concentration:
$$
\mathbb {P} \left[ Z > \frac {\mathbb {E} [ Z ]}{2} \right] \geq \frac {1}{4} \cdot \frac {\mathbb {E} [ Z ] ^ {2}}{\mathbb {E} [ Z ^ {2} ]}.
$$
This turns out to suffice for our purposes—provided we can show a hypercontractivity property for  $Z$ , namely that  $\mathbb{E}[Z^2] \leq \Theta(1) \mathbb{E}[Z]^2$  (as well as that  $\mathbb{E}[Z] = \Theta(1)$ , which is just a second moment constraint).
Our main algorithmic idea is to use a sum-of-squares (SOS) program to check hypercontractivity of the random variable  $Z$ . To do so, we crucially leverage a result due to [KS17] stating that any  $D$  that has bounded Poincaré constant is certifiably hypercontractive in the SOS framework (and it turns out this extends to  $D_T$  as well). This means that we can run a certain polynomial-time semidefinite program that checks hypercontractivity of  $Z$  over the sample, and whenever  $D$  is in fact Poincaré, we are guaranteed that the test will pass with high probability (see Proposition 3.5). This is sufficient to ensure that the stationary point  $\mathbf{w}$  we have computed is indeed close in angular distance to  $\mathbf{w}^*$ .
In order to finally arrive at our main results, we need to run further tests which ensure that the disagreement between our computed  $\mathbf{w}$  and any (unknown) optimum  $\mathbf{w}^*$  is bounded by the angle between them, i.e.,  $\mathbb{P}_{\mathbf{x} \sim D}[\mathrm{sign}(\langle \mathbf{w}, \mathbf{x} \rangle \neq \mathrm{sign}(\langle \mathbf{w}^*, \mathbf{x} \rangle)] \leq O(\angle (\mathbf{w}, \mathbf{w}^*))$  (see Lemma 3.1). This in turn guarantees that  $\mathbf{w}$  has error  $O(\mathrm{opt}) + \epsilon$ . We stress that while [GKSV23] introduced similar testers for the special case of Gaussian marginals, our tests succeed universally with respect to a broad family of distributions including some heavy-tailed distributions (see Definition 2.1). From a technical perspective, prior to our work, such tests either produced a suboptimal bound, or required estimating the operator norms of a polynomial number of random matrices formed using rejection sampling. We significantly simplify this approach by showing that it is sufficient to estimate the operator norm of a single random matrix. Finally, to obtain our improved results for the Massart setting, it turns out that the proof admits certain simplifications that guarantee final error  $\mathrm{opt} + \epsilon$  while also allowing a wider range of Poincaré distributions.
Related Work. There is a large body of work on agnostic learning algorithms for halfspaces that run in fully polynomial time. We briefly mention only those that are most closely relevant to our work; please see [BH21] for a survey as well as [GKSV23, Sec 1.1] for further related work. Following a
One may wonder if it is possible to test whether the low-degree moments of the input marginal  $D$  match any distribution in a family  $\mathcal{D}$  (e.g., all strongly log-concave distributions) without directly comparing to a specific  $D^{*}$ . This is a reduction to testing whether a given (approximate) low-degree moment tensor lies within a large set of target low-degree moment tensors, and would indeed suffice for universally testable learning. This general problem, however, seems highly challenging to solve directly.
long line of work on distribution-specific agnostic learners for halfspaces [KLS09, ABL17, Dan15, BZ17, YZ17, Zha18, ZSA20, ZL21], the work of [DKTZ20a] introduced a particularly simple approach for the Massart setting, based solely on non-convex SGD. This work, which sets the template that our approach also follows, achieved the information-theoretically optimal error of  $\mathrm{opt} + \epsilon$  for origin-centered Massart halfspaces over a wide range of structured distributions (and was later extended to general halfspaces by  $[\mathrm{DKK}^{+}22])$ . The non-convex SGD approach was then generalized by [DKTZ20b] to show an  $O(\mathrm{opt}) + \epsilon$  guarantee for the fully agnostic setting.
The testable learning model was introduced by the work of [RV23], who showed a tester-learner for halfspaces achieving error opt +  $\epsilon$  in time  $d^{\widetilde{O}(1/\epsilon^4)}$  for the case where the target marginal is Gaussian. Subsequently, [GKK23] provided a general algorithmic framework based on moment-matching for this problem, and showed a tester-learner for halfspaces only requiring time  $d^{\widetilde{O}(1/\epsilon^2)}$  with respect to any fixed strongly log-concave marginal (matching known lower bounds for ordinary agnostic learning over Gaussian marginals [GGK20, DKZ20, DKPZ21, DKR23]).
The most closely relevant work to the present one is that of [GKSV23] (see also  $\mathrm{[DKK^{+}23]}$ ), who showed fully polynomial-time tester-learners for halfspaces achieving error  $O(\mathrm{opt}) + \epsilon$  in the agnostic setting and  $\mathrm{opt} + \epsilon$  in the Massart setting for the case where the target marginal is the Gaussian. As detailed in the technical overview, their tests rely crucially on moment-matching and are tailored to a specific target marginal. By contrast, our tests check hypercontractivity using an SOS program and succeed universally for a wide class of certifiably hypercontractive distributions.
Certifying distributional properties such as hypercontractivity is an important aspect of a large body of work on robust algorithmic statistics using the SOS framework. We will not attempt to summarize this literature here and direct the reader to [KS17, BK21] for overviews of related work, as well as to  $\mathrm{[FKP^{+}19]}$  for a textbook treatment. The notion of certifiable anti-concentration has also been studied (see e.g. [KKK19, RY20, BK21]), but it turns out not to be directly useful for our purposes as it is only known to hold for distributions satisfying very strong conditions such as rotational symmetry.
Limitations and Further Work. Open directions in testable learning (and universally testable learning) include the design of (efficient) tester-learners for concept classes other than the class of halfspaces, e.g., functions of halfspaces or neurons with other activations (like ReLU or sigmoid).
Notation and Terminology. For what follows, we consider  $D_{\mathcal{X}\mathcal{Y}}$  to be an unknown joint distribution over  $\mathcal{X} \times \mathcal{Y}$  from which we receive independent samples, and its marginal on  $\mathcal{X}$  will be denoted by  $D_{\mathcal{X}}$ . In particular  $\mathcal{X} = \mathbb{R}^d$ , and labels will lie in  $\mathcal{Y} = \{\pm 1\}$ . We will use  $\mathcal{C}$  to denote a concept class mapping  $\mathbb{R}^d$  to  $\{\pm 1\}$ , which throughout this paper will be the class of halfspaces or functions of halfspaces over  $\mathbb{R}^d$ . We use  $\mathrm{opt}(\mathcal{C}, D_{\mathcal{X}\mathcal{Y}})$  to denote the optimal error  $\inf_{f \in \mathcal{C}} \mathbb{P}_{(\mathbf{x}, y) \sim D_{\mathcal{X}\mathcal{Y}}} [f(\mathbf{x}) \neq y]$ , or just opt when  $\mathcal{C}$  and  $D_{\mathcal{X}\mathcal{Y}}$  are clear from context. We recall that in Massart noise model, the labels satisfy  $\mathbb{P}_{y \sim D_{\mathcal{X}\mathcal{Y}} | \mathbf{x}}[y \neq \mathrm{sign}(\langle \mathbf{w}^*, \mathbf{x} \rangle) | \mathbf{x}] = \eta(\mathbf{x})$ , with  $\eta(\mathbf{x}) \leq \eta < \frac{1}{2}$  for all  $\mathbf{x}$ . When we have adversarial noise (i.e., when we are in the agnostic model), the labels can be completely arbitrary. In both cases, the goal is to produce a hypothesis whose error is competitive with opt. We use  $\mathbb{E}$  to denote the expectation of a random variable in brackets (or, correspondingly,  $\mathbb{P}$  for the probability of an event), either over the unknown joint distribution or over the empirical distribution with respect to a sample  $S$  (e.g.,  $\mathbb{E}_{Z \in S}[f(Z)] = \frac{1}{|S|} \sum_{Z \in S} f(Z)$ ).
Definitions and Distributional Assumptions. For the problem of learning halfspaces in the agnostic and in Massart noise models, any of the known polynomial algorithms that achieve computationally optimal guarantees require that the marginal distribution has at least the following nice properties previously defined by, e.g., [DKTZ20b].
Definition 2.1 (Nice Distributions). For a given constant  $\lambda \geq 1$ , we consider the class of  $\lambda$ -nice distributions over  $\mathbb{R}^d$  to be the distributions that satisfy the following properties:
1. For any unit vector  $\mathbf{v}$  in  $\mathbb{R}^d$  the distribution satisfies  $\mathbb{E}[\langle \mathbf{v},\mathbf{x}\rangle^2 ]\in [\frac{1}{\lambda},\lambda ]$  .(bounded spectrum)
2. For any two dimensional subspace  $V$ , the corresponding marginal density  $q_{V}(\mathbf{x})$  satisfies  $q_{V}(\mathbf{x}) \geq 1 / \lambda$  for any  $\| \mathbf{x}\|_2 \leq 1 / \lambda$ . (anti-anti-concentration)
3. For any two dimensional subspace  $V$ , the corresponding marginal density  $q_{V}(\mathbf{x})$  satisfies  $q_{V}(\mathbf{x}) \leq Q(\| \mathbf{x} \|_{2})$  for some function  $Q: \mathbb{R}_{+} \to \mathbb{R}_{+}$  such that  $\sup_{r \geq 0} Q(r) \leq \lambda$  and also  $\int_{r=0}^{\infty} r^{k} Q(r) dr \leq \lambda$ , for any  $k = 1, 3, 5$ . (anti-concentration and concentration)
In the testable learning framework, however, corresponding results provide testable guarantees with respect to target marginals that are isotropic strongly log-concave [GKSV23], which is a strictly stronger condition than the one of Definition 2.1 (see Proposition 2.3 below). We now provide the standard definition of (strongly) log-concave distributions.
Definition 2.2 ((Strongly) Log-Concave Distributions [SW14]). We say that a distribution over  $\mathbb{R}^d$  is  $(\beta$ -strongly) log-concave, if its density can be written as  $e^{-\varphi}$ , where  $\varphi$  is a  $(\beta$ -strongly) convex function on  $\mathbb{R}^d$  (for some  $\beta > 0$ ).
Proposition 2.3 (Log-Concave Distributions are Nice [LV07]). There exists a universal constant  $\lambda \geq 1$  such that any isotropic log-concave distribution is  $\lambda$ -nice.
In this work, we provide universally testable guarantees with respect to the class of nice distributions with bounded Poincaré constant (see Definition 2.4 below).
Definition 2.4 (Poincaré Distributions). For a given value  $\gamma >0$ , we say that a distribution over  $\mathbb{R}^d$  is  $\gamma$ -Poincaré, if  $\mathrm{Var}(f(\mathbf{x}))\leq \gamma \cdot \mathbb{E}[\| \nabla f(\mathbf{x})\| _2^2 ]$  for any differentiable function  $f:\mathbb{R}^d\to \mathbb{R}$
Although it is not clear whether one can efficiently obtain testable guarantees for the problem of learning noisy halfspaces under nice marginals (which is known to be an efficiently solvable problem in the non-testable setting [DKTZ20a, DKTZ20b]), by restricting our attention to nice distributions that, additionally, have bounded Poincaré constant, we obtain efficient learning results, even in the universally testable setting. Our results are strictly stronger than the ones in [GKSV23], since we capture isotropic strongly log-concave distributions universally, due to Proposition 2.3 and the fact that strongly log-concave distributions are also Poincaré, as per Proposition 2.5 below.
Proposition 2.5 (Strongly Log-Concave Distributions are Poincaré, [SW14, Proposition 10.1]). Any  $\frac{1}{\gamma}$ -strongly log-concave distribution is  $\gamma$ -Poincaré.
Furthermore, under a long-standing conjecture about the geometry of convex bodies [KLS95], our results capture the family of all isotropic log-concave distributions.
Conjecture 2.6 (Kannan-Lovasz-Simonovits Conjecture [KLS95] reformulation from [LV18]). There is a universal constant  $\gamma >0$  for which any isotropic log-concave distribution is  $\gamma$ -Poincaré.
3 Universal Testers
In this section, we present two basic testers that constitute the basic building blocks of the universal tester-learners we provide in the next section. The testers in this section might be of independent interest and their appeal is that they succeed even when the distribution in their input is unspecified up to certain bounds on a number of its statistics. In fact, the family of distributions for which each such tester succeeds is of infinite size, even non-parametric.
3.1 Universal Tester for Bounding Local Halfspace Disagreement
First, we present a universal tester that checks, given a parameter vector  $\mathbf{w}$ , whether a set of samples  $S$  is such that bounding the angular distance of  $\mathbf{w}$  from an optimum parameter vector, implies that the corresponding halfspace disagrees with the (unknown) optimum halfspace only on a bounded fraction of points in  $S$ . This property ensures that if  $\mathbf{w}$  is close to the optimum parameter vector, then it is also an approximate empirical risk minimizer. The tester universally accepts samples from nice distributions with high probability (Definition 2.1).