SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 3.1 (Universally Testable Bound for Local Halfspace Disagreement). Let  $D_{\mathcal{X}\mathcal{Y}}$  be a distribution over  $\mathbb{R}^d \times \{\pm 1\}$ ,  $\mathbf{w} \in \mathbb{S}^{d - 1}$ ,  $\theta \in (0,\pi /4]$ ,  $\lambda \geq 1$  and  $\delta \in (0,1)$ . Then, for a sufficiently large constant  $C$ , there is a tester that given  $\delta$ ,  $\theta$ ,  $\mathbf{w}$  and a set  $S$  of samples from  $D_{\mathcal{X}}$  with size at least  $C \cdot \left(\frac{d^4}{\theta^2\delta}\right)$ , runs in time poly  $(d,\frac{1}{\theta},\frac{1}{\delta})$  and satisfies the following specifications:
(a) If the tester accepts  $S$ , then for every unit vector  $\mathbf{w}' \in \mathbb{R}^n$  satisfying  $\angle(\mathbf{w}, \mathbf{w}') \leq \theta$  we have
$$
\underset {\mathbf {x} \sim S} {\mathbb {P}} [ \mathrm {s i g n} (\langle \mathbf {w} ^ {\prime}, \mathbf {x} \rangle) \neq \mathrm {s i g n} (\langle \mathbf {w}, \mathbf {x} \rangle) ] \leq C \cdot \theta \cdot \lambda^ {C}
$$
(b) If the distribution  $D_{\mathcal{X}}$  is  $\lambda$ -nice, the tester accepts  $S$  with probability  $1 - \delta$ .
The proof of Lemma 3.1 simplifies and improves the proof of a similar but weaker result in [GKSV23] (see their Proposition 4.5). The initial tester exploited the observation that the probability of disagreement between two halfspaces can be upper bounded by a sum of products, where each product has two terms: one corresponding to the probability of falling in a (known) strip orthogonal to  $\mathbf{w}$  and one corresponding to the probability of having large enough inner product with some unknown vector orthogonal to  $\mathbf{w}$ , conditioned in the (known) strip. The first term can be controlled by estimating the probability of falling in a (known) strip, while the second follows by Chebyshev's inequality, after estimating the largest eigenvalue of the covariance matrix conditioned in the known strip. This approach introduces a number of complications, including the fact that conditioning requires rejection sampling, which, in turn requires a lower bound on the probability of falling inside each strip. We propose a simpler tester that controls all of the terms of the sum simultaneously by estimating the largest eigenvalue of a single covariance matrix (without conditioning). Upper and lower bounds on the eigenvalues of random symmetric matrices can be universally tested with testers that are guaranteed to accept when the elements of the matrix have bounded second moments (spectral tester of Proposition A.2). We present our full proof in Appendix B.1.
3.2 Universally Testable Weak Anti-Concentration
We now provide an important universal tester, which ensures that for a given vector  $\mathbf{w}$ , a sample set  $S$  and any unknown unit vector  $\mathbf{v}$  orthogonal to  $\mathbf{w}$ , among the samples falling within a (known) strip orthogonal to  $\mathbf{w}$ , at least a constant fraction is absolutely correlated with  $\mathbf{v}$  by a constant. In other words, the tester ensures that the conditional empirical distribution is weakly anti-concentrated in every direction. The tester universally accepts nice distributions that have bounded Poincaré constant.
Lemma 3.2 (Universally Testable Weak Anti-Concentration). Let  $D$  be a distribution over  $\mathbb{R}^d$ . Then, there is a universal constant  $C > 0$  and a tester that given a unit vector  $\mathbf{w} \in \mathbb{R}^d$ ,  $\delta \in (0,1)$ ,  $\gamma > 0$ ,  $\lambda \geq 1$ ,  $\sigma \leq \frac{1}{2\lambda}$  and a set  $S$  of i.i.d. samples from  $D$  with size at least  $C \cdot \frac{d^4}{\sigma^2\delta} \log(d) \lambda^C$ , runs in time poly $(d,\lambda,\frac{1}{\sigma},\frac{1}{\delta},\log(\frac{1}{\gamma}))$  and satisfies the following specifications
(a) If the tester accepts  $S$ , then for any unit vector  $\mathbf{v} \in \mathbb{R}^d$  with  $\langle \mathbf{v}, \mathbf{w} \rangle = 0$  we have
$$
\underset {\mathbf {x} \in S} {\mathbb {P}} \left[ | \langle \mathbf {v}, \mathbf {x} \rangle | \geq \frac {1}{C \lambda^ {C}} \left| | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \sigma \right. \right] \geq \frac {1}{C \lambda^ {C} \gamma^ {4}}
$$
(b) If  $D$  is  $\gamma$ -Poincaré and  $\lambda$ -nice, then the tester accepts  $S$  with probability at least  $1 - \delta$ .
The proof of Lemma 3.2 is based on a simple fact from probability that is true for any non-negative random variable and ensures that the mass assigned to the tails is lower bounded by the ratio of the square of its expectation to the second moment.
Proposition 3.3 (Paley-Zygmund Inequality). For any non-negative random variable  $Z$ , we have
$$
\mathbb {P} [ Z > \mathbb {E} [ Z ] / 2 ] \geq \frac {1}{4} \cdot \frac {\mathbb {E} [ Z ] ^ {2}}{\overline {{\mathbb {E}}} [ Z ^ {2} ]}
$$
In the special case where  $Z$  follows the distribution of  $\langle \mathbf{v},\mathbf{x}\rangle^2$  conditioned on  $|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma$  for some unitary orthogonal vectors  $\mathbf{v}$ ,  $\mathbf{w}$ , some  $\sigma >0$  and some random variable  $\mathbf{x}$  whose distribution is, say, 1-nice (see Definition 2.1), one can show that  $\mathbb{E}[Z]$  is lower bounded by a constant and  $\mathbb{E}[Z^2 ]$  is upper bounded by another constant, so  $Z$  assigns a non-trivial mass to a set that is bounded away from zero. This property is useful in the context of learning noisy halfspaces, as we show in the following section (see Proposition 4.2 and Lemma 4.3). However, testing algorithms that check whether such a property holds for given  $\mathbf{w}$  and  $\sigma$ , are guaranteed to succeed when the marginal distribution has, additionally, bounded Poincaré constant. The main part of the proof that requires a bounded Poincaré constant, is testing whether  $\mathbb{E}[Z^2 ]$  is bounded uniformly over the set of unit vectors  $\mathbf{v}$  orthogonal to  $\mathbf{w}$ , since  $Z^{2} = \langle \mathbf{v},\mathbf{x}\rangle^{4}$ , where  $\mathbf{v}$  is unknown. We use the following result from [KS17].
Proposition 3.4 (Certifiable Hypercontractivity of Poincaré Distributions, Theorem 4.1 in [KS17]). Let  $\delta \in (0,1)$ ,  $\gamma > 0$  and let  $D$  be a  $\gamma$ -Poincaré distribution over  $\mathbb{R}^d$ . Let  $S$  be a set of independent samples from  $D$  with size at least  $(2d\log(4d/\delta))^4$ . Consider the constrained maximization problem
$$
\arg \max  _ {\| \mathbf {v} \| _ {2} = 1} \mathbb {E} _ {\mathbf {x} \in S} [ \langle \mathbf {v}, \mathbf {x} \rangle^ {4} ] \tag {3.1}
$$
Then, the optimum solution of the degree-4 sum-of-squares relaxation of the problem (3.1) has value at most  $C\gamma^4$  for some universal constant  $C$ , with probability at least  $1 - \delta$  over the sample  $S$ .
Using Proposition 3.4, we are able to provide a universal tester for bounding the empirical fourth moments. The tester solves an appropriate SDP relaxation of the (hard) problem [HL13] of finding the direction with maximum fourth moment and is guaranteed to succeed if  $\mathbf{x}$  has Poincaré parameter bounded by a known value.
Proposition 3.5 (Hypercontractivity Tester). Let  $D$  be a distribution over  $\mathbb{R}^d$ . Then, there is a tester that given  $\delta \in (0,1)$ ,  $\gamma > 0$  and a set  $S$  of i.i.d. samples from  $D$  with size at least  $(2d\log(4d/\delta))^4$ , runs in time  $\mathrm{poly}(d, \log \frac{1}{\delta}, \log \frac{1}{\gamma})$  and satisfies the following specifications
(a) If the tester accepts  $S$ , then for any unit vector  $\mathbf{v} \in \mathbb{R}^d$  we have
$$
\underset {\mathbf {x} \in S} {\mathbb {E}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {4} \right] \leq C \cdot \gamma^ {4}, w h e r e C i s o s e m o u n i v e r s a l c o n s t a n t.
$$
(b) If the distribution  $D$  is  $\gamma$ -Poincaré, then the tester accepts  $S$  with probability at least  $1 - \delta$ .
Proof. The tester does the following:
1. Solves a degree-4 sum-of-squares relaxation of problem (3.1) up to accuracy  $\gamma^4$ . (For a formal definition of the relaxed problem, see Problem (2.3) in [KS17].)
2. If the solution has value larger than  $(C - 1)\gamma^4$ , then reject. Otherwise accept.
The computational complexity of the tester is poly  $(|S|,d,\log \frac{1}{\gamma})$  , since the problem it solves can be written as a semidefinite program [Sho87, Par00, Nes00, Las01].
If the tester accepts  $S$ , then we know that the optimal solution of the relaxed problem is at most  $C\gamma^4$  and we also know that any solution of the initial problem (3.1) has value at most equal to the value of the relaxation. Therefore  $\mathbb{E}[\langle \mathbf{v},\mathbf{x}\rangle^4 ]\leq C\gamma^4$ , for any  $\mathbf{v}\in \mathbb{S}^{d - 1}$ .
On the other hand, if the true distribution  $D$  is  $\gamma$ -Poincaré, then, with probability at least  $1 - \delta$ , we have that the solution found in step 3.2 has, with probability at least  $1 - \delta$ , value at most  $C' \gamma^4$  for some universal constant  $C'$ , due to Proposition 3.4. In order to ensure that the tester will accept with probability at least  $1 - \delta$ , it suffices to pick  $C = C' + 1$ .
We provide the full proof of Lemma 3.2, in Appendix B.2. The tests we perform include a spectral tester that accepts with high probability when the distribution of  $\mathbf{x}$  is nice (similar to the spectral tester used for Lemma 3.1), a tester of the probability that  $|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma$  and the hypercontractivity tester of Proposition 3.5.
4 Universal Tester-Learners for Halfspaces
In this section, we present our main result on universally testable learning of halfspaces.
Theorem 4.1 (Efficient Universal Tester-Learner for Halfspaces). Let  $D_{\mathcal{X}\mathcal{Y}}$  be any distribution over  $\mathbb{R}^d \times \{\pm 1\}$ . Let  $\mathcal{C}$  be the class of origin centered halfspaces in  $\mathbb{R}^d$ . Then, for any  $\lambda \geq 1$ ,  $\gamma > 0$ ,  $\epsilon > 0$  and  $\delta \in (0,1)$ , there exists an universal tester-learner for  $\mathcal{C}$  w.r.t. the class of  $\lambda$ -nice and  $\gamma$ -Poincaré marginals up to error  $\mathrm{poly}(\lambda) \cdot (1 + \gamma^4) \cdot \mathrm{opt} + \epsilon$ , where  $\mathrm{opt} = \min_{\mathbf{w} \in \mathbb{S}^{d - 1}} \mathbb{P}_{D_{\mathcal{X}\mathcal{Y}}}[y \neq \mathrm{sign}(\langle \mathbf{w}, \mathbf{x} \rangle)]$ , and error probability at most  $\delta$ , using a number of samples and running time  $\mathrm{poly}(d, \lambda, \gamma, \frac{1}{\epsilon}, \log \frac{1}{\delta})$ .
Moreover, if the noise is Massart with given rate  $\eta < 1/2$ , then the algorithm achieves error opt + ε with time and sample complexity poly(d, λ, γ,  $\frac{1}{\epsilon}$ ,  $\frac{1}{1 - 2\eta}$ , log  $\frac{1}{\delta}$ ).
Our proof follows a surrogate loss minimization approach that has been used for classical learning of noisy halfspaces [DKTZ20a, DKTZ20b] as well as classical (non-universal) testable learning [GKSV23]. In particular, the algorithm runs Projected Stochastic Gradient Descent (see A.5) on a surrogate loss whose stationary points are shown to be close to optimum parameter vectors under certain distributional assumptions. In the regular testable learning setting, given a stationary point, the above property can be tested with respect to any (fixed and known) target strongly log-concave marginal as shown by [GKSV23]. For such a stationary point, more tests are used in order to ensure
bounds on local halfspace disagreement. We provide some delicate refinements of the proofs in [GKSV23] that enable us to substitute their testers with the universal testers of Section 3.
We use the following surrogate loss function which was also used in [GKSV23].
$$
\mathcal {L} _ {\sigma} (\mathbf {w}; D _ {\mathcal {X Y}}) = \underset {(\mathbf {x}, y) \sim D _ {\mathcal {X Y}}} {\mathbb {E}} \left[ \ell_ {\sigma} \left(- y \frac {\langle \mathbf {w} , \mathbf {x} \rangle}{\| \mathbf {w} \| _ {2}}\right) \right], \tag {4.1}
$$
In Equation (4.1), the function  $\ell_{\sigma}$  is a smoothed version of the step function as in Proposition A.4.
In order to analyze the properties of the stationary points of the surrogate loss, we provide the following refinement of results implicit in [GKSV23, DKTZ20a, DKTZ20b]. We show that the gradient of the surrogate loss is lower bounded by the difference between certain quantities that are controlled by the marginal distribution (see Figure 1). We stress that we do not use any assumptions for the marginal distribution in this step. Prior work included similar bounds, but the corresponding quantities were different. We need to be more precise and provide the following result, whose proof is based on two dimensional geometry and can be found in Appendix C.1.
Proposition 4.2 (Modification from [GKSV23, DKTZ20a, DKTZ20b]). For a distribution  $D_{\mathcal{X}\mathcal{Y}}$  over  $\mathbb{R}^d\times \{\pm 1\}$  let opt be the minimum error achieved by some origin-centered halfspace and  $\mathbf{w}^{*}\in \mathbb{S}^{d - 1}$  a corresponding vector. Consider  $\mathcal{L}_{\sigma}$  as in Equation (4.1) for  $\sigma >0$  and let  $\eta < 1 / 2$ . Let  $\mathbf{w}\in \mathbb{S}^{d - 1}$  with  $\angle (\mathbf{w},\mathbf{w}^{*}) = \theta < \frac{\pi}{2}$  and  $\mathbf{v}\in \mathrm{span}(\mathbf{w},\mathbf{w}^{*})$  such that  $\langle \mathbf{v},\mathbf{w}\rangle = 0$  and  $\langle \mathbf{v},\mathbf{w}^{*}\rangle < 0$ . Then, for some universal constant  $C > 0$  and any  $\alpha \geq \frac{\sigma}{2\tan\theta}$  we have  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};D_{\mathcal{X}\mathcal{Y}})\| _2\geq A_1 - A_2 - A_3,$  where
$$
\begin{array}{l} A _ {1} = \frac {\alpha}{C \cdot \sigma} \cdot \mathbb {P} \left[ | \langle \mathbf {v}, \mathbf {x} \rangle | \geq \alpha a n d | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{6} \right] \\ A _ {2} = \frac {C}{\tan \theta} \cdot \mathbb {P} \left[ | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{2} \right] a n d A _ {3} = \frac {C}{\sigma} \cdot \sqrt {\mathrm {o p t}} \cdot \sqrt {\mathbb {E} \left[ \langle \mathbf {v} , \mathbf {x} \rangle^ {2} \cdot \mathbb {1} _ {\{| \langle \mathbf {w} , \mathbf {x} \rangle | \leq \frac {\sigma}{2} \}} \right]} \\ \end{array}
$$
Moreover, if the noise is Massart with rate  $\eta$ , then  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};D_{\mathcal{X}\mathcal{Y}})\| _2\geq (1 - 2\eta)A_1 - A_2$
If the marginal distribution is nice, then the quantities  $A_{1}, A_{2}$  and  $A_{3}$  are such that  $\sigma$  can be chosen accordingly so that stationary points of the surrogate loss (or their inverses) are close to some optimum vector (see Proposition A.3 for properties of nice distributions). We use some simple tests (e.g., estimate the probability of falling in a strip,  $\mathbb{P}[|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma /2]$  and appropriate spectral testers) as well as our universal tester for weak anti-concentration (see 3.2) to establish bounds on quantities  $A_{1}, A_{2}$  and  $A_{3}$  which ensure that the desired property holds for a given vector  $\mathbf{w}$ , under no distributional assumptions. The tester in the following result universally accepts nice distributions with bounded Poincaré parameter. The formal proof can be found in Appendix C.2.
Lemma 4.3 (Universally Testable Structure of Surrogate Loss). Let  $D_{\mathcal{X}\mathcal{Y}}$  be any distribution over  $\mathbb{R}^d \times \{\pm 1\}$ . Consider  $\mathcal{L}_{\sigma}$  as in Equation (4.1). Then, there is a universal constant  $C > 0$  and a tester that given a unit vector  $\mathbf{w} \in \mathbb{R}^d$ ,  $\delta \in (0,1)$ ,  $\eta < 1/2$ ,  $\gamma > 0$ ,  $\lambda \geq 1$ ,  $\sigma \leq \frac{1}{C\lambda^c}$  and a set  $S$  of i.i.d. samples from  $D_{\mathcal{X}\mathcal{Y}}$  with size at least  $C \cdot \frac{d^4}{\sigma^2\delta} \log(d)\lambda^C$ , runs in time  $\mathrm{poly}(d,\lambda,\frac{1}{\sigma},\frac{1}{\delta},\log(\frac{1}{\gamma}))$  and satisfies the following specifications
(a) If the tester accepts  $S$ , then, the following statements are true for the minimum error  $\mathsf{opt}_S$  achieved by some origin-centered halfspace on  $S$  and the optimum vector  $\mathbf{w}_S^* \in \mathbb{S}^{d-1}$
- If the noise is Massart with associated rate  $\eta$  and  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S)\| _2\leq \frac{1 - 2\eta}{C\lambda^C\gamma^4}$  then either  $\angle (\mathbf{w},\mathbf{w}_S^*)\leq \frac{C\lambda^C(1 + \gamma^4)}{1 - 2\eta}\cdot \sigma$  or  $\angle (-\mathbf{w},\mathbf{w}_S^*)\leq \frac{C\lambda^C(1 + \gamma^4)}{1 - 2\eta}\cdot \sigma$
- If the noise is adversarial with  $\mathrm{opt}_S \leq \frac{\sigma}{C\lambda^C}$  and  $\|\nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S)\|_2 < \frac{1}{C\lambda^C\gamma^4}$  then either  $\angle (\mathbf{w},\mathbf{w}_S^*) \leq C\lambda^C (1 + \gamma^4) \cdot \sigma$  or  $\angle (-\mathbf{w},\mathbf{w}_S^*) \leq C\lambda^C (1 + \gamma^4) \cdot \sigma$ .
(b) If the marginal  $D_{\mathcal{X}}$  is  $\lambda$ -nice and  $\gamma$ -Poincaré, then the tester accepts  $S$  with probability at least  $1 - \delta$ .
We now give the algorithm for  $\delta \gets 1/3$  since we can reduce the probability of failure with repetition (repeat  $O(\log \frac{1}{\delta})$  times, accept if the rate of acceptance is  $\Omega(1)$  and output the halfspace achieving the minimum test error among the halfspaces returned).
The algorithm receives  $\lambda \geq 1, \gamma > 0, \epsilon > 0$  and  $\eta \in (0, 1/2) \cup \{1\}$  (say  $\eta = 1$  when we are in the agnostic case) and does the following for some appropriately large universal constants  $C_1, C_2 > 0$ .
1. First, initialize  $E = \frac{\epsilon}{C_1\lambda^{C_1}}$ , and let  $\Sigma$  be a list of real numbers and  $A$  be a positive real number, where  $\Sigma$  and  $A$  are defined as follows. If  $\eta = 1$ , then  $\Sigma$  is an  $\frac{E}{C_1\lambda^{C_1}}$ -cover of the interval  $\left[0, \frac{1}{C_1\lambda^{C_1}}\right]$  and  $A = \frac{1}{C_1\lambda^{C_1}\gamma^4}$ . Otherwise, let  $\Sigma = \left\{\frac{E \cdot (1 - 2\eta)}{C_1\lambda^{C_1}(1 + \gamma^4)}\right\}$  and  $A = \frac{1 - 2\eta}{C_1\lambda^{C_1}\gamma^4}$ .
2. Draw a set  $S_{1}$  of  $C_2\left(\frac{\lambda d}{\gamma\epsilon}\right)^{C_2}$  i.i.d. samples from  $D_{\mathcal{X}\mathcal{Y}}$  and run PSGD, as specified in Proposition A.5 with  $\epsilon \gets A$ ,  $\delta \gets \frac{\delta}{C_1}$  on the loss  $\mathcal{L}_{\sigma}$  for each  $\sigma \in \Sigma$ .
3. Form a list  $L$  with all the pairs of the form  $(\mathbf{w},\sigma)$  where  $\mathbf{w}\in \mathbb{S}^{d - 1}$  is some iterate of the PSGD subroutine performed on  $\mathcal{L}_{\sigma}$ .
4. Draw a fresh set  $S_{2}$  of  $C_2\left(\frac{\lambda d}{\gamma\epsilon}\right)^{C_2}$  i.i.d. samples from  $D_{\mathcal{X}\mathcal{Y}}$  and compute for each  $(\mathbf{w},\sigma)\in L$  the value  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S_{2})\|_{2}$ . If, for some  $\sigma \in \Sigma$ ,  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S_2)\| _2 > A$  for all  $(\mathbf{w},\sigma)\in L$ , then reject.
5. Update  $L$  by keeping for each  $\sigma \in \Sigma$  only one pair of the form  $(\mathbf{w},\sigma)$  for which we have  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S_2)\| _2\leq A$ .
6. Run the following tests for each  $(\mathbf{w},\sigma)\in L$ . (This will ensure that part (a) of Lemma 4.3 holds for each of the elements of  $L$ , i.e., that any stationary point of the loss  $\mathcal{L}_{\sigma}$  that lies in  $L$  is angularly close to the empirical risk minimizer<sup>2</sup>.)
- If  $\mathbb{P}_{(\mathbf{x},y) \in S_2} [|\langle \mathbf{w}, \mathbf{x} \rangle| \leq \frac{\sigma}{6}] \leq \frac{\sigma}{C_1 \lambda^{C_1}}$  or  $\mathbb{P}_{(\mathbf{x},y) \in S_2} [|\langle \mathbf{w}, \mathbf{x} \rangle| \leq \frac{\sigma}{2}] > \sigma \cdot C_1 \lambda^{C_1}$ , then reject.
- Compute the  $(d - 1) \times (d - 1)$  matrices  $M_{S_2}^+$  and  $M_{S_2}^-$  as follows:3
$$
M _ {S _ {2}} ^ {+} = \underset {(\mathbf {x}, y) \in S _ {2}} {\mathbb {E}} \left[ (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T} \cdot \mathbb {1} _ {\{| \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{2} \}} \right]
$$
$$
M _ {S _ {2}} ^ {-} = \mathbb {E} _ {(\mathbf {x}, y) \in S _ {2}} \left[ (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T} \cdot \mathbb {1} _ {\{| \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{6} \}} \right]
$$
- Reject if the maximum singular value of  $M_{S_2}^+$  is greater than  $\sigma \cdot C_1\lambda^{C_1}$ .
- Reject if the minimum singular value of  $M_{S_2}^-$  is less than  $\frac{\sigma}{C_1 \lambda^{C_1}}$ .
- Run the hypercontractivity tester on  $S' = \{\mathrm{proj}_{\perp \mathbf{w}} \mathbf{x} : (\mathbf{x}, y) \in S_2 \text{ and } |\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma\}$ , i.e., solve an appropriate SDP (see Prop. 3.5 with  $\gamma \gets \gamma$ ,  $\delta \gets \delta / C_1$ ) and reject if the solution is larger than a specified threshold.
7. Set  $\theta = \frac{(1 + \gamma^4)\sigma}{A\gamma^4}$ , and run the following tests for each pair of the form  $(\mathbf{w},\sigma)$  and  $(- \mathbf{w},\sigma)$  where  $(\mathbf{w},\sigma) \in L$ . (This will ensure that part (a) of Lemma 3.1 is activated, i.e., that the distance of a vector from the empirical risk minimizer is an accurate proxy for the error of the corresponding halfspace.)
- If  $\mathbb{P}_{(\mathbf{x},y)\in S_2}[|\langle \mathbf{w},\mathbf{x}\rangle |\leq \theta ] > C_1\lambda^{C_1}\theta$  then reject.
- Compute the  $(d - 1) \times (d - 1)$  matrix  $M_{S_2}$  as follows:4
$$
M _ {S _ {2}} = \underset {(\mathbf {x}, y) \in S _ {2}} {\mathbb {E}} \left[ \sum_ {i = 2} ^ {\infty} \frac {(\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T}}{(i - 1) ^ {2}} \mathbb {1} \{| \langle \mathbf {w}, \mathbf {x} \rangle | \in [ (i - 1) \theta , i \theta) \} \right]
$$
- If  $\| M_S\|_{\mathrm{op}} > C_1\theta \lambda^{C_1}$ , then reject.
8. Otherwise, accept and output the vector  $\mathbf{w}$  that achieves the smallest empirical error on  $S_{2}$  among the vectors in the list  $L$ .
This concludes the algorithm. The full proof of Theorem 4.1 may be found in Appendix C.3.
2Or the same holds for the inverse vector.
3The operator  $\mathrm{proj}_{\perp \mathbf{w}}:\mathbb{R}^d\to \mathbb{R}^{d - 1}$  projects vectors on the hyperplane orthogonal to  $\mathbf{w}$ .
Note that only at most  $|S_{2}|$  many terms below are non-zero, hence  $M_{S_2}$  can be computed efficiently.
We wish to thank the anonymous reviewers of NeurIPS 2023 for their constructive feedback. Aravind Gollakota was at UT Austin while this work was done, supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Adam R. Klivans was supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Konstantinos Stavropoulos was supported by NSF award AF-1909204, the NSF AI Institute for Foundations of Machine Learning (IFML), and by scholarships from Bodossaki Foundation and Leventis Foundation. Arsen Vasilyan was supported in part by NSF awards CCF-2006664, DMS-2022448, CCF-1565235, CCF-1955217, CCF-2310818, Big George Fellowship and Fintech@CSAIL. Part of this work was done while Arsen Vasilyan was visiting UT Austin.