SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use  $\mathbf{E}_{\mathbf{x}\sim D}[\mathbf{x}]$  for the expectation of the random variable  $\mathbf{x}$  according to the distribution  $D$  and  $\mathbf{Pr}[E]$  for the probability of event  $E$ . For simplicity of notation, we may omit the distribution when it is clear from the context. For  $\mu \in \mathbb{R}^d$ ,  $\boldsymbol{\Sigma} \in \mathbb{R}^{d\times d}$ , we denote by  $\mathcal{N}(\mu, \boldsymbol{\Sigma})$  the  $d$ -dimensional Gaussian distribution with mean  $\mu$  and covariance  $\boldsymbol{\Sigma}$ . For  $(\mathbf{x},y) \in \mathcal{X}$  distributed according to  $D$ , we denote  $D_{\mathbf{x}}$  to be the marginal distribution of  $\mathbf{x}$ . Let  $f: \mathbb{R}^d \mapsto \{\pm 1\}$  be a boolean function and  $D$  a distribution over  $\mathbb{R}^d$ . The degree-1 Chow parameter vector of  $f$  with respect to  $D$  is defined as  $\mathbf{E}_{\mathbf{x}\sim D}[f(\mathbf{x})\mathbf{x}]$ . For a halfspace  $h(\mathbf{x}) = \mathrm{sign}(\mathbf{v} \cdot \mathbf{x})$ , we say that  $\mathbf{v}$  is the defining vector of  $h$ .
Moment-Matching In what follows, we use the phrase "A distribution  $D$  on  $\mathbb{R}^d$  matches  $k$  moments with a distribution  $Q$  up to error  $\Delta$ ". Similarly to [GKK22], we formally define approximate moment-matching as follows.
Definition 1.4 (Approximate Moment-Matching). Let  $k \in \mathbb{N}$  be a degree parameter and let  $\mathcal{M}(k, d)$  be the set of  $d$ -variate monomials of degree up to  $k$ . Moreover, let  $\Delta \in \mathbb{R}_+^{|\mathcal{M}(k, d)|}$  be a slack parameter (indexed by the monomials of  $\mathcal{M}(k, d)$ ), satisfying  $\Delta_0 = 0$ . We say that two distributions  $D, Q$  match  $k$  moments up to error  $\Delta$  if  $|\mathbf{E}_{\mathbf{x} \sim D}[m(\mathbf{x})] - \mathbf{E}_{\mathbf{x} \sim Q}[m(\mathbf{x})]| \leq \Delta_m$  for every monomial  $m(\mathbf{x}) \in \mathcal{M}(k, d)$ . When the error bound  $\Delta$  is the same for all monomials we overload notation and simply use  $\Delta$  instead of the parameter  $\Delta$ .
2 Weak Testable Proper Agnostic Learning
As our starting point, we give an algorithm that performs testable proper learning of homogeneous halfspaces in the presence of adversarial label noise with respect to the Gaussian distribution. The main result of this section is given below and its proof can be found in Appendix A.
Proposition 2.1 (Proper Testable Learner with Adversarial Label Noise). Let  $D$  be a distribution on labeled examples  $(\mathbf{x},y)\in \mathbb{R}^d\times \{\pm 1\}$  with  $\mathbf{x}$ -marginal  $D_{\mathbf{x}}$ . Suppose that there exists a unit vector  $\mathbf{v}^{*}\in \mathbb{R}^{d}$  such that  $\operatorname{Pr}_{(\mathbf{x},y)\sim D}[\mathrm{sign}(\mathbf{v}^{*}\cdot \mathbf{x})\neq y]\leq \mathrm{opt}$ . There exists an algorithm (Algorithm 2) that given  $\tau, \eta \in (0,1)$ ,  $N = d^{\tilde{O}(1/\eta^2)}\log(1/\tau)$  i.i.d. unlabeled samples from  $D_{\mathbf{x}}$ , queries the labels of  $O(d/\eta^2)\log(d/\tau)$  of them, and runs in time  $\mathrm{poly}(d,N)$  and does one of the following:
- The algorithm reports that the  $\mathbf{x}$ -marginal of  $D$  is not  $\mathcal{N}(\mathbf{0},\mathbf{I})$ .
- The algorithm outputs a unit vector  $\mathbf{w} \in \mathbb{R}^d$ .
With probability at least  $1 - \tau$  the following holds: (1) if the algorithm reports anything, the report is correct, and (2) if the algorithm returns a vector  $\mathbf{w}$ , it holds  $\| \mathbf{v}^* - \mathbf{w}\|_2 \leq C_{\mathrm{A}} \sqrt{\mathrm{opt} + \eta}$ , where  $C_{\mathrm{A}} > 0$  is an absolute constant.
A couple of remarks are in order. First, notice that if the algorithm outputs a vector  $\mathbf{w}$ , we only have the guarantee that  $\| \mathbf{v}^* - \mathbf{w}\|_2$  is small — instead of that the hypothesis halfspace  $h_{\mathbf{w}}(\mathbf{x}) = \mathrm{sign}(\mathbf{w} \cdot \mathbf{x})$  achieves small 0-1 error. Nonetheless, as we will show in the next section, conditioned on  $D$  passing some test, the error of the halfspace  $h_{\mathbf{w}}$  will be at most opt plus a constant multiple of  $\| \mathbf{v}^* - \mathbf{w}\|_2$  (see Lemma 3.1). Second, unlike the testable improper learners in [RV23, GKK22] — which achieve error of  $\mathrm{opt} + \eta$  with similar running time and sample complexity — our testable proper learner achieves the weaker error guarantee of  $O(\sqrt{\mathrm{opt} + \eta})$ . This suffices for our purposes for the following reason: in the context of our localization-based approach, we only need an efficient proper weak learner that achieves sufficiently small constant error. This holds for our proper testable learner, as long as both opt and  $\eta$  are bounded above by some other sufficiently small constant.
To obtain a proper learner, we proceed to directly estimate the defining vector  $\mathbf{v}^*$  of the target halfspace  $h^{*}(\mathbf{x}) = \mathrm{sign}(\mathbf{v}^{*}\cdot \mathbf{x})$ , where we assume without loss of generality that  $\mathbf{v}^*$  is a unit vector. The following simple fact relating the degree-1 Chow-parameters of a homogeneous halfspace and its defining vector will be useful for us.
Fact 2.2 (see, e.g., Lemma 4.3 of [DKS18]). Let  $\mathbf{v}$  be a unit vector and  $h(\mathbf{x}) = \mathrm{sign}(\mathbf{v} \cdot \mathbf{x})$  be the corresponding halfspace. If  $\mathbf{x}$  is from  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , then we have that  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}(\mathbf{0},\mathbf{I})}[h(\mathbf{x})\mathbf{x}] = \sqrt{2 / \pi}\mathbf{v}$ .
To apply Fact 2.2 in our context, we need to overcome two hurdles: (i) the  $\mathbf{x}$  marginal of  $D$  is not necessarily the standard Gaussian, and (ii) the labels are not always consistent with  $h^{*}(\mathbf{x})$ . The second issue can be circumvented by following the approach of [DKS18]. In particular, if the  $\mathbf{x}$  marginal of  $D$  is indeed Gaussian, we can just treat  $D$  as a corrupted version of  $(\mathbf{x}, h^{*}(\mathbf{x}))$ , where  $\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$  and estimate the Chow parameters robustly.
To deal with the first issue, we borrow tools from [GKK22]. At a high level, we certify that the low-degree moments of  $D_{\mathbf{x}}$  — the  $\mathbf{x}$  marginal of  $D$  — approximately match the corresponding moments of  $\mathcal{N}(0,\mathbf{I})$  before estimating the Chow parameters. Importantly, this testing procedure uses only unlabeled samples. To establish the correctness of our algorithm, we show that, for any distribution  $B$  that passes the moment test, the Chow parameters of a halfspace under  $B$  will still be close to its defining vector. We state the lemma formally below. Its proof can be found in Appendix A.
Lemma 2.3 (From Moment-Matching to Chow Distance). Fix  $\eta >0$ . Let  $k = C\log (1 / \eta) / \eta^2$  and  $\Delta = \frac{1}{kd^k}\left(\frac{1}{C\sqrt{k}}\right)^{k + 1}$ , where  $C > 0$  is a sufficiently large absolute constant. Let  $B$  be a
distribution whose moments up to degree  $k$  match with those of  $\mathcal{N}(\mathbf{0},\mathbf{I})$  up to additive error  $\Delta$ . Let  $h(\mathbf{x}) = \mathrm{sign}(\mathbf{v}\cdot \mathbf{x})$  be a halfspace. Then we have that  $\left\| \mathbf{E}_{\mathbf{x}\sim B}[h(\mathbf{x})\mathbf{x}] - \sqrt{\frac{2}{\pi}}\mathbf{v}\right\|_2\leq O(\sqrt{\eta})$ .
With Lemma 2.3 in hand, we know it suffices to estimate the Chow parameters of  $h^*$  with respect to  $D_{\mathbf{x}}$ . This would then give us a good approximation to  $\mathbf{v}^*$  conditioned on  $D_{\mathbf{x}}$  indeed having its low-degree moments approximately match those of  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ . We use the following lemma, which estimates the Chow parameters robustly under adversarial label noise.
Lemma 2.4. Let  $\eta, \tau \in (0,1)$  and  $G$  be a distribution over  $\mathbb{R}^d \times \{\pm 1\}$  such that  $\mathbf{E}_{\mathbf{x} \sim G_{\mathbf{x}}}[\mathbf{x}\mathbf{x}^{\top}] \preccurlyeq 2\mathbf{I}$ . Let  $\mathbf{v} \in \mathbb{R}^d$  be a unit vector such that  $\mathbf{Pr}_{(\mathbf{x},y) \sim G}[\mathrm{sign}(\mathbf{v} \cdot \mathbf{x}) \neq y] \leq \eta$ . Then there exists an algorithm that takes  $N = O(d / \eta^2) \log(d / \tau)$  labeled samples from  $G$ , runs in time poly  $(N)$ , and outputs a vector  $\mathbf{w}$  such that  $\| \mathbf{E}_{\mathbf{x} \sim G_{\mathbf{x}}}[\mathrm{sign}(\mathbf{v} \cdot \mathbf{x})\mathbf{x}] - \mathbf{w}\|_2 \leq O(\sqrt{\eta})$  with probability at least  $1 - \tau$ .
The proof follows from standard arguments in robust mean estimation and the details can be found in Appendix C. We remark that the number of labeled samples required by this procedure is (nearly) linear with respect to  $d$  for constant  $\eta$ , which is important for us to obtain (nearly) optimal label complexity overall.
We are now ready to describe our testable proper learner. In particular, it first certifies that the moments of the underlying distribution match with the standard Gaussian and then draws i.i.d. samples to estimate the Chow parameter. It then follows from Fact 2.2, Lemmas 2.3 and 2.4 that the resulting Chow vector, conditioned on that the moment test pass, is closed to the defining vector of the optimal halfspace. We given the pseudo-code of the algorithm and its analysis, which also serves as the proof of Proposition 2.1, in the end of Appendix A.
3 Efficient Testable Learning of Halfspaces
In this section, we give our tester-learner for homogeneous halfspaces under the Gaussian distribution, thereby proving Theorem 1.3. Throughout this section, we will fix an optimal halfspace  $h^*(\mathbf{x}) = \mathrm{sign}(\mathbf{v}^* \cdot \mathbf{x})$ , i.e., a halfspace with optimal 0-1 error.
The structure of this section is as follows: In Section 3.1, we present a tester which certifies that the probability of the disagreement region between two halfspaces whose defining vectors are close to each other is small under  $D_{\mathbf{x}}$ . In Section 3.2, we present and analyze our localization step and combine it with the tester from Section 3.1 to obtain our final algorithm.
3.1 From Parameter Distance to 0-1 Error
For two homogeneous halfspaces  $h_{\mathbf{u}}(\mathbf{x}) = \mathrm{sign}(\mathbf{u}\cdot \mathbf{x})$  and  $h_{\mathbf{v}}(\mathbf{x}) = \mathrm{sign}(\mathbf{v}\cdot \mathbf{x})$ , where  $\mathbf{u},\mathbf{v}$  are unit vectors, if  $D_{\mathbf{x}}$  is the standard Gaussian,  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , we can express the probability mass of their disagreement region as follows (see, e.g., Lemma 4.2 of [DKS18]):
$$
\Pr_ {\mathbf {x} \sim D _ {\mathbf {x}}} \left[ h _ {\mathbf {u}} (\mathbf {x}) \neq h _ {\mathbf {v}} (\mathbf {x}) \right] \leq O \left(\| \mathbf {u} - \mathbf {v} \| _ {2}\right). \tag {1}
$$
Hence, learning homogeneous halfspaces under Gaussian marginals can often be reduced to approximately learning the defining vector of some optimal halfspace  $h^*$ . This is no longer the case if  $D_{\mathbf{x}}$  is an arbitrary distribution, which may well happen in our regime. We show in this section that it is still possible to "certify" whether some relationship similar to the one in Equation (1) holds.
In particular, given a known vector  $\mathbf{v}$ , we want to make sure that for any other vector  $\mathbf{w}$  that is close to  $\mathbf{v}$ , the mass of the disagreement region between the halfspaces defined by  $\mathbf{v}$ ,  $\mathbf{w}$  respectively is small. To do so, we will decompose the space into many thin "slabs" that are stacked on top of each other in the direction of  $\mathbf{v}$ . Then, we will certify the mass of disagreement restricted to each of the slab is not too large. For slabs that are close to the halfspace  $\mathrm{sign}(\mathbf{v} \cdot \mathbf{x})$ , we can check these slabs must not themselves be too heavy. For slabs that are far away from the halfspace, we use the observation that the points in the disagreement region must then have large components in the subspace perpendicular to  $\mathbf{v}$ . Hence, as long as  $D$  has its second moment bounded, we can bound the mass of the disagreement region in these far-away slabs using standard concentration inequality. Formally, we have the following lemma, whose proof can be found in Appendix B.
Lemma 3.1 (Wedge Bound). Let  $D_{\mathbf{x}}$  be a distribution over  $\mathbb{R}^d$ . Given a unit vector  $\mathbf{v}$  and parameters  $\eta, \tau \in (0,1/2)$ , there exists an algorithm (Algorithm 1) that draws i.i.d. samples from  $D_{\mathbf{x}}$ , runs in
Input: Sample access to a distribution  $D_{\mathbf{x}}$  over  $\mathbb{R}^d$ ; tolerance parameter  $\eta > 0$ ; unit vector  $\mathbf{v} \in \mathbb{R}^d$ ; failure probability  $\tau \in (0,1)$ .
Output: Certifies that for all unit vectors  $\mathbf{w}$  such that  $\| \mathbf{w} - \mathbf{v}\| _2\leq \eta$  it holds that  $\mathbf{Pr}_{\mathbf{x}\sim D_{\mathbf{x}}}[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\neq \mathrm{sign}(\mathbf{w}\cdot \mathbf{x})]\leq C\eta$ , for some absolute constant  $C > 1$ , or reports that  $D_{\mathbf{x}}$  is not  $\mathcal{N}(\mathbf{0},\mathbf{I})$ .
1. Set  $B = \lceil \sqrt{\log(1 / \eta)} / \eta \rceil$
2. Let  $\widetilde{D}$  be the empirical distribution obtained by drawing  $\mathrm{poly}(d,1 / \eta)\log (1 / \tau)$  samples from  $D_{\mathbf{x}}$
3. For integers  $-B - 1 \leq i \leq B$ , define  $E_{i}$  to be the event that  $\{\mathbf{v} \cdot \mathbf{x} \in [i\eta, (i + 1)\eta]\}$  and  $E_{B + 1}$  to be the event that  $\{| \mathbf{v} \cdot \mathbf{x} | \geq \sqrt{\log(1 / \eta)}\}$ .
4. Verify that  $\sum_{i = -B - 1}^{B + 1}\left|\mathbf{Pr}_{\mathcal{N}(\mathbf{0},\mathbf{I})}[E_i] - \mathbf{Pr}_{\widetilde{D}}[E_i]\right| \leq \eta$
5. Let  $S_{i}$  the distribution of  $\widetilde{D}$  conditioned on  $E_{i}$  and  $S_{i}^{\perp}$  be  $S_{i}$  projected on the subspace orthogonal to  $\mathbf{v}$ .
6. For each  $i$ , verify that  $S_{i}^{\perp}$  has bounded covariance, i.e., check that  $\mathbf{E}_{\mathbf{x} \sim S_{i}^{\perp}}[\mathbf{x x}^{\top}] \preccurlyeq 2\mathbf{I}$ .
Algorithm 1: Wedge-Bound
time poly  $(d,1 / \eta)\log (1 / \tau)$ , and reports either one of the following: (i) For all unit vectors  $\mathbf{w}$  such that  $\| \mathbf{w} - \mathbf{v}\| _2\leq \eta$  it holds  $\mathbf{Pr}_{\mathbf{x}\sim D_{\mathbf{x}}}[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\neq \mathrm{sign}(\mathbf{w}\cdot \mathbf{x})]\leq C\eta$ , for some absolute constant  $C > 1$ . (ii)  $D_{\mathbf{x}}$  is not the standard Gaussian  $\mathcal{N}(\mathbf{0},\mathbf{I})$ . Moreover, with probability at least  $1 - \tau$ , the report is accurate.
3.2 Algorithm and Analysis: Proof of Theorem 1.3
We employ the idea of "soft" localization used in [DKS18]. In particular, given a vector  $\mathbf{v}$  and a parameter  $\sigma$ , we use rejection sampling to define a new distribution  $D_{\mathbf{v},\sigma}$  that "focuses" on the region near the halfspace  $\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})$ .
Fact 3.2 (Rejection Sampling, Lemma 4.7 of [DKS18]). Let  $D$  be a distribution on labeled examples  $(\mathbf{x},y)\in \mathbb{R}^d\times \{\pm 1\}$ . Let  $\mathbf{v}\in \mathbb{R}^{d}$  be a unit vector and  $\sigma \in (0,1)$ . We define the distribution  $D_{\mathbf{v},\sigma}$  as follows: draw a sample  $(\mathbf{x},y)$  from  $D$  and accept it with probability  $e^{-(\mathbf{v}\cdot \mathbf{x})^2\cdot (\sigma^{-2} - 1) / 2}$ . Then,  $D_{\mathbf{v},\sigma}$  is the distribution of  $(\mathbf{x},y)$  conditional on acceptance. If the  $\mathbf{x}$ -marginal of  $D$  is  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , then the  $\mathbf{x}$ -margins of  $D_{\mathbf{v},\sigma}$  is  $\mathcal{N}(\mathbf{0},\mathbf{\Sigma})$ , where  $\boldsymbol {\Sigma} = \mathbf{I} - (1 - \sigma^2)\mathbf{v}\mathbf{v}^T$ . Moreover, the acceptance probability of a point is  $\sigma$ .
We now briefly discuss the sample and label complexities of this rejection sampling procedure. Since the acceptance probability is  $\sigma$ , we need roughly  $1 / \sigma$  samples from  $D$  to simulate one sample from  $D_{\mathbf{v},\sigma}$ . On the other hand, to simulate one labeled sample from  $D$ , one only need 1 label query in addition to the unlabeled samples as one can query only the points that are accepted by the procedure. This makes sure that rejection sampling is efficient in terms of its label complexity.
The main idea of localization is the following. Let  $\mathbf{v}$  be a vector such that  $\| \mathbf{v} - \mathbf{v}^* \|_2 \leq \delta$ . Suppose that we use localization to the distribution  $D_{\mathbf{v},\delta}$ . If we can learn a halfspace with defining vector  $\mathbf{w}$  that achieves sufficiently small constant error with respect to the new distribution  $D_{\mathbf{v},\delta}$ , we can then combine our knowledge of  $\mathbf{w}$  and  $\mathbf{v}$  to produce a new halfspace with significantly improved error guarantees under  $D$ . The following lemma formalizes this geometric intuition.
Lemma 3.3. Let  $\mathbf{v}^*$ ,  $\mathbf{v}$  be two unit vectors in  $\mathbb{R}^d$  such that  $\| \mathbf{v} - \mathbf{v}^*\|_2 \leq \delta \leq 1/100$ . Let  $\boldsymbol{\Sigma} = \mathbf{I} - (1 - \delta^2)\mathbf{v}\mathbf{v}^T$  and  $\mathbf{w}$  be a unit vector such that  $\left\|\mathbf{w} - \frac{\boldsymbol{\Sigma}^{1/2}\mathbf{v}^*}{\|\boldsymbol{\Sigma}^{1/2}\mathbf{v}^*\|_2}\right\|_2 \leq \zeta \leq 1/100$ . Then it holds  $\left\|\frac{\boldsymbol{\Sigma}^{-1/2}\mathbf{w}}{\|\boldsymbol{\Sigma}^{-1/2}\mathbf{w}\|_2} - \mathbf{v}^*\right\|_2 \leq 5(\delta^2 + \delta \zeta)$ .
We defer the proof to Appendix C. Below, we provide a few useful remarks regarding the relevant parameters.
Remark 3.4. Observe that in Lemma 3.3 we require that the distance of  $\mathbf{v}$  and  $\mathbf{v}^*$  is smaller than  $1/100$ . While this constant is not the best possible, we remark that some non-trivial error is indeed necessary so that the localization step works. For example, assume that  $\mathbf{v}$  and  $\mathbf{v}^*$  are orthogonal, i.e.,  $\mathbf{v} \cdot \mathbf{v}^* = 0$ , and that  $\boldsymbol{\Sigma} = \mathbf{I} - (1 - \xi^2)\mathbf{v}\mathbf{v}^T$  for some  $\xi \in [0,1]$ . Observe that  $\boldsymbol{\Sigma}^{1/2}$  scales vectors by a factor of  $\xi$  in the direction of  $\mathbf{v}$  and leaves orthogonal directions unchanged. Similarly, its inverse  $\boldsymbol{\Sigma}^{-1/2}$  scales vectors by a factor of  $1/\xi$  in the direction of  $\mathbf{v}$  and leaves orthogonal directions unchanged. Without loss of generality, assume that  $\mathbf{v} = \mathbf{e}_1$ ,  $\mathbf{v}^* = \mathbf{e}_2$  where  $\mathbf{e}_1, \mathbf{e}_2$  are the two standard basis vectors. Then  $\boldsymbol{\Sigma}^{1/2}\mathbf{v}^* = \mathbf{v}^*$ . Moreover, assume that  $\mathbf{w} = a\mathbf{e}_1 + b\mathbf{e}_2$  (with  $a^2 + b^2 = 1$ ). We observe that  $\| \mathbf{w} - \boldsymbol{\Sigma}^{1/2}\mathbf{v}^* / \| \boldsymbol{\Sigma}^{1/2}\mathbf{v}^* \|_2\|^2 = \| \mathbf{w} - \mathbf{v}^* \|_2^2 = 2 - 2b$ . However, we have that  $\mathbf{s} = \boldsymbol{\Sigma}^{-1/2}\mathbf{w} = (a/\xi)\mathbf{e}_1 + b\mathbf{e}_2$ . Therefore,  $\| \mathbf{s} / \| \mathbf{s} \|_2 - \mathbf{v}^* \|_2^2 = 2 - 2b / \sqrt{(a/\xi)^2 + b^2}$ . Notice that for all  $\xi \in [0,1]$  it holds that  $\mathbf{s} / \| \mathbf{s} \|_2$  is further away from  $\mathbf{v}^*$  than  $\mathbf{w}$ , i.e., rescaling by  $\boldsymbol{\Sigma}^{-1/2}$  worsens the error.
We are now ready to present an iterative testing/learning procedure that will serve as the main component of our algorithm. At a high level, we first use Algorithm 2 from Proposition 2.1 to learn a vector  $\mathbf{v}$  that is close to  $\mathbf{v}^*$  in  $\ell_2$ -distance up to some sufficiently small constant. Then we localize to the learned halfspace and re-apply Algorithm 2 to iteratively improve  $\mathbf{v}$ . In particular, we will argue that, whenever the learned halfspace is still significantly suboptimal, the algorithm either detects that the underlying distribution is not Gaussian or keeps making improvements such that  $\mathbf{v}$  gets closer to  $\mathbf{v}^*$  conditioned on  $\mathbf{v}$  still being sub-optimal. The pseudocode and analysis of the algorithm can be found in Appendix C.
Lemma 3.5. Suppose  $\left\| \mathbf{v}^{*} - \mathbf{v}^{(t)}\right\|_{2}\leq \delta \leq 1 / 100$ . There is an algorithm (Algorithm 3) that with probability at least  $1 - \tau$ , either (i) correctly reports that the  $\mathbf{x}$ -marginal of  $D$  is not  $\mathcal{N}(\mathbf{0},\mathbf{I})$  or (ii) computes a unit vector  $\mathbf{v}^{(t + 1)}$  so that either  $\left\| \mathbf{v}^{*} - \mathbf{v}^{(t + 1)}\right\|_{2}\leq \delta /2$  or  $\left\| \mathbf{v}^{*} - \mathbf{v}^{(t)}\right\|_{2}\leq C\mathrm{opt}$ , where  $C > 0$  is an absolute constant. Furthermore, the algorithm draws  $N = \mathrm{poly}(d,1 / \delta)\log (1 / \tau)$  many unlabeled samples and uses  $O(d\log (d / \tau))$  label queries.
After running Algorithm 3 for at most a logarithmic number of iterations, we know  $\mathbf{v}$  must be close to  $\mathbf{v}^*$ . Then, we can use Algorithm 1 from Lemma 3.1 to certify that the disagreement between the learned halfspace and  $h^{*}(\mathbf{x})$  is small. One technical difficulty is that we do not know precisely when to terminate the update and each update does not necessarily monotonically bring  $\mathbf{v}$  closer to  $\mathbf{v}^*$ . As a result, the process only yields a list of vectors  $\mathbf{v}^{(1)},\dots \mathbf{v}^{(t)}$  with the guarantee that at least one of them is sufficiently closed to  $\mathbf{v}^*$ . A natural idea is to estimate the 0-1 errors of the halfspaces defined by the vectors in the list and simply pick the one with the smallest empirical error. Naively, we need to estimate the errors up to an additive  $\epsilon$  and this may take up to  $\Omega (1 / \epsilon^2)$  labeled samples, which is exponentially worse than our goal in terms of the dependency on  $\epsilon$ . Nonetheless, we remark that comparing the errors of two halfspaces which differ by some multiplicative factors can be done much more efficiently in terms of its label complexity. In particular, it suffices for us to compare the errors of the two halfspaces restricted to the area in which the predictions made by them differ. Under this conditioning, the difference between the errors then gets magnified to some constant, making comparisons much easier. Hence, we can run a tournament among the the halfspaces, which reduces to perform pairwise comparisons among the lists, to obtain our final winner hypothesis. The result is summarized in the following lemma.
Lemma 3.6 (Tournament). Let  $\epsilon, \tau \in (0,1)$  and  $D$  a distribution over  $\mathbb{R}^d \times \{\pm 1\}$ . Given a list of halfspaces  $\{h^{(i)}\}_{i=1}^k$ , there is an algorithm that draws  $N = \Theta(k^2 \log(k / \tau) / \epsilon)$  i.i.d. unlabeled samples from  $D_{\mathbf{x}}$ , uses  $\Theta(k^2 \log(k / \tau))$  label queries, runs in time poly $(d, N)$  and outputs a halfspace  $\tilde{h}$  satisfying that  $\mathbf{Pr}_{(\mathbf{x}, y) \sim D}[\tilde{h}(x) \neq y] \leq 10 \min_i \mathbf{Pr}_{(\mathbf{x}, y) \sim D}[h^{(i)}(x) \neq y] + \epsilon$ .
We provide the pseudocode of the algorithm and its analysis in Appendix C. The proof of Theorem 1.3 then follows.
Our final algorithm (Algorithm 5) simply iteratively applies the localized update routine (Algorithm 3) and then runs a tournament (Algorithm 4) among the  $O(\log(1/\epsilon))$  candidate halfspaces obtained from the iterations. The pseudocode and its analysis, which serves as proof of our main theorem, can be found at the end of Appendix C.
4 Acknowledgements
ID was supported by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), and a DARPA Learning with Less Labels (LwLL) grant. DK was supported by NSF Medium Award CCF-2107547 and NSF Award CCF-1553288 (CAREER). NZ was supported in part by NSF award 2023239, NSF Medium Award CCF-2107079, and a DARPA Learning with Less Labels (LwLL) grant.