SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. If  $\mathbf{x} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ , then  $\mathbf{v} \cdot \mathbf{x} \sim \mathcal{N}(0,1)$ . If we concatenate the values of  $\operatorname{Pr}_{\mathcal{N}(\mathbf{0},\mathbf{I})}[E_i]$  into a vector, it can be viewed as the discretization of  $\mathcal{N}(0,1)$  into  $2B + 3$  many buckets. On the other hand,  $\operatorname{Pr}_{\tilde{D}}[E_i]$  is an empirical version of this discrete distribution composed of  $N$  i.i.d. samples where  $N = \mathrm{poly}(d,1/\eta)\log(1/\tau)$ . Since we can learn any discrete distribution with support  $n$  up to error  $\eta$  in total variation distance with  $\Theta(n/\eta^2)\log(1/\tau)$  samples with probability at least  $1 - \tau$ , it follows that Line 4 will pass with high probability as long as we take more than  $\Theta(B/\eta^2)\log(1/\tau) \leq \mathrm{poly}(d,1/\eta)\log(1/\tau)$  many samples.
For Line 6, we remark that  $S_{i}^{\perp}$  is the empirical version of a  $(d - 1)$ -dimensional standard Gaussian. Since the empirical mean and the empirical covariance concentrates around the true mean and
covariance with probability at least  $1 - \tau$  if one takes more than  $\Theta (d^2 /\eta^2)\log (1 / \tau)$  many samples, it follows that Line 4 will pass with high probability as long as we take more than  $\Theta (d^{2} / \eta^{2})\log (1 / \tau)\leq$  poly(d, 1/η) log(1/τ) many samples.
Suppose that both lines pass. We claim that this implies the following: for all unit vectors  $\mathbf{w}$  such that  $\| \mathbf{w} - \mathbf{v}\|_2 \leq \eta$  it holds
$$
\Pr_ {\mathbf {x} \sim \widetilde {D}} \left[ \operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x}) \right] \leq C \eta . \tag {10}
$$
for some absolute constant  $C > 1$ . Given this, we can deduce that the same equation must also hold for  $D_{\mathbf{x}}$  with high probability — albeit with a larger constant  $C'$ . To see this, we remark that the left hand side of the equation can be treated as the error of the halfspace  $\mathrm{sign}(\mathbf{w} \cdot \mathbf{x})$  if the true labels are generated by  $\mathrm{sign}(\mathbf{v} \cdot \mathbf{x})$ . Since the VC-dimension of the class of homogeneous halfspaces is  $d$ , we have that the error for all  $\mathbf{w}$  under  $D_{\mathbf{x}}$  is well-approximated by that under  $\widetilde{D}$  up to an additive  $\eta$  with probability at least  $1 - \tau$  given  $N = \mathrm{poly}(d, 1 / \eta) \log(1 / \tau)$  many samples. Hence, conditioned on Equation (10), it holds with probability at least  $1 - \tau$  that
$$
\operatorname * {P r} _ {\mathbf {x} \sim D _ {\mathbf {x}}} [ \operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x}) ] \leq (C + 1) \eta ,
$$
for all  $\mathbf{w}$
We now proceed to show Equation (10) holds if the Algorithm 1 did not terminate on Lines 4 and 6. Conditioned on Line 6, for any unit vector  $\mathbf{u} \in \mathbb{R}^d$  that is orthogonal to  $\mathbf{v}$ , we have  $|\mathbf{E}[\mathbf{u} \cdot \mathbf{x}]| \leq O(1)$  and  $\operatorname{Var}[\mathbf{u} \cdot \mathbf{x}] \leq 2$ . Using Chebyshev's inequality, for any  $\alpha > 0$ , it holds
$$
\Pr_ {\mathbf {x} \sim S _ {i} ^ {\perp}} \left[ | \mathbf {u} \cdot \mathbf {x} | \geq \alpha \right] \leq O \left(\frac {1 + \eta^ {2}}{\alpha^ {2}}\right). \tag {11}
$$
We can now bound  $\mathbf{Pr}_{\mathbf{x}\sim \widetilde{D}}[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\neq \mathrm{sign}(\mathbf{w}\cdot \mathbf{x})]$  for an arbitrary unit vector  $\mathbf{w}$  satisfying  $\| \mathbf{w} - \mathbf{v}\| _2\leq \eta$ . We proceed to rewrite  $\mathbf{w}$  as  $(1 - \gamma^{2})^{1 / 2}\mathbf{v} + \gamma \mathbf{u}$  for some unit vector  $\mathbf{u}\in \mathbb{R}^d$  that is orthogonal to  $\mathbf{v}$  and  $\gamma \in (0,\eta)$ . Denote  $\gamma^\prime = \gamma /(1 - \gamma^2)^{1 / 2}$ , then the event that  $\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\neq \mathrm{sign}(\mathbf{w}\cdot \mathbf{x})$  implies that  $\gamma^\prime |\mathbf{u}\cdot \mathbf{x}|\geq |\mathbf{v}\cdot \mathbf{x}|$ . Therefore, we have that
$$
\underset {\mathbf {x} \sim S _ {i}} {\operatorname {P r}} [ \operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x}) ] \leq \underset {\mathbf {x} \sim S _ {i}} {\operatorname {P r}} [ \gamma^ {\prime} | \mathbf {u} \cdot \mathbf {x} | \geq | \mathbf {v} \cdot \mathbf {x} | ] \leq \underset {\mathbf {x} \sim S _ {i} ^ {\perp}} {\operatorname {P r}} [ \gamma^ {\prime} | \mathbf {u} \cdot \mathbf {x} | \geq i \eta ] \leq O \left(\frac {1 + \eta^ {2}}{i ^ {2}}\right), \tag {12}
$$
where in the second inequality we use the definition of  $S_{i}$ , and in the third inequality we use that  $\gamma \leq \eta$  and Equation (11). We now bound from above the total disagreement probability between  $\mathbf{w}$  and  $\mathbf{v}$  under  $\widetilde{D}$ . We have that
$$
\begin{array}{l} \mathbf {P r} _ {\mathbf {x} \sim \widetilde {D}} \left[ \operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x}) \right] \leq \mathbf {P r} _ {\mathbf {x} \sim \widetilde {D}} \left[ | \mathbf {v} \cdot \mathbf {x} | \geq \sqrt {\log (1 / \eta)} \right] + \mathbf {P r} _ {\mathbf {x} \sim \widetilde {D}} \left[ | \mathbf {v} \cdot \mathbf {x} | \leq \eta \right] \\ \sqrt {\log (1 / \eta)} / \eta \\ + \sum_ {| i | > 1} \mathbf {P r} _ {\mathbf {x} \sim S _ {i}} \left[ \operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x}) \right] \mathbf {P r} _ {\mathbf {x} \sim \tilde {D}} [ (i - 1) \eta \leq \mathbf {x} \cdot \mathbf {v} \leq i \eta ] \\ \leq 6 \eta + O (\eta) \sum_ {| i | > 1} ^ {\sqrt {\log (1 / \eta)} / \eta} \frac {1 + \eta^ {2}}{i ^ {2}} \leq O (\eta), \\ \end{array}
$$
where we used that  $\mathbf{Pr}_{\mathbf{x} \sim \widetilde{D}}[|\mathbf{x} \cdot \mathbf{v}| > \sqrt{\log(1 / \eta)}] \leq 3\eta$  and  $\mathbf{Pr}_{\mathbf{x} \sim \widetilde{D}}[|\mathbf{x} \cdot \mathbf{v}| < \eta] \leq 3\eta$ , since in Line 4 we verified that the probabilities  $\mathbf{Pr}_{\mathbf{x} \sim \widetilde{D}}[\mathbf{v} \cdot \mathbf{x} \in [(i - 1)\eta, i\eta]$  are close to the probabilities under  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  and hence bounded by  $O(\eta)$ , Equation (12), and the fact that the series  $\sum_{i} \frac{1}{i^2}$  is convergent and less than  $\pi^2 / 6$ .
C Omitted Proofs of localization and main theorem
Proof of Lemma 3.3. Since  $\| \mathbf{v} - \mathbf{v}^* \|_2 \leq \delta$ , we can write
$$
\mathbf {v} ^ {*} = \frac {1}{\sqrt {1 + \kappa^ {2}}} (\mathbf {v} + \kappa \mathbf {u}) \tag {13}
$$
for some  $\kappa \in [0,\delta ]$  and some unit vector  $\mathbf{u}$  perpendicular to  $\mathbf{v}$ . By definition,  $\Sigma^{1 / 2}$  shrinks in the direction of  $\mathbf{v}$  by a factor of  $\delta$  and leaves other orthogonal directions unchanged. Hence, it holds
$$
\boldsymbol {\Sigma} ^ {1 / 2} \mathbf {v} ^ {*} = \frac {1}{\sqrt {1 + \kappa^ {2}}} \left(\delta \mathbf {v} + \kappa \mathbf {u}\right).
$$
Then, using the triangle inequality, we obtain
$$
\gamma := \left\| \boldsymbol {\Sigma} ^ {1 / 2} \mathbf {v} ^ {*} \right\| _ {2} \leq \frac {1}{\sqrt {1 + \kappa^ {2}}} \left(\delta \| \mathbf {v} \| _ {2} + \kappa \| \mathbf {u} \| _ {2}\right) \leq \frac {2 \delta}{\sqrt {1 + \kappa^ {2}}} \leq 2 \delta ,
$$
since  $\kappa$  is upper bounded by  $\delta$ . Since  $\left\| \mathbf{w} - \frac{\mathbf{\Sigma}^{1/2}\mathbf{v}^*}{\|\mathbf{\Sigma}^{1/2}\mathbf{v}^*\|_2} \right\|_2 \leq \zeta$ , we can write
$$
\mathbf {w} = \frac {\boldsymbol {\Sigma} ^ {1 / 2} \mathbf {v} ^ {*}}{\left\| \boldsymbol {\Sigma} ^ {1 / 2} \mathbf {v} ^ {*} \right\| _ {2}} + a \mathbf {v} + b \mathbf {u} ^ {\prime},
$$
for some  $|a|, b \in [0, \zeta]$  and  $\mathbf{u}'$  perpendicular to  $\mathbf{v}$ . We can multiply both sides by  $\gamma$  and get
$$
\gamma \mathbf {w} = \boldsymbol {\Sigma} ^ {1 / 2} \mathbf {v} ^ {*} + a \gamma \mathbf {v} + b \gamma \mathbf {u} ^ {\prime},
$$
$$
\gamma \boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w} = \mathbf {v} ^ {*} + a \gamma / \delta \mathbf {v} + b \gamma \mathbf {u} ^ {\prime}.
$$
Using Equation (13), we then have
$$
\gamma \boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w} = \left(\frac {1}{\sqrt {1 + \kappa^ {2}}} + a \frac {\gamma}{\delta}\right) \mathbf {v} + \frac {\kappa}{\sqrt {1 + \kappa^ {2}}} \mathbf {u} + b \gamma \mathbf {u} ^ {\prime}. \tag {14}
$$
Let  $\lambda := \frac{1}{\sqrt{1 + \kappa^2}} + a\gamma/\delta$  be the coefficient before  $\mathbf{v}$ . We next identify the range of  $\lambda$ .
Claim C.1. It holds that  $\lambda \in [1 - \delta - 2\zeta, 1 + 2\zeta]$ .
Proof. Recall that  $\kappa \in [0,\delta], |a| \in [0,\zeta]$  and  $\gamma \in [0,2\delta]$ . If we view  $\lambda$  as a function of  $\kappa, a, \gamma$ , it is minimized when  $\kappa = \delta$ ,  $a = -\zeta$ ,  $\gamma = 2\delta$ , which then gives
$$
\lambda \geq \frac {1}{\sqrt {1 + \delta^ {2}}} - 2 \zeta \geq 1 - \delta - 2 \zeta ,
$$
where in the second inequality we use the fact that  $\frac{1}{\sqrt{1 + x^2}} \geq 1 - x$  for  $x \geq 0$ . On the other hand,  $\lambda$  is maximized when  $\kappa = 0, a = \zeta, \gamma = 2\delta$ , which gives  $\lambda \leq 1 + 2\zeta$ . Hence, we can conclude that  $\lambda \in [1 - \delta - 2\zeta, 1 + 2\zeta]$ .
We multiply both sides of Equation (14) by  $\frac{1}{\lambda\sqrt{1 + \kappa^2}}$ , which gives
$$
\begin{array}{l} \frac {\gamma}{\lambda \sqrt {1 + \kappa^ {2}}} \pmb {\Sigma} ^ {- 1 / 2} \mathbf {w} = \frac {1}{\sqrt {1 + \kappa^ {2}}} \mathbf {v} + \frac {\kappa}{\lambda (1 + \kappa^ {2})} \mathbf {u} + b \frac {\kappa}{\lambda \sqrt {1 + \kappa^ {2}}} \mathbf {u} ^ {\prime} \\ = \mathbf {v} ^ {*} + \left(\frac {\kappa}{\lambda (1 + \kappa^ {2})} - \frac {\kappa}{\sqrt {1 + \kappa^ {2}}}\right) \mathbf {u} + b \frac {\gamma}{\lambda \sqrt {1 + \kappa^ {2}}} \mathbf {u} ^ {\prime}, \\ \end{array}
$$
where in the second equality we use Equation (13). We then bound from above and below the norm, and we get that
$$
\left| \left\| \frac {\gamma}{\lambda \sqrt {1 + \kappa^ {2}}} \mathbf {\Sigma} ^ {- 1 / 2} \mathbf {w} \right\| _ {2} - 1 \right| \leq \frac {\kappa}{1 + \kappa^ {2}} \left| \frac {1}{\lambda} - \sqrt {1 + \kappa^ {2}} \right| + \frac {b \gamma}{\lambda \sqrt {1 + \kappa^ {2}}},
$$
where we used triangle inequality. Note that
$$
\left| \left(1 / \lambda\right) - \sqrt {1 + \kappa^ {2}} \right| \leq \left| 1 / \lambda - 1 \right| + \left| 1 - \sqrt {1 + \kappa^ {2}} \right| \leq (\delta + 2 \zeta) / (1 - \delta - 2 \zeta) + \kappa
$$
and that  $\kappa \leq \delta$ . Therefore, we obtain that
$$
\left| \left\| \frac {\gamma}{\lambda \sqrt {1 + \kappa^ {2}}} \boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w} \right\| _ {2} - 1 \right| \leq 4 (\delta^ {2} + \delta \zeta).
$$
Input: Sample access to a distribution  $D$  over labeled examples; unit vector  $\mathbf{v} \in \mathbb{R}^d$ ; parameters  $\tau, \eta, \delta > 0$ .
Output: Either reports that  $D_{\mathbf{x}}$  is not  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  or computes a unit vector  $\mathbf{v}'$  such that: either  $\| \mathbf{v}^* - \mathbf{v}' \|_2 \leq \delta / 2$  or  $\| \mathbf{v}^* - \mathbf{v}' \|_2 \leq O(\mathrm{opt})$
1. Let  $D_{\mathbf{v},\delta}$  be the distribution obtained by running Rejection Sampling with parameters  $\mathbf{v}$  and  $\delta$
2. Check the acceptance probability is within  $[\delta / 2, 3\delta / 2]$ . Otherwise, report  $\mathbf{x}$ -margins of  $D$  is not  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ .
3. Let  $G$  be the distribution obtained by applying the transformation  $\pmb{\Sigma}^{-1/2}$  on the  $\mathbf{x}$  marginals of  $D_{\mathbf{v},\delta}$  where  $\pmb{\Sigma} = \mathbf{I} - (1 - \delta^2)\mathbf{v}\mathbf{v}^\top$ .
4. Run Algorithm 2 on  $G$  with accuracy  $\eta = 1 / (20000C_{\mathrm{A}}^2)$  to obtain a unit vector  $\mathbf{w}$ .
5. If the algorithm reports the marginal of  $G$  is not  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , terminate and report it.
6. Set  $\mathbf{v}' = \Sigma^{1/2}\mathbf{w} / \left\| \Sigma^{1/2}\mathbf{w} \right\|_2$  and return  $\mathbf{v}'$ .
Algorithm 3: Testable Localized-Update
Let  $A = \left\| \frac{\gamma}{\lambda\sqrt{1 + \kappa^2}}\pmb{\Sigma}^{-1 / 2}\mathbf{w}\right\|_2$ . We have that
$$
\left\| \frac {\boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w}}{\left\| \boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w} \right\| _ {2}} - \mathbf {v} ^ {*} \right\| _ {2} \leq \| \mathbf {v} ^ {*} \| _ {2} | 1 - 1 / A | + | 1 / A - 1 | \leq 5 (\delta^ {2} + \delta \zeta).
$$
Proof of Lemma 3.5. For simplicity, we denote  $\mathbf{v}^{(t)}$  as  $\mathbf{v}$ . We apply the Rejection Sampling procedure from Fact 3.2 in the direction of  $\mathbf{v}$  with  $\sigma = \delta$ . If the  $\mathbf{x}$ -marginal of  $D$  is  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , the acceptance probability of  $D_{\mathbf{v},\delta}$  is exactly  $\delta$ . We then estimate the acceptance probability with accuracy  $\epsilon$ ; if it is not lying inside the interval  $[\delta /2,3\delta /2]$  (see Line 2 of Algorithm 3), we report that the  $\mathbf{x}$ -marginal of  $D$  is not standard normal and terminate.
Conditioned on the event that the algorithm did not terminate, we can get an (unlabeled) sample from the  $\mathbf{x}$ -marginal of  $D_{\mathbf{v},\delta}$ , using  $O(1 / \delta)$  unlabeled samples from  $D_{\mathbf{x}}$ . To get a labeled sample from  $D_{\mathbf{v},\delta}$ , we only need 1 additional label query. Note that, under the distribution  $D_{\mathbf{v},\delta}$ , the error of the  $\mathbf{v}^*$  is
$$
\begin{array}{l} \Pr_ {(\mathbf {x}, y) \sim D _ {\mathbf {v}, \delta}} [ \operatorname {s i g n} (\mathbf {v} ^ {*} \cdot \mathbf {x}) \neq y ] = \Pr_ {(\mathbf {x}, y) \sim D} [ \operatorname {s i g n} (\mathbf {v} ^ {*} \cdot \mathbf {x}) \neq y | (\mathbf {x}, y) \text {i s a c c e p t e d} ] \\ \leq \Pr_ {(\mathbf {x}, y) \sim D} [ \operatorname {s i g n} (\mathbf {v} ^ {*} \cdot \mathbf {x}) \neq y ] / \Pr_ {(\mathbf {x}, y) \sim D} [ (\mathbf {x}, y) \text {i s a c c e p t e d} ] \\ \leq 2 \operatorname {o p t} / \delta , \tag {15} \\ \end{array}
$$
where we used that the probability of the acceptance is at least  $\delta /2$ . Denote by  $G$  the distribution of  $(\pmb{\Sigma}^{-1 / 2}\mathbf{x},y)$ , where  $(\mathbf{x},y)\sim D_{\mathbf{v},\delta}$  and  $\pmb {\Sigma} = \mathbf{I} - (1 - \delta^2)\mathbf{v}\mathbf{v}^\top$ . We note that if the  $\mathbf{x}$ -marginal of  $D$  were the standard normal, then  $\mathbf{x}$ -marginal of  $G$  is the standard normal. Hence, we can apply the algorithm Algorithm 2 from Proposition 2.1. Under the transformed distribution, we have that the new optimal vector  $(\mathbf{v}^{*})^{\prime}:= \pmb{\Sigma}^{1 / 2}\mathbf{v}^{*} / \| \pmb{\Sigma}^{1 / 2}\mathbf{v}^{*}\|_{2}$ .
From Proposition 2.1, running Algorithm 2 on the normalized distribution  $G$  with error parameter  $\eta \leq 1 / (2000C_{\mathrm{A}}^2)$  and failure probability  $\tau$  consumes  $\mathrm{poly}(d)\log (1 / \tau)$  unlabeled samples and  $O(d\log (d / \tau))$  labeled samples from  $G$ . Since the rejection sampling procedure for  $G$  accepts points with probability at least  $\Omega (\epsilon)$  Therefore, it consumes  $\mathrm{poly}(d,1 / \epsilon)\log (1 / \tau)$  samples from  $D$  and uses  $O(d\log (d / \tau))$  additional label queries. Conditioned on the event that it succeeds (which happens with probability at least  $1 - \tau$ ), it either (i) reports that the x-marginal of  $G$  is not standard Gaussian (ii) returns a unit vector  $\mathbf{w}$  such that
$$
\left\| \mathbf {w} - \left(\mathbf {v} ^ {*}\right) ^ {\prime} \right\| _ {2} \leq C _ {\mathrm {A}} \sqrt {\underset {(\mathbf {x} , y) \sim G} {\operatorname * {P r}}} [ \operatorname {s i g n} \left(\left(\mathbf {v} ^ {*}\right) ^ {\prime} \cdot \mathbf {x}\right) \neq y ] + \eta . \tag {16}
$$
In case (i), we can directly report that  $\mathbf{x}$ -marginal of  $D$  is not  $\mathcal{N}(\mathbf{0},\mathbf{I})$  since the  $\mathbf{x}$ -marginal of  $G$  ought to be  $\mathcal{N}(\mathbf{0},\mathbf{I})$ .
In case (ii), we claim that at least one of the following hold (a)  $\mathbf{v}$  before the localized update is already good enough, i.e.  $\| \mathbf{v}^{*} - \mathbf{v}\|_{2}\leq 40000C_{\mathrm{A}}^{2}\mathrm{opt}$ ; (b) it holds  $\| (\mathbf{v}^{*})^{\prime} - \mathbf{w}\|_{2}\leq 1 / 100$ . Suppose that (a) does not hold; we will show that it then must hold  $\| (\mathbf{v}^{*})^{\prime} - \mathbf{w}\|_{2}\leq 1 / 100$ . Since  $\| \mathbf{v}^{*} - \mathbf{v}\|_{2}\leq \delta$  and  $\| \mathbf{v}^{*} - \mathbf{v}\|_{2} > 40000C_{\mathrm{A}}^{2}$  opt, we have that  $\mathrm{opt}\leq \delta /(40000C_{\mathrm{A}}^{2})$ . Furthermore, from Equation (15) we have that  $\mathbf{Pr}_{(\mathbf{x},y)\sim D_{\mathbf{v},\delta}}[\mathrm{sign}(\mathbf{v}^{*}\cdot \mathbf{x})\neq y]\leq 2\mathrm{opt} / \delta$ , it then follows that  $\mathbf{Pr}_{(\mathbf{x},y)\sim G}[\mathrm{sign}((\mathbf{v}^{*})^{\prime}\cdot \mathbf{x})\neq y]\leq 2\mathrm{opt} / \delta \leq 1 / (20000C_{\mathrm{A}}^{2})$ . Substituting this into Equation (16) then gives
$$
\left\| \mathbf {w} - \left(\mathbf {v} ^ {*}\right) ^ {\prime} \right\| _ {2} \leq 1 / 1 0 0.
$$
Using our assumption that  $\| \mathbf{v} - \mathbf{v}^*\| _2\leq \delta < 1 / 100$  , we can apply Lemma 3.3, which gives that
$$
\left\| \frac {\boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w}}{\left\| \boldsymbol {\Sigma} ^ {- 1 / 2} \mathbf {w} \right\| _ {2}} - \mathbf {v} ^ {*} \right\| _ {2} \leq \delta / 2.
$$
Hence, we set  $\mathbf{v}^{(t + 1)} = \left\| \frac{\pmb{\Sigma}^{-1 / 2}\mathbf{w}}{\|\pmb{\Sigma}^{-1 / 2}\mathbf{w}\|_2}\right\| _2$  and this completes the proof.
Input: Sample access to  $D_{\mathbf{x}}$  over unlabeled examples and query access to the labels of the samples drawn; a list of  $k$  halfspaces  $\{h^{(i)}\}_{i=1}^k$ ; parameters  $\tau, \epsilon > 0$ .
Output: finds a halfspace  $\tilde{h}$  whose error is nearly optimal among the list.
1. Let  $C$  be a sufficiently large constant.
2. For  $i,j = 1\cdots k$
(a) Take  $N_{1}\coloneqq C\log (k / \tau) / \epsilon$  samples  $D_{\mathbf{x}}$
(b) Skip the iteration if there are fewer than  $N_{2} \coloneqq 0.1C\log (k / \tau)$  samples such that  $h^{(i)}(\mathbf{x}) \neq h^{(j)}(\mathbf{x})$ . Denote the samples satisfying the condition as  $S_{1}$ .
(c) Query the labels of a random subset of samples from  $S_{1}$  of size  $N_{2}$ . Denote the set of labeled samples as  $S_{2}$
(d) Compute the empirical errors of  $h^{(i)}, h^{(j)}$  using  $S_{2}$ .
(e) If the empirical errors differ by more than  $1/4$ , mark the one with larger error as sub-optimal.
3. Return any halfspace among the list that has not been marked as sub-optimal or declare failure if there is not any.
Algorithm 4: Tournament
Proof of Lemma 3.6. Fix a halfspace  $h^{(i)}$ . Suppose there is another halfspace  $h^{(j)}$  such that
$$
\mathop{\Pr}\limits_{(\mathbf{x},y)\sim D}[h^{(i)}(x)\neq y] > 10\mathop{\Pr}\limits_{(\mathbf{x},y)\sim D}[h^{(j)}(x)\neq y] + \epsilon .
$$
We argue the halfspace will be marked as sub-optimal with probability at least  $\tau / \log(1/\epsilon)$ . For convenience, denote  $a := \mathbf{Pr}_{(\mathbf{x},y) \sim D}[h^{(i)}(x) \neq y \land h^{(i)}(x) \neq h^{(j)}(x)]$ ,  $b := \mathbf{Pr}_{(\mathbf{x},y) \sim D}[h^{(j)}(x) \neq y \land h^{(i)}(x) \neq h^{(j)}(x)]$  and  $m := \mathbf{Pr}_{\mathbf{x} \sim D_{\mathbf{x}}}[h^{(i)}(x) \neq h^{(j)}]$ . Notice that  $a, b$  are the errors of the two halfspaces from the area in which their predictions differ and  $m$  is the mass of the area. Then, it holds
$$
a + b = \underset {(\mathbf {x}, y) \sim D} {\mathbf {P r}} [ h ^ {(i)} (\mathbf {x}) \neq h ^ {(j)} (\mathbf {x}) \wedge y = h ^ {(i)} (\mathbf {x}) ] + \underset {(\mathbf {x}, y) \sim D} {\mathbf {P r}} [ h ^ {(i)} (\mathbf {x}) \neq h ^ {(j)} (\mathbf {x}) \wedge y \neq h ^ {(i)} (\mathbf {x}) ] = m,
$$
$$
a - b = \mathbf {P r} _ {(\mathbf {x}, y) \sim D} \left[ h ^ {(i)} (\mathbf {x}) \neq y \right] - \mathbf {P r} _ {(\mathbf {x}, y) \sim D} \left[ h ^ {(j)} (\mathbf {x}) \neq y \right] \geq 9 \mathbf {P r} _ {(\mathbf {x}, y) \sim D} \left[ h ^ {(j)} (x) \neq y \right] + \epsilon > 9 b.
$$
This further implies that  $a / m - b / m > 4 / 5$ . Notice that  $a / m, b / m$  are precisely the errors of  $h^{(i)}$ ,  $h^{(j)}$  under the conditional distribution of  $D$  restricted to the area in which  $h^{(i)}(\mathbf{x}) \neq h^{(j)}(\mathbf{x})$ . Hence,
with  $\Theta (\log (k / \tau))$  many samples, the corresponding empirical errors will differ by at least  $1 / 4$  with probability at least  $\tau /k$ .
On the other hand, suppose
$$
\mathop{\bf P r}\limits_{(\mathbf{x},y)\sim D}[h^{(i)}(x)\neq y]\leq \mathop{\bf P r}\limits_{(\mathbf{x},y)\sim D}[h^{(j)}(x)\neq y].
$$
This implies that  $a / m < b / m$ . As a result, the algorithm either skips the iteration if there are fewer than  $\Theta(\log(k / \tau))$  many labeled samples from the conditional distribution or the empirical error of  $h^{(i)}$  will not be less than that of  $h^{(j)}$  by more than 1/4 with probability at least  $\tau / k^2$  if there are enough samples.
The correctness of the lemma then follows from union bound. Finally, we remark  $m$ , the mass of the area we condition to, should be at least  $\epsilon$  if the errors of the two halfspaces differ by at least  $\epsilon$ . In that case, it takes on average  $\Theta(1/\epsilon)$  unlabeled samples and 1 additional label query to simulate one labeled sample from the conditional distribution. Therefore, it suffices if the algorithm takes  $O(k^2 \log (k / \tau) / \epsilon)$  many unlabeled samples from  $D_{\mathbf{x}}$  and use  $\Theta(k^2 \log (k / \tau))$  many additional label queries. This concludes the proof of Lemma 3.6.
Input: Sample access to a distribution  $D_{\mathbf{x}}$  over unlabeled examples and query access to the labels of the samples drawn;  $\epsilon >0$  ; unit vector  $\mathbf{v}\in \mathbb{R}^d$
Output: Either reports that  $D_{\mathbf{x}}$  is not  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  or computes hypothesis  $h$  such that  $\operatorname{Pr}_{(\mathbf{x}, y) \sim D}[h(\mathbf{x}) \neq y] = O(\mathrm{opt})$ .
1. Set  $\tau = (\epsilon /(C\log (1 / \epsilon)))$  for a sufficiently large constant  $C > 0$ . Set  $\eta = 1 / (20000C_{\mathrm{A}})$  where  $C_\mathrm{A}$  is the constant from Proposition 2.1.
2. Run Algorithm 2 on  $D$  with accuracy  $\eta$  to obtain unit vector  $\mathbf{v}^{(0)}$ .
3. If Algorithm 2 reports that  $D_{\mathbf{x}}$  is not  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ , then report it and terminate.
4. For  $t = 0 \dots \log(1/\epsilon)$ : set  $\delta = (1/100)2^{-t}$  and run Algorithm 3 with parameters  $\mathbf{v}^{(t)}, \delta, \eta$  to obtain  $\mathbf{v}^{(t+1)}$ .
5. For  $i = 0 \dots \log(1/\epsilon)$ :
(a) Run Algorithm 1 with  $\mathbf{v}^{(i)}$  and  $\eta = j\epsilon$  for  $j \in [1/\epsilon]$  on the  $\mathbf{x}$ -marginals of  $D$ .
(b) Set  $h^{(i)}(\mathbf{x}) = \mathrm{sign}(\mathbf{v}^{(i)}\cdot \mathbf{x})$ .
6. Return the hypothesis obtained from running Algorithm 4 on the list of hypothesis  $\{h^{(i)}\}_{i=1}^{\log(1/\epsilon)}$  on  $D$  with accuracy  $\epsilon$  and failure probability  $\tau/10$ .
Algorithm 5: Testable Localization