SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[ABL17] P. Awasthi, M. F. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. J. ACM, 63(6):50:1-50:27, 2017.
[BBL06] M. Balcan, A. Beygelzimer, and J. Langford. Agnostic active learning. In Proceedings of the 23rd international conference on Machine learning, pages 65-72, 2006.
[BBZ07] M. Balcan, A. Broder, and T. Zhang. Margin based active learning. In Learning Theory: 20th Annual Conference on Learning Theory, COLT 2007, San Diego, CA, USA; June 13-15, 2007. Proceedings 20, pages 35-50. Springer, 2007.
[BL13] M. Balcan and P. Long. Active and passive learning of linear separators under log-concave distributions. In Conference on Learning Theory, pages 288-316. PMLR, 2013.
[Dan15] A. Daniely. A PTAS for agnostically learning halfspaces. In Proceedings of The  $28^{th}$  Conference on Learning Theory, COLT 2015, pages 484-502, 2015.
[Dan16] A. Daniely. Complexity theoretic limitations on learning halfspaces. In Proceedings of the 48<sup>th</sup> Annual Symposium on Theory of Computing, STOC 2016, pages 105–117, 2016.
$\left[\mathrm{DGJ}^{+}10\right]$  I. Diakonikolas, P. Gopalan, R. Jaiswal, R. Servedio, and E. Viola. Bounded independence fools halfspaces. SIAM Journal on Computing, 39(8):3441-3462, 2010.
$\left[\mathrm{DKK}^{+}21\right]$  I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Agnostic proper learning of halfspaces under gaussian margins. In Proceedings of The  $34^{\text{th}}$  Conference on Learning Theory, COLT, 2021.
[DKMR22] I. Diakonikolas, D. M. Kane, P. Manurangsi, and L. Ren. Cryptographic hardness of learning halfspaces with massart noise. CoRR, abs/2207.14266, 2022.
[DKN10] I. Diakonikolas, D. M. Kane, and J. Nelson. Bounded independence fools degree-2 threshold functions. In FOCS, pages 11–20, 2010.
[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Proceedings of The  $34^{\text{th}}$  Conference on Learning Theory, COLT, 2021.
[DKR23] I. Diakonikolas, D. M. Kane, and L. Ren. Near-optimal cryptographic hardness of agnostically learning halfspaces and relu regression under gaussian marginals. CoRR, abs/2302.06512, 2023.
[DKS18] I. Diakonikolas, D. M. Kane, and A. Stewart. Learning geometric concepts with nasty noise. In Proceedings of the  $50^{\text{th}}$  Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, pages 1061-1073, 2018.
[DKTZ20a] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with massart noise under structured distributions. In Conference on Learning Theory, COLT, 2020.
[DKTZ20b] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Non-convex SGD learns halfspaces with adversarial label noise. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
[DKZ20] I. Diakonikolas, D. M. Kane, and N. Zarifis. Near-optimal SQ lower bounds for agnostically learning halfspaces and ReLUs under Gaussian marginals. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
[GGK20] S. Goel, A. Gollakota, and A. R. Klivans. Statistical-query lower bounds via functional gradients. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
[GKK22] A. Gollakota, A. Klivans, and P. Kothari. A moment-matching approach to testable learning and a new characterization of rademacher complexity. arXiv preprint arXiv:2211.13312, 2022. To appear in STOC'23.
[GKSV23] A. Gollakota, A. R. Klivans, K. Stavropoulos, and A. Vasilyan. An efficient tester-learner for halfspaces. CoRR, abs/2302.14853, 2023.
[Hau92] D. Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation, 100:78-150, 1992.
[KKMS08] A. Kalai, A. Klivans, Y. Mansour, and R. Servedio. Agnostically learning halfspaces. SIAM Journal on Computing, 37(6):1777-1805, 2008. Special issue for FOCS 2005.
[KLS09] A. Klivans, P. Long, and R. Servedio. Learning Halfspaces with Malicious Noise. Journal of Machine Learning Research, 10:2715-2740, 2009.
[KSS94] M. Kearns, R. Schapire, and L. Sellie. Toward Efficient Agnostic Learning. Machine Learning, 17(2/3):115-141, 1994.
[Ros58] F. Rosenblatt. The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65:386-407, 1958.
[RV23] Ronitt Rubinfeld and Arsen Vasilyan. Testing distributional assumptions of learning algorithms. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 1643-1656, 2023.
[She21] J. Shen. On the power of localized perceptron for label-optimal learning of halfspaces with adversarial noise. In International Conference on Machine Learning, pages 9503-9514. PMLR, 2021.
[Tie22] S. Siegel. Hardness of agnostically learning halfspaces from worst-case lattice problems. CoRR, abs/2207.14030, 2022.
[Val84] L. G. Valiant. A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing (STOC), pages 436-445. ACM Press, 1984.
A Omitted Proofs for Weak Testable Proper Agnostic Learning
Proof of Lemma 2.3. It suffices to show that for any unit vector  $\mathbf{u} \in \mathbb{R}^d$ , the following holds:
$$
\left| \mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] \right| \leq O (\sqrt {\eta}).
$$
The following fact expresses a real number  $a$  as an integral of the sign function.
Fact A.1. For any  $a \in \mathbb{R}$ , it holds  $a = \frac{1}{2} \int_{0}^{\infty} (\mathrm{sign}(a - t) + \mathrm{sign}(a + t)) \, \mathrm{d}t$ .
We apply Fact A.1 to the term  $\mathbf{u} \cdot \mathbf{x}$ , which gives
$$
\begin{array}{l} \left| \underset {\mathbf {x} \sim B} {\mathbf {E}} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] \right| = \frac {1}{2} \left| \underset {\mathbf {x} \sim B} {\mathbf {E}} \left[ h (\mathbf {x}) \int_ {t \geq 0} (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t))   \mathrm {d} t \right] \right. \\ \left. - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} \left[ h (\mathbf {x}) \int_ {t \geq 0} \left(\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)\right) \mathrm {d} t \right] \right| \\ = \frac {1}{2} \Bigg | \int_ {t \geq 0} \left(\mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) ] \right. \\ \left. - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} \left[ h (\mathbf {x}) \left(\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)\right) \right]\right) \mathrm {d} t \Bigg |, \\ \end{array}
$$
where in the last line we switch the order of the integral of  $t$  and  $\mathbf{x}$  by Fubini's theorem. We then split the above integral over  $t$  into two parts based on the magnitude of  $t$  ( $t > 1 / \sqrt{\eta}$  versus  $0 \leq t \leq 1 / \sqrt{\eta}$ ) and apply the triangle inequality:
$$
\begin{array}{l} \left| \mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] \right| \\ \leq \frac {1}{2} \left| \int_ {0 \leq t \leq 1 / \sqrt {\eta}} \left(\underset {\mathbf {x} \sim B} {\mathbf {E}} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) ] - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) ]\right) d t \right| \\ + \frac {1}{2} \left| \int_ {0 \leq t \leq 1 / \sqrt {\eta}} \left(\mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t) ]\right) d t \right| \\ + \frac {1}{2} \Bigg | \int_ {t \geq 1 / \sqrt {\eta}} \left(\mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) ] \right. \\ \left. - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} \left[ h (\mathbf {x}) \left(\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)\right) \right]\right) \mathrm {d} t \Bigg |. \tag {2} \\ \end{array}
$$
We start by bounding the integral for  $t \geq 1 / \sqrt{\eta}$ .
Lemma A.2 (Chow-Distance Tail). Let  $Q$  be distribution over  $\mathbb{R}^d$  with  $\mathbf{E}_{\mathbf{x}\sim Q}[\mathbf{x}\mathbf{x}^\top ]\preccurlyeq 2\mathbf{I}$ . Moreover, let  $g(\mathbf{x}):\mathbb{R}^d\mapsto \mathbb{R}$  be a bounded function, i.e.,  $|g(\mathbf{x})|\leq 1$  for all  $\mathbf{x}\in \mathbb{R}^d$ . It holds
$$
\left| \int_ {t \geq 1 / \sqrt {\eta} ^ {\mathbf {x} \sim Q}} \mathbf {E} [ g (\mathbf {x}) (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) ] \mathrm {d} t \right| \leq O (\sqrt {\eta}).
$$
Proof. We split the expectation into two parts based on the relative sizes of  $|\mathbf{u} \cdot \mathbf{x}|$  and  $t$ . Specifically, we can write:
$$
\begin{array}{l} \left| \int_ {t \geq 1 / \sqrt {\eta}} \mathbf {E} _ {\mathbf {x} \sim Q} \left[ g (\mathbf {x}) \left(\mathrm {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \mathrm {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)\right) \right] \mathrm {d} t \right| \\ \leq \left| \int_ {t \geq 1 / \sqrt {\eta} ^ {\mathbf {x} \sim Q}} \mathbf {E} [ g (\mathbf {x}) (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) \mathbb {1} \{| \mathbf {u} \cdot \mathbf {x} | \geq t \} ] d t \right| \\ + \left| \int_ {t \geq 1 / \sqrt {\eta} ^ {\mathbf {x} \sim Q}} \mathbf {E} [ g (\mathbf {x}) (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) \mathbb {1} \{| \mathbf {u} \cdot \mathbf {x} | \leq t \} ] d t \right|. \tag {3} \\ \end{array}
$$
For the second term in Equation (3), we rely on the following observation: when  $|\mathbf{u}\cdot \mathbf{x}|\leq t$ , the quantities  $\mathbf{u}\cdot \mathbf{x} - t$  and  $\mathbf{u}\cdot \mathbf{x} + t$  have opposite signs. Hence, we conclude the integrand is 0 everywhere and therefore the second term is also 0. For the first term, we have
$$
\begin{array}{l} \left| \int_ {t \geq 1 / \sqrt {\eta}} \mathbf {E} _ {\mathbf {x} \sim Q} [ g (\mathbf {x}) (\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) \mathbb {1} \{| \mathbf {u} \cdot \mathbf {x} | \geq t \} ] \right| \\ \leq \int_ {t \geq 1 / \sqrt {\eta}} \mathbf {E} _ {\mathbf {x} \sim Q} \left[ \left| g (\mathbf {x}) \left(\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)\right) \mathbb {1} \{| \mathbf {u} \cdot \mathbf {x} | \geq t \} \right| \right] \\ \leq \int_ {t \geq 1 / \sqrt {\eta}} \mathbf {E} _ {\mathbf {x} \sim Q} \left[ 2 \mathbb {1} \{| \mathbf {u} \cdot \mathbf {x} | \geq t \} \right] \leq 4 \int_ {t \geq 1 / \sqrt {\eta}} \frac {1}{t ^ {2}} \leq O (\sqrt {\eta}), \\ \end{array}
$$
where the first inequality follows from the triangle inequality, the second inequality uses the fact that the  $\mathrm{sign}(\cdot)$  function is at most 1 and the third inequality follows from Chebyshev's inequality using the fact that the  $\mathbf{E}[\mathbf{xx}^{\top}] \preccurlyeq 2\mathbf{I}$ . Combining our analysis for the two terms in Equation (3), we can then conclude the proof of Lemma A.2.
Using the triangle inequality and applying Lemma A.2 on the distributions  $B$  and  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , we have that
$$
\begin{array}{l} \frac {1}{2} \Bigg | \int_ {t \geq 1 / \sqrt {\eta}} \left(\mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) (\mathrm {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \mathrm {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)) ] \right. \\ \left. - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} \left[ h (\mathbf {x}) \left(\operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) + \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t)\right) \right]\right) \mathrm {d} t \Bigg | \leq O (\sqrt {\eta}). \tag {4} \\ \end{array}
$$
We then turn our attention to the terms
$$
\left| \int_ {0 \leq t \leq 1 / \sqrt {\eta}} \left(\mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) ]\right) d t \right|. \tag {5}
$$
$$
\left| \int_ {0 \leq t \leq 1 / \sqrt {\eta}} \left(\mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} + t) ]\right) d t \right|. \tag {6}
$$
To bound Equations (5) and (6), we need the following fact from [GKK22].
Fact A.3 (Theorem 5.6 of [GKK22]). Let  $h: \mathbb{R}^d \mapsto \{\pm 1\}$  be a function of  $p$  halfspaces, i.e.,  $h(\mathbf{x}) = g(h_1(\mathbf{x}), \dots, h_p(\mathbf{x}))$  where  $h_i$  are halfspaces and  $g: \{\pm 1\}^p \mapsto \{\pm 1\}$ . For any  $k \in \mathbb{N}$ , let  $\Delta = \frac{\sqrt{p}}{2k} \frac{1}{d^k} \left( \frac{1}{C' \sqrt{k}} \right)^{k+1}$  for some sufficiently large absolute constant  $C' > 0$ . Then, for any distribution  $B$  whose moments up to order  $k$  match those of  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  up to  $\Delta$ , we have
$$
\left| \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} [ h (\mathbf {x}) ] - \underset {\mathbf {x} \sim B} {\mathbf {E}} [ h (\mathbf {x}) ] \right| \leq \frac {1}{\sqrt {k}} \sqrt {p} \left(C \log \left(\sqrt {p k}\right)\right) ^ {2 p}.
$$
for some constant  $C > 0$
For a fixed  $t$ , note that  $h(\mathbf{x}) \mathrm{sign}(\mathbf{u} \cdot \mathbf{x} - t)$  is a function of two halfspaces. Moreover, from the assumptions of Lemma 2.3, the distributions  $B$  and  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  match  $k = C \log(1 / \eta) / \eta^2$  moments up to error  $\Delta = \frac{1}{kd^k} \left( \frac{1}{C \sqrt{k}} \right)^{k + 1}$ , where  $C > 0$  is a sufficiently large absolute constant. Therefore, applying Fact A.3 and the triangle inequality gives
$$
\begin{array}{l} \frac {1}{2} \left| \int_ {0 \leq t \leq 1 / \sqrt {\eta}} \left(\underset {\mathbf {x} \sim B} {\mathbf {E}} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) ] - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} [ h (\mathbf {x}) \operatorname {s i g n} (\mathbf {u} \cdot \mathbf {x} - t) ]\right) \mathrm {d} t \right| \\ \leq O (1) \int_ {0 \leq t \leq 1 / \sqrt {\eta}} \eta \mathrm {d} t = O (\sqrt {\eta}). \tag {7} \\ \end{array}
$$
Similarly, we can show that Equation (6) is bounded by  $O(\sqrt{\eta})$ . Substituting the bounds from Equations (4) and (7) into Equation (2) then gives
$$
\left| \underset {\mathbf {x} \sim B} {\mathbf {E}} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] - \underset {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} {\mathbf {E}} [ h (\mathbf {x}) \mathbf {x} \cdot \mathbf {u} ] \right| \leq O (\sqrt {\eta}).
$$
Since  $\mathbf{u}$  is chosen as an arbitrary unit vector, this implies that
$$
\left\| \mathbf {E} _ {\mathbf {x} \sim B} [ h (\mathbf {x}) \mathbf {x} ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} [ h (\mathbf {x}) \mathbf {x} ] \right\| _ {2} \leq O (\sqrt {\eta}).
$$
Combining this with Fact 2.2 concludes the proof of Lemma 2.3.
Proof of Lemma 2.4. We first show that  $\left\| \mathbf{E}_{\mathbf{x}\sim G_{\mathbf{x}}}\left[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\mathbf{x}\right] - \mathbf{E}_{(\mathbf{x},y)\sim G}[y\mathbf{x}]\right\| _2\leq O(\sqrt{\eta})$  . For any unit vector  $\mathbf{u}$  , we have that
$$
\begin{array}{l} \underset {\mathbf {x} \sim G _ {\mathbf {x}}} {\mathbf {E}} [ \operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) \mathbf {u} \cdot \mathbf {x} ] - \underset {(\mathbf {x}, y) \sim G} {\mathbf {E}} [ y \mathbf {u} \cdot \mathbf {x} ] = \underset {(\mathbf {x}, y) \sim G} {\mathbf {E}} [ (\operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) - y) \mathbf {u} \cdot \mathbf {x} ] \\ \leq \sqrt {\underset {(\mathbf {x} , y) \sim G} {\mathbf {E}} [ (\operatorname {s i g n} (\mathbf {v} \cdot \mathbf {x}) - y) ^ {2} ]} \underset {\mathbf {x} \sim G _ {\mathbf {x}}} {\mathbf {E}} [ (\mathbf {u} \cdot \mathbf {x}) ^ {2} ] \\ \leq 4 \sqrt {\eta}, \\ \end{array}
$$
where we used the Cauchy-Schwarz inequality and the fact that  $\mathbf{Pr}_{(\mathbf{x},y)\sim G}[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\neq y]\leq \eta$ . Therefore, we have that  $\left\| \mathbf{E}_{\mathbf{x}\sim G_{\mathbf{x}}}\left[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\mathbf{x}\right] - \mathbf{E}_{(\mathbf{x},y)\sim G}\left[y\mathbf{x}\right]\right\|_2\leq 4\sqrt{\eta}$ . Let  $(\mathbf{x}^{(1)},y^{(1)}),\ldots ,(\mathbf{x}^{(N_1)},y^{(N_1)})$  be samples drawn from  $G$ , where  $N_{1} = O(d / \eta^{2})$ . Then, let  $\widetilde{\mathbf{w}} = (1 / N_1)\sum_{j = 1}^{N_1}y^{(j)}\mathbf{x}^{(j)}$ . From Chebyshev's inequality, we know the  $i$ -th coordinate of  $\widetilde{\mathbf{w}}$  should satisfy  $\mathbf{Pr}[|\widetilde{\mathbf{w}}_i - \mathbf{E}_{(\mathbf{x},y)\sim D}[y\mathbf{x}]|\geq \eta /\sqrt{d} ]\leq 4d / (N_1\eta^2)\leq 1 / 4$ . Therefore, using the standard median technique, we can find a  $\mathbf{w}_i^{\mathrm{median}}$ , so that  $\mathbf{Pr}[|\mathbf{w}_i^{\mathrm{median}} - \mathbf{E}_{(\mathbf{x},y)\sim D}[y\mathbf{x}]|\geq \eta /\sqrt{d} ]\leq \tau /d$  using  $N_{2} = O(N_{1}\log (d / \tau))$  samples. Let  $\mathbf{w} = (\mathbf{w}_1^{\mathrm{median}},\dots ,\mathbf{w}_d^{\mathrm{median}})$ , then we have that  $\left\| \mathbf{E}_{(\mathbf{x},y)\sim G}[y\mathbf{x}] - \mathbf{w}\right\|_2\leq O(\eta)$  with probability at least  $1 - \tau$ . Then, using the triangle inequality, we have that  $\left\| \mathbf{E}_{\mathbf{x}\sim G_{\mathbf{x}}}\left[\mathrm{sign}(\mathbf{v}\cdot \mathbf{x})\mathbf{x}\right] - \mathbf{w}\right\|_2\leq O(\sqrt{\eta})$ , which concludes the proof of Lemma 2.4.
Proof of Proposition 2.1. Let  $k, \Delta, N$  be defined as in Algorithm 2. If  $D_{\mathbf{x}}$  is  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ , the moments up to degree  $k$  of the  $\mathbf{x}$ -marginal of the empirical distribution  $\widehat{D}_N$  (obtained after drawing  $N$  i.i.d. samples from  $D_{\mathbf{x}}$ ) are close to those of  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  up to additive error  $\Delta$  with probability at least  $1 - \tau / 10$ .
If Algorithm 2 did not terminate on Line 3, we then have that the moments up to degree  $k$  of  $\widehat{D}_N$  are close to those of  $\mathcal{N}(\mathbf{0},\mathbf{I})$  up to additive error  $\Delta$  with probability at least  $1 - \tau /10$ . Then, applying Lemma 2.3 with  $B = \widehat{D}_N$  and  $h(\mathbf{x}) = \mathrm{sign}(\mathbf{v}^*\cdot \mathbf{x})$ , we get that
$$
\left\| \mathbf {E} _ {\mathbf {x} \sim \hat {D} _ {N}} [ \operatorname {s i g n} (\mathbf {v} ^ {*} \cdot \mathbf {x}) \mathbf {x} ] - \sqrt {2 / \pi} \mathbf {v} ^ {*} \right\| _ {2} \leq O (\sqrt {\eta}). \tag {8}
$$
Now, suppose we were to query the labels of all points in  $\widehat{D}_N$ . We could then construct another empirical distribution  $\widehat{L}_N$  over labeled samples. By our assumption, the error of  $\mathrm{sign}(\mathbf{v}^*\cdot \mathbf{x})$  under
Input: Sample access to a distribution  $D_{\mathbf{x}}$  over unlabeled samples and query access to the labels of the drawn samples according to  $D$ ; certification range  $\eta$ ; failure probability  $\tau$ .
Output: Either reports that the  $D_{\mathbf{x}}$  is not  $\mathcal{N}(\mathbf{0},\mathbf{I})$ ; or returns a unit vector  $\mathbf{w}\in \mathbb{R}^d$  such that  $\| \mathbf{v}^{*} - \mathbf{w}\|_{2}\leq C_{\mathrm{A}}\sqrt{\mathrm{opt} + \eta}$ .
1. Set  $k = C\log (1 / \eta) / \eta^2$  and  $\Delta = \frac{1}{kd^k}\left(\frac{1}{C\sqrt{k}}\right)^{k + 1}$ , where  $C > 0$  is a sufficiently large absolute constant.
2. Draw  $N = d^{Ck\log k}\log (1 / \tau)$  samples from  $D_{\mathbf{x}}$  and construct the empirical distribution  $\widehat{D}_N$ .
3. Certify that the moments of  $\widehat{D}_N$  up to degree  $k$  match with those of  $\mathcal{N}(\mathbf{0},\mathbf{I})$  up to error  $\Delta$ .
4. If the above does not hold; report that  $D_{\mathbf{x}}$  is not  $\mathcal{N}(\mathbf{0}, \mathbf{I})$  and terminate.
5. Draw  $N' = O(d / \eta^2) \log (d / \eta)$  samples from  $\hat{D}_N$  and query their labels. Denote the resulting labeled samples as  $S$ .
6. Use algorithm from Lemma 2.4 on  $S$  and obtain  $\mathbf{w}$ . Return  $\mathbf{w} / \| \mathbf{w}\| _2$ .
Algorithm 2: Proper Testable Learner
$D$  is at most opt. Hence, the error of  $\mathrm{sign}(\mathbf{v}^*\cdot \mathbf{x})$  under  $\widehat{L}_N$  is at most  $\mathrm{opt} + \eta$  with probability at least  $1 - \tau /10$ . Next, notice that the set of samples  $S$  in Line 5 are exactly i.i.d. samples from  $\widehat{L}_N$ . Hence, applying Lemma 2.4 on  $\widehat{L}_N$  then gives that, with probability at least  $1 - \tau /10$ , the vector  $\mathbf{w}$  computed on Line 6 of Algorithm 2 satisfies
$$
\left\| _ {\mathbf {x} \sim (\hat {L} _ {N}) _ {\mathbf {x}}} \left[ \operatorname {s i g n} \left(\mathbf {v} ^ {*} \cdot \mathbf {x}\right) \mathbf {x} \right] - \mathbf {w} \right\| _ {2} \leq O \left(\sqrt {\operatorname {o p t} + \eta}\right). \tag {9}
$$
Notice that the  $\mathbf{x}$ -marginal of  $\widehat{L}_N$  is exactly  $\widehat{D}_N$ . Hence, combining Equations (8) and (9), we get that  $\left\| \mathbf{w} - \sqrt{2 / \pi} \mathbf{v}^* \right\|_2 \leq O\left(\sqrt{\mathrm{opt} + \eta}\right)$ , as desired.
B Proof of Wedge Bound Lemma 3.1
Proof of Lemma 3.1. Recall that  $\widetilde{D}$  is the empirical distribution made up of  $N$  i.i.d. samples from  $D_{\mathbf{x}}$  where  $N = \mathrm{poly}(d,1 / \eta)\log (1 / \tau)$ . We consider  $\widetilde{D}$  restricted to a set of thin "slabs" stacked on each other in the direction of  $\mathbf{v}$ . More formally, we define  $S_{i}$  to be the distribution of  $\widetilde{D}$  conditioned on  $\mathbf{v}\cdot \mathbf{x}\in [(i - 1)\eta ,i\eta ]$ , for  $i\in [-\sqrt{\log(1 / \eta)} /\eta ,\sqrt{\log(1 / \eta)} /\eta ]$ , and  $S_{i}^{\perp}$  to be the distribution  $S_{i}$  projected into the subspace orthogonal to  $\mathbf{v}$ .
Suppose that  $D_{\mathbf{x}}$  is indeed  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ . Then the distribution  $\widetilde{D}$  is the empirical distribution formed by samples taken from  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ . In this case, it is easy to see that both Line 4 and 6 of Algorithm 1 pass with high probability.
Claim B.1. Assume that  $D_{\mathbf{x}} = \mathcal{N}(\mathbf{0},\mathbf{I})$ . Then the tests at Line 4 and 6 of Algorithm 1 pass with probability at least  $1 - \tau /10$