SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Claim 3.4. It holds that  $|J|\leq 4|\cot (\theta)|\exp (-t^2)\exp \big(|\cos (\theta)|t^2\big)$
Proof of Claim 3.4. Note that from Claim C.7, we have that  $c_{i} = 2\mathrm{He}_{i - 1}(t)\exp (-t^{2} / 2) / \sqrt{i}$ , hence, it holds that  $J = 4\cos (\theta)\exp (-t^2)\sum_{i = 1}^{\infty}i^{-1}\mathrm{He}_{i - 1}^2 (t)\cos^{i - 1}\theta$ . We use the following fact.
Fact 3.5 (Mehler Formula, see, e.g. [Foo78]). For  $|\rho| < 1$  and  $x, y \in \mathbb{R}$ , it holds that
$$
\exp \left(- \frac {1}{2} \frac {\rho}{1 - \rho^ {2}} (x - y) ^ {2}\right) = \sqrt {1 - \rho^ {2}} \sum_ {k \geq 0} \rho^ {k} \mathrm {H e} _ {k} (x) \mathrm {H e} _ {k} (y) \exp \left(- \frac {1}{2} \frac {\rho}{1 + \rho^ {2}} (x ^ {2} + y ^ {2})\right).
$$
Applying Fact 3.5 for  $\rho = \cos (\theta)$  and  $x = y = t$ , we get that
$$
\begin{array}{l} \left| J \right| = 4 \big | \cos \theta \exp (- t ^ {2}) \sum_ {i \geq 1} i ^ {- 1} \mathrm {H e} _ {i - 1} ^ {2} (t) \cos^ {i - 1} \theta \big | \leq 4 \big | \cos \theta \big | \exp (- t ^ {2}) \sum_ {i \geq 0} \mathrm {H e} _ {i} ^ {2} (t) \big | \cos \theta \big | ^ {i} \\ = 4 | \cot \theta | \exp (- t ^ {2}) \exp \left(\frac {| \cos (\theta) | t ^ {2}}{(1 + \cos^ {2} \theta)}\right). \\ \end{array}
$$
This completes the proof of Claim 3.4.
Using Claim 3.4, we get that
$$
\left| \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ f _ {\mathbf {v}} (\mathbf {x}) f _ {\mathbf {u}} (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ f _ {\mathbf {v}} (\mathbf {x}) ] \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ f _ {\mathbf {u}} (\mathbf {x}) ] \right| \leq 4 | \cot (\theta) | \exp (- t ^ {2}) \exp \left(| \cos (\theta) | t ^ {2}\right),
$$
completing the proof of Lemma 3.3.
We associate each  $\mathbf{v}$  and  $\mathbf{u}$  to a distribution  $D_{\mathbf{v}}$  and  $D_{\mathbf{u}}$ , constructed as above. The following lemma provides explicit bounds on the correlation between the distributions  $D_{\mathbf{v}}$  and  $D_{\mathbf{u}}$ . Recall that the pairwise correlation of two distributions with cdfs  $D_{1}, D_{2}$  with respect to a distribution with cdf  $D$  is defined as  $\chi_{D}(D_{1}, D_{2}) + 1 \coloneqq \int_{x \in \mathcal{X}} D_{1}(x) D_{2}(x) / D(x)$  (see Definition C.1). We have the following lemma (see Appendix C.4 for its proof):
Lemma 3.6. Let  $D_0$  be a product distribution distributed as  $\mathcal{N} \times \{\pm 1\}$ , where  $\mathbf{Pr}_{(\mathbf{x},y) \sim D_0}[y = 1] = \mathbf{Pr}_{(\mathbf{x},y) \sim D_{\mathbf{v}}} [y = 1] = p$ . We have  $\chi_{D_0}(D_{\mathbf{v}}, D_{\mathbf{u}}) \leq 2(1 - 2\eta)(\mathbf{E}[f_{\mathbf{v}}(\mathbf{x}) f_{\mathbf{u}}(\mathbf{x})] - \mathbf{E}[f_{\mathbf{v}}(\mathbf{x})]\mathbf{E}[f_{\mathbf{u}}(\mathbf{x})])$  and  $\chi^2(D_{\mathbf{v}}, D_0) \leq (1 - 2\eta)(\mathbf{E}[f_{\mathbf{v}}(\mathbf{x})] - \mathbf{E}[f_{\mathbf{v}}(\mathbf{x})]^2)$ .
For any  $c \in (0,1/2)$ , there exists a set  $\mathcal{S}$  of  $2^{\Omega(d^{c})}$  unit vectors in  $\mathbb{R}^{d}$  such that for any pair  $\mathbf{v} \neq \mathbf{u} \in \mathcal{S}$  satisfies  $|\mathbf{v} \cdot \mathbf{u}| < d^{-1/2 + c}$  (Fact C.4). We associate each  $\mathbf{v} \in \mathcal{S}$  with  $f_{\mathbf{v}}$  and a distribution  $D_{\mathbf{v}}$  and denote  $\mathfrak{D} = \{D_{\mathbf{v}}, \mathbf{v} \in \mathcal{S}\}$ . By the definition of  $\mathcal{S}$  and Lemma 3.3, for any  $\mathbf{v}, \mathbf{u} \in \mathcal{S}$ , we have that  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x}) f_{\mathbf{u}}(\mathbf{x})] \leq 4d^{-1/2 + c} \exp(-t^{2}) \exp((t / d^{1/4 - c/2})^{2}) + \mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x})]\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{u}}(\mathbf{x})]$ . Since  $t / d^{1/4 - c/2} \leq 1/2$  by assumption, we get that  $|\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x}) f_{\mathbf{u}}(\mathbf{x})] - \mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x})]\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{u}}(\mathbf{x})]| \leq d^{-1/2 + c} \exp(-t^{2})$ . By Lemma 3.6, it follows that  $\chi_{D_0}(D_{\mathbf{v}}, D_{\mathbf{u}}) \leq C(1 - 2\eta) \exp(-t^{2}) d^{-1/2 + c}$  and  $\chi^2(D_{\mathbf{v}}, D_0) \leq C(1 - 2\eta) \exp(-t^{2}/2)$ , where  $C > 0$  is an absolute constant. From standard SQ machinery (see, e.g., Lemma C.3), we have that any SQ algorithm that solves the decision problem  $\mathcal{B}(\mathfrak{D}, D_0)$ , requires either  $2^{\Omega(d^{c})}$  queries, or at least one query to VSTAT ( $\exp(t^{2}) d^{1/2 + c}$ ). Noting that  $p = O(\exp(-t^{2}/2)/t)$  (by Fact B.1) completes the proof of Theorem 3.2.
ID was supported by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), and a DARPA Learning with Less Labels (LwLL) grant. JD was supported by NSF Award CCF-2007757 and by the U. S. Office of Naval Research under award number N00014-22-1-2348. DK was supported by NSF Medium Award CCF-2107547 and NSF Award CCF-1553288 (CAREER). PW was supported in part by NSF Award CCF-2007757. NZ was supported in part by NSF award 2023239, NSF Medium Award CCF-2107079, and a DARPA Learning with Less Labels (LwLL) grant.
[Fel16] V. Feldman. Statistical query learning. In Encyclopedia of Algorithms, pages 2090-2095. 2016.
$\left[\mathrm{FGR}^{+}17\right]$  V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. Statistical algorithms and a lower bound for detecting planted cliques. J. ACM, 64(2):8:1-8:37, 2017.
[FGV17] V. Feldman, C. Guzman, and S. S. Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Philip N. Klein, editor, Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017, pages 1265-1277. SIAM, 2017.
[Foa78] D. Foata. A combinatorial proof of the mehler formula. Journal of Combinatorial Theory, Series A, 1978.
[FS97] Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139, 1997.
[God55] H. J. Godwin. On generalizations of tchebychef's inequality. Journal of the American Statistical Association, 50(271):923-945, 1955.
[HY15] S. Hanneke and L. Yang. Minimax analysis of active learning. J. Mach. Learn. Res., 16:3487-3602, 2015.
[Kea98] M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983-1006, 1998.
[MSN06] P. Massart and E. Nedelec. Risk bounds for statistical learning. Ann. Statist., 34(5):2326-2366, 10 2006.
[Ros58] F. Rosenblatt. The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65:386-407, 1958.
[Val84a] L. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.
[Val84b] L. G. Valiant. A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing (STOC), pages 436-445. ACM Press, 1984.
[Vap98] V. Vapnik. Statistical Learning Theory. Wiley-Interscience, New York, 1998.
[YZ17] S. Yan and C. Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 1056-1066, 2017.
[ZL21] C. Zhang and Y. Li. Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise. In Proceedings of The 34th Conference on Learning Theory, COLT, 2021.
[ZLC17] Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient Langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, COLT 2017, pages 1980–2022, 2017.
[ZSA20] C. Zhang, J. Shen, and P. Awasthi. Efficient active learning of sparse halfspaces with arbitrary bounded noise. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
Organization The appendix is organized as follows: In Appendix A, we provide additional comparison to prior work. In Appendix B, we present the full version of Section 2, completing proofs and providing supplementary lemmas omitted in the main body. In Appendix C, we start with background on Hermite polynomials and the SQ model, followed by omitted proofs from Section 3. Finally, in Appendix D, we prove our lower bound for low-degree polynomial testing.
A Comparison with Previous Work
Here we provide a more detailed comparison with the very recent work of  $\mathrm{[DDK^{+}23]}$ , which gives an algorithm and an SQ lower bound for learning  $\gamma$ -margin halfspaces in the presence of RCN. We start by noting that the SQ-hard instance of  $\mathrm{[DDK^{+}23]}$  is based on a discrete distribution on the hypercube. Consequently, neither the construction nor its analysis have any implications on the Gaussian setting studied here. More generally, the margin assumption intuitively captures a much more general family of distributions than the Gaussian distribution (though, formally speaking, the two assumptions are incomparable, as the Gaussian only exhibits an approximate margin property). On the positive side,  $\mathrm{[DDK^{+}23]}$  gives an efficient algorithm for learning margin halfspaces in the presence of RCN with sample complexity  $\widetilde{O}(1 / (\gamma^2\epsilon^2))$ . It is important to note that the homogeneity (i.e., origin-centered) assumption provably does not help for the margin case — i.e., the case of homogeneous halfspaces with a margin is as hard as the case of general halfspaces with a margin. In sharp contrast, our algorithm in this work has sample complexity that crucially depends on the unknown bias  $p$  of the target halfspace — interpolating between  $\widetilde{O}(d / \epsilon)$  and  $\widetilde{O}(d / \epsilon^2)$ .
B Full Version of Section 2
Throughout this paper, we frequently use the following fact. For  $\mathbf{x}$  drawn from the standard normal distribution, the threshold  $t$  in the definition of the halfspace can be related to the bias of  $f(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^{*}\cdot \mathbf{x} + t)$ , as follows.
Fact B.1 (Komatsu's Inequality). For any  $t \in \mathbb{R}$ , the bias  $p$  of a halfspace described by  $f(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^* \cdot \mathbf{x} + t)$  can be bounded as:
$$
\sqrt {\frac {2}{\pi}} \frac {\exp (- t ^ {2} / 2)}{t + \sqrt {t ^ {2} + 4}} \leq p \leq \sqrt {\frac {2}{\pi}} \frac {\exp (- t ^ {2} / 2)}{t + \sqrt {t ^ {2} + 2}}.
$$
We begin by stating our main result, in Theorem 1.3 below, and providing a high-level summary of the main algorithm, in Algorithm 3. We note that both the assumption that  $t \geq 0$  and that  $t \leq \sqrt{2\log((1 - 2\eta) / \epsilon)}$  are without loss of generality. For the former, it is by the simple symmetry of the standard normal distribution that the entire argument translates into the case  $t < 0$ , possibly by exchanging the meaning of  $+1$  and  $-1$  labels. For the latter, we note that when the bias is small, i.e., for  $p \leq \epsilon / (2(1 - 2\eta))$ , a constant hypothesis suffices. Thus, only the cases covered by Theorem 1.3 are of interest to us. For the rest of the section, we assume that  $\eta$  is known a priori; we show in Appendix B.3 an efficient way to estimate it without increasing the sample complexity.
Theorem 1.3. (Main Algorithmic Result) There exists an algorithm that, given  $\epsilon, \delta \in (0,1/2)$  and  $N$  samples from a distribution  $\mathcal{D}$  satisfying Definition 1.1, runs in time  $O(dN/\epsilon^2)$  and returns a hypothesis  $h \in \mathcal{C}$  such that with probability at least  $1 - \delta$ , it holds  $\mathrm{err}_{0-1}^{\mathcal{D}}(h) \leq \eta + \epsilon$ . The sample complexity of the algorithm is  $N = \tilde{O}\left(\frac{d}{(1-2\eta)\epsilon} + \frac{d}{\max(p(1-2\eta), \epsilon)^2}\right) \log(1/\delta)$ .
Remark B.2. The sample complexity stated in Theorem 1.3 has two components: the first depends on  $\epsilon$ , and the second depends on the bias  $p$ . As we demonstrate in Section 3, the  $1/p^2$  term is required for computationally efficient SQ algorithms and low-degree tests. Moreover, the term  $\widetilde{O}(d\log(1/\delta)/((1 - 2\eta)\epsilon))$  is information-theoretically optimal, even when  $p = 1/2$ , as shown, e.g., in [HY15].
At a high level, our main algorithm consists of three main subroutines, as summarized in Algorithm 3. The subroutines are specified in the rest of the section, where we analyze the sample complexity and the runtime of our algorithm, and, as a consequence, prove Theorem 1.3. In more detail, the
Algorithm 3 Main Algorithm
1: Input:  $\delta$ ,  $\eta$ ,  $\epsilon$ , sample access to distribution  $\mathcal{D}$
2:  $[\mathbf{w}_0,\hat{p} ] = \mathrm{Initialization}(\delta ,\eta ,\epsilon);\epsilon ' = \epsilon /(1 - 2\eta)$
3:  $t_0 = \sqrt{2\log(1 / \hat{p})}$ ,  $M = 8\left[\frac{\sqrt{2(\log(4 / \hat{p}))} - \sqrt{2\log(1 / \hat{p})}}{(\epsilon')^2}\right] + 1$
4: Draw  $N_{2} = O\left(\frac{d\log(1 / \delta)\log(1 / \epsilon^{\prime})}{(1 - 2\eta)^{2}\epsilon^{\prime}}\right)$  samples  $\{(x^{(i)},y^{(i)})\}_{i = 1}^{N_2}$  from  $\mathcal{D}$
5: for  $m = 1:M$  do
6:  $t_m = t_0 + (m - 1)\frac{(\epsilon')^2}{8}, \gamma_m = \frac{\epsilon'}{2}\exp(t_m^2 / 2)$
7:  $\widehat{\mathbf{w}}_m = \mathrm{Optimization}(\mathbf{w}_0, t_m, \gamma_m, \eta, \{(x^{(i)}, y^{(i)})\}_{i=1}^{N_2})$
8: end for
9:  $[\widehat{\mathbf{w}}_{\mathrm{out}},t_{\mathrm{out}}] = \mathrm{Testing}((\widehat{\mathbf{w}}_1,t_1),(\widehat{\mathbf{w}}_2,t_2),\ldots ,(\widehat{\mathbf{w}}_M,t_M))$
10: return  $\widehat{\mathbf{w}}_{\mathrm{out}}$ $t_\mathrm{out}$
Initialization subroutine (specified in Algorithm 4) ensures that we can choose a vector  $\mathbf{w}_0$  from the unit sphere that forms a sufficiently small angle with the target vector  $\mathbf{w}^*$ , using  $\widetilde{O}(d\log(1/\delta)/((1 - 2\eta)^2p^2))$  samples. This can be seen as warm-start for the main optimization procedure (specified in Algorithm 5). Crucially, the Initialization procedure does not require knowing the bias  $p$  of the target halfspace; instead, it estimates this parameter to a constant factor.
The Optimization procedure is run for different guesses of the threshold  $t$  ( $O(\log (1 / \epsilon) / \epsilon^2)$  of them). At a high level, it can be seen as a variant of Riemannian (sub)gradient descent on the unit sphere, applied to the empirical LeakyReLU loss — defined as  $\text{LeakyReLU}_{\lambda}(u) = (1 - \lambda)u\mathbb{1}\{u \geq 0\} + \lambda u\mathbb{1}\{u < 0\}$  — with parameter  $\lambda$  set to  $\eta$ ,  $u = \mathbf{w} \cdot \mathbf{x}$ , and with samples restricted to a band, namely  $a < |\mathbf{w} \cdot \mathbf{x}| < b$  — with  $a$  and  $b$  chosen as functions of the guess for the threshold  $t$ . The band restriction is key in avoiding  $d / \epsilon^2$  dependence in the sample complexity, and instead only requiring order- $(d / \epsilon)$  samples to be drawn for the empirical LeakyReLU loss subgradient estimate. For a sufficiently accurate estimate  $\hat{t}$  of  $t$  (which is satisfied by at least one of the guesses for which the Optimization procedure is run), we argue that there is a sufficiently negative correlation between the empirical subgradient and the target weight vector  $\mathbf{w}^*$ . This result, combined with the Initialization result, then enables us to inductively argue that the distance between the weight vector constructed by the Optimization procedure and the target vector  $\mathbf{w}^*$  contracts and becomes smaller than  $\epsilon$  within order-  $\log (1 / \epsilon)$  iterations. This result is quite surprising, since the LeakyReLU loss is nonsmooth (it is, in fact, piecewise linear) and we do not explicitly bound its growth outside the set of its minima (i.e., we do not prove a local error bound, which would typically be used to prove linear convergence). Thus, the result as ours is impossible using black-box results for nonsmooth optimization. Additionally, we never even explicitly use the LeakyReLU loss function or prove that it is minimized by  $\mathbf{w}^*$ ; instead, we directly prove that the vectors  $\mathbf{w}$  constructed by our procedure converge to the target vector  $\mathbf{w}^*$ . On a technical level, our result is enabled by a novel inductive argument, which we believe may be of independent interest.
Finally, since each run of the Optimization subroutine returns a different hypothesis, with only the one(s) with the guess of  $t$  being sufficiently accurate satisfying our theoretical guarantee, we need a principled approach to selecting a hypothesis with the target error guarantee. This is achieved in the Testing procedure, which simply draws a fresh sample and selects a hypothesis with the lowest test error. By a standard result due to [MN06], such a hypothesis satisfies our target error guarantee.
B.1 Initialization Procedure
We begin this section with Lemma 2.1, which shows that for any  $\kappa >0$ , given  $N_{1} = \widetilde{O} (d / (\kappa^{4}p^{2}(1 - 2\eta)^{2})\log (1 / \delta))$  i.i.d. samples from  $\mathcal{D}$ , we can construct a good initial point  $\mathbf{w}_0$  that forms an angle at most  $\kappa$  with the target weight vector  $\mathbf{w}^*$ . For our purposes,  $\kappa$  should be of the order  $\frac{1}{t}$ . For  $t\leq 2\sqrt{\log(1 / \epsilon')}$ , where  $\epsilon^{\prime} = \epsilon /(1 - 2\eta)$ , we can ensure that  $N_{1} = \widetilde{O} (d / (p^{2}(1 - 2\eta)^{2})\log (1 / \delta))$ . The downside of the lemma, however, is that the number of samples  $N_{1}$  requires at least approximate knowledge of the bias parameter  $p$  (or, more accurately, of  $e^{-t^2 /2}$ ). We address this challenge by arguing (in Lemma 2.2) that we can estimate  $p$  using the procedure described in Algorithm 4, without increasing the total number of drawn samples by a factor larger than order-log  $(1 / \epsilon ')$ .
Algorithm 4 Initialization
1: Input:  $\delta, \eta, \epsilon > 0$
2: Let  $j = 0$
3: repeat
4:  $j\gets j + 1$
5:  $p_j\gets 1 / 2^j$
6: Draw  $n_j = \left\lceil \frac{32\pi d(\log(1/\delta) + \log\log((1 - 2\eta)/\epsilon))}{(1 - 2\eta)^2p_j^2} \right\rceil$  new samples from  $\mathcal{D}$
7:  $\mathbf{u}_j\gets \frac{1}{n_j}\sum_{i = 1}^{n_j}y^{(i)}\mathbf{x}^{(i)}$
8: until  $\| \mathbf{u}_j\| _2\geq \frac{3}{4}\sqrt{\frac{2}{\pi}} (1 - 2\eta)p_j$  or  $(1 - 2\eta)p_{j}\leq \epsilon$
9: Let  $\hat{p} = 2p_{j}$ $\kappa = 1 / (5\sqrt{2(\log(4) + \log(1 / \hat{p}))})$
10: Draw  $N_{1} = \lceil \frac{64\pi d\log(2 / \delta)}{\kappa^{4}(1 - 2\eta)^{2}\hat{p}^{2}}\rceil$  new samples from  $\mathcal{D}$
11:  $\mathbf{u}\gets \frac{1}{N_1}\sum_{i = 1}^{N_1}\mathbf{x}^{(i)}y^{(i)}$
12:  $\mathbf{w}_0\gets \mathbf{u} / \| \mathbf{u}\| _2$
13: return  $\mathbf{w}_0, \hat{p}$
Lemma 2.1 (Initialization via Chow Parameters). Given  $\kappa > 0$ , define  $p_t = e^{-t^2 / 2}$ ,  $N_1 = O(d / (\kappa^4 p_t^2 (1 - 2\eta)^2) \log(1 / \delta))$  and let  $(\mathbf{x}^{(i)}, y^{(i)})$  for  $i \in [N_1]$  be i.i.d. samples drawn from  $\mathcal{D}$ . Let  $\mathbf{u} = \frac{1}{N_1} \sum_{i=1}^{N_1} \mathbf{x}^{(i)} y^{(i)}$  and  $\mathbf{w}_0 = \mathbf{u} / \| \mathbf{u} \|_2$ . Then, with probability  $1 - \delta$ , we have  $\theta(\mathbf{w}_0, \mathbf{w}^*) \leq \kappa$ .
Proof. We start by showing that  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]$  is parallel to  $\mathbf{w}^*$  and has nontrivial magnitude, and then draw our conclusions from there. In particular, we prove that
$$
\underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ y \mathbf {x} ] = \sqrt {\frac {2}{\pi}} (1 - 2 \eta) p _ {t} \mathbf {w} ^ {*}. \tag {1}
$$
To do so, observe first that for any vector  $\mathbf{v}$  in the orthogonal complement of  $\mathbf{w}^*$  (i.e., such that  $\mathbf{v} \cdot \mathbf{w}^* = 0$ ),  $\mathbf{x} \cdot \mathbf{v}$  (projection of  $\mathbf{x}$  onto  $\mathbf{v}$ ) is independent of  $\mathbf{x} \cdot \mathbf{w}^*$ , as  $\mathbf{x}$  is drawn from  $\mathcal{N}$ . Thus,  $\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[y\mathbf{x}] \cdot \mathbf{v} = 0$ , which means that  $\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[y\mathbf{x}]$  is either a zero vector or parallel to  $\mathbf{w}^*$ . To determine the magnitude of this vector, we next look at the projection of  $\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[y\mathbf{x}]$  onto  $\mathbf{w}^*$ . Using that  $\mathbf{w}^* \cdot \mathbf{x}$  is a one-dimensional standard normal random variable, we have
$$
\begin{array}{l} \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ \mathrm {s i g n} (\mathbf {w} ^ {*} \cdot \mathbf {x} + t) \mathbf {w} ^ {*} \cdot \mathbf {x} ] = \underset {z \sim \mathcal {N}} {\mathbf {E}} [ \mathbb {1} \{z \geq - t \} z - \mathbb {1} \{z <   - t \} z ] \\ = \underset {z \sim \mathcal {N}} {\mathbf {E}} [ \mathbb {1} \{- t <   z <   t \} z ] + 2 \underset {z \sim \mathcal {N}} {\mathbf {E}} [ \mathbb {1} \{z \geq t \} z ] \\ = 2 \underset {z \sim \mathcal {N}} {\mathbf {E}} [ \mathbb {1} \{z \geq t \} z ] \\ = \frac {2}{\sqrt {2 \pi}} \int_ {t} ^ {+ \infty} z e ^ {- z ^ {2} / 2} \mathrm {d} z = - \frac {2}{\sqrt {2 \pi}} \int_ {t} ^ {+ \infty} \mathrm {d} \left(e ^ {- z ^ {2} / 2}\right) \\ = \sqrt {\frac {2}{\pi}} e ^ {- t ^ {2} / 2} = \sqrt {\frac {2}{\pi}} p _ {t}. \\ \end{array}
$$
Recalling the definition of  $y$ , we now obtain Equation (1). We then conclude from Equation (1) that  $\| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]\| _2 = \sqrt{\frac{2}{\pi}} (1 - 2\eta)p_t$ , as  $\| \mathbf{w}^*\| _2 = 1$ , by assumption.
The next step is to show that  $\mathbf{u} = \frac{1}{N_1}\sum_{i=1}^{N_1}y^{(i)}\mathbf{x}^{(i)}$  concentrates near its expectation. Note that  $y\mathbf{x}$  is  $\sqrt{2}$ -sub-Gaussian as  $\mathbf{x}$  is standard normal, therefore, by the multiplicative Hoeffding bound,
$$
\begin{array}{l} \mathbf {P r} \left[ \left\| \frac {1}{N _ {1}} \sum_ {i = 1} ^ {N _ {1}} y ^ {(i)} \mathbf {x} ^ {(i)} - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ y \mathbf {x} ] \right\| _ {2} \geq \frac {\kappa^ {2}}{8} \| \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ y \mathbf {x} ] \| _ {2} \right] \leq 2 \exp \left(- \frac {N _ {1} \kappa^ {4} \| \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ] \| _ {2} ^ {2}}{1 2 8 d}\right) \\ \leq 2 \exp \left(- \frac {N _ {1} \kappa^ {4} (1 - 2 \eta) ^ {2} p _ {t} ^ {2}}{6 4 \pi d}\right). \\ \end{array}
$$
Thus, choosing  $N_{1} = \lceil \frac{64\pi d\log(2 / \delta)}{\kappa^{4}(1 - 2\eta)^{2}p_{t}^{2}}\rceil$  suffices to guarantee that  $\| \mathbf{u} - \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]\| _2\leq$ $\frac{\kappa^2}{8}\| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]\| _2$  with probability at least  $1 - \delta$
It remains to bound  $\theta (\mathbf{w}_0,\mathbf{w}^*)$ , where  $\mathbf{w}_0 = \mathbf{u} / \| \mathbf{u}\| _2$ . Since  $\mathbf{w}^*$  is a unit vector, we have that with probability at least  $1 - \delta$
$$
\begin{array}{l} \cos \theta (\mathbf {w} _ {0}, \mathbf {w} ^ {*}) = \frac {\mathbf {u} \cdot \mathbf {w} ^ {*}}{\| \mathbf {u} \| _ {2}} \geq \frac {\left(\mathbf {u} - \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ]\right) \cdot \mathbf {w} ^ {*} + \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ] \cdot \mathbf {w} ^ {*}}{\| \mathbf {u} - \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ] \| _ {2} + \| \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ] \| _ {2}} \\ \geq \frac {(1 - \kappa^ {2} / 8) \| \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ] \| _ {2}}{(1 + \kappa^ {2} / 8) \| \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ y \mathbf {x} ] \| _ {2}} = 1 - \frac {\kappa^ {2} / 4}{1 + \kappa^ {2} / 8}, \\ \end{array}
$$