SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Input:  $\mathbf{w}_0, \hat{\hat{t}}, \hat{\gamma}, \eta, N_2$  i.i.d. samples  $(\mathbf{x}^{(i)}, y^{(i)})$  from  $\mathcal{D}$
2:  $\mu_0\gets \frac{(1 - 4\rho)\sqrt{2\pi}}{16(1 - 2\eta)};\rho \gets 0.00098;P(\hat{t},\hat{\gamma})\gets \mathbf{Pr}_{z\sim \mathcal{N}}[-\hat{t}\leq z\leq -\hat{t} +\hat{\gamma} ]$
3: for  $k = 0$  to  $K$  do
4: Let  $\mathcal{E}(\mathbf{w}_k,\hat{t})\coloneqq \{\mathbf{x}: - \hat{t}\leq \mathbf{w}_k\cdot \mathbf{x}\leq -\hat{t} +\hat{\gamma}\}$
5: Let  $\mathbf{g}(\mathbf{w}_k;\mathbf{x}^{(i)},y^{(i)}) = \frac{1}{2} ((1 - 2\eta)\mathrm{sign}(\mathbf{w}_k\cdot \mathbf{x}^{(i)} + \hat{t}) - y^{(i)})\mathrm{proj}_{\mathbf{w}_k^\perp}(\mathbf{x}^{(i)})$
6:  $\widehat{\mathbf{g}} (\mathbf{w}_k)\gets \frac{1}{N_2}\sum_{i = 1}^{N_2}\mathbf{g}(\mathbf{w}_k;\mathbf{x}^{(i)},y^{(i)})\frac{\mathbb{1}\{\mathbf{x}^{(i)}\in\mathcal{E}(\mathbf{w}_k,\hat{t})\}}{P(\hat{t},\hat{\gamma})}$
7:  $\mu_{k}\gets \mu_{k - 1}(1 - \rho)$
8:  $\mathbf{w}_{k + 1}\gets \frac{\mathbf{w}_k - \mu_k\widehat{\mathbf{g}}(\mathbf{w}_k)}{\|\mathbf{w}_k - \mu_k\widehat{\mathbf{g}}(\mathbf{w}_k)\|_2}$
9: end for
10: return  $\mathbf{w}_{K + 1}$
generality. For the former, it is by the simple symmetry of the standard normal distribution that the entire argument translates into the case  $t < 0$ , possibly by exchanging the meaning of  $+1$  and  $-1$  labels. For the latter, we note that when the bias is small, i.e., for  $p \leq \epsilon / (2(1 - 2\eta))$ , a constant hypothesis suffices.
2.1 Initialization Procedure
We begin this section with Lemma 2.1, which shows that given  $N_{1} = \widetilde{O}(d / (\kappa^{4}p^{2}(1 - 2\eta)^{2})\log(1 / \delta))$  i.i.d. samples from  $\mathcal{D}$ , we can construct a good initial point  $\mathbf{w}_0$  that forms an angle at most  $\kappa$  with the target weight vector  $\mathbf{w}^*$ . For our purposes,  $\kappa$  should be of the order  $1 / t$ . For  $t \leq \sqrt{2\log(1 / \epsilon')}$  where  $\epsilon' = \epsilon / (1 - 2\eta)$ , we can ensure that  $N_{1} = \widetilde{O}(d / (p^{2}(1 - 2\eta)^{2})\log(1 / \delta))$ . The downside of the lemma, however, is that the number of samples  $N_{1}$  requires at least approximate knowledge of the bias parameter  $p$  (or, more accurately, of  $e^{-t^2 / 2}$ ). We address this challenge by arguing (in Lemma 2.2) that we can estimate  $p$  using the procedure described in Algorithm 4, without increasing the total number of drawn samples by a factor larger than order-  $\log(1 / \epsilon')$ .
Lemma 2.1 (Initialization via Chow Parameters). Given  $\kappa > 0$ , define  $p_t = e^{-t^2 / 2}$ ,  $N_1 = O(d / (\kappa^4 p_t^2 (1 - 2\eta)^2) \log (1 / \delta))$  and let  $(\mathbf{x}^{(i)}, y^{(i)})$  for  $i \in [N_1]$  be i.i.d. samples drawn from  $\mathcal{D}$ . Let  $\mathbf{u} = \frac{1}{N_1} \sum_{i=1}^{N_1} \mathbf{x}^{(i)} y^{(i)}$  and  $\mathbf{w}_0 = \mathbf{u} / \| \mathbf{u} \|_2$ . Then, with probability  $1 - \delta$ , we have  $\theta(\mathbf{w}_0, \mathbf{w}^*) \leq \kappa$ .
We now leverage Lemma 2.1 to argue about the correctness of implementable Initialization procedure, stated as Algorithm 4 in Appendix B.1, where the proofs for this subsection can be found.
Lemma 2.2. Consider the Initialization procedure described by Algorithm 4 in Appendix B.1. If  $0 \leq t \leq \sqrt{2\log((1 - 2\eta) / \epsilon)}$ , then with probability at least  $1 - \delta$ ,  $\exp(-t^2 / 2) \leq \hat{p} \leq 4\exp(-t^2 / 2)$ . The algorithm draws a total of  $\widetilde{O}\left(\frac{d\log(1 / \delta)}{\max\{(1 - 2\eta)p, \epsilon\}^2}\right)$  samples and ensures that  $\theta(\mathbf{w}_0, \mathbf{w}^*) \leq \min\{\frac{1}{5t}, \frac{\pi}{2}\}$ .
2.2 Optimization
As discussed before, our Optimization procedure (Algorithm 2) can be seen as Riemannian subgradient descent on the unit sphere. Crucial to our analysis is the use of subgradient estimates from Line 5 and Line 6, where we condition on the event that the samples come from a thin band, defined in Line 4. Without this conditioning, the algorithm would correspond to projected subgradient descent of the LeakyReLU loss on the unit sphere. The conditioning effectively changes the landscape of the loss function being optimized, which cannot be argued anymore to even be convex, as the definition of the band depends on the weight vector  $\mathbf{w}$  at which the vector  $\widehat{\mathbf{g}}(\mathbf{w})$  is evaluated. Nevertheless, as we argue in this section, the optimization procedure can be carried out very efficiently, even exhibiting a linear convergence rate. To simplify the notation, in this section we denote the conditioned distribution  $\mathcal{D}|_{\mathcal{E}(\mathbf{w},\hat{t})}$  by  $\mathcal{D}(\mathbf{w},\hat{t})$ . We carry out the analysis assuming the estimate  $\hat{t}$  is within additive  $\epsilon^2$  of the true threshold value  $t$ ; as argued before, this has to be true for at least one estimate  $\hat{t}$  for which the Optimization procedure is invoked.
In the following lemma, we show that if the angle between a weight vector  $\mathbf{w}$  and the target vector  $\mathbf{w}^*$  is from a certain range, we can guarantee that  $\mathbf{g}(\mathbf{w})$  is sufficiently negatively correlated with  $\mathbf{w}^*$ . This condition is then used to argue about progress of our algorithm. The upper bound on  $\theta$  will hold initially, by our initialization procedure, and we will inductively argue that it holds for all iterations. The lower bound, when violated, will imply that the distance between  $\mathbf{w}$  and  $\mathbf{w}^*$  is small, in which case we would have converged to a sufficiently good solution  $\mathbf{w}$ .
Lemma 2.3. Fix any  $\epsilon' \in (0,1)$ . Suppose that  $0 \leq t \leq \sqrt{2\log(1/\epsilon')}$  and  $\mathbf{w} \in \mathbb{R}^d$  is such that  $\|\mathbf{w}\|_2 = 1$ , and  $\theta = \theta(\mathbf{w},\mathbf{w}^*)$  satisfies the inequality  $\epsilon' \exp(t^2/2) \leq \theta \leq 1/(5t)$ . If  $|\hat{t} - t| \leq \epsilon'^2/8$  and  $\hat{\gamma} = (1/2)\epsilon' \exp(\hat{t}^2/2)$ , then  $\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}(\mathbf{w},\hat{t})}[\mathbf{g}(\mathbf{w};\mathbf{x},y) \cdot \mathbf{w}^*] \leq -(1 - 2\eta) \sin \theta/(2\sqrt{2\pi})$ .
Since, by construction,  $\mathbf{g}(\mathbf{w})$  is orthogonal to  $\mathbf{w}$  (see Line 5 in Algorithm 2), we can bound the norm of the expected gradient vector by bounding  $\mathbf{g}(\mathbf{w})\cdot \mathbf{u}$  for some unit vectors  $\mathbf{u}$  that are orthogonal to  $\mathbf{w}$  using similar techniques as in Lemma 2.3. To be specific, we have the following lemma.
Lemma 2.4. Under the assumptions of Lemma 2.3,  $\left\| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}(\mathbf{w},\hat{t})}[\mathbf{g}(\mathbf{w};\mathbf{x},y)]\right\| _2\leq \frac{(1 - 2\eta)}{\sqrt{2\pi}}.$
The last technical ingredient that we need is the following lemma which shows a uniform bound on the difference between the empirical gradient  $\widehat{\mathbf{g}} (\mathbf{w})$  and its expectation (for more details, see Lemma B.6 and Corollary B.8 in Appendix B).
Lemma 2.5. Consider the learning problem from Definition 1.1. Let  $\epsilon', \hat{t}, \hat{\gamma}$  be parameters satisfying the conditions of Lemma 2.3. Let  $\delta \in (0,1)$ . Then using  $\widetilde{O}(d\log(1/\delta)/((1-2\eta)^2\epsilon'))$  samples to construct  $\widehat{\mathbf{g}}$ , for any unit vector  $\mathbf{w}$  such that  $\epsilon' \exp(t^2/2) \leq \theta(\mathbf{w}, \mathbf{w}^*) \leq 1/(5t)$ , it holds with probability at least  $1-\delta$ :  $\|\widehat{\mathbf{g}}(\mathbf{w}) - \mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}(\mathbf{w},\hat{t})}[\mathbf{g}(\mathbf{w})]\|_2 \leq (1/4)\|\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}(\mathbf{w},\hat{t})}[\mathbf{g}(\mathbf{w})]\|_2$ .
We are now ready to present and prove our main algorithm-related result. A short roadmap for our proof is as follows. Since Algorithm 1 constructs a grid with grid-width  $\epsilon^{\prime 2} / 8$  that covers all possible values of the true threshold  $t$ , there exists at least one guess  $\hat{t}$  that is  $\epsilon^{\prime 2}$ -close to the true threshold  $t$ . We first show that to get a halfspace with error at most  $\epsilon^{\prime}$ , it suffices to use this  $\hat{t}$  as the threshold and find a weight vector  $\mathbf{w}$  such that the angle  $\theta (\mathbf{w},\mathbf{w}^{*})$  is of the order  $\epsilon^{\prime}$ , which is exactly what Algorithm 2 does. The connection between  $\theta (\mathbf{w},\mathbf{w}^{*})$  and the error is conveyed by the inequality  $\operatorname{Pr}[\mathrm{sign}(\mathbf{w}\cdot \mathbf{x} + t)\neq \mathrm{sign}(\mathbf{w}^{*}\cdot \mathbf{x} + t)]\leq (\theta (\mathbf{w},\mathbf{w}^{*}) / \pi)\exp (-t^{2} / 2)$ ; see Appendix B. Let  $\mathbf{w}_k$  be the parameter generated by Algorithm 2 at iteration  $k$  for threshold  $\hat{t}$ . We show that  $\theta (\mathbf{w}_k,\mathbf{w}^*)$  converges to zero at a linear rate. To this end, we prove that under our carefully devised step size  $\mu_{k}$ , there exists an upper bound on  $\| \mathbf{w}_k - \mathbf{w}^*\| _2$ , which contracts at each iteration. Note that since both  $\mathbf{w}_k$  and  $\mathbf{w}^*$  are on the unit sphere, we have  $\| \mathbf{w}_k - \mathbf{w}^*\| _2 = 2\sin (\theta (\mathbf{w}_k,\mathbf{w}^*) / 2)$ . Essentially, this implies that Algorithm 2 produces a sequence of parameters  $\mathbf{w}_k$  such that  $\theta (\mathbf{w}_k,\mathbf{w}^*)$  converges to 0 linearly, under this threshold  $\hat{t}$ . Thus, we can conclude that there exists a halfspace among all halfspaces generated by Algorithm 1 that achieves  $\epsilon^\prime$  error with high probability.
Theorem 2.6. Consider the learning problem from Definition 1.1. Fix any unit vector  $\mathbf{w}_0\in \mathbb{R}^d$  such that  $\theta (\mathbf{w}_0,\mathbf{w}^*)\leq \min (1 / (5t),\pi /2)$ . Fix any  $\epsilon ,\delta >0$ . Let  $\hat{t} >0$  be a threshold such that  $|\hat{t} -t|\leq \epsilon^{2} / (8(1 - 2\eta)^{2})$ , and let  $\hat{\gamma} = \epsilon /(2(1 - 2\eta))\exp (\hat{t}^{2} / 2)$ . Then Algorithm 2 uses  $N_{2} = \widetilde{O}\big(d / ((1 - 2\eta)\epsilon)\log (1 / \delta)\big)$  samples from  $\mathcal{D}$ , has runtime  $\tilde{O} (N_2d)$ , and outputs a weight vector  $\mathbf{w}$  such that  $h(\mathbf{x}) = \mathrm{sign}(\mathbf{w}\cdot \mathbf{x} + \hat{t})$  satisfies  $\operatorname *{Pr}[h(\mathbf{x})\neq y]\leq \eta +\epsilon$  with probability at least  $1 - \delta$ .
Proof. Let  $\epsilon' = (\epsilon / 1 - 2\eta)$ , and denote by  $\mathbf{w}_k$  the vector produced by the algorithm at  $k^{\text{th}}$  iteration for threshold  $\hat{t}$ . For any unit vector  $\mathbf{w}$  and  $|\hat{t} - t| \leq \epsilon'^2 / 8$ , it holds  $\mathbf{Pr}[\mathrm{sign}(\mathbf{w} \cdot \mathbf{x} + \hat{t}) \neq \mathrm{sign}(\mathbf{w}^* \cdot \mathbf{x} + t)] \leq \epsilon'^2 / (4\sqrt{2\pi}) + \theta(\mathbf{w}, \mathbf{w}^*) / \pi \exp(-t^2 / 2)$  (see Appendix B.2 for more details). Therefore, it suffices to find a parameter  $\mathbf{w}$  such that  $\theta(\mathbf{w}, \mathbf{w}^*) \leq \pi \epsilon' \exp(t^2 / 2)$ . Note that since both  $\mathbf{w}$  and  $\mathbf{w}^*$  are unit vectors, we have  $\|\mathbf{w} - \mathbf{w}^*\|_2 = 2\sin(\theta / 2)$ , indicating that it suffices to minimize  $\|\mathbf{w} - \mathbf{w}^*\|_2$  efficiently. As proved in Section 2.1, we can start with an initial vector  $\mathbf{w}_0$  such that  $\theta(\mathbf{w}_0, \mathbf{w}^*) \leq 1 / (5t)$  by calling Algorithm 4 (in Appendix B.1). Denote  $\theta_k = \theta(\mathbf{w}_k, \mathbf{w}^*)$  and consider the case when  $\theta_k \geq \epsilon' \exp(t^2 / 2)$ . We establish the following claim:
Claim 2.7. Let  $C_1 \coloneqq (1 - 2\eta) / \sqrt{2\pi}$ . Drawing  $N_2 = \widetilde{O}(d\log(1/\delta) / ((1 - 2\eta)^2\epsilon'))$  samples from distribution  $\mathcal{D}$ , we have that if  $\theta_k \geq \epsilon' \exp(t^2/2)$  then with probability at least  $1 - \delta$ :  $\| \mathbf{w}_{k+1} - \mathbf{w}^* \|_2^2 \leq \| \mathbf{w} - \mathbf{w}^* \|_2^2 - (C_1/2)\mu_k \sin \theta_k + 4C_1^2\mu_k^2$ .
It remains to choose the step size  $\mu_{k}$  properly to get linear convergence. By carefully designing a shrinking step size, we are able to construct an upper bound  $\phi_{k}$  on the distance of  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k\| _2$  using Claim 2.7. Importantly, by exploiting the property that both  $\mathbf{w}$  and  $\mathbf{w}^*$  are on the unit sphere, we show that the upper bound is contracting at each step, even though the distance  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k\| _2$  could be increasing. Concretely, we have the following lemma.
Lemma 2.8. Let  $\rho = 0.00098$  and  $\phi_k = (1 - \rho)^k$ . Then, setting  $\mu_k = (1 - 4\rho)\phi_k / (16C_1)$  it holds  $\sin (\theta_k / 2) \leq \phi_k$  for  $k = 1, \dots, K$ .
Proof. Let  $\phi_k = (1 - \rho)^k$  where  $\rho = 0.00098$ . This choice of  $\rho$  ensures that  $32\rho^2 + 1020\rho - 1 \leq 0$ . We show by induction that choosing  $\mu_k = (1 - 4\rho)\phi_k / (16C_1) = (1 - \rho)^k (1 - 4\rho) / (16C_1)$ , it holds  $\sin(\theta_k/2) \leq \phi_k$ . The condition certainly holds for  $k = 1$  since  $\theta_1 \in [0, \pi/2]$ . Now suppose that  $\sin(\theta_k/2) \leq \phi_k$  for some  $k \geq 1$ . We discuss the following 2 cases:  $\phi_k \geq \sin(\theta_k/2) \geq \frac{3}{4}\phi_k$  and  $\sin(\theta_k/2) \leq \frac{3}{4}\phi_k$ . First, suppose  $\phi_k \geq \sin(\theta_k/2) \geq \frac{3}{4}\phi_k$ . Since  $\sin(\theta_k/2) \leq \sin\theta_k$ , it also holds  $\sin\theta_k \geq \frac{3}{4}\phi_k$ . Bringing in the fact that  $\| \mathbf{w}_{k+1} - \mathbf{w}^* \|_2 = 2\sin(\theta_{k+1}/2)$  and  $\| \mathbf{w}_k - \mathbf{w}^* \|_2 = 2\sin(\theta_k/2)$ , as well as the definition of  $\mu_k$ , the conclusion of Claim 2.7 becomes:
$$
\begin{array}{l} (2 \sin (\theta_ {k + 1} / 2)) ^ {2} \leq (2 \sin (\theta_ {k} / 2)) ^ {2} - (C _ {1} / 2) \mu_ {k} \sin \theta_ {k} + 4 C _ {1} ^ {2} (1 - 4 \rho) \phi_ {k} \mu_ {k} / (1 6 C _ {1}) \\ \leq 4 \phi_ {k} ^ {2} - 3 C _ {1} \mu_ {k} \phi_ {k} / 8 + C _ {1} (1 - 4 \rho) \mu_ {k} \phi_ {k} / 4 = 4 \phi_ {k} ^ {2} (1 - (1 + 8 \rho) (1 - 4 \rho) / 5 1 2), \\ \end{array}
$$
where in the second inequality we used  $\sin \theta_{k} \geq \frac{3}{4}\phi_{k}$  and in the last equality we used the definition of  $\mu_{k}$  by which  $\mu_{k} = (1 - 4\rho)\phi_{k} / (16C_{1})$ . Since  $\rho$  is chosen so that  $32\rho^{2} + 1020\rho - 1 \leq 0$ , we have:
$$
\sin \left(\theta_ {k + 1} / 2\right) \leq \phi_ {k} \sqrt {1 - (1 + 8 \rho) (1 - 4 \rho) / 5 1 2} \leq (1 - \rho) \phi_ {k} = (1 - \rho) ^ {k + 1},
$$
as desired. Next, consider  $\sin (\theta_k / 2) \leq (3 / 4)\phi_k$ . Recall that  $\mathbf{w}_{k + 1} = \mathrm{proj}_{\mathbb{B}}(\mathbf{w}_k - \mu_k\widehat{\mathbf{g}} (\mathbf{w}_k))$  and  $\mathbf{w}_k \in \mathbb{B}$ , where  $\mathbb{B}$  is the unit ball<sup>2</sup>; therefore,  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k\| _2 \leq \| \mathbf{w}_k - \mu_k\widehat{\mathbf{g}} (\mathbf{w}) - \mathbf{w}_k\| _2 = \mu_k\| \widehat{\mathbf{g}} (\mathbf{w}_k)\| _2$  by the non-expansiveness of the projection operator. Furthermore, applying Lemma 2.5 and Lemma 2.4, it holds that  $\| \widehat{\mathbf{g}} (\mathbf{w}_k)\| _2 \leq (5 / 4)\| \mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}(\mathbf{w},t)}[\mathbf{g}(\mathbf{w})]\| _2 \leq 2(1 - 2\eta) / \sqrt{2\pi}$ , i.e., we have  $\| \widehat{\mathbf{g}} (\mathbf{w}_k)\| _2 \leq 2C_1$ ; therefore,  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k\| _2 \leq 2\mu_kC_1$ , which indicates that:
$$
2 \left(\sin \left(\theta_ {k + 1} / 2\right) - \sin \left(\theta_ {k} / 2\right)\right) = \left\| \mathbf {w} _ {k + 1} - \mathbf {w} ^ {*} \right\| _ {2} - \left\| \mathbf {w} _ {k} - \mathbf {w} ^ {*} \right\| _ {2} \leq \left\| \mathbf {w} _ {k + 1} - \mathbf {w} _ {k} \right\| _ {2} \leq 2 \mu_ {k} C _ {1}.
$$
Since we have assumed  $\sin (\theta_k / 2)\leq (3 / 4)\phi_k$  , then it holds:
$\phi_{k + 1} - \sin (\theta_{k + 1} / 2)\geq (1 - \rho)\phi_k - \phi_k + \phi_k - \sin (\theta_k / 2) - (1 - 4\rho)\phi_k / 16\geq 3(1 - 4\rho)\phi_k / 16 > 0,$  since we have chosen  $\mu_{k} = (1 - 4\rho)\phi_{k} / (16C_{1})$  . Hence, it also holds that  $\sin (\theta_{k + 1} / 2)\leq \phi_{k + 1}$
Lemma 2.8 shows that  $\sin (\theta_k / 2)$  converges to 0 linearly. Therefore, using  $N_{2} = \widetilde{O}(d\log(1/\delta) / ((1 - 2\eta)^{2}\epsilon'))$  samples, after  $K = O((1/\rho)\log(1 / (\exp(t^{2}/2)\epsilon')) = O(\log(1/\epsilon'))$  iterations, we get a  $\mathbf{w}_K$  such that  $\theta_K \leq 2\sin(\theta_K / 2) \leq \epsilon' \exp(t^{2}/2)$ . Let  $h(\mathbf{x}) \coloneqq \mathrm{sign}(\mathbf{w}_K \cdot \mathbf{x} + \hat{t})$ . Then it holds that the disagreement of  $h(\mathbf{x})$  and  $f(\mathbf{x})$  is bounded by  $\mathbf{Pr}[h(\mathbf{x}) \neq f(\mathbf{x})] \leq \epsilon'$  (see Appendix B for more details). Finally, since  $\mathrm{err}_{0-1}^{\mathcal{D}}(h) = \mathbf{Pr}_{(\mathbf{x},y) \sim \mathcal{D}}[h(\mathbf{x}) \neq y] = \eta + (1 - 2\eta)\mathbf{Pr}_{\mathbf{x} \sim \mathcal{D}_{\mathbf{x}}}[h(\mathbf{x}) \neq \mathrm{sign}(\mathbf{w}^{*} \cdot \mathbf{x} + t)]$ , for any  $h: \mathbb{R}^d \mapsto \{\pm 1\}$ , to get misclassification error at most  $\eta + \epsilon$  (with respect to the  $y$ ), it suffices to use  $\epsilon' = \epsilon / (1 - 2\eta)$ . Therefore, we get  $\mathbf{Pr}_{(\mathbf{x},y) \sim \mathcal{D}}[\mathrm{sign}(\mathbf{w}_K \cdot \mathbf{x} + \hat{t}) \neq y] \leq \eta + \epsilon$ , using  $N_2 = \widetilde{O}(d\log(1/\delta) / ((1 - 2\eta)\epsilon))$  samples. Since the algorithm runs for  $O(\log(1/\epsilon))$  iterations, the overall runtime is  $\widetilde{O}(N_2d)$ . This completes the proof of Theorem 2.6.
Proof Sketch of Theorem 1.3. From Lemma 2.2, we get that with  $\widetilde{O}(d / ((1 - 2\eta)^2 p^2))$  samples our Initialization procedure (Algorithm 4) produces a unit vector  $\mathbf{w}_0$  so that  $\theta(\mathbf{w}_0, \mathbf{w}^*) \leq \min(1/(5t), \pi/2)$  with high probability. We construct a grid of  $(\epsilon^2 / (8(1 - 2\eta)^2)$ -separated values, containing all the possible values of the threshold  $t$  of size roughly  $\sim 1/\epsilon^2$ . We run Algorithm 2 for each possible choice of the threshold  $t$ . Conditioned on the choice of  $\hat{t}$  and  $\mathbf{w}_0$  that satisfies the assumptions of Theorem 2.6, Algorithm 2 outputs a weight vector  $\widehat{\mathbf{w}}$  so that  $\mathbf{Pr}_{(\mathbf{x}, y) \sim \mathcal{D}}[\mathrm{sign}(\widehat{\mathbf{w}} \cdot \mathbf{x} + \hat{t}) \neq y] \leq \eta + \epsilon$ . Using standard concentration facts, we have that with a sample size of order  $\widetilde{O}(d / ((1 - 2\eta)\epsilon))$  from  $\mathcal{D}$ , we can output the hypothesis with the minimum empirical error with high probability.
This is true because  $\widehat{\mathbf{g}} (\mathbf{w}_k)$  is orthogonal to  $\mathbf{w}_k$ , and thus  $\| \mathbf{w}_k - \mu_k\widehat{\mathbf{g}} (\mathbf{w}_k)\| _2 > 1$ , meaning that projections onto the unit ball and the unit sphere are the same in this case.
3 SQ Lower Bound for Learning Gaussian Halfspaces with RCN
To state our SQ lower bound theorem, we require the following standard definition.
Definition 3.1 (Decision/Testing Problem over Distributions). Let  $D$  be a distribution and  $\mathfrak{D}$  be a family of distributions over  $\mathbb{R}^d$ . We denote by  $\mathcal{B}(\mathfrak{D},D)$  the decision (or hypothesis testing) problem in which the input distribution  $D^{\prime}$  is promised to satisfy either (a)  $D^{\prime} = D$  or (b)  $D^{\prime}\in \mathfrak{D}$ , and the goal of the algorithm is to distinguish between these two cases.
Theorem 3.2 (SQ Lower Bound for Testing RCN Halfspaces). Fix  $c \in (0,1/2)$  and let  $d \in \mathbb{N}$  be sufficiently large. For any  $p \geq 2^{-O(d^c)}$ , any SQ algorithm that learns the class of (at most)  $p$ -biased Gaussian halfspaces on  $\mathbb{R}^d$  in the presence of RCN with  $\eta = 1/3$  to error less than  $\eta + p/3$  either requires queries to VSTAT  $(\widetilde{O}(d^{1/2 - c}/p^2))$ , or needs to make at least  $2^{\Omega(d^c)}$  statistical queries.
We note that our SQ lower bound applies to a natural testing version of our learning problem. By a standard reduction (see Lemma C.9), it follows that any learning algorithm for the problem requires either  $2^{\Omega(d^{c})}$  many queries or at least one query to VSTAT  $(\widetilde{O}(d^{2c - 1/2} / p^{2}))$ . We also note that the established bound is tight for the corresponding testing problem (see Appendix C.6).
Proof of Theorem 3.2. For any unit vector  $\mathbf{v} \in \mathbb{R}^d$ , we define the LTF  $f_{\mathbf{v}}(\mathbf{x}) = \mathrm{sign}(\mathbf{v} \cdot \mathbf{x} - t)$  where  $t > 0$  and denote  $p = \mathbf{Pr}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x}) = 1]$ . Let  $D_{\mathbf{v}}$  be the distribution on  $(\mathbf{x}, y)$  with respect to  $f_{\mathbf{v}}$  with the random variable  $y$  supported on  $\{\pm 1\}$  as follows:  $\mathbf{Pr}[y = f_{\mathbf{v}}(\mathbf{x}) \mid \mathbf{x}] = 1 - \eta$  and  $\mathbf{x}$  is distributed as standard normal. Denote by  $A_{\mathbf{v}}$  the distribution  $D_{\mathbf{v}}$  conditioned on  $y = 1$  and by  $B_{\mathbf{v}}$  the distribution  $D_{\mathbf{v}}$  conditioned on  $y = -1$ . It is easy to see that
$$
A _ {\mathbf {v}} (\mathbf {x}) = G (\mathbf {x}) (\eta + (1 - 2 \eta) \mathbb {1} \left\{f _ {\mathbf {v}} (\mathbf {x}) > 0 \right\}) / (\eta + (1 - 2 \eta) p)
$$
$$
B _ {\mathbf {v}} (\mathbf {x}) = G (\mathbf {x}) (1 - \eta - (1 - 2 \eta) \mathbb {1} \left\{f _ {\mathbf {v}} (\mathbf {x}) > 0 \right\}) / (1 - \eta - (1 - 2 \eta) p).
$$
Fix unit vectors  $\mathbf{v}$ ,  $\mathbf{u} \in \mathbb{R}^d$  and let  $\theta$  be the angle between them. We bound from above the correlation between  $f_{\mathbf{v}}(\mathbf{x})$  and  $f_{\mathbf{u}}(\mathbf{x})$ . Our main technical lemma is the following:
Lemma 3.3. Let  $f_{\mathbf{v}}(\mathbf{x})$  and  $f_{\mathbf{u}}(\mathbf{x})$  defined as above. Then it holds
$$
\left| \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ f _ {\mathbf {v}} (\mathbf {x}) f _ {\mathbf {u}} (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ f _ {\mathbf {v}} (\mathbf {x}) ] \mathbf {E} _ {\mathbf {x} \sim \mathcal {N}} [ f _ {\mathbf {u}} (\mathbf {x}) ] \right| \leq 4 | \cot (\theta) | \exp (- t ^ {2}) \exp \left(| \cos (\theta) | t ^ {2}\right).
$$
Proof of Lemma 3.3. We start by calculating the Hermite coefficients of the univariate function  $\mathrm{sign}(z - t)$ . We will use the fact that  $\mathbf{E}_{z\sim \mathcal{N}}[\mathrm{sign}(z - t)\mathrm{He}_i(z)] = 2i^{-1 / 2}\mathrm{He}_{i - 1}(t)\exp (-t^2 /2)$  (see Claim C.7). Let  $c_{i}$  be the Hermite coefficient of degree  $i$ . Without loss of generality (due to the rotational invariance of the Gaussian distribution), we can assume that  $\mathbf{v} = \mathbf{e}_1$  and  $\mathbf{u} = \cos \theta \mathbf{e}_1 + \sin \theta \mathbf{e}_2$ . Using standard algebraic manipulations and orthogonality arguments (see Claim C.5 for more details), we have that
$$
\underset {\mathbf {x} \sim \mathcal {N}} {\mathbf {E}} [ f _ {\mathbf {v}} (\mathbf {x}) f _ {\mathbf {u}} (\mathbf {x}) ] = \underset {\mathbf {x} _ {1}, \mathbf {x} _ {2} \sim \mathcal {N}} {\mathbf {E}} [ \mathrm {s i g n} (\mathbf {x} _ {1} - t) \mathrm {s i g n} (\cos \theta \mathbf {x} _ {1} + \sin \theta \mathbf {x} _ {2} - t) ] = \sum_ {i \geq 0} \cos^ {i} \theta c _ {i} ^ {2}.
$$
Note that  $\mathrm{He}_0(\mathbf{x}) = 1$ , therefore  $c_{0} = \mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x})]$ . Therefore, we have that
$$
\underset {\mathbf {x} \sim \mathcal {N}} {\mathbf {E}} [ f _ {\mathbf {v}} (\mathbf {x}) f _ {\mathbf {u}} (\mathbf {x}) ] = \sum_ {i \geq 1} \cos^ {i} \theta c _ {i} ^ {2} + \underset {\mathbf {x} \sim \mathcal {N}} {\mathbf {E}} [ f _ {\mathbf {v}} (\mathbf {x}) ] \underset {\mathbf {x} \sim \mathcal {N}} {\mathbf {E}} [ f _ {\mathbf {u}} (\mathbf {x}) ].
$$
Let  $J = \sum_{i\geq 1}\cos^i\theta c_i^2$ . To complete the proof, it remains to bound the  $|J|$ . We show the following: