SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr [ \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) \neq \operatorname {s i g n} (\mathbf {w} ^ {*} \cdot \mathbf {x} + t) ] \\ = \Pr [ \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t), \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t) = \operatorname {s i g n} \left(\mathbf {w} ^ {*} \cdot \mathbf {x} + t\right) ] \\ + \Pr [ \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) = \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t), \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t) \neq \operatorname {s i g n} \left(\mathbf {w} ^ {*} \cdot \mathbf {x} + t\right) ] \\ \leq \Pr [ \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t) ] + \Pr [ \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t) \neq \operatorname {s i g n} (\mathbf {w} ^ {*} \cdot \mathbf {x} + t) ]. \\ \end{array}
$$
Since  $|\hat{t} - t| \leq \epsilon'^2 / 8$ , it holds
$$
\begin{array}{l} \mathbf {P r} [ \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) \neq \operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + t) ] = \mathbf {P r} [ - \max  \{t, \hat {t} \} \leq \mathbf {w} \cdot \mathbf {x} \leq - \min  \{t, \hat {t} \} ] \\ \leq \frac {2 | \hat {t} - t |}{\sqrt {2 \pi}} \leq \frac {\epsilon^ {\prime 2}}{4 \sqrt {2 \pi}}. \\ \end{array}
$$
In addition, as shown in Fact B.9,  $\mathbf{Pr}[\mathrm{sign}(\mathbf{w}\cdot \mathbf{x} + t)\neq \mathrm{sign}(\mathbf{w}^{*}\cdot \mathbf{x} + t)]\leq \frac{\theta(\mathbf{w},\mathbf{w}^{*})}{\pi}\exp (-t^2 /2);$  thus,
$$
\Pr \left[ \operatorname {s i g n} \left(\mathbf {w} \cdot \mathbf {x} + \hat {t}\right) \neq \operatorname {s i g n} \left(\mathbf {w} ^ {*} \cdot \mathbf {x} + t\right) \right] \leq \frac {\epsilon^ {\prime 2}}{4 \sqrt {2 \pi}} + \frac {\theta (\mathbf {w} , \mathbf {w} ^ {*})}{\pi} \exp \left(- t ^ {2} / 2\right). \tag {13}
$$
Therefore, it suffices to find a parameter  $\mathbf{w}$  such that  $\theta (\mathbf{w},\mathbf{w}^{*})\leq \pi \epsilon^{\prime}\exp (t^{2} / 2)$ . Note that since both  $\mathbf{w}$  and  $\mathbf{w}^*$  are unit vectors, we have  $\| \mathbf{w} - \mathbf{w}^{*}\|_{2} = 2\sin (\theta /2)$ , indicating that it suffices to minimize  $\| \mathbf{w} - \mathbf{w}^{*}\|_{2}$  efficiently. As proved in Lemma 2.1 and Lemma 2.2, we can start with an initial vector  $\mathbf{w}_0$  such that  $\theta (\mathbf{w}_0,\mathbf{w}^*)\leq 1 / (5t)$  by calling Algorithm 4. Starting from this  $\mathbf{w}_0$ , we show that  $\| \mathbf{w}_k - \mathbf{w}^*\| _2$  contracts linearly whenever the angle between  $\mathbf{w}_k$  and  $\mathbf{w}^*$  is larger than  $\epsilon^\prime \exp (t^2 /2)$ , thus we reach the required upper bound for this angle within a logarithmic number of steps. Denote  $\theta_{k} = \theta (\mathbf{w}_{k},\mathbf{w}^{*})$  and consider the case when  $\theta_{k}\geq \epsilon^{\prime}\exp (t^{2} / 2)$ .
Proof. Observe first that since  $\widehat{\mathbf{g}}(\mathbf{w}_k)$  is orthogonal to  $\mathbf{w}_k$ , we have  $\|\mathbf{w}_k - \mu_k \widehat{\mathbf{g}}(\mathbf{w}_k)\|_2^2 = \|\mathbf{w}_k\|_2^2 + \mu_k^2 \|\widehat{\mathbf{g}}(\mathbf{w}_k)\|_2^2 \geq 1$ , thus normalizing  $\mathbf{w}_k - \mu \widehat{\mathbf{g}}(\mathbf{w}_k)$  is equivalent to projecting  $\mathbf{w}_k - \mu_k \widehat{\mathbf{g}}(\mathbf{w}_k)$  to the unit ball  $\mathbb{B}$ . Since we have assumed  $\mathbf{w}^* \in \mathbb{B}$ , by the non-expansiveness of the projection operator and  $\mathrm{proj}_{\mathbb{B}}(\mathbf{w}^*) = \mathbf{w}^*$ , we have:
$$
\begin{array}{l} \left\| \mathbf {w} _ {k + 1} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} = \left\| \frac {\mathbf {w} _ {k} - \mu_ {k} \widehat {\mathbf {g}} (\mathbf {w} _ {k})}{\| \mathbf {w} _ {k} - \mu_ {k} \widehat {\mathbf {g}} (\mathbf {w} _ {k}) \| _ {2}} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} = \left\| \operatorname {p r o j} _ {\mathbb {B}} \left(\mathbf {w} _ {k} - \mu_ {k} \widehat {\mathbf {g}} (\mathbf {w} _ {k})\right) - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} \\ \leq \left\| \mathbf {w} _ {k} - \mu_ {k} \widehat {\mathbf {g}} \left(\mathbf {w} _ {k}\right) - \mathbf {w} ^ {*} \right\| _ {2} ^ {2}. \\ \end{array}
$$
Thus, expanding the squared norm on the right-hand side yields:
$$
\begin{array}{l} \left\| \mathbf {w} _ {k + 1} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} \leq \left\| \mathbf {w} _ {k} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} - 2 \mu_ {k} \widehat {\mathbf {g}} (\mathbf {w} _ {k}) \cdot \left(\mathbf {w} _ {k} - \mathbf {w} ^ {*}\right) + \mu_ {k} ^ {2} \| \widehat {\mathbf {g}} (\mathbf {w} _ {k}) \| _ {2} ^ {2} \\ = \left\| \mathbf {w} _ {k} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} + 2 \mu_ {k} \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w} _ {k}, \hat {\iota})} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right) \cdot \mathbf {w} ^ {*} \right] \tag {14} \\ + 2 \mu_ {k} (\widehat {\mathbf {g}} (\mathbf {w} _ {k}) - \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w} _ {k}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w} _ {k}) ]) \cdot \mathbf {w} ^ {*} + \mu_ {k} ^ {2} \| \widehat {\mathbf {g}} (\mathbf {w} _ {k}) \| _ {2} ^ {2} \\ \end{array}
$$
where in the first equality we used the fact that  $\widehat{\mathbf{g}} (\mathbf{w}_k)$  and  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}(\mathbf{w}_k,\hat{t})}[\mathbf{g}(\mathbf{w}_k)]$  are both orthogonal to  $\mathbf{w}_k$ . Without loss of generality (because of the rotational invariance), assume  $\mathbf{w}_k = \mathbf{e}_1$  and
$\mathbf{w}^{*} = \cos \theta \mathbf{e}_{1} + \sin \theta \mathbf{e}_{2}$ . Then, again by the fact that both  $\widehat{\mathbf{g}} (\mathbf{w}_k)$  and  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}(\mathbf{w}_k,\hat{t})}[\mathbf{g}(\mathbf{w}_k)]$  are orthogonal to  $\mathbf{w}_k$ , we have
$$
\begin{array}{l} \left(\widehat {\mathbf {g}} \left(\mathbf {w} _ {k}\right) - \underset {(\mathbf {x}, y) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right)\right]\right) \cdot \mathbf {w} ^ {*} = \sin \theta_ {k} \left(\widehat {\mathbf {g}} \left(\mathbf {w} _ {k}\right) - \underset {(\mathbf {x}, y) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right)\right]\right)\left. \right. \cdot \mathbf {e} _ {2} \\ \leq \sin \theta_ {k} \left\| \widehat {\mathbf {g}} (\mathbf {w} _ {k}) - \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w} _ {k}, \hat {t})} {\mathbf {E}} \left[ \mathbf {g} (\mathbf {w} _ {k}) \right] \right\| _ {2}. \\ \end{array}
$$
Thus, further invoking Lemma 2.3, we have:
$$
\begin{array}{l} \left\| \mathbf {w} _ {k + 1} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} \leq \left\| \mathbf {w} _ {k} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} - 2 \mu_ {k} \frac {(1 - 2 \eta) \sin \theta_ {k}}{2 \sqrt {2 \pi}} \\ + 2 \mu_ {k} \sin \theta_ {k} \| \widehat {\mathbf {g}} (\mathbf {w} _ {k}) - \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w} _ {k}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w} _ {k}) ] \| _ {2} + \mu_ {k} ^ {2} \| \widehat {\mathbf {g}} (\mathbf {w} _ {k}) \| _ {2} ^ {2}. \tag {15} \\ \end{array}
$$
Corollary B.8 (or Lemma 2.5) implies that with  $N_{2} = \widetilde{O}(d\log(1/\delta)/((1 - 2\eta)^{2}\epsilon'))$  samples in total, for any unit vector  $\mathbf{w}_{k}$  satisfying  $\epsilon' \exp(t^{2}/2) \leq \theta_{k} \leq 1/(5t)$  with probability at least  $1 - \delta$ , it holds:
$$
\left\| \widehat {\mathbf {g}} \left(\mathbf {w} _ {k}\right) - \underset {(\mathbf {x}, y) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right) \right] \right\| _ {2} \leq \frac {1}{4} \| \underset {(\mathbf {x}, y) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right) \right] \| _ {2}. \tag {16}
$$
Recall that we have shown in the proof of Lemma 2.4 that  $\| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}(\mathbf{w}_k,\hat{t})}[\mathbf{g}(\mathbf{w}_k)]\| _2\leq \frac{1 - 2\eta}{\sqrt{2\pi}}$  therefore, Equation (16) further gives that with probability at least  $1 - \delta$ :
$$
\begin{array}{l} \left\| \widehat {\mathbf {g}} \left(\mathbf {w} _ {k}\right) \right\| _ {2} \leq \left\| \widehat {\mathbf {g}} \left(\mathbf {w} _ {k}\right) - \underset {(\mathbf {x}, y) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right) \right] \right\| _ {2} + \left\| \underset {(\mathbf {x}, y) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} {\mathbf {E}} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right) \right] \right\| _ {2} \\ \leq \frac {5}{4} \left\| _ {\left(\mathbf {x}, y\right) \sim \mathcal {D} \left(\mathbf {w} _ {k}, \hat {t}\right)} \left[ \mathbf {g} \left(\mathbf {w} _ {k}\right) \right] \right\| _ {2} \leq \frac {2 (1 - 2 \eta)}{\sqrt {2 \pi}}. \tag {17} \\ \end{array}
$$
Thus, plugging Equation (16) and Equation (17) back into Equation (15), we get that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \left\| \mathbf {w} _ {k + 1} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} \leq \left\| \mathbf {w} _ {k} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} - 2 \mu_ {k} \frac {1 - 2 \eta}{2 \sqrt {2 \pi}} \sin \theta_ {k} + 2 \mu_ {k} \frac {1 - 2 \eta}{4 \sqrt {2 \pi}} \sin \theta_ {k} + \mu_ {k} ^ {2} \frac {2 (1 - 2 \eta) ^ {2}}{\pi} \\ \leq \left\| \mathbf {w} _ {k} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} - \mu_ {k} \frac {1 - 2 \eta}{2 \sqrt {2 \pi}} \sin \theta_ {k} + \mu_ {k} ^ {2} \frac {2 (1 - 2 \eta) ^ {2}}{\pi}. \tag {18} \\ \end{array}
$$
Let  $C_1 \coloneqq \frac{1 - 2\eta}{\sqrt{2\pi}}$ . Then Equation (18) is simplified to:
$$
\left\| \mathbf {w} _ {k + 1} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} \leq \left\| \mathbf {w} - \mathbf {w} ^ {*} \right\| _ {2} ^ {2} - \frac {C _ {1}}{2} \mu_ {k} \sin \theta_ {k} + 4 C _ {1} ^ {2} \mu_ {k} ^ {2}, \tag {19}
$$
completing the proof of this claim.
It remains to choose the step size  $\mu_{k}$  properly to get linear convergence. By carefully designing a shrinking step size, we are able to construct an upper-bound  $\phi_{k}$  on the distance of  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k\| _2$  using Claim 2.7. Importantly, by exploiting the property that both  $\mathbf{w}$  and  $\mathbf{w}^*$  are on the unit sphere, we show that the upper bound is contracting at each step, even though the distance  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k\| _2$  could be increasing. Concretely, we have the following claim.
Proof. Let  $\phi_k = (1 - \rho)^k$  where  $\rho = 0.00098$ . This choice of  $\rho$  ensures that  $32\rho^2 + 1020\rho - 1 \leq 0$ . We show by induction that choosing  $\mu_k = (1 - 4\rho)\phi_k / (16C_1) = (1 - \rho)^k (1 - 4\rho) / (16C_1)$ , it holds  $\sin (\theta_k / 2) \leq \phi_k$ . The condition certainly holds for  $k = 1$  since  $\theta_1 \in [0, \pi / 2]$ . Now suppose that  $\sin (\theta_k / 2) \leq \phi_k$  for some  $k \geq 1$ . We discuss the following 2 cases:  $\phi_k \geq \sin (\theta_k / 2) \geq \frac{3}{4}\phi_k$  and  $\sin (\theta_k / 2) \leq \frac{3}{4}\phi_k$ .
First, suppose  $\phi_k \geq \sin(\theta_k/2) \geq \frac{3}{4}\phi_k$ . Since  $\sin(\theta_k/2) \leq \sin \theta_k$ , it also holds  $\sin \theta_k \geq \frac{3}{4}\phi_k$ . Bringing in the fact that  $\| \mathbf{w}_{k+1} - \mathbf{w}^* \|_2 = 2\sin(\theta_{k+1}/2)$  and  $\| \mathbf{w}_k - \mathbf{w}^* \|_2 = 2\sin(\theta_k/2)$ , as well as the definition of  $\mu_k$ , Equation (19) becomes:
$$
\begin{array}{l} (2 \sin (\theta_ {k + 1} / 2)) ^ {2} \leq (2 \sin (\theta_ {k} / 2)) ^ {2} - \frac {C _ {1}}{2} \mu_ {k} \sin \theta_ {k} + 4 C _ {1} ^ {2} \frac {(1 - 4 \rho)}{1 6 C _ {1}} \phi_ {k} \mu_ {k} \\ \leq 4 \phi_ {k} ^ {2} - \frac {C _ {1}}{2} \mu_ {k} \frac {3}{4} \phi_ {k} + \frac {C _ {1} (1 - 4 \rho)}{4} \mu_ {k} \phi_ {k} \\ = 4 \phi_ {k} ^ {2} - \frac {C _ {1}}{4} \left(\frac {1}{2} + 4 \rho\right) \frac {1 - 4 \rho}{1 6 C _ {1}} \phi_ {k} ^ {2} \\ = 4 \phi_ {k} ^ {2} \left(1 - \frac {(1 + 8 \rho) (1 - 4 \rho)}{5 1 2}\right), \\ \end{array}
$$
where in the second line we used  $\sin \theta_{k} \geq \frac{3}{4}\phi_{k}$  and in the third line we used the definition of  $\mu_{k}$  by which  $\mu_{k} = (1 - 4\rho)\phi_{k} / (16C_{1})$ . Since  $\rho$  is chosen so that  $32\rho^{2} + 1020\rho - 1 \leq 0$ , we have:
$$
\begin{array}{l} \sin (\theta_ {k + 1} / 2) \leq \phi_ {k} \sqrt {1 - \frac {(1 + 8 \rho) (1 - 4 \rho)}{5 1 2}} \\ \leq \phi_ {k} \left(1 - \frac {(1 + 8 \rho) (1 - 4 \rho)}{1 0 2 4}\right) \leq (1 - \rho) \phi_ {k} = (1 - \rho) ^ {k + 1}, \\ \end{array}
$$
Next, consider  $\sin (\theta_k / 2) \leq \frac{3}{4} \phi_k$ . Recall that  $\mathbf{w}_{k + 1} = \mathrm{proj}_{\mathbb{B}}(\mathbf{w}_k - \mu_k \widehat{\mathbf{g}}(\mathbf{w}_k))$  and  $\mathbf{w}_k \in \mathbb{B}$ ; therefore,  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k \|_2 \leq \| \mathbf{w}_k - \mu_k \widehat{\mathbf{g}}(\mathbf{w}) - \mathbf{w}_k \|_2 = \mu_k \| \widehat{\mathbf{g}}(\mathbf{w}_k) \|_2$  by the non-expansiveness of the projection operator. Furthermore, applying Equation (17), we have  $\| \widehat{\mathbf{g}}(\mathbf{w}_k) \|_2 \leq 2C_1$ ; therefore,  $\| \mathbf{w}_{k + 1} - \mathbf{w}_k \|_2 \leq 2\mu_k C_1$ , which indicates that:
Since we have assumed  $\sin (\theta_k / 2)\leq \frac{3}{4}\phi_k$  , then it holds:
$$
\begin{array}{l} \phi_ {k + 1} - \sin \left(\theta_ {k + 1} / 2\right) \geq (1 - \rho) \phi_ {k} - \phi_ {k} + \phi_ {k} - \sin \left(\theta_ {k} / 2\right) - \mu_ {k} C _ {1} \\ \geq - \rho \phi_ {k} + \frac {1}{4} \phi_ {k} - \frac {1 - 4 \rho}{1 6} \phi_ {k} = \frac {3 (1 - 4 \rho)}{1 6} \phi_ {k} > 0, \\ \end{array}
$$
since we have chosen  $\mu_{k} = (1 - 4\rho)\phi_{k} / (16C_{1})$ . Hence, it also holds that  $\sin (\theta_{k + 1} / 2) \leq \phi_{k + 1}$ .
Lemma 2.8 shows that  $\sin (\theta_k / 2)$  converges to 0 linearly. Therefore, using  $N_{2} = \widetilde{O}(d\log(1/\delta)/((1 - 2\eta)^{2}\epsilon'))$  samples, after  $K = O\left(\frac{1}{\rho}\log(1 / (\exp(t^{2} / 2)\epsilon'))\right) = O(\log(1 / (p\epsilon'))$  iterations, we get a  $\mathbf{w}_K$  such that  $\theta_K \leq 2\sin(\theta_K / 2) \leq \epsilon' \exp(t^{2} / 2)$ . Let  $h(\mathbf{x}) \coloneqq \mathrm{sign}(\mathbf{w}_K \cdot \mathbf{x} + \hat{t})$ . Equation (13) then implies that the disagreement of  $h(\mathbf{x})$  and  $f(\mathbf{x})$  is bounded by:
$$
\mathbf {P r} [ h (\mathbf {x}) \neq f (\mathbf {x}) ] \leq \frac {\epsilon^ {\prime 2}}{4 \sqrt {2 \pi}} + \frac {\epsilon^ {\prime}}{\pi} \leq \epsilon^ {\prime}.
$$
Furthermore, for any boolean function  $h: \mathbb{R}^d \mapsto \{\pm 1\}$  it holds
$$
\mathrm {e r r} _ {0 - 1} ^ {\mathcal {D}} (h) = \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {P r}} [ h (\mathbf {x}) \neq y ] = \eta + (1 - 2 \eta) \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {P r}} [ h (\mathbf {x}) \neq \mathrm {s i g n} (\mathbf {w} ^ {*} \cdot \mathbf {x} + t) ].
$$
Thus, to get misclassification error at most  $\eta +\epsilon$  (with respect to the  $y$ ), we only need to use  $\epsilon^{\prime} = \epsilon /(1 - 2\eta)$ , and we finally get that  $\mathbf{Pr}_{(\mathbf{x},y)\sim \mathcal{D}}[\mathrm{sign}(\mathbf{w}_K\cdot \mathbf{x} + \hat{t})\neq y]\leq \eta +\epsilon$ , using  $N_{2} = \widetilde{O} (d\log (1 / \delta) / ((1 - 2\eta)\epsilon))$  samples. Since the algorithm runs for  $O(\log (1 / \epsilon))$  iterations, the overall runtime is  $\widetilde{O} (N_2d)$ . This completes the proof of Theorem 2.6.
Proof of Theorem 1.3. From Lemma 2.2, we get that with  $\widetilde{O}(d\log(1/\delta)/((1 - 2\eta)^2p^2))$  samples Algorithm 4 produces a unit vector  $\mathbf{w}_0$  so that  $\theta(\mathbf{w}_0,\mathbf{w}^*) \leq \min(1/(5t),\pi/2)$ .
Since our guesses of the threshold  $t_m$ ,  $m \in [M]$  form a grid of  $(\epsilon^2 / (8(1 - 2\eta)^2)$ -separated values on the interval  $[\sqrt{2\log(1/\hat{p})}, \sqrt{2\log(4/\hat{p})}] \ni t$ , which covers all possible values of the true threshold  $t$ , there exists a  $\bar{m} \in [M]$  such that  $|t_{\bar{m}} - t| \leq \epsilon^2 / (8(1 - 2\eta)^2)$ . Thus, the condition of Theorem 2.6 is satisfied by at least one input threshold. Given  $\mathbf{w}_0$ , let  $\widehat{\mathbf{w}}_m$ ,  $m \in [M]$  be the weight vector produced by Algorithm 5 at call  $m = 1, \dots, M$ . From Theorem 2.6, we know that with  $\widetilde{O}(d\log(1/\delta) / ((1 - 2\eta)\epsilon))$  samples, with probability at least  $1 - \delta$  we get a list of halfspaces  $\{h_m(\mathbf{x}) : h_m(\mathbf{x}) = \mathrm{sign}(\widehat{\mathbf{w}}_m + t_m), m = 1, \dots, M\}$  so that
$$
\min  _ {h _ {m}, m \in [ M ]} \mathbf {P r} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ h _ {m} (\mathbf {x}) \neq y ] \leq \eta + \epsilon .
$$
Finally, to pick the optimal hypothesis from the list, we utilize the following fact.
Fact B.10 (Equation (7) in [MN06]). Let  $\mathcal{D}$  be a distribution over  $\mathbb{R}^d \times \{\pm 1\}$ . Let  $\mathcal{F}$  be a concept set of boolean functions with VC dimension at most  $d$ . Let  $\widehat{\mathcal{D}}$  be the empirical distribution obtained by drawing  $\widetilde{O}(d\log(1/\delta)/((1-2\eta)\epsilon))$  samples from  $\mathcal{D}$ . Then it holds that
$$
\min  _ {f \in \mathcal {F}} \underset {(\mathbf {x}, y) \sim \widehat {\mathcal {D}}} {\mathbf {P r}} [ f (\mathbf {x}) \neq y ] \leq \min  _ {f \in \mathcal {F}} \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {P r}} [ f (\mathbf {x}) \neq y ] + \epsilon .
$$
Using the above fact and a sample size of  $N_{3} = \widetilde{O}(d\log(1/\delta) / ((1 - 2\eta)\epsilon))$  from  $\mathcal{D}$ , we output the hypothesis with the minimum empirical error. By Fact B.10, we have that this will introduce an error at most  $\epsilon$  with probability at least  $1 - \delta$ . Since  $M = O(1/\epsilon^{2})$ , the total number of calls of Algorithm 5 in Algorithm 3 is  $O(1/\epsilon^{2})$ , and the runtime is  $\widetilde{O}(Nd/\epsilon^{2})$ .
Algorithm 6 Testing Procedure
Input: Hypothesis weight vectors  $\widehat{\mathbf{w}}_1, \widehat{\mathbf{w}}_2, \dots, \widehat{\mathbf{w}}_m$  and thresholds  $t_1, t_2, \dots, t_m$
Draw  $N_{3}$  samples  $\{(x^{(i)},y^{(i)})\}_{i = 1}^{N_3}$  from  $\mathcal{D}$
Calculate the test error (the fraction of misclassified points) for  $h_m(\mathbf{x}) = \mathrm{sign}(\widehat{\mathbf{w}}_m \cdot \mathbf{x} + t_m)$ ,  $m \in [M]$ , using  $\{(\mathbf{x}^{(i)}, y^{(i)})\}_{i=1}^{N_3}$
Let  $h_{\bar{m}}(\mathbf{x}) = \mathrm{sign}(\hat{\mathbf{w}}_{\bar{m}}\cdot \mathbf{x} + t_{\bar{m}})$ ,  $\bar{m}\in [M]$  be the halfspace with smallest empirical error.
return  $\widehat{\mathbf{w}}_{\bar{m}},t_{\bar{m}}$
B.3 The Case Where Both  $\eta$  and  $p$  are Unknown
Throughout this section, we carried out the analysis assuming knowledge of the noise parameter  $\eta$ . We now show how to relax this requirement, without changing the sample complexity (up to constant factors) and only affecting the algorithm runtime by a factor  $1 / \epsilon$ . In the following lemma, we show that with  $\widetilde{O}(d\log(1/\delta) / (p^2(1 - 2\eta)^2))$  samples we can compute constant factor estimates of the values of  $p$  and  $1 - 2\eta$ , which suffice for determining the correct number of samples to draw in all three subprocedures of our main algorithm (i.e., we can correctly determine  $N_1$ ,  $N_2$ , and  $N_3$ ).
Lemma B.11. There is an algorithm that uses  $\widetilde{O}(d\log(1/\delta)/(p^2(1-2\eta)^2))$  samples, and with probability at least  $1-\delta$  outputs estimates  $\hat{p}, \hat{\eta}$ , so that  $C\hat{p} \geq p \geq \hat{p}$  and  $C(1-2\hat{\eta}) \geq (1-2\eta) \geq (1-2\hat{\eta})$ , where  $C > 0$  is a sufficient large absolute constant.
Proof Sketch. We note that Algorithm 4, can, in fact, be used to get an estimate of  $(1 - 2\eta)p$  instead of only  $p$ . Therefore, Algorithm 4 outputs  $\hat{z}$  so that  $2\hat{z} \geq (1 - 2\eta)p \geq \hat{z}$ .
We assume for simplicity that  $f(\mathbf{x})$  is positively biased, i.e.,  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f(\mathbf{x})] \geq 0$ . Note that  $\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[\mathbb{1}\{y = b\}] = (1 - 2\eta)\mathbf{Pr}_{\mathbf{x} \sim \mathcal{N}}[f(\mathbf{x}) = b] + \eta$ , where  $b \in \{\pm 1\}$ . Because  $f(\mathbf{x})$  is positively biased, we have that  $\mathbf{Pr}_{\mathbf{x} \sim \mathcal{N}}[f(\mathbf{x}) = 1] \geq 1/2$ . Denote the random variable  $Z$  as  $Z = \mathbb{1}\{y = 1\} - 1/2$ . Note that  $\mathbf{E}[Z] \geq (1/2)(1 - 2\eta)(\mathbf{Pr}[f(\mathbf{x}) = 1] - 1/2)$ . Note that if  $p$  is less than a sufficiently small constant, then  $\mathbf{E}[Z] \geq (1/4)(1 - 2\eta)$ , whereas if  $p = 1/2$ , this expectation does not give any useful information. Note that by standard Chernoff bounds, using  $O(N\log(1/\delta))$  samples, where  $N$  is a parameter, we can get estimates  $\widehat{Z}$ , so that  $\mathbf{E}[Z] \geq \widehat{Z} - \sqrt{1/N}$ . By letting  $N = O(1/\widehat{z})$ , we can distinguish between the cases that  $(1 - 2\eta)(\mathbf{Pr}[f(\mathbf{x}) = 1] - 1/2) \leq \widehat{z}$ , in
which case we have that  $p$  is close to  $1/2$ , therefore  $\hat{z}$  is an estimate of  $1 - 2\eta$  that satisfies our requirements. Otherwise, we are in the case where  $\mathbf{E}[Z] \geq (1/4)(1 - 2\eta)$ . Therefore, we run the following algorithm: In each round  $s$ , we draw  $N_s = O(2^s \log (\log (1/\epsilon)/\delta))$  samples, and we check whether  $\widehat{Z}_s - \sqrt{1/N_s} \geq 1/2\widehat{Z}_s$ . If  $\widehat{Z}_s - \sqrt{1/N_s} < 1/2\widehat{Z}_s$ , we continue, otherwise we stop and return  $1/2\widehat{Z}_s$ , which is an effective lower bound of  $(1 - 2\eta)$ . The number of rounds is at most  $\log (1/\epsilon)$ , so by union bound, the probability of success is at least  $1 - \delta$ . After we have estimated an effective lower bound for  $(1 - 2\eta)$ , we can get an estimate for the value of  $p$ , using the estimator  $\hat{z}$  (recalling that  $2\hat{z} \geq (1 - 2\eta)p \geq \hat{z}$ ).
While Lemma B.11 is sufficient for ensuring that the number of samples our algorithm draws is not higher than when assuming the knowledge of  $\eta$ , it is not sufficient for correctly translating the 0-1 error and guaranteeing that it is bounded by  $\eta + O(\epsilon)$ . However, it is not hard to verify that if we run the Optimization procedure for an estimate  $\hat{\eta}$  that is within  $\pm \epsilon$  of  $\eta$ , then the correct 0-1 error bound of  $\eta + O(\epsilon)$  would follow. This is resolved by simply running the entire optimization component of the algorithm (including all calls to Algorithm 2) for a grid of  $\epsilon$ -separated values of  $\eta$  in the range  $(0, 1/2)$ , which must contain the true value of  $\eta$ . It is immediate that this increases the runtime (and the number of hypothesis halfspaces) by a factor  $O(1/\epsilon)$ . Yet the same number of samples suffices for the optimization and testing, as all that we require is that at least one hypothesis is constructed using estimates of  $\eta$  and  $t$  that are sufficiently close to their true values (by order- $\epsilon$  and order- $\epsilon^2$ , respectively, as discussed before).
C Omitted Content from Section 3
C.1 Background on Hermite Polynomials