SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Combining with the derived upper bounds on  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}\left[\mathbf{x}_2\mathbb{1}\{\mathcal{E}_1\}\right]$  and  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}\left[\mathbf{x}_2\mathbb{1}\{\mathcal{E}_3\}\right]$  completes the proof.
Combining Claim B.4 and Claim B.3, we have  $\mathbf{E}_{\mathbf{x} \sim \mathcal{D}_{\mathbf{x}}}[\mathbf{x}_2 \mathbb{1}\{\mathcal{E}(\mathbf{w}, \hat{t}), \mathbf{x}_2 \sin \theta \leq -t - \mathbf{x}_1 \cos \theta\}] \leq -\frac{\mathbf{Pr}[\mathcal{E}(\mathbf{w}, \hat{t})]}{2\sqrt{2\pi}}$ . Thus, plugging this result back to Equation (4) and then combining with Equation (3), we complete the proof of the lemma.
Since, by construction,  $\mathbf{g}(\mathbf{w})$  is orthogonal to  $\mathbf{w}$  (see Line 5 in Algorithm 5), we can bound the norm of the expected gradient vector by bounding  $\mathbf{g}(\mathbf{w})\cdot \mathbf{u}$  for some unit vectors  $\mathbf{u}$  that are orthogonal to  $\mathbf{w}$  using similar techniques as in Lemma 2.3. To be specific, we have the following lemma.
Proof. First, we show that for any vector that is orthogonal to both  $\mathbf{w}$  and  $\mathbf{w}^*$ , the expected gradient of  $\mathbf{g}(\mathbf{w}) = \mathbf{g}(\mathbf{w};\mathbf{x},y)$  is zero. As a consequence,  $\bar{\mathbf{E}} [\mathbf{g}(\mathbf{w})]$  must lie in the 2-dimensional space spanned by  $\mathbf{w}$  and  $\mathbf{w}^*$ . To see that, observe first that
$$
\begin{array}{l} \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {v} ] = \frac {1 - 2 \eta}{\Pr [ \mathcal {E} (\mathbf {w} , \hat {t}) ]} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} [ \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {w} ^ {*} \cdot \mathbf {x} + t \leq 0 \} \mathbf {v} \cdot \operatorname {p r o j} _ {\mathbf {w} ^ {\perp}} (\mathbf {x}) ] \\ = \frac {1 - 2 \eta}{\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} [ \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {w} ^ {*} \cdot \mathbf {x} + t \leq 0 \} ] \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} [ \mathbf {v} \cdot \mathbf {x} ] = 0, \\ \end{array}
$$
where we used  $\mathrm{proj}_{\mathbf{w}^{\perp}}(\mathbf{x}) = \mathbf{x} - (\mathbf{x}\cdot \mathbf{w})\mathbf{w}$  and  $\mathbf{v}\cdot \mathbf{w} = 0$  , the fact that  $\mathbf{v}\cdot \mathbf{x}$  is independent of  $\mathbf{w}\cdot \mathbf{x}$  and  $\mathbf{w}^{*}\cdot \mathbf{x}$  , and that  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}[\mathbf{v}\cdot \mathbf{x}] = 0$  . Furthermore, by construction,  $\mathbf{g}(\mathbf{w})$  is orthogonal to  $\mathbf{w}$  . This indicates that  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[\mathbf{g}(\mathbf{w})]$  is parallel to  $\mathbf{w}^*$  , as both  $\mathbf{w}$  and  $\mathbf{w}^*$  are unit vectors. Since  $\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}$  is rotation invariant, we can assume  $\mathbf{w} = \mathbf{e}_1$  and  $\mathbf{w}^{*} = \cos \theta \mathbf{e}_{1} + \sin \theta \mathbf{e}_{2}$  . We thus only need to bound  $|\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}(\mathbf{w},\hat{t})}[\mathbf{g}(\mathbf{w})]\cdot \mathbf{e}_2|$  . Recall that  $\mathcal{D}(\mathbf{w},\hat{t})$  is the distribution conditioned on the band  $\mathcal{E}(\mathbf{w},\hat{t}) = \{\mathbf{x}: - \hat{t}\leq \mathbf{w}\cdot \mathbf{x}\leq -\hat{t} +\hat{\gamma}\}$  ; hence, by the definition of  $\mathbf{g}(\mathbf{w})$  , we have
$$
\begin{array}{l} \left| \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {e} _ {2} ] \right| = \frac {\left| \mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {e} _ {2} \mathbb {1} \{\mathcal {E} (\mathbf {w} , \hat {t}) \} ] \right|}{\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]} \\ = \frac {(1 - 2 \eta)}{\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]} | _ {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} \mathbf {E} [ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \sin \theta \mathbf {x} _ {2} \leq - t - \cos \theta \mathbf {x} _ {1} \right\} ] |. \tag {9} \\ \end{array}
$$
To proceed, we discuss the cases where  $t \leq 1$  and  $t \geq 1$ , following similar steps as in Lemma 2.3.
Claim B.5. Under the assumptions of Lemma 2.3, if  $t \geq 1$ , then
$$
\left\| \mathbf {E} _ {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w}, \hat {t})} [ \mathbf {g} (\mathbf {w}; \mathbf {x}, y) ] \right\| _ {2} \leq \frac {1 - 2 \eta}{\sqrt {2 \pi}}.
$$
Proof. As shown in the proof of Claim B.3, when  $t \geq 1$  the condition  $\mathbf{x}_2 \leq -t / \sin \theta - \mathbf{x}_1 \cot \theta$ ,  $-\hat{t} \leq \mathbf{x}_1 \leq -\hat{t} + \hat{\gamma}$  implies that  $\mathbf{x}_2 \leq 0$ . Hence, in this case we have
$$
\mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\} \leq \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq 0 \right\},
$$
and we can further conclude that
$$
\begin{array}{l} \left| \mathbf {E} _ {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \right] \right| = \mathbf {E} _ {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} \left[ - \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \right] \\ \leq \mathbf {E} _ {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} [ - \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq 0 \right\} ] \\ = \frac {\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{\sqrt {2 \pi}}. \\ \end{array}
$$
Plugging this back into Equation (9) yields the claimed result.
When  $t \leq 1$ , we use a slightly different decomposition
$$
\mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - \hat {t} / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\} = \mathbb {1} \left\{\mathcal {E} _ {1} ^ {\prime} \right\} + \mathbb {1} \left\{\mathcal {E} _ {2} ^ {\prime} \right\},
$$
$$
\mathcal {E} _ {1} ^ {\prime} = \mathcal {E} (\mathbf {w}, \hat {t}) \cap \left\{\mathbf {x} _ {2} \leq - t / \sin \theta - \mathbf {x} _ {1} \cot \theta , \mathbf {x} _ {2} \leq 0 \right\},
$$
$$
\mathcal {E} _ {2} ^ {\prime} = \mathcal {E} (\mathbf {w}, \hat {t}) \cap \left\{0 \leq \mathbf {x} _ {2} \leq - t / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\}.
$$
By the definitions of these two events, we have
$$
\underset {\mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {1} ^ {\prime} \} \right] \leq 0, \quad \underset {\mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {2} ^ {\prime} \} \right] \geq 0.
$$
Since in the proof of Lemma 2.3 we have shown that
$$
\underset {\mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - \hat {t} / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\} \right] \leq 0,
$$
it must hold that
$$
\begin{array}{l} \left| \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - \hat {t} / \sin \theta - \mathbf {x} _ {1} \cot \theta \} \right] \right| = \left| \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {1} ^ {\prime} \} \right] + \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {2} ^ {\prime} \} \right] \right| \\ \leq - \mathbf {E} _ {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} _ {1} ^ {\prime} \right\} \right]. \\ \end{array}
$$
Since  $\mathbb{1}\{\mathcal{E}_1'\} \leq \mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t}),\mathbf{x}_2\leq 0\}$ , we thus have
$$
\left| \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - \hat {t} / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\} \right] \right| \leq \frac {\operatorname* {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{\sqrt {2 \pi}}.
$$
Plugging this back into Equation (9) completes the proof.
The following lemma establishes a uniform convergence result for the empirical subgradient.
Lemma B.6. Consider the learning problem defined in Definition 1.1. Fix  $\epsilon'$ ,  $\delta \in (0,1/2)$ , and let  $\alpha$  be any absolute constant in  $(0,1)$ . Consider the following class of functions for  $(\mathbf{x},y) \sim \mathcal{D}$ :
$$
\mathcal {F} = \left\{\mathbf {g} ^ {\prime} (\mathbf {w}): \mathbf {w} \in \mathbb {R} ^ {d}, \| \mathbf {w} \| _ {2} = 1, \epsilon^ {\prime} \exp (t ^ {2} / 2) \leq \theta (\mathbf {w}, \mathbf {w} ^ {*}) \leq 1 / (5 t) \right\}, \quad w h e r e
$$
$$
\mathbf {g} ^ {\prime} (\mathbf {w}; \mathbf {x}, y) = \frac {1}{2} \big ((1 - 2 \eta) \mathrm {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) - y \big) \mathrm {p r o j} _ {\mathbf {w} ^ {\perp}} (\mathbf {x}) \mathbb {1} \left\{\mathbf {w} \cdot \mathbf {x} \in [ - \hat {t}, - \hat {t} + \hat {\gamma} ] \right\},
$$
and where  $\hat{t}$  satisfies  $|\hat{t} - t| \leq \epsilon'^2 / 8$  and  $\hat{\gamma} = (1/2)\epsilon'\exp(\hat{t}^2 / 2)$ . Then, using  $N = \widetilde{O}\left(\frac{d\log(1 / (\delta))}{(1 - 2\eta)^2\epsilon'}\right)$  samples from  $\mathcal{D}$  with probability at least  $1 - \delta$ , for any  $\mathbf{g}'(\mathbf{w}, \hat{t}) \in \mathcal{F}$  it holds
$$
\left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {g} ^ {\prime} (\mathbf {w}; \mathbf {x} ^ {(i)}, y ^ {(i)}) - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}; \mathbf {x}, y) ] \right\| _ {2} \leq \alpha \| \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}; \mathbf {x}, y) ] \| _ {2}.
$$
Proof. For simplicity, we will use  $\mathbf{g}'(\mathbf{w})$  to denote  $\mathbf{g}'(\mathbf{w};\mathbf{x},y)$  and we further define
$$
\widehat {\mathbf {g}} ^ {\prime} (\mathbf {w}) := \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {g} ^ {\prime} (\mathbf {w}; \mathbf {x} ^ {(i)}, y ^ {(i)}).
$$
By definition,  $\mathbf{g}'(\mathbf{w})$  is orthogonal to  $\mathbf{w}$ , hence so is  $\widehat{\mathbf{g}}'(\mathbf{w})$ . As already argued in the proof of Lemma 2.4,  $\mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[\mathbf{g}'(\mathbf{w})]$  is also orthogonal to  $\mathbf{w}$ . Thus,  $\|\widehat{\mathbf{g}}'(\mathbf{w}) - \mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[\mathbf{g}'(\mathbf{w})]\|_2$  is determined by  $(\widehat{\mathbf{g}}'(\mathbf{w}) - \mathbf{E}_{(\mathbf{x},y) \sim \mathcal{D}}[\mathbf{g}'(\mathbf{w})]) \cdot \mathbf{w}'$ , where  $\mathbf{w}'$  is a unit vector that is orthogonal to  $\mathbf{w}$ .
Fix unit vectors  $\mathbf{w},\mathbf{w}^{\prime}\in \mathbb{R}^{d}$  with  $\mathbf{w}\cdot \mathbf{w}^{\prime} = 0$  . We are going to make use of the following variant of Bernstein's inequality (see, e.g., [God55]).
Fact B.7. Let  $X_{1},\ldots ,X_{N}$  be zero mean i.i.d. random variables. Assume that for some positive reals  $L,\sigma >0$  , it holds that  $\mathbf{E}[|X_i|^k ]\leq (1 / 2)\sigma^2 L^{k - 2}k!$  . Then, for any  $x\in (0,\sqrt{N\sigma^2} /(2L))$
$$
\mathbf {P r} \left[ \left| \sum_ {i = 1} ^ {N} X _ {i} \right| \geq 2 x \sqrt {N \sigma^ {2}} \right] \leq \exp (- x ^ {2}).
$$
We show that the random variable  $\mathbf{g}'(\mathbf{w})\cdot \mathbf{w}'$  satisfies the assumptions of Fact B.7. For any  $k\geq 2$  using the definition of  $\mathbf{g}'$  which enforces  $\mathbf{g}'(\mathbf{w};\mathbf{x},y) = 0$  whenever  $\mathbb{1}\{\mathbf{w}\cdot \mathbf{x}\in [-\hat{t}, - \hat{t} +\hat{\gamma} ]\} = 0$  we have that
$$
\begin{array}{l} \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} \left[ \left| \mathbf {g} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} \right| ^ {k} \right] \leq \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbb {1} \left\{\mathbf {w} \cdot \mathbf {x} \in [ - \hat {t}, - \hat {t} + \hat {\gamma} ] \right\} \right] \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \left| \mathbf {w} ^ {\prime} \cdot \mathbf {x} \right| ^ {k} \right] \\ \leq \mathbf {P r} [ \mathbf {w} \cdot \mathbf {x} \in [ - \hat {t}, - \hat {t} + \hat {\gamma} ] ] C ^ {k - 2} k! , \\ \end{array}
$$
where the last inequality comes from the fact that for a \(\sqrt{2}\)-sub-Gaussian variable \(z\) (e.g., a Gaussian random variable), it holds \(\mathbf{E}[|z|^k] \leq (\sqrt{2} e\sqrt{k})^k \leq C_1C_2^{k-2}k!\) for some absolute constants \(C_1\) and \(C_2\), and we choose \(C\) to be a large enough multiple of \(C_1, C_2\). Let \(\sigma^2 = \mathbf{Pr}[\mathbf{w} \cdot \mathbf{x} \in [-\hat{t}, -\hat{t} + \hat{\gamma}]\]\n\nNote that since \(\hat{\gamma} = \frac{1}{2}\epsilon' \exp(\hat{t}^2/2)\), we have \(\sigma^2 \geq \hat{\gamma} \exp(-\hat{t}^2/2) = \frac{1}{2}\epsilon'\). Then, the condition of Fact B.7 is satisfied with \(\sigma^2 \geq \frac{1}{2}\epsilon'\).
Next, we show that we can bound  $\| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[\mathbf{g}'(\mathbf{w})]\| _2$  from below. In Lemma 2.3 we showed that when  $\epsilon^{\prime}\exp (t^{2} / 2)\leq \theta (\mathbf{w},\mathbf{w}^{*})\leq 1 / (5t),|\hat{t} -t|\leq \epsilon^{\prime 2} / 8$  and  $\hat{\gamma} = \frac{1}{2}\epsilon^{\prime}\exp (\hat{t}^{2} / 2)$  , it holds
$$
\begin{array}{l} \underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {w} ^ {*} ] = \frac {\mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {w} ^ {*} \mathbf {1} \{\mathbf {w} \cdot \mathbf {x} \in [ - \hat {t} , - \hat {t} + \hat {\gamma} ] \} ]}{\mathbf {P r} [ \mathbf {w} \cdot \mathbf {x} \in [ - \hat {t} , - \hat {t} + \hat {\gamma} ] ]} \\ = \frac {\mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} \left[ \mathbf {g} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {*} \right\}}{\sigma^ {2}} \leq - \frac {(1 - 2 \eta) \sin \left(\theta \left(\mathbf {w} , \mathbf {w} ^ {*}\right)\right)}{2 \sqrt {2 \pi}}. \tag {10} \\ \end{array}
$$
Let  $\tilde{\mathbf{w}}$  be a unit vector orthogonal to  $\mathbf{w}$  such that  $\mathbf{w}^{*} = \mathbf{w}\cos (\theta (\mathbf{w},\mathbf{w}^{*})) + \tilde{\mathbf{w}}\sin (\theta (\mathbf{w},\mathbf{w}^{*}))$ . Then, recalling the fact that  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[\mathbf{g}'(\mathbf{w})]$  is also orthogonal to  $\mathbf{w}$ , we have:
$$
\begin{array}{l} \| \mathbf {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) ] \| _ {2} = \sup  _ {\mathbf {v} \in \mathbb {R} ^ {d}: \| \mathbf {v} \| _ {2} = 1} \mathbf {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) ] \cdot \mathbf {v} \\ \geq - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} \left[ \mathbf {g} ^ {\prime} (\mathbf {w}) \right] \cdot \tilde {\mathbf {w}} \\ = - \frac {\mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) ] \cdot \mathbf {w} ^ {*}}{\sin \theta (\mathbf {w} , \mathbf {w} ^ {*})} \\ \geq \frac {1 - 2 \eta}{2 \sqrt {2 \pi}}, \\ \end{array}
$$
where in the last line we used Equation (10). Thus, as  $\sigma^2 \leq 1$  (it is defined as a probability), we conclude that
$$
\left\| \mathbf {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} \left[ \mathbf {g} ^ {\prime} (\mathbf {w}) \right] \right\| _ {2} \geq \frac {(1 - 2 \eta)}{2 \sqrt {2 \pi}} \sigma^ {2}. \tag {11}
$$
Applying Fact B.7, we now get that for any  $\mathbf{w}' \in \mathbb{R}^d$ ,  $\| \mathbf{w}' \|_2 = 1$ ,
$$
\mathbf {P r} \left[ \left| \widehat {\mathbf {g}} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} \left[ \mathbf {g} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} \right] \right| \geq 2 x \sqrt {\sigma^ {2} / N} \right] \leq \exp (- x ^ {2}).
$$
Choosing  $x = \frac{\alpha(1 - 2\eta)}{2\sqrt{2\pi}}\sqrt{\sigma^2N}$ , where  $\alpha \in (0,1)$  is an absolute constant, yields
$$
\mathbf {P r} \left[ \left| \widehat {\mathbf {g}} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} ] \right| \geq \frac {\alpha (1 - 2 \eta)}{2 \sqrt {2 \pi}} \sigma^ {2} \right] \leq \exp (- N \alpha^ {2} (1 - 2 \eta) ^ {2} \sigma^ {2} / (8 \pi)).
$$
In particular, since the above inequality holds for any unit  $\mathbf{w}'$ , using Equation (11), it follows that
$$
\begin{array}{l} \Pr \left[ \left| \widehat {\mathbf {g}} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) \cdot \mathbf {w} ^ {\prime} ] \right| \geq \alpha \| \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) ] \| _ {2} \right] \\ \leq \exp \left(- N \alpha^ {2} (1 - 2 \eta) ^ {2} \sigma^ {2} / (8 \pi)\right). \tag {12} \\ \end{array}
$$
It remains to show that Equation (12) holds for all functions in  $\mathcal{F}$ . To do that, we apply the union bound along all the directions  $\mathbf{w}'$ , all the hypothesis  $\mathbf{w}$ . A cover for these parameters will be of order  $(1 / \epsilon')^{O(d)}$ . Hence, we have that for any unit vector  $\mathbf{w} \in \mathbb{R}^d$  with  $\epsilon' \exp(t^2/2) \leq \theta(\mathbf{w}, \mathbf{w}^*) \leq 1/(5t)$ ,
$$
\begin{array}{l} \Pr \left[ \| \widehat {\mathbf {g}} ^ {\prime} (\mathbf {w}) - \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) ] \| _ {2} \geq \alpha \| \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} ^ {\prime} (\mathbf {w}) ] \| _ {2} \right] \\ \leq \exp (O (d \log (1 / \epsilon^ {\prime}))) \exp (- N \alpha^ {2} (1 - 2 \eta) ^ {2} \sigma^ {2} / (8 \pi)) \leq \delta , \\ \end{array}
$$
where in the last inequality, we used that  $\sigma^2 \geq \frac{1}{2}\epsilon'$ , and  $N \geq \widetilde{O}(d\log(1/\delta) / (\epsilon'(1 - 2\eta)^2))$ .
Recall that for some fixed  $\mathbf{w}$  and  $\hat{t}$ , we have defined the empirical gradient vector in Line 6 as:
$$
\widehat {\mathbf {g}} (\mathbf {w}) = \frac {1}{N _ {2} P (\hat {t} , \hat {\gamma})} \sum_ {i = 1} ^ {N _ {2}} \mathbf {g} (\mathbf {w}; \mathbf {x} ^ {(i)}, y ^ {(i)}) \mathbb {1} \left\{\mathbf {x} ^ {(i)} \in \mathcal {E} (\mathbf {w}, \hat {t}) \right\},
$$
where  $P(\hat{t},\hat{\gamma}) = \mathbf{Pr}_{z\sim \mathcal{N}}[z\in [-\hat{t}, - \hat{t} +\hat{\gamma} ]] = \mathbf{Pr}[\mathcal{E}(\mathbf{w},\hat{t})]$ , since  $\mathbf{w}$  is a unit vector and  $\mathbf{w}\cdot \mathbf{x}$  follows standard Gaussian. Thus,  $\widehat{\mathbf{g}} (\mathbf{w}) = \widehat{\mathbf{g}}^{\prime}(\mathbf{w}) / \mathbf{Pr}[\mathcal{E}(\mathbf{w},\hat{t})]$ . In addition, by definition we know that
$$
\underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) ] = \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) ] / \mathbf {P r} [ \mathcal {E} (\mathbf {w}, \hat {t}) ],
$$
and so Lemma B.6 immediately implies the following corollary.
Corollary B.8. Consider the learning problem from Definition 1.1. Let  $\epsilon', \delta, \hat{t}, \hat{\gamma}$  be parameters satisfying the condition of Lemma B.6 and choose  $\alpha = 1/4$ . Then using  $\widetilde{O}(d\log(1 / (\delta)) / ((1 - 2\eta)^2\epsilon'))$  samples to construct  $\widehat{\mathbf{g}}$ , for any unit vector  $\mathbf{w}$  such that  $\epsilon' \exp(t^2 / 2) \leq \theta(\mathbf{w}, \mathbf{w}^*) \leq 1 / (5t)$ , it holds with probability at least  $1 - \delta$ :  $\|\widehat{\mathbf{g}}(\mathbf{w}) - \mathbf{E}_{(\mathbf{x}, y) \sim \mathcal{D}(\mathbf{w}, \hat{t})}[\mathbf{g}(\mathbf{w})]\|_2 \leq (1/4)\|\mathbf{E}_{(\mathbf{x}, y) \sim \mathcal{D}(\mathbf{w}, \hat{t})}[\mathbf{g}(\mathbf{w})]\|_2$ .
We are now ready to present and prove our main algorithm-related result. A short roadmap for our proof is as follows. Since Algorithm 3 constructs a grid with grid-width  $\epsilon'^2 /8$  that covers all possible values of the true threshold  $t$ , there exists at least one guess  $\hat{t}$  that is  $\epsilon'^2$ -close to the true threshold  $t$ . We first show that to get a halfspace with error at most  $\epsilon'$ , it suffices to use this  $\hat{t}$  as the threshold and find a weight vector  $\mathbf{w}$  such that the angle  $\theta(\mathbf{w},\mathbf{w}^*)$  is of the order  $\epsilon'$ , which is exactly what Algorithm 5 does. The connection between  $\theta(\mathbf{w},\mathbf{w}^*)$  and the error is conveyed by the following fact:
Fact B.9 (see, e.g., Lemma 4.2 of [DKS18]). Under the standard normal distribution, it holds:
$$
\mathbf {P r} [ \mathrm {s i g n} (\mathbf {w} \cdot \mathbf {x} + t) \neq \mathrm {s i g n} (\mathbf {w} ^ {*} \cdot \mathbf {x} + t) ] \leq \frac {\theta (\mathbf {w} , \mathbf {w} ^ {*})}{\pi} \exp (- t ^ {2} / 2).
$$
Let  $\mathbf{w}_k$  be the parameter generated by Algorithm 5 at iteration  $k$  for threshold  $\hat{t}$ . We show that  $\theta(\mathbf{w}_k, \mathbf{w}^*)$  converges to zero at a linear rate. To this end, we prove that under our carefully devised step size  $\mu_k$ , there exists an upper bound on  $\|\mathbf{w}_k - \mathbf{w}^*\|_2$ , which contracts at each iteration. Note that since both  $\mathbf{w}_k$  and  $\mathbf{w}^*$  are on the unit sphere, we have  $\|\mathbf{w}_k - \mathbf{w}^*\|_2 = 2\sin(\theta(\mathbf{w}_k, \mathbf{w}^*) / 2)$ . Essentially, this implies that Algorithm 5 produces a sequence of parameters  $\mathbf{w}_k$  such that  $\theta(\mathbf{w}_k, \mathbf{w}^*)$  converges to 0 linearly, under this threshold  $\hat{t}$ . Thus, we can conclude that there exists a halfspace among all halfspaces generated by Algorithm 3 that achieves  $\epsilon'$  error with high probability.
Theorem 2.6. Consider the learning problem from Definition 1.1. Fix any unit vector  $\mathbf{w}_0\in \mathbb{R}^d$  such that  $\theta (\mathbf{w}_0,\mathbf{w}^*)\leq \min (1 / (5t),\pi /2)$ . Fix any  $\epsilon ,\delta >0$ . Let  $\hat{t} >0$  be a threshold such that  $|\hat{t} -t|\leq \epsilon^{2} / (8(1 - 2\eta)^{2})$ , and let  $\hat{\gamma} = \epsilon /(2(1 - 2\eta))\exp (\hat{t}^2 /2)$ . Then Algorithm 2 uses  $N_{2} = \widetilde{O}\big(d / ((1 - 2\eta)\epsilon)\log (1 / \delta)\big)$  samples from  $\mathcal{D}$ , has runtime  $\widetilde{O} (N_2d)$ , and outputs a weight vector  $\mathbf{w}$  such that  $h(\mathbf{x}) = \mathrm{sign}(\mathbf{w}\cdot \mathbf{x} + \hat{t})$  satisfies  $\operatorname *{Pr}[h(\mathbf{x})\neq y]\leq \eta +\epsilon$  with probability at least  $1 - \delta$ .
Proof. Let  $\epsilon' = \frac{\epsilon}{1 - 2\eta}$ , and denote  $\mathbf{w}_k$  as the parameter produced by the algorithm at  $k^{\text{th}}$  iteration under threshold  $\hat{t}$ . Observe that for any unit vector  $\mathbf{w}$ :