SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Gamma_ {p} := \left\{j \in [ 2 m ]: (- 1) ^ {j} = p, G _ {j} (0, 1) (\gamma - \rho) - B _ {j} (0, 1) (\gamma + \rho) \geq \frac {2 \lambda_ {w}}{\eta} \right\}, \\ \Theta_ {p} := \{j \in \Gamma_ {p}: G _ {j} (0, 1) (\gamma + \rho) - B _ {j} (0, 1) (\gamma - \rho) \leq 1 - \gamma + \rho \}. \\ \end{array}
$$
These sets are useful in that neurons in  $\Gamma_p$  have predictable activation patterns during the early phase of training. Furthermore, if  $i$  is the index of a corrupted point which activates a neuron in  $\Theta_{y_i}$  at initialization, then this point will continue activating this neuron throughout the early phase of training. Concentration argument shows that  $\Gamma_p$  and  $\Theta_p$  are sufficiently significant subsets of  $[2m]_p^2$  with high probability. In summary, for benign overfitting we say we have a good initialization if i)  $\max_{i\neq l}|\langle \mathbf{n}_i,\mathbf{n}_l\rangle |\leq \frac{\rho}{1 - \gamma}$ , ii) for some small constant  $\alpha \in (0,1)$  then  $|\Gamma_p|\geq (1 - \alpha)m$  for  $p\in \{-1,1\}$ , and iii) for each  $i\in S_F$  there exists a  $j\in [2m]$  such that  $(-1)^{j} = y_{i}$  and  $i\in \mathcal{A}_j^{(0)}$ .
Lemma 3.2. Under the assumptions of Theorem 3.1 and assuming we have a good initialization, suppose at some iteration  $t_0$  the loss of every clean point is bounded above by  $a \in \mathbb{R}_{\geq 0}$ , while the loss of every corrupted point is bounded above by  $b \in \mathbb{R}_{\geq 0}$ . Then for all  $t \geq t_0$  the total number of clean and corrupt updates which occur after  $t_0$  are upper bounded as follows,
$$
G (t _ {0}, t) \leq C n \left(\frac {a + b k \gamma}{\eta}\right), \quad B (t _ {0}, t) \leq C k \left(\frac {b + a n \gamma}{\eta}\right).
$$
Because these upper bounds are independent of  $t$  then we may conclude that training reaches a steady state after a finite number of iterations. In particular, this means every point either has zero loss or activates no neurons. To prove the network achieves zero loss we need only show that every training point activates at least one neuron after the last training update. This property is simple to prove for clean points: indeed, if  $i \in S_T$  then  $i$  activates every neuron in  $\Gamma_{y_i}$  after the first iteration. An inductive argument then shows  $i$  activates a neuron in every subsequent iteration. Showing that every corrupt point activates a neuron at the end of training is not as simple, and requires a more careful consideration of the training dynamics. To this end we say a neuron is a carrier of a training point between iterations  $t_0$  and  $t$  if  $i \in \mathcal{A}_j^{(\tau)}$  for all  $\tau \in [t_0, t]$ . In order to prove the network fits the corrupt data we need to show each corrupt point  $(\mathbf{x}_i, y_i)$  has a carrier neuron in  $\Theta_{y_i}$  throughout training. If too many clean points activate such a neuron, then it is possible it will eventually cease to carry any corrupt points and if a corrupt point loses all of its carrier neurons then it cannot be fitted. We show this event cannot occur by studying the activation patterns of neurons in  $\Gamma := \Gamma_1 \cup \Gamma_{-1}$ .
Lemma 3.3. Let the assumptions of Theorem 3.1 hold and suppose we have a good initialization. Let  $j \in \Gamma$  and  $t > 0$  be an iteration such that no point achieves zero loss at or before this iteration. For a point  $i \in S_T$ , then  $i \in \mathcal{A}_j^{(t)}$  iff  $i \sim j$ . For a point  $i \in S_F$  with  $i \not\sim j$ ,  $i \in \mathcal{A}_j^{(t)}$  iff  $i \in \mathcal{A}_j^{(1)}$ .
The next lemma bounds the activations just before any points achieve zero loss.
Here we use  $[2m]_{+1}$  to refer to the even indices, or those neurons with positive output weights, while  $[2m]_{-1}$  the odd indices, or those neurons with negative output weights.
Lemma 3.4. Under the assumptions of Theorem 3.1 and assuming we have a good initialization, there is an iteration  $\mathcal{T}_1 \leq \frac{C}{\eta m[1 + (\gamma + \rho)(n - k)]}$  before any point achieves zero loss where the following hold for a constant that varies from line to line.
1. For all  $p \in \{-1, 1\}$ ,  $j \in \Gamma_p$ ,  $i \sim j$ , and  $i \in S_T$ , then  $\langle \mathbf{w}_j^{(\mathcal{T}_1)}, \mathbf{x}_i \rangle \geq cm^{-1}$ .
2. For all  $p \in \{-1, 1\}$ ,  $j \in \Gamma_p$ ,  $i \not\sim j$ , and  $i \in S_T$ , then  $\langle \mathbf{w}_j^{(\mathcal{T}_1)}, \mathbf{x}_i \rangle \leq -cn\gamma m^{-1}$ .
3. For all  $i\in S_T$  , then  $\ell (\mathcal{T}_1,\mathbf{x}_i)\leq c$
Due to the fact that clean points are the majority and all of them push the network in the same signal direction, then immediately after  $\mathcal{T}_1$  the loss of clean points is small and clean points activate all neurons in the relevant  $\Gamma_p$  strongly. Furthermore, once the loss of a clean point is small it stays small. In subsequent iterations, if the number of corrupt updates since  $\mathcal{T}_1$  is also small, approximately  $C\varepsilon n\gamma / (\eta(\gamma + \rho))$ , then each clean point will activate on all but an  $\varepsilon$  proportion of neurons in the relevant  $\Gamma_p$ . As the hinge loss switches off the updates from a point once it reaches zero loss, eventually clean points do not participate in every iteration. Furthermore, when they do participate their updates are spread over a large proportion of the neurons. This ensures that most neurons in  $\Theta_p$  cannot receive too many clean updates in isolation, thereby ensuring carrier neurons continue to carry corrupted points throughout training.
Lastly, the generalization result follows from the near orthogonality of the noise components of both the training and test data. Indeed, using the same concentration bound, a test point satisfies the same inner product noise condition as the training data with high probability.
Lemma 3.5. Consider a test label  $y \in \{-1,1\}$  and point  $\mathbf{x} := y\sqrt{\gamma}\mathbf{v} + \sqrt{1 - \gamma}\mathbf{n}$ , where  $\mathbf{n} \sim \mathrm{Uniform}(S^{d-1} \cap \mathrm{span}\{\mathbf{v}\}^{\perp})$  is mutually i.i.d. from the training sample. Assume the conditions of Theorem 3.1 hold and that we have a good initialization. In addition, suppose that  $|\langle \mathbf{n}, \mathbf{n}_l \rangle| < \frac{\rho}{1 - \gamma}$  for all  $l \in [2n]$ , then  $yf(T_{end}, \mathbf{x}) > 0$ .
3.2 Non-benign overfitting
The next theorem states a harmful overfitting result: for sufficiently small  $\gamma$  the network achieves again zero loss on both the clean and corrupt data after a finite number of iterations, but the probability of misclassification is bounded from below by a constant. A detailed proof of this Theorem along with the associated lemmas is provided in Appendix D.
Theorem 3.6. Let Assumption 1 hold and further assume  $m \geq C\log (n / \delta)$ ,  $\rho \leq cn^{-1}$ ,  $\eta < 1 / (2mn)$  and  $\gamma \leq \frac{c}{\sqrt{nd}}$ . Then with probability at least  $1 - \delta$  over the randomness of the dataset and network initialization the following hold.
$$
\mathbb {P} (\operatorname {s g n} (f (\mathcal {T} _ {e n d}, \mathbf {x})) \neq y) \geq \frac {1}{8}.
$$
We remark that the above result holds for  $n \geq 1$  and any  $k$ . Indeed, in this regime the noise components dominate the training dynamics and we therefore expect the performance of the network on test points to be close to random. We re-emphasize that, unlike in the data model used by Frei et al. (2022) and Xu & Gu (2023), there does exist a classifier with perfect generalization error for arbitrarily small  $\gamma$ . The significance of Theorem 3.6 is that under the data model considered GD results in a suboptimal classifier.
Proof sketch. Similar to the proof of Theorem 3.1, in the context of non-benign overfitting we say the initialization is "good" if  $\max_{i\neq l}|\langle \mathbf{n}_i,\mathbf{n}_l\rangle |\leq \frac{\rho}{1 - \gamma}$  and if each point in the training sample
activates a neuron of the same sign. Under the conditions of Theorem 3.6 it can be shown that a good initialization in this context happens with high probability.
Lemma 3.7. In addition to the conditions of Theorem 3.6, suppose we have a good initialization and that for some iteration  $t_0$  then  $\ell(t_0, \mathbf{x}_i) \leq a$  for all  $i \in [2n]$ . Then  $T(t_0, t) \leq \frac{Cna}{\eta}$ .
As for the benign overfitting case, we need to show that each training point activates a neuron after the last training iteration. Under the assumptions on  $\gamma$  it can be shown that the loss of a point decreases during every iteration it participates in, regardless of the status and activations of other points in the training sample. All that remains is to lower bound the generalization error. To this end observe for a test point  $(\mathbf{x},y)$  that
$$
y \left(f \left(\mathcal {T} _ {\text {e n d}}, \mathbf {x}\right) - f \left(\mathcal {T} _ {\text {e n d}}, - \mathbf {x}\right)\right) = \sum_ {j = 1} ^ {2 m} y (- 1) ^ {j} \langle \mathbf {w} _ {j} ^ {\left(\mathcal {T} _ {\text {e n d}}\right)}, \mathbf {x} \rangle .
$$
If the right-hand-side of this equality is negative we can conclude that either  $\mathbf{x}$  or  $-\mathbf{x}$  is misclassified. That this event is true with probability lower bounded by a constant in turn follows by appropriately upper bounding the norm of the network weights in the signal subspace, as well as lower bounding the norm of the network weights in the noise subspace.
3.3 No-overfitting
The following theorem illustrates that for  $\gamma$  larger than the upper bound required for benign overfitting, then after convergence, which occurs in a finite number of iterations, only the clean points achieve zero loss. By contrast, the corrupt points cease to activate any neurons and are thus zeroed by the network. The network also achieves asymptotically optimal test error. A detailed proof of this theorem along with the associated lemmas is provided in Appendix E.
Theorem 3.8. Let Assumption 1 hold and further assume  $m \geq 2$ ,  $n \geq C \log \left(\frac{m}{\delta}\right)$ ,  $\rho \leq c\gamma$  and  $cn^{-1} \leq \gamma \leq ck^{-1}$ . Then there exists a sufficiently small step-size  $\eta$  such that with probability at least  $1 - \delta$  over the randomness of the dataset and network initialization we have the following.
1. The training process terminates at an iteration  $\mathcal{T}_{end} \leq \frac{Cn}{\eta}$ .
2. For all  $i\in S_T$  then  $\ell (\mathcal{T}_{end},\mathbf{x}_i) = 0$  while  $\ell (\mathcal{T}_{end},\mathbf{x}_i) = 1$  for all  $i\in S_F$
3. The generalization error satisfies
We remark that the upper bound on  $\gamma$  allows us to re-deploy the same proof technique used to prove convergence in the benign overfitting case, thereby ensuring the training process converges within a finite number of iterations. We conjecture this upper bound can be relaxed but leave such an analysis to future work.
Proof sketch. In the context of no-overfitting we identify a "good" initialization as one for which  $\max_{i\neq l}|\langle \mathbf{n}_i,\mathbf{n}_l\rangle |\leq \frac{\rho}{1 - \gamma}$  and  $\Gamma = \Gamma_{-1}\cup \Gamma_{+1} = [2m]$ . Under the conditions of Theorem 3.8 it can be shown a good initialization in this context occurs with high probability, furthermore the resulting activation pattern early during training is simple to characterize.
Lemma 3.9. Suppose that the conditions of Theorem 3.8 hold and that we have a good initialization. Consider an arbitrary  $j \in [2m]$  and iteration  $2 \leq t \leq \mathcal{T}_0$  occurring before a point has achieved zero loss. Then  $i \in \mathcal{A}_j^{(t)}$  iff  $i \sim j$ .
Next we bound the activations of the training points just before  $\mathcal{T}_0$ , the iteration at which any training points first achieve zero loss. In the following we use  $F_{1}, F_{2}$  and  $F_{3}$  as placeholders for expressions depending on the data and model parameters. Here, for the sake of conveying the ideas in the proof we do not write them in full and refer the reader to Supplementary Material.
Lemma 3.10. Suppose that the conditions of Theorem 3.8 hold and that we have a good initialization, then there is an iteration  $\mathcal{T}_1$  before any point achieves zero loss such that
$$
\langle \mathbf {w} _ {j} ^ {(\mathcal {T} _ {1})}, \mathbf {x} _ {i} \rangle \leq \frac {F _ {1}}{m} i f i \in \mathcal {S} _ {F}, i \sim j,
$$
$$
\langle \mathbf {w} _ {j} ^ {\left(\mathcal {T} _ {1}\right)}, \mathbf {x} _ {i} \rangle \geq \frac {F _ {2}}{m} i f i \in \mathcal {S} _ {T}, i \sim j,
$$
$$
\langle \mathbf {w} _ {j} ^ {\left(\tau_ {1}\right)}, \mathbf {x} _ {i} \rangle \leq - \frac {F _ {3}}{m} i f i \not \sim j.
$$
Next we seek to ensure the activation patterns remain mostly fixed: in particular, we show  $i \in \mathcal{A}_j^{(t)}$  if  $i \in S_T$  and  $i \sim j$ , while  $i \notin \mathcal{A}_j^{(t)}$  if  $i \not\sim j$ .
Lemma 3.11. Suppose that the conditions of Theorem 3.8 hold and that we have a good initialization. In addition, for  $a, b \in \mathbb{R}$  assume there is a time  $t_0$  such that  $\ell(t_0, \mathbf{x}_i) \leq a$  for all  $i \in S_T$  and  $\phi(\langle \mathbf{w}_j^{(t_0)}, \mathbf{x}_i \rangle) \leq b$  for all  $i \in S_F$  and  $i \sim j$ . If  $i \in S_T$ ,  $i \sim j$  implies  $i \in \mathcal{A}_j^{(\tau)}$  and  $i \not\sim j$  implies  $i \notin \mathcal{A}_j^{(\tau)}$  for all  $\tau$  satisfying  $t_0 \leq \tau < t$ , then
$$
B _ {j} (t _ {0}, t) \leq \frac {C k}{\eta} \left(b + \frac {a}{m}\right), \quad \sum_ {j \sim s} G _ {j} (t _ {0}, t) \leq \frac {C (a + m b)}{\gamma \eta}.
$$
As before, this update bound is finite and iteration-independent, therefore GD converges provided the assumptions on the activation patterns are not violated. Furthermore, if these activation patterns do hold, then every clean point activates a neuron and no corrupt point activates a neuron of the same label sign. Therefore, under the assumption on the activation pattern, at convergence clean points achieve zero loss while corrupt points have non-zero loss, i.e., they activate no neurons. It therefore suffices to prove the condition on the activation pattern, which we show holds as long as
$$
\min  \left\{\frac {F _ {3}}{m}, \frac {F _ {2}}{m} \right\} \geq C k (\gamma + \rho) \eta \left(\frac {F _ {1} + 1 - F _ {2}}{m}\right) \geq \eta (\gamma + \rho) B _ {j} (t _ {0}, t).
$$
As  $C$  does not depend on the parameters, we can ensure this condition holds by letting  $Ck(\gamma + \rho)$  be sufficiently small. With  $\rho \leq cn^{-1}$ , we show it suffices that  $\gamma < ck^{-1}$ . Finally, the generalization result follows in a fashion almost identical to that used for Lemma 3.5.
3.4 Comparison of results
We compare the differing regimes of our results side-by-side with those of Frei et al. (2022); Xu & Gu (2023) in Table 1. We note that comparisons are not like-for-like as Frei et al. (2022) consider smooth, leaky ReLU and logistic loss, Xu & Gu (2023) a generalized family of activation functions, which includes ReLU, and logistic loss, and this paper ReLU and hinge loss. Furthermore, in addition to differences in the noise distribution discussed in Section 2.1, Frei et al. (2022); Xu & Gu (2023) assume a data model where the norm of each data point is approximately proportional to  $\sqrt{d}$ . We therefore re-scale their results in order to make comparison with this work in which all data points have unit norm.
Taken together these results suggest, at least under the type of data model considered, that benign overfitting occurs for signal strengths proportional to between roughly  $1 / \sqrt{dn}$  and  $1 / n$ . Furthermore, our results also suggest that above approximately  $1 / n$  one might expect to see a transition to no-overfitting, while below approximately  $1 / \sqrt{nd}$  a transition to harmful overfitting. We provide preliminary supporting experiments in the Supplementary Material. We again remark that the latter is non-trivial in our setting as for all  $\gamma > 0$  the classifier  $h(\mathbf{x}) = \mathrm{sign}(\langle \mathbf{v},\mathbf{x}\rangle)$  always has perfect accuracy.
$^3$ Frei et al. (2022) show overfitting results for smaller  $\gamma$ , but assume a data model where benign overfitting is possible only when  $1 / (\gamma \sqrt{nd})$  is asymptotically zero, implicitly showing non-benign overfitting.
Table 1: across all results  $k \leq cn$  while  $d \geq Cn^2 \log (n / \delta)$  for (Frei et al., 2022), Xu & Gu (2023) and Theorem 3.1.
Frei et al. (2022)	Xu & Gu (2023)	Theorem 3.1	Theorem 3.6	Theorem 3.8
n≥C·	log(1/δ)	log(m/δ)	log(1/δ)	1	log(m/δ)
m≥C·	1	log(n/δ)	log(n/δ)	log(n/δ)	1
γ≤c·	1/n	1/n	1/n	1/√nd	1/k
γ≥C·	1/√nd	√log(md/nδ)/nd	√log(n/δ)/d	0	1/n
Result	Benign3	Benign	Benign	Non-benign	No-overfit
Developing a theoretical description of benign overfitting in neural networks is a highly nascent area, with mathematical results available only for very limited data models. Furthermore, the conditions describing the transitions between overfitting versus non-overfitting and benign versus non-benign even in these simplified settings are yet to be fully characterized. The goal of this work was to address this issue as well as explore the impact of using the hinge loss. In particular, and admittedly for a simple data model, we prove three different training outcomes, corresponding to non-benign overfitting, benign overfitting and no-overfitting, based on conditions on the margin of the clean data. Our analysis also differs significantly from prior works due to the fact the ratio of loss between different training points can be unbounded and the implicit bias of using hinge loss versus exponentially tailed loss is poorly understood.
Limitations and future work: the key limitation of this work is the restrictiveness of the data model. In particular, as in prior and related works we use a near-orthogonal noise model and assume a rank one signal, we also place additional conditions on the noise distribution. In addition to generalizing the signal and noise model as well as improving the bounds required for our results to hold, we believe the following themes are important areas for future research: first relaxing the near orthogonal noise condition, second exploring data models beyond those which are linearly separable, third investigating the role and impact of depth.
EG, WS and DN were partially supported by NSF DMS 2011140 and NSF DMS 2108479. EG was also partially supported by NSF DGE 2034835.
Appendix A Properties of the data and network at initialization
For each of our results to hold we require certain properties on both the network weights and training sample to hold at initialization. Here we bound the probabilities of these events in turn. Later, for each specific setting we combine the relevant conditions using the union bound.
First, and in order to prove convergence, we require the noise components of the training sample to be approximately orthogonal to one another.
Lemma A.1. Let  $\rho, \delta \in (0, 1)$ . Given a sequence  $(\mathbf{n}_i)_{i=1}^{2n}$  of mutually i.i.d. random vectors with  $\mathbf{n}_i \sim U(\mathbb{S}^{d-1} \cap \text{span}(\mathbf{v})^\perp)$  for all  $i \in [2n]$ , then assuming  $d \geq \max \left\{3, 3\rho^{-2}\ln\left(\frac{2n^2}{\delta}\right)\right\}$
$$
\mathbb {P} \left(\bigcap_ {i, l \in [ 2 n ], i \neq \ell} \{\left| \langle \mathbf {n} _ {i}, \mathbf {n} _ {\ell} \rangle \right| \leq \rho \}\right) \geq 1 - \delta .
$$
Proof. Consider two pairs of mutually i.i.d. random vectors  $\mathbf{n},\mathbf{n}^{\prime}\sim U(\mathbb{S}^{d - 1}\cap \mathrm{span}(\mathbf{v})^{\perp})$  and  $\mathbf{u},\mathbf{u}^{\prime}\sim U(\mathbb{S}^{d - 2})$ , observe
$$
\langle \mathbf {n}, \mathbf {n} ^ {\prime} \rangle \stackrel {{d}} {{=}} \langle \mathbf {u}, \mathbf {u} ^ {\prime} \rangle .
$$
Due to the fact that  $\mathbf{u}$  and  $\mathbf{u}'$  are independent as well as the rotational invariance of  $U(\mathbb{S}^{d-2})$  it follows that
$$
\langle \mathbf {u}, \mathbf {u} ^ {\prime} \rangle \stackrel {{d}} {{=}} \langle \mathbf {u}, \mathbf {e} _ {1} \rangle ,
$$
where  $\mathbf{e}_1 \coloneqq [1,0..0]^T$ . Let  $\mathrm{Cap}(\mathbf{e}_1,\rho) \coloneqq \{\mathbf{z} \in \mathbb{S}^{d-2} : \langle \mathbf{e}_1,\mathbf{z} \rangle \geq \rho\}$  denote the spherical cap of  $\mathbb{S}^{d-2}$  centered on  $\mathbf{e}_1$ . As  $d \geq 3$  then from Ball (1997)[Lemma 2.2] it follows that
$$
\mathbb {P} (| \langle \mathbf {n}, \mathbf {n} ^ {\prime} \rangle | \geq \rho) = \mathbb {P} (\mathbf {u} \in \operatorname {C a p} (\mathbf {e} _ {1}, \rho)) \leq \exp \left(- \frac {(d - 1) \rho^ {2}}{2}\right) \leq \exp \left(- \frac {d \rho^ {2}}{3}\right).
$$
Applying the union bound
$$
\begin{array}{l} \mathbb {P} \left(\bigcap_ {i, \ell \in [ 2 n ], i \neq \ell} \{\left| \langle \mathbf {n} _ {i}, \mathbf {n} _ {\ell} \rangle \right| \leq \rho \}\right) = 1 - \mathbb {P} \left(\bigcup_ {i, \ell \in [ 2 n ], i \neq \ell} \{\left| \langle \mathbf {n} _ {i}, \mathbf {n} _ {\ell} \rangle \right| \geq \rho \}\right) \\ \geq 1 - 2 n ^ {2} \mathbb {P} \left(\left| \langle \mathbf {n} _ {i}, \mathbf {n} _ {\ell} \rangle \right| \geq \rho\right) \\ \geq 1 - 2 n ^ {2} \exp \left(- \frac {d \rho^ {2}}{3}\right). \\ \end{array}
$$
Setting  $\delta \geq 2n^2\exp \left(-\frac{d\rho^2}{3}\right)$  and rearranging we arrive at the result claimed.
In addition to requiring the approximate orthogonality property on the training data, our approach also requires a large proportion of the neurons at initialization to satisfy particular conditions in regard to the number of clean versus corrupt activations. To this end, we introduce the following terms where  $p \in \{-1, 1\}$ .
- Let  $\Gamma_p := \{j : (-1)^j = p, G_j(0,1)(\gamma - \rho) - B_j(0,1)(\gamma + \rho) \geq \frac{2\lambda_w}{\eta}\}$  denote the set of neurons with output weight  $(-1)^p$  which have more clean points activating them than corrupt ones at initialization. We will show that these sets of neurons have a predictable behavior early during training before any clean points achieve zero loss. We further let  $\Gamma = \Gamma_1 \cup \Gamma_{-1}$ .
- Let  $\Theta_p := \{j \sim \Gamma_p : G_j(0,1)(\gamma + \rho) - B_j(0,1)(\gamma - \rho) < 1 - \gamma + \rho\} \subset \Gamma_p$ . For our benign overfitting result we will show that neurons in this subset are able to carry corrupt points throughout training, eventually, at least in the overfitting setting, enabling them to achieve zero loss. We further let  $\Theta = \Theta_1 \cup \Theta_{-1}$ .
First we show  $\Gamma$  accounts for a significant proportion of neurons. To this end we first provide the following result.
Lemma A.2. Define  $\mu \coloneqq \frac{2k}{n + k}$  and assume  $\kappa \in (0,1)$  satisfies  $\kappa >\mu$ . Given an arbitrary neuron  $\mathbf{w}_j\sim U(\mathbb{S}^{d - 1})$ , we say that a collection of training points is  $(\varepsilon ,\kappa)$ -good iff both  $T_{j}(0,1)\geq 1$  and  $B_{j}(0,1) < \kappa T_{j}(0,1)$  with probability at least  $1 - \epsilon$  over the randomness of the neuron. There exist positive constants  $C,c$  such that if  $\delta \coloneqq \exp (-cn(\kappa -\mu^2))$  and  $n\geq C$  then with probability at least  $1 - \frac{\delta}{\epsilon}$  the training sample is  $(\varepsilon ,\kappa)$ -good.