SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. First we establish certain pieces of notation specific to what follows: we say a point  $\mathbf{x}$  is positive iff  $\langle \mathbf{x}, \mathbf{v} \rangle > 0$  and is negative iff  $\langle \mathbf{x}, \mathbf{v} \rangle < 0$ . We use  $S^{+}$  and  $S^{-}$  to denote these sets of points respectively. Note by construction, see (1), clean and corrupt points of the same sign are mutually i.i.d. As here we only ever consider one neuron and the activations of the training sample on this neuron at initialization, we also drop both the subscript  $j$  as well as the argument parentheses on the counting functions. We also use  $\pm$  superscripts to denote the subsets corresponding to activations from positive and negative points respectively: as examples  $T$  is used as shorthand for the total number of activations,  $B^{+}$  is the number corrupt positive activations and  $G^{-}$  is the number of clean negative activations.
First by the symmetry of the distribution of  $\mathbf{w}$ ,  $\mathbb{P}(\langle \mathbf{w},\mathbf{v}\rangle >0) = \mathbb{P}(\langle \mathbf{w},\mathbf{v}\rangle < 0) = \frac{1}{2}$ . As a result
$$
\begin{array}{l} \mathbb {P} ((B <   \kappa T) \cap (T > 0)) = \frac {1}{2} \mathbb {P} ((B <   \kappa T) \cap (T > 0) \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0) \\ + \frac {1}{2} \mathbb {P} ((B <   \kappa T) \cap (T > 0) \mid \langle \mathbf {w}, \mathbf {v} \rangle <   0). \\ \end{array}
$$
As the analysis and results derived under either condition will prove identical under reversal of the signs involved, without loss of generality we let  $\langle \mathbf{w},\mathbf{v}\rangle >0$  . Using the union bound
$$
\mathbb {P} ((B <   \kappa T) \cap (T > 0) \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0) \geq 1 - \mathbb {P} (T = 0 \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0) - \mathbb {P} (B \geq \kappa T \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0),
$$
therefore it suffices to upper bound the two probabilities on the right-hand-side.
Observe if  $\langle \mathbf{w},\mathbf{v}\rangle >0$  then for  $\mathbf{x}\in S^{+}$  we have  $\mathbb{P}(\langle \mathbf{x},\mathbf{w}\rangle) > 1 / 2$  and for  $\mathbf{x}\in S^{-}$  we have  $\mathbb{P}(\langle \mathbf{x},\mathbf{w}\rangle) < 1 / 2$ . By the mutual independence of the preactivations  $(\langle \mathbf{x},\mathbf{w}_j\rangle)_{i = 1}^{2n}$  then  $\mathbb{P}(T = 0|\langle \mathbf{w},\mathbf{v}\rangle >0)\leq (1 / 2)^n$ . Consider now a slightly different data model, in which a training sample consists of  $n - k$  clean positive points and  $2k$  corrupt positive points. Abusing notation, we let  $\zeta$  denote the event that we are instead drawing our training sample in this manner and also that  $\langle \mathbf{w},\mathbf{v}\rangle >0$ . In this setting  $T^{+} = T$  and furthermore the event  $B < \kappa T$  is equivalent to  $B^{+} < \kappa T^{+}$ . Again, as the preactivations are mutually independent the number of positive activations can be lower bounded using a binomial distribution with probability  $1 / 2$ . Applying a Chernoff bound it follows that
$$
\mathbb {P} \left(T ^ {+} <   \frac {n + k}{4} \mid \zeta\right) \leq \exp \left(- \frac {n + k}{1 6}\right).
$$
Furthermore, observe sampling positive points which activate  $\mathbf{w}_j$  is equivalent to uniformly sampling without replacement  $T^{+}$  points from  $S^{+}$ . Let  $Z_{\ell} = 1$  iff the  $\ell$ -th element sampled from  $S^{+}$  is corrupt and is 0 otherwise. Using a variant of Hoeffding's bound for sampling without replacement (see for example Proposition 1.2 of Bardenet & Maillard (2015))
$$
\mathbb {P} \left(B ^ {+} \geq \kappa T ^ {+} \mid \zeta\right) = \mathbb {P} \left(\frac {1}{T ^ {+}} \sum_ {\ell = 1} ^ {T ^ {+}} Z _ {\ell} - \mu \geq \kappa - \mu\right) \leq \exp \left(- 2 T ^ {+} (\kappa - \mu) ^ {2}\right).
$$
$$
\begin{array}{l} \mathbb {P} (B \geq \kappa T | \langle \mathbf {w}, \mathbf {v} \rangle > 0) \leq \mathbb {P} \left(B ^ {+} \geq \kappa T ^ {+} \mid \zeta\right) \\ \leq \mathbb {P} \left(B ^ {+} \geq \kappa T ^ {+} \mid T ^ {+} \geq \frac {n + k}{4}, \zeta\right) + \mathbb {P} \left(T ^ {+} <   \frac {n + k}{4} \mid \zeta\right) \\ \leq 2 \exp \left(- \frac {(n + k) (\kappa - \mu) ^ {2}}{1 6}\right). \\ \end{array}
$$
Combining these results it follows that
$$
\begin{array}{l} \mathbb {P} \left(\left(B <   \kappa T\right) \cap (T > 0) \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \geq 1 - \mathbb {P} (T = 0 \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0) - \mathbb {P} (B \geq \kappa T \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0) \\ \geq 1 - 3 \exp \left(- \frac {(n + k) (\kappa - \mu) ^ {2}}{1 6}\right). \\ \end{array}
$$
Therefore, there exist constants  $C, c > 0$  such that if  $n \geq C$  then
$$
\mathbb {P} \left(\left(B <   \kappa T\right) \cap (T > 0) \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \geq 1 - \delta .
$$
Note if instead we condition on the event  $\langle \mathbf{x},\mathbf{w}\rangle < 0$  then swapping the roles of the negative and positive points in the argument above gives the same outcome. As a result
$$
\mathbb {P} ((B <   \kappa T) \cap (T > 0)) \geq 1 - \delta .
$$
For convenience let  $X \coloneqq (\mathbf{x}_i)_{i=1}^{2n}$  denote the training sample and  $X_{\epsilon, \kappa}^c \coloneqq \{X : \mathbb{P}_w((B \geq \kappa T) \cup (T = 0)) > \epsilon\}$  the set of training samples which are not  $(\epsilon, \kappa)$ -good. Note here that the subscript  $w$  indicates randomness over the neuron alone, furthermore by construction
$$
\mathbb {P} \left(\left(B \geq \kappa T\right) \cup (T = 0) \mid X \in X _ {\epsilon , \kappa} ^ {c}\right) \geq \epsilon .
$$
Furthermore, as
$$
\delta \geq \mathbb {P} \left(\left(B \geq \kappa T\right) \cup (T = 0)\right) \geq \mathbb {P} \left(\left(B \geq \kappa T\right) \cup (T = 0) \mid X \in X _ {\epsilon , \kappa} ^ {c}\right) \mathbb {P} \left(X \in \chi_ {\epsilon , \kappa} ^ {c}\right),
$$
then it follows that  $\mathbb{P}\left(X\in X_{\epsilon ,\kappa}^{c}\right)\leq \frac{\delta}{\epsilon}$ . As a result we conclude that the probability of drawing a  $(\epsilon ,\kappa)$ -good training sample is at least  $1 - \frac{\delta}{\epsilon}$ .
Based on Lemma A.2, the following lemma bounds the probability that the cardinality of  $\Gamma_p$  is large. We note that the result presented here on non-overfitting requires  $|\Gamma_p| = m$  while the result on benign overfitting that  $|\Gamma_p| \geq (1 - \alpha)m$  for some small constant  $\alpha \in (0, 1)$ .
Lemma A.3. Suppose  $n \geq 15k$ ,  $\lambda_w \leq \eta \frac{\gamma - \rho}{4\gamma}$  and  $\gamma \geq 4\rho$ . Then there exist positive constants  $C, c > 0$  such that if  $n \geq C$  the following are true.
1.  $\mathbb{P}(|\Gamma_p| = m) \geq 1 - m \exp(-cn)$ .
2. With  $\alpha \in (0,1)$  then
$$
\mathbb {P} \left(\left| \Gamma_ {p} \right| \geq (1 - \alpha) m\right) \geq 1 - \exp (- c n).
$$
Proof. Again as here we only ever consider the activations at initialization, we write  $T_{j}(0,1)$  as  $T_{j}$ ,  $G_{j}(0,1)$  as  $G_{j}$  and  $B_{j}(0,1)$  as  $B_{j}$ . Let  $p \in \{-1,1\}$  and consider an arbitrary neuron  $j$  such that  $(-1)^{j} = p$ , by definition if
$$
G _ {j} (\gamma - \rho) - B _ {j} (\gamma + \rho) \geq \frac {2 \lambda_ {w}}{\eta}
$$
we may conclude  $j \in \Gamma_p$ . Rearranging this expression, equivalently  $j \in \Gamma_p$  if
$$
B _ {j} + \frac {\lambda_ {w}}{\eta} \leq \frac {\gamma - \rho}{2 \gamma} T _ {j}.
$$
As  $\frac{\lambda_w}{\eta} \leq \frac{\gamma - \rho}{4\gamma}$ , then membership to  $\Gamma_p$  is guaranteed as long as  $T_j \geq 1$  and  $B_j \leq \frac{\gamma - \rho}{4\gamma} T_j$ . Note by the assumptions of the lemma  $\mu := \frac{2k}{n + k} \leq \frac{1}{8}$  and  $\frac{\gamma - \rho}{4\gamma} \geq \frac{3}{16}$ . Conditioning on the event we draw a  $(\varepsilon, \frac{3}{16})$ -good training sample then the probability that  $j \notin \Gamma_p$  is at most  $\epsilon$  by Lemma A.2. Furthermore, with the training sample fixed the activations of each neuron are mutually independent. Let  $X = (\mathbf{x}_i)_{i=1}^{2n}$  denote the draw of the training sample and  $X_{\epsilon,\kappa} = \{X : \mathbb{P}_w((B \geq \kappa T) \cup (T = 0)) \leq \epsilon\}$  the set of  $(\varepsilon,\kappa)$ -good training samples. In what follows we assume  $\kappa = 3/16$  and let  $\epsilon = \exp(-cn)$  where  $c < 16^{-2}$  is a sufficiently small positive constant, then by Lemma A.2 there exist constants  $C, c$  such that if  $n \geq C$
$$
\mathbb {P} \left(X \in X _ {\epsilon , \kappa}\right) \geq 1 - \frac {\delta}{\epsilon} \geq 1 - \exp (- c n).
$$
The first result follows by applying the union bound,
$$
\begin{array}{l} \mathbb {P} \left(\left| \Gamma_ {p} \right| = m\right) \geq \mathbb {P} \left(\left| \Gamma_ {p} \right| = m \mid X \in X _ {\epsilon , \kappa}\right) \mathbb {P} \left(X \in X _ {\epsilon , \kappa}\right) \\ \geq (1 - m \exp (- c n)) (1 - \exp (- c n)) \\ \geq 1 - m \exp (- c n). \\ \end{array}
$$
For the second result, let  $\alpha' \in (0,1)$  denote the smallest scalar satisfying both  $\alpha < \alpha'$  and  $\alpha'm \in [m]$ . Observe
$$
\begin{array}{l} \mathbb {P} \left(\left| \Gamma_ {p} \right| <   (1 - \alpha) m \mid X \in X _ {\epsilon , \kappa}\right) = \mathbb {P} \left(\exists \mathcal {J} \subset [ 2 m ] _ {p}, \left| \mathcal {J} \right| = \alpha^ {\prime} m: j \notin \Gamma_ {p} \forall j \in \mathcal {J} \mid X \in X _ {\epsilon , \kappa}\right) \\ \leq \left( \begin{array}{c} m \\ \alpha^ {\prime} m \end{array} \right) \epsilon^ {\alpha^ {\prime} m} \\ \leq \left(\frac {\epsilon e}{\alpha^ {\prime}}\right) ^ {\alpha^ {\prime} m}. \\ \end{array}
$$
As  $\alpha$  is a constant, there exists positive constants  $C, c$  such that if  $n \geq C$  then
$$
\frac {\epsilon e}{\alpha} = \exp (- c n + 1 + \log (1 / \alpha^ {\prime})) \leq \exp (- c n).
$$
$$
\begin{array}{l} \mathbb {P} \left(\left| \Gamma_ {p} \right| \geq (1 - \alpha) m\right) \geq \mathbb {P} \left(\left| \Gamma_ {p} \right| \geq (1 - \alpha) m \mid X \in X _ {\epsilon , \kappa}\right) \mathbb {P} \left(X \in X _ {\epsilon , \kappa}\right) \\ \geq \left(1 - \exp (- c m n \alpha^ {\prime})\right) (1 - \exp (- c n)) \\ \geq 1 - \exp (- c n) \\ \end{array}
$$
We now turn our attention to establishing the conditions required at initialization on the corrupt points for the result on benign overfitting. To this end, in the following two lemmas we introduce the notion of an  $\epsilon$ -fine training sample and lower bound the probability of drawing one. Then, by conditioning on drawing such a training sample, we lower bound the cardinality of a set of neurons, which we denote  $\Lambda$ , which satisfy a property related to  $\Theta$ .
Lemma A.4. Assume  $\gamma \leq \frac{4}{5n}$  and  $\gamma \geq 5\rho$ . We say a training sample is  $\epsilon$ -fine if for a random neuron  $\mathbf{w}_j$  the inequality  $G_j < \frac{4}{10} T_j + \frac{5}{8} n$  holds with probability at least  $1 - \epsilon$ . There exist constants  $C, c > 0$  such that if  $n \geq C$  then with probability at least  $1 - \epsilon^{-1} \exp(-cn)$  the training sample is  $\epsilon$ -fine.
Proof. This lemma is analogous to Lemma A.2 and to this end we reuse much of the same notation. In particular, recall a point  $\mathbf{x}$  is positive iff  $\langle \mathbf{x},\mathbf{v}\rangle >0$  and is negative iff  $\langle \mathbf{x},\mathbf{v}\rangle < 0$ . Note all points with the same sign are mutually i.i.d. by construction. We use  $S^{+}$  and  $S^{-}$  to denote these sets of positive and negative points respectively. As here we only consider a single random neuron  $\mathbf{w}_j$  and the activations at initialization of the training sample on it, we also drop both the subscript  $j$  as well as the argument parentheses on the counting functions. We also use  $\pm$  superscripts to denote the subsets corresponding to activations from clean and corrupt points. As indicative examples of our notation going forward, we denote  $\mathbf{w}_j$  as  $\mathbf{w}$ ,  $T$  is used as shorthand for the total number of activations while  $B^{+}$  and  $G^{-}$  are the number of corrupt positive and clean negative activations respectively.
By the symmetry of the distribution of  $\mathbf{w}$ ,  $\mathbb{P}(\langle \mathbf{w},\mathbf{v}\rangle >0) = \mathbb{P}(\langle \mathbf{w},\mathbf{v}\rangle < 0) = \frac{1}{2}$ . As a result
$$
\begin{array}{l} \mathbb {P} \left(G <   \frac {4}{1 0} T + \frac {5}{8} n\right) = \frac {1}{2} \mathbb {P} \left(G <   \frac {4}{1 0} T + \frac {5}{8} n \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \\ + \frac {1}{2} \mathbb {P} \left(G <   \frac {4}{1 0} T + \frac {5}{8} n \mid \langle \mathbf {w}, \mathbf {v} \rangle <   0\right). \\ \end{array}
$$
As the analysis and results derived under either condition will prove identical under reversal of the signs involved, without loss of generality we let  $\langle \mathbf{w},\mathbf{v}\rangle >0$ . Consider this problem for a slightly different data model, in which a training sample consists of  $2(n - k)$  clean, positive points and  $k$  negative points. Abusing notation, we let  $\zeta$  denote the event that we are instead drawing our training sample in this manner and also that  $\langle \mathbf{w},\mathbf{v}\rangle >0$ . Clearly
$$
\mathbb {P} \left(G \geq \frac {4}{1 0} T + \frac {5}{8} n \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \leq \mathbb {P} \left(G ^ {+} \geq \frac {4}{1 0} T ^ {+} + \frac {5}{8} n \mid \zeta\right).
$$
In this setting, only positive points activate  $\mathbf{w}$ , therefore all points which activate  $\mathbf{w}$  are identically distributed. As a result, sampling positive points which activate  $\mathbf{w}$  is equivalent to uniformly sampling without replacement  $T^{+}$ points from  $\mathcal{S}^{+}$ . Let  $Z_{\ell} = 1$  if the  $\ell$ -th element sampled from  $\mathcal{S}^{+}$ is clean
and is 0 otherwise. Define  $\mu = \frac{2(n - k)}{2n - k}$ , then again using a variant of Hoeffding's bound for sampling without replacement (Bardenet & Maillard, 2015)[Proposition 1.2] and as long as  $\frac{4}{10} + \frac{5n}{8T^{+}} > \mu$ , we have
$$
\begin{array}{l} \mathbb {P} \left(G ^ {+} \geq \frac {4}{1 0} T ^ {+} + \frac {5}{8} n \mid \zeta\right) = \mathbb {P} \left(\frac {1}{T ^ {+}} G ^ {+} \geq \frac {4}{1 0} + \frac {5 n}{8 T ^ {+}} \mid \zeta\right) \\ = \mathbb {P} \left(\frac {1}{T ^ {+}} \sum_ {l = 1} ^ {T ^ {+}} Z _ {l} - \mu \geq \frac {4}{1 0} + \frac {5 n}{8 T ^ {+}} - \mu \mid \zeta\right) \tag {4} \\ \leq \exp \left(- 2 T ^ {+} \left(\frac {4}{1 0} + \frac {5 n}{8 T ^ {+}} - \mu\right) ^ {2}\right). \\ \end{array}
$$
We proceed to lower and upper bound  $T^{+}$ , to this end we first lower and upper bound the probability that a positive point activates  $\mathbf{w}$  conditioned on the event  $\langle \mathbf{w}, \mathbf{v} \rangle > 0$ . Let  $\mathbf{x} = \sqrt{\gamma} \mathbf{v} + \sqrt{1 - \gamma} \mathbf{n}$  be a fixed positive point, then by the symmetry of the distribution of  $\mathbf{w}$
$$
\begin{array}{l} \mathbb {P} \left(\langle \mathbf {w}, \mathbf {x} \rangle > 0 | \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) = \mathbb {P} \left(\sqrt {\gamma} \langle \mathbf {w}, \mathbf {v} \rangle + \sqrt {1 - \gamma} \langle \mathbf {w}, \mathbf {n} \rangle > 0 | \langle \mathbf {w}, \mathbf {v} \rangle <   0\right) \\ \geq \mathbb {P} (\langle \mathbf {w}, \mathbf {n} \rangle > 0) \\ = \frac {1}{2}. \\ \end{array}
$$
For convenience, let  $E_{1}$  denote the event  $\langle \mathbf{w},\mathbf{x}\rangle >0$ ,  $E_{2}$  the event  $\langle \mathbf{w},\mathbf{v}\rangle >0$  and  $E_{3}$  the event  $|\langle \mathbf{w},\mathbf{v}\rangle |\leq \varphi$  for some arbitrary  $\varphi \in (0,1)$ . For the upper bound observe
$$
\begin{array}{l} \mathbb {P} \left(\langle \mathbf {w}, \mathbf {x} \rangle > 0 \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) = \mathbb {P} \left(E _ {1} \mid E _ {2}\right) \\ = \mathbb {P} \left(E _ {1} \mid E _ {2}, E _ {3}\right) \mathbb {P} \left(E _ {3} \mid E _ {2}\right) + \mathbb {P} \left(E _ {1} \mid E _ {2}, E _ {3} ^ {c}\right) \mathbb {P} \left(E _ {3} ^ {c} \mid E _ {2}\right) \\ \leq \mathbb {P} \left(E _ {1} \mid E _ {2}, E _ {3}\right) + \mathbb {P} \left(E _ {3} ^ {c} \mid E _ {2}\right). \\ \end{array}
$$
Let  $\operatorname{Cap}(\mathbf{n},\varphi) \coloneqq \{\mathbf{z} \in \mathbb{S}^{d - 1} : \langle \mathbf{n},\mathbf{z}\rangle \geq \varphi\}$  denote the spherical cap of  $\mathbb{S}^{d - 1}$  centered on  $\mathbf{n}$ . As  $d \geq 3$  and  $\mathbf{w} \sim U(\mathbb{S}^{d - 1})$  then from Ball (1997)[Lemma 2.2] it follows that
$$
\mathbb {P} (| \langle \mathbf {w}, \mathbf {u} \rangle | \geq \varphi) = \mathbb {P} (\mathbf {w} \in \operatorname {C a p} (\mathbf {u}, \varphi)) \leq \exp \left(- \frac {d \rho^ {2}}{3}\right).
$$
$$
\begin{array}{l} \mathbb {P} \left(E _ {1} \mid E _ {2}, E _ {3}\right) = \mathbb {P} (\sqrt {\gamma} \langle \mathbf {w}, \mathbf {v} \rangle + \sqrt {1 - \gamma} \langle \mathbf {w}, \mathbf {n} \rangle > 0 \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0, | \langle \mathbf {w}, \mathbf {v} \rangle | \leq \varphi) \\ \leq \mathbb {P} \left(\langle \mathbf {w}, \mathbf {n} \rangle \geq - \sqrt {\frac {\gamma}{1 - \gamma}} \varphi\right) \\ = \frac {1}{2} + \frac {1}{2} \mathbb {P} \left(| \langle \mathbf {w}, \mathbf {n} \rangle | \leq \sqrt {\frac {\gamma}{1 - \gamma}} \varphi\right) \\ \leq 1 - \frac {1}{2} \exp \left(- \frac {d \gamma \varphi^ {2}}{3 (1 - \gamma)}\right). \\ \end{array}
$$
Therefore, and noting under the assumptions of the lemma that  $\frac{\gamma}{1 - \gamma} \leq \frac{4}{n}$ ,
$$
\begin{array}{l} \mathbb {P} \left(\langle \mathbf {w}, \mathbf {x} \rangle > 0 | \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \leq 1 + \exp \left(- \frac {d \varphi^ {2}}{3}\right) - \frac {1}{2} \exp \left(- \frac {d \gamma \varphi^ {2}}{3 (1 - \gamma)}\right) \\ \leq 1 + \exp \left(- \frac {d \varphi^ {2}}{3}\right) - \frac {1}{2} \exp \left(- \frac {4 d \varphi^ {2}}{3 n}\right). \\ \end{array}
$$
Set  $\varphi^2 = \frac{3\sqrt{n}}{d}$ , then
$$
\mathbb {P} \left(\langle \mathbf {w}, \mathbf {x} \rangle > 0 | \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \leq 1 + \exp \left(- \sqrt {n}\right) - \frac {1}{2} \exp \left(- \frac {4}{\sqrt {n}}\right).
$$
Letting  $\omega \in (0,1 / 2)$  be an arbitrary constant, then as long  $n\geq \max \{\ln^2\left(\frac{1}{\omega}\right),16\ln^{-2}\left(\frac{1}{1 - \omega}\right)\}$
$$
\mathbb {P} \left(\langle \mathbf {w}, \mathbf {x} \rangle > 0 \mid \langle \mathbf {w}, \mathbf {v} \rangle > 0\right) \leq \frac {1}{2} + \frac {\omega}{2}.
$$
In order to take advantage of concentration the upper bound on  $T^{+}$  must be greater than  $\mathbb{E}[T^{+}]$ . On the other hand, if the upper bound is too large then the condition  $\frac{4}{10} + \frac{5n}{8T_{j}^{+}} > \mu$  will be compromised.
As  $\mu < 1$  if
$$
T ^ {+} = \sum_ {i \in S ^ {+}} \mathbb {1} (i \in \mathcal {A} ^ {(0)}) \leq \frac {1 0 0}{9 6} n
$$
then this condition is not compromised. Setting  $\omega = \frac{1}{48}$ , if  $n \geq 16\ln^{-2}\left(\frac{48}{47}\right) \eqqcolon C$  then
$$
\mathbb {E} \left[ T ^ {+} \right] = (2 n - k) \mathbb {P} (i \in \mathcal {A} ^ {(0)}) \leq \frac {9 8}{9 6} n
$$
and therefore, applying a Chernoff bound,
$$
\mathbb {P} \left(T ^ {+} \leq \frac {9 9}{9 6} n \mid \zeta\right) \geq 1 - \exp \left(- \frac {n}{1 9 9 0 0}\right).
$$
Under the same conditions, to lower bound  $T^{+}$  we again apply a Chernoff, which gives
$$
\mathbb {P} \left(T ^ {+} \geq \frac {2 (n - k)}{4} \mid \zeta\right) \geq 1 - \exp \left(- \frac {2 (n - k)}{1 6}\right).
$$
Therefore, there exists a small positive constant  $c$  and a large constant positive constant  $C$  such that if  $n \geq C$  then
$$
\mathbb {P} \left(\frac {2 (n - k)}{4} \leq T ^ {+} \leq \frac {9 9}{9 6} n \mid \zeta\right) \geq 1 - \exp (- c n).
$$
In combination with the bound in (4), from this result it follows that there also exists a sufficiently small constant  $c > 0$  such that
$$
\mathbb {P} \left(G _ {j} ^ {+} \geq \frac {4}{1 0} + \frac {5}{8} n \mid \zeta , \frac {2 (n - k)}{4} \leq T ^ {+} \leq \frac {9 9}{9 6} n\right) \leq \exp (- c n).
$$
As a result, there exist positive constants  $C, c$  such that if  $n \geq C$  then
$$
\begin{array}{l} \mathbb {P} \left(G \geq \frac {4}{1 0} T + \frac {5}{8} n \mid \langle \mathbf {w}, \mathbf {v} \rangle <   0\right) \\ \leq \mathbb {P} \left(G ^ {+} \geq \frac {4}{1 0} T ^ {+} + \frac {5}{8} n \mid \zeta\right) \\ \leq \mathbb {P} \left(G ^ {+} \geq \frac {4}{1 0} T ^ {+} + \frac {5}{8} n \mid \zeta , \frac {2 (n - k)}{4} \leq T ^ {+} \leq \frac {9 9}{9 6} n\right) \mathbb {P} \left(\frac {2 (n - k)}{4} \leq T ^ {+} \leq \frac {9 9}{9 6} n \mid \zeta\right) \\ + \mathbb {P} \left(G ^ {+} \geq \frac {4}{1 0} T ^ {+} + \frac {5}{8} n \mid \zeta , \frac {2 (n - k)}{4} > T ^ {+} \text {o r} T ^ {+} > \frac {9 9}{9 6} n\right) \mathbb {P} \left(\frac {2 (n - k)}{4} > T ^ {+} \text {o r} T ^ {+} > \frac {9 9}{9 6} n \mid \zeta\right) \\ \leq \mathbb {P} \left(G ^ {+} \geq \frac {4}{1 0} T ^ {+} + \frac {5}{8} n \mid \zeta , \frac {2 (n - k)}{4} \leq T ^ {+} \leq \frac {9 9}{9 6} n\right) \mathbb {P} \left(\frac {2 (n - k)}{4} \leq T ^ {+} \leq \frac {9 9}{9 6} n \mid \zeta\right) \\ + \mathbb {P} \left(\frac {2 (n - k)}{4} \geq T ^ {+} \text {o r} T ^ {+} \geq \frac {9 9}{9 6} n \mid \zeta\right) \\ \leq \exp (- c n). \\ \end{array}
$$
Note if instead  $\langle \mathbf{x},\mathbf{v}\rangle < 0$ , then swapping the roles of the negative and positive points in the argument outlined above gives the same result. Therefore, under the assumptions of the lemma,
$$
\mathbb {P} \left(G <   \frac {4}{1 0} T + \frac {5}{8} n\right) \geq 1 - \exp (- c n).
$$
Let  $X = (\mathbf{x}_i)_{i=1}^{2n}$  denote the training sample and  $X_{\epsilon}^{c} = \{X : \mathbb{P}_{w}\left(G \geq \frac{4}{10} T + \frac{5}{8} n\right) > \epsilon\}$  the set of training samples which are not  $\epsilon$ -fine. Note the subscript  $w$  above indicates randomness over the neuron  $w$  alone. Clearly by construction
$$
\mathbb {P} \left(G \geq \frac {4}{1 0} T + \frac {5}{8} n \mid X \in X _ {\epsilon} ^ {c}\right) \geq \epsilon .
$$
$$
\exp (- c n) \geq \mathbb {P} \left(G \geq \frac {4}{1 0} T + \frac {5}{8} n\right) \geq \mathbb {P} \left(G \geq \frac {4}{1 0} T + \frac {5}{8} n \mid X \in X _ {\epsilon} ^ {c}\right) \mathbb {P} (X \in X _ {\epsilon} ^ {c}),
$$
then it follows that  $\mathbb{P}\left(X\in X_{\epsilon}^{c}\right)\leq \epsilon^{-1}\exp (-cn)$ . As a result we conclude that there exist positive constants  $C,c$  such that if  $n\geq C$  then the probability of drawing an  $\epsilon$ -fine training sample is at least  $1 - \epsilon^{-1}\exp (-cn)$ .
Lemma A.5. Assume  $\gamma \leq \frac{4}{5n}$ ,  $\gamma \geq 5\rho$  and let  $\Lambda := \{j \in [2m] : G_j < \frac{\gamma - \rho}{2\gamma} T_j + \frac{1}{2\gamma}\}$ . There exists a positive constant  $C$  such that for any  $\delta \in (0,1)$  if  $n \geq C \log(1/\delta)$  then
$$
\mathbb {P} (| \Lambda | > 1. 7 5 m) \geq 1 - \delta .
$$
Proof. To bound the size of  $\Lambda$  with high probability we follow a similar approach used to bound the size of  $\Gamma_p$  with high probability. As  $\gamma \leq \frac{4}{5n}$  and  $\rho \leq \frac{\gamma}{5}$ , then observe
$$
\frac {\gamma - \rho}{2 \gamma} T _ {j} + \frac {1}{2 \gamma} \geq \frac {4}{1 0} T _ {j} + \frac {5}{8} n.
$$
Therefore,  $j \in \Lambda$  if  $G_{j} < \frac{4}{10} T_{j} + \frac{5}{8} n$ . Conditioned on the event that the training sample is  $\epsilon$ -fine for  $\epsilon := \exp(-cn)$  for some sufficiently small constant  $c$ , then with the data fixed the preactivations of the data on each neuron are mutually independent and identically distributed by construction. As such, in this setting the events  $(G_{j} < \frac{4}{10} T_{j} + \frac{5}{8} n)_{j=1}^{2m}$  are also mutually independent. Let  $X = (\mathbf{x}_{i})_{i=1}^{2n}$  denote the training sample and  $X_{\epsilon}$  the set of  $\epsilon$ -fine training samples, then