SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} 1 = N _ {\boldsymbol {\theta}} (\mathbf {x} _ {s}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma (\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (- \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}) + \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) \\ \geq \lambda_ {s} + \sum_ {j = 1} ^ {n / 2} \sigma \left(- \sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) + \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) \\ \geq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + \lambda_ {s} - \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \\ \geq \frac {1}{4} + \lambda_ {s} \left(1 - m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}}\right), \\ \end{array}
$$
where we used Lemma B.6 to show that there is at least one  $k \in \{1, \dots, n/2\}$  with  $\sigma_{s,k}' = 1$ . Using that  $mn\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{2}$  for a large enough constant  $c_3$  and rearranging the terms above we have that  $\lambda_s \leq \frac{6}{4} \leq 3mn$  since  $m, n \geq 1$ .
Case II. Assume  $s \in I_{-}$ . Take  $\lambda_r \coloneqq \arg \max_{i \in I_+} \lambda_i$ . By Lemma B.5 there is at least one neuron  $k \in \{n/2 + 1, \ldots, n\}$  with  $\sigma_{s,k}' = 1$ . We have that:
$$
\begin{array}{l} \frac {1}{4} \leq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \\ \leq \sum_ {j = 1} ^ {n / 2} \sum_ {i \in [ m ]} \lambda_ {i} y _ {i} \sigma_ {i, j} ^ {\prime} - \sum_ {j = n / 2 + 1} ^ {n} \sum_ {i \in [ m ]} \lambda_ {i} y _ {i} \sigma_ {i, j} ^ {\prime} \leq m n \lambda_ {r} - \lambda_ {s}. \\ \end{array}
$$
Rearranging the terms we get  $\lambda_s \leq mn\lambda_r - \frac{1}{4} \leq mn\lambda_r$ . Note that  $\mathbf{x}_r$  lies on the margin, otherwise  $\lambda_s = 0$ , which similarly to Case I is a contradiction. Again, using a similar analysis to Case I we get:
$$
\begin{array}{l} 1 = N _ {\boldsymbol {\theta}} (\mathbf {x} _ {r}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) + \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) \\ \geq \lambda_ {r} + \sum_ {j = 1} ^ {n / 2} \sigma \left(- \sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) + \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) \\ \geq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + \lambda_ {r} - \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \\ \geq \frac {1}{4} + \lambda_ {r} - \lambda_ {r} m ^ {2} n ^ {2} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}}, \\ \end{array}
$$
By our assumption,  $\frac{m^2n^2\log(d)2\sqrt{2}}{\sqrt{d}} \leq \frac{1}{2}$  for a large enough constant  $c_3$ . Hence, rearranging the terms we get  $\lambda_r \leq \frac{6}{4}$ . This means that  $\lambda_s \leq 2mn\lambda_r \leq 3mn$ .
We now turn to calculate the output of  $N_{\theta}$  on the sample  $\mathbf{x}$ . Suppose we sample  $\mathbf{x} \sim \mathrm{Unif}(\mathbb{S}_{d-1})$  by Lemma D.1 we have with probability  $> 1 - \epsilon$  that  $|\mathbf{x}_i^\top \mathbf{x}| \leq \sqrt{\frac{2 \log\left(\frac{m}{\epsilon}\right)}{d}}$  for every  $i \in [m]$ . We condition on this event for the rest of the proof. Note that for every positive neuron  $j$  we have that:
$$
\begin{array}{l} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) - \sigma \left(b _ {j}\right) = \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} + b _ {j}\right) - \sigma \left(b _ {j}\right) \\ \geq \sigma \left(- \lambda_ {s} m \sqrt {\frac {2 \log \left(\frac {m}{\epsilon}\right)}{d}} + b _ {j}\right) - \sigma (b _ {j}) \\ \geq - \lambda_ {s} m \sqrt {\frac {2 \log \left(\frac {m}{\epsilon}\right)}{d}}. \\ \end{array}
$$
Similarly, for every negative neuron  $j$  we have:
$$
\sigma (b _ {j}) - \sigma (\mathbf {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}) \geq - \lambda_ {s} m \sqrt {\frac {2 \log \left(\frac {m}{\epsilon}\right)}{d}}.
$$
In both bounds above we used that  $\lambda_{s}$  is the largest among the  $\lambda_{i}$ 's. Combining both bounds, we have that:
$$
\begin{array}{l} N _ {\boldsymbol {\theta}} (\mathbf {x}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) + \\ + \sum_ {j = 1} ^ {n / 2} \sigma \left(b _ {j}\right) - \sum_ {j = 1} ^ {n / 2} \sigma \left(b _ {j}\right) + \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(b _ {j}\right) \\ \geq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} - \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {m}{\epsilon}\right)}{d}} \\ \geq \frac {1}{4} - 3 m ^ {2} n ^ {2} \sqrt {\frac {2 \log \left(\frac {m}{\epsilon}\right)}{d}}. \\ \end{array}
$$
We use that  $\frac{3m^2n^2\log(d)\sqrt{2}}{\sqrt{d}} \leq \frac{1}{8}$  for a large enough constant  $c_3$ , hence  $N_{\theta}(\mathbf{x}) \geq \frac{1}{4} - \frac{1}{8} > 0$ , this finishes the proof.
C Proofs from Section 4.2
C.1 Proof of Proposition 4.4.
We first note that there exists  $j \in \{1,2\}$  with  $v_{j} < 0$ , since otherwise the network wouldn't be able to classify samples with a negative label (which exist by assumption). Assuming without loss of generality that  $v_{2} < 0$ , for any  $\mathbf{w}_{1}$  we also have that  $\operatorname*{Pr}_{\mathbf{x} \sim \mathrm{Unif}(\mathbb{S}^{d - 1})}[\mathbf{w}_{1}^{\top}\mathbf{x} \leq 0] = \frac{1}{2}$ . If this even occurs, then  $N_{\theta}(\mathbf{x}) \leq v_{2}\sigma (\mathbf{w}_{2}\top \mathbf{x}) \leq 0$ .
C.2 Proof of Proposition 4.5
Suppose  $[n] = J_{+} \cup J_{-}$  are such that
$$
N _ {\boldsymbol {\theta}} (\mathbf {x}) = \sum_ {j \in J _ {+}} v _ {j} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x}\right) + \sum_ {j \in J _ {-}} v _ {j} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x}\right),
$$
where  $v_{j} \geq 0$  for  $j \in J_{+}$  and  $v_{j} < 0$  for  $j \in J_{-}$ . Then, for any choice of  $(\mathbf{w}_j)_{j \in J_+}$ :
$$
\Pr_ {\mathbf {x} \sim \operatorname {U n i f} \left(\mathbb {S} ^ {d - 1}\right)} \left[ N _ {\boldsymbol {\theta}} (\mathbf {x}) \leq 0 \right] \geq \Pr_ {\mathbf {x} \sim \mathbb {S} ^ {d - 1}} \left[ \forall j \in J _ {+}, \mathbf {w} _ {j} ^ {\top} \mathbf {x} <   0 \right] \geq \frac {1}{2 ^ {| J _ {+} |}} \geq \frac {1}{2 ^ {n}}.
$$
C.3 Proof of Proposition 4.6
Assume without loss of generality that  $I_{-} = [k]$ . Consider the weights  $\mathbf{w}_i = \mathbf{x}_i$ ,  $v_{i} = -1$  for every  $i\in I_{-}$ , and  $\mathbf{w}_{k + 1} = \sum_{i\in I_+}\mathbf{w}_i$ ,  $v_{k + 1} = 1$ . For this network we have  $y_{i}N_{\theta}(\mathbf{x}_{i}) = 1$ , thus all points lie on the margin. In addition, it is not difficult to see that this network satisfies the other KKT conditions with  $\lambda_{i} = 1$  for every  $i\in [m]$ . Note that  $\operatorname*{Pr}_{\mathbf{x}\sim \mathrm{Unif}(\mathbb{S}^{d - 1})}[\mathbf{w}_{k + 1}^{\top}\mathbf{x} < 0] = \frac{1}{2}$  and  $\operatorname*{Pr}_{\mathbf{x}\sim \mathrm{Unif}(\mathbb{S}^{d - 1})}[\forall i\in I_{-},\mathbf{w}_{i}^{\top}\mathbf{x} < 0] = \frac{1}{2^{k}}$ , since all the  $\mathbf{x}_i$  are orthogonal. Thus, we get that
$$
\operatorname * {P r} _ {\mathbf {x} \sim \operatorname {U n i f} \left(\mathbb {S} ^ {d - 1}\right)} \left[ N _ {\boldsymbol {\theta}} (\mathbf {x}) \leq 0 \right] \geq \operatorname * {P r} _ {\mathbf {x} \sim \operatorname {U n i f} \left(\mathbb {S} ^ {d - 1}\right)} \left[ \mathbf {w} _ {k + 1} ^ {\top} \mathbf {x} <   0 \right] - \operatorname * {P r} _ {\mathbf {x} \sim \operatorname {U n i f} \left(\mathbb {S} ^ {d - 1}\right)} \left[ \forall i \in I _ {-}, \mathbf {w} _ {i} ^ {\top} \mathbf {x} <   0 \right] \geq \frac {1}{2} - \frac {1}{2 ^ {k}}.
$$
D Additional probabilistic lemmas
D.1 Proof of Lemma A.1
Clearly, the third (no collisions) condition holds almost surely, so it suffices to analyze the gaps between samples. The distribution of distances between uniformly random points on a segment is well studied [Pyke, 1965, Holst, 1980] and the lemma can be derived by known results. Nonetheless we provide a proof for completeness.
Denote by  $\Delta_1 \leq \Delta_2 \leq \dots \leq \Delta_{m+1}$  the ordered spacings  $x_1, (x_2 - x_1), \ldots, (x_m - x_{m-1}), (1 - x_m)$ . With this notation, note that  $E_x$  occurs if  $\Delta_{m+1} \leq \frac{\log(8(m+1)/\delta)}{m+1}$  and  $\Delta_{m/8} \geq \frac{1}{10m}$ . We will show that each of these conditions holds with probability at least  $1 - \frac{\delta}{4}$ , under which we would conclude by the union bound. Let  $Z_1, \ldots, Z_{m+1} \stackrel{iid}{\sim} \mathrm{Exp}(1)$  be unit mean exponential random variables, and denote their ordering by  $Z_{(1)} \leq \dots \leq Z_{(m+1)}$ . The main well known observation which we use is that for any  $j \in [m+1]: \Delta_j \stackrel{d}{=} \frac{Z_{(j)}}{\sum_{i=1}^{m+1} Z_i}$  (see Holst, 1980). Hence for any  $t > 0$ :
$$
\begin{array}{l} \Pr \left[ (m + 1) \Delta_ {j} - \log (m + 1) \leq t \right] \\ = \Pr \left[ Z _ {(j)} - \log (m + 1) \leq t + (t + \log (m + 1)) \left(\frac {\sum_ {i = 1} ^ {m + 1} Z _ {i}}{m + 1} - 1\right) \right]. \\ \end{array}
$$
Note that  $\frac{\sum_{i=1}^{m+1} Z_i}{m+1} - 1 \xrightarrow{p} 0$  as  $\mathbb{E}\left[\frac{\sum_{i=1}^{m+1} Z_i}{m+1}\right] = 1$  and  $\operatorname{Var}\left[\frac{\sum_{i=1}^{m+1} Z_i}{m+1}\right] = \frac{m+1}{(m+1)^2} \xrightarrow{m \to \infty} 0$ . Furthermore,
$$
\begin{array}{l} \Pr \left[ Z _ {(j)} - \log (m + 1) \leq t \right] = \Pr \left[ Z _ {(j)} \leq t + \log (m + 1) \right] = \Pr \left[ Z _ {(1)}, \dots , Z _ {(j)} \leq t + \log (m + 1) \right] \\ = \left(1 - e ^ {- t - \log (m + 1)}\right) ^ {j} = \left(1 - \frac {e ^ {- t}}{m + 1}\right) ^ {j}. \\ \end{array}
$$
By introducing the change of variables  $r = \frac{t + \log(m + 1)}{m + 1}$  we conclude that
$$
\lim  _ {m \to \infty} \Pr [ \Delta_ {j} \leq r ] = \left(1 - \frac {e ^ {- (m + 1) r + \log (m + 1)}}{m + 1}\right) ^ {j}.
$$
It remains to plug in our parameters of interest. For  $j = m + 1$  we get
$$
\lim  _ {m \rightarrow \infty} \Pr [ \Delta_ {m + 1} \leq r ] = \lim  _ {m \rightarrow \infty} \left(1 - \frac {e ^ {- (m + 1) r + \log (m + 1)}}{m + 1}\right) ^ {m + 1} = \lim  _ {m \rightarrow \infty} \exp \left(- e ^ {- (m + 1) r + \log (m + 1)}\right).
$$
Noting that for  $r = \frac{8\log(8(m + 1) / \delta)}{m + 1}$  it holds that  $e^{-(m + 1)r + \log (m + 1)} \to 0$  so we can use the Taylor approximation  $\exp (z) \approx 1 + z$  and conclude that
$$
\lim  _ {m \rightarrow \infty} \Pr [ \Delta_ {m + 1} \leq r ] = 1 - e ^ {- (m + 1) r + \log (m + 1)} = 1 - \frac {\delta}{8},
$$
where the last equality holds for  $r = \frac{\log(8(m + 1) / \delta)}{m + 1}$ . Overall for  $m$  larger than some numerical constant, the left hand side is larger than  $1 - \delta / 4$  as required.
The second condition follows a similar computation for  $j = \frac{m + 1}{8}$ ,  $r = \frac{\log(m / 8\log(8 / \delta))}{8(m + 1)}$ , while using the fact  $\operatorname*{Pr}\left[\Delta_{m / 8} > \frac{1}{10(m + 1)}\right] = 1 - \operatorname*{Pr}\left[\Delta_{m / 8} \leq \frac{1}{10(m + 1)}\right]$ .
D.2 Concentration bounds for vectors on the unit sphere
Here we present some standard concentration bounds for uniformly sampled points on the unit sphere in high dimension.
Lemma D.1 (Lemma 2.2 from Ball, 1997).  $\operatorname*{Pr}_{\mathbf{u},\mathbf{v}\sim \mathrm{Unif}(\mathbb{S}^{d - 1})}[|\mathbf{u}^{\top}\mathbf{v}| \geq t] \leq 2\exp \left(\frac{-dt^2}{2}\right)$ .
Lemma D.2 (Exercise 4.7.3 from Vershynin, 2018). Let  $\mathbf{x}_1,\ldots ,\mathbf{x}_m\sim \mathrm{Unif}(\mathbb{S}^{d - 1})$ , and denote by  $X$  the matrix whose  $i$ 'th row is  $\mathbf{x}_i$ . Then, there is a universal constant  $c > 0$  such that:
$$
\Pr \left[ \| X X ^ {\top} - I \| \geq \frac {c}{d} \cdot \left(\sqrt {\frac {d + t}{m}} + \frac {d + t}{m}\right) \right] \leq 2 e ^ {- t}.
$$