SlowGuess/NeurIPS_2023_OCR huggingface dataset:
That is, the output weights are all fixed to be  $\pm 1$ , equally divided between the neurons. We will use the stationarity condition (Eq. (2)) freely throughout the proof. For our network it means that we can write for every  $j \in [n]$ :
$$
\mathbf {w} _ {j} = v _ {j} \sum_ {i \in [ m ]} \lambda_ {i} \sigma_ {i, j} ^ {\prime} y _ {i} \mathbf {x} _ {i}
$$
$$
b _ {j} = v _ {j} \sum_ {i \in [ m ]} \lambda_ {i} \sigma_ {i, j} ^ {\prime} y _ {i},
$$
where  $\sigma_{i,j}^{\prime} = \mathbb{1}(\mathbf{w}_{j}^{\top}\mathbf{x}_{i} + b_{j} >)$ ,  $v_{j} = 1$  for  $j\in \{1,\dots ,n / 2\}$  and  $v_{j} = -1$  for  $j\in \{n / 2 + 1,\ldots ,n\}$ .
The next lemma shows that all the biases are positive. Note that this lemma relies only on that the dimension is large enough, and that Property 1 of the data holds:
Lemma B.5. Assume that  $d \geq 8m^4\log \left(\frac{2m^2}{\delta}\right)$ . Then for every  $j \in [n]$  we have  $b_{j} \geq 0$ .
Proof. Assume on the way of contradiction that  $b_{j} < 0$  for some  $j \in [n]$ . We assume without loss of generality that  $j \in \{1, \dots, n/2\}$ , the proof for  $j \in \{n/2 + 1, \dots, n\}$  is done similarly. Denote by  $I_{+}' = \{i \in I_{+} : i \in I_{-} \}$  and  $I_{-}' = \{i \in I_{-} : i \in I_{+}\}$ . We can write:
$$
b _ {j} = \sum_ {i \in [ m ]} \lambda_ {i} \sigma_ {i, j} ^ {\prime} y _ {i} = \sum_ {i \in I _ {+} ^ {\prime}} \lambda_ {i} - \sum_ {i \in I _ {-} ^ {\prime}} \lambda_ {i}.
$$
Note that  $\lambda_{i}\geq 0$  for every  $i$ , hence  $I_{-}^{\prime}$  is non empty, otherwise  $b_{j}\geq 0$ . Take  $\lambda_r = \arg \max_{i\in I_-'}\lambda_i$  we have:
$$
0 <   w _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j} = b _ {j} + \sum_ {i \in I _ {+} ^ {\prime} \cup I _ {-} ^ {\prime}} \lambda_ {i} y _ {i} \mathbf {x} _ {r} ^ {\top} \mathbf {x} _ {i}
$$
$$
<   - \lambda_ {r} + \sum_ {i \in I _ {+} ^ {\prime} \cup I _ {-} ^ {\prime} \setminus \{r \}} \lambda_ {i} y _ {i} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r},
$$
where we used that  $b_{j} < 0$  and  $\| \mathbf{x}_r\| = 1$ . Rearranging the terms and using Property 1:
$$
\begin{array}{l} \lambda_ {r} <   \sum_ {i \in I _ {+} ^ {\prime} \cup I _ {-} ^ {\prime} \setminus \{r \}} \lambda_ {i} | y _ {i} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | \\ \leq \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {i \in I _ {+} ^ {\prime} \cup I _ {-} ^ {\prime} \backslash \{r \}} \lambda_ {i} \\ \leq \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {i \in I _ {+} ^ {\prime} \cup I _ {-} ^ {\prime}} \lambda_ {i}. \\ \end{array}
$$
Denote  $\lambda_s := \arg \max_{i \in I_+^{\prime} \cup I_-^{\prime}} \lambda_i$ , then from the above we have shown that  $\lambda_r \leq \lambda_sm\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}}$ , since  $|I_+^\prime \cup I_-^\prime| \leq m$ . In particular,  $s \in I_+^\prime$ , otherwise,  $\lambda_r = \lambda_s$  which means that  $\lambda_r < 0$  since
$m\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} < 1$  which is a contradiction. We can now write:
$$
\begin{array}{l} b _ {j} = \sum_ {i \in I _ {+} ^ {\prime}} \lambda_ {i} - \sum_ {i \in I _ {-} ^ {\prime}} \lambda_ {i} \\ \geq \sum_ {i \in I _ {+} ^ {\prime}} \lambda_ {i} - \sum_ {i \in I _ {-} ^ {\prime}} \lambda_ {r} \\ \geq \lambda_ {s} - \lambda_ {s} m ^ {2} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \\ = \lambda_ {s} \left(1 - m ^ {2} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}}\right). \\ \end{array}
$$
By our assumption on  $d$ , we have that  $m^2 \sqrt{\frac{2 \log \left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{2}$ , hence  $b_j \geq \frac{\lambda_s}{2}$ . But, by our assumption,  $b_j < 0$  which is a contradiction since  $\lambda_s \geq 0$ .
The following lemma is a general property of the KKT conditions:
Lemma B.6. Let  $i \in I_{+}$  (resp.  $i \in I_{-}$ ) with  $\lambda_i > 0$ . Then, there is a neuron  $k$  with positive output weight (resp. negative output weight)  $s.t \sigma_{i,k}' = 1$ .
Proof. We prove it for  $i \in I_{+}$ , the other case is similar. Assume otherwise, that is for every neurons with positive output  $j$  we have  $\sigma_{i,j}^{\prime} = 0$ , that is  $\mathbf{w}_j^\top \mathbf{x}_i + b_j \leq 0$  by the definition of  $\sigma_{i,j}^{\prime}$ . This means that  $N_{\theta}(\mathbf{x}_i) \leq 0$ , since all the positive neurons are inactive on  $\mathbf{x}_i$ , which is a contradiction to  $N_{\theta}(\mathbf{x}_i) \geq 1$ .
The next lemma shows that if the bias terms are smaller than  $\frac{1}{4}$ , then the  $\lambda_{i}$ 's for  $i \in I_{+}$  cannot be too small.
Lemma B.7. Assume that  $d \geq 32m^4 n^2 \log \left( \frac{2m^2}{\delta} \right)$ , and that  $\sum_{j=1}^{n/2} b_j - \sum_{j=n/2+1}^{n} b_j \leq \frac{1}{4}$ . Then, for every  $i \in I_+$  we have  $\lambda_i \geq \frac{1}{4n}$ .
Proof. Take  $r \in I_{+}$ , we have:
$$
\begin{array}{l} 1 \leq N _ {\boldsymbol {\theta}} (\mathbf {x} _ {r}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma \left(\lambda_ {r} \sigma_ {r, j} ^ {\prime} + \sum_ {i \in I \backslash \{r \}} \lambda_ {i} y _ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \lambda_ {r} \sigma_ {r, j} ^ {\prime} - \sum_ {i \in I \backslash \{r \}} \lambda_ {i} y _ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ \leq \sum_ {j = 1} ^ {n / 2} \sigma \left(\lambda_ {r} + \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \lambda_ {r} - \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) \tag {18} \\ \end{array}
$$
We will bound the two terms above. For the first term, note that all the terms inside the ReLU are positive, since by Lemma B.5 we have  $b_{j} \geq 0$ , and  $\lambda_{i} \geq 0$  by Eq. (4) hence we can remove the ReLU
function. Using Property 1 we can bound:
$$
\begin{array}{l} \sum_ {j = 1} ^ {n / 2} \sigma \left(\lambda_ {r} + \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) \leq \frac {n}{2} \lambda_ {r} + \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {j = 1} ^ {n / 2} \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} + \sum_ {j = 1} ^ {n / 2} b _ {j} \\ \leq \frac {n}{2} \lambda_ {r} + \frac {n}{2} \cdot \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {i \in I \backslash \{r \}} \lambda_ {i} + \sum_ {j = 1} ^ {n / 2} b _ {j}, \tag {19} \\ \end{array}
$$
where we used that  $\sigma_{i,j}^{\prime}\leq 1$ . For the second term in Eq. (18) we use the fact that the ReLU function is 1-Lipschitz to get that:
$$
\begin{array}{l} - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \lambda_ {r} - \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) \\ = - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \lambda_ {r} - \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + \sum_ {j = n / 2 + 1} ^ {n} \left| \lambda_ {r} + \sum_ {i \in I \backslash \{r \}} \lambda_ {i} \sigma_ {i, j} ^ {\prime} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | \right| \\ \leq - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + \frac {n}{2} \lambda_ {r} + \frac {n}{2} \cdot \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {i \in I \backslash \{r \}} \lambda_ {i}, \tag {20} \\ \end{array}
$$
where we again used Lemma B.5 to get that  $b_{j} \geq 0$ , and Property 1. Combining Eq. (19) and Eq. (20) with Eq. (18) we get that:
$$
1 \leq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + n \lambda_ {r} + n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {i \in I \backslash \{r \}} \lambda_ {i}.
$$
Rearranging the terms above, and using our assumption on the biases we get that:
$$
\frac {3}{4} \leq n \lambda_ {r} + n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \sum_ {i \in I \backslash \{r \}} \lambda_ {i}.
$$
If  $\lambda_r \geq \frac{1}{4n}$  then we are done. Assume otherwise, then  $n\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \sum_{i \in I \setminus \{r\}} \lambda_i \geq \frac{1}{2}$  and  $\lambda_r < \frac{1}{4n}$ . Denote  $\lambda_s := \arg \max_{i \in [m]} \lambda_i$ , then we have that  $\sum_{i \in [m] \setminus \{r\}} \lambda_i \leq m\lambda_s$ , which by the above inequality means that  $\lambda_s \geq \frac{\sqrt{d}}{2mn\sqrt{2\log\left(\frac{2m^2}{\delta}\right)}}$ . We split into cases:
Case I. Assume  $s \in I_{+}$ . By Lemma B.6 there is  $k \in \{1, \dots, n/2\}$  with  $\sigma_{s,k}' = 1$ . For the  $k$ -th neuron we have:
$$
\begin{array}{l} \sigma \left(\mathbf {w} _ {k} ^ {\top} \mathbf {x} _ {s} + b _ {k}\right) = \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, k} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {k}\right) \\ \geq \sigma \left(\lambda_ {s} - \sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {k}\right) \\ \geq \sigma \left(\lambda_ {s} - m \lambda_ {s} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} + b _ {k}\right), \tag {21} \\ \end{array}
$$
and note that since  $\lambda_s \geq 0$ ,  $m\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} < 1$  and  $b_j \geq 0$  this neuron is active. For every other neuron  $j$  with a positive output weight we have:
$$
\begin{array}{l} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sigma \left(b _ {j}\right) = \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sigma \left(b _ {j}\right) \\ \geq \sigma \left(\sum_ {i \in [ m ]} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) - \sigma (b _ {j}) \\ \geq \sigma \left(- m \lambda_ {s} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} + b _ {j}\right) - \sigma (b _ {j}) \\ \geq - m \lambda_ {s} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}}, \tag {22} \\ \end{array}
$$
where we used that  $\sigma$  is 1-Lipschitz and that  $\lambda_{s}$  is the largest among the  $\lambda_{i}$ 's. For a neuron with a negative output weight, we can write:
$$
\begin{array}{l} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) = \sigma \left(- \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) \\ \leq \sigma \left(- \sum_ {i \in I \setminus \{s \}} \lambda_ {i} | {\bf x} _ {i} ^ {\top} {\bf x} _ {s} | + b _ {j}\right) \leq b _ {j} + m \lambda_ {s} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \\ \end{array}
$$
In total, combining the above bound with Eq. (21) and Eq. (22) we have that:
$$
\begin{array}{l} 1 = N _ {\boldsymbol {\theta}} (\mathbf {x} _ {s}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma (\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}) + \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n / 2} \sigma (b _ {j}) \\ \geq - \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} + \lambda_ {s} + \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \\ \end{array}
$$
By our assumption on  $d$  we have that  $mn\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{2}$ . Hence, rearranging the terms above, we get that:
$$
\sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \leq 1 - \frac {\lambda_ {s}}{2}.
$$
But then, looking at the output of the network on  $\lambda_r$  (recall that by our assumption  $\lambda_r \leq \frac{1}{4n}$ ) we get:
$$
\begin{array}{l} 1 \leq N _ {\boldsymbol {\theta}} (\mathbf {x} _ {r}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) + \\ + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq \sum_ {j = 1} ^ {n / 2} \sigma \left(\lambda_ {r} + \sum_ {i \in I \backslash \{r \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \sum_ {i \in I \backslash \{r \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) + \\ + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq \frac {n \lambda_ {r}}{2} + \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} + \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \\ \leq \frac {1}{8} + \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} + 1 - \frac {\lambda_ {s}}{2} \\ \leq \frac {9}{8} + \lambda_ {s} \left(m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} - \frac {1}{2}\right). \tag {23} \\ \end{array}
$$
By our assumption  $mn\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{4}$ , hence rearranging the terms above we get that  $\lambda_s \leq \frac{1}{2}$  which is a contradiction to  $\lambda_s \geq \frac{\sqrt{d}}{2mn\sqrt{2\log\left(\frac{2m^2}{\delta}\right)}} > 1$ .
Case II. Assume  $s \in I_{-}$ . By Lemma B.6 there is  $k \in \{n/2 + 1, \ldots, n\}$  with  $\sigma_{s,k}' = 1$ . Note that since  $\lambda_s \neq 0$  we have that  $N_{\theta}(\mathbf{x}_s) = -1$  (i.e. this sample is on the margin). We have that:
$$
\begin{array}{l} - 1 = N _ {\boldsymbol {\theta}} (\mathbf {x} _ {s}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) + \\ + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq \sum_ {j = 1} ^ {n / 2} \sigma \left(\sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\lambda_ {s} - \sum_ {i \in I \backslash \{s \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {s} | + b _ {j}\right) + \\ + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} - \lambda_ {s} \\ \leq \frac {1}{4} + \lambda_ {s} \left(m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} - 1\right), \tag {24} \\ \end{array}
$$
where we used that  $mn\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{2}$  and  $b_{j} \geq 0$  by Lemma B.5, hence in the second to last inequality the term inside the ReLU is positive. In addition, we used that the ReLU function is 1-Lipschitz, and that  $\lambda_s$  is the largest among the  $\lambda_i$ 's. Since  $mn\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{2}$ , rearranging the terms above gives us that  $\lambda_s \leq \frac{10}{4}$ , which is a contradiction to that  $\lambda_s \geq \frac{\sqrt{d}}{2mn\sqrt{2\log\left(\frac{2m^2}{\delta}\right)}} \geq 3$ .
To conclude, both cases are not possible, hence  $\lambda_r \geq \frac{1}{4n}$ , which is true for every  $r \in I_+$ .
Lemma B.8. Assume that  $d \geq 16m^4 n^4 \log \left( \frac{2m^2}{\delta} \right)$ , and that  $\sum_{j=1}^{n/2} b_j - \sum_{j=n/2+1}^{n} b_j \leq \frac{1}{4}$ . Then, for every  $i \in I_-$  we have  $\lambda_i \leq \frac{10}{4}$ .
Proof. Take  $\lambda_s \coloneqq \arg \max_{i \in [m]} \lambda_i$ . We split into two cases:
Case I. Assume  $s \in I_{-}$ . Note that  $\lambda_s > 0$ , otherwise  $\lambda_i = 0$  for every  $i$ , which means that  $N_{\pmb{\theta}}(\mathbf{x})$  is the zero function. Hence,  $\mathbf{x}_s$  lies on the margin, and also by Lemma B.6 there is a neuron  $j$  with a negative output weight such that  $\sigma_{s,j}' = 1$ . By the same calculation as in Eq. (24) we have:
$$
\begin{array}{l} - 1 = N _ {\boldsymbol {\theta}} (\mathbf {x} _ {s}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {s} + b _ {j}\right) \\ \leq \frac {1}{4} + \lambda_ {s} \left(m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} - 1\right) \\ \end{array}
$$
Using our assumption that  $mn\sqrt{\frac{2\log\left(\frac{2m^2}{\delta}\right)}{d}} \leq \frac{1}{2}$  and rearranging the above terms we get:  $\lambda_s \leq \frac{10}{4}$  which finishes the proof.
Case II. Assume  $s \in I_{+}$ . Denote by  $\lambda_r \coloneqq S \arg \max_{i \in I_{-}} \lambda_i$ . By Lemma B.6 there is at least one neuron  $k$  with a positive output weight such that  $\sigma_{s,k}' = 1$ , for this neuron we can bound:
$$
b _ {k} = \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, k} ^ {\prime} \geq \lambda_ {s} - m \lambda_ {r}.
$$
Every neuron  $j$  with a negative output weight can be bounded similarly by:
$$
b _ {j} = - \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \leq m \lambda_ {r}.
$$
Combining the above bounds, and using that  $b_{j} \geq 0$  by Lemma B.5 we can bound:
$$
\begin{array}{l} \frac {1}{4} \geq \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \\ \geq \lambda_ {s} - m \lambda_ {r} - \sum_ {j = n / 2 + 1} ^ {n} m \lambda_ {r} \\ \geq \lambda_ {s} - m n \lambda_ {r}. \\ \end{array}
$$
Rearranging the terms we get that:  $\lambda_s \leq mn\lambda_r + \frac{1}{4}$ . If  $\lambda_r \leq 1$  we are finished, otherwise  $\lambda_s \leq 2mn\lambda_r$  and also  $\mathbf{x}_r$  lies on the margin since  $\lambda_r > 0$ , hence. By doing a similar calculation to Case I we can write:
$$
\begin{array}{l} - 1 = N _ {\boldsymbol {\theta}} (\mathbf {x} _ {r}) = \sum_ {j = 1} ^ {n / 2} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) \\ = \sum_ {j = 1} ^ {n / 2} \sigma \left(\sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(- \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} + b _ {j}\right) + \\ + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq \sum_ {j = 1} ^ {n / 2} \sigma \left(\sum_ {i \in I \backslash \{r \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) - \sum_ {j = n / 2 + 1} ^ {n} \sigma \left(\lambda_ {r} - \sum_ {i \in I \backslash \{r \}} \lambda_ {i} | \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {r} | + b _ {j}\right) + \\ + \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) - \sum_ {j = n / 2 + 1} ^ {n} \sigma (b _ {j}) \\ \leq - \lambda_ {r} + \sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} + \lambda_ {s} m n \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}} \\ \leq - \lambda_ {r} + \frac {1}{4} + 2 \lambda_ {r} m ^ {2} n ^ {2} \sqrt {\frac {2 \log \left(\frac {2 m ^ {2}}{\delta}\right)}{d}}. \\ \end{array}
$$
By the assumption on  $d$  we have  $2m^{2}n^{2}\sqrt{\frac{2\log\left(\frac{2m^{2}}{\delta}\right)}{d}} \leq \frac{1}{2}$ . Thus, rearranging the terms above and using these bounds we get that  $\lambda_r \leq \frac{10}{4}$ .
Lemma B.9. Assume that  $p \leq \frac{c_1}{n^2}$ ,  $m \geq c_2 n \log \left( \frac{1}{\delta} \right)$  and  $d \geq 32m^4 n^4 \log \left( \frac{2m^2}{\delta} \right)$ . Then,  $\sum_{j=1}^{n/2} b_j - \sum_{j=n/2+1}^{n} b_j > \frac{1}{4}$ .
Proof. Assume on the way of contradiction that  $\sum_{j=1}^{n/2} b_j - \sum_{j=n/2+1}^{n} b_j \leq \frac{1}{4}$ . By Lemma B.7 we have that  $\lambda_i \geq \frac{1}{4n}$  for every  $i \in I_+$ , and by Lemma B.8 we have that  $\lambda_i \leq \frac{10}{4}$  for every  $i \in I_-$ . By
Lemma B.6, for every  $i \in I_{+}$  there is some  $j \in \{1, \dots, n/2\}$  with  $\sigma_{i,j}' = 1$ . This means that:
$$
\begin{array}{l} \sum_ {j = 1} ^ {n / 2} b _ {j} = \sum_ {j = 1} ^ {n / 2} \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \\ \geq \frac {\left| I _ {+} \right|}{4 n} - \sum_ {j = 1} ^ {n / 2} \sum_ {i \in I _ {-}} \lambda_ {i} \geq \frac {\left| I _ {+} \right|}{4 n} - \frac {1 0 n \left| I _ {-} \right|}{8}. \\ \end{array}
$$
In a similar manner, we can bound:
$$
\begin{array}{l} \sum_ {j = n / 2 + 1} ^ {n} b _ {j} = \sum_ {j = n / 2 + 1} ^ {n} \sum_ {i \in [ m ]} y _ {i} \lambda_ {i} \sigma_ {i, j} ^ {\prime} \\ \geq \sum_ {j = n / 2 + 1} ^ {n} \sum_ {i \in I _ {-}} \lambda_ {i} \geq - \frac {1 0 n | I _ {-} |}{8}. \\ \end{array}
$$
Combining the two bounds above we get that:
$$
\sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \geq \frac {| I _ {+} |}{4 n} - \frac {1 0 n | I _ {-} |}{4}.
$$
But, by our assumptions we have that  $m \geq 2n$  (for a large enough constant), and by Lemma B.4 we have  $|I_{-}| \leq \frac{cm}{n^{2}}$  and  $|I_{+}| \geq \left(1 - \frac{c}{n^{2}}\right)m$  for some universal constant  $c > 0$  which we will later choose. Plugging these bounds to the displayed equation above, we get that:
$$
\sum_ {j = 1} ^ {n / 2} b _ {j} - \sum_ {j = n / 2 + 1} ^ {n} b _ {j} \geq \frac {1}{2} \cdot \left(1 - \frac {c}{n ^ {2}}\right) - \frac {1 0 c}{2} \geq \frac {1}{2} - \frac {1 1 c}{2},
$$
where in the last inequality we used that  $n \geq 1$ . Taking  $c$  to be a small enough constant (i.e.  $c < \frac{1}{22}$ ), we get that  $\sum_{j=1}^{n/2} b_j - \sum_{j=n/2+1}^{n} b_j > \frac{1}{4}$ , which is a contradiction.
Proof of Theorem 4.3. By Lemma B.9 we have that  $\sum_{j=1}^{n/2} b_j - \sum_{j=n/2+1}^{n} b_j > \frac{1}{4}$ . Denote by  $\lambda_s := \arg \max_{i \in [m]} \lambda_i$ , we would first like to show that  $\lambda_s \leq 3mn$ . We split into two cases:
Case I. Assume  $s \in I_{+}$ . Note that  $\mathbf{x}_s$  lies on the margin, otherwise  $\lambda_i = 0$  for every  $i \in I$ , which means that  $N_{\theta}$  is the zero predictor, this contradicts the assumption that  $N_{\theta}$  classifies the data correctly. Using a similar analysis to Case I in Lemma B.7 we have: