SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.1. Denote by  $E_{x}$  the event in which the samples  $(x_{i})_{i = 1}^{m}\sim \mathcal{D}_{x}^{m}$  satisfy
- (maximal gap isn't too large)  $d_{\mathrm{max}} \coloneqq \max_{i \in [m - 1]} (x_{i + 1} - x_i) \leq \frac{\log(8(m + 1) / \delta)}{m + 1}$ .
- (most gaps aren't too small)  $\left| \left\{ i \in [m - 1] : x_{i + 1} - x_i < \frac{1}{10(m + 1)} \right\} \right| < \frac{m + 1}{8}$ .
- (no collisions)  $\forall i \neq j \in [m] : x_i \neq x_j$ .
Then there exists absolute  $m_0 \in \mathbb{N}$  such that  $\forall m \geq m_0: \operatorname{Pr}_{S_x \sim \mathcal{D}_x^m}[E_x] \geq 1 - \frac{\delta}{4}$ .
Proof. Deferred to Appendix D.
Following the lemma above, we continue by conditioning on the probable event  $E_{x}$ , after which we will conclude the proof by the union bound. We now state a lemma due to Safran et al. [2022] which is crucial for our analysis.
Lemma A.2 (Lemma E.6. Safran et al., 2022). Suppose that  $i < j$  are such that  $y_{i}, y_{i+1}, \ldots, y_{j} = -1$ . Then in the interval  $[x_{i}, x_{j}]$  there are at most two points at which  $N_{\theta}'$  increases.
We derive the following corollary:
Corollary A.3. Suppose that  $y_{i},y_{i + 1},\ldots ,y_{i + 4} = -1$ . Then there exists  $i\leq \ell \leq i + 3$  for which  $N_{\pmb{\theta}}|_{[x_{\ell},x_{\ell +1}]} < 0$ .
Proof. Assume towards contradiction that  $y_{i}, \ldots, y_{i+4} = -1$ , yet for any  $i \leq \ell \leq i + 3$  there exists  $z_{\ell} \in (x_{\ell}, x_{\ell+1})$  such that  $N_{\theta}(z_{\ell}) \geq 0$ . Recall that  $N_{\theta}(x_{i}), \ldots, N_{\theta}(x_{i+3}) \leq -1$  by Eq. (3). Thus for each  $i \leq \ell \leq i + 2$ , looking at the segment  $(z_{\ell}, z_{\ell+1}) \ni x_{\ell+1}$  we see that  $N_{\theta}(z_{\ell}) > 0$ ,  $N_{\theta}(x_{\ell+1}) \leq -1$ ,  $N_{\theta}(z_{\ell}) > 0$ . In particular, by the mean value theorem, any such segment must contain a point at which  $N_{\theta}'$  increases. Obtaining three such points which are distinct contradicts Lemma A.2.
We assume without loss of generality that  $m$  is divisible by 5 and split the index set  $[m]$  into groups consecutive five indices: we let  $I_1 = \{1, \dots, 5\}$ ,  $I_2 = \{6, \dots, 10\}$  and so on up to  $I_{m/5}$ . Denoting by  $\mu$  the (one dimensional) Lebesgue measure we get that under the event  $E_x$  it holds that
$$
\begin{array}{l} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \Pr_ {x \sim \mathcal {D} _ {x}} [ N _ {\boldsymbol {\theta}} (x) <   0 ] \right] = \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ \mu (x: N _ {\boldsymbol {\theta}} (x) <   0) ] \\ \geq \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \sum_ {i \in [ m - 1 ]} \mu \left(x _ {i + 1} - x _ {i}\right) \cdot \mathbb {1} \left\{N \mid_ {\left[ x _ {i}, x _ {i + 1} \right]} <   0 \right\} \right] \\ = \sum_ {i \in [ m - 1 ]} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \left(x _ {i + 1} - x _ {i}\right) \cdot \mathbb {1} \left\{N | _ {[ x _ {i}, x _ {i + 1} ]} <   0 \right\} \right] \\ = \sum_ {i \in [ m / 5 ]} \sum_ {l \in [ I _ {i} ]} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \left(x _ {l + 1} - x _ {l}\right) \cdot \mathbb {1} \left\{N | _ {[ x _ {l}, x _ {l + 1} ]} <   0 \right\} \right] \\ \end{array}
$$
$$
\begin{array}{l} \left. \right.\left[ \text {C o r o l l a r y} A. 3 \right] \geq \sum_ {i \in [ m / 5 ]} \sum_ {l \in \left[ I _ {i} \right]} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \min  _ {\ell \in I _ {i}} \left(x _ {\ell + 1} - x _ {\ell}\right) \cdot \mathbb {1} \left\{\forall \ell \in I _ {i}: y _ {\ell} = - 1 \right\}\right] \\ = \sum_ {i \in [ m / 5 ]} \sum_ {l \in [ I _ {i} ]} \min  _ {\ell \in I _ {i}} (x _ {\ell + 1} - x _ {\ell}) \cdot \Pr \left[ y _ {i}, y _ {i + 1}, \dots , y _ {i + 4} = - 1 \right] \\ \geq \widetilde {c p} ^ {5}, \\ \end{array}
$$
where the last inequality follows from our conditioning on  $E_{x}$ . To see why, note that the sum  $\sum_{i\in [m / 5]}\sum_{l\in [I_i]}\min_{\ell \in I_i}(x_{\ell +1} - x_\ell)$  is lower bounded by the sum of the  $m / 5$  smallest gaps, yet under  $E_{x}$  this sum contains at least  $\Omega (m)$  summands larger than  $\Omega (1 / m)$  — hence it is at least some constant. We conclude that as long as  $E_{x}$  occurs we have  $\mathbb{E}_{S_y\sim \mathcal{D}_y^m}\left[\operatorname *{Pr}_{x\sim \mathcal{D}_x}\left[N_\pmb {\theta}(x) < 0\right]\right]\geq cp^5$ . Moreover, we see by the analysis above that if a single label  $y_{l}$  for some  $l\in I_i\subset [m]$  is changed, this can affect  $N_{\pmb{\theta}}(x)$  only in the segment  $[x_{\mathrm{min}_{I_i}\ell},x_{\mathrm{max}_{I_i}\ell}]$  which is of length at most  $7d_{\mathrm{max}} = O\left(\frac{\log(m / \delta)}{m}\right)$ . Thus we can apply McDiarmid's inequality to obtain that under  $E_{x}$ , with probability at least  $1 - \delta /4$ :
$$
\operatorname * {P r} _ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) <   0 \right] \geq c \left(p ^ {5} - \sqrt {\frac {\log (m / \delta)}{m}}\right).
$$
Overall, by union bounding over  $E_{x}$  the inequality above holds with probability at least  $1 - \delta /2$ , which proves the desired lower bound.
We now turn to prove the upper bound. Let  $N^{*}(\cdot)$  be a 2-layer ReLU network of minimal width  $n^{*} \in \mathbb{N}$  that classifies the data correctly, namely  $y_{i}N^{*}(x_{i}) > 0$  for all  $i \in [m]$ . Note that  $n^{*}$  is uniquely defined by the sample while  $N^{*}$  is not. Furthermore,  $n^{*}$  is upper bounded by the number of neighboring samples with different labels. Hence,
$$
\begin{array}{l} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ n ^ {*} ] \leq \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ | \{i \in [ m - 1 ]: y _ {i} \neq y _ {i + 1} \} | ] \tag {6} \\ = (m - 1) \cdot \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ \mathbb {1} \left\{y _ {1} \neq y _ {2} \right\} ] \\ = (m - 1) \cdot \Pr_ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ y _ {1} \neq y _ {2} ] \\ = 2 (m - 1) p (1 - p) = O (p m). \\ \end{array}
$$
We conclude that the expected width of  $N^{*}$  (as a function of the sample) is at most  $n^{*} = O(pm)$ . By Safran et al. [2022, Theorem 4.2], this implies  $N_{\theta}$  belongs to a class of VC dimension  $O(n^{*}) = O(pm)$ . Thus by denoting the 0-1 loss  $L^{0 - 1}(N_{\theta}) = \operatorname{Pr}_{(x,y)\sim \mathcal{D}}[\mathrm{sign}(N_{\theta}(x))\neq y]$  and invoking a standard VC generalization bound we get
$$
\mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ L ^ {0 - 1} (N _ {\boldsymbol {\theta}}) \mid n ^ {*} \right] \leq \underbrace {\frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {1} \left\{\mathrm {s i g n} (N _ {\boldsymbol {\theta}} (x _ {i})) \neq y _ {i} \right\}} _ {= 0} + O \left(\sqrt {\frac {n ^ {*}}{m}}\right) = O \left(\sqrt {\frac {n ^ {*}}{m}}\right).
$$
By Jensen's inequality  $\mathbb{E}[\sqrt{n^*} ]\leq \sqrt{\mathbb{E}[n^*]}\lesssim \sqrt{pm}\Rightarrow \mathbb{E}[\sqrt{n^* / m} ]\lesssim \sqrt{p}$ , so by the law of total expectation
$$
\mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ L ^ {0 - 1} (N _ {\boldsymbol {\theta}}) ] = \mathbb {E} _ {n ^ {*}} \left[ \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} [ L ^ {0 - 1} (N _ {\boldsymbol {\theta}}) \mid n ^ {*} ] \right] \lesssim \mathbb {E} _ {n ^ {*}} \left[ \sqrt {\frac {n ^ {*}}{m}} \right] \lesssim \sqrt {p}. \tag {7}
$$
In order to relate the bound above to the clean test error, note that
$$
\begin{array}{l} L ^ {0 - 1} (N _ {\boldsymbol {\theta}}) = \operatorname * {P r} _ {(x, y) \sim \mathcal {D}} [ \operatorname {s i g n} (N _ {\boldsymbol {\theta}} (x)) \neq y ] \\ = (1 - p) \cdot \Pr_ {(x, y) \sim \mathcal {D}} [ N _ {\boldsymbol {\theta}} (x) \leq 0 | y = 1 ] + p \cdot \Pr_ {(x, y) \sim \mathcal {D}} [ N _ {\boldsymbol {\theta}} (x) > 0 | y = - 1 ] \\ \geq \underbrace {(1 - p)} _ {\in [ \frac {1}{2}, 1 ]} \cdot \Pr_ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) \leq 0 \right] \\ \geq \frac {1}{2} \cdot \Pr_ {x \sim \mathcal {D} _ {x}} [ N _ {\boldsymbol {\theta}} (x) \leq 0 ], \\ \end{array}
$$
$$
\mathbb {E} _ {\mathcal {S} _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \Pr_ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) \leq 0 \right] \right] \leq \mathbb {E} _ {\mathcal {S} _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ 2 L ^ {0 - 1} \left(N _ {\boldsymbol {\theta}}\right) \right] \stackrel {{\text {E q .} (7)}} {{\lesssim}} \sqrt {p}.
$$
This can be seen by considering a network representing the linear spline of the data, for which it suffices to set a neuron for adjacent samples with alternating signs.
As in our argument for the lower bound, we now note by Eq. (6) that flipping a single label  $y_{l}$  for some  $l \in [m]$  changes  $n^{*}$  by at most 1, hence changing the test error by at most  $O(1 / m)$ . Therefore we can apply McDiarmid's inequality and see that under the event  $E_{x}$ , with probability at least  $1 - \delta /4$ :
$$
\operatorname * {P r} _ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) \leq 0 \right] \leq C \left(\sqrt {p} + \sqrt {\frac {\log (1 / \delta)}{m}}\right).
$$
Overall, by union bounding over  $E_{x}$  the inequality above holds with probability at least  $1 - \delta /2$ , which proves the upper bound and finishes the proof.
Figure 3: Illustration of the proof of Theorem 3.2 in case there is a single non-linearity along  $[x_i, x_{i+1}]$ . If the network is not linear along  $[x_i, x_{i+1}]$ , one of the cases illustrated in the top row (in blue) must occur. In each case, the dashed green perturbation classifies correctly by altering exactly two neurons while reducing the parameter norm. Moreover, if the network is linear along  $[x_i, x_{i+1}]$ , yet  $N_\theta(x_i) < -1$  or  $N_\theta(x_{i+1}) > 1$ , one of the cases illustrated in the bottom row must occur. In either case, the dashed green perturbation classifies correctly by altering exactly two neurons while reducing the parameter norm.
Throughout the proof, given a sample  $S = (x_i, y_i)_{i=1}^m \sim \mathcal{D}^m$  we denote  $S_x = (x_i)_{i=1}^m$ ,  $S_y = (y_i)_{i=1}^m$ , and assume without loss of generality that  $x_1 \leq x_2 \leq \dots \leq x_m$ . Denote by  $E_x$  the event in which the samples  $(x_i)_{i=1}^m \sim \mathcal{D}_x^m$  satisfy
$$
d _ {\max } := \max  _ {i \in [ m - 1 ]} (x _ {i + 1} - x _ {i}) \leq \frac {\log (8 (m + 1) / \delta)}{m + 1},
$$
and recall that by Lemma A.1 we have  $\operatorname{Pr}_{S_x \sim \mathcal{D}_x^m}[E_x] \geq 1 - \frac{\delta}{4}$  for  $m$  larger than an absolute constant. Therefore from here on throughout the proof we condition on  $E_x$ , after which we can conclude using the union bound.
For any point  $x \in (0,1)$ , denote by  $i_x$  the maximal index  $i \in [m]$  such that  $x_i \leq x$ . Let  $A_x$  denote the event in which  $y_{i_x - 1} = 1$ ,  $y_{i_x} = -1$  and  $y_{i_x + 1} = 1$ . Note that for any  $x$  we have  $\operatorname{Pr}_{S_y \sim \mathcal{D}_y^m}[A_x] = p(1 - p)^2 \geq \frac{1}{4} p$ , so we get
$$
\begin{array}{l} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \Pr_ {x \sim \mathcal {D} _ {x}} [ N _ {\boldsymbol {\theta}} (x) <   0 ] \right] = \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} \left[ \mathbb {1} \left\{N _ {\boldsymbol {\theta}} (x) <   0 \right\} \right] \\ \geq \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} \left[ \mathbb {1} \left\{N _ {\boldsymbol {\theta}} (x) <   0 \right\} \mid A _ {x} \right] \cdot \operatorname * {P r} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} [ A _ {x} ] \\ \geq \frac {p}{4} \cdot \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \Pr_ {x \sim \mathcal {D} _ {x}} [ N _ {\boldsymbol {\theta}} (x) <   0 \mid A _ {x} ] \right]. \tag {8} \\ \end{array}
$$
We aim to show that  $\operatorname{Pr}_{x\sim \mathcal{D}_x}[N_\pmb{\theta}(x) < 0|A_x] = \Omega(1)$ . In order to do so, note that conditioning the uniform measure  $\mathcal{D}_x = \mathrm{Unif}([0,1])$  on the event  $A_x$  results in a uniform measure over the (union of) segments between  $-1$  labeled samples to  $+1$  labeled samples that also satisfy the additional property that the neighboring sample on their left is labeled  $+1$ . We will show that if  $\pmb{\theta}$  is a local minimum of Problem (1), then along any such segment  $N_{\pmb{\theta}}(x)$  is linear from  $-1$  to  $+1$ :
Proposition A.4. Let  $i \in [m - 1]$  be such that  $x_{i - 1} < x_i < x_{i + 1}$  with  $y_{i - 1} = 1$ ,  $y_i = -1$  and  $y_{i + 1} = 1$ . If  $\pmb{\theta}$  is a local minimum of Problem (1), it holds that  $N_{\pmb{\theta}}(x_i) = -1$ ,  $N_{\pmb{\theta}}(x_{i + 1}) = 1$  and  $N_{\pmb{\theta}}(\cdot)$  is linear over  $(x_{i}, x_{i + 1})$ .
In particular, the proposition above shows that
$$
\operatorname * {P r} _ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) <   0 \mid A _ {x} \right] = \frac {1}{2},
$$
which plugged into Eq. (8) gives
$$
\mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \Pr_ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) <   0 \right] \right] \geq \frac {p}{8}.
$$
Moreover, noting that any local minimum is in particular a KKT point, we saw in the proof of Theorem 3.1 that under the event  $E_{x}$ , changing a single label  $y_{l}$ ,  $l \in [m]$  can change the test error by at most  $O(d_{\mathrm{max}}) = O(\log (m / \delta) / m)$ . By McDiarmid's inequality this finishes the proof of the desired lower bound.
As for the upper bound, for  $x \in (0,1)$  we denote by  $B_{x}$  the event in which  $y_{i_x} = 1 = y_{i_x + 1} = 1 -$  namely,  $x$  is between two positively labeled samples. Note that for any  $x$  we have  $\operatorname{Pr}_{S_y \sim \mathcal{D}_y^m}[B_x] = (1 - p)^2 \geq 1 - 2p \Rightarrow \operatorname{Pr}_{S_y \sim \mathcal{D}_y^m}[B_x^c] \leq 2p$ . We will show that if  $\theta$  is a local minimum of the margin maximization problem, then  $B_{x}$  implies that  $N_{\theta}(x) \geq 0$ .
Proposition A.5. Let  $i \in [m - 1]$  be such that  $y_{i} = y_{i + 1} = 1$ , and let  $x \in [x_i, x_{i + 1}]$ . If  $\pmb{\theta}$  is a local minimum of Problem (1), it holds that  $N_{\pmb{\theta}}(x) \geq 0$ .
$$
\mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} [ \mathbb {1} \left\{N _ {\boldsymbol {\theta}} (x) <   0 \right\} | B _ {x} ] = 0,
$$
so we get
$$
\begin{array}{l} \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}} \left[ \Pr_ {x \sim \mathcal {D} _ {x}} \left[ N _ {\boldsymbol {\theta}} (x) <   0 \right] \right] = \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} \left[ \mathbb {1} \left\{N _ {\boldsymbol {\theta}} (x) <   0 \right\} \right] \\ = \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} \left[ \mathbb {1} \left\{N _ {\boldsymbol {\theta}} (x) <   0 \right\} \mid B _ {x} \right] \cdot \Pr_ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} [ B _ {x} ] \\ + \mathbb {E} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} \left[ \mathbb {1} \left\{N _ {\boldsymbol {\theta}} (x) <   0 \right\} \mid B _ {x} ^ {c} \right] \cdot \operatorname * {P r} _ {S _ {y} \sim \mathcal {D} _ {y} ^ {m}, x \sim \mathcal {D} _ {x}} \left[ B _ {x} ^ {c} \right] \\ \leq 0 + 1 \cdot 2 p = 2 p. \\ \end{array}
$$
As in the lower bound proof, recalling that any local minimum is in particular a KKT point, we saw in the proof of Theorem 3.1 that under the event  $E_{x}$ , changing a single label  $y_{l}$ ,  $l \in [m]$  can change the test error by at most  $O(d_{\mathrm{max}}) = O(\log (m / \delta) / m)$ . Hence applying McDiarmid's inequality proves the upper bound thus finishing the proof.
Proof of Proposition A.4. Throughout the proof we fix  $i \in [m]$  for which the conditions described in the proposition hold, and we assume without loss of generality that the neurons are ordered
with respect to their activation point:  $-\frac{b_1}{w_1} \leq -\frac{b_2}{w_2} \leq \dots \leq -\frac{b_n}{w_n}$ . Moreover, we may assume without loss of generality that  $w_1, \ldots, w_n \geq 0$ . Indeed, note that Eq. (1) is invariant under the transformation  $v_j \gets -v_j$ ,  $w_j \gets -w_j$ ,  $b_j \gets -b_j$  which does not affect neither the parameter nor the parameterized network in function space. Hence, any local minimum of Eq. (1) corresponds to a local minimum with  $w_1, \ldots, w_n \geq 0$ . Lastly, we will make frequent use of the following simple observation. For differentiable  $x$  we have
$$
N _ {\boldsymbol {\theta}} ^ {\prime} (x) = \sum_ {j = 1} ^ {n} v _ {j} \cdot \underbrace {w _ {j} \mathbb {1} \left\{w _ {j} x + b _ {j} \geq 0 \right\}} _ {\geq 0}, \tag {9}
$$
so if  $N_{\pmb{\theta}}'(z_1) < N_{\pmb{\theta}}'(z_2)$  for some  $z_1 < z_2$ , then there must exist  $j \in [n]$  with  $v_j > 0$  and  $z_1 < -\frac{b_j}{w_j} < z_2$ . Similarly,  $N_{\pmb{\theta}}'(z_1) > N_{\pmb{\theta}}'(z_2)$  implies that there exists  $j \in [n]$  with  $v_j < 0$ ,  $z_1 < -\frac{b_j}{w_j} < z_2$ .
We split the proof of Proposition A.4 into two lemmas.
Lemma A.6.  $N_{\pmb{\theta}}(\cdot)$  is linear over  $(x_i, x_{i+1})$ .
Proof. Recall that  $N_{\pmb{\theta}}(x_i) \leq -1$  and  $N_{\pmb{\theta}}(x_{i + 1}) \geq 1$ , so  $N_{\pmb{\theta}}$  increases along the segment  $(x_{i}, x_{i + 1})$ . Thus,  $N_{\pmb{\theta}}(\cdot)$  is not linear over  $(x_{i}, x_{i + 1})$  only if (at least) one of the following occurs: (1) There exist  $z_1, z_2 \in (x_i, x_{i + 1})$  such that  $z_1 < z_2$  and  $0 < N_{\pmb{\theta}}'(z_1) < N_{\pmb{\theta}}'(z_2)$ ; (2) there exist  $z_1, z_2 \in (x_i, x_{i + 1})$  such that  $z_1 < z_2$ ,  $N_{\pmb{\theta}}'(z_1) > 0$  and  $N_{\pmb{\theta}}'(z_1) > N_{\pmb{\theta}}'(z_2)$ ; or (3) there exists a single  $z \in (x_i, x_{i + 1})$  at which  $N_{\pmb{\theta}}$  is non-differentiable, such that  $N_{\pmb{\theta}}'|_{(x_i, z)} \leq 0$  and  $N_{\pmb{\theta}}'|_{(z, x_{i + 1})} > 0$ . We will show either of these contradict the assumption that  $\pmb{\theta}$  is a local optimum of the margin maximization problem.
Case (1). The assumption on  $z_{1}, z_{2}$  implies that there exists  $j_{1} \in [n]$  such that  $-\frac{b_{j_1}}{w_{j_1}} \in (z_1, z_2)$  and  $v_{j_1} > 0$ . Let  $j_{2} > j_{1}$  be the minimal index  $j \in \{j_1 + 1, \dots, n\}$  for which  $v_{j} < 0$ . For some small  $\delta > 0$ , consider the perturbed network
$$
\begin{array}{l} N _ {\boldsymbol {\theta} _ {\delta}} (x) := \sum_ {j \in [ n ] \backslash \{j _ {1}, j _ {2} \}} v _ {j} \sigma \left(w _ {j} \cdot x + b _ {j}\right) \tag {10} \\ \left. + (1 - \delta) v _ {j _ {1}} \sigma \left(w _ {j _ {1}} \cdot x + \left(b _ {j _ {1}} - \frac {\delta}{1 - \delta} \left(\frac {w _ {j _ {1}} b _ {j _ {2}}}{w _ {j _ {2}}} - b _ {j _ {1}}\right)\right)\right)\right) \\ + \left(1 + \delta \frac {v _ {j _ {1}} w _ {j _ {1}}}{v _ {j _ {2}} w _ {j _ {2}}}\right) v _ {j _ {2}} \sigma \left(w _ {j _ {2}} \cdot x + b _ {j _ {2}}\right). \\ \end{array}
$$
It is clear that  $\| \pmb{\theta} - \pmb{\theta}_{\delta}\| \stackrel {\delta \to 0}{\longrightarrow}0$ , and we will show that for small enough  $\delta$  the network above still satisfies the margin condition, yet has smaller parameter norm. To see why the margin condition is not violated for small enough  $\delta$ , notice that  $N_{\pmb{\theta}_{\delta}}(x) = N_{\pmb{\theta}}(x)$  for all  $x\notin (x_i, - \frac{b_{j_2}}{w_{j_2}})$  so in particular  $N_{\pmb{\theta}_{\delta}}(x_l) = N_{\pmb{\theta}}(x_l)$  for all  $l\in [i]$ , as well as for all  $x_{l}\geq -\frac{b_{j_{2}}}{w_{j_{2}}}$ . Furthermore, by minimality of  $j_{2}$  we note that there cannot exist  $y_{k} = -1$  for  $k$  such that  $x_{k}\in (-\frac{b_{j_{1}}}{w_{j_{1}}}, - \frac{b_{j_{2}}}{w_{j_{2}}})$ . A direct computation gives that  $N_{\pmb{\theta}_{\delta}}\geq N_{\pmb{\theta}}$  along this segment, so overall the margin condition is indeed satisfied for the
<sup>7</sup>We can assume that such  $j_{2}$  exists, by otherwise discarding  $j_{2}$  in the rest of the proof which would work verbatim. Notably, the only case in which there does not exist such  $j_{2}$  is when  $x_{i}$  is the last sample to be labeled -1.
entire sample. As to the parameter norm, we have
$$
\begin{array}{l} \| \boldsymbol{\theta}_{\delta}\|^{2} = \sum_{j\in [n]\setminus \{j_{1},j_{2}\}}(v_{j}^{2} + w_{j}^{2} + b_{j}^{2}) + (1 - \delta)^{2}v_{j_{1}}^{2} + w_{j_{1}}^{2} + \left(b_{j_{1}} - \frac{\delta}{1 - \delta}\left(\frac{w_{j_{1}}b_{j_{2}}}{w_{j_{2}}} -b_{j_{1}}\right)\right)^{2} \\ + \left(1 + \delta \frac {v _ {j _ {1}} w _ {j _ {1}}}{v _ {j _ {2}} w _ {j _ {2}}}\right) ^ {2} v _ {j _ {2}} ^ {2} + w _ {j _ {2}} ^ {2} + b _ {j _ {2}} ^ {2} \\ = \sum_ {j \in [ n ] \backslash \{j _ {1}, j _ {2} \}} \left(v _ {j} ^ {2} + w _ {j} ^ {2} + b _ {j} ^ {2}\right) + (1 - 2 \delta) v _ {j _ {1}} ^ {2} + w _ {j _ {1}} ^ {2} + b _ {j _ {1}} ^ {2} - \frac {2 \delta}{1 - \delta} b _ {j _ {1}} \left(\frac {w _ {j _ {1}} b _ {j _ {2}}}{w _ {j _ {2}}} - b _ {j _ {1}}\right) \\ + \left(1 + 2 \delta \frac {v _ {j _ {1}} w _ {j _ {1}}}{v _ {j _ {2}} w _ {j _ {2}}}\right) v _ {j _ {2}} ^ {2} + w _ {j _ {2}} ^ {2} + b _ {j _ {2}} ^ {2} + O (\delta^ {2}) \\ = \sum_ {j \in [ n ]} \left(v _ {j} ^ {2} + w _ {j} ^ {2} + b _ {j} ^ {2}\right) - 2 \delta \left(v _ {j _ {1}} ^ {2} + \frac {b _ {j _ {1}}}{1 - \delta} \left(\frac {w _ {j _ {1}} b _ {j _ {2}}}{w _ {j _ {2}}} - b _ {j _ {1}}\right) - \frac {v _ {j _ {1}} w _ {j _ {1}}}{v _ {j _ {2}} w _ {j _ {2}}}\right) + O (\delta^ {2}). \\ \end{array}
$$
$$
\begin{array}{l} \left\| \boldsymbol {\theta} \right\| ^ {2} - \left\| \boldsymbol {\theta} _ {\delta} \right\| ^ {2} = 2 \delta \left(v _ {j _ {1}} ^ {2} - \frac {v _ {j _ {1}} w _ {j _ {1}} v _ {j _ {2}}}{w _ {j _ {2}}} - \frac {b _ {j _ {1}}}{1 - \delta} \left(b _ {j _ {1}} - \frac {w _ {j _ {1}} b _ {j _ {2}}}{w _ {j _ {2}}}\right)\right) + O (\delta^ {2}) \\ \Rightarrow \| \boldsymbol {\theta} \| ^ {2} = \| \boldsymbol {\theta} _ {\delta} \| ^ {2} + 2 \delta \left(v _ {j _ {1}} ^ {2} - \frac {v _ {j _ {1}} w _ {j _ {1}} v _ {j _ {2}}}{w _ {j _ {2}}} - \frac {b _ {j _ {1}}}{1 - \delta} \left(b _ {j _ {1}} - \frac {w _ {j _ {1}} b _ {j _ {2}}}{w _ {j _ {2}}}\right)\right) + O (\delta^ {2}). \tag {11} \\ \end{array}
$$
By construction,  $v_{j_1} > 0$  and  $v_{j_2} < 0$  hence  $-\frac{v_{j_1}w_{j_1}v_{j_2}}{w_{j_2}} > 0$ . Moreover,  $-\frac{b_{j_1}}{w_{j_1}} > 0$  and  $w_{j_1} > 0$  so  $-\frac{b_{j_1}}{1 - \delta} > 0$  for  $\delta < 1$ . Lastly, recall that  $j_2 > j_1 \Rightarrow -\frac{b_{j_2}}{w_{j_2}} \geq -\frac{b_{j_1}}{w_{j_1}} \Rightarrow b_{j_1} - \frac{w_{j_1}b_{j_2}}{w_{j_2}} \geq 0$ . Overall, plugging these into Eq. (11) shows that  $\| \pmb{\theta} \|^2 > \| \pmb{\theta}_\delta \|^2$  for small enough  $\delta$ , contradicting the assumption the  $\pmb{\theta}$  is a local minimum.
Case (2). Since  $N_{\pmb{\theta}}(x_{i-1}) \geq 1$  and  $N_{\pmb{\theta}}(x_i) \leq -1$ ,  $N_{\pmb{\theta}}'$  must be negative somewhere along the segment  $(x_{i-1}, x_i)$ . Yet,  $N_{\pmb{\theta}}'(z_1) > 0$  so there must exist  $j_1 \in [n]$  such that  $-\frac{b_{j_1}}{w_{j_1}} \in (x_{i-1}, z_1)$  and  $v_{j_1} > 0$ . Moreover, the assumption on  $z_1, z_2$  implies that there exists  $j_2 \in [n]$  such that  $-\frac{b_{j_2}}{w_{j_2}} \in (z_1, z_2)$  and  $v_{j_2} < 0$ . For some small  $\delta > 0$ , consider the perturbed network
$$
\begin{array}{l} N _ {\boldsymbol {\theta} _ {\delta}} (x) := \sum_ {j \in [ n ] \backslash \{j _ {1}, j _ {2} \}} v _ {j} \sigma \left(w _ {j} \cdot x + b _ {j}\right) \tag {12} \\ + (1 - \delta) v _ {j _ {1}} \sigma \left(w _ {j _ {1}} \cdot x + b _ {j _ {1}}\right) + v _ {j _ {2}} \sigma \left(\left(w _ {j _ {2}} + \frac {\delta v _ {j _ {1}} w _ {j _ {1}}}{v _ {j _ {2}}}\right) \cdot x + \left(b _ {j _ {2}} + \frac {\delta v _ {j _ {1}} b _ {j _ {1}}}{v _ {j _ {2}}}\right)\right). \\ \end{array}
$$