SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Denote  $m' = \lfloor m / 6 \rfloor$ . For any  $i \in \{0, \dots, n - 1\}$ , the set  $\mathcal{A}_i = \{j \in \{1, \dots, m'\} \mid u_j \in [v_i, v_{i + 1}]\}$  is empty with probability  $(1 - (v_{i + 1} - v_i))^{m'} \leqslant (1 - \Delta v)^{m'}$ . Thus by the union bound, the probability that at least one of  $\mathcal{A}_1, \dots, \mathcal{A}_n$  is empty is upper bounded by  $n(1 - \Delta v)^{m'}$ .
We now check that  $n(1 - \Delta v)^{m^{\prime}}\leqslant \delta /18$  . Indeed,
$$
m ^ {\prime} = \left\lfloor \frac {m}{6} \right\rfloor \geqslant \frac {m}{6} - 1 \geqslant \frac {3 + \log n + \log \frac {1}{\delta}}{\Delta v} \geqslant \frac {\log n + \log \frac {1 8}{\delta}}{\Delta v} \geqslant - \frac {\log n + \log \frac {1 8}{\delta}}{\log (1 - \Delta v)},
$$
where we use  $\Delta v \leqslant 1, 3 \geqslant \log(18)$ , and  $\log(1 - \Delta v) \leqslant -\Delta v < 0$ . This gives the desired inequality.
In other words, the probability that at least one of the intervals  $[v_i, v_{i+1}]$  contains none of the  $u_1, \ldots, u_{m'}$  is bounded by  $\delta / 18$ . As a consequence, by the union bound, the probability that at least one of the intervals  $[v_i, v_{i+1}]$  contains strictly less than 6 of the  $u_1, \ldots, u_m$  is bounded by  $\delta / 3$ , i.e.,  $\mathbb{P}(A^c) \leqslant \delta / 3$ .
(b) Recall that by convention,  $u_0 = -\frac{\eta}{2}$  and  $u_{m + 1} = 1 + \frac{\eta}{2}$ . For all  $i \in \{0, \dots, m + 1\}$ , denote  $I_i = (u_i - D, u_i + D)$ . Denote  $F_j$  the event "  $u_j \in I_i$  for some  $i \in \{0, \dots, m + 1\}$ ,  $i \neq j$ ". Note that  $B^c = \cup_{j=1}^{m} F_j$ .
Fix  $j = 1, \ldots, m$ . By conditioning on  $u_i$  for all  $i \in \{0, \ldots, m + 1\}$ ,  $i \neq j$ , we see that  $\mathbb{P}(F_j) \leqslant 2(m + 1)D$ . By the union bound,
$$
\mathbb {P} \left(B ^ {c}\right) \leqslant 2 m (m + 1) D \leqslant \frac {\delta}{3}.
$$
(c) Take  $i \in \{1, \dots, n - 1\}$ . For convenience, we define the random variable  $u_i^{\mathrm{L}}$  (resp.  $u_i^{\mathrm{R}}$ ) on the full probability space by setting  $u_i^{\mathrm{L}} = 0$  (resp.  $u_i^{\mathrm{R}} = 1$ ) when there is no neuron on the left (resp. the right) of  $v_i$ . We compute the joint cumulative distribution function of  $(u_i^{\mathrm{L}}, u_i^{\mathrm{R}})$  (with a convenient change of inequality): for all  $0 \leqslant y \leqslant v_i \leqslant z \leqslant 1$ ,
$$
\mathbb {P} \left(u _ {i} ^ {\mathrm {L}} \leqslant y, u _ {i} ^ {\mathrm {R}} \geqslant z\right) = \mathbb {P} (\forall j \in \{1, \dots , m \}, u _ {j} \notin [ y, z ]) = (1 - (z - y)) ^ {m}.
$$
We observe that the joint cumulative distribution function of  $(u_i^{\mathrm{L}}, u_i^{\mathrm{R}})$  is a smooth function of  $(y, z)$  when  $(y, z) \in (0, v_i) \times (v_i, 1)$ . Note that the events  $E_i$  and  $\{(u_i^{\mathrm{L}}, u_i^{\mathrm{R}}) \in (0, v_i) \times (v_i, 1)\}$  are equal up to a null set. Therefore, on this event,  $(u_i^{\mathrm{L}}, u_i^{\mathrm{R}})$  is an absolutely continuous random variable with density  $g: (0, v_i) \times (v_i, 1) \to \mathbb{R}$ ,
$$
g (y, z) = - \frac {\partial^ {2}}{\partial y \partial z} \mathbb {P} (u _ {i} ^ {\mathrm {L}} \leqslant y, u _ {i} ^ {\mathrm {R}} \geqslant z) = m (m - 1) \left(1 - (z - y)\right) ^ {m - 2}.
$$
$$
\begin{array}{l} \mathbb {P} \left(C _ {i} ^ {c} \cap E _ {i}\right) = \mathbb {P} \left(\left\{\left| u _ {i} ^ {\mathrm {R}} + u _ {i} ^ {\mathrm {L}} - 2 v _ {i} \right| \leqslant D \right\} \cap E _ {i}\right) \\ = \int_ {\left\{0 <   y <   v _ {i} <   z <   1 \right\}} m (m - 1) \left(1 - (z - y)\right) ^ {m - 2} \mathbf {1} _ {\left\{| y + z - 2 v _ {i} | \leqslant D \right\}} d y d z. \\ \end{array}
$$
We make the change of variables  $\theta = z - y$ ,  $\nu = z + y$ .
$$
\begin{array}{l} \mathbb {P} \left(C _ {i} ^ {c} \cap E _ {i}\right) = \frac {m (m - 1)}{2} \int_ {\left\{0 <   \frac {\nu - \theta}{2} <   v _ {i} <   \frac {\nu + \theta}{2} <   1 \right\}} \left(1 - \theta\right) ^ {m - 2} \mathbf {1} _ {| \nu - 2 v _ {i} | \leqslant D} d \theta d \nu \\ \leqslant \frac {m (m - 1)}{2} \left(\int_ {0} ^ {1} (1 - \theta) ^ {m - 2} \mathrm {d} \theta\right) \left(\int_ {- \infty} ^ {\infty} \mathbf {1} _ {| \nu - 2 v _ {i} | \leqslant D} \mathrm {d} \nu\right) \\ = D m. \\ \end{array}
$$
Using  $m \geqslant 24 / \Delta v \geqslant 24n$ , we have
$$
\sum_ {i = 1} ^ {n - 1} \mathbb {P} \left(C _ {i} ^ {c} \cap E _ {i}\right) \leqslant (n - 1) D m \leqslant \frac {\delta}{2 4 \times 6} \leqslant \frac {\delta}{3}.
$$
B Proofs of the results of the main text
Let us lower-bound the smallest eigenvalue of  $H_{\eta}(u)$  which is equal to
$$
\min_{\| a\| = 1}a^{\top}H_{\eta}(u)a  .
$$
Now for  $a \in \mathbb{R}^{m+1}$  such that  $\|a\| = 1$ ,
$$
a ^ {\top} H _ {\eta} (u) a = \sum_ {i, j = 0} ^ {m} a _ {i} a _ {j} \int_ {0} ^ {1} \sigma_ {\eta} (x - u _ {i}) \sigma_ {\eta} (x - u _ {j}) \mathrm {d} x = \int_ {0} ^ {1} \left(\sum_ {i = 0} ^ {m} a _ {i} \sigma_ {\eta} (x - u _ {i})\right) ^ {2} \mathrm {d} x.
$$
Since  $\Delta u > 2\eta$  (because  $u \in \mathcal{U}$ ) and  $u_0 = -\eta/2$ ,  $u_{m+1} = 1 + \eta/2$ , the intervals  $[u_i + \eta/2, u_{i+1} - \eta/2]$  for  $i \in \{0, \dots, m\}$  are disjoint and included in  $[0,1]$ . Thus
$$
a ^ {\top} H _ {\eta} (u) a \geqslant \sum_ {i = 0} ^ {m} \int_ {u _ {i} + \eta / 2} ^ {u _ {i + 1} - \eta / 2} \left(\sum_ {i = 0} ^ {m} a _ {i} \sigma_ {\eta} (x - u _ {i})\right) ^ {2} \mathrm {d} x.
$$
Since  $\sigma(x) = 0$  if  $x < -1/2$  and  $\sigma(x) = 1$  if  $x > 1/2$ , we have that  $\sigma_{\eta}(x) = 0$  if  $x < -\eta/2$  and  $\sigma_{\eta}(x) = 1$  if  $x > \eta/2$ . Further recall that the  $u_i$  are ordered in increasing order. As a consequence,
$$
\begin{array}{l} a ^ {\top} H _ {\eta} (u) a \geqslant \sum_ {i = 0} ^ {m} \int_ {u _ {i} + \eta / 2} ^ {u _ {i + 1} - \eta / 2} \left(\sum_ {k = 0} ^ {i} a _ {k}\right) ^ {2} d x \\ = \sum_ {i = 0} ^ {m} \left(u _ {i + 1} - u _ {i} - \eta\right) \left(\sum_ {k = 0} ^ {i} a _ {k}\right) ^ {2} \\ \geqslant \frac {\Delta (u)}{2} \sum_ {i = 0} ^ {m} \left(\sum_ {k = 0} ^ {i} a _ {k}\right) ^ {2}, \tag {12} \\ \end{array}
$$
where in the last step, we used that  $\Delta(u) > 2\eta$  and thus  $u_{i+1} - u_i - \eta \geqslant \Delta(u) - \eta \geqslant \Delta(u) - \Delta(u)/2 = \Delta(u)/2$ . Now, denote  $c_0 = 0$  and  $c_i = \sum_{k=0}^{i-1} a_k$ . Then  $\|a\| = 1$  writes
$$
\sum_ {i = 0} ^ {m} \left(c _ {i + 1} - c _ {i}\right) ^ {2} = 1.
$$
$$
\sum_ {i = 0} ^ {m} (c _ {i + 1} - c _ {i}) ^ {2} = \sum_ {i = 0} ^ {m} c _ {i + 1} ^ {2} + \sum_ {i = 0} ^ {m} c _ {i} ^ {2} - 2 \sum_ {i = 0} ^ {m} c _ {i + 1} c _ {i} \leqslant 4 \sum_ {i = 0} ^ {m + 1} c _ {i} ^ {2}.
$$
$$
\sum_ {i = 0} ^ {m + 1} c _ {i} ^ {2} \geqslant \frac {1}{4},
$$
which shows in conjunction with (12) that the smallest eigenvalue of  $H_{\eta}(u)$  is lower-bounded by  $\frac{\Delta u}{8}$ .
To show that  $G(u) = (\nabla_u L_\eta)(a_\eta^*(u), u)$  is Lipschitz-continuous on  $\mathcal{U}_\eta$ , we show that it is differentiable on  $\mathcal{U}_\eta$  and that its derivatives are uniformly bounded. The chain rule gives
$$
\frac {\partial G _ {j}}{\partial u _ {k}} = \sum_ {l = 0} ^ {m} \frac {\partial a _ {\eta , l} ^ {*}}{\partial u _ {k}} (u) \frac {\partial^ {2} L _ {\eta}}{\partial u _ {j} \partial a _ {l}} (a _ {\eta} ^ {*} (u), u) + \frac {\partial^ {2} L _ {\eta}}{\partial u _ {j} \partial u _ {k}} (a _ {\eta} ^ {*} (u), u).
$$
From (10), using that  $\sigma$  is twice continuously differentiable, it can be checked that  $\frac{\partial L_{\eta}}{\partial u_j}$  is differentiable in both its arguments and its derivatives are uniformly upper-bounded when  $a$  is bounded. Furthermore, for  $u \in \mathcal{U}_{\eta}$ ,
$$
\left\| a _ {\eta} ^ {*} (u) \right\| \leqslant \left\| H _ {\eta} (u) ^ {- 1} \right\| \left\| b _ {\eta} (u) \right\| \leqslant \frac {8 M \sqrt {m + 1}}{\Delta (u)},
$$
by Lemma 1 and Proposition 1. Finally, according to Lemma 3,  $a_{\eta}^{*}$  is differentiable with derivatives uniformly upper-bounded on  $\mathcal{U}_{\eta}$ . This concludes the proof.
In this proof, we denote  $u_{i}^{\mathrm{L}}(\tau)$  (resp.  $u_{i}^{\mathrm{R}}(\tau)$ ) the position at time  $\tau$  of the neuron that is at initialization closest to  $v_{i}$  to the left (resp. the right). Note that because of the movement of the neurons, it could be that  $u_{i}^{\mathrm{L}}$  (resp.  $u_{i}^{\mathrm{R}}$ ) does not remain the neuron closest to the left (resp. the right) throughout the dynamics. Our proof discusses when this phenomenon occurs. Similarly, denote  $u_{i}^{\mathrm{LL}}$  (resp.  $u_{i}^{\mathrm{RR}}$ ) the neuron second closest to the left (resp. the right) of  $v_{i}$ . Since the initialization is  $D$ -good, note that all these neurons are distinct.
Denote  $\overline{T}$  the minimal time  $\tau \in [0, T_{\max})$  such that  $\Delta(u(\tau)) \leqslant D/2$  or there are less than two neurons in some piece  $[v_i, v_{i+1}]$  of  $f^*$ . Note that by assumption,  $\Delta(u(0)) \geqslant D > D/2$  and there are at least 6 neurons in each interval at initialization, thus  $\overline{T} > 0$ . Furthermore, using the trivial inequalities  $M \geqslant \Delta f/2$ ,  $m + 1 \geqslant 1$  and  $\eta^{1/2} \geqslant \eta$ , we have  $\frac{D}{2} = \frac{2^{11/2} M \sqrt{m + 1} \sqrt{\eta}}{\Delta f} \geqslant 8\eta > 2\eta$ . Recall that  $2\eta$  is the quantity defining the set  $\mathcal{U}_{\eta}$  supporting the maximal solution of the equation (8). As a consequence, we do have  $\overline{T} < T_{\max}$ . At the end of the proof, we check that  $\mathcal{T} < \overline{T}$ , by controlling carefully the movement of each neuron.
Let us first bound the difference between the dynamics of  $u$  and the dynamics that we would have if at each time  $\tau$ , the weights  $a$  were given by  $a_0^*(u(\tau))$ , the best approximation of  $f^*$  by a piecewise constant function with subdivision  $u(\tau)$ . For any  $\tau < \overline{T}$  and  $j \in \{1, \ldots, m\}$ , by Lemma 6, we have
$$
\begin{array}{l} \left| \frac {\mathrm {d} u _ {j}}{\mathrm {d} \tau} (\tau) + \frac {\partial L _ {\eta}}{\partial u _ {j}} (a _ {0} ^ {*} (u (\tau)), u (\tau)) \right| \\ = \left| \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {\eta} ^ {*} (u (\tau)), u (\tau)\right) - \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u (\tau)), u (\tau)\right) \right| \\ \leqslant 2 M (\sqrt {m + 1} + 1) \left\| a _ {\eta} ^ {*} (u (\tau)) - a _ {0} ^ {*} (u (\tau)) \right\| + \sqrt {m + 1} \left\| a _ {\eta} ^ {*} (u (\tau)) - a _ {0} ^ {*} (u (\tau)) \right\| ^ {2}. \tag {13} \\ \end{array}
$$
We are therefore led to bounding  $\| a_{\eta}^{*}(u(\tau)) - a_{0}^{*}(u(\tau)) \|$ , as follows:
$$
\begin{array}{l} \left\| a _ {\eta} ^ {*} (u (\tau)) - a _ {0} ^ {*} (u (\tau)) \right\| \leqslant \frac {2 ^ {4} M \sqrt {m + 1} \eta}{\Delta (u (\tau))} \quad \tag {byLemma7} \\ \leqslant \frac {2 ^ {5} M \sqrt {m + 1} \eta}{D} \quad (\text {s i n c e} \Delta (u (\tau)) \geqslant D / 2) \\ = \frac {D (\Delta f) ^ {2}}{2 ^ {8} M \sqrt {m + 1}} \quad \text {(b y d e f i n i t i o n o f D)}. \\ \end{array}
$$
Then the first term in (13) is less than
$$
\frac {(\sqrt {m + 1} + 1) D (\Delta f) ^ {2}}{2 ^ {7} \sqrt {m + 1}} \leqslant \frac {D (\Delta f) ^ {2}}{2 ^ {6}},
$$
and the second term in (13) is less than
$$
\frac {D ^ {2} (\Delta f) ^ {4}}{2 ^ {1 6} M ^ {2} \sqrt {m + 1}} \leqslant \frac {D (\Delta f) ^ {2}}{2 ^ {1 4}}, \quad \text {u s i n g} D \leqslant \Delta (u (0)) \leqslant 1, \Delta f \leqslant 2 M \text {a n d} m + 1 \geqslant 1.
$$
Hence we obtain, for any  $\tau < \overline{\mathcal{T}}$  and  $j \in \{1, \dots, m\}$ ,
$$
\left| \frac {\mathrm {d} u _ {j}}{\mathrm {d} \tau} (\tau) + \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u (\tau)), u (\tau)\right) \right| \leqslant \frac {D (\Delta f) ^ {2}}{6 0} =: \Delta g \tag {14}
$$
Now, let us examine how the neurons move, by leveraging Lemma 4 that gives exact formulae for  $\frac{\partial L_{\eta}}{\partial u_j}(a_0^*(u(\tau)), u(\tau))$ . First, if  $u_j$  is not next to a discontinuity,  $\frac{\partial L_{\eta}}{\partial u_j}(a_0^*(u(\tau)), u(\tau)) = 0$ , hence
$$
\left| u _ {j} (\tau) - u _ {j} (0) \right| \leqslant (\Delta g) \tau .
$$
Let us now study what happens next to a discontinuity  $v_{i}$ . Denote  $(\delta f)_{i} = f_{i}^{*} - f_{i - 1}^{*}$ . W.l.o.g., assume that
$$
u _ {i} ^ {\mathrm {R}} (0) - v _ {i} > v _ {i} - u _ {i} ^ {\mathrm {L}} (0).
$$
In the reverse case, the proof is the same by swapping the roles of  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$ , and of  $u_{i}^{\mathrm{LL}}$  and  $u_{i}^{\mathrm{RR}}$ . We are going to show that the dynamics of  $u_{i}^{\mathrm{L}}$  are divided into two phases. Define  $\mathcal{T}_{i}$  as the minimal  $\tau \in [0, \overline{T}]$  such that  $u_{i}^{\mathrm{L}}(\tau) = v_{i} - \eta / 2$ . In the first phase  $[0, \mathcal{T}_{i}]$ , we have  $u_{i}^{\mathrm{L}}(\tau) < v_{i} - \frac{\eta}{2}$  and  $u_{i}^{\mathrm{L}}$  moves towards  $v_{i}$ . In the second phase  $[\mathcal{T}_{i}, \overline{\mathcal{T}}]$ , we show below that  $u_{i}^{\mathrm{L}}(\tau) \in [v_{i} - \eta, v_{i} + \eta]$ . Note that we can have  $\mathcal{T}_{i} = 0$  if  $u_{i}^{\mathrm{L}}(0) \geqslant v_{i} - \frac{\eta}{2}$ . It is also possible that  $\mathcal{T}_{i} = \infty$  a priori; this means that the second phase does not exist. We show below that this case does not happen. Figure 7 depicts the two phases.
(a) Phase 1
(b) Phase 2
Figure 7: Dynamics of the neurons next to a discontinuity  $v_{i}$ . In the first phase,  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$  move towards  $v_{i}$ , until the closest neuron (in this case  $u_{i}^{\mathrm{L}}$ ) reaches the interval of size  $\eta$  centered in  $v_{i}$ . In the second phase,  $u_{i}^{\mathrm{L}}$  remains in an interval of size  $2\eta$  around  $v_{i}$ , and none of the neurons move significantly.
Beginning by the first phase, we have, while  $u_{i}^{\mathrm{L}}(\tau) < v_{i} - \frac{\eta}{2}$  and  $u_{i}^{\mathrm{R}}(\tau) > v_{i} + \frac{\eta}{2}$ , according to Lemma 4,
$$
\frac {\partial L _ {\eta}}{\partial u _ {i} ^ {\mathrm {L}}} (a _ {0} ^ {*} (u (\tau)), u (\tau)) = - \frac {1}{2} \frac {(u _ {i} ^ {\mathrm {R}} (\tau) - v _ {i}) ^ {2} (\delta f) _ {i} ^ {2}}{(u _ {i} ^ {\mathrm {R}} (\tau) - u _ {i} ^ {\mathrm {L}} (\tau)) ^ {2}},
$$
$$
\frac {\partial L _ {\eta}}{\partial u _ {i} ^ {\mathrm {R}}} (a _ {0} ^ {*} (u (\tau)), u (\tau)) = \frac {1}{2} \frac {(v _ {i} - u _ {i} ^ {\mathrm {L}} (\tau)) ^ {2} (\delta f) _ {i} ^ {2}}{(u _ {i} ^ {\mathrm {R}} (\tau) - u _ {i} ^ {\mathrm {L}} (\tau)) ^ {2}}.
$$
For ease of computation, let  $d_i^{\mathrm{L}}(\tau) = v_i - u_i^{\mathrm{L}}(\tau)$  and  $d_i^{\mathrm{R}}(\tau) = u_i^{\mathrm{R}}(\tau) - v_i$  be the distances between the neurons and  $v_i$ . Then, by (14),
$$
\begin{array}{l} \frac {\mathrm {d} d _ {i} ^ {\mathrm {R}}}{\mathrm {d} \tau} (\tau) + \frac {\mathrm {d} d _ {i} ^ {\mathrm {L}}}{\mathrm {d} \tau} (\tau) \leqslant - \frac {1}{2} \frac {\left(\left(d _ {i} ^ {\mathrm {R}} (\tau)\right) ^ {2} + \left(d _ {i} ^ {\mathrm {L}} (\tau)\right) ^ {2}\right) (\delta f) _ {i} ^ {2}}{\left(d _ {i} ^ {\mathrm {L}} (\tau) + d _ {i} ^ {\mathrm {R}} (\tau)\right) ^ {2}} + 2 \Delta g \\ \leqslant - \frac {(\Delta f) ^ {2}}{4} + 2 \frac {D (\Delta f) ^ {2}}{6 0} \leqslant - \frac {(\Delta f) ^ {2}}{5} \\ \end{array}
$$
since  $D \leqslant \Delta(u(0)) \leqslant 1$ . Thus, in some time less than  $\mathcal{T} = \frac{6}{(\Delta f)^2}$ ,  $d_i^{\mathrm{R}}(\tau) + d_i^{\mathrm{L}}(\tau) \leqslant \eta$ , that is, either  $u_i^{\mathrm{L}}$  reaches  $v_i - \frac{\eta}{2}$  or  $u_i^{\mathrm{R}}$  reaches  $v_i + \frac{\eta}{2}$ . Let us check that the second event cannot actually
happen: while  $u_{i}^{\mathrm{L}}(\tau) < v_{i} - \frac{\eta}{2}$  and  $u_{i}^{\mathrm{R}}(\tau) > v_{i} + \frac{\eta}{2}$ , we also have
$$
\begin{array}{l} \frac {\mathrm {d} d _ {i} ^ {\mathrm {R}}}{\mathrm {d} \tau} (\tau) - \frac {\mathrm {d} d _ {i} ^ {\mathrm {L}}}{\mathrm {d} \tau} (\tau) \geqslant \frac {\left(\left(d _ {i} ^ {\mathrm {R}} (\tau)\right) ^ {2} - \left(d _ {i} ^ {\mathrm {L}} (\tau)\right) ^ {2}\right) (\delta f) _ {i} ^ {2}}{\left(d _ {i} ^ {\mathrm {L}} (\tau) + d _ {i} ^ {\mathrm {R}} (\tau)\right) ^ {2}} - 2 \Delta g \\ = \frac {\left(d _ {i} ^ {\mathrm {R}} (\tau) - d _ {i} ^ {\mathrm {L}} (\tau)\right) \left(\delta f\right) _ {i} ^ {2}}{d _ {i} ^ {\mathrm {L}} (\tau) + d _ {i} ^ {\mathrm {R}} (\tau)} - 2 \Delta g. \\ \end{array}
$$
By condition (c) of Definition 2 and by (14), we have  $d_i^{\mathrm{R}}(0) - d_i^{\mathrm{L}}(0)\geqslant D = \frac{60\Delta g}{(\Delta f)^2}\geqslant \frac{60\Delta g}{(\delta f)_i^2}$ , and furthermore  $d_{i}^{\mathrm{L}}(\tau) + d_{i}^{\mathrm{R}}(\tau)\leqslant 1$ . An easy reasoning then shows that  $d_i^{\mathrm{R}} - d_i^{\mathrm{L}}$  is increasing. Therefore  $u_{i}^{\mathrm{R}}$  must remain further away from  $v_{i}$  than  $u_{i}^{\mathrm{L}}$ .
In summary, we showed that there exists some time  $\mathcal{T}_i \leqslant \mathcal{T}$  when  $u_i^{\mathrm{L}}(\mathcal{T}_i) = v_i - \frac{\eta}{2}$ , which marks the end of the first phase, and we also have
$$
d _ {i} ^ {\mathrm {R}} \left(\mathcal {T} _ {i}\right) - d _ {i} ^ {\mathrm {L}} \left(\mathcal {T} _ {i}\right) \geqslant d _ {i} ^ {\mathrm {R}} (0) - d _ {i} ^ {\mathrm {L}} (0) \geqslant D.
$$
Moving on to the study of the second phase, let us show that  $u_{i}^{\mathrm{L}}(\tau)$  stays in the interval  $[v_i - \eta ,v_i + \eta ]$  for  $\tau \in [\mathcal{T}_i,\overline{\mathcal{T}})$ . Consider any  $\tau \leqslant \overline{\mathcal{T}}$  such that  $u_{i}^{\mathrm{L}}(\tau) = v_{i} - \eta$ . Then we have by (14) and Lemma 4
$$
\frac {\mathrm {d} u _ {i} ^ {\mathrm {L}}}{\mathrm {d} \tau} (\tau) \geqslant \frac {\left(u _ {i} ^ {\mathrm {R}} (\tau) - v _ {i}\right) ^ {2} (\delta f) _ {i} ^ {2}}{\left(u _ {i} ^ {\mathrm {R}} (\tau) - v _ {i} + \eta\right) ^ {2}} - \Delta g \geqslant \Delta g, \tag {15}
$$
where the second upper bound comes from the fact that we have  $u_{i}^{\mathrm{R}}(\tau) - v_{i} \geqslant \frac{D}{2} - \eta$  since  $\Delta(u(\tau)) \geqslant D/2$ , and furthermore,  $x \mapsto \frac{x^{2}}{(x + \eta)^{2}}$  is increasing, hence
$$
\frac {(u _ {i} ^ {\mathrm {R}} - v _ {i}) ^ {2} (\delta f) _ {i} ^ {2}}{(u _ {i} ^ {\mathrm {R}} - v _ {i} + \eta) ^ {2}} \geqslant \left(\frac {\frac {D}{2} - \eta}{\frac {D}{2}}\right) ^ {2} \Delta f ^ {2} \underset {(D / 2 \geqslant 2 \eta)} {\geqslant} \frac {(\Delta f) ^ {2}}{4} \geqslant 2 \Delta g.
$$
Equation (15) implies that  $u_{i}^{\mathrm{L}}(\tau) \geqslant v_{i} - \eta$  for all  $\tau \in [\mathcal{T}_i, \overline{\mathcal{T}})$ . Similarly, consider any  $\tau \leqslant \overline{\mathcal{T}}$  such that  $u_{i}^{\mathrm{L}}(\tau) = v_{i} + \eta$ . Note that, for such a  $\tau$ ,  $u_{i}^{\mathrm{L}}(\tau)$  is now on the right of  $v_{i}$ , and the neurons flanking  $v_{i}$  are  $u_{i}^{\mathrm{LL}}$  and  $u_{i}^{\mathrm{L}}$ . Thus we have by (14) and Lemma 4
$$
\frac {\mathrm {d} u _ {i} ^ {\mathrm {L}}}{\mathrm {d} \tau} (\tau) \leqslant - \frac {(v _ {i} - u _ {i} ^ {\mathrm {L L}} (\tau)) ^ {2} (\delta f) _ {i} ^ {2}}{(v _ {i} + \eta - u _ {i} ^ {\mathrm {L L}} (\tau)) ^ {2}} + \Delta g \leqslant - \Delta g,
$$
where the second lower bound unfolds similarly as previously. This shows that  $u_i^{\mathrm{L}}(\tau) \leqslant v_i + \eta$  for all  $\tau \in [\mathcal{T}_i, \overline{\mathcal{T}})$ .
We now check that  $\mathcal{T} < \overline{\mathcal{T}}$ , that is, for all  $\tau \leqslant \mathcal{T}$ ,  $\Delta(u(\tau)) > D/2$  and there are at least two neurons in each interval  $[v_i, v_{i+1}]$ . Starting with the first condition, we say that neurons  $u_j$  and  $u_k$  collide if  $|u_j(\tau) - u_k(\tau)| = D/2$  for some  $\tau \leqslant \mathcal{T}$ . Let us show that no pair of neurons collide.
We start by showing that there is no collision between  $u_{i}^{\mathrm{LL}}$  and  $u_{i}^{\mathrm{L}}$ . In the first phase  $[0, \mathcal{T}_i]$ , we have  $\frac{\mathrm{d}u_i^{\mathrm{LL}}}{\mathrm{d}\tau}(\tau) \leqslant \Delta g$ . Recall that we also have  $\frac{\mathrm{d}u_i^{\mathrm{L}}}{\mathrm{d}\tau}(\tau) \geqslant -\Delta g$  and thus for  $\tau \leqslant \mathcal{T}_i$ ,
$$
u _ {i} ^ {\mathrm {L}} (\tau) - u _ {i} ^ {\mathrm {L L}} (\tau) \geqslant u _ {i} ^ {\mathrm {L}} (0) - u _ {i} ^ {\mathrm {L L}} (0) - 2 \mathcal {T} \Delta g \geqslant \frac {4 D}{5}
$$
since  $u_{i}^{\mathrm{L}}(0) - u_{i}^{\mathrm{LL}}(0) \geqslant D$  and  $\mathcal{T}\Delta g = D / 10$  by definition of  $\mathcal{T}$  and  $\Delta g$ . As a consequence,  $u_{i}^{\mathrm{LL}}$  and  $u_{i}^{\mathrm{L}}$  do not collide during the first phase, and we have
$$
u _ {i} ^ {\mathrm {L L}} \left(\mathcal {T} _ {i}\right) \leqslant u _ {i} ^ {\mathrm {L}} \left(\mathcal {T} _ {i}\right) - \frac {4 D}{5} = v _ {i} - \frac {\eta}{2} - \frac {4 D}{5}. \tag {16}
$$
In the second phase, we can have  $u_{i}^{\mathrm{L}} \in [v_{i}, v_{i} + \eta]$  in which case  $u_{i}^{\mathrm{LL}}$  becomes the neuron flanking  $v_{i}$  to the left and  $u_{i}^{\mathrm{L}}$  the neuron flanking to the right. Then (14) and Lemma 4 give
$$
\frac {\mathrm {d} u _ {i} ^ {\mathrm {L L}}}{\mathrm {d} \tau} \leqslant \frac {(u _ {i} ^ {\mathrm {L}} (\tau) - v _ {i}) ^ {2} (\delta f) _ {i} ^ {2}}{(u _ {i} ^ {\mathrm {L}} (\tau) - u _ {i} ^ {\mathrm {L L}} (\tau)) ^ {2}} + \Delta g \leqslant \frac {1 6 \eta^ {2} M ^ {2}}{D ^ {2}} + \Delta g.
$$
Of course, this bound also holds when  $u_{i}^{\mathrm{L}} \in [v_{i} - \eta, v_{i}]$ , because then  $\frac{\mathrm{d}u_{i}^{\mathrm{LL}}}{\mathrm{d}\tau} \leqslant \Delta g$ . Thus, in the second phase  $\tau \in [\mathcal{T}_i,\mathcal{T}]$ , by the previous upperbound and the fact that  $u_{i}^{\mathrm{L}}(\tau) \geqslant v_{i} - \frac{\eta}{2}$ ,
$$
\begin{array}{l} u _ {i} ^ {\mathrm {L}} (\tau) - u _ {i} ^ {\mathrm {L L}} (\tau) \geqslant v _ {i} - \frac {\eta}{2} - \left(u _ {i} ^ {\mathrm {L L}} \left(\mathcal {T} _ {i}\right) + (\tau - \mathcal {T} _ {i}) \left(\frac {1 6 \eta^ {2} M ^ {2}}{D ^ {2}} + \Delta g\right)\right) \\ \geqslant \frac {4 D}{5} - \mathcal {T} \left(\frac {1 6 \eta^ {2} M ^ {2}}{D ^ {2}} + \Delta g\right), \\ \end{array}
$$
by (16). Let us now upper-bound each of the last two terms by  $\frac{D}{10}$  to conclude. By definition of  $D$ ,
$$
\eta = \frac {(\Delta f) ^ {2} D ^ {2}}{2 ^ {1 3} (m + 1) M ^ {2}}.
$$
$$
\frac {1 6 \eta^ {2} M ^ {2} \mathcal {T}}{D ^ {2}} = \frac {3 (\Delta f) ^ {2} D ^ {2}}{2 ^ {2 1} (m + 1) ^ {2} M ^ {2}} \leqslant \frac {D}{1 0}
$$
using the definition of  $\mathcal{T}$ ,  $D \leqslant \Delta(u(0)) \leqslant 1$ ,  $\Delta f \leqslant 2M$  and  $m + 1 \geqslant 1$ . Finally,  $\mathcal{T} \Delta g = D / 10$ . Thus  $u_i^{\mathrm{LL}}$  and  $u_i^{\mathrm{L}}$  do not collide.
We now show that  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$  do not collide. In the first phase  $\tau \in [0, \mathcal{T}_i^-]$ , we have
$$
u _ {i} ^ {\mathrm {R}} (\tau) - u _ {i} ^ {\mathrm {L}} (\tau) \geqslant u _ {i} ^ {\mathrm {R}} (\tau) - v _ {i} = d _ {i} ^ {\mathrm {R}} (\tau) \geqslant d _ {i} ^ {\mathrm {R}} (\tau) - d _ {i} ^ {\mathrm {L}} (\tau) \geqslant D.
$$
As a consequence,  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$  do not collide during the first phase, and we have
$$
u _ {i} ^ {\mathrm {R}} \left(\mathcal {T} _ {i}\right) \geqslant D + u _ {i} ^ {\mathrm {L}} \left(\mathcal {T} _ {i}\right) = D + v _ {i} - \frac {\eta}{2}. \tag {17}
$$
In the second phase,  $u_{i}^{\mathrm{R}}$  plays a role symmetric to  $u_{i}^{\mathrm{LL}}$ : it can be, or not, the neuron closest to the right of  $v_{i}$ , depending on whether  $u_{i}^{\mathrm{L}} \in [v_{i} - \eta, v_{i}]$  or  $u_{i}^{\mathrm{L}} \in [v_{i}, v_{i} + \eta]$ . As for  $u_{i}^{\mathrm{LL}}$ , we can show that in any case, for  $\tau \in [\mathcal{T}_i, \mathcal{T}]$ ,
$$
\frac {\mathrm {d} u _ {i} ^ {\mathrm {R}}}{\mathrm {d} \tau} \geqslant - \frac {1 6 \eta^ {2} M ^ {2}}{D ^ {2}} - \Delta g.
$$