SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus one concludes as before: for  $\tau \in [\mathcal{T}_i,\mathcal{T}]$ , by the previous lowerbound and the fact that  $u_i^{\mathrm{L}}(\tau)\leqslant v_i + \frac{\eta}{2}$
$$
u _ {i} ^ {\mathrm {R}} (\tau) - u _ {i} ^ {\mathrm {L}} (\tau) \geqslant u _ {i} ^ {\mathrm {R}} (\mathcal {T} _ {i}) - (\tau - \mathcal {T} _ {i}) \left(\frac {1 6 \eta^ {2} M ^ {2}}{D ^ {2}} + \Delta g\right) - \left(v _ {i} + \frac {\eta}{2}\right).
$$
Then, by (17),
$$
u _ {i} ^ {\mathrm {R}} (\tau) - u _ {i} ^ {\mathrm {L}} (\tau) \geqslant D - \eta - \mathcal {T} \left(\frac {1 6 \eta^ {2} M ^ {2}}{D ^ {2}} + \Delta g\right) > \frac {D}{2},
$$
where the last lower-bound unfolds similarly as for  $u_{i}^{\mathrm{LL}}$  and  $u_{i}^{\mathrm{L}}$ . Thus there is no collision between  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$ .
The reader can check that all other pairs of neurons do not collide, including those involving  $u_0 = -\eta / 2$  and  $u_{m+1} = 1 + \eta / 2$ . In fact, the proof is easier than for  $u_i^{\mathrm{LL}}$ ,  $u_i^{\mathrm{L}}$  and  $u_i^{\mathrm{L}}$ ,  $u_i^{\mathrm{R}}$  because the discontinuity at  $v_i$  attracts these neurons together.
Furthermore, we proved that before time  $\mathcal{T}$  at most one neuron can escape on each side of a piece  $[v_i,v_{i + 1}]$  of  $f$ . Since we start with at least four (and even six) neurons per piece, there is always before  $\mathcal{T}$  at least two neurons per piece.
This shows that  $\mathcal{T} < \overline{\mathcal{T}}$ , and we also proved that at time  $\mathcal{T}$ , all discontinuities have finished their first phase, hence there is a neuron at distance less than  $\eta$  from each discontinuity of the target function.
Take  $C = 2^{-19}$ . Then by assumption of Theorem 2,
$$
\eta \leqslant \frac {\delta^ {2} (\Delta f) ^ {2}}{2 ^ {1 9} M ^ {2} (m + 1) ^ {5}}.
$$
Moreover, by the definition of  $D$  from Proposition 3,
$$
\eta = \frac {(\Delta f) ^ {2} D ^ {2}}{2 ^ {1 3} M ^ {2} (m + 1)}.
$$
$$
D ^ {2} \leqslant \frac {\delta^ {2}}{2 ^ {6} (m + 1) ^ {4}},
$$
and in consequence
$$
D \leqslant \frac {\delta}{6 (m + 1) ^ {2}}.
$$
Then Lemma 10 shows that the initialization is  $D$ -good with probability at least  $1 - \delta$  (since the  $D$ -good property is monotonous in  $D$ ).
Hence, with probability at least  $1 - \delta$ , according to Proposition 3, the maximal solution to (8) is defined at least until  $\mathcal{T}$  and at that time, there is a neuron at distance less than  $\eta$  from each discontinuity of the target function. Furthermore,  $3\eta \leqslant \frac{1}{m + 1} \leqslant \frac{1}{n} \leqslant \Delta v$ , hence Lemma 9 applies. This implies that
$$
\int_ {0} ^ {1} \left| f ^ {*} (x) - f (x; a ^ {*} (u (\mathcal {T})), u (\mathcal {T})) \right| ^ {2} \mathrm {d} x \leqslant 6 M ^ {2} \eta n.
$$
The assumption on  $\eta$  allows to conclude that the upper-bound is less than  $\xi$ .
Remark 2. We did not try to optimize the value of  $C$  since our goal was to show convergence to a global optimum and the dependency of the dynamics on the parameters (for instance, it is remarkable that  $\mathcal{T}$  does not depend on  $\xi$ ).
B.5 Proof of Proposition 4
For  $s \leqslant t$ , Proposition 1 holds since for  $\Delta(u(s)) \geqslant 16\eta > 2\eta$ . Thus  $a_{\eta}^{*}(u(s))$  is well-defined and verifies
$$
\nabla_ {a} L _ {\eta} \left(a _ {\eta} ^ {*} (u (s)), u (s)\right) = 0.
$$
Let, for  $s \leqslant t$ ,  $V(s) = \| a(s) - a_{\eta}^{*}(u(s))\|$ . Recall that, by (11),
$$
\nabla_ {a} L _ {\eta} (a, u) = H _ {\eta} (u) a - b _ {\eta} (u).
$$
Hence, for  $s \leqslant t$ ,
$$
\begin{array}{l} \langle a (s) - a _ {\eta} ^ {*} (u (s)), \nabla_ {a} L _ {\eta} (a (s), u (s)) \rangle \\ = \left\langle a (s) - a _ {\eta} ^ {*} (u (s)), \nabla_ {a} L _ {\eta} (a (s), u (s)) - \nabla_ {a} L _ {\eta} \left(a _ {\eta} ^ {*} (u (s)), u (s)\right) \right\rangle \\ = \langle a (s) - a _ {\eta} ^ {*} (u (s)), H _ {\eta} (u (s)) (a (s) - a _ {\eta} ^ {*} (u (s))) \rangle \\ \geqslant \frac {\Delta (u (s))}{8} V (s) ^ {2} \\ \geqslant \frac {D}{1 6} V (s) ^ {2}, \\ \end{array}
$$
where the first lower bound is a consequence of Proposition 1. Then we have, for any  $s \leqslant t$ ,
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} s} \left(\frac {1}{2} V (s) ^ {2}\right) = \left\langle a (s) - a _ {\eta} ^ {*} (u (s)), \frac {\mathrm {d} a}{\mathrm {d} s} (s) - \frac {\mathrm {d}}{\mathrm {d} s} a _ {\eta} ^ {*} (u (s)) \right\rangle \\ = \left\langle a (s) - a _ {\eta} ^ {*} (u (s)), - \nabla_ {a} L _ {\eta} (a (s), u (s)) - \frac {\mathrm {d}}{\mathrm {d} s} a _ {\eta} ^ {*} (u (s)) \right\rangle \\ \leqslant - \frac {D}{1 6} V (s) ^ {2} + \left\| \frac {\mathrm {d}}{\mathrm {d} s} a _ {\eta} ^ {*} (u (s)) \right\| V (s). \\ \end{array}
$$
Let us now upper bound the norm appearing in the second term. We first have by the chain rule
$$
\frac {\mathrm {d}}{\mathrm {d} s} a _ {\eta} ^ {*} (u (s)) = \frac {\partial a _ {\eta} ^ {*}}{\partial u} (u (s)) \frac {\mathrm {d} u}{\mathrm {d} s} (s).
$$
By Lemma 3 (which holds since for  $\Delta(u(s)) \geqslant 16\eta > 2\eta$ ),
$$
\left\| \frac {\partial a _ {\eta} ^ {*}}{\partial u} (u (s)) \right\| \leqslant \frac {8}{\Delta (u (s))} \big (2 (m + 1) \| a _ {\eta} ^ {*} (u (s)) \| + M \big).
$$
$$
\left\| \frac {\mathrm {d} u}{\mathrm {d} s} (s) \right\| \leqslant \varepsilon \| \nabla_ {u} L _ {\eta} (a (s), u (s)) \|.
$$
By Lemma 8,
$$
\left\| \nabla_ {u} L _ {\eta} (a (s), u (s)) \right\| \leqslant \sqrt {m + 1} \| a (s) \| ^ {2} + M \| a (s) \|. \tag {18}
$$
$$
\left\| a (s) \right\| \leqslant \left\| a _ {\eta} ^ {*} (u (s)) \right\| + \left\| a (s) - a _ {\eta} ^ {*} (u (s)) \right\| = \left\| a _ {\eta} ^ {*} (u (s)) \right\| + V (s).
$$
By Lemmas 6 and 7, which apply since  $\Delta(u(s)) > 2\eta$  and since there are at least two positions  $u_j(s)$  in each interval  $[v_i, v_{i+1}]$  for  $s \leqslant t$ ,
$$
\begin{array}{l} \left\| a _ {\eta} ^ {*} (u (s)) \right\| \leqslant \left\| a _ {0} ^ {*} (u (s)) \right\| + \left\| a _ {0} ^ {*} (u (s)) - a _ {\eta} ^ {*} (u (s)) \right\| \\ \leqslant 2 M \sqrt {m + 1} + \frac {1 6 M \sqrt {m + 1} \eta}{\Delta (u (s))} \\ \leqslant 2 M \sqrt {m + 1} + \frac {3 2 M \sqrt {m + 1} \eta}{D} \\ \leqslant 3 M \sqrt {m + 1}, \\ \end{array}
$$
where the last upper bound is implied by the assumption  $D \geqslant 32\eta$ .
Now define  $T_{\mathrm{max}} = \inf \left\{s \geqslant 0, V(s) > 3M\sqrt{m + 1}\right\}$  and assume  $s \leqslant \min(t, T_{\mathrm{max}})$  so that  $V(s) \leqslant 3M\sqrt{m + 1}$ . Then we proved that  $\| a(s) \| \leqslant 6M\sqrt{m + 1}$ . Going back to (18), we deduce that
$$
\left\| \nabla_ {u} L _ {\eta} (a (s), u (s)) \right\| \leqslant 3 6 M ^ {2} (m + 1) ^ {3 / 2} + 6 M ^ {2} \sqrt {m + 1} \leqslant 2 ^ {6} M ^ {2} (m + 1) ^ {3 / 2}. \tag {19}
$$
$$
\left\| \frac {\mathrm {d}}{\mathrm {d} s} a _ {\eta} ^ {*} (u (s)) \right\| \leqslant \frac {2 ^ {9} M ^ {2} (m + 1) ^ {3 / 2}}{\Delta (u (s))} \big (6 M (m + 1) ^ {3 / 2} + M \big) \varepsilon \leqslant \frac {2 ^ {1 3} M ^ {3} (m + 1) ^ {3}}{D} \varepsilon .
$$
All in all,
$$
\frac {\mathrm {d}}{\mathrm {d} s} \Big (\frac {1}{2} V (s) ^ {2} \Big) \leqslant - \frac {D}{1 6} V (s) ^ {2} + \frac {2 ^ {1 3} M ^ {3} (m + 1) ^ {3}}{D} \varepsilon V (s).
$$
$$
\frac {\mathrm {d}}{\mathrm {d} s} (V (s)) = \frac {1}{V (s)} \frac {\mathrm {d}}{\mathrm {d} s} \left(\frac {1}{2} V (s) ^ {2}\right) \leqslant - \frac {D}{1 6} V (s) + \frac {2 ^ {1 3} M ^ {3} (m + 1) ^ {3}}{D} \varepsilon .
$$
By Gronwall's inequality, for all  $s \leqslant \min(t, T_{\max})$
$$
\begin{array}{l} V (s) \leqslant \exp^ {- \frac {D}{1 6} s} V (0) + \frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon \left(1 - \exp^ {- \frac {D}{1 6} s}\right) (20) \\ \leqslant \exp^ {- \frac {D}{1 6} s} V (0) + \frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon . (21) \\ \end{array}
$$
Finally note that  $V(0) = \| a_{\eta}^{*}(0)\| \leqslant 2M\sqrt{m + 1}$  and  $\frac{2^{17}M^3(m + 1)^3\varepsilon}{D^2} \leqslant 2M\sqrt{m + 1}$  by the assumption of the Proposition on  $\varepsilon$ . Hence (20) implies that for all  $s \leqslant \min(t, T_{\max})$ ,  $V(s)$  is a (weighted) average of two terms less than  $2M\sqrt{m + 1}$  hence it is less than  $2M\sqrt{m + 1}$ . This shows that  $T_{\max} \geqslant t$ , which concludes the proof since (21) is then valid for  $s = t$ .
In the proof, we take  $C_1 = 2^{-21}$  and  $C_2 = 2^{-36}$ . Denote
$$
D = \frac {\delta}{6 (m + 1) ^ {2}}.
$$
Lemma 10 shows that the initialization is  $D$ -good with probability at least  $1 - \delta$ . In the following, we study the case where this event happens.
Denote  $\overline{T}$  the minimal time  $t > 0$  such that  $\Delta(u(t)) \leqslant D/2$  or there are less than two neurons in some piece  $[v_i, v_{i+1}]$  of  $f^*$  or  $\|a(t)\| > 7M\sqrt{m + 1}$ . Note that  $\overline{T} > 0$  since the initialization is  $D$ -good. By Lemma 8,  $\nabla_u L_\eta$  and  $\nabla_a L_\eta$  are Lipschitz-continuous on compacts, hence the solution of the gradient flow is well defined for  $t < \overline{T}$  since  $\overline{T}$  defines a compact set of parameters.
Then all the assumptions of Proposition 4 are satisfied on the time interval  $[0, t]$  for any  $t < \overline{T}$ . More precisely, the assumptions that do not come directly from the definition of  $\overline{T}$  are the lower bound for  $D$  and the upper bound for  $\varepsilon$ . The lower bound for  $D$  comes from
$$
D = \frac {\delta}{6 (m + 1) ^ {2}} \geqslant 3 2 \eta \tag {22}
$$
by (3) and the simple bounds  $\delta \leqslant 1, \Delta f \leqslant 2M, m + 1 \geqslant 1$ . The upper bound for  $\varepsilon$  comes from (3) since
$$
\varepsilon \leqslant \frac {\delta^ {3} (\Delta f) ^ {2}}{2 ^ {3 6} M ^ {4} (m + 1) ^ {1 9 / 2}} \leqslant \frac {\delta^ {2}}{3 6 \cdot 2 ^ {1 6} M ^ {2} (m + 1) ^ {1 3 / 2}} = \frac {D ^ {2}}{2 ^ {1 6} M ^ {2} (m + 1) ^ {5 / 2}},
$$
where the second upper bound uses  $m \geqslant 0$ ,  $\delta \leqslant 1$  and  $\Delta f \leqslant 2M$ . Therefore, according to Proposition 4,
$$
\left\| a (t) - a _ {\eta} ^ {*} (u (t)) \right\| \leqslant 3 M \sqrt {m + 1} \exp^ {- \frac {D}{1 6} t} + \frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon , \tag {23}
$$
Furthermore, the proof of Proposition 4 actually implies that
$$
\left\| a _ {\eta} ^ {*} (u (t)) \right\| \leqslant 3 M \sqrt {m + 1} \quad \text {a n d} \quad \| a (t) \| \leqslant 6 M \sqrt {m + 1}. \tag {24}
$$
The second bound implies that the condition  $\| a(t) \| > 7M\sqrt{m + 1}$  in the definition of  $\overline{T}$  is actually never active. Let us distinguish between two phases: letting
$$
T _ {0} = \frac {1 6}{D} \log \left(\frac {2 ^ {1 6} M ^ {2} (m + 1) ^ {3}}{\delta (\Delta f) ^ {2}}\right) = \frac {9 6 (m + 1) ^ {2}}{\delta} \log \left(\frac {2 ^ {1 6} M ^ {2} (m + 1) ^ {3}}{\delta (\Delta f) ^ {2}}\right),
$$
then the first phase corresponds to  $t \leqslant T_0$  and the second phase for  $t \geqslant T_0$ .
Analysis of the first phase. In the first phase, each neuron moves at most by
$$
\varepsilon T _ {0} \max  _ {j} \left| \frac {\partial L _ {\eta}}{\partial u _ {j}} (a (t), u (t)) \right| \leqslant \varepsilon T _ {0} \| \nabla_ {u} L _ {\eta} (a (s), u (s)) \| \leqslant 2 ^ {6} \varepsilon T _ {0} M ^ {2} (m + 1) ^ {3 / 2},
$$
where the second upper bound comes from (19) in the proof of Proposition 4. This quantity is less than  $\frac{D}{8}$  if
$$
\frac {6 1 4 4 (m + 1) ^ {7 / 2} M ^ {2}}{\delta} \log \left(\frac {2 ^ {1 6} M ^ {2} (m + 1) ^ {3}}{\delta (\Delta f) ^ {2}}\right) \varepsilon \leqslant \frac {\delta}{4 8 (m + 1) ^ {2}}.
$$
Let us check this condition: we have
$$
\begin{array}{l} \frac {6 1 4 4 (m + 1) ^ {7 / 2} M ^ {2}}{\delta} \log \left(\frac {2 ^ {1 6} M ^ {2} (m + 1) ^ {3}}{\delta (\Delta f) ^ {2}}\right) \varepsilon \\ = \frac {1 6 \cdot 6 1 4 4 (m + 1) ^ {7 / 2} M ^ {2}}{\delta} \log \left(\frac {2 M ^ {1 / 8} (m + 1) ^ {3 / 1 6}}{\delta^ {1 / 1 6} (\Delta f) ^ {1 / 8}}\right) \varepsilon \\ \leqslant \frac {1 6 \cdot 6 1 4 4 (m + 1) ^ {7 / 2} M ^ {2}}{\delta} \log \left(\frac {4 M (m + 1)}{\delta \Delta f}\right) \varepsilon , \\ \end{array}
$$
since  $m + 1 \geqslant 1$ ,  $\delta \leqslant 1$ , and  $2M / \Delta f \geqslant 1$ , hence  $(2M / \Delta f)^{1/8} \leqslant 2M / \Delta f$ . Next, upper-bounding  $\log(x)$  by  $x$ , we have, by (3),
$$
\begin{array}{l} \frac {7 6 8 (m + 1) ^ {7 / 2} M ^ {2}}{\delta} \log \left(\frac {2 ^ {1 6} M ^ {2} (m + 1) ^ {3}}{\delta (\Delta f) ^ {2}}\right) \varepsilon \leqslant \frac {6 4 \cdot 6 1 4 4 (m + 1) ^ {9 / 2} M ^ {3}}{\delta^ {2} \Delta f} \varepsilon \\ \leqslant \frac {6 1 4 4 \delta (\Delta f)}{2 ^ {2 9} M (m + 1) ^ {5}} \\ \leqslant \frac {\delta}{4 8 (m + 1) ^ {2}} \\ \end{array}
$$
using  $\Delta f \leqslant 2M$  and  $m \geqslant 0$ . Note that the upper bound  $2^6\varepsilon T_0M^2 (m + 1)^{3 / 2} \leqslant D / s$  also implies that
$$
T _ {0} \leqslant \frac {D}{2 ^ {9} \varepsilon M ^ {2} (m + 1) ^ {3 / 2}} \leqslant \frac {1}{2 \varepsilon (\Delta f) ^ {2}} = \frac {T}{1 2} \tag {25}
$$
since  $m \geqslant 0$ ,  $D \leqslant 1$  and  $\Delta f \leqslant 2M$ . Since each neuron moves by at most  $D/8$  in the time interval  $[0, T_0]$  and since  $\Delta(u(0)) \geqslant D$ , we deduce that
$$
\Delta \left(u \left(T _ {0}\right)\right) \geqslant \frac {3}{4} D. \tag {26}
$$
Similarly, by condition (c) of the definition of a  $D$ -good vector, for all discontinuities  $v_{i}$ ,
$$
\left| u _ {i} ^ {\mathrm {R}} (0) + u _ {i} ^ {\mathrm {L}} (0) - 2 v _ {i} \right| \geqslant D,
$$
$$
\left| u _ {i} ^ {\mathrm {R}} \left(T _ {0}\right) + u _ {i} ^ {\mathrm {L}} \left(T _ {0}\right) - 2 v _ {i} \right| \geqslant \frac {3}{4} D. \tag {27}
$$
Furthermore, there at least four neurons on each piece of  $f$  at  $T_0$ , because at most one neuron can move out of each piece by either side between 0 and  $T_0$ .
Analysis of the second phase. Let
$$
\Delta a = \frac {D (\Delta f) ^ {2}}{2 ^ {9} M \sqrt {m + 1}} = \frac {\delta (\Delta f) ^ {2}}{6 \cdot 2 ^ {9} M (m + 1) ^ {5 / 2}}.
$$
In the second phase  $t \geqslant T_0$ , we are able to control by  $\Delta a$  the distance between  $a(t)$  and the weights  $a_0^*(u(t))$  that are the best approximation of  $f^*$  by a piecewise affine function with subdivision  $u(t)$ . To show this, first note that the first term in (23) is smaller than  $\frac{\Delta a}{4}$  when
$$
3 M \sqrt {m + 1} \exp^ {- \frac {D}{1 6} t} \leqslant \frac {\Delta a}{4}.
$$
$$
t \geqslant \log \left(\frac {1 2 M \sqrt {m + 1}}{\Delta a}\right) \frac {1 6}{D}.
$$
which is implied by  $t \geqslant T_0$ . Furthermore, the second term in (23) is smaller than  $\frac{\Delta a}{4}$  because, by definition of  $D$  and by (3),
$$
\frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon = \frac {3 6 \cdot 2 ^ {1 7} M ^ {3} (m + 1) ^ {7}}{\delta^ {2}} \varepsilon \leqslant \frac {6 ^ {2} \delta (\Delta f) ^ {2}}{2 ^ {1 9} M (m + 1) ^ {5 / 2}} = \frac {6 ^ {3} \Delta a}{2 ^ {1 0}} \leqslant \frac {\Delta a}{4}.
$$
Hence, for all  $T_0 \leqslant t < \overline{T}$ ,
$$
\left\| a (t) - a _ {\eta} ^ {*} (u (t)) \right\| \leqslant \frac {\Delta a}{2}.
$$
Furthermore, note that the assumption of Lemma 7 applies for  $t < \overline{T}$  since  $\Delta(u(t)) \geqslant \frac{D}{2} > 2\eta$  by (22). Therefore, by Lemma 7 and by (3),
$$
\begin{array}{l} \left\| a _ {\eta} ^ {*} (u (t)) - a _ {0} ^ {*} (u (t)) \right\| \leqslant \frac {2 ^ {4} M \sqrt {m + 1}}{\Delta (u (t))} \eta \\ \leqslant \frac {2 ^ {5} M \sqrt {m + 1}}{D} \eta \\ = \frac {2 ^ {5} \cdot 6 M (m + 1) ^ {5 / 2}}{\delta} \eta \\ \leqslant \frac {6 \delta (\Delta f) ^ {2}}{2 ^ {1 6} M (m + 1) ^ {5 / 2}} \\ = \frac {6 ^ {2} \Delta a}{2 ^ {7}} \leqslant \frac {\Delta a}{2}. \\ \end{array}
$$
By the triangular inequality, we deduce the upper bound that we were after, that is
$$
\left\| a (t) - a _ {0} ^ {*} (u (t)) \right\| \leqslant \Delta a.
$$
As in the proof of Proposition 3, we can now control the distance between the true dynamics and the one that we would have if the weights were equal to  $a_0^*(u)$ . Namely, for any  $T_0 \leqslant t \leqslant \overline{T}$  and  $j \in \{1, \dots, m\}$ , by Lemma 6 (which applies since  $\Delta(u(t)) > 2\eta$  by (22)), we have
$$
\begin{array}{l} \left| \frac {\mathrm {d} u _ {j}}{\mathrm {d} t} (t) + \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u (t)), u (t)\right) \right| \\ = \left| \frac {\partial L _ {\eta}}{\partial u _ {j}} (a (t), u (t)) - \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u (t)), u (t)\right) \right| \\ \leqslant 2 M (\sqrt {m + 1} + 1) \| a (t) - a _ {0} ^ {*} (u (t)) \| + \sqrt {m + 1} \| a (t) - a _ {0} ^ {*} (u (t)) \| ^ {2}. \\ \end{array}
$$
The first term is less than
$$
2 M (\sqrt {m + 1} + 1) \Delta a = \frac {(\sqrt {m + 1} + 1) D (\Delta f) ^ {2}}{2 ^ {8} \sqrt {m + 1}} \leqslant \frac {D (\Delta f) ^ {2}}{2 ^ {7}},
$$
and the second term is less than
$$
\sqrt {m + 1} (\Delta a) ^ {2} = \frac {D ^ {2} (\Delta f) ^ {4}}{2 ^ {1 8} M ^ {2} \sqrt {m + 1}} \leqslant \frac {D (\Delta f) ^ {2}}{2 ^ {1 6}},
$$
using  $D \leqslant \Delta(u(0)) \leqslant 1$ ,  $\Delta f \leqslant 2M$  and  $m + 1 \geqslant 1$ . Hence we obtain, for any  $T_0 \leqslant t \leqslant \overline{T}$  and  $j \in \{1, \ldots, m\}$ ,
$$
\left| \frac {\mathrm {d} u _ {j}}{\mathrm {d} t} (t) + \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u (t)), u (t)\right) \right| \leqslant \frac {D (\Delta f) ^ {2}}{1 2 0}.
$$
We are therefore in a situation very similar to the proof of Proposition 3, starting from (14). One can check that all the arguments used in the proof also apply here. On top of the estimate above that resembles (14), the crucial facts that make the argument of Proposition 3 work here are the bounds (26) and (27) as well as the fact that there are at least four neurons on each piece  $f$  at  $T_0$ , which together are very similar to the conditions ensuring that  $u(0)$  is  $D$ -good in the proof of Proposition 3. Another key point is (25), ensuring that a time at least equal to  $11T / 12$  remains after the first phase of this proof, which is enough time for the dynamics described in the proof of Proposition 3 to unfold.
This yields that  $T < \overline{T}$ , and that at time  $T$ , there is a neuron at distance less than  $\eta$  from each discontinuity of  $f^*$ . Furthermore,  $3\eta \leqslant \frac{1}{m + 1} \leqslant \frac{1}{n} \leqslant \Delta v$ , hence Lemma 9 applies. Thus
$$
\int_ {0} ^ {1} \left(f _ {\eta} (x; a _ {\eta} ^ {*} (u (T)), u (T)) - f ^ {*} (x)\right) ^ {2} \mathrm {d} x \leqslant 6 M ^ {2} \eta n \leqslant \frac {\xi}{2},
$$
where the second upper bound comes from  $n \leqslant m + 1$  and from (3). Furthermore, by (24) and by Lemma 8,
$$
\begin{array}{l} \left| L _ {\eta} (a (T), u (T)) - L _ {\eta} \left(a _ {\eta} ^ {*} (u (T)), u (T)\right) \right| \leqslant \sqrt {m + 1} \left(6 M (m + 1) + M\right) \| a (T) - a _ {\eta} ^ {*} (u (T)) \| \\ \leqslant 1 6 M (m + 1) ^ {3 / 2} \| a (T) - a _ {\eta} ^ {*} (u (T)) \|. \\ \end{array}
$$
Let us show that this term is less than  $\xi /4$ . Recall that, by (23),
$$
\left\| a (T) - a _ {\eta} ^ {*} (u (T)) \right\| \leqslant 3 M \sqrt {m + 1} \exp^ {- \frac {D}{1 6} T} + \frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon .
$$
By definition of  $D$  and  $T$ , by using  $\exp(-x) \leqslant 1 / x$  for  $x \geqslant 1$  and by (3),
$$
\begin{array}{l} 1 6 M (m + 1) ^ {3 / 2} \cdot 3 M \sqrt {m + 1} \exp^ {- \frac {D}{1 6} T} = 4 8 M ^ {2} (m + 1) ^ {2} \exp \left(- \frac {\delta}{1 6 (m + 1) ^ {2} (\Delta f) ^ {2} \varepsilon}\right) \\ \leqslant \frac {4 8 \cdot 1 6 M ^ {2} (m + 1) ^ {4} (\Delta f) ^ {2}}{\delta} \varepsilon \\ \leqslant \frac {4 8 (\Delta f) ^ {2} \delta}{2 ^ {3 1} M ^ {2} (m + 1) ^ {9 / 2}} \xi \\ \leqslant \frac {\xi}{8} \\ \end{array}
$$
using  $\Delta f\leqslant 2M$ $\delta \leqslant 1$  , and  $m + 1\geqslant 1$  . Furthermore, by (3), we get that
$$
1 6 M (m + 1) ^ {3 / 2} \cdot \frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon = \frac {3 6 \cdot 2 ^ {2 1} M ^ {4} (m + 1) ^ {1 7 / 2}}{\delta^ {2}} \varepsilon \leqslant \frac {\xi}{8}.
$$
We therefore obtain the sought  $\xi /4$  upper-bound and can conclude that
$$
\begin{array}{l} \int_ {0} ^ {1} \left(f _ {\eta} (x; a (T), u (T)) - f ^ {*} (x)\right) ^ {2} \mathrm {d} x \leqslant \int_ {0} ^ {1} \left(f _ {\eta} (x; a _ {\eta} ^ {*} (u (T)), u (T)) - f ^ {*} (x)\right) ^ {2} \mathrm {d} x \\ + 2 \left| L _ {\eta} (a (T), u (T)) - L _ {\eta} \left(a _ {\eta} ^ {*} (u (T)), u (T)\right) \right| \\ \leqslant \xi . \\ \end{array}
$$
C Experimental details and additional experiments
Our code is available at https://github.com/PierreMarion23/two-timescale-nn.
Numerical illustration in the setting of Section 2. To obtain Figures 3 and 4, we use the parameters of Table 1. For Figure 5, we use the parameters of Table 2.
Name	Value
m	20
ε	2·10-5
η	4·10-3
P	1.8·108
h	10-5
Additive noise	Uniform on [−1, 1]
Table 1: Parameters of Figures 3 and 4.
Name	Value
m	20
ε	1
η	4·10-3
P	106
h	10-5
Additive noise	Uniform on [−1, 1]
Table 2: Parameters of Figure 5.
Our target function is defined by  $f^{*} = 1$  on  $[0.,0.2]$ ,  $[0.35,0.5]$ ,  $[0.65,0.8]$ ,  $f^{*} = 2$  on  $[0.5,0.65]$  and  $f^{*} = 4$  elsewhere. The activation function is a piecewise cubic polynomial defined by  $x \mapsto \min(\max(4(x + 0.5)^3, 0), 0.5) + \min(\max(0.5 - 4(0.5 - x)^3, 0), 0.5)$ .
We re-run the same SGD experiment as above twenty times, and plot in Figure 8 the average  $L2$  distance to the target as a function of  $\varepsilon$ , averaging over the initialization randomness and SGD randomness. This confirms that, in our setting, the SGD is able to recover the target function in the two-timescale regime ( $\varepsilon \ll 1$ ), but fails outside of the two-timescale regime ( $\varepsilon = 1$ ). The transition between the two regimes seems to occur for  $\varepsilon \approx 0.1$ .
The number of iterations in Table 1 is much larger than the one in Table 2. There are two levels of analysis to explain this difference. The most straightforward reason is that the positions  $u$  evolve at a speed  $\varepsilon h$ , which is much smaller in Table 1. However, one may note that it is not larger by a
Figure 8:  $L_{2}$  distance with the target as a function of  $\varepsilon$  in the 1D experiment, with 20 repeats
factor  $1 / \varepsilon = 50,000$  but "only" by a factor  $\simeq 200$ . To understand why, we have to get more into the details of the dynamics, to understand the order of magnitude of the number of steps required before convergence. In the two-timescale regime, the limiting factor for convergence is the movement of the positions  $u$ . At each step, they move by an order of  $\varepsilon h\simeq 10^{-10}$ . The positions need to move on a scale of  $5\cdot 10^{-2}$  to align with the discontinuities of the target. Hence the required number of steps is  $0.05 / (2\cdot 10^{-10})\simeq 2\cdot 10^{8}$ . In the standard regime, on the contrary, the limiting factor for convergence is the movement of the weights  $a$ . They move by an order of  $h\simeq 10^{-5}$  at each step, and they need to move by a distance of  $\simeq 5$ , necessitating  $\simeq 5\cdot 10^{5}$  steps to achieve convergence.
Finally, note that it is possible to increase  $h$  in Table 1 while keeping the same behavior (in our experiment,  $h$  is kept to the same value as in Table 2 in order to facilitate the comparison). More precisely, taking  $h = 10^{-3}$  in Table 1 yields similar results while dividing the computational cost by 100.
Higher dimensions. In 2D, we use the parameters of Table 3 and 4. The positions are initialized uniformly over  $[0,1]$  and the weights uniformly over  $[0,3]$ . The target is
$$
f ^ {*} (x, y) = \mathbf {1} _ {x \geqslant 0. 3} + \mathbf {1} _ {x \geqslant 0. 5} + \mathbf {1} _ {x \geqslant 0. 7} + \mathbf {1} _ {y \geqslant 0. 3} + \mathbf {1} _ {y \geqslant 0. 5} + \mathbf {1} _ {y \geqslant 0. 7}.
$$
The activation function is the same as in the one-dimensional case. We use SGD with batch size  $10^{3}$ .
Name	Value
m	10
ε	1
η	10-2
P	300
h	1.0
Additive noise	None