SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Raphael Berthier
raphael.berthier@epfl.ch
We study the training dynamics of shallow neural networks, in a two-timescale regime in which the stepsizes for the inner layer are much smaller than those for the outer layer. In this regime, we prove convergence of the gradient flow to a global optimum of the non-convex optimization problem in a simple univariate setting. The number of neurons need not be asymptotically large for our result to hold, distinguishing our result from popular recent approaches such as the neural tangent kernel or mean-field regimes. Experimental illustration is provided, showing that the stochastic gradient descent behaves according to our description of the gradient flow and thus converges to a global optimum in the two-timescale regime, but can fail outside of this regime.
Artificial neural networks are among the most successful modern machine learning methods, in particular because their non-linear parametrization provides a flexible way to implement feature learning (see, e.g., Goodfellow et al., 2016, chapter 15). Following this empirical success, a large body of work has been dedicated to understanding their theoretical properties, and in particular to analyzing the optimization algorithm used to tune their parameters. It usually consists in minimizing a loss function through stochastic gradient descent (SGD) or a variant (Bottou et al., 2018). However, the non-linearity of the parametrization implies that the loss function is non-convex, breaking the standard convexity assumption that ensures global convergence of gradient descent algorithms.
In this paper, we study the training dynamics of shallow neural networks, i.e., of the form
$$
f (x; a, u) = a _ {0} + \sum_ {j = 1} ^ {m} a _ {j} g (x; u _ {j}),
$$
where  $m$  denotes the number of hidden neurons,  $a = (a_0, \dots, a_m)$  and  $u = (u_1, \dots, u_m)$  denote respectively the outer and inner layer parameters, and  $g(x; u)$  denotes a non-linear function of  $x$  and  $u$ . The novelty of this work lies in the use of a so-called two-timescale regime (Borkar, 1997) to train the neural network: we set stepsizes for the inner layer  $u$  to be an order of magnitude smaller than the stepsizes of the outer layer  $a$ . This ratio is controlled by a parameter  $\varepsilon$ . In the regime  $\varepsilon \ll 1$ , the neural network can be thought of as a fitted linear regression with slowly evolving features
$g(x;u_j),j = 1,\ldots ,m$  : this reduction enables us to precisely describe the movement of the inner layer parameters  $u_{j}$
Our approach proves convergence of the gradient flow to a global optimum of the non-convex landscape with a fixed number  $m$  of neurons. The gradient flow can be seen as the simplifying yet insightful limit of the SGD dynamics as the stepsize  $h$  vanishes. Proving convergence with a fixed number of neurons contrasts with two other popular approaches that require to take the limit  $m \to \infty$ : the neural tangent kernel (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019; Zou et al., 2020) and the mean-field approach (Chizat and Bach, 2018; Mei et al., 2018; Rotskoff and Vanden-Eijnden, 2018; Sirignano and Spiliopoulos, 2020). As a consequence, this paper is intended as a step towards understanding feature learning with a moderate number of neurons.
While our approach through the two-timescale regime is general, our description of the solution of the two-timescale dynamics and our convergence results are specific to a simple example showcasing the approach. More precisely, we consider univariate data  $x \in [0,1]$  and non-linearities of the form  $g(x;u_j) = \sigma (\eta^{-1}(x - u_j))$ , where  $u_{j}$  is a variable translation parameter,  $\eta$  is a fixed dilatation parameter, and  $\sigma$  is a sigmoid-like non-linearity. Finally, we restrict ourselves to the approximation of piecewise constant functions.
Organization of this paper. In Section 2, we detail our setting and state our main theorem on the convergence of the gradient flow to a global optimum. Section 3 articulates this paper with related work. Section 4 provides a self-contained introduction to the two-timescale limit  $\varepsilon \to 0$ . We explain how it simplifies the analysis of neural networks, and provides heuristic predictions for the movement of neurons in our setting. Section 5 gives a rigorous derivation of our result. We prove convergence first in the two-timescale limit  $\varepsilon \to 0$ , then in the two-timescale regime with  $\varepsilon$  small but positive. Section 6 presents numerical experiments showing that the SGD dynamics follow closely those of the gradient flow in the two-timescale regime, and therefore exhibit convergence to a global optimum. On the contrary, SGD can fail to reach a global optimum outside of the two-timescale regime.
2 Setting and main result
We present a setting in which a piecewise constant univariate function  $f^{*}:[0,1]\to \mathbb{R}$  is learned with gradient flow on a shallow neural network. Our notations are summarized on Figure 1. We begin by introducing our class of functions of interest.
Definition 1. Let  $n\geqslant 2$ $\Delta v\in (0,1)$ $\Delta f > 0$  and  $M\geqslant 1$  .We denote  $\mathcal{F}_{n,\Delta v,\Delta f,M}$  the class of functions  $f^{*}:[0,1]\to \mathbb{R}$  satisfying the following conditions:
-  $f^{*}$  is piecewise constant: there exists
$$
0 = v _ {0} <   v _ {1} <   \dots <   v _ {n - 1} <   v _ {n} = 1
$$
and  $f_0^*, \ldots, f_{n-1}^* \in \mathbb{R}$  such that
$$
\forall x \in (v _ {i}, v _ {i + 1}), f ^ {*} (x) = f _ {i} ^ {*},
$$
for all  $i\in \{1,\ldots ,n\} ,v_{i} - v_{i - 1}\geqslant \Delta v,$
for all  $i\in \{1,\dots ,n - 1\} ,|f_i^* -f_{i - 1}^* |\geqslant \Delta f,$
for all  $i\in \{0,\dots ,n - 1\}$ $|f_i^* |\leqslant M$
Figure 1: Notations of the paper. The target  $f^{*}$  is in blue and the neural network  $f(\cdot; a, u)$  in orange.
Let us now define our class of neural networks. Consider  $\sigma : \mathbb{R} \to \mathbb{R}$  an increasing, twice continuously differentiable non-linearity such that  $\sigma(x) = 0$  if  $x \leq -1/2$ ,  $\sigma(x) = 1$  if  $x \geq 1/2$ , and  $\sigma - 1/2$  is odd. Then, our class of shallow neural networks is defined by
$$
f (x; a, u) = a _ {0} + \sum_ {j = 1} ^ {m} a _ {j} \sigma_ {\eta} (x - u _ {j}), \quad \sigma_ {\eta} (x) = \sigma (\eta^ {- 1} x),
$$
where  $0 < \eta \leqslant 1$  measure the sharpness of the non-linearity  $\sigma_{\eta}$ . Note that the inner layer parameter  $u_{j}$  determines the translation of the non-linearity; no parameterized multiplicative operation on  $x$  is performed in this layer. We refer to the parameter  $u$  as the "positions" of the neurons (or, sometimes, simply as the "neurons") and to the parameter  $a$  as the "weights" of the neurons. We define the quadratic loss as
$$
L (a, u) = \frac {1}{2} \int_ {0} ^ {1} \left(f ^ {*} (x) - f (x; a, u)\right) ^ {2} \mathrm {d} x.
$$
We use gradient flow on  $L$  to fit the parameters  $a$  and  $u$ : they evolve according to the dynamics
$$
\frac {\mathrm {d} a}{\mathrm {d} t} (t) = - \nabla_ {a} L (a (t), u (t)), \quad \frac {\mathrm {d} u}{\mathrm {d} t} (t) = - \varepsilon \nabla_ {u} L (a (t), u (t)), \tag {1}
$$
where  $\varepsilon$  corresponds to the ratio of the step sizes of the two iterations.
Main result. By leveraging the two-timescale regime where  $\varepsilon$  is small, our theorem shows that, with high probability, a neural network trained with gradient flow is able to recover an arbitrary piecewise constant function to an arbitrary precision. The proof is relegated to the Appendix.
Theorem 1. Let  $\xi, \delta > 0$ , and  $f^*$  a piecewise constant function from  $\mathcal{F}_{n,\Delta v,\Delta f,M}$ . Assume that the neural network has  $m$  neurons with
$$
m \geqslant \frac {6}{\Delta v} (4 + \log n + \log \frac {1}{\delta}). \tag {2}
$$
Assume that, at initialization, the positions  $u_{1}, \ldots, u_{m}$  of the neurons are i.i.d. uniformly distributed on  $[0, 1]$  and their weights  $a_{0}, \ldots, a_{m}$  are equal to zero.
Then there exists  $Q_{1} > 0$  and  $Q_{2} > 0$  depending on  $\xi, \delta, m, \Delta f, M$  such that, if
$$
\eta \leqslant Q _ {1}, \quad \varepsilon \leqslant Q _ {2}, \tag {3}
$$
then, with probability at least  $1 - \delta$ , the solution to the gradient flow (1) is defined at least until  $T = \frac{6}{\varepsilon(\Delta f)^2}$ , and
$$
\int_ {0} ^ {1} | f ^ {*} (x) - f (x; a (T), u (T)) | ^ {2} \mathrm {d} x \leqslant \xi .
$$
Further,  $Q_{1} = \frac{C_{1}}{M^{2}(m + 1)}\min \left(\frac{\delta^{2}(\Delta f)^{2}}{(m + 1)^{4}},\xi\right)$  and  $Q_{2} = \frac{C_{2}\delta^{2}}{M^{4}(m + 1)^{17 / 2}}\min \left(\frac{\delta(\Delta f)^{2}}{m + 1},\xi\right)$  for some universal constants  $C_1,C_2 > 0$
For this result to hold, the inequality (2) requires the number of neurons in the neural network to be large enough. Note that the minimum number of neurons required to approximate the  $n$  pieces of  $f^{*}$  is equal to  $n$ . If the length of all the intervals is of the same order of magnitude, then  $\Delta v = \Theta(1/n)$  and thus the condition is  $m = \Omega(n(1 + \log n + \log 1/\delta))$ . In this case, condition (2) only adds a logarithmic factor in  $n$  and  $\delta$ . Moreover, the lower bound on  $m$  does not depend on the target precision  $\xi$ . Thus we observe some non-linear feature learning phenomenon: with a fixed number  $m$  of neurons, gradient flow on a neural network can approximate any element from the infinite-dimensional space of piecewise constant functions to an arbitrary precision.
The recovery result of Theorem 1 is provided under two conditions (3). The first one should not surprise the reader: the condition on  $\eta$  enables the non-linearity to be sharp enough in order to approximate well the jumps of the piecewise constant function  $f^{*}$ . The novelty of our work lies in the condition on  $\varepsilon$ , that we refer to as the two-timescale regime. This condition ensures that the step sizes taken in the positions  $u$  are much smaller than the step sizes taken in the weights  $a$ . As a consequence, the weights  $a$  are constantly close to the best linear fit given the current positions  $u$ . This property decouples the dynamics of the two layers of the neural network; this enables a sharp description of the gradient flow trajectories and thus the recovery result shown above. This intuition is detailed in Section 4.
Two-timescale regime. Systems with two timescales, or slow-fast systems, have a long history in physics and mathematics, see Berglund and Gentz (2006, Chapter 2) for an introduction. In particular, iterative algorithms with two timescales have been used in stochastic approximation and optimization, see Borkar (1997) or Borkar (2009, Section 6). For instance, they are used in the training of generative adversarial networks, to decouple the dynamics of the generator from those of the discriminator (Heusel et al., 2017), in reinforcement learning, to decouple the value function estimation from the temporal difference learning (Szepesvári, 2010), or more generally in bilevel optimization, to decouple the outer problem dynamics from the inner problem dynamics (Hong et al., 2023). However, to the best of our knowledge, the two-timescale regime has not been used to show convergence results for neural networks.
Layer-wise learning rates. Practitioners are interested in choosing learning rates that depend on the layer index to speed up training or improve performance. Using smaller learning rates for the first layers and higher learning rates for the last layer(s) improves performance for fine-tuning (Howard and Ruder, 2018; Ro and Choi, 2021) and is a common practice for transfer learning (see, e.g., Li et al., 2022). Another line of work proposes to update layer-wise learning rates depending on the norm of the gradients on each layer (Singh et al., 2015; You et al., 2017; Ko et al., 2022). However, they aim to compensate the differences across gradient norms in order to learn all the parameters at the same speed, while on the contrary we enjoy the theoretical benefits of learning different speeds.
Theory of neural networks. A key novelty of the analysis of this paper is that we show recovery with a fixed number of neurons. We now detail the comparison with other analyses.
The neural tangent kernel regime (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019; Zou et al., 2020) corresponds to small movements of the parameters of the neural network. In this case, the neural network can be linearized around its initial point, and thus behaves like a linear regression. However, in this regime, the neural network can approximate only a finite dimensional space of functions, and thus it is necessary to take  $m \to \infty$  to be able to approximate the infinite-dimensional space of piecewise constant functions to an arbitrary precision.
The mean-field regime (Chizat and Bach, 2018; Mei et al., 2018; Rotskoff and Vanden-Eijnden, 2018; Sirignano and Spiliopoulos, 2020) describes the dynamics of two-layer neural networks in the regime  $m \gg 1$  through a partial differential equation on the density of neurons. This regime is able to describe some non-linear feature learning phenomena, but does not explain the observed behavior with a moderate number of neurons. In this paper, we show that in the two-timescale regime, only a single neuron aligns with each of the discontinuities of the function  $f^{*}$ . However, it should be noted that the neural tangent kernel and mean-field regimes have been applied to show recovery in a wide range of settings, while our work is restricted to the recovery of piecewise constant functions. Extending the application of the two-timescale regime is left for future work.
Our work includes a detailed analysis of the alignment of the positions of the neurons with the discontinuities of the target function  $f^{*}$ . This is analogous to a line of work (see, e.g., Saad and Solla (1995); Goldt et al. (2020); Veiga et al. (2022)) interested in the alignment of a "student" neural network with the features of a "teacher" neural network that generated the data, for high-dimensional Gaussian input. In general, the non-linear evolution equations describing this alignment are hard to study theoretically. On the contrary, thanks to the two-timescale regime and to the simple setting of this paper, we are able to give a precise description of the movement of the neurons.
Our study bears high-level similarities with the recent work of Safran et al. (2022). In a univariate classification setting, they show that a two-layer neural network achieves recovery with a number of neurons analogous to (2): inversely proportional to the length of the smallest constant interval of the target, up to logarithmic terms in the number of constant intervals and in the failure probability. However, the two papers have different settings: Safran et al. (2022) consider classification with ReLU activations while we consider regression with sigmoid-like activations. More importantly, the authors do not use the two-timescale regime. Instead, by a specific initialization scale, they ensure that the neural network has a first lazy phase where the positions of the neurons do not move significantly. For the second rich phase, they describe the implicit bias of the limiting point; this approach does not lead to an estimate of the convergence time while the fine description of the two-timescale limit does.
Finally, a related technique is the so-called layerwise training, which consists in first training the inner layer with the outer layer fixed, and then doing the reverse. This setup has been used in theoretical works to show convergence of (stochastic) gradient descent in a feature learning regime with a moderate number of neurons (Abbe et al., 2023; Damian et al., 2022). The two-timescale regime can be seen as a refinement of this technique since we allow both layers to move simultaneously instead of sequentially, which is closer to practical setups.
4 A non-rigorous introduction to the two-timescale limit
This section introduces the core ideas of our analysis in a non-rigorous way. Section 4.1 introduces the limit of the dynamics when  $\varepsilon \rightarrow 0$ , called the two-timescale limit. Section 4.2 applies the two-timescale limit to predict the movement of the neurons.
4.1 Introduction to the two-timescale limit
Let us consider the gradient flow equations (1) and perform the change of variables  $\tau = \varepsilon t$ :
$$
\frac {\mathrm {d} a}{\mathrm {d} \tau} = - \frac {1}{\varepsilon} \nabla_ {a} L (a, u), \quad \frac {\mathrm {d} u}{\mathrm {d} \tau} = - \nabla_ {u} L (a, u). \tag {4}
$$
In the two-timescale regime  $\varepsilon \ll 1$ , the rate of the gradient flow in the weights  $a$  is much larger than then the rate in the positions  $u$ . Note that  $L$  is marginally convex in  $a$ , and thus, for a fixed  $u$ , the gradient flow in  $a$  must converge to a global minimizer of  $a \mapsto L(a, u)$ . More precisely, assume that  $\{\sigma_{\eta}(. - u_1), \dots, \sigma_{\eta}(. - u_m)\}$  forms an independent set of functions in  $L^2([0,1])$ . Then the global minimizer of  $a \mapsto L(a, u)$  is unique; we denote it as  $a^*(u)$ . In the limit  $\varepsilon \to 0$ , we expect that  $a$  evolves sufficiently quickly with respect to  $u$  so that it converges instantaneously to  $a^*(u)$ . In other words, the gradient flow system (4) reduces to its so-called two-timescale limit when  $\varepsilon \to 0$ :
$$
a = a ^ {*} (u), \quad \frac {\mathrm {d} u}{\mathrm {d} \tau} = - \nabla_ {u} L \left(a ^ {*} (u), u\right). \tag {5}
$$
The two-timescale limit considerably simplifies the study of the gradient flow system because it substitutes the weights  $a$ , determined to be equal to  $a^{*}(u)$ . However, showing that (4) reduces to (5) requires some mathematical care, including checking that  $a^{*}(u)$  is well-defined.
Remark 1 (abuse of notation). Equation (5) contains an ambiguous notation: does  $\nabla_u L(a^*(u), u)$  denote the gradient in  $u$  of the map  $L^*: u \mapsto L(a^*(u), u)$  or the gradient  $(\nabla_u L)(a, u)$  taken at  $a = a^*(u)$ ? In fact, by definition of  $a^*(u)$ , both quantities coincide:
$$
(\nabla_ {u} L ^ {*}) (u) = \left(\frac {\mathrm {d} a ^ {*}}{\mathrm {d} u} (u)\right) ^ {\top} (\nabla_ {a} L) (a ^ {*} (u), u) + (\nabla_ {u} L) (a ^ {*} (u), u) = (\nabla_ {u} L) (a ^ {*} (u), u),
$$
where  $\frac{\mathrm{d}a^*}{\mathrm{d}u}$  denotes the differential of  $a^*$  in  $u$  and we use that, by definition of  $a^*(u)$ ,  $(\nabla_a L)(a^*(u), u) = 0$ . This is a special case of the envelope theorem (Border, 2015, Sec. 5.10).
The discussion in this section is not specific to the setting of Section 2. Using the two-timescale limit to decouple the dynamics of the outer layer  $a$  and the inner layer  $u$  is a general tool that may be used in the study of any two-layer neural network. We chose the specific setting of this paper so that the two-timescale limit (5) can be easily studied, thereby showcasing the approach. The next section is devoted to a sketch of this study.
4.2 Sketch of the dynamics of the two-timescale limit
In this section, in order to simplify the exposition of the behavior of the two-timescale limit (5), we consider the limiting case  $\eta \rightarrow 0$ . Note that this is coherent with Theorem 1 that requires  $\eta$  to be small. This limit is a neural network with a non-linearity equal to the Heaviside function
$$
\sigma_ {0} (x) = 0 \quad \text {i f} x <   0, \qquad \sigma_ {0} (x) = 1 / 2 \quad \text {i f} x = 0, \qquad \sigma_ {0} (x) = 1 \quad \text {i f} x > 0.
$$
Note that  $\sigma_0$  would be a poor choice of non-linearity in practice: as its derivative is 0 almost everywhere, the positions  $u$  would not move. However, it is a relevant tool to get an intuition about the dynamics of our system for a small  $\eta$ . Moreover, as we will see in Section 6, the dynamics sketched here match closely those of the SGD (with  $\eta > 0$ ).
Figure 2: Sketch of the dynamics of the neurons in the two-timescale limit with a Heaviside nonlinearity. Only the neurons next to a discontinuity of the target move.
The set  $\{1, \sigma_0(. - u_1), \dots, \sigma_0(. - u_m)\}$  generates the space of functions that are piecewise constant with respect to the subdivision  $\{u_1, \dots, u_m\}$ . Furthermore, if  $u_1, \dots, u_m$  are distinct and in  $(0, 1)$ , then this set is an independent set of functions in  $L^2([0, 1])$ . Thus  $a^*(u)$  is well defined and represents the coefficients of the best piecewise constant approximation of  $f^*$  with subdivision  $\{u_1, \dots, u_m\}$ .
This quantity is straightforward to describe under the mild additional assumption that there are at least two neurons  $u_{j}$  in each interval  $(v_{i - 1}, v_{i})$  between two points of discontinuity of  $f^{*}$ . For each  $1 \leqslant i \leqslant n$ , let  $u_{i}^{\mathrm{L}}$  denote the largest position of neurons below  $v_{i}$  and  $u_{i}^{\mathrm{R}}$  denote the smallest position above  $v_{i}$  (with convention  $u_{0}^{\mathrm{R}} = 0$  and  $u_{n + 1}^{\mathrm{L}} = 1$ ). By assumption,  $0 = u_{0}^{\mathrm{R}} < u_{1}^{\mathrm{L}} < u_{1}^{\mathrm{R}} < \dots < u_{n}^{\mathrm{L}} < u_{n}^{\mathrm{R}} < u_{n + 1}^{\mathrm{L}} = 1$  are distinct. A simple computation then shows the following identities:
- for all  $i \in \{1, \dots, n\}$ , for all  $x \in (u_i^{\mathrm{L}}, u_i^{\mathrm{R}})$ ,  $f(x; a^*(u), u) = \frac{v_i - u_i^{\mathrm{L}}}{u_i^{\mathrm{R}} - u_i^{\mathrm{L}}} f_{i-1}^* + \frac{u_i^{\mathrm{R}} - v_i}{u_i^{\mathrm{R}} - u_i^{\mathrm{L}}} f_i^*$ ,
and for all  $i\in \{1,\ldots ,n + 1\}$  , for all  $x\in (u_{i - 1}^{\mathrm{R}},u_i^{\mathrm{L}})$ $f(x;a^{*}(u),u) = f_{i - 1}^{*}$
where we recall that  $f_{i}^{*}$  denotes the value of  $f^{*}$  on the interval  $(v_{i}, v_{i+1})$ . Figure 2 illustrates the situation. Moreover, the loss  $L(a^{*}(u), u)$ , which is half of the square  $L^2$ -error of this optimal approximation, can be written
$$
L \left(a ^ {*} (u), u\right) = \frac {1}{2} \sum_ {i = 1} ^ {n - 1} \frac {\left(v _ {i} - u _ {i} ^ {\mathrm {L}}\right) \left(u _ {i} ^ {\mathrm {R}} - v _ {i}\right)}{u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}} \left(f _ {i} ^ {*} - f _ {i - 1} ^ {*}\right) ^ {2}. \tag {6}
$$
The dynamics of the two-timescale limit (5) corresponds to the local optimization of the subdivision  $u$  in order to minimize the loss (6). A remarkable property of this loss is that it decomposes as a sum of local losses around the jump points  $v_{i}$  for  $i \in \{1, \dots, n-1\}$ . Each element of the sum involves only the two neurons located at  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$ . As a consequence, the dynamics of the two-timescale limit (5) decompose as  $n$  independent systems of two neurons  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$ : for all  $i \in \{1, \dots, n-1\}$ ,
$$
\frac {\mathrm {d} u _ {i} ^ {\mathrm {L}}}{\mathrm {d} \tau} = - \frac {\mathrm {d} L}{\mathrm {d} u _ {i} ^ {\mathrm {L}}} (a ^ {*} (u), u) = + \frac {1}{2} \frac {(u _ {i} ^ {\mathrm {R}} - v _ {i}) ^ {2}}{(u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}) ^ {2}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}) ^ {2},
$$
$$
\frac {\mathrm {d} u _ {i} ^ {\mathrm {R}}}{\mathrm {d} \tau} = - \frac {\mathrm {d} L}{\mathrm {d} u _ {i} ^ {\mathrm {R}}} (a ^ {*} (u), u) = - \frac {1}{2} \frac {(v _ {i} - u _ {i} ^ {\mathrm {L}}) ^ {2}}{(u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}) ^ {2}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}) ^ {2}.
$$
All neurons other than  $u_{1}^{\mathrm{L}}, u_{1}^{\mathrm{R}}, \ldots, u_{n-1}^{\mathrm{L}}, u_{n-1}^{\mathrm{R}}$  do not play a role in the expression (6), thus they do not move in the two-timescale limit (5). The position  $u_{i}^{\mathrm{L}}$  moves right and  $u_{i}^{\mathrm{R}}$  moves left, until one of them hits the point  $v_{i}$ . This shows that the positions of the neurons eventually align with the jumps of the function  $f^{*}$ , and thus that the function  $f^{*}$  is recovered.
5 Convergence of the gradient flow