SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Parameters of Figure 6a.
Name	Value
m	10
ε	10-2
η	10-2
P	5·103
h	1.0
Additive noise	None
Table 4: Parameters of Figure 6b.
As in the one-dimensional case, we re-run the experiment twenty times and report the results in Figure 9a.
We perform a similar experiment in 10D, using the same setup. The target is
$$
f ^ {*} (x _ {1}, \ldots , x _ {1 0}) = \sum_ {i = 1} ^ {1 0} \mathbf {1} _ {x _ {i} \geqslant 0. 3} + \mathbf {1} _ {x _ {i} \geqslant 0. 5} + \mathbf {1} _ {x _ {i} \geqslant 0. 7}.
$$
We use SGD with batch size  $10^{4}$ . The results in terms of  $L_{2}$  distance are reported in Figure 9b.
(a) In 2D
(b) In 10D
Figure 9:  $L_{2}$  distance with the target as a function of  $\varepsilon$  in the higher-dimensional experiment, with 20 repeats.
ReLU networks. The goal of this experiment is to use ReLU activations to approximate piecewise-affine targets in 1D. The target is a ReLU network with positions  $u = [0.3, 0.5, 0.7]$  and weights  $a = [1.0, -2.0, 3.0]$ . We use the same setup as for the 2D experiment described above. The results are reported in Figure 10.
(a) After training in the standard regime
(b) After training in the two-timescale regime
(c)  $L_{2}$  distance with the target as a function of  $\varepsilon$ , 20 repeats
Figure 10: Experiment with ReLU activation and piecewise-affine targets. In the standard regime, some kinks of the target are not covered by neurons. This is not the case in the two-timescale regime.
D Additional remarks
D.1 Counter-example in the case of non-uniform initialization
It is easy to craft an example where gradient flow in the two-timescale limit does not converge to the global minimum if the positions of the neurons are not drawn uniformly at random at initialization. Take for instance the case where there are three pieces in the target, the leftmost piece at level 1, the second one at level  $-1$  and the third one at level 0. Consider the case where all neurons are in the third piece at initialization. Then (in the two-timescale limit) the bias and the weights are all instantly equal to zero, which is the solution of the optimization problem in  $a$ . Looking at (10), this shows that the positions do not move, and thus that the neurons remain in this local minimum.
This example may seem artificial, but in fact, more generally, the case where two consecutive pieces are not covered by any neuron often corresponds to a local minimum of the two-timescale dynamics. This is why we require neurons on every piece at initialization to avoid falling in this local minimum.
D.2 Ideas of proof for SGD
The proof of Theorem 1 consists in bounding the difference between the actual dynamics and the dynamics that we would have if the weights were given by  $a_0^*(u(t))$ , that is, by the weights in the
two-timescale limit with  $\eta = 0$ . This requires to bound two errors terms, one coming from the fact that  $\eta > 0$ , and one coming from the fact that  $\varepsilon > 0$ . By making  $\eta$  and  $\epsilon$  small enough, we ensure that both terms are small, which in turn allows to describe the trajectories of the weights and of the positions throughout the dynamics.
Moving from gradient flow to stochastic gradient descent induces two additional error terms: a discretization error, and a noise error. However, taking small enough step size allows to bound both error terms and thus ensures that (with high probability) the SGD dynamics are close to the gradient flow dynamics. For this reason, we believe that our proof structure (namely, bounding the difference with the two-timescale limit as in Proposition 4, then showing that this bound allows to describe the trajectories of the weights and of the positions as in Theorem 1) should adapt to the SGD case, at the condition that the step size is small enough.