- We provide a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training, and show that it applies whenever the initialisation scale  $\lambda$  is below a precise bound which is polynomial in the network width  $m$  and exponential in the training dataset cardinality  $n$ . Moreover, our analysis applies for any input dimension  $d > 1$ , for any  $n \geq d$  (otherwise exact learning of the teacher neuron may not be possible), for any  $m$ , and without assuming any specific random distribution for the initialisation. In particular, we demonstrate that the role of the overparameterisation in this setting is to ensure that initially at least one hidden neuron with a positive last-layer weight has in its active half-space at least one training point.
- We show that, during a first phase of the training, all active hidden neurons with a positive last-layer weight get aligned to a single direction which is positively correlated with all training points, whereas all active hidden neurons with a negative last-layer weight get turned away from all training points so that they deactivate. In contrast to the orthogonal dataset case where the sets of training points that are in the active half-spaces of the neurons are essentially constant during the training, in our correlated setting this first phase in general consists, for each neuron, of a different sequence of stages during which the cardinality of the set of training points in its active half-space gradually increases or decreases, respectively.
- We show that, during the rest of the training, the bundle of aligned hidden neurons with their last-layer weights, formed by the end of the first phase, grows and turns as it travels from near the origin to near the teacher neuron, and does not separate. To establish the latter property, which is the most involved part of this work, we identify a set in predictor space that depends only on  $\lambda$  and the training dataset, and prove: first, that the trajectory of the bundle stays inside the set; and second, that this implies that the directional gradients of the individual neurons are such that the angles between them are non-increasing.
- We prove that, after the training departs from the initial saddle, which takes time logarithmic in  $\lambda$  and linear in  $d$ , the gradient satisfies a Polyak-Lojasiewicz inequality and consequently the loss converges to zero exponentially fast.
- We prove that, although for any fixed  $\lambda$  the angles in the bundle of active hidden neurons do not in general converge to zero as the training time tends to infinity, if we let  $\lambda$  tend to zero then the networks to which the training converges have a limit: a network of rank 1, in which all non-zero hidden neurons are positive scalings of the teacher neuron and have positive last-layer weights. This establishes that gradient flow from an infinitesimal initialisation is implicitly biased to select interpolator networks of minimal rank. Note also that the limit network is identical in predictor space to the teacher neuron.
- We show that, surprisingly, among all networks with zero loss, there may exist some whose Euclidean norm is smaller than that of any network of rank 1. Moreover, we prove that this is the case if and only if a certain condition on angles determined by the training dataset is satisfied. This result might be seen as refuting the conjecture of Boursier et al. [2022, section 3.2] that the implicit bias to minimise Euclidean parameter norm holds beyond the orthogonal setting, and adding some weight to the hypothesis of Razin and Cohen [2020]. The counterexample networks in our proof have rank 2 and make essential use of the ReLU non-linearity.
- We perform numerical experiments that indicate that the training dynamics and the implicit bias we theoretically established occur in practical settings in which some of our assumptions are relaxed. In particular, gradient flow is replaced by gradient descent with a realistic learning rate,