SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Global minimum: We follow the exact same reasoning as for the second saddle. We now have  $s_{t_3}$  equal to the vector  $(-1, 1)$  and the iterates must jump to a point  $\beta_3$  such that  $(i) \beta_3[1] \leq 0$ ,  $\beta_3[2] \geq 0$  (K2 and K3),  $(ii) \nabla L(\beta_3)[1] \leq 0$ ,  $\nabla L(\beta_3)[2] \geq 0$  (K1),  $(iii)$  for  $i \in \{1, 2\}$ , if  $\beta_3[i] \neq 0$  then  $\nabla L(\beta_3)[i] = 0$  (K4). Again, these are the optimality conditions of the following problem:
$$
\operatorname *{arg  min}_{\beta [1]\leq 0,\beta [2]\geq 0}L(\beta).
$$
$\beta^{\star}$  is the unique minimiser of this problem and  $\beta_{3} = \beta^{\star}$ . For  $t \geq t_{3}$  we have  $s_{t} = s_{t_{3}}$  and Eq. (12) is satisfied for all following times: the iterates do not have to move anymore.
3.2 Presentation of the full saddle-to-saddle algorithm
We can now provide the full algorithm (Algorithm 1) which computes the jump times  $(t_1, \ldots, t_p)$  and saddles  $(\beta_0 = \mathbf{0}, \beta_1, \ldots, \beta_p)$  as the values and vectors such that the associated piecewise constant process satisfies Eq. (12) for all  $t$ . This algorithm therefore defines our limiting process  $\tilde{\beta}^\circ$ .
Algorithm 1 in words. The algorithm is a concise representation of the steps we followed in the previous section to construct  $\tilde{\beta}^{\circ}$ . We explain each step in words below. Starting from  $k = 0$ , assume we enter the loop number  $k$  at the saddle  $\beta_{k}$  computed in the previous loop:
- The set  $\mathcal{A}_k$  contains the set of coordinates "which are unstable": by having a non-zero derivative, the loss could be decreased by moving along each one of these coordinates and one of these coordinates will have to activate.
Algorithm 1: Successive saddles and jump times of  $\lim_{\alpha \to 0}\tilde{\beta}^{\alpha}$
Initialise:  $(t,\beta ,s)\gets (0,0,0)$    
while  $\nabla L(\beta)\neq 0$  do   
 $\begin{array}{rl} & {\mathcal{A}\leftarrow \{j\in [d],\nabla L(\beta)(j)\neq 0\}}\\ & {\Delta \leftarrow \inf \{\delta >0\mathrm{s.t.}\exists i\in \mathcal{A},s(i) - \delta \nabla L(\beta)(i) = \pm 1\}}\\ & {(t,s)\leftarrow (t + \Delta ,s - \Delta \cdot \nabla L(\beta))}\\ & {\beta \leftarrow \arg \min L(\beta)\text{where}\beta \in \left\{\beta \in \mathbb{R}^d\text{s.t.}\begin{array}{ll}\beta_i\geq 0\text{if} s(i) = +1\\ \beta_i\leq 0\text{if} s(i) = -1\\ \beta_i = 0\text{if} s(i)\in (-1,1) \end{array} \right\}} \end{array}$    
end   
Output: Successive values of  $\beta$  and  $t$
- The time gap  $\Delta_{k}$  corresponds to the time spent at the saddle  $\beta_{k}$ . It is computed as being the elapsed time just before (K1) breaks if the coordinates do not jump.
- We update  $t_{k+1} = t_k + \Delta_k$  and  $s_{k+1} = s_k - \Delta_k \nabla L(\beta_k)$ :  $t_{k+1}$  corresponds to the time at which the iterates leave the saddle  $\beta_k$  and  $s_{k+1}$  constrains the signs of the next saddle  $\beta_{k+1}$ .
- The solution  $\beta_{k+1}$  of the constrained minimisation problem is the saddle to which the flow jumps to at time  $t_{k+1}$ . The optimality conditions of this problem are such that Eq. (12) is maintained for  $t \geq t_{k+1}$ .
Various comments on Algorithm 1. First we point out that any solution  $\beta_{c}$  of the constrained minimisation problem which appears in Algorithm 1 also satisfies  $\beta_{c} = \arg \min_{\beta [i] = 0}$  for  $i\notin \mathrm{supp}(\beta_c)$ $L(\beta)$  as in Eq. (4): the algorithm hence indeed outputs saddles as expected. Up until now we have never checked whether the algorithm's constrained minimisation problem has a unique minimum. This is crucial otherwise the assignment step would be ill-defined. Showing the uniqueness is non-trivial and is guaranteed thanks to the general position Assumption 1 on the data (see Proposition 7 in Appendix D.1). In this same proposition, we also show that the algorithm terminates in at most  $\min \left(2^{d},\sum_{k = 0}^{n}\binom {d}{k}\right)$  steps, that the loss strictly decreases at each step and that the final output  $\beta_{p}$  is the minimum  $\ell_1$  -norm solution. These last two properties are expected given the fact that the algorithm arises as being the limit process of  $\tilde{\beta}^{\alpha}$  which follows the mirror flow Eq. (10).
Links with the LARS algorithm for the Lasso. Recall that the Lasso problem [46, 15] is formulated as:
$$
\beta_ {\lambda} ^ {\star} = \underset {\beta \in \mathbb {R} ^ {d}} {\arg \min } L (\beta) + \lambda \| \beta \| _ {1}. \tag {13}
$$
The optimality condition of Eq. (13) writes  $-\nabla L(\beta_{\lambda}^{\star}) \in \lambda \partial \| \beta_{\lambda}^{\star} \|_1$ . Now notice the similarity with Eq. (12): the two would be equivalent with  $\lambda = 1 / t$  if the integration on the left hand side of Eq. (12) did not average over the whole trajectory but only on the final iterate, in which case  $-\int_0^t \nabla L(\tilde{\beta}_t^\circ) \mathrm{d}s = -t \cdot \nabla L(\tilde{\beta}_t^\circ)$ . Though the difference is small, the trajectories of our limiting trajectory  $\tilde{\beta}^\circ$  and the lasso path  $(\beta_{\lambda}^{\star})_{\lambda}$  are quite different: one has jumps, whereas the other is continuous. Nonetheless, the construction of Algorithm 1 shares many similarities with that of the Least Angle Regression (LARS) algorithm [19] (originally named the Homotopy algorithm [39]) which is used to compute the Lasso path. A notable difference however is the fact that each step of our algorithm depends on the whole trajectory through the vector  $s$ , whereas the LARS algorithm can be started from any point on the path.
3.3 Outputs of the algorithm under a RIP and gap assumption on the data.
Unlike previous results on incremental learning, complex behaviours can occur when the feature matrix is ill designed: several coordinates can activate and deactivate at the same time (see Appendix A for various cases). However, if the feature matrix satisfies the  $2r$ -restricted isometry property (RIP) [14] and there exists an  $r$ -sparse solution  $\beta^{\star}$ , the visited saddles can be easily approximated using Algorithm 1. We provide the precise characterisation below.
Sparse regression with RIP and gap assumption. (RIP) Assume that there exists an  $r$ -sparse vector  $\beta^{\star}$  such that  $y_{i} = \langle x_{i},\beta^{\star}\rangle$ . Furthermore we assume that the feature matrix  $X\in \mathbb{R}^{n,d}$  satisfies the  $2r$ -restricted isometry property with constant  $\tilde{\varepsilon} < \sqrt{2} -1 < 1 / 2$ : i.e. for all submatrix  $X_{s}$  where we extract any  $s\leq 2r$  columns of  $X$ , the matrix  $X_{s}^{\top}X_{s} / n$  of size  $s\times s$  has all its eigenvalues in the interval  $[1 - \tilde{\varepsilon},1 + \tilde{\varepsilon} ]$ . (Gap assumption) Furthermore we assume that the  $r$ -sparse vector  $\beta^{\star}$  has coordinates which have a "sufficient gap". W.l.o.g we write  $\beta^{\star} = (\beta_{1}^{\star},\dots ,\beta_{r}^{\star},0,\dots ,0)$  with  $|\beta_1^\star |\geq \ldots \geq |\beta_r^\star | > 0$  and we define  $\lambda \coloneqq \min_{i\in [r]}(|\beta_i^* | - |\beta_{i + 1}^*\mid)\geq 0$  which corresponds to the smallest gap between the entries of  $|\beta^{\star}|$ . We assume that  $5\tilde{\varepsilon}\| \beta^{\star}\|_{2} < \lambda /2$  and we let  $\varepsilon \coloneqq 5\tilde{\varepsilon}$ .
A classic result from compressed sensing (see Candes [13, Theorem 1.2]) is that the  $2r$ -restricted isometry property with constant  $\sqrt{2} - 1$  ensures that the minimum  $\ell_0$ -minimisation problem has a unique  $r$ -sparse solution which is  $\beta^{\star}$ . This means that Algorithm 1 will have  $\beta^{\star}$  as final output and the following proposition shows that we can precisely characterise each of its outputs when the data satisfies the previous assumptions.
Proposition 2. Under the restricted isometry property and the gap assumption stated right above, Algorithm 1 terminates in  $r$ -loops and outputs:
$$
\beta_ {1} = (\beta_ {1} [ 1 ], 0, \dots , 0)
$$
$$
w i t h \quad \beta_ {1} [ 1 ] \in \left[ \beta_ {1} ^ {\star} - \varepsilon \| \beta^ {\star} \|, \beta_ {2} ^ {\star} + \varepsilon \| \beta^ {\star} \| \right]
$$
$$
\beta_ {2} = \left(\beta_ {2} [ 1 ], \beta_ {2} [ 2 ], 0, \dots , 0\right)
$$
$$
\text {w i t h} \left\{ \begin{array}{l} \beta_ {2} [ 1 ] \in \left[ \beta_ {1} ^ {\star} - \varepsilon \| \beta^ {\star} \|, \beta_ {1} ^ {\star} + \varepsilon \| \beta^ {\star} \| \right] \\ \beta_ {2} [ 2 ] \in \left[ \beta_ {2} ^ {\star} - \varepsilon \| \beta^ {\star} \|, \beta_ {2} ^ {\star} + \varepsilon \| \beta^ {\star} \| \right] \end{array} \right.
$$
$$
\beta_ {r - 1} = \left(\beta_ {r - 1} [ 1 ], \dots , \beta_ {r - 1} [ r - 1 ], 0, \dots , 0\right)
$$
$$
w i t h \beta_ {r - 1} [ i ] \in \left[ \beta_ {i} ^ {\star} - \varepsilon \| \beta^ {\star} \|, \beta_ {i} ^ {\star} + \varepsilon \| \beta^ {\star} \| \right]
$$
$$
\beta_ {r} = \beta^ {\star} = (\beta_ {1} ^ {\star}, \dots , \beta_ {r} ^ {\star}, 0, \dots , 0),
$$
at times  $t_1, \ldots, t_r$  such that  $t_i \in \left[\frac{1}{|\beta_i^*| + \varepsilon\|\beta^*\|}, \frac{1}{|\beta_i^*| - \varepsilon\|\beta^*\|}\right]$  and where  $\|\cdot\|$  denotes the  $\ell_2$  norm.
Informally, this means that the algorithm terminates in exactly  $r$  loops and outputs jump times and saddles roughly equal to  $t_i = 1 / |\beta_i^\star|$  and  $\beta_i = (\beta_1^\star, \dots, \beta_i^\star, 0, \dots, 0)$ . Therefore, in simple settings, the support of the sparse vector is learnt a coordinate at a time, without any deactivations. We refer to Appendix D.2 for the proof.
4 Convergence of the iterates towards the process defined by Algorithm 1
We are now fully equipped to state our main result which formalises the convergence of the accelerated iterates towards the limiting process  $\tilde{\beta}^{\circ}$  which we built in the previous section.
Theorem 2. Let the saddles  $(\beta_0 = 0, \beta_1, \ldots, \beta_{p-1}, \beta_p = \beta_{\ell_1}^{\star})$  and jump times  $(t_0 = 0, t_1, \ldots, t_p)$  be the outputs of Algorithm 1 and let  $(\tilde{\beta}_t^\circ)_t$  be the piecewise constant process defined as follows:
$$
(S a d d l e s)
$$
$$
\tilde {\beta} _ {t} ^ {\circ} = \beta_ {k}
$$
$$
f o r \quad t \in \left(t _ {k}, t _ {k + 1}\right) a n d \quad 0 \leq k \leq p, \quad t _ {p + 1} = + \infty .
$$
The accelerated flow  $(\tilde{\beta}_t^\alpha)_t$  defined in Eq. (9) uniformly converges towards the limiting process  $(\tilde{\beta}_t^\circ)_t$  on any compact subset of  $\mathbb{R}_{\geq 0} \setminus \{t_1, \ldots, t_p\}$ .
Convergence result. We recall that from a technical point of view, showing the existence of a limiting process  $\lim_{\alpha \to 0} \tilde{\beta}^{\alpha}$  is the toughest part. Theorem 2 provides this existence as well as the uniform convergence of the accelerated iterates towards  $\tilde{\beta}^{\circ}$  over all closed intervals of  $\mathbb{R}$  which do not contain the jump times. We highlight that this is the strongest type of convergence we could expect and a uniform convergence over all intervals of the form  $[0,T]$  is impossible given that the limiting process  $\tilde{\beta}^{\circ}$  is discontinuous. In Proposition 3, we give an even stronger result by showing a graph convergence of the iterates which takes into account the path followed between the jumps. We also point out that we can easily show the same type of convergence for the accelerated weights  $\tilde{w}_t^\alpha \coloneqq w_{t^\alpha (t)}^\alpha$ . Indeed, using the bijective mapping which links the weights  $w_{t}$  and the predictors  $\beta_{t}$  (see Lemma 1 in Appendix C), we immediately get that the accelerated weights  $(\tilde{u}^{\alpha},\tilde{v}^{\alpha})$  uniformly converge towards the limiting process  $(\sqrt{|\tilde{\beta}^{\circ}|},\mathrm{sign}(\tilde{\beta}^{\circ})\sqrt{|\tilde{\beta}^{\circ}|})$  on any compact subset of  $\mathbb{R}_{\geq 0}\backslash \{t_1,\ldots ,t_p\}$ .
Estimates for the non-accelerated iterates  $\beta_{t}^{\alpha}$ . We point out that our result provides no speed of convergence of  $\tilde{\beta}^{\alpha}$  towards  $\tilde{\beta}^{\circ}$ . We believe that a non-asymptotic result is challenging and leave it as future work. Note that we experimentally notice that the convergence rate quickly degrades after each saddle. Nonetheless, we can still write for the non-accelerated iterates that  $\beta_{t}^{\alpha} = \tilde{\beta}_{t / \ln (1 / \alpha)}^{\alpha} \sim \tilde{\beta}_{t / \ln (1 / \alpha)}^{\circ}$  as  $\alpha \to 0$ . Hence, for  $\alpha$  small enough the iterates  $\beta_{t}^{\alpha}$  are roughly equal to 0 until time  $t_1 \cdot \ln (1 / \alpha)$  and the minimum  $\ell_1$ -norm interpolator is reached at time  $t_p \cdot \ln (1 / \alpha)$ . Such a precise estimate of the global convergence time is rather remarkable and goes beyond classical Lyapunov analyses which only leads to  $L(\beta_{t}^{\alpha}) \lesssim \ln (1 / \alpha) / t$  (see Proposition 4 in Appendix C).
Natural extensions of our setting. More general initialisations can easily be dealt with. For instance, initialisations of the form  $u_{t=0} = \alpha \mathbf{u}_0 \in \mathbb{R}^d$  lead to the exact same result as it is shown in [50] (Discussion after Theorem 1) that the associated mirror still converges to the  $\ell_1$ -norm. Initialisations of the form  $[u_{t=0}]_i = \alpha^{k_i}$ , where  $k_i > 0$ , lead to the associated potential converging towards a weighted  $\ell_1$ -norm and one should modify Algorithm 1 by accordingly weighting  $\nabla L(\beta)$  in the algorithm. Also, deeper linear architectures of the form  $\beta_w = w_+^D - w_-^D$  as in [50] do not change our result as the associated mirror still converges towards the  $\ell_1$ -norm. Though we only consider the square loss in the paper, we believe that all our results should hold for any loss of the type  $L(\beta) = \sum_{i=1}^{n} \ell(y_i, \langle x_i, \beta \rangle)$  where for all  $y \in \mathbb{R}$ ,  $\ell(y, \cdot)$  is strictly convex with a unique minimiser at  $y$ . In fact, the only property which cannot directly be adapted from our results is showing the uniform boundedness of the iterates (see discussion before Proposition 5 in Appendix C).
4.1 High level sketch of proof of  $\tilde{\beta}^{\alpha}\rightarrow \tilde{\beta}^{\circ}$  which leverages an arc-length parametrisation
In this section, we give the high level ideas concerning the proof of the convergence  $\tilde{\beta}^{\alpha}\rightarrow \tilde{\beta}^{\circ}$  given in Theorem 2. A full and detailed proof can be found in Appendix E. The main difficulty stems from the non-continuity of the limit process  $\tilde{\beta}^{\circ}$ . To circumvent this difficulty, a clever trick which we borrow to [18, 36] is to "slow-down" time when the jumps occur by considering an arc-length parametrisation of the path. We consider the  $\mathbb{R}_{\geq 0}$  arclength bijection  $\tau^{\alpha}$  and leverage it to define the 'appropriately slowed down' iterates  $\hat{\beta}_{\tau}^{\alpha}$  as:
$$
\hat {\beta} _ {\tau} ^ {\alpha} = \tilde {\beta} _ {\hat {t} ^ {\alpha} (\tau)} ^ {\alpha} \quad \text {w h e r e} \quad \hat {t} _ {\tau} ^ {\alpha} = \left(\tau^ {\alpha}\right) ^ {- 1} (\tau) \quad \text {a n d} \quad \tau^ {\alpha} (t) = t + \int_ {0} ^ {t} \| \dot {\hat {\beta}} _ {s} ^ {\alpha} \| \mathrm {d} s.
$$
This time reparametrisation has the fortunate but crucial property of leading to  $\dot{\hat{t}}^{\alpha}(\tau) + \| \hat{\beta}_{\tau}^{\alpha}\| = 1$  by a simple chain rule, which means that the speed of  $(\hat{\beta}_{\tau}^{\alpha})_{\tau}$  is uniformly upperbounded by 1 independently of  $\alpha$ . This behaviour is in stark contrast with the process  $(\tilde{\beta}_t^\alpha)_t$  which has a speed which explodes at the jumps. This change of time now allows us to use Arzelà-Ascoli's theorem to extract a subsequence which uniformly converges to a limiting process which we denote  $\hat{\beta}$ . Importantly,  $\hat{\beta}$  enables to keep track of the path followed between the jumps as we show that its trajectory has two regimes:
$$
\begin{array}{l} \text {S a d d l e s :} \hat {\beta} _ {\tau} = \beta_ {k} \qquad \text {C o n n e c t i o n s :} \dot {\hat {\beta}} _ {\tau} = - \frac {| \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau})}{\| | \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau}) \|}. \end{array}
$$
The process  $\hat{\beta}$  is illustrated on the right: the red curves correspond to the paths which the iterates follow during the jumps. These paths are called heteroclinic orbits in the dynamical systems literature [31, 3]. To prove Theorem 2, we can map back the convergence of  $\hat{\beta}^{\alpha}$  to show that of  $\tilde{\beta}^{\alpha}$ . Moreover from the convergence  $\hat{\beta}^{\alpha} \to \tilde{\beta}$  we get a more complete picture of the limiting dynamics of  $\tilde{\beta}^{\alpha}$  as it naturally implies the convergence of the graph of the iterates  $(\tilde{\beta}_t^\alpha)_t$  converges towards that of  $(\hat{\beta}_{\tau})_{\tau}$ . The graph convergence result is formalised in this last proposition.
Proposition 3. For all  $T > t_p$ , the graph of the iterates  $(\hat{\beta}_t^\alpha)_{t \leq T}$  converges to that of  $(\hat{\beta}_{\tau})_{\tau}$ :
$$
\operatorname {d i s t} \left(\left\{\tilde {\beta} _ {t} ^ {\alpha} \right\} _ {t \leq T}, \left\{\hat {\beta} _ {\tau} \right\} _ {\tau \geq 0}\right) \underset {\alpha \rightarrow 0} {\longrightarrow} 0 \quad (H a u s d o r f f. d i s t a n c e)
$$
5 Further discussion and conclusion
Link between incremental learning and saddle-to-saddle dynamics. The incremental learning phenomenon and the saddle-to-saddle process are often complementary facets of the same idea and refer to the same phenomenon. Indeed for gradient flows  $\mathrm{d}w_{t} = -\nabla F(w_{t})\mathrm{d}t$ , fixed points of the dynamics correspond to critical points of the loss. Stages with little progress in learning and minimal movement of the iterates necessarily correspond to the iterates being in the vicinity of a critical point of the loss. It turns out that in many settings (linear networks [30], matrix sensing [8, 41]), critical points are necessarily saddle points of the loss (if not global minima) and that they have a very particular structure (high sparsity, low rank, etc.). We finally note that an alternative approach to realising saddle-to-saddle dynamics is through the perturbation of the gradient flow by a vanishing noise as studied in [6].
Characterisation of the visited saddles. A common belief is that the saddle-to-saddle trajectory can be found by successively computing the direction of most negative curvature of the loss (i.e. the eigenvector corresponding to the most negative eigenvalue) and following this direction until reaching the next saddle [26]. However this statement cannot be accurate as it is inconsistent with our algorithm in our setting. In fact, it can be shown that this algorithm would match the orthogonal matching pursuit (OMP) algorithm [42, 17] which does not necessarily lead to the minimum  $\ell_1$ -norm interpolator. In [7], which is the closest to our work and the first to prove convergence of the iterates towards a piece-wise constant process, the successive saddles are entirely characterised and connected to the Lasso regularisation path in the underparameterised setting. Recently, [9] extended the diagonal linear network setting to diagonal parametrisations of the form  $f_{u\odot v}$ , but at the cost of stronger assumptions on the trajectory.
Adaptive Inverse Scale Space Method. Following the submission of our paper, we were informed that Algorithm 1 had already been proposed and analysed in the compressed sensing literature. Indeed it exactly corresponds to the Adaptive Inverse Scale Space Method (aISS) proposed in [11]. The motivations behind its study are extremely different from ours and originate from the study of Bregman iteration [12, 40, 52] which is an efficient method for solving  $\ell_1$  related minimisation problems. The so-called inverse scale space flow which corresponds to Eq. (12) in our paper can be seen as the continuous version of Bregman iteration. As in our paper, [11] show that this equation can be solved through an iterative algorithm. We refer to [51, Section 2] for further details. However we did not find any results in this literature concerning the uniqueness of the constrained minimisation problem due to Assumption 1, nor on the maximum number of iterations, the behaviour under RIP assumptions and the maximum number of active coordinates.
Subdifferential equations and rate-independent systems. As in Eq. (12), subdifferential inclusions of the form  $\nabla L(\beta_t) \in \frac{\mathrm{d}}{\mathrm{d}t} \partial h(\beta_t)$  for non-differential functions  $h$  have been studied by Attouch et al. [4] but for strongly convex functions  $h$ . In this case, the solutions are continuous and do not exhibit jumps. On another hand, [18, 36, 37] consider so-called rate-independent systems of the form  $\partial_q E(t, q_t) \in \partial h(\dot{q}_t)$  for 1-homogeneous dissipation potentials  $h$ . Examples of such systems are ubiquitous in mechanics and appear in problems related to friction, crack propagation, elastoplasticity and ferromagnetism to name a few [35, Ch. 6 for a survey]. As in our case, the main difficulty with such processes is the possible appearance of jumps when the energy  $E$  is non-convex.
Conclusion. Our study examines the behaviour of gradient flow with vanishing initialisation over diagonal linear networks. We prove that it leads to the flow jumping from a saddle point of the loss to another. Our analysis characterises each visited saddle point as well as the jumping times through an algorithm which is reminiscent of the LARS method used in the Lasso framework. There are several avenues for further exploration. The most compelling one is the extension of these techniques to broader contexts for which the implicit bias of gradient flow has not yet been understood.
Acknowledgments. S.P. would like to thank Loucas Pillaud-Vivien for introducing him to this beautiful topic and for the many insightful discussions. S.P. also thanks Quentin Rebjock for the many helpful discussions and Johan S. Wind for reaching out and providing the reference of [11]. The authors also thank Jérôme Bolte for the discussions concerning subdifferential equations, Aris Daniilidis for the reference of [32], as well as Aditya Varre and Mathieu Even for proofreading the paper.
Organisation of the Appendix.
1. In Appendix A, we give the experimental setup and provide additional experiments.
2. In Appendix B, we prove Proposition 1 and provide additional comments concerning the unicity of the minimisation problem which appears in the proposition.
3. In Appendix C, we provide some general results on the flow.
4. In Appendix D, we prove Proposition 2 and give standalone properties of Algorithm 1.
5. In Appendix E, we explain in more detail the arc-length parametrisation explained in the main text as well as prove Theorem 2 and Proposition 3.
6. In Appendix F, we provide technical lemmas which are useful to prove the main results.
A Experimental setup and additional: experiments, extension, related works.
Experimental setup and additional experiments. For each experiment we generate our dataset as  $y_{i} = \langle x_{i},\beta^{\star}\rangle$  where  $x_{i} = \mathcal{N}(0,H)$  for a diagonal covariance matrix  $H$  and  $\beta^{\star}$  is a vector of  $\mathbb{R}^d$ . Gradient descent is run with a small step size and from initialisation  $u_{t = 0} = \sqrt{2}\alpha \mathbf{1}\in \mathbb{R}^{d}$  and  $v_{t = 0} = \mathbf{0}$  for some initialisation scale  $\alpha >0$ .