SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(5) The algorithm outputs the minimum  $\ell_1$ -norm solution. Let  $\beta^{\star}$  be the output of the algorithm after  $p$  iterations. Notice that by the definition of the successive sets  $I_{k,\pm}$  and of the constraints on the minimisation problem, we have that at each iteration  $s_k \in \partial \| \beta_k\|_1$ . Therefore  $s_p \in \partial \| \beta^{\star}\|_1$ . Also, recall from part (1) of the proof that  $s_p \in \operatorname{span}(x_1, \ldots, x_n)$  which means that there exists  $r \in \mathbb{R}^n$  such that  $s_p = X^\top r$ . Putting the two together we get that  $X^\top r \in \partial \| \beta^{\star}\|_1$ , this condition along with the fact that  $L(\beta^{\star}) = \min L(\beta)$  are exactly the KKT conditions of  $\arg \min_{\beta \in \arg \min L} \| \beta \|_1$ .
To put our upperbound on the number of iterations into perspective, the worst-case number of iterations for the LARS algorithm is  $(3^d + 1)/2$  [34]. Hence Algorithm 1 has fewer iterations in the worst-case setting. Whether an exponential dependency in the dimension is inevitable for Algorithm 1 is unknown and we leave this as future work.
However, when the number of samples is much smaller than the dimension we lose the exponential dependency. Indeed, for  $\varepsilon \coloneqq n / d\leq 1 / 2$  , we have the upperbound  $\sum_{k = 0}^{n}\binom {d}{k}\leq 2^{H(\varepsilon)d}$  where  $H(\varepsilon) = -\varepsilon \log_2(\varepsilon) - (1 - \varepsilon)\log_2(1 - \varepsilon)$  is the binary entropy. Since for  $\varepsilon \leq 1 / 2$ $H(\varepsilon)\leq -2\varepsilon \log_2(\varepsilon)$  , we get the upperbound  $\sum_{k = 0}^{n}\binom{d}{k}\leq 2^{H(\varepsilon)d}\leq (\frac{d}{n})^{2n}$  , which is much better than  $2^{d}$
As mentioned several times, for general feature matrices  $X$  complex behaviours can occur with coordinates deactivating and changing sign several times. Here we show that for simple datasets which have a feature matrix  $X$  that satisfy the restricted isometry property (RIP) [14], we can simply determine the jump times and the saddles as a function of the sparse predictor which we seek to recover.
The non-realistic but enlightening extreme case of the RIP assumption is to consider that the feature matrix is such that  $X^{\top}X / n = I_{d}$ . In this case, by letting  $\beta^{\star}$  be the unique vector such that  $y = \langle x,\beta^{\star}\rangle$  and assuming that  $\beta^{\star} = (\beta_{1}^{\star},\dots ,\beta_{r}^{\star},0,\dots ,0)$  with  $|\beta_1^\star | > \dots >|\beta_r^\star | > 0$ , then the loss writes  $L(\beta) = \| \beta -\beta^{\star}\|_{2}^{2} / 2$  and one can easily check that Algorithm 1 would terminate in  $r$  loops and output exactly  $t_i = \frac{1}{|\beta_i^\star|}$  and  $\beta_{i} = (\beta_{1}^{\star},\ldots ,\beta_{i}^{\star},0,\ldots ,0)$  for  $i\leq r$  (the case where several coordinates of  $\beta^{\star}$  are strictly equal can also be treated: for example if  $\beta_1^\star = \beta_2^\star$  then the first output of the algorithm is directly  $\beta_{1} = (\beta_{1}^{\star},\beta_{2}^{\star},0,\dots ,0)$ ).
We now recall the more realistic RIP setting which is an adaptation of the previous observation.
Sparse regression with RIP and gap assumption. (RIP) Assume that there exists an  $r$ -sparse vector  $\beta^{\star}$  such that  $y_{i} = \langle x_{i},\beta^{\star}\rangle$ . Furthermore we assume that the feature matrix  $X\in \mathbb{R}^{n,d}$  satisfies the  $2r$ -restricted isometry property with constant  $\tilde{\varepsilon} < \sqrt{2} -1 < 1 / 2$ : i.e. for all submatrix  $X_{s}$  where we extract any  $s\leq 2r$  columns of  $X$ , the matrix  $X_{s}^{\top}X_{s} / n$  of size  $s\times s$  has all its eigenvalues in the interval  $[1 - \tilde{\varepsilon},1 + \tilde{\varepsilon} ]$ . (Gap assumption) Furthermore we assume that the  $r$ -sparse vector  $\beta^{\star}$
has coordinates which have a "sufficient gap". W.l.o.g we write  $\beta^{\star} = (\beta_{1}^{\star},\dots,\beta_{r}^{\star},0,\dots,0)$  with  $|\beta_1^\star |\geq \ldots \geq |\beta_r^\star | > 0$  and we define  $\lambda \coloneqq \min_{i\in [r]}(|\beta_i^\star | - |\beta_{i + 1}^\star |)\geq 0$  which corresponds to the smallest gap between the entries of  $|\beta^{\star}|$ . We assume that  $5\tilde{\varepsilon}\| \beta^{\star}\|_{2} < \lambda /2$  and we let  $\varepsilon \coloneqq 5\tilde{\varepsilon}$ .
A classic result from compressed sensing (see Candes [13, Theorem 1.2]) is that the  $2r$ -restricted isometry property with constant  $\sqrt{2} - 1$  ensures that the minimum  $\ell_0$ -minimisation problem has a unique  $r$ -sparse solution which is  $\beta^{\star}$ . Furthermore it ensures that the minimum  $\ell_1$ -norm solution is unique and is equal to  $\beta^{\star}$ . This means that Algorithm 1 will have  $\beta^{\star}$  as a final output.
We now recall the result which characterises the outputs of Algorithm 1 when the data satisfies the previous assumptions.
$$
\text {w i t h} \left\{ \begin{array}{l} \beta_ {2} [ 1 ] \in [ \beta_ {1} ^ {\star} - \varepsilon \| \beta^ {\star} \|, \beta_ {1} ^ {\star} + \varepsilon \| \beta^ {\star} \| ] \\ \beta_ {2} [ 2 ] \in [ \beta_ {2} ^ {\star} - \varepsilon \| \beta^ {\star} \|, \beta_ {2} ^ {\star} + \varepsilon \| \beta^ {\star} \| ] \end{array} \right.
$$
at times  $t_1, \ldots, t_r$  such that  $t_i \in \left[\frac{1}{|\beta_i^\star| + \varepsilon\|\beta^\star\|}, \frac{1}{|\beta_i^\star| - \varepsilon\|\beta^\star\|}\right]$  and where  $\|\cdot\|$  denotes the  $\ell_2$  norm.
Proof. In all the proof  $\| \cdot \|$  denotes the  $\ell_2$  norm  $\| \cdot \|_2$ . For simplicity we assume that  $\beta_i^\star > 0$  for all  $i \in [r]$ , the proof can easily be adapted to the general case. We first define  $\xi := X^\top X / n - I_d$ . By the restricted isometry property, for any  $k \leq 2r$ , we have that any  $k \times k$  square matrix extracted from  $\xi$  which we denote  $\xi_{kk}$  has its eigenvalues in  $[-\tilde{\varepsilon}, \tilde{\varepsilon}]$ . It also means that the eigenvalues of  $(I_k + \xi_{kk})^{-1} - I_k$  are in  $[\frac{1}{1 + \tilde{\varepsilon}} - 1, \frac{1}{1 - \tilde{\varepsilon}} - 1] \subset [-2\tilde{\varepsilon}, 2\tilde{\varepsilon}]$ .
We now proceed by induction with the following induction hypothesis:
-  $\beta_{k - 1}$  has its support on its  $(k - 1)$  first coordinates with  $|\beta_{k - 1}[i] - \beta_i^\star |\leq 5\tilde{\varepsilon} ||\beta^\star ||$  for  $i < k$
$t_k \in \left[\frac{1}{\beta_k^\star + 5\tilde{\varepsilon}\|\beta^\star\|}, \frac{1}{\beta_k^\star - 5\tilde{\varepsilon}\|\beta^\star\|}\right]$  and  $s_{t_k}[k] = 1$
-  $s_{t_k}[i] \in [t_k(\beta_i^\star - 5\tilde{\varepsilon} \| \beta^\star \|), t_k(\beta_i^\star + 5\tilde{\varepsilon} \| \beta^\star \|)] \subset (-1, 1)$  for  $i > k$
From the recurrence hypothesis, the output of the algorithm at step  $k$  is hence  $\beta_{k} = \arg \min L(\beta)$  under the constraint  $\beta [i] \geq 0$  for  $i \leq k$  and  $\beta [i] = 0$  otherwise. We first search for the solution of the minimisation problem without the sign constraint and still (abusively) denote it  $\beta_{k}$ : we will show that it turns out to satisfy the sign constraint and that it is therefore indeed  $\beta_{k}$ .
In the following, for a vector  $v$ , we denote by  $v[: k]$  its  $k$  first coordinates. Setting the  $k$  first coordinates of the gradient to 0, we get that  $[X^{\top} X(\beta_k - \beta^\star)][:k] = \mathbf{0}$ , which leads to  $(I_k + \xi_{kk})\beta_k[:k] = \beta^\star[:k] + [\xi\beta^\star][:k]$ , which gives:
$$
\begin{array}{l} \beta_ {k} [: k ] = \left(I _ {k} + \xi_ {k k}\right) ^ {- 1} \left(\beta^ {\star} [: k ] + \left[ \xi \beta^ {\star} \right] [: k ]\right) \\ = \beta^ {\star} [: k ] + [ \xi \beta^ {\star} ] [: k ] + v _ {1} \\ \end{array}
$$
where from the bound on the eigenvalues of  $(I_k + \xi_{kk})^{-1} - I_k$  and  $\| \xi \beta^{\star}\| \leq \tilde{\varepsilon}\| \beta^{\star}\|$
$$
\begin{array}{l} \left\| v _ {1} \right\| \leq 2 \tilde {\varepsilon} \left\| \beta^ {\star} [: k ] + \left[ \xi \beta^ {\star} \right] [: k ]\right) \| \\ \leq 2 \tilde {\varepsilon} (\| \beta^ {\star} \| + \| \xi \beta^ {\star} \|) \\ \leq 2 \tilde {\varepsilon} \left(\| \beta^ {\star} \| + \tilde {\varepsilon} \| \beta^ {\star} \|\right) \\ \leq 4 \tilde {\varepsilon} \| \beta^ {\star} \|. \\ \end{array}
$$
$$
\beta_ {k} [: k ] = \beta^ {\star} [: k ] + v _ {2}
$$
where  $v_{2} = [\xi \beta^{\star}][:k] + v_{1}$  hence  $\| v_{2}\|_{\infty}\leq \| v_{2}\| \leq 5\tilde{\varepsilon}\| \beta^{\star}\|$ . Notice that from the definition of  $\lambda$  and the fact that  $5\tilde{\varepsilon}\| \beta^{\star}\| < \lambda /2$  we have that  $\beta_{k}[:k]\geq 0$  coordinate-wise, hence verifying the sign constraint. Also note that  $\| \beta_k\| \leq \| \beta^\star \| +5\tilde{\varepsilon}\| \beta^\star \| \leq 4\| \beta^\star \|$ .
For  $t \geq t_k$ ,  $s_t = s_{t_k} - (t - t_k)\nabla L(\beta_k)$ , and  $[\nabla L(\beta_k)][:k] = 0$  therefore  $s_t[:k] = s_{t_k[:k]}$ . Now for  $i > k$ ,  $[- \nabla L(\beta_k)]_i = n^{-1}[X^\top X(\beta^\star - \beta_k)]_i = \beta_i^\star + [\xi (\beta_k - \beta^\star)]_i$ . Now since  $(\beta_k - \beta^\star)$  is  $r$ -sparse we have that:
$$
\begin{array}{l} \| \xi (\beta_ {k} - \beta^ {\star}) \| _ {\infty} \leq \| \xi (\beta_ {k} - \beta^ {\star}) \| \\ \leq \tilde {\varepsilon} \| \beta_ {k} - \beta^ {\star} \| \\ \leq \tilde {\varepsilon} (\| \beta_ {k} \| + \| \beta^ {\star} \|) \\ \leq 5 \tilde {\varepsilon} \| \beta^ {\star} \| <   \lambda / 2, \tag {21} \\ \end{array}
$$
Now from the fact that  $s_t[i] = s_{t_k}[i] + (t - t_k)\beta_i^\star + (t - t_k)[\xi(\beta_k - \beta^\star)]_i$  and using the recurrence hypothesis:  $s_{t_k}[i] \in [t_k(\beta_i^\star - 5\tilde{\varepsilon} \| \beta^\star \|), t_k(\beta_i^\star + 5\tilde{\varepsilon} \| \beta^\star \|)]$ , we get (using the bound Eq. (21)) that  $s_t[i] \in [t(\beta_i^\star - 5\tilde{\varepsilon} \| \beta^\star \|), t(\beta_i^\star + 5\tilde{\varepsilon} \| \beta^\star \|)]$ . From the "separation assumption" we have that  $5\tilde{\varepsilon} \| \beta^\star \| < \lambda / 2$  and therefore the next coordinate to activate is necessarily the  $(k + 1)^{th}$  at time  $t_{k+1}$  with  $s_{t_{k+1}}[k + 1] = 1$  and:
$$
t _ {k + 1} \in \left[ \frac {1}{\beta_ {k + 1} ^ {\star} + 5 \tilde {\varepsilon} \| \beta^ {\star} \|}, \frac {1}{\beta_ {k + 1} ^ {\star} - 5 \tilde {\varepsilon} \| \beta^ {\star} \|} \right].
$$
This proves the recursion. The algorithm cannot stop before iteration  $r$  as  $\beta^{\star}$  is the unique minimiser of  $L$  that has at most  $r$  non-zero coordinates. But it stops at iteration  $r$  as  $\beta^{\star}$  is the unique minimiser of  $L(\beta)$  under the constraints  $\beta_{i} \geq 0$  for  $i \leq r$  and  $\beta_{i} = 0$  otherwise.
E Proof of Theorem 2 and Proposition 3 through the arc-length parametrisation
In this section, we explain in more details the arc-length reparametrisation which circumvents the apparition of discontinuous jumps and leads to the proof of Theorem 2. The main difficulty to show the convergence stems from the non-continuity of the limit process  $\tilde{\beta}^{\circ}$ . Therefore we cannot expect uniform convergence of  $\tilde{\beta}^{\alpha}$  towards  $\tilde{\beta}$  as  $\alpha \to 0$ . In addition,  $\tilde{\beta}^{\circ}$  does not provide any insights into the path followed between the jumps.
Arc-length parametrisation. The high-level idea is to "slow-down" time when the jumps occur. To do so we follow the approach from [18, 36] and we consider an arc-length parametrisation of the path, i.e., we consider  $\tau^{\alpha}$  equal to:
$$
\tau^ {\alpha} (t) = t + \int_ {0} ^ {t} \| \dot {\hat {\beta}} _ {s} ^ {\alpha} \| \mathrm {d} s.
$$
In Proposition 6, we showed that the full path length  $\int_0^{+\infty}||\dot{\beta}_s^\alpha ||\mathrm{d}s$  is finite and bounded independently of  $\alpha$ . Therefore  $\tau^{\alpha}$  is a bijection in  $\mathbb{R}_{\geq 0}$ . We can then define the following quantities:
$$
\hat {t} _ {\tau} ^ {\alpha} = (\tau^ {\alpha}) ^ {- 1} (\tau) \quad \text {a n d} \quad \hat {\beta} _ {\tau} ^ {\alpha} = \tilde {\beta} _ {\hat {t} ^ {\alpha} (\tau)} ^ {\alpha}.
$$
By construction, a simple chain rule leads to  $\hat{t}^{\alpha}(\tau) + \| \hat{\beta}_{\tau}^{\alpha}\| = 1$ , which means that the speed of  $(\hat{\beta}_{\tau}^{\alpha})_{\tau}$  is always upperbounded by 1, independently of  $\alpha$ . This behaviour is in stark contrast with the process  $(\tilde{\beta}_t^\alpha)_t$  which has a speed which explodes at the jumps. It presents a major advantage as we can now use Arzelà-Ascoli's theorem to extract a converging subsequent. A simple change of variable shows that the new process satisfies the following equations:
$$
- \int_ {0} ^ {\tau} \dot {\hat {t}} _ {s} ^ {\alpha} \nabla L \left(\hat {\beta} _ {s} ^ {\alpha}\right) \mathrm {d} s = \nabla \tilde {\phi} _ {\alpha} \left(\hat {\beta} _ {\tau} ^ {\alpha}\right) \quad \text {a n d} \quad \dot {\hat {t}} _ {\tau} ^ {\alpha} + \| \dot {\hat {\beta}} _ {\tau} ^ {\alpha} \| = 1 \tag {22}
$$
started from  $\hat{\beta}_{\tau}^{\alpha} = 0$  and  $\hat{t}_0 = 0$ . The next proposition states the convergence of the rescaled process, up to a subsequence.
Proposition 8. Let  $T \geq 0$ . For every  $\alpha > 0$ , let  $(\hat{t}^{\alpha}, \hat{\beta}^{\alpha})$  be the solution of Eq. (22). Then, there exists a subsequence  $(\hat{t}^{\alpha_k}, \hat{\beta}^{\alpha_k})_{k \in \mathbb{N}}$  and  $(\hat{t}, \hat{\beta})$  such that as  $\alpha_k \to 0$ :
$$
\left(\hat {t} ^ {\alpha_ {k}}, \hat {\beta} ^ {\alpha_ {k}}\right)\rightarrow (\hat {t}, \hat {\beta}) \quad i n \left(C ^ {0} ([ 0, T ], \mathbb {R} \times \mathbb {R} ^ {d}), \| \cdot \| _ {\infty}\right) \tag {23}
$$
$$
\left(\dot {\hat {t}} ^ {\alpha_ {k}}, \dot {\hat {\beta}} ^ {\alpha_ {k}}\right)\rightharpoonup \left(\dot {\hat {t}}, \dot {\hat {\beta}}\right) \quad i n L _ {1} [ 0, T ] \tag {24}
$$
Limiting dynamics. The limits  $(\hat{t},\hat{\beta})$  satisfy:
$$
- \int_ {0} ^ {\tau} \dot {\hat {t}} _ {s} \nabla L (\hat {\beta} _ {s}) \mathrm {d} s \in \partial \| \hat {\beta} _ {\tau} \| _ {1} \quad a n d \quad \dot {\hat {t}} _ {\tau} + \| \dot {\hat {\beta}} _ {\tau} \| \leq 1 \tag {25}
$$
Heteroclinic orbit. In addition, when  $\hat{\beta}_{\tau}$  is such that  $|\hat{\beta}_{\tau}| \odot \nabla L(\hat{\beta}_{\tau}) \neq 0$ , we have
$$
\dot {\hat {\beta}} _ {\tau} = - \frac {| \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau})}{\| | \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau}) \|} \quad a n d \quad \dot {\hat {t}} _ {\tau} = 0. \tag {26}
$$
Furthermore, the loss strictly decreases along the heteroclinic orbits and the path length  $\int_0^T\| \dot{\hat{\beta}}_\tau \| \mathrm{d}\tau$  is upperbounded independently of  $T$ .
Proof. Differentiating Eq. (22) and from the Hessian of  $\tilde{\phi}_{\alpha}$  we get:
$$
\begin{array}{l} \dot {\hat {\beta}} _ {\tau} ^ {\alpha} = - \dot {\hat {t}} _ {\tau} ^ {\alpha} (\nabla^ {2} \tilde {\phi} _ {\alpha} (\hat {\beta} _ {\tau} ^ {\alpha})) ^ {- 1} \nabla L (\hat {\beta} _ {\tau} ^ {\alpha}) \\ = - (1 - \| \dot {\hat {\beta}} _ {\tau} ^ {\alpha} \|) (\nabla^ {2} \tilde {\phi} _ {\alpha} (\hat {\beta} _ {\tau} ^ {\alpha})) ^ {- 1} \nabla L (\hat {\beta} _ {\tau} ^ {\alpha}). \\ \end{array}
$$
Therefore taking the norm on the right hand side we obtain that
$$
\| \dot {\hat {\beta}} _ {\tau} ^ {\alpha} \| = \frac {\| (\nabla^ {2} \tilde {\phi} _ {\alpha} (\hat {\beta} _ {\tau} ^ {\alpha})) ^ {- 1} \nabla L (\hat {\beta} _ {\tau} ^ {\alpha}) \|}{1 + \| (\nabla^ {2} \tilde {\phi} _ {\alpha} (\hat {\beta} _ {\tau} ^ {\alpha})) ^ {- 1} \nabla L (\hat {\beta} _ {\tau} ^ {\alpha}) \|},
$$
$$
\dot {\hat {\beta}} _ {\tau} ^ {\alpha} = - \frac {\left(\nabla^ {2} \tilde {\phi} _ {\alpha} \left(\hat {\beta} _ {\tau} ^ {\alpha}\right)\right) ^ {- 1} \nabla L \left(\hat {\beta} _ {\tau} ^ {\alpha}\right)}{1 + \left\| \left(\nabla^ {2} \tilde {\phi} _ {\alpha} \left(\hat {\beta} _ {\tau} ^ {\alpha}\right)\right) ^ {- 1} \nabla L \left(\hat {\beta} _ {\tau} ^ {\alpha}\right) \right\|}. \tag {27}
$$
Subsequence extraction. By construction Eq. (22) we have  $\dot{\hat{t}}_{\tau}^{\alpha} + \|\dot{\hat{\beta}}_{\tau}^{\alpha}\| = 1$ , therefore the sequences  $(\dot{\hat{t}}^{\alpha})_{\alpha}, (\dot{\hat{\beta}}^{\alpha})_{\alpha}$  as well as  $(\hat{t}^{\alpha})_{\alpha}, (\hat{\beta}^{\alpha})_{\alpha}$  are uniformly bounded on  $[0, T]$ . The Arzelà-Ascoli theorem yields that, up to a subsequence, there exists  $(\hat{t}, \hat{\beta})$  such that  $(\hat{t}^{\alpha_k}, \hat{\beta}^{\alpha_k}) \to (\hat{t}, \hat{\beta})$  in  $(C^{0}([0, T], \mathbb{R} \times \mathbb{R}^{d}), \| \cdot \|_{\infty})$ . Since  $\|\dot{\hat{\beta}}_{\tau}^{\alpha}\|, \|\dot{\hat{t}}_{\tau}^{\alpha}\| \leq 1$  we have, applying the Banach-Alaoglu theorem, that up to a new subsequence
$$
\left(\dot {\hat {t}} ^ {\alpha_ {k}}, \dot {\hat {\beta}} ^ {\alpha_ {k}}\right) \stackrel {*} {\rightharpoonup} \left(\dot {\hat {t}}, \dot {\hat {\beta}}\right) \text {i n} L _ {\infty} (0, T) \tag {28}
$$
and  $\| \dot{\hat{\beta}}_{\tau}\| \leq \lim \inf_{\alpha_k}\| \dot{\hat{\beta}}_{\tau}^{\alpha_k}\| \leq 1$  and thus  $\dot{\hat{t}}_{\tau} + \| \dot{\hat{\beta}}_{\tau}\| \leq 1$ :
$$
\int_ {0} ^ {T} \| \dot {\hat {\beta}} _ {\tau} \| \mathrm {d} \tau \leq \int_ {0} ^ {T} \lim  _ {\alpha_ {k}} \inf  _ {\alpha_ {k}} \| \dot {\hat {\beta}} _ {\tau} ^ {\alpha_ {k}} \| \mathrm {d} \tau \leq \int_ {0} ^ {+ \infty} \lim  _ {\alpha_ {k}} \inf  _ {\alpha_ {k}} \| \dot {\hat {\beta}} _ {\tau} ^ {\alpha_ {k}} \| \mathrm {d} \tau \leq \lim  _ {\alpha_ {k}} \inf  _ {\alpha_ {k}} \int_ {0} ^ {+ \infty} \| \dot {\hat {\beta}} _ {\tau} ^ {\alpha_ {k}} \| \mathrm {d} \tau <   C,
$$
where the third inequality is by Fatou's lemma. Note that since  $[0,T]$  is bounded then it also implies the weak convergence in any  $L_{p}(0,T)$ ,  $1\leq p < \infty$ . Since  $(\hat{\beta}^{\alpha})$  converges uniformly on  $[0,T]$ , and  $\nabla L$  is continuous, we have that  $\nabla L(\hat{\beta}^{\alpha})$  converges uniformly to  $\nabla L(\hat{\beta})$ . Since  $\dot{\hat{t}}^{\alpha_k}\rightharpoonup \dot{\hat{t}}$  in  $L_{1}(0,T)$ , passing to the limit in the equation  $\nabla \tilde{\phi}_{\alpha}(\hat{\beta}_{\tau}^{\alpha}) = -\int_{0}^{\tau}\dot{\hat{t}}_{s}^{\alpha}\nabla L(\hat{\beta}_{s}^{\alpha})\mathrm{d}s$  leads to
$$
- \int_ {0} ^ {\tau} \dot {\hat {t}} _ {s} \nabla L (\hat {\beta} _ {s}) d s \in \partial \| \hat {\beta} _ {\tau} \| _ {1},
$$
due to Lemma 2.
Recall from Eq. (27) and the definition of  $\tilde{\phi}_{\alpha}$  that:
$$
\dot {\hat {\beta}} _ {\tau} ^ {\alpha} = - \frac {\sqrt {\hat {\beta} _ {\tau} ^ {\alpha} + \alpha^ {4}} \odot \nabla L \left(\hat {\beta} _ {\tau} ^ {\alpha}\right)}{1 / \ln (1 / \alpha) + \left\| \sqrt {\hat {\beta} _ {\tau} ^ {\alpha} + \alpha^ {4}} \odot \nabla L \left(\hat {\beta} _ {\tau} ^ {\alpha}\right) \right\|}. \tag {29}
$$
Hence assuming that  $\hat{\beta}_{\tau}$  is such that  $\| |\hat{\beta}_{\tau}| \odot \nabla L(\hat{\beta}_{\tau}) \| \neq 0$ , we can ensure that  $\| |\hat{\beta}_{\tau'}| \odot \nabla L(\hat{\beta}_{\tau'}) \| \neq 0$  for  $\tau' \in [\tau, \tau + \varepsilon]$  and  $\varepsilon$  small enough. We have then  $\frac{\sqrt{\hat{\beta}_{\tau'}^{\alpha} + \alpha^4} \odot \nabla L(\hat{\beta}_{\tau'})}{1 / \ln(1 / \alpha) + \|\sqrt{\hat{\beta}_{\tau'}^{\alpha} + \alpha^4} \odot \nabla L(\hat{\beta}_{\tau'})\|}$  converges uniformly toward  $-\frac{|\hat{\beta}_{\tau'}| \odot \nabla L(\hat{\beta}_{\tau'})}{\|\hat{\beta}_{\tau'}| \odot \nabla L(\hat{\beta}_{\tau'})\|}$  on  $[\tau, \tau + \varepsilon]$ . Using the dominated convergence theorem, we have  $\int_{\tau}^{\tau + \varepsilon} \frac{\sqrt{\hat{\beta}_{\tau'}^{\alpha} + \alpha^4} \odot \nabla L(\hat{\beta}_{\tau'})}{1 / \log(1 / \alpha) + \|\sqrt{\hat{\beta}_{\tau'}^{\alpha} + \alpha^4} \odot \nabla L(\hat{\beta}_{\tau'})\|} \mathrm{d}\tau' \to \int_{\tau}^{\tau + \varepsilon} \frac{|\hat{\beta}_{\tau'}| \odot \nabla L(\hat{\beta}_{\tau'})}{\|\hat{\beta}_{\tau'}| \odot \nabla L(\hat{\beta}_{\tau'})\|} \mathrm{d}\tau'$ . We therefore obtain  $\dot{\hat{\beta}}_{\tau} = -\frac{|\hat{\beta}_{\tau}| \odot \nabla L(\hat{\beta}_{\tau})}{\|\hat{\beta}_{\tau}| \odot \nabla L(\hat{\beta}_{\tau})\|}$  in  $L_1[0, T]$ . Consequently  $\| \dot{\hat{\beta}}_{\tau} \| = 1$  and  $\dot{\bar{t}}_{\tau} = 0$ .
Proof that the loss strictly decreases along the heteroclinic orbits.
Assume  $\hat{\beta}_{\tau}$  is such that  $|\hat{\beta}_{\tau}| \odot \nabla L(\hat{\beta}_{\tau}) \neq 0$ , then the flow follows
$$
\dot {\hat {\beta}} _ {\tau} = - \frac {| \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau})}{| | \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau}) | |}
$$
Letting  $\gamma (\tau) = \frac{1}{\|\hat{\beta}_{\tau}|\odot\nabla L(\hat{\beta}_{\tau})\|}$  we get:
$$
\mathrm {d} L (\hat {\beta} _ {\tau}) = - \gamma (\tau) \sum_ {i} | \hat {\beta} _ {\tau} (i) | \odot [ \nabla L (\hat {\beta} _ {\tau}) ] _ {i} ^ {2} \mathrm {d} \tau <   0,
$$
because  $|\hat{\beta}_{\tau}| \odot \nabla L(\hat{\beta}_{\tau})^2 \neq 0$ .
Borrowing terminologies from [18], we can distinguish two regimes: when  $\dot{\hat{\beta}}_{\tau} = 0$ , the system is sticked to the saddle point. When  $\dot{\hat{t}}_{\tau} = 0$  and  $\| \dot{\hat{\beta}}_{\tau}\| = 1$  the system switches to a viscous slip which follows the normalised flow Eq. (26). We use the term of heteroclinic orbit as in the dynamical systems literature since in the weight space  $(u,v)$  it corresponds to a path with links two distinct critical points of the loss  $F$ . Since  $\dot{\hat{t}}_{\tau} = 0$ , this regime happens instantly for the original  $t$  time scale (i.e. a jump occurs).
From Proposition 8, following the same reasoning as in Section 3, we can show that the rescaled process converges uniformly to a continuous saddle-to-saddle process where the saddles are linked by normalized flows.
Theorem 3. Let  $T > 0$ . For all subsequences defined in Proposition 8, there exist times  $0 = \tau_0' < \tau_1 < \tau_1' < \dots < \tau_p < \tau_p' < \tau_{p+1} = +\infty$  such that the iterates  $(\hat{\beta}_{\tau}^{\alpha_k})_\tau$  converge uniformly on  $[0,T]$  to the following limit trajectory:
$$
(" S a d d l e") \quad \hat {\beta} _ {\tau} = \beta_ {k} \quad f o r \tau \in \left[ \tau_ {k} ^ {\prime}, \tau_ {k + 1} \right] w h e r e 0 \leq k \leq p
$$
$$
\dot {\hat {\beta}} _ {\tau} = - \frac {| \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau})}{\| \hat {\beta} _ {\tau} | \odot \nabla L (\hat {\beta} _ {\tau}) \|} \quad f o r \tau \in [ \tau_ {k + 1}, \tau_ {k + 1} ^ {\prime} ] w h e r e 0 \leq k \leq p - 1
$$
where the saddles  $(\beta_0 = 0, \beta_1, \ldots, \beta_p = \beta_{\ell_1}^{\star})$  are constructed in Algorithm 1. Also, the loss  $(L(\hat{\beta}_{\tau}))_{\tau}$  is constant on the saddles and strictly decreasing on the orbits. Finally, independently of the chosen subsequence, for  $k \in [p]$  we have  $\hat{t}_{\tau_k} = \hat{t}_{\tau_k'} = t_k$  where the times  $(t_k)_{k \in [p]}$  are defined through Algorithm 1.
Proof. Some parts of the proof are slightly technical. To simplify the understanding, we make use of auxiliary lemmas which are stated in Appendix F. The overall spirit follows the intuitive ideas given in Section 3 and relies on showing that Eq. (25) can only be satisfied if the iterates visit the saddles from Algorithm 1.
We let  $\hat{s}_{\tau} := -\int_{0}^{\tau}\hat{t}_{s}\nabla L(\hat{\beta}_{s})\mathrm{d}s$ , which is continuous and satisfies  $\hat{s}_{\tau} \in \partial \| \hat{\beta}_{\tau}\|_{1}$  from Eq. (25). Let  $S = \{\beta \in \mathbb{R}^{d}, |\beta| \odot \nabla L(\beta) = 0\}$  denote the set of critical points and let  $(\beta_{k}, t_{k}, s_{k})$  be the successive values of  $(\beta, t, s)$  which appear in the loops of Algorithm 1.