SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We do a proof by induction: we start by assuming that the iterates are stuck at the saddle  $\beta_{k - 1}$  at time  $\tau \geq \tau_{k - 1}^{\prime}$  where  $\hat{t}_{\tau_{k - 1}^{\prime}} = t_{k - 1}$  and  $\hat{s}_{\tau_{k - 1}^{\prime}} = s_{k - 1}$  (recurrence hypothesis), we then show that they can only move at a time  $\tau_{k}$  and follow the normalised flow Eq. (26). We finally show that they must end up "stuck" at the new critical point  $\beta_{k}$ , validating the recurrence hypothesis.
Proof of the jump time  $\tau_{k}$  such that  $\hat{t}_{\tau_k} = t_k$  : we set ourselves at time  $\tau \geq \tau_{k - 1}^{\prime}$ , stuck at the saddle  $\beta_{k - 1}$ . Let  $\tau_{k}\coloneqq \sup \{\tau ,\hat{t}_{\tau}\leq t_{k}\}$ , we have that  $\tau_{k} < \infty$  from Lemma 3. Note that by continuity of  $\hat{t}_{\tau}$  it holds that  $\hat{t}_{\tau_k} = t_k$ . Now notice that  $\hat{s}_{\tau} = \hat{s}_{\tau_{k - 1}^{\prime}} - (\hat{t}_{\tau} - \hat{t}_{\tau_{k - 1}^{\prime}})\nabla L(\beta_{k - 1}) = s_{k - 1} - (\hat{t}_{\tau} - t_{k - 1})\nabla L(\beta_{k - 1})$ . We argue that for any  $\varepsilon >0$ , we cannot have  $\hat{\beta}_{\tau} = \beta_{k - 1}$  on  $(\tau_{k},\tau_{k} + \varepsilon)$ . Indeed by the definition of  $\tau_{k}$  and from the algorithmic construction of time  $t_k$ , it would lead to  $|\hat{s}_{\tau}(i)| > 1$  for some coordinate  $i\in [d]$ , which contradicts Eq. (25). Therefore the iterates must move at the time  $\tau_{k}$ .
Heterocline leaving  $\beta_{k-1}$  for  $\tau \in [\tau_k, \tau_k']$ : contrary to before, our time rescaling enables to capture what happens during the "jump". We have shown that for any  $\varepsilon$ , there exists  $\tau_\varepsilon \in (\tau_k, \tau_k + \varepsilon)$ , such that  $\hat{\beta}_{\tau_\varepsilon} \neq \beta_{k-1}$ . From Lemma 4, since the saddles are distinct along the flow, we must have that  $\hat{\beta}_{\tau_\varepsilon} \notin S$  for  $\varepsilon$  small enough. The iterates therefore follow a heterocline flow leaving  $\beta_{k-1}$  with a speed of 1 given by Eq. (26). We now define  $\tau_k' := \inf \{\tau > \tau_k, \exists \varepsilon_0 > 0, \forall \varepsilon \in [0, \varepsilon_0], \hat{\beta}_{\tau + \varepsilon} \in S\}$  which corresponds to the time at which the iterates reach a new critical point and stay there for at least a small time  $\varepsilon_0$ . We have just shown that  $\tau_k' > \tau_k$ . Now from Proposition 8, the path length of  $\hat{\beta}$  is finite, and from Lemma 4 the flow visits a finite number of distinct saddles at a speed of 1. These two arguments put together, we get that  $\tau_k' < +\infty$  and also  $\hat{\beta}_{\tau_k' + \varepsilon} = \hat{\beta}_{\tau_k'}$ ,  $\forall \varepsilon \in [0, \varepsilon_0]$ . On another note, since  $\dot{\hat{t}}_\tau = 0$  for  $\tau \in [\tau_k, \tau_k']$  we have  $\hat{t}_{\tau_k'} = \hat{t}_{\tau_k}(= t_k)$  as well as  $\hat{s}_{\tau_k} = \hat{s}_{\tau_k'}(= s_k)$ .
Proof of the landing point  $\beta_{k}$ : we now want to find to which saddle  $\hat{\beta}_{\tau_k'} \in S$  the iterates have moved to. To that end, we consider the following sets which also appear in Algorithm 1:
$$
I _ {\pm , k} := \left\{i \in \{1, \dots , d \}, \text {s . t .} \hat {s} _ {\tau_ {k} ^ {\prime}} (i) = \pm 1 \right\} \quad \text {a n d} \quad I _ {k} = I _ {+, k} \cup I _ {-, k}. \tag {30}
$$
The set  $I_{k}$  corresponds to the coordinates of  $\hat{\beta}_{\tau_k'}$  which "are allowed" (but not obliged) to be activated (i.e. non-zero). For  $\tau \in [\tau_k', \tau_k' + \varepsilon_0]$  we have that  $\hat{s}_{\tau} = \hat{s}_{\tau_k'} - (\hat{t}_{\tau} - t_k)\nabla L(\hat{\beta}_{\tau_k'})$ . By continuity of  $\hat{s}$  and the fact that  $\hat{s}_{\tau} \in \partial \| \hat{\beta}_{\tau_k'}\|_1$ , the equality translates into:
if  $i\notin I_k,\hat{\beta}_{\tau_k'}(i) = 0$
- if  $i \in I_{+,k}$ , then  $[\nabla L(\hat{\beta}_{\tau_k'})]_i \geq 0$  and  $\hat{\beta}_{\tau_k'}(i) \geq 0$
- if  $i \in I_{-,k}$ , then  $[\nabla L(\hat{\beta}_{\tau_k'})]_i \leq 0$  and  $\hat{\beta}_{\tau_k'}(i) \leq 0$
for  $i\in I_k$  , if  $\hat{\beta}_{\tau_k^{\prime}}(i)\neq 0$  , then  $[\nabla L(\hat{\beta}_{\tau_k^{\prime}})]_i = 0$
One can then notice that these conditions exactly correspond to the optimality conditions of the following constrained minimisation problem:
$\arg \min$  （20  $L(\beta)$  (31)
$$
\begin{array}{l} \beta_ {i} \geq 0, i \in I _ {k, +} \\ \beta_ {i} \leq 0, i \in I _ {k, -} \\ \beta_ {i} = 0, i \notin I _ {k} \\ \end{array}
$$
We showed in Proposition 7 that the solution to this problem is unique and equal to  $\beta_{k}$  from Algorithm 1. Therefore  $\hat{\beta}_{\tau} = \beta_{k}$  for  $\tau \in [\tau_k',\tau_k' + \varepsilon_0]$ . It finally remains to show that  $\hat{\beta}_{\tau} = \beta_{k}$  while  $\tau \leq \tau_{k + 1}$ , where  $\tau_{k + 1} := \sup \{\tau ,\hat{t}_{\tau} = t_{k + 1}\}$ . For this let  $\tau \in [\tau_k',\tau_{k + 1}]$ , notice that for  $i\notin I_k$ , we necessarily have that  $\hat{\beta}_{\tau}(i) = \beta_{k}(i) = 0$ , otherwise we break the continuity of  $\hat{s}_{\tau}$ . Similarly, for  $i\in I_{k, + }$ , we necessarily have that  $\hat{\beta}_{\tau}(i)\geq 0$  and for  $i\in I_{k, - },\hat{\beta}_{\tau}(i)\leq 0$  for the same continuity reasons. Now assume that  $\hat{\beta}_{\tau}(I_k)\neq \beta_k(I_k)$ . Then from Lemma 4 and continuity of the flow,  $\exists \tau^{\prime}\in (\tau_{k}^{\prime},\tau)$  such that  $\hat{\beta}_{\tau^{\prime}}\notin S$  and there must exist a heterocline flow Eq. (26) starting from  $\beta_{k}$  which passes through  $\beta_{\tau '}$ . This is absurd since along this flow the loss strictly decreases, which is in contradiction with the definition of  $\beta_{k}$  which minimises the problem Eq. (31).
Theorem 3 enables to prove without difficulty Theorem 2 which we recall below. Indeed we can show that any extracted limit  $\hat{\beta}$  maps back to the unique discontinuous process  $\tilde{\beta}^{\circ}$ .
(Saddles)  $\tilde{\beta}_t^\circ = \beta_k$  for  $t\in (t_k,t_{k + 1})$  and  $0\leq k\leq p$ $t_{p + 1} = +\infty$
Proof. We directly apply Theorem 3, let  $\alpha_{k}$  be the subsequence from the theorem. Let  $\varepsilon >0$ , for simplicity we prove the result on  $[t_1 + \varepsilon ,t_2 - \varepsilon ]$ , all the other compacts easily follow the same line of proof. Note that since  $\hat{t}^{\alpha_k}(\tau_1')\to t_1$  and  $\hat{t}^{\alpha_k}(\tau_2)\to t_2$ , for  $\alpha_{k}$  small enough  $\hat{t}^{\alpha_k}(\tau_1')\leq t_1 + \varepsilon$  and  $\hat{t}^{\alpha_k}(\tau_2)\geq t_2 - \varepsilon$ , by the monotonicity of  $\tau^{\alpha_k}$ , this means that for  $\alpha_{k}$  small enough,  $\tau_1'\leq \tau^{\alpha_k}(t_1 + \varepsilon)$  and  $\tau_{2}\geq \tau^{\alpha_{k}}(t_{2} - \varepsilon)$ . Therefore
$$
\begin{array}{l} \sup  _ {t \in [ t _ {1} + \varepsilon , t _ {2} - \varepsilon ]} \| \tilde {\beta} _ {t} ^ {\alpha_ {k}} - \beta_ {1} \| = \sup  _ {t \in [ t _ {1} + \varepsilon , t _ {2} - \varepsilon ]} \| \hat {\beta} ^ {\alpha_ {k}} (\tau_ {\alpha_ {k}} (t)) - \beta_ {1} \| \\ = \sup  _ {\tau \in \left[ \tau^ {\alpha_ {k}} \left(t _ {1} + \varepsilon\right), \tau^ {\alpha_ {k}} \left(t _ {2} - \varepsilon\right) \right]} \| \hat {\beta} ^ {\alpha_ {k}} (\tau) - \beta_ {1} \| \\ \leq \sup  _ {\tau \in \left[ \tau_ {1} ^ {\prime}, \tau_ {2} \right]} \| \hat {\beta} ^ {\alpha_ {k}} (\tau) - \beta_ {1} \|, \\ \end{array}
$$
which goes uniformly to 0 following Theorem 3. Since this result is independent of the subsequence  $\alpha_{k}$ , we get the result of Theorem 2.
We restate and prove Proposition 3 below.
Proposition 3. For all  $T > t_p$ , the graph of the iterates  $(\tilde{\beta}_t^\alpha)_{t \leq T}$  converges to that of  $(\hat{\beta}_{\tau})_{\tau}$ :
Proof. For  $\alpha$  small enough, we have that  $\hat{t}_{\tau_p'}^\alpha \leq t_p + \varepsilon \leq T$
$$
\begin{array}{l} \sup  _ {\tau \geq 0} d (\hat {\beta} _ {\tau}, \{\tilde {\beta} _ {t} ^ {\alpha} \} _ {t \leq T}) = \sup  _ {\tau \leq \tau_ {p} ^ {\prime}} d (\hat {\beta} _ {\tau}, \{\tilde {\beta} _ {t} ^ {\alpha} \} _ {t \leq T}) \\ \leq \sup _ {\tau \leq \tau_ {p} ^ {\prime}} \| \hat {\beta} _ {\tau} - \tilde {\beta} _ {\hat {t} _ {\tau} ^ {\alpha}} \| \\ = \sup _ {\tau \leq \tau_ {p} ^ {\prime}} \| \hat {\beta} _ {\tau} - \hat {\beta} _ {\tau} ^ {\alpha} \| \underset {\alpha \to 0} {\longrightarrow} 0, \\ \end{array}
$$
according to Theorem 3.
$$
\begin{array}{l} \sup  _ {t \leq T} d (\tilde {\beta} _ {t} ^ {\alpha}, \{\hat {\beta} _ {\tau^ {\prime}} \} _ {\tau^ {\prime}}) = \sup  _ {\tau \leq \tau_ {T} ^ {\alpha}} d (\hat {\beta} _ {\tau} ^ {\alpha}, \{\hat {\beta} _ {\tau^ {\prime}} \} _ {\tau^ {\prime}}) \\ \leq \sup  _ {\tau \leq \tau_ {T} ^ {\alpha}} \| \hat {\beta} _ {\tau} ^ {\alpha} - \hat {\beta} _ {\tau} \| _ {\alpha \to 0} \quad 0, \\ \end{array}
$$
according to Theorem 3, which concludes the proof.
F Technical lemmas
The following lemma describes the behaviour of  $\nabla \tilde{\phi}_{\alpha}(\beta^{\alpha})$  as  $\alpha \to 0$  in function of the subdifferential  $\partial \| \cdot \| _1$ .
Lemma 2. Let  $(\beta^{\alpha})_{\alpha >0}$  such that  $\beta^{\alpha}\underset {\alpha \to 0}{\longrightarrow}\beta \in \mathbb{R}^{d}$
- if  $\beta_{i} > 0$  then  $[\nabla \tilde{\phi}_{\alpha}(\beta^{\alpha})]_i$  converges to 1
- if  $\beta_{i} < 0$  then  $[\nabla \tilde{\phi}_{\alpha}(\beta^{\alpha})]_{i}$  converges to  $-1$ .
Moreover if we assume that  $\nabla \tilde{\phi}_{\alpha}(\beta^{\alpha})$  converges to  $\eta \in \mathbb{R}^d$ , we have that:
-  $\eta_{i} \in (-1, 1) \Rightarrow \beta_{i} = 0$
-  $\beta_{i} = 0 \Rightarrow \eta_{i} \in [-1,1]$ .
Overall, assuming that  $(\beta^{\alpha},\nabla \tilde{\phi}_{\alpha}(\beta^{\alpha}))\underset {\alpha \to 0}{\longrightarrow}(\beta ,\eta)$ , we can write:
$$
\eta \in \partial \| \beta \| _ {1}.
$$
$$
\begin{array}{l} [ \nabla \tilde {\phi} _ {\alpha} (\beta^ {\alpha}) ] _ {i} = \frac {1}{2 \ln (1 / \alpha)} \operatorname {a r c s i n h} \left(\frac {\beta_ {i} ^ {\alpha}}{\alpha^ {2}}\right) \\ = \frac {1}{2 \ln (1 / \alpha)} \ln \left(\frac {\beta_ {i} ^ {\alpha}}{\alpha^ {2}} + \sqrt {\frac {(\beta_ {i} ^ {\alpha}) ^ {2}}{\alpha^ {4}} + 1}\right). \\ \end{array}
$$
Now assume that  $\beta_i^\alpha \to \beta_i > 0$ , then  $[\nabla \tilde{\phi}_\alpha (\beta^\alpha)]_i\to 1$ , if  $\beta_{i} < 0$  we conclude using that  $\arcsinh$  is an odd function. All the claims are simple consequences of this.
The following lemma shows that the extracted limits  $\hat{t}$  as defined in Proposition 8 diverge to  $\infty$ . This divergence is crucial as it implies that the rescaled iterates  $(\hat{\beta}_{\tau})_{\tau}$  explore the whole trajectory..
Lemma 3. For any extracted limit  $\hat{t}$  as defined in Proposition 8, we have that  $\tau - C \leq \hat{t}_{\tau}$  where  $C$  is the upperbound on the length of the curves defined in proposition 6.
$$
\tau^ {\alpha} (t) = t + \int_ {0} ^ {t} \| \dot {\bar {\beta}} _ {s} ^ {\alpha} \| d s.
$$
From Proposition 6, the full path length  $\int_0^{+\infty}||\dot{\beta}_s^\alpha ||\mathrm{d}s$  is finite and bounded by some constant  $C$  independently of  $\alpha$ . Therefore  $\tau^{\alpha}$  is a bijection in  $\mathbb{R}_{\geq 0}$  and we defined  $\hat{t}_{\tau}^{\alpha} = (\tau^{\alpha})^{-1}(\tau)$ . Furthermore  $\tau^{\alpha}(t)\leq t + C$  leads to  $t\leq \hat{t}^{\alpha}(t + C)$  and therefore  $\tau -C\leq \hat{t}^{\alpha}(\tau)$  for all  $\tau \geq 0$ . This inequality still holds for any converging subsequence, which proves the result.
Under a mild additional assumption on the data (see Assumption 2), we showed after the proof of Proposition 1 in Appendix B that the number of saddles of  $F$  is finite. Without this assumption, the number of saddles is a priori not finite. However the following lemma shows that along the flow of  $\hat{\beta}$  the number of saddles which can potentially be visited is indeed finite.
Lemma 4. The limiting flow  $\hat{\beta}$  as defined in Proposition 8 can only visit a finite number of critical points  $\beta \in S \coloneqq \{\beta \in \mathbb{R}^d, \beta \odot \nabla L(\beta) = \mathbf{0}\}$  and can visit each one of them at most once.
Proof. Let  $\tau \geq 0$ , and assume that  $\hat{\beta}_{\tau} \in S$ , i.e., we are at a critical point at time  $\tau$ . From Proposition 1, we have that
$$
\begin{array}{l} \hat {\beta} _ {\tau} \in \quad \arg \min  \quad L (\beta), \tag {32} \\ \beta_ {i} = 0 \text {f o r} i \notin \operatorname {s u p p} (\hat {\beta} _ {\tau}) \\ \end{array}
$$
Let us define the sets
$$
I _ {\pm} := \{i \in \{1, \dots , d \}, \text {s . t .} \hat {s} _ {\tau} (i) = \pm 1 \} \quad \text {a n d} \quad I = I _ {+} \cup I _ {-}.
$$
The set  $I$  corresponds to the coordinate of  $\hat{\beta}_{\tau}$  which "are allowed" (but not obliged) to be non-zero since from Eq. (25),  $\mathrm{supp}(\hat{\beta}_{\tau}) \subset I$ . Now given the fact that the sub-matrix  $X_{I} = (\tilde{x}_{i})_{i \in I} \in \mathbb{R}^{n \times \operatorname{card}(I)}$  is full rank (see part (1) of the proof of Proposition 7 for the explanation), the solution of the minimisation problem (32) is unique and equal to  $\beta[\xi] = (X_{\xi}^{\top} X_{\xi})^{-1} X_{\xi}^{\top} y$  and  $\beta[\xi^{C}] = 0$  where  $\xi = \mathrm{supp}(\hat{\beta}_{\tau})$ . There are  $2^{d} = \mathrm{Card}(P([d]))$  (where  $P([d])$  contains all the subsets of  $[d]$ ) number of constraints of the form  $\{\beta_{i} = 0, i \notin \mathcal{A}\}$ , where  $\mathcal{A} \subset [d]$ , and  $\hat{\beta}_{\tau}$  is the unique solution of one of them.  $\hat{\beta}_{\tau}$  can therefore take at most  $2^{d}$  values (very crude upperbound). There is therefore a finite number of critical points which can be reached by the flow  $\hat{\beta}$ . Furthermore, from Proposition 8, the loss is strictly decreasing along the heteroclinic orbits, each of these critical points can therefore be visited at most once.