SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Figure 1 and Figure 4 (Left):  $(n, d, \alpha) = (5, 7, 10^{-120})$ ,  $H = I_d$ ,  $\beta^{\star} = (10, 20, 0, 0, 0, 0, 0) \in \mathbb{R}^7$ .
- Figure 4 (Right):  $(n, d, \alpha) = (6, 6, 10^{-10})$ ,  $H = \mathrm{diag}(1, 10, 10, 10, 10, 10) \in \mathbb{R}^{6 \times 6}$ ,  $\beta^{\star} = (1, 0, 0, 0, 0, 0, 0) \in \mathbb{R}^{6}$ .
Figure 3 (Left):  $(n,d,\alpha_{1},\alpha_{2}) = (7,2,10^{-100},10^{-10}),H = I_{d},\beta^{\star} = (10,20)\in \mathbb{R}^{7}$
- Figure 3 (Right):  $(n,d,\alpha) = (3,3,10^{-100})$ ,  $X$  is the square root matrix of the matrix  $((20,6,-1.4),(6,2,-0.4),(-1.4,-0.4,0.12)) \in \mathbb{R}^{3\times 3}$ ,  $\beta^{\star} = (1,9,10)$ .
Figure 3: Left: Visualisation of the uniform convergence of  $\tilde{\beta}^{\alpha}$  towards  $\tilde{\beta}^{\circ}$  as  $\alpha \to 0$ .  $\alpha_{1} = 10^{-100} \ll \alpha_{2} = 10^{-10}$  Right: In some cases, 2 coordinates can activate at the same time. Note that the time axis is in log-scale for better visualisation.
Figure 4: Complex dynamics can occur. Left and right: Coordinates are not monotonic and the number of active coordinates neither as several coordinates can deactivate at the same time. The piecewise constant process plotted in black is the limiting process  $\tilde{\beta}^{\circ}$  predicted by our theory.
Proof. Non-existence of maxima / non-global minima. This is a simpler version of results which appear in [30], for the sake of completeness we provide here a simple proof adapted to our setting. The intuition follows the fact that if there existed a local maximum / non-global minimum for  $F$  then this would translate to the existence of a local maximum / non-global minimum for the convex loss  $L$ , which is absurd.
Assume that there exists a local maximum  $w^{\star} = (u^{\star}, v^{\star})$ , i.e. assume that there exists  $\varepsilon > 0$  such that for all  $w = (u, v)$  such that  $\| w - w^{\star} \|_2^2 \leq \varepsilon$ ,  $F(w) \leq F(w^{\star})$ . We show that this would imply that  $\beta^{\star} = u^{\star} \odot v^{\star}$  is a local maximum of  $L$ , which is absurd.
The mapping  $g:(u,v)\mapsto (u\odot v,\sqrt{(u^2 - v^2) / 2})$  from  $\mathbb{R}_{\geq 0}^{d}\times \mathbb{R}^{d}\to \mathbb{R}^{d}\times \mathbb{R}_{\geq 0}^{d}$  is a bijection with inverse
$$
g ^ {- 1}: (\beta , \alpha) \mapsto \left(\sqrt {\alpha^ {2} + \sqrt {\beta^ {2} + \alpha^ {4}}}, \operatorname {s i g n} (\beta) \odot \sqrt {- \alpha^ {2} + \sqrt {\beta^ {2} + \alpha^ {4}}}\right). \tag {14}
$$
Also notice that  $F(g^{-1}(\beta, \alpha)) = L(\beta)$  for all  $\beta$  and  $\alpha$ . Now let  $\tilde{\varepsilon} > 0$  and let  $\beta \in \mathbb{R}^d$  such that  $\| \beta - \beta^\star \|_2^2 \leq \tilde{\varepsilon}$ , then for  $(u, v) = g^{-1}(\beta, \alpha_*)$  where  $\alpha_* = \sqrt{((u^\star)^2 - (v^\star)^2)/2}$  we have that:
$$
\begin{array}{l} \left\| (u, v) - \left(u ^ {\star}, v ^ {\star}\right) \right\| _ {2} ^ {2} = 2 \left\| \left(\sqrt {\alpha_ {*} ^ {4} + \beta^ {2}} - \sqrt {\alpha_ {*} ^ {4} + \beta^ {\star 2}}\right) ^ {2} \right\| _ {1} \\ \leq 2 \| \beta^ {2} - \beta^ {\star 2} \| _ {1} \\ = 2 \| (\beta - \beta^ {\star}) ^ {2} + 2 (\beta - \beta^ {\star}) \beta^ {\star} \| _ {1} \\ \leq 2 \| (\beta - \beta^ {\star}) ^ {2} \| _ {1} + 2 \| \beta^ {\star} \| _ {\infty} \| \beta - \beta^ {\star} \| _ {1} \\ \leq 2 (1 + \sqrt {d} \| \beta^ {\star} \| _ {\infty}) \tilde {\varepsilon} \\ \leq \varepsilon \\ \end{array}
$$
where the last inequality is for  $\tilde{\varepsilon}$  small enough. This means that  $L(\beta) = F(w) \leq F(w^{\star}) = L(\beta^{\star})$  and  $\beta^{\star}$  is a local maximum of  $L$ , which is absurd.
The exact same proof holds to show that there are no local minima of  $F$  which are not global minima.
Critical points. The gradient of the loss function  $F$  writes:
$$
\nabla_ {w} F (w) = \left( \begin{array}{c} \nabla_ {u} F (w) \\ \nabla_ {v} F (w) \end{array} \right) = \left( \begin{array}{c} \nabla L (\beta) \odot v \\ \nabla L (\beta) \odot u \end{array} \right) \in \mathbb {R} ^ {2 d}.
$$
Therefore  $\nabla F(w_{c}) = \mathbf{0} \in \mathbb{R}^{2d}$  implies that  $\nabla L(\beta_{c}) \odot \beta_{c} = \mathbf{0} \in \mathbb{R}^{d}$ . Now consider such a  $\beta_{c}$  and let  $\mathrm{supp}(\beta_{c}) = \{i \in [d] \text{ such that } \beta_{c}(i) \neq 0\}$  denote the support of  $\beta_{c}$ . Since  $[\nabla L(\beta_{c})]_{i} = 0$  for  $i \notin \mathrm{supp}(\beta_{c})$ , we can therefore write that
$$
\beta_{c}\in \operatorname *{arg  min}_{\beta_{i} = 0\text{for} i\not\in\operatorname {supp}(\beta_{c})}L(\beta).
$$
Furthermore we point out that since  $\mathrm{supp}(\beta_c) \subset [d]$ , there are at most  $2^d$  distinct sets  $\mathrm{supp}(\beta_c)$ , and therefore at most  $2^d$  values  $F(w_{c}) = L(\beta_{c})$ , where  $w_{c}$  is a critical point of  $F$ .
Additional comment concerning the uniqueness of  $\arg \min_{\beta_i = 0,i\not\in\mathrm{supp}(\beta_c)}L(\beta)$
We point out that the constrained minimisation problem (4) does not necessarily have a unique solution, even when  $\beta_{c}$  is not a global solution. Though not required for any of our results, for the sake of completeness, we show here that under an additional mild assumption on the data, we can ensure that the minimisation problem (4) which appears in Proposition 1 has a unique minimum when  $L(\beta_c) > 0$ . Under this additional assumption, there is therefore a finite number of saddles  $\beta_{c}$ . Recall that we let  $X\in \mathbb{R}^{n\times d}$  be the feature matrix and  $(\tilde{x}_1,\ldots ,\tilde{x}_d)$  be its columns. Now assume temporarily that the following assumption holds.
Assumption 2 (Assumption used just in this short section). Any subset of  $(\tilde{x}_1, \ldots, \tilde{x}_d)$  of size smaller than  $\min(n, d)$  is linearly independent.
One can easily check that this assumption holds with probability 1 as soon as the data is drawn from a continuous probability distribution, similarly to [47, Lemma 4]). In the following, for a subset  $\xi = \{i_1,\ldots ,i_k\} \subset [d]$ , we write  $X_{\xi} = (\tilde{x}_{i_1},\dots,\tilde{x}_{i_k})\in \mathbb{R}^{n\times k}$  (we extract the columns from  $X$ ). For a vector  $\beta \in \mathbb{R}^d$  we write  $\beta [\xi ] = (\beta_{i_1},\dots,\beta_{i_k})$  and  $\beta [\xi^C ] = (\beta_i)_{i\notin \xi}$ . We distinguish two different settings:
- Underparametrised setting  $(n \geq d)$ : in this case, for any  $\xi = \{i_1, \ldots, i_k\} \subset [d]$ , then  $\beta^{\star} := \operatorname*{argmin}_{\beta_i = 0, i \notin \xi} L(\beta)$  is unique. Indeed we simply set the gradient to  $0$  and notice that due to Assumption 2, there exists a unique solution, indeed it is  $\beta^{\star}$  such that  $\beta^{\star}[\xi] = (X_{\xi}^{\top} X_{\xi})^{-1} X_{\xi}^{\top} y$  and  $\beta^{\star}[\xi^{C}] = 0$ .
- Overparametrised setting  $(d > n)$ : Global solutions: arg  $\min_{\beta \in \mathbb{R}^d} L(\beta)$  is an affine space spanned by the orthogonal of  $(x_1, \ldots, x_n)$  in  $\mathbb{R}^d$ . Since  $\operatorname{span}(\tilde{x}_1, \ldots, \tilde{x}_d) = \mathbb{R}^n$  from Assumption 2, any  $\beta^\star \in \arg \min_{\beta \in \mathbb{R}^d} L(\beta)$  satisfies  $X\beta^\star = y$  and  $L(\beta^\star) = 0$ . "Saddle points": now let  $\beta_c \in \mathbb{R}^d$  be such that we can write  $\beta_c \in \arg \min_{\beta_i = 0, i \notin \operatorname{supp}(\beta_c)} L(\beta)$  and assume that  $L(\beta_c) > 0$  (i.e., not a global solution), then: (1)  $\beta_c$  has at most  $n$  non-zero entries, indeed if it were not the case, then  $y$  would necessarily belong to  $\operatorname{span}(\tilde{x}_i)_{i \in \operatorname{supp}(\beta_c)}$  due to the assumption on the data, and this would lead to  $L(\beta_c) = 0$ , (2) therefore, similar to the underparametrised case, arg  $\min_{\beta_i = 0, i \notin \operatorname{supp}(\beta_c)} L(\beta)$  is unique, equal to  $\beta_c$ , and we have that  $\beta_c[\xi] = (X_\xi^\top X_\xi)^{-1} X_\xi^\top y$  and  $\beta_c[\xi^C] = 0$  where  $\xi = \operatorname{supp}(\beta_c)$ .
Thus, in both the underparametrised and overparametrised settings, the minimisation problem (4) appearing in Proposition 1 has a unique minimum when  $L(\beta_c) > 0$  and Assumption 2 holds.
C General results on the iterates
In the following lemma we recall a few results concerning the gradient flow Eq. (5):
$$
\mathrm {d} w _ {t} = - \nabla F (w _ {t}) \mathrm {d} t, \tag {15}
$$
where  $F$  is defined in Eq. (3) as:
$$
F (w) := L (u \odot v) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(\langle u \odot v, x _ {i} \rangle - y _ {i}\right) ^ {2}.
$$
Lemma 1. For an initialisation  $u_0 = \sqrt{2}\alpha$ ,  $v_0 = 0$ , the flow  $w_t^\alpha = (u_t^\alpha, v_t^\alpha)$  from Eq. (15) is such that the quantity  $(u_t^\alpha)^2 - (v_t^\alpha)^2$  is constant and equal to  $2\alpha^2\mathbf{1}$ . Furthermore  $u_t^\alpha > |v_t^\alpha| \geq 0$  and therefore from the bijection Eq. (14) we have that:
$$
u _ {t} ^ {\alpha} = \sqrt {\alpha^ {2} + \sqrt {(\beta_ {t} ^ {\alpha}) ^ {2} + \alpha^ {4}}}, v _ {t} ^ {\alpha} = \mathrm {s i g n} (\beta_ {t} ^ {\alpha}) \odot \sqrt {- \alpha^ {2} + \sqrt {(\beta_ {t} ^ {\alpha}) ^ {2} + \alpha^ {4}}}.
$$
Proof. From the expression of  $\nabla F(w)$ , notice that the derivative of  $(u_t^\alpha)^2 - (v_t^\alpha)^2$  is equal to  $\mathbf{0}$  and therefore equal to its initial value.
Since  $(u_{t}^{\alpha})^{2} - (v_{t}^{\alpha})^{2} = (u_{t}^{\alpha} + v_{t}^{\alpha})(u_{t}^{\alpha} - v_{t}^{\alpha}) > 0$ , by continuity we get that  $u_{t}^{\alpha} + v_{t}^{\alpha} > 0$  and  $u_{t}^{\alpha} - v_{t}^{\alpha} > 0$  and therefore  $u_{t}^{\alpha} > |v_{t}^{\alpha}|$ .
In this section we consider the accelerated iterates Eq. (9) which follow:
$$
\mathrm {d} \nabla \tilde {\phi} _ {\alpha} \left(\tilde {\beta} _ {t} ^ {\alpha}\right) = - \nabla L \left(\tilde {\beta} _ {t} ^ {\alpha}\right) \mathrm {d} t, \quad \text {w h e r e} \quad \tilde {\phi} _ {\alpha} := \frac {1}{\ln (1 / \alpha)} \cdot \tilde {\phi} _ {\alpha} \tag {16}
$$
with  $\tilde{\beta}_{t = 0} = \mathbf{0}$  and where  $\phi_{\alpha}$  is defined Eq. (7).
Proposition 4. For all  $\alpha >0$  and minimum  $\beta^{\star}\in \arg \min_{\beta}L(\beta)$ , the loss values  $L(\tilde{\beta}_t^\alpha)$  and the Bregman divergence  $\mathrm{D}_{\tilde{\phi}_{\alpha}}(\beta^{\star},\tilde{\beta}_{t}^{\alpha})$  are decreasing. Moreover
$$
L \left(\tilde {\beta} _ {t} ^ {\alpha}\right) - L \left(\beta^ {\star}\right) \leq \frac {\tilde {\phi} _ {\alpha} \left(\beta^ {\star}\right)}{2 t}, \tag {17}
$$
$$
L \left(\frac {1}{t} \int_ {0} ^ {t} \tilde {\beta} _ {s} ^ {\alpha} \mathrm {d} s\right) - L \left(\beta^ {\star}\right) \leq \frac {\tilde {\phi} _ {\alpha} \left(\beta^ {\star}\right)}{2 t}. \tag {18}
$$
Proof. The loss is decreasing since:  $\frac{\mathrm{d}}{\mathrm{d}t} L(\tilde{\beta}_t^\alpha) = \nabla L(\tilde{\beta}_t^\alpha)^\top \dot{\beta}_t^\alpha = -\dot{\tilde{\beta}}_t^\alpha^\top \nabla^2 \tilde{\phi}_\alpha (\tilde{\beta}_t^\alpha) \dot{\tilde{\beta}}_t^\alpha \leq 0$ .
$\frac{\mathrm{d}}{\mathrm{d}t}\mathrm{D}_{\tilde{\phi}_{\alpha}}(\beta^{\star},\tilde{\beta}_{t}^{\alpha}) = -\nabla L(\tilde{\beta}_{t}^{\alpha})^{\top}(\tilde{\beta}_{t}^{\alpha} - \beta^{\star}) = -2(L(\tilde{\beta}_{t}^{\alpha}) - L(\beta^{\star}))$  (since  $L$  is the quadratic loss), therefore the Bregman distance is decreasing. We can also integrate this last equality from 0 to  $t$ , and divide by  $-2t$ :
$$
\begin{array}{l} \frac {1}{t} \int_ {0} ^ {t} L (\tilde {\beta} _ {s} ^ {\alpha}) \mathrm {d} s - L (\beta^ {\star}) = \frac {\mathrm {D} _ {\tilde {\phi} _ {\alpha}} (\beta^ {\star} , \beta_ {0} ^ {\alpha} = \mathbf {0}) - \mathrm {D} _ {\tilde {\phi} _ {\alpha}} (\beta^ {\star} , \beta_ {t} ^ {\alpha})}{2 t} \\ \leq \frac {\tilde {\phi} _ {\alpha} (\beta^ {\star})}{2 t}. \\ \end{array}
$$
Since the loss is decreasing we get that  $L(\tilde{\beta}_t^\alpha) - L(\beta^\star) \leq \frac{\tilde{\phi}_\alpha(\beta^\star)}{2t}$  and from the convexity of  $L$  we get that  $L\left(\frac{1}{t}\int_{0}^{t}\tilde{\beta}_{s}^{\alpha}\mathrm{d}s\right) - L(\beta^{\star}) \leq \frac{\tilde{\phi}_{\alpha}(\beta^{\star})}{2t}$ .
In the following proposition, we show that for  $\alpha$  small enough, the iterates are bounded independently of  $\alpha$ . Note that this result unfortunately only holds for the quadratic loss, we expect it to hold for other convex losses of the type  $L(\beta) = \frac{1}{n}\sum_{i}\ell(y_i,\langle x_i,\beta\rangle)$  where  $\ell(y,\cdot)$  is strictly convex has a unique root at  $y$  but we don't know how to show it. Also note that bounding the accelerated iterates  $\tilde{\beta}^{\alpha}$  is equivalent to bounding the iterates  $\beta^{\alpha}$  since  $\tilde{\beta}_t^\alpha = \beta_{\ln(1/\alpha)t}^\alpha$ .
Proposition 5. For  $\alpha < \alpha_0$ , where  $\alpha_0$  depends on  $\beta_{\ell_1}^{\star}$ , the iterates  $\tilde{\beta}_t^\alpha$  are bounded independently of  $\alpha$ :
$$
\left\| \tilde {\beta} _ {t} ^ {\alpha} \right\| _ {\infty} \leq 3 \left\| \beta_ {\ell_ {1}} ^ {\star} \right\| _ {1} + 1
$$
Proof. From Eq. (16), integrating and using that  $L$  is the quadratic loss, we get:
$$
\nabla \tilde {\phi} _ {\alpha} (\tilde {\beta} _ {t} ^ {\alpha}) = \frac {t}{n} X ^ {\top} (y - X \bar {\beta} _ {t} ^ {\alpha}) = - \frac {t}{n} X ^ {\top} X (\bar {\beta} _ {t} ^ {\alpha} - \beta^ {\star}),
$$
where we recall that  $X \in \mathbb{R}^{n \times d}$  is the input data represented as a matrix and where we denote the averaged iterate by  $\tilde{\beta}_t^\alpha = \frac{1}{t} \int_0^t \tilde{\beta}_s^\alpha \, \mathrm{d}s$ . Thus we get
$$
\nabla \tilde {\phi} _ {\alpha} \left(\tilde {\beta} _ {t} ^ {\alpha}\right) ^ {\top} \left(\tilde {\beta} _ {t} ^ {\alpha} - \beta^ {\star}\right) = - \frac {t}{n} \left(\bar {\beta} _ {t} ^ {\alpha} - \beta^ {\star}\right) ^ {\top} X ^ {\top} X \left(\tilde {\beta} _ {t} ^ {\alpha} - \beta^ {\star}\right). \tag {19}
$$
By convexity of  $\tilde{\phi}_{\alpha}$  we have  $\tilde{\phi}_{\alpha}(\beta_{t}^{\alpha}) - \tilde{\phi}_{\alpha}(\beta^{\star}) \leq \nabla \tilde{\phi}_{\alpha}(\beta_{t}^{\alpha})^{\top}(\beta_{t}^{\alpha} - \beta^{\star})$ . By the Cauchy-Schwarz inequality, we also have  $(\bar{\beta}_t^\alpha -\beta^\star)^\top X^\top X(\beta_t^\alpha -\beta^\star) \leq \| X(\beta_t^\alpha -\beta^\star)\| \| X(\bar{\beta}_t^\alpha -\beta^\star)\|$ . Using Proposition 4:  $\| X(\beta_t^\alpha -\beta^\star)\| ^2\leq n\tilde{\phi}_{\alpha}(\beta^{\star}) / t$  and  $\| X(\bar{\beta}_t^\alpha -\beta^\star)\| ^2\leq n\tilde{\phi}_{\alpha}(\beta^{\star}) / t$  we can further bound the right hand side of Eq. (19) as
$$
- \frac {t}{n} \left(\bar {\beta} _ {t} ^ {\alpha} - \beta^ {\star}\right) ^ {\top} X ^ {\top} X \left(\beta_ {t} ^ {\alpha} - \beta^ {\star}\right) \leq \tilde {\phi} _ {\alpha} \left(\beta^ {\star}\right).
$$
Thus it yields
$$
\tilde {\phi} _ {\alpha} \left(\beta_ {t} ^ {\alpha}\right) - \tilde {\phi} _ {\alpha} \left(\beta^ {\star}\right) \leq \tilde {\phi} _ {\alpha} \left(\beta^ {\star}\right).
$$
From [50] (proof of Lemma 1 in the appendix) we get that for
$$
\alpha <   \min \left\{1, \sqrt {\| \beta \| _ {1}}, (2 \| \beta \| _ {1}) ^ {- 1} \right\}
$$
$$
\tilde {\phi} _ {\alpha} (\beta) \leq \frac {3}{2} \| \beta \| _ {1},
$$
and for all  $\alpha < \exp(-d/2)$ :
$$
\begin{array}{l} \tilde {\phi} _ {\alpha} (\beta) \geq \| \beta \| _ {1} - \frac {d}{\ln (1 / \alpha^ {2})} \\ \geq \| \beta \| _ {1} - 1, \\ \end{array}
$$
which finally leads for
$$
\alpha <   \alpha_ {0} := \min  \left\{1, \sqrt {\| \beta_ {\ell_ {1}} ^ {\star} \| _ {1}}, \left(2 \| \beta_ {\ell_ {1}} ^ {\star} \| _ {1}\right) ^ {- 1}, \exp (- d / 2) \right\}
$$
to the result.
The following proposition shows that we can bound the path length of the flow  $\tilde{\beta}^{\alpha}$  independently of  $\alpha$ . Keep in mind that the path length of  $\tilde{\beta}^{\alpha}$  is equivalent to that of  $\beta^{\alpha}$  as the first is just an acceleration of the second:  $\tilde{\beta}_t^\alpha = \beta_{\ln (1 / \alpha)}^\alpha t$ .
Proposition 6. For  $\alpha < \alpha_0$  where  $\alpha_0$  is the same as in Proposition 5, the path length of the iterates  $(\beta_t^\alpha)_{t \geq 0}$  is bounded independently of  $\alpha > 0$ :
$$
\int_ {0} ^ {+ \infty} \| \dot {\beta} _ {t} ^ {\alpha} \| \mathrm {d} t <   C,
$$
where  $C$  does not depend on  $\alpha$ . Hence the path length of the accelerated flow  $\tilde{\beta}^{\alpha}$  is also bounded independently of  $\alpha$ .
Proof. Having shown that the iterates  $\beta_t^\alpha$  are bounded independently of  $\alpha$ , it also implies that the iterates  $w_t = (u_t, v_t)$  are bounded following Lemma 1. Since the loss  $w \mapsto F(w)$  is a multivariate polynomial function, it is a semialgebraic function and we can consequently apply the result of Kurdyka [32, Theorem 2] which grants that
$$
\int_ {0} ^ {+ \infty} \| \dot {w} _ {t} \| \mathrm {d} t <   C,
$$
where the constant  $C$  only depends on the loss and on the bound on the iterates. We further use that  $\dot{\beta} = \dot{u}\odot v + u\odot \dot{v}$  and  $\| \dot{u}\odot v + u\odot \dot{v}\| \leq C_1(\| \dot{u}\| +\| \dot{v}\|)$  using that  $u$  and  $v$  are bounded and  $\| \dot{u}\| +\| \dot{v}\| \leq C_2\| \dot{w}\|$  using the equivalence of norms. Therefore  $\int_0^{+\infty}\| \dot{\beta}_t^\alpha \| \mathrm{d}t < C$  for some  $C$  which is independent of the initialisation scale  $\alpha$ .
D Standalone properties of Algorithm 1
D.1 "Well-definedness" of Algorithm 1 and upperbound on its number of loops
Notice that this proposition highlights the fact that Algorithm 1 is on its own an algorithm of interest for finding the minimum  $\ell_1$ -norm solution in an overparametrised regression setting. We point out that the provided upperbound on the number of iterations is very crude and could certainly be improved.
Proposition 7. Algorithm 1 is well defined: at each iteration (i) the attribution of  $\Delta$  is well defined as  $\Delta < +\infty$ , (ii) the constrained minimisation problem has a unique solution and the attribution of the value of  $\beta$  is therefore well-founded. Furthermore, along the loops: the iterates  $\beta$  have at most  $n$  non-zero coordinates, the loss is strictly decreasing and the algorithm terminates in at most  $\min(2^d, \sum_{k=0}^{n} \binom{d}{k})$  steps by outputting the minimum  $\ell_1$ -norm solution  $\beta_{\ell_1}^\star := \underset{\beta \in \arg \min L}{\arg \min} \| \beta \|_1$ .
Proof. In the following, for the matrix  $X$  and for a subset  $I = \{i_1,\dots ,i_k\} \subset [d]$ , we write  $X_{I} = (\tilde{x}_{i_{1}},\ldots ,\tilde{x}_{i_{k}})\in \mathbb{R}^{n\times k}$  (we extract the columns from  $X$ ). For a vector  $\beta \in \mathbb{R}^d$  we write  $\beta_{I} = (\beta_{i_{1}},\dots ,\beta_{i_{k}})$ .
(1) The constrained minimisation problem has a unique solution: we follow the proof of [47, Lemma 2]. Following the notations in Algorithm 1, we define  $I = \{i \in [d], |s_i| = 1\}$  and we point out that after  $k$  loops of the algorithm, the value of  $s$  is equal to  $s = -(\Delta_1 \nabla L(\beta_0) + \dots + \Delta_k \nabla L(\beta_{k-1})) \in \operatorname{span}(x_1, \ldots, x_n)$ . We can therefore write  $s = X^\top r$  for some  $r \in \mathbb{R}^n$ .
Now assume that  $\ker(X_I) \neq \{0\}$ . Then, for some  $i \in I$ , we have  $\tilde{x}_i = \sum_{j \in I \setminus \{i\}} c_j \tilde{x}_j$  where  $c_j \in \mathbb{R}$ . Without loss of generality, we can assume that  $I \setminus \{i\}$  has at most  $n$  elements. Indeed, we can otherwise always find  $n$  elements  $\tilde{I} \subset I \setminus \{i\}$  such that  $\tilde{x}_i = \sum_{j \in \tilde{I}} c_j \tilde{x}_j$ . Rewriting the previous equality, we get
$$
s _ {i} \tilde {x} _ {i} = \sum_ {j \in I \backslash \{i \}} \left(s _ {i} s _ {j} c _ {j}\right) \left(s _ {j} \tilde {x} _ {j}\right). \tag {20}
$$
Now by definitions of the set  $I$  and of  $r$ , we have that  $\langle \tilde{x}_j, r \rangle = s_j \in \{+1, -1\}$  for any  $j \in I$ . Taking the inner product of Eq. (20) with  $r$ , we obtain that  $1 = \sum_{j \in I \setminus \{i\}} (s_i s_j c_j)$ . Consequently, we have shown that if  $\ker(X_I) \neq \{0\}$ , then we necessarily have for some  $i \in I$ ,
$$
s _ {i} \tilde {x} _ {i} = \sum_ {j \in I \setminus \{i \}} a _ {j} (s _ {j} \tilde {x} _ {j}),
$$
with  $\sum_{j\in I\setminus \{i\}}a_j = 1$ , which means that  $s_i\tilde{x}_i$  lies in the affine space generated by  $(s_j\tilde{x}_j)_{j\in I\setminus \{i\}}$ . This fact is however impossible due to Assumption 1 (recall that without loss of generality we have that  $I\setminus \{i\}$  has at most  $n$  elements, and trivially less than  $d$  elements). Therefore  $X_I$  is full rank, and  $\operatorname {Card}(I)\leq n$ . Now notice that the constrained minimisation problem corresponds to  $\arg \min_{\substack{\beta_i\geq 0,i\in I_+\\ \beta_i\leq 0,i\in I_-}}\| y - X_I\beta_I\| _2^2$ . Since  $X_I$  is full rank, this restricted loss is strictly convex and the constrained minimisation problem has a unique minimum.
(2)  $\Delta < +\infty$ : Notice that the optimality conditions of
$$
\beta = \operatorname *{arg  min}_{\substack{\beta_{i}\geq 0,i\in I_{+}\\ \beta_{i}\leq 0,i\in I_{-}\\ \beta_{i} = 0,i\notin I}}\| y - X_{I}\beta_{I}\|_{2}^{2},
$$
are (i)  $\beta$  satisfies the constraints, (ii) if  $i\in I_{+}$  (resp  $i\in I_{-}$ ) then  $[- \nabla L(\beta)]_i\leq 0$  (resp  $[- \nabla L(\beta)]_i\geq 0)$  and (iii) if  $\beta_{i}\neq 0$  then  $[\nabla L(\beta)]_i = 0$ . One can notice that condition (ii) ensures that at each iteration, for  $\delta \leq \Delta_k$ ,  $s_{k - 1} - \delta \nabla L(\beta_{k - 1})\in [-1,1]$  coordinate wise. Also, if  $L(\beta_{k - 1})\neq 0$ , then a coordinate of the vector  $|s_{k - 1} - \delta \nabla L(\beta_{k - 1})|$  must necessarily hit 1, this value of  $\delta$  corresponds to  $\Delta_k$ .
(3) The loss is strictly decreasing: Let  $I_{k-1,\pm}$  and  $I_{k,\pm}$  be the equicorrelation sets defined in the algorithm at step  $k-1$  and  $k$ , and  $\beta_{k-1}$  and  $\beta_k$  the solutions of the constrained minimisation problems. Also, let  $i_k$  be the newly added coordinate which breaks the constraint at step  $k$  (which we assume to be unique for simplicity). Without loss of generality, assume that  $s_k(i_k) = +1$ . Since the sets
$I_{k - 1, + }\setminus (I_{k, + }\setminus \{i_k\})$  and  $I_{k - 1, - }\setminus I_{k, - }$  are (if not empty) only composed of indexes of coordinates of  $\beta_{k - 1}$  which are equal to 0, one can notice that  $\beta_{k - 1}$  also satisfies the new constraints at step  $k$ . Therefore  $L(\beta_{k})\leq L(\beta_{k - 1})$ . Now since  $[- \nabla L(\beta_{k - 1})]_{i_k} > 0$ , from the strict convexity of the restricted loss on  $I_{k}$ , this means that  $\beta_{k}(i_{k}) > 0$  (which also means that newly activated coordinate  $i_k$  must activate), and therefore  $\beta_{k - 1}\neq \beta_{k}$  and  $L(\beta_{k}) < L(\beta_{k - 1})$ .
(4) The algorithm terminates in at most  $\min \left(2^d, \sum_{k=0}^{n} \binom{d}{k}\right)$  steps: Recall that we showed in part (1) of the proof that at each iteration  $k$  of the algorithm,  $I_k$  as at most  $\min(n, d)$  elements. Since  $\operatorname{supp}(\beta_k) \subset I_k$ , we have that  $\beta_k$  has at most  $\min(n, d)$  non-zero elements, also recall that we always have  $\beta_k = \arg \min_{\beta_i = 0, i \notin \operatorname{supp}(\beta_k)} L(\beta)$  (we here have unicity of this minimisation problem following part (1) of the proof). There are hence at most
$$
\sum_ {k = 0} ^ {\min  (n, d)} \binom {d} {k} = \min  \left(2 ^ {d}, \sum_ {k = 0} ^ {n} \binom {d} {k}\right)
$$
such minimisation problems. The loss being strictly decreasing, the algorithm cannot output the same solution  $\beta$  at two different loops, and the algorithm must terminate in at most  $\min \left(2^d,\sum_{k = 0}^{n}\binom {d}{k}\right)$  iterations by outputting a vector  $\beta^{\star}$  such that  $\nabla L(\beta^{\star}) = 0$ , i.e.  $\beta^{\star}\in \arg \min L(\beta)$ .