SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Scott Pesme EPFL
scott.pesme@epfl.ch
Nicolas Flammarion EPFL
In this paper we fully describe the trajectory of gradient flow over 2-layer diagonal linear networks for the regression setting in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minimum  $\ell_1$ -norm solution. We explicitly characterise the visited saddles as well as the jump times through a recursive algorithm reminiscent of the LARS algorithm used for computing the Lasso path. Starting from the zero vector, coordinates are successively activated until the minimum  $\ell_1$ -norm solution is recovered, revealing an incremental learning. Our proof leverages a convenient arc-length time-reparametrisation which enables to keep track of the transitions between the jumps. Our analysis requires negligible assumptions on the data, applies to both under and overparametrised settings and covers complex cases where there is no monotonicity of the number of active coordinates. We provide numerical experiments to support our findings.
Strikingly simple algorithms such as gradient descent are driving forces for deep learning and have led to remarkable empirical results. Nonetheless, understanding the performances of such methods remains a challenging and exciting mystery: (i) their global convergence on highly non-convex losses is far from being trivial and (ii) the fact that they lead to solutions which generalise well [53] is still not fully understood.
To explain this second point, a major line of work has focused on the concept of implicit regularisation: amongst the infinite space of zero-loss solutions, the optimisation process must be implicitly biased towards solutions which have good generalisation properties for the considered real-world prediction tasks. Many papers have therefore shown that gradient methods have the fortunate property of asymptotically leading to solutions which have a well-behaving structure [38, 24, 16].
Aside from these results which mostly focus on characterising the asymptotic solution, a slightly different point of view has been to try to describe the full trajectory. Indeed it has been experimentally observed that gradient methods with small initialisations have the property of learning models of increasing complexity across the training of neural networks [29]. This behaviour is usually referred to as incremental learning or as a saddle-to-saddle process and describes learning curves which are piecewise constant: the training process makes very little progress for some time, followed by a sharp transition where a new "feature" is suddenly learned. In terms of optimisation trajectory, this corresponds to the iterates "jumping" from a saddle of the training loss to another.
Several settings exhibiting such dynamics for small initialisation have been considered: matrix and tensor factorisation [44, 27], simplified versions of diagonal linear networks [23, 7], linear networks [22, 45, 26], 2-layer neural networks with orthogonal inputs [10], learning leap functions with 2-layer neural networks [1] and matrix sensing [2, 33, 28]. However, all these results require
Figure 1: Gradient flow  $(\beta_{t}^{\alpha})_{t}$  with small initialisation scale  $\alpha$  over a 2-layer diagonal linear network (for the precise experimental setting, see Appendix A). Left: Training loss across time, the learning is piecewise constant. Middle: The magnitudes of the coordinates are plotted across time: the process is piecewise constant. Right: In the  $\mathbb{R}^3$  space in which the iterates evolve (the remaining coordinates stay at 0), the iterates jump from a saddle of the training loss to another. The jumping times  $t_i$  as well as the visited saddles  $\beta_{i}$  are entirely predicted by our theory.
restrictive assumptions on the data or only characterise the first jump. Obtaining a complete picture of the saddle-to-saddle process by describing all the visited saddles and jump times is mathematically challenging and still missing. We intend to fill this gap by considering diagonal linear networks which are simplified neural networks that have received significant attention lately [50, 48, 25, 43, 20] as they are ideal proxy models for gaining a deeper understanding of complex phenomenons such as saddle-to-saddle dynamics.
1.1 Informal statement of the main result
In this paper, we provide a full description of the trajectory of gradient flow over 2-layer diagonal linear networks in the limit of vanishing initialisation. The main result is informally presented here.
Theorem 1 (Main result, informal). In the regression setting and in the limit of vanishing initialisation, the trajectory of gradient flow over a 2-layer diagonal linear network converges towards a limiting process which is piecewise constant: the iterates successively jump from a saddle of the training loss to another, each visited saddle and jump time can recursively be computed through an algorithm (Algorithm 1) reminiscent of the LARS algorithm for the Lasso.
The incremental learning stems from the particular structure of the saddles as they correspond to minimisers of the training loss with a constraint on the set of non-zero coordinates. The saddles therefore correspond to sparse vectors which partially fit the dataset. For simple datasets, a consequence of our main result is that the limiting trajectory successively starts from the zero vector and successively learns the support of the sparse ground truth vector until reaching it. However, we make minimal assumptions on the data and our analysis also holds for complex datasets. In that case, the successive active sets are not necessarily increasing in size and coordinates can deactivate as well as activate until reaching the minimum  $\ell_1$ -norm solution (see Figure 1 (middle) for an example of a deactivating coordinate). The regression setting and the diagonal network architecture are introduced in Section 2. Section 3 provides an intuitive construction of the limiting saddle-to-saddle dynamics and presents the algorithm that characterises it. Our main result regarding the convergence of the iterates towards this process is presented in Section 4 and further discussion is provided in Section 5.
2 Problem setup and leveraging the mirror structure
Linear regression. We study a linear regression problem with inputs  $(x_{1},\ldots ,x_{n})\in (\mathbb{R}^{d})^{n}$  and outputs  $(y_{1},\dots ,y_{n})\in \mathbb{R}^{n}$ . We consider the typical quadratic loss:
$$
L (\beta) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(\left\langle \beta , x _ {i} \right\rangle - y _ {i}\right) ^ {2}. \tag {1}
$$
We make no assumption on the number of samples  $n$  nor the dimension  $d$ . The only assumption we make on the data throughout the paper is that the inputs  $(x_{1},\ldots ,x_{n})$  are in general position. In order to state this assumption, let  $X\in \mathbb{R}^{n\times d}$  be the feature matrix whose  $i^{th}$  row is  $x_{i}$  and let  $\tilde{x}_j\in \mathbb{R}^n$  be its  $j^{th}$  column for  $j\in [d]$ .
Assumption 1 (General position). For any  $k \leq \min(n, d)$  and arbitrary signs  $\sigma_1, \ldots, \sigma_k \in \{-1, 1\}$ , the affine span of any  $k$  points  $\sigma_1 \tilde{x}_{j_1}, \ldots, \sigma_k \tilde{x}_{j_k}$  does not contain any element of the set  $\{\pm \tilde{x}_j, j \neq j_1, \ldots, j_k\}$ .
This assumption is slightly technical but is standard in the Lasso literature [47]. Note that it is not restrictive as it is almost surely satisfied when the data is drawn from a continuous probability distribution [47, Lemma 4]. Letting  $S = \arg \min_{\beta}L(\beta)$  denote the affine space of solutions, Assumption 1 ensures that the minimisation problem  $\min_{\beta^{\star}\in S}\| \beta^{\star}\| _1$  has a unique minimiser which we denote  $\beta_{\ell_1}^{\star}$  and which corresponds to the minimum  $\ell_1$ -norm solution.
2-layer diagonal linear network. In an effort to understand the training dynamics of neural networks, we consider a 2-layer diagonal linear network which corresponds to writing the regression vector  $\beta$  as
$$
\beta_ {w} = u \odot v \text {w h e r e} w = (u, v) \in \mathbb {R} ^ {2 d}. \tag {2}
$$
This parametrisation can be interpreted as a simple neural network  $x \mapsto \langle u, \sigma(\mathrm{diag}(v)x) \rangle$  where  $u$  are the output weights, the diagonal matrix  $\mathrm{diag}(v)$  represents the inner weights, and the activation  $\sigma$  is the identity function. We refer to  $w = (u,v) \in \mathbb{R}^{2d}$  as the weights and to  $\beta := u \odot v \in \mathbb{R}^d$  as the prediction parameter. With the parametrisation (2), the loss function  $F$  over the parameters  $w = (u,v) \in \mathbb{R}^{2d}$  is defined as:
$$
F (w) := L (u \odot v) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(\left\langle u \odot v, x _ {i} \right\rangle - y _ {i}\right) ^ {2}. \tag {3}
$$
Though this reparametrisation is simple, the associated optimisation problem is non-convex and highly non-trivial training dynamics already occur. The critical points of the function  $F$  exhibit a very particular structure, as highlighted in the following proposition proven in Appendix B.
Proposition 1. All the critical points  $w_{c}$  of  $F$  which are not global minima, i.e.,  $\nabla F(w_{c}) = \mathbf{0}$  and  $F(w_{c}) > \min_{w} F(w)$ , are necessarily saddle points (i.e., not local extrema). They map to parameters  $\beta_{c} = u_{c} \odot v_{c}$  which satisfy  $|\beta_{c}| \odot \nabla L(\beta_{c}) = \mathbf{0}$  and:
$$
\beta_ {c} \in \underset {\beta [ i ] = 0 \text {f o r} i \notin \operatorname {s u p p} \left(\beta_ {c}\right)} {\arg \min } L (\beta) \tag {4}
$$
where  $\mathrm{supp}(\beta_c) = \{i\in [d],\beta_c[i]\neq 0\}$  corresponds to the support of  $\beta_{c}$
The optimisation problem in Eq. (4) states that the saddle points of the train loss  $F$  correspond to sparse vectors that minimise the loss function  $L$  over its non-zero coordinates. This property already shows that the saddle points possess interesting properties from a learning perspective. In the following we loosely use the term of 'saddle' to refer to points  $\beta_c \in \mathbb{R}^d$  solution of Eq. (4) that are not saddles of the convex loss function  $L$ . We adopt this terminology because they correspond to points  $w_c \in \mathbb{R}^{2d}$  that are indeed saddles of the non-convex loss  $F$ .
Gradient Flow and necessity of "accelerating" time. We minimise the loss  $F$  using gradient flow:
$$
\mathrm {d} w _ {t} = - \nabla F (w _ {t}) \mathrm {d} t, \tag {5}
$$
initialised at  $u_0 = \sqrt{2}\alpha \mathbf{1} \in \mathbb{R}_{>0}^d$  with  $\alpha > 0$ , and  $v_0 = \mathbf{0} \in \mathbb{R}^d$ . This initialisation results in  $\beta_0 = \mathbf{0} \in \mathbb{R}^d$  independently of the chosen weight initialisation scale  $\alpha$ . We denote  $\beta_t^\alpha \coloneqq u_t^\alpha \odot v_t^\alpha$  by the prediction iterates generated from the gradient flow to highlight its dependency on the initialisation scale  $\alpha^1$ . The origin  $\mathbf{0} \in \mathbb{R}^{2d}$  is a critical point of the function  $F$  and taking the initialisation  $\alpha \to 0$  therefore arbitrarily slows down the dynamics. In fact, it can be easily shown for any fixed time  $t$ , that  $(u_t^\alpha, v_t^\alpha) \to \mathbf{0}$  as  $\alpha \to 0$ , indicating that the iterates are stuck at the origin. Therefore if we restrict ourselves to a finite time analysis, there is no hope of exhibiting the observed saddle-to-saddle behaviour. To do so, we must find an appropriate bijection  $\tilde{t}_\alpha$  in  $\mathbb{R}_{\geq 0}$  which "accelerates" time (i.e.,  $\tilde{t}_\alpha(t) \underset{\alpha \to 0}{\longrightarrow} +\infty$  for all  $t$ ) and consider the accelerated iterates  $\beta_{\tilde{t}_\alpha(t)}^\alpha$  which can escape the saddles. Finding this bijection becomes very natural once the mirror structure is unveiled.
<sup>1</sup>We point out that the trajectory of  $\beta_t^\alpha$  exactly matches that of another common parametrisation  $\beta_w \coloneqq \frac{1}{2} (w_+^2 - w_-^2)$ , with initialisation  $w_{+,0} = w_{-,0} = \alpha \mathbf{1}$ .
2.2 Leveraging the mirror flow structure
While the iterates  $(w_{t}^{\alpha})_{t}$  follow a gradient flow on the non-convex loss  $F$ , it is shown in [5] that the iterates  $\beta_{t}^{\alpha}$  follow a mirror flow on the convex loss  $L$  with potential  $\phi_{\alpha}$  and initialisation  $\beta_{t=0}^{\alpha} = \mathbf{0}$ :
$$
\mathrm {d} \nabla \phi_ {\alpha} \left(\beta_ {t} ^ {\alpha}\right) = - \nabla L \left(\beta_ {t} ^ {\alpha}\right) \mathrm {d} t, \tag {6}
$$
where  $\phi_{\alpha}$  is the hyperbolic entropy function [21] defined as:
$$
\phi_ {\alpha} (\beta) = \frac {1}{2} \sum_ {i = 1} ^ {d} \left(\beta_ {i} \operatorname {a r c s i n h} \left(\frac {\beta_ {i}}{\alpha^ {2}}\right) - \sqrt {\beta_ {i} ^ {2} + \alpha^ {4}} + \alpha^ {2}\right). \tag {7}
$$
Unveiling the mirror flow structure enables to leverage convex optimisation tools to prove convergence of the iterates to a global minimiser  $\beta_{\alpha}^{\star}$  as well as a simple proof of the implicit regularisation problem it solves. As shown by Woodworth et al. [50], in the overparametrised setting where  $d > n$  and where there exists an infinite number of global minima, the limit  $\beta_{\alpha}^{\star}$  is the solution of the problem:
$$
\beta_ {\alpha} ^ {\star} = \underset {y _ {i} = \langle x _ {i}, \beta \rangle , \forall i} {\arg \min } \phi_ {\alpha} (\beta). \tag {8}
$$
Furthermore, a simple function analysis shows that  $\phi_{\alpha}$  behaves as a rescaled  $\ell_1$ -norm as  $\alpha$  goes to 0, meaning that the recovered solution  $\beta_{\alpha}^{\star}$  converges to the minimum  $\ell_1$ -norm solution  $\beta_{\ell_1}^{\star} = \arg \min_{y_i = \langle x_i,\beta \rangle}\| \beta \| _1$  as  $\alpha$  goes to 0 (see [49] for a precise rate). To bring to light the saddle-to-saddle dynamics which occurs as we take the initialisation to 0, we make substantial use of the nice mirror structure from Eq. (6).
Appropriate time rescaling. To understand the limiting dynamics of  $\beta_t^\alpha$ , it is natural to consider the limit  $\alpha \to 0$  in Eq. (6). However, the potential  $\phi_\alpha$  is such that  $\phi_\alpha(\beta) \sim \ln(1/\alpha) \| \beta \|_1$  for small  $\alpha$  and therefore degenerates as  $\alpha \to 0$ . Similarly, for  $\beta \neq 0$ ,  $\| \nabla \phi_\alpha(\beta) \| \to \infty$  as  $\alpha \to 0$ . The formulation from Eq. (6) is thus not appropriate to take the limit  $\alpha \to 0$ . We can nonetheless obtain a meaningful limit by considering the opportune time acceleration  $\tilde{t}_\alpha(t) = \ln(1/\alpha) \cdot t$  and looking at the accelerated iterates
$$
\tilde {\beta} _ {t} ^ {\alpha} := \beta_ {\vec {t} _ {\alpha} (t)} ^ {\alpha} = \beta_ {\ln (1 / \alpha) t} ^ {\alpha}. \tag {9}
$$
Indeed, a simple chain rule leads to the "accelerated mirror flow":  $\mathrm{d}\nabla \phi_{\alpha}(\tilde{\beta}_{t}^{\alpha}) = -\ln \left(\frac{1}{\alpha}\right)\nabla L(\tilde{\beta}_{t}^{\alpha})\mathrm{d}t$ . The accelerated iterates  $(\tilde{\beta}_{t}^{\alpha})_{t}$  follow a mirror descent with a rescaled potential:
$$
\mathrm {d} \nabla \tilde {\phi} _ {\alpha} \left(\tilde {\beta} _ {t} ^ {\alpha}\right) = - \nabla L \left(\tilde {\beta} _ {t} ^ {\alpha}\right) \mathrm {d} t, \quad \text {w h e r e} \quad \tilde {\phi} _ {\alpha} := \frac {1}{\ln (1 / \alpha)} \cdot \phi_ {\alpha}, \tag {10}
$$
with  $\tilde{\beta}_{t=0} = 0$  and where  $\phi_{\alpha}$  is defined Eq. (7). Our choice of time acceleration ensures that the rescaled potential  $\tilde{\phi}_{\alpha}$  is non-degenerate as the initialisation goes to 0 since  $\tilde{\phi}_{\alpha}(\beta) \underset{\alpha \to 0}{\sim} \| \beta \|_1$ .
3 Intuitive construction of the limiting flow and saddle-to-saddle algorithm
In this section, we aim to give a comprehensible construction of the limiting flow. We therefore choose to provide intuition over pure rigor, and defer the full and rigorous proof to the Appendix E. The technical crux of our analysis is to demonstrate the existence of a piecewise constant limiting process towards which the iterates  $\tilde{\beta}^{\alpha}$  converge to. The convergence result is deferred to the following Section 4. In this section we assume this convergence and refer to this piecewise constant limiting process as  $(\tilde{\beta}_t^\circ)_t$ . Our goal is then to determine the jump times  $(t_1,\dots ,t_p)$  as well as the saddles  $(\beta_0,\ldots ,\beta_p)$  which fully define this process.
To do so, it is natural to examine the limiting equation obtained when taking the limit  $\alpha \to 0$  in Eq. (10). We first turn to its integral form which writes:
$$
- \int_ {0} ^ {t} \nabla L \left(\tilde {\beta} _ {s} ^ {\alpha}\right) \mathrm {d} s = \nabla \tilde {\phi} _ {\alpha} \left(\tilde {\beta} _ {t} ^ {\alpha}\right). \tag {11}
$$
Provided the convergence of the flow  $\tilde{\beta}^{\alpha}$  towards  $\tilde{\beta}^{\circ}$ , the left hand side of the previous equation converges to  $-\int_0^t\nabla L(\tilde{\beta}_s^\circ)\mathrm{d}s$ . For the right hand side, recall that  $\tilde{\phi}_{\alpha}(\beta)\stackrel {\alpha \to 0}{\sim}\| \beta \| _1$ , it is therefore natural to expect the right hand side of Eq. (11) to converge towards an element of  $\partial \| \tilde{\beta}_t^\circ \| _1$ , where we recall the definition of the subderivative of the  $\ell_{1}$ -norm as:
$$
\partial \| \tilde {\beta} \| _ {1} = \{1 \} \text {i f} \tilde {\beta} > 0, \{- 1 \} \text {i f} \tilde {\beta} <   0, [ - 1, 1 ] \text {i f} \tilde {\beta} = 0.
$$
The arising key equation which must satisfy the limiting process  $\tilde{\beta}^{\circ}$  is then, for all  $t\geq 0$
$$
- \int_ {0} ^ {t} \nabla L \left(\tilde {\beta} _ {s} ^ {\circ}\right) \mathrm {d} s \in \partial \| \tilde {\beta} _ {t} ^ {\circ} \| _ {1}. \tag {12}
$$
We show that this equation uniquely determines the piecewise constant process  $\tilde{\beta}^{\circ}$  by imposing the number of jumps  $p$ , the jump times as well as the saddles which are visited between the jumps. Indeed the relation described in Eq. (12) provides 4 restrictive properties that enable to construct  $\tilde{\beta}^{\circ}$ . To state them, let  $s_t = -\int_0^t\nabla L(\tilde{\beta}_s^\circ)\mathrm{d}s$  and notice that it is continuous and piecewise linear since  $\tilde{\beta}^{\circ}$  is piecewise constant. For each coordinate  $i\in [d]$ , it holds that:
(K1)  $s_t[i] \in [-1, 1]$  (K2)  $s_t[i] = 1 \Rightarrow \tilde{\beta}_t^\circ[i] \geq 0$  and  $s_t[i] = -1 \Rightarrow \tilde{\beta}_t^\circ[i] \leq 0$
(K3)  $s_t[i] \in (-1, 1) \Rightarrow \tilde{\beta}_t^\circ[i] = 0$  (K4)  $\tilde{\beta}_t^\circ[i] > 0 \Rightarrow s_t[i] = 1$  and  $\tilde{\beta}_t^\circ[i] < 0 \Rightarrow s_t[i] = -1$
To understand how these conditions lead to the algorithm which determines the jump times and the visited saddles, we present a 2-dimensional example for which we can walk through each step. The general case then naturally follows from this simple example.
3.1 Construction of the saddle-to-saddle algorithm with an illustrative  $2d$  example.
Let us consider  $n = d = 2$  and data matrix  $X \in \mathbb{R}^{2 \times 2}$  such that  $X^{\top}X = ((1,0.2),(0.2, -0.2))$ . We consider  $\beta^{\star} = (-0.2,2) \in \mathbb{R}^{2}$  and outputs  $y = X\beta^{\star}$ . This setting is such that the loss  $L$  has  $\beta^{\star}$  as its unique minimum and  $L(\beta^{*}) = 0$ . Furthermore the non-convex loss  $F$  has 3 saddles which map to:  $\beta_{c,0} := (0,0) = \arg \min_{\beta_i = 0,\forall i} L(\beta), \beta_{c,1} := (0.2,0) = \arg \min_{\beta[2] = 0} L(\beta)$  and  $\beta_{c,2} := (0,1.6) = \arg \min_{\beta[1] = 0} L(\beta)$ . The loss function  $L$  is sketched in Figure 2 (Left). Notice that by the definition of  $\beta_{c,1}$  and  $\beta_{c,2}$ , the gradients of the loss at these points are orthogonal to the axis they belong to. When running gradient flow with a small initialisation over our diagonal linear network, we obtain the plots illustrated in Figure 2 (Middle and Right). We observe three jumps: the iterates jump from the saddle at the origin to  $\beta_{c,1}$  at time  $t_1$ , then to  $\beta_{c,2}$  at time  $t_2$  and finally to the global minimum  $\beta^{\star}$  at time  $t_3$ .
Figure 2: Left: Sketch of the  $2d$  loss. Middle and right: Outputs of gradient flow with small initialisation scale: the iterates are piecewise constant and  $s_t$  is piecewise linear across time. We refer to the main text for further details.
Let us show how Eq. (12) enables us to theoretically recover this trajectory. A simple observation which we will use several times below is that for any  $t' > t$  such that  $\tilde{\beta}^{\circ}$  is constant equal to  $\beta$  over the time interval  $(t, t')$ , the definition of  $s$  enables to write that  $s_{t'} = s_t - (t' - t) \cdot \nabla L(\beta)$ .
Zeroth saddle: The iterates are at the saddle at the origin:  $\tilde{\beta}_t^\circ = \beta_0 \coloneqq \beta_{c,0}$  and therefore  $s_t = -t \cdot \nabla L(\beta_0)$ . Our key equation Eq. (12) is verified since  $s_t = -t \cdot \nabla L(\beta_0) \in \partial \| \beta_0 \|_1 = [-1, 1]^d$ . However the iterates cannot stay at the origin after time  $t_1 \coloneqq 1 / \| \nabla L(\beta_0) \|_\infty$  which corresponds to the time at which the first coordinate of  $s_t$  hits  $+1$ :  $s_{t_1}[1] = 1$ . If the iterates stayed at the origin after  $t_1$ , (K1) for  $i = 1$  would be violated. The iterates must hence jump.
First saddle: The iterates can only jump to a point different from the origin which maintains Eq. (12) valid. We denote this point as  $\beta_{1}$ . Notice that:
-  $s_{t_1}[2] = -t_1 \cdot \nabla L(\beta_0)[2] \in (-1, 1)$  and since  $s_t$  is continuous, we must have  $\beta_1[2] = 0$  (K3)
-  $s_{t_1}[1] = 1$  and hence for  $t \geq t_1$ ,  $s_t[1] = 1 - (t - t_1)\nabla L(\beta_1)[1]$ . We cannot have  $\nabla L(\beta_1)[1] < 0$  (K1), and neither  $\nabla L(\beta_1)[1] > 0$  since otherwise  $s_t[1] \in (-1, 1)$  and  $\beta_1 = 0$  (K3)
The two conditions  $\beta_{1}[2] = 0$  and  $\nabla L(\beta_{1})[1] = 0$  uniquely defines  $\beta_{1}$  as equal to  $\beta_{c,1}$ . We now want to know if and when the iterates jump again. We saw that  $s_t[1]$  remains at the value  $+1$ . However since  $\beta_{1}$  is not a global minimum,  $\nabla L(\beta_{1})[2] \neq 0$  and  $s_t[2]$  hits  $+1$  at time  $t_2$  defined such that  $-(t_1\nabla L(\beta_0) + (t_2 - t_1)\nabla L(\beta_1))[2] = 1$ . The iterates must jump otherwise (K1) would break.
The iterates cannot jump to  $\beta^{\star}$  yet! As the second coordinate of the iterates can activate, one could expect the iterates to be able to jump to the global minimum. However note that  $s_t$  is a continuous function and that  $s_{t_2}$  is equal to the vector (1, 1). If the iterates jumped to the global minimum, then the first coordinate of the iterates would change sign from  $+0.2$  to  $-0.2$ . Due to (K4) this would lead  $s_t$  jumping from  $+1$  to  $-1$ , violating its continuity.
Second saddle: We denote as  $\beta_{2}$  the point to which the iterates jump.  $s_{t_2}$  is now equal to the vector  $(1,1)$  and therefore  $(i)\beta_{2}\geq 0$  (coordinate-wise) from (K2 and K3) and the continuity of  $s$ . Since  $s_t = s_{t_2} - (t - t_2)\nabla L(\beta_2)$ , we must also have:  $(ii)\nabla L(\beta_2)\geq 0$  from (K1) (iii) for  $i\in \{1,2\}$ , if  $\beta_{2}[i]\neq 0$  then  $\nabla L(\beta_2)[i] = 0$  from (K4). The three conditions  $(i)$ ,  $(ii)$  and  $(iii)$  precisely correspond to the optimality conditions of the following problem:
$$
\operatorname *{arg  min}_{\beta [1]\geq 0,\beta [2]\geq 0}L(\beta).
$$
The unique minimiser of this problem is  $\beta_{c,2}$ , hence  $\beta_{2} = \beta_{c,2}$ , which means that the first coordinate deactivates. Similar to before, (K1) is valid until the time  $t_3$  at which the first coordinate of  $s_t = s_{t_2} - (t - t_2)\nabla L(\beta_2)$  reaches  $-1$  due to the fact that  $\nabla L(\beta_2)[1] > 0$ .