Title: Saddle-to-Saddle Dynamics in Diagonal Linear Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: gradient flow

saddle-to-saddle

diagonal linear network

incremental learning
Abstract: In this paper we fully describe the trajectory of gradient flow over $2$-layer diagonal linear networks for the regression setting in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minimum $\ell_1$-norm solution. We explicitly characterise the visited saddles as well as the jump times through a recursive algorithm reminiscent of the LARS algorithm used for computing the Lasso path.  Starting from the zero vector, coordinates are successively activated until the minimum $\ell_1$-norm solution is recovered, revealing an incremental learning. Our proof leverages a convenient arc-length time-reparametrisation which enables to keep track of the transitions between the jumps. Our analysis requires negligible assumptions on the data, applies to both under and overparametrised settings and covers complex cases where there is no monotonicity of the number of active coordinates. We provide numerical experiments to support our findings.

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Abstract

In this paper we fully describe the trajectory of gradient flow over 2 -layer diagonal linear networks for the regression setting in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minimum ℓ 1 -norm solution. We explicitly characterise the visited saddles as well as the jump times through a recursive algorithm reminiscent of the LARS algorithm used for computing the Lasso path. Starting from the zero vector, coordinates are successively activated until the minimum ℓ 1 -norm solution is recovered, revealing an incremental learning. Our proof leverages a convenient arc-length time-reparametrisation which enables to keep track of the transitions between the jumps. Our analysis requires negligible assumptions on the data, applies to both under and overparametrised settings and covers complex cases where there is no monotonicity of the number of active coordinates. We provide numerical experiments to support our findings.

1 Introduction

Strikingly simple algorithms such as gradient descent are driving forces for deep learning and have led to remarkable empirical results. Nonetheless, understanding the performances of such methods remains a challenging and exciting mystery: (i) their global convergence on highly non-convex losses is far from being trivial and (ii) the fact that they lead to solutions which generalise well [53] is still not fully understood.

To explain this second point, a major line of work has focused on the concept of implicit regularisation: amongst the infinite space of zero-loss solutions, the optimisation process must be implicitly biased towards solutions which have good generalisation properties for the considered real-world prediction tasks. Many papers have therefore shown that gradient methods have the fortunate property of asymptotically leading to solutions which have a well-behaving structure [38, 24, 16].

Aside from these results which mostly focus on characterising the asymptotic solution, a slightly different point of view has been to try to describe the full trajectory. Indeed it has been experimentally observed that gradient methods with small initialisations have the property of learning models of increasing complexity across the training of neural networks [29]. This behaviour is usually referred to as incremental learning or as a saddle-to-saddle process and describes learning curves which are piecewise constant: the training process makes very little progress for some time, followed by a sharp transition where a new 'feature' is suddenly learned. In terms of optimisation trajectory, this corresponds to the iterates "jumping" from a saddle of the training loss to another.

Several settings exhibiting such dynamics for small initialisation have been considered: matrix and tensor factorisation [44, 27], simplified versions of diagonal linear networks [23, 7], linear networks [22, 45, 26], 2 -layer neural networks with orthogonal inputs [10], learning leap functions with 2 -layer neural networks [1] and matrix sensing [2, 33, 28]. However, all these results require

25

120

100

α

ln(1

/

α

)

t

t

Accelerated iterates

=

β

β

0

5

10

15

20

Accelerated iterates

˜

β

α

t

=

β

α

ln(1

/

α

)

t

t

1

t

2

t

3

Time

t

Non-activated coordinates

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

20

40

60

80

Accelerated iterates

˜

β

α

t

=

β

α

ln(1

/

α

)

t

t

1

t

2

t

3

Time

t

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Time

t

0

5

10

15

20

First saddle

Second saddle

Third saddle

Global minimum

Iterates in        space

β

0

β

1

β

2

Train losses

L

(

β

α

t

)

Iterates

β

α

t

Saddles:

β

/star

De-activating

coordinate

R

3

β

α

t

[1]

β

α

t

[2]

β

α

t

[3]

0

.

3

Time

t

Time

0

.

0

0

.

1

0

.

2

0

.

4

0

.

5

0

.

6

t

restrictive assumptions on the data or only characterise the first jump. Obtaining a complete picture of the saddle-to-saddle process by describing all the visited saddles and jump times is mathematically challenging and still missing. We intend to fill this gap by considering diagonal linear networks which are simplified neural networks that have received significant attention lately [50, 48, 25, 43, 20] as they are ideal proxy models for gaining a deeper understanding of complex phenomenons such as saddle-to-saddle dynamics.

1.1 Informal statement of the main result

In this paper, we provide a full description of the trajectory of gradient flow over 2 -layer diagonal linear networks in the limit of vanishing initialisation. The main result is informally presented here.

Theorem 1 (Main result, informal) . In the regression setting and in the limit of vanishing initialisation, the trajectory of gradient flow over a 2 -layer diagonal linear network converges towards a limiting process which is piecewise constant: the iterates successively jump from a saddle of the training loss to another, each visited saddle and jump time can recursively be computed through an algorithm (Algorithm 1) reminiscent of the LARS algorithm for the Lasso.

The incremental learning stems from the particular structure of the saddles as they correspond to minimisers of the training loss with a constraint on the set of non-zero coordinates. The saddles therefore correspond to sparse vectors which partially fit the dataset. For simple datasets, a consequence of our main result is that the limiting trajectory successively starts from the zero vector and successively learns the support of the sparse ground truth vector until reaching it . However, we make minimal assumptions on the data and our analysis also holds for complex datasets . In that case, the successive active sets are not necessarily increasing in size and coordinates can deactivate as well as activate until reaching the minimum ℓ 1 -norm solution (see Figure 1 (middle) for an example of a deactivating coordinate). The regression setting and the diagonal network architecture are introduced in Section 2. Section 3 provides an intuitive construction of the limiting saddle-to-saddle dynamics and presents the algorithm that characterises it. Our main result regarding the convergence of the iterates towards this process is presented in Section 4 and further discussion is provided in Section 5.

2 Problem setup and leveraging the mirror structure

2.1 Setup

Linear regression. We study a linear regression problem with inputs ( x 1 , . . . , x n ) ∈ ( R d ) n and outputs ( y 1 , . . . , y n ) ∈ R n . We consider the typical quadratic loss:

<latexi sh

1_b

64="m

0zC8n

v

K

A

+

roGqLU

>

FX

c

VD

SgNBEJ

P

Z

I

9

Q

M

d

O

jw

5

T

p

Y

3

/

u

y

R

k

W

7

2

f

H

<latexi sh

1_b

64="UgLHzPw

d

FI3n cf

v

R

9A

>

B

V

NS8

EJ

r

q/

0

G

K

D

T

m7

Q

y5

2+

u

M

W

o

Z

k

j

C

X

Y

p

O

<latexi sh

1_b

64="j9

r

Z/dc

q

Wn

P

Y

Fz

Ov

>A

B

V

NS8

EJ3

X

Kk

0

G

2

M

y7

LuRC

+o

w

g

D

I

U

p

H

m

T

5

Q

f

<latexi sh

1_b

64="LA2Nw+dC

y

KGZmjBVIEz

>

n

c

S8

9

0

PR

g

D

T

7

QvH

5

u

M

f

F

W

o

k

O

p

Y

X

˜

α

<latexi sh

1_b

64="T

oCwBI

ud

3O

AvVp7KQ

XM

>

+

c

DLS

N

FJ

U

2

W

gq

r

m

Z

9

Yf

G

Hj

/

8

n

z

5

R

P

y

0

E

k

<latexi sh

1_b

64="

n+85XZrA

T

3ogH

y

k

c

>

CD

V

JS

NBEO

2

Rj

IpzK

9GL

Y

M

0

Q

m

7

q

f/

v

d

u

w

W

P

U

F

<latexi sh

1_b

64="UgLHzPw

d

FI3n cf

v

R

9A

>

B

V

NS8

EJ

r

q/

0

G

K

D

T

m7

Q

y5

2+

u

M

W

o

Z

k

j

C

X

Y

p

O

<latexi sh

1_b

64="j9

r

Z/dc

q

Wn

P

Y

Fz

Ov

>A

B

V

NS8

EJ3

X

Kk

0

G

2

M

y7

LuRC

+o

w

g

D

I

U

p

H

m

T

5

Q

f

<latexi sh

1_b

64="p

Sgm+R

UX

v

8

jH7

ud

q

>A

B

n

c

VDL

N

EJy

/

Y

o

G

I

W

P

Z

K

z

C9FO

Q

M

w

f

3

2T

k

0

r

5

<latexi sh

1_b

64="LA2Nw+dC

y

KGZmjBVIEz

>

n

c

S8

9

0

PR

g

D

T

7

QvH

5

u

M

f

F

W

o

k

O

p

Y

X

<latexi sh

1_b

64="T

oCwBI

ud

3O

AvVp7KQ

XM

>

+

c

DLS

N

FJ

U

2

W

gq

r

m

Z

9

Yf

G

Hj

/

8

n

z

5

R

P

y

0

E

k

<latexi sh

1_b

64="Xo53Ew ypKN

O8

cC

mu/jB

>A

VDJSg

2

R

I0

FT

vH

M

k

Qq

r

n

G

L

U

P

7

z

Z

fW

Y

+

d

9

<latexi sh

1_b

64="/+8

5PD

Fu

Igw02ZMUqzG

c

>A

B

n

dV

LS

N

EJy

r

f

9

Y

K3o

X

p

k

H

7

v

Q

R

O

W

C

T

j

m

<latexi sh

1_b

64="/9N

TG5FyIPCS

+

2RnmE

DLfc

>A

B7

V

8

q

vg

Q

WM

w

K

p

Y

d

X

j

k

u

O

Z

z

0

<latexi sh

1_b

64="LAQM0v

V

pKIBr

G

q5

>

7n

c

NS8

E

3U

/

9

m

F

Z

J

R

PCj

d

zX

Y

H

+

O

T

u

f

g

w

D

<latexi sh

1_b

64="Y

SCG

2k

HvXcNVB

PdW

0o

>A

83

DL

F

pr

f

Z

u

gq

R

j

9

J

m

O

+

/

n

M

z

T

K

7

UI

E

Q

w

5

y

<latexi sh

1_b

64="I7AX9

RY

Q8

m3V

U/

>

B

n

c

N

EJ

r

q

v

WMF

w

K

p

2

Td

f

D

+

P

j

k

Cyu

O

Z

G

5

z

L

<latexi sh

1_b

64="0

R

CMg

U

7wj3BK

Wuo+YPZLEI

>A

8n cdVDJS

N

O2

X

G

y

p

vH

Q

9

k

F

r

q

f

5

/

z

T

m

We make no assumption on the number of samples n nor the dimension d . The only assumption we make on the data throughout the paper is that the inputs ( x 1 , . . . , x n ) are in general position . In order to state this assumption, let X ∈ R n × d be the feature matrix whose i th row is x i and let ˜ x j ∈ R n be its j th column for j ∈ [ d ] .

̸

Assumption 1 (General position) . For any k ≤ min( n, d ) and arbitrary signs σ 1 , . . . , σ k ∈ {-1 , 1 } , the affine span of any k points σ 1 ˜ x j 1 , . . . , σ k ˜ x j k does not contain any element of the set {± ˜ x j , j = j 1 , . . . , j k } .

This assumption is slightly technical but is standard in the Lasso literature [47]. Note that it is not restrictive as it is almost surely satisfied when the data is drawn from a continuous probability distribution [47, Lemma 4]. Letting S = arg min β L ( β ) denote the affine space of solutions, Assumption 1 ensures that the minimisation problem min β ⋆ ∈S ∥ β ⋆ ∥ 1 has a unique minimiser which we denote β ⋆ ℓ 1 and which corresponds to the minimum ℓ 1 -norm solution.

2-layer diagonal linear network. In an effort to understand the training dynamics of neural networks, we consider a 2 -layer diagonal linear network which corresponds to writing the regression vector β as

This parametrisation can be interpreted as a simple neural network x ↦→⟨ u, σ (diag( v ) x ) ⟩ where u are the output weights, the diagonal matrix diag( v ) represents the inner weights, and the activation σ is the identity function. We refer to w = ( u, v ) ∈ R 2 d as the weights and to β := u ⊙ v ∈ R d as the prediction parameter . With the parametrisation (2), the loss function F over the parameters w = ( u, v ) ∈ R 2 d is defined as:

Though this reparametrisation is simple, the associated optimisation problem is non-convex and highly non-trivial training dynamics already occur. The critical points of the function F exhibit a very particular structure, as highlighted in the following proposition proven in Appendix B.

Proposition 1. All the critical points w c of F which are not global minima, i.e., ∇ F ( w c ) = 0 and F ( w c ) > min w F ( w ) , are necessarily saddle points ( i.e., not local extrema). They map to parameters β c = u c ⊙ v c which satisfy | β c |⊙∇ L ( β c ) = 0 and:

where supp( β c ) = { i ∈ [ d ] , β c [ i ] = 0 } corresponds to the support of β c .

̸

The optimisation problem in Eq. (4) states that the saddle points of the train loss F correspond to sparse vectors that minimise the loss function L over its non-zero coordinates . This property already shows that the saddle points possess interesting properties from a learning perspective. In the following we loosely use the term of 'saddle' to refer to points β c ∈ R d solution of Eq. (4) that are not saddles of the convex loss function L . We adopt this terminology because they correspond to points w c ∈ R 2 d that are indeed saddles of the non-convex loss F .

initialised at u 0 = √ 2 α 1 ∈ R d > 0 with α > 0 , and v 0 = 0 ∈ R d . This initialisation results in β 0 = 0 ∈ R d independently of the chosen weight initialisation scale α . We denote β α t := u α t ⊙ v α t the prediction iterates generated from the gradient flow to highlight its dependency on the initialisation scale α 1 . The origin 0 ∈ R 2 d is a critical point of the function F and taking the initialisation α → 0 therefore arbitrarily slows down the dynamics. In fact, it can be easily shown for any fixed time t , that ( u α t , v α t ) → 0 as α → 0 , indicating that the iterates are stuck at the origin. Therefore if we restrict ourselves to a finite time analysis, there is no hope of exhibiting the observed saddle-to-saddle behaviour. To do so, we must find an appropriate bijection ˜ t α in R ≥ 0 which 'accelerates' time ( i.e. ˜ t α ( t ) -→ α → 0 + ∞ for all t ) and consider the accelerated iterates β α ˜ t α ( t ) which can escape the saddles. Finding this bijection becomes very natural once the mirror structure is unveiled.

2.2 Leveraging the mirror flow structure

While the iterates ( w α t ) t follow a gradient flow on the non-convex loss F , it is shown in [5] that the iterates β α t follow a mirror flow on the convex loss L with potential ϕ α and initialisation β α t =0 = 0 :

where ϕ α is the hyperbolic entropy function [21] defined as:

Unveiling the mirror flow structure enables to leverage convex optimisation tools to prove convergence of the iterates to a global minimiser β ⋆ α as well as a simple proof of the implicit regularisation problem it solves. As shown by Woodworth et al. [50], in the overparametrised setting where d > n and where there exists an infinite number of global minima, the limit β ⋆ α is the solution of the problem:

Furthermore, a simple function analysis shows that ϕ α behaves as a rescaled ℓ 1 -norm as α goes to 0 , meaning that the recovered solution β ⋆ α converges to the minimum ℓ 1 -norm solution β ⋆ ℓ 1 = arg min y i = ⟨ x i ,β ⟩ ∥ β ∥ 1 as α goes to 0 (see [49] for a precise rate). To bring to light the saddle-tosaddle dynamics which occurs as we take the initialisation to 0 , we make substantial use of the nice mirror structure from Eq. (6).

̸

Appropriate time rescaling. To understand the limiting dynamics of β α t , it is natural to consider the limit α → 0 in Eq. (6). However, the potential ϕ α is such that ϕ α ( β ) ∼ ln(1 /α ) ∥ β ∥ 1 for small α and therefore degenerates as α → 0 . Similarly, for β = 0 , ∥∇ ϕ α ( β ) ∥→ ∞ as α → 0 . The formulation from Eq. (6) is thus not appropriate to take the limit α → 0 . We can nonetheless obtain a meaningful limit by considering the opportune time acceleration ˜ t α ( t ) = ln(1 /α ) · t and looking at the accelerated iterates

Indeed, a simple chain rule leads to the 'accelerated mirror flow': d ∇ ϕ α ( ˜ β α t ) = -ln ( 1 α ) ∇ L ( ˜ β α t )d t . The accelerated iterates ( ˜ β α t ) t follow a mirror descent with a rescaled potential:

with ˜ β t =0 = 0 and where ϕ α is defined Eq. (7). Our choice of time acceleration ensures that the rescaled potential ˜ ϕ α is non-degenerate as the initialisation goes to 0 since ˜ ϕ α ( β ) ∼ α → 0 ∥ β ∥ 1 .

3 Intuitive construction of the limiting flow and saddle-to-saddle algorithm

In this section, we aim to give a comprehensible construction of the limiting flow. We therefore choose to provide intuition over pure rigor, and defer the full and rigorous proof to the Appendix E. The technical crux of our analysis is to demonstrate the existence of a piecewise constant limiting process towards which the iterates ˜ β α converge to. The convergence result is deferred to the following Section 4. In this section we assume this convergence and refer to this piecewise constant limiting process as ( ˜ β ◦ t ) t . Our goal is then to determine the jump times ( t 1 , . . . , t p ) as well as the saddles ( β 0 , . . . , β p ) which fully define this process.

To do so, it is natural to examine the limiting equation obtained when taking the limit α → 0 in Eq. (10). We first turn to its integral form which writes:

0

0

0

.

.

.

02

02

00

0

.

00

0

0

.

.

02

00

0

0

0

Provided the convergence of the flow ˜ β α towards ˜ β ◦ , the left hand side of the previous equation converges to -∫ t 0 ∇ L ( ˜ β ◦ s )d s . For the right hand side, recall that ˜ ϕ α ( β ) α → 0 ∼ ∥ β ∥ 1 , it is therefore natural to expect the right hand side of Eq. (11) to converge towards an element of ∂ ∥ ˜ β ◦ t ∥ 1 , where we recall the definition of the subderivative of the ℓ 1 -norm as:

The arising key equation which must satisfy the limiting process ˜ β ◦ is then, for all t ≥ 0 :

We show that this equation uniquely determines the piecewise constant process ˜ β ◦ by imposing the number of jumps p , the jump times as well as the saddles which are visited between the jumps. Indeed the relation described in Eq. (12) provides 4 restrictive properties that enable to construct ˜ β ◦ . To state them, let s t = -∫ t 0 ∇ L ( ˜ β ◦ s )d s and notice that it is continuous and piecewise linear since ˜ β ◦ is piecewise constant. For each coordinate i ∈ [ d ] , it holds that:

To understand how these conditions lead to the algorithm which determines the jump times and the visited saddles, we present a 2 -dimensional example for which we can walk through each step. The general case then naturally follows from this simple example.

3.1 Construction of the saddle-to-saddle algorithm with an illustrative 2 d example.

Let us consider n = d = 2 and data matrix X ∈ R 2 × 2 such that X ⊤ X = ((1 , 0 . 2) , (0 . 2 , -0 . 2)) . We consider β ⋆ = ( -0 . 2 , 2) ∈ R 2 and outputs y = Xβ ⋆ . This setting is such that the loss L has β ⋆ as its unique minimum and L ( β ∗ ) = 0 . Furthermore the non-convex loss F has 3 saddles which map to: β c, 0 := (0 , 0) = arg min β i =0 , ∀ i L ( β ) , β c, 1 := (0 . 2 , 0) = arg min β [2]=0 L ( β ) and β c, 2 := (0 , 1 . 6) = arg min β [1]=0 L ( β ) . The loss function L is sketched in Figure 2 ( Left ). Notice that by the definition of β c, 1 and β c, 2 , the gradients of the loss at these points are orthogonal to the axis they belong to. When running gradient flow with a small initialisation over our diagonal linear network, we obtain the plots illustrated Figure 2 ( Middle and Right ). We observe three jumps: the iterates jump from the saddle at the origin to β c, 1 at time t 1 , then to β c, 2 at time t 2 and finally to the global minimum β ⋆ at time t 3 .

α

( ˜

˜

α

∫

L

( ˜

β

α

t

)

0

1

2

Accelerated iterates

˜

β

α

t

˜

β

α

(1)

˜

β

α

(2)

-

1

1

s

α

t

:=

-

∫

t

0

∇

L

( ˜

β

α

t

)d

s

s

α

(1)

s

α

(2)

Time

t

Time

t

Accelerated iterates

˜

β

◦

t

Train losses

L

( ˜

β

α

t

)

0

1

2

Accelerated iterates

˜

β

α

t

˜

β

α

(1)

˜

β

α

(2)

1

1

s

α

t

:=

-

∫

t

0

∇

L

( ˜

β

α

t

)d

s

s

α

(1)

s

α

(2)

s

t

:=

-

∫

t

0

∇

L

( ˜

β

◦

s

)d

s

t

1

t

2

t

3

t

1

t

2

t

3

+1

1

β

0

β

1

β

2

β

/star

0

.

02

Train losses

L

( ˜

β

α

t

)

1

2

Accelerated iterates

˜

β

◦

t

˜

β

◦

(1)

˜

β

◦

(2)

1

s

α

t

:=

-

∫

t

0

Train losses

L

β

t

)

0

0

1

2

Accelerated iterates

β

◦

t

˜

β

◦

(1)

˜

β

◦

(2)

0

-

1

1

s

t

:=

-

0

∇

L

β

t

)d

s

s

(1)

s

(2)

β

c,

0

β

c,

1

β

c,

2

β

/star

-∇

L

(

β

c,

1

)

-∇

L

(

β

c,

1

)

Sketch of

the loss

[1]

[2]

[1]

[2]

Time

-

t

Time

t

t

0

-

0

Time

t

0

0

0

Time t Time t Time Let us show how Eq. (12) enables us to theoretically recover this trajectory. A simple observation which we will use several times below is that for any t ′ > t such that ˜ β ◦ is constant equal to β over the time interval ( t, t ′ ) , the definition of s enables to write that s t ′ = s t -( t ′ -t ) · ∇ L ( β ) .

0

0

α

( ˜

Train losses

<latexi sh

1_b

64="Z/j2p gB

G

RLAMmo

0w+

C

>

H

cdV

9S

N

E

3z

8

vqIWF

Q

DT

Y

J

7

X

U

f

n

P

K

5

r

k

O

u

y

<latexi sh

1_b

64="0

R

CMg

U

7wj3BK

Wuo+YPZLEI

>A

8n cdVDJS

N

O2

X

G

y

p

vH

Q

9

k

F

r

q

f

5

/

z

T

m

<latexi sh

1_b

64="NBUTkHz

F

Y

+EOGX

Z

A

K

>

9

c

VDLSg

Jy

r

f

oP

Q

u

n

M

v

w

W

p

j

3

70

C

q

m

5

2

/

R

I

d

8

<latexi sh

1_b

64="XE

3jwTY0H

A

L

Q

7K

r

>

B9

c

V

NS8

W

O

J

k

U

F

2

o

+

z

u

g

d

CI

M

/

Z

D

5

G

y

m

v

n

p

f

q

P

R

<latexi sh

1_b

64="Z/j2p gB

G

RLAMmo

0w+

C

>

H

cdV

9S

N

E

3z

8

vqIWF

Q

DT

Y

J

7

X

U

f

n

P

K

5

r

k

O

u

y

<latexi sh

1_b

64="ES2ARGVLK

3

BdmZ

Nou

w

>

9H

D

g

Oy

r

fUY

P

Q

n

M

v

W

p

F

+

k

z

70

C

q

8X

5

J

/

I

<latexi sh

1_b

64="/9N

TG5FyIPCS

+

2RnmE

DLfc

>A

B7

V

8

q

vg

Q

WM

w

K

p

Y

d

X

j

k

u

O

Z

z

0

<latexi sh

1_b

64="o

H

+8B

U

W/EC

DY

y

>A

I

c

Z

SgN

7

jF

X

pT

n

KM

0u

Q

G

q

w5

P

3

O

r

R

m

d

z

V

9

f

2

<latexi sh

1_b

64="UgLHzPw

d

FI3n cf

v

R

9A

>

B

V

NS8

EJ

r

q/

0

G

K

D

T

m7

Q

y5

2+

u

M

W

o

Z

k

j

C

X

Y

p

O

<latexi sh

1_b

64="I7AX9

RY

Q8

m3V

U/

>

B

n

c

N

EJ

r

q

v

WMF

w

K

p

2

Td

f

D

+

P

j

k

Cyu

O

Z

G

5

z

L

<latexi sh

1_b

64="j9

r

Z/dc

q

Wn

P

Y

Fz

Ov

>A

B

V

NS8

EJ3

X

Kk

0

G

2

M

y7

LuRC

+o

w

g

D

I

U

p

H

m

T

5

Q

f

<latexi sh

1_b

64="T

oCwBI

ud

3O

AvVp7KQ

XM

>

+

c

DLS

N

FJ

U

2

W

gq

r

m

Z

9

Yf

G

Hj

/

8

n

z

5

R

P

y

0

E

k

<latexi sh

1_b

64="85HX

w/RfPqM3B

Wnv yDm

k

>A

7

c

V

NS

EJ

O

K

9

j0

rGC

Y

2

d

+

o

z

Z

g

p

I

Q

F

T

U

<latexi sh

1_b

64="LA2Nw+dC

y

KGZmjBVIEz

>

n

c

S8

9

0

PR

g

D

T

7

QvH

5

u

M

f

F

W

o

k

O

p

Y

X

<latexi sh

1_b

64="EoXNz nvSm2Mf

PB

IG

rkg

>A

8

c

V

3U

q/

9L

0

R

K

O

WD

T

J

Cj

Y

Z

puwH

d

7

5

y

F

Q

+

<latexi sh

1_b

64="5cdU

C

X

2

Dyo

>A

B

n

V

NS8

E

3

r

q/

9L

PR

K

g

wm7

QvH

u

M

W

z

F

I

O

Y

Z

f

<latexi sh

1_b

64="5Byz

NRrEc7m

f

wJ9j

X

>A

n

V

S8

3

W

O

Kk

0

G

/Y

M

2

d

C

P

+o

U

gq

D

Q

FI

v

ZT

p

H

<latexi sh

1_b

64="

HWX

k2B

R

N

E

Pvnd/V

U

>A

CJ3

c

Sy

O

z

+

T

Q

Iro

YM

Z

8

f

9

D

q

L

5

j

F

m

0

G

p

<latexi sh

1_b

64="UgLHzPw

d

FI3n cf

v

R

9A

>

B

V

NS8

EJ

r

q/

0

G

K

D

T

m7

Q

y5

2+

u

M

W

o

Z

k

j

C

X

Y

p

O

<latexi sh

1_b

64="j9

r

Z/dc

q

Wn

P

Y

Fz

Ov

>A

B

V

NS8

EJ3

X

Kk

0

G

2

M

y7

LuRC

+o

w

g

D

I

U

p

H

m

T

5

Q

f

<latexi sh

1_b

64="5cdU

C

X

2

Dyo

>A

B

n

V

NS8

E

3

r

q/

9L

PR

K

g

wm7

QvH

u

M

W

z

F

I

O

Y

Z

f

<latexi sh

1_b

64="5Byz

NRrEc7m

f

wJ9j

X

>A

n

V

S8

3

W

O

Kk

0

G

/Y

M

2

d

C

P

+o

U

gq

D

Q

FI

v

ZT

p

H

<latexi sh

1_b

64="T

oCwBI

ud

3O

AvVp7KQ

XM

>

+

c

DLS

N

FJ

U

2

W

gq

r

m

Z

9

Yf

G

Hj

/

8

n

z

5

R

P

y

0

E

k

<latexi sh

1_b

64="LA2Nw+dC

y

KGZmjBVIEz

>

n

c

S8

9

0

PR

g

D

T

7

QvH

5

u

M

f

F

W

o

k

O

p

Y

X

<latexi sh

1_b

64="yucw n5

8

RT

GK

oDI

>A

B

X

V

NS

EJ3

W

O

Qk

U

P

q9g

0

7

/

M

H

j

r

+

d

f

Z

p

z

F

Y

<latexi sh

1_b

64="k pRQ

3u

GJ

q

/zX

mjo2

WS

>A

B

c

V

N

8

E

O

rH

M

+wF

K

v

YTd

C

D

9

5

P

g

I

7

f

Ly

U

0

Z

n

∇

L

t

β

α

( ˜

t

)d

s

Zeroth saddle: The iterates are at the saddle at the origin: ˜ β ◦ t = β 0 := β c, 0 and therefore s t = -t ·∇ L ( β 0 ) . Our key equation Eq. (12) is verified since s t = -t ·∇ L ( β 0 ) ∈ ∂ ∥ β 0 ∥ 1 = [ -1 , 1] d . However the iterates cannot stay at the origin after time t 1 := 1 / ∥∇ L ( β 0 ) ∥ ∞ which corresponds to the time at which the first coordinate of s t hits +1 : s t 1 [1] = 1 . If the iterates stayed at the origin after t 1 , (K1) for i = 1 would be violated. The iterates must hence jump.

First saddle: The iterates can only jump to a point different from the origin which maintains Eq. (12) valid. We denote this point as β 1 . Notice that:

s t 1 [2] = -t 1 · ∇ L ( β 0 )[2] ∈ ( -1 , 1) and since s t is continuous, we must have β 1 [2] = 0 (K3)

s t 1 [1] = 1 and hence for t ≥ t 1 , s t [1] = 1 -( t -t 1 ) ∇ L ( β 1 )[1] . We cannot have ∇ L ( β 1 )[1] < 0 (K1), and neither ∇ L ( β 1 )[1] > 0 since otherwise s t [1] ∈ ( -1 , 1) and β 1 = 0 (K3)

̸

The two conditions β 1 [2] = 0 and ∇ L ( β 1 )[1] = 0 uniquely defines β 1 as equal to β c, 1 . We now want to know if and when the iterates jump again. We saw that s t [1] remains at the value +1 . However since β 1 is not a global minimum, ∇ L ( β 1 )[2] = 0 and s t [2] hits +1 at time t 2 defined such that -( t 1 ∇ L ( β 0 ) + ( t 2 -t 1 ) ∇ L ( β 1 ))[2] = 1 . The iterates must jump otherwise (K1) would break.

The iterates cannot jump to β ⋆ yet! As the second coordinate of the iterates can activate, one could expect the iterates to be able to jump to the global minimum. However note that s t is a continuous function and that s t 2 is equal to the vector (1 , 1) . If the iterates jumped to the global minimum, then the first coordinate of the iterates would change sign from +0 . 2 to -0 . 2 . Due to (K4) this would lead s t jumping from +1 to -1 , violating its continuity.

̸

Second saddle: We denote as β 2 the point to which the iterates jump. s t 2 is now equal to the vector (1 , 1) and therefore (i) β 2 ≥ 0 (coordinate-wise) from (K2 and K3) and the continuity of s . Since s t = s t 2 -( t -t 2 ) ∇ L ( β 2 ) , we must also have: (ii) ∇ L ( β 2 ) ≥ 0 from (K1) (iii) for i ∈ { 1 , 2 } , if β 2 [ i ] = 0 then ∇ L ( β 2 )[ i ] = 0 from (K4). The three conditions (i) , (ii) and (iii) precisely correspond to the optimality conditions of the following problem:

The unique minimiser of this problem is β c, 2 , hence β 2 = β c, 2 , which means that the first coordinate deactivates. Similar to before, (K1) is valid until the time t 3 at which the first coordinate of s t = s t 2 -( t -t 2 ) ∇ L ( β 2 ) reaches -1 due to the fact that ∇ L ( β 2 )[1] > 0 .

̸

Global minimum: We follow the exact same reasoning as for the second saddle. We now have s t 3 equal to the vector ( -1 , 1) and the iterates must jump to a point β 3 such that (i) β 3 [1] ≤ 0 , β 3 [2] ≥ 0 (K2 and K3), (ii) ∇ L ( β 3 )[1] ≤ 0 , ∇ L ( β 3 )[2] ≥ 0 (K1), (iii) for i ∈ { 1 , 2 } , if β 3 [ i ] = 0 then ∇ L ( β 3 )[ i ] = 0 (K4). Again, these are the optimality conditions of the following problem:

β ⋆ is the unique minimiser of this problem and β 3 = β ⋆ . For t ≥ t 3 we have s t = s t 3 and Eq. (12) is satisfied for all following times: the iterates do not have to move anymore.

3.2 Presentation of the full saddle-to-saddle algorithm

We can now provide the full algorithm (Algorithm 1) which computes the jump times ( t 1 , . . . , t p ) and saddles ( β 0 = 0 , β 1 , . . . , β p ) as the values and vectors such that the associated piecewise constant process satisfies Eq. (12) for all t . This algorithm therefore defines our limiting process ˜ β ◦ .

Algorithm 1 in words. The algorithm is a concise representation of the steps we followed in the previous section to construct ˜ β ◦ . We explain each step in words below. Starting from k = 0 , assume we enter the loop number k at the saddle β k computed in the previous loop:

The set A k contains the set of coordinates "which are unstable": by having a non-zero derivative, the loss could be decreased by moving along each one of these coordinates and one of these coordinates will have to activate.

̸

̸

The time gap ∆ k corresponds to the time spent at the saddle β k . It is computed as being the elapsed time just before (K1) breaks if the coordinates do not jump.

We update t k +1 = t k +∆ k and s k +1 = s k -∆ k ∇ L ( β k ) : t k +1 corresponds to the time at which the iterates leave the saddle β k and s k +1 constrains the signs of the next saddle β k +1

The solution β k +1 of the constrained minimisation problem is the saddle to which the flow jumps to at time t k +1 . The optimality conditions of this problem are such that Eq. (12) is maintained for t ≥ t k +1 .

Various comments on Algorithm 1. First we point out that any solution β c of the constrained minimisation problem which appears in Algorithm 1 also satisfies β c = arg min β [ i ]=0 for i/ ∈ supp( β c ) L ( β ) as in Eq. (4): the algorithm hence indeed outputs saddles as expected. Up until now we have never checked whether the algorithm's constrained minimisation problem has a unique minimum. This is crucial otherwise the assignment step would be ill-defined. Showing the uniqueness is non-trivial and is guaranteed thanks to the general position Assumption 1 on the data (see Proposition 7 in Appendix D.1). In this same proposition, we also show that the algorithm terminates in at most min(2 d , ∑ n k =0 ( d k ) ) steps, that the loss strictly decreases at each step and that the final output β p is the minimum ℓ 1 -norm solution. These last two properties are expected given the fact that the algorithm arises as being the limit process of ˜ β α which follows the mirror flow Eq. (10).

Links with the LARS algorithm for the Lasso. Recall that the Lasso problem [46, 15] is formulated as:

The optimality condition of Eq. (13) writes -∇ L ( β ⋆ λ ) ∈ λ∂ ∥ β ⋆ λ ∥ 1 . Now notice the similarity with Eq. (12): the two would be equivalent with λ = 1 /t if the integration on the left hand side of Eq. (12) did not average over the whole trajectory but only on the final iterate, in which case -∫ t 0 ∇ L ( ˜ β ◦ t )d s = -t · ∇ L ( ˜ β ◦ t ) . Though the difference is small, the trajectories of our limiting trajectory ˜ β ◦ and the lasso path ( β ⋆ λ ) λ are quite different: one has jumps, whereas the other is continuous. Nonetheless, the construction of Algorithm 1 shares many similarities with that of the Least Angle Regression (LARS) algorithm [19] (originally named the Homotopy algorithm [39]) which is used to compute the Lasso path. A notable difference however is the fact that each step of our algorithm depends on the whole trajectory through the vector s , whereas the LARS algorithm can be started from any point on the path.

3.3 Outputs of the algorithm under a RIP and gap assumption on the data.

Unlike previous results on incremental learning, complex behaviours can occur when the feature matrix is ill designed: several coordinates can activate and deactivate at the same time (see Appendix A for various cases). However, if the feature matrix satisfies the 2 r -restricted isometry property (RIP) [14] and there exists an r -sparse solution β ⋆ , the visited saddles can be easily approximated using Algorithm 1. We provide the precise characterisation below.

Sparse regression with RIP and gap assumption. (RIP) Assume that there exists an r -sparse vector β ⋆ such that y i = ⟨ x i , β ⋆ ⟩ . Furthermore we assume that the feature matrix X ∈ R n,d satisfies the 2 r -restricted isometry property with constant ˜ ε < √ 2 -1 < 1 / 2 : i.e. for all submatrix X s where we extract any s ≤ 2 r columns of X , the matrix X ⊤ s X s /n of size s × s has all its eigenvalues in the interval [1 -˜ ε, 1 + ˜ ε ] . (Gap assumption) Furthermore we assume that the r -sparse vector β ⋆ has coordinates which have a 'sufficient gap'. W.l.o.g we write β ⋆ = ( β ⋆ 1 , . . . , β ⋆ r , 0 , . . . , 0) with | β ⋆ 1 |≥ . . . ≥ | β ⋆ r | > 0 and we define λ := min i ∈ [ r ] ( | β ⋆ i |-| β ⋆ i +1 | ) ≥ 0 which corresponds to the smallest gap between the entries of | β ⋆ | . We assume that 5˜ ε ∥ β ⋆ ∥ 2 < λ/ 2 and we let ε := 5˜ ε .

A classic result from compressed sensing (see Candes [13, Theorem 1.2]) is that the 2 r -restricted isometry property with constant √ 2 -1 ensures that the minimum ℓ 0 -minimisation problem has a unique r -sparse solution which is β ⋆ . This means that Algorithm 1 will have β ⋆ as final output and the following proposition shows that we can precisely characterise each of its outputs when the data satisfies the previous assumptions.

Proposition 2. Under the restricted isometry property and the gap assumption stated right above, Algorithm 1 terminates in r -loops and outputs:

Informally, this means that the algorithm terminates in exactly r loops and outputs jump times and saddles roughly equal to t i = 1 / | β ⋆ i | and β i = ( β ⋆ 1 , · · · , β ⋆ i , 0 , . . . , 0) . Therefore, in simple settings, the support of the sparse vector is learnt a coordinate at a time, without any deactivations. We refer to Appendix D.2 for the proof.

4 Convergence of the iterates towards the process defined by Algorithm 1

Weare now fully equipped to state our main result which formalises the convergence of the accelerated iterates towards the limiting process ˜ β ◦ which we built in the previous section.

Theorem 2. Let the saddles ( β 0 = 0 , β 1 , . . . , β p -1 , β p = β ⋆ ℓ 1 ) and jump times ( t 0 = 0 , t 1 , . . . , t p ) be the outputs of Algorithm 1 and let ( ˜ β ◦ t ) t be the piecewise constant process defined as follows:

The accelerated flow ( ˜ β α t ) t defined in Eq. (9) uniformly converges towards the limiting process ( ˜ β ◦ t ) t on any compact subset of R ≥ 0 \{ t 1 , . . . , t p } .

Convergence result. We recall that from a technical point of view, showing the existence of a limiting process lim α → 0 ˜ β α is the toughest part. Theorem 2 provides this existence as well as the uniform convergence of the accelerated iterates towards ˜ β ◦ over all closed intervals of R which do not contain the jump times. We highlight that this is the strongest type of convergence we could expect and a uniform convergence over all intervals of the form [0 , T ] is impossible given that the limiting process ˜ β ◦ is discontinuous. In Proposition 3, we give an even stronger result by showing a graph convergence of the iterates which takes into account the path followed between the jumps. We also point out that we can easily show the same type of convergence for the accelerated weights ˜ w α t := w α ˜ t α ( t ) . Indeed, using the bijective mapping which links the weights w t and the predictors β t (see Lemma 1 in Appendix C), we immediately get that the accelerated weights (˜ u α , ˜ v α ) uniformly converge towards

Estimates for the non-accelerated iterates β α t . We point out that our result provides no speed of convergence of ˜ β α towards ˜ β ◦ . We believe that a non-asymptotic result is challenging and leave it as future work. Note that we experimentally notice that the convergence rate quickly degrades after each saddle. Nonetheless, we can still write for the non-accelerated iterates that β α t = ˜ β α t/ ln(1 /α ) ∼ ˜ β ◦ t/ ln(1 /α ) as α → 0 . Hence, for α small enough the iterates β α t are roughly equal to 0 until time t 1 · ln(1 /α ) and the minimum ℓ 1 -norm interpolator is reached at time t p · ln(1 /α ) . Such a precise estimate of the global convergence time is rather remarkable and goes beyond classical Lyapunov analysises which only leads to L ( β α t ) ≲ ln(1 /α ) /t (see Proposition 4 in Appendix C).

Natural extensions of our setting. More general initialisations can easily be dealt with. For instance, initialisations of the form u t =0 = α u 0 ∈ R d lead to the exact same result as it is shown in [50] (Discussion after Theorem 1) that the associated mirror still converges to the ℓ 1 -norm. Initialisations of the form [ u t =0 ] i = α k i , where k i > 0 , lead to the associated potential converging towards a weighted ℓ 1 -norm and one should modify Algorithm 1 by accordingly weighting ∇ L ( β ) in the algorithm. Also, deeper linear architectures of the form β w = w D + -w D -as in [50] do not change our result as the associated mirror still converges towards the ℓ 1 -norm. Though we only consider the square loss in the paper, we believe that all our results should hold for any loss of the type L ( β ) = ∑ n i =1 ℓ ( y i , ⟨ x i , β ⟩ ) where for all y ∈ R , ℓ ( y, · ) is strictly convex with a unique minimiser at y . In fact, the only property which cannot directly be adapted from our results is showing the uniform boundedness of the iterates (see discussion before Proposition 5 in Appendix C).

4.1 High level sketch of proof of ˜ β α → ˜ β ◦ which leverages an arc-length parametrisation

In this section, we give the high level ideas concerning the proof of the convergence ˜ β α → ˜ β ◦ given in Theorem 2. A full and detailed proof can be found in Appendix E. The main difficulty stems from the non-continuity of the limit process ˜ β ◦ . To circumvent this difficulty, a clever trick which we borrow to [18, 36] is to 'slow-down' time when the jumps occur by considering an arc-length parametrisation of the path . We consider the R ≥ 0 arclength bijection τ α and leverage it to define the 'appropriately slowed down' iterates ˆ β α τ as:

This time reparametrisation has the fortunate but crucial property of leading to ˙ ˆ t α ( τ ) + ∥ ˙ ˆ β α τ ∥ = 1 by a simple chain rule, which means that the speed of ( ˆ β α τ ) τ is uniformly upperbounded by 1 independently of α . This behaviour is in stark contrast with the process ( ˜ β α t ) t which has a speed which explodes at the jumps. This change of time now allows us to use Arzelà-Ascoli's theorem to extract a subsequence which uniformly converges to a limiting process which we denote ˆ β . Importantly, ˆ β enables to keep track of the path followed between the jumps as we show that its trajectory has two regimes:

The process ˆ β is illustrated on the right: the red curves correspond to the paths which the iterates follow during the jumps. These paths are called heteroclinic orbits in the dynamical systems literature [31, 3]. To prove Theorem 2, we can map back the convergence of ˆ β α to show that of ˜ β α . Moreover from the convergence ˆ β α → ˆ β we get a more complete picture of the limiting dynamics of ˜ β α as it naturally implies the convergence of the graph of the iterates ( ˜ β α t ) t converges towards that of ( ˆ β τ ) τ . The graph convergence result is formalised in this last proposition.

β

/star

β

0

β

1

β

2

˙ ˆ

β

∝ -|

ˆ

β

| /circledot∇

L

( ˆ

β

)

Connections

( ˆ

β

τ

)

τ

Proposition 3. For all T > t p , the graph of the iterates ( ˜ β α t ) t ≤ T converges to that of ( ˆ β τ ) τ :

<latexi sh

1_b

64="Smj5VR

/v

P8

DL

oHB

A

>

COX

cd

N

y

E

3

Gr+g

TQ

0

z

u

MJ

k

q

p

9

I

f

K

W

Y

2

F

Zn

w

U

7

<latexi sh

1_b

64="0

R

CMg

U

7wj3BK

Wuo+YPZLEI

>A

8n cdVDJS

N

O2

X

G

y

p

vH

Q

9

k

F

r

q

f

5

/

z

T

m

<latexi sh

1_b

64="+3wDNc9

v

2

ukCEr5

UA

/Y

>

B8X

V

S

q

L

0

GPR

K

g

m7

H

T

Q

M

f

F

dW

o

Z

y

I

J

z

j

n

p

O

<latexi sh

1_b

64="n fdc9mwPMjQ

/S

5

K

0

g

>A

B8X

V

N

EJ3

W

O

k

rGC

2

y7

LuR

+

D

o

z

Y

q

U

FI

H

p

T

Z

v

<latexi sh

1_b

64="

O

jI02Y

Bgz kSEQ

7W

rKu8

>A

C

H

cdV

9

wN

3

/

T

P

o

m

f

p

+FR

L

GX

M

v

y

J

U

5

Z

D

q

n

<latexi sh

1_b

64="BPwGv

r

Vn

2f+

D

W

>A

X

c

NS

EJ3U

q/

9L

0

R

K

m

H

T

5

Q

u

F

o

Z

kp

j

z

C

Y

y

I

O

5 Further discussion and conclusion

Link between incremental learning and saddle-to-saddle dynamics. The incremental learning phenomenon and the saddle-to-saddle process are often complementary facets of the same idea and refer to the same phenomenon. Indeed for gradient flows d w t = -∇ F ( w t )d t , fixed points of the dynamics correspond to critical points of the loss. Stages with little progress in learning and minimal movement of the iterates necessarily correspond to the iterates being in the vicinity of a critical point of the loss. It turns out that in many settings (linear networks [30], matrix sensing [8, 41]), critical points are necessarily saddle points of the loss (if not global minima) and that they have a very particular structure (high sparsity, low rank, etc.). We finally note that an alternative approach to realising saddle-to-saddle dynamics is through the perturbation of the gradient flow by a vanishing noise as studied in [6].

Characterisation of the visited saddles. A common belief is that the saddle-to-saddle trajectory can be found by successively computing the direction of most negative curvature of the loss (i.e. the eigenvector corresponding to the most negative eigenvalue) and following this direction until reaching the next saddle [26]. However this statement cannot be accurate as it is inconsistent with our algorithm in our setting. In fact, it can be shown that this algorithm would match the orthogonal matching pursuit (OMP) algorithm [42, 17] which does not necessarily lead to the minimum ℓ 1 -norm interpolator. In [7], which is the closest to our work and the first to prove convergence of the iterates towards a piece-wise constant process, the successive saddles are entirely characterised and connected to the Lasso regularisation path in the underparameterised setting. Recently, [9] extended the diagonal linear network setting to diagonal parametrisations of the form f u ⊙ v , but at the cost of stronger assumptions on the trajectory.

Adaptive Inverse Scale Space Method. Following the submission of our paper, we were informed that Algorithm 1 had already been proposed and analysed in the compressed sensing literature. Indeed it exactly corresponds to the Adaptive Inverse Scale Space Method (aISS) proposed in [11]. The motivations behind its study are extremely different from ours and originate from the study of Bregman iteration [12, 40, 52] which is an efficient method for solving ℓ 1 related minimisation problems. The so-called inverse scale space flow which corresponds to Eq. (12) in our paper can be seen as the continuous version of Bregman iteration. As in our paper, [11] show that this equation can be solved through an iterative algorithm. We refer to [51, Section 2] for further details. However we did not find any results in this literature concerning the uniqueness of the constrained minimisation problem due to Assumption 1, nor on the maximum number of iterations, the behaviour under RIP assumptions and the maximum number of active coordinates.

Subdifferential equations and rate-independent systems. As in Eq. (12), subdifferential inclusions of the form ∇ L ( β t ) ∈ d d t ∂h ( β t ) for non-differential functions h have been studied by Attouch et al. [4] but for strongly convex functions h . In this case, the solutions are continuous and do not exhibit jumps. On another hand, [18, 36, 37] consider so-called rate-independent systems of the form ∂ q E ( t, q t ) ∈ ∂h ( ˙ q t ) for 1 -homogeneous dissipation potentials h . Examples of such systems are ubiquitous in mechanics and appear in problems related to friction, crack propagation, elastoplasticity and ferromagnetism to name a few [35, Ch. 6 for a survey]. As in our case, the main difficulty with such processes is the possible appearance of jumps when the energy E is non-convex.

Conclusion. Our study examines the behaviour of gradient flow with vanishing initialisation over diagonal linear networks. We prove that it leads to the flow jumping from a saddle point of the loss to another. Our analysis characterises each visited saddle point as well as the jumping times through an algorithm which is reminiscent of the LARS method used in the Lasso framework. There are several avenues for further exploration. The most compelling one is the extension of these techniques to broader contexts for which the implicit bias of gradient flow has not yet fully been understood.

Acknowledgments. S.P. would like to thank Loucas Pillaud-Vivien for introducing him to this beautiful topic and for the many insightful discussions. S.P. also thanks Quentin Rebjock for the many helpful discussions and Johan S. Wind for reaching out and providing the reference of [11]. The authors also thank Jérôme Bolte for the discussions concerning subdifferential equations, Aris Daniilidis for the reference of [32], as well as Aditya Varre and Mathieu Even for proofreading the paper.

References

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory , pages 2552-2623. PMLR, 2023.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

Peter Ashwin and Michael Field. Heteroclinic networks in coupled cell systems. Arch. Ration. Mech. Anal. , 148(2):107-143, 1999.

H. Attouch, J. Bolte, P. Redont, and M. Teboulle. Singular Riemannian barrier methods and gradient-projection dynamical systems for constrained optimization. Optimization , 53(5-6): 435-454, 2004.

Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 468-477. PMLR, 18-24 Jul 2021.

Yuri Bakhtin. Noisy heteroclinic networks. Probab. Theory Related Fields , 150(1-2):1-42, 2011.

Raphaël Berthier. Incremental learning in diagonal linear networks. arXiv preprint arXiv:2208.14673 , 2022.

Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Global optimality of local search for low rank matrix recovery. In Advances in Neural Information Processing Systems , volume 29, 2016.

Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind. Transformers learn through gradual rank increase. arXiv preprint arXiv:2306.07042 , 2023.

Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow reLU networks for square loss and orthogonal inputs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Martin Burger, Michael Möller, Martin Benning, and Stanley Osher. An adaptive inverse scale space method for compressed sensing. Mathematics of Computation , 82(281):269-299, 2013.

Jian-Feng Cai, Stanley Osher, and Zuowei Shen. Split bregman methods and frame based image restoration. Multiscale modeling & simulation , 8(2):337-369, 2010.

Emmanuel J Candes. The restricted isometry property and its implications for compressed sensing. Comptes rendus mathematique , 346(9-10):589-592, 2008.

E. Candès, J. Romberg, and T. Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics , 59(8):1207-1223, 2006.

Scott Shaobing Chen, David L Donoho, and Michael A Saunders. Atomic decomposition by basis pursuit. SIAM review , 43(1):129-159, 2001.

Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 1305-1338. PMLR, 09-12 Jul 2020.

Geoff Davis, Stephane Mallat, and Marco Avellaneda. Adaptive greedy approximations. Constructive approximation , 13:57-98, 1997.

Messoud A. Efendiev and Alexander Mielke. On the rate-independent limit of systems with dry friction and small viscosity. J. Convex Anal. , 13(1):151-167, 2006.

Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. Least angle regression. 2004.

Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s)gd over diagonal linear networks: Implicit regularisation, large stepsizes and edge of stability. arXiv preprint arXiv:2302.08982 , 2023.

Udaya Ghai, Elad Hazan, and Yoram Singer. Exponentiated gradient meets gradient descent. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the 31st International Conference on Algorithmic Learning Theory , volume 117 of Proceedings of Machine Learning Research , pages 386-407. PMLR, 08 Feb-11 Feb 2020.

Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely. The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations , 2020.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems , volume 30, 2017.

Jeff Z HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory , pages 2315-2357. PMLR, 2021.

Arthur Jacot, François Ged, Berfin ¸ Sim¸ sek, Clément Hongler, and Franck Gabriel. Saddle-tosaddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933 , 2021.

Liwei Jiang, Yudong Chen, and Lijun Ding. Algorithmic regularization in model-free overparametrized asymmetric matrix factorization. arXiv preprint arXiv:2203.02839 , 2022.

Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon S Du, and Jason D Lee. Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing. arXiv preprint arXiv:2301.11500 , 2023.

Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems , 32, 2019.

Kenji Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information Processing Systems , volume 29, 2016.

M. Krupa. Robust heteroclinic cycles. J. Nonlinear Sci. , 7(2):129-176, 1997.

Krzysztof Kurdyka. On gradients of functions definable in o-minimal structures. Ann. Inst. Fourier (Grenoble) , 48(3):769-783, 1998.

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations , 2021.

Julien Mairal and Bin Yu. Complexity analysis of the lasso regularization path. arXiv preprint arXiv:1205.0079 , 2012.

Alexander Mielke. Evolution of rate-independent systems. Evolutionary equations , 2:461-559, 2005.

Alexander Mielke, Riccarda Rossi, and Giuseppe Savaré. Modeling solutions with jumps for rate-independent systems on metric spaces. Discrete Contin. Dyn. Syst. , 25(2):585-615, 2009.

Alexander Mielke, Riccarda Rossi, and Giuseppe Savaré. Variational convergence of gradient flows and rate-independent evolutions in metric spaces. Milan Journal of Mathematics , 80: 381-410, 2012.

Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953 , 2017.

Michael R Osborne, Brett Presnell, and Berwin A Turlach. A new approach to variable selection in least squares problems. IMA journal of numerical analysis , 20(3):389-403, 2000.

Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin. An iterative regularization method for total variation-based image restoration. Multiscale Modeling & Simulation , 4(2):460-489, 2005.

Dohyung Park, Anastasios Kyrillidis, Constantine Carmanis, and Sujay Sanghavi. Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 65-74. PMLR, 20-22 Apr 2017.

Yagyensh Chandra Pati, Ramin Rezaiifar, and Perinkulam Sambamurthy Krishnaprasad. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar conference on signals, systems and computers , pages 40-44. IEEE, 1993.

Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. In Advances in Neural Information Processing Systems , 2021.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In International Conference on Machine Learning , pages 8913-8924. PMLR, 2021.

Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences , 116 (23):11537-11546, 2019.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

Ryan J. Tibshirani. The lasso problem and uniqueness. Electron. J. Stat. , 7:1456-1490, 2013.

Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems , 32, 2019.

Johan S Wind, Vegard Antun, and Anders C Hansen. Implicit regularization in ai meets generalized hardness of approximation in optimization-sharp results for diagonal linear networks. arXiv preprint arXiv:2307.07410 , 2023.

Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 3635-3673. PMLR, 09-12 Jul 2020.

Yi Yang, Michael Möller, and Stanley Osher. A dual split bregman method for fast l1 minimization. Mathematics of computation , 82(284):2061-2085, 2013.

WYin, S Osher, D Goldfarb, and J Darbon. Bregman iterative algorithms for l1-minimization with applications to compressed sensing: Siam journal on imaging sciences, 1, 143-168. LIST OF FIGURES , 2008.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017.

Organisation of the Appendix.

In Appendix A, we give the experimental setup and provide additional experiments.

In Appendix B, we prove Proposition 1 and provide additional comments concerning the unicity of the minimisation problem which appears in the proposition.

In Appendix C, we provide some general results on the flow.

In Appendix D, we prove Proposition 2 and give standalone properties of Algorithm 1.

In Appendix E, we explain in more detail the arc-length parametrisation explained in the main text as well as prove Theorem 2 and Proposition 3.

In Appendix F, we provide technical lemmas which are useful to prove the main results.

A Experimental setup and additional: experiments, extension, related works.

Experimental setup and additional experiments. For each experiment we generate our dataset as y i = ⟨ x i , β ⋆ ⟩ where x i = N ( 0 , H ) for a a diagonal covariance matrix H and β ⋆ is a vector of R d . Gradient descent is run with a small step size and from initialisation u t =0 = √ 2 α 1 ∈ R d and v t =0 = 0 for some initialisation scale α > 0 .

Figure 1 and Figure 4 (Left): ( n, d, α ) = (5 , 7 , 10 -120 ) , H = I d , β ⋆ = (10 , 20 , 0 , 0 , 0 , 0 , 0) ∈ R 7 .

Figure 3 (Left): ( n, d, α 1 , α 2 ) = (7 , 2 , 10 -100 , 10 -10 ) , H = I d , β ⋆ = (10 , 20) ∈ R 7 .

Figure 4 (Right): ( n, d, α ) = (6 , 6 , 10 -10 ) , H = diag(1 , 10 , 10 , 10 , 10 , 10) ∈ R 6 × 6 , β ⋆ = (1 , 0 , 0 , 0 , 0 , 0 , 0) ∈ R 6 .

Figure 3 (Right): ( n, d, α ) = (3 , 3 , 10 -100 ) , X is the square root matrix of the matrix ((20 , 6 , -1 . 4) , (6 , 2 , -0 . 4) , ( -1 . 4 , -0 . 4 , 0 . 12)) ∈ R 3 × 3 , β ⋆ = (1 , 9 , 10) .

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Time

t

0

5

10

15

20

Accelerated iterates

˜

β

α

t

=

β

α

ln(1

/α

)

t

˜

β

α

1

t

(1)

˜

β

α

1

t

(2)

˜

β

α

2

t

(1)

˜

β

α

2

t

(2)

10

2

10

1

10 0

10 1

10 2

Time

t

0

2

4

6

8

10

Accelerated iterates

t

=

ln(1/

)

t

t

(1)

t

(2)

t

(3)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

-

5

0

5

10

15

20

Accelerated iterates

˜

β

α

t

=

β

α

ln(1

/α

)

t

˜

β

α

t

(1)

˜

β

α

t

(2)

˜

β

α

t

(3)

˜

β

α

t

(4)

˜

β

α

t

(5)

˜

β

α

t

(6)

˜

β

α

t

(7)

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

Time

t

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Accelerated iterates

˜

β

α

t

=

β

α

ln(1

/α

)

t

˜

β

α

t

(1)

˜

β

α

t

(2)

˜

β

α

t

(3)

˜

β

α

t

(4)

˜

β

α

t

(5)

˜

β

α

t

(6)

Time

t

B Proof of Proposition 1

Proposition 1. All the critical points w c of F which are not global minima, i.e., ∇ F ( w c ) = 0 and F ( w c ) > min w F ( w ) , are necessarily saddle points ( i.e., not local extrema). They map to parameters β c = u c ⊙ v c which satisfy | β c |⊙∇ L ( β c ) = 0 and:

where supp( β c ) = { i ∈ [ d ] , β c [ i ] = 0 } corresponds to the support of β c .

̸

Proof. Non-existence of maxima / non-global minima. This is a simpler version of results which appear in [30], for the sake of completeness we provide here a simple proof adapted to our setting. The intuition follows the fact that if there existed a local maximum / non-global minimum for F then this would translate to the existence of a local maximum / non-global minimum for the convex loss L , which is absurd.

Assume that there exists a local maximum w ⋆ = ( u ⋆ , v ⋆ ) , i.e. assume that there exists ε > 0 such that for all w = ( u, v ) such that ∥ w -w ⋆ ∥ 2 2 ≤ ε , F ( w ) ≤ F ( w ⋆ ) . We show that this would imply that β ⋆ = u ⋆ ⊙ v ⋆ is a local maximum of L , which is absurd.

The mapping g : ( u, v ) ↦→ ( u ⊙ v, √ ( u 2 -v 2 ) / 2) from R d ≥ 0 × R d → R d × R d ≥ 0 is a bijection with inverse

Also notice that F ( g -1 ( β, α )) = L ( β ) for all β and α . Now let ˜ ε > 0 and let β ∈ R d such that ∥ β -β ⋆ ∥ 2 2 ≤ ˜ ε , then for ( u, v ) = g -1 ( β, α ∗ ) where α ∗ = √ (( u ⋆ ) 2 -( v ⋆ ) 2 ) / 2 we have that:

where the last inequality is for ˜ ε small enough. This means that L ( β ) = F ( w ) ≤ F ( w ⋆ ) = L ( β ⋆ ) and β ⋆ is a local maximum of L , which is absurd.

The exact same proof holds to show that there are no local minima of F which are not global minima.

Critical points. The gradient of the loss function F writes:

̸

Therefore ∇ F ( w c ) = 0 ∈ R 2 d implies that ∇ L ( β c ) ⊙ β c = 0 ∈ R d . Now consider such a β c and let supp( β c ) = { i ∈ [ d ] such that β c ( i ) = 0 } denote the support of β c . Since [ ∇ L ( β c )] i = 0 for i / ∈ supp( β c ) , we can therefore write that

Furthermore we point out that since supp( β c ) ⊂ [ d ] , there are at most 2 d distinct sets supp( β c ) , and therefore at most 2 d values F ( w c ) = L ( β c ) , where w c is a critical point of F .

Additional comment concerning the uniqueness of arg min β i =0 ,i ̸∈ supp( β c ) L ( β ) .

We point out that the constrained minimisation problem (4) does not necessarily have a unique solution, even when β c is not a global solution. Though not required for any of our results, for the sake of completeness, we show here that under an additional mild assumption on the data, we can ensure that the minimisation problem (4) which appears in Proposition 1 has a unique minimum when L ( β c ) > 0 . Under this additional assumption, there is therefore a finite number of saddles β c . Recall that we let X ∈ R n × d be the feature matrix and (˜ x 1 , . . . , ˜ x d ) be its columns. Now assume temporarily that the following assumption holds.

Assumption 2 (Assumption used just in this short section) . Any subset of (˜ x 1 , . . . , ˜ x d ) of size smaller than min( n, d ) is linearly independent.

One can easily check that this assumption holds with probability 1 as soon as the data is drawn from a continuous probability distribution, similarly to [47, Lemma 4]). In the following, for a subset ξ = { i 1 , . . . , i k } ⊂ [ d ] , we write X ξ = (˜ x i 1 , . . . , ˜ x i k ) ∈ R n × k (we extract the columns from X ). For a vector β ∈ R d we write β [ ξ ] = ( β i 1 , . . . , β i k ) and β [ ξ C ] = ( β i ) i/ ∈ ξ . We distinguish two different settings:

Underparametrised setting ( n ≥ d ) : in this case, for any ξ = { i 1 , . . . , i k } ⊂ [ d ] , then β ⋆ := argmin β i =0 ,i ̸∈ ξ L ( β ) is unique. Indeed we simply set the gradient to 0 and notice that due to Assumption 2, there exists a unique solution, indeed it is β ⋆ such that β ⋆ [ ξ ] = ( X ⊤ ξ X ξ ) -1 X ⊤ ξ y and β ⋆ [ ξ C ] = 0 .

Overparametrised setting ( d > n ) : Global solutions: arg min β ∈ R d L ( β ) is an affine space spanned by the orthogonal of ( x 1 , . . . , x n ) in R d . Since span(˜ x 1 , . . . , ˜ x d ) = R n from Assumption 2, any β ⋆ ∈ arg min β ∈ R d L ( β ) satisfies Xβ ⋆ = y and L ( β ⋆ ) = 0 . "Saddle points": now let β c ∈ R d be such that we can write β c ∈ arg min β i =0 ,i / ∈ supp( β c ) L ( β ) and assume that L ( β c ) > 0 (i.e., not a global solution), then: (1) β c has at most n non-zero entries, indeed if it were not the case, then y would necessarily belong to span(˜ x i ) i ∈ supp( β c ) due to the assumption on the data, and this would lead to L ( β c ) = 0 , (2) therefore, similar to the underparametrised case, arg min β i =0 ,i / ∈ supp( β c ) L ( β ) is unique, equal to β c , and we have that β c [ ξ ] = ( X ⊤ ξ X ξ ) -1 X ⊤ ξ y and β c [ ξ C ] = 0 where ξ = supp( β c ) .

Thus, in both the underparametrised and overparametrised settings, the minimisation problem (4) appearing in Proposition 1 has a unique minimum when L ( β c ) > 0 and Assumption 2 holds.

C General results on the iterates

In the following lemma we recall a few results concerning the gradient flow Eq. (5):

where F is defined in Eq. (3) as:

Lemma 1. For an initialisation u 0 = √ 2 α , v 0 = 0 , the flow w α t = ( u α t , v α t ) from Eq. (15) is such that the quantity ( u α t ) 2 -( v α t ) 2 is constant and equal to 2 α 2 1 . Furthermore u α t > | v α t |≥ 0 and therefore from the bijection Eq. (14) we have that:

Proof. From the expression of ∇ F ( w ) , notice that the derivative of ( u α t ) 2 -( v α t ) 2 is equal to 0 and therefore equal to its initial value.

In this section we consider the accelerated iterates Eq. (9) which follow:

with ˜ β t =0 = 0 and where ϕ α is defined Eq. (7).

Proposition 4. For all α > 0 and minimum β ⋆ ∈ arg min β L ( β ) , the loss values L ( ˜ β α t ) and the Bregman divergence D ˜ ϕ α ( β ⋆ , ˜ β α t ) are decreasing. Moreover

Proof. The loss is decreasing since: d d t L ( ˜ β α t ) = ∇ L ( ˜ β α t ) ⊤ ˙ β α t = -˙ ˜ β α ⊤ t ∇ 2 ˜ ϕ α ( ˜ β α t ) ˙ ˜ β α t ≤ 0 .

d d t D ˜ ϕ α ( β ⋆ , ˜ β α t ) = -∇ L ( ˜ β α t ) ⊤ ( ˜ β α t -β ⋆ ) = -2( L ( ˜ β α t ) -L ( β ⋆ )) (since L is the quadratic loss), therefore the Bregman distance is decreasing. We can also integrate this last equality from 0 to t , and divide by -2 t :

Since the loss is decreasing we get that L ( ˜ β α t ) -L ( β ⋆ ) ≤ ˜ ϕ α ( β ⋆ ) 2 t and from the convexity of L we get that L ( 1 t ∫ t 0 ˜ β α s d s ) -L ( β ⋆ ) ≤ ˜ ϕ α ( β ⋆ ) 2 t .

In the following proposition, we show that for α small enough, the iterates are bounded independently of α . Note that this result unfortunately only holds for the quadratic loss, we expect it to hold for other convex losses of the type L ( β ) = 1 n ∑ i ℓ ( y i , ⟨ x i , β ⟩ ) where ℓ ( y, · ) is strictly convex has a unique root at y but we don't know how to show it. Also note that bounding the accelerated iterates ˜ β α is equivalent to bounding the iterates β α since ˜ β α t = β α ln(1 /α ) t .

Proposition 5. For α < α 0 , where α 0 depends on β ⋆ ℓ 1 , the iterates ˜ β α t are bounded independently of α :

Proof. From Eq. (16), integrating and using that L is the quadratic loss, we get:

where we recall that X ∈ R n × d is the input data represented as a matrix and where we denote the averaged iterate by ¯ β α t = 1 t ∫ t 0 ˜ β α s d s . Thus we get

By convexity of ˜ ϕ α we have ˜ ϕ α ( β α t ) -˜ ϕ α ( β ⋆ ) ≤ ∇ ˜ ϕ α ( β α t ) ⊤ ( β α t -β ⋆ ) . By the Cauchy-Schwarz inequality, we also have ( ¯ β α t -β ⋆ ) ⊤ X ⊤ X ( β α t -β ⋆ ) ≤ ∥ X ( β α t -β ⋆ ) ∥∥ X ( ¯ β α t -β ⋆ ) ∥ . Using Proposition 4: ∥ X ( β α t -β ⋆ ) ∥ 2 ≤ n ˜ ϕ α ( β ⋆ ) /t and ∥ X ( ¯ β α t -β ⋆ ) ∥ 2 ≤ n ˜ ϕ α ( β ⋆ ) /t we can further bound the right hand side of Eq. (19) as

From [50] (proof of Lemma 1 in the appendix) we get that for

Thus it yields then:

and for all α < exp( -d/ 2) :

which finally leads for

to the result.

The following proposition shows that we can bound the path length of the flow ˜ β α independently of α . Keep in mind that the path length of ˜ β α is equivalent to that of β α as the first is just an acceleration of the second: ˜ β α t = β α ln(1 /α ) t .

Proposition 6. For α < α 0 where α 0 is the same as in Proposition 5, the path length of the iterates ( β α t ) t ≥ 0 is bounded independently of α > 0 :

where C does not depend on α . Hence the path length of the accelerated flow ˜ β α is also bounded independently of α .

Proof. Having shown that the iterates β α t are bounded independently of α , it also implies that the iterates w t = ( u t , v t ) are bounded following Lemma 1. Since the loss w ↦→ F ( w ) is a multivariate polynomial function, it is a semialgebraic function and we can consequently apply the result of Kurdyka [32, Theorem 2] which grants that

where the constant C only depends on the loss and on the bound on the iterates. We further use that ˙ β = ˙ u ⊙ v + u ⊙ ˙ v and ∥ ˙ u ⊙ v + u ⊙ ˙ v ∥≤ C 1 ( ∥ ˙ u ∥ + ∥ ˙ v ∥ ) using that u and v are bounded and ∥ ˙ u ∥ + ∥ ˙ v ∥≤ C 2 ∥ ˙ w ∥ using the equivalence of norms. Therefore ∫ + ∞ 0 ∥ ˙ β α t ∥ d t < C for some C which is independent of the initialisation scale α .

D Standalone properties of Algorithm 1

D.1 'Well-definedness' of Algorithm 1 and upperbound on its number of loops

Notice that this proposition highlights the fact that Algorithm 1 is on its own an algorithm of interest for finding the minimum ℓ 1 -norm solution in an overparametrised regression setting. We point out that the provided upperbound on the number of iterations is very crude and could certainly be improved.

Proposition 7. Algorithm 1 is well defined: at each iteration (i) the attribution of ∆ is well defined as ∆ < + ∞ , (ii) the constrained minimisation problem has a unique solution and the attribution of the value of β is therefore well-founded. Furthermore, along the loops: the iterates β have at most n non-zero coordinates, the loss is strictly decreasing and the algorithm terminates in at most min(2 d , ∑ n k =0 ( d k ) ) steps by outputting the minimum ℓ 1 -norm solution β ⋆ ℓ 1 := arg min β ∈ arg min L ∥ β ∥ 1 .

Proof. In the following, for the matrix X and for a subset I = { i 1 , . . . , i k } ⊂ [ d ] , we write X I = (˜ x i 1 , . . . , ˜ x i k ) ∈ R n × k (we extract the columns from X ). For a vector β ∈ R d we write β I = ( β i 1 , . . . , β i k ) .

(1) The constrained minimisation problem has a unique solution: we follow the proof of [47, Lemma 2]. Following the notations in Algorithm 1, we define I = { i ∈ [ d ] , | s i | = 1 } and we point out that after k loops of the algorithm, the value of s is equal to s = -(∆ 1 ∇ L ( β 0 ) + · · · + ∆ k ∇ L ( β k -1 )) ∈ span( x 1 , . . . , x n ) . We can therefore write s = X ⊤ r for some r ∈ R n .

Now assume that ker( X I ) = { 0 } . Then, for some i ∈ I , we have ˜ x i = ∑ j ∈ I \{ i } c j ˜ x j where c j ∈ R . Without loss of generality, we can assume that I \ { i } has at most n elements. Indeed, we can otherwise always find n elements ˜ I ⊂ I \ { i } such that ˜ x i = ∑ j ∈ ˜ I c j ˜ x j . Rewriting the previous equality, we get

Now by definitions of the set I and of r , we have that ⟨ ˜ x j , r ⟩ = s j ∈ { +1 , -1 } for any j ∈ I . Taking the inner product of Eq. (20) with r , we obtain that 1 = ∑ j ∈ I \{ i } ( s i s j c j ) . Consequently, we have shown that if ker( X I ) = { 0 } , then we necessarily have for some i ∈ I ,

with ∑ j ∈ I \{ i } a j = 1 , which means that s i ˜ x i lies in the affine space generated by ( s j ˜ x j ) j ∈ I \{ i } . This fact is however impossible due to Assumption 1 (recall that without loss of generality we have that I \ { i } has at most n elements, and trivially less that d elements). Therefore X I is full rank , and Card( I ) ≤ n . Now notice that the constrained minimisation problem corresponds to arg min β i ≥ 0 ,i ∈ I + β i ≤ 0 ,i ∈ I -∥ y -X I β I ∥ 2 2 . Since X I is full rank, this restricted loss is strictly convex and the constrained minimisation problem has a unique minimum.

(2) ∆ < + ∞ : Notice that the optimality conditions of

̸

are (i) β satisfies the constraints, (ii) if i ∈ I + (resp i ∈ I -) then [ -∇ L ( β )] i ≤ 0 (resp [ -∇ L ( β )] i ≥ 0 ) and (iii) if β i = 0 then [ ∇ L ( β )] i = 0 . One can notice that condition (ii) ensures that at each iteration, for δ ≤ ∆ k , s k -1 -δ ∇ L ( β k -1 ) ∈ [ -1 , 1] coordinate wise. Also, if L ( β k -1 ) = 0 , then a coordinate of the vector | s k -1 -δ ∇ L ( β k -1 ) | must necessarily hit 1 , this value of δ corresponds to ∆ k .

̸

(3) The loss is strictly decreasing: Let I k -1 , ± and I k, ± be the equicorrelation sets defined in the algorithm at step k -1 and k , and β k -1 and β k the solutions of the constrained minimisation problems. Also, let i k be the newly added coordinate which breaks the constraint at step k (which we assume to be unique for simplicity). Without loss of generality, assume that s k ( i k ) = +1 . Since the sets

̸

̸

I k -1 , + \ ( I k, + \ { i k } ) and I k -1 , -\ I k, -are (if not empty) only composed of indexes of coordinates of β k -1 which are equal to 0 , one can notice that β k -1 also satisfies the new constraints at step k . Therefore L ( β k ) ≤ L ( β k -1 ) . Now since [ -∇ L ( β k -1 )] i k > 0 , from the strict convexity of the restricted loss on I k , this means that β k ( i k ) > 0 (which also means that newly activated coordinate i k must activate ), and therefore β k -1 = β k and L ( β k ) < L ( β k -1 ) .

(4) The algorithm terminates in at most min ( 2 d , ∑ n k =0 ( d k ) ) steps: Recall that we showed in part (1) of the proof that at each iteration k of the algorithm, I k as at most min( n, d ) elements. Since supp( β k ) ⊂ I k , we have that β k has at most min( n, d ) non-zero elements, also recall that we always have β k = arg min β i =0 ,i / ∈ supp( β k ) L ( β ) (we here have unicity of this minimisation problem following part (1) of the proof). There are hence at most

̸

such minimisation problems. The loss being strictly decreasing, the algorithm cannot output the same solution β at two different loops, and the algorithm must terminate in at most min ( 2 d , ∑ n k =0 ( d k ) ) iterations by outputting a vector β ⋆ such that ∇ L ( β ⋆ ) = 0 , i.e. β ⋆ ∈ arg min L ( β ) .

(5) The algorithm outputs the minimum ℓ 1 -norm solution. Let β ⋆ be the output of the algorithm after p iterations. Notice that by the definition of the successive sets I k, ± and of the constraints on the minimisation problem, we have that at each iteration s k ∈ ∂ ∥ β k ∥ 1 . Therefore s p ∈ ∂ ∥ β ⋆ ∥ 1 . Also, recall from part (1) of the proof that s p ∈ span( x 1 , . . . , x n ) which means that there exists r ∈ R n such that s p = X ⊤ r . Putting the two together we get that X ⊤ r ∈ ∂ ∥ β ⋆ ∥ 1 , this condition along with the fact that L ( β ⋆ ) = min L ( β ) are exactly the KKT conditions of arg min β ∈ arg min L ∥ β ∥ 1 .

To put our upperbound on the number of iterations into perspective, the worst-case number of iterations for the LARS algorithm is (3 d +1) / 2 [34]. Hence Algorithm 1 has fewer iterations in the worst-case setting. Whether an exponential dependency in the dimension is inevitable for Algorithm 1 is unknown and we leave this as future work.

However, when the number of samples is much smaller than the dimension we lose the exponential dependency. Indeed, for ε := n/d ≤ 1 / 2 , we have the upperbound ∑ n k =0 ( d k ) ≤ 2 H ( ε ) d where H ( ε ) = -ε log 2 ( ε ) -(1 -ε ) log 2 (1 -ε ) is the binary entropy. Since for ε ≤ 1 / 2 , H ( ε ) ≤ -2 ε log 2 ( ε ) , we get the upperbound ∑ n k =0 ( d k ) ≤ 2 H ( ε ) d ≤ ( d n ) 2 n , which is much better than 2 d .

D.2 Proof of Proposition 2

As mentioned several times, for general feature matrices X complex behaviours can occur with coordinates deactivating and changing sign several times. Here we show that for simple datasets which have a feature matrix X that satisfy the restricted isometry property (RIP) [14], we can simply determine the jump times and the saddles as a function of the sparse predictor which we seek to recover.

The non-realistic but enlightening extreme case of the RIP assumption is to consider that the feature matrix is such that X ⊤ X/n = I d . In this case, by letting β ⋆ be the unique vector such that y = ⟨ x, β ⋆ ⟩ and assuming that β ⋆ = ( β ⋆ 1 , . . . , β ⋆ r , 0 , . . . , 0) with | β ⋆ 1 | > · · · > | β ⋆ r | > 0 , then the loss writes L ( β ) = ∥ β -β ⋆ ∥ 2 2 / 2 and one can easily check that Algorithm 1 would terminate in r loops and output exactly t i = 1 | β ⋆ i | and β i = ( β ⋆ 1 , . . . , β ⋆ i , 0 , . . . , 0) for i ≤ r (the case where several coordinates of β ⋆ are stricly equal can also be treated: for example if β ⋆ 1 = β ⋆ 2 then the first output of the algorithm is directly β 1 = ( β ⋆ 1 , β ⋆ 2 , 0 , . . . , 0) ).

We now recall the more realistic RIP setting which is an adaptation of the previous observation.

Sparse regression with RIP and gap assumption. (RIP) Assume that there exists an r -sparse vector β ⋆ such that y i = ⟨ x i , β ⋆ ⟩ . Furthermore we assume that the feature matrix X ∈ R n,d satisfies the 2 r -restricted isometry property with constant ˜ ε < √ 2 -1 < 1 / 2 : i.e. for all submatrix X s where we extract any s ≤ 2 r columns of X , the matrix X ⊤ s X s /n of size s × s has all its eigenvalues in the interval [1 -˜ ε, 1 + ˜ ε ] . (Gap assumption) Furthermore we assume that the r -sparse vector β ⋆

has coordinates which have a 'sufficient gap'. W.l.o.g we write β ⋆ = ( β ⋆ 1 , . . . , β ⋆ r , 0 , . . . , 0) with | β ⋆ 1 |≥ . . . ≥ | β ⋆ r | > 0 and we define λ := min i ∈ [ r ] ( | β ⋆ i |-| β ⋆ i +1 | ) ≥ 0 which corresponds to the smallest gap between the entries of | β ⋆ | . We assume that 5˜ ε ∥ β ⋆ ∥ 2 < λ/ 2 and we let ε := 5˜ ε .

A classic result from compressed sensing (see Candes [13, Theorem 1.2]) is that the 2 r -restricted isometry property with constant √ 2 -1 ensures that the minimum ℓ 0 -minimisation problem has a unique r -sparse solution which is β ⋆ . Furthermore it ensures that the minimum ℓ 1 -norm solution is unique and is equal to β ⋆ . This means that Algorithm 1 will have β ⋆ as a final output.

We now recall the result which characterises the outputs of Algorithm 1 when the data satisfies the previous assumptions.

Proposition 2. Under the restricted isometry property and the gap assumption stated right above, Algorithm 1 terminates in r -loops and outputs:

at times t 1 , . . . , t r such that t i ∈ [ 1 | β ⋆ i | + ε ∥ β ⋆ ∥ , 1 | β ⋆ i |-ε ∥ β ⋆ ∥ ] and where ∥·∥ denotes the ℓ 2 norm.

Proof. In all the proof ∥·∥ denotes the ℓ 2 norm ∥·∥ 2 . For simplicity we assume that β ⋆ i > 0 for all i ∈ [ r ] , the proof can easily be adapted to the general case. We first define ξ := X ⊤ X/n -I d . By the restricted isometry property, for any k ≤ 2 r , we have that any k × k square matrix extracted from ξ which we denote ξ kk has its eigenvalues in [ -˜ ε, ˜ ε ] . It also means that the eigenvalues of ( I k + ξ kk ) -1 -I k are in [ 1 1+˜ ε -1 , 1 1 -˜ ε -1] ⊂ [ -2˜ ε, 2˜ ε ] .

We now proceed by induction with the following induction hypothesis:

β k -1 has its support on its ( k -1) first coordinates with | β k -1 [ i ] -β ⋆ i |≤ 5˜ ε ∥ β ⋆ ∥ for i < k

t k ∈ [ 1 β ⋆ k +5˜ ε ∥ β ⋆ ∥ , 1 β ⋆ k -5˜ ε ∥ β ⋆ ∥ ] and s t k [ k ] = 1

s t k [ i ] ∈ [ t k ( β ⋆ i -5˜ ε ∥ β ⋆ ∥ ) , t k ( β ⋆ i +5˜ ε ∥ β ⋆ ∥ )] ⊂ ( -1 , 1) for i > k

From the recurrence hypothesis, the output of the algorithm at step k is hence β k = arg min L ( β ) under the constraint β [ i ] ≥ 0 for i ≤ k and β [ i ] = 0 otherwise. We first search for the solution of the minimisation problem without the sign constraint and still (abusively) denote it β k : we will show that it turns out to satisfy the sign constraint and that it is therefore indeed β k .

In the following, for a vector v , we denote by v [: k ] its k first coordinates. Setting the k first coordinates of the gradient to 0 , we get that [ X ⊤ X ( β k -β ⋆ )][: k ] = 0 , which leads to ( I k + ξ kk ) β k [: k ] = β ⋆ [: k ] + [ ξβ ⋆ ][: k ] , which gives:

where from the bound on the eigenvalues of ( I k + ξ kk ) -1 -I k and ∥ ξβ ⋆ ∥≤ ˜ ε ∥ β ⋆ ∥ :

Therefore

where v 2 = [ ξβ ⋆ ][: k ] + v 1 hence ∥ v 2 ∥ ∞ ≤ ∥ v 2 ∥≤ 5˜ ε ∥ β ⋆ ∥ . Notice that from the definition of λ and the fact that 5˜ ε ∥ β ⋆ ∥ < λ/ 2 we have that β k [: k ] ≥ 0 coordinate-wise, hence verifying the sign constraint. Also note that ∥ β k ∥≤ ∥ β ⋆ ∥ +5˜ ε ∥ β ⋆ ∥≤ 4 ∥ β ⋆ ∥ .

For t ≥ t k , s t = s t k -( t -t k ) ∇ L ( β k ) , and [ ∇ L ( β k )][: k ] = 0 therefore s t [: k ] = s t k [: k ] . Now for i > k , [ -∇ L ( β k )] i = n -1 [ X ⊤ X ( β ⋆ -β k )] i = β ⋆ i +[ ξ ( β k -β ⋆ )] i . Now since ( β k -β ⋆ ) is r -sparse we have that:

Now from the fact that s t [ i ] = s t k [ i ] + ( t -t k ) β ⋆ i +( t -t k )[ ξ ( β k -β ⋆ )] i and using the recurrence hypothesis: s t k [ i ] ∈ [ t k ( β ⋆ i -5˜ ε ∥ β ⋆ ∥ ) , t k ( β ⋆ i + 5˜ ε ∥ β ⋆ ∥ )] , we get (using the bound Eq. (21)) that s t [ i ] ∈ [ t ( β ⋆ i -5˜ ε ∥ β ⋆ ∥ ) , t ( β ⋆ i + 5˜ ε ∥ β ⋆ ∥ )] . From the 'separation assumption' we have that 5˜ ε ∥ β ⋆ ∥ < λ/ 2 and therefore the next coordinate to activate is necessarily the ( k +1) th at time t k +1 with s t k +1 [ k +1] = 1 and:

This proves the recursion. The algorithm cannot stop before iteration r as β ⋆ is the unique minimiser of L that has at most r non-zero coordinates. But it stops at iteration r as β ⋆ is the unique minimiser of L ( β ) under the constraints β i ≥ 0 for i ≤ r and β i = 0 otherwise.

E Proof of Theorem 2 and Proposition 3 through the arc-length parametrisation

In this section, we explain in more details the arc-length reparametrisation which circumvents the apparition of discontinuous jumps and leads to the proof of Theorem 2. The main difficulty to show the convergence stems from the non-continuity of the limit process ˜ β ◦ . Therefore we cannot expect uniform convergence of ˜ β α towards ˜ β as α → 0 . In addition, ˜ β ◦ does not provide any insights into the path followed between the jumps.

Arc-length parametrisation. The high-level idea is to 'slow-down' time when the jumps occur. To do so we follow the approach from [18, 36] and we consider an arc-length parametrisation of the path, i.e., we consider τ α equal to:

In Proposition 6, we showed that the full path length ∫ + ∞ 0 ∥ ˙ β α s ∥ d s is finite and bounded independently of α . Therefore τ α is a bijection in R ≥ 0 . We can then define the following quantities:

By construction, a simple chain rule leads to ˙ ˆ t α ( τ ) + ∥ ˙ ˆ β α τ ∥ = 1 , which means that the speed of ( ˆ β α τ ) τ is always upperbounded by 1 , independently of α . This behaviour is in stark contrast with the process ( ˜ β α t ) t which has a speed which explodes at the jumps. It presents a major advantage as we can now use Arzelà-Ascoli's theorem to extract a converging subsequent. A simple change of variable shows that the new process satisfies the following equations:

started from ˆ β α τ = 0 and ˆ t 0 = 0 . The next proposition states the convergence of the rescaled process, up to a subsequence.

Proposition 8. Let T ≥ 0 . For every α > 0 , let ( ˆ t α , ˆ β α ) be the solution of Eq. (22) . Then, there exists a subsequence ( ˆ t α k , ˆ β α k ) k ∈ N and ( ˆ t, ˆ β ) such that as α k → 0 :

Limiting dynamics. The limits ( ˆ t, ˆ β ) satisfy:

Heteroclinic orbit. In addition, when ˆ β τ is such that | ˆ β τ |⊙∇ L ( ˆ β τ ) = 0 , we have

̸

Furthermore, the loss strictly decreases along the heteroclinic orbits and the path length ∫ T 0 ∥ ˙ ˆ β τ ∥ d τ is upperbounded independently of T .

Proof. Differentiating Eq. (22) and from the Hessian of ˜ ϕ α we get:

Therefore taking the norm on the right hand side we obtain that and therefore

Subsequence extraction. By construction Eq. (22) we have ˙ ˆ t α τ + ∥ ˙ ˆ β α τ ∥ = 1 , therefore the sequences ( ˙ ˆ t α ) α , ( ˙ ˆ β α ) α as well as ( ˆ t α ) α , ( ˆ β α ) α are uniformly bounded on [0 , T ] . The Arzelà-Ascoli theorem yields that, up to a subsequence, there exists ( ˆ t, ˆ β ) such that ( ˆ t α k , ˆ β α k ) → ( ˆ t, ˆ β ) in ( C 0 ([0 , T ] , R × R d ) , ∥·∥ ∞ ) . Since ∥ ˙ ˆ β α τ ∥ , ∥ ˙ ˆ t α τ ∥≤ 1 we have, applying the Banach-Alaoglu theorem, that up to a new subsequence

and ˙ ˆ β τ lim inf α k ˙ ˆ β α k τ 1 and thus ˙ ˆ t τ + ˙ ˆ β τ 1 :

where the third inequality is by Fatou's lemma. Note that since [0 , T ] is bounded then it also implies the weak convergence in any L p (0 , T ) , 1 ≤ p < ∞ . Since ( ˆ β α ) converges uniformly on [0 , T ] , and ∇ L is continuous, we have that ∇ L ( ˆ β α ) converges uniformly to ∇ L ( ˆ β ) . Since ˙ ˆ t α k ⇀ ˙ ˆ t in L 1 (0 , T ) , passing to the limit in the equation ∇ ˜ ϕ α ( ˆ β α τ ) = -∫ τ 0 ˙ ˆ t α s ∇ L ( ˆ β α s )d s leads to due to Lemma 2.

Recall from Eq. (27) and the definition of ˜ ϕ α that:

Hence assuming that ˆ β τ is such that ∥| ˆ β τ |⊙∇ L ( ˆ β τ ) ∦ = 0 , we can ensure that ∥| ˆ β τ ′ |⊙∇ L ( ˆ β τ ′ ) ∦ = 0 for τ ′ ∈ [ τ, τ + ε ] and ε small enough. We have then √ ˆ β α τ ′ + α 4 ⊙∇ L ( ˆ β α τ ′ ) 1 / ln(1 /α )+ ∥ √ ˆ β α τ ′ + α 4 ⊙∇ L ( ˆ β α τ ′ ) ∥ converges uniformly toward -| ˆ β τ ′ |⊙∇ L ( ˆ β τ ′ ) ∥| ˆ β τ ′ |⊙∇ L ( ˆ β τ ′ ) ∥ on [ τ, τ + ε ] . Using the dominated convergence theorem, we have ∫ τ + ε τ √ ˆ β α τ ′ + α 4 ⊙∇ L ( ˆ β α τ ′ ) 1 / log(1 /α )+ ∥ √ ˆ β α τ ′ + α 4 ⊙∇ L ( ˆ β α τ ′ ) ∥ d τ ′ → ∫ τ + ε τ | ˆ β τ ′ |⊙∇ L ( ˆ β τ ′ ) ∥| ˆ β τ ′ |⊙∇ L ( ˆ β τ ′ ) ∥ d τ ′ . We therefore obtain ˙ ˆ β τ = -| ˆ β τ |⊙∇ L ( ˆ β τ ) ∥| ˆ β τ |⊙∇ L ( ˆ β τ ) ∥ in L 1 [0 , T ] . Consequently ∥ ˙ ˆ β τ ∥ = 1 and ˙ ˆ t τ = 0 .

Proof that the loss stricly decreases along the heteroclinic orbits.

Assume ˆ β τ is such that | ˆ β τ |⊙∇ L ( ˆ β τ ) = 0 , then the flow follows

Letting γ ( τ ) = 1 ∥| ˆ β τ |⊙∇ L ( ˆ β τ ) ∥ we get:

because | ˆ β τ |⊙∇ L ( ˆ β τ ) 2 = 0 .

̸

̸

Borrowing terminologies from [18], we can distinguish two regimes: when ˙ ˆ β τ = 0 , the system is sticked to the saddle point. When ˙ ˆ t τ = 0 and ∥ ˙ ˆ β τ ∥ = 1 the system switches to a viscous slip which follows the normalised flow Eq. (26). We use the term of heteroclinic orbit as in the dynamical systems literature since in the weight space ( u, v ) it corresponds to a path with links two distinct critical points of the loss F . Since ˙ ˆ t τ = 0 , this regime happens instantly for the original t time scale ( i.e. a jump occurs).

From Proposition 8, following the same reasoning as in Section 3, we can show that the rescaled process converges uniformly to a continuous saddle-to-saddle process where the saddles are linked by normalized flows.

Theorem 3. Let T > 0 . For all subsequences defined in Proposition 8, there exist times 0 = τ ′ 0 < τ 1 < τ ′ 1 < · · · < τ p < τ ′ p < τ p +1 = + ∞ such that the the iterates ( ˆ β α k τ ) τ converge uniformly on [0 , T ] to the following limit trajectory :

where the saddles ( β 0 = 0 , β 1 , . . . , β p = β ⋆ ℓ 1 ) are constructed in Algorithm 1. Also, the loss ( L ( ˆ β τ )) τ is constant on the saddles and strictly decreasing on the orbits. Finally, independently of the chosen subsequence, for k ∈ [ p ] we have ˆ t τ k = ˆ t τ ′ k = t k where the times ( t k ) k ∈ [ p ] are defined through Algorithm 1.

Proof. Some parts of the proof are slightly technical. To simplify the understanding, we make use of auxiliary lemmas which are stated in Appendix F. The overall spirit follows the intuitive ideas given in Section 3 and relies on showing that Eq. (25) can only be satisfied if the iterates visit the saddles from Algorithm 1.

We let ˆ s τ := -∫ τ 0 ˙ ˆ t s ∇ L ( ˆ β s )d s , which is continuous and satisfies ˆ s τ ∈ ∂ ∥ ˆ β τ ∥ 1 from Eq. (25). Let S = { β ∈ R d , | β |⊙∇ L ( β ) = 0 } denote the set of critical points and let ( β k , t k , s k ) be the successive values of ( β, t, s ) which appear in the loops of Algorithm 1.

We do a proof by induction: we start by assuming that the iterates are stuck at the saddle β k -1 at time τ ≥ τ ′ k -1 where ˆ t τ ′ k -1 = t k -1 and ˆ s τ ′ k -1 = s k -1 (recurrence hypothesis), we then show that they can only move at a time τ k and follow the normalised flow Eq. (26). We finally show that they must end up 'stuck' at the new critical point β k , validating the recurrence hypothesis.

Proof of the jump time τ k such that ˆ t τ k = t k : we set ourselves at time τ ≥ τ ′ k -1 , stuck at the saddle β k -1 . Let τ k := sup { τ, ˆ t τ ≤ t k } , we have that τ k < ∞ from Lemma 3. Note that by continuity of ˆ t τ it holds that ˆ t τ k = t k . Now notice that ˆ s τ = ˆ s τ ′ k -1 -( ˆ t τ -ˆ t τ ′ k -1 ) ∇ L ( β k -1 ) = s k -1 -( ˆ t τ -t k -1 ) ∇ L ( β k -1 ) . We argue that for any ε > 0 , we cannot have ˆ β τ = β k -1 on ( τ k , τ k + ε ) . Indeed by the definition of τ k and from the algorithmic construction of time t k , it would lead to | ˆ s τ ( i ) | > 1 for some coordinate i ∈ [ d ] , which contradicts Eq. (25). Therefore the iterates must move at the time τ k .

̸

Heterocline leaving β k -1 for τ ∈ [ τ k , τ ′ k ] : contrary to before, our time rescaling enables to capture what happens during the 'jump'. We have shown that for any ε , there exists τ ε ∈ ( τ k , τ k + ε ) , such that ˆ β τ ε = β k -1 . From Lemma 4, since the saddles are distinct along the flow, we must have that ˆ β τ ε / ∈ S for ε small enough. The iterates therefore follow a heterocline flow leaving β k -1 with a speed of 1 given by Eq. (26). We now define τ ′ k := inf { τ > τ k , ∃ ε 0 > 0 , ∀ ε ∈ [0 , ε 0 ] , ˆ β τ + ε ∈ S } which corresponds to the time at which the iterates reach a new critical point and stay there for at least a small time ε 0 . We have just shown that τ ′ k > τ k . Now from Proposition 8, the path length of ˆ β is finite, and from Lemma 4 the flow visits a finite number of distinct saddles at a speed of 1 . These two arguments put together, we get that τ ′ k < + ∞ and also ˆ β τ ′ k + ε = ˆ β τ ′ k , ∀ ε ∈ [0 , ε 0 ] . On another note, since ˙ ˆ t τ = 0 for τ ∈ [ τ k , τ ′ k ] we have ˆ t τ ′ k = ˆ t τ k (= t k ) as well as ˆ s τ k = ˆ s τ ′ k (= s k ) .

Proof of the landing point β k : we now want to find to which saddle ˆ β τ ′ k ∈ S the iterates have moved to. To that end, we consider the following sets which also appear in Algorithm 1:

The set I k corresponds to the coordinates of ˆ β τ ′ k which 'are allowed' (but not obliged) to be activated ( i.e. non-zero). For τ ∈ [ τ ′ k , τ ′ k + ε 0 ] we have that ˆ s τ = ˆ s τ ′ k -( ˆ t τ -t k ) ∇ L ( ˆ β τ ′ k ) . By continuity of ˆ s and the fact that ˆ s τ ∈ ∂ ∥ ˆ β τ ′ k ∥ 1 , the equality translates into:

if i ∈ I + ,k , then [ ∇ L ( ˆ β τ ′ k )] i ≥ 0 and ˆ β τ ′ k ( i ) ≥ 0

if i ∈ I -,k , then [ ∇ L ( ˆ β τ ′ k )] i ≤ 0 and ˆ β τ ′ k ( i ) ≤ 0

for i ∈ I k , if ˆ β τ ′ k ( i ) = 0 , then [ ∇ L ( ˆ β τ ′ k )] i = 0

̸

One can then notice that these conditions exactly correspond to the optimality conditions of the following constrained minimisation problem:

We showed in Proposition 7 that the solution to this problem is unique and equal to β k from Algorithm 1. Therefore ˆ β τ = β k for τ ∈ [ τ ′ k , τ ′ k + ε 0 ] . It finally remains to show that ˆ β τ = β k while τ ≤ τ k +1 , where τ k +1 := sup { τ, ˆ t τ = t k +1 } . For this let τ ∈ [ τ ′ k , τ k +1 ] , notice that for i / ∈ I k , we necessarily have that ˆ β τ ( i ) = β k ( i ) = 0 , otherwise we break the continuity of ˆ s τ . Similarly, for i ∈ I k, + , we necessarily have that ˆ β τ ( i ) ≥ 0 and for i ∈ I k, -, ˆ β τ ( i ) ≤ 0 for the same continuity reasons. Now assume that ˆ β τ ( I k ) = β k ( I k ) . Then from Lemma 4 and continuity of the flow, ∃ τ ′ ∈ ( τ ′ k , τ ) such that ˆ β τ ′ / ∈ S and there must exist a heterocline flow Eq. (26) starting from β k which passes through β τ ′ . This is absurd since along this flow the loss strictly decreases, which is in contradiction with the definition of β k which minimises the problem Eq. (31).

̸

E.1 Proof of Theorem 2

Theorem 3 enables to prove without difficulty Theorem 2 which we recall below. Indeed we can show that any extracted limit ˆ β maps back to the unique discontinuous process ˜ β ◦ .

Theorem 2. Let the saddles ( β 0 = 0 , β 1 , . . . , β p -1 , β p = β ⋆ ℓ 1 ) and jump times ( t 0 = 0 , t 1 , . . . , t p ) be the outputs of Algorithm 1 and let ( ˜ β ◦ t ) t be the piecewise constant process defined as follows:

The accelerated flow ( ˜ β α t ) t defined in Eq. (9) uniformly converges towards the limiting process ( ˜ β ◦ t ) t on any compact subset of R ≥ 0 \{ t 1 , . . . , t p } .

Proof. We directly apply Theorem 3, let α k be the subsequence from the theorem. Let ε > 0 , for simplicity we prove the result on [ t 1 + ε, t 2 -ε ] , all the other compacts easily follow the same line of proof. Note that since ˆ t α k ( τ ′ 1 ) → t 1 and ˆ t α k ( τ 2 ) → t 2 , for α k small enough ˆ t α k ( τ ′ 1 ) ≤ t 1 + ε and ˆ t α k ( τ 2 ) ≥ t 2 -ε , by the monotonicity of τ α k , this means that for α k small enough, τ ′ 1 ≤ τ α k ( t 1 + ε ) and τ 2 ≥ τ α k ( t 2 -ε ) . Therefore

which goes uniformly to 0 following Theorem 3. Since this result is independent of the subsequence α k , we get the result of Theorem 2.

E.2 Proof of Proposition 3

We restate and prove Proposition 3 below.

Proposition 3. For all T > t p , the graph of the iterates ( ˜ β α t ) t ≤ T converges to that of ( ˆ β τ ) τ :

Proof. For α small enough, we have that ˆ t α τ ′ p ≤ t p + ε ≤ T

according to Theorem 3.

Similarly:

according to Theorem 3, which concludes the proof.

F Technical lemmas

The following lemma describes the behaviour of ∇ ˜ ϕ α ( β α ) as α → 0 in function of the subdifferential ∂ ∥·∥ 1 .

if β i > 0 then [ ∇ ˜ ϕ α ( β α )] i converges to 1

if β i < 0 then [ ∇ ˜ ϕ α ( β α )] i converges to -1 .

Moreover if we assume that ∇ ˜ ϕ α ( β α ) converges to η ∈ R d , we have that:

β i = 0 ⇒ η i ∈ [ -1 , 1] .

Overall, assuming that ( β α , ∇ ˜ ϕ α ( β α )) -→ α → 0 ( β, η ) , we can write:

Now assume that β α i → β i > 0 , then [ ∇ ˜ ϕ α ( β α )] i → 1 , if β i < 0 we conclude using that arcsinh is an odd function. All the claims are simple consequences of this.

The following lemma shows that the extracted limits ˆ t as defined in Proposition 8 diverge to ∞ . This divergence is crucial as it implies that the rescaled iterates ( ˆ β τ ) τ explore the whole trajectory..

Lemma 3. For any extracted limit ˆ t as defined in Proposition 8, we have that τ -C ≤ ˆ t τ where C is the upperbound on the length of the curves defined in proposition 6.

Proof. Recall that

Proof. We have that

From Proposition 6, the full path length ∫ + ∞ 0 ∥ ˙ β α s ∥ d s is finite and bounded by some constant C independently of α . Therefore τ α is a bijection in R ≥ 0 and we defined ˆ t α τ = ( τ α ) -1 ( τ ) . Furthermore τ α ( t ) ≤ t + C leads to t ≤ ˆ t α ( t + C ) and therefore τ -C ≤ ˆ t α ( τ ) for all τ ≥ 0 . This inequality still holds for any converging subsequence, which proves the result.

Under a mild additional assumption on the data (see Assumption 2), we showed after the proof of Proposition 1 in Appendix B that the number of saddles of F is finite. Without this assumption, the number of saddles is a priori not finite. However the following lemma shows that along the flow of ˆ β the number of saddles which can potentially be visited is indeed finite.

Lemma 4. The limiting flow ˆ β as defined in Proposition 8 can only visit a finite number of critical points β ∈ S := { β ∈ R d , β ⊙∇ L ( β ) = 0 } and can visit each one of them at most once.

Proof. Let τ ≥ 0 , and assume that ˆ β τ ∈ S , i.e., we are at a critical point at time τ . From Proposition 1, we have that

Let us define the sets

The set I corresponds to the coordinate of ˆ β τ which 'are allowed' (but not obliged) to be non-zero since from Eq. (25), supp( ˆ β τ ) ⊂ I . Now given the fact that the sub-matrix X I = (˜ x i ) i ∈ I ∈ R n × card( I ) is full rank (see part (1) of the proof of Proposition 7 for the explanation), the solution of the minimisation problem (32) is unique and equal to β [ ξ ] = ( X ⊤ ξ X ξ ) -1 X ⊤ ξ y and β [ ξ C ] = 0 where ξ = supp( ˆ β τ ) . There are 2 d = Card( P ([ d ])) (where P ([ d ]) contains all the subsets of [ d ] ) number of constraints of the form { β i = 0 , i / ∈ A} , where A ⊂ [ d ] , and ˆ β τ is the unique solution of one of them. ˆ β τ can therefore take at most 2 d values (very crude upperbound). There is therefore a finite number of critical points which can be reached by the flow ˆ β . Furthermore, from Proposition 8, the loss is strictly decreasing along the heteroclinic orbits, each of these critical points can therefore be visited at most once.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper characterizes the trajectory of gradient flow on 2-layer diagonal linear networks for linear regression tasks. Specifically, the paper considers the model parameterized as $x \\mapsto \\langle u \\odot w, x \\rangle$ in the linear regression setting. By interpreting the gradient flow on the nonconvex loss as a mirror flow, the authors show, in the limit of vanishing initialization, that the gradient flow dynamics jump between various saddles before converging to the minimum $\\ell_1$ norm solution. The paper characterizes the exact times of these jumps (after appropriate rescaling based on the initialization size $\\alpha$ in the $\\alpha \\rightarrow 0$ limit), as well as the location of these saddles; these times and locations can be computed via algorithm 1. As a consequence, the paper shows that each saddle is a solution to a constrained minimization problem where some subset of coordinates are fixed 0. Under a RIP assumption on the data, the paper shows that new coordinates are added sequentially at each saddle, thus demonstrating incremental learning.\n\nSTRENGTHS:\n- The paper appears to be technically sound, and is well-written and easy to understand.\n- The quadratically parameterized regression problem has been well studied in prior works (Woodworth et al. 2020, Azulay et al. 2021, and others), which show that the gradient flow can be characterized by mirror descent and converges to the minimum $\\ell_1$ solution in the $\\alpha \\rightarrow 0$ limit. However, the current paper exactly characterizes the limit of the gradient flow trajectory under minimal data assumptions, which is a novel analysis. The main technical novelty here is proving that as $\\alpha \\rightarrow 0$ the trajectory converges to a piecewise constant process. \n- While many prior works on implicit regularization only focus on properties of the solution at convergence, the current paper can describe the saddle points along the trajectory. I find it to be a significant contribution that the successive saddles which can be visited by gradient flow can be characterized (by Algorithm 1) explicitly. In particular, the observation that coordinates can be deactivated has not appeared in prior work and is quite interesting.\n- Finally, I find this paper to be of moderate significance, as the quadratically parameterized linear regression setting is a common toy problem to understand the implicit regularization effect of gradient descent more generally.\n\nWEAKNESSES:\nThere are a couple (minor) weaknesses of the current theory.\n- The analysis is limited to the quadratically parameterized regression setting, which while despite possessing rich implicit regularization behavior is still far from more practical settings in which implicit regularization occurs.\n- One weakness of the current paper is that the jump times and saddle locations are only implicitly defined via Algorithm 1. The paper does show that under an RIP assumption that the jump times are $1/\\beta^*_s$ and the saddles correspond to incrementally learning more coordinates. However, it is difficult to interpret the intermediate iterates of Algorithm 1 more generally, which reduces the impact of this paper.\n- Another weakness is that the entirety of the analysis is done in the $\\alpha \\rightarrow 0$ limit, rather than at some small (but finite) initialization $\\alpha$.\n\nMinor typos:\n- line 133 “diferred” → “deferred”\n- line 318 “independant” → independent”\n\nQUESTIONS:\n- In general, what can one say about successive saddles? It would be interesting to understand more fine grained properties about the sequence of saddles, such as how many coordinates can change at each step and under what conditions coordinates can get deactivated. I find it unlikely that Algorithm 1 would loop through all possible subsets of activated coordinates, and instead find it more likely that coordinates would be deactivated less frequently. I thus think the paper would benefit from additional investigation (either theoretically or empirically) into how the set of activated coordinates changes between saddles.\n- What can be said in the case when $\\alpha$ is small but isn’t taken to 0? Can one say something quantitatively about whether saddle-to-saddle dynamics occur, or the rate at which the trajectory limits to the piecewise constant trajectory $\\tilde \\beta^\\circ$?\n\nLIMITATIONS:\nLimitations/Broader Impact are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work concerns the behavior of gradient flow of 2-layer diagonal linear neural networks when the initialization scale goes to zero. The authors show that this limiting behavior is governed by a piecewise constant trajectory consisting of jumps from saddle to saddle.\n\nSTRENGTHS:\nThe incremental learning and saddle-to-saddle phenomenon is interesting. Although the model considered in this paper is equivalent to linear models, the dynamics that the authors uncover is nonetheless nontrivial and of broader interest. The writing quality is excellent and the technical material is presented in an intuitive to understand way.\n\nWEAKNESSES:\nIs the final point $\\beta_p$ in Theorem 2 the minimum $\\ell_1$ solution to $L$? Theorem 2 does not explicitly state it.\n\nMuch of the later analysis is regarding the $\\beta$'s. But the model is stated in terms of the $u$'s and $v$'s. Can the authors comment on the behavior of the $u$'s and $v$'s? (Or at least point me to where this was discussed, in case I missed it).\n\nline 40 \"diagonal linear networks which are simplified neural networks that have received significant attention lately...\". Could the authors provide a slightly expanded discussion of what these prior work did? Since this paper is specifically about diagonal linear networks, this part of the related work I believe is especially important.\n\nQUESTIONS:\nWhat is the significance of the connection to the Homotopy algorithm? I'm not familiar with it and would appreciate if the authors can explain a bit more why this connection matters.\n\nIf the authors can address this and the weaknesses above, I'll be happy to raise the score to a 7.\n\nLIMITATIONS:\nI do not believe the authors addressed the limitations, at least not in an explicit way. I think the paper would be greatly improved if it can touch upon the comment brought up on line 78 that the activation function is the identity. The authors make it very clear that the work is about linear networks, as many other prior works have done. I feel like if the authors can provide some form of experimental results/explanation of whether this phenomenon appear (or does not appear) where $\\sigma$ is nonlinear would be greatly beneficial. That said, I think the paper's technical contribution outweighs this.\n\nIf the authors can address this and all of the above, I'll be happy to raise the score to an 8.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper analyzes a two-layer diagonal linear network, which is a linear regression model where the linear weight is parameterized as the point-wise product of two weight vectors. It is shown that, with vanishing initialization, gradient flow will jump between saddle points of the training loss and eventually reach the $\ell_1$-minimum-norm solution. This paper further provides an algorithm to calculate all the saddle points.

STRENGTHS:
It is very important to understand the training dynamics of neural networks, and this paper analyzes a novel and interesting saddle-to-saddle phenomenon. Moreover, Algorithm 1 in this paper can compute all the jump times and saddle points, giving a complete characterization of the gradient flow trajectory. The writing is also clean, with illustrative examples and graphs.

WEAKNESSES:
The main weakness is that the two-layer diagonal linear network model (specifically, the point-wise product of two weight vectors) is too simple. If there is some evidence that some of form of the saddle-to-saddle phenomenon also happens in practical networks, this paper will be more convincing.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
In this paper, the authors studied the training dynamics of gradient flow that minimizes mean-square loss with 2-layer diagonal linear networks and data in general position. Under the small initialization (initialization scale goes to 0), the authors showed that the limiting dynamics follows a saddle-to-saddle dynamics (jump from one saddle point to another saddle point) until reach the min-$\ell_1$-norm interpolator. They gave an algorithm that can compute the visited saddle points. The results generalized the previous works on incremental learning. Experiments are also provided to verify the results.

STRENGTHS:
1.	The paper is clearly written and easy-to-follow. The proof sketch and example are given to help the readers to understand the proof easier.
2.	Understanding the training dynamics of neural networks/nonlinear models is an interesting and important problem.
3.	This paper gives a precise characterization of the training dynamics that jumps between saddle points. This recovers and goes beyond some of the previous works on the incremental learning for linear diagonal networks.
4.	The technique of reparametrized time to “accelerate” time seems to be interesting and might be of independent interest.

WEAKNESSES:
1.	The current paper focuses on the linear diagonal linear networks. It would be interesting to see if such analysis could be generalized to other more complicated problems.

QUESTIONS:
1.	In Theorem 2, it seems that the final solution is min-$\ell_1$-norm interpolator, so I think that means we are implicitly assuming the input dimension $d$ is at least the number of samples $n$?

LIMITATIONS:
The limitation is discussed in the paper. This is a theoretical work and therefore does not seem to have negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the saddle-to-saddle dynamics in Diagonal Linear Networks.\nThe authors present solid theoretical understanding. \nThey show that over 2-layer diagonal linear network, gradient flow starting with vanishing initialization visits then jump from saddles  jumps from a saddle of the training loss to another until reaching the minimum $\\ell_1$-norm solution.\n\nSTRENGTHS:\nThe paper is clear and well-written. \nUnderstanding the training dynamics of gradient descent over neural networks is a significant theoretical issue. \nParticularly, the phenomenon of saddle-to-saddle during neural network training remains mysterious, and this paper makes a valuable contribution by offering a solid theoretical analysis for 2-layer diagonal linear networks.\nThe proof presented in the paper incorporates innovative techniques, including mirror flow and time-reparametrization.\n\nWEAKNESSES:\nThe article's significance may be constrained by the focus on 2-layer diagonal linear networks, which have limited representation abilities and are no better than linear models. \nHowever, this limitation is not a major concern since the problem itself is non-convex, even in this simplified scenario.\n\nQUESTIONS:\n1. I would like to know how much the vanishing initialization would affect the saddle-to-saddle phenomenon. Will saddle-to-saddle dynamics also occur if a practical initialization is used?\n\n2. As shown in the theory and Figure 2, the recovery of coordinates is sequential, which seems to be similar to the Coordinate Descent algorithm. In this setting of 2-layer diagonal linear networks, are GD and CD inherent related?\n\nLIMITATIONS:\nThe analysis in this study focus on 2-layer diagonal linear networks, which have limited representation abilities and exhibit some special properties such as Prop 1.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}