Title: Implicit Bias of Mirror Flow on Separable Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Implicit bias

Mirror descent

Classification
Abstract: We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised ‘at infinity’ and have many possible solutions; we study which solution is preferred by the algorithm depending on the mirror potential. For exponential tailed losses and under mild assumptions on the potential, we show that the iterates converge in direction towards a $\phi_\infty$-maximum margin classifier. The function $\phi_\infty$ is the horizon function of the mirror potential and characterises its shape ‘at infinity’. When the potential is separable, a simple formula allows to compute this function. We analyse several examples of potentials and provide numerical experiments highlighting our results.

Implicit Bias of Mirror Flow on Separable Data

Abstract

We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised 'at infinity' and have many possible solutions; we study which solution is preferred by the algorithm depending on the mirror potential. For exponential tailed losses and under mild assumptions on the potential, we show that the iterates converge in direction towards a ϕ ∞ -maximum margin classifier. The function ϕ ∞ is the horizon function of the mirror potential and characterises its shape 'at infinity'. When the potential is separable, a simple formula allows to compute this function. We analyse several examples of potentials and provide numerical experiments highlighting our results.

1 Introduction

Heavily over-parametrised yet barely regularised neural networks can easily perfectly fit a noisy training set while still performing very well on unseen data [Zhang et al., 2017]. This statistical phenomenon is surprising since it is known that there exists interpolating solutions which have terrible generalisation performances [Liu et al., 2020]. To understand this benign overfitting, it is essential to take into account the training algorithm. If overfitting is indeed harmless, it must be because the optimisation process has steered us towards a solution with favorable generalisation properties.

From this simple observation, a major line of work studying the implicit regularisation of gradient methods has emerged. These results show that the recovered solution enjoys some type of low norm property in the infinite space of zero-loss solutions. Gradient descent (and its variations) has therefore been analysed in various settings, the simplest and most emblematic being that of gradient descent for least-squares regression: it converges towards the solution which has the lowest ℓ 2 distance from the initialisation [Lemaire, 1996]. In the classification setting with linearly separable data, iterates of gradient methods must diverge to infinity to minimise the loss. Therefore, the directional convergence of the iterates is considered and Soudry et al. [2018] show that gradient descent selects the ℓ 2 -max-margin solution amongst all classifiers.

Going beyond linear settings, it has been observed that an underlying mirror-descent structure very recurrently emerges when analysing gradient descent in a range of non-linear parametrisations [Woodworth et al., 2020, Azulay et al., 2021]. Providing convergence and implicit regularisation results for mirror descent has therefore gained significant importance.

In this context, for linear regression, Gunasekar et al. [2018] show that the iterates converge to the solution that has minimal Bregman distance to the initial point. Turning towards the classification setting, an apparent gap emerges as there is still no clear understanding of what happens: Can directional convergence be characterised in terms of a max-margin problem? If so, what is the associated norm? Quite surprisingly, this question remains largely unanswered, as it is only understood for L -homogeneous potentials [Sun et al., 2023]. Our paper bridges this gap by formally characterising the implicit bias of mirror descent for separable classification problems.

<latexi sh

1_b

64="o

C

rKk

Nf

d

WJ

H

0

>A

B7n

c

VDLS

EOy

U

9

P

M

v

wT

Z

I

m

qF

Qj

8

/

R

G

3

z

p

5

X

+

<latexi sh

1_b

64="9

2

E

jG

uK

pYSM+m

JWro

>A

B7n

c

V

N

8

3

X

O

Q

PR

g

D

0

w

/

vH

T

5

Z

f

z

d

q

L

U

C

y

F

I

k

<latexi sh

1_b

64="

qNF

wSk+W

J

D

A

Y

TL8

G

>

C

n

cdV

gM

3

r

0E

j

u

K

oy

Z

Q

9

v

X

z

2

p

IHB

5/

m

U

R

f

O

7

P

<latexi sh

1_b

64="

yNgfSO0z3R

7XU

/vV5c

p

>A

C

d

FB

H

Zq9

j

D

K

G2T

QW

+

Y

nJ

m

8

k

w

E

I

r

P

M

L

o

u

<latexi sh

1_b

64="gfD

G5HdC2uj

9mXU+

3

ABwE

>

c

V

NS8

J

W

O

I

0

r

F

v

p

7

YT

PQ

/

k

z

n

L

q

R

Z

M

y

<latexi sh

1_b

64="w3BVAYK2/O

U

u

EC50

>

cd

DLSgN

Jz

G

Mr

k

v

I

7

o

8

Q

ZX

y

Pp

F

+

W

R

H

T

nq

m

<latexi sh

1_b

64="7OSw2N89Xo

P5ZD

V

+

qgk

>A

C

3

cd

L

BEJz

G

Mr

0v

HQ

F

U

IY

R

m

j

Wf

K

u

y

T

p

n

/

<latexi sh

1_b

64="5g

w

UWNq

BMZjpE

2K8zXT

>A

CJ3

cdVDLS

F

rP

km

IQ

u

G

f

n

0

O

H

7

R

/o

9

y

+

<latexi sh

1_b

64="Fc

D8zC

v

2IWM

OU

w

>A

B

H

V

NS

J3

X

9

j

7

y

m

Z

/Q

k

Y

n

d

+

Gq

r

g

R

5

u

f

<latexi sh

1_b

64="2A7

5

w

9k

0W

M

3v

8I

g

>

C

X

cdVBNS

J

F

n

Zf pu

z

E

H

R

U

moy

/

+

L

T

r

PD

q

j

Y

G

O

Q

K

<latexi sh

1_b

64="RI+

g

8

Kn d/

9

rGN

UVO

c

>A

B

3

SwM

EJ2

X7

T

C

k

Wz0

v

YD

Z

F

q

m

L

H

p

5Q

y

j

f

u

o

P

<latexi sh

1_b

64="

BIMZ5vq7

UD

nuP

Sz

8O

G

>A

X

cdV

L

g

F

3

2

k

w

K

Yj

N

r

f0

E

yJ

H

/

T

R

9

m

W

Q

C

o

p

+

-

{

¯

β

∈

R

2

: min

i

y

i

〈

x

i

,

¯

β

〉

= 1

}

Iterations

10

-

2

1

10

5

y

= +1

y

=

1

¯

β

∞

Limiting directions

- max-margins

φ

∞

t

φ

∞

∝ ‖ · ‖

2

y

i

x

i

10

-

6

Train losses

(Log log scale)

Mirror descent with:

φ

MD

1

(

β

) =

∑

i

=1

,

2

cosh(

β

i

)

φ

(

β

) =

‖

β

‖

2

2

φ

MD

2

(

β

) =

∑

i

=1

,

2

β

i

arcsinh(

β

i

)

-

√

β

2

i

+1

L

(

β

t

)

L

(

β

MD

1

t

)

L

(

β

MD

2

t

)

¯

β

MD

2

∞

¯

β

MD

1

∞

φ

MD

2

∞

∝ ‖ · ‖

1

φ

MD

1

∞

∝ ‖ · ‖

∞

1.1 Informal statement of the main result

For a separable dataset ( x i , y i ) i ∈ [ n ] , we study the mirror flow d ∇ ϕ ( β t ) = -∇ L ( β t )d t with potential ϕ : R d → R and an exponential tailed classification loss L . We prove that β t converges in direction towards the solution of the ϕ ∞ -maximum margin solution where the (asymmetric) norm ϕ ∞ captures the shape of the potential ϕ 'at infinity' (see Figure 2 for an intuitive illustration).

Theorem 1 (Main result, Informal) . There exists a horizon function ϕ ∞ such that for any separable dataset, the normalised mirror flow iterates ¯ β t := β t / ∥ β t ∥ converge and satisfy:

Our result holds for a large class of potentials ϕ and recovers previous results obtained for ϕ = ∥ · ∥ p p [Sun et al., 2022] and for L -homogeneous potentials [Sun et al., 2023]. For general potentials, showing convergence towards a maximum margin classifier is much harder because, in stark contrast with homogeneous potentials, ϕ 's geometry changes as the iterates diverge. To capture the behaviour of ϕ at infinity, we geometrically construct its horizon function ϕ ∞ . By considering ϕ 's successive level sets (and re-normalising them to prevent blow up), we show that under mild assumptions, these sets asymptotically converge towards a limiting horizon set S ∞ . The horizon function ϕ ∞ is then simply the asymmetric norm which has S ∞ as its unit ball (see Figure 3 for an illustration). In addition, when the function ϕ is 'separable' and can be written ϕ ( β ) = ∑ i φ ( β i ) for a real valued function φ , then a very simple and explicit formula enables to calculate ϕ ∞ (Theorem 3).

The paper is organised as follows. The classification setting as well as the assumptions on the loss and the potential are provided in Section 2. The proof sketch and an intuitive construction of the horizon function are given in Section 3. In Section 4, we state the formal definition and results. Simple examples of horizon potentials and numerical experiments supporting our claims are finally given in Section 5.

1.2 Relevance of mirror descent and related work

We first outline the motivations for understanding the implicit regularization of mirror descent and discuss related works that contextualize our contribution within the machine learning context.

Relevance of studying mirror descent in the context of machine learning. Though mirror descent is not per se an algorithm used by machine learning practitioners, it proves to be a very useful tool for theoreticians in the field. Indeed, when analysing gradient descent (and its stochastic and accelerated

<latexi sh

1_b

64="WHZu/rQp

K

vn8

7jVq5

o

>A

C

3

d

BNS

E

f2

y

RPJ

9

m

kX

wO

F

L

0

+g

U

M

Y

D

z

G

T

<latexi sh

1_b

64="8z

0d

KW3k

B5n9Q

+

g

E

I

>A

C

X

c

V

NS

Ur q/

L

T2up

j

m

G

Y

O

Fv yJ

f

D

P

w

Z

H

o

R

M

7

<latexi sh

1_b

64="Qr

5K0CvBj

UZROP

MWX/nNw

g

>A

3

cdV

S8

E

f2q

y

m+

H

I

L

u

D

7Y

Tk

J

zG

9

<latexi sh

1_b

64="5DE3+M

Xn9WJV

f

u

gkmwSQ

>A

B7

cd

L

F

U

2vq

I

p

GK

o

0

Z

C

P

/

Yj

O

z

y

r

H

8

R

T

N

<latexi sh

1_b

64="GEpTJ

5kY

m

u

zVf7cPSwC

>A

X

d

BN

F

Q

K

j

2I

Z

DL

U

g/

r

W

3

M

n8

o

+

q

R

vy

9

O

H

0

<latexi sh

1_b

64="oQpvWym

A

Y

5

M

JuX

Dk

>

CP

c

VBN

E

8

7

TG

H

LZ

S

/

O

2

F

+wf

I

n

0

K

9q

r

3d

R

j

z

<latexi sh

1_b

64="5

qc

+

r2ZwGJW

8M

KFNQ

>A

CHn dVB

u

T

k

E

7

/

m

D

L

PS

Y

zy

3

po

I

g

v

R

9

O

X

f

U

j

0

<latexi sh

1_b

64="C

TO

vY/H

ZWVzqmn2

k

+7

>A

I

cd

DL

FPXwK

Q8

0

R

5

E

N

9p

S

f

o

3

g

U

B

y

J

G

M

r

ju

<latexi sh

1_b

64="3

NrMEn

/w+VU

B

mK

5F

IG0

>A

8

c

S

q

9L

C

z

W

Q

Z

2

O

k

R

H

v

p

df

u

X

7

o

J

g

T

D

Y

P

j

y

<latexi sh

1_b

64="W

kX

f

JREjP

Y

w

>A

B7n cdVDLSgN

O

r

9

Z

M

F

m

H

v

8

GC

qK

+

u

3

Q

z

T

0

variants) on neural-network architectures, an underlying mirror-descent structure very recurrently emerges. Then, results for mirror descent enable to prove convergence as well as characterise the implicit bias of gradient descent for these architectures. Diagonal linear networks, which are ideal proxy models for gaining insights on complex deep-learning phenomenons, is the most notable example of such an architecture. The hyperbolic entropy potential naturally appears and enables to prove countless results: implicit bias of gradient descent in regression [Woodworth et al., 2020, Vaskevicius et al., 2019] and in classification [Moroshko et al., 2020], effect of stochasticity [Pesme et al., 2021] and momentum [Papazov et al., 2024], convergence of gradient descent and effect of the step-size [Even et al., 2023], saddle-to-saddle dynamics [Pesme and Flammarion, 2023]. Unveiling an underlying mirror-like structure goes beyond these simple networks as they also appear in: matrix factorisation with commuting observations [Gunasekar et al., 2017, Wu and Rebeschini, 2021], fully connected linear networks [Azulay et al., 2021, Varre et al., 2023] and 2 -layer ReLU networks [Chizat and Bach, 2020]. Building on these examples, Li et al. [2022] investigate the formal conditions that ensure the existence of a mirror flow reformulation for general parametrisations, extending previous results by Amid and Warmuth [2020a,b].

Gradient descent in classification. Numerous works have studied gradient descent in the classification setting. For linear parametrisations, separable data and exponentially tailed losses, Soudry et al. [2018] prove that GD converges in direction towards the ℓ 2 -maximum margin classifier and provides convergence rates. A very fine description of this divergence trajectory is conducted by Ji and Telgarsky [2018] and a different primal-dual analysis leading to tighter rates is given by Ji and Telgarsky [2021]. Similar results are proven for stochastic gradient descent by Nacson et al. [2019c]. In the case of general loss tails, Ji et al. [2020] prove that gradient descent asymptotically follows the ℓ 2 -norm regularisation path. A whole 'astral theory' is developed by Dudík et al. [2022] who provide a framework which enables to handle 'minimisation at infinity'. Beyond the linear case, Lyu and Li [2020] proves for homogeneous neural networks that any directional limit point of gradient descent is along a KKT point of the ℓ 2 -max margin problem. A weaker version of this result was previously obtained by Nacson et al. [2019a]. Furthermore, convergence results for linear networks are provided by Yun et al. [2021]. Finally, for 2 -layer networks in the infinite width limit, assuming directional convergence, Chizat and Bach [2020] proves that the limit can be characterised as a max-margin classifier in a certain space of functions.

1.3 Notations

We provide here a few notations which will be useful throughout the paper. We let [ n ] be the integers from 1 to n . We denote by Z ∈ R n × d the feature matrix whose i th line corresponds to the vector y i x i . When not specified, ∥ · ∥ corresponds to any (definable) norm on R d . For a convex function h , ∂h ( β ) denotes its subdifferential at β : ∂h ( β ) = { g ∈ R d : h ( β ′ ) ≥ h ( β ) + ⟨ g, β ′ -β ⟩ , ∀ β ′ ∈ R d } . For any scalar function f : R → R and vector u ∈ R p , the vector f ( u ) ∈ R p corresponds to the component-wise application of f over u . We denote by σ : R n → R n the softmax function equal to σ ( z ) = exp( z ) / ∑ n i =1 exp( z i ) ∈ ∆ n where ∆ n is the unit simplex. For a convex potential ϕ , we denote D ϕ ( β, β 0 ) the Bregman divergence equal to ϕ ( β ) -( ϕ ( β 0 ) + ⟨∇ ϕ ( β 0 ) , β -β 0 ⟩ ) ≥ 0 .

2 Problem set-up

We consider a dataset ( x i , y i ) 1 ≤ i ≤ n with points x i ∈ R d and binary labels y i ∈ {-1 , 1 } . We choose a loss function ℓ : R → R and seek to minimise the empirical risk

We propose to study the dynamics of mirror flow, which is the continuous-time limit of the mirror descent algorithm [Beck and Teboulle, 2003]. Mirror descent is a generalisation of gradient descent to non-Euclidean geometries induced by a given convex potential function ϕ : R d → R . The method generates a sequence ( ˆ β k ) k ≥ 0 with ˆ β 0 = β 0 ∈ R d and

When the step size γ goes to 0, the mirror descent iterates approach the solution ( β t ) t ≥ 0 to the following differential equation:

initialised at β 0 . Studying the mirror flow (MF) leads to simpler computations than its discrete counterpart, and still allows to obtain rich insights about the algorithm's behaviour.

We now state our standing assumptions on the loss function ℓ and potential ϕ .

Assumption 1. The loss ℓ satisfies:

ℓ is convex, twice continuously differentiable, decreasing and lim z → + ∞ ℓ ( z ) = 0 .

ℓ has an exponential tail, in the sense that ℓ ( z ) ∼ z →∞ -ℓ ′ ( z ) ∼ z →∞ exp( -z ) .

The first part of the assumptions is very general and ensures that the empirical loss L can be minimised 'at infinity'. The exponential tail is crucial: it enables to identify a unique maximum margin solution towards which the iterates converge in direction, independently of the considered loss. Both the exponential ℓ ( z ) = exp( -z ) and the logistic loss ℓ ( z ) = ln(1 + exp( -z )) satisfy the conditions. On the other hand, losses with polynomial tails do not satisfy the second criterion. Similar assumptions on the tail appear when investigating the implicit bias of gradient descent for separable data [Soudry et al., 2018, Nacson et al., 2019b, Ji et al., 2020, Ji and Telgarsky, 2021, Chizat and Bach, 2020].

Assumption 2. The potential ϕ : R d → R satisfies:

ϕ is twice continuously differentiable, strictly convex and coercive.

for every c ∈ R ≥ 0 and β 2 ∈ R d , the sub-level set { β 1 ∈ R d , D ϕ ( β 2 , β 1 ) ≤ c } is bounded.

∇ 2 ϕ ( β ) is positive-definite for all β ∈ R d .

∇ ϕ diverges at infinity: lim ∥ β ∥→∞ ∥∇ ϕ ( β ) ∥ = + ∞ .

The first two points of the assumption are commonly used to ensure well-posedness of mirror descent [Bauschke et al., 2017]. The third one is necessary in continuous time to ensure the existence and uniqueness over R ≥ 0 of a solution to the (MF) differential equation (in particular, we want to avoid the solution 'blowing up in finite time' ; see Lemma 2 in Appendix A). The coercive gradient assumption is crucial for our main result and we discuss it in more depth in Section 6.

Finally, we assume that the dataset is linearly separable.

Assumption 3. There exists β ⋆ ∈ R d such that y i ⟨ β ⋆ , x i ⟩ > 0 for every i ∈ [ n ] .

Notice that such β ⋆ 's correspond to minimisation directions: L ( λβ ⋆ ) λ →∞ -→ 0 . Under the three previous assumptions, we can show that the mirror flow iterates ( β t ) t ≥ 0 minimise the loss while diverging to infinity.

Proposition 1. Considering the mirror flow ( β t ) t ≥ 0 , the loss converges towards 0 and the iterates diverge: lim t →∞ L ( β t ) = 0 and lim t →∞ ∥ β t ∥ = + ∞ .

The proof relies on classical techniques used to analyse gradient methods in continuous time and we defer the proof to Appendix A. We now turn to the main question addressed in this paper:

Among all minimising directions β ⋆ , towards which does the mirror flow converge?

We initially offer a heuristic and intuitive answer to this question, setting the stage for the formal construction of the implicit regularisation problem.

3 Intuitive construction of the implicit regularisation problem

In this section, we give an informal presentation and proof sketch of our main result. A fully rigorous exposition is then provided in Section 4.

Preliminaries. Assume here for simplicity that ℓ ( z ) = exp( -z ) . The mirror flow then writes

with q ( β t ) = σ ( -Zβ t ) , where σ is the softmax function and Z the matrix with rows ( y i x i ) i ∈ [ n ] . Note that q ( β t ) belongs to the unit simplex ∆ n .

We simplify the differential equation by performing a time rescaling, which does not change the asymptotical behaviour. As θ : t ↦→ ∫ t 0 L ( β s )d s is a bijection in R ≥ 0 (see Lemma 4), we can speed up time and consider the accelerated iterates ˜ β t = β θ -1 ( t ) . 1 By the chain rule, we have

and therefore

From now on, we drop the tilde notation and assume that a change of time scale has been done. We want to characterise the directional limit of the diverging iterates β t . To do so, we study their normalisation ¯ β t := β t ∥ β t ∥ . As they form a bounded sequence, and q ( β t ) ∈ ∆ n is also bounded, we can extract a subsequence 2 ( ¯ β t s , q ( β t s )) s ∈ N , with lim s →∞ t s = ∞ converging to some limit ( ¯ β ∞ , q ∞ ) . By the Césaro average property, 1 t s ∫ t s 0 q ( β s )d s also converges towards q ∞ . Equation (1) then yields

Observe that q ( β t ) = σ ( -Zβ t ) and the softmax function σ approaches the argmax operator at infinity. Hence, as β t diverges, we expect that q ( β t ) k → 0 for coordinates k for which ( -Zβ t ) k is not maximal, i.e. ( Zβ t ) k not minimal. This observation is made formal in the following lemma. Its proof is straightforward and is given in Appendix A.

Lemma 1. Assume that ( ¯ β t s , q ( β t s )) s →∞ -→ ( ¯ β ∞ , q ∞ ) . It holds that:

In words, coordinates of q ∞ which do not correspond to support vectors of ¯ β ∞ must be zero. Our goal is now to uniquely characterise ¯ β ∞ as the solution of a maximum margin problem.

3.1 Warm-up: gradient flow

As a warm-up, let us consider standard gradient flow, which corresponds to mirror flow with potential ϕ = ∥ · ∥ 2 2 / 2 . In this case, Equation (2) becomes β t s /t s → Z ⊤ q ∞ . Since the normalised iterates satisfy ¯ β t s → ¯ β ∞ , we get

Now notice that this equation along with the slackness conditions from Lemma 1 exactly correspond to the optimality conditions of the following convex minimisation problem:

<latexi sh

1_b

64="

A

0

8MqjVu krd

P

/J2f gO

>

B

X

c

DLS

N

E

z

G

H

Y

yKRI9

m

5

T

7

Q

v

F

w+

3

p

W

nZ

C

U

o

<latexi sh

1_b

64="mc25pIqjX

MZ

W

y

F

CHz

o

>A

dV

7SgNB

J3

G

r

L

R

u

E

w

k

D

/Y

f

+T

O

U9

n

8

Q

K

0

P

v

<latexi sh

1_b

64="qIS

H

CugKVLDY

d

89T

J

EU

>A

B/3

c

N

F

2v

O

GzW

R

k

Zm

Mp

o

Q

+

w

n

X

y

5

r

j

0

f

7

P

<latexi sh

1_b

64="YCg

ZWqR

j/

UI

Oncr

XHS

u

E

>A

B

dVDL

N

FJ3

P

ovQ

k

9

T

p

0

z

y

+

5

7

M

f

8

wG

K2

m

<latexi sh

1_b

64="Y7

0

G

p8

RZW

I

cw

>A

B

X

V

NS

M

EJ

+

T

C

uk

q

f

y

L

K

n/

v

3

O

9

m

F

r

odH

D

z

U

<latexi sh

1_b

64="

3VoRGUcT

S2z

f

YCNM

Hn0

>A

d

DL

FJ

rP

u

vQ

9

j

Z

p

Ey

g

+

X

O

5

7

W

8

w

K

m

q

<latexi sh

1_b

64="ocv

9KW

RCw0Q

HB

I

/

8A

>

dV

7SgN

FJ3

G

r

L

j

m

+

E

y

k

z

D2

Y

f

T

O

U

M

n

u

P

p

q

X

<latexi sh

1_b

64="UA

+ToX

Mw

ZP

/3mGgkq7

9W

>

B

cdVDLS

N

FJ

r

y

Cu

E

vY

z

K

j

f8

0

n

Q

H

p

5

2

I

O

R

<latexi sh

1_b

64="2WRFCdc

5M3S

K

oGwk

0L

Eg

>A

B+n

V

N

8

J

Xq

v

U

7

y

m

pZ

I/

P

j

z

Y

D

ur

O

T

H

Q

9

f

φ

(1)

(

β

) = 100

φ

(1)

(

β

) = 10

φ

(2)

(

β

) = 10

φ

(1)

(

β

) = 1

φ

(2)

(

β

) = 1

φ

(2)

(

β

) = 100

φ

(1)

φ

(2)

φ

(2)

∞

φ

(1)

∞

Furthermore, the ℓ 2 -unit ball being strictly convex, Problem (3) has a unique solution to which ¯ β ∞ must therefore be equal. Importantly, notice that Problem (3) uniquely defines the limit of any extraction on the normalised iterates ¯ β t : the normalised iterates ¯ β t must therefore converge towards the ℓ 2 -maximum margin. We recover the implicit regularisation result from Soudry et al. [2018]:

3.2 General potential: introducing the horizon function ϕ ∞

We now tackle general potentials ϕ . In the general case, the challenge of identifying the max-margin problem to which the iterates converge in direction stems from the fact that if the potential ϕ is not L -homogeneous 3 , its geometry changes as the iterates diverge. More precisely, its sub-level sets S c := { β ∈ R d , ϕ ( β ) ≤ c } change of shape as c increase, as illustrated by Figure 2 (Left).

However, we can hope that these sets have a limiting shape at infinity, meaning that the normalised sub-level sets ¯ S c := S c /R c where R c := max β ∈ S c ∥ β ∥ converge to some limiting convex set S ∞ as c →∞ . We can then construct an asymmetric norm 4 ϕ ∞ which has S ∞ as its unit ball. In words, ϕ ∞ captures the shape of ϕ at infinity. This informal construction is made rigorous in Section 4.1. We state here the crucial consequence of this construction.

Corollary 1. The horizon function ϕ ∞ is such that for any sequence β t diverging to infinity for which β t ∥ β t ∥ and ∇ ϕ ( β t ) ∥∇ ϕ ( β t ) ∥ both converge, then:

for some strictly positive factor λ .

Using this construction, we can derive the optimality conditions satisfied by ¯ β ∞ . From the convergence in Equation (2) and that of ¯ β t → ¯ β ∞ , applying Corollary 1, we obtain that:

Up to a positive multiplicative factor (which is irrelevant due to the positive homogeneity of the quantities involved), this condition along with Lemma 1 are exactly the optimality conditions of the convex problem

The limiting direction ¯ β ∞ must therefore belong to the set of its solutions. Assuming that this set contains a single element of norm 1 (we refer to the next section for comments concerning the uniqueness), we deduce that the iterates ¯ β t must converge towards it:

<latexi sh

1_b

64="jZfVO+

m

GJHDLY

9cwr

Fu

20o

>A

B

n

NS8

E

3U

q/

CR

Xkp

v

5

d

K

M

P

W

T

y

g

Q

I

<latexi sh

1_b

64="w/PzXKI

V

TBE

mY

F

>A

7n cd

DLS

N

3

W

ur

G5

Q

C

08

p

j

f

ZJ

HMv9

q2

O

y

g

R

<latexi sh

1_b

64="ZWLBn7

8f

YVF

0

k3mR+

c

>A

d

D

S

N

2pr

u

ovgK

T

X

O

y

G

J

H

M

IQ

Cj

9

5

E

P

/

w

U

<latexi sh

1_b

64="Jq

DLBGRPWK

3Sn9Yk vAF7

c

>

dV

N

2pr

f

Z

u

o

g

+

/Q

jC

T

0m

E

z

MX

y

5

8

H

U

O

I

w

<latexi sh

1_b

64="zYN8EJQ0mfyrK

/WI

cP

jHn

>A

B9

dVDLS

F

3

u

2

XR

g

C

p

Z

G

q

+T

O

7

U

v

M

k

5

w

o

<latexi sh

1_b

64="/cd yu

Km0

qPOnZ

VGow

JHXE

>A

B9

DLS

N

F

3U

2v

k

g

7Mr

T

W

R

5

I

Q

z

f

8

+

C

Y

j

p

<latexi sh

1_b

64="3cFmzq+YP

B

d

CS

kA

K

>

9H

VDJ

gN

E

2

W

OX

j

R

y

G

w

T

U

M

/

v0

n

Q

8

p

Z

5

f

IL

o

7

r

u

<latexi sh

1_b

64="

3Dc

w

ZGUOyf8

IR+

M057

o

>A

B

V

NS

EJ

W

X

K

9

j

r

/Y

2

d

C

n

z

k

gq

F

p

m

T

Q

H

<latexi sh

1_b

64="Mvw/

oWU

I

AfJyQz

K+2ck

j

>

CEH

VDLSg

F

3

q

XR

9

Bu

n

5N

P

Y

O

Z

r

7

p

0

8

Sub-level sets

S

c

3

S

c

2

S

c

1

¯

S

c

2

¯

S

c

1

¯

S

c

3

Normalised

Sub-level sets

S

∞

φ

∞

∝ ‖ · ‖

1

4 Main result: directional convergence towards the ϕ ∞ -max margin

We now state our formal results, starting with the precise construction of the horizon function ϕ ∞ , followed by the theorem showing convergence of the iterates towards the ϕ ∞ -max-margin.

4.1 Construction of the horizon function ϕ ∞

We first define the horizon shape of a potential ϕ , and provide sufficient conditions for its existence. Then, we use this shape to construct a horizon function ϕ ∞ , which allows the interpretation of the directional limits of gradients of ϕ at infinity. The proofs require technical elements from variational analysis to ensure that the limits are well-defined; these are deferred to Appendix B.

Horizon shape. Assume w.l.o.g. that ϕ (0) = 0 . For c ≥ 0 , consider the sublevel set:

which is nonempty and compact by coercivity of ϕ . We can then define the normalised sublevel set:

By construction, the set ¯ S c belongs to the unit ball. We are interested in the limit of ¯ S c as c →∞ . Definition 1. We say that ϕ admits a horizon shape if the family of normalized sublevel sets ( ¯ S c ) c> 0 defined in Equation (4) converges to some compact set S ∞ as c →∞ for the Hausdorff distance. In addition, we say that this shape is non-degenerate if the origin belongs to the interior of S ∞ .

The Hausdorff distance is a natural distance on compact sets [see Rockafellar and Wets, 1998, Section 4.C., for a definition]. In Proposition 2, we prove the existence of the horizon shape for a large class of functions which contains all the potentials with domain R d encountered in practice. Although the horizon shape is guaranteed to exist for most functions, we cannot a priori prove that it is non-degenerate, as the normalized sub-levels ¯ S c can become 'flat' as c →∞ . 5 Given the technical complexity associated with this case, we now focus exclusively on non-degenerate horizon shapes.

Horizon function. If ϕ admits a non-degenerate horizon shape S ∞ , we define its horizon function as the Minkowski gauge [Rockafellar and Wets, 1998, Section 11.E] of S ∞ :

By construction, the horizon function ϕ ∞ is an asymmetric norm and its sub-level sets correspond to scaled versions of S ∞ [see Rockafellar and Wets, 1998, Section 11.C, for more properties]. For example, in the case of the horizon shape S ∞ illustrated in Figure 3, the corresponding horizon function ϕ ∞ is proportional to the ℓ 1 -norm. Although the construction of ϕ ∞ presented here is rather abstract, we show in Theorem 3 that for separable potentials defined over R d , it can be computed with an explicit formula. Though different, our definition of the horizon function shares many similarities with the classical concept of horizon function from convex analysis [Rockafellar and Wets, 1998]. We discuss the links between the two notions at the end of Section 4.3.

4.2 Main result: directional convergence of the iterates towards the ϕ ∞ -max-margin

We can now state our main result which fully characterises the directional convergence of mirror flow.

Theorem 2. Assume that ϕ admits a non-degenerate horizon shape and let ϕ ∞ be its horizon function. Assuming that the following ϕ ∞ -max-margin problem has a unique minimiser, then the mirror flow normalised iterates ¯ β t = β t ∥ β t ∥ converge towards a vector ¯ β ∞ and

where the symbol ∝ denotes positive proportionality.

Remark on the uniqueness of the margin problem. If the unit ball of ϕ ∞ is strictly convex, then the ϕ ∞ -max-margin problem has a unique solution. However, in the general case, there may exist an infinity of solutions and weak but ad hoc assumptions on the dataset are required to guarantee uniqueness. For instance, if ϕ ∞ is proportional to the ℓ 1 -norm, a common assumption which ensures uniqueness is assuming that the data features are in general position [Dossal, 2012].

4.3 Assumptions guaranteeing the existence of ϕ ∞

Our main result, presented in Theorem 2 relies on the existence of a horizon shape, S ∞ , as described in Definition 1. From this shape, the asymmetric norm ϕ ∞ is constructed.

Weshow here that the existence of S ∞ is ensured for a large class of 'nice' functions, specifically those definable in o-minimal structures [Dries, 1998]. For the reader unfamiliar with this notion, this class contains all 'reasonable' functions used in practice, such as polynomials, logarithms, exponentials, and 'reasonable' combinations of those. This is a typical assumption used for instance to prove the convergence of optimisation methods through the Kurdyka-Łojasiewicz property [Attouch et al., 2011].

Proposition 2. If any of the three following conditions hold: (i) ϕ is a finite composition of polynomials, exponentials and logarithms, (ii) ϕ is globally subanalytic, (iii) ϕ is definable in a o-minimal structure on R ; then ϕ admits a horizon shape S ∞ .

The proof is technical and we defer it to Appendix B. Although the previous proposition ensures the existence ϕ ∞ for a wide range of potentials, it does not offer a direct method for computing it. In the following, we show that for potentials that are both separable and even, a simple formula exists, allowing for the direct calculation of ϕ ∞ .

We note that φ is a bijection over R ≥ 0 , and denote by φ -1 its inverse. We consider the function φ -1 ◦ ϕ , which can be seen as a renormalisation of ϕ . It has the same level sets as ϕ and ensures that lim η → 0 ηφ -1 ( ϕ ( ¯ β/η )) exists in R > 0 for all ¯ β . These two observations lead to the following theorem.

Assumption 4. The potential ϕ is separable, in the sense that there exists φ : R → R ≥ 0 such that ϕ ( β ) = ∑ d i =1 φ ( β i ) . We assume that φ satisfies Assumption 2, that it is definable in a o-minimal structure on R and that it is an even function. W.l.o.g. we assume that φ (0) = 0 .

Theorem 3. Under Assumption 4, the potential ϕ admits a non-degenerate horizon shape and its horizon function is such that there exists λ > 0 such that for every ¯ β ∈ R d :

We use this simple formula when computing ϕ ∞ for various potentials in the next section.

Remark on previous notions of horizon function. In the convex analysis literature, the horizon function is typically defined as ϕ ∞ ( ¯ β ) = lim η → 0 ηϕ ( ¯ β/η ) [Rockafellar and Wets, 1998, Laghdir and Volle, 1999]. In our context, this definition would yield a function which equals + ∞ everywhere except at the origin. In contrast, our definition ensures that ϕ ∞ attains finite values over R d . The distinction stems from our way of normalising the level sets by R c in Section 4.1, or alternatively, from the composition by φ -1 in the separable case. The two constructions would coincide only if ϕ was Lipschitz continuous, which is at odds with Assumption 2.

<latexi sh

1_b

64="9YR+LugTvS

K

o

5y2

O

8

>A

B

n

c

VD

J

E

zF

I

jH

k

Z

w

d

M

C

/

PG

W

3

r

p

f

7

q

NU

X

Q

m

0

<latexi sh

1_b

64="u+

Qnk

L

8

jf

UO/

c0vA

Y

>

CDX

V

7SgNBFJ3

G

Mr

m

z

W

2

R

y

Z

d

9

5

p

w

H

I

q

E

Po

K

<latexi sh

1_b

64="9YR+LugTvS

K

o

5y2

O

8

>A

B

n

c

VD

J

E

zF

I

jH

k

Z

w

d

M

C

/

PG

W

3

r

p

f

7

q

NU

X

Q

m

0

<latexi sh

1_b

64="72UG9

K/m

f

HDwWM

YrT

S

Q

>A

C

c

V

gNBFJ3

L

Zj

I

E

0

X

k

R

d

+

8

P

Ov

5

z

o

p

y

u

q

n

<latexi sh

1_b

64="p

j

/Gf8cFYJVqD

L

Pv

+U

>A

C

n

SgM

3

r

d

RXZ

u

w

K

2

H

o

I

79

0

E

y

z

5

O

N

Q

B

T

<latexi sh

1_b

64="3SZI8X

u

gAV5

r

H

7

vq

D

>

B

n

c

N

EJ

U

/

9L

0

F

Q

oWy2m

p

w

G

M

K

fT

d

+

R

O

Y

<latexi sh

1_b

64="K7gI

0EUWTun

S+

2vXA

w

8

>

CD3

c

V

L

M

F

rP

d

km

J

RQZ

zY

y

5

/

N

G

f

9

o

O

q

H

B

pj

<latexi sh

1_b

64="

IU7y

+

NB

L2

K89rZg

d

/

>A

CEn

c

V

S

FJ3

G

M

mkz

T

R

w

Y

Qj

v

5

q

0

W

o

D

H

Of

X

P

u

p

<latexi sh

1_b

64="EV+8HgyoUGA

ck

n

X

9NDw

>

CB

JS

O2

W

KM

QL

m

Y

I

Z

d

z0F

j

/

P

fv

3

p

R

r

q

u

T

5

7

<latexi sh

1_b

64="WHZu/rQp

K

vn8

7jVq5

o

>A

C

3

d

BNS

E

f2

y

RPJ

9

m

kX

wO

F

L

0

+g

U

M

Y

D

z

G

T

<latexi sh

1_b

64="c qpL/dTHzOS9Gr

2

KUw

5v

Y

>A

C

3

ZD

N

FI

n

V

uBo

g

0

X

W

m

M

k

y

P

8

+

E

7

J

j

R

fQ

<latexi sh

1_b

64="8z

0d

KW3k

B5n9Q

+

g

E

I

>A

C

X

c

V

NS

Ur q/

L

T2up

j

m

G

Y

O

Fv yJ

f

D

P

w

Z

H

o

R

M

7

<latexi sh

1_b

64="Qr

5K0CvBj

UZROP

MWX/nNw

g

>A

3

cdV

S8

E

f2q

y

m+

H

I

L

u

D

7Y

Tk

J

zG

9

250

500

250

β

MD

1

t

β

t

Iterate trajectories in the

-plane

ℝ

2

First coordinate

Second

coordinate

β

MD

2

t

Iterate trajectories in the

- open unit disk

ℝ

2

under the rescaling

π

(

β

) =

β

1 +

‖

β

‖

2

π

(

β

MD

1

t

)

π

(

β

t

)

π

(

β

MD

2

t

)

¯

β

MD

1

∞

¯

β

∞

¯

β

MD

2

∞

Rescaled trajectories

for visualisation purposes

(arrows mark the progression of iterates over time, diverging outward)

5 Applications and experiments

In this section, we illustrate our main result using various potentials.

Homogeneous potentials. We first consider potentials ϕ which are L -homogeneous, i.e., there exists L > 0 such that for all c > 0 and β ∈ R d , ϕ ( cβ ) = c L ϕ ( β ) . In this case, the sublevel sets ¯ S c are all equal. It follows that ϕ ∞ ∝ ϕ 1 /L . An important example is the case of ϕ = ∥ · ∥ p p where ∥ · ∥ p corresponds to the ℓ p -norm with p > 1 , for which we get that ϕ ∞ ∝ ∥ · ∥ p and we recover the result from Sun et al. [2022, 2023].

Hyperbolic-cosine entropy potential. Finally, we consider ϕ MD 1 ( β ) = ∑ d i =1 (cosh( β i ) -1) . Applying Theorem 3, we get that ϕ ∞ ∝ ∥ · ∥ ∞ .

Hyperbolic entropy potential. The hyperbolic entropy potential: ϕ MD 2 ( β ) = ∑ d i =1 ( β i arcsinh( β i ) -√ β 2 i +1 -1) plays a central role in works considering diagonal linear networks [Woodworth et al., 2020, Pesme and Flammarion, 2023]. Applying Theorem 3, we obtain that ϕ ∞ ∝ ∥ · ∥ 1 and we recover the result from Lyu and Li [2020] and Moroshko et al. [2020].

Experimental details concerning Figure 1. As shown in Figure 1 (Middle), we generate 40 points with positive labels and 40 points with negative labels. Starting from β 0 = 0 , we run mirror descent with the exponential loss ℓ ( z ) = exp( -z ) and with the three following potentials:

Wefirst observe in Figure 1 (Left) that the training loss converges to zero, as predicted by Proposition 1, with a convergence rate that varies across different potentials. Moreover, as illustrated in Figure 1 (Middle and Right), the iterates converge in direction towards their respective unique ϕ ∞ -max margin solutions associated with the following geometries:

Therefore, by employing various potentials, we can induce different implicit biases, leading to distinct generalisation properties depending on the data distribution. The trajectories of the mirror descent iterates are shown and commented in Figure 4.

6 Conclusion and limitations

In this paper, we offer a comprehensive characterisation of the implicit bias of mirror flow for separable classification problems. This characterisation is framed in terms of the horizon function associated with the mirror descent potential, leveraging the asymptotic geometry induced by the potential. Note that we did not cover the discrete mirror descent algorithm; we believe the analysis would extend without additional difficulties compared to the continuous counterpart.

Extensions and open problems. Our results being purely asymptotic, characterising the rate at which the normalised iterates converge towards the maximum-margin solution is an open direction for future research. Furthermore, we note that our analysis does not cover potentials that are defined only on a strict subset of R d (such as the log-barrier and the negative entropy), and with possibly non-coercive gradients. This class of potentials is of interest as it arises when investigating deep architectures, such as diagonal linear networks of depth D > 2 . In this setting, it is known that gradient flow on the weights lead to a mirror flow on the predictors with a certain potential ϕ D [Woodworth et al., 2020]. Interestingly, the potentials ϕ D have non-coercive gradients and their horizon functions do not depend on the depth D as they are all proportional to the ℓ 1 -norm. The predictors are, however, known to converge in direction towards a KKT point of the non-convex ℓ 2 /D -max-margin problem [Lyu and Li, 2020] which can be different from the ℓ 1 -max-margin problem [Moroshko et al., 2020]. This observation highlights that our coercive gradient assumption is necessary for our result to hold. However, extending our analysis beyond this assumption is a promising direction for understanding gradient dynamics in deep architectures.

Acknowledgments and Disclosure of Funding

S.P. would like to thank Loucas Pillaud-Vivien for the helpful discussions at the beginning of the project as well as Pierre Quinton for the careful proofreading of the paper. The authors also thank Jérôme Bolte and Edouard Pauwels for the precious help on the existence of the Hausdorff limit of definable families, as well as Nati Srebro and Lénaïc Chizat for the remarks on the limitations of our result for potentials with non-coercive gradients. This work was supported by the Swiss National Science Foundation (grant number 212111).

References

Charalambos Aliprantis and Kim Border. Infinite Dimensional Analysis . Springer Berlin, Heidelberg, 2006.

Ehsan Amid and Manfred K Warmuth. Winnowing with gradient descent. In Conference on Learning Theory , pages 163-182. PMLR, 2020a.

Ehsan Amid and Manfred KK Warmuth. Reparameterizing mirror descent as gradient descent. Advances in Neural Information Processing Systems , 33:8430-8439, 2020b.

Attouch, X. Goudou, and P. Redont. The heavy ball with friction method, i. the continuous dynamical system: Global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipitive dynamical system. Communications in Contemporary Mathematics , 2(1): 1-34, 2000.

Hedy Attouch and Gerald Beer. On the convergence of subdifferentials of convex functions. Archiv der Mathematik , 1993.

Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semialgebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods. Mathematical Programming , 2011.

Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning , pages 468-477. PMLR, 2021.

Heinz H Bauschke, Jérôme Bolte, and Marc Teboulle. A descent lemma beyond lipschitz gradient continuity: first-order methods revisited and applications. Mathematics of Operations Research , 42(2):330-348, 2017.

Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Jérôme Bolte, Aris Daniilidis, and Adrian Lewis. Tame functions are semismooth. Mathematical Programming , 2007.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory , pages 1305-1338. PMLR, 2020.

Combari and L. Thibault. On the graph convergence of subdifferentials of convex functions. Proceedings of the American Mathematical Society , 1998.

Richard Courant and Fritz John. Introduction to Calculus and Analysis . Springer New York, 1989.

Charles Dossal. A necessary and sufficient condition for exact sparse recovery by l1 minimization. Comptes Rendus Mathematique , 350(1-2):117-120, 2012.

P. D. van den Dries. Tame Topology and O-minimal Structures . Cambridge University Press, 1998.

Miroslav Dudík, Robert E Schapire, and Matus Telgarsky. Convex analysis at infinity: An introduction to astral space. arXiv preprint arXiv:2205.03260 , 2022.

Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s)gd over diagonal linear networks: Implicit bias, large stepsizes and edge of stability. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems , 30, 2017.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning , pages 1832-1841. PMLR, 2018.

Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300 , 2018.

Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via a primal-dual analysis. In Algorithmic Learning Theory , pages 772-804. PMLR, 2021.

Ziwei Ji, Miroslav Dudík, Robert E Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Conference on Learning Theory , pages 2109-2136. PMLR, 2020.

Beata Kocel-Cynk, Wiesław Pawłucki, and Anna Valette. A short geometric proof that hausdorff limits are definable in any o-minimal structure. advg , 2014.

Mohammed Laghdir and Michel Volle. A general formula for the horizon function of a convex composite function. Archiv der Mathematik , 1999.

B Lemaire. An asymptotical variational principle associated with the steepest descent method for a convex function. Journal of Convex Analysis , 3:63-70, 1996.

Zhiyuan Li, Tianhao Wang, Jason D Lee, and Sanjeev Arora. Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent. Advances in Neural Information Processing Systems , 35:34626-34640, 2022.

Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems , 33:8543-8552, 2020.

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations , 2020.

Edward Moroshko, Blake E Woodworth, Suriya Gunasekar, Jason D Lee, Nati Srebro, and Daniel Soudry. Implicit bias in deep linear classification: Initialization scale vs training accuracy. Advances in neural information processing systems , 33:22182-22193, 2020.

Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning , pages 4683-4692. PMLR, 2019a.

Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3420-3428. PMLR, 2019b.

Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3051-3059. PMLR, 2019c.

Hristo Papazov, Scott Pesme, and Nicolas Flammarion. Leveraging continuous time to understand momentum when training diagonal linear networks. In International Conference on Artificial Intelligence and Statistics , pages 3556-3564. PMLR, 2024.

Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. Neurips , 2023.

Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems , 34, 2021.

Tyrrell Rockafellar and Roger J. B. Wets. Variational Analysis . Springer Berlin Heidelberg, 1998.

Ralph Tyrell Rockafellar. Convex Analysis . Princeton University Press, 1970.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 2018.

Haoyuan Sun, Kwangjun Ahn, Christos Thrampoulidis, and Navid Azizan. Mirror descent maximizes generalized margin and can be implemented efficiently. Advances in Neural Information Processing Systems , 35, 2022.

Haoyuan Sun, Khashayar Gatmiry, Kwangjun Ahn, and Navid Azizan. A unified approach to controlling implicit regularization via mirror descent. arXiv preprint arXiv:2306.13853 , 2023.

Lou Van den Dries and Chris Miller. Geometric categories and o-minimal structures. 1996.

Aditya Vardhan Varre, Maria-Luiza Vladarean, Loucas Pillaud-Vivien, and Nicolas Flammarion. On the spectral bias of two-layer linear networks. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems , 32, 2019.

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory , pages 3635-3673. PMLR, 2020.

Fan Wu and Patrick Rebeschini. Implicit regularization in matrix sensing via mirror descent. Advances in Neural Information Processing Systems , 34:20558-20570, 2021.

Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations , 2021.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR , 2017.

Organisation of the Appendix.

In Appendix A, we provide the proofs of the existence and uniqueness of (MF), of the convergence of the loss, the divergence of the iterates and the proof of Lemma 1.

In Appendix B, we provide all the proofs concerning the construction of the horizon shape and that of our main Theorems 2 and 3.

A Proofs of properties of the mirror flow in the classification setting

We start by proving the existence and uniqueness of (MF) over R ≥ 0 . The proof is standard and relies on ensuring that the iterates do not diverge in finite time.

Lemma 2. For any initialisation β 0 ∈ R d , there exists a unique solution defined over R ≥ 0 which satisfies (MF) for all t ≥ 0 and with initial condition β t =0 = β 0 .

Proof. From Assumption 2, we have that ϕ is differentiable, strictly convex and its gradient is coercive. Consequently, ∇ ϕ is bijective over R d (see Rockafellar [1970], Theorem 26.6). Furthermore, the Fenchel conjugate ϕ ∗ is differentiable over R d and ( ∇ ϕ ) -1 = ∇ ϕ ∗ .

To prove the existence and uniqueness of a global solution of (MF), we first consider the following differential equation:

Since L is C 2 , ∇ L is Lipschitz on all compact sets. Furthermore, since ∇ 2 ϕ is p.s.d., ∇ ϕ ∗ = ( ∇ ϕ ) -1 is C 1 and therefore Lipschitz on all compact sets. Hence ∇ L ◦ ∇ ϕ ∗ is Lipschitz on all compact sets and from the Picard-Lindelöf theorem, there exists a unique maximal ( i.e. which cannot be extended) solution ( u t ) satisfying eq. (5) such that u t =0 = ∇ ϕ ∗ ( β 0 ) . We denote [0 , T max ) the intersection of this maximal interval of definition (which must be open) and R ≥ 0 . Our goal is now to prove that T max = + ∞ . To do so, we assume that T max is finite and we will show that this leads to a contradiction due to the fact that the iterates β t cannot diverge in finite time. Let β t := ∇ ϕ ∗ ( u t ) and notice that β t is therefore the unique solution satisfying (MF) over [0 , T max ) with β t =0 = β 0 .

with initial condition u t =0 = ∇ ϕ ∗ ( β 0 ) .

Bounding the trajectory of β t over [0 , T max ) . Pick any β ∈ R d and notice that by convexity of L :

Where L min is a lower bound on the loss. Integrating from 0 to t < T max we get:

Therefore, due to Assumption 2, the iterates β t are bounded over [0 , T max ) . The proof from here is standard (see e.g. Attouch et al. [2000], Theorem 3.1): from eq. (5) we get that ˙ u t is bounded over [0 , T max ) and sup t ∈ [0 ,T max ) ∥ ˙ u t ∥ =: C < + ∞ which means that ∥ u t -u t ′ ∥ ≤ C | t -t ′ | . Hence lim t → T max u t =: u ∞ must exist. Applying the Picard-Lindelöf again at time T max with initial condition u ∞ violates the initial maximal interval assumption. Therefore T max = + ∞ which concludes the proof.

We now recall and prove classical results on the mirror flow in the classification setting.

Proposition 1. Considering the mirror flow ( β t ) t ≥ 0 , the loss converges towards 0 and the iterates diverge: lim t →∞ L ( β t ) = 0 and lim t →∞ ∥ β t ∥ = + ∞ .

Proof. The loss is decreasing. d d t L ( β t ) = ⟨∇ L ( β t ) , ˙ β t ⟩ = -⟨∇ 2 ϕ ( β t ) -1 ∇ L ( β t ) , ∇ L ( β t ) ⟩ ≤ 0 , where the inequality is due to the convexity of the potential ϕ .

Convergence of the loss towards 0 . Now consider the Bregman divergence between an arbitrary point β and β t :

which is such that:

where the inequality is by convexity of the loss. Integrating and due to the decrease of the loss, we get that:

Since this is true for all point β , we get that L ( β t ) ≤ inf β ∈ R d L ( β ) + D ϕ ( β,β 0 ) t . It remains to show that the right hand term goes to 0 as t goes to infinity. To show this, let ε > 0 , by the separability assumption we get that there exists β ⋆ such that min y i ⟨ x i , β ⋆ ⟩ > 0 . Since L ( λβ ⋆ ) -→ λ →∞ 0 , we can choose λ big enough such that L ( λβ ⋆ ) < ε and then t λ large enough such that 1 t λ D ϕ ( λβ ⋆ , β 0 ) < ε . The loss therefore converges to 0 .

Divergence of the iterates.

For all i ∈ [ n ] , ℓ ( y i ⟨ x i , β t ⟩ ) ≤ L ( β t ) -→ t →∞ 0 . Due to the assumptions on the loss, this translates into y i ⟨ x i , β t ⟩ -→ t →∞ ∞ , hence ∥ β t ∥ -→ t →∞ + ∞ .

Introducing a few notations. Before giving two important lemmas, we provide a few notations. Recall that Z is the data matrix of size n × d whose i th row is y i x i , we then have that ∇ L ( β ) = Z ⊤ ℓ ′ ( Zβ ) , where ℓ ′ is applied component wise. We now denote by q ( β ) the vector in R n equal to:

Notice that due to ℓ > 0 , ℓ ′ < 0 , ℓ -1 is increasing and ℓ ′ is decreasing, we have that q ( β ) > 0 and that for all i 0 ∈ [ n ] ,

Therefore q ( β ) ∈ (0 , 1] n .

We further denote

This way we can simply write ∇ L ( β t ) = -a t Z ⊤ q t with q t := q ( β t ) . Integrating the flow (MF), we write:

Two lemmas. In the following lemma we recall and prove that a coordinate q ∞ [ k ] must be equal to 0 if datapoint x k is not a support vector of ¯ β ∞ .

Lemma 3. For some function C t →∞ , if the iterates ¯ β t = β t C t converge towards a vector which we denote ¯ β ∞ and q t converges towards a vector q ∞ ∈ [0 , 1] n . Then it holds that:

In words, q ∞ [ k ] = 0 if x k is not a support vector.

Proof. Recall that

From Proposition 1, we have that min i ∈ [ n ] y i ⟨ x i , ¯ β ∞ ⟩ > 0 and we denote this margin as γ . Now consider k ∈ [ n ] which is not a support vector, i.e, y k ⟨ x k , ¯ β ∞ ⟩ > min i ∈ [ n ] y i ⟨ x i , ¯ β ∞ ⟩ and without loss of generality assume that y 1 ⟨ x 1 , ¯ β ∞ ⟩ = min 1 ≤ i ≤ n y i ⟨ x i , ¯ β ∞ ⟩ . We denote by δ = ⟨ y k x k -y 1 x 1 , ¯ β ∞ ⟩ > 0 the gap. Then

We write ¯ β t = ¯ β ∞ + r t where ( r t ) t ≥ 0 ∈ R d converges to 0 . For t big enough, we have that y k ⟨ x k , ¯ β t ⟩ ≥ y k ⟨ x k , ¯ β ∞ ⟩ -δ 4 and y 1 ⟨ x 1 , ¯ β t ⟩ ≤ y 1 ⟨ x 1 , ¯ β ∞ ⟩ + δ 4 . Therefore for t large enough, since ℓ ′ is negative and increasing:

where the last term converge to 0 due to the exponential tail of -ℓ ′ and that C t →∞ .

We here reformulate and prove Lemma 1.

Proof. Recall that ∇ ϕ ( β t ) = ∇ ϕ ( β 0 ) + Z ⊤ ∫ t 0 a ( β s ) q ( β s )d s , therefore

Lemma 4 (Reformulation of Lemma 1) . Denoting a ( β t ) := -ℓ ′ ( ℓ -1 ( ∑ i ℓ ( y i ⟨ x i , β t ⟩ ))) > 0 , we have that ∫ t 0 a ( β s )d s -→ t →∞ + ∞ . For ℓ ( z ) = exp( -z ) , this translates to ∫ t 0 L ( β s )d s →∞ .

Where the first inequality is due to the triangle inequality and the second to the fact q ( β ) ∈ (0 , 1] n . Since the iterates diverge, we have from Assumption 2 that ∥∇ ϕ ( β t ) ∥ -→ t →∞ ∞ and therefore that

B Differed proofs on the construction of ϕ ∞

As mentioned in the main text, the following property highlights the fact that all 'reasonable' potentials have a horizon shape.

Proposition 2. If any of the three following conditions hold: (i) ϕ is a finite composition of polynomials, exponentials and logarithms, (ii) ϕ is globally subanalytic, (iii) ϕ is definable in a o-minimal structure on R ; then ϕ admits a horizon shape S ∞ .

Proof. Note that points (i) and (ii) are particular cases of (iii) [Dries, 1998, Bolte et al., 2007]. If h is definable in a o-minimal structure, then so is the sublevel set S c for c > 0 , and so is the normalization factor R c since it can be defined in first-order logic as

Therefore, ( ¯ S c ) c> 0 if a definable family of definable and compact sets. Then so is the family ( ¯ S t -1 ) t ∈ (0 , 1] . Since all the sets belong to the unit ball of R d , they lie in the sets of compact subsets of B (0 , 1) . This set is compact for the Hausdorff metric [Aliprantis and Border, 2006, Thm 3.85]; therefore, there exists a sequence ( t k ) k ∈ N such that t k → 0 and ( ¯ S t -1 k ) k ∈ N converges to some set ¯ S .

We can then apply Corollary 2 of Kocel-Cynk et al. [2014], which states that there exists a definable arc γ : (0 , 1] → (0 , 1] such that lim τ → 0 γ ( τ ) = 0 and ¯ S = lim τ → 0 ¯ S γ ( τ ) -1 . This implies that the limit ¯ S is uniquely defined and therefore that lim t → 0 ¯ S t -1 = ¯ S .

The next corollary is a more general restatement of Corollary 1. It shows that the construction of ϕ ∞ enables to take the limit lim t ∇ ϕ ( β t ) t ∝ ∂ϕ ∞ ( ¯ β ∞ ) .

Corollary 2. Assume that ϕ admits a non-degenerate horizon shape S ∞ . Then its horizon function ϕ ∞ satisfies the following properties.

ϕ ∞ is convex and finite-valued on R d ,

Let ( β s ) s> 0 be a continous sequence such that when s →∞ :

Then ¯ g is proportional to a subgradient of ϕ ∞ at ¯ β :

Proof. The sequence of sets ( ¯ S c ) is contained in the compact ball B (0 , 1) ; therefore, Hausdorff convergence is equivalent to Painlevé-Kuratowski convergence [Rockafellar and Wets, 1998, Section 4.C]. Hence, as ( ¯ S c ) are convex, so is their limit S ∞ [Rockafellar and Wets, 1998, Prop 4.15]. It follows that ϕ ∞ is convex [Rockafellar and Wets, 1998, Ex 3.50].

Since S ∞ is non-degenerate, there exists a radius r 0 such that B (0 , r 0 ) ⊂ S ∞ , which implies that ϕ ∞ ( β ) is finite-valued for every β .

To prove point (ii), consider the sequence of functions ( η c ) c> 0 formed by the indicators of convex sets ¯ S c :

Note that the epigraph of η c is ¯ S c × R + ; these sets also converge to S ∞ × R + [Rockafellar and Wets, 1998, Ex 4.29], from which we conclude that function η c converge epigraphically to the indicator function η ∞ of S ∞ ( η ∞ = I S ∞ ). We can then apply Attouch's theorem [Attouch and Beer, 1993, Combari and Thibault, 1998] ensuring that the graph of the subdifferentials of η c

converge in Painlevé-Kuratowski sense to the graph G ( ∂η ∞ ) of subdifferential of η ∞ . This means that if a sequence ( β c , g c ) c> 0 such that ( β c , g c ) ∈ G ( ∂η c ) for every c > 0 converges, then its limit belongs to G ( ∂η ∞ ) .

Consider now a sequence ( β s ) s> 0 satisfying the conditions described in (ii). Since it diverges to infinity and ϕ is coercive, we have ϕ ( β s ) →∞ , and we may assume w.l.o.g that ϕ ( β s ) > 0 for all s .

We have by definition of sublevel sets that β s ∈ S ϕ ( β s ) , and therefore the gradient ∇ ϕ ( β s ) belongs to the normal cone of S ϕ ( β s ) at β s (indeed, the gradient is orthogonal to the level sets; see e.g., [Courant and John, 1989, Chapter 1.5]). Since the normal cone of a convex set is the subdifferential of its indicator [Rockafellar, 1970, Section 23], we thus have

Consider now the normalized levels sets as defined in (4). Denoting

we have ¯ β s ∈ ¯ S ϕ ( β s ) and thus by simple rescaling (8) becomes

Since ∂I ¯ S c is a cone (the normal cone to ¯ S c ), this also holds for any positive multiple of ∇ ϕ ( β s ) . We deduce that for every s > 0

Note that since ¯ β s belongs to the normalized level sets, this sequence is bounded. We can extract a subsequence ( ¯ β s k , ∇ ϕ ( β s k ) ∥∇ ϕ ( β s k ) ∥ ) k ≥ 0 which converges to a limit point ( ˆ β, ˆ g ) . By the previous remark on graphical convergence of subdifferentials, we have ( ˆ β, ˆ g ) ∈ G ( ∂I S ∞ ) , i.e.,

We need to prove that ˆ β is not 0 . Since ϕ is strictly convex, the level set { ϕ ( β ) = c } is exactly the boundary of the sublevel set { ϕ ( β ) ≤ c } . Therefore, β s lies on the boundary of S ϕ ( β s ) , and hence so does ¯ β s lie on the boundary of ¯ S ϕ ( β s ) . Since 0 is in the interior of S ∞ , it also belongs to the interior of ¯ S ϕ ( β s ) for s larger than some s 0 . Then, there exists r 0 > 0 such that B (0 , r 0 ) ⊂ ¯ S ϕ ( β s ) for s ≥ s 0 . By definition of boundary, we then have for s ≥ s 0 ∥ ¯ β s ∥ > r 0 , which leads to ∥ ˆ β ∥ > 0 .

To achieve the desired result; we need to relate ( ˆ β, ˆ g ) to ( ¯ β, ¯ g ) . First, notice that by construction we have necessarily ˆ g = ¯ g = lim s →∞ ∇ ϕ ( β s ) / ∥∇ ϕ ( β s ) ∥ . Then, note that

Taking the norm of the second limit, we have ∥ ˆ β ∥ = lim k →∞ ∥ β s k ∥ R ϕ ( βs k ) . Injecting back in the first limit yields

Therefore, (9) becomes

This means that ¯ g belongs to the normal cone of S ∞ at ∥ ˆ β ∥ ¯ β [Rockafellar, 1970, Sec. 23]. We note the level set { β : ϕ ∞ ( β ) ≤ ϕ ∞ ( ∥ ˆ β ∥ ¯ β ) } is exactly τS ∞ for some τ > 0 . We use Corollary 23.7.1 from Rockafellar [1970] which states that if a vector is in the normal cone of the level set of ϕ ∞ , then it must be a positive multiple of a subgradient. This implies that that there exists λ ≥ 0 such that

Finally, λ > 0 since ∥ ¯ g ∥ = 1 , and ∂ϕ ∞ ( ∥ ˆ β ∥ ¯ β ) = ∂ϕ ∞ ( ¯ β ) by positive homogenity of ϕ ∞ .

We can now prove our main result, which we restate here.

Theorem 2. Assume that ϕ admits a non-degenerate horizon shape and let ϕ ∞ be its horizon function. Assuming that the following ϕ ∞ -max-margin problem has a unique minimiser, then the mirror flow normalised iterates ¯ β t = β t ∥ β t ∥ converge towards a vector ¯ β ∞ and

where the symbol ∝ denotes positive proportionality.

The proof essentially follows exactly the same lines as in Section 3 but taking into account the fact that the loss is not exactly the exponential one.

Proof. Recall the definitions of the quantities a t and q t given above Equation (7) which enable to write:

Similar to the time change we performed in Section 3, we consider θ ( t ) = ∫ t 0 a s d s . From Lemma 4, θ is a bijection over R ≥ 0 and perform the time change ˜ β t = β θ -1 ( t ) . Due to the chain rule, after the time change and dropping the tilde notation we obtain:

where ¯ q t := 1 t ∫ t 0 q s d s corresponds to the average of ( q s ) s ≤ t .

Extracting a convergent subsequence: We now consider the normalised iterates ¯ β t = β t ∥ β t ∥ and up to an extraction we get that ¯ β t → ¯ β ∞ . Since q t is a bounded function, up to a second extraction, we have that q t → q ∞ , and the same holds for its average: ¯ q t → q ∞ . Taking the limit in Equation (10) we immediately obtain that:

We can now directly apply Corollary 2 and there exists λ > 0 such that:

The end of the proof is then as explained in Section 3.

Finally we recall and prove Theorem 3 which provides a simple formula for the horizon function in the case of separable potentials.

Theorem 3. Under Assumption 4, the potential ϕ admits a non-degenerate horizon shape and its horizon function is such that there exists λ > 0 such that for every ¯ β ∈ R d :

Dividing by t we get:

which also means that

Proof. Lipschitzness, upper and lower boundedness. For η > 0 , let us denote by h η : β ↦→ η · φ -1 ( ϕ ( β/η )) and notice that ∇ h η ( β ) = ( φ ′ ( β k /η ) φ ′ ( φ -1 ( ∑ i φ ( β i /η ))) ) k ∈ [ d ] ≥ 0 Since φ ≥ 0 and that φ -1 and φ ′ are increasing we get that that ∇ h η ( β ) ∈ [0 , 1] d . Therefore ( h η ) η> 0 are uniformly Lipschitz-continuous. Consequently, for all β , h η ( β ) is upper-bounded independently of η . Lastly, since φ ≥ 0 , notice that h η ( β ) ≥ min i | β i | > 0 for all β = 0 .

̸

Point-wise and epi-convergence of h η . For all ¯ β , by composition, η ↦→ η · φ -1 ( ϕ ( ¯ β/η )) is a definable function, the monotonicity Lemma [Van den Dries and Miller, 1996] (Theorem 4.1) ensures that it has a unique limit in R which we denote h 0 ( β ) . From the uniform Lipschitzness of h η , we get that ( η, β ) ∈ R ≥ 0 × R d ↦→ h η ( β ) is continuous. Hence for all sequence η k → 0 , we get that h η k epi-converges to h 0 . Therefore (epi h η k ) k converges in the Painlevé-Kuratowski sense towards epi h η 0 .

Link between the level sets of h η and those of ϕ . To conclude the proof it remains to notice that for all c ≥ 0 :

Therefore letting η c = 1 /φ -1 ( c ) we get that

This simply means that η c is an appropriate normalising quantity, it replaces the normalisation by the radius of S c . Since { ¯ β ∈ R d , h η c ( ¯ β ) ≤ 1 } converges in the Painlevé-Kuratowski sense towards { ¯ β ∈ R d , h 0 ( ¯ β ) ≤ 1 } , we get that R c η c · ¯ S c converges towards the same set. However, with our previous construction, we also have that ¯ S c converges towards ¯ S ∞ . The sets ¯ S ∞ and { ¯ β ∈ R d , h 0 ( ¯ β ) ≤ 1 } are therefore proportional and h 0 ∝ ϕ ∞ which concludes the proof.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Theorems 2 and 3 give the results which are claimed in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We included a "Limitations" paragraph at the end of the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The main assumptions on the loss, potential and dataset are given at the beginning of the paper. If an additional assumption is required, it is clearly stated in the result. The proofs are all deferred to the appendix but we clearly state in the main text where they can be found.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide extremely toy examples which are very easy to reproduce. The dataset is shown in Figure 1 and, as stated in Section 5, we perform mirror descent over the dataset with the exponential loss.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code behind the experiments is straightforward and can easily be reproduced.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The considered potentials and loss are given. The value of the step-size is not given as it does not have any relevance.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The use of error bars is not relevant in our work: there is no source of stochasticity.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: It is clear that our experiments can easily be reproduced by any computer.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked the Code of Ethics and our work respects it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Given the theoretical aspect of our work, we don't think are work leads to any direct positive nor negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There is no use of existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors study the implicit bias of the mirror descent algorithm, from the perspective of the optimization trajectory of the continuous flow version. They propose the conceptions of horizon shape and horizon function $\\phi_\\infty$ to help characterize the properties of mirror flow at infinity. Since $\\phi_\\infty$ defines a norm, they prove that the mirror flow will eventually converge in direction to a max $\\phi_\\infty$-margin solution under the linear exponential-tail classification setting. Their findings contribute to a deeper understanding of the inherent characteristics of mirror descent across a wide range of potential functions.\n\nSTRENGTHS:\n1. The result of this work is solid, containing the general class of potential functions, and the authors derive a calculation rule of $\\phi_\\infty$ for a general class of potential functions.\n2. The paper is informative and well-structured, particularly in section 3. By using the example of gradient descent within the framework established by the preceding lemmas, which is a special case of mirror descent, the authors clearly outline the reasons why mirror flow will eventually converge in direction without complicated formulas.\n\nWEAKNESSES:\n1. Since mirror descent is not so popular in the practice of machine learning problems,  there could be more discussion about the implications of their results. For example, Figure 1 is really interesting as it reveals that the mirror descent shares the same structure of implicit bias with the steepest descent [1], what is the essence of such similarities?\n2. The setting of an infinitely small learning rate, i.e., optimization flow,  might be a little strong under a simple linear classification problem compared to the previous works. I suggest the authors state the technical challenges of the discrete optimization process of mirror descent.\n3. I might be wrong, but it seems not strict to apply the Bolzano–Weierstrass theorem to an uncountably infinite set at page 5, line 160 and 61.\n\n[1] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pages 1832–1841. PMLR, 2018.\n\nQUESTIONS:\n1. I'm really curious about whether this result could be extended to the q-homogenous neural networks like [1], cause it seems this result was also derived from the KKT condition of the lagrangian function.\n2. I guess there might be no minus in line 461 for the first equality.\n3. Could the author explain why they need to prove the uniqueness of the flow solution in Lemma 2, since this was not covered in the previous work, like [1][2]. Moreover, why do the authors need to prove that $\\int_0^t a(\\beta_s)ds \\to \\infty$ for Lemma 1.\n4. What is the definition of $h_\\infty$ in line 535 and $h(\\cdot)$? Moreover, could the author explain more about how to derive formula (8) ?\n\n[1] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020.\n\n[2] B.Wang, Q. Meng,W. Chen, and T.-Y. Liu. The implicit bias for adaptive optimization algorithms on homogeneous\nneural networks. In International Conference on Machine Learning, pages 10849–10858. PMLR, 2021.\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the implicit bias of mirror flow (the continuous-time counterpart of mirror descent) in the context of binary classification for linearly separable data. Given that the problem has infinitely many solutions, obtained at infinity, the authors aim to identify which solution is achieved through mirror flow. Assuming an exponential tail on the loss function, the authors demonstrate that mirror flow converges directionally to a maximum margin classifier, where the margin is characterized by a horizon function of the mirror potential. This result extends many existing works, and numerical experiments are provided to verify the findings.\n\nSTRENGTHS:\n1. The paper is well-written and well-organized. Despite the technical nature of the analysis and results, the paper is relatively easy to follow.\n2. The paper is also well-motivated. Although mirror descent is not commonly used as an algorithm for training neural networks, analyzing its convergence is valuable for understanding the implicit bias of gradient descent in various neural network architectures.\n3. I did not verify the details of the proof. However, the paper provides several motivating examples, including the quadratic potential corresponding to gradient flow, which makes the results quite convincing.\n4. The main results extend several prior works.\n\nWEAKNESSES:\nAlthough the authors have stated that the convergence rate is left for future study, it would be beneficial to provide at least empirical evidence of the convergence rate. The authors mentioned in line 294 that the convergence rate varies across different potentials.\n\nQUESTIONS:\n1. In the main result, Theorem 2, the conclusion that the normalized mirror flow converges to a vector \\(\\bar{\\beta}_{\\infty}\\) is drawn even when the \\(\\phi_{\\infty}\\)-max-margin problem does not necessarily have a unique solution. Can the authors provide more insight on this? If I understand correctly, in the motivating example, \\(\\bar{\\beta}_{\\infty}\\) is a subsequence limit. If the \\(\\phi_{\\infty}\\)-max-margin problem does not have a unique solution, this result cannot be extended to the whole limit, unlike in the gradient flow case with quadratic potential.\n2. More explanation could be provided for Figure 1. In particular, it would be interesting to see the trajectories of the mirror flows on the plane, rather than only showing the limit.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This manuscript examines the implicit bias of mirror descent on a classification problem when the dataset is linearly separable. Assuming a coercive gradient, it demonstrates that the implicit bias is characterized by the shape of the level set of the mirror potential near infinity. Their analysis successfully recovers existing results for p-norm potentials and identifies the implicit bias of the potentials emerging in the analysis of linear neural networks. Additionally, it leaves the characterization of the implicit bias when the gradient is not coercive as an interesting open problem.

STRENGTHS:
I think the paper is very well-written and has a solid contribution.

It addresses an important problem, aiming to understand the implicit bias of neural networks. Prior work has shown that the dynamics of linear networks can be characterized by mirror descent, highlighting the relevance of this study.

WEAKNESSES:
NA

QUESTIONS:
In line 123, the authors suggested that the logistic loss satisfies the conditions in Assumption 1. However, it is clear that the logistic loss does not have an exponential tail. Could they clarify whether this is a mistake or if there is an underlying argument supporting their claim?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the asymptotic behaviour of the mirror descent (MD) algorithm for a linear classification task. It is shown that the classifier (hyperplane orthogonal to $\\beta$) will be a max-margin classifer, where the margin is determined by some unknown horizon function $\\phi_\\infty$. This works extend prior work which consider $\\ell_p$ and homogeneous potential functions for MD, and shows this result for very general $\\phi$.\n\nSTRENGTHS:\nThe paper makes an interesting statement about behaviour of mirror descent on classification tasks, will minimal assumptions. In doing so, it takes a big step and extends previous work to the cover general potential functions.\nThe paper is well written and the figures help with understanding the concepts of convergence.\n\n---\n*While I could understand the paper, this is not my area of research. I do not find myself fit to evaluate the paper on soundness, relevance to the sub-field and importance of contributions.*\n\nWEAKNESSES:\n- The paper does not characterize $\\phi_\\infty$ in terms of the bregman potential $\\phi$ (and other relevant entities). \nThe main result expresses that there exists some function, that is minimized by $\\bar \\beta_\\infty$, the direction of the classifier as $t\\rightarrow \\infty$. \nI think this limits the relevance and strength of the result. For instance, this does not help with interpretability compared to the case where we can prove the optimization algorithm converging to a max-margin classifier wrt the $\\ell_2$ norm.\n\n- I am not sure about relevance and use-cases of the mirror descent algorithm with very general potentials. As far as I know, typically, a small set of norm-based or entropic (neg-ent, tsallis, etc) are used within applications of ML. So while the theorem makes an interesting statement for an optimization standpoint, I'm not sure how relevant it is for the ML community. The theorem is also not entirely relevant to the pure optimization community since it's for the specific case of linear classification with finite data.\n---\n*While I could understand the paper, this is not my area of research. I do not find myself fit to evaluate the paper on soundness, relevance to the sub-field and importance of contributions.*\n\nQUESTIONS:\n- Are there any classes of potential functions (other than the norms and $L$-homogeneous ones), for which $\\phi_\\infty$ may be calculated or approximated?\n\n- Beyond gradient descent, is there any work that quantifies the rates of convergence to max-margin classifiers? Is this even possible?\n\nLIMITATIONS:\nThe limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}