Title: Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability
Venue: NeurIPS 2023
Accepted: yes
Keywords: gd; implicit bias; edge of stability
Abstract: Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen et al., 2021], where the stepsizes are set to be large, resulting in non-monotonic losses induced by the GD iterates. This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime. Despite the presence of local oscillations, we prove that the logistic loss can be minimized by GD with any constant stepsize over a long time scale. Furthermore, we prove that with any constant stepsize, the GD iterates tend to infinity when projected to a max-margin direction (the hard-margin SVM direction) and converge to a fixed vector that minimizes a strongly convex potential when projected to the orthogonal complement of the max-margin direction. In contrast, we also show that in the EoS regime, GD iterates may diverge catastrophically under the exponential loss, highlighting the superiority of the logistic loss. These theoretical findings are in line with numerical simulations and complement existing theories on the convergence and implicit bias of GD for logistic regression, which are only applicable when the stepsizes are sufficiently small.

Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability

Abstract

Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen et al., 2021], where the stepsizes are set to be large, resulting in non-monotonic losses induced by the GD iterates. This paper studies the convergence and implicit bias of constantstepsize GD for logistic regression on linearly separable data in the EoS regime. Despite the presence of local oscillations, we prove that the logistic loss can be minimized by GD with any constant stepsize over a long time scale. Furthermore, we prove that with any constant stepsize, the GD iterates tend to infinity when projected to a max-margin direction (the hard-margin SVM direction) and converge to a fixed vector that minimizes a strongly convex potential when projected to the orthogonal complement of the max-margin direction. In contrast, we also show that in the EoS regime, GD iterates may diverge catastrophically under the exponential loss, highlighting the superiority of the logistic loss. These theoretical findings are in line with numerical simulations and complement existing theories on the convergence and implicit bias of GD for logistic regression, which are only applicable when the stepsizes are sufficiently small.

1 Introduction

Gradient descent (GD) is a foundational algorithm for machine learning optimization that motivates many popular algorithms. Theoretically, the behavior of GD is well understood when the stepsize is small. In this regard, one of the most classic results is the descent lemma (see, e.g., Section 1.2.3 in Nesterov et al. [2018]):

Lemma (Descent lemma, simplified version) . Suppose that sup w ∥ ∥ ∇ 2 L ( w ) ∥ ∥ 2 ≤ β 1 , then

When the targeted function is smooth (such as logistic regression) and the stepsize is small ( 0 < η < β/ 2 ), the descent lemma ensures a monotonic decrease of the function value by performing each GD step. Building upon this, a sequence of iterates produced by GD with small stepsizes provably minimizes the function value in various settings (see, e.g., Lan [2020]).

For a more modern example, a recent line of research has established the implicit bias of GD with small stepsizes (see Soudry et al. [2018], Ji and Telgarsky [2018] and references thereafter). Specifically, they consider GD for optimizing logistic regression (besides other loss functions) on

90

0

200

400

600

800

1000

iteration

10

3

10

2

10

1

10 0

10 1

10 2

training loss

=0.1

=0.01

0

50

100

150

200

250

300

iteration

0

25

50

75

100

125

150

175

200

sharpness

=0.1

=0.01

0

200

400

600

800

1000

iteration

60

65

70

75

80

85

test accuracy

=0.1

=0.01

(a) Training loss

(b) Sharpness

(c) Test accuracy

linearly separable data. When the stepsizes are sufficiently small, the GD iterates are shown to decrease the risk monotonically (by a variant of the descent lemma); moreover, the GD iterates tend to align with a direction that maximizes the ℓ 2 -margin of the data [Soudry et al., 2018, Ji and Telgarsky, 2018]. The margin-maximization bias of small-stepsize GD sheds important light on understanding the statistical benefits of GD, as a large margin solution often generalizes well [Bartlett et al., 2017, Neyshabur et al., 2018].

Nonetheless, in practical machine learning optimization, especially in deep learning, the empirical risk (or training loss) often varies non-monotonically (while being minimized in the long run) - the local risk oscillation is not only caused by the algorithmic randomness but is more an effect of using large stepsizes , as it happens for deterministic GD (with large stepsizes) as well [Wu et al., 2018, Xing et al., 2018, Lewkowycz et al., 2020, Cohen et al., 2021]. This phenomenon is showcased in Figures 1(a) and 2(a), and is referred to by Cohen et al. [2021] as the edge of stability (EoS). The observation sets a non-negligible gap between practical and theoretical GD setups, where in practice, GD is run with large stepsizes that lead to local risk oscillations, but in theory, GD is only considered with sufficiently small stepsizes, predicting a monotonic risk descent (with a few exceptions, which will be discussed later in Section 2). A tension remains to be resolved:

Is the convergence of risk under local oscillation merely a 'lucky' occurrence, or is it predictable based on theory?

Contributions. In this work, we study the behaviors of GD in the EoS regime in arguably the simplest setting for machine learning optimization - logistic regression on linearly separable data. We show that with any constant stepsize, while the induced risks may oscillate locally, GD must minimize the risk in the long run at a rate of O (1 /t ) , where t is the number of iterates. In addition, we show that the direction of the GD iterates (with any constant stepsize) must align with a max-margin direction (the hard-margin SVM direction) at a rate of O (1 / log( t )) . These results explain how GD minimizes a risk non-monotonically, and complement existing theories [Soudry et al., 2018, Ji and Telgarsky, 2018] on the convergence and implicit bias of GD, which are only applicable when the stepsizes are sufficiently small.

Some additional notable contributions are

We also show that, when projected to the orthogonal complement of the max-margin direction, the GD iterates (with any constant stepsize) converge to a fixed vector that minimizes a strongly convex potential at a rate of O (1 / log( t )) . This characterization is conceptually more interpretable than an existing version [Soudry et al., 2018].

We show that in the EoS regime, GD can diverge catastrophically if the logistic loss is replaced by the exponential loss. This is in stark contrast to the small-stepsize regime, where the behaviors

of GD are known to be similar under any exponentially-tailed losses including both the logistic and exponential losses [Soudry et al., 2018, Ji and Telgarsky, 2018]. The difference in the EoS regime provides insights into why the logistic loss is preferable to the exponential loss in practice.

From a technical perspective, we develop a new approach for analyzing GD with large stepsizes. Our approach views the GD iterates as a coupling of two orthogonal iterates, one along a maxmargin direction and the other along the orthogonal complement of the max-margin direction. The former iterates tend to infinity and the latter iterates approximate 'imaginary' GD iterates that minimize a strongly convex potential with a decaying stepsize scheduler, controlled by the former iterates. Our techniques for analyzing large-stepsize GD can be of independent interest.

2 Related Works

In this section, we discuss papers related to our work.

Implicit bias. We first review a set of papers on the implicit bias of GD (with small stepsizes).

Along this line, Soudry et al. [2018] are the very first to show that GD converges along a max-margin direction when minimizing the empirical risk of an exponentially-tailed loss function (such as the logistic and exponential losses), a linear model, and linearly separable data. Then, an alternative analysis is provided by Ji and Telgarsky [2018], which also deals with non-separable data. These two works directly motivate us for considering GD for logistic regression on linearly separable data. However, there are at least three notable differences between our work and theirs. Firstly, their results only apply to GD with small stepsizes, while our results apply to GD with any constant stepsize. Secondly, their theories predict no difference between the logistic and exponential losses (as they are limited to the small-stepsize regime). Quite surprisingly, we prove that in the EoS regime, GD can diverge catastrophically under the exponential loss. Thirdly, from a technical viewpoint, their implicit bias analysis is built upon the risk convergence analysis, which further relies on a monotonic risk descent argument, hence only applies to small stepsizes. In comparison, we come up with a new approach that allows analyzing the implicit bias under risk oscillations; the long-term risk convergence is simply a consequence of the implicit bias results. Hence our techniques can accommodate any constant stepsize. See Section 5 for more discussions.

Subsequent works have extended the results by Soudry et al. [2018], Ji and Telgarsky [2018] to other algorithms such as momentum-based GD [Gunasekar et al., 2018a, Ji et al., 2021] and SGD [Nacson et al., 2019c], and homogenous but non-linear models [Gunasekar et al., 2017, Ji and Telgarsky, 2019, Gunasekar et al., 2018b, Nacson et al., 2019a, Lyu and Li, 2020] and non-homogenous models [Nacson et al., 2019a]. All these theories require the stepsizes to be small or even infinitesimal, in a regime away from our focus, the EoS regime.

It is worth noting that Nacson et al. [2019b] consider GD with an increasing stepsize scheduler that achieves a faster margin-maximization rate than constant-stepsize GD. However, their stepsize at each iteration is still appropriately small, resulting in a monotonic risk descent by a variant of the descent lemma.

Edge of stability. The risk oscillation phenomenon has been observed in several deep learning papers [Wu et al., 2018, Xing et al., 2018, Lewkowycz et al., 2020], and the work by Cohen et al. [2021] coins the term, edge of stability (EoS), that formally refers to it. In the remainder of this part, we focus on reviewing the current theoretical progress in understanding EoS.

Zhu et al. [2023] rigorously characterize EoS for a two-dimensional function ( u, v ) ↦→ ( u 2 v 2 -1) 2 . Chen and Bruna [2022] study EoS for a one-dimensional function u ↦→ ( u 2 -1) 2 and for a special two-layer single-neuron network. Similar to these two works, Kreisler et al. [2023] study EoS in a 1-dimensional linear network. Ahn et al. [2022a] consider functions ( u, v ) ↦→ ℓ ( uv ) , where ℓ is assumed to be convex, even, and Lipschitz; notably, they show a statistical gap between the small-stepsize regime and the EoS regime. Finally, Even et al. [2023], Andriushchenko et al. [2023] consider the regularization effect of large stepsizes in a diagonal linear network. Compared to their settings, our problem, i.e., logistic regression, is a natural machine-learning problem with fewer artifacts (if any).

EoS has also been theoretically investigated for general functions [Ma et al., 2022, Ahn et al., 2022b, Damian et al., 2022, Wang et al., 2022b], but these theories are often subject to subtle assumptions that are hard to interpret or verify. Specifically, Ma et al. [2022] require the function to grow

'subquadratically'. Ahn et al. [2022b] assume the existence of a 'forward invariant subset' near the set of minima of the function. Damian et al. [2022] assume a negative correlation between the gradient direction and the largest eigenvalue direction of the Hessian. Wang et al. [2022b] consider a two-layer neural network but require the norm of the last layer parameter and the sharpness to change in the same direction along the GD trajectory. Indirectly connected to EoS, the work by Kong and Tao [2020] shows a chaotic behavior of GD with a non-small stepsize when optimizing a 'multi-scale' loss function. In comparison, our assumptions are more natural and interpretable.

Besides, the work by Lyu et al. [2022] considers EoS induced by GD for scale-invariant loss, e.g., a network with normalization layers and weight decay, and the work by Wang et al. [2022a] shows a balancing effect in matrix factorization induced by GD with a constant stepsize that is nearly 4 / ∥∇ 2 L ( w 0 ) ∥ 2 and is larger than 2 / ∥∇ 2 L ( w 0 ) ∥ 2 . The objectives in their works are different from ours, i.e., logistic regression.

The unstable convergence has also been studied for normalized GD [Arora et al., 2022] and regularized GD [Bartlett et al., 2022]. These algorithms are apart from our focus on the vanilla GD.

Finally, the work by Liu et al. [2023] considers logistic regression with non-separable data (such that the objective is strongly convex), where GD with sufficiently large stepsize diverges. In contrast, we consider logistic regression with separable data, where GD with an arbitrarily large stepsize still converges.

3 Preliminaries

We use x ∈ R d to denote a feature vector and y ∈ {± 1 } to denote a binary label, respectively. Let ( x i , y i ) n i =1 be a set of training data. Throughout the paper, we assume that ( x i , y i ) n i =1 is linearly separable [Soudry et al., 2018].

Assumption 1 (Linear separability) . Assume there is w ∈ R d such that y i x ⊤ i w > 0 for i = 1 , . . . , n .

Let w ∈ R d be the parameter of a linear model. In logistic regression , we aim to minimize the following empirical risk

We study a sequence of iterates ( w t ) t ≥ 0 produced by constant-stepsize gradient descent (GD), where w 0 denotes the initialization and the remaining iterates are sequentially generated by:

where η > 0 is a constant stepsize. We are especially interested in a regime where η is very large such that L ( w t ) oscillates as a function of t . For the simplicity of presentation, we will assume that w 0 = 0 . Our results can be easily extended to allow general initialization.

The following notations are useful for presenting our results.

Definition 1 (Margins and support vectors) . Under Assumption 1, define the following notations:

(A) Let γ be the maxℓ 2 -margin (or max-margin in short), i.e.,

(B) Let ˆ w be the hard-margin support-vector-machine (SVM) soluion, i.e.,

It is clear that ˆ w exists and is uniquely defined (see, e.g., Section 5.2 in Mohri et al. [2018]). Note that ∥ ˆ w ∥ 2 = 1 /γ and ˆ w / ∥ ˆ w ∥ 2 is a max-margin direction. Also note that by duality, ˆ w can be written as (see, e.g., Section 5.2 in Mohri et al. [2018])

(C) Let S be the set of indexes of the support vectors, i.e.,

(D) If there exists non-support vector ( S ⊊ [ n ] ), let θ be the second smallest margin, i.e.,

It is clear from the definitions that θ > γ > 0 . In addition, from the definitions we have

In addition to Assumption 1, we make the following two mild assumptions to facilitate our analysis.

Assumption 2 (Regularity conditions) . Assume that:

Assumption 2 is only made for the convenience of presentation. In particular, Assumption 2(A) can be made true for any dataset by scaling the data vectors with a factor of max i ∥ x i ∥ 2 . Without Assumption 2(B), our theorems still hold under a minor revision by replacing all the vectors of interests with their projections to span { x i , i = 1 , . . . , n } .

Assumption 3 (Non-degenerate data) . In addition to Assumption 1, assume that

Assumption 3 has been used in Soudry et al. [2018] (see their Theorem 4), which requires that the support vectors span the dataset and are associated with strictly positive dual variables. Assumption 3(B) holds almost surely for every linearly separable dataset sampled from a continuous distribution according to Appendix B in Soudry et al. [2018]. Assumption 3 provides convenience to our analysis, but we conjecture it might not be necessary. Removing/relaxing Assumption 3 is left as a future work.

3.1 Space Decomposition

Conceptually, our analysis is built on a novel space decomposition viewpoint, which relies on the following lemma.

Lemma 3.1 (Non-separable subspace) . Suppose that Assumptions 1, 2, and 3 hold. Then ( x i , y i ) i ∈S is not linearly separable in the subspace orthogonal to the max-margin direction ˆ w / ∥ ˆ w ∥ 2 . That is, for every v such that ⟨ v , ˆ w ⟩ = 0 , there exist i, j ∈ S such that y i · ⟨ x i , v ⟩ < 0 , y j · ⟨ x j , v ⟩ > 0 .

Proof of Lemma 3.1. By Assumption 3 and ⟨ v , ˆ w ⟩ = 0 , we have

By Assumptions 2 and 3 we have

̸

so there must exist i ∈ S such that y i x ⊤ i v = 0 . Without loss of generality, assume that y i x ⊤ i v < 0 . Then since α i > 0 for i ∈ S by Assumption 3, there must exist j ∈ S such that y j x ⊤ j v > 0 .

Lemma 3.1 shows that, although the dataset can be (linearly) separated by ˆ w , it cannot be separated by any vector orthogonal to ˆ w . This motivates us to decompose the d -dimensional ambient space into a 1 -dimensional 'separable' subspace and a ( d -1) -dimensional 'non-separable' subspace. This idea is formally realized as follows.

Fix d -1 orthogonal vectors f 1 , . . . , f d -1 ∈ R d such that ( ˆ w / ∥ ˆ w ∥ 2 , f 1 , . . . , f d -1 ) forms an orthogonal basis of the ambient space R d . Then define two projection operators :

The two operators together define a natural space decomposition, i.e., R d = P ( R d ) ⊕ P ( R d ) . Moreover, ( P ( x i ) , y i ) n i =1 are linearly separable with an maxℓ 2 -margin γ according to Definition 1, and ( ¯ P ( x i ) , y i ) i ∈S (hence ( ¯ P ( x i ) , y i ) n i =1 ) are non-separable according to Lemma 3.1. So the decomposition of space can also be understood as the decomposition of data features into 'maxmargin features' and 'non-separable features'.

In what follows, we will call P ( R d ) the max-margin subspace and ¯ P ( R d ) the non-separable subspace , respectively. In addition, we define a 'margin offset' that quantifies to what extent the 'non-separable features' are not separable.

Definition 2 (Margin offset for the non-separable features) . Under Assumptions 1, 2, and 3, it holds that ( ¯ P ( x i ) , y i ) i ∈S is non-separable. Let b be a margin offset such that

Then b > 0 due to the non-separability. The definition immediately implies that:

Comparison to Ji and Telgarsky [2018]. The work by Ji and Telgarsky [2018] also conducts space decomposition (see their Section 2). However, our approach is completely different from theirs. Firstly, they consider a non-separable dataset but we consider a linearly separable dataset. Secondly, at a higher level, they decompose the 'dataset' (into two subsets), while we decompose the 'features' (into two kinds of features). More specifically, Ji and Telgarsky [2018] first group the non-separable dataset into the 'maximal linearly separable subset' and the complement, non-separable subset, then decompose the ambient space according to the subspace spanned by the non-separable subset and its orthogonal complement. In comparison, we consider a linearly separable dataset and decompose the ambient space according to a max-margin direction (i.e., P ) and its orthogonal complement (i.e., ¯ P ).

4 Main Results

We are now ready to present our main results. All proofs are deferred to Appendix C. To begin with, we provide the following theorem that captures the behaviors of constant-stepsize GD for logistic regression on linearly separable data.

Theorem 4.1 (The implicit bias of GD for logistic regression) . Suppose that Assumptions 1, 2, and 3 hold. Consider ( w t ) t ≥ 0 produced by ( GD ) with initilization 2 w 0 = 0 and constant stepsize η > 0 . Then there exist positive constants c 1 , c 2 , c 3 > 0 that are upper bounded by a polynomial of { e η , e n , e 1 /b , 1 /η, 1 / ( θ -γ ) , 1 /γ, e θ/γ } but are independent of t , such that:

(A) The risk is upper bounded by

(B) In the max-margin subspace,

(C) In the non-separable subspace,

(D) In addition, in the non-separable subspace,

where G ( · ) is a strongly convex potential defined by

0

25

50

75

100

125

150

175

200

iteration

10

12

10

10

10

8

10

6

10

4

10

2

10 0

10 2

10 4

risk

=0.01

=0.1

=10

=10 2

=10 3

0

10

20

30

40

50

iteration

10

8

10

6

10

4

10

2

10 0

10 2

sharpness

=0.01

=0.1

=10

=10 2

=10 3

(a) Empirical risk

(b) Sharpness

Note that Theorem 4.1 applies to GD with any positive constant stepsize, therefore allowing GD to be in the EoS regime. We next discuss the implications of Theorem 4.1 in detail.

Risk minimization. Theorem 4.1(A) guarantees that the GD iterates minimize the logistic loss at a rate of O (1 /t ) for any constant stepsize, even for those large stepsizes that cause local risk oscillations. This result explains the risk convergence of GD in the EoS regime, as illustrated in Figure 2, and is also consistent with the observations in neural network experiments (see Figure 1).

Margin maximization. Theorem 4.1(B) shows that the GD iterates, when projected to the maxmargin direction, tend to infinity at a rate of O (log( t )) . Moreover, Theorem 4.1(C) shows that the GD iterates, when projected to the non-separable subspace, are uniformly bounded. These two results together imply that the direction of the GD iterates will tend to a max-margin direction, i.e., the hard-margin SVM direction, at a rate of O (1 / log( t )) . Therefore, the implicit bias of GD that maximizes the ℓ 2 -margin is consistent in both the EoS regime and the small-stepsize regime [Soudry et al., 2018, Ji and Telgarsky, 2018].

Iterate convergence in the non-separable subspace. Theorem 4.1(D) shows that the GD iterates, when projected to the non-separable subspace, converge to the minimizer of a strongly convex potential G ( · ) . Here, G ( · ) measures the exponential loss of a parameter on the support vectors with their non-separable features. This provides a more precise characterization of the implicit bias of GD: the direction of the GD iterates converges to the hard-margin SVM direction, moreover, the limit of the projections of the GD iterates to the orthogonal complement to the hard-margin SVM direction minimizes the exponential loss on the non-separable features of the support vectors.

Comparison to Theorem 9 in Soudry et al. [2018]. Theorem 9, in particular, equation (18), in Soudry et al. [2018] indirectly characterizes the convergence of GD iterates in the non-separable subspace. It reads in our notations that: ˜ w := lim t →∞ ( w t -ˆ w log( t ) ) exists and satisfies

In Appendix A, we show that Theorem 4.1(D) is equivalent to condition (1) in terms of describing ¯ P ( w ∞ ) . Despite their equivalence, (1) is less interpretable than Theorem 4.1(D), as (1) entangles an effect of P ( w ∞ ) with ¯ P ( w ∞ ) , while Theorem 4.1 completely decouples P ( w ∞ ) and ¯ P ( w ∞ ) . In particular, (1) seems to suggest ¯ P ( w ∞ ) to be a function of stepsize η since ˜ w depends on η . However, this is only an illusion brought by the lack of interpretability of (1); it is clear that ¯ P ( w ∞ ) is independent of η according to Theorem 4.1(D).

Exponential loss. Until now, our theory for GD is consistent for large and small stepsizes. However, this is a particular benefit thanks to the design of the logistic loss, and may not hold for other losses. Our next result suggests that, in the EoS regime where the stepsizes are large, GD can diverge catastrophically under the exponential loss.

Theorem 4.2 (The catastrophic divergence of GD under the exponential loss) . Consider a dataset of two samples, where

It is clear that ( x i , y i ) i =1 , 2 is linearly separable and (1 , 0) is the max-margin direction. Consider a risk defined by the exponential loss:

Let ( w t , ¯ w t ) t ≥ 0 be the iterates produced by GD with constant stepsize η for optimizing L ( w, ¯ w ) . If

then:

(A) L ( w t , ¯ w t ) →∞ .

(B) w t →∞ .

(C) For every t ≥ 0 , | ¯ w t | ≥ 2 γw t .

(D) Moreover, the sign of ¯ w t flips every iteration.

As a consequence, ( w t , ¯ w t ) t ≥ 0 diverge in terms of either magnitude or direction; in particular, the direction of ( w t , ¯ w t ) t ≥ 0 cannot converge to the max-margin direction (which is (1 , 0) ).

Theorem 4.2 shows that with a large constant stepsize, the GD iterates no longer minimize the risk defined by the exponential loss and no longer converge along the max-margin direction. In fact, the directions of the GD iterates flip every step, thus the direction of the GD iterates necessarily diverges , resulting in no meaningful implicit bias at all.

In the EoS regime, large-stepsize GD still behaves nicely under the logistic loss (Theorem 4.1) but can behave catastrophically under the exponential loss (Theorem 4.2). From a mathematical standpoint, this difference is rooted in the fact that the gradient of the logistic loss is uniformly bounded while the gradient of the exponential loss could be extremely large. From a practical standpoint, it provides insights into why the logistics loss (and its multi-class version, the cross-entropy loss) is preferable to the exponential loss in practice.

The different behaviors of large-stepsize GD under the logistic and exponential losses also sharply contrast the EoS regime with the small-stepsize regime. Because in the small-stepsize regime, the convergence and implicit bias of GD are known to be similar under any exponentially-tailed losses, including the logistic and exponential losses [Soudry et al., 2018, Ji and Telgarsky, 2018].

5 Techniques Overview

The proofs of Theorems 4.1 and 4.2 are deferred to Appendix C. In this section, we explain the proof ideas of Theorem 4.1 by analyzing a simple dataset considered in Theorem 4.2 (the treatment to the general datasets can be found in Appendix B). But this time we work with the logistic loss instead of the exponential loss, that is,

Then the GD iterates can be written as

where

For simplicity, assume that

Different from Soudry et al. [2018], Ji and Telgarsky [2018], our approach begins with showing the implicit bias (despite that the risk may oscillate). The long-term risk convergence is then simply a consequence of the implicit bias results.

Step 1: ( ¯ w t ) t ≥ 0 is uniformly bounded. Observe that ¯ g t and ¯ w t always share the same sign and that | ¯ g t | ≤ 1 , so we have

By induction, we get that ( | ¯ w t | ) t ≥ 0 is uniformly bounded by max {| ¯ w 0 | , η } = Θ(1) .

Step 2: w t ≈ log( t ) /γ . We turn to study the max-margin subspace. It is clear that g t ≤ 0 for every t ≥ 0 . So we have w t ≥ 0 by induction. Moreover, we have

where the last inequality is because | ¯ w t | is uniformly bounded. We also have

where the third inequality is because e -¯ w t + e ¯ w t ≥ 1 and the last equality is because w t ≥ 0 . Putting these together, we have

Step 3: ¯ g t ≈ exp( -γw t ) · ∇ G ( ¯ w t ) . We turn back to the non-separable subspace. Note that ¯ g t is an odd function of ¯ w t . Without loss of generality, let us assume ¯ w t ≥ 0 in this part. Notice that

Then we have

where the inequality is by (3), and G ( ¯ w ) := e ¯ w + e -¯ w is defined as in Theorem 4.1(D). On the other hand, since | ¯ w t | is bounded and w t is increasing (and tends to infinity), there must exist a time t 0 such that e -γw t ≤ e -| ¯ w t | for every t ≥ t 0 . Then for t ≥ t 0 we have

where the first inequality is by (3) and e -γw t ≤ e -¯ w t , and the last inequality is because we assume ¯ w t ≥ 0 . Putting these together, and using (2), we obtain that

Step 4: a modified descent lemma. Using (4) and Taylor's expansion, we have

Step 5: the convergence of ¯ w t . What remains is adapted from classic convergence arguments. Choose ¯ w ∗ = arg min G ( · ) , then

where the equality is by (4), and the inequality is because of the convexity of G ( · ) , | ¯ w t | ≤ Θ(1) , and (4). Taking a telescoping sum, we have

Combing the above with (5) and using η t ≈ Θ(1) /t from (4), we get

Finally, since ∑ T t = t 0 η t ≥ Θ(1) · (log( T ) -log( t 0 )) according to (4), we get that G ( ¯ w T ) -G ( ¯ w ∗ ) ≤ Θ(1) / (log( T ) -log( t 0 )) .

Step 6: risk convergence. The long-term risk convergence result can be easily established by making use of the implicit bias results we have obtained so far.

6 Conclusion

We consider constant-stepsize GD for logistic regression on linearly separable data. We show that with any constant stepsize, GD minimizes the logistic loss; moreover, the GD iterates tend to infinity when projected to a max-margin direction and tend to a fixed minimizer of a strongly convex potential when projected to the orthogonal complement of the max-margin direction. We also show that GD with a large stepsize may diverge catastrophically if the logistic loss is replaced by the exponential loss. Our theory explains how GD minimizes a risk non-monotonically.

Acknolwdgement

We thank the anonymous reviewers for their helpful comments and Alexander Tsigler for pointing out several typos. VB is partially supported by the Ministry of Trade, Industry and Energy(MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the International Cooperative R&D program. JDL acknowledges the support of the ARO under MURI Award W911NF-11-1-0304, the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, NSF CIF 2212262, ONR Young Investigator Award, and NSF CAREER Award 2144994.

References

Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the" edge of stability". arXiv preprint arXiv:2212.07469 , 2022a.

Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning , pages 247-257. PMLR, 2022b.

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning , pages 903-925. PMLR, 2023.

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning , pages 948-1024. PMLR, 2022.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Peter L Bartlett, Philip M Long, and Olivier Bousquet. The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. arXiv preprint arXiv:2210.01513 , 2022.

Lei Chen and Joan Bruna. On gradient descent convergence beyond the edge of stability. arXiv preprint arXiv:2206.04172 , 2022.

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=jh-rTtvkGeM .

Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022. URL https://openreview.net/forum?id=enoU_Kp7Dz .

Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s) gd over diagonal linear networks: Implicit regularisation, large stepsizes and edge of stability. arXiv preprint arXiv:2302.08982 , 2023.

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning , pages 1832-1841. PMLR, 2018a.

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems , 31, 2018b.

Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300 , 2018.

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/ forum?id=HJflg30qKX .

Ziwei Ji, Nathan Srebro, and Matus Telgarsky. Fast margin maximization via dual acceleration. In International Conference on Machine Learning , pages 4860-4869. PMLR, 2021.

Lingkai Kong and Molei Tao. Stochasticity of deterministic gradient descent: Large learning rate for multiscale objective function. Advances in Neural Information Processing Systems , 33:2625-2638, 2020.

Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 17684-17744. PMLR, 23-29 Jul 2023.

Guanghui Lan. First-order and stochastic optimization methods for machine learning , volume 1. Springer, 2020.

Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218 , 2020.

Chunrui Liu, Wei Huang, and Richard Yi Da Xu. Implicit bias of deep learning in the large learning rate phase: A data separability perspective. Applied Sciences , 13(6):3961, 2023.

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations , 2020. URL https://openreview. net/forum?id=SJeLIgBKPS .

Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=xp5VOBxTxZ .

Chao Ma, Daniel Kunin, Lei Wu, and Lexing Ying. Beyond the quadratic approximation: The multiscale structure of neural network loss landscapes. Journal of Machine Learning , 1(3):247267, 2022. ISSN 2790-2048. doi: https://doi.org/10.4208/jml.220404. URL http://global-sci. org/intro/article_detail/jml/21028.html .

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning , pages 4683-4692. PMLR, 2019a.

Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3420-3428. PMLR, 2019b.

Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3051-3059. PMLR, 2019c.

Yurii Nesterov et al. Lectures on convex optimization , volume 137. Springer, 2018.

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=Skz_WfbCZ .

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1): 2822-2878, 2018.

Yuqing Wang, Minshuo Chen, Tuo Zhao, and Molei Tao. Large learning rate tames homogeneity: Convergence and balancing effect. In International Conference on Learning Representations , 2022a. URL https://openreview.net/forum?id=3tbDrs77LJ5 .

Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along GD trajectory: Progressive sharpening and edge of stability. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022b. URL https://openreview.net/forum?id=thgItcQrJ4y .

Lei Wu, Chao Ma, et al. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems , 31, 2018.

Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770 , 2018.

Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamics with a minimalist example. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=p7EagBsMAEO .

A On the Equivalence between Theorem 4.1(D) and (1)

Note that ˜ w in (1) contains components in both the max-margin and non-separable subspaces, and we need to disentangle those two components.

Under the coordinate system that defines P and ¯ P , we can represent a vector v ∈ R d as

Then for i ∈ S , we have

So (1) is equivalent to

Recall that ∑ i ∈S α i = 1 /γ 2 according to Definition 1. So focusing on ¯ P , the above is equivalent to the following condition on ¯ P ( ˜ w ) :

Here we ignore a shared normalization factor.

Now, recall from Assumption 3(B) that ( α i ) i ∈S are such that

Note that as long as ∑ i ∈S α i = 1 /γ 2 , we have

Now consider ¯ P . Note that ¯ P ( ˆ w ) = 0 by the choice of ¯ P , then apply ¯ P on both sides of the above equation, we get

Under (7), (6) is equivalent to the following condition on ¯ P ( ˜ w ) :

which is precisely the first-order condition of

Hence we have shown that: the condition that ˜ w satisfies (1) is equivalent to the condition that ¯ P ( ˜ w ) minimizes the strongly convex potential G ( · ) .

B The Behaviors of Constant-Stepsize GD

B.1 Notation Simplifications

Without loss of generality, we assume that

Otherwise, we replace y i with 1 and x i with y i · x i , respectively, and the following analysis does not change.

Then the risk becomes

Rotating the hard-margin SVM solution. Note that the ( GD ) iterates (under linear models) are rotation equivariant. Specifically, let R be an orthogonal matrix, then applying R on both sides of ( GD ), we obtain

which is equivalent to the GD iterates under changes of variables, w ← Rw and x ← Rx .

Therefore, without loss of generality, we can apply a rotation to the dataset such that ˆ w ∥ e 1 . Then for v ∈ R d ,

Slightly abusing notations, in what follows we will write

Then we have

So the loss can be written as:

So ( GD ) can be written as:

The above two recursions capture the GD iterates projecting to the max-margin and non-separable subspaces, respectively.

B.2 Boundedness of GD in the Non-Separable Subspace

We first show that ( ¯ w t ) t ≥ 0 stay bounded for every fixed stepsize η .

Lemma B.1 (Positiveness of w t ) . Suppose that Assumption1 holds. Consider ( w t ) t ≥ 0 defined by (8) with constant stepsize η > 0 . Then for every t ≥ 0 , it holds that w t ≥ 0 .

Proof. Recall that

We only need to show that ∇ w L ( w, ¯ w ) ≤ 0 . This is because

where

Similarly, we define

Lemma B.2 (A recursion of ∥ ¯ w t ∥ 2 ) . Suppose that Assumptions 1, 2, and 3 hold. Consider ( ¯ w t ) t ≥ 0 defined by (8) with constant stepsize η > 0 . Then for every t ≥ 0 , there exists j ∈ [ n ] such that

As a direct consequence,

Proof. We first make a few useful notations. Fix a time index t .

Let k be the index of the 'most negatively classified' support sample, i.e.,

Let j be the index of the 'most negatively classified' sample, i.e.,

then by Definition 2 it holds that

Then

In particular, we must have since

We remark that it is possible that k = j .

Step 0: an iterate norm recursion. Recall that

Then

Step 1: gradient norm bounds. By definition, we have

Here, we use a property of logistic loss that the gradient is uniformly bounded. Therefore, we have

Step 2: cross-term bounds. We aim to show that the negative parts in the cross-term can cancel both the positve parts in the cross-term and the squared gradient norm term.

̸

Note that the following holds for either j = k or j = k :

The first term in (15) can be bounded by

The second term in (15) can be bounded by

since

Now, bringing (16), (17), and (18) into (15), we obtain

Here, we use a property of logistic loss that the gradients from incorrectly classified samples dominate the gradients from correctly classified samples.

Step 3: iterate norm recursion bounds. Using (14) and (19), we can obtain

We have completed the proof.

Lemma B.3 (Boundedness of ¯ w ) . Suppose that Assumptions 1, 2, and 3 hold. Consider ( ¯ w t ) t ≥ 0 defined by (8) with constant stepsize η > 0 . Then for every t ≥ 0 , it holds that

Proof. We prove the claim by induction. Clearly, ∥ ¯ w 0 ∥ 2 = 0 ≤ max { 4 n/b, ηn 2 /b } + ηn . Now suppose that

and discuss the following two cases:

If ∥ ¯ w t ∥ 2 ≤ max { 4 n/b, ηn 2 /b } , then by triangle inequality

Else, we have

which implies

This completes the induction.

B.3 Divergence of GD in the Max-Margin Subspace

Definition 3 (Some loss measurements in the non-separable subspace) . Under Assumptions 1, 2, and 3, we define the following notations:

(A) Define two loss functions

In the case where S = [ n ] , we define H ( ¯ w ) = 0 .

(B) Define

It is clear that G min ≥ 1 since (¯ x i ) i ∈S are non-separable by Definition 2.

(C) Define

It is clear that G ( ¯ w ∗ ) = G min . Moreover, it holds that ∥ ¯ w ∗ ∥ 2 ≤ W max by Lemma B.4.

(D) Recall that ∥ ¯ w t ∥ 2 ≤ W max according to Lemma B.3. We then define

It is clear that

and that G max , H max are polynomials on e η , e n , and e 1 /b , and are independent of t . Lemma B.4. For the ¯ w ∗ in Definition 3, it holds that

Proof. By Definition 2, there exists j ∈ S such that

On the other hand, by the definition of ¯ w ∗ , we have

Therefore, we have e b ·∥ ¯ w ∗ ∥ 2 ≤ n, that is, ∥ ¯ w ∥ ≤ log( n ) /b ≤ W

We now consider ( w t ) t ≥ 0 .

Lemma B.5. Suppose Assumptions 1, 2, and 3 hold. Then for every t ≥ 0 , it holds that

Proof. Recall that

We only need to provide upper and lower bounds on -∇ w L ( w t , ¯ w t ) . The lower bound is because:

The upper bound is because:

since

x

i

γ >

0

by Definition 1

since

e

t

/

(1 +

e

t

)

0

.

∗ 2 max .

which implies that

{

5 min

1

t

, e

}

Proof. Observe that

which implies that w t is increasing. Furthermore, we have

which implies that

We then get

We have completed the proof.

Lemma B.6 (A lower bound on w t ) . Suppose Assumptions 2, 1, and 3 hold. Then it holds that

As a direct consequence, it holds that

Lemma B.7 (An upper bound on w t ) . Suppose Assumptions 1, 2, and 3 hold. Then it holds that

As a direct consequence, it holds that

Proof. Observe that

Let

Recall that w t is increasing according to (20). So we have

(21) and (23) together imply that

Then for t ≥ t 0 , we have

which implies

Therefore, for t ≥ t 0 , we have

Note that the above also holds for 0 ≤ t ≤ t 0 according to (22). We have completed the proof.

B.4 Convergence of GD in the Non-Separable Subspace

We show that the vanilla gradient on the non-separable subspace, ∇ ¯ w L ( w t , ¯ w t ) , can be understood as the gradient on a modified loss with a rescaling factor, e -γw t ∇ G ( ¯ w t ) , ignoring higher order errors.

Lemma B.8 (Gradients comparison lemma) . Suppose Assumptions 1, 2, and 3 hold. Then it holds that

As a direct consequence, for every vector ¯ v ∈ R d -1 , it holds that

Proof. Recall that

By the triangle inequality, we have

The ( ♣ ) term can be bounded by

The ( ♡ ) term can be bounded by

by triangle inequality since ∥ ¯ x i ∥ 2 ≤ 1 by Assumption 2

Bringing the bounds on the ( ♣ ) and ( ♡ ) into (25), we obtain

We have shown the first conclusion. The second conclusion follows from the first conclusion: for every v ∈ R d -1 ,

We have completed the proof.

Lemma B.9 (A gradient norm bound) . Suppose Assumptions 1, 2, and 3 hold. Then it holds that

Proof. The inequality is because:

The next lemma shows that the function value is 'non-increasing' ignoring higher order terms. Lemma B.10 (A modified descent lemma) . Suppose Assumptions 1, 2, and 3 hold. Then it holds that

As a direct consequence of the above and Lemma B.6, it holds that

where θ/γ > 1 by Definition 1 and c 0 is a polynomial on { e η , e n , e 1 /b , 1 η , 1 θ -γ , 1 γ , e θ/γ } and is independent of t , given by

Proof. Note that

Recall that ∥ ¯ w t ∥ 2 ≤ W max . So we have

Then we can apply Taylor's theorem to obtain that

Next we use Lemma B.8 with v = -∇ G ( ¯ w t ) to get The cross-term is bounded by

Using the above and the gradient norm bound from Lemma B.9, we get that

where in the last inequality we use that G max ≥ G min ≥ 1 by Definition 3. From the above we have

The summation is small by Lemma B.6, because

Inserting the above into (27) completes the proof.

We now prove the convergence of the iterates projected on the non-separable subspace.

Lemma B.11 (Convergence on the non-separable subspace) . Suppose Assumptions 1, 2, and 3 hold. Then it holds that

where c 1 > 0 is a polynomial on { e η , e n , e 1 /b , 1 η , 1 θ -γ , 1 γ , e θ/γ } and is independent of T .

Proof. The proof is conducted in several steps.

Step 1: one-step function value bound. Observe that

For the cross-term, we apply Lemma B.8 with v = -( ¯ w t -¯ w ∗ ) to obtain

where the second inequality is by Lemma B.3, and the last inequality is by Lemma B.4. Using the above and the gradient norm bound from Lemma B.9, we get that

By the convexity of G ( · ) , we have

So we get

where we use η ≤ W max := max { 4 n/b, ηn 2 /b } + ηn in the last inequality.

Step 2: the sum of function values stays bounded. Observe that

Taking telescope summation over (30), we obtain

where

is a constant (a polynomial on { e η , e n , e 1 /b , 1 η , 1 θ -γ , 1 γ , e θ/γ } and is independent of t ) defined in Lemma B.10.

Step 3: function value decreases, approximately. For T ≥ t ≥ 1 , we have

which implies that

Step 4: the last function value is small. Taking summation of (32) over t = 2 , . . . T , we get

We also have

Putting these together, we get

where

is a polynomial on { e η , e n , e 1 /b , 1 η , 1 θ -γ , 1 γ , e θ/γ } . So for T ≥ 3 , we have

where c 1 is a polynomial on { e η , e n , e 1 /b , 1 η , 1 θ -γ , 1 γ , e θ/γ } and is independent of T .

C Proofs Missing from the Main Paper

C.1 Proof of Theorem 4.1

Proof of Theorem 4.1. Theorem 4.1 is a consequence of our analysis in Appendix B.

(C) is because of Lemma B.3.

(D) is because of Lemma B.11.

(A) is because of the following:

where the last inequality is because that

by Lemma B.6 and that ∑ n i =1 exp( -w ⊤ t x i ) is uniformly bounded by a constant by Definition 3.

C.2 Proof of Theorem 4.2

Proof of Theorem 4.2. The GD iterates can be written as

We claim that: for every t ≥ 0 ,

1. w t ≥ 0 .

2. | ¯ w t | ≥ 1 .

3. | ¯ w t | ≥ 2 γw t .

We prove the claim by induction. For t = 0 , it holds by assumption. Now suppose that the claim holds for t and consider the case of t +1 .

w t +1 ≥ 0 holds since w t +1 ≥ w t by (33) and w t ≥ 0 by the induction hypothesis.

| ¯ w t +1 | ≥ 1 holds because

To prove that | ¯ w t +1 | ≥ 2 γw t , first observe that

Then we have

We have completed the induction.

Finally, we prove the claims in Theorem 4.2 using the above results.

(B) is because of from (33).

We have already proved (C) by induction.

To show (D), without lose of generality, let us assume ¯ w t ≥ 0 , then

We can repeat the above argument to show that ¯ w t +1 > 0 if ¯ w t ≤ 0 .

0

2000

4000

6000

8000

10000

iteration

10

4

10

3

10

2

10

1

10 0

10 1

10 2

training loss

=0.1

=0.01

0

50

100

150

200

0

2

4

6

8

10

(a) Training loss

̸

0

2000

4000

6000

8000

10000

iteration

10

8

10

7

10

6

normalized margin

=0.1

=0.01

(b) Normalized margin for homogenous network

To show (A), we apply w t →∞ and that | ¯ w t | ≥ 2 γw t :

We have completed all the proofs.

D Experimental Setups

Neural network experiments. We randomly sample 1 , 000 data from the MNIST 3 dataset as the training set and use the remaining data as the test set. The feature vectors are normalized such that each feature is within [ -1 , 1] .

We use a fully connected network with the following structure

The network is initialized with Kaiming initialization. We use the cross-entropy loss.

We consider constant-stepsize GD with two types of stepsizes, η = 0 . 1 and η = 0 . 01 .

The results are presented in Figure 1.

Logistic regression experiments. We randomly sample 1 , 000 data with labels '0' and '8' from the MNIST dataset as the training set. The feature vectors are normalized such that each feature is within [ -1 , 1] .

We use a linear model without bias. So the number of parameters is 784 . The model is initialized from zero. We use the binary cross-entropy loss, i.e., the logistic loss.

We consider constant-stepsize GD with various stepsizes.

The results are presented in Figure 2.

Additional experiments. We conduct additional experiments on the margin maximization effect of large stepsize GD on a homogenous neural network. Results are presented in Figure 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study gradient descent for logistic regression on linearly separable data, with a focus on large stepsizes, and in particular stepsizes in the edge of stability (EoS) regime.\nWith the logistic loss, they show that for *any* positive stepsize, the empirical loss is minimized as the number of GD steps increases, and the limit direction maximises the margin. Explicit rates are given.\nThe authors also provide counterexamples for the exponential loss: if the stepsize is too large, the loss diverges and the iterates cannot converge to the maxmargin direction.\n\nSTRENGTHS:\nThe authors study a hot topic (Edge of Stability) on an overall very well studied problem (logistic regression). \nIn these lines of work, the results are rather strong: proving convergence for any stepsize, beyond those that provide a nice descent lemma, seemed like a challenging problem, and this forms a strong contribution.\n\nThe other contributions (comparison between logistic and exp. losses, , the technical analysis) are also independently of interest.\n\nOverall, the paper is neatly written and easy to follow.\n\nWEAKNESSES:\nThere are no main concerns or weaknesses to this paper, in terms of results or presentation.\nHowever, if I had to give one, I would say that since the implicit bias is the same whatever the stepsize is (it is thus the same for infinitesimal  ones and for stepsizes at the EoS), Theorem 1 extends previous results on the implicit bias of logistic regression, but does not give insight onto what happens at the edge of stability, and what drives generalization to increase. Could this be commented by the authors ?\n\nAlso, here are some recent references that study the EoS phenomenon on other problems, and that should be mentionned in the related works (they are all quite recent):\n\nIn contrast to the logistic regression problem studied in this submission, large stepsizes on regression problem lead to different solutions for GD and SGD when at the edge of stability:\n*(S) GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability*, Even et al. 2022.\n\nThe following paper also studies *scalar* linear networks (a scalar parametered by a product $xy$), as done in Ahn et al. 2022a,b and Zhu et al. 2022 cited in the submission, but in a more precise way: *Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond*.\n\nFinally, *SGD with Large Step Sizes Learns Sparse Features*, Andriushchenko et al. 2022 study large stepsizes SGD and generalization (more experimental piece of work).\n\nQUESTIONS:\nI have no further questions.\n\nLIMITATIONS:\nNo such limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the convergence and implicit bias of logistic regression on linearly separable datasets with arbitrarily large step sizes.  The paper proves that gradient descent converges _for any_ constant step size, even those which result in non-decreasing loss curves.  Similar to the regime of small step sizes, gradient descent converges along the direction of the max-margin linear separator.\n\nIn contrast to logistic loss, the paper proves that for exponential loss there exist datasets and step sizes where the optimizer fails to converge, shedding light on why logistic loss may be preferable in practice over exponential loss.\n\nSTRENGTHS:\n- It's interesting (and far from obvious a priori) that GD with _any_ constant step size will converge.\n- The separation between logistic and exponential losses sheds light on why the former is preferred in practice (since both losses are equivalent under the small step size regime studied in previous works)\n- The separable / non-separable orthogonal decomposition seems like it might be useful for other analyses of logistic regression on separable datasets.\n\nWEAKNESSES:\n- Minor: would Assumption 3 hold if the dataset were the last-layer features from a neural net has undergone neural collapse, or would such a dataset not satisfy assumption 3?\n\nQUESTIONS:\n- Could you include some numerical examples of convergence at extremely large step sizes?  I am curious to see what the loss curve looks like.\n\n- The following recent paper should be discussed: \n\nChunrui Liu, Wei Huang, and Richard Yi Da Xu.  \"Implicit Bias of Deep Learning in the Large Learning Rate Phase: A Data Separability Perspective.\"  Applied Sciences, 2023.  https://www.mdpi.com/2076-3417/13/6/3961\n\nThe paper does not seem to be on arxiv, and the venue is not a standard one ... nevertheless, the paper is quite related.\n\nIn their Proposition 1, that paper argues that for logistic regression on _non-separable datasets_ (or, more generally, for any strongly convex optimization problem), gradient descent fails to converge for large enough step sizes.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper studies the convergence of gradient descent with large step sizes beyond the classical stepsize suggest by smoothness-based convergence. Across the iterates, the risk here exhibits non-monotonic behaviour and yet converges in the long range.  To show these phenomenon, the paper rigorously studies the gradient descent with large step-size in the case of logistic regression with linearly separable data.

STRENGTHS:
a) The paper studies logistic regression in the case of separable data. Even in this simpler case, the authors prove a very strong result. It is shown that gradient descent with **any** constant stepsize converges. 

b) For gradient descent with any step-size, the implicit bias towards the max $\ell_2$-margin solution is shown. 

c) A novel proof technique was introduced to handle any stepsize and it was clearly presented with a simple example. In contrast to existing works, the implicit bias is shown which leads to convergence in risk.

WEAKNESSES:
a) The proof is tailor made for linear regression and does not work beyond that for any convex functions.

QUESTIONS:
a) In the case of exponential loss, it is shown that it can diverge with a large stepsize for a simple example. Do you think there exists a sub-class of problems where exponential loss can converge with any step size?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research investigates the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime. The authors assert that despite local oscillations, the logistic loss can be minimized by GD with constant stepsize over a long time scale. \tExperiments on neural networks and logistic regression verify the theory and illustrate the non-monotonic convergence and implicit bias of GD.\n\nSTRENGTHS:\n- Provides novel theoretical analysis of GD allowing constant step sizes, complementing existing theories requiring small step sizes. This better matches practical deep learning where large steps are used.\n- Establishes implicit bias results for GD with any constant step size. Shows GD aims to maximize the margin regardless of step size.\n- The analysis technique of decomposing GD iterates into orthogonal components is innovative. It allows handling large step sizes that cause loss oscillations.\n- Experiments on neural networks and logistic regression align with theory and illustrate non-monotonic convergence and implicit bias.\n- Shows logistic loss is better behaved than exponential loss for large-stepsize GD, providing practical insights.\n\nWEAKNESSES:\n- The analysis is limited to logistic regression on linearly separable data. Extending it to more complex models like neural networks is not clear.\n- Some assumptions made, like linearly separable data and non-degenerate support vectors, may be restrictive and not hold in practice. For example, wrongly labeled samples are common.\n\nQUESTIONS:\nHow to define linearly separable in neural network literature?\n\nIt is clear that when you use a large learning rate, the neural network goes far away from the linear regime. How can you make a compensation to use the current model to understand the non-linearity in neural networks?\n\nLIMITATIONS:\nThis work makes valuable theoretical contributions but has limitations in terms of connection to practice. Extending the analysis to address these limitations remains an open challenge.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
This work investigates the implicit bias in linear regression with logistic loss with large learning rates. It shows the convergence result of logistic loss for any learning rates, as long as an counter-example of exponential loss to show the gap between these two losses.

STRENGTHS:
1. The motivation and setting are good: large learning rates (i.e., Edge of Stability) are a rising topic recently. As pointed out by the authors, it requires reconsideration of previous results of implicit regularization with small learning rates. To my knowledge, this work is the first to redo one of the bias results in the standard setting.

2. The gap between logistic and exponential losses is good: typically implicit-bias works handle these two similarly, but this work shows that logistic loss can contain arbitrarily large learning rates while exponential loss has an upper bound.

WEAKNESSES:
1. The covered setting is somehow simple in lack of detailed discussions of what impact logistic loss + large learning rates have on more complicated settings possibly. For instance, does the max-margin bias in deep homogenous model (Lyu and Li) hold any more? This kind of impact could be verified with visualized experiments, with the theoretical development left as future works.

QUESTIONS:
As above in weakness.

Typos & minor suggestions:
1. below 299: it is $\nabla G$ instead of $G$ in the inner product.

2. in Figure 2, the bound of $2/\eta=0.2$ can be plotted in a more clear way, such as log.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23