A Minimalist Example of Edge-of-Stability and Progressive Sharpening
Authors: Liming Liu
Zixuan Zhang
Simon Shaolei Du
Tuo Zhao
Venue: NeurIPS 2025 poster ()
Keywords: edge of stability
progressive sharpening
gradient descent
training dynamics
nonconvex optimization
linear network
Primary area: optimization
TL;DR: A new minimalist example to understand the Edge of Stability and Progressive Sharpening phenomenon

Abstract: Recent advances in deep learning optimization have unveiled two intriguing phenomena under large learning rates: Edge of Stability (EoS) and Progressive Sharpening (PS), challenging classical Gradient Descent (GD) analyses. Current research approaches, using either generalist frameworks or minimalist examples, face significant limitations in explaining these phenomena. This paper advances the minimalist approach by introducing a two-layer network with a two-dimensional input, where one dimension is relevant to the response and the other is irrelevant. Through this model, we rigorously prove the existence of progressive sharpening and self-stabilization under large learning rates, and establish non-asymptotic analysis of the training dynamics and sharpness along the entire GD trajectory. Besides, we connect our minimalist example to existing works by reconciling the existence of a well-behaved "stable set" between minimalist and generalist analyses, and extending the analysis of Gradient Flow Solution sharpness to our two-dimensional input scenario. These findings provide new insights into the EoS phenomenon from both parameter and input data distribution perspectives, potentially informing more effective optimization strategies in deep learning practice.

A Minimalist Example of Edge-of-Stability and Progressive Sharpening
Liming Liu Georgia Institute of Technology 11iu6060gatech. edu
Zixuan Zhang Georgia Institute of Technology zzhang31050gatech. edu
Simon Shaolei Du University of Washington ssduucs. washington, edu
Tuo Zhao
Georgia Institute of Technology
tourzhao@gatech.c edu
Abstract
Recent advances in deep learning optimization have unveiled two intriguing phe- nomena under large learning rates: Edge of Stability (FoS) and Progressive Sharp- ening (PS), challenging classical Gradient Descent (GD) analyses. Current research approaches, using either generalist frameworks or minimalist examples, face signifi- cant limitations in explaining these phenomena. This paper advances the minimalist approach by introducing a two-layer network with a two-dimensional input, where dimension is relevant the response and the other is irrelevant. Through this model, we rigorously prove the existence of progressive sharpening and self- stabilization under large learning rates, and establish non-asymptotic analysis of the training dynamics and sharpness along the entire GD trajectory. Besides, we connect minimalist example to existing works by reconciling the existence well- bell-behaved "stable set" between minimalist and generalist analyses, and extending the analysis of Gradient Flow Solution sharpness our two-dimensional input scenario. These findings provide new insights into the EoS phenomenon from both parameter and input data distribution perspectives, potentially informing more effective optimization strategies in deep learning practice.
Introduction
Deep learning has revolutionized many fields, from computer vision to natural language processing. However, this progress has also posed significant challenges to classical optimization theory. Most classical gradient descent (GD) analysis assumes small learning rates for easing convergence analysis. Consider minimizing a smooth loss function I.(0) with respect the parameter 0, classical analyses show that when choosing a learning rate such that S()) < 2/1,, where S(0) denotes the largest eigenvalue of the Hessian matrix (2)) the optimization is "stable" and the loss function decreases monotonically to guarantee convergence Nesterov 2013
Recent works such as Cohen et al. 2021 however, have observed that such stability assumption does not hold when training modern neural networks with GD. In particular, they summarize two specific phenomena: The first one is called "Progressive Sharpening" "(PS), that is, S(0), which is also referred to as the "sharpness" in Cohen et al. [2021]. keeps increasing until it reaches the instability threshold 2/n during training; The second one is called "Edge of Stability is, the sharpness hovers at the instability threshold 2/1 after progressive sharpening, with the loss decreases continuously and nonmonotonically. These two phenomena undoubtedly challenge the classical analyses and have already attracted the attention many researchers.
that
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Current research on understanding these phenomena has developed along two lines. The first line purses a so-called "generalist analysis" frameworks that, while being generic, rely on hard-to verify assumptions. For example, [ et al. [022b] analyze GD for training two-layer wide neural networks. By characterizing the norms of the second-layer weights, they prove four-stage behavior, covering the PS and EoS phenomena. However, their analysis requires assumptions that are hard to verify for two-layer neural networks, e.g., the sharpness being upper bounded, and only works for extremely wide settings, which diverges from practical scenarios. Damian et al. [2022] prove a similar four-stage behavior based on a general loss function, and their analysis also relies on hard-to-verify assumptions, such the existence of progressive sharpening and the existence of a certain well-behaved "stable set", which is doubted and showed badly-behaver for scalar networks in [Keisserr et al., [023]] Overall speaking, such analysis provide results similar to real experiments. but their hard-to- verity assumptions significantly restrict their applicability to practice. Relaxing these assumptions is also extremely challenging.
The second line focuses on minimalist examples, offering more concrete and intuitive insights without requiring specific theoretical assumptions. For instance, Zhu et al. [2022] study 2d slices of a 4-layer linear scalar network and prove convergence to a minimum with sharpness slightly below to 2/n, and Wang et al.
[0023] consider the behavior of sharpness with GD on special class of 2-layer scalar networks (with nonlinear activation) in the form of F F(ry). However, these works suffer from two drawbacks: (I) They can only characterize the asymptotic sharpness of the converged minimum. Due to the lack of analyzing sharpness for the entire trajectory, they cannot provide more desirable nonasymptotic guarantees for the PS and EoS stages: (2) The setting of scalar networks is over-simplified compared with practice, and the obtained results cannot explain the role of the input data dimension in the EoS stage.
Besides, another notable example of the minimalist analysis is Kreisler et al. [2023]. which analyze a variant of sharpness. Specifically, they introduce a concept of Gradient Flow Solution (GFS), and prove a monotonic decrease in the sharpness of GFS at the EoS stage for scalar networks. In addition to the aforementioned drawbacks, Kreisler et al.] [2023] also suffer from another drawback: The sharpness of GFS does not directly transfer to that of GD trajectory, thus providing no immediate explanation for the EoS phenomenon.
Here we aim to address the limitations of the minimalist analysis by providing more nontrivial example: A two-layer neural network of width one with two-dimensional input - in particular, one input dimension is relevant to the response, and the other one is irrelevant. We establish nonasymptotic analysis of the training dynamics along the entire trajectory: (1) We prove the existence of progressive sharpening and self-stabilization under large learning rates; (2) We provide sharpness guarantees for the entire trajectory, showing that GD trajectory will never exceed a sharpness upper bound; (3) We prove that the mon-monotyn decreasing loss is essentially monotonically decreasing when projected to the only relevant dimension. Through such theory, we provides new insights of why EoS happens from the perspective of both parameters and input data distribution.
Moreover, we highlight two connections of our theory to existing works: (1) We reduce the gap on the existence of a well-hehaved "stable set" between the minimalist and generalist analyses. Specifically,
Kreisler et al. (2023] prove that the stable set hypothesized in Damian et al.] [2022) can be disjoint in
the scalar networks studied by Zhu et al. [022]. ). Kreister et al. [002]] which essentially violates the
assumption of Damian et al. [2022). Therefore, the projected GD considered in Damian et al. [0022] cannot smoothly decrease the loss toward zero. In contrast, we prove that in our considered two-layer neural network with the two-dimensional input admits nontrivial well-behaved set, which is the subset of the stable set defined in Damian et al. [022]. This indicates a potential separation between the bivariate and scalar inputs for linear networks; (2) We extend the analysis of Kreisler et al. [023] and provide the monotonic decrease of GFS sharpness for our considered two-layer neural network with the two-dimensional input.
2 Related Works
The asymptotic property of GD sharpness was first mentioned in Wu et al. [2018] as an empirical observation, that is, the minimal sharpness that the GD trajectory with learning rate 7) ultimately converges to is always around 2 Cohen et al. [202]] made a comprehensive empirical study on the sharpness of the entire GD trajectory. To be more specific, they summarized two phenomena:
")
2

---

"progressive sharpening" and "edge of stability", which means the sharpness of gradient descent with a learning rate 7 will first increase to 2 and then stabilize at such a scale during the entire training process. They also illustrated that the training loss of GD with learning rate 1) can non-monotonically decrease, even when the stable condition, sharpness À S (where 7) is the learning rate), is not satisfied. The non-monotonic decay property of the training loss with GD has also been observed in various other settings [Wu et al. 2018 Arora et al. 2018 Xing et al. 2018 J Jastrzebski et al. 2020
Lewkowycz et al. 2020 Wang et al. 2021 Li et al. 2022a
Recently, several works have attempted to comprehend the mechanism behind EoS with different
loss functions under various assumptions [Ahn et al., 2022 Ma et al., 2022 Arora et al., 2022 Lyu
jt al. 2022 Li al. 2022b Zhu al. 2022]. From a landscape perspective, Ma et al. [022]]
defined special subquadratic property of the loss function and proved that EoS occurs based this
assumption. AAhn et al. [2022] followed this landscape property and studied the unstable convergence
behavior of GD. Both Arora et al. [0222] and Lyu et al. [0022] investigated the implicit bias on the
sharpness of GD in some general loss function.
There are also some works trying to investigate
EoS phenomenon on highly simplified settings. Zhu 2022 proved the asymptotic sharpness of the converged minimum will be close to 2 Wang
ct al.
et al. 2023 consider the behavior of sharpness with GD on special class of 2-layer scalar networks (with nonlinear activation) in the form of F F(zy). and also get a asymptotic result and show some other behavior beyond BoS their setting. Agarwala et al. [2022] investigate second order regression models, get asymptotic result similar to Zhu et al. 2022], and a result loosely related to "progressive sharpening". Chen and Bruna [0233 investigate the behavior beyond edge of stability in various simplified examples. Kreisler et al. [023]] consider the setting of scalar networks, and show new concept gradient flow solution (GFS) will decrease during EoS.
Another line of work [Lewkowycz et al. 2020 Wang et al. 2021] focuses on the implicit bias introduced by large learning rates. [eewkowyz et al. [0200] first proposed the "catapult phase" regime similar to the EoS, where loss does not diverge even if the sharpness is larger than 2 Wang
et al. 2021 analyze the balance effect of GD with large learning rate for matrix factorization problems. More recently, [Li et al. [2022b] provided a theoretical analysis of the sharpness along the gradient descent trajectory in a highly overparameterized two-layer linear network setting under some hard-to-verify assumptions during the training process. Damian et al. [022] followed Li et al.) (2022b) to develop general theory of self-stabilization also under some hard-to-verinn verify assumptions. Moreover, they proposed a concept called "constrained trajectory" to show that the trajectory of GD with large learning rate deviates from the gradient flow, which was firstly observed by Dastrzebski
ct al. (2020) and confirmed by Coohen et al. [2021].
Setup
In this paper, we study a regression problem with two-dimensional input x = (2), E and scalar response y = (x) IR For analytical simplicity, we suppose
211 22
0, A1
0 2)) and f"(x)) F2,
N
A 2 K, where K is a large enough absolute constant. Here a is an inrelevant feature We suppose X (or noise) with large scale, while small-scale x2 fully determines the response
Although the setting appears relatively simple, using the rotation invariance property of GD on neural networks, which means that rotating the input data by an orthogonal transformation leaves the gradient descent dynamics unchanged, non-diagonal data covariance matrix. In addition, exploiting parameter symmetry allows us to extend this framework to handle more than two features. See Appendix for details.
can extend this setting to dimensiaimeli spaces with
To learn the target function f", we use a two-layer width-one linear network with weights = (a, B2) E 3
f(x; = 0,3201 + a:B222.
(3.1)
Then the population square loss is given by
1 2 Ez.y [w f(x;0))3] = (1(03)) + 22 - 1
L(0)= =
(3.2)
3

---

The Hessian matrix (f) of L(0) can be written as:
21 + 2252
2A1a81 Aaa²
2220
- 12
H(0) =
2
2A20.32 X2
0
Definition 3.1 (Sharpness of L(0)). We define the largest eigenvalue of (0) as the sharpness parameter of L(O). We denote it by S(0).
Our setting is partly motivated by Rosenfeld and Risteski [2023], which demonstrated that the oscillations observed during the EoS stage in image classification tasks are primarily driven by large-magnitude features in the input data. In particular, such features, such as the background color in CIFAR-10 images, exhibit little correlation with the true labels and effectively act as noise during training. Building on their observations, our work provides rigorous theoretical foundations that formalize and clarify their informal explanations, demonstrating how the EoS phenomenon and the behaviors hypothesized in Rosenfeld and Risteski [2023] emerge. A more detailed discussion is provided in Appendix
We consider the following constrained optimization problem in our setting:
min L(0) where θ = (0 |3 s c] AE
The threshold 1 (A2//A1) is chosen based the condition number A2/A1, which reflects the scale disparity between input features. In our setting, 31 associated exclusively with the high-variance, noise-dominated feature I1. while 32 contributes fitting the informative signal. As shown in (3.1). increasing |31| disproportionately amplifies the effect of noise without improving alignment with the target. In contrast, a scales both $1 and 3g simultancously, and thus amplifies noise and signal together, preserving the relative structure of the model's response.
In our setting, the instability in training is caused by large-scale noise. However, when |81] is too large, the noise is excessively amplified, and the properties of the dynamics go beyond the limits of existing theoretical tools, current optimization theories mainly focus on convergence behavior and lack appropriate tools for handling instability. Therefore, although we strive to use new methods to analyze the unstable optimization process, we must impose constraints to ensure it does not exceed the theoretical limits. Our numerical experiments in Figure | show that even on this constrained problem, the optimization process remains highly unstable. In Section G we demonstrate that the behavior of the constrained optimization process is consistent with that of the unconstrained one.
The update rule for GD with learning rate 1j in this constrained problem is: A(t + 1) II (0(t) r)(L)(()))
where II is the projection function. We initialize the weights within the initialization set (1),,, which defined as the set of satisfying
1.1 V
/
2
v V
3
1 a
S sas
$ V A11
max
a)
S
A1
20
20a
X2 (1 a:32) S 87
X2,3 (1
a:32)
500A1a
\1a
Our analyses focus on sufficiently large learning rate 77 E 122 A Xg where ((())) is nonempty.
Notably, this initialization allows us to explore the training dynamics across wide range of learning rates, all starting from the same initial point. Additional details are provided in Appendix D
The GD dynamics of our model exhibits interesting EoS phenomena. As shown in Figure I we observe that while the loss decreases over long timescales, exhibits non-monotonic behavior with periodic spikes. Meanwhile, the sharpness grows and oscillates around 2/1,, with rapid alternation between progressive sharpening and self-stabilization phases. These characteristies align with the EoS phenomena observed in Cohen et al. [202]] and Damian et al. [2022], and extend beyond the scalar network setting studied in Zhu et al. 2022 We provide further discussion in Section
4 Theoretical Results
In this section, we develop a nonasymptotic analysis of the GD dynamics throughout the entire trajectory. In Section 4.1 we identify three distinct phases in the GD dynamics, proving the existence

---

70-
logltess)
sharpmess
40
30-
10-
2000
4000
6000
8000
10000
20000
4000
6000
8000
10000
epoch
epoch
Figure 1; Set X1 = 100 and X2 = 0.01. We train our model with 1/20 for 10000 iterations.
of progressive sharpening and self-stabilizattion Furthermore, in Section we establish that the loss moo--monotooically) converges to zero, with a rate depending explicitly on the input data variance. In our analysis, we use learning late 17 E 2 To./A,, l, and initialization within the set X' (())) unless otherwise specified. Our choice of is sufficiently large guarantee the occurrence of EoS.
4.1 Gradient Descent Dynamics
To begin with, we present the three distinct phases identified in the GD dynamics.
Phase 1: Progressive sharpening before EoS. The loss decreases monotonically, while the loss sharpness gradually increases, yet remains below the stability threshold 2/n-
Phase 2.1: Progressive sharpening during EoS. As the dynamics enters the EoS (Edge of Stability) stage, the sharpness continues to increase monotonically and exceeds the stability threshold 2/n.
Phase 2.2: Self-stabilization during EoS. In this phase of the EoS stage, the dynamics self-stabilize as the sharpness decreases monotonically until it falls below 2/n.
In addition to proving the existence of these phases, also show that throughout the entire training process, the sharpness controlled within a range close to 2/1,, and that gradient descent eventually converges minimum whose sharpness is strictly upper bounded by 2/n.
Theorem 4.1 (Global Convergence). For any > and e > 0, there exists a time T(5, <), such that for any & 2 T(6, <), have
2+8
L(((t)) S and S(O(t)) <
(4.1)
To rigorously characterize the GD dynamics, we first present the following properties of the parameters ((t) = (a(t), 3 (t), 32())):
Lemma 4.2. For 2 use vtt) to denote the eigenvector corresponding to the largest eigenvalue
of ((((t)), we have:
ii) X (t) S(0(1) & a (ii) (O,
(0) > 0.9; (iii) 11(t+1)>BB1 (2()).
We make the following remarks for Lemma 4.2
Sharpness dominated by Property (i) demonstrates that a1 dominates the sharpness S(0) along the trajectory, suggesting that analyzing suffices to reveal the dynamics of sharpness.
Instability 31 The top eigenvector v(t) of ((((t) identifies the direction of maximum curvature in L(((t)). When the sharpness is as large 2/1, updates along v(t) can induce oscillatory instability in the GD dynamics. Property (ii) reveals that the oscillating direction v( v(() closely aligns with B1. Furthermore, we will show that 31 induces spikes in the loss L. in Section 4.2
Monotonic Increase of 32 In contrast 31 B2 stable direction. This stability difference a&L 89, (0) Aza is much smaller than 8 (0) A Property (iii) confirms this stability, proving that 32 increases monotonically 73, throughout the GD trajectory.
arises from their respective second-order partial derivatives:
5

---

Now we are ready to characterize the sharpness along the GD trajectory, and demonstrate the existence of progressive sharpening and EoS in the following theorem.
Theorem 4.3. Let I to be the first time such that Ai (T)) 1.5 (Here Ti can be oc). Then for any 0 < < Ti, a > increases monotonically, and
1.1 S S(O(t)) v
1.7
7
"
Fort 2 T1, we have
1.5 < S(O(t)) -
4.7 1
7
Theorem 4.3 guarantees that the phenomenon of progressive sharpening before entering the EoS stage (Phase 1) and the sharpness bounded near 2/1) during EoS (Phase 2). Specifically, when Tı, the sharpness keeps growing, as evidentiated by the monotonic increase of a, yet stays below 2/1,. When t z Ti, the sharpness bounded from below and above, near 2/1,, indicating the GD trajectory stays in the flat region and never escapes to a sharper region during EoS.
Furthermore, prove the progressive sharpening (Phase 2.1) and self- seff-stabilization (Phase 2.2) during EoS, when 0- (0,31, B2) is initialized within more stable (1))
22 -a02) }.
{(n) = (a, 32) E X(n) a v1.5/(Amn) 82 S 0.2/a, Bf
50
Here (n) forms a significant subset of X1). represents relatively flat region X Then can derive the following theorem:
Theorem 4.4 (Edge of Stability). Let 0(0) (())). There exists T2 T3, such that:
(Progressive Sharpening) For & [0, T2], we have
2.37
a(t + 1) > att). and S(O(T2))
1
(Self-stabilization) Fort t E (T2, Tz]. we have
a(l + 1) < all); and S(O(Ts)) 2.01
"
Here we demonstrate the monotonicity of a. The monotonicity of S(0) presents a more complex analytical challenge due to the absence of an explicit expression. Nevertheless, as demonstrated in Lemma 4.2 magnitude of S(0) is tightly bounded both above and below by a². While we do not provide strict proof of S(0)'s monotonicity, these bounds effectively characterize its behavior with considerable precision.
Theorem 4.4 gives a characterization of progressive sharpening and self-stabilization, which are consistent with the empirical observation in Damian et al. [2022] and [i et al. (2022b). When T2, sharpness tends to increase as increases monotonically. and exceeds the stability threshold 2/n. When T. t S T3, the sharpness drops, as a decrease monotonically. Additionally, we empirically demonstrate that during EoS (Figure []. the GD dynamics repeatedly alternates between progressive sharpening (Phase 2.1) and self-stabilization (Phase 2.2).
4.2 Loss Decay Rate
In this section, we estimate a monotonic decay rate of the non-monotonic loss. Recall the loss function defined in (3.2):
L(Ø) = (a
(4.2)
2
L1: oscillatory tern
L2: convergence term
The loss L. has two components, an oscillatory term L, and a convergence term Tz2. The first term I corresponds to learning the irrelevant feature I1, and contains oscillatory parameter 31, as shown

---

L less
0.006
L loss
loss
0.005
[ loss
0.004
0.003
0.002
0.001
-10
0.000
2000
4000
8000
8000
10000
2000
4000
6000
8000
10600
epcch
spoch
Figure 2: Same setting as Figure 1 In the left figure we plot L (0) and [(0) in log scale. We can see that [(0) nicely reflects the decay rate for (o).. The slope of the red dashed line is 2 log 222) ). which nicely reflect the decrease rate of L(0). In the right figure we plot (0) and
La (0). In most time L, (0) near zero unless spikes occur.
in (ii) of Lemma 4.2 Empirically, L1 exhibits drastic fluctuations, but rapidly drops to a level near zero every time the GD trajectory goes back to the stable region where S(0) < 2/ 2/1, as demonstrated in Figure 2 We thereby deduce that L1 contributes to the spikes appearing in the loss dynamics, while L2 dominates the loss descent.
Notably, L1 barely influences the overall loss descent rate. This motivates us to focus on L2 to estimate the monotonic decay rate. Our numerical experiments in Figure 2 indicate that Lg well approximates the descent trend of I.. However, I.2 still non-monotonic, due to the fluctuations of a. Thereby, we consider
[(0) = (11 (2
VA17
by restricting 22/(A11). We first show that T. is good estimate for T2: Lemma 4.5. Let 8(0) E (n)) For any
f T4, where T4 min 2 0 (1(()) 2
(,5///1//), we have (6) < (()) S 3.). Moreover, for any S S Tς such that att) S /2((>m), we have 0. (0) - L(0) < (0).
Lemma 4.5 proves that (0) can controlled by i.)0), especially when GD trajectory is in the stable region. Next, we present the decay rate for L:
Theorem 4.6. Let 0(0) E X1)).. For any 0 S < T4, with T4 defined in Lemma 4.5 we have:
42g 2 İ(0(t + 1)) (1
A2 2 A1
<
((((t))
Theorem 4.6 proves L decays in linear rate upper bounded by A2/11 The rate becomes faster as A2/A, grows larger. This reflects how the relative scale between features influences the loss decay. As \ increases, the model becomes more sensitive to the relevant feature ø2, leading to faster loss descent. In addition, Figure 2 shows that the rate (1 2 which lies between the upper and lower bounds in Theorem 4.6 precisely estimates the decreasing speed of L()).
Our analysis primarily focuses on the interval [0, T2). While this formulation might appear constrained, it captures substantial period of the optimization process, yielding valuable insights into the GD dynamics. Notably, Figure @euuree@emmooe demonstrates that the estimated decay rate maintains its validity well beyond the theoretically analyzed timeframe, suggesting broader applicability of our findings.
Connection with Existing Results
We present the connections our theory to the gradient flow solution (GFS) proposed in Kreisler et al. (2023), and the "constrained trajectory" in Damian et al. [2022]
7

---

5.1 Connection with Gradient Flow Solution
Firstly, we introduce gradient flow (GF), which is the continuous form of GD: ((t) - For any initialization 6, define the gradient flow solution (GFS) Scr(O) (0) as the limit of the GF trajectory, and denote o(0) as the GFS sharpness, i.c. the loss sharpness at SGF(0).
Kreisler et al. 2023 study the GFS sharpness along the GD trajectory on scalar networks, proving its monotonic decrease. We extend this analysis to our two-layer neural network with two-dimensional input. Herein, we verify that GD reduces the GFS sharpness in this more general setting, while further elucidating the distinction between GFS sharpness and GD sharpness.
The following Lemma characterizes the GFS sharpness when the GF starts at 0(0) E xx)).
Lemma 5.1. Let 1) [2/A1: , For any ((0),, ((0) E x satisfying a(0) (2)))), there exists 31 (0) R such that ((0) = (a(0), 81 (0), (0)) E (n)) Moreover, the GF initialized at ((0) converges to solution with the following sharpness:
1
(((()) 2 (1 4000 )A1-
Lemma 5.1 reveals that when initialized from subset of X1)).. GF converges to sharp minima with sharpness lower bounded by (1 2006 A1. This result show that our analysis is beyond that in Kreisler al. [2023). as they requires ((0) I 2v2 In particular, given 10yA1 Xi" when 2 5/A1. the GFS sharpness x(0) > 4/n. While GD with large 1j converges to flat minima whose sharpness is bounded by 2/1 (Theorem 4.1.. GF exhibits different trajectories and converges to solutions with greater sharpness.
"
Next analyze GF initialized from points (()) along GD trajectory. The following theorem demonstrates that GD decreases the GFS sharpness, which is consistent with the observations in Kreisler et al. (2023).
120
lower bound
up bound
100
80
Theorem 5.2. For any 2 O, we have
60
<(((1)) 2 1 (t)
A1
+
m/1
1
(t))
40
2n
2
opoch
((((t) < 4.2 (1)
A1
4
+
1/11
4.2
+
Figure We choose vj = 12. and show the evolvement upper and lower bound in Theorem 5.2
21)
and both the upper and lower bounds monotonically decrease with time / 2 0.
Here the monotonic decrease of the upper and lower bound in Theorem [5. is because the monotonic increase of 32 (t), proved in Lemma 4.2 (iii). In Figure 3 we plot the upper and lower bound in Theorem This demonstrates that both bounds do decrease monotonically with time, and implies a near-monotonic decrease of the GFS sharpness.
<
0.5
0.8
1.0
1.8
1.6
1.4
OF oe:
Figure 4 illustrates GF trajectories initiated from various points along the GD path. Recall that greater a indicates higher sharpness, as shown in (i) of Lemma 4.2) The plot demonstrates how the GFSs transition from the highly sharp Minimizer to Minimizer 3 with moderate sharpness 2/n-
$
Mereent Manwer Stati faat)
0.6
5.2 Connection with Constrained Trajectory
Figure 4: Same setting as Figure We plot GFs starting from different points on the GD trajectory and the minimizers these GFs converge to.
In this section, we draw connections to the "constrained trajectory" framework introduced by
8

---

Damian et al. [0022] who show that GD at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint S(Ø) < 2/n. Specifically, they consider PGD on a so-called
stable set M, which is defined as
M(n) = 0: -
5())
2 and VL(0)
(0))
where u(0) the cigenvector associated with the largest eigenvalue of Hessian (0). Then the PGD on M is formulated as
0g ITM (0o) and 04 MM (0! (0.--NV..01))) ()TT.
where the projection on M is defined MT) == argmingre.M ||0 0'| The trajectory of the PGD is referred to as "constrained trajectory".
As mentioned in Damian et al. [2022], the constrained trajectory is mainly determined by the sharpness condition, S(0) S 2/n,, while the other condition VL(0) u(0) of M is included
to ensure the constrained trajectory stable, without affecting the stationary points of PGD on M. Following the same principle, choose our stable set
M (n - 0: V sas
2 Ann :1 = = aB2 5
The first condition a corresponds directly to maintaining sharpness near 2/11 (Lemma 4.2 The second condition sets the oscillatory term 31 to zero. Moreover, the third condition ensures parameter in M (1) is not far from solution manifold. The following lemma proves M C M, and provides explicit update rule for the PGD Mt.
Lemma 5.3. Let r E [2/X1, 0.1] and 8(0) (1)) Then Mt(n) is subset of M(n), and the PGD on Mt(n) has the following update: (t) and
2
at(t + 1) Clip a (VL(((())),
V
1711
al(t - 1) =t)) 1/ Bo (t)),
where Clip(æ, c) = sign(x) minn(1x],,
Lemma 5.3 implies that the PGD on M can be regarded as GD with 31 restricted to zero and a clipped. We next characterize the PGD trajectory, derive its convergence rate during EoS, and demon- strate its value in understanding GD dynamics.
0.5
0.6
0.8
0.9
1.0
1.6
Theorem 5.4. Let == minft 2 0 (t)) = /2/(nA1). For any such that 0 s < 7, the PGD on M satisfies
1.4
1.2
Gelamn Mariil Minrie Mevribe
a (t + 1) > (t) and L(0t(t + 1)) (t))
For any l 2 we have
L(01(t (t + 1)) (1 1 2A2 A1
2 L(B (t)) (5.1)
0.67
5.4orrem5544shoooo the loss monotonically decreases along the constrained trajectory, and provides an explicit convergence rate. We can see that this rate lies between the upper bound and lower bound in Theorem 4.66 and in Figure? 2 we can see that (5.I) predicts the decay rate of precisely.
Figure 5: Same stetting as Figure We plot GD trajectory with learning rate the GF and constrained trajectory starting from the same initialization.
20 well as
The constrained trajectory analysis complements our approach in Section 4.2 in an interesting way. Both methods aim understand the core convergence behavior by handling the oscillatory dynamics, but through different means. The constrained trajectory directly enforees stability by projecting 0

---

4.2 studies the convergence through Lg (defined in (4.2) ). while allowing the oscillatory parameter 3 to vary naturally. Despite these distinct approaches, both analyses arrive at similar convergence rates, indicating that (1 barely impacts the loss descent and reinforcing our understanding of the fundamental dynamics at the EoS.
onto a stable set where 5 = O. while our previous analysis in Section
We also visualize the constrained trajectory in Figure 5
We show that GD first "sharpens" to the unstable region where is large, and then "self-stabilize" to the stable region. After that, another sharpening starts and the trajectory enters the next cyclical behavior. As illustrated in Figure [ during EoS stage, GD trajectory will follow the constrained trajectory rather that GF trajectory.
Discussion
6.1 Alignment with the EoS Phenomena in Practice
To our best knowledge, the phenomenon in our setting, including the periodic progressive sharpening
and self seff-stabilization, and the periodic loss spikes, have not been observed in previous minimalist
settings [Zhu et al., [2022 Kreisler et al. 2023 Wang et al. 2023 Chen and Bruna 2023). Our setting
is based on the two dimensional input data with two features in different scale. This is different
from most previous minimalist settings such as Zhu et al. [022] Kreisler et al. [223] where they
suppose scalar input. Recall that in our setting, suppose large-scat scalesscal&&& but irrelevant feature and
small scale and relevant feature. Such construction is the key for our phenomenon, as the model has the trend to progressive sharpening (to increase its norm) for fitting the small scale feature, but the large-scale irrelevant feature (or noise) will provide a regularization for the model, which provides a self-stabilization effect.
We argue that observations in our settings are more similar to practical results Cohen et al. (2021),
Damian et al. (2022) than observation in Zhu et al. (2022), Kreisler et al. [2023]. In experiments from Cohen et al. [202]], [amiian et al. [2022] it's easy to see the periodic progressive sharpening, liff-taailization and the loss spikes, see Figure 6 and 7 These phenomenon are also observed in our
synthetic setting. In contrast, in the scalar network setting in Zhu et al. [2022). J Kreisler et al. (2023), J.
the sharpness jump up and down around 2 and their loss function will monotonically decrease after first a few steps. See Figure S and Figure 9 for the experiments on scalar networks.
6.2 Trade-off in Large Learning Rates
An interesting trade-off emerges in our setting: larger learning rates tend drive the optimization toward flatter minima, yet such minima are often located farther from the initialization. More
specifically, since a remains close to V Xyoj
at both initialization and convergence, while 31 starts
from small value and converges to O, the distance between initialization and the final minimal is primarily determined by 82, and its scale is approximately on the order of ((2) O O(VAr)). Thus a larger learning rate 1) results in greater distance between the initialization and the final minimum.
This introduces a tension between the step size and the distance to convergence, which implies that a larger learning rate does not necessarily lead faster convergence. While this conclusion is consistent with previous observations in the literature, our work highlights a distinct underlying mechanism. In convex settings, where the minimum unique, slower convergence under large step sizes is typically attributed to overshooting. In non-convex settings, although minima are not unique, prior work generally focuses on the per-step decrease in objective value, rather than the specific minimum reached.
In contrast, our analysis explicitly shows that larger learning rates converge to flatter-and thus more distant-minima. This perspective reveals a new form of trade-off: the increase in step size is counteracted by the increased distance the final solution, providing a novel explanation for the non-monotonic relationship between learning rate and convergence speed. We believe this insight offers fresh angle for understanding the dynamics of optimization in non-convex landscapes, and opens the door to future work on characterizing such trade-off's in more general settings.
10

---

References
Atish Agarwala, Fabian Pedregosa, and Jeffrey Pennington. Sccond-order regression models exhibit progressive sharpening to the edge of stability. arXiv preprint arXiv:2210.04488 2022.
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient
descent. In International Conference on Machine Learning, pages 247-257. PMLR, 2022.
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. arXiv preprint arXiv: 18t.. 03981,91111 2018.
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pages 948-1024. PMLR, 2022.
Lei Chen and Joan Bruna. Beyond the edge of stability via two-step gradient updates. In International Conference on Machine Learning, pages 4330-4399. PMLR, 2023.
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on
neural networks typically occurs the edge of stability. arXiv preprint arXiv:2103 103i::200330066. 00065, 2021.
Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209. 15594, 2022.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09572, 2020.
Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. arXiv preprint 13314 2023.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl- Dick-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02211. 2020.
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In International Conference Machine Learning, pages 12656-12684 PMLR, 2022a.
Zhouzi Li, Zixuan Wang, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability. arXiv preprint arXiv:2207.12678 2022b.
Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normal- ization layers: Sharpness reduction. Advances in Neural Information Processing Systems, 35: 34689-34708, 2022.
Chao Ma, Lei Wu, and Lexing Ying. The multiscale structure of neural network loss functions: The effect on optimization and origin. arXiv preprint arXiv:2204. /136, 2022.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
Elan Rosenfeld and Andrej Risteski. Outliers with opposing signals have an outsized effect on neural network optimization. arXiv preprint arXiv:2311.04160 2023.
Yuqing Wang, Minshuo Chen, Tuo Zhao, and Molei Tao. Large learning rate tames homogeneity: Convergence and balancing effect. arXiv preprint 2r 2021.
Yuqing Wang, Zhenghao Xu, Tuo Zhao, and Molei Tao. Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult. arXiv preprint arXiv:2310.17087 2023.

---

Lei Wu, Chao Ma, et al. How sgd selects the global minima in over parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31, 2018.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. walk with sgd. arXiv preprint arXiv:1802.0877,, 2018.
Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding -tga-of-stability training dynamics with a minimalist example. arXiv preprint arX1v:22100032294 2022.
12

---

A Phenomena in other works
We begin by examining an experiment from Cohen [021], in which they train two-layer tanh network to approximate Chebyshev polynomial. The observed phenomenon closely resembles our findings. The sharpness initially increases, then rapidly decreases low level, before growing again and entering a cyclic pattern. Large spikes occur and quickly subside during this process. Figure illustrates their results.
train loss (iteration)
sharpness (iteration)
0.3
sharpness
f =
train
0.2
f 2/8
n =
0.1
0.0
500
1000
1500
2000
500
1000
1500
2000
iteration
iteration
Figure 6: Experiment from Cohen al. 2021 A two-layer tanh network is trained to approximate a and
Chebyshev polynomial using Gradient Descent (GD) with learning rates r7
2022b and Damian et al. 2022 For instance, Damian (ct al. 2022] train a transformer with MSE Loss on the SST2 dataset. We present their results in Figure
Similar observations are reported in Li et al.
0.67
Sharpness
Loss
2050
0.66
0.65
2000
500
1000
500
1000
Step
Step
Figure 7: Experiment from Damian et al. 2022
transformer model is trained on the Stanford Sentiment Treebank 2 (SST-2) dataset using Mean Squared Error (MSE) as the loss function.
These experiments from various studies align closely with our numerical experiments. However, we note a discrepancy between these practical experiments and those presented in Zhu et al. 2022 and Kreisler et [2023). their experiments on scalar networks, the loss oscillates during the initial steps but subsequently decreases monotonically. The sharpness does not exhibit progressive sharpening process; instead, it fluctuates around / from the initialization. Figures 8 and 9 provide detailed illustrations of these observations. Based on these findings, we posit that the Eddge of Stability (EoS) phenomenon differs from that described Zhu al. 2022) and Kreisler al. 2023
B More Experiments and Details
B.1 Visualization of Practical Model Parameters in 3D Space
We visualize part of the GD trajectory with different learning rates, as well as the Gradient Flow (GF) trajectory, while training four-layer MLP on a binary classification problem using the CIFAR- 10 dataset. The GD trajectory closely follows the GF trajectory before entering the EoS regime. Subsequently, the GD trajectory enters an unstable region, beginning to oscillate, and its long-term movement shifts to new direction that differs from the GF direction. This behavior is similar to GD's behavior observed in our results presented in Figure Additionally, observe that GD with
13

---

10a2
2/8
14.
2/10
Loss
10
7 = 2/12
12
10-10
Sharpness
A= 10
10
A=8
7/ 2/8
2/10
7 2/12
4
200
400
600
800
1000
Iterations
Figure 8: Experiment from Zhu al. [2022). Gradient Descent (GD) applied to the loss function (1 2² which represents special case of scalar network setting. The learning rates are and 2.
chosen be 8 10
12.
10-1-
Loss
Amax
Sharpness
2/n
Loss
10-4
101
10-7
0
200
400
600
0
200
400
600
Iteration
Iteration
Figure 9: Experiment from Kreisler et al. 2023 A depth-4 scalar network is trained using Gradient
Descent (GD) for 10s steps with learning rate 0.2. The loss function is defined as ?
smaller learning rates follows the GF for a longer distance before beginning to change direction and oscillate in new region.
C Extension of the Setting
Here some extensions of our setting. First, by exploiting the rotational invariance of Gradient Descent in neural networks, we can assume that the input data æ E Id satisfies
= 1E + T212,
where U1 U2 E Fd, ||u
||22|| and (u1, U2) O. The pair (I1, and the target function f" Rdd follow the same assumptions as in original setting:
x1
N
A 2))) and f f*(x) 22.
x2
Then, the covariance matrix of the input given by
E Fixz" X \ of + X2U2217
This covariance can be a non-diagonal matrix in ddd indicating that our setting naturally extends to high-dimensional and non-diagonal data distributions.
In addition, the current setting can be extended to multiple features. Specifically, we can extend the input from (æ1, Ø2) to (I1, I2, 23), where 23 is independent of both æ1 and F2, and has the same
14

---

0.20
-0.2
V1
0.15
-0.4
-0.6
V2
0.05
0.10
-0.8
-1.0
-1.2
0.00
-0.10
-0.15
0.05
0.00
-0.10
-0.1
with Ir-1/15
GD with 1r-1/30 Gradient Flow Initia lization
Figure 10: Visualization of GD trajectory for a four layers MI.P training on Cifar10
22 + 23, and correspondingly extend the parameter vector from (o, B1, B2) to (a,, 31, 32, 83). The loss function then becomes
distribution 22. We redefine the target
3a + 22 1)2 >2 1)2
By symmetry, if the initialization satisfies (0) = 32(()) then 3r(1) = (4( for all Under this condition, the loss simplifies to
(1 + (2>2) - 1))
which corresponds exactly to the original setting with X₁ \i and X/ = 2d2, and the training dynamics remain equivalent.
Although this construction may appear somewhat artificial, demonstrates that introducing an additional independent variable with small scale has similar effect on the learning dynamics as increasing the scale of xz. If the initialization does not satisfy 82(0) = (O)),, the analysis becomes more complex, but the overall behavior remains closely aligned with the original setting.
D Theoretical Properties of Initialization
Our initialization allows explore the GD dynamics across wide range of learning rates, all starting from the same initial point. This stands in stark contrast to previous minimalist analyses such as Zhu al. [2022] and Chen and Bruna 2023), where their initializations permit only narrow selection of learning rates.
To this, define special set Y C 3 that is independent of the learning rate:
X X
v3
22
B7
X2
2V5
sa<1, max a, 10a
B2 < a
<
<
500
a3
\ιa
We then have the following proposition:
15

---

Proposition D.1. For any 0 E Y, there exists learning rate 1) - A2 rate r [0.55, 1], (rr)).
such that for any decay
Proof. Since 21a² 2 20 A2 we know that there exists a learning rate nf such that \ia²n 2. First, we show that E (n))
(i) 11 = and a > 0, so the condition
Ain a V Ain
is satisfied.
( S 32 is satisfied by the definition of Y. Since B2
v3
know
82 2 v33 Since
" x have
JonAi 20
I0o
3
S B2. So the condition max 20
20a
3
0
$ Bg
a
is
satisfied.
(iii) The condition for 81 is naturally satisfied by the definition of Y. Next, note the following property that for E [0.55, 1]. we have:
VornAi
Vond
1.1 S 2 2,
20
20
Therefore, is apparent that if 0 e X(n).. then for E [0.55, 1), 0 E Xrr)).
E Proofs for Section
E.1 Proof of Lemma 4.2 and Theorem
To simplify the proof, we assume K, which is the absolute constant in our setting for the lower bound of 2,2 satisfying K 10 First we prove the following Proposition:
Proposition E.1. For any 2 0, let f to be the first time such that Amaa? 2 1.5, then have:
1.5 2 (t) S 4.19 for t 2 P.
(E.1)
and (1) keeps increase before 7 (7 can be oo).
Proof. Note that use learning rate 77 S 0.1. The update equation can be written as:
alt + 1) = aa() 17 (AI - - (
(E.2)
B1(1+1) = -Clip (i((1) 1/ (1)8(()) 2 -
(E.3)
82(1 1) = (2(() + 112a((1((11 (1)) where Clip(z, = sign(z) min [x], We can get that:
(E.4)
(1 - att+++++)))) + 1)) < (1 (tt)
(E.5)
[ - 71)2a2()) 1)22 (100(()))(((((((()))0))))))))())
Note that the following are the initialization conditions:
V6n 3
1
1.1 (211a((0) 2, max
20a a()) - 32(00
(E.6)
20
(<0)'
A (0) (00) & (0)
(1 a())
(E.7)
a0)
\1a(1)
We set &t to be the largest time such that for any S t we have a and:
Maza 2 (t) + 3, (t) < 0.01.
We will first prove Proposition E.1 for 1. 5 + For < 1t + we know that in
E..5 we have:
,
1 - n2za² (t)
- Xga(t) att))2(t) 2 0.01
0.
4
16

---

Since (2 < 1, based on E.5) we can see that a(t) < 1 for S tf + 1. Therefore, if
Proposition E.1 holds for S et 1,
we will get axl)) > 0 for
S + 1, and we have:
nd + 1//2 (1) (t)
1/A (I)a?(1)
5
1
S
102 S 0.011,
X 1.1-102
which means t tt + thus tf oc. Therefore, it suffices to prove Proposition E.1 for S
which yields t" = oc and (t) < for all t.
Let to be the last time that \17)a((to) 2, with co. If a² does not increase monotonically before la, let i be the first time such that a² (41 + 1) < a² (t1), where 4 2 1. We have that for any
t < t1, (t) < 1 and:
1 1-0.01
0.01
4 2
(1 - ---(())2((t) 1 a(t + 1) + +1 S 0.85,
1) < 1+ ndd
a(11) (1+ >
a(1) (1+ 400 (002) a(l).
4a²(t)
471
When =1 - 1, we have:
(1 m))a2 (1+ 471na2(1))
1
1 7 0.98.
> 0.01
Thus, for 1:
Xina (t) 2 1.95.
Moreover, for t == 1 1, we have:
A211
R(1)) S (1) S
7.8"
Before tu. B² decreases monotonically. and when n)1 < 1.9, B8 decreases with an exponential
rate of at least 0.81. Using Bernoulli's inequality, have:
inf (nd1a²) 2 1.9 (1- StStoo
5
> 1.5.
7.8
Let 7 be the first time (1) 2 1.5. Then a²(1) monotonically increases before 7 and infigesto (TA)a?(1)) 2 1.5. This also holds if a²(() increases monotonically before lg- Next, we suppose to < oo to analyze what will happen if GD enters the unstable regime.
Let to be the first time after to such that n) 2, where to I co. If co, it is apparent that lim 2 = 2. Therefore, there will exist t < co, such that
A (ta
sup (m)1a?(t)). toct5t2
For to <1S 12, we have:
d2 X 1n
2))))
V d (1-)))((((((0((8))))))))))))))))
2
We can deduce that for t < < 4, [3) < ) Thus, we know that before 41, the update equation is the same as GD without any constraints. We set undefined coefficients n, 72 and an undetermined time t" such that:
Am (a (t + - a? 2 for t' < S to.
Xm
< n2
for to < tz.
where n1 > n2. We will estimate n2, n22, and iu later. Then we will have: Xina +t) 2 1.1 +
and \1na ?tt +
9m1
) 2 2. Then we have:
10
9mn- -10
to II (inja(()) 11)
+i
/2TT
(11)
10
2
II
10
0
71
Bmg
/T/1
n
e 3
17

---

We suppose that X177a (t1) = A + where À > 1. Then we will have (1) - - A
_
and
n2
(1 1)m²) 1.So we have:
(1)aa²() (1)--) 2 (Amz))
A
n2
II i1
(A- 1)22
ex-1
12ng
Therefore, we can obtain:
10
TO
A
2x
II
(Arna (1)n 2
10ng + 12m
1=
Since att + 1) & (t)) + -
we have:
A1 + 1) &
Auna (1) + 21/ Ao ( (106:000000000)-)000000000000000
S (t) + A122n 2
1223n1
2 +
16)2na²(t) S (t) + 200 + 1600
Given that 17 0.1, we can set n and ny such that m n2) 100. Since we know:
min {a(())2(1)) 2 max {a()))2(1)) 2 0.15, sStt
"SSiSo
can derive:
- sine (1))((0(11(())))))))) max att) at)) 2 2
1
suppose 1* is the largest time before to such that: o(to) If ((() S 50 (2)02(())
{1a(t)3((1) > 50 x232(t) (t)) for t=t 1.
Since before lg. we have (P(L 1) 2 (11--))33(()) we will have:
1 (1) a(t" )3²(t))) s {)a((T))0(()))
1 -(t))))0-)()()((()) 50
5000
Based on the definition of t", we obtain:
mine-sesa (t (()
1
max (a² (t+ (a at))
2.04
In conclusion, we can set n and n2 such that < 2.04.
Combining all the above equations, we have:
102/9
9m,/10
A
n2
V2A
100 when
A 3.18.
e
e--1
(10ma
Since a(1) 2 a(l*) and (1l*))22 (1-(((1111(111)))))1 (+())((((((=(((((((())))))) - we have:
(1n3((1) (41) > 10 Xnn3(
10'n
> 5. 103
2
(11((())8((16)) > d -2332110(1 - ((t)))2(1o)) then we can see:
Amns((t))a((tt)
10 (inn)(ttt))aa (tv)
10tn
5 103
-
18

---

Therefore, we have att + 1) < a(tz), which contradicts the definition of t1. So we can obtain 3.19. Consequently, we have proved that:
sup (nd1a? (t)) An +1 (t)) + 0.01 2 4.19 < 4.2. l<<1512
Next, we suppose L2 < oc to examine the consequences. Using the property of the constraint on 31 we can obtain lower bound for (²( as:
with constant c= 2 A2
Xn + 1) 2 (1 - >)nc)) A17a (t). Let t3 be the first time such that Anna (t) 1.7. is easy to see that:
Anna² (ts) 2 1.7 (1 - Anno2)22
Evidently, for 2 1s. we have:
1 (t + 1) = 3 + 1).
Therefore, by using Bernoulli's inequality, we obtain that for t 2 ts:
(1))a((1= 2 1.7 (1 2
Using the condition that n7 & DvAIA < 20
we get ( 2 1.5 for 2 Is- Then we
know (t + 1) 2 (t) for tz.
Let 44 be the first time after l3 such that a²(1) (1) starts to increase again. We will discuss whether 8 (14) will satisfy the initialization condition. The left side is immediately due to the definition of 44. Let
4 1. Since we have:
B2 a ((44)22((4) 2 (0.09 .7²X3 ) (1 25- (2(1)) - ((())(())(())((((44)))
the right side of the initial condition simply due to:
pl((a) 2 4 (P((W)
A2 ((E)))8((11)
4A1
A2a2 aa(a))
500
Since 32(0) 2 (0), induce that 32(1) 2 a(t) for all Since 82 will increase for all time,
we know B2(ta)= 2
VonA 20
So we get (2(4) 2 20 Then 44 will satisfy initialization condition. Therefore, we can repeat
the above proof from 14. So we know that Proposition E.1 is true for < + 1. Then we obtain that it and (2() for all t.
F. Proof Lemma 4.2
Since we have proved that (())3((1)) < 1 all f, according to the update rule of 32 (E.4). we can conclude that 31 + > 32(() for all This completes the proof of Lemma 4.2 (ii).
Next based on Lemma 4.2 (ii), we will prove Lemma 4.2 (i). Recall that the Hessian matrix for
0 = (a, 31, 32) is given by:
A1.87 + A232
2A1a31 X1a²
2d2aBB - -1
H(0)
2)
2220.32
Azaz
19

---

Since H0)2 = >1a², it is apparent that S(Ø) 2 A1a². Let us define matrices A and B as follows:
2x10s))) == 2220,82
22
2d20r32 X2
A =
2
X2
A2a2
We use || to denote the maximum singular value of any matrix M. It is apparent that:
S(0) < ||All + ||
Since we have proved that 0 < <a < 1, for B we have:
Bl12 = max {
|223382 + X20 ()) + 222 -
-1SIs1
S 12 + 22aa + A2.
Thus have:
|B1|
de
A2
X2 X
a
at
a1
- A
A2n
Aa
V)2
dn 0.0111.
10vA
It is also easy to see from the above equation that:
Tr(B)=
>2233 + >2a 2 0.012 2
For ||A||2, without loss of generality, suppose $1 2 0. Since we already know that:
1
v10
631
VAn7
let u = (t, /1 -V--/t (2) with E [-1 We then have:
lu Aul S a
12
87 a
/1 4 +2 (1-
-
()
+4
31
a
Thus have:
Iu= Aul
S (2 +
(1--)) (1
1
(1
2
< 1.105.
36
Therefore, we get ||Al|22 5 1.105 x a² Consequently, we obtain:
S(0) < ||A||2 | S (0.0111+ 1.105) Aia < 1.12 21a
Thus we have proved Lemma 4.2 (i).
Next we will prove Lemma 4.2 (ii) based on Lemma 4.2 (i)ii.). Following the notation in the previous
section, we aim to estimate the second eigenvalue of A, denoted as kj. We have:
k = tr(A)) Amax
is straightforward prove that Amax (A) 2 \1a² + +1137 = tr(A); thus, kn S 0. Let hig denote the second eigenvalue of B. We have:
k2
(20012 A2
0
22a
-2)
>0011
A2
2 -22-
0
2
Therefore, if (8) has any negative eigenvalue k then 2 ky
- -22. Let U1, uy, and Vy denote the
three eigenvectors of (9) and O) We express
== -1 + Φ2 + 0343,
where of + + 0z = 1. We then have:
\1a² = =(HH( < of Amxx (H())) + (tr(H(0)) - Amax (H())) - k) 03) < of (Amax (A) + Amax (B)) + (Ir(H(() Amax (A) - ki + A2) o
I of (1.105111 2 0.01201\ja + (tr (B) + A1) +1)
< 0 1.12 (0.012 0.001 )1 x
2
2 (1.120 0.011 (1)
Thus 1.12Ø7 + 0.013(1 - of) 2 1. This implies Φi 2 0.9, which means cos(e1,, u)] 2 0.9.
Therefore, we have completed the proof of Lemma 4.2 (ii)
20

---

E.1.2 Proof of Theorem 4.3
Theorem 4.3 is direct corollary of Proposition E. and Lemma 4.2 (ii).
E.2 Proof of Theorem 4.1
Based on Lemma 4.2 we have 32 O. Using the condition that (t) < and the lower bound of a in Theorem we can conclude that for any > 0, there exists & such that:
a(t))3(tt) 2 1 for 2 t..
(E.8)
Otherwise, (t) would approach infinity. If there exists T such that:
Anna² 22 for T2T,
(E.9)
then we have:
(1 2 (TTT > 0 for 12 T*
Based on 2 and (t)) < 1, can derive 82(()
A1 Using equation (E.8) it is
Ain straightforward to prove that there exists Ti > T such that:
A2 (t)) < A 3T )a(() for t TT
2
Thus, we have:
all + 1) 1) a(1) (T
which contradicts equation E.. Therefore, we have proven that there exist infinitely many t such that \1Taa² < 2.
there exists a > which does not satisfy the above proposition, then for any t > to. we have Any 248 Combining this with our previous conclusion, we find that there exist infinitely many
B2(t) < V t such that:
2
2 2 8
(1)
<V Am
2 5
This immediately contradicts Equation E.8 when we
V
2+0
2
So we have now proved
that for any 8 > 0 and > 0, there exists time T(5, e) such that for any 2 T(Ō,e):
a (t) 2 5,
1 a=t) S.
(E.10)
Finally, we will prove that for any d O, there exists time Ta such that (t) < d for t TT. If this were not the case, there would be infinitely many such that Xi70? (t) 2 1.9. This implies that
(2()) V Aun for any t. We set E > 0 such that:
AiTPEs = (1+ 12c + ) ) (I-bind v1.9
A1 (11-
When t > e(),e) and (t) 2 d, we have:
a(t + (t (t+ 1) (1+ A2ci )2c (2+) (1- +
AinRei
A1
v1.9
y.
This immediately contradicts our previous conclusion when we set =1 - > 0. Therefore, for any > there exists a time Ta such that (1) < d for > Ta. Combining this with Equation (F. we can conclude that for any 0, there exists a time T. such that for t 2 T., L(((t)) <
21

---

We now know that for any and 8, there exists T(8, e) such that for 2 c() we have:
(t) - 2 + 6, 1Bt(t)s.e.
Recall that for (a, 82)) we have:
S(0) max A (tga + t1 + 22 (tza + t112)² subject to
+ + + < 1.
When 31 = 0, 0. 1 - X xmj < 2 6, and 0 S B2 S _.4
we have:
S(0) max X1 2 + A (1za + (1 subject to
1 + =
max X + A2 + ( + +g) subject to 1
Since 2233 v 2 v \1 - 1009
we get that A2a + X2 < Aaa Thus, when 31 = 0.
100
0, 1 15
- naa
and < 32 S a
we have ||H())12 a 1 245
7
We can see that set A = {(a, 82, t1, t2, ts) a > 0,1 S \uaa² & 2 2 0 s 5 a 2 1} is compact in TR. Now we define:
F(a. 31, 32 21 t2, t3) = A1 (tza + +1332)² + A2 (tza + t1,222² ((31) = max
F(a, 31, 82, t 12, t3).
(2,
It is apparent that F is continuous function. Thus, based Berge's 's Maximum Theorem, g is also a continuous function. Since g(0) < 2 + ö, we know there exists es such that when [31 (1) S 6s. we have g(3)) < (2 + 25) Therefore, we know that for t > T(,, ((()))) s (2 + 28)/n.
thus completing the proof.
E.3 Proof of Theorem
We consider the following initialization:
1.1 S $ m))1a3(0)= S 1.5, [2(00) 2 max
VGrjAi 3
20 22aa(0) {(0)
22 500)1a(0) (1 a(0)32(0)) < 30) S
a0) (1 (0))
Based on the result and proof in Theorem 4.3 know that for any t 2 0, (t)) 2 0.15.
Let be the largest time such that a consistently increases before T, Ti the largest time such that (L)) & 0.85 for /. < Ti, T2 be the first time such that 2ua 2 2, and T3 be the first time such that
Bi(1) x² 50)
(1 a(1)
Since (1(1) < (0), we know T3 > min (1, T2). For / S T3. we have the following equation:
(t( a(t)
[2( 50)
< max
a(0) 49
Thus have:
50 a²(1)
(11)22(1) < 49 a0 (0)
Then, based on the range of (0)) and the range of (1) in the previous theorem, we have (TT))32(T) S 0.78, which yields Ti Ty. Thereby, we get T2 < Ta < Tí. We also know T > T3. Based on the definition of Tı, we know:
maxogrs7, (1))2(1) (1-0)))1))))20))) 2 miny;ge57, -(t))(()(1a((()2(()) ((t)B2())
22

---

Then we can easily obtain:
maxosIST. (a 11 a2)
minT (a - +1)- 2.1.
Similar to the previous proof, we set ng and n4 such that:
10 o 100r(aa(t+1)) - -t
for 0sts T2
R3
for T2 <tst
n4
Then we have min, m4) 2 100 and
na 2
1
n4
21
Similar to the previous proof, we obtain:
[100 2 (i) 1) <
T2
2
n5
A
n2
VV2
2A
10ng tug
12ng
0.1
when x= 1.37.
e
ed-1
If (T3) S 2.37, based on the fact that T3 S Ti, we have:
1 x²82(0)
1
-01(0))000( ((0){{())))
(T3)) S
50))a(0) (1 a(0))3(00)
100
100
50),a²(0)
1
(2)) (0))2((0) 1
-
$
40
50)1a²((33)
20
50A1a²
1 (T)) ((TT))
20
50))a((3)
This contradicts the definition of T3. Thus, we conclude that (Ta) > 2.37, which implies A1noa((T) (T) > 2.37 and S(0(T))
2.37
7
Since 2 2.37 for T3 S t < T, if 2 T3 + 9, we have either BT +8 2 2
AVAi
A
or:
()3)(13)( ((
(1-
BT 8)) 2 1.3716
50)
ATa)Ta)
322a (Ts) (2 a (Ts))
\Ta3(T3) 3. Aga(Ts + 8)3 + (1 (Ts + 8)3
ATa 8)
A1a(T3 +8)
In the first case, is also easy to prove that:
V
A2 4
Ara 2 + () +8)
(T 8)332 + 8) ((3 + 8)32(33 + 8)).
4nyA1
This shows that T' T +8, which contradicts assumption. Therefore, T3+ +8. Based on the limited movement speed of aB2,, we can easily prove that (TT) x (T)) + 4n/ S 0.82.
So we know that Ti 2 T.
Let T4 2 T be the first time such that
(Ta) >
Note that for the constraint constant we have:
Vx
Td
4/21
Let T3 be the first time after T such that 217ja² 2. Note that T3 < oc; otherwise, L(0) cannot converge, which contradicts Theorem 4.1 For T < < Tr, we have:
a(1 + 1) < a()) + 1/22.22 -1 (()))2(()), a(t + 1) 2 a(t) 1 (1d 2 a(t) - 400 (t).
23

---

Thus, for T Tt < Tá, we have:
(1+ (1- 1 "
) N11o
(E.11)
We already know that
(T( (2
(Ta(T) (1 ()-aa(T))22())
If 2 + let (T) 2 2.37. Then we have the following property:
(z - () 400 )²= 1)2 2
Using this property, we will have either (3((T + 2) 2 vd2 or:
A/A1
((T+2)2 (2 1)2 (1
22 -
(f 2 XT)
400
(T) (2T)
Thus, we get T4 2 T - 2, which leads to contradiction. Therefore, T4 S T + 2. Then, using (E.11)
we know T3 > T4.
For < T4 < Tg. we have
alt +1)+1)
=1+ (x23)((11--(()81100) a(l)
S1+ -x
400 <
S 1 +
a(l)
\17ja(11)
800'
1 1 82 1)
1 + 1//2.
(1
Based on the fact that 0.15 < a (2) s 0.82 and (T + 1) 2 1.37 1.37 (T)), is easy to prove
that for T < t T4, have a 1) a(t).
For T4 S
Ts. we have:
p(t) 2 (T(T2) -
Since 0.15 - (T)) < 0.85, we know:
(2
(1 -1-a(th al)))())2((t) (2()(1) (1)32(())
A1a(t)
)tna((t)
- ((1))2((((1a(())))
Amna((T) (Ma(()))(()1--a(()) (T)
-(((())(( Anna((t)
< 2 \ina?(T)
(1T)(( (TT
(T) a(T) < sf(t).
Thus, for 74 < < Ts. we have alt + 1) < at). Therefore, we conclude that for T < Ts. have att +1++) < att).
Let T3 < Ta S oo be the first time after T3 such that a(t + 1) 2 a(t). We know that a1 (Ts) < "
Recall that:
Ans8f + A2
2d10ß
2)20 - -12
H(0)-
2)10011
21a²
2222.32 - ーA
0
Aaa
T3 oc, then we know that |H(((T)))ll = 21T)((T)) < n
For the following part, we suppose To c0. At To. have:
(TT(TT) < (1 (TTa),
24

---

If (T6) < 1.75 then we directly have ||H(O(T))) & 1.12 1.75
<
2.
Next we suppose
(To) 2 L75 We then have:
(i)))) -
A271
[ 7
(To)
In the following proof, for conciseness, we use to denote (Ta). Let u = (11, t2, 13) TT We have:
T H(O)u S (A, + 12 + A + A1a + 41 + 222(22223) - 1)t1ts
2 (A1 - 22 + X2)1 + X1a + (A2a² + 12) + 4110811112
s (A15) - 22 + >2)th + X1a + (Aza² + 12) + 5 + 42
5
= (A139 + 12 + 2 + 5
\1 )イ + (Aza² + 5 + (Aza² 2 + +12)133
We note that:
2
2za 12 <
A2
4
2
2137 + 1202 + X2 +
21122 2
-
+
A2
+
21a
7
a²
5
7
2
2.01
a1 + +5 <
71
7
Thus, we know that for any u E R3, T (() S
2.01 Therefore, S(0(T))) S 2.01
E.4 Proof for Lemma 4.5
We have the following inequalities:
din
4.2 (2(
5 (1)
1 -
1.1 B2( Ain
2 (1)
2,32(1)
Airy (1
2
X
Using the condition
V Xim
B2( S 1, we obtain:
1
4.2
B2(()
4.2
22
1
2
B2(t)
V
2
11
82(t)
1 1.1
2
2
32(1)
V
2
Therefore, we can conclude:
[(0)
1
0.75 <
<
$ 3.3.
4 1 111
2
(
((1
4.2
2
V
Moreover, when (t) <
V A we have i(0)= < Lz)
2
E.5
Proof of Theorem 4.6
Based on the initialization and Theorem 4.3, we know that v28.(0
2 0.15 for t 2 0. According to
vAn
the condition given by Theorem 4.3 we only need to consider such that v28((1) condition, we can derive:
< 0.5. Using this
V2
22 Vdn
V2
22
1
< <a(()(1 a(1)3(()) 2
1
2 V
VAn
Vd
25

---

Thus we have:
(2(t) + 22
Vi
1- v232(t)
- 12( (2(t) + A22
2v2n
(1- (t)
(1
V2A1
vxun
V1
Viy1
Then we obtain:
( 1 (1- A1
412
(t)
(2
2 (1 1
A2
)
(1 1
v2
Vnn
VAn
A
Finally, we have:
1
11g) A1
)
2
(1
( 1-
2 (t) Vay
(2
(1-
d2
2
(2)
2
VA17
X
VAn
Therefore, we can observe that the decrease speed is exp( k22 for 1 I k S 4, and it independent of the learning rate 7). This indicates that the condition number of the input covariance matrix significantly affects the training speed in EoS regime.
X
F Proof of Section 5
F1 Proof of Lemma
First, we consider minimal 0- = (1, B2). We have B1 = and aB2 1. Let u tt)) IRS such that ||u||2 = 1. Then we have:
x² (²H(00))v )u A (12a)² + A2(13a + 11222² S 21a221 + >2(a² + +1)) + (2))
Since ff + 1/ = 1, we can see:
+ (1 - -2) S max{Aia² 22a2 + 22
Other other hand, when (0, we have S(0) 21 When /1 we have 5(0) Xza² + 22.33. Therefore, we know that
82
1/
Va=+03
0 +
S(0) max{A1a², A2a + +202).
Thus, we have
A2
S(0) = \ia² <> >11a 2 >2aa² + 22283 + a
>
d1
Aa
A2
A2
+ a² - 82
\ A2
A2
Now we consider gradient flow (GF) and let (B(0)) = (a(oo), 31 (DO), 32(0))) Since we know
that GF preserves layer norm difference as shown in Theorem 2.1 in Du et al. 2018)) we can get:
a²(2x) - (20) = =² - (0) - (0) < 0, (2()) = 1.
(F.I)
(F.2
Note that (1((0 <
Azn 4
We can then obtain:
a²( - (0) X2 S a - 12(00) S s0²(0) - (2)).)
(F3)
4
Based on the fact that a²) 2 x..
1 and (00) < 1, we have:
a²(00 - - A 10
V
Vxz
A
A1 A2
VA
10
40
A
A2
Aa
Thus get that d(0(0)) >1 (oo). Now we need to estimate the scale of a (o0).
26

---

Recall that a0) (0) _ 0. Using F.2 and a²()o) - -32(00) S Y, we can get a 2
y
Using (.2 and (x)) 32 (00) 2
v
we can get
2
40
Vx 40vA1
+ V 4 (r
)²
+ 77
a
20vAr
40vA1
2
Thus we can obtain:
A1 + 4
v2a
X1 / 72)
40
- <o(0(0)) = (00) 2
2
2
Thus when 0, we have:
VA122
(0(0)) 2 A1
2
A
40
4000
F.2 Proof of Lemma 5.2
x -
Let ((0(t)) = (a<(o)), (1)) Using the constraint constant
we obtain:
2x
Vx2 - - a² - (0))
a² - ()()
AV
Similar the proof of Lemma obtain:
5.1
derive o(0(t)) >1 (co). Using Theorem 4.3
4.2 (t).
32 (t) s a (00) - (20) (2(x) v n7\1
1
Following the proof in Lemma 5.1 we can derive:
1
1 - (2(()
2
a² (xx)
2
33(1) +
44
2
1/11
1/11
4.2
4.2
a²
(20)
- 2
(3((1)
+
4
- (3((1)
n111
11111
Consequently we obtain:
2
\17)3((1)
\1p
4+
mAi
1
<(e(t)) 2
+
217
2
2
4.2 (11
A
4+
TA1
12
(3
(((()) <
20
Since (2(() is monotonically increasing, both 7A1
() and
4.2 (1)
decrease monotonically.
nd1
Given that 7 + / + 72 is monotonically increasing function, we conclude that the two-sided bound of (0(()) a monotonically decreasing function
F.3 Proof of Lemma 53
Recall that the definition of is:
D = {0: 1 < Ainaa < -;00 a32 1}.
0,0
Using the expression of (0), for 0 E D, we know 3 = 0, and we can obtain:
22
2A2a A1
H(0)
0
21a
0
2220,32
2za
27

---

Let (0,1.0) E R We then have:
HO)u X1auu.
The other two eigenvalues are those of the matrix:
X2
2A2082 - A2 Aoa
B =
32 X2
Recall that we have proved in Appendix E.1.I that when a,32 < 1:
Amax
B
E B| $ X2 Agaa + 22-
Based on the condition in D that X 217
2 and 0 < a,3 < 1, we have:
62
1
< < X/17².
a
a
Thus, we obtain:
Amax
S (B) \2 a²² + +dza² + A2
s (A122122 + d2 + 127) a
0.0111 A1a2.
Therefore, 21 is the largest cigenvalue of (0) and u is its corresponding eigenvector. Thus, [[)(1] àι 2 < 2. Recall that the gradient of 31 is 2 31, which will be 0 when 31 0.
") Therefore, when 81 O, we have VL(0) u O. This proves that is in the "stable set".
Now we will prove that the update equation for the constrained trajectory is:
+ +2223000 - (185()) V +)(++))= Clip (at() CCip
2 m
(F.4)
(t = =)(()) + 1)22 (1) (1)(1 - a (1)
(F.5)
It is important note that (L) = 0 by the definition of ID. Based on the definition of ID and the initialization condition, we know al (0) (0) < First, we want to prove that if a (t) (t) < 1, then we will have F.4 and F.5). as well as a at(t + 1) + 1) < 1. Suppose at (t) < 1, then we have:
1 - (11()) 1223((1) (1:001111000 (1-a (1-a10)8(0)) (3)(() + (1) (1000(111)))) (1-a1(0))(()) +
= (1 - \211 A2 (a)(1)) + 21(1)) X3,7 (1)((()) (1 (1-a (1)((())) (1)00))))))
When a (t) < 1, based on the definition of D, we have:
A27 (a (t)² + =l(()) < (1)2 Azn 2h₂
S Xι
+ >1 < 0.011,
at(t)²
23n2
(1) (1 (1=aa(t)3((t)) -
< 0.001.
4
Thus, we know that if af (1) 1, then
(1-a . (8/(0) + m2 (aaa((((100/0) (1-01(0)3(0))) <1.
is then apparent that:
Clip (at(t) + (t) (1- (1-07())8(()) ./
2
ndi
(2(()) + 122 (1) (1-8(())8(()) <1.
28

---

Therefore, we know that:
Clip + (1-
nxi )
+ (1)- (1-a((0)8((0) D.
It is then apparent that:
CCip (100++))((((1100000000 (1-a)()8(0) .v 2
9 )o.
(5(() + m2za (1) (001))))))))))) (1-01(088(0))
= =Ilb (a'(t) + m223((t) (1-))(())))))))
(1))) + M2 (1) (1 (1=0(()))(0))
Thus, we have shown that if a (t) < we have (F44) and (5), as well as a (++1) 1. Therefore, can prove that for all and (F.) will hold by induction.
F44 Proof of Theorem 5.4
Based Equation F.4 Equation
F.5 and (t())((())
know that for all I we have:
al (t+ 2a B)(t + 1) > B5()).
Since we know that
1 (1-a (1=) > 0, 2
L (0)
then for all & we have
L (0t (t + 1)) < L (f))
Next, based on the definition of i and a + 1) 2 (t), we know that for all 2 E, we have a1(1)= = v
2 X.n Therefore, for 2 7 we have:
1
2
(0) 22 1
31(()
V
Amn
1)(++1)= (t) + v
2
(1-V
1
2
83(1)
Amn
Based on these two equations, it is easy prove that
1
/
2
(0t + 1))
A2
1
V
8)(t
1)
2
A1T
2\₂
2
\2
(1-V
1
81(()
X1
2
Aun
2\2
L (0t(1).
X
G Gradient Descent Dynamies for Unconstrained Optimization Problem
We present training results for the original optimization problem without constraints. The experiments use the same initialization set and corresponding learning rate as defined in Section The training dynamics of GD remain consistent with those observed in our constrained optimization setting, For example, the original problem, with learning rate of r = 30
I and an initialization from X ).) we trained the model using gradient descent for 15,000 steps. As shown in Figure [1 the results closely resemble the results reported in our main text: the training loss exhibits sharp spikes, and the sharpness displays recurring pattern of progressive sharpening followed by self-stabilization.
29

---

-52
logibss)
7.5
sharpmess
-102
-12.
15.2
-37.5
2003
6200
s200
8933
1000
1200s
36000
2020
100
6002
9595
1000 apocT
12000
Sacco 36000
02921
0.350
0.22
0.800
0.20
0.18
0.750
0.04
0.24
0.700
2000
4002
6000
8999
100ce
12000
10902
36000
2300
a030
6010
8003 10003
12000 upact
1800
Figure 11: In the original optimization problem, we A 100 and A2
We choose learning
rate r) 30 and choose initialization from X' (()) Then we train the model using GD for 17k steps.
H Discussion about the insights we provide
The key difference between our work and Rosenfeld and Risteski [2023] is that Rosenfeld and
Risteski [023] is empirical study offering insightful but informal hypotheses, while work
provides rigorous theoretical justification for the same phenomena, clarifies previously vague or
informal aspects of their explanation, and explains how the behaviors hypothesized in [1] arise.
Toward Understanding Progressive Sharpening: Rosenfeld and Risteski [2023] attempts to link sharpening with "sensitivity" of model, but it remains unclear which quantity precisely drives this sensitivity or why it increases progressively. Our work precisely characterizes the behavior of landscape sharpness in setting:
We prove it is tightly controlled by the final-layer L2 norm,
Sharpness grows linearly with the largest input eigenvalue A
And the 1.2 norm increases monotonically before a threshold, driving progressive sharpening.
This concretely explains why sharpening occurs and when it halts.
Why Dynamics Diverge After Initially Converging: Rosenfeld and Risteski [2023] speculates that optimizing for the relevant signal eventually amplifies the irrelevant but large scale feature e.g., the "sky" feature images, but offers no explanation for why this will happen. We show this phenomenon arises mechanistically in our setting:
Optimizing the relevant feature increases output-layer parameter a,
Which in turn amplifies the loss on irrelevant dimensions via 0,31,
This triggers divergence along the 31 direction.
Thus, offer a quantitative mechanism underlying the divergence pattern observed in Rosenfeld and Risteski [2023]
Why and How Dynamics Return to Stability: Rosenfeld and Risteski 2023 proposes two possible
(and incompatible) hypotheses: either the network "downweightss the inrelevant feature, flattening the valley" "catapults different basin," without determining which applies and why this will occur. Our analysis proves that:
Divergence eventually causes to decrease due negative gradients induced by large B1,
30

---

This leads to monotonic self self-stabilization, which pulls the sharpness back below the threshold,
No basin-hopping is needed result shows stabilization within the same basin. Hence, we explain both onset and recovery of instability.
In summary, our work builds a rigorous foundation for understanding Edge-of-Stabiliyy through theoretically grounded minimalist example. We clarify several previously open questions raised in Rosenfeld and Risteski [2023]. Our contribution lies not in proposing entirely new phenomena, but formalizing and proving mechanisms that were previously only hypothesized. Our framework also explains other empirical observations, such as the monotonic sharpness decrease of GFS in Kreisler et al. [2023], thus offering unified theoretical lens on EoS dynamics across settings with heterogeneous data distributions.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract accurately reflects the paper's contributions. It clearly states the development of minimalist model, the rigorous analysis of progressive sharpening and ceff-stabilizatinn, and the connection to broader theoretical frameworks, all of which are substantiated in the paper. The scope and claims are well-aligned with the technical content.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: For the feasibility of theoretical analysis, we introduce certain modifications. We discuss these modifications and verify that they do not affect our empirical results.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
32

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide complete theoretical result, and the complete proofs are included in Section Section E and Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our experiments simple numerical simulations designed to verify thcoretical results and illustrate the properties of our setting. We provide detailed descriptions of the experimental setup, making all results reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
33

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our experiments all numerical simulations, specific dataset is required. If the paper is accepted, we will provide our code, although it is very simple.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
gededdeii///////e/eeeeeeessssse/ CodeSubmiee isi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
primarily the choices of Justification: We provide the details of numerical experiments pepermmentt-pimmailly \1, A2 and learning rate
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Our experiments are deterministic and not related to statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our numerical simulations can be run directly on CPU and do not require any special hardware.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpus//neurips.. httpp//ppspcppubu.ttttttccccaaae
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is theoretical. There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
35

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
38


Reviews:
[{"review_id": "BzHo8iOpL9", "reviewer": "Reviewer_ApKd", "summary": "The paper analyzes the Edge-of-Stability and Progressive Sharpening phenomena for two-layer networks with a two-dimensional input, a hidden dimension, and an output dimension of width 1.", "questions": "# suggestions\n1. The paper cites the arXiv version for many papers, and not the conference version. For example, for [1,2,3], the paper cites the arXiv version instead of [1,2,3].", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. The minimalist example the paper analyzes exhibits the Edge-of-Stability phenomenon in a closer way than most other minimalist examples analyzed in other papers.\n2. The paper proves that for the minimalist example, many of the phenomena related to the Edge-of-Stability occur. That includes progressive sharpening, self-stabilization, and global convergence.\n3. The paper use an approximation for the loss, and shows that the approximation decays at a geometric rate.\n4. The paper shows the connection with previous works, namely:\n    1. The paper's minimalist example setting is motivated by [1], where they demonstrate that the oscillations during the Edge-of-Stability stage in image classification tasks are driven by “large magnitude” features in the input data. The paper's goal is to demonstrate the Edge-of-Stability in such setting.\n    2. They show upper and lower bounds on the GFS sharpness studied in Kreisler et al. [2]. They show that both bounds decrease monotonically, which indicates that the GFS sharpness also decreases monotonically, as demonstrated in Kreisler et al. [2].\n    3. They introduce a subset $\\mathcal{M}^{\\dagger}(\\eta)$ of the stable set $\\mathcal{M}(\\eta)$ studied in Damian et al. [3]. They show that for the projected gradient descent on $\\mathcal{M}^{\\dagger}(\\eta)$, the loss decreases monotonically.\n\n# Weaknesses\n1. While the paper does analyze the minimalist example extensively, in my opinion, it does not introduce a new understanding that can be applied to other problems that exhibit the Edge-of-Stability phenomenon.\n2. Some of the theorems and lemmas only hold up to some step number. Thus, only partially proving the phenomena related to the Edge-of-Stability.\n\n[1] Elan Rosenfeld and Andrej Risteski. Outliers with opposing signals have an outsized effect on neural network optimization. In International Conference on Learning Representations (ICLR), 2023.  \n[2] Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In International Conference on Machine Learning (ICML), 2023.  \n[3] Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In International Conference on Learning Representations (ICLR), 2023.", "was_revised": true, "final_justification": "My main concerns were whether the paper introduces a new understanding of the EoS phenomena, and the similarity between this paper and [1]. I have increased my score as the authors have addressed these concerns in their rebuttal.\n\n[1] Elan Rosenfeld and Andrej Risteski. Outliers with opposing signals have an outsized effect on neural network optimization. ICLR 2023.", "rebuttal": "Thank you for your review and letting us know your concerns! We will carefully address your concerns and questions below: \n\n**Weakness 1: While the paper does analyze the minimalist example extensively, in my opinion, it does not introduce a new understanding that can be applied to other problems that exhibit the Edge-of-Stability phenomenon.**\n\nWe understand your concern, but we believe our paper does introduce broadly applicable insights. As far as we know, our paper is the first theoretical work to establish how data distribution heterogeneity drives the EoS phenomenon.\n\nSpecifically, we study the setting where one feature $x_1$ is large-scale but irrelevant, while another $x_2$ is small-scale and relevant. Such construction is essential to reproducing key EOS behaviors including periodic progressive sharpening and self-stabilization, which has **not** been observed in previous minimalist settings (See Section 6.1). More interestingly, we show the decoupled behavior of parameters -- $\\beta_1$ associated with $x_1$ oscillates, while $\\beta_2$ corresponding to $x_2$ increases monotonically (Lemma 4.2). We also prove how the condition number $\\lambda_2 / \\lambda_1$ affects the loss decay rate (Theorem 4.6). \n\nThese theoretical results offer valuable insights on the central role of data distribution in driving EoS, a factor overlooked in prior minimalist examples where only a scalar input is considered [2,5]. Moreover, our work provides a new perspective for studying EoS in multi-feature settings, and establishes a concrete foundation for extending theoretical understanding beyond current frameworks.\n\n**Weakness 2: Some of the theorems and lemmas only hold up to some step number. Thus, only partially proving the phenomena related to the Edge-of-Stability.**\n\nWe acknowledge that some of our results hold for certain steps. This is due to the inherent difficulty of analyzing unstable and discrete optimization dynamics over long horizons. Importantly, this limitation does not compromise our core contributions. We rigorously prove the global convergence of the unstable dynamics and establish an upper bound on the sharpness of the final minimum (Theorem 4.1). We also demonstrate the existence of the EoS phenomenon throughout the entire training process (Theorem 4.3). Additionally, we show that the monotonic decrease of GFS sharpness observed in [2] persists in our example across all steps (Theorem 5.2). Together, these results provide a solid and novel theoretical foundation for understanding EoS.\n\n---\n\nThank you again for your review and for your suggestion! If you have any further questions or concerns, we would be happy to address them.\n\n**Reference**\n\n[1] Elan Rosenfeld and Andrej Risteski. Outliers with opposing signals have an outsized effect on neural network optimization. ICLR 2023.\n\n[2] Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. ICML 2023.\n\n[3] Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. ICLR 2023.\n\n[4] Jeremy M. Cohen and Simran Kaur and Yuanzhi Li and J. Zico Kolter and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021.\n\n[5] Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, Rong Ge. Understanding Edge-of-Stability Training Dynamics with a Minimalist Example. ICLR 2023"}, {"review_id": "PeiddNZkCn", "reviewer": "Reviewer_Yj6C", "summary": "This paper presents a simple problem to analytically derive the well-known phenomenon of Edge of Stability (EoS). The authors consider the training of a two-layer linear neural network of width one in a regression task. Under specific settings and assumptions, they show that progressive sharpening happens, followed by EoS, which is divided into two parts: sharpening and self-stabilization, which occur periodically. They also prove the global convergence of gradient descent in the large learning rate regime for this case.", "questions": "1) Can you generalize this result for GD without projection? Because in practice, EoS happens even without projection\n2) It seems that the problem is symmetric between the relevant and irrelevant parts. Can we switch the dominance such that the relevant part would be much larger?\n3) Is it possible to generalize the proofs by making the relevant and irrelevant parts of the input vectors instead of scalars? Here, we will have the same structure of the Hessian, but with blocks. In this way, we can derive a high-dimensional optimization problem that exhibits an EoS.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Well written\n- Try to follow the practice\n- Good connection with prior work\n- A simple problem that exhibits EoS, which one can analyze\n\nWeaknesses:\n- Too many constraints on the problem: initialization, condition number, learning rate, projection, etc. Namely, the conditions of the results are very specific, while the EoS happens in a much wider setting. Even in this simple problem.\n- Low-dimensional (High-dimensional problems can also be simple)", "was_revised": true, "final_justification": "The paper presents an easy optimization problem that exhibits EoS phenomenon. The authors addressed my questions.", "rebuttal": "Thank you for your review and for recognizing our work! We will do our best to answer your questions and address your concerns.\n\n**Question 1. Can you generalize this result for GD without projection? Because in practice, EoS happens even without projection?**\n\nThe projection is mainly introduced to facilitate our theoretical analysis rather than being fundamental to the phenomenon itself. Given the highly unstable dynamics inherent to EoS, existing theoretical tools are limited, necessitating certain modifications to make rigorous proofs tractable. We are actively exploring more effective theoretical approaches to analyze such instabilities. However, as we discussed in lines 145–152 and Appendix F, the same EoS phenomenon occurs in our setting even without projection.\n\n**Question 2. It seems that the problem is symmetric between the relevant and irrelevant parts. Can we switch the dominance such that the relevant part would be much larger?**\n\nNo, if we make such a change, numerically the EoS phenomenon would no longer occur. Due to space limitations, we are unable to provide a complete theoretical proof, but we can briefly outline the intuition as follows: If we switch the dominance such that the relevant part becomes much larger — or more simply, if we swap the scales of $x_1$ and $x_2$, then the loss function becomes:  \n$$\n\\frac{1}{2}\\lambda_2(\\alpha\\beta_{1})^2 +\\frac{1}{2}\\lambda_1(\\alpha\\beta_{2}-1)^2\n$$  where $\\lambda_2 \\ll \\lambda_1$. Now $\\beta_1$ becomes the stable direction and $\\beta_2$ becomes the sharp direction. By symmetry, we may assume without loss of generality that $\\alpha > 0$. Then, when the sharp direction $\\beta_2$ starts to oscillate, for every two steps, there is always one step where the gradient of $\\alpha$ is negative (the step when $\\alpha \\beta_2 > 1$). This means that once the dynamic starts to oscillate in the sharp direction, progressive sharpening can no longer occur, which is different from the numerical results in our original setting, where progressive sharpening and self-stabilization occur alternately and continuously.\n\n**Question 3. Is it possible to generalize the proofs by making the relevant and irrelevant parts of the input vectors instead of scalars?**\n\nYes, it is possible to generalize the proofs by extending the relevant and irrelevant parts of the input vectors from scalars to vectors. Here we provide a simple idea: we change the input from $(x_1, x_2)$ to $(x_1/2, x_1/2, x_2/2, x_2/2)$. Then the Hessian matrix becomes a $4 \\times 4$ matrix, which contains two $2 \\times 2$ blocks. We can prove that, in certain cases, the optimization dynamics in this high-dimensional space are fully equivalent to those in our setting.\n\nThere are also more general ways to extend our setting to high dimension. Due to the rotational invariance of gradient descent, for any high-dimensional data, as long as there are only two features and these two features satisfy the conditions we mentioned, whether these two features are scalars or high-dimensional vectors, we can use rotational invariance to reduce them equivalently to our setting.\n\n**Weakness 1. Too many constraints on the problem: initialization, condition number, learning rate, projection, etc. Namely, the conditions of the results are very specific, while the EoS happens in a much wider setting. Even in this simple problem.**\n\nWe fully understand your concern here. The projection in our theory is introduced to facilitate theoretical analysis, as such an optimization discrete and highly unstable process is very difficult to analyze theoretically. As demonstrated in the Appendix F, the dynamics without projection exhibit similar phenomena. \n\nFurthermore, we emphasize that other constraints—such as those on initialization and learning rate—are essential to enable the Edge-of-Stability (EoS) phenomenon. Even in practical neural networks, EoS does not arise under arbitrary initialization or learning rate choices, and this limitation is even more pronounced in our simplified setting.\n\nWe want to emphasize that when dealing with such complex and unstable optimization problems, introducing certain constraints to limit the scope of theoretical analysis is often necessary. Moreover, most of our constraints are not strict; they still allow for a considerable range of variability.\n\n**Weakness 2. Low-dimensional (High-dimensional problems can also be simple)**\n\nPlease see our answer in **Question 3**.\n\n---\n\nThank you again for your review and your recognition of our work! If you have any further questions or concerns, we would be happy to discuss them with you."}, {"review_id": "t5STvWSX80", "reviewer": "Reviewer_mhh3", "summary": "This paper presents a novel minimalist analysis of progressive sharpening (PS) and edge of stability (EoS) phenomena in gradient descent (GD). Specifically, the authors study a two-layer network of width one with a two-dimensional input consisting of one relevant feature and one dummy dimension. Among existing minimalist examples, this setup most closely reproduces both PS and EoS behaviors, thereby providing a clear insight into the mechanisms behind them. Furthermore, the authors connect their results to existing works (Damian et al., 2022) and (Kreisler et al., 2023), offering a broader perspective on these phenomena.", "questions": "- line 72-79: Could you elaborate on this point in more detail? I am having a hard time following the argument. Which part of the main section is this related to?\n- line 139: I understand the choice of $c$ being related to the condition number $\\lambda_2/\\lambda_1$, but why exactly  $c= \\frac{1}{2}(\\lambda_2/\\lambda_1)^{1/4}$?\n- line 154: Is choice of the initialization set $\\chi(\\eta)$ an artifact of the proof? Does this imply that for certain initialization either PS or EoS might not occur?\n- line 221: Why is $\\tilde{\\chi}(\\eta)$ more stable than $\\chi(\\eta)$?\n- line 247: I am not following the statement that $L_1$ drops to near zero whenever the GD goes back the stable region.\n- line 363: What do you mean by the statement that \"large-scale irrelevant feature will provide a regularization\"?", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Stregnths**\n- Constructing a minimalist example with a large-scale irrelevant feature, inspired by the findings of (Rosenfeld and Risteski, 2023), is both insightful and thought-provoking. The subsequent theoretical analysis confirms that this setting induces both PS and EoS behaviors more completely, unlike the existing minimalist analyses.\n- While the result itself may appear limited, the authors strengthen its impact by connecting it to two promising prior analyses of PS and EoS, thereby contributing to a more cohesive understanding of the phenomena. First, they show that GD reduces the sharpness of the gradient flow solution in their minimalist setting, extending the observation made in (Kreisler et al. 2023) for a scalar network. Second, they examine the behavior of GD along the constrained trajectory, inspired by (Damian et al., 2022), where the loss is shown to decrease monotonically. (I would appreciate if the authors could provide a clearer motivation of relating their work to (Kreisler et al. 2023) and (Damian et al., 2022), ideally in the introduction or at the beginning of Section 5.)\n\n**Weaknesses**\n- The justification on constraining the $\\beta_1$ value appears insufficient, despite some explanations provided by the authors. The statement that \"current optimization theories cannot handle instability\" while the paper is analyzing instability is somewhat unclear. While the intent seems to be that bounding $\\beta_1$ becomes intractable without constraints, this point warrants further elaboration. In fact, it seems that bounding $\\beta_1$ could have been a core theoretical contribution.\n- The analysis is limited to a specific set of initialization $\\chi(\\eta)$. It is unclear why this restriction is necessary from a theoretical standpoint. What would happen both practically and theoretically if this constraint was relaxed?", "was_revised": true, "final_justification": "While the paper has some limitations, these are well addressed, and the work makes a valuable contribution that merits publication.", "rebuttal": "Thank you for your review and for recognizing our work! We also appreciate the concerns and questions you have raised. We have carefully read the weaknesses and questions you raised, and we noticed that your question already covers the weaknesses you pointed out. Therefore, we will address your concerns by carefully answering your question.\n\n**Question 1. line 72-79: Could you elaborate on this point in more detail? I am having a hard time following the argument. Which part of the main section is this related to?**\n\nWe apologize for not indicating which part of the main text this discussion corresponds to, which may have caused your confusion.\n\nLine 72 -79 mainly corresponds to Sections 5.1 and 5.2 of the main text. In Section 5.1, we prove that on our constructed bivariate input $(x_1, x_2)$, the linear network exhibits the same phenomenon as in [1], namely that the GFS sharpness decreases monotonically.\n\nIn Section 5.2, we prove that under our bivariate input setting, the “stable set” assumed and studied in [2] does exist, and the result from [2]—that the loss decreases monotonically and smoothly over this stable set—also holds in our setting. This contrasts sharply with the result in [1], where it was shown that no well-behaved stable set exists under their setting.\n\n**Question 2. line 139: I understand the choice of being related to the condition number $\\lambda_2/\\lambda_1$, but why exactly $c = \\frac{1}{2}(\\lambda_2/\\lambda_1)^{\\frac{1}{4}}$?**\n\nThe constant $c$ does not need to be exactly $\\frac{1}{2}(\\lambda_2/\\lambda_1)^{\\frac{1}{4}}$, but it should be $O((\\lambda_2/\\lambda_1)^{\\frac{1}{4}})$.\n\nWhen the dynamic is unstable and $\\beta^2_1$ becomes large, we can approximately write the update rule for $\\alpha$ as:\n\n$$\n\\alpha(t+1) \\,\\approx\\, \\alpha(t) - \\eta\\lambda_1\\beta^2_{1}(t)\\alpha(t) = (1 - \\eta\\lambda_1\\beta^2_1(t))\\alpha(t)\n$$\n\nSo when $\\beta_1$ reaches the constraint $c$, considering the range of $\\eta$, $\\alpha$ fast decays at a rate bounded by $1 - c^2\\sqrt{\\lambda_1/\\lambda_2}$. Note that we have already stated in the main text that the sharpness of the Hessian is essentially $\\lambda_1\\alpha^2$. This brings us to a point that is difficult to handle using existing optimization theory: when the second order Hessian matrix varies quickly, how can we analyze the entire dynamics? We were not able to find a satisfactory method to handle this, so we had to restrict $c^2 \\sqrt{\\lambda_1 / \\lambda_2}$ to be within a constant. This implies that $c^2$ needs to be in $\\mathcal{O}(\\sqrt{\\lambda_2 / \\lambda_1})$, and therefore $c$ needs to be in $\\mathcal{O}((\\lambda_2 / \\lambda_1)^{1/4})$.\n\n**Question 3. line 154: Is choice of the initialization set $\\mathcal{X}(\\eta)$ an artifact of the proof? Does this imply that for certain initialization either PS or EoS might not occur?**\n\nThe initialization set $\\mathcal{X}(\\eta)$ is a technical requirement for our rigorous proofs rather than a fundamental limitation of the phenomenon. Numerical experiments demonstrate that EoS occurs for initializations in a set much larger than $\\mathcal{X}(\\eta)$. But of course, for certain initializations, either PS or EoS might not occur. For example, if the initialization is directly near a very flat minimal, then GD will simply converge to that minimal; if it is in an extremely sharp region, GD may diverge immediately. We think this should hold for any theoretical or practical model.\n\n**Question 4. line 221: Why is $\\tilde{\\mathcal{X}}(\\eta)$ more stable than $\\mathcal{X}(\\eta)$?**\n\nRecalling that $\\lambda_1\\alpha^2$ corresponds to sharpness, in set $\\mathcal{X}(\\eta)$ we require $\\alpha < \\sqrt{\\frac{2}{\\lambda_1\\eta}}$, while in set $\\tilde{\\mathcal{X}}(\\eta)$ we further require $\\alpha < \\sqrt{\\frac{1.5}{\\lambda_1\\eta}}$, which means $\\tilde{\\mathcal{X}}(\\eta)$ is a subset of $\\mathcal{X}(\\eta)$ with lower sharpness. In optimization theory, regions with lower sharpness are generally considered to be more stable. Therefore, we call $\\tilde{\\mathcal{X}}(\\eta)$ as a more stable subset of $\\mathcal{X}(\\eta)$.\n\n**Question 5. line 247: I am not following the statement that $L_1$ drops to near zero whenever the GD goes back the stable region.**\n\nThis is an empirical observation, though we acknowledge Figure 2 may not clearly illustrate this behavior. We will include a clearer figure in future revisions.\n\nIn fact, note that the update equation for $\\beta_1$ is given by\n\n$$\n\\beta_1(t+1) = (1 - \\lambda_1\\alpha(t)^2\\eta)\\beta_1(t)\n$$\n\nwhere $\\lambda_1\\alpha(t)^2 \\approx S(\\theta(t))$, so $\\beta_1^2$ decreases rapidly when $S(\\theta) < 2/\\eta$, which in turn causes $L_1 = \\frac{1}{2}\\lambda_1\\alpha^2\\beta^2_1$ to decrease quickly as well.\n\n**Question 6. line 363: What do you mean by the statement that \"large-scale irrelevant feature will provide a regularization\"?**\n\nThe large-scale irrelevant feature $x_1$ and its corresponding parameter $\\beta_1$ effectively constrain the sharpness. This is because the update equation for $\\alpha$ is\n\n$$\n\\alpha(t+1) = \\alpha(t) - \\eta\\lambda_1\\beta^2_{1}(t)\\alpha(t) + \\eta \\lambda_2\\beta_{2}(t)(1-\\alpha(t)\\beta_{2}(t))\n$$\n\nso when $\\beta_1$ becomes large due to oscillation, it suppresses the growth of $\\alpha$, thereby preventing the model from entering regions of high sharpness ($\\approx \\lambda_1\\alpha^2$). In this sense, it plays a regularization-like role. We apologize for not making this point clear. We will add a clarification in this part in the future revision.\n\n**Weakness 1. The justification on constraining the $\\beta_1$ value appears insufficient, despite some explanations provided by the authors...**\n\nPlease see our answer in **Question 2.**\n\n**Weakness 2. The analysis is limited to a specific set of initialization $\\mathcal{X}(\\eta)$. It is unclear why this restriction is necessary from a theoretical standpoint. What would happen both practically and theoretically if this constraint was relaxed?**\n\nPlease see our answer in **Question 3.**\n\n---\n\nWe sincerely thank you again for your recognition of our work! If you have any further questions or concerns, we would be very happy to discuss them with you!\n\n**Reference**\n\n[1] Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. *Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond*. ICML 2023.\n\n[2] Alex Damian, Eshaan Nichani, and Jason D Lee. *Self-stabilization: The implicit bias of gradient descent at the edge of stability*. ICLR 2023."}, {"review_id": "Mn85ZLz8to", "reviewer": "Reviewer_pR9a", "summary": "The authors construct a simple, analytically tractable example that exhibits key phenomena observed in deep learning training dynamics, specifically, Edge-of-Stability behavior and Progressive Sharpening when using large learning rates. Their model consists of a two-layer linear network trained with gradient descent on a two-dimensional input, separating task-relevant and irrelevant features. They provide a non-asymptotic, step-by-step analysis demonstrating that the training process self-stabilizes after progressively sharpening: the sharpness of the loss landscape consistently hovers near the critical $2/\\eta$ threshold, even as the loss decreases in a non-monotonic fashion. They connect this to notions such as stable set previously established in the literature.", "questions": "- Is the study specific to this simple two dimensional example or could it be generalized to larger dimensions and different target functions?\n- What happens if the K in line 124 is not very large ? Like for example if we suppose a standard gaussian ?\n- Can we conduct such an analysis without supposing a distribution on the data ?\n- Is a sharpness of $2/\\eta$ considered sharp or flat ?\n- The times T1 T2 T3 T4 can be a bit hard to grasp. Can you say for example to which epoch do they correspond on Figure 2?\n- In line 377, you say that the learning rate affect the distance between initialization and minima? Is it the case in practice? Aren't the weights initialized independently of the learning rate unlike in this setting ?\n- What would change if we use SGD or other optimizers used in practice to train deep neural networks?\n\nSome minor typos:\n- line 200 $\\alpha$ instead of $\\alpha_1$\n- line 283, should it be $\\phi(\\theta(t))$ ?\n- line 338 \"constrained\" instead of \"contrained\"\n- line 368 \"spikes\" instead of \"spiles\"", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Interesting theoretical study, which explains the EoS phenomena observed in deep learning in a tractable easy-to-interpret setting.\n- Clear assumptions and statements and helpful discussions.\n\nWeaknesses: \n- The 2D minimalist example, while elegant, may omit complexities in deep, realistic architectures. The main challenge is generalization of these results.\n- The study relies on strong assumptions. It is unclear how robust the analysis is to more generic settings.\n- The study concerns population loss while in practice we only have access to empirical risk.", "was_revised": true, "final_justification": "The paper presents good results, but it relies on strong assumptions.", "rebuttal": "Thank you for your review and for recognizing our work! We will do our best to answer your questions and address your concerns.\n\n**Question 1. Is the study specific to this simple two dimensional example or could it be generalized to larger dimensions and different target functions?**\n\nYes, there are many ways to extend our setting to the high-dimensional case. For example, due to the rotational invariance of gradient descent, for any high-dimensional data, as long as there are only two features and these two features satisfy the conditions we mentioned, whether these two features are scalars or high-dimensional vectors, we can use rotational invariance to reduce them equivalently to our setting.\n\nIn addition, there are also some methods that allow our setting to be extended to the multi-feature case. For example, we can change the input from $(x_1, x_2)$ to $(x_1, x_2, x_3)$ where $x_3$ is independent with $x_1, x_2$ and have the same distribution with $x_2$. Then we can prove that, in certain cases, the optimization dynamics in this high-dimensional space are fully equivalent to those in our setting.\n\n**Question 2. What happens if the K in line 124 is not very large ? Like for example if we suppose a standard gaussian ?**\n\nIn our numerical experiments, we found that similar phenomena generally occur when $K \\geq 20$. However, the imbalance in feature scales is the key to the emergence of these phenomena. If the data is changed to standard Gaussian (where $K=1$), the numerical results would be significantly different — in fact, the behavior in that case resembles more close to the results reported in [1].\n\n**Question 3. Can we conduct such an analysis without supposing a distribution on the data ?**\n\nOur analysis can be generalized beyond the Gaussian distribution.\nWe only require a distribution with zero mean and covariance matrix\n$\\begin{bmatrix}\n    \\lambda_1 & 0 \\\\\\\\\n    0 & \\lambda_2\n\\end{bmatrix}$\nto ensure that our loss function remains unchanged.\n\n**Question 4. Is a sharpness of $2/\\eta$ considered sharp or flat ?**\n\nIn our description, sharp and flat are defined with respect to a fixed learning trajectory $\\eta$. We treat $2/ \\eta$ as a threshold for $\\eta$: sharpness values smaller than $2/\\eta$ are considered flat, while those greater than $2/\\eta$ are considered sharp. We do not explicitly define the case where the sharpness is exactly equal to $2/\\eta$. However, since $2/\\eta$ is a single value, the probability that the sharpness exactly equals this value during training is zero. Therefore, the lack of a definition at this single point has no practical impact.\n\n**Question 5. The times T1 T2 T3 T4 can be a bit hard to grasp. Can you say for example to which epoch do they correspond on Figure 2?** \n\nThis is a great suggestion — we will include a figure showing $T_1$, $T_2$, $T_3$, and $T_4$ in future versions. Due to the NeurIPS rebuttal constraints, we are currently unable to include the figure. In Figure 1 (we refer to it as Fig. 1 since it is clearer), $T_1$ is approximately at epoch 600, $T_2$ around epoch 2200, $T_3$ around epoch 2215, and $T_4$ around epoch 3100.\n\n**Question 6. In line 377, you say that the learning rate affect the distance between initialization and minima? Is it the case in practice? Aren't the weights initialized independently of the learning rate unlike in this setting ?**\n\nIn practice, similar phenomena may indeed occur, and we have some preliminary results supporting this observation. We have conducted some simple experiments by training fully connected networks on CIFAR-10 (with standard initialization methods that are independent of the learning rate). In these experiments, EoS phenomenon will occur, and larger learning rates indeed tend to find solutions that are farther from the initialization. However, verifying whether this holds more broadly may require further practical experiments.\n\n**Question 7. What would change if we use SGD or other optimizers used in practice to train deep neural networks?**\n\nThere are some empirical studies on the EOS phenomenon in SGD, which find that small batches stabilize the sharpness at a level much lower than $\\frac{2}{\\eta}$. Therefore, the EoS of small batches is a very different story. For the EoS phenomenon under other optimizers, even empirical studies are currently limited, and there is a lack of sufficient insight for theoretical analysis. In our current work, we restrict our analysis to full-batch gradient descent, as the optimization dynamics in our setting are already highly unstable and challenging to analyze even in the full-batch regime. However, since our setting is sufficiently simple, it is indeed possible to study the EoS behavior of other optimizers within this framework. We will consider exploring these possibilities in future work.\n\n**Weakness 1. The 2D minimalist example, while elegant, may omit complexities in deep, realistic architectures.**\n\nPlease see our answer in **Question 1.**\n\n**Weakness 2. The study relies on strong assumptions. It is unclear how robust the analysis is to more generic settings.**\n\nWe acknowledge that our setting is a minimalist example has some assumptions. However, as we demonstrated in our response to Question 1, our setting can be easily extended to more general high-dimensional cases. In addition, we want to argue that some assumptions—such as those on initialization and learning rate—are actually necessary to ensure that the Edge-of-Stability (EoS) phenomenon can occur. Even for practical networks, EoS does not emerge under arbitrary initialization or learning rate choices, and this is even more true in our simplified setting. Moreover, the assumption on the condition number is in fact a core contribution of our work, as we found that the phenomena we demonstrate only emerge under distributions with relatively high condition numbers.\n\n**Weakness 3. The study concerns population loss while in practice we only have access to empirical risk.**\n\nWe acknowledge this point; however, we argue that this is a very common trick in theoretical analysis. For example, [1] [2] considers a simpler single-input setting, and [3] also replaces the empirical loss with the population loss in their final theoretical results. Theoretically, when the batch size is sufficiently large, optimization on the empirical loss closely approximates optimization on the population loss.\n\n---\n\nThank you again for your review and your recognition of our work! If you have any further questions or concerns, we would be happy to discuss them with you.\n\n**Reference**\n\n[1] Xingyu Zhu and Zixuan Wang and Xiang Wang and Mo Zhou and Rong Ge. Understanding Edge‐of‐Stability Training Dynamics with a Minimalist Example. ICLR 2021.\n\n[2] Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. ICML 2023.\n\n[3] Kwangjun Ahn and Sébastien Bubeck and Sinho Chewi and Yin Tat Lee and Felipe Suarez and Yi Zhang. Learning Threshold Neurons via the “Edge of Stability”. NeurIPS 2023."}, {"review_id": "i1Av3mCE2G", "reviewer": "Reviewer_EK41", "summary": "The authors theoretically study optimization of a two-layer, linear, width-1, scalar-output neural network with 2-dimensional inputs using projected gradient descent. The objective is the square population loss, where the data population has covariance with one large eigenvalue corresponding to noise (which plays no role in determining the target value), and one small eigenvalue corresponding to signal (which entirely determines the target value). The authors demonstrate that optimizing this objective with a moderately large learning rate exhibits progressive sharpening, followed by a transient edge of stability phase, followed by monotonic linear convergence of the loss to a global minimum. The authors also compare their setting and results with those of previous work.", "questions": "No questions.", "limitations": "Limitations (e.g. simplicity of the model) are not explicitly addressed by the authors. If the paper is accepted for publication, I would recommend including a discussion of limitations and directions for future work in the extra space provided.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n - The theory demonstrates all the features of large learning rate training demonstrated in Cohen et al 2021, specifically progressive sharpening and edge of stability. To my knowledge, it is the first to provide properly-grounded guarantees of these phenomena.\n - The theory indicates how the structure of the data plays a role in determining the unstable dynamics.\n\nWeaknesses:\n - The setting is quite limited and simplistic, but no more so than others considered in papers published at venues of equivalent prestige to NeurIPS.\n - The setting studied is projected gradient descent, rather than gradient descent. In this sense, the theory is a departure from the phenomena indicated in Cohen et al.", "was_revised": true, "final_justification": "I was already convinced that the work was a substantial contribution to the literature on edge of stability. The authors pointed out some ways in which their ideas could be generalised, which I think point to promising directions for future research. My rating remains a 5 (Accept).", "rebuttal": "Thank you for your review and for recognizing our work! We believe your evaluation of our paper is very objective. We will do our best to address your concerns below.\n\n**Weakness 1. The setting is quite limited and simplistic, but no more so than others considered in papers published at venues of equivalent prestige to NeurIPS.**\n\nYes, our setting is a simple example. However, as you mentioned, such minimalist examples are very common at NeurIPS and other top-tier conferences, and our setting is not simpler than those. At the same time, our setting is not a mere replication of previous work, as we introduce a new data distribution that connects to the EoS phenomenon, reproducing EoS behavior closer to practice and providing new insights.  \nMoreover, due to the rotational invariance of gradient descent, our setting can be easily extended to high-dimensional inputs with two features.\n\nIn addition, there are also some simple methods that allow our setting to be extended to the multi-feature case. We simply change the input from $(x_1, x_2)$ to $(x_1, x_2, x_3)$ where $x_3$ is independent with $x_1, x_2$ and has the same distribution as $x_2$. We also change the target $y$ to be $x_2 + x_3$. Next, we extend the parameters from $(\\alpha, \\beta_1, \\beta_2)$ to $(\\alpha, \\beta_1, \\beta_2, \\beta_3)$. Then the loss becomes:  \n$$\n\\frac{1}{2}\\lambda_1(\\alpha\\beta_{1})^2 +\\frac{1}{2}\\lambda_2(\\alpha\\beta_{2}-1)^2 + \\frac{1}{2}\\lambda_2(\\alpha\\beta_{3}-1)^2\n$$  Then, by symmetry, if our initialization satisfies $\\beta_2(0) = \\beta_3(0)$, then for any $t$ we have $\\beta_2(t) = \\beta_3(t)$, and the loss becomes equivalent to  \n$$\n\\frac{1}{2}\\lambda_1(\\alpha\\beta_{1})^2 +\\frac{1}{2}(2\\lambda_2)(\\alpha\\beta_{2}-1)^2\n$$  which corresponds to the case $\\lambda^{'}_1 = \\lambda_1$, $\\lambda^{'}_2 = 2\\lambda_2$ in the original setting with $(\\lambda^{'}_1, \\lambda^{'}_2)$, and the dynamics are essentially equivalent. Therefore, although our setting is simple, it can in fact be extended to more general cases.\n\n**Weakness 2. The setting studied is projected gradient descent, rather than gradient descent. In this sense, the theory is a departure from the phenomena indicated in Cohen et al.**\n\nWe acknowledge that the use of projection introduces a gap between our theoretical setting and practical implementations. However, the optimization dynamics in our setting are discrete and highly unstable, making the theoretical analysis extremely challenging. To make the problem more tractable, we introduce projection as a necessary simplification to enable a rigorous theoretical study. Our numerical experiments show that similar phenomena still occur without projection, indicating that our insights likely generalize beyond the projected setting. Extending the analysis to the unprojected case remains an important but technically demanding direction, which we aim to pursue in future work.\n\n---\n\nWe sincerely thank you again for your recognition of our work! If you have any further questions or concerns, we would be very happy to discuss them with you!"}]

Author rebuttal:

Meta-review: The committee reached a consensus by which this is a strong and insightful submission. The work provides a clean and minimalist study of edge-of-stability and progressive sharpening phenomena, offering both theoretical clarity and empirical demonstrations. Reviewers appreciated the novelty of the construction, the quality of the analysis, and the clarity of the presentation. While some concerns were raised about the scope and broader applicability, the authors effectively addressed many of these in the rebuttal, strengthening confidence in the contribution. Overall, this paper advances our understanding of phenomena of broad interest to the community. I recommend acceptance.
Decision: Accept (poster)