Title: A Minimalist Example of Edge-of-Stability and Progressive Sharpening
Venue: NeurIPS 2025
Accepted: yes
Keywords: edge of stability

progressive sharpening

gradient descent

training dynamics

nonconvex optimization

linear network
Abstract: Recent advances in deep learning optimization have unveiled two intriguing phenomena under large learning rates: Edge of Stability (EoS) and Progressive Sharpening (PS), challenging classical Gradient Descent (GD) analyses. Current research approaches, using either generalist frameworks or minimalist examples, face significant limitations in explaining these phenomena. This paper advances the minimalist approach by introducing a two-layer network with a two-dimensional input, where one dimension is relevant to the response and the other is irrelevant. Through this model, we rigorously prove the existence of progressive sharpening and self-stabilization under large learning rates, and establish non-asymptotic analysis of the training dynamics and sharpness along the entire GD trajectory. Besides, we connect our minimalist example to existing works by reconciling the existence of a well-behaved "stable set" between minimalist and generalist analyses, and extending the analysis of Gradient Flow Solution sharpness to our two-dimensional input scenario. These findings provide new insights into the EoS phenomenon from both parameter and input data distribution perspectives, potentially informing more effective optimization strategies in deep learning practice.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors theoretically study optimization of a two-layer, linear, width-1, scalar-output neural network with 2-dimensional inputs using projected gradient descent. The objective is the square population loss, where the data population has covariance with one large eigenvalue corresponding to noise (which plays no role in determining the target value), and one small eigenvalue corresponding to signal (which entirely determines the target value). The authors demonstrate that optimizing this objective with a moderately large learning rate exhibits progressive sharpening, followed by a transient edge of stability phase, followed by monotonic linear convergence of the loss to a global minimum. The authors also compare their setting and results with those of previous work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - The theory demonstrates all the features of large learning rate training demonstrated in Cohen et al 2021, specifically progressive sharpening and edge of stability. To my knowledge, it is the first to provide properly-grounded guarantees of these phenomena.\n - The theory indicates how the structure of the data plays a role in determining the unstable dynamics.\n\nWeaknesses:\n - The setting is quite limited and simplistic, but no more so than others considered in papers published at venues of equivalent prestige to NeurIPS.\n - The setting studied is projected gradient descent, rather than gradient descent. In this sense, the theory is a departure from the phenomena indicated in Cohen et al.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nLimitations (e.g. simplicity of the model) are not explicitly addressed by the authors. If the paper is accepted for publication, I would recommend including a discussion of limitations and directions for future work in the extra space provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors construct a simple, analytically tractable example that exhibits key phenomena observed in deep learning training dynamics, specifically, Edge-of-Stability behavior and Progressive Sharpening when using large learning rates. Their model consists of a two-layer linear network trained with gradient descent on a two-dimensional input, separating task-relevant and irrelevant features. They provide a non-asymptotic, step-by-step analysis demonstrating that the training process self-stabilizes after progressively sharpening: the sharpness of the loss landscape consistently hovers near the critical $2/\\eta$ threshold, even as the loss decreases in a non-monotonic fashion. They connect this to notions such as stable set previously established in the literature.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Interesting theoretical study, which explains the EoS phenomena observed in deep learning in a tractable easy-to-interpret setting.\n- Clear assumptions and statements and helpful discussions.\n\nWeaknesses: \n- The 2D minimalist example, while elegant, may omit complexities in deep, realistic architectures. The main challenge is generalization of these results.\n- The study relies on strong assumptions. It is unclear how robust the analysis is to more generic settings.\n- The study concerns population loss while in practice we only have access to empirical risk.\n\nQUESTIONS:\n- Is the study specific to this simple two dimensional example or could it be generalized to larger dimensions and different target functions?\n- What happens if the K in line 124 is not very large ? Like for example if we suppose a standard gaussian ?\n- Can we conduct such an analysis without supposing a distribution on the data ?\n- Is a sharpness of $2/\\eta$ considered sharp or flat ?\n- The times T1 T2 T3 T4 can be a bit hard to grasp. Can you say for example to which epoch do they correspond on Figure 2?\n- In line 377, you say that the learning rate affect the distance between initialization and minima? Is it the case in practice? Aren't the weights initialized independently of the learning rate unlike in this setting ?\n- What would change if we use SGD or other optimizers used in practice to train deep neural networks?\n\nSome minor typos:\n- line 200 $\\alpha$ instead of $\\alpha_1$\n- line 283, should it be $\\phi(\\theta(t))$ ?\n- line 338 \"constrained\" instead of \"contrained\"\n- line 368 \"spikes\" instead of \"spiles\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel minimalist analysis of progressive sharpening (PS) and edge of stability (EoS) phenomena in gradient descent (GD). Specifically, the authors study a two-layer network of width one with a two-dimensional input consisting of one relevant feature and one dummy dimension. Among existing minimalist examples, this setup most closely reproduces both PS and EoS behaviors, thereby providing a clear insight into the mechanisms behind them. Furthermore, the authors connect their results to existing works (Damian et al., 2022) and (Kreisler et al., 2023), offering a broader perspective on these phenomena.\n\nSTRENGTHS AND WEAKNESSES:\n**Stregnths**\n- Constructing a minimalist example with a large-scale irrelevant feature, inspired by the findings of (Rosenfeld and Risteski, 2023), is both insightful and thought-provoking. The subsequent theoretical analysis confirms that this setting induces both PS and EoS behaviors more completely, unlike the existing minimalist analyses.\n- While the result itself may appear limited, the authors strengthen its impact by connecting it to two promising prior analyses of PS and EoS, thereby contributing to a more cohesive understanding of the phenomena. First, they show that GD reduces the sharpness of the gradient flow solution in their minimalist setting, extending the observation made in (Kreisler et al. 2023) for a scalar network. Second, they examine the behavior of GD along the constrained trajectory, inspired by (Damian et al., 2022), where the loss is shown to decrease monotonically. (I would appreciate if the authors could provide a clearer motivation of relating their work to (Kreisler et al. 2023) and (Damian et al., 2022), ideally in the introduction or at the beginning of Section 5.)\n\n**Weaknesses**\n- The justification on constraining the $\\beta_1$ value appears insufficient, despite some explanations provided by the authors. The statement that \"current optimization theories cannot handle instability\" while the paper is analyzing instability is somewhat unclear. While the intent seems to be that bounding $\\beta_1$ becomes intractable without constraints, this point warrants further elaboration. In fact, it seems that bounding $\\beta_1$ could have been a core theoretical contribution.\n- The analysis is limited to a specific set of initialization $\\chi(\\eta)$. It is unclear why this restriction is necessary from a theoretical standpoint. What would happen both practically and theoretically if this constraint was relaxed?\n\nQUESTIONS:\n- line 72-79: Could you elaborate on this point in more detail? I am having a hard time following the argument. Which part of the main section is this related to?\n- line 139: I understand the choice of $c$ being related to the condition number $\\lambda_2/\\lambda_1$, but why exactly  $c= \\frac{1}{2}(\\lambda_2/\\lambda_1)^{1/4}$?\n- line 154: Is choice of the initialization set $\\chi(\\eta)$ an artifact of the proof? Does this imply that for certain initialization either PS or EoS might not occur?\n- line 221: Why is $\\tilde{\\chi}(\\eta)$ more stable than $\\chi(\\eta)$?\n- line 247: I am not following the statement that $L_1$ drops to near zero whenever the GD goes back the stable region.\n- line 363: What do you mean by the statement that \"large-scale irrelevant feature will provide a regularization\"?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents a simple problem to analytically derive the well-known phenomenon of Edge of Stability (EoS). The authors consider the training of a two-layer linear neural network of width one in a regression task. Under specific settings and assumptions, they show that progressive sharpening happens, followed by EoS, which is divided into two parts: sharpening and self-stabilization, which occur periodically. They also prove the global convergence of gradient descent in the large learning rate regime for this case.

STRENGTHS AND WEAKNESSES:
Strengths:
- Well written
- Try to follow the practice
- Good connection with prior work
- A simple problem that exhibits EoS, which one can analyze

Weaknesses:
- Too many constraints on the problem: initialization, condition number, learning rate, projection, etc. Namely, the conditions of the results are very specific, while the EoS happens in a much wider setting. Even in this simple problem.
- Low-dimensional (High-dimensional problems can also be simple)

QUESTIONS:
1) Can you generalize this result for GD without projection? Because in practice, EoS happens even without projection
2) It seems that the problem is symmetric between the relevant and irrelevant parts. Can we switch the dominance such that the relevant part would be much larger?
3) Is it possible to generalize the proofs by making the relevant and irrelevant parts of the input vectors instead of scalars? Here, we will have the same structure of the Hessian, but with blocks. In this way, we can derive a high-dimensional optimization problem that exhibits an EoS.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the Edge-of-Stability and Progressive Sharpening phenomena for two-layer networks with a two-dimensional input, a hidden dimension, and an output dimension of width 1.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The minimalist example the paper analyzes exhibits the Edge-of-Stability phenomenon in a closer way than most other minimalist examples analyzed in other papers.\n2. The paper proves that for the minimalist example, many of the phenomena related to the Edge-of-Stability occur. That includes progressive sharpening, self-stabilization, and global convergence.\n3. The paper use an approximation for the loss, and shows that the approximation decays at a geometric rate.\n4. The paper shows the connection with previous works, namely:\n    1. The paper's minimalist example setting is motivated by [1], where they demonstrate that the oscillations during the Edge-of-Stability stage in image classification tasks are driven by “large magnitude” features in the input data. The paper's goal is to demonstrate the Edge-of-Stability in such setting.\n    2. They show upper and lower bounds on the GFS sharpness studied in Kreisler et al. [2]. They show that both bounds decrease monotonically, which indicates that the GFS sharpness also decreases monotonically, as demonstrated in Kreisler et al. [2].\n    3. They introduce a subset $\\mathcal{M}^{\\dagger}(\\eta)$ of the stable set $\\mathcal{M}(\\eta)$ studied in Damian et al. [3]. They show that for the projected gradient descent on $\\mathcal{M}^{\\dagger}(\\eta)$, the loss decreases monotonically.\n\n# Weaknesses\n1. While the paper does analyze the minimalist example extensively, in my opinion, it does not introduce a new understanding that can be applied to other problems that exhibit the Edge-of-Stability phenomenon.\n2. Some of the theorems and lemmas only hold up to some step number. Thus, only partially proving the phenomena related to the Edge-of-Stability.\n\n[1] Elan Rosenfeld and Andrej Risteski. Outliers with opposing signals have an outsized effect on neural network optimization. In International Conference on Learning Representations (ICLR), 2023.  \n[2] Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In International Conference on Machine Learning (ICML), 2023.  \n[3] Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In International Conference on Learning Representations (ICLR), 2023.\n\nQUESTIONS:\n# suggestions\n1. The paper cites the arXiv version for many papers, and not the conference version. For example, for [1,2,3], the paper cites the arXiv version instead of [1,2,3].\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-11"}