On the Convergence Direction of Gradient Descent
Authors: Shuo Chen
Xiaolong Li
Jiaying Peng
Yao Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Gradient Descent
Edge of Stability
Convergence Direction
Primary area: optimization

Abstract: Gradient descent (GD) is a fundamental optimization method in deep learning, yet its asymptotic directional properties remain less understood. In this paper, we prove that if GD converges, its trajectory either aligns toward a fixed direction or oscillates along a specific line. The fixed-direction convergence occurs under small learning rates, while the oscillatory convergence behavior emerges for large learning rates. This result offers a new lens for understanding long-term GD dynamics. Experimentally, we find that this directional convergence behavior also appears in stochastic gradient descent (SGD) and Adam. Furthermore, we discuss how these theoretical findings regarding oscillatory convergence might offer a perspective on the sharpness dynamics observed in the Edge of Stability (EoS) regime. Our work provides both theoretical clarity and practical insight into the behavior of dynamics for multiple optimization methods.

Published as a conference paper at ICLR 2026
ON THE CONVERGENCE DIRECTION OF GRADIENT DESCENT
Shuo Chen
Xiaolong Li Institute of Information Science Beijing Jiaotong University Beijing, China lixl@bjtu edu.ccn.eeeccccc.
Institute of Information Science
Beijing Jiaotong University Beijing, China schen1307 wwx.aol....................of... com
Jiaying Peng
Yao Zhao
School of Mathematical Science
Institute of Information Science
Capital Normal University Beijing, China
Beijing Jiaotong University
Beijing, China yzhao@bjtu.edu
jiay saaynngenggecnu.. edu. cn
ABSTRACT
Gradient descent (GD) is fundamental optimization method in deep learning. yet its asymptotic directional properties remain less understood. In this paper, we prove that GD converges, its trajectory either aligns toward a fixed direction oscillates along specific line. The fixed-direction convergence occurs under small learning rates, while the oscillatory convergence behavior emerges for large learning rates. This result offers new lens for understanding long-term GD dy- namics. Experimentally, we find that this directional convergence behavior also appears in stochastic gradient descent and Adam. Furthermore, discuss how these theoretical findings regarding oscillatory convergence might offer perspec- tive the sharpness dynamics observed in the Edge of Stability (BoS) regime. Our work provides both theoretical clarity and practical insight into the behavior of dynamics for multiple optimization methods as well EoS.
INTRODUCTION
Gradient descent (GD) is one of the most extensively studied optimization algorithms. While clas- sical analysis typically focuses small learning rates, the precise structure of the GD trajectory remains less understood. Recent empirical lines of research, such as the observations regarding the Edge of Stability (EoS) (Cohen et al. 202T), highlight that GD exhibit complex, non-monotonic dynamics where the loss fluctuates over short timescales. Motivated by these rich dynamical behav- iors, this work focuses the asymptotic directional properties of GD, investigating theoretically how the trajectory aligns or oscillates relative to the learning rate.
A common framework for analyzing GD assumes that the function to be minimized is convex and L-smooth (Nesterov 2003). Formally, let f D C IR" + R be a differentiable function defined on domain D, satisfying the L-smooth condition, i.c., for all x, y D.
[V)(() ff('l < L|| - 1//
Conventionally, GD dynamics (x 1220 is defined as,
Fh++ F(*x);
where
F(x)
f(x))
By the classical descent lemma (Ahn et al. 2022), this framework ensures stable convergence and
allows for well-established convergence rate, the learning rate satisfies 0 < 1) 2/L. However, the condition on L-smooth qquation[]reqee requires that the constant L is sufficiently large. As a result, this condition usually fails neural networks.

---

Published as a conference paper at ICLR 2026
GD Trajectory Point
2.5
GD Trajectory
Initial
2.0
10
1.5
625
0.5
0.0
0.5
372
278
-05
0.5
1,0
2.0
60e
0.5
0.0
0.5
10
1.5
2.0
2,5
3.0
CD Trajectory
GD Trajectory
0.5
Initial Point
Initial Foint
0.05
0.5
224
1790
$1925
0005
128
0010
4.375
527M
0.0
630
-0.20
-025
Figure 1: GD trajectories for f(r, x2/2 2y² with 7 0.1 (upper left, the convergence direction exists as v-axis) and y = 0.42 (upper right, the convergence direction alternates along y- axis), and for f(rr,, y) x²² 2 2y² xyy² y with 1/ 0.1 (lower left, the convergence direction exists as r-axis) and 0.42 (lower right, the convergence direction alternates along y-axis).
Moreover, GD usually requires learning rate satisfying et al. 2022). where A, is the largest eigenvalue of the Hessian matrix at the local minimum. On the other hand, empirical results show that GD often continues to make progress even when the learning rate exceeds this bound at the first several GD iterations. More specifically. 2/n is smaller than the sharpness of æg for the first several 2g 's in the GD trajectory. Here, the sharpness is defined as the largest eigenvalue of the Hessian matrix 2gg- This observation raises fundamental questions about the underlying mechanisms that govern GD's long-term behavior and convergence. Many works have been devoted analyzing the GD dynamics from the perspective of stability, loss landscape.
7 < 2/An for convergence Ahn
and optimization trajectories (Ahn et al. 2022 Zhu et al. 2023 Damian et al. 2023 Lee & Jang
2023 Chen al. 2024 Cai al. (024).
In this work, the convergence direction of GD is theoretically investigated. Specifically, depending on the learning rate, we prove that, if convergent, the GD trajectory either aligns with fixed di- rection oscillates along specific line. To illustrate our idea, consider simple convex quadratic function [ax y) x 2 + 2y², in which (0, 0) the unique global minimum. In this case, the GD dynamics are given by Zg = (1 - a) and yk (1 4 yo- Consider then the convergence direction, i.e., the limit of normalized vector
(IK, yk)
UR
(x )ll
We have, for 2o # and Yo 0. lim Uy (sign(xo)) if < 17 2/5, and limg--se u (0, sign(yo)) if 2/5 < 17 < 1/2 (see Figure for illustration). Moreover, such phenomenon also holds if small disturbation exists. For example, consider quadratic function
2

---

Published as a conference paper at ICLR 2026
with high-order disturbation, f(2, y) = =2/2 2n² xxy² y², The similar results arise up under the similar settings (see Figure ass well).
This result suggests that GD iterates align with a particular direction when approaching the mini- mum, rather than converging in arbitrary directions. Our analysis generalizes this observation by characterizing the exact convergence direction of the GD trajectory, providing a refined understand- ing of GD's ss asymptotic behavior.
Our key contributions summarized as follows:
Convergence direction of GD: We show that when GD converges, its trajectory admits conver-
gence direction that depends on the learning rate Specifically, if 0 < 17 < 2/ + Xa), the GD trajectory aligns toward a fixed direction. In contrast, 2/(A1 + Xn) 1) < 2/ Xa, the GD
trajectory exhibits directional oscillations along specific line. Here, \1 and A, are the smallest and largest eigenvalues of the Hessian matrix at the local minimum respectively.
Insights for modern optimization methods: We observe that popular optimization algorithms such as stochastic gradient descent (SGD) and Adam also exhibit similar alignment behaviors in practice. Our findings may inform the design of new optimization algorithms that explicitly exploit this directional alignment.
2 RELATED WORKS
central theoretical foundation of our work builds on the celebrated proof of the Gradient Conjec-
ture, originally posed by René Thom and proven in Parusinski et al. (2000). The conjecture concerns the behavior of gradient flow trajectories near critical points of real analytic functions. Specifically, consider trajectory æt)) of the gradient vector field of a real analytic function f. with x (t) - Zo as
1 + 00. The Gradient Conjecture asserts that the normalized secants, lim, .r(t) [=] converge.
Actually, Parusinski et al. 2000) not only prove this conjecture but establishes a stronger statement: the radial projection of the trajectory onto the unit sphere has finite length. Their argument lever- ages Lojasiewicz inequalities, asymptotic critical values, and the construction of control functions. Notably, functions of the form g(x) = f(x)/r/ -a-r" grow fast enough along trajectories guar- antee directional convergence, where ||=]|| . is a small positive constant, / is a rational number, and a is a negative constant. In Xing al.] (2018), the authors analyze the trajectory of SGD in neural networks and observe that consecutive gradient steps tend to follow a valley-like structure in the loss landscape, often moving in directions that alternate or oscillate. Their results highlight that the directionality of gradient steps carries meaningful geometric information about the optimization path, even across non-convex and noisy settings. In Morchdi et al. (2023), the authors investigate the phenomenon of gradient oscillation in neural network training by analyzing the correlation between consecutive gradient directions. They observe that a significant portion optimization progress oc- curs during periods when consecutive gradient steps exhibit strong negative correlation, highlighting the complex and often non-monotonic behavior of gradient directions in practical training dynamics. Inspired by these works, try extend the continuous-time Gradient Conjecture GD. We show that if convergent, the GD iterates exhibit analogous directional behavior, which either converge to fixed direction or oscillate along line. This connection provides a bridge between the geometry of continuous gradient flow and discrete optimization dynamics.
The phenomenon of EoS is an interesting topic in understanding GD dynamics. In Cohen et al. (2021), the authors first observe that GD frequently operates in regime where the largest eigenvalue of the Hessian hovers around the critical threshold 2/n, challenging traditional stability analyses. This discovery motivates further research into how optimization methods interact with sharpness and stability. key extension of EoS appears in Sharpness- Aware Minimization (SAM) (Long & via
Bartleti 2024). In that work, the authors show that SAM dynamically adjusts the Hessian the gradient and perturbation radius, stabilizing training and promoting flatter minima. Moreover, Dai et al. (2024) discusses the impact of the batch normalization layer toward EoS. Recent work by Grimmer (2024) highlights that GD with periodically long steps can achieve provably faster convergence, despite violating descent at individual iterations.
Other works explore GD or SGD dynamics in multiple regimes under large learning rates (Hoffer et al. 2017 Li et al. 2019 Wu et al. 2023). Related studies examine curvature-aware learning rates
3

---

Published as a conference paper at ICLR 2026
Wang & Ma 2024 Cohen al. 2024), and nonmonotone line search method (Fx et al. 2024). which adaptively adjust learning rates near the EoS without manual tuning. The works (Ma et al. 2022 Lee & Jang 2023) further show that loss landscapes subquadratic near minima and propose interaction-aware sharpness measures for mini-batch settings. At the same time, using momentum may modify GD's EoS behavior. In PPhunyaphibarn al. (2024), the authors show that Polyak's s momentum leads to sharp curvature drops ("catapults") and raises the maximum stable sharpness, stabilizing training while promoting flatter minima. Another important direction is parameter averaging, and Nitanda et al. (2024) proves that averaged SGD smooths noisy updates and implicitly biases toward wider minima. More research on implicit bias can be found in Arora er al. (2022); Nacson et al. (2023). This complements EoS findings, as averaging helps mitigate the instability caused by large learning rates and sharp directions. Recent work also links EoS with multi-phase dynamics. For instance, Brana et al. (2022); Caa et al. (2024) observe that large learning rates in neural networks may induce the EoS phase and gradually more stable phases. In Yang et al. (024), the authors show that untuned SGD can achieve optimal rates but is sensitive to unknown smoothness constants, leading to gradient explosions. In Wang & Ma (2024). the authors identify dynamic transitions in gradient flow training. suggesting EoS interacts with deeper phase structures in training. More studies on dynamic of different phases can be found in Damian et al. (2023).
Thomas et al. 2020
Finally, the role of noise in SGD ties to EoS. For instance, MMlayoff et al. (2021) links large learning rates with smoother learned functions and shows that stable solutions depend on depth, supporting the regularization view of EoS. More research on the connection of geometry and dynamics can be found in Lee et al. 2016); Zhu et al. (2019): Martens (2020); Wann & Wul (2023). Overall, these studies show that normalization, momentum, averaging, and noise all shape the dynamics of training near EoS, highlighting the role of instability-driver behavior in modern optimization.
3 MAIN RESULTS
Consider a loss function f C (IR) and its GD dynamics defined in equation 2 and equation with learning rate 17. For an isolated local minimum xx of f. consider the GD trajectory {æx J220 converging to x. Specifically, we define V, = {xo E R= : lim Tg x }. Moreover, assume that for any subset W C R" with zero measure, (W) also has zero measure, where F is defined in equation 2 The zero measure condition of GD dynamics is a conventional assumption for technical reasons (Ahn et al.) 2022 Chen al. 2024). Then, our main result can be summarized as the following theorem.
Theorem 1: For the above defined function f. suppose the eigenvalues (AI...., do of V2 (a*)
satisfy 0 < A1 < X2 S S An-1 < <n,, for almost all initial points æn E Vn,
If0 < 1 < (/111 + >N,), the convergence direction of GD exists, i... lim
exists.
R-000 || 1 T
If (/(AA+A,)) lim (-1)*
1
2/An, the alternative convergence direction of GD exists, i.e..
exists.
k-+0
In this theorem, the condition for eigenvalues means that ʃ is locally strong convex function at æx. Moreover, for given > 0, the initial point set V,, is an open subset of IR" (Chen et al.) 2024). Next, we proceed to sketch the proof and the complete proof provided in the appendies.
3.1 PROOF SKETCH WITH < n 2/(x1 Am.
Without loss of generality, we assume x = O. Furthermore, we may assume that the Hessian matrix 22 (() is diagonal (see Appendix A for details), i.e., V f0) = diag(A1..... A,)) Define
1 nA,1 b b= max { 1)A:i)
2
Since 17 < 2/(11 (A,) < 2/2x, = 1/X,, follows that a > b 2 0. Let Fn be the i-th component of æh. then the GD iteration can be expressed componentwisely as, for each 1 < < n,
#k+1,; = 2b,i - f = (1 - 1)),) - (x()),
(6)
where gi is defined by, for x = (#1,..., Fn) E R"
(:(2)) 1/ (x)()))
(7)
4

---

Published as a conference paper at ICLR 2026
For simplicity, denote 9k.i (x(x))) throughout the following proof. Clearly, 9: (RR) and thus Gl. is twicely differentiable function with respect to Zy.- Moreover, by GD iteration equa- tion 2 Ik. can be regarded twicely differentiable function of xn- By the above preparation, first establish a forward-invariant set for GD and provide key estimates for Ok,i- These results are summarized in the lemma below.
Lemma 1: For the function f defined in Theorem 1, there exists an open set Ω C V, containing
xx 0 and a constant C' 0 such that for any initial point 2t E Q, the following properties hold,
For all 2 0, 3 E Q
For all k 2 and
S i n, 9 . S 2|
0ga. Brans s C|
For all k 2 and
S i.j S n,
The proof of Lemma draws upon the main theorem derived in Chen et al. (2024). which is stated below for clearity.
Theorem 2: [Theorem 2, Chen et al.
2024 Let f E (IR") and x* is local minimum. Suppose that there exists r > 0 such that f(x) [(x') holds for any x satisfying || (zz-a0 x=I = r. Moreover, assume that Vf(x) o for all E B(x",r)
{==}. If < n < 2/An. where X, is the largest eigenvalue of fa*), then there exists an open set U C B(x", r) containing x" such that the following forward-invariance property holds,
ZEU - -((()==n(((()) E..
Here, B (x",, r) is the open ball centered at x' with radius and B (xx,,)) is its closure. The foward-invariange is one of the most important properties for GD, which states that once the GD trajectory falls in to the forward-invariane set U, it will never escape from it.
Noticed that Lemma remains valid for all 0 < r < 2/An, and it will be utilized in the proof for both cases of small or large learning rates. Its proof is detailed in Appendix B. Next, leveraging Ω and C from Lemma 1, we select a constant > 0 satisfying
B(O,e) E) Ω Ce (a nCs)² <a, 8
3n2C'
With this choice, we first prove that the Theorem holds for almost all initial points æo E B(0,s). We now provide the following lemma.
Lemma 2: Consider the loss function f defined Theorem with learning rate 0 < < < 2/(Ar + A,). For the constant defined in equation 9 the following forward-invariance properties holds for 1 0,
vo B(0,E)
æ E B(O,s).
(10)
By this lemma, we see that BO, is also forward invariant set, i.c. the GD trajectories dived into B(0 =) remain confined within =). The detailed proof of Lemma 2 is provided in Appendix C.
Having established these lemmas, we now prove Thcorem for the case of small learning rate 0 < < 2/(11 + An). Firstly, define
S ao E B(0,E) Vk 2 0, |
<
(11)
One can prove that S has zero measure (the detailed proof of this technical issue is provided in Appendix D). Then we only need to consider 2o E B(0,s) \. S. Specifically, by definition of S in
equation 11 we only need to consider the initial point æo E B(0,s) such that there exists k* 2 0,
(12)
In this case, it's easy to see that [z .
2 |aga for 2 < i S n. Thus, by Cauchy's 's inequality,
| 1|2 x no?- .1 Then, according to equation 6 and Lemma 1,
| 2 |rae.al 1gx-,1| | C || 1| 2 |oeell nC
(13)

---

Published as a conference paper at ICLR 2026
Moreover, for 2 n, also by equation and Lemma 1,
[0 - [1 - 77 [ .al + lok ,I S b (ak + C. | 112 S bl lo .l + nC
(14)
follows that
1 S b bo nan 1)Cu
(15)
Based on equation
12 and equation
15 we have,
Ik +1 S b/222/ b + renn 1)Ca
(16)
Then, by equation 9 equation 13 and equation 16. with |ma S
| < <e,
IK +1.1 2
(17)
In this way, by Lemma 2, as ||æz||| E, we know that ||> < holds for any & 2 0. Then, by induction, we conclude that for all h k*,
| 2
(18)
Moreover, to ensure our conclusion, we have to show if ug-,1 > O. then for all l: 2 k* , 1 Actually, if x > 0, then by equation 6 equation and the same induction in equation [3]
>> 0.
1k+1,1
a - nCa221 2 (a nCE) DA.1 0.
(19)
The above claim can be then proved inductively. In the same way, if xx-,1 < O, then for all k 2 k", Tk,1 < 0. this situation, we know that I * 0 for all 2 k*, if Tbt,1 . * O. Furthermore, with equation 18 based on the same induction of equation 13 and | S s, we have, for all k: 2 k*,
|R41,1| 2 a | - nCa 2 (a - nCs) |=
(20)
Next, similarly, with equation 18 based on the same induction of equation for k: k and 2 S i S n, we have,
Fettal S b [oa,a + C ||=,.||" < b b|rr,,l + nCr
(21)
Since satisfies equation that q b/(a nCe)" 2 < i < n, we have,
then a nCs > b, one can pick satisfying < 2 such
1. Then, according to equation 20 and equation 21 for k
k and
1
nC |
[k
nC|xx, 2-a
1k+1,1 12 I
(a nCe)
(a - nCe)
|
a
q
(a - nC&) a.
(22)
Since | 1 < 1, equation
22
can be reduced
| [141111
S 9 | [IrA,1]
nC
+ (a nCE)aa.
(23)
Still, by induction, equation 23 implies that, for all k: 2 k" and 2 < S n
b-
nC (a - nCs)a
S
Σ 1=0
q'.
(24)
0k,1 a
k- o <1/(1 v). we have, for all k 2 k- and 2 < <
As Ei=0
nC lak-,l + (1- (a nCE)"
|x,11 18 S b
(25)

---

Published as a conference paper at ICLR 2026
Hence, there exists constant C1 0 such that the following estimation holds for k 2 k- and [
O. Therefore, based on 2 S i n, |_,,]| I C₁. As a result, for all 2 S i S n, lima Fbil
2k,
the above induction, on one hand, if 2 # 0.
lim
(sign(ZR-.1.. 0,.
(26)
This situation corresponds to the case where the GD trajectory eventually becomes dominated by the direction of the eigenvector corresponding to the smallest eigenvalue of V2 f2*")) This behavior is intuitive and expected for almost all initial points in (0,, e).
On the other hand, if T = O, we see that Zg. 0 since To $ S, meaning that the GD trajectory reached x= = 0 after only finite iterations. Define
WE = (ao E B(0,e) as 0}:
(27)
then we have To E Wt. According to the assumption on the GI dynamics in Theorem 1, W
is a zero measure set since W₁ = F= (10)), and thus the set SU (k) has zero measure.
In this way, for each initial point æo =0, \ (SU (UR=oWis) Wi)), according to equation 26 the convergence direction of GD exists. Finally, return to the set Vy, define
W (wo E Vn 2 = or o& E S).
(28)
is clear that the measure of W3 is zero, thus W also has zero measure. Then for each initial point E V,, (Uk2o Wa), ),, as its GD trajectories will finally enter the set B(0,a)M(C (UR Wx)), the theorem finally proved.
The detailed proof of Theorem 1 with large learning rate is provided in Appendix E.
3.2 SHARPNESS OSCILLATION AND DISCUSSION OF EOS
Consider a function f IR" + R satisfying the assumptions of Theorem I with a local min-
imum x=" 0. Suppose that the Hessian 72 f() have distinct eigenvalues satisfying 0
A < A2 < ... < An By classical matrix perturbation theory (Kato) 1995] (Hom Johnson 2013 Bhatia 2013 we know that there exists neighborhood U of xx such that the eigenvalues A ..... An (x) of f²f(#) are differentiable functions of æ, and X (xx) <... < (z)).. Then, by
Taylor expansion, we have, for the maximal eigenvalue X, (x) and x E U,
(x) = A, (0) + TTo x + +[[
(29)
in which, by definition, (0) = A, is the maximal eigenvalue at O and w (0). Accordingly, by Theorem 1, the GD trajectory 1e20 converges or alternativeyy converges along a certain direction E RR. As result, we may conclude that: For small learning rate 1 (0, 2/) + An)), there exists constant C, such that
(xxx = An + C2 + o(11=3+1)) I))
(30)
Therefore, ignoring the higher order term of || we have, Am (xx)) X, or (xxg) 1 An depend- ing on the sign of Czy- For large learning rate 77 E (2/(Az + An), An), there exists constant C such that
(ass) = An, + (-1) + o([|=x]])
(31)
Similarly, ignoring the higher order term of |æ we have, for instance, when C, > 0, An (xx)
will oscillate converge to Xn. i.e., (x2k) + X, and (x t >n, Moreover, as An. 2/1 < A1 + Xn, compared with the above case, the sharpness Xm (x22)) is more likely larger than 2/11 for the case of small k, i.e., for the first several Fg GD trajectory. The above discussion may provide new insights to the sharpness fluctuation phenomenon of EoS.
We now present an example to illustrate our idea. Consider here a loss function of two variables, f(x, y) _²² + +22/ + x²y + 2², which has a local minimum (0, 0) with λ 1 and X2 2. The GD dynamics and sharpness evolution under different learning rates are shown in Figure In Figure 2a and 2b] the leamingg rate is taken as 17 = 0.1 (/ A²) 2/3. In these two figures, the GD trajectory converges to the minimum along e = (0, F1), and the maximum eigenvalue X2(x)) converges monotonically to X2 2, consistent with our theoretical prediction for small
2nl
7

---

Published as a conference paper at ICLR 2026
GO Tatectory
60 Telectory
GD Trajectry
Intial Point Local Nirimum
Iriial Pait Local Nisimum
Intal Pork Local Nirimm
-25
Starpnccs
Sharoness
Sharpnoss
(a) 17 0.1, C₁
(b) 0.1, Cr =
(c) 0.95, C2
Figure 2: GD behavior on f(z, = = 22 z2y + x² under different learning rates. The top row shows the GD trajectories, and the bottom row shows the evolution of the sharpness along GD trajectory.
learning rates. On the other hand, in Figure 2c the learning rate 0.95 satisfies 2/(A1 + A2) 2/3 < 7 < 2/22 = 1. As a result, the GD trajectory exhibits oscillations along the sharpest direction (). Correspondingly, the maximum eigenvalue of the Hessian oscillates around the asymptotic value λ2, and may occasionally exceed the theoretical upper bound 2/1 s 2. 20105, which shown the dashed red line in this figure. This fluctuation is manifestation of the EoS phenomenon, but the long-term convergence of the oscillation envelope toward X supports the conclusion of our theorem. More results about this experiment are provided in Appendix
3.3 CONVERGENCE DIRECTIONS FOR MODERN OPTIMIZERS
Having rigorously established in Theorem that GD trajectories admit a convergence direction, we are curious whether this phenomenon might extend more widely used optimization algorithms in deep learning. Surprisingly. our experiments reveal that the existence of convergence directions is not confined to the deterministic and idcalized setting vanilla GD. Similar behaviors also emerge when using SGD with momentum and Adam in multi-class classification tasks.
To explore this, we conduct experiments on the CIFAR-I0 dataset, using a convolutional neural network architecture (the details provided in Appendix G). We train this network using two widely adopted optimization methods: SGD with momentum and Adam. For each optimizer, we monitor both the training loss and the directional alignment of successive update steps throughout training. In particular, we track the cosine similarity between consecutive GD parameter updates, defined
cos (AE++1, Art) = (Amr, Am
(32)
|Az ||
where Ary 3b+1 Ts. which captures whether the optimizer's trajectory aligns with stable direction over time. The results are shown in Figure 3 For all optimization methods, cos (A Axx) tends to when the loss function converges stably in descending manner, which indicates the convergence direction emerges and supports our claim.
This finding has important implications for understanding of optimization dynamics. It suggests that the asymptotic alignment behavior may reveal several promising research directions. The con- sistency of convergence directions across algorithms could inform better learning rate schedules,
8

---

Published as a conference paper at ICLR 2026
Loss
Convergence Direction Alignment
Do0 Similatl
GD
0.
Courne
60
Iterations
Berations
Cosine Simberkky
0.23
SGD
0.00
SGD
Covite Yalue
1730
450
L8/5
Cosino Srakkrrr
Adam
Adam
Cosine Value
3000
Figure 3:
Convergence directions for GD, SGD, and Adam. Each row presents the training loss (left) and the convergence direction alignment (right) for a different optimizer.
initializations, or optimizers. Understanding their emergence, especially in adaptive methods, may offer insights into convergence and generalization.
4 CONCLUSION
In this paper, we investigate the asymptotic directional behavior of GD and establish new theoretical results. Motivated by the Gradient Conjecture from continuous-time gradient flow, prove that under mild assumptions, the discrete-time GD also admits a similar directional property. Moreover, discuss the oscillating behavior of sharpness by deriving the variation of eigenvalues near the minimum. Furthermore, we also find that this result holds for SGD and Adam.
This work opens several promising directions for future research.
We only consider the locally strong convex function in the proposed theorem. It is important to investigate whether directional convergence can extended more general functions and other optimization algorithms, such momentum-based or adaptive methods.
A guantitative study of convergence speed in the direction could offer sharper theoretical guaran- tees and practical insights. Such results may inform the design of new optimization algorithms that leverage directional behavior for improved efficiency generalization. Understanding these aspects could help unify optimization methods under a broader theoretical framework.
An intriguing future direction is explore whether the phenomenon of convergence directions observed in GD is further amplified or structurally modified under the use of periodically long steps, suggested by the mon-monoto but globally contracting patterns in Grimmer (2024).
Building upon the Kurdyka-t.ojasiewice (KL.) inequality based works proposed in Qiu et al. (2024); Attouch et al. (2010). a promising direction is to extend the analysis to functions de- fined with the KL condition. Another interesting line of work is investigate the finite-time behavior and generalization performance of stochastic optimization under the approximate de- scent paradigm. Finally, integrating the KL inequality with data-depnndent sampling strategies or overparameterized models could provide new insights into convergence dynamics in modern machine learning.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China under Grants 62372037 and U24B20179.
REFERENCES
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gra- dient descent. In In Proceedings of the 39th International Conference Machine Learning, pp- 247-257, 2022.
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In Proceedings of the 39th International Conference on Machine Learning, pp. 948-1024, 2022.
Hédy Attouch, Jérôme Bolte, Patrick Redont, and Antoine Soubeyran. Proximal alternating mini- mization and projection methods for nonconvex problems: An approach based on the kurdyka- lojasiewicz inequality. Math. Oper. Res., 35(2):438 457, 2010.
Rajendra Bhatia. Matrix Analysis. Springer New York, 2013.
Yuhang Cai, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter L. Bartlett. Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization. In Advances in Neural Information Processing Systems, volume 37, 2024.
Shuo Chen, Jiaying Peng, Xiaolong Li, and Yao Zhao. On the unstable convergence regime of gradient descent. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11373-11380, 2024.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
Jeremy Cohen et al. Adaptive learning rates in the edge of stability. In Proceedings of the Interna- tional Conference on Artificial Intelligence and Statistics, 2024.
Yan Dai, Kwangjun Ahn, and Suvrit Sra. The crucial role of normalization in sharpness-awan minimization. In Advances in Neural Information Processing Systems, volume 36, 2024.
Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization:: The implicit bias of gradient descent at the edge of stability. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
Curtis Fox, Leonardo Galli, Mark Schmidt, and Holger Rauhut. Nonmonotone line searches operate at the edge of stability. In Proceedings of the OPT 2024 Workshop on Optimization for Machine Learning. 2024.
Benjamin Grimmer. Provably faster gradient descent via long steps. SIAM J. Optim., 34(3):2588- 2608, 2024.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017.
Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2nd edition, 2013.
Tosio Kato. Perturbation Theory for Linear Operators. Springer Berlin Heidelberg, 1995.
Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Proceedings of the 29th Annual Conference on Learning Theory, 2016.
10

---

Published as a conference paper at ICLR 2026
Seunghyeon Lee and Chulhee Jang. A new characterization of the edge of stability based sharpness measure aware of batch gradient distribution. In Proceedings of the 11th Interaational Conference on Learning Representations (ICLR), 2023.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neurul Information Processing Systems, volume 32, 2019.
Philip M. Long and Peter L. Bartlett. Sharpness-aware minimization and the edge of stability. Journal of Machine Learning Research, 25(179):1-20, 2024.
Chao Ma, Daniel Kunin, Lei Wu, and Lexing Ying. Beyond the quadratic approximation: The multiscale structure of neural network loss landscapes. Journal of Machine Learning, 2022.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 21(16)):-77, 2020.
Chedi Morchdi, Yi Zhou, Jie Ding, and Bei Wang. Exploring gradient oscillation in deep neural network training. In Annual Allerton Conference on Communication, Control, and Computing, pp. 1-7. IEEE, 2023.
Rotem Mulayoff, Tomer Michaeli, and Daniel Soudry. The implicit bias minima stability: A view from function space. In Advances in Neural Information Processing Systems, volume 34, 2021.
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability in multivariate shallow ReLU networks. In Proceedings of the 1Ith International Conference on Learning Representations (ICLR), 2023.
Yurii Nesterov. Introductory Lectures on Comex Optimization: A Basic Course. Springer US, 2003.
Atsushi Nitanda, Ryuhei Kikuchi, Shugo Maeda, and Denny Wu. Why is parameter averaging beneficial in SGD? An objective smoothing perspective. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, pp. 3565-3573, 2024.
Adam Parusinski, Tadeusz Mostowski, and Krzysztof Kurdyka. Proof of the gradient conjecture of R. Thom. Annals of Mathematics, 152(3):763-795) 2000.
Prin Phunyaphibarn, Junghyun Lee. Bohan Wang. Huishuai Zhang, and Chulhee Yun. Gradient descent with Polyak's momentum finds flatter minima via large catapults. In Proceedings of the 2nd Workshop on High-dimensional Learning Dynamics (Hil.D) at ICMI. 2024, 2024.
Junwen Qiu, Bohao Ma, Xiao Li, and Andre Milzarek. A kl-based analysis framework with appli- cations to non-descent optimization methods. CoRR, flss22006.0227.. 2024.
Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Pierre-Antoine Manzagol, Yoshua Ben- gio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect op- timization and generalization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pp. 3503-3513, 2020.
Mingze Wang and Chao Ma. Understanding multi-phase optimization dynamics and rich nonlinear behaviors of ReLU networks. Advances in Neural Information Processing Systems, volume 36, 2024.
Mingze Wang and Lei Wu. A theoretical analysis of noise geometry in stochastic gradient descent. arXiv preprini 0rXiv:2110.0066 2023.
Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along GD trajectory: Progressive sharp- ening and edge of stability. In Advances Neural Information Processing Systems, volume 35, 2022.
Jingfeng Wu, Vladimir Braverman, and Jason D. Lee. Implicit bias of gradient descent for logis- tic regression at the edge of stability. In Advances in Neural Information Processing Systems, volume 36, 2023.

---

Published as a conference paper at ICLR 2026
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with SGD. CoRR, abs/1802.08770, 2018.
Junchi Yang, Xiang Li, Ilyas Fatkhullin, and Niao He. Two sides of one coin: the limits of untuned SGD and the power of adaptive methods. In Advances in Neural Information Processing Systems, volume 36, 2024.
Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamies with a minimalist example. In Proceedings of the 11th International Confer- ence on Learning Representations (ICLR), 2023.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochas- tic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Proceedings of the 36th International Conference Machine Learning, pp. 7654-7663, 2019.
12


Reviews:
[{"review_id": "cgHV5P3Nb2", "reviewer": "Reviewer_UY6e", "summary": "The paper establishes the result on direction of GD convergence to the minima (with the $2/(\\lambda_1 + \\lambda_n)$ as the threshold between existence of direction and direction alternating at every step. A perturbative explanation for sharpness oscillations is given. Experiments conducted to show a presence of oscillations in the NN dynamics under GD/SGD/Adam.", "questions": "- What is the difference between your experiments and those of Xing et al.?\n- How does your sharpness oscillations results compare to those of Damian et al. “Self-Stabilization…”", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The proof is correct and carefully written, adding the control for higher-order terms; the experiments are clear in what they are measuring and well-explained.", "weaknesses": "Theory:\n    - unfortunately, there is no real novelty in the proven theoretical result (Theorem 1), as it is known in the classical iterative methods literature, see e.g. [1], Chapter 4. That is, your result is known for quadratics, the nonlinear extension is a technicality.\n    - Considering that this is the main part of the paper (see the details about empirical part below), this lack of novelty constitutes enough of a ground for rejection.\nConnection to EoS:\n    - The above phenomenon is not really related to EoS. EoS happens also (and specifically!) when dynamics are away from minima - and that’s almost the point of it, as the training continues as the network is in EoS (see original Cohen’s EoS, 2021). Your result is asymptotic near a strict minima, which makes them only connected because both are about oscillations, and that’s it\n    - there they are always zero (and negative) eigenvalues present in the spectrum of the Hessian of NN, see [2] or [3]. This makes the bound $2/(\\lambda_1 + \\lambda_n) = 2/\\lambda_n$, therefore eliminating the regime you term “large learning rate regime”\n    - Considering that you are discussing the oscillations of top eigenvalue during the EoS oscillations, you should contrast your work with Damian et al. “Self-Stabilization…” (2022), as this is precisely what this work is doing - showing how and why the sharpness oscillates and produced the self-stabilization\nEmpirics:\n    - the empirics are basically the experiments of “A Walk with SGD” of Xing et al. (2018), albeit with Adam added. They are measuring the exact same quantity, showing the oscillations. Therefore, there is no novelty in the experiments either. In particular, the GD experiments show the EoS phenomena (as later described by Cohen et al al. in EoS paper)\n    - Moreover, the experiments are unrelated to the theorem you are proving, see the above point about EoS being away from minima — that is, the oscillations start when we are away from minima, and there is no inherent reason why the oscillations in the experiments are the same oscillations that are talked about in the theorem. Therefore, it is unclear how the experiments support your finding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2IIlw4G4n", "reviewer": "Reviewer_i7ue", "summary": "The paper studies the asymptotic direction of gradient descent (GD) near an isolated local minimum. It proves that, if the GD sequence converges, then its normalized iterates converge either to a fixed direction (for smaller learning rates) or alternate between two opposite directions along a line (for larger but still stable learning rates). The boundary between these regimes is $2/(\\lambda_1+\\lambda_n)$, where $\\lambda_1$ and $\\lambda_n$ are the smallest and largest eigenvalues of $\\nabla^2f(x^*)$. The authors connect this to the Edge of Stability (EoS) by showing that, via a first-order expansion of $\\lambda_{\\max}(\\nabla^2f(x_k))$, sharpness can oscillate even as the loss converges. Empirically, toy 2D examples illustrate the two regimes, and preliminary experiments suggest similar directional alignment behavior for SGD and Adam on CIFAR-10 via cosine similarity of successive updates.", "questions": "1. Can you include more experiments as discussed in Weaknesses?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper pinpoints a clean, eigenvalue-driven dichotomy for gradient descent near an isolated local minimum: when $0<\\eta<2/(\\lambda_1+\\lambda_n)$, the normalized iterates converge to a fixed direction, whereas for $2/(\\lambda_1+\\lambda_n)<\\eta<2/\\lambda_n$ they alternate between two opposite directions. This result is stated cleanly (Theorem 1) with transparent assumptions and offers an interpretable link to the “edge-of-stability” phenomenon.\n\nThe analysis is technically careful: the argument builds an invariant neighborhood around the minimizer, bounds higher-order terms so the smallest-eigenvalue component dominates for sufficiently small $\\eta$, and identifies a measure-zero exceptional set of initialization. \n\nThe exposition is clear and visual: Sec. 3.2 connects directional behavior to sharpness oscillations via a local expansion of the largest Hessian eigenvalue, and Figs. 1 and 2 make the two regimes concrete on toy problems. Although preliminary, the empirical section suggests similar directional alignment patterns for SGD and Adam on CIFAR-10, hinting at broader relevance beyond plain GD.\n\nTo sum up:\n1. The paper has strong mathematical analysis with clear assumptions, statements and proofs.\n2. The work includes experimental work that compares the proposed method with other works.\n3. The writing is generally clear with nice flow.", "weaknesses": "My only concern is that the empirical validation is limited and largely qualitative: it relies on toy 2D landscapes and a single CIFAR-10 configuration, without uncertainty quantification, ablations, or explicit tracking of the normalized iterate direction toward a limiting vector/pair. Strengthening the evidence with additional toy problems and a broader deep-learning suite, e.g., multiple architectures on CIFAR-10 and CIFAR-100, would make the claims more compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cL1dNH79X5", "reviewer": "Reviewer_dfgz", "summary": "This paper study the convergence in direction of the iterates of gradient descent. The main results of the papers identifies two regimes. If the step size $\\eta$ verifies $0<\\eta<\\frac2{\\lambda_1+\\lambda_n}$ where $\\lambda_1$ and $\\lambda_n$ are respectively the smallest and the largest eigenvalues of the Hessian of the objective function $f$ at a critical point, the iterates of gradient descent converge in direction. If $\\frac2{\\lambda_1+\\lambda_n}<\\eta<\\frac2{\\lambda_n}$, the iterates exhibit an alternating convergence direction. The paper then discusses the implications of these results for the edge of stability and provides numerical experiments suggesting that the theoretical findings extend to other optimizer such as SGD with momentum and Adam.", "questions": "* **Q1**: In the experiments with neural networks, how do the learning rates compare with the sharpness? It seems that only the convergence direction occurs. Do SGD and Adam exhibit alternative convergence direction with higher step sizes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* **S1**: The authors establish convergence direction of gradient descent. The result is sound and novel to my knowledge.\n\n* **S2**: The established results provide insight on the behavior of the sharpness during the optimization trajectory.\n\n* **S3**: Numerical experiments suggest that the theoretical results established for GD may extend to SGD with momentum and Adam.", "weaknesses": "* **W1**: The experiments appear to be based on a single run. Since some of the compared algorithms are stochastic, multiple runs should be conducted. The results should then report the average or median cosine metric along with standard deviations or quantiles to reflect variability and ensure statistical reliability.\n\n* **W2**: The code of the experiments is not provided, hindering reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9DHCNfur3y", "reviewer": "Reviewer_Cqt9", "summary": "This paper studies the asymptotic behavior of gradient descent as it converges to a minimizer, on generic objective functions. The main assumption is that the objective is strongly convex around the minimizer (i.e. the Hessian at the minimizer has no zero eigenvalues).   The paper proves that there exactly two possible modes of convergence: either gradient descent converges along a specific direction (i.e. cosine distance between successive iterates is $\\approx 1$), or it oscillates along a particular line (i.e. cosine distance between successive iterates is $\\approx -1$).   They prove that the first case happens when $0 < \\eta < 2 / (\\lambda_1 + \\lambda_n)$ and that the second case happens when $2 / (\\lambda_1 + \\lambda_n) < \\eta < 2 / \\lambda_n$, where $\\lambda_1$ and $\\lambda_n$ are the largest and smallest Hessian eigenvalues at the minimizer.\n\nI think the basic intuition comes from the case of a quadratic objective.  Here, the rate of convergence along each eigenvector direction is $1 - \\eta \\lambda_i$, where $\\lambda_i$ is the eigenvalue.  Asymptotically, we will be converging along the eigenvector for which $|1 - \\eta \\lambda_i|$ is largest.  There are two possible cases: \n  - if $\\eta < 2 / (\\lambda_1 + \\lambda_n)$, then this direction will be the eigenvector corresponding to the _smallest_ eigenvalue $\\lambda_n$ and the contraction factor $1 - \\eta \\lambda_n$ will be positive and we will converge along that particular direction\n  - if $\\eta > 2 / (\\lambda_1 + \\lambda_n)$, then this direction will be the eigenvector corresponding to the _largest_ eigenvalue $\\lambda_1$, and the contraction factor $1 - \\eta \\lambda_1$ will be negative and we will oscillate along that line.\n\nThe paper basically makes this intuition precise for general objectives.", "questions": "- What happens when the learning rate is greater than $2/\\lambda_n$ -- can you prove that convergence is impossible?\n- For the experiments in section 3.3, I think you used cross-entropy loss.  For cross-entropy loss, optimizers generally leave EOS before the end of training (see Cohen et al '21, Appendix C), which is what we see here.  I think that if you used MSE loss, you might see the other, oscillatory convergence regime.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper nicely tackles an interesting problem in a fully rigorous manner.", "weaknesses": "The asymptotic behavior of an optimizer near a minimum may not be that relevant for practical deep learning tasks, where practitioners generally care about the behavior of the optimizer far from any minimum.\n\nAnother weakness is that the paper only studies the case where gradient descent converges to a point, whereas it is also possible for gradient descent to wind up in a orbit where it oscillates indefinitely, for example see [Chen and Bruna '23] or [Ghosh et al '25].\n\nLei Chen and Joan Bruna.  \"Beyond the edge of stability via two-step gradient updates.\"  ICML '23.\n\nAvrajit Ghosh, Soo Min Kwon, Rongrong Wang, Saiprasad Ravishankar and Qing Qu.  \"Learning Dynamics of Deep Linear Networks Beyond the Edge of Stability.\"  ICLR '25.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V45ubLAGe1", "reviewer": "Reviewer_YPEn", "summary": "Extending to the discrete setting a result known for the gradient flow, the authors characterize how gradient descent, assuming it converges to some local minima, will converge to this limit. They show that, depending on stepsize, it will either converge along a fixed direction, or oscillate along it. Numerical experiments are provided on toy examples to illustrate this result, and also to observe empirically, on a training neural network task, how behave the angle between consecutive iterates for several optimizers. A section is devoted to the connection of their work with the Edge of Stability (EoS) phenomenon.", "questions": "- On figure 2)a), as the convergence occurs along the y-axis direction, why the sharpness does not converge to 1 instead of 2 ?\n\n- Under equation (5), the justification that $a > b $ seems evasive to me. Could you elaborate ?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The presentation is clear, the paper well organized. I think not assuming global smoothness a priori and instead characterizing the convergence according to the eigenvalue of the Hessian at a local minima is an interesting viewpoint, such that the main result (Theorem 1) is interesting.", "weaknesses": "I express severe doubts about the relevance of the connection established between the author's work and the EoS phenomenon, which is presented as a central contribution. \n\n\n- In their main result, the authors assume the function is $C^3$, and that gradient descent converges to a strict local minimizer. Because of the $C^3$ assumption, there exists a vicinity of this local minimizer such that the function is strongly convex and $L$-smooth for some $L$, with $L$ as close as we want to $\\lambda_n$. \nAs gradient descent is assumed to converge to this local minimizer, the algorithm eventually enters this vicinity, and stay inside it. With this regard:\n\n     (i) It is implicitly assumed that asymptotically, the algorithm lies in a strongly convex, $L$-smooth landscape, with $\\eta < \\frac{2}{L}$. \n\n     (ii) In this setting, even if the sharpness exhibit oscillations , the loss still decreases monotonically, as it seems to be observed on figure 2.c. This is not consistent with the typical behavior of the EoS regime, where the loss is not monotonically decreasing, but only decreases in the long run.\n\n     (iii) According to [1], a quadratic growth property does not fit the local behavior of neural network losses around minimizers, whose growth is more likely to be sub linear. Such a setting is not covered by the implicit \"local strong convexity\" assumption of the authors.\n\nFor those reasons, it is not clear to me whether this work improves or not our understanding of the EoS phenomenon. I believe the claimed connection should be substantiated more clearly.\n\n[1] Helusive traductionao Ma, Lei Wu, and Lexing Ying. The multiscale structure of neural network loss functions: The\neffect on optimization and origin\n\nMinor remarks \n- The labels on figure 1 are very small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)