Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region
Authors: Shuang Liang
Guido Montufar
Venue: ICLR 2026 Poster (2026)
Keywords: large step size
gradient descent
matrix factorization
convergence
implicit bias
chaos
fractal basin boundary
Primary area: optimization
TL;DR: Gradient descent with near-critical step sizes enters a chaotic regime, characterized by sensitivity to initialization, fractal convergence regions, and absence of simple implicit biases.

Abstract: We examine gradient descent in matrix factorization and show that under large step sizes the parameter space develops a fractal structure. We derive the exact critical step size for convergence in scalar-vector factorization and show that near criticality the selected minimizer depends sensitively on the initialization. Moreover, we show that adding regularization amplifies this sensitivity, generating a fractal boundary between initializations that converge and those that diverge. The analysis extends to general matrix factorization with orthogonal initialization. Our findings reveal that near-critical step sizes induce a chaotic regime of gradient descent where the training outcome is unpredictable and there are no simple implicit biases, such as towards balancedness, minimum norm, or flatness.

Published as a conference paper at ICLR 2026
GRADIENT DESCENT WITH LARGE STEP SIZES: CHAOS AND FRACTAL CONVERGENCE REGION
Shuang Liang UCLA angshuang@g ucla edu
Guido Montúfar UCLA & MPI MiS
montufarêmatio ucla a edu
ABSTRACT
We examine gradient descent in matrix factorization and show that under large step sizes the parameter space develops a fractal structure. We derive the exact critical step size for convergence in scalar-vector factorization and show that criticality the selected minimizer depends sensitively on the initialization. Moreover, we show that adding regularization amplifies this sensitivity, generating a fractal boundary between initializations that converge and those that diverge. The analysis extends to general matrix factorization with orthogonal initialization. Our findings reveal that near-critical step sizes induce chaotic regime of gradient descent where the training outcome is unpredictable and there are no simple implicit biases, such as towards balancedness, minimum norm, or flatness.
INTRODUCTION
Understanding the properties of gradient descent non-convex overparametrized optimization has been a central pursuit in modern machine learning. The step size, or learning rate, is a critical factor determining the dynamics and convergence of gradient descent optimization. In particular, it has major influence on the returned solution and its generalization performance (Nar and Sastry 2018
Jastrzebski et al. 2020 Lewkowycz et al. 2020; Cohen et al., 2021). Large step sizes have been associated with flat and balanced minimizers of the training objective (Wu et al. [2088] Wang et al.) 2022, Menon 2025), sparse feature representations (Nacson et al., 2022] Andrisshchenke et al. 2023), smooth solution functions (Mulayoff and Michaeli 2000
et al.
2023). and improved Yet, the theoretical
generalization (Ba et al., [022] Qiao et al. [024]] [Sadridinov et al.] 2024).
understanding of large step sizes remains limited, even in simple convex settings. Our investigation is motivated by two fundamental questions:
Given an initial parameter, what is the critical (largest) step size that allows convergence?
What kind of implicit biases are induced by gradient descent with near-critical step size?
Addressing these questions is challenging, since large step sizes can produce highly complex, non- monotonic, and even chaotic trajectories. In particular, trajectories may not converge stationary points but instead enter periodic or chaotic oscillations (Chen and Bruna 2023) Chen et al. 2002b)] Ghosh et al. 2025), converge
statistical distribution (Kong and Tao 2020); trajectories that eventually converge to a minimizer may still undergo chaotic oscillations during early training (Zhu
al. 2023 Kreisler et al., 2023] Song and Yun (023); ; and trajectories with nearby initializations
et al. (025). Moreover, empirically, the of step sizes and the set of initializations leading to convergence can form fractal structures (see, respectively, [Sohl-Dickstein, 2024| Zhu et al. 2023). In this work, we provide precise answers the above questions in the context matrix factorization problems with rigorous theoretical characterizations.
can diverge exponentially from one another (Hermman et al. 2022 Jiménez-Gonzále
We begin by examining gradient descent in simplified problem to factor scalar target as the inner product of two vectors. We show that two striking phenomena emerge at near-critical step sizes: (i) initializations in arbitrarily small sets can converge to global minimizers with arbitrarily large norm, sharpness imbalance, to a saddle, indicating lack of simple implicit biases; and (ii) the set of initializations that converge, that is, the convergence region, has fractal structure, indicating that the convergence unpredictable near the boundary (see Figure Interestingly, while

---

Published as a conference paper at ICLR 2026
One coordinate of the rinal minimizer
Balanced minimierr Imbotonced minimizer
global Convergence boundery
Random AxiS2
16
Random Axis;
Figure 1: The training outcome of gradient descent depends sensitively on initializations near the convergence boundary. Left: Gradient descent applied to L(u, v) = (u v- 1)² 2 0.3([|u|] 2| with (u, v) E IR.10 Shown is random two-dimensional slice of 10 Gray points are initializations leading to divergence; other points are colored by coordinate value of the converged minimizer. The convergence boundary has a fractal structure. Right: Gradient descent applied to L(u, v) (ue -oo--1. 1)a with (u, v) I Shown are balanced minimizer p (green star), its neighborhood Or (blue disk), and the preimage of Op under GD (blue region with dashed boundary). An imbalanced minimizer (green diamond) is shown similarly, with its neighborhood and preimages depicted orange. The convergence boundary is smooth but the converged point is sensitive initialization.
in the unregularized setting the convergence region is regular (in the almost everywhere sense), it becomes fractal once regularization introduced. Further quantifying the unpredictability of the training outcome, we show that the topological entropy of the gradient descent system is at least log 3, Also, we show that the fractal nature of the boundary of the convergence region is captured by self-similar curve whose fractal dimension is estimated as 1.249. To our knowledge, beyond the univariate training loss setting studied by Kong and Tao (2020): Chen et al. (2024b), this the first rigorous characterization of chaos in gradient descent optimization.
We then extend our analysis to general matrix factorization by showing that, when the initialization lies in subspace defined by of orthogonal conditions, the gradient descent dynamics decouples into several independent scalar factorization dynamics. Hence, all results established for scalar factorization remain valid this subspace. This includes the commonly used identity initialization, as well as the training of linear residual networks (Hardt and Ma 2017 Bartlett et al. 2018).
We further analyze the mechanisms underlying these phenomena and trace them a folding behavior of the update map GD(O) = v/ LO)): the map GD sends a region C onto a superset of C in multi-fold covering manner. Consequently, if C contains a convergence boundary that is invariant under GD, then the boundary exhibits self- self-similarity; moreover, GD is mixing on the boundary, giving rise to the sensitivity to initialization. We show that for general neural networks with polynomial activations GD indeed acts covering map the parameter space outside measure-7ero set.
Overall, our results imply that criticiitical step sizes place gradient descent in a chaotic regime where the training outcome is unpredictable: infinitesimal perturbations the initial conditions can lead to substantially different outcomes. This contrasts sharply with the stable dynamics observed at smaller step sizes. We empirically validate the presence of chaos in matrix factorization with general initializations, deep matrix factorization, and deep ReLU networks trained on real-world datasets.
MAIN CONTRIBUTIONS
The goal of this article to provide rigorous insights into the dynamics of gradient descent with large step sizes in matrix factorization. Our contributions can be summarized as follows:
We derive the exact critical step size for convergence in scalar factorization and show that the convergence region is equal almost everywhere to a bounded and smooth domain. At critical step sizes, we show that infinitesimal perturbations the initialization can lead to global minimizers with arbitrarily large norm, or to a saddle point. Moreover, for initializations randomly sampled from arbitrarily small sets, the distribution of the sharpness the converged minimizers has
2

---

Published as a conference paper at ICLR 2026
support containing all possible values below 2/n,, which evidences chaos at distributional level. Also, we show that the topological entropy of gradient descent system is at least log 3.
We show that when using le regularization, gradient descent selects either the minimal distance solution the maximal distance solution among all global minimizers. At critical step size, infinitesimal perturbations of the initialization can switch this selection from one to the other. Furthermore, adding l regularization yields fractal convergence boundary whose geometry is captured by a self-similar shape in R22 after symmetry reduction.
We extend these results to general matrix factorization with initializations in a subspace that includes the identity initialization. In particular, gradient descent exhibits chaos and fractal convergence boundary on this subspace.
We reason that chaos arises from a folding behavior of the gradient descent update map. For general neural networks with polynomial activations, we show that gradient descent is covering map on each connected component of the parameter space after removing a measure-zero set. We empirically validate these chaotic phenomena in deep ReLU networks trained on real-world data.
1.2 RELATED WORK
Gradient Descent Dynamics Under Large Step Sizes A main line of research on large-step-size
gradient descent focuses on the non-monotonic convergence of the loss and its impact on the final
model. Key perspectives include the Edge of Stability (Cohen et al., 2021
Ma et al. 2022 Agarwala
et al. 2023 Damian et al., 2023 Ahn et al. [022 [023] Zhu et al. 2023 Wang et al. 2023) and the
catapult phenomenon (ee et al. [2200] Kalra and Barkeshli, 2023 Meltzer and Liu 2023
Zhu et al.. [024a b). Compared
these works,
analysis extends to even larger (near-crticaa)
step sizes. Another line of work shows how a large step size can enhance feature learning in one step of gradient descent (Ba al.| [022]] Dandi et al., 2024 Moniri et al. 2025), also comparing different parametrizations (Sonthalia ct al. 2025). Similar observations about the role of the step size in feature learning have also been made for SGD (Andriushchenko et 2023 Lu et al. 2024) and pre-training (Sadrtdinov et al. 2024.. Ziyin et al. (2022) observed that for a certain range of step sizes, SGD can have undesirable behavior, such as convergence to local maxima. For linear networks, Kreisler et al.) (003) identified a monotonically decreasing quantity (sharpness) along the gradient descent trajectories. Waan et al. (0022) showed large step size induces an implicit bias towards balanced minimizers in matrix factorization. Crăciun and Ghoshdastidar (2024) proved the existence of step size threshold above which the algorithm diverges. A similar problem is studied
by Marion and Chizat (0244) Large-step-size gradient descent has also been investigated in logistic
regression (Wu and Su 2023 Wu et al.
2024 Meng et al. 2024), and some of the analysis has been
further extended to shallow networks (Cai al. 2024).
empirically observed chaos, specifically positive finite-time Lyapunov exponents, for several variants of steepest descent methods. The phenomenon named period-doubling bifurcation roule to chaos has been widely observed in recent literature (Kong and Tao, 2020; [Cen and Bruna 2023 Chen et al.] 2024b; Meng et al. 2024 Danovski et al. [2244] Ghosh al. (2025). Among them, only Kong and Tao (2020); Chen al. (2024b) provided rigorous analyses for the chaotic dynamics. They showed the emergence of Li- Yorke chaos, i.e., the existence of periodic orbits of arbitrary periods, for univariate training losses. In comparison, our setting is high-dimensional. Additionally, we establish not only the existence of all periodic orbits, but also the sensitivity of the limiting point to initialization, which is more relevant to practical optimization, particularly the implicit bias of the optimization algorithm.
Chaos in Optimization Van Den Doel and Ascher (2012)
2 PRELIMINARIES
We focus on the following shallow matrix factorization problem with lg regularization:
min L(Ø) 1 V (=(U,V)
À (IUTI) + 1IVI
(1)
where À 2 0, U.U.V E Rd and the target matrix Y E Rd xd, is diagonal matrix. The diagonality
of Y is weak assumption that can be achieved by reparametrization. Specifically, for arbitrary Y consider the singular value decomposition Y Py LT and the rotations U ú U.. and
3

---

Published as a conference paper at ICLR 2026
== =V. The objective then becomes the target diagonal. Moreover, the optimization dynamics in minimizing LUÜ V) are identical to those in minimizing L(U, V) up to the rotation (see details in Appendix (
+1VI))) + where
Gradient optimization in problem (I) has been extensively studied, especially in small-step-size Arora et al. 2019 Yun et al.) [02T] Du et al. 2018 Li et al. 202T, Min et a 2023: Chen et al., (024a). Its landscape enjoys a favorable structure: the global minimum attained and every stationary point is either a global minimizer or a strict saddle (Li et al. 2019b Valavi et al. 2020, Zhou et al. 2022). Nevertheless, this problem retains a key complexity differ substantially (although they yield the same and-to-end matrix). In particular, in the unregularized case, both the parameter norm and the imbalance I|U' T VV || can be arbitrarily large on the set of minimizers. This makes the problem a natural testbed for studying how hyperparameter choices affect the implicit biases of parameter optimization algorithms. We note that other forms of regularization have also
regimes (see, e.g.. [aaxe et al. 2014,AArora
typical of neural network optimization: global minimizers
| +
been studied in the literature, such as ||U - VV |
(Tu et al. 2016 Ge et al. 2017).
We consider gradient descent with constant step size 17 to solve problem [
Ut+1 = U2- (V, U,- YT) n/Vu,, V(+1 V₁ - nUU(U V) nAV.
We define the gradient descent update map (0) = 0 - L() so that (U++1, Vi+1)) = (U,, Vo). The basin of attraction of stationary point 6* of is the set of all initializations that converge to 8", {0: lim GD N (0) = 8*) 80 The convergence region for step size 17, D,, is the union of the basins of attraction of all global minimizers. The critical step size 17" (Ū, V) for an initial- ization (Ū, V) is defined as 17" (Ū, F)= sup {1): lim GD, (0) E M), i.e., the supremum of the step sizes that allow convergence to a global minimizer, where M = {0: L(0) = ming L(0') denotes the set of all global minimizers. A set is said to be invariant under GD,, if (S) C S.
We introduce notions for describing fractal geometry. A fractal is typically defined as shape that exhibits self- similariit and fine structure at arbitrarily small scales. Formally, we say a set S C R is self-similar with degree & if there exist k homeomorphisms, de: S - S, 1,.... k. that satisfy (i) S = = So (S) and (ii) there exists an open set O C S such that _ (O) C O and (o) { are pairwise disjoint. Condition (i) states that S can be covered by k smaller copies of itself. Condition (ii), which is known as the open set condition, ensures that those copies do not overlap much. This definition is closely related to an Iterated Function System (IFS), standard tool for amalyzing fractals (see, e.g., Hutchinson 1981 Falconer 2013). However, unlike IFS where the maps are required to be contractive and the set S to be compact, the shapes considered in our study may be unbounded.
Finally, we introduce notions related to chaos. Although there is no universal definition of chaos, one common characterization of chaos is the sensitivity to initialization, which is often known as the butterfly effect. In the context of optimization, this means that infinitesimal perturbations of the initial condition can lead to substantially different training outcomes (c.g.. turning convergence into divergence, or shifting convergence from one minimizer to another qualitatively different minimizer). In dynamical systems, key measure of chaos is the topological entropy. Informally, the topological entropy h (F) of a dynamical system F measures the exponential growth rate of the number of distinct trajectories of F as a function of the trajectory length. We defer the formal definition to Appendix B. A positive topological entropy is widely regarded as hallmark of chaos (see, e.g., Katok et al. 1995 Robinson 1998. Vries 2014). In this paper, we adopt the above notions for fractals and chaos. We note that different definitions and settings exist, and discuss their relation to our study in Appendix A
3 SIMPLIFIED MATRIX FACTORIZATION
In this section, we study gradient descent in the special case of problem 1 where d,, 1, i.e., factorizing scalar y as the inner product of two vectors u." u.u. This and similar scalar factorization settings have served as canonical models for understanding large-step-sizz dynamics (Lewkowyzz
ct
2020 Waan et al.) 2022; Kreisler et al. 2023 Ahn et al.) 2023 Zhu et al. 2023 Kalra
et al. 2025). Compared to these, our analysis extends to the full spectrum of step sizes, rather than
restricting to bounded step sizes. Proofs for results in this section are in Appendix
In dynamical systems the basin of attraction is often defined for attractors. Here extend the terminology include all stationary points, such as saddles, for simplicity of presentation.
4

---

Published as a conference paper at ICLR 2026
3.1
CHAOS AT LARGE STEP SIZE
Consider the unregularized scalar factorization problem:
min (RR)
L(0)
2 (u yy)
(2)
where u, v E I2 2 1 and y IR Problem 2 retains several key complexities of the general prob- lem (I), including non-convexity, high- sigh-dimensionality non-Lipschitz gradients, and an unbounded set of global minimizers M (0: 0= = y).
In the following result, we characterize the critical step size for problem 22 and the emergence of chaos under critical step sizes. We use B(0, E) to denote the ball of radius & centered
Theorem Consider gradient descent with step size 1j for solving problem 22. The following holds:
Critical Step Size: For almost all initializations (ū, B) E R2dd, the algorithm converges to global
minimizer if n < n" (ū, 8) and fails to converge to any minimizer if n > 17" (ū, 0), where the critical step size is given by (when y = 0, we adopt the convention 1/0 +o0):
8
17" (ū, min-
| 1012 + v (1all + |1 16y(ū
Therefore, when v satisfies vlyl < 1, the convergence region Dr, is equal almost everywhere to D2 (u, v) E RRdd: 1|u12 + 11l1 + ((llul + |=]l])2 16y(u² Tu y) 8
Sensitivity to Initialization: Fix a step size y that satisfies nyyl < Let Tmin
min {[[0]] : O E M) be the minimal norm over all global minimizers. Given arbitrary 0 E ÖD' E, K > O and E Pimins 00), there exist " E B(0,e) c) such that, as N tends to infinity, GD (0') converges to global minimizer with norm 7, GD (0") converges to global minimizer with | - we ||r > K, and GD (0"") converges to (0, 0), which is a saddle when y # O.
Trajectory Complexity: Assume nlyl < 1. The topological entropy of the gradient descent system GD, satisfies hi ,,D,,,, 2 log3. Moreover, GD,, has periodic orbits of any positive integer period.
Theorem provides a necessary and sufficient convergence condition for gradient descent in prob- lem The critical step size (3) consists of two terms. The first term arises because: when 1/yl > 1, all global minimizers become unstable and only attract measure-zero set. The second term character- izes the full convergence region, which is equal almost everywhere to an ellipsoid D), (see right panel of Figure T.. Note that this convergence result is global: it holds for all step sizes 17 and initializations 8.. except for a null set in the (n, 0o) -space. In contrast, the previous work of Wang et al. (2022) only showed convergence for step sizes in the range < 1/(3]y]), and initializations in a strict subset of 7). A comparison between the proof techniques used for Theorem and those used by Wang et al. 2022) is provided in Appendix E.2
D' (see Figure
By Theorem ! gradient descent in problem 22) exhibits strong form sensitivity to initial condition. Note that in problem (2). the squared norm of the parameter coincides with the loss sharpness Amax (V²L) at global minimizers (see Appendix E.1.2 Hence, Theorem I shows that at critical step size, infinitesimal perturbations of the initialization can send the trajectory to a minimizer with arbitrarily large norm, sharpness or imbalance, or a saddle. This a hallmark of unpredictability: is impossible to reduce the error in the prediction of the converging point by improving the precision in the specification of the initialization. We remark that this form reachability from an arbitrarily small range of initial values is familiar in chaos theory, such as the Julia sets in complex systems and the Wada basin boundaries (see, c.g., Devancy and Eckmann| [9877] Nusse and Yorke 1996] Aguirre et al. 2001). However, these classical frameworks depend on properties not satisfied by setting, such& as complex differentiability or invertibility of the system map, and thus do not apply here.
Theorem I also quantitatively measures chaos in gradient descent in problem 2): the topological entropy is positive and is at least log 3. Roughly speaking. the number of distinct gradient descent trajectories of length N grows at rate of 3 (further interpretation for topological entropy is provided
5

---

Published as a conference paper at ICLR 2026
5 2/0
Derelit
4 [V"
0.22
206
1 1v
Figure 2: Gradient descent applied L(u,e) (wv 1)² with (u, R?. Left: Blue lines and purple lines represent the basins attraction of unstable minimizers and of the saddle (0, 0), respectively. Right: Initializations square). However, the distributions of the squared norm and imbalance of the converged minimizer (u", u*), and of the number of iterations to reach loss below 10 have wide supports.
evenly sampled from small set intersecting aD, (the blue
in Appendix B.1). Beyond entropy, another aspect trajectory complexity is shown: gradient descent admits periodic orbits of any positive integer period. This is closely related to Li- Yorke chaos and was shown for gradient descent in univariate loss functions (Kong and Tao 2020
Chen et al., 2024b).
While Theorem D identifies the convergence region up measure-zeno set, now provide complete description of the region. In Appendix [..11] we show that every minimizer with squared norm larger than 2/1 Lyapunov-unstable and that the basins attraction of unstable minimizers and of the saddle have measure zero. By Theorem | these measure-zero basins intersect 0D', " in arbitrarily small neighborhoods and exhibit fractal structure (see left panel of Figure 2 The full convergence region D,, is then the union of the smooth domain Di and the basins of unstable minimizers, excluding the basin of the saddle (0,0) when # O.
The next result shows that gradient descent in problem (2) exhibits chaos even after removing the measure-zero basins of attraction of unstable minimizers, i.e., it is chaotic at the distributional level.
Theorem 2. Under the same conditions and notations as Theorem | and assuming nlul < 1, given arbitrary 0 @D > 0 and open sub-interval E C [Timin: 2/TI/. there exists a set of positive measure c E(0,)) such that for any 6' E O. GD (0') converges to a solution with norm in E.
Consider initializations randomly sampled from distribution whose support contains arbitrarily small neighborhood of the convergence boundary. Theorem then implies that the sharpness (or norm) of the converged minimizer must have support containing the entire interval (Ymin 2/1)) (see right panel of Figure 2 In other words, at near-critical step size and even after discarding the basins of unstable minimizers, the best prediction for the final sharpness is the entire interval (Tmins 2/n).
3.2 REGULARIZATION INDUCES FRACTAL CONVERGENCE BOUNDARY
Consider the scalar factorization problem with & regularization:
min R=(
L( u v v) (1ul1 111111)
(4)
where u, Rdd d 2 1, 0 and y E R. The added regularization makes the set of global minimizers a bounded set. In particular, for problem 4 MM- = [u sgn())u, || lys - ー} when << ly] and M 0), when À 2 ly]- Regularization is commonly used to mitigate unbounded minimizers and to establish convergence results (Cabral et al. [2013] Ge al. 2017]] Li et al. (001a)). However, and rather remarkably, we will show that for the regularized problem the global dynamics of gradient descent becomes the limiting point of convergent trajectories unpredictable but also the convergence itself.
more unpredictable than for the unregularized problem: not only is
The predictability of convergence depends the geometry of the boundary of the convergence region. Two difficulties arise in analyzing this geometry: (i) the presence of the basin of the saddle; and (ii) the high-dimensionality of the convergence boundary. Specifically, observe that the basin

---

of attraction of the saddle point intricately penetrates D, and creates topological boundaries "within" Dry (see Figure 8 However, such boundaries are not of interest, since they do not separate points inside D, from points outside, i.e., both sides lie in D,,- This motivates to instead consider the boundary of D D U.S., where S, is the basin of the saddle. Note, the smooth domain D' in the Published as a conference paper at ICLR 2026
unregularized problem plays analogous role in clarifying the geometry of convergence region.
The second difficulty is the high dimensionality of the boundary aD" C CR2d. To address this, we identify and reduce the symmetry in "D" We introduce the map T: IR.2d - RR2 T(u, v) = (u vv v. ||ul1 + || (2 The fiber of T, i.e., the preimage of a point in T (R diffeomorphic to gd-1 x gd and hence has regular shape (see Appendix D. In the following, we show that gradient descent dynamics are captured by their evolution across these fibers.
2d), ). is generically manifold
Proposition 3. Let (u,, 122020 denote the gradient descent trajectory in problem 4 with X 2 O. Let (ze, wi) T(e, va). There exists a planar map F: IR² E that only depends on T7, X, y such that (#++1, Wi+11 (141,WW(+1) (ze, wi) converges to T(M). and it converges to (0,0) if and only if (2t, we) converges to (0,0).
= F(z,, we) holds for all t 2 0. In particular, (u,, v1) converges to .M if and only if
The map F describes how the gradient descent trajectory evolves across the fibers T. Its formulation is given in Appendix D By Proposition all points lying in the same fiber share the same convergence behavior. Therefore, roughly, the boundary aD", can be constructed by attaching fibers of T to the projected boundary ;( ): an example will be given below. Note, as the fibers are generically smooth manifolds, any geometric complexity of OD", will be captured by T(OD")-
In the following result, we show that the convergence boundary of problem 44) has a self-similar structure and gradient descent exhibits sensitivity to initialization near the convergence boundary.
Theorem 4. Consider gradient descent with step size ni for problem (4) with 0 < X S m1/(
lyl. 1. Consider the map T(u, v) = (u TT,, v. 1 + ||=]12). Let S, be the basin of attraction of (0,0), which is saddle if X < lul, and let D Dn U S.p. The following holds:
Self-similarity: So has measure zero and T(DD)) is self-iimilar with degree three.
Unboundedness: When y 0, there exist constants a, b >> O such that almost all initializations (ū, 8) with |ū 8| < cxxp(=b(([a]] + [[ converge to a global minimizer.
Sensitivity to Initialization: For any 0 E Day the algorithm converges either to the closest global minimizer p (0) = arg M |lp A|? or the farthest p+) = arg maxpeM ||p 0||
Moreover, there exist infinitely many points on aD", such that for any open set O containing such
(f") converges to point, there exist f 8 E O such that GD, (f') converges to p (6") and GD n p² (0"), as N tends to infinity.
Theorem 4 indicates that the convergence boundary for problem 4 0D" can be constructed by attaching libers of 7' to self-similar set T(āD)) with degree three. T(D) is displayed in the left panel of Figure 3 and its box-counting dimension is estimated to be 1.249, as shown in the right panel. The case of d = 1, i.e., u, u E IR, is shown in the middle panel of Figure There, the fibers of T generically consist of four discrete points. 8D" is then constructed by attaching four points to T(OD"). Hence, aD", is simply the union of four copies of T(DD"). The fractal boundary marks unpredictability in convergence: given an initial point near the boundary, it almost impossible to determine whether the point is inside or outside the convergence region. This unpredictability is also quantified by the box-counting dimension, as explained in Appendix B..2]
Theorem 4 shows that when y O, the convergence region has unbounded interior up to null set. This sharply contrasts with the convergence region in the unregularized case, which coincides almost everywhere with a bounded domain. By Theorem gradient descent converges provided that the
TT decays exponentially fast as function of the squared norm ||ul[ ||||| Geometrically, this creates an outward spike in the convergence region. Then, the self-similarity replicates this spike infinitely many times and at multiple scales, giving rise to the spiky convergence boundary observed in Figure 3 Although Theorem 4 shows the unboundedness only for the case = O, we observe qualitatively the same geometry for general targets (for an example, see left panel in Figure Mp.
When 0 E Dy, both minpem llp 012 and maxpeM 1p 0112 admit unique solutions. Morcover, p² (0) # p (0) when À lv]-
7

---

Published as a conference paper at ICLR 2026
T
Box-counting dimension 1.249
Minimizer, Minimizers
1.240x 1 0.477
2.99.9968
0/
log(1/e)
Figure 3: Gradient descent is applied to T. v) = (ur / 2 where (u, v) R.2. Left: The projected convergence boundary T@ØD)) is self-similar with degree three: it is covered by three smaller copies of itself (green, red, blue). Middle: The convergence boundary 0D" consists of four replicates of T (, separated by gray lines. The only two minimizers are shown as red triangle and a blue circle. Points are colored red if they converge to the red triangle, and blue if they converge to the blue circle. Right: The box-counting dimension of ((D)") is estimated as 1.249.
Fractal basin boundaries have been extensively studied in dynamical systems (see, e.g., Grebogi et al. 1983b McDonald et al., I985). However, these classical approaches are either largely case-specific or rely properties that do not hold in our settings, for instance, invertibility of the system map. A more detailed discussion is in Appendix A To our knowledge, our result provides the first rigorous characterization of a fractal convergence region in the context of machine learning optimization.
Theorem 4 also shows that, although regularization eliminates unbounded global minimizers, the selected minimizer remains unpredictable. Specifically, while the algorithm always selects either the minimal distance solution or the maximal distance solution, this selection becomes unpredictable the convergence boundary, both choices occur in arbitrarily small sets (see middle panel of Figure 3). This stands in contrast with gradient descent under small step sizes, which typically exhibits a iistance-minimizzaioon bias (see, e.g.. Gunasekar al. 2018 Boursier al. 2022). Indeed, in the following, we show that with sufficiently small step size, gradient descent in problem always selects the minimal distance solution.
Theorem 5. Under the same conditions and notations as Theorem and letting Quu, v) ||u|
[1u122 + / (11u122 + 11u112)2 16y(u -),, the following holds for almost all initializations (ū, 8): If 8/(4) + (ūū then gradient descent converges a global minimizer; 1f 4/(4A Q(ū, 8)), then the particular minimizer it converges to is p (ū, 0).
Finally, we present another implication of the chaos in gradient descent. In Appendix we show for the case d = 1, i.e., L(u, v) = {uv - y2 + 2 (u² + u²), u, u E R, X 2 0, that, any continuous dynamical invariant must be constant. In particular, the imbalance u v², which is known to be (approximately) preserved under gradient descent with small step sizes (Du et al., 2018 Arora et al. 2019). fails dramatically under large step sizes. Although this result does not directly extend to the case 2, we anticipate that the chaos strongly constrains the form of dynamical invariants.
4 MATRIX FACTORIZATION AND BEYOND
In this section, first extend results in Section 3 to general matrix factorization with orthogonal initializations. We then analyze the underlying mechanism that gives rise to the chaotic phenomena, and discuss to what extent this mechanism may extend to more general settings. Finally, present experiments showing chaos in a real-world machine learning setting.
4.1 MATRIX FACTORIZATION WITH ORTHOGGNNLL INITIALIZATIONS
Consider gradient descent in matrix factorization 1 with initializations in the following subspace:
W = {U, V) IR 2dd-dy (u', u²) (u', v²) = (v', w²) = 0, Vi + i
(5)
where u', oe denote the ith column of U, V. The subspace W includes several commonly studied initialization schemes, such as the scaled identity initializations Ú aTa, V BTa (Chou et al.
8

---

Published as a conference paper at ICLR 2026
2024
Ghosh et al. 2025), zero-asymmeerri initialization (Wu al. 2019), and those used in training linear residual networks (Hardt and Ma 2017] Bartlett et al. 2018). J. Note that in the low-rank setting d < dy. constraints in require at least dy d pairs of (ur, e') to be initialized to zero.
key observation is that results of Section precisely characterize the gradient descent dynamics on W: with initialization in W, the trajectory remains in W, and the dynamics decouple column- - each pair (u", v") evolves independently according to the scalar factorization dynamics with target ys. the ith diagonal entry of Y. Thus, all results in Section 3 extend verbatim, and gradient descent exhibits chaos and a fractal convergence boundary on W (see details in Appendix G).
A full characterization of the dynamics outside W requires additional investigations, for instance, whether W attracts or repels nearby trajectories. However, we point out that, the presence of chaos on W already suggests that the global dynamics can be unpredictable. For instance, due to the continuity of GD,, we expect that initializations in the vicinity of the convergence boundary in W. will inherit the sensitivity to initial conditions, least during the initial phase before potentially escaping W. This phenomenon is known as transient chaos in dynamical systems (see, e.g., [Tel 1990). Experimentally, we observe that the chaotic phenomena indeed persist for general initializations, and also in deep matrix factorization (Appendix []. A rigorous characterization of gradient descent dynamics in these settings is an intriguing direction that we leave for future work.
4.2 GENERAL MECHANISM BEHIND CHAOS
We now explain the mechanism giving rise to chaos in scalar factorization. For simplicity, consider the case d = 1: L(u, v) (uee y)² /2 + A(u² + /2 with (u, u) R In this setting, there exists a set C C R22 such that GD., behaves as a 3-covering map from C onto GD,,(C) C C. Roughly speaking, GD., stretches and folds C cover GC,,(C) three times. Furthermore, C contains the convergence boundary 0D,, y The boundary satisfies (8D,) aD,,, meaning that OD, can be stretched and folded three times cover itself. thereby exhibiting self-similarity. Additionally, due to its folding behavior, GD, , is transitive on ,D, i.c., points on the boundary OD, are mixed under the iterations of GD, (via multiple stretches and folds). This mixing leads to the sensitivity to initializations near the boundary. A visualization of these properties are provided in Figure 6
The key ingredient above is the existence of a set C in the parameter space on which GD,, acts as a covering map. In fact, the described chaotic phenomena arise given such set C and any invariant subset A C C that satisfies (A) = A, a condition typically satisfied by the boundary of a basin of attraction. Then ask: When does such set C exist? The following result partially addresses this for general neural networks with polynomial activation functions.
Proposition 6. Let fo(-) be a polynomial neural network of arbitrary depth and width, parametrized by 0 E I.. Consider loss function L(0) ΣØ= e(fa (2z), (,)) with polynomial loss E and training data (I Mi) }= For all 1) > O except for most finitely many values, there exists a measure-zero set K. C Rp such that GD,,
b is a covering map on any connected component of RP \Kap
The proof is provided in Appendix G We remark that, while Proposition 6 suggests general folding behavior of GD2, it does not address whether GD,,(C) (C) contains C and whether C contains a basin boundary. We leave these intriguing questions for future research.
4.3 CHAOS AND FRACTALS IN NEURAL NETWORKS
We show that the chaotic phenomena persist in real-world training settings. We trained depth-three ReLU network on a 2-class subset of CIFAR-10 (Krizhevsky et al., 2009). In Figure 4 we consider the mean squared error and report how the step size affects the norm and sharpness of the parameter that is returned at the end of training. For gradient descent without momentum, two distinct regimes of step sizes can be observed: (i) EoS regime: both the final norm and sharpness lie close to a smooth curve, indicating predictability. In particular, the final sharpness is close to 2/1,, aligning with JotheC et al. (2021). (ii) Chaotic regime: the norm and sharpness become sensitive to the step size, indicating chaos and unpredictability, In particular, the final sharpness spans almost all values below 2/17, aligning with our Theorem 2 A similar phenomenon appears for gradient descent with Polyak momentum (Polyak] 1964). However, in the small-step-sic regime, the final sharpness forms a cluster rather than aligning with (2 + 2.8) as predicted by Cohen et al. (2021). Experiments with cross-entropy loss provided in Appendix where a similar two-regime phenomenon is observed.
9

---

Published as a conference paper at ICLR 2026
Vanilla GD
Varilla GD
momestum 0
momentum
Grobal miiimum
Gisbal minimum
Eo5 TagueT
Chaplic rig me
Falled attain oprimality
Falled attain optimalty
Fitell Harr
Fmi Sereen
regime
regime
S1o0 ole
048
Stop
Stop
Stop siz0
Figure 4: GD without and with momentum in training depth-3 ReLU network on subset of CIFAR-10 for 5000 iterations under mean squared error. The initialization is randomly sampled and then kept fixed across all panels. At large step sizes, the final norm and final sharpness sensitive to step size, Dashed black lines show the final sharpness predicted by Cohen et al. (202)
In Figure 5 we show how the parameter initialization of gradient descent affects the final loss and sharpness when using weight decay. For both quantities, we observe fractal structures in the parameter space, indicating that the training outcome is highly sensitive to the initialization. Further experiments in Appendix show qualitatively similar fractal patterns also appear without weight decay. Experiment details of Figures and 5 are provided in Appendix J
Final loss
7comeriin
Final sharpreess
7oomed-n
Random Asile
Random Aulsa
-2
Random
Random Axiss
Figure 5: GD with weight decay in training a depth-3 ReLU network on a subset of CIFAR-10 for 3000 iterations. Shown is a random two-dimensional slice of the parameter space. Each initial parameter is colored by the value the final loss and final sharpness, respectively. Left two panels: fractal basins of global minimizers (dark blue), sub-optimal solutions (white), and region leading to divergence (dark red). Right two panels: the final sharpness is sensitive to the initialization, spanning wide range of values below 2/n. Gray points initializations from which the algorithm diverged.
CONCLUSION
We offered a rigorous characterization of gradient descent with large step sizes in matrix factorization. Our results reveal two striking phenomena: near the convergence boundary, the selection of the minimizer is umpredictable, and adding regularization can induce a fractal convergence boundary that makes the convergence itself unpredictable. As driver of this complexity, we suggested a covering map structure exhibited by the gradient descent update map the parameter space.
Limitations Although our characterizations substantially expand the state of knowledge in non- convex overparametrized optimization in the particular setting of matrix factorization, further research needed to rigorously characterize the dynamics of large-step-size gradient descent in other settings, such general initializations, deep matrix factorization, or neural networks with nonlinear activation functions. We believe the contributed insights can aid in the development of such programs.
Future Directions We showed that at large step sizes there may not exist any simple algorithmic biases, but observed that biases could still be studied in a distribution sense. Further analyzing the properties of the distribution over global minimizers that is induced by a distribution of initializations an interesting direction for future work. particular, are there cases in which the distribution is uniform over subset of minimizers, or cases in which it will concentrate in predictable way?
10

---

Published as a conference paper at ICLR 2026
Reproducibility Statement Code to reproduce experiments is made available at https
com/ http::ahic...ataaaaaaaaaoaaoaann
ACKNOWLEDGMENT
This project has been supported by NSF CCF-2212520 and NSF DMS-2145630. GM was partially supported by DARPA AIQ project 12R00112520011 DFG SPP 2298 project 464109215, and BMFTR in DAAD project 57616814 (SECAI). We thank Ryan Tibshirani for insightful discussions.
REFERENCES
Agarwala, A., Pedregosa, F., and Pennington, (2023). Second-order regression models exhibit progressive sharpening the edge of stability. In Proceedings of the 40th International Conference on Machine Learning, pages 169-195.
Aguirre, J., Vallejo. C., and Sanjuán, M. A. (2001), Wada basins and chaotic invariant sets in the hénon-heiles system. Physical Review E, 64(6):066208.
Aguirre, J., Viana, R. L.,, and Sanjuán, M. A. (2009). Fractal structures in nonlinear dynamics. Reviews of Modern Physics, ((1):333-386.
Ahn, K., Bubeck, S., Chewi, S., Lee, T.,, Suarez, F., and Zhang, Y. (2023). Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems, 36:19540- 19569.
Ahn, K., Zhang. J., and Sra, S. (2022). Understanding the unstable convergence of gradient descent. In International conference on machine learning, pages 247-257. PMLR.
Andriushchenko, M., Varre, A. V., Pillaud-Vivien, L., and Flammarion, N. (2023). SGD with large step sizes learns sparse features. In International Conference Machine Learning, pages 903-99 925- PMLR.
Arora, S., Cohen, N., Golowich, N., and Hu, W. (2019). A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations.
Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, (2022). High-dimensional asymptotics of feature learning: How one gradient step improves the representation. In Advances in Neural Information Processing Systems.
Bartlett, P., Helmbold, D., and Long, P. (2018). Gradient descent with identity initialization effi- ciently learns positive definite linear transformations by deep residual networks. In International conference on machine learning, pages 521 5-3000. PMIR.
Boursier, E., Pillaud- Vivien, L., and Flammarion, N. (2022). Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs. Advances Neural Information Processing Systems, volume 35, pages 20105-200 20122111 18. Curran Associates, Inc.
Bowen, R., Ruelle, D., and Chazottes, J. (2008). Equilibrium States and the Ergodic Theory of Anoxov Diffeomorphismss Lecture Notes in Mathematics. Springer Berlin Heidelberg.
Cabral, R., De Torre, F., Costeira, J. P., and Bernardino, A. (2013). Unifying nuclear norm and bilinear factorization approaches for low-rank matrix decomposition. In Proceedings of the IEEE international conference on computer vision, pages 2488-2495.
Cai. Y., Wu, J., Mei, S., Lindsey, M., and Bartlett, P. (2024). Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization, Advances in Neural Information Processing Systems, 37:71306-71351
Chen, H., Chen, X., Elmasri, M.. and Sun, Q. (2024a). Gradient descent in matrix factorization: Understanding large initialization. In Uncertainty Artificial Intelligence, pages 619-647. PMLR.
Chen, L. and Bruna. J. (2023). Beyond the edge of stability via two-step gradient updates. In International Conference Machine Learning, pages 4330-4391. PMLR.

---

Published as a conference paper at ICLR 2026
Chen, P., Jiang, R., and Wang, P. (2025). A complete loss landscape analysis of regularized deep matrix factorization. arXiv preprint arXiv:2506.23344,
Chen, X., Balasubramanian, K., Ghosal, P., and Agrawalla, B. K. (2024b). From stability to chaos: Analyzing gradient descent dynamics in quadratic regression. Transactions on Machine Learning Research.
Chou, H.-H., Gieshoff, C., Maly, J., and Rauhut, H. (2024). Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. Applied and Computational Harmonic Analysis, 68:101595.
Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. (2021). Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations.
Crăciun, A. and Ghoshdastidar, D. (2024). On the convergence of gradient descent for large learning rates. arXiv preprint arXiv:2402. 13108.
Damian, A., Nichani, E., and Lee, D. (2023). Self-stabilization:: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations.
Dandi, Y., Krzakala, F., Loureiro, B., Pesce, L.., and Stephan, L. (2024). How two-layer neural networks learn, one (giant) step a time. Journal of Machine Learning Research, 25(349):1-65.
Danovski, K., Soriano, M. C., and Lacasa, L. (2024). Dynamical stability and chaos in artificial neural network trajectories along training. Frontiers in Complex Systems, 2:1367957.
De Melo, W. and Van Strien, S. (2012). One-dimensional dynamics, volume 25, Springer Science & Business Media.
Devaney, R. L. and Eckmann, J.-P. (1987). An introduction to chaotic dynamical systems.
Du, S. S., Hu, W., and Lee, J. D. (2018). Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31.
Elaydi, S. N. (2007). Discrete chaos: with applications in science and engineering. Chapman and Hall/CRC.
Falconer, K. (2013). Fractal geometry: mathematical foundations founddattoiallunoooonn and applications. John Wiley & Sons.
Ge, R., Jin, C., and Zheng, Y. (2017). No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In International conference on machine learning, pages 1233-1242. PMLR.
Ghosh, A.. Kwon, S. M., Wang. R., Ravishankar, S., and Qu, Q. (2025). Learning dynamics of deep matrix factorization beyond the edge of stability. In The Second Conference on Parsimony and Learning (Recent Spotlight Track).
Golubitsky, M. and Stewart, (2003). The symmetry perspective: from equilibrium chaos in phase space and physical space, volume 200. Springer Science & Business Media.
Grebogi, C., McDonald, S. W., Ott, E.,, and Yorke, J. A. (1983a). Final state sensitivity: an obstruction to predictability. Physics Letters A, 99(9):115-418
Grebogi, C., Ott, E.,, and Yorke, J. A. (1983b). Fractal basin boundaries, long-lived chaotic transients, and unstable pair bifurcation. Physical Review Letters, 50(1))993.
Gunasekar, S., Lee, J.. Soudry, D., and Srebro, N. (2018). Characterizing implicit bias in terms of optimization geometry. In Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 1832-1841. PMLR.
Hardt, M. and Ma, T. (2017). Identity matters in deep learning. International Conference on Learning Representations.
12

---

Published as a conference paper at ICLR 2026
Hermaann, L., Granz, M., and Landgraf, T. (2022). Chaotic dynamics are intrinsic to neural network training with sgd. Advances in Neural Information Processing Systems, 35:5219-5229.
Hunt, B. R., Ott, E., and Rosa Jr, E. (1999). Sporadically fractal basin boundaries of chaotic systems. Physical review letters, 82(18):3597.
Hutchinson, E. (1981). Fractals and self similarity. Indiana University Mathematics Journal, 30(5):713-747.
Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho, K., and Geras, K. (2020). The break-even point on optimization trajectories deep neural networks. International Conference on Learning Representations.
Jelonek, Z. (2002). Geometry of real polynomial mappings. Mathermaticche Zeitschrift, 239 333.
Liménez-Gonzálee P., Soriano, M. C., and Lacasa, L. (2025). Leveraging chaos in the training of artificial neural networks. arXiv preprint arXiv:2506.08552 085233.
Kalra, D. S. and Barkeshli, M. (2023). Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width. Advances in Neural Information Processing Systems, 36:5 1661112215566..
Kalra, D. S., He. T., and Barkeshli, M. (2025). Universal sharpness dynamics in neural network training: Fixed point analysis, edge of stability, and route to chaos. In The Thirteenth International Conference on Learning Representations.
Katok, A., Katok, A., and Hasselblatt, B. (1995). Introduction to the modern theory of dynamical systems. Cambridge university press.
Kong. L. and Tao, M. (2020). Stochasticity of deterministic gradient descent: Large learning rate for multiscale objective function. Advances in neural information processing systems, 33:2625-2638.
Kreisler, I., Nacson, M. S., Soudry, D., and Carmon, Y. (2023). Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In International Conference on Machine Learning, pages 17684-17744, PMLR.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny i2009..
Lee. J. (2000). Introduction to topological manifolds. Springer.
Lee, J. (2012). Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer New York.
Lewkowycz, A., Bahri, Y., Dyer, E., Sohl-Dickstein, J., and Gur-Ari, G. (2020). The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022188
Li, Q., Zhu, Z., and Tang, G. (2019a). The non-convex geometry of low-rank matrix optimization. Information and Inference: Journal of the IMA, 8(1):51-96.
Li, T.-Y. and Yorke, J. A. (1975). Period three implies chaos. The American Mathematical Monthly, 82(10):985-992.
Li, X., Lu, J., Arora, R., Haupt, J., Liu, H., Wang, Z., and Zhao, T. (2019b). Symmetry, saddle points, and global optimization landscape of nonconvex matrix factorization. IEEE Transactions on Information Theory, (5(6):3499-3144
Li, Z., Luo, Y., and Lyu, K. (2021). Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning, In International Conference on Learning Representations.
Lu, M., Wu, B., Yang, X., and Zou, D. (2024). Benign oscillation of stochastic gradient descent with large learning rate. In The Tweifth International Conference on Learning Representations.
Ma, C., Wu, L., and Ying, L. (2022). The multiscale structure of neural network loss functions: The effect optimization and origin. arXiv preprint arXiv:2204. /326, 14.
13

---

Published as a conference paper at ICLR 2026
Marion, P. and Chizat, L. (2024). Deep linear networks for regression are implicitly regularized towards flat minima. In The ehirty- Annual Conference on Neural Information Processing Systems.
McDonald, S. W., Grebogi, C., Ott, E., and Yorke, A. (1985), Fractal basin boundaries. Physica D: Nonlinear Phenomena, 12(2):125-153.
Meltzer, D. and Liu, (2023). Catapult dynamics and phase transitions in quadratic nets. arXiv preprint arXiv:2301.07737
Meng. S. Y., Orvieto, A., Cao, D. Y., and De Sa, C. (2024). Gradient descent on logistic regression with non-separable data and large step sizes. arXiv preprint arXiv:2406.0500
Menon, G. (2025). The geometry of the deep linear network. In XIV Symposium on Probability and Stochastic Processes, pages 1-47, Cham. Springer Nature Switzerland.
Min, H., Vidal, R., and Mallada, E. (2023). On the convergence of gradient flow multi-layer linear models. In International Conference on Machine Learning, pages 24850-24887. PMLR.
Moniri, B., Lee, D., Hassani, H., and Dobriban, E. (2025). A theory of non-linear feature learning with one gradient step in two-layer neural networks.
Mulayoff, R. and Michaeli, T. (2020). Unique properties of flat minima in deep networks. In International conference on machine learning, pages 7108-7118. PMLR.
Nacson, M. S., Mulayoff, R., Ongie, G., Michaeli, T., and Soudry, D. (2023). The implicit bias of minima stability in multivariate shallow ReLU networks. The Eleventh International Conference Learming Representations.
Nacson, M. S., Ravichandran, K., Srebro, N., and Soudry, (2022). Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pages 16270-16295. PMLR.
Nar, K. and Sastry, S. (2018). Step size matters in deep learning, Advances in Neural Information Processing Systems, 31.
Nusse, H. E. and Yorke, J. A. (1996). Wada basin boundaries and basin cells. Physica D: Nonlinear Phenomena, 9023):242-211.
Polyak, B. (1964). Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17.
Ponomarev, S. (1987). Submersions and preimages of sets of measure zero. Siberian Mathematical Journal, 28(1):153-16:
Qiao, D., Zhang, K., Singh, E., Soudry, D., and Wang, Y.-X. (2024). Stable minima cannot overfit in univariate ReLU networks: Generalization by large step sizes. Advances in Neural Information Processing Systems, 37:94163-94208.
Robinson, C. (1998). Dynamical systems: stability, symbolic dynamics, and chaos. CRC press.
Rosa Jr, and Ott, E. (1999). Mixed basin boundary structures of chaotic systems. Physical Review F, 59(3)
Sadrtdinov, I., Kodryan, M., Pokonechny, E.,, Lobacheva, E., and Vetrov, D. P. (2024). Where do large learning rates lead us? Advances in Neural Information Processing Systems, 37:58445-58479,
Saxe, A. M., McClelland, L.,, and Ganguli, S. (2014). Exact solutions the nonlinear dynam- ics of learning in deep linear neural networks. In 2nd International Conference on Learning Representations, Conference Track Proceedings.
Smale, S. (1967). Differentiable dynamical systems. Bulletin of the American Mathematical Society, 7376)7747-
14

---

Published as a conference paper at ICLR 2026
Sohl-Dickstein, J. (2024). The boundary of neural network trainability is fractal. arXiv preprint arXivv2402.06184.
Song, M. and Yun, C. (2023). Trajectory alignment: Understanding the edge of stability phenomenon via bifurcation theory. In Thirty-seventh Conference on Neural Information Processing Systems.
Sonthalia, R., Murray, M., and Montúfar, (2025). Low rank gradients and where find them, In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Tél. T. (1990). Transient chaos. Directions in chaos. 3:149-211.
Tu, S., Boczar, R., Simchowitz, M., Soltanolkotabi, M., and Recht, B. (2016). Low-rank solutions of linear matrix equations via procrustes flow. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 964-973 PMLR.
Valavi, H., Liu, S., and Ramadge, P. (2020). Revisiting the landscape of matrix factorization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1629-1638. PMLR.
Van Den Doel, K. and Ascher, U. (2012). The chaotic nature of faster gradient descent methods. Journal of Scientific Computing, 51 51)5:560--58.
Vries, J. (2014). Topological dynamical systems: an introduction to the dynamics of continuous mappings, volume 59, Walter De Gruyter.
Wang, Y., Chen, M., Zhao, T., and Tao, M. (2022). Large learning rate tames homogeneity: Conver- gence and balancing effect. In International Conference Learning Representations.
Wang, Y., Xu, Z., Zhao, T., and Tao, M. (2023). Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult. In NeurlPS 2023 Workshop on Mathematics of Modern Machine Learning.
Wu, J., Bartlett, L.., Telgarsky, M., and Yu, B. (2024). Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency. In Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pages 5019-5073. PMLR.
Wu, I.., Ma, C., al. (2018). How SGD selects the global minima in pwer-parameteriized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31.
Wu, L. and Su, J. (2023). The implicit regularization of dynamical stability in stochastic gradient descent. International Conference on Machine Learning, pages 37656-37684. PMLR.
Wu, L., Wang, Q., and Ma, C. (2019). Global convergence of gradient descent for deep linear residual networks. Advances in Neural Information Processing Systems, 32.
Xu, Z., Min, H., Tarmoun, S., Mallada, E.,, and Vidal, R. (2023). Linear convergence of gradient de- scent for finite width over parametrized linear networks with general initialization. In International Conference on Artificial Intelligence and Statisties, pages 2262 22-4. PMLR.
Ye, T. and Du, S. S. (2021). Global convergence of gradient descent for asymmetric low-rank matrix factorization. Advances in Neural Information Processing Systems,
Yun, C., Krishnan, S., and Mobahi, H. (2021). A unifying view implicit bias in training linear neural networks. In International Conference on Learning Representations.
Zhou, J., Li, X., Ding, T., You, C., Qu, Q., and Zhu, Z. (2022). On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In International Conference on Machine Learning, pages 27179-27202. PMLR.
Zhu, L., Liu, C., Radhakrishnan, A., and Belkin, M. (2024a). Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning. In International Conference on Machine Learning, pages 62476-62509. PMLR.
15

---

Published as a conference paper at ICLR 2026
Zhu, L., Liu, C., Radhakrishnan, A., and Belkin, M. (2024b). Quadratic models for understanding catapult dynamics of neural networks. In The Twelfth International Conference on Learning Representations.
Zhu, X., Wang, Z., Wang. X., Zhou, M., and Ge, (2023). Understanding edge-of-stability training dynamics with a minimalist example. In International Conference on Learning Representations.
Ziyin, L., Li, B., Simon, J. B., and Ueda, M. (2022). SGD can converge to local maxima. In International Conference on Learning Representations.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendix is organized into the following sections.
Appendix
Relation to classical theory for chaos and fractals
Appendix
Measure of chaos in dynamical systems
Appendix
Diagonality of the target matrix
Appendix
Quotient dynamics of gradient descent
Appendix
Proofs for Section
S
Appendix
Non-existence of continuous dynamical invariant
Appendix
General matrix factorization
Appendix
Experiment details
Appendix
Additional experiments on matrix factorization
Appendix
Additional experiments on real-world data
A RELATION TO CLASSICAL THEORY FOR CHAOS AND FRACTALS
A.1 CHAOTIC DYNAMICAL SYSTEMS
A common definition of chaotic dynamical systems is as follows (Devaney and Eckmann 1987).
dynamical system F: X + X, where X is the state space, is chaotic if: (i) it is sensitive to initialization, (ii) it is topological transitive, and (iii) periodic points are dense in X. Here, sensitivity to initialization requires that there exists 8 > 0 such that for any E X and any neighborhood of x, there exists N > and y in the neighborhood such that dist PNN (x), EN (y)) 8. This is weaker than the property we presented in Theorem T and Theorem 4 when the converged points of two trajectories are different, the trajectories must differ by a positive difference at some time N, but not vice versa. In Proposition 15] we show that the boundary aD', as defined in Theorem | is invariant under gradient descent. In Proposition
18 we show that when restricted to åD the gradient descent system semi-conjugate to a one-dimensional system that precisely Devaney chaotic. However, we show in Proposition 20 that the original gradient descent system is not Devaney chaotic when 1 2 as it fails to be topological transitive.
A system F: X + X is topological transitive if for any pair of non-empty open sets U, V, there exists such that FN (U) n + 2. family of dynamical systems that exhibit transitivity is the family of Axiom-A diffeomorphisms (Smale 1967). These dynamical systems where the set of wondering points is hyperbolic and is equal to the closure of the set of periodic points. A closed set A is hyperbolic if it is forward invariant and at each point E A the tangent space of the ambient space splits as a direct sum of stable and unstable subspaces. Axiom-A diffeomorphism is always transitive each of its basic sets (Bowen al., 20088] Chapter 3). However, gradient update maps typically are not expected to satisfy this definition as they in general are not global diffeomorphisms.
known that an
Another definition of chaotic dynamical system is by Li and Yorke (1755.. They considered a dynamical system F: X + X with X R being an interval chaotic if F has a periodic orbit with period three. They showed that if such periodic orbit exists, then (i) F' has periodic orbit with any period; (ii) there exists an uncountable set J such that, for every p. E with p + 4.
lim sup | (p) F (4)) O, lim 1 (p) P (q) 0, N
N-00
and (iii) for every E S and a periodic point E J,
sup |F (p) N (4) >
N-
In Proposition 18
showed that the restricted system GD,lar; is semi-conjugate to one- dimensional system that is Li- Yorke chaotic. In general, Devaney chaos and Li- Yorke chaos do not imply each other. For a detailed comparison between different notions of chaotic dynamical systems, see the work of Elaydi (0077).
17

---

Published as a conference paper at ICLR 2026
A.2 FRACTAL BASIN BOUNDARY
The fractal convergence boundary studied in this work falls into more general notion, called fractal basin boundary. The seminal classification given by McDonald et al. (1985) divides fractal basin boundaries into three categories: quasicircles, locally connected but not quasicircles, and locally disconnected. The most regular type, quasicircle, is typical in the Julia sets of complex analytic maps. However, noted by McDonald et al. (1985), , properties of complex analytic maps do not generalize to real maps, and hence quasicircle uncommon in real systems. We observe that the convergence boundary in our study falls in the second category, locally connected but quasicircles. This type of boundary has been observed in several planar maps, i.e., dynamical systems defined on regions of IRa A well-known example is the following system:
In+1
AgTnn mod (1), Un+1 = XyUn cos (.
The basin boundary of this system is precisely the Weierstrass curve. McDonald et al. (1985) argued that typical characteristic of this type of boundaries is the local stratification structure, which also appears in the convergence region in our case (see left panel in Figure M). To our knowledge, however, all examples of locally connected boundaries appearing in the literature, including those presented by McDonald et al. 1985): Hunt et al. (999); Rosa Jr and Onl (1999), are bounded, whereas the convergence region
is shown be unbounded. Classical approaches do not apply to our study, as most of those theoretical studies are case-specific. The last category has the most complicated structure and, noted by [Aguirre al. (2009), turns out to appear more commonly in physical systems. Boundaries in this category typically exhibit Cantor set structure. Examples include the famous Hénon map and the horseshoe map. For recent review of the fractal boundaries, we refer readers to Aguirre et al. (2009).
B MEASURE OF CHAOS IN DYNAMICAL SYSTEMS
We introduce two measures of chaos in dynamical systems, In Appendix B.1 we introduce the topological entropy of dynamical system and, Appendix B.2 we discuss how the fractal dimension of the basin boundary implies unpredictability.
B. TOPOLOGICAL ENTROPY
Let F: X X dynamical system, where is the state space with a metric The idea behind topological entropy is to measure how fast the number of "distinct" trajectories increases the trajectory length increases. To measure the difference between two trajectories of length N, consider
dx y)
max USISN-
(()
Then, the number of "distinct" trajectories of length is measured by
r(N,s) max {| dx(x, e, Vx, y S, x + y)
where [S] is the number of elements in S. The topological entropy of F, denoted h(F), measures the
exponential growth rate of r(N,e) as increases. Specifically, l((F) is defined as follows:
log (NN,))
h(F)=
lim limsup N
s-+0+
We give an example to provide more intuition. Consider the following symbolic dynamical systems:
a: {0, 1 + {0, 10
(SoS$182 ...) ($182...)
Here the state space 1) 0 denotes the set of all infinite sequence of two symbols 0 and 1, whose metric defined by
5 s/
d((sos) ... (s/s ...
2
0=
The system is called the full-shift two symbols. Despite its simple definition, this system is unpredictable and chaotic (see, e.g.. Devaney and Eckmann, 1987). In particular, periodic points are dense in the state space, and, there exists trajectory that is dense in the state space, i.e., there is
18

---

Published as a conference paper at ICLR 2026
single trajectory that can come arbitrarily close to any point. In terms of predictability, consider two points $ = (8p81 ...) = (s/s² ...) that have the same first m elements, but differ starting from the (m + 1)th element. By the definition of the distance, However, for all 2 m, we have dio (s), o (s)) > 1/2. Therefore, even two initial points are arbitrarily close to each other,
have d(s, s') S Σ}= 1/2 1/2" +1
can not make any prediction on how close their trajectories will remain the long term. This unpredictability stems from the richness of "distinct" trajectories. In fact, can show that h(()) log 2 > (see, e.g., Vries 2014). Note, the topological entropy of the gradient descent in matrix factorization is at least log 3 (Theorem)
B.2 BOX-COUNTING DIMENSION AND UNPREDICTABILITY
There have been numerous investigations discussing how non-integer fractal dimension implies unpredictabilitty in dynamical systems (see, e.g.. Tél 1990 Aguirre et al. 2009)) Here we provide brief introduction to this topic.
Recall that the box-counting dimension of a set S is defined as the following limit if it exists:
DR(S)= = lim log((V(s)) E- log(1/e)
where ((e) is the number of boxes of side length & needed to cover the set S. For a dynamical system F: X X where X TR is the state space, let D3 be the box-counting dimension of basin boundary and D be the topology dimension of the state space. Consider a collection of trajectories and randomly perturb their initial points by scale s. Let f(s) denote the fraction of the trajectories that converge to a different point, i.e., whose initial point lies in different basin of attraction after the perturbation. Thus, fE) can be roughly viewed as the chance of making an error in predicting the converged point when the precision in specifying the initial point is e. In general, the following scaling relation holds (Grebogi et al. 1983a):
f(e) ~
DB D-D2,
where D D- DB known as the uncertainty exponent. When the boundary smooth, we have DB = D - 1 and thus f(z) E, i.c. the accuracy of the prediction of the converged point is proportional to the precision on the initial point. In contrast, when the boundary has a non-integer dimension, we have D - < DB < D and hence D - DB < 1. This implies that a substantial increase in the precision in specifying the initial point leads only very small increase in the accuracy of the prediction. This marks sensitivity to initialization and unpredictability. Note, the box- counting dimension the projected boundary T(8D") is estimated as 1.249, yielding an uncertainty exponent 2 1.249 0.751 (see Section 3.2)
C DIAGONALITY OF THE TARGET MATRIX
We show that in matrix factorization 1 one may assume without loss of generality that the target matrix is diagonal. This simplification a standard technique that has been widely adopted in the literature.
Let Y P. be the singular value decomposition of Y, where Pr, O(d,) and Ey E Rdy is diagonal. Consider the change of coordinates = ÜPJ and V VQJ. Recall that the U U-update in minimizing LUU, ) is given by
U211 U. U U. UYT 1/AUr
In the new coordinate, we have
Ú U141Pr
UUPY
nAU.Py
(6)
Ü. - -nV:QT - - 172ú,
U. - VVV.VV U, - -
19

---

Published as a conference paper at ICLR 2026
On the other hand, since the Frobenius norm is invariant under left- right-multiplication by orthogonal matrices, the loss function in the new coordinates is given by
1 _ L(U, V) PP Vo P - 2I + A
2 (10RT + 11
2
1 - Ey - 2
(1UŪ)) ( +(V()))
2
Note the update iteration 66) coincides with the Ü-update in minimizing Ī(Ū, V) with gradient descent. An analogous calculation shows the same holds for the V V-update. Therefore, one may directly study the gradient descent dynamics in minimizing L(U, V).
D QUOTIENT DYNAMICS OF GRADIENT DESCENT
We show that the gradient descent dynamics in the scalar factorization problem can be described by quotient system, and we further establish key properties of this system.
Consider the map
T: R
IR? T(u, v) (u y, 1/ 1 + 1112))
Note that this definition differs from the one introduced in Section 3.2 by constant shift of -y- where y R is target scalar of problem (2). This adjustment is made purely for convenience in presenting the proof. All results stated here extend trivially to the original formulation.
We will show that the gradient descent dynamics are fully captured by their evolution across the fibers of the map T. In other words, different initializations in the same fiber produce qualitatively identical trajectories. This reflects an inherent symmetry of the system. The quotient system factors out this symmetry and describes the fiber-wise dynamics. The term quotient dynamical system is borrowed from the theory of equivariant dynamical systems (see, e.g.. Golubitsky and Stewart 2003).
D.1 QUOTIENT DYNAMICAL SYSTEM
We first introduce two properties of the map (i) the preimage any measure-zero set has measure zero and (ii) the fiber of T is generically a smooth manifold.
Proposition 7. The preimage of any measure-ze set under the map T a measure-zero set.
Proof. By Ponomarev (987), it suffices show the map T is submersion almost everywhere, i.e., the Jacobian of T has rank two almost everywhere. Notice that
JT w)
u 2v
2u
Hence, rank(JT) < if and only if there exists C F 0 such that 2u and cu 2v. This gives cvv = 2cu 4u, and hence, c c= +2 or u == 0. The set {u = 0} has zero measure. When c = +2, have tu which also yields measure-7ero set. This completes the proof.
0
Proposition 8. For almost all (2, w) T(R2d) [(z,w) E RR: 2 2/2 + v1). T- w) is diffeomorphic sd-1 x gd-1
Proof. Notice that
||u + v112 w 2(= v), ||u- o12 w 2(= w). Consider the linear bijection + and = u - v. follows that
(2, (p. q): |r = w 2(2 + v), |1a112 w 2(z 8)
Thus, the fiber is diffeomorphic 1d-1 1 g whenever o + 2(2 y) * 0. Note this only fails at measure-zero set, which completes the proof.
Li
Next, we prove Proposition 3. In the terminology of dynamical systems, this result shows that the gradient descent system GD,, semi-conjugate to planar system f under the map T.
20

---

Published as a conference paper at ICLR 2026
Proposition 9 (Proposition 3 Let (u,, v1)220 denote the gradient descent trajectory in problem with X 2 O. Let (2t: w/)
4
= I Twe. ve). Consider the map f IR2 - IR? defined by
((x) 2 = 11²² 23 + n + (11-n))n
1 n2)2 - mu + >)>>>>>w))) Xw)) + ym² 2
((1 - (1-1 )f - - (1a(11---(((( (=
(7)
We have that (2e+1, W(+1) = f(2,, wes) holds for all t 1..00 0. In particular, (u, v4) converges to M if and only if (Ze, We) converges to (MM, and it converges to (0, 0) if and only if (2t, we) converges to (-
Proof. To ease the notation, let (2, = (2,, w/) and (x', w/) = (21+1, Wc+1) for arbitrary We have that
x' = (u') 1'= -
(u- a224 n/uu) T - n- -n--n---aall 1 y
2 m 2n/u" u
= wsw 2np(2 v) + (2) + x (2 y)
= 2 + y2² + ((1 - n/)² - mw > + yn² x² - 2ym
Also, we have that
w' ||''112 + ||,011
(u 12 x2) T (u - 120 - nnuu) + (v nuu nnuu) (v (nzu n)d=)
= ((1 - 2 +
- 41/²(1 - (=) + +
Note that, the loss function solely depends uT v y and ||u 2 + ||0112- Thus (ue, U2) converges
to .M if and only if (Za, wy) converges to 7(M).. Also, note that, T(u, = (-y, 0) if and only
if u = 0. Thus (ue, v/) converges (0, 0) if and only if (=,w;) converges (-w, 0). This completes the proof.
To further simplify the analysis, we consider the change of coordinates o(z, w) = (nz nw). Note that, under the map o(z, w) = (1z2, mw), the system f. as defined in 7 is topologically conjugate to
2 nyz 2 + ((1 17/22 - + n>w)= yn X2 2ym >
F
2 w
00 foo
w
((1 - 1 +2 - 42 - (= + TU)
μz² + -
2pr
u1= -z(1 () 14)
(8)
where we let μ my and == 11λ. With Cauchy-Schward inequality, it is straightforward to verify that the state space of F is
Ω = [2, E R w 2 2|= + ml]
The system F has two parameters, μ and v, whereas f has three, 17, y, A. Therefore, we instead study the system F. Note, trajectories of F and those of only differ by a scale. Thus all results for F extend trivially to f.
D.2 PROPERTIES OF THE QUOTIENT DYNAMICS
We show that the map F, as defined in g is a proper map, i.e., the preimage of any compact set is compact.
Proposition 10 (Properness). When 0 < / < 1 |pl. the map F is proper on Ω.
Proof. Consider [[(,,, wx)ll + oo for a sequence of points (zz, wk). Let (2fe w/t) F(zz, wk).
Assume (2/g w/)) stays bounded. Since (zk, wx) is unbounded and Ω is a cone, one must have
wk + oc. Notice that
w/6 ((1 ([]=2)² + =m)w 42x(1 (=( + 1))
21

---

Published as a conference paper at ICLR 2026
To make w/ bounded, 2 has to be unbounded. However, wy 2/2x + pol.
w/ 2 ((1 1/)² + zi)wn 4|2x| 12x + pl| |1 vl ((1 - (1] =z)wE 2|=x|
wis |1-10 v|
2 wa (|zzl - (1 =))²
Since wk: |z are unbounded, w/ has to be unbounded, which yiclds a contradiction. This completes the proof.
Consider the function Q defined as follows
Q: Ω + IR, Q(z, = w + +/V2 /22
16μz.
We will frequently use Q as a Lyapunov-like function to study the dynamics of F.
In the following result, describe the level set structure of the function Q.
Lemma 11 (Level-set structure). Consider Q2z, w) = w + vw² 16 Then, Q(z, w) 2 4/pl for all (z, w) E Q. Moreover, we have that
Ifr A/pl, then for all (z, w) E 22 Q(2, w) if and only if 2sgn(u)(=+ and w S 41p4] Q(2.w) > holds for all other points.
r > 41p1, then for all (z, w) E Ωt Q(z, w) is less than, equal to, larger than r if and only if
-16pz p2 2rw is less than, equal to, larger than 0, respectively.
Proof. When 2|2 s4/.
w² 16pz 2 4(2 + u1) 16pz 4(= - -1)² 2
Therefore, Q is well-defined in f2
When μ = O, we have that Q(2, w) = 2w. The claimed results clearly hold. In the sequel, consider + /w² 16pz and
* 0. Let Q(z, w) z (16)/(16H) and w = (r + s2/2. Notice that
/22 16pz. Then we have
2 2 (r + s)(r s) 2w 2 w² 16 2 0.
Since 2 2/2 + pl. have 2² > 4(2 11)² and hence
r + ž 4(
T5
+ uu)
2
16p
(9)
- (p² 16p²²) 16 2) S
+ 2 4/14/ |s|
4/1
We have that for (z, w) Ω
v 2 161 4/p4|
w² 16μz = 4/p1 w
+ 2 16ps = (4/14) w)2 w I 4/
w (1) 11), w 4/p)
Therefore,
{Q(2, w) = 4/p/} = (w = 2sgn (z) + p), S 4/μl} c 02, and {Q(z, w) > 1|11} = Ω (Q(z, w) = 4/p1}-
Now consider 4/p4/. When =2(= 14), we have
-16uz - ,2 2rw 0 +> r 4u
22

---

Published as a conference paper at ICLR 2026
When w -2 + p)), we have
-16pz - 12 + 2rw = 0+ w T 4
2
Therefore, the line - 16 p² + 2rw 0 intersect 0R two points, whose coordinates
are باا Since r > 4/11,, we have that 2
+# < r always holds. This implies that, for all
(2,w) w) E on { {- 16p p2 2rw s 0}. we have w 0. Thus, have that for (2, w) E Ω
and 4/pl
" + w² 16 <
+ w 167z T w
+ 2² 161z < (r w)² + 16μz - y² + 2rw < O.
The above clearly holds when < is changed
> This completes the proof.
We identify three invariant sets of the quotient system F. A set S c Q2 is said to be an invariant set under F if F(S) S.
Lemma 12 (Invariant boundary). The boundary 82 consists of two lines: {w = 2(2 + p)) w 2 0} and (w- -2(2 + 2), w 2 0}. Each of the lines an invariant set of F. Meanwhile, when 0 S V < 1 |ul, the set (Q= 4/pl} is invariant under F.
Proof. Let (z', [zz w). By direct computation, we have that
w' 2(=' + (1) = (w - -== + = -)²
(10)
w/ + 2(2² + (4) = (w +
1)2
It follows that if (2, w) E 02 = (w = (1), F(z, w) an.
According to Lemma I {Q- 4|μ|} {w 2sgn(n)) (2 + ni), w E [0, 4|al]3 (s + p), we have that the w--uddaee is given by
When
w' w (sgn(())
(1)
w)² (1- 11--))
w 2w (ggn(u)
2 1) (1 v)
w( w A x(w).
1 + v 1|m)2
2
We will analyze the image set of k(O0, 4|ul]). Clearly. the minimum of k<[0, 4/pl) is k(0) = 0. Let A = =1 V |n/. We have that x/ s/(w) = 0 if w = -2A or w = -2A/3. Notice that when 0 <1 11-1|), we have 4/pl S 224. Therefore, the maximum of sx[0, 4/ml]) is either k(4/pl) or x(-2A/3). When 4/p > -2//3, we have (1 - v)/5 < |M|| S 11-1 v. Notice that
2A 8(1 - + 14)3
R(
3
27
Viewing x<= x(=3A) as cubic function of [K|,, we have that, as 1 - V 0, k( =(- is convex on (1-v)/5 < |||| < 1 v.. Therefore, to show k( 22 ) < 4/pa for (1 w)/5 < |M| & v, it suffices
to show this holds when μ (1 - v1/5 and p
V. Notice that
1 1- - < 4 8 (1 v)
1
1- v
+ (1 - (1=
25 27
~ 1.56.
27
5
5
2 63
and that
27
-/ (1 - - + v)3 4(1 - v) + (1 (1- v)= v)
< 16'
27
which are all satisfied. Therefore, x( x(-3A/3) < 41pl when 4/p| > -2A/3. Meanwhile, we have
x(4]μl) 4|p (Mk) 13
Since < 1 - |pa. we have that -1 < ||| 1 + < 0 and that << <(4]pl) < 4/p]. Therefore, the
image set of x<(0,44) 4/pl]) is contained [0, //]] This means that the set (Q = 4|pl] } is invariant under F. which completes the proof.
n
23

---

Published as a conference paper at ICLR 2026
In the sequel, present two important properties of the map F, which will be used in the proof of our main results. In the following result, we identify the region on which a single update of F leads to decrease, or increase in the value of Q.
Lemma 13 (Monotonicity region). Assume 0 S V < 1- 1m4. When v = 0, we have that, for (2,w) E Q: (i) Q(F(z, w)) = Q(2, w) if and only (z, w) lies in the set ww= 4)
{w = 2sgn(p)) + p), w S 4|p17:
and (ii) If (z, w) & Z, we have (Q(F(z, w)) - Q(z, w) (w - uuz 42) -)
When > O, we have that, for (z, w) E 22 (i) Q( w)) S Q(2, w) if and only if (z, w) lies in the set
2²2 + +2v U < 11/2 - 2v +2²)
4 (v - 1)
12 -2v + 22 2² > -v² + 2v
z(v 1)
which contains {Q(2, w) <8 - Av]; and (ii) Q( w)) = (2, w) if and only if (2, w) lies in the sel
= =414] U {wi (W- - 11=² - 2v + v²)
4 ( - 1)
_ a ()(2,
22 > -2² + 2v
2(V - 1)
v² -v 2v + x2
Proof. Let (z', = F(2, w). Assume that / > 0. Note the case of μ < 0 can be proved
via an analogous procedure. Let r = Q(2 w) and let s w /w² - 16 When we have that (( w)) = Q(2, w) always holds, by Lemma 12 Consider r 4/pl. Using Lemma 11 we have that the sign of Q(z', w') Q(z, w) is the same as that of the inner product
4|pl,
between the vector pointing from (z, w) to w") and the normal vector (-8p,, Q(2, w)) of the line
- -16 Q(z w)² + 2Q(2, w)w 0, which is given
(-8p)(= =) + Q(z, (ww' w)
8p(23 + u2²² + (v² - 2v w + vw)z + v² F 2μv)+
Q(z, () 2v 2 = 42(1 v) + 1n)
(1I)
x 1² (r² 16p (r2 8rs² (-1+ 256y 2 (-2 2 u)v)
x 2r2 2 + 8vrs² 8rs² + 256 2,2 -
When V = 0, the above is equal to s2 ((r 8). By noticing that > 4/1l > 0, that the sign of r is the same as that of w μzz 4 by Lemma and that = 0 if and only if 2 = O, we have all the results for v =0.
When V > 0. has the same sign as
211(22 + v² - 20) - (1 v)>(w V - 16
We have that for (2, w) E S2
(2 12 2v) - -(1-v)a(w- Vw2 16Hz) S 50}
=f2² S v22 20) U { w < _1(²² 2v + v2)
4: (v 1)
22 -2² 2 + 20 }
z(v - 1)
122 - 2v + 22
and
(2/1(2² + +12
22)) - -2v)
-v)=(w - V - 16112) = 0}
[-= = 11(22 1/=2_ - 22 + v²)
42 - 1)
22 >- -v² + 22
2(v 1)
1/2 2y + 22
Next, we show that {Q < 88- - - is contained in the above set. Notice that 8 > 4(1m) + v) always
holds when 0 < 1 11/1). So 8 - 4v 4/pa and, by Lemma 11 the level set (Q = 8 -2/ is on
the line
8p
8 - 4v
w- 8 4v
+
2
24

---

Published as a conference paper at ICLR 2026
Then it suffices to show that when 2² > -v² + 2v, the following holds
p==² - 2 +2)
4:2 1)
8p 4u
8-
4v
1² 2v + x2
8-44"
2
0.
(12)
By direct computation, we have that the level set {Q = Av} intersects 0n at = v).
Hence, by the cone structure of f we have 2² < (2 2) if < 8 4v. Therefore, multiplying
=(2² + + - 2v) to both sides of the above inequality and assuming > O, we have that the inequality is equivalent to
(2- (-11=² + (4 - 6y + 2v²) n 1
2v + 1²)) 2 0
(v - (v
+ (- + 12 (- + (4- 6v 2v )z uu( 22 + v²)) 2 0
(13)
u2² + -4- - 6v + 2v²)= 2- n 1 2v + v²)) 2 0.
The symmetry axis of the parabola is (v [) 22) / > Since 1 μe the symmetry axis lies in (2 - u, +oo). Notice that
n2² (4 6y 2v pl-22 2v + -
-======~~------0 /==-~==22 20
+ 2( - v) v2 + +2v 0,
which is satisfied. Therefore,
13) holds and 12 holds. The case of < 0 can be proved with similar procedure. Thus, we have that {Q < 8 4v} is inside the set {Q(2z, w) 2 (2,(z, w))}.
Finally, when μ O, we have that Q(2, w) 2w. We have that
Q(F(z, - Q(2, w) w(2² (1 - (1- 422 - v)
4= + v) w(22 2v v²)-
is straightforward verify that the claimed results hold for this case. This completes the proof.
In the following result, we characterize the preimage map of F, which in general a multi-valued map.
Proposition 14 (Preimage structure). Assume 0 S < 1 - |u]. Consider the sets
{(z, w) [2 Q(2, w) > 6 4v}
Ag = (2, E 6:: < V v-
A2 {(z, w) E f.: 1-v).
The restrictions Flc)(Ao)) Flat(A)) are homeomorphisms onto Q. Moreover, there exists homeo- morphisms Go: Ω > cll Ao), G1: cB( + Gi (c)( [(2, w) Ω: (l(A)) such that F G; is an identity map on the domain of Gi for 0,1,2.
|1|$1 < v}, G2: Ω
Proof. Notice that the critical points of F lie in the set
{det F w) = (1 + - + w 3 2 - 22μ 2v + wv + +2²)= = =0}. (14)
Since |u| < 1 - v, the bottom tip of ? (-p, 0), lies in 1,- - v). Therefore, Ag is bounded by w = 22 + μ1) and = V- - Notice that the parabola 1 - w - 3=2² - 22μ - 22 + wy + v2 =
intersects = -( + μ1) at = = 24 + // and we have
(- +1/
+ 1
which is satisfied by assumption. Therefore, det JF vanishes nowhere on Ap
We next show that F(Ag) = Ω?. For an arbitrary (20. wo) E 22, (z, w) E F (20. wo) if (2, w) solves the following system
1d + μ2² + ((1 v)² w + vw) + v2 2pv 20
(15)
((1 - u) + =²)w - 4= - 12 + 1T) = wig
25

---

Published as a conference paper at ICLR 2026
For + O, solving [5] is equivalent solving
2 + p2² + (1 - 2) + 12
2pv 20
4x(1 (2 (2+ + +1) ww
(1 -)
z² + (1 v)2
which is equivalent to solving the following quintic equation
p(2) = 20 + +22 - 2(1 - 1) + ((-22v² 2v² + 4v - 3) z0)2
(16)
+ (v - 1))(1² -3-² + 3v + wo - 1) + (v 1)2())((( (uD - 2) 20) =
Notice that p( p--00)== -oo and p(-1 + = (v- 1) (wo 2(0 + 1²) > 0. Hence, has at least
one root in (-00- -1 v). By Lemma 12 particular, by (10), we have that, when viewing as a map on 2
(a2) = a2 f== =(1- w)}, [[),
and (2) C S
Therefore, the above root of p corresponds to one preimage in Ag. This means that F(Ao) =
For any compact set K C !2" Note K is also compact in Q. Since F is proper by Proposition 10 F-)(K) is compact. Since K n an Ø and a(l2) 5 F(aAo). we have (K) n 0.A Ø. Therefore,
= F F (K) n Ag = = (K) n cl())), which is a closed in F (K). As closed subset of compact space is compact, we have (F) (FIA,) (o.)-((P (K) is compact. Hence, FlA, is
proper map. Since ? is simply-connected, by Hadamard Inverse Function theorem, we have that FlA, is homeomorphism from Ao to S2%
We now show that F maps A. bijectively to on Since Ao C cc(())., we have that F(An) Ωo (Fo)) Since cl(AA) is compact, therefore, Al))(A)): (lo) - 52 is proper, and hence is closed (see, e.g., Lee 2000, Theorem 4.95). Therefore, F(cl( Ao)) is closed set that contains Ω. Hence, cl(s2)) = Q2 C F(c)()))). Since F(Ag) = Ω2, we have an C F(AAo), which means FlaA, is onto 02. We next show it also injective. By Lemma 12, we know F maps {z v v- 1} to {w- = 2(z + [u}] and maps {w= = 2( - } to itself. When 2 V 1, the w-update under F is given by
+ )) w/ w((1 w) + (-1+
4(1 1111+++ #++)
which is linear in w. Therefore, F ==1-v must be an injection. When w = -2(z + n)) the w-update under F is given by
w' w w - 1 + + +)²
2
1 As a function w, w' have two critical points, w = 2(1 v) and w == (1- v). Notice that
1 intersects an (v 1, - p v)). Then when (z, w) E cl(An), we have W 2 2(1 - v). Since 1 - p V/ > O, we have that the above w' is monotonic with w. Therefore, Fla(()) is an injection. It follows that, FlaA, is bijection A and Fla((o)) is bijection to Ω. Since FleK(Ao) Aa) is closed map, its inverse is continuous. Therefore, Fla((A)) is homeomorphism. The proof for A2 is similar and thus is omitted.
Finally, we analyze the behavior of F, as map onto B. We show that every points in B are regular values. Note that,
1 1- 3-
2214 2v + 12
4v
> 0
1- -
(17)
+ 32 2uz (3v²
8v + 5) 0.
For this parabola of 2, we have
(24)² - 4. 3(3 - 8v + 5) <
1 3(3v 5)(v 1)
(1 - v)² < <3 5) - v)
7
+ < 7.
26

---

Published as a conference paper at ICLR 2026
Therefore, we have that (17 holds and that
Q2, - 3z 2z11- 2v + v²
<6- 4v
1-
0 1 3z - -2=Ø 2v++1 +12
1 1- --3²-- -2z -2 + v²
)2 16yz < 6 du
1-
1
+ 16μz + +2(6- - -41) 3z²
22p 2v v2 - (6- (6) < 0
1 1
+ (6z - 9)2² -2p(Av 5)2 + (2v² 9r2 + 13v 6) < 0.
For this new parabola of 2, we have its discriminant is negative
15 - 4v) - 12(3 2v) 3v + <
+ (1 - (5-
2v) -v) 1) - 2) <0
+ 32, 182r² 331 191 < O.
By differentiation computation, claim that the last equation holds when [0, 1]. Therefore, prove that the maximum Q value on the parabola :: 1 - w 32² 2zμ 22 + wy + v² = 0 is at most 6 4v. Notice that all the critical values of given by 00 J F().. But what we have shown
and Lemma 13 we have
Q(2, w) I 4u, V(2, w) E F(.).
Therefore, have that every points in B are regular values.
Now we show that [ (z, w) = 3 for (z, w) E B. To this end, we first consider a special point: I (-p + p(1 - v/ w0 (1 =)²) F(0, w") for some w" such that x² E B. Notice that the w-coordinate of x² tends to infinity as w' tends to infinity. Hence, w" can be arbitrarily large while keeping x² B, i.e., (x²²) > -v. We show that |F (x=) 3. Plugging 2o u1 v)²
and w00 = w² (1 v)² to (66] gives
= x + uz² 22v 1) - -3p(v (
(18)
+ x² + 22² = =v - 1)) + 3y4z (v--1)(1++++a
The left-hand side is a continuous function and thus has a finite upper bound when E [[-1+v, 1 v]. The right-hand side is a parabola, whose symmetry axis is -3//4. Hence, it's global minimum
9 - - 1)2 - 1}
Notice that this quantity tends to 00 as w* tends to +o0. Hence, for large enough w", equation (18) does not have solution on 1-1 + p, v/. It follows that Fx"(x") does not have any element in {2 E [-1 + v, 1 w/} except (0, w*)) By what we have shown, Flei(A)) and F Fle((A)) are bijections onto Ω. Hence, F--(<*) have exactly one element in An and exactly one in A2. Therefore, = [x-1(=*) Now consider any other point in B and path connecting x² and that point. Since every point in B is regular value, by the stack of records theorem, the function |F |F- (F- is locally constant. (Stack of records theorem requires the domain be compact and this can be achieved by confining F on w < W for some large enough W so that the image contains the path. This is guaranteed by properness of F.) Note the path is compact and thus F (z, w)) is constant on the entire B and hence is 3.
Given (z, w) E B, F-) w) we already know there are exactly one point in Ag and exactly one point in A2. Hence, the third point must lie in {(2, w) E Ω: |z| < 1--- w). We define this map by G1: > {(2, w) E Ω: |2| <I-) and let A₁ G, (B). By what we have shown, det J vanishes nowhere on Aj. Note by construction of A,, Fl,, (A1) B. With similar treatment used for Ao. we can show FA,, is proper. As B is simply connected, we have that F, when restricted to A1, is homeomorphism to B. As shown above, F is bijection from |=| = +(1 v) to a2. Moreover, we claim that G can be extended to {Q - 6 42) in a bijective manner, as one can choose C' slightly smaller than 4v and apply the same analysis for {(z, w) E 82 1, > C} as we did for B. Hence, F. when restricted to (I,A) is a bijection onto cB. Note, Fla(A,) is proper and hence its inverse is continuous. Therefore, Fl Flc((Ar) is a homeomorphism. This completes the proof.
27

---

Published as a conference paper at ICLR 2026
E PROOFS FOR SECTION 3
In this section, we present the proof' of our main results. The key idea is to first analyze the quotient dynamical system introduced in Appendix D and then translate the conclusions back to the original gradient descent system.
E.1
UNREGULARIZED PROBLEM
Preliminary results are first presented in Appendix E.1.1 and the proof of Theorem | given in Appendix 1.2
E.1.1 PRELIMINARY RESULTS
As discussed in Appendix system F:
the gradient descent system GD2 is semi-conjugate to the following
23 + p2² ww +
F
(= + 1)w 4z m
w
where μ yr denote the parameter of the system and the state space F
[w 2 2| ml}.
In the following several results, we characterize the long term behavior of the orbits of F. We will use the terms trajectory and orbit interchangeably. We say an orbit {*k, Wx converges to a set $ ((22,, wk), S) O, where d(z, 5) yes d(x, y). Unless stated otherwise, we use (2', w/) to denote F(z, w)
Proposition 15 (Long-Term Term Dynamics). Assume |pl S 1. Given an initial condition (zo. wo) E S we have that:
If Wi < paz + 4, the orbit stays in {w pzz + 4) and converges
(w 2sgn(p)) + u), w S 4/u U {2 00.
If woo pizo + 4, the orhit either diverges, in the sense that Wik co, or converges {= 0) in finite steps.
If wo = p2z- + 4, the orbit stays in {w pzz + 4].
Q(F(2, w)) Q(2, Let (zk, wk) pk (zo, wy) for k > 1. When < 420 + 4, by Lemma 13 we have Q(2+11, Wx+1) < Q(z,, wx) for all 0. Hence, the trajectory stays in the region [Q < 8) {w < uz + 4]. Since Q we) is monotonic and is non-negative, it converges to some finite value and AQ wx) converges to zero. By Lemma [3 we have that
Proof. Consider the function AQ(2, w)
(AQ 0) = (= 0) (w = pz pz 43 U (w (z + p), S 4|nl]
If (2k, wk) does not converges Z, there exists Eu such that for any K there exists such that d((z,, wx), 2) > En- Note that, the function AQ. when restricted to the compact set {w S juz + d((z, w). Z) 2 60). is non-positive and continuous. Thus, obtains its maximal value and the maximum is strictly negative. Hence, d((2,, wx), Z) Eo implies that AQ (2k, wx) < for some > O, which contradicts the fact that AQ - Therefore, (2k, wx) converges to
K
zn (w < p2 + 4] = (= 0} U (w = (sgn(n)( + p1), w S 4|p1}-
When wi > 1450 + 4, similarly, we have that Q(z++1, Wk-1) 2 Q(zz, wx). Hence, (Fx, ws) stays in the region (w > juz + 4} for all & > O. Hence, Q(za, wa) either diverges to infinity converges to a finite value. it diverges, Wi must also diverge, since the function Q, when restricted to (w S w) for any fixed w. is continuous and hence is upper bounded. If Q(2k, wk) converges to some finite value, then AQ converges to zero and the trajectory must remain within the compact region {w < C} for some C 0. Using arguments similar those above, have that (zk, wx) must converge to
28

---

Published as a conference paper at ICLR 2026
Z Zn {w u2 44 f2= = 0}. Now assume the convergence is in infinite steps, i..,, |eyl # 0 for all E N. Then the sequence |zx+1/ 2 is well defined and converges to one. Notice that we have
- 1 + MEz W. + 1 | + p ww 1||-
(19)
Note as 2 + 0, the above lower bound is dominated by |un 1/. Since {w p 4} n f2 0}- = {(0, w): w 2 4]. Bxl2 is lower bounded by - 8 for some > 0. This contradicts the fact that |zz+1/zz| converges to one. Hence, the convergence must occur within finite steps.
Finally, the result for the case wi pizo 4 directly comes from Lemma 13 This completes the proof.
Proposition 16 (Convergence). When |u| > 1, almost all initializations does not converge to {z 1 0}. When |a] < 1, almost all initializations with Q1z, w) < 8 converges and almost all initializations with Q(2, w) > diverges.
Proof. Consider ||| 1. By Proposition 15 when Q(z, w) 8, initializations either converge to {z = 0} in finite steps or diverge. Notice that converging within finite steps means that the initialization lies in the set
U F 0}))
As the Jacobian of has full rank almost everywhere, the above iet measure-zero set (Ponomarey 1987). Hence, almost all initializations with Q > 8 diverge.
When Q < 8, by Proposition 15
we have that the orbit converges to f2 0} or to (Q 4|pl]
{w 2sgn(n) ())(( + p), w < 4 p Notice that, when w = (z) + μu), the w-update under F given by
1 w (w-2ww2220 2222211
w = x(=)(w)
(20)
Consider k as a one-dimensional dynamical system defined on [0, m|]. For this one-dimensional
system, it is straight forward to obtain that, there are two fixed points: w 0 and = 2 jc], and also that, when |ب|| < 1. all orbits converge to w = 2|p4| except the one with initial value w = 0. Note that
W = 0 corresponds to the fixed point (-u, 0) of F. The Jacobian of F at (-H, 0) has eigenvalues (1+ x)² (-1 + 2) Therefore, (-u, 0) is hyperbolic fixed point. By the local stable manifold theorem and the fact that (w = sgn(n)) (= + nu}} is invariant under F, we have that the basin of
attraction of (-p,,) 0) can be given by
0)- =
{w = - sgn(()() + +M))),
for some small neighborhood O of of (-H, 0). This set is a measure-zere set, since the Jacobian of F has full rank almost everywhere. Therefore, for all initializations lies in {Q(2, w) < 8} (--,, ps 0), the
orbit converges to {= 0} or, to iQ 4/r4} \ ((-H,,0 O)). Consider any fixed initialization (=o. wp) is a hyperbolic fixed point, the orbit does not have an accumulation point in some neighborhood of (-xu,00 0). Therefore, the omega-limit set w(zo, wo) C {Q 4|p } ((-1, 0)}. The omega-limit set is non-empty, as the orbit is always bounded. Note, for any m E w(2p, Wo)), we have that 1a (m) + (0, -2(p1) as N tends to infinity, as explained above. Since the omega-limit set is forward invariant under F and closed, we have that (0. 22(m4) E w wo). ). It follows that the orbit visits arbitrarily small neighborhood O of (0. -2|pl) some time k. Take the neighborhood as O { (1((z,)) w) 4//=|| < E, |z| < $). Notice that
in the second case. Since (-1,, 0)
-++1
1x + μス w/k + 1 < 1 p + |wkk 1/.
Zk
Since |22 μ2| + |w- + 2/pl 1| as (z, w) + (0, 2/ and |2|m] - 1| < 1. we can choose
& small enough such that
< 1 - 8 for some 5 > 0. Therefore, |=++1| < | Meanwhile, note that the Q value monotonically decreases the orbit is in (Q 8). Hence, for all 2 k.
(zj, ()) E O and $1t1 Therefore, have that almost all initializations with Q converge to {2 0).
< 8. It follows that 2j + 0 and the orbit converges to {2 = 0).
2j
Finally, consider ||| > 1. Since 2|u|| = 2sgn (z) (0 (), we have that inf { {w: (0, w) E .) = 2/p >
2. Using arguments similar to those in Proposition [5] in particular in (19). we have that converging to any global minimizer must occur within finite steps. As shown above, those initializations form a measure-zero set. This completes the proof.
29

---

Published as a conference paper at ICLR 2026
From the proofs of the preceding two propositions, obtain the following corollary.
Corollary 17. Consider gradient descent with step in problem (2) Any global minimizer with ||u]l2 + ||a||² 2 2 2/n is an unstable minimizer, i.e., it repels orbits its neighborhood. Consequently, inirializations that converge to such minimizer form measure- set. Moreover, when |p] >> 1, i.e, nlyl > all global minimizers are unstable.
Next, we analyze dynamics on the invariant set (Q(2, 8) = {w paz + 4]. Observe that when restricted this set, reduces to the following one-dimensional system:
32, F(=) = 2 μz² z(pz + 4) -333,, 2 =E [-22).
The following result shows that F is chaotic dynamical system.
Proposition 18 (Chaotic Boundary Dynamics). Assume |e| 1. The system F on == [ is Devaney chaotic and has topological entropy log 3. Moreover, there exists periodic orbits with any period and thus F is also Li- Yorke chaotic.
We first seek simpler system which is topologically conjugate to F. Notice that F is continuous map from [-2, 2] to itself. Consider vo(()) = z/2, which a homeomorphism from [-2, 2] to [-1, 1]. and Fi()) = 4z - 32, which a continuous map from [-1, 1] to itself. We have that
Proof.
o (o(2) 1( 2
3z
23 3z
vo o F()).
Hence, F is conjugate to Fj. Now consider (b2)) [-1, 1] [-1, 1, and
sin( 2), which is a homcomorphism from
3z + 2, -32,
x E |-1,-1/35
F2(2) =
x E ( 1/3, 1/3);
3z 2,
x E [1/3, 1].
which is continuous map from [-1, to 1-,, 1]. We have that for E 1-1,,-1/33
3T F1 ((2) 4sin 2 -2 sinn 2) sin 3
= (3z (3z+ 22) v F2(z).
Similarly, one can verify that Fg a 3/ = vp o F2 also holds on (-1/3, 1/3) and [1/3, 1]. Hence, Fg is topologically conjugate to F.
Note Fz is a piecewise linear continuous map with slope equal to +3. Hence, the topological entropy of F2 is equal to log 3 (De Melo and Van Strien 2012, Corollary of Theorem 7.2). For a univariate map on compact interval, positive topological entropy implies Devaney chaotic (Elaydi) 2007
Theorem 3.13). Due to the conjugacy. have that F is also Devaney chaotic. Also, conjugacy preserves topological entropy (Robinson 1998 Theorem 1.7, Ch.8). Therefore, h(F) log 3.
We now show the existence of periodic orbit with any period. According to the Yorke Thcorem and Yorke 1975), sufficient condition is that there exists a point x such that F (z) < x < F2(x) <
F2 (z). Consider x = -577. We have that F2(x) = -1/7, (x)) = 3/7, and (x) )) = 5/77. Due
to the conjugacy, also has periodic orbit with any period and is Li- Yorke chaotic. This completes the proof.
We translate Proposition to the original gradient system GD,,
Proposition 19. Assume μu| < 1. We have that h(GD,) 2 h h(GD, aD;)) ) 2 log 3, and GD,, admits periodic orbits with any period.
Proof. By Lemma 13 OD is invariant under GD,, Since the map T(u,v) (z, w) is semi-
conjugacy between GD, DlyDD, and F, we have that ((GD,loo;) 2 ((F)) log 3 (see, e.g., Robinson 1998 Theorem 1.7, Ch 8), where the equality comes from Proposition 18 Meanwhile, since 0D' is
an invariant subset of 2d we have that h(GD,, Z ((DDlle ) (see, e.g.. Vries 2014 Proposition 8.1.7).
30

---

Published as a conference paper at ICLR 2026
Next, we show the existence of all periodic orbits. As shown in Proposition 18
F is conjugate to Fgg
32 + 2, -32,
E lo |-1, -0,,-1170 -133;;
(=)
E 11 (-1/3, 1/3); 12 [1/3. 1].
(21)
3z 2,
Note F2 is a piecewise linear map defined on [-1, 1], with each piece mapping onto the whole interval
[-1, 1J. With classical analyses in symbolic dynamics (see, e.g., Devaney and Eckmann, 1987 Ch
1.7), we have that F2, when restricted to [-1, 1] {ar [ 1, 1]: F2 (x) + +1, ANJ. is conjugate to the shift operator a, defined on the following set
{s {0, 2. s is not eventually constant or 2},
such that (ss) = (8081882 ... = ($1$2 ..) ). The conjugacy is given by the map Ç:
((x) (8081 ".), by sy=1 (
where E {0, and 1 is defined in
Now we construct periodic orbits (wx, Uk k20 with periodicity K for arbitrary K. Consider T(u,, 0x) = (2k, Wx).. Consider new coordinates in ue-space, given by (u + v, u v), which preserves all dynamical behaviors of GDD,, Notice that
uk+1 + Ub+1 = uk 2 + Uk 1uE = - myx (ux + ux).
(22)
2k+1
Ub+1
uy 22 2k + 17244 = (1 + mex) Uz)..
It follows that, during training us + uk and U Uy must lie in the minedimnnioona space spanned by uo + th and ug Un respectively. Meanwhile, shown in Proposition 8 if T v²) T(u, v). then |u' + v'll = ||u + vlll and ||u' - o'll = ||uu vll- Also, it is clear that if T(u v') + T(u, v). (u', v²) # (u, v). Therefore, whenever (zx, w&& w2)220 is K-orbit and,
# (k E (o..................... ...
- 1]: 1 11th < 0}, # (k E (0, ... -.,,,....111111 13: 1 726 <
are two even numbers, where #S is the number of elements in the set S, we have that (ux, Uk v2)x20 is a K-orbit. Notice that, under the conjugacy between f and F2, these two numbers correspond to the numbers of visits of the Fz-orbit to 12 and Lo. i... the number of 2's and 0's appearing in the symbol 8. For = 2, take (zo, Wo) = ( (- 4 - 2μ). Notice that (-2, 4 2/4) is a fixed point under F. and that, 1 T750 y > 0 but 1 + 7)50 va < O. Therefore, (ug, vx)x20 is 2-orbit. For K = 3. take (20, Wo) be the element corresponding to (001 001...), which is a 3-orbit in the symbolic system. As there are two 0's and zero 2's, (uk: Uk)k2 is a 3-orbit. For K 4, take (2u, wo) be the element corresponding to (0011 0011 ....) ). and we have that (ux, Uk v2)x20 is a 4-orbit. For K = 2N + 1, take (zo, wo) corresponding to the repetition of 2N 0's and one 1; for K = 2N 2, take (zo, wn)
corresponding to the repetition of 2NN O's and two I's. By this, we have that GD,, admits all periodic orbits, which completes the proof.
In the following, we show that, the original gradient descent system is not chaotic in the sense of Devaney when d 2 2.
Proposition 20. Assume that n/yl < 1 and d 2 2. The system GD2laD, not topological transitive.
Proof. Let (ux, vk)220 be orbit under GD,, Consider new coordinates pk uk + Us and
Is Ug 7g for 1: 2 0. Recall (22) and notice that 2ạ u, U/ - y -y (1lp&11 || //4 Thus, (Px. 9x) evolves autonomously. Since (u, v) + (p, q) is a homeomorphism, the un-system is topological transitive if and only if the pq-system is.
Notice that, if 4 n([|u]]
[|-l(2) y T
v). we have that
||u|| 2 + |1e1
n/9l
4 2 7[|uu12 + =1u113) - 7
= r7(1
)(1lu11 + 11112))
2
2
31

---

Published as a conference paper at ICLR 2026
where we used that |u. Tul vl I (I|u1l2 + l/ell1)2.. Since nlyl < 1, we have that ||u1 + 110112 < 8/n. Therefore.
(u, E DD, + ||ul12 + ||012 + v (11ul12 ||u112)2 16y(u Jo w-
8
1]
1n[llll1 + [1w112) - viyy("' Touvv y)
(23)
+ (u T vJ)
nIa
u2y/22 /2/2-111 Id n/a
11 1
1/-y²
-n Ia
The last equation is quadratic form and the eigenvalues of the coefficient matrix are 17 + 2n each with multiplicity d. Therefore, when 7/yl < 1, the quadratic form is positive definite and defines smooth ellipsoid of dimension 2d - 1. Notice that (u, v) + (p. q) = (u + U, - v) is a bijective linear transformation. Thus, 8D' is still an ellipsoid in pn-coordinates.
Now fix any (yo, (0) E OD, and an open neighborhood = XU, of (po, 40) where po E Up C Bd and go E U C Rd. According to (22), we have that
UK O CC
{ep: p Up, E ER {dq: q Ug. d E R) C R2d
Clearly, when 2 and U is small enough such that (0, 0) E U, is not dense in R2dd and there exists an open set V such that V aD, # Ø and C V Therefore, GD,løD, is not transitive in po-coordinates. This completes the proof.
We proceed to show that when the initialization is near the boundary, the orbit can visit any point in the state space.
Proposition 21. Assume lu| < 1. Given any (2", w²) S and any open set O C Q such that on on w) = 8} & Ø, there exists N 2 0 and (z, w) E O such that Eo (z, = (z", w²).
Proof. As in Proposition 14 let Go: Ω - (Ao) denote the inverse of F FleitA))) Let mg = (a w²)
and m. = (ma) for k 2 1. We first show that limg- My = mn = (-2, 4- 2μ)). Note for all A 1, m& $ (Q - 4|p1]- Therefore, by Lemma 13 we know that ((m²) either stays at 8 or monotonically approaches 8. Hence, as we shown in the proof of Proposition 15 my must converge the compact set
Z {AQ 0) (= 0) (w μz + 4 fw 2sgn(n)(2 w), 4/p1}
As Gl) C cl(A)). me must converge to the set
Z m = (= -1, = μ1z + 4}
follows that the omega-limit set (mo)) C (Ac)(Ap). Notice that, when restricted to {w = +2 the system F reduces to F(2) = 23 -3,, z E [- Then GolZne((A)) corresponds to the branch of f-1 whose image is [-2, -1] 1]. Using the conjugacy between F and F2 shown in Proposition 18 it's clear that all orbits under Go (2ne((A)) converge to = -2. Therefore, for any E w(mo). GØ (q) + 2- 4- 2n) as N + co. Since the omega-limit set is invariant and closed, we have that (-2,4 2p.) w(m)), which implies that my visits an arbitrarily small neighborhood of (-2.4 - 2μ).. Note, the eigenvalues of JF 4 - 2p) are 9 and 5 - 2p. Hence, (-2, 4 2p) attracts all orbits of Go in some neighborhood of itself. Hence, my - m as k Dc.
Now consider any open set O that satisfies On n = 8} On {w = pz + 4} is not empty. We show that there exists x E and n such that F" (a:) = m². Recall that Fl(q=s) is topologically conjugate to the piece-wise linear map F2 as shown in Proposition 18 For Fz. we have that
2j 3$ j 0, 0........ 33).
F ((+1)) Ukzof-14
which is dense in [-1,1]. By symmetry arguments, we have that the preimage of -1 is dense. Therefore, using the conjugacy, we have that there exists O and such that F- (m") ie.. (x) = m Notice {Q = 8} C cl(B), by Proposition 14] we have that there exists .., in. e {0, 1,2) such that Gin,,
G2 (m") Since the composition of G, 's is continuous, there
32

---

Published as a conference paper at ICLR 2026
exists neighborhood Ô of m" such that Gi., o...o G., (Ö) O. Since my m.. there exists n2 such that m = GD (mo) E Ô. Taken together, we have that
Giny o Gi Go (mo) _ mn O.
By the definitions of G,'s in Proposition
14 follows that p pm (mi) mo,
which completes the proof.
E.1.2 PROOF OF THEOREM AND THEOREM 2
Proof of Theorem ! According to Proposition 7 any measure-zero event in system F corresponds
to measure-zero event in system GD,, According to Proposition the orbit of GD,, converges to {u" y if and only if the orbit of F converges to {= 0}. and the former converges to (0, 0) if
and only the latter converges to (-y, 0). According to Proposition 16 when |/| < 1 and for almost
all initializations (z, w), the orbit converges to {a = 0} if (z, w) Notice that μ = my and due to the conjugacy
Q(2, w) <8 + v(| + (10112) + 18 (1 3 + 2|xl3)) 16ny nuu -v <8
8 "
||u1 + ||0112 + v (1101 + || 16y (u v y)
Also, by Proposition 16 when Q(z, w) This gives the critical step size 3
| almost all initializations do not converge.
We now show the sensitivity to initialization. Consider any open neighborhood W C R2dd such that W n OD' + Ø. Notice that the Jacobian of the map T drops rank if and only {u = = Also, as shown in (23). OD, defines a smooth ellipsoid of dimension 2d - 1. Notice that, {u = =v} is the union of two linear subspace with dimension d. Therefore, since 2d - 1 2 d and an ellipsoid is curved everywhere, there always exists a point E W n 0D', {ur - and neighborhood -v} Ø. The Jacobian of T is full rank at all points in W, so by constant rank theorem, T(W) is an open set. Meanwhile, T(0) E T(aD')) Hence, under
W of f such that W C W and W n
the conjugacy (8) we have T(W) n fw= = μz 4] + Ø. According to Proposition 21 there exists (z', w'), (2" w") T(W) such that FN w') converges (0, w²) with any w" E 12/. oo) and F& (z", w") converges to (-M, 0), as N tends to infinity. Therefore, there exists 0', 0" E W such that
N GD n (0') converges to global minimizer with squared norm in [2|y]. oo) and GD
N (0") converges
to (0, 0). Notice that
|||| + |||| 1 > 2||u|| ||||||
2 2/u v/.
Therefore, the minimal squared norm at {u T v = y} is 2|yl Also, notice that
|uu T vv I1. Tr((uu" vu T) (us ve = ||u||* + |||||* -2(22 21u (1]ul] + ||0||/]2
2(u vv)
(||ul|² + || 2(u T1)
Hence, any global minimizer, the imbalance is lower bounded by the squared norm. Hence, by what we have shown, arbitrarily large imbalance can be also attained by initializations in W.
Finally, the lower bound for the topological entropy and the existence of periodic orbit of any period directly come from Proposition 19 This completes the proof.
Next, present basic property for the unregularized scalar factorization problem: the sharpness coincides with the squared norm at the set of global minimizers. This has been proved by Wang et al. (2022). Below restate their results.
33

---

Published as a conference paper at ICLR 2026
Proposition 22 (Wang et al. 2022 Theorem F.2). For the unregularized scalar factorization
problem 2 the eigemvalues of the Hessian L are =(u v - y), each with multiplicity
and 2 21 ||ol? + v //|//] ||]|3) + 4(u y)² + 8(u' "0u'v- y)u v). Consequently, when
we have that
Xmax
Tr = 1| 11111
Proof of Theorem 2 the -yssstem, there exists a positive-measure set of initializations that converge to {(O, w): w E E.). Notice that
Assume E = (ee,e') C (Ymin: 2/1j. Let E' = (en, e'n).. We first prove that, in
1
|= + pazk wk. + +1 2
(24)
Zk
equivalent to
Wik 2 2x + p.zk and Wk S 2 + μa + 2.
Notice that (i) the convex parabola = 2² + μ2z + 2 intersects the w-axis at (0,2), (ii) the level set {Q (z, w) = cf are straight lines intersecting the w-axis at e/2). Therefore, there exists 5 E (0, 1), C' E (4/u4,, 4) such that all points in
T == ((2, w) Ω: |=| < <, (,, C}
satisfies |zx+1/2k| r for some constant ? E (0, 1). Notice that, this holds for all sufficiently small ó and thus we fix y. For (Zk, wx)) E '" we have that -= 1 < 8. Mcanwhile, since Qz*++1 We+1) S Qzz,, wx) by Lemma 13 we have that (2K+1, wk-1) E T. It follows that if the initialization (zo, w)) is in T, the entire trajectory remains within T. Hence, we have that l S |= for all k: 2 0, and that
Wk++1 - wkl =
1x - 4) - 42xp1 S |Ex/?/wx 4 + 4|al| S Cilaxl
where C1 is independent of 5. We have that,
[wo - wol S
2
<
1 1-
k=
Hence, Wi00 wol - -> Then, there must exist a sufficiently small sub-interval E" C and sufficiently small 8 such that, for all initializations in
A = {(2, w) Ω: |a| < 8, /w| E").
which is positive-measure set, the trajectory converges to {(0,w) w E E').
Consider any open neighhorhood W C Rd2d such that W n n * Ø. As shown in the above proof of Theorem 1 there always exits a point E W NOD' \ fu = ナf} and neighborhood W of 0 such that W W, T has full rank W, and T(W) is an open set satisfying T(W)nfu fw uuz + 4) Ø. According to Proposition 21 there exists (2', w²) E T(W) such that FN (z', w²) E A. By continuity of FN, there exists neighbor A' of (z', w') satisfying A' C T(W) and FN (A') A. Since Thw W + T(W) is continuous, T (A') is open in W and has positive measure. By the conjugacy
g and what we have shown, for all points in T (A'), the final norm lies in E. This completes the proof.
Lastly, we discuss the technical novelties in the proof of Theorem in comparison with the techniques of Wang et al. (2022). While their analysis is also based on a careful study of the gradient descent system, is restricted to specific subset of the parameter space. In comparison, on a global analysis of the system. The key ingredients are: (i) we identified an equivariance symmetry in GD,, and the corresponding quotient dynamical system defined by the map F (Proposition 9): (ii) we analyzed all the branches of the inverse map F-1 (which is multi-valued map) and carefully studied the inverse dynamics defined by F-1 (Proposition 14 and Proposition 21 (iii) we used symbolic dynamics to show that GD,,, when restricted to the convergence boundary, semi-conjugate to Devaney chaotic system (Proposition (8) and that GD,, admits periodic orbits with any period (Proposition 19p.
approach is built
34

---

Published as a conference paper at ICLR 2026
E.2
REGULARIZED PROBLEM
Similar to the previous section, preliminary results are first presented in Appendix E.2. and the proof' of Theorem and Theorem is given in Appendix E.2.2
E.2.1 PRELIMINARY RESULTS
Unless stated otherwise, we use (z', w') to denote F(z, w). We first show that when the step size is small enough, the quotient dynamics F is predictable.
Proposition 23. Assume < < lpl. For almost all (2,w) 9 if w) < 4u, we have that F (z, w:) converges to T(M) as N + co. If (z, w) < 4 - 4u, for any (u, v) that satisfies T(u,v) v) (z, w). GD (u, v) converges to p (u, v), as defined in Theorem
N -
13 we have that Q(F (z, w)) monotonically decreases, Since Q(F (z, w)) is non-negative, it converges to some finite value. It follows that (2, converges to
Proof. By Lemma
{AQ = 0) n {Q < 8 - 4w), which is equal to {Q = 41u1} according to Lemma 13 Note {Q= 4/p } {w (s + pu), w S 4/ul} by Lemma I1 The w-update under F on {Q- = pu } is given by
xk=)(w) = wl 1 + w- 1m|)
When |p| v, k(w) has two fixed points on 10,4/41
0 = 2(ml-v)). is straight forward
to obtain that w = 2[1m4 - z/) attracts all orbits on [0, [4[] except the one with initial value w = 0.
Note = 0 corresponds
the saddle (-H,,), 0). where the Jacobian of F has eigenvalues: (1+ p w)²
and (-1+ /z 100--)) 2/²² Hence, (-p, 0) is a hyperbolic fixed point of F. Note also that, w = 2(144 v) corresponds to (-sgn(u)) 2(μm] - v)),, which the only element of ((M). The Jacobian of F at
this point has eigenvalues: (-1 + 2x)² and 1 2|4] + 2v. Hence. (~sgn( (1) 2(1u] - -v)) attracts
nearby orbits. Then with the omega-limit set arguments similar to those in the proof of Proposition 16
we have that, for all initializations that do not lie in the basin of attraction of (-14,) 0):
WoD {w = 22sgn()) + +1)3,,
where O is a neighborhood of (-p,, 0), the orbit converges to T(M). The above basin has measure- zero since the Jacobian of F has full rank almost everywhere. Therefore, for almost all initializations with Q w) < 8 - 4v, the orbit converges to T(M). When |4| S 1/, k(w) has only one fixed point w = O. which attracts all orbits [0, 4/pl]. Note in this case, (-H, 0) attracts nearby orbits, since both eigenvalues of J have norm smaller than one. Then, with the omega-limit set arguments, we have that for all initializations with Q(z, w) < 8 4v, the orbit converges to T(M).
Next, we identify the converged point under GD,, when Q < 4 - 4v. Without loss of generality, assume y 2 O. Let (Zk, wx) 2220 be the orbit and 7 (uz, Un) (Zx, wx). Consider new coordinates
p = (u + v)/V2 and q = (u v)/ Note that when y > X, the set of global minimizers is given by
au = u, || = y - A] = (q= = 0, ||pl = 2(y - A)}-
Notice that
V2. ph+1 up+1 + uk+1 Uk 72 nuuu + Us - 12 77dRR
= (1 1 - 172) + Ux) = (1 - n - n1a) Pc-
Under the conjugacy we have
Pk = poll - žj z -
v)
(25)
Therefore, the converged point is either (v2(y- - X) | 0) or 2(v X) TPoll
0). Since the global minimizer is given by {q = 0, ||1l2 = 2(y - - the former point is the minimal distance solution and the latter point is the maximal distance solution, under the py-coordinates. Note the
change of coordinates which preserves distance. Therefore the same statement holds in the ww-coordinates.
(u+w)// /22 and (u -o-r//22 is given by an orthogonal transformation
Note that,
Q(z, w) = 4 4v + 2pp 2(1 - u)² (1 v)w 0.
35

---

Published as a conference paper at ICLR 2026
The above line intersects with 0Q at (1- ( and (1v-1--(v-11+4)) Therefore, for all initializations that satisfy Q(2, w) < 4--48 4v, we have |2| <1
v. Meanwhile, since 4 - 4v < 8 4v, we have that the {Q < 4 - 42) is forward invariant by Lemma 13 i.e., |a| < - holds the entire orbit. This implies that, when (= Wo) < 4 4v, we have that + 2j - v > 0 for all j 2 0. By (25), the converged minimizer in pg-coordinate has to be ( (v2(y= >) Poll 0), which is the minimal distance solution. This completes the proof.
We now proceed to show the projected boundary T(DD") is self-similar.
Proposition 24 (Self-similarity Assume 0 < < 1 |t). The boundary T(aD")) is self-similar with degree three.
Proof. We use D for D", for notation simplicity. First, we prove that T(āD) (T( Notice that, by Proposition 9 orbit under GD,, converges to M or to the saddle if and only if the corresponding orbit under F' converges to T(M) or to (-H, 0), respectively. This implies that T T D and T(D*) (D)) Since T' is continuous, we have that OT (T-1(((D)) = 0D c T (8T(D). Hence, T@D C (D()..
For the other direction, consider any point y $ T(aD). Then we have T (v) n 8D = Ø. Note T (y) is connected and compact by Proposition 8 Thus, there exists an open neighborhood O of T(D) or T(O) c T(D") = T(DY.. We claim that for any yf that is sufficiently close to y. y' E TOO). Notice that, if & an we have that (v) n {u = -v} = Ø0. Thus, JT is non-singular at (y) and, by constant rank theorem, T(O)
T- (g) such that C D O C D Hence, T(O)
contains neighborhood of y, which gives the claim. If 82, without loss of generality, assume that y = - μ4, wo) for some we 2 0. Then F- (v) {u=uu = ||u]2 = wo/2). Consider any (u', e") E T-T:- (g/). As shown in the proof of Proposition g when I1y/ - y/l tends to zero, ||' - -1'1 must tend to 0 and ||vd' + 12 to 2ung. This implies that (u', v') tends to F- F-1(y). Thus, if y/ is
sufficiently close to y. we have T-1(y) C O and thus y' E TO)). Now, given that T(O) c T(D) or T(O) c T(D), we have y & (D) Hence, (T(D) C T(D) and (T = T(OD).
Next, we prove that F(OT(D)) = OT(D). Let A = =(DD). Notice that, by Proposition 23 {Q(z, w) <8- 4v) basin of attraction, A, is an open set due to the continuity of P. It follows that 0A C A C {Q 2 8 -2). Now consider any point x E F= (Q.).. (aA). We claim that a small enough neighborhood O of æ is mapped to neighborhood of F(x). Note the claim holds trivially if x is a regular point of F. Assume that det Jxxx) = 0 and recall that the critical points of is given in (4). Since F(x) E OA C A and (A)( C A we have that x E A C (Q> 8 - 4v}. Hence, as we shown in the proof of Proposition|// 14 x E dAg or x E dAg, and the claim holds according to Proposition [4 Now, since F(x) E OA, F(O) contains point y E A and point E A By the previous claim, F n O + Ø and F (x) no + Ø. As F-:FFAAA (A) C A and F-1(A9) C A x E aA and F-1 F-1(AA) OA. Since F is surjective by Proposition 14 we have Fo F FF-(OA)) ŌA. It follows that
is an attracting neighborhood of T(M) U ((,,,( Thus, the corresponding
0A = Fo=FoFF (-A c F(AA).
For the other direction, note that for any E 0A, since E cl(A) and F is continuous, we have that F(y) E (A)) cl(A). Therefore, y E cl(A) A cl(AA) \ A. Since F-'(A) C A, F(y) & A. Then we have that F()) E cl(A) A = 0A and F(AA) C dA. Therefore, F(0A) = OA.
Since F(AA)= = 0. and AA c (Q 2 4v). Proposition 14 gives that OT(D") U (.
where G,'s 's are homeomorphisms. As shown in Proposition 14 G, (G,)) is empty whenever i 4 j. Therefore, T(8D)) is self-similar with degree three. This completes the proof.
In the following, show that D, has an unbounded interior, up measure-zero set.
Proposition 25 (Unboundedness)) When μ 0,0 0 < < there exists a. 0 such that, for almost all initializations that lie in {(2, w) E Ω: |z| < a exxp(-bbr (bbw)), the orbit converges to the minimizer.
36

---

Published as a conference paper at ICLR 2026
Proof. Let (z', w²) ) F(z, w). Note, when μt 0, the unique global minimizer of L corresponds to (0, 0). Let a =1-22 v. Then 0 < a < 1. Assume that |a| < a exp(-bux) for some a, b > 0. We aim
to show that |2/| exx((-butt Notice that
2 |2 (a. aw)=
|z| 1=|3 (a² + aw))zz
S a exp( -3bw) a -bo aexp(-bbr)e aw).
Also, we have
w' = w(2² + a²² 42²a S (w + 4aja² exp(-21ru) ) + wa.
Note a eexp(-bull decreases w' increases. Then,
exxp( (-bbww) |='l
b((w + 4a))a exp( (lnno wa))) > as exp(-3hno) + aexp(-bu)\a aww)
exeep
b(w 4a:)a² exp(-22np) (2mm)) -ep((-a > a² exp(-330g + exp(-bur) buy) ow)
exp
b(w 4a)aa exp( 22ww) a² exeep((a² ( 3)bw) + exp((a²
(ab 1 aww
exp
= b(w + 4cr)a² exp( (hmr) > a² exp(((2 - 3)bbo) + exp((a² 1)bre)(a² aw).
(26)
Let
p(w) 1-1 b(- explaaaaaxxxp-022a0eee q(w) a²
(eljjw)+exp(pp texp((a²-11)xm)) +aw).
Note that
1 p((w) -a babbepp( (eexp(-2bbb 2bwr) 2b(w + 4a))
α 2bw + Sab.
Therefore p(w) decreases from (-oc, wo) and increases on [wo, +00), where we = (1 bcbb)/(b))
Let b > 1/(8.). Then we have that
min p(w) = p(0) = 4baab. wE(0,+50)
Note also that
= = exp((a² - 1)bw) -a2(a"33)6esx)) exbeep((2bb) + (a² 11)(aa² + aww) +a)
x <a² -3)bexp( (2bp- + (a² - 1)bar (a² 1)bo² +a
x a²(aa² - 1iibexp(-2bre + ((a² - 1)aw + (a² - 1)a² ) b b+a
Note, that a² (a² 3)bexpt e (bw) 0 always holds. Also, since (a² - D)a + (a² - 1)a² < 0
when w 2 0, we can choose sufficiently large so that q'(w) < 0 holds on (0, +00). Then
((w) = g(0) = a² + +
max =c)
Now fix b. Note 1 - 4 + and a² + a² + a² as a + O. We can always select small enough such that (0) p(0). Then have that p(w) > g(w) holds for all holds. Therefore, the set {|z] < a exp(-bw)) is forward invariant under F. Due to the exponential
and hence (26)
decay, we can always select a small enough and large enough such that {12] < a eexp(-bw)) C
{Q(2, w) > Q(F(z, w))}. where the latter set is given in Lemma 13, Therefore, in this exponential cone, Q decreases monotonically. Then using similar arguments to those Proposition 23 we have that all almost all initializations in this cone converge to the minimizer. This completes the proof. Li
In the following, we show that when the initialization is near the boundary, the orbit can visit any point in the space.
37

---

Published as a conference paper at ICLR 2026
Proposition 26. Assume 0 S < min 2.2 1 - |m[}- Consider arbitrary point my = (2, w) S2. When 14 2 0, som Cg (mo) (-2 v, 4 2(v + n)). When la O, N- G2 (mo) (2 - v, + 2(μ - v)).
Proof. We prove the case μ 2 0. Note that the case of < 0 can be proved via analogous procedures.
Let (z', w,) = F(2, w) and my = G3 for k 2 1. Consider the function E(z, w) = w w 2 n)).
We have
E(F(z, w)) E(z, w) =w' + 2(2 p)) - 2(2 + 1u)
(27)
[ww + 2(= + ()) + 2)(2 + v).
Note for (2,w) E Ω2 2(z + n) 2 O. Also, 2 0 > -v. Then for w > 2(z + pn), we have that
E(F(z, w)) - B(z, w) - ール
We have that m, E (Ao) for k 2 Hence, the 2-coordinate of m& is smaller than v for all
k 2 1. Since // < 1/2, v - 1< < -v. Hence, my E (E(F(2, w)) > E(z, w)} for all & 2 1. Note (m²))220 is a backward orbit. Thus, E(ms) monotonically decreases. Since lower bounded by on Ω2 [(ma)) converges to some finite value E", For contradiction, assume mg is unbounded. Note that, given arbitrary mi E cl(Ao). the set
()(,,))= S ( < M)
bounded for any M > O. Hence, we have the z-coordinate of my tends to negative infinity. Note that for sufficiently small negative 2 and (z, w) E S2 we have
w w + 4z(= (= + v) w=2² +-2+
4=(= + n1((1 -)
2² (-2+
42(: + +)((1 vv
-((2+1) 22 =² (-2 v)y
- (-2--v)) 2z(1 - v)
2 +2 101--10 v)z (2 - v) > 0,
which clearly holds as z tends oc. Therefore, my must lie in the region (w' > w) for all k: 2 K for some finite K > O. Note this implies that the w-coordinate of my starts to decrease from all k 2 K. This conflicts the fact that my is unbounded, as 9 n {w < C} is bounded for any C' > 0. Hence, m& is bounded.
According to (27,, my must converge to cloA)) ((w 2(z + p)) + - v)) - = cc((A) n [(w + 2(2 n1 - 0}.
Otherwise, assume my K for all ls and some compact set K. The function FFF w)) Fz(z, w) is continuous, so if my does not converge to its zero level set, E'(m)) - 1(m=-1) is bounded below and (mm2) can not converge.
Note, when restricting to {w = -2 + 11). the w-update under F is given by
w' k(w)= w w
- 42) w 2 0.
Solving <(w)= = w, we obtain that has two fixed points is straight forward to obtain that all backward orbits of x converges to w = 4 - 2(u + 1) except the one initialized at w 0. Meanwhile, note that = 4 2(μ + v) corresponds to (v-22,24 - v)). The eigenvalues of the Jacobian of F at this point are: 5 2/ - 2v, (-3 2.) Therefore, the backward orbits of F, i.e., forward orbits of Gg. are locally attracted by (v 2, 4 2(μ v)). Then,
0:
and w = 4 - 2( + v). It
using omega-limit set arguments similar to those in the proof of Proposition 16 we have that my converges to (-2 + v, 4- 2(v + p)). This completes the proof.
Using Proposition 26 we show that for the gradient descent system, the converged minimizer is unpredictable when the initialization is near the boundary.
38

---

Published as a conference paper at ICLR 2026
Proposition 27. Assume 0 S < m 1-1--p1 |m|)- Consider & = (-2 + 2, - 2(v + pt) and & (2 + 2(n- 12). Fori = 1,2, we have that UN F-N (Ex) has infinitely many points and FN-OF-N(E) ". When y for any open set O such that O n F
# 2, there exists (,,,w),(e (x") E O such that, for any (u', e'), (u", v²) that satisfy T(u', v') = (z', w') and T(u", x²")= = w" we have GD,
n (u', vt) converges to p* (u', et) and GD,, of (u" v") converges to p (u", v.). When y < the same result holds for any open set O such that O n NN (Ex)) # Ø.
Proof. We prove the case y 2. 0. Note that the case of y < 0 can be proved via analogous procedures.
We first show that = F- NN (E,) has infinitely many points and = N (G.) C (T(D)) Notice
that, & lies in the set {(z, w) E Q: w -2 + n) By Proposition 12
this set is invariant under
F, where the w-update under F is given by
w' = k(w) = w w
2 - 1-1
By analyzing the one-dimensional cubic map K, it is straight forward to obtain that, as N + co, for all w with w < 22 + v), '~ (w) + 0 and, for all w with w > 4 22 v)) x>V (w) - -0. Note that w converging to zero corresponds to zw-orbit converging to (-p,, 0) and tre-orbit converging to the (0, Therefore, we have & & 8T (D)) = T(aD"), where the equality was shown in the proof of Proposition 24 Note, Q(5i)
4v > 4v. Therefore, by Proposition 14, Therefore,
N F N (G) = (G,, o .... a (GI): Vh: 2 1,4, E (0, 1,2), Vi},
(28)
where Ge 's are homeomorphisms. By the construction of G, the cardinality of this set is infinity. Also, each G; is homeomorphism, any point in this set belongs to T(aD").
Next, we show that for any open set O such that O n U -N (EI) * R, there exists (z', w'), (2", w") O satisfying the claimed properties. When y S X, L has unique mini- mizer (0, 0) and the result holds according to Proposition 26 Now consider y > X. Let (ug, 0k) o2)a20 be a gradient descent orbit that converges to global minimizer, and (Za, wx) T(uk, UR). As we shown in the proof of Proposition 23
the converged minimizer is the minimal distance solution if
(( wo) # {j 2 1 1- 2, - v < 0}
1-2,
is an even number; and the converged minimizer is the maximal distance solution if the above is an odd number.
Let m. (-v, 2( v)). so that {m.] = T(M). Assume that O (G,, .... Gi, (E1)) for some fixed i.. .dg. By the continuity of G1, o .... G2y(Et) (1), there exists a neighborhood Ô 3 E such that Gj o G.2 (O) C O. By Proposition 26 there exists N' > 0 such that G (9) C Ó. Then we have that
(x',w') ) 2 Gi, o.... oi. o GoOa (m.) EO
and
(,","") a G. By construction, pk+N w') m, and o++N'+1(2" w") m.. Then it suffices to that one of
Gi G G2(mm)) E0.
Ç(2', w') and ((=", w²) is odd and the other even. To see this, notice that {w -2(2 + n)} is
forward-invariant. Then, as m² & {w -2(2 + n)}, the orbit starting from (z', w²) and from
(z", wn) can not visit {w = -22 ()). Therefore, by noticing that Go( C {2 {=<11 <=<1- 11-10 v}.
GI {w = 22(2 + ())) c {= [=<1-) 111-) -v.. G2(2 (W== -2(z +1)) C { > 1- - -v.. and
m, E (2 < 111--0 v}. we have that
1-v).
<(z', # (1sisk: ij = 2) <(>", wn) = # (1 S s k: in = +1.
This completes the proof.
E.2.2 PROOFS OF THEOREM 4 AND THEOREM 5
Proof of Theorem 4 By Proposition S, = T (T(S,)) and T(S,) is the basin of attraction of
the point (-11, 0) for system F. As shown in the proof of Proposition 23 T(S.) has measure zero.
39

---

Published as a conference paper at ICLR 2026
By Proposition 7 S, also has measure zero. The projected boundary T(D)) is self similar with degree three by Proposition 24 The unboundedness is given by Proposition [2
When y 2 O, consider the set H = T -o (N.FFF(N))) where &1 is defined in Proposition 27 By Proposition 27|and and since is surjective, H has infinitely many elements. By Proposition 9 T (T(DD)) DD" By Proposition 27 -N= F-N(E)) (5i) T(OD"). Together, have that H == (UR (-N))) C CD"
Consider any open neighborhood W c R2dd such that WNH o Ø. We will show that T(W) contains an open neighborhood O such that O n (UÑ=o PONNNN(66 (EI) + Ø. Notice the Jacobian of the map T drops rank and only if = u.. If W W [u= =e} = 2, then by constant rank theorem, 7 is locally projection, which gives the claim. If [u -t} o Ø, then, without loss of generality, assume We show that for any point (z', w') E Ω T
W = ((uo, uo), 6). Then T(ug. uo) = =(1uol that sufficiently close is surjective, there exists (u', r') such that T(u', v/) (='. w²), Note whenever (z', w') tends to
2||uol 2) there exists a preimage under T in W. Note,
22ll we have w' + 2z' = ||u' + + tends 4||uo 1 and w' 22' = ||u e' tends to O. Therefore, (u', 2²) tends to {u= v. ||]|| = -|luol]. Note, the map T is invariant under rotation. Therefore, with proper rotation, we can select (u',v') such that, as (z', w²) tends to (I]vol]², 2 2|luol?²).
it tends to {u- U, un) = (wo, un). Thus, such (w', v²) lies in W. This gives the claim.
Finally, by Proposition 27 there exist 0', 0" E W such that GD (0") converges to p=') and GD y (0") converges to p (0'). The case of < can be proved analogously using Proposition
This completes the proof.
Proof of Theorem 5 The results directly come from Proposition 9 Pnnd Proposition 23
F NON-EXISTENCE OF CONTINUOUS DYNAMICAL INVARIANT
Consider the scalar factorization problems:
1 (wv y)² + (u² + 1²),
min T((0) 0= (1.
(29)
where A 2 0 and u, v. E R. We show that there is no simple quantity that remains invariant during training.
dynamical invariant is map defined on the parameter space of the model whose values remain unchanged along optimization trajectories. Formally, for gradient descent applied to problem (2)..
map [u,, v): R + R with & 2 is a 5-approximate invariant if [ N (ū. ō)) - [(ū, T) S 8
holds for all N becomes a strict invariant. Invariants and approximate invariants have been used extensively to analyze the optimization dynamics of gradient flow and gradient descent non-convex optimization problems. Particularly, for problem 4 without regularization, the imbalance [U, V) = UUT VV is a well-known invariant of gradient flow (Du al. 2018) and an approximate invariant of gradient descent with small step sizes (Arora et al.) 2019;; Ye and Du 2021 Xu et al. 2023). In contrast, the following result shows that no simple invariants exist under large step sizes.
and initializations (ū, 5) E Rdd where is norm R When 5 = 0. T
Theorem 28 (Non-Existence of Simple Dynamical Invariants). Consider gradient descent with step size applied to problem (29) with 0 < X < min((1/n) - lvl. 1/m2)) If I(u, u): R² + I continuous 6-approximate invariant, then (up(a.)).(w) ('",e" [1, I(u', o')ll S 26. Consequently, the only continuous invariants are the constant functions.
Proof. We use the notation F, P, 2, w as stated in the conjugacy ô-approximate invariant. For any > O, there exist 0' 8" such that
Assume / is a continuous
sup [|T(0') - I(8") (v.
[1T(u, v) ((u','')1-
Without loss of generality, assume y 0. Now fix any point 0 T 2/n.4/n-11 2(y + A)).
Under the conjugacy (8), we have that T()) = (v-2) (v-2,4 2,4-2((- v))+ Then according to Proposition 26
for the regularized case and Proposition 21 for the unregularized case, in any neighborhood O of
40

---

Published as a conference paper at ICLR 2026
T(f), there exists ,' and N', NN' such that FN (5") T(0") and FN (5") T( Using similar arguments as those in Appendix E.2.2 and in Appendix E.1.2 we have that there exists T(6"). Next, we show that 0' and ? (0") = 0" To see this, notice that, for any (u, v), ts IR? (u, u) (s, t) if and only T(,, u) = (s, and, the two pairs, + and + t, and, and - f, have the same sign. Consider the change of coordinates p = (u + v)/v2 and q (u / 2 Let (us, 02)220 denote an orbit under GD,. By direct computation, we have that
0", H such that T N' (f") = T and TGD, N"
(0"))
@" can be chosen such that GD n
GD (0*) = and GD NM
pk = poll 2j v) Therefore, the sign of pv is fully determined by whether
"p = # (i E (0, (0, NN' - 1} 1 23}
even odd. Similarly, we have
qk = qo + 22 - -v),
and the sign of qN is fully determined whether
ny # (j E (0, No - 1) 1 v> 2j)
even or odd. Notice that we can take e as follows
g' Go G2 (T(O')),
where Go and G2 are defined as in Proposition 14 and my E {0, 1}. and m, is a sufficiently large number. Since the image Go lies out side {2 - w). increasing m, does not affect 7pp Also, since the image of G2 is contained in {z > v). one can always select mp from {0, 1) to make np even or odd as needed. Now fix my. Since the image of Gy is contained fa < v 1]. one can always select sufficiently large my to make ng even odd needed. Consequently, by appropriate choices of mp and my. the signs of pv and qN can be made arbitrary. This implies that, one can always select 8' such that GD, N (8') 0'. A similar statement holds for 0".
Since is 6-invariant, we have:
[1((0') - 1(0")) 11(00)))0000)0))))))))))))))11 1(0") 28 sup [4 llu,, v')l - 28. (wr")
Notice that 0', 0" can be arbitrarily close to 0. Since is continuous at 0 and is arbitrary, we have that
sup (w'))
[I((u,)) I(u','))l 28,
which completes the proof.
G GENERAL MATRIX FACTORIZATION
We present the extensions of the results in Section general matrix factorization.
In the following, we present the extension of Theorem to unregularized matrix factorization. Theorem 29 (Unregularized Matrix Factorization). Consider gradient descent with step size applied to problem (1) with and d dy. Let Y Diag(y). :Vd.) Consider the set
W (U, v) E 2d:ddy (u,, u²) (u' v²) = ( 0, Vi Vi j}
(30)
where u', of denote the ith column of matrices U, V. Assume the initialization (U,V) E W. The following holds:
Critical Step Size: Define the critical step size
1
8
()" V)
min min
[ys]
|| 11
v (1|
|| 11 16y, 16 (
v)
41

---

Published as a conference paper at ICLR 2026
For almost all initializations (under surface measure on W), the algorithu converges global minimum if < 77 (U.V... and does not converge to a global minimum if n > of (U, V). Therefore, when n) satisfies n||||l2 < L, the convergence region restricted to W, D n W, is equal almost everywhere (under surface measure on W) to the following set:
D' (0-((1))) (U..V E W: 1121 + 11021 + V (1lw 11) - 16y((w') Uc) < of
Vi
Sensitivity to Initialization: Fix step size y that satisfies 7|l 7||y[|2 < 1. Given arbitrary 0 a (here boundary is taken with respect to the subspace topology on W), E O and K2 > O, there exist 8' 8", ,0,00 9" E B(0,e) E) such that, as N tends to infinity, GD (6') converges to a global minimizer with norm larger than K1, GD (0") converges to a global minimizer with lIUUT - VV ||r > K2, and GD (0"H) converges to a stationary point, which is saddle point when min min(l > 0.
Trajectory Complexity: Assume n/ y||l1|22 < 1. The topological entropy of the gradient descent system GD, satisfies ,(DD,, 2 log 3. Moreover, GD,, has periodic orbits of any positive integer
period.
All of the above results follow directly from Theorem I and Proposition 31 dynamical system F; X + X, if S C. X is an invariant set, i.c., F() C S, then we have
We remark that, for
M(F)= 2 h( ((Fls). This gives the result for topological entropy.
We now present the extensions of Theorem 4 and Theorem 5 to regularized matrix factorization.
Theorem 30 (Regularized Matrix Factorization). Consider gradient descent with step size n) for problem Let Y Diag(y1...- .Jd,), ..,,), Assume that X S minj=1,... .yy (11nn)n)///)))) (wsl, 1/ 1/(2n)}- Let W be defined as in (30). Assume the initialization (U, (U,V)= E W. Consider the map TU, V) || 2 - 11) Let Sn denote the set of initializations (U, V) that converges to (0, 0). Let D", = Dy U. The following holds:
(('') v²,
Self-similarity: For any i E {1,.... ,dy... T. (O(D", n W)) is self-similar with degree three (here boundary is taken with respect the subspace topology W).
Unboundedness: When Y = 0, there exist constants a.b >> O such that almost all initializations (..V) E EW (under surface measure on W) with ((²) e+| < axxpp (b(p + (11)) ( for all
(1..... ,dy converge a global minimizer.
Sensitivity to Initialization: Let (u,, uf)ez denote the gradient descent trajectory of the pair (u', ot), with (ugr v8) = (w², D²). Let M; denote the set of global minimizers for the scalar problem L4(u, v) = (u v w- ); + 2/2 ([[ull2 + +1vll2)) Then M == Mi x .... x Ma, where W denotes the set of global minimizers for problem (4). We have that, for any (U, Dy nw and any i E (1, d as t tends to infinity, (ut, vt) converges either to
p (ug v6) = arg min (,,r)EM;
[1u,,)) - (u6: 11?
or to
pt pu v6) arg max
[1u,,) v) - (ub. 112
(u.+)EM,
Moreover. there exist infinitely many points on (D", n W) (here boundary is taken with respect the subspace topology on W) such that for any open set O containing such point, there exist i E {1.... dyy, (U', V*), (U", V") O such that, tends to infinity, (uh,, ,o,") converges to i.11 p (ul' v16 and (ut ,. 3. converges to pt tu v00 3.").
Stable Dynamics Under Small Step Size: Consider the function
Quu = ||ll + ||o12 + / (1ull + [|||13]22 16y(u - y).
Then the following holds for almost all initializations (Ü, V) W (under surface measure on W): If n < min;=1,... ,d, 8/(4X + Q(ūi, T)),, then gradient descent converges global minimizer: If 7 < min;=1,-. d, 4/(4X Q(a', U²)), then for all (ut, vf) converges p (ub, vt)-
42

---

Published as a conference paper at ICLR 2026
All of the above results follow directly from Theorem 4 Theorem and Proposition 31 We remark that, while the above Theorems are presented for initializations in W, chaotic phenomena are observed under generalization initializations. Experiments are provided in Appendix !
Proposition 31. Consider gradient descent with step size n, for problem ① with d 2 dy- Consider the set W {(U, V) 1,dd,, (u', u²) {u', vd) = (v', = O, vi o fj}. where w', v² denote the ith column of matrices U.V. The set W is ffrward-invariin
i.e. (W,, C W. Moreover, if the initialization (U, V) E W. then for i 1,.... the trajectory of the columns (u', v²) is identical the trajectory of gradient descent applied the scalar factorization problem Li(u, v) (u y1)² |
(1|u] || with step size yi and initialization (u², vT)).
Proof of Proposition 37 Recall the update:
U411 U² VV, U. YT) n2U,, V4+ V1 nUU(U, V. -Y) nAV..
For (U,, V,) E W, we have that
U111 = U₁ nV,, U. nVy -nAU
U. 1 Σ 0 (T))) .U. + - nWUU.
Therefore, for j = 1, ,yy
dy
w//11 = ur - Σ o (xt) (xA) ng .ndu
k=1
= w/ ()()
ut 1. mnclr
ug
v,) .uul.
Therefore, the one-step u² uppute aligns with that in scalar factorization problem. Similarly, we can show this holds for w p/-update. Now it suffices to verify that W is forward invariant. Assume (Un, Vi) E W. Notice that both w/_1 and u/41 are linear combinations of uf and vy. Then it clear that
(u/41 () (u/+1 vft = (ve+2> v/ =0
whenever j * This completes the proof.
The gradient descent update map GD, is non-invertible in general. Nevertheless, we show that the parameter space can be partitioned into small pieces, so that when restricted to each piece, GD,, has simple behavior.
Proof of Proposition 6 For notational simplicity, we let G = GD Under the assumptions, G is map with polynomial coordinates and det JG is polynomial. Then either det JG is the zero function or has a zeeo zero locus. To reject the first case, it suffices to have that 0. Note V²L(O) is fixed positive semi-definite matrix. Then if 1 is not the inverse of one of the eigenvalues of V ((). we have det (G * O. Therefore, for all 0 except for finitely many values, det JG has measure- zero zero locus. For such 17. G is non-constant map. By Jelonek (2002), there exists a semi-algebraic, measure-zero set C R. such TR \5 is a proper map. Let S' G( JG = 0)) denote the set of critical values of G. Then S' has measure zero by Sard's theorem and is also semi-algebraic. Since JG is non-zero almost everywhere, by Ponomarey 1987], K,, = G G G-'(("))) is measure-zero set. Since K.,, is semi-algebraie, IRP Kr, has finitely many connected components. Fix connected component C. For any compact set K C G(C), since K n.s Ø, (G( ( is compact. Meanwhile, since OC c (S) UG" and K n (S S') 2, (Glc)-I(K) (Gl (G))))) (K) n cl(C) is compact. Therefore, Glc is a proper map between connected manifolds that has full-rank Jacobian everywhere. Hence, Glc a smooth covering map (see, e.g.. Lee 2012). This completes the proof.
det JG(0)= det(I - L()))
that, Gl RR' C-((S)
43

---

Published as a conference paper at ICLR 2026
H EXPERIMENT DETAILS
For Figure Ieft pancl, we consider the problem L(,, e) = (u v - 1)² + 0.3( + IellB) with (u, v) E I2 t We randomly sampled two orthogonal unit vectors in 10 Viewing the two vectors as new axes, we evenly sampled 600 initial points in the range [-4, 2 We then ran gradient descent with step size 1 for 1000 iterations. The training stops if the loss is below Lunin + +10- where Lmin is the global minimum or if it is above 100. For Figure right panel. we consider L(u, v) (wv 1)2 with (u, v) E IR2. We evenly sampled 8002 initial points in the range [-4.5, 4.5)². We ran gradient descent with step size 0.2 for 6 iterations and recorded the final squared distances to the two minimizers. mi = (1, 1) and m2 = (2.9, 1/2.9). Viewing the final distances as functions of the initial point, we used the "contourf" function from the Matplotlib package (version 3.5.2) draw the sublevel sets of the distances. For the minimizer ma, we drew the sublevel set of [o, 0.15) to get the preimage of GD EED=6(BB (B(m1, vO.15)). For the minimizer 1/2, we drew the sublevel set of [0, 0.25) to get the preimage of GD (BB V..25)).
For Figure 2 we consider T(u, v) = (wv - 1)2 with (u, v) E RR2, For the left panel, we evenly sampled 800 initial points in the range [-4.5, 4.5 and ran gradient descent with step size 0.2 for 6 iterations. To visualize the basin for unstable minimizers, note, as shown in Corollary [7 converging to unstable minimizers can only occur within finitely many steps. We therefore recorded the final loss value and used the "contour" function from the Matplotlib package (version 3.5.2) to collect points in the level set of 0 for the loss. Those points correspond to convergence a global minimizer within 6 or less steps. We then filtered out and visualized points that converge to an unstable global minimizer, i.c., minimizer with squared norm larger than 2/n (see Corollary [7
In Figure 2 to visualize the basin for the saddle (0, note, as shown in the proof of Proposition 16 -sgn(y)v} for some neighborhood of (0, 0). Then
this basin can be given by - - (On we also recorded the final distance to the set {u = -0} and used the "contour" function from the Matplotlib package (version 3.5.2) to collect points in the level set of 0 for the distance. Then we filtered out the points that lie in D', (as defined in Theorem D]. This yield the basin associated with the saddle. To justify this procedure, note, as shown in Proposition 15 any point outside 'D' either converges to minimizer within finite steps or diverges. Also note, by the analysis in Lemma [2 points on (u = -gn(y)) } either converge to the saddle or diverge. For the right panel of Figure 2 we evenly sampled 800 points in [-0.9, -0.6] x [-4.55,-44 -4.25). We ran gradient descent with step size 0.2 for 250 iterations. The training stops if the loss value is below 10-8 or above 100.
For Figure 3 we consider L(u, v) = (we - 0.5)² /2 + 0.1 + x²) where (u, v) E R.2. For the left panel, we consider the dynamical system defined by F (see Proposition 3 with
1, = 0.2 and y = 1. In the zw-space, we evenly sampled 20002 initial points in 225, 3] x [0, 10) and filtered out those in (w 2 2]= We applied F200 to those sampled points and filtered out initial points that lead to loss value below Lmiu + 10-5 where Luuin is the global minimum of L. Those points come from the projected convergence region T(D"). Then we used the
function from
the SciPy package (version 1.9.1) to find the boundary of those points. The coloring of the boundary is based on the preimage structure of F. which is described in Proposition 14 For the middle panel, we evenly sampled 8 initial points in [-4,4] 4]2 and ran gradient descent for 100 iterations. For the right panel, we estimated the box-counting dimension for the boundary points found in the left panel. We first normalized these points to fit within [0. 1j2. We then computed the number of boxes N()) needed to cover all the points, with the box width & ranging from 1/2² to 1/25. We then performed linear regression on log NE) versus (og).
I ADDITIONAL EXPERIMENTS ON MATRIX FACTORIZATION
The folding behavior of GD in scalar factorization In Figure 6 we illustrate the folding behavior
of the map GD,, in L(u, v) = (uv - 1)2/2 with (u, v) E I²2 and 1/ = 0.2. The map GD,, is a 3-covering map from the light blue region C' in the left panel to the light orange region GC, in the middle panel. Notice that C C GD, and C contains the convergence boundary (black ellipsoid). The right panel shows that GD,, is transitive on the boundary: the trajectory of an initialization this boundary appears to wander along it in a seemingly random manner. For theoretical justifications of these behaviors, see Proposition 14 and Proposition 18
44

---

Published as a conference paper at ICLR 2026
Figure 6: Folding behavior of the map GD,, in L(u, (ur 1)2/2 with (u, v) R2 and 1j 0.2. In all panels, the black ellipsoid is the convergence boundary and the green hyperbola the set of global minimizers. The orange line in the middle panel is the set Crit, which consists of all critical values of GD, The blue lines in the left panel are the set GD,, (Crit), i.e., the preimage of critical values. The map GD2 is 3-covering map from the light blue region in the left panel to the light orange region in the middle panel. The right panel shows the training trajectory for initialization on the convergence boundary.
Comparison with Wang et al. (2022) Figure 7 visual comparison between our Theo- rem I and Wang et al. (2022, Theorem 3.1), for the objective L(u, v) (we y)2 /2 with (u, R?.
Recall that the convergence condition provided in the work of Wang et al. (2022) is:
4
ni (ū, 5) min
(31)
31yl
|
| + 41yl
The left penal of Figure 7 shows the case 7 1, 0.3. Here, the first condition in 31 is satisfied.
As shown, initializations satisfying the second condition < 4/(u² + 2² + 4/yl) form a strict
subset of the convergence region D in Theorem The right panel shows the case r = 1, = 0.9.
Note that this setting falls outside the analysis of Wang al. (2022). as n) > 1/33/yl). Meanwhile, initializations satisfying y < 4/(u² + 2 + 4/pl) form an even smaller subset of D
1 0.3
Theorem Global minimbers
Theorem 1 Global minimirers
(2022)
(2022)
-2
Theorem 3.1). Gradient descent with step size is applied L(u, v) (wv / with (u. v) R22 In both panels, the light blue region is the convergence region D', described in Theorem and the red region is the set of initializations satisfying 7) < 4/(2² + u² 4|pl), conditions required by Wang et al. (002).
Figure 7: Comparison between Theorem 1 and Wang et al. 2022
How convergence boundary and basin of saddle evolve with In Figure we illustrate how the convergence boundary and the basin of attraction of the saddle point evolve as the regularization parameter increases the scalar factorization problem. As shown in the figure, and consistent with our theoretical results, the convergence boundary is smooth (in the almost where sense) when À = 0. When À is just above zero, the boundary is close to a smooth and bounded set, with the fractal spikes thin that they barely visible. As À increases, the fractal structure becomes more pronounced, and the spikes gradually get wider. Also, the basin of attraction the saddle does not separate points inside the convergence region from points outside.
45

---

Published as a conference paper at ICLR 2026
= mm0.01
A
Am0.5
Figure 8: Gradient descent is applied to L(u, - (ue 0.8) [ + v2), where u. E R and À E {0, 0.01, 0.1, 0.5). Blue points represent initializations that converge to global minimizer; uncolored points represent initializations that do note converge. Red lines represent the basin of attraction of the saddle point (0,
Unregularized matrix factorization with general initialization In Figure descent for shallow matrix factorization without regularization under general initialization. We observe that, on a random slice of the parameter space, the convergence boundary is non-smooth, suggesting that a smooth convergence boundary is a special property of the invariant subspace W. This also implies that, globally, the critical step size might depend intricately on the initialization. However, sensitivity to initialization is common: on all the random slices, the converged minimizer is unpredictable near the convergence boundary. This suggests that chaotic dynamics always exists near the global convergence boundary.
gradient
Bansom Arbic
Rovdor
Rolo.)
ANRO!
Unregularized
Figure 9: Gradient descent is applied to LUU, V) V - where U. E R and Y is diagonal matrix whose diagonal elements are randomly sampled from [0, 1]. Four randomly sampled two-dimensional slices of the parameter space IR 40 are shown. The points are colored according to the squared Frobenius norm of the converged minimizer; uncolored points represent initializations that do not converge.
Regularized matrix factorization with general initialization In Figure TO descent for shallow matrix factorization with regularization under general initialization. We observe that in the random slices of the parameter space, the convergence boundary exhibits fractal-like geometry, although it appears to be less spiky than the boundary in scalar factorization. Also, as shown the figure, sensitivity initialization persist under general initialization. Together, Figure and Figure suggest that chaos and unpredictability are global properties of gradient descent in shallow matrix factorization.
gradient
Deep matrix factorization In Figure II we ran gradient descent in depth-three matrix factoriza- tion under generalization initialization. For deep matrix factorization we observe that already for the unregularized problem, the convergence boundary exhibits fractal-like structure, as fine-scale structures emerge. We report how the squared norm of the converged minimizer depends on the initialization, for two random slices the parameter space, As shown in the figure, while points near the origin converge to minimizers of small norm, sensitivity to initialization occurs in the vicinity of the boundary. For the regularized problem, we observe that not only the convergence boundary has a fractal-like structure, but the convergence region also becomes disconnected, with intricate connected components. The disconnectedness can be explained by the emergence of local minimizers, which attracts nearby trajectories, and non-strict saddles, which trap trajectories for long periods before they
46

---

Published as a conference paper at ICLR 2026
Par Randee Aodo
Sandom Adic
Random Addaa
Random Adtas
Random Adtta
Regularized
Gradient descent applied L(U,V) -Y 0.25 IVI))) where U, V E _5xx and Y = Diag(0.9, 0.8). Four randomly sampled two-dimensional slices of the parameter space _ are shown. The points are colored according the one coordinate value of the converged minimizer; uncolored points represent initializations that do not converge.
Figure 10:
escape. For detailed discussion the landscape geometry of regularized deep matrix factorization, see the work of Chen et al. 2025). Additionally, we observe sensitivity to initialization near the convergence boundary.
Squate nimss
Rundor ldlag
Fandon
Aandon
fancom
Unregularized
Regularized
Figure II: Gradient descent is applied to L(U, V) - -UV + +IV
[[W])? where U, V. W 2=2 and Y = Diag((..9 (09, 0.5). The left two panels show two randomly sampled two-dimensional slices of the parameter space R212 for the unregularized problem. Points colored according to the squared norm of the converged minimizer. The right two panels show the same random slices for the regularized problem. Points are colored according one coordinate of the converged minimizer. In all panels, uncolored points represent initializations that do not converge global minimizer.
J ADDITIONAL EXPERIMENTS ON REAL-WORLD DATA
Chaotic regime vs. stall-step-size regime In Figures 4 and 12 we considered a 2-class subset of CIFAR-10, each containing 25 randomly sampled images. We trained neural networks with two hidden layers, each with 100 neurons, for 5000 epochs. Whenever Polyak momentum is used, 0.9. For mean-squared error, the training stops if the training loss is below 10-6 or is above 10s; For cross-entropy, the training stops if the training loss is below 10-2 or is above 10a. The results for cross entropy are shown in Figure 12 Note when the pross-entropy is employed, the final sharpness is lower than the 2/1 curve, which aligns with the observations of Cohen et al. (2021). We also clearly observe two distinet step-size regimes, associated with EoS and Chaos, respectively, similar to what we had seen for the mean squared error in Figure
Fractal structure in parameter space In Figures 5 and 13 we considered a 2-class subset of CIFAR-10, each containing 25 randomly sampled images. We trained neural networks with two hidden layers, each with 25 neurons, for 3000 epochs under mean squared error. The weight decay parameter for Figures 5 and is set as 10-3 and 0, respectively. The step size is set as r) = 0.05. We randomly sampled two orthogonal unit vectors in the parameter space and kept them fixed as the coordinate axes for the random slice. On that slice, we evenly sampled 300 x 300 points from [32, 42] x [16, 24] (with respect to the random axes) and used them initializations. The training stops if the training loss below x 10-4 or above 10*. Notice that in Figure 13 the fractal structures qualitatively similar those in Figure
47

---

Published as a conference paper at ICLR 2026
Payak momeruum
meyaccmmmmntum 3
11.
aptimality
estan optimaltty
110
20n
000
642 cats 615
8:M
6.34
Figure 12: GD without or with momentum in training depth-3 ReLU network on subset of CIFAR-10 for 5000 iterations using the cross-entropy loss. The initialization is randomly sampled once and then kept fixed across all panels. At large step sizes, the final norm and final sharpness are highly sensitive to the step size.
Final loss
Final sharpness
Random Axi5,
Randomy AxiS3
253
203
Random Axis:
Random Axis:
Figure 13: GD without weight decay in training depth-3 ReLU network subset of CIFAR-10 for 3000 iterations. Shown is a random wwo-dimensional slice of the parameter space. Each point is parameter initialization, colored according to the final value of the loss (left) and sharpness (right). Gray points are initializations from which the algorithm diverges.
48


Reviews:
[{"review_id": "bFmAhb7s22", "reviewer": "Reviewer_Nc47", "summary": "This paper studies regularized matrix factorization problems with gradient descent (GD). The authors investigate what happens when GD is run with large step sizes, a regime often associated with improved generalization in deep learning, yet poorly understood theoretically. \n\nThey first start from a simplified matrix factorization, in which the target matrix is reduced to a scalar. In particcular, for the unregularized scalar matrix factorization the authors characterize the critical step size below which the GD converges almost surely, sensitivity to initialization, and topological entropy of the GD system. For the regularized case the GD training dynamics exhibit fractal convergence boundadry.\n\nThen they discuss the possibility to extend their results to general matrix factorization problems. The paper is concluded with some discussions on limitations and future directions.", "questions": "1. Although Section 4 covers the case when initialization matrices are rescaling of identities, there are still many cases not covered and the orthogonality conditions seems to be strong. Could the authors could provide some comments on the main challenges of extending the one-dimensional case to the high-dimensional case.\n\n2. Could the authors provide some discussions on whether or not the chaos or factal geometry can be observed in real-world machine learning problems?\n\n3. Any challenges of extending the GD dynamics to SGD dynamics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide a solid theoretical analysis for matrix factorization with gradient descent. The main results reveal that the training dynamics can exhibit chaos and fractal geometry when stepsizes vary.", "weaknesses": "1. The main results are limited to gradient descent on matrix factorization problems. In practice there is still a gap between this type of optimizatin problem and large-scale machine learning problems with stocahstic gradient methods.\n\n2. For the general matrix factorization problems, the authors only provide partial results -- the initialization of the matrices should satisfy a special orthogonality condition, under which the trajectories of the column vectors can be characterized using the scalar factorization case. This type of condition might be a bit strong and not covering all cases. This may also indicate the intrinsic difficulties of extending the scalar case to the high dimensional case.\n\n3. The experiments provided are limited to synthetic simulations, and there seems no detailed discussions between the theoretical results on chaos, fractal geometry and empirical observations from real-world problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKNq7QZ2xf", "reviewer": "Reviewer_t5Rh", "summary": "This paper analyzes gradient descent (GD) for both regularized and unregularized (shallow) matrix factorization, which simplifies to a scalar factorization problem under some orthogonal conditions. For the unregularized case, the authors prove the critical step size needed to converge to a global minimizer, as well as the conditions under which GD fails to converge to a minimizer. They state that this new result also tightens existing ones found in the literature.\nFurthermore, they show how GD is sensitive to initialization for this problem, and how at the critical step size, infinitesimal perturbations of the initialization can send the trajectory to a minimizer with arbitrarily large norm, sharpness, or imbalance, or to a saddle point. For the regularized problem, they show that the global dynamics of GD become even more unpredictable than for the unregularized problem. This unpredictability depends on the geometry of the boundary of the convergence region, which they show is self-similar and hence has a fractal structure.", "questions": "I have listed a few questions in the weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper presents new insights into the geometry of the convergence boundary for matrix factorization, which simplifies to a scalar factorization problem. It also presents a dichotomy in the geometry between regularized and unregularized scalar factorization: for the regularized problem, the convergence region (up to a measure-zero set) has an unbounded interior, whereas for the unregularized case, it coincides almost everywhere with a bounded domain. To my knowledge, this is novel.\n\nFurthermore, the authors present critical step size regions that determine convergence or failure to converge. I am not sure exactly how novel this part is, however (which I discuss in the following section). Nonetheless, I believe that the mathematical tools used to derive the results could be useful for the broader research community.", "weaknesses": "- I am somewhat unsure about the novelty of the step size bound for convergence in the unregularized case. Looking at the first term in Equation (3), I believe it comes from the inverse of the sharpness. If we then consider the results from Ghosh et al. (2025), specifically the case where $L=2$ and the target matrix is rank-1, this seems to reduce to the scalar factorization case. For this, they then prove that using a learning rate of $\\eta > 1 / |y|$ induces oscillations (and thus prevents convergence). Could the authors verify if this is the case? I do think that the second term in Equation (3) could be new, and it warrants more discussion than simply stating it depends on the convergence region. There should also be more discussion on what is done differently to improve the Wang et al. (2022) results, as mentioned in Line 255.\n\n- The paper's language is sometimes overly strong; for example, in line 257, the phrase \"precise description of chaos in gradient descent.\" This is a description of GD on a specific problem, not a general one.\n\n- I found the mathematical language to be overly complicated at times. For example, if I understand correctly, Proposition 5 seems to state that if we start from an orthogonal initialization, GD is invariant. This allows a simplification to the loss function in lines 162-163, which then reduces to the scalar factorization case. If so, this section could be simplified to improve readability. I also found lines 363-373 difficult to follow, though this may just be my interpretation.\n\n- Finally, a limitation is the simplicity of the studied model. While I do not expect derivations for the settings in lines 478-480, experimental results would be beneficial. For example, does adding regularization to deep matrix factorization alter the results of Ghosh et al. (2025), and do any of the results from this paper translate to that setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSWzkNzxQl", "reviewer": "Reviewer_V6Q9", "summary": "The authors examine the dynamics of gradient descent with step sizes at (or near) the critical value associated with a given initialization. In particular, they analyze the scalar factorization problem in both its regularized and unregularized forms. \nIn the unregularized case, they rigorously determine the critical step size for any given initialization and show that the behavior near the convergence boundary is chaotic, both in the norm of the minimizer, the imbalance between the two factorization vectors, and the nature of the final point (minimum or strict saddle). They also derive a lower bound for the topological entropy of gradient descent in this setting. \nIn the regularized case, they rigorously show that the convergence boundary is fractal, implying that convergence itself is unpredictable near the boundary, and provide an intuitive explanation of the mechanism, based on a partition of the parameter space. Here too, they demonstrate that the dynamics initialized near the boundary are chaotic, showing in particular that gradient descent converges either to the nearest or the farthest global minimizer, and that infinitesimal perturbations can flip this selection. \nFinally, they show that in the general matrix factorization problem, there exists a non-trivial set of initializations for which the problem decomposes into parallel sub-dynamics of scalar factorizations, allowing the extension of the previous results to this case.", "questions": "Beyond the numerical experiments for general initializations in the appendix, can you provide some results or intuition on whether the qualitative mechanisms and intuitions developed in this paper for a simple optimization landscape persist in more complex settings, where, for example, many local minima are present (either in theoretical settings or in experimental findings in realistic machine learning settings)?\n\nDo you have any insights on whether similar chaotic or fractal behaviors might appear for adaptive or momentum-based optimizers (e.g., Adam) near critical hyperparameters?\n\nIn Theorem 3, unboundedness is shown only in the case of $y = 0$, but it is then stated in the main text that, in general, the convergence region has an unbounded interior. Even though the caption of Figure 3 states that this behavior is qualitatively observed in general, I think it should be made clearer in the main text that this property, in the general $y \\neq 0$ setting, is only observed numerically.\n\nIn the figure captions, the parameter space is referred to as $(x, y)$ in the unbounded case. I think this causes confusion, since in the main text $(u, v)$ is used in both cases, and $y$ is used for the target of the factorization.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The setting studied in this work allows for a rigorous analysis of gradient descent when critical step sizes are used, with a thorough characterization of the chaotic nature of the dynamics and of the unpredictability of both convergence itself and the final point. \nAs stated by the authors, this can be connected with broader lines of research in realistic machine learning settings, such as the edge\nof stability, and to the best of my knowledge, this is the first rigorous work on the characterization of chaos for gradient descent.", "weaknesses": "The setting studied in this paper is very specific, namely the scalar version of the matrix factorization problem. As stated by the authors, this can be extended to the general matrix factorization only with orthogonal initializations, which correspond to the case where the problem actually decomposes into many parallel scalar ones. \nIt would strengthen the impact to clarify how these mechanisms extend in more general settings, or even how the intuition provided by this simple setting is useful when moving to more realistic scenarios where gradient descent with critical step sizes is used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zeq5VPFoJd", "reviewer": "Reviewer_F5Q8", "summary": "In this paper, the authors theoretically examine the dynamics of full-batch gradient descent (GD) with large step-sizes on a shallow matrix factorization problem\n\n$$ 1/2 \\Vert UV^T - Y \\Vert_F^2 $$\n\nIn contrast to previous work, they particularly focus on critical learning rates, i.e., learning rates for which GD converges just about and exhibits chaotic behavior. Their main results are as follows:\n\nFor the simplified setting of scalar factorization (the matrix $Y$ is a scalar $y$ which is decomposed into the inner product of two vectors $u,v$) and fixed learning rate $\\eta$, Theorem 1\n\n- characterizes the region of initializations $D_\\eta$ for which GD converges, \n- shows that around any point on the boundary of D one finds initializations such that GD converges to arbitrarily large/unbalanced solutions or to the saddle $(u,v) = (0,0)$, and\n- lower bounds the topological entropy of the dynamical system defined by GD characterizing it to be chaotic\n\nIn the setting of Theorem 1, but with additional $\\ell_2$-regularization of $u$ and $v$, and fixed learning rate $\\eta$, Theorem 3 shows that \n\n- the outer boundary of $D_\\eta$ has fractal structure,\n- for $y = 0$, the set of initializations converging to global optimality is unbounded (although the set of global minimizers is bounded), and\n- on the boundary of $D_\\eta$ the limit is highly sensitive to the choice of initialization.\n\nFinally, in Proposition 5 they show that Theorem 1 and 3 can be lifted to matrix valued $y$ by restricting to initializations in a particular slice of the high dimensional space.\n\nEmpirical simulations support their results.", "questions": "On the one hand, I think it is an interesting line of study to understand the connection of GD with large learning rates and chaotic systems. On the other hand, the results of the paper are strongly limited to scalar factorization and even in this case, most of the effects can only be observed on a (quasi)-null set of the successful initializations which is apparently confirmed by parts of the discussion (see points below). Furthermore, parts of the results raise doubts on whether topological entropy is the right measure to identify chaotic behavior (see points below). Consequently, I lean towards not accepting the submission. Due to time constraints, I could not check the proofs in detail.\n\nThe following points might be considered to improve the draft:\n\n- l. 092: „the map GD sends a region C containing multiple minimizers onto a larger region that contains C“ -> Is this a general statement or restricted to a specific objective function? It seems to me that this does not hold in general. Consider, e.g., a suitably scaled version of the objective function $f(x) = 1/4 x^4 - 1/2 x^2$ with derivative $f’(x) = x^3 - x = (x-1)(x+1)x$, and C = [-2,2]. Then the map GD should map C to a subset instead of a superset.\n\n- ll. 184/185: „…the supremum of the step sizes that allow convergence, where…“ -> shouldn’t it be „…supremum of the step sizes that allow convergence to global optimizers…“? At least this is the formal definition of $\\eta(U,V)$. Furthermore, I would suggest to formally define $\\mathcal{M}$ at least once in clear math notation.\n\n- ll. 238/239: The explicit form of $D_{\\eta}'$ seems to be incorrect for $y = 0$. In this case according to (3), convergence should hold for a.e. initialization and every $\\eta$ such that $D_{\\eta}'$ should be equal to the whole space a.e.\n\n- l. 247: The statement claims that for $y = 0$ and any learning rate (or $y \\neq 0$ and arbitrarily small learning rates) GD is a chaotic system. This appears counter-intuitive to me. Shouldn’t GD for extremely small learning rates be absolutely non-chaotic? Either there is a mistake in the statement or the use of topological entropy to measure chaos of GD seems inappropriate.\n\n- ll. 293-311: The discussion about instability of points with norm larger than $2/\\eta$ and the empirical observation that converging GD realizations always reach stationary points with norm less than $2/\\eta$ aligns with existing work on sharpness regularization due to large step-sizes (since in this toy setting sharpness and parameter norm coincide). It thus considerably weakens the relevance of the theoretical statement in Theorem 1 that one can reach limit points with arbitrary high norm/sharpness. \n\n- l. 298: „…the basin of the saddle…“ \n\n- l. 325: „…the presence of a basin…“\n\n- l. 340: The notation {z = y} is very imprecise\n\n- l. 358: In the definition of $p^-$ and $p^+$, the argmin/argmax is not always a singleton, or is it? If not, it would be cleaner to write $p^- \\in$ argmin …\n\n- l. 415: „…when the step size…“\n\n- l. 436: „…applied to the scalar factorization…“\n\n- l. 480: „..development of such a program.“", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Examining this extreme region of initialization/learning rate leads to beautiful connections between GD and fractals/chaotic systems", "weaknesses": "- The rigorous analysis is strongly restricted to shallow scalar factorization (Proposition 5 only extends the insights to a null set of the matrix factorization space)\n- The results are only relevant for a (quasi-)null set of those initializations for which GD converges", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)