Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures
Authors: Yedi Zhang
Andrew M Saxe
Peter E. Latham
Venue: ICLR 2026 Poster (2026)
Keywords: simplicity bias
learning dynamics
gradient flow
Primary area: learning theory
TL;DR: We present a theoretical framework that explains a simplicity bias arising from saddle-to-saddle learning dynamics for a general class of neural networks, incorporating fully-connected, convolutional, and attention-based architectures.

Abstract: Neural networks trained with gradient descent often learn solutions of increasing complexity over time, a phenomenon known as simplicity bias. Despite being widely observed across architectures, existing theoretical treatments lack a unifying framework. We present a theoretical framework that explains a simplicity bias arising from saddle-to-saddle learning dynamics for a general class of neural networks, incorporating fully-connected, convolutional, and attention-based architectures. Here, simple means expressible with few hidden units, i.e., hidden neurons, convolutional kernels, or attention heads. Specifically, we show that linear networks learn solutions of increasing rank, ReLU networks learn solutions with an increasing number of kinks, convolutional networks learn solutions with an increasing number of convolutional kernels, and self-attention models learn solutions with an increasing number of attention heads. By analyzing fixed points, invariant manifolds, and dynamics of gradient descent learning, we show that saddle-to-saddle dynamics operates by iteratively evolving near an invariant manifold, approaching a saddle, and switching to another invariant manifold. Our analysis also disentangles data-induced and initialization-induced saddle-to-saddle dynamics. In particular, the former leads to low-rank weights while the latter to sparse weights. Equipped with the theory, we predict the effects of data distribution and weight initialization on the duration and number of plateaus in learning. Overall, our theory offers a framework for understanding when and why gradient descent progressively learns increasingly complex solutions.

Published as a conference paper at ICLR 2026
SADDLE-TO-SADDLE DYNAMICS EXPLAINS A SIMPLICITY BIAS ACROSS NEURAL NETWORK ARCHITECTURES
Yedi Zhang Andrew Saxe Peter E. Latham"
!Gatsby Computational Neuroscience Unit, University College London
PSainsbury Wellcome Centre, University College London
"Correspondence: yedi@gatsby..cl.ac.uk
ABSTRACT
Neural networks trained with gradient descent often learn solutions of increasing complexity over time, a phenomenon known as simplicity bias. Despite being widely observed across architectures, existing theoretical treatments lack uni- fying framework. We present a theoretical framework that explains a simplicity bias arising from saddle-to-saddle learning dynamics for a general class of neu- ral networks, incorporating fully-connected, convolutional, and attention-based based architectures. Here, simple means expressible with few hidden units, i.e., hid- den neurons, convolutional kernels, or attention heads. Specifically, we show that linear networks learn solutions of increasing rank, ReL.U networks learn solu- tions with an increasing number of kinks, convolutional networks learn solutions with an increasing number of convolutional kernels, and self-attention models learn solutions with an increasing number of attention heads. By analyzing fixed points, invariant manifolds, and dynamics of gradient descent learning, we show that Modlee-t--sadde dynamics operates by iteratively evolving near invari- ant manifold, approaching a saddle, and switching to another invariant manifold. Our analysis also disentangles data-induced and initialization-induee saddle-to- saddle dynamics. In particular, the former leads to low-rank weights while the latter to sparse weights. Equipped with the theory, we predict the effects of data distribution and weight initialization on the duration and number of plateaus in learning. Overall, our theory offers framework for understanding when and why gradient descent progressively learns increasingly complex solutions.
INTRODUCTION
Deep neural networks trained with gradient descent often learn functions of increasing complexity over the course of training (Arpit et al., 2017; Kalimeris et al., 2019; Rahaman et al., 2019; Saxe et al.. 2019; Refinetti et al., 2023; Bhattamishra et al., 2023; Abbe et al., 2023). This dynamical simplicity bias has been observed across architectures (Shah et al., 2020; Teney et al., 2022; Edel- man et al., 2024), tasks (Karkada et al., 2025; Wurgaft et al., 2025; Wang & Pehlevan, 2025), and training paradigms ranging from supervised (Rahaman et al., 2019) to reinforcement (Schaul et al., 2019) and self-supervised learning (Simon et al., 2023). A particularly striking manifestation is siage-like dynamics: extended plateaus in loss alternating with bursts of rapid improvement as net- works progress through increasingly complex input-output maps (Saxe al., 2014; 2019). These dynamics, known as "saddle-to-saddle"" dynamics because they can result from trajectories passing near a sequence of saddle points (Jacot et al., 2022; Berthier, 2023; Pesme & Flammarion, 2023), have been documented in deep linear networks (Saxe et al., 2014; 2019; Gissin et al., 2020; Ja- cot et al., 2022), two-layer and deep ReLU networks (Maennel et al., 2018; Boursier et al., 2022; Chistikov et al., 2023; Wang & Ma, 2023; Kumar & Haupt, 2024; Zhang et al., 2025a: Wu et al., 2025: Bantzis et al., 2026), and self-attention models (Boix-Adsera et al., 2023: Rende et al., 2024; Geshkovski et al., 2024; Zhang et al., 2025b; Varre et al., 2025), and have been hypothesized to be universal (Ziyin et al., 2025; Kunin et al., 2025). Yet the same architectures can also exhibit smooth, exponential training dynamics, simply by changing the initialization (Jacot et al., 2018: 88;; Tu al.,

---

Published as a conference paper at ICLR 2026
Loss landscape cartoon
Linear fullly-cormect net
ReLU fully-commett
Linear self-attention
Gradient descent trajectory
manifolds
effective (phaserD88)
2000
2000
1000
2000
Weights
menifold: (8)
Linear convolutonal net
E ReLU convolutional net
Quadratic net
1.0
1.0
1.0
Loss curve example
0.5
0.5
Traning loss
0.0
0.5
1000
2000
1000
2000
1000
2000
04
94-
0.8
2000
Epoch
04 04
-24 04
.0.
088 0
69
Figure/ Saddle-to-sadde dynamics occurs in the gradient descent training of a wide range of architectures and leads to dynamical simplicity bias. (A) Saddle-to-saddl dynamics on a cartoon loss landscape. The cyan and yellow curves represent invariant manifolds, on which the network implements input-output maps expressible by the architecture with one and two units, respectively. In general, vaddle-to-sadde dynamics operates by repeating: i) during the plateau, escaping from a saddle associated with a width-h network onto invariant manifold with effective width (h 1); ii) during the rapid transition phase, approaching a fixed point on that manifold, which is saddle associated with width-(h 1) network. This figure shows repeats of this process. (B-G) Loss and weight dynamics for various architectures. Each panel shows the loss during training (top), and the first-layer weights during the intermediate plateau (bottom left, phase 3 in panel A) and the end of learning (bottom right). The first-layer weights to each hidden unit are two-dimensional and plotted as black dots. During the intermediate plateau, all networks visit saddle, at which the input- output map of the network can be expressed by the architecture with only one unit. The network then converges to a stable fixed point, at which the input-output map expressible with two units. The weight structures in BC, DE, and FG correspond to three categories of weight configurations of fixed points in Thcorem 1; see Section for details. A video version of this figure is provided at URL. Dynamics with other two-layer architectures and deep networks are provided in Figures to 5. Experimental details are provided in Appendix I.
2024; Kunin et al., 2024); and more broadly, the emergence of stage-like dynamics can hinge on the data distribution (Yoshida & Okada, 2019; Goldt al., 2020) and architectural choices (Orhan & Pitkow, 2018).
These diverse findings raise foundational questions about the nature of dynamical simplicity bias in deep neural networks. Is there a universal mechanism driving stage-like dynamics, or a collection of architecture-speeffic mechanisms? Is there a principled link between stages and simplicity, such that earlier stages in learning are simpler? And if simplicity does underlie these dynamics, what is the operative notion of simplicity, and how does it reflect an architecture's 's inductive bias?
Here we answer these questions. We show that for range of architectures, including linear net- works, ReLU networks, convolutional networks, quadratic networks, and linear self-attention (Fig- ure IB-G), there is universal mechanism, saddle-to-sadaa dynamics, driving stage-like learning. The operative notion of simplicity is the number of effective units the architecture, i.e., hidden neurons, convolutional kernels, attention heads. In particular, first we show that fixed points in the loss landscape are recursively embedded: fixed points of smaller networks are embedded in saddle points of larger networks, yielding nested hierarchy of saddles. Second, we show that saddle points are connected by invariant manifolds along which a larger network behaves like a smaller one, preserving simplicity along the connecting trajectories. Third, the link between saddle-
2

---

Published as a conference paper at ICLR 2026
to-saddle dynamics and simplicity arises from the interplay of the saddle hierarchy and timescale separation. Specifically, timescale separation steers dynamics toward invariant manifolds associated with simple input-output maps, thereby controlling the complexity increment at each stage. We also disentangle data-induced and intiiaizzaton-induee
timescale separation, showing that the former leads to low-rank weights (Figure IB,C) while the latter leads to sparse weights (Figure 1F,G). To- gether, this theory paints a unified picture of embedded saddles, invariant manifolds, and dynamics which give rise simplicity bias across architectures, and predicts when instead non-stage-like behavior will arise.
Related work. We are inspired by line of pioneering research that began with the seminal work of Fukumizu & Amari (2000) and continued in subsequent studies (Inoue et al., 2003; Amari et al., 2006; Wei et al., 2008; Amari et al., 2011; Fukumizu et al., 2019; Simsek et al., 2021; Zhang al., 2021). In particular, Fukumizu & Amari (2000) first discovered a hierarchy of fixed points in two- layer fully-connectoo
nonlinear neural networks. While their fixed points could, in principle, be ex- tended to convolutional and attention- -aseed architectures, they did not explore this, as convolutional architectures had not been popularized and attention-base architectures had not been invented. We study the fixed points across fully-connectedd convolutional, and attention- baaed architectures. Fur- ther, we go beyond fixed points to study invariant manifolds and andle--o-sadee dynamics, with implications for simplicity bias. more detailed literature review is provided in Appendix A.
2 NETWORK SETUP
Let f(x) represent a neural network with input a E R.P. We focus on one layer in the network with H units and trainable parameters 01.HI
f(z: 01.H) = Fout
Σ
(gnn (x:); u,);; i=1
where 0 =
0; u1
Here (omr()) and (.) represent the processing after and before this layer, which are usually deeper and shallower layers of the network. The weights u, E R.N. U. R.V, and thus 0g E N.N.NNN on the right because fully-connected layer) or matrix (as in a self-attention layer). The network output f(x; 01.H) can be a scalar or vector. We will specify their dimensionality when we make them concrete.
We place the second-layer weight
u;) may be a scalar (as in a
The definition of layer in Equation (1) incorporates major architectures. For fully-connected layer, unit is a hidden neuron: 6(2; w. b) = aww x + b) where a--) is the activation func- tion and w. b are the weight and bias. For a convolutional layer, unit is a convolutional kernel: 2) where denotes convolution. For a self-attention layer, unit is an attention head: of Z; K, Q) T@smax ZQKT Z Z where smax(.) denotes row-wise wise softmax and K, K are the key and query weights. A self-attention layer fits into our definition as follows,
Φ(2; u) = o
ATTN(Z) smax ZQK T z Zzz = I @ ZZQK Z (V) Φ K, Q)u. (2) We note that this is not common notation for self-attention; present it solely to show that Equation (1) incorporates self-attention. Hence, statements we will make about Equation (1) apply to fully-connected, convolutional, and self-attention architectures.
Let pz : Up 1p be a supervised learning training set. The training loss is averaged over the training set L = } Σp- Eyyn. (x,,),, where the loss function is second order differentiable with respect to (x), including common choices like squared error loss. The parameters are trained with gradient flow the training loss,
0c
OC
0f(x)
(3)
00
0f())
00
Gradient flow captures the behavior of gradient descent in the limit of small learning rate.
Definition 1. A point 0* is fixed point of the gradient flow dynamics in Equation (3) if 0C
30 g-
0.
3 Loss LANDSCAPE: EMBEDDED FIXED POINTS
In this section, we establish that saddles generally exist in networks described by Equation (1). We show that fixed point of narrow network gives rise set of fixed points in a wider network.
3

---

Published as a conference paper at ICLR 2026
These fixed points constructed by embedding the narrow network into the wider network, as formalized in Theorem 1.
Theorem (Embedded fixed points). If a network defined by Equation (1) with (H - 1) units has fixed point 0 i:(H-1) yielding an input-outpat mup (x)), then there exists Ov.H E S such that network with H units implements the same map (x) and 01.H a fixed point.
We construct F1:H by setting the first (H - units to 0 i(HH-1) and modifying them as follows.
(i) For any &, the set S includes
uh ui UH = Vn U; =1 (w)o No ER, -
(4)
(i) If Uezo such that vz, (=;; Ugero) = O, the set S includes
u Uzero: UH
0.
(5)
(mii If (z; is degree-I homogeneous in that is Va E FF 60 ou) (z; u), where F for general homogeneous functions, and = R2 for positively homogeneous functions, e.g., the ReLU activation iottaatiionuntioonnn the set includes
u Yuu vs =o0; -1- (u_7,%)),
e FiiE {1. , 1}. (6) No R.R.
(iv) If p(z; u) is linear in that is degree-I homogeneous, Va E R, p( anp( 2; u), and additive, 6(2; u.) + p(2; = Φ( z; u, + uz), the set S includes
H-1
#-1
UH =
Tu, u UH =
Σ
Yo Vi
Your Yap E R,
1=1
U.
Yuu
Yo Vj
i= 1...... 1.
The proof of Theorem which is provided in the Appendix E,, consists of two steps. First, verify that for the weight configurations given above, the width-H network implements the same input-output map as the width-(IH - 1) network. Second, show that gradients of the weights in the width-ff network are either equal or proportional to those in the width-(H1 1) network, which are zero.
Remark 1. Equation (4) is valid for any activation function Φi while the rest valid for with specific properties, implying that certain properties of o give rise to a larger set of embedded fixed points weight space. Equations (4) and (5) were first discovered by Fukumizu & Amari (2000). We extend these two constructions with Equations (6) and (7). This extension is crucial for studying learning dynamics, as the saddles visited during learning turn out fall under Equations (5) (7) but not Equation (4).
By induction, obtain Corollary 2 by repeatedly applying Theorem I to embed multiple units in one layer and embed units in multiple layers of deep network, with each layer defined by Equation (1).
Corollary 2. If Lepth-L network with hu units in layer 1 = -. 1....... L) has a fixed point yielding an input-output map (xx),, then for a depth-L network with Hi 2 bu units in each layer, there exist weight configurations such that the network implements the same map (xx and the weight configurations are fixed points.
Theorem and Corollary 2 indicate that the global minima of a narrow network, even they incur nonzero training loss, remain fixed points of the gradient flow dynamics in any wider network with the same architecture. For example, the global minimum of width-1 network typically lacks the ex- pressivity to fit the training set and thus incurs nonzero loss. In wide network capable of achieving zero loss, the fixed points corresponding to the width-1 network global minimum are either saddles local minima. They guaranteed to be saddles in deep linear networks with rank-r (r 2 1) target maps (Baldi & Hornik, 1989; Kawaguchi, 2016) and, under mild conditions, are saddles in general architectures (Fukumizu & Amari, 2000; Fukumizu et al., 2019).
In Figure 1, we show six cases where the network first visits a saddle, corresponding to a solution expressible by the architecture with single unit. The network then converges stable fixed point,
4

---

Published as a conference paper at ICLR 2026
corresponding to a solution expressible with two units. The fixed points visited during learning fit into three different categories in Theorem 1. In panels (B,C), the fixed points visited during learning described by Equation (7), corresponding to rank-one and rank-two weights. In panels (D.E), the fixed points are described by Equation (6), corresponding to one and two rays of proportional weights. In panels (F,F), the fixed points are described by Equation (5), corresponding one two units with large weights with the rest being near zero.
INVARIANT MANIFOLD: EFFECTIVELY NARROW NETWORKS
An invariant manifold of dynamical system is manifold such that any point starting on remains on the manifold under the system's 's evolution. In Theorem 3, show that for gradient flow dy- namics of the class of neural networks we consider, invariant manifolds always exist. Further, these invariant manifolds correspond weight configurations that make the network effectively narrower than its actual width.
Theorem (Invariant manifolds). Let T be any time such that one of the following conditions (i)- (iv) holds in a network defined by Equation (1). Then, in each case, the stated relationship between the weights is preserved for all 2 T under gradient flow dynamics:
(i) For any &. two units have equal weights: 0. = 0--
(ū) If Uzero such that Vz, Φ(2; Unero) = 0, a unit has zero weights: v; = 0, u; = Uzero-
(ii)) If o(z( u) is homogeneous in u. two units have proportional weights: 0, -0,, 1 E F.
(iv) If ((2;; u.) is linear in u, any number of units have lineur dependence: O, Σjtt
The precise definitions of homogeneity and linearity are given in Theorem
The proof of Theorem 3 is provided in the Appendix and is relatively straightforward. For ex- ample, when 0, 0y. the gradients of O, and 0, are equal and thus they stay equal for all future time. The invariant manifolds are larger in weight space when ø has zero, homogeneity or linearity properties, similar to the enlarged set of embedded fixed points in Theorem 1.
When the weights of network lie on an invariant manifold, its onput- map is expressible with fewer units than its actual width: simply remove the i-th unit and appropriately modify the remaining weights (see Appendix F3). Further, we can have more than one constraints; e.g., 0, = 02 and 0g O₂- Each added constraint reduces the effective width by 1. Hence, when weights evolve on an invariant manifold, the simplicity of the network's 's input-output map is constrained by the effective width associated with the invariant manifold, rather than the actual width.
The invariant manifolds indicate that there exist gradient flow paths connecting pairs of embedded fixed points defined in Theorem 1 (see Appendix F.4). Following such path corresponds to an dynamics. To see this, starting from an embedded fixed point with
iteration of saddle-to-sadee effective width h, we may apply a carefully chosen small perturbation that moves the weights onto the invariant manifold with effective width (h + 1). This perturbation corresponds to breaking exactly one constraint. By Theorem 3, the dynamics then remains on the invariant manifold for all time, eventually converging to a fixed point on it, that is, embedded fixed point with effective width (h + 1). This process is one saddle-to-saddle transition: from the saddle with effective width h to the saddle with (h 1). We illustrate this process in Figure IA. In the next section, we develop heuristic arguments showing that the gradient flow dynamics can, in some cases, naturally evolve near such saddle-to-saddle paths the invariant manifolds.
5 SADDLE-TO BADDLE-TO-SADDLE DYNAMICS
The embedded fixed points (Section 3) and invariant manifolds (Section 4) hold for general architec- tures defined by Equation (1). To analyze learning dynamics, however, we must work with concrete architectures. We focus two-layer networks where ((æ; u) is homogeneous polynomial in the weights u. studying the linear and quadratic cases in detail. The linear case includes fully-connected linear networks and convolutional linear networks. The quadratic case includes quadratic networks (defined by Equation (71)) and linear attentttentinn Both types of architectures exhibit saddle-to- saddle dynamies, but their mechanisms differ. We show that the mechanism in the linear case is
5

---

Published as a conference paper at ICLR 2026
timescale separation between directions across all units due to the distribution of the data, while the the mechanism in the quadratic case is a timescale separation between units due to initialization.
5.1 LINEAR CASE: TIMESCALE SEPARATION BETWEEN DIRECTIONS
Consider two-layer network in which c<æ;; u) is linear in the weights u,
f(x; O1.H)
Σ v.ua xxx) Wz, where v E ER, u, R.
(8)
Here =(x) denotes any function of the input æ, as (ax; u) is linear in u but not necessarily linear in a.. The gradient flow dynamics of Equation (8) trained on squared loss is
0; = Eyz - WE Ui,
úg = (E WE.;)
Vi,
i 1=1... H.
where the data statistics are Σyz 1 Σ P
P 2x=1 y2 E. E 2x2 When the weights (cc). (.) O(e),i 1,.... H. the first terms in
are initialized to be small, i.c., 2 (0)
Equation (9) dominate: Σy: - WEgs = Eye + -E The weights thus approximately evolve as a linear dynamical system (Equation (10)), which we analyze in Theorem 4.
Theorem (Timescale separation between directions). Consider the linear dynamical system
ú; = Eyz éí = = Us 1. ,..
(10)
Let the singular value decomposition of Eye be given by Eyz = SkquTk: D -inniN Nu) with singular values 81 2 ... 2 Sp. and let the largest singular value 81 have multiplicity S r < DD. Let the initial weights be sampled independently from a Gaussian distribution (00,2 2) with small c. When the projection of the weights on the span of the top r singular vectors reaches (1), that is
PP.1| = 0(1), where P =
qk
r O,
=
(11)
Tk
Uu
the projection the remaining subspace || P)O.11 ((tt-er+s/e)) almost surely.
We provide the proof in Appendix G.2 and the intuition here. The second and first-layer weights V; u4 grow exponentially along the singular vectors Qs, respectively, the rate gL Relative to the dominant growth rate "1 along the top singular vectors, the components along other singular vectors decay as c(*x=11) k r 1,... D. Consequently, during the early phase, the weights become increasingly aligned with the top singular vectors and thus approximately rank-r. Taking r as an example, the weights become approximately rank-one; specifically, v, aligns with qi, and u, aligns with r for every i.
Theorem 3 implics that rank-r weights constrain a linear network to an invariant manifold corre- sponding effective width Since the carly phase dynamics drives the weights to be approxi- mately rank-r, the network evolves near the invariant manifold and approaches a fixed point on it. This is the first iteration of saddle-to-saal to-dddle dynamics. In weight space, the weights move from the initial saddle at zero to the second saddle. In function space, the network learns a more complex solution, changing from a constant zero function to rank-r projection of the target linear map.
Subsequent iterations of saddle-to-saddle dynamics operate similarly. The dynamics near a rank-r saddle, corresponding plateau in the loss, is again approximately linear dynamical system
b;
Eyz Ui; i, = Ey Us, 1= 1,.......
(12)
where Σyz is Eyz projected onto rank- - subspace; see Appendix G.3. Via the same rea- soning as Theorem 4, the weights grow the fastest along the top singular vectors of Σ Low-rank weight growth will again place a linear network near an invariant manifold with few more effective units, guiding the dynamics toward a fix point on that manifold.
To summarize, in the linear case, distinct singular values of the input-output correlation matrix induce a timescale separation between weight growth along different directions. If all singular values distinct, the timescale separation leads to approximately rank-one weight growth during loss plateau, causing the escape path from a saddle to closely follow an invariant manifold with one more effective unit.

---

Published as a conference paper at ICLR 2026
5.2 QUADRATIC CASE: TIMESCALE SEPARATION BETWEEN UNITS
We now consider a two-layer network in which o(;; u) is quadratic in the weights u.
H 01:H) = Σ i Z(z)u, where Vs E R. U, E RD. P..ZE E RDD.
(13)
Here Z(x) denotes any function of the input æ. For example, linear self-attention fits into Equa- tion (13) with (xx) being cubic function of the input æ, and <ææ; u. a quadratic function of the key and query weights u = [vec( vec)())))))) has saddle-to-saddle dynamics and involves non-closed-fomm solutions. The gradient flow dynamics of Equation (13) trained on squared loss is given by Equation (44). Near small initialization, the quadratic terms in Equation (44) dominate. In Proposition 5, we analyze the approximate dynamics and show that one unit with the largest initialization grows much faster than the rest.
vec( We consider the scalar output case because already
Proposition 5 (Timescale separation between units). Consider the dynamical system
è uu Eyzui U₁ 20, Eyzu, ==
(14)
Assume Enz is symmetric and has both positive and negative eigenvalues. Let the initial weights be sampled independently from Gaussian distribution N(0, 22) with small e. When weights in one of the units reaches O(1), the rest of the units is (e)) almost surely.
We provide derivations in Appendix H.2 and the intuition here. The intuition that the quadratic dynamics in Equation (14) is tich-get-richh process. We can get flavor of such dynamics by considering the simplest quadratic dynamics, Úū = u?, which has the solution
(1)
1)
i= 1.,,.,, H.
(15)
(0)
By solving for with i and j. we can write (t()) in terms of (t))) as
22 0,
(t) (0
1 [t(()]
(16)
(0)
Assuming initial conditions of order O(c), for example (0) NO0, 2²), and letting by be the unit with the largest initial value, we see that when (t) ~ O(1). the other units are still small: (1) ~ (()) for i * j. Thus, under quadratic dynamics úū = =u? distinct initial conditions of the units induce timescale separation in their growth. Although the general case, analyzed in Appendix H.2, is more complicated, the timescale separation between units essentially comes from the same mechanism.
In Theorem 3(i), we showed that if p(x; = O' then nonzero weights in one unit and zero weights in the rest of the units constrain network to an invariant manifold with effective width one. Since the early dynamics drives one unit to grow much faster than the rest, the network evolves near the invariant manifold with effective width one and approaches fixed point it. This is the first dynamics. Subsequent iterations operate similarly. Starting near the
iteration of saddlc-to-sadde first saddle, one unit has nonzero weights and (H 1) units still have small weights. The dynamics near the first saddle drives one of the (H - 1) units to grow much faster than the rest. Hence, the escape path from the first saddle again approximately follow the invariant manifold with two effective units, steering the dynamics toward fixed point on that manifold. This process repeats.
For o(z; u) that is quadratic in u and has c(æ; 0) Vz, the distinct initial weights in each unit induce a timescale separation between the weight growth in different units. One unit grows much faster than the rest, causing the escape path from saddle to closely follow an invariant manifold with one more effective unit.
Higher-order polynomial activation. If ((;; u) is a homogeneous polynomial of degree p > 2 in the weights u, we conjecture that there is still a timescale separation between units, possibly even stronger than the quadratic case. Our intuition is that the dynamics near zero has a similar flavor to the scalar dynamics, é; = uf- By similar reasoning to Proposition 5, the unit with the largest initialization grows much faster than the rest, causing a timescale separation between units. The dynamics in the cubic (p- 3) case is consistent with intuition, shown in Figure 4G.
7

---

Published as a conference paper at ICLR 2026
Loss curves of different network width H
B
Loss curves of power law data with different exponent
linear fully-connected net
linear self-attention
linear fully-connected net
linear self-attention
1.0
H 800
1,0
1:25
1.0
1.25
400
K= 1.0
== 1.0
H 200
km 0.76
K=0.7 0.75
0.5
100
0.5
0.5
0.5
0.5
H 50
0.26
+ 0.25
H 25
0.0
0.0
0.0
0.0
0.0
2000
4000
0
10000
20000
0
2000
4000
5000
10000
Loss of different initialization structure
Loss curves of different initialization scale
1.0
small lsotroplc
1.0
10-
lerge
large 2
10
0.5
berge rank-3
0.5
10-2
0.0
0.0
2000
4000
1000 2000
3000
4030
Figure 2: The effect of network width, data distribution, and initialization on learning dynamics. Singular values of Eyz (linear network) or positive singular values of Σn (linear self-attention) follow a power law, 8, = n ,,=,, n = 1, 3, and are normalized such that Σ Sn = 1. (A) In- creasing the number of units И has little effect the loss curves of linear networks, but shortens the plateaus in linear self-attention. Ι = for both models. (B) Decreasing the power law exponent shortens the plateaus in both linear networks and linear self-attention. Setting = 0 eliminates plateaus in linear networks but does not eliminate plateaus in linear self-attention. H is 100 for linear networks and 25 for linear self-attention. (C) Linear networks with small isotropic initializa- tion or large low-rank initialization exhibit saddle-to-sadde dynamics. The loss landscape cartoon illustrates large rank-r weights places linear network near an invariant manifold with effective units and thus approaches saddles during learning. (D) Increasing the scale of isotropic random initialization shortens the plateaus. x 1 for panels C,D.
General nonlinear activation. If @5 < u) is a general nonlinear activation function, we can Taylor With small initialization, u se O, the early dynamics near ini-
expand (a;; u) around tialization is dominated by the lowest-order non-vanishing term in the Taylor expansion, assuming the data statistic associated with that term is nonzero. For example, in two-layer fully-connected tanh network, the lowest-order von-vinishing term is the linear term. The tanh network thus de- velops rank-one weights in the early phase near initialization, similar to Theorem However, the subsequent dynamics is not necessarily saddle-to-saddle, since rank-one weights do not gen- erally correspond to invariant manifolds for tanh networks; see Figure 4D. By comparison, a two-layer collecconnected network with activation ø(æ; u) = u x tanh(u u x). the lowest-order non-vanishing term is quadratic. The network thus has a timescale separation between units similar to Proposition 5, and exhibits saddle-to-saddi dynamics shown in Figure 4F.
IMPLICATIONS
We now validate our theory and demonstrate its predictive power by examining how the network width, data distribution, and initialization affect learning dynamics.
Effect of network width. Our analysis in Section 5.1 shows that in linear networks, the timescale separation occurs between directions across all units. Consequently, increasing the number of units in linear networks has little effect on the dynamics, provided there are enough units to learn all directions. In contrast, the analysis in Section 5.2 implies that increasing the number of units in net- works where <(x;; u.) that is quadratic in u can shorten the plateaus. That is because the timescale separation in the quadratic case occurs between learning different units due to their distinct initial values. When sampling initial weights from fixed distribution, increasing the number of weights reduces the gaps between adjacent samples, thereby shortening the plateaus. Simulations in Fig- ure 2A confirm our theoretical prediction. In this case, increasing the number of heads of linear self-attention, for which o( o(x; u) is quadratic in u, speeds up learning, while increasing the width of fully- colnecc linear networks does not. This demonstrates an interesting, theoretically grounded advantage of scaling up linear self-attention over scaling up fully-connected linear networks.
8

---

Published as a conference paper at ICLR 2026
Effect of data distribution. In linear networks, the timescale separation in learning different di- rections arises from the distinct singular values of Σys. In Figure 2B, we let the singular values of Σye follow power law. As expected, decreasing the power law exponent narrows the gaps between singular values, thereby shortening the plateaus. When the exponent is 0, all the singular values are equal, eliminating the plateaus except the initial one corresponding to the escape from the saddle at zero, In this case, the largest singular value has muitiplicity r = D in Theorem 4, causing the solution to jump directly from effective width 0 to D, skipping the stages in between. By contrast, in networks for which 6( u) is quadratic in u, the timescale separation is due to the distinct initial values in the units. Therefore, setting the positive singular values of Σyg be equal shortens but does not eliminate plateaus. Simulations with linear self-attention in Figure 2B confirm our prediction.
Effect of initialization structure. According to our theory, to have saddle-to-saddlt dynamics the initialization must be near an invariant manifold, and the escape path from saddles must follow an invariant manifold. Perhaps surprisingly, however, initializing near saddle is not necessary con- dition. In Figure 2C we initialize the weights near an invariant manifold but away from saddles; for linear networks, this corresponds to large low-rank weights with small perturbation. As predicted, learning undergoes maddle-to-saddee dynamics. Because the initialization is away from saddles, there is not a plateau at the start; the loss first drops exponentially and then exhibits plateaus followed by sigmoid-shaped drops. To our knowledge, this regime has not previously been observed. If we initialize near the invariant manifold associated with exactly the required number of effective units, loss undergoes a rapid exponential drop. even though the network learns a solution with low-rank weights, which is the feature learning solution in linear networks (Dominé al., 2025). This result adds nuance to the common view that exponential loss curves are often a hallmark of lazy learning (Jacot et al., 2018; Chizat et al., 2019).
Effeet of initialization scale. We examine the effect of initialization scale when using an isotropic Gaussian distribution, a common choice in practice. As shown in Figure 2D, increasing the ini- tialization scale gradually shortens the plateaus. Saddle-to-saddle dynamics becomes weaker in the sense that the learning trajectory does not approach the saddles as closely as it does with small ini- tialization. For intermediate initialization scales, plateaus are less pronounced, yet the network still approximately learns solution of increasing complexity, similar to the case with small initialization. In architectures that have saddle-to- so-sdde dynamics, we conjecture that the distance from the initial weights to invariant manifolds associated with low effective width determines the strength of feature learning. This criterion can be viewed as an extension of prior beliefs, in which the relative scale of initial weights across layers (Dominé et al., 2025) or the rank of initial weights (Liu et al., 2024) were thought to determine the strength of feature learning.
7 DISCUSSION
We studied the gradient flow dynamics of a broad class of architectures, analyzing fixed points, invariant manifolds, and dynamics near fixed points. Our theoretical framework reveals a general mechanism for saddle-to-saddle dynamics and provides definition of simplicity that reflects the inductive biases of different architectures. When a network exhibits saddle-to-saddie dynamics, it recruits one or a few new effective units during each transition and learns solutions of increasingly complexity, where complexity is measured by the minimal number of units required for the archi- tecture express the solution. On high level, we identify a mechanism behind the intuition that a neural network can decompose task into smaller pieces and learn piece by piece over time. The leamnigg process sometimes reconstructs the network's 's own architecture, one unit at a time.
Condition for saddle-to-adddle dynamics. Saddle-to-saddle dynamics depends on two conditions: (i) the escape path from saddles closely follows invariant manifolds with few additional effective units; and (ii) the initialization is close to an invariant manifold with fewer effective units than needed to attain zero loss. As an example violating the first condition, two-layer tanh networks with small initialization develop rank-one weights during the early phase. This is because the tanh function is approximately linear near zero, and thus the early dynamics is approximately a linear dynamical system, similar to Theorem 4. However, since tanh is not homogeneous, rank-one weights do not correspond to an invariant manifold with effective width one. Consequently, tanh networks not guided to approach the saddle with one effective unit, and probably do not have saddle-to-

---

Published as a conference paper at ICLR 2026
saddle dynamics in general. As an example violating the second condition, large isotropic random initialization is almost surely away from invariant manifolds. Thus, neural networks with large dynamics. A special case violating
random initialization generally do not exhibit saddle-to-sade the second condition is when an architecture has full expressivity with a single unit, such as linear networks with scalar input scalar output (Shamir, 2019), and linear self-attention with merged key and query weights (Zhang et al., 2025b).
Deep networks. The fixed points and invariant manifolds in Sections 3 and 4 apply general deep networks defined by Equation (I), whereas the analysis of dynamics in Section 5 only applies to two-layer networks. Nonetheless, many deep newtorks still exhibit poddle-to-saddee dynamics, with some showing a timescale separation between directions and some between umits, as shown in Figure 5. Although general treatment of deep network dynamies is beyond the scope of this paper, we propose a conjecture for predicting which type of timescale separation (between directions or units) arises within layer of deep network. We conjecture that the order of the activation function o (a)); u.). whether is linear or quadratic in u; continues to predict learning behaviors, including the type of the timescale separation and the effects of width and data distribution. In deep networks, Im (a) in Equation (1) may involve weights that are not specific to any individual unit of the layer under consideration, i.e., weights in shallower layers not indexed by i. For example, let us consider the second hidden layer of depth-3 linear fully-connected network:
H Σ u u.)()(())
f(x)
E ini In where gin(x) = WWx
(17)
where W is the first-layer weight matrix.' Since (g (x); u.) = un Jin (x)( is linear in u, we pre- dict a timescale separation between directions similar to Section 5. and that the weights acquire an additional rank during each saddle-to-saddle transition. This is consistent with the existing literature (Gidel et al., 2019; Gissin et al., 2020) and our simulations in Figure
We further note that deep networks introduce several new questions that do not arise in the two- layer setting. deep networks visit a sequence of embedded fixed points and learn increasingly complex solutions by recruiting additional effective units, which layers recruit additional units at each increase in complexity? This question is particularly interesting for transformers, which have self-attention, fully-connected layers, and skip connections. With skip connections, deep network may also learn increasingly complex solutions by recruiting additional layers. This possibility seems consistent with the literature on layer pruning showing that large-scale transformers maintain their performance when removing up to half of the deeper layers and performing a small amount of finetuning (Gromov et al., 2025). Another work modeled the increasingly complex solutions of transformer by increasing the width of its fully-connected layers (Wurgaft et al., 2025).
Exhaustiveness of fixed points and invariant manifolds. Although we have not identified any fixed points or invariant manifolds beyond Proposition 5 and Theorem 3, it remains an open question whether these are exhaustive. If not, under what conditions do they become so? If the fixed points are exhaustive under reasonable assumptions, they would provide a useful diagnostic: each plateau during training would indicate that the network is implementing a solution expressible by narrower sub-ne nub-network. Moreover, the fixed points and invariant manifolds we describe arise solely from the network architecture and thus hold for any training data set. A further question is whether particular data sets can induce more fixed points or invariant manifolds than the data-agnostic ones (Zhao et al., 2023; Misof et al., 2025).
Other architectures and learning rules. At its core, our theory exploits the permutation symmetry of units in feed-forward neural networks defined by Equation (I). Permutation symmetry exists beyond feed-forward architectures and supervised learning rules. Indeed, stage-like learning curves have been observed in recurrent neural networks (Proca et al., 2025; Ger & Barak, 2025), and other learning rules, such reinforcement learning (Schaul et al., 2019). self-supervised learning (Simon et al., 2023), and predictive coding (Innocenti et al., 2024). This suggests the possibility of an even broader theory that incorporates these architectures and learning rules, with progressive permutation symmetry breaking as unifying explanation for progressive learning behaviors.
AA depth-3 linear network differs from linear self-attention, D" XQ,K, X XV,, because in linear self-attention all weights are indexed by and thus cannot be absorbed into (z)).).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Samuel Liebana, Lock van Rossem, Erin Grant, Stefano Sarao Mannelli, Mâté Lengyel, Valentina Njaradi, Aaditya K. Singh, Andrew Lampinen, and Jin Hwa Lee for helpful conversations, and anonymous reviewers for their constructive feedback.
We thank the following funding sources: Gatsby Charitable Foundation (GAT3850 and GAT4058) to Y7. AS, and PEL:; Sainsbury Wellcome Centre Core Grant from Wellcome (219627/7/19/22) to AS; Schmidt Science Polymath Award to AS. AS a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.
REFERENCES
Emmanuel Abbe, Enric Boix Adserå, and Theodor Misiakiewiez. Sgd learning on neural net- works: leap complexity and saddle-to-sadde. dynamics. In Gergely Neu and Lorenzo Rosasco (eds.), Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceed- ings of Machine Learning Research, pp. 2552-2623, PMLR, 12-15 Jul 2023. URL https: i/proceedings.mlr.pprrree www.mm....reesvveee.m v195/ html.
El Mehdi Achour, François Malgouyres, and Sébastien Gerchinovitz. The loss landscape of deep linear neural networks: a second-order analysis. Journal of Machine Learning Reseurch, 25(242): 1-76, 2024. URL http::/jtmlr..ee oeeeeorggggeeeeeeeeee http://ir.rrggppp/e////esea 223/23- 0493.html.
Madhu Advani, Andrew M. Saxe, and Haim Sompolinsky. fiighddimensional dynamics of gener- alization error in neural networks. Neural Networks, 132:428-446, 2020. ISSN 0893-6080. doi: https://doi.org// wwt.://orr.....1ooeeeeeeeee.ee 2020 08.0088..222. URL httpp://www.scceeneediieee e/ e//aridce/pp//0093000000011111
/scienc
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamies learning in neuromanifolds. Neural Computation, 1805):1007-10655, 05 2006. ISSN 0899-76677. doi: 10001....00.
2aa.1622ncco2200.18......
URL http:://ool.oor//.....o
Shun-ichi Amari, Tomoko Ozeki, Florent Cousseau, and Haikun Wei. Dynamics of learning in hierarchical models - singularity and milnor attractor. In Rubin Wang and Fanji Gu (eds.), Ad- vances in Cognitive Neurodynamics (II), pp. 3-9, Dordrecht, 2011. Springer Netherlands. ISBN 088-904441-995-40
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 233-242. PMLR, 06-11 Aug 2017. URL URL hatpas//pocceeddngg mlr. mlr, vrres//v0// arpit htmml.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022.
Yuri Bakhtin. Noisy heteroclinic networks. Probability theory and related fields, 150(1):1 42, 2011.
Pierre Baldi and Kurt Horik.. Neural networks and principal component analysis: Learning from examples without local minima. Newral Nenworks, 2(1):53-58, 1989. ISSN 0893-6080. doi:
http://o...rr///1100000000300099 nee/aartcce//p//833000000000011
URL https:/ hitts://wwwr http://wwwwwcccceeeo www.weccencedirett.oomm//
Ioannis Bantzis, James B Simon, and Arthur Jacot. Saddle-to-saddee dynamics in deep relu net- works: Low-rank bias in the first saddle escape. In The Fourteenth International Conference on Learning Representations, 2026. URL httpps://openrevieww http:///ppmeeeeeeeaeeeeeoonnnmd/ net/foormmtdd
Raphaël Berthier. Incremental learning in diagonal linear networks. Journal of Machine Learning Research, 24(171):1-26 2023. URL http:/ httttttpppppppp//eeeeeeeee 1392. html.
11

---

Published as a conference paper at ICLR 2026
Raphaël Berthier, Andrea Montanari, and Kangjie Zhou. Learning time-scales in two-layers neural networks. Foundations of Computational Mathematics, pp. 1-84, 2024.
Raphaël Berthier. Diagonal linear networks and the lasso regularization path, 2026. URL https: Srrzii..rrgabb/250000 18766.
Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transform- ers and their ability to learn sparse Boolean functions. In Anna Rogers, Jordan Boyd-Graben and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics ( (Volume I: Long Papers), pp- 5767-5791, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. N0.18653/v1// 12/33//1/222333 acl-ooog...lll URL
bttpp://aclaantooos org/ www.////////ee/e/////////// acl-long..111111 317/.
Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind. Trans- formers learn through gradual rank increase. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 24519-2455. Curran Associates, Inc., 2023, URL htttps://proceedinggs wwwpeeedddinnneeuis..ooommm cc/aaper Paper- Conference.per
Etienne Boursier and Nicolas Flammarion. Simplicity bias and optimization threshold in two- layer ReLU networks. Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julics Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd In- lernational Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 5241-5275. PMLR, 13-19 Jul 2025a. URL https:/ / http:////ooceeeegsse//ee mlr malr.pres/// v2 67 07/boursier25a... html.
Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is two-edged sword. Journal of Machine Learning Research, 26(183): 2025b. URL http:
hwtpl/rrrrgg/pperr///ee 2/2-44////////---/ 1523 html.
Etienne Boursier, Loucas PILLAUD- VIVIEN,,, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 20105-20118 18. Curran Associates, Inc., 2022. URL https://proced ags.neurips.co wwweurips.ccppaat aper- Oper-Conference.pde
Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth Interna- tional Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2205-2211. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijcai22221/333 URL Main Track.
https:/ /ttts://doi.org org/ 10. 2...2493//// 11963/ijcai..oo 2AA./20223300
Ping-yeh Chiang, Renkun Ni, David Yu Miller, Arpit Bansal, Jonas Geiping, Micah Goldblum, and Tom Goldstein. Loss landscapes are all you need: Neural network generalization can be explained without the implicit bias of gradient descent. In The Eleventh International Conference on Learning Representations, 2023. URL Zy9.
.net// .ett/forrumlldd
Dmitry Chistikov, Matthias Englert, and Ranko Lazic. Learning neuron by shallow relu net- work: Dynamics and implicit bias for correlated inputs. In A. Oh, Naumann, Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information P Processing Systems, volume 36, pp. 23748-23760. Curran Associates, Inc., 2023, URL https: https:/pproceedigg..n Paper-Confe Confe
ips.cc/papes sence.pdf.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpe://proceedingg wwwp:rrrwwenniinnnoirrr...o 3266b29657725455-Paapae httpggg..eorrppp...pppap..r saapper_illespppprr/// 2tt////11///////////// fllle//611c5578433.
12

---

Published as a conference paper at ICLR 2026
Kamaludin Dingle, Chico Camargo, and Ard A Louis. imput-outph maps strongly biased towards simple outputs. Nature communications, 91)):611, 2018.
Clémentine Carla Juliette Dominé, Nicolas Anguita, Alexandra Maria Proca, Lukas Braun, Daniel Kunin, Pedro A. M. Mediano, and Andrew M Saxe. From lazy to rich: Exact learning dynamics in deep linear networks. In The Thirteenth International Conference on Learning Representations, 2025. URL https:///penrevieeeeeee http://orrnmmmiiieaaacooooo
Ezra Edelman, Nikolaos Tsilivis, Benjamin L. Edelman, Eran Malach, and Surbhi Goel. The evolution of statistical induction heads: In-context learning markov chains. In A. Globerson, in Neural Information Processing Systems, volume 37, pp. 64273-64311. Curran Associates, Inc., 2024. doi: 10.55202/009901722005. URL ptteee//pprocccednngg. www.ips.eerrr.cooo cc/ papprrappp..p faaearffflllseee//eaea/ere ottp://e770bbbeeee26000000000060 Pappeeaaeeeeeeeeeaaaaaaaaeaaaaaa Pww.aaaprrrerrneeeepaaaaaa
L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, 7ac Hatfield-Dodss, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superpo- sition, 2022. URL hitpptss/traaasffoomeee http://ttaansfore//eeee//// circutttrriictts..e. pubp2022/toy totttoooooiiideeo.mmmoomo
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://open review.net/forume
Spencer Frei, Niladri S. Chatterji, and Peter I. Bartlett. Random feature amplification: Feature learning and generalization neural networks. Journal of Machine Learning Research, 24(303): 1-49, 2023a. URL ttttp://jmii wng/paapers/224/22- 1132.112222 htmml
Spencer Frei, Gal Vardi, Peter Bartlett, Nathan Srebro, and Wei Hu. Implicit bias in leaky reLU net- works trained on high-dimensional data. In The Eleventh International Conference on Learning Representations, 2023b. URL hwtphhttt///ppeeeee /owpneeveeeeeeeeeeeeeeeeee net/forum httd///onummm--JJpppeeeeEEEW
Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of
multilayer perceptrons. Neural Networks, 13(3):317-327, 2000. ISSN 0893-6080. doi: https:
//tb:./g/00..0116600033400000000 atticl/ppi/$09988690000009
URL
Kenji Fukumizu, Shoichiro Yamaguchi, Yoh-ichi Mototake, and Mirai Tanaka. Semi-flat min- ima and saddle points by embedding neural networks overparameterization.. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL nww.ip...enusisccoo www.rippsscpppppprrppp files/ filles/paeer/219/ /tt////eeeeeeeee 4445559686888000066 6 ed2de90d330acbba. Pacieeebbba-aaeee....
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30583-30598. Curran Associates, Inc., 2022. URL https://proceeeinggs neur pps.cc/paper paper ppper__ilesspele http://///.2ee////222aaaaaaaaaa2 ference. pdf.
ips.cc/
Paper- Con
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https://openrevieet net
Yoav Ger and Omri Barak. Learning dynamies of RNNs in closed-loop environments. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: // //ooprreview.nee http://..w..eoooomm/d/eeee..
Borjan Geshkovski. Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metastability in the self-attentic model, 2024. URL https:// /ttp///wwwworr/b////////////
13

---

Published as a conference paper at ICLR 2026
Nikhil Ghosh, Song Mei, and Bin Yu. The three stages of learning dynamics in high- dimensionni kernel methods. International Conference on Learning Representations, 2022. URL https: wwtp://tee/ooommmd-aeooa
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien Implicit regularization of discrete gradient dynamics in linear neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https:/ ptttps://proceeddnggg www.eeeiigg..eeueeeeee wwwwrips.c/ppappeeee Paper, pdf.
files/p
Daniel Gissin, Shai Shalev-Shwariz, and Amit Daniely. The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations, 2020.
URL https:/
Margalit Glasgow. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. In The Twelfth International Conference on Learning Representations, 2024. URL
Micah Goldblum, Marc Anton Finzi, Keefer Rowan, and Andrew Gordon Wilson. Position: The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings the 4 International Conference Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15788-15808. PMLR, 21-27 Jul 2024. URL https:/ httppas//ppcceedingg. wwa.ingg........... www.:prees/23333 /ttpa//////////o/mmm/m22/ gol.dbumm24..a.mmmmmm html.
Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdehorová. Modeling the influence of data structure learning in neural networks: The hidden manifold model. Phys. Rev. X. 10: 041044, Dee 2020. doi: 10.100/PhySReex.10.44 10.041444. URL 1103/PhysRevx.10.o 10.041444..
wpw.pp/p....pp.................p /oi/100.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Conference on Learning Representations, 2025. URL httpps///pperreiee /oggm/rrrumd
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9995- 10006. Curran Associates, Inc., 2020. URL httpp::/pprcceeding neurips.cc/pappe files/p Paper.pda
G. E. Hinton, J. L. McClelland, and D. E. Rumelhart. Distributed representations, pp. 77-109. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimiz- ing the description length of the weights. Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT '93, pp. 5-13, New York, NY, USA, 1993. Associa- tion for Computing Machinery. ISBN 0897916115. doi: 10. 10.1145/168304.160 168306. URL https:/ /tt.:/iiiooooo 111.1111111111111111111
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997. http://ccoo.999..... 1997.9.1.1. URL http::////oo.oo11111111110
ISSN 0899-7667. doi: 10.1162/neco. 9.1.1.
1997.
David Holzmüller and Ingo Steinwart. Training two-layer relu networks with gradient descent is inconsistent. Journal of Machine Learning Research, 23(181):1-82, 2022. URL http://jmlr.o rg/ ht//aapers/v23//// 2001/00000300000 830.83000mmmm0 html.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early- time learning dynamics of neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17116-17128. Curran Associates, Inc., 2020. URL https:/ / heurips.cc/pappeep Pap.edra.aaaa...ff..
14

---

Published as a conference paper at ICLR 2026
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
Francesco Innocenti, EI Mchdi Achour, Ryan Singh, and Christopher L. Buckley. Only strict saddles in the energy landscape of predictive coding networks? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 33649-53683 Curran Associates, Inc., 2024. URL httpe://prrcoedi ngs. wgs.meerips.cc/ paperrapppepp.r files/ fttle//paperr2222///////////// http:///oooo...eo.t..ooooooaaat. er- St-CConfrrence.pde
Masato Inoue, Hyeyoung Park, and Masato Okada. On-line learning theory of soft committee ma- chines with correlated hidden units -steepsst gradient descent and natural gradient descent-. Jour- nal of the Physical Society of Japan, (2(4):8558114 2003. https://doi.org/ http://d...rr1111111/////////
doi: 10.1 10.1143/JPSI.72.805 URL
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- eralization neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceeedngg www.eeddiinnnnnnnss...e htt.c/pppppefflllppapppaaeeeeeee files/paper/2 .df.
Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity, 2022.
URL https://arxiv
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In net/f
International Conference on Learning Representations, 2019. URL orum?id=HJfg300
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Infor- mation Processing Systems, volume 33, pp. 17176-17186. Curran Associates, Inc., 2020. URL httpe://proceeding.. neurips.cc/papet httpp///ee.aaaaoooooooooooooooaa Pap.e444cc2be-PPpp......
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions increasing complexity. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL wwwpsss..eiisseeeeeeeee wwtpcccpppppeeliieeee Daalls/paper/2000 http///eeeeee2233559999/9// 7c806a895ecc5e25-Paaa
https:// Papert..-papp.ppppee
Katie Kang, Amrith Setlur, Claire Tomlin, and Sergey Levine. Deep neural networks tend to extrap- olate predictably. In The Twelfth International Conference on Learning Representations, 2024.
URL htttps//opperreeeee
net/forum (odd:ummmmddjjoooorooo)
Dhruva Karkada, James Simon, Yasaman Bahri, and Michael R DeWeese. Closed-form training dynamics reveal learned features and linear structure in word2vec-likk models. In The Thirty- ninth Anmual Conference on Neural Information Processing Systems, 2025. URL https://open review
Kenji Kawaguchi. Deep learning without poor local minima. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL htttss:/pproceeediiii httee/2229922667122444415aaaaaa2 Paw.a//aaa222999oroaar.rpo. Paper.dd.....p............ .df.
meurips.c//par
files/ files/paper/2
Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks nearly-orthogonal data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 30167-30221. Curran Associates, Inc., 2023. URL https: https:/pprocceiiggssss neur ips.co cc/ ppape__fille wwwpss/aapee////// paper 2aaper/2023/550 /.0.:0//00//..0.....0.0202222333 ference. pdf.
Papee-a--a----------------e Con
15

---

Published as a conference paper at ICLR 2026
Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in wwo-homogeneous neural networks. Transactions on Machine Learning Research, 2024, ISSN 2835-8856. URL httpss//openrrvieewnee// htwp://wwweetooommmdddd-reeree
Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37. pp. 81157- 81203. Curran Associates, Inc., 2024. URL https://pproceeding www.wg........u.oom cc/ cttpppprccpppprprpeeeepee _fles//lls_ppp_
Conference.
Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R DeWeese, Surya Ganguli, and Nina Miolane, Alternating gradient flows: A theory of feature learning in two-layer neural networks. In The Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems, 2025. URL
net wottffoummiddi 17LKKO
Thomas Laurent and James von Brecht. Deep linear networks with arbitrary loss: All local minima are global. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp- 2902-2907. PMLR, 10-15 Jul 2018. URL https://://// ttttps://prcceedinng. mlr. .html.
www.prees.///a///ttem..e press/v80 Saaees/v00//aaaa v80/laurent18a
Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. In International Conference on Learning Representations, 2022. URL. https://open review.net
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmaan, 1989. URL https: ///proceedingg.neeeeee wwwweee...ppaaaeeeeeeee cc/ cc/paper lappppapalle//a//llllllllaaeeeel ftills/paeer////e http////h/.eeeeeeeaaaa//0000aaa0 81277658-Paper.p2
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2021. URL
Yuhan Helena Liu, Aristide Baratin, Jonathan Cornford, Stefan Mihalas, Eric Todd SheaBrown, and Guillaume Lajoie. How connectivity structure shapes rich and lazy learning in neural circuits. In The Twelfth International Conference on Learning Representations, 2024. URL hstps://open review.net fide://foommmddsllsss ?id Y
Haihao Lu and Kenji Kawaguchi. Depth creates no bad local minima, 2017. URL hwwph:httt//rreee /aaxiv.. org/ 1atp//112..00800.
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at infinite-widde limit. Journal of Machine Learming Research, 22(71):1-47, 2021. URL http://jmlr.orz org/ httppaarrrrggpppprr/eeeeeeeeeeee 22/p///122222222222222222 html..
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent two-layer Margin maximization and simplicity bias. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1 er sr_ffles/ppal
Curran Associates, Inc., 2021. URL httpss//prroceedings nwwrips.ccceeeeeee httpps.ccpppppp features, 2018 URL htt.a///wppppoo.oo....cc./aaaaaa
Paper....d
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network
wwwp://///////oooooooooo attpaa///111.03..... 083667
Pierre Marion and Raphaël Berthier. Leveraging the two-timescale regime to demonstrate conver- gence of neural networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 64996- 65029. Curran Associates, Inc.. 2023. URL http://pproeeednng.eeeuee www.eeddiiiinniuiiseeeooo newpeps.ccpppaprrrrr pappr_ffieeee//pppppppppp
.df.
16

---

Published as a conference paper at ICLR 2026
Hancheng Min, Enrique Mallada, and Rene Vidal. Early neuron alignment in two-layer reLU net- works with small initialization. In The Twelfth Interationall Conference on Learning Represen- tations, 2024. URL https:/ https://openrevieww
Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, and Ard Louis. Is sgd a bayesian sampler? well, almost. Journal of Machine Learning Research, 22( 64, 2021. URL http:/ http:://mmmrrooo// org
[papers/v22/20-1 (00-676..tll..
Chris Mingard, Henry Rees, Guillermo Valle-Pérez, and Ard A Louis. Deep neural networks have an inbuilt occam's 's razor. Nature Communications, 16(1):220, 2025. doi: https://doi.org/110 1033// s4 941467-024-548133 URL http://///dbborr/0000000000///1/
Philipp Misof, Pan Kessel, and Jan E Gerken. Equivariant neural tangent kemelss. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.). Proceedings of the 42nd International Conference Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 44470-44503, PMLR, 13-19 Jul 2025. URL https:/ htttpp://pprceeeee proceedinge...... mlr, wttitrpress/eee press/v267 http://v6//mmmmmo /wwso//is.aaaa////////// html..
Emin Orhan and Xaq Pitkow. Skip connections eliminate singularities. In International Conference on Learning Representations, 2018. URL
WCZ.
Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 7475-7505. Curran Associates, Inc., 2023. URL httpps//pproocediiggnaeuurrr..ep Pagp:aaadd..aaaPPaaa--Cooaaaaaa
files/
Leonardo Petrini, Francesco Cagnetta, Eric Vinden-Eijnden and Matthicu Wyart. Learning sparse features can lead to overfitting in neural networks. In S. Koyejo. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9403-9416. Curran Associates, Inc., 2022. URL httphtttss//rrrceeiigg..ee s.ce/paper httpg///pppooooo////..a../0o0060 rence.pdf.
Confe
Mary Phuong and Christoph H Lampert. The inductive bias of relu networks on orthogonally separable data, In International Conference on Learning Representations, 2021. URL https: / //openreview.ng /
Alexandra Maria Proca, Clémentine Carla Juliette Dominé, Murray Shanahan, and Pedro A. M. Mediano. Learning dynamics in linear recurrent neural networks. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp. Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 49900-49902. PMLR, 13-19 Jul 2025. URL https:/ httpe://ppcceeddngg mlr aalr.ppess//ee press/v267/ v267 wtt./proc225a.... /ttp:/pa..aaammmmmmm
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019, URL https:/ ppoceedings..mm mlr press/v97 /tt/aaaaaaaaaaaaam html.
Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with SGD learn distributions of increasing complexity. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. pp. 28843-28863. PMLR, 23-29 Jul 2023. URL httpp:://rroceedings mlr, press/ v202/refinetti23a. htmm. html.
Riccardo Rende, Federica Gerace, Alessandro Laio, and Sebastian Goldt. A distributional simplicity bias in the learning dynamics of transformers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems,
17

---

Published as a conference paper at ICLR 2026
volume 37, pp. 96207-96228. Curran Associates, Inc., 2024. URL URL https//ppp// ca/ppperr http:///aarce/..../.c/c//sss//aa Paper- Con ference.pdf.
Jirko Rubruck, Jan Philipp Bauer, Andrew M Saxe, and Christopher Summerfield. Early learning of the optimal constant solution neural networks and humans. 8th Annual Conference on
Cognitive Computational Neuroscience, 2025. URL https:///ppenrertew htwp//rwwwweeeee///ouuumeoa foree//oommmddx yu486HRh.
Roei Sarussi, Alon Brutzkus, and Amir Globerson. Towards understanding learning in neural net- works with linear teachers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pp. 9313-9322. PMLR, 18 24 Jul 2021. URL https://proceedings www.ggs.................. press/ v139/sarussi2n html.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. In The Second International Conference Learn- ing Representations, 2014. URL httpss//opeerevvite net/forum _id=_ __tpddd=_wwWpDDFFF _C.
Andrew M. Saxe, James L.. McClelland, and Surya Ganguli. mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 (53):11337 2019. doi: 10. 107.3/0pas..18200 1820226116. URL https:/ / /www, pnw.. org/doi /abs/10.1073/pnaa pnas. wttpa............... 1820226116.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of plateaus in deep reinforcement learning, 2019. URL httpp:://arxiv.oog//aaaaae http://www.orr/ab////..11.... 11455.
Jürgen Schmidhuber. Discovering neural nets with low kolmogorov complexity and high gener- alization capability. Neural Networks, 10(5):857-873, 1997. ISSN 0893-6080. doi: https:
1oooooorg//..1101116000933--6609 #tticae/pp/S099660096000777
URL /ttps://a www.scienccdireet.com//eeeeeeeee
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 9573-9585. Curran Associates, Inc., 2020. URL https::/prroceedingg..u wwwpssnerrippcco www.ips..uepppppeeepeeee cecp/appr_aae es/ he/ppaprr/202//55
pdf.
Ohad Shamir. Exponential convergence time of gradient descent for one-dimensional deep linear neural networks. In Alina Beygelzimer and Daniel Hsu (eds.). Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pp. 2691-2713. PMLR, 25-28 Jun 2019. URL htttps://proceedinng/ mlr, mulll.press//eeee press/v999 w99//999/miii shamiir1a. htmll
James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Leurning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 31852-31876. PMLR, 23-29 Jul 2023. URL
mlr whtiirpres///e ppwss//v02/simmaaeeeee
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerst- ner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9722-9732. PMLR, 18-24 Jul 2021. URL http:ss///roceediingg. mlr, watr.prrss/vv v1 39/simsek21 h.mmmel21a..mml.. html.
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10153-10161. PMLR, 18-24 Jul 2021. URL https:/ http///pooeeeen.eccco mlr vataes//vvv///////e http///119///ttttae /ttt///mmmmnaaattaaa html.
18

---

Published as a conference paper at ICLR 2026
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16761-16772, June 2022.
Damien Teney, Armand Mihai Nicolicioiu, Valentin Hartmann, and Ehsan Abbasnejad. Neural red- shift: Random networks are not random functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4786-4796, June 2024.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In Shipra Agrawal and Francesco Orabona (eds.), Proceedings of The 34th Inter- national Conference on Algorithumic Learning Theory, volume 201 of Proceedings of Machine Learning Research, pp. 1429-1459, PMLR, 20 Feb-23 Feb 2023, URL https///proceedings. mlr www.ppees/v////immm/ree html.
Nikita Tsoy and Nikola Konstantinov. Simplicity bias of two-layer networks beyond linearly sepa- rable data. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 48728- 48767. PMLR, 21-27 Jul 2024. URL https:/ httpas////occeeddeeggeee mlr http////3//tttoy2y5// html.
Zhenfeng Tu, Santiago Aranguri, and Arthur Jacot. Mixed dynamics in linear networks: Unifying the lazy and active regimes. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tom- Advances
czak, and C. Zhang Advancess in Neural Information Processing Systems, volume 37, pp- 106059-106104 Curran Associates, Inc., 2024. URL https://prroceeednggss http://erreeiiiin.nerrereerepeee htt.lecccppaae paper fills//ppper2024// 21t.0////000..1.6600000005555010 pappaa/aeeeeaea33aa---aaaee- Coweee-Cooeeeenccccom pdf.
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the murameter-functioo map is biased towards simple functions. In International Conference Learning Representations, 2019. URL httpp://opeereeeteeee
net fidrrrrmmm
Aditya Varre, Gizem Yüce, and Nicolas Flammarion. Learning in-context n-grams with trans- formers: Sub-n-grams are near-stationary points. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Pro- ceedings of Machine Learning Research, pp. 60924-60963. PMLR, 13-19 Jul 2025. URL https:// httpss///poccedingg...o mlr press/v267 wvarr225a.htm..
Binxu Wang and Cengiz. Pehlevan. An analytical theory of spectral bias in the learning dynamics of diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://opemrerr//
net het//orrumiddd-3000 h
Mingze Wang and Chao Ma. Understanding multi-phase optimization dynamics and rich nonlinear behaviors of reLU networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL attps://openrevieet httppp///pppmmeeeeeeeeee//onnnnn
Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari. Dynamics of learning near singularities in layered networks. Neural Computation, (033):8138444 03 2008. ISSN 0869. doi: 211162/eco.2007..210 2007.12-06-411-. URL 007 10--144064414.
Stephen Wiggins. Invariant Manifolds: Linear and Nonlinear Systems, pp. 28-70. Springer New York, New York, NY, 2003, ISBN 978-0-387-211449-9 doi: 10. 11110070-387--1749--5--- URL /ttp:////oo..rg/100000000/001000
Zhengqing Wu, Berfin Simsek, and François Gaston Ged. Loss landscape of shallow reLU-like neural networks: Stationary points, saddle escape. and network embedding. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreviiet net/f orum?/d=ogKEELcvvV
Daniel Wurgaft, Ekdeep Singh Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, and Noah Goodman. In-context learning strategies emerge rationally. In The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025. URL attpss//oppenrviiii net/f
19

---

Published as a conference paper at ICLR 2026
Yu Yang, Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman. Identifying spurious biases early in training through the lens of simplicity bias. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference Artificial Intelli- gence and Statistics, volume 238 of Proceedings of Machine learning Research, pp. 2953-2961. PMLR, 02-04 May 2024. URL http::// pttp://///oreeeings. mlr.r.gsmmmmrrrrmmmm. wwt.:mmwl..preeeeee wwwps/resseeeee/eeeee h
Yuki Yoshida and Masato Okada. Data-dependence of plateau phenomenon in learning with neu- ral network - statistical mechanical analysis. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL. https:// http///oocdnnn.nnuurrrrree nwwrips.cc/eeeeeeee ccppaper paper.pdr.
Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, and Nicolas Flammarion. Incremental learning of sparse attention patterns in transformers. In EurlPS 2025 Workshop on Principles of Generative Modeling (PriGM), 2025. URL http::://ppenreeeieww http::///geeaarreevve.eeerrruurr wttp:///ooommmmmdd/=22222/2//
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Global optimality conditions for deep neural networks. In International Conference on Learning Representations, 2018. URL https://openreview..a
et B//forum//d-BBBkCC......
Yaoyu Zhang, Zhongwang Zhang. Tao Luo, and Zhiqin J Xu. Embedding principle of loss landscape of deep neural networks. M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 14848- 14859. Curran Associates, Inc., 2021. URL htttps:/procceednggg hwtppprrwrnnnnnnniiii....o wwwrrps.ccppppprrrrr... p aper/2021 /ttp:////.ccc2227777777122222222 att.://51..777.aaaaa000000eaaaee
Yedi Zhang, Peter Latham, and Andrew M Saxe. Understanding unimodal bias in multimodal deep linear networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver. Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st Interna- tional Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Re- search, pp. 59100-59125. PMLR, 21-27 Jul 2024. URL https:/ http://ccccceeeessee mlr malr.press/v2 35/zhang24aa html.
Yedi Zhang, Andrew M Saxe, and Peter E. Latham. When are bias-free relu networks effectively linear networks? Transactions on Machine Learning Research, 2025a. ISSN 2835-8856. URL
https://openreview...t wwt.:/nneettoot/ddeee...... Ucpfdn66666
Yedi Zhang, Aaditya K Singh, Peter E. Latham, and Andrew M Saxe. Training dynamics of in- context learning in linear attention. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste- Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.). Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 76047-76087. PMLR, 13-19 Jul 2025b. URL https:/ http://ecoceee...mmmm t.press/v261 www.ga/nbnn..mmmmmmmmm html.
Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy. Symmetries, flat min- ima, and the conserved quantities of gradient flow. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevvew ntt//oourt/ddddeeeeooou 9lcp
Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang, Parameter symmetry potentially unifies deep learning theory, 2025. URL
20

---

Published as a conference paper at ICLR 2026
Table of Contents
1
Introduction
2 Network Setup
3
Loss Landscape: Embedded Fixed Points
Invariant Manifold: Effectively Narrow Networks
5 Saddle-to-Saddle Dynamics
5.1 Linear case: timescale separation between directions
6
5.2
Quadratic case: timescale separation between units
Implications
7
Discussion
A
Additional Related Work
22
A.I
Saddle-to-saddle dynamics
22
A.2 Incremental learning other settings
23
A.3
Simplicity bias
23
B Additional Figures
24
B.1
Learning dynamics two-layer networks
24
B.2
Learning dynamics in deep networks
26
B.3
The effect of skip connection
27
Additional Discussion
28
Gradient Calculations
Embedded Fixed Points
30
F
Invariant Manifolds
34
F.1
Definition of invariant manifolds
34
F2
Proof of invariant manifolds
F33
Expressivity invariant manifolds
................................
35
F.4
The embedded fixed points on invariant manifolds
36
Dynamics of Linear Networks
38
G.I
Gradient flow equations
................................
38
G.2 Proof of timescale separation
38
G.3 Fixed points of linear networks
39
H Dynamies of Quadratic Networks
H.l Gradient flow equations
41
H.2 Derivations for timescale separation
Implementation Details
44
21

---

Published as a conference paper at ICLR 2026
Appendix
A ADDITIONAL RELATED WORK
A.1 SADDLE-TO-SADDA DYNAMICS
Though saddle-to-sadde dynamics is a recurring phenomenon in the theory literature on learning dynamics, many questions remain open. The only case in which saddle-to-saddle dynamies has been proven for the full trajectory is the diagonal linear network in the limit of small initialization (Berthier, 2023; 2026; Pesme & Flammarion, 2023). For fully-connecte
linear networks, saddle- to-saddle dynamics has been shown under white input covariance and small spectral initialization (Saxe et al., 2014; 2019; Gidel et al., 2019; Gissin et al., 2020; Li et al., 2021). Outside this setting, the phase of visiting the first saddle from small initialization is well understood in linear networks (Jacot et al., 2022), while visiting subsequent saddles is not. For linear self-attention, Zhang et al. (2025b) showed saddle-to-saddle dynamics, but several phenomena remain unexplained. such as why it occurs even when the eigenvalues thought to govern the duration of plateaus are equal, and how increasing the number of heads affects the dynamics. While our work does not provide rigorous proofs for the technical open questions, we offer explanatory insights into these phenomena, and generate novel predictions on how network width, data statistics, and initialization affect saddle-to- saddle dynamics (Section 6).
There is also some work that did not focus on saddee dynamics but included relevant anal- ysis. Zhang et al. (2024) showed that multimodal deep linear networks exhibit -addeee dynamics. In multimodal deep linear networks with two input modalities, the saddle corresponds to unimodal solution that is learning to fit the output only using of the faster-to-learn modal- ity. Rubruck et al. (2025) showed that two-layer linear networks with bias terms in the first layer exhibit saddle-to-saddle dynamics. Under some conditions, the first saddle corresponds to learning an optimal constant solution (Kang et al., 2024) that is the mean of the target output regardless of input.
Phenomena related to the timescale separation between directions in Theorem 4 have been examined in prior studies and are often discussed as weight alignment (Ji & Telgarsky, 2019; 2020; Atanasov et al., 2022). In seminal work on the learning dynamics of deep linear networks (Saxe et al., 2014), aligned weights were introduced as an ansatz, with the analysis assuming aligned initial weights rather than deriving alignment from small isotropic initialization. This ansatz is often referred to in the linear network literature as the "spectral initialization" assumption; see Table of Tarmoun et al. (2021) for a list of common initialization assumptions for linear networks. Later, Atanasov et al. (2022) analyzed the early phase dynamics of two-layer linear networks with scalar output from small isotropic initialization. They showed that the weights become increasingly aligned with a rank-one direction in the early phase, coined as "silent alignment". Our Theorem 4 extends the "silent alignment" to the vector output case, and recovers it when the output is scalar. A useful note is that fully-connected linear networks with scalar output or scalar input do not have nonzero saddles, since a width-one network already has full expressivity. Consequently, they do not exhibit saddle-to-saddle dynamics except for escaping the saddle at zero.
Hu et al. (2020) showed that two-layer nonlinear networks from a particular symmetric initialization have similar dynamics to a linear model on the input in the early phase of training. Because the network outputs zero for any input at their symmetric initialization, their analysis is related to our analysis of linear network near small initialization (Section 5. Our Theorem 4 also leverages the fact that the network output close to zero and the dynamics approximately linear in the weights.
Kunin et al. (2025) proposed an algorithmic framework to capture staircase learning curves in two- layer networks. We go beyond the algorithmic level by presenting a theoretical framework that analyzes embedded fixed points, invariant manifolds, and two different timescale separation mech- anisms. We complement their work by addressing the question of why gradient descent dynamics behaves similarly to their algorithm. Further, our results on fixed points and invariant manifolds are not limited to two-layer networks; they apply to deep networks defined by Equation (1). We also disentangle data-induced and iawallizaaiinnnnndueee
saddle-to- toddad dynamics, and show that the former leads to low-rank weights while the latter leads to sparse weights. These two mechanisms were not distinguished in prior literature.
22

---

Published as a conference paper at ICLR 2026
Ziyin et al. (2025) proposed hypothesis that learning dynamics is generally symmetry-to symmetry. The aadde-to-saadlee dynamics in our work is related to permutation symmetry between the umits. Our work makes a theoretical case for their hypothesis and identifies the conditions under which saddle-to-saddle dynamics occurs.
A.2 INCREMENTAL LEARNING IN OTHER SETTINGS
Incremental learning, characterized by earlier phases corresponding to simpler solutions, has been examined in several other theoretical settings. Cao et al. (2021); Ghosh et al. (2022) studied the spectral bias in the neural tangent kernel regime. In the kernel regime, eigenfunctions with larger eigenvalues are learned faster. This behavior differs from saddle-to-saddle dynamics, as the network in the kernel regime neither visits saddles nor exhibits plateaus during learning. Instead, the train- ing loss decreases throughout learning, with faster decay early in learning and slower decay later. Outside the kernel regime, Abbe et al. (2023) studied a layer-wise training setup, in which the first and second layers trained separately. Marion & Berthier (2023); Berthier al. (2024) studied a two-timescale regime, in which the second-layer weights are trained with a much larger learning rate than the first-layer weights. In comparison, analysis focuses the learning dynamics of standard gradient descent outside the neural tangent kernel regime.
A.3 SIMPLICITY BIAS
In this paper, we focus on the dynamical simplicity bias; that is, learning increasingly complex solutions over the course of training. A broader, longstanding body of theoretical and experimen- tal research has explored the "stationary" simplicity bias, independent of training dynamics. Early studies (Hinton & van Camp, 1993; Hochreiter & Schmidhuher, 1997) connected generalization to the minimum description length of the weights, suggesting that flat minima correspond to simple solutions that potentially generalize well. A line of work based algorithmic information theory (Schmidhuber, 1997; Valle-Perez et al., 2019; Dingle et al., 2018; Mingard et al., 2021; 2025; Gold- blum al., 2024; Teney et al., 2024) showed that standard architectures with randomly sampled weights are biased toward (Huh et al., 2023) documented that both randomly initialized networks and trained networks exhibit a bias toward input-output maps with low effective rank embeddings. These findings motivated volume hypothesis: neural networks have a simplicity bias arising from the loss landscape; specifi- cally, the simple solutions occupy larger volume in the weight space than the complex ones (Huh et al., 2023; Chiang et al., 2023). Our work on dynamical simplicity bias complements the station- ary simplicity bias literature by examining how gradient descent dynamics drives the progression of solution complexity over time.
maps with low Kolmogorov complexity. Empirical work
According to the free lunch theorem, single inductive bias is universally beneficial. Thus, the simplicity bias can be either advantageous or detrimental depending on the task. Several studies (Hermann & Lampinen, 2020; Shah et al., 2020; Petrini al., 2022; Yang et al., 2024) have shown that favoring simple solutions may harm generalization when the simple solution relies on simple but spurious features, whereas more complex but robust features yield better generalization. For example, convolutional neural networks often prefer to classify objects by texture rather than shape, even though the classifier relying on shape can generalize better (Geirhos et al., 2019). Moreover, studies on two-layer ReLU networks (Holzmüller & Steinwart, 2022; Tsoy & Konstantinov, 2024; Boursier Flammarion, 2025a;b) demonstrated that simplicity bias can sometimes cause optimiza- tion difficulties, where the first-layer weights align with limited set of spurious directions when omnidirectional weights are required to reach global minima.
23

---

Published as a conference paper at ICLR 2026
B ADDITIONAL FIGURES
B.1 LEARNING DYNAMICS IN TWO-LAYER NETWORKS
fully-connect network
digit 0,1
digit 0,8
digit 5,7
digit 3,5
Lowk
0.5-
0.5-
0.6
0.5
8V
0.0
0.0
0.0
0.0
500
500
1000
500
1000
1000
ReLU fully-connect network
digit 0,1
digit 0,8
digit 5,7
digit 3,5
1.0
1.0
8V
5-5-
0.5
0.5
0.5
www.
8V
500
1000
500
1000
500
1000
500
1000
Figure 3: Saddle-to-saddle dynamics two-layer fully-connected linear and ReLU networks trained
for binary classification of MNIST digits. The input dimension is 28 x 28 784, the hidden layer width is 1000, and the target outputs are two-dimensional one-hot vectors. The intermediate plateau is longer when the to distinguish than digits 0/1. The colored curves represent the top three singular values of the first- layer weight matrix, U E ER 1010000784, Consistent with our theory, the growth of the first and second singular values coincides with the first and second abrupt drops in the training loss, respectively, corresponding to the increase in the effective width. The third largest singular value is close to zero, meaning the rank of the first-layer weight matrix is at most two, approximately. Details: The batch size is 64. The learning rate is 0.01. The initial weights are sampled independently from N(,, 10-12).
digits are harder distinguish. For example, digits 3/5 are harder
Table I: Singular values of MNIST binary classification data.
Digit pair
0,1
0,8
5.7
3.5
Singular value governing the first plateau
4.20
5.21
4.13
4.62
Singular value governing the second platcau
3.90
5.21
3,26
1.38
In Figure 3, we show the learning dynamics of two-layer linear and ReLU networks trained for binary classification of MNIST digits. Despite being noisier than the learning dynamics synthetic datasets, the plateaus and abrupt drops the training loss are still pronounced. The abrupt drop in loss also coincides with the growth of singular value of the first-layer weight matrix, implying that an increase in the effective width can explain the observed dynamics. We show the singular values governing the duration of the first and second plateaus of the linear network dynamics in Table 1, which are the first singular values of Σyz and (I eje, )E respectively. Here is the first cigenvector of the matrix E -.EE gas see Lemma 6. The sizes of the singular values approximately match the duration of the plateaus Figure 3.
24

---

Published as a conference paper at ICLR 2026
0th-order
1st-order Sinusold network
2nd-order
3rd-order
Softmax self-attention
d(); (a) w sin m)
Cubic network
Loss curves
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
10000
20000
2000
4000
0
15000
30000
20000
40000
Weights
1/
1.5
14
0.8
0.-
-1.6
-16
-1.6
1.0
-1.6
145
-1.5
1.5
-14
1.5
-0.
0.5
-0.
0.5
B Sigmaid network
Tanh network
0(2) u)
TTx tanh (wi Tx)
o m) [uTxa 2 taanh (u (TTx))
1.0
1.0
1.0
1.0
0.5
0.5-
0.5-
0.5
0.0
0.0
0.0
0.0
0
4000
8000
2000
4000
0
5000
10000
0
30000
60000
0.4
166
1.4
1.6
6-
1.6
1.6
-2A 5
0.4
94 0
0.4
-1.8
1.5
1.5
1.0
1.5
1.5
-1.6
1.0
Figure 4: Learning dynamics in two-layer networks with other activation functions. Each panel shows the loss during training (top), and the first-layer weights right after the first abrupt loss drop (bottom left) and at the end of learning (bottom right). The first-layer weights to each hidden unit are two-dimensional and plotted as black dots. (A) The softmax self-attention model is the same as lin- car self-attention in Figure IF, except for adding the softmax activation function. The training data set is the same as that of Figure IF (B) Two-layer fully-connected sigmoid network, i.e., of xx; u) sigmoid (u") T z). (C) Two-layer molyy-connected sinusoid network, i.e., o(z; u) = sin (u T x). (D) Two-layer collecconnected tanh network, i.c., 6(x; u) tanh (u æ). (E.F.H) Two-layer fully- connected networks with the given activation functions. (G) Two-layer fully-connectee cubic net- work, i... ((æ;; u) (u x²². Details: Except for panel A, the training set is generated by a width-2 teacher network with the same activation function, y 6b <x; u.)) + o(x; uz), x E B E RR. The input is sampled from (0, I). For panels B-F, the teacher network has / = [1. [1,0] . = [0. T. For panels G,H, the teacher network has , [1. 0] u [0. 1]T The number of training samples is 8192. The learning rate is 0.02. The initial weights are sampled independently from No 10 10-12) for panels A-D, and from (0, 0.0052) for panels E-H. The width is H- 50 for panels A-D, and H- = 10 for panels E-H.
In Figure 4, we plot the loss and weights dynamics for two-layer networks with other activation functions. If we Taylor expand o(z; u) around u O, the lowest-order non-vanishing terms the Oth-order, Ist-order, 2nd-order, and 3rd-order terms for panels (A,B), (C,D), (E,F), and (G,H), respectively. In panels (C,D), the networks develop rank-one weights in the early dynamics, since the sinusoid and tanh activation functions are approximately linear around zero. After the first abrupt drop in loss, our theory cannot predict the dynamics anymore because rank-one weights do not generally correspond to embedded fixed points or invariant manifolds for sinusoid and tanh networks. In panels (E,F.G,H), the networks undergo saddle-t-ssaddle dynamics, similar to the quadratic networks studied in Section 5.2. This is because dynamics with the 2nd-order and 3rd- order terms exhibit a timescale separation between units, as discussed at the end of Section 5. Indeed, the weights dynamics in panels (E,,, show that there are one and two units with large weights with the rest being near zero during the intermediate plateau and at convergence, respectively.
25

---

Published as a conference paper at ICLR 2026
B.2
LEARNING DYNAMICS DEEP NETWORKS
A Linear fully-connect net
ReLU fully-connect net
E
Linear transformer
1.0
1.0
1.0
Loss curves
0.5
0.51
0.5-
0.0
0.0
0.0
20000
40000
20000
40000
30000
60000
Weights
0.4
04
0.4
0.4
0.8-
0.8
-44
-04
-34
-0.5
-0.5
0.4
-0.
0.8
Linear convolutional net
ReLU convolutional net
F Quadratic net
1.0
1.0
1.0
0.5
0.0
0
20000
40000
30000
60000
20000
40000
0.8
-0.4
-0.4
404
-04
0.8
0.2
Figure 5: Learning dynamics in deep networks. Each panel shows the loss over training time (top), and the first-layer weights right after the first abrupt loss drop (bottom left) and at the end of learning (bottom right). The first-layer weights to each hidden unit are two-dimensional and plotted as black dots. The training sets in panels A,C,E are the same as those in Figure 1(B.DFF.. The training sets in panels B,D.F split the scalar output in Figure (C,,, into two-dimensional vector output. (A) Three-layer linear fully-connected network. (B) The network has a convolutional linear layer as the first hidden layer and cullecconnected linear layer the second hidden layer. (C) Three-layer ReLU fully-connecee layer and fully-connected ReL.U layer the second hidden layer. (E) One-layer linear transformer, consisting of one linear self-attention layer and two fully-connected linear layers. (F) The network has fully-connected layer with quadratic activation the first hidden layer and fully-connected linear layer as the second hidden layer. Details: The number of training samples is 8192. The learning rate is 0.02. The initial weights are sampled independently from (0, 0.005²) for panels A-E, and from NO0 0.05²) for panel F. The width is H 50 for panels A-D, and H 10 for panels E.F.
network. (D) The network has a convolutional ReLU layer as the first hidden
In Figure 5, we present the learning dynamics of deep networks with various architectures, in com- parison with the two-layer architectures Figure 1. Similar to two-layer networks, the deep net- works also exhibit in panels (A.B) correspond to the embedded fixed points in Equation (7), those in panels (C,D) cor- respond to Equation (6), and those in panels (E,F) correspond to Equation (5). The effective width of the first layer during the intermediate plateau and at convergence is one and two, respectively.
dynamics. The weight structures indicate that the visited saddles
We let the output of the networks in Figure 5(B.D.F) be two-dimensional, rather than one- dimensional as in Figure ICC,E,G), due to considerations of expressivity. If the output is one- dimensional, the second colnecco linear layer can achieve full expressivity with an effective width of one. This would make the second aaddee-t--aaddle transition (if there is onc) different from the first: in the first transition, the effective width of both layers increases by one, whereas in the second transition only the effective width of the first layer increases. We leave this interesting problem to future research.
26

---

Published as a conference paper at ICLR 2026
B.3
THE EFFECT OF SKIP CONNECTION
Schematic of architectures
Deep fully-connect
Deep ReLU fully-connect
no skip
skip
skip 2
linear time axis
Output
Output
Output
1.0
skip
1.0
skip 2
FC layer
FC layer
layer
no skip
no 5kip
0.5
FC layer
FC layer
FC layer
FC layer
FC layer
FC layer
0.0
0.0
25000
50000
50000
100000
Input x
Input
Input
logarithm
1.0
1.0
0.5
skip
skp
Wkpp
0.0
no skip
skip
10
10
10"
Figure 6: Saddle-to-saddle dynamics in deep fully-comnected networks with skip connections. (A) Schematic of three four-layer fully-connected networks: one with no skip connection, one with skip connection that skips one layer, and one with skip connection that skips two layers. The three linear networks are defined in Equation (18). (B,C) Loss curves of linear and ReLU networks with skip connections, plotted using linear time (top row) and logarithmic time (bottom row) axes. All networks exhibit saddle-to-saddle dynamics, with the network that skips more layers learning faster. With small initialization, shallower linear networks learn faster (Saxe et al., 2019). In the network without a skip connection, all four layers must escape the zero fixed point and learn. In the network that skips layer, the second-layer weights remain near zero while the other three layers escape the zero fixed point and learn, yielding dynamics similar to a three-layer network. In the network that skips two layers, only the first and last layers need learn, yielding dynamics similar to two-layer network.
In Figure 6, the networks are four-layer linear networks with skip connections, defined as
no skip: ((x)
(18a)
skip f(x) W4Wa Wi æ + Wi
(18b)
skip 2: f(x) WaWW W W + WiΦ)
(18c)
where the input R2 and the weights Wa E 2R2 W3, W₂ E 50x00 W1 E 5050x
All three networks defined in Equation (18) exhibit saddle-to-sadae dynamics when trained from small initialization, with the network that skips more layers learning faster. This because weights in the skipped layers can remain near zero while weights in other layers escape the zero fixed point and learn. When the skipped layers are effectively unused, the network behaves like a shallower net- dynamics. Furthermore, since shallower linear networks learn faster (Saxe et al., 2019), networks that skip more layer also learn faster. The dynamics in ReLU networks with skip connections is similar.
work consisting only of the unskipped layers and exhibits saddle-to-sadee
27

---

Published as a conference paper at ICLR 2026
C ADDITIONAL DISCUSSION
ReLU activation function. Because the ReLU activation function is piece-wise linear, we con- jecture that ReLU networks trained from small initialization have timescale separation between different directions, similar to the mechanism in linear networks. Indeed, prior studies have found that the direction that maximizes the correlation with the output grows the fastest, that the direction arg m -1 (ReLU(u z)y). This was known as quantizing (Maennel et al., 2018), condensing (Luo al., 2021), random feature amplification (Frei al., 2023a), and studied in many other theo- retical works on the learning dynamics of ReLU networks (Le & Jegelka, 2022; Petrini et al., 2022; Timor et al., 2023; Kou al., 2023; Glasgow, 2024; Min et al., 2024).
Reduction of high-dimensions learning dynamics. The invariant manifolds in Theorem 3 can also be useful for reducing high-dimensional learning dynamics. Whenever network evolves near an invariant manifold during a learning phase, Theorem 3 suggests that the full learning dynam- ics may be well approximated by the dynamics of a narrower network. For example, if a width-2 homogeneous network has nearly proportional weights, 01 = 702. dynamics in 0₂ and 02 may be well approximated by lower-dimensional dynamics only in 0,. Many prior analyses have suc- cessfully reduced high-dimensional learning dynamies for individual architectures using similar idea, including deep linear networks (Saxe et al., 2014; 2019; Advani et al., 2020), ReLU networks (Phuong Lampert, 2021; Sarussi al., 2021: Lyu et al., 2021; Frei et al., 2023b; Tsoy & Kon- stantinov, 2024; Zhang et al., 2025a), and self-attention (Zhang et al., 2025b; Yüksel et al., 2025).
Distributed localized features. Identifying which type of fixed points network visits among Equations (4) to (7) may be of interest to representation learning (Hinton et al., 1986; Elhage et al., 2022) and pruning problems (LeCun et al., 1989; Frankle & Carbin, 2019; Gromov et al., 2025). The fixed points described by Equations (4), (6) and (7) correspond to networks with distributed, nonlo- cal, or polysemantic features, while fixed points described by Equation (5) correspond networks with localized or monosemantic features. Networks with localized features and simplicity bias can be more easily pruned. In Section 5, we showed that the timescale separation between directions due to data distribution gives rise to distributed features, while the timescale separation between units gives rise to localized features.
Technical future directions. The goal of this paper is to establish theoretical framework and vali- date its predictive power. To serve this goal, we have prioritized the completeness of the framework, times relying on heuristies and empirical observations. Several interesting technical questions remain open. First, how close must a point in weight space be to an invariant manifold in order to approach fixed point on that manifold before leaving the manifold? Quantifying this could enable rigorous proofs of saddle-to-saddle dynamics in a wider range of architectures, extending beyond di- agonal linear networks (Berthier, 2023; 2026; Pesme& & Flammarion, 2023). Second, is the sequence of saddles visited during training Markovian? That is, can the next saddle be inferred solely from the current one, independent of earlier ones? In dynamical systems literature, it has been shown that certain saddle-o-saadle transitions are Mon-Markovian (Bakhtin, 2011). 1).
28

---

Published as a conference paper at ICLR 2026
D GRADIENT CALCULATIONS
We write down the gradients of the weights 01s for the network defined in Equation (1). We denote
H
f(z) 9oun
where Ç Σ (am (z): u,)u,
(19)
The variables have dimensionality
E R.. . E... (( E RNNON. SER
Recall that the training loss defined as
P
P
-= P -
Σ
Ely f(x)))
(20)
P
#=1
p=1
Using the chain rule, the gradient flow dynamics of vz can be written
OC In,
1
P
Oc
Hlp oc
1 P
de, aç
E
blga (20); u.)
(21)
On:
The gradient flow dynamics of u, involves matrix-by-vedt derivatives, which generally require tensor notations. To avoid introducing tensor notations, instead write the gradient entrywise, for n 1,''" Nu.
OC
P
OC
Pep Oc
P
dplgin u.)
Oe, 0C
v
(22)
Ouinn
P #=1
Duin
Ouin
p=
29

---

Published as a conference paper at ICLR 2026
E
EMBEDDED FIXED POINTS
Here we prove Theorem 1.
Proof. We have that 8 11) is a fixed point of the gradient flow dynamics of the width- 1) network, that is
7c
OC
0f
0,
0,
1= H
(23)
00: 0,=0;
(x)
00;
We denote
H-1 (z)( uz)03.
-- Σ
(24)
The input-output map the width- (H 1) network f*(x)) = Ioa (())
We prove the four statements one by one.
(i) For any o. analyze Equation (4).
First, the width-H network implements the same input-outpur map as the width-(H - 1) network, that is f(xr) Doa where
H
5 (x) uy)
1=1
H-1
= ((gm(()) (x): un wn + ol9m(z): +
Σ
(x)) u,)
j=1jf
H
7o 7o)
Σ
((((())
j 15
p(gmn (z);
Second, we calculate the gradients of the weights in the width-// network. For the units with unmodified weights, the gradients are the same as the width- (H 1) network
dc 0f(x) 0, = 0f(x) 00, (x) a0;
dc (f) = j 1..... H. 1, i
For the i-th unit, the gradients can be expressed using Equations (21) and (22) as
Oe, Σ
dl
P
(g (x u.)
Oc
P
agnn (x
0j = 0, oc
P=1
1
aplgm (pp)): u:) ŌP,
ún
e2
P
Ougn Oc
1
P
Jolgm (x))) (z)))
= -- %v)
P 1=1 Ousn aç*
(1 %o i
0, 1, No.
30

---

Published as a conference paper at ICLR 2026
Similarly, for the new H unit, the gradients
Dbp
8z = 0,
P
1=1
1
T d (xx));
Dep. ōc*
UHH 9o
Yo 0, 1, Nu.
P
Ouin
(ii) For & such that V/z, (2;; Ugrro) O, analyze Equation (5).
The width- H network implements the same input-output map as the width-(H - -1 network, that is (ax - Jou (C) where
H Σ 2
H-1
pf u,; = ((gi(()); Uxero ) + Σ (a (z): = 5*
Because the first (H - 1) units have unmodified weights, their gradients are the same those in the width-(H 1) network, which are zero. For the new H-th unit, the gradients can be expressed using Equations (21) and (22)
Lann (x Uzero
T
O,
P
OC
p=1
1
O0lgin uT)T
Obp
üH
0,
n 1, Nu-
P
OuH,n
i If é(; u) is degree-1 homogencous in u, analyze Equation (6).
The width-/t H network implements the same input-output map as the width-(H 1) network, that is f (xx) Jou (C) where
# 5
Σ ( 1=1
H
(()) (() - notoo +
Σ
((n(());
u3)03
J=1,
( + -1)(1-ur)))))
Σ
((mn(z):
(=)(());
S
Because the units with indices j = 1,... ......,A И 1.j * have unmodified weights, their gradients the those in the width-(H 1) network, which For the i-th
31

---

Published as a conference paper at ICLR 2026
unit the gradients can be expressed using Equations (21) and (22)
Hlp 8c*
0,
(g a xx)): :);;)
0,-
0,
T 00(gin (I u.) Hll
P #=1
Ōuin
(1 70 (2 Ol,
- Σ
Ousnm
O5
p
(1 Yu 7o 1
0, 1,.., Nu.
By Euler's homogeneous function theorem, the partial derivative of the homogeneous function (12; u) has the following property
0Φ(2: u)
0Φ(2 22 u)
n = 1, ... Nu
(25)
Oun
Oun
u=
Thus, for the new Hith unit, the gradients are
Olp
06p (()
0y
p (xx): :
03 0,
oc
P
Oc*
0 ppp)): u OP, T
üHH
P
H
duhtn
O5
P
do(gi (ww): Ya uu) 06p
U
Ouia 00
To u O, n= 1, Nu.
(iv) If ((2;; u) is linear u, we analyze Equation (7).
The width-44 H network implements the same input-output map as the width-(H1 1) network, that is (xx = your(S (C) where
1
5 Σ
(x)( UN
(z);
#-1
H
H
H
(x); );
Σ
You u
Σ
Yor 2i
+
Σ
( (x):
0j Ya,
Σ =²=1
Noge Vzz
1=1
H Σ
#-1
Tu Nop (0(gn((()))) 2e +
p( (x); uz)03
Σ
Yag Tu Mlginnlnnn (
dggin (2);
5
32

---

Published as a conference paper at ICLR 2026
For 1, .. . the gradients can be expressed using Equations (21) and (22) as
1
dl,
0;
Σ
(f (s))); u;)
P
Oc
1
Σ
1i
Yuu Σ
H-1
To, m3
(gn (wx); (e)):us ui)
Ol,. dç
Ouin
1=1
5=1
P
0 Gin (x ): ui)
Olp OC
H
(elgin (x, TT ael
Σ
T
v
Yu.
Pos
0j
Ousn
Jujn
Oc
P=1
j=1
k=1
Uij
+
Yog U5,
j=1
= -,
n
Nu-
We leverage the degree-1 homogeneous and additive properties of linearity, which yields
M-1
H-
z:
Yu,
ui
Yu. (=; u,))
(26)
1=1
Thus, for the new H-th unit, the gradient for vtr
P
Ol
vH =
Σ
( (zx)): uH)
P
Oç
1
P
Iin (xn( Σ
Yuu Yu; un
Oep
P
O5
1-1
Hly Mann (x));
Yuu
P
05
1=1
H-1
Yuu oj
For d(2; u) that is linear in u, the partial derivative, u. Thus, the gradient for uy is
@d(zzu) Dan
is function that does not involve
1
delgi ( T dl
üH
P
UH
JuHn
0C
H
P
dolgm (20 (xpp): uH) dgp
Yuu
Ouhnn
Oç
p=1
H
Σ
Ye, i
1=1
0.
n
N
Hence, in all the four cases, the weights of the width-If network given by Theorem satisfy that the input-output map is the same as the width-(H 1) network and the gradients are zero.
33

---

Published as a conference paper at ICLR 2026
F INVARIANT MANIFOLDS
F.1 DEFINITION OF INVARIANT MANIFOLDS
We provide the definition of invariant manifolds used in this work.
Definition 2 (Invariant manifold).
M R is an invariant set under the dynamical system
h()) if for any ((0) On E M have ( M for all E R. An invariant set M C R" is
an invariant manifold if M has the structure of differentiable manifold.
In this work, we focus on particular class invariant manifolds defined via the constraint ((0) O; see Theorem 3. For defining invariant manifolds in more generality, we refer the readers to Wiggins (2003).
If trajectory heteroclinic orbit in the dynamical systems literature (Bakhtin, 2011). The saddle-to-san saddle transi- tions in our setup can be viewed as heteroclinic orbits.
invariant manifold connects two distinct fixed points as - +oo,, it is called
F.2 PROOF OF INVARIANT MANIFOLDS
We here prove Theorem 3.
Proof. We prove the four statements one by one. Recall that 0, is defined in Equation (1), which is the stacked second-layer layer and first-layer weights in the i-th unit
U
=
26
(i) For any o, two units have equal weights: 0, = =0;-
We write down the dynamics of the difference (0, - 0,) and substitute in 0, O,, Using Equation (21), the dynamics of (v; wj) given by
1 P
Oe 0.
cu 01)
Σ
(2 (lgm(( u;))
aç
Using Equation (22), the dynamics of (Ui,n Ujn) for =,... N₂ is given by
d dt (Ui. Uj.n) = P
P
Odlin (2p)) u:) adh (2.)); Ougz
Ol, ac
0.
v.
= 0.
Oui
(ii) If 3 Unero such that Vz, ((2; Unero) = 0, a unit has zero weights: v, = 0, u4 Uzorn-
Substituting v; 0. u. Usero into Equations (21) and (22), obtain
P
Dlp O,
ú, = P
Σ
Lgg (1) (Ipp): Usero)
Oc
==1
1
u:)
OP,
0,
1,..,, Nr,
Ougn
ac
(iii) If (T)(2; u) is homogeneous in u, units have proportional weights: 0, =70,7
34

---

Published as a conference paper at ICLR 2026
Using Equation (21) and the degree-1 homogeneous property, by E F, ((2 (0(2; u)) the dynamics of (v; - -ywu) is given by
dt (v; (1;=)
d
1
Σ ==1
(T(n(())) - (z0 u,)) T 0l,
(((m(Ix): -ainn(())))wwa)p)p))p T 06,
P
ōc
0.
Using Equations (22) and (25), the dynamics of (wi,n - mm forn =
1on==1,.. 1,... Nu is given by
1
Poblei((3,,.11 ub))
Ola
d (1in -)
v:
y0,
P
Jug,
aus,
oç
Holgin xx)): u.) Oe,
u1 - y0,)
P
Juin
0
#=1
O.
(iv) If ((2; u) is linear in u, any number of units have linear dependence: 0, = Eji 7z0
We leverage the degree-1 homogeneous and additive properties of linearity, which yields
(z; u.) Σ Σnd((z = 2; Σnau U
(27)
j
Using Equations (21) and (27), we obtain the dynamics of (1; Ejjt 75003)
T
d
dea
0, Σ 1505
Σ
Σ
df
P
-
OC
j
1=1
OPp
Iin (m((pp): uu
Σ Cny
P
Oc
p=1
Slp Oç
Mg (2x)):))
P
0.
For ((2; u) that is linear in u, the partial derivative @o(=;u) is a function that does not involve
Sie
u. Thus, using Equation (22), obtain the dynamics of (ui.. Ejjt Uz ) for
.. Nu.
d Ui, Engugin
1
Oe,
0. Oc
Ouin
jti
F.3 EXPRESSIVITY ON INVARIANT MANIFOLDS
When the weights of a network lie on an invariant manifolds, its input-output map is expressible by the architecture with fewer units than its actual width.
35

---

Published as a conference paper at ICLR 2026
(i) For any o, when two units have equal weights, 0, ,, remove the i-th unit and multiply Vy by 2, obtaining a network with one less unit that expresses the same input-output map.
(ii) For & such that Vz, (z;; Uzero) 0, when a unit has zero weights, 0, Uzeros we can just remove this unit, obtaining a network with one less unit that expresses the same input-output map.
iiii For ((z; u) that is degree-1 homogeneous in u. when two units have proportional weights, 0: 70,, we can remove the i-th unit and multiply v by (1 72), obtaining a network with one less unit that expresses the same input-output map.
(iv) For (0;; u) that is linear in u, when there is a linear dependence 0, = Ej 750 we can remove the i-th unit and modify the second-layer weights in all remaining units as follows
o Dj + 7/j
Σ Vy. s'
(28)
This new width- -H network expresses the same input- map as the original width- network
H Σ j=1Øøi
6f ( ( 20
Σ
Hgg (x)
Uj + Ys
E vy
jti
8'
Σ
cegn (x)) + Gin [in(x)); Σ T3
Σ Yy
8'
Σ
(g (a) uy) + & (gin (x)); ue;)
H
Σ
(z)
F.4 THE EMBEDDED FIXED POINTS ON INVARIANT MANIFOLDS
The set embedded fixed points lying on the invariant manifolds given in Theorem 3 is subset of the embedded fixed points given in Theorem 1. Here we specify the subset of embedded fixed points that lie on invariant manifolds.
(i) When To 1/2 in Equation (4), the embedded fixed point has uy = u,,, UH of /2, which on the invariant manifold of 0μ 0, Theorem 3())
(ii) For & such that Vz, 6(2; Uzoro) 0, it is clear that u Usero, Uh is on the invariant manifold in Theorem 3(i).
(iii For ((2; u) that is degree- I homogeneous in u, when 7 = 7T/((1 + + Equation (6), the embeded fixed point has
U = Tre Ui = To
V; = Vj.
(29)
1 Yu
which is on the invariant manifold of 0g 0, in Theorem ((ii).
(iv) For (12; u) that is linear in u, let us first rearrange Equation (7) by substituting v; o/ Yue Un into UH = - Yo, of and obtaining an expression U in terms of {v,
H-11
#
H
UH
Yop (wx + Yu,UH) VH)
No
(30)
1 - Σ11 1==1 To, Yug im1
1=1
36

---

Published as a conference paper at ICLR 2026
When Yv. /vus(11 + Σ
H11 2j=1 Yo, in Equation (7), the embedded fixed point has
11
U
Yue Mi:
(31a)
1m1
1
VH
Yop 82
11
1
j=1
Yos Yus i=1
H-1
7v
H-1 12
H 1-1
2 Tu5
1
jj=1 14 75p
j=1
H
Tu, V/
(31b)
which is on the invariant manifold of 0μ Nu.
H-1
0, in Theorem 3(iv).
37

---

Published as a conference paper at ICLR 2026
G DYNAMICS OF LINEAR NETWORKS
G.1 GRADIENT FLOW EQUATIONS
We derive the gradient flow equations given in Equation (9).
For 1, H, the gradient flow dynamics on squared loss, Ely. 9)
1 - is given by
P OC
0.
P
(z()
Ovi
p=1
Up
W 2/, 2 us
y W Σ 2
OC 0f(zm)
ú
P
0f(z,)
du.
#=1
P
1
W 22
#=1
Σ 2p2 WT
P
Recall that the data statistics are defined as
Σy | E Up
Σ.;
1 p
Σ 2
Substituting in the data statistics, obtain the gradient flow equations in Equation (9), which are
Ui i= 1...... H. yE WE uis d (E WE
G.2 PROOF OF TIMESCALE SEPARATION
We here prove Theorem 4, that is the timescale separation between directions in a linear dynamical system.
Proof. The linear dynamical system in Equation (10) can be written
0 ya
Σ
i = MO,, where M
yz
0
(32)
u
The symmetric matrix M D positive eigenvalues, negative eigenvalues, and (N, + N2, 2D) zero eigenvalues. The nonzero eigenvalues are the singular values of Eyz and their negative
M
8k
qk Pk
M
qs Tk
8k
9k Tk
k= 1......, D.
(33)
The exact time-course solution Equation (10) is
(1)
v; ui
(t)
9k rkk
+ buze
Hut
-k -Tk
+ Ex,
i= 1, H,
(34)
38

---

Published as a conference paper at ICLR 2026
where the constants Cki: dys projections of the initial weights onto the eigenvectors with nonzero eigenvalues, and & is the initial weights projected onto the eigenspace with zero eigenvalue
4k (0) + r u: (0))
(35a)
2
1
by
O vi(0)
rk u, (0)):
(35b)
2
&. = (0)
Cki
y rk
+
bbi
qs
(35c)
1m
Recall that the projection matrix P. defined in Equation (11), corresponds to the rank-r subspace spanned by the top singular vectors. The projection of the weights this subspace has E norm
q [*k
PO.1| =
Chee
2
e011 PO;
(36)
Rs
k=1
The time takes for ||PO||| reach O(1)
T
In [PO(
(37)
At time T, the projection of the weights on the null-space of P has lg norm
(qx)
[ - ()0,) =
Σ
C
bare
9k
E:
Tk
-Tk]
k=r+1
A=1
D
c 22nnT +
2s + 11E/112
Ai
kmr+1
km
D
< T
2
+
Σ
62 + ||&
ki
kmr+1
km
PO.(0)]
1 2I
[|(I - P)0
II(I [II-P) 2e01
(38)
[P)]
Because the each entry of the initial weight 0; (0) is independently sampled from.N(0) e?), the norm [00( Oc) and the ratio (
[P]] O(1). Thus, we have
[I (T)) O( (I- P))I 2or =0 00 2r+1
(39)
Hence, time T, [())) 0(), while || (T)I O ((e2-5011/51) is still small.
G.3 FIXED POINTS OF LINEAR NETWORKS
In Lemma 6, we specify all the fixed points in the linear network learning dynamics in Equation (9). Lemma Denote the eigenvectors of the symmetric matrix Ey yz as e₁ E RN,, k
1.... N., arranged in descending order of their associated eigenvalues. There min(N, NN, Nu) nonzero eigenvalues. The sufficient and necessary condition for a set of weights to be a fixed point of the dynamics in Equation (9) is
most D
Σ v.u.
Σ exee
E span(ex)eeA... u; E span {E KEA." (40)
1=1
KE
where Ar is set of indices Ar C (1,2,... .., D), |A =r.
39

---

Published as a conference paper at ICLR 2026
Proof. The proof can be found in the seminal work by Baldi & Hornik (1989) or follow-up work (Kawaguchi, 2016; Lu & Kawaguchi, 2017; Yun et al., 2018; Laurent von Brecht, 2018; Achour et al., 2024).
The dynamics near a fixed points defined in Equation (40) is approximately linear dynamical system, for i 1,... H.
8; EEyz W us
Eye Σ
exei
EeekEyeh Eg Eze
u, E yy Us
(41a)
kEA,,
T
ú; = Exyz W
Us
Enys
Σ
ExeR Enz 2i Exz
Us ΣT Ui.
(41b)
REA,,
where Eng is Eyz projected onto a rank-(D r) subspace
Σy = Σ
exey Eyz-
(42)
kg.
When defining A, in Lemma 6, there are (2) possible choices of r indices out of D, assuming the eigenvalues are distinct. Each choice produces different input-output linear map. Thus, there are (2) embedded fixed points of effective width in linear networks, if we count fixed points by the distinct input-output maps they implement. When a linear network undergoes saddle- to-saddle dynamics and approaches fixed points of effective width r 0, ...... D sequentially, determining which one of the (2) fixed points it approaches at each stage non-trivial open problem. Even in diagonal linear networks, specifying the sequence of visited saddles requires non-trivial work (Berthier, 2023; 2026; Pesme & Flammarion, 2023).
Remark 2.
40

---

Published as a conference paper at ICLR 2026
H DYNAMICS OF QUADRATIC NETWORKS
H. GRADIENT FLOW EQUATIONS
The gradient flow dynamics of Equation (13) trained squared loss, ((v.0) 2 -g) is given
by
1
OC
am(m))
P
(z()
H
Vp
Σ
U Z
ui zu
P
j=1
P
H 11: Σ
P
1
U1 Zz
Σ
uu
Zi us:
P
P
1=1
==1
w=1
OL
4.
P
(z
Ous
W=1
P
H
Up
Uj Un
P
#=1
j=1
P
1
2
P
U/
u; Σ
V2
P
Σ
m. Zn ug.
p=1
1=1
m=1
Denote the data statistics as
1 P P p=1
Ege
1
Σ
Uu Zur
Ezz
Σ
vec (ec(Zz,) (ec
(43)
P
#=1
The dynamics can be written as
è = u Egeu, U4 Σ ey(u, @ T Ezz(u, @ uu).
(44a)
ú 2z; Du
20
(u 8 uu)) Σzz(u @ Ip),
(44b)
where @ denotes the Kronecker product.
H.2 DERIVATIONS FOR TIMESCALE SEPARATION
We here provide the derivations for Proposition 5, that is the timescale separation between units in quadratic dynamics.
To reduce clutter, we omit the index i in Equation (14) for now; will put it back when need it. We study the dynamics
i Ey á Euz
(45)
Step I: reduction one-dimensional dynamics.
Denote the eigenvalues and eigenvectors of the symmetric matrix Eyz
ΣgzTh SkTK, == L.... D.
(46)
We change variables by projecting the (orthonormal) eigenvectors of Eyz.
O
V2 rh u == I=,...,,
(47)
41

---

Published as a conference paper at ICLR 2026
where the factor of 2 is for convenience only. Since time has arbitrary unit, can let t t/2. Then, the dynamics of u and v can be expressed in terms of the new coordinates d1,' ap and v,
é
(48a)
à = U8
h ==
,.,,...,1
(48b)
This set of equations admits a conservation law,
d
8²
o
2u
Sk
20
Σ
syas
(49)
dt
Thus, their difference at initialization is conserved throughout training
v(t)² Σ
=00)
a (0)
(50)
We also notice relationship between the ok
dam
Sm Am
am (t)
In (t) ax (0)
In
(51)
da
0A
8m
(0)
Thus, different (t) can be expressed in terms of each other,
am (t)
8k/8m
(t) = =a(
(52)
am (0)
Using Equations (50) and (52), and defining
(t) = (t)
(53)
(0)
we can express vtt) in terms of Tm (t),
w(1)² = v(0)
Σ k=1
(0)(0)((TT))) (1)2sx/5m - 1)
(54)
At this point m is arbitrary; we will set it to a particular value shortly. Substituting Equation (54) into Equation (48b) and using Equation (53), we obtain one-dimensional and separable differential equation for T.
fm mm sign (v(0) 8. Tm
x(0)2
Σ k=1
(0) ( 7m
25t/ /sm
(55)
with initial condition nm (0) = 1. Reducing the (D + 1)-dimensional dynamics in Equation (45) to the one-dimensional separable differential equation in Equation (55) may be of independent interest.
Step 2: bounding the growth time.
Let us choose m to maximize sign(u(O)) 8m If (0) positive, the chosen m maximizes 8m: it is negative the chosen minimizes 8m (and thus maximize |sm], assuming that there both negative and positive eigenvalues). either case, max, ($t/8,) = 1. And with this maximization, the prefactor becomes smm)-
We are interested in the time it takes for am (L) to become large; that happens when L ~ 1/am/00 (0), which is large for small initial conditions. The time it takes for that to happen, denoted Linat- is bounded by the time it takes for Tm (t) to go to infinity, giving us
dx
timaa < too
(56)
Ism
x v v0)2 v
+ ER_ (0) 25uu/sm mmmmm 1)
42

---

Published as a conference paper at ICLR 2026
So far we have ignored the dependence unit, i. However, each has associated with it different foo. Let us use too to denote the different times, arranged in increasing order.
o fo < ... < 10,
(57)
We will also add subscript i all the other variables well. At time 6,,, we know that Tm,1 1) is large, but what about Tm. (loc,,) for i >> 1? That is given implicitly by
(Pmaa)
dm
Innal
(58)
T (1) + Eb aR (0) 20 20n/smms 1)
Note that 5m acquired a subscript i, because it is either the maximum or minimum eigenvalue, depending on the sign of u0))). Rearranging terms slightly and performing small amount of algebra, this can be written as
dı
&mm (0)l(/oo, - Annal)
(59)
(Pana))
T / (T(
where
u0) (0))
(0, (t 2.20k//mm 11) 1) (1)
(0):
(x) π²
+
(60)
bymo
We can bound the left hand side,
dm
dr π
(61)
Fmmellaa) TT /1+1+ (T)
VI+ V min,i
(mmal
+ 1p min, Tm, (funa)
where min the minimum value of (T(T)) Inserting this into Equation (59) then gives us
1
Ty. Tm. (Liial)
622)
/1 min. am (0) timull
To determine the size of the right hand side, we first note that because the spread in initial condi- tions is O(1), the relative spread in the too is ((1). Second, the loc, scale inversely with initial conditions (see Equation (56)), whose typical size we denote Consequently,
1 0
8m,, a Bo (0))()xo.) (imal)
(63)
from which it follows that nn, (f O(1). And so, given the definition of n (t) in Equa- tion (53), we have am (finat) ~ e. Thus, when the variables associated with one of the units become (1), the variables associated with all the other units O(e).
43

---

Published as a conference paper at ICLR 2026
I IMPLEMENTATION DETAILS
Videos the learning dynamics in Figure are provided in the supplementary material. For all models, sample the initial weights from M/ <²) and train them with squared loss
C Σ My (xx))
11 12x
(64)
Linear fully-connected network (Figure IB).
The network is defined as
f(x) = Σ v.u. a,
where v, u, x E IR, 50.
(65)
The training set (a Yo P x=1 generated as
N 1 1
Yn
W Tye
Zy
Here P 8192, c 10-6, and the learning rate is 0.01.
Linear convolutional network (Figure IC). The network defined as
f(x)
Σ Tea
U12
422
0
where x E IR$,, H. = 50.
(66)
0
142
Here the first layer is one-dimensional convolutional layer with kernel size 2, stride 2, and padding We set /1 50. The pytorch code for setting up this layer
torch nn. nn.
out out_channels=55,
Bernel_aize==
stride=2,
padding=0,
dilation=1,
groups=1,
bias=False)
The training set {æp: ypt / == is generated as
0
0
0
0
1
= w" Iy Du ~ N
0
0
w"
V5
0
Here 87-2, 10 and the learning rate is 0.01.
ReLU fully-connected network (Figure ID).
The network defined as
"1
f(x) = Σ vz ReLU(u)
where U; E R, 2, x0 E R.? H = 50.
(67)
The training set is an orthogonal input dataset used in Boursier et al. (2022, Figure 3) and Zhang et al. (2025a, Figure 4). It contains two data points
1/1 1,
0.5
-1
22
1/2
1.
44

---

Published as a conference paper at ICLR 2026
Here P= c, = 10-6, and the learning rate is 0.01.
ReLU convolutional network (Figure IE). The network is defined as
Σ Jos 2/2) ReLU ((rd
f(x)
442
where IR,, H 50,
(68)
0
u:2
which is the same as Equation (66) except for the ReLU activation function. The training set fa Uy |p=1 is generated as
8/2
w"
20
V5
It contains four data points
2 0
0
31
0000
22
0
3y
2/2
24
0
2
Here P= c 10-6, and the learning rate 0.01.
Linear self-attention (Figure 1F).
The model is defined
H S(X) x + Σ 1XXXKTQ..
(69)
where
V ((D+1)x(D+1) K,, Q, RR((D) X E (R(++I)((++),
(70)
Here D is the embedding dimension, N is the context length, and Rt is the rank of each attention head. We train the linear self-attention model on an in-context linear regression task (Garg et al., 2022; Zhang et al., 2025b). The training set {X u. guglp=1 is generated as
x, = Ea.1
Ln.2
Cu.N w. Kn,N
Ep..
Up.g = w Fpg.9
w Lp,1
Wp Fx,2
0
and
Fn.n 4. I).
Wp ~ NNO I),
1 n. 1= ...
Here D= = 2, 32, R H 10, 8192, 0.005, and the learning rate is 0.02.
Quadratic network (Figure 1G).
The network is defined as
f(x)
Σ vi (u. xx)
where is u. IR?
(71)
The training set {æp, p is generated as
0 "
u p.p) + un T xzp 8/
Fy. N( I). i
0
1
Here H 10, P= 8192, c 0.005, and the learning rate is 0.04.
In linear self-attention and the quadratic network, set H 10, not 50 the other architec- tures, because a large makes the plateaus very short for these two architectures. This effect was discussed in Section 6 and validated with simulations in Figure 2A.
45


Reviews:
[{"review_id": "a6ol76OYsN", "reviewer": "Reviewer_9Lv2", "summary": "This paper presents a unifying theoretical framework explaining simplicity bias in gradient descent dynamics through what the authors call saddle-to-saddle dynamics. The authors argue that across diverse architectures—fully connected, convolutional, ReLU, quadratic, and self-attention networks—training often progresses by traversing a sequence of saddles, each corresponding to increasingly complex solutions. The work generalizes earlier analyses of linear networks, introducing formal results on embedded fixed points and invariant manifolds to explain why networks successively recruit additional “effective units.” The paper also provides clear implications for how initialization, network width, and data distribution affect the number and duration of learning plateau.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper offers a broad, architecture-agnostic framework that connects dynamical simplicity bias phenomena seen across many model types. The idea of interpreting learning stages as saddle-to-saddle transitions is elegant and theoretically grounded.  \n2. The analysis of fixed points, invariant manifolds, and timescale separation (linear vs. quadratic cases) is mathematically sound, and the simulations (Figure 1 & 2) convincingly demonstrate the predicted dynamics and validate the theory’s implications for initialization and data statistics.  \n3. Despite the depth of the mathematics, the exposition is clean, well-structured, and includes intuitive visualizations that aid understanding.", "weaknesses": "1. The simulations, while well-chosen, are limited to small synthetic examples. It would strengthen the paper to show whether the predicted stage transitions are visible in real-world or larger-scale training runs.  \n2. Some proofs depend on idealized gradient flow dynamics and homogeneity assumptions that may not hold under practical stochastic optimization with large step size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9fbfp4taiB", "reviewer": "Reviewer_Peza", "summary": "The paper argues that saddle-to-saddle dynamics can induce an implicit bias toward simpler solutions across diverse architectures. Leveraging permutation symmetry, the authors show that a saddle point in a narrower architecture remains a saddle point when extended to a wider one, and they identify new conditions under which such saddles can exist in larger networks. Furthermore, the paper demonstrates that, under these conditions, the dynamics evolve within an invariant manifold, suggesting that saddle-to-saddle dynamics correspond to transitions between invariant manifolds via perturbations. \n\nMore specifically, the authors unify multiple architectures within a single framework and exploit the fact that permutation symmetry is shared across them. Their analysis of saddle-to-saddle dynamics also reveals a benefit of increasing the hidden dimension HHH in self-attention architectures—a contrast to the fully connected case—which they further validate empirically.", "questions": "## First Question\nDo the authors have corresponding results for all deep architectures in Figure 4?\n\n ## Second Question\nFrom the analysis related to Figures 2A and 2B, authors argue that the difference between linear attention and fully connected networks arise from the quadratic dependence of $\\phi(x, u)$ on $u$. However, I believe the observation of Figures 2A and 2B arise from the depth and initialization. \n\nCan the authors repeat the experiments of Fig 2 A and B for 3-layer diagonal-ish linear network \n\n$f(x) =  \\sum_j \\sum_{i=1}^Ha_{ij}b_{ij}c_{ij}x_j$ \n\nusing the same initialization as the self-attention network in the paper? Here, $a_{ij},b_{ij}, c_{ij} \\in \\mathbb{R}$ and $x_j$ are the entries of the input. The model, while similar to a quadratic network, does not have $u^2$ dependence. It only shares that there are three products of parameters ($abc$ instead of $vu^2$) to express $f$.\n\nGreater effective depth—i.e., more multiplicative parameters—typically increases sensitivity to initialization, as suggested by the more Heaviside-like saddles observed with larger parameter products (e.g., your Figure 4). Notably, the self-attention and quadratic networks use a larger initialization $\\epsilon=0.005$, compared to $\\epsilon=10^{-6}$ for fully connected and convolutional networks; I interpret this as a practical choice to escape a longer initial saddle plateau caused by more products of small initial weights (three for quadratic/attention vs. two for fully connected/convolution).\n\nWhen depth-induced sensitivity is combined with a larger (yet still “rich-regime”) initialization, increasing $H$ may simply allow the model to sample slightly larger initial effective parameters, thereby shortening the transient around the saddle. If a 3-layer diagonal-like network exhibits similar sensitivity to $H$ and initialization as observed in the self-attention networks, the analysis should emphasize the role of multiplicative parameterization rather than attributing the phenomenon primarily to the activation form only (i.e. a special case). If no such effect is observed, the current interpretation is convincing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a unifying perspective on the widely assumed simplicity bias underlying saddle-to-saddle dynamics across various architectures and, importantly, elucidates its connection to permutation symmetry. This formalization and unification constitute the core contribution of the work.\n\nAlthough the analysis is based on a heuristic scenario, it is consistent with both experimental observations and existing understanding of saddle-to-saddle dynamics. The analysis further reveals an unexpected benefit of increasing $H$ in self-attention architectures.\n\nThe paper is clearly written, and the illustrative examples effectively convey the key ideas.", "weaknesses": "While I fully agree with the reasoning presented in Section 5 and consider it a sufficient contribution, several steps remain heuristic. \nFor instance, the transition between invariant manifolds is not thoroughly explored: while the paper studies the existence of invariant manifolds and the dynamics within them, it offers limited explanation for why the dynamics should adhere to these manifolds in the first place. Proposition 5 states that if one variable takes a larger value, the others must be small, yet it does not clarify why one should dominate in the first place. The argument is intuitively consistent with the rich-get-richer dynamics of layerwise structures, but making this connection more explicit—especially in settings where many neurons can fit the target — has been a central focus of works such as Kunin et al. (2025), which the paper also cites. \n\nWhile I acknowledge that the analysis in Section 5 is restricted to two-layer networks, I believe the current analysis can be potentially misleading, particularly regarding Figures 2A and 2B and the conclusion that the saddle-to-saddle mechanism differ for different architectures. As the authors conjecture in line 354, I believe that the degree or effective depth — the number of times the learnable parameters are multiplied to achieve $f(x)$ — is the main cause of the sensitivity to $H$ and initialization: a 3-layer network with linear dependence on $\\phi$ would likely exhibit similar sensitivity to $H$ and initialization as the 2-layer self-attention networks.\n\nThe intuitive description of Proposition 5 (line 331) and the conjecture on depth (line 354) seem to suggest that the number of times learnable parameters are multiplied — the effective depth — is the key factor, and that the self-attention and quadratic networks simply obtain an additional effective depth from $\\phi \\propto u^2$. While I believe the analysis is correct for two-layer networks, the authors could improve clarity on what specifically causes the sensitivity to $H$ and initialization, especially as the authors already discuss the limitations of Section 5 regarding depth. Please see my question 2 for a verifiable experiment. \n\nThe empirical section could also be strengthened by providing result for all architectures (convolutional, attention, quadratic) in Figures 4, to make it consistent to Figure 1. This is important as the theory claims to unify dynamics across diverse deep architectures.  Additionally, one or two experiments on real dataset would significantly strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9d8uFGbDZ", "reviewer": "Reviewer_S6X7", "summary": "This work attempts to explain the emergence of saddle-to-saddle dynamics in\nthe training of neural networks by gradient descent via invariant manifolds. The\ninvariant manifolds in question correspond to the number of effective units of the\nnetwork. In order for saddle-to-saddle dynamics to emerge, the authors argue\nthat initialization must be close to an invariant manifold on which zero loss is\nunattainable and that the escape path from saddles closely follows invariant\nmanifolds.\n\nConcretely, a result on the embedding of local minima in smaller networks\nas saddles within larger networks is proven, extending [7]. Theorem 3 provides\na group of properties invariant under gradient flow. The theoretical results\non saddle-to-saddle dynamics focus on two-layer linear and quadratic (in the\nweights) networks. Timescale separation is argued to occur in linear networks\ndue to the distribution of data and in quadratic networks (analysis restricted\nto scalar targets) due to the initialization. Based on this characterization of\nthe invariant manifolds visited during the training, the authors provide some\nconjectures on learning dynamics.", "questions": "The claim that the Theorems 1 and 3 together imply saddle-to-saddle dynamics\nof increasing network complexity seems quite strong. On this point I have the\nfollowing four questions:\n\n1) Can the authors justify that there do not exist many other invariant\nmanifolds under gradient flow? If others do exist, why could the dynamics not\ntake us between these manifolds?\n\n2) Can the authors more clearly justify the statement ”invariant manifolds\nindicate that there exist gradient flow paths connecting pairs of embedded fixed\npoints defined in Theorem 1”? Unless I’m missing something simple this may\nneed a reference/proof/discussion.\n\n3) Can the authors justify that the learning between saddles is incremental?\nIt is not apparent to me that one could not ’skip’ several stages of complexity,\nor that (noting the numerical errors and stochastic effects in practice) in\ngeneral some perturbations may not push the flow away from following saddle-to-saddle dynamics.\n\n4) Could the authors explain how their theory interacts with times when\none does not escape a simple representation, sometimes at the cost of failing to\nminimize the loss function. Papers I have in mind here are [4, 3].\n\nI then have three more points:\n\n5) In 300 ”Subsequent iterations of saddle-to-saddle dynamics operate similarly.” Providing a full account of visited saddles in the case of a diagonal\nnetwork [11] shows that the saddles one visits in this setting follow iterates\nof AISSM [5], and it is non-trivial to prove. Is this setting simpler owing to\nthe comments in paragraph 4 of A.1 noting ”Linear networks with scalar input\nor scalar output do not have the embedded fixed points described by Theorem\n1 except the zero fixed point”? Otherwise it seems like there might still be more to say.\n\n6) Could the authors briefly explain how [14] differs in setup to allow saddle-to-saddle dynamics, as compared with comments in Sect 7?\n\n7) Typo: 311: follow\n\nReferences:\n\n[1] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. NeurIPS 2019.\n\n[2] Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, and\nJoshua Susskind. Transformers learn through gradual rank increase, NeurIPS 2023.\n\n[3] Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer\nnetworks training is a two-edged sword. Journal of Machine Learning Research, 26(183):1-75, 2025.\n\n[4] Etienne Boursier and Nicolas Flammarion. Simplicity bias and optimization\nthreshold in two-layer ReLU networks. ICML 2025.\n\n[5] Martin Burger, Michael Moller, Martin Benning, and Stanley Osher. An\nadaptive inverse scale space method for compressed sensing. Mathematics\nof Computation, 82(281):269-299, 2013.\n\n[6] Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic\nStephan. How two-layer neural networks learn, one (giant) step at a time. Journal of Machine Learning Research 25:1-165, 2024.\n\n[7] K Fukumizu and S Amari. Local minima and plateaus in hierarchical\nstructures of multilayer perceptrons. Neural networks, 13(3):317-327, 2000.\n\n[8] Gauthier Gidel, Francis R. Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in deep linear neural networks. NeurIPS 2019.\n\n[9] Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit\nbias of gradient descent on linear convolutional networks. NeurIPS 2018.\n\n[10] Michael Kleinman, Alessandro Achille, and Stefano Soatto. Critical\nlearning periods emerge even in deep linear networks. ICLR 2024.\n\n[11] Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. NeurIPS 2023.\n\n[12] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions\nto the nonlinear dynamics of learning in deep linear neural networks. ICLR 2014.\n\n[13] Zhenfeng Tu, Santiago Tomas Aranguri Diaz, and Arthur Jacot. Mixed dynamics in linear networks: Unifying the lazy and active regimes. NeurIPS 2024.\n\n[14] Yaoyu Zhang, Zhongwang Zhang, Tao Luo, and Zhiqin J Xu. Embedding principle of loss landscape of deep neural networks. NeurIPS 2021.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The structure of the paper is good and related literature is thoroughly documented. Equations (6) and (7) in Theorem 1 provide new examples of embedded fixed points which may be of independent interest. Theorem 3 shows\nthat the properties of units being zero, have equal weights, have proportional\nweights in the case of $\\phi$ homogeneous in $u$, or have linearly dependent weights\nin the case of $\\phi$ linear in $u$ are preserved under gradient flow. I found these\nresults to be interesting. In Appendix F.1 it is shown that on such invariant\nmanifolds the network may be represented with one with fewer units. Therefore saddle-to-saddle dynamics between such manifolds could indicate learning\nsequentially more complex networks. The proofs of these results are clear and I\ncan find no issue with them. Theorem 4 and Proposition 5 are presented clearly\nand their proofs (if one accepts the setup) seem to be correct and concise, with\nobservations on different drivers of saddle-to-saddle dynamics in the two considered settings. Proposition 5 seems to be both novel and interesting. The\nmotivation for scaling up width in transformers is a good application and experiments align with the theory. More generally, claims are supported with well\npresented and relevant experiments. The authors provide conjectures on when\nsaddle-to-saddle dynamics fail to emerge and broader settings exhibiting permutation symmetry where saddle-to-saddle dynamics may emerge. That some\nfuture research directions are motivated by this work is positive.", "weaknesses": "I am concerned that Theorem 4 is not novel. As the authors note in Appendix\nA.1, the incremental learning dynamics of linear networks are quite well understood. [12] and [8] seem to cover the same ground as Theorem 4, and closely related work includes [1, 9, 10, 6, 13]. Could the authors be more explicit in\nexplaining the novelty of this theorem?\n\nIt is noted that one of the conditions for saddle-to-saddle dynamics is that\nescape paths closely follow invariant manifolds. The authors do not discuss at\nlength when this may be satisfied or violated. As this is posited as key for\nthe emergence of such dynamics the paper would be improved with some more\nexamples and heuristic commentary, even if a general theoretical treatment of\nthis condition is infeasible.\n\nThe procedure of linearizing about fixed points and studying this dynamical\nsystem is not valid once some of the weights have begun to move away from the\ncritical point. Questions of rigour notwithstanding, the theoretical setting is\nquite restrictive in that it only considers networks of depth 2 and with no non-linear activation. This setting is arguably more restrictive than, for example, [2], although I appreciate that a diagonal parametrisation is used here (as an\naside the authors might like to consider including this reference?). The authors\nsuggest why tanh networks fail to exhibit saddle-to-saddle dynamics, but a\nwider discussion of how the results apply to commonly used activations (e.g.\nleaky ReLU) would strengthen the paper. Similarly, a discussion of the role\nof depth would make the paper more relevant. It seems to me that Appendix\nB illustrates that saddle-to-saddle dynamics can still occur in deeper networks\nwith non-linear activations, but without much comment on differences with the\nlinear two-layer case (or justification if there is no difference).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kppX92IeH", "reviewer": "Reviewer_axwH", "summary": "The submitted work analyzes saddle to saddle dynamics in neural networks. Specifically, the authors introduce a unifying framework for analyzing various architectures, including FCNs, CNNs and Attention. \n\nThe key results are:\n1. Fixed points of narrow networks are embedded inside wider networks.\n2. Mechanism behind saddle to saddle dynamics\n3. Definition of 'simple' in different networks: for instance, low rank in linear networks\n\nThese results have interesting implications:\n1. While network width does not affect the time scale separation in linear networks, adding more attention heads in shortens the plateaus\n2. The power law exponent of the data results in longer plateaus\n3. Shorter plateaus result from increases initialization scale", "questions": "* Do the authors have an intuition about how saddle to saddle dynamics changes on adding residual connections? At least empirically?\n* Can the authors clarify how their results on time scale separation differs from works like [1]\n\n[1] Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has several interesting contributions:\n1. Different architectures learn differently: linear networks are more data driven where quadratic models (such as attention) are sensitive to initialization scale.\n2. Conditions for Saddle-to-saddle dynamics\n\nOverall, the paper is well written has significant contributions towards our understanding of neural network dynamics and its dependence on architecture and dataset properties.", "weaknesses": "* The work is limited to simple setups: feedforward networks, gradient flow and synthetic datasets", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7k3TL9GRXN", "reviewer": "Reviewer_8EAS", "summary": "This paper provides a dynamical explanation for the simplicity bias observed in neural networks trained with SGD. The authors argue that when a neural network is initialized with small weights (i.e., near the origin of the loss landscape), it evolves closely along an invariant manifold corresponding to a width-$h$ network. It can then transition to an invariant manifold corresponding to a network of width $h+1$, which results in a steep decrease in loss.\n\n- Their central contribution is that, for this initialization, the saddle-to-saddle dynamics corresponds to a one-by-one activation of neurons. The network starts by fitting a single neuron, evolving closely along the corresponding invariant manifold until the optimal one-neuron solution is found. However, this minimum from a one-neuron perspective is a saddle point for the full-width network. The network then transitions to activate another dormant neuron and evolves closely along a two-neuron invariant manifold. This process repeats.\n\n- While the embedding of width-$(h-1)$ networks within width-$h$ networks is well-known, the authors demonstrate this for CNNs and attention layers and connect the phenomenon to the dynamics of SGD. They also provide novel propositions regarding timescale separation.", "questions": "- What happens to the dynamics if there are loss-induced invariant manifolds? For instance, in a teacher-student setup where the target function (the teacher) has the same symmetries as the student network, how would this affect the dynamics? In general, how does the data set and e.g. the data set size influence the dynamics and generalization.\n\n- There is a long history of research on simplicity bias (especially: https://www.jmlr.org/papers/volume22/20-676/20-676.pdf, https://www.nature.com/articles/s41467-024-54813-x). How does the notion of simplicity bias presented here connect to these prior findings?\n\n- Based on these results, what are potential new definitions for the lazy vs. rich regimes?\n\n- How can this framework be connected to the kernel perspective on spectral bias?\n\n- How do the results change when we remove the parameter symmetries? (https://neurips.cc/virtual/2024/poster/93573)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work connects saddle-to-saddle dynamics to the stage-wise learning of neurons along invariant manifolds, proving a specific version of simplicity bias. While the embedding of width-$h$ neural networks in the loss landscape of width-$(h+1)$ NNs is well-known, the authors extend these results to CNNs and attention layers.\n\n- They prove a timescale separation for the saddle-to-saddle dynamics, which is driven by the target function for linear neural networks but by the weight magnitude in the quadratic case.\n\n- One of the most interesting points is the new understanding of the lazy vs. rich regime. The dynamics within these regimes (i.e., saddle-to-saddle dynamics along invariant manifolds) can be distinct from strong feature learning, such as learning a low-rank solution. This has the potential to provide a much more detailed account of SGD dynamics.", "weaknesses": "- A more formal introduction to dynamical systems, perhaps in a short paragraph or an appendix, would be helpful. For instance, the term \"invariant manifold\" is used but never formally defined.\n\n- There is a significant body of literature on spectral bias in kernels (e.g. https://arxiv.org/pdf/1912.01198), such as the Neural Tangent Kernel (NTK). The paper is missing a discussion on how its results connect to spectral bias in function space. Although the mechanism in the infinite-width limit is different, it would be valuable to explain these differences and explore how the two perspectives could be combined.\n\n- A clearer study, even if only empirical, on how the structure of the target function influences learning and saddle-to-saddle dynamics would be beneficial. For example, what happens with single-index models or k-sparse parity? How does the data distribution $p(x)$ (e.g., normal, uniform, or spiked) affect the results? It's unclear to what extent the observed dynamics are solely a result of how neural networks are initialized and trained. Since the gradient is taken with respect to the loss function, it would be highly interesting to understand how data-induced invariant manifolds (see questions) might arise and interact with the model-induced ones.\n\n- A clearer and more detailed phase diagram of the lazy and rich regimes would be nice, incorporating the new insight that the distinction is more about saddles and invariant manifolds than just the initialization scale and learing rate.\n\n-  There is a typo on Line 424.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)