Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape
Authors: Ioannis Bantzis
James B Simon
Arthur Jacot
Venue: ICLR 2026 Poster (2026)
Keywords: Saddle-to-Saddle
Implicit bias
Low-rank bias
Bottleneck rank
Primary area: learning theory
TL;DR: Starting from a small initialization, DNN escape the plateau around the origin with most of its layer being approximately rank 1.

Abstract: When a deep ReLU network is initialized with small weights, gradient descent (GD) is at first dominated by the saddle at the origin in parameter space. We study the so-called escape directions along which GD leaves the origin, which play a similar role as the eigenvectors of the Hessian for strict saddles. We show that the optimal escape direction features a \textit{low-rank bias} in its deeper layers: the first singular value of the $\ell$-th layer weight matrix is at least $\ell^{\frac{1}{4}}$ larger than any other singular value. We also prove a number of related results about these escape directions. We suggest that deep ReLU networks exhibit saddle-to-saddle dynamics, with GD visiting a sequence of saddles with increasing bottleneck rank.

Published as a conference paper at ICLR 2026
SADDLE-TO-SADDLY DYNAMICS IN DEEP RELU NETWORKS: LOW-RANK BIAS IN THE FIRST SADDLE ESCAPE
Ioannis Bantzis
James B. Simon
EPFL
UC Berkeley and Imbue Berkeley and San Francisco, USA james. s simon@berke edd.
Lausanne, Switzerland
ioannis bant wwseaaisslliii epfl.
Arthur Jacot
Courant Institute, NYU
New York, USA arthur. @nyu.ee
ABSTRACT
When a deep ReLU network is initialized with small weights, gradient descent (GD) is at first dominated by the saddle at the origin in parameter space. We study the so-called escape directions along which GD leaves the origin, which play a similar role as the cigenvectors of the Hessian for strict saddles. We show that the optimal escape direction features a low-rank bias in its deeper layers: the first singular value of the f-th layer weight matrix is at least éå larger than any other singular value. We also prove a number of related results about these escape directions. We suggest that deep ReLU networks exhibit maddle-to-saden dynamics, with GD visiting a sequence of saddles with increasing bottleneck rank (Jacot 2023a).
INTRODUCTION
In spite of the groundbreaking success of Deep Neural Networks (DNNs), the training dynamics of GD in these models remain ill-understood, especially when the number of hidden layers is large. significant step in our understanding is the (relatively recent) realization that there exist multiple regimes of training in large neural networks: kernel or lazy regime, where DNNs simply implement kernel methods w.r.t. the Neural Tangent Kernel (NTK) (Jacot et al.) 2018 Du et al.) 2019 Allle--Zh al. 2019), and active (or rich) regime characterized by the presence of feature learning (Chizat & Bach 2018a Rotskoff' & Ganden-Bijnden [2018] Chizat & Bach (0188) and some form of sparsity such low-rank bias (Li et al. 2020 Gunasekar al. [017]] Arora et al., 2019a).
Arora
The kernel regime is significantly simpler than the active one because the dynamics can be linearized
around the initialization (Jacot et al. 2018 Lee al., 2019). and the loss is approximately quadratic/convex in the region traversed by GD (Jacot et al. 2020) (it also satisfies the PL inequality (Liu et al., al. 2020). This makes it possible to prove strong convergence guarantees (Du al.) 2019) Allen-Zhu et al. 2019) and apply generalization bounds from the kernel methods literature almost directly (Arora al.) 201 Bordelon al. 2020). Our understanding the kernel regime essentially complete, but some functions are unlearnable in the kernel regime yet learnable in the
active regime (Bach] 2017 Ghorbani et al. 2020
There are arguably many active regimes corresponding different ways to leave the kernel regime. including small weight initialization (Woodworth et al., 2020), large learning rate (Lewkowyez el al.
2020: Damian al. 2022). large noise in training (Smith al. 2021 Pesme al. 2021 Vivien
et al. 2022; Wang Jacot 2024), late training with the cross entos-yy loss (Ji Te telgarsky 2018
Chizat & Bach 2020), and weight decay (E al. 2019 Ongie al. 2020 Jacot 2023a).
"ooannis Bantzis supported by a scholarship for graduate studies from the Onassis Foundation.

---

Published as a conference paper at ICLR 2026
We will focus the effect of initialization scale, where a phase change from kernel regime to active regime occurs as the variance of the initial weights decays towards zero. Here again we can distinguish two active regimes (Luo et al. 202T): the mean-field regime which lies right at transition between regimes (Chizat & Bach| 20188) Rotskoff & Vanden-Hijnden 2018 Mei et al. 2018). and the
saddle-to-saddle regime (Saxe al. Jacot al. 2022 Pesme & Flammarion
2023
Boursier
ct al. 2022) for even smaller initialization.
The mean-field limit was first described for shallow networks (Chizat & Bach 2018a; Rotskoff &
Vanden-Eijnden 2018| Mei et al. 2018), and has more recently been extended to the deep case (Araújo et al. 2019 Bordelon & Pehlevan 2022). A limitation of these approaches that the limiting dynamics remain complex, especially in the deep case where they are described by algorithms that are not only very costly in the worst case (Bordelon & Pehlevan 2022; Yang Hu, 2020p, but also difficult to interpret and reason about. This high complexity could be explained by the fact that the mean-fiel field limit is critical, i.e. it lies exactly at the transition between kernel and active, and therefore it must capture the complexity of both of those regimes, as well as of the whole spectrum of intermediate dynamics.
1.1 SADDLE- TO---DDDAADD DYNAMICS
This motivates the study of the saddle-to-saddle regime for even smaller initializations. As the name suggests, this regime characterized by GD visiting a number of saddles before reaching a global minimizer. Roughly speaking, because of the small initializations, GD starts in the vicinity of the saddle which lies at the origin in parameter space and remains stuck there for a number of steps until it finds an escape direction, leading to sudden drop in the loss. This escape direction exhibits form of approximate sparsity (amongst other properties) that is preserved by GD. At this point, the level of sparsity can either be enough to fit the training data in which case the loss will drop to zero and training will stop, but if the network is 'too sparse' to fit the data, GD will approach another saddle a lower cost (which is locally optimal given the sparsity) before escaping along less sparse escape direction. GD can visit a sequence of saddles before reaching a final network that fits the data while being as sparse as possible. This has been described as performing greedy algorithm (Li et al. 2020) where one tries to find the best data-fit fit with a sparsity constraint that is gradually weakened until the training data can be fitted.
Such incremental learning dynamics were first observed in diagonal linear networks (Saxe et al. 2014, 2019, Gidel et al. 2019) (and by extension to linear Convolutional Neural Networks, which are diagonal nets in Fourier space), before being extended to linear fully-comnected networks (Arora cct al. 2019a Li et al., 2020 Jacot et al., 2022: Tu et al. 2024 Kunin et al. 2024). These result in coordinate sparsity of the learned vector for diagonal networks and rank sparsity of the learned matrix for fully-connected networks.
For nonlinear networks, the focus has been mainly on shallow networks, where condensation effect is observed, wherein groups of neurons end up having the same activations (up to scaling). Roughly speaking, in the first escape direction, first group of hidden neurons comes out first, all with the same activation (up to scaling), in the subsequent saddles, new groups can emerge or an existing group can split in two (Chizat & Bach 2018a) (though sometimes they may fail to split leading to problems (Boursier& & Flammarion] (024)). This condensation effect leads to a form of sparsity, since each group then behaves as single neuron, thus reducing the effective number of neurons (Luo et al. 2021 (Simse et al. (021). These dynamics could be understood as implicitly implementing a Frank Wolfe algorithm (Bach) 2017): alternating between finding new neurons to 'add' to the mix, and then tuning the mixing weights to get the best possible fit (Kunin et al. 2025).
To our knowledge, all prior theoretical analysis of saddle-to-saddle dynamies in deep nonlinear networks rely on an equivalence to deep linear networks, which can arises with differentiable nonlinearities (e.g. arctan) allowing for a Taylor approximation of the origin (Bai et al., 2022). or in specific settings where the ReLUs do not change signs (Zhang et al.) 2025). This leads to
low-rank bias, where all layers are rank-one in the first escape direction. Saddle-to-see saddee dynamics with multiple plateaus/saddles have been observed empirically in deep ReLU networks trained on supervised (Atanasov et al.] 2024) and self-supervised (Simon et al. 2023) tasks, and these empirics motivate present theoretical study.
2

---

Published as a conference paper at ICLR 2026
1.2 BOTTLENECK RANK INCREMENTAL LEARNING
Surprisingly, show more complex rank sparsity structure in deep ReLU networks: the majority of layers are rank-one (or approximately so), with possibly a few high-rank layers at the beginning of the network, in contrast linear nets, shallow ReLU networks, and deep nets with differentiable nonfinearity where all layers are rank-one in the first escape.
This fits into the bottleneck structure and related bottleneck rank (BN-rank) observed in large depth ReLU networks trained with weight decay (Jacot 2023a bJ Wen & Jacot 2024 Jacot & Kaiser (024). where almost all layers share the same low rank, with a few higher rank layers located close the input and output layers. Additionally, in the middle low-rank layers ("inside the bottleneck") ") the preactivations are approximately non-negative, so that the ReLU approximates the identity.
The bottleneck rank Rank (f)) is a notion of rank for finite piecewise linear functions f. defined as the minimal integer ke such that can be decomposed f = h o g with intermediate dimension k* (Jacot, 2023a). For large depths, it is optimal in the sense of minimizing the parameter norm to represent f with bottleneck structure, where the first few high-dim. layers represent g. followed by many rank k* layers representing the identity on the dimension k* intermediate representation, before using the last few layers to represent
Our results imply that the first escape direction of deep ReLU networks has BN-rank one, because almost all layers are approximately rank-one except a few high rank layers in the beginning. This is a "half" bottleneck structure, since it lacks high dimensional layers before the outputs, but it still fits within the BN-rank theory, suggesting that the BN-rank is the correct notion of sparsity in deep ReLU networks (rather than the traditional notion of rank).
We conjecture that deep ReLU networks exhibit similar coldle-to-saddee dynamics as linear networks, with the distinction that it is the BN-rank that gradually increases rather than the traditional rank.
1.3 CONTRIBUTIONS
In this paper, we give description of the saddle the origin in deep ReLU networks, and the possible escape directions that GD could take as it escapes this first saddle. As in (Jacot et al.) (022). each escape direction can be assigned an escape speed, and we show that the optimal escape speed non-decreasing in depth (Proposition 3.2).
We then prove in Theorem 3.1 that the optimal escape directions feature a low-rank bias that gets stronger in deeper layers (i.e. layers closer to the output layer). More precisely the weight matrix Wa and activations Zi over the training set for f = 1,.., L are t ap-aappoxiiaately rank 1 in the sense that their second singular value is O(e- 4) times smaller than the first. Furthermore, deeper layers are also more linear, i.e. the effect of the ReLU becomes weaker.
Finally, we provide in Section 4 an example of a simple dataset whose optimal escape direction has the following structure: the first weight matrix is rank two, followed by rank-one matrices. This shows that the structure of our first result where the first layers are not approximately rank-one but the deeper layers are is not an artifact of our proof technique and reflects real examples. This contrasts with previous saddle-to- -o-addde dynamics, where all layers are approximately rank-one in the first escape direction.
2 SADDLE AT THE ORIGIN
We represent the training dataset Ii,..., fully-connected neural network of depth L with widths no din, nl...., nr dout and ReLU nonlinearity (z) = max(2, (z, 0}. The ne x N dimensional matrices of activation Z and preactivation Ze at the ε-th layer then defined recursively as
Rdio as a dip x N matrix X. We consider a
2 = X
Z, Wi
2 (Ze),
3

---

Published as a conference paper at ICLR 2026
for the ny X ng-1 weight matrix We ( =,.,..... I,...,L. L. The weight matrices Wi...., Wz the parameters of the network, and we concatenate them into single vector of parameters of dimension P = De 1- The outputs of the network the last layer's preactivations Y, Zr.
We consider general cost C. R + R that takes the network outputs Yo and returns the loss ( = C(Y)). The parameters 8(t) are then trained with gradient flow (GF) on the loss L
0,0(1) VVC((()
starting from random initialization 6g ~ (0, ()) for small
One can easily check that the origin 0 is a critical point of the loss. Our analysis will focus on the neighborhood of this saddle, and for such small parameters the outputs Yo will be small, we can therefore approximate the loss as
(1) where VC(0) an ny N matrix. Since we only care about the dynamics of gradient flow, the first term can be dropped. We will therefore mainly focus on the localized loss Lo( = Tr] writing = (C()) for simplicity.
(()) C(V)= C(0)) TT [VC(O)")]] O(I)))))
The localized loss Lo can be thought of as resulting from zooming into origin. It captures the loss in the neighborhood of the origin. Note that since the ReLU is not differentiable, neither the loss at the origin, that cannot However, this loss has the advantage being homogeneous with degree L. i.. Lo = Co)). which will be key in our analysis. A derivation of equation T and of this homogeneity can be found in Section A.1 of the Appendix.
the traditional strategy of approximating Co by polynomial.
2.1 GRADIENT FLOW ON HOMOGENEOUS LOSSES
On a homogeneous loss, the GF dynamics decompose into dynamics of the norm ||0]] and of the normalized parameters 0/l0011 (see Proposition A.I for detailed derivation):
(+))(())l ((t))V))(( =111000)0- 0,0(t) = (1-))))))-(((((())))))))+
where we used Euler's 's homogeneous function theorem: 8TH Vo( = LCo . In our setting the parameter norm is very small ||0]| << 1, which implies that the dynamics of will be much faster than that of the norm ||@]| because |[(())] >> ||@(t)][--1
Notice that (I 1 00T) is the projection to the tangent space of the sphere at 8, which implies that the normalized parameters are doing projected GF over the unit sphere on the Co loss (up to a prefactor of [ which be interpreted speed up of the dynamics for larger norms).
Therefore, may reparametrize time t(g), such that g(t) f [10(g)) - which correspond switching time-dependent learning rate T, 110(g))1-22 obtain the dynamics:
0, (1(()) -T.11(()) 0,0(4) - -1--00000")VV<0000
We can therefore solve for (()) on its own, and the norm [[Ø(q)] then takes the form
( ||((a)] exp (-L co(0(o1))d11)
If needed, these solutions can then be translated back in I-time, using the formula
((g) = 1 [10(a))))~ = 11(00)11-1 [" exp [TT.. (1(T--2) 2) f Lo(0(12))d1)) dhp-
4

---

Published as a conference paper at ICLR 2026
2.2 ESCAPE DIRECTIONS AND THEIR SPEEDS
Assuming convergence of the projected gradient flow O(g). for all initializations zo there will be time qi where ((qi) will be close to critical point of Lo restricted to the sphere, i.e. point 0 such that (I- f* 0*
V Lou(d") = O. We call these escape directions (assuming (u 0), because
once it is reached, (()) remains approximately constant while the parameter norm grows fast.
Definition 2.1. An escape direction is vector on the sphere p E / such that (
-sp for some & E IR,, which we call the escape speed associated with p. We switch from the unit sphere to the radius LL sphere as it will lead to cleaner formulas.
An optimal escape direction p² L2 is an escupe direction with the largest speed 8* > O. is minimizer of Lo restricted to 1112
p' E arg ming Co(p).
If the parameters start aligned with an escape direction Bo x p. then GF on the localized loss will diverge towards infinity in straight line with rate determined by the depth L and the escape speed Proposition 2.2. Considering gradient flow on the localized loss Co. if at some time to the parameter satisfies
@(to) = p with p E 1 and Vlu(p) = - p. then for all t 2 to the normalized direction remains constant, and the norm [|@(t)] satisfies
(110(6))1 2- + (2 LL)L (1- 10)) tt)) if 1 + [0({)]I [[e(to)] exp 2 (t - -to))
i 2.
Of course, the localized loss Lo is only good approximation as long as the outputs Yo are small. This will apply up to some escape time (r) which is the first time the outputs satisfy Yell r. Since [|Ya II S 1|Wzllop... ... .. X|| S
L
[10
||||| r we know that
[10(to)]1 2-1 - -
if 2
t (r) - to
(L 2) 1 (log VLr log [(((tt)))),
if 2.
After this escape time, we expect the localized GF diverge from the true GF: the localized GF diverges towards infinity (in finite time when L > 2). while the true GF typically slows down as it approaches another saddle or a minima. This paper focuses on the dynamics before the escape time.
In general, we do not start aligned with an escape direction, but since the normalized parameters 0(4) follow GF restricted to the sphere, they will converge to an escape direction, at which point a similar explosion of the norm will take place.
Note that the dynamics of Ō(4) (in reparametrized g-time) are unaffected by multiplying the initialization On by factor >> 0. Therefore the time q of convergence to escape direction is independent ax, and at the time y1, the parameter norm will depend linearly on a: [1(())) ll = Ca that the Taylor approximation
for some C' > 0. We can therefore always choose small enough (Equation M remains valid up to the time of convergence q1-
The convergence of GF to escape directions is essentially the same the convergence to Karush- (KKT) points as GD diverges towards infinity with the cross-entropy loss Ji & Telgarsky (2018): ; Chizat & Bach (2020). The techniques used in these papers could be used to extend the present GF analysis to GD convergence.
3
Low RANK BIAS AND APPROX. LINEARITY OF THE ESCAPE DIRECTIONS
The main result of this paper is that at the optimal escape directions, the deeper layers (i.e. for large E) approximately low-rank and have almost nonlinearity effect:
5

---

Published as a conference paper at ICLR 2026
Theorem 3.1. Consider optimal escape direction
0" arg min Tr G Yo)
[01|=
with optimal speed s =i Tr Gr|G' Yo), then for all layers with 8 > d² we have:
Exs (We) Eizg (2,) Et Σx2 (
12 - 2
< c e
1Ze
cl
where (A) is the i-th largest singular value of A and c |X
log
In the rest of the section, we will prove result that shows that the optimal escape speed 8* is increasing in depth, thus controlling the constant e in depth. This guarantees that the condition c² holds for all but finitely many of the initial layers of the network. We then present sketch of proof for the Theorem, stating a few intermediate results that are of independent interest.
3.1 OPTIMAL SPEED IS INCREASING IN DEPTH
The bounds of Theorem 3.1 are strongest when the optimal escape direction s* is large. Thankfully, the optimal escape speed increasing in I::
Proposition 3.2. Given depth L network with Co(0) so for so 0 and |101| L, can construct a network of depth + k for any k 2 with parameters that satisfies |10'112 = +
and Lo( I (0).. Therefore, the optimal escape speed s* (I.) is a non-decreasing function.
Furthermore, in the deeper network, we have Rank( ZLr) = Rank WL = 1 for all L' 2 and Z Zi, for all L' L.
To construct the deeper network, first transform the last weights W,, to be rank-one (this is possible without increasing Lo), and we then add rank-one weights in the additional layers. Some very similar structures have been used in previous work (Jacot 2023a Bai et al. 2022)
3.2 SKETCH OF PROOF
To prove Theorem 3.1 first show that if the inputs approximately rank-one, then the optimal escape will be approximately rank-one all layers:
Proposition 3.3. Consider the minimizer H = arg min PSL Tr TT(GTTo((ue' + x)]
where u, e
R.a u, v 2 0 entry wise and ||x|| S for some e Then for all we have:
Ees (WW)) Exs (2(7)
12
Ze Zell
|
<8
(2( Σ (2)))
7
G
This also implies that a hidden representation is approximately rank-one in one layer lo. then must also be approximately rank-one in all subsequent layers lo- We can prove the existence of many such low-rank layers assuming the escape speed is large enough:
Proposition 3.4. Assuming Tr(G 7ı.) v -80 for some constant 8p > and ||01| < then for any ratio p E (0, 1) there are least (1 L layers that are approximately rank-one the sense that
E,>o 87 (27)
[IX||~|||||]
1 PL
< log
Σ 87 (27)
80
The proof of Theorem 3.1 therefore goes as follows: for any e = pL, Proposition 3.4 implies that there least (1 L L - layers that approximately rank-one. The earliest such layer to must satisfy &n < L. Proposition 3.3 implies that all layers after 4o must be approximately rank-one,
including the l-th layer.

---

Published as a conference paper at ICLR 2026
15.
5000)
5/W))
05
step
(a) Training Loss
(b) After the first saddle escape
(c) Final Iteration
Figure 1: Deeper layers show stronger bias toward low-rank structure than earlier layers MNIST. Left: Training loss over training time. Vertical lines indicate the specific iterations at which singular values extracted. Center and Right: Top 10 singular values of the weight matrices per layer& for layers 1-6 including input and output layer.
The two propositions are also independent interest. Proposition 3.3 gives an example of inputs where all layers are low-rank, not just the deeper layers. Proposition 3.4 applies to any parameter with fast enough escape speed, not just to the optimal escape direction, and guarantees a similar low rank structure. Interestingly, in contrast to the other results, it does not say anything about where those low-rank layers are.
3.3 EMPIRICAL RESULTS ON MNIST
We empirically confirm the presence of low-rank structure in networks trained the MNIST dataset. Specifically, we train a 6-layer fully connected network without bias and with small initialization.
Figure highlights two distinct saddle points during training. After escaping the first saddle, observe the emergence of single dominant singular value in every layer, with this effect being particularly pronounced in the deeper layers (layers 4-6). While theoretical analysis explains the behavior after the first saddle escape, our experiments reveal that, towards the end of training, a second dominant singular value appears. This suggests that the rank of the weight matrices increases following subsequent saddle escapes. A detailed visualization of the singular value evolution each layer is provided in Figure 3 in the Appendix.
THE OPTIMAL ESCAPE DIRECTION IS NOT ALWAYS EXACTLY RANK ONE
Our discussion has thus far consisted of results which paint the picture that deep ReLU networks trained from small initialization first escape the origin in direction which is approximately rank one in each weight matrix. Much of our labor has been in identifying suitable notions of "approximately rank one." Before concluding, it is worth asking: do we actually need such notions? In fact, if one performs straightforward numerical experiments on simple datasets, one will often find that the first escape direction is exactly rank one in each layer. Might we hope that the optimal escape direction is fact always exactly rank one?
In this section, we provide simple counterexample in which the optimal escape direction is rank two the first layer. We then give numerical experiments which show that (projected) gradient descent actually finds this rank-two solution.
Example 1 (Rank-two optimal escape direction). Consider the unit circle dataset (x1))=1=
(sin( ( cos ( N )) with alternating loss gradients = ((-1))) Let = Consider training depth-three bias-free ReLU MLP with hidden width at least four from small initialization on this dataset. Then the optimal rank-one escupe direction has speed 81 = /2 - R 0.414, but there exists better rank-two escape direction with speed s =
Such alternating loss gradients can result straightforwardly from, for example, targets Y ((-1))) and the usual squared loss.
7

---

Published as a conference paper at ICLR 2026
Unit circle dataset
Escape simulations (width 16)
Escupe simulation resulls
1.00
0.5
Successful fraction
V2
0.75
0.50
-1-
0.25
0.0
10000 step
20000
width 10
21
Figure 2: Depth-3 neural networks find rank-two escape directions a toy dataset. Left: visualization of the dataset. Red and blue points have loss gradient values respectively. Center: several training objective under the parameter norm constraint ||Ø] best achievable value for rank-one weights, are colored blue and deemed successful. Right: as width increases, the fraction of successful runs increases. See Figure for visualization of the training runs at all widths.
and = -1, of projected gradient descent the first-order loss Runs whose objective exceeds v2 the
Proof. Our network has weight matrices W,, W2, Wa which parameterize the network function as Y, = W3a o W20 Wi X. As discussed in Subsection [.2 we wish to minimize the escape speed = -rr Yel such that Σ, [[ell 3. We know from homogeneity that the minimizer will have [Wel = for all f.
If we additionally constrain all three weight matrices to be rank-one, then width-one ReLU network can achieve the same maximal escape speed (a network with only rank layers can only represent 'one neuron functions': Y, = uo (v" x) for some vectors
independently of depth). Taking into account the positivity ReLU, we need only study a width-one network with Wi = [cos(()) ( sin (Φ)) for some & [0, 2xr. W2 [1]. and W3- [+1]. The only degree of freedom remaining is the angle 6 to which W1 attuned. We solve this ID optimization problem in Appendix D.I. finding that the optima fall Φ 7 for j E Z. giving speed 81 = v2 R 0.414.
Without such rank-one constraint, can improve this speed. We use only four neurons in the first hidden layer and one neuron the second hidden layer (setting all incoming and outgoing weights to other neurons to zero) and choose the following weights for the active neurons:
0
W1
W₂ 11 -1 1-1, W3 =[]]
-1
0
This gives speed $2 1
This counterexample shows that the optimal escape direction may in fact be non-rank-ome and thus it is reasonable to search for sense in which, for sufficiently deep network, the optimal escape direction is approximately rank one
4.1 NUMERICAL, EXPERIMENTS: WIDE NETWORKS FIND THE OPTIMAL, ESCAPE DIRECTION
Of course, the existence of such non-rank-one optimal escape direction is only interesting if gradient descent actually finds it. In this case, does. We train networks of varying width with projected gradient descent to minimize the loss on the sphere ||0 2 3. As shown in Figure 2 wider networks are more likely to converge the faster, rank-two escape direction.
It is worth noting that there may exist even faster escape direction than the rank-two solution we identify (though we doubt it; numerical experiments), but in any may he assured that the fastest escape direction not rank one.
8

---

Published as a conference paper at ICLR 2026
5 DISCUSSION: SADDLE-TO-SADDA DYNAMICS
The results of this paper only describe the very first step of a much more complex training path. They describe the escape from the first saddle at origin, but is likely that the full dynamics might visit the neighborhood of multiple saddles, as is the case for linear networks (Jacot et al.) 2022, Li et al., 2020) or shallow ReLU networks (Abbe et al.) [021]] 2022). We now state a few conjectures/hypotheess, which should be viewed as possible next steps towards the goal of describing the complete Saddle-to-Saddle dynamics:
(1) Large width GD finds the optimal escape direction: Our numerical experiments suggest that wider networks are able to find the optimal escape direction with GD, even when this optimal escape direction has some higher rank layers. The intuition is that the more neurons, the more likely it is that a subset of neurons implement a 'circuit' that is similar to an optimal escape direction, and that this group will out-compete the other neurons and end up dominating. Note that even in shallow networks, finding this optimal escape direction is known to be NP-hard (Bach (017), which implies that an exponential number of neurons might be required in the worst case.
(2) Exact rank-one at most layers: Inspired by our illustrating example, we believe that it is likely that the optimal escape directions might only have finite number of high-rank layers at the beginning. followed by rank-one identity layers until the outputs.
Note that if we assume that the optimal speed direction s" (L), plateaus after a certain Lu, i.e. s* (L) =s* (Lo) for all L 2 Lo. then Proposition 3.2 already implies that there is an optimal escape direction where layers f 2 T.n are rank 1. Conversely, if there is an optimal escape directions with only rank-one layers after the To-th layer, then s" (I.) s* (T-o) for all I, 2 To-
(3) rank-one layers remain rank-one until the next saddle:
Assuming that GD does find the optimal escape direction, it will have approximately rank-one layers as it escapes the saddle. The next step is to show that these layers remain approximately rank-one until reaching a second saddle.
In linear networks, this follows from the fact that the optimal escape direction rank-one and balanced (i.e. W.T, We-1 W for all layers E), and that the space of rank-one and balanced network is an invariant space under GF.
The ReLU case is more difficult because we have only approximately rank-one layers. More precisely to guarantee that there is a layer that e-approximately rank-one, we need both a small initialization and large depth, in contrast to linear networks where a small enough initialization is sufficient. Our second conjecture would help with this aspect.
The next difficulty is to show that the approximate rank-one layers remain for sufficient amount of time. The key tool to prove this in linear networks is balancedness. ReLU networks only satisfy weak balancedness in general i.e. diag(W? Wi) diag(W, - Wy".), however the stronger balancedness applies at layers where the pre-activations have non-negative entries: 2, 2 0.
(4) BN-rank incremental learning The final goal is to prove that these Saddle-to-Saddle dynamics allow ReLU networks to implement a form of greedy low BN-rank algorithm, which searches first among BN-rank one functions, then among gradually higher rank functions, stopping at the smallest BN-rank sufficient to fit the data.
Again, this is inspired by an analogy to linear network, which implement greedy low-rank algorithm to minimize the traditional rank. In parameter space, the GD dynamics visits sequence of saddles of increasing rank. It starts close to the saddle the origin (the best rank fit),, before escaping along a rank-one direction until reaching a rank-one critical point (locally optimal rank-one fit). If the loss zero at this point, the GD dynamics stop, otherwise this best rank-one fit is a saddle where GD platcaus for some time until escaping along a rank 2 direction, and on (Jacot et al. 2022p.
The so-called Bottleneck rank Rank N.N (f) (Jacot 2023a) is the smallest integer & such that f can be represented as the composition of two functions f= h y with inner dimension k. Several recent papers have shown how the BN rank plays a central role in deep ReLU networks trained with decay/L2-reguulee particular, these works observe the emergence of a bottleneck structure as the depth grows, where all middle layers of the network share the same low rank (discarding small singular values of Wt), which
(Jacot 2023a bi Wen & Jacot 2024: Jacot & Kaiser 2024). In

---

Published as a conference paper at ICLR 2026
equals the BN rank of the network, with only finite number of high-rank layers at the beginning and end of the network.
Our results can be interpreted as saying that the optimal escape direction of the saddle at the origin exhibits a 'hhlf- bottleeetenek' (because there are only nigh-dimensional layers at the beginning of the network, not the end) with BN-rank one. This suggests that the Saddle-to-Sadeee dynamies in deep ReLU networks could correspond to a greedy low-BN-rank search, where the BN-rank increases gradually between each plateau/saddle. Interestingly, previous theoretical analysis of the bottleneck structure were only able to prove the existence of low-rank layers but not necessarily locate them (Jacot 2023b), our ability prove that the deeper layers are all approximately low-rank is therefore significant improvement over the previous proof techniques.
possible that in contrast to linear network, the complete Saddle-to-SaddBe dynamics would require both small initialization and large depth. This matches our numerical experiments in Figure Tand Figure 4 in the appendix, where we observe more distinct plateaus depth 6 layer compared to depth 4 layer. This suggests that in contrast to linear networks, where the plateaus can be made longer and more distinct by taking smaller initialization, for ReLU networks we need to also increase the depth to achieve the same effect.
REFERENCES
Emmanuel Abbe, Enric Boix-Adserå, Matthew Stewart Brennan, Guy Bresler, and Dheeraj Mysore Nagaraj. The staircase property: How hierarchical structure can guide deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL hittps://pperre// wwt.reeieeeeeee......e.... www.netfforrmt forumm id=
EirrFciApc
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-stairease property: necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp. 4782-4887. PMLR, 2022.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Dyego Araújo, Roberto I Oliveira, and Daniel Yukimura. A mean-field limit for certain deep neural networks. arXiv preprint
2019.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019a. ISSN 1049-5258.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019b.
Alexander Atanasov, Alexandru Meterez, James B Simon, and Cengiz Pehlevan. The optimization landscape of sgd across the feature learning strength. arXiv preprint arXiv:2410.046-22 2024.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-68) 2017.
Zhiwei Bai, Tao Luo, Zhi-Qin John Xu, and Yaoyu Zhang. Embedding principle in depth for the loss landscape analysis of deep neural networks. arXiv preprint arXiv:2205. 13283, 2022.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35:32240-32256, 2022.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. arXiv preprint arXiv:2002.02561 2020.
Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is a two-edged sword. arXiv preprint arXiv:2401.1079) 2024.
10

---

Published as a conference paper at ICLR 2026
Etienne Boursier, Loucas Fillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow reLU networks for square loss and orthogonal inputs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL htt.ppss///peeeeiiiitttt
net forrmm? d=1.74c- 1xxQ1
Lénaïc Chizat and Francis Bach. On the Global Convergence of Gradient In
Descent for Over-parameterizee Models using Optimal Transport.
Advances in Neural Information Processing Systems 31, pp. 3040-3050. Curran Associates, Inc., 2018a.
URL http:: wwwppppprrr.ss....pooappp..mseem www..iss........o.....m cctps//eee.iissccpppe/e.//e.
pdf
Lenaic Chizat and Francis Bach. A note on lazy training in supervised differentiable programming. arXiv preprint 1ef2:::811.979555 2018b.
Lénaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 1305 1338. PMLR, 09-12 Jul 2020. URL http: http://proceecinggs mlr. mmlr.rress/v15/ chi Shizatt0a.hhan
Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint 0rXiv:2209.155000 2022.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes pecepparameterized neural networks. In International Conference Learning Representations, 2019. URL
Weinan E, Chao Ma, and Lei Wu. Barron spaces and the compositional function spaces for neural network models. arXiv preprint arXiv: 09009 2019.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 14820-14830, 2020.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization discrete gradient dynamics in linear neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- (uché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http::htt////ototse
neur ps cc/
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6152-6160, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Arthur Jacot. Implicit bias of large depth networks: notion of rank for nonlinear functions. In The Eleventh International Conference on Learning Representations, 2023a. URL https
net. wtt.e//wwww..eeettoooeiee------
0B
Arthur Jacot. Bottleneck structure in learned features: Low-dimension vs regularity tradeoff. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 23607-23629, Curran Associates, Inc., 2023b. http///ccte///prreeeinggg.n wwwp:eeeeeeddeeeeeneeeeee.eeeeee ceppeppe http:eeerriieeeeeeeeee//20 Eile/ (lleAA/ecccBBee003311111111111e2
.df
Arthur Jacot and Alexandre Kaiser. Hamiltonian mechanics of feature learning: Bottleneck structure in leaky resnets, 2024. URL http:/sss///xxi..og////ea//////e
2453317573
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kerel:: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems 31, pp.
Curran Associates. Inc., 2018. URL http:tt:://pae www.aaprss.ippee nww.. wttpp///p.......ccccpp....ppp 8076-neural- neuralu-ll---- -aagentt wtw.rraeeeooaaarrcoeeoooeeeeeeaa fd

---

Published as a conference paper at ICLR 2026
Arthur Jacot, Franck Gabriel, and Clément Hongler. The asymptotic spectrum of the hessian of
dnn throughout training. In International Conference on Learning Representations, 2020. URL
net/ Forumiid--
Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity, 2022.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. CoRR, abs/1810.02033 2018. URL http:::/aaxii..rrrg//////// wwwporggg/abbaooeeeeeeeeeeeeaeee 02032
Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems, 37:81157-81203 2024.
Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R WeWeese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A theory of feature learning in two-layer neural networks. arXiv preprint arXiv:2506.066: 2025.
Jachoon Lec, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in neural information processing systems, pp. 8572-8583, 2019.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022 02.11.. 2020.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2020.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Toward theory of optimization for over-parameteriized systems of non-linear equations: the lessons of deep learning. arXiv preprint arXiv:2003.000307, 2020.
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at width limit. Journal of Machine Learning Research, 22(71):1-47, 2021. URL /tttp://mmll http://mmrr.o../r../////e/ httt:://hhh..oooorapppsssss/////
wttp:/ppppppppppppppeeeeeepppp watper//v22220022 11m..mmmmmmm
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 116633))7766--7777 2018.
Greg Ongie. Rebecca Willetr, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. International Conference on Learning Representations, 2020.
http://ppnnrrreeeeeeeeerrrrnnnna ii 11N 1NP xHKDH
Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 7475-7505. Curran Associates, Inc., 2023. URL https::/ hwwp://wwwee.nnnnneeiii..o...e wtw.reoe//e//ooee.e........oooo.
waales/paper/// FLle///0881111550000033331111112 www.aapppppaaccccceee.e.eaeeeeee pdf
Scott Pesme, Loucas Pillaud-Vin Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: provable benefit of stochasticity. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 29218-29230. Curran Associates, Inc., 2021. URL /ttpsp/cpp///proeeiiiin...ooo www.lngggnnuuuuuene
files/ 2ttp:///e_iieee/appe//22022 pptp///pppr///////////
12

---

Published as a conference paper at ICLR 2026
Grant Rotskoff and Eric Vanden-Eijndent Parameters interacting particles: long time convergence and asymptotic error scaling of neural networks. In
Advances in Neural Information Processing Systems 31. Pp. 7146-7155.
Curran Associates, Inc., 2018. URL. htttp://papers.niss.
7945- Saas-paarmeters--a inw.rncttnng
pdf
Andrew M Saxe, James L McClelland, and Surya Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 (2)):11557--11546, 2019. ISSN 0027-8424. doi:
URL https
www. org/ wwwttotttttttccccco http://oootntttt/////i1/
James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pp. 31852-31877 1876. PMLR, 2023.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning mlr
Research, pp. 9722-9732 PMLR, 18-24 Jul 2021. URL. press/ /vas/vl.5 / mmsimmseklll. ht.l
Samuel Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176. 2021.
Zhenfeng Tu, Santiago Aranguri, and Arthur Jacot. Mixed dynamics in linear networks: Unifying the lazy and active regimes. to appear at NeurlPS, 2024.
Loucas Pillaud Vivien, Julien Reygner, and Nicolas Flammarion. Label noise (stochastic) gradient descent implicitly solves the lasso for quadratic parametrisation. In Conference Learning Theory, pp. 2127-2159. PMLR, 2022.
Zihan Wang and Arthur Jacot. Implicit bias of SGD in lz-regularized linear DNNs: One-way jumps from high to low rank. In The Twelfth International Conference on Learning Representations, 2024.
net htt.://www..ttrrrrroo.ooooooooo.
Yuxiao Wen and Arthur Jacot. Which frequencies do CNNs need? Emergent bottleneck structure in feature learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 52779- 52800. PMLR, 21-27 Jul 2024. URL httpps://ppoceeeeeeeeete wen24d. html
mmlr. www.p/reessv355
Blake Woodworth, Suriya Gunasekar, Pedro Savarese, Edward Moroshko, Itay Golan, Jason Lee, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models, 2020.
URL httphhtpss//peeeeiiiiieee
Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks, 2020.
Yedi Zhang, Andrew M Saxe, and Peter E Latham. When are bias-free relu networks effectively linear networks? Transactions on Machine Learning Research, 4:1-36, 2025.
13

---

Published as a conference paper at ICLR 2026
A PROOFS OF THEOREMS
A.1 SMALL DERIVATIONS
Let us first few derivations of equations stated in the main.
First, Equation
|
which follows from Taylor expansion of the cost C' w.r.t. the output labels:
(
C(Y))
-C(0) + Σ OY. CoYo) + (11+)1))))
C(()) + Tr [VC(O) + +(1))))))))
As concrete example,wa we have for the Mean Squared Error C(Y) # - Y*1 for some 'true" labels Y", the equivalent three terms are:
(()) N Y Y
N 18
2 N
Tr(y
N Yol
since VC((0) = - x Y
Second, the homogeneity of localized loss (0) = Tr(CC(0)7] follows from the fact that
all parameters rescaled by factor of À then the first activation will be multiplied by A (because (x) = o((W)))) = \o(W ), the second activation by X² (because ar2 (ar o ((A (x))) (W (x))) and so until the outputs which are multiplied by A4, The
localized loss will have the same prefactor of X thanks to the linearity of the trace:
Co(20) THVC(())( = Lo).
A.2 GRADIENT FLOW HOMOGENEOUS LOSSES
Proposition A.1. On a homogeneous loss, the GF dynamics decompose into dynamics of the norm ||@]| and of the normalized parameters 0 0/100:
0.)10(()]| = -0()) = -11000011---00000
0,0(t) =
(Lo(O()))
Proof. Since Co satisfies gradient flow with respect to 0, we have
d0 = =
dt
Because Lo is L-homogeneous, Euler's homogeneous function theorem implies:
(0)
Now, define the normalized parameter
0
|
Differentiating 0 with respect to time using the quotient rule yields:
dō d
0
de de
|00|
de
dt
dt ||@||
|012
Substituting (o) gives:
dg
||0|
0
dt
|
14

---

Published as a conference paper at ICLR 2026
To compute di ell note that
||0|| = 0)1/
Differentiating, we obtain:
d |0||
1
0 d0
|0| 0T ( (-VC
dt
|00|
dt
Using the homogeneity property 0 (O) (0). this simplifies to:
1
L Lo(
dl
|10|
Substitute this back into the expression for
de
di
d0
(o)(
|0l| + Co(
dt
||011
This can be simplified as:
d8 Co)
L Lo))-
dt ||0||
|013
We wish to express the right- hand side in terms of 0. Using the scaling property of the gradient for homogeneous function, we have
(o) \
and recalling that we finally obtain: d0
0 (0) = Lo())),
- -100 10000-00000000 (11) (Co(O).
di
A.3 EXPLOSION IN ESCAPE DIRECTION
Proposition A.2. If some time to the parameter satisfies
O(to) = with p 122P-1 and VLo(p) - then all 12 to the normalized direction remains constant, and the norm [[({t]] satisfies
(110(to)11 2-1. +(2 Lt (t) [|0(ta)] exp (2s (xp(2s
# 2,
[[())]]
- (t-to))
if =
Proof. Using the chain rule we have
0(t) d0
de |0(0)) = [((())]
| 0(t) (V.
di
Using Euler's theorem,
0(1) (Lo L Lo((t))
we obtain
L di |0(1)) = ||Ø())] (o(1))
Since ((t) (I0())](((t) and by homogeneity
(o()(1) 110(0)11 L. (0()),
15

---

Published as a conference paper at ICLR 2026
and because ((t) O(to) for all t 2 to with ((((tt)) -s we deduce
Lo(O(t)) -8 -|(((t)1
Substituting this back, we have
11(001 = [((()]] (-+100001*) L. 10(00)
di
Defining R(()) = []O(t) the above becomes the separable ordinary differential equation
dR
L R(tn) = [[0(to)]l-
dt
Case 1: L o 2. We separate variables:
R RI-L dR = Is dt.
Integrate from to
R(t)
R²- dR
Lidr.
R(to)
The left-hand side integrates to
2
R(1)2-L 12
2 L
2- L
R(to)
Hence,
R(t)² 2(t))-L100 R(to) 2-L
L8 (1 lo)-
2-1.
Solving for R(t) gives
R(t)2-1- = (1122-1=R((tt)) =R(to)) + +(2
L)L (t to),
or equivalently,
[|({t]] (110(to)) 2-L + (2 L)L (t - to)) st.
Case 2: L = 2. The ODE reduces to which is linear. Its unique solution is
dR
2s R. dt
R(t) R(to) exp(2 (1 to)),
that is,
||@(t)] ||0(to)] exp 22 (t to)).
A.4 OPTIMAL SPEED IS INCREASING IN DEPTH
Proposition A.3. Given depth network with Co(0) - 80 0 and 1011 = L, we can construct a network of depth L + k for any k 2 1 with parameters 0' that satisfies 1|0' = L +
and Co(0") < Co(0). Therefore, the optimal escape speed s" (L) is a non non-decreasing function.
Furthermore, in the deeper network, we have Rank(Zz]) = Rank WT = for all L' > L and Zy 27, for all L' L.
Proof. We denote with Wx.: the i-th column of WN and with Wz, the i-th row of W.. We can decompose the trace using the columns Wi,s and rows WI-1t in the following way:
Tr Zı] = Σ Tr WI, (WL-1,
16

---

Published as a conference paper at ICLR 2026
The negative contribution is entirely due to the Wz matrix the application of the activation function yields a non-negative matrix. For this sum there exists some i" [wr] that maximizes the negative contribution that for all i E [wz]]
Tr(G' WW 2)] & TH(G" W WWL-
where # denotes the normalized vector F ||||| We define network of depth L + i using the following matrices We:
WL-
Σ W.L.alllWL-.
WL-1=
WL*K
W W. L. 0 0
0).
0 0
0 0 0
We
l L ...,, L+ k
:
0
We We (= 1,2, ..., 2
We observe that the trace of the new network is lower equal to the trace of the original network:
Tr GT = Tr =
We
= =Tr[G (W.-- 7.
Σ
Wr.
(r.11,
"z
I. (w.,o((
Tr WW
Tr Gr Zz).
The norm of the network is
L-2
wz
11012
Σ
Well +2 Σ
WL
L-1,6el +
i=1
|Wall +
t=1
L
17

---

Published as a conference paper at ICLR 2026
A.5
Low RANK BIAS
A.5.1
WEAK CONTROL
Before presenting results will describe simple lemma that's ' useful to proofs.
Lemma A.4. For depth-L network with ||02| || I L we have that
Well s
Proof. This essentially follows from the AM-GM inequality:
11012
L )
1
We
})}
Welle
and using the bounded norm assumption
|1011 $
1 = 12
L
Proposition A.5. Given that Tr[G' Σr] < 880, for some constant so > 0 and [101] < L then for
any ratio p (0,1) there are least (1 D)L layers that are approximately rank in the sense that the singular values s; of Z satisfy
Ei> 8
log || X|| r
log |G|| F
log 80
2
(3)
Σi 82
pL
Proof. We expand the activations,
| lP:
1
L- 1
|
2 1-1 op
| 11
1
F
1Z
2
7 11
t=1
t=
Where the operator norm of matrix is its largest singular value. Since Zr = W,Zg we have
2
< || ||
So by using lemma A.4
17
12p L-1
|2g_
112p
L-1
L-1 II 8=
|27 12
1
||
1 op
|2
E=
On the other hand we have that
1 I
1 7z
TH 2r.
since the inner product is always lower than the norm product.
Now by combining the above we get
18

---

Published as a conference paper at ICLR 2026
Tr Zz]²
Taking the log on both sides,
| 1 GG
Σ log 23 1
log 2 18 S log
Tr|GT Zr
By contradiction, we see that for any ratio p E (0, 1), there can be at most p7 layers where
log || + log
||
log 80
log Ze log [e
pl.
That is there is at least (1 - L layers where
S of
1e 12
log || F
log ||]]] log 80
S
2og ZZllr - 2 log Zell S
Σ Σ121
1z 2
PL
A.5.2 STRONG CONTROL ON ALMOST RANK 1 INPUT
The following result shows that the input of the network is approximately rank-1, 1, here encoded as uv" + X, where u. are non non-negative entry-wise vectors and X is matrix of small norm ||||e c, then all layers are approximately Rank-1 I too at the optimal escape direction.
Proposition A.6. Consider the minimizer 0* = arg min 1011 Tr Tr[G Yow + x)] where u, E R", u, v 2 0 entry wise and ||X|| I for some e 0. Thenfor all we have:
Eez (WW)) (
1
Ze
||]||
<
((W)) Σ12 (Z())
1 7
8
|G||Fe
Proof. In the case where the input is only the rank 1 matrix uo
that
Gr Yo(uTT] Tr (.TT)) = G Yo(u)
and therefore the minimum is achieved when the aligmment is maximized:
m T - ||
When |0|2 < it is true that
[Yo(u YoluoT X)Ile & II Well Se
t=1
consequence of the Cauchy-Schward inequality.
We can also see that
Tr + +x]] [T Yo(uT) )]] s |G]|ec.
At the minimum 0+ = -arg min Tr T
YoluoT + +X) we observe that
Tr Tr[G Yo- Two - xx] < [ Yg(uv + +x] < -||Gv|||u|| + |G]|
19

---

Published as a conference paper at ICLR 2026
where 0 arg min Tr |G' Yo(u=") On the other direction we get
Tr Yo. (uvJ +x)]] 2 - Tr[G'' Yo. (ue (u=T)] - | FE
- -- (w)ll - |G
(4)
where we used the Schwarz inequality in the last line. Combining the two, get
Yg (u)ll
2|
|
|GG|||||||
and since |101² < L it is also true that the LHS of the above inequality is upper bounded by 1.
can also see that
[[ (a)il Nai ||
1 (uo + X)Il I + XII
[ (uv T) -
1
Uw T| 1| +
1
Toll
and by using the above inequality get
2 G G
(uv + +XII
| F |Gu |||
To 1|
2(
| )
I + X||
Now expand the activations,
(uv" + XXIl
L II
7
| ||
|uw + +XI
Z
and using the fact that IIe |WWllr 1
We W.ZZe1lle
12%-
L
[We
- 111
127-111 12
πI
II
(5)
W 11 IllF
1g
l|
We split the norm of the activations using the inequalities
We WWZ7-ille IF
Well 27 IIF
Wellop |
W | 1
[Well F 12
and
||,
|| 12-11lop Welle|lZg_,ll
1|08_1lop
|ell
||2g_|11
so we get
Wellop | [ell |2g_ 1| 1Ze-ill
12g1ll - 2(
||G|| |Gullllu||
II min
|all|al|| )..
By squaring and rearranging the terms get for the first ratio:
20

---

Published as a conference paper at ICLR 2026
Ei>g (Wa))
|
<4(
||l|||||| )
Σiz1$ (We)
| ||]
which further simplifies to
Eize (We)
|G|| Gull|u
<
E Eiz1 87 (Wi)
Using inequality 4 we observe that:
|Gr||lbu|| = =
and hence
Σiz2 of (W,) Ei (We)
|||||
S8 8* G
We proceed similarly for the singular values of Ze.
For the matrices Ze and 2z we note that their Frobenius inner product is zero, so
ZZel = 12 Ze 1
Re-arranging gives the inequality
|2
Zell?
|
<
/Ze
|
A.5.3 STRONG CONTROL
We can combine the above two statements to show that at the maximum escape speed, the final layers will be almost rank-1. 1. To prove that we first apply proposition(A.5] find layer L that is almost rank-1. 1. We need an additional lemma that ensures that we can select non-negative singular vectors for the largest singular value of the activation Zz,- Then we can apply proposition A.6 conclude that all layers& 2 to will be approximately rank-
Lemma A.7. For A Rmxn with non-negative entries and $1 its largest singular value we can find right and left singular values u1 Un for $1 which are non-negative entry-wise.
Proof. The right singular vector for $1 satisfies
A Au 8141
and since A is non-negative A T A is also non-negative. We can now apply an extended version of the Perron-Frobenius theorem for non-negative matrices select the eigenvector ii 2 0 entry-wise. Now we select
Auj
U1
81
which is a left singular vector as it satisfies Av = 8 Uj and since A, u are non negntegat is also non-negative.
21

---

Published as a conference paper at ICLR 2026
Theorem A.8 (Theorem 3.1 in the main). Consider optimal escape direction
0' arg min Tr Tr Yo) |01l=t
with optimal speed s" minlooia=1 Tr G Yo). then for all layers i we have:
Eizg* (We) Eizg (zg)
1 Zelle
38
l-
Σ121 8t (We)
Σii1 (Zg)
8 - cl
where /2|1X|| G|| PV log X log |G|| log 8*.
Proof. We denote Y2 (X) = WW X)... the network when only the layers from l to l2 are applied.
Using proposition A.5 we can find layers to < lı ... en S L that satisfy 3 and are approximately 1ank-1. We can select the minimum those, to-
Because the argument is valid for at least (1 L)L of the L total layers, the earliest layer to must occur on or before the pL-th layer.
It is true that 2% is non-negative entry-wise and so can apply lemma A.7 to find non-negative singular vectors U1 U1 that additionally satisfy
log X +
log |G]|
log s*
11 91 (1 1i )1
of < 2
pl.
im 2
2 X log
X F +
log
|
log s*
pl.
We now use the fact that for the layers & 2 to we are at the optimal escape direction 0*
0"
arg min 00
Tr(G 1
We do that since:
min 11 TT TT(GTTY) Yg] < minn t1 P=LLE Tr[G
TOGTYL.too L (2to)
We can now apply proposition A.6 on the sub-network VL For all layers l to we have that:
Σi>g (We) (We)
Σ s (Z7) Σx21 (Zzg))
12
Zell
Ei21 (W,))
VpL. VpL
where | GllF
2 log
||
We see that, since p E (0. 1) was chosen arbitrarily, we can choose p 8 for any f. Then since to 5 pL l the above inequality will hold for any f-th layer with f > c.
I
22

---

Published as a conference paper at ICLR 2026
B MNIST TRAINING DETAILS
We train a 6-layer fully connected neural network (multilayer perceptron, MLP) without biases on the MNIST dataset, using the cross-entropy loss. The network comprises one input layer, four hidden layers, and one output layer. Each hidden layer contains 1000 neurons. The weight matrices have the following dimensions:
Input layer: Wi E R 787x
Hidden layers: W; 11100010000 for i 2,3, 4,
Output layer: We R 1000
The weights are initialized from normal distribution with mean 0 and standard deviation 1/ 1/1000.
We train the model for 1000 epochs using a batch size 32. The learning rate at each step is adjusted dynamically according to:
10
Ir(()
[(t))
where
10(1)12 Σ IWi(1
and
|| || denotes the Frobenius norm.
Each MNIST image is normalized using the dataset-wide mean and standard deviation of the pixel values:
x/255 /
This standardization ensures that the input distribution has approximately zero mean and unit variance, which helps stabilize training.
A more complete picture of how the singular values of the weight matrices evolve during training is presented in
We repeated the same experiment with depth-4 fully connected network and we report our findings in figure
23

---

Published as a conference paper at ICLR 2026
Layer
Layer
Layer
Log Singular Value
Log Sinqulas Value
Log Singui ar Value
101-
10"
1.0
1.8
10
1.2
1.4 step
1.0
1.6
3.0
step
step
Layer 4
Layer 5
Layer
Log Singular Value
Singular Valuee
Log Singular Valuee
107
10a
101
30-
10"
10-1
10
1.6
3.0
10
1.2
1.4
1.6
16
1.6
1.0
step
step
Step
1.5
0.5
step
Figure 3: Deeper layers show stronger bias toward low-rank structure than earlier layers on MNIST. Top two rows: Top singular values of the weight matrices for layers 1-6 including input and output layer over training time. Bottom: Training loss trajectory MNIST.
C CIFAR-10 10 TRAINING DETAILS
We train 6-layer fully connected neural network without biases the CIFAR-IO dataset, using the cross-entropy loss. The network comprises one input layer, four hidden layers, and one output layer. Each hidden layer contains 1000 neurons. The weight matrices have the following dimensions:
Input layer: W1 E R 1024x1000
Hidden layers: W; E 1000x 1000 for i = 2, 4,5
Output layer: W E IR 10000110
The weights are initialized from normal distribution with mean 0 and standard deviation 1/ 1/1000.
We train the model for 5000 epochs using a batch size 32. The learning rate at each step is adjusted dynamically according to:
40
Ir(()
[((())
where
6 10(1))12 = Σ (W)(
and || denotes the Frobenius norm.
Each CIFAR-10 10 image x is normalized using the dataset-wide mean f and standard deviation a of the pixel values:
24

---

Published as a conference paper at ICLR 2026
Layer 1
Layer 2
2.0-
3.0-
Singular Value
Singular Value
2.5
1.5-
1.5
1.0
0.5-
0.5
0.0
100000
300000
500000
100000
300000
500000
step
step
Layer 3
Layer 4
3.5
3.0
Singular Value
Singular Value
3.0
2.5-
2.5-
2.0
1.5
1.0-
0.0
0.0
100000
300000
500000
100000
300000
500000
step
step
2.0
1.0
0.5
0
100000
300000
500000
step
Figure 4: Depth-4 MLP with small initialization on MNIST. Top two rows: Top 10 singular values of the weight matrices for layers 1-4 including input and output layer over training time. Bottom: Training loss trajectory on MNIST.
x/ x/255
/
more complete picture how the singular values of the weight matrices evolve during training is presented in
We repeated the same experiment with depth-4 fully connected network we report our findings in figure 6 For the depth-4 network used the learning rate Ir(t) [[0(e] ensure convergence.
0,01 and 10000 epochs to
25

---

Published as a conference paper at ICLR 2026
Layer
Layer
Layer
10
10a
10
10-3
Step
Step
step
Layer
Layer 5
Layer
102
10
105
105
10-3
10a4
10"4
2
Step
Step
5,(W,)
6-
1
2
3
5
6
2.0
1.5
0.5
0.0
0
2
7
step
Figure 5: Deeper layers show a stronger bias toward low-rank structure than earlier layers on CIFAR- 1-0. Top two rows: Top 10 singular values of the weight matrices for layers 1-6 including input and output layer over training time in logarithmic scale. Third Row: Top 10 singular values of the weight matrices We across layers 1-6 (including input/output), Left: After the first saddle point escape. Right: At the end of training. Bottom: Training loss trajectory on CIFAR-10.
D SUPPORTING MATERIAL FOR SECTION 4
D.1 FINDING THE MAXIMAL RANK-ONE ESCAPE SPEED
Picking up the argument from the proof sketch of Example I we have network function equal to
f(X) = (W, (W,X), X), where W = [cos(o) sin [o) and the sign is chosen to give a positive escape speed. Applied the dataset of Example T and noting that most four points will have nonzero
26

---

Published as a conference paper at ICLR 2026
function value at given time, finds escape speed is equal to
cos { π 4 cos(E) cos ( 4
cos
{
π 7 2
(6)
where ç Φ mod ) See Figure
7 for depiction of this periodic function. Its maximal value of
v2- falls at multiples -
27

---

Published as a conference paper at ICLR 2026
Layer 1
Layer
10".
109.
10
10
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
step
step
Layer 3
Layer 4
10a
10-2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8 step
1.0
1.2
1.4
1.6
step
166
8
6
s,(W,)
5;(W,)
2
2
3
1
1
2.3
2.2
2.0
2.1
2.0
1.9
1.0
1.8
1.7
0.5
1st saddle escape 2nd saddle
1.6
0.0
0.2
0.4
0.6
1.0
1.2 1.4
0.0
0.2
0.4
0.6
0.8 step
1.0
1.2
1.4
step
Figure 6: Depth-4 MLP with small initialization on CIFAR-10. Top two rows: Top 10 singular values of the weight matrices for layers 1-4 including input and output layer over training time in logarithmic scale during the initial 1000 epochs. Third Row: Top 10 singular values of the weight matrices per layer for layers including input and output layer after the first saddle escape and at the end of training. Bottom: Training loss curves on CIFAR-10 10. Left: The initial 1000 epochs. Right: The full training run of 10000 epochs.
28

---

Published as a conference paper at ICLR 2026
0.4
0.3
escapa speed s
0.2
0.1
0.0
3x 2
2T
Figure 7: Visualization of Equation
6
width 4
width 8
width = 16
width = 332
0.5
loss
20000
0
20000
20000
20000
epoch
epoch
epoch
epoch
width 64
width 128
width 256
width 512
0.5
0.5
0.5
0.5
loss
0.0
0.0
0.0
0.0
0
20000
0
20000
20000
0
20000
epoch
epoch
epoch
epoch
width 1024
successfuul free
loss
0.5
20000
101
10a
epoch
width
Figure 8: Visualization of all training runs of projected gradient descent on Example shows all training runs in the experiment of Figure 2
This plot
29


Reviews:
[{"review_id": "NESUMQBcqx", "reviewer": "Reviewer_gV27", "summary": "This paper analyzes how gradient flow (GF) escapes the saddle point at the origin when training deep ReLU networks initialized with small weights. The authors characterize the optimal escape direction of GF and show that it exhibits a low-rank structure in the deeper layers, where the leading singular value of layer $ \\\\ell $ is at least $ \\\\ell^{1/4} $ larger than the other singular values combined. They further suggest that deep ReLU networks follow saddle-to-saddle dynamics, where the weights transition between saddles of progressively higher rank during training.", "questions": "1. Please explain the relation between the optimal escape direction from the saddle at the origin and the actual dynamics of GD. \n2. Can you give a way to convert the speed defined by GF to GD?\n3. In line 189, it says that you consider the case of adaptive learning rate, where $ \\\\eta = \\\\| \\\\theta \\\\|^{2-L}  $. Wouldn't it be extremely large near the origin for deep networks? \n4. In the equation of the escape time (line 234), the dependency of $  t(r\\_1) -t\\_0$ on $ r $ for deep networks, given by $ r\\^{2-L} $, is strange. It predicts that crossing smaller thresholds $ r $ takes a longer time (since $ L > 2 $), which does not make sense. Is there a problem with this result?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper studies an interesting topic of escaping saddle points. The problem setting and assumptions are clearly presented, and the authors support their analysis with illustrative examples and well-designed figures. Overall, the paper is clearly written and effectively communicates its main ideas.", "weaknesses": "Although the authors depicted a detailed picture of the dynamics of GD in the vicinity of the saddle point at the origin, I am still not convinced that this is an actual phenomenon for the following reasons.\n\n**Initialization magnitude:**\nThe current initialization schemes avoid initialization at the plateau of origin. In other words, using the standard initializations, we would not encounter such a setting.\n\n**GF vs. GD:**\nThe analysis presented in the paper applies to GF. However, these results cannot be translated directly to GD. Specifically, GD with non-vanishing step size exhibits richer dynamics with dynamical stability constraints along its trajectory (e.g., Edge of Stability). Moreover, the optimal escape speed should be adjusted according to the stability constraints.\n\n**Optimial vs. typical escape direction:**\nThe presented results on low rank are about the optimal escape direction. However, there is no proof that GD (or GF) will follow this escape trajectory. It might be the case that the typical escape routes of GD are non-optimal. In this regard, I don't understand the claim of GD bias towards low rank solutions, since there is no guarantee that GD will follow this optimal direction.\n\n**Effect on the fully trained model:**\nThe main result deals with the state of the model in the first stage of training. I am not sure what effect a low-rank model early in the training has on the fully trained model. I am not convinced by the claim of saddle-to-saddle dynamics without any proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfZeQVgZxq", "reviewer": "Reviewer_bYb4", "summary": "Training of neural networks can happen in two main regimes: lazy and rich, the latter further split into an active and a mean-field regime.\nThis paper studies the early training dynamics of deep ReLU networks initialized with small weights, focusing on the active training regime, specifically the saddle-to-saddle phase where gradient flow (GF) escapes from the origin saddle in parameter space.\nAfter introducing a *localized loss* approximating the true loss at small scale, norm and spherical dynamics are separated with the latter evolving faster and identifying escape directions, which are minima of the loss constrained to a sphere and therefore points where the norm dynamics can take the lead.\nThe theoretical results of the paper revolve around conditions imposed on the parameter at escape directions, in particular a low rank bias (seen through singular values of weights and activations matrices) and the collapse of inner activation layer to identity layer, with both these facts amplified with depth of the layer.\n\nA validation numerical experiments is provided and a counter-example is given to disprove a conjecture that the escape direction is always rank one and, finally, the authors propose conjectures about saddle-to-saddle dynamics.", "questions": "I have a few questions:\n1. **On the parameter norm $r$ of line 228**: can $r$ be estimated to know the scale at which the approximation of GF by the localized loss ceases to hold ?\n1. **On the MNIST experiment**: $1000$ hidden neurons are used in each of the 6 layers. In my experience,this level of expressivity exceeds what is needed to fit MNIST with an MLP.\nIs there a particular reason for choosing this large width?\nMore generally, the training setup seems carefully tuned: many training steps ($1000$ epochs of relatively small batches), learning rate scheduling, input standardization.\nHow necessary are these design choices ?\nCould you comment on the sensitivity of the results to the training details ? (Except the initialization scale since it is the focus of the paper)\n1. **On figure 3**: what do you observe if you plot $\\frac{s_i}{s_1}$ i.e. the ratio of each singular values by the highest one ?\nThis ratio plot might make it easier to visualize the dominance of the top singular value discussed in the paper.\n1. **On conjecture 4**: the paper seems to be a step toward a full characterization of incremental learning in ReLU networks beyond the first saddle.\nIn your view, what are the most promising long-term practices this line of work could lead to, e.g. improved interpretability via hierarchical learning, overfitting diagnostics, redundancy-based architecture search, or new training strategies that separately address the spherical and explosion phases (as in Kunin et al., 2025)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The submission appears technically sound, with a mathematically precise yet elegant formalism. The main claims are well supported by both theoretical proofs in the appendix and numerical validations (though not extensive).\nThe paper is clearly written and well structured, conveying a complete and coherent message.\nWhile the authors do not provide code, the experimental setup is described in sufficient detaisl in Appendix B to allow reproduction, except for the unspecified method used to compute the top 10 singular values, which I think should be precised.\nOverall, I have a positive impression of this work and consider it to be of high quality.\n\nThe analysis of where the rank-one layer are located, along with the proposal of a half-bottleneck structure, is particularly insightful as it helps reveal (during training) where the network’s interesting computations occur and therefore help with interpretability.\n\nWith the insights provided by the paper, there are clear ways forward, specifically, extending the analysis beyond the first saddle escape and exploring how learning might understood iteratively across subsequent saddles.\nThese potential research paths are further reinforced by the conjectures presented in the discussion section.", "weaknesses": "The paper presents a descriptive rather than prescriptive analysis and combined with the theoretical treatment, it limits the immediate practical consequences but the approach is appropriate given the paper’s objectives.\nAlso, the experimentation are limited in scope and the setup on MNIST seems a bit specific (see the related question below).\nI think the paper could benefit from a stronger connection to practical implications, even long term, to help engage a casual reader: although the introduction situates the work well within related literature, the paper overall offers few explicit takeaways or long-term motivations for practitioners. (see last question)\n\nLimitations are:\n- The setup does not include biases\n- The main result applies for optimal escape directions, however it is not guaranteed that the found escape direction will be optimal under gradient flow (though with the nuance that another result, proposition 3.4 which applies more broadly)\n- The formalism is continuous while in practice discrete gradient descent is used\n- Emphasis on deep networks:\n    - the condition $l>c^2$ is proven to hold on all but finitely many initial layers; which is not a clear practical information\n    - the bound is in $l^{\\frac{1}{4}}$ which is not so strong (ratio $\\approx 3$ for depth $l=100$), which is nuanced by experiments where the effect is clearly visible\n\n## Minor and additional feedback\n- line 453: \"is greedily searched by first searching among BN-rank one functions\" consider rephrasing to avoid repetition e.g. \"is greedily search, first among BN-rank one functions\"\n- line 96: the word \"step\" is a bit ambiguous, something like \"Dynamics with multiple saddle-to-saddle sequences\" could be clearer.\n- line 109: consider changing \"activation\" to \"preactivation\"\n- notation overlap in sections 2.1 and 2.2: $s$ denotes both the reparamtrized time and the escape speed\n- line 203 & 239: in my understanding this dynamics holds because $\\|\\theta(t)\\|$ evolves more slowly than $\\overline\\theta(t)$ as it can be seen in equations lines 190. If this is correct, I would recommend precising it explicitly somewhere.\n- theorem 3.1 is named A.8 is appendix\n- line 323: appendix 3 but appendices are labelled with letters\n- line 427: probably you meant \"escape speed\" instead of \"escape direction\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fR3RNWFrdc", "reviewer": "Reviewer_Bqc1", "summary": "This paper gives a description of the saddle at the origin in deep ReLU networks, and the possible escape directions that GD could take as it escapes this first saddle. It were shown that the optimal escape directions feature a low-rank bias that gets stronger in deeper layers, and the optimal escape speed is non-decreasing in depth. A simple dataset is presented whose optimal escape direction is rank two for the first weight matrix. Finally, a few conjectures/hypotheses describing the complete Saddle-to-Saddle dynamics are discussed.", "questions": "see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The theoretical analysis seems sound, although I did not check the details of proof.\n2. Both optimal escape direction and speed are analysed. \n3. Figure 1 intuitively illustrates the saddle escape behavior and the low-rank bias feature on the MNIST dataset.", "weaknesses": "1. In experiments, the low-rank bias feature is only demonstrated on the MNIST dataset. Results on more datasets are preferred.\n2. The motivation is not clear enough. Why should we study the saddle-to-saddle regime for very small initializations? In practice, the weights are initialized with, e.g. Kaiming initialization, which may not fall into the very small initialization regime.\n3. What is the practical implications of low-rank bias feature for network training? Does it mean pathological loss landscape? How to use it to escape the saddle more effectively and efficiently? \n4. The complete Saddle-to-Saddle dynamics are only given as hypotheses without further proofs or empirical evidences. The main technical contribution is focused on the first saddle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WiX8mIRsby", "reviewer": "Reviewer_Y4Ze", "summary": "The paper provides an analytical study of the low-rank bias observed in the first escape direction in multi-layer ReLU networks. In fact, when the network is initialized with small weights, it needs to escape from the saddle point present at the origin, and the direction of this escape was previously shown to be biased towards low ranks. The paper formally proves that the escape direction is biased to have approximately rank 1 in deeper layers. A simple neural network trained on MNIST validates the theory and an analytical counterexample is provided to confirm that the rank of the optimal escape direction can be higher than one and thus an approximate notion of rank one is necessary.", "questions": "- At L171 it is claimed that \"since the ReLU is not differentiable, neither is the loss at the origin, so that we cannot use the traditional strategy of approximating L0 by a polynomial.\". How is the approximation of Eq. (1) justified in light of this fact? \n- How much do the authors think the result extend (if they do) to the case of regularized losses or other activation functions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The background literature in Section 1 is very well presented. The reader gets a clear picture of this line of research, how previous works contributed to it, and how this paper fits into the narrative.\n- The main results (Theorem 3.1. and Proposition 3.2.) are novel and provide an interesting theoretical advancement in the context of understanding saddle-to-saddle dynamics.", "weaknesses": "Although both theoretical and experimental examples were given to corroborate the claim, the latter are very limited in scope, with only one experiment performed on one simple architecture and dataset. I recognize that this is mainly a theoretical work, but a more robust numerical evaluation would have been welcome to validate the theory.\n\nThe results mainly concern the specific case of standard ReLU MLPs with no regularization (otherwise the loss is not homogeneous) and trained with gradient flow. These are standard assumptions in the literature but their strict nature limits the scope of the results.\n\nI believe that the authors could improve the readability of the paper for a reader that is not an expert in this line of research:\n- The approximation around the origin of Eq. (1) should be derived in the text, in the appendix, or a reference where such derivation is made should be referenced.\n- The same holds for the homogeneity of the loss at L173. A good reference plus an intuition on where it comes from would be enough I think.\n- The derivation of the equation at L180-181 (which should be numbered) in the appendix should be referenced in the main text. This should hold for all other derivations in the appendix.\n\n### Minor weaknesses\n- Please provide full forms of all abbreviations before using them, e.g. DNN (L025), NTK (L029), CNN (L076)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)