Intrinsic training dynamics of deep neural networks
Authors: Sibylle Marcotte
Gabriel Peyré
Rémi Gribonval
Venue: ICLR 2026 Poster (2026)
Keywords: gradient flow
path-lifting
intrinsic lower dimensional dynamic
conservation laws
implicit bias
Primary area: learning theory

Abstract: A fundamental challenge in the theory of deep learning is to understand whether gradient-based training can promote parameters belonging to certain lower-dimensional structures (e.g., sparse or low-rank sets), leading to so-called implicit bias. As a stepping stone, motivated by the proof structure of existing implicit bias analyses, we study when a gradient flow on a parameter $\theta$ implies an intrinsic gradient flow on a ``lifted'' variable $z = \phi(\theta)$, for an architecture-related function $\phi$. We express a so-called intrinsic dynamic property and show how it is related to the study of conservation laws associated with the factorization $\phi$. This leads to a simple criterion based on the inclusion of kernels of linear maps, which yields a necessary condition for this property to hold. We then apply our theory to general ReLU networks of arbitrary depth and show that, for a dense set of initializations, it is possible to rewrite the flow as an intrinsic dynamic in a lower dimension that depends only on $z$ and the initialization, when $\phi$ is the so-called path-lifting. In the case of linear networks with $\phi$, the product of weight matrices, the intrinsic dynamic is known to hold under so-called balanced initializations; we generalize this to a broader class of {\em relaxed balanced} initializations, showing that, in certain configurations, these are the \emph{only} initializations that ensure the intrinsic metric property. Finally, for the linear neural ODE associated with the limit of infinitely deep linear networks, with relaxed balanced initialization, we make explicit the corresponding intrinsic dynamics.

Published as a conference paper at ICLR 2026
INTRINSIC TRAINING DYNAMICS OF DEEP NEURAL NETWORKS
Sibylle Marcotte ENS PSL Univ. s sibylle lle wwm.cotteeeess.... fr
Gabriel Peyré
CNRS, ENS PSL Univ. gabriel prerieelppereeeeeee
Rémi Gribonval
Inria, CNRS, ENS de Lyon,
Université Claude Bernard Lyon 1. LIP, UMR 5668, 69342, Lyon cedex 07, France remi www.iggribbialioooooo @inria.fr
ABSTRACT
A fundamental challenge in the theory of deep learning is to understand whether gradient-based training can promote parameters belonging to certain lower- dimensional structures (e.g., sparse or low-rank sets), leading to so-called implicit bias. As stepping stone, motivated by the proof structure of existing implicit bias analyses, we study when a gradient flow on a parameter 0 implies an intrinsic gradient flow
"lifted" variable (0)), for architecture-rellted function db We express a so-called intrinsic dynamic property and show how it is related to the study of conservation laws associated with the factorization ø. This leads a simple criterion based on the inclusion of kernels of linear maps, which yields a necessary condition for this property to hold. We then apply our theory to general ReLU networks of arbitrary depth and show that, for dense set of initializations, it is possible to rewrite the flow as an intrinsic dynamic in a lower dimension that depends only on and the initialization, when Φ is the so-called path-lifting. In the case of linear networks with o, the product of weight matrices, the intrinsic generalize
dynamic is known to hold under so-called balanced initializations; this to broader class of relaxed balanced initializations, showing that, in certain configurations, these the only initializations that ensure the intrinsic metric property. Finally, for the linear neural ODE associated with the limit of infinitely deep linear networks, with relaxed balanced initialization, make explicit the corresponding intrinsic dynamics.
INTRODUCTION
A central question in deep learning theory is how the complexity of gradient- based training can give rise to simpler, lower-dimensional dynamics. In this work, we explore when the gradient flow on parameters 0 naturally induces gradient flow on "lifted" variable <00), where captures structural aspects of the model.
Intrinsic lifted flow. The study of optimization flows arising in the training of neural networks often benefits from the identification of lowcr-dimensional intrinsic dynamics. Specifically, due to the natural symmetries of linear and ReLU networks, is of considerable interest rewrite parameter flow f(t) in terms an representation 2t)) = ((O(t)), using a suitable architecture- reparametrization ø (often called lifting) that factors out certain symmetries.
When dissected, the most advanced recent results characterizing the implicit bias induced by gradient- based optimization algorithms notably rely on two key analysis ingredients: (i) establishing that

---

Published as a conference paper at ICLR 2026
the dynamics of 2tt) is intrinsic, i..., that it can be expressed Riemannian gradient flow with metric depending only on 2 and the initial parameters 8(0): (ii) further proving that this flow on x(t) admits mirror flow representation. With the combination of these two ingredients one gains access to powerful analytical tools rooted in convex optimization theory, allowing explicit characterization of the induced implicit bias. In particular, prior research has successfully leveraged mirror flow formulations to rigorously demonstrate implicit regularization effects, such as sparsity in scalar linear neural networks and two-layer networks with a single neuron (Gunasekar et al., 2018). as well as maximum-man margii classification for logistic regression problems in separable data scenarios (Soudry
et al. 2018 Chizat& & Bach, 2020).
Recent work by Li ct al. (2022) identifies sufficient conditions under which (i)+ can both be established, requiring that the parametrization ø be commuting. However, this commuting condition is rarely satisfied in practical scenarios. This work focuses on characterizing weaker conditions ensuring that the flow on =tt) is still driven by an intrinsic Riemannian gradient flow (but not necessarily mirror flow anymore), which we believe is an important step forward and a starting point for future investigations encompassing variants of (ii) with warped mirror flows (Azulay et al. 2021) for practical (deep) network architectures. A first sufficient condition for (i), introduced by Marcotte et al. (2023). demands merely that the parametrization be involutive. Marcotte et al. (2023) have shown that this weaker condition applies specifically to the parametrization used in two-layer ReI.U networks (Stock & Gribonval 2022). As we will see, a consequence of the analysis conducted in our paper is the extension of this result to arbitrary DAG ReLU networks (Gonon et al. 2024).
Conservation laws. The functions conserved during the training dynamics play a crucial role in establishing that the dynamics of =(t) is governed by an (intrinsic) Riemannian metric that depends only on z and the initialization Ou == (()). Indeed, when a trajectory (()) is known to remain within level sets {0: h(0)= = h(0))) where is a velued- conserved function, the dynamics are effectively restricted to a manifold of lower dimension that is entirely determined by the initialization. A particularly important class of conserved functions along these trajectories is given by the conservation laws associated with certain prchitcctrre-eeendder parametrization şi, concept introduced in Marcotte et al. (2023). These laws depend exclusively on Φ. and notably, in the context of neural network training dynamics, they represent quantities preserved across trajectories irrespective of the initial conditions or the training dataset. In the specific case of linear and ReLU neural networks, these conservation laws correspond exactly to previously known "canonical" conserved functions identified in Du et al.
(2018): et al. (2019), as demonstrated by |Marcotte let al. (023);; see also Le & Jegelka| (2022) for canonical conservation laws for general DAG ReLU architectures. Furthermore, Marcotte et al.
(023) establish that if the parametrization é is involutive, there exist sufficiently many scalar conservation laws to fully rewrite the original trajectory (t) in terms of ((((t) and the initial conditions alone. In the linear network case, when so-called balanced conditions (a notion introduced in Arora et al. (019)) are satisfied (i.e., when the initialization sets all canonical conservation laws (Chitour et al. 2018) to zero, h(0)) 0), it becomes possible to rewrite the flow in terms of z = x(0), where & corresponds to the product of weight matrices, as an intrinsic Riemannian metric (Arora et al. 2018 Bah et al.) (022). Moreover, Achhuur et al. (2025) extended this result to linear convolational networks in the case of a mean squared loss, but this time for arbitrary initializations, with the Riemannian metric depending on the initialization. For linear networks and in the particular case when the loss function is the square loss, Bah et al. (2022) show that the trajectory evolves the manifold of matrices having some fixed rank under ba lnced condition. Still in the square-loss setting, and in the case of two-layer linear networks, Tarmoun et al.) (2021): Braun et al. (2022); Dominé et al. (2025) exploit the conservation laws obtain an exact closed-form expression for (t) under specific configurations, whereas Varre et al. (2023) use the same laws to analyse an implicit bias of this dynamic.
Our main contributions. We first define the notion of intrinsic dynamic property (Definition 2.6). then the notion of intrinsic metric property (Definition 2.10) and finally the one of intrinsic recover- ability property (Definition 2.15), and we show the implications (Lemma 2.11 and Lemma (.16):
Intrinsic Recoverability = Intrinsic Metric = Intrinsic Dynamic.
We then provide a simple criterion that characterizes the intrinsic recoverability property (Theo- rem 2.17.. ). and show (Proposition) 2.21) that this criterion is quasi equivalent to the Frobenius property, which is slightly weaker than involutivity (Definition 2.20p. We prove that the so-called path- path-lifting (Gonon et al. (024) reparametrization for general ReLU networks of arbitrary depth satisfies this property (Theorem 3.T. characterizing a dense set of initializations of general ReL.U network
2

---

Published as a conference paper at ICLR 2026
that satisfies the intrinsic recoverability property (Corollary 3.3), as illustrated by characterization of the intrinsic dynamic of 3-layer neural network (Proposition (35)). Next, by establishing a necessary condition for the intrinsic metrie property to hold based on the study of kernels of linear mappings Theorem [.14] we show that the intrinsic metric property fails to hold for the natural reparametrizatioos corresponding to 2-layer linear networks (resp. of attention layers), unless the initialization satisfies the relaxed balanced condition introduced in Definition (Theorem 3.9). We then show that relaxed balanced initializations do satisfy the intrinsic metric property, not only in 2 2-layer networks (Theorem 3.8) but also in linear networks of arbitrary depth (Theorem 3.11). and we characterize the resulting intrinsic dynamic. Finally, we extend our analysis to the infinite-depth limit of linear networks. We show that set of functions is conserved along the trajectory (Proposi- tion 3.12.. . and, in contrast to the case of L > 2-layer, we derive closed-form expression for the metric in the case of relaxed balanced initializations (Theorem 3.13
2 DYNAMICS OF OVER- EVER-PARAMETTE PER--PAAAAMERIEEI MODELS
In most machine learning models, overparameterization occurs due inherent symmetries (such as rescaling) within the parameter space A E ... In practice, this redundancy can be factored out through a function (often called lifting (Candès al. 2013 Gonon al. (0224) that captures these symmetries. Although the resulting lifted variable
(0)) E ER² often lives in higher
dimension d >> D, it also belongs to lower dimensional manifold 2 of dimension d' < D, and provides a representation of the essential structure of the model. We consider parameters 0(t) E R
that satisfy the gradient flow dynamic with some initialization 0y:
0(1) -=((((((())) 0(0) 00 0o
to minimize the function E. In machine learning, (()) is typically defined as the empirical average over training samples (Ia;; yi) of a quantity that depends on the output g(0, xx) of neural network with weights and biases collected in the parameter vector 0. The function g(0, x) can be locally reparameterized via the above rechitectureddeenddee for the global loss L. This is the starting point of our analysis, captured via the following assumption:
lifting d(0), leading to the same factorization
Assumption 2.1 (Local reparameterization of via d E CRRR ). on an open set Q S P). There is a function f E (12, I) such that
4O E St {(O) - f()(0).
(2)
The following examples illustrate common choices of & for various neural network architectures.
Example 2.2 (Linear neural networks). For a two-layer network with r hidden neurons and 0 = (U,V) E Raxr x mx (where D (n + m)r), the model g(0,z) = UV is factorized via the map ΦLin ()) == UVT E Raxm, thus the empirical risk can also be factorized by ória. This extends to I, layers where 0 = (Uz,... U, with g(0,r) .x) UL ... U1x and draa( = Un U1-
The resulting factorization of holds globally on Ω = D
Example 2.3 (ReLU neural networks). Consider g(0. (00.)) = (Vo())> x), with (8) = (max (34, 0)); the ReLU activation function. Denote 0 = (U, V) with U = (u1.... ur) E Raxr V
(1i,".. 2,) E R. (so that = (n-+m)r). Given training vectors x; E Th consider f C R the set of all parameters 0 = =U, V) such that v x; + 0 for every i. Then, 0 101(c,_ > 0) Ej is locally constant over 0 E Ω. and the model go (2:) can be factorized by the reparametrization (ReLL(() = (u,u) ) (here = rmn) using g(0, x)) = Σ3 $, with (0) == u,v, so can be factorized by ΦROLU with some forward function f: the reparametrization ( contains r has d' < D r) associated to "local" f valid in a neighborhood of 0. A similar factorization is possible for deeper ReLU networks (cf Neyshabur et al. (2015); Stock & Gribonval (2022); Gonon al. (2024) and we still write RRLLU, as further discussed in the proof of Theorem 3.1
matrices of size n m (each of rank most one, so in particular
Example 2.4 (Attention layer). For an attention layer, the input E R.N Xxdim is the vertical stacking of N tokens 2(() E Iddim The layer output is
g(8, X) = softmax (XQ KXT T O E
R Xxdim
with (softmax(A)),; = exp(A.;) ΣN exp(A,t)
with Q, K, V.OO E d xdim We use the reparameterization Oxce (6) == (ő), where Φi = Q K
and Φ == VTO, such that g00 X)= = softmax X o1 XT) ) as done by Marcotte et al. (2025).
3

---

Published as a conference paper at ICLR 2026
Thus, similarly to the linear case Example 2.2 l can be globally factorized by Ωeee exhibits no dependency on the specific parameter configuration 8y- This naturally extends to multiple attention layers by concatenating the corresponding factorizations.
2.1 DYNAMICS OF LIFTED PARAMETERS: TO BE OR NOT TO BE INTRINSIC?
This paper addresses a fundamental question underlying much of the efforts to characterize the implicit bias of gradient-based methods: under what conditions does the gradient flow dynamics equation in parameter space 0 lead to a dynamics on the lifted parameters z == (((()) that can be expressed as intrinsic gradient flow on z? This is crucial when attempting to establish that z(t) follows a mirror flow (Gunasekar et al., 2017), , which is key step to characterize the implicit bias of gradient- based optimization. We specifically examine when 2(t) follows flow with respect to Riemannian metric which, by definition depends only on 2 (and the initial parameter configuration 0o), thereby eliminating explicit dependence on the parameter trajectory O()).
A starting point of the analysis is that, under Assumption 2.1 and by the chain rule when 0(t) E Ω
(t) = @ε(t(t))((t) = -0 T Vf(((t)).
(3)
Thus our goal is to understand when the symmetric, positive semi-definite matrix
M(0)= == 0)()))0(0)
(4)
(corresponding the so-called path kernel (Gebhart et al. 2021) when is the path-lifting associated to ReLU networks) can be solely expressed in terms of and Bo during the trajectory, i.c. do we have a function K- Ko, such that (o(t)) (x)))? When this is the case equation becomes
att) = - KK (
(5)
an ordinary differential equation (ODE) which is Riemannian flow (Boumal 2023) for the metric K K=1(=) (or sub-Riemannian flow for the pseudo-inverse K (z) when ((z) is not invertible),
hence associated to an intrinsic dynamic on the lifted parameters =!).
As illustrated next, rewriting M (0(L)) as a function of =(() along the trajectory ((1) is indeed possible for simple linear networks, with function K--) that depends on the initialization 0g-
Example 2.5 (A simple linear network). Consider g(0, x) wur, with 0 = [u, v) R. Rm and (0) = wv E R. (ef Example 2.2 Then M (0) (+)(0)5(0)) = vr + Id During the X (as h(0) == ||a|12 is conserved AAora et al. 2019
trajectory 2² ||0|2 ug |
and as vv" T = u -u-222 2zz we have (u²)² Xu² || = 0 so that 2² = A+ /+ As a result
along the whole trajectory we have M (0) = Ke (2) so that zt) satisfies the оδE equation 5 with
2
A X2 2|2 Idm:
Ke, (z) X
22
vz.
22 +
4 ||
In partieular when
1 one has (2, V(0g of)2 + 4=2 hence é == -VX 422 f()).
See Section B for more comments on that example. In the above example the function Ka,- as its notation suggests, only depends on the initialization 0g but not on the function such that { = f Φ. In machine learning scenarios, ff typically captures dependence on the training dataset. The intrinsic metric Ka, (z) thus captures parts of the dynamics of 2(t) due to the network architecture (via Φ) and of the training algorithm (the gradient flow equation T irrespective of the dataset and the learning Ko (z)). This
task (of course the latter still play role via the [=()) term in the ODE motivates the introduction of the following definition.
Definition 2.6 (Intrinsic dynamic property). O, verifies the intrinsic dynamic property with respect to on some open set Ω 9 fp, if there is Ko, Rd -> Rddd such that, for every E C2, the maximal solution 0()) of equation D on Z with E = fo o 6 satisfies M (0(t)) = Ka (Q(()))) for each t.
2.2 CONSERVATION LAWS
Example 2.5 illustrates a phenomenon that we will systematically exploit in our analysis: with the typical reparameterizations & mentioned above, there exists a vector-valued function h : 0 +> h(0) E N that is conserved along the trajectory and allows to exhibit function Ke, such that M (0(t)) = Ko, (2(t)) along the trajectory. As these will play a key role in our analysis we now remind the essential concepts related to conservation laws.
4

---

Published as a conference paper at ICLR 2026
We denote @1, ...... , &d the d coordinate functions of the reparameterization o: R 6(0) Rd
C (IR ). Since Φ yields a factorization of the loss, functions h such that Vh(0) 1 (0) for each i and each 8 remain constant along the trajectory. This has been thoroughly analyzed, see e.g.
Marcotte et al. 2023 2024), using the following definition.
Definition 2.7 (Conservation law for d). A function E (2,, R) is a conservation law for & on Ω if for any 0 E S2 one has (h())))(() = O, i.e. for each E 92 and i, (V).(0), Vh(0)) = 0.
Proposition 2.8. Under Assumption 2.1 on E, & and Q if E C¹ (S,, R) satisfies (o()) on S2, then h remains constant during the trajectory (t) of equation [ for any initialization dg E Ω2
The conservation laws associated with a given parameterization 6 have been almost exhaustively studied for parameterizations corresponding to linear networks, ReLU networks, and attention layers. In particular, prior work has shown that all conservation laws for ø in the cases of ReLU (cf Example 2.3 and linear (cf Example (.2p networks (see Marcotte et al. (2023) as well as for an attention layer (see bee|Mattotte et al. (2025) are captured by the following proposition (Marcotte) al. 2023). In Le & Jegelka (2022), the authors also give canonical conservation laws for general DAG ReLU architectures. We recall these conservation laws in Lemma F.9 This has been proven theoretically for two-layer networks and empirically validated for deeper architectures using symbolic computation (see Marcotte al. 2023)) One of the contributions of this paper (see Corollary 3.4 and Corollary E.15) is to establish the completeness of conservation laws for é associated with any DAG ReLU network. It worth noticing that all conservation laws in such cases polynomials.
Proposition 2.9 (Conservation laws for classical on R),,. Consider O (UL.... U1) and (0) = Ur
... ......
U1 from Example 2.2 (resp. (REIM from Example 2.3). The functions
h; : 0 U141 - U,U. (resp. h, 0 + Diag 1 U141 - U,U,T)
are conservation laws dran (resp. &nowu). Similarly, considering = (Q, K, V, O) and OALt from Example 2.4 h: 0 (QQ KK VVT 00*) is set of conservation laws for dxsr.
2.3 INTRINSIC DYNAMICS VIA CONSERVATION LAWS Given conservation laws h0) for o. any trajectory ((t) satisfying equation ! remaain at all times on the set
Mag 10 h(0) h(0n)):
(6)
determined by f.. This holds true for any function f such that = f o 6 (hence, in machine learning: for any task/loss and any dataset, provided that the network model is (locally) factorized via Φ).
To establish the existence of a function Ko, (.) such that M(((t)) = Ko, (=(t)) on the whole trajectory, natural relaxation is thus establish a related equality the whole set Mo, rather than only on a specific trajectory. This leads to the following definition and its immediate consequence.
Definition 2.10 (Intrinsic metric property). We say that Og verifies the intrinsic metric property with respect on an open set U ? Oo if there exists conservation laws h()) IR-N for & and function Ko. E C² [Rdd (dxd) such that M (0) = Koo (o()) for each 0 E MM. nu
Lemma 2.11. If 0o verifies the intrinsic metric property 2.10 on U with respect to o, then it also verifies the intrinsic dynamic property on U with respect to c.
Remark 2.. is not difficult check on all examples considered in this paper that if bu satisfies the intrinsic metric property with respect to & on some open set U, then any 0/ E MM, also satisfies the property on properly modified open set U', with the same function K. This function thus only depends on h(0)), and we denote it Kn ((())) when needed to highlight this fact.
Remark 2.13. Lemma 2.11 remains valid with a slightly weakened version of Definition 2.10 where Ks, is not required to be smooth. Yet, since the existence of a smooth solution to the resulting intrinsic ODE equation 5 simplified when Ka, is C\,, we chose to include this in the definition.
The following theorem (proved in Section C) establishes a necessary condition for the intrinsic metric property to hold. We use it to show that the property does not always hold for linear networks.
Theorem 2.14. Consider h C¹ (IR", NN,, & E CR. R and du R such that the matrix 0h(0) E NDD has constant rank on Ma, n U with U e to an open subset of R and Ma,- = h ((h(O))) Then (i) (ii), where
5

---

Published as a conference paper at ICLR 2026
(i) There exists open set O C (Mas NU) and map Ki. E CO, R.xd) such that: (ii)
( Ka, (o(0), e MM, nu; kerdo(0) n ker c (O),
40 Mo. nu.
A trivial case where equation 7 holds when the intersection of kernels on the left hand side is zero:
kerŌo(0) n kerôh(O) {0]-
(8)
This stronger assumption can in fact be shown to imply the intrinsic metric property (see Theorem 2.17 in the upcoming section), and we will show (cf Corollary (3.3 that, with @RaLu associated to general DAG (directed acyclic graph) ReLU networks of any depth, there exists set of conservation laws such that equation [ inddeed holds for any initialization. This implies the intrinsic metric property and therefore the intrinsic dynamic property irrespective of the initialization for ReLU networks with
ΦROLU-
For linear networks with more than one hidden neuron, equation 8 does not hold (as consequence of Steps 2,3 in the proof of Theorem and we will show that equation 7 does not always hold. Nevertheless, certain initializations, known as balanced conditions (Arora et al. (219)), are known to satisfy the intrinsic dynamic property with respect to the reparametrisation ΦLin (cf' (Arora et al., 2018, Theorem 1). (Bah al. 2022 Lemma 2)). In this paper. we generalize this result to so-called relaxed balanced initializations (see Definition Moreover, we show that in certain configurations, relaxed balanced initializations are exactly the only ones that satisfy the intrinsic metric property (cf
Theorem 3.8 and Theorem 3.9
2.4 INTRINSIC RECOVERABILIT
In this section we consider a stronger condition called intrinsic recoverability property which requires not only that M (O(l)) can be rewritten as function of =(1) and the initialization, but also that at each point of the trajectory f(t) itself can be fully expressed in terms of ztf and the initialization fg. In other words, in this scenario, {(t) can be completely recovered from the parameterization 6 and the initialization alone. hence the name. As we will establish, this apparently strong property indeed holds when equation is satisfied, which is always the case for ReI.U networks.
2.4.1 INTRINSIC RECOVERABILITY IMPLIES INTRINSIC METRIC
Definition 2.15 (Intrinsic recoverability property). We say that 9n verifies the intrinsic recoverability property on an open set U 3 0, with respect to o, if there exists conservation laws h(0) E RRN for ø and function I(.) E C² (TR" R.W, IR DR such that I()(0), h())) for each 0 E U.
When this property holds, each E Mc. satisfies M(0) = MTT()(0) h())] = ([L()()), h(0o))) Kn(o))(((O) (with Kn(a)()) == M[ M[[(,, hence the following result.
Lemma 2.16. If fg satisfies the intrinsic recoverability property on an open set U 3 Og with respect to 6, then Gg satisfies the intrinsic metric property on U with respect to é.
The intrinsic recoverability property is equivalent to equation & (see Section D for a proof):
Theorem 2.17. Given ④ E CR D Red) and Og E RR,, the following are equivalent: (i) there are conservation laws h E C2 (St, N ) for o on a neighborhood $2 of fo such that equation & holds for each 0 E Ma, n Ω2 (ii) there is open set U C Q on which Oo satisfies the intrinsic recoverability property Definition 2.15 (and thus the intrinsic metric property Definition 2.10) with respect to &b
2.4.2 THE FROBENIUS PROPERTY IS ALMOST EQUIVALENT TO INTRINSIC RECOVERABILTTE
We are interested in equation g as it implies the intrinsic recoverability property, and thus an intrinsic dynamic. It may not seem obvious a priori how to verify whether such a condition can hold, nor how to construct suitable conservation laws h in practice. Intuitively, one should select as many conservation laws as possible while ensuring they remain independent, in a specific sense (Marcotte) et al. 2023, Definition 28) As shown by Marcotte et al. (2023), knowing the maximal number of such conservation laws can be checked using Lie brackets of the associated vector fields. We recall the relevant definitions and explain how this criterion applies in our setting.
Definition 2.18 (Lie brackets). Given two vector fields XI. X2 E C (e, RR),, the Lie bracket XxIX2) is the vector field defined by [x1: X2) (0) (x)())x - Oxi (0))

---

Published as a conference paper at ICLR 2026
Definition 2.19 (Generated Lie algebra). Given some function space W C C (O, (",R", the generated Lie algebra of W is the smallest subspace of (O) (+, that contains W and that is stable under Lie brackets, and is denoted Lie(W).
The trace at 0 E θ of any set W c C (0, of vector fields is defined as the linear space
(()) span [x(0) e W C
and for any infinitely smooth & we denote W = span V (-),,,, 1 C
Definition 2.20 (Frobenius property). A C function o satisfies the Frobenius property Ω if for all 0 E Q, Lie(W ()) = W (0). This property is slightly weaker than involutivity (Isidori 1995).
The following proposition (proved in Section E relates this property the intrinsic dynamic property of fg. In particular, as the Frobenius property does not hold for Φtin (Marcotte et al.) [023] Proposition I.1), is not possible to have the intrinsic recoverability for linear networks with classical @Lin-
Proposition 2.21. We have the following implications (1) = (ti) = (táā): (1) é satisfies the
Frobenius property on Ω and the trace of ' a has its dimension that is constant on 9 (ii) For any
80 E Q. there exists conservation laws h for & on a neighborhood U C S2 of tn such that for each 0 E Mo, equation 8 holds; (fii) d satisfies the Frobenius property on 22
Link with the commuting property. The result presented in Li et al. (202)) can be recovered as a special case of Proposition 2.2T] the authors require the reparametrization á to be commuting, meaning that for all pairs es Øi. the Lie bracket V Vd. Vo, is zero. In this setting, & naturally satisfies the Frobenius property, and the result of al. (2022) establishes an even stronger property: the dynamics 6(0) form mirror flow. In particular,
worth noting that diagonal networks satisfy that their parametrization (product of the diagonals) is commuting (as all coordinates functions separable). which thus (Li et al. (022) implies a mirror flow dynamic, and thus an implicit bias (see e.g. Azulay et al. (2021). ). In contrast, we consider a weaker condition; we seek only to determine whether the dynamics on <(()) can be expressed intrinsically as Riemannian gradient flow.
APPLICATION FOR GENERAL RELU NETWORKS AND LINEAR NETWORKS
We now show that the intrinsic recoverability property is satisfied for any initialization for the parametrization & associated to a large class of (deep) ReLU neural networks. While this result is already known in the two-layer case (Marcotte et al. [023 Examples 3.5 and 3,8), here we establish it for the general model of ReLU networks of Gonon et al. (2024), associated to a directed acyclic graph ww/iincarmmax-polll/
(DAG) of any depth, including skip connexions and arbitrary mixes of activations. We first establish that deeLu satisfies the Frobenius property (see Section F. for a proof):
Theorem 3.1. The paraneterization UReLU used for ReLU neural networks with any DAG architecture (see Gonon et al. 2204)) and our Example 2.3) satisfies the Frobenius property on (R)) (0))
The following theorem (proved in Section F2 for general DAG ReLU networks) refines this result by allowing some zero entries in the parameters.
Theorem 3.2. Consider an MLP with at least two neurons in each layer (including input and output layers), and denote θ C IB the set of all parameters with at most one zero component. On this set, which is open, dense, and with a single connected component, the parameterization @newu of et al. 2024) satisfies the Frobenius property.
This leads to the following corollary (proved under weaker assumptions in Section F3 which guarantees the existence of a maximal set of conservation laws big enough to ensure the intrinsic recoverability property.
Corollary 3.3. For a general DAG ReLU network (resp. for an MLP with at least two neurons per layer), every element 0 of the set (R. \ {0))² (resp. of the set θ of Theorem 3.2) admits an open neighborhood U in this set on which the intrinsic recoverability property (and thus the intrinsic dynamic property with respect dsel)) holds.
Moreover, the known conservation laws given in Proposition 2.9 are independent on θ from Theo- rem 3.2 (actually on an even larger set containing θ). Thus, they yield m independent conservation laws on O where corresponds to the number of hidden neurons. To verify that these are indeed
7

---

Published as a conference paper at ICLR 2026
the only ones, one must check that the trace of W supported by Marcotte et al. 2023), which confirms that Lic drwen) (0) = W dhean (0) has dimension
one has dimension D m; while it is empirically
D m when sampling random values of f. as well as random dimensions and depths, the following corollary establishes it theoretically (see Section F.4 for proof).
Corollary 3.4. Let us consider a MLP architecture with at least two neurons per layer, and with m hidden neurons. Then the m independent conservation laws for ΦReLc given in Proposition 2.9 are exhaustive on the set θ from Theorem 3.2 there is no more conservation laws than these ones.
As concrete example the following proposition (proved in Section G provides the first closed-form formula for the intrinsic dynamic for a three-layer ReLU network with sealar input and output.
Proposition 3.5. For a 3-layer ReLU MLP with scalar input/output, the factorization @ReLu read
Z = ΦReLU( u. V, w) == diag(u) V ddag(w) E R=
Wy o 0 Vi,j), and with u E IR",, V E Ruxmm and E IRm.. Define θ == {uu, V. V.w) u4 Wij
consider the n + m conservation laws h(0) := ((u} - -, Vo (w} Σ. )-))) for QROLU
Every 8g @ satisfies the intrinsic dynamics with respect to ORELL, which reads 2 K6, (2) f()) with 2 = vec(Z) ()) corresponding to
2 = -diiag( fZ))Z") diag(a) - Z (iiag((() VZ( ding( Z diag( ddiag (ddiga Z
(Z))..
where: for any matrix M, ddiag(M) == diag (Diag(M)) where Diag(M) extracts its diagonal as a vector and diag(u) is the diagonal matrix with entries of o; and b) the vectors a = az 7, h(0))) E
Ro and [ = (7, h00)) E Re0 (uniquely determined by Z and h(0o)) satisfy
a² 22-12 di) -1, -ooa = =0, -²- - diag(a)
O 10B== 0,
(10)
with 2 E Rx the element-wise square on the matrix Z E RRux and with X E R", E R such that h(0)) 14). When X, pa O, equation entirell characterizes (a,,3)
3.1 DEEP LINEAR NEURAL NETWORKS AND LINEAR NEURAL ODES
For L-layer linear networks, 6 = (U1..... UL) and the yields factorization via CReLU. leading to an intrinsic dynamic by the results of the previous section.
formalism (Gonon et al. 2024)
more common, however, to consider the dynamics of Z == &r.in (0r.) = Ur. U1, since dLin
is more efficient than ΦRaLU in terms of dimension reduction. We now analyze the dynamics of (t).
The gradient flow 02 ((((L) gives the evolution of Zr (see e.g. (Bah et al.
2022 Lemma 2)):
L Σ S, (T, with { Sy Un U UJ41" UE 3=1
S1. Id,
17 -
(11)
T, = UT. U,-1
U1,
Ti = Id.
The metric (8L) on z1 = vec(Zz) is thus entirely characterized by (S)())): T Ti+1 (BL)) Definition 3.6 (Relaxed balanced condition). We say that 02 = (UL, ... U2) satisfies the relaxed balanced condition if there exists x= (Ax)s RR- such that
U U.U. ATT
AI -SiSL-1
(12)
(O-)balanced conditions Baa et al.
2022 Def 1) (Arora et al. 2019 Def 1) correspond À 0.
Remark 3.7. It is worth noting that Dominé et al. (2025) used this exact same condition and called it the balanneedd condition. However, the defimition of X-balanced condition is already used (see (Arora et al. 2019 Def by the literature to refer to the weaker condition |U, U U.U,T S >, Other
works (see e.g. [Tarmoun et al. (202T); ) |Braun et al. (2022): Varre et al.] (203)) use stronger conditions
on the initializations, that in particular satisfy the relaxed balanced conditions of Definition 3.6
3.1.1 DEEP LINEAR NEURAL NETWORKS
We first detail the study of the two-layer case, and then generalize it to the deep case.
Matrix factorization. We consider the two-layer case where 0 == (U E gurr R= and with Z = Olin (0) == UVT E R nx We assume ((t) satisfies the gradient flow equation with 8(0) (U,=0, Vo= We denote S = UTO V.LoV== E R
When written x m matrix, we denote Z instead of and also view ff(Z) as matrix.
8

---

Published as a conference paper at ICLR 2026
If Bu satisfies the balanced condition equation 12 S 0, then (Arora et al. 2018 Theorem 1) (Bah
et al. 2022 Lemma 2) 8g satisfies the intrinsic metric property with respect to Orin and
= 3=-VZZTvf(2 -
(13)
We generalize this result (see Section HJ a proof) broader class of initializations: all initializa- tions satisfying the relaxed balanced condition equation 12 possess the intrinsic metric property.
Theorem 3.8. Consider 8g == (Ut=o, Vi=o) where both U1=o E B and Vi=o E TRxr have full rank S min(n, m.), and assume S = AId, for some X R. Then, on neighborhood it of o:=
1 2 = -Il ldn + 22 4ZZ
>
[-
X Idm + V22ldm + +42
Vf(Z) (1
(14)
where a is the orthogonal projector on rangeA.
Note that equation 13 corresponds indeed to equation 14 with À and that min(n, (n, m) is necessary to have 5= Ald, for some À * O. Note also that Theorem 3.8 generalizes to the case r S min (n, m) the expression obtained in (Dominé al. 2025 Theorem 5.2) for the special case r min(n.m) (if Dominé et al. (025) focus in general on the squared loss, the proof of their Theorem 5.2 does not rely on the use of this specific loss: this result can be applied for any loss, as ours). The following theorem shows that the relaxed balanced condition is actually necessary condition when r S max(rr m) to have the strong intrinsic dynamic property (see Definition 10 in Section I variant of Definition 226 where is piecewise C2. Its proof (see Section Ip relies on showing the non-inclusion of the kernels of equation]] 7
Assume that both Ut-o E RRax and V RRmxr have full rank and that S. max(n, m). If == Uz=QUi=0 Vi=oV== Vi= o Ald,, then 8g does not satisfy the strong intrinsic dynamic property with respect to ØLin-
Theorem 3.9. Let 0o == (U Vo-o).
The case r > max (n. m) is still open. For n m and any r, the following proposition (proved in Section D shows that all initializations do satisfy the intrinsic metric property with respect to ØLin-
Proposition 3.10. Let 0 == vu with u E IR' and E R. Then == (0) = (u, e) E R. We denote S = un=ou ot=ovIo E Brxr, Then one has
2tr( tr())² (z))
In particular, it is important to note that the two-layer linear analysis allows these results to be applied directly to networks composed of attention layers (Example 2.4).
Deep linear neural networks. Consider linear neural networks of arbitrary depth, with square weight matrices 02 (UL...,U1)) U, R=xn, Indeed, this is precisely the main case where non-trivial relaxed balanced condition (A; * 0) can occur at every layer i, since n, = and thus n, min (n:-1, 14+1)) for all i. The following theorem (proved in Section K) generalizes Theorem 3.8 to this setting. In the case of balanced conditions (A = 0), our theorem recovers the dynamics described in (Arora et al. 2018 Theorem 1), (Bah et al., 2022, Lemma 2).
Theorem 3.11. If (0) satisfies the relaxed balanced condition (Definition 3.6) with X (Xx); then during the trajectory (1) of equation [ the mutrices in equation 77 satisfy (I)
(()((())U)())) and (T))((t) (t)( Ut)),, where 2,(x) := M 4- (I ax) with ao = O and ag == Ek= for k 1, ... L 1 and (z) := 1 by) with bo = 0 and by = - di. Moreover U.U,J (resp. UJ Ui) is the unique rool of Z, QU(U,
(resp. of 21
(U, U1)) with spectrum lower bounded by maxosk<L-1 ak (resp. by maxosk5L-2 bz). This implies that all matrices in equation are are entirely characterized by Zn and the initialization, hence (0) satisfies the intrinsic dynamic property on RDD with respect to dran-
3.1.2 INFINITELY DEEP LINEAR NETWORKS
We next consider the limit when I. - +o0 of deep linear residual networks with parameters Uk = Id,, + Ag. and thus focus on the analysis of the parameter 0 = (As) 1). where A, R which corresponds to linear neural ODEs model (introduced by Chen et al. (2018)). Remarkably, our theoretical approach still applics in this regime, and yields a closed-form formula for the metric. We thus study the dynamics of parameters O(1) E X' where X' corresponds to the Banach

---

Published as a conference paper at ICLR 2026
space (C' (1,, ",,"), || Ile)) where llfller == max |f ll }. and such that the trajectory
t + 0(1) = (A, (4)) (4))) is the solution of the gradient flow on f(O), given by the (family of
coupled) ODEs
Vs [0, 1],
84 (t)== with (t(
3A, ((()) E R
(15)
where we assume that the loss function é + Lipschitz X' (to ensure by the Cauchy-Lipschiee theorem that indeed there exists unique maximal solution 0(-) E (10, 7), .) of equation 15 with given ((0)).
R is such that +
( (2
aA. s0)) is locally
As an infinite-depth analog of Zr = UL ... U1 given any 0 E X we consider [0. 1] Z, Z. E ==*x the unique global solution (as = (As)sElo.11 E 2) of the state equation
1 = A, Zo= Idn-
(16)
The analog to Assumption 2.1 is to assume that ((0) =(Zz=1 with f E C², and now want to know if it is possible to rewrite the dynamies =1 (t) as an intrinsic dynamic that only depends on
de
(t) and the initialization ((0). The following proposition (see Section for a proof) gives set
of conserved functions during all trajectories of equation 15
Proposition 3.12. For any
[0, 1]. consider h, 0 = (A.)ac(0.1) EX + A' A
[A] E Raxa, where we denote A', = As. Then for any 8 (0. 1], one has for any l: h, (0(t)) h, (e(0)), where O(t) is the maximal solution of equation 15 with initialization ((0).
Moreover, the following theorem (see Section M for a proof) shows that for relaxed balanced initializations, the evolution of Z (t) 7,5= (1) is intrinsic it depends only on Zi and the initialization 0(0):
Theorem 3.13. If the initialization (0) satisfies that for each & E [0, h, (0(0) X()) for
some X(-) E C ([0, 1], RR), then one has
L.' (ZzZT)²- (Z [xpp)()))2
with (3) = (1-8) (1) (1)- (1-s)) s (1)+ where 1/2 : 8 [0. fo So A(1-v)dudu
and 12 : E [0. + J0 So ((u)dudu. If X-) = (balanceed-onddition)) then 7( 0.
In sense, this theorem captures the infinite-depth limit (L. - +00) of Theorem 3.1 while offering the key advantage of an explicit closed-fore expression for the associated metric.
CONCLUSION
In this paper, we investigated when high-dimensional gradient flows can be recast as intrinsic Riemannian flows in diver-dimeniional spaces. Our results show that such reductions are always possible for ReLU networks under path-lifting parametrization, and for linear networks under relaxed balanced initializations. A central contribution is our analysis of the "path-lifting metric", recently introduced and still largely unexplored object, for which we provide an intrinsic characterization in the l-yer case. Extending this analysis to deeper or more general architectures could shed new light on the geometry of gradient dynamics for general ReLU networks.
ACKNOWLEDGMENTS
The work of G. Peyré was supported by the French government under the management of Agence Nationale de la Recherche part of the "Investissements d' d'avenir" program, reference ANR-19- P3IA-0001 (PRAIRIE 31A Institute) and by the European Research Council (ERC project WOLF). The work of R. Gribonval was supported by the AllegroAssai ANR-19-CHIA-OONE project of the French Agence Nationale de la Recherche and by the SHARP ANR project ANR-23-PEIA-ONN the context of the France 2030 program. The authors thank Stephan Thomassé, Gilles Blanchard and Laurent Jacques for discussions on rank properties of DAGs that helped simplify the proof of Proposition F7 in Section
in
F2
10

---

Published as a conference paper at ICLR 2026
REFERENCES
El Mchdi Achour. Kathlén Kohn, and Holger Rauhut. The riemannian geometry associated to gradient flows of linear convolutional networks, 2025, URL https::/arxiv.ont abs/
2507.08 06367
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit accelcration by overparamcterization. In International Conference on Machine Learning, pp. 244 253. PMLR, 2018.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2019.
Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 668-477 PMLR, 18-24 Jul 2021.
Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenbery. Learning deep linear neural networks: Riemannian gradient flows and convergence global minimizers. Information and Inference: A Journal of the IMA, 11(1): (1(1):307-35,, 2022.
R. H. Bartels and G. W. Stewart. Algorithm 432 [c2]: Solution of the matrix equation ax xb [f4]. Commun. ACM, (5(9)) 826, September 1972. ISSN 0001-0782. doi: 10.1145/3611533.61152.. URL httphhttss:/// 1t/p./......01111111111111/1111. 361582.
Berthelin. Equations différentielles. Enseignement des mathématiques. Cassini, 2017. ISBN
9782842252298. http::tttt///ooo wtw.a//ee....oooooooooooooaaaaaa
Nicolas Boumal. An introduction to optimization smooth manifolds. Cambridge University Press, 2023.
Lukas Braun, Clémentine Dominé, James Fitzgerald, and Andrew Saxe. Exact learn- ing dynamics of deep linear networks with prior knowledge. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35. pp. 6615-6629 Curran Associates, Inc.. Files/paper/2022
wwurips. fil file222bb22111513300775311222222
2022. URL
Emmanuel Candès, Thomas Strohmer, and Vladislav Voroninski. PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex Programming. Communications on Pure and Applied Mathematics, 66(8):1241-12744 2013. doi: 10.1002//pu221432. URL Publisher: Wiley Subscription Services,
hitp:://dk doi. wrg/10. 1002 21422
Inc., A Wiley Company.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and
R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018 URL Utt/httpps//pprroeiinn...eerrrre
ec ccppaper carpaper_rleesseeaaaeaaeesees filesspaper) Paw.:886669339-9-P.pppeeee Paper.
Yacine Chitour, Zhenyu Liao, and Romain Couillet. A geometric approach of gradient descent algorithms linear neural networks. arXiv preprint arXiv: 1.055668 2018.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conf. on Learning Theory, pp. 1305-1338. PMIR, 2020.
Clémentine Carla Juliette Dominé, Nicolas Anguita, Alexandra Maria Proca, Lukas Braun, Daniel Kumin, Pedro A. M. Mediano, and Andrew M Saxe. From lazy to rich: Exact learning dynamics in deep linear networks. In The Thirteenth International Conference on Learning Representations, 2025. URL / ew wwt. wwt.e//eeerrrrmmmmomooooooooooo

---

Published as a conference paper at ICLR 2026
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in Neural Information Processing Systems, 31, 2018.
Thomas Gebhart, Udit Saxena, and Paul Schrater. A Unified Paths Perspective for Pruning at Ini- tialization, January 2021.
10552
10552 arXiv:2101.10552
[es].
Petra Gleiss, Josef Leydold, and Peter Stadler. Circuit bases of strongly connected digraphs. Discus- siones Mathematicae Graph Theory, 23(2):241-260, 2003.
Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, and Rémi Gribonval. A path-norm toolkit for modern networks: consequences, promises and challenges. In The Twelfth International Conference on Learning Representations, 2024. htwpt::ttppppeeeeeeeeee forr forum? Id=h112vUIYYk
Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization, Advances in Neural Information Processing Systems, 30, 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference Machine Learning, pp. 1832-1841. PMLR, 2018.
A Isidori. Nonlinear system control. New York: Springer Verlag, 61:225-236, 1995.
Velimir Jurdjevic. Geometric Control Theory. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 1997. ISBN 9780521495028.
Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. In International Conference on Learning Representations, 2022. URL https:// openreview n http://orrumnnxxeeeeeceooo
Zhiyuan Li, Tianhao Wang, Jason D Lee, and Sanjeev Arora. Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 34626-34640. Curran Associates, Inc., 2022.
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: Conservation laws for gradient flows. Advances in neural information processing systems, 36, 2023.
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Keep the momentum: Conservation laws beyond euclidean gradient flows. In 41st International Conference on Machine Learning, 2024.
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Transformative or conservative? conservation laws for resnets and transformers. In 42nd International Conference on Machine Learning (ICML 2025), 2025.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-Based Capacity Control in Neural Net- works. In Proceedings The 2015. URL httpp:://pooceedi.nett mlr. wwtpppsssssswwseseppeee 1928-
The 28th Conference on Learning Theory, pp. 1376-1401. PMLR, June
html ISSN:
L.S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and F. Mishchenko. The mathematical theory of optimal processes. Wiley, NY, 1962.
Daniel Soudry. Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822 2878, 2018
Pierre Stock and Rémi Gribonval. An Embedding of ReLU Networks and Analysis of their Identifiability. Constructive Approximation, 2022. doi: 10. 1111007/00336-022--05555 Publisher: Springer Verlag.
12

---

Published as a conference paper at ICLR 2026
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics gradient flow in overparameterized linear models. In Internationaa Conference on Machine Learning, pp. 10153-10161. PMLR, 2021.
Aditya Vardhan Varre, Maria-Luiza Vladarean, Loucas Viviie, and Nicolas Flammarion. On the spectral bias of two-layer linear networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htccp:://peenreiieeenntt//eeeeee ptt.pruuuniiiddddd.......
13

---

Published as a conference paper at ICLR 2026
A A TABLE THAT SUMMARIZES WHICH PARAMETRIZATIONS CAN BE USED TO ANALYZE WHICH TYPE OF NEURAL NETWORK, ALONG WITH THE CORRESPONDING RESULTS.
Paramctrization
@Lin
ΦRELU
Network type
Linear network
IMP: only for relaxed balanced dg
IMP: for all dg
Dimension: d' < D
d' D - k R D
DAG ReLU
N/A
IMP: for all 6g
d' D 1 D IMP = intrinsic metric property ; dimension of the manifold of o(0); k= hidden neurons
Table The table summarizes which parametrizations can be used analyze which rype of neural network, along with the corresponding results.
B MORE COMMENTS ON EXAMPLE 2.5
In example equation 2.5 when À +00 Kop/ / > Id (the Euclidean metric). When À 0, one has Ks. (z) = || |=|||
25 for every a 0, and in particular, by the uniqueness result in the Cauchy-
Lipschitz theorem, 0 is reachable only if 20 0. When X - -o, Ko,/ |X] + //githan tiggthub.. com/ htt.scccccsciiaae www.byllema/nnntinnnen wtriiiitttrrinnnnnaeaea cesiiinn__dyamiicomm c for numerical illustrations of these different behaviors.
=z" [|]| See https:
C PROOF OF THBOREM 2.14
Theorem 2.14. Consider h E C³ (IB ,,,,,,,,,,, & E (RR and Oo RRD such that the matrix @h( E NxD has constant rank on Mo, nU with U e Oy an open subset of RDD and Ma,- = h ([h(Ou))) Then (i) = (ii), where
(i) There exists open set O 3 d n U) and map Ko E CO, Rdxd) such that:
M (o()), kerōo(0) n kerôh(0) C kerêM(O), 4O E Ma. nu.
v0 E Me. nU;
(ii)
Proof. - (ii).
Assume (i) and fix 0 E Ma, n and a vector v E ker ao(() n ker Ōh()). Applying the chain rule in the ambient space DR (possible as ker @h(0) To,o, because 0h (f) has its rank that remains constant on Me, N by hypothesis) gives
@M) dKs (o(0) (0o(0) v) 0K2, (0) O,
hence E ker (0) and (ii) holds.
D PROOF OF THEOREM 2.17
Theorem 2.17. Given o E CR. RR)) and fu E R., the following are equivalent: (i) there are conservation laws h E C¹ (Ω, RN ) for each 9 E Ma, n Ω2 (ii) there is an open set S Q on which d, satisfies the intrinsic recoverability 2.7ettioon...11...
neighborhood Ω2 of fu such that equation & holds for
property Definition 2.15 (and thus the intrinsic metric property Definition 2.10 with respect to dh
14

---

Published as a conference paper at ICLR 2026
Proof. We first show (t) =
(it). We assume (1). Observe that given any 0 R equation 8
equivalent (by rank theorem) to
( ah(d)
rank
D.
(17)
By smoothness of Φ and h, if equation 17 holds at 0g it also holds in whole neighborhood U of 0g. By the implicit function theorem, denoting F(0) = (0(0), h())), implies that 0 can be expressed on U as 0 F- (0(0). h(0)) (()(0), h(O)..
We now show (ii) - (i). We assume (ii). Then on U one has [()(0), h())). Thus on Me, nu, one has f [()(0), h(0o)). We now fix some Me, n U' and we consider vector V E ker 0e(0) n ker @h)
Applying the chain rule in the ambient space IR on I' gives
t == Idpu = Op(or h(0))) (0o(0) v)) (0) -0 =0
and thus U = O.
E PROOF OF PROPOSITION 2.21
Proposition 2.21. We have the following implications (i)
(ti) - (fii): (1) 6 satisfies the
Frobenius property on Q and the trace of V has its dimension that is constant on Q (it) For any
to S2 there existy conservation laws for d neighborhood U C S of On such that for each A E Me, equation 8 holds; (iii) 6 satisfies the Frobenius property on Ω
Proof. (i)
=
= (1i) is direct consequence of the proof of (Marcotte et al.
2023 Proposition 3.7).
(dit). Let us assume (ii). We fix 0g. Then by assumption on U 3 6o- [(h(0), ((0). and by using Proposition 2.21 one has ker (6(0) nker dh()) (0) on Ma, nu. Thus equation 17 holds an open neighborhood O of to- As h are conservation laws for e on U, one has on n that dim (W (0)) = D - rO).
We now show (ti) -
But as for any conservation law h for Φ, one has (Vh(O), Vo,(0)) = (Vh(O), (0) = (Vh(O), V Vo)](0)) O, one has necessarly dimLie (06) S D rankōh (b) and as one also has D rank dimW (8o) I dimLieW, (0u) then has W (On) Lie (Bw). This holds for any Og. which concludes the proof.
O
F PROOF OF THEOREM 3.1 THEOREM 3.2 COROLLARY 3.3 AND
COROLLARY 3.4
F.1 PROOF OF THEOREM 3.1
Theorem 3.1. The parameterization tRat used for ReLU neural networks with any DAG architecture (see Gonon et al. 2024) and our Example 2.3 satisfies the Frobenius property on (R)) {O))))
Proof. We consider parametrization o + (o)( where all Φ; are monomials in (0,, ,p) E (R)(0)) i.e. (2) = -1 0z Moreover, if a variable O, appears in some coordinate with exponent (5) O. then every other coordinate that contains 0g uses the same (k) (8)
exponent a a; These assumptions indeed satisfied for the path-lifting parametrization &Rew associated to general ReLU networks (Gonon et al. 2024, Stock Gribonval] 2022), associated to directed acyclic graph (DAG) of any depth, inc mellding skip connexions and arbitrary mixes of activations (and even slight generalizations of max- max-poolingg.
Now let consider two indices i,j [l. Denote I (resp. J) the subset of all indices l such that o (i) + 0 (resp. a.") * 0). By abuse of notation we write i njj (resp. i'ú etc.) the set I J (resp. J) and denote Bnny etc. the restriction of 0 to the corresponding entries. In particular, one can decompose
(Bing: Ost Ogr (ung)))
15

---

Published as a conference paper at ICLR 2026
We write
(.(0) - diny (0rng) (
o,( - Oinu (fins) @5)
where (-ny(-) is the maximal monomial factoring both (x and (5(+), and on (resp. xzre) ,) is the unique monomial such that
6x(-) =
x.(+) ) (-) 0)1(())
Then one has:
Vongons Φini Veni
V
Vo,(0)-
and Voy(0)= Dini Vogi
and
8 Poijdany Von Voins 0
VoiniVol O² Poidoss
8 (0) =
0
0 0
0
0
Thus
Poinjo Vdl Voins 0
Voins Vol 02 Pony
0
0
V Voinglgrs
0
8 (o)
0
Pins Vogu
0
8 ding V Voino (oooss Vony 0
@2 V Temidey Voav Velnsll 0
and similarly one has:
0 @ Voing
(, = one
VΦla Wannil ?
Finally one has:
Vo. Vo,1(0) = -Von (einilVaaae V || Valya Vidino doo
0 -vdnydo Vogions
But as:
0
Bj Vans - on Vogne
Vor on, Vo, ôing
As Φing # 0 (indeed 0 E (TR) (RR ( then has
Dj Voini on Vogni
E W (0) and thus
Vo vo, E (O).
16

---

Published as a conference paper at ICLR 2026
F.2 PROOF OF THEOREM 3.2
we introduce the notion of prumable DAGs and characterize them, before showing (see Corollary F.11) that the rank of (+(+) is almost everywhere equal to that of @o(1D)
To prove Theorem 3.2
F.2.1 PRUNABLE DAGS
Definition F.1. Given a DAG G -V, with vertices V and edges E and, for each edge i E E we denote Gj -V, EJ) with E E (i) the smaller DAG obtained by removing edge i. The input neurons of G' (neurons without any antecedent) are always also input neurons of G and similarly for the output neurons of G (neurons without any successor), and we say that G is prunable at edge if G has no additional input/output neurons. The graph G' is said to be prumable everywhere (resp. somewhere/noweer/ it is prunable at every (resp. at some/at no) edge i.
Recalling that, by definition, path in graph is a sequence of consecutive edges starting from an input neuron and ending an output neuron, we denote P, == {p: i} the of all paths of G containing edge i E E, and Pf the other paths of G.
Lemma F.2. The set Pí is always subset of the paths of G/ Moreover, G is prumable at edge i, then P. exactly the set of paths of G
Proof. If p P. then its edges are distinct from hence p is a path of G Conversely, when G is prunable at i, G2 has the same input and output neurons as G, hence if p is path of G2 it is a path of G with edges in E,, i.e. with no edge equal to i, i.e. p P..
The following straightforward characterizations will be useful.
Lemma F.3. A graph G is prunable edge if. and only if. the output neuron of edge has at least another incoming edge, and the input neuron of edge has at least another outgoing edge.
Corollary F.4. graph G is prunable everywhere if, and only if, every non-output neuron has at least two successors, and every non-inpul neurons has least two antecedents.
Corollary F.5. graph G associated multilayer perceptron (MLP) of depth L with layer widths No, 0 < < L, is prunable everywhere if, and only if. N₂ 2 for every layer E (including inpul and output layers).
F.2.2 RANK PROPERTIES
Lemma F6.. The parameterization 6 = dReLU used for any general DAG ReLU neural network satisfies that for any E (R) (R) has rank(d@(f)) rank(do((p)), where 1 is the vector with all its coordinates equal to 1.
Proof. By definition of the component op of Φb (dbp) for each path p as a product Gonon al. (024)) one has:
Oøp (0)6, (p()
00,
In particular for = 1p one gets:
S = db(1p)) (L(sept)o.
and thus:
D De(0yS (o(
(18)
where Do(0) diag(o(0)) and D, diag(O). Note that equation 18 holds for each 0 P. Finally, as all coordinates of 0 are in IR RR)(0), both Doto) and Dg are invertible and thus rank (S)
rank()o(0)). which concludes the proof.
Proposition F7. For any DAG ReLU architecture, one has rank()00n::c(1))) 2 D m with m the number of hidden neurons and D) the number of parameters.
17

---

Published as a conference paper at ICLR 2026
Proof. Without loss of generality we prove the result connected DAG, Indeed, if the DAG has C' connected components, Gg. 1 & S C, each with D2 edges and my hidden neurons, then Σx D3 D and Σ. My m the result for the whole graph follows from the result on each component: up to row/column permutations, it is easy to check that the matrix @p(1p) is the block-diagonal concatenation of the corresponding matrices for each component, hence its rank is
(D, - mm)) = D m..
Assume that G is connected DAG, and denote S' = ( DelDD) :F. the incidence matrix of G restricted to F, the set of all full paths (i.e., paths starting from an input neuron and ending at an output neuron). As a direct consequence of the definition of S', one gets: range(de(11)) 2 rango(S"). Thus, to conclude the proof it is enough to show that range(S") = D m.
We now show that range(S") Proposition 6)) that in a strongly coannectedddirecteer graph G' = (E', .V', the rank of the cycle matrix i [V*| 1, where the cycle matrix has as many rows as the number of cycles in G' and each row has binary entries indicating which edge (indexing the columns of C') belongs to the corresponding cycle. To exploit this fact, we complete G (as DAG, has no cycle!) into a strongly connected graph G" with cycle matrix C' satisfying rank(S") = rank and |E"| |E| + Vout
D m., We will use the known fact (see (Gless et al., 2003
IVinl + 1, [V' = [ + 2, so that rank(S')= = rank(()) = E - [V] + |Voue + Vial = D m as claimed, since |V| (/Vott) + Vmml). Above, we denoted Vin (resp. Vout) the set of input (resp. output) neurons of G.
The construction is as follows: we add two fresh neurons μ. (V' == VU {u, v/); the first one is a "source" " neuron with an outgoing edge towards each input neuron in Vin the second is a "sink" neuron with an incoming edge from each output neuron in Vout: another edge is added y - p.. It is not difficult to check that cycles of G' are in bijection with paths in G: each cycle e of G' passes through exactly one input neuron and one output neuron, and its restriction to the edges of G is precisely a path p between these neurons. Vice-versa, every path in G. starting at y E Vin and ending at 7 Vous is uniquely prolongated into a cycle of G" by adding the three edges 7 + v. V- ~> 14, and μ + With slight abuse of notation we will thus identify e and p.
Equipped with this construction we first show that S' Ci:, E], so that rank(S') I rank(C). Indeed, S" has binary entries S" S"(p,e) el = ppae. so that 5'[p, e| = C(c, e] for every path and e E E.
To conclude, we only need to show rank(C) S rank(S'). For this we prove that each column Cia, e' E E'' \ E, is in the span of S'. Denoting out(7) (resp. in(7) the set of outgoing (resp. incoming) edges of neuron 7 E V, and suc(r) (resp. ant(?) the set of successors (resp. antecedent) neurons of y, i.c neurons connected via an outgoing (resp. incoming) edge, we distinguish three cases:
- y with y E Vin some input neuron; a cycle e contains this edge e' if, and only if, the corresponding path p of G starts at the input neuron 7, equivalently if there exists some neuron 7' E suc(7) such that p contains the edge 7 > >'' As can only pass through at most one of the outgoing edges of y, obtain Clc, e'] = Eecout(s) S' e/. and since this holds for every e we get Cl:,et Eecout(>) 8'(s.e) e). showing indeed that Cl: e'] belongs to the span of S";
-'= v, with y E Vous; similarly, Cl: = - S'|:,el:
p'=-- μr this edge appears in all cycles hence Cl:, e/] = 1. Since we can partition the cycles according to which edge e" = μ 7. 7 Vın they contain, we have 1 = Eyev, Cl:s/1 -> 7]. and since the first case have established that such Cl:.1 + 7 belong to the span of S", so does Cl:,e']=1 11
Corollary F8. For any DAG ReLU architecture, one has rank (00ReI((111) = D - m with m the number of hidden neurons and D the number of parameters.
Before showing this result, we first establish that the m "canonical" conservation laws (Le & Jegelka 2022 associated to each hidden neuron 1 < < m are independent on set that contains D.
a directed graph where for every nodes pav there exists a directed path starting y and ending v
18

---

Published as a conference paper at ICLR 2026
Lemma F.9. For any DAG ReLU architecture, the "canonical" conservation laws obtained in Le
& Jegelka 2022) associated to each hidden neuron 1 j I m, expressed (denoting 8 the bias of
neuron J. succlg)/ /ant(j) its outgoing/nncomnn edges, and 8 the corresponding weights) as
h₂(0) -
0
Σ (Eant(f)
0g -
(19)
iEsucc(()
are independent in the neighhorhood of each parameter such that Vhy(0) # O for all j (so in particular around any 0 E Q with θ the set of Theorem 3.2).
Proof. Let us consider a parameter such that Vhz(O) * O for all hidden neurons j.
For each edge i μ + has
20, if
ah 00,
(0)
20, j=
0
otherwise
and similarly for each hidden neuron j' with a bias
Thy (0) { 20 -220
j=
10 otherwise
002-
denoting ((0) (hy( RT the Jacobian matrix 0h(0)= R has one two nonzero entries per row: each row corresponds to one edge or one bias, each column to a hidden neuron, and row has two nonzero entries at the input and output neurons of the corresponding edge (resp. one nonzero entry at the neuron corresponding to the bias).
By standard linear algebra one can show that ah(O) has trivial kernel, hence its column are linearly independent. By continuity this remains the case for the m columns of 0h(6') in neighborhood of This exactly the definition of independent conservation laws.
We now are able to prove Corollary
F.8
Proof. By Lemma F. there exists (at least) independent conservation laws around 1p- Let us denote them hú, 1 <is S m. By definition of a conservation law (Definition 2.7 h of @Ret we have that (0₂ E ker(@ (0)), < sis m, and by definition of the independence of conservation laws, the vectors (h), S < m, are linearly independent. It follows that
dim(ker (0)kele0))) 2 m.
By the rank-nullity theorem this implies
rank a BeLU (ID) < D m,
and thus, by Proposition
gets
rank rank(O@
1n)) D m.
Corollary F.10. Consider DAG ReLU architecture with D parameters and hidden neurons.
Assume that the underlying DAG G is prumable at edge i. If satisfies 03 # 0 for every j + i, then Bank(O@rezule raan(a00 (doRezo =
Proof of Corollary
F.10
By Lemma F66
the result holds if 0, + 0, hence we focus the case where
= Denote & dhetu. and P. {p= p 3 i} the set of all paths containing the edge
1st step: show that rank (a(()) 2 rank rank (1)(1n)) Corollary F38
m. As a preliminary, observe that d (0) = O if, and only if, E P., and that, denoting 5 = Oo(1 1 D). we have
E P.,j Hi,
dép (0)
(0)
p E P.,j
00;
(0
p $ P,,j +
0
p $ P..j =i.
19

---

Published as a conference paper at ICLR 2026
Thus one write by blocks:
0Ø(0) (o ".)
(20)
where u = (0p) (i) PEP, has coordinate equal to zero, 8' = (0)) D0) = diag( (8)))
Dø == diag((") is invertible, and S" == S {.].. Thus, we have
ran())))))) = rank (orr = =1 rank
Let us now consider the smaller DAG G. obtained by removing edge i from the graph G. Since G is prunable at i, by Lemma F.2 the set P exactly the set of all paths of G so that S' (1 with o' the path-lifting associated to Gy, which is a DAG with D' = D parameters and m' m hidden neurons. By Corollary F88 we obtain:
Corollary rank()ø(11))
rank(O((0)) 1 + rank(S') 2 1+ 1) m D
2d step: We now show that ran(() S (ank()@(1n)) Indeed
rank(0o(1D)) =: rank(S)= 2 rank(Dste)
Equation 18
rank()))()Doo) 2 rank((0(()) -
where in the last incquality used again Equation 20 recalling that is the only coordinate of O.
3rd step: Conclusion.
Using the two first steps, one has rank(do(())
(rank(0o(1p)), ro01)))))))))))))) + 1} = {D m, D + 1 m). Assume by contradiction rank(0o(0)) = D + m, i.e., there exists a subset of D + - m linearly independent columns of 0o(0). Then, by the continuity of there exists open neighborhood U of 0 such that on U, these columns remain linearly independent, hence rank(O@ner (O0nnt (0)) 2 D D+1 m for every 0' E U. In particular for each 0' in the non-empty set U (0)) D.
have rank(O((0')) D + m,
which contradicts Lemma F.6 This concludes the proof.
Corollary F11. Consider DAG ReLU architecture with D parameters that is prumable everywhere. For any with at most one zero component, one has rank (drkk((( = rank(Bd)sosall (1D)) D m, with m the number of hidden neurons.
Proof. By Corollary F.8 we have rank (1p)) = D- m with m the number of hidden neurons and the number of parameters. Since 0 has at most one zero component and since the underlying graph is prunable at edge i (as it is everywhere by assumption), we conclude using Corollary F.10
In particular, for an MLP architecture, this corollary reads:
Corollary F.12. Consider MLP architecture with D parameters where every layer (including imput and output ones) contains at least two neurons. For any 0 with at most one zero component, one has rank(0ó (oo 0)) rank(o@ =
D - m, with m the number of hidden neurons.
Proof. By Corollary F.8 we have rank(ddretin (1n))) ) D m with m the number of hidden neurons and the number of parameters. Since has at most one zero component, there an edge in the graph G underlying the MLP architecture such that 8, * 0 for each + i. Moreover, since focus on an MLP, by Corollary [55 and our assumption on the layer widths, the underlying graph is prunable everywhere, hence it is prunable at edge i. We conclude using Corollary E.10
F.2.3 PROOF OF THEOREM 3.2
We can now establish Theorem
3.2
fact, we prove a more general version of Theorem
3.2
Theorem F.13. Consider a DAG ReLU architecture that is prunable everywhere, and denote θ C R the set of all parameters with at most zero component. On this set, which is open, dense, and with a single connected component, the parameterization ózatu of Gonon et al. (2024) satisfies the Frobenius property.
20

---

Published as a conference paper at ICLR 2026
Proof. It is each to check that è is dense open set with single connected component. Moreover for any 8 E 9,, denoting & = @eeLv. we have
= rank(O((0)) Comollar rank (ank(O@(1p)) =:
dimLie(W (0) 2
To conclude we need to prove that this is indeed an equality. Assume by contradiction that there exists a parameter E such that dimLie(W ()1) > d. Then, by standard continuity arguments, this will remain the case on an open neighborhood U of 0j. which is absurd by Theorem 3.1 as the Frobenius propery holds on (IR (0)) dense open subset of t
F.3 PROOF OF COROLLARY
We prove a more general version of Corollary
3.3
Corollary F.14. For a general DAG ReLU network (resp. for DAG ReLU network prunable everywhere), every element the set (R {0}) (resp. of the set θ of Theorem Theoree F.13) admits open neighborhood U in this set on which the intrinsic recoverability property (and thus the intrinsic dynamic property with respect to (newv) holds.
Proof. Let us consider a DAG ReLU network (resp. a DAG ReLU architecture prunable everywhere). We can apply Lemma F.6 (resp. Corollary [.I1] to obtain open dense é (resp. with a single connected component) on which the dimension of the trace of W remains constant. By Theorem 3.1 (resp. Theorem F.T3), ΦBaLU satisfies the Frobenius property on é. By Proposition 2.21 every 0g E + admits neighborhood U which it satisfies the intrinsic recoverability property with respect ΦRLLU- By Lemma 2.16 such parameter Og also satisfies the intrinsic metric property on U with respect ΦRELD-
F.4 PROOF OF COROLLARY
We prove a more general version of Corollary 3.4
Corollary F.15. Let consider DAG ReLU architecture that is prumable everywhere, and with hidden neurons. Then the m independent conservation laws for dRew given in Proposition 2.9 are exhaustive the θ from Theorem F.13 there is more conservation laws than these ones.
constant and equal to range (ddeev = D m by Theorem 3.2 and Corollary F.11 Then by (Marcotte et al. 2023 Theorem 3.3), there exists exactly independent conservation laws locally on Thus the m independent conservation laws for ΦRELL on θ given in Lemma F.9 are exhaustive.
Proof. On O, we know that the dimension of Lie Aouca (0)
G PROOF OF PROPOSITION 3.5
Lemma G.1. Let Y Raxm Then there exists unique pair (a,, =::0) of vectors E R Ri such that
a Ydiag(B) -1 and B Y (idaaaa))
Proof. Define the mappings
S(B) = / diag T(a) y diag(a)-
Let Da(a,a') = log(a/a')ll>> denote the Thompson metric (R>o)" (and similarly Dm on (Rso)") where R² is the set of positive real numbers. It is known that ((R>o)", Dn) (resp. ((Ro))" Dm)) complete metric space. The linear operator Y is I-Lipschitz from ((Rso)" Dm) to (R>u)"", (D))), according to the Birkhoff contraction theorem. Moreover, the square root function -Lipsccitz in this metric. Hence, the composition S o T is 1 :contracting with respect to D By the Banach fixed-poim theorem, there exists a unique fixed point of o T, which implies the existence and uniqueness of the pair (ax, (3) solving the original equations.
21

---

Published as a conference paper at ICLR 2026
Proposition 3.5. For 3-layer ReLU MLP with scalar input/outpus, the factorization @ReIL reads
ΦReLU(u, V, w) == diag(u) V diag(w) R=
with u E RR, V E RTxm and w E IRT Define == {(u, V, w) : u. ViJ wy * ji and consider the n + m conservation laws h(0) := (u - Σj vi) (w} Σ: (=))=)) for ReLL
Every O E Q satisfies the intrinsic dynamics with respect to @RoTZ which reads 2 = Ko (z) (()) with vec(Z) corresponding
2 -ddiag( V f(Z)Z") diag(a) Z diag((a)) (Z(2) diag(B)- diag(3) -1 ddiag(2 (Z V vf(2))
where: for any matrix M, ddiaagM) == diag (inagDia((M)) where Diag(M) extracts its diagonal as a vector and diag(u) is the diagonal matrix with entries of v; and b) the vectors a = oz h(0)))
R and B (ZZ, h(0)) E Re5o (uniquely determined by 7 and h(fo)) satisfy
a² - -1 (3) - x B - - diag(a)-~1m @ B = 0,
(10)
with I2 2 E "m the clement-wise square the matrix Z R= and with À E R", μ R" such that h(0g) = (2, μ)). When À, μ = 0, equation 10 entirelyy characterizes (a, 3).
Proof Given the general definition of CReLU (see e.g. Neyshabur et al. (0015) ; Stock & Gribonval 2002): Gonon et al. )2024)), we study the factorization map
6(u, V. diag(u) V diag(w),
where u e R", paxm w Rm with u, Wj # 0.
Step 1: Gradient flow in parameters.
Let f: Ruxm + R and define the loss E(u, V, w) - f [ōlu, V, w)). Writing Z Φ(u, V, w) and its - Oul, V = -ōvl, i = - 6 is:
gradient G = (Z), the gradient-flow ODE equation
Diag diag(w) V
V' = diag(u) G (iag(w).
à Diag diag(u) (u) G),
Step 2: Induced flow 2.
Since Z = diag(u) u)) Vdiag(w), we have
2 = diag( i) Vvdigg())) Veng(())dia((())) + Vdiag())
Substituting the above yields
Ż = -ddiaggG diag(w)
diag(u²) G diag(w²) diag(u)Vddiagg(V diag(u))))
where we set ddiag = diag (Diag(M))
Eliminating V via V diag(u) Zudiig())) (possible as u1 Wy # on )) and using ddiag( M diag(a)) = dM) diag(a) one obtains
= -ddiaa (Gz diag(u Z diag(u²) G diag(w²) - Z diag(w (w 2) ddiag(Z (TTG) G).
Moreover by Corollary 3.3 there exists conservation laws h and function I' such that 0 = (u, V, w) [()(0), h()) = [ZZ, h(t)) so that a := u² and /3 w² (entrywise multiplication) can both be expressed as functions ax Z, h(f)) and (2 h()).. Below we explicit such conservation laws and characterize properties of a and [3.
Step 3: Conserved quantities and elimination of ox, G.
The flow equation preserves the following n + m conservation laws:
V2
fJ
j=1
When written m matrix, we denote Z instead of and also view ff(Z) as matrix.
22

---

Published as a conference paper at ICLR 2026
jj 1, ,...,, m
V Pj:
for given constants À (u², w²) > is solution of the coupled system
R" and Rm determined by tg. Since Vij Zzz/(u,00y), then
m 2
0
X u = 0,
j=1
Z u
2²
w
H4 wg =0 O.
In vector-matrix form (with entrywise squaring):
= u²
B = w²
a² "=-12""diag(31) where 2 is the elementwise square of Z and is the element- -iise product.
A o = 0,
B - (1213) diag(a) 1 - p 3 0,
Special case = 0, μ 0. Then the system reduces to
a² (1Z(²) diag(3) (aagg(3-11mm 1m. == = tiag((a)=11) 1n.
(21)
By Lemma G. with Y 1 (possible as Zag u, Vijw, * since θ), the exists a unique solution (a,, B) 0 of the system equation 21
In the scalar case (n |2/2/3,
1) with 12 2² scalar, the solution is a B (1212)2/3
H PROOF OF THEOREM 3.8
Theorem 3.8. Consider 0g == (Ut=0. Vi=o) where both U-=o E Eaxr and Vi=o E Rmxr have full rank r S min(n, mi) and assume S AId, for some TR. Then, on a neighborhood Si of 0:=o:
X ldn +
1 [/XTT, 777 3
X ldm /22ld 447T2
2 -11
(2] 2TZ
(14)
where 1IA the orthogonal projector on rangeA.
Proof. Step 1:
rank of Z.
As r S min(n, m) and as both Ut=o E Rxx and Vi=o E Rmxr have full rank equal to r, remains the case in neighborhood Q of 0o = (Urzo, Vi= and it is also the case for Z UV.
Step A quadratic equation for = TUT. Compute
zz = UV VV = -U(TVV)UT
With the hypothesis
VTU-VVV Ald, get V - Ald,, hence
77 U AT TU
NUUT p2 XP.
Thus P satisfies the quadratic matrix equation
p2 XP T7 = 0.
(22)
Step 3: Simultaneous diagonalisation and scalar reduction.
Write Z' := ZZT.
Because
P= UUT
z' UV= V)UT,
23

---

Published as a conference paper at ICLR 2026
and U U differs from V only by scalar multiple of the identity, have (U (U VVTUUU U). Encapsulating by U and U yields PZ' = P. Hence P and are simultaneousy diagonalisable: there exists orthogonal matrix W E _=== such that
P W diag(o)........ W,
Z' W diag(Hi..... .......... ,.,, WT
with o. μ4 2 and where we assume 01 2 2 nn and Ln 2 2 pn.
In the common eigenbasis, equation 22 becomes for every
of Ao; 14 = O.
Its two roots are
of t / X2 + Aps
2
By the first step, already has that Z2 for any > r: 0; 14 0 that o; o,; and that for any S т, o, 0 and μ4 > 0. Thus /x² + 4p, > |A], the "_" root is negative, while P = UUT is positive-semidefinie. Therefore a; o for i S r. Let us define IT = W diag( 1,0, 1. O)WT the orthogonal projector on range(Z) rango(ZZ) TZT). It follows
that:
1 X Idn V 4ZZT
P 11z
(23)
Step 4: The expression for Q = VV A fully analogous computation gives
z = -VT VVVVV + VI Q² + XQ,
so that Q satisfies
2Q 2 = 0.
(24)
Because Q and T == zT commute, they share an orthonormal eigenbasis in which equation 24
reduces
+ -4 (n:20, 0, μ4 2).
By the first step, one already has that on Ω for any i> r: = 0 and that for any i is r. * 0
and μ1 # O. For S the positive root (as vX2 4/4 |A] is
+
22 + 4p4
2
so that
Q -2 2 x
ldm +
l2 + 4T
(25)
with 7 2T and where I the orthogonal projector on range 7 2).
Step 5: Uniqueness and conclusion In both cases equation
23 lequatiinn 25 are the only solutions
consistent with UUT ž and VVT > and with rank(Z) on Ω. Finally one has on Ω::
Ž = -UU Zf(Z) Vf(Z)VVT
A ldm + 1 22ddm +42 = -I ldn + 2 ATId, + 4 T VS(X) VS(X)
2
2
which concludes the proof.
I PROOF OF THEOREM 3.9
We first show the following lemma:
Lemma L11 If # AId, with a real symmetric matrix, then there exists a skew-symmetric matrix A such that [A # O.
24

---

Published as a conference paper at ICLR 2026
Proof. Let assume S * AId, (in particular necessarily). Thus there are at least two distinct eigenvalues of á and μ associated to the eigenvectors x and y. Then A = zy skew-symmetric matrix that satisfies:
yz
* 0 is a
[A, S] (zy" yr "
)s - S(zy" - yz
x(Sy) T (((x)))))))))))))))) T - -(Sx)y T + (Sy)a* as symmetric
= pay' _ -oyr -5xy + wyy
(wyy T + yy ) + 0.
μ # 8, and which concludes the proof.
We introduce a slightly stronger version of Definition 2.6 involving piecewise C² functions f. Definition I.2 (Strong intrinsic dynamic property). 8, verifies the strong intrinsic dynamic property with respect to & on some open set Q 9 8o, if there is Ko, I2 + Rdxd such that: if (.) E Co satisfies 8(0) = 8 and ((t) = - -V((((t)) with = ff o. where f is piecewise C?, whenever O(1) E 1 and f is differentiable at c(0(1)), then M(O(()) O(1) E Ω.
(o(((()) holds for each such that
Theorem 3.9. Let 0o (Uz=o, Vo-o). Assume that both U E RRaxx and V E Raxx have full rank and that r S max(n, IfS = Un-o VooV # AId,, then Og does not satisfy the strong intrinsic dynamic property with respect to Ulim-
Proof. A. First we show that such an initialization does not satisfy the intrinsic metric property. In light of the necessary condition of Theorem 2.14 we will first characterize (0rāM(0) for any 0 = (U, V). Then, with h(0) = U VT V and s(0) = drrsn(0) = UVT, we will exhibit a subspace V of (0) n ke) such that V g (0). We will then conclude using the needed calculus and Theorem 2.14
Step 1: Characterization of M) for any 0 = (U
By equation 11 (with L = 2, U2 = U,U = VT, one can write M (e) vec(UUT + XVVT) ) for any matrix X E nx Using the Kronecker product and the fact that (A@))yec(X) vec( BXAT) ). this expression can be rewritten as:
((0 Id & (UUT) + (VVT) & Id..
(26)
Thus differentiating equation yields that for any (H, K) of the same dimensions as (U, V) we have
0M (0), K) Idm & (UH + HU + (VK + KV ) @ Id,, and thus: (H, K) E (e)
and only if Idn & (UH + HHU - + TV & Idz, We now show that
kerāM( = (H,, K): THERUH + HUT = pId, V + KVT = =uIdm (27)
The converse inclusion is clear. We now prove the direct inclusion. Let us consider (H, K) E (O), then one has Id s (UHT + HUT)
-VVK T + KVT) & Id,, Still using that (A @ (X) = vec( BXA") and denoting U = UH + HUT and V' = (VK + KV this exactly means that for any matrix X xxm one has U'X -XV'T To conclude, we only need to show that this implies the existence of p. 4 E IR such that U" =,dd, and V = II'Id.... since in
since the equality U'X "VT then also implies μ = pL. This immediate if V this case U' must also be equal to zero U"X = 0 for every X. Assume now that V is non-zero so there exists a vector such that 4"T v + 0. Considering any such and any vector u, and setting X = uv! we have
(U'u)eJ T = -
= =u(""To)"
hence Uuu is colinear with u. Since this holds for any choice of u, we deduce indeed that U is proportional to Id,, A similar reasoning yields that V Id This concludes the proof of equation 27
Step 2: Characterization of a subspace V C ke) kerō Since h(0) = UTU VVV and (00) = T we have
Oh(0). K) =
(H(0).(H,K) UK + HVT
25

---

Published as a conference paper at ICLR 2026
and one easily check that for any 0 such that h(0) S we have
v:= ( -VA T
UA AER* (A) +
C (t) n kcrā
Step 3: Proof that V & (0) The fact that matrix A E R"x satisfies
+ (A + +
A A S
equivalent
U - U (
with As (resp. AA) the symmetric (resp. skew-symmetric) part of A (so that A = As + An). Denote S, (resp. A,) the set of symmetrie (resp. skew-symmetric) matrices and the Lyapunov operator defined by:
L: L: ((A))= As -
ES,
We obtain
== (-V(As--An) (As, A., L(As)
= |AA,
As S f Ald,, by Lemma I.1 there exists a skew-symmetric matrix A E A, such that A S] + O. As U U + V is positive definite (as either U or V has full column-rank) its eigenvalues X; 0 satisfy A; + Xy * 0, so (see e.g. Bartels & Stewart (972) ) in particular the Lyapunov operator: L: S, + S, is invertible. Since A. SJ S SDA S,, we obtain that there exists
As * 0 such that I(As)= , S). This particular choice As and A exhibits a parameter We now show that 0' (O) We proceed by contradiction: if 0' E kerêM (0) then, by equation 27 there exists E IR such that U(A + A)U pdd, and V(A +AVT puldm that say
0' = (UA, -VAT) that satisfies 0' V C (err@s())kkerhil
S = pldn and 2V =puldm.
(28)
When r S max(m, n) and since U, V are full rank, at least one of the two matrices U or V is full column rank r. Without loss of generality let us assume that U is full column rank. Then U U is invertible and we deduce that,
2 M(UTU)-
(29)
Moreover (as we indeed show below) range(U n range(VT) * {0). then by considering U
T y & for some E R and y 1T one deduces from equation 28 that ~||=11 = 2r U' == = Asz = 2y V VAsV = =llvvl and thus = O. Hence As = by equation 29 contradicting L(As) [A S + O, which shows that Ø' & (c)
Thus we only need to prove that one has rangeU rrageV * [00, and indeed:
(imnrang dim(rangc(VTT))
rank(U
(V -dangrange(U" rango(V
=rank(U)
Frank(V)
(aoge((UT)]T)))))
+ rank
0.
where we used in the last line that max(n,
Step 4: Conclusion.
As both U-=u and Vi=o have full rank remains the case in a neighborhood Q of fg. Moreover as < max(n, m) then one of the two matrices has full column rank on Ω. In particular the vertical
(
concatenation has full rank (equal to r)) on Ω as r S max(n, m) S n + m.
26

---

Published as a conference paper at ICLR 2026
Since ( has full rank on S2 by (Marcotte al. 2023 Proposition 4.2 and Corollary 4.4) the vector-valued function h contains a complete set of conservation laws.
We now show by contradiction that for any ! _ Ω2 8, does not satisfy the intrinsic metric property on f2'. . Let us assume there cxists a neighborhood Q S 92 of tn and a set of conservation laws ho for & and a function Ke, such that (0) = Ke (ø(8) for each 0 E MM n where Ma = {0: (f) = (o))))) As the family of conservation laws h is complete on f2 (and in particular on (') and Lie W (0) has constant dimension on S? (and thus on 22') by (Marcotte al. 2023
Proposition 4.3), using (Marcotte et al. 2025 Proposition 2.12) yields that Ma = 10 ho (0)
ho0)) Mbg = {0: : h(0)- h(0p)) 3. Thus the function Ke, also satisfies M() Ko (o())
on Mb, hence h satisfies assumption 2) of Theorem 2. 14 As the rank of θh(()) is constant on St'
we deduce by Theorem 2.14 the inclusion equation 7 which contradicts the previous step.
B. Finally we show that such a Og does not satisty the strong intrinsic dynamic property either.
This is direct consequence of the following lemma.
Lemma L.3. Consider a two-layer linear network parameterized by 0 = VU, x00) = (lin(0) =
U1=u))
UVT, and an initialization On- = (Ur_o, Vt-o) such that the vertical concatenation
has full
Vi=o)
rank. If 0g satisfies the strong intrinsic dynamic property with respect to è on S2 then also satisfies the intrinsic metric property with respect to & some open neighborhood Di 0n-
Consider the function Ka, which existence is guaranteed by the fact that Og satisfies the strong intrinsic dynamic property. We will use the Nhow-Rashesski theorem to show that there is a neighborhood ! S Ω of 6g such that Me, n Ω2 is attainable by patching a finite - V wn (0x().).) each initiated at the ending point of the previous one,
Proof of Lemmal 1.3
number of trajectories On (t) defined via fields
wx(+) F (w( = 05 (OT((T)V (-)V
E C
We will further show that there is piecewise C2 function of such that this is feasible with continuous trajectory f(t) such that O(t)
o á for each t such O(t) Ω and f is differentiable at ((((t)). The strong intrinsic dynamic property of to with respect to ø thus yields M(O(t))= Ka, (o(((t)) every time, and in particular (0) (e,((()) (ø(0)) at the end point of the trajectory. This shows that 0g satisfies the intrinsic metric property with respect to e on !
v((((t)) with
To exploit the algebra calculus, since every w()) E F can be written as
theorem, we first observe that J F,, and that by standard Lie
w(()= Σ (0)))()
we have (i) C (i) where W. = span (v)())) Vice-versa, considering e, E Rd the i-th canonical vector, since f((d) = (e, Φ) is c with Vf(Ø) = C,, we get wx(-) :=
Do (VVL()) (-) E F. hence Lie(W,) (0) C (i). Moreover, (1=D)) Vi-o
has full
rank it remains the case in neighborhood ?! C Ω of Co. With the same arguments as in step 4 of the proof of Theorem 3.9 above, the vector-vat valued function contains a complete set of conservation laws of é and by (Marcotte al. 2023 Propositions 4.2 and 4.3) one thus has (i) = Lie (0) = (TMMi (with Ta the tangent plane at #) for each E .Mb, nΩ'. Finally, choose an open neighborhood s2' C S2" of 0g such that Mr. O S2' is connected set: all the assumptions of the -as--aseevvkki theorem (Jurdjevic 1997 Theorem 3) hold on Ω' thus the attainable sets of F from Og on S2' is exactly M/ n.' This means that for every 0 E Mi n there is a trajectory ((t),z [0. 7] such that ((t) E Q2 at every time, ((0) 6g. A(T)
and [] = Up(tk, fx+1) with (tt) = wy(((()) for t E (tx, tk+1), with Wik E F. Without loss of generality the trajectory does not self-in inffiinterset (if it does at times < T',
shorten it by concatenating the trajectories on [0, 7] and [7', TT). Since there are ( functions fa. such that urk V(Ja o)), and since the trajectory ((1) does not self-intersect, there is piecewise ( function
27

---

Published as a conference paper at ICLR 2026
that matches each fü on each piece of the trajectory 8()), hence 6(t) V((((()) with é = foo. for every E (th: tx+1) and every k.
J PROOF OF PROPOSITION 3.10
Proposition 3.10. Let == (u. with u E I'' and E RR. Then = = )ran(0) (u, E R. We
denote S = u==0% Ut=00 E gxx, Then one has à (()
Proof. Since 0e(()= = [c" ,u" we have
0o(0))o(0)) ||u||2 + ||u11
Since h(0) == uu vo
T is a conservation law of ΦLin for every 0 = (u, v) on the trajectory one
has: S = uu vv and therefore 5 ||u|22ue zuv" zou + || Thus
(r(52) || + ||||| 2z².
As one also has: (1u/12 ||012)] tr( one has:
(8*())8(())T)) (l|u|1 + || = 2([lu|| $ + ||o|() - (1/u12 - 1||13)22
2(tr(5²²) + 222) tr()))
21r((5) 4=² tr(s))
which concludes the proof.
K PROOF OF THEOREM 3.11
Theorem 3.11. (0) satisfies the relaxed balanced condition (Definition 3.6 with X = (A:);
then during the trajectory (t) of equation [ the matrices in equation 11 satisfy ())
(U(((()) and (t) = (U) (t)), where Q₃(x) == 11 1- (x - ax) with an 0 and ag Σ d for k 1,... L 1 and Rz(x) == IR by) with by and by = -K Ax. Moreover ULUI (resp. U, U1) is the unique root of Z = Qo(ULU (resp. of 7, = Rt.1(U;) U1)) with spectrum lower bounded by maxogksL-1 ag (resp. by maxogksL-2 by). This implies that all matrices in equation (aaiioonTTrrree are entirely characterized by Z and the initialization, hence (0) satisfies the intrinsic dynamic property on R.D. with respect to órin-
Proof. Let us first outline the main steps of the proof. We first show that the equalities 2 = Qo(UzU?)) and 2,, Zr. = (U, U1) hold on the whole trajectory. Then we prove that this implies the expression of S, (resp. of Tj) in terms of U.UT (resp. of U.T along the whole trajectory too. Finally we show that along the whole trajectory ULU] and U,UT Un (and therefore all Sy's 's and T2 's are entirely characterized by Zr, = Crin(O (0,) and the initial conditions (captured by A). This will thus imply that (0) satisfies the intrinsic dynamic property RD with respect to dlan.
Step 1: Expression of Z as polynomial in UNUT
Since UJ+1 U,UJT is a set of conservation laws for Grin. the fact that the relaxed balanced conditions equation hold at initialization implies that they hold along the whole trajectory.
We prove by induction on 1 5 < that Ze := Ue... U1 satisfies ZiZT = P((U.UT) for some and Pz(z) xPt-1(æ >x-1) for 2 f & L.
polynomial Pt of degree that satisfy P (z) For = 1 we trivially have Z₁ = U, hence the result is true. Now consider & < and assume
that the result holds true for e 1. Since 7, U+Ze-1 we have
ZeZ, U.(Ze- -UP
equation
U.Pr. -
28

---

Published as a conference paper at ICLR 2026
where used equation 12 for i = = - Denoting Pt-1(z) Pt-1(z - Xe-1) we obtain 7 = (U? U,,, P(-I(U,U,) P. This concludes the induction.
Given the recursion formula for Ps, another easy induction yields
k Me-i) 1585L.
Pa(x) = -T
(30)
==1
Specializing é L we obtain Pt. = Qo as claimed.
Step 2: Expression of Sj (resp. of 2Σ Zz and T2) as a polynomial in UzUZ (resp. in U1).
It is direct consequence of the first step, as we now explain. To show the result Sj. consider the new variable 0' = (V²_j:--.. U1) == (Un....,, Uz+1) and Z' == - .. = U.....U ...UU+ With these notations we have Sj Z'z'T, and the relaxed balanced conditions imply that:
(U(+1) T//11 - U((UU) XIdd, iiis
where X' = (N----1-... X) = (Ar-1)...., A1+1). ). By the first step we obtain the desired expres- sion.
Similar computations with 0' = (UT ..... UJ-1). 1) Z' = UT ... UJ_1 and X = (-A1...., show the desired expression for T, Z' TTT and z ZL as well.
Step 3: Characterization of UnUZ via Ze and the initial conditions. The proof that U, U1 is characterized by Zz (in fact Z Zz) and the initial conditions is similar and therefore omitted.
By the first step we have Z = QU(ULU?) hence ULU is indeed matrix root of this equation. As both matrices Z, and U'U are real symmetric, the above expression shows that we can reduce to the scalar study of their eigenvalues.
As we show below, a consequence of the relaxed balancedness conditions equation I2 is that all cigenvalues of the positive semi-definite matrix ULU belong to the interval 7 [max(0, aj...........................Q 1,, oc). Thus, considering any eigenvalue e 2 0 of the positive matrix Zz
enough to show that the polynomial equation R(X) = Qo(X) e admits a unique root in this interval.
The existence of root in is a consequence of the mean value theorem, since (a.
-e S 0 and lim R(x) +00. To prove uniqueness, we proceed by contradiction: assume that R(X) admits two distinct roots Ø1 < 82 Rolle's theorem R' (X) (X) has root in JI,, xz/. This contradicts the fact that, by the construction of Qo and Rolle's s theorem, all roots of (X) are contained in the open interval (min(0, 01, -.., .L, max(0, a1 -L-1))))))))
By
To conclude the proof, we show that indeed all eigenvalues of ULU, belong to = [max(0, ai....,,............ aL-1), oc). Denote o, = inf sp(U,U,)), < < L. Since each matrix Ug is square and positive semi-definite, we have sp(U,U,") = sp U.) S [0, oo) for every 1 < i i L, and by equation 12 we also have sp(U++1U
4 SPUUU,,)) hence 0++1 = o, + X 2 0 for 1 S i < L - 1. An easy recursion shows that o; 2 max(O, ΣØ= X3) for 1 < S L. hence the result.
We now anticipate a slight generalization part of the results of Theorem 3.11 that will be used later in the proof of Theorem B.13
Lemma K.1 (Perturbed relaxed balanced condition). Consider matrices (Ux) C R and scalars L-11 Denoting h L/L, define
C = max(1, (1, max [1ll),
C2 == max Auu
(31)
:== L max
U+U 122 Ar Id
(32)
0SkSL-2
29

---

Published as a conference paper at ICLR 2026
Fix j E {O....,L- 2} and recall that Sj (UL-1 ... ".)--)(U- (.ut--)"--+++)"'' Define do =
and, for 2 1, ag : h²2 Ei AL-1-i Co = 2C', Ci = (Ct + nh² 1)/h. Then
1-1-(i+)
max ||S}
II k=0
(UL-1UL-) - - < Coee C + +c(i)
(33)
Before proving this lemma, we state the following lemma, as will be essential in the proof of Lemma K.I. it provides a uniform bound on the Lipschitz constant of a class of polynomials.
Lemma K.2 (Uniform Lipschitz bound). Consider Co > O, C1 > O. For any 0 < h S 1. any integer 1 s d < 1/h, any degree-d polynomial
Qa(x)
II (x cn); k=1
with maxa |cal I Coh, and any matrices A, A+ E BR()) == {X ||||| S R(h)} where R(h) = + Cih and where || denotes the Frobenius norm, one has
K |A), with K = K(Co, C1) Che Co(1+Ci) + +ec1
[0a(A+ Ca(A)II S
(34)
h
Proof. Step 1: Scalar Lipschitz constant on the ball Bg. For any matrix polynomial Q(x)
Em=o xm one has, denoting DQ(X) H Em= am Σy=o H
Q(A+A)- Q(A) = / DQ DQ((++1)1A) [DQ(X)[HII &
S (I(IX VX, HA
where (It) = E laml m m-1 (we used here that the spectral norm is bounded by the Frobenius norm).
Step 2: Bounding LQ, (R(h)). Exploiting the coefficient-root relation on Qa that is unitary yields
|amm| < (a) 3m where 3 Coh for any 0
1. Since Fn for any R >>> 0 obtain
(Q) S B Σ
m( m(3R)" m- R d3(1 2R) + d.d-1
m
Insert d 1 d < 1/h, 3 Coh. Since R(h) + CiA < R(1) 1 C1 (as h S 1) we get:
Loa (R(h))
1
Coh Coh (L((+CCoR)))
- Co CCOR(N)
ei
C CoeCo(+CC)) t
h
h
where the exponential bound uses (1 1)afe S for 1 0. We define K ((Co,Ci) ColL+C)) Ci
Step 3: Conclusion. Applying the integral formula for Step with the bound from Step gives [|Qa(A + A) Qa(A)]l S (K/h) ||||
for every A with 4,4+ A BR(B). which is equation 34
We now prove Lemma K.1
Work with the truncated sequence (U,, "ig...,UN Uy) (Uj+1,.... Ur.-2, UT,1), where N L- j. Define Ze
Proof. Step 0: Reindexing.
UY... U for < E N
and Mj = U.UT Then Sy Z-
We also observe that by the definition of v) in equation 1 < f N
32 since = 1/L, we have for each
U. Ug- M U. UU-UULUUU T 12 -e Id, + r2-11
UL-
|| s 121 (35)
30

---

Published as a conference paper at ICLR 2026
Step 1: Polynomial representation with perturbation. We prove by induction that
E = ZiZi - (Mee) satisfies || - EKCU hn
(36)
where the polynomials Pt are defined by
Pi (z) =
Pa(x) = Pt-1 -b-b) (2 < e N),
with by h2 M (matching the re-indexed sequence). and the constant K is obtained by Lemma K.2 applied to the constants Co = 2C2, and C₁ (C2 + nhh 1)/h.
Base case Trivial: Z1 U/, so Z M1 (M) ) and E1 O. Induction step. Assume equation holds at rank & - 1. Since Ze = U2Z U Z = U2( -Z =
+ Fe-
By induction hypothesis and the fact that the spectral norm is bounded by the Frobenius norm, the second term of the right hand side is bounded as
| - C (e 1)KC2( hm S (e (e-1)0 Khn,
hence we only need to show that
(U P.(M))ll C. K hn-
Write Qe-1(x) PE-1(1 11-1). From equation 35 and the definition of M = -1-1--1
one gets
M = UTTU%
U/ - be- Id,, - R'D 112-1l1 S 112n.
Hence
UPPt-1(Mc--UU U,Q6-1 UU--U -A-i--)
-U'QI-IU,T +UZ(Qe U2 Qu-
=M: Qc-
Thus to conclude the induction step we only need to show that
10;(Qe-i(U; U. rre1)) Qu-1(U; U3) S CfK K hn.
By the definition of C1, the matrices A = U?TU", A- satisfy max (I A|I AII < ||| ||||| < C + h2n < + Cyh. Moreover, with the same induction that has led to equation 30 the polynomial (ar) has all its roots bounded by T.h²Cx, hence Qe-1(x) == Pt-1(1 by has all its roots bounded by (T. 1)hiCC < 2Cxh Coh, therefore we can apply Lemma K.2 to obtain, with K = KCCo, C1) CoeCO( cc:
K 110=1 U/ r(-1) QU- UiLl - h
|112
s K hn
(U, Q-- (U- U
s C.K. byj
Co21
Of K
hmps
which concludes the induction.
Step 2: Factorisation of Px. With the same induction that has led to equation
30
we have
N-1
(x
II (o ak),
au = 0, ax
h
1-1
1-
Applying equation 36 with
N and recalling S₁ = Zn I yields
N-1
S = (M + Ex
II
(MN az In) + Ex,
k=0
where | < CBN NKhn < (1+ (,, exp(Ci)
Since MN U UU.-UU-1 T, recover equation 33 claimed.
31

---

Published as a conference paper at ICLR 2026
L PROOF OF PROPOSITION B. 3.12
Proposition 3.12. For any s E [0, 11, consider h, : : = (A,))E(u,1) E E
[A] A.] E Ruxn, where we denote A', Ag. Then for any [0.1], one has for any t:
(((t)) = h, (0(0),, where (t) is the maximal solution of equation 15 with initialization ((0).
Proof. For convenience we recall the state equation equation 16 for Z. where a [0, 1] indicates depth:
dZ.
A. Ze. Zo Idn fixed,
(37)
ds
and recall that the objective function is factorized by ((0) f(Z,=1), where the parameters the family 0 = [AA, E [0, 1]}.
Let 0: [t [0 + 0(1) E x E (Io, be the solution of the gradient flow given by the
family of coupled ODE equation 15
aA, (t) = (t),), with (t) = Oe a
0A, ((t)).
vs E 1],
(38)
with given initialization ((0). Our goal is to show that & (0(t)) 0.
Step I: Computations of o (O()t))
For any 8 E [0, 1), one has by definition
hg (0())) (t)
(OA,(1))
+ [A.(0).
+
Ox
Os
Taking the t-derivative yields
(8.,(t)
à (OA.(1))
+ 2 (L(()),A.0))
(O(tt)
Ot
Ot Ös
Ōt
Ōs
Ot
(tA.t))
OA.(t) Öt
+
(A.(0)T, .A(01
(39)
Ös
Ōt
Ös
Ot
where the exchange of derivatives is justified in Section L.I
Moreover one has
0t [t) AA((t) = (AA(1)
.A.(t) + [A.(t)T, aA.(()
Ot
Ot
Thus by using equation 38
(.,(t)) Ot
= (t)
we obtain
Os ( (-gc(t)
+ (0 - -oo((t) [11++++00000 +1-00(0) (o,(t),,A.0) [A.(t)T,- [c(())
(0()))
0t
Os
(g.(t)
(0g.(t)
T
- [ta(t) ,AA(t)) - [A.(t) (t)). (A,)))).a..t))..
(40)
Øs
Os
The remaining task show that the sum of these terms cancels, using an expression of the gradient.
Step 2: An expression of (t) using the adjoint equation.
To compute the gradient g. ã we introduce the adjoint variable Pontryagin et al. 1962) (.
ar
which satisfies the adjoint equation
0f (t,(t) - A.(t) Ar(t) = az (z))t)
(41)
Øs
32

---

Published as a conference paper at ICLR 2026
Moreover it satisfies shown in Section L.2
(1) = A,(t) Z (t)
(42)
Step 3: Compute 9x (t).
By differentiating equation 42
with respect to s, we get:
Üs (t) (Aa(()
(1)
Ös Z,(t) (,(
Ös
Then by using the adjoint equation equation
41 and the state equation equation 37 one has
Ös (t) = - " (.())).())) + A.(() A
(43)
A.((t) (t) (t)
(44)
-[A.(t)) .g((t))
(45)
Taking the transpose,
( 0s (a((1) --) A.(1) + a.) = [A.()...())) (A.()))A.(())))
(46)
Step 4: Conclusion. By substituting the computed expressions into equation 40 one obtains as claimed that
i (O())) 0.
L.I WE NOW DETAIL EQUATION 39
Theorem L.1 (Commutation of mixed derivatives). Let
X' -c'( I"*x"),, Ilfllx max(11flle llo))
and set B ( 1], R2xx) with the sup-norm || || = || 1oxo Denote X + B, + 8 the spatial derivative. Suppose 0(-) e (", and write s), [0(t)[(s). Then
the mixed derivatives
0,0,A(t,s) and 0,0.A(t,s)
exist for every (t, E [0, x [0, 1] and coincide:
0,0,A(t. s) =),,AA((( (t.
the map s + 0,0,A(t,, s) is continuous.
Proof. Step I: is continuous. For every f E X. ||I[|a = || S max [[0' ||,
so Dllop < 1; hence a bounded and thus a continuous linear map.
Step 2: Temporal differentiability is preserved by D. The fact that the function 0 (valued in the Banach space X) is C means precisely that its (Fréchet) derivative @(t) 0,0(t) X' exists for each and the map + 0(L) is continuous from [0, T) to X.
Applying the continuous and linear operator D yields by linearity
D(O(t + h)) D(O(t)) D (t + h) f(t)
h
h
33

---

Published as a conference paper at ICLR 2026
for every & [0, T] and h small enough such that + h [0, T], and since by continuity of D the right hand side tends ((((t) when h 0, the left hand side also has a limit, showing that
dt (D(O(t))) = D(((t))
for every t E [O.
(47)
Thus the mixed derivative O,O,A(t,.)) exists as an element of
Step 3: symmetry of the mixed derivatives. Evaluating the identity equation 47 above pointwise in and writing A(t, s) [0(t)](s) gives
a, s) (s()(t)) a, (+(t)) (s) 0,0.A(t,s).
Hence the two mixed derivatives exist everywhere and are equal.
Step 4: continuity of + (tt, Since ((t) E X for each t, its derivative -> 0, [é(t)] (s)
is continuous. As 0, (s) 0,0,A(t, s), by the previous step, this exactly means that A,t, s) is continuous.
L.2
WE NOW SHOW EQUATION 42
More precisely, to show equation 42 we will both prove that
0. (1) = (Z.(1) (2,0))))) (VT(2)))) ()(2(())..0)
(48)
and that
= (2
(49)
which will indeed give equation 42
We briefly explain why for given t the matrix (t) never loses its invertibility when s E [0.1] varies, by showing that the determinant can never reach 0. As
O,Z,(t) A.(t) Zt).).
Zo(t) Id,
Jacobi's rule gives
d
det Z.(1) = in)A(()) det Z,(1,, det Zo(1) = 1.
ds
Solving this scalar ODE,
det Z.(1) = expl (r(A,(1)) dr 40, e [0.1].
Therefore (t) E GT(n) for every s.
Since t is fixed, in the following we lighten notations by dropping it from the equations. The proof of equation 49 is direct by showing that A, and 1 2, vS(Zi) satisfy the same ODE equation 41 with the same value at 1. Thus we only need to show equation 48
Proof. To show equation 48 we will use Riesz theorem to identify the expression the gradient. We thus will consider the Hilbert space
1 = 1,2 (10, 1), RTx"),
(U, V.V) L==) = ( V.) ds,
in which the parameter 0 = (A, E RRxx, 8 E [0, 1]) E Raxn) =: X C I,2 lives.
We recall that (0) the unique solution of the state equation equation 16
0,Z. = A.Z., Zo Id,,
Vs E [0. 1).
(50)
and that the cost is factorized by the flow map (8) with smooth scalar field f Rxx E, i.e. ((0) (Z,
34

---

Published as a conference paper at ICLR 2026
1st step: expression of the Gateaux variation of the flow
Let 0 = [A, 5 E [0, | E X' be fixed and pick an arbitrary 60 E X. For E R define the perturbed
coefficient 0 := 0 + 80, denoting its components 0 {Ag s [0, 1]}. Denote by 2 = (05) the flow that satisfies the associated ODE:
0,2g = A 2g = Idn
Vs E [0,
(51)
As e, Z + A C² th function (s, + Z is C² using the Cauchy-Lipschitz theorem with parameter. In particular for any & E [0. 11, c+ Z is C Introduce the first variation
8Z,
-z
A.:
ds
==0
which corresponds to the Gateaux derivative of ('' + (0)) at 0 in the direction = 80. We now show that A, satisfies the following inhomogeneous ODE:
0.D. = A,A, + 8A, Zso
= 0,
E [0, 1].
(52)
where 6A. = d. de A5
Indeed let us consider of = 24-2, for any 0 < <ES1 1. In particular one has of A. Moreover one has:
L. = / (A Au Zu = 10¹ B Zudu + [ Ang
(53)
A ==0 dAr (as + A C²) and where 8 E [0, 1] B is continuous (at O, we define B = 5AA)) as + Ar is C²), and thus is bounded on [0, by a constant that does not depend on E. By dominated convergence, when 0 in equation 53 one obtains the limit:
where B A. -:=1 satisfies B
d
5-0 de
A. = I. (A. + SAu Z,)du,
which coincides with the unique solution of equation 52
Since Zg is a solution for the homogencous part 0,Z, = A.Z, with Zo = Id,, by the variation-of- parameters method, one obtains (as Ao = 0):
A, Uo 1 /- Z 6A, b, dr.
Evaluating at s = 1 gives
[' 7, 7 8A, 7 dr.
821 = -1==1==
(54)
Ju
2d step: Differential of and identification of the gradient.
Because f E 2 (RRxx, ,R) its (Fréchet) differential at M E Ruxa is Df(M)[!
H)), Applying the chain rule to l Zi with the Gateaux differentials De @ and in the direction 80, one obtains,
THER*N
(55)
Dae(0) De (Z) (0)) [ 82,
But as by hypothesis both é and f are Frechet differentiaabl,, one has:
DE(0))0) D [2))
Using equation 55 with M 7 (0) and H 87,
De(0))50)
(56)
By inserting the expression of 8Z, from equation 54 into equation 56 one has:
DE(0)(50) in (f()) Z, 5.A, Z dr.
35

---

Published as a conference paper at ICLR 2026
Because tr(RS) tr(SR), get
tx ((Z)) 6,2,"0A, tr Vf((Z) 212 21,,1 )A,),
and thus by defining for each
(57)
one finally has
DE(0)(50) = 1
-10 tr(G 8A,) dr = (G, (0,,0))22
By Riesz theorem, the gradient in L (i.e. the Fréchet gradient) is the unique element G E I verifying [58) (G, (6) 1. for every 80:
Vi(0)= G.
The transpose in equation 57 finally yields the required formula.
L.3 LINK WITH CONSERVATION LAWS FINTTE DEPTH (INFORMAL).
We assume that 0z = (UL, U1) satisfies the relaxed balanced conditions:
U - U.U = H; L
then as Uy = Id + 2, As, using that A&t1 = AA + A/4 + o ( we get:
H₁
1
L = (Id + Ax+1) ) (Id + Ak+1) - (Id + AA) AAd + AA
T(Id
L
L
L
AT+1 + Ac41 A& + AT
A
Afr Act1
+
L
I,2
1,2
AC
+ +AA-
AsA + +AA Ak
+0
1,2
T2
Now (0)
+
12
I,2
Thus h2 such that Ns. (0) H + o(1. In particular if 0z satisfies the quasi balanced condition
A ld.
U4LU -UU,
1
then one can choose ht as:
h.(0) = A(s)Idn,
with À function such that A(SE) condition.
Ag. We say in that case that 0 satisfies the relaxed balanced
M PROOF OF THEOREM 3.13
M.1 PROOF OF THE THEOREM
Theorem 3.13. the initialization 0(0) satisfies that for each (0(()) X(s)ld, for some X(-) E ((O 1), R), then one has
2, [ ( (x2 2.)dd,
with (s)) = (1-s) (1)-11 - (/)+) where vi : [0. 1] J S A(1-c)dudu and 1/2 [0, 1] Jo fo ((v)dudu. If x-)) 0 (baalnced-comdditon) then 7(-) 0.
(1-
36

---

Published as a conference paper at ICLR 2026
Proof. For any and any integer L we define Sp == 8/8 I. for k 0.... L and:
(+) = (t) + hu,, (Ox,)))) with h == and Xo(t) = 1x-
Since this corresponds exactly to the Euler explicit method with step for the ODE
(L) (1,)),,( Zo(() Idn:
E [0, 1];
one has for any and L (computations postponed in Section M.2 :
sup 0
(tx(t) Z. (t)]l (hh)
(58)
[[0,Z((t) - (,Xx((t)) = O(h),
(59)
with || I| any matrix norm on Rax=, the implicit constant in the notation ((-) is independent and L while it can depend t.
We now fix some t and observe that (t) = (x) with
U²(() Id,, + A. (t). so that (from now on we drop the t variable for brevity)
(60)
0,X, = h 1-1
I-1
U1111--11 >>>>>> j
Uo
2=0
By equation 15 and the relation equation 48 (shown in Section
L.2
we have for any [0, 1]
0,Ag 9. 2
hence
L-1 0,X, - Σ (x1 1=0
As UL-1 ... U1+1 = XLX3+1 j+1 = (21 + (Z,)) + ((h)) = 2, (h) since the invertibility of Z, and continuity of 7, implies that || is uniformly bounded) and Z 7-1 Id, + (h) (since 7agt1 = Zzg + hA., 7., + (h²)), we deduce that
55t) 72 TT (Z,,1) ( T = 1(77 7-11
[(UI-I + (Ih) + O(h)))
UU++)) OO(h),
where in the last line we used that since with any relevant matrix norm since max, |UA|] = 11 O(h) (1/L) we have [|U U,l s [1 (11L)] O(1. Similarly we also have = (1) hence
L-1
0,8, = Σ (UL-1... U,-)(-L ... ..,1)) Uf Un
h Σ o(h)
1=0
j=U
O(k) since 1/1.
Similarly as U--1... ... Up Xx hence
X, = Z2,+ (h)) by equation 58 we get Z,, = (U,-1 ... U.) +0(h) +00h))
0,X, = = 170 (.
4-1
= h
- -00 -0)
==
(61)
37

---

Published as a conference paper at ICLR 2026
We also have
U Uk+uU Uk+1 UxU (Id,, + As (Id, + - - (Id, + hA. (Tdn -hAbw))
M(Ab + A M(A, + +A1e) 112 (Ash Ase) - h² 1/( 11
1A + + -Asu + Ag (h2)
=1ahse (0) + (12)
= >sA(ss))dd + o (12))
(62)
as (0) satisfies the quasi- baaai-ballaned condition and A As, + +hA''s + o (h), where the implicit
(1) 1) is still independent of and L as s E [0, 1 A', is continuous on the compact set [0, 1] and is thus uniformly continuous.
(11) function in the notation o(h?) = h²o
By Lemma K.1 with 2
{(s)) and one has:
L-1-(j-1
II k=0
(UL-IU1-1 - axld,)) + FL-1-16
(63)
with
b Σ
60 =0
a = h
A(sL11 s;) for 21, and |EL-1-5ll S Kn
(64)
i=1
where K (Co exp(Ci) exp(Co((C + exp(Cii) with Co = 20s, Ci (CB + nyli 1)/h,
Cv == maxmma max (k C^ = max |As
(65)
k
& - > Id, ||-
= L2 max 0gk$L-2
(66)
As A(-) is continuous, C2 S || < oo for any L. Similarly, we already used that as s E [0,1] + A, is continuous, Cp = 1 + O(h, and thus C2
(h), again with implicit constant independent of Moreover by equation 62 Th = n/L² = o(h²), and we obtain Ci (C) nph² 1)/h (O(h) + o(h2))/h O(I), hence Ci is bounded uniformly. Finally
we obtain
max E o(1)
(67)
where the implicit function o(1) still independent of L.
We denote
F(UL-1UTT1) II (U--UU - IdId,)
(68)
and use the shorthand A := A/(() A., (t) Raxn for 0 < L 1. Since Us- = Id, + h.A
with h 1/T. and St-1 I
1-1
1+1 for each integer i, have (using Riemann
38

---

Published as a conference paper at ICLR 2026
integration X(+) is continuous): since all the matrices in the product equation 68 commute
P,(Ur-1U1-1)) exp
log log(Ur-1UT_- as Id.dd))
64
exp
Σ log
Id,, +
A + A
(1)
1
L Σ x(1 X(1
#1
Id,
L
L
k=
=16" ((--))d++(k
A + A
I.10
exp
L
I.
A(1 v)do Id, o
T,
k=
exp
83(A1 + +
Σ/ k=0
X(1 - v)d- lId, (1)
= So A(1 ((I-r)duduta())
denote 1 8 E [0, 1] + J0² Jo X(1 v)dudu. By equation 63 and the above derivations one has
Z, = lim
X = lim FL-1 FL-I(UL-1U7-)) exp((A1 + +A) v1(() Id,), 1.+00
+00
and thus
A + A = (Z + (1 Id,
Thus
F2 FU(LL-IU[11) (2,27) exp (s, (1))1111)) - (1,)))
(69)
Similarly before and by adapting the proof of Lemma K. one gets:
j-1
(U--1-.. .......... U.) [U Uo bdld,) o(1) U,) o(1)
(70)
with by = - E X(si) and by 0, and where denote
j-1
G)(UT [U, bældn)
(71)
Similarly above:
GUT Uo) (o) exp
Σ
[U Up brldm))
k=
Ao + A.
k-1
exp
log
Id, +
+
+ I
I
Σ
((s
Id,
L
k=0
i==
[= ((1)
Ag + A
1 S*
exp
L
A(v)do Id, +
T.
k=
85 (Ao + +10)+
1
Σ/
J M(vddo Id, + o(1)
exp
T
K==
=So 18 Xv)dodu+o(1)
39

---

Published as a conference paper at ICLR 2026
We denote 1/2 8
[0. + So So (())dvdu. By equation and the above derivations we have
Z 11
lim X X
lim GL(U)) Uo) exp((Ao Ag 1b2(1) Idn),
1.0++000
1-++00
and thus
Ao + Aj = log 2 21) -1) 1dn
follows that
(U+ Uo) (Z ' exp( 5 +b()
(72)
Finally, combining equation 63 63-equation 67 equatiinnnnnnooooo 68 equation 69 and equation 70 Dequutiio
71
equation 72
we obtain
0,
61
h
1.. (.) U.) +
5
L-1
= h
(())) 0
5=0
= J. (/2 +ol),
where
s1/2(1) 76)) (1 (1) 11 s1))(()) (22(6).
Thus one has:
(1 = f (2 ( [xx]6
which concludes the proof.
M.2 PROOF EQUATION 58 EQUATION
59
We now show that equation 58 equation
59 hold for any L.
Proof. First recall that
(t) = (t) + h.4,, (t),
with Xo(()= = Id,, This corresponds exactly to the Euler explicit formulation of the ODE:
(, (t), Zo(t)
Id s sE (0.11 1]
(73)
with step h 1/L.
We now show both items at once. We fix some t. Set
Z2 Y
Ildr W(0)
W(s) =
R
Y. = 0=Z,s
0
so that
_ W(s) ( A.(1) O (s)).
d
(74)
ds
(t) A,
The corresponding explicit-Eule Euler discretization with step 1/L reads (1)
W Wn + h
(t) Are (t) WE.
(75)
which coincides component-wise with the recursions for Xx and T3 = 0.
At
Because the right-hand side of equation 74 (s, W) +
O,A,(t) As(t)) W is C³ (indeed both
+ As(t) and AtAs(t) C² (cf Theorem L.1 for each 1), the Fuler explicit scheme
converges at order one (see e.g. (Berthelin) 2017 Proposition 10.30)):
anax
W (sx) W& O(h).
USkšL
In particular one get that for any k: Xx(t) Z., (1) + (hh) and (reading the second bloc row the final index k: = L):
||a,z..1() (1) |_ Tzl O(h).
40

---

Published as a conference paper at ICLR 2026
N
LLM USAGE
The authors this paper used Large Language Models to aid and polish the writing of this paper and tool to make some of the proofs.
41


Reviews:
[{"review_id": "0c8EeP4IKc", "reviewer": "Reviewer_cmEp", "summary": "This paper introduces a unifying theoretical framework to support implicit bias-type analyses of deep neural networks. The paper's goal is to establish when the gradient flow on the original parameters $\\theta$ implies a corresponding flow on a lifted variable $z = \\phi(\\theta)$ depending on initialization $\\theta(0)$ and the architecture --- this is defined formally through the \"intrinsic dynamic\" property. The authors define two further properties of the pair $\\theta(0)$ and $\\phi$, which can ensure intrinsic dynamics on $z$. These properties are related through a chain of implications, with necessary and/or sufficient conditions provided for parts of this chain to hold. Finally, the framework is applied to establish\n\n * that any initialization $\\theta_0$ of DAG ReLU networks of arbitrary depth satisfies the intrinsic dynamic property with respect to the path-lifting $\\phi$\n * that the relaxed balanced initialization for arbitrarily deep fully connected linear neural networks (LNN) with square weights satisfies the intrinsic dynamic property with respect to a commonly used reparametrization $\\phi$. This result is extended to infinitely deep LNNs under the same initialization for which the dynamics $\\dot{z}$ is made explicit.\n * the explicit dynamics of $\\dot{z}$ for a 3 layer ReLU MLP.\n * the explicit dynamics of $\\dot{z}$ for two-layer fully connected linear networks with relaxed balanced initialization", "questions": "Please see the above section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper makes a solid contribution to the line of work studying the implicit bias of NNs. The reasons are multifold:\n\n* The proposed framework is novel and provides a unified approach for establishing that a reparametrization of the weights admits intrinsic dynamics, which is essential to implicit bias analyses. This result is very useful as, to my knowledge, existing literature relies on ad-hoc approaches that come with a difficult-to-track fine-grained variation in assumptions. While the difficulty of making $\\dot{z}(t)$ explicit remains, this framework can provide shortcuts to validating potential architecture-initialization-reparametrization combinations. \n* To the best of my knowledge, the related literature is appropriately covered. Connections to prior work are precise and detailed, and paint a clear picture of the current state of theoretical progress in this area.\n* The proofs are presented clearly and, though I have not examined every detail, their main steps appear correct.\n* The paper is well-structured and thoughtfully written, mathematically sound, and with a good balance of grounding examples among the dense theoretical results. The authors did a great job at condensing a high amount of information in a way that is easy to follow and pleasant to read.", "weaknesses": "A balancing discussion on the limitations of this framework / current results is mostly missing. For example,\n * What is the main difficulty in extending Theorem 4.3 to arbitrary $r$? \n * The ReLU architecture and $\\phi_{\\mathrm{ReLU}}$ pair seems to allow for stronger properties leading to intrinsic dynamics, i.e., intrinsic recoverability, compared to linear neural networks with relaxed balanced $\\theta_0$ and $\\phi_{\\mathrm{Lin}}$. Is it coincidental, or does it point to a deeper-rooted limitation of the factorization $\\phi_{\\mathrm{Lin}}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gg2Q4DEAym", "reviewer": "Reviewer_E9V6", "summary": "The paper defines a three increasingly stronger conditions that all imply the existence of intrinsic dynamics. These conditions require the knowledge of the set of invariants / conserved quantities, however a necessary and an equivalent condition that is more easily computable is given for two of these conditions.\n\nThe authors then use this framework to give a number of example of intrinsic dynamics, in linear networks and ReLU networks.", "questions": "- Are there any model type where you believe that these techniques could discover some yet unknown invariants and lead to simplified dynamics?\n- There are very few invariants in ReLU networks (the number of invariant is the squared root of the number of parameters), which suggests that there is no hope to just use invariants to obtain truly lower dimensional dynamics, do you still see hope? The biggest successes at finding simplified dynamics in DNNS is taking infinite width limits, e.g. NTK / mean-field limits. Can these fit in your framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper presents a number of ideas that englobe many previous result in the same framework, and seems to also show that one cannot do better than these previous results (there are no more invariants than those already known).\n\nSome of the intrinsic dynamics provided are to my knowledge new, though they are all very close to already known works.", "weaknesses": "This paper is a bit of a typical example of \"proof by abstract non-sense\", it mainly proves mainly already existing results using very abstract (and arguably quite complex) tools. I appreciate that this type of approach can tell us that there are not more invariants and therefore simplifications than the ones we already knew, but I am not convinced that I need all these tools to find the next invariants on a new model, because it seems that people have been able to identify these invariants naively without issues.\n\nFor linear networks, the relaxed balanced conditions has also been used in this paper https://arxiv.org/abs/2405.17580, which also proves that this condition arises naturally in the infinite width limit. The infinite depth linear network dynamics description is more novel, but it is very similar to the two layer case.\n\nFor ReLU networks, the invariants are just the differences between the norms of the incoming and outcoming weights, which were known for a while too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lO1M6oMoTe", "reviewer": "Reviewer_1Gdc", "summary": "The goal of this paper is to find whether a high-dimensional gradient flow can be written as an intrinsic Riemannian flows in lower-dimensional spaces. The authors analyzed three related properties: intrinsic dynamics, metric, and recoverability. They then connect these properties: if we have iinstrinsic recoverability (connected with conservation laws), then the intrinsic metric property is satisfied, which then \"verifies\" the intrinsic dynamic property. The application to ReLU networks and that to linear networks are also discussed.", "questions": "1. Can the authors provide some novel insights about neural network behavior that was unknown before applying this new framework?\n\n2. One motivation of studying the intrinsic flow suggested by the authors is the connection to implicit bias. Then how do the results appeared in this paper inspire the study of the implicit bias? As one advantage of the framework is the relaxation of the requirement of the mirror flow representation, which in fact helps finding the implicit bias, I'm a bit confused about this question.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper has demonstrated a high level of mathematical rigor. The authors clearly define the involved properties formally, and use a series of lemmas and theorems to formally establish their connections. I find the results provided in Theorem 3.3 technically sound, which provides a viable way to investigate the possibility of studying the low-dimensional intrinsic Riemannian flow. The scope of the framework is somewhat general.\n\nIn addition, the application to deep linear networks (the relaxed balanced initialization compared to prior works) is interesting, which explains why this kind of initialization is necessary rather than only a clever or convenient choice.", "weaknesses": "1. However, in my view, this paper does not provide fundamentally new insight. The authors indeed broaden the scope of the connection between conservation laws and intrinsic metric for low-dimensional flow, however, this idea has been broadly discussed by prior works (e.g., Bah et al., (2022); Marcotte et al. (2023)) and the authors do not simplify the description. Hence, there lacks an adequate motivation for what this new framework can provide. As a result, I think the scope of contribution of this paper is limited (providing a new characterization for the path-lifting metric). \n\n2. Another major weaknesses is the presentation. I think this paper is very challenging to read. It is too dense. The chain of definitions, lemmas, and theorems is very long, yet the main flow is constantly interrupted by examples and propositions that have already been discussed by other works. This is even more confusing when I find that the authors in fact have one separate section for examples. In addition, the link between high-level concepts is difficult to follow, and there lacks sufficient intuitive explanation for, e.g., why these high-level properties should be equivalent and can be connected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4j3UlBvYyU", "reviewer": "Reviewer_xeEd", "summary": "This paper studied conditions under which the gradient flow in minimizing $\\ell(\\theta)$ can be rewritten as an autonomous flow in $z=\\phi(\\theta)$ for some reparametrization map $\\phi$. Based on conservation laws, the authors derived and analyzed two sufficient conditions: the intrinsic metric property and the intrinsic recoverability property. They showed for general ReLU networks that, parameter points in a dense set admit an intrinsic flow in a neighborhood. For linear networks, they showed that the map from all weight matrices to the end-to-end matrix satisfies the intrinsic metric property under relaxed balanced initializations, but not in general. The authors derived explicit forms of the intrinsic flow for linear networks and an infinitely deep linear network with relaxed balanced initialization, and a depth-three scalar ReLU network.", "questions": "* Is Assumption 2.1 not satisfied by $\\phi_{\\mathrm{ReLU}}$? Consider $g(x,\\theta)= u \\sigma(v^\\top x)=\\mathbb{1}(v^\\top x)uv^\\top x$. When $\\theta$ lies on {$v^\\top x=0$}, $\\phi_{\\mathrm{ReLU}}(\\theta)=u v^\\top$ is not a reparametrization in any neighborhood of $\\theta$, since the indicator function $\\mathbb{1}(v^\\top x)$ is not determined by $u v^\\top$. \n\n* How is the gradient flow solution defined for ReLU networks, for example, the \"maximal solution\" in Definition 2.6 and in Proposition 3.10? In particular, how is the non-differentiability of the ReLU activation at zero addressed? Does a gradient flow solution exist on $t\\in[0,+\\infty)$ for any initialization, and is the solution unique?\n\n* (Following the previous question) Does Proposition 3.10 indicate that, regardless of how the derivative of ReLU at zero, $\\sigma'(0)$, is defined, the flow in $Z$ is always well-defined on $\\Theta$, and the definition of $\\sigma'(0)$ only affects $\\nabla f(Z)$ in the $Z$-flow? \n\n\n**Minor suggestions**\n\n* I find Equation (7) quite interesting. It would be helpful if the authors could provide a geometric interpretation of this condition, in particular an interpretation of the kernel of $\\partial M$. \n\n* In the abstract and the Contributions, there are claims similar to \"For general ReLU networks, we show that, for any initialization, it is possible to rewrite the flow as an intrinsic dynamic.\" According to Corollary 3.9, I would suggest using \"for a dense set of initializations\" instead of \"any initialization\", as these two notions can differ a lot.  \n\n* It might make more sense to group Sections 2, 3.1, and 3.2 together as they concern the general framework, and group Sections 3.3 and 4 together as they concern specific networks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed framework provides a systematic way to study the intrinsic dynamics of gradient flow. It continues two lines of works:\n\n* The paper considered reparameterized flows that are not necessarily mirror flows, thereby extending the framework in Li et al. (2022), and providing an intermediate step for applying tools developed in Azulay et al. (2021). \n\n* It presented several descriptions of the intrinsic metric and intrinsic recoverability properties based on conservation laws, which I find natural and insightful. This connects with the framework in Marcotte et al. (2023). \n\nI therefore believe this work is valuable for the community studying optimization dynamics and implicit bias. \n\nAdditionally, the intrinsic dynamics derived for the depth-three ReLU network are interesting and have not been identified in prior work, to my knowledge.", "weaknesses": "* The negative results for linear network are not strong to me. First, only the sufficient condition, intrinsic metric property, is declined, not the existence of intrinsic dynamic. Second, only a specific reparametrization map, $\\phi_{\\mathrm{lin}}$, is considered. It remains unclear whether $\\phi_{\\mathrm{lin}}$ induces an intrinsic flow, and if not, whether other reparameterization maps might. \n\n* The intrinsic flow for ReLU networks is established only locally, which makes its utility unclear. Current analyses of implicit bias, even those not relying on mirror flows, still require the reparametrized flow to be globally defined, for example those in Azulay et al. (2021).\n\n**Initial recommendation**\n\nDespite the above points, I find this paper valuable to the community and my initial recommendation is acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)