Topology and geometry of the learning space of ReLU networks: connectivity and singularities
Authors: Marco Nurisso
Pierrick Leroy
Giovanni Petri
Francesco Vaccarino
Venue: ICLR 2026 Poster (2026)
Keywords: learning dynamics
topology
neural networks
ReLU networks
geometry
symmetry
loss landscape
gradient
singularity
connectedness
Primary area: learning theory

Abstract: Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.

Published as a conference paper at ICLR 2026
TOPOLOGY AND GEOMETRY OF THE LEARNING SPACE OF RELU NETWORKS:
CONNECTIVITY AND SINGULARITIES
Marco Nurisso's Pierrick Leroy',, Giovanni Petri & Francesco Vaccarino' 1 DDepartment of Mathematical Sciences, Politecnico di Torino, Turin, Italy NPLab, Network Science Institute, Northeastern University London, London, UK
ABSTRACT
Understanding the properties of the parameter space in feed-forward ReLU net- works is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLUU networks built on general directed acyclic graph (DAG) architectures: the (dis connectensss of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.
INTRODUCTION
The success of deep learning has spurred extensive research into the geometry and dynamics of neural network training. While classical results primarily focus on layered architectures, many modern networks adopt more flexible structures, such as directed acyclic graphs (DAGs), arising either from design or from pruning and compression strategies. These architectures challenge existing theory and necessitate new tools to understand their training behavior, particularly in the presence of non-smooth, homogeneous activations like ReLU.
In this paper, study two fundamental training pathologies in DAG-based ReLU networks: the (dis)coomectedness the training- invariant parameter space, and the presence singularities within it. Our analysis is grounded in the observation that training with gradient flow networks with homogeneous activations gives rise to symmetry-inducee conservation laws. These laws constrain learning trajectories to algebraic variety-refent
the invariant der-nefined by system of quadratic equations dependent on the network's ' topology and initialization.
Our contributions are as follows:
We derive an elegant formulation for the conservation laws that arise during gradient flow training of ReLU networks result of rescaling symmetries, in the general of DAG-based architectures.
We extend previous results on shallow networks (Nurisso et al. 2024) by studying the geometry and topology of the invariant set the general architecture case, providing necessary and sufficient conditions for its connectedness based on network bottlenecks and balance constraints.
Equal contribution. Corresponding authors (marco. sso, pierrick. 1eroyy [po1 ito.

---

Published as a conference paper at ICLR 2026
We identify and analyze singularities of the invariant set, showing that they correspond to disconnected sub-networks, and prove that they are unreachable under standard gradient flow from generic initializations.
We propose nuclear norm- norm-based regularizer that promotes convergence to singular config- urations, thereby enabling differentiable, structure-agnostic pruning. In our experiments, observe that 1.1 regularization-dees induces similar singular behavior as our dedicated regularizer, and therefore also fosters effective lossless pruning.
not explicitly targeting neuron caansty--mpirriee
Taken together, our results shed light on the interplay between network topology and optimization geometry. They also offer principled pathway for designing pruning mechanisms that exploit the structure of the optimization space rather than relying solely on heuristic sparsity constraints.
1.1 RELATED WORKS
DAG neural networks. General feedforward architectures can be formalized as directed acyclic graphs (DAGs) (Gori et al. 2023 Hwang Tuung 2023 Chirag Agarwal et al.] 2021), or more abstractly Armmnta & Jodoin 2021), though this perspective remains relatively under- explored. DAG structures support variants of topological sorting (Kahn 1962), which recover the notion of layers (Boccato et al. 2024a Chirag Agarwal et al. (021). Both natural and synthetic neural systems align well with this broader formalism (Boccato et al.
2024b Milano et al. 2023).
DAG-like networks can also emerge through unstructured pruning (sce below) or via the sampling Sinplingg of
sparse subnetworks, as in the lottery ticket hypothesis (Frankle & Carbin 2018
Liu et al. 2018 You
al. 2019).
Pruning. Pruning methods are typically classified as structured or unstructured (Hoefler et al. 2021 Cheng al. 2024). Structured pruning targets entire groups of parameters, such neurons or channels (Yuan & Lin 2006 Nonnenmacher et al.] (202T). while unstructured pruning removes
individual weights (Han al. 2015 Frantar & Alistarh 2023), often at the cost of hardware
inefficiency. Sparsity can be induced iteratively during training (Lin et al. 2020; Jin et al. (2016). or
through differentiable techniques and regularization (Savarese et al.) [020 Pan et al.) [016]] Kang
& Han 2020). Recent efforts like any-structural pruning Fang et al. (2023) aim to unify pruning
strategies into general frameworks. Our work approaches pruning through the geometry of the optimization landscape: nuclear norm regularizer naturally promotes sparsity across arbitrary structures in DaG-based networks, though current computational limitations restrict its practical use.
Singularities and deep learning, Singular Learning Theory [Watanabe) 2009; 2007) blends statisti- cal learning with algebraic geometry, treating singularities as central to the learning process in the Bayesian framework, when working with non-identifiable models such neural networks. Recent works have applied its tools to describe modern neural network architectures (Wei et al. 2022 Lau al. 2023 Furrann & Lau, 2024). Singularities try (Marchetti et 2005), which examines the space of functions realizable by networks-often termed the neuro-manifold. The influence of singularities on learning has been recognized for decades: early work (Amari & Ozeki, 2001 Amari et al.) 2001 2006) analyzed their impact on gradient descent in simplified settings. Their relation network topology has also been studied; for instance, skip connections are known to reduce singularities (Orhan & Pitkow 2017).
also foundational neuro-algebraic geome-
Training dynamies of ReLU networks. A large body of work analyzes the gradient flow and descent dynamics of networks with homogeneous activations, including convergence guarantees (Sirignano Spiliopoulos] 2020 Mei et al., 2018 Rotskoff & Vanden-Fijnden (002)) and implicit bias prop- erties (Boursier et al. 2022] Chizat & Bach L020,[Lyuu et al. 2021 (ouudyy et al. 2018). ReL.Uss
non-smoothness poses analytic challenges (Eberle et al. 2021), yet its positive homogeneity enables rescaling symmetries (Dinh et al. (017) and conservation laws (Neyshabur et al.) 2015 Marcotte
et al. 22023] Kunin et al. 22000] Zhao et al. 2022 Tanaka et al. 2020 N Nurisso et al. (024). The discrete-ting time setting of gradient descent has also received attention (Feng et al. 2019 Smith et al.)
2021 Kumin et al. 2020). More broadly, symmetry principles continue to shed light on deep learning
phenomena (Grigsby et al. 2023 Gluch & Urbanke 2021 Bronstein al. 2021
Ziyin et al. 2025
2

---

Published as a conference paper at ICLR 2026
a
V/
Vo
Rescaling symmetry
Invariant set
b
d
Conservation laws Cp
B
-1-1 1
Figure 1: Example of feed-forwand DAG architecture G. b. The incidence matrix B of G with rows associated to input and output neurons removed. c. Visualization of the rescaling symmetry of ReLU neurons. The initialization determines the balance value C. (0, f of every hidden neuron, which characterizes the shape of the invariant set (e).
2 SETUP AND NOTATION
In this section, we start by introducing our notation for the neural network topology. We then review known results on symmetries of ReLU networks and their associated conservation laws, reformulating them in our compact notation, Next, we introduce the notion of invariant ser, whose properties are further investigated in section 3 first its connectedness, and then its singularities, each part including numerical experiments. We conclude and discuss limitations in section 4
DAG neural networks. Consider a general computational graph G describing feed-forward neural network architecture (see the related work section L.I) G is directed, acyclic graph (DAG) on a set of nodes V, called neurons, with edges E. We identify a subset of neurons V1 S V containing the input neurons, such that no edges neurons such that no edges are going out of the elements in Vo. We assume that V, Vo 2, i.i. no neurons have empty neighbors. We write @V denote the of input and output neurons V/ u Vo. All the other nodes V V are the hidden nodes which to be the fundamental computational unit of the neural network V V 0V (Figure [a). For any node o we call Anc(v) the set of its ancestors, i.e. nodes w V such that there exists a path in G from w to and Desc(v) its descendants, i.e. the nodes w such that there exists a path
entering the elements of Vi, and subset Vo V of output
U1
Un in G.
Each edge (i,j) E E has a parameter ((.)) R associated with it and, when data is passed through the network, hidden nodes V sum the values of their incoming edges, apply the ReLU function and output to each outgoing edge the resulting value multiplied by the edge parameter. As is standard in the literature (see e.g. 5.1 in Bishop & Nasrabadi (2006)), ), one can also consider biases in this setup by adding "virtual" input neuron whose input is fixed to I and adding edges from it to every hidden neuron.
We call parameter space θ the set of all parameters, i.c. the vector space of real functions over the edges E - R, O RR and we write (a 0) to indicate the input-output function encoded by G with parameters A. Throughout the paper, it will be convenient to formulate the results by describing the connectivity structure with the incidence matrix B of Gi (Bondy & Murty 1979). B | is a standard object in graph theory that describes how each edge is connected to its endpoints. Its elements are defined as follows: Be,(i.)) 1 if e j. Bu.(i.d) -1 if i and otherwise. See, for example, the DAG in Figure [ and its associated incidence matrix (with rows associated to nodes in @V removed) in Figure
Symmetries of ReLU networks. In this work, following Du al. 2018), study the properties of neural network where the activation function a is homogeneous, namely (x) = o' (æ) z for every and for every element of the sub-differentie of (2x)) if a is non-differentiibll at The commonly used ReLU (o()) max (2. 0}) and Leaky ReLU (z)) = max {2, y2) with S 7 s 1) activation functions satisfy this property.
is well known that the geometry of the parameter space θ is heavily influenced by the properties of the activation function. Some activation functions and specific neural network modules induce
3

---

Published as a conference paper at ICLR 2026
some symmetries in the parameter space, i.e., transformations h of the parameters that do not change the function encoded by the network fc(:;0) = ( 0) (Zhao et al. 2025). In the case of homogeneous activations, the most critical symmetry is given by rescaling (Neyshabur et al. 2015). In fact, the input weights of any hidden neuron can be rescaled by positive scalar a > 0, provided that its output weights are rescaled by the inverse a-1 This result is well-known for single and multi-layer networks and holds even in general DAG architectures as it is defined at single node. We write this as the action of the group I., of positive real numbers on each local parameter space O2 == (0(z))) (x,y) E E and z or 2} by means of Ta (f) Ta (((())))) (f(w.))) (f( ((ad(,,)) v (2 (w.))))) (Figure I
Local conservation laws under gradient flow. The presence of symmetries in the neural network's s parameter-function map induces the presence of same-loss sets of the loss landscape. Let indeed fa(:0): I R D ((1..)) U/) E Rd x RR);"" be training dataset and L θ R be loss function which depends on the parameters only through the output of the neural network that is
J.-1
L(0)=
Σ (fc((z;0).
where l: Ro RVol R is differentiable.
Let us now assume that we train the network using the continuous-time analog of gradient descent i.c. gradient flow (GF), ((1) -vo := -g((()),, where V VeL(O(t)) is the Clarke sub-differential (Clarke et al., 2008). Given that the loss function depends on the parameters only through fa. its value at & must be constant over the orbit of rescaling. This, together with the fact that the gradient of differentiable function a point is orthogonal to the level set at that point, means that the gradient orthogonal to the orbit under the action of rescaling, at any parameter 0 where T(O) is differentiable. This orthogonality condition constrains the possible values the gradient and, by extension, the possible gradient flow trajectories.
This orthogonality condition can be shown to be
(0, g(0)
Σ ((w))((, 1i3-0
E (co))((.5)) =0 fo-zz
for every hidden neuron V (see Section A.2 for details); the gradient modulated by the parameter ()) @ 9 is conserved when passing through each hidden
values a network flow, as the quantity g neuron, where @ denotes the element- wise Hadamard product between vectors. This result is well known and obtained, with a different approach, in e.g. Tanaka et al. (2020).
We propose conveniently re-write the gradient conditions all hidden nodes using a variation of the incidence matrix B of G.
Proposition 1. Let B e RIMHE be the incidence matrix ofG with the rows associated with input and outpul nodes removed; then (0, (0)) O Ve V is equivalent to
B(0 O (()) =.
Proof. The proof follows directly from the definition of the incidence matrix. At any hidden node V. if we denote by 0g the weight of any e E, and by ge the e-th component of g(),, then it holds
B()og(0)). Bu,Bagee=
Σ (( - Σ ((()))9)))) = ((,g(0)) 0. Eb-oy
ecB
f:e-j
Invariant sets.
Equation 2). implics that some quantities are conserved under gradient flow optimization or, equivalently, that the learning trajectories constrained lower-dimensional subset of the parameter space.
This means that do not include regularization terms which depend explicitly on the parameters.
4

---

Published as a conference paper at ICLR 2026
Proposition 2. Let fa; be initialized with parameters 8(0) such that B0(0)² 2 cC RR with 8(0)² the element-wise square of the vector 8(0), then, for every t 2 O, it holds that BO(t)²
Proof.
d Bo(1)² = B 0000 = -2 ((())) = =0.
dt
dt
This result (visualized in Figure [). note, is a general, elegant re-writing of the well-known neuron- wise conservation law (Du et al. 2018 Liang et al., 2019; Kunin et al. 2020 Saxe et al. 2013). As we will sec, this is not a mere notational feature, as this formulation reveals precious insights into the relationship between the training dynamics and the neural network's s graph structure. We now define the invariant set as the set the training trajectories are constrained to due to the conservation laws: the network is initialized in the invariant set, it will remain in it until the end of training.
Definition 1 (Invariant set, generalization of Nurisso et al. (2024)). Given yeV, call invariant set the set Ho (c) s O of the solutions of the system of polynomial equations BB
If we look at the single equation associated with hidden neuron v V, we see that B6 = can be
written
Eo.o Eons c. 1+v
jov
which corresponds to a hyperbolic quadric hypersurface in the local parameter space of v, Og (Figure Te). From the graph's 's point of view, we can interpret this as stating that the vector of squared parameters 8² is akin unconstrained sources/sinks and hidden nodes supplying or demanding some flow according to the value and sign of e (Ford Fulkerson, 1962).
fluid flowing through the edges of G. with input/output nodes acting as
In Nurisso et al. (2024), it is shown that for shallow networks, the geometrical structure of Ho(c) is simple, as the total invariant set factorizes into the Cartesian product of the neurons' quadric hypersurfaces. In the general case (MLP or DAG), the situation is much more complex because the equations of Hc(c) are coupled: parameters associated with internal edges appear in multiple equations.
The invariant set (c) which is an algebraic variety (albeit we do not know whether it is reduced or not), is interesting object because it lies in between the redundant but more "concrete" parameter space and the abstract function space (or neuromanifold (Calin) 2020 Kohn 2024)) the model's implemented function lives in. In fact, fixed any c, no parameters in Ho(c) are observationally equivalent w.r.t. rescalings, that is fa(=0)= = fa(; (0") (0.)).) for no rescaling, thus making the invariant set good proxy for the function space Fo == (fa(=; () O). Nevertheless, as discussed in Nurisso et al. (202)), different values of e correspond to different topologies of Hc((), meaning that Fe does not provide the full picture to understand the learning process. (c((c also has its limits: it might not be identifiable with the functions it contains. Indeed, for two isomorphic nodes i e G with # j. permuting their input and output weights will yield an observationally equivalent parametrization. And if i and j are such that cy. then both parameters will be in (c), and so the parameterization map 8 + fo(::) will not be injective into Fg.
3 GEOMETRY AND TOPOLOGY OF THE INVARIANT SET
The study of the geometric and topological properties of Hc((c) (Definition 1 can give us interesting loss and data-independent insights into the training processes.
Ho(e) is the set of solutions of a system of degree-two polynomial equations, each one corresponding to a quadric. Despite the apparent simplicity, studying general intersections of quadries is not casy (de Medrano 2023) but, in our case, the specific structure of Ho(c) greatly simplifies the process. In fact, the equations _Ø² e correspond to system of coaxial quadric hypersurfaces, meaning that they contain only squares and no mixed terms of the form 0.02. This fact allows us to employ some powerful recent results in topology (de Medrano 2023).
5

---

Published as a conference paper at ICLR 2026
a
Out-battlezeck
b
(c), # 2!?
d.1
It- Be-botsleucc
vB
v3
d.2
Optimisation space neuren 4, 4.8, -3 Optiisation space neuron 4, 8, -.=1.
Trajectory
Start Point X End hant
Disconaected
ii
Connereed
Cu
8(3.4)
0(2,4)
Disconmected
Figure 2: Overview of connectedness. a. In- and but-bottlenecks G. b. The non-emptiness of
Ho(c) is guaranteed if every hidden neuron has input and output edges. c. Different connectedness conditions and intuitive visualizations of the associated algebraic varieties for an out-bottleneck Numerical experiment showcasing training dynamics in a connected and disconnected scenario for DAG network with hidden nodes (d.1).
3.1 NON-EMPTINESS
The first result concerns the non-emptiness of (G for a given e or, from the other point of view, what the possible balance values that can appear from initialization are.
Proposition 3 (Feasible balance). For all c RM, one has Ha(c) + Ø.
This result (proven in Section A.3) means that any balance configuration is achievable through parameter initialization, provided that no neurons are excluded from the computations. From the point of view of network flows, this tells that it is possible build flow 0 that satisfies any supply and demand on the hidden nodes.
the hidden neurons
3.2 CONNECTEDNESS
Nurisso et al. (2024) showed that, for some values of c, the invariant set of a shallow ReLU neural network is disconnected. This means that network initialized in one connected component cannot reach an optimum located in another through gradient flow. Here, we show that the conditions for connectedness in the general DAG resemble the for the shallow case, with additional pathological cases resulting from particular graph topologies.
To find out whether Ho (c) is connected not, will use the following proposition, adapted to case from de Medrano (2023).
Proposition 4 (de Medrano (2023 Proposition 4.7). Ho(c) is connected if and only if. for every
e E, Ho. (c)
where G_ EV {e)).
In other words, Ho (c) is connected it is "robust enough" so that the deletion of single edges does not change the possibility of satisfying the supply and demand conditions of c. From this observation, together with Proposition 3 we see that the cases in which there is disconnection must necessarily come from the presence neurons with only one input or output connection.
Definition 2 (Bottleneck neurons). hidden neuron V an in-bottleneck if deg (v) and an out-bottleneck if deg" (v) = We denote with V V the sets of in and out-bottleneck nodes, respectively. For any out-bottleneck neuron v V call Anc(v) the set of its pure ancestors. i.e. ancestors w Anc(v) such that any path from w to Vo passes through v. Analogously, any in- bootlleneck defines a set of pure descendants Desc(u) v) containing neurons w E Desc(v) such that any path from V, to passes through v. Among the pure ancestors of out-boutleneck say

---

Published as a conference paper at ICLR 2026
set T c AAn(() is closed by forward edges the inclusion UueT (u) N (u) T holds, where N (u) denotes the out-neighhors of u. The analogous notion of closedness by backward edges is obtained by considering descendants and in-neighbors instead.
The removal of the single connection of a bottleneck neuron (Figure 2.b) will disconnect it, and the set of its pure uncestors/descendants will be effectively cut out from the network's ' computation: either because they receive no imputs from V, or because they produce no output to Vo.
It turns out that it is possible to derive complete characterization of connectedness and disconnect- edness, leveraging tools from network flow theory.
Theorem 1. (c) is connected if and only if Vou E VB. VT c Ane(v) s.t. T closed by forward edges Ever Cu 2 0 and Vu e VB. VT c Desc(v) s.t. T closed by backward edges Euer Cu. S O.
Proof. The proof is fairly technical and can be found in Section A.4
Intuitively, disconnectedness is caused by bottleneck nodes such that cutting their single edge makes the balances (supplies/demands) of their pure ancestors/descendants unfeasible. For instance, let us look the out-bottleneck in Figure -e: (i) C₂ O means that e requires more flow coming out than in. This is not feasible because there are no other output connections. (ii) Ca 0 means that requires more flow coming than out, which is always feasible for the small shallow network (ii). More generally, this is feasible in bigger networks unless (iii) there is forward closed set T for which Σ Cş < 0, as predicted by Theorem [ Intuitively, there is too much flow coming in T than be absorbed before Mechanistically, there is disconnection whenever the output/input weight of a bottleneck neuron cannot change sign through gradient flow.
Two immediate corollaries follow, clarifying the problem of connectedness in most practical cases. Corollary 1. G has no bottleneck neurons, then Ho (c) is connected.
Note that bottleneck neurons are rare in MLPs because their existence implies the presence of layer with only one node. The exceptions are given by neural networks built to solve binary classification or scalar regression problems, where there is single output neuron, and the neurons in the last layer are all out-bottlenecks. In that case, we adapt Theorem T finding that the results of Nurisso et al.) (2024) nicely carry over the multi-layer case.
Corollary 2. If G is fully-comnected MLP such that all hidden layers contain more than one neuron, then Hc(c) is connected if and only if c. 2 0 VU e Vīz and cu S O Vu e Viz.
Proof. The result follows from Theorem T by noticing that in a fully connected architecture, the only pure ancestor/pure descendant of node is the node itself.
A practical implication of this section is that the expressivity of ReLU networks can be reduced at initialization to the extent that they lose their universal approximation capability. That is, some functions become immediately unreachable, regardless of the chosen loss function or dataset.
Numerical experiments We illustrate Theorem T on the toy DAG neural network shown in Figure 11, implemented using dedicated software (Boccato et al.) 2024a) and in the gradient descent setting. Neuron 4 will be the out-bottleneck of interest. All hidden neurons (in black) have ReLU activation. With the initialization given by the values in the figure, we have c= (C2,c3, C4) (6) - 1,3). There are 3 forward closed sets of nodes, the largest being T = (2,3.4) with ExeT C = Σ. C = 0 - 4. Therefore, if at initialization 0 (0) < V2, then ΣkeT Cs < 0 and the optimization space will disconnect at neuron 4. Concretely, it means that the balance condition at neuron 4 will forbid sign switches of ((.55)-
We try to make the model learn the function f : (x1, + - + x22) for positive inputs. The only way to output negative values is if ((4.5) < O, and the optimum will not be reachable for (0) 1 < 2 as shown on the right plot of Figure L.2, while the optimum is reachable (and is reached) for 0, (0) = /3 22, as depicted on the left plot. Additional experiments can be found in Section A.9.1
7

---

Published as a conference paper at ICLR 2026
a
Role) Bease
Prop. ol inglle valee alon
d
Slean trat lows elong prezing
G
W Yaww
rucear rom
How
mabler
13000 20300 3000
5000
Mumber Neurate
10
Figure 3: a. singularity of the invariant set corresponds to a configuration where set neurons cut out from input and outputs. b. Visualization of the training dynamics on an invariant set with singularities. c. Proportion of null singular values along training for shallow network with 20 hidden neurons with and without regularization. d. Test losses function of the number of neurons pruned. Shaded regions in c. and d. denote confidence intervals over 50 independent trainings that have converged to low loss solution.
3.3 SINGULARITIES
While singularities in the neuromanifold i.e. the function space have been extensively studied (Amari & Ozekil 2001] Amari al.) [2001] 2006) Henny et al.) (2024), in this part propose complementary perspective by analyzing singularities of the invariant set Ha(c), which sits in the optimization (parameter) space.
Singularities are sub-networks. In algebraic geometry, a singularity refers to a point in a variety where the tangent space is not well-defined. Mathematically, if variety X is given by the system of n equations (()) 0 IR", singularities correspond to the points æ where the Jacobian matrix (xx) Dg(x) has its rank reduced from the maximal over X. When the rank of the Jacobian is maximal, the point is instead said to be regular.
In the case of the invariant set, the Jacobian matrix is computed by taking the derivative of BB w.r.t. 0, resulting in
Jc(0)= Bdd
From Equation (4), we see how the Jacobian matrix has the same structure of the graph incidence matrix B, with each of the edges (the columns) weighted by its associated parameter 6. Therefore, follows that the only way in which the matrix can have a lower rank is when some of the values of 0 are 0, i.e., when some of the edges are cut from the graph.
Theorem 2 (Singularities disconnect neurons). If G' is the undirected graph obtained from G by gluing together input and output nodes and neglecting edge direction, and E()) S E is the set of edges with zero weight O. = o E then
Ja = -M1
where CC(G) the number of connected components of G.
Proof. The proof uses tools from discrete topology and is performed by relating the rank of Jc( to the kernel of a weighted graph Laplacian. The derivation can be found in Section A.5
Therefore, follows that rank (G)) |V| for regular parameters, and its rank decreases for parameters whose zero edges disconnect some hidden neurons from both input and output. If group of neurons is such that it is not connected by any path to both input and output neurons, it means that it is useless component of the network as it takes part in any computation, both in the feed-forward and in the back-propagatinn propagation phases. Singularities, therefore, correspond to effective sub-networks of the original neural network, as shown on Figure Bh. This echoes similar results connecting sub-networks to the singular points of the neuromanifold (Trager et al.) 2019 Shahverdi 2024 AArevann al. 2025).
This observation can be leveraged to prove the following result.
Proposition 5 (Singularities are invariant under GF). Let 0(tp) Ho(c) and W S be discon- nected set of nodes time lo. that is, ((,,)) 0 for (u.u) E with u W and v W or vice versa. Then, later time 1. > ln. ((,,)) (L) still holds and W is still disconnected.
8

---

Published as a conference paper at ICLR 2026
Proof. Intuitively, the result is proved by noticing that, in a singular 8, the activation of the neurons in W will be O, meaning that backpropagation will assign zero gradients to all the edges (u. v), v W A u W. The full proof can be found in Section A.6
Proposition 5 means that if the parameter reaches a singularity, then it cannot escape from it: once a network module has been killed, it cannot be revived.
Singularities are rare. One would be tempted to think that the presence and invariance of singular- ities could provide the explanation for the neural network performing an automatic model selection through the progressive movement from one singularity to another, smaller one. We show here, however, that this picture does not hold for the singularities of the invariant set for two reasons: 1. Given a random initialization, the probability of (c(c) having singularities is O. 2. If (c(c) has singularities, then the learning trajectory cannot reach them in finite time.
Proposition 6, Let c R!V, If Hc(c) admits singularities then there exists a subset of hidden neurons W s V such that Evew Co = =0.
Proof. By definition, a singularity identifies a disconnected set of nodes W & V. If we denote the edges inside W by Ew = {e (u, v) E u, v E W). we have that Evew C. = Ees 6 02 0. This is because each edge in Ew is shared by exactly 2 nodes of W and all other edges in or out of W have weight 0.
To have a singularity, therefore, exact equality condition on must hold (Figure 3 If we sample the initial parameter with any initialization scheme where each parameter is independently sampled from the real numbers TR, we see that the probability that set of neurons will have sum exactly zero must be zero. Moreover, a stronger statement than the one of Proposition can be derived.
Proposition 7. Under GF optimization, we have that rank J (@(0)) rank J (((t)) voste
This result (Section A.7) tells us that a gradient flow trajectory cannot fall into singularity in finite time. Together, Propositions 6aand immlless that singularities generally don't exist in the optimization space when using common initializations and, even with a specifically chosen initialization (c= 0), they are effectively unreachable under gradient flow.
Inducing singularities. As shown above, singularities can allow the model to perform "self-pruning" but they are in general hard to reach. To actively drive the training dynamics towards them, we explore the use of regularization. An underlying motivation is the application of differentiable pruning in a very general way, entirely agnostic to the DAG topology. The idea is illustrated by Figure 3p.
natural approach to target the singularities given by Theorem iss to directly penalize the number of neurons which are connected through paths the input or output. To formalize this, we can leverage the Jacobian of (c)(), (c) 2Bdiag(O)) Promoting singularities corresponds to maximizing the dimension of the tangent space of the invariant set, which translates to minimizing the rank of Jc Since the rank is non-differentiable function, we instead penalize the nuclear norm [Jc(O)] the sum of its singular values- as a smooth surrogate (Zhao 2012).
the
Numerical experiments As illustrative example, test our approach the Breast Cancer dataset (Wolberg et al.) 1993) using a range of MLP architectures-shalleee and deep, with or without biases and skip connections. To approximate continuous gradient flow, we train with SGD using a small step size (0.001), comparing nuclear norm regularization against LI and L2. Tracking the Jacobian rank during training confirms that the nuclear norm consistently drives the model toward singularities (fig. Bp). For instance, a shallow network can disconnect around 18 of its 20 hidden units, whereas 1.2 and unregularized training leave all neurons active. Surprisingly, LI regularization performs similarly to the nuclear norm, despite being traditionally associated with parameter (not neuron) sparsity. This suggests that LI may implicitly promote singularities, and, while a precise theoretical understanding is outside the scope of this paper, we provide an empirical analysis in Section A.10 Finally, reaching singular configurations guarantees the ability to perform lossless pruning. and while L2 is already quite robust to pruning, disconnecting active neurons introduces modifications to the implemented function (fig. (3.). All experimental details can be found in Section A.8 additional experiments in Section A.9.2

---

Published as a conference paper at ICLR 2026
4 CONCLUSION
In this work, we investigated two underexplored pathologies in the training of ReLU neural networks defined over directed acyclic graphs (DAGs); the (dis)connectedness the invariant parameter space and the emergence of singularities within it. By leveraging the symmetry properties of homogeneous activations and analyzing the associated conservation laws under gradient flow, we provided a complete characterization of the invariant set an algebraic variety constrained by quadratic balance conditions.
Our topological analysis revealed that disconnections in the optimization space dictated by the presence of bottleneck neurons and an imbalance flow conditions. We further demonstrated that singularities correspond effective subnetworks, and although gradient flow trajectories cannot reach them in finite time, their role can be leveraged to improve structured pruning. We introduced nuclear norm regularizer that promotes convergence toward such configurations. Surprisingly, we observed that L1 regularization can induce comparable effects, hinting at deeper connection between sparsity and singularity-drive pruning.
Limitations. The limitations of our work mainly lie the fact that the theoretical analysis fully based on the assumption of using neural networks with homogeneous activation functions trained with gradient flow on an unregularized loss. Other training algorithms (like Adam) and regularizers not subject to the same conservation laws described here. Likewise, in discrete settings the conservation laws only hold approximately with bigger stepsizes, incurring in bigger violations the laws.
ACKNOWLEDGMENTY
M.N. acknowledges the project PNRR-NGEU, which has received funding from the MUR DM 352/2022. G.P. acknowledges partial support by ERC Consol idator Grant RUNES (Grant no. 101171380) and the MSCA Doctoral Network BeyondTTheddee(Grig Grrnt no. 101 20085).. This study carried out within the FAIR Future Artificial Intelligence Research and received funding from the European Union Next-GenerationEUU (PIANO NAZIONALE DI RIPRESA RESILIENZA (PNRR) MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 D.D. 1555 1/10/2002, PE00000013), This manuscript reflects only the authors' views and opinions; neither the European Union nor the European Commission can be considered responsible for them.
REFERENCES
Shun-ichi Amari and Tomoko Ozeki. Differential and algebraic geometry of multilayer perceptrons. IEICE transactions on fundamentals of electronics, communications and computer sciences, 84(1): 31-38, 2001.
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Geometrical singularities in the neuromanifold of multilayer perceptrons. Advances in neural information processing systems, 14, 2001.
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamics of learning in neuromanifolds. Neural computation, 18(5): 1007-165,, 2006.
Yossi Arjevani. Joan Bruna, Joe Kilecl, Elzbieta Polak, and Matthew Trager. Geometry and optimiza- tion of shallow polynomial networks. arXiv preprint arXiv:2501.06074, 2025.
Marco Armenta and Pierre-Mare Jodoin. The representation theory of neural networks. Mathematics, 9(24):3216, 2021.
Christopher Bishop and Nasser M. Nasrabadi. Pattern recognition and machine learning, volume 4.
Springer, 2006.
Tommaso Boccato, Matteo Ferrante, Andrea Duggento, and Nicola Toschi. 4ward: A relayering strategy for efficient training of arbitrarily complex directed acyclic graphs. Neurocomputing, 568: 127058, 2024a.
10

---

Published as a conference paper at ICLR 2026
Tommaso Boccato, Matteo Ferrante, Andrea Duggento, and Nicola Toschi. Beyond multilayer perceptrons: Investigating complex topologies in neural networks. Neural Networks, 171:215-22 2024b.
John Adrian Bondy and Uppaluri Siva Ramachandra Murty, Graph theory with applications. north- Holland, 1979.
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamies of shallow relu networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems, 38:20105-20118, 2022.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges. arXiv preprint arXiv:2104.13470 2021.
Ovidiu Calin. Neuromanifolds. In Deep Learning Architectures: A Mathematical Approach, pp. 465-504. Springer, 2020.
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Chirag Agarwal Chirag
Agarwal, Joe Klobusicky Joe Klobusicky, and Dan Schonfeld Dan Schon- feld. Convergence of backpropagation with momentum for network architectures with skip connections. Journal of Computational Mathematics, 39(1):147-158, January 2021. ISSN 0254- 9409. doi: 10.4208/jem. 1912- 1-0-1220--2999 1912-m2018-0277
URL http: doi org/10 4208/
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory, pp. 1305-1338. PMLR, 2020.
Francis H Clarke, Yuri S Ledyaev, Ronald Stern, and Peter R Wolenski. Nonsmooth analysis and control theory, volume 178. Springer Science Business Media, 2008
Santiago López de Medrano. Topology and Geometry of Intersections of Ellipsoids in R n, volume 361. Springer Nature, 2023.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Simon Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
Simon Eberle, Arnulf Jentzen, Adrian Rickert, and Georg S Weiss. Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with relu activation. arXiv preprint arXiv:2 10..08/8888..... 08/06, 2021.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16091-161011 2023.
Yuanyuan Feng, Tingran Gao, Lei Li, Jian-Guo Liu, and Yulong Lu. Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation. arXiv preprint /oXiv: 2019.
L. R. Ford and D. R. Fulkerson. Flows in Networks. Princeton University Press, 1962. ISBN 125399.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 180300333,, 2018.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.

---

Published as a conference paper at ICLR 2026
Zach Furman and Edmund Lau. Estimating the local learning coefficient at scale. arXiv preprint arXiv:2402.03668, 2024.
Grzegorz Gluch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprini arXiv:2104.0555 2021.
Marco Gori, Alessandro Betti, and Stefano Melacci. Machine Learning: constraint-based approach. Elsevier, 2023.
Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks. In International Conference Machine Learning, pp- 11734-11760 PMLR, 2023.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:/5 2015.
Nathan W Henry, Giovanni Luca Marchetti, and Kathlén Kohn. Geometry of lightning self atteentinoo Identifiability and dimension. arXiv preprint arXiv:2408. 17221, 2024.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
Wen-Liang Hwang and Shih- Shuo- Tung. Analysis of function approximation and stability of general dnns in directed acyclic graphs using un-rectifying analysis. Electronics, 12(18):3858, 2023.
Xiaojie Jin, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.00522:: 2016.
Kahn. Topological sorting large networks. Commun. ACM, 5(11)) November 1962.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International conference on machine learning, pp. 5122-5131. PMLR, 2020.
Kathlén Kohn. The geometry of the neuromanifold. Collections, 57(06), 2024.
Daniel Kunin, Javier Sagastuy Sagastuy-Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:2012.04728. 2020.
Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning aware complexity measure. arXiv preprint arXiv:2308. /2/08, 2023.
coefficient: A singularity- aware
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference artificial intelligence and statistics, pp. 188-896. PMLR, 2019.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint arXiv:2006.07253, 2020.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of network pruning. arXiv preprint trXiv:8110005270 2018.
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems, 34:12978-12991, 2021.
Giovanni Luca Marchetti, Vahid Shahverdi, Stefano Mereta, Matthew Trager, and Kathlén Kohn. An invitation to neuroalgebraic geometry. arXiv preprint arXiv:2501.18915. 2025.
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: Conservation laws for gradient flows. Advances in neural information processing systems, 36: 63210-63221, 2023.
12

---

Published as a conference paper at ICLR 2026
Song Mei, Andrea Montanari, and Phan-Mini Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7655-776 2018.
Gianluca Milano, Alessandro Cultrera, Luca Boarino, Luca Callegaro, and Carlo Ricciardi. Tomogra- phy of memory engrams in self-organizing nanowire connectomes, Nature Communicutions, 14 (1):5723, 2023.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. arXiv preprint arXiv:2110.11395, 2021.
Marco Nurisso, Pierrick Leroy, and Francesco Vaccarino. Topological obstruction to the training of shallow relu neural networks, Advances in Neural Information Processing Systems, 37:35358- 35387, 2024.
A Emin Orhan and Xaq Pitkow. Skip connections eliminate singularities. arXiv preprint 171100170..09115. 2017.
Wei Pan, Hao Dong, and Yike Guo. Dropneuron: Simplifying the structure of deep neural networks.
arXiv preprint arXiv: (6666007366 2016.
Grant Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 75(9): 1939,1935,10 2022.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in neural information processing systems, 13:11380-1199 2020.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint 13/26113 2013.
Vahid Shahverdi. Algebraic complexity and neurovaricty of linear convolutional networks. arXiv preprint T.T.:22401.161133 2024.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics, 80(2):72--752, 2020.
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. Journal of Machine Learning Research, 19(70):1-57, 2018.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377-6388, 2020.
Matthew Trager, Kathlén Kohn, and Joan Bruna, Pure and spurious critical points: of linear networks. arXiv preprint arXiv: 2019.
geometric study
Sumio Watanabe. Almost all learning machines are singular. 2007 IEEE Symposium on Foundations of Computational Intelligence, pp. 383-388. IEEE, 2007.
Sumio Watanabe. Algebraic geometry and statistical learning theory, volume 25. university press, 2009.
Cambridge
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's good. IEEE Transactions on Neural Networks and Learning Systems, 34(12):10473-100 10486, 2022.
13

---

Published as a conference paper at ICLR 2026
William Wolberg, Olvi Mangasarian, Nick Street, and W. Street. Breast cancer wisconsin (diagnostic). UCI Machine Learning Repository, 1993. DOL:
/
Haoran You, Chaojian Li, Pengfei Xu, Y. Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard Baraniuk. Drawing early-bird tickets: Towards more efficient training of deep networks. ArXiv, abs/1909. 11957, 2019. URL https:://///. http::/:/app.sssanniii........ www.stemaaiiiiieeeeeee ar org/ CorpusID:202888B2
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(1):49-67 2006.
Bo Zhao, lordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy. Symmetries, flat minima, and the conserved quantities of gradient flow. arXiv preprint arXiv:2210.172/66 2022.
Bo Zhao, Robin Walters, and Rose Yu. Symmetry in neural network parameter spaces. arXiv preprint arXiv:2506. 10018.301.... 2025.
Yun-Bin Zhao. An approximation theory matrix rank minimization and its application to quadratic equations. Linear Algebra and its Applications, 437(1):77-93, 2012.
Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang. Parameter symmetry breaking and restoration determines the hierarchical learning in systems. arXiv preprint arXiv:2502.053000 2025.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIX / SUPPLEMENTAL MATERIAL
A.1 LLM USAGE
LLMs (ChatGPT) were used aid polishing the paper after writing.
A.2 GEOMETRIC DERIVATION OF THE NETWORK FLOW EQUATION
Let 0 e R be parameter configuration. Let focus single hidden neuron i, which associate the neuron-wise rescaling action TX (0) that, for any > 0 acts as T ()) = (l)) T" (O))) = (v.f) on the parameters associated to the edges coming in and out of respectively, and leaves the other elements of 0 unchanged. 7 (0) rescaling of the neuron v. follows from the fact that we are dealing with ReLUU networks that the loss function L is constant over 0, L(T L(0) Ya 0.
R+ is the orbit of 0 under
is not zero on all the edges coming in and out of v, we have that action T is free, meaning that T = 0 > T is a smooth manifold admitting tangent space at
1, implying that the orbit 7 0 is diffeomorphic to R., Therefore, the orbit
The fact that L is constant over O means that O () is contained into the level set of L at O. The gradient of a function at a point is always orthogonal to the level set it is contained in, meaning that, in particular, the gradient g()) will be orthogonal to the tangent space of T at 0.
To derive a vector generating this I-dimensional tangent space, it enough to consider the equation describing T"0, differentiate w.r.t. and evaluate = 1.
d
(0) d aB((.)) = =
(1)
da
(i,0) da a=1
a=1
d (o)
d
0(s,0) -0(,))
(2)
da a-1
(v,j) da a-11
(o)
=f {i.
(3)
da
a=1
(i+)
(4)
Orthogonality of the gradient w.r.t this vector can be written as
d
Tz(0)
((0)=0 =0
(5)
da a-1
Σ (.)))()))))) Σ ((.))(()(ss)) h j
Σ
0 0.g(0) =0
(6)
=0
(0.g (0.g(0)).
A.3 PROOF OF PROPOSITION 3
We start from the following result from Medrano (2023).
Proposition 8 (ide Medrano 2023) Proposition 4.1). The following are equivalent:
1. H((c) is non-empty.
2. c lies in the convex cone generated by the columns of B, c Co
The non-emptiness of the invariant set then completely described by the following proposition.
Proposition 9. Co( (3) R/

---

Published as a conference paper at ICLR 2026
Induced network flow (FP)
[0,ee]
0
(A)
Gu
Intaented]
[6)=]]
no
[0,00
[6 mo
[o.w]]
Anc(v)
10,
[6x,6e]
Figure 4: Visualization of the induced network flow problem (FFt) at bottleneck node u é Vn. In the right panel, we depict the internal ares in gray, the incoming ares in orange, the source ares in blue, the sink ares in red and the circulation are in black.
Proof. To show that, for any c [R! the system Bz c admits non-negative solution I, we take a constructive approach and build explicitly.
Let us initialize Hi by assigning the value 1 to every edge (zu)e Ve E. Let us define the balance vector at initialization, co B1 which, in general, will be different from c, Co # c. For each hidden neuron e V, we make an adjustment to To that corrects the balance value at e.
If (cu) Co. define 6, C. (co) v- Let consider path Pe going from the input neurons V1 (which exists by definition of G) and the vector p, R supported it, that is such that assigns value to each edge in and to all other edges. Consider now x' 20 + 8, pez which is non- negaavvte because 8, 2 0. The balance cy == (Bx'), ww will remain unchanged in all nodes w different from v, cy (cn) because, in such nodes, same quantity is added one edge coming in and one edge going out. In the new balance value will be
6. = (Bx', (Bxo). (B1 (co). 6, (co) c. (co) Cu.
instead (cn)v. > Co, we define 5, = (cn) Gv and add the vector 1 p. supported on a path that connects the output neurons Vo. Once again, in all nodes except for the balance is left unchanged, while in V
(Bx') (BBr))++(BBBB))) (B1p.). (co). 5, = c ((co) c+) c
Repeating this process for any node e V. allows to correct the balance de Cp at all nodes, giving non-negative solution x' to the system Bx
A.4 PROOF OF THEOREM 1
The problems associated with network flow have been extensively studied in the literature (Ford and Fulkerson 1962). The idea of the proof map the statement network flow feasibility problem and then apply well-known results in the field.
We first report here the more general version of the result from de Medrano (2033) that we reported in Proposition 4
Proposition 10 (Ide Medrano 2023) Proposition 4.7). The following are equivalent:
1. Ho(c) is connected.
2. Co( BL: for all i, where B-; denotes the matrix B with the i-th column removed and Co(A) is the convex cone generated by the columns of A.
A.4.1 INDUCED NETWORK FLOW PROBLEM
Let us consider a general ferd-forward network as in Section and an out- aut-bottleneck node v e V We will prove that connectedness is equivalent to the existence of solutions network flow problem associated with each bottleneck node. We now build a flow feasibility problem on a subgraph G in "circulation form". using the notation Fathabaddi and Ghiyasvand (0077
2

---

Published as a conference paper at ICLR 2026
Let Vä. construct directed multi- -raph G, Ag) in the following way.
The node set N₂ Anc(u) {s,t). where and are two extra nodes not from G. The of ares Ag contains the following sets of edges:
the internal ares A. {e- (v1, 01, 02 Anc(u)):
the incoming arcs A, = (s, ( 3(v1, u2) e E with in Anc(v), U2 e Anc(u))
the source ares Ag (s. w)) Anc(v).cw < <0]:
the sink arcs A₁ = (w, t) w Anc( v), Cw > 0): the circulation are s), to adopt the problem's "circulation form" (Fathabadi and Ghiyas vand 2007).
Therefore A. A. A, U. u s(t,s)}. ((t,
In the general network flow problem, each = (u, w) is assigned a lower bound le Lww and a possibly infinite upper bound m. Muw on the possible values the flow on them. In case, fix lower and upper bounds follows. The flow the internal and incoming and the st) are are required to be non-negative: L = 0 and m. = oo for e E A, U {tt, s)). The source are constrained
carry a flow equal the negative of the e value of their endpoints: l. = M. = Can Ve = (8, w) A.. The sink ares are constrained by the value of their starting point: m. = Cw Ve (w,f) As..
The network (Ne., A,) is said feasible if there exists a real function on the edges, called flow and denoted f: A + such that
1. at each node the flow is conserved Em(u,n)e ((www) - E=(mm,)en S(w,s) = Vw N;
2. for each edge the bounds are respected lg $ fe S me Vee A,.
Remembering that all this construction is built around an nut-bottleneck v, call this problem (FF:) for "flow feasibility at u Vg"" and we visualize the construction in Figure 4 The analogous problem for in-bottleneck w VB is converted the same formulation by reversing arrows in the DAG and changing the sign of the e vector, we call it (FF.).
A.4.2 TRANSLATION OF THE CHARACTERIZATION BY NETWORK FLOW PROBLEMS
Having defined the induced problems (FF:) and (FF,),, we have first equivalence lemma: Lemma 1.
HC(C) connected = Vu é VB. (FF,) has solution and, Vw VB. (FF..) has a solution.
Proof. (=) We start with the forward implication, assuming that the invariant set is connected and building a solution for every flow feasibility problem associated to bottleneck neurons.
Assume that (c(C) is connected. Then, by the characterization of connectedness in Proposition 10 know that e Co Co(B..) Vi. This tells that removing any column from B, there still exists x E IR. !E|-1 such that B_
We let e Vg and ask whether (FF;) has solution. Recall that there one-to-one correspondence between columns of B and edges of the DAG.
We choose the column to remove to be the one corresponding to the unique out-edge e* of o and get solution vector x² |E|-1 to B_
3

---

Published as a conference paper at ICLR 2026
} - R.. Using z², we explicitly build solution T the problem (FF)) with node and are sets (N, A) as described above.
x² can be seen function: x² : E {e²}
f:A + R.
x (c)
if A A; if (s, As
-Cu
(9)
Cu
if (u.t) A
E, fe(e)
if (t,s)
Note that we used the fact that incoming edges in GD (Nu, A,) can be mapped one-to-one with edges from V Anc(v) Anc(v). To check that this flow is feasible, have to check the two conditions of conservation and boundedness. For u N {8,1), the conservation of flow is assured by the fact that vx" is a member of (c) and therefore respects the balance conditions with balance Cu. This value Cu. depending on its sign, is accounted for Gy by the ares going to s or For t, conservation is assured by the definition of f and for it is assured by summing conservation equations for all nodes in N (s,
Checking the boundedness condition is immediate for e Ao A,, since x" has only non- negativee values. For source and sink ares, by definition of f they are set with the only possible value, and f(,.s) is non-negative as sum of non-negative terms.
The same reasoning can be applied for any in-bottleneck u Vg-
(-) We now prove the reverse implication and assume Yo e Vg, (FF)) has solution and analogously for (FF.). By Proposition 3 we know that Ha(c) non-empty. To show that is connected, we need to show that c Co(B_,) Vi.
Stated differently, we need to show that when removing an edge, we can still find a solution e ER that satisfies every node's balance condition in the DAG.
Let us pick an edge = (v, 0') E E, remove it, and distinguish 3 cases.
1. If V and v' é Vø. the new DAG still has the property that each hidden node is contained in one path from input V, to output Vo. so we can apply the non-emptiness property on this new DAG and get solution.
2. If e V3 or u' e Vi but not both, we focus on V V. We will construct a function x: E:EN [e*) R, such that 8* / respects all balance conditions the new DAG G_et.
We start by splitting the edge set parts: E {er} = E, Ey L Eo- are defined as follows. For generic edge (u, w) E {e*}, we let e E, if u, w Ane(v) (v) i.e. e is an edge whose nodes are not involved in the network flow problem (FF.). We let e Ey u. (nc(v), that is if both nodes are involved in (FFt). Lastly, Eg- if Anc(v) and w Anc(v) i.e. if the edge a hybrid edge connecting a node not involved in (FF;) node involved in it. Notice that the case u e Anc(v) and w $ Anc(u) does not exist by definition of the pure ancestor set.
where the edge subsets
The next step consists crafting 3 functions on the edges and then gluing them together to obtain a solution. By non-emptiness, again, know that if F., Ø, we can find solution 0 the DAG restricted to Auc(v) i.e. an independent solution for the DAG where we have removed the bottleneck and its pure ancestors. We define:
0
ife E.
((0))(())
(10)
if E.
Then, by denoting with fr solution of (FF.)) ), we define:
0
ife E, ife E,
2)(c) (e)
(11)
4

---

Published as a conference paper at ICLR 2026
Finally, for the hybrid edges in Eo-f. we leverage the fact that they be mapped one-to-one to the incoming edges in (FF Just like we did for E,, we thus assign to each one of them the value of the solution of the network flow problem. Let us denote Za- the function which does this assignment for edges in Fo-f and zero elsewhere.
At this point, the function 0: e+ (xa((e)) 2z ( Ie- respects balance conditions for nodes in Anc(v), this is assured by the xy + Io--f part being a solution (FFF) ). also respects the balance for nodes which are not pure ancestors and are not connected to pure ancestors, by the definition of (e).
For the other nodes u Anc(v) such that 3(, w) E with Anc(v), the balance might not be immediately respected. Indeed, the values assigned to the edges in Eo-f will disrupt the balance given by the Zo part of the function.
Luckily, each of these disruptions may be resolved locally, without influencing the balance of the other nodes. Given any u é Anc(v). we compute its balance c₂ = Ee(u.w)EEN(r) 02 (v..))
Em((uu))eR((e") (u.,w))
If é > Cu. we pick any path p in G_.. from to the output nodes p= (P),... PP). Pı u, pn Vo. Let I be the indicator function which assigns to the edges in p and 0 the other edges. Note that this path exists because u is not a pure ancestor of and thus removing e* does not disconnect from the output nodes. If we define 6' 00 (c,, c we see that the balance of 6' at is
Σ
(G.
Σ Ca (ure dy - cy Cu Cus
(e("
while it is left unchanged any other node in the path because the quantity added to its inputs is the same as the one added to the outputs.
If ca Cu instead, can pick any path p in G_e- from input nodes to Vr,Pn u and define 0' 00 (c., 1, fix the balance
.Pi.....,PP,pp
Repeating this process for every hidden node, we are able to find a function 0* on the edges of G=e- which satisfies the balance equation at every node, meaning that B_ (0*))(0))))) )2+)²=
3. If U e VB n Vg. we have that Ane(v) n Desc(v) u) (u) and so no edge is shared between the two deal with being in- and nat-bootleneck independently.
sets of nodes and
Now that we have a characterization of connectedness in terms of flow feasibility we move on to study this feasibility with the positive cut method. To avoid cluttering, we refer (FF.) denote either (FF)) ) or FFF)) as these are the same problem, differing only in their origin.
We now resort to the following classic result:
Proposition 11 (Hoffman's 's theorem (Hoffman) 1958), reported in Fathabadi and Ghiyasvand 2007
A network with conservation and boundedness constraints with non-negative lower bounds is feasible if and only for every non-trivial partition (S,T) SuU =, called a cut, we have V(S, (S, s O. where:
V(S,T)
Σ Li
Σ
maj
(12)
jeT
JeS
Equivalently, we have that (FF.) does not have a solution if and only if there exists a strictly positive cut:
(FF.) does not have solution 38, c N, SUSTT= V(S.T) 0
(13)
Lemma 2. Let (S,T) be a strictly positive cut i.e. V(STT) > 0. Then
{8,t) c S OR {s,t) cT
Proof. If s S, then t otherwise the are (t, s) goes from T'to to S and it has infinite upper bound. E T, then e T otherwise there is no are with strictly positive lower bound entering T and V(S, 0.
5

---

Published as a conference paper at ICLR 2026
Lemma 3. Let be the bottleneck of the problem (FF.)) and G2 A) the induced network flow problem. Let (u, w) & A with u, w E N {s,t}. Then
uu T = w T
V(S,T)
(14)
w S u S
We say that T is forward closed and backward closed.
Proof. We have that {8, = Anc(v). This means that (u, w) is an internal are so Muw = co. Both statements in S. otherwise V(S, T)) -o0. In other words an are starting in T must end in T, and an are ending in S must start in S.
proved by observing that there cannot be internal are starting in T and ending
Lemma 4. Let (S, be strictly positive cut, Then
{8,1) C
(15)
Proof. From Lemma we know {8,1) C OR OR {8,1] c. Let be the bottleneck of the problem (FF)) and G. (A, the induced network flow problem. We proceed by contradiction and suppose that {s,t} c T. Let us pick an arbitrary node w e N {s, Anc(v). In the DAG, any path going backward from will necessarily pass through an incoming edge the path leave Anc(v), otherwise would be disconnected from the input. Formally,
E N {s,t) 32 e N (s,t) Anc(w), (s, A,
So must be in 7 otherwise (s, would go from T to S with co. By the forward closure property of T from Lemma w T.
Since N > {s,t) f) was picked arbitrarily, it follows that N ts c T. By hypothesis {s,t] c T and so N which impossible since the partition (S,T) must be non-trivial.
Finally, we prove the main theorem. Proposition 12.
Vu VB, VT Ano(v) s.1. T. closed by forward edges, Exer Cu 2 VE V VT Desc( () s.t. T closed by backward edges, Ever Cu $
HG(c) is connected
(16)
Proof. Let e V (FFt) has no solution
V(S,))
3(S,T), forward closed, V(S,T) >
(Lemma 3
37 Anc(v) forward closed, V(N T,T)>
(Lemmas
37 c Ane(r) forward closed,
Σ45
Σ
mij
0
(Equation 12
GET
KT
JET
JU
(17)
In the third row, we used the fact that S = N T. Now, notice that we can rewrite the first sum as Ever lij -E wer Cus because the only non-null lower bounds leaving S are those of source arcs and jel <0
these belong to nodes having Cu 0. The second sum can be rewritten as ΣieT m.j Σ ne' Cu as jef consequence of the forward stability of T: the only edges leaving T are sink arcs from T i.e. from
c2>0
nodes having Gu > 0.
This allows to rewrite the equivalence obtained above
(P FT) has no solution <> 37 с Anc(v) forward closed,
Σ -C Σ Ca >0
WET
wel'
Cu
0u>0
(18)
37 Aue(v) forward closed,
Σ Cu
ueT

---

Published as a conference paper at ICLR 2026
Therefore by negating both statements:
(FF,) has solution VT Anc(v) forward closed,
Σ
Cu 2
(19)
ueT
The complementary statement for in-bottlenecks is obtained by reversing the arrows and following similar steps, and we conclude by using Lemma [
E VB, (FF)) has a solution
Hc((c) is connected
Vo VB, (FF)) has solution
(20)
V VB, VT Anc(v) forward closed, Eve Cn 2
E VB, VT Desc(v) backward closed, EveT Cu $ 0

---

Published as a conference paper at ICLR 2026
A.5 PROOF OF THEOREM
Let us compute rank Jc:
First, observe that if 0. + O Ve e F., then rank Jc(0) rank B) as diag(f) is invertible. Therefore, the rank of the Jacobian can decrease if and only some parameters 0, i.c. when some edges effectively removed from the computational graph.
Define now diag (0) 1/0, if 0. 0 else 0 and E()) (e E 0. 0} the of zero-weight edges. Observe that
rank (() = rank (ddiag())))))) ()) rank Bdiag (O)ddii(()diig))) ddia (0) rank raak(BBe(o)) (B-s(0)).
where Beg B with the columns corresponding to edges in & put to 0. Now, it holds that
(ank(B) rank( -VI dim ker kcr(Bl (BIc(o)):
(21)
where the last equality follows from the rank nullity theorem.
To relate dim ker (er(B" (B=))))) to the topological properties of G, we briefly introduce concepts stemming from relative homology (Hatcher 2002).
Let Co( be the vector space of real functions on the nodes of G (all neurons) e Co(G)
an V V. These functions are customarily called the O-chains (of G). Let Ci be the vector space of real functions the edges of G, y (G) = y, E R; these are called the I-chains (on G). can see the incidence matrix (with the rows associated with all nodes included) as the matrix representation of the linear operator from 1-chains to O-chains B: CG() + Co(G). If 11, is
the indicator function on node U E V and = EeEE Be
By Σ
Σ
Ue
Σ
Ve I
In this setting, ker B is called the 0-th cohomology group of G' and denoted by HG); and dim ker T is proven to equal to the number CC(G) of connected components of G.
Let us now pick the subset of input and output nodes @V c V and define the space of relative O-chains Co(G, Ov) Co(aV) i.e. the quotient space of the functions the nodes modulo the space of functions on the input and output nodes. This means that we identify two O-chains e and e' iff c(v) (() for all internal nodes u e V == aV. An element in Co will thus be an equivalence class
Co(G)
[x] Co(G,OV) = [z] = Σ 1
Σ
x.L1
LueV
veV
as, by definition of quotient vector space, we have that [1.]
ve av.
The incidence matrix B, therefore, induces relative incidence matrix (:C,(G) + Co(G)
Ca(VV) as
By= [By]. From this, we see that
By = Byy
Σ
Σ ye Σ Ye
1
(22)
veV
(0m(110) AeeV
Σ
Σ Ve Σ
Ve
1
(23)
DeV-a==i (=(W,)).EE c=(v.a).uVV
meaning that the relative incidence matrix can be represented with the incidence matrix with the rows associated with nodes in @v removed, i.c., the we used in the text.
In this setup ker ßJ is known as the 0-th relative cohomology group (G, of the pair (G, av) and, given this characterization, we can resort to Proposition 2.22 in Hatcher (2002), and deduce that
dim H (G-E(()) (V) dim ( -
8

---

Published as a conference paper at ICLR 2026
where G/ØG is the graph G with nodes in 0V all glued together, and G-E(o) is the graph G with the edges in E() removed. We can therefore go back to Equation (21) and state that
rankJ(()) = |V] dim HD HG (G-E(e)/OV) + 1- --Vl://)))-------C--------oo--- -C((CCC()))/)) +1
meaning that the rank of the Jacobian is less than its maximum value of [V] only when the number of connected components of the quotient graph G- is greater than 1. This happens if and only if removing edges in (0) disconnects set of nodes from both input and output.
A.6 PROOF OF PROPOSITION
5
To prove the result, we will prove that, if 02 = 0. (to) 0 for every edge e (u, v) with u u W for every edge (u, as above.
W, u W, the same holds for the gradients: Is (t())
Let us denote by Po.Vo the set of all paths from node v to a node in Vo i.e p E Pu.Vo = p= (ui...., )n,)) with u1 v. Un, Vo and (ui, ui+1) E, for all
Let av, En be the activation and pre-activation of neuron v, respectively
Σ ((.,)) ag =0=0((z)) (u.))E
The chain rule allows to decompose the gradient of the loss flowing through neuron in the contributions all path from v to an output neuron as follows:
ÖL da. kzu:
Σ
(24)
oa,
p- (a1 P.Po Daunp
Dzu; Da.:
1-2
For any neuron in the disconnected set u W, it holds that any path to the output y Pv,Vo contains an edge ep such that 0e,- If we notice that Deu.-1
Gew; 0(,,, 1.#;)s we have that, for any path p. the product in Equation (24 will contain ep and therefore its value will be 0 AL = Vu W. Sa.
Let us now prove that this implies that the edges that disconnect W have zero gradient.
First, let (u, w) E with V W and w W.
0L 0L daw 0,
Je 00e Daw 00.
because W.
Let = (w, e E with w e W and v
W. aL da, OL Sa, Ozv
ØL
00
Da, a0. Da, Oz, 00.
where Oz. 00 have 0 activations. Therefore W
aw. Given that W is also disconnected from the input nodes V,, any node inside must
aw and ge 0.
A.7 PROOF OF PROPOSITION
7
Let 0(1) (((((())))))))))))))))()))) be the evolution of the parameter configuration under gradient flow.
Let Jo(0) 2.diag(() and let us derive the evolution equation for JG.
d Jo(t) di
d 2 Bdiag(((t)) 2Bdiiag(((t) = = --BBdia(((((((()) di
(25)
turns out that we can simplify this equation by leveraging the connections between rescaling symmetries and GF dynamics. In fact, Equation 8 of Kunin et al. (2000) states that rescaling symmetries induce a relation between the gradient and the Hessian H of the loss function. If b. is the transpose of the row of B associated with u E V, we have that
H()) O bw)) g by V
(26)

---

Published as a conference paper at ICLR 2026
This follows because b, has a value of 11 on the edges incoming in and +1 the edges outgoing from v. Gathering the identities of Equation (26) associated to all hidden neurons, we get the following matrix equation:
(d)) + Bdiag(o(8))
(27)
Plugging Equation (27 in Equation
25 finally get
dt (t) = (2) Jc()))(()))
(28)
iec the Jacobian evolution is dictated by the Hessian matrix.
It is known (e.g. Blanes al. (2009) page 15) that the solution of Equation 28 can be written in the following way
Ja(t) J6( expl (S. H(s)ds).
where T exp(Jo H(s)ds) is the time-ordered matrix exponential
T Texp( H())ds)=
= Σ Sh dt' S." "tz f.-0 H()
n=0
which ensures that the terms of the exponential series are in the right order, as the matrices H(4) (12) may not commute.
Finally, in Theorem 4 of Blanes et al. (2009). is shown that the time-ordered matrix exponential can be written standard matrix exponential and is therefore invertible for any finite t. This means that rank (t)) = rank(Jc(O)), thus concluding the proof.
10

---

Published as a conference paper at ICLR 2026
A.8
EXPERIMENTAL DETAILS
Here we provide additional details regarding experiments described in the main paper.
A.8.1
DISCONNECTED NEURONS AND PRUNING
Figures 3 show, respectively, the proportion of null singular values during training and the evolution of test loss under neuron pruning for a shallow, bias-free architecture. We report Figure results for a broader range architectures discussed in the main text. Both nuclear norm and LI regularization promoted neuron sparsity (fig.5) left column). Note that skip connections made neuron pruning more challenging as illustrated previous literature (Fang et al. 2023). Below, we describe our two pruning strategies; however in both cases, nuclear norm and LI remain the most robust, vanilla the most fragile, and L2 intermediate (fig. 5 center and right columns).
Architectural details. We evaluate six architectures: shallow network with 20 hidden units, with and without bias; and four multilayer perceptrons (MLPs) with three layers of 10 hidden units-varyyng by the presence bias and whether a skip connection links the first and second hidden layers.
Details on methodology. All architectures were trained from scratch for 5000 epochs using SGD with learning rate of 10-3, Regularization strengths empirically tuned to balance low task loss and singular value minimization::.... = 0.05 (nuclear norm), a 10, and 0 20. The total loss defined L Leesk Oreg For each architecture, we sampled 30 runs that converged to low train loss models to analyze training dynamics and pruning robustness. Singular values below 10 were considered null. The dataset was preprocessed. All experiments ran on CPU over approximately 50 hours.
Pruning. We iteratively remove entire neurons (i.e., groups of parameters) from trained models and measure the degradation in performance, measured by the test loss. For each hidden neuron k we compute principled pruning score
Sk ( C (K,))E
Σ 02y)
(29)
(N.J)E
which is the product of the L2 norms of its input and output weights. This score is low for (nearly) disconnected neurons and invariant under neuron rescaling, making it robust reparameterization. However, a max-based score,
= Max Σ P2 Σ 022))
(30)
(1,))E
(b..))EE
produced slightly different results and is reported as well for comparison.

---

Published as a conference paper at ICLR 2026
Architacture: shallow, noblas
singular values along training
Test loss along mul, pruning
Test loss along max pruning
088
20-
nuccear norm
0.4
0A
vorilo
30000
0.0
0.0
Pruned Neuron
Pruned Neurom
Architecture: bias
singular values training
Test along
less pruning
0.8
0.8
nudder varill
Tist
0.2
02
0
10000 20000 30000 40000 50000 Thaning step
Maurors
Architecture: mip_ mip_noblas__ti nobiee_nnsskip
singular values along training
along pruning
along pruning
0.8
0.8
25
20
15-
Test
0.4
0.4
10
0.2
0
10000 20009 30000
40000 50000
22 202026
20 20220222
Thaning step
Pr heurons
Pruned Neuronn
Architecture: mip mip_bias_ poskip
singular values along training
along pruning
along max pruning
50
0.8
25
00
15
Test
0.4
0.4
t0
0
10000 20009 30000
40000 50000
202030
Trainng stop
Neuron
Pruned heurcns
Architecture: mip noblas skip
singular values along training
Test loss along mul. pruning
Test loss along max pruning
0.8
13
00
Rost
8.
0A
10
10000
20200 sab0s Trainng stop
40000 50000
Pruncd heurons
Pruncd hourons
Architecture: mip blas sidp
singular values along training
Test loss along mul, pruning
Test loss along max pruning
6.8
00
Oxcommecee
0.4
DA-
10
10000
20200
30505 40000
53020
0.2
Traning
Prancd Neurons
Pruncd hourons
Figure 5: (Rows) architectures (Left column) number of almost 0 singular values (threshold: 10-3) along training. (Center column) pruning neurons on trained networks using sş (multiplicative) and (Right column) s (maximum) scores
12

---

Published as a conference paper at ICLR 2026
A.8.2 TRAINING DYNAMICS
For one training of shallow bias-free network, we present the evolution of key quantities during training to highlight how different regularizations affect their behavior.
As shown in Figure both nuclear Oe to zero- consiseettt with the fact that disconnecting neuron in fully connected layer requires its input and output weights to vanish, making C₁ null by definition (Equation (3). Without regularization. Cj values, which should be fixed under continuous gradient flow, can increase due to the effect of discrete optimization steps. These steps cause a gradual drift out of Ho (c), as the update vector deviates from the ideal optimization path. Notably, in this unregularized case, e₁ values remain more stable. Figure present the complementary view singular values.
and Lı regularization drive many neurons' balance numbers
values training
leat norm
vanilla
1.5
6
10
10
307
12
10'
10
307
12
10
10
3.54
302
10
10 10
30"
10
Trining St6p
Training St6p
Training S86p
Training 5869
Figure 6: C values during training of a shallow, bias-free network. e [0, is the neuron index.
Singular values a along training run
nuclear norm
vanilla
L2
1.00-
0.75
0.50
0.25
0.00
Figure 7: Singular values of the Jacobian B diag(O) during training of shallow, bias-free network. k [o, 20] is the neuron index.
13

---

Published as a conference paper at ICLR 2026
A.9
ADDITIONAL EXPERIMENTS
In this part, we present additional experiments that replicate our findings different contexts for completeness.
A.9.1 CONNECTIVITY
Besides the toy model presented in the paper which studies a DAG structure, we replicated connect- edness results for MLPs on synthetic and real data.
Synthetic data. The synthetic setup consisted of a MLP trying to learn sum the components of
2 2-dimensional input vector where each component is drawn randomly from [0, 1]. To solve the task, least one of the neurons in the last layer must have a positive output weight. When the parameter space is disconnected, stated by Corollary 2 initializing
wrong connected component creates a pathology in which the network may blocked from reaching the optimum, as shown on Figure 2 preventing the loss decrease below threshold even the training For the synthetic summing task, this happens because a negative C prevents any sign flip for a neuron. Training losses for 10 layers, 0.5 million parameters MLP are shown Figure X
Training Loss
Ce: Negative, Signs: Negative
C:: Default, Signs: Negative
Gel Negative, Signs: Default
Default, Signs: Default
Training Loss
10"
100
200
300
400
500
Epochs
Figure 8: Training loss for 4 possible initializations of a MLP withb 10 layers. Initializations only differ in the last layer output weights. Cg and signs refer the last layer. Blue curve: when the training space is disconnected (negative (&) and initialization is on a bad component (negative output sign for last layer neurons), the model is unable to learn correctly.
Real data. We also replicate the learning obstruction real data, both for MNIST and for the ViT features obtained from a pretrained vision transformer model (without any finetuning) on a classical high resolution dataset (Elson et al.) (207)) of cats and dogs images treated as 224 by 224 pixels and projected to 20 dimensions with UMAP. Both these tasks are set up as binary classification: the cats and dogs is already binary dataset, while for MNIST we modify the labels to predict whether not the digit is equal or above 5. For both datasets, it is easy for a standard MLP to achieve a low train loss reported Figure | except when the optimization space is disconnected (cy- 0) and the initialization done on a component which does not contain parameters able to predict both classes (e.g. negative output weights on last layer).
14

---

Published as a conference paper at ICLR 2026
MNIST
VT features Cals Dogs
0.7
Defastt Signz: Nepative
1.75
G, Defazlt Signu: Negetive
0.-
Nogutive Signs, Defout
Gji Negetive. 5 Sinns: Defbutt
Default, Signs. Defuutt
Gel Default: Slyns. Defut
0.5
1.25
703 533
www
Trinig L230
0.4
wwm.m
1.00
0.50
0.00
500
1000
1500
2000
3000
2000
3000
4200
5000
6000
Steps
Steps
(a)
(b)
Figure 9: Training losses for 4 types of initializations differing only in the last layer output weights. (a) Training loss MNIST. The model is MIP with 3 hidden layers containing 100, 50 and 10 neurons. (b) Training loss on ViT features extracted from a dataset of cats and dogs images. The model is an MLP with 3 hidden layers containing 20, 50 and 20 neurons.
A.9.2 SINGULARITIES
In addition to the experiment conducted on the Brest Cancer dataset the main paper and further described Section A.8 we obtained similar results in two others contexts: the classification of cats and dogs from VIT features discussed in Section (ection[A.9.1] and a more challenging facial attributes prediction task from face recognition model features, for which we used the CelebA dataset (Liu et al. 2015).
The ViT feature task is easily solved with almost perfect test accuracy by three layer MLP having 2500 parameters and 90 total neurons. Adding in L1 regularization achieved 80% pruned neurons, while adding the Jacobian regularizer around 90%, both without a significative loss of accuracy compared to the vanilla training shown Figure I0
Train Task Loss
Singular Loss
Loss
10:1
Nuclear
0.6
Vanilia
0.2
10-2
10a
1000
2000
1000
2000
1000
2000
Training step
Training step
Training step
Proportion pruned neurons
Test Task
Test Accuracy
1.0
1.00
0.98
Nuclear
0.4
Vanilla
0.2
0.96
L1
0.0
1000
2000
1000
2000
1000
2000
Training step
Training step
Training step
Figure 10: Key metrics along training for binary classification task taking as input ViT features. Colored curved denote different regularizers.
For the more difficult task of predicting facial attributes like "lipstick" or "gender" from features of pretrained face recognition models, test accuracy varied depending the attribute. However both L1 and the Jacobian regularization performed on par with the vanilla i.. unregularized model. Depending the attribute, I.1 achieved [80-90]% neuron sparsity, while the Jacobian regularizer
15

---

Published as a conference paper at ICLR 2026
achieved [85-95]% neuron sparsity. Again, significative impact test accuracy observed, as reported on Figure I1
Arched_ Arched_Fyeb ryobrows
Heavy Makeup
Wearing, Lipstick
Mali
2000
1900
1000
2000
Brueed neuress
0.75
0.75
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.00
0.8
3.0
scorrey
0.5
0.6
0.6
0.4
1.000
10:0
60
40-
20
20
10000
1.000
2000
2000
1000
0.30
p.05
0.05
0.05
0.00
0.00
1000
2000
1200
2000
1000
2000
1000
2000
Steps
Steps
Steps
Steps
Figure 11: Key metrics along training for prediction task aiming predicting whether not facial attribute was present on picture from features extracted with face recognition model (here: FaceNet). Colored curves correspond different regularizers. Dynamic pruning row 2.
16

---

Published as a conference paper at ICLR 2026
A.10 EMPIRICAL COMPARISON BETWEEN LI AND NUCLEAR REGULARIZATION
In the main text we observe that both the nuclear norm and L1 regularizers prune similar number of neurons (see also Figure Bt, Figure 5 Figure 10 and Figure [TI. Here, give empirical evidence key distinctions between these regularizers. We first show that both regularizers cannot be explained by a null model and then illustrate that the nuclear norm preserves edges on active neurons, achieving strong group regularization while L1 more aggressive, also pruning edges belonging to active neurons.
A.10.1 NULL MODEL
In this part we use a simple null model to rule out the explanation that LI regularization achieves neuron sparsity solely due its known edge sparsity mechanism. The null model works as follow: we estimate the probability PL1 for a generic edge to be dropped after training. Intuitively, we choose an edge before training and observe after training under L1 regularization if was dropped or not. To decide if the edge was dropped or not, we use threshold of 10 which corresponds to a clear peak in the parameters values distribution, stable over multiple orders of magnitude. Then, starting from the initial computational graph (i.e. before training, with all edges), we compute analytically the expected number disconnected neurons if every edge is dropped with probability PLı. Results are reported Figure T le]t. We repeat the same analysis for the nuclear norm regularization, using the same threshold we obtain another null model with probability of dropping a random edge of Pnuc and report the expected number of disconnected neurons on Figure [2 right. In summary, both null models cannot explain the number of pruned neurons by the number of pruned edges, meaning that in both cases there must be other underlying mechanisms. The underlying mechanism is explicit in the case of the nuclear norm regularization, since neurons are directly targeted, but remains veiled in the case of Ll. Note also that the null model is an even worse explanation in the singular regularization case, indicating a stronger alternative mechanism.
singular
855.01
08
85N
68
26.0
19
26.6
(a)
(b)
Figure 12: Probability of having at least & pruned neurons under the null model for (a) nuclear norm regularizer (b) LI regularizer. The red dotted vertical line the observed number of pruned neurons, the blue curve the analytic probability of pruning at least & neurons under the null model.
A.10.2 ACTIVE WEIGHTS AND PRUNED NEURONS
To further investigate the difference between L1 and nuclear norm, we turn our attention to the distribution of parameters magnitude, which is plotted on Figure left for 4 representative trainings. For the singular regularization we observe clear separation the magnitude of parameters belonging to pruned and active neurons, and this is not the case for LI regularization. This means that the nuclear norm either drops completely a neuron (i.e. all its edges at the same time) or keeps the neuron active. In contrast, the separation is less clear for L1: there many inactive weights on active neurons. Figure [3 right shows the aggregated results for 30 models of each type, where we observe an overlap of the distribution for L.1 only.
17

---

Published as a conference paper at ICLR 2026
Figu Fommatter each preved
a
Figure 13: a. Absolute value of input parameters for the hidden layer shallow network with bias. b. Absolute value of parameters (including bias) for the same network architecture 30 independent runs.
REFERENCES
Sergio Blanes, Fernando Casas, Jose-Angel Oteo, and José Ros. The magnus expansion and some of its applications. Physics reports, 470(5-6):151-23333 2009.
Santiago López. de Medrano. Topology and Geometry of Intersections of Ellipsoids in R n, volume 361. Springer Nature, 2023.
Jeremy Elson, John (JD) Douceur, Jon Howell, and Jared Saul. Asirra: A captcha that exploits interest-aligned manual image categorization. In Proceedings of 14th ACM Conference on Computer and Communications Security (CCS). Association for Computing Machinery, Inc., Octo- ber 2007. URL https::/www//t http://ct....oceeeeee//////aaaaa
asirra a captcha that orgaa
interest allteeeest-ligned---ee manual image categorizzatinn
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16091-16101, 2023.
Hassan Salchi Fathabadi and Mehdi Ghiyasvand. A new algorithm for solving the feasibility problem of a network flow. Applied Mathematics and Computation, 192(2):429 438, 2007.
L. R. Ford and D. R. Fulkerson. Flows in Networks. Princeton University Press, 1962. ISBN 9780691625393.
Allen Hatcher. Algebraic topology. Cambridge University Press, Cambridge, 2002. ISBN 0-521- 79160-X; 0-521-79540-0.
Alan J Hoffman. Some recent applications of the theory of linear inequalities to extremal combinato- rial analysis. New York, NY, pages 113-117, 1958.
Daniel Kunin, Javier Sagastuy-reena, Surya Ganguli, Daniel LKK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:2012.04728, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
18


Reviews:
[{"review_id": "QDl0LSwCbp", "reviewer": "Reviewer_pzFJ", "summary": "The paper studies invariant sets for gradient flow training of DAG-based ReLU architectures and singularities within those invariant sets.", "questions": "On one hand, singularities could be a concern, because they cannot be escaped once reached. On the other hand you suggest that they may be desirable in the sense that they can be seen as the model performing some automatic pruning during training (and indeed you suggest that one may want to induce singularities intentionally). May question is whether there could not be a worry that inducing singularities prematurely limits the model and prevents it from later converging to more favorable solutions which require all neurons (or at least some of the prematurely pruned ones).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is very well written and provides some insights on properties of the training dynamics.", "weaknesses": "To me the results seem to be relatively minor and easy extensions of previous results. The authors suggest that formulating these conservation laws with the use of the incidence matrix of the DAG gives significant new insight. But as far as I can see, the main insight is that there a singularities when parts of the graph become disconnected, which does not seem to be surprising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8LyWSQ16cV", "reviewer": "Reviewer_j6Nf", "summary": "This paper studies the properties of the parameter space of ReLU networks, notably in order to decide whether this space is connected and/or contains singularities, which are relevant questions to consider when targeting an optimally trained network, or to prune the network without losing performance/expressiveness, respectively.\n\nThe authors consider the framework of Directed Acyclic Graphs (DAGs), which is more general than layered architectures,and focus in this paper on properties of homogeneous activation functions, in particular here the ReLU activation function.\n\nThey show a complete characterization of the connectedness of the learning space under GF with any given initialization, analysing to this end the role of bottleneck vertices in the network (that is, vertices with only one out-going arc, or only one in-coming arc) and the balance conditions (invariant under GF once the initialization is done) on related sets of vertices.\n\nMoreover, the authors study singularities, namely parts of the learning space where part of the network stops contributing to the computation. They prove a link between the existence of such singularities and the already mentioned balance conditions, and that even when the conditions are gathered, a GF algorithm will not stumble upon a singularity in finite time. \nThe authors circumvent this impossibility to favor \"self-pruning\" by using regularization, and provide numerical experiments showing which regularization helps driving the model towards singularities.", "questions": "p3, discussing on re-scaling: Do you assume here and in the rest of the paper that all biases are 0?\n\np4, top of the page: do you have any other requirement on $\\\\ell$, other than it being differentiable? For instance $\\\\ell(x,x)=0$ ?\n\np5, Definition 1: $\\\\theta^2$ is the vector obtained from $\\\\theta$ by squaring each individual element? Or do you here implicitly use some other product?\n\np5, Proposition 3: the point of view of network flows can be obtained in a simpler way as what is done in Appendix A.3. Indeed, since the source and sinks have unconstrained flows, it would suffice to initialize all edge weights with 1, and then correct the balance for each node $u$ with a simple edition of the weights along a path from an arbitrary source to an arbitrary sink going through $u$. Does the algebraic point of view give, in some way, more insight for this paper?\n\np6, Theorem 1: I think the proof could take a shortcut (following the idea of the precedent remark, and the intuition-providing  text at the beginning of page 7: first prove that if the conditions are not satisfied, it is unfeasible to satisfy the responsible set of vertices, and if it is, show that fixing first the edge weights incident to $Anc(v)$ (or $Desc(v)$) to satisfy the local constraints, and then construct the rest of the solution greedily without editing any edge incident to $Anc(v)$ (which is then possible by definition of this set, anything outside is on at least one path from source to sink avoiding $Anc(v)$). \nFor the trivial case where the deleted $e$ does not correspond to a bottleneck, the result is immediate.\nThen, the Proposition 4 directly yields the result.\nIs there a reason for taking the long and more technical way?\n\np6, Figure 2d: the figure is a good illustration of the proved theorem. I don't understand however, what the additional experiments on real data (Appendix A.9.1) bring to the paper, since it needs no further empirical demonstration that the space is disconnected. As I am less acquainted with the experimental side, could you indicate what I am missing here?\n\np8, Proposition 6: is the converse known to be true/false? \n\np9: When and why is self-pruning interesting to have? I understand why one wants to self-prune when the initialization was made such that some singularity exist, but is there an advantage in how expressive the network can be when initialized with a reachable singularity, versus when initialized such that none can be attained by GF?\n\n\nTypos and Suggestions:\n\np3, Symmetries of ReLU networks: $\\\\sigma$ is not introduced, which could be done by adding \"the activation function\" in front.  Moreover, the formulae for ReLU and Leaky ReLU are both wrong: ReLU: $\\\\sigma(z)=\\\\max\\\\{z, 0\\\\}$ and Leaky ReLU: $\\\\sigma(z)=\\\\max\\\\{z, \\\\gamma z\\\\}$.\n\np3, Local conservation laws under gradient flow: the variables $d$ and $e$ are clear from context but should be defined nonetheless. \n\np6, Definition 2: prefer the use of \"...with $V^-_B$, $V^+_B$ the sets..., respectively.\"\n\np6, Figure 2c: (ii) this case is misleading, since it is not obvious without the text that the case (iii) can \"override\" it. It also is technically not true, since there could be a completely independent vertex $v'$ in the network making the space disconnected. Maybe find an alternate formulation meaning roughly \"a priori connected\".\n\np7, Corollary 2: unless some intermediate layer has a single neuron! It might not be an interesting case, but the soundness of the corollary requires excluding it.\n\np7: \"Concretely, it means that the balance condition ... will forbid sign switches ...\" This is the important intuition behind this section, it would be valuable to highlight it more, and potentially to merge it with the previous paragraph. \n\n\n\nI am open to updating my rating of the paper, depending on the answers provided by the authors.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Provides a sound and thorough theoretical analysis of the connectivity of learning space for ReLU-activated DAGs Networks trained under GF after arbitrary initialization.\n\nTheoretical analysis of the conditions of existence of singularities, and of the possibility to reach them, complemented with experimental results on tools to reach these singularities in practice.", "weaknesses": "The results on connectivity might be achievable with simpler tools and less technicality.\n\nThe experimental part on connectivity does not bring anything to the discussion. \n\nThe introduction of some notions and symbols is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6BjbXTkBb", "reviewer": "Reviewer_Fprd", "summary": "This paper studies the feed-forward ReLU networks defined over directed acyclic graphs, examining the (dis)connectedness of the parameter space and the existence of singularities within it. The conservation laws under gradient flow are identified. Due to the disconnectedness of certain parameter configurations, certain singularities are unreachable, reducing the expressivity of ReLU networks at initialization.", "questions": "Is there a particular reason to use the uncommon notation of double angle brackets $《》$? I struggled to understand it from a short inline definition given in line 177. I also didn't know if this notation is essential for reading and understanding the main results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is relatively well-written and polished. Illustrative figures are provided to accompany the theoretical results and aid understanding.\n- The theoretical formulation is clean.\n- The result on the disconnectedness of the parameter space is somewhat surprising. The implication of losing expressivity at initialization seems interesting.\n- Some numerical experiments are conducted to validate theoretical results.", "weaknesses": "- I am wondering whether the disconnected case occurs in fully-connected ReLU networks or not, since the example network given in Figure 2(d.1) does not look like a fully-connected network. If the disconnection only occurs in networks that are not fully connected, then the statement in line 358 may be inaccurate: \"the expressivity can be reduced to the extent that they lose their universal approximation capability\"; because ReLU networks that are not fully connected are not universal approximators to begin with. Please feel free to correct me if I have misunderstood your results.\n- In line 423, the authors state that: \"given a random initialization, the probability of $\\mathcal H_G(c)$ having singularities is zero.\" I trust that this statement itself is correct. However, it doesn't necessarily mean that the gradient flow/descent cannot go near singularities. It's quite common that ReLU networks can have saddle-to-saddle dynamics, in which the gradient flow path passes near a sequence of fixed points [1]. In those cases, even though the dynamics from random initialization never puts the parameters exactly in an invariant set, going near those fixed point is still a very prominent, if not the most prominent, trait of the learning dynamics. If I didn't misunderstand the result, the paragraph \"singularities are rare\" should probably come with more nuance or caveat -- \"probability of having singularities being zero\" doesn't mean that learning dynamics doesn't go near singularities.\n- The conservation laws arising from symmetries are also studied in [2]. I am wondering how their results relate to yours results in \"local conservation laws under gradient flow\" in line 160.\n- It might be useful to also discuss the limitation of studying gradient flow in place of SGD. Because the quantities that obey conservation laws under gradient flow can actually be time-varying in SGD [3,4].\n\n[1] Boursier et al. \"Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs.\" NeurIPS 2022.\n\n[2] Ziyin. \"Symmetry induces structure and constraint of learning.\" ICML 2024.\n\n[3] Liu et al. \"Noise and fluctuation of finite learning rate stochastic gradient descent.\" ICML 2021.\n\n[4] Chen et al. \"Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks.\" NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bYzYMcFjn", "reviewer": "Reviewer_aTae", "summary": "The paper presents their study on the function spaces parameterized by polynomial neural networks (i.e., those whose activation functions are polynomial). There are two main contributions: identifiability and singularity of functions in the neuromanifold (i.e., functions representable by neural networks). For the former, the authors show that for generic functions in neuromanifold, the set of parameters realizing these functions is at most finitely many or singleton, for Multi-Layer Perceptrons (MLP) and Convolutional Neural Networks (CNN) architectures respectively. For the latter, they characterize singularities as functions realized by sparse subnetworks and links this discovery to the sparsity bias of MLPs.", "questions": "1. In section 3.2, the link between optimization on the parameter space and on the neuromanifold is rather hand-waving. I wonder if there is a real relation between these two (under suitable conditions).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper are generally well-written and the results are well-presented. While I do not dive into the proof, their results look sound to me. Two contributions are mathematically interesting and suggest further following work.", "weaknesses": "Several points deserves to be further polished:\n1. Since most architectures use ReLU, I find that it is better to connect the current results to the ReLU cases (authors did admit this limitation in section 5).\n2. The bound on the degree of the activation in Theorem 4.1 is vacuous in the dimensions of the neural network architecture. Hence, I am not sure if this result reflects what we truly observe in practice.\n3. If I understand it correctly, the definition of critically exposed implies that there exists a positive probability that mappings $u$ admit a weight in a critically exposed set as critical points of the training dynamics (provided that we have sufficiently data). However, since we are unable to quantify this probability, they might be negligible and might vanish when dimension increases. I am not sure if we can use this notion to explain the so-called ``bias towards sparse subnetworks'' as in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)