Topological obstruction to the training of shallow ReLU neural networks
Authors: Marco Nurisso
Pierrick Leroy
Francesco Vaccarino
Venue: NeurIPS 2024 poster (2024)
Keywords: learning dynamics
topology
two-layer neural networks
ReLU networks
geometry
symmetry
loss landscape
gradient flow
Primary area: optimization
TL;DR: For some initializations, the space of possible gradient flow trajectories of a shallow ReLU neural network is disconnected, resulting in the training being impossible.

Abstract: Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings.
This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. 
When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments.

Topological obstruction to the training of shallow ReLU neural networks
Marco Nurisso
Pierrick Leroy
Politecnico di Torino & CENTAI Institute Torino,
Politecnico di Torino Torino, 10100 ITALY pierrick. peerrikk.erooypolitooo.
marco www..mmmosssssottooo.oo
Francesco Vaccarino
Politecnico Torino
Torino, 10100 ITALY
francesco aracceco..accarrinooooooto.ooo.
Abstract
Studying the interplay between the geometry of the loss landscape and the optimiza- tion trajectories of simple neural networks is fundamental step for understanding their behavior in more complex settings. This paper reveals the presence of topo- logical obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature the ReLU acti- vation function constrains the training trajectories to lie product of quadric hypersurfaces whose shape depends on the particular initialization of the network's s parameters. When the neural network's output is single scalar, prove that these quadrics can have multiple connected components, limiting the set of reachable pa- rameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments.
Introduction
Training a neural network consists of navigating the complex geometry of the loss landscape to reach of its deepest valleys. Gradient descent and its variants are, by far, the most commonly used algorithms to perform this task. While technically correct, the standard picture of the parameter space as Euclidean space with the trajectory rolling down the loss's surface in the steepest direction towards a minimum is slightly misleading because different choices of parameters can be observationally equivalent i.e. encode the same function [10]. The observational equivalence of parameters shape the loss landscape by imposing specific geometric structures on the parameter space. Minima are isolated points but digeesiimnsion manifolds with complex geometry [7
and the loss function's 's gradients and Hessian are constrained to obey some specific laws 46 27 Gradient-bant based
optimization methods, where the parameters are updated by performing discrete steps the gradient's direction, are thus very much dependent on the
geometry 12 29].
In this work, provide topological perspective the constraints induced by some groups of network symmetries on the optimization trajectories. Topology is field of mathematics that studies the properties of a space that are preserved under continuous deformations. Our main goal is find and quantify in topological terms the impossibility of the training trajectories to freely explore the parameter space and get from any initialization an optimal parameter. This idea is formalized in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the topological notion of commectedness and, in particular, with the O-th Betti number, which counts the number of connected components the space is composed of. The presence, or the absence, of topological obstructions in the parameter space does not depend on the particular loss function or the training data but is intrinsic to the interplay between the geometry and the topology of the parameter space under the action of groups of symmetries inducing observationally equivalent networks.
Main contributions.
Our main contributions are the following.
1. We find that, for two-layer neural networks, the gradient flow trajectories lie on an invariant set, which can be factored the product of quadric hypersurfaces.
2. We analytically compute its Betti numbers, i.e., the number of connected components, holes, and higher-dimensional cavities.
3. We find that the invariant set can be disconnected when the network's output dimension is I, leading clear topological obstruction.
4. We find that the obstruction is caused by "pathological" neurons that cannot change the sign of their output weights when trained with gradient flow.
5. We discuss the relation between the invariant set and the network's 's symmetries, finding that if we consider permutations, the number of effective connected components scales linearly in the number of pathological neurons.
6. We perform numerical validations on controlled toy scenarios, displaying the effect of obstruction in practice.
2 Related work
large body of work studies gradient flow and gradient descent optimization of one hidden layer networks with homogeneous activations. Convergence properties have been found for wide networks with bounded density at initialization [31. The implicit regularization provided is studied
37 43
under various assumptions on: orthogonal input data [4), initialization scale [30 4 wide (overpa-
rameterized regime) and infinitely wide [6) linearly separable data [30 45
Deeper linear networks
24 have also been studied.
These works focus on proving convergence and understanding which (optimal) solution is found, whereas our work investigates the shape of the optimization space and focuses cases where the optimum might not be reachable from a given initialization.
Closer to our work, Safran et al. [39 studies two-layer ReLU binary classifiers with single input and output, counting the number of their piecewise- linear components after training. Eberle et al. [ focuses on the differential challenge posed by the ReLU activation function and studies properties like the uniqueness of the solution of gradient flow differential equation for a given initialization.
ReLU activation is a nonnegative homogeneous function, meaning that particular weight rescalings do not change the neural network's function. This is at the heart of the counterargument measures made by Dinh et al. [10) which shows that the Hessian eigenvalues can be made arbitrarily large in this way. Neyshabur al. (34] explores the effect such rescalings can have on the gradient, proposing rescaling-invane invariant regularization, and Pittorino et al. [36] employs them to define invariant flatness measures. Generally speaking, neural networks possess symmetries (20], and symmetries influence the geometry of training. Du et al. [12 studies how symmetry leads ReLU networks automatically balance the neurons' weights. Kunin al. 277. Zhao et al. [51] study how it constrains the gradient and Hessian matrix, leading to conservation laws w.r.f. gradient flow and Tanaka et (46) leverages it to propose a network pruning scheme. Ziyin [52] studies general mirror-reflect symmetries of the loss function and their effect on the weights of the trained network.
flatness
Other conserved quantitics stem from batch normalization' 's scale invariance (23 47]. The transition from gradient flow to finite step size gradient descent breaks the conservation laws, resulting in altered trajectories 14
Numerous works have explored the geometry and topology of the loss landscape to obtain insight into a neural network's training behavior. Motivated by the striking experimental observation that low
loss points can be connected by simple curves [II] [8] or line segments 40, [6 [5], a large body of
literature tries to understand this phenomenon of mode connectivity under the topological lens of
the connectedness of the loss function's 's sublevel sets [1 35 26), especially for overparameterized Another line of work approaches the connectivity of minima from another
neural networks [9 point of view, studying the presence [50] 38 48) or absence (28) of spurious minima, i.c. minima

---

which are not global. Bucarelli et al. [5] analytically derives bounds on the sum of the Betti numbers of the loss landscape's 's sublevel set. Topological data analysis methods have also been exploited to numerically study the shape of the loss landscape [ 22
3 Setup and preliminaries
3.1 One-hidden layer neural network
Unless otherwise stated, all vectors are column vectors, that is, x (xI...., ra) T E IR Rdxl Let us consider a two-layer neural network f(.0)= R R specified by the function
f(;;0) (
(1)
where r E IR" is the input, 0 (W(), ((2)) with W(()) e Ridd and W( E Rexx are the parameters, a: IR is the component-wise activation function and is the number of neurons in the hidden layer. Notice that we consider a network with no biases, as it allows us a discussion with lighter
notation. The case with biases is discussed Appendix E
In this work, following [12), we focus on the case where a is homogeneous, namely (x) x for every z and for every element of the sub-differential o' (z) if a is non-differentiable The commonly used ReLU (o))) max {z, 0f) and Leaky ReLU (a))) = max {2,7} with 0 y s 1) activation functions satisfy this property.
We call parameter space the vector space e = {0 (W(1,, W(2) w()) Ridd w(2) e R
It will also be convenient examine the single hidden neurons and their associated parameters for the following discussions.
Proposition 1. For the two-layer neural network defined in Equation 1 1,....,,, let (e11,,12......4 en) be the canonical basis of Rixt (Ox = (exa (((2) (W(D) w(22) E 0 then = @ @ O2
Let
and
O
Details of the proof are provided in Appendix A Fixing {I..... .. we can consider O₂ as the parameter space of the k-th hidden neuron, which consists of the inputs and output weights of neuron k. namely the rows and columns of W(1) and W(2), respectively. For simplicity, when we
work in O₁ we write w((2), e&a (1)) and wx(2) wx Interestingly, the decomposition of Proposition Tonly holds for two-layer neural networks and will be crucial to the formulations of this paper's results.
3.2 Symmetries and observationally equivalent networks
is well known that the properties of the activation function heavily influence the geometry of the parameter space O The activation function's commutativity with some classes of transformations can result in the latter having no effect on the function implemented by the neural network. This means that, in general, the mapping from the parameter space the hypothesis class of functions is not injective. Following the terminology Dinh et al. [IO), we say that two parameters 01, O2 θ observationally equivalent, if they encode the same function f(;00;) f(, (.,02) and write 0, 02.
In the case homogeneous activations (ReLU or Leaky ReLU), we describe two kinds of transfor- mations that send parameter into an observationally equivalent one.
Neuron rescaling. The input weights of hidden neuron can be rescaled by a positive scalar > provided that its output weights are rescaled by the inverse a (top panel of Figure Ia), We formalize this as the action of the group R, of positive real numbers O
T:R. O₂ O
(a, 02) - (as) = (a (aaww ((() ...(2)) W())
This action can be naturally extended to the space of all parameters by considering the possibility of rescaling all hidden neurons simultaneously by different factors. If a (a1...., .4) R
T.(0) (d ((2) diag(a)
(3)
3

---

b way
H(2)
Neuron rescaling
71
H(d
Ta
(
00
Neuron permutation
Ps
Figure a. Depiction of the two group actions acting on the space of the network's parameters: the neuron rescaling of Equation ( (top) and the neuron permutation of Equation (4) (bottom). b. Depiction of the geometry of the parameter space induced by the rescaling invariance of ReLU
networks. The dotted lines denote the orbits T(0) while the solid lines represent the invariant sets H((c) associated with and the one associated with its rescaled version 6'. Notice how the gradient of the loss g(f) is tangent to H(cc and orthogonal to T(f).
Given that (az)) (=) when a E IR., we see how 0~ (+).
We write T(0) to denote the orbit of parameter under the action of T, i.e. the set of all parameters obtained from 0 by arbitrarily rescaling the neurons T(0) = (T. (0) }
Permutations of the neurons. Besides rescaling, we can obtain an observationally equivalent network by permuting the hidden neurons in such way as to preserve their input and output weights (bottom panel of Figure
Given the symmetric group on I elements G of the permutations of {1,...,.}, we write the action
P.G. x
T, 0) - (0) (R, w(2) W()
where R2 is the x row-permutation matrix associated the permutation æ.
Given that the activation function a is applied component-wiset we have that commutes with Rx namely
f(x; Pa(O)) = (( (R R, (())))))))00)000)))0))) (w) and thus (f) ~ because R = (R=)-1
Having defined these two actions, say that 0 and 0' are observationally equivalent by rescalings and permutations if can obtained from 0 by finite sequence of actions T and P or, equivalently thanks to Lemma I in the Appendix, if there exists a rescaling and permutation such that 0' Px T. (0). this case, we write 0 T 0'.
3.3 Conserved quantities and the invariant hyperquadries
The presence of symmetries in the neural network's parameter-funntion map results specific geometric structure in the loss landscape. Let indeed D {(x,, y4) E Rd x R [ of N input-output pairs and fix a loss function L θ E which depends the parameters only through the output the neural network (T). that
Ji-1 be training set
N L(0) N & ((((s;))
where R: E + is differentiable. In this work, as empirical risk minimization, we consider the continuous time version of the gradient descent (GD) algorithm (with learning rate h > 0)
0G-1 0, hveL(0.)
(6)

---

named gradient flow (GF), and defined as
d 0(1) =0)= -((((() de
where VoL(((t)) is the Clarke sub- di-fferefettilll 7 which takes into account the parameters 0 where L(0) is von-differentiable. Given that the loss function depends on the parameters only through f. its value at & must be constant over the orbit T()). This, together with the fact that the gradient of a differentiable function at a point is orthogonal to the level set at that point, means that
( T (0)
(8)
any parameter where L(O) is differentiable, as represented in Figure Dp. This orthogonality condition constrains the possible values of the gradient and, by extension, the possible gradient flow trajectories. particular, as proven Liang et [29]. Tanaka et al. [46] Equation (8) equivalent to
d Σ
(L) 9ki Σ
6
(
9jk
=0 vk= vk=l.....
(9)
For convenience of notation, we define, for l = I,...,,, I, the following bilinear forms on θ which help us describe the geometry induced by the rescaling symmetry. If A (W()) ((()) and rj (V(1), V(2)), we define
(0, n
Σ
W
Σw (2) jk
(10)
5k
j-1
which, notice, only depends on the k-th row of W(1)) and k-th column of 22)) meaning that we can cquivalently see it as a bilinear form on Og. Og. together with
( is a Eseudd-EEcclide space.
With the notation given by Equation (10), we see that Equation 9 can be simply rewritten as (0))()))) > for every neuron k. This condition, akin to orthogonality w.r.t. the bilinear form of Equation (10). implies that, under gradient flow optimization,
(0, 0) = 2k0, = -(g = vk= L,..,,..,,.
(11)
This result, first obtained in Saxe et al. 41 for linear networks and discussed Du et al. [12. Liang et al. (29], Kunin et al. (27), tells us that the rescaling symmetry results in the quantities 9% being conserved. This means that the difference between the Euclidean norm of the inputs and the outputs is constant for each neuron throughout the GF training trajectory. Moreover, under the condition of homogeneity of the activation function, Du et al. 112 proves that Equation (TT) holds even at non-differentiable points of I. and in the case of multiple layers.
Invariant sets. Assume that at the initialization dn we have (fo. 0u)) = for all k. then Equa- tion (IT) implies that the GF trajectory will lie on the set characterized by the system of equations This subset is mapped to itself under the GF dynamics by Equation
(0,0) 10,03H Cg. for A: = ......I I.
(see Fi) and constitutes the main object of our study.
Definition (Invariant set). Given e= (cI....,cc) ct). we call invariant set the subset H(c) θ given by the equations (0, 0)) Vk = I.....l.
If we look at each single equation (i.c. to each hidden neuron), we see that Equation II can be written
(w (w))) =c
(12)
which corresponds to hyperquadric (or quadric hypersurface) in O₁. We denote with ((cx) O this hypersurface and call it the imvariant hyperquadric associated to the k-th hidden neuron.
Here C₁. IR takes the role of a label associated with the k-th hidden neuron, which, we see in the next section, plays a key role in specifying the shape of Q(ex). ) Figure 2a shows how, for d 2 and 1, ((cx) is an hyperboloid with 1 sheet (connected) if 0 and 2 sheets if c₁ < O.
5

---

Topology of the invariant set
As discussed above, Equation (11 tells that gradient flow trajectories can't 't explore the whole space è but are constrained to lie on the invariant set (c). The values of c, in turn, depend on the initialization and, we see from Equation (12), quantify the balance between the norms of input and output weights in every hidden neuron.
The goal of this section is to provide topological characterization of H(c) that can tell us something about the presence or absence of fundamental obstructions to the network's 's training process. With obstruction, mean the impossibility of a GF trajectory to travel freely from point to the other in H(c). We refer the reader to Appendix for an essential overview of some of the topological concepts that we rely on in the next paragraphs.
Counting high-dimensional holes. Our topological characterization will be framed using Betti numbers. Betti numbers are well-known topological invariants given by sequence of natural numbers that intuitively encode the number higher-dimensionall holes and cavities present in space. In particular, the O-th Betti number of space X, (X( corresponds to the number of connected components of X and thus will be fundamental for our goal of identifying obstructions.
The invariant set (c) is given as the set of solutions of polynomial equations of degree 2 sharing no variables. Furthermore, in the setting of two-layer neural networks, we can leverage the fact that the parameter space can be decomposed into the parameter spaces of the hidden neurons. This, in turn, allows us to decompose the invariant set as the product of the neurons' invariant hyperquadries, greatly simplifying our study.
Lemma In a two-layer ReLU neural network, the invariant set H(c) is homeomorphic to the Cartesian product of the hidden neurons' invariant hyperquadrics, that is
H(c) = Q(ci) x... Q(ci).
(13)
Lemma | tells us that Morcover, the hyperquadrics we encounter here are well-studied objects for which the next proposition (proven in Appendix D.2) gives topological characterization.
understand the topology of H((c by studying independently its factors.
Proposition 2. If c > O, Q(ex) is topological manifold homeomorphic to R x Sd-11 Ifex < ((cx) a topological manifold homeomorphic to Rdd S Ifc, = 0, Q(0) is contractible space.
Leveraging the decomposition of Lemma I and the characterization of the factors given by Proposi- tion [ we can explicitly compute all the Betti numbers of the invariant set. We give the next result in terms of the Poincaré polynomial of Hc)., namely the polynomial whose coefficients are the Betti numbers (see Appendix [).
Theorem 1. Let L,, be the number of positive, negative, and zero components of respectively. The Poincaré polynomial of H()) is given by
PH(c) (z) (1+1411)4 (1+2--1)
(14)
This result, which is proven in Appendix D.3 contains a wealth of topological information as it gives us the exact number of holes and cavities of any order, depending on the network's hyperparameters (d, e) and initialization (4,, I.). In the rest of this work, we focus only on the 0-th Betti number as the non-comnectedness of Hc) provides clear obstruction to the GF trajectories.
Connectedness of the invariant set. With regard to the connectedness of H()), we can leverage Theorem to obtain the exact number of connected components.
Corollary 1. The 0-th Betti number 3o of H(c), corresponding to the number of its connected components, is given by
1
if d, d
24.
if 1,e>1
Bg
(15)
2
fd>1,e=1
2/,
idd=le==1
Proof. This can be directly obtained from the coefficient of degree of the Poincaré polynomial obtained through Theorem

---

Ce <
(cc)
C > 0
b
H(C))
Effective componetts
C++
C+
C_+
TaP+(0.)
W
(m3)
(w2))
Figure 2: a. The invariant hyperquadric ((cx) of a neuron with two inputs (d 2) and one output (e in the cases where eæ 0 (left) and сą > (right). b. Depiction of the invariant set H(c) in the case where I_ so that there are 2 4 connected components. C== denotes the connected component such that = (1), The blue lines separate the different effective components of H(c).
What we see in Equation (15) is that most cases, the invariant is connected, and gradient flow has no topological limitations in exploring the whole of H(c). Instead, when the hidden neurons have only input only one output, the space fragmented into several components whose number scales exponentially in 4 or I., respectively.
Let us focus the more interesting
where d
Corollary 2. If the output of a two-layer ReLU neural network is a single scalar = 1, its input has dimension d >> 1, and the initial parameter Og is such that (0o, Ooh 0 for l. 0 hidden neurons, then the H()) is disconnected and has 2 connected components.
This means that neurons initialized with the norm of their outgoing weight strictly greater than their incoming weights' norm responsible for disconnecting the space. We now precisely identify which connected component a parameter belongs to and clarify the meaning of the obstruction.
Proposition 3. Let 1, > and 0 H(c) with such that Chg:-.., Ck. 0 while e& 0 for all other k. Let (² (w ...... w(()) Iilx-_ be the vector whose components are the compo- nents of (2) l associated to c. O. Then the vector s()) = (sign( (w(2)) )...,, sign. w:())) identifies uniquely the component 0 belongs namely: O and 0' belong to the sume connected component of H(c) if and only if s()) = s( s66'.
Proposition proven in Appendix D.4 implies that s0)) does not change when we move in C on continuous curve such as the one given by gradient flow. This gives us an interesting interpretation of the topological obstruction: gradient flow cannot change the signs of the outgoing weights of the hidden neurons & such that сę < 0 (see Appendix[Gi for an intuitive explanation of the phenomenon). This same observation is also mentioned in Boursier and Flammarion [3]. Proposition 3 extends one of the results of Boursier et al. which proves that the same also holds when C₁ (balanced initialization).
By also considering Corollary I one obtains that a clever initialization of the parameters given by (66. fn > 0 4k 1,..., can prevent the issue by ensuring the connectedness of the invariant set. We also find that under common initialization schemes such as Xavier [19] and Kaiming (2]] the probability of having pathological neurons is negligible when the input dimension and number of hidden neurons is high (see Appendix
Taking symmetries into account
Corollary 2 states that neurons & such that eæ < 0 are "pathological", in the sense that they are respon- sible for disconnecting the invariant set into several components, whose number scales exponentially in the number of those neurons. This result gives grim picture of the possibility of actually
7

---

optimizing the neural network: if the initial parameter 8g is in particular connected component and the global optimum 8, lies in another, then any gradient flow trajectory will not be able reach 0, because it will be constrained in its connected component.
This result, however, provides only with partial picture of the parameter space's geometry. is priori possible that the training trajectory, moving in its connected component, reaches parameter Ç. which itself is optimal obstruction given by the non-connectedness would be only apparent.
is observationally equivalent to (C 02). In this case, the topological
To take this fact into account, define the following notion.
Definition 2 (Effective component). Let 0 Hc(c) and C(0) be its connected component therein, We define its effective component (0) as the union of the connected component of all O' such that o' T O. So that (0) := U C(0").
Figure 2 gives picture which clarifies the definition, showing a space with 4 connected components that has only 3 effective components. If the optimum 8, belongs to the same effective component as the initialization, then possible to reach a parameter that is observationally equivalent to (through permutations and rescalings).
We present useful result which tells us that the action of rescaling of Equation 3 can take any non-degenerate parameter e H((c to any other invariant set (c') for every c' e This means that any invariant set can realize all the neural network's 's functions.
Proposition 4. For every Oı E R and for every Os O₂ such that (1(1) w.()) * 0, there exists unique og E R, such that Top (0x) E 2(cx). W O and w # 0, then the same holds for every Ck < while, if ((() and W(2) & 0, it holds for every cp 0.
&
The proof can be found in Appendix D.5 with the formula of the specific which realizes the rescaling.
The following theorem leverages the power of Proposition to give necessary and sufficient conditions for and 0' to belong to the same effective component.
Let e E! and I_ be the number of neurons such that e Assume that 1. 2 1. Let C, C' y H()) be two distinct connected components of H(c) such that s(0) s, vθ e C, and s(f')= s', vo' C². Then, the following statements are equivalent:
Theorem 2. Let > 1 and
1. for every e C there exists 0' C" such that 0 T. 0';
2. Σ
The theorem, proven in Appendix [.6 tells us that, while connected components are identified by s. the effective components are identified only by the values of Σ, 8; or, equivalently, by the distribution of + s. Therefore, we find that the number of effective components scales much slower than the exponential growth of the number of connected components given by Corollary
Corollary 3. The number of effective components of ((() is given by 1 L..
Proof. Theorem tells us that two connected components C.C" belong to the same effective compo- nent if and only if their associated sign vectors s, s' (-1,1)' have the same sum. The number of effective components will thus equal the number of different values that the sum Σ s; can have. If s; = 1 Vi then Σ/ s = L. Each switch of a component to -1 decreases the sum's 'ss value by until reaches the minimum -l_.. Therefore, the total number of values of the sum will be 1 LL.
6 Empirical Validation
Task, dataset, and model setup. We display here toy example, showing how the initialization of the model can cause a topological obstruction, making the optimum unreachable.
We consider the function F(I1, 22) (xx, +2)), which will be our ground-truth. Next, we generate datasct of 8000 points (z,, F Fz+)) by sampling x; U([0, Our model, depicted in Figure 3a)
8

---

a
O(a)
<
((sa)
62 <
C
Bad instiaiiation
Goad intiillietiio
((c))
a < O
(c))
Optimizattion
The small 2-layer neural network architecture considered. The hidden neurons' parameter spaces, together with the invariant hyperquadrics associated with hidden neurons (left) and 2 (right), for an initialization with topological obstruction (top) and without it (bottom). The colored curves represent the gradient descent trajectories from initialization (0) up to t. 500 optimization steps. The loss curves for the bad (obstructed) and good initializations.
Figure 3: Visualization of the experimental setup described in Section 6
a one hidden layer neural network with 2 hidden neurons, ReLU activations and no biases. All the weights are initialized by independently sampling from U[VV v2]). From the task and the network's i architecture, it is clear that at least one of the output weights has to be negative to approximate F' correctly.
To standardize our results, we apply the rescaling of Proposition 4 and relocate the initial parameters to an observationally equivalent one in the invariant set (c)) with Gj (-0.1, controlling the sign of the weights on the last layer. We allow ourselves to do these two manipulations to control the experiments while only marginally modifying the network initialization, avoiding the introduction of massively unbalanced weights, which could change the dynamics, as shown in Neyshabur et al. [34] Finally, we train the network using gradient descent on the MSE loss with a small learning rate of h = 0.01. This limits the variations of values to less than one percent along training, giving us good approximation of gradient flow.
Results. We initialize different models and collect all states and losses. First, when we initialize the model with an "unlucky" configuration, namely e (-0.1,-0.1) (the space has 4 connected components) and s()) (+1,+1), we find that the trajectories are confined to the positive region of their invariant hyperquadrie. resulting in a poor approximation of F, as can see Figure (top) and in the loss of Figure 3e. Instead, with an initial configuration such that e= (-0.1,+0.1) connected components) and s(t) (+), the model can leverage the connectedness of ((c²) to leamn F by flipping the sign of the second neuron's 's output weight (Figure 3p bottom right).
A more realistic experiment. We present here further experiment to show how the topological obstruction can be hindrance more realistic setting. We consider a simple binary classification task on the well-known breast cancer dataset [49]. which we try to solve by fitting one-layer ReLU neural network trained to minimize the BCE loss. We vary the number of hidden neurons and, for each I, we change the number of non-pathological neurons 4 (neurons with c 0) from 0 to 1. We repeat the experiment with 100 different random initializations and show how the model's s average performance changes when the degree of disconnectedness of its invariant set is varied. The result, on the left panel of Figure 4 clearly shows the presence of "gradient" in performance, where increasing the number of non-pathologice neurons decreases the average value of the test loss after training. The right panel of Figure 4 moreover, shows how the impact of the obstruction depends on the number of non con-patholog neurons and not on their fraction over the total number of hidden neurons.

---

reroraee
seeeage
0.29 0.28
0.22
0.35
0.33
0.37 0.35
0.5
0:43
0.38
0.36
0.36 0.38
0.50
0.53
0.25
0.23
0.19
0.42
0:35
09
0:49
0.49
9.69
0.46 0.49
0.40
0.6
0.05
0.64
05
0.65
0.05
02
0.64
0.03
0.35
8
0.69
0.69
0.30
2
9
16
236
Figure 4: Left. Average test BCE loss two-layer ReLU neural network trained on the breast cancer dataset over 100 different initializations for each pair (I,4,,, 2.... . and 4, S l, of numbers of hidden neurons and non-pathological neurons. Right. the y-axis displays the percentage of non-pathological neurons.
Conclusions
In this paper, we have given analytical results that clarify the nature of the constraints imposed by gradient flow on the parameter space of two-layer neural network with homogencous activations. In the case of single scalar output, which appears in tasks such binary classification and scalar regression, we identified initial conditions that lead to a topological obstruction in the form of the parameter space's fragmentation into multiple connected components. This is caused by pathological neurons whose output weights cannot change their sign during training. Moreover, if one also considers the network's symmetries under permutations of the hidden neurons, find that most of the connected components are equivalent. The number of effective components of the resulting space scales linearly with the number of pathological neurons, contrasting with the exponential growth of the number of connected components obtained without considering the permutation symmetries.
As shown in the last numerical experiment, the lack of non- non-pathological neurons hinders learning, when the network's 's width is scaled. Our probabilistic analysis outlined Appendix [ that with common initialization schemes, the probability of creating pathological neuron decreases rapidly with increased inner layer width. Therefore, the combination of specific initialization schemes and large number of hidden neurons (beyond the minimum required to solve task) appears make this obstruction unlikely in practice. This work describes a simple safeguard to avoid obstructions, which can, for instance, discourage the usage of initialization schemes that result in the proliferation of pathological neurons.
however, shows
8 Limitations
The main limitation of the work is the network's architecture, which is limited to only one hidden layer. Considering multiple layers, we can still define rescalings and permutations and find invariant hyperquadries for each hidden neuron. The issue emerges in the fact that these hyperquadrics are not "independent" anymore, and the invariant set cannot be factored into the product of the ((c)). This intuitively results from the fact that in the multi-layer case, each weight in the hidden layers is shared by two neurons.
The second limitation that our study focuses on gradient flow optimization. This idealized situation doesn't take into account the fact that moderate step size of gradient descent and stochastic gradient descent can break the conservation of (0, (0,0% and make the parameters drift away from the invariant set 2 Moreover, popular optimizers like ADAM [25] update the parameters employing the gradients previous iterations so their trajectories will be constrained to lie on H(c) as we defined
The inclusion of regularization terms in the loss function, such as bp regularizations, also breaks the invariance to rescalings.
10

---

Acknowledgements
M.N. acknowledges the project PNRR-NGEU, which has received funding from the MUR DM 352/2022.
V.V. would like to thank the Isaac Newton Institute for Mathematical Sciences for the support and hospitality during the programme Hypergraphs: Theory and Applications when work on this paper was undertaken. This work was supported by: EPSRC Grant Number EP/V52 1929/1
This study carried out within the FAIR Future Artificial Intelligence Research and received funding from the European Union Text-GenerationEU (PIANO NAZIONALE DI RIPRESA
RESILIENZAA (PNRR) MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 D.D. 1555 11/10/2022, PE00000013), This manuscript reflects only the authors' views and opinions; neither the European Union nor the European Commission can be considered responsible for them.
References
[1] Serguei Barannikov, Daria Voronkova, Ilya Trofimov, Alexander Korotin, Grigorii Sotnikov, and Evgeny Burnaev. Topological obstructions in neural networks learning. arXiv preprint MxXi::0012.18344 2020.
[2] David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009. 2020.
[3] Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is two-edged sword. arXiv preprint arXiv:2401.1079,, 2024.
[4] Etienne Boursier, Loucas Pillaud- Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems, 35:20105-20118 2022.
[5] Maria Sofia Bucarelli, Giuseppe Alessio D' Inverno, Monica Bianchini, Franco Scarselli, and Fabrizio Silvestri. A topological description of loss surfaces based on betti numbers. arXiv preprint arXiv:2401. 038244, 2024.
[6] Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory, pages 1305-1338. PMLR, 2020.
[7] Francis H Clarke, Yuri S Ledyaev, Ronald Stern, and Peter Wolenski. Nonsmooth analysis and control theory, volume 178. Springer Science & Business Media, 2008
[8] Y Cooper. The critical locus of overparameterized neural networks. arXiv preprint arXiv:2005.02/11 2020.
[9] Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint 1804. 10200, 2018.
[10] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
[11] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learming, pages 1309-1318 88 PMLR, 2018.
[12] Simon Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
[13] Simon Eberle, Arnulf Jentzen, Adrian Riekert, and Georg S. Weiss. Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with relu activation. Electronic Research Archive, 2023.

---

[14] Yuanyuan Feng, Tingran Gao, Lei Li, Jian-Guo Liu, and Yulong Lu. Uniform- -ne--in--tiiim time weak error analysis for stochastic gradient descent algorithms via diffusion approximation. arXiv preprint arXiv: arXiv:/902. 00035, 2019.
[15] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861. 2020.
[16] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning. pages 3259-3269. PMLR, 2020.
[17] C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimiza- tion. arXiv preprint arXiv:1611.01444 2016.
[18] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
[19] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-22222222 256. JMLR Workshop and Conference Proceedings, 2010.
[20] Grzegorz Gluch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprint arXiv:2 104.02500 2021.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing duman-leved performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[22] Stefan Horoi, Jessie Huang, Bastian Rieck, Guillaume Lajoie, Guy Wolf, and Smita Krish- naswamy. Exploring the geometry and topology of neural network loss landscapes. In Interna- tional Symposium on Intelligent Duta Analysis, pages 171-184. Springer, 2022.
[23] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[24] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2019. URL hatprrmmmmddHHfffggg000
[25] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, www.sssiiannnnioiieeeeeee abs/1412.6680,, 2014. htLlatttps//ggi..seeniccccchhh
arg/CorpusID 6628106
[26] Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32, 2019.
[27] Daniel Kunin, Javier Sagastuy- Brena,, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint 20Xi:22012.04728, 2020.
[28] Shiyu Liang, Ruoyu Sun, Yixuan Li, and Rayadurgam Srikant. Understanding the loss surface of neural networks for binary classification. In International Conference on Machine Learning, pages 2835-2843. PMLR, 2018
[29] Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics, pages 888 198.896.79 PMLR, 2019.
[30] Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances Neural Information Processing Systems,
2021.
12

---

[31] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33): E7665-E7771, 2018.
[32] James R Munkres. Elements of algebraic topology. CRC press, 2018.
[33] J.R. Munkres. Topology. Featured Titles for Topology. Prentice Hall, Incorporated, 2000. ISBN 9780131816299.
[34] Behnam Neyshabur, Russ Salakhutdinov. and Nati Srebro. Path-sgd: Path normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
[35] Quynh Nguyen. On connected sublevel sets in deep learning. In International conference on machine learning. pages 4790-47999 PMLR, 2019.
[36] Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning, pages 17759-17781. PMLR, 2022.
[37] Grant Rotskoff and Eric Vanden- Sajden-iinden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 75(9):1889-195) 1935, 2022.
[38] Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International conference machine learning, pages 4433 4433-4441. PMLR, 2018.
[39] Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems, 35:32667-32679, 2022.
[40] Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706. 04454, 2017.
[41] Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:
2013.
[42] Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning,
pages 9722-9732. PMLR, 2021.
[43] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis neural networks: A law of large numbers. SIAM Journal Applied Mathematics, 80(2):725-752, 2020.
[44] Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021.
[45] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19 170):1-57, 2018.
[46] Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377-63887 2020.
[47] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint 17X1v:1706.053500 2017.
[48] Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, 20(133):1-34, 2019.
[49] William Wolberg, Olvi Mangasarian, Nick Street, and W Street. Breast cancer wisconsin (diagnostic). UCI Machine Learning Repository, 10:C5DW2B, 1995.
13

---

[50] Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. arXiv preprint
2018.
[51] Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy. Symmetries, flat minima, and the conserved quantities of gradient flow. arXiv preprint arXiv:2210.172I1 2022.
[52] Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Conference on Machine Learning, 2024.
14

---

A Parameter spaces
Let (en ,12....,................ cu) and (eI.... ei) be the canonical bases of the spaces R and R², respectively and
i₂ ((ExWW()) w(() exx) (W(), W(2) E ce.
OR notice, is the subspace of θ consisting of the weight matrices W()) with null rows except for the k-th one, and weight matrices W((2 with null columns except for the k-th one. We can check that ① θ @ ... @ ,₁ because, if is the x identity matrix, Ces I that
E(we W( E(ena ex))
Moreover, θ = 0 x x.. x b via the linear isomorphism
= (W()) w(e)) (0+))-1 ((ex&W(1), w(2) ekk)))))))))) k=1
This, we see, is equivalent to decomposing the neural network of Equation 1 into the computations of the single hidden neurons. Indeed, let f(x;0x)= == f(x, (exx (1), (()))r)) then, considering that CkeCkk eek and that (CCwv) (v) it holds that
f(z;0x) = ((2 ( wk 1....,I.
Therefore Ex (z) ((;;0).
B Primer on topology
Here, recall some basic facts about the topology required to understand the paper's 's results. A self-consistent introduction is outside this work's scope, so we refer the interested reader to more complete expositions in Munkres 33 32
Topological manifold. An r-dimensional topological manifold is topological space X which lo- cally looks like the Euclidean space IR". More formally, for each p e X, there exists a neighbourhood U of p and homeomorphism mapping U to open subset of I
Contractible space. A topological space X is contractible if it can continuously deform to point p X. This means that there exists a continuous map
F: x -
such that F(x,0) x and F(x, 1) p for every x e X.
Betti numbers. Betti numbers formalize the notion of the hole topological space and extend it to describe digger-aimmnsionall cavities. The general idea is that one can associate sequence of Abelian groups named homology groups to any space X, which encodes rich information about the higher-dimensiinal cavities in X. For what we are concerned here, the rank of the k-th homology group is called the k-th Betti number 3x (X). 3& (X) counts number k-dimensional holes in the space: (X) count the number of connected components, (X) the number of "circular" holes and (X) the number voids or cavities.
A contractible space X is connected and cannot have any holes, and thus its Betti numbers are (x( and (x) Vi> 0.
Betti numbers are topological invariants, meaning they are preserved when a space is transformed via homeomorphism, namely bijective, continuous map with continuous inverse.
Poincaré polynomials.
The Poincaré polynomial of a topological space X is the polynomial whose k-th coefficient is given by the k-th Betti number
px (x) 3o(X) (1 (2
15

---

Künneth formula. Künneth's ss theorem describes ussembling the homology groups of Cartesian product of spaces X Y from the homology groups of the factors X.Y. One of its corollaries tells us that if we care only about Betti numbers, simple relation holds between the Poincaré polynomials of Y and the ones of X and namely,
(xx px (x)))x((z)
Types of connectedness. In topology, there are several kinds of connectedness, Two of them are particularly important for this work.
1.) A topological space X is connected if it cannot be divided into two disjoint non-empty open sets. If it is not connected, the connected component of a point x X is given by the union of all connected subsets of X which contain z.
A topological space equal to the Cartesian product of two spaces X Y Z is connected if and only and are both connected.
2.) A topological space is path-connected if, for every pair of points I,y X, there exists continuous curve y [0. 1] + X such that (0) = x, y(1) = y. The path-component of is the set of all e X such that a continuous curve exists connecting x: to
This second notion is more relevant our setting, where care about the possible destinations of the optimization trajectories.
Path-connectedness implies connectedness, but not the opposite. There are situations, however, where these two notions are equivalent. For example, when X a topological manifold, X is connected and only if X is path connected.
Notice that the 0-th Betti number Bo (X) counts the number of connected components but, in general, not the number of path components. With Lemma [ we prove that these two notions are equivalent for our object of study.
C Extra propositions and lemmas
Lemma 2. The invariant set Hc)) is connected if and only if it is path connected.
Proof. Lemma tells us that H(c) 2(a)) x 2(ci).
Let us focus on a particular ((cx).
When Cj 0, Proposition 2 tells us that Q(cx) is topological manifold, and thus, it is connected if and only it is path connected.
When 0, Q(0) is not topological manifold but contractible, implying that it is connected. Let us prove that also path-connected.
Let Oz. 0% (0)) and define the curve y [0, 1] 2(0)
Na (t) = t. 0%
such that Ne.0 (0)= 0,, 72:0 (1) 0. 7a,o (t) (0) for every te [0, 1] because
(T: (t), o (t) h ()
Therefore, the segment from 0g to 0 belongs to (()).
A continuous curve from 0% 6g can be thus obtained by
1k:00 (2t)
if [0, 1 if [1,1]
Tk:er 7o:0 (1)
1h:00 (2 2t)
which is continuous because "ok: (1) Nk:00 (1)= 0. Therefore 2(0) is path connected.
Finally, if (c) is connected, then all of its factors ((cx) are connected, which, in turn, true and only if they are path-connected. A product of path-connected space is again path-connected, and therefore H(e) path-connected. The other implication true because path-connectedness implies connectedness, thus concluding the proof.
16

---

Lemma 3 (Interchange of rescalings and permutations), Let 0 6, a E R2, and π G₁, then, if á R
TaPa(()=PPiI(()))
(16)
Proof.
Given that
(0) (diag( ( R2 R, R= (( RR R_diag(a)"))
we need to prove that diag(a) R, diag(a).
(dtag(a)Rx))) Σ diag( (a) (Ra) = as
if = (1))
otherwise
Let us pick a generic à E R,.
=(0) if t1)
āāj
(Radiia()))) = Σ (Rz) (i)) (Ba =
otherwise
Let us consider the inverse permutation π
so that π - ii =. Then, if R
ā = a = aq
and thus we get that diag(a) Rz R,,iia((()) ()).
D Proofs
D.1 Proof of Lemma 1
Proposition D tells us that the invariant set can be decomposed as the direct sum of the single hidden neurons' parameter spaces. This means that, for every 0 e O, there exist unique 01 E O,...., 0 e θ such that
Proof.
i == 01 + 02
0:.
Therefore, we have a linear isomorphism :p: 0 x ... O + +
:: (0I,...,0)) - 01 + 01 =0
The invariant set a subset of O, which is given as the set solutions of equations (),0)) Ck l 1.....,.. I. Notice that each of these equations involves set of variables that appear only that particular equation. These variables are exactly the ones which belong to O In fact (0,0% = 0x) Therefore, given 01 ((ci)...., E Q(cr) we have that
( .......................... 01) Σ
O₂ H(c).
Rm1
On the opposite, given 0 Hc)) have that
x (0) = (f1....,, 01) ((c:) x... Q(ci).
Therefore, H(c) is in bijection with 2( x... Q(ci) through & which, being a linear isomorphism. implies also that H(c) and Q(c.) x... Q(c:) are homeomorphic.
D.2 Proof of Proposition
Proof. Let us consider the three cases separately. If ct > 0, Q(cx) is defined by the equation
( E
(2(2))2
3k =c = 1 [((2) Ck.
17

---

where ||-| p is the Frobenius norm of a matrix, i.e., the square root of the sum of the squares of its elements. This can be rewritten as
( (1)
|| F
V
ck
(17)
where V C + [ (2)
because C₁ O. We define the map h: 2(c+) + 8 x RR as
(
(w w(
((22)
(18)
Ck
where, notice, the first component belongs to the sphere S because of Equation (17 and W R. This map is bijective, differentiable and has the following inverse h 5d-1 R 2(k)
h- (u, 2) (V which is differentiable. Therefore, h is diffeomorphism from 2(cx) to S R".
c* (V
Jull u.
Ck < O, we write the equation of 2(cx) as
w (2)
( 2
where (( and W(()) have switched their role to guarantee the term on the right to be positive. The diffeomorphism is now built analogously to Equation (18) as a map h 2(cx) ge-1 x Rdd
If cæ = O, we prove that 2(0) is contractible space. To do that, we exhibit a homotopy equivalence between ((0) and the point i.e. a continuous map p: [0, l]x 2(0) 2(0) such that p(0, 0₂) Oz and p( 1, 02) = 0 VO Q(0). The map is defined in the following way:
p(A,0b) 0x) (1-A)0k. A)0x.
This is continuous and well-defined because
Kp(A, 0x), p(A, 0x)))= = (((1 A)0,, (1 2)081x = (1 (0x, 0x39 =0, meaning that p(,, 02) (()) for every 02 @(0) and for every A E [0, 1J.
D.3 Proof of Theorem
Proof. An implication of the Künneth formula is that the Poincaré polynomial of the Cartesian product two spaces equal to the product of their Poincaré polynomials:
Pxxy (x) Px (x)py (3).
Starting from Proposition 2 we can apply this result to ((ck).
PR- (x (x)ps= (x)
c₁ >
Po(a.) = Pad (z) (x)
if c < 0
(19)
if c =
because a contractible space has connected component and all its other Betti numbers equal to
Moreover, we know that RR" is contractible for any n and its Poincaré polynomial is pro (x) 1. The Poincaré polynomial of the sphere Sth is given by psn (x) 1 x".
Equation (19 becomes
+ x-1
if c > if c₁ < 0. if C+ =
Po(a)) =
1 1
(20)
Given that Lemma 1 tells us that H(c) can be factored into the product of the ((cx), apply Künneth formula and find that
PH(c) Pe(c.) (z) = (1 2 11)
(21)
18

---

D.4 Proof of Proposition 3
Proof. In the following, we exploit Lemma 2 and use the terminology connected and path-connected interchangeably.
Let us first prove that s(0) s(t') means that 0 and #' belong to the same connected component. We do this by explicitly building a continuous curve 5: [0. 1] H(c) such that 6(0) 0 and 6(1) = 0'.
Let us proceed by leveraging the homeomorphism from H()) and (cc) x... x ((ci) and consider the different components of ó in the invariant hyperquadrics associated to each neuron d
If Cy 2 O, we know that ((cx) is path-connected and therefore we fix (t) to any continuous curve in ((ck.) such that (0) 0g and (1) 0z
C < for E K 1/1.....A. 1L) E ( .l......]................. we define the curve "v. : [0, + 2(cx.) with 1) ( s...... ...., (1- ())
Rel s(0). Ch. + +11 (1-t) W (1-))3
(1)
for every i= I.....L.
76. is continuous curve which connects the point 74:0 (0) Os, with 7:.8 (1) (0) we define (1:0(1) == Neo (1 t), which the same curve as Ve: traversed in the opposite direction, we can define the curve
SR. (t) Yuo
i.c. the curve which travels on Y:o0 for E [0, 2 and on Tear for te [ 1]. for i= I.......
Notice now that òx, (0) = On, and 6R, (1) = 0% Moreover ōR, is continuous because
T (1) s00):
(0,)
under the hypothesis that (0) = s(0".
Finally, we found continuous curve 5 ( 51) such that 5(t) H(c) Vt [0, and 5(0)= = 0, 8(1) = 8', Therefore, and belong to the same connected component.
Let us now prove that if 0 and #' belong to the same connected component, then s()) s(0')
Let 7 [0, 1] H(C) be continuous curve in Hc() such that 7(0) and 711 0'
For each (k1..... ,k, } such that Gj O, we know that 7x (1) е Q((+) means that
(x(1)= ( 7k1 (t).....
(1)
(1) Rnd (t), su st((t) Ck
7/ 10)2
(1) for some function (t) (-1,1} such that sx(0) <()) and (1) x
Assume, by contradiction, that k(0) -s((') )x- Assume also that s(f); +1 and s(f') -1.
Notice that e <0 implies that p(t) -ck | 0.
(2) st)))(tt)) then, is a continuous function such that 7 (²)()) > 0 and 7/ (1) The function 7 and thus, by the intermediate value theorem, there exists 6, (0, such that 7/2
(2)
(2) (f..) 0.
But Tx (1) # for every as 8 (1) 1-1,1) and - Ck + | ( 1F >
Repeating the argument for s(0) ; -1 then prove by contradiction that s(0) s(0')a for all K, thus concluding the proof.
D.5 Proof of Proposition
Proof. We have by Equation
10 and Equation
(.)
(O), (0)) )x = - o
(1)
d
7 Σ (w
c
2- Cx ===0
1
a j=1
19

---

By renaming A
d Σ
(
[ pr
C= Σ ) jol
=|
and multiplying by az. >
jh
we have: i-1
Aa cgak -C=0
(22)
2 Solving for og gives us:
A CK 4AC 2 4AC'> 0
VA
0
2A
Given that we want or > O, we discard the negative solution. The other is positive because and thus v A |cx/-
oz
Ck V
2A
Of which we keep the positive solution only, with its full expression being:
1 1
(2)
Ck + c
Og
(23)
| (
Hence, if a (4) with ag given by Equation (23 we get that (Ta()).T((0))+ c mk= I....,
Let us consider the pathological cases W w =0.
((())) 0, (x & * 0 then A 4=0,Co 0,C 0. Therefore, we have that Equation 22 becomes
-cxa =
which has solutions if and only C < In that case 0g neuron with zero input weights and nonzero output weights can be rescaled only to the invariant hyperquadrics with 6
This means that a hidden
If w = 0, ((() * 0 then C = 0,A 0. Therefore, we have that Equation
22 becomes
a
Ck
which has solutions if and only if C₁ 0. In that case ag
w This means that hidden
neuron with zero output weights and nonzero input weights can be rescaled only to the invariant hyperquadrics with C
((() = and w-(2) 0, then 0x = 2(0) and it cannot be rescaled to any other invariant hyperquadric. &
D.6 Proof of Theorem
Proof. Let us first prove that, if for every 0 e C there exists a 0' C" such that 0 TP 0' then
Σ 50)) s(0):-
F=
First, Lemma tells us that can interchange rescaling and permutation if we permute the rescaling factors accordingly. This means we can reduce any composite action of rescalings and permutations to the action of single rescaling and a single permutation.
20

---

Let f é C and 6' & C" such that 0 8. Then there exist E IR and e Gy such that
TaP H'
(24)
This means that (0) and 6' belong to the same invariant and, specifically, to the same connected component. Therefore,
s (Ta,Pa0)) =00))
Notice that
= (T)
because Ta does not change the sign of (2 it acts by scaling it by positive factors. Let focus on
s (0) sign sign((1(2) R²))))
If the neurons of f such that C 0 are indexed by ,ı, ....... kL_ we will have that the neurons of Px (0) such that Cg O are indexed by x (ki), (122) ... (k..) Therefore
(O)); sign( (2) =(k:) (s(0)
for some permutation т_ E G Therefore, Equation 24 implies that
s(0') = =)p: s RZ.
The action of rescaling and permutation only reshufile the label of the connected component. This means that
((s(O)R := E s()) =
L Es((), Σ s(0)).
s(0) = s(0)) = El
1=1
Let us prove the other implication. Let s, s' e RR- such that Σ
8
[-1
i-1
Given that their sum is equal, and s' have the same number of and and thus there exists permutation G such that s' = sR
Let ( be the permutation which permutes the neurons such that c according to π_ and leaves the others fixed. In this way s (0)) = sR s'.
Px however, doesn't belong to H() but to another invariant set given by H(R,,c).
Applying Proposition 4 can find rescaling (<<)) R such that T. (=) H(c))
Since neurons such that Cą 2 0, are left unchanged by the permutation, we can apply the proposition and we rescale them with or. 1. The permuted neurons are the ones such that C₁. < O, namely the ones whose weights satisfy [w(3)]
[w O, meaning that w²(2) O.
As noted above, the action of the rescaling doesn't change the sign vector, and thus
s(T,P,(O) (0) =s!
If we name 6' : (x() this result means that we found 6' T 0 such that o' e C", thus concluding the proof.
E Including biases
Let us consider the case where we include biases. The resulting two-layer neural network can be written
f(; (z ((2) o (W +(()) b(2)
(25)
where ((1) l and ((2) R
To work with this extended set of parameters, we re-define the space
e (0= (W()) (2)) ((2))}
21

---

and the single hidden neuron spaces
(1) b (0x (e beabbe W }
where the second bias term ((2) does not appear because it is not directly involved with the computa- tions of the hidden neurons. This means that we can write
+ O x ... x O x RR
where II is included to describe the parameters in 6(2),
The neuron rescaling action now acts on the biases ((2) as well as the weights:
T:R. O O₂ (a, (x) ((,)) = (aW .bb - (
(26)
(1)
and can be extended to the whole space of parameters
R x
+
(27)
(0,0) " T.(0) (diag(a)) ,diag(a)b (iiii)/aW)dddi((((bb
(C diag((a)) ((00))
Once again, find that T,0 0.
In this more general case, we can rewrite the bilinear form include the biases. 0 (W()) 600, wr(2, b(2) and 17 = (V(1,, p(1, ((2,, (2)), we define
C (0, n> Σ w(Wv( (1) Σ ki + 2
(28)
jk JR
and see that, once gradient flow optimization, we have a conservation condition like the one of Equation
(0(t). - I....,
vt>
Once again, we call ((cx.) the hypersurface of O₂ which satisfies the equation (0,0); = c and H(cx) the set in è defined by (0,0)) c Vk 1,...,l.
With this in mind, it is not hard to extend the results of Proposition and Theorem Which turn out to be slightly modified.
Proposition 5. If ck 0, Q(ck) is topological manifold homeomorphic to R x sd. If ck O. ((cc) is lopological manifold homeomorphic to Rd Se-1, Cz 0, 2(0) is contractible.
In this case, we can factor the space of parameters H()) as
H(c) Q(c.) x
where the last factor is due to the freedom in choosing the values of ((2).
Proposition 6. Let C& + 0 Vk = l,..., Let 1,,L_, la be the number of positive, negative and zero elements of respectively. The Poincuré polynomial of H()) is given by
PH(c) (z) (1))=((1+))))))) =1+ (1+2 11
(29)
Corollary 4. The 0-th Betti number Bo(c) of H((), corresponding to the number of its connected components, given by
ife>1
Bo(c) 2/- =
(30)
The result we obtain is similar to Corollary althouugh slightly modified by the fact that having single input neuron does not cause H((c to become disconnected anymore. In the case of 1, therefore, the picture presented in the main text is left unchanged.
22

---

Probability of H(c) being disconnected
Xavier normal initialization
Kaiming normal initialization
1.0
1.0
7
7
0.8
0.8
13
13
19
0.6
19
0.6
25
25
0.4
0.4
31
31
0.2
0.2
0.0
0.0
13
19
25
31
37
43
49
13
19
25
31
37
43
49
Figure 5: Probability of the topological obstruction function of the number of input d and hidden neurons, when the initial weights are sampled with Xavier normal (left) and Kaiming normal (right) initialization schemes.
Probability of obstruction
Let us consider the following question: what is the probability having disconnected invariant given a realistic initialization.?
Consider one-layer ReLU neural network with and assumee that the weights sampled inde-
pendently of one another from a normal distribution Wx() (N yk, i, w( ~ (0, o2) vk,j. will be disconnected if and only if there exists
From Corollary 2 we know that the invariant set H((c) ki
hidden neuron satisfying Σ (w(())2 < (Wr(2)2 Given independence of the initial weight
sampling, this probability can be computed as
P[obstruction]
PI (W i-1
where F' the cumulative distribution function the Fisher-Snedecor distribution.
Having obtained this general expression, we can specify it to two common initialization schemes.
We obtain Kaiming initialization 21 with of 2/d, o3 2/1 resulting in P|obstruction]:
Fi,((1)..
We obtain Xavier normal initialization 19 with of 2/ 1), 2/(1++++ +1) resulting in [foottuuutioo]] Fi,a( deld
d=l
We plot these two expressions in Figure We how, for large values of d, the probability of obstruction quickly falls to 0 for any number of hidden neurons. Instead, we see an opposite trend for small values of d: the probability of disconnectedness grows with 1. Moreover, it is interesting to notice that the region of high obstruction probability is much larger for Xavier initialization than for Kaiming initialization, further showing why the latter is preferred when working with ReLU networks.
G Intuition on the occurrence of obstruction
We can give some intuition on why there no obstruction for multiple outputs. First, we consider single hidden neuron k, with d incoming weights and a single output = 1. If the neuron is pathological, we have that
Σ (w (2
Since the weights w:(0) (t) continuous curves in time, for w2(2) to change sign, its value needs to pass through but, under the condition above, this cannot happen its square is always positive.
23

---

Consider now multiple outputs > 1, resulting in the conservation condition being
2 ( ki
Σ 3-1
(
jk
Now, any component W W: can change sign by passing through 0 because the other components can compensate for it by increasing their magnitude to keep the condition satisfied.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
All the claims in the abstract are supported by analytical results and numerical experiments.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of analysis in Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

All the assumptions of the theoretical results are clearly specified and the complete proofs are written Appendix D
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The "Task, dataset and model setup." paragraph of Section clearly explains our experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Given the simplicity our setup, we think that releasing the code would be unnecessary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccpppllciggudeeeeeeeeeeeee Cooee//dddeeebmimcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experiment's details are explained in Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our simple experiment has the goal of displaying the meaning the analytical results and, thus, think it doesn't 't require error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiment was performed on a simple laptop.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We reviewd the NeurIPS Code of Ethics and confirm that we conform to it in every repect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Being purely theoretical work, we believe that it doesn't 't havesignificantt social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We release no data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: We use no existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We release
assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "Cf5IgRASDY", "reviewer": "Reviewer_JxYP", "summary": "Summary of the paper:\n\n* The paper undertakes a theoretical 'reachability' analysis for neural\n  networks trained with gradient flow, identifying 'obstructions' separating\n  certain regions of parameter space from each other.\n* The setting is one-hidden-layer networks with homogeneous activation\n  functions, including ReLUs. The main paper considers the biasless case but\n  the authors extend their results to the case with biases in the appendices,\n  with only minor modifications to the results.\n* The authors consider training such networks with gradient flow and identify\n  that regardless of the (output-dependent) loss function that gradient flow\n  conserves a certain norm of the weights confining the training trajectory\n  to a subset of the parameter space they call the 'invariant set' which is a\n  product of hyperquadrics associated with each hidden unit in the network.\n* Through topological analysis the authors derive the Poincaré polynomial of\n  such invariant sets and in turn their zeroth Betti number, in effect\n  counting the number of connected components in the invariant set.\n  * These components are of interest because if gradient flow is initialised\n    in one component it is impossible for the training trajectory to find its\n    way to another, and training is limited to the best solution available in\n    the initial component.\n  * In this setting the zeroth Betti number (the number of connected\n    components) is only one for networks with multidimensional inputs and\n    outputs, indicating no topological obstructions.\n  * In contrast, for networks with scalar inputs or outputs, it is\n    exponential in the number of hidden units (with a base depending on the\n    network initialisation), indicating many topological obstructions.\n  * (For networks with biases, the space is connected if and only if the\n    network has a multidimensional output).\n* The authors then revise their count to take rescaling and permutation\n  symmetries into consideration.\n  * The reasoning is that if gradient flow can only explore its initial\n    connected component, this is only a problem when the component is\n    actually missing network functions rather than when it is merely missing\n    some (but not all) implementations of those functions. Perhaps only one\n    reachable implementation of the target function is needed.\n  * The result is that there are fewer 'effectively' connected components,\n    linear rather than exponential in the number of hidden units (with\n    constants again depending on the initialisation).\n  * However, the point is that there are still some topological obstructions\n    to worry about.\n* The authors provide a clear 'proof of concept' of their work in the form of\n  a numerical experiment simulating gradient flow by learning a small neural\n  network with gradient descent using a small learning rate. The results\n  clearly show that the learning trajectory is confined to the invariant set\n  and obstructed from reaching the global minimum as predicted.\n\nI wrote a bit of a long review, so here is also a summary of my review:\n\n* The main contribution, demonstrating the existence of topological\n  obstructions to gradient flow, appears novel, interesting, and an important\n  contribution to our understanding of the structure of parameter space.\n* I also think the paper is exceptionally well written and clear and the\n  analysis is elegant. I think the authors have discussed the limitations of\n  the work well. I am left with only a small number of questions.\n* My main concern is that the results show topological obstructions only\n  occur under some strong assumptions, and in fact they do not arise in\n  more situations. I find this somewhat at odds with the framing of the\n  contribution.\n\nOverall, I think the work is worthy of acceptance because it is elegant,\ninteresting, informative, and thought provoking, but I maintain concerns\nabout whether the framing is accurate.", "questions": "1. The authors have presented a clear interpretation of the source of the\n   topological obstruction in terms of the relationship between each\n   unit's weights (lines 232 onwards). Do the authors have a similarly\n   enlightening interpretation of the breakdown of the topological\n   obstructions in the case with multidimensional outputs?\n\n2. Another assumption of the work is that the loss function depends only on\n   the network weights via the output of the network. Am I correct in\n   understanding that if the loss function includes a regularisation term\n   that depends on, for example, an $L_p$ norm of the weights, then this\n   assumption would be violated and the analysis would not hold because\n   gradient flow would no longer always be perpendicular to the invariant\n   sets? This seems fine but might be worth acknowledging.\n\n3. Appendix E extends the analysis to networks including biases, but stops\n   after deriving the zeroth Betti number in this setting. Could you please\n   clarify whether the analysis of section 5 is unchanged in this setting?\n\n4. The contribution list does not mention the 'proof of concept' numerical\n   experiments. It seems to me that these experiments are a reasonably\n   important part of the paper's contribution and its evaluation to warrant\n   mentioning in the introduction (as well as the abstract).\n\n5. Line 118: The authors describe the transformations as sending parameters\n   to different but observationally equivalent parameters. It is typically\n   true that the parameter produced is different but for some parameters or\n   for some transformations the transformations don't change the parameter.", "limitations": "The paper accurately describes the contribution including its limitations.\nThese limitations are adequately and carefully disclosed in multiple places\nin the main text and discussed in detail in appendix F.\n\n1. One main limitation is the simplicity of the architecture, with only one\n   hidden layer. The authors discuss barriers to extending the work to the\n   multi-layer case, namely the interaction between layers, which is left to\n   future work.\n2. A second main limitation is the fundamental assumption of gradient flow.\n   The authors explain that the conservation laws that are crucial to their\n   analysis do not hold for discretised gradient descent.\n3. A third main limitation is that the topological obstructions only exist in\n   the case of networks with a scalar output. This is clearly acknowledged in\n   the abstract and throughout the paper, but I think the authors could have\n   more thoroughly discussed the implications of this restriction, namely\n   that in many cases we should not expect topological constructions at all.\n\nIf the work is to be accepted I would encourage the authors to include the\nvaluable discussion in appendix F in the main text, and to extend this\ndiscussion along the lines of (3).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I thank the authors for submitting their elegant and well-presented theoretical\nanalysis exploring an important topic.\n\n* Topological analysis of gradient flow is an interesting and apparently\n  novel approach that stands to make a meaningful contribution to the field's\n  theoretical understanding of the 'global' structure of parameter space.\n\n* The work clearly exhibits 'topological obstructions' in a restricted\n  setting, which is a thought-provoking phenomenon (though I have some\n  reservations about its relevance in practical settings, see below).\n\n* I appreciate the elegant and rigorous application of powerful, established\n  tools from the field of topology. The theoretical results are also\n  supported with a clear, small-scale proof-of-concept numerical experiment.\n\n* The presentation of the theory and experiments is very clear and\n  accessible. The figures are carefully designed and informative. The\n  technical details of the framework and results are complete and made the\n  paper quite accessible to me, even though I have limited knowledge of\n  topology (the primer in the appendix was appreciated).\n\n* Clear, thorough acknowledgement of related and prior work in the dedicated\n  section and throughout the paper.", "weaknesses": "The authors have convincingly demonstrated the existence of topological\nobstructions under the stated assumptions. My main concern about this work is\nthe relevance of topological obstructions to deep learning practice.\n\n1. I am not so concerned about the restriction to single-hidden-layer\n   networks. The results may not generalise to the multi-layer case, or\n   perhaps they will, it seems worth looking into with future work. However,\n   I am concerned that *even in the single-hidden-layer setting* the main\n   disconnectedness result holds only for networks with scalar output. The\n   authors clearly show that networks with multidimensional outputs do not\n   show obstructions.\n\n    With this in mind, do the authors believe that the topological\n    obstructions they have identified are relevant to deep learning practice?\n    If so, can they clarify this point?\n\n2. Furthermore, I have concerns about the gap between gradient flow and\n   discrete gradient descent training. The authors explain that discretised\n   gradient methods are not confined to the same invariant sets as gradient\n   flow. It seems possible that this means discrete gradient methods count\n   circumvent topological obstructions, even if they exist in practical\n   settings.\n\n    1. Do the authors conjecture that the identified obstructions have\n       implications for gradient descent training?\n    \n    2. If the authors would support such an 'obstruction hypothesis', I would\n       invite them to consider explicitly stating the hypothesis in their\n       introduction and/or conclusion.\n    \n    3. The authors could also consider extending their simple numerical\n       experiments to explore whether and to what extent the demonstrated\n       obstruction affects training with larger learning rates or\n       momentum-based training methods.\n\nPending the authors' clarification of these points, the framing of the work\nis somewhat confusing to me. It appears to me that the authors have shown\nthat topological obstructions to gradient flow are actually quite a\nrestricted phenomenon. An alternative framing for the paper's results would\nbe to say that they show that deep learning is often *not* held back by\ntopological obstructions. Could the authors comment on this alternative\nframing?\n\nI don't think this concern is fatal to the paper. The topological analysis\napproach and most of the results are still informative and worth sharing even\nif the topological obstructions themselves end up not arising in more\npractical settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Colleague, thank you very much for your useful and insightful review.\n\n---\n\n**Weaknesses**\n\n**W1.**\n\nAt the moment it is hard to foresee if and how this can be relevant to deep learning practice as the results need to be extended to more general settings.\nProvided the same holds also for multilayer networks, then we could see this obstruction being relevant as MLPs are often employed as building blocks of commonly used architectures like CNNs.\n\nFocusing on the single-layer case, we think that the impact of obstruction is also related to its width and the particular task at hand: if the NN is large and the task is not hard, it will probably be able to solve it without employing its pathological neurons, provided they are not too many.\n\nAssuming, however, that the network is not too large, there are specific settings where obstructions could be a hindrance.\nAs you highlight, the case of scalar outputs is a limited scenario that is nonetheless relevant in tasks like binary classifications.\nAs you can see from part 2 of the general rebuttal (Fig. 2 in the extra pdf file), we observe that, by training with GD, increasing the number of pathological neurons at initialization reduces the average classification performance.\n\n**W2.1**\n\nThe precise relation with finite-step size GD is an interesting direction that we will certainly explore in the future. From the simple preliminary experiments we performed, we tend to see that, even when the trajectories drift away from the invariant set, such drift is usually not able to push them over the obstruction. The experiment described in part 3 of the general rebuttal shows, however, that this behavior can occur for some values of the learning rate, which are not too low nor too high. Our conjecture is that, in general, this is more likely to happen when the initial value of $c_k$ is negative but small so that the connected components are close to one another.\n\n**W2.2**\n\nGiven that all of our analysis is based on the loss functions' invariance w.r.t. to rescaling and the parameters evolving through GF, we think that the addition of momentum would break the setup and result in trajectories that are not constrained on $\\mathcal{H}(c)$. \n\nRegarding the effects of higher learning rates, please refer to part 3 of the general rebuttal, where we show a simple experiment in which the trajectories circumvent the obstruction. \n\n**W2.3.**\n\nWe agree that, from this first set of results, it seems that the topological obstruction induced by symmetries is a restricted phenomenon. However, the fact that there is the possibility of a fundamental obstruction to training is not something that is commonly known or expected by the community. Therefore, we believe that the framing we gave to the paper is justified, as it serves to highlight this possibility. Moreover, while our analysis clearly identifies obstructions in some cases, it is, in principle, hard to exclude that extending the results to more general settings will not uncover other obstructions of a similar kind. Stating that \"deep learning is often not held back by topological obstructions\" would, therefore, be a bit optimistic at this point, even if the probabilistic analysis of part 1 of the general rebuttal seems to corroborate this conjecture for two-layer networks.\n\n---\n\n**Questions.**\n\n**Q1.**\n\nYes, we can give some intuition on why there is no obstruction for multiple outputs. Let us first consider a single hidden neuron, with incoming weights $w^{(1)}\\in\\mathbb{R}^d$ and one single output $w^{(2)}\\in\\mathbb{R}^d$. If the neuron is pathological, we have that $||w^{(1)}||^2_2 - (w^{(2)})^2 = c$ with $c<0$. Since the weights $w(t)$ are continuous, for $w^{(2)}$ to change sign, its value needs to pass through 0 but, under the condition above, this cannot happen as $(w^{(2)})^2 = ||w^{(1)}||^2_2 - c >0 \\implies w^{(2)}\\neq 0$. Consider now multiple outputs $w^{(2)}\\in\\mathbb{R}^e$, resulting in the condition being  $||w^{(1)}||^2_2 - ||w^{(2)}||^2_2 = c$. Now, any component of $w^{(2)}$ can change sign by passing through 0 because the other components can compensate for it by increasing their magnitude to keep the condition satisfied.\n\n**Q2.**\n\nYes, this is correct. If we add $L_p$ regularization, then the loss function is no longer invariant to the action of the rescaling transformation, and thus, the learning trajectory is, in general, not constrained anymore to lie on $\\mathcal{H}(c)$.\n\n**Q3.**\n\nYes, all the results shown in Section 5 also hold when biases are taken into account. This comes from the fact that, from the formal point of view, biases are the same as the weights of $W^{(1)}$. We can thus treat a two-layer NN with biases as a biasless two-layer NN with $d+1$ instead of $d$ inputs. If we extend the permutation action of Eq. (4) to also permute biases, the results will be left practically unchanged except for the first hypothesis of Theorem 2, which will now become $d\\geq 1$ instead of $d>1$.\n\n**Q4.**\n\nThank you for the comment. If accepted, we will add it to the final version of the paper.\n\n**Q5.**\n\nThank you for the observation. We consider rephrasing that sentence to make it more precise, e.g., \"We describe two kinds of transformations whose orbits are made of observationally equivalent parameters only.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OPcV6ZFuO8", "reviewer": "Reviewer_n2cB", "summary": "This paper studies the topological obstruction in the loss landscape of two-layer ReLU neural networks under gradient flow. Using conserved quantities in gradient flow, the authors define invariant sets, to which gradient flow trajectories are constrained. They then show that when the input or output of the network is a scalar, the invariant set may have more than one connected components depending on initialization. This leads to a topological obstruction, as gradient flow cannot reach a global minimum if parameters are initialized in the component that does not contain one. After taking scaling and permutation symmetry into account, the number of effective components scales linearly with the number of pathological neurons, which are neurons that cannot change the sign of their output weight. On a toy example with two hidden neurons, the authors fully characterize the invariant sets and show theoretically and empirically that gradient flows that do not start in the same effective component as the global minimum cannot reach the global minimum.", "questions": "- For ReLU networks with one hidden layer, does every invariant set contain a global minimum? If not, then even if an invariant set is connected, it is not guaranteed that all gradient flows starting from points on this set can reach a global minimum. This is not the type of obstruction considered in the paper, but I am curious about whether there are other considerations when choosing the initialization, besides ensuring the connectedness of the invariant set.\n- Assume that an invariant set contains a global minimum. Do all gradient flows in this invariant set converge to a global minimum?\n- For a wider two-layer ReLU network, using common initialization methods (such as Xavier initialization), what is the probability that the invariant set has more than one effective components? What is the probability that the initialization falls in the component that does not contain a global minimum? Is it nonzero?\n- Do the rest of Betti numbers, other than $\\beta_0$, provide useful information about the loss landscape and the training process?\n\n**Minor issues / suggestions**\n- Line 135-136: While allowing a finite sequence of actions is arguably more intuitive, it might suffice to just define observationally equivalence as being related by a composition of one $T$ and one $P$. As a consequence of Lemma 3 in the appendix, for any finite sequence of T and P, there exists a sequence of T and P with length at most 2 that produces the equivalent transformation.\n- In Equation 11, is the second expression missing a factor of 2?\n- The sentence in line 483 is not grammatically correct.", "limitations": "The authors are upfront about the limitations and include a limitation section in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper takes a novel approach of studying the topology of loss landscapes by examining invariant sets. While many work studies the loss landscape by proving topological properties of sub-level sets, this paper pursues an orthogonal direction by analyzing the topological properties of the set of reachable parameters by training trajectories. This approach combines the study of loss landscape and optimization dynamics, which leads to the original discovery of topological obstructions to the training of ReLU networks.\n- The paper provides concrete examples where gradient flows from certain initializations cannot converge to a global minimum and gives practical suggestions on how to choose initializations to avoid this issue, at least for two-layer ReLU networks. \n- The results on when invariant sets can be disconnected and the number of effective components provide valuable information on the loss landscape. These results could potentially lead to a new line of work on the trainability of various architectures under common optimization algorithms.\n- The paper is very well written. Setups and mathematical concepts are explained in an accessible way without losing rigor. The theorems and their implications are explained clearly. The visualizations (Figure 1-3) are informative and provide clear intuitions. The toy example in Section 6 is simple and effectively demonstrates a case where topological obstruction prevents gradient flow from reaching the global minimum.", "weaknesses": "The scope of the analysis, which is restricted to two layer ReLU neural network and gradient flow, is rather limited.\n- While ReLU is a popular activation function, two-layer networks are rare in today’s machine learning models. The results in the paper do not readily extend to deeper networks, where invariant sets no longer factorize easily into product spaces.\n- Since SGD is known to have an implicit bias characterized by a drift of $c$, training trajectories in practical settings (with non-infinitesimal learning rate) will not stay in the same invariant set. Even if they stay close to an invariant set throughout training, the noise in each step may take the parameters to a different component. \n\nIf extending the analysis beyond this setup is challenging, additional experiments on different settings could also help illustrate that the topological obstruction is a realistic hindrance in practice. For example, for deeper ReLU networks, it might be possible to investigate empirically whether there exist initializations that lead to similar obstructions as observed in Figure 3c.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Colleague, thank you very much for your useful and insightful review.\n___\n**Weaknesses.**\n\n**W1.**\n\nIt is true that our results are limited to one-layer neural networks but we believe that, even if they cannot be generalized to the multi-layer case, they could be interesting when studying architectures which employ one-layer NNs as building blocks (like CNNs).\n\n**W2.**\n\nIndeed, both SGD and GD with finite learning rate will drift away from the invariant set and, if $c_k$ is small enough, we find that this drift can push the trajectory to circumvent the obstruction (see part 3 of the general rebuttal). We stress, however, that from the point of view of our analysis, there are no differences between continuous time GD and continuous time SGD. This comes from the fact that the latter can be seen as a GF with respect to a different loss which depends only on a subset of the dataset (a mini batch) and thus is also invariant to the same rescaling action.\n\n___\n**Questions.**\n\n**Q1.** \n\nThank you for the interesting question. The answer is to be found in Proposition 4. What it tells us is that, for every non-degenerate parameter $\\theta$, there is a rescaling that takes $\\theta$ to an observationally equivalent one. This means that every invariant set contains copies of all (non-degenerate) parameters and, therefore, the global minimum too.\n\n**Q2.**\n\nUnfortunately, we cannot answer this question a priori as the actual GF trajectory will depend on the specifics of the loss function, i.e., its functional form and the dataset used. By staying completely general, we cannot exclude that the training will get stuck in a local minimum, even if the initialization is in the right component. What we can say is that if $\\mathcal{H}(c)$ is not connected, then, surely, a global optimum in another component cannot be reached.\n\n**Q3.**\n\nFor the same reasons described in the reply above, we cannot, staying completely general, quantify whether a global minimum will be reached. We can, however, compute some statistics about the topological properties of the invariant set under common initialization schemes. Please see part 1 of the general rebuttal for a thorough answer about the probability of connectedness.\n\n**Q4.**\n\nThis is a very interesting question for which we do not have a clear answer yet. While it is clear how connectedness can affect the learning trajectory, it is not clear if and how higher-order holes can affect it.\n\n**Minor issues/suggestions.**\n\nThank you for pointing out these mistakes, which we will amend in the camera-ready proof, hopefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XQTSk750bF", "reviewer": "Reviewer_hKtn", "summary": "In this paper, authors analyze the performance of gradient-descent optimization over a two-layer neural network. Authors reveal the presence of obstructions in the loss landscape and explore their topology. Finally, they identify the cause of those optimization obstructions in the so-called `pathological’ neurons—neurons that can’t change the sign of their output weight.", "questions": "See Weaknesses section.", "limitations": "Yes, the limitations are addressed. This is a purely theoretic work, dealing with a somewhat simplified example of a two layer fully-connected neural network.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) An interesting and novel approach.\n\n2) A step up from well studied gradient descent optimization of one hidden-layer networks.\n\n3) Solid theoretical background and strict mathematical proofs of main results.", "weaknesses": "1) This paper identifies pathological neurons that disrupt the performance of gradient descent over ReLU activations; however, nothing is said about how such obstructions can be avoided or how their effect can be mitigated.\n\n2) The scope of the research is limited to neural networks with two hidden layers.\n\n3) The possible benefits for further research and practical applications of the achieved results are somewhat vague.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Colleague, thank you very much for your useful and insightful review.\n___\n**Weaknesses.**\n\n**W1.** \n\nAs we mention in lines  245 and 246, one can easily control the initialization to have $c_k\\geq 0$ for every hidden neuron by employing, for example, Proposition 4. Once that holds we have that the invariant set is connected and the topological obstruction is avoided. Besides, please take a look at part 1 of the general rebuttal, where we address the probability of having obstructions under some of the more common initialization schemes.\n\n**W2.**\n\nWe agree that being restricted to two-layer neural networks is the main limitation of our work. We are currently working on extending the results to multiple layers, and we are obtaining encouraging results. However, we believe that the two-layer case provides an elegant and important proof of concept on the possibility of studying topological obstructions to learning. Moreover, while most architectures are not two-layer neural networks, most of them contain two-layer networks as building blocks, and therefore, we can conjecture that obstructions like the ones studied in our work can occur even in that case.\n\n**W3.**\n\nWhile ours is a theoretical work, we believe that the main practical implication is that, when training neural networks with scalar output, one should check the values of $c_k$ for each hidden neuron and, if possible, choosing an initialization that ensures $c_k>0$. In that way, in fact, we can be sure that no topological obstruction to learning of this kind can occur.  \n\nThe future research directions, in our opinion, are many and interesting. First, the extension to the multilayer case is the most natural one and can also provide insights into other architectures, like CNNs, which can be seen as subnetworks of MLPs. Second, the literature provides us with other symmetries, like the scaling symmetry given by batch-norm and the translation symmetry given by adding a softmax before the loss. These symmetries induce constraints which can be studied with our framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tmzYJevmQM", "reviewer": "Reviewer_9QVd", "summary": "This paper considers the landscape topology of one hidden layer networks with non-negative homogeneous activations. The main result of this work is that in some cases, the loss landscape may consist of disconnected components that cannot be traversed by gradient flow dynamics. This leads to an obstruction, in the sense that certain initial conditions will result in closed paths away from the global minimum. They further count the number of effective components (in the sense of being equivalent under the symmetries of the problem) and show that this number grows linearly with the number of hidden neurons that cannot change their sign during the dynamics.\nThe authors provide a full analytical treatment of the problem, supplemented by a simple toy example.", "questions": "Comments and questions below:\n\n1) L52: “studied by under various…” incoherent sentence.\n\n2) L51: “activation” should be “activations” I assume.\n\n3) In Sec. 3.3, the authors consider an arbitrary dataset and a general empirical loss function. The analysis does not depend on these details and is therefore truly a property of the network and the initialization, and so these results should hold for any task. Is this correct?\n\n4) The experimental setup in Sec. 6 is incomplete - the loss is never specified (MSE, CE?), it would be nice  to see (empirically) that the results do not depend on it.\n\n5) Can the authors comment on the relation of their work and Refs. [1,2]?\n\n6) What happens to the topological structure if we consider the discrete GD instead of GF? the results should hold for small learning rate (as shown numerically), but there is a possible catapult regime (Ref. [3]) at large learning rate, will the paths not be restricted to connected parts in this case? could the catapult effect cause a transition between disconnected effective manifolds?\n\n[1] https://arxiv.org/pdf/2401.10791\n\n[2] https://arxiv.org/pdf/2402.05626\n\n[3] https://arxiv.org/abs/2003.02218", "limitations": "The authors discuss their limitations in a dedicated section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I believe the paper has several strengths, in particular:\n\n- **Soundness and clarity** - The authors review some known results, and build upon them in an instructive way, that allows the reader to understand precisely what has been done. Additionally, they provide proofs, as well as intuition for all of their results.\n\n- **Novelty** - As far as I know, this topological analysis is new, with the closest work being Ref. [1], which is still very different. I believe this work can shed light on the training dynamics of GD in this setting.\n\n\nReferences:\n\n[1] - https://arxiv.org/pdf/2401.10791", "weaknesses": "The main weakness of this work is its **scope**. The work studies in great detail a very specific architecture, but it is unclear how interesting their conclusions are for either theorists or practitioners. Additionally, their numerical/toy example is extremely simple, and it is a bit hard to see how their results extend to more complicated settings, even though they discuss some of these aspects in their limitations section. It would be useful to include more complicated examples, even in the single hidden layer case, as well as contrast these results with cases in which the activations are not ReLU-like, i.e., commenting on how these results break down.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Colleague, thank you very much for your useful and insightful review.\n___\n**Questions.**\n\n**Q1,Q2.**\n\nThank you for the corrections.\n\n**Q3.** \n\nThat is correct. Particular activations or datasets may induce other symmetries but our results are independent of their choice.\n\n**Q4.**\n\nThank you for pointing out the lack of specificity about the loss function, which we will fix in the final version of the paper. We indeed used the MSE loss. You can appreciate the generality w.r.t. the loss function in the experiment described in part 2 of the general rebuttal where the BCE loss is employed. \n\n**Q5.**\nThe main relation of [1] with our work is in their Lemma 1 (balancing in GF) and the subsequent observation on the constant sign of the the weight on the outgoing edge, when $e=1$, which is the same observation proved in our Corollary 3. Nevertheless, while it is not difficult to prove corollary 3 as a consequence of the balance equations in the case e=1, our topological approach not only shows that the obstruction happens if and only if e=1, but also it explains in precise mathematical terms why the obstruction appears or not.\n\nIn [2], the authors study one-layer networks with homogeneous activation functions and, in the case of MSE loss, they derive a condition for a stationary point to be a local minimum. \nThis condition prescribes that there must be no *escape neurons* which, intuitively, provide escape directions from a saddle point.\nIn the case of a scalar output ($e=1$), they prove that this condition is necessary and sufficient and that escape neurons are characterized by having $W^{(2)}_k = 0$.\nThe general goals of the paper are different from ours, as we don't make any assumption on the loss function and, therefore, we don't aim at characterizing its local or global minima. However, it would be interesting to study the relations between the two works, in particular how our *pathological neurons* relate to the escape neurons.\n\n**Q6.**\n\nPlease refer to part 3 of the general rebuttal, where we show one example of gradient descent circumventing the obstruction.\nAt the moment, it is difficult to know if this is due to the catapult mechanism, and we think that understanding it is an interesting future research direction.\n___\n\n**Weaknesses**\n\n**W1.** *The main weakness of this work is its scope. The work studies in great detail a very specific architecture, but it is unclear how interesting their conclusions are for either theorists or practitioners.*\n\nThis shallow network as a model has limitations, but our work could become a theoretical starting point for studying other architectures in a similar way. Indeed, single-layer ReLU neural networks are commonly embedded as sub-networks of architectures like CNNs, which could show obstructions of the same kind studied here. \nThe implications are not direct on the practical side, but initialization schemes could benefit from circumventing the possibility of obstruction by having $c_k>0$ for every hidden neuron.\n\n**W2.** *Additionally, their numerical/toy example is extremely simple, and it is a bit hard to see how their results extend to more complicated settings, even though they discuss some of these aspects in their limitations section. It would be useful to include more complicated examples, even in the single hidden layer case, as well as contrast these results with cases in which the activations are not ReLU-like, i.e., commenting on how these results break down.*\n\nPlease refer to part 2 of the general rebuttal, where we describe a further, more realistic experiment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "87xd4VPhZx", "reviewer": "Reviewer_Gc1k", "summary": "This works studies how topological properties affect loss landscape of neural networks, showing that the loss can be divided into disconnected regions where reaching one region from the other using GF is impossible. The theory is shown to have consequence for understanding the training of simple networks", "questions": "Does the results still hold when there is a bias term？", "limitations": "The paper could also benefit by discussing more of its limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Studying topological properties of the loss landscape is interesting and should be encouraged, the theory is solid and convincing", "weaknesses": "This paper has quite a few weaknesses in its current form, and these prevent me from recommending acceptance at this stage\n\n1. Prior works. (a) it is unclear how different this work is from Safran et al. [35]. This point is never discussed, and this makes it difficult for me to judge the novelty of this work. (b) Incorrect reference. Eq. (11) appears the earliest in https://arxiv.org/abs/1312.6120, not in Du et al. [11]. (c) emergence of topological obstruction due to permutation/rescaling symmetries have been studied in https://arxiv.org/abs/2309.16932, and the authors need to compare the results to clarify what is novel in the present manuscript\n\n2. The main result applies only to two layer bias-less ReLU nets, which I feel to be too weak\n\n3. The meaningful implications (corollary 1) is only relevant when either the input or the output dimension is 1, which makes the theory very unlikely to the relevant to practice\n\n4. The experiment also has very limited scope. If the authors can show that the effect they studied is relevant for training a much larger model on a more realistic dataset, I would be more convinced of the relevance and contribution of the theory", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Colleague, thank you very much for your useful and insightful review. \n___\n**Weaknesses.**\n\n**W1.**\n- **(a)** Thanks for this question, an answer to which we believe can improve the quality of our paper. There are several differences, namely: in [35], the input-output layers are restricted to be 1-dimensional; second, they are interested in analyzing the number of PL components for the trained networks, i.e., the decomposition of the Input Space into convex polytopes over which the function associated with the NN is linear. We are rather trying to understand the geometry and the topology of the weights space under GF training. We will insert this clarification in the main text.\n- **(b)** A result close to Eq. (11) is indeed found there, although applied to *linear* neural networks without ReLU activations. We nonetheless agree that it should be mentioned in the main text as preceding Du et al. [11].\n- **(c)** We thank you for highlighting this work, which we were not aware of at the time we submitted ours. Nevertheless, following your pointer, we noticed that it has now been published in ICML. In that paper, the authors study general mirror-reflect symmetries (which include rescaling) of the loss function and derive the constraints that they impose on the gradient and, as a consequence, on gradient-based optimization. While they derive interesting results from their framework, like the emergence of sparsity, their work doesn't deal with the topology of such constraints, which is the main goal of our paper. We will be happy to include it in the references and point out what we have just observed.\n\n**W2.**\n\nAs we mention in line 91 of the paper, we discuss the inclusion of biases in Appendix E. The results we find are mostly left unchanged as biases can be treated in the same way as the parameters of the input weights, effectively resulting in a network with $d+1$ inputs.\nThis shallow network as a model has limitations but our work could become a theoretical stepstone for studying other architectures. Indeed, this framework may be found inside other models (CNNs) or mechanisms (Graph Attention Network, need to check). Besides, there is an overly extended scientific literature on shallow ReLU networks, as can be appreciated in ref. A B C, and, in particular, in the find survey on shallow with a lot of citations, which will be added to our reference also in answering your important question.\n\n**W3.** \n\nCorollary 1 is a mathematical fact independent of the data nature, and its meaning relies on being a structural characteristic of shallow ReLU tout-court. Concerning the likelihood of facing these situations, there are two distinct considerations to be done: first, there are multiple widely used tasks where the output is a single scalar, for example, binary classification and scalar regression, and there is boundless literature on this architecture; on the other hand, we agree the input dimension equal to 1 may be of theoretical interest only.\n\n**W4.**\n\nPlease find in part 2 of the general rebuttal the results of a further experiment that displays the effect of obstruction in a more realistic setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful and thorough assessments of our work.

Prompted by some of the reviewers' questions, we performed three additional experiments to clarify some of the points raised.

### **1. Probability of obstruction**
Let us consider the following question: *what is the probability of having a disconnected invariant set given a realistic initialization?* 

Consider a one-layer neural network with $e=1$ and assume that the weights are sampled independently of one another from a normal distribution 
$W^{(1)}\_{ki}\sim\mathcal{N}(0,\sigma_1^2)\ \forall k,i$ and $W^{(2)}\_{k}\sim\mathcal{N}(0,\sigma_2^2)\ \forall k,j$. 

From Corollary 2, we know that the invariant set $\mathcal{H}(c)$ will be disconnected if and only if there exists a hidden neuron satisfying the following $\sum_{i=1}^d (W^{(1)}_{ki})^2 < (W^{(2)}_k)^2$.

Given the independence of weight sampling, this probability can be computed as 
$\mathbb{P}[\text{obstruction}] = 1-\mathbb{P}[\sum_{i=1}^d (W^{(1)}_{ki})^2 > (W^{(2)}_k)^2]^l$,

 which it is not hard to show to be equal to $1-(F_{1,d}(d\sigma_1^2/\sigma_2^2))^l$, where $F$ is the cdf of the Fisher-Snedecor distribution.

Having obtained this general expression, we can specify it to two common initialization schemes. 

- We obtain *Kaiming initialization* [1] with $\sigma_1^2 = 2/d, \sigma_2^2 = 2/l$ resulting in $\mathbb{P}[\text{obstruction}] = 1 - F_{1,d}(l)^l$;
- We obtain *Xavier normal initialization* [2] with $\sigma_1^2 = 2/(d+l), \sigma_2^2 = 2/(1+l)$ resulting in $\mathbb{P}[\text{obstruction}] = 1 - F_{1,d}(\frac{d+ld}{d+l})^l$.


We plot these two expressions in Figure 1 of the extra file we uploaded. 
We can clearly see how, for large values of $d$, the probability of obstruction quickly falls to 0 for any number of hidden neurons.
Instead, for small values of $d$, we see an opposite trend: the probability of disconnectedness grows with $l$. 
Moreover, it is interesting to notice that the region of high obstruction probability is much larger for Xavier initialization than for Kaiming initialization, further showing that the latter is preferred when working with ReLU networks.

As a complement, we also observe that, by using the binomial distribution, it is possible to show that the probability of having $2^B$ disconnected regions, assuming the same sampling scheme described above, is  ${ l \choose B}p^B p^{l-B}$, where $p=1-F_{1,d}(z)$, with $z$ the opportune argument depending on the variances or the layers' sizes.



### **2. A more realistic experiment**
We present here a further experiment to show how the topological obstruction presented in the paper can be a hindrance in a more realistic setting.
We consider a simple binary classification task on the well-known  *breast cancer* dataset [3] which we try to solve by fitting one-layer ReLU neural networks trained to minimize BCE loss.
We vary the number of hidden neurons $l$ and, for each $l$, we change the number of neurons $k$ such that $c_k>0$ from 1 to $l$. 
By repeating the experiment 100 times, we can show how the model's average performance changes when the degree of disconnectedness of its invariant set is varied.
The result, on the left panel of Fig. 2 clearly shows the presence of a "gradient" in performance, where increasing the number of positive $c_k$ (non-pathological neurons) tends to decrease the average value of the test loss after training.

### **3. Finite step size**
The results presented in our work hold when the network is trained with continuous-time GF so it is natural to wonder what changes when we consider finite-step size gradient descent.
While a formal or thorough numerical analysis is outside the scope of this work, we show in a simple setting how *it is possible* for a large enough step size to make a trajectory abandon the initial $\mathcal{H}(c)$ and circumvent the obstruction. 
We consider the same numerical setup of Section 6 and initialize the parameters on the wrong component of a $\mathcal{H}(c)$ with $c_1=c_2=-0.03$ so that it results in disconnection while also being small enough to make the component close and facilitate the trajectory's escape.
In Fig. 3 of the supplementary pdf, we see how raising the learning rate high enough can change the $c_k$ of the first neuron, pushing it over the obstruction and allowing it to reach the global minimum close to the other component.

- [1] *Delving deep into rectifiers: Surpassing human-level performance on imagenet classification -  He, K., Zhang, X., Ren, S. \& Sun, J. (2015).*
- [2] *Understanding the difficulty of training deep feedforward neural networks - Glorot, X. \& Bengio, Y. (2010).*
- [3] *Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository - Wolberg, W., Mangasarian, O., Street, N. \&  Street, W. (1995).*

Meta-review: The work studies the properties of the loss surfaces of two-layer neural networks with homogenous activations (e.g., ReLU). The authors prove that in such networks, the gradient flow trajectories lie on an invariant set, which can be factored as the product of quadric hypersurfaces. The authors derive the properties of such sets and, importantly, show that for neural networks with a single output neuron, such sets may be disconnected, which creates obstacles for gradient flow and makes it (sometimes) impossible to reach the global optimum.

The reviewers found the results of the paper to be interesting and inspiring, and the overall exposition to be very clear and accessible. The key concepts are well illustrated with the figures and experimental illustrations. The area chair after reading the paper is also on the positive side. The key weakness pointed out by the reviewers is the limited scope, as the paper (primarily) studies a particular setting (continuous gradient flow, two-layer MLP only, homogenous activations, primary focus on the scalar output, etc.). However, it is likely that some of the theoretical advancements developed in the current paper can be extended to more general cases of neural networks. As a result, this could help to broaden the theoretical understanding of neural networks and help to advance the field of deep learning from the practical side as well (e.g., NN initialization, optimization).

Overall, there has been a large amount of fruitful discussions of authors with the reviewers, with clear suggestions on how and what to further improve in the paper, in particular, to highlight the open problems and potential future research in this direction. Upon acceptance, the area chair requests the authors to spend sufficient time incorporating these discussions into the paper accordingly.
Decision: Accept (poster)