Title: Topological obstruction to the training of shallow ReLU neural networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: learning dynamics

topology

two-layer neural networks

ReLU networks

geometry

symmetry

loss landscape

gradient flow
Abstract: Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings.
This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. 
When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments.

Topological obstruction to the training of shallow ReLU neural networks

Abstract

Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings. This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments.

1 Introduction

Training a neural network consists of navigating the complex geometry of the loss landscape to reach one of its deepest valleys. Gradient descent and its variants are, by far, the most commonly used algorithms to perform this task. While technically correct, the standard picture of the parameter space as Euclidean space with the trajectory rolling down the loss's surface in the steepest direction towards a minimum is slightly misleading because different choices of parameters can be observationally equivalent i.e. encode the same function [10]. The observational equivalence of parameters shape the loss landscape by imposing specific geometric structures on the parameter space. Minima are not isolated points but high-dimensional manifolds with complex geometry [17, 9, 42] and the loss function's gradients and Hessian are constrained to obey some specific laws [46, 27]. Gradient-based optimization methods, where the parameters are updated by performing discrete steps in the gradient's direction, are thus very much dependent on the symmetry-induced geometry [12, 29].

In this work, we provide a topological perspective on the constraints induced by some groups of network symmetries on the optimization trajectories. Topology is a field of mathematics that studies the properties of a space that are preserved under continuous deformations. Our main goal is to find and quantify in topological terms the impossibility of the training trajectories to freely explore the parameter space and get from any initialization to an optimal parameter. This idea is formalized in

the topological notion of connectedness and, in particular, with the 0-th Betti number , which counts the number of connected components the space is composed of. The presence, or the absence, of topological obstructions in the parameter space does not depend on the particular loss function or the training data but is intrinsic to the interplay between the geometry and the topology of the parameter space under the action of groups of symmetries inducing observationally equivalent networks.

Main contributions. Our main contributions are the following.

We find that, for two-layer neural networks, the gradient flow trajectories lie on an invariant set, which can be factored as the product of quadric hypersurfaces.

We analytically compute its Betti numbers, i.e., the number of connected components, holes, and higher-dimensional cavities.

We find that the invariant set can be disconnected when the network's output dimension is 1, leading to a clear topological obstruction.

We find that the obstruction is caused by 'pathological' neurons that cannot change the sign of their output weights when trained with gradient flow.

We discuss the relation between the invariant set and the network's symmetries, finding that if we consider permutations, the number of effective connected components scales linearly in the number of pathological neurons.

We perform numerical validations on controlled toy scenarios, displaying the effect of obstruction in practice.

2 Related work

A large body of work studies gradient flow and gradient descent optimization of one hidden layer networks with homogeneous activations. Convergence properties have been found for wide networks [37, 43] with bounded density at initialization [31]. The implicit regularization provided is studied under various assumptions on: orthogonal input data [4], initialization scale [30, 4], wide (overparameterized regime) and infinitely wide [6], linearly separable data [30, 45]. Deeper linear networks [24] have also been studied.

These works focus on proving convergence and understanding which (optimal) solution is found, whereas our work investigates the shape of the optimization space and focuses on cases where the optimum might not be reachable from a given initialization.

Closer to our work, Safran et al. [39] studies two-layer ReLU binary classifiers with single input and output, counting the number of their piecewise-linear components after training. Eberle et al. [13] focuses on the differential challenge posed by the ReLU activation function and studies properties like the uniqueness of the solution of a gradient flow differential equation for a given initialization.

ReLU activation is a nonnegative homogeneous function, meaning that particular weight rescalings do not change the neural network's function. This is at the heart of the counterargument to flatness measures made by Dinh et al. [10], which shows that the Hessian eigenvalues can be made arbitrarily large in this way. Neyshabur et al. [34] explores the effect such rescalings can have on the gradient, proposing a rescaling-invariant regularization, and Pittorino et al. [36] employs them to define invariant flatness measures. Generally speaking, neural networks possess symmetries [20], and symmetries influence the geometry of training. Du et al. [12] studies how symmetry leads ReLU networks to automatically balance the neurons' weights. Kunin et al. [27], Zhao et al. [51] study how it constrains the gradient and Hessian matrix, leading to conservation laws w.r.t. gradient flow and Tanaka et al. [46] leverages it to propose a network pruning scheme. Ziyin [52] studies general mirror-reflect symmetries of the loss function and their effect on the weights of the trained network. Other conserved quantities stem from batch normalization's scale invariance [23, 47]. The transition from gradient flow to finite step size gradient descent breaks the conservation laws, resulting in altered trajectories [14, 2, 27, 44].

Numerous works have explored the geometry and topology of the loss landscape to obtain insight into a neural network's training behavior. Motivated by the striking experimental observation that low loss points can be connected by simple curves [11, 18] or line segments [40, 16, 15], a large body of literature tries to understand this phenomenon of mode connectivity under the topological lens of the connectedness of the loss function's sublevel sets [17, 35, 26], especially for overparameterized neural networks [9, 8, 42]. Another line of work approaches the connectivity of minima from another point of view, studying the presence [50, 38, 48] or absence [28] of spurious minima, i.e. minima

which are not global. Bucarelli et al. [5] analytically derives bounds on the sum of the Betti numbers of the loss landscape's sublevel set. Topological data analysis methods have also been exploited to numerically study the shape of the loss landscape [1, 22].

3 Setup and preliminaries

3.1 One-hidden layer neural network

Unless otherwise stated, all vectors are column vectors, that is, x = ( x 1 , . . . , x d ) ⊺ ∈ R d ≅ R d × 1 . Let us consider a two-layer neural network f (⋅ , θ ) ∶ R d → R e specified by the function

where x ∈ R d is the input, θ = ( W ( 1 ) , W ( 2 ) ) with W ( 1 ) ∈ R l × d and W ( 2 ) ∈ R e × l are the parameters, σ ∶ R → R is the component-wise activation function and l is the number of neurons in the hidden layer. Notice that we consider a network with no biases, as it allows us a discussion with lighter notation. The case with biases is discussed in Appendix E.

In this work, following [12], we focus on the case where σ is homogeneous , namely σ ( x ) = σ ′ ( x ) ⋅ x for every x and for every element of the sub-differential σ ′ ( x ) if σ is non-differentiable at x . The commonly used ReLU ( σ ( z ) = max { z, 0 } ) and Leaky ReLU ( σ ( z ) = max { z, γ } with 0 ≤ γ ≤ 1 ) activation functions satisfy this property.

We call parameter space the vector space Θ = { θ = ( W ( 1 ) , W ( 2 ) ) ∣ W ( 1 ) ∈ R l × d , W ( 2 ) ∈ R e × l } .

It will also be convenient to examine the single hidden neurons and their associated parameters for the following discussions.

Proposition 1. For the two-layer neural network defined in Equation (1) . Let k = 1 , . . . , l , let ( e 11 , e 12 , . . . , e ll ) be the canonical basis of R l × l and Θ k = { θ k = ( e kk W ( 1 ) , W ( 2 ) e kk ) ∣ ( W ( 1 ) , W ( 2 ) ) ∈ Θ } ⊂ Θ , then Θ = Θ 1 ⊕/uni22EF⊕ Θ l .

Details of the proof are provided in Appendix A. Fixing k ∈ { 1 , . . . , l } , we can consider Θ k as the parameter space of the k -th hidden neuron, which consists of the inputs and output weights of neuron k , namely the rows and columns of W ( 1 ) and W ( 2 ) , respectively. For simplicity, when we work in Θ k , we write W ( 1 ) k ∶= e kk W ( 1 ) and W ( 2 ) k ∶= W ( 2 ) e kk . Interestingly, the decomposition of Proposition 1 only holds for two-layer neural networks and will be crucial to the formulations of this paper's results.

3.2 Symmetries and observationally equivalent networks

It is well known that the properties of the activation function heavily influence the geometry of the parameter space Θ . The activation function's commutativity with some classes of transformations can result in the latter having no effect on the function implemented by the neural network. This means that, in general, the mapping from the parameter space to the hypothesis class of functions is not injective. Following the terminology in Dinh et al. [10], we say that two parameters θ 1 , θ 2 ∈ Θ are observationally equivalent , if they encode the same function f (⋅ ; θ 1 ) = f (⋅ , θ 2 ) and write θ 1 ∼ θ 2 .

In the case of homogeneous activations (ReLU or Leaky ReLU), we describe two kinds of transformations that send a parameter θ into an observationally equivalent one.

Neuron rescaling. The input weights of a hidden neuron can be rescaled by a positive scalar α > 0 provided that its output weights are rescaled by the inverse α -1 (top panel of Figure 1a). We formalize this as the action of the group R + of positive real numbers on Θ k :

This action can be naturally extended to the space of all parameters by considering the possibility of rescaling all hidden neurons simultaneously by different factors. If α = ( α 1 , . . . , α l ) ∈ R l +

Given that σ ( az ) = aσ ( z ) when a ∈ R + , we see how θ ∼ T α ( θ ) .

We write T ( θ ) to denote the orbit of a parameter θ under the action of T , i.e. the set of all parameters obtained from θ by arbitrarily rescaling the neurons T ( θ ) = { T α ( θ ) ∶ α ∈ R l + } .

Permutations of the neurons. Besides rescaling, we can obtain an observationally equivalent network by permuting the hidden neurons in such a way as to preserve their input and output weights (bottom panel of Figure 1a).

Given the symmetric group on l elements S l of the permutations of { 1 , . . . , l } , we write the action

where R π is the l × l row-permutation matrix associated to the permutation π .

Given that the activation function σ is applied component-wise, we have that it commutes with R π , namely

Having defined these two actions, we say that θ and θ ′ are observationally equivalent by rescalings and permutations if θ ′ can be obtained from θ by a finite sequence of actions of T and P or, equivalently thanks to Lemma 3 in the Appendix, if there exists a rescaling α and a permutation π such that θ ′ = P π ○ T α ( θ ) . In this case, we write θ rp ∼ θ ′ .

3.3 Conserved quantities and the invariant hyperquadrics

The presence of symmetries in the neural network's parameter-function map results in a specific geometric structure in the loss landscape. Let indeed D = {( x i , y i ) ∈ R d × R e } N i = 1 be a training set of N input-output pairs and fix a loss function L ∶ Θ → R which depends on the parameters only through the output of the neural network (1), that is

where ℓ ∶ R e × R e → R is differentiable. In this work, as empirical risk minimization, we consider the continuous time version of the gradient descent (GD) algorithm (with learning rate h > 0 )

named gradient flow (GF), and defined as

where ∇ θ L ( θ ( t )) is the Clarke sub-differential [7] which takes into account the parameters θ where L ( θ ) is non-differentiable. Given that the loss function L depends on the parameters only through f , its value at θ must be constant over the orbit T ( θ ) . This, together with the fact that the gradient of a differentiable function at a point is orthogonal to the level set at that point, means that

at any parameter θ where L ( θ ) is differentiable, as represented in Figure 1b. This orthogonality condition constrains the possible values of the gradient and, by extension, the possible gradient flow trajectories. In particular, as proven in Liang et al. [29], Tanaka et al. [46], Equation (8) is equivalent to

For convenience of notation, we define, for k = 1 , . . . , l , the following bilinear forms on Θ , which help us describe the geometry induced by the rescaling symmetry. If θ = ( W ( 1 ) , W ( 2 ) ) and η = ( V ( 1 ) , V ( 2 ) ) , we define

which, notice, only depends on the k -th row of W ( 1 ) and k -th column of W ( 2 ) meaning that we can equivalently see it as a bilinear form on Θ k . Θ k , together with ⟪⋅ , ⋅⟫ k is a pseudo-Euclidean space .

With the notation given by Equation (10), we see that Equation (9) can be simply rewritten as ⟪ θ, g ( θ )⟫ k = 0 for every neuron k . This condition, akin to orthogonality w.r.t. the bilinear form of Equation (10), implies that, under gradient flow optimization,

This result, first obtained in Saxe et al. [41] for linear networks and discussed in Du et al. [12], Liang et al. [29], Kunin et al. [27], tells us that the rescaling symmetry results in the quantities ⟪ θ, θ ⟫ k being conserved. This means that the difference between the Euclidean norm of the inputs and the outputs is constant for each neuron throughout the GF training trajectory. Moreover, under the condition of homogeneity of the activation function, Du et al. [12] proves that Equation (11) holds even at non-differentiable points of L and in the case of multiple layers.

Invariant sets. Assume that at the initialization θ 0 we have ⟪ θ 0 , θ 0 ⟫ k = c k , for all k , then Equation (11) implies that the GF trajectory will lie on the set characterized by the system of equations ⟪ θ, θ ⟫ k = c k for k = 1 , . . . , l . This subset is mapped to itself under the GF dynamics by Equation (11) (see Figure 1b) and constitutes the main object of our study.

Definition 1 (Invariant set) . Given c = ( c 1 , . . . , c l ) , we call invariant set the subset H( c ) ⊆ Θ given by the equations ⟪ θ, θ ⟫ k = c k ∀ k = 1 , . . . , l .

If we look at each single equation (i.e. to each hidden neuron), we see that Equation (11) can be written as

which corresponds to a hyperquadric (or quadric hypersurface) in Θ k . We denote with Q( c k ) ⊆ Θ k this hypersurface and call it the invariant hyperquadric associated to the k -th hidden neuron.

Here c k ∈ R takes the role of a label associated with the k -th hidden neuron, which, we see in the next section, plays a key role in specifying the shape of Q( c k ) . Figure 2a shows how, for d = 2 and e = 1 , Q( c k ) is an hyperboloid with 1 sheet (connected) if c k > 0 and 2 sheets if c k < 0 .

4 Topology of the invariant set

As we discussed above, Equation (11) tells us that gradient flow trajectories can't explore the whole space Θ but are constrained to lie on the invariant set H( c ) . The values of c , in turn, depend on the initialization and, we see from Equation (12), quantify the balance between the norms of input and output weights in every hidden neuron.

The goal of this section is to provide a topological characterization of H( c ) that can tell us something about the presence or absence of fundamental obstructions to the network's training process. With obstruction, we mean the impossibility of a GF trajectory to travel freely from one point to the other in H( c ) . We refer the reader to Appendix B for an essential overview of some of the topological concepts that we rely on in the next paragraphs.

Counting high-dimensional holes. Our topological characterization will be framed using Betti numbers . Betti numbers are well-known topological invariants given by a sequence of natural numbers that intuitively encode the number of higher-dimensional holes and cavities present in space. In particular, the 0-th Betti number of a space X , β 0 ( X ) corresponds to the number of connected components of X and thus will be fundamental for our goal of identifying obstructions.

The invariant set H( c ) is given as the set of solutions of l polynomial equations of degree 2 sharing no variables. Furthermore, in the setting of two-layer neural networks, we can leverage the fact that the parameter space can be decomposed into the parameter spaces of the hidden neurons. This, in turn, allows us to decompose the invariant set as the product of the neurons' invariant hyperquadrics, greatly simplifying our study.

Lemma 1. In a two-layer ReLU neural network, the invariant set H( c ) is homeomorphic to the Cartesian product of the hidden neurons' invariant hyperquadrics, that is

Lemma 1 tells us that we can understand the topology of H( c ) by studying independently its factors. Moreover, the hyperquadrics we encounter here are well-studied objects for which the next proposition (proven in Appendix D.2) gives a topological characterization.

Proposition 2. If c k > 0 , Q( c k ) is a topological manifold homeomorphic to R e × S d -1 . If c k < 0 , Q( c k ) is a topological manifold homeomorphic to R d × S e -1 . If c k = 0 , Q( 0 ) is a contractible space.

Leveraging the decomposition of Lemma 1 and the characterization of the factors given by Proposition 2, we can explicitly compute all the Betti numbers of the invariant set. We give the next result in terms of the Poincaré polynomial of H( c ) , namely the polynomial whose coefficients are the Betti numbers (see Appendix B).

Theorem 1. Let l + , l -, l 0 be the number of positive, negative, and zero components of c , respectively. The Poincaré polynomial of H( c ) is given by

This result, which is proven in Appendix D.3, contains a wealth of topological information as it gives us the exact number of holes and cavities of any order, depending on the network's hyperparameters ( d, e ) and initialization ( l + , l -). In the rest of this work, we focus only on the 0-th Betti number as the non-connectedness of H( c ) provides a clear obstruction to the GF trajectories.

Connectedness of the invariant set. With regard to the connectedness of H( c ) , we can leverage Theorem 1 to obtain the exact number of connected components.

Corollary 1. The 0 -th Betti number β 0 of H( c ) , corresponding to the number of its connected components, is given by

Proof. This can be directly obtained from the coefficient of degree 0 of the Poincaré polynomial obtained through Theorem 1.

(4)

0>%

0<

0k

0k

(w)²+(w)²-(w)²

=Ck

What we see in Equation (15) is that in most cases, the invariant set is connected, and gradient flow has no topological limitations in exploring the whole of H( c ) . Instead, when the hidden neurons have only one input or only one output, the space is fragmented into several components whose number scales exponentially in l + or l -, respectively.

Let us focus on the more interesting case where d > 1 and e = 1 .

Corollary 2. If the output of a two-layer ReLU neural network is a single scalar e = 1 , its input has dimension d > 1 , and the initial parameter θ 0 is such that ⟪ θ 0 , θ 0 ⟫ k < 0 for l -> 0 hidden neurons, then the set H( c ) is disconnected and has 2 l -connected components.

This means that neurons initialized with the norm of their outgoing weight strictly greater than their incoming weights' norm are responsible for disconnecting the space. We now precisely identify which connected component a parameter θ belongs to and clarify the meaning of the obstruction.

Proposition 3. Let e = 1 , d > 1 , and θ ∈ H( c ) with c such that c k 1 , . . . , c k l -< 0 while c k ≥ 0 for all other k . Let W ( 2 ) -∶= ( W ( 2 ) k 1 , . . . , W ( 2 ) k l -) ∈ R 1 × l -be the row vector whose components are the components of W ( 2 ) ∈ R 1 × l associated to c k < 0 . Then the vector s ( θ ) = ( sign ( W ( 2 ) k 1 ) , . . . , sign ( W ( 2 ) k l -)) identifies uniquely the component θ belongs to, namely: θ and θ ′ belong to the same connected component of H( c ) if and only if s ( θ ) = s ( θ ′ ) .

Proposition 3, proven in Appendix D.4, implies that s ( θ ) does not change when we move in C on a continuous curve such as the one given by gradient flow. This gives us an interesting interpretation of the topological obstruction: gradient flow cannot change the signs of the outgoing weights of the hidden neurons k such that c k < 0 (see Appendix G for an intuitive explanation of the phenomenon). This same observation is also mentioned in Boursier and Flammarion [3]. Proposition 3 extends one of the results of Boursier et al. [4] which proves that the same also holds when c k = 0 (balanced initialization).

By also considering Corollary 1, one obtains that a clever initialization of the parameters given by ⟪ θ 0 , θ 0 ⟫ k = c k > 0 ∀ k = 1 , . . . , l can prevent the issue by ensuring the connectedness of the invariant set. We also find that under common initialization schemes such as Xavier [19] and Kaiming [21] the probability of having pathological neurons is negligible when the input dimension and number of hidden neurons is high (see Appendix F).

5 Taking symmetries into account

Corollary 2 states that neurons k such that c k < 0 are 'pathological', in the sense that they are responsible for disconnecting the invariant set into several components, whose number scales exponentially in the number of those neurons. This result gives us a grim picture of the possibility of actually

optimizing the neural network: if the initial parameter θ 0 is in a particular connected component and the global optimum θ ∗ lies in another, then any gradient flow trajectory will not be able to reach θ ∗ because it will be constrained in its connected component.

This result, however, provides us only with a partial picture of the parameter space's geometry. It is a priori possible that the training trajectory, moving in its connected component, reaches a parameter ζ , which itself is optimal as it is observationally equivalent to θ ∗ ( ζ ∼ θ ∗ ). In this case, the topological obstruction given by the non-connectedness would be only apparent.

To take this fact into account, we define the following notion.

Definition 2 (Effective component) . Let θ ∈ H( c ) and C ( θ ) be its connected component therein. We define its effective component Eff ( θ ) as the union of the connected component of all θ ′ such that θ ′ rp ∼ θ . So that Eff ( θ ) ∶= ⋃ θ ′ rp ∼ θ C ( θ ′ ) .

Figure 2b gives a picture which clarifies the definition, showing a space with 4 connected components that has only 3 effective components. If the optimum θ ∗ belongs to the same effective component as the initialization, then it is possible to reach a parameter that is observationally equivalent to it (through permutations and rescalings).

We present a useful result which tells us that the action of rescaling of Equation (3) can take any non-degenerate parameter θ ∈ H( c ) to any other invariant set H( c ′ ) for every c ′ ∈ R l . This means that any invariant set can realize all the neural network's functions.

Proposition 4. For every c k ∈ R and for every θ k ∈ Θ k such that W ( 1 ) k , W ( 2 ) k ≠ 0 , there exists a unique α k ∈ R + such that T α k ( θ k ) ∈ Q( c k ) . If W ( 1 ) k = 0 and W ( 2 ) k ≠ 0 , then the same holds for every c k < 0 , while, if W ( 1 ) k ≠ 0 and W ( 2 ) k = 0 , it holds for every c k > 0 .

The proof can be found in Appendix D.5 with the formula of the specific α which realizes the rescaling.

The following theorem leverages the power of Proposition 4 to give necessary and sufficient conditions for θ and θ ′ to belong to the same effective component.

Theorem 2. Let d > 1 and e = 1 . Let c ∈ R l and l -be the number of neurons such that c k < 0 . Assume that l -≥ 1 . Let C,C ′ ⊆ H( c ) be two distinct connected components of H( c ) such that s ( θ ) = s , ∀ θ ∈ C, and s ( θ ′ ) = s ′ , ∀ θ ′ ∈ C ′ . Then, the following statements are equivalent:

1. for every θ ∈ C there exists θ ′ ∈ C ′ such that θ rp ∼ θ ′ ;

The theorem, proven in Appendix D.6, tells us that, while connected components are identified by s , the effective components are identified only by the values of ∑ i s i or, equivalently, by the distribution of ± 1 in s . Therefore, we find that the number of effective components scales much slower than the exponential growth of the number of connected components given by Corollary 1.

Corollary 3. The number of effective components of H( c ) is given by 1 + l -.

Proof. Theorem 2 tells us that two connected components C,C ′ belong to the same effective component if and only if their associated sign vectors s, s ′ ∈ {-1 , 1 } l -have the same sum. The number of effective components will thus equal the number of different values that the sum ∑ l -i = 1 s i can have. If s i = 1 ∀ i then ∑ l -i = 1 s = l -. Each switch of a component to -1 decreases the sum's value by 2 until it reaches the minimum -l -. Therefore, the total number of values of the sum will be 1 + l -.

6 Empirical Validation

Task, dataset, and model setup. We display here a toy example, showing how the initialization of the model can cause a topological obstruction, making the optimum unreachable.

We consider the function F ( x 1 , x 2 ) = -( x 1 + x 2 ) , which will be our ground-truth. Next, we generate a dataset of 8000 points ( x i , F ( x i )) by sampling x i ∼ U ([ 0 , 1 ] 2 ) . Our model, depicted in Figure 3a)

a

W(1)

Q(c1)

C1<0

Q(c2)

0>

11

W()

12

w(2)

#0,(0)

#

W(1)

21

0(t)

02(t) 02(0)

W(1)

22

2

W;(2)

2

2.5-

Bad initialization-

Good initialization

2.0-

Q(c1)

0>

Q(c2)

℃2>0

1.5

(0)T

:W(2)

#0,(0)

1.0-

0(tx)

02(0)

0.5

0.0-

0

100

200

300

400

500

02(tx)

Optimization steps

is a one hidden layer neural network with 2 hidden neurons, ReLU activations and no biases. All the weights are initialized by independently sampling from U ([- √ 2 , √ 2 ]) . From the task and the network's architecture, it is clear that at least one of the output weights has to be negative to approximate F correctly.

To standardize our results, we apply the rescaling of Proposition 4 and relocate the initial parameters to an observationally equivalent one in the invariant set H( c ) with c k ∈ {-0 . 1 , 0 . 1 } , controlling the sign of the weights on the last layer. We allow ourselves to do these two manipulations to control the experiments while only marginally modifying the network initialization, avoiding the introduction of massively unbalanced weights, which could change the dynamics, as shown in Neyshabur et al. [34]. Finally, we train the network using gradient descent on the MSE loss with a small learning rate of h = 0 . 01 . This limits the variations of c k values to less than one percent along training, giving us a good approximation of gradient flow.

Results. We initialize different models and collect all states and losses. First, when we initialize the model with an 'unlucky' configuration, namely c = (-0 . 1 , -0 . 1 ) (the space has 4 connected components) and s ( θ ) = (+ 1 , + 1 ) , we find that the trajectories are confined to the positive region of their invariant hyperquadric, resulting in a poor approximation of F , as we can see in Figure 3b (top) and in the loss of Figure 3c. Instead, with an initial configuration such that c = (-0 . 1 , + 0 . 1 ) (2 connected components) and s ( θ ) = (+ 1 , + 1 ) , the model can leverage the connectedness of Q( c 2 ) to learn F by flipping the sign of the second neuron's output weight (Figure 3b bottom right).

A more realistic experiment. We present here a further experiment to show how the topological obstruction can be a hindrance in a more realistic setting. We consider a simple binary classification task on the well-known breast cancer dataset [49], which we try to solve by fitting a one-layer ReLU neural network trained to minimize the BCE loss. We vary the number of hidden neurons l and, for each l , we change the number of non-pathological neurons l + (neurons with c k > 0 ) from 0 to l . We repeat the experiment with 100 different random initializations and show how the model's average performance changes when the degree of disconnectedness of its invariant set is varied. The result, on the left panel of Figure 4, clearly shows the presence of a "gradient" in performance, where increasing the number of non-pathological neurons decreases the average value of the test loss after training. The right panel of Figure 4, moreover, shows how the impact of the obstruction depends on the number of non-pathological neurons and not on their fraction over the total number of hidden neurons.

2

3

4

5

6

7

8

9

l

9

8

7

6

5

4

3

2

1

0

l

+

0.27

0.29

0.28

0.33

0.33

0.31

0.35

0.33

0.37

0.35

0.43

0.38

0.36

0.36

0.38

0.45

0.44

0.42

0.44

0.43

0.42

0.55

0.49

0.49

0.49

0.49

0.46

0.49

0.59

0.59

0.56

0.6

0.6

0.57

0.54

0.57

0.65

0.64

0.65

0.65

0.65

0.62

0.64

0.63

0.69

0.69

0.69

0.69

0.69

0.69

0.69

0.69

Average BCE Loss

4

16

64

256

l

1.0

0.5

0.0

l

+/

l

0.41

0.22

0.2

0.19

0.53

0.25

0.23

0.19

0.69

0.69

0.69

0.69

Average BCE Loss

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.2

0.3

0.4

0.5

0.6

7 Conclusions

In this paper, we have given analytical results that clarify the nature of the constraints imposed by gradient flow on the parameter space of a two-layer neural network with homogeneous activations. In the case of a single scalar output, which appears in tasks such as binary classification and scalar regression, we identified initial conditions that lead to a topological obstruction in the form of the parameter space's fragmentation into multiple connected components. This is caused by pathological neurons whose output weights cannot change their sign during training. Moreover, if one also considers the network's symmetries under permutations of the hidden neurons, we find that most of the connected components are equivalent. The number of effective components of the resulting space scales linearly with the number of pathological neurons, contrasting with the exponential growth of the number of connected components obtained without considering the permutation symmetries.

As shown in the last numerical experiment, the lack of non-pathological neurons hinders learning, even when the network's width is scaled. Our probabilistic analysis outlined in Appendix F, however, shows that with common initialization schemes, the probability of creating a pathological neuron decreases rapidly with increased inner layer width. Therefore, the combination of specific initialization schemes and a large number of hidden neurons (beyond the minimum required to solve a task) appears to make this obstruction unlikely in practice. This work describes a simple safeguard to avoid obstructions, which can, for instance, discourage the usage of initialization schemes that result in the proliferation of pathological neurons.

8 Limitations

The main limitation of the work is the network's architecture, which is limited to only one hidden layer. Considering multiple layers, we can still define rescalings and permutations and find invariant hyperquadrics for each hidden neuron. The issue emerges in the fact that these hyperquadrics are not 'independent' anymore, and the invariant set cannot be factored into the product of the Q( c k ) . This intuitively results from the fact that in the multi-layer case, each weight in the hidden layers is shared by two neurons.

The second limitation is that our study focuses on gradient flow optimization. This idealized situation doesn't take into account the fact that moderate step size of gradient descent and stochastic gradient descent can break the conservation of ⟪ θ, θ ⟫ k and make the parameters drift away from the invariant set [2]. Moreover, popular optimizers like ADAM [25] update the parameters employing the gradients at previous iterations so their trajectories will not be constrained to lie on H( c ) as we defined it.

The inclusion of regularization terms in the loss function, such as ℓ p regularizations, also breaks the invariance to rescalings.

References

Serguei Barannikov, Daria Voronkova, Ilya Trofimov, Alexander Korotin, Grigorii Sotnikov, and Evgeny Burnaev. Topological obstructions in neural networks learning. arXiv preprint arXiv:2012.15834 , 2020.

David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009.11162 , 2020.

Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is a two-edged sword. arXiv preprint arXiv:2401.10791 , 2024.

Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems , 35:20105-20118, 2022.

Maria Sofia Bucarelli, Giuseppe Alessio D'Inverno, Monica Bianchini, Franco Scarselli, and Fabrizio Silvestri. A topological description of loss surfaces based on betti numbers. arXiv preprint arXiv:2401.03824 , 2024.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory , pages 1305-1338. PMLR, 2020.

Francis H Clarke, Yuri S Ledyaev, Ronald J Stern, and Peter R Wolenski. Nonsmooth analysis and control theory , volume 178. Springer Science & Business Media, 2008.

Y Cooper. The critical locus of overparameterized neural networks. arXiv preprint arXiv:2005.04210 , 2020.

Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv:1804.10200 , 2018.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019-1028. PMLR, 2017.

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning , pages 1309-1318. PMLR, 2018.

Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems , 31, 2018.

Simon Eberle, Arnulf Jentzen, Adrian Riekert, and Georg S. Weiss. Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with relu activation. Electronic Research Archive , 2023.

Yuanyuan Feng, Tingran Gao, Lei Li, Jian-Guo Liu, and Yulong Lu. Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation. arXiv preprint arXiv:1902.00635 , 2019.

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems , 33:5850-5861, 2020.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint arXiv:1611.01540 , 2016.

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems , 31, 2018.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Grzegorz Głuch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprint arXiv:2104.05508 , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Stefan Horoi, Jessie Huang, Bastian Rieck, Guillaume Lajoie, Guy Wolf, and Smita Krishnaswamy. Exploring the geometry and topology of neural network loss landscapes. In International Symposium on Intelligent Data Analysis , pages 171-184. Springer, 2022.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations , 2019. URL https://openreview. net/forum?id=HJflg30qKX .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106 .

Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems , 32, 2019.

Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:2012.04728 , 2020.

Shiyu Liang, Ruoyu Sun, Yixuan Li, and Rayadurgam Srikant. Understanding the loss surface of neural networks for binary classification. In International Conference on Machine Learning , pages 2835-2843. PMLR, 2018.

Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics , pages 888-896. PMLR, 2019.

Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems , 34:12978-12991, 2021.

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences , 115(33): E7665-E7671, 2018.

James R Munkres. Elements of algebraic topology . CRC press, 2018.

J.R. Munkres. Topology . Featured Titles for Topology. Prentice Hall, Incorporated, 2000. ISBN 9780131816299.

Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems , 28, 2015.

Quynh Nguyen. On connected sublevel sets in deep learning. In International conference on machine learning , pages 4790-4799. PMLR, 2019.

Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning , pages 17759-17781. PMLR, 2022.

Grant Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics , 75(9):1889-1935, 2022.

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International conference on machine learning , pages 4433-4441. PMLR, 2018.

Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems , 35:32667-32679, 2022.

Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454 , 2017.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013.

Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning , pages 9722-9732. PMLR, 2021.

Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics , 80(2):725-752, 2020.

Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176 , 2021.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research , 19 (70):1-57, 2018.

Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems , 33:6377-6389, 2020.

Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350 , 2017.

Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research , 20(133):1-34, 2019.

William Wolberg, Olvi Mangasarian, Nick Street, and W Street. Breast cancer wisconsin (diagnostic). UCI Machine Learning Repository , 10:C5DW2B, 1995.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. arXiv preprint arXiv:1802.03487 , 2018.

Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy. Symmetries, flat minima, and the conserved quantities of gradient flow. arXiv preprint arXiv:2210.17216 , 2022.

Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Conference on Machine Learning , 2024.

A Parameter spaces

Let ( e 11 , e 12 , . . . , e ll ) and ( e 1 , . . . , e l ) be the canonical bases of the spaces R l × l and R l , respectively and

Θ k , notice, is the subspace of Θ consisting of the weight matrices W ( 1 ) with null rows except for the k -th one, and weight matrices W ( 2 ) with null columns except for the k -th one. We can check that Θ = Θ 1 ⊕/uni22EF⊕ Θ l , because, if I l is the l × l identity matrix, ∑ e kk = I l so that

Moreover, Θ ≅ Θ 1 ×/uni22EF× Θ l via the linear isomorphism

This, we see, is equivalent to decomposing the neural network of Equation (1) into the computations of the single hidden neurons. Indeed, let f ( x ; θ k ) ∶= f ( x, ( e kk W ( 1 ) , W ( 2 ) e kk )) , then, considering that e kk e kk = e kk and that σ ( e kk v ) = e kk σ ( v ) , it holds that

Therefore ∑ k f ( x ; θ k ) = f ( x ; θ ) .

B Primer on topology

Here, we recall some basic facts about the topology required to understand the paper's results. A self-consistent introduction is outside this work's scope, so we refer the interested reader to more complete expositions in Munkres [33, 32].

Topological manifold. An n -dimensional topological manifold is a topological space X which locally looks like the Euclidean space R n . More formally, for each p ∈ X , there exists a neighbourhood U of p and a homeomorphism mapping U to an open subset of R n .

Contractible space. A topological space X is contractible if it can continuously deform to a point p ∈ X . This means that there exists a continuous map

such that F ( x, 0 ) = x and F ( x, 1 ) = p for every x ∈ X .

Betti numbers. Betti numbers formalize the notion of the hole in a topological space and extend it to describe higher-dimensional cavities. The general idea is that one can associate a sequence of Abelian groups named homology groups to any space X , which encodes rich information about the higher-dimensional cavities in X . For what we are concerned here, the rank of the k -th homology group is called the k -th Betti number β k ( X ) . β k ( X ) counts the number of k -dimensional holes in the space: β 0 ( X ) count the number of connected components, β 1 ( X ) the number of 'circular' holes and β 2 ( X ) the number of voids or cavities.

A contractible space X is connected and cannot have any holes, and thus its Betti numbers are β 0 ( X ) = 1 and β i ( X ) = 0 ∀ i > 0 .

Betti numbers are topological invariants , meaning they are preserved when a space is transformed via a homeomorphism, namely a bijective, continuous map with continuous inverse.

Poincaré polynomials. The Poincaré polynomial of a topological space X is the polynomial whose k -th coefficient is given by the k -th Betti number

Künneth formula. Künneth's theorem describes assembling the homology groups of a Cartesian product of spaces X × Y from the homology groups of the factors X,Y . One of its corollaries tells us that if we care only about Betti numbers, a simple relation holds between the Poincaré polynomials of X × Y and the ones of X and Y , namely,

Types of connectedness. In topology, there are several kinds of connectedness. Two of them are particularly important for this work.

1.) A topological space X is connected if it cannot be divided into two disjoint non-empty open sets. If it is not connected, the connected component of a point x ∈ X is given by the union of all connected subsets of X which contain x .

A topological space equal to the Cartesian product of two spaces X = Y × Z is connected if and only if Y and Z are both connected.

2.) A topological space is path-connected if, for every pair of points x, y ∈ X , there exists a continuous curve γ ∶ [ 0 , 1 ] → X such that γ ( 0 ) = x, γ ( 1 ) = y . The path-component of x is the set of all y ∈ X such that a continuous curve exists connecting x to y .

This second notion is more relevant to our setting, where we care about the possible destinations of the optimization trajectories.

Path-connectedness implies connectedness, but not the opposite. There are situations, however, where these two notions are equivalent. For example, when X is a topological manifold, X is connected if and only if X is path connected.

Notice that the 0-th Betti number β 0 ( X ) counts the number of connected components but, in general, not the number of path components. With Lemma 2, we prove that these two notions are equivalent for our object of study.

C Extra propositions and lemmas

Lemma 2. The invariant set H( c ) is connected if and only if it is path connected.

Proof. Lemma 1 tells us that H( c ) ≅ Q( c 1 ) × /uni22EF×Q( c l ) .

Let us focus on a particular Q( c k ) .

When c k ≠ 0 , Proposition 2 tells us that Q( c k ) is a topological manifold, and thus, it is connected if and only if it is path connected.

When c k = 0 , Q( 0 ) is not a topological manifold but contractible, implying that it is connected. Let us prove that it is also path-connected.

Let θ k , θ ′ k ∈ Q( 0 ) and define the curve γ ∶ [ 0 , 1 ] → Q( 0 )

such that γ k ; θ ( 0 ) = θ, γ k ; θ ( 1 ) = 0 . γ k ; θ ( t ) ∈ Q( 0 ) for every t ∈ [ 0 , 1 ] because

Therefore, the segment from θ k to 0 belongs to Q( 0 ) .

A continuous curve from θ k to θ ′ k can be thus obtained by

which is continuous because γ k ; θ ( 1 ) = γ k ; θ ′ ( 1 ) = 0 . Therefore Q( 0 ) is path connected.

Finally, if H( c ) is connected, then all of its factors Q( c k ) are connected, which, in turn, is true if and only if they are path-connected. A product of path-connected space is again path-connected, and therefore H( c ) is path-connected. The other implication is true because path-connectedness implies connectedness, thus concluding the proof.

Lemma 3 (Interchange of rescalings and permutations) . Let θ ∈ Θ , α ∈ R l + and π ∈ S l , then, if ˜ α = R π -1 ( α )

we need to prove that diag ( α ) R π = R π diag ( ˜ α ) .

Let us pick a generic ˜ α ∈ R l + .

Let us consider the inverse permutation π -1 so that π -1 ( j ) = i if π ( i ) = j . Then, if ˜ α = R π -1 α ,

and thus we get that diag ( α ) R π = R π diag ( ˜ α ) .

D Proofs

D.1 Proof of Lemma 1

Proof. Proposition 1 tells us that the invariant set can be decomposed as the direct sum of the single hidden neurons' parameter spaces. This means that, for every θ ∈ Θ , there exist unique θ 1 ∈ Θ 1 , . . . , θ l ∈ Θ l such that

Therefore, we have a linear isomorphism φ ∶ Θ 1 ×/uni22EF× Θ k → Θ

The invariant set is a subset of Θ , which is given as the set of solutions of l equations ⟪ θ, θ ⟫ k = c k k = 1 , . . . , l . Notice that each of these equations involves a set of variables that appear only in that particular equation. These variables are exactly the ones which belong to Θ k . In fact ⟪ θ, θ ⟫ k = ⟪ θ k , θ k ⟫ k .

Therefore, given θ 1 ∈ Q( c 1 ) , . . . , θ l ∈ Q( c l ) we have that

On the opposite, given θ ∈ H( c ) we have that

Therefore, H( c ) is in bijection with Q( c 1 )×/uni22EF × Q( c l ) through φ which, being a linear isomorphism, implies also that H( c ) and Q( c 1 ) × /uni22EF×Q( c l ) are homeomorphic.

D.2 Proof of Proposition 2

Proof. Let us consider the three cases separately. If c k > 0 , Q( c k ) is defined by the equation

Proof. Given that

where ∥⋅∥ F is the Frobenius norm of a matrix, i.e., the square root of the sum of the squares of its elements. This can be rewritten as where √ c k +∥ W ( 2 ) k ∥ 2 F > 0 because c k > 0 . We define the map h ∶ Q( c k ) → S d -1 × R e as

where, notice, the first component belongs to the sphere S d -1 because of Equation (17) and W ( 2 ) k ∈ R e . This map is bijective, differentiable and has the following inverse h -1 ∶ S d -1 × R e →Q( k )

which is differentiable. Therefore, h is a diffeomorphism from Q( c k ) to S d -1 × R e .

If c k < 0 , we write the equation of Q( c k ) as

where W ( 1 ) k and W ( 2 ) k have switched their role to guarantee the term on the right to be positive. The diffeomorphism is now built analogously to Equation (18) as a map h ∶ Q( c k ) → S e -1 × R d .

If c k = 0 , we prove that Q( 0 ) is a contractible space. To do that, we exhibit a homotopy equivalence between Q( 0 ) and the point 0 , i.e. a continuous map p ∶ [ 0 , 1 ]×Q( 0 ) → Q( 0 ) such that p ( 0 , θ k ) = θ k and p ( 1 , θ k ) = 0 ∀ θ k ∈ Q( 0 ) . The map is defined in the following way:

This is continuous and well-defined because

D.3 Proof of Theorem 1

Proof. An implication of the Künneth formula is that the Poincaré polynomial of the Cartesian product of two spaces is equal to the product of their Poincaré polynomials:

Starting from Proposition 2, we can apply this result to Q( c k ) .

because a contractible space has 1 connected component and all of its other Betti numbers equal to zero.

Moreover, we know that R n is contractible for any n and its Poincaré polynomial is p R n ( x ) = 1 . The Poincaré polynomial of the sphere S n is given by p S n ( x ) = 1 + x n .

Equation (19) becomes

Given that Lemma 1 tells us that H( c ) can be factored into the product of the Q( c k ) , we apply Künneth formula and find that

D.4 Proof of Proposition 3

Proof. In the following, we exploit Lemma 2 and use the terminology connected and path-connected interchangeably.

Let us first prove that s ( θ ) = s ( θ ′ ) means that θ and θ ′ belong to the same connected component. We do this by explicitly building a continuous curve δ ∶ [ 0 , 1 ] → H( c ) such that δ ( 0 ) = θ and δ ( 1 ) = θ ′ .

Let us proceed by leveraging the homeomorphism from H( c ) and Q( c 1 )×/uni22EF × Q( c l ) and consider the different components of δ in the invariant hyperquadrics associated to each neuron δ = ( δ 1 , . . . , δ l ) .

If c k ≥ 0 , we know that Q( c k ) is path-connected and therefore we fix δ k ( t ) to any continuous curve in Q( c k ) such that δ k ( 0 ) = θ k and δ k ( 1 ) = θ ′ k .

for every i = 1 , . . . , l -.

γ i ; θ is a continuous curve which connects the point γ i ; θ ( 0 ) = θ k i with γ i ; θ ( 1 ) = ( 0 , s ( θ ) i √ -c k i ) .

If we define ¯ γ i ; θ ( t ) ∶= γ i ; θ ( 1 -t ) , which is the same curve as γ i ; θ but traversed in the opposite direction, we can define the curve

i.e. the curve which travels on γ i ; θ for t ∈ [ 0 , 1 2 ] and on ¯ γ i ; θ ′ for t ∈ [ 1 2 , 1 ] , for i = 1 , . . . , l -.

Notice now that δ k i ( 0 ) = θ k i and δ k i ( 1 ) = θ ′ k i . Moreover δ k i is continuous because under the hypothesis that s ( θ ) = s ( θ ′ ) .

Finally, we found a continuous curve δ = ( δ 1 , . . . , δ l ) such that δ ( t ) ∈ H( c ) ∀ t ∈ [ 0 , 1 ] and δ ( 0 ) = θ, δ ( 1 ) = θ ′ . Therefore, θ and θ ′ belong to the same connected component.

Let us now prove that if θ and θ ′ belong to the same connected component, then s ( θ ) = s ( θ ′ ) .

Let γ ∶ [ 0 , 1 ] → H( c ) be a continuous curve in H( c ) such that γ ( 0 ) = θ and γ ( 1 ) = θ ′ .

For each k ∈ K = { k 1 , . . . , k l -} such that c k < 0 , we know that γ k ( t ) ∈ Q( c k ) means that

for some function s k t 1 , 1 such that s k 0 s θ k and s k 1 s θ k

Assume, by contradiction, that s θ k s θ k . Assume also that s θ k 1 and s θ k 1 .

Notice that c k < 0 implies that p ( t ) ∶= √ -c k +∥ γ ( 1 ) k ∥ 2 F > 0 .

( ) ∈ {-} ( ) = ( ) ( ) = ( ′ ) . ( ) = -( ′ ) ( ) = + ( ′ ) = -

The function γ ( 2 ) k ( t ) = s k ( t ) p ( t ) , then, is a continuous function such that γ ( 2 ) k ( 0 ) > 0 and γ ( 2 ) k ( 1 ) < 0 and thus, by the intermediate value theorem, there exists t ∗ ∈ ( 0 , 1 ) such that γ ( 2 ) k ( t ∗ ) = 0 .

Repeating the argument for s ( θ ) k = -1 we then prove by contradiction that s ( θ ) k = s ( θ ′ ) k for all k ∈ K , thus concluding the proof.

D.5 Proof of Proposition 4

Proof. We have by Equation (10) and Equation (2):

By renaming A = d ∑ i = 1 ( W ( 1 ) ki ) 2 = ∥ W ( 1 ) k ∥ 2 F , C = e ∑ j = 1 ( W ( 2 ) jk ) 2 = ∥ W ( 2 ) k ∥ 2 F and multiplying by α 2 k > 0 we have:

Solving for α 2 k gives us:

Given that we want α k > 0 , we discard the negative solution. The other is positive because ∆ > c 2 k and thus √ ∆ > ∣ c k ∣ .

Of which we keep the positive solution only, with its full expression being:

Hence, if α = ( α 1 , . . . , α l ) with α k given by Equation (23), we get that ⟪ T α ( θ ) , T α ( θ )⟫ k = c k ∀ k = 1 , . . . , l .

Let us consider now the pathological cases W ( 1 ) k = 0 or W ( 2 ) k = 0 .

If W ( 1 ) k = 0 , W ( 2 ) k ≠ 0 then A = 0 , C ≠ 0 . Therefore, we have that Equation (22) becomes

which has solutions if and only if c k < 0 . In that case α k = ∥ W ( 2 ) k ∥ F √ -c k . This means that a hidden neuron with zero input weights and nonzero output weights can be rescaled only to the invariant hyperquadrics with c k < 0 .

If W ( 2 ) k = 0 , W ( 1 ) k ≠ 0 then C = 0 , A ≠ 0 . Therefore, we have that Equation (22) becomes

which has solutions if and only if c k > 0 . In that case α k = √ c k ∥ W ( 1 ) k ∥ F . This means that a hidden neuron with zero output weights and nonzero input weights can be rescaled only to the invariant hyperquadrics with c k > 0 .

If W ( 1 ) k = 0 and W ( 2 ) k = 0 , then θ k = 0 ∈ Q( 0 ) and it cannot be rescaled to any other invariant hyperquadric.

D.6 Proof of Theorem 2

Proof. Let us first prove that, if for every θ ∈ C there exists a θ ′ ∈ C ′ such that θ rp ∼ θ ′ then l -∑ i = 1 s ( θ ) i = l -∑ i = 1 s ( θ ′ ) i .

First, Lemma 3 tells us that we can interchange rescaling and permutation if we permute the rescaling factors accordingly. This means we can reduce any composite action of rescalings and permutations to the action of a single rescaling and a single permutation.

Let θ ∈ C and θ ′ ∈ C ′ such that θ rp ∼ θ ′ . Then there exist α ∈ R l + and π ∈ S l such that

This means that T α P π ( θ ) and θ ′ belong to the same invariant set and, specifically, to the same connected component. Therefore,

Notice that

because T α does not change the sign of W ( 2 ) as it acts by scaling it by positive factors. Let us focus on

If the neurons of θ such that c k < 0 are indexed by k 1 , k 2 , . . . , k l -, we will have that the neurons of P π ( θ ) such that c k < 0 are indexed by π ( k 1 ) , π ( k 2 ) , . . . , π ( k l -) . Therefore

for some permutation π -∈ S l -. Therefore, Equation (24) implies that

The action of rescaling and permutation can only reshuffle the label s of the connected component.

This means that

Let us now prove the other implication. Let s, s ′ ∈ R l -such that l -∑ i = 1 s i = l -∑ i = 1 s ′ i .

Given that their sum is equal, s and s ′ have the same number of + 1 and -1 and thus there exists a permutation π -∈ S l -such that s ′ = sR ⊺ π -.

Let π ∈ S l be the permutation which permutes the neurons such that c k < 0 according to π -and leaves the others fixed. In this way s ( P π ( θ )) = sR ⊺ π -= s ′ .

P π ( θ ) , however, doesn't belong to H( c ) but to another invariant set given by H( R π c ) .

Applying Proposition 4 we can find a rescaling α = α ( π ) ∈ R l + such that T α P π ( θ ) ∈ H( c ) .

Since neurons such that c k ≥ 0 , are left unchanged by the permutation, we can apply the proposition and we rescale them with α k = 1 . The permuted neurons are the ones such that c k < 0 , namely the ones whose weights satisfy ∥ W ( 1 ) k ∥ 2 F -∥ W ( 2 ) k ∥ 2 F < 0 , meaning that W ( 2 ) k ≠ 0 .

As noted above, the action of the rescaling doesn't change the sign vector, and thus

If we name θ ′ ∶= T α P π ( θ ) this result means that we found a θ ′ rp ∼ θ such that θ ′ ∈ C ′ , thus concluding the proof.

E Including biases

Let us consider the case where we include biases. The resulting two-layer neural network can be written as

where b ( 1 ) ∈ R l and b ( 2 ) ∈ R e .

To work with this extended set of parameters, we re-define the space

and the single hidden neuron spaces

where the second bias term b ( 2 ) does not appear because it is not directly involved with the computations of the hidden neurons. This means that we can write

where R e is included to describe the parameters in b ( 2 ) .

The neuron rescaling action now acts on the biases b ( 1 ) as well as the weights:

and can be extended to the whole space of parameters

Once again, we find that T α θ ∼ θ .

In this more general case, we can rewrite the bilinear form to include the biases. If θ = ( W ( 1 ) , b ( 1 ) , W ( 2 ) , b ( 2 ) ) and η = ( V ( 1 ) , p ( 1 ) , V ( 2 ) , p ( 2 ) ) , we define

and see that, once gradient flow optimization, we have a conservation condition like the one of Equation (9)

Once again, we call Q( c k ) the hypersurface of Θ k which satisfies the equation ⟪ θ, θ ⟫ k = c k and H( c k ) the set in Θ defined by ⟪ θ, θ ⟫ k = c k ∀ k = 1 , . . . , l .

With this in mind, it is not hard to extend the results of Proposition 2 and Theorem 1 which turn out to be slightly modified.

Proposition 5. If c k > 0 , Q( c k ) is a topological manifold homeomorphic to R e × S d . If c k < 0 , Q( c k ) is a topological manifold homeomorphic to R d × S e -1 . If c k = 0 , Q( 0 ) is contractible.

In this case, we can factor the space of parameters H( c ) as

where the last factor is due to the freedom in choosing the values of b ( 2 ) .

Proposition 6. Let c k ≠ 0 ∀ k = 1 , . . . , l . Let l + , l -, l 0 be the number of positive, negative and zero elements of c , respectively. The Poincaré polynomial of H( c ) is given by

Corollary 4. The 0 -th Betti number β 0 ( c ) of H( c ) , corresponding to the number of its connected components, is given by

The result we obtain is similar to Corollary 1 although slightly modified by the fact that having a single input neuron does not cause H( c ) to become disconnected anymore. In the case of e = 1 , therefore, the picture presented in the main text is left unchanged.

Xaviernormalinitialization

1

7

0

3

9

0

5

1

0

7

3

9

1

7

13

19

25

37

43

40

Kaimingnormalinitialization

7

0

3

9

0

25

0

31

37

43

6t

1

7

13.19

25

31

37

43

49

F Probability of obstruction

Let us consider the following question: what is the probability of having a disconnected invariant set given a realistic initialization?

Consider a one-layer ReLU neural network with e = 1 and assume that the weights are sampled independently of one another from a normal distribution W ( 1 ) ki ∼ N( 0 , σ 2 1 ) ∀ k, i , W ( 2 ) k ∼ N( 0 , σ 2 2 ) ∀ k, j . From Corollary 2 we know that the invariant set H( c ) will be disconnected if and only if there exists a hidden neuron satisfying ∑ d i = 1 ( W ( 1 ) ki ) 2 < ( W ( 2 ) k ) 2 . Given independence of the initial weight sampling, this probability can be computed as

where F is the cumulative distribution function of the Fisher-Snedecor distribution.

Having obtained this general expression, we can specify it to two common initialization schemes.

We obtain Kaiming initialization [21] with σ 2 1 = 2 / d, σ 2 2 = 2 / l resulting in P [ obstruction ] = 1 -F 1 ,d ( l ) l .

We obtain Xavier normal initialization [19] with σ 2 1 = 2 /( d + l ) , σ 2 2 = 2 /( 1 + l ) resulting in P [ obstruction ] = 1 -F 1 ,d ( d + ld d + l ) l .

We plot these two expressions in Figure 5. We can see how, for large values of d , the probability of obstruction quickly falls to 0 for any number of hidden neurons. Instead, we see an opposite trend for small values of d : the probability of disconnectedness grows with l . Moreover, it is interesting to notice that the region of high obstruction probability is much larger for Xavier initialization than for Kaiming initialization, further showing why the latter is preferred when working with ReLU networks.

G Intuition on the occurrence of obstruction

We can give some intuition on why there is no obstruction for multiple outputs. First, we consider a single hidden neuron k , with d incoming weights and a single output e = 1 . If the neuron is pathological, we have that

Since the weights W ( a ) ij ( t ) are continuous curves in time, for W ( 2 ) 1 k to change sign, its value needs to pass through 0 but, under the condition above, this cannot happen its square is always positive.

Consider now multiple outputs e > 1 , resulting in the conservation condition being

Now, any component W ( 2 ) jk can change sign by passing through 0 because the other components can compensate for it by increasing their magnitude to keep the condition satisfied.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the claims in the abstract are supported by analytical results and numerical experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our analysis in Section 8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions of the theoretical results are clearly specified and the complete proofs are written in Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The 'Task, dataset and model setup.' paragraph of Section 6 clearly explains our experimental setup.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Given the simplicity of our setup, we think that releasing the code would be unnecessary.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experiment's details are explained in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our simple experiment has the goal of displaying the meaning of the analytical results and, thus, we think it doesn't require error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Our experiment was performed on a simple laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewd the NeurIPS Code of Ethics and confirm that we conform to it in every repect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Being a purely theoretical work, we believe that it doesn't havesignificantt social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We release no data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We use no existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We release no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This works studies how topological properties affect loss landscape of neural networks, showing that the loss can be divided into disconnected regions where reaching one region from the other using GF is impossible. The theory is shown to have consequence for understanding the training of simple networks

STRENGTHS:
Studying topological properties of the loss landscape is interesting and should be encouraged, the theory is solid and convincing

WEAKNESSES:
This paper has quite a few weaknesses in its current form, and these prevent me from recommending acceptance at this stage

1. Prior works. (a) it is unclear how different this work is from Safran et al. [35]. This point is never discussed, and this makes it difficult for me to judge the novelty of this work. (b) Incorrect reference. Eq. (11) appears the earliest in https://arxiv.org/abs/1312.6120, not in Du et al. [11]. (c) emergence of topological obstruction due to permutation/rescaling symmetries have been studied in https://arxiv.org/abs/2309.16932, and the authors need to compare the results to clarify what is novel in the present manuscript

2. The main result applies only to two layer bias-less ReLU nets, which I feel to be too weak

3. The meaningful implications (corollary 1) is only relevant when either the input or the output dimension is 1, which makes the theory very unlikely to the relevant to practice

4. The experiment also has very limited scope. If the authors can show that the effect they studied is relevant for training a much larger model on a more realistic dataset, I would be more convinced of the relevance and contribution of the theory

QUESTIONS:
Does the results still hold when there is a bias term？

LIMITATIONS:
The paper could also benefit by discussing more of its limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the landscape topology of one hidden layer networks with non-negative homogeneous activations. The main result of this work is that in some cases, the loss landscape may consist of disconnected components that cannot be traversed by gradient flow dynamics. This leads to an obstruction, in the sense that certain initial conditions will result in closed paths away from the global minimum. They further count the number of effective components (in the sense of being equivalent under the symmetries of the problem) and show that this number grows linearly with the number of hidden neurons that cannot change their sign during the dynamics.\nThe authors provide a full analytical treatment of the problem, supplemented by a simple toy example.\n\nSTRENGTHS:\nI believe the paper has several strengths, in particular:\n\n- **Soundness and clarity** - The authors review some known results, and build upon them in an instructive way, that allows the reader to understand precisely what has been done. Additionally, they provide proofs, as well as intuition for all of their results.\n\n- **Novelty** - As far as I know, this topological analysis is new, with the closest work being Ref. [1], which is still very different. I believe this work can shed light on the training dynamics of GD in this setting.\n\n\nReferences:\n\n[1] - https://arxiv.org/pdf/2401.10791\n\nWEAKNESSES:\nThe main weakness of this work is its **scope**. The work studies in great detail a very specific architecture, but it is unclear how interesting their conclusions are for either theorists or practitioners. Additionally, their numerical/toy example is extremely simple, and it is a bit hard to see how their results extend to more complicated settings, even though they discuss some of these aspects in their limitations section. It would be useful to include more complicated examples, even in the single hidden layer case, as well as contrast these results with cases in which the activations are not ReLU-like, i.e., commenting on how these results break down.\n\nQUESTIONS:\nComments and questions below:\n\n1) L52: “studied by under various…” incoherent sentence.\n\n2) L51: “activation” should be “activations” I assume.\n\n3) In Sec. 3.3, the authors consider an arbitrary dataset and a general empirical loss function. The analysis does not depend on these details and is therefore truly a property of the network and the initialization, and so these results should hold for any task. Is this correct?\n\n4) The experimental setup in Sec. 6 is incomplete - the loss is never specified (MSE, CE?), it would be nice  to see (empirically) that the results do not depend on it.\n\n5) Can the authors comment on the relation of their work and Refs. [1,2]?\n\n6) What happens to the topological structure if we consider the discrete GD instead of GF? the results should hold for small learning rate (as shown numerically), but there is a possible catapult regime (Ref. [3]) at large learning rate, will the paths not be restricted to connected parts in this case? could the catapult effect cause a transition between disconnected effective manifolds?\n\n[1] https://arxiv.org/pdf/2401.10791\n\n[2] https://arxiv.org/pdf/2402.05626\n\n[3] https://arxiv.org/abs/2003.02218\n\nLIMITATIONS:\nThe authors discuss their limitations in a dedicated section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
In this paper, authors analyze the performance of gradient-descent optimization over a two-layer neural network. Authors reveal the presence of obstructions in the loss landscape and explore their topology. Finally, they identify the cause of those optimization obstructions in the so-called `pathological’ neurons—neurons that can’t change the sign of their output weight.

STRENGTHS:
1) An interesting and novel approach.

2) A step up from well studied gradient descent optimization of one hidden-layer networks.

3) Solid theoretical background and strict mathematical proofs of main results.

WEAKNESSES:
1) This paper identifies pathological neurons that disrupt the performance of gradient descent over ReLU activations; however, nothing is said about how such obstructions can be avoided or how their effect can be mitigated.

2) The scope of the research is limited to neural networks with two hidden layers.

3) The possible benefits for further research and practical applications of the achieved results are somewhat vague.

QUESTIONS:
See Weaknesses section.

LIMITATIONS:
Yes, the limitations are addressed. This is a purely theoretic work, dealing with a somewhat simplified example of a two layer fully-connected neural network.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the topological obstruction in the loss landscape of two-layer ReLU neural networks under gradient flow. Using conserved quantities in gradient flow, the authors define invariant sets, to which gradient flow trajectories are constrained. They then show that when the input or output of the network is a scalar, the invariant set may have more than one connected components depending on initialization. This leads to a topological obstruction, as gradient flow cannot reach a global minimum if parameters are initialized in the component that does not contain one. After taking scaling and permutation symmetry into account, the number of effective components scales linearly with the number of pathological neurons, which are neurons that cannot change the sign of their output weight. On a toy example with two hidden neurons, the authors fully characterize the invariant sets and show theoretically and empirically that gradient flows that do not start in the same effective component as the global minimum cannot reach the global minimum.\n\nSTRENGTHS:\n- This paper takes a novel approach of studying the topology of loss landscapes by examining invariant sets. While many work studies the loss landscape by proving topological properties of sub-level sets, this paper pursues an orthogonal direction by analyzing the topological properties of the set of reachable parameters by training trajectories. This approach combines the study of loss landscape and optimization dynamics, which leads to the original discovery of topological obstructions to the training of ReLU networks.\n- The paper provides concrete examples where gradient flows from certain initializations cannot converge to a global minimum and gives practical suggestions on how to choose initializations to avoid this issue, at least for two-layer ReLU networks. \n- The results on when invariant sets can be disconnected and the number of effective components provide valuable information on the loss landscape. These results could potentially lead to a new line of work on the trainability of various architectures under common optimization algorithms.\n- The paper is very well written. Setups and mathematical concepts are explained in an accessible way without losing rigor. The theorems and their implications are explained clearly. The visualizations (Figure 1-3) are informative and provide clear intuitions. The toy example in Section 6 is simple and effectively demonstrates a case where topological obstruction prevents gradient flow from reaching the global minimum.\n\nWEAKNESSES:\nThe scope of the analysis, which is restricted to two layer ReLU neural network and gradient flow, is rather limited.\n- While ReLU is a popular activation function, two-layer networks are rare in today’s machine learning models. The results in the paper do not readily extend to deeper networks, where invariant sets no longer factorize easily into product spaces.\n- Since SGD is known to have an implicit bias characterized by a drift of $c$, training trajectories in practical settings (with non-infinitesimal learning rate) will not stay in the same invariant set. Even if they stay close to an invariant set throughout training, the noise in each step may take the parameters to a different component. \n\nIf extending the analysis beyond this setup is challenging, additional experiments on different settings could also help illustrate that the topological obstruction is a realistic hindrance in practice. For example, for deeper ReLU networks, it might be possible to investigate empirically whether there exist initializations that lead to similar obstructions as observed in Figure 3c.\n\nQUESTIONS:\n- For ReLU networks with one hidden layer, does every invariant set contain a global minimum? If not, then even if an invariant set is connected, it is not guaranteed that all gradient flows starting from points on this set can reach a global minimum. This is not the type of obstruction considered in the paper, but I am curious about whether there are other considerations when choosing the initialization, besides ensuring the connectedness of the invariant set.\n- Assume that an invariant set contains a global minimum. Do all gradient flows in this invariant set converge to a global minimum?\n- For a wider two-layer ReLU network, using common initialization methods (such as Xavier initialization), what is the probability that the invariant set has more than one effective components? What is the probability that the initialization falls in the component that does not contain a global minimum? Is it nonzero?\n- Do the rest of Betti numbers, other than $\\beta_0$, provide useful information about the loss landscape and the training process?\n\n**Minor issues / suggestions**\n- Line 135-136: While allowing a finite sequence of actions is arguably more intuitive, it might suffice to just define observationally equivalence as being related by a composition of one $T$ and one $P$. As a consequence of Lemma 3 in the appendix, for any finite sequence of T and P, there exists a sequence of T and P with length at most 2 that produces the equivalent transformation.\n- In Equation 11, is the second expression missing a factor of 2?\n- The sentence in line 483 is not grammatically correct.\n\nLIMITATIONS:\nThe authors are upfront about the limitations and include a limitation section in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary of the paper:\n\n* The paper undertakes a theoretical 'reachability' analysis for neural\n  networks trained with gradient flow, identifying 'obstructions' separating\n  certain regions of parameter space from each other.\n* The setting is one-hidden-layer networks with homogeneous activation\n  functions, including ReLUs. The main paper considers the biasless case but\n  the authors extend their results to the case with biases in the appendices,\n  with only minor modifications to the results.\n* The authors consider training such networks with gradient flow and identify\n  that regardless of the (output-dependent) loss function that gradient flow\n  conserves a certain norm of the weights confining the training trajectory\n  to a subset of the parameter space they call the 'invariant set' which is a\n  product of hyperquadrics associated with each hidden unit in the network.\n* Through topological analysis the authors derive the Poincaré polynomial of\n  such invariant sets and in turn their zeroth Betti number, in effect\n  counting the number of connected components in the invariant set.\n  * These components are of interest because if gradient flow is initialised\n    in one component it is impossible for the training trajectory to find its\n    way to another, and training is limited to the best solution available in\n    the initial component.\n  * In this setting the zeroth Betti number (the number of connected\n    components) is only one for networks with multidimensional inputs and\n    outputs, indicating no topological obstructions.\n  * In contrast, for networks with scalar inputs or outputs, it is\n    exponential in the number of hidden units (with a base depending on the\n    network initialisation), indicating many topological obstructions.\n  * (For networks with biases, the space is connected if and only if the\n    network has a multidimensional output).\n* The authors then revise their count to take rescaling and permutation\n  symmetries into consideration.\n  * The reasoning is that if gradient flow can only explore its initial\n    connected component, this is only a problem when the component is\n    actually missing network functions rather than when it is merely missing\n    some (but not all) implementations of those functions. Perhaps only one\n    reachable implementation of the target function is needed.\n  * The result is that there are fewer 'effectively' connected components,\n    linear rather than exponential in the number of hidden units (with\n    constants again depending on the initialisation).\n  * However, the point is that there are still some topological obstructions\n    to worry about.\n* The authors provide a clear 'proof of concept' of their work in the form of\n  a numerical experiment simulating gradient flow by learning a small neural\n  network with gradient descent using a small learning rate. The results\n  clearly show that the learning trajectory is confined to the invariant set\n  and obstructed from reaching the global minimum as predicted.\n\nI wrote a bit of a long review, so here is also a summary of my review:\n\n* The main contribution, demonstrating the existence of topological\n  obstructions to gradient flow, appears novel, interesting, and an important\n  contribution to our understanding of the structure of parameter space.\n* I also think the paper is exceptionally well written and clear and the\n  analysis is elegant. I think the authors have discussed the limitations of\n  the work well. I am left with only a small number of questions.\n* My main concern is that the results show topological obstructions only\n  occur under some strong assumptions, and in fact they do not arise in\n  more situations. I find this somewhat at odds with the framing of the\n  contribution.\n\nOverall, I think the work is worthy of acceptance because it is elegant,\ninteresting, informative, and thought provoking, but I maintain concerns\nabout whether the framing is accurate.\n\nSTRENGTHS:\nI thank the authors for submitting their elegant and well-presented theoretical\nanalysis exploring an important topic.\n\n* Topological analysis of gradient flow is an interesting and apparently\n  novel approach that stands to make a meaningful contribution to the field's\n  theoretical understanding of the 'global' structure of parameter space.\n\n* The work clearly exhibits 'topological obstructions' in a restricted\n  setting, which is a thought-provoking phenomenon (though I have some\n  reservations about its relevance in practical settings, see below).\n\n* I appreciate the elegant and rigorous application of powerful, established\n  tools from the field of topology. The theoretical results are also\n  supported with a clear, small-scale proof-of-concept numerical experiment.\n\n* The presentation of the theory and experiments is very clear and\n  accessible. The figures are carefully designed and informative. The\n  technical details of the framework and results are complete and made the\n  paper quite accessible to me, even though I have limited knowledge of\n  topology (the primer in the appendix was appreciated).\n\n* Clear, thorough acknowledgement of related and prior work in the dedicated\n  section and throughout the paper.\n\nWEAKNESSES:\nThe authors have convincingly demonstrated the existence of topological\nobstructions under the stated assumptions. My main concern about this work is\nthe relevance of topological obstructions to deep learning practice.\n\n1. I am not so concerned about the restriction to single-hidden-layer\n   networks. The results may not generalise to the multi-layer case, or\n   perhaps they will, it seems worth looking into with future work. However,\n   I am concerned that *even in the single-hidden-layer setting* the main\n   disconnectedness result holds only for networks with scalar output. The\n   authors clearly show that networks with multidimensional outputs do not\n   show obstructions.\n\n    With this in mind, do the authors believe that the topological\n    obstructions they have identified are relevant to deep learning practice?\n    If so, can they clarify this point?\n\n2. Furthermore, I have concerns about the gap between gradient flow and\n   discrete gradient descent training. The authors explain that discretised\n   gradient methods are not confined to the same invariant sets as gradient\n   flow. It seems possible that this means discrete gradient methods count\n   circumvent topological obstructions, even if they exist in practical\n   settings.\n\n    1. Do the authors conjecture that the identified obstructions have\n       implications for gradient descent training?\n    \n    2. If the authors would support such an 'obstruction hypothesis', I would\n       invite them to consider explicitly stating the hypothesis in their\n       introduction and/or conclusion.\n    \n    3. The authors could also consider extending their simple numerical\n       experiments to explore whether and to what extent the demonstrated\n       obstruction affects training with larger learning rates or\n       momentum-based training methods.\n\nPending the authors' clarification of these points, the framing of the work\nis somewhat confusing to me. It appears to me that the authors have shown\nthat topological obstructions to gradient flow are actually quite a\nrestricted phenomenon. An alternative framing for the paper's results would\nbe to say that they show that deep learning is often *not* held back by\ntopological obstructions. Could the authors comment on this alternative\nframing?\n\nI don't think this concern is fatal to the paper. The topological analysis\napproach and most of the results are still informative and worth sharing even\nif the topological obstructions themselves end up not arising in more\npractical settings.\n\nQUESTIONS:\n1. The authors have presented a clear interpretation of the source of the\n   topological obstruction in terms of the relationship between each\n   unit's weights (lines 232 onwards). Do the authors have a similarly\n   enlightening interpretation of the breakdown of the topological\n   obstructions in the case with multidimensional outputs?\n\n2. Another assumption of the work is that the loss function depends only on\n   the network weights via the output of the network. Am I correct in\n   understanding that if the loss function includes a regularisation term\n   that depends on, for example, an $L_p$ norm of the weights, then this\n   assumption would be violated and the analysis would not hold because\n   gradient flow would no longer always be perpendicular to the invariant\n   sets? This seems fine but might be worth acknowledging.\n\n3. Appendix E extends the analysis to networks including biases, but stops\n   after deriving the zeroth Betti number in this setting. Could you please\n   clarify whether the analysis of section 5 is unchanged in this setting?\n\n4. The contribution list does not mention the 'proof of concept' numerical\n   experiments. It seems to me that these experiments are a reasonably\n   important part of the paper's contribution and its evaluation to warrant\n   mentioning in the introduction (as well as the abstract).\n\n5. Line 118: The authors describe the transformations as sending parameters\n   to different but observationally equivalent parameters. It is typically\n   true that the parameter produced is different but for some parameters or\n   for some transformations the transformations don't change the parameter.\n\nLIMITATIONS:\nThe paper accurately describes the contribution including its limitations.\nThese limitations are adequately and carefully disclosed in multiple places\nin the main text and discussed in detail in appendix F.\n\n1. One main limitation is the simplicity of the architecture, with only one\n   hidden layer. The authors discuss barriers to extending the work to the\n   multi-layer case, namely the interaction between layers, which is left to\n   future work.\n2. A second main limitation is the fundamental assumption of gradient flow.\n   The authors explain that the conservation laws that are crucial to their\n   analysis do not hold for discretised gradient descent.\n3. A third main limitation is that the topological obstructions only exist in\n   the case of networks with a scalar output. This is clearly acknowledged in\n   the abstract and throughout the paper, but I think the authors could have\n   more thoroughly discussed the implications of this restriction, namely\n   that in many cases we should not expect topological constructions at all.\n\nIf the work is to be accepted I would encourage the authors to include the\nvaluable discussion in appendix F in the main text, and to extend this\ndiscussion along the lines of (3).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}