Topological obstruction to the training of shallow ReLU neural networks

Authors: marco  nurisso
pierrick  leroy
francesco  vaccarino
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:48:57

Abstract: Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings.
This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. 
When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments.

Main paper: 
 introduction: 
 Training a neural network consists of navigating the complex geometry of the loss landscape to reach one of its deepest valleys. Gradient descent and its variants are, by far, the most commonly used algorithms to perform this task. While technically correct, the standard picture of the parameter space as Euclidean space with the trajectory rolling down the loss's surface in the steepest direction towards a minimum is slightly misleading because different choices of parameters can be observationally equivalent i.e. encode the same function [10] . The observational equivalence of parameters shape the loss landscape by imposing specific geometric structures on the parameter space. Minima are not isolated points but high-dimensional manifolds with complex geometry [17, 9, 42] and the loss function's gradients and Hessian are constrained to obey some specific laws [46, 27] . Gradient-based optimization methods, where the parameters are updated by performing discrete steps in the gradient's direction, are thus very much dependent on the symmetry-induced geometry [12, 29] .In this work, we provide a topological perspective on the constraints induced by some groups of network symmetries on the optimization trajectories. Topology is a field of mathematics that studies the properties of a space that are preserved under continuous deformations. Our main goal is to find and quantify in topological terms the impossibility of the training trajectories to freely explore the parameter space and get from any initialization to an optimal parameter. This idea is formalized in the topological notion of connectedness and, in particular, with the 0-th Betti number, which counts the number of connected components the space is composed of. The presence, or the absence, of topological obstructions in the parameter space does not depend on the particular loss function or the training data but is intrinsic to the interplay between the geometry and the topology of the parameter space under the action of groups of symmetries inducing observationally equivalent networks.Main contributions. Our main contributions are the following.1. We find that, for two-layer neural networks, the gradient flow trajectories lie on an invariant set, which can be factored as the product of quadric hypersurfaces. 2. We analytically compute its Betti numbers, i.e., the number of connected components, holes, and higher-dimensional cavities. 3. We find that the invariant set can be disconnected when the network's output dimension is 1, leading to a clear topological obstruction. 4 . We find that the obstruction is caused by "pathological" neurons that cannot change the sign of their output weights when trained with gradient flow. 5. We discuss the relation between the invariant set and the network's symmetries, finding that if we consider permutations, the number of effective connected components scales linearly in the number of pathological neurons. 6. We perform numerical validations on controlled toy scenarios, displaying the effect of obstruction in practice. 
related work: 
 A large body of work studies gradient flow and gradient descent optimization of one hidden layer networks with homogeneous activations. Convergence properties have been found for wide networks [37, 43] with bounded density at initialization [31] . The implicit regularization provided is studied under various assumptions on: orthogonal input data [4] , initialization scale [30, 4] , wide (overparameterized regime) and infinitely wide [6] , linearly separable data [30, 45] . Deeper linear networks [24] have also been studied.These works focus on proving convergence and understanding which (optimal) solution is found, whereas our work investigates the shape of the optimization space and focuses on cases where the optimum might not be reachable from a given initialization.Closer to our work, Safran et al. [39] studies two-layer ReLU binary classifiers with single input and output, counting the number of their piecewise-linear components after training. Eberle et al. [13] focuses on the differential challenge posed by the ReLU activation function and studies properties like the uniqueness of the solution of a gradient flow differential equation for a given initialization.ReLU activation is a nonnegative homogeneous function, meaning that particular weight rescalings do not change the neural network's function. This is at the heart of the counterargument to flatness measures made by Dinh et al. [10] , which shows that the Hessian eigenvalues can be made arbitrarily large in this way. Neyshabur et al. [34] explores the effect such rescalings can have on the gradient, proposing a rescaling-invariant regularization, and Pittorino et al. [36] employs them to define invariant flatness measures. Generally speaking, neural networks possess symmetries [20] , and symmetries influence the geometry of training. Du et al. [12] studies how symmetry leads ReLU networks to automatically balance the neurons' weights. Kunin et al. [27] , Zhao et al. [51] study how it constrains the gradient and Hessian matrix, leading to conservation laws w.r.t. gradient flow and Tanaka et al. [46] leverages it to propose a network pruning scheme. Ziyin [52] studies general mirror-reflect symmetries of the loss function and their effect on the weights of the trained network.Other conserved quantities stem from batch normalization's scale invariance [23, 47] . The transition from gradient flow to finite step size gradient descent breaks the conservation laws, resulting in altered trajectories [14, 2, 27, 44] .Numerous works have explored the geometry and topology of the loss landscape to obtain insight into a neural network's training behavior. Motivated by the striking experimental observation that low loss points can be connected by simple curves [11, 18] or line segments [40, 16, 15] , a large body of literature tries to understand this phenomenon of mode connectivity under the topological lens of the connectedness of the loss function's sublevel sets [17, 35, 26] , especially for overparameterized neural networks [9, 8, 42] . Another line of work approaches the connectivity of minima from another point of view, studying the presence [50, 38, 48] or absence [28] of spurious minima, i.e. minima which are not global. Bucarelli et al. [5] analytically derives bounds on the sum of the Betti numbers of the loss landscape's sublevel set. Topological data analysis methods have also been exploited to numerically study the shape of the loss landscape [1, 22] .3 Setup and preliminariesQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our analysis in Section 8. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
one-hidden layer neural network: 
 Unless otherwise stated, all vectors are column vectors, that is,EQUATIONwhere x ∈ R d is the input, θ = (W (1) , W (2) ) with W (1) ∈ R l×d and W (2) ∈ R e×l are the parameters, σ ∶ R → R is the component-wise activation function and l is the number of neurons in the hidden layer. Notice that we consider a network with no biases, as it allows us a discussion with lighter notation. The case with biases is discussed in Appendix E.In this work, following [12] , we focus on the case where σ is homogeneous, namely σ(x) = σ ′ (x) ⋅ x for every x and for every element of the sub-differential σ ′ (x) if σ is non-differentiable at x. The commonly used ReLU (σ(z) = max {z, 0}) and Leaky ReLU (σ(z) = max {z, γ} with 0 ≤ γ ≤ 1) activation functions satisfy this property.We call parameter space the vector space Θ = {θ = (W (1) , W (2) ) | W (1) ∈ R l×d , W (2) ∈ R e×l }.It will also be convenient to examine the single hidden neurons and their associated parameters for the following discussions. Proposition 1. For the two-layer neural network defined in Equation (1). Let k = 1, . . . , l, let (e 11 , e 12 , . . . , e ll ) be the canonical basis of R l×l and Θ k = {θ k = (e kk W (1) , W (2) e kk ) | (W (1) , W (2) ) ∈ Θ} ⊂ Θ, then Θ = Θ 1 ⊕ ⋯ ⊕ Θ l .Details of the proof are provided in Appendix A. Fixing k ∈ {1, . . . , l}, we can consider Θ k as the parameter space of the k-th hidden neuron, which consists of the inputs and output weights of neuron k, namely the rows and columns of W (1) and W (2) , respectively. For simplicity, when we work in Θ k , we write W (1) k ∶= e kk W (1) and W(2) k ∶= W (2) e kk . Interestingly, the decomposition of Proposition 1 only holds for two-layer neural networks and will be crucial to the formulations of this paper's results.It is well known that the properties of the activation function heavily influence the geometry of the parameter space Θ. The activation function's commutativity with some classes of transformations can result in the latter having no effect on the function implemented by the neural network. This means that, in general, the mapping from the parameter space to the hypothesis class of functions is not injective. Following the terminology in Dinh et al. [10] , we say that two parameters θ 1 , θ 2 ∈ Θ are observationally equivalent, if they encode the same function f (⋅; θ 1 ) = f (⋅, θ 2 ) and write θ 1 ∼ θ 2 .In the case of homogeneous activations (ReLU or Leaky ReLU), we describe two kinds of transformations that send a parameter θ into an observationally equivalent one.Neuron rescaling. The input weights of a hidden neuron can be rescaled by a positive scalar α > 0 provided that its output weights are rescaled by the inverse α -1 (top panel of Figure 1a ). We formalize this as the action of the group R + of positive real numbers on Θ k :EQUATIONThis action can be naturally extended to the space of all parameters by considering the possibility of rescaling all hidden neurons simultaneously by different factors. If α = (α 1 , . . . , α l ) ∈ R l + T α (θ) = (diag(α)W (1) , W (2) diag(α) -1 ). Given that σ(az) = aσ(z) when a ∈ R + , we see how θ ∼ T α (θ).We write T (θ) to denote the orbit of a parameter θ under the action of T , i.e. the set of all parameters obtained from θ by arbitrarily rescaling the neuronsT (θ) = {T α (θ) ∶ α ∈ R l + }.Permutations of the neurons. Besides rescaling, we can obtain an observationally equivalent network by permuting the hidden neurons in such a way as to preserve their input and output weights (bottom panel of Figure 1a ).Given the symmetric group on l elements S l of the permutations of {1, . . . , l}, we write the actionEQUATIONwhere R π is the l × l row-permutation matrix associated to the permutation π.Given that the activation function σ is applied component-wise, we have that it commutes with R π , namely 1) x) = W (2) σ(W (1) x) = f (x; θ) and thusEQUATIONP π (θ) ∼ θ because R ⊺ π = (R π ) -1 .Having defined these two actions, we say that θ and θ ′ are observationally equivalent by rescalings and permutations if θ ′ can be obtained from θ by a finite sequence of actions of T and P or, equivalently thanks to Lemma 3 in the Appendix, if there exists a rescaling α and a permutation π such that θ ′ = P π ○ T α (θ). In this case, we write θ rp ∼ θ ′ .The presence of symmetries in the neural network's parameter-function map results in a specific geometric structure in the loss landscape. Let indeedD = {(x i , y i ) ∈ R d × R e } N i=1be a training set of N input-output pairs and fix a loss function L ∶ Θ → R which depends on the parameters only through the output of the neural network (1) , that isEQUATIONwhere ℓ ∶ R e × R e → R is differentiable. In this work, as empirical risk minimization, we consider the continuous time version of the gradient descent (GD) algorithm (with learning rate h > 0)EQUATIONnamed gradient flow (GF), and defined asEQUATIONwhere ∇ θ L(θ(t)) is the Clarke sub-differential [7] which takes into account the parameters θ where L(θ) is non-differentiable. Given that the loss function L depends on the parameters only through f , its value at θ must be constant over the orbit T (θ). This, together with the fact that the gradient of a differentiable function at a point is orthogonal to the level set at that point, means thatEQUATIONat any parameter θ where L(θ) is differentiable, as represented in Figure 1b . This orthogonality condition constrains the possible values of the gradient and, by extension, the possible gradient flow trajectories. In particular, as proven in Liang et al. [29] , Tanaka et al. [46] , Equation ( 8) is equivalent toEQUATIONkiEQUATIONjk g(2)EQUATIONFor convenience of notation, we define, for k = 1, . . . , l, the following bilinear forms on Θ, which help us describe the geometry induced by the rescaling symmetry. If θ = (W (1) , W (2) ) and η = (V (1) , V (2) ), we defineEQUATIONjk V(2) jk (10) which, notice, only depends on the k-th row of W (1) and k-th column of W (2) meaning that we can equivalently see it as a bilinear form on Θ k . Θ k , together with ⟪⋅, ⋅⟫ k is a pseudo-Euclidean space.With the notation given by Equation (10) , we see that Equation ( 9) can be simply rewritten as ⟪θ, g(θ)⟫ k = 0 for every neuron k. This condition, akin to orthogonality w.r.t. the bilinear form of Equation (10) , implies that, under gradient flow optimization,EQUATIONThis result, first obtained in Saxe et al. [41] for linear networks and discussed in Du et al. [12] , Liang et al. [29] , Kunin et al. [27] , tells us that the rescaling symmetry results in the quantities ⟪θ, θ⟫ k being conserved. This means that the difference between the Euclidean norm of the inputs and the outputs is constant for each neuron throughout the GF training trajectory. Moreover, under the condition of homogeneity of the activation function, Du et al. [12] proves that Equation ( 11) holds even at non-differentiable points of L and in the case of multiple layers.Invariant sets. Assume that at the initialization θ 0 we have ⟪θ 0 , θ 0 ⟫ k = c k , for all k, then Equation (11) implies that the GF trajectory will lie on the set characterized by the system of equations ⟪θ, θ⟫ k = c k for k = 1, . . . , l. This subset is mapped to itself under the GF dynamics by Equation (11) (see Figure 1b ) and constitutes the main object of our study. Definition 1 (Invariant set). Given c = (c 1 , . . . , c l ), we call invariant set the subset H(c) ⊆ Θ given by the equations ⟪θ, θ⟫ k = c k ∀k = 1, . . . , l.If we look at each single equation (i.e. to each hidden neuron), we see that Equation (11) can be written asEQUATIONwhich corresponds to a hyperquadric (or quadric hypersurface) in Θ k . We denote with Q(c k ) ⊆ Θ k this hypersurface and call it the invariant hyperquadric associated to the k-th hidden neuron.Here c k ∈ R takes the role of a label associated with the k-th hidden neuron, which, we see in the next section, plays a key role in specifying the shape of Q(c k ). Figure 2a shows how, for d = 2 and e = 1, Q(c k ) is an hyperboloid with 1 sheet (connected) if c k > 0 and 2 sheets if c k < 0.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Given the simplicity of our setup, we think that releasing the code would be unnecessary.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
topology of the invariant set: 
 As we discussed above, Equation (11) tells us that gradient flow trajectories can't explore the whole space Θ but are constrained to lie on the invariant set H(c). The values of c, in turn, depend on the initialization and, we see from Equation (12) , quantify the balance between the norms of input and output weights in every hidden neuron.The goal of this section is to provide a topological characterization of H(c) that can tell us something about the presence or absence of fundamental obstructions to the network's training process. With obstruction, we mean the impossibility of a GF trajectory to travel freely from one point to the other in H(c). We refer the reader to Appendix B for an essential overview of some of the topological concepts that we rely on in the next paragraphs.Counting high-dimensional holes. Our topological characterization will be framed using Betti numbers. Betti numbers are well-known topological invariants given by a sequence of natural numbers that intuitively encode the number of higher-dimensional holes and cavities present in space.In particular, the 0-th Betti number of a space X, β 0 (X) corresponds to the number of connected components of X and thus will be fundamental for our goal of identifying obstructions.The invariant set H(c) is given as the set of solutions of l polynomial equations of degree 2 sharing no variables. Furthermore, in the setting of two-layer neural networks, we can leverage the fact that the parameter space can be decomposed into the parameter spaces of the hidden neurons. This, in turn, allows us to decompose the invariant set as the product of the neurons' invariant hyperquadrics, greatly simplifying our study. Lemma 1. In a two-layer ReLU neural network, the invariant set H(c) is homeomorphic to the Cartesian product of the hidden neurons' invariant hyperquadrics, that isEQUATIONLemma 1 tells us that we can understand the topology of H(c) by studying independently its factors. Moreover, the hyperquadrics we encounter here are well-studied objects for which the next proposition (proven in Appendix D.2) gives a topological characterization.Proposition 2. If c k > 0, Q(c k ) is a topological manifold homeomorphic to R e × S d-1 . If c k < 0, Q(c k ) is a topological manifold homeomorphic to R d × S e-1 . If c k = 0, Q(0) is a contractible space.Leveraging the decomposition of Lemma 1 and the characterization of the factors given by Proposition 2, we can explicitly compute all the Betti numbers of the invariant set. We give the next result in terms of the Poincaré polynomial of H(c), namely the polynomial whose coefficients are the Betti numbers (see Appendix B). Theorem 1. Let l + , l -, l 0 be the number of positive, negative, and zero components of c, respectively. The Poincaré polynomial of H(c) is given byEQUATIONThis result, which is proven in Appendix D.3, contains a wealth of topological information as it gives us the exact number of holes and cavities of any order, depending on the network's hyperparameters (d, e) and initialization (l + , l -). In the rest of this work, we focus only on the 0-th Betti number as the non-connectedness of H(c) provides a clear obstruction to the GF trajectories.Connectedness of the invariant set. With regard to the connectedness of H(c), we can leverage Theorem 1 to obtain the exact number of connected components. Corollary 1. The 0-th Betti number β 0 of H(c), corresponding to the number of its connected components, is given byβ 0 = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ 1 if d, e > 1 2 l+ if d = 1, e > 1 2 l- if d > 1, e = 1 2 l++l-if d = 1, e = 1 (15)Proof. This can be directly obtained from the coefficient of degree 0 of the Poincaré polynomial obtained through Theorem 1. What we see in Equation ( 15) is that in most cases, the invariant set is connected, and gradient flow has no topological limitations in exploring the whole of H(c). Instead, when the hidden neurons have only one input or only one output, the space is fragmented into several components whose number scales exponentially in l + or l -, respectively.Let us focus on the more interesting case where d > 1 and e = 1. Corollary 2. If the output of a two-layer ReLU neural network is a single scalar e = 1, its input has dimension d > 1, and the initial parameter θ 0 is such that ⟪θ 0 , θ 0 ⟫ k < 0 for l -> 0 hidden neurons, then the set H(c) is disconnected and has 2 l-connected components.This means that neurons initialized with the norm of their outgoing weight strictly greater than their incoming weights' norm are responsible for disconnecting the space. We now precisely identify which connected component a parameter θ belongs to and clarify the meaning of the obstruction. k l-) ∈ R 1×l-be the row vector whose components are the components of W (2) ∈ R 1×l associated to c k < 0. Then the vector s(θ) = (sign(W(2) k1 ), . . . , sign(W (2) k l-))identifies uniquely the component θ belongs to, namely: θ and θ ′ belong to the same connected component of H(c) if and only if s(θ) = s(θ ′ ). Proposition 3, proven in Appendix D.4, implies that s(θ) does not change when we move in C on a continuous curve such as the one given by gradient flow. This gives us an interesting interpretation of the topological obstruction: gradient flow cannot change the signs of the outgoing weights of the hidden neurons k such that c k < 0 (see Appendix G for an intuitive explanation of the phenomenon). This same observation is also mentioned in Boursier and Flammarion [3] . Proposition 3 extends one of the results of Boursier et al. [4] which proves that the same also holds when c k = 0 (balanced initialization).By also considering Corollary 1, one obtains that a clever initialization of the parameters given by ⟪θ 0 , θ 0 ⟫ k = c k > 0 ∀k = 1, . . . , l can prevent the issue by ensuring the connectedness of the invariant set. We also find that under common initialization schemes such as Xavier [19] and Kaiming [21] the probability of having pathological neurons is negligible when the input dimension and number of hidden neurons is high (see Appendix F). 
taking symmetries into account: 
 Corollary 2 states that neurons k such that c k < 0 are "pathological", in the sense that they are responsible for disconnecting the invariant set into several components, whose number scales exponentially in the number of those neurons. This result gives us a grim picture of the possibility of actually optimizing the neural network: if the initial parameter θ 0 is in a particular connected component and the global optimum θ * lies in another, then any gradient flow trajectory will not be able to reach θ * because it will be constrained in its connected component. This result, however, provides us only with a partial picture of the parameter space's geometry. It is a priori possible that the training trajectory, moving in its connected component, reaches a parameter ζ, which itself is optimal as it is observationally equivalent to θ * (ζ ∼ θ * ). In this case, the topological obstruction given by the non-connectedness would be only apparent.To take this fact into account, we define the following notion. Definition 2 (Effective component). Let θ ∈ H(c) and C(θ) be its connected component therein. We define its effective component Eff(θ) as the union of the connected component of all θ ′ such thatθ ′ rp ∼ θ. So that Eff(θ) ∶= ⋃ θ ′ rp ∼ θ C(θ ′ ).Figure 2b gives a picture which clarifies the definition, showing a space with 4 connected components that has only 3 effective components. If the optimum θ * belongs to the same effective component as the initialization, then it is possible to reach a parameter that is observationally equivalent to it (through permutations and rescalings).We present a useful result which tells us that the action of rescaling of Equation ( 3) can take any non-degenerate parameter θ ∈ H(c) to any other invariant set H(c ′ ) for every c ′ ∈ R l . This means that any invariant set can realize all the neural network's functions.Proposition 4. For every c k ∈ R and for everyEQUATION≠ 0, there exists aunique α k ∈ R + such that T α k (θ k ) ∈ Q(c k ). If W (1) k = 0 and W (2) k ≠ 0, then the same holds for every c k < 0, while, if W (1) k ≠ 0 and W (2) k = 0, it holds for every c k > 0.The proof can be found in Appendix D.5 with the formula of the specific α which realizes the rescaling.The following theorem leverages the power of Proposition 4 to give necessary and sufficient conditions for θ and θ ′ to belong to the same effective component. Theorem 2. Let d > 1 and e = 1. Let c ∈ R l and l -be the number of neurons such that c k < 0.Assume that l -≥ 1. Let C, C ′ ⊆ H(c) be two distinct connected components of H(c) such that s(θ) = s, ∀θ ∈ C, and s(θ ′ ) = s ′ , ∀θ ′ ∈ C ′ . Then, the following statements are equivalent: 1. for every θ ∈ C there existsθ ′ ∈ C ′ such that θ rp ∼ θ ′ ; 2. l- ∑ i=1 s i = l- ∑ i=1 s ′ iThe theorem, proven in Appendix D.6, tells us that, while connected components are identified by s, the effective components are identified only by the values of ∑ i s i or, equivalently, by the distribution of ±1 in s. Therefore, we find that the number of effective components scales much slower than the exponential growth of the number of connected components given by Corollary 1. Corollary 3. The number of effective components of H(c) is given by 1 + l -.Proof. Theorem 2 tells us that two connected components C, C ′ belong to the same effective component if and only if their associated sign vectors s, s ′ ∈ {-1, 1} l-have the same sum. The number of effective components will thus equal the number of different values that the sum ∑ l-i=1 s i can have. Ifs i = 1 ∀i then ∑ l- i=1 s = l -.Each switch of a component to -1 decreases the sum's value by 2 until it reaches the minimum -l -. Therefore, the total number of values of the sum will be 1 + l -. 
empirical validation: 
 Task, dataset, and model setup. We display here a toy example, showing how the initialization of the model can cause a topological obstruction, making the optimum unreachable.We consider the function F (x 1 , x 2 ) = -(x 1 + x 2 ), which will be our ground-truth. Next, we generate a dataset of 8000 points (x i , F (x i )) by samplingx i ∼ U ([0, 1]2 ). Our model, depicted in Figure 3a ) is a one hidden layer neural network with 2 hidden neurons, ReLU activations and no biases. All the weights are initialized by independently sampling fromEQUATION). From the task and the network's architecture, it is clear that at least one of the output weights has to be negative to approximate F correctly.To standardize our results, we apply the rescaling of Proposition 4 and relocate the initial parameters to an observationally equivalent one in the invariant set H(c) with c k ∈ {-0.1, 0.1}, controlling the sign of the weights on the last layer. We allow ourselves to do these two manipulations to control the experiments while only marginally modifying the network initialization, avoiding the introduction of massively unbalanced weights, which could change the dynamics, as shown in Neyshabur et al. [34] . Finally, we train the network using gradient descent on the MSE loss with a small learning rate of h = 0.01. This limits the variations of c k values to less than one percent along training, giving us a good approximation of gradient flow.Results. We initialize different models and collect all states and losses. First, when we initialize the model with an "unlucky" configuration, namely c = (-0.1, -0.1) (the space has 4 connected components) and s(θ) = (+1, +1), we find that the trajectories are confined to the positive region of their invariant hyperquadric, resulting in a poor approximation of F , as we can see in Figure 3b (top) and in the loss of Figure 3c . Instead, with an initial configuration such that c = (-0.1, +0.1) (2 connected components) and s(θ) = (+1, +1), the model can leverage the connectedness of Q(c 2 ) to learn F by flipping the sign of the second neuron's output weight (Figure 3b A more realistic experiment. We present here a further experiment to show how the topological obstruction can be a hindrance in a more realistic setting. We consider a simple binary classification task on the well-known breast cancer dataset [49] , which we try to solve by fitting a one-layer ReLU neural network trained to minimize the BCE loss. We vary the number of hidden neurons l and, for each l, we change the number of non-pathological neurons l + (neurons with c k > 0) from 0 to l. We repeat the experiment with 100 different random initializations and show how the model's average performance changes when the degree of disconnectedness of its invariant set is varied. The result, on the left panel of Figure 4 , clearly shows the presence of a "gradient" in performance, where increasing the number of non-pathological neurons decreases the average value of the test loss after training. The right panel of Figure 4 , moreover, shows how the impact of the obstruction depends on the number of non-pathological neurons and not on their fraction over the total number of hidden neurons.  
conclusions: 
 In this paper, we have given analytical results that clarify the nature of the constraints imposed by gradient flow on the parameter space of a two-layer neural network with homogeneous activations.In the case of a single scalar output, which appears in tasks such as binary classification and scalar regression, we identified initial conditions that lead to a topological obstruction in the form of the parameter space's fragmentation into multiple connected components. This is caused by pathological neurons whose output weights cannot change their sign during training. Moreover, if one also considers the network's symmetries under permutations of the hidden neurons, we find that most of the connected components are equivalent. The number of effective components of the resulting space scales linearly with the number of pathological neurons, contrasting with the exponential growth of the number of connected components obtained without considering the permutation symmetries.As shown in the last numerical experiment, the lack of non-pathological neurons hinders learning, even when the network's width is scaled. Our probabilistic analysis outlined in Appendix F, however, shows that with common initialization schemes, the probability of creating a pathological neuron decreases rapidly with increased inner layer width. Therefore, the combination of specific initialization schemes and a large number of hidden neurons (beyond the minimum required to solve a task) appears to make this obstruction unlikely in practice. This work describes a simple safeguard to avoid obstructions, which can, for instance, discourage the usage of initialization schemes that result in the proliferation of pathological neurons. 
limitations: 
 The main limitation of the work is the network's architecture, which is limited to only one hidden layer. Considering multiple layers, we can still define rescalings and permutations and find invariant hyperquadrics for each hidden neuron. The issue emerges in the fact that these hyperquadrics are not "independent" anymore, and the invariant set cannot be factored into the product of the Q(c k ). This intuitively results from the fact that in the multi-layer case, each weight in the hidden layers is shared by two neurons.The second limitation is that our study focuses on gradient flow optimization. This idealized situation doesn't take into account the fact that moderate step size of gradient descent and stochastic gradient descent can break the conservation of ⟪θ, θ⟫ k and make the parameters drift away from the invariant set [2] . Moreover, popular optimizers like ADAM [25] update the parameters employing the gradients at previous iterations so their trajectories will not be constrained to lie on H(c) as we defined it.The inclusion of regularization terms in the loss function, such as ℓ p regularizations, also breaks the invariance to rescalings.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [No] Justification: Our experiment was performed on a simple laptop.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We reviewd the NeurIPS Code of Ethics and confirm that we conform to it in every repect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Being a purely theoretical work, we believe that it doesn't havesignificantt social impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We release no data or models that have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: We use no existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Let (e 11 , e 12 , . . . , e ll ) and (e 1 , . . . , e l ) be the canonical bases of the spaces R l×l and R l , respectively and Θ k = {(e kk W (1) , W (2) e kk ) | (W (1) , W (2) ) ∈ Θ} ⊂ Θ.Θ k , notice, is the subspace of Θ consisting of the weight matrices W (1) with null rows except for the k-th one, and weight matrices W (2) with null columns except for the k-th one. We can check that Θ = Θ 1 ⊕ ⋯ ⊕ Θ l , because, if I l is the l × l identity matrix, ∑ e kk = I l so that (W (1) , W (2) EQUATIONk , Wk ) = ∑ k (e kk W (1) , W (2) e kk ).Moreover, Θ ≅ Θ 1 × ⋯ × Θ l via the linear isomorphism θ = (W (1) , W (2) ) ↔ (θ k ) l k=1 = ((e kk W (1) , W (2) e kk )) l k=1 . This, we see, is equivalent to decomposing the neural network of Equation ( 1) into the computations of the single hidden neurons. Indeed, let f (x; θ k ) ∶= f (x, (e kk W (1) , W (2) e kk )), then, considering that e kk e kk = e kk and that σ(e kk v) = e kk σ(v), it holds that f (x; θ k ) = W (2) e kk σ(W (1) x) ∀k = 1, . . . , l.Therefore ∑ k f (x; θ k ) = f (x; θ).Here, we recall some basic facts about the topology required to understand the paper's results. A self-consistent introduction is outside this work's scope, so we refer the interested reader to more complete expositions in Munkres [33, 32] .Topological manifold. An n-dimensional topological manifold is a topological space X which locally looks like the Euclidean space R n . More formally, for each p ∈ X, there exists a neighbourhood U of p and a homeomorphism mapping U to an open subset of R n . Contractible space. A topological space X is contractible if it can continuously deform to a point p ∈ X. This means that there exists a continuous mapF ∶ X × [0, 1] → X such that F (x, 0) = x and F (x, 1) = p for every x ∈ X.Betti numbers. Betti numbers formalize the notion of the hole in a topological space and extend it to describe higher-dimensional cavities. The general idea is that one can associate a sequence of Abelian groups named homology groups to any space X, which encodes rich information about the higher-dimensional cavities in X. For what we are concerned here, the rank of the k-th homology group is called the k-th Betti number β k (X). β k (X) counts the number of k-dimensional holes in the space: β 0 (X) count the number of connected components, β 1 (X) the number of "circular" holes and β 2 (X) the number of voids or cavities.A contractible space X is connected and cannot have any holes, and thus its Betti numbers are β 0 (X) = 1 and β i (X) = 0 ∀i > 0.Betti numbers are topological invariants, meaning they are preserved when a space is transformed via a homeomorphism, namely a bijective, continuous map with continuous inverse.Poincaré polynomials. The Poincaré polynomial of a topological space X is the polynomial whose k-th coefficient is given by the k-th Betti number p X (x) = β 0 (X) + β 1 (X)x + β 2 (X)x 2 + . . . . Künneth formula. Künneth's theorem describes assembling the homology groups of a Cartesian product of spaces X × Y from the homology groups of the factors X, Y . One of its corollaries tells us that if we care only about Betti numbers, a simple relation holds between the Poincaré polynomials of X × Y and the ones of X and Y , namely,p X×Y (x) = p X (x)p Y (x).Types of connectedness. In topology, there are several kinds of connectedness. Two of them are particularly important for this work.1.) A topological space X is connected if it cannot be divided into two disjoint non-empty open sets. If it is not connected, the connected component of a point x ∈ X is given by the union of all connected subsets of X which contain x.A topological space equal to the Cartesian product of two spaces X = Y × Z is connected if and only if Y and Z are both connected.2.) A topological space is path-connected if, for every pair of points x, y ∈ X, there exists a continuous curve γ ∶ [0, 1] → X such that γ(0) = x, γ(1) = y. The path-component of x is the set of all y ∈ X such that a continuous curve exists connecting x to y. This second notion is more relevant to our setting, where we care about the possible destinations of the optimization trajectories.Path-connectedness implies connectedness, but not the opposite. There are situations, however, where these two notions are equivalent. For example, when X is a topological manifold, X is connected if and only if X is path connected.Notice that the 0-th Betti number β 0 (X) counts the number of connected components but, in general, not the number of path components. With Lemma 2, we prove that these two notions are equivalent for our object of study. Let us focus on a particular Q(c k ).When c k ≠ 0, Proposition 2 tells us that Q(c k ) is a topological manifold, and thus, it is connected if and only if it is path connected.When c k = 0, Q(0) is not a topological manifold but contractible, implying that it is connected. Let us prove that it is also path-connected.Let θ k , θ ′ k ∈ Q(0) and define the curve γ ∶ [0, 1] → Q(0) γ k;θ (t) = t ⋅ θ k such that γ k;θ (0) = θ, γ k;θ (1) = 0. γ k;θ (t) ∈ Q(0) for every t ∈ [0, 1] because ⟪γ k;θ (t), γ k;θ (t)⟫ k = t⟪θ, θ⟫ k = 0.Therefore, the segment from θ k to 0 belongs to Q(0).A continuous curve from θ k to θ ′ k can be thus obtained byγ k;θ ′ γ k;θ (t) ∶= { γ k;θ (2t) if t ∈ [0, 1 2 ] γ k;θ ′ (2 -2t) if t ∈ [ 1 2 , 1] which is continuous because γ k;θ (1) = γ k;θ ′ (1) = 0. Therefore Q(0) is path connected.Finally, if H(c) is connected, then all of its factors Q(c k ) are connected, which, in turn, is true if and only if they are path-connected. A product of path-connected space is again path-connected, and therefore H(c) is path-connected. The other implication is true because path-connectedness implies connectedness, thus concluding the proof.EQUATIONProof. Given thatT α P π (θ) = (diag(α)R π W (1) , W (2) R ⊺ π diag(α) -1 ) we need to prove that diag(α)R π = R π diag(α). (diag(α)R π ) ij = l ∑ k=1 diag(α) ik (R π ) kj = α i (R π ) ij = { α i if j = π(i) 0 otherwise . Let us pick a generic α ∈ R l + . (R π diag(α)) ij = l ∑ k=1 (R π ) ik diag(α) kj = (R π ) ij αj = { αj if j = π(i) 0 otherwise .Let us consider the inverse permutation π -1 so that π -1 (j) = i if π(i) = j. Then, if α = R π -1 α, αj = α π -1 (j) = α iand thus we get that diag(α)R π = R π diag(α).Proof. Proposition 1 tells us that the invariant set can be decomposed as the direct sum of the single hidden neurons' parameter spaces. This means that, for every θ ∈ Θ, there exist uniqueθ 1 ∈ Θ 1 , . . . , θ l ∈ Θ l such that θ = θ 1 + θ 2 + ⋯ + θ l .Therefore, we have a linear isomorphismφ ∶ Θ 1 × ⋯ × Θ k → Θ φ ∶ (θ 1 , . . . , θ l ) ↦ θ 1 + ⋯ + θ l = θ.The invariant set is a subset of Θ, which is given as the set of solutions of l equations ⟪θ, θ⟫ k = c k k = 1, . . . , l. Notice that each of these equations involves a set of variables that appear only in that particular equation. These variables are exactly the ones which belong to Θ k . In fact ⟪θ,θ⟫ k = ⟪θ k , θ k ⟫ k . Therefore, given θ 1 ∈ Q(c 1 ), . . . , θ l ∈ Q(c l ) we have that φ(θ 1 , . . . , θ l ) = l ∑ k=1 θ k ∈ H(c).On the opposite, given θ ∈ H(c) we have thatφ -1 (θ) = (θ 1 , . . . , θ l ) ∈ Q(c 1 ) × ⋯ × Q(c l ).Therefore, H(c) is in bijection with Q(c 1 ) × ⋯ × Q(c l ) through φ which, being a linear isomorphism, implies also that H(c) and Q(c 1 ) × ⋯ × Q(c l ) are homeomorphic.Proof. Let us consider the three cases separately. Ifc k > 0, Q(c k ) is defined by the equation d ∑ i=1 (W (1) ki ) 2 - e ∑ j=1 (W (2) jk ) 2 = c k ⇐⇒ ∥W (1) k ∥ 2 F -∥W (2) k ∥ 2 F = c k ,where ∥⋅∥ F is the Frobenius norm of a matrix, i.e., the square root of the sum of the squares of its elements. This can be rewritten asEQUATION)whereEQUATIONEQUATIONwhere, notice, the first component belongs to the sphere S d-1 because of Equation ( 17) and W(2) k∈ R e . This map is bijective, differentiable and has the following inverse h-1 ∶ S d-1 × R e → Q( k ) h -1 (u, x) = ( √ c k + ∥v∥ 2 F u, x) which is differentiable. Therefore, h is a diffeomorphism from Q(c k ) to S d-1 × R e . If c k < 0, we write the equation of Q(c k ) as ∥W (2) k ∥ 2 = √ -c k + ∥W (1) k ∥ 2 2where W(1) k and W(2) k have switched their role to guarantee the term on the right to be positive. The diffeomorphism is now built analogously to Equation (18) as a map h ∶ Q(c k ) → S e-1 × R d .If c k = 0, we prove that Q(0) is a contractible space. To do that, we exhibit a homotopy equivalence between Q(0) and the point 0, i.e. a continuous map p ∶[0, 1]×Q(0) → Q(0) such that p(0, θ k ) = θ k and p(1, θ k ) = 0 ∀θ k ∈ Q(0).The map is defined in the following way:p(λ, θ k ) = (1 -λ)θ k . This is continuous and well-defined because ⟪p(λ, θ k ), p(λ, θ k )⟫ k = ⟪(1 -λ)θ k , (1 -λ)θ k ⟫ k = (1 -λ) 2 ⟪θ k , θ k ⟫ k = 0, meaning that p(λ, θ k ) ∈ Q(0) for every θ k ∈ Q(0) and for every λ ∈ [0, 1].Proof. An implication of the Künneth formula is that the Poincaré polynomial of the Cartesian product of two spaces is equal to the product of their Poincaré polynomials:p X×Y (x) = p X (x)p Y (y).Starting from Proposition 2, we can apply this result to Q(c k ).EQUATION)because a contractible space has 1 connected component and all of its other Betti numbers equal to zero.Moreover, we know that R n is contractible for any n and its Poincaré polynomial is p R n (x) = 1. The Poincaré polynomial of the sphere S n is given by p S n (x) = 1 + x n .Equation (19) becomesEQUATION)Given that Lemma 1 tells us that H(c) can be factored into the product of the Q(c k ), we apply Künneth formula and find that pEQUATIONProof. In the following, we exploit Lemma 2 and use the terminology connected and path-connected interchangeably.Let us first prove that s(θ) = s(θ ′ ) means that θ and θ ′ belong to the same connected component. We do this by explicitly building a continuous curve δ ∶ [0, 1] → H(c) such that δ(0) = θ and δ(1) = θ ′ .Let us proceed by leveraging the homeomorphism from H(c) and Q(c 1 )×⋯×Q(c l ) and consider the different components of δ in the invariant hyperquadrics associated to each neuron δ = (δ 1 , . . . , δ l ).If c k ≥ 0, we know that Q(c k ) is path-connected and therefore we fix δ k (t) to any continuous curve inQ(c k ) such that δ k (0) = θ k and δ k (1) = θ ′ k . If c k < 0 for k ∈ K ∶= {k 1 , . . . , k l-} ⊆ {1, . . . , l}, we define the curve γ i;θ ∶ [0, 1] → Q(c ki ) with γ i;θ (t) = ⎛ ⎝ (1 -t)W (1) ki1 , . . . , (1 -t)W (1) kil , s(θ) i √ -c ki + (1 -t) 2 ∥W (1) ki ∥ 2 F ⎞ ⎠for every i = 1, . . . , l -.γ i;θ is a continuous curve which connects the point γ i;θ (0) = θ ki with γ i;θ (1) = (0, s(θ) i √ -c ki ).If we define γi;θ (t) ∶= γ i;θ (1t), which is the same curve as γ i;θ but traversed in the opposite direction, we can define the curve δ ki (t) = γi;θ ′ γ i;θ (t) i.e. the curve which travels on γ i;θ for t ∈ [0, 1 2 ] and on γi;θ ′ for t ∈ [ 1 2 , 1], for i = 1, . . . , l -. Notice now that δ ki (0) = θ ki and δ ki (1) = θ ′ ki . Moreover δ ki is continuous becauseγ i;θ (1) = (0, s(θ) i √ -c ki ) = (0, s(θ ′ ) i √ -c ki ) = γi;θ ′ (0)under the hypothesis that s(θ) = s(θ ′ ).Finally, we found a continuous curve δ = (δ 1 , . . . , δ l ) such that δ(t) ∈ H(c) ∀t ∈ [0, 1] and δ(0) = θ, δ(1) = θ ′ . Therefore, θ and θ ′ belong to the same connected component.Let us now prove that if θ and θ ′ belong to the same connected component, then s(θ) = s(θ ′ ).Let γ ∶ [0, 1] → H(c) be a continuous curve in H(c) such that γ(0) = θ and γ(1) = θ ′ .For each k ∈ K = {k 1 , . . . , k l-} such that c k < 0, we know that γ k (t) ∈ Q(c k ) means thatEQUATIONEQUATIONkd (t), s k (t) √ -c k + ∥γ (1) k ∥ 2 F γ (2) k (t))for some function s k (t) ∈ {-1, 1} such that s k (0) = s(θ) k and s k (1) = s(θ ′ ) k .Assume, by contradiction, that s(θ) k = -s(θ ′ ) k . Assume also that s(θ) k = +1 and s(θ′ ) k = -1. Notice that c k < 0 implies that p(t) ∶= √ -c k + ∥γ (1) k ∥ 2 F > 0.The function γEQUATIONk (0) > 0 and γk (1) < 0 and thus, by the intermediate value theorem, there exists t * ∈ (0, 1) such that γ k (t * ) = 0. But γ (2) k (t) ≠ 0 for every t, as s k (t) ∈ {-1, 1} and √ -c k + ∥γ (1) k ∥ 2 F > 0.Repeating the argument for s(θ) k = -1 we then prove by contradiction that s(θ) k = s(θ ′ ) k for all k ∈ K, thus concluding the proof.Proof. We have by Equation (10) and Equation ( 2):EQUATIONjk ) 2 -c k = 0 By renaming A = d ∑ i=1 (W (1) ki ) 2 = ∥W (1) k ∥ 2 F , C = e ∑ j=1 (W (2) jk ) 2 = ∥W (2) k ∥ 2 Fand multiplying by α 2 k > 0 we have:EQUATIONSolving for α 2 k gives us:∆ = c 2 k + 4AC ≥ 4AC > 0 α 2 k = c k ± √ ∆ 2AGiven that we want α k > 0, we discard the negative solution. The other is positive because∆ > c 2 k and thus √ ∆ > |c k |. α k = ± √ c k + √ ∆ 2AOf which we keep the positive solution only, with its full expression being:EQUATIONEQUATIONEQUATIONHence, if α = (α 1 , . . . , α l ) with α k given by Equation ( 23), we get that ⟪T α (θ),T α (θ)⟫ k = c k ∀k = 1, . . . , l.Let us consider now the pathological cases WEQUATION≠ 0 then A = 0, C ≠ 0. Therefore, we have that Equation (22) becomes-c k α 2 k -C = 0which has solutions if and only if c k < 0. In that caseα k = ∥W (2) k ∥ F √ -c k .This means that a hidden neuron with zero input weights and nonzero output weights can be rescaled only to the invariant hyperquadrics with c k < 0.EQUATION≠ 0 then C = 0, A ≠ 0. Therefore, we have that Equation ( 22) becomesEQUATION. This means that a hidden neuron with zero output weights and nonzero input weights can be rescaled only to the invariant hyperquadrics with c k > 0.If W (1) k = 0 and W (2) k = 0, then θ k = 0 ∈ Q(0)and it cannot be rescaled to any other invariant hyperquadric.Proof. Let us first prove that, if for every θ ∈ C there exists a θ′ ∈ C ′ such that θ rp ∼ θ ′ then l- ∑ i=1 s(θ) i = l- ∑ i=1 s(θ ′ ) i .First, Lemma 3 tells us that we can interchange rescaling and permutation if we permute the rescaling factors accordingly. This means we can reduce any composite action of rescalings and permutations to the action of a single rescaling and a single permutation.Let θ ∈ C and θ ′ ∈ C ′ such that θ rp ∼ θ ′ . Then there exist α ∈ R l + and π ∈ S l such thatEQUATIONThis means that T α P π (θ) and θ ′ belong to the same invariant set and, specifically, to the same connected component. Therefore,s(T α P π (θ)) = s(θ ′ ).Notice that s(T α P π (θ)) = s(P π (θ))because T α does not change the sign of W (2) as it acts by scaling it by positive factors. Let us focus on s(P π (θ)) = sign((W (2) R ⊺ π ) -). If the neurons of θ such that c k < 0 are indexed by k 1 , k 2 , . . . , k l-, we will have that the neurons of P π (θ) such that c k < 0 are indexed by π(k 1 ), π(k 2 ), . . . , π(k l-). Therefores(P π (θ)) i = sign(W (2) π(ki) ) = (s(θ)R ⊺ π-) ifor some permutation π -∈ S l-. Therefore, Equation (24) implies thats(θ ′ ) = s(P π (θ)) = s(θ)R ⊺ π-.The action of rescaling and permutation can only reshuffle the label s of the connected component. This means thats(θ)R ⊺ π-= s(θ ′ ) ⇒ l- ∑ i=1 (s(θ)R ⊺ π-) i = l- ∑ i=1 s(θ ′ ) i ⇒ l- ∑ i=1 s(θ) i = l- ∑ i=1 s(θ ′ ) i .Let us now prove the other implication. Let s, s′ ∈ R l-such that l- ∑ i=1 s i = l- ∑ i=1 s ′ i .Given that their sum is equal, s and s ′ have the same number of +1 and -1 and thus there exists a permutation π -∈ S l-such that s ′ = sR ⊺ π-. Let π ∈ S l be the permutation which permutes the neurons such that c k < 0 according to π -and leaves the others fixed. In this way s(P π (θ)) = sR ⊺ π-= s ′ . P π (θ), however, doesn't belong to H(c) but to another invariant set given by H(R π c).Applying Proposition 4 we can find a rescaling α = α(π) ∈ R l + such that T α P π (θ) ∈ H(c). Since neurons such that c k ≥ 0, are left unchanged by the permutation, we can apply the proposition and we rescale them with α k = 1. The permuted neurons are the ones such that c k < 0, namely the ones whose weights satisfy ∥W(1) k ∥ 2 F -∥W (2) k ∥ 2 F < 0, meaning that W (2) k ≠ 0.As noted above, the action of the rescaling doesn't change the sign vector, and thuss(T α P π (θ)) = s(P π (θ)) = s ′ .If we name θ ′ ∶= T α P π (θ) this result means that we found a θ ′ rp ∼ θ such that θ ′ ∈ C ′ , thus concluding the proof.Let us consider the case where we include biases. The resulting two-layer neural network can be written as f (x; θ) = W (2) σ(W (1) x + b (1) ) + b (2) ,where b (1) ∈ R l and b (2) ∈ R e .To work with this extended set of parameters, we re-define the space Θ = {θ = (W (1) , b (1) , W (2) , b (2) )} and the single hidden neuron spacesEQUATIONk , e kk b(1)EQUATIONk e kk )}where the second bias term b (2) does not appear because it is not directly involved with the computations of the hidden neurons. This means that we can writeΘ ≅ Θ 1 × ⋯ × Θ l × R ewhere R e is included to describe the parameters in b (2) .The neuron rescaling action now acts on the biases b (1) as well as the weights:EQUATIONk , αbk ,EQUATIONand can be extended to the whole space of parameters 1) , diag(α)b (1) , W (2) diag(α) -1 , b (2) ).T ∶ R l + × Θ → Θ (α, θ) ↦ T α (θ) = (diag(α)W ((EQUATIONOnce again, we find that T α θ ∼ θ.In this more general case, we can rewrite the bilinear form to include the biases. If θ = (W (1) , b (1) , W (2) , b (2) ) and η = (V (1) , p (1) , V (2) , p (2) ), we defineEQUATIONki + bkEQUATION(2) jk (28) and see that, once gradient flow optimization, we have a conservation condition like the one of Equation ( 9) ⟪θ(t), θ(t)⟫ k = c k ∀t > 0 ∀k = 1, . . . , l.Once again, we call Q(c k ) the hypersurface of Θ k which satisfies the equation ⟪θ, θ⟫ k = c k and H(c k ) the set in Θ defined by ⟪θ, θ⟫ k = c k ∀k = 1, . . . , l.With this in mind, it is not hard to extend the results of Proposition 2 and Theorem 1 which turn out to be slightly modified.Proposition 5. If c k > 0, Q(c k ) is a topological manifold homeomorphic to R e × S d . If c k < 0, Q(c k ) is a topological manifold homeomorphic to R d × S e-1 . If c k = 0, Q(0) is contractible.In this case, we can factor the space of parameters H(c) asH(c) ≅ Q(c 1 ) × ⋯ × Q(c l ) × R e ,where the last factor is due to the freedom in choosing the values of b (2) . Proposition 6. Let c k ≠ 0 ∀k = 1, . . . , l. Let l + , l -, l 0 be the number of positive, negative and zero elements of c, respectively. The Poincaré polynomial of H(c) is given byEQUATIONCorollary 4. The 0-th Betti number β 0 (c) of H(c), corresponding to the number of its connected components, is given byEQUATIONThe result we obtain is similar to Corollary 1 although slightly modified by the fact that having a single input neuron does not cause H(c) to become disconnected anymore. In the case of e = 1, therefore, the picture presented in the main text is left unchanged. Let us consider the following question: what is the probability of having a disconnected invariant set given a realistic initialization?Consider a one-layer ReLU neural network with e = 1 and assume that the weights are sampled independently of one another from a normal distribution Wki ∼ N (0, σ 2 1 ) ∀k, i , W∼ N (0, σ 2 2 ) ∀k, j. From Corollary 2 we know that the invariant set H(c) will be disconnected if and only if there exists a hidden neuron satisfying∑ d i=1 (W (1) ki ) 2 < (W (2) k ) 2 .Given independence of the initial weight sampling, this probability can be computed asP[obstruction] = 1 -P[ d ∑ i=1 (W (1) ki ) 2 > (W (2) k ) 2 ] l = 1 -(F 1,d (dσ 2 1 /σ 2 2 )) l ,where F is the cumulative distribution function of the Fisher-Snedecor distribution.Having obtained this general expression, we can specify it to two common initialization schemes.• We obtain Kaiming initialization [21] withσ 2 1 = 2/d, σ 2 2 = 2/l resulting in P[obstruction] = 1 -F 1,d (l) l .• We obtain Xavier normal initialization [19] withσ 2 1 = 2/(d + l), σ 2 2 = 2/(1 + l) resulting in P[obstruction] = 1 -F 1,d ( d+ld d+l ) l .We plot these two expressions in Figure 5 . We can see how, for large values of d, the probability of obstruction quickly falls to 0 for any number of hidden neurons. Instead, we see an opposite trend for small values of d: the probability of disconnectedness grows with l. Moreover, it is interesting to notice that the region of high obstruction probability is much larger for Xavier initialization than for Kaiming initialization, further showing why the latter is preferred when working with ReLU networks.We can give some intuition on why there is no obstruction for multiple outputs. First, we consider a single hidden neuron k, with d incoming weights and a single output e = 1. If the neuron is pathological, we have thatEQUATIONSince the weights W 1k to change sign, its value needs to pass through 0 but, under the condition above, this cannot happen its square is always positive. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The work studies the properties of the loss surfaces of two-layer neural networks with homogenous activations (e.g., ReLU). The authors prove that in such networks, the gradient flow trajectories lie on an invariant set, which can be factored as the product of quadric hypersurfaces. The authors derive the properties of such sets and, importantly, show that for neural networks with a single output neuron, such sets may be disconnected, which creates obstacles for gradient flow and makes it (sometimes) impossible to reach the global optimum.

The reviewers found the results of the paper to be interesting and inspiring, and the overall exposition to be very clear and accessible. The key concepts are well illustrated with the figures and experimental illustrations. The area chair after reading the paper is also on the positive side. The key weakness pointed out by the reviewers is the limited scope, as the paper (primarily) studies a particular setting (continuous gradient flow, two-layer MLP only, homogenous activations, primary focus on the scalar output, etc.). However, it is likely that some of the theoretical advancements developed in the current paper can be extended to more general cases of neural networks. As a result, this could help to broaden the theoretical understanding of neural networks and help to advance the field of deep learning from the practical side as well (e.g., NN initialization, optimization).

Overall, there has been a large amount of fruitful discussions of authors with the reviewers, with clear suggestions on how and what to further improve in the paper, in particular, to highlight the open problems and potential future research in this direction. Upon acceptance, the area chair requests the authors to spend sufficient time incorporating these discussions into the paper accordingly.

Reviews: review: paper_summary: paper_summary: Summary of the paper:

* The paper undertakes a theoretical 'reachability' analysis for neural
  networks trained with gradient flow, identifying 'obstructions' separating
  certain regions of parameter space from each other.
* The setting is one-hidden-layer networks with homogeneous activation
  functions, including ReLUs. The main paper considers the biasless case but
  the authors extend their results to the case with biases in the appendices,
  with only minor modifications to the results.
* The authors consider training such networks with gradient flow and identify
  that regardless of the (output-dependent) loss function that gradient flow
  conserves a certain norm of the weights confining the training trajectory
  to a subset of the parameter space they call the 'invariant set' which is a
  product of hyperquadrics associated with each hidden unit in the network.
* Through topological analysis the authors derive the Poincaré polynomial of
  such invariant sets and in turn their zeroth Betti number, in effect
  counting the number of connected components in the invariant set.
  * These components are of interest because if gradient flow is initialised
    in one component it is impossible for the training trajectory to find its
    way to another, and training is limited to the best solution available in
    the initial component.
  * In this setting the zeroth Betti number (the number of connected
    components) is only one for networks with multidimensional inputs and
    outputs, indicating no topological obstructions.
  * In contrast, for networks with scalar inputs or outputs, it is
    exponential in the number of hidden units (with a base depending on the
    network initialisation), indicating many topological obstructions.
  * (For networks with biases, the space is connected if and only if the
    network has a multidimensional output).
* The authors then revise their count to take rescaling and permutation
  symmetries into consideration.
  * The reasoning is that if gradient flow can only explore its initial
    connected component, this is only a problem when the component is
    actually missing network functions rather than when it is merely missing
    some (but not all) implementations of those functions. Perhaps only one
    reachable implementation of the target function is needed.
  * The result is that there are fewer 'effectively' connected components,
    linear rather than exponential in the number of hidden units (with
    constants again depending on the initialisation).
  * However, the point is that there are still some topological obstructions
    to worry about.
* The authors provide a clear 'proof of concept' of their work in the form of
  a numerical experiment simulating gradient flow by learning a small neural
  network with gradient descent using a small learning rate. The results
  clearly show that the learning trajectory is confined to the invariant set
  and obstructed from reaching the global minimum as predicted.

I wrote a bit of a long review, so here is also a summary of my review:

* The main contribution, demonstrating the existence of topological
  obstructions to gradient flow, appears novel, interesting, and an important
  contribution to our understanding of the structure of parameter space.
* I also think the paper is exceptionally well written and clear and the
  analysis is elegant. I think the authors have discussed the limitations of
  the work well. I am left with only a small number of questions.
* My main concern is that the results show topological obstructions only
  occur under some strong assumptions, and in fact they do not arise in
  more situations. I find this somewhat at odds with the framing of the
  contribution.

Overall, I think the work is worthy of acceptance because it is elegant,
interesting, informative, and thought provoking, but I maintain concerns
about whether the framing is accurate.
strength_weakness: strength_weakness: The authors have convincingly demonstrated the existence of topological
obstructions under the stated assumptions. My main concern about this work is
the relevance of topological obstructions to deep learning practice.

1. I am not so concerned about the restriction to single-hidden-layer
   networks. The results may not generalise to the multi-layer case, or
   perhaps they will, it seems worth looking into with future work. However,
   I am concerned that *even in the single-hidden-layer setting* the main
   disconnectedness result holds only for networks with scalar output. The
   authors clearly show that networks with multidimensional outputs do not
   show obstructions.

    With this in mind, do the authors believe that the topological
    obstructions they have identified are relevant to deep learning practice?
    If so, can they clarify this point?

2. Furthermore, I have concerns about the gap between gradient flow and
   discrete gradient descent training. The authors explain that discretised
   gradient methods are not confined to the same invariant sets as gradient
   flow. It seems possible that this means discrete gradient methods count
   circumvent topological obstructions, even if they exist in practical
   settings.

    1. Do the authors conjecture that the identified obstructions have
       implications for gradient descent training?
    
    2. If the authors would support such an 'obstruction hypothesis', I would
       invite them to consider explicitly stating the hypothesis in their
       introduction and/or conclusion.
    
    3. The authors could also consider extending their simple numerical
       experiments to explore whether and to what extent the demonstrated
       obstruction affects training with larger learning rates or
       momentum-based training methods.

Pending the authors' clarification of these points, the framing of the work
is somewhat confusing to me. It appears to me that the authors have shown
that topological obstructions to gradient flow are actually quite a
restricted phenomenon. An alternative framing for the paper's results would
be to say that they show that deep learning is often *not* held back by
topological obstructions. Could the authors comment on this alternative
framing?

I don't think this concern is fatal to the paper. The topological analysis
approach and most of the results are still informative and worth sharing even
if the topological obstructions themselves end up not arising in more
practical settings.
questions: questions: 1. The authors have presented a clear interpretation of the source of the
   topological obstruction in terms of the relationship between each
   unit's weights (lines 232 onwards). Do the authors have a similarly
   enlightening interpretation of the breakdown of the topological
   obstructions in the case with multidimensional outputs?

2. Another assumption of the work is that the loss function depends only on
   the network weights via the output of the network. Am I correct in
   understanding that if the loss function includes a regularisation term
   that depends on, for example, an $L_p$ norm of the weights, then this
   assumption would be violated and the analysis would not hold because
   gradient flow would no longer always be perpendicular to the invariant
   sets? This seems fine but might be worth acknowledging.

3. Appendix E extends the analysis to networks including biases, but stops
   after deriving the zeroth Betti number in this setting. Could you please
   clarify whether the analysis of section 5 is unchanged in this setting?

4. The contribution list does not mention the 'proof of concept' numerical
   experiments. It seems to me that these experiments are a reasonably
   important part of the paper's contribution and its evaluation to warrant
   mentioning in the introduction (as well as the abstract).

5. Line 118: The authors describe the transformations as sending parameters
   to different but observationally equivalent parameters. It is typically
   true that the parameter produced is different but for some parameters or
   for some transformations the transformations don't change the parameter.
limitations: limitations: The paper accurately describes the contribution including its limitations.
These limitations are adequately and carefully disclosed in multiple places
in the main text and discussed in detail in appendix F.

1. One main limitation is the simplicity of the architecture, with only one
   hidden layer. The authors discuss barriers to extending the work to the
   multi-layer case, namely the interaction between layers, which is left to
   future work.
2. A second main limitation is the fundamental assumption of gradient flow.
   The authors explain that the conservation laws that are crucial to their
   analysis do not hold for discretised gradient descent.
3. A third main limitation is that the topological obstructions only exist in
   the case of networks with a scalar output. This is clearly acknowledged in
   the abstract and throughout the paper, but I think the authors could have
   more thoroughly discussed the implications of this restriction, namely
   that in many cases we should not expect topological constructions at all.

If the work is to be accepted I would encourage the authors to include the
valuable discussion in appendix F in the main text, and to extend this
discussion along the lines of (3).
paper_summary: paper_summary: This paper studies the topological obstruction in the loss landscape of two-layer ReLU neural networks under gradient flow. Using conserved quantities in gradient flow, the authors define invariant sets, to which gradient flow trajectories are constrained. They then show that when the input or output of the network is a scalar, the invariant set may have more than one connected components depending on initialization. This leads to a topological obstruction, as gradient flow cannot reach a global minimum if parameters are initialized in the component that does not contain one. After taking scaling and permutation symmetry into account, the number of effective components scales linearly with the number of pathological neurons, which are neurons that cannot change the sign of their output weight. On a toy example with two hidden neurons, the authors fully characterize the invariant sets and show theoretically and empirically that gradient flows that do not start in the same effective component as the global minimum cannot reach the global minimum.
strength_weakness: strength_weakness: The scope of the analysis, which is restricted to two layer ReLU neural network and gradient flow, is rather limited.
- While ReLU is a popular activation function, two-layer networks are rare in today’s machine learning models. The results in the paper do not readily extend to deeper networks, where invariant sets no longer factorize easily into product spaces.
- Since SGD is known to have an implicit bias characterized by a drift of $c$, training trajectories in practical settings (with non-infinitesimal learning rate) will not stay in the same invariant set. Even if they stay close to an invariant set throughout training, the noise in each step may take the parameters to a different component. 

If extending the analysis beyond this setup is challenging, additional experiments on different settings could also help illustrate that the topological obstruction is a realistic hindrance in practice. For example, for deeper ReLU networks, it might be possible to investigate empirically whether there exist initializations that lead to similar obstructions as observed in Figure 3c.
questions: questions: - For ReLU networks with one hidden layer, does every invariant set contain a global minimum? If not, then even if an invariant set is connected, it is not guaranteed that all gradient flows starting from points on this set can reach a global minimum. This is not the type of obstruction considered in the paper, but I am curious about whether there are other considerations when choosing the initialization, besides ensuring the connectedness of the invariant set.
- Assume that an invariant set contains a global minimum. Do all gradient flows in this invariant set converge to a global minimum?
- For a wider two-layer ReLU network, using common initialization methods (such as Xavier initialization), what is the probability that the invariant set has more than one effective components? What is the probability that the initialization falls in the component that does not contain a global minimum? Is it nonzero?
- Do the rest of Betti numbers, other than $\beta_0$, provide useful information about the loss landscape and the training process?

**Minor issues / suggestions**
- Line 135-136: While allowing a finite sequence of actions is arguably more intuitive, it might suffice to just define observationally equivalence as being related by a composition of one $T$ and one $P$. As a consequence of Lemma 3 in the appendix, for any finite sequence of T and P, there exists a sequence of T and P with length at most 2 that produces the equivalent transformation.
- In Equation 11, is the second expression missing a factor of 2?
- The sentence in line 483 is not grammatically correct.
limitations: limitations: The authors are upfront about the limitations and include a limitation section in the appendix.
paper_summary: paper_summary: In this paper, authors analyze the performance of gradient-descent optimization over a two-layer neural network. Authors reveal the presence of obstructions in the loss landscape and explore their topology. Finally, they identify the cause of those optimization obstructions in the so-called `pathological’ neurons—neurons that can’t change the sign of their output weight.
strength_weakness: strength_weakness: 1) This paper identifies pathological neurons that disrupt the performance of gradient descent over ReLU activations; however, nothing is said about how such obstructions can be avoided or how their effect can be mitigated.

2) The scope of the research is limited to neural networks with two hidden layers.

3) The possible benefits for further research and practical applications of the achieved results are somewhat vague.
questions: questions: See Weaknesses section.
limitations: limitations: Yes, the limitations are addressed. This is a purely theoretic work, dealing with a somewhat simplified example of a two layer fully-connected neural network.
paper_summary: paper_summary: This paper considers the landscape topology of one hidden layer networks with non-negative homogeneous activations. The main result of this work is that in some cases, the loss landscape may consist of disconnected components that cannot be traversed by gradient flow dynamics. This leads to an obstruction, in the sense that certain initial conditions will result in closed paths away from the global minimum. They further count the number of effective components (in the sense of being equivalent under the symmetries of the problem) and show that this number grows linearly with the number of hidden neurons that cannot change their sign during the dynamics.
The authors provide a full analytical treatment of the problem, supplemented by a simple toy example.
strength_weakness: strength_weakness: The main weakness of this work is its **scope**. The work studies in great detail a very specific architecture, but it is unclear how interesting their conclusions are for either theorists or practitioners. Additionally, their numerical/toy example is extremely simple, and it is a bit hard to see how their results extend to more complicated settings, even though they discuss some of these aspects in their limitations section. It would be useful to include more complicated examples, even in the single hidden layer case, as well as contrast these results with cases in which the activations are not ReLU-like, i.e., commenting on how these results break down.
questions: questions: Comments and questions below:

1) L52: “studied by under various…” incoherent sentence.

2) L51: “activation” should be “activations” I assume.

3) In Sec. 3.3, the authors consider an arbitrary dataset and a general empirical loss function. The analysis does not depend on these details and is therefore truly a property of the network and the initialization, and so these results should hold for any task. Is this correct?

4) The experimental setup in Sec. 6 is incomplete - the loss is never specified (MSE, CE?), it would be nice  to see (empirically) that the results do not depend on it.

5) Can the authors comment on the relation of their work and Refs. [1,2]?

6) What happens to the topological structure if we consider the discrete GD instead of GF? the results should hold for small learning rate (as shown numerically), but there is a possible catapult regime (Ref. [3]) at large learning rate, will the paths not be restricted to connected parts in this case? could the catapult effect cause a transition between disconnected effective manifolds?

[1] https://arxiv.org/pdf/2401.10791

[2] https://arxiv.org/pdf/2402.05626

[3] https://arxiv.org/abs/2003.02218
limitations: limitations: The authors discuss their limitations in a dedicated section.
paper_summary: paper_summary: This works studies how topological properties affect loss landscape of neural networks, showing that the loss can be divided into disconnected regions where reaching one region from the other using GF is impossible. The theory is shown to have consequence for understanding the training of simple networks
strength_weakness: strength_weakness: This paper has quite a few weaknesses in its current form, and these prevent me from recommending acceptance at this stage

1. Prior works. (a) it is unclear how different this work is from Safran et al. [35]. This point is never discussed, and this makes it difficult for me to judge the novelty of this work. (b) Incorrect reference. Eq. (11) appears the earliest in https://arxiv.org/abs/1312.6120, not in Du et al. [11]. (c) emergence of topological obstruction due to permutation/rescaling symmetries have been studied in https://arxiv.org/abs/2309.16932, and the authors need to compare the results to clarify what is novel in the present manuscript

2. The main result applies only to two layer bias-less ReLU nets, which I feel to be too weak

3. The meaningful implications (corollary 1) is only relevant when either the input or the output dimension is 1, which makes the theory very unlikely to the relevant to practice

4. The experiment also has very limited scope. If the authors can show that the effect they studied is relevant for training a much larger model on a more realistic dataset, I would be more convinced of the relevance and contribution of the theory
questions: questions: Does the results still hold when there is a bias term？
limitations: limitations: The paper could also benefit by discussing more of its limitations

Comments: title: Thank you for the comments
Reply to the Authors
Thanks for response
Thank you for the clarifications
Relevance of topological obstructions to deep learning practice [2/2]
Relevance of topological obstructions to deep learning practice [1/2]
Thanks for the clarification
Thanks for the detailed reply
comment: Dear Colleague, thank you for the positive assessment of our work and for the insightful comments.

**1. Characterising when discrete GD bridges obstructions.**

The preliminary experiment we performed, prompted by the reviewers' questions, was just meant to show that there are circumstances in which it is possible for gradient descent trajectories to circumvent the obstruction, and should not be taken as a complete characterization of the phenomenon. The argument "not too low nor too high" roughly meant that if the learning rate is too low, the trajectory closely approximates GF dynamics, which prevents it from escaping the invariant set. Conversely, if the learning rate is set too high, GD may exhibit erratic behavior. 

We think that analyzing the effect of discretization is a very interesting research direction which deserves to be treated in detail elsewhere. As for when the bridge phenomenon happens, an educated guess would be that the value of the learning rate should be compared with the minimal distance between the components of the invariant hyperquadric i.e. $2|c|$.

**2. On regularisation.**

Thank you for the comment. We will add this observation in the limitations section.

**3. On biases.**

The key point is that the work presented in this paper does not make any assumptions on the inputs, whether they can vary or they are constant (like for biases). As you can see in Eq. (26), the rescaling transformation (and also permutation) is not concerned with the inputs and acts on biases just like it acts on the weights of the first layer $W^{(1)}$. Therefore, weights and biases will be treated in exactly the same way and the results obtained still hold. 

**4. On single-hidden-layer networks as building blocks.**

Your assessment is indeed correct. In the case of a network with a final two-layer ReLU module, the interaction with the prior layers would need to be taken into consideration for our topological analysis. 

**5. On the position of the limitations section.**

We thank you for the suggestion. We agree that the limitation section, expanded with the observations prompted by the reviews, needs to be included in the main paper and, if accepted, we will do so in the final version. 

**6. Making explicit the open question of topological obstructions.**

The driving interest behind our work is undoubtedly related to this "obstruction hypothesis", but we feel it is too early to pinpoint the potential practical applications. 
We hope the community will find our paper intriguing because it studies a fundamental structural phenomenon of neural network architectures and, secondarily, because of the possibility of it leading to practical improvements to deep learning practices (if these obstructions are found to have an impact).

On this second possibility, an insight we gather from the classification task presented in part 2 of the general rebuttal, is that the presence of only pathological neurons seems to hinder learning, even when the network's width is scaled. However, our probabilistic analysis shows that with common initialization schemes, the probability of creating a pathological neuron decreases rapidly with increased inner layer width. Therefore, the combination of specific initialization schemes and an abundance of hidden neurons (beyond the minimum required to solve a task) appears to make this obstruction unlikely in practice, which might explain why this phenomenon has not been observed. 
This work describes a simple safeguard to avoid obstructions, which could, for instance, discourage looking for new initialization schemes that result in a proliferation of pathological neurons.

Should the paper be accepted, we will include these observations and further clarify our motivations in the introduction and conclusions sections.
I thank the authors for their detailed reply, as well as their global reply and further experiments.

I believe the paper should be accepted, but due to its limited scope, and after reading other reviews, including that of Reviewer JxYP, which addressed regularization and the "building block" argument etc., I believe that a higher score is not warranted. 

I do recommend that the authors delve deeper into the finite learning rate in future works, and especially the lr=0.533 example that was found, to understand how the obstruction is circumvented as a theoretically motivated reason for using large learning rates.
Thank you for addressing my comments and questions. I do not have further questions and will maintain my positive rating.
Thank you for answering my questions and providing clarifications; I have also carefully read the reviews from other reviewers (and rebuttals to them). I found this paper quite interesting and I lean towards thinking it worthy of acceptance. Unfortunately, my prior experience in the optimization theory is rather limited, and I will hold on my current score for this paper (6).
### Framing

**5. On the position of the limitations section:** I think the differences of architecture and learning algorithm between the setting studied in this paper and settings of relevance in deep learning practice are substantial and deserve to be explicitly addressed within the main text of this paper if it is to be accepted.

I think the authors have acknowledged these differences and their appendix F does a good job of summarising them. I would repeat the suggestion from my review, that the authors should consider promoting this material from appendix F into the main text.

Since my review the authors have also presented a page worth of additional figures, and, also accounting for the space required to describe these experiments, they may face a shortage of space if they were to attempt to put all of this material into a 10-page version of the submission along with the discussion of limitations. However, I still think it is worth considering making a prominent place for a brief explicit acknowledgement of these limitations (e.g. in a named section or subsection) because this is one of the most important considerations for readers of the submitted work to be made aware of. (Of course, the authors are welcome to use their judgement to decide how to organise the final version of the paper, if accepted.)

**6. Making explicit the open question of topological obstructions:** I questioned the authors' framing around topological obstructions, given that their results imply that these appear in quite restricted circumstances. I appreciate the authors' response which was as follows:

> We agree that, from this first set of results, it seems that the topological obstruction induced by symmetries is a restricted phenomenon. However, the fact that there is the possibility of a fundamental obstruction to training is not something that is commonly known or expected by the community. Therefore, we believe that the framing we gave to the paper is justified, as it serves to highlight this possibility. Moreover, while our analysis clearly identifies obstructions in some cases, it is, in principle, hard to exclude that extending the results to more general settings will not uncover other obstructions of a similar kind.

I agree that the finding of topological obstructions in a restricted setting is worth presenting to the community and worth further investigation.

Upon taking another glance at the paper I can't fault the authors for anything they have said. They consistently carefully qualify every claim so as to avoid overstating their results. They have been upfront about the limitations of the direct implications of their results in their paper and in their rebuttals. In my opinion, the paper is a masterclass in precise communication of theoretical results.

However I am left with a distinct impression that while the authors have not said anything wrong, something important remains *unsaid*---namely that the central motivation of their work is the hypothesis that topological obstructions may hinder training in practical deep learning. If I am not mistaken, this is never stated in the paper, even though it has got to be the strongest reason to share this work with the NeurIPS community (right?).

I think the authors don't need to be afraid to make this claim, even though the answer lies in future work. Their results and experiments have comprehensively and elegantly answered this question in the limited setting of shallow ReLU networks. More than a toy model, their analysis made to look effortless with the powerful tools of topology has cut out an example for future work on this topic to follow.

I therefore ask:

1. Do the authors agree that the reason people should care about their work is that topological obstructions may be relevant to deep learning with practical algorithms and architectures (or not, whatever the case may be, but it would be nice to know either way)?
2. Are the authors willing to state their answer to question (1) prominently in their paper (for example in the introduction or conclusion, for example alongside the discussion of limitations as per my previous point)?

[Part 2 of 2]
It appears that the current consensus among reviewers is that this paper should be (weakly) accepted, with the primary remaining concern being about the relevance of the phenomenon of topological obstructions in practice (noting this is my impression and I don't speak for the other reviewers).

On the question of relevance I have a couple of follow-up questions after reading the reviews and rebuttals and studying the new gradient descent experiments. I organise my questions by two key points of departure between the theoretical setting where topological obstructions found by the authors and those settings common in deep learning practice: on the learning algorithm (gradient flow) and on the architecture (scalar-output single-hidden-layer ReLU networks). I conclude with some suggestions on the paper's framing.

### Learning algorithms

**1. Characterising when discrete GD bridges obstructions:** The additional experiments show that the phenomenon of discrete gradient descent training bridging the obstruction "can occur for some values of the learning rate, which are not too low nor too high" [from a rebuttal].
I would like to question the basis of this "not to low nor too high" characterisation.

In the attachment I see that the `lr=0.533` run clearly bridges the obstruction in terms of loss and parameter-space trajectory. The `lr=0.6` run appears not to have bridged the obstruction (though I can't actually see where it went in the parameter space?), and this must be the basis for "not too low nor too high", but this is the only higher learning rate tested. I don't know that this is sufficient basis for characterising the bridging phenomena in this way.

**2. On regularisation:** In my review I noted that learning with weight-based regularisation would violate the assumptions of the analysis, and the authors agreed with this observation.

This difference would appear to me to be more severe than the difference between gradient flow and (stochastic) gradient descent, since it is not just about discretisation---regularised parameter updates may not even be *locally* orthogonal to the invariant set.

This limitations are implicit in the assumptions listed throughout the analysis, but I would invite the authors to explicitly acknowledge this limitation at the level of the discussion of appendix F (alongside their existing acknowledgement of the difference with momentum-based methods).

### Architectures

**3. On biases:** I was not completely convinced by the authors' reply to my question about the case with biases. It's true that networks with biases are the same as networks with additional input units, but those additional input units have to have their inputs restricted to a constant function. Thus analysis of network functions including, e.g., invariance, has to proceed in a subspace of the expanded architecture's function space. In general such a restriction can create new symmetries.

I think this probably doesn't undermine the analysis, since the analysis is mainly based on the existence of certain symmetries which will not be removed by adding additional restricted dimensions. I would need to think about it more carefully to be sure.

I would like to ask, have the authors taken this complexity into account in the claims in the paper that the results generalise to networks with biases?

**4. On single-hidden-layer networks as building blocks:** A couple of times in the rebuttals the authors said that while single-hidden-layer networks are not commonly used in practice as a stand-alone architecture, they are commonly used as building blocks of other architectures. The authors cite CNNs as examples and I might add their use in transformer blocks and that an MLP can be viewed as a composition of
single-hidden-layer networks in some sense.

However, I wanted to point out that in such cases typically the building block layers have multiple outputs (with the exception of a final layer in an MLP or CNN used for binary classification). The authors have shown that such layers *would not* unilaterally suffer topological obstructions. Therefore, while the authors' analysis could be informative in studying architectures composed of hidden layers, their key finding won't necessarily generalise (there would have to be some other component of the architecture that restores the topological obstruction to this layer or introduces its own obstruction). In the case of networks with a final layer with a scalar output, the final layer may have a topological obstruction, as long as the interaction between this and prior layers does not allow it to be circumvented.

I understand that the definitive answer would have to be left to future work, but I wonder, would the authors agree with this preliminary assessment of the prospects of that future work?

[Part 1 of 2]
I thank the authors for their clarifying rebuttals. I especially thank the authors for their intuitive explanation of the resolution of the obstruction in the multi-output case. I invite them to include this and their other clarifications in future revisions of the paper. The three new experiments are also a very welcome addition to the submission.

At the moment I am maintaining my recommendation. I retain some of my concerns about the framing and relevance of the results. I am about to post a top-level comment on this topic since concerns about relevance were also raised by a few other reviewers.
The reply addresses my main concerns. Although I still think the main problem is with its unclear relevance. I will raise the contribution rating to fair and the score to 6