Learning on a Razor’s Edge: Identifiability and Singularity of Polynomial Neural Networks
Authors: Vahid Shahverdi
Giovanni Luca Marchetti
Kathlén Kohn
Venue: ICLR 2026 Poster (2026)
Keywords: identifiability
singularities
critical points
neuromanifolds
polynomial activation
algebraic geometry
Primary area: learning theory
TL;DR: We discuss identifiability of MLPs and CNNs with a generic polynomial activation, and relate the singularities of their neuromanifolds to subnetworks and sparsity bias.

Abstract: We study function spaces parametrized by neural networks, referred to as neuromanifolds. Specifically, we focus on deep Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs) with an activation function that is a sufficiently generic polynomial. First, we address the identifiability problem, showing that, for almost all functions in the neuromanifold of an MLP, there exist only finitely many parameter choices yielding that function. For CNNs, the parametrization is generically one-to-one. As a consequence, we compute the dimension of the neuromanifold. Second, we describe singular points of neuromanifolds. We characterize singularities completely for CNNs, and partially for MLPs. In both cases, they arise from sparse subnetworks. For MLPs, we prove that these singularities often correspond to critical points of the mean-squared error loss, which does not hold for CNNs. This provides a geometric explanation of the sparsity bias of MLPs. All of our results leverage tools from algebraic geometry.

Published as a conference paper at ICLR 2026
LEARNING ON A RAZOR'S EDGE: IDENTIFIABILITY AND SINGULARITY OF POLYNOMIAL NEURAL NETWORKS
Vahid Shahverdi *, Giovanni Luca Marchetti * & Kathlén Kohn *
Department of Mathematics
KTH Royal Institute of Technology
Stockholm, Sweden
(vahidsha, g glma, kathlen @kth.se
ABSTRACT
We study function spaces parametrized by neural networks, referred neu- romanifolds. Specifically, we focus on deep Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs) with activation function that is a sufficiently generic polynomial. First, we address the identifiability problem, showing that, for almost all functions in the neuromanifold of an MLP, there ex- ist only finitely many parameter choices yielding that function. For CNNs, the parametrization generically one-to-one. As a consequence, we compute the dimension of the neuromanifold. Second, describe singular points of neuro- manifolds. We characterize singularities completely for CNNs, and partially for MLPs. In both cases, they arise from sparse subnetworks. For MIPs, prove that these singularities often correspond to critical points the mean-squared er- ror loss, which does not hold for CNNs. This provides geometric explanation of the sparsity bias of MLPs. All of our results leverage tools from algebraic geometry.
Figure 1: Subnetworks define singular points (orange) of the neuromanifold.
INTRODUCTION
Machine learning models parametrize spaces of functions, often referred neuromanifolds (Calin 2020 Marchetti et al, 2025). Their geometry governs several learning aspects, ranging from expressivity and sample complexity to the training dynamics. Neuromanifolds tical learning theory- where they are referred to as hypothesis spaces-and and information geometry Amari 2016), and have been analyzed from the perspective of various mathematical disciplines.
central statis-
"guul contribution.

---

Published as a conference paper at ICLR 2026
A classical question in machine learning is identifiabilityy the problem of describing the parame- ters that correspond to the same function. Several works have formally addressed this question for specific activation functions, such as Tanh (Fefferman et al.] 1994), Sigmoid (Vlačié & Böleskei 2022), ReI.U (Grigsby et al. 2023; Bona-Pellissier et al. 2003). and monomials (Finkel et al.) 2024 'sym- Usevich et al. 2025). Identifiailltyy can be interpreted characterizing the redundancies metries' of the parametrization of the neuromanifold (Lim et al. 2024). The degree of redundancy determines the dimension of the neuromanifold, which, in turn, measures the sample complexity of the corresponding model (Cucker & Smale, 2002). Moreover, non-identifiable functions often correspond to singularities of the neuromanifold. These are special points where the model is not a smooth manifold, such as edges, cusps, or seff-intersectionss As advocated by the field of Singular Learning Theory [Watanabe] 2009), singular points play an important role for learning, e.g., in terms of generalization error (Wei et al. 2022) and training stability (Amari et al. 2006 Wei et al. 2008
In this work, we investigate the identifiability and singularities of both Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs). Instead of focusing function, we consider a large class of activation functions, namely, generic polynomials of large degree. This is motivated by the fact that polynomials can approximate arbitrary continuous func- tions, and thus, polynomial neural networks approximate arbitrary networks. Crucially, considering polynomial activations makes geometric analysis more feasible for two reasons. First, polynomial ML.ss are the only ones whose neuromanifolds live in finite-dimensional ambient spaces (Leshno al. 1993 Theorem 1). Second, allows us to employ tools from algebraic geometry (Kileel 2019 Trager et al. 2020). The latter is a rich field of mathematics that is particularly suitable for analyzing symmetries and singular spaces. Polynomial networks are the central focus of neu- roalgebraic geometry (Marchetti et al., (025) an emerging field bridging algebraic geometry and theoretical deep learning, where our work naturally fits.
single activation
et al.
Our main results apply to sufficiently generic polynomial activation functions of large degree:
Identifiability: We prove that almost all functions in the neuromanifold of an MLP correspond to only finitely many parameters (Theorem 4.1). This result is in line with the widespread belief that general MLPs have discrete parameter symmetries (coming from permutations of neurons in each layer). However, the only available proof' of this claim are for Tanh and Sigmoid activations (Fefferman et al., 1994, Vlačić & Bölcskei 2022). Our result extends the identifiability studies for monomial activations (Finkel et al. 2024| Usevich et al.) 2025) to the broad class of generic polynomials. In particular, we deduce that the dimension of the neuromanifold coincides with the number of parameters. This resolves the dimension conjecture by (Kileel et al. (019) in general form. For CNNs, we prove a stronger result: almost all functions are uniquely identifiable, i.e., they correspond to single choice of parameters (Theorem 4.4
We consider sparse subnetworks, where a subset of neurons is inactive. We prove that these subnetworks (under appropriate assumptions) are singular points of the neuromanifold, both for MLPs (Theorem 4.2) and CNNs (Theorem 4.6). We then introduce the notion of critically exposed parameter sets, that contain critical points the mquars error loss with positive prob- ability over the dataset (Definition (5. We prove that subnetworks of MLPs are critically exposed (Theorem while they are not in the case of CNNs (Proposition 4.5).
Singularities:
Our results can be interpreted from the perspective of sparsity bias- the empirically-ohservedd phe- nomenon that neural networks tend to discard neurons during their training process, effectively converging to a function parametrized by smaller subarchitecture. This is closely related to the popular 'lottery ticket hypothesis' (Frankle & Carbin 2 suggestion that the model implic- itly identifies and amplifies subnetworks that are particularly efficient for the given task, resulting in sparse representations of data. Our singularity results provide a geometric explanation of these phenomena. Indeed, our notion of critically exposedness formalizes the idea of a bias of the training dynamics towards the given set, since critical points are local attractors for stochastic gradient flows (Chen et al. 2023). Our results critical exposedness of subnetworks indicate that the sparsity bias is exhibited by MLPs, but not by pure single-channel CNNs. This is in line with recent empirical findings for convolutional architectures (Blumenfeld et al. 2020).
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Algebraic geometry of deep learning. As anticipated, a line of research in theoretical deep learning-recente termed "neuroalgebraic geometry' (Marchetti et al. neuromanifolds of polynomial neural networks through the lens of algebraic geometry. Motivated by the fact that polynomials can approximate arbitrary continuous functions, the goal of neuroalgebraic geometry is to approach fundamental problems in machine learning via tools from algebra. One such problem is identifiability, which represents a core focus of neuroalgehraic geometry. Several models have been analyzed in the literature, ranging from MLPs with linear (Trager et al. (2020) and monomial activations (Kileel et al., 2019 Finkel et al. 2024; Kubjas et al. 2024; Marchetti et al.) 2024), to CNNs with linear (Kohn et al. 2022, 2024 Shahverdi 2024) and monomial activations Shahverdi et al. 2025), to (un-normalized) attention based networks (Henry et al. 2025). ) In this work, contribute to this line of research by addressing identifiability for MLPs and CNNs with (generic) polynomial activation functions. Singularities are also central in neuroalgebraic geometry, due their role in the learning process. However, they formally understood only for linear MIPss (Trager et al. 2020) and monomial CNNs (Shahverdi et al.) 2025). In both cases, they coin- cide with subnetworks the corresponding architecture. The general relation between singularities and subnetworks is conjectural (Marchetti et al., (025). In this work, we show that subnetworks often define singular points for MLPs and CNNs, further sedimenting this relation.
the study of
Implicit sparsity bias. Several works have theoretically analyzed the tendency of neural networks to converge to sparse data representations. A line of research (Woodworth et al. 20200: Nacson al., 2022 Pesme et al., 2021 Andriushchenko
al., (023) has shown that, for deep diagonal linear networks, the training dynamics, when initialized around the origin, implicitly penalizes the
l norm of the weights, inducing sparsity in the representation. A similar bias towards low-rank solutions has been shown for deep linear networks trained with a regularized loss (Kunin et al.) 2019 Ziyin et al., 2022 Wang & Jacot 2024). Closely related to our work, these biases have been recently reformulated in terms of subnetworks (Chen et al., (023). Most of the previous literature focuses on simple models, and on tools and ideas from (stochastic) dynamical systems theory. In contrast, we promote a purely geometric perspective, explaining the subnetwork bias in terms of the singularities and the parametrization of the neuromanifold. Moreover, we focus on general models- specifically, deep networks with polynomial activations. This generality is enabled by the powerful tools from algebraic geometry.
Singular learning theory. The role of singularities in machine learning and especially their ef- feet on the training dynamics (Amari al. [2066] WWii et al.) 2008 has been promoted by Singular Amari
Learning Theory (SLT) [Watanabe ( [2009] 2007] Amari et al. |000 Wei et al. (022)). The latter falls
into the more general framework of information geometry, focusing on the Riemannian geometry of the Fisher information metric. Crucially, however, the notion of singularity in SLT drastically differs from the one adopted in this work. Following the formalism of neuroalgebraic geometry, we consider singular points of the neuromanifold in the classical algebro-geometric sense-see see Section 4.2 and Appendix A In contrast, singularities in SLT are parameters where the metric tensor, once pulled back, is degenerate. This happens due to extra degrees of freedom in parameter space, or due to criticalities of the parametrization. Both these cases can lead to singular or smooth points of the neuromanifold. Vice versa, a singular point can arise in neuromanifolds that parametrized regularly, where the pulled-back metric tensor is non-degenerate; this is the case for polynomial CNNs, we will discuss in Section 4.4 Therefore, the two notions of singularity are different and incomparable. In a sense, our results extend ideas similar to the ones from SLT to a function space perspective, i.e., in terms of the geometry of the neuromanifold.
3 BACKGROUND
In this section, we overview the basic notions around algebraic geometry, deep neural networks and their neuromanifolds.
3

---

Published as a conference paper at ICLR 2026
3.1 ZARISKI TOPOLOGY
Throughout this work, we will leverage on methods from (polynomial) algebra and algebraic geom- etry. Specifically, will often the Zariski topology, which we recall here (in the context of real vector spaces).
Definition 1. Fix a positive integer n. The Zariski topology on R is the topology whose closed sets are the algebraic ones, i.c., solution sets of systems of polynomial equations in variables.
Now, the Zariski topology exhibits striking property: it is irreducible, i..., any non-empty open set dense. Moreover, dense sets the Zariski topology are also dense the Euclidean one. As such, for Zariski-open set, it suffices to find single element in it, to conclude that 'almost all' points of IR belong to it. Based on this, elements belonging to open sets are referred to as generic. We will leverage on irreducibility in several arguments, which will allow to obtain results that hold for generic objects (e.g., generic parameters, and generic activation functions).
3.2 NEUROMANIFOLDS AND SUBNETWORKS OF MLPs
Fix function a: R, sequence of L 1 positive integers do, do...., dr, and, for every i 1,..., L, matrix W; Idd
Definition 2. A Multi-Layer Perceptron (MLP) with architecture d (do...., ., dr), activation func- tion and weights W (W1, ..... WL) is the map fw: Rd + Rd given by the composition:
fw Wr ................................ o0o.
(1)
where a is applied coordinate-wise.
We now introduce the function spaces parametrized by neural networks. Let W + be the parameter space of an MI.,, and yp: W 3 W > /w be its parametrization map.
R4 x
Definition The neuromunifold of an MLP with architecture d and activation function a image of the parametrization yp, i.c. Ma. = [fw W E WJ,
(2)
As anticipated, from now on, we assume that a is (univariate) polynomial of some degree r > 1:
(x)
axx"
In this case, fw is a (multivariate and vector-valued) polynomial of degree at most y--1 i.e.. Ma. C V. where V is the space of all polynomial maps RR: -> Rd of degree at most م In particular, the ncuromanifold lives in an ambient space V of functions, which is mitee-dimensional and linear. This is unique polynomial activations (Leshno et al. 1993, Theorem 1). Moreover, it follows immediately from the Tarski- Seidenberg theorem that Ma.- is a semi-algebraic variety, i.e., it can be defined by polynomial equalities and inequalities in V. The inequalities be removed by taking the closure of Ma, in the Zariski topology of V.
We also introduce the notion of subnetwork of MLP, which will be central in our results. For every i = I,...,, 1, let A, C {I,...,,,, d:) and define A = (A1,
Definition 4. parameter W E W is an A-subnewwork if for every 1,..., L - 1 and every j E A,, the j-th column of W411 vanishes. We say that W is a strict subnetwork if, moreover, the j-th row of W, vanishes for j E A;.
For simplicity, we set Ag Ar 0, i.e., we not allow subnetworks obtained by removing input or output neurons.
3.3 OPTIMIZATION
We consider a regression problem with mean squared error objective. To this end, let D be dataset, i.e., finite subset D c RRd x Rdz representing input-output pairs. The corresponding
4

---

Published as a conference paper at ICLR 2026
loss Lw: V + Rgo is given by:
E 11f(x) y/1².
(4)
(ANjED
An MLP is trained on the dataset DD by minimizing LD o over W. Since the loss quadratic,
equation 4 can be rephrased as LD Q( ((f-u), u), where Q is quadratic form on V and V. both depending on DD We refer to Trager et al. (020);; Shahverdi et al. (2025) ; Kubjas et al. (2024) for extended discussions around this, including precise expressions for Q and (depending on D). These works also show that for large and generic 'D, the quadric Q is non non-degenerate and is generic in V. Therefore, with enough data, training the network amounts to optimizing a non-degenerate distance from generic point in the ambient space of the neuromanifold.
Since optimization is typically performed via gradient descent, we will be interested in the critical points (in parameter space) of the loss, where its gradient vanishes. These correspond to the equi- libria of the gradient flow, and are therefore the stationary points of gradient descent. For loss La(l)== Q - u) as above, it follows from the chain rule that W E W is a critical point of L. up if, and only if, fw - u orthogonal according to the scalar product induced by Q to the image of the differential of yp based at W. This geometrie interpretation of criticality will be crucial in some of our arguments.
Note that focus whether point is critical, disregarding the type of criticality, i.e., (local) minima/maxima and saddles. While this enables a purely-geometric treatment, studying the type of criticality is interesting problem, since local minima are the actual (local) attractors of the gradient flow. However, this is more subtle, and goes beyond the scope of this work - see Section Moreover, it worth mentioning that when noise is taken into account the dynamics, arbitrary critical points can become (local) attractors, in a probabilistic sense. This is general principle from stochastic processes, and has been discussed in the context of deep learning by Chen et al. (2023).
4 RESULTS
In this section, we present our main results. We first focus on MLPs, and then consider CNNs.
4.1 IDENTIFIABILITY OF MLPS
We now discuss identifiability of MLPs which, from the perspective of algebraic geometry, means to investigate the fibers of the parametrization map yp. Formally, the fiber of yp at W is defined as x (fw) (W' E W fw fw:).
To begin with, observe that if W is an A-subnetwork, then fw coincides with the function obtained by removing from the architecture the neurons of the i-th layer with indices in A,, resulting in an MLP with architecture (dg - AAul: dp - |Azl). This implies that fw is independent of the di entries of the j-th of W, for j E A. In particular, strict subnetworks parametrize the same function as the corresponding non-strict ones. Therefore, the fiber of 4 has a large fiber at W a fact has been observed in the SLT literature (Amari et al. 2006 Wei et al.) 2008 Orhan & Pitkow 2018 More precisely, this fiber has dimension dim(y= ((w)) Σ/-1
A,
In our main result, we prove that, if the activation a is generic and of large-enough degree, the fiber of yp at generic W W is instead finite, i.e., O-dimensional. In other words, we show generic finite identifiability. This is equivalent, by the fiber-dimension theorem (Shafarevich) 2013 Chap. 1.6.3), to that the dimension of the neuromanifold equal the number parameters. The dimension of the neuromanifold is a fundamental invariant measuring the expressivity and sample complexity of the model (Kileel et al., 2019 Marchetti et al. (023). From a mathematical perspective, computing the dimension is an involved problem. For polynomial MLPs, it was originally conjectured in (Kileel jet al., al., 2019), and only recently established in (Finkel et al., 2024 Usevich et al. (225)) for monomial activations a (ar) = x". We extend those result to more general polynomial activations. Note that monomial MLPs have infinite generic fibers, arising from neuron- wise scalings.
Theorem 4.1. Suppose that di > 1 i O.... L- 1, and let a be a generic polynomial of large enough degree r > 0 (depending d). Then, the generic fiber of the parametrization map yp
5

---

Published as a conference paper at ICLR 2026
finite. In particular,
dim(Maa,) dim [
(5)
Proof. Since the result highly technical, we summarize the proof here. A full proof is provided in Appendix B
We first show that it suffices to prove the result for a single activation. From the properties of the Zariski topology, it then follows that the result holds for a generic one. We then pick a particular activation of large degree r whose coefficients are extremely sparse, i.e., the a, equation vanishes for many 0 I i S r. For this activation, we argue that some homogeneous components of Jw (seen as multivariate polynomial in its input) coincide with MLPs with the same weights W, but where the activation function is a monomial. This implies that the fiber of & W is contained in the intersection the fibers MLPs with monomial activation. As mentioned above, the generic fibers of these networks are known; they coincide with permutations of neurons each layer, and neuron- wise rescalings. In order to conclude, we show that the intersection the fibers of the monomial MLPs induces equations that the rescalings must satisfy. Via torie geometry prove that those equations imply that there are only finitely many possible rescalings, concluding the proof.
L
Note that an explicit bound on the degree can be extracted from the proof. Namely, by combining Lemma B.I and Theorem B.3 it suffices that
(6m))(4-11( where 2 mnaxfdj.... 1 This bound large, and likely to be extremely loose. Still, the existence of any bound suffices for polynomial approximation purposes, since it is possible to approximate (continuous) functions with polynomials of arbitrary-large degree see Remark
4.2 SINGULARITIES OF MLPs
Here, we show that subnetworks may yield singularities of the neuromanifold. To this end, recall that point in variety is singular if its tangent space has exceeding dimension, i.c. larger than the dimension of the variety; see Section A for a formal definition. Intuitively, singularities are special points where variety exhibits degeneracy
Figure 2 for an illustration.
Before stating our result, we discuss a simple yet illustrative case, where subnetworks are known to coincide with singularities (Trager et al. 2020 Marchetti et al. 2025). Consider linear MLP, meaning that (xx)- = x is the identity polynomial. Suppose that the architecture exhibits a "bottle- neck', i.e., do, dr. d = min;:....,,.-11 d, Then the neuromanifold contains all the linear maps _d + dd of rank at most d-a space known as determinantal variety. The geometry of the lat- ter is well understood: the singular points coincide exactly with the maps of rank strictly less than d. These can be represented subnetworks, for example, by choosing A,, where d; d, of an appropriate cardinality.
We now state the main result of this section, establishing singularity of subnetworks, under an ar- chitectural assumption similar to the bottleneck from the example above.
Theorem 4.2. Suppose that a has more than dim(Ma.. non-vanishing coefficients, i.e., + ffr at least dim (Mi(Ma.)) + indices $ For every = 1,..., I.- - 1, fix proper subsets A, C (1,....,dy ).). Suppose that for some O...., L 2, A441 + 0 and d; |Ail S dg |Ayl for all j If W E W is A-subnetwork, then fw is singular point of M....
Proof. We prove the claim for a generic A-subnetwork W. This is sufficient since the singular locus is Zariski closed. For simplicity of notation, we assume dj tte same proof extends to the general case. Consider the index i E (O...., 2} provided by the hypothesis. For E A44 and & & A the chain rule implies:
Öe
W) = do
Σ AEA.
Wii o sw-11)
(6)
OW 2
In particular, it must hold that dim(Ma..)

---

Published as a conference paper at ICLR 2026
where À is polynomial, and W (W1..... W.). This expression defines vector in the tangent space of Ma,o at fw. Due to the genericity of W, we have that + O.
Now, fw and À are unaffected when the j-th row of W changes. As this row varies, the hypothe- sis on the coefficients of implies that the polynomials a CigA W [i, f] z4) over 1gd. span a linear space of dimension larger than dim(Ma.,a) Due to the hypothesis on d, |A₃ for j S i, the rank of the weight matrices W, is at least d; - [A,] for generic W.. Since, moreover, the image of the activation function is, least, a half-line in R. conclude that the image of o fw has non-empty interior in Rd in the Euclidean topology. By irreduciiility of the Zariski topology, this set is Zariski dense. In other words, fw is dominant map onto Rd.1 in the Zariski topology. This implies that if some polynomials over Rd Iddl are linearly independent, then they remain such over dR after pre-composing them by o fwitthis this follows from the general principle of duality in algebraic geometry (Dieudonne & Groothendeck]] [971 I, Corollary 1.2.7). In conclu- sion, the polynomials from equation 6 generatte a linear space of dimension larger than dim(Ma..) implying that fw is singular point.
Theorem 4.2 leaves open the question of whether all the singularities of the neuromanifold are parametrized by subnetworks see Section [ for a discussion. As mentioned above, this is true for linear MLPs, i.e., when (xx) = I. Moreover, we provide an explicit example in the appendix (Section D.I). where we manually compute the singular locus (of the Zariski closure of the neu- romanifold) for small network with a cubical activation, and show that subnetworks exhaust all singularities.
Remark 4.1. Theorem and 4.2 can be potentially extended beyond polynomial activations via polynomial approximation, as anticipated in Section []] We sketch an informal argument here rigorous proof would require details from functional analysis, and we leave it for future investigation (see Section 5). Both theorems are based on showing that some derivatives of y are linearly inde- pendent. Namely, Theorem 4. relies on showing that the Jacobian Jw is full-rank (for generic W), while the proof of Theorem 4.2 argues that some derivatives of yp at parameters the same fiber are linearly independent (see equation 6). Now, suppose that a is not a polynomial, but belongs to a space F of smooth functions where polynomials (of large degree) are dense. Examples are the space of smooth functions on compact domain with uniform norm, where density follows from the approximation theorem, or the space of functions that are analytic around 0 with the {2-distance on their Taylor coefficients, which can be approximated by polynomials by truncat- ing their Taylor series. It is natural to expect that linear independence is an an open condition in F. Indeed, since linear independence of functions is guaranteed when their Wronskian is non-vanishing (Bender& & Orszag 2013), openness follows assuming that the Wronskian is continuous over F. This means that if the theorems hold for an activation a E F, then they hold in neighborhood of (with respect to the topology of J). Since we know that they hold for generic polynomials of large degree, which are dense in F we conclude that the theorems hold in a dense open subset of T, i.c., for "almost all' a E I' (in an appropriate functional sense, depending on the topology of 9).
4.3 EXPOSEDNESS OF MIPS
Our next main result is concerned with the role of subnetworks in the optimization process. Moti- vated by Section 3.3 we consider objectives of the form Lz((f) = Q( -u), where f,u V and Q is a non-degenerate quadratic form over V. We now introduce a central notion concerned with biases of the optimization process towards subsets of the parameter space W. The notion is general, since it applies to any algebraic map yp: W + V. In particular, it can be used for any polynomial machine leaming model, including neural networks with polynomial activations of arbitrary architecture.
Definition 5. A subset 5 C W is critically exposed if the set
Ug = (u E FW E S ( = 0}
has a non-empty interior in V.
In the above, the notion of interior is intended with respect to the Euclidean topology of V, since the latter is the standard one in applications. However, all the following results will hold in the Zariski topology of V. As discussed in Section this will result in stronger statements. In fact, by irreduciilityy of the Zariski topology, if Us has non-empty interior, then it is dense. In this case,
7

---

Published as a conference paper at ICLR 2026
intuitively, 'almost all' u's belong to it, meaning that S will contain critical points almost certainly. Therefore, from now on we stick to the Zariski topology.
Intuitively, the notion of critically exposedness formalizes the presence of a bias towards S in the optimization process. Since u depends on the dataset, the weights in a critically exposed set are equilibria of the training dynamics for non-negligible amount of data. More concretely, if is sampled randomly from a full-support distribution over V, then the weights in S will be equilibria with positive probability.
Remark 4.2. If the Jacobian Jw of yp at W vanishes some weights W E W, then W critical for every E V (actually, for any loss function). In particular, the singleton S = {W} is critically exposed, and Us = V. In the case of MLPs with (0) = O, this holds for the zero weights W = 0.
Exposedness can be rephrased in geometric termss From the discussion at the end of Section 3.3 it follows that W E W is a critical point of Lu, if, and only if. the image of Jw is orthogonal to fw u, according to the scalar product over V induced by Q. In other words, the locus of u for which a given W is critical coincides with the translated orthogonal complement fw im(Jw of the image of the Jacobian. As consequence,
Us
U fw + im(Jwp)+. WES
(8)
The right-hand side is the union of a family of affine subspaces of V indexed by S. Exposedness amounts to the statement that this union has full dimension dim(Us) = dim(V). Below, we leverage on this geometric strategy to show that strict subnetworks of MLPs are critically exposed. As we shall see in the next section, this drastically differs for convolutional architectures, for which the dimensionality of the union Us is typically deficient.
Theorem 4.3. Suppose (0) = 0. For every = ,,...,,,- 1- fix A C (I....,d,} d.). with |A, < d, Let S W be the set of strict A-subnetworks. Then any open set of S is critically exposed.
Proof. Fix a strict A-subnetwork W. From the chain rule applied to equation I it follows that
Oep
if E A or j E Ag-1,
(W)
aw, [..
Dpls &
(W)
otherwise.
In the above, sp S denotes the restriction o to S. Therefore, V(C., o (p) W is obtained
by padding (L slss) (W) with vanishing entries. In particular, if W is critical point for
Lu o plss is critical for L., o &o as well. Hence, im(Jwpls) in equation contaiis the embedded normal bundle of (the smooth locus of) (SS). This bundle has
im L and so the union Us
full dimension dim(V), even when restricted to the image of an open set T C S. This shows that Ur is full-dimensionall and exposedness of T follows.
O
Remark 4.3. Theorem 4.3 holds also for activations with (0) * O, if one uses a different definition of strict A-subnetworks W: namely, for every layer i and every j E A,, the j-th column of W vanishes and the j-th row of W; coincides with its k-th row for some k & A, These repeated rows in the weight matrices lead to repeated columns in the Jacobian Jwip, instead of O-columns as in equation
The proof of Theorem 4.3 shows actually more: since Jw is obtained by adding vanishing columns to Jwipls. the gradient descent dynamics of & & and of & o øls are isomorphic over S for any differentiable loss L: V - IR. Put simply, from a dynamical perspective, strict subnetworks are equivalent when seen as embedded in W or as MLPs with a smaller architecture. Similar consider- ations, with analogous Jacobian arguments, have been made in (Chen et al. 2023).
4.4 COMPARISON WITH CONVOLUTIONAL NETWORKS
So far, we have considered neural networks with a fully-connected architecture. In this section, instead discuss convolutional networks classical architecture originated in computer vision (Fukushima 1979 LeCun et al.) 1995). We start by recalling the basic definitions. To this end, fix positive integers k. 8, d' E N representing filter size, stride, and output dimension respectively. The
8

---

Published as a conference paper at ICLR 2026
MLP
CNN
Figure 2: Illustration of the different types of singularities (orange) arising in the neuromanifolds of ML.s and CNNs.
convolution between a filter w RR and an input vector x E Rdd with s(d' 1) + k, the
vector w Ky x E IR' defined for 1 I i as:
(w * x) Σ
wlil x<s(( 1))
(10)
Note that we consider one-dimensional convolutions for simplicity of notation, but the theory holds similarly for higher-dimensee ones. Indeed, our arguments build upon (Shavverdi et al.] (2025).
whose results apply verbatim, with the same proofs, higher-dimensional convolutions.
Now, fix sequences k,s Z211. d z such that d; s,(d,-1 + ks) for all i, and vectors W (w,,.... w... E D
Definition 6. A Convolutional Neural Network (CNN) with architecture (k, d) and weights w is the map fw: Re Rd given by: fw (x) wL *av. (-.. tsz o(w1 *,, x)).
(11)
where a is applied coordinate-wind
Similarly to MLPs, via abuse of notation, we denote by W + i=1 Rh: the parameter space, by pp: W 3 w ト fw the parametrization map, and its mmage-ie.., the neuromanifold--yy MK..d.
The geometry of the neuromanifold is well understood for monomial activation functions (xx) = x" (Shahverdi al. 2025| Kohn al. 2022). Below, we extend the main results from (Shahverdi et al. 2025) to general polynomial activations with (0) = 0. Specifically, we show that & is regular in W. & (0), meaning that its Jacobian Jw has full rank for every w E W (0). The reason exclude the fiber of the zero function is that, similarly to MLPs, the Jacobian is rank deficient at w if fw = (see Remark 4.2h.
Theorem 4.4. Let a be generic polynomial of large enough degree >>>>>>>>>> 0 (depending L) with (0) 0. Then the parametrization map &p restricted to W \. x (0)) is regular, generically one-to-one, and its remaining fibers are finite. In particular, dim(Ms...d..)
dim(W)=IX = kg-
The proof is provided in Appendix C.I 4.44 isfied by the parametrization of an MLP (cf. Theorem 4.1) As such, it has important consequences terms of singularities and exposedness. Specifically, since the parametrization yp regular, does not induce spurious critical points of the loss in parameter space (Trager et al.) 2020). i.e., all critical parameters w (away from y (0)) yield critical functions fw. Since moreover all fibers of (excluding 0) are finite, we conclude that the singular points of Mr.s.d.r \ (0) are nodal, i.e., they arise from celffintersections. Differently from the case of MLPs, these singularities are of mild type- see Figure for illustration, and do not cause critically exposedness. In fact, we now show that, once the fiber of 0 has been excluded, no algebraic set can be critically exposed. This paints a completely different picture than in the case of MLPs (cf. Theorem 4.3).
stronger property than the one sat-
Proposition 4.5. Assume the hypotheses of Theorem 4.4 Let C. W be an open set of an algebraic variety strictly contained in W. 1fO & (PS), then S is not critically exposed.

---

Published as a conference paper at ICLR 2026
The proof is provided in Appendix C.3 define equilibria for the training dynamics of CNNs has been sometimes observed empirically. For example, it has been established that initializing CNNs with almost all vanishing weights does not result in a collapsed dynamics. Instead, the network recovers from the initialization, and learns well (Blumenfeld et al. 2020).
The fact that, differently from MLPs, subnetworks do not
Nevertheless, subnetworks of CNNs can yield singular points of the neuromanifold. In fact, we show now that all singularities arise from subnetworks in the case of convolutional architectures. Since CNNs exhibit a weight-sharing pattern, the notion of subnetwork is different than for MLPs. Specifically, we wish subnetworks to be reparametrizable by smaller architecture, i... by smaller filters. We will think of the latter as being represented by filters that are padded by vanishing entries on the left or on the right. This leads to the following definition. Pick integers AI,..., Ar, such that 0 S A; I k; for every i, and define A = (A1...., At).
Definition 7. A parameter E W is an A-subnetwork if for each = I,..., I, w: []] = 0 holds either for all j = 1...., A, or for all j k; A, + I,..., ky The subnetwork is proper if A, > for at least one i.
Given an A -subbetwork, we denote by & E Z the cardinality of A,, equipped with a positive or neg- ative sign corresponding to whether w; padded with zeros on the left or on the right, respectively. We also recursively define io = 0, [ = L + [4-1 44:44111ss-1 1/s for i 2 1. The proof of the following claim is provided in Appendix C.2 alongside an example illustrating the condition on the &, in Appendix D.2
(0). Then fw is a singular point of Mx.s.d,o if. and only if. w is proper A-subnetwork such that i E Z for all i, and it = 0.
Theorem 4.6. Assume the hypotheses of Theorem 4.4 Let w E W x
CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
We have considered neural networks with an activation function that is sufficiently-generic poly- nomial. By leveraging arguments from algebraic geometry, we have studied questions related to identifiability, singularity, and exposedness for MLPs and CNNs. Overall, this work addresses open problems in neuroalgebraic geometry (Marchetti et al. 2024), and provides a novel geometric per- spective on the sparsity bias of neural networks. Yet, it is subject to limitations, as outlined below.
states that subnet- works of MLPs can parametrize singular points of the neuromanifold. However, it is unclear whether this describes all singularities. For deep linear MLPs (Trager et al.) (2200) and for deep polynomial CNNs (see Theorem 4.6), all singularities come from subnetworks. The problem of full charac- terization of the singularities of polynomial MLPs left open. Similarly, a complete classification of critically exposed parameter subsets for polynomial MLPs, beyond the subnetworks described in Theorem 4.3 and Remark 4.3) is crucial for a complete understanding of the biases of deep networks.
Complete characterization singularities and exposedness. Theorems 4.2
Type of critical points. Our analysis has only considered whether weights are critical points of the objective, disregarding the type of criticality (local minimum/maximun all critical points are equílibria of the dynamics, local minima correspond to the (local) attractors. Thus, it would be interesting to incorporate the type of critical points in our analysis; in particular, to understand whether subnetworks are local minima for nen-nggli amounts of u e V.
saddle). Although
Beyond the algebraic. Throughout this work, in order to leverage on tools from algebraic geom- etry, we have focused on neural networks with polynomial activations. However, several popular activation functions are either piece-wise polynomial (c.g., ReLU), or completely non-algebraic (e.g., Tanh and SoftMax). Extending our results to the non-polynomial case therefore important. As mentioned in Section 1 a promising strategy to this end is polynomial approximation; since polynomials can (locally) approximate arbitrary continuous functions, neuromanifolds neural networks can be approximated by algebraic ones. This approach is, generally speaking, part of the research program of neuroalgebraic geometry (Marchetti et al., 2024), and has been often fruitful for extending results for polynomial networks beyond the algebraic domain (Boullé et al.) 2020 Zhang & Kileel, 2023). To this end, we have sketched an argument in Remark 4.1 However, rigorous proof comprising all the functional analysis details is left for future investigation.
general
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Pro- gram (WASP) funded by the Knut and Alice Wallenberg Foundation.
REFERENCES
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Shun-ichi Amari, Tomoko Ozeki, and Hyeyoung Park. Learning and inference in hierarchical mod- els with singularities. Systems and Computers in Japan, 34(7):34 42, 2003.
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamics of learning in neuromanifolds. Neural computation, 18(5): 107--165,, 2006.
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivii and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learn- ing. pp. 903-925, PMLR, 2023.
Carl M Bender and Steven A Orszag, Advanced mathematical methods for scientists and engineers I: Asymptotic methods and perturbution theory. Springer Science & Business Media, 2013.
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond signal propagation: is feature diver- sity necessary in deep neural network initialization? International Conference on Machine Learning, pp. 960-969 PMLR, 2020.
Joachim Bona- Pellissee,, François Bachoc, and François Malgouyres. Parameter identifiability of deep feedforward relu neural network. Machine Learning, 112(11)444114445 2023.
Nicolas Boullé, Yuji Nakatsukasa, and Alex Townsend. Rational neural networks. Advances in neural information processing systems, 33:14243-14251 2020.
Ovidiu Calin. Deep learning architectures. Springer, 2020.
Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gra- dient noise attracts sgd dynamics towards simpler subnetworks. Advances in Neural Information Processing Systems, 36:35027-35063, 2023.
Felipe Cucker and Steve Smale. On the mathematical foundations of learning. Bulletin of the American mathematical society, 39(1):1-49, 2002.
Jean Alexandre Dieudonne and Alexandre Grothendieck. Éléments de géométrie algébrique, volume 166. Springer Berlin Heidelberg New York, 1971.
Charles Fefferman et al. Reconstructing a neural net from its output. Revista Matemática Iberoamer- icana, 10(3):507-556, 1994.
Bella Finkel, Jose Israel Rodriguez, Chenxi Wu, and Thomas Yahl. Activation thresholds and ex- pressiveness of polynomial neural networks. arXiv preprint arXiv:2408. 04569, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Kunihiko Fukushima. Neural network model for a mechanism of pattern recognition unaffected by shift in vosition--necconitnee IEICE Technical Report, A, 62(10):658-66 665, 1979.
Daniel R. Grayson and Michael E, Stillman. Macaulay2, software system for research in algebraic
geometry. Available at hittp://wee wwwp/wwwwww...m...// wwtp:/maccco www.uuuauuuucmmmm..mmm www.amaaaaa....ommmm
Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks. In International Conference Machine Learning, pp. 11734-11766 PMLR, 2023.
Nathan W. Henry, Giovanni Luca Marchetti, and Kathlén Kohn. Geometry of lightning self-
attention: Identifiability and dimension. International Conference on Learning Representa- fions, 2025.
11

---

Published as a conference paper at ICLR 2026
Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.
Kathlén Kohn, Thomas Merkh, Guido Montúfar, and Matthew Trager. Geometry of linear convolu- tional networks. SIAM Journal on Applied Algebra and Geometry, 6(3):368-406, 2022.
Kathlén Kohn, Guido Montúfar, Vahid Shahverdi, and Matthew Trager. Function space and critical points of linear convolutional networks. SIAM Journal on Applied Algebra and Geometry, 8(2): 333-362, 2024.
Kaie Kubjas, Jiayi Li, and Maximilian Wiesmann. Geometry of polynomial neural networks. Alge- braic Statistics, 15(2):295-33) 2222,, 2024.
Daniel Kunin, Jonathan Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regu- larized linear autoencoders. In International conference on machine learning, pp. 3560-3569. PMLR, 2019.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-667, 1993.
Derek Lim, Theo Putterman, Robin Walters, Haggai Maron, and Stefanie Jegelka. The empirical im- pact of neural parameter symmetries, or lack thereof. Advances in Neural Information Processing Systems, 37:28322-28335 2024.
Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of learning: Universal fourier features emerge in invariant networks. In The Thirty Seventh Annual Conference on Learning Theory, pp. 3775-3797. PMLR, 2024.
Giovanni Luca Marchetti, Vahid Shahverdi, Stefano Mereta, Matthew Trager, and Kathlén Kohn. Algebra unveils deep learning an invitation to neuroalgebraic geometry. In International Con- feerecce Machine Leurning, 2025. URL BRRLhhtts:///axttta http//axxi....ga...aa/a/ htlp://bb/225111.. 2501 18915
Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pp. 16270-16295. PMLR, 2022.
A. Emin Orhan and Xaq Pitkow. Skip connections eliminate singularities. In International Confer- ence on Learning Representations, 2018.
Scott Pesme, Loucas Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34:29218-2222 22221182222000 2021.
Mesut Şahin. Rational points of lattice ideals toric variety and toric codes. Finite Fields and Their Applications, 90: 102226, 2023.
Igor. R. Shafarevich. Basic Algebraic Geometry Vol 1: Varieties in projective space. Springer, New York, 3rd edition, 2013.
Vahid Shahverdi. Algebraic complexity and neurovariety of linear convolutional networks. arXiv preprint arXiv:2401.166113 2024.
Vahid Shahverdi, Giovanni Luca Marchetti, and Kathlén Kohn. On the geometry and optimization of polynomial convolutional networks. In Artificial Intelligence and Statistics, 2025.
Matthew Trager, Kathlén Kohn, and Joan Bruna. Pure and spurious critical points: geometric study of linear networks. In International Conference on Learning Representations, 2020.
Konstantin Usevich, Clara Dérand, Ricardo Borsoi, and Marianne Clausel. Identifiability of deep polynomial neural networks. arXiv preprint arXiv:2506. 251093506..7093, 2025.
12

---

Published as a conference paper at ICLR 2026
Verner Vlačić and Helmut Bölcskei. Neural network identifiability for family of sigmoidal non- linearities. Constructive Approximation, 55(1):173-22 224,, 2022.
Zihan Wang and Arthur Jacot. Implicit bias of sgd in (eeglrrggllrr linear dnns: One-way jumps from high low rank. In International Conference on Learning Representations, 2024.
Sumio Watanabe. Almost all learning machines are singular. In 2007 IEEE Symposium on Founda- fions of Computational Intelligence, pp. 383-388. IEEE, 2007.
Sumio Watanabe. Algebraic geometry and statistical learning theory, volume 25, Cambridge uni- versity press, 2009.
Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari. Dynamics of learning near singularities in layered networks. Neural computation, 20(3):813-833, 2008.
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's s good. IEEE Transactions on Neural Networks and Learning Sys- lems, 34(12): 10473- 2022.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 5635-3673, PMLR, 2020.
Yifan Zhang and Joe Kileel. Covering number of real algebraic varieties and beyond: Improved bounds and applications. arXiv preprint arXiv:2311.05116, 2023.
Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of deep linear network. Advances in Neural Information Processing Systems, 25:24446-24458, 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A SINGULARITY AND CRITICALITY
In this section, we aim to clarify the distinction between singular points of a variety and critical points of its parametrization. Although both notions involve degeneracies, they can arise from dif- ferent mechanisms and have different implications for optimization. Singular points are geometric features of the variety itself, reflected in the dimension of the tangent space, while critical points depend on the differential of the chosen parametrization.
To formally define singularities of variety X S RR. we assume that X is irreducible (i.c., it cannot be written the union of two proper non-empty subvarietics) and consider its vanishing ideal Ix that consists of all polynomials in R R[21..... Fn, that vanish along X. We then fix a generating set Jacobian matrix J whose (i, j-th entry is
(f1, -f.....,JJ. fa) Ix for that ideal and compute the 0f. For almost every point on X (i.e., except for p on some proper subvariety), the rank of (p) is Bay"
the same. Those are the smooth points of X, and at those points the kernel of (p) is the tangent space of X p. At the remaining points on X, the rank the Jacobian drops; those are the singular points. They form subvariety of X, defined by the ideal Ix and the x minors of the Jacobian, where e is the rank of (p) smooth point p X.
Through the classical examples of nodal and cuspidal cubic curve, we now illustrate how the concepts of singularities on varieties and critical points of parametrizations differ, and how tangent spaces play key role in their characterization.
1.5
2.0
1.5
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
-1.0
-1.0
-1.5
-2.0
-1.5
0.0
0.5
1.0
1.5
2.0
(a) Nodal cubic
(b) Cuspidal cubic
Figure 3: Two singular cubic curves.
Example A.1. Consider the parametrization of two singular cubic curves:
: R + IR? w:: R PR-RR R
+ (t2 1, t(t² - 1)) (nodal cubic y2 = x x2 + 1)),
(12)
t (t,
(cuspidal cubic y² = x²)
(13)
In both cases, the origin is a singular point of the curve; see Figure 3 For the nodal cubic, every point the curve except the origin has fiber of size one. At the origin, however, the fiber is unusual, since it contains two distinct parameters: g (0, 0) = {-1, 1}. In contrast, the cuspidal parametrization is injective, i.e., s (z, y) consists of single point for every (z, E im()).
The Jacobians of both parametrizations are
Jep 12t
3t²
1].
(14)
J. (21 31
(15)
Thus, &p is regular at every (i.e., the Jacobian never vanishes), while & is critical at 0 (where the Jacobian [0, 0). In other words, the cuspidal parametrization is not regular at the cusp, although does not have any unusual fibers; its singularity arises solely from the drop in rank of the Jacobian.
14

---

Published as a conference paper at ICLR 2026
To algebraically that the origin is indeed the only singular point of both curves, compute the
Jacobian of their defining equations f(z y) = x² + 1) y² and g(x, y) = x² y². respectively:
J [3x² + 2x
-2y],
(16)
Jg
[3x² - -2y].
(17)
Both vanish precisely when (z, (0, 0). At all other points the curves, the kemnl of the Jabobian is the 1-dimensional tangent space. At the singularity (0,0), 0). the Zariski tangent space is the whole 2-dimensional ambient plane R2a For the node, the plane is spanned by the distinct tangent directions corresponding to / = -1 and t= I, with slopes and 11, respectively. For the cusp, however, the 2-dimensional fangent space arises from the vanishing of all first-order terms, which corresponds geometrically to infinite curvature at the origin. In particular, since Jo 0. this tangent space cannot be detected solely from the diffential of the parametrization.
This example illustrates that critical points of a parametrization can cause singularities the image variety, but that not all singularities of parametrized varieties arise like that. Moreover, not every critical point of parametrization leads to singular point on the image. This be for instance seen by projecting the cuspidal curve in Figure 3b onto y-axis: Composing the cuspidal parametrization with that projection leads parametrization everywhere) that is not regular at / = 0. All in all, critical points of parametrizations and singularities of their image varieties are subtly related, but non implies the other in general.
t of the line ER' (which is clearly smooth
The effect that different types of singularities have when optimizing loss over high-dimensional algebraic varieties is largely unknown. Cuspidal-type singularities, arising from a rank drop of the parametrization's 's Jacobian, may appear both as spurious (Trager et al., 2020) and genuine critical points of the loss. In the case of MLPs, we show that these types of singularities are critically exposed. In contrast, nodal-type singularities do not exhibit this behavior. This reflects the main difference between the singular points of MLPs and CNNs (see Figure ( since we show that all singularities of CNNs are of nodal type.
B PROOF OF THEOREM 4.
In this section, we prove Theorem
We begin with some technical results.
Lemma B.1. Let (x) = 9L +... + & be polynomial activation function such that 33 > 8/-1 for every 1, and B₂ 1. Define 0g, (x) B.. Then for all weights W E W, the MLP can be decomposed as:
fw(x) Σ Wiog,W
R(x),
(18)
where the remainder R(x) does not contain any monomial of degree 8/-² for every j 1,....A L.
Proof. It suffices to show that monomials of degree 8/-1 appear solely in the term
Wrop, WT-10 W Wi.2...... -g, Wi (x))..
(19)
We verify this by tracking the degrees generated layer by layer.
First, note that the monomials of degrees 8--1 and s clearly only generated by the terms
WLOg, W WL-10 Wi.,,,,,WWW-..... as (i)) and WLOS, WL- WL-2 0z Wi()),
(20)
respectively. Thus, we focus our analysis on degrees 3/-1 1 for 1 < L.
We proceed by induction on the number of layers k. where < < L. Specifically, we claim that at layer k monomials of degree s/ appear exclusively through the composition
WLa,, WE-10T ".. ". (
(21)
15

---

Published as a conference paper at ICLR 2026
Moreover, claim that the closest degrees immediately preceding and succeeding 03-1 given by (3) 1)55 + BJ-1 and (s8-2 1)31 + BJ+1. respectively.
For 2, note that the degrees appearing at the second layer are
8
82
B BL,
(22)
and thus R(x) = 0 at this stage.
For the induction hypothesis, assume that the claim holds for some layer & < L. That is, mono- mials of degrees 3 are solely generated by equation 21 at layer k, and the sequence of degrees generated has the form
1 B2 - B B
(23)
< 3 8k-1- - + Bi+1 < ... <
1/- 1 32 + BL-1 < 3 B/-1
where intermediate terms represent monomials whose degrees lie strictly between the bounds above. For the inductive step, consider the (k: + 1lllayer composition, expressed by
We
Σ WK+ WW
(24)
To obtain monomials of degree exactly 8 we analyze each summand individually. For any choice of Øg with j * i, we claim that no monomial of degree B emerges. Hence, the only valid choice is j = i. To see this, we consider two cases
j < i In this case, to obtain a term of degree at least 3 in the multinomial expansion of 0g,
we have to use at least one the monomials of Weo ( of degree larger than 8h-1 By equation 23 the smallest possible degree is 8 13 + B However, by our assumption, this is strictly larger than 3 2 BB.
j In this case, since the smallest degree appearing in WEo... (Wi(2) according to equation 23 the smallest degree after passing through σg, 3 > 33 > 31-1 Hence, all terms are of degree larger than 13
(x) is B
Moreover, the neighboring degrees of B in Jz WLO ... WWi are
(3: - 1) (B) 3, + 3 8 = --+
(B1 - (8) B1 + B141).
(25)
= -
This exactly confirms the inductive step, hence the uniqueness of the decomposition in equation 18
Lemma B.2. Let E1 > C2 be positive integers of opposite parity. For i = 1..... L. let W Rdixd, be generic. For = 1,2, define
A, = {D,,, D,,WWDD7.'.
W.D Dj1k E diag (R) Vk 1, I,..., L 1).
(26)
where diagā, (R) is the set of real invertible diagonal matrices of size dy x de. Then A A consists exactly of those tuples in Ag for which each D1,k A1,ed is multiple of the identity for some \1 E IR. \ {0) that satisfy
1-2 6
1-33
A1.1
X
1
(27)
Proof. Take (Wi,... Wr E A n A2. By definition, there exist diagonal invertible matrices Dj.k such that for K = 2,... L 1:
W D1, D2,1 W1,
Wr D We DT
D2 W
(28)
16

---

Published as a conference paper at ICLR 2026
Since generic matrix admits diagonal symmetries, D1,1 D2,1 D. For A 2 the second equality becomes
D1, W2 = W2 DA,
(29)
where = 61 - C2. Left multiplication by diagonal matrix scales rows, while right multiplication the same scalar multiple of
scales columns. Thus, since W2 is generic, both DA and D2,2 the identity. Since A is odd, this yields D = L I and 2,. = Xi D1,2 for some 11 E R \ (0}.
By plugging this relation into the equality for & and repeating the same argument shows that D1,2 \1,2,, and that D2.3 >T X D1.3. Inductively, for every t 1,..., L- 1 we get
-
D1.-1=
D2.
112
1 DI,k-
(30)
At the last layer, the equality W,, W.D-I
WT. Wi. gives D2,1-11 11 Using the equality
for k L 1-1 in equation 30
get
Aei L-3 -
D
11y
112
2T
(31)
Hence, DI,L-1 is also a scalar multiple of the identity matrix, say A1,L-1I, and thus
1.2
(32)
Conversely, direct computation reveals that these scalar multiples of the identity yield indeed point in the intersection A, n A2.
Next, we recall a result from literature on identifiability of MLPs with monomial activation func- tions.
Theorem B.3 (Finkel et al. 20244) Suppose that (xx) = x², with 2 6m2 6m, where 2 (d,,,.,,,,........... For generic W E W, the fiber :p (Jw) consists of weights of the form
(i,,, Pr2Dr.222D,5 P.T.
WLD W.DI,L-1PP... W.D.L--P.TL-)
(33)
where Pt. are permutation matrices and D are invertible diagonal matrices of size dj dy.
We can finally prove the desired theorem.
Proof of Theurem 4.7 fix a sufficiently large degree r. It sufficient show that there exists a polynomial a of degree for which the theorem holds. Indeed, y having finite fibers is equivalent to its Jacobian attaining the maximal rank dim(W) at generic weights W. The condition that the Jacobian has maximum rank dim (W) is open with respect to the Zariski topology in the coefficient space of a. Since open sets are dense, this condition holds for one polynomial activation of degree holds for generic one.
Consider the sparse polynomial
L
(xx) = Σ
x
(34)
iml
where 3, > 84 for all (note that Br = r). Set 0g (I) = B: Then, by Lemma B.1 the output of the MIP uniquely decomposes as
WLog, WLO1,WW-10,W 0z (1() + (remaining terms).
(35)
Each monomial term in equation 35 is an MLP with monomial activation 8 By Theorem B.3
if choose $1 2 6m? 6m, where 2 max(d1,.. dz-1). then the generic fiber of the parametrization map of such an MLP consists of weights of the form
D,, Wa PT
W.D PT W.D.L.PP.LL.. P.......)))
(36)
17

---

Published as a conference paper at ICLR 2026
where Pt, are permutation matrices and Di, invertible diagonal matrices of size d, x dz. For generic parameters W, the fiber (fw) is contained in the intersection of the sets of such tuples. In other words, every tuple the fiber &p (fw) can be expressed as in equation 36 for every i i= I....... L.
Since W is generic and each Di,; diagonal, it follows that all the corresponding permutation matrices must coincide at the intersection. That is, we have Pt. Pr.j for all i, k. Since we wish to show finiteness of fibers, without loss of generality, we may assume that these permutation matrices are identities.
Now, by comparing tuples equation 36 for different indices i, Lemma B.2 implies that each diag- onal matrix reduces scalar multiple of the identity, that is, Dz.; > I for some Xi.j IR.. Moreover, the intersection is characterized explicitly by the polynomial system:
? - 11=0,
B2 LLL
L,2
(37)
22,1
L2
81 L-L-2
- 0.
We use the lattice-ideal approach in toric geometry to show that this system has finitely many solu-
tions; see (Saiin 2023 Section 7). Let A[i,j] = Then A is Vandermonde matrix of size
(L - 1) L
with determinant
det =
II
(8, - (1) #
(38)
1SKjEL-1
Using the Smith normal form, there exist matrices U.V E GLL-1 such that
U JAV-
(39)
where l 1 l2 ... LL-1- Letting A (AL,I )L,L-1) E (C*)4-1, define new coordinates = (WI...., WL-1) via
L-1
Wy = II L
(40)
Since V is unimodular, the map + is an algebraic automorphism of the torus. In these new ft-1 coordinates, the system equation 37 is equivalent to w =1 .... W 1, which has exactly l... 6 oo solutions in (C*) 6-1, Hence, (fw( is finite, and the claimed dimension follows directly from the fiber-dimension theorem.
C POLYNOMIAL CONVOLUTIONAL NETWORKS
In this section, we provide technical details for the proofs in Section 4.4 We first establish general result on univariate polynomials (zx) = Σi=o a,z".
Lemma C.1. Suppose that > 2 and that ar, ar-1 # O. Moreover, let uu E R \ {0} and R be such that, for all R.
d Ag....,
ALuL of - (f
(A121))...)) uu mcouuL- of... o(uxx)...)) ((uu)))...uuuzx.........
(41)
Then, = =1 1.
Proof. Consider the following rational function:
P(x) [ o(x)
(42)
which is well-defined wherever (xx) # 0. Define inductively:
Go(() A1u1z
[Ho(v) = uix,
(43)
Gx(z)= = A Gxx)1(()))
(x(()) = ux+1o (Hz-
18

---

Published as a conference paper at ICLR 2026
The left-hand side of equation 41 coincides with GL-1(z) Yauy (G)-2(1)) while the right-
hand side (x) ugo (H)-)(z)) Now, by differentiating equation 41 and then dividing by equation 41 we obtain
Gy 2(H1.22 GL-2
1
P(GI.-2)
(44)
HL-2
By applying the chain rule iteratively, the above identity equivalent to
L-2
1-2
II P(G)(1)) = II (H)())) k=
(45)
A=
We assume for contradiction that A + for some k. Let min 0 < h: S --2 21 Ac+1 * 1)-
For every < m, we have that G₁ = Hj. Moreover, Gm (x) Am+1 Hm (x) with Am+1 # 1. In particular, equation 45 becomes
L-2
1-2 II P(Ha(z))
II P(Ga(=)) k=
(46)
Now, set B ; 9r-1, As oc, the rational function P satisfies P(x) - 2 +=xxx=2) Thus,
Br
for every k: 0, we have
B
3 Hx(x))
P(Gx(x)) = + (x(x))
+ O(r Gdd
P(Ha(=))=r) -r+
+ Ox -x-g2dg
(ddoHH)) (47)
Since deg Gk = deg H = r the terms of order up to -p in equation 46 are
L-2
L-2
3
B ) + O(x 0x- (48)
II k==+1 7) (r+ + Gm(r) + O 111--
II
(r+
Hm(
k=m+1
Using Gm(x)= = =m++1 Hm (z)) we obtain
Am+1 Oz-ra-1))
B Am+1 Hm(
(49)
Since the left hand-side is of order 0xx and 3 of by hypothesis, conclude that Am++ which is a contradiction
1.
Corollary C.2. Suppose that r > 2, ag = and ar,, ar-1 + O. Moreover, let w1 E IR .R......, Wt E R&L be non-zero filters and let λ1 dL E ER be such that, for all 1R:
ALWL *sa. (... o(A111 ts, x)) = wL tsx o (... * oww t, x))
(50)
Then, A1 -.. d =1 1.
Proof. Let i = 1,..., dg be the smallest index such that the variable 2, appears in one of the monomials of equation 50 with non-zero coefficient. Substituting zj + 0 for all * i in equation 50 yields the equality of univariate polynomials in equation 41 where ug is the first non-zero entry of the filter Wk. Thus, the claim follows from Lemma C.1
C. PROOF OF THEOREM
Proof. We first show regularity. Similarly to the proof of Theorem 4.1 we will do so for a specific activation function, which will imply the statement for generic one. Consider the polynomial activation defined by
(xx) 3t ... "
(51)
where 81 > 1, 8r = r, and 3, > -5--111 for all j > 1. We will show that the parametrization map oo with this activation is regular W \ x ( Note that p0-(0) consists precisely of those filter tuples where one of the filter is zero, because convolving with a non-zero filter a full-rank linear map (Shahverdi et al. 2025
Lemma 4.1).
19

---

Published as a conference paper at ICLR 2026
By Lemma B.1 the CNN output decomposes uniquely as
Sw wL *sz 0g (...*ay *o2 03 (wi May x)) + (remaining terms).
(52)
From (Shahverdi al. 2025 Proposition A.2), if fw 7 O, the kernel of the differential of the CNN parametrization with monomial activation oıs, is explicitly given by the subspace
A, = (,,, 1, + A.2))u,,....
+ \LL--1)ur-1, (-x.,,,...
- -A,,L-1 Xi.5 E ]]
(53)
Using the decomposition equation 52 we deduce that the kernel of the Jacobian Jw is contained in the intersection N Ai. Now, let Vy E Rh contain the coefficients of the filters wp in A, and define
7/ (8 .p/- ..... ... T)
(54)
Then, (7), 03)) 0. Hence, in
1 1 A,. we must have
1/6 BL
BL
8111 $1=7
3L-1
UL 0.
(55)
81-1
B1-2
B1
Since 3, - B, + for all i * j. the matrix in equation 55 is full-rank Vandermonde matrix, which is invertible. Therefore, the only solution is ey = 0. It follows that n A, = {0}, and hence the parameterization map oo has an injective differential over W 1>p ((0). In other words, op is an immersion away from yp (0).
Since yp is an immersion at generic weights w E W, it necessarily has finite fibers. We wish to show that the generic fiber is singleton. To this end, choose weights
(W1,..., w1., where w; (1, ...,,,..1. 1) for all i. From (Shahverdi et al. 2025 Theorem 4.6), it follows that the fiber p (fw) is contained in {(A
A; E KJ. But then Corollary C.2 implies that ( = {w} is a singleton. Since is regular at w, there is an open neighborhood around w such that every fiber remains singleton. Since open sets are dense in the Zariski topology, we conclude that the generic fiber is a singleton.
C.2 PROOF OF THEBREM 4.6
Proof. Since op is regular, generically one-to-one and has finite fibers over W\ ((), the property that fw is singular point is equivalent to the corresponding fiber yp ( not being singleton.
We start by assuming that fwr * a singular point, i.e., there is another filter vector w' such that fw fw. By Corollary C.2 w' differs from by more than just layerwise scalings by constants.
The monomials of maximal degree appearing in fw() are
WL *az. oy (... *2 (y *a, .))= w'y *ez. Tr (...tez " *a x))
(56)
In particular, and w' also give rise to the same monomial CNN. Now. (Shahverdi et al. 2025
Theorem 4.6) states that w is a proper A-subnetwork for some A, and that the filters in w are shifted versions of the filters in These shifts need satisfy i, Z and i, 0 according to (Shahverdi et al. 2025 Remark 4.2).
For the converse direction, we consider a proper A-subnetwork w that satisfies the assumptions of the statement. To this end, define new weights w' = (wi..... ....., w²,) as:
[ (w:l4: + 1}, w: [4 + ..................... ., w. [ka], ........0 0), // { (0. ........00 0, w: [1]. w. [2]..... w; lk + t.]),
2 0, 4 5 0.
(57)
Since each filter w, is non-zero and t, 0 for some it holds that # w'. Since &z = 0, by leveraging on the equivariance property convolutions, it is an immediate calculation to verify that fw fw', desired.
20

---

Published as a conference paper at ICLR 2026
C.3 PROOF OF PROPOSITION 4.5
dim(V) - d. for every W E S. Since has finite fibers, equation Baciiionnddee defines a family of affine subspaces of V, where only finite number of subspaces is indexed by a given point in (oS).. Therefore, dim(Us) I dim( + dim(V) dim(MMx.a.d.o. Since is contained an algebraic variety strictly contained W, dim((p(S)) < dim(Mx....r We conclude that dim(Us) dim(V), implying that Us must have empty interior.
Proof. Since if regular, dim (im (Jw:))
D EXPLICIT EXAMPLES
In this section, we discuss explicit examples of both MLPs and CNNs.
D.1 MLP
In the following example, we compute the singular locus of a shallow MLP with polynomial activation, and verify that the subnetworks are critically exposed. The computations are performed via Macaulay2 (Grayson & Stillman) symbolic algebra software.
Consider the shallow MLP with architecture (2, 2, 1) and activation (<r) = x²3 + x² and parametrization
o {(c d) (c.)) (aPe +e (3a²be + df) aña²
(58)
+ (a2e + ce (3ab² 3cd f) 21
+ (2abe + 2cdf) uι + (b'e + x + (b + +f
We identify the output with its coefficient vector in IR relative to the lex-ordered basis (rf, of of. 21 0132, xy 2z), and write
s (
le, =t =t -
- (t))...,tt).
(59)
where t, denotes the coefficient of the i-th basis monomial. Then the Zariski closure Ma.o of the neuromanifold Ma IR7 is the hypersurface 0, where
F 2taf tot4ts 6t2tata 96 + 212t 6t,4477.
(60)
The singular locus of Ma. is the subvariety defined by VF = 0. After simplifying the equations, this locus a 3-dimensional variety cut out by the following polynomials:
3tzto 2t447,
3tats t2t7,
tats - 2t2t7,
t2 - 67,47,
(61)
36246
1at4 - 3t,t7,
1244 - 9t,
21zl3 3t to.
2 3t
These equations precisely correspond to the subnetworks, that is, to those weight matrices W (W1, W2) for which either e = 0, or f = 0, or = and =
We now show that the parameter points corresponding these singularities critically exposed. To this end, note that the Jacobian the parametrization with respect to (a, b, c, d, e, f) is
3a
0
3c f
0
a
6abe
3a e
Godf
3c f
3a
3cd
2ac
0
2cf
a²
?
Jwip
362e
6abe
3df² f
6cdf
3ab²
3ad
(62)
2be
2ae
2df
2c
2ab
2cd
0
36?e
3d f
b3
df
26e
2df
62
d2
21

---

Published as a conference paper at ICLR 2026
and evaluating at points with 0 gives
3a
a
e
6abe
3a
0
ba
3dd
2ae
0
0
a²
362e
6abe
3ab²
3ed²
(63)
2be
2ae
0
0
2ab
2od
362e
6²
d3
2be
6²
d²
The tangent directions to the subnetwork f = 0 are (a, b, c, d, e); the c, d columns vanish here, so the effective Jacobian the subnetwork is the 7 matrix formed by the (a, b, eocouumss, which we denote by - The 3 3 minor using the first three rows equals ea Hence, for general W with f = 0 and * and * O, we have that
rank Jy 3,
(64)
which shows that
dim u IR?
- = 4.
(65)
As consequence, the right-hand side of equation has dimension 3+4 = 7, which equals the ambi- ent dimension TR?", This shows, from a is critically exposed.
perspective, that the set of subnetworks
D.2 CNN
Here, illustrate the technical condition in Theorem 4.6 for minimal example. We consider the shallow CNN
[d el
a
b
(66)
0
0
of stride 2, with filters = la b
c] W2
ld e]. with imput = [x]
32
T3
24 xs]
and with activation oyy) = y² + y. Using Macaulay2, we can easily check that the network parametrization map yp regular (away from the parameters w1 function) and almost everywhere injective.
wo mapping to the zero
We can obtain many subnetworks following Definition e.g., by setting a O, b or = 0, or a = b = e 0. Out of those subnetworks, we will see now that only the latter constitutes singularity.
The subnetwork = = = 0 corresponds to the function x + (-o(cEs), which can be embedded two different ways as a subnetwork into the original architecture:
[d 0
)
0
x = 10 .
c
(67)
These two subnetworks are in the same fiber of the network parametrization map s.. Since yp is regular and generically one-to-one, this means that the (two equivalent) subnetworks yield a sin- gularity on the neuromanifold. This can also be verified via explicit calculations in Macaulay2: For that, we first compute a vanishing ideal the neuromanifold and then check that the function + d ((cra) is indeed singularity by seeing that the Jacobian of the ideal at that function has lower rank than the expected rank.
For each of the other mentioned subnetworks, namely 0 a b 0 0, there is unique way of embedding the subnetwork into the original architecture (assuming that all unspecified pa- rameters non-zero). Thus, they do not cause singularities the neuromanifold. Again, this be verified in Macaulay2 by computing that the rank the Jacobian of the vanishing ideal of the neuromanifold is the expected one.
We can easily check that this behavior is encoded in the combinatorial condition on the &, given in Theorem 4.6
22

---

Published as a conference paper at ICLR 2026
a 0
e a b
f2)
(1. 2
(2, 1)
(0. -1)
(2, 0)
The general intuition is that, for a subnetwork to embeddable in several ways into given archi- tecture, the added zeroes in one layer have to be "canceled out" in the next layer (compatible with the stride).
23


Reviews:
[{"review_id": "r5uQaRqOIk", "reviewer": "Reviewer_nwJE", "summary": "The paper studies the function spaces (neuromanifolds) parameterized by deep MLPs and CNNs with generic high-degree polynomial activations. It proves finite identifiability for MLPs (generic outputs come from finitely many parameters) and generic one-to-one identifiability for CNNs, hence the dimension equals the number of parameters in both settings. It then characterizes singular points of these neuromanifolds: (i) for MLPs, many subnetworks yield singularities and are shown to be critically exposed (they occur as critical points of squared-loss for a set of targets with nonempty interior); (ii) for CNNs, all singularities are precisely the subnetworks with edge zero-padding that satisfy an integrality constraint, and such sets are not critically exposed (away from the zero fiber). This geometric picture explains sparsity bias in MLPs but not in single-channel CNNs and resolves a long-standing dimension conjecture from prior work on polynomial networks. Figures 1--2 visualize how subnetworks create singular points and how MLP cuspidal-type vs. CNN nodal singularities differ.", "questions": "Non-polynomial activations: Can the polynomial approximation idea in Remark 4.1 be made quantitative (e.g., stability of singularity types under uniform approximation on compact sets)? Which parts of identifiability/exposedness survive in the ReLU or tanh settings?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: Moves from monomial/linear models to generic polynomial activations; formalizes critical exposure; delivers complete CNN singularity characterization. \n* Quality: Careful use of fiber-dimension, Vandermonde invertibility, and toric lattice ideals; clean separation between parametrization criticality and image singularity (Appendix A). Proof architecture is transparent via modular lemmas. \n* Clarity: Clear definitions of neuromanifolds/subnetworks; optimization setup with quadratic loss; intuitive figures (Fig. 1--2) and didactic examples (nodal vs. cuspidal curves in Fig. 3) to illustrate singular behaviors. \n* Significance: Fixes dimension = parameter count generically for both MLPs and CNNs; connects geometry to sparsity bias and to the presence/absence of spurious critical points across architectures.", "weaknesses": "* Generality vs. practicality: Many results require very high polynomial degree and generic coefficients. Concrete degree thresholds are not explicit beyond \\(r \\gg 0\\) (depends on architecture). Giving quantitative bounds (even conservative ones) would improve applicability. \n* Model scope: CNN analysis is single-channel, 1D; multi-channel and higher-D details are asserted “similarly” but not proved. Because modern CNNs are multi-channel, a pathway or obstacles to generalization would be valuable. \n* Singularity coverage in MLPs: The paper shows many singularities arise from subnetworks but leaves open whether all singularities do (contrast with linear MLPs and with CNNs, where a full characterization is given). Clarifying non-subnetwork singularities would strengthen the picture. \n* Type of critical points: Criticality vs. local minima vs. saddles is not analyzed; given the optimization motivation (sparsity bias), even partial results or conjectures on stability types would be informative. \n* Beyond polynomial activations: While approximation arguments are discussed (Remark 4.1), formal extension to ReLU/Tanh/Softmax is left for future work; clarifying which parts port over under approximation limits would broaden impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5uzOMaIJpn", "reviewer": "Reviewer_ZhpW", "summary": "The paper studies the function spaces (“neuromanifolds”) of deep MLPs and CNNs with polynomial activations, using tools from algebraic geometry. The core results:\n\nIdentifiability. For MLPs with a generic sufficiently high-degree polynomial activation, the parametrization is generically finite-to-one; hence the dimension of the neuromanifold equals the number of parameters (Theorem 4.1). For CNNs, it is generically one-to-one and regular off the zero fiber (Theorem 4.4).\n\nSingularities. Subnetworks (deactivating neurons/filters) yield singular points: fully characterized for CNNs (Theorem 4.6) and partially for MLPs (Theorem 4.2).\n\nOptimization bias. The paper introduces “critically exposed” parameter sets. Strict subnetworks of MLPs are critically exposed for quadratic losses (Theorem 4.3), but not for CNNs (Proposition 4.5). This gives a geometric account of sparsity bias in MLPs.", "questions": "1) Can you sketch the proof of  the limit argument suggested in Remark 4.1: under what conditions do MLP/CNN singularities persist under uniform polynomial approximation of non-polynomial activations?\n\n2) Are there settings where subnetwork critical points in MLPs are provably local minima with non-negligible measure?\n\n\n3) Can dimension counts for U_S (Eq. 8) yield measure bounds for how often subnetworks are equilibria?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Conceptual advancement. Provides a clean algebraic–geometric framework for identifiability and singularity in deep networks with polynomial activations, extending prior results beyond monomials.\n\nGenerality of MLP result. Finite identifiability for generic polynomials closes a dimension conjecture (dimension = #params) and generalizes known tanh/sigmoid cases to large-degree polynomials (Theorem 4.1).\n\nCNN result. The regularity and generic injectivity of CNN parametrizations off the zero fiber (Theorem 4.4) is technically strong and explains why CNN singularities are mild and do not create optimization equilibria.\n\nSparsity bias lens. The critical exposedness notion and proofs (Theorem 4.3 vs Prop. 4.5) give a principled account of why MLPs tend to collapse to sparse subnetworks whereas CNNs typically recover from near-zero initializations—matching known empirical phenomena.", "weaknesses": "Activation assumptions: Many results require “generic” high-degree polynomials (often with σ(0)=0 and nonzero top coefficients). Practical popular activations (ReLU, GELU, tanh) are non-polynomial; while the authors argue approximation plausibility (Remark 4.1, §5), formal transfer to non-polynomial nets is not proven.\n\nOptimization link: While “critically exposed” is compelling and geometric, the paper doesn’t classify whether exposed subnetworks are local minima vs saddles, which matters for SGD dynamics and generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j4sU5E1Vib", "reviewer": "Reviewer_WScj", "summary": "The paper investigates the geometry of neural networks with polynomial activation functions through algebraic geometric tools. The authors claim three main contributions:\nIdentifiability results: For MLPs with generic polynomial activations, almost all functions have finitely many parameter representations; for CNNs, the parametrization is generically one-to-one.\nSingularity characterization: Sparse subnetworks constitute singular points of neuromanifolds for both MLPs and CNNs.\nCritical exposedness: Subnetworks of MLPs are \"critically exposed\" (contain critical points of the loss with positive probability), providing a geometric explanation for sparsity bias. CNNs do not exhibit this property.", "questions": "Add concrete examples: Provide explicit small examples (e.g., 2-layer networks) where singularities and exposedness can be computed directly, if possible.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Important theoretical questions: The paper addresses fundamental issues in deep learning theory - identifiability, singularities, and sparsity bias - that have significant implications for understanding optimization and generalization.\nNovel geometric perspective: Connecting sparsity bias to singularities of neuromanifolds is creative and could provide new insights into the lottery ticket hypothesis.\nArchitectural comparison: The distinction between MLP and CNN geometry regarding critical exposedness is novel and aligns with empirical observations about their different behaviors.", "weaknesses": "The paper relies heavily on citations in a way that makes the intuition of the proofs difficult to follow . I would appreciate a more self contained mathematical exposition. I would be happy to improve my rating if more exposition was provided, as well as the following are addressed. \n\nIssues with specific proofs: \nTheorem 4.1 (MLP Identifiability)\n- The constraint β₁ > 6m² - 6m appears without much justification. Why this specific bound?\n- The dependence on this proof and a good bit of discussion in the paper on the Zariski topology necessitates a more thorough exposition/ explanation of this topological space, as it is a rather different topology than commonly used in Learning. \n\nTheorem 4.2 (MLP Singularities)\n- There should be more said about the dominance of σ ◦ fW′  . In particular it isnt clear that the image of σ ◦fW′ having a non-empty interior is sufficient to show Zariski density .\n\nTheorem 4.6 (CNN Singularities) \nMore said at all points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuNPRK7d6S", "reviewer": "Reviewer_jF6i", "summary": "The paper studies the (dis)connectedness and the existence of singularities in the parameter space of ReLU neural networks whose architectures admit a DAG computational graph representation. The authors then leverage these results to study whether these elements can exert impact on the dynamics of gradient flow (GF) of standard training algorithms.", "questions": "1. Do Proposition 6 - 7 imply that singularities are truly rare? It seems to me that the limit of GF can still be a singularity (or a sparse subnetwork). If the GF dynamics does not bias towards sparse subnetworks, do you have any idea which points are preferable for the convergence of GF?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I generally like the paper. Here are several remarkable points:\n1. The conservative law for ReLU networks in the DAG case is very elegant. \n2. Theorem 1 makes a very nice connection to flow problem in graph theory.\n3. Theorem 2 - Proposition 5 to 7 provide a very clean picture on the dynamics of gradient flow in the existence of singularities", "weaknesses": "1. I have a hard time distinguishing what are the main contributions and what are already proved in the literature. Authors might want to re-organize the section 2, and credit properly all the results (theorems, propositions, definitions) if they are ever taken/inspired by previous works.\n\n2. Do the author forget to define the notion of stable by forward/backward edges in the announcement of Theorem 1? Otherwise, I believe that Theorem 1 needs rephrasing to be easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tnlGsfcDD", "reviewer": "Reviewer_RBeS", "summary": "The paper studies the function spaces corresponding to CNNs and MLPs with polynomial activations using tools from algebraic geometry. Due to polynomial activations, these function spaces are manifolds. The authors show that almost everywhere, each function on this manifold corresponds to at-most finitely many parameter values of the network (the identifiability problem) and stronger result for CNNs (almost everywhere uniformly identifiable). The authors also characterize the singularities of the neuromanifolds. \n\n\nI want to note that I have no training in algebraic geometry and most of the results and ideas in this paper appeared opaque to me. I have set my confidence level to be low to reflect this fact.", "questions": "- Theorem 4.1 requires $r$ to be large enough. Is there a quantitative estimate about how large it should be? \n\n- In Theorem 4.2 it is not clear what \"enough coefficients of \\sigma are non-vanishing\" means. \n\n- The term \"generic polynomial\" is used throughout the paper, but I could not gather the meaning of this term. This could be because I am not trained in algebraic geometry.", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles the important question about the class of functions represented by neural networks. While standard results with non-linear non-polynomial activation functions consider approximation properties, this considers a different of view: that of characterizing the set of functions exactly represented by neural network families. The paper answers some deep questions about this function space such as identifiability and singularity.", "weaknesses": "- The authors claim that the identifiability results were previously only known for sigmoid and tanh activation functions. Given that they are more widespread in ML than polynomial activations, I would say that the current results are not as relevant to the community. \n\n- It does not directly give us relevant insights into practically relevant activation functions such as ReLU or sigmoid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)