Identifiability of Deep Polynomial Neural Networks
Authors: Konstantin Usevich
Ricardo Augusto Borsoi
Clara Dérand
Marianne Clausel
Venue: NeurIPS 2025 oral ()
Keywords: polynomial neural networks
identifiability
tensor decompositions
uniqueness
Primary area: theory
TL;DR: We provide necessary and sufficient conditions for the identifiability deep polinomial neural networks.

Abstract: Polynomial Neural Networks (PNNs) possess a rich algebraic and geometric structure. However, their identifiability-a key property for ensuring interpretability-remains poorly understood. In this work, we present a comprehensive analysis of the identifiability of deep PNNs, including architectures with and without bias terms. Our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability. As special cases, we show that architectures with non-increasing layer widths are generically identifiable under mild conditions, while encoder-decoder networks are identifiable when the decoder widths do not grow too rapidly compared to the activation degrees. Our proofs are constructive and center on a connection between deep PNNs and low-rank tensor decompositions, and Kruskal-type uniqueness theorems. We also settle an open conjecture on the dimension of PNN's neurovarieties, and provide new bounds on the activation degrees required for it to reach the expected dimension.

Identifiability of Deep Polynomial Neural Networks
Konstantin Usevich', Ricardo Borsoi, Clara Dérand, Marianne Clausel Université de Lorraine, CNRS, CRAN Nancy, F-54000, France
firstname oagtrmmmm__sstaammuuuooooooaaaaa
Abstract
Polynomial Neural Networks (PNNs) possess rich algebraic and geometric struc- ture. However, their identifiability a key property for ensuring interpretability- remains poorly understood. In this work, we present a comprehensive analysis of the identifiability of deep PNNs, including architectures with and without bias terms. Our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability. As special cases, we show that architectures with non-increasing layer widths are generically identifiable under mild conditions, while encoder-decoder networks are identifiable when the decoder widths do not grow too rapidly compared to the activation degrees. Our proofs are constructive and center on a connection between deep PNNs and low-rank tensor decomposi- tions, and Kruskal-type uniqueness theorems. We also settle open conjecture the dimension of PNN's neurovarieties, and provide new bounds on the activation degrees required for it reach the expected dimension.
Introduction
Neural network architectures which use polynomials as activation dunctions-polynonii neural networks (PNN>-have emerged architectures that combine competitive experimental performance (capturing high-order interactions between input features) while allowing a fine grained theoretical analysis. On the one hand, PNNs have been employed in many problems in computer vision I image representation 4. physics [5) and finance ( to name a few. On the other hand, the geometry of function spaces associated with PNNs, called neuromanifolds, can be analyzed using tools from algebraic geometry. Properties of such spaces, such as their dimension, shed light on the impact of a PNN architecture (layer widths and activation degrees) on the expresvivity of feedforward, convolutional and self-attention PNN architectures (7-11) They also determine the landscape of their loss function and the dynamics of their training process
12 13]
Moreover, PNNs are also closely linked to low-rank tensor decompositions [1+ 8] which play fundamental role in the study of latent variable models due their identifiability properties [1)] In fact, single-outpu 2-layer PNNs are equivalent to low-rank symmetric tensors 77.. Identifiability whether the parameters and, consequently, the hidden representations of a NN can be determined from its response up to some equivalence class of trivial ambiguities such as permutations of its neurons-is key question in NN theory 32). Identifiability is critical to ensure interpretability in representation learning 33 to provably obtain disentangled representations (36). and in the study of causal models 37 It is also critical to understand how the architecture affects the inference process and to support manipulation or "stitching" of pretrained models and representations Moreover, it has important links to learning and optimization of PNNs 44
35 38
Corresponning author
In this version, the appendices have been reworked for better readability. Appendix E explains the changes between the submitted and the camera-ready version.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Identifiability of deep PNNs is intimately linked to the dimension of their so-called neurovarieties: when this dimension reaches the effective parameter count, the number of possible parametrizations is finite, which means the model is finitely identifablee and the neurovariety is said to be non-defective. In addition, many PNN architectures admit only a single parametrization (i.e., they are globally identifiable. This has been investigated for specific types of self- self-attention [9] and convolutional [8] layers, and feedforward PNNs without bias [T]. However, current results for feedforward networks only show that finite identifiability holds for very high activation degrees, or for networks with the same widths in every layer ΑЛ A standing conjecture is that this holds for any PNN with degrees at least quadratic and non non-increasing layer widths [II], which parallels identifiability results of ReLU networks 29]. However, a general theory of identifiability of deep PNNs is still missing.
1.1 Our contribution
We provide comprehensive analysis of the identifiability of deep PNNs considering monomial activation functions. We prove that an L-layer PNN is finitely identifiable if every 2-layer block composed by pair of two successive layers is finitely identifiable for some subset of their inputs. This surprising result tightly links the identifiability of shallow and deep polynomial networks, which is key challenge in the general theory of NNs. Moreover, our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability.
As special cases, we show that architectures with non-increasing layer widths (i.e., pyramidal nets) are generically identifiable, while encoder-decoder (bottleneck) networks are identifiable when the decoder widths do not grow too rapidly compared to the activation degrees. We also show that the minimal activation degrees required to render a PNN identifiable (which is equivalent to its activation thresholds) is only linear in the layer widths, compared to the quadratic bound in [T] Theorem 18]. These results not only settle but generalize conjectures stated in IT. Moreover, we also address the case of PNNs with biases (which was overlooked in previous theoretical studies) by leveraging homogenization procedure.
Our proofs are constructive and are based on a connection between deep PNNs and partially symmetric canonical polyadic tensor decompositions (CPD). This allows us to leverage Kruskal-type uniqueness theorems for tensors to obtain identifiability results for 2-layer networks, which serve as the building block in the proof of the finite identifiability of deep nets, which is performed by induction. Our results also shed light on the geometry of the nenrovarieties, they lead to conditions under which its dimension reaches the expected (maximum) value.
1.2 Related works
Polynomial NNs: Several works studied PNNs from the lens of algebraic geometry using their asso- ciated neuromanifolds and neurovarieties (7) (in the emerging field of neuroalgebraic geometry [41) and their close connection to tensor decompositions. Kileel et al. 7 studied the expressivity or feed- forward PNNs in terms of the dimension of their neurovarictics. An analysis of the neuromanifolds
for several architectures was presented in [10]. Conditions under which training losses do not exhibit bad local minima or spurious valleys were also investigated
13 12 42] The links between training 2-layer PNNs and low-rank tensor approximation 13 as well as the biases of gradient descent 43
have been established.
Recent work computed the dimensions of neuromanifolds associated with special types of self- attention 9 and convolutional [8] architectures, and also include identifiability results. For feed- forward PNNs, finite identifiability was demonstrated for networks with the same widths in every layer [T]. while stronger results are available for the 2-layer case with more general polynomial activations 44.. Finite identifiability also holds when the activation degrees are larger than a so-called activation degree threshold [I). Recent work studied the singularitics of PNNs with activations consisting of the sum of monomials with very high activation degrees [45). PNNs are also linked to factorization machines 446]; this led to the development of efficient tensor-based learning algorithms
[4] 48 Note that other types of non-monomial polynomial-tyse activations 49 30 5, 51 have shown excellent performance; however, the geometry of these models is not well known.
NN identifiability: Many studies focused on the identifiability 2-layer NNs with tanh, odd, and
ReI.U activation functions 23 Moreover, algorithms to learn 2-layer NNs with unique parameter
recovery guarantees have been proposed (see, e.g., 52 53
however, their extension to NNs with
or more layers is challenging and currently uses heuristics (54). Identifiability of deep NNs under

---

weak genericity assumptions was first studied in the pioneering work of Fefferman (24] for the case of the tanh activation function through the study of its singularities. Recent work extended this result to more general sigmoidal activations 22 26 Various works focused on deep ReI.U nets, which are piecewise linear (2X;; they have been shown to be generically identifiable the number of neurons per layer is non- non-increasing (29] Recent work studied the local identifiability of ReI.U nets [30H32]. Identifiability has also been studied for latent variable/causal modeling. leveraging different types of assumptions (e.g., sparsity, statistical independence, etc.) [55H60], Note that although some of these works tackle deep NNs, their proof techniques are completely different from our approach and do not apply to the case of polynomial activation functions.
Tensors and NNs: Low-rank tensor decompositions had widespread practical impact in the com- pression of NN weights (61 651. Moreover, their properties also played key role in the theory of NNs 18] This includes the study of the expressivity of convolutional [66] and recurrent (67
NNs, and the sample complexity of reinforcement learning parametrized by low-rank transition and reward tensors [69 70_ The decomposability of low-rank symmetric tensors was also paramount in establishing conditions under which 2-layer NNs can (or cannot (71) be learned in polynomial time and in the development of algorithms with identifiability guarantees [52 72, 73 It was also used to study identifiability of some deep linear networks 774. However, the use of tensor decompositions in the studying the identifiability of deep nonlinear networks has not yet been investigated.
2 Setup and background
2.1 Polynomial neural networks: with and without bias
Polynomial neural networks functions R - Re represented feedforward networks with bias terms and activation functions of the form (-( = (-)". Our results hold for both the real and complex valued case (F R. C), thus, and we prefer keep the real notation for simplicity. Note
that we allow the activation functions to have a different degree Ty for each layer.
Definition (PNN). polynomial neural network (PNN) with biases and architecture (d (do, d1, ................................ dL),² r (ri..... -L)))) is map Rdd + Rdd, given by a feedforward neural network
PNN [0] = PNN,10] = fL Pri-. fL-1 P o...o Pr. o fi,
(1)
where fi(x)) = W x + b, are affine maps, with W; E Rd:xdi-1 being the weight matrices and by E Red the biases, and the activation functions Pr : Rdd + Rd, defined as pr(z) == (x),)..,,²)) are monomial. The parameters O are given by the entries of the weights W. and biases bú, iee,
0 (w, b), (W 2.... WE), b b= (b1, bu)
(2)
The vector of degrees r is called the activation degree of PNN, [0] (we often omit the subscript if is clear from the context).
PNNs are algebraic maps and are polynomial vectors, where the total degree is Trotal = F1 TL-1- that is, they belong to the polynomial space ( ) xdz, where Pa,r denotes the space of d- variate polynomials of degree S r. Most previous works analyzed the simpler case of PNNs without bias, which we refer as homogeneous. Due to its importance, we consider it explicitly.
Definition 2 (hPNN). A PNN is said to be a homogenous PNN (hPNN) when it has no biases (b. = 0 for all e I...., L), and is denoted as
hPNN [un) hPNN, [w] == WL o Pre-i o W L-1 Pre-z Pr. o W Its parameter set is given by w = (W 1, W 2..... ....
It is well known that such PNNs are in fact homogeneous polynomial vectors and belong to the polynomial space (Hessreeeat) Xds, where Ha, C Par denotes the space of homogeneous d-variate polynomials of degree r. hPNNs whose properties can be used in their theoretical analysis.
also naturally linked to tensors and tensor decompositions,
Example 3 (Running example). Consider an hPNN with - r (2) and d 2, 2). In such a case the parameter matrices are given as
W 2 =
bi
b12
W
011
012
413
b21
b22
021
022
A23
3

---

and the hPNN hPNN, [w] is vector polynomial that admits the expression
b1
bb2 b22
p(œ) = W 2P2 W iz) =
(Q11œ1 + 01232 + a1313)² +
(a2101 + 022T2 + d2323)
b2
the only monomials that can appear are of the form æf with + k- thus p is vector of d homogeneous polynomials in 3 variables (in our notation, p E (Ha,.2)2)
2.2 Equivalent PNN representations
It is known that the PNNs admit equivalent representations (i.e., several parameters 0 leading to the same function). Indeed, for each hidden layer we can (a) permute the hidden neurons, and (b) rescale the input and output to each activation function since for any o 0, (at)" a"t". These transformations lead to different sets parameters that leave the PNN unchanged. We can characterize all such equivalent representations in the following lemma (provided in 7 for the case without biases).
Lemma 4. Let PNN 10) be PNN with 1 as in ② Let also De E Rddedd be any invertible diagonal matrices and Ps Zdexde (E = 1, 1,..., L 1) be permutation matrices, and define the transformed parameters as
W + P,D.W, D D.TL-PTT PT..
with Do and PL = DL = I by convention. Then the modified parameters W', define exactly the same network, i.e. PNNa,r = P for the parameter vector
0' = ((W1, W2 W'D, (b/, b2...., ,,)),,
0 and O' are linked with such transformation, they are called equivalent (denoted O O'). Example 5 (Example 3
continued). In Example we can take any a, B + 0 to get
bb
B 2b12
hPNN d [w]
(a0m121 taa122zt )
(Ba21 + +B02222 )²
b21
B 8-2
which correspond to rescaling rows of W 1 and corresponding columns of W we additionally permute them, get W PDW1, W W2D "Tp with D
[8 3] and P [P
This characterization of equivalent representations allows to define when PNN is unique. Definition 6 (Unique and finite-to-one representation). The PNN = PNN a.r 0 (resp. hPNN hPNN d, (w]) with parameters 0 (resp. w) is said have unique representation if every other representation satisfying p- PNNa.- (resp. p hPPNN,, (w')) is given by equivalent set of parameters, i.e., 0' ~ 0 (resp. w ~ w) in the sense of Lemma (i.e., they can be obtained from the permutations and elementwise scalings in Lemma 4
Similarly, a PNN only finitely many non equivalen representations, that is, the set (6' PNNN,, a. [0'] = p} (resp.
PNNa,,-O0 (resp. hPNN p hPNN (w)) is fiitt-ttone if admits
{w': hPNNN, a [w'] p contains fiittll many non-equivalent parameters.
Example (Example continued). Thanks to links with tensor decompesitions and their uniqueness, it is known that the hPNN in Example 3 has unique representation if W is invertible and W, full
row rank (rank 2), see Proposition 35 in Section 4.2
2.3 Identifiability and link to neurovarieties
An immediate question is which PNN/hPNN architectures are expected to admit only a single (or finitely many) non-equivalent representations? This question can be formalized using the notions of global and finite identifiability, which considers a general set of parameters.
Definition 8 (Global and finite identifiability). The PNN (resp. hPNN) with architecture (d,r) is said be globally identifiable if for general choice of (w, b) RR (edd--1+)) (resp. E R dede 1) (i.e.. for all choices of parameters except for of Lebesgue measure zero), the network PNN [0] (resp. hPNN (w]) has a unique representation.
Similarly, the PNN (resp. hPNN) with architecture (d. r) is said to be finitely identifiable for general choice of 0, (resp. w) the network PNNa, 0 (resp. aPNNaar is finitetoo-one (i.e., admits only finitely many non-equivalent representations).

---

In the following, we use the term "identifiable" to refer to finite identifiability unless stated otherwise. Note also that the notion of finite identifiability much stronger than the related notion of local identifiability (i.e., a model being identifiable only in a neighborhood of parameterization). Example 9 (Example 7 continued). From Example 7 we see that the hPNN architecture with d= (3, 2, 2), (2) is ideettfiable due to the fact that generic matrices W and W2 are full runk.
Note that Definition 8 excludes set of parameters of Lebesgue measure zero. Thus, for an identifiable architecture such as the one mentioned in Example O there exists rare sets of pathological parameters for which the hPNN is non-unique (e.g., weight matrices containing collinear rows).
With some abuse of notation, let hi h.NNNa.r be the map taking w to hPNNN,, [w]. Then the image of hPPNN,, [- [+] is called neuromanifold, and the neurovariety Vdrr , is defined its closure in the
Zariski topology [ The study of neurovarieties and their properties is a topic of recent interest 44111 More details are given in Appendix [ An important property for our case is the link between identifiability of an hPNN, the dimension its neurovariety, and the rank of its Jacobian.
Proposition 10. The architecture hPNNarrH is finitely identifiable if and only if the dimension of r is equal to the effective number of parameters, i.e, dim V.r far=
Ei=ı dide-1-Ee 1aah---ee -Ei=1 de. In such case, Va. is said be nondefective. Equivalently, the rank of the Jucobian of the map hPNNar11 is maximal and equal to Eizz dede-1 Et=i de at general parameter w.
3 Main results
3.1 Main results on the identifiability of deep hPNNs
Although several works have studied the identifiability of 2-laaerr NNs, tackling the case of deep networks is significantly harder. However, when we consider the opposite statement, i.e., the non- identifiability of network, it is much easier show such connection: in deep network with L >>>> layers, the lack of identifiability of any 2-layer subnetwork (formed by two consecutive layers) clearly implies that the full network is not identifiable. What our main result shows is that, surprisingly, under mild additional conditions the converse is also true for hPNNs: if the every 2-layer subnetwork identifiable for some subset of their inputs, then the full network identifiable well. This is formalized in the following theorem.
Theorem 11 (Localization theorem). Let (Ido...., dL), (f1...., be the hPNN format. For = o...., L - 2 denote de == min (da.....de) Then the following holds true: if for all & = 1...., L the two-layer architecture hP de is finitely identifiable, then the L-layer architecture hPNN is finitely identifiable as well.
The technical proofs are relegated to the appendices. This key result shows a strong relation between the finite identifiability of shallow and deep hPNNs. However, as we move into the deeper layers, the identifiability conditions required by Theorem 11 are stricter than in the shallow case, since the number of inputs is reduced to de. This can lead to requirement of larger activation degrees to guarantee identifiability compared to the shallow case.
Theorem 1
allows us to derive identifiability conditions for hPNNs using the link between 2-layer hPNNs and partially symmetric tensor decompositions and their generic uniqueness based on classical Kruskal-type conditions. We use the following sufficient condition for the identifiability of shallow networks.
Proposition 12 (Sufficient condition for identifiability of 2-layer hPNN), Let do, d1 2 2, d2 2 1 be the layer widths and r 2 2 such that
2d, (nin(d2,d40 d1)
r min(d,, do)
(4)
Then the 2-layer hPNN with architecture ((do, d1, dz),r) is globally identifiable.
Remark 13. If the above condition is satisfied for every 2-layer architecture ((de-1, 1, de. de+1), re),
{= 1=1.... 1...., L 1, then Theorem 17 implies that the L-layer hPNN is finitely identifiable for the T--layer architecture rd.
Li.., the smallest algebraic variety that contains the image of the map hPNNa.r(.
5

---

Remark 14. Note that for the single output case dp = 1, Equation means the activation degree in the last layer must satisfy TI.1 2 in contrast to ry 2 for & 1.
Remark 15 (Our bounds are constructive). We note that the condition (4) for identifiability is not the best possible (and can be further improved using much stronger results on generic uniqueness of decompositions, see e.g. 775 Corollary 37]). However, the bound (4) is constructive, and we can use standard polynomial-lime lensor algorithms recover the paramelers of the 2-layer hPNN.
3.2 Implications for specific architectures
Proposition 12 has direct implications for the finite identifiability of several architectures of practical interest, including pyramidal and bottleneck networks, and for the activation thresholds of hPNNs, as shown in the following corollaries.
Corollary 16 (Pyramidal hPNNs are always identifiable). The hPNNs with architectures containing non-increasing layer widths (except possibly the last layer), i.e., do di N ...d--1 2 and dı 2 L, are finitely identifiable for any degrees satisfying
(i) .i..... 7L-1 22 if dp 2 2: or (ii)
FL-2 2 2, TL-1 23 if dp =1
Note that, due to the connection between the identifiability of hPNNs and the neurovarieties presented in Proposition 10 direct consequence of Corollary [6]is is that the neurovariety Yar has expected
dimension. This settles a recent conjecture presented in Section 4]. This implication is explained in detail in Appendix A
Instead of seeking conditions the layer widths for a fixed (or minimal) degree, a complementary perspective to determine what are the smallest degrees Fę such that a given architecture d is finitely
identifiable. Following the terminology introduced in [1I]. refer to those values the activation degree thresholds for identifiability an hPNN. An upper bound is given in the following corollary: Corollary 17 (Activation degree thresholds for identifiability). For fixed layer widths d = (do.... .... ....,).)))))) with de 2 2, = O....,L L - 1, the hPNNs with architectures (d, (FI....,TL-1)) are finitely identifiable for any degrees satisfying
rt 2 2dg 1.
Note that due to Proposition 10 the result in this corollary implies that the neurovariety Var- has expected dimension. This means that (2dg 1) is also universal upper bound to the so-called activation thresholds for hPNN expressiveness introduced in [I]. The existence of such activation degree thresholds was conjectured in 7 and recently proved in [11 Theorem 18], but the for quadratic in de bound (the bound Corollary 17 is linear).
Remark 18 (Admissible layer sizes). The possible sizes in deep network are tightly linked with Ti rt (for general Te, a similar bound O(d)))) follows from link with tensor decompositions (761). Therefore, to allow for larger layer widths, we need to have higher-degree activations.
the degree of the activation. For example,
identifiability is impossible if de de-1 -+1)
enlightening to consider the admissible layer widths when taking into account the joint effect of layer widths and degrees. By doing this, Proposition 12 can be leveraged to yield identifiability conditions for the case of bottleneck networks, as illustrated in the following corollary.
Corollary 19 (Identifiability of bottleneck hPNNs). Consider the "bottleneck" architecture with
do 2 d1 2 2 do S dbb1 S... dd
and do 2 2. Suppose that Tb. 2 and that the decoder part satisfies de É E 1b+1, L-1), 1]. Then the bottleneck hPNN is finitely identifiabl.
T < db 1 for
This shows that encoder-decoder hPNNS architectures are identifiable under mild conditions on the layer widths and decoder degrees, providing polynomial networks-based counterpart to previous studies that analyzed linear autoencoders 777 78).
Note that the width of the bottleneck layer d, constrains the entire decoder part of the architecture; the degrees re, e 2 are constrained according to the width dn. The presence of bottlenecks has also been shown to affect the expressivity of hPNNs in (7] Theorem 19]: for d, 2do 2 there exists number never reaches that of the ambient space) for any choice of widths di...., d,-1, do+1......d dL.
of layers such that for r; 2 2 and do 2 2, the hPNN neurovariety is non-filling (i.e., its dimension

---

3.3 PNNs with biases
The identifiability of general PNNS (with biases) can be studied via the properties of hPNNs. The simplest idea is truncation (i.e., taking only higher-order terms of the polynomials), which eliminates biases from PNNs. Such an approach was already taken in [4 for shallow PNNs with general polynomial activation, and is described in Appendix D.3] We will follow different approach based on the well-known idea of homogenization: we transform PNN to an equivalent hPNN with structured parameters keeping the information about biases at the expense of increasing the layer widths. Our key result is to show how this can be used to study the identifiability of PNNs with bias terms. The following correspondence is well-known.
Definition 20 (Homogenization). There is a one-to-one mapping between polynomials in d variables of degree and homogeneous polynomials of the same degree in d + 1 variables. We denote this mapping Par + Mar-1r by homog(.), and it acts as follows: for every polynomial E Pa. p homog(p) Ha+1, (that is p(z1,..., Ed-1))
1) is the unique homogeneous polynomial in
+ 1 variables such that
,dd, 1) ((d))..,
Example 21, For the polynomial p E 22 in variables (I1, (2) given by x(21. azi b 22 czi + ez + f + 9: its homogenization p- = homog(p) E A2,2 in variables (z), (2,,22, 23) is
p(11, 02; 03) = a + br + ca2 + ex133 + fiznz + gay: and we can verify that p(1, 21, x2) = p(11, 22).
Similarly, we extend homogenization to polynomial vectors, which gives the following. Example 22. Let f(x) = W2rr, (Wi b1) + bg. and define extended matrices as
W
W1 bi 1
R (((+++))(()+))) W₂ IW2 b2] E (d
Then its homogenization f homog(J f) an hPNN of format (do + 1, + 1,)
f(क) = W 2P W 1
where z [auu, T1..... Foly Ido+il T, so that f(11, .......... Ido 1) f(11. Dd,).
The construction in Example 22 similar to the well-known idea augmenting the network with an artificial (constant) input. The following proposition generalizes this example to the case of multiple layers, by "propagating" the constant input.
Proposition 23, Fix the architecture
(F1,..... (fi...., TL-1) and d (do...., dr). Then polynomial vector p (P do jxdz. admits PNN representation p = PNNa,-((w, b)] with (w. b) as in (2) if and only if its homogenization p homog(p) admits an hPNN decomposition for the same activation degrees r and extended d = (do + .....,dI-1. + 1, d,d P hPNN d. [0).
w (W.........
WL), with matrices given as
W.
bi
E (iddtt))xdd-1tt) l L,
W
0 1
W.
bi.
((dd) t=
That is, PNNs are one-to-one correspondence to hPNNs with increased number of inputs and structured weight matrices.
Uniqueness of PNNs from homogenization: An important consequence of homogenization is that the uniqueness of the homogenized hPNN implies the uniqueness of the original PNN with bias terms, which is key result support the application of our identifiability results to general PNNs.
Proposition 24. If hPNN, from Proposition 23 is unique (resp. finite-to-one) (without taking into account the structure), then the original PNN representation PNN, b)] is unique (resp. fonte-to-onee.
hPNN
7

---

The proposition follows from the fact that we can always fix the permutation ambiguity for the "artificial" imput.
Remark 25. Despite the one-to-one correspondence, for generic properties (e.g., finite identifiability) we cannot immediately apply the results from the homogeneous case, because the matrices W g are structured (they form a set of measure zero inside
However, can prove that the identifiability of the hPNN implies the identifiability of the PNN.
Lemma 26. Let the 2-layer hPNN architecture be finitely (resp. globally) identifiable for ((do 1, d1 + 1, 1,d2), r1). Then the PNN architecture with widths (do, d1, d2) and degree v1 also finitely (resp. globally) identifiable.
Using Lemma and specializing the proof of Theorem we obtain the following result:
Proposition 27. Let ((da,...,dz.) (FI....,FFI-1)) be the PNN format. For l 0,... , L denote de = minfdo.......................... ....... Then the following holds true: If for all (= l,....., L- each two-layer architecture hPNN ( (1-++11 is finitely identifiable, then the L-layer PNN with architecture (d, r) is finitely identifiable as well.
In particular, we have the following bounds for generic uniqueness.
Corollary 28. Let ((do...., dL), (.......L-1)) be such that de 2 1, and re 2 satisfy
2(de + 1) minnde + 1, de+1)
Tt
(in(de,de-1)
then the L-layer PNN with architecture (d,r) is finitely identifiable (and globally identifiable if L 2).
Remark 29. For general PNNs with bias, similar conclusions hold to the ones in the hPNN case. In particular, for fixed layer widths de 1, the activation threshold for PNN architecture (d, r) becomes 2 2dg + 1. Also, pyramidal PNNs are identifiable in degree 2.
A distinctive feature of PNNs with bias is that they can be identifiable even for architectures with layers containing single hidden neuron: for de 1 and de+1 2 and/or de-1 the condition in Corollary 28 is still satisfied when rę 2 2.
Proofs and main tools
Our main results in Theorem translates the identifiability conditions of deep hPNNs into those of shallow hPNNs. Our results are strongly related to the decomposition of partially symmetric tensors (we review basic facts about tensors and tensors decompositions and recall their connection between to hPNNs in later subsections). More details are provided in the appendices, and we list key components of the proof below.
4.1 Identifiability of deep PNNs: necessary conditions
Increasing hidden layers breaks uniqueness. The key insight is that if we add to any architecture a neuron in any hidden layer, then the uniqueness of the hPNN is not possible, which formalized as following lemma (whose proof is based, in its turn, tensor decompositions).
hPNNN,ww) be an hPNN of format (do..... .d, .d....,,z)) Then for any i there exists an infinite number of representations of hPNNs p hPNN, [wn] with architecture (do....,de de + +,...,,d dz). In particular, the augmented hPNN is not unique (and is not finite-to-oner
Lemma 30. Let
Internal features of a unique hPNN are linearly independent. Lemma 30 (as linear dependence would allow for pruning neurons).
This is an easy consequence of
Lemma 31. For d (do..... dr), let p hPNN, [w] have unique (or fonete L-layers decomposition. Consider the output at any E-th internal level < after the activations
(x)
Pr. W W..... Pr. Wi(x).
(5)
Then the elements of (x())) (x)
ge,d, (x)) are linearly independent polynomials.
8

---

Identifiability for hPNNs and Kruskal rank. Identifiability of 2-layer hPNNs, or equivalently uniqueness of CPD strongly related to the concept of Kruskal rank of a matrix that we define below. Definition 32. The Kruskal rank of matrix A (denoted krank (rank[A)] (A}) ) is the maximal number k such
that any columns of are linearly independent.
This is in contrast with the usual rank, which is the maximal l such that there exist k linearly independent columns. Therefore krank(A) S rank Note that krank [}} 2 means that none of the pairs of columns of A are linearly dependent (no columns are pairwise collinear). Using the notion of Kruskal rank, we can state a necessary condition on weight matrices for identifiability of hPNNs, which is generalization of the well-known necessary condition for the uniqueness of CPD tensor decompositions (6) (i.e., shallow networks), and is corollary of Lemma 330 and Lemma 31
Proposition 33. As in Lemma 31 let the widths be d = (do..... dz), and p hPNN, [w] have unique (or finite-to-on tonee) L-layers decomposition. Then we have that for all 1....,
krank (W]) 2, krank E+1} Z 1,
where krank W }+1} 2 1 simply means that W1+1 does not have zero columns.
4.2 Shallow hPNNs and tensor decompositions
An order-s tensor" T E R. is an s-way multidimensional array (more details are provided in Appendix B.2 and more background on tensors can be found in [14) It is said have d-term CPD (canonical polyadic decomposition) if it admits a decomposition into d rank-1 terms
T- an @ ... & ass.j for di.j E IR",, with & being the tensor (outer) product. The CPD is
also written compactly as = [A1, A2...., A,] for matrices A, [a.,,,.... ai,a) E IR, xd T is said to be (partially) symmetric if it is invariant any permutation of (a subset) of its indices 779 Concretely, we will consider tensors T partially symmetric on dimensions E (2...., 8), with CPD that is also partially symmetric, i.e., with A,, i 2 2 satisfying A = Ag
A.. Our main
proof' strongly rely on results of [7] on the connection between hPNN and tensors decomposition in the shallow (i.e., 2-layer) case (see also (79).
Proposition 34. There is a one-to-one mapping between partially symmetric tensors FF E Rd2xdox--xx4e and polynomial vectors f E (Hodo.)) Xd2, )xd2, which can be written as
F f(z)= F())
with F()) E dd the first unfolding of F. Under this mapping, the partially symmetric CPD
F= [W2, ....... .wil
(6)
is mapped hPNN 2p,, (W iz). Thus, uniqueness of hhPN (do da), W2)! is equivalent to uniqueness of the partially symmetric CPD of F.
Thanks to the link with the partially symmetric CPD, we prove the following Kruskal- -baed sufficient condition for uniqueness (which is counterpart of Proposition B3).
Proposition 35. Let Pw satisfying do d1 2, d2 > 1. Assume that 2 2, krank [2) 2 1, krank[W[] and that:
W2er, (W (x) be 2-layer hPNN with layer sizes (do, d1, d2)
2dj krank W2)
krank(W]] -
then the 2-layer hPNN P (x) is unique (or equivalently, the CPD of (6) is unique).
Remark 36. For 2-layer hPNNs (L 2), when the activation degree r is high enough Proposition 33 gives both necessary and sufficient conditions for uniqueness due to Proposition 35
Remark 37. Proposition 35 forms the basis of the proof of Proposition 12 which comes from the fact that the Kruskal rank of generic matrix is equal to its smallest dimension.
Remark 38. Proposition
35 based on basic ( (Kruskal) uniqueness conditions 82). As mentioned in Remark 15 by using more powerful results on generic uniqueness [83 84), we can obtain better
bounds for identifiability of 2-layer PNNs. For example, for "bottleneck" " architectures (as in
Corollary 19. the results of [83 Thm 1.11-12] imply that for degrees T = 2, identifiability holds for
decoder layer sizes satisfying a weaker condition de
(dy - Dydy (instead of de < do 1).

---

4.3 Proof of the main result
The proof of Theorem D proceeds by induction over the layers e =. L. The key idea is based on a procedure that allows us to prove finite identifiability of the L-th layer given the assumption that the previous layers are identifiable. For this, we introduce a map (last layer map)
v lgq. W := W LP (q(21..... .Fdo).
(7)
where q the vector polynomial of degree linear layer. Then the L-laaer hPNN is composition:
ri ... TL-2 2, representing the output of the (I- 1
hPNN, (0, Wt z.] = क [NNN
0), Wr), for 0 (WI..... WL-1).
To obtain finite identifiability, we look at the Jacobian of the composite map. The key to this recursion to show that the Jacobian (q. W,,) (Jacobian of & with respect the input polynomial vector and W 1.) is of maximal possible rank. For this, we construct a "certificate" of finite identifiability Q realized by hPNN hPNN
(11.... hPNN
0 but of simpler structure which inherits identifiability of a shallow
Remark 39. For dj. = 1, maximality of the rank for Jw W 1) is closely related to nondefectivity of the variety of sums of powers of forms, which often proved by establishing Hilbert genericity of an ideal generated by the elements of q (a question raised in Fröberg conjecture, e.g., 1851).
A key limitation of our techniques is that they only allow for establishing finite identifiability for deep PNNs. There exist recent results linking finite and global identifiability, [75] 86] but only for additive decompositions (shallow case). We state, however, the following conjecture.
Conjecture 40. Under the assumptions of Theorem M7 the L-layer hPNN is globally identifiable.
Note that the conjecture may be valid only for global identifiability (i.e., for a generic choice of parameters) and not for uniqueness, since it is not true that the composition of unique shallow hPNNs yield a unique deep hPNNs, as shown by the following example.
(x1 - zz)2|² T. Example 41. Consider two polynomials: p(z1,02) = [<2] + xz²2
We see that this
polynomial vector admits two different representations
1 W 222 W (W2)))
p = Ip2( W 222((Tr)) = W
2
with
1 -1
W₂
-1
W
-1
which are not equivalent. However, each 2-layer subnetwork is unique (see Example
7
Discussion
In this paper, we presented a comprehensive analysis of the identifiability of deep feedforward PNNs by using their connections to tensor decompositions. Our main result the localization of identifiability, showing that deep PNNs are finitely identifiable if every 2-layer subnetwork is also finitely identifiable for subset of their inputs. Our results can be also useful for compression (pruning) neural networks as they give an indication about the architectures that are not reducible. An important perspective is also to understand when two different identifiable PNN architectures can represent the same function, as the identifiable representations can potentially occur for different non-compatible formats (e.g., a PNN in format d = (2, 4, 4, 2) could be potentially pruned to two different identifiable representations, say, d (2, 3, 4, 2) and d = (2, 2, 2)).
While our results focus on the case of monomial activations, we believe that this approach can be ex- tended for establishing theoretical guarantees for other types of architectures and activation functions. key first step in addressing general polynomial activations
In fact, the monomial case constitutes (see, e.g., [55) which, in turn, can approximate most commonly used activations on compact sets. Moreover, the close connection between PNNs and partially symmetric tensor decompositions (which benefit from efficient computational algorithms based on linear algebra (87)) can also serve as support for the development of computational algorithms based on tensor decompositions for training deep PNNs. In fact, tensor decompositions have been combined with the method of moments to learn small NN architectures (see, e.g., 52 {8) ) extending such approaches for training deep PNNs with
finite datasets is an important direction for future work.
10

---

Acknowledgments
This work was supported in part by the French National Research Agency (ANR) under grants ANR- 23-CE23-0024, ANR-23-CE94-0001 by the PEPR project CAUSALI-T-AI, and by the National Science Foundation, under grant NSF 2316420.
References
[1] Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou. P-nets: Deep polynomial neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[2] Grigorios Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference on Computer Vision, pages 692-716. Springer, 2022.
[3] Mohsen Yavartanoo, Shih-Hsuan Hung, Reyhaneh Neshatavar, Yue Zhang, and Kyoung Mu Lee. Polynet: Polynomial neural network for 3D shape recognition with polyshape representation. In International conference on 3D vision (3DV), pages 1014-1023, IEEE, 2021.
[4] Guandao Yang, Sagie Benaim, Varun Jampani, Kyle Genova, Jonathan Barron, Thomas Funkhouser, Bharath Hariharan, and Serge Belongie. Polynomial neural fields for subband decomposition and manipulation. In Advances in Neural Information Processing Systems, 2022. /KtL/tttpps//pppeeeeeee....
[5] Jie Bu and Anuj Karpatne. Quadratic residual networks: A new class of neural networks for solving forward and inverse problems in physics involving PDEs. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 675-683, SIAM, 2021.
[6] Sarat Chandra Nayak and Bijan Bihari Misra. Estimating stock closing indices using GA- weighted condensed polynomial neural network. Financial Innovation, 4(1):21, 2018.
[7] Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.
[8] Vahid Shahverdi, Giovanni Luca Marchetti, and Kathlén Kohn. On the geometry and op- timization of polynomial convolutional networks. AISTATS 2025, 2025. arXiv preprint arXiv:2410.00722.
[9] Nathan W Henry, Giovanni Luca Marchetti, and Kathlén Kohn. Geometry of lightning self- attention: Identifiability and dimension. ICLR 2025, 2025. arXiv preprint arXiv:2408. 17221.
[10] Kaie Kubjas, Jiayi Li, and Maximilian Wiesmann. Geometry of polynomial neural networks. Algebraic Statistics, 15(2):295-335 328, 2024. arXiv:2402.00949.
[11] Bella Finkel, Jose Israel Rodriguez, Chenxi Wu, and Thomas Yahl. Activation degree thresholds and expressiveness of polynomial neural networks. Algebraic Statistics, 16(2): 13--10,, 2025. arXiv:2408.04569.
[12] Samuele Pollaci. Spurious valleys and clustering behavior of neural networks. In International Conference on Machine Learning, pages 28079-28099, PMLR, 2023.
[13] Yossi Arjevani. Joan Bruna, Joe Kileel, Elzbieta Polak, and Matthew Trager. Geometry and optimization of shallow polynomial networks. arXiv preprint arXiv:2501.06074 2025.
[14] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51 (3):455-500, 2009.
[15] Nicholas D Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E Papalex- akis, and Christos Faloutsos. Tensor decomposition for signal processing and machine learning. IEEE Transactions signal processing, 65(13):3551-3582 2017.

---

[16] Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, Danilo P Mandic, al. Tensor networks for dimensionality reduction and large-scale optimization: Part low-rank tensor decompositions. Foundations and Trends@ in Machine Learning, ((4-5):249-42)) 2016.
[17] Aditya Bhaskara, Moses Charikar, and Aravindan Vijayaraghavan. Uniqueness of tensor decompositions with applications to polynomial identifiability. In Conference on Learning Theory, pages 742-778. PMLR, 2014.
[18] Ricardo Borsoi, Konstantin Usevich, and Marianne Clausel. Low-rank tensor decompositions for the theory of neural networks. IEEE Signal Processing Magazine, 2026.
[19] Animashree Anandkumar, Rong Ge. Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of machine learning research, 15: 2773-2832, 2014.
[20] Héctor J Sussmann. Uniqueness of the weights for minimal feedforward nets with given input-output map. Neural networks, 5(4):589-593, 1992.
[21] Francesca Albertini and Eduardo D) Sontag. For neural networks, function determines form. Neural networks, 6(7):955-999) 1993.
[22] Francesca Albertini, Eduardo D Sontag, and Vincent Maillot. Uniqueness of weights for neural networks. Artificial Neural Networks for Speech and Vision, pages 115-125, 1993.
[23] Henning Petzka, Martin Trimmel, and Cristian Sminchisescu. Notes on the symmetries of 2- layer ReLU-networks, In Proceedings of the northern lights deep learning workshop, volume 1, pages 1-6, 2020.
[24] Charles Fefferman. Reconstructing neural net from its output. Revista Matemática Iberoamer- icana, 10(3):507-555 1994.
[25] Verner Vlačić and Helmut Bölcskei. Affine symmetries and neural network identifiability. Advances in Mathematics, 376:107485 2021.
[26] Verner Vlačić and Helmut Bölcskei. Neural network identifiability for a family of sigmoidal nonlinearities. Constructive Approximation, 55(1):173-224, 2022.
[27] Flavio Martinelli, Berfin Şimşek, Wulfram Gerstner, and Johanni Brea. Expand-and-clee cllusssterr parameter recovery of neural networks. In Proceedings of the 41st International Conference Machine Learning, pages 34895-34919, 2024.
[28] David Rolnick and Konrad Kording. Reverse-engineering deep ReLU networks. In International Conference on Machine Learning, pages
PMLR, 2020.
[29] Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of ReLU networks. In 8th International Conference Learning Representations, 2020.
[30] Pierre Stock and Rémi Gribonval. An embedding of ReLU networks and an analysis of their identifiability. Constructive Approximation, pages 1-47, 2022.
[31] Joachim Bona-Pellissie François Malgouyres, and François Bachoc. Local identifiability of deep ReLU neural networks: the theory. Advances neural information processing systems, 35:27549-2766 2022.
[32] Joachim Bona-Pellissiet François Bachoc, and François Malgouyres. Parameter identifiability of a deep feedforward ReLU neural network. Machine Learning, 1 11((11):43114499 2023.
[33] Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning, 2021.
[34] Quanhan Xi and Benjamin Bloem-Reddy Indeterminacy in generative models: Characterization and strong identifiability. International Conference on Artificial Intelligence and Statistics, pages 69 12-3999.. PMLR, 2023.
12

---

[35] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:1 893-11905, 2022.
[36] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International conference on machine learning, pages 41 14-
4124. PMLR, 2019.
[37] Aneesh Komanduri, Xintao Wu, Yongkai Wu, and Feng Chen. From identifiable causal representations to controllable counterfactual generation: A survey on causal generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openreviex. wttp://wwww.eeeetuummmmiii Pi=oouuummP-PUUUZNNem
[38] Akira Ito, Masanori Yamada, and Atsutoshi Kumagai. Linear mode connectivity between multiple models modulo permutation symmetries. In Forty-second International Conference Machine Learning, 2025.
[39] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023.
[40] Sumio Watanabe. Algebraic geometry and statistical learning theory, volume 25. Cambridge university press, 2009.
[41] Giovanni Luca Marchetti, Vahid Shahverdi, Stefano Mereta, Matthew Trager, and Kathlén Kohn. Position: Algebra unveils deep learning invitation neuroalgebraic geometry. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URL
[42] Abbas Kazemipour, Brett W Larsen, and Shaul Druckmann. Avoiding spurious local minima in deep quadratic networks. arXiv preprint arXiv:2001.00098, 2019.
[43] Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, and Volkan Cevher. The spectral bias of polynomial neural networks. In International Conference Learning Representations, 2022. URL /tttps://openrevverr..... wttpriiwww.eeeeffooeeeeeeeeeee ww-.orLLLLPPPPPLLLSSST
[44] Pierre Comon, Yang Qi, and Konstantin Usevich. Identifiability of an X-rank decomposition of polynomial maps. SIAM Journal on Applied Algebra and Geometry, 1(1):388 414, 2017.
[45] Vahid Shahverdi, Giovanni Luca Marchetti, and Kathlén Kohn. Learning on a razor's edge: the singularity bias of polynomial neural networks. arXiv preprint arXiv:2505.1 11855005.111...... 2025.
[46] Steffen Rendle. Factorization machines. In IEEE International conference data mining,
pages 995-1000. IEEE, 2010.
[47] Mathieu Blondel, Masakazu Ishihata, Akinori Fujino, and Naonori Ueda. Polynomial networks and factorization machines: New insights and efficient training algorithms. In International Conference on Machine Learning, pages 850-858. PMLR, 2016.
[48] Mathieu Blondel, Vlad Niculae, Takuma Otsuka, and Naonori Ueda. Multi-output polynomial networks and factorization machines. Advances in Neural Information Processing Systems, 30, 2017.
[49] Li-Ping Liu, Ruiyuan Gu, and Xiaozhe Hu. Ladder polynomial neural networks. arXiv preprint arXiv:2106. 13834, 2021.
[50] Feng-Lei Fan. Mengzhou Li, Fei Wang, Rongjic Lai, and Ge Wang. On expressivity and trainability of quadratic networks. IEEE Transactions Neural Networks and Learning Systems, 2023.
[51] Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Polynomial composition activations: Unleashing the dynamics of large language models. In The Thirleenth International Conference on Learning Representations, 2025. URL https:::p /ttpsr///ppeeiiieeeeeee het/forrumiddChoiiie
13

---

[52] Majid Janzamin, Hanie Sedghi, and Anima Anandkumar, Beating the perils of non non-convexity; Guaranteed training of neural networks using tensor methods. arXiv preprint orXiv:1506.08k 08473, 2015.
[53] Massimo Fornasier, Jan Vybíral, and Ingrid Daubechies. Robust and resource efficient identifi- cation shallow neural networks by fewest samples. Information and Inference: Journal of the IMA, (M2))625-6955 2021.
[54] Christian Fiedler, Massimo Fornasier, Timo Klock, and Michael Rauchensteiner. Stable recovery of entangled weights: Towards robust identification of deep neural networks from minimal samples. Applied and Computational Harmonic Analysis, 62:123- 2023.
[55] Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics, 76(1):1-33, 2024.
[56] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ICA: A unifying framework. In International conference on artificial intelligence and statistics, pages 2207-2217. PMLR, 2020.
[57] Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances Neural Information Processing Systems, 36:486033486663 2023.
[58] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned rep- resentations. In International Conference on Machine Learning, pages 9030-9039, PMLR, 2021.
[59] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. Advances in neural information processing systems, 35:16411-16422, 2022.
[60] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35: 1687--11-,,,,, 2022.
[61] V Lebedev, Y Ganin, M Rakhuba, I Oseledets, and V Lempitsky. Speeding-up convolutional neural networks using fine-tuned CP-decomposition. In Proc. 3rd International Conference on Learning Representations (ICLR), 2015.
[62] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Adv. Neur. Inf. Proc. Syst. 28, 2015.
[63] Xingyi Liu and Keshab K Parhi. Tensor decomposition for model reduction neural networks: A review. IEEE Circuits and Systems Magazine, 23(2):8-2) 2-, 2023.
[64] Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In Proc. 16th European Conference on Computer Vision (ECCV), pages 522-539 Glasgow, UK 2020. Springer.
[65] Emanuele Zangrando, Steffen Schotthöfer, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Geometry-aware training of factorized layers in tensor Tucker format. Proceedings, Adv. Neur. Inf. Proc. Syst., 2024.
[66] Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on learning theory, pages 698-728. PMLR, 2016.
[67] Maude Lizaire, Michael Rizvi-Martel, Marawan Gamal, and Guillaume Rabusseau. tensor decomposition perspective on second-order RNNs. In Forty-first International Conference on Machine Learning, 2024.
[68] Valentin Khrulkov, Alexander Novikov, and Ivan Oseledets. Expressive power of recurrent neural networks. In ICLR, 2018
14

---

[69] Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg. Jean Kossaifi, Shimon Whiteson, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agent reinforcement learning. In International Conference on Machine Learning. pages 7301-7312. PMIR, 2021.
[70] Sergio Rozada, Santiago Paternain, and Antonio G Marques. Tensor and matrix low-rank value- function approximation in reinforcement learning. IEEE Transactions on Signal Processing, 2024.
[71] Marco Mondelli and Andrea Montanari. On the connection between learning two-layer neu- ral networks and tensor decomposition. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1051-1060. PMLR, 2019.
[72] Rong Ge. Rohith Kuditipudi, Zhize Li, and Xiang Wang. Learning two-layer neural networks with symmetric inputs. In International Conference on Learning Representations, 2019.
[73] Pranjal Awasthi, Alex Tang, and Aravindan Vijayaraghavan. Efficient algorithms for learning 2epth-2 neural networks with general ReLU activations. Adv. Neur. Inf. Proc. Syst. 34:1344 13455- 13496, 2021.
[74] François Malgouyres and Joseph Landsberg. Multilinear compressive sensing and an application to convolutional linear networks. SIAM Journal on Mathematics of Data Science, 1(3):466444 475, 2019.
[75] Alex Casarotti and Massimiliano Mella. From non-defectivity to identifiability. Journal of the European Mathematical Society, 25(3):913-99) 1-11-9991111111 2022.
[76] Joseph M Landsberg. Tensors: Geometry and applications, volume 128. American Mathemati- cal Soc., 2012.
[77] Daniel Kunin, Jonathan Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. In International Conference on Machine Learning, pages 3560-3569. PMLR, 2019.
[78] Xuchan Bao, James Lucas, Sushant Sachdeva, and Roger B Grosse. Regularized linear au- toencoders recover the principal components, eventually. Advances in Neural Information Processing Systems, 33:6971-6988 2020.
[79] Pierre Comon, Gene Golub, Lek-Heng Lim, and Bernard Mourrain. Symmetric tensors and symmetric tensor rank. SIAM Journal on Matrix Analysis and A 2008
[80] Nicholas D Sidiropoulos and Xiangqian Liu. Identifiability results for blind beamforming in incoherent multipath with small delay spread. IEEE Transactions on Signal Processing, 49(1): 228-236, 2001.
[81] Ignat Domanov and Lieven De Lathauwer. On the uniqueness of the canonical polyadic decomposition of third-order tensors-Part Part II: Uniqueness of the overall decomposition. SIAM Journal on Matrix Analysis and Applications, 34(3):876-90) 2013.
[82] Nicholas D. Sidiropoulos and Rasmus Bro. On the uniqueness of multilinear decomposition of N-way arrays. Journal of Chemometrics, 14(3):229-239, 2000.
[83] Ignat Domanov and I Lieven De Lathauwer. Generic uniqueness conditions for the canonical polyadic decomposition and INDSCAL. SIAM Journal on Matrix Analysis and Applications, 3674)1157711589 2015.
[84] Hirotachi Abo and Maria Chiara Brambilla. On the dimensions of secant varieties of Segre- Veronese varieties. Annali di Matematica Pura ed Applicata, 192(1):61-92 2013.
[85] Ralf Fröberg, Samuel Lundqvist, Alessandro Oneto, and Boris Shapiro. Algebraic stories from one and from the other pockets. Arnold Mathematical Journal, 137)6137-160, 2018
[86] Alex Massarenti and Massimiliano Mella. Bronowski's conjecture and the identifiability of projective varieties. Duke Mathematical Journal, 173(17):323333116 2024.
15

---

[87] Kim Batselier and Ngai Wong, Symmetric tensor decomposition by an iterative eigendecompo- sition algorithm. Journal of Computational and Applied Mathematics, 308:69-82 2016.
[88] Samet Oymak and Mahdi Soltanolkotabi. Learning deep convolutional neural network via tensor decomposition. Information and Inference: A Journal of the IMA, 1033)1031-1071, 2021.
[89] Jacek Bochnak, Michel Coste, and Marie-Françoise Roy. Real algebraic geometry, volume 36. Springer, 2013.
[90] Paul Breiding, Fulvio Gesmundo, Mateusz Michatck, and Nick Vannicuwenhoven. Algebraic compressed sensing. Applied and Computational Harmonic Analysis, 65:374-406, 2023,
[91] Yang Qi, Pierre Comon, and Lek-HHeg Lim. Semialgebraic geometry of nonnegative tensor rank. SIAM Journal on Matrix Analysis and Applications, 3754):155-11580 2016. SIAM.
[92] Samuel Lundqvist, Alessandro Oneto, Bruce Reznick, and Boris Shapiro. On generic and maximal k-ranks of binary forms. Journal of Pure and Applied Algebra, 223(5):0022-200) 2079, 2019.
[93] Alexander Taveira Blomenhofer. On uniqueness of power sum decomposition. SIAM Journal on Applied Algebra and Geometry, 0(1):211-233 2025.
[94] Alex Massarenti and Massimiliano Mella. The Bexxande-HHirchowizz theorem for neurovari- cties. arXiv preprint 1rXiv:2511.19703, 2025.
16

---

A roadmap to the appendices
The appendices of the paper contain background on tensor decompositions and neurovarieties, the proofs of the technical results, well as discussion the changes between the originally submitted and final version of the paper. They are organized as follows:
Appendix/Alpresenee background neurovarieties for homogeneous PNNs. This is crucial part for understanding the link between finite identifiability of an hPNN, the dimension of neurovariety and the rank of the Jacobian of its parametrization map.
Appendix B contains the main technical tools used in the proof the localization theorem and follows the structure of Section In particular, it presents the proof' of necessary conditions for uniqueness (Section 4.1). background on tensor decompositions and Kruskal-base sufficient conditions for the identifiability of 2-layer hPNNs (Section
Appendix C presents the proof of the localization theorem (Theorem and its conse- quences for several hPNN architectures, as well as some supporting technical results.
Appendix Beaadixxxppue presents the proofs for the case of PNNs with biases. Appendix D.3 discusses the idea of Iruncation, an alternative approach to tackle the PNNs with biases.
Appendix E discusses necessary and sufficient conditions for the identifiability of hPNNs, as well as changes between the originally submitted and the final version of the paper which were done to correct mistake in the proof of one of the main results.
A Homogeneous PNNs and neurovarieties
hPNNs are often studied through the prism of neurovarieties, using their algebraic structure. Our results have direct implications on the expected dimension the neurovarieties, as explained in this appendix.
A.1 Neurovaricties and dimension
An hPNN architecture (d, r) defines map hPNN from the weight tuple w = (W1, 1..... Wz) to (polynomial) function space ;:
hPNN
hPNN drr [w) R dede - H.
The space H is the space of length-dy vectors of homogeneous polynomials of degree Ttotal T172 ............ TL-1 1 in do variables:
:= ->oo
thus H is a finite-dimensional vector space of dimension
N = dim( dr. (do + Ttotal - 1)
Protal
which follows from the fact that dim (.r,)) = (dtr-l).
The key observation is that hPNNNar-1] polynomial-in-teeee -nt-r-------eeeeeerr map. which has important implication on the space of networks with given architecture. The image Im Im(hPNNa, called neuromanifold, is semi-algebraic set The properties of Im a.r.N are tightly linked to the properties of the neurovariety Va, denned as the closure of Tm d,- (-) the Zariski topology, i.e., the smallest algebraic variety aagebraic se containing Im In(hPNN,,1)) The key property the dimension of the neurovariety which is equal to the dimension of the neurovariety
88 Prop.
2.8.2).
"The appendices have been reorganized and reworked for better readability.
88 Def. 1. set by polynomial equations and inequalities.
89
Def. 2.1:4]: a set cut out by polynomial equations.
'roughly defined as the dimension of the tangent space general point, see 89 $2.8] for details.
17

---

The properties of neurovarieties depend on the field (i.e., results can differ between R or C), and we focus on the real case. However, most of the results can be translated the complex case as well. We mostly follow [90] Section 4], and an overview semialgebraic sets can be also found in [9] (see [9]
for a detailed account).
The following upper bound on dim adr the bound was presented in
dim drr I min
Σ
dede-1
Σ de
dim
(8)
output space dimension
degrees of Freedom
If there is an equality in the bound (8). we say that the neurovariety has expected dimension. There are two fundamentally different cases when the expected dimension is reached.
Expressive case.
If the right bound is reached, i.e., the neurovariety:
dim ar dim ( =d, do + Ttotal -1
Ptotal
the hPNN is expressive, and the neurovaricty Var is said to be thick 77, as it fills the whole function space H (and thus the neuromanifold is positive Lebesgue measure). In particular, this implies that (see 77] Proposition 5]) any homogeneous polynomial vector from H (i.e., of degree Ttotal with dy inputs and di outputs, with degrees fixed as r1 = 122 _... = rL-1) can be represented as an hPNN with layer widths (do, 2d1, ..... 2d1-1, dz) and the same activation degrees.
Identifiable case. The left bound E d - de) follows from the presence of equiva- lences defined in Lemma (i.e.. the size of the vector w minus the number of independent rescalings) and defines the number of effective parameters of the representation (this is explained the following subsections). Moreover, the left bound is reached if and only if the hPNN architecture is finitely identifiable:
Proposition 10 The architecture hPNNN is finitely identifiable if and only if the dimension of V equal to the effective number of parameters, i.e., dim = a - drde-1- -/=1 de. In such case, Var said to be nondefective. Equivalently, the rank of the Jacobian of the map hPNN maximal and equal to EL. - - de at general parameter w.
Proposition 10 is central to the proof of the main results of paper. The proof of Proposition 10 relies on properties of fibers of polynomial maps and is reviewed in the next subsection, together with the Jacobian of the parameterization.
A.2 Polynomial maps and fiber dimension
We recall some key facts on the polynomial maps and their images. We begin by highlighting the link between dimensions of semialgebraic sets and the Jacobian of the polynomial maps.
Lemma A.1. Let yp : T R. be a polynomial map, and denote by (0) the n m Jacobian matrix. Let
ro == max rax
0
Then we have that:
1. rank (0) = ro for generic O (i.e., for all O E Th except set of Lebesgue measure zero, where the rank of the Jacobian is strictly less than ro).
2. To is equal to the dimension of Im((p) and (Zariski) closure:
To dim [Im(p)) = dim (Im(p))-
The proof of Lemma A.I is given in 90 Theorem 4.7] and the preceding paragraph (in 190), the number To is called generic rank of the parameterization yp). mainly follows from semicontinuity of the rank of matrix.
18

---

Remark A.2 (On genericity). Due to the algebraic structure of 9. the genericity statement in Lemma A. is much stronger: in fact, the set of points where rank{J,((0} # Ta a semialgebraic subset of Rm of dimension strictly less than m. The same holds for all generic statements and definitions in the paper (such as finite identifiabiliyy global identifiability, etc.), see the definition of genericity in 190, Definition 4.1).
Remark A.3. The right bound for neurovariety dimension ( follows essentially from Lemma A. indeed, in the case sp(.) = hPNNd, l.b. rank(Jp) does not exceed the dimension of the ambient space of up (equal to dim(H)).
The following lemma is key for linking finite identifiability to the dimension of the neurovariety. Lemma A.4 (Fiber dimension). Let ip R R be polynomial map, that To (Im((I(())) Then the dimension of its generic fiber is equal to m To, that is, for generic E IRm,, the preimage pp (x)( is semialgebraic set with
dim P (x) = m To-
Lemma A.4 is well known specialists, but in the literature it is mostly formulated for the complex case (see 900 Theorem 4.7]). For the real field it is special case [90] Theorem 4.9].
A particular case is when To m, in which case Lemma A.4 implies finiteness of the fiber:
Corollary A.S. The following two statements are equivalent:
For general 0 E IR"", rank(J ran""r
mm;
For general 0 Im, the fiber (i.e., the preimage x - (<p(f))) consists of finite number of points.
Proof. The statement follows from Lemma A.4 specialized to (ro m) and from the fact that O-dimensional semialgebraic sets are collections of finite number of points.
Finally we make the following remark that is very commonly used. Corollary A.6. rank (0o)) = m foror some 0g E TR"", then rank (J,()) m for generic O.
Proof. This directly follows from Lemma A.1 since in Lemma A.1 is equal to m.
Remark A.7. Corollary A.6 implies that finding a single point with full column rank Jacobian implies finitieness of the generic fiber.
A.2.1 The case of neurovarieties
The first implication Lemma A.4 the left upper bound in K). based on the following lemma from 77. for which we provide a short proof for completeness.
Lemma A.8 (77) Lemma 13]). For a general parameter w (W W L), the set of equivalent hPNN representations in Lemma is semialgebraic and of dimension El=1 de.
Proof. First, note that the set of equivalent representations is of dimension most LL-1 de (by the number of parameters). Consider a general w = (W W....., W L), so that the first column of each r, for 1,..., of vectors 11 E RR ...., -, E Iddz-1 having elementwise the same signs as Ug. Then there exist matrices D, so that the equivalent weight matrices We D.W D rt-1 have 8. exactly as their first columns. Thus the set of equivalent representations is exactly of dimension Σ[-1 de-
1, equal ny RRd does not have zero elements. Now take any collection
Remark A.9. The left upper bound in () simply follows from Lemma A.8 (as written in M Lemma 13]): indeed, the dimension of the fiber of hPNN must be least E1-1 de. This implies, by Lemma A.4
L
rank (J+(O))
Σ de dd--dee- Σ de
which is exactly the right dimension bound in by Lemma
A.I
19

---

Note that Proposition 10 will exactly consider the case when the equality is reached in ( for generic 0. Similarly Corollary [A.6] the following corollary of Lemma implies that for the case of neurovarieties it suffices to find a single set of parameters wr where the Jacobian of the parameterization of maximal rank to guarantee finite identifiability of hPNN architecture. This will be used in the proofs to give certificate of finite idenitifiability.
Corollary A.10. If there exists particular point Og such that equality is achieved in 9 then the equality in achieved for generic 0.
Proof. Since there exists such a Og then the To defined in Lemma A.
satisfies
L Ede - Σ
L-1
2
de.
(10)
1=1
t=1
But from (9. ro must be bounded from above by the same number. Therefore the equality for to is achieved in 10
A.3 Proof of the proposition
Proof of Proposition 10 We denote (p() hPNN [-]] for simplicity (so that = Eí dede-1 and n dim H)) and consider separately the "only if" + ) and "if" (= parts.
Assume that for a generic w the fiber yp (pw consists of finite number of equivalence classes, thus it is finite union of non-intersecting semialgebraic subsets of dimension E/11 de. Therefore, by 88 Theorem 2.8.5] the whole fiber - (vpww) has the dimension equal to Σt=1 de as well, hence dim arr Σ dzde-1 - Σi=1 de.
+ The proof follows a similar argument as in the proof of 90 Theorem 4.9]. We consider (Zariski open) subset of parameters without zero values 2 (R {0)) It can be shown that
the preimage of the image of its complement == p (K 2)) is a (semialgebraic) set of measure zero. Therefore for the set 2/ ạ 2 the preimage of the image is contained in 2/
(p((()) V
Note that any w E 2 can be brought (by diagonal scaling and permutation) to an equivalent form:
W. = We ]
W. E 1ixe-1)
(11)
for all Ł= 2,...., L where the reduced W parameterize the classes of equivalent parameters in V up to permutation. Now denote wे (W1, W 2...... W L) and define ww = (W w....., W.) with We as (II. Consider the following map
# + hPNN a.r (wr()).
Then if the generic fiber of 1a is finite, this will imply that on W', the fiber of the map contains finitely many equivalence classes. For this, note that the Jacobian of 1 is just a submatrix of the Jacobian of yp with exactly m E/-1 de columns. We will show that is full rank at generic point W.
Consider the following map
E (W W...W....... W....., DI.... D2) + (W W2..,, WL)
defined as
W. Dr
1...
D D-
W.
for é - 2,.., L (with the convention that D,. Idz.
Consider particular wy constructed as above (by normalization of a w E W). Then for a neighbor- hood V of wi and a neighbourhood y of (Id,...., I ... the map & is diffeomorphism from l x y to open neigbourhood of the corresponding wo w wwwo).
20

---

Consider the composition o o Then at the point (W1, 1, W WL, I.. ) we have that (i) the derivatives with respect to D at identity matrices are zero and (ii) the Jacobian of & with respect to up coincides with the Jacobian of v. hence it must have full column rank (m ES de) which is equal to the dimension of the neurovariety. Hence, the fiber of is finite, which implies finite identifiability of go
B
Main tools for the proof
This appendix contains the main technical tools used in the proof the localization theorem. It is organized in three subsections, following the same structure as in Section
presents the proofs of necessary conditions for uniqueness corresponding to Section 4.1 in the main body of the paper;
Appendix B.1
Appendix [8.2 presents background on tensor decompositions and the proof of Proposition 34 from the main body of the paper, which shows the link between 2-layer hPNNs and partially symmetric tensors;
Appendix B.3 presents Kruskal-based sufficient conditions for the identifiability of 2-layer hPNNs (Propositions 35 and 12 in the main paper).
B.1 Necessary conditions for uniqueness
In this subsection prove the key lemmas stated in Section 4.1 (Lemmma 30 and Lemma 31). These results give necessary conditions for the uniqueness of an hPNN terms of the minimality of an unique architectures and the independence (nondredundancy) of its internal representations.
Lemma 30
Let p = hPNN,[w)) be an hPNN of format (do. de...... ........,dd..... Then for any &
there exists an infinite number of representations of hPNNs p = (PNN,[w) with architecture (do...., de + dr... In particular, the augmented hPNN is not unique (and not finite-to-one),
Proof of Lemma 30 Let (Wo. 0. W w) the weight matrices associated with the representation of format (do, de..... ....,d,, dr) the hPNN = hPNN, [w]. By assumptions on the dimensions, the two matrices Wt E Reexde and W c+1 E Rddiixde read
W e+1 = W1
wal ; where, for each i, w; Rd++1
We V1
va.]" where, for each i, v. E Rdc-1,
& Without loss of generality, let assume none of the w, is zero vector and set
We
0
0i va. R 1)xdd
in which we add row of zeroes to W.. In this case, we can take the following family of matrices defined for any u E Id:
[u wi
wa, E xdd+)x
Then, we have that for any choice of u and for any z,
(u)
W WeiPr. (Wez) (Wez)
W Wet1p, (Wez).
(u)
The matrices W 6+1 and W E+1 for u + O have a different number of zero columns and cannot permutation/rescaling of each other, constituting different representations of the same hPNN In fact, every choice of u' that is not collinear to u and w, = 1,..., de leads to a different non-equivalent representation of p. Thus, have infinite number of coniviuiillnnn representations
(W 0, ... W W & W (u)
841..... Wz)
of format (do...., de + 1,..., dr) for the hPNN p = hPNN, [w].
ootherwise, we can replace zero vector w. with randomly chosen non-zero vector and set the corresponding 0
21

---

Lemma 30 can be seen as form of minimality or irreducibility of unique hPNNs, as it shows that unique h IPNN does not admit smaller (i.e., with a lower number of neurons) representation.
Lemma 31 For the widths d = (do...., dr),, let p = hPNN,, [wo] be a unique I.-layers decomposi- tion. Consider the vector output any f-th internal level < T. after the activations
qr(2)) = Pri Weo.... Pri Wi(x).
Then the elements (x) [
qtdd (z)) are linearly independent polynomials.
Proof of Lemma 37 By contradiction, suppose that the polynomials qe. (z), .... It.d, (æ) are lin- early dependent. Assume without loss of generality that, e.g., the last polynomial qe,d, (a)) can expressed as a linear combination of the others. Then, there exists a matrix B TR (dz=1) so that
qe,1 (x)
p W Pri-1 o... Pre4 We-1B
: ge -1 (x))
i.e. the hPNN p admits representation of size d (do: ..... de 1..... dr) with parameters
(W1.... ...... W We+1 B..B..... B.... W.). Therefore, by Lemma 30 its original representation is not unique, which is contradiction.
Using Lemma 30 and Lemma 31 we can prove the conditions on the Kruskal ranks of weight matrices that are necessary for uniqueness. These conditions are based on the notion of Kruskal rank which we recall from [15].
Definition 32 The Kruskal rank of a matrix (denoted krank( (A)) is the maximal number k such that any columns of A are linearly independent.
Note that the following two cases of particular interest also have simple equivalent interpretations:
krank{A) 2 is equivalent to saying that matrix A has columns;
krank( 2 2 is equivalent to saying that no pair of the columns of matrix A are collinear.
As in Lemin 31 let the widths be d = (do...., dr.), and = hPNN, [w] have a unique (or fimite-t--one) L-layers decomposition. Then we have that for all e 1..... L
Proposition 33
krank [ 2, krank (eek E+1} 2 1,
where krank Wetle e+1) 2 1 simply means that W We+ does not have zero columns.
Proof of Proposition 33 Suppose that krank WT] < 2. Then we have that at level L, the vector
(x()) of internal features defined in (5) contains linearly dependent or zero polynomials, which
violates Lemma 31
Similarly if krank(W e+1) = 0, then the neuron corresponding to the zero column be pruned to obtain a representation with (dg - 1) neurons at the f-th level, which implies loss of uniqueness by Lemma 30 and thus leads contradiction.
B.2 Background tensors
In this appendix, we first present background on tensors and the CP tensor decomposition, and demonstrate the link between hPNNs and the partially symmetric CPD (Proposition 34 in the main paper).
B.2.1 Basics on tensors and tensor decompositions
Notation. The order of tensor the number of dimensions, also known as ways or modes. Vectors (tensors of order one) are denoted by boldface lowercase letters, e.g., Matrices (tensors of order two) are denoted by boldface capital letters, e.g., A. Higher-order tensors (order three or higher) are denoted by boldface Euler script letters, e.g., X.
22

---

Unfolding of tensors. The p-th unfolding (also called mode-p unfolding) of a tensor of order
T Im X is the matrix (P() of size mp (m1m2 mp-17pp11 ma) defined as
T
bpd
T
Mom where j = +
(i
m
n=1 nfp
Epp
We give an example of unfolding extracted from
14
Let the frontal slices of X E 3x be
1
4
7
10
13
16
19
22
X
-
2
5
8
11
X
14
17
20
23
3
6
9
12
15
18
21
24
Then the three mode-n unfoldings of X' are
1
4
10
13
16
19
22
X11
2
5
8
11
14
17
20
23
3
6
9
12
15
18
21
24
1
3
13
14
15
(²
4
5
6
16
17
18
7
9
19
20
21
10
11
12
22
23
24
X
2
4
6
10
11
12
13
14
15
16
17
18
22
23
24
Symmetric and partially symmetric tensors. A tensor of order 8, T E Rmm X=-Xm, is said to be
symmetric if mi =.... my and for every permutation a of sl.
T f2 i, =T hect) The tensor T T x-, is said to be partially symmetric along the modes (p + 1, 1,...,8) for p < 8 if Mp+1
m, and for every permutation of 1,......
T An 1, =T (p.---ppo
Mode products. The p-mode (matrix) product of T E Rmm xm XXm, denoted by T o A and is of size my ... mp-1 J mp-1 X... mg. It is defined as
with a matrix A E R/xmp
mp
T A ig thomel
T
A jáp
R-term decomposition. An R-term canonical polyadic decomposition (CPD) of tensor T is decomposition tensor as a sum of rank-1 tensors [4 [5], that is
(4)
T
i=1
where, for each E {I,...,s) s). a, E Im and @ denotes the tensor (outer) product operation. Alternatively, denote A (p)
a (p)
a (p) m and the corresponding CPD as
T [A (1) (2)
A [.)
When T is partially symmetric along the modes (p + +.+11... s)...,s). for A (p+1) A (p+2)
its CPD satisfying A (*) is called partially symmetric CPD. The case of fully symmetric tensors (i.e., tensors which are symmetric along all their dimensions) deserves special attention [79].
The symmetric CPD of fully symmetric tensor T E Rmxmx.. is defined as
T
= Us a 8 ... ...8800 ai 1=1
where u, E IR are real-valued coefficients. With slight abuse of notation, represent it compactly using the same notation as an order-(n (n + tensor of size x m X ... X m, as
T = [u, A, A],
where u E Rlxm is a x m matrix (i.e., row vector) containing the coefficients u, that is, u4 Uy i ,,...,R. R.
In the definition of CPD, do not require H to be minimal (thus R is necessarily equal to tensor rank).
23

---

B.2.2 Link between hPNNs and partially symmetric tensors
Recall that Hdd.r denotes the space of do-variate homogeneous polynomials of degree S r. The following proposition, originally presented in Section of the main body of the paper, formalizes the link between polynomial vectors and partially symmetric tensors.
There is one-to-one mapping between partially symmetric lensors FF Rdax Xdo and polynomial vectors f E (Hodo.r) (dx,, which can be writlen as 10
Proposition 34
F f(x)) FF
with F(l) E Rddxxd the first unfolding of F. Under this mapping, the partially symmetric CPD
== [2 W...... .W]]
mapped hPNN W 2Pr Wwx). Thus, uniqueness of hPNN (do da W2)] is equivalent to uniqueness of the partially symmetric CPD of F..
Proof. We distinguish the cases, d2 and da 2. We begin the proof by the more general case d2 2 2.
Case d2 2 2. Denoting by u, E Rd2 the i-th column of W2 and E RRdo the i-th row of W1, the relationship between the 2-layer hPNN and tensor F can be written explicitly as
f(x)= W WPrr(W ( 1x)
41
u, (v. x))"
Σ 1=1
u,(v (w To
=
= W2
=())
where @ denotes the Khatri-Rao product. The equivalence of the last expression and the first unfolding of the order- + 1) tensor F can be found in [14].
The special case d2 = 1. When dg = 1, the columns of W2 E 1xd; are scalars values u, E R, i 1.....d. dj. In this case, (WT @ ... @ WT) W becomes equivalent to the vectorization of F, which is fully symmetric tensor of order with factors W and coefficients [W2h1, i 1.....d..
B.3
Kruskal-based conditions for the uniqueness and identifiability of 2-layer networks
B.3.1 Sufficient conditions for uniqueness
The direct links between 2-layer (L 2) hPNNs and partially symmetric CPDs in Proposition 34 allows us obtain sufficient conditions for their uniqueness by means of Kruskal-based uniqueness results for the CPD, which we recall in the following lemma.
Lemma B.1 (Kruskal's theorem, s-way version R-term CPD with A e XR such that
82), Thm. 3). Lei = (2 A (2) ( A, (0) the
krank(A (0) 2 2R (s
(12)
Then the CP decomposition of 'T is unique up to permutation and scaling ambiguities, that is, for any
alternative CPD A A (2)
(1)
A (s)
there exist a permutation matrix II and invertible
diagonal matrices A1, Ag ........ A, such that
À (6)
A 10 IIA,
for 1,...,8.
1oin the definition of f. the tensor Sr viewed de vector when multiplied by F())

---

Now we prove Proposition 35 giving sufficient conditions for uniqueness in the case L = 2.
Proposition 35 Let pm (x) W2Pr, (W (z) be 2-layer hPNN with W RRdixdo and W &d and layer sizes (do, d1, d2) satisfying do, d1 2, d2 2 1. Assume that r 2 2, krank W 2) 1, krank[W]] 2 2 and that:
2d1 - krank( (W2)
krank [
then the 2-layer hPNN Pwo (a) is unique (or equivalently, the CPD of F 66 is unique).
One can apply Proposition to show that the 2-layer hPNN P (x) is in one-to-one correspondence with the order r + partially symmetric tensor
Proof of Proposition 35
[W W ..I.. F
(13)
thus, the uniqueness of hpNN, W 1. W2] is equivalent to that of the CP-decomposition of F in By Lemma B.1 the dy-tem CP decomposition of 'T is unique provided that
krank 3 + krank w 2 2d1- +r.
By noting that krank| W [rank(W]] > 1 and rearranging the terms, we obtain the desired result.
Note that for the case dg 2 (i.e., hPNNs with at least two outputs), Proposition 35 gives conditions that may hold for quadratic activation degrees 2 2. On the other hand, for networks with single output (i.e., d2 1). it requires 2 3.
B.3.2 Sufficient conditions for identifiability
Equipped with the sufficient conditions for the uniqueness of 2-layer hPNNs obtained in Proposi- tion 35
we can now prove the generic identifiability result stated in Proposition 12
Proposition 12
Let du, d1
d2 2 1 be the layer widths and 2 2 such that
2d, min(d1,,d d2)
r
min do)
Then the 2-layer hPNN with architecture ((do, di d2), (r)) is globally identifiable.
Proof of Proposition 12
For general matrices W E Rd xdo and Wa E Rd2xd, we have
krank W [ = min di),
krank W min d2, d1).
Moreover, do. d1 2, d2 2 implies that generically krank [W] 2 and krank(Wz2 > This along with 4 means that the assumptions in Proposition 35 are satisfied generically (for all parameters except for a set of Lebesgue measure zero). Thus, the hPNN with architecture ((ds, d2), (r)) is globally identifiable.
C Proof of the localization theorem
This appendix contains the main proofs of the localization theorem (Theorem 11 for deep hPNNs, as well as supporting lemmas and auxiliary technical results. We also provide proofs of the corollaries that specialize this result for several choices of architectures (e.g., pyramidal, bottleneck) and to the activation thresholds, discussed in Section 3.2 of the main paper.
Results from the main paper: Theorem Corollaries [6 19 and 17
25

---

Roadmap of the proof: The proof of the localization theorem requires some setup. The main idea, as briefly sketched in Section [4.3 (cction[4.3jor of the main paper, is construct recursion for Jacobian of the parameter map, and to certify that it has maximal rank (generically). This relies crucially on the properties of the neurovarieties associated to an hPNN as explained in Appendix A particular on Proposition [0] and Lemmal A.4 which link the the finite identifiability of the hPNN to the rank of its Jacobian. The proof of the main result presented towards the end of this appendix, in Appendix C.7, and proceeds by induction. However, it requires several technical tools which are build in the subsections that precede it.
Appendix C.I starts with some preparatory results on the rank of the Jacobian of 2-layer hPNN, setting the base case.
Appendix C.2 defines the so-called last layer map (i.e., the map that composes a dg variate polynomial with one hPNN layer) and illustrates the structure of its Jacobian by means of detailed example.
Appendix C.3 presents key proposition which establishes certificate show that the Jacobian of the last layer map has maximal rank, and before proceeding to the proof, illustrates the result with an example.
Appendix C.4 introduces some additional notation and setup which will be used in the proof the key proposition.
presents the proof of the key proposition for the special case when the number of input variables do is equal to the number of variables used in the certificate (equal to the smallest bottleneck in the network).
Appendix C.5
Appendix gives the proof of the key proposition in the general case when the number of input neurons do can be larger than the number of variables the certificate.
Appendix C.7 contains the proof of the localization theorem.
Finally, Appendix C.8 presents the proofs for the results concerning the implications of the localization theorem different hPNN architectures.
Simplifying the notation:
In Appendices C. to we denote the number of input neurons by m. the number of hidden neurons in the second-to -o-t layer by and the number of output neurons as n. For two-layer networks, we denote the first- and second-layer weight matrices by V and W, respectively.
C.1 Preparatory lemmas rank of Jacobian of 2-layer PNN
Lemma C.1. Let (m, d, n) and r be such that the 2-layer hPNN with architecture ((m, d. n).r) fimitely identifiable (resp. the partially symmetric d-term CPD is generically unique). Then for general matrices V, W the Jacobian of the map ((V. W) = hPNN, (V, W)], given by
Jo= Jo W) J J
has maximal possible rank:
rank (m 1)d, and also its submatrix containing the derivatives with respect to elements of V is full column rank:
(14)
rank rank(J[ = mnd.
(15)
Proof. The first statement follows from dimension of the neurovariety (that is, (m + n 1)d), and the second statement follows from the fact that the subset of pairs (V, W) with W given as
W
W (
W
parameterizes an open subset of the neurovariety (i.e., due to the scaling ambiguity, almost any pair of V and W can be reduced to such a form). As shown in the proof of Proposition 10 (specialized to (W,, W2) = (V, W)), the reduced Jacobian is full column rank:
rank ( (1g))
J W
} = md (n 1)d,
26

---

where 1o denotes the Jacobian with respect to W. Note tnis implies that all the submatrices are full column rank and, as therefore JV
is full column rank.
Remark C.2. The conditions in Lemmal C.1 are satisfied, for example, if the Kruskal-bass generic uniqueness conditions are satisfied (see Proposition [2].
Before giving the elements of the main proof, we provide an example of explicit Jacobian computation for the map hPNNd,r which will be the guiding example for the proof of identifiability.
Example C.3 (Simplest architecture). Consider example (m, d, n) (2, 2, 2). r and denote
the elements of V and W
[1
31
W1,1 W2,1
W1,2
V =
W
02
32
W2.2
so the hPNN map (V, W) = hPNN d.r VV W is given by
(V, W) wi (atz) 3(x2)²² 2 w2(00201 + 32
where w1, w2 w2 denote the columns of the matrix W:
W1,1
W1.2
W2,1
W2.2
The image of ip lives in the space of vector polynomials (H2.2)* x2 (of dimension 6). therefore, the blocks of the Jacobian J'V) and J (W) are of sizes 6 x 4. The matrix J has as its columns derivatives with respect to 0i and By. for (1,2) which are, respectively:
dip 2w 2 + B182),
Oe
2w 22 (1122).
(16)
da,
03,
Let us choose the canonical basis of (H2.2 ) x²2 as e
zz ie [1,2), L 10, 1,2), where e; are
unit vectors. Then the block v represented the matrix form as:
Be Daj
0e 331
de Dug
33g
c
W W1,101
0
W W2,202
0
01
W1,131
W1,101
W1,282 0
W1,20'
J = (2)
Wi,131
W W1,2832 0
e2#
W2,101
0
W2,202
e2
W2,131
W2,101
W2.282
W2.200
W2,1
0
W2.282
The block (W) contains the derivatives with respect to Wi.j. i,j E {1, which are:
ej(ar + 3,
(17)
aw.,
In the same monomial basis, the matrix can be expressed as
Bie JW
DW2,1
Dag
Ddg
0
0
20 3
2a 32
0
((W)
2
87
B2
0
e
0
af
0
a2
0
201
91
0
20 32
e
0
B7
0
8
Remark C.4. is easy to show why For this example, we choose particular and W
14 and 15
are satistfied for the architecture in Example C.3
be identity matrices, which gives us
2
0
0
0 0
0
0
0
0
( J(m]
0
0
0
0
1
0
0
1
0
0
0
0
2
0
0
0
0
1
27

---

It is easy to see that the left block (matrix ) hus rank 4, and the total mairix has rank 6 (2 + 1)2. Therefore, by Corollaries A.6 and A.10 (14) and (15) are satisfied generically.
We will also need an explicit form of the Jacobian in the general case, which is generalization of the expression in Example C,3
Remark C.5. Let (m, d, n). v, V and W be in Lemma C.7 denote V/ E mh and w E R
With some abuse of notation
V
Wdl,
W
wa)
and let 2 =[21
2ml
has the form be the input variables. Then the hPNN pp = hPNNd,
M V. W](=) = Σ w
(18)
==1
Therefore, we have that derivatives with respect to the elements of the matrix W can be expressed as
dep 0.p
oW. (ww;); (v)
(19)
where is the i-th unit vector in IR, and, with respect to elements of V, we have
Oe de rzz wy(w (1
(20)
,V a(v)e
Note that Lemma C.I concerns the dimensions of linear spaces spanned by the sets of polynomials in 20
Also,
(20 and 19 are generalizations of (16) and (7). respectively.
C.2
Jacooiin of composition of polynomial maps
The goal this subsection, is to exhibit the structure of the composition polynomial NN-like maps and their Jacobians. Consider outer layer of an hPNN, which is denoted as
W pr (gi......
In order see what happens when substitute variables q1,-.., qa by do-variate polynomials ((11...,, E (Has. xd we introduce the following definition (which corresponds to 7
Definition C.6 (Last layer map). Let W E Raad be n x d matrix E N. We define the map da that transforms a vector of R-degree do-variate polynomial as follows:
vp: (H x Raad + (Hoo. R²) Ro)
(q(2),..., Id.), W) + Glg, W| W p. TJ))),
and denote the Jacobian with respect the parameters (and its blocks)
Jg W)
(1(0)
j(W
where J(q) has d( R columns and (WW db has ni columns.
Example C.7. Similarly to Example C.3 we take the case n = 2, d 2, 2, and denote
W Wi w2). Then the last layer map becomes
(qq,, 42) ungh w242
Consider special case do 2, R = 2 so that ib maps (q1, 92) E (H22) to vector polynomial in (H2,))** x2, and let the imput polynomials be parameterized as
(2 22) = (2.0) n² 24,
21 + (0.2) 23
j E {1,2},
where of ((1.22), (in. 12) {(2, 0), (1, (0.2)) are the coefficient of gj next to monomials x/tx'33 whose blocks are described below.
Then the Jacobian (g W) is a 10 x 10 matri IT]
since H6.() 5.
28

---

The block (o is 10 x 6 matrix, whose columns are the 6 polynomials (similarly to (16):
Oip
Ōqj 0af (6))5) 2wo 22 ( (2)( j (1.2), (2), (1..))
((2,0) (1.1), (0.2)
(21)
In the canonical basis is given as j
54
Puj 12.07
the2
E1:Ty
P0g
(0.25
(2.0)
W1.20 (2.0)
c
Wi,1q
0
W1.19 (1,1)
W1,14; (2,0)
W1,292 (1.1)
W W1,29p (2,0)
=12122
W1,191
(0.2)
W1,1 (1.1)
Wi,191
(2.0)
W1.202 (0.2)
W Wi,20
(1.1)
W (2,0)
21212
0
W1,19 (0,2)
W1,1 (1,1)
W W1,292
(0,2)
W1.29p (1,1)
(2)
0
0
W1,141 (0,2)
0
Wi,292
(0.2)
exai
W2,191
(2,0)
(2.0) 22.202
0
0
=2
W2.1q (1,))
W2.19 (2.0)
W2,20 (1,1)
W2,292
(2.0
(2,0)
0
e
W2.191
(0.2)
W2,191 (1.1)
W2,191 (2.0)
W2.242 (U.2)
W2,202 (1,1)
W2,292 (2,0)
W2,141 (0,2)
W2,141 (1.1)
W2,29 (0,2)
W2,203 (1,1)
(0.2)
(0,2)
0
W2,141
0
0
W2,292
The second block, similarly to [7 }. is 10 x 4 matrix whose columns are
Oip
aW.j e: (0)(,,,,,)2) i (1,2),
(22)
and has similar structure that J in Example C.3 (it will be explicitly shown in the next example).
C.3 A certificate of maximal rank for the Jacobian of the last layer
The following proposition gives condition for when the Jacobian of the last layer map has maximal rank, based constructing certificate.
Proposition C.8 (Certificate of last layer map). Let m,d, n and r 2 2 be fixed, and the matrices
V gaxm and W E gaxd be such that the equalities (14) (5) are satisfied. Fix do 2 m, R
and consider the polynomial vector (II......... Fmm E (A m. C (Had. xd defined as
xa
7(x) = V
(23)
Then we have that the evaluation of the Jacobian of the last layer map ib (see Definition C.6 the
point (q, W) is of maximal possible rank:
(y W)) d(n + do + R ( R11)
(24)
R
and its submatrix containing derivatives with respect to is full column rank
rank(J (q. W)) d do + R 11-1
(25)
Before proving Proposition C.8
we give an illustrative example of the Jacobian of the last layer map
evaluated the certificate q.
Example C.9 (Example C.3
continued). We continue Examples C.3 and C.7 In this case, the vector
polynomial q from Proposition C.8
reads
m (11, x2
[an + B1x3)
(01, =
03 11, F22
=
(a2a7 + (2x3))

---

I.e,, using the notation of Example
C.3 the coefficients of the polynomials are
(2.0)
a 1.11
01..21 0,1,0,11)
11 (0.2)) (az,, 0, 02).
42
42
42
Specializing Example C.7
(and removing factor for simlicity), we get
dv
dvl
dut (2.00
0N 14,27
10,23
ou2 (2.07
0u2 (4,27
dug (0,2)
W1,101
0
W 12.20²
0
e aaaaa
0
W W1.101
W1 W1.202
W 11
0
W
W
0
W 1,220
3
W1 21.2
1
(9)
qg W)
612
0
0
0
0
W
12,2/
2
W 12
2,200
0
0
W2,101
0
W2,20' 0
0
12
12,10'1
22
W2,201
0
12,181 0
0
W2,2 0
0
e222
W2.1
W2.2
(W) The matrix 1
then, according
22
becomes
8w
UWa,1
8W1,2
0
O
201 31
20
0
(W))
g W)
18?
O
33
e
af
a2
e
0
0
0
0
e
2a
31
201
ermaz
0
0
0
89
$2
The crux of the proof of Proposition C.8 is the following observation. Ie stack together matrices
j(W) and permute the rows and columns as follows, we get the block diagonal +
=
2
matrix
Da
ov [2.0T
10.23
22.07
10.27
DWL.1
JW2,1
DW1,2
DWL.3
(1.1)
ejat
Wi,r0
of
erajas #2
Wi,8:
W1,
W1
W,
202
2ax
20
e
W1,1
0
W12
0
W2,10n
W2,202
2
J
Wa.1
W2,
W2.922
W2.002
20
2cr
ega
Wa.18
W3
68
Wi,101
W1,202
e
0
Wi,88: Wi,88
W2,101
W2,20g
Wa,18, Wa.2%a
We see that the top-left block of the matrix nothing but the matrix
(
(W)
where is as in Example C.3 thus it has rank 6, and its left 4 columns are linearly independent. Moreover, its bottom-right block can be viewed as submatrix (( (taking first and third columns, for instance), and therefore has full column rank? 2.
Thus matrix Jo has rank 8 6 + 2 and 1.9 has rank 6 4+2 2.
30

---

C.4 Extra notation for the proof of the proposition
In order prove Proposition C.8
we introduce extra notation for the columns of Jg. We first let
W = w Wdl as in Remark C.5
so we can express
q. W
w (g)))
==1
Already this, similarly to (19 gives us
aw. + a(w,): v e:(g;)",
and we denote the linear space spanned by these polynomials (i.e., the range of J (
((W) = span aaw.s 1,== = range(J
Now we look into details of the structure of the matrix f(ر Let i = (f1,..., ddo) E I be a multi-index that rums over
I= (i == (i1 idg) i1...., id 2 and i + + id = R}
so that the coefficients polynomial q E Hdo., can be numbered by the elements in ((I1,..,,aal
Σ iEI
1 ... Fd
ido
Then, the columns ) for q(x) = =x)
[a(x))] are given by the polynomials
f (,0(2)) = du which are precisely generalizations of 21 We denote the spaces spanned by such polynomials
(q. W) (rx)) ... To
(w((a))++ == j= d, i
(26)
d 43)
( = span (=)))
and their span (the range 1(9) as
((9) span [ jiez range {(
Example C.10. notation of Example C.7 I {(2, 0), (1, 1), (0, 2)). In this case, we have
Oip
On
((q.(2.0)) span
(2.0)
0q (2,0)
((a((1.1)) span
0qg (1,1)
00g
(1,1)
((q.(022) span
(0,2)
042
(0.2)
which correspond the columns {1,4), (2, f3, 6), respectively, of the matrix J(C).
Remark C.11. Proving Proposition C.8 (i.e., proving that (4
25 hold) is equivalent showing
that
do + R -
')
dim ( = d(n - 1) +
(27)
R
dim ((a) = (do du + R - 1)
(28)
R
respectively.
The strategy of proving that the dimensions of these subspaces are maximal is to show that the individual subspaces 3) are orthogonal under some conditions (which similar to bringing J into the block- biock-diagonal form in Example C.9
31

---

C.5 Proof of the propasition on certificate: case m do
We first prove the proposition for the case when the number of input variables dg equal to the number of variables m used in the certificate.
Proof of Proposition C.8 (case m do). Recall that in the notation of the previous subsection we need calculate
dimspan span (C(W), ,span (( (((
dim
Now let us consider these subspaces for particular choice of of the form 23
We have that
j.a from (26) have the form
(m) (m) (21,.... Imm
iil mud Rt) Aml mod R).
m
polynomial in x
Therefore get that ((9.8) ((q.,e) unless one of the following conditions holds:
(i,L) C To
with Io == -(R,,,........... 0). (0, R, O, o).... (0, . R)}. For the same reasons we get
((W) (c(.)))))) for all To.
Therefore, we get
rank(Jø] dim span (CIW), spann ((a.)) (Span(C HiEZo)
Σ dim(
+
TET\Ta
Let us look at those dimensions separately. Denote
[2]
2m] with
21 = xf.
x
so that for q of the form 23 it holds
1 = UJz.
Then, for i E I To it is easy to see that
dim(C(a.)) = dim span ((w, (w,,0)))) (0, )]-1) =d,
where the last equality follows from Lemma and (20). By doing the same substitution, we obtain that
span (((W), spant [ ((9. (( = span ((::(]))))) ndd
Jij=1' "))" ( d.m
),0=1
which is exactly the set of vectors in 199 20). Therefore, by Lemma C.I we have dim span (L(W) (Cla.e) )eEIs) (n 1)d md, and dim span ( C (l),e) leEI = md.
(29)
(30)
Taking into account that
R + m 1a-1) 1)
# (Tn) =
m
and #(T)
R
this proves 27 for dg = m. Equality 28 (for dy = m) can be proved similarly using the fact that
rank ( (J(O) (q. W)} dim span ( (( EET
Σ TE
dim ((a.(().)))
md d(# # d(#
32

---

C.6 Proof of the propasition: extending to the case of more variables
Proof of Proposition C.8 cass m du). We denote by Im (with some abuse of notation) the multi- indices that correspond to the monomials that depend only on , Tmi
Im = fi E I 1mth....... ddo = 0)
and we define
L. span (( Jietm: ( Lest == span (C (C( From the first part of the proof (case do), we have already proved that
JEI
R + m - 1
dimspan (c ( d(n - 1) +c
(31)
R
and
dim C( R m
R .
(32)
What is left to show is that adding Lext to these subspaces does not drop the rank.
Since the particular choice of q - (II1,,.., depends only on the m variables, thanks to (26 we have
Iddo
T Tm+l
Fm
J do
0do
polynomial in Jm
This immediately implies that ((a.8) 1 C(a.e) if (Im+11...., Ado)) * .do)) as well as ((9.0) ((W) if (fm41,.... ido) ) K O. Therefore, we get
( L(g) Lext and span (c(9),2(W))
span (c ((()) ① Lext:
and, consequently, just need to show that Lezt is of maximal dimension. To show this, split I' Im into a direct sum according to the degrees of the last dg m variables:
dim Cent
Σ
dim (e))
=
imt1 20
656m+1 +... +ldo SR
where
((g = span ((... www..f........i.m............... )(1 ((1..... ):6 20. 21 +
But then, for a fixed (fm+1,..., ido) such that bm+1 +... + ddo = Ro S R, the dimension this
subspace is equal to
dim (mg. = dim spanfz) ... dd de0
w ) (ā))"-1}
R-
dimspan am wy (ū,))
f 20
t...ttmm It Ro
dim diimspan Roth w. (0,(
Ro
20
but the latter set of polynomials linearly independent because subset of the basis vectors of ((9) which are linearly independent by (32). Therefore we get Lext is of maximal possible dimension (the spanning columns linearly independent).
C.7 Localization theorem
Theorem 11 (Localization theorem) Let ((do....,din dr..), vI...)) be the hPNN format. For
{ - 0. 0.....,L L 2 denote de min(do...... .d.......d.. de). Then the following holds true: if for all & 1...,, the two-layer architecture hPNN (d, dede 12) is finitely identifiable, then the
L-layer architecture hPNN (. is finitely identifiable as well.
33

---

Proof. (Proof of Theorem We prove the theorem by induction.
Base: L = 2 The base of the induction is trivial since the case L the full hPNN consists in a 2-layer network.
Induction step: (L - k - 1) + (L k)
Assume that the statement holds for k 1.
Now consider the case L k.
With some abuse of notation, let (W i...,, Wt L-),,, so that w = (0, W 1) and denote
R = ri -.
Let be the one defined in Proposition C.8 but given for the last subnetwork, so that n = di, d = d TL-1. W = WL. Then we have that
w == hPNN (r......--)) 100, W t.] ())()), Wr. where h0) = hPNN (m........))0))
Therefore, by the chain rule
( q=h(0)) (O)
(()
J (pw))=
q=h(0)
J1 (ww)
J2
Now are going show that the matrices have necessary rank for generic 0. For this, note by the induction assumption, for generic 0, we have
L-2
L-2
rank (Jn)(
Σ dide+1
Σ de.
Now we show the ranks for other matrices. Observe that
R + do (R+ 1 + (dz - -1)d-1 (33)
rank( (()) q=)(())
(W) q=)(())
R
due to the essential ambiguities. But then if we find particular point 0, where rank maximal for = h0) then the rank (33) will be maximal for generic 0.
But then, let m = dr.-1 min fiinddo dr 1 and consider the following matrices:
I
0
I
0
W
0
0
1-2
0
0
and
W
WL-1=
V
0
for V E RRdz 1mm generic. Then we get that for 1 I...... wr-2)
h()) V
exactly as in Proposition identifiability of hPNN (åg and2 1.de).rz f.l).. Therefore, rank in (33 will be maximal for generic (0, W z) and also
C.8 (whose conditions are satisfied by the assumption on finite
rank ( ((0)) ( dr-1
R do - 1)
R
for generic 0 (i.e., the matrix is full rank).
34

---

This leads to (ank(Ji(w)) = (0) for generic 0. Finally, we have that
rank p(w) = rank{J](O)) + rank [I
JI(0)_ J2(0)}
2 rank(J + rank
J2(
spen (5(ر) gomnoo))
1-2 d=dz+1
de + (dr. - 1)d
Σ d=de+1
Σ
de:
where Uky denotes the orthogonal projection onto a subspace U. On the other hand,
L-1
rank P.. (w)) -
Σ
dede+1
Σ
de
1=
8=1
due to presence of ambiguities (bound ( Hence, equality holds and therefore the
neurovariety has expected dimension.
C.8 Implications of the localization theorem
Corollary 16 (Pyramidal hPNNs are always identifiable) The hPNNs with architectures containing
non con-increasing layer widths (except possibly the last layer), i.e., do 2 dj Z ..d111 2. and
d, 1, are finitely identifiable for any degrees satisfying (i) F 2 if dz 2; or (ii) .2....,TL-2. 2 2, FL-1 2 if 1.
Proof. (Proof of Corollary
16) This follows from Theorem I and the following facts:
For such choice of de, de de for all l - ..... --
Network (de-1, de, de+1) with d 2 de is identifiable (by Proposition 12 for:
- Fr 2 2, in case d 2 2;
- rz 2 3, in case de+1 = 1.
Corollary 17 (Activation degree thresholds for identifiability) For fixed layer widths (do.....da, with de 2 2, == O...., I.....I--.. the hPNNs with architectures (d, (L.-1) are identifiable for any degrees satisfying
re 2 2de
Proof of Corollary 17 Note that the assumptions guarantee that de 2 2. Then the Kruskal bound (in Proposition 12 for identifiability of (de-1, de, de41) can be bounded as
2d min(de, de+1) < 2d - 1. min de-1) -
therefore. for re 2 2dg- the hPNN (de-1, 1. de, det1), re is identifiable, which implies the identifia- bility of the L-layer architecture by Theorem
Corollary 19 (Identifiability of bottleneck hPNNs) Consider the "bottleneck" architecture with d 2 d1 2 ... ds S db+1 .... S dr,
and de 2 2. Suppose that Ti,...,To 2 and that the decoder part satisfies de S do for i E {b I.... L 1}. Then the bottleneck hPNN is finitely identifiable.
35

---

Proof of Corollary 19 This follows from Theorem 11 and the following facts:
For layers & E {I...., by (the encoder part), we have de = de and thus identifiability of de, de+1) holds for 2 2 (the same argument in the pyramidal case).
For layers & E {b + 11,+1.... 1,..,, 1.} (the decoder part), we have de = d, and thus identifiability of (de-1, de, de-1) holds for
de
Tt
db -1'
which, after rearranging, gives the desired result.
D Analyzing case of PNNs with biases
This appendix contains the proofs and supporting technical results for the identifiability results of PNNs with bias terms presented in Section 3.3 of the main paper. We start by establishing the relationship between PNNs and hPNNs and their uniqueness by means of homogeneization. We then prove main finite identifiability results showing that finite identifiability of 2-layer subnetworks of the homogeneized PNNs is sufficient to guarantee the finite identifiability of the original PNN.
Results from the main paper: Definition 20 Propositions
23
24
Lemma
26 and Corollary 28
D.1 The homogeneization procedure: the hPNN associated to a PNN
Our homogeneization procedure is based on the following lemma:
Definition 20 There is one-to-one mapping between (possibly inhomogeneous) polynomials in d variables of degree r and homogeneous polynomials of the same degree + variables. We denote this mapping Pa. d,y Haa+1r by homog(.), ), and it acts follows: for every polynomial p P Ldr. p = homog(p) E Ha411 is the unique homogeneous polynomial in d + variables such that
pla 1) =p( p(E T.).
Proof of Definition 20
Let p be a possibly inhomogeneous polynomial in d variables, which reads
p(z1 Fa) =
Σ la|Sr
ba 2 ... 2
for a (01,..., Qa). One sets
p(.1 p(.1....7d..)) Fd. 2)
Σ Sr
barl 01 ...
x. od 0-1
which satisfies the required properties.
Associating an hPNN to a given PNN: Now we prove that for each polynomial p admitting a PNN representation, its associated homogeneous polynomial admits hPNN representation. This is formalized in the following result.
Proposition 23 Fix the architecture (fi,..,,TI.ll. TL-1) and d = (do...., dr). Then polynomial vector E (oooraaal) yxdz admits PNN representation = PNN,,--(w b)] with (w, b) as 22) if and only if its homogenization p homog(p) admits an hPNN decomposition for the same activation degrees r and extended d = (do + ......d.-11- + 1, dr,,, P hPNN [,, [w).
T (W -W1,...
WL), with matrices given as
W. be
E ((((++))(de
g L,
W.
1
WL
by g(de)x((d--tt))
t=L.
36

---

Proof of Proposition 23 Denote Pi(z) = Pri (W 1 + b1).
Let x
x RR Observe first
that
P1 (3) 2²1
Pri W [
We proceed then by induction on L 2 1. The case trivial. Assume that L. 2. Then
Pi ( 2
W 2Pr (W (
W₂
W 2 + + 5mb2. b2.
Specializing at 2 = 1, we recover
W 2P (x) b2 =p(x)= p(x, 1).
hence
W2pr W 1 = -(
For the induction step, assume that g hPNN (d++1....dd ++1,de). W 2, .,... w the homo- geneization q PNN (d) (d)....d))) ((), 2..... WL), (b,..., bL))]. By assumption,
(1 ")) q(p) (z. 1)) - (x), p((),,
p(x, 1) 7
which completes the proof.
Proposition 24 JfhPNN, [i)] from Proposition 23 is unique an hPNN (without tuking into account the structure), then the original PNN representation PNN,-(w: , b)] is unique.
Proof of Proposition 24 Suppose hPNN, [w] is unique (or finite-to-one), where i is structured as Proposition 23 Note that any equivalent (in the sense of Lemma specialized for hPNN,, (w))
parameter vector w (W i...., W L) realizing the same hPNN must satisfy
W.D.W,D.Y. -L-1 T PE-1.
l e< I
(34)
W WED P
L-11
t= L.
for permutation matrices Ps and invertible diagonal matrices De, with Po = Do = I. We are going
to show that bringing We , to the form
W
b
l L
W
0
1
(35)
[W'L
b
t= L.
that does not introduce extra ambiguities besides the ones for PNN (given in Lemma
By Proposition 33 for e I..... the extended matrices satisfy krank( (We))(WWW))))))))))) 2 (as well for any equivalent krank[ (W, T) 2 2). This implies that the matrix W, , contains only single row of the form [0 .0....0i (which is its last row). Therefore in order for W, to be of the form 35)
the matrices P1. D2 must be the form
0
0
P
D
0
0 1
Iterating this process for = 2, L
1, we impose constraints of the form
07 D
0
P 0 1
1
This implies that (W,,8a b() and (We, be) must be linked in Lemma
4
Now suppose that hPNN, [ir] is finite-to-one. Then the same reasoning applies to all alternative (non-equivalent) parameters w that are realized by a PNN, because Proposition| 23 holds for every solution. Since there are finitely many equivalence classes, the corresponding PNN representation also finite-to-one.
37

---

D.2 Generic identifiability conditions for PNNs with bias terms
Lemma 26 Let the 2-layer hPNN architecture ((dg + 1. 1.d1 + 1, d2), (r1)) be finitely (resp. globally) identifiable. Then the PNN architecture with widths (do, d1, d2) and degree on is also finitely (resp. globally) identifiable.
Proof of Lemma 26 By Proposition 24 we just need to show that for general (W2, b2, W b1), the following hPNN unique (finite-t--one)
p(z) W2
b2] Pr, ( (Wa) 1
(36)
with W1 given
W = W1 bj 1
We see that W, lies in subspace of (d1 + 1) x (do + 1) matrices.
We use the following fact: by multilinearity, both uniqueness and finite-to-one properties of an hPNN are invariant under multiplication of WN on the right by any nonsingular (do + 1) x (do + 1) matrix
Q. We note that the image of the polynomial map
[(do-1)x(d+3))
R x
x Rd R(d++1x(do+1))
(Q, W
which is surjective, and its image is dense. Therefore, identifiability (resp. finite identifiability) holds except some set of measure zero in then it also hold for W constructed from almost all (W1, 1,, pairs. For example, for finite identifiability this explained by the fact that there is a smooth point of the hPNN neurovariety corresponding to the parameters (W2 w1),
Proposition 27 Let ...o....,e dL), (L-1)) be the PNN format. For 6 O....,L. denote de min do....,,d: .de}- Then the following holds true: If for all é 1.....I, each two- is finitely identifiable, then the T-layer PNN with architecture (d, r) is finitely identifiable as well.
layer architecture iiPN (5.1-+1 -.d
For the proof of the main proposition, we need the following lemma.
Lemma D.1. Global (resp. finite) identifiability of an hPNN of format ((m, d, rt),r) implies (resp. finite) identifiaiiliyy of the hPNN in format ((m, d, n + k), r) for any k > 0.
Proof. Let the parameters of the larger hPNN be such that
A
W₂ =
A E Raxd B BE RRRd WT.
B
so that
hhPNN (m.d. W 1, A]
Ao, ( a Ba, (x)
hPNN (m) W1, W2]
hPNN (m. W 1, B
But then assume that hPNN (m.d.n). ( 1, A is finite-to-one. Then by Lemma 31 have that the elements of qı (x) o. (W (z) are linearly independent, hence the linear system
hPNN (m.d.)).- [W, B] ] (q,(a))
has the unique solution, equal to B. Note that for (W1, W2), the subset of parameters (W1, A) is also generic, hence global (resp. finite) identifiability for widths (m, d, n) implies global (resp. finite) identifiability for widths (m, d, n + k).
Proof of Proposition 27
We are going prove that under the condition of the theorem, two hPNN architectures for degrees r and widths
(do 1,..., de-1 + 1,) and (do 1,..., . de 1 + 1, d2 1) are finitely identifiable.
We proceed by induction, similarly as in Theorem
38

---

Base: L = The base of the induction follows is trivial since it is the 2-layer network, and from Lemma D.l for the architecture (dz-1 + 1, de + 1, de+1 + 1).
Induction step: (L k - 1) + (L k) Assume that the statement holds for L
k. As in the proof of Theorem [1| we set 0 (W I....., W i). so that w (0, Wr., where We is as in Proposition 23 and
h 1, Now consider the case
denote R = TL-2- The difference is that the parameters are now 0 == (()), where
0 (W W 1- b
Let di be the one defined in Proposition C.8 but given for the last subnetwork, that TL-1, W
n dr, d
W L- Then we have that
p W
hPNN (5] v(h(0(0)). W).
where h()) hPNN
10
Again, by the chain rule
g(a)
g=)( (0())
J ( q=((0()))
Jz(0)
J W)
I
(w)
(2(
where the matrix in the right hand side is full column rank. Therefore, we just need to show that the left hand side matrix is full column rank for particular 0 0(0). But, for this, remark that we can use almost the same construction example Proposition C.8 but choosing
slightly different matrices: θ
W L-11 with
0
0 0
W
0
Im
and
W L-1
10
V'l.
where in Lemma
C.1 we can choose generie V
structured as
(V')
[('))
V
Indeed, we need this to be smooth point (i.e., full rank Jacobian of W Pre-1 (V'x:), which is full rank for generic (VV) (V') by the same argument as in the proof of Lemma 26
But such O' indeed belongs the image of ((0) they share the needed structure, which completes the proof.
Corollary 28 Let ((do....,
TT-1)) be such that de 2 1, and re 2 2 satisfy rt
2(d, + 1-i1)mnddd, + 1,11,, de+1)
min(de, de-1)
then the L-layer PNN with architecture (d, is finitely identifiable (and globally identifiable when = 2).
Proof of Comollary 28 sition 27
This directly follows from combining Lemma 26 Proposition T2 and Propo-
39

---

D.3 Truncation of PNNs with bias terms
In this appendix. we describe an alternative (to homogenization) approach to prove the identifiability of the weights We of I, b]] based on truncation. The key idea is that the truncation of PNN is an hPNN, which allow one to leverage the uniqueness results for hPNNs. However, we note that unlike homogeneization, truncation does not by itself guarantees the identifiability of the bias terms bg.
For truncation, we use leading terms of polynomials, i.e. for E Pa. we define lt(p) E Hay the homogencous polynomial consisting of degree-r terms of p:
Example D.2. For bivariate polynomial 2,2 given by
p(11, x2) ax₃ + bz102 + cx2 + ez1 + f 9:
its truncation = =t(pp] E H2 becomes
x2) = axi + 1 + cu.
In fact It).) is orthogonal projection dar Ha also apply [t to vector polynomials coordinate-ve wise. Then, PNNs with biases can be treated using the following lemma.
Lemma D.3. Let p = PNNa,- b)] be a PNN with bias terms. Then its truncation the hPNN with the same weight matrices
It (t = hPNN
Proof. The statement follows from the fact that It (((((()))) = It (((()()))) Indeed, this implies [t(((1,))) m) + e)") ((v, ((), which can applied recursively PNN PNNa.-((uu a.-((w, b)].
Example D.4. Consider a 2-layer PNN
f(z) 2 (W www++bbb b,) + b2.
(37)
Then its truncation is given by
1t (+))) = W (W aæ)
This idea well-known and fact was used in to analyze identifiability of a 2-layer network with arbitrary polynomial activations.
Remark D.5. Thanks to Lemma D.3 the idensifiabiiity results obtained for hPNNS can be directly applied. Indeed, obtain identifiablity of weights, under the same assumptions listed for the hPNN case. However, this does not guarantee identifiabiliyy of biases, which was achieved using homogeneization.
E Localization theorem: necessary and sufficient conditions for identifiability
This appendix has been added to the camera ready version on the request of the program committee. It explains the changes between the original submission and the camera-ready version.
Our main technical result in Theorem gives sufficient conditions for finite identifiability of deep hPNNs based on identifiability of two-layer subnetworks. In an earlier (submitted) version of the paper, the following results were claimed.
[w), w (W 1, ..... WL) L) be L-layer hPNN with architecture (d. r) satisfying dg...., dL-1 2 2, d, 2 and fi...., TL-1 2 2. Then, hPNN, |w] is unique according Definition if and only for every f 1...., L 1 the 2-layer subnetwork hPNN (ro)l We+1) is unique as well.
Claim (A, specific uniqueness). Let
This strong claim implied another claim on identifiability of hPNN architectures, which can be seen as a counterpart of the current Theorem 11
Claim (B, identifiability). The L-layer hPNN with architecture (d, r) satisfying do....,dl, 2 dr. > 1 and ri...., .T.. 2 2 is identifiable according to Definition 8 if and only if for every the 2-layer subnetoork with architecture (14-11,e,, de+1), (re)) identifiable as
{= 1,..., I. well.
40

---

smallest
bottleneck
de
do (input)
dp (output)
de-1
de
de+1
Figure 1: (from NeurIPS poster) Necessary and sufficient conditions for identifiability of an L-layer PNN. Blue: necessary conditions, i.e., "only if" part of claim B (identifiability of the ((de-1, de, de+1), (re)) subnetwork), Red: sufficient condition as given by Theorem I] (identifiabil- ity of the ((de-1, de, det1), (re)) subnetwork).
We note that the "only if"" part always holds (as argued in the beginning of Section 3.1 non- uniqueness of any 2-layer subnetwork implies non-uniqueness of the overall network. The relation between necessary and sufficient conditions for identifiability illustrated in Fig. [
Thus, both claims (A) and (B) were in fact aiming to answer the following questions:
(A) Does uniqueness of all 2-layer subnetworks (TNN(r.((WW We We+1)l imply uniqueness of the overall network (r ()((W)(WWWWWWW 1...., W
(r1....-L
(B) Does identifiabilit of all ((d--1, 1, de, 1,+1), (rr)) 2-layer urchitectures imply the identifiabil- ity of the overall architecture ((do, ................................ dz), (T1.....FL-1)))
We show below that the answer these questions is negative, both for specific uniqueness (uniqueness of a particular choice of parameters) and generic uniqueness (identifiability given architecture), which motivated the update of the paper.
E.1 Supporting examples
Absence of specific uniqueness (counterexample claim (A)). Consider the simplest archi- tecture with d 2,). Proposition 12 Example 41 from the last section of the paper provides example of specific network of the format (d, r violating claim (A), We provide below an expanded version of this example.
22, for which the conditions of Theorem I1 are verified due
Example
(No specific uniqueness). Consider two polynomials:
x
p
Note that
?
2
P2(211 r2), therefore this polynomial vector can be written
p(x) = P2 W 222(2r)
for the following choice of weight matrix:
11 -1
W₂
that have
p(x) = 1202 (W, 22)2(2))
(2.NN(222)( [II, W2, I2)]
where 1 is the identity matrix. On the other hand, we can use the expansions
af + 2² (x) + x22 + (x) x²
2
(0) + x2))
(x) x2
2x1x2
2
41

---

and the fact that
(अ$ - (z + xz)² (2x12²)²
to show that there exists an alternative hPNN expansion of p(z), summarized as
p(z) = W3P2 W (2
hP 2 W2, Wa).
where
W 1 0 -1
We see that the two representations are not equivalent: (I2, W2. I2) / (W 1 W2. W3), as W cannot be obtained from scaling and permutations of rows of Iz-
On the other hand, all the matrices in the expansions (I W 2. Wa) are 2 2 invertible and thus, for example, the networks hPNN W2)] and hPNN (2) (W2, [2] have unique representations (similarly to Example(7) More precisely, all matrices (12, Wa 2, Wa) as well as their transposes have their rank and Kruskal rank both equal to 2, and therefore the conditions of Lemma B.1 are satisfied.
Absense of generic uniqueness (counterexample claim (B)). Example 41 is not just isolated example that can be circumvented by looking at a generic parameter set, as shown in the following example.
Example E.1 (No generic identifiability without further assumptions). We provide counterexample to the conjecture that localization holds in full generality in the generic sense based on the count of dimension. Consider the following architecture:
d = (2,3,3,,) and == (3,3).
It is easy to see that the subnetworks ((do, d2), ri) and ((d1, d2, dz), r2) both satisfy the Kruskal- based criterion in Proposition as
32 2d, min(m2, d1)
2d2 min (d3, d2) 5 min d1
3
min(d1,
2
so both subnetworks are identifiable. However, due to Proposition 10 for the global network (d, r)
be identifiable the dimension of its associated neurovariety must be equal to
dodp + d,d2 + d2d3 di d2 12.
However, the image of hPNN ddr) [:] is in the space of degree-9 homogeneous bivariate polynomials, and therefore the neuromanifold (and the neurovariety) lies in H2s- But H2,9 has dimension 10, thus arrive at contradiction with the identifiability of the 3-layer network.
E.2 Statement of changes
In the camera-ready version, the claims (A) and (B) have been replaced with Theorem [I which uses stricter condition. This replacement preserves the main conclusions and contributions of the original paper, notably:
1. The localization of identifiability identifiability 2-layer subnetworks (composed by two consecutive layers) is sufficient to guarantec identifiability of a deep L-layer polynomial network;
2. As a consequence, uniqueness theorems for tensors can be leveraged prove identifiability of deep PNNs; for example, well-known Kruskal theorems imply:
a) that pyramidal networks (and their generalizations) are identifiable in degrees 2 2: b) linear bounds the so-called activation degree thresholds (i.e., identifiability holds for degrees linear in the layer widths);
3. Identifiability of networks with biases is implied by identifiability of (augmented) bias-free PNN architectures.
42

---

Drawbacks:
Despite the fact that our main conclusions still hold, the amended version of the localization theorem lead to the following changes:
The theorem and the corresponding corollaries for deep architectures concern generic properties (and not specific) and finite identifiability (instead of global identifiability..
Theorem 11 requires stronger assumption on 2-layer subnetworks: not only each 2-layer block needs to be identifiable, but also with possibly smaller number of inputs.
This stronger condition weakened the result for networks with a bottleneck layer, but keeps the same conclusion (that is, decoder network needs to have higher degrees compared to the encoder in order to allow for increasing the layer widths).
In the following, we explain the mistake in the original proof of Theorem 11 and discuss the current challenges to extending the amended proof to the localization of global identifiability.
E.3 Remark on the mistake in the original proof and related problems
The mistake in the original proof of Theorem 11 concerned equations (11)-(13) in the original paper (Section A.2.2 of the original supplementary materials), in the induction step of the theorem (going from L - 1 to L layers). The original argument is based constructing the polynomial vector (z) using the flattening operation x 60r' + 2, -yg.'' - R (d))" The issue is that the equation (12, original paper) is only valid on subset of z (z structured as a tensor power) and thus does not imply (13, original paper) we originally claimed. The absence of this implication broke the inductive argument, requiring the proof to be amended.
An interpretation of this issue is that the flattening destroys the structure from lower layers. In fact, the flattening mapping corresponds
projection appearing in the computation of the decomposition of polynomials as sums of powers of forms (see the commutative diagram in [92] Section 4)), which makes such computation (decomposition as a sum of powers of polynomials) very difficult and currently an open problem in general, unless additional knowledge can be used [93]
Our new proof still proceeds similarly by induction (going from L- to L layers), where the induction step is related to showing non-defectivity (finite identifiability) of a subvariety of variety of powers of forms, thus connecting
subtle questions in algebraic geometry, such as Fröherg's 's conjecture (22 (the latter not solved in full generality, see [85) for an account of recent progress). Extending finite identifiability to identifiability seems challenging, at least with the techniques we are aware of; very recent work in algebraic geometry [75] [66 shows that this transition finite identifiability implies global identifiability) is possible for the so-called X-rank decompositions, but this result is only applicable to shallow polynomial networks. We are not aware of any systematic progress in the direction of non-additive structures, of which deep PNNs a special case. Thus, the transition from finite to global identifiability of deep PNNs was left as an open conjecture (Conjecture 40) in the remera-ready version of the paper. We hope that future progress in the field of algebraic geometry will provide the adequate tools to settle this challenging problem [2
Whiile preparing the update of the camera-ready ready version of the paper, we became aware of a recent preprint 94 that claims to prove much stronger (in many cases) result than Theorem 11 and claims global identifiability
well.
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We provide clear description of our contributions in Section I.1 in the
introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our paper is of theoretical nature, we outline all the assumptions used to derive the results very clearly.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
44

---

Answer: [Yes]
Justification: We clearly state the assumptions used in each theoretical result, and complete proofs are provided in an appendix in the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification: Our paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
45

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: Our paper is theoretical and does not include experiments, thus does not require data code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
ollicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pblle pwwppppguuuiieeeeep httpa///uuudess/eeeuuuuiiiis
(CodeSubiisss wwtplubmmsssionii ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: Our paper is theoretical and does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification: Our paper is theoretical and does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
46

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Our paper is theoretical and does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification:
Our research adhered to every aspect of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper is theoretical in nature (we study the identfaabillyyluiqqenes of neural networks). Thus, our results advance the understanding of the behavior of neural networks. We highlighted the potential positive impacts of our identifiability results on the interpretability of neural networks in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
47

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
48

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
49

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in our research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwpurrppsscccoofeeeeeeeee cc/Conferencess///////eeeccc/eee for what should or should be described.
50


Reviews:
[{"review_id": "Mv8AnnOeyd", "reviewer": "Reviewer_Mfr2", "summary": "The paper considers identifiability of the parameters of deep polynomial neural networks. Identification is an important, and often less studied, property of NNs as it is directly linked to the potential to interpret the NNs weights. The authors show that for an equivalence class up to permutations, deep PNN are identified if and only if each 2-layer subnetwork is also identified, extending existing results for 2-layer networks. The paper's main result follows from (1) a rank condition to ensure identifiability in the 2-layer case and (2) an induction argument to extend it to the L layer case.  The authors show how their results can be applied to a variety of structures and given conditions to check whether a PNN structure can be identified.", "questions": "Some smaller questions beyond the two questions in the weakness section:\n\n1. In definition 8 why do we additionally introduce the requirement that the parameters cannot be chosen on sets of measure zero? Is this also a requirement for definition 6 and is the set of parameters in the equivalence class defined by Lemma 4 measure zero? \n\n2. Is Proposition 13 new? I would be helpful to relate it to previous work (e.g. [7]) and to the recurrent example and how it relates to the full rank condition of the weight matrices. \n\n\n* Some typos: for eg. Lemma 32 *of with* architecture", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The topic of identification of general L-layer deep PNN is important as it is a less studied topic, but crucial to understand interpretability of NNs.\n\n2. The main result that 2-layer uniqueness can be extended to L-layer uniqueness appears correct, slightly surprising at first sight, and potentially a great insight. \n\n3. The paper's review of how the identification results may be used to check different architectures may be potentially useful for researchers in choosing how to construct PNNs.\n\nWeaknesses:\n1. A bit more discussion on how interpretability relates to identification, given that the result is up to permutation matrices would help make the paper more relevant. How large is the equivalence class? Are there other equivalence classes for which a similar result could have been proven? \n2. The paper presents many results and it is often slightly unclear (beyond Theorem 11) what is their contribution versus building on existing work. Furthermore, what is the relevance of the polynomial activations for the arguments? It seems that the results and proof methods would work generically for different activations, and if such, do similar results exist for NNs with other activations. More discussion on this would be appreciated.", "was_revised": true, "final_justification": "see comment.", "rebuttal": "We would like to thank the Reviewer for the careful evaluation of our work and for the valuable suggestions for improvements. In the following, we provide responses and clarifications to the weaknesses or questions raised by the Reviewer.\n\nQ1: **How interpretability relates to identification and size of the equivalence class**:  Thank you for the insightful question. In fact, the equivalence class of the parameters of a neural network (i.e., a set of parameter ambiguities which will lead to a representation of the same function) depends  on choice of the activation function. In a more general setting, this has been elegantly formulated through the notion of \"intertwiner groups\" studied in [34], which characterizes classes of equivalent parameters for various activations. \n    \nFor the case of polynomial neural networks with monomial activation, the equivalence class  described in Lemma 4 consists of permutations of neurons and rescaling of the input and the output of the activation function. If a PNN is unique/identifiable, then this equivalence class describes all possible representations of the same network with the same number of neurons. For the case of the polynomial activation function of general shape (not just a monomial), the equivalence class reduces to the set of permutations of neurons (which is unavoidable in any MLP), and no rescaling is possible [Shahverdi et al., 2025].\n    \nFrom a practical perspective, if we assume the weight matrices to be properly normalized to fix the scaling ambiguity, identification is linked to interpretability in the sense that for an identifiable network, any alternative representation will be a permuted version of the original one, whereas for any non-identifiable model the parameters and representations could be completely different. Moreover, identifiability plays an important role in model comparison, \"stitching\" or averaging, which has been used in several recent works [34], [Ito et al., 2025], [Ainsworth et al., 2022]. In such works, one finds a permutation (and possibly rescaling) that best aligns the hidden representations of different pretrained models in order to properly perform model comparison, averaging, uncertainty quantification, among other tasks. For this procedure to yield meaningful results, identification is essential.\n    \nWe will emphasize the role of identification in the revised manuscript, as well as the influence of the equivalence class of parameters.\n\n[Shahverdi et al., 2025] Shahverdi, V., Marchetti, G. L., and Kohn, K. (2025). Learning on a Razor's Edge: the Singularity Bias of Polynomial Neural Networks. arXiv preprint arXiv:2505.11846.\n\n[34] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep-.-\nlearning models and their internal representations. Advances in Neural Information Processing\nSystems, 35:11893–11905, 2022\n\n[Ito et al., 2025] Ito, A., Yamada, M., and Kumagai, A. (2025) Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries. In Forty-second International Conference on Machine Learning.\n\n[Ainsworth et al., 2022] Ainsworth, S. K., Hayase, J., and Srinivasa, S. (2022). Git re-basin: Merging models modulo permutation symmetries. The Eleventh International Conference on Learning Representations.\n\n\nQ2: **Novelty of the results vs. building on prior work, existing results for other activation functions**: \nWe thank the Reviewer for this important question. The main novelty of the paper is on two key results, namely, Theorem 11 (our localization theorem) about the uniqueness of deep PNNs, and Corollary 12 on their identifiability, as well as the key results which are used in their proofs. The uniqueness of homogeneous PNNs in the 2-layer case is already known in the literature, but, to the best of our knowledge, there exists no previous localization results that allowed one to leverage them in the study of deep PNNs. In particular, all results on the uniqueness of deep homogeneous PNNs and (shallow and deep) PNNs with biases are completely novel.\n\nRegarding the relevance of polynomial activations, the main advantage is that this is the case of deep NNs with nonlinear activations that are amenable to analysis with powerful methods of algebraic geometry. This is a very active research direction, as witnessed by a spark of recent results on geometry of the related neuromanifolds, expressivity of the networks, optimization landscape and properties of critical points [37]. For our paper, polynomial (monomial) activations are crucial for key steps in the proofs. Moreover, these techniques can serve as a key first step in the study of general polynomial activations, see the recent preprint that appeared after submission of this article [Shahverdi et al., 2025]. We also believe that an extension to other activation functions (for example, analytic) is feasible, for example, by using polynomial approximations, but this would require additional work and major changes in the proof techniques.\n    \nFinally, we highlight that some identifiability results for deep networks also exist for other architectures, for example, for networks with ReLU [29,30] or tanh [23] activation functions. However, the proof techniques used in these works are completely different, and do not apply to the case of polynomial activation functions. Moreover, an important characteristic of our proofs is that they are constructive and can be exploited as a first step in the development of algorithms to learn the weight matrices of a PNN. We will also emphasize this distinction in the related work section of the revised manuscript.\n\n[7] Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.\n\n[11] Bella Finkel, Jose Israel Rodriguez, Chenxi Wu, and Thomas Yahl. Activation thresholds and expressiveness of polynomial neural networks. arXiv preprint arXiv:2408.04569, 2024.\n\n[23] Charles Fefferman. Reconstructing a neural net from its output. Revista Matemática Iberoamericana, 10(3):507–555, 1994.\n\n[29] Pierre Stock and Rémi Gribonval. An embedding of ReLU networks and an analysis of their identifiability. Constructive Approximation, pages 1–47, 2022.\n\n[30] Joachim Bona-Pellissier, François Malgouyres, and François Bachoc. Local identifiability of deep ReLU neural networks: the theory. Advances in neural information processing systems, 35:27549–27562, 2022.\n\n[37] Giovanni Luca Marchetti, Vahid Shahverdi, Stefano Mereta, Matthew Trager, and Kathlén Kohn.\nAn invitation to neuroalgebraic geometry. arXiv preprint arXiv:2501.18915, 2025\n\n\nQ3: **Why identifiability definitions exclude a set of measure zero**: In the definition of identifiability/generic uniqueness (Definition 8), the requirement that for a network to be identifiable uniqueness must hold for all points except possibly for some subset of measure zero serves to eliminate some pathological cases which would appear had the definition allowed the weights to be completely arbitrary. A special case of parameters to exclude is the subnetworks (i.e., networks representable with fewer neurons in at least one layer). This corresponds to weight matrices that have zero rows or columns, or collinear rows. In fact, such networks corresponds to the singular points of the underlying neurovarieties [Shahverdi et al., 2025].\n\nHowever, we emphasize that our paper does not only treat generic uniqueness. Definition 6 concerns only a single set of parameters, and says whether a specific network representation is unique. The main theorem (Theorem 11) also applies to specific uniqueness, and does not assume genericity of the parameters. We will clarify these definitions and their implications in the revised version of the paper.\n\nQ4: **Relate proposition 13 to results in [7] and rank condition**:  Thank you for this comment and for the suggestion for clarifying the relation between this proposition and previous work and the link to the recurrent example. Indeed, Proposition 13 is related to the identifiability of 2-layer homogeneous PNNs, which was known in the literature (the link between 2-layer homogeneous PNNs and a CP tensor decomposition was already established in [7]). Thus, Proposition 13 (and Proposition 38, corresponding to the case of fixed weight matrices) are direct consequences of the uniqueness of this CP decomposition, which is known in the literature (in fact, stronger conditions are available for identifiability of homogeneous PNNs in the 2-layer case, see Remark 17). We emphasize that the novelty in our paper lies in the constructive proof for the uniqueness and identifiability of deep PNNs (as well as of shallow and deep PNNs with biases), which comes from the uniqueness of the CP decomposition, see the proof of Proposition 38. This is totally novel and, in turn, supports the development of algorithms, since learning deep PNNs can then be performed by computing the CPD of its corresponding tensor. This will be clarified in the revised version of the paper. We will also relate this proposition to the running example, to show how it applies to its corresponding architecture in the revised paper.\n\nQ5: **Typos**: Thank you very much for pointing this out, we will fix this and other typos in the revised paper."}, {"review_id": "3U0jLiocn6", "reviewer": "Reviewer_vryW", "summary": "This paper investigates the identifiability of deep Polynomial Neural Networks (PNNs). Identifiability of a network means the ability of identify the network parameters from its functional input-outputs. \nThe central is a two-fold \"localization theorem\" which proves that the global identifiability of a deep PNN is equivalent to the identifiability of all its constituent 2-layer sub-networks. The proof is relatively elementary and easy to follow, nevertheless elegant and clean. Leveraging this result, the authors connect the of identifiability for a two layer network to the uniqueness of partially symmetric tensor decompositions. This framework allows them to derive new, explicit conditions under which specific deep PNN architectures, such as pyramidal and bottleneck networks, are generically identifiable. The paper also provides improved, linear bounds on the activation degree thresholds required for identifiability and extends its analysis to PNNs with bias terms via a homogenization technique.", "questions": "- From what I understand, all the proofs of inedibility rely on this \"general\" configuration weights, that are everywhere except on some zero measure subset. While this is very true for initialization, I wonder if network that has been trained will remain in this general configuration or not? \n- Do authors have any ideas to generalize their results to new activations that are not homogenous, namely GeLU, tanh, etc? Or the proof method here completely breaks down? \n- While the algebra shown here is quite interesting, the polynomial basis is really not a good abstraction for real activations. But something like Hermite polynomials are in fact a natural basis there. Is it conceivable to extend these proof methods to some other type of basis rather than polynomials? \n- While the issue of identifiability is very interesting, another topic of high interesting to ML audience would be the complexity aspect. meaning, even if we know a network is identifiable, can we find it (reconstruct it) in a reasonable time? Can we turn one of these i Another aspect would be, the  tightly related topic is also the sample complexity? As in, how many samples do we need to identify a network?  Can authors comment on this? \n- Could the authors please comment on the relationship between their proof strategy and Fefferman's? Is there any link at all or not?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# strengths\n\n- The paper provides a rigorous and comprehensive analysis of deep PNN identifiability. The specific conditions derived for pyramidal and bottleneck networks (Corollaries 18 and 19) are novel and valuable to researchers in this specific domain.\n\n- The work goes beyond mere existence proofs. It provides explicit, checkable inequalities (e.g., Proposition 13) that link architecture choices (layer widths, activation degrees) to the identifiability property. The connection to tensor decomposition is well-executed and powerful.  I found the link between identifiability and uniqueness of tensor decomposition highly interesting and novel (although I am not sure if it is generally novel). \n\n- In general settings, The paper provides an improvement on the activation degree thresholds required for identifiability, showing a linear dependence on layer width rather than the previously conjectured quadratic one (Corollary 22). This is a clear, measurable advance over very recent work in the field.\n\n- I found paper to be quite well-written and logically structured. The definitions are clear, and the arguments, while technical, are laid out in a way that is followable by a reader. \n\n# Weaknesses \nTwo of the key weaknesses have to do with contextualization and background of this work: \n- A similar argument to the localization theorem is present in \"Reconstructing a neural net from its output\" by Fefferman (1994). Fefferman established an analogous recursive, layer-by-layer proof strategy for the identifiability of deep sigmoidal networks. While the mathematical tools are different and the setup (sigmoid vs polynomial activations), the core conceptual principle of reducing a deep problem to a sequence of shallow ones is not new. The failure to cite and discuss this foundational work that predates this work by 3 decades, seems to be a major oversight.\n- The \"homogeneization procedure\" for handling bias terms is presented as a novel approach. Namely in lines 56-57: \"Moreover, we also address the case of PNNs with biases (which was overlooked in previous theoretical studies).\" In fact, this method is a standard, textbook technique known as using homogeneous coordinates, widely applied across computer science and machine learning books for decades. The novelty lies in proving it preserves identifiability for PNNs, not in inventing the method itself. This distinction is important to make and the current phrasing is somewhat misleading. \n- Limited Scope of Application: The results are specific to PNNs with monomial activations. While this is an important theoretical model, its direct applicability to the broader landscape of modern neural networks (which primarily use ReLU, GeLU, etc.) is limited. Given that earlier works, such as Fefferman's work from 30 years ago on sigmoidal activation, there ought to be more general ways of identifiability that go beyond polynomial networks.", "was_revised": true, "final_justification": "Both based on the original submission and throughout the discussion, I maintain my highly favorable view (5:accept, unchanged from the original rating) of this submission and consider it to be a valuable contribution. \n\nAs stated in my original review, I only raised some points regarding references and contextualizing the work, plus  some questions that were more to clarify the work and some novelty aspects of the work. Several of the discussion points, including those in response to my review, deem to be interesting points worth being integrated into the work, which I presume the authors will do if accepted.", "rebuttal": "We would like to thank the Reviewer for the careful evaluation of our work and for the valuable suggestions for improvements. In the following, we provide responses and clarifications to the weaknesses or questions raised by the Reviewer.\n\nQ1: **Relationship to the argument present in Fefferman (1994)**: We agree with the Reviewer that the property proved by Fefferman is similar to our localization result for polynomial networks. We want nonetheless to emphasize the difference between the objects in consideration. Fefferman's paper indeed focuses on the case of a very specific analytic function (the tanh activation), which is far from being polynomial, and makes use of clever analytic techniques (study of singularities), which only work, though, for activation functions with specific properties (real analytic, finite limit at infinity, meromorphic extension to $\\mathbb C$). Also note that Fefferman's arguments would not work in our situation, since monomial activations do not fulfill his requirements (having finite limit at infinity in one direction of the complex plane). In contrast, our proof is based on purely algebraic arguments, well suited to tackle the monomial case. We would also like to add that an exciting future research direction is to tackle the analytic activation case using general (non-monomial) polynomial activations. This would indeed encompass Fefferman's result, and our results on monomial polynomial activations constitute a crucial first step in this analysis.\n\nWe also would like to emphasize that the very powerful Fefferman's result provides only a sufficient condition for identifiability. Theorem 9 in Fefferman's work [23] excludes a lot of parameters (though a discrete set) and is not very practical. In our case, not only the proofs are constructive, but also they provide if and only if condition for uniqueness, provided the degree of the activation is high enough (no collinear rows in weight matrices and no zero columns).  \n\nWe would like to thank the reviewer for mentioning this important paper. Indeed, although Fefferman's paper was already cited in the introduction of the original paper (reference [23]), there is not enough credit given to it, and we will correct this in the revised version.\n\n[23] C. Fefferman. Reconstructing a neural net from its output. Revista Matemática Iberoamericana, 10(3):507–555, 1994.\n\n\nQ2: **Novelty of the homogenization procedure**: We agree with the Reviewer that homogeneization itself is a fairly standard technique in machine learning and mathematics. We highlight, as the Reviewer correctly notes, that the contribution in our paper lies in leveraging it to demonstrate the uniqueness of the resulting deep PNN architecture by reducing it to a homogeneous PNN. The challenge comes from the fact that neither the weight matrices nor the input signals and hidden activations can be assumed to be distributed according to an absolutely continuous measure, which precludes the use of various standard arguments. This difficulty brought on considerable work to tackle NNs with biases even in the case of 2-layer NNs (see for example [68]). Nonetheless, we appreciate the Reviewer's remark that the current phrasing be potentially misleading in the original paper, and we will therefore revise it in the final manuscript to make this distinction clear to the reader.\n\n[68] P. Awasthi et al. Efficient algorithms for learning depth-2 neural networks with general ReLU activations. Adv. Neur. Inf. Proc. Syst., 34:13485–13496, 2021.\n\n\nQ3: **Limited scope of application**: We note that considering PNNs with monomial activations in fact is a crucial first step in tackling general polynomial activations -- which in turn can approximate all classical activation functions over compact sets. This is a direction we will investigate for a follow up work. For more details, see also the answer to Q5 just below, and also the answer to Q2 by Reviewer Ux2R.\n\n\nQ4: **Reliance on a general configuration of weights**: We highlight that our paper include both uniqueness (fixed architecture) and identifiability (generic uniqueness) results. The  conditions for uniqueness of a given architecture are in fact reasonably weak in general: we only require that (for high enough degrees) the weight matrices (with weaker conditions for the first and last layers) have no collinear (or, in the more pathological case, zero) rows or columns, otherwise the network would have redundant neurons that could be suppressed without changing its response. The generic identifiability conditions require the matrices to be sampled from any distribution that is absolutely continuous with respect to the Lebesgue measure. This requirement is fairly weak, and serves to exclude weights that fall in such pathological cases (e.g., some weight matrices with zero/collinear rows/columns). The conditions of uniqueness at fixed architecture are in fact helpful to diagnose the resulting trained network: if it can accurately represent the training data, and yet its weight matrices do not satisfy simple conditions such as having no collinear columns, then this means that the considered architecture could be potentially reduced to a smaller one. If we properly train a PNN to approximate data representable by an identifiable PNN architecture, then the weights should not converge networks with such \"degenerate\" weight matrices because otherwise the trained network would not be able to represent the data coming from the original identifiable one. These reasons lead us to believe that these conditions are indeed reasonably mild. We will include a more detailed discussion of the implication of these assumptions in the revised version of the paper.\n\nQ5: **Generalizing the results to non-homogeneous activations (GeLU, tanh, etc)**: On the one hand, our proof techniques are considerably different than those of previous works focused on tanh or sigmoidal activations; thus, directly generalizing them for such activation functions appears to be highly nontrivial. On the other hand, considering monomial activation functions is a necessary and crucial step to tackle the PNNs with general polynomial activations. This general polynomial case, which is of significant practical interest, can be potentially tackled with an extension of our proof techniques, where our identifiability results for the case of monomial activations should be an essential part of the analysis. See also the answer to Q2 by Reviewer Ux2R.\n\nQ6: **Extension to Hermite polynomials or other bases**: This is a very interesting question. This is related to the study of the identifiability of PNNs whose activation functions are general polynomials. As mentioned in the previous response, our results on monomial activations serve as a crucial first step to tackle the uniqueness of PNNs with general polynomial activations, which can encompass Hermite polynomial bases. This is a topic we are interested in pursuing in a future work. We will clarify this perspective in the revised version of our paper.\n\nQ7: **Complexity aspects**: Thank you for this insightful question. An important and distinctive aspect of our proofs is that they are constructive, and the parameters of a PNN can be computed from the canonical polyadic decomposition of its associated tensor. While the order of this tensor can be potentially high, under some conditions there exist very efficient algorithms based on linear algebra to compute the factor matrices (see, for example, [Batselier et al., 2016]). While the question of sample complexity is also of great practical interest, our analysis concerns identifiability in the noiseless setting, which is a first stepping stone to study the finite sample case. We find these questions very exciting and plan to tackle algorithmic and sample complexity issues in upcoming work, for instance, following an approach similar to the one used in [Oymak et al., 2021], which leverage the calculation of moment tensors from empirical data.\n\n[Batselier et al., 2016] K. Batselier et al. Symmetric tensor decomposition by an iterative eigendecomposition algorithm. Journal of Computational and Applied Mathematics, 308, 69-82. (2016)\n\n[Oymak et al., 2021] S. Oymak et al. Learning a deep convolutional neural network via tensor decomposition. Information and Inference: A Journal of the IMA, 10(3), 1031-1071 (2021).\n\n\nQ8: **Link between our proof and the one in Fefferman (1994)**: Please see the response to Q1 just above."}, {"review_id": "okPVttb9vC", "reviewer": "Reviewer_gdGi", "summary": "This paper develops a comprehensive theory of when deep polynomial neural networks (PNNs) are identifiable, i.e., when their weights can be recovered uniquely from their input-output map. The key insight is the localization theorem, which shows that a deep PNN is globally identifiable if and only if every consecutive two-layer block is identifiable. Building on this, the authors derive simple, constructive conditions (involving layer widths and activation degrees) guaranteeing identifiability for a wide range of architectures, including pyramidal nets and encoder-decoder nets. Their bounds on activation degrees improve prior quadratic requirements to linear in layer widths, and they settle several conjectures in recent literature.", "questions": "- Could the authors expand on the work’s relevance to practitioners?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The theoretical contribution is strong. The paper establishes a clear and modular criterion for identifiability of deep PNNs and resolves prior open conjectures on identifiability and neurovariety dimensions. The activation threshold bounds for identifiability are shown to be linear rather than quadratic, improving prior results significantly.\n- The proofs are constructive, offering parameter recovery via tensor decomposition algorithms.\n- The results are general enough to cover architectures with and with bias, and of different types such as non-increasing width and encoder-decoder networks.\n\nWeaknesses\n- The paper does not present numerical experiments or simulations to illustrate the tightness of theoretical bounds or demonstrate recovery of network weights in practice. \n- Computing Kruskal ranks and performing CPD uniqueness tests can be computationally intensive for large-scale networks. The paper does not discuss algorithmic costs, heuristics, or approximate checks for use on real-world PNNs.", "was_revised": true, "final_justification": "The author's response has addressed my concerns, especially on relevance to practitioners.", "rebuttal": "We would like to thank the Reviewer for the careful evaluation of our work and for the valuable suggestions for improvements. In the following, we provide responses and clarifications to the weaknesses or questions raised by the Reviewer.\n\nQ1: **Lack of numerical experiments**: We agree that the paper would benefit from numerical experiments. However, our main goal to clearly present the uniqueness/identifiability result and its proof in a digestible form, which we really hope will be very useful for community. We emphasize, however, that the constructive nature of our proofs make it feasible to develop algorithms based on linear algebra to recover the parameters of an (h)PNN. We will include a discussion on numerical algorithms and how they can be derived from the proposed proofs in the revised version of the manuscript.\n\nQ2: **Computationally intensive to compute Kruskal ranks**: Thank you for this important question. We note that we do not actually need to compute Kruskal ranks to verify several architectures of practical interest. For various architectures, for example, those with sufficiently high degrees, we just need to check whether the weight matrices have collinear rows/columns. Moreover, in many cases (for high degrees or numbers of neurons not being too large) the corresponding tensor representation of a PNN can be decomposed using efficient large-scale algorithms based on linear algebra, such as the one in [Batselier et al., 2016]. This is a topic we plan to study in a followup work. We will include a discussion about the development of computationally efficient algorithms in the revised version of the paper.\n\n[Batselier et al., 2016] K. Batselier et al. Symmetric tensor decomposition by an iterative eigendecomposition algorithm. Journal of Computational and Applied Mathematics, 308, 69-82. (2016)\n\nQ3: **Expand on the relevance to practitioners**: We thank the Reviewer for this interesting question. As we already mentioned in the answer to Question 1 of Reviewer Mfr2, interpretability of neural networks is tightly linked to uniqueness of its parameters.\nOur work could be also a guide for the design of pruning strategies, aiming at building identifiable architectures from a non-identifiable given one while preserving its original functional response. This work can be an useful to guide for compression.\n\nFurthermore, as also mentioned in the answer to Question 1 from Reviewer Mfr2, identifiability plays an important role in model comparison, \"stitching\" or averaging. This has been used in several recent works [34], [Ito et al., 2025], [Ainsworth et al., 2022], where one finds a permutation (and possibly rescaling) that best aligns the hidden representations of different pretrained models to perform model comparison, averaging, uncertainty quantification, among other tasks. For this procedure to yield meaningful results, identification is essential.\n\nFinally, identifiability/uniqueness also has direct implications on the geometry of PNNs, since it is directly linked to the dimension of its associated neurovariety as discussed in Appendix B of our paper.\n\nWe will emphasize the role of identification in the revised manuscript, as well as the influence of the equivalence class of parameters.\n\n[34] C. Godfrey et al. On the symmetries of deep-learning models and their internal representations. Advances in Neural Information Processing Systems, 35:11893–11905, 2022\n\n[Ito et al., 2025] A. Ito et al. (2025). Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries. In Forty-second International Conference on Machine Learning.\n\n[Ainsworth et al., 2022] S. K. Ainsworth et al. . Git re-basin: Merging models modulo permutation symmetries. The Eleventh International Conference on Learning Representations. (2022)"}, {"review_id": "9euMwTsDGQ", "reviewer": "Reviewer_f8tx", "summary": "This paper studies the identifiability of the parameters of single-hidden-layer neural networks with polynomial activation functions. The main result shows that under mild conditions on the activation degree and input distribution, a polynomial network with kk hidden units is identifiable from the input-output function it computes. The authors characterize when two parameter settings define the same function and give conditions under which all parameters are uniquely recoverable up to symmetry. Their analysis builds on algebraic tools, including properties of polynomial ideals and ranks of function spaces.", "questions": "The main result is mathematically clean and addresses an important foundational question—identifiability—in a tractable setting. Although the class of networks studied (shallow, polynomial activations) is of limited practical use today, the result itself is of theoretical interest and may inform future work on understanding parameter-function mappings in broader network families. \n\n\n- Include a short discussion of whether and how the results extend to real analytic or piecewise-linear activations.\n- Consider a brief experiment or example showing what identifiability looks like in practice—e.g., a visualization of function-equivalent but parameter-distinct networks failing under certain assumptions.\n- Clarify the implications for learning: does identifiability imply anything about convergence or optimization in practice for this class?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Among the strengths are a clear, elegant identifiability result: The core result on the identifiability of shallow polynomial networks is cleanly stated and technically sound. It adds clarity to a fundamental question—when is the function computed by a neural network uniquely determined by its parameters?—in a setting amenable to analysis. Furthermore, the authors use appropriate tools from algebraic geometry and multilinear algebra to derive their results. The arguments are accessible while still mathematically rigorous. Finally, the paper is clearly written, especially for a mathematically inclined audience. The structure, notation, and logical flow make the main result easy to follow.\n---\nAmong the weaknesses, I think the relevance is limited. The results concern networks with monomial activations, which are never used in modern deep learning. The authors do not discuss how the identifiability properties extend—or fail to extend—to more realistic activation functions (e.g., ReLU, tanh). As such, the results may have limited direct impact on practical neural network design or training. Furthermore while this is a theory paper, even a small numerical experiment (e.g., verifying identifiability or its failure in small examples) would make for better readability.  The identifiability of polynomial systems is closely related to classical results like Bézout’s Theorem, which concerns the number of solutions to systems of polynomial equations. Finally, the paper does not explore whether the results might extend to deeper networks, approximate identifiability (in the presence of noise), or probabilistic identifiability. These are directions of  interest in both learning theory and applied statistics.", "was_revised": true, "final_justification": "The paper is good and technically sound, though the results have limited applicability and it is difficult to say howto extend them. From my personal point of view, the main weakness is not a technical one, it is about the research topic. Let us assume that the goal of the community  is to understand fundamental principles of learning and time is short to achieve this goal. Then I feel that identifiability of polynomial networks is not a high priority topic. My grade follow from this belief not by problems with the mathematics of the paper.", "rebuttal": "We would like to thank the Reviewer for the careful evaluation of our work and for the valuable suggestions for improvements. In the following, we provide responses and clarifications to the weaknesses or questions raised by the Reviewer.\n\n**Response to weaknesses**: First we would like to emphasize that the proposed techniques constitute a crucial first step in the study of the identifiability of more general architectures, such as general polynomial networks (which can provide good approximations to activations commonly used by practitioners), see the answer to Q2 by Reviewer Ux2R for more details. Moreover, our results also address networks of arbitrary depth. Furthermore, we also address the question of uniqueness of a specific set of NN parameters (Theorem 11), which is a much stronger property than identifiability (Corollary 12).\n\nWe also agree that approximate or probabilistic identifiability are of great interest in ML theory and applied statistics. However, a first stepping stone to these analysis requires understanding the identifiability in the noiseless case, which is the aim of our paper. We do provide an a clear and easy-to-digest answer to this question and advance the understanding of their neurovarieties in a wide range of settings. Thus, while approximate or probabilistic identifiability are are exciting and challenging topics, these will be better suited to be addressed in a dedicated future work.\n\nQ1: **Role of Bézout's theorem**: We thank the Reviewer for this very interesting remark.\nBézout's Theorem is indeed is very classical and fundamental result in algebraic geometry. Briefly, Bézout's theorem asserts that a **general** system of $n$ polynomial equations in $n$ variables has a finite number of solutions. However, for several reasons, it is not applicable in our situation.   \n- The uniqueness property can indeed be written as a system of polynomial equations. Nevertheless to fulfill the requirements of Bézout's theorem, we would need to have more independent equations than the number of variables, that is, a \"small\" number of neurons in the hidden layers compared to the others parameters (degree, extremal widths). This is generally not the case in applications.  \n- Bézout's theorem holds for **general** polynomials, i.e. chosen out of a set of zero Lebesgue measure. However, this set is not explicit: in other words, Bézout does not provide an explicit manner to check whether our uniqueness property (or even identifiability) holds for the equations arising from a given PNN.   \n- It is also important to notice that for **monomial** activations, as we explained in the paper, we can only have uniqueness/finiteness **up to the trivial ambiguities**, and there always exist infinitely many equivalent decompositions. Results like Bézout's theorem do not apply directly in this context, and we would need more advanced algebro-geometric techniques to tackle this very important question.\n\nQ2: **Relevance and extension to more common (real analytic or piecewise-linear activations) activations**: On the one hand, our proof techniques are considerably different than those of previous works focused on special cases of analytic functions such as tanh or sigmoidal activations; thus, directly generalizing our proofs to general real analytic functions appears to be highly nontrivial. On the other hand, considering monomial activation functions is a necessary and crucial step to tackle the case of PNNs with general polynomial activations. In a future work, we plan to tackle this general polynomial case, which is of significant practical interest, by extending our proof techniques. We will include a discussion about possible research directions to extend our results to NNs with more general polynomial activation functions in the revised version of the paper.\n\nQ3: **Experiment or example showing what identifiability looks like in practice**: Thank you for the valuable suggestion. In order to better illustrate identifiable and non-identifiable networks in practice, we consider will include additional examples in the revised paper. In particular, we will include an example to compare PNNs that are 1) unique and thus identifiable, 2) identifiable but non-unique (i.e., with pathological weight matrices, such as collinear rows), and 3) non-identifiable (and thus, also not unique).\n\nQ4: **Implications of identifiability on learning or optimization**: We thank the Reviewer for this interesting question. In fact, identifiability has many important links to learning and optimization of PNNs. In particular, identifiability provides important information on the dimension of their associated neurovariety and neuromanifold, which, in turn, has deep implications on their expressivity, sample complexity, and on the geometry of their learning landscapes, as noted in previous works [7] and [9]. Moreover, a key result from singular learning theory (see [Watanabe, 2009] for background) is that non-identifiable networks are precisely singular points of the neurovariety, and in fact, learning algorithms are biased towards such solutions [Shahverdi et al., 2025]. Thus, it is very important to characterize which networks are non-identifiable [Shahverdi et al., 2025]. We will comment on the implications of identifiability to learning in the revised version of the paper.\n\n[Shahverdi et al., 2025] Shahverdi, V., Marchetti, G. L., and Kohn, K. (2025). Learning on a Razor's Edge: the Singularity Bias of Polynomial Neural Networks. arXiv preprint arXiv:2505.11846.\n\n[Watanabe, 2009] S. Watanabe, \"Algebraic Geometry and Statistical Learning Theory\", Cambridge University Press, 2009.\n\n[7] J. Kileel et al. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.\n\n[9] N. W Henry et al. Geometry of lightning self-attention: Identifiability and dimension. ICLR 2025, 2025. arXiv preprint arXiv:2408.17221."}, {"review_id": "e6mvuYf9j1", "reviewer": "Reviewer_Ux2R", "summary": "This manuscript develops an algebraic-geometric theory of global identifiability for MLPs with monomial activation functions, specifically $x^r$. The arguments are based on the relationship between partially symmetric canonical polyadic tensor decomposition and monomial MLPs. The main contributions of the paper are:\n - An L-layer monomial MLP is globally identifiable if and only if every consecutive two-layer block is identifiable (Theorem 11).\n - Establishing a lower bound for the activation degree at which monomial MLPs become identifiable (Propositions 13 and 15). This bound is linear in terms of layer widths, improving the quadratic bound provided in [1].\n -  Providing separate analyses for pyramidal architectures and those featuring bottleneck layers.\n- Extending the analysis to monomial MLPs with bias through a homogenization technique, which to my knowledge was not considered in previous literatures.\n\n[1] Finkel et al., Activation thresholds and expressiveness of polynomial neural networks, 2024.", "questions": "-  In lines 37-38, the authors claim that results in [1, 2, 3] focus on local identifiability rather than global identifiability. However, these references appear to describe the generic fiber of the parameterization, such as the proof of Theorem 18 in [1] and the birationality result in [2]. Could the authors clarify why this is considered local rather than global identifiability?\n- In Section 4.3, regarding shallow networks, the locus of functions admitting unique decomposition is described via the Kruskal rank condition. Does this locus correspond exactly to functions in monomial MLPs without any neurons being inactive? Is this statement true for deep case based on theorem 11?\n\n [2] Shahverdi et al., On the geometry and optimization of polynomial convolutional networks, 2024.\n\n [3] Henry et al., Geometry of lightning self-attention: Identifiability and dimension, 2024.", "limitations": "As mentioned above, the limitations regarding the choice of monomial activation are not discussed. I recommend the authors to include it.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The manuscript including the appendix is overall well-written, and the method used to reduce the analysis of deep monomial MLPs to shallow ones is very elegant. \n\n- Although the results have limited implications for real-world applications, the approach has the potential to be extended to the analysis of MLPs with arbitrary polynomial activations. This is particularly important in real-world applications, as any continuous activation function can be approximated by polynomials over a compact set.\n\n- Another notable aspect is the inclusion of bias in the analysis. Prior works on monomial MLPs such as [1] ignored bias, which posed a limitation for applying algebraic geometry techniques to general neural network analysis.\n\n # Weaknesses\n- The authors refer to \"polynomial\" activation function in the abstract and introduction without specifying that they mean \"monomial,\" which could lead to confusion. I suggest for clarity, the term monomial being mentioned as this is the primary setup for this paper.\n\n- From an empirical perspective, the paper’s impact appears limited. This point is not sufficiently discussed, and the potential for generalizing the results to arbitrary polynomial activations—an extension that could substantially enhance the paper’s relevance—is not addressed.\n\n- Lastly, there is a minor typo on line 271: \"Example 51\" should be corrected to \"Example 5.\"\n\nOverall, I highly recommend this work for its novel methodology and its potential impact in the context of the algebraic analysis of deep neural netwoks.", "was_revised": true, "final_justification": "The paper tackles an important and long-standing problem, improving and extending existing results, and providing new techniques. It is mathematically rigorous and original. Therefore, I believe that the work deserves acceptance.", "rebuttal": "We would like to thank the Reviewer for the careful evaluation of our work and for the valuable suggestions for improvements. In the following, we provide responses and clarifications to the weaknesses or questions raised by the Reviewer.\n\nQ1: **Tackling monomial vs. general polynomial activations**: This is a good point and we totally agree with the Reviewer, however, we also emphasize that, in all papers starting from the paper by Kileel, Trager, and Bruna (reference [7]), the term \"polynomial neural networks\" refers to neural networks with monomial activations, and it is a standard terminology used in the literature. Nevertheless, we will clarify that the results focus on monomial activations instead of general polynomial activations when describing the setup of the revised paper, which should better contextualize our contribution, and also in the discussion of the limitations of the paper. We would also like to emphasize that the study of the monomial case is a fundamental first step in tackling more general polynomial activations, as further clarified in the answer to the following question.\n\n[7] J. Kileel et al. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.\n\n\nQ2: **Empirical impact and generalization to arbitrary polynomial activations**: We thank the Reviewer for the suggestion. We agree that an extension to consider general polynomial activations instead of monomial ones would greatly improve the practical impact of our results. We do believe such an extension is possible. In fact, the monomial case is the key to address the general polynomial case. For instance, the very recent paper [Shahverdi et al., 2025] (published May 17th, 2025, after the submission of our paper) considers more general polynomial activations and they first rely on results for the monomial case. This is an exciting subject which we plan to investigate in a followup work.\n\n[Shahverdi et al., 2025] V. Shahverdi et al. Learning on a Razor's Edge: the Singularity Bias of Polynomial Neural Networks. arXiv preprint arXiv:2505.11846. (2025).\n\n\nQ3: **Typo in line 271**: Thank you very much for pointing this out, we will fix it in the revised paper.\n\n\nQ4: **Local identifiability rather than global identifiability in [1, 2, 3]**: The reviewer is right, this was an incorrect wording from our side. Indeed, the proof of Theorem 18 in [11] contains a result on global identifiability (for a high degree though, so weaker than our results, as we mentioned in the introduction), but the statement of the theorem is about local identifiability (more precisely, about dimension of the neurovariety, which implies that the parameterization map is finite-to-one, subject to scaling ambiguities). References [8] and [9] indeed also contain results  about global identifiability, but for other/particular architectures (convolutional and attention-like layers) and thus are not applicable to general PNNs. What we meant is that identifiability is not the main focus of these papers, but we believe it is a central concept that  is key to understanding the geometry of PNNs. We do apologize for this incorrect wording and we will correct it in the revised paper.\n\n\nQ5: **Kruskal rank condition and MLPs without inactive neurons**: Thank you for the insightful question. The Kruskal rank conditions used in our results requires that, for a 2-layer network with weight matrices $W_1$ and $W_2$ for the first and second layer, respectively, $W_1$ has no collinear rows, and $W_2$ has no zero columns. This in turn corresponds to networks that are **irreducible** in the sense that the activations in the hidden layer are linearly independent, which is slightly stronger than being \"active\"/nonzero. \n\nIn fact, we have that for high enough degrees, uniqueness is achieved **if and only if** the weight matrices satisfy these Kruskal rank constraints (these necessary conditions become sufficient). This is also true for the case of deep (L-layers) networks due to the localization theorem. We will improve the discussion of the implication of the Kruskal rank conditions in the revised version of the paper."}]

Author rebuttal:

Meta-review: The paper presents a comprehensive theory of identifiability of deep neural networks with (high-degree) monomial activation functions. The central result is a localization theorem (Theorem 11), which establishes that identifiability of this network is equivalent to the identifiability of each consecutive two-layer subnetwork. It establishes constructive conditions based on activation degrees and layer widths, improving previous bounds. The analysis leverages tensor decomposition techniques and extends to networks with bias terms, resolving an open conjecture on neurovariety dimensions. The reviewers do raise concerns over the applicability to modern networks (e.g. with relu, gelu activations) is questionable. Nevertheless, the reviewers still find the result important and hope that follow-up work can be conducted to connect to more applicable networks. I agree with the limited applicability and hope to see follow-up work connectin them with more general activation functions or with the recent results on polynomial representations, i.e. networks without activation functions such as Multilinear Operator Networks or Deep tree tensor networks for image recognition.
Decision: Accept (oral)