Title: Identifiability of Deep Polynomial Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: polynomial neural networks

identifiability

tensor decompositions

uniqueness
Abstract: Polynomial Neural Networks (PNNs) possess a rich algebraic and geometric structure. However, their identifiability-a key property for ensuring interpretability-remains poorly understood. In this work, we present a comprehensive analysis of the identifiability of deep PNNs, including architectures with and without bias terms. Our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability. As special cases, we show that architectures with non-increasing layer widths are generically identifiable under mild conditions, while encoder-decoder networks are identifiable when the decoder widths do not grow too rapidly compared to the activation degrees. Our proofs are constructive and center on a connection between deep PNNs and low-rank tensor decompositions, and Kruskal-type uniqueness theorems. We also settle an open conjecture on the dimension of PNN's neurovarieties, and provide new bounds on the activation degrees required for it to reach the expected dimension.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript develops an algebraic-geometric theory of global identifiability for MLPs with monomial activation functions, specifically $x^r$. The arguments are based on the relationship between partially symmetric canonical polyadic tensor decomposition and monomial MLPs. The main contributions of the paper are:\n - An L-layer monomial MLP is globally identifiable if and only if every consecutive two-layer block is identifiable (Theorem 11).\n - Establishing a lower bound for the activation degree at which monomial MLPs become identifiable (Propositions 13 and 15). This bound is linear in terms of layer widths, improving the quadratic bound provided in [1].\n -  Providing separate analyses for pyramidal architectures and those featuring bottleneck layers.\n- Extending the analysis to monomial MLPs with bias through a homogenization technique, which to my knowledge was not considered in previous literatures.\n\n[1] Finkel et al., Activation thresholds and expressiveness of polynomial neural networks, 2024.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The manuscript including the appendix is overall well-written, and the method used to reduce the analysis of deep monomial MLPs to shallow ones is very elegant. \n\n- Although the results have limited implications for real-world applications, the approach has the potential to be extended to the analysis of MLPs with arbitrary polynomial activations. This is particularly important in real-world applications, as any continuous activation function can be approximated by polynomials over a compact set.\n\n- Another notable aspect is the inclusion of bias in the analysis. Prior works on monomial MLPs such as [1] ignored bias, which posed a limitation for applying algebraic geometry techniques to general neural network analysis.\n\n # Weaknesses\n- The authors refer to \"polynomial\" activation function in the abstract and introduction without specifying that they mean \"monomial,\" which could lead to confusion. I suggest for clarity, the term monomial being mentioned as this is the primary setup for this paper.\n\n- From an empirical perspective, the paper’s impact appears limited. This point is not sufficiently discussed, and the potential for generalizing the results to arbitrary polynomial activations—an extension that could substantially enhance the paper’s relevance—is not addressed.\n\n- Lastly, there is a minor typo on line 271: \"Example 51\" should be corrected to \"Example 5.\"\n\nOverall, I highly recommend this work for its novel methodology and its potential impact in the context of the algebraic analysis of deep neural netwoks.\n\nQUESTIONS:\n-  In lines 37-38, the authors claim that results in [1, 2, 3] focus on local identifiability rather than global identifiability. However, these references appear to describe the generic fiber of the parameterization, such as the proof of Theorem 18 in [1] and the birationality result in [2]. Could the authors clarify why this is considered local rather than global identifiability?\n- In Section 4.3, regarding shallow networks, the locus of functions admitting unique decomposition is described via the Kruskal rank condition. Does this locus correspond exactly to functions in monomial MLPs without any neurons being inactive? Is this statement true for deep case based on theorem 11?\n\n [2] Shahverdi et al., On the geometry and optimization of polynomial convolutional networks, 2024.\n\n [3] Henry et al., Geometry of lightning self-attention: Identifiability and dimension, 2024.\n\nLIMITATIONS:\nAs mentioned above, the limitations regarding the choice of monomial activation are not discussed. I recommend the authors to include it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the identifiability of the parameters of single-hidden-layer neural networks with polynomial activation functions. The main result shows that under mild conditions on the activation degree and input distribution, a polynomial network with kk hidden units is identifiable from the input-output function it computes. The authors characterize when two parameter settings define the same function and give conditions under which all parameters are uniquely recoverable up to symmetry. Their analysis builds on algebraic tools, including properties of polynomial ideals and ranks of function spaces.\n\nSTRENGTHS AND WEAKNESSES:\nAmong the strengths are a clear, elegant identifiability result: The core result on the identifiability of shallow polynomial networks is cleanly stated and technically sound. It adds clarity to a fundamental question—when is the function computed by a neural network uniquely determined by its parameters?—in a setting amenable to analysis. Furthermore, the authors use appropriate tools from algebraic geometry and multilinear algebra to derive their results. The arguments are accessible while still mathematically rigorous. Finally, the paper is clearly written, especially for a mathematically inclined audience. The structure, notation, and logical flow make the main result easy to follow.\n---\nAmong the weaknesses, I think the relevance is limited. The results concern networks with monomial activations, which are never used in modern deep learning. The authors do not discuss how the identifiability properties extend—or fail to extend—to more realistic activation functions (e.g., ReLU, tanh). As such, the results may have limited direct impact on practical neural network design or training. Furthermore while this is a theory paper, even a small numerical experiment (e.g., verifying identifiability or its failure in small examples) would make for better readability.  The identifiability of polynomial systems is closely related to classical results like Bézout’s Theorem, which concerns the number of solutions to systems of polynomial equations. Finally, the paper does not explore whether the results might extend to deeper networks, approximate identifiability (in the presence of noise), or probabilistic identifiability. These are directions of  interest in both learning theory and applied statistics.\n\nQUESTIONS:\nThe main result is mathematically clean and addresses an important foundational question—identifiability—in a tractable setting. Although the class of networks studied (shallow, polynomial activations) is of limited practical use today, the result itself is of theoretical interest and may inform future work on understanding parameter-function mappings in broader network families. \n\n\n- Include a short discussion of whether and how the results extend to real analytic or piecewise-linear activations.\n- Consider a brief experiment or example showing what identifiability looks like in practice—e.g., a visualization of function-equivalent but parameter-distinct networks failing under certain assumptions.\n- Clarify the implications for learning: does identifiability imply anything about convergence or optimization in practice for this class?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a comprehensive theory of when deep polynomial neural networks (PNNs) are identifiable, i.e., when their weights can be recovered uniquely from their input-output map. The key insight is the localization theorem, which shows that a deep PNN is globally identifiable if and only if every consecutive two-layer block is identifiable. Building on this, the authors derive simple, constructive conditions (involving layer widths and activation degrees) guaranteeing identifiability for a wide range of architectures, including pyramidal nets and encoder-decoder nets. Their bounds on activation degrees improve prior quadratic requirements to linear in layer widths, and they settle several conjectures in recent literature.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The theoretical contribution is strong. The paper establishes a clear and modular criterion for identifiability of deep PNNs and resolves prior open conjectures on identifiability and neurovariety dimensions. The activation threshold bounds for identifiability are shown to be linear rather than quadratic, improving prior results significantly.\n- The proofs are constructive, offering parameter recovery via tensor decomposition algorithms.\n- The results are general enough to cover architectures with and with bias, and of different types such as non-increasing width and encoder-decoder networks.\n\nWeaknesses\n- The paper does not present numerical experiments or simulations to illustrate the tightness of theoretical bounds or demonstrate recovery of network weights in practice. \n- Computing Kruskal ranks and performing CPD uniqueness tests can be computationally intensive for large-scale networks. The paper does not discuss algorithmic costs, heuristics, or approximate checks for use on real-world PNNs.\n\nQUESTIONS:\n- Could the authors expand on the work’s relevance to practitioners?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the identifiability of deep Polynomial Neural Networks (PNNs). Identifiability of a network means the ability of identify the network parameters from its functional input-outputs. \nThe central is a two-fold \"localization theorem\" which proves that the global identifiability of a deep PNN is equivalent to the identifiability of all its constituent 2-layer sub-networks. The proof is relatively elementary and easy to follow, nevertheless elegant and clean. Leveraging this result, the authors connect the of identifiability for a two layer network to the uniqueness of partially symmetric tensor decompositions. This framework allows them to derive new, explicit conditions under which specific deep PNN architectures, such as pyramidal and bottleneck networks, are generically identifiable. The paper also provides improved, linear bounds on the activation degree thresholds required for identifiability and extends its analysis to PNNs with bias terms via a homogenization technique.\n\nSTRENGTHS AND WEAKNESSES:\n# strengths\n\n- The paper provides a rigorous and comprehensive analysis of deep PNN identifiability. The specific conditions derived for pyramidal and bottleneck networks (Corollaries 18 and 19) are novel and valuable to researchers in this specific domain.\n\n- The work goes beyond mere existence proofs. It provides explicit, checkable inequalities (e.g., Proposition 13) that link architecture choices (layer widths, activation degrees) to the identifiability property. The connection to tensor decomposition is well-executed and powerful.  I found the link between identifiability and uniqueness of tensor decomposition highly interesting and novel (although I am not sure if it is generally novel). \n\n- In general settings, The paper provides an improvement on the activation degree thresholds required for identifiability, showing a linear dependence on layer width rather than the previously conjectured quadratic one (Corollary 22). This is a clear, measurable advance over very recent work in the field.\n\n- I found paper to be quite well-written and logically structured. The definitions are clear, and the arguments, while technical, are laid out in a way that is followable by a reader. \n\n# Weaknesses \nTwo of the key weaknesses have to do with contextualization and background of this work: \n- A similar argument to the localization theorem is present in \"Reconstructing a neural net from its output\" by Fefferman (1994). Fefferman established an analogous recursive, layer-by-layer proof strategy for the identifiability of deep sigmoidal networks. While the mathematical tools are different and the setup (sigmoid vs polynomial activations), the core conceptual principle of reducing a deep problem to a sequence of shallow ones is not new. The failure to cite and discuss this foundational work that predates this work by 3 decades, seems to be a major oversight.\n- The \"homogeneization procedure\" for handling bias terms is presented as a novel approach. Namely in lines 56-57: \"Moreover, we also address the case of PNNs with biases (which was overlooked in previous theoretical studies).\" In fact, this method is a standard, textbook technique known as using homogeneous coordinates, widely applied across computer science and machine learning books for decades. The novelty lies in proving it preserves identifiability for PNNs, not in inventing the method itself. This distinction is important to make and the current phrasing is somewhat misleading. \n- Limited Scope of Application: The results are specific to PNNs with monomial activations. While this is an important theoretical model, its direct applicability to the broader landscape of modern neural networks (which primarily use ReLU, GeLU, etc.) is limited. Given that earlier works, such as Fefferman's work from 30 years ago on sigmoidal activation, there ought to be more general ways of identifiability that go beyond polynomial networks.\n\nQUESTIONS:\n- From what I understand, all the proofs of inedibility rely on this \"general\" configuration weights, that are everywhere except on some zero measure subset. While this is very true for initialization, I wonder if network that has been trained will remain in this general configuration or not? \n- Do authors have any ideas to generalize their results to new activations that are not homogenous, namely GeLU, tanh, etc? Or the proof method here completely breaks down? \n- While the algebra shown here is quite interesting, the polynomial basis is really not a good abstraction for real activations. But something like Hermite polynomials are in fact a natural basis there. Is it conceivable to extend these proof methods to some other type of basis rather than polynomials? \n- While the issue of identifiability is very interesting, another topic of high interesting to ML audience would be the complexity aspect. meaning, even if we know a network is identifiable, can we find it (reconstruct it) in a reasonable time? Can we turn one of these i Another aspect would be, the  tightly related topic is also the sample complexity? As in, how many samples do we need to identify a network?  Can authors comment on this? \n- Could the authors please comment on the relationship between their proof strategy and Fefferman's? Is there any link at all or not?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers identifiability of the parameters of deep polynomial neural networks. Identification is an important, and often less studied, property of NNs as it is directly linked to the potential to interpret the NNs weights. The authors show that for an equivalence class up to permutations, deep PNN are identified if and only if each 2-layer subnetwork is also identified, extending existing results for 2-layer networks. The paper's main result follows from (1) a rank condition to ensure identifiability in the 2-layer case and (2) an induction argument to extend it to the L layer case.  The authors show how their results can be applied to a variety of structures and given conditions to check whether a PNN structure can be identified.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The topic of identification of general L-layer deep PNN is important as it is a less studied topic, but crucial to understand interpretability of NNs.\n\n2. The main result that 2-layer uniqueness can be extended to L-layer uniqueness appears correct, slightly surprising at first sight, and potentially a great insight. \n\n3. The paper's review of how the identification results may be used to check different architectures may be potentially useful for researchers in choosing how to construct PNNs.\n\nWeaknesses:\n1. A bit more discussion on how interpretability relates to identification, given that the result is up to permutation matrices would help make the paper more relevant. How large is the equivalence class? Are there other equivalence classes for which a similar result could have been proven? \n2. The paper presents many results and it is often slightly unclear (beyond Theorem 11) what is their contribution versus building on existing work. Furthermore, what is the relevance of the polynomial activations for the arguments? It seems that the results and proof methods would work generically for different activations, and if such, do similar results exist for NNs with other activations. More discussion on this would be appreciated.\n\nQUESTIONS:\nSome smaller questions beyond the two questions in the weakness section:\n\n1. In definition 8 why do we additionally introduce the requirement that the parameters cannot be chosen on sets of measure zero? Is this also a requirement for definition 6 and is the set of parameters in the equivalence class defined by Lemma 4 measure zero? \n\n2. Is Proposition 13 new? I would be helpful to relate it to previous work (e.g. [7]) and to the recurrent example and how it relates to the full rank condition of the weight matrices. \n\n\n* Some typos: for eg. Lemma 32 *of with* architecture\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}