Title: The Computational Complexity of Counting Linear Regions in ReLU Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: ReLU Neural Networks

Linear Regions

Counting Complexity
Abstract: An established measure of the expressive power of a given ReLU neural network is the number of linear regions into which it partitions the input space. There exist many different, non-equivalent definitions of what a linear region actually is. We systematically assess which papers use which definitions and discuss how they relate to each other. We then analyze the computational complexity of counting the number of such regions for the various definitions. Generally, this turns out to be an intractable problem. We prove NP- and \#P-hardness results already for networks with one hidden layer and strong hardness of approximation results for two or more hidden layers. Finally, on the algorithmic side, we demonstrate that counting linear regions can at least be achieved in polynomial space for some common definitions.


Peer reviews:
comments: SUMMARY:
This paper studies the computational complexity of counting linear regions of ReLU neural networks. It's shown that while such problem is computationally feasible for one-hidden-layer networks, it becomes inefficient for networks with more than one hidden layers.

STRENGTHS AND WEAKNESSES:
**Main strength:** the results are novel and technically interesting. In particular, the authors show a sharp phase transition between one and more layers.

**Main weakness:** my main concern is on the relevance and significance to the machine learning community. (1) counting the number of linear regions is not a popular (and not a good one in my opinion, e.g. the highly self-repeating "saw-tooth" function) metric for the expressivity of neural networks. (2) deciding the computational complexity of such problem, is even less relevant, since I find no reason to consider it at all in practice.

**Other comments:** the improvement over the result of Wang 2022 seems incremental. The proof uses a similar framework, and the new regime covered isn't significant, since most practical neural networks have depth at least $O(\log n)$, already covered by Wang 2022. I believe it would be beneficial to discuss more on your technical novelties.

In section 3, you claim to unify the six definitions, while I found no statements supporting this claim (unless you can somehow show $R_1=O(R_6)$, or rewrite all six with a single notion). This makes this section look somewhat deviating.

QUESTIONS:
I believe this paper would be a better fit for TCS venues.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 3
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the computational complexity of counting the number of linear regions induced by ReLU neural networks. The authors identify and formalize six non-equivalent definitions of linear regions used in the literature. They prove that exact counting is #P-hard even for networks with one hidden layer, and that approximate counting is NP-hard for networks with two or more hidden layers. They also show that the decision problem (e.g., whether the number of regions exceeds a threshold) is NP-complete for networks with at least two hidden layers. Finally, they provide a positive result: exact counting is in FPSPACE for some of the definitions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper offers a clear and systematic organization of existing literature by identifying and formalizing six non-equivalent definitions of linear regions, resolving ambiguities that have led to inconsistencies in prior work.\n2. It establishes new and strong computational hardness results, including #P-hardness for shallow networks and NP-/inapproximability results for deep networks under various definitions.\n\nWeaknesses:\n1. The results are based entirely on worst-case complexity analysis, which may not fully reflect the structure of practically trained networks. For instance, gradient-based training often induces fewer regions, potentially making counting easier in practice.\n2. The parameterized complexity of the problem is not explored; fixed-parameter tractability in terms of the number of neurons, depth, or region count remains an open direction.\n\nQUESTIONS:\n1. Do the hardness results still hold in the absence of bias terms? It appears that the SAT-based reductions used in the paper rely on biases to construct bounded and localized nonzero regions. If the ReLU networks are restricted to be bias-free, do the same hardness results—particularly the NP-hardness and inapproximability for deep networks—still apply? Clarifying this would help understand whether the complexity arises inherently from the activation patterns or from representational power added by biases.\n2. Have the authors considered the parameterized complexity of counting linear regions? While the paper establishes strong worst-case hardness results (e.g., #P-hardness for 1-layer networks and ETH-based lower bounds in input dimension), it would be interesting to analyze whether the problem admits fixed-parameter tractable algorithms with respect to natural parameters such as the total number of neurons s(N), the network depth d, or the number of regions R.\n\nLIMITATIONS:\n1. The paper exclusively analyzes the worst-case computational complexity of counting linear regions. While this is important from a theoretical standpoint, it may not fully capture the behavior of ReLU networks encountered in practice—particularly those trained using gradient-based methods, which are often observed to induce relatively few linear regions.\n\n2. Although the authors show that region counting is in FPSPACE for some definitions, the algorithms presented are mainly of theoretical interest and not designed for practical implementation. There is little discussion of heuristics, approximations, or tractable special cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers fully-connected ReLU neural networks. It is well-known that the function associated to such a network is piece-wise linear. The paper studies the number of pieces (called linear regions) in the input space, where the function is linear. The paper provides 6 precise mathematical definitions of linear regions from the literature. Then, for these definitions, the paper provides results of a Theoretical-Computer-Science (TCS) nature on the computational complexity of counting the linear regions. There are positive and negative results.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\nThe paper appears to be mathematically precise and rigorous. I agree with the authors that being mathematically precise about the definition and manipulation of a linear region is important, as without clarity, mistakes are quickly made, as the authors point out in the literature review. The results obtained by the authors appear to be wide, as they cover 6 existing definitions of linear regions, as well as several notions of complexity from TCS. I think studying theoretical properties of ReLU networks is important since they are fundamental in deep learning. \n\nWEAKNESSES The paper is a bit dry, probably because there are many results and because of the page limit. There does not seem to be machine learning intuition as to why different definitions lead to different computational complexities. The reading is not user-friendly to readers that are not experts in complexity theory, in my opinion.\n\nQUESTIONS:\nCan you exploit the extra page, if there is a camera ready step for the paper, to add \n- intuition about the Machine Learning impact of the results, especially on the differences between definitions\n- pedagogical explanations on the complexity theory concepts, for a general audience \n\nWhy is it written \"definitions X and Y\" instead of \"Definitions X and Y\" in the paper?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically investigates the computational complexity of counting linear regions in ReLU neural networks, a key measure of their expressive power. The authors first identify six non-equivalent definitions of linear regions used in literature, clarifying their relationships through a unified taxonomy. For shallow networks with only one hidden layer, they show that while detecting multiple regions is polynomial-time solvable, exact counting is #P-hard across all definitions. For deep networks, they significantly strengthen prior work by proving NP-hardness for any constant number of layers at least 2 and establishing strong inapproximability results. Notably, they demonstrate that even deciding whether a deep network has larger than 1 region is NP-complete. On the positive side, they prove polynomial-space upper bounds for three definitions. The work highlights fundamental limitations in analyzing neural networks through linear regions while providing rigorous complexity classifications. Key technical innovations include novel reductions from SAT/#SAT and careful handling of definitional subtleties that had caused inconsistencies in prior literature.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality\n\n1.The paper is technically rigorous, with well-constructed proofs and reductions (e.g., #P-hardness for shallow networks, NP-hardness for deep networks).\n\n2.The authors carefully address inconsistencies in prior definitions of linear regions (Section 3, Table 1), which strengthens the theoretical foundation.\n\n3.The upper-bound result (Theorem 6.1, polynomial-space counting) is a meaningful contribution, showing that some variants of the problem are not as intractable as others.\n\nClarity\n\n1.The paper is well-structured, with clear distinctions between different definitions of linear regions and their implications.\n\n2.The writing is precise, though some proofs in the appendix are dense (as expected for a theoretical paper).\n\nSignificance\n\n1.The results have broad implications for understanding the expressive power and complexity of neural networks.\n\n2.The hardness results (Theorems 4.2, 5.2, 5.7-5.8) suggest fundamental limitations in analyzing ReLU networks via linear regions.\n\nOriginality\n\n1.The systematic comparison of six definitions of linear regions is novel and clarifies ambiguities in prior work.\n\nWeaknesses:\n\nQuality\n\n1.Some hardness results (e.g., Theorem 5.8) rely on somewhat artificial constructions (summing multiple copies of a network), which may limit practical implications.\n\n2.The paper does not explore whether certain restricted architectures (e.g., networks trained via gradient descent) might admit efficient counting algorithms, despite mentioning this as future work.\n\nClarity\n\n1.The distinction between Definitions 4 (open connected regions) and 5 (closed connected regions) could be explained more intuitively, as their differences are subtle.\n\n2.The proof sketches are helpful, but some steps (e.g., in Lemma 5.6) remain quite technical and may be hard to follow without careful reading.\n\nSignificance\n\n1.While the theoretical contributions are strong, the practical implications are less clear. The hardness results suggest that exact counting is infeasible, but do not explore whether heuristic or approximate methods (beyond worst-case analysis) could be useful in practice.\n\n2.The focus is purely on ReLU networks; extensions to other activation functions (e.g., sigmoid, GeLU) are not discussed.\n\nOriginality\n\n1.The paper builds heavily on prior work (e.g., Wang (2022), Hanin & Rolnick (2019)), and while it refines and extends these results, the core idea of studying the complexity of linear regions is not entirely new.\n\n2.The upper-bound result (Theorem 6.1) is somewhat incremental, as it mainly shows that some variants of the problem are in PSPACE rather than EXPTIME.\n\nQUESTIONS:\n1.Could the authors discuss whether their results imply limitations for empirical methods (e.g., sampling-based counting)? A brief discussion of practical workarounds (e.g., restricted network classes) would strengthen impact.\n\n2.The analysis is limited to ReLU networks. Do similar hardness results hold for smooth activations (e.g., sigmoid, GeLU)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies the problem of counting linear regions expressed by ReLU networks, from the point of view of computational complexity. The paper provides worst-case complexity results for the decision problem of determining whether a ReLU net has $K>1$ regions, and the function problem of counting linear regions.\n\nThe authors identify non-equivalent notions of affine and activation regions in the literature and relate them. Then, they provide worst-case reductions for the decision and function problems considered, for shallow and deep MLPs.\n\nThe main results can be summarized as follows:\nFor shallow networks, the authors show that the decision problem can be solved in polynomial time, while the counting problem cannot.\nFor deeper networks, time-complexity bounds are provided establishing hardness of linear region counting. Importantly, the authors present hardness results even for approximate region counting and discuss some implications for “average-case” problem instances with randomized algorithms.\nFinally, improving upon prior work, the paper shows that for some notions of affine region, the function problem can be solved in polynomial space (and therefore in exponential time).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper presents a nuanced discussion of worst-case computational complexity of linear region counting, under different non-equivalent notions of linear region.\n2. Reductions to known hard problems (SAT and cell-counting for oriented hyperplane arrangements) are presented formally and clearly.\n3. The work provides clear (counter-)examples illustrating the main technical nuances (main figures as well as appendix C). The gist of most proofs is included in the main paper.\n4. The paper shows that even approximate region counting is hard for MLPs, and touches upon hardness under randomized polynomial algorithms.\n\n## Weaknesses\n1. While it is very important to establish hardness results, worst-case problem instances are limited in that they don’t provide intuition on average-case hardness for certain problem distributions that are relevant in practice. Indeed, the hypothesis space of ReLU MLPs is very large and those worst-case solutions might be rare in practice. \n\n2. With the exception of Corollary 5.5, the dependency of the worst-case bounds on the depth and width of neural networks cannot be read directly from the main text, but only from the appendix. I believe such bounds are of independent interest beyond “region counting”, and so it would be useful to include them in the main paper.\n\n3. Other established works in the literature (Hanin and Rolnick, ICML’19 and NeurIPS’19), argue that, under certain assumptions, “worst-case complexity” instances are isolated (discrete) points in the hypothesis space of ReLU nets (see for instance Lemma 12 in Hanin and Rolnick, NeurIPS’19). It would be useful to relate the worst-case bounds to their results (see Questions below).\n\n4. The paper ends rather abruptly by sketching Algorithms 1 and 2. I think it is important to include a final discussion, relating the paper’s technical results with the broader literature (see Questions below). This was partially done under “Limitations”, in section 1.1.\n\n\n### Justification of the given scores\n\n**Quality:** The overall technical quality of the paper is high, with clear reductions to known hard problems. I particularly appreciate the discussion on computational hardness for randomized algorithms.\n\n**Clarity:** The paper is overall clear. Most concepts are clearly introduced in Section 2, making the paper accessible to interested but non-specialized readers. However, the dependency on input dimension, network width, and depth is sometimes not immediately clear from the main text, and can be found only in the appendix. This could be improved.\n\n**Significance:** The paper presents worst-case reductions for the decision and function formulations of linear region counting, for several notions of linear region as well as for shallow and deep MLPs. The work addresses some important technical inconsistencies within prior work, which are of interest beyond the narrower scope of the paper. \n\nMost importantly, the paper discusses the hardness of exact and approximate region counting. Since region discovery appears under different guises in the ML literature (from density estimation problems to estimating Lipschitz continuity and robustness), I believe the work provides a solid base on which further theoretical analysis can be made, for specific problem distributions.\n\n**Originality:** The work identifies a gap in the literature, whereby only limited technical results formally quantify hardness of region-counting problems (most notably Wang, 2022). The main technical novelty in terms of Mathematical proofs lies in constructing ReLU networks satisfying a given SAT condition, and then extending said construction to networks with depth $L > 1$. The proof technique was borrowed from Katz et al. (2017) as acknowledged by the authors.\n\nOverall, I appreciate the technical contributions advanced by the work, and the only limitations I see are with the paper’s broader impact beyond the “region counting” niche.\n\nQUESTIONS:\nI outline below a few technical questions that should be considered by the authors in order to broaden the impact of the work. I believe it is not necessary for the authors to fully address these questions in order for the paper to be accepted. However, I strongly encourage the authors to include a discussion of some of the points below in the next revision of the paper.\n\n1. Corollary 5.5 establishes an exponential relation between input space dimensionality and the hardness of region counting. Said dependency also appears in classical results on hyperplane arrangements – most notably Zaslavsky’s Theorem –  and is connected to the well known “curse of dimensionality” in ML. I think this relation should be discussed more explicitly by the authors. \n\n2. In the main text, it would be ideal to make explicit the dependency on $n_i$ and $L$ of Theorem 5.2, Lemma 5.3, and Theorem 5.7. Particularly, how does complexity scale as a function of $n_i$ vs $L$?\n\n3. Hanin and Rolnick (NeurIPS’19) argue that worst-case complexity solutions in the hypothesis space of ReLU nets under random Gaussian initialization are discrete points. I wonder if the worst-case solutions you construct in your proofs fall under the same category, or whether they are stable under small perturbations of the network’s parameters.\n\n4. I think it is of great value to draw connections between the complexity results presented in the paper and other problems in the literature that entail knowing the linear regions partition expressed by a ReLU network. A non-exhaustive list is:\n  * Estimating the Lipschitz constant of ReLU networks entails minimizing a quadratic form over the underlying polytopal complex (Virmaux and Scaman, 2018).\n  * Density estimation is related to the size of linear regions (Polianskii et al, 2022).\n  * Computing the spectral decay of a ReLU network (Rahaman, et al. 2019)\n\nOther minor points:\n\n* The labeling of R1, …, R6 in the caption of Figure 1 does not match the definitions given in Section 3. Indeed, the labeling used in the caption even contradicts Theorem 3.1. This is easily fixable.\n\n* It is common in Machine Learning literature to group hypotheses and assumptions in a dedicated paragraph, before stating technical results.  To further improve clarity and accessibility of the work, some assumptions and conjectures from computational complexity could be summarized in a dedicated section (e.g. in the appendix). It would be useful to include such a section (e.g. in the appendix) to recall common assumptions such as NP$\\ne$RP.\n\n### References\n\n* Complexity of linear regions in deep networks. Hanin and Rolnick. ICML 2019.\n* Deep relu networks have surprisingly few activation patterns. Hanin and Rolnick. NeurIPS 2019.\n* Lipschitz regularity of deep neural networks: analysis and efficient estimation. Virmaux and Scaman. NeurIPS 2018.\n* Voronoi density estimator for high-dimensional data: Computation, compactification and convergence. Polianskii et al. ICML 2022.\n* On the spectral bias of neural networks. Rahaman et al. ICML 2019.\n\nLIMITATIONS:\nLimitations of the work are properly discussed in Section 1.1. To further broaden the impact of the paper and improve contextualization of the findings in relation to the broader literature, I would have appreciated a closing discussion section describing how the technical results relate to other problems associated with linear region counting. A series of pointers is provided under “Questions” above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}