Title: On Universality Classes of Equivariant Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Geometric Deep Learning

Theory for Equivariant Neural Networks

Expressiveness

Approximation Theory
Abstract: Equivariant neural networks provide a principled framework for incorporating symmetry into learning architectures and have been extensively analyzed through the lens of their *separation power*, that is, the ability to distinguish inputs modulo symmetry. This notion plays a central role in settings such as graph learning, where it is often formalized via the Weisfeilern&ndash;Leman hierarchy. In contrast, the *universality* of equivariant models&mdash;their capacity to approximate target functions&mdash;remains comparatively underexplored. 
In this work, we investigate the approximation power of equivariant neural networks beyond separation constraints.
We show that separation power does not fully capture expressivity: models with identical separation power may differ in their approximation ability.
To demonstrate this, we characterize the universality classes of shallow invariant networks, providing a general framework for understanding which functions these architectures can approximate. Since equivariant models reduce to invariant ones under projection, this analysis yields sufficient conditions under which shallow equivariant networks fail to be universal.
Conversely, we identify settings where shallow models do achieve separation-constrained universality.
These positive results, however, depend critically on structural properties of the symmetry group, such as the existence of adequate normal subgroups, which may not hold in important cases like permutation symmetry.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical investigation into the approximation capabilities (universality) of equivariant neural networks, arguing that this property should be considered distinct from the more commonly studied separation power. The authors develop a framework for analyzing the universality classes of shallow invariant networks by connecting them to the theory of generalized ridge functions. This framework characterizes the functions a network can approximate via a set of differential equations they must satisfy.\n\nThe main contribution is a formal characterization of the universality classes for shallow invariant networks, and a demonstration for concrete examples (shallow CNNs and PointNets), that models with identical separation power can have different approximation capabilities. This directly shows that separation power is not a complete measure of expressivity. Further, the authors generalize prior universality results and show that shallow networks can be universal if the symmetry group possesses suitable normal subgroups, a condition that highlights structural limitations for important cases like the permutation group.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper's central message (separation power does not fully capture expressivity) is a nice conceptual contribution to the field. Much of the theoretical work on GNNs and other equivariant models has focused on separation, often using it as a proxy for overall expressivity. This paper convincingly demonstrates the limitations of that view.\n2. The authors build a rigorous mathematical framework leveraging sophisticated tools from approximation theory (specifically, the theory of ridge functions and their characterization via differential operators). The main theorems are precise, and the proofs provided in the appendix appear sound to me (although I did not check everything carefully).\n\n**Weaknesses**\n\n1. The analysis is limited to shallow networks. While this is a necessary starting point for a deep theoretical analysis, it remains a key limitation. The authors point out that depth can be crucial for achieving universality in practice, but how exactly the interplay between depth, separation, and approximation power unfolds in deeper models remains a major open question.\n2. The main characterization theorem (Theorem 13) and its corollaries are elegant but highly technical. For a practitioner wanting to analyze a new architecture, verifying the required conditions could be very challenging. The primary value of the theory may lie more in the conceptual insights it provides than as a plug-and-play analysis tool.\n3. The fact that the paper is highly technical and dense in theory makes it comparatively hard to read. Given the topic, this is unavoidable to some degree, but I think the paper could benefit from illustrations that allow readers to develop an intuition about relevant concepts.\n\nQUESTIONS:\n1. The results clearly highlight the limitations of shallow permutation-equivariant networks. You contrast this with prior work showing depth-3 PointNets are universal. Could you speculate on how your analytical framework could be extended to deeper networks? Would a recursive application of the differential operator analysis for each layer be sufficient, or would a fundamentally different approach be necessary?\n2. Your results suggest that for groups like $S_n$, standard shallow equivariant layers are not universal. What are the practical takeaways for an ML engineer designing a permutation-equivariant model? Should they always build deep models? Or does your theory suggest alternative designs for the hidden layers that could achieve universality even with limited depth?\n3. Instead of strictly enforcing symmetry by a particular architectural design, an alternative approach is to use an unconstrained architecture and learn the symmetries of interest from (augmented) data. Assuming such a model has learned the symmetries sufficiently well (i.e. it is approximately equivariant), would it have similar limitations? Phrased differently, is it possible to derive a sort of \"tradeoff\" between achievable approximation accuracy and equivariance for a (shallow) unconstrained model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the universality of equivariant neural nets, and particularly its relationship with their separation power, that is, their ability to distinguish elements in different equivalence class. After a (lengthy) introduction on the different notions, they exhibit an example of networks that have the same separation power for point clouds, but different universality (one is universal, the others are not). Their main theorems pertains to necessary/sufficient conditions for universality or failure thereof, and the vanishing of certain differential operators on classes of function.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- this negative results are quite original in this active literature. I believe that, at least the concept, is an important step towards a better understanding of separation, universality, and the building of invariant/equivariant architectures with such properties.\n\nWeaknesses\n- the main weakness of the paper is that it is extremely notation-heavy and quite hard to parse. It' feels like each line introduces new notations up until the very end, making the results quite obfuscated and hard to judge. Examples, main theorems, etc. feel rushed: for instance, one of the main theorem is about differentiability, and this notion is never introduced before the theorem itself! I feel like this paper would really benefit from being journal length, or from being in a much more focused and example-heavy way.\n\n- with so many notations, some of them clash, eg $M_d$ and $M_h$ with integers $d$ and $h$, and some are redundant, I believe neural nets are re-defined many times along the way. It's very easy to lose track.\n\n- proposition 9 is the core of the idea and it is a bit confusing. width-1 convolutional networks are mixed with permutation-invariance (which is weird), and the proof is quite confusing, for such seemingly simple architectures. Examples like this should be more explicitely written.\n\n- related to this, another weakness is that, as of now, all the notions and theorems feel a bit too \"heavy\" for the rather simple counter-example that is the core of the authors' demonstration. For instance, the authors introduce a quite convoluted notion of \"separation class\" and \"separation-constrained universality\", and then only apply it to maximal separation over point clouds. Could other, less trivial examples be found?\n\n- The authors claim that equivariance can be summarized to invariance, but it seems a bit imprecise. They correctly that, *for negative results*, this is the case; of course failure to be invariant-universal is also failure to be equivariant-universal. But the converse is a bit brushed off.\n\n- A large branch of proofs of universality precisely proves it by using separability + Stone-Weierstrass theorem. It could be useful to include it in a remark.\n\nIn conclusion, I believe this paper starts an important line of results, but would really benefits from being longer, with more examples that are less trivial, and more time to discuss the two \"main\" theorems, which are rushed at the end of the paper used notions that were not introduced in the rest of the paper.\n\nQUESTIONS:\nSee above, mostly related to invariance vs. equivariance and other examples of \"true\" constrained separability (the given example uses maximal separability).\n\nLIMITATIONS:\nLimitations are appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper revisits the relationship between approximation power and separation power for equivariant neural networks. While most prior work focuses on separation power—the ability to distinguish inputs up to symmetry—the authors ask whether networks that separate equally well can also approximate equally well. By characterizing the universality classes of equivariant networks, it shows that separation power alone cannot fully describe approximation power: models with identical separation power may differ in their approximation ability. Particularly, it exhibits three popular set models whose separation power is maximal yet whose approximation power is strictly nested.

STRENGTHS AND WEAKNESSES:
**Strengths**
- It was known that universal approximation implies maximal separation power. This work provides another piece, suggesting that the opposite is not true: maximal separation power does not necessarily lead to universal approximation. 
- This theoretical finding implies that function approximation power and speration power of equivalence classes are fundamentally distinct. Therefore, model analysis and architectural design should address separation and approximation as distinct criteria, instead of relying on just one of them.



**Weaknesses**
- The main theoretical results are mainly restricted to shallow networks
- There are no empirical validations 
- The notion is heavy to read

QUESTIONS:
See **Weaknesses**.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
Observes that we should care about more than universality “up-to-separation” of equivariant neural networks, we should also care about their capacity to approximate a function. Characterize universality for shallow models. Show that there are families of models (e.g. shallow PointNets and CNNs with a filter width of 1) that differ in their approximation capacity but have identical ability to separate inputs mod symmetry.

STRENGTHS AND WEAKNESSES:
## Strengths

Offers an interesting, novel perspective on the universality of equivariant networks. Construct real examples where separation and universality differ. The proofs are relatively clear and the assumptions are generally stated.

## Weaknesses 

- Importance: the theorems target depth-2 models. It seems non-trivial to extend to deep architectures (and motivate this distinction), in-part because compositions of equivariant layers have significant approximation capacity.
- Similarly: is there a reason why universality captures certain intuitions better than separability and why should this matter? 
- Minor: In the conclusion, you state that you formulate shallow invariant networks as superpositions of ridge functions. You do not explicitly connect these in the main body/cite your appendix.

QUESTIONS:
- For Thm 18, Stone–Weierstrass also requires point-separation?
- The n=3 PointNet case seemed rush. Could you perhaps add a bit more detail here.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a theoretical framework to characterize the universality of shallow equivariant neural networks, without assuming universal components like MLPs. For this, concepts of separation power of equivariant networks are leveraged and links between separation power and universality and obtained. Necessary criteria for universality are derived, along with examples of applications of these ideas to widely used architectures.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper is well-written and overall a great read despite the heaviness of the theoretical framework and notations\n- The results of the paper generalize and extend several existing results in a non-trivial and very interesting way. They have practical implications and can be used to show explicitly if some architectures satisfy universal criteria. As such they will be useful and important to the geometric deep learning community\n- The limitations of the analysis are clearly stated\n\n## Weaknesses\n- Some related works are not mentioned, notably [1,2], which formalize concepts related to separation constrained universality for graphs. Another line of work sidesteps the issues of expressivity of equivariant neural networks by adaptation of universal architectures [3,4] and is worth mentioning. \n- The paper is clearly written, but understanding could potentially be improved by including some figures (they could fit with the additional page or in appendix). One recommendation is for example to represent the constructions for the layer spaces (equation 1) using coloured bipartite graphs as is done for example in [5] (This paper is also worth citing as it introduced a formulation close to equation 1). I think this would allow to visualize all the examples provided in the paper in an intuitive way.\n- The paper would also be improved if the authors could add short proof sketches and find some intuition that is captured by each of the main theorems. In their current form the statements are a bit dry.\n\n[1] Azizian and Lelarge, Expressive Power of Invariant and Equivariant Graph Neural Networks, 2021\n\n[2] Chen et al., On the equivalence between graph isomorphism testing and function approximation with gnns, 2019\n\n[2] Puny et al., Frame averaging for invariant and equivariant network design, 2021\n\n[3] Kaba et al., Equivariance with learned canonicalization functions, 2022\n\n[5] Ravanbakhsh et al. Equivariance through parameter-sharing, 2017\n\n[6] Pacini et al. Separation Power of Equivariant Neural Networks, 2025\n\nQUESTIONS:\n- One component that I felt would make the story more complete for me is a characterization of conditions for $C_\\rho = C_G$, e;.g. when does separation constrained universality coincide with universality? I think this would help in understanding the implications of Theorem 18. Are results from [6] enough for this? If yes, it would be nice to include them in the discussion.\n- If I understand correctly, the framework does not fully generalize to the equivariant case, since only non-universality (and not universality) transfers from invariance to equivariance. Is that right? If yes, this should be mentioned more explicitely.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}