On Universality of Deep Equivariant Networks
Authors: Marco Pacini
Mircea Petrache
Bruno Lepri
Shubhendu Trivedi
Robin Walters
Venue: ICLR 2026 Poster (2026)
Keywords: Geometric Deep Learning
Theory for Equivariant Neural Networks
Expressiveness
Approximation Theory
Primary area: learning on graphs and other geometries & topologies
TL;DR: We show that depth is decisive for universality. We derive separation-constrained universality results for invariant and equivariant networks, unifying and extending prior work.

Abstract: Universality results for equivariant neural networks remain rare. Those that do exist typically hold only in restrictive settings: either they rely on regular or higher-order tensor representations, leading to impractically high-dimensional hidden spaces, or they target specialized architectures, often confined to the invariant setting. This work develops a more general account. For invariant networks, we establish a universality theorem under separation constraints, showing that the addition of a fully connected readout layer secures approximation within the class of separation-constrained continuous functions. For equivariant networks, where results are even scarcer, we demonstrate that standard separability notions are inadequate and introduce the sharper criterion of *entry-wise separability*. We show that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime. Together with prior results showing the failure of universality for shallow models, our findings identify depth and readout layers as a decisive mechanism for universality, additionally offering a unified perspective that subsumes and extends earlier specialized results.

Published as a conference paper at ICLR 2026
ON UNIVERSALITY OF DEEP EQUIVARIANT NETWORKS
Marco Pacini"
Mircea Petrache Bruno Leprit Shubhendu Trivedi
Robin Walters 1
ABSTRACT
Universality results for equivariant neural networks remain rare. Those that do exist typically hold only in restrictive settings: either they rely regular or higher-order tensor representations, leading to impractically high-dimensional hidden spaces, or they target specialized architectures, often confined to the invariant setting. This work develops a more general account. For invariant networks, we establish a universality theorem under separation constraints, showing that the addition of a fully connected readout layer secures approximation within the class of separation- constrained continuous functions. For equivariant networks, where results are even scarcer, we demonstrate that standard separability notions are inadequate and intro- duce the sharper criterion of entry-wise separability. We show that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime. Together with prior results showing the failure of universality for shallow models, our findings identify depth and readout layers as a decisive mechanism for universality, additionally offering a unified perspective that subsumes and extends earlier specialized results.
INTRODUCTION
Symmetry has emerged as a key organizing principle in deep learning, Equivariant neural networks encode symmetry by ensuring that transformations of the input are mirrored by corresponding transformations of the output. This inductive bias has proven successful in applications ranging from vision and molecular modeling to representation learning on graphs and manifolds (Cohen & Welling 2016 Kondor & Trivedi, 2018 Bronstein et al. 2021).
An important concern, however, is whether the introduced inductive biases may impose additional undesired constraints beyond symmetry. In this direction, the majority of work focuses the study of expressivity, a broad notion that intuitively reflects the capacity approximate arbitrarily complex target functions. This notion admits different formalizations, but two main approaches are currently investigated in the literature. The first directly tackles universality, understood here as the problem of approximating all symmetty-compatibb target functions (Ravanbakhsh 2020: Keriven & Peyré 2019 Maron et al.] [2019b] Sonoda et al. (022). which, however, often requires models with impracticall large intermediate representations. The second approach concerns the ability of models to distinguish input pairs, their separation power, an ability that also constrains the functions they can approximate. Separation has been extensively studied in the graph learning community through the lens of the Weisfeiler- 2019, Maron et al., 2019a), and more recently in a general equivariant setting (Joshi et al.) [0033] Pacini et al. 2024). In particular, Chen et al. (2019) and Joshi et al. (2023) present initial approaches to universality that explicitly account for separation constraints. They establish universality results up to Weisteiler-Leman and orbit separation, respectively, thereby furnishing the first cases of separation-constrainne universality.
family of models to represent
test (Morris et al.
nuanced landscape for the interaction between separation and universality. For instance, they present examples of invariant shallow architectures with the same separation power but different approximation power, showing that although separation is
However, Pacini et al. (2025) suggest
* Uniiverity of Trento; Fondazione Bruno Kessler mpacini8 fbk.eu
I PPC Chile. mpetrache@uc... cl.
$ Fondazione Bruno Kessler. lepri@fbx.
5 shhbhenddu@csail ceaicuuccssal...m mit.edu
Northeastern University. r. wal wer.@ehrrhhessteem..cd

---

Published as a conference paper at ICLR 2026
necessary condition for approximation, it may fail to be sufficient one. Nevertheless, Zaheer et al.] (0177), Qi et al. (2017), and [Segol & Lipman (2020) show that adding fully connected readout layers increasing the depth of this limited class of architectures transforms them into universal models up to separation. This suggests that depth and readout layers may play a crucial role in achieving separation-constraine universality and, more generally, in efficiently enhancing approximation power. In this paper, we shed light on this phenomenon by investigating the role of depth in separation-constraane offer unified framework that goes beyond earlier architecture-speet a separation separation-coostrriineed universality theorem for invariant networks, showing that models with fully connected readouts can approximate every continuous function consistent with their separation relation (Section 4.. We then turn to the equivariant setting, where a simple example shows that standard separability is too coarse to characterize universality. To address this, we introduce the notion of entry-wise separability (Section 5.1]. Intuitively, instead of considering the separation relation of the entire function, examine all the separation relations of its projections onto individual output coordinates simultaneously. With this notion in place, we prove two entry-wise separation- constrained universality theorems. These results establish that deep equivariant networks achieve universality either when depth is sufficient to stabilize separation or when specific output layers act, in the equivariant setting, as surrogates of fully connected readouts in the invariant case (Section[5.2) In summary, our results identify depth and readouts as key factors for universality across broad classes of invariant and equivariant architectures. They clarify the role of separation in approximation and subsume earlier results restricted to shallow or architecture-specffic settings.
universality, both in the invariant and equivariant regimes, and results. Our first result is
We summarize our main contributions below:
We establish a epparation-constrainee universality theorem for invariant networks (Theo- rem showing that the addition of a fully connected readout guarantees approximation within the separation-constrained class.
We introduce the concept of entry-wise separability and demonstrate, via Example that S
standard separability fails to capture the universality class of equivariant networks.
Building on this refinement, we prove two entry-wise anparation-constraiee universality theorems, showing that equivariant networks achieve universality either once entry-wise separation relations stabilize with depth (Theorem 2]. or when equipped with specific readout layers (Theorem 3).
2 RELATED WORK
Equivariant architectures have emerged as a principled framework to incorporate symmetry into ma- chine learning (Cohen & Welling, 2016 Kondor volution, variety of techniques have been developed to enforce equivariance in over-parameterrieed mechanisms, including approximate equivariance (Finzi et al. Petrachte & Trivedi 2023), tensor- and polynomial methods (Thomas et al. 2018). and hybrid These approaches have been extended to different data structures, such point clouds (Fuchs et al. 2020), graphs (Victor Gareia Satorras et al.. (021). I), and simplicial complexes (Battiloro et 2025). Thanks to this versatility. these models were able to adapt to diverse symmetry-sensiiiee domains, including high-energy physics (Bogatskiy al. 2020), structural biology and drug discovery (Jumper et al. 2021), robotics (Huang et al. 2023). and medical imaging (Lafarge et al. 202T).
Kondo & Trivedi, [2018] Bronstein et 202T). Beyond con-
or hierarchical eepresentation-leamiii
Petrache
polynomial models (Dym & |ortler
However, due to this heterogeneity of the landscape, a principled understanding of how such biases affect models remains fragmented and far from complete. Most work focuses on expressivity. the capacity of a model class to represent arbitrarily complex target functions. [Ravnbaahhsh (2020) and [Soodaa et al. (2022) show that shallow architectures with regular hidden representations can approximate any equivariant map. However, these results require hidden spaces whose dimension scales with group size, making them impractical. Similarly, Maaroo et al. (2019b) establish universality for invariant networks with high-order tensor representations, though such constructions remain far from practical use. Among architectures used in practice, graph neural networks occupy a prominent role in the geometric deep learning literature. The study of their expressivity has been carried out test, which enables fine-grained
primarily through the lenses of separation, via the Weisfeiler-Lene understanding of the model's separation capabilities. This focus is justified by the result of Chen et al.
2

---

Published as a conference paper at ICLR 2026
(011 who establish universality up Weisfeiler- Leman separation. Building this, Joss,Joshi et al.
(023 extended the result to geometric graph neural networks, establishing universality up to orbit separation. While these universality results remained confined to the invariant setting. the separation power of equivariant networks is now well characterized (Geerts) 2020 Geerts & Reutter [022] Pacini al. 2024), and depth plays central role. However, Pacini
(2255 recently showed that
depth and additional readout layers play a more subtle role in approximation relative to separation, demonstrating that they can change the class of functions that are approximable without altering separation. This stands in stark contrast to the theory of classical neural networks, where depth is known to improve parameter efficiency but not the class of approximable functions (Tellarsky 2016 Yarotsky 2017 2018).
We extend this literature in two ways. First, we prove that adding a fully connected readout layer is sufficient to achieve connsriin- universality for invariant neural networks. Second, we introduce and analyze entry-wise separability, showing that provides the appropriate refinement for extending separation-constrainee universality to equivariant networks. We present these results in mathematical framework that is general enough to encompass prior settings while precisely capturing the underlying phenomena.
PRELIMINARIES
3.1 GROUPS AND EQUIVARIANCE
We study functions that behave consistently under prescribed transformations. Among them, some are naturally formalized by the algebraic structure of group: sets of transformations closed under composition and containing inverses and an identity. While group theory provides the natural framework for reasoning about symmetry, in the context of neural networks it is convenient to reformulate these ideas in linear-algebraic terms. This is achieved through representation theory, which encodes abstract group elements as matrix actions on vector spaces (Serre) 1977).
Permutation representations play central role in this work. They arise when group G acts on a finite set X, where the action is given by an identification of G as subset of permutations of X. Let IR denote the space of real-valued functions on X, and for each æ E define the indicator e, E Rx by (x)(x) = 1 and (y) 0 for y * z. The collection {ez)eex forms a canonical basis of X The associated permutation representation is the linear action on V = R given by g(ez) Cggz where gr is the result of g acting on for g E G, X.
V and W are permutation representations of G. a map Φ V + W is called G-equivariant when <(gu) go(v) for all g E G, E V. We denote by Hom(V, W) the space of linear maps and by Homa (V, W) the subspace of G-equivariant linear maps. Similarly, Aff(V, W) denotes the space of affine maps and o W) S AV, W) the subset of G-equivariant affine maps. All these spaces are real vector spaces under pointwise addition and scalar multiplication.
Further preliminaries are provided in Appendix A
3.2 LAYER SPACES, NEURAL SPACES & EQUIVARIANT NEURAL NETWORKS
We now describe equivariant neural networks, which are model classes with group equivariant layers. Throughout, we restrict attention to networks equivariant under the action of a finite group, with layers given by permutation representations and equipped with arbitrary point-wise continuous activations. We begin by introducing the notion of layer space, namely space of affine maps subject to additional constraints- such as cquivariance requirements or restrictions on the set of permissible which will serve as the fundamental building block of the neural architectures under consideration.
Definition (Layer Spaces). Let G be a finite group acting on a finite set X, let X be the permuta- fion representation associated with this action, and let V be another permutation representation of G. A layer space is a subset M C (V, Rx). ). In this work, we focus on spaces of the form
M 4 Σ
I (v)
Σ Uj 1x, j=1
F1. Ibe 91 BE R
(I)
3

---

Published as a conference paper at ICLR 2026
where ,'...., of Homm(V the sets X1...., Xt denote all the orbits of X under the G-action, and Lx, := -=Ex, ez ez -..... with {ez bae the canonical basis of R
Example 1. We give some examples of layer spaces, I,, I, C, and P. which will be used throughout the manuscript as running references in our analysis of the universality phenomena. These layer spaces correspond widely used architectures in geometric deep learning and illustrate how standard models naturally fit into the general form 7
(i) Linear Layer: Linear layers in standard neural networks are given by elements of ARR, R.) For the action of any group, we can define the set of affine linear maps between trivial representations L:= AI,, RR, whose relevance will become clearer later, for instance, in relation
(ii) Invariant Layer:
Let G be a finite group acting on a finite set X, and Rx denote the associated permulation representation. We denote by lK the trivial real representation of G. The space of G-invariant affine maps from Rxx to R is denoted by == (R ,R). If
X = X1 ..... L. Xe is the orbit decomposition of X, then we have the characterization
I= UH Σ
x v + 21..... E.
(iil) Convolutional Layer: Standard convolutional layers correspond to maps equivariant with respect to the cyclic group G Zn x Zns acting by the standard cyclic permutations on the product X = [n] x [n]. and can be naturally formulated within the framework of permutation representations. Here consider the generalization to general finite G acting on finite X, and focus on convolutional layers with filter width 1 between general permutation representations R.X. These can be written the form of follows.
C= c==fur urt rid + Σ y:lx. 21 De E
(2)
Note that here C (R X for any action of G X, where X X1 L....U U denotes the orbit decomposition of X. The same setting can be extended to wider filters, but for ease of exposition, will not be used this paper.
(iv) PointNet Layer: Sum Sum-pooling PointNet layers (Qi et al. 2017) are designed to process unordered collections, such point clouds, by enforcing permutation equivariance. In the simplest case, an input configuration of n real elements is represented as a vector a E RR" where we identify R [R[*]] = R This definition extends analogously the general case with muliidiimensioal features. Equivariant PointNet layers act on such inputs using maps in the space Affs., (R", R.). Zaheer al. (2017) characterized this space
featue,
P : ++ (0) id +x2llT) + yl
Ø1, 01, 22, E R
where [n]
[1, 1]T.
We restrict our study to point point-wise activations, also referred in the literature component-wise entry-wise activations.
Definition (Point-wise Activation). Let + R be a nonlinear activation. Given a permutation representation RR of group G, define the associated point-wise activation ă I R by ö (Exex axl2)) = -rEx ((0))ez. Wherever the usage is unambiguous from context, we will denote both and by the symbol.
Throughout the paper, assume that the activation function RR -> IR. is non-polynomial, focus on universality results for architectures fixed depth.
We state the definition of neural network and of the functional space of fixed neural architecture, which we call a neural space, also referred to in the literature as a neuromanifold (Calin 2020).
4

---

Published as a conference paper at ICLR 2026
Definition 3 (Neural Networks and Neural Spaces), Let G be group and Vo..... ..... Va be permutation representations of G. For each i = 1...., d, let M, be a layer space in (V, Vi). For 2 2, the neural space associated with layers Mi..... Ma and activation is defined recursively by
(M .-... Ma) { 10 o........... . oØ* &² M, for each i = -.....,, .d).
Any of E No (M....., Ma) is called a neural network with layers M,...., Ma and activation o.
3.3 UNIVERSALITY CLASSES AND SEPARATION
We aim to characterize the class of continuous functions approximable by neural networks with fixed architecture. We generalize the notion of universality class introduced by Pacini et al. (2025) for shal- low networks, networks of arbitrary depth. Before giving formal defimition, we introduce an aux- iliary notion which plays the role of width in classical (non-equivariant) universality results (Pinkus) 1999), since it can be interpreted as the dimension of intermediate invariant feature representations. V and W are permutation representations of a finite group G and M S (V, W) is as defined in (1 then, for each h, k N we define Mixh the subspace of Aff & R W R²):
M = -{ {{ (011..., IKk + fi ................................ Σÿ= fmo Súj E 1, h.j = 1, .....
(3)
Example 2. Recall that the definition of L Af, R), the layer space k is the set of all affine maps from Re to Rh, namely (ff ) Since L = Aff (R, R) where G acts trivially on R. this layer space can be interpreted as the space of affine G-equivariant maps between trivial representations. In this sense, the multiplicities k and h correspond to the widths of intermediate representations in the standard neural network setting.
With the above notation in place, we can now provide a general definition of universality classes. Intuitively, universality class consists of all functions that can be uniformly approximated on compact sets by neural networks of given architecture, with variable multiplicities of layer spaces. Definition 4 (Universality Classes). The universality class Uo (M Ma) associated with the layer spaces M1,... Ma is defined as
Ug (M...., Ma) U No(M)xh, M xh2 MM KERd-1
Ma. 2xhs d-1
,, 1x1),
where the overline denotes closure in the topology of uniform convergence on compact sets.
Invariant networks are inherently unable to distinguish between elements in the same group orbits, but additional undesired separability constraints may arise when dealing with neural networks with particular architectures employed in practice. A prominent example is given by graph neural networks, which are known to be subject to separation constraints equivalent to the Weisfeiler-Leman test (Chen et al. 2019). To study the universality classes arising from architectures employed in practice, we must therefore take these separability constraints into account. We will use the following natural definitions of separation and of separation- consstrann- universality.
Definition (Separation-Consse Constrained Universality). Let S {f - be a family of functions. We say that U separates two points a, 3 E V if there exists f E U such that f(a) + f(B). The set of pairs that cannot be distinguished by any f E U induces an equivalence relation:
p(U)= {(a 3) E V X | f(a)= = f(3) for all f E eU}.
We say that U is separation-constraiiie functions that preserve the equivalence relation p = p(2)), namely
universal if it approximates exactly the class of continuous
Cv(VWW = (f E CVV.W) f(a) = f(ß) whenever (a, E pp.
Note that separability is necessary condition for uniform approximation sequence of functions with prescribed separation power p converges only to functions that also respect p. In other words, CV, W) is closed subset of C(V, W) in the topology of uniform convergence on compact sets.
compact sets: any
As noted in Sections I and the literature universality for equivariant neural networks is typically architecture-ee architettrredependeen decendent often focusing on the invariant case, and when general, relying on impractically large intermediate representations.
5

---

Published as a conference paper at ICLR 2026
Prior Work. Here summarize, to the best of our knowledge, known universality results, recasting them within a unified framework of universality classes and separation-conat constrained approximability.
1. The classical universality theorem of Pinkus [999., which states that neural networks can approximate any continuous function, translates in this framework as Ug (L, L) = C(R,, IR).
2. Segol & Lipman (020) show that a simplified version of 3-layer PointNets, where convolu- tional filters of width I appear only certain layers (see Examples [in (v). universal. This, in turn, implies that full 3-layer PointNets are universal in the class of continuous Sn equiivarian functions. Namely, Uo (C, P,C) = (P, P, Cs., (R"" ,I",
3. Ravanbakhsh (020) shows that shallow equivariant networks with regular representations hidden layers universal among G-equivariant functions. Namely, for permutation representations and W. (M, N) = Co(V.W) where M = (V, RD) and N Aff W).
and
4. Joshi et al. (2023) show that models expressive enough distinguish all G-orbits be- come universal in the invariant sense once augmented with shallow neural network head. Namely, if the neural space (M ,a, I) separates G-orbits in R", then the associated universality class, augmented with a shallow network head, satisfies Uo (M,..., Ma, J, L.) Co(RR, RR).
5. Geerts 2020); Maron et al. (2019a); [Chen et al. (2019) show that graph neural networks and invariant graph networks (Maron et al.) 2018) can approximate any continuous in- variant function with the same separation power as the Weisfeiler- Leman test. Namely, for the layer space M Affs, (RR") ) (RT)) which processes k-order relational structures equivariantly, Uo (M, M, 1, L) Ca-WLa ((R*))) (IR")(0)) that is, the set
times of continuous functions with the same separation power as the k- WL test after d iterations.
6. Moreover, Pacini et al. (2025) show that in some cases the final trivial layer, as in the two previous examples, is necessary for separation-constrainee universality when certain repre- sentations are involved. Namely, they prove that Uo (C,I) Ç. (P,1)) Ç Cs., (R", R). even though these spaces exhibit the same separation power: p(u (C, T)) = (P, PCCs., (R", ,R))
4 SEPARATIINN-COSTRAANNS UNIVERSALITY FOR INVARIANT NETWORKS
In this section, we establish a general result on separation-constralee universality (Definition 5 for invariant neural networks, extending prior works on invariant universality (see Prior Work 4 and 5) In particular, we prove that pathological mismatches between separation power and approximation power (see Prior Work 6 can always be resolved by adding fully connected readout layer.
Theorem 1.
Let MI...., Ma be layer spaces as defined in Definition 2 land recall that I denotes layer space of invariant linear functions, namely, a subset of the layer space from Example 77 p pl Ma.1)) Then
Ser
(4)
Proof of Theorem Note that by Theorem in Pacini ct al. (024)) and the remark following it, p is preserved under the extension from NG (M,...., MA, 1, L) to Up (M,..., Ma, 1, L) and from No Ma, I) to Uo (M1.... Ma, I), therefore
(M Ma, I)) = p(NV (M,,.... Ma. L)).
Hence we get U, (M,,..., Ma, I.) S CVVV) and we only have to prove the opposite inclusion. Given functions fi...., fn E C(V, RR. define their parallelization as F₁ = (fi,..., fn): V - R Fa(x)) (fi(z),... (x)),, and set
A = (n Fn C(R)). A = ono (G. [,,],,,,XXX,I]]))
(5)
KEN

---

Published as a conference paper at ICLR 2026
Note that by the universal approximation theorem, AA. An An From now will take
F {fr heN to be family of functions such that p(F) =
We get via a result from the appendix
that
CV(V)
Lemma
H
An.
U MEN
A/.
(6)
HEN
Define
N, =
U
NG
M
xkka pkaxh) for each h E N.
RENd-1
Then we can write
Ub (M, M,, = U
(M - xks I I
KENA+1
NG M 1xks Thaxh) 8 No Ixx1I)
=
KENd12
U KEN
JOSISEN, ee NNToo
KEN
Equation
Al
Equation 6
CV(V)
AEN
To prove the above inclusion, if fh (M Ma, then their parallelization (Ji,...,Ja belongs to N₁ by Lemmal from the appendix. The last equality holds because of
Equation and Corollary 2 since there exists family of networks F [ff] such that fa E No ( Md, 1) for each h E N and p(F) P, and we can use this family to define Ake
UNIVERSALITY OF EQUIVARIANT NEURAL NETWORKS
In this section, extend the previous results the equivariant setting. However, important differ- ences between the invariant and equivariant cases emerge: in Section 5.1 we show that the standard form of the separation relation as in Definition 5 often fails faithfully characterize equivariant universality classes, requiring us to introduce the notion of entry-wise separation (Definition 6). In Section [52] we establish universality theorems analogous Theorem showing that the outcome crucially depends on the choice of output space.
5.1 ENTRY-WISE SEPARATION
Here, we study equivariant functions by reducing the problem to the analysis of suitable invariant functions, thereby connecting our setting to the results of Section 4 The main tool for this reduction the projection onto output coordinates. More precisely, let be a finite group acting on the finite set X. For X consider the stabilizer of x, given by G2 Stabe(x) ;= {g G' gæ x). and the linear projection T. X > I onto the z-th coordinate. Then n.. induces the pushforward map
TzA Co CV
(V)
f
- To f.
Since the vector of projections satisfies (Tz) idgx, it follows that )aex acts the identity on CV(V, x. Thus, the study of universality for equivariant maps reduces to the problem of synchronous universality of the invariant projection maps. However, below Proposition I shows that the interaction between equivariance and the global separation is non-trivial when projecting functions onto different output entries.
Proposition 1. Let p = p(N) be the separation relation of a.family of equivariant neural networks N. The restriction of n₂ to
C,, : CCd.
is surjective onto Ca.. (V), the space of Gz-invariant functions with separation relation p.
7

---

Published as a conference paper at ICLR 2026
The proof for Proposition and of all subsequent results may be found in the Appendix.
Proposition shows that, after projection onto a single output coordinate, the space of equivariant functions with separation is constrained by stricter relation. This relation combines P with the G₂ --iivvariance relation, which identifies elements within each Gz-orbit. However, the following example shows that this stricter condition remains insufficient correctly characterize the universality classes associated with equivariant architectures.
Example (Separation for CNNs). Let C' be the layer space of convolutional filters with width 1 defined in Example / ii For the purpose of this example, it is sufficient to restrict C' go from IR" to I" with Sn acting in the standard way on RR. Hence, (2) becomes
C:= { rid ~ + y1
2. &.yER, R
Consider the universality class for 2 2:
UC(C) = Uo CC,,..,, 9).
d timex
We can show (see Proposition 5 that
(²(C) = [((11,.... Rn)) Note that idga E (C). Then, (C) is the trivial separation relation, namely (CU) ((x,2) x ]. Thus, the target space of apparation-constraiin Cs,, (R", ).). However, 7 shows that Ug (C) S Cs., (R",, (R", for each this case eeparation-constrainee universality can never be attained, regardless of depth d.
C(R)) & ff
R*).
(7)
universality is Cs..p (R",
Or equivalently, in
Example 3 shows that characterizing equivariant universality classes requires a finer notion of separability, which now define.
Definition 6 (Entry-wise Separation). Let G be a finite group acting on a finite set X = (I1..... Inh and let Rx denote the associated permutation representation. Let V be another permutation representation over G and N neural space of functions in (V,VRR Let ne : :R R be the linear projection onlo the x-th component for each x E X. Define the family of separation relations
Pz(N)= = {(a, 3) E V X f()) = f
(8) for all E N).
for each x X. We define the entry-wise separation relation as the collection of separation relations T(N) (px, (N), ,. (N)).
We define the set of continuous functions that respect P as
C,(V,B*)= = {s C(V, f | ( V(U1, U2) E Pa E X].
If a universality class with entry-wise separation p coincides with Cp. we call it entry-wise separation universal.
Note that p(N) Pzz (A) ..nn....pp..... (N), the standard separation relation is implied by the entry- wise separation relations. That is N C (V, C Cocr) (V, As noted in Section 3.3 separation is a necessary condition for approximation, and now we see entry-wise separation IS necessary as well. Note that in certain cases entry-wise separation reduces entirely to the standard separation relation, for instance the invariant case where G acts trivially on R, more simply when p(N)= Pz, (N)
P=., (A). Yet, Example 3 shows that entry-wise separation can, in fact, be strictly stronger than standard separation. Indeed, on the one hand 7 gives
(C) = (x....., II) + (xx) E C(R) S Cstabe, (1) (R) (R",RR)
while on the other hand, have UC) C, (RR" R). If we denote IR" R R=-1 here P1 { ((x1, (y1, v)) (R Rn-1)² x1 y1 }. Analogous results hold for the other pi. with i = 2....,.. n. This proves the following proposition and shows that the universality class in Example can be completely characterized by entry-wise separation universality.
Proposition 2. Define p (C)). Then, (C) CR",, R.).
8

---

Published as a conference paper at ICLR 2026
5.2 ENTRY-WISH SEPARATION CONSTRAINED UNIVERSALITY
Now we are ready to state universality results under the more general notion of entry-wise separability discussed in Section
Theorem 2. Let Vo.....V, Vn. be permutation representations of a finite group G. Let X be finite G-set and R* ils associated permutation representation. Let M1y..., ......,M, My be layer spaces in 1,...:J, =o..:h and let M be a layer space Aff (RR IRx) containing the identity map. Let d be such that
(V-1,, V.). for
p: (M Mf, M, M)) = (M My, M. M..,,
(8)
times
d+ times
Then,
Uo (M........ M M. ,..., M C
d+1 times
In other words, repeating the oulput layer beyond the separation-stabilization threshold ensures entry-wise separation-constrainee constrained universality.
Since by Theorem of Pacini al. (2024), separation known to stabilize after certain depth,
obtain the following corollary.
Corollary Assume the notation of Theorem 2 There exists natural number D for which Uo (M,...., My M, M) is entry-wise separation-constraine universal for each d 2 D.
d times
In a different direction, we can show that entry-wise separation-constraineee universality can be achieved when the output layer is a convolutional filter of width 1, without the requirement of sufficient depth as in Theorem and Corollary D This is formalized as follows.
Theorem 3. Let Vo...., V3 permutation representations of a finite group G. Let X be a finite G-set and RX its associated permutation representation. Let M1..... M, be layer spaces in (V- Vi) for =I...., J, and let C be a layer space Aff (RR.* xR of convolutional filters with width as defined in Example 7 Then Uo (M C) Cp(V), where p p(uo (MM...,,.... M,,
Note that when C is defined on a one-dimensional space, we have = L, and 4 becomes the invariant layer space 1, In this case, Theorem 3 which is formulated in the equivariant setting, specializes Theorem D the corresponding result in the invariant setting.
At first sight, it may be tempting to compare Theorem 2 and Theorem Badd conclude that Theorem a stronger statement. However, is important to note that adding the layer space the end does not change the entry-wise separation power of the model class, whereas adding a certain number of M layers may increase it. Theorem eexpicitly accounts for this effect.
Theorem and Corollary may be particularly relevant for their practical implications: they ensure that maximal expressivity reached finite depth and rule out the possibility of unbounded improvement. Theorem
the other hand, is instrumental in recovering known results such as (Segol & Lipman 2020). It also shows that universality stabilization in Theorem 2 and Corollary occur the same depth entry-wise separation stabilization, revealing that the threshold in Theorem 2 is not always optimal.
Remark 1. Thanks to Theorem 3 we can easily recover the universality result of Segoo & Lip
man 2020 Namely, U4
Uo (P, P, P) Cs, CR"", remains verify that T No G.,CC. C.C)) separates Stabs, (1) orbits in R", which follows directly from Lemma (Ap- pendix
Note that this shows that the depth threshold required for separation-stability in Theorem pro- vides a sufficient, but not necessary, condition for universality. Indeed, pV, RC RCC
plu, (V. WW). so separation has not stabilized, yet entry-wise separation universality is already achieved. However, determining in general when separation stabilization takes place is difficult problem. Corollary guarantees that maximal expressivity reached after a finite number of steps and then saturates. This result supports the intuition that increasing depth enhances expressivity. Less

---

Published as a conference paper at ICLR 2026
intuitively, it also shows that beyond certain threshold, saturation occurs and further increases in depth no longer affect the universality class.
Remark 2. Theorem 3 marks a significant difference between the equivariant and the invariant cases.
p(u, (P,1)) pCCs., (R8", R)), the corresponding universality classes satisfy Ue (C, Ç 1) & Cs., (R" R). These strict inequalities are proved via characterization through differential operators. In the equivariant case, we have (C, C) Ç Ur(P. C Cs, (IR", Re), yet as we showed here, both spaces can be characterized terms entry-wise separation, without resorting to the differential operator characterization. Note that we expect this to be phenomenon specific to networks with output layers in c ( Output spaces in A R). or more generally in (R Ry), may instead require characterization in terms of differential operators for arbitrary finite G-sets Y.
Indeed, Pacini al. (2025) shows that, although
LIMITATIONS
Our results characterize universality of deep invariant and equivariant networks under separation constraints, but several limitations remain which provide avenues for future work. First, the theory applies to networks with point-wise activations and permutation representations. Extending the analysis to other types of representations or more general nonlinearities may require different approaches. Second, our universality theorems asymptotic and do not provide quantitative approximation rates sample complexity bounds, which are important for understanding expressivity in practice. Finally, have not addressed optimization or trainability: while depth is shown be sufficient for universality, when such networks can be efficiently trained remains an open question.
7 CONCLUSIONS
We established new universality results for deep invariant and equivariant networks. For invariance, we proved that depth is sufficient to guarantee universality within the class of cetaraaion-respectii functions. For equivariance, we introduced the refined concept of entry-wise separability and showed that, once entry-wise relations stabilize, deep equivariant networks achieve universality. Taken together, these results unify and extend prior shallow or architecture-specffe universality theorems, highlighting depth as a general mechanism for universality in equivariant models. We hope this framework provides a basis for future advances in the analysis and design of expressive, symmetry- aware, neural networks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The work of Mircea Petrache was supported by National Center for Artificial Intelligence CENIA FB210017, Basal ANID. The work of Bruno Lepri was partially supported by the following projects: Horizon Europe Programme, grants #10112 12- 0237-ELIAS and # #101 -20763-TANGO, Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them.
REPRODUCIBILITY STATEMENT
This work is purely theoretical and contains experiments datasets. All results are formally stated as theorems, propositions, or corollaries, and complete proofs are provided in the main text and appendices. Definitions and assumptions are explicitly stated to ensure mathematical clarity, and we reference relevant prior results where appropriate. As such, all claims in the paper can be fully verified by checking the provided proofs.
ETHICS STATEMENT
This work is theoretical and does not involve experiments with human subjects, sensitive data, or deployment of models in real-world applications. We therefore do not foresee any direct ethical concern.
REFERENCES
Claudio Battiloro, Ege Karaismailoglu, Mauricio Tec, George Dasoulas, Michelle Audirac, and Francesca Dominici. E(n) Equivariant Topological Neural Networks, February 2025. URL
15429 arXiv:2405.15422 [es].
Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz Group Equivariant Neural Network for Particle Physics. In Proceedings of the 37th International Conference on Machine Learning, pp. 992-1002. PMLR, November 2020. BtLLitttss//pproco ott.://ccag....ooo.ooooo//oooooa html html 3498.
mlr wttp://ssll//o/asssssssssseeeeee
ISSN: 2640-
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv:2 13478 [es, stat), May 2021. URL http://aaxxw..or/ab/////aaa
http://arxiv. www.orr///bss..... /210g//bb//02..... 13478 arXiv: 2104.13478.
Ovidiu Calin. Deep Learning Architectures: A Mathematical Approach. Springer Publishing Company, Incorporated, 1st edition, 2020. ISBN 978-3-030-36720-6.
Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with GNNs, May 2019. URL http:://axix
Btq/abs/1190s 1905.11.... 12560 arXiv:1905. 12560 [cs, stat].
Taco Cohen and Max Welling. Group Equivariant Convolutional Networks. In Proceedings The The
33rd Internationaal Conference on Machine Learning, pp. 2990-2999, PMLR, June 2016. URL m
https::/proceedingg http://eeeaaaaa........er//////s wwtpp//ese/e////.c//eeeea/e/ html ISSN: 1938-7228.
Nadav Dym and Steven J. Gortler. Low Dimensional Invariant Embeddings for Universal Geometric Learning, May 2022. URL
02956 arXiv:2205. 02956 [cs,
math].
Marc Finzi, Gregory Benton, and Andrew G Wilson. Residual Pathway Priors for Soft Equiv- ariance Constraints. In Advances in Neural Information Processing Systems, volume 34, pp. 30037-30049, Curran Associates, Inc., 2021. htt.///nnoi./u/tt////////oooiaaa
.tmml

---

Published as a conference paper at ICLR 2026
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3) Transformers: 3D
Roto-Translation Equivariant Attention Networks, Advances in Neural Information Processing Systems, 33:1970-1980 2020. URL
www.:eddwwnnnnnnnoooooeoe.e.m.e. nwwrips. Cc/ cc/paper/ paper/
2020/hash/15231arcooo 9d1 2bb.aaa3333111222252555533333333 Abst htt. htm huttm
source=chatgpt.cm com
Floris Geerts. The expressive power of kth-order invariant graph networks, July 2020. URL
12035 1xXiv:2007120035 [es, math, stat].
Floris Geerts and Juan L Reutter. EXPRESSIVENESS AND APPROXIMATION PROPERTIES OF GRAPH NEURAL NETWORKS. pp. 43, 2022.
Haojie Huang, Owen Lewis Howell, Dian Wang, Xupeng Zhu, Robert Platt, and Robin Walters.
Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D. October 2023. URL https
//openreview.oo wwt.:/neeee/uuteeeeeeeee
Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Lio. On the Ex- pressive Power of Geometric Graph Neural Networks. International Conference of Learning Representations, 2023, URL peattps:///penneviieee
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596(787)):583 589, August 2021. ISSN 1476-4687 doi: 10. 00000/016661100211 021-00819--22222220 URL Utths::s: www. nwwuratueee...o. wttp://ttaaaeesaaaaeeeoo 021 ture Publishing Group.
Publisher: Na-
Nicolas Keriven and Gabriel Peyré. Universal Invariant and Equivariant Graph Neural Net- works. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL http://///.aapaaaa.pa..ccccappss
html
Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In Proceedings of the 35th International Conference on Machine Learning, pp. 2747-2755. PMLR. July 2018. URL
mlr wwwppppwwssssspeeeee httpa/0///oooor11
html
ISSN: 2640-3498.
Maxime W. Lafarge, Erik J. Bekkers, Josien P. W. Pluim, Remeo Duits, and Mitko Veta. Roto- translation equivariant convolutional networks: Application to histopathology image analysis. Medical Image Analysis, 68:101849, February 2021. ISSN 1361-8415. doi: 10. 10.1016/j.media. 2020. 101849. URL https:: www.wwwccie...... encedi rect. com/ sessscieeeee scc.enc//atticce//pi/.
S1 9136184152030213
Haggai Maron, Heli Ben Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and Equivariant Graph Networks. International Conference on Learning Representations, September 2018. URL
https: // //tp://////eeeevveeeeeee iew .ne.//e///aa forum? Ftt///oommmmmixxx 72
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably Powerful Graph Networks. International Conference of Learning Representations, 2019a.
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the Universality of Invariant Networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 4363- 4371. PMLR, May 2019b. htLhttt::/// http:::s//ppocceeennge mlr wwwpppwwwss.ee...p http/////mmaro11111 html
ISSN: 2640-3498.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-Ordes Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4602 4609, July 2019. ISSN 2374-3466 2159-5399, doi: 10. 10.16/9/aaal h.. URL httpp: ortpojs/rr////////// index php AAA
4484
12

---

Published as a conference paper at ICLR 2026
Marco Pacini, Xiaowen Dong, Bruno Lepri, and Gabriele Santin. Separation Power of Equiv- ariant Neural Networks, December 2024. URL 22Xi:2240600866 [es].
08966
Marco Pacini, Gabriele Santin, Bruno Lepri, and Shubhendu Trivedi. On Universality Classes of Equivariant Networks, June 2025. URL http://arxiv http:/iwworr/bb///55/50 02293 arXiv:2506.0299) [cs].
Mircea Petrache and Shubhendu Trivedi. ApproximatioonGGeneralliinn Trade-off' under (Approxi- mate) Group Equivariance. Advances in Neural Information Processing Systems, 36:61936-61959
files/paper 3222///4111330012201111100001110 org.a/aeeesotttaoooo.a..aaaaa.ar Confreence
pcppspppeppppee_..eseppee.e. December URL http:///wwtttteappeeeeeaa.aaaaaa
html
Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:
143-195, January 1999. ISSN 1474-0508, 0962-4929. doi: 10:1077/S096229299002999 URL
https: www. www.::.aabriiiie.o.o wtt.eiiieee.....oooooeeoooeeeeee
Charles R. Qi, Su, Hao, Mo, Kaichun, and Guibas, Leonidas J. PointNet: Deep Learning Point Sets
for 3D Classification and Segmentation. In 2017 IEEE Conference on Computer Vision and Puttern
Recognition (CVPR), pp. 77-85, Honolulu, HI, July 2017. IEEE. ISBN 978- 1-531-----------11 doi: 1..09/CVPR.2017... URL BRRLhtttp://eeexplorr ieee wwwpg///oueet/////o/oooo 80994999/
Siamak Ravanbakhsh. Universal Equivariant Multilayer Perceptrons. In Proceedings of the 37th
International Conference on Machine Learning, pp. 7996 8006. PMLR, November 2020. URL
https: btttp::/pprceeddingg mlr http://wooeocaooo..oeeeeeee///// 3498.
html ISSN: 2640-
Nimrod Segol and Yaron Lipman. On Universal Equivariant Set Networks, January 2020. URL
http:// wttpv//aavvvvv.oo//.a/ org/ abs/ 02421 1rXiv:1910.0242) [cs, stat].
Pean-Pierre Serre. Linear Representations of Finite Groups, volume 42 Graduate Texts in Mathematics. Springer, New York, NY, 1977. ISBN 978-1-4684-94600 978-1-4684-9458-
7. doi: 10. 11119/9978---64-9955
URL http:: http:///iin..sspinnggo.. springer. wttp:eerr.om.............0...
978 1-84-14684--8877
Sho Sonoda, Isao Ishikawa, and Masahiro Ikeda. Universality of group convolutional neural networks based ridgelet analysis groups In Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS '22, pp. 38680-38694, Red Hook, NY, USA, November 2022. Curran Associates Inc. ISBN 978-1-71322 978-1-71387-108-8
Matus Telgarsky. Benefits of depth in neural networks, May 2016. URL wtl.httpp//arxiv.orm abs/1602. 04485
[cs, stat].
Nathanicl Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation and translation-eqqiiariann neural networks for 3D point clouds, May 2018. URL /tt.///ap.///aa.....///aa///////
08219 arXiv:1802.08219 [cs].
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) Equivariant Graph Neural Net- works. Proceedings of the 38th International Conference on Machine Learning, pp. 9323-9332.
m.r wtt.mm.tw......////.m htt.://///attrra2211. PMLR, July 2021. URL https: http://ccececong....pppesssssaaa
html ISSN: 2640-3498.
Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:10 94:103-114, October 2017. ISSN 0893-6080. doi: 10. 10.1016//.eeumee22007......00 URL https
www htt.://cccccacccccccccaacaaaaaaa
Dmitry Yarotsky. Optimal approxiiapp continuous functions by very deep ReLU networks, June arXiv:1802.03620 [cs].
2018. Utt.ittpp:/arriiv..rrg//
13

---

Published as a conference paper at ICLR 2026
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In Advances in Neural Information Processing Systems, volume 30.
Curran Associates, Inc., 2017. URLhttps: /twpppaapaaaaaa/aaaaaee nips ct cc/paper.p...p wiles/papes 2017/hash/ faaaaash/222244 7d wttp:/aaaeeeee.eeeaa0aaaaaaaaea. aat
A
PRELIMINARIES
Definition 7 (Group). A group is set G together with binary operation G G - G such that:
Associativity: for all h, k E G we have (g- ht) = g- (h k).
Identity element: there exists element G such that g g for every g G.
Inverses: for every g G there exists an element g 1 G such that g g g =-
The group is finite if has finitely many elements.
We next recall the notion of group homomorphism, which is structure-preserving map between groups.
Definition 8 (Homomorphism). Let G and H be groups. A functin é - H called a group homomorphism if. for all g. h G, it satisfies
6 h) ((g) <(h).
Definition 9 (Group Actions). Let G be a group and let X be set. A group action of G X is
map
$ G X -
often written as 6g(x) = (gg,x)) for E G and I E X, such that:
Identity: 6e = idx, where e is the identity element of G.
. Compatibility: for all g, h E G we have 0g tb. Ógh.
In practice, we frequently denote the action by or simply gr instead of (g(x).
A set X together with group action of called a Geset Equivalently, X a G-set if there exists an action G x X + X satisfying the identity and compatibility conditions above.
central notion in our analysis that of a map between G-sets that respects the group action, leading the following definition.
Definition 10 (Equivariance). Let X and Y be G-sets. function f X Y is G-equivariant for all g E G and x E X, we have
f(g x) = g. f(x).
Definition 11 (Group Representations) Let G be group and let V be vector space over R. A representation of G on a group homomorphism
G GL(V),
where GL(V) denotes the group of invertible linear maps V - V. Given such a homomorphism, we obtain linear action of G V by setting
gu (v) for g G, vEV.
When V and W G-representations, denote by Homg V, W) the space of G-equivariant linear maps + W, and by Aff W) the set of G-equivariant affine maps - W.
Note that Home V, W) is vector space. Indeed, 0 Home (V, W), and for any f.g Homg W) and a, 9 e R, the linear combination a + .g is still in (omg(V W). The same property holds for Aff (V, W).
14

---

Published as a conference paper at ICLR 2026
Definition 12 (Permutation Representations). Let X be a finite and let G be finite group acting on X. The associated permutation representation of G is the linear action of G on Rx defined on the standard basis {er)zex by
g(ex)
Eg.z for all g G, IEX.
We now make explicit how this fits the general notion of representation. Proposition 3. Let X and G be as above and set V == Rx For each E G there is a unique linear map
((g) : -V
such that (eg) Gax for all X. Then the map
Φ G - g 0(
is a representation G on V.
Proof. Since fex is basis of V = Rx for each E G there exists unique linear map ((g) V- V such that ((g)(ez) Cyrr for all x E X. Moreover,
s (o))( = <0 (eggz) = (g-1-(9x)) e2
so (1 is the inverse of ((g) and ((g) E GL(().
For g, h E G and any x E X we have
(e( = ((n) = eg. (g.(h.r) ((gh) ((gh)(ez)
hence ((g)((h) = (igh) and Φ : G + GL(V) is group homomorphism.
Moreover, after choosing ordering X {aI,... 2n and identifying V R", each ((g) is
represented by a permutation matrix Pg R with entries
(Py)y if g 2z 2i, and (Py)s otherwise.
In particular, Par = Py and P. In. Pg is group homomorphism into GL,,, (R).
B SEPARATION UNIVERSALITY FOR INVARIANT NETWORKS
We recall the notation introduced in equation 3 for the subspace M of Aff (RY)
{
(01, .Kk + EE f15(3)) E5 (mo(z)) fo M, 1,... h,j 1,...,k
Mkxh
Thus, M C Aff & R & RA). In particular, for each fij in the above definition, we write
fa)(*) Aiaæ bij. where Aij and bij denote respectively the linear part and the translational part of fij. With this notation, the linear and translational parts of an element in M can be written respectively
A11
A12
A1
Σ
byj
A21
Am
Az
Σ
b2j
and
LAn
Asa
Ana
Σ クー bno
The following definitions and lemmas are used for the proof of Theorem 1
Lemma 1. Let f1,..., fn E N,,,.... Ma, 1) then their parallelization (fi,..., fi) belongs to Uieneto No. (M M¹a 1xh).
15

---

Published as a conference paper at ICLR 2026
Proof of Lemmal 7 Let consider the case h 2: for h 2, the proof is analogous.
Let fi: f2 E (M, Ma). Then each affine layer at depth in (f1, fe) is a block diagonal matrix whose first block is the i-th layer of fi and the second is the i-th layer of f2; similar analysis holds for the bias terms. Hence, this layer belongs to M for i 1. For i 1, the first linear layer of (fi f2) is block column matrix where each block is the first layer of fi and f2; again, similar analysis holds for the bias terms. Hence, this layer belongs to M1 This shows that
(f1, f2) E (N M2x² M2x2) =
U
No (M ..... Maa-ix3
kEN2-1
Definition 13. Let M1 Mi...,, Ma be layer spaces. Let be bases for the layer space Mi, and define
M == Spang B;
for each = I....,d. d. Define rational neural spaces as follows:
No (M, == (N
M
Note that N₂ (MI..... Ma) depends on the choice of the bases ....... Bd.
Lemma 2. In the notation of Definition 13
p ................................ Ma)) = p(No
Ma)).
Therefore, (M, does not depend on the choice of bases Bi...., Ba-
Proof of Lemma
By the continuity of the parametrization map and the density of M in M,.
Lemma 2 implies the following corollary.
Corollary 2. There exists countable family F =ffiNEN (M such that
p(F) = (MN,.,,.......
Lemma 3. Let V Rd with its usual topology and let p be a closed equivalence relation on V. For
family F [nnJNEN of continuous maps fn V- - 1h such that p(F) p. Then the set
A:
-UU U ACS fa)l A ((R" ] n21
((R))))
is dense in CV.V). Or equivalently, for every h E C₂(V) there exist nn 1 oc and Ane E C(Rmmn R) such that (
(f1...,, h.
Proof of Lemma 3 For x E V set F(x) = (Jn(x))nEN. Fix a compact K C V. Since each fn is continuous, F(K) is compact in the product VN, Note that p {(r,y) E 12 F(x) F(y)). so that the map : K/p + ((K) defined by <(xx)) F(x) is well defined. Furthermore continuous and bijective, and since K/p is compact Hausdorff and F(K) Hausdorff, & is homeomorphism. Hence every h E C,( factors uniquely as
h = H Flx
for a unique H E C(F(K).RR)
Let n. : ( + (Rm)
T0 be the projection onto the first a coordinates. Note that R)
A =
A A Inl : A
n21
Then A is sub-algebra of ((F()) containing constants. We next prove that A separates points. Indeed, v. E F(K) with + y/. then there exists j with yj # v/j: define the continuous scalar function p 1T + such p(u) (u, yj - Vn)) and note that p(y;) # p(y,)) and choose (((R")")) given by A(z,,...,
p(z)). This function A lies A and satisfies (A o (AOT))() (3) # (A desired. Now we may use the that A C(F(K)) in the uniform norm, concluding the proof.
theorem, which gives
L
16

---

Published as a conference paper at ICLR 2026
C UNIVERSALITY OF EQUIVARIANT NEURAL NETWORKS
C.I ENTRY-WISE SEPARATION
In this section, we study equivariant functions by reducing the problem to the analysis of particular invariant functions, thereby extending the previous results. The tools used for this reduction are suitable projections onto output coordinates, together with reconstruction maps that allow us to recover the entire function from single projection. For the sake of presentation, we begin by considering the case where G acts transitively on X. Let E X, and let G denote the stabilizer of x. Let 7T Rx + be the linear projection on the w-th coordinate in IR' We obtain pushforward map of defined as
Wxe
Let . It be transversal for G/Gz, that is, choice of representatives for classes in G/Gz. We define the reconstruction map as
(V) Co(V 0f fH U fg. v)egue)
If the action of G on X is transitive, then reconstruction map 0% is well-defined, continuous linear operator such that
Proposition 4.
(i) aza -h = idcco. (V)
(ii) 0 ZT idco(V.
Proof. Choosing different representative for each 9i means choosing an element gi h for an arbitrary h E Gr. For f E Ca, (V), the Gæ-invariance of f implies
f((g: h)- ffh--ffh-h- g; ')'' f(g v).
Then Cgchr = Cgg since h E G.. As consequence, the choice of representatives 91.... It does not affect (f). Next, we prove that (f) is G-equivariant: indeed, for E G we have
(Z))))))))
Σ A95 gu)) Egiz
Σ f(; 10) cg-1 w
=g g. Σ f(g 10 0) cg1x
-g- -g-00()(0)
where in the second equality we use the fact that -1. .1...... . 9: is another transversal for G/G These observations prove that 0r is well-defined. is continuous and linear since it the composition of continuous and linear functions. We can choose 91 (f) is simply fw),, proving (i). To prove (ii), notice that for x E X, the set 91..... Qt is transversal of G/Gz if and only if 9ıæ, ge0 is the G-orbit of r. Thus for each C(V. R can write
in which case the z-th coefficient in
f(v) = Σ Tg Tg (1)
(9)
i=1
17

---

Published as a conference paper at ICLR 2026
Now for f E CV can conclude (ii) follows:
=x f(u) =
xz feg UV) cga
i=
Te (gi -1 f(u)) egi
Tagar (v) Eggar
Equation y
f(v).
Proposition says that Tz- is linear homeomorphism, hence, a function class N is dense in CG(V, and only if Tz- (N) is. This means that we can restrict ourselves to the study of function families of type n. (N), which are similar to the study conducted in Section 3.3
Proof of Proposition The claim follows directly from Proposition applying The yields one inclusion, while the reconstruction map yields the other.
Remark 3 (Linear case). In particular, in the affine case we obtain,
AV R*) + Affa. (V) f ne f.
Tig
Note that characterizing Affa, (V, IR) reduces to computing VG,, If RY for finite G-set Y, then we just need to compute the orbits of G₂ on Y.
The previous observations extend, with minor modifications, to the non-transitive case, which we address next. Let X Y1 L...LY, be the decomposition of X into G-orbits. For each 1, ,, denote by T&Y, R-
+ R Yi the standard projection. Consider the maps
Φ: Co(V,R*) + Co(VV Beleee...CCCCCC.. CC((V, f (Tgn TRY..
and
Co(V, W: (..I",))..CCC(V(RR" Co(K, (fI..., (3) - -+ f1(z) f.
Note that a homeomorphism onto its image. Let ,,.,.., be points with I; E Y, Then, for each i 1,...,8, s, have
na; Co(V,R*) = nz; C.(V.R").
Moreover, for each i 1,...,,,
Co(V,R = 0% T2; Co(V.RX) = CG., (V,R).
Therefore, understanding Co RX) reduces understanding Co., (V, IR) for each 1, ... 8, and then reconstructing C&(V, RR via ......., 9r and V. In particular, since we focus on closed linear subspaces l S CV ) it enough to study the subspaces u for = 1....,&, 8, namely universality classes invariant neural networks.
Proposition The following equality is true:
Ur (C.....9 = (x11..... Fn. ) +
(S ç Cs. (R", (10)
d times
18

---

Published as a conference paper at ICLR 2026
Proof of Proposition S neural space No (C Chxk).
We start by considering the case d and then study the more general
Recall X(C') Span {z + idgx 2). Elements in C can be represented affine maps z Br + c where B and have the following block representations
b, id
and
c=
by id
Ch. 1
While elements in Chh*k can be represented as affine maps + Ax + where d E R and
01.1 idgx
41A idgx
A 2 idgx,
dR.i idgx
ahhh
idgx
where Å = [or,y] E R
Given E X and s I,... h. we can write elements 0 E No (C Cthck
h Σ
(z.(x)= Ao
Ozjo (6/z4 + cj)
5=1
for some az by, E R. But note that
(x,(()) Σ
a (b,z; + (j) (x)
(11)
==1
where
(v) = Σ os (byy + cz).
That is, & E (R"",,,, obtain the proof of the theorem for the case d = For d 2, it suffices to note that the composition of spaces of the type No (C (hhk)) again yields elements of the same type. This concludes the proof.
IR,, In other words, taking the limit as h + co and setting = 1, we
C.2 ENTRY-WISE SEPARATION CONSTRAINED UNIVERSALITY
Proof of Theorem
Recall that
Aff R.*)
Thanks to Proposition to prove Theorem it suffices to show that, for each æ X,
Ze- U6 uo .......... My, C) (V).
where
= p(T== Uo (M1, ..... C))
Fix x E X, and define Ps = Tg. C. Then
T²- Uo (M..... Mr, C* = -.- Uo (MI,..... My, nT. C*)
= U, (M,
M,, P2)
Uo (MI.....
Mf My-1; Tg. M I.).
For the second equality, note that
P. =
= HUU (Au A.
={U+> Am (v) + A.RR).
19

---

Published as a conference paper at ICLR 2026
For the third equality, recall that the pointwise activation ő is defined by
Σ
Σvve))
Uuex
(((2) Cx+ xEX
Σ
ZEX
and observe that, for each X, have the commutation relation
TT
ne O
Hence, the final activation we have
{ ( Am (ā()) + H (
( + AR.
and note that, in this way, the final layer becomes the space of arbitrary affine maps of the real line, namely L. Finally, note that My S (V, for some permutation representation V. Thus, by Remark 3 we have
Txz M Affa, RV,
and therefore nrt My is space of Gi-invariant affine functions. Thanks to Theorem we obtain
Nw. Uo (M, My. C) Uog (,, M5-1, T.. M,, L) C.. (V),
where
Pa p(T Uo (M My (T) p(u o L)). This concludes the proof.
For brevity, we will prove all subsequent results in terms of Us (M,..., M) or similar forms. The
times
same results, however, extend verbatim to the more general setting Uo (M,,... Ma. M,..., M. d times
Proof of Theorem
Note that
Ug M.C) C M)
d times
d+1 times
Therefore,
pl Uo (M...., M)) s (M....
M. S (M M))
d+1 tines
times
times
By hypothesis.
:: p( (uo M....,,M (M,
times
and hence
p ( (M.... M,....,,,.Y M,C))
as well. Then, by Theorem
Un M, M, M, = C,())
times
Since functions realized by neural networks continuous, it follows that
Ur (M, M) 1OS CVV.
d+1 times
Finally, observe that
C,(V) Us (M,
(M,...,M. M. (M. M) C CV()),
d times
d4 times
which yields the claim.
20

---

Published as a conference paper at ICLR 2026
Lemma 4. Let X, Y, Z be metric spaces, and ler g C(X,Y and F C(Y, 2). The following identities hold.
PG = p(G = 6
where the closure is taken with respect to uniform convergence on compact sets.
Proof. For convenience, recall the definition
GaF (fog f f
We start by proving the first equality. Since G CGOF, we have
F(G6F) S (GGFF).
For the reverse inclusion, let (x, y) E ( F), that
f g()) = glv)
for each F and g E eg
Fix f E F and let E G. By definition of the closure, there exists sequence (ON)NEN S 9 such that In + uniformly on compact sets. For each n E
f(gn(x)) = f(g.
and by continuity of / we obtain f((()) = f((())) Since ʃ F and E & were arbitrary, (x, E p(g&F,
i.e.,
FGG S (GG
This proves the first equality.
We now prove the second equality. Since G C [G.F, we have 6 S (GGGF). For the reverse inclusion, let (x. v) E p F). that
ʃ g()) f g(v) for each F and g E G.
Fix 9 and let f E F. By definition of the closure, there exists a sequence (fn)NEN S F such that fn + uniformly on compact sets. For each n E N.
fn(g(z)) = fn(gw)), hence taking limits yields f(((()) = f((l)). Since E 9 and f E F were arbitrary, (α,1) E PlGFF)
i.e.,
PGGF) C (G
This concludes the proof.
Lemma S. Let M be layer space. Thenforr each k; E No we have
(kxl 6 pixh C plexh.
Proof. Let & E ckx1 Then & can be written as
d(z),..., 2x) (1()) ... du (xx)
for some $1..... &n E C.
Let pixh, Then va be written
((x)) ()), (x(z))
for some vi ....... tu E P. The composition v0 o E Ckx1 Mixh can be written
s Ó( xR) [ tbe (x,)))
where olo0 E M for each and j since each Φ; is a scalar multiple of the identity plus a translational part.
21

---

Published as a conference paper at ICLR 2026
Lemma 6. The projection पे (C, P.C) separates Stabs,, (r)-orbits in R
Proof. Without loss of generality, assume = 1, and note that
पं (C, P,C) Uo P,,,P,
where Pi C.C. By Lemma
obtain
U UNo(CL c
U
No N p\x)) U
No(CLxx, pkxh, pfx1).
KEN
AEN
A
Then, by Lemma
5 have
((Ur(C,PPP)) pl U No pixh
p pAX1)
K.NEN
C ( Wol , ppx1))
KEN
KEN
Since functions in Uo PC, Stabs, (1)-invariant, they can at separate Stabs, (1)- orbits. Therefore, it suffices to show that Uo (C,C) 6 P1) separates these orbits. We know by Proposition 5 that
(C, (z), (f1,..,, xn) (f(x)) ( f(an)) ʃ C(R)).
We now characterize elements in UV (P, P1). Before proceeding, recall that
C { x d #1 A.p R)
Then
===10== { + (A μ1) AE = (x de A, ER..
Elements in (P, are limits of functions of the form
nTz)= Acão B(x),
where
bi,1 id +b1,2 1
B
A [ai. TT ... an ag-ep
b id tbn 1
for arbitrary h 2 1. Then, for x (01, :.......2... Fn).
n(0)= Σ a of (bm121 + b22 + +... +.... xn)) (21 + +....x..) +xx),
where C ( hxll).
1). Hence,
(P, Pi) = [1i, (Ti,...,,,)) vn) + 2 + +...))am) SEC(R?)
Therefore,
u u= u= (m [g(x)), ...))
+... g(xn))
| |EEC((?) g e C(R)).
Note that the family
{ (oI...., xnn) + ((x1)) +.... g(xx)))))))))))())) C(R)
separates x (x) ......... an) and (Vi,.... Un) if and only if there exists permutation S,
such that Yy- Moreover, separates x and y whenever 21 # Vı, for instance by choosing id and fu, v) Thus, U separates x and y if and only if there exists permutation 7 E Stabs, (1)
such that I = 73. This concludes the proof.
22


Reviews:
[{"review_id": "N59t8ViW76", "reviewer": "Reviewer_pabC", "summary": "This paper develops a unified theory of separation-constrained universality for deep invariant and equivariant neural networks built from permutation representations and point-wise activations.\n\n* For invariant networks, adding a fully connected readout guarantees universality within the class of continuous functions that respect the model’s separation relation (Theorem 1).\n* For equivariant networks, the authors show that standard separation is too coarse and introduce entry-wise separation; universality holds either once entry-wise separation stabilizes with depth (Theorem 2) or by appending a width-1 convolutional readout (Theorem 3). These results recover and generalize several prior architecture-specific theorems and identify depth and readout layers as the decisive mechanisms for universality.", "questions": "- Can the authors outline how their proofs might be adapted to yield approximation rates or width/depth trade-offs, even under simplifying assumptions (e.g., Lipschitz target functions on compact domains)? \n\n- Given a working equivariant model, how should one diagnose whether entry-wise separation has stabilized in practice (e.g., via synthetic distinguishability tests), and how does one choose a minimal readout ensuring universality while controlling parameter count?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Originality: Introduces entry-wise separation and proves that it exactly characterizes universality classes for broad equivariant architectures---cleanly explaining why standard separation can fail. \n* Quality: Precise definitions (layer spaces, neural spaces, universality classes), clear statement of Theorems 1--3, and complete proofs; careful reduction from equivariant to invariant projections via stabilizers. \n* Clarity: Running examples (PointNet-style \\(P\\), convolution \\(C\\), invariant \\(I\\)) make abstract statements tangible; Example 3 vividly shows the need for entry-wise separation. \n* Significance: Unifies and extends prior results (Maron et al., Ravanbakhsh, Segol & Lipman, Chen et al., Joshi et al.) and translates them into a common separation-constrained viewpoint with practical design levers (depth and readout).", "weaknesses": "* Scope limitations: Results are restricted to finite groups and permutation representations with point-wise activations; many modern models rely on continuous groups (e.g., \\(E(n)\\), \\(SO(3)\\)) and richer nonlinearities (gates, attention). The authors acknowledge this as future work, but a discussion of which parts of the proofs break (e.g., reconstruction via stabilizers, existence of enough invariant functionals) would sharpen the limits. \n* Non-quantitative nature: The theorems are existential; no rates, width bounds, or explicit depth thresholds are provided (Theorem 2 uses a stabilization depth from prior work). Concrete bounds for common groups (e.g., \\(S_n\\), dihedral groups) and standard blocks would increase practical impact. \n* Readout specificity: The readouts that guarantee universality are either fully connected (invariant) or width-1 convolution (equivariant). While natural, it would help to clarify how far this extends to other linear equivariant heads used in practice (e.g., steerable bases, attention-style heads), and whether the “identity-containing” condition on \\(M\\) can be weakened. \n* Examples breadth: Example 3 focuses on width-1 convolutions; complementary examples for graph and manifold settings (beyond the WL context) could better demonstrate the generality of entry-wise separation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8Lh2I5NeM", "reviewer": "Reviewer_zQ23", "summary": "This paper develops a general theory for universality in invariant and equivariant neural networks.  \nFor invariant networks, the authors show that adding a fully connected readout layer ensures approximation of all continuous functions consistent with the network’s separation relation.  \nFor equivariant networks, they argue that the usual notion of separation is too weak, introducing the new concept of *entry-wise separation*, which examines separability for each output coordinate.  \nThey then prove two theorems: universality holds (i) once entry-wise separation stabilizes with depth, or (ii) when the network includes a width-1 convolutional readout.  \nOverall, the paper identifies **depth** and **readout layers** as the key factors controlling universality and provides a unified theoretical framework that connects and extends prior architecture-specific results. :contentReference[oaicite:0]{index=0}", "questions": "- Could the authors explicitly assume a non-linear activation in the readout MLP and restate Theorem 1 accordingly? Would Theorems 2 and 3 need similar clarification?  \n- Are there examples showing the exact depth at which entry-wise separation stabilizes for common groups (e.g., $S_n$)?  \n- Would entry-wise separation remain the correct notion if the representation were not a permutation one (e.g., vector or tensor field representations)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- **Originality:** Entry-wise separation is a simple but fresh idea that fixes a real gap in how we reason about equivariant universality. The unifying view across invariant and equivariant settings is valuable.\n- **Quality** The Example~3 counterexample is instructive and motivates the new definition well.\n- **Clarity:** The big picture is clear: depth stabilizes separation, and readouts enable universal approximation within the correct class. The relation to prior work is well positioned.\n- **Significance:** Results inform model design (when to use a readout layer and how deep to go). They also explain why some shallow models fail even if they have the same separation power.", "weaknesses": "- **Lack of assumption:** Theorem 1 (and also other theorems) fails, for example, if $\\sigma = {\\rm id}$.  \n- **Sparse definitions:** The permutation representation and Eq. (3) are only informally presented. Adding a clear algebraic and matrix-form definition would remove ambiguity.  \n- **Scope:** The analysis is limited to permutation representations and point-wise activations; potential extensions to other representations are only mentioned briefly.  \n- **No quantitative results:** The paper proves existence of universality but gives no approximation rates or bounds on the required depth threshold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "seUXXjeQWg", "reviewer": "Reviewer_gzFd", "summary": "The authors show a separation-constrained universality theorem for invariant networks, and they extend the result to the case of equivariant networks by introducing the notion of entry-wise separability.", "questions": "- In theorem 1, how large can $k$ become? In the standard neural network case, we can obtain arbitrary well-approximated function by increasing the number of the hidden layers. How does $k$ depend on the input dimension and $|G|$?\n- Similarly, in Theorem 2, can we determine optimal value of $d$? Since $M$ depends on the functions $\\phi^1,\\ldots,\\phi^k$, are there any relationships between $d$ and $k$?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors theoretically investigate separation-constrained universality for invariant and equivariant settings. They introduce a notion called entry-wise separability and show a connection between the depth of the neural networks and the universality. The results are solid and the topic is interesting.", "weaknesses": "The notation should be consistent. For example, $h$ represents a natural number in some cases, a function in other cases, and an element in $G$ in other cases. Since this paper focuses on theory, the consistency of the notation improves the readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "35FCGQFNbO", "reviewer": "Reviewer_hN6i", "summary": "This paper addresses the theoretical foundations of universality in equivariant neural networks, a topic for which rigorous results remain limited. Existing universality theorems often rely on highly restrictive assumptions—such as regular or higher-order tensor representations that induce prohibitively large hidden dimensions, or architectures that are restricted to the invariant case. The authors develop a more general and unified theoretical framework that encompasses both invariant and equivariant networks. For invariant architectures, they prove a universality theorem under separation constraints, showing that the inclusion of a fully connected readout layer enables universal approximation within the class of separation-constrained continuous functions. For equivariant networks, where existing results are even rarer, the paper identifies a key limitation of standard separability assumptions and introduces the concept of entry-wise separability, a stronger and more suitable criterion. The authors prove that equivariant networks achieve universality within this regime, provided they have sufficient depth or appropriate readout layers. Together with earlier results showing the failure of universality for shallow models, this work highlights depth and readout structure as fundamental mechanisms for universality. The framework also unifies and extends prior specialized results, offering a clearer theoretical understanding of how equivariant architectures can approximate complex functions while maintaining symmetry constraints.", "questions": "Here are reviewer-style questions that incorporate your points:\n1.\tGeneralization error and sample complexity. Your results are asymptotic in function approximation. Can the framework yield generalization bounds—e.g., via Rademacher or covering-number control of the neural spaces—expressed in terms of the entry-wise separation profile, the depth at which separation stabilizes, and the size/structure of the readout? In particular, can you derive sample complexity or margin-based bounds that certify when added depth/readout improves test error rather than only representation capacity?\n2.\tFunction spaces beyond continuity. Universality is stated over continuous functions. Can you extend the results—or obtain rates—for other spaces such as L^p, Hölder/Sobolev/Besov, BV, or Barron-type spaces? Is entry-wise separability compatible with norms/seminorms in these spaces so that one can state approximation theorems with explicit error decay under smoothness or spectral assumptions?\n3.\tConnections to integral representations of shallow nets. Classical approximation theory links shallow networks to integral representations (ridgelet transform, Barron spaces, Radon-type decompositions). Is there a formal relationship between your entry-wise separability (or separation stabilization with depth) and the spectral measures that appear in ridgelet/Barron analyses? For instance, can depth plus an appropriate readout be interpreted as enforcing or approximating certain ridgelet spectra or moment constraints, and do your results recover known rate results in Barron regimes or suggest new ones for equivariant settings?\n4.\tOperational criteria. Practically, how can one estimate or test entry-wise separability (and its stabilization depth) from finite data or from an architecture’s design (e.g., filter classes, tensor orders)? Are there computable proxies (e.g., spectral norms of intertwiners, orbit counts, WL-width analogs) that permit a-priori generalization guarantees or architecture selection?\n5.\tRobustness and misspecification. Can your theory quantify how approximation/generalization degrades under model misspecification (e.g., when the true target violates the assumed separation relations slightly), or under noisy labels/inputs? Are there stability bounds for the entry-wise notion analogous to Lipschitz-type robustness?\n6.\tFrom existence to construction. Your theorems are existential. Do they imply constructive schemes (e.g., explicit readout designs or depth schedules) that achieve provable approximation rates and generalization bounds in the above function spaces, perhaps by leveraging ridgelet-inspired initialization or kernel/NTK limits adapted to entry-wise separability?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper tackles a compelling and important direction—building theory for symmetry-aware (equivariant/invariant) neural networks—and does so with clarity and breadth. The authors provide a unified, architecture-agnostic framework that cleanly separates the roles of separation, depth, and readout layers, resolving long-standing confusion about why models with identical separation power can differ in approximation power. The introduction of entry-wise separability is a crisp and genuinely new lens that both diagnoses the failure of standard separability to characterize equivariant universality and supplies the right notion under which strong theorems can be proved. The results are conceptually satisfying and practically meaningful: for invariants, a fully connected readout restores universality within the separation-constrained class; for equivariants, universality emerges either after finite depth stabilization or via specific output layers that act as equivariant analogues of readouts. These statements not only subsume and extend prior specialized results but also give a reassuring message for practice—maximal expressivity is reached at finite depth, avoiding the specter of unbounded “depth chasing.” The presentation is mathematically careful (clear definitions, clean statements, explicit assumptions) and the work advances the field by turning an intuitively appealing idea—that symmetry should guide learning—into a rigorous set of principles that explain when and how symmetry-respecting networks are universal.", "weaknesses": "While the paper offers a clean and unified treatment, its contribution risks being perceived as incremental given a substantial body of prior work on universality for symmetry-aware models (invariant and equivariant networks, WL-based expressivity, PointNet/graph settings, and regular-representation proofs). The central message—that depth and readout layers restore universality under separation constraints—resonates with, and in places refines, known intuitions; however, the practical novelty over existing universality theorems may feel limited without sharper separations or new, unexpected consequences. Methodologically, the results hinge on point-wise activations and permutation representations under finite groups, leaving unclear how far the theory extends to continuous/compact Lie groups, non-pointwise nonlinearities, or mixed representation stacks that are common in modern E(3)/SO(3) models. The proposed entry-wise separability is mathematically convenient but may be difficult to verify or estimate in practice, and its relationship to measurable statistics used by practitioners (e.g., tensor orders, message-passing depth, spectral gaps) is not yet operationalized. The universality guarantees are asymptotic and do not provide approximation rates, sample complexity, or robustness to model misspecification—key levers for assessing when added depth/readouts actually help. Moreover, the theory presumes or requires stabilization of separation with depth but offers limited guidance for identifying the stabilization threshold for concrete architectures. Finally, the paper is purely theoretical: without constructive recipes, empirical probes, or optimization analysis, it remains uncertain how the results translate into trainable, parameter-efficient designs under real-world constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)