Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
Authors: Yuzhu Chen
Tian Qin
Xinmei Tian
Fengxiang He
Dacheng Tao
Venue: Accepted by TMLR ()

Abstract: Equivariant neural networks encode the intrinsic symmetry of data as an inductive bias, which has achieved impressive performance in wide domains. However, the understanding to their expressive power remains premature. Focusing on 2-layer ReLU networks, this paper investigates the impact of enforcing equivariance constraints on the expressive power. By examining the boundary hyperplanes and the channel vectors, we constructively demonstrate that enforcing equivariance constraints could undermine the expressive power. Naturally, this drawback can be compensated for by enlarging the model size -- we further prove upper bounds on the required enlargement for compensation. Surprisingly, we show that the enlarged neural architectures have reduced hypothesis space dimensionality, implying even better generalizability.

Published in Transactions on Machine Learning Research (04/2026)
Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
Yuzhu Chen
cydrau@maillsstc.ee...rr
University of Science and Technology of China
Tian Qin"
tgin@@mail.uttc.edu....
University of Science and Technology of China
Xinmei Tian
Sinmei@rust.eddu.cr
University of Science and Technology of China
Fengxiang He
F. H.HeWed.ac. uk
University of Edinburgh
Dacheng Tao Nanyang Technologioal University
dacheng. tao mao@ntu.edu.co
Reviewed on OpenReview: https: // openreviee net/ forum? 25bJ44Bre4
Abstract
Equivariant neural networks encode the intrinsic symmetry of data as an inductive bias, which has achieved impressive performance in wide domains. However, the understanding to their expressive power remains premature. Focusing on 2-layer ReLU networks, this paper investigates the impact of enforcing equivariance constraints on the expressive power. By examining the boundary hyperplanes and the channel vectors, constructively demonstrate that enforcing equivariance constraints could undermine the expressive power. Naturally, this drawback can be compensated for by enlarging the model size we further prove upper bounds on the required enlargement for compensation. Surprisingly, we show that the enlarged neural architectures have reduced hypothosis space dimensionality, implying even better generalizability.
1 Introduction
In scaling up the capacity of machine learning models, leveraging the intrinsic symmetry in data, if it exists, would be a much more efficient approach than simply scaling up model size; see good examples in point clond processing (Qi et al., 2017 Li ct al.
2018 Fuchs ct al. 2020 Chen et al.
2021 visual tasks
(Cohen Welling 2016 Xu et al., 2023), graph
neural networks Veličkovic al.
Kanalsoulis & 2016 Eismann al. 2021 Kondor 2025 reinforcement
2017
Ribeiro (024)) physics and chemistry (Faber al.
When learning these symmetries, classic methods must be shown numerous transformed (sometimes manually) data examples, such as rotated or translated images, and be enforced to make saariann//quivariantt predictions, which is usually computationally expensive and requires vast amounts of data (Bronstein et al. 2021). Equivariant noural networks address those challenges through cither new architectures or optimization mothods to enforce symmetries (Cohen & Welling 2016), which a useful approach is to enforce each layer of the werwork be equivariant.
learning and decision making (Wang al. 2022
Qin et al. 2022) amongst others.
Despite the impressive performance, it remains unclear whether enforcing equivariance comes at the cost of reduced expressive power. This paper investigates this trade-off by analyzing whether restricting the
Botth authors contributed equally.

---

Published in Transactions on Machine Learning Research (04/2026)
Figure An equivariant function that satisfies s((a, b) )- s((b, T) and b((a,b)T) s((-a, D) The left subfigure is 3D plot, while the right subfigure 2D Contour map.
hypothesis space to equivariant models increases the expected risk when approximating an equivariant target function sxx). Specifically, we examine whether the following inequalities hold:
AEO inf Exzr[1]Fool -
inf REenGEN:
Be-r[1|Fo(e s(x)
and BEBNGENs
inf
Bx-rllFo(o)) s
inf E [ s(x)
#EenLENs
Here, @ is the set of network parameters (usually L matrices) and Fe is the neural network with parameter 0. We consider two types of equivariant networks: GENs (General Equivariant Networks), where the global mapping x + Fo(x) is required to be equivariant, and LENs (Tayer-wise Equivariant. Networks), a stricter subset where equivariance enforced at every layer the network. In practice, constructing LENs is a common approach to achieving GENs (He al. 2021
Boundary hyperplancs and channel vectors. We first investigate how these equivariance constraints (imposed at the level of input-output mapping) fundamentally restrict the architectural structure of neural networks. For two-layer ReL.U networks, we focus the geometric properties of the boundary hyperplanes and channel vectors of equivariant networks. We prove that GEN requires boundary hyperplanes to be symmetric. While for LENs, we demonstrate constructive property: any LEN can be reformulated into another L,NN that maintains the same model size but possesses a symmetric set of channel vectors. These geometric rigiditics serve as the foundation for our subsoquent analysis, enabling us to quantify how much model size enough for approximating a larget function.
Drawback of enforcing cquivariance. Following the insight of the above analysis, we demonstrate a scenario where equivariance strictly hurts the expressive power. Specifically, we construct an invariant target function R² and establish two key findings: (1) for two-layer ReLU networks with a single neuron, general networks (GNs) achieve strictly lower expected error compared to general equivariant networks (GENs), and (2) for two-layer ReLU networks that can arbitrarily approximate the target function, layer-wise equivariant networks (LENs) require a strictly larger model size than GENs.
Compensations via enlarging model size. Inversely, we prove that an increase in model size can fix the compromised expressive power. When the target function is invariant, and the data distribution is symmetric, both in terms of group G, enlarging the model size any equivariant network by |G] times can compensate for the expected loss over a 2-layer neural network. Moreover, doubled model size allows LENs to represent all invariant functions of GENs. We further prove that the constructed LEN with [G] times
2

---

Published in Transactions on Machine Learning Research (04/2026)
enlarged model size surprisingly has a less complex hypothesis space, implying even improved generalizability. Together with the comparable expressive power, this result renders LENS a competitive solution.
To the best of our knowledge, this paper provides the first in-depth theoretical examination of the expressive power of layer-wise equivariant neural networks, demonstrating that enforcing equivariance has drawback in expressive power. We also show that an appropriately larger model size can compensate for the loss expressive power and potentially enhance model snteralliaablityyy underlining the advantages of incorporating layer-wise equivariance into model architectures.
2 Related Works
Enforcing equivariance. Typical approaches for obtaining equivariant neural network are (1) design-
ing oquivariant architectures (Cohen & Welling 2017 Satorras et al., 2021
Trang
et al. 2024 and (2)
introducing equivariance constraints into the optimization (Winter et al. 2022 Tang 2022
Pertigkiozoglou
et al. 2024
Equivariant neural architectures. Cohen & Welling 2016); Shin et al. (016 design group convolu- tional neural networks in the group spaces as the equivariant variants of the convolutional neural networks, which is extended to the homogeneous spaces by [Cohn et al. (2019). Similar approaches have also been employed in the design of other popular equivariant networks, such as graph neural networks (Klicpera et al.,
2020 Aykent & Xia 2025
transformers (Hutchinson et al. 2021 Islam et al. 2025), and dilfusion models
[Hoogeboom et al.
2022
Wan et al. 2025) Hao et al. 2025) leverage permutation equivariance in vision
transformer for producing "human "human-impents imperceptible, machine-recognizablee images.
Symmetry-aware optimization. In this stream, equivariance constraints are introduced in optimization, as either regularisers or constraints, to ensure the equivariance, usually termed as steerable neural networks (Cohen Welling, 2017). Other good examples are about enforcing the symmetry group SN in steerable graph networks (Maron et al. 2018), the rotation and Euclidean groups (Cx, in stcerable convolutional neural networks (Weiler et al. 2018 Weiler & Cesa 2019). and Lie symmetry in differential equations Sachound-Sadegl) et al. 2023 Jiang et al.)
2025 A major obstacle is the high computational costs to
[2021 address this issues with an algorithm with polynomial computational complexity on the sum of the number of discrete generators and the dimension of the group, which divides the problem into some independent subproblems and adopts Krylov method to compute nullspaces.
compute the equivariant basis Finzi et al.
Theoretical analysis. Theoretical studies have shown that equivariant neural networks have siguificant advantages, in terms of generalizability, convergence, and approximation. [Sannai al. (2021) prove im- proved generalization error bounds of equivariant models. Qin et al. (2022) leverage orbit averaging to construct a projection mapping from the original hypothesis space to the permutation--qquivllent counter- part, underpinning reduced hypothesis complexity and better generalizability. Lawrence et al. 2022 prove
that linear group convolutional neural networks trained by gradient descent for binary classification converge to solutions with low-rank Fourier matrix coefficients based on the results via implicit bias. Zaheer et al.
(2017): Maron et al. (019);; Yarotsky 2021 Pacini et al. (0255 prove the universal approximation ability of equivariant models by various approximation techniques, while Ravanbakhsh (2020) utilize group averaging to obtain an equivariant approximator prove the universal approximation ability. In addition, Elesedy & models have smaller expected loss when the target function is invariant/ anvariant/equivariantt equivariant. Azizian & mare lelarge (2021) study the expressive power of invariant eevariant/equvvee /equivariant graph neural networks. On the other hand, Pettrche & Trivedi (2023) focus on another approach that relaxes the strict constraints to partial and approximate equivariance, and provide bounds on the approximation error and generalization error. In contrast to the aforementioned studies, this paper focuses more on the goometric propertics of ReLU networks and their expressive power. We prove that with the same model size, equivariance constraints impair expressive power. For the observed benefits of equivariance, we attribute them to the reduced hypothesis complexity, which holds even if the model size is larger.
Zaidi 2021 prove that invariant/equivea
3

---

Published in Transactions on Machine Learning Research (04/2026)
3 Problem Settings
General networks (GNs). GNs refer to general neural networks without any design for realizing the oquivariance. In this paper, we focus on two-layer networks, with a hidden layer of width m (also the number of neurons), representing mapping from Ik -> IR. The network is parameterized by 0, including the weight matrix W(ll) e Rmx* for the hidden layer, and the weight matrix W(2) Rdxm for an output layer. With the popular nonlinear activation ReLU a, the GN maps any imput E IK" to the output: F(x)= = W (W(()) For the brevily, we denole by og the i-th column of W(1), by By the j-th row of W((2 and w(), and collect these parameters (with m neurons) into a set Om. In this formulation, the output of a GN can be formulated as follows:
F F(x) = ( W
E ()),
(1)
i=1
where (a, 2) is the inner product axx. We note that the total number of parameters in the GN is nm + md. As this count is proportional to the hidden layer size m, we use m to characterize the model size.
General equivariant networks (GENs). GENs represent a subset of GNs that constrain the output function F to be equivalent with respect to a given group representation. A group representation for a group G formally defined a homomorphism p: G GL(m), where GL(m) is the group of all invertible matrices Rmxm A function F is considered equivariant if, for given imput and output representations p and & respectively, it adheres to the condition F F Py o, o F for all group elements g G. A GN whose function F satisfies this property is categorised as a GEN. Specifically, when ø id, the function F is called an invariant function and the networks are called invariant networks. In this paper, we assume that group G is finite and focus on invariant functions.
Liser-wise equivariant networks (LENs). LENs represent a subset of GNs that constrain each layer to be equivalent with respect to given group representation. Specifically, given the group representation tb in the hidden layer, LENs require that for all E G,
0g (()) = (22) o bhy
v 00 = o o and bg o W = w(1)) a Pg.
As each layer is constrained to be equivariant to maintain the symmetry in the data, the whole network is equivariant. Specifically, for every group element E G, we have
F Pg w o ((!) o Pg o o W(!)
oooo,
( o bhy oo W((1) = og ( ooo w(1) = 6, o F.
In practice, constructing LENs is a common approach to achicving GENs He et al. 2021
Expressive power. The expressive power, of GNs, GFNs, and LENs is defined the minimum expected loss for approximating target function Naturally, smaller minimum expected loss means better expressive power. Since GENs are in a subset of GNs, and LENs have inluitively more constraints, we have the following straightforward results: for any target function s, any input distribution P E AR", and any number of neurons m,
inf Ez-r(]FF((x) s(x) S Bge..GNN
Ex (1F -())))) S inf eEe..GEN:
inf
Em-P(l (()))
DeemLLENs
where 9 represents the parameter of a two-layer RcLU network =0= = (W(2), W("))), and Fg is the forward function given parameter 0, and O,, is the parameter set of ma-hidden-layer networks. We reuse the notations GNs (also for GENs and LENs) to denote the respective parameter sets that makes Fe GN (also for GEN and LEN). This paper investigates whether equivariance constrains the expressive power of two-layer networks. Specifically, we study whether there exists some equivariant function and m such that
inf BGN.. Ea (x) - s(x)11 Dee.nGEN:
inf Eg-r [|] Fo(x) - s(x)]12) and
inf E (o) s(x) inf Ezwr [( s(x)
9Ee,,nGENs
,ee,nLEEN e nLENs
4

---

Published in Transactions on Machine Learning Research (04/2026)
Figure 2: A visualization of the feature function F of F(x, v) o(x) + -o(-y) + o(-x + y), where the left subfigure is Fi and the right figure is F2. As shown, there are two boundary hyperplanes: = 0, 0, = x.
The first inequality studies whether being equivariant is necessary for being the optimal approximator if the target function is equivariant. The second is on whether enforcing layer-wise equivariance the optimal way compare the expressive power of GNs, GENs,
to incorporate equivariance. To ensure fair comparison, and LENs under the same architecture (e.g., the same weight-sharing scheme) and the same model size. We also note that it is reasonable to assume the target function to be equivariant, because equivariant networks are designed for equivariant lasks.
4
Boundary Hyperplanes and Channel Vectors
To establish a clear foundation for our analysis expressive power, we discuus two tools, symmetry boundary hyperplanes and symmetry channel vectors, in this section.
4.1 GENs Imply Symmetric Boundary Hyperplanes
Naturally, ReLU networks are also piece-wise linear, given ReLU functions are piece-wise linear:
m
m
F(x) -
Σ Bio ((a;,
Σ
3 x
Σ
an x = (F(x),)),
(2)
i(ooa)20
Vil(o,a))0
where F(x) = Ditan T is the feature function of F and is piece- wise constant.
We define the boundary hyperplane as the hyperplane whose feature function values are almost surely different on its two distinct sides. For simplicity, we assume that all hyperplanes contain and have dimension n - 1. The formal definition is shown below.
Definition 4.1 (boundary hyperplane). Let F be an oulput function of two-layer ReLU network and F(x) be the corresponding feature function. We call hyperplane M boundary hyperplane if and only if for all E M and & M:
lim F(x + 8v) # lim F(x 8u). 5-+0+
6-20
It not necessary define the feature function F on the boundary hyperplanes. For any given function
F, define it only the set S (x F is linear in B(x,r), 3r > 0), where B(r,r) {xr

---

Published in Transactions on Machine Learning Research (04/2026)
||||| < rt is the open ball centering at with radius The value of F(x) is determined uniquely by the Riesz representation theorem. The complementary set S is the union of all boundary hyperplanes, whose intersection with the hyperplanes M = óv. For all x E S that satisfies (I, as) # 0 for all i E [m/] F(x) must be - B.a.. Therefore, if the feature function changes when crossing through the same hyperplane, the active set {i: az) 0} must change and the increment of the feature function is Eitor.. 8:a which is then summarized Theorem 4.2 channel vectors.
This theorem can be viewed as equivalent definition by the
Theorem 4.2. Let F = Ei= (,)((a))))) x)) be an output function. Then, hyperplane M is boundary hyperplane if and only for all v & M:
E
agft +
Σ
ag
(a, M)
((a.)) M))
where (a) are defined as (a,, 2) for all x E M. Especially, if M is boundary hyperplane, there ewists some channel vector such that (a, M) 0.
Boundary hyperplane provides powerful analytical tool for understanding the internal mechanisms of GENs. We prove that for a GEN, the set of its boundary hyperplanes is symmetric with respect to all group representations. Specifically, for any given boundary hyperplane M, then for any E G, the transformed hyperplane PyM = {p x E M] also a boundary hyperplane of the GEN, because
lim F(p + 6p.u) = (pg)) lim F(x ov) + (og))= lim F(r - -u) = lim F(pyr 0p,u). 8-00
8-0+
8-0+
á 6-00
where x is a point on the hyperplane M and v is any vector not in M. The inequality holds because the group representation is always invertible. We note that M is not always another hyperplane. In fact, for the identity g = P = M.
This property hold for all group elements g E G, implying the of all transformed hyperplanes, {py : e G), is a subset of the GEN's total set of boundary hyperplanes. Tt also holds for all boundary hyperplanes M, which means the GEN's total set of boundary hyperplanes is symmetry under p. We sunmarize this finding in the following theorem.
Theorem 4.3. For GEN on group G, ils boundary hyperplane set M is closed under group representation Pg for all G.
This theorem has direct implications for the expressive power of GENs. When approximating target function, the network must capture its discontinuous, i.e., boundary hyperplanes. Due to the symmetry requirement, even a single learned hyperplane M generates a whole family of hyperplanes, {pgM g E G}. The size of this set provides a lower bound on the number of boundary hyperplanes the GEN must use to approximate the target function effectively. According Theorem 4.2. larger number of required hyperplanes, in turn, necessitates greater number of neurons in the network.
4.2 LENs further Imply Symmetric Channel Vectors
LENs represent more constrained subset of GENs, enforcing symmetry the layer level rather than across the network a whole. Although the symmetry property for boundary hyperplane is fundamental property of all GENs, the stricter conditions imposed by the LEN structure provide deeper insights. At least, mentioned in Section 3 LEN needs one admitted group representation and two weight matrices in the corresponding intertwining space.
We first consider the property of admitted group representation, i.c., o o db This commutation property is not. trivial and places a significant restriction on the form that the group representation yp can take. Specifically, for RcLU, the following lemma provides a complete characterization of all such admitted representations.
Lemma 4.4 (Godfrey et al. 2022]; Bökman & Kahl 2023 ) For ReLU, group representation tb
is admitted if and only for all g G, Vy is generalized permulation matrir with exclusively posilive

---

Published in Transactions on Machine Learning Research (04/2026)
non-zero entries. That is, each row and each column v must contain ezactly one non-zero element, and that. element must be posilive.
Proof. We first prove that each row of tig has at most one non-zero entry for all g E G. Assume for contradiction that exists g E G where row of 4h has at least two non-zero entries, (1g)) and (Vy)ik Without loss of generality, we assume (4))a I (#g)ss- Admitted & means that for input vector (z))(());
Cj - Ck: (tiee)) (#))..- However, the other side we have (o((g
we have (o(())z))
((Wgg)) (vg)x)) O, which contradicts our initial assumption that the entries were non-zero. Therefore, each row of vi contains at most one non-zero entry.
Each row has exactly one non-zero entry because by is invertible, which also means each entry has at least one non-zero row, making vi a generalized permutation matrix. Morcover, when (%g))s o 0, we can set = Ej. Then (V)a))) (x)) implies that (1bi)) vig- From the definition of ReLU, (V)g)sj is positive for all i.
Lemma implies that the action of any 4i on a vector is a composition of a permutation and a positive scaling its coordinates. For each g E G, we can identify a unique permutation Pg that maps the coordinate indices. Specifically, (i) is the index of the only non-zero element in row i of matrix v.g A special case of admitted group representation occurs when all non-zero entries of 4/ is 1. In this scenario, each bh is standard permutation matrix. We refer to this to as permuting representation.
Following the property of admitted representation, we now examine how it interacts with the other constraints of invariant LENs. The final linear layer, represented by W(2) = [8,, Aml,, is required to satisfy the condition (2(2 &o = (2) for LENs. This requirement reveals a doeper constraint on up and the geometry of (2)), Specifically, we have the following lemma:
Lemma 4.5. For admilled 1/ in an invariant LEN and any connected pair of indices (i.j) (erists g E G making (i) = j), we have ||3,118, = 113,113/. Besides, all h E G with Pr( = j shares the same positive (1b))sg
Proof. Consider a pair (i.j) that makes thiy + 0 for some * h. Admitted vb means that for input ej, we have By = W = W(P) byy W()), e, = 3 for all v, including v. and vhn It shows that Bg and By share the same direction, meaning [|:3, [8 = | It also shows that thi does not change if tb switches from th to u/,, i.e., (15))s (Vn)as-
In light of Lemma 4.5 we say B and B3 are in the samn orbit if they share the same direction, i.e., |13y11 || The lemma demonstrates that the symmetry constraint requires all weight vectors in the same orbit to lie in the same direction in vector space.
Aside from the constraint on W LENs also impose a condition on W W(1) should belong to the intertwiner space defined by and p. Combining it with the constraints on sb and W(2), we provide the following theorem.
Theorem 4.6. For any p-invariant LEN Fe, there exists an equivalent p-invariant LEN, Fg of the same size that computes the ezact same function (Fg(x) Fo(x)) for all z), where the channel vectors set of the rewritten network ((āz)) is closed under the action of the transposed group representation pT. That is, for any channel vector āj and any group element g G, pg â is also channel vector of F.g
The proof is included in Appendix D The theorem means that we can rewrite any LEN into another LEN of the same size, where the channel vectors of the rewritten LEN exhibit an explicit symmetry in pT. In the view of input-output mapping, the rewritten LEN is equivalent to the given LEN. For the rewritten LEN, the requirement of symmetry channel vectors means that one single channel vector ā; implicitly defines a subset. of channel vectors (0, a. : g E G}. The net work size is then smaller and bounded to accommodate all these channel vectors.
We note that this constraint stricter than the one for GENs. A GEN requires the symmetry for bound- ary hyperplanes while LEN requires that for channel vectors, where one boundary hyperplane has to be
7

---

Published in Transactions on Machine Learning Research (04/2026)
determined by at least one channel vector, but different channel vectors could lead to one hyperplane. For instance, the channel vectors a and -a are distinct vectors but define the exact same boundary hyperplane. A GEN has the flexibility to use scaled versions of the same vector normal, whereas a LEN, in its symmetric form, must treat each parallel vector as a distinct channel vector.
5 Drawback of Equivariance in Expressive Power
In this section, we show our results on how enforcing equivariance hurts the expressive power. The proofs are through constructing a target function on TR.2 + R. When approximating this target function, we show that for neural networks with one neuron, i.e. in parameter than GENs. For n-nwuron-netwoorss, GNs and GENs can fully approximate this target function, while LENs cannot.
11, GNs have strictly smaller expected loss
Example 5.1. Consider group G consists of 4 elements e,g, h, b.gh where g2 = 112 and g h h gh. One group representation is as follows
-1
Pe
1
Pg
0
Ph
0 -1 and Pgh -1 0
Given representation consider target function 8:
R
(a,b)* max([a), 161, la 61),
which is an invariant target function because
a
6
max( |a|, 16 al) =(
(a) and
SO Pg
b
0
Ph
-b)
max(1 al. b/. 1b al)
We note that has another form: (((a,b) (a)) + (-b) abb-a)) a).
Figure D provides a 3D plot and 2D contour map to visualize the effects of the imposed symmetry. Figure displays the feature function of the symmetry group element. Building on this example, we demonstrato how the constraints of GENs can limit expressive power and how this limitation further pronounced in LENs. Specifically, we mainly focus on comparisons of GENs vs. GNs on the 1-neuron setting, and LENs vs GENs on the 3-neuron setting.
One-neuron networks. The output function of the neural network becomes Ba((a, x)). A typical one- neuron network applics the ReLU activation function to each dimension of the input. Formally, when (1, 0) and B = 1, F((a,b) T) o(a) is an approximation and F is GN with one neuron. Similarly, when a 10, and 3 = 1, F((a, T) ) o(b) is also a GN approximation. Thus, for all input distribution P with Pra-p|I7 = (0,0) 0)] < 1, we have
inf #EBinGNs
() sx)) < min Em-rla(() , (e-rlo(()) s(x)11) < Eb-r(1((2)11
(3)
For GENS, we have fpxx) = f(x). Combining with the definition of feature function F, we have (F(oz), px) =
(p" F(pr). = (F) which implies p" F F for all p. Let p = Ph. we have for all
Bo(la, F(x) (pT Fo pz)()) -Bo((a,-22
where al least one of x)) and o((a, -x)) is zero, indicating F = 0 for all x. It shows that the only one-neuron GEN zero, which leads to strictly lower expressive power then GNs. Specifically, we have that for all input distribution P with Pr [I= (0.0) 1,
inf (l) s(x)11) E1> inf 0c
El) s(x)11]
(4)
0eGinGEN:
8

---

Published in Transactions on Machine Learning Research (04/2026)
Three-neuron networks. Three-neuron GNs can represent defined in Example 5.1 because the form (((abb))) aaa) (-b)))))))))))) + +a(b- a) is already three-neuron network. Due to the equivariance property of 8, this network is also a GEN. Therefore, three neurons are sufficient for both GNs and GENs to achieve s.
However, for LENs, we show that three neurons are not enough. According to Theorem 4.6 for any LEN that represents 8, there exists a rewritten LEN with symmetric channel vectors of the same size that also perfectly represents
Since s has three boundary hyperplanes: x 0, y 0, and - y = 0, they should also be the boundary hyperplane of the rewritten LEN, which implies the existence of at least three channel vectors: e1 (1, 0), c2(0, 1), and c)(1,1) with c102c3 # 0. Since the channel vectors are symmetric, P c1, (1,0) = -c (1, 0)also forms channel vector; as do (0, and -c,(1,1) As result, the original LEN must contain at least six channel vectors, i.e., LENs require at least six neurons to represent s. Finally, we show that six neurons are sufficient, as the following LEN has exact six neurons and represents s:
1 Fo((a, b) - + + + a) ((aaaa)) -b].
6 Compensation to Enforcing Equivariance
In the previous sections, we demonstrated that for a fixed model size, the expressive power of LENs are strictly lower than that of GENs, which are also strictly less expressive than GNs. A natural question arises: can this limitation in expressive power be compensated for by enlarging the model size? This section affirms that cnlarging the model size is indoed a viable solution. We will cxamine this through both realizable and non-realizable cases. Further, we show that, surprisingly, the enlarged networks can have even reduced hypothesis complexity, which leads to better generalizability.
6.1 Realizable Case
We first consider the realizable case, where the target function can be perfectly represented by GN with finite number of neurons. By definition, also be represented by finite-neuron GEN, When it comes to LEN, the conclusion is not straightforward.
Firstly, we show that with a larger model size, there always exists a LEN that can represent any representable mapping of finite-neuron GENs, To demonstrate this, consider target GEN-representable function defined as a(x) = -m ((a((() x)). We can construct a LEN, denoted by F(x), that perfectly replicates s(z). The weights for this constructed LEN are defined follows:
(2 = G
Br
, and
|G|
W = 1p' a..... Poja
pm Am..... Paaa aml
This construction ensures that the weights adhere to the necessary equivariance conditions. In the first layer, for all E G, Pa. (ax, Pa. (x)) = (pTP Pg. 2) = (o ak,z), implying the existence of a permutation tb that maps the neuron index corresponding to Po. to the index for PT 0 This 1a is admitted because of Theorem 4.4 In the second layer, we have that the is for all p oi, indicating its invariance with respect to V.
This constructed LEN is rewritten version of because of its invariance and equivariance; specifically,
F(x)
σ (e] ai,x))
gEG
3:0 ((as:Py2))
gEG
(( = s(x),
G
gEG
9

---

Published in Transactions on Machine Learning Research (04/2026)
which confirms that our constructed LEN perfectly represents the target function This result can be extended to handle any equivariant functions and multi-layer architectures, detailed in Appendix F
Building on this, we utilize the relationship between the number of boundary hyperplanes and that of channel vectors to study how many neurons are required to let LEN approach GEN. For GENs or LENs, the number of channel vectors is larger than the number of boundary hyperplanes, according to Theorem 1.2 Inversely, for e GENs, if there are boundary hyperplanes of some output function k, we can construct some specific 3 = and a; for i E [k] such that f- - -i=1 (a., has no boundary hyperplanes, which then can be represented
compress neurons to make the number of channel vectors equal to the number of boundary hyperplanes plus two.
linear function (ag, æ) = (((00, =)) a((-ao( z)). Therefore, for GENs,
As for LENs, we prove the lollowing lemma that deals with all symmetric channel vectors as a whole. Lemma 6.1. Let 3 8: Egec olgg oi, x)) be on output function of LENs with as + 0 for all [6]. Denote M.- = (0g au/lle of aill =g E G} as the normalized orbit induced by 04. Then, we have Mj = M, or Mj M Mj- = & for all i.j E [q. Moreover, if M4 = -Mi, we have
Eo(loaas x)) 2)) Kig Σ Eolloay,.
gEG
gEG
d,,z)) such that M, n M, = & for all i,j E [e]. Furthermore, for each Mr, we can then merge all parallel channel vectors, since
which enables us to reformulate the output function as Ei E. o (19) (pp],,,))))
Eo a x)) |Stab(ā.)
Σ
gEG
gEG/Stabla.)
where Stab(a) {g E G pga a} is the stabilizer subgroup with respected to a. As result, in the EpEc/staMa.) o(pp
equivalent form Ei= B/
x)), there not two channel vectors of the same
direction.
we can construct LENs whose channel vectors are not parallel. Hence, the number of channel vectors is no more than double the number of boundary hyperplancs plus two. Therefore, the minimum required model size of LENs approximately summarized in the following theorem with some technical modifications.
According to Lemma 6.1
more than double that of GENs, which is
Theorem 6.2. LENs require al most double model size lo represent all invarianl oulpul funclions of GENs.
Theorem 6.2 indicates that enforcing layer-wise equivariance would be competitive among all methods of incorporating equivariance. We note that the conclusion of Theorem 6.2 can alternatively be derived from the work of Agrawal & Ostrowski (2022), where canonical form of GENs is defined, and doubling of model size makes each layer equivariant. In contrast, we present an independent constructive proof in Appendix G. which utilizes group averaging and the orbit merging of channel vectors. Our proof places greater emphasis on the intrinsic geometric properties of LENs, and this geometric perspective further underpins the analysis of expressive power for LENs in Section 6.2
6.2 Non-realizable Case
We now consider the non-rcalizable problem. With the same model size, enforcing equivariance may hurt the expressive power. We prove that when the networks have a larger model size, LENs may achieve comparable and even better expressive power. We start with the following theorem, which shows comparable expressive power when the model size is enlarged |G] times.
Theorem 6.3 (Elesedy & Zaidi (2021h). Let s be any invariant target function. Denote by Q the projection of f to the equivariant space, defined as Qf 2 GG Egec I Pg- If the data distribution is symmetric, then we have that
(II(f())-s()))) - s(x)) < B[Il(()) - s(x))
for all approzimator f. Specifically, if f is an output function of GNs, Of can be an output function of BENs/LENg with a |G] times model size, implying that GENs/LE /LENs can fir the expressive power gap and achieve smaller expected loss by larger model size.
10

---

Published in Transactions on Machine Learning Research (04/2026)
When the GN has an output function f(x) Ei=1 Bio((aa, the weight matrices of GENs/LENs can be formulated as
W =
31
B
,
and
G
|G]
G
w = l a......
Paia 01
Pgi am ................................ Pojci aml?,
which implies comparable expressive power while requiring a model size proportional to G Meanwhile, the weight matrices can be generated in two steps: (1) copy the same number of original channel vectors, and (2) generate all symmetric channel vectors to make the network equivariant. Therefore, although LENs have larger model size, they have the same number of learnable parameters. As result, the dimensionality of the hypothesis space be reducod by selecting fower original channel vectors by constraining them to subspace, while maintaining comparable expressive power.
If we use the method in Lemma 6.1 to compress the networks, we can further get an even better result. More prociscly, we can let the weight matrices be
w = B1, BI....,
, Smh,
and
C₁
Cm
W() = 1 01,
Paic, a1,... Pams Om: Pm 0ml
where ag constrained to satisfy Ho a G] = Cs, and Oúj is chosen to satisfy (o'., a : [Ci]] {p] a g G}. Similarly, this model still has comparable expressive power when targeting invariant target functions, but a lower-dimensional hypothesis space, since we constrain all first-layer channel vectors within (sub-)space.
6.3 Enlarged LEN with Smaller Hypothesis Complexity
While increasing model size allows LENs to achieve comparable expressive power to GNs, a natural concern is that this may harm the generalizability. In this section, we revisit Example 5.1 to demonstrate that despite requiring more neurons, LENs can still maintain a less complex hypothesis space than GNs. This lower complexity significant it suggests that LENs can achieve better gencralizability.
As discussed in Section LEN with six neurons and GN with three neurons represent the target function. With the permutation representation defined
0
1
0
1
0
0
1
0
1
4/%
1
0
0
1
1 0
Then the output function of the LEN can be formulated as
bl + c2y) o +(c2++0y)))
+bllo(-ca- czy) + c1y)
+bzla(e32 cay) a cay)].
From this formulation, space is RR. In contrast, threc-neuron GN is defined as Σ ( c2t), which involves 9 parameters, resulting in a larger parameter space of IR
determine that is characterized by five parameters, meaning its parameter
The key insight here that although LENs may have a larger model size to maintain the expressive power, their inherent structural constraints could lead to lower-dimensional hypothesis space. Consequently, this reduced complexity implies that LENs are expected exhibit superior generalizability compared standard GNs in this scenario.
11

---

Published in Transactions on Machine Learning Research (04/2026)
7 Conclusions
Equivariant neural networks leverage data symmetrics structural inductive bias, leading to significant success across various applications. Despite their popularity, their expressive power is not yet fully under- stood. This study evaluates how enforcing equivariance influences the expressive power of two-layer ReLU networks. By analyzing boundary hyperplancs and channel vectors, we show that these constraints can inherently limit model's expressive power. We demonstrate, however, that this limitation can be compen- sated for by increasing model scale, through formal upper bounds for the necessary expansion. Notably, our findings reveal that these larger equivariant architectures possess a reduced hypothesis space dimensionality, suggesting superior generalization potential.
References
Devanshu Agrawal and James Ostrowski. A classification of g-invariant shallow neural networks. Advances in Neural Information Processing Systems, 35:13679 13679-13669,, 2022.
Tara Akhound-Sadegh, Laurence Perrcault-Levassene Johannes Brandstetter, Max Welling, and Siamak Ravanbakhsh. Lie point symmetry and plysics-informed networks. In Advances in Neural Information Processing Systems, volume 36, pp. 92468-42455 2023.
Sarp Aykent and Tian Xia. Gotennet: Rethinking efficient 3d equivariant graph neural networks. In The Thirteenth International Conference on Learning Representations, 2025.
Waiss Azizian and mare lelarge. Expressive power of invariant and equivariant graph neural networks. In International Conference on Learning Representations (ICLR), 2021.
Georg Bökman and Fredrik Kahl. Investigating how relu-networks encode symmetries. Advances in Neural Information Processing Systems, 36:13720 13744, 2023.
Michacl M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges. arXiv preprint arXio:2104.1 134778 2021.
Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant, point network for 3d point cloud analysis. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning (ICML). PMLR, 2016.
Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations (ICLR), 2017.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cmns on homogeneous spaces. In Neural Information Processing Systems (NeurIPS). 2019.
Stephan Eismann, Raphael JL Townshend, Nathaniel Thomas, Milind Jagota, Bowen Jing, and Ron O Dror. Hierarchical, rotation-equivariant neural networks to select structural models of protein complexes. Proteins: Structure, Function, and Bioinformaties, 2021.
Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In Inter- national Conference on Machine Learning (ICML), 2021.
Felix A Faber, Alexander Lindmaa, O Anatole Von Lilienfeld, and Rickard Armiento. Machine learning energies of million elpasolite (a b 2 6) crystals. Physical review letters, 2016.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing oquivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning (ICML). PMI.R, 2021.
12

---

Published in Transactions on Machine Learning Research (04/2026)
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. In Advances in neural information processing systems, volume 33, pp. 1970- 1981, 2020.
Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetrics of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:11893 11905, 2022.
Fusheng Hao, Fengxiang He, Yikai Wang, Fuxiang Wu, Jing Zhang, Jun Cheng, and Dacheng Tao. Human- imperceptible, reconialee images. In International Joint Conference Artificial Intelligence, 2025.
Lingshen He, Yuxuan Chen, Yiming Dong, Yisen Wang, Zhouchen Lin, et al. Efficient equivariant network. In NIPS, 2021.
Emiel Hoogeboom, Victor Garcia Satorras, Clement Vignac, and Max Welling, Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning (ICML). PMLR, 2022.
Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. In International Conference Machine Learning (ICML). PMLR, 2021.
Mohammad Mohaiminul Islam, Rishabh Anand, David R. Wessels, Friso de Kruiff, Thijs P Kuipers, Rex Ying, Clara Sánchez, Sharvarce Vadgama, Georg Bökman, and Erik J Bekkers. Platonic transformers: A solid choice for equivariance. arXiv preprint arXiv:2510.05511 03..11,1,, 2025.
Xuelian Jiang, Tongtian Zhu, Yingxiang Xu, Can Wang, Yeyu Zhang, and Fengxiang He. Lie symme- try net: Preserving conservation laws in modelling financial market dynamics via differential equations. Transactions on Machine Learning Research, 2025.
Charilaos I Kanatsoulis and Alejandro Ribeiro. Graph neural networks are more powerful than we think. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7550-7554. IEEE, 2024.
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations (ICLR), 2020.
Risi Kondor. The principles behind equivariant neural networks for physics and chemistry. Proceedings of the National Academy of Sciences, 122(41):e241565612 2025.
Hannah Lawrence, Kristian Georgiev, Andrew Dienes, and Bobak Kiani. Implicit bias of lincar equivariant networks. In International Conference on Machine Learning (ICML), 2022.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. In Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint ar Xrx: 1,1,0000,,, 2018.
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International Conference on Machine Learning (ICML). PMLR, 2019.
Marco Pacini, Mircea Petrache, Bruno Lepri, Shubhendu Trivedi, and Robin Walters. On universality of deep equivariant networks. arXiv preprint STTT:2510011811, 2025.
Stefanos Portigkiozoglon, Evangelos Chatzipantazis, Shubhendu Trivodi, and Kostas Daniilidis. Improv- ing equivariant model training via constraint relaxation. In Advances in Neural Information Processing Systems, volume 37, pp. 83497-83520, 2024.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Mircea Petrache and Shubhendu Trivedi. Approsinatioo--eneallaaaaaa trade-offs under (approximate) group equivariance. Advances in Neural Information Processing Systems, 36:61936 36:61936-61955, 2023.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Tian Qin, Fengxiang He, Dingfeng Shi, Wenbing Huang, and Dacheng Tao. Benefits of permutation- equivariance in auction mechanisms. In Newral Information Processing Systems (NewrIPS), 2022.
Siamak Ravanbakhsh. Universal equivariant multilayer perceptrons. In International Conference on Machine Learning (ICML). PMLR, 2020.
Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invari- ant/ ant/equivariant deep networks via quotient feature spaces. In Uncertainty in Artificial Intelligence (UAT). PMLR, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian Fuchs, Ingmar Posner, and Max Welling. E(n) equiv- ariant normalizing flows for molecule generation in 3d. arXiv preprint arXie:2105.0000 2021.
Hoo-Chang Shin, Holger R Roth, Mingchen Gao, Le Lu, Ziyue Xu, Isabella Nogues, Jianhua Yao, Daniel Mollura, and Ronald M Summers. Deep convolutional neural networks for computer-aided detection: Cun archilectures, dalaset characteristics and transfer learning. IEEE trasaclions on medical imaging, 2016.
Junqi Tang. Equivariance regularization for image reconstruction. arXiv preprint orXim:2202.05002 2022.
Thuan Anh Trang, Nhat Khang Ngo, Daniel T Levy, Thieu Ngoc Vo, Siamak Ravanbakhsh, and Truong Son Hy. E (3)-equivariant mesh neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 748-756. PMLR, 2024.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph allention networks. arXiv preprint arXiv:1710. 10903, 2017.
Boyan Wan, Yifei Shi, Xiaohong Chen, and Kai Xu. Equivariant diffusion model with a5-group neurons for joint pose estimation and shape reconstruction. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2025.
Dian Wang, Robin Walters, and Robert Platt. SO(()-cquivariant reinforcement learning. In International Conference on Learning Represenlations, 2022.
Maurice Weiler and Gabriele Cesa. General e -euueqqrrrr sleerable cnns. In Neural Information Processing Systems (NeurIPS), 2019.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Robin Winter, Marco Bertolini, Tuan Le, Frank Noc, and Djork-Arné Clovert. Unsupervised lcarning of group invariant and equivariant representations. In Advances in Neural Information Processing Systems, volume 35, pp. $1942-31958 2022.
Renjun Xu, Kaifan Yang, Ke Liu, and Fengxiang He. E(q)-equivariannt vision transformer. In Uncertainty in Artificial Intelligence, pp. 2356 2366. PMLR, 2023.
Dmitry Yarotsky. Universal approximations of invariant maps by noural networks. Constructive Approzi- malion, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexan- der Smola. Deep sets. arXiv preprint arXiv: (0f:::173.066 703.06114, 2017.
14

---

Published in Transactions on Machine Learning Research (04/2026)
A Proof of Theorem 4.2
Proof. We verify that the boundary hyperplane is well-defined. Firstly, we should check the definition of feature function F. Given the function F = Enm Bio ([a;, r)), we could consider the set S = {or E IR" (as. 2) * 0, e E [m]]. Of course its complementary set S is the union of some hyperplanes 1 Mr, where M, (x x) = 0}. Then for any E 5, one of (as, .) > 0 and {a,, x) < 0 holds. For small enough 0 or (a < holds for all y E B(x, As a result, we have B(x,r) S. While every (as, v) does not change its sign for y E B(x,,)) r), the output function F is linear in B(x, Especially by Riesz representation theorem, there exists unique vector such that F(y) (v, on B(x,r). Combining with that F(x) = Ei(a,,,)>0 S,(0;, x), we have F(x) = Eijan,a)o 0,57 for all Z E S. Morcover, for every hyperplane M * M; for all E [m] and any continuous distribution m on it, we have x E E almost surely.
T, (a),
For every E M of any given hyperplane M and v & M, we have F(x + ōv) is a constant when 5 > 0 is small enough, since x + dv would not pass through any hyperplanes M, as ō goes to O. As a result, F(x+ converges to that constant uniformly, bounded by E= ||0,, Therefore, dominated convergence theorem ensures that lim (x + 80)] = Ellim F(x + 8v)]. Moreover, if x + du is not in the the hyperplanes M, (it almost surely holds), x E S implies lim Fly + óv) Since lim F(x + ōv) is piece-wist constant, the expectation is well-defined.
constant for all y E (xx,r) n M for small radius r.
For given M, we can compare lim F(x++v)) and lim F(x 6v) immediately. If & M, for all i E [m], we have 2 E S and then ++v and x- z-ôv are both in Bz,,r) C. S for small 6, resulting in that F(x++ = F(x- dv) for small 8 > O. As a result, if M * M, for all i E [m]] x is in S and therefore F(x+ = F(x ōv) almost surely, so that lim [[F(x + 5v)] = lim F[x - 8v)]. Otherwise, we can let F = E 20,
and thus
lim E[m òv)] = lim -v)].
Meanwhile, let G denote F - F =
x)), we have
g(r - 50) - G(x -ōv)- = Σ a
Σ
a Vo E M.
ki(a,)))00
Therefore, we have
E Elg(x + ou) G(x 5u)] = Σ
Σ agt.
8:(o..0) <0
As a result, the hyperplane M is boundary hyperplane if and only if Ei(or))> ., # Exjonm O.8T.
Moreover, if no ag satisfies that (as, M) O, the hyperplane M is not a boundar hyperplane since Eetornnso ag = 0 Enjoraa< a,B Inversely, if M is a boundary hyperplane, there exists some channel vector a such that M) = 0.
The proof is completed. In addition, we would like to use the feature function to characterize the boundary hyperplane for future exlension to multi-layer conditions, where the equivariant condition would be more complex. However, the feature functions can be similarly defined and therefore the boundary can be shaped.
B Proof of Lemma 4.4
For simplicity, we denote e E " as the vector (O...., 1,..,, ,,T,..,)),, where the i-th entry and other entries are all O. Since vi a = o thy we have wor(x) (vx) for all E R".
If there are two nonzero entries thy < the in the same row of Wig we can set == Ej ex and then we have (z)) = (Wgez); = bay while (o(w)=)); (thi vhx) = 0. That implics Vij = O, which contradicts. Thus, each row has at most one nonzero entry. Hence, there are at most n nonzero entries of thy, which are in different rows.
Since by is invertible, each row and each column of vi have at least one nonzero entry. It implics that there are least n nonzero entries of vig. Thus, there are exactly n nonzero entries of vy and by is generalized permutation matrix diag(A), (1 . A.m) Pg
15

---

Published in Transactions on Machine Learning Research (04/2026)
Moreover, when Pay + O, we set Cj. Then ((úx) (0)(x) implics that o(biy) Vay- From the definition of ReLU, we have that thij >> 0. The proof is completed.
C Proof of Theorem 4.3
Proof. Let be an invariant output function that can be represented as En invariant, can be also reformulated
Since F is
m
F=
OF
olaa,, Pyz)))
SEG
Then the feature function can be
F(x)
E
T 00
G
La a1,))50
Then, for every transformed point Po we have
F(pg2) |Gl
Σ
ph aaa
Σ
= |G| a
am.)>0
=(p') G
p af (og) (pz)-1
L.te #)>0
As result, F is somehow "equivariant'. Moreover, let us consider the definition of boundary hyperplanes,
we have
lim 6-0
+ pg)) lim (x = (0
lim E[F(x + u)],
6+0+
8-1
and
lim EFF(pxt- pat)] = lim (x- (0g)) lim E[F(x u)], 5-01
8-0
where v & M and Pou & PgM. Therefore, M is a boundary hyperplanc if and only if P is boundary hyperplane. Besides, if we consider the equivariant condition, M is a boundary hyperplane if and only
Σ i.eepp (..)>5 (o) M))=0
P #
Σ
s,he a, (0) M)=
By contrast, that for MyM is
Σ
T a +
Σ
Ph T.a
T
i,bolo ") (g) a...g M3=0
(p) a).
The LHH is equal to
Σ
p a
Σ
(P)) of =0z)
Σ
Lhi as 200
hnisph 02.0 io M))=0
(e M)=0
(P) M)
while the RHS equal to
Σ
p TT
Σ
( a, = -02)
Σ
PT.
i.hii #0 (e) M)=
i.hslgh ahalppa..) co (0), M))=
M))0
As result, M and PyM have the equivariant conditions be boundary hyperplanes, causing that M is a boundary hyperplane if and only if 0g M is boundary hyperplane. The proof is completed.
16

---

Published in Transactions on Machine Learning Research (04/2026)
D Proof of Theorem 4.6
Proof. Due to W o Pg = 4/ o W(), and thus we have
w
Σ
v/
w(
Pg.
|G
DEG
Then can be rewritten as:
F(x) [81 ... Bm]
([a]
am
-[31 ... ......mm
a1
aml)
paæ
G
g
gEG
=[3 13....Baa Bm]
p
G
(0%) Py
pgtt
9EG
1
Bio
Σ
(12(1)
pgx
G
(1tg)..P.00)
1=1
9EG
B.
|3311
T
(pp(())
Pg
G
||3:11
(11g),
(DEG
((5
T
B
og 1
|.B.|1
c
x
==1
BEG
Denote 3
8 [.lI
and á; = En P3 () we get F(x) = Σ (,0
For new parameters, we have
1
P a. Ph.
Pg (
G
gEG
T gg
Ph. (1)
9EG
T
(1, (000))))
G
gEG
0P (1))
which is a satisfied rewritten LEN.
E Proof of ENs' Universal Represetation Ability
Proof. We construct the following ENs to represent all equivariant output functions of GNs. Let the output function of GNs be ( o (W.))...)) Let the weight matrices of ENs be
W(4) = G
(6b (WL)
2o w())
[G|
W/00- diag (()
(
Ae = 2.3.
[a|
W (WW11)
Pen ...., (W(1)
Poya y²)T.
IG
17

---

Published in Transactions on Machine Learning Research (04/2026)
where the group representation & in the intermediate layers is the corresponding permutation representation acting on each axis eg. as Vggg: -g.g-.. As result, we have, A4 = 2,3... 2.3.... L-1,
W Pg.
W Paro
W Pg. W Pgz
W Pgag
W Pg
Pg
W Psia
W Pg a19l
[WO Plia
1
diag W
W yo diag(
W())
W
didi W(O
W 1/ W(,
[
[G]
|G]
W
1
(0g)
(())
Pga w(1)
(6
(())
910 W(1)
G
G
919
IGI
|G]
1 |G|
(4)
W(l))
|G|
As result, the constructed network has the same output function QF and equivariant layers, renders EN. The proof is completed.
F Proof of Lemma 6.1
G} and N = 3/11//10 31 g G}. where a and are two nonzero vectors in R", Then we prove that M NNN- = @ or M = N.
Proof. Denote M = a
If MONN& # 0, there exist group elements 9o. hg E such that P a/llllp all = pho 3/1lph B|1. Then, for any element paa/llpaaal] all E M,, there exists Phoso 3/110" 11ll" h E N such that
Iph 311
p Pho 1 3- 0g p P
0g a and
hogo 9
1 90 ax
lP hogo i 31
l arl
It implies that M C N. Inversely, and N C M, we have M = N. Moreover, when denoting M a M |M]. If there exist g, h E G such that a////lp"aal = p////lpp all, we have Df paa- cpp a with some positive number since they have the same direction. Then, we have of = ca and
prove that N C. M in the same way. Hence, combining M C N
G}, we also prove that
With ) = (0gh-1)) (1 ca =
Meanwhile, since (gh Gl = E G, we have - = 15 a == a. It implies that d! = and thus o
c = 1. Finally, we have that of = p a
+
Tl
of Tip all which indicates that [M [M]. Therefore, if
M N, we have M kN for some ls >
In particular, if there exist two parallel channel vectors aaa and of aj, assume o kpp a, for some l: > O and then we have
O = 10g (p' a.) : g) (0g (kp) g k(2 ( kO,
Moreover, ENs can merge these channels as
Σ Bio a x)) 3 3,0(103 x)) = Σ (kB, + 33)0
gEG
gEG
As result, we can find an EN of the same output function such that P as 1p5 asll # a for all
18

---

Published in Transactions on Machine Learning Research (04/2026)
Furthermore, can define the stabilizer subgroup Stab(a) with
{g : Pg ast and therefore G Stab(a) @ M f01 = Staaba))
Σ a()
E Σ
E
a))
gEG
hEG/Stab(a) gh gh-t EStab(a)
hEG/Staa(())
The proof is completed.
G Proof of Theorem 6.2
Proof. We consider the following conditions with given invariant output function of GNs F(x) Ex= (,o((a;; x). Denote the number of boundary hyperplane of F(x) by N.
If N < m -1 1, we can construct the EN of Lemma 6.1 where we let the output function be formulated as (F(x)= = GG] Ex Bi Lgea ((P) 0s, x)). Lemma 6.1 implics that the number of channel vectors m' after
merging is at most double the number of boundary hy hyperplanes plus 2, and therefore,
m' s 2N +2 2(m + 2= 2m.
If N = there are boundary hyperplanes MI,... Mm and can assume (a;, M2 for all E [m] without loss of generality. Theorem 4.3 indicates that all the boundary hyperplanes are symmetric, enable us to classifier them as S1 = { where each set collects all symmetric boundary hyperplancs. Then consider the orbits O = { a, g e G}, we have O; = O for all i, in the same classification set S. Lemma 6.1 implies that the sum C E 9 EnEr (10g ay, x)) can be formulated by the sum of at most 2m items, resulting in m' s 2m also.
M S = {M M with kn = 0 and k441 = m,
The proof is completed.
19


Reviews:
[{"review_id": "u24U08nBHq", "reviewer": "Reviewer_CP2c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies differences in expressive power of General Equivariant Nets (GENs) and Layer-wise Equivariant Nets (LENs), compared to non-equivariant General Networks (GNs), where all the networks are assumed to be 2-layer ReLU networks. They show that in some cases, GENs and LENs clearly lead to higher error than GNs, and that LENs need to have more neurons to approximate functions that GENs can for the same error value. By increasing the number of neurons in LENs to at most 2x the number of neurons in GENs, the expressive capacity of the LEN becomes comparable to the GEN. \n\nStrengths\n1. As the paper is fully theoretical, it is easy to see that it is technically sound.\n2. The paper addresses some interesting questions about expressivity of equivariant networks.\n3. I appreciate the authors constructing simple, but very illustrative examples, to help understand the points about loss of expressivity with equivariance, and how LENs can have the same capacity as GENs with at most 2x the neurons, and still have better generalization. \n\nWeaknesses:\n1. As a researcher who does not do a lot of theoretical work, it is not clear to me how useful these results are in practice. All the analysis is for finite groups and for 2-layer ReLU networks. The results mostly seem to agree with general intuitions, which is good, and it is also good to have actual proofs rather than just intuitions, but I don't see broader use of the results. It would be nice if the authors can provide some explanations to put the paper in broader context. \n\n2. The authors should try and improve the Related Work section. Under \"Theoretical Analysis\", many papers are listed, but no connections to this paper are mentioned, how are all these previous papers related to this submission? The authors may find this paper to be of value to discuss as well, which also considers symmetries in data and misspecified symmetries: https://arxiv.org/abs/2305.17592\n\n3. No proof is provided for Theorem 6.3, I can see that it is true, still good to provide a full proof. I think this paper has some similar ideas: https://arxiv.org/abs/1804.10306\n\n4. The analysis uses number of neurons as the way to measure the size of the networks. This is not really what people think of when they think of sizes of network, the number of learnable parameters that matters more. This is obvious when weight sharing is involved, as in most equivariant networks (like CNNs). So, counting the number of neurons in LENs, and then saying that LEN is still more generalizable because it actually has fewer learnable parameters seems trivial to me (section 6.3). Hope the authors can clarify why the number of neurons is important to count, rather than the learnable parameters.\n\n5. This is not a real weakness, but it would be good if the authors could have a small addition to discuss the impact of training data and the training dynamics on choice of model architectures, as those are not considered in the paper.\n\n6. There seem to be a few typos in the paper. For example, at the beginning of section 3, I think the dimensions of W1 and W2 should be transposed for the dimensions to make sense.", "claims_and_evidence_yn": "Yes", "claims_explanation": "All the claims in the paper are mathematical in nature, and they come with mathematical theorems. I have gone through the proofs of all the theorems and lemmas, and they all appear to be correct.", "audience_yn": "Yes", "audience_explanation": "Yes, TMLR audience includes researchers working on equivariant neural networks. They are likely to find this paper interesting.", "requested_changes": "Please address the weaknesses listed above.", "broader_impact_concerns": "No such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tPwt1weAqC", "reviewer": "Reviewer_RR2G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses equivariant two-layer ReLU-networks. In particular, the discussion centers on the fact that restricting each linear layer to be equivariant under a permutation representation does not cover all possible equivariant functions representable by the base architecture at a fixed width. The paper notes that one can compensate for this by increasing the network width.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There are some slight inaccuracies, but they seem to be easily fixable. \n\n1. Some results are stated in terms of invarance only, it would help the reader to be clearer about which results regard invariance vs equivariance.\n2. The last paragraph on page 5 is unclear. It is stated that \"*We prove that for a GEN, the set of its boundary hyperplanes are symmetric with respect to all group representations.*\" However, I believe that the intent is that the symmetry is w.r.t. the specific group representation $\\rho$ that the GEN is equivariant under.\n3. The first equation in page 6 is stated in terms of invariance. It seems to me like a $\\phi$ should be involved for this to cover the equivariant case. Specifically, $\\mathcal F(\\rho x) = \\rho^{-T} \\mathcal F(x) \\phi^T$.\n4. At the bottom of page 10, the following sentence is unclear to me. \"*Similarly, this model still has comparable expressive power, but a lower-dimensional hypothesis space, since we constrain all first-layer channel vectors within a (sub-)space.*\" Having comparable expressive power seems contradictory to having a smaller hypothesis space. Is the intended meaning that the model has comparable expressive power amongst equivariant functions, but a smaller hypothesis space when considering all functions?", "audience_yn": "No", "audience_explanation": "The findings in the paper are interesting to the geometric deep learning community, but the presentation of the findings should be improved. Most of the findings are already covered in the literature. The relevant papers should be discussed and compared to. It needs to be clear what the presented findings contribute that has not been priorly covered.\n\n1. Most importantly, in [A] all invariant shallow ReLU-networks are classified and it is shown that they can always be rewritten such that the layers are equivariant under signed permutation representations. Furthermore, these signed permutation representations can be written as permutation representations by doubling the network width. Theorem 6.2 in the submission therefore seems to follow directly from the results in [A]. The same is true for Section 5.\n2. Lemma 4.4 is well-known, see for instance [B]. A similar proof to the one in the submission appears in [C].\n3. Theorem 6.3 is covered in [D].\n\n[A] Agrawal & Ostrowski, A Classification of G-Invariant Shallow Neural\nNetworks, NeurIPS 2022\n\n[B] Godfrey et al, On the Symmetries of Deep Learning Models and their Internal Representations, NeurIPS 2022\n\n[C] Bökman & Kahl, Investigating how ReLU-networks encode symmetries, NeurIPS 2023\n\n[D] Elesedy & Zaidi, Provably Strict Generalisation Benefit for Equivariant Models, ICML 2021", "requested_changes": "For me to recommend acceptance of this submission, the points listed under **evidence** above should be clarified and the presented findings should be contextualized within the literature.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6wIvZeSIvm", "reviewer": "Reviewer_AX99", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper draws attention to shortcomings of equivariant neural networks. The main claims is that there is a trade off for restricting the\nhypothesis space to equivariant models increases the expected risk when approximating targets. To address this, the authors compared risks for GN, GEN, and LEN models and also showed that the symmetry requirements incurred larger amount to neurons to have similar expressive power with unconstrained models. I think novelty of their position and theoretical rigor are noticeable strength of this work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This is a theoretical work evidenced by proofs. I did not checked the entire steps, but I think most of the arguments are sound at least in the outlined example. Most of its claims can be intuitively understood by ML practitioners.\n\nHowever, I am currently somewhat doubtful for implication of their claim and whether their claim holds in practice. \n* I believe the entire proofs are based on all models consider the identical risk minimization problem i.e. $\\inf_\\theta \\mathbb{E} p \\lVert F_\\theta(x) - x\\rVert^2$. However, there are plenty of cases that equivariant models and nonequivariant models cannot be directly compared with same cost functions and data distributions can be drawn from manifolds equipped with symmetry that can be only trained with equivariant models. \n* In the same vein, I think the authors arguments are trivial in the sense that nonequivariant models are **specialized** to solve **unconstrained** risk minimization while equivariant models are not. Therefore, I believe that it would be much meaningful if the analysis focused on highlighting the trade-off of enforcing equivariance, rather than one-sided drawback of equivalent nets, where it is apparent that the general cost minimization is was not likely a preferred objective at the fist place.\n* It will be much appreciated if the analysis contain extension to 3D spaces e.g. rotation group.", "audience_yn": "Yes", "audience_explanation": "Equivalent neural nets are an active field of study in machine learning and its applications.", "requested_changes": "* Please put appropriate justification for comparing expressive power with unconstrained risks.\n* Some practical group convolution manages to cut their computation costs with efficient modeling that can be only possible with equivariance. I suggest the authors to extend the conclusion section with limitations to properly represent their theory in the real-world.\n* Please hints how these theoretical findings in 2D can be applied to  group equivariant 3D  deep learning models such as SO(3).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)