G-RepsNet: A Lightweight Construction of Equivariant Networks for Arbitrary Matrix Groups
Authors: Sourya Basu
Suhas Lohit
Matthew Brand
Venue: Accepted by TMLR ()

Abstract: Group equivariance is a strong inductive bias useful in a wide range of deep learning tasks. However, constructing efficient equivariant networks for general groups and domains is difficult. Recent work by Finzi et al. directly solves the equivariance constraint for arbitrary matrix groups to obtain equivariant MLPs (EMLPs). But this method does not scale well and scaling is crucial in deep learning.
Here, we introduce Group Representation Networks (G-RepsNets), a lightweight equivariant network for arbitrary matrix groups with features represented using tensor polynomials. The key insight in our design is that using tensor representations in the hidden layers of a neural network along with simple inexpensive tensor operations leads to scalable equivariant networks. Further, these networks are universal approximators of functions equivariant to orthogonal groups. We find G-RepsNet to be competitive to EMLP on several tasks with group symmetries such as $O(5)$, $O(1, 3)$, and $O(3)$ with scalars, vectors, and second-order tensors as data types. 
On image classification tasks, we find that G-RepsNet using second-order representations is competitive and often even outperforms sophisticated state-of-the-art equivariant models such as GCNNs and $E(2)$-CNNs. To further illustrate the generality of our approach, we show that G-RepsNet is competitive to G-FNO and EGNN on N-body predictions and solving PDEs respectively, while being efficient.

Published in Transactions on Machine Learning Research (05/2025)
G- G-RepsNet: A Lightweight Construction of Equivariant Net- works for Arbitrary Matrix Groups
Sourya Basu" AWS AI Labs
basusouryo@@gaallcco com
Suhas Lohit Mitsubishi Electric Research Laboratorics
slohit@hmerl.
Matthew Brand
brand@merl.com com
Mitsubishi Electric Research Laboratories
Reviewed on OpenReview: https: // openeeveew......... net/ formm? id= kIeYngOuf0
Abstract
Group equivariance is a strong inductive bias useful in a wide range of deep learning tasks. However, constructing efficient equivariant networks for general groups and domains is difficult. Recent work by Finzi et al. (2021b) directly solves the equivariance constraint for arbitrary matrix groups to obtain equivariant MLPs (EMLPs), but this method does not scale well and scaling crucial in deep learning. Here, we introduce Group Representation Networks (G-RepsNets), lightweight equivariant network for arbitrary matrix groups with features represented using lensor polynomials. The key insight in our design is that using tensor representations in the hidden layers of a neural network along with simple inexpensive tensor operations loads to scalable equivariant networks. Further, these networks are universal approximators of functions equivariant to orthogonal groups. We find G-RepsNet. be competitive to EMLP on several tasks with group symmetries such as O(5), (11,)), and 0(3) with scalars, vectors, and second-order tensors as data types. On image classification tasks, find that G-RepsNet using second-order reprosentations is competitive and often even outperforms sophisticated state-of-the-art equivariant models such as GCNNs (Cohen & Welling 2016a) and E(2)-CNNs (Weiler & Cesa 2019 To further illustrate the generality of our approach, we show that G-RepsNet is competitive to G-FNO (Helwig et al. 2023 and EGNN (Satorras et al.) 2021) on N-body predictions and solving PDEs respectively, while being efficient. Code will be released hhtpps: //g:: htt///eeerreeeeerrc///eeeeelplle
1 Introduction
Group equivariance plays a key role in the success of several popular architectures such as translation equivariance in Convolutional Neural Networks (CNNs) for image processing (LeCun et al. 1989), 3D rotational equivariance in Alphafold2 (Jumper et al. 2021), ) and equivariance to general discrete groups in Group Convolutional Neural Networks (GCNNs) (Cohen & Welling, 2016a).
But designing efficient equivariant networks can be challenging because they both require domain-specific knowledge and can be computationally inefficient. E.g., there are several works designing architectures for different groups such as the special Euclidean group SE(3) (Fuchs el al. 2020). special Lorentz group 0(1,3) (Bogatskiy et al.] 2020), discrete Euclidean groups (Cohen & Welling 2016a Ravanbakhsh et al. 2017), etc. Moreover, some of these networks can be computationally inefficient, prompting the design of
simpler and lightweight equivariant networks such as EGNN (Satorras et al. 2021 for graphs and vector
neurons (Deng et. al. 2021 for point cloud processing.
*Sourya Basu was intern MERL when this work was performed

---

Published in Transactions on Machine Learning Research (05/2025)
Finzi et al. 2021b) propose algorithm to construct equivariant MLPs (EMLPs) for arbitrary matrix groups when the data is provided using tensor polynomial representations. This method directly computes the basis of the equivariant MLPs and requires minimal domain knowledge. However, using the computed oquivariant basis can be computationally expensive, and it is impractical to use them for practical datasets such as images and point clouds, as noted by the authors (Finzi el al. 2021a). It is also noted in prior works (uuhh et al. 2020 Thomas et al. 2018) that using equivariant basis, even for simple groups such as SO(3), can be computationally expensive and is impractical to use them to scale up to large datasets. Hence, we propose lightweight construction of equivariant networks which is inexpensive, and yet is competitive to EML.Ps for toy datasets and scales up to larger datasets of practical importance.
To this end, we introduce Group Representation Network (G-RepsNet), which replaces scalar representation from classical neural networks with tensor representations of different orders to obtain expressive equivariant networks. We use the same tensor polynomial representations as EMLP to represent the features in our network. But unlike EMLP, we only use inexpensive tensor operations such as tensor addition and tensor multiplication to construct our network. We show that even with these simple operations, we obtain a universal network for orthogonal groups. EMLPs are empirically known to be computationally expensive Further, we empirically show that G-RepsNet provides competitive results to existing statc-of-the-ae equivariant models and even outperforms them in several cases while having a simple and efficient design.
Our proposal generalizes Vector Neurons (Dng et al. 2021 which use first-order 0(3) tensor representations to obtain equivariance to the ((3) group. contrast, G-RepsNet is a construction which is equivariant to arbitrary matrix groups, universal for orthogonal groups, and uses higher-order tensor polynomial representations, while being computationally efficient. The main contributions as well as the summary of our results are detailed below.
1. We propose novel lightweight construction of equivariant architectures. We call them G-RepsNets, which is are class of computationally efficient architectures equivariant to arbitrary matrix groups and easy to construct.
2. We show that G-RepsNets are universal approximators of equivariant functions for orthogonal groups.
3. On synthetic datasets from Finzi et al. (2221b), we show that G-RepsNet is computationally much more efficient than EMLP and also performs competitively to EMLP across different groups such as O55), O(3), and O(1,3) 3) using scalars, vectors, and second-order tensor representations.
4. We show that G-RepsNet with second-order tensor representations outperforms sophisticated state- of-the-art equivariant networks for image classification such as GCNNs (Cohen & Welling
2016a) and E(2)-CNNs (Weiler & Cesa. 2019) when trained from scratch, and equitune (Basu et al. 2023b when used with pretrained models.
5. G-RepsNet is competitive to G-FNO (Helwig et al. 2023 and EGNN Satorras et al., 2021 on N-body predictions and solving PDEs, respectively, while being computationally efficient.
2 Related Work
Parameter sharing A popular method for constructing group equivariant architectures involves sharing learnable parameters in the network to guarantee equivariance, e.g. CNNs (TeCun et al. 1989), GCNNs (Cohen & Welling 2016a; Kondor & Trivedi 2018), Deepsets (Zaheer et al., 2017), etc. However, all these methods are restricted to discrete groups, unlike our work which can handle equivariance to arbitrary matrix groups.
Steerable networks Another popular approach for constructing group equivariant networks is by first computing a basis of the space of equivariant functions, then lincarly combining these basis vectors to construct an equivariant network. This method can also handle continuous groups. Several popular architectures employ this method, e.g. steerable CNNs (Cohen & Welling 2016b), ((2)-CNNs (Weiler & Cesa 2019), Tensor Field Networks (Thomas et al. 2018),, tranntttrassoommes (Fuchs et al. 2020). EMIPPs (Finzi et al. 2021b) etc. But, these methods are computationally expensive and, thus, often replaced by efficient equivariant architectures for specific models, e.g., F((n) equivariant graph neural networks (Satorras et al. 2021 for
2

---

Published in Transactions on Machine Learning Research (05/2025)
D&L H4y
DXLF GHRgsthet
Inpir epeosettlio Dec.at)
Sec
Time par spocch (sa0))
Ty Layer
Tiat
Zplayer
Lyyaye
090
Imputs
Outputs
Tran
(a)
(b)
(c)
Figure (a) Summary of G-RepsNet layer construction with example inputs of types To, Ti, and T2, and outputs of the same types. Each layer consists of three subcomponents: i) imput feature representation shown as Ti ii) converting tensor types appropriately shown using arrows from T; to Tj. and iii) neural processing the converted tensors using appropriate neural networks, as discussed in Section 4 (b) and (c) provide comparisons of the loss and wall time of G-RepsNels with EMLPs Finzi et al. 2021b and MLPs for an eqqiivarar regression described in Section 5.1
graphs and vector neurons DDng et al. 2021 for point cloud processing. More comparisons with EMLPs are provided in Appondix A Kondor et al. (2018) propose using stecrable higher-order permutation reprosentation to obtain a permutation-innvriant graph neural networks. In contrast, we use higher-order tensors for arbilrary matrix groups, work with arbitrary base models such as CNNs, Fourier Neural Operators (FNOs) (Li et al, 2021), etc., and show that our architecture is universal approximator for functions equivariant to orthogonal groups.
Representation-based methods A simple alternative to using steerable networks for continuous networks is to construct equivariant networks by simply representing the data using group representations, only using scalar weights combine these representations, and using non-linearities that respect their equivariance. Works that use representation-baet universal scalars (Villar et al. 2021). Vector neurons restricted to first-order tensors and Universal Scalars face scaling issues, hence, mostly restricted to synthetic experimenls. More comparisons with universal scalars
methods include vector neurons (Deng et al. 2021) for 0(3) group and
are provided in Appendix
A
Frame averaging Yet another approach to obtain group equivariance is to use framo-averaging (Yarotsky 2022 Puny et al., 2021), where averaging over equivariant frames corresponding to each input performed obtain equivariant outputs. This method works for both discrete and continuous groups but requires the construction of these frames, either fixed by design as in Puny et al. (2021): : Basu et al. (2023b) learned using auxiliary equivariant neural networks as in Kaba el al. (023). Our method is, in general, different from this approach since our method does not involve averaging over any frame or the use of auxiliary equivariant. networks. For the special case of discrete groups, the notion of frame averaging is closely related to both parameter sharing as well as representation methods. Hence, in the context of equituning (Basu et al. 2023b), we show how higher-order tensor representations can directly be incorporated into their frame-averaging method.
3 Group and Representation Theory
A group is set G along with binary operator ':, such that the axioms of group are satisfied: a) closure:
91 32 G for all 91,92 92 E G, b) associalivity: (g) 92) 93 =91 (g2 ga) for all ni 92, E G, c) identity: there exists e E G such that e = -- for any E G, d) inverse: for every g E G there exists g E G such that g = g-
For a given set X, a group action a group G on X' is defined via a map a : G X X such that a(e, 2) = x for all x E X, and agg), a(g2,))) = a(g: 92, .) for all g1. 91 E G and x X, where e is the identity element of G. When clear from context, we write a(g,z) simply as ga. Given function f D, call the function to be G-equivariant if f(gr) gf(x) for all g G and X.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Let GL(m) represent the group of all invertible matrices of dimension m. Then, for a group G, linear group representation of G is defined as the map p : G GT(m) such that p(g)92) = ((())((()) and p(e) -1, 1 the identity matrix. A group representation of dimension m is a linear group action on the vector space R...
For finite group G, the (left) regular representation p over a vector space is a linear representation over V that is freely generated by the elements of G, i.e,, the elements of G can be identified with a basis of V. Further, p(g) can be determined by its action the corresponding basis of V, p(g) : h + gh for all h e G. For designing G- G-RepsNet, note that the size of regular representation is equal to the size of G. The size of tensor product represcntations considered here, m, can be written as m |G] x d for some integer d. We call the first dimension of size |G as the group channel dimension.
We call any linear group representation other than the regular representation as non-regular group representation. Examples of such representations include representations written as a Kronecker sum of irreducible representations (basis of a group representation). In the design of G-RepsNet, we use regular representation for finite groups and non-regular representations for continuous groups. Given some base linear group representation p(g) for g E G on some vector space V, we construct tensor representations by applying Kronecker sum @ Kronecker product @, and tensor dual " Each of these tensor operations on the vector spaces leads to corresponding new group actions. The group action corresponding to V* becomes p(g-1) Let Pi (g) and p2(g) for g E G be group actions on vector spaces V1 and V/2 respectively. Then, the group action on Vi @ 1/2 is given by (i @ (2( and that on V1 @ 1/2 is given by (i @ p2(g).
We denote the tensors corresponding to the base representation p T1 tensors, i.e., tensors of order one, and To denotes scalar. In general, T, denotes a tensor of order m. Further, Kronecker product of tensors Tm and Tn gives tensor Tm+= of order + n. We use the notation T to denole times Kronecker product of Tm tensors. Kronecker sum of two tensors of types Tm and Tn gives tensor of type Tm + Tu Finally, Kronecker sum of r tensors of the same type Tm is written as rTm.
(n) refers to the orthogonal group which is the group of dimension n that preserves the distance in Euclidean space of dimension n. The group elements of (n)) can be identified with orthogonal matrices Q of dimension X which satisfy QTQ QQT In, where I, is the identity matrix. Similarly, the Lorentz group ((1, n) is the group of all isometries of n-dimensional spacetime that leave the origin fixed, where the distance is computed using the Minkowski metric.
4 G-RepsNet Architecture
Here, we describe the general design of the G-RepsNet architecture. Each layer of G-RepsNet consists of three subcomponents: i) representing features using appropriate tensor representation (Section 4.1p, ii) converting tensor lypes of the input represent (Section 4.2 and iii) processing these converted lensors (Section 4.3)
Finally, Section 4.4 discusses some properties of our network along with existing architectures that are special cases of NeRepsNet. Now describe these subcomponents in detail.
4.1 Input Feature Representations
We employ two techniques to obtain input tensor representations for networks with regular and non-regular representations as described below.
Regular representation is favorable to use for small finite groups, c.g., cyclic group C, of discrete rotations of 360 degrees. For input features for regular representations, we simply use the input features obtained from the E(2)-CNNs (Weiler & Cesa. 2019), but any regular representation works with our model. Thus, if we are given an image of dimension B x C H x W, the Ti regular representation is of dimension |G] x B x C x H x W, where [G], B, C, H, W are the group channel dimension, batch size, channel size, height, and width, respectively. Similarly, for any tensor of type Tú, the group channel dimension of size G!..
Regular represcntation:
n
Non-regular representation: Non-regular representation is useful for all contimuous groups as well as large finite groups, e-g. (0(n) group of rotations, S(n) group of permutations. For non-regular representations,
4

---

Published in Transactions on Machine Learning Research (05/2025)
usually, the data is naturally provided in suitable tensor representations, e.g., position and velocity data of particles from the synthetic datasets in Finzi et al. (2021b) are provided in the form of Kronecker sum of irreducible representations of groups such as O(n). In all our experiments using continuous groups, the inputs are already provided as tensor representations using the appropriate irreducible representations. Here, we call the dimension of the matrix representation of the group as the group channel dimension. E.g., for the (0(2) group elements represented as 2 x matrices, we have the group channel dimension equals to 2.
4.2 Tensor Conversion
Crucial to our architecture is the tensor conversion component. The input to cach layer in Fig. la is given as a concatenation of tensors of varying orders. But T,-layer in Fig. Ia only processes tensors of order T. Thus, to process tensors of order T3. j # i, they must first. be converted to tensors of order T₃ and then passed to the Ti-layer. Our tensor conversion algorithm is described next.
When i >> O, we convert tensors of type Т₃ to tensors of type T, by first writing i kj + Fr, where k = [i/j]- Then, we obtain T; from Т₃ and Ti as Trss 60 TT." When using non-regular representations, we assume that the input to the G-RepsNet model always consists of some tensors with Zi representations, which is not a strong assumption that helps keep our construction simple and also encompasses all experiments from Finzi et al. (2021b). We do not convert tensors of type T, to Ti for 0 < i < as (a) it requires tensor dccomposition, which can be expensive practice, and (b) we already obtain universality for orthogonal groups without it.
When i i= 0, convert each imput of type T3 to type To by using an appropriate invariant operator, e.g. Euclidean norm for Euclidean groups, or averaging over the group channel dimension for regular groups. These design choices keep our design lightweight as well as expressive as we show both theoretically as well as empirically. Details on processing these imputs are described next.
4.3
Neural Processing
Now we discuss how the various T:-layers are constructed and how they process the input tensor features that have been converted to T; tensor types. We use different techniques for regular and non-regular tensor representations.
Regular representation: Recall that regular representations for tensors of type T, have group channel dimensions equal to |G]', where |G] the size of the group. For tensors of dimension (IG) x B) x C x H W, we treat the group channel dimension just like the batch dimension and process the (IG B) inputs in parallel through the same model. Here we are free to choose any model of our choice for any of the Tı-layers, e.g., MLP, CNNs, FNOs, etc. We call these models of choice our base model just like used in frame-averaging (Puny et al. 2021 and equitune (Basu et al. 2023b).
Non-regular representation: Here, we impose certain restrictions on what models can be used for Ti-layers and how to use them. First, the To-layer passes all the tensors of type To or scalars through a neural network such as an MLP or a CNN. Since the inputs are invariant scalars, the outputs are always invariant and thus, there are no restrictions on the neural network used for the Ta-layer, i.e., they may also use non-linearisies. Now we describe how to process the tensors of type T; for i i>0 0.
Let us call the output from the To-layer as Yr.- For T--layer with i > 0, we first multiply the input with learnable weight matrix along the dala dimension, i.e. the dimensions other than the group channel dimension and batch dimension, with no point-wisc non-linearities or bias terms. This ensures that the output is equivariant just as in Vector Neurons (Deng et al. 2021). E.g., if the input to T-llyee for the S0(2) group is x x n, then we multiply the input with a matrix of dimension n x m to get an output of dimension B x x m. Let us call the output from this linear layer as HT,-
Yr. Then, to mix the T₁ tensors with the To tensors better, we update Hy, as Hr, = Hr, * inv(Hr,)) where inv(-) is a group-invariant function such as the Euclidean norm for Euclidean group. We note here that this mixing function bears similarity to the bilinear layers used in EMI.Ps (Finzi et al. 2021b). However, we
5

---

Published in Transactions on Machine Learning Research (05/2025)
differ in its motivation. In EMLPs, the motivation is solely to enhance performance, whereas. in design, it is motivated from the to make the design universal for orthogonal groups (see Appendix C
Finally, we pass HT, through another linear layer without any bias or pointwise non-linearities to obtain Yr, This miring of various tensor types is crucial to make our network expressive and is required for the universality of our network.
Both (a) tensor conversion, that allows higher order tensor features and (b) tensor mixing that combines information among tensors of different orders/types are novel aspects of the proposed G-RepsNet architecture that generalize earlier works (see Appendix A (Deng et al. 2021 Zaheer et al. 2017 Basu et al. 2023b) Our
proofs on universality of G-RepsNets to orthogonal groups (see Appendix
C make use of this tensor-mixing
step to show that no theoretical representation ability is lost in spite of the simple construction, in the case of orthogonal groups.
4.4 Properties
Equivariance: For regular reprosentations, any group action applied to the input appears as a permutation in the group channel dimension. Further, since the data processed in parallel along the group channel dimension, the output permutes accordingly, making our model equivariant. For non-regular representation, the To layer only processes invariant tensors and, hence, preserves equivariance of the overall model. Moreover, the Ti-layers simply perform linear combination of tensors, making the overall model equivariant. A proof for equivariance of our model is given in Appendix B
Here, we show that our models are universal approximators of equivariant functions for orthogonal groups. This ensures that our models are expressive. For models constructed for regular representations, easy lo verify that there exist G-RepsNets that are universal approximators of equivariant functions. To that end, note that restricting G-RepsNet to only Ti tensors and taking an average with group inverses along the group channel dimension gives group symmctrization in equitune (Basu et al. 2023b Yarotsky 2022). It is well known that the symmetrization of universal approximators such as MLPs give universal approximators of equivariant functions (Yarotsky 2022). It follows that G-Reps.Nets are universal approximators of equivariant functions for regular representations. Note that even though our models using features of type Ti themselves are universal approximators, we illustrate empirically that higher order tensors significantly boost the performance of G-RepsNet with regular representation.
Universality:
For non-regular representations, we provide simple constructive proofs showing the universality properties of the G-RepsNet architecture. We first show that G-RepsNet approximate arbitrary invariant scalar functions of vectors from O(d) and O d) groups. Then, extend the proof to vector-valued functions for the same groups. First, recall the Fundamental Theorem of Invariant Theory for O(d) as described in Lemma
Lemma 1 [Wey 1946)): ) A function of vector inputs returns an invariant scalar if and only if it can be wrillen us a function ondy of the invariant sealar products of the inpul vectors. That is, given inpul veclors X2,....... X,), X2 E Rdd any invariant sealar function h RRdxn is R can be written as
X.)) X2.,X22.....
f((X,, XIXixx)
(1)
where (X,, Xj) denotes the inner product between X, and Xj, and f is an arbilrary function.
similar result. holds for the (11 group. In Thm. we show that G-RepsNet approximate arbitrary invariant scalar functions for ((d) or ((1,d) groups. The main idea of the proof is to show that G-RepsNet can aulomatically compule the necessary inner products (X,, X,) in equation D and the function ff in equation I can be approximated using an MLP in the To layer. Detailed proof is provided in Appendix
As mentioned in Villar al. 2021
Theorem 1. For given Ti inputs (X1, X2,..,, Xa) corresponding to O(d) or O(1,d) group, X₂ E Rd any invariant scalar function [Rdxn + R, there exists a G-RepsNet model that can approzimate h.
Similarly, this result can be extended vector functions as described in Thm. The proof for Thm.
2 is
also constructive and is provided in Appendix C

---

Published in Transactions on Machine Learning Research (05/2025)
Theorem 2. For given T1 inputs (X1, Xg...., ,x)) corresponding to O(d) O(1,d) group, X, R any equivariant vector function h: Rdxx + RR, there cxists G-RepsNet model that can approcimate h.
Finally, in Appendix A.2 harmonic networks (Worrall et al. 2017), Deepsets (Zaheer et al. 2017), and equitune (Basu et al.
we show how several popular models such Vector Neurons Deng et al.
202 2023b
are special cases of G-RepsNet.
5 Applications and Experiments
We conduct several experiments to show that our model is competitive with state-of-the-are equivariant models across various domains while also being casy to design and computationally efficient.
For non-regular representation, we provide two experiments: i) in Section 5.1 we compare G-RepsNet with EMLP on synthetic datasets, which encompass equivariance to several groups such as O(5), O(3), and ((1,3) and involves tensors of different orders; ii) in Section 5.2 we compare G-RepsNet with EGNN on N-body dynamics prediction. Note that the groups considered are restricted to orthogonal groups, even though G-RepsNets work for arbitrary matrix groups just. like EMLPs. This is because we directly take the datasets from Finzi et al. 2021bb; Satorras et al. (2021. which are restricted to orthogonal groups.
For regular representation, we provide two more experiments: i) for image classification we show in Section 5.3 that G-RepsNet with higher-order tensors and CNNs the base model can outperform popular equivariant. models such as GCNNs and E(2)-CNNs when trained from scratch and equitune when used with pretrained models; ii) for solving PDEs, in Appendix 5.4 we construct G-RepsNet with FNOs as the base model, where we find G-RepsNets is competitive to more sophisticated equivariant models such as G-FNOs (Helwig ct al. 2023 while being much faster than them.
5.1 Comparison with EMLPs
Datasets: We consider three regression tasks from Finzi et al. (2021b): ii5)-invariant task, (3u)-qquivar task, and (1,33) invariant task. In (6))-invrriant regression, we have input X = {2:)2=1 of type 27, and z/ x,ae") of type Tb. Finally, for the ((1.3)-equivatle
output f(11, = sin (1|=1])) - || /2 + |> of type To. Then, for 0(3) equivaaai task we have input X {m,,, ==) of type 5T. 5Ti corresponding to masses and their positions. The output the inertia matrix I E, (=]'xa
task, we use the electron-muon scattering (e + - + (-) task from Finzi et al. (0021)) originally from Bogatskiy et al. (2020). Here, the input is of type (2,), corresponding to the four momenta of input and output electron and muon, and the output is the matrix element of type To.0) (Finzi et al. 2021bb.
Model design: For all the experiments here, we design G-RepsNet using different tensor represontations in each of the models depending on the application. We call the number of tensors in a hidden layer as its channel size. We fix a channel size of 100.
((D)-invariant model: The input consists of two tensors of Ti type that are passed through the first layer consisting of To-layers and Ta-layers similar to vector neurons shown in Fig. 6a but our design differs from vector neurons in that we use simple Euclidean norm to compute the To converted tensors instead of dot. product used by vector neurons. All T; layers are made of MLPs. The number of output tensors is equal to the channel size, and the channel sizes used for our experiments in discussed in Section 5.1 This is followed by three similar layers consisting of Taylayers and Ti-layers, all of which take as input T₁ tensors, and output tensors of the same type. Additionally, these layers use residual connections as shown in Fig. 5a Finally, the Ti tensors are converted to To tensors by taking their norms, which are passed through a final To-layer that. gives the output.
More preciscly, in our experiments, we consider a model with 5 learnable lincar layers with no bias terms, where the dimensions of the layers are (2 x 100, 100 x 100, 100 x 100, 100 x 100, 100 x 1). The input of type 27, is of dimension (2 x 5). The input is first passed through the first layer of dimension 2 x 100 to obtain a hidden layer output of type 100T, Then, this output is also converted to type 100T, by simply taking the norm. Thus, we have a tensor of type 100T, + 1007, Finally, we convert this tensor of type 100(T, + Ti) 1007; by simply multiplying the 100T, scalars with the 1007; vectors. This is basically simplified version
7

---

Published in Transactions on Machine Learning Research (05/2025)
10-
1.0
1.4
1.2
MJ
1.0
Pest MIS loss
MSE
0.6
0.4
0.4
0.-
0.2
00
600
200
Tain data size
Train data size
Yain data size
(a)
(b)
(c)
2021b) and MLPs for (a) (())-invaii Figure 2: Comparison of G-RepsNets with EMLPs (Finzi et al.
synthetic regression task with input type 27i and output type To, (b) -quiiivaan regression with input
as masses and positions of 5 point masses using representation of lype 57 + 57, and output as the inertia
matrix of type T2, (c) (O,,, 3)-invariant regression computing the matrix element in electron-muor particle scattering with input of type 4T, and output of type To-
Table 1: Comparison of MLP, G-RepsNet and EMLP on tasks using non-regular representation. Train time per epoch (in seconds), test time per sample (in milliseconds) and number of parameters for models with the same channel size of 384 for training datasets of size 1000 are provided.
Model
MLP
G-RepeNet
EMLP
Task
# params
Train time [4] Test time (ms)
# params
Train Time (4)
Test time [ms)
# params
Train time (+)
Test time (ms)
(3))-Invrrton
300,250
0,0083
0.33
30,300
0.013
0.52
480.604
3.00
7.73
03)-oquisariant
307, 200
0.0087
0.32
233,200
0.084
1.55
572.121
3.19
7.13
S0 3iimvrrnntt
302,583
0,0080
0.33
30,500
0.049
1,05
446.688
2,86
8.01
This gives a tensor of type 1007;, which is the input for the next layer. We repeat the same process of converting to To and back to T1 for the next two layers. For the final two layers, we convert all the tensors to scalars of type 100T, and process through the last two layers and use ReLU activation function in between.
of the tensor mixing process described in Section 4.3
0(3) equivvvaar model: The model contains four layers that take in input tensors cach of type To + Ti and output single tensor of lype T2. A detailed description of the four layers are as follows.
Pirst layer: Let the input and output of the first layer be XT. XT, and HT. HT, HT,, respectively. Here, Xr, denotes tensors of type T; and similarly for Hr.-
To compute HT, we first convert Xy, to type To by taking its norm and concatenating it with X7.- Let us assign this concatenated value to HT. Then, the final value of HT, is obtained by passing HT, through two lincar layers with a ReLU activation in between.
To compute Hr, we simply perform W2 Wi(X,, //1W//())/ Il) as the tensor mixing process from Section 4.3 where W,, W2 are single linear layers with no bias terms. To compute HT, we first convert Xr, to type T2 by multiplying it with an identity matrix of dimension of Xy,, Let us call this HT Then, we convert Xr, to type T2 by taking the outer product with itself. Let us call this HT., We concatenate HT and HT.. and call this Hr,- Then, we update H7g as follows. We simply perform W2(Hr, * (Hr,)))/ (Hr,)) as the tensor mixing process from Section 1.3. where Wi, W2 are single-layered linear layers with no bias terms. In all cases, the number of tensors obtained is equal to the channel size used for the experiments discussed in Section 5.1
Second and third layers: The first layer above is followed by two layers of input and output types To TT+ + T2. Let the input and output of the this type of layer be Xm, Xr,, Xy, and Hr. Hr, Hr,, respectively. Here, Xr. denotes tensors of type T; and similarly for HT. To compute HT,, we first convert Xr, and X7, to type To by taking its norm and concatenate it to XT,- Let us assign this concatenated value to HI,- Then, the final value of Hr, is obtained by passing HT, through two linear layers with a ReLU activation in between.
The rest of the computations for obtaining HT, and HT, are identical to the first layer, which is described below for completeness.
8

---

Published in Transactions on Machine Learning Research (05/2025)
300
EMLP G-Pepslet
DNLF
DNU
GiRepoles
G 6-tepoliet
230
HJP
HPP
2.0
HPP
2.00
1.5
1.0
1.00
1000
1.0
030
030
000
0.0
000
420
420
420
(a)
(b)
(c)
Figure 3: Times per epoch (in seconds) for different MLPs, G-RepsNets, and EMLPs for varying dataset sizes. Here the figures correspond the experiments described next. a) 0(5)-invariant synthetic regression task with imput type 27; and output type To, (b) 0(3) equivaarant regression with imput masses and positions of 5 point masses using representation of type 57o + 571 and output as the inertia matrix of type T2, (c) SO(1,3) (((1,,))-nvande 3)-inaariant regression computing the matrix element in electron-muon particle scattering with input of type 47' and output of lype To-
Table 2:
Comparison of EGNN and G-RepsGNN the equivvaaiaa 5-body dynamics prediction task.
Note that G-RepsGNN is constructed by simply replacing the representation in the GNN architecture from Gilmer et al. 2017 with T1 representations along with tensor mixing from Section is a specialized GNN designed for F(())equivariant tasks.
4.3
whereas EGNN
Model
Test Loss
Forward Time (ms)
Numbor of parameters
EGNN
0.0069
1.76
69215
G-RepsGNN (ours)
0.0049
2.02
106288
To compute Hr, we simply perform W2(Hr, W/ where W1, W2 are single linear layers with no bias terms.
as the mixing process from Section
To compute HT. we first convert XTo to lype T2 by multiplying it with an identity matrix of dimension of XT. Let us call this HT Then, we convert XT, to type T2 by taking the outer product with itself. Let call this HT.. We concatenate HTm, HIT,, and XTa, and call this lly,, Then, we update Hr, as follows. We simply perform W2(Hr, W1 ( Wi, W2 are single-layered linear layers with no bias terms.
the tensor mixing process from Section 4.3 where
These layers also use residual connections similar to the ones shown in Fig. 5a
Fourth layer: Finally, the T2 tensors of the output of the penultimate layer are passed through a final T2 layer, which gives the final output.
-ivaaannn model This design is identical to the design of the i(5)-invariant network above except for few changes: a) the invariant tensors is obtained using Minkowski norm instead of the Euclidean norm, b) the number of channels are decided by the number of channels chosen for this specific experiment in Section
Experimental results: We train MLPs, EMLPs, and G-RepsNet on the datasets discussed above for 100 epochs. Further details on the hyperparameters are given Scction [.I]] From Fig. 2 we find that across all the lasks, G-RepsNets perform competitively to EMLPs and significantly outperform non-equivariant MLPs. Moreover, Fig. and Tab. 1 show that G-Reps are computationally much more efficient than EMLPs, while being only slightly more expensive than naive MLPs. This shows that G-RepsNet provide competitive performance to EMLPs on equivariant tasks. Morcover, the lightweight design of G-RepsNets motivates its use in larger datasets.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Table shows mean (sud) of classification accuracies on Ro190-CTFAR10 dataset for T-GG-RepSCNN, T-- TZ-G-RepsCNN, GCNN, and T2-G-RepsGCNN for 100 epochs. The base model used here is Resnet18, and
the T2 layers in the T3 models are added only in the last layer as described in Appendix D.2 Results are
over 3 seeds.
Dataset \\  Model
CNN
T Ti-G-RepSCNN
TZ-G-RepSCNN
GCNN
D-GGRRepSGNNN
Rot90-CIFAR10 72.8 (0.2)
79.5 (0.2)
80.4 (0.4)
73.8 (0.9) 76.6 (0.5)
5.2 Modelling Dynamic N-Body System with GNNs
We consider the problem of predicting the dynamics of N charged particles given their charges and initial positions, where the symmetry group for equivariance is the orthogonal group O(3). Each particle is placed at a node of a graph 9 {V,E), where V and & are the sets of vertices and edges. We use the N-body dynamics dataset from Satorras et al. (2021), where the task is to predict the positions of N = 5 charged particles after T = 1000 steps given their initial positions E R3x5, velocities E IR3x6, , and charges E {-1,1)3.
Dataset details:
Model design and experimental setup: Let the edge attributes of G be asj. and let h be the node feature of node uy V at layer I of a message passing neural network (MPNN). An MPNN as defined by Gilmer et al. (2017) has an edge update, may = de (h,, hl a.j) and node update h,*1 (n(,, m.), m, Ejen()) Maj. where the and on are MLPs corresponding to edge and node updates, respectively.
We design G-RepsGNN by making small modifications to the MPNN architecture. In our model, we use two edge updates for To and T1 tensors, respectively, and one node update for Ti update. The two edge updates are Mig.7, = Te (Illell, I [ an)) Maj.T, = -e.T, (h1,, hi, aij), where |I-II obtains To tensors from Ti tensors for the Euclidean group, Ve,ra (.) is To-layer MLP, and Φe,ri (-) is a Ti-layer made of an MLP without any pointwise non-linearities or biases. The final edge update is obtained as mij = Mij,T maj.r./ [I m l|- Finally, the node update is given by h!#1
@n.7; (hg, m.), where m, EjeN(o) my and Φ1 (-) is an MLP without any pointwise non-lincarities or biases. Thus, the final node update is a Ti tensor. We compare G-RepsGNN with EGNN (Satorras et al. 2021 which is a popular equivariant graph neural network.
We closely followed Satorras et al. (2021) to generate the dataset: we used 3000 trajectories for train, 2000 trajectorics for validation, and 2000 for test. Both EGNN and G-RepsGNN models have layers and were trained for 10000 epochs, same as in Satorras et al. (0211)
Results and Observations: From Tab. 2 we find that even though EGNN is specialized architecture for the task, G-RepsGNN performs competitively to EGNN. Note that here the comparison is made to EGNN since it is a computationally efficient expressive equivariant model just like G-RepsGNN, although restricted for processing graphs. Here our goal is not to achieve state-of-the-art results on this task but to simply show that our model is competitive with popular models even with minimal design changes to the uon-equivariant base model MPNN. Further results of test losses and forward times for various other models are reported in Tab. 7 in Appendix F.T for completeness. Since G-RepsGNN has a comparable computational complexity to EGNN, it is computationally much more efficient than many specialized group equivariant architectures that use spherical harmonics for -(uuvvauuvaaiaaee as noted from Tab.
5.3 Second-Order Image Classification
We perform two sets of experiments: i) the first, we train various image classification models from scratch and compare them with G-RepsNet and ii) we perform equivariant finetuning of non-equivariant models with higher-order tensors, hence extending the equivariant finetuning method of Basu et al. (2023b) to sccond-order tensors.
Dataset: For ablation studies to understand the effect of second-order tensors in image classification and for experiments involving training from scratch, we test on different datasets obtained by applying random rotations to the CIFARIO datasct. When the random rotations are a multiple of 360 for integer n, we t call the dataset Rot 360 CIIARR1,, else if the rotations are by arbitrary angles in (0, 360), we simply call it Rot-CIFAR10.. These rotations are applied to ensure that the dataset exhibits the C, of multiples of
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table Table shows mean (std) lest accuracies for equituning using a pretrained Resnet with RoL90-CIFARIO and Galaxy10. For the T2 models, the T2 layers are added only in the last layer.
Dataset's Model
Finetune
Ti-Equitune
T-- Tz-Equit
Rot90-CIFAR.I
82.7 (0.5)
88.1 (0.3)
89.6 (0.3)
Galaxy10
76.9 (3.2)
79.3 (1.6)
80.7 (4.0)
0.56
Test Accuracy
0.52
CNN
TL-G-RepSCNN
180'
0.48
Re-G-RN
270'
T2-GCNN
1
2
7
Tj Features
Tp Features
Layer Position
(a)
(b)
Figure 4: In (a), we analyze the performance of a rot90-equivariant CNN with convolutional layers and 5 fully connected layers on CIFAR-C Here, T2 representations are introduced in layer E [l,..., 8). We find that using T/ representations in the final layers of the CNN easily outperforms non-equivariant CNNs as well as traditional equivariant representations with T₁ reprosentations. (b) shows the Tj and T2 features obtained from one channel of pretrained Resnet corresponding to T Ti-qquitune and Tz-equitune, respectively.
360 -degree rolations or SO(2) symmetry, which helps lest the working of equivariant networks. For our experiments on equivariant finetuning, we test on Rot90-CIFAR10 and Galaxy10 without any rotations. We do not apply rotations on Galaxy10 since naturally has the C4 symmctry.
Experimental setup and model design: We first design ro190-equivariant CNN with 3 conv layers followed by 5 fully connected layers and train it from scratch on CIFAR10 with random rotations. We use Ti representations for the first layers and use T22 representations for the rest, where the T2 representations are obtained by simple outer product of the Tj representation in the group channel dimension. Fig. 5 shows a simple way to add residual connections in G-RepsNet as well as a general architecture for CNNN Tt. is easy to verify the equivariance maintained for both Ti and T2 for regular representations. to see that because of our simple design, the number of parameters for the base CNN, T CeNG and T2- 12-G-RepsCNN are exactly the same (about 150K parameters, and takes only about 0.8ms for a forward pass Nvidia A5500). We train each model for 10 epochs. The results reported in Fig. 4a indicate that using Th representations in the later layers of the same network significantly outperforms both non-equivariant well as equivariant basba CNNs. Fig. 4b provides visualization of the T2 features compared to the T2 features. The TiK or T2 features are convered to C,,-invarian To features by taking the mean along the group dimension. Hyperparameter values are provided in Appendixs D.2 We compare with bascline equivariant
is also easy
architectures such as GCNNs (Cohen Welling 2016a) and E (2)-CNNNs (Weiler & Cesa 2019).
First, for comparison with GCNNs, we use CNN well as GCNN our base model for constructing G-RepsNets. We use the resnet architecture (He et al., 2016) our CNNs. For GCNNs and E(2)-CNNs, we simply replace the convolutions with group convolutions (Cohen & Welling 201 and E(2)-CNNs (Weiler & Cesa 2019), respectively in the same CNN and adjust the channel sizes to ensure nearly equal number of parameters.
We design two G-RepsCNN architectures: a) Ce-G-Re,, where each layer has a T2 representation, and b) 77-G-RepsCNN, where all the layers except the last layer use T, representation and the last layer uses T₃ representation. Using GCNN the base model, we construct TZ-G-RepsGCNN, by simply replacing the CNN model with GCNN in the Ty Ge-G-Rep,NNN Note that do not construct T--GNN as it results in
11

---

Published in Transactions on Machine Learning Research (05/2025)
Table Table shows mean (std) of classification accuracies on Rot-CIFAR10 dataset (CTFARIO with random rotations in (-180", +180*) for various G-RepsCNNNs and E(2)-CNNs with different group equivariances, and tensor orders. he base model used here is Resnet18, and the T2 layers in the T2 models are added only in the last layer as described in Appendix D.2 fixed seeds.
All models are trained for 100 epochs and results are over 3
Model CNN
Equivariance
Tensor Orders
Test Acc.
65.21 (0.4)
T T-GG-RepCCNN
C8
(T:)
73.4 (0.4)
T2- Tz-G-RepsCNN
C8
(2)
73.8 (0.4)
E(2)-CNN
C8
(7:)
49.6 (1.6)
T2- E2-)
C8
(Ti, T2)
57.3 (1.4)
Ti-G-RepCCNN
C16
(Th)
73.8 (0.1)
Tz-G-RepSCNN
C16
(Ti,
75.2 (0.5)
E(2)-CNN
C16
(Ti)
46.8 (0.8)
T-- TEN
C16
(Ti. T2)
55.4 (1.9)
To-Layer
xk
(n k)
Ti
Ti-Layer
T₁
T2
Ti-Layer
T2
Tz-Layer
T2
To Layer
To Outputs
Inputs
Outputs
Inputs
(a) Simple Tj residual connections
(b) T2-CNN design
Figure 5: (a) shows simple way to add residual connections in Net. (b) shows the architecture used for T2 CNNs and equituning, where the first k layers are made of Ti-layers to extract features, then the extracted features are converted in T2 tensors, which are then processed by Ty-layers. Finally To tensors, i.e., scalars are obtained as the final output.
the same model as GCNN. These models are compared on the Rot90-CIFAR10 dataset. The base architecture for constructing the G-RepsCNNs is ResNet18 and the T2 layers are only used in the last layer. Here, we note that, if we decompose the features G-RepsCNNs in terms of irreps, our design can be seen as special case of Steerable CNNs (Weiler & Cesa. 2019), except that for constructing G-RepsCNN, we do not need to analytically solve for the equivariant layers. Moreover, in some cases such as SE(3) (tanfor)rrns Fuchs et al. (2020), even when the analytical solution is given, they can be computationally expensive.
For comparison with E(2)-CNNs, we perform a similar comparison as that with GCNNs. Here we work with NOt-CIFAR1O dataset. This is because E(2)-CNNs are equivariant to larger groups than that of 90-degree rotations, so, we want to test the model's capabilities for these larger group symmetries. Here we build four variants of G-RepsCNNs: CNGG-eepSNNN, T CN-G-Rops cach for both Cs and Cis equivariance. Here Cn for {8, 16) corresponds to the groups of 360 " -degree rotations. The layer representations for CNN are all Ti tensors of the On group. Whereas for G-RepsCNN, all layer representations except the last layer are Ti representations and the last layer uses T2 representations. Note that T C2- is the same as the traditional E(2)-CNN for the Cn group, which has Ti representation at each layer. T2-G-Rep (2) has Ti representations for each layer except for the last layer that uses a T2 representation. That is, the base architecture for constructing the ())- is E(2)-CNN with similar number of parameters and the Tr layers are only used in the last layer. Please check Appendix D.2 for additional details on architecture.
For experiments on equivariant finetuning, we take the equituning algorithm of Basu et al. (023b) that uses ti representations and extend it to use T2 representations in the final layers. We use pretrained Resnet18 as
12

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6:
The table shows mean (sld) of percentage relative mean square errors over 3 seeds, and mean training time (in seconds) per epoch over 5 epochs and mean testing time per sample (in milliscconds) over
100 samples for solving PDEs with FNOs. G-RepsI'NOs use regular 7i representations with FNO as the base model. Both FNO and G-RepsFNO contain about 468K trainable parameters while G-FNO is much larger with about 3.4M parameters.
Model
PNO
G-RepalNNO
G-FNO
Datuset
MSE
Train time (s)
Test time (ms)
MSE
Train Time
Tist time (ms))
MSB
Train time (s)
Test time (mus)
NS
8.41 (0.4)
49.8
7.46
5:31 (0.2)
53.9
7.71
4.78 (0.4)
100.9
21.20
NS-Sym
4.21 (0.1)
19.2
5:31
2.92 (0.1)
20.8
7.52
2.24 (0.3)
43.8
20.99
our non-equivariant base model and perform non-equivarianl finetuning and equivariant finetuning with Ti and T2 representations. Additional experimental details are provided in Appendix D
Results: In Tab. 3 and Tab. 5 we provide the results for training from scratch. From Tab. 3 and
we make two key observations: a) Te-G-RepsCNNN are competitive and oflen outperform the baselines GCNNs and E(2)-CNNs, b) T2 features, when added to the baselines to obtain Tz-G-RepsGCNNG and Tz- T2-G-RepsE((2CNNN)
they outperform the original Ti counterpart for both Cs and Cie equivariance. This shows the importance of higher-order tensors in image classification. Thus, we not only provide competitive performance to baselines using our models but also improve the results from these baselines by adding T2 features in them. Finally, from Tab. 4 we find that on both 1ot90-CIFAR10 and Galaxy10, Tz-oquitune casily oulperforms equitune, confirming the importance of T2 features.
5.4 Solving PDEs with FNOs
Datasets and Experimental Setup: We consider two versions of the incompressible Navier-Stokes equation from Helwig et al. (2023); Li ct al. (2021). The first version is Navier-Stokes equation without any symmetry (NS dataset) in the data, and a second version that does have 90º rolation symmetry (NS-SYM dataset). The general Navier-Stokes equation considered is written as,
(xw(x,t) + u(x, t)- vw(x, = vAw(a, t) + f(x), V u=((zt))= and w<x 0) = wo(x),
(2)
where w(z, E IR denotes the vorticity at point (z,,), wo(x) is the initial velocity, u(æ,t) R² is the velocity at (x, (), and y == 10-1 is the viscosity coefficient. I' denoles an external force affecting the dynamics of the fluid. The task here is to predict the vorticity at all points on the domain x E [0, 1]2 for some t, given the previous values of vorticity at all point on the domain for previous T steps. As stated by Helwig ct al. (2023), when f is invariant with respect to 90º rotations, then the solution equivariant, otherwise not. We use the same forces f as Helwig et al. (2023). For non-invariant force, we use [((,, = 0.1 (2x + x22) + cos (2 (m) + x2))) and as invariant force, we use fine = 0.1 (4πZι) + cos (4TTæ2)). We use T' 20 previous steps inputs for the NS dataset and T = 10 for NS-SYM and predict for t T + 1, same as in Helwig et al. 20223 We train our models with batch size 20 and learning rate 10-3 for 100 epochs.
Model design: We use the FNO and G-FNO models directly from Helwig et al. (023)). And we construct G-RepsFNO by directly using Tj representation corresponding to the C4 group of 90 rotations for all the features. Further, G-RepsFNO uses FNO as the base model.
Results and Observations: In Tab. G. we find that G-RepsFNO clearly outperforms traditional FNOs on both datasets NS and NS-SYM. Note that the NS dataset does not have rot90 symmetries and yet G-RepsFNOS outperform FNOs showing that using equivariant representations may be more expressive for tasks without any obvious symmetrics as was also noted in several works such as Cohen & Welling (2016a); Helwig et al. (2023). Moreover, we find that the G-RepsFNO models perform competitively with the more sophisticated, recently proposed, G-FNOs. Thus, we gain benefits of equivariance by directly using equivariant representations on non-equivariant base models and making minimal changes to the architecture. Further, in Tab. d we show that G-RepsFNOs are computationally much more efficient than the more sophisticated G-FNOs.
13

---

Published in Transactions on Machine Learning Research (05/2025)
6 Limitations
Here we provide the limitations to our method to the best of our knowledge.
Our method constructs lightweight networks equivariant to arbitrary matrix groups. However, the universality of our method is limited to orthogonal groups as discussed in $. 4.4 Universality for efficient practical equivariant architecture to general matrix groups is challenging open problem and left for future research. Nevertheless, since orthogonal groups form large class of groups of practical importance, we believe that the scalability of our network compared to equivariant networks as general as ours, e.g. Finzi el al. (2021b), Villar et al. 22021) makes it an important equivariant network construction method.
While the GeRepsNet architecture is indeed general and applicable to arbitrary matrix groups, the function that is invariant to group action may not be easy to define or compute in general, unlike the case of orthogonal groups (nn and 0)),
Tensor operations used in construction in Section can be expensive for tensor of very high order. But in the majority of applications we use tensor multiplications only when the applications require such higher-order tensor features, e.g. in Section 5.1 In such cases, the applications themselves require higher-order features in the network to achieve good performance. E.g. the (3)-eqquivaiiant regression task in Section 5. has socond-order tensor outputs, hence, any equivariant network for this task must have higher-order tensors and hence, are computationally more expensive than non- equivariant networks. In sccond-order image classification in Section 5.3 we only use second-order tensor in the final few layers where it leads to significant improvement in performance and the feature dimensions are relatively not too large, hence, maintaining comparable computational complexity (ef. Tab.
10 to the equivariant networks using first-order features.
The G-RepsNet architecture is designed to be simple and efficient (and universal for orthogonal groups) which means that we avoid tensor decomposition (e.g., required if T2 tensors need to be converted to 7i). However, this leads to higher memory requirements. In cases with more extreme memory constraints, lensor decomposition may be worth despite the slower speed.
7 Conclusion
We present G-RepsNet,, a lightweight yet expressive architecture designed to provide equivariance to arbitrary matrix groups. We find that G-RepsNet gives competitive performance to EMLP on various invariant and oquivariant regression tasks taken from Finzi et al. (2021b), at much less computational expensc. For image classification, we find that G-RepsNet with second-order lensors outperforms existing equivariant models such as GCNNs and E(2)-CNNs well methods such as equitune when trained using pretrained models such Resnet. Further illustrating the simplicity and generality of our design, show that using simple first-order tensor representations in G-RepsNet achieves competitive performance to specially designed equivariant networks for several different domains. We considered diverse domains such as PDE solving and N-body dynamics prediction using FNOs and MPNNs, respectively, as the base model.
References
Sourya Basu, Pulkit Katdare, Prasanna Sattigeri, Vijil Chenthamarakshan, Katherine Rose Driggs-Camphan Payel Das, and Lav R Varshney. Efficient equivariant transfer learning from pretrained models. In seirth- Conference Neural Information Processing Systems, 2023a.
Sourya Basu, Prasanna Sattigeri, Karthikeyan Natesan Ramamurthy, Vijil Chenthamarakshann Kush R Varshney, Lav R Varshney, and Payel Das. Equi-tuning: Group equivariant fine-tuning of pretrained models. In Proceedings of the AAAI Conference Artificial Intelligence, 2023b.
Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pp. 992-1002. PMLR, 2020.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. In International Conference on Learning Representations, 2022.
Таco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning, pp. 2990 2999. PMLR, 2016a.
Taco S Cohen and Max Welling. Stecrable cmns. In International Conference on Learning Representations, 2016b.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the TEFE/CV International Conference on Computer Vision, pp. 12200 12209, 2021.
Marc Finzi, Max Welling, and Andrew Gordon Wilson. EMLP github. https://github.cr com com/mffmm//ooooooooo mf inzi/ equivariant-MLP 2021a. Accessed: 2025-03-13.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning, pp- 3318 3328. PMLR, 2021b.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970 1981, 2020.
Justin Gilmer, Samuel S Schocnholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp. 1272-1272. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Jacob Helwig, Xuan Zhang, Cong Fu, Jerry Kurtin, Stephan Wojtowytsch, and Shuiwang Ji. Group equivariant fourier neural operators for partial differential equations. In International Conference on Machine Learning, 2023.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Newral networks, 2(5):359- 1989.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tumyasuvnakool,, Russ Bates, Augustin Zidck, Anna Potapenko, ct al. Highly accurate protein structure prediction with alphafold. Nalure, 596(7873):583 589, 2021.
Sékou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equiv- ariance with learned canonicalization functions. In International Conference on Machine Learning, pp- 15546 15566. PMLR, 2023.
Jinwoo Kim, Dat Tien Nguyen, Ayhan Suleymanzade, Hycokjun An, and Seunghoon Hong. Learning probabilistic symmetrization for architecture agnostic equivariance. In Thirly-seventh Conference Newral Information Processing Systems, 2023. URL
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi-body systems with symmetric energies. arXie preprint arXio:1910.00 1900000900 2019.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pp. 2747-2755. PMLR, 2018.
Risi Kondor, Hy Truong Son, Horace Pan, Brandon Anderson, and Shubhendu Trivedi. Covariant composi- tional networks for learning graphs. arXio preprint ar 801.00444 2018.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural compulation, (4):541 551, 1989.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for paramctric partial differential equations. In International Conference on Learning Representations, 2021.
Arnab Kumar Mondal, Siba Smarak Panigrahi, Sékou-Oumat Kaba, Sai Rajeswar, and Siamak Ravanbakhsh. Equivariant adaptation of large pretrained models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Omri Puny, Matan Atzmon, Edward J Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2021.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing. In International Conference Machine Learning, pp. 2892 2901. PMLR, 2017.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks, In International Conference on Machine Learning, pp. 1823-9332 PMLR, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-oqquivriian neural networks for 3d point clouds. arXiv preprint (rXii:1802.082) 2018.
Soledad Villar, David W Hogg, Kate Foorey-Fisher, Weichi Yao, and Ben Blun-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems, 34:28848 28863, 2021.
Maurice Weiler and Gabriele Cesa. General E (2)eqquvvri steerable cnns. Advances in neural information processing systems, 32, 2019.
Hermann Weyl. The classical groups: their invoriants and representations, volume 45. Princcton university press, 1946.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEFF conference computer vision and pattern recognition, pp. 5028 5037, 2017.
Dmitry Yarotsky. Universal approximations invariant maps by noural networks. Constructive Approzimation, 55(1):407 474, 2022.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in newral information processing systems, 30, 2017.
Appendix
A Additional Details on Related Works
A.1 EMLPs and Universal Scalars
EMLPs Given the input and output types for some matrix group, the corresponding tensor representations can be derived from the given base group representation Using these lensor representations, one can solve
for the space of linear equivariant functions directly from the obtained equivariant constraints corresponding the tensor representations. Finzi et al. (2021b) propose an elegant solution solve these constraints by computing the basis of the linear equivariant space and construct an equivariant MLP (EMLP) from the computed basis. Our work is closest to this work as use the same data representations as Finzi al.
16

---

Published in Transactions on Machine Learning Research (05/2025)
To-Layer
Tj
Ty-Layer
Tj
Ij
Ti-Layer
Ti
Inputs
Outputs
Inputs
Outputs
(a) Vector neurons (0(3) only)
(b) Equitune
Figure 6: (a) and (b) show layers from vector neurons Deng et al. 20211 and equitune Basu et al. 2023b
which are special cases of G-RepsNet...
(2021b), but we propose a much simpler architecture for equivariance to arbitrary matrix groups. Because of the simplicity of our approach, we are able to use it for several larger datasets, which is in contrast to Finzi cet al. (2021b), where the experiments are mostly restricted to synthetic experiments. Moreover, using these bases are in general known to be computationally expensive (Fuchs et al. 2020
Universal scalars Villar al. (2021) propose method to circumvent the need to explicitly use these equivariant bases. The First Fundamental Theorem of Invariant Theory for the Euclidean group O(d) states that "a function of vector inputs returns an invariant scalar if and only if it can be written as a function only of the invariant scalar products of the input vectors' (Weyl 1946). Taking inspiration from this theorem and a related theorem for equivariant vector functions, Villar et al. (2021) characterize the equivariant functions for various Euclidean and
groups. They further motivate the construction of neural networks taking the invariant scalar products of given tensor data as inputs. However, the number of invariant scalars for N tensors in data point grows as N hence, making an impractical method for most real life machine learning datasets. Hence, their experiments
also mostly restricted to synthetic datasets like in EMLP.
Morcover, Villar et al. 2021 S. 5) show that even though the number of resulting scalars grows proportional to 2 when the data is of dimension d, approximately N x (d + 1) number of these scalars is sufficient to construct the invariant function. But, it might not be trivial to find this subset of scalar for real life datasets such as images. Hence, we propose to use deeper networks with equivariant features that directly take the N tensors input, instead of N scalar inputs, which also circumvent the need to use equivariant bases. Additional related works and comparisons in Section
A.2 Special Cases and Related Designs
Here, we look at existing group equivariant architectures popular for their simplicity that are special cases or closely related to our general design.
Vector Neurons Popular for its lightweight (0(()-cquivariant applications such as point cloud, the vector neurons (Dng et al.) 2021) serve as a classic example of special cases of our design as illustrated in Fig. 6a Their Ti-layer simply consists of a linear combination Ti inputs without bins terms, same as ours. The To-layer first converts the Ti tensors into To tensors by taking inner products. Then, pointwise non-linearities are applied to the To tensor and then mixed with the Tj tensors, by multiplying them with Tj tensors and further linearly mixing the Ti tensors.
Harmonic networks Harmonic networks H-nets WWrrall et al. 2017 employ a similar architecture to ours and vector neurons, but specialized for the S0(2) group. They also take as input Ti inputs, then oblain the To scalars by computing the Euclidean norms of the inputs. All non-linearities are applied only lo the scalars. The Ti tensors are processed using linear circular cross-correlations that preserve equivariance. Further, higher order tensors are obtained by chained-cross correlations. The use of cross-corrclation is very different from our design but it designed in a similar spirit of building tensors of various orders and construct simple, yet expressive equivariant. features.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Deepsets Deepsets Zaheer al. 2017) is popular architecture equivariant to permutations. Here, we show how Deepsets can be constructed using non-regular representation for the S(n) group of permutations using the G-RepsNet construction in $. 4
First, we recall that the Deepsets architecture. Suppose the imput to the Deepsets is given by X = [X1, X2, Xs], then the output of each layer of Deepsets is given by Y [Yi, 2,, Ya), where Y, = W1(X,) + W2(2 X;), where W1 and W2 are two learnable weight matrices. Usually, we also have pointwise activation functions used on the output, which we ignore here since they do not affect the equivariance of permulation groups. It. is easy to verify that the Deepsets layer is equivariant to permutations on X, since, if we replace X; by Xj in the input for A j, then Y, is replaced by Y2 in the output.
Now, consider the construction of G-RepsNet corresponding to the S(n) group using non-regular representation from Section 4 For the input X = [X1, X2, Xs), the group channel dimension is equal to 1. We first obtain the To and Ti representations of the data as XT. Σi=1 X, Xr, X2, Xs]. Then, we obtain Yr, = [W1 (xr)), 1], which is a concatenation of the invariant term W1 (x()))) and a scalar independent of the input. Similarly, obtain the Hr, = [1, (2x)), where (2(XT,) == [W2(X1), (2())), W2(X3)] Now, we get Y = -T, * Yr.- = [Wi W2((,,)) )l- From Y/, it is easy to obtain Yr, by summing the two components (XT,), and (XT,,)) of Y/, Thus, establishing the similarity between Deepsets and G-RepsNet for the permutation group. Note that here we used the inv(.) as a constant, which is a valid choice, to obtain a similar form of the output as Deepsets.
Equitune Finally, recent works on frame-averaging such equitune and related symmetrization tech- niques (asu et al. 2023a Kim et al. 2023 Kaba et al. 2023 Mondal et al.) 2023) construct equivariant architecture by performing some sort of averaging over groups. This can be seen as using a regular Ti representation as the input and output type as illustrated in Fig. 6b These works have mainly focused on exploring the potential of equivariance in pretrained models. In this work, we further explore the capabilities of regular Tj representations and find their surprising benefits in equivariant tasks. Moreover, this also inspires us to explore beyond regular Ti representations, e.g., find T2 representations can yield better results than Ti representations when used in the final layers of a model for image classification.
B Proof of Equivariance
Here we provide the proof of cquivariance of a G- G-RepsNet layer to matrix groups. Further, since stacking equivariant layers preserve the equivariance of the resulting model, the equivariance of the G-RepsNet model follows directly. The argument is similar to the proof of equivariance of vector neurons to the SO(3) group.
First, consider regular representation. Note from Section that the group channel dimension is treated like a batch dimension in regular representations for discrete groups. Thus, any permutation in the input naturally appears in the output, hence, producing equivariant output.
Now we consider non-regular representations. Assuming that the input to a G-RepsNet layer consists of tensors of types To: T Ti....,77 Tn, we first note that the output of the To-layer in Fig. Ta is invariant, following which we find that the T,-layer outputs equivariant T, tensors.
The output of the layer is clearly invariant since all the inputs to the network are of type To, which are already invariant.
Now, we focus Ty-layer. Recall from Section that T, layer consists only of linear networks without any bias terms pointwise non-linearities. Suppose the linear net work is given by stack of linear matrices. We show that any such linear combination performed by a matrix preserves equivariance, hence, stacking these matricos would still preserve oquivariance of the output. Let the input tensor of type T, be X E Rexk i.e,, we have e tensors of type T; and size of the representation of each tensor equals to k. Consider a matrix W E Re' which multiplied with X gives Y = W x X E Re'xk, where Y linear combination of the e input tensors each of type T,. Let the group transformation on the tensor T. be given by G E R Then the group transformed input is given by X' = X x G e Bexk, The output of X' through the Ty-layer is given by Y' = W& x X x G E R xk = (W x X) x G = Y x G, where the second last equality follows from the associativity property of matrix multiplication. Thus, each Ti-layer is equivariant.
18

---

Published in Transactions on Machine Learning Research (05/2025)
C On the Universality of the G-RepsNet Architecture
Proof to Thm. [ Let the tensors of type i at layer be written as Hi Given input (x1,X2 . E R of type Ti, we construct a G-RepsNett architecture that can approximate h by taking help from the approximation properties of multi-layered perceptron (Hornik et al. 1989).
Let the first layer consist only of Ta-layers, i.e., linear layers without any bias terms such that the obtained hidden layer Hi is of dimension Rdx(x*+n) and consists of the Ti tensors X; + X, for all i, E (1,...,9) n n.. This can be obtained by a simple linear combination. Now, construct the
and X; for all i second layer by first taking the norm of all the 71 tensors, which gives ||x + ||X, + x (X,, X,) for all i.J E (1....,7o n and ||x;|| || for all E (1,...,,,) n). Then, using a simple linear combination of the converted To tensors give (X,,X) for all i.j E (1....,... Finally, passing (X,, X,) for all i, E {1.....m) through an MLP gives Hg. Now, from the universal approximation capability of MLPs, can approximate f from equation ! Thus, we obtain the function h from Lem. I
Now, recall from Villar al. 2021
statement similar to Lem. but for vector functions.
Lemma 2 (Villar et al. (2021)) ). A function of vector inputs returns an equivariant vector if and only if it can be written linear combination of invariant scalar functions times the input vectors. That is, given inpul veclors (X1, X Xa), X; E Rd, any equivarianl. veclor funclion h : Rdxx - IR can be wrillen as
n
X2,, Xa) - Σ
(3)
where (X,, x,) denotes the inner product between X, and Xj, and fis are some arbitrary functions.
The proof closely follows the proof for Thm. D Let the tensors of type at layer I be written Hi- Given input (X1, X2...,, E Rd of type Ti, we construct G-RepsNet architecture that approximate h by taking help from the approximation properties of multi-layered perceptron Hornik
Proof to Thm. 2
ct al. 1989).
Let the first layer consist only of Ti- Ti-layers, i.e., linear layers without any bias terms such that the obtained hidden layer H. is of dimension (dx(1++*) and consists of the Ti tensors X, + X, for all i,j E [I....,n} n and X, for all i [I,...,n).. nJ. This can be obtained by a simple linear combination.
Let the second layer consist of both a To layer and T1 layer. Let the To layer output, Hy be (X,, X2) for all i E (1....,n) nn) and || for E (1,...,n) in similar way done in the proof for Thm. 1 And let the Ti layer output, H1, be X; for E nI.
Again, let the third layer also consist of a To layer and Ti layer. Let the To layer consist of MLPs approximating the output ||X x fe((X,, xz)R)=) for t E {1,...,n} nn. Denote [|xell x fi((X, ,, X,Y;, -- as H3... Then, let the Ti layer consist of first mixing the scalars H with X, as described in Section as
H = X₂ x ||xell
where H&d for t E [I, represent the output of the Ti layer of the third layer. Note that from the universal approximation properties of MLPs (Hrrikk et al. 1989), we get that H approximates X, x S(((,, Xy),z=)- Finally, the fourth layer consists of a single Ti layer that sums the vectors H33' for E n.....,)), which combined with Lem. 2 concludes the proof.
Thus, we find that simple architocture can universally approximate invariant scalar and equivariant vector functions for the Odd) or (11 groups. This is reminiscent of the universality property of single-layered ML.P. However, in practice, deep neural networks are known to have better representational capabilities than
19

---

Published in Transactions on Machine Learning Research (05/2025)
Table Test loss and running time for various neural network architectures for the 5-body dynamics prediction task. The results for SEGNN are taken from Brandstetter et al. (2022). rest are taken from Satorras et al. 2021).
Model
Test Loss
Forward Time
Linear
0.0819
0.0001
SE (3) Transformer Fuchs et al. (2020)
0.0244
0.1346
Tensor Field Network Toomas al.
(2018)
0.0155
0.0343
Graph Neural Network
Gilmer el. (2017)
0.0107
0.0032
Radial Field Network
Köhler al.
2019
0.0104
0.0039
EGNN Satorras et al. (2021
0.0071
0.0062
SEGNN Brandstetter
(2022
0.0043
0.0260
single-layered MLP. In similar way, in practice, we design deep equivariant networks using the G-RepsNet architecture that provides good performance on a wide range of domains.
D Additional Experimental Details
D.1 Comparison with EMLPs
Here we provide the learning rate and model sizes used for the experiments on comparison with EMLPs in Soction 5.1
For each task and model, we choose model sizes between small (with channel size 100) and large (with channel size 384). Similarly, we choose the learning rate from {10-3, 3 x 10 10-3).
In the (6))-invariant regression task, for MLPs and EMLPs, size 384. Whereas for G-RepsNets, we use a learning rate of 10 and channel size 100.
learning rate of 3 10-3 and channel
For the equivvaiarr task, we use learning rate 10 3 and channel size 384 for all the models.
For the O(1,3) ((1,3)-ime -iiviiari regression task, we use learning rate of 3 x 10-3 for all the models. Further, we use a channel size of 384 for MLPs and EMLPs, whereas for G-RepsNets, a channel size of 100 was chosen as it gives better result.
D.2 Second-Order Image Classification
Training CNNs from scratch: The CNN used for the ablation experiments for the plot Fig. 4a consists of convolutional layers each with kernel size 5, and output channel sizes 6, 16, and 120, respectively. Following the convolutional layers are fully connected layers, each consisting of features of dimension of 120. For training from scratch, we train each model for 10 epochs, using stochastic gradient descent with learning rate of 10-3, momentum of 0.9. Further, we also use stepLR learning rate scheduler with y of 0.1, step size of 7, which reduces the learning rate by factor of y after every step size number of epochs.
Comparison GCNNs and F(22-CNNs when trained from scratch: For each of the models, we use the Resnet18 architecture (cither with naive convolutions or group convolutions). We train each model for 100 epochs using stochastic gradient descent, with learning rate 10-3, momentum of 0.9, weight decay of 5 10-4
Sccond-Order Finctuning: For finetuning the pretrained Resnet18, we use epochs, using stochastic gradient descent with learning rate of 10-3, momentum of 0.9. For equivariant finetuning with T2 representa- tions, we first extract Ti featured from the pretrained model same as done for equituning (Basu et al. 2023b), following which we convert to T2 representations using a simple outer product. Once the desired features are obtained, we pass it through two fully connected layers with ReLU activation function in between obtain the final classification output.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Table 9: Table shows mean training time (in minutes) taken by each of the models per epoch for the experiments in Tab. 3 We note that Tz-G-RepsCNN computationally comparable 7--G-RepSCNN the second-order features in TN-G-RepsCNN are only used in the final layers with small dimensions. Similarly, GCNN is computationally comparable to T2-G-RepsGCH G-RepsGCNN.
Dataset \\  Model
CNN Ti-G-RepSONN
T2- Tz-G-RepsCNN
GCNN T-- TZ-G-RepSGCNN
Rot90-CIFAR10 1.07
3.19
3.21
5.50
5.50
Table 10: Table shows mean Training time (in minutes) laken by each of the models per epoch for the experiments in Tab. [
We note that TZ-G-RepSCNN is computationally comparable Ti-G-RepsCNN the second-order features in T2-G-RepSCNN are only used in the final layers with small dimensions. Similarly, CNN)-CNN is computationally comparable to (2-G-RepsE(2)CNN,
Model
Equivariance
Tensor Orders
Training time (min)
CNN
1.55
Gi-G-RepSCNN
C8
(Ti)
6.77
T--G-RepSCNN
C8
(Tr)
6.8
E(2)-CNN
C8
(Ti)
5.52
T2- (2)
C8
(2)
5.52
CepS-ReN
C16
(Ti)
9.80
CNNG-RepSCNN
C16
(Ti, T2)
9.17
CNN)-CNN
C16
(Ti)
6.81
T (2=G-RODSE(2)-CCN
C16
T2)
6.81
E Additional Results
E.1
Comparison of time for forward passes in GNN models
We present the results of forward pass times for various equivariant and non-equivariant graph neural network models in Tab. 7 taken directly from Satorras et al. [2021). It shows that networks constructed from equivariant bases such as tensor field networks (TFNs) and (E(()--auivariint transformers can be significantly slower than non-equivariant graph neural networks.
E.2 Additional results on compute required for second-order image classification using GCNNs, E(2)-CNNs
Tab. and 10 show the average training time taken per epoch for various models for the experimental results in Tab. and respectively.
21


Reviews:
[{"review_id": "QhqPFwVumv", "reviewer": "Reviewer_mbNm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work presents a novel approach to constructing equivariant networks using operations on tensor representations. It generalizes multiple related works into a unified framework while showing good performance and efficiency. It is applicable to arbitrary symmetry groups (provided a tensor representation) and exhibits universality for orthogonal groups, making it broadly applicable to various domains and tasks with different symmetries.\n\nThe breadth of experiments is appreciated, but each one seems to compare the proposed architecture(s) to a single related work and sometimes a non-equivariant baseline. Are there any other relevant architectures to compare against for each experiment? \n\nEach of the “model design” sections are quite heavy to read in paragraph form, making them lack clarity. I would recommend instead presenting each in pseudocode, then using more natural language to describe them at a higher level. Including a codebase in addition to this change would further clarify exactly how your methods work.\n\nGenerally, the comparisons of G-RepsNet to related works is useful to show how it generalizes many of them. However, some relationships are not clear, partially due to the lack of clarity already mentioned as well as some missing connections. Specifically, how does your proposed invariant gating (Sec. 4.3 Neural Processing, 4th paragraph) compare to the Gated Nonlinearities proposed in Weiler et al. (2018) and then used in EMLPs (Finzi et al., 2021)? Also, tG-RepsNet seems more similar to vector neurons than to EMLPs, yet the latter seems to take precedence in multiple sections of the paper.\n\nTo strengthen your paper, you should include experimentation for groups beyond orthogonal groups in the non-regular representation case. Even if related works don’t provide such datasets, you can find, collect, or generate your own datasets to show the breadth of your proposed methods. The permutation symmetry group is an obvious candidate that at least Zaheer et al. (2017) has already provided datasets and experiments for.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces G-RepsNet, a new type of equivariant neural network that uses tensor representations and simple tensor operations to achieve generality, efficiency, and scalability. G-RepsNet is proven to be a universal approximator for functions equivariant to orthogonal groups, and it outperforms existing methods on various tasks, including image classification and N-body dynamics prediction, while being computationally efficient. This work provides a seemingly significant advancement in building equivariant neural networks, making them more practical for a wider range of applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses noted above.\n\nAvoid putting claims and analysis in the caption of tables and figures. For example, any statement with the word “outperforms” should be in the main text, rather than the caption. I would shorten the caption for Table 2 to “Table 2: Mean (std) test accuracies for equituning using a pretrained Resnet with Rot90-CIFAR10 and Galaxy10.” and move the other statements to the main text. Please apply the same to all captions.\n\nPlease use `\\citep{}` for parenthetical citations and `\\citet{}` for textual citations. Many of your citations, as currently written, should be parenthetical rather than textual, and some vice versa. For example, \" equivariant models such as GCNNs Cohen & Welling (2016a) and E(2)-CNNs Weiler & Cesa (2019).\" -> \" equivariant models such as GCNNs (Cohen & Welling, 2016a) and E(2)-CNNs (Weiler & Cesa, 2019).\" and “such as (Fuchs et al., 2020; Thomas et al., 2018)” -> “such as Fuchs et al. (2020) and Thomas et al. (2018)”\n\nI like the brief background given by the Section 3 “Group and Representation Theory” to help a wider audience understand the rest of the work. To follow this level of definition, the notation $O(n)$ for orthogonal groups should be explicitly defined. Other group notation definitions are also quite scattered: there’s a few in Section 4.1 “Input Feature Representations”, so I would recommend adding a paragraph to the “Group and Representation Theory” section to consolidate them to one place.\n\nCan Tables 4 and 11 be merged?\n\nGenerally, there are quite a few typos and errors. Please proofread your work. Here are some fixes.\n\nAbstract: “MLPs (EMLPs). But this method does not scale well and” -> “MLPs (EMLPs), but this method does not scale well, and” Generally, do not begin sentences with “but”.\n\nSec. 1 Introduction, paragraph 1: “can be challenging both because they require” -> “can be challenging because they both require”\n\nSec. 1 Introduction, paragraph 3: “.as is noted by the authors eml.”: fix this citation.\n\nSec. 1 Introduction, paragraph 4: “computationally expensive Further” -> “computationally expensive. Further”\n\nSec. 1 Introduction, enumerated list: “which is are a class” -> “which are a class”\n\nSec. 2 Related Works, paragraph 2 Steerable networks: “are provided in Sec. A.” -> “are provided in Appendix A.” Using the word “Appendix” helps clarify that this section is found at the end of the paper.\n\nSec. 4.3 Neural Processing, final paragraph: “groups (see C make use” -> “groups (see Appendix C) make use”\n\nSec. 4.4 Properties, paragraph 2: “note that restricting G-RespNet to” -> “note that restricting G-RepsNet to”\n\nFigure 2 caption: the caption text overlaps with the subfigure labels.\n\nSec. 5.1 Comparison with EMLPs, paragraph 2 Model design: “with 5 layers where use different tensor” -> “with 5 layers, using different tensor”\n\nAppendix E.4. Table 9: “GRpesFNOs”->”GRepsFNOs”\n\nEntire paper: be consistent about whether there is a dash in “G-RepsNet” or not. A quick search shows 123 occurrences with the dash and 22 without (not including in figure image files).", "broader_impact_concerns": "No broader impact statement included: I don't think one is necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ML3ce8wiP4", "reviewer": "Reviewer_Pq7k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The simplicity of the proposed architecture allows it to be adapted to a wide range of equivariant architectures, as demonstrated in the experiments.\n- The provided universality result indicates that the simplicity of the proposed architecture doesn't hurt its generality in the case of orthogonal groups. While this result is not generalized to arbitrary matrix groups, it is still quite significant, especially since the orthogonal groups cover the symmetries of a large portion of the commonly used equivariant tasks.\n- The experimental results show that the proposed G-RespNets can achieve comparable results to the general EMLP in terms of performance while requiring significantly lower computational resources. Additionally, it compares favorably with other architectures specialized to equivariance in specific symmetry groups.\n\nWeaknesses:\n- One of the main weaknesses of the proposed approach is that the tensor product operation results in at least a quadratic increase in the dimension of the resulting representations. This can hurt the authors' scalability claims.\n- Additionally, the choice not to perform tensor decomposition means that the higher-order tensors can contain correlated or redundant information. For example, a type-two tensor can contain a scalar part (trace), an antisymmetric part (vector), and a symmetric traceless part. While previous approaches that operate with irreducible representations can handle each of these parts separately, the proposed G-RespNet will need to process the information as a single entity with 9 correlated components. Could this redundancy necessitate larger latent representations to encode the same information compared to an approach that decomposes tensors into irreducibles?\n- The design of the tensor mixing layer assumes the existence of both a scalar feature $Y_{T_0}$ and an invariant function inv(.). While in most groups with unitary representations this assumption is trivial and can be satisfied by using an appropriate norm, it can get more complex in the case of arbitrary matrix groups that are non-compact and do not have unitary representations. Is there a general solution for efficiently computing this invariant function inv(.) in the general case of arbitrary matrix groups?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a novel framework for constructing efficient networks that are equivariant to arbitrary group transformations. Specifically, the authors propose a lightweight neural network architecture that utilizes tensor representations, which are processed and combined only using tensor addition and tensor multiplication operations. These simple operations preserve the equivariant properties of the network while requiring fewer computational resources compared to previous works, such as EMLP which constrains the linear operations between different equivariant representations by using a basis of equivariant weight matrices. After introducing the proposed architecture of Group Representation Networks (G-RepsNet), the authors show that it is a universal approximator in the case of orthogonal matrix groups. In the experimental section, they showcase how the proposed network desing can achieve performance comparable to the computationally expensive EMLP on a set of synthetic datasets. Additionally, they provide comparisons with other specialized equivariant architectures, showing improved performance in tasks such as image classification, Nbody simulation, and finding solutions for the 2D Navier-Stokes equation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some possible changes that could further support the claims of this work are the following:\n- A discussion about the assumption of the existence of an easy and efficient way to compute invariant functions in the case of arbitrary matrix groups. Is this computation always as easy as computing a vector norm?\n- A discussion regarding the tradeoffs of using redundant information in higher-order tensor representations versus decomposing the input tensors into irreducibles. While the simplicity of the tensor representations provides benefits in the speed of the method, can it hurt the expressivity of the model given a fixed memory budget?\n-  More experiments showcasing the scaling of the method when it uses tensors $T_i$ with $i\\geq 3$, which could further support the scalability claims of the authors.\n\nWhile the proposed changes are not necessary for acceptance, I believe they will help improve the presentation of the significance of this work.", "broader_impact_concerns": "There are no concerns on the ethical implications of the work that would require adding a Broader Impact Statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VWC4AM9JyM", "reviewer": "Reviewer_wEwX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the work is that building more efficient equivariant networks is important, as a common drawback of equivariant architectures is their high computational demand. The proposed networks seem simple, and this is a contribution as many equivariant networks are quite complicated.\n\n### Weaknesses related to presentation\nSince TMLR does not require novelty, there is no reason not to be charitable and relate as much as possible to prior work in the method description. This would improve the reader's experience significantly. Further, several tables referred to in the main text appear in the appendix, which is unnecessary as there is no page limit for TMLR. Please refer to the requested changes below for detailed comments.\n\n### Experimental weaknesses\nI will focus on the experiments on image data in Section 5.3, as these fall within my area of expertise.\n1. The reported test accuracies for CIFAR-10 in Table 4 are very low. This immediately catches the eye of the reader and makes it difficult to assess the merits of the proposed method. For reference, in 2010, 79% was considered good (https://www.cs.toronto.edu/~kriz/conv-cifar10-aug2010.pdf) and in 2014, results above 80% were standard (https://code.google.com/archive/p/cuda-convnet/). Cohen & Welling got over 90% accuracy in 2016 with their G-CNNs. Thus, results below 80% (including for G-CNNs!) stand out to the reader. I do not see a reason not to have better architectures in these experiments. CIFAR-10 with rotations should not be more difficult than ordinary CIFAR-10 for rotation-invariant models.\n2. I have trouble understanding exactly how $T_2$-features are mapped to $T_0$-features in the networks. Earlier in the paper, it is stated  that $T_0$-features are obtained through \"an appropriate invariant operator, e.g. Euclidean norm for Euclidean groups, or averaging over the group channel dimension for regular groups.\" But I did not find information on what is used for the experiments in Section 5.3.\n3. I did not find information on runtime or number of parameters for the different models in the experiments in Section 5.3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes GRepsNet, a way to construct equivariant networks. \n\nAs in prior work, features are collected in terms of which representation acts on them. The main idea is to, given a feature $X$ with associated representation/\"feature type\" $\\rho:G\\to\\mathrm{GL}(\\mathbb{R}^K)$, write the feature $X$ as a $K\\times C$-tensor, where the representation dimension $K$ and \"channel dimension\" $C$ are kept apart. Then linear layers can be applied from the right, e.g. mapping $X\\mapsto X W$, where $W$ is a learnable $C\\times D$ weight matrix so that the output has new channel dimension $D$ and the same feature type as the input. The contribution of the present paper is to use this idea for quite general representations (when $\\rho$ is the 3-dim irrep of $\\mathrm{SO}(3)$, the idea is the well-known \"Vector Neurons\"-approach).\n\nThe paper proposes stacking a couple of linear layers as described above, interspersed with simple layers that mix feature types to build equivariant networks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical changes for recommending acceptance\n### On alignment with prior work and delineating the novelty of the work\n1. For irreps (of real type), the proposed linear layers are general (due to Schur's lemma, see e.g. B.5.10 in [1]), but for other representations, the proposed linear layers do not represent maximally general linear layers. The present approach seems related to separable group convolutions as proposed in [2] and [3]. However, it is not intuitive to me why \"separating out\" higher tensor powers of the regular representation, and further not having any linear layer over the group axis, is a good idea.\n2. Tensor products of regular representations decompose into direct sums of regular representations. For instance, $T_2$ features for the group of $90$ degree rotations decompose into four $T_1$'s. It is, therefore, unclear what the benefit of using $T_2$ features in the image classification experiment is. Is it the nonlinearity introduced by tensoring $T_1$ features? Is it the fact that we get four times as many features as input to the final classification layer? Or is it something else? Specifically, statements like \"This shows the importance of higher-order tensors in image classification.\" at the end of Section 5.3, are therefore not evident. \n3. The tensor products of representations considered as feature types in this work decompose into irreps. So, all considered network features can be described in terms of irreps. This makes all CNNs considered here special cases of steerable CNNs, and it would help the reader if they were described as such.\n4. As mentioned earlier, it is not clear to me how the mapping to invariant features is done in the CNN experiments.\n5. I believe that the layers merging $T_0$ and higher-order tensors could be seen as special cases of the bilinear layers by Finzi et al.\n\n### Experimental details\n1. Move the figures and tables referenced from the main text to the main text. E.g. Fig. 6a and Table 8.\n2. Include runtime and the number of parameters for all models.\n3. If $T_2$ outperforms $T_1$, why not use $T_3$, $T_4$ etc?\n\n### Technical background\nThe comments refer to sentences in Section 3.\n1. \"Then, for a group G, the linear group representation of G is defined as\" -> *a* linear group representation\n2. \"the size of regular representation is proportional to the size of G.\" -> is *equal* to the size of G.\n3. \"Hence, the size of any representation, m, can be written as m = |G| × d for some integer d.\" Not the size of any representation, but specifically the tensor product representations considered here.\n\n# Non-critical changes, but highly recommended\n1. Change to networks that achieve > 90% test accuracy in the CIFAR-10 experiments.\n\n# References\n[1] Weiler, Maurice, Patrick Forré, Erik Verlinde, and Max Welling. \"Equivariant and Coordinate Independent Convolutional Networks.\" A Gauge Field Theory of Neural Networks (2023).\n\n[2] Lengyel, Attila, and Jan van Gemert. \"Exploiting learned symmetries in group equivariant convolutions.\" In 2021 IEEE International Conference on Image Processing (ICIP), pp. 759-763. IEEE, 2021.\n\n[3] Knigge, David M., David W. Romero, and Erik J. Bekkers. \"Exploiting redundancy: Separable group convolutional networks on lie groups.\" In International Conference on Machine Learning, pp. 11359-11386. PMLR, 2022.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a consensus in reviewers that this paper introduces an interesting method and deserves to be accepted.
Decision: Accept (poster)