Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?
Authors: Jiacheng Cen
Anyi Li
Ning Lin
Yuxiang Ren
Zihe Wang
Wenbing Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Geometric Graph Neural Networks
High Degree Steerable Vectors
Primary area: graph_neural_networks

Abstract: Equivariant Graph Neural Networks (GNNs) that incorporate E(3) symmetry have achieved significant success in various scientific applications. As one of the most successful models, EGNN leverages a simple scalarization technique to perform equivariant message passing over only Cartesian vectors (i.e., 1st-degree steerable vectors), enjoying greater efficiency and efficacy compared to equivariant GNNs using higher-degree steerable vectors. This success suggests that higher-degree representations might be unnecessary. In this paper, we disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric structures, including $k$-fold rotations and regular polyhedra. We theoretically demonstrate that equivariant GNNs will always degenerate to a zero function if the degree of the output representations is fixed to 1 or other specific values. Based on this theoretical insight, we propose HEGNN, a high-degree version of EGNN to increase the expressivity by incorporating high-degree steerable vectors while maintaining EGNN's efficiency through the scalarization trick. Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric structures, but also shows substantial improvements on more complicated datasets such as $N$-body and MD17. Our theoretical findings and empirical results potentially open up new possibilities for the research of equivariant GNNs.

Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?
Jiacheng Cen 12 Anyi Li Ning Lin Yuxiang Ren Zihe Wang 2, Wenhing Huang
1 Gaoling School of Artificial Intelligence, Renmin University of China
2 Beijing Key Laboratory of Big Data Management and Analysis Methods
2012 Laboratories, Huawei Technologies, Shanghai {jiacc. cn. _i_ li_anyi, minglino00oouuttll @outlook. com; renyuxiang (etryyuxiinnnghhuaaaee wwwwwwuuww..c.cco com; wang. zihe@ruc. edu. en; hwenbing@126. com
Abstract
Equivariant Graph Neural Networks (GNNs) that incorporate the E(3) symmetry have achieved significant success in various scientific applications. As one of the most successful models, EGNN [1] leverages a simple scalarization technique perform equivariant message passing over only Cartesian vectors (i.e., 1st- degree steerable vectors), enjoying greater efficiency and efficacy compared to equivariant GNNs using higher-degree steerable vectors. This success suggests that higher-degree representations might be unnecessary. In this paper, we disprove this hypothesis by exploring the expressivity of equivariant GNNs symmetric structures, including k-fold rotations and regular polyhedra. We theoretically demonstrate that equivariant GNNs will always degenerate a zero function if the degree of the output representations is fixed to 1 or other specific values. Based on this theoretical insight, we propose HEGNN, a high-degree version of EGNN to increase the expressivity by incorporating high-degree steerable vectors while still maintaining EGNN's s advantage through the scalarization trick. Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on a toy dataset consisting of symmetric structures, but also shows substantial improvements other complicated datasets without obvious symmetry, including N-body and MD17. Our study potentially showcase an effective way modeling high-degree representations in equivariant GNNs.
Introduction
Molecules, proteins, crystals, and many other scientific data can be effectively modeled and rep- resented through geometric graphs [2-8]. This type data structure encapsulates not only node characteristics and edge information but also 3D vector (such as position, velocity, etc.) for each node. To process geometric graphs, equivariant Graph Neural Networks (GNNs) have been developed, which undergo equivariant message passing over nodes, conforming the E(3) SE(3) symmetry of physical laws. These models have achieved remarkable successes in a lot of scientific tasks, such physical dynamics simulation [9-11] 11], molecular generation [12-15] and protein design [16-18].
Pioneer equivariant GNNs [19-22] derive high-degree steerable representations beyond scalars and 3D coordinates with the help of spherical harmonics and conduct equivariant message passing between representations of different degrees through the Clebsch-Gordee (CG) tensor product. While these high-degree models are able to approximate any function of fully connected geometric graphs in theory [23], they usually suffer from expensive computational costs in practice. In contrast,
Wenbing Huang is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

k-fold (odd)
k-fold (even)
Tetrahedron Cube (Hexahedron) Octahedron
Dodecahedron
Icosahedron
Common symmetric graphs. Equivariant GNNs on symmetric graphs will degenerate to zero function if the degree of their representations is fixed as
Figure I:
EGNN [1] leverages simple scalarization technique allow equivariant message passing over only 3D vectors (i.e. the Ist-degree steerable features). Specifically, the scalarization technique first encodes 3D vectors into scalars invariant messages, which are passed as geometric messages after the multiplication with the 3D vectors to recover the orientation information. Despite its simplicity, EGNN achieves remarkably better efficacy and efficiency against conventional high-degree models for broad range of applications [24; 25]. Such successes suggest that higher-degree representations might be unnecessary.
In this paper, we challenge and disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric graphs. Fig. illustrates the examples of k-fold rotations and regular polyhedra, which are invariant to rotations up to certain rotating angles. Taking the cube for example, conducting 90º rotation around the axes crossing the center of the orientation unchanged. Interestingly. by making use of group theory, we theoretically prove that any equivariant GNN (after translating the coordinate center to the origin and conducting graph- level readout) on these symmetric graphs will degenerate to a zero function if the degree of their representations is fixed be The direct deduction of this theorem is that EGNN can only output zero 3D vector no matter how we rotate the input graph, indicating that EGNN totally loses the recognition ability of orientation. Additionally, this statement points out the limitation of the methods that rely constructing global features for symmetric graphs (e.g. frames in frame averaging [26; 27], virtual nodes in FastEGNN [28]), equivariant pooling in EGHN [29], and meshes in Neural PdM [30]), since it is impossible to output another non-collinear 3D vector except the center coordinate.
opposite faces keeps its shape and
Based on the above theoretical insights, we propose novel equivariant GNN model termed HEGNN*, which enhances EGNN by incorporating high-degree steerable vectors while inheriting the desired benefit from EGNN through the scalarization trick. In summary, our contributions are as follows:
We theoretically investigate the expressivity reduction issue of equivariant GNNs on sym- metric graphs.
We propose HEGNN. to further incorporate high-degree steerable representations into EGNN. Moreover, since the equivariant message passing process between different-degree representations is conducted via inner products, it shares the same benefit as EGNN, com- pared traditional high-degree models.
Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric graphs, but also shows substantial im- provements on more complicated datasets without explicit symmetry, such as N-body and MD17.
2 Related Works
Equivariant GNNs. Equivariant GNNs can be divided into two classes: scalarization- based models and high-degree steerable models [31]. Scalarization-batt
models adopt norms inner products to convert equivariant 3D vectors into invariant scalars, which are considered as coefficients to linearly combine 3D vectors for node update. EGNN [1] is the first work falling into this category. Concurrently, PAINN [32] further enhances the expressive ability of the model by introducing multi-channel equivariant features. On the contrary, high-degree steerable models (e.g. TFN [19],
2See Appendix A.3 further discussion.
*Code available at https: //gi gigthub..a com/ cow/GLAD- GLLD--LAADRR BALAAA--UC/CAE RUC/HEGNN.

---

SEGNN [33] and SE(3)- (ranssoranff))))) use spherical harmonics to ensure the equivariance of message passing, and realize interaction between steerable features of different degrees through CG tensor products. Our HEGNN also uses high-degree steerable features, but it leverages the scalarization trick for the interaction between steerable features of different degrees, thus leading to more expressivity than EGNN and less computational cost than other high-degree models.
Expressivity of Equivariant GNNs. The theoretical expressivity of equivariant GNNs is initially explored by [23], which proves the university of the high-degree steerable model, i.e., TEN [19], over cully-connecee with just spherical representations other than the full (O(3) representations that are required in the proof of [23]. More recently, the GWL framework [35] extends the Weisfeiler-Lehmal (WL) test into a geometric version [36] to study the expressive power of geometric GNNs operating on sparse graphs from the perspective of discriminating geometric graphs. Different from all the above works, our paper investigates the expressivity of equivariant GNNs on symmetric graphs, and demonstrates the necessity of involving high-degree representations. Although the GWL test paper [35] has experimentally compared different models k-fold structures that allowed to rotate only in the 2D space, the conclusions of this paper are proved both theoretically and experimentally, Moreover, our discussions cover a full range of examples including k-folds (rotation in 3D space) and regular polyhedra.
geometric graphs. GemNet [34] further demonstrates that the universality holds
3 Theoretical Analyses
In this section, we first present the necessary preliminaries related geometric graphs and group representation. Then, we define and illustrate typical examples of symmetric graphs. Finally, we will discuss when equivariant GNNs will degenerate a zero function on symmetric graphs.
3.1 Preliminaries
Geometric graph. A geometric graph of N nodes is defined as G = (H. (H. where H
{h; E p and X = { E R$ J are node features and 3D coordinates, respectively; A E NNN represents the adjacency matrix and can be assigned with edge features eij if necessary. Throughout our theoretical analyses in this section, we assume the node features and edge features to be identical for all.
Transformation of geometric graph. We are interested in the transformations of geometric graph G with respect to a group G, which defined as G. for E 8 and denoting the group action. For instance, g G can be explained as translation, rotation, reflection of the coordinates
X. These transformations form a 3D Euclidean group denoted E(3), and its subgroup without translation is called the orthogonality group O(3). With the aid of group representation p(g), the transformation of coordinate á is represented p(g)æz. For example, orthogonal matrices are the trivial representations of O(3), that is, the orthogonal transformation of vector E is represented by Oz with O 3x3 being an orthogonal matrix. Besides, there are other representations of ((3). such as the irreducible representations which will be detailed below.
Equivariance. Let X' and y be the input and output vector spaces, respectively. A function f: + yYi is called equivariant with respect to group @ if
Vg E 6, (px(())) = (x))
where Px and By are the group representations in the input and output spaces, respectively. Since we can eliminate the translation effect by simply translating the center of all coordinates to the origin, we only discuss equivariance with respect to 0(3) in this section. In other words, we default that the center of X is at the origin.
Irreducible representations and steerable features. 0(3) consists of rotation and inversion, implying 0(3) SO(3) x C.. where SO(3) is the rotation group and C, {c, denotes the inverse group. We first discuss the irreducible representations of SO(3). For each rotation E SO(3), its irreducible representations are Wigner-D matrices ( E x(2++1)x(2++1 of different degree I N [21; 37]. When = 1, it becomes the common rotation matrix R. acting on the 3D coordinate space. Under the irreducible representations, the equivariant constraint in Eq. (1) turns
3

---

into = y( = Yn (z) offer unique and complete set of function bases satisfying the equivariant constraint. We further define a modulated spherical harmonics as f = ( (at ( by adding continuous radial function & IR + R of vector norm || I1 for re-scaling. Such function f and its output (z)) are called type-l steerable function and steerable feature, respectively.
if the output degree is I. According to [38], spherical harmonics
We now deduce the irreducible representations from SO(3) to O(3). Note that spherical harmonics satisfy (-z)) = (- (-1'r( in other words, they are inverse- -quivvr-quuvvariant when / is odd, but inverse-invariant when I is even. We thus specify the group representation of ((3)
p (rm) = の (m) (t),
where (m) = 1 for m = (the identity) and (0) (m)) = (-1)' if m = i (the inverse). Readers can refer the discussion in e3nn [39] with another representation method by using the concept of parity and construct this through methods such as Clebsch-Gordan (CG) tensor product [40]. For concision, the type-l steerable feature is denoted as (()) with a tilde notation.
3.2 Symmetric Graph
In $ 1, we present that k-fold rotations and regular polyhedra exhibit certain symmetries. In this subsection, we formally describe them via the notion of the symmetric graph.
Definition 3.1 (Symmetric Graph). A geometric graph G is called a symmetric graph, if there exists finite and nontrivial subgroup S S ((3). S * {c). satisfying that Vh E 55. h G == G. All subgroups making& G symmetric yields set (g),, and all geometric graphs that symmetric w.n.t. S) constitute a set denoted as (J)).
Here h. 9 g= G is defined in the graph level. Particularly for the coordinates X E 3xN,, it implies that vO E 5. 3 & SN OX = X P and PA = AP, where Sy is the permutation group of order N. Essentially, rotating the coordinates of symmetric graph leads copy of this graph up to a different permutation of the nodes.
Without considering inversion, the finite subgroups of SO(3) only cyclic group Cn, dihedral group D,,, tetrahedral group T, octahedral group O, and Icosahedral group [41]. We provide several examples of symmetrie graphs as follows.
Example 3.2 (k--flds). On the one hand, for a geometric graph G corresponding to a 2k-fold with nodes [cos(1 // a/kk), sin(i o)) the inverse group C, and the dihedral group Dax (rotation around z-axis with angle π a/k: and reflection around the axis connecting the midpoints of opposite sides or the axis connecting opposite vertices), are symmetric groups on G. namely, Cr D2x E HG(g). On the other hand, for geometric graph G corresponding to (2k: + 1 -fold with nodes {(cos(i- 2 /2k + 1), sin(i- 2T 22k + 1)), 0) 1 HG)) includes the dihedral group D2k+1 1 but without the inverse group C..
Example 3.3 (Regular Polygons). The symmetric groups of regular polygons in the plane and regular prisms in space include the dihedral group D,,- Regular tetrahedra are symmetric with respect to three rotation axes of the second order and four axes of the third order, corresponding elements. Regular hexahedra (cubes) and the regular octahedra (which are dual to each other and share the same symmetric groups) are symmetric about six axes of the second order, four axes of the third order, and three axes of the fourth order, corresponding to 24 group elements. Regular dodecahedra and regular icosahedra (which are also dual each other) are symmetric about six axes of the fifth order, ten axes of the third order, and fifteen axes of the second order, corresponding to 60 group elements. Additionally, except tetrahedra, all other four regular polygons are central symmetric, indicating that C2 is their common symmetric group.
12 group
3.3 Equivariant GNNs on symmetric graphs
We now demonstrate that equivariant GNNs symmetric graphs will encounter the issue of ex- pressivity degeneration. Here, we assume that the graph functions permutation of the nodes. This fits the case when we add readout layer to all nodes globally or just focus on the message passing process for each node individually.
we explore are invariant to the
We first derive crucial theorem that greatly facilitates our analyses.

---

Theorem 3,4. Suppose that f is an ((3))equivrrian function on geometric graphs, regarding the group representation p defined in Eq. (2). Then. for any symmetric graph G induced by the group 5 S 0()), namely, Vg E G()), we always have
(9) = (5))
(3)
Here we have defined group average (5)) = [.T I
E
Eq. (3) is interesting and it shows that the function f is symmetric with respect to the group average
(5). More importantly, it indicates an linear equation (12+1 - (5)) (g) = 0, where
1241 E (1+++)))()) is the identity matrix. We can immediately attain the following conclusion.
Theorem 3.5. If and only if the matrix I (5) is non-singular, the 0(3))eqqnivariant f(i) is always a zero function on G, namely.
(G) = O, VG G()).
A more general version of Theorem 3.5 is that the output space of f()) corresponds to the null
space of the matrix 121+1 p (5)), indicating that dim (f)) = (21 + 1 rank (I2i+1 - ( (5))
Therefore, even the function f()) will not exactly reduce to a zero function when I241 - (5) is
singular, its output space still limited to subspace and suffers from diminished expressivity owing to the symmetry of the input geometric graph.
In practice, it is difficult to determine if the matrix I24+1 - p00 (5) singular. This determination becomes easier we can show that the group average (5)) is equal to the zero matrix. Fortunately, we have the following property.
Theorem 3.6. For finite group S) with its representation p()), (()) is a zero matrix (i.e., (5)) 0) if and only if tr(p (5)) = =.. In this case, f() 0, VG G(5).
According to Theorem 3.6, we calculate the trace of the group average for each symmetric graph of interest and check if the trace is equal zero. We summarize the conclusions for k-fold structures and regular polyhedra in Table 1. We find that when 1, f() 0 for all cases. In addition, the function degenerates when is odd, if the symmetric graph is induced by the inverse group C,, We defer more details of the calculations in the Appendix. Compared to the conclusions drawn by the GWL paper [35] which only experimentally discusses the folold structures under 2D rotations, here apply rigorous theoretical derivations to analyze more cases besides k-folds, regarding more symmetric subgroups of (3)).
Table 1: Expressivity degeneration of equivariant GNNs on symmetric graphs.
Symmetric Graph g
Symmetry Group 5 E H[(g)
I leading to (g) = 0
2k-fold
C₁ D2k D2s+1 T
I is odd
(2k: + 1)-fold
1< 2k + 1 and is odd
Tetrahedron
lE (1,2,
Cube//ctaheedron
C, O
[= 2 or is odd
Dodeecaledrom//cooaheccoon
C, I
L {2, 4, 8, 14) or / is odd
The Proposed HEGNN
The analyses in the last section imply the necessity of involving the representations with more and higher degrees in equivariant GNNs. Therefore, we propose HEGNN by further conducting the update of high-degree steerable features upon EGNN [1]. As illustrated in Fig. 2, HEGNN is composed of the three key components: initialization of high-degree steerable features, calculation of cross-degree invariant messages, and aggregation of neighbor messages, the latter two of which are conducted over multiple layers. We depict each component separately in what follows.
Initialization of high-degree steerable features. Given geometric graph & (H, (H; where each node contains only type-0 feature h; and type-1 feature H. we first obtain the initialization
5

---

R2
Coocatt
Oth Degree
1st Degree
Mixed Degrees
Nes
Operator Network
M2lt
(a) EGNN
F2
Uph Inbeeding
RC
Teasor Product
4B5
(b) TFN
(c) HEGNN
Figure 2: The different architectures of our HEGNN, EGNN [1] and TEN [19]. HEGNN exploits the scalarization trick inspired by EGNN to enable steerable features to interact between different degrees, avoiding the high computational cost of using CG tensor products TFN.
of high-degree steerable features {ύī N by using spherical harmonics on normalized relative coordinates. In detail, aggregate spherical harmonics from all neighbors
Σ
ao (mii,inn)) (1)
(5)
inin
[N(O)
JEN(()
where mij,init inmt (hi, hi, eij, eij,de)) is invariant sealar with Pmaiintt being arbitrary MultiLayer Perceptron (MLP), and (ii) denotes the neighbors of 1
Calculation of cross-degree invariant messages. EGNN [1] employs scalarization trick by transforming the relative coordinate 3; Ty (the usage of relative coordinates for translation invariance) into an invariant scalar via the vector norm, which will be used to compute invariant message for both node features and coordinates. We generalize this scalarization trick the case of high-degree steerable features. To be specific, we carry out the inner product between 5/" and a!!))
for each degree / individually, resulting in an invariant scalar 2ig Then, we get the invariant message between node i and j. namely m, after undergoing an MLP of all invariant quantities. The above processes are summarized as follows:
di = - - 1b
0 0))). mij = Pm hh, eaj,
(1)
(6)
where + refers to concatenation. It should be noted that the form of SO3KRATES introduced in the concurrent work [42] is equivalent to the expression for z Eq, (6). Furthermore, our scalarization trick simplifies the formulation by bypassing the Clebsch-Gordan coefficients, making it more straightforward and casier to understand.
Aggregation of neighbor messages. With the invariant message mij at hand, we then update h, F., ((()) via message aggregation over all neighbors. We define Ah; Aæ, Ao!!) as the residues, which are calculated by:
Ah; Ph
his [W(1)
Σ mij
AE,
Σ pa(mi)) -E,), JEN())
(7)
[(iI
JEN(()
Av
Σ JEN())
P8 (m:y)
)-
(w.0) (-(0- -60-)
(8)
'Eq. (5) is unable derive aseudo-vectors such as torque angular momentum, which are type-1 steerable features but invariant reflection. To address this issue, further conduct CG tensor product between anst and æt; æ yield the steerable feature of desired symmetry.
(1)

---

(1) both output a ID scalar. Note that where 9hi ie, Pe are different MLPs, and Pa, Po
the application of Eq. (8) for all degrees can be compactly rewritten as L A pa(mm))
@ (o - ) in the form of CG tensor product with the weights Pp (m(j)) Oi Ps- This form can be easily implemented using existing libraries such as e3nn. o3. MaallyyoometeeTennnoPPeee
1 [V(0) ΣjEN()) 18 1g
[39]. The resulting residues are used for the update:
h; h; + Ah,; T.. 5. + AT,,
In addition, can augment the update of 2 with 1st-degree feature of leading to उ. ã; A + (1) (h;) )o which yet is not explored in our experiments for the sake of simplicity. The final output of h, and H. can be used for the node-level invariant prediction and equivariant prediction, nodes.
respectively. We can also obtain a graph-level prediction by further adding a readout layer of all
We now analyze the expressivity of HEGNN. Apparently, by including high-degree features, HEGNN is able to avoid the loss of expressive ability even on symmetric graphs. Moreover, when tackling general geometric graphs, HEGNN is capable of characterizing the complete angle information of the input graph, if its maximal degree L is sufficiently large. For concision and without losing the generality, we assume the steerable features o(2) are initialized with only spherical harmonics without the weights Peiinit in Eq. (5). Let Ti.- (zt -z2// T.ll. the inner product (0) be expanded as follows
(!)
can
Σ (EN(()
(it))
Σ LEN())
y (Zze)
4T
Σ (EN(() KEN())
p()) ((Tis, Zzx)),
(10)
21 +1
where P()). R + R is Legendre polynomial of degree I, and Eq. (10) is based on the properties
of spherical harmonics that (z), ( (g)) = 4T / + 1) p()) ((0( 3)), ||ج|| = | = 1. We have the following result.
Theorem 4.1. For any geometric graph, there exists bijection between the set of in- [ais ner products {zig Jimi given by Eg. (10) and the set of edge angles Aij = (0is.st = arccos(@ie: (e))seEN()))ENN))
Theorem 4. states that the inner products of full degrees can recover the information of all angles between each pair of edges, affirming the enhanced expressivity of our HEGNN. The proof is derived mainly based on the fact that Legendre polynomials are orthogonal polynomial bases which can injectivly represent the set Aij thanks to Newton's 's identities. The details are deferred to the appendix. Although the upper-bound of the degree in Theorem 4.1 grows rapidly with the graph size, will be shown in our experiments that HEGNN with only L S 6 is sufficient to outperform traditional models like EGNN [1] and TFN [19] in practice.
5
Experiment
5.1 Expressivity on Symmetric Graphs
Design of experiments: To experimentally verify the conclusion we proved above, we design more comprehensive experiment based on code in [35]. This experiment uses four k-fold structures (k E (2, 5, 10)) and five convex regular polyhedra shown in Fig. test objects, and the center of each is at the origin. In detail, an arbitrary rotation in 3D is acted on such symmetric structures called Go which ensures the geometric graph after rotation called G1 does not coincide with the original one. The goal of our experiments is to check whether different equivariant neural networks can distinguish Go and GT
The models we select include two models that only use Cartesian coordinates: EGNN and GVP- GNN; and two models that use high-degree steerable features: TFN and MACE. However, TFN and MACE (denoted as TFN/ACEE;5)) always use all degrees I E (O,..., L}, so it is not clear which
7

---

degree(s) of steerable features distinguish the two geometric graphs. In our HEGNN, all steerable features corresponding to unwanted degrees could be masked during initialization Eq. (5), and we let HEGNN== I. be HEGNN with only Ith-degree steerable features. Additionally, to align with TEN/MACE, we also test the performance of HEGNN with all E {0, I,.., [ donated as HEGNNIs L- Following the settings in [35], the output of each graph is the concatenation of invariant scalars, coordinates, and high-degree steerable features pooling among all nodes. We then map this spliced vector to two-dimensional vector and input it into a simple classifier to determine whether the equivariant graph neural network can distinguish Go from G1.
The results on k-fold are deferred to Appendix for saving space, and the results on regular polyhedra are shown in Table From Table we can know steerable features in which degree could not distinguish specific symmetry structure, and both results on k-fold and regular polyhedra are also in perfect agreement with our conclusions. Models (EGNN and GVP-GNN) only with Cartesian vectors cannot distinguish any symmetric graph at all. Taking HEGNN as an example, since (()(()) = 0, vS) E {T, O, J. no matter which kind of regular polyhedron, f(8) could only output thus failing distinguish the structures.
Results:
Table 2: Regular polyhedra.
Rotational symmetry Octahedron
GNN Layer
Tetrahedron
Cube
Dodecahedron
Icosahedron
Cart.
E-GNN
50.0 +0.
50.0 =00
50.0 =00
50.0 +0.0
50.0 =00
GVP-GNN:
50.0
50.0 100
50.0 100
50.0 1..
50.0 1.0
HEGNN==1
50.0 +0.0
50.0 =00
50.0 +00
50.0 0.D
50.0 +0.000...
HEGNN.
50.0 0.0
50.0 200
50.0
4.0
50.0 10.00.0.0
Single Type Spherical
HEGNN
100.0 0.0
50.0 =00
50.0 =00
50.0 10.0+au.
50.0 0.0
HEGNN 1
100.0
90.0 =300
90.0 =30.0
50,0
50,0
HEGNN 5
50.0 0.0
50.0
50.0 000
50.0
50.0
HEGNN 8
100.0 0.0
100.0 =..0
100.0 0.0
100.0 =
100.0 0.0
HEGNN T
100.0 0.0
50.0 =000
50.0 =00
50.0
50,0 0.0
HEGNN
100.0 0.0
90.0 130.0
90.0 130.0
50.0 1.0
50.0 1.0
HEGNN 9-9
100.0 0.0
50.0 =00
50.0 +000
50.0
50.0
HEGNN =10
100.0
100.0 n.0
95.0
100.0 2.0
100.0
HEGNN 11
100.0
50.0 +00
50.0 =00
50.0
50.0
HEGNW/TEN/MACE 122
50.0 0.0
50.0 =.00
50.0
50.0 30.0
50.0 0.0
Sph.
HEGNN/TEN/MAEE. Beg3
100.0
50.0
50.0 200
50.0 0.D
50.0 =..00.0..
HEGGNNTTNMACEE
100.0
100.0 n.0
100.0
50.0
50.0
HEGNNNTTNMACE
100.0 0.0
100.0 0.0
100.0 0.0
100.0 0.0
100.0 0.0
5.2 Physical Dynamies Simulation
Datasets: We benchmark our HEGNN in two scenarios, including: N-body system [43] is a dataset generated from simulations. In our simulations, each system contains N charged particles with random charge e (0, 1}. whose movements are driven by Coulomb forces. To verify the efficiency and effectiveness of our HEGNN on datasets of different sizes, we select N from (5, 20, 50, 100). We use 5000 samples for training, 2000 for validation, and 2000 for testing. The task is estimate the positions of the N particles after 1,000 timesteps. MD17 [44] dataset contains trajectory data for eight molecules generated through molecular dynamics simulations. The goal of this experiment is to predict the future positions of the atoms based on their current state. We follow the dataset partitioning scheme from [45], splitting the dataset into 500/2000/2000 frame pairs for training, validation and testing, respectively. All experiments are run on a single NVIDIA A100-80G GPU.
Baselines: To demonstrate the advantages of our HEGNN over both models with scalarization techniques and models with high-degree steerable vectors at the same time, our baseline needs to consider the selection issues of both models simultaneously. Therefore, we select some representative models as baselines, including the invariant RF [46], the cquivariant EGNN [1], TFN [19] and SE(3)- Tr. [20]. In addition, select classical models such as Linear dynamics [1], the non-equivariant
'It should be noted here that because only 0 ~ 11-th sttrre spherical harmonics can be used in e3nn [39]. we only measure the models with to 1 11 dehree here, and in Appendix B.3, We have given a new verification method.
8

---

Message Passing Neural Network (MPNN) [47], the invariant SchNet [48], and the equivariant GVP-GNN [49] for the b-body experiments. For MD17 experiments, we also select GMN [45].
Metrics: 1. Loss function: We use Mean Squared Error (MSE) to measure the accuracy of the prediction results in both experiments. 2. Inference time: Given that the N-body system contains data of varying sizes, we test the inference time of each model on this dataset. The inference time for each model is calculated relative to the benchmark, which is the inference time of EGNN the corresponding scale.
The main results of N --body system simulation presented in Table 3. From these results, we observe the following: Overall performance: Our HEGNN significantly outperforms other models across datasets of all sizes. Although EGNN [1] performs better than high-degree steerable models like TFN or (E(3) - -33--sssforeerrr than EGNN, which show that the method of HEGNN introducing high-degree steerable features is more effective. Stability: Although the performance of the model (HEGNN1cc)) using high-degree steerable features declines slightly when the geometric graph is small, overall, HEGNN performs better than other models. 3. Inference time: Our model's inference time is significantly faster than that of high-degree steerable models like TEN, reflecting the simplicity and efficiency of our HEGNN.
Results N-Body systems:
this task, our HEGNN is still better
Results on MD17: The main results MD17 experiment are shown Table 4, with some data sourced from [45]. From these results, draw the following insights: 1. Overall performance: Our HEGNN outperforms other models on six out of eight molecules. The effect on the remaining Two molecules is only not good GMN [45] and this because GMN introduces additional knowledge such as chemical bonds. Advantage high-degree vectors: Most of the best results obtained on HEGNNI indicating that the of high-degree steerable features enhance model expression capabilities.
Table 3: MSE and time-consuming ratio with EGNN [1] on N-body system.
5-body
20-body
50-body
100-body
MSE
Relative Time
MSE
Relative Time
MSE
Relative Time
MSE
Relative
(x10
(x10-*
(x10
(x10 ?)
Time
Linear
7.72
0.01
10.12
0.02
11.81
0.02
12.69
0.01
MPNN [47]
1.80
0.49
2.50
0.51
2.96
0.50
3.55
0.45
SchNet [48]
11.31
17.72
6.24
22.14
31.63
22.14
27.04
RF [46]
1.51
0.54
3.41
0.65
4.75
0.67
5.72
0.49
GVP-GNN [49]
7.26
2.36
5.76
2.38
7.07
2.42
7.55
2.33
EGNN [1]
0.65
1.00
1.01
1.00
1.00
1.00
1.36
1.00
TFN
1.49
2.69
1.86
3.19
2.20
2.87
3.42
6.58
TEN
1.76
3.91
1.87
4.54
1.94
4.89
OOM
SE(3)TTa32
3.24
4.94
3.19
5.88
2.54
5.97
2.33
5.15
HE
0.52
1.77
0.79
1.84
0.88
1.60
1.13
1.45
HEGNN k2
0.47
1.88
0.78
1.94
0.90
1.71
0.97
1.55
HEGNN 1g
0.48
2.11
0.80
2.23
0.84
1.84
0.94
1.61
HEGNN b56
0.69
2.14
0.86
2.43
0.96
2.18
0.86
1.90
Table Prediction error (x10-2) -2) on MD17 dataset. Results averaged across runs.
Aspirin
Benzene
Ethanol
Malonaldehyde Naphthalene Salicylic
Toluene
Uracil
RF
10.94_001
103.72+1.2 4.64
13.93
0.50
1.23
10.93
0.64:0o1
EGNN
14.41
62.404033
4.64#0m
13.64
0.47
1.02+0.2
11.78
0.64+ao1
EGNNReg
13.82 61.68+0037
6.06+0.0
13.49
0.63+
1.68 +000
11.05 #0.01
0.66++0.1
GMN
10.14=0.03
48.1240.0
4.83 +0.m
13.1a
0.40+aot
0.91 +0.01
10.22
0.59+a.o
TFN
12.37=0.18
58.48+11
4.81 10.04
13.624008
0.494001
1.03
10.89
0.84+..0
SB(3)-TT.g2
11.12-0.06
68. +
4.74
13.89+..oz
0,52+aor
1.13
10.88
0.79+a.z
HEGNN
10.32.0.58
6765
4.63
12.85
0.38
1.90
10.56.2.10
0.56:10.0
HEGNN
10.04=0.45
61.80
4.63
12.85
0.39
0.90
10.56
0.55
HEGNN
10.20=0023
4.63 62.82.1425
12.85
0.37
0.94
10.55
0.52:1001
HEGNNI36
9.94
59.9345m
4.62
12.85
0.37+amz
0.88
10.56
0.54

---

5.3 Perturbation Experiment
In practical scenarios, slight perturbations (such as molecular vibrations) can disrupt strict symmetry. potentially mitigating the conclusions outlined in Theorems 3.5 and 3.6. We therefore designed this perturbation experiment for simple study and were surprised to find that HEGNN can still bring better robustness through the introduction of high-degree steerable features.
Design of experiments: We take the tetrahedron as example and compare the cases of EGNN, HEGNN,=3, and HEGNN1s3 when adding noise perturbations with results in Table 5. Here, 6 represents the ratio of noise, and the modulus of the noise obeys NN [B|l Tell] f).
Table 5: Results for perturbation experiment.
E = 0.01
E = 0.05
0.10
= 0.50
EGNN
50.0 0.0
45.0 = 15.0
65.0 22.9
60.0 20.0
HEGNN
100.0 0.0
100.0 0.0
100.0 = 0.0
100.0 0.0
HEGNNI33
100.0
100.0 0.0
100.0
100.0 0.0
Results: It can be observed that the performance of EGNN is slightly improved in the presence of noise (from 50% when = 0.01 to 60% when = 0.5), while HEGNN demonstrates better robustness. Even though symmetry-breaking factors will make the geometric graph deviate from the symmetric state, the deviated graph is still roughly symmetric. In other words, the outputs of equivariant GNNs on the derivated graphs keep close to zero if the degree value is chosen to those in Table I, which will still lead to defective performance.
6 Conclusion
In this paper, we challenged the prevailing notion that higher-degree steerable vectors are unnecessary for achieving expressivity in equivariant Graph Neural Networks (GNNs). Through rigorous theo- retical analysis, we demonstrated that equivariant GNNs constrained to Ist-degree representations inevitably degenerate to zero functions when applied to symmetric structures, such as k-fold rotations and regular polyhedra. To address this limitation, we introduced HEGNN, a high-degree extension of the EGNN model. HEGNN enhances expressivity by integrating higher-degree steerable vectors while retaining the efficiency of the original model through a scalarization technique. Our extensive empirical evaluations various datasets, including the symmetric toy dataset, N-body, and MD17, validate our theoretical predictions. HEGNN not only adheres to our theoretical insights but also exhibits significant performance improvements existing models. These findings underscore the critical role of higher-degree representations in fully leveraging the potential of equivariant GNNs.
Acknowledgment
This work was jointly supported by the following projects: the National Science and Technology Major Project under Grant 2020AAA0107300, the National Natural Science Foundation of China (No. 62376276, No. 62172422); Beijing Nova Program (No. 20230484278); Beijing Outstanding Young Scientist Program (No. BJJWZYJHO122 2019 2019100020098,, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (23XNK]19);
Public Computing Cloud, Renmin University of China.
References
[1] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning, pages 9323-9332. PMLR, 2021.
[2] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.
[3] Rui Jiao, Xiangzhe Kong, Ziyang Yu, Wenbing Huang, and Yang Liu. Equivariant pretrained transformer for unified geometric learning on dulti-domain 3d molecules. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024.
10

---

[4] Jun Wu, Xiangzhe Kong, Ningguan Sun, Jing Wei. Sisi Shan, Fuli Feng, Feng Wu, Jian Peng. Linqi Zhang. Yang Liu, et al. Better prior distribution for antibody design. Available at SSRN 4909414, 2024.
[5] Thorben Frank, Oliver Unke, and Klber-Robert Müller. So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems. Advances in Neural Information Processing Systems, 35:29400-29413, 2022.
[6] Yusong Wang, Tong Wang. Shaoning Li, Xinheng He, Mingyu Li, Zun Wang, Nanning Zheng, Bin Shao, and Yan Liu. Enhaneing geometric representations for molecules with equivariant vector-scalar interactive message passing. Nature Communications, 15(1):313, 2024.
[7] Runfa Chen, Jiaqi Han, Fuchun Sun, and Wenhing Huang. Subequivariant graph reinforcement learning in 3d environments. In International Conference on Machine Learning, pages 4545- 4565. PMLR, 2023.
[8] Runfa Chen, Ling Wang, Yu Du, Tianrui Xue, Fuchun Sun, Jianwei Zhang, and Wenbing Huang. Subequivariant reinforcement learning in 3d multi-entity physical environments. Forty-first International Conference on Machine Learning. 2024.
[9] Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. Advances in Neural Information Processing Systems, 35:26256-2666 26268, 2022.
[10] Liming Wu, Zhichao Hou, Jirui Yuan, Yu Rong, and Wenbing Huang. Equivariant spatio- temporal attentive graph networks to simulate physical dynamics. Advances in Neural Informa- tion Processing Systems, 36, 2024.
[11] Jiaqi Han, Minkai Xu, Aaron Lou, Haotian Ye, and Stefano Ermon. Geometric trajectory diffusion models. arXiv preprint 1xXi:2440.110221 2024.
[12] Yuxuan Song. Jingjing Gong, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei- Wei-Ying Ma. Unified generative modeling of 3d molecules with bayesian flow networks. In The Twelfth International Conference on Learning Representations, 2023.
[13] Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pages 38592-38610. PMLR, 2023.
[14] Yanru Qu, Keyue Qiu, Yuxuan Song, Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei-Ying Ma. Molcraft: Structure-based drug design in continuous parameter space. In Forty-first International Conference on Machine Learning, 2024.
[15] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2022.
[16] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):008--111
2023.
[17] John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang. Ahmed Ismail, Vin- cent Frappier, Dana Lord, Christopher Ng-Thow- Erik R Van Vlack, al. Illuminating protein space with programmable generative model. Nature pages 1-9, 2023.
[18) Ziyang Yu, Wenbing Huang, and Yang Liu. Rigid protein-protein docking via equivariant elliptic-paraboloid interface prediction. In The Twelfth International Conference on Learning Representations, 2024.
[19] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint 1rXiv:1802.0000 2018.

---

[20] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (rnntraasfrrmm::: 3d troostranslttion equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1999 1981, 2020.
[21] Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.
[22] Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications, 14(1):579, 2023.
[23] Nadav Dym and Haggai Maron. On the universality of rotation equivariant point cloud networks arXiv preprini urXiv:2010.02449, 2020.
[24] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867-8887. PMLR, 2022.
[25] Minkai Xu, Jiaqi Han, Aaron Lou, Jean Kossaifi, Arvind Ramanathan, Kamyar Azizzadenesheli, Jure Leskovec, Stefano Ermon, and Anima Anandkumar. Equivariant graph neural operator for modeling 3d dynamics. arXiv preprint arXiv:2401. 11.00011110.... 2024.
[26] Omri Puny, Matan Atzmon, Edward Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2021.
[27] Alexandre Agm Duval, Victor Schmidt, Alex Hernandez-Garcia, Santiago Miret, Fragkiskos D Malliaros, Yoshua Bengio, and David Rolnick. Faenet: Frame averaging equivariant gnn for materials modeling. In International Conference on Machine Learning, pages 9013-9033.
PMLR, 2023.
[28] Yuelin Zhang, Jiacheng Cen, Jiaqi Han, Zhiqiang Zhang, Jun Zhou, and Wenbing Huang. Improving equivariant graph neural networks on large geometric graphs via virtual nodes learning. In Forty-first International Conference on Machine Learning. 2024.
[29] Jiaqi Han, Wenbing Huang, Tingyang Xu, and Yu Rong. Equivariant graph dierarchy-basee neural networks. Advances Neural Information Processing Systems, 35:9176-911 798777 2022.
[30] Yusong Wang, Chaoran Cheng, Shaoning Li, Yuxuan Ren, Bin Shao, Ge Liu, Pheng-Ani Heng, and Nanning Zheng. Neural p² m: A long-range interaction modeling enhancer for geometric gnns. Advances in Neural Information Processing Systems, 2024.
[31] Jiaqi Han, Jiacheng Cen, Liming Wu, Zongzhao Li, Xiangzhe Kong, Rui Jiao. Ziyang Yu, Tingyang Xu, Fandi Wu, Zihe Wang, al. A survey of geometric graph neural networks: Data structures, models and applications. arXiv preprint arXiv:2403.00485 2024.
[32] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377-9388. PMLR, 2021.
[33] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. In International Conference on Learning Representations, 2021.
[34] Johannes Gasteiger, Florian Becker, and Stephan Gunnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, (86799-6802, 2021.
[35] Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Lio. On the expressive power of geometric graph neural networks. International Conference on Machine Learning, pages 15330-15355. PMLR, 2023.
12

---

[36] Boris Weisfeiler and Andrei Leman. The reduction of graph to canonical form and the algebra which appears therein. nti, Series, 2(9):12-16, 1968.
[37] Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gābor Csányi Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:1 15:11423-111 1436,, 2022.
[38] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
[39] Mario Geiger and Tess Smidt.
c3nn: Euclidean neural networks arXiv preprint
arXiv:2207.09453. 2022.
[40] David J Griffiths and Darrell Schroeter. Introduction to quantum mechanics. university press, 2018.
Cambridge
[41] Lev Davidovich Landau and Evgenii Mikhailovich Lifshitz. Quantum mechanics: relativistic theory, volume 3. Elsevier, 2013.
[42] J Thorben Frank, Oliver T Unke, Klaus-Robert Müller, and Stefan Chmiela. A euclidean trans- former for fast and stable machine learned force fields. Nature Communications, 15(1):6539, 2024.
[43] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning. pages 2688-2697 PMLR, 2018.
[44] Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller, Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e660311,, 2017.
[45] Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equiv- ariant graph mechanics networks with constraints. In International Conference on Learning Representations, 2022.
[46] Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi-body systems with symmetric energies. arXiv preprint arXiv: 2019.
[47] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1263-1272. PMLR, 2017.
[48] Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24), 2018.
[49] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
[50] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pages 5583-5608, PMLR, 2022.
[51] James F Epperson. An introduction to numerical methods and analysis. John Wiley & Sons, 2013.
[52] Michael Engel. Point group analysis in particle simulation data. arXiv preprint arXiv:2106. 14846, 2021.
[53] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in Neural Information Processing Systems, 30, 2017.
13

---

[54] Hannah Lawrence, Vasco Portilheiro, Yan Zhang, and Tekou-Ouman Kaba. Improving equiv- ariant networks with probabilisticc symmetry breaking. In ICML 2024 Workshop on Geometry- grounded Representation Learning and Generative Modeling, 2024.
[55] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molecular graphs. Advances in Neural Information Processing Systems, 35:650-664 2022.
[56] Kelin Xia and Guo-We Wei Wei. Persistent homology analysis of protein structure, flexibility, and folding. International journal for numerical methods in biomedical engineering. 30(8):814 844, 2014.
[57] Floor Eijkelboom, Rob Hesselink, and Erik J Bekkers, E(n) equivariant message passing simplicial networks. In International Conference on Machine Learning, pages 9071-9081. PMLR, 2023.
[58] Claudio Battiloro, Ege Karaismailoğlu, Mauricio Tec, George Dasoulas, Michelle Audirac, and Francesca Dominici. E (n) equivariant topological neural networks. arXiv preprint arXiv:2405. 15429, 2024.
[59] Ralf Drautz. Atomic cluster expansion for accurate and transferable interatomic potentials. Physical Review B, (9(1):11444 2019.
[60] Genevieve Dusson, Markus Bachmayr, Gábor Csányi, Ralf Drautz, Simon Etter, Cas van Der Oord, and Christoph Ortner. Atomic cluster expansion: Completeness, efficiency and stability. Journal of Computational Physics, 454: 110446, 2022.
[61] Xiangzhe Kong, Wenbing Huang, Zhixing Tan, and Yang Liu. Molecule generation by principal subgraph mining and assembling. Advances in Neural Information Processing Systems, 35:2550- 2563, 2022.
[62] Xiangzhe Kong, Wenbing Huang, and Yang Liu. Generalist equivariant transformer towards 3d molecular interaction learning. In Forty-first International Conference on Machine Learning, 2023.
[63] Xiangzhe Kong, Wenbing Huang, and Yang Liu. Full-atom peptide design with geometric latent diffusion. Advances in Neural Information Processing Systems, 2024.
[64] Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. Roto-translated local coordinate frames for interacting dynamical systems. Advances in Neural Information Processing Systems, 34:6417-6429, 2021.
[65] Yi-Lun Liao, Brandon M Wood, Abhishek Das, and Tess Smidt. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. In The Twelfth International Conference on Learning Representations, 2024.
[66] Ziqiao Meng, Liang Zeng, Zixing Song, Tingyang Xu, Peilin Zhao, and Irwin King. Towards geometric normalization techniques in se(3) equivariant graph neural networks for physical dynamies simulations. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), pages 5981-5989, 2024.
[67] Zian Li, Xiyuan Wang, Yinan Huang, and Muhan Zhang. Is distance matrix enough for geometric deep learning? Advances in Neural Information Processing Systems, 36, 2024.
[68] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023.
[69] Weiliang Luo, Gengmo Zhou, Zhengdan Zhu, Yannan Yuan, Guolin Ke, Zhewei Wei, Zhifeng Gao, and Hang Zheng. Bridging machine learning and thermodynamics for accurate k prediction. JACS Au, 2024.
[70] Gengmo Zhou, Zhen Wang, Feng Yu, Guolin Ke, Zhewei Wei, and Zhifeng Gao. S-molsearch: 3d sepi-super contrastive learning for bioactive molecule search. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
14

---

[71] Fanmeng Wang, Hongteng Xu, Xi Chen, Shuqi Lu, Yuging Deng, and Wenbing Huang. Mper- former: An se (3) transformer- based molecular perceptron. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pages 25222252, 2023.
[72] Fanmeng Wang, Wentao Guo, Minjie Cheng, Shen Yuan, Hongteng Xu, and Zhifeng Gao. Mmpolymer: A multimodal multitask pretraining framework for polymer property prediction. In Proceedings of the 33rd ACM International Conference Information and Knowledge Management, CIKM '24, 2024.
[73] Johannes Klicpera, Janek Groß, and Stephan Günnnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020.
[74] Angxiao Yue, Dixin Luo, and Hongteng Xu. A plug-and-play quaternion message-passing module for molecular conformation representation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 16633-16641, 2024.
[75] Yang Zhang, Wenbing Huang, Zhewei Wei, Ye Yuan, and Zhaohan Ding. Equipocket: an (3)-equivariant geometric graph neural network for ligand binding site prediction. In Forty-first International Conference on Machine Learning. 2024.
[76] Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu, Energy -hergyymotivated equivariant pretraining for 3d molecular graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8096-8104, 2023.
[77] Ziyang Yu, Wenbing Huang, and Yang Liu. Force-guided bridge matching for full-atom time-coarsened dynamics of peptides. arXiv preprint 1XX1::2408.11226. 2024.
[78] Qi Li, Rui Jiao, Liming Wu, Tiannian Zhu, Wenbing Huang, Shifeng Jin, Yang Liu, Hongming Weng, and Xiaolong Chen. Powder diffraction crystal structure determination using generative models. arXiv preprint arXiv:2409.04F 04727, 2024.
[79] Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[80] Rui jiao, Xiangzhe Kong, Wenbing Huang, and Yang Liu. 3d strueture prediction of atomic systems with flow-based direct preference optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[81] Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations, 2022.
[82] Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full-atom antibody design. In Proceedings of the 40th International Conference on Machine Learning, pages 17409-17429, 2023.
[83] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
15

---

A Theoretical Details
A.1 Equivariannc//lvariance of HEGNN
In this section, we demonstrate the equivariance of our HEGNN. order to further illustrate the connection between our HEGNN, EGNN, and TFN, general proof is given here.
Table 6: Comparison between HEGNN and the scalarization-based model representing EGNN [1], and the high-degree steerable model representing TFN [19]. HEGNN combines the scalarization trick of EGNN that only uses invariant scalars (Oth degree steerable features) interact between steerable features corresponding to different degrees, avoiding the high computational cost of using CG tensor products in TFN.
EGNN (1]
TEN [19]
HEGNN (Ours)
m. Pm (h,, h1 Cig. di DiLL 2i
Msg
my Om (h,, Cyy
miy a!!h)
org
wids) y
Rii ||
m, pe(m.y) (Ø, xz)
my pe(m)s) - (oh, z.g)
5 v. (m))) (6.")
EsENio maj
m, EjeN(: my
Agg
m Eserro mi
(2)
m. EieN(o mhs
m. 0. E)ENio m.y
!)) = 0; ESeNis mh
Δh: Ph (h, m.)
A/: (h,,, m.)
Upd
Aet (b) m!
AZ; m.
AZ: mi
Ao
Theorem A.1 (Equivariance/Invarinnc of HEGNN), h, 5 in HEGNN is E(3) invariant, ã; is E(3) equivariant. addition, all 6 are 0(3) equivariant and translation invariant when is odd; all 0f is 50(3) equivariant and inversion/translatinn invariant to when 1 even.
Proof. Consider a sequence composed of functions (s:: (((11) - x200 equivariant to same
group G) the equivariance lead to an interesting property that
PN ..... Pi+1 Px (h) ...
PN Pj+1 Pxti (h) ..... $11
holds for all i,j f1,2.... ... N and 5), which means that the group elements h can freely exchanged in the composite sequence of equivariant functions. In particular, if one of the equivariant functions (e.g. (px) is replaced by an invariant function, the group element will be absorbed, which means
N
Pk: o o PP++1 Pxcco (h)sp ..... o yp1 PPN G P11
holds for all h 5 but only i E 21, k}. Although pN ypk is still equivariant, because the group elements must be input starting from øı, the overall yN o ... 1o is still an invariant function. That is to say, to conclude that the entire HEGNN is equivariant, only need to prove that HEGNN equivariant in initialization and each layer.
The initialization of HEGNN is based on spherical harmonics, which is similar to TFN. Spherical harmonies are inherently equivariant, that is.
y yO (R.T)= D (tt)
Note that variables participating in the coefficient in Eq. (5) are all invariant scalars, so the initialization of HEGNN is consistent with the spherical harmonic function. Note that for Cartesian vectors, they can be aligned by arranging the spherical harmonics of 1st degree [19], that is,
T x Y (1) (z).
From this perspective, EGNN can also be considered to be initialized using spherical harmonics, but this step is omitted because the value is proportional to the input Cartesian vector.
It is worth explaining that spherical harmonics are inversion invariant when is even, that is,
Y (1) (mz) Y (4) (z).
16

---

Equivariant ones are not necessarily better than invariant ones. When we need to predict pseudovectors (such as moments), we need to inversion invariant Ist-degree steerable features, because pseudovectors are inversion invariant. This is why introducing the cross product # y (the result is inversion invariant) into a linear combination can only build a SF(3) equivariant network, but not F(() network [50].
In fact, inversion equivariann/invariant high degree steerable features can be obtained by calculating the CG tensor product of spherical harmonics and inversion equivariant Cartesian vectors like 2; He [39]. Moreover, the equivariance at each layer is also easy to prove and the internal Wigner-D matrix can be extracted through the CG tensor product [31].
(b)6 - [(D (b)6(i))
w Bgg
(D")))))()))
When the output result is an invariant scalar, the Wigner-D matrix degenerates into a trivial repre- sentation 1. Norm and inner product are all special cases of this type, so equivariance is established. From this perspective, EGNN and HEGNN are equivalent to using only the weight coefficients of (l. I) + and (0, I) + Similar ideas include the steerable MLPs in [33].
A.2 Other Proofs
Theorem A.2 (Theorem 3.4). Suppose that f()) is an -(3))-quivariant function on geometric graphs, regarding the group representation pd)) defined in Eq. (2), Then, for any symmetric graph G induced by the group S 0(3), namely, VG E GG)), we always have
( = 600 (5)
(11)
Here we have defined group average (5)) T5 Enes (b)-
Proof. If f()) is (e(-)vevvvariant, then is also a Seequivariant, thus
(0) ( (())
(() |5
1
Σ ( 9)- BE
1 Σ (()) |55
_ p(b) hEn
(5
DES
Theorem A.3 (Theorem 3.5). If and only if the matrix I2t+ (5)) is non-singular, the 0(3)- equivariant function f()) is always a zero function G, namely.
VG E G(5). (T) =0
(12)
Proof. From Theorem 3.4, we know that
= (())) = (I4+1- (0)) (() = 0,
and the theorem holds for basic knowledge of lincar algebra.
Theorem A.4 (Theorem 3.6). For a finite group 5 with its representation p()), (() is a zero matrix (i.e., (5) = 0) if and only ( (5)) 0. In this case, ( = 0, Vg E G(5).
Proof. It is obvious that (S)) 0 mention the main diagonal, and we only to prove (((p((())) 0 - ( 0.
tr(o((()) (5)) 0 since all elements are zero not to
basic fact is h 5 = 5,, thereby
0 (b)p (5))
Now we can use group average and get
(s)) = (5)
17

---

Such operation can be repeated many times, so that (o)(5)) (5))
AA E
Now we calculate the trace for each matrix and find
(((0()))
(5)) =0
Ak E
By Newton's identity [51]. all eigenvalues of the matrix (( are O, that is, the matrix is a zero matrix.
Table 7: The traces of symmetric groups based on [52]. Trace for polyhedral groups can be calculated by ( = [t/r] + 61l mod r] with repeat length where is a string only with or 1. For exmaple, for Tetrahedral group T, D (T) 5/6] b|5 mod 6| = 0 + 0 0.
Group
Notation
Data for Wigner-D matrix traces D (H)
Reflection group
C
(2l 1). di mod 2,0
Cyclic group
Cn
2[1/n] +
Dihedral group
Dv
[1/n| + ôt mod 2,0
Tetrahedral group
T
r= 6
b = 100110
Octahedral group
O
r= 12
b 100010101110
Icosahedral group
I
30 b 10000000100101100111111111111111
Theorem A.5 (Theorem 4.1). For any geometric graph, there exists bijection between the
[hugl set of inner products { Re given by Eg. (10) and the set of edge angles Au; = (Oisst = se)t,,,e)))
Proof. Note that the Legendre polynomial is set of orthogonal polynomial bases, and there is bijection to the power function polynomial space, that is
Assl p() (cos 0n)
Asgl Σ n=1
span
span
cos
On
1=0
a=
where M is any non-negative integer represents the degree of the polynomial space. Moreover, from the knowledge of Newton's identities, the space of power sums can be converted to space of elementary symmetric polynomials as
M
} M
[Aagl Σ
span
cos On
span
Σ
II cos Onx
Isny <ng mm S Aagl
a=0
a=0
From Vieta's 's formulas, when M Aigl. with the M polynomial in the space of elementary symmetric polynomials being coefficients, we can build a AAi -dderre polynomial with {cos 0 E A as its all roots?. Since all angles are in [0, x). the cosine uniquely determines the angle value, and the proposition is established.
A.3 Further Discussion
Our theory in fact shows that the degeneration of global features certain degree (in Table are inevitable symmetric geometric graphs. This raises two points worth discussing:
1. The degree indicated degenerate not necessarily produce non-zero representation, which may still be affected by the model form and the edge situation.
2. There are some tricks to get around this degeneration: for example, making the output set relaxing equivariance constraints (e.g. probabilistic symmetry breaking [54]).
The trick comes from Lemma in DeepSet [53].
18

---

It is worth mentioning that outputting set can solve most problems, although such operators may be quite intractable implement in computer systems. The failure cases of frame averaging [26; 27] and Neural P [30] which depend on singular value decomposition or eigenvalue decomposition, is caused by the non-unique matrix decomposition. Some other examples include ComENet [55], which uses the scatter scatter_ming () operator in PyTorch to extract the nearest neighbors, making it intractable to handle the situation where multiple neighbors are simultancously closest, which is quite common in chemical molecules (e.g. -CH3, -NH2).
Moreover, from Theorem 4.1, we show the expressivity our HEGNN, which is able to recover the information of all angles between each pair of edge. However, it should be noted that |Ail may be an extremely large number, which is unacceptable practical applications achieve completeness. The same problem also arises in discussions based on CG tensor product models (e.g. TFN (19]). such as discussions based on D-spanning [23], because sufficiently high-degree However, in terms of actual results, the performance of both our HEGNN and TFN is remarkable. From this perspective, how to bridge the gap between completeness and actual performance with features of limited channels and degrees is question worth considering.
unacceptable.
To get the ball rolling, raise interesting question here. Is there performance gap between this type of purely mathematical representation and other features based on physical and biochemical prior knowledge?
Purely mathematical representation: topological characteristics [56-58], cluster expansion basis [59; 60), subgraph blocks [61-63], frames [26; 27; 64), normalization operators [65; 66];
Features based on physical biochemical prior knowledge: distance matrix [67-72], chemical bond length, angle and dihedral angle [55; 73-75), force [76; 77), fractional coordinates [78-80], canonical ordering [81; 82).
In fact, some these features are directly related (e.g. frames and fractional coordinates). How to construct effective and interpretable pure mathematical features based those with prior knowledge will become key point in network design, and we will consider further exploration future work.
B More Experimental Details and Results
B.1 Comparison parameters between models
Like EGNN [1], different features use different numbers of channels, so the inference time does not obviously reflect the time complexity of O( We list the details of our HEGNN of different degree Table Intuitively, we add one of each steerable feature on the basis of EGNN (using 64 invariant scalars and 2 Cartesian vectors, i.e. coordinate and velocity).
Table 8: Channels for steerable features of different degrees and total dimensions of HEGNN of different degrees.
HEGNNI
HEGNNI
HEGNN 133
HEGNNI
Channel for (0)
65
65
65
65
Channel for ((1)
3
3
3
3
Channel for ((2)
1
1
1
Channel for 5(3)
1
1
Channel for (4)
1
Channel for (G)
1
Channel for (6)
1
Total dimensions
74
79
86
119
Table 9 shows the number of parameters and inference time of different models.
19

---

Table 9: Parameters and inference time (on 100-body dataset) of EGNN and other high-degre models.
Parameters
1 S
1 2
S
EGNN
134. 1k
HEGNN
160.3k
160.9k
161.5k
163.2k
TFN
3.7M
9.6M
19.5M
86.6M
SEGNN
228. 1k
244.1k
254.9k
288.1k
MACE
14.8M
38.0M
77.0M
342.5M
Inference Time (10 (10-3s)
[ <<
1 S
1 S
I 6
EGNN
0.57
HEGNN
0.82
0.88
0.91
1.08
TEN
3.75
2.66
00M
SEGNN
1.33
1.7
1.98
22.33
MACE
22.10
125,87
261.11
OOM
B.2
Experiment on k-fold Structure.
The results on the k-fold structure are completely consistent with the conclusion of Table 1. The few results that cannot reach 100% are due to the small number of training epochs, so the classifier fails to perform perfect classification. In fact, the accuracy of models can achieve 100.00z oa after increasing the number of training rounds of the model like 500 epochs. Since from 2nd-degree steerable features can distinguish Go and G1, and HEGNNTFN/MACE&s. contain 2nd 2nd-degree steerable features when I 3, the extra results are hidden but all their values are 100.00 ao.
Table 10: k-fold symmetric structures.
Rotational symmetry fold
GNN Layer
2 fold
fold
10 fold
Cart.
E-GNN-=1
50.0 0.0
50.0 =0.0
50.0
50.0 =.00
GVP-GNN
50.0 0.0
50.0 000
50.0
50.0 0.0
HEGNN =1
50.0
50.0 0.0
50.0
50.0 #.0
Single Type Spherical
HEGNN
95.0 #15.0
95.0 +15.0
95.0 215.0
95.0 =15.0
HEGNN 3
50.0
75.0 #40.31
50.0 10.000..
50.0 +.00
HEGNN
100.0 0.0
95.0 +15.0
95.0 +15.0
95.0 +15.0
HEGNN 5
50.0
100.0
100.0
50.0 0.0
HEGNN
100.0 0.0
95.0 +15.0
100.0 0.0
95.0 +15.0
HEGNN =7
50.0 0.0
95.0
90.0 =30.0
50.0 2.00
HEGNN
100.0 0.0
90.0 =30.0
85.0 +32.02
85.0 =32.02
HEGNN =9
50.0 0.0
100.0
100.0 0.0
50.0 +.00
HEGNN
100.0 0.0
100.0 0.0
100.0 0.0
100.0 4.0
HEGNN
50.0 0.0
100.0
95.0 415.0
50.0 +.00
Sph.
HEGNN//EE//MCCE:/
50.0 0.0
50.0 + 0.0
50.0 = 0.0
50.0 0.0
HEGNN/TENMACE <2
100.0
100.0
100.0 0.0
100.0
B.3 Further Verification on Regular Polyhedra
Implementation details: Since the e3nn [39] library only implements spherical harmonics up to 11th-degree, we verified the expressivity of our HEGNN in disguise. In this experiment, we measure the expressivity of our HEGNN by calculating whether the high-degree feature i:") updates on regular polyhedra. Namely, if (a(4) 0, then HEGNN= loses the expressivity. This judgment is necessary and sufficient. We calculated the sum of all degree vectors from L 1 to 30, which
20

---

covers the maximum repeat length in Table 7. We implemented the calculation of the high-degree features based on scipy [83] library.
Results: Our calculation results are shown in Table 11 which completely consistent with the theoretical results in Table 7. The simple experiment shows that the of spherical harmonics E, y Fe), function graph, will actually vanish regular polyhedra for some integer degree
Table 11: Expressivity analysis of HEGNN using sums of spherical hurmonics. Here, "True" indicates that our HEGNN can distinguish the orientations, meaning the norm of the sum of spherical harmonics greater than "False" in the table means that distinction can be made, with the norm of the corresponding spherical harmonic being less than 10 30-3
Rotational symmetry
L
Tetrahedron
Cube
Octahedron
Dodecahedron
Icosahedron
False False
False
False
False False
False
2
False
False
False
3
True
False
False
False
False
True
True
True
False
False
5
False
False
False
False
False
6
True
True
True
True
True
True
False
False
False
False
8
True
True
True
False
False
True
False
False
False
False
10
True
True
True
True
True
11
True
False
False
False
False
12
True
True
True
True
True
13
True
Falsc
False
False
False
14
True
True
True
False
False
15
True
False
False
False
False
16
True
True
True
True
True
17
True
False
False
False
False
18
True
True
True
True
True
19
True
False
False
False
False
20
True
True
True
True
True
21
True
False
False
False
False
22
True
True
True
True
True
23
True
False
False
False
False
24
True
True
True
True
True
25
True
False
False
False
False
26
True
True
True
True
True
27
True
False
False
False
False
28
True
True
True
True
True
29
True
False
False
False
False
30
True
True
True
True
True
B.4 Results on other dataset settings
The N --body dataset setting of this paper refers to FastEGNN [28]*. that is, 5,000 samples are used as the training set (instead of 3,000). We tested the situation of using different data segmentation in Table 12. We also add ClofNet [50], local frame based scalarization method and SEGNN [33] (select according to the optimal situation the paper) and MACE [37] (1 2), two classic high-degree steerable models for comparison.
'nttps://githut Baw.aabocco/GLL--UUUEBBBBC
21

---

Table 12: Results of N-body dataset under partitions.
N-booy(x10-3)
5-body
20-body
50-body
100-body
train/validdeestt33////2
EGNN
0.71
1.08
1.16
1.29
ClofNet
0.89
1.79
2,40
2.94
ClofNet-vel
0.84
1.50
2.28
2.67
GMN
0.67
1.21
1.18
2.55
MACE
1.43
1.93
2.20
2.51
SEGNN
1.81
2.67
3,44
NaN
HEGNN 1
0.64
0.84
0.92
1.04
HEGNN
0.69
0.89
1.13
0.94
HEGNN 13
0.58
1.04
0.92
1.04
HEGNN 156
0.77
1.06
1.02
1.18
truin/validdeesttSWWWW/
ClofNet
0.80
1.49
2.28
2.77
ClofNct-vel
0.78
1.45
2.22
2.77
GMN
0.52
0.98
1.04
1,21
MACE
1.13
1.60
2,41
3.38
SEGNN
1.68
2.63
3,30
NaN
HEGNN 1-1
0.52
0.79
0.88
1.13
HEGNN 122
0.47
0.78
0.90
0.97
HEGNN 1333
0.48
0.80
0.84
0,94
HEGNN 156
0.69
0.86
0.96
0.86
Note that SEGNN in Table 12 does not show the effect in the original paper. We also tried to reproduce the dataset in the original paper [33], and the effect of SEGNN on 5-body dataset can be reproduced. However, see Table 13, it also shown that SEGNN performs poorly on larger datasets.
Table 13: Comparison between EGNN, SEGNN and HEGNN N-body from [33]
N-body(x10-3)
5-body
20-body
50-body
100-body
EGNN
0.71
1.04
1.15
1.31
SEGNN
0.50
6.61
9.34
13.46
HEGNN 1S1
0.71
0.97
0.93
1,22
HEGNN 122
0.65
0.91
1.05
1.14
HEGNN 13
0.63
0.99
1.05
1.27
HEGNN 156
0.72
1.05
1.11
1.28
We found that the GMN-L method proposed in [29] generally performs the best on MD17 dataset, but our HEGNN-6 also achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance.
22

---

Table 14: Prediction error (x10-3) MD17 dataset. Results averaged across 3 runs.
Aspirin
Benzene
Ethanol
Malonaldehyde Naphthalene Salicylic
Toluene
Uracil
10.94_001
103.72:1..0 2.64
13.93
0.50
1.23
10.93
0.64.000
EGNN
=4.4
62,40
4.64
13.64
0.47
1.02
11.78
0.64
EGNNReg
13.82=019 17.684037
6.062001
13,49
0.63
1.68
11.05
0.661001
GMN
10.14=003
42:00:
4.83
13.11
0.40
0.00
10.22:
0.59
GMN-L
9.76##0.1
54.17
4.63
12.82
0.41+to1
0.88
10.45
4.59
TFNi22 12.37=0.18
18.48
4.81
10004
13.62:1008
0.49
1.03
10.89
0.84
SE(3)-Taag2
11.12:0.06
68.
10667
1.74
13.89
0,52
1.13
10.88
0.79
HEGNNI11
10.32=0.58
62.53
4.63
12.85
0.38
0.90
10.56
0.56+
HEGNNI
10.04=0.45
61,801592
4.63 10001
12.85
0.39
0.91 1:006
10.56:1..0
0,55
HEGNN
10.20
4228
4.63 1.001
12.85
0,37
0.94
10.55
0.52:10001
HEGNNI
9.94
59.9315211
4.62
12.854000
0.37
0,88
10.56
0.54 1:000
B.5 Expressiveness of initialization layer
Note that both EGNN [1] and HEGNN151 only use Cartesian vectors. However, in Table 3, the effect of the latter is greatly improved. We speculate that there are two possible factors for this improvement: 1) the extra layer of message passing brought in by Eq. (5); 2) the multi-channel Cartesian vectors (see Table 8). Therefore, we tested the effect of LIGGNNNg--lleert and the results are shown in Table 15. From the improvement, first factor contributes more.
Table 15: Comparison between EGNN and HEGNN on N-body.
N-hody(x10-3)
5-body
20-body
50-body
100-body
EGNN-4layers
0.65
1.01
1.00
1.36
HEGNN
0.63
0.98
0.96
1.31
0.52
0.79
0.88
1.13
C
Limitation
Our current experiments are mainly limited to testing on small molecules and have not been verified large-scale molecules large-scale physical systems. Whether HEGNN is effective large-scale geometric graph data sets remains to be verified.
D Broader Impact
Our research belongs the field of AI for Science. The HEGNN proposed in this article is expected to better model scientific problems, thereby promoting the development of nigher-precision and efficient AI scientific models.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction nicely summarize the theoretical contributions mentioned in $ 3, the model design mentioned in 4, and the experimental results mentioned in $ 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Our current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. Whether our HEGNN is effective on large-scale geometric graph data sets remains be verified.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our main theoretical results are given in $ 3 and the proofs are given in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details of our experimental results are in "Design of experiments" in $ 5.1 and "Datasets" in $ 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
https://githin Bt////LLLARRUHHHONNAA
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
oolooooooollyloooooolooooommllll for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/
wttppcccppp//ccccceeee..e.cl..ll pubppp/ppppgeeee wwticc/ggg/////eieecieeccccccse/ /CdeeS/mmm sa i aassooooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details have listed in "Design of experiments" in $511 and "Datasets" in $ 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the variance of the model runs in Table 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We use NVIDIA A100-80G, which has written in $ 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:////nrraappeeeecaec/ee///// csQuidel ines?
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our research belongs the field of AI for Science. The HEGNN proposed in this article is expected to better model scientific problems, thereby promoting the develop- ment of nigher-preciste and efficient AI scientific models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in our paper have used with citation.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. paperswithcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: bttpss///iihhle GLAD-RUC/EEENN
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/modell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "CjUMqNnKTp", "reviewer": "Reviewer_op9c", "summary": "The paper studies the benefit of using higher order steerable features in geometric GNNs and theoretically identifies classes of symmetric geometric graphs where methods using only order-1 (or low order) features are guaranteed to fail.\nWith this in mind, the authors propose a simple and efficient way to integrate higher order features in the EGNN architecture.", "questions": "Eq 7, 8: do I understand correctly that this kind of model can not pass any information between features $\\tilde{v}^{(l)}$ of different orders $(l)$ beyond the invariant quantities used to compute $m_{ij}$? Then, that essentially means that, after the initial features are extracted via the first convolution with spherical harmonics in Eq. 5, *the rest of model* is invariant to independent rotations by O(3) of each feature $\\tilde{v}^{(l)}$, right?\nThis seems to suggest there should be a gap in performance with models like TFN or SEGNN, which instead allow for different frequencies (features of different orders) to interact inside the model. Can you elaborate on this?\n\n\nSec 5.2: it seems important to compare with SEGNN [21] since that work was also motivated by the idea of adapting EGNN to support higher order steerable features, if I remember correctly.\n\n\nTable 3: I would expect the model size to scale quadratically with the maximum frequency $L$ since $\\sum_{l=0}^L (2l+1) = (L+1)^2$, but the runtimes in table 3 do not show this pattern. Why is the case?\n\n\nTable 4: what are the SOTA on these datasets? It would be good to include the performance achieved by some previous works to give an idea of whether these are competitive results or not\n\n\nWhy introducing Theorem 3.5 when Theorem 3.6 seems more general and practical? Also, I think these theorems hold for any compact (not necessarily finite) subgroups by using the Peter-Weyl theorem.\n\n\nLine 226: is this a more efficient way to implement this fairly sparse operation?", "limitations": "Some limitations are addressed in the supplementary material.\n\nFor completeness, I think the authors could comment on the possible gap in expressiveness with other steerable methods like TFN and SEGNN which allow for features of different orders to interact (see my first question above).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Clearly written and well motivated paper.\n\nThe proposed architecture seems a valid alternative to existing steerable architectures, enabling the use of higher order features with reduced computational complexity.\n\nThe paper also includes a few interesting theoretical insights about the failure cases of EGNN.", "weaknesses": "While the overall novelty is a bit limited (the message that higher order features are necessary for expressiveness has been theoretically and empirically explored in a number of previous works in the literature), the theoretical derivation of the failure cases and the proposed architecture can be useful contributions to the community. \n\nSee questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!  We provide the following responses to your concerns:\n\n> **Q1: The performance gap between scalarization-based models (e.g. HEGNN) and high-degree steerable models (e.g. TFN, SEGNN).**\n\nIt is true that our HEGNN exclusively passes invariant quantities (inner products of high-degree representations) between features $\\tilde v^{(l)}$ of different degrees $l$, unlike the expensive CG tensor product used in TFN or SEGNN, which considers all possible interactions between different frequencies. Our model can be seen as a generalization of the scalarization trick in EGNN to high-degree representations. While the scalarization trick might somehow sacrifice model expressivity in theory, it has shown significantly better efficacy and efficiency in practice compared to conventional high-degree models, as demonstrated by EGNN paper and also our experiments here. Additionally, Theorem 4.1 indicates that passing inner products of full degrees is sufficient to recover the information of all angles between each pair of edges, affirming the theoretical expressivity of our HEGNN in characterizing the geometry of the input structure. \n\n\n\n> **Q2: Comparison with SEGNN is important**.\n\nNice suggestion! We have additionally compared our model with SEGNN in the table below, using the default settings from the public code. It is observed that SEGNN performs significantly worse than our model (in Table S1). We conjecture that the equivariant non-linear functions applied to the CG tensor products in SEGNN make it difficult to converge to a desirable solution during training, resulting in suboptimal performance. We will include these results in the revised version.\n\n\n\n> **Q3: The relationship between runtimes and model degree.**\n\nThank you for this valuable observation. While the model size does scale quadratically with the maximum frequency $L $, this does not mean that the exact runtimes in Table 3 follow the same pattern. By leveraging PyTorch, which is based on CUDA toolkit, computations for different frequencies can be largely parallelized. As a result, the actual implementation cost is lower than the quadratic complexity.\n\n\n> **Q4: SOTA on N-body and MD-17**.\n\nThanks for the reminder. For the N-body dataset, we follow the settings in FastEGNN [a], which was recently published and can be regarded as the SOTA model. As shown in the table below, in all cases from 5-body to 100-body, our HEGNN models generally outperform FastEGNN, verifying the effectiveness of our model's design. \n\nFor the MD-17 dataset, we apply the evaluation protocol from the GMN paper [b]. We thoroughly checked the methods in the citations of this paper and found that the GMN-L method proposed in [b] generally performs the best, thus consider it the SOTA method. Our HEGNN-6 achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance.\n\n[a] Zhang Y, Cen J, Han J, et al. Improving Equivariant Graph Neural Networks on Large Geometric Graphs via Virtual Nodes Learning.\n\n[b] Huang W, Han J, Rong Y, et al. Equivariant graph mechanics networks with constraints.\n\n**Table S1:** Results on N-body\n|N-body($\\times10^{-2}$)|**5-body**|**20-body**|**50-body**|**100-body**|\n|-|-|-|-|-|\n|FastEGNN|0.66|0.81|1.03|0.99|\n|SEGNN|1.68|2.63|3.30|Nan|\n|HEGNN≤1|0.52|0.79|0.88|1.13|\n|HEGNN≤2|**0.47**|**0.78**|0.90|0.97|\n|HEGNN≤3|0.48|0.80|**0.84**|0.94|\n|HEGNN≤6|0.69|0.86|0.96|**0.86**|\n\n**Table S2:** Results on MD-17\n|MD-17|Aspirin|Benzene|Ethanol|Malonaldehyde|Naphthalene|Salicylic|Toluene|Uracil|\n|-|-|-|-|-|-|-|-|-|\n|GMN|10.14±0.03|**48.12±0.40**|4.83±0.01|13.11±0.03|0.40±0.01|0.91±0.01|**10.22±0.08**|0.59±0.01|\n|GMN-L|**9.76±0.11**|54.17±0.69|4.63±0.01|**12.82±0.03**|0.41±0.01|**0.88±0.01**|10.45±0.04| 0.59±0.01|\n|HEGNN≤1|10.32±0.58|62.53±7.62|4.63±0.01|12.85±0.01|0.38±0.01|0.90±0.05|10.56±0.10|0.56±0.02|\n|HEGNN≤2|10.04±0.45|61.8±5.92|4.63±0.01|12.85±0.01|0.39±0.01|0.91±0.06|10.56±0.05|0.55±0.01|\n|HEGNN≤3|10.20±0.23|62.82±4.25|4.63±0.01|12.85±0.02|**0.37±0.01**|0.94±0.10|10.55±0.16|**0.52±0.01**|\n|HEGNN≤6|9.94±0.07|59.93±5.21|**4.62±0.01**|12.85±0.01|**0.37±0.02**|**0.88±0.02**|10.56±0.33|0.54±0.01|\n\n\n\n> **Q5: The relationship between Theorems 3.5 and 3.6.**\n\nThanks for your comment. In fact, Theorem 3.5 addresses the more general case, while Theorem 3.6 is a special case for practical convenience (since $I-0=I$ is full-rank).\n\nFor practical purposes, we consider only the finite symmetric group because only geometric graphs with one or two nodes can exhibit infinite symmetric groups, representing single atoms or diatomic molecules in physics [a]. We do not consider these cases due to their trivial topological structure in realistic tasks. Generalization to compact subgroups can be easily achieved by replacing summation with integration, with volume elements chosen to be normalized and invariant to group action. \n\nFrom a mathematical perspective, Theorems 3.5 and 3.6 essentially calculate the dimension of the invariant space of a group. This can be achieved by calculating the trace of the projection map of the subgroup [b]. Generalization to compact subgroups is more naturally done by introducing the projection map of compact subgroups, as directly found in [b]. However, for readability, we have not delved deeply into representation theory in this paper. In our future work, we may present our results and further findings using the language of representation theory.\n\n[a] Landau, L. D., and E. M. Lifshitz. Quantum Mechanics: Non-Relativistic Theory.\n\n[b] Fulton, William, and Joe Harris. Representation Theory. \n\n> **Q6: Efficiency of operators in Line 226.**\n\nYes, this is a more efficient way. In this form, we are able to apply e3nn, which is the most commonly used library for processing spherical harmonics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WLzVE962KH", "reviewer": "Reviewer_7wT3", "summary": "This paper studies the necessity of higher-degree features in geometric graph neural networks (i.e. graph neural networks processing data embedded in three-dimensional space), focusing on the ability to recognize data with non-trivial rotational inner-symmetry such as k-folds and regular polygons, and based on the findings, proposes and validates an extension of the E(n)-GNN architecture. Theoretically, the authors first state in Theorem 3.4 that any O(3)-equivariant function on an inner-symmetric graph must produce an output which has an identical inner-symmetry. This immediately puts a restriction on the possible space of outputs an equivariant function can produce, more specifically as the invariant subspace under the action of the inner-symmetry (Theorem 3.5). It follows that certain inner-symmetric inputs and certain choices of output spaces (i.e. degrees $l$) leads to degenerate output space {0} (Theorem 3.6 and Table 1). Consequently, it follows that using a sufficiently high-degree (even) features leads to non-degenerate output spaces of layers, hence be able to encode e.g. orientations of the input. The authors empirically validate the findings on expressive power in Section 5.1, and then demonstrate good performance of the proposed model based on high-degree features and invariant scalarization on n-body for n <= 100 and MD17 datasets.", "questions": "Please see the Weaknesses section.", "limitations": "The authors have discussed the limitations in the checklist, but not in the main text. I encourage the reader to move the discussions to a separate section in the main text or Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- S1. The paper is well-written and easy to follow.\n- S2. The claims on expressive power regarding symmetric inputs in Section 3.3 is correct, and is original as far as I can confirm; the closest related work is [1], and the authors have clarified the differences of their approach which seems technically correct. This claim is verified by the synthetic experiments in Section 5.1.\n- S3. The model proposed by the authors based on the theoretical claims is original as far as I am aware, and it seems useful in practice (Section 5.2-5.3), both in terms of performance and efficiency (slightly more costly than EGNN, but still cheaper than other networks involving higher-degree tensors such as TFN; but it is unclear how the approach compares to MACE, see W4 in the Weaknesses section). In particular, the fact that the model performs well on 100-body simulation task is interesting.\n\n[1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024)", "weaknesses": "- W1. While technically different from [1], this paper conveys a similar message: the use of higher-degree tensors in geometric neural networks is necessary to obtain higher expressive powers which is necessary for certain problems. Given this, one may argue that the GWL hierarchy in [1] is more general (as it handles general non-symmetric inputs as well). Furthermore, inputs with rotational inner-symmetry have been investigated in Section 5.2 of [1] to argue in favor of the higher-degree tensor features, precisely using the task of encoding rotations of the input; while the types of inner-symmetries considered in this work is more general (Table 1), the finding is fundamentally not very different, which can be understood as a weakness of the paper.\n- W2. It was not entirely clear to me why the capability of encoding rotations of inner-symmetric inputs would be important for practical learning tasks, although the experiments seem to imply so. For example, on n-body systems, I believe it is very unlikely that one would encounter symmetric inputs including the ones given in Table 1, due to symmetry-breaking factors such as numerical imprecision, simulation errors, and noise. I have similar concerns regarding chemical structures (e.g. MD17), as slight perturbations to the atomic coordinates are sufficient to eliminate the inner-symmetries (if they exist) and hence not experience the problems given in Theorem 3.5 and 3.6. This implies a gap between the theoretical arguments on the expressive power given in Section 3, and empirical results given in Sections 5.2 and 5.3.\n- W3. While Theorem 4.1 says that the proposed architecture can recover the information of all angles between each pair of edges, it does not directly address the original problem given in Theorem 3.5 and 3.6.\n- W4. I am not sure why MACE [2], while also using higher-degree features and already used for experiments in Tables 1 and 2, is not included for comparison in Tables 3 and 4.\n\n[1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024)\n\n[2] Batatia et al. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!  We provide the following responses to your concerns:\n\n> **W1. Differences and connections with GWL.**\n\nThank you for raising the comparison with the GWL paper [1]. Here, we would like to further highlight the difference between [1] and our paper: \n\n1. Different Motivations: the GWL paper aims to investigate the expressivity of different geometric GNNs from the perspective of WL test, whereas our paper focuses more on exploring the necessity of involving high-degree representations. The GWL test paper has discussed rotational inner-symmetry, but the analyses are only derived experimentally.  Upon the formal definition of symmetric graphs, we are able to derive rigorous and theoretical results to explain the expressivity degeneration of equivariant GNNs on symmetric graphs in Table 1. Besides rotational inner-symmetry, we have also investigated regular polyhedra, the derivations of which are not trivial and rely heavily on our derived Theorem 3.6.  \n2. Different Evaluation Scopes: while the GWL paper only discusses the phenomenon of rotational inner-symmetry degradation in a single section, we thoroughly explore more general kinds of symmetric graphs in our experiments. In addition, to evaluate the practical effectiveness of our proposed model, we conduct experimental comparisons on the N-body and MD-17 tasks. The performances align with our theoretical findings in Theorem 3.6 and Theorem 4.1, verifying the strong expressive power of our model.\n\n> **W2. Stability of symmetric structures under perturbations.** \n\nThanks for your question. We agree that it is unlikely to encounter exactly symmetric inputs in practice, but this does not indicate that our theoretical analyses are NOT practically meaningful. Even though symmetry-breaking factors will make the geometric graph deviate from the symmetric state, the deviated graph is still roughly symmetric. In other words, the outputs of equivariant GNNs on the derivated graphs keep close to zero if the degree value is chosen to be those in Table 1 according to Theorem 3.5 and 3.6, which wil still lead to defective performance. To explore this further, we take the tetrahedron as an example and compare the cases of EGNN, HEGNN$\\_{l= 3}$, and HEGNN$\\_{l\\leq 3}$ when adding noise perturbations.\n\n**Table S3:** Resluts under perturbations\n|                   | $\\varepsilon=0.01$ | $\\varepsilon=0.05$ | $\\varepsilon=0.10$ | $\\varepsilon=0.50$ |\n| ----------------- | ------------------ | ------------------ | ------------------ | ------------------ |\n| EGNN              | 50.0 ± 0.0         | 45.0 ± 15.0        | 65.0 ± 22.9        | 60.0 ± 20.0        |\n| HEGNN$_{l=3}$     | 100.0 ± 0.0        | 100.0 ± 0.0        | 100.0 ± 0.0        | 100.0 ± 0.0        |\n| HEGNN$_{l\\leq 3}$ | 100.0 ± 0.0        | 100.0 ± 0.0        | 100.0 ± 0.0        | 100.0 ± 0.0        |\n\nHere, $\\varepsilon$ represents the ratio of noise, and the modulus of the noise obeys $\\mathcal{N}(0,\\varepsilon\\cdot\\mathbb{E}[||\\vec x-\\vec x_c||]\\cdot I)$. It can be observed that the performance of EGNN is slightly improved in the presence of noise (from $50$% when $\\varepsilon=0.01$ to $60$% when $\\varepsilon=0.5$), while HEGNN demonstrates better robustness. The symmetry-breaking factors you mentioned are very interesting, and the results above will be included in our revised paper. \n\n> **W3. The difference between Theorems 3.5, 3.6, and Theorem 4.1.**\n\nWe are sorry for any potential confusion. Since our proposed HEGNN has involved high-degree representations, it does directly address the original problem given in Theorem 3.5 (3.6). The purpose of introducing Theorem 4.1 is to further demonstrate the enhanced expressivity of our HEGNN by using the inner products of full degrees. In other words, Theorem 3.5 (3.6), and Theorem 4.1 discuss different aspects of the benefits by our model. \n\n> **W4. The performance of MACE on the N-body dataset and the MD-17 dataset should be provided.**\n\nThanks for your reminder. We have additionally tested MACE on the N-body dataset. Due to the expensive running cost, we reduce the channel number of MACE from 64 to 8 (but out of memory still occurs in the 100-body case). Please note that MACE also resorts to CG tensor products similar to TFN, and it is thus consistent that its performance is worse than our model.  \n\n**Table S4:** Results of MACE and HEGNN on N-body dataset\n| N-body ($\\times 10^{-2}$) | 5-body   | 20-body  | 50-body  | 100-body |\n| :--------------------------- | :------- | :------- | :------- | :------- |\n| MACE                         | 1.13     | 1.60     | 2.41     | 3.38     |\n| HEGNN$\\_{l\\leq1}$| 0.52     | 0.79     | 0.88     | 1.13     |\n| HEGNN$\\_{l\\leq2}$| **0.47** | **0.78** | 0.90     | 0.97     |\n| HEGNN$\\_{l\\leq3}$| 0.48     | 0.80     | **0.84** | 0.94     |\n| HEGNN$\\_{l\\leq6}$| 0.69     | 0.86     | 0.96     | **0.86** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "24ImIIXVy8", "reviewer": "Reviewer_vhjv", "summary": "This paper challenges the prevailing notion that high-degree steerable representations are unnecessary in equivariant Graph Neural Networks (GNNs). The authors provide theoretical analysis showing that equivariant GNNs constrained to 1st-degree representations degenerate to zero functions when applied to symmetric structures like k-fold rotations and regular polyhedra. To address this limitation, they propose HEGNN, a high-degree extension of the EGNN model that incorporates higher-degree steerable vectors while maintaining efficiency through a scalarization technique. The authors evaluate HEGNN on symmetric toy datasets, N-body systems, and MD17 molecular dynamics datasets, demonstrating improved performance over existing models.", "questions": "1. Can the authors conduct additional experiments with different tasks or application domains to further support the claims presented in the paper? This could help demonstrate the broader applicability and necessity of high-degree representations in equivariant GNNs.\n2. How does HEGNN compare to more recent state-of-the-art methods on standardized N-body system benchmarks, such as those used in the ClofNet paper and subsequent works?\n3. Can the authors provide a more detailed analysis of the computational efficiency of HEGNN compared to other high-degree models and EGNN? This could include training times, memory usage, and scaling behavior with respect to the number of particles and degree of representations.\n4. Have the authors explored the performance of HEGNN on tasks beyond position prediction, such as force field prediction or other molecular property predictions? This could help strengthen the argument for the necessity of high-degree representations.", "limitations": "The authors acknowledge that their current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. It remains to be verified whether HEGNN is effective on large-scale geometric graph datasets.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a clear and well-motivated research question, challenging an existing assumption in the field of equivariant GNNs.\n2. The theoretical analysis is thorough and provides valuable insights into the limitations of low-degree representations for symmetric structures.\n3. The proposed HEGNN model offers a promising approach to incorporating high-degree representations while maintaining computational efficiency.\n4. The experimental results on symmetric toy datasets align well with the theoretical predictions, providing empirical support for the main claims.\n5. The paper includes a good discussion of the trade-offs between expressivity and efficiency in equivariant GNN models.", "weaknesses": "1. The experimental comparisons on the N-body system are limited and use different splits and variations compared to existing literature. This makes it difficult to directly compare HEGNN's performance to state-of-the-art methods. Including comparisons with more recent baselines (e.g., ClofNet, GCPNet, SaVeNet) and using standardized benchmarks would strengthen the empirical evaluation.\n2. The experiments are primarily focused on predicting future positions of particles/atoms. While this is a relevant task, it may not fully demonstrate the necessity or advantages of high-degree representations across a wider range of equivariant GNN applications. Additional experiments on different tasks or domains could provide more comprehensive evidence to support the paper's claims.\n3. Some relevant baselines are missing from the comparisons. For example, GMN-L is not included in the MD17 dataset experiments, despite outperforming the proposed method on several targets. Similarly, GMN is missing from the N-body systems experiment. Including these baselines would provide a more complete picture of HEGNN's performance relative to existing methods.\n4. The paper could benefit from a more extensive discussion of the computational trade-offs involved in using high-degree representations. While the authors mention that HEGNN is more efficient than traditional high-degree models, a more detailed analysis of the computational costs and scaling behavior would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!  We provide the following responses to your concerns:\n\n> **W1 & Q2: HEGNN should be tested on conventional dataset splits and compared with new baselines such as ClofNet.**\n\nNice suggestion! We have additionally conducted experiments on standard N-body benchmarks (train/valid/test = 3k/2k/2k), and compared our method with more state-of-the-art methods including ClofNet as well as its variant ClofNet-vel [a], MACE [b] and SEGNN [c], thanks to the availability of their open-source code. The results on standard split and our original protocol are reported in Table S5. It is observed that our HEGNN clearly outperforms existing methods in both cases, indicating the general effectiveness of our model. \n\n> **W2 & Q1 & Q4: Is it possible to test the performance of HEGNN in different prediction targets and different application fields to further illustrate its versatility?**\n\nThanks for your valuable suggestion. We choose the prediction of future atoms as our task, as it is equivariant, indicating that the input and output spaces share the same coordinate system. With this task, we are capable of evaluating if the output of the model can retain the full geometry including orientation information of the input after multi-layer message passing. We understand that additional experiments on different tasks or domains (e.g. force field prediction or molecular property prediction) are helpful, which, yet, are not the main focus of this paper and are better left for future exploration. \n\n> **W3: It is necessary to supplement baselines such as GMN and GMN-L.**\n\nThanks for the reminder. As suggested, we have further included the results of GMN-L in the experiments on MD17 in Table S6, and added the comparisons with GMN in the N-body experiment in Table S5. Our HEGNN-6 generally outperforms GMN, and achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance. These results will be added to the revised paper.\n\n**Table S5:** Results of N-body dataset under two partitions\n|N-body($\\times10^{-2}$)|5-body|20-body|50-body|100-body|\n|-|-|-|-|-|\n|train/valid/test=3k/2k/2k|||||\n|EGNN|0.71|1.08|1.16|1.29|\n|ClofNet|0.89|1.79|2.40|2.94|\n|ClofNet-vel|0.84|1.50|2.28|2.67|\n|GMN|0.67|1.21|1.18|2.55|\n|MACE|1.43|1.93|2.20|2.51|\n|SEGNN|1.81|2.67|3.44|Nan|\n|HEGNN$\\_{l\\leq1}$|0.64|**0.84**|0.92|1.04|\n|HEGNN$\\_{l\\leq2}$|0.69|0.89|1.13|**0.94**|\n|HEGNN$\\_{l\\leq3}$|**0.58**|1.04|**0.92**|1.04|\n|HEGNN$\\_{l\\leq6}$|0.77|1.06|1.02|1.18|\n|train/valid/test=5k/2k/2k|||||\n|GMN|0.52|0.98|1.04|1.21|\n|ClofNet|0.80|1.49|2.28|2.77|\n|ClofNet-vel|0.78|1.45|2.22|2.62|\n|MACE|1.13|1.60|2.41|3.38|\n|SEGNN|1.68|2.63|3.30|Nan|\n|HEGNN$\\_{l\\leq1}$|0.52|0.79|0.88|1.13|\n|HEGNN$\\_{l\\leq2}$|**0.47**|**0.78**|0.90|0.97|\n|HEGNN$\\_{l\\leq3}$|0.48|0.80|**0.84**|0.94|\n|HEGNN$\\_{l\\leq6}$|0.69|0.86|0.96|**0.86**|\n\n**Table S6:** Results on MD-17 with GMN & GMN-L\n|MD-17|Aspirin|Benzene|Ethanol|Malonaldehyde|Naphthalene|Salicylic|Toluene|Uracil|\n|-|-|-|-|-|-|-|-|-|\n|GMN|10.14±0.03|**48.12±0.40**|4.83±0.01|13.11±0.03|0.40±0.01|0.91±0.01|**10.22±0.08**|0.59±0.01|\n|GMN-L|**9.76±0.11**|54.17±0.69|4.63±0.01|**12.82±0.03**|0.41±0.01|**0.88±0.01**|10.45±0.04|0.59±0.01|\n|HEGNN$\\_{l\\leq1}$|10.32±0.58|62.53±7.62|4.63±0.01|12.85±0.01|0.38±0.01|0.90±0.05|10.56±0.10|0.56±0.02|\n|HEGNN$\\_{l\\leq2}$|10.04±0.45|61.8±5.92|4.63±0.01|12.85±0.01|0.39±0.01|0.91±0.06|10.56±0.05|0.55±0.01|\n|HEGNN$\\_{l\\leq3}$|10.20±0.23|62.82±4.25|4.63±0.01|12.85±0.02|**0.37±0.01**|0.94±0.10|10.55±0.16|**0.52±0.01**|\n|HEGNN$\\_{l\\leq6}$|9.94±0.07|59.93±5.21|**4.62±0.01**|12.85±0.01|**0.37±0.02**|**0.88±0.02**|10.56±0.33|0.54±0.01|\n\n\n> **W4 & Q3: Comparison of various indicators at runtime between HEGNN and other high-degree steerable models.**\n\nThanks! The main difference between our HEGNN and traditional high-degree models is that we employ inner products for message exchange between the representations of different degrees, while TFN resorts to CG tensor products to consider all possible interactions between different representations. By denoting the maximum degree as $L$, the complexity of our inner products is equal to $\\textstyle\\sum_{l=0}^L(2l+1)=(L+1)^2=O(L^2)$, whereas the complexity of CG tensor products is derived as $O(L^6)$ by [a]. The following tables further report the inference times and model sizes of the high-degree models in the 100-body case of the N-body dataset. It is verified that our HEGNN obtains better performance with lower computation cost, compared to TFN, SEGNN, and MACE. \n\n**Table S7:** Parameters and inference times of models\n|**Parameters of Models**|$l\\leq1$|$l\\leq2$|$l\\leq3$|$l\\leq6$|\n|------------------------|-----------|-----------|-----------|-----------|\n|EGNN|134.1k|--|--|--|\n|HEGNN|160.3k|160.9k|161.5k|163.2k|\n|TFN|--|9.6M|19.5M|86.6M|\n|SEGNN|228.1k|244.1k|254.9k|288.1k|\n|MACE|14.8M|38.0M|77.0M|342.5M|\n|**Inference Times ($10^{-2} s$) of Models**|$l\\leq1$|$l\\leq2$|$l\\leq3$|$l\\leq6$|\n|EGNN|0.57|--|--|--|\n|HEGNN|0.82|0.88|0.91|1.08|\n|TFN|--|3.75|2.66|OOM|\n|SEGNN|1.33|1.70|1.98|22.33|\n|MACE|22.10|125.87|261.11|OOM|\n\n[a] Passaro S, Zitnick C L. Reducing SO (3) convolutions to SO (2) for efficient equivariant GNNs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5aUgUHqNT", "reviewer": "Reviewer_3KcU", "summary": "The present work pertains to analysing and improving Graph Neural Networks operating on geometric graphs that possess E(3) (Euclidean) symmetries, i.e. the tasks of interest are invariant/equivariant to rotations, translations and reflections. The paper challenges the assumption that it is sufficient to use E(3)-equivariant features of 1st degree (the term “degree” is related to the dimensionality of the group representation to which the features are equivariant - 1st-degree features are 3-dimensional and are equivariant to 3D rotations/reflections), as done in EGNN, Satorras et al., ICML’21, which works quite well in practice and is computationally efficient. The authors construct a series of counterexample symmetric geometric graphs (i.e. graphs which, when rotated by certain angles, remain intact up to permutations of their vertices) which are shown to be challenging for geometric GNNs. It is shown that, for certain degrees $l$, a GNN can only produce output features equal to zero, therefore showing that the degree is crucial for expressivity and that $l=1$ is insufficient for some cases. To address this limitation, they combine ideas from EGNN and TFN, Thomas et al., arxiv’18: They create a GNN, similar to EGNN, enriched with auxiliary features of higher degree, using spherical harmonics (similar to TFN). To keep the computational complexity low, they produce the messages used for the updates of node features/positions/auxiliary features via scalarisation using inner products, simplifying the messages used in TFN. The expressivity of the method is theoretically analysed, showing the ability to recover angles, and empirically tested, showing improved performance in several synthetic and real-world benchmarks.", "questions": "- Section 3.1/Section 4: It is unclear to me why the authors introduced the modulation of the spherical harmonics (re-scaling). How does this affect expressivity?\n- Section 3.1: Is the representation of O(3) – Eq. (2) – conventional or introduced here for the first time? In the first case, could the authors provide a reference? In the second case, it would be more appropriate to precisely show that this is indeed a representation for completeness.\n- As far as I understand, Eq. (3) refers to graph-level features. Perhaps this should be explicitly mentioned to improve readability.\n- What information is lost from the scalarization trick of Eq. (6) compared to CG tensor products? To what extent does this limit expressivity?\n- How do the authors explain that in certain cases in Table 2, HEGNN does not achieve 100% performance (although it should according to the theory)? Additionally, why is the std so high (15% and 30%)?\n- In table 3, it seems that HEGNN with $l\\leq 1$ always outperforms EGNN, which is also 1st-degree. How do the authors explain this?\n- It is unclear to me how equivariance to reflections is handled by this method. How does the discussion at the end of section 3.1 affects the operators used?\n- Table 5 is quite helpful, perhaps consider moving it to the main text. \n\n**Minor**: I have the impression that there were a few typos throughout the text. I suggest performing a proof-reading (examples: L78 typo: university --> universality, L266 type: donated --> denoted).", "limitations": "I recommend that the authors devote a separate section to discuss the limitations of their work. Currently, I could locate only a limited discussion in the appendix with regards to not testing in large-scale systems.\n\nI do not foresee any negative societal impact.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Significance**. The studied problem is important for various physical sciences using data-driven methods and finding better trade-offs between performance and computational complexity is key to their progress. The present paper improves this trade-off both theoretically and empirically. \n\n**Novelty**.\n  - *Theoretical results*. The underlying theory illustrating the expressivity barriers of low-degree representations (theorems 3.4-3.6) is simple and concise, yet insightful and of independent technical interest (can be used for other types of symmetric objects to find out properties of neural representations and illustrate limitations of modelling choices). Additionally, theorem 4.1. backs the proposed method with theoretical arguments, hinting that is probably a good trade-off between expressivity and computational complexity (although the former is not completely characterised).\n   - *Method*. As far as I know, the proposed approach to incorporate higher-degree representations into message-passing is original. Additionally, it is easy to understand and implement, which may facilitate reproducibility.\n\n**Presentation**. Apart from a few exceptions (see weaknesses), the paper is well-explained, with carefully chosen notation and progressive illustration of the different results (from the theoretical motivation to the model improvement and empirical evaluation).\n\n**Empirical results & Execution**.  The experimental outcomes are in agreement with the theory on synthetic data, while the method is shown to generalise better than its competitors in real-world data, with additional computational efficiency advantages. Overall, this work is well-executed: starting from a characterisation of the limitations of existing works, then proposing a remedy (that gets the best of both worlds from two different families of methods) and finally testing it both on the counterexamples, as well as in real-world scenarios, validating the theory.", "weaknesses": "**Clarity: limits audience and creates some issues with contextualization to related work**. Although not a major weakness, there are some issues with the clarity of the presentation, which I think should be addressed to improve the exposition of the paper to a wider audience. \n  - In the introduction and the related work section, the authors mention several concepts without providing explanations which might make the text hard to follow, especially for readers not experts in geometric GNNs and Euclidean symmetries. \n    - For instance: *x-degree steerable representations* - this is a notion at the heart of the paper, so it should probably be intuitively explained early on, e.g. by summarising the formal definitions at the end of section 3.1.\n    - *Clebsch-Gordan tensor product* - this concept is not explained in the paper. However, I do not think that the authors should assume familiarity with it. The reader needs to obtain a clear understanding due to the following: the main methodological innovation of the present work is to replace this operation with inner products to improve computational efficiency. I think that the authors should explain in a separate subsection the differences between these two operators and discuss what they sacrifice by not incorporating CG products in their architecture.\n-\tExamples 3.2. and 3.3. need further elaboration as they are the main motivation behind the proposed methodology. E.g. why do even folds are not symmetric w.r.t. the cyclic group? Perhaps explain what the dihedral group is. Why are regular polygons symmetric w.r.t. the dihedral group?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!  We provide the following responses to your concerns:\n\n> **W1: More detailed background knowledge and conceptual explanations are needed to improve clarity.**\n\nThank you for your valuable suggestions! We will further summarize the formal definitions of steerable representations at the end of Section 3.1, and introduce the concept of CG tensor product in Section 3.1 and compare its difference with Eq.(6) in Section 4.  Furhter eblations will be added in Examples 3.2 and 3.3. We sincerely thank you once again, and will further enhance the clarity of our paper by providing more explanations related to geometric GNNs, Euclidean symmetries, and Examples 3.2 and 3.3.\n\n\n\n> **Q1: The significance of re-scaling on spherical harmonics.**\n\nThanks for your comments! The re-scaling part corresponds to the radial function, which is widely used in previous papers such as TFN. Here, for convenience, we combine the radial function into the formulation of the spherical harmonics to improve the readability of our paper. \n\n\n\n> **Q2: Origin of O(3) group representation.**\n\nNice suggestion! This representation method is conventional and can be found in many physics-related literature, such as [a,b]. More general and theoretical content of representation theory can be found in some pure mathematical books, such as [c,d]. We will add the necessary citations around Eq.(2) in the revised version.\n\n[a] Landau, L. D., and E. M. Lifshitz. Quantum Mechanics: Non-Relativistic Theory. \n\n[b] Chen, J. Q., Jialun Ping, and Fan Wang. Group Representation Theory for Physicists. \n\n[c] Weyl, Hermann. The Classical Groups: Their Invariants and Representations. \n\n[d] Fulton, William, and Joe Harris. Representation Theory. \n\n\n\n> **Q3: It is helpful to explicitly point out that Eq. (3) refers to graph-level features.**\n\nThank you for raising this point, which is very important for improving the readability of our paper. We will clearly mention this in subsequent revisions.\n\n\n\n> **Q4: What information is lost in the scalarization trick compared to the CG tensor product?**\n\nThanks for this valuable question! Our HEGNN exclusively passes invariant quantities (inner products of high-degree representations) between features $\\tilde v^{(l)}$ of different degrees $l$, unlike the expensive CG tensor product used in TFN, which considers all possible interactions between different frequencies. Our model can be seen as a generalization of the scalarization trick in EGNN to high-degree representations. While the scalarization trick might somehow sacrifice model expressivity in theory, it has shown significantly better efficacy and efficiency in practice compared to conventional high-degree models, as demonstrated by EGNN paper and also our experiments here. Additionally, Theorem 4.1 indicates that passing inner products of full degrees is sufficient to recover the information of all angles between each pair of edges, affirming the theoretical expressivity of our HEGNN in characterizing the geometry of the input structure. As suggested, we will add a new paragraph for the comparision between Eq.(6) and CG tensor products in Section 4. \n\n\n\n> **Q5: Explanation of the results in Table 2.**\n\nThank you for your comments. The experiment setup in Table 2 is designed from the GWL paper [a]. It requries to first embed the input graphs through an equivariant neural network (e.g. EGNN, HEGNN, TFN), and then classify them through a simple classifier.  As for the phenomenon that HEGNN does not achieve 100% in some cases, it is probably because the classifier is not trained well. We observe that accuracy can be improved to 100%, if increasing the number of training epochs to a sufficiently large value (e.g. 2,000).\n\nThe high std is due to the settings used in the GWL paper. There are only 10 test groups in total, and each group is to classify two graphs into two classes. The classification accuracy can only be 0%, 50%, and 100% for each group, which might exlain why the std will be relatively large.\n\n[a] Joshi C K, Bodnar C, Mathis S V, et al. On the expressive power of geometric graph neural networks.\n\n\n> **Q6: The reason for the  performance gap between EGNN and HEGNN$\\_{l\\leq 1}$.**\n\nVery insightful comment! We speculate that why HEGNN with $l\\leq 1 $ outperforms EGNN is mainly owing to HEGNN initialization (Eq.(5)), which is equivalent to an additional massage passing layer, and increases the depth of the neural network. To show this, we additionally test the effects of EGNN-4layer, HEGNN-3layer, and HEGNN-4layer on the N-body dataset, and the results are as follows. All HEGNNs only used 1-st  steerable feature. It can be found that the performance of EGNN-4layer is very close to HEGNN-3layer.\n\n**Table S8:** Comparison between EGNN and HEGNN on N-body\n|N-body ($\\times10^{-2}$)|5-body|20-body|50-body|100-body|\n|-|------|-|-|-|\n|EGNN-4layer|0.65|1.01|1.00|1.36|\n|HEGNN$\\_{l\\leq1}$-3layer|0.63|0.98|0.96|1.31|\n|HEGNN$\\_{l\\leq1}$-4layer|0.52|0.79|0.88|1.13|\n\n\n> **Q7: How to achieve reflection equivariance with HEGNN?**\n\nThank you for this valuable question. Since the steerable features are initialized by spherical harmonics in Eq.(5), reflection equivariance is associated with the transformation $(-1)^lD^{{l}}(\\mathfrak r)$, where $D^{{l}}(\\mathfrak r)$ is the rotation representation. In other words, the sign of the steerable feature changes if $l$ is odd, and keeps unchanged if $l$ is even.  Besides, during the message passing in our model, all steerable features only interact with the scalar (i.e. the inner product), without changing their original equivariance. Hence, reflection equivariance is always represented by $(-1)^lD^{{l}}(\\mathfrak r)$ throughout our model. \n\n\n> **Q8: It is suggested to move Table. 5 to the main text.**\n\nThank you for your suggestion. We will include it in the main text in the revised version.\n\n\n> **Minor: Some typos.**\n\nThank you very much. We will proof-read our paper again and have the typos fixed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We sincerely thank all reviewers and ACs for their time and efforts on reviewing the paper. We are very glad that the reviewers recognized the problems we studied, the theories we proposed, and the models we built, and their comments really gave us a lot of inspiration and enlightenment.

For the symmetric structure problem we studied and the theory we proposed, the reviewers thought it was well motivated (Reviewer op9c, vhjv), the theoretical analysis was correct (Reviewer 7wT3), thorough (Reviewer vhjv), insightful and of independent technical interest (Reviewer 3KcU). And our proposed HEGNN, a new method of introducing steerable features through scalarization technique, was evaluated as original (Reviewer 7wT3, 3KcU), promising (Reviewer vhjv), enable the use of higher order features with reduced computational complexity (All Reviewers). The whole article is consistent with the theory and experimental results (Reviewer 7wT3, vhjv), and clear and easy to understand (Reviewer 7wT3, 3KcU).

We also appreciate the reviewers for the insightful comments. To address their concerns, we have added additional experiments as follows.

**Table S1** shows the results of FastEGNN, SEGNN and HEGNN on the N-body dataset with different numbers of particles.

**Table S2****/S6** shows the experimental results of HEGNN, GMN & GMN-L on the MD-17 dataset.

**Table S3** compares HEGNN with EGNN on symmetric graphs when adding noise perturbations, in order to anlalyse how the symmetry-breaking factors influence the performance.

**Table S4** shows the results of HEGNN and MACE on the N-body dataset with different numbers of particles.

**Table S5** compares HEGNN with more relevant baselines under two different protocols on the N-body dataset.

**Table S7** reports the inference times and model sizes of high-degree models.

**Table S8** compares the performance between 4-layer EGNN, 3-layer HEGNN$\_{l\leq 1}$, and 4-layer HEGNN$\_{l\leq 1}$ on the N-body dataset, to explain why  HEGNN$\_{l\leq 1}$ is better than EGNN.

Meta-review: Based on the positive reviews and constructive discussion, I recommend accepting this paper. The work presents novel theoretical insights into equivariant Graph Neural Networks and proposes an innovative model (HEGNN) that balances expressivity and efficiency. Experimental results support the theoretical claims and show improvements over existing methods. While minor revisions were suggested, the reviewers agreed on the paper's technical soundness and valuable contribution to the field.
Decision: Accept (poster)