Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?

Authors: jiacheng  cen
anyi  li
ning  lin
yuxiang  ren
zihe  wang
wenbing  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:22:55

Abstract: Equivariant Graph Neural Networks (GNNs) that incorporate E(3) symmetry have achieved significant success in various scientific applications. As one of the most successful models, EGNN leverages a simple scalarization technique to perform equivariant message passing over only Cartesian vectors (i.e., 1st-degree steerable vectors), enjoying greater efficiency and efficacy compared to equivariant GNNs using higher-degree steerable vectors. This success suggests that higher-degree representations might be unnecessary. In this paper, we disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric structures, including $k$-fold rotations and regular polyhedra. We theoretically demonstrate that equivariant GNNs will always degenerate to a zero function if the degree of the output representations is fixed to 1 or other specific values. Based on this theoretical insight, we propose HEGNN, a high-degree version of EGNN to increase the expressivity by incorporating high-degree steerable vectors while maintaining EGNN's efficiency through the scalarization trick. Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric structures, but also shows substantial improvements on more complicated datasets such as $N$-body and MD17. Our theoretical findings and empirical results potentially open up new possibilities for the research of equivariant GNNs.

Main paper: 
 introduction: 
 Molecules, proteins, crystals, and many other scientific data can be effectively modeled and represented through geometric graphs [2] [3] [4] [5] [6] [7] [8] . This type of data structure encapsulates not only node characteristics and edge information but also a 3D vector (such as position, velocity, etc.) for each node. To process geometric graphs, equivariant Graph Neural Networks (GNNs) have been developed, which undergo equivariant message passing over nodes, conforming to the E(3) or SE(3) symmetry of physical laws. These models have achieved remarkable successes in a lot of scientific tasks, such as physical dynamics simulation [9] [10] [11] , molecular generation [12] [13] [14] [15] and protein design [16] [17] [18] .Pioneer equivariant GNNs [19] [20] [21] [22] derive high-degree steerable representations beyond scalars and 3D coordinates with the help of spherical harmonics and conduct equivariant message passing between representations of different degrees through the Clebsch-Gordan (CG) tensor product. While these high-degree models are able to approximate any function of fully connected geometric graphs in theory [23] , they usually suffer from expensive computational costs in practice. In contrast, k-fold (odd) k-fold (even)Tetrahedron Cube (Hexahedron) Octahedron Dodecahedron Icosahedron 1Figure 1 : Common symmetric graphs. Equivariant GNNs on symmetric graphs will degenerate to a zero function if the degree of their representations is fixed as 1.EGNN [1] leverages a simple scalarization technique to allow equivariant message passing over only 3D vectors (i.e. the 1st-degree steerable features). Specifically, the scalarization technique first encodes 3D vectors into scalars as invariant messages, which are passed as geometric messages after the multiplication with the 3D vectors to recover the orientation information. Despite its simplicity, EGNN achieves remarkably better efficacy and efficiency against conventional high-degree models for a broad range of applications [24; 25] . Such successes suggest that higher-degree representations might be unnecessary.In this paper, we challenge and disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric graphs. Fig. 1 illustrates the examples of k-fold rotations and regular polyhedra, which are invariant to rotations up to certain rotating angles. Taking the cube for example, conducting 90 • rotation around the axes crossing the center of the two opposite faces keeps its shape and orientation unchanged. Interestingly, by making use of group theory, we theoretically prove that any equivariant GNN (after translating the coordinate center to the origin and conducting graphlevel readout) on these symmetric graphs will degenerate to a zero function if the degree of their representations is fixed to be 1. The direct deduction of this theorem is that EGNN can only output a zero 3D vector no matter how we rotate the input graph, indicating that EGNN totally loses the recognition ability of orientation. Additionally, this statement points out the limitation of the methods that rely on constructing global features for symmetric graphs2 (e.g. frames in frame averaging [26; 27] , virtual nodes in FastEGNN [28] ), equivariant pooling in EGHN [29] , and meshes in Neural P3 M [30] ), since it is impossible to output another non-collinear 3D vector except the center coordinate.Based on the above theoretical insights, we propose a novel equivariant GNN model termed HEGNN 3 , which enhances EGNN by incorporating high-degree steerable vectors while inheriting the desired benefit from EGNN through the scalarization trick. In summary, our contributions are as follows:• We theoretically investigate the expressivity reduction issue of equivariant GNNs on symmetric graphs.• We propose HEGNN, to further incorporate high-degree steerable representations into EGNN. Moreover, since the equivariant message passing process between different-degree representations is conducted via inner products, it shares the same benefit as EGNN, compared to traditional high-degree models.• Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric graphs, but also shows substantial improvements on more complicated datasets without explicit symmetry, such as N -body and MD17.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract and introduction nicely summarize the theoretical contributions mentioned in § 3, the model design mentioned in § 4, and the experimental results mentioned in § 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Equivariant GNNs. Equivariant GNNs can be divided into two classes: scalarization-based models and high-degree steerable models [31] . Scalarization-based models adopt norms or inner products to convert equivariant 3D vectors into invariant scalars, which are considered as coefficients to linearly combine 3D vectors for node update. EGNN [1] is the first work falling into this category. Concurrently, PAINN [32] further enhances the expressive ability of the model by introducing multi-channel equivariant features. On the contrary, high-degree steerable models (e.g. TFN [19] ,SEGNN [33] and SE(3)-Transformer) use spherical harmonics to ensure the equivariance of message passing, and realize interaction between steerable features of different degrees through CG tensor products. Our HEGNN also uses high-degree steerable features, but it leverages the scalarization trick for the interaction between steerable features of different degrees, thus leading to more expressivity than EGNN and less computational cost than other high-degree models.Expressivity of Equivariant GNNs. The theoretical expressivity of equivariant GNNs is initially explored by [23] , which proves the university of the high-degree steerable model, i.e., TFN [19] , over fully-connected geometric graphs. GemNet [34] further demonstrates that the universality holds with just spherical representations other than the full SO(3) representations that are required in the proof of [23] . More recently, the GWL framework [35] extends the Weisfeiler-Lehman (WL) test into a geometric version [36] to study the expressive power of geometric GNNs operating on sparse graphs from the perspective of discriminating geometric graphs. Different from all the above works, our paper investigates the expressivity of equivariant GNNs on symmetric graphs, and demonstrates the necessity of involving high-degree representations. Although the GWL test paper [35] has experimentally compared different models on k-fold structures that are allowed to rotate only in the 2D space, the conclusions of this paper are proved both theoretically and experimentally. Moreover, our discussions cover a full range of examples including k-folds (rotation in 3D space) and regular polyhedra.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Our current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. Whether our HEGNN is effective on large-scale geometric graph data sets remains to be verified.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: We use NVIDIA A100-80G, which has written in § 5.2. 
theoretical analyses: 
 In this section, we first present the necessary preliminaries related to geometric graphs and group representation. Then, we define and illustrate typical examples of symmetric graphs. Finally, we will discuss when equivariant GNNs will degenerate to a zero function on symmetric graphs.Geometric graph. A geometric graph of N nodes is defined as G := H, ⃗ X; A , where H :={h i ∈ R C H } N i=1 and ⃗ X := {⃗ x i ∈ R 3 } N i=1are node features and 3D coordinates, respectively; A ∈ R N ×N represents the adjacency matrix and can be assigned with edge features e ij if necessary. Throughout our theoretical analyses in this section, we assume the node features and edge features to be identical for all.Transformation of geometric graph. We are interested in the transformations of a geometric graph G with respect to a group G, which is defined as g • G, for g ∈ G and • denoting the group action. For instance, g • G can be explained as translation, rotation, or reflection of the coordinates ⃗ X. These transformations form a 3D Euclidean group denoted as E(3), and its subgroup without translation is called the orthogonality group O(3). With the aid of group representation ρ(g), the transformation of a coordinate ⃗x is represented as ρ(g)⃗ x. For example, orthogonal matrices are the trivial representations of O(3), that is, the orthogonal transformation of a vector ⃗x is represented by O⃗ x with O ∈ R 3×3 being an orthogonal matrix. Besides, there are other representations of O(3), such as the irreducible representations which will be detailed below.Equivariance. Let X and Y be the input and output vector spaces, respectively. A functionEQUATIONwhere ρ X and ρ Y are the group representations in the input and output spaces, respectively. Since we can eliminate the translation effect by simply translating the center of all coordinates to the origin, we only discuss equivariance with respect to O(3) in this section. In other words, we default that the center of ⃗ X is at the origin.Irreducible representations and steerable features. O(3) consists of rotation and inversion, implying O(3) = SO(3) × C i , where SO(3) is the rotation group and C i = {e, i} denotes the inverse group. We first discuss the irreducible representations of SO(3). For each rotation r ∈ SO(3), its irreducible representations are Wigner-D matrices D (l) (r) ∈ R (2l+1)×(2l+1) of different degree l ∈ N [21; 37] . When l = 1, it becomes the common rotation matrix R r acting on the 3D coordinate space. Under the irreducible representations, the equivariant constraint in Eq. ( 1) turnsinto f (l) (R r ⃗ x) = D (l) (r)f (l) (⃗ x),if the output degree is l. According to [38] , spherical harmonicsY (l) = [Y (l) m (⃗ x)] lm=-l offer a unique and complete set of function bases satisfying the equivariant constraint. We further define a modulated spherical harmonics asf (l) (⃗ x) = φ(∥⃗ x∥) • Y (l) (⃗ x/∥⃗ x∥)by adding a continuous radial function φ : R + → R of vector norm ∥ • ∥ for re-scaling. Such a function f l and its output f l (⃗ x) are called type-l steerable function and steerable feature, respectively. We now deduce the irreducible representations from SO(3) to O(3). Note that spherical harmonics satisfy Y (l) (-⃗ x) = (-1) l Y (l) (⃗ x); in other words, they are inverse-equivariant when l is odd, but inverse-invariant when l is even. We thus specify the group representation of O(3) asEQUATIONwhere σ (l) (m) = 1 for m = e (the identity) and σ (l) (m) = (-1) l if m = i (the inverse). Readers can refer to the discussion in e3nn [39] with another representation method by using the concept of parity and construct this through methods such as Clebsch-Gordan (CG) tensor product [40] . For concision, the type-l steerable feature is denoted as ṽ(l) with a tilde notation.In § 1, we present that k-fold rotations and regular polyhedra exhibit certain symmetries. In this subsection, we formally describe them via the notion of the symmetric graph. Hereh • G = G is defined in the graph level. Particularly for the coordinates ⃗ X ∈ R 3×N , it implies that ∀O ∈ H, ∃P ∈ S N , O ⃗ X = ⃗XP and P A = AP , where S N is the permutation group of order N . Essentially, rotating the coordinates of a symmetric graph leads to a copy of this graph up to a different permutation of the nodes.Without considering inversion, the finite subgroups of SO(3) are only cyclic group C n , dihedral group D n , tetrahedral group T , octahedral group O, and Icosahedral group I [41] . We provide several examples of symmetric graphs as follows.Example 3.2 (k-folds). On the one hand, for a geometric graph G corresponding to a 2k-fold with nodes {(cos(i • π/k), sin(i • π/k), 0)} 2k-1 i=0 , the inverse group C i and the dihedral group D 2k (rotation around z-axis with angle π/k, and reflection around the axis connecting the midpoints of opposite sides or the axis connecting opposite vertices), are symmetric groups on G, namely, C i , D 2k ∈ H(G). On the other hand, for a geometric graph G corresponding to a (2k + 1)-fold with nodes {(cos(i • 2π/(2k + 1), sin(i • 2π/(2k + 1)), 0)} 2k i=0 , H(G) includes the dihedral group D 2k+1 but without the inverse group C i .Example 3.3 (Regular Polygons). The symmetric groups of regular polygons in the plane and regular prisms in space include the dihedral group D n . Regular tetrahedra are symmetric with respect to three rotation axes of the second order and four axes of the third order, corresponding to 12 group elements. Regular hexahedra (cubes) and the regular octahedra (which are dual to each other and share the same symmetric groups) are symmetric about six axes of the second order, four axes of the third order, and three axes of the fourth order, corresponding to 24 group elements. Regular dodecahedra and regular icosahedra (which are also dual to each other) are symmetric about six axes of the fifth order, ten axes of the third order, and fifteen axes of the second order, corresponding to 60 group elements. Additionally, except tetrahedra, all other four regular polygons are central symmetric, indicating that C i is their common symmetric group.We now demonstrate that equivariant GNNs on symmetric graphs will encounter the issue of expressivity degeneration. Here, we assume that the graph functions we explore are invariant to the permutation of the nodes. This fits the case when we add a readout layer to all nodes globally or just focus on the message passing process for each node individually.We first derive a crucial theorem that greatly facilitates our analyses. Theorem 3.4. Suppose that f (l) is an O(3)-equivariant function on geometric graphs, regarding the group representation ρ (l) defined in Eq. (2). Then, for any symmetric graph G induced by the group H ≤ O(3), namely, ∀G ∈ G(H), we always haveEQUATIONHere we have defined group average as ρ (l) (H) := 1 |H| h∈H ρ (l) (h).Eq. ( 3) is interesting and it shows that the function f (l) is symmetric with respect to the group average ρ (l) (H). More importantly, it indicates an linear equation I 2l+1 -ρ (l) (H) f (l) (G) = 0, where 2l+1) is the identity matrix. We can immediately attain the following conclusion.I 2l+1 ∈ R (2l+1)×(Theorem 3.5. If and only if the matrixI 2l+1 -ρ (l) (H) is non-singular, the O(3)-equivariant function f (l)is always a zero function on G, namely,EQUATIONA more general version of Theorem 3.5 is that the output space of f (l) corresponds to the null space of the matrix I 2l+1 -ρ (l) (H), indicating that dim(f (l) ) = (2l + 1) -rank I 2l+1 -ρ (l) (H) . Therefore, even the function f (l) ) will not exactly reduce to a zero function when I 2l+1 -ρ (l) (H) is singular, its output space is still limited to a subspace and suffers from diminished expressivity owing to the symmetry of the input geometric graph.In practice, it is difficult to determine if the matrix I 2l+1 -ρ (l) (H) is singular. This determination becomes easier if we can show that the group average ρ (l) (H) is equal to the zero matrix. Fortunately, we have the following property.Theorem 3.6. For a finite group H with its representation ρ (l) , ρ (l) (H) is a zero matrix (i.e., ρ (l) (H) = 0) if and only if tr(ρ (l) (H)) = 0. In this case, f (l) (G) ≡ 0, ∀G ∈ G(H).According to Theorem 3.6, we calculate the trace of the group average for each symmetric graph of interest and check if the trace is equal to zero. We summarize the conclusions for k-fold structures and regular polyhedra in Table 1 . We find that when l = 1, f (1) ≡ 0 for all cases. In addition, the function degenerates when l is odd, if the symmetric graph is induced by the inverse group C i . We defer more details of the calculations in the Appendix. Compared to the conclusions drawn by the GWL paper [35] which only experimentally discusses the k-fold structures under 2D rotations, here we apply rigorous theoretical derivations to analyze more cases besides k-folds, regarding more symmetric subgroups of O(3).Table 1 : Expressivity degeneration of equivariant GNNs on symmetric graphs.EQUATIONThe Proposed HEGNNThe analyses in the last section imply the necessity of involving the representations with more and higher degrees in equivariant GNNs. Therefore, we propose HEGNN by further conducting the update of high-degree steerable features upon EGNN [1] . As illustrated in Fig. 2 , HEGNN is composed of the three key components: initialization of high-degree steerable features, calculation of cross-degree invariant messages, and aggregation of neighbor messages, the latter two of which are conducted over multiple layers. We depict each component separately in what follows.Initialization of high-degree steerable features. Given a geometric graph G H, ⃗ X; A where each node contains only type-0 feature h i and type-1 feature ⃗x i , we first obtain the initialization The different architectures of our HEGNN, EGNN [1] and TFN [19] . HEGNN exploits the scalarization trick inspired by EGNN to enable steerable features to interact between different degrees, avoiding the high computational cost of using CG tensor products in TFN.EQUATIONof high-degree steerable features {ṽ(l)i } L l=0 by using spherical harmonics on normalized relative coordinates. In detail, we aggregate spherical harmonics from all neighbors asEQUATIONwhereEQUATIONij is an invariant scalar with φ m,init being an arbitrary MultiLayer Perceptron (MLP), and N (i) denotes the neighbors of i4 .x i -⃗ x j (the usage of relative coordinates is for translation invariance) into an invariant scalar via the vector norm, which will be used to compute invariant message for both node features and coordinates. We generalize this scalarization trick to the case of high-degree steerable features. To be specific, we carry out the inner product between ṽ(l) i and ṽ(l)jfor each degree l individually, resulting in an invariant scalar z (l) ij . Then, we get the invariant message between node i and j, namely m ij after undergoing an MLP of all invariant quantities. The above processes are summarized as follows:EQUATIONwhere refers to concatenation. It should be noted that the form of SO3KRATES introduced in the concurrent work [42] is equivalent to the expression for z (l) ij in Eq. ( 6). Furthermore, our scalarization trick simplifies the formulation by bypassing the Clebsch-Gordan coefficients, making it more straightforward and easier to understand.Aggregation of neighbor messages. With the invariant message m ij at hand, we then update h i , ⃗x i , ṽ(l) i via message aggregation over all neighbors. We define ∆h i , ∆⃗ x i , ∆ṽi as the residues, which are calculated by:EQUATION∆ṽEQUATIONwhereEQUATIONṽ are different MLPs, and φ ⃗ x , φṽ both output a 1D scalar. Note that the application of Eq. ( 8) for all degrees can be compactly rewritten asL l=0 ∆ṽ (l) i = 1 |N (i)| j∈N (i) 1⊗ φṽ(mij ) cg L l=0 ṽ(l) i - ṽ(l) jin the form of CG tensor product with the weightsφ ṽ (m ij ) := L l=0 φ (l)ṽ . This form can be easily implemented using existing libraries such as e3nn.o3.FullyConnectedTensorProduct [39] . The resulting residues are used for the update:EQUATIONIn addition, we can augment the update of ⃗ x i with 1st-degree feature ṽ(1) i , leading to ⃗x i = ⃗ x i + ∆⃗ x i + ϕ (1) ṽ (h i )ṽ (1)i , which yet is not explored in our experiments for the sake of simplicity. The final output of h i and ⃗x i can be used for the node-level invariant prediction and equivariant prediction, respectively. We can also obtain a graph-level prediction by further adding a readout layer of all nodes.We now analyze the expressivity of HEGNN. Apparently, by including high-degree features, HEGNN is able to avoid the loss of expressive ability even on symmetric graphs. Moreover, when tackling general geometric graphs, HEGNN is capable of characterizing the complete angle information of the input graph, if its maximal degree L is sufficiently large. For concision and without losing the generality, we assume the steerable features ṽi are initialized with only spherical harmonics without the weights φ (l) ṽ,init in Eq. ( 5). Let ⃗x is = (⃗ x i -⃗ x s )/∥⃗ x i -⃗ x s ∥, the inner product z (l)ij can be expanded as followsEQUATIONwhere P (l) : R → R is Legendre polynomial of degree l, and Eq. ( 10) is based on the properties of spherical harmonics that⟨Y (l) (⃗ x), Y (l) (⃗ y)⟩ = 4π/(2l + 1) • P (l) (⟨⃗ x, ⃗ y⟩), ∥⃗ x∥ = ∥⃗ y∥ = 1.We have the following result. Theorem 4.1. For any geometric graph, there exists a bijection between the set of inner products {z(l) ij } |Aij |l=1 given by Eq. (10) and the set of edge angles A ij = {θ is,jt := arccos⟨⃗ x is , ⃗x jt ⟩} s∈N (i),t∈N (j) .Theorem 4.1 states that the inner products of full degrees can recover the information of all angles between each pair of edges, affirming the enhanced expressivity of our HEGNN. The proof is derived mainly based on the fact that Legendre polynomials are orthogonal polynomial bases which can injectivly represent the set A ij thanks to Newton's identities. The details are deferred to the appendix. Although the upper-bound of the degree in Theorem 4.1 grows rapidly with the graph size, it will be shown in our experiments that HEGNN with only L ≤ 6 is sufficient to outperform traditional models like EGNN [1] and TFN [19] in practice.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
expressivity on symmetric graphs: 
 Design of experiments: To experimentally verify the conclusion we proved above, we design a more comprehensive experiment based on code5 in [35] . This experiment uses four k-fold structures (k ∈ {2, 3, 5, 10}) and five convex regular polyhedra shown in Fig. 1 as test objects, and the center of each is at the origin. In detail, an arbitrary rotation in 3D is acted on such symmetric structures called G 0 which ensures the geometric graph after rotation called G 1 does not coincide with the original one. The goal of our experiments is to check whether different equivariant neural networks can distinguish G 0 and G 1 .The models we select include two models that only use Cartesian coordinates: EGNN and GVP-GNN; and two models that use high-degree steerable features: TFN and MACE. However, TFN and MACE (denoted as TFN/MACE l≤L ) always use all degrees l ∈ {0, . . . , L}, so it is not clear which degree(s) of steerable features distinguish the two geometric graphs. In our HEGNN, all steerable features corresponding to unwanted degrees could be masked during initialization in Eq. ( 5), and we let HEGNN l=L be a HEGNN with only lth-degree steerable features. Additionally, to align with TFN/MACE, we also test the performance of HEGNN with all l ∈ {0, 1, . . . , L} donated as HEGNN l≤L . Following the settings6 in [35] , the output of each graph is the concatenation of invariant scalars, coordinates, and high-degree steerable features pooling among all nodes. We then map this spliced vector to a two-dimensional vector and input it into a simple classifier to determine whether the equivariant graph neural network can distinguish G 0 from G 1 .The results on k-fold are deferred to Appendix for saving space, and the results on regular polyhedra are shown in Table 2 . From Table 1 , we can know steerable features in which degree could not distinguish specific symmetry structure, and both results on k-fold and regular polyhedra are also in perfect agreement with our conclusions. Models (EGNN and GVP-GNN) only with Cartesian vectors cannot distinguish any symmetric graph at all. Taking HEGNN l=5 as an example, since D (5) (H) = 0, ∀H ∈ {T, O, I}, no matter which kind of regular polyhedron, f (5) could only output 0 thus failing to distinguish the structures. Datasets: We benchmark our HEGNN in two scenarios, including: N -body system [43] is a dataset generated from simulations. In our simulations, each system contains N charged particles with random charge c i ∈ {0, 1}, whose movements are driven by Coulomb forces. To verify the efficiency and effectiveness of our HEGNN on datasets of different sizes, we select N from {5, 20, 50, 100}.We use 5000 samples for training, 2000 for validation, and 2000 for testing. The task is to estimate the positions of the N particles after 1,000 timesteps. MD17 [44] dataset contains trajectory data for eight molecules generated through molecular dynamics simulations. The goal of this experiment is to predict the future positions of the atoms based on their current state. We follow the dataset partitioning scheme from [45] , splitting the dataset into 500/2000/2000 frame pairs for training, validation and testing, respectively. All experiments are run on a single NVIDIA A100-80G GPU.Baselines: To demonstrate the advantages of our HEGNN over both models with scalarization techniques and models with high-degree steerable vectors at the same time, our baseline needs to consider the selection issues of both models simultaneously. Therefore, we select some representative models as baselines, including the invariant RF [46] , the equivariant EGNN [1] , TFN [19] and SE(3)-Tr. [20] . In addition, we select classical models such as Linear dynamics [1] , the non-equivariant Message Passing Neural Network (MPNN) [47] , the invariant SchNet [48] , and the equivariant GVP-GNN [49] for the N -body experiments. For MD17 experiments, we also select GMN [45] .Metrics: 1. Loss function: We use Mean Squared Error (MSE) to measure the accuracy of the prediction results in both experiments. 2. Inference time: Given that the N -body system we use contains data of varying sizes, we test the inference time of each model on this dataset. The inference time for each model is calculated relative to the benchmark, which is the inference time of EGNN at the corresponding scale.Results on N -Body systems: The main results of N -body system simulation are presented in Table 3 .From these results, we observe the following: 1. Overall performance: Our HEGNN significantly outperforms other models across datasets of all sizes. Although EGNN [1] performs better than high-degree steerable models like TFN or SE(3)-Transformer in this task, our HEGNN is still better than EGNN, which show that the method of HEGNN introducing high-degree steerable features is more effective. 2. Stability: Although the performance of the model (HEGNN l≤6 ) using high-degree steerable features declines slightly when the geometric graph is small, overall, HEGNN performs better than other models. 3. Inference time: Our model's inference time is significantly faster than that of high-degree steerable models like TFN, reflecting the simplicity and efficiency of our HEGNN.The main results of MD17 experiment are shown in Table 4 , with some data sourced from [45] . From these results, we draw the following insights: 1. Overall performance:Our HEGNN outperforms other models on six out of eight molecules. The effect on the remaining two molecules is only not as good as GMN [45] and this is because GMN introduces additional knowledge such as chemical bonds. In practical scenarios, slight perturbations (such as molecular vibrations) can disrupt strict symmetry, potentially mitigating the conclusions outlined in Theorems 3.5 and 3.6. We therefore designed this perturbation experiment for a simple study and were surprised to find that HEGNN can still bring better robustness through the introduction of high-degree steerable features.We take the tetrahedron as an example and compare the cases of EGNN, HEGNN l=3 , and HEGNN l≤3 when adding noise perturbations with results in Table 5 . Here, ε represents the ratio of noise, and the modulus of the noise obeys Results: It can be observed that the performance of EGNN is slightly improved in the presence of noise (from 50% when ε = 0.01 to 60% when ε = 0.5), while HEGNN demonstrates better robustness. Even though symmetry-breaking factors will make the geometric graph deviate from the symmetric state, the deviated graph is still roughly symmetric. In other words, the outputs of equivariant GNNs on the derivated graphs keep close to zero if the degree value is chosen to be those in Table 1 , which will still lead to defective performance.N (0, ε • E[∥⃗ x -⃗ x c ∥] • I). 
conclusion: 
 In this paper, we challenged the prevailing notion that higher-degree steerable vectors are unnecessary for achieving expressivity in equivariant Graph Neural Networks (GNNs). Through rigorous theoretical analysis, we demonstrated that equivariant GNNs constrained to 1st-degree representations inevitably degenerate to zero functions when applied to symmetric structures, such as k-fold rotations and regular polyhedra. To address this limitation, we introduced HEGNN, a high-degree extension of the EGNN model. HEGNN enhances expressivity by integrating higher-degree steerable vectors while retaining the efficiency of the original model through a scalarization technique. Our extensive empirical evaluations on various datasets, including the symmetric toy dataset, N -body, and MD17, validate our theoretical predictions. HEGNN not only adheres to our theoretical insights but also exhibits significant performance improvements over existing models. These findings underscore the critical role of higher-degree representations in fully leveraging the potential of equivariant GNNs. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Our research belongs to the field of AI for Science. The HEGNN proposed in this article is expected to better model scientific problems, thereby promoting the development of higher-precision and efficient AI scientific models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper poses no such risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All existing assets used in our paper have used with citation. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Equivariance/Invariance of HEGNNIn this section, we demonstrate the equivariance of our HEGNN. In order to further illustrate the connection between our HEGNN, EGNN, and TFN, a general proof is given here.Table 6 : Comparison between our HEGNN and the scalarization-based model representing EGNN [1] , and the high-degree steerable model representing TFN [19] . HEGNN combines the scalarization trick of EGNN that only uses invariant scalars (0th degree steerable features) to interact between steerable features corresponding to different degrees, avoiding the high computational cost of using CG tensor products in TFN.EGNN [1] TFN [19] HEGNN (Ours)Msg m ij = ϕ m (h i , h j , e ij , d 2 ij ) ⃗ m ij = φ ⃗ x (m ij ) • (⃗ x i -⃗ x j ) m(L) ij = ṽ(L) i ⊗ W (dij ) cg Y (L) ⃗ x ij ∥⃗ x ij ∥ m ij = φ m (h i , h j , e ij , d 2 ij , l∈L z (l) ij ) ⃗ m ij = φ ⃗ x (m ij ) • (⃗ x i -⃗ x j ) ⃗ v (l) ij = φ (l) ṽ (m ij ) • (ṽ (l) i - ṽ(l) j ) Agg m i = α i j∈N (i) m ij ⃗ m i = α i j∈N (i) ⃗ m ij m(L) i = α i j∈N (i) m(L) ij m i = α i j∈N (i) m ij ⃗ m i = α i j∈N (i) ⃗ m ij m(l) i = α i j∈N (i) m(l) ij Upd ∆h i = φ h (h i , m i ) ∆⃗ x i = ⃗ m i ∆v (L) i = m (L) i ∆h i = φ h (h i , m i ) ∆⃗ x i = ⃗ m i ∆ṽ (l) i = m(l) i Theorem A.1 (Equivariance/Invariance of HEGNN). h i , ṽ(0) i in HEGNN is E(3) invariant, ⃗ x i is E(3) equivariant.In addition, all ṽ(l) i are O(3) equivariant and translation invariant when l is odd; all ṽ(l) i is SO(3) equivariant and inversion/translation invariant to when l even.Proof. Consider a sequence composed of functions {φ i : X (i-1) → X (i) } N i=1 equivariant to a same group H, the equivariance lead to an interesting property thatφ N • • • • • φ i+1 • ρ X (i) (h)φ i • • • • • φ 1 = φ N • • • • • φ j+1 • ρ X (j) (h)φ j • • • • • φ 1 ,holds for all i, j ∈ {1, 2, . . . , N } and h ∈ H, which means that the group elements h can be freely exchanged in the composite sequence of equivariant functions. In particular, if one of the equivariant functions (e.g. φ k ) is replaced by an invariant function, the group element h will be absorbed, which meansφ N • • • • • φ k • • • • • φ i+1 • ρ X (i) (h)φ i • • • • • φ 1 = φ N • • • • • φ 1 . holds for all h ∈ H but only i ∈ {1, 2, . . . , k}. Although φ N • • • • • φ k is still equivariant, because the group elements must be input starting from φ 1 , the overall φ N • • • • • φ 1 is still an invariant function.That is to say, to conclude that the entire HEGNN is equivariant, we only need to prove that HEGNN is equivariant in initialization and each layer.The initialization of HEGNN is based on spherical harmonics, which is similar to TFN. Spherical harmonics are inherently equivariant, that is,Y (l) (R r ⃗ x) = D (l) (r)Y (l) (⃗ x).Note that variables participating in the coefficient in Eq. ( 5) are all invariant scalars, so the initialization of HEGNN is consistent with the spherical harmonic function. Note that for Cartesian vectors, they can be aligned by arranging the spherical harmonics of 1st degree [19] , that is,⃗ x ∝ Y (1) (⃗ x).From this perspective, EGNN can also be considered to be initialized using spherical harmonics, but this step is omitted because the value is proportional to the input Cartesian vector.It is worth explaining that spherical harmonics are inversion invariant when l is even, that is,Y (l) (m⃗ x) = Y (l) (⃗ x).Equivariant ones are not necessarily better than invariant ones. When we need to predict pseudovectors (such as moments), we need to inversion invariant 1st-degree steerable features, because pseudovectors are inversion invariant. This is why introducing the cross product ⃗ x × ⃗ y (the result is inversion invariant) into a linear combination can only build a SE(3) equivariant network, but not a E(3) network [50] .In fact, inversion equivariant/invariant high degree steerable features can be obtained by calculating the CG tensor product of spherical harmonics and inversion equivariant Cartesian vectors like ⃗ x i -⃗x c [39] . Moreover, the equivariance at each layer is also easy to prove and the internal Wigner-D matrix can be extracted through the CG tensor product [31] . (l) .D (l) (h)ṽ (l) = D (l1) (h)ṽ (l1) ⊗ W cg D (l2) (h)ṽ (l2)When the output result is an invariant scalar, the Wigner-D matrix degenerates into a trivial representation 1. Norm and inner product are all special cases of this type, so equivariance is established. From this perspective, EGNN and HEGNN are equivalent to using only the weight coefficients of (l, l) → 0 and (0, l) → l. Similar ideas include the steerable MLPs in [33] .Theorem A.2 (Theorem 3.4). Suppose that f (l) is an O(3)-equivariant function on geometric graphs, regarding the group representation ρ (l) defined in Eq. (2). Then, for any symmetric graph G induced by the group H ≤ O(3), namely, ∀G ∈ G(H), we always haveEQUATIONHere we have defined group average asρ (l) (H) := 1 |H| h∈H ρ (l) (h). Proof. If f (l) is O(3)-equivariant, then it is also a H-equivariant, thus f (l) (G) = 1 |H| h∈H f (l) (h • G) = 1 |H| h∈H ρ(h)f (l) (G) =   1 |H| h∈H ρ(h)   f (l) (G) = ρ(H)f (l) (G).Theorem A.3 (Theorem 3.5). If and only if the matrix I 2l+1 -ρ (l) (H) is non-singular, the O(3)equivariant function f (l) is always a zero function on G, namely,EQUATIONProof. From Theorem 3.4, we know thatf (l) (G) = ρ(H)f (l) (G) ⇐⇒ I 2l+1 -ρ (l) (H) f (l) (G) = 0,and the theorem holds for basic knowledge of linear algebra.Theorem A.4 (Theorem 3.6). For a finite group H with its representation ρ (l) , ρ (l) (H) is a zero matrix (i.e., ρ (l) (H) = 0) if and only if tr(ρ (l) (H)) = 0. In this case,f (l) (G) ≡ 0, ∀G ∈ G(H).Proof. It is obvious that ρ (l) (H) = 0 =⇒ tr(ρ (l) (H)) = 0 since all elements are zero not to mention the main diagonal, and we only to prove tr(ρ(l) (H)) = 0 =⇒ ρ (l) (H) = 0. A basic fact is h • H = H, thereby ρ (l) (h)ρ (l) (H) = ρ (l) (H).ρ (l) (H) 2 = ρ (l) (H)Such operation can be repeated many times, so thatρ (l) (H) k = ρ (l) (H), ∀k ∈ N + .Now we calculate the trace for each matrix and findtr ρ (l) (H) k = tr ρ (l) (H) = 0, ∀k ∈ N + .By Newton's identity [51] , all eigenvalues of the matrix ρ(H) are 0, that is, the matrix is a zero matrix.Table 7 : The traces of symmetric groups based on [52] . Trace for polyhedral groups can be calculated by Theorem A.5 (Theorem 4.1). For any geometric graph, there exists a bijection between the set of inner products {zD (l) (H) = ⌊l/(l) ij } |Aij |l=1 given by Eq. ( 10) and the set of edge anglesA ij = {θ is,jt := ⟨⃗ x is , ⃗ x jt ⟩} s∈N (i),t∈N (j) .Proof. Note that the Legendre polynomial is a set of orthogonal polynomial bases, and there is a bijection to the power function polynomial space, that is span   |Aij | n=1 P (l) (cos θ n )    M l=0 = span    |Aij | n=1 cos α θ n    M α=0, where M is any non-negative integer represents the degree of the polynomial space. Moreover, from the knowledge of Newton's identities, the space of power sums can be converted to space of elementary symmetric polynomials asspan    |Aij | n=1 cos α θ n    M α=0 = span    1≤n1<n2<...nm≤|Aij | k ν=1 cos θ n k    M α=0From Vieta's formulas, when M = |A ij |, with the M + 1 polynomial in the space of elementary symmetric polynomials being coefficients, we can build a |A ij |-degree polynomial with {cos θ | θ ∈ A ij } as its all roots 7 . Since all angles are in [0, π), the cosine uniquely determines the angle value, and the proposition is established.Our theory in fact shows that the degeneration of global features of a certain degree (in Table 1 ) are inevitable on symmetric geometric graphs. This raises two points worth discussing:1. The degree not indicated to degenerate not necessarily produce a non-zero representation, which may still be affected by the model form and the edge situation. 2. There are some tricks to get around this degeneration: for example, making the output a set or relaxing equivariance constraints (e.g. probabilistic symmetry breaking [54] ).It is worth mentioning that outputting a set can solve most problems, although such operators may be quite intractable to implement in computer systems. The failure cases of frame averaging [26; 27] and Neural P 3 M [30] which depend on singular value decomposition or eigenvalue decomposition, is caused by the non-unique matrix decomposition. Some other examples include ComENet [55] , which uses the scatter_min() operator in PyTorch to extract the nearest neighbors, making it intractable to handle the situation where multiple neighbors are simultaneously closest, which is quite common in chemical molecules (e.g. -CH 3 , -NH 2 ).Moreover, from Theorem 4.1, we show the expressivity of our HEGNN, which is able to recover the information of all angles between each pair of edge. However, it should be noted that |A ij | may be an extremely large number, which is unacceptable in practical applications to achieve completeness.The same problem also arises in discussions based on CG tensor product models (e.g. TFN [19] ), such as discussions based on D-spanning [23] , because a sufficiently high-degree D is unacceptable. However, in terms of actual results, the performance of both our HEGNN and TFN is remarkable.From this perspective, how to bridge the gap between completeness and actual performance with features of limited channels and degrees is a question worth considering.To get the ball rolling, we raise an interesting question here. Is there a performance gap between this type of purely mathematical representation and other features based on physical and biochemical prior knowledge? In fact, some of these features are directly related (e.g. frames and fractional coordinates). How to construct effective and interpretable pure mathematical features based on those with prior knowledge will become a key point in network design, and we will consider further exploration in future work.Like EGNN [1] , different features use different numbers of channels, so the inference time does not obviously reflect the time complexity of O(L 2 ). We list the details of our HEGNN of different degree in Table 8 . Intuitively, we add one of each steerable feature on the basis of EGNN (using 64 invariant scalars and 2 Cartesian vectors, i.e. coordinate and velocity). Channel for ṽ(0) 65 65 65 65 Channel for ṽ(1)EQUATIONTotal dimensions 74 79 86 119Table 9 shows the number of parameters and inference time of different models. The results on the k-fold structure are completely consistent with the conclusion of Table 1 . The few results that cannot reach 100% are due to the small number of training epochs, so the classifier fails to perform perfect classification. In fact, the accuracy of models can achieve 100.00± 0.0 after increasing the number of training rounds of the model like 500 epochs. Since from 2nd-degree steerable features can distinguish G 0 and G 1 , and HEGNN/TFN/MACE l≤L contain 2nd-degree steerable features when l ≥ 3, the extra results are hidden but all their values are 100.00± 0.0.Table 10 : k-fold symmetric structures. Implementation details: Since the e3nn [39] library only implements spherical harmonics up to the 11th-degree, we verified the expressivity of our HEGNN in disguise. In this experiment, we measure the expressivity of our HEGNN by calculating whether the high-degree feature ṽ(l) i updates on regular polyhedra. Namely, if ∆ṽ (L) i = 0, then HEGNN l=L loses the expressivity. This judgment is necessary and sufficient. We calculated the sum of all degree vectors from L = 1 to 30, which Note that SEGNN in Table 12 does not show the effect in the original paper. We also tried to reproduce the dataset in the original paper [33] , and the effect of SEGNN on 5-body dataset can be reproduced. However, see Table 13 , it is also shown that SEGNN performs poorly on larger datasets. We found that the GMN-L method proposed in [29] generally performs the best on MD17 dataset, but our HEGNN-6 also achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance. Note that both EGNN [1] and HEGNN l≤1 only use Cartesian vectors. However, in Table 3 , the effect of the latter is greatly improved. We speculate that there are two possible factors for this improvement: 1) the extra layer of message passing brought in by Eq. ( 5); 2) the multi-channel of Cartesian vectors (see Table 8 ). Therefore, we tested the effect of HEGNN l≤1 -3layers, and the results are shown in Table 15 . From the improvement, the first factor contributes more. Our current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. Whether our HEGNN is effective on large-scale geometric graph data sets remains to be verified.Our research belongs to the field of AI for Science. The HEGNN proposed in this article is expected to better model scientific problems, thereby promoting the development of higher-precision and efficient AI scientific models.NeurIPS Paper ChecklistSee Appendix A.3 for further discussion.Code is available at https://github.com/GLAD-RUC/HEGNN.Eq. (5) is unable to derive pseudo-vectors such as torque or angular momentum, which are type-1 steerable features but invariant to reflection. To address this issue, we can further conduct CG tensor product between ṽ(l) i,init and ⃗ xi -⃗ xj to yield the steerable feature of desired symmetry.https://github.com/chaitjo/geometric-gnn-dojo.It should be noted here that because only 0 ∼ 11-th-degree spherical harmonics can be used in e3nn[39], we only measure the models with up to 11th-degree here, and in Appendix B.3, We have given a new verification method.The trick comes from Lemma 6 in DeepSet[53].https://github.com/GLAD-RUC/FastEGNN. 


Decision: Based on the positive reviews and constructive discussion, I recommend accepting this paper. The work presents novel theoretical insights into equivariant Graph Neural Networks and proposes an innovative model (HEGNN) that balances expressivity and efficiency. Experimental results support the theoretical claims and show improvements over existing methods. While minor revisions were suggested, the reviewers agreed on the paper's technical soundness and valuable contribution to the field.

Reviews: review: paper_summary: paper_summary: The paper studies the benefit of using higher order steerable features in geometric GNNs and theoretically identifies classes of symmetric geometric graphs where methods using only order-1 (or low order) features are guaranteed to fail.
With this in mind, the authors propose a simple and efficient way to integrate higher order features in the EGNN architecture.
strength_weakness: strength_weakness: While the overall novelty is a bit limited (the message that higher order features are necessary for expressiveness has been theoretically and empirically explored in a number of previous works in the literature), the theoretical derivation of the failure cases and the proposed architecture can be useful contributions to the community. 

See questions below.
questions: questions: Eq 7, 8: do I understand correctly that this kind of model can not pass any information between features $\tilde{v}^{(l)}$ of different orders $(l)$ beyond the invariant quantities used to compute $m_{ij}$? Then, that essentially means that, after the initial features are extracted via the first convolution with spherical harmonics in Eq. 5, *the rest of model* is invariant to independent rotations by O(3) of each feature $\tilde{v}^{(l)}$, right?
This seems to suggest there should be a gap in performance with models like TFN or SEGNN, which instead allow for different frequencies (features of different orders) to interact inside the model. Can you elaborate on this?


Sec 5.2: it seems important to compare with SEGNN [21] since that work was also motivated by the idea of adapting EGNN to support higher order steerable features, if I remember correctly.


Table 3: I would expect the model size to scale quadratically with the maximum frequency $L$ since $\sum_{l=0}^L (2l+1) = (L+1)^2$, but the runtimes in table 3 do not show this pattern. Why is the case?


Table 4: what are the SOTA on these datasets? It would be good to include the performance achieved by some previous works to give an idea of whether these are competitive results or not


Why introducing Theorem 3.5 when Theorem 3.6 seems more general and practical? Also, I think these theorems hold for any compact (not necessarily finite) subgroups by using the Peter-Weyl theorem.


Line 226: is this a more efficient way to implement this fairly sparse operation?
limitations: limitations: Some limitations are addressed in the supplementary material.

For completeness, I think the authors could comment on the possible gap in expressiveness with other steerable methods like TFN and SEGNN which allow for features of different orders to interact (see my first question above).
paper_summary: paper_summary: This paper studies the necessity of higher-degree features in geometric graph neural networks (i.e. graph neural networks processing data embedded in three-dimensional space), focusing on the ability to recognize data with non-trivial rotational inner-symmetry such as k-folds and regular polygons, and based on the findings, proposes and validates an extension of the E(n)-GNN architecture. Theoretically, the authors first state in Theorem 3.4 that any O(3)-equivariant function on an inner-symmetric graph must produce an output which has an identical inner-symmetry. This immediately puts a restriction on the possible space of outputs an equivariant function can produce, more specifically as the invariant subspace under the action of the inner-symmetry (Theorem 3.5). It follows that certain inner-symmetric inputs and certain choices of output spaces (i.e. degrees $l$) leads to degenerate output space {0} (Theorem 3.6 and Table 1). Consequently, it follows that using a sufficiently high-degree (even) features leads to non-degenerate output spaces of layers, hence be able to encode e.g. orientations of the input. The authors empirically validate the findings on expressive power in Section 5.1, and then demonstrate good performance of the proposed model based on high-degree features and invariant scalarization on n-body for n <= 100 and MD17 datasets.
strength_weakness: strength_weakness: - W1. While technically different from [1], this paper conveys a similar message: the use of higher-degree tensors in geometric neural networks is necessary to obtain higher expressive powers which is necessary for certain problems. Given this, one may argue that the GWL hierarchy in [1] is more general (as it handles general non-symmetric inputs as well). Furthermore, inputs with rotational inner-symmetry have been investigated in Section 5.2 of [1] to argue in favor of the higher-degree tensor features, precisely using the task of encoding rotations of the input; while the types of inner-symmetries considered in this work is more general (Table 1), the finding is fundamentally not very different, which can be understood as a weakness of the paper.
- W2. It was not entirely clear to me why the capability of encoding rotations of inner-symmetric inputs would be important for practical learning tasks, although the experiments seem to imply so. For example, on n-body systems, I believe it is very unlikely that one would encounter symmetric inputs including the ones given in Table 1, due to symmetry-breaking factors such as numerical imprecision, simulation errors, and noise. I have similar concerns regarding chemical structures (e.g. MD17), as slight perturbations to the atomic coordinates are sufficient to eliminate the inner-symmetries (if they exist) and hence not experience the problems given in Theorem 3.5 and 3.6. This implies a gap between the theoretical arguments on the expressive power given in Section 3, and empirical results given in Sections 5.2 and 5.3.
- W3. While Theorem 4.1 says that the proposed architecture can recover the information of all angles between each pair of edges, it does not directly address the original problem given in Theorem 3.5 and 3.6.
- W4. I am not sure why MACE [2], while also using higher-degree features and already used for experiments in Tables 1 and 2, is not included for comparison in Tables 3 and 4.

[1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024)

[2] Batatia et al. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2023)
questions: questions: Please see the Weaknesses section.
limitations: limitations: The authors have discussed the limitations in the checklist, but not in the main text. I encourage the reader to move the discussions to a separate section in the main text or Appendix.
paper_summary: paper_summary: This paper challenges the prevailing notion that high-degree steerable representations are unnecessary in equivariant Graph Neural Networks (GNNs). The authors provide theoretical analysis showing that equivariant GNNs constrained to 1st-degree representations degenerate to zero functions when applied to symmetric structures like k-fold rotations and regular polyhedra. To address this limitation, they propose HEGNN, a high-degree extension of the EGNN model that incorporates higher-degree steerable vectors while maintaining efficiency through a scalarization technique. The authors evaluate HEGNN on symmetric toy datasets, N-body systems, and MD17 molecular dynamics datasets, demonstrating improved performance over existing models.
strength_weakness: strength_weakness: 1. The experimental comparisons on the N-body system are limited and use different splits and variations compared to existing literature. This makes it difficult to directly compare HEGNN's performance to state-of-the-art methods. Including comparisons with more recent baselines (e.g., ClofNet, GCPNet, SaVeNet) and using standardized benchmarks would strengthen the empirical evaluation.
2. The experiments are primarily focused on predicting future positions of particles/atoms. While this is a relevant task, it may not fully demonstrate the necessity or advantages of high-degree representations across a wider range of equivariant GNN applications. Additional experiments on different tasks or domains could provide more comprehensive evidence to support the paper's claims.
3. Some relevant baselines are missing from the comparisons. For example, GMN-L is not included in the MD17 dataset experiments, despite outperforming the proposed method on several targets. Similarly, GMN is missing from the N-body systems experiment. Including these baselines would provide a more complete picture of HEGNN's performance relative to existing methods.
4. The paper could benefit from a more extensive discussion of the computational trade-offs involved in using high-degree representations. While the authors mention that HEGNN is more efficient than traditional high-degree models, a more detailed analysis of the computational costs and scaling behavior would be valuable.
questions: questions: 1. Can the authors conduct additional experiments with different tasks or application domains to further support the claims presented in the paper? This could help demonstrate the broader applicability and necessity of high-degree representations in equivariant GNNs.
2. How does HEGNN compare to more recent state-of-the-art methods on standardized N-body system benchmarks, such as those used in the ClofNet paper and subsequent works?
3. Can the authors provide a more detailed analysis of the computational efficiency of HEGNN compared to other high-degree models and EGNN? This could include training times, memory usage, and scaling behavior with respect to the number of particles and degree of representations.
4. Have the authors explored the performance of HEGNN on tasks beyond position prediction, such as force field prediction or other molecular property predictions? This could help strengthen the argument for the necessity of high-degree representations.
limitations: limitations: The authors acknowledge that their current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. It remains to be verified whether HEGNN is effective on large-scale geometric graph datasets.
paper_summary: paper_summary: The present work pertains to analysing and improving Graph Neural Networks operating on geometric graphs that possess E(3) (Euclidean) symmetries, i.e. the tasks of interest are invariant/equivariant to rotations, translations and reflections. The paper challenges the assumption that it is sufficient to use E(3)-equivariant features of 1st degree (the term “degree” is related to the dimensionality of the group representation to which the features are equivariant - 1st-degree features are 3-dimensional and are equivariant to 3D rotations/reflections), as done in EGNN, Satorras et al., ICML’21, which works quite well in practice and is computationally efficient. The authors construct a series of counterexample symmetric geometric graphs (i.e. graphs which, when rotated by certain angles, remain intact up to permutations of their vertices) which are shown to be challenging for geometric GNNs. It is shown that, for certain degrees $l$, a GNN can only produce output features equal to zero, therefore showing that the degree is crucial for expressivity and that $l=1$ is insufficient for some cases. To address this limitation, they combine ideas from EGNN and TFN, Thomas et al., arxiv’18: They create a GNN, similar to EGNN, enriched with auxiliary features of higher degree, using spherical harmonics (similar to TFN). To keep the computational complexity low, they produce the messages used for the updates of node features/positions/auxiliary features via scalarisation using inner products, simplifying the messages used in TFN. The expressivity of the method is theoretically analysed, showing the ability to recover angles, and empirically tested, showing improved performance in several synthetic and real-world benchmarks.
strength_weakness: strength_weakness: **Clarity: limits audience and creates some issues with contextualization to related work**. Although not a major weakness, there are some issues with the clarity of the presentation, which I think should be addressed to improve the exposition of the paper to a wider audience. 
  - In the introduction and the related work section, the authors mention several concepts without providing explanations which might make the text hard to follow, especially for readers not experts in geometric GNNs and Euclidean symmetries. 
    - For instance: *x-degree steerable representations* - this is a notion at the heart of the paper, so it should probably be intuitively explained early on, e.g. by summarising the formal definitions at the end of section 3.1.
    - *Clebsch-Gordan tensor product* - this concept is not explained in the paper. However, I do not think that the authors should assume familiarity with it. The reader needs to obtain a clear understanding due to the following: the main methodological innovation of the present work is to replace this operation with inner products to improve computational efficiency. I think that the authors should explain in a separate subsection the differences between these two operators and discuss what they sacrifice by not incorporating CG products in their architecture.
-	Examples 3.2. and 3.3. need further elaboration as they are the main motivation behind the proposed methodology. E.g. why do even folds are not symmetric w.r.t. the cyclic group? Perhaps explain what the dihedral group is. Why are regular polygons symmetric w.r.t. the dihedral group?
questions: questions: - Section 3.1/Section 4: It is unclear to me why the authors introduced the modulation of the spherical harmonics (re-scaling). How does this affect expressivity?
- Section 3.1: Is the representation of O(3) – Eq. (2) – conventional or introduced here for the first time? In the first case, could the authors provide a reference? In the second case, it would be more appropriate to precisely show that this is indeed a representation for completeness.
- As far as I understand, Eq. (3) refers to graph-level features. Perhaps this should be explicitly mentioned to improve readability.
- What information is lost from the scalarization trick of Eq. (6) compared to CG tensor products? To what extent does this limit expressivity?
- How do the authors explain that in certain cases in Table 2, HEGNN does not achieve 100% performance (although it should according to the theory)? Additionally, why is the std so high (15% and 30%)?
- In table 3, it seems that HEGNN with $l\leq 1$ always outperforms EGNN, which is also 1st-degree. How do the authors explain this?
- It is unclear to me how equivariance to reflections is handled by this method. How does the discussion at the end of section 3.1 affects the operators used?
- Table 5 is quite helpful, perhaps consider moving it to the main text. 

**Minor**: I have the impression that there were a few typos throughout the text. I suggest performing a proof-reading (examples: L78 typo: university --> universality, L266 type: donated --> denoted).
limitations: limitations: I recommend that the authors devote a separate section to discuss the limitations of their work. Currently, I could locate only a limited discussion in the appendix with regards to not testing in large-scale systems.

I do not foresee any negative societal impact.

Comments: title: Rebuttal Acknowledgment
Post rebuttal
comment: Dear Reviewers and Area Chairs,

We would like to extend our heartfelt gratitude for your dedicated efforts, insightful feedback, and constructive suggestions. Your nice comments and thoughtful suggestions are truly commendable.

Through our discussions and the reviewers' responses, we convice we have addressed the major concerns raised by reviewers. Our work received positive comments from all reviewers both before and after the rebuttal, and now there are two reviewers have given clear acceptance recommendations (Reviewer 7wT3 & 3KcU). This outcome has greatly benefited and encouraged us, and we would like to thank all of you for your valuable support! 

Our theory strictly proved the necessity of high-degree steerable features, while our novel framework, HEGNN, proposed a promising method based on scalarization-trick to introduce high-degree steerable features with both efficacy and efficiency. We have made our complete code and training details publicly available, ensuring transparency and reproducibility. The rebuttal points raised during this process will be thoughtfully incorporated into the final version.

Once again, we express our sincere appreciation for your time and dedication in reviewing our work. Your insightful input has significantly contributed to the refinement of our manuscript. Thank you!
Thank you for keeping your positive recommendation on our work. We will definitely and gladly revise our manuscript according to your inspiring suggestions and those of the other reviewers.
I thank the authors for their responses and the additional experiments provided. I strongly encourage them to update their manuscript in several parts as per the reviewers' suggestions (e.g. extra explanations to improve clarity, clearer comparison with CG tensor product to illustrate their technical contribution, additional experiments suggested by reviewer vhjv, computational complexity/runtime discussion suggested by reviewer op9c). I maintain my initial score and positive evaluation of this paper and recommend acceptance.
Thank you for your constructive feedback and willingness to raise the score, which greatly inspired us. We will definitely incorporate your insightful suggestions into our revisions.
Thank you for the comprehensive response. The fact that GWL has only empirically investigated the inner-symmetric inputs is something I have missed and indeed addresses the original limitation I have raised. The results in Table S3 seem interesting and important; I propose that the authors include it in the main text in the revision, rather than including it in the Appendix. It would have been great if the performance of MACE on MD-17 was also included, but I guess the results are still valid as it involves comparisons to current SOTA (response to Q4 of reviewer op9c). I have adjusted my score accordingly.
Thank you for your positive recognition of our paper. We will definitly add the details and results in the final version of our paper.  Regarding your mention of our limited novelty, we found that we did not explain it clearly in our previous response. Please allow us to add a few points here.

- Although there have been some works studying the importance of using high-degree features, we are the first to rigorously explain the expressivity barriers of low-degree representations. Theorem 3.5 and Theorem 3.6  are simple and concise, yet insightful and of independent technical interest, as also pointed out by Reveiwer 3KcU. 
- To the best of our knowledge, our HEGNN model is the first to exploit high-degree steerable features using the scalarization trick. This approach not only surpasses prior higher-order models in efficiency but also achieves superior results in experimental results. Besides, its easy implementation may facilitate reproducibility.
- In Theorem 4.1, we demonstrate that employ inner products of full degree can recover the information of all angles between each pair of edges, thus affirming the theoretical expressivity of our HEGNN in capturing the geometry of the input structure. 

Once again, thank you very much for your valuable comments and suggestions that help improve our paper.
Thanks for the reply, I think that cleared most of my doubts!
I encourage the authors to include these details and results in the final version of the manuscript.

I still feel like the novelty is limited, but the paper includes some interesting insights and the proposed method seems effective and seems to be thoroughly evaluated. For these reasons, I maintain my positive recommendation.
We sincerely appreciate your further comments. We provide more explanations to address your concerns below.

> **Q1: The relationship between model time consumption and degree.**

Your question is both valuable and inspiring! We sincerely apologize for our previous unconsidered response. Upon re-evaluating the complexity of our model, we now believe that your insight is correct—the model's bottleneck lies not only in the value of the degree $L$ but also in the number of channels. As illustrated in the following table, for our models with different $L$ values, the number of channels for type-0 features is fixed to 65, while it is no more than 3 for higher-type features. In other words, the computations for type-0 features account for the majority of the overall running cost. This could explain why the runtimes in Table 3 do not scale quadratically with respect to $L$. Additionally, there are other factors that affect the model's time consumption, including the initialization in Eq. (5) and the coefficient calculations in Eqs. (5) and (8).

Thank you once again for your constructive comment! We will definitely include the above explanations around Table 3 in the revised paper.

**Table S9:** Total dimensions used in HEGNN of different degrees

||HEGNN$\_{l\leq1}$|HEGNN$\_{l\leq2}$|HEGNN$\_{l\leq3}$|HEGNN$\_{l\leq6}$|
|-|-|-|-|-|
|Type-0|65|65|65|65|
|Type-1|3|3|3|3|
|Type-2|-|1|1|1|
|Type-3|-|-|1|1|
|Type-4|-|-|-|1|
|Type-5|-|-|-|1|
|Type-6|-|-|-|1|
|Total Dim|74|79|86|119|

> **Q2: The gap in performance of SEGNN between our results and original paper.**

Yes, the performance discrepancy of SEGNN can be attributed to the use of a different dataset than the one employed in the original SEGNN paper [21]. In our paper, we utilized the dataset constructed by [a] to provide a more thorough evaluation of the compared methods. This dataset includes a wider range of scenarios, spanning from 5 bodies to 100 bodies, beyond the 5-body case studied in [21]. It is important to note that [a] employed different preprocessing and dataset splitting compared to [21], which may explain why SEGNN's performance differs even in the same 5-body scenario as [21].

In addition to the dataset in [21], here we have conducted additional experiments using the SEGNN dataset, following the official code from the SEGNN paper for dataset construction. We then re-evaluate the performance of EGNN, SEGNN, and our proposed HEGNN models across various scenarios, ranging from 5 bodies to 100 bodies. The results are presented in the following table. 

**Table S10:** Comparison between EGNN, SEGNN and HEGNN on N-body from [21]

|N-body ($\times 10^{-2}$)|5-body|20-body|50-body|100-body|
|-|-|-|-|-|
|EGNN|0.71|1.04|1.15|1.31|
|SEGNN|**0.50**|6.61|9.34|13.46|
|HEGNN$\_{l\leq1}$|0.71|0.97|**0.93**|1.22|
|HEGNN$\_{l\leq2}$|0.65|**0.91**|1.05|**1.14**|
|HEGNN$\_{l\leq3}$|0.63|0.99|1.05|1.27|
|HEGNN$\_{l\leq6}$|0.72|1.05|1.11|1.28|

As observed, SEGNN's performance closely aligns with the results reported in its original paper for the 5-body case (0.50 vs. 0.43), supporting the reliability of our implementation. However, SEGNN shows significantly higher losses in scenarios with a larger number of bodies, indicating a decline in performance as task complexity increases. We conjecture that the steerable nonlinear functions used in SEGNN could make the learning process more difficult when the number of bodies increases. 

Although our proposed HEGNN model performs worse than SEGNN in the 5-body case, it consistently exhibits lower loss across various configurations, demonstrating its superior ability to handle increasing task complexity. 

We appreciate your valuable attention to this detail and will include the above results and analyses to further strengthen the validity of our findings.

[a] Huang W, Han J, Rong Y, et al. Equivariant graph mechanics networks with constraints.
I thank the authors for the detailed reply.

Regarding Q3, the ratio between the models' sizes for $l\leq6$ and $l\leq1$ is larger than 10 (since $(1+1)^2=4$ vs $(6+1)^2=49$). It still seems surprising to me that a model 12 times larger is only 30% slower. Am I missing something? Is it possible that the model's bottleneck is somewhere else? Or, are other hyper-parameters changed in the model across different values of $L$?


Regarding Q4 and Table S1, the (N=5 body) performance reported for SEGNN seems very different from the one in Table 1 in the original paper [21] (1.68 vs 0.43), why is that the case? is the dataset different from the one used in [21]?