Title: Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Geometric Graph Neural Networks

High Degree Steerable Vectors
Abstract: Equivariant Graph Neural Networks (GNNs) that incorporate E(3) symmetry have achieved significant success in various scientific applications. As one of the most successful models, EGNN leverages a simple scalarization technique to perform equivariant message passing over only Cartesian vectors (i.e., 1st-degree steerable vectors), enjoying greater efficiency and efficacy compared to equivariant GNNs using higher-degree steerable vectors. This success suggests that higher-degree representations might be unnecessary. In this paper, we disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric structures, including $k$-fold rotations and regular polyhedra. We theoretically demonstrate that equivariant GNNs will always degenerate to a zero function if the degree of the output representations is fixed to 1 or other specific values. Based on this theoretical insight, we propose HEGNN, a high-degree version of EGNN to increase the expressivity by incorporating high-degree steerable vectors while maintaining EGNN's efficiency through the scalarization trick. Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric structures, but also shows substantial improvements on more complicated datasets such as $N$-body and MD17. Our theoretical findings and empirical results potentially open up new possibilities for the research of equivariant GNNs.

Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?

Abstract

Equivariant Graph Neural Networks (GNNs) that incorporate the E(3) symmetry have achieved significant success in various scientific applications. As one of the most successful models, EGNN [1] leverages a simple scalarization technique to perform equivariant message passing over only Cartesian vectors (i.e., 1stdegree steerable vectors), enjoying greater efficiency and efficacy compared to equivariant GNNs using higher-degree steerable vectors. This success suggests that higher-degree representations might be unnecessary. In this paper, we disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric structures, including k -fold rotations and regular polyhedra. We theoretically demonstrate that equivariant GNNs will always degenerate to a zero function if the degree of the output representations is fixed to 1 or other specific values. Based on this theoretical insight, we propose HEGNN, a high-degree version of EGNN to increase the expressivity by incorporating high-degree steerable vectors while still maintaining EGNN's advantage through the scalarization trick. Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on a toy dataset consisting of symmetric structures, but also shows substantial improvements on other complicated datasets without obvious symmetry, including N -body and MD17. Our study potentially showcase an effective way of modeling high-degree representations in equivariant GNNs.

1 Introduction

Molecules, proteins, crystals, and many other scientific data can be effectively modeled and represented through geometric graphs [2-8]. This type of data structure encapsulates not only node characteristics and edge information but also a 3D vector (such as position, velocity, etc.) for each node. To process geometric graphs, equivariant Graph Neural Networks (GNNs) have been developed, which undergo equivariant message passing over nodes, conforming to the E(3) or SE(3) symmetry of physical laws. These models have achieved remarkable successes in a lot of scientific tasks, such as physical dynamics simulation [9-11], molecular generation [12-15] and protein design [16-18].

Pioneer equivariant GNNs [19-22] derive high-degree steerable representations beyond scalars and 3D coordinates with the help of spherical harmonics and conduct equivariant message passing between representations of different degrees through the Clebsch-Gordan (CG) tensor product. While these high-degree models are able to approximate any function of fully connected geometric graphs in theory [23], they usually suffer from expensive computational costs in practice. In contrast,

k

-fold (odd)

k

-fold (even)

Cube (Hexahedron)

Octahedron

Dodecahedron

Icosahedron

EGNN [1] leverages a simple scalarization technique to allow equivariant message passing over only 3D vectors ( i.e. the 1st-degree steerable features). Specifically, the scalarization technique first encodes 3D vectors into scalars as invariant messages, which are passed as geometric messages after the multiplication with the 3D vectors to recover the orientation information. Despite its simplicity, EGNN achieves remarkably better efficacy and efficiency against conventional high-degree models for a broad range of applications [24; 25]. Such successes suggest that higher-degree representations might be unnecessary.

In this paper, we challenge and disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric graphs. Fig. 1 illustrates the examples of k -fold rotations and regular polyhedra, which are invariant to rotations up to certain rotating angles. Taking the cube for example, conducting 90 ◦ rotation around the axes crossing the center of the two opposite faces keeps its shape and orientation unchanged. Interestingly, by making use of group theory, we theoretically prove that any equivariant GNN (after translating the coordinate center to the origin and conducting graphlevel readout) on these symmetric graphs will degenerate to a zero function if the degree of their representations is fixed to be 1. The direct deduction of this theorem is that EGNN can only output a zero 3D vector no matter how we rotate the input graph, indicating that EGNN totally loses the recognition ability of orientation. Additionally, this statement points out the limitation of the methods that rely on constructing global features for symmetric graphs 2 ( e.g. frames in frame averaging [26; 27], virtual nodes in FastEGNN [28]), equivariant pooling in EGHN [29], and meshes in Neural P 3 M [30]), since it is impossible to output another non-collinear 3D vector except the center coordinate.

Based on the above theoretical insights, we propose a novel equivariant GNN model termed HEGNN 3 , which enhances EGNN by incorporating high-degree steerable vectors while inheriting the desired benefit from EGNN through the scalarization trick. In summary, our contributions are as follows:

We theoretically investigate the expressivity reduction issue of equivariant GNNs on symmetric graphs.

We propose HEGNN, to further incorporate high-degree steerable representations into EGNN. Moreover, since the equivariant message passing process between different-degree representations is conducted via inner products, it shares the same benefit as EGNN, compared to traditional high-degree models.

Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric graphs, but also shows substantial improvements on more complicated datasets without explicit symmetry, such as N -body and MD17.

2 Related Works

Equivariant GNNs. Equivariant GNNs can be divided into two classes: scalarization-based models and high-degree steerable models [31]. Scalarization-based models adopt norms or inner products to convert equivariant 3D vectors into invariant scalars, which are considered as coefficients to linearly combine 3D vectors for node update. EGNN [1] is the first work falling into this category. Concurrently, PAINN [32] further enhances the expressive ability of the model by introducing multi-channel equivariant features. On the contrary, high-degree steerable models ( e.g. TFN [19],

1

SEGNN [33] and SE(3)-Transformer) use spherical harmonics to ensure the equivariance of message passing, and realize interaction between steerable features of different degrees through CG tensor products. Our HEGNN also uses high-degree steerable features, but it leverages the scalarization trick for the interaction between steerable features of different degrees, thus leading to more expressivity than EGNN and less computational cost than other high-degree models.

Expressivity of Equivariant GNNs. The theoretical expressivity of equivariant GNNs is initially explored by [23], which proves the university of the high-degree steerable model, i.e. , TFN [19], over fully-connected geometric graphs. GemNet [34] further demonstrates that the universality holds with just spherical representations other than the full SO(3) representations that are required in the proof of [23]. More recently, the GWL framework [35] extends the Weisfeiler-Lehman (WL) test into a geometric version [36] to study the expressive power of geometric GNNs operating on sparse graphs from the perspective of discriminating geometric graphs. Different from all the above works, our paper investigates the expressivity of equivariant GNNs on symmetric graphs, and demonstrates the necessity of involving high-degree representations. Although the GWL test paper [35] has experimentally compared different models on k -fold structures that are allowed to rotate only in the 2D space, the conclusions of this paper are proved both theoretically and experimentally. Moreover, our discussions cover a full range of examples including k -folds (rotation in 3D space) and regular polyhedra.

3 Theoretical Analyses

In this section, we first present the necessary preliminaries related to geometric graphs and group representation. Then, we define and illustrate typical examples of symmetric graphs. Finally, we will discuss when equivariant GNNs will degenerate to a zero function on symmetric graphs.

3.1 Preliminaries

Geometric graph. A geometric graph of N nodes is defined as G := ( H , ⃗ X ; A ) , where H := { h i ∈ R C H } N i =1 and ⃗ X := { ⃗ x i ∈ R 3 } N i =1 are node features and 3D coordinates, respectively; A ∈ R N × N represents the adjacency matrix and can be assigned with edge features e ij if necessary. Throughout our theoretical analyses in this section, we assume the node features and edge features to be identical for all.

Transformation of geometric graph. We are interested in the transformations of a geometric graph G with respect to a group G , which is defined as g · G , for g ∈ G and · denoting the group action. For instance, g · G can be explained as translation, rotation, or reflection of the coordinates ⃗ X . These transformations form a 3D Euclidean group denoted as E(3) , and its subgroup without translation is called the orthogonality group O(3) . With the aid of group representation ρ ( g ) , the transformation of a coordinate ⃗ x is represented as ρ ( g ) ⃗ x . For example, orthogonal matrices are the trivial representations of O(3) , that is, the orthogonal transformation of a vector ⃗ x is represented by O ⃗ x with O ∈ R 3 × 3 being an orthogonal matrix. Besides, there are other representations of O(3) , such as the irreducible representations which will be detailed below.

Equivariance. Let X and Y be the input and output vector spaces, respectively. A function f : X → Y is called equivariant with respect to group G if

where ρ X and ρ Y are the group representations in the input and output spaces, respectively. Since we can eliminate the translation effect by simply translating the center of all coordinates to the origin, we only discuss equivariance with respect to O(3) in this section. In other words, we default that the center of ⃗ X is at the origin.

Irreducible representations and steerable features. O(3) consists of rotation and inversion, implying O(3) = SO(3) × C i , where SO(3) is the rotation group and C i = { e , i } denotes the inverse group. We first discuss the irreducible representations of SO(3) . For each rotation r ∈ SO(3) , its irreducible representations are Wigner-D matrices D ( l ) ( r ) ∈ R (2 l +1) × (2 l +1) of different degree l ∈ N [21; 37]. When l = 1 , it becomes the common rotation matrix R r acting on the 3D coordinate space. Under the irreducible representations, the equivariant constraint in Eq. (1) turns

into f ( l ) ( R r ⃗ x ) = D ( l ) ( r ) f ( l ) ( ⃗ x ) , if the output degree is l . According to [38], spherical harmonics Y ( l ) = [ Y ( l ) m ( ⃗ x )] l m = -l offer a unique and complete set of function bases satisfying the equivariant constraint. We further define a modulated spherical harmonics as f ( l ) ( ⃗ x ) = φ ( ∥ ⃗ x ∥ ) · Y ( l ) ( ⃗ x / ∥ ⃗ x ∥ ) by adding a continuous radial function φ : R + → R of vector norm ∥ · ∥ for re-scaling. Such a function f l and its output f l ( ⃗ x ) are called typel steerable function and steerable feature , respectively. We now deduce the irreducible representations from SO(3) to O(3) . Note that spherical harmonics satisfy Y ( l ) ( -⃗ x ) = ( -1) l Y ( l ) ( ⃗ x ) ; in other words, they are inverse-equivariant when l is odd, but inverse-invariant when l is even. We thus specify the group representation of O(3) as

where σ ( l ) ( m ) = 1 for m = e (the identity) and σ ( l ) ( m ) = ( -1) l if m = i (the inverse). Readers can refer to the discussion in e3nn [39] with another representation method by using the concept of parity and construct this through methods such as Clebsch-Gordan (CG) tensor product [40]. For concision, the typel steerable feature is denoted as ˜ v ( l ) with a tilde notation.

3.2 Symmetric Graph

In § 1, we present that k -fold rotations and regular polyhedra exhibit certain symmetries. In this subsection, we formally describe them via the notion of the symmetric graph.

̸

Definition 3.1 (Symmetric Graph) . Ageometric graph G is called a symmetric graph, if there exists a finite and nontrivial subgroup H ≤ O(3) , H = { e } , satisfying that ∀ h ∈ H , h · G = G . All subgroups making G symmetric yields a set H ( G ) , and all geometric graphs that are symmetric w.r.t. H constitute a set denoted as G ( H ) .

Here h · G = G is defined in the graph level. Particularly for the coordinates ⃗ X ∈ R 3 × N , it implies that ∀ O ∈ H , ∃ P ∈ S N , O ⃗ X = ⃗ XP and PA = AP , where S N is the permutation group of order N . Essentially, rotating the coordinates of a symmetric graph leads to a copy of this graph up to a different permutation of the nodes.

Without considering inversion, the finite subgroups of SO(3) are only cyclic group C n , dihedral group D n , tetrahedral group T , octahedral group O , and Icosahedral group I [41]. We provide several examples of symmetric graphs as follows.

Example 3.2 ( k -folds) . On the one hand, for a geometric graph G corresponding to a 2 k -fold with nodes { (cos( i · π/k ) , sin( i · π/k ) , 0) } 2 k -1 i =0 , the inverse group C i and the dihedral group D 2 k (rotation around z -axis with angle π/k , and reflection around the axis connecting the midpoints of opposite sides or the axis connecting opposite vertices), are symmetric groups on G , namely, C i , D 2 k ∈ H ( G ) . On the other hand, for a geometric graph G corresponding to a (2 k +1) -fold with nodes { (cos( i · 2 π/ (2 k +1) , sin( i · 2 π/ (2 k +1)) , 0) } 2 k i =0 , H ( G ) includes the dihedral group D 2 k +1 but without the inverse group C i .

Example 3.3 (Regular Polygons) . The symmetric groups of regular polygons in the plane and regular prisms in space include the dihedral group D n . Regular tetrahedra are symmetric with respect to three rotation axes of the second order and four axes of the third order, corresponding to 12 group elements. Regular hexahedra (cubes) and the regular octahedra (which are dual to each other and share the same symmetric groups) are symmetric about six axes of the second order, four axes of the third order, and three axes of the fourth order, corresponding to 24 group elements. Regular dodecahedra and regular icosahedra (which are also dual to each other) are symmetric about six axes of the fifth order, ten axes of the third order, and fifteen axes of the second order, corresponding to 60 group elements. Additionally, except tetrahedra, all other four regular polygons are central symmetric, indicating that C i is their common symmetric group.

3.3 Equivariant GNNs on symmetric graphs

We now demonstrate that equivariant GNNs on symmetric graphs will encounter the issue of expressivity degeneration. Here, we assume that the graph functions we explore are invariant to the permutation of the nodes. This fits the case when we add a readout layer to all nodes globally or just focus on the message passing process for each node individually.

We first derive a crucial theorem that greatly facilitates our analyses.

Theorem 3.4. Suppose that f ( l ) is an O(3) -equivariant function on geometric graphs, regarding the group representation ρ ( l ) defined in Eq. (2) . Then, for any symmetric graph G induced by the group H ≤ O(3) , namely, ∀G ∈ G ( H ) , we always have

Here we have defined group average as ρ ( l ) ( H ) := 1 | H | ∑ h ∈ H ρ ( l ) ( h ) .

Eq. (3) is interesting and it shows that the function f ( l ) is symmetric with respect to the group average ρ ( l ) ( H ) . More importantly, it indicates an linear equation ( I 2 l +1 -ρ ( l ) ( H ) ) f ( l ) ( G ) = 0 , where I 2 l +1 ∈ R (2 l +1) × (2 l +1) is the identity matrix. We can immediately attain the following conclusion.

Theorem 3.5. If and only if the matrix I 2 l +1 -ρ ( l ) ( H ) is non-singular, the O(3) -equivariant function f ( l ) is always a zero function on G , namely,

A more general version of Theorem 3.5 is that the output space of f ( l ) corresponds to the null space of the matrix I 2 l +1 -ρ ( l ) ( H ) , indicating that dim( f ( l ) ) = (2 l +1) -rank ( I 2 l +1 -ρ ( l ) ( H ) ) . Therefore, even the function f ( l ) ) will not exactly reduce to a zero function when I 2 l +1 -ρ ( l ) ( H ) is singular, its output space is still limited to a subspace and suffers from diminished expressivity owing to the symmetry of the input geometric graph.

In practice, it is difficult to determine if the matrix I 2 l +1 -ρ ( l ) ( H ) is singular. This determination becomes easier if we can show that the group average ρ ( l ) ( H ) is equal to the zero matrix. Fortunately, we have the following property.

Theorem 3.6. For a finite group H with its representation ρ ( l ) , ρ ( l ) ( H ) is a zero matrix ( i.e. , ρ ( l ) ( H ) = 0 ) if and only if tr( ρ ( l ) ( H )) = 0 . In this case, f ( l ) ( G ) ≡ 0 , ∀G ∈ G ( H ) .

According to Theorem 3.6, we calculate the trace of the group average for each symmetric graph of interest and check if the trace is equal to zero. We summarize the conclusions for k -fold structures and regular polyhedra in Table 1. We find that when l = 1 , f (1) ≡ 0 for all cases. In addition, the function degenerates when l is odd, if the symmetric graph is induced by the inverse group C i . We defer more details of the calculations in the Appendix. Compared to the conclusions drawn by the GWL paper [35] which only experimentally discusses the k -fold structures under 2D rotations, here we apply rigorous theoretical derivations to analyze more cases besides k -folds, regarding more symmetric subgroups of O(3) .

4 The Proposed HEGNN

The analyses in the last section imply the necessity of involving the representations with more and higher degrees in equivariant GNNs. Therefore, we propose HEGNN by further conducting the update of high-degree steerable features upon EGNN [1]. As illustrated in Fig. 2, HEGNN is composed of the three key components: initialization of high-degree steerable features, calculation of cross-degree invariant messages, and aggregation of neighbor messages, the latter two of which are conducted over multiple layers. We depict each component separately in what follows.

Initialization of high-degree steerable features. Given a geometric graph G ( H , ⃗ X ; A ) where each node contains only type-0 feature h i and type-1 feature ⃗ x i , we first obtain the initialization

h

ij

∆

h

i

/vector

x

ij

∆

/vector

x

i

Concat

MLP

Agg

Agg

Norm

Scalar

Mult.

/vector

x

ij

/vector

x

ij

˜

v

(

l

)

ij

∆˜

v

(

l

)

ij

Sph Embedding

Tensor Product

Agg

h

ij

∆

h

i

/vector

x

ij

∆

/vector

x

i

˜

v

(

l

)

i

˜

v

(

l

)

j

∆˜

v

(

l

)

i

Concat

MLP

Agg

Agg

Norm

Scalar

Mult.

Inner

Product

Tensor

Sub.

Scalar

Mult.

Agg

h

0th Degree

/vector

x

1st Degree

˜

v

(

l

)

Op /

Net

Operator / Network

(a) EGNN

(b) TFN

(c) HEGNN

of high-degree steerable features { ˜ v ( l ) i } L l =0 by using spherical harmonics on normalized relative coordinates. In detail, we aggregate spherical harmonics from all neighbors as

where m ij, init = φ m , init ( h i , h j , e ij , d 2 ij ) is an invariant scalar with φ m , init being an arbitrary MultiLayer Perceptron (MLP), and N ( i ) denotes the neighbors of i 4 .

Calculation of cross-degree invariant messages. EGNN [1] employs a scalarization trick by transforming the relative coordinate ⃗ x i -⃗ x j (the usage of relative coordinates is for translation invariance) into an invariant scalar via the vector norm, which will be used to compute invariant message for both node features and coordinates. We generalize this scalarization trick to the case of high-degree steerable features. To be specific, we carry out the inner product between ˜ v ( l ) i and ˜ v ( l ) j for each degree l individually, resulting in an invariant scalar z ( l ) ij . Then, we get the invariant message between node i and j , namely m ij after undergoing an MLP of all invariant quantities. The above processes are summarized as follows:

where ⊕ refers to concatenation. It should be noted that the form of SO3KRATES introduced in the concurrent work [42] is equivalent to the expression for z ( l ) ij in Eq. (6). Furthermore, our scalarization trick simplifies the formulation by bypassing the Clebsch-Gordan coefficients, making it more straightforward and easier to understand.

Aggregation of neighbor messages. With the invariant message m ij at hand, we then update h i , ⃗ x i , ˜ v ( l ) i via message aggregation over all neighbors. We define ∆ h i , ∆ ⃗ x i , ∆˜ v ( l ) i as the residues, which are calculated by:

where φ h , φ ⃗ x , φ ( l ) ˜ v are different MLPs, and φ ⃗ x , φ ( l ) ˜ v both output a 1D scalar. Note that the application of Eq. (8) for all degrees can be compactly rewritten as ⊕ L l =0 ∆˜ v ( l ) i = 1 |N ( i ) | ∑ j ∈N ( i ) 1 ⊗ φ ˜ v ( m ij ) cg ( ⊕ L l =0 ( ˜ v ( l ) i -˜ v ( l ) j )) in the form of CG tensor product with the weights φ ˜ v ( m ij ) := ⊕ L l =0 φ ( l ) ˜ v . This form can be easily implemented using existing libraries such as e3nn.o3.FullyConnectedTensorProduct [39]. The resulting residues are used for the update:

In addition, we can augment the update of ⃗ x i with 1st-degree feature ˜ v (1) i , leading to ⃗ x i = ⃗ x i + ∆ ⃗ x i + ϕ (1) ˜ v ( h i )˜ v (1) i , which yet is not explored in our experiments for the sake of simplicity. The final output of h i and ⃗ x i can be used for the node-level invariant prediction and equivariant prediction, respectively. We can also obtain a graph-level prediction by further adding a readout layer of all nodes.

We now analyze the expressivity of HEGNN. Apparently, by including high-degree features, HEGNN is able to avoid the loss of expressive ability even on symmetric graphs. Moreover, when tackling general geometric graphs, HEGNN is capable of characterizing the complete angle information of the input graph, if its maximal degree L is sufficiently large. For concision and without losing the generality, we assume the steerable features ˜ v (1) i are initialized with only spherical harmonics without the weights φ ( l ) ˜ v , init in Eq. (5). Let ⃗ x is = ( ⃗ x i -⃗ x s ) / ∥ ⃗ x i -⃗ x s ∥ , the inner product z ( l ) ij can be expanded as follows

where P ( l ) : R → R is Legendre polynomial of degree l , and Eq. (10) is based on the properties of spherical harmonics that ⟨ Y ( l ) ( ⃗ x ) , Y ( l ) ( ⃗ y ) ⟩ = 4 π/ (2 l +1) · P ( l ) ( ⟨ ⃗ x , ⃗ y ⟩ ) , ∥ ⃗ x ∥ = ∥ ⃗ y ∥ = 1 . We have the following result.

Theorem 4.1. For any geometric graph, there exists a bijection between the set of inner products { z ( l ) ij } | A ij | l =1 given by Eq. (10) and the set of edge angles A ij = { θ is,jt := arccos ⟨ ⃗ x is , ⃗ x jt ⟩} s ∈N ( i ) ,t ∈N ( j ) .

Theorem 4.1 states that the inner products of full degrees can recover the information of all angles between each pair of edges, affirming the enhanced expressivity of our HEGNN. The proof is derived mainly based on the fact that Legendre polynomials are orthogonal polynomial bases which can injectivly represent the set A ij thanks to Newton's identities. The details are deferred to the appendix. Although the upper-bound of the degree in Theorem 4.1 grows rapidly with the graph size, it will be shown in our experiments that HEGNN with only L ≤ 6 is sufficient to outperform traditional models like EGNN [1] and TFN [19] in practice.

5 Experiment

5.1 Expressivity on Symmetric Graphs

Design of experiments: To experimentally verify the conclusion we proved above, we design a more comprehensive experiment based on code 5 in [35]. This experiment uses four k -fold structures ( k ∈ { 2 , 3 , 5 , 10 } ) and five convex regular polyhedra shown in Fig. 1 as test objects, and the center of each is at the origin. In detail, an arbitrary rotation in 3 D is acted on such symmetric structures called G 0 which ensures the geometric graph after rotation called G 1 does not coincide with the original one. The goal of our experiments is to check whether different equivariant neural networks can distinguish G 0 and G 1 .

The models we select include two models that only use Cartesian coordinates: EGNN and GVPGNN; and two models that use high-degree steerable features: TFN and MACE. However, TFN and MACE (denoted as TFN/MACE l ≤ L ) always use all degrees l ∈ { 0 , . . . , L } , so it is not clear which

degree(s) of steerable features distinguish the two geometric graphs. In our HEGNN, all steerable features corresponding to unwanted degrees could be masked during initialization in Eq. (5), and we let HEGNN l = L be a HEGNN with only l th-degree steerable features. Additionally, to align with TFN/MACE, we also test the performance of HEGNN with all l ∈ { 0 , 1 , . . . , L } donated as HEGNN l ≤ L . Following the settings 6 in [35], the output of each graph is the concatenation of invariant scalars, coordinates, and high-degree steerable features pooling among all nodes. We then map this spliced vector to a two-dimensional vector and input it into a simple classifier to determine whether the equivariant graph neural network can distinguish G 0 from G 1 .

Results: The results on k -fold are deferred to Appendix for saving space, and the results on regular polyhedra are shown in Table 2. From Table 1, we can know steerable features in which degree could not distinguish specific symmetry structure, and both results on k -fold and regular polyhedra are also in perfect agreement with our conclusions. Models (EGNN and GVP-GNN) only with Cartesian vectors cannot distinguish any symmetric graph at all. Taking HEGNN l =5 as an example, since D (5) ( H ) = 0 , ∀ H ∈ { T, O, I } , no matter which kind of regular polyhedron, f (5) could only output 0 thus failing to distinguish the structures.

5.2 Physical Dynamics Simulation

Datasets: We benchmark our HEGNN in two scenarios, including: N -body system [43] is a dataset generated from simulations. In our simulations, each system contains N charged particles with random charge c i ∈ { 0 , 1 } , whose movements are driven by Coulomb forces. To verify the efficiency and effectiveness of our HEGNN on datasets of different sizes, we select N from { 5 , 20 , 50 , 100 } . We use 5000 samples for training, 2000 for validation, and 2000 for testing. The task is to estimate the positions of the N particles after 1,000 timesteps. MD17 [44] dataset contains trajectory data for eight molecules generated through molecular dynamics simulations. The goal of this experiment is to predict the future positions of the atoms based on their current state. We follow the dataset partitioning scheme from [45], splitting the dataset into 500/2000/2000 frame pairs for training, validation and testing, respectively. All experiments are run on a single NVIDIA A100-80G GPU.

Baselines: To demonstrate the advantages of our HEGNN over both models with scalarization techniques and models with high-degree steerable vectors at the same time, our baseline needs to consider the selection issues of both models simultaneously. Therefore, we select some representative models as baselines, including the invariant RF [46], the equivariant EGNN [1], TFN [19] and SE(3)Tr. [20]. In addition, we select classical models such as Linear dynamics [1], the non-equivariant

Message Passing Neural Network (MPNN) [47], the invariant SchNet [48], and the equivariant GVP-GNN [49] for the N -body experiments. For MD17 experiments, we also select GMN [45].

Metrics: 1. Loss function: We use Mean Squared Error (MSE) to measure the accuracy of the prediction results in both experiments. 2. Inference time: Given that the N -body system we use contains data of varying sizes, we test the inference time of each model on this dataset. The inference time for each model is calculated relative to the benchmark, which is the inference time of EGNN at the corresponding scale.

Results on N -Body systems: The main results of N -body system simulation are presented in Table 3. From these results, we observe the following: 1. Overall performance : Our HEGNN significantly outperforms other models across datasets of all sizes. Although EGNN [1] performs better than high-degree steerable models like TFN or SE(3) -Transformer in this task, our HEGNN is still better than EGNN, which show that the method of HEGNN introducing high-degree steerable features is more effective. 2. Stability : Although the performance of the model (HEGNN l ≤ 6 ) using high-degree steerable features declines slightly when the geometric graph is small, overall, HEGNN performs better than other models. 3. Inference time : Our model's inference time is significantly faster than that of high-degree steerable models like TFN, reflecting the simplicity and efficiency of our HEGNN.

Results on MD17: The main results of MD17 experiment are shown in Table 4, with some data sourced from [45]. From these results, we draw the following insights: 1. Overall performance: Our HEGNN outperforms other models on six out of eight molecules. The effect on the remaining two molecules is only not as good as GMN [45] and this is because GMN introduces additional knowledge such as chemical bonds. 2. Advantage of high-degree vectors: Most of the best results are obtained on HEGNN l ≤ 6 , indicating that the use of high-degree steerable features can enhance model expression capabilities.

5.3 Perturbation Experiment

In practical scenarios, slight perturbations (such as molecular vibrations) can disrupt strict symmetry, potentially mitigating the conclusions outlined in Theorems 3.5 and 3.6. We therefore designed this perturbation experiment for a simple study and were surprised to find that HEGNN can still bring better robustness through the introduction of high-degree steerable features.

Design of experiments: We take the tetrahedron as an example and compare the cases of EGNN, HEGNN l =3 , and HEGNN l ≤ 3 when adding noise perturbations with results in Table 5. Here, ε represents the ratio of noise, and the modulus of the noise obeys N (0 , ε · E [ ∥ ⃗ x -⃗ x c ∥ ] · I ) .

Results: It can be observed that the performance of EGNN is slightly improved in the presence of noise (from 50% when ε = 0 . 01 to 60% when ε = 0 . 5 ), while HEGNN demonstrates better robustness. Even though symmetry-breaking factors will make the geometric graph deviate from the symmetric state, the deviated graph is still roughly symmetric. In other words, the outputs of equivariant GNNs on the derivated graphs keep close to zero if the degree value is chosen to be those in Table 1, which will still lead to defective performance.

6 Conclusion

In this paper, we challenged the prevailing notion that higher-degree steerable vectors are unnecessary for achieving expressivity in equivariant Graph Neural Networks (GNNs). Through rigorous theoretical analysis, we demonstrated that equivariant GNNs constrained to 1st-degree representations inevitably degenerate to zero functions when applied to symmetric structures, such as k -fold rotations and regular polyhedra. To address this limitation, we introduced HEGNN, a high-degree extension of the EGNN model. HEGNN enhances expressivity by integrating higher-degree steerable vectors while retaining the efficiency of the original model through a scalarization technique. Our extensive empirical evaluations on various datasets, including the symmetric toy dataset, N -body, and MD17, validate our theoretical predictions. HEGNN not only adheres to our theoretical insights but also exhibits significant performance improvements over existing models. These findings underscore the critical role of higher-degree representations in fully leveraging the potential of equivariant GNNs.

7 Acknowledgment

This work was jointly supported by the following projects: the National Science and Technology Major Project under Grant 2020AAA0107300, the National Natural Science Foundation of China (No. 62376276, No. 62172422); Beijing Nova Program (No. 20230484278); Beijing Outstanding Young Scientist Program (No. BJJWZYJH012019100020098), the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (23XNKJ19); Public Computing Cloud, Renmin University of China.

References

Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning , pages 9323-9332. PMLR, 2021.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.

Rui Jiao, Xiangzhe Kong, Ziyang Yu, Wenbing Huang, and Yang Liu. Equivariant pretrained transformer for unified geometric learning on multi-domain 3d molecules. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design , 2024.

Jun Wu, Xiangzhe Kong, Ningguan Sun, Jing Wei, Sisi Shan, Fuli Feng, Feng Wu, Jian Peng, Linqi Zhang, Yang Liu, et al. Better prior distribution for antibody design. Available at SSRN 4909414 , 2024.

Thorben Frank, Oliver Unke, and Klaus-Robert Müller. So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems. Advances in Neural Information Processing Systems , 35:29400-29413, 2022.

Yusong Wang, Tong Wang, Shaoning Li, Xinheng He, Mingyu Li, Zun Wang, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Enhancing geometric representations for molecules with equivariant vector-scalar interactive message passing. Nature Communications , 15(1):313, 2024.

Runfa Chen, Jiaqi Han, Fuchun Sun, and Wenbing Huang. Subequivariant graph reinforcement learning in 3d environments. In International Conference on Machine Learning , pages 45454565. PMLR, 2023.

Runfa Chen, Ling Wang, Yu Du, Tianrui Xue, Fuchun Sun, Jianwei Zhang, and Wenbing Huang. Subequivariant reinforcement learning in 3d multi-entity physical environments. In Forty-first International Conference on Machine Learning , 2024.

Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. Advances in Neural Information Processing Systems , 35:26256-26268, 2022.

Liming Wu, Zhichao Hou, Jirui Yuan, Yu Rong, and Wenbing Huang. Equivariant spatiotemporal attentive graph networks to simulate physical dynamics. Advances in Neural Information Processing Systems , 36, 2024.

Jiaqi Han, Minkai Xu, Aaron Lou, Haotian Ye, and Stefano Ermon. Geometric trajectory diffusion models. arXiv preprint arXiv:2410.13027 , 2024.

Yuxuan Song, Jingjing Gong, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei-Ying Ma. Unified generative modeling of 3d molecules with bayesian flow networks. In The Twelfth International Conference on Learning Representations , 2023.

Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning , pages 38592-38610. PMLR, 2023.

Yanru Qu, Keyue Qiu, Yuxuan Song, Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei-Ying Ma. Molcraft: Structure-based drug design in continuous parameter space. In Forty-first International Conference on Machine Learning , 2024.

Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations , 2022.

Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature , 620(7976):1089-1100, 2023.

John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Ng-Thow-Hing, Erik R Van Vlack, et al. Illuminating protein space with a programmable generative model. Nature , pages 1-9, 2023.

Ziyang Yu, Wenbing Huang, and Yang Liu. Rigid protein-protein docking via equivariant elliptic-paraboloid interface prediction. In The Twelfth International Conference on Learning Representations , 2024.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems , 33:1970-1981, 2020.

Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications , 13(1):2453, 2022.

Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications , 14(1):579, 2023.

Nadav Dym and Haggai Maron. On the universality of rotation equivariant point cloud networks. arXiv preprint arXiv:2010.02449 , 2020.

Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning , pages 8867-8887. PMLR, 2022.

Minkai Xu, Jiaqi Han, Aaron Lou, Jean Kossaifi, Arvind Ramanathan, Kamyar Azizzadenesheli, Jure Leskovec, Stefano Ermon, and Anima Anandkumar. Equivariant graph neural operator for modeling 3d dynamics. arXiv preprint arXiv:2401.11037 , 2024.

Omri Puny, Matan Atzmon, Edward J Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations , 2021.

Alexandre Agm Duval, Victor Schmidt, Alex Hernandez-Garcia, Santiago Miret, Fragkiskos D Malliaros, Yoshua Bengio, and David Rolnick. Faenet: Frame averaging equivariant gnn for materials modeling. In International Conference on Machine Learning , pages 9013-9033. PMLR, 2023.

Yuelin Zhang, Jiacheng Cen, Jiaqi Han, Zhiqiang Zhang, Jun Zhou, and Wenbing Huang. Improving equivariant graph neural networks on large geometric graphs via virtual nodes learning. In Forty-first International Conference on Machine Learning , 2024.

Jiaqi Han, Wenbing Huang, Tingyang Xu, and Yu Rong. Equivariant graph hierarchy-based neural networks. Advances in Neural Information Processing Systems , 35:9176-9187, 2022.

Yusong Wang, Chaoran Cheng, Shaoning Li, Yuxuan Ren, Bin Shao, Ge Liu, Pheng-Ann Heng, and Nanning Zheng. Neural p 3 m: A long-range interaction modeling enhancer for geometric gnns. Advances in Neural Information Processing Systems , 2024.

Jiaqi Han, Jiacheng Cen, Liming Wu, Zongzhao Li, Xiangzhe Kong, Rui Jiao, Ziyang Yu, Tingyang Xu, Fandi Wu, Zihe Wang, et al. A survey of geometric graph neural networks: Data structures, models and applications. arXiv preprint arXiv:2403.00485 , 2024.

Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning , pages 9377-9388. PMLR, 2021.

Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. In International Conference on Learning Representations , 2021.

Johannes Gasteiger, Florian Becker, and Stephan Gunnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems , 34:6790-6802, 2021.

Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Lio. On the expressive power of geometric graph neural networks. In International Conference on Machine Learning , pages 15330-15355. PMLR, 2023.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. nti, Series , 2(9):12-16, 1968.

Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems , 35:11423-11436, 2022.

Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems , 31, 2018.

Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453 , 2022.

David J Griffiths and Darrell F Schroeter. Introduction to quantum mechanics . Cambridge university press, 2018.

Lev Davidovich Landau and Evgenii Mikhailovich Lifshitz. Quantum mechanics: nonrelativistic theory , volume 3. Elsevier, 2013.

J Thorben Frank, Oliver T Unke, Klaus-Robert Müller, and Stefan Chmiela. A euclidean transformer for fast and stable machine learned force fields. Nature Communications , 15(1):6539, 2024.

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning , pages 2688-2697. PMLR, 2018.

Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances , 3(5):e1603015, 2017.

Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant graph mechanics networks with constraints. In International Conference on Learning Representations , 2022.

Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi-body systems with symmetric energies. arXiv preprint arXiv:1910.00753 , 2019.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning , pages 1263-1272. PMLR, 2017.

Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics , 148(24), 2018.

Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations , 2020.

Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning , pages 5583-5608. PMLR, 2022.

James F Epperson. An introduction to numerical methods and analysis . John Wiley & Sons, 2013.

Michael Engel. Point group analysis in particle simulation data. arXiv preprint arXiv:2106.14846 , 2021.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in Neural Information Processing Systems , 30, 2017.

Hannah Lawrence, Vasco Portilheiro, Yan Zhang, and Sékou-Oumar Kaba. Improving equivariant networks with probabilistic symmetry breaking. In ICML 2024 Workshop on Geometrygrounded Representation Learning and Generative Modeling , 2024.

Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molecular graphs. Advances in Neural Information Processing Systems , 35:650-664, 2022.

Kelin Xia and Guo-Wei Wei. Persistent homology analysis of protein structure, flexibility, and folding. International journal for numerical methods in biomedical engineering , 30(8):814-844, 2014.

Floor Eijkelboom, Rob Hesselink, and Erik J Bekkers. E(n) equivariant message passing simplicial networks. In International Conference on Machine Learning , pages 9071-9081. PMLR, 2023.

Claudio Battiloro, Ege Karaismailo˘ glu, Mauricio Tec, George Dasoulas, Michelle Audirac, and Francesca Dominici. E (n) equivariant topological neural networks. arXiv preprint arXiv:2405.15429 , 2024.

Ralf Drautz. Atomic cluster expansion for accurate and transferable interatomic potentials. Physical Review B , 99(1):014104, 2019.

Genevieve Dusson, Markus Bachmayr, Gábor Csányi, Ralf Drautz, Simon Etter, Cas van Der Oord, and Christoph Ortner. Atomic cluster expansion: Completeness, efficiency and stability. Journal of Computational Physics , 454:110946, 2022.

Xiangzhe Kong, Wenbing Huang, Zhixing Tan, and Yang Liu. Molecule generation by principal subgraph mining and assembling. Advances in Neural Information Processing Systems , 35:25502563, 2022.

Xiangzhe Kong, Wenbing Huang, and Yang Liu. Generalist equivariant transformer towards 3d molecular interaction learning. In Forty-first International Conference on Machine Learning , 2023.

Xiangzhe Kong, Wenbing Huang, and Yang Liu. Full-atom peptide design with geometric latent diffusion. Advances in Neural Information Processing Systems , 2024.

Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. Roto-translated local coordinate frames for interacting dynamical systems. Advances in Neural Information Processing Systems , 34:6417-6429, 2021.

Yi-Lun Liao, Brandon M Wood, Abhishek Das, and Tess Smidt. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. In The Twelfth International Conference on Learning Representations , 2024.

Ziqiao Meng, Liang Zeng, Zixing Song, Tingyang Xu, Peilin Zhao, and Irwin King. Towards geometric normalization techniques in se(3) equivariant graph neural networks for physical dynamics simulations. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI) , pages 5981-5989, 2024.

Zian Li, Xiyuan Wang, Yinan Huang, and Muhan Zhang. Is distance matrix enough for geometric deep learning? Advances in Neural Information Processing Systems , 36, 2024.

Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations , 2023.

Weiliang Luo, Gengmo Zhou, Zhengdan Zhu, Yannan Yuan, Guolin Ke, Zhewei Wei, Zhifeng Gao, and Hang Zheng. Bridging machine learning and thermodynamics for accurate p k a prediction. JACS Au , 2024.

Gengmo Zhou, Zhen Wang, Feng Yu, Guolin Ke, Zhewei Wei, and Zhifeng Gao. S-molsearch: 3d semi-supervised contrastive learning for bioactive molecule search. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Fanmeng Wang, Hongteng Xu, Xi Chen, Shuqi Lu, Yuqing Deng, and Wenbing Huang. Mperformer: An se (3) transformer-based molecular perceptron. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages 2512-2522, 2023.

Fanmeng Wang, Wentao Guo, Minjie Cheng, Shen Yuan, Hongteng Xu, and Zhifeng Gao. Mmpolymer: A multimodal multitask pretraining framework for polymer property prediction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , CIKM '24, 2024.

Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations , 2020.

Angxiao Yue, Dixin Luo, and Hongteng Xu. A plug-and-play quaternion message-passing module for molecular conformation representation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 16633-16641, 2024.

Yang Zhang, Wenbing Huang, Zhewei Wei, Ye Yuan, and Zhaohan Ding. Equipocket: an e (3)-equivariant geometric graph neural network for ligand binding site prediction. In Forty-first International Conference on Machine Learning , 2024.

Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu. Energy-motivated equivariant pretraining for 3d molecular graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 8096-8104, 2023.

Ziyang Yu, Wenbing Huang, and Yang Liu. Force-guided bridge matching for full-atom time-coarsened dynamics of peptides. arXiv preprint arXiv:2408.15126 , 2024.

Qi Li, Rui Jiao, Liming Wu, Tiannian Zhu, Wenbing Huang, Shifeng Jin, Yang Liu, Hongming Weng, and Xiaolong Chen. Powder diffraction crystal structure determination using generative models. arXiv preprint arXiv:2409.04727 , 2024.

Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion. Advances in Neural Information Processing Systems , 36, 2024.

Rui jiao, Xiangzhe Kong, Wenbing Huang, and Yang Liu. 3d structure prediction of atomic systems with flow-based direct preference optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations , 2022.

Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full-atom antibody design. In Proceedings of the 40th International Conference on Machine Learning , pages 17409-17429, 2023.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

A Theoretical Details

A.1 Equivariance/Invariance of HEGNN

In this section, we demonstrate the equivariance of our HEGNN. In order to further illustrate the connection between our HEGNN, EGNN, and TFN, a general proof is given here.

Table 6: Comparison between our HEGNN and the scalarization-based model representing EGNN [1], and the high-degree steerable model representing TFN [19]. HEGNN combines the scalarization trick of EGNN that only uses invariant scalars (0th degree steerable features) to interact between steerable features corresponding to different degrees, avoiding the high computational cost of using CG tensor products in TFN.

Theorem A.1 (Equivariance/Invariance of HEGNN) . h i , ˜ v (0) i in HEGNN is E(3) invariant, ⃗ x i is E(3) equivariant. In addition, all ˜ v ( l ) i are O (3) equivariant and translation invariant when l is odd; all ˜ v ( l ) i is SO (3) equivariant and inversion/translation invariant to when l even.

Proof. Consider a sequence composed of functions { φ i : X ( i -1) →X ( i ) } N i =1 equivariant to a same group H , the equivariance lead to an interesting property that

holds for all i, j ∈ { 1 , 2 , . . . , N } and h ∈ H , which means that the group elements h can be freely exchanged in the composite sequence of equivariant functions. In particular, if one of the equivariant functions ( e.g. φ k ) is replaced by an invariant function, the group element h will be absorbed, which means

holds for all h ∈ H but only i ∈ { 1 , 2 , . . . , k } . Although φ N ◦ · · · ◦ φ k is still equivariant, because the group elements must be input starting from φ 1 , the overall φ N ◦ · · · ◦ φ 1 is still an invariant function. That is to say, to conclude that the entire HEGNN is equivariant, we only need to prove that HEGNN is equivariant in initialization and each layer.

The initialization of HEGNN is based on spherical harmonics, which is similar to TFN. Spherical harmonics are inherently equivariant, that is,

Note that variables participating in the coefficient in Eq. (5) are all invariant scalars, so the initialization of HEGNN is consistent with the spherical harmonic function. Note that for Cartesian vectors, they can be aligned by arranging the spherical harmonics of 1st degree [19], that is,

From this perspective, EGNN can also be considered to be initialized using spherical harmonics, but this step is omitted because the value is proportional to the input Cartesian vector.

It is worth explaining that spherical harmonics are inversion invariant when l is even, that is,

Equivariant ones are not necessarily better than invariant ones. When we need to predict pseudovectors (such as moments), we need to inversion invariant 1st-degree steerable features, because pseudovectors are inversion invariant. This is why introducing the cross product ⃗ x × ⃗ y (the result is inversion invariant) into a linear combination can only build a SE(3) equivariant network, but not a E(3) network [50].

In fact, inversion equivariant/invariant high degree steerable features can be obtained by calculating the CG tensor product of spherical harmonics and inversion equivariant Cartesian vectors like ⃗ x i -⃗ x c [39]. Moreover, the equivariance at each layer is also easy to prove and the internal Wigner-D matrix can be extracted through the CG tensor product [31].

When the output result is an invariant scalar, the Wigner-D matrix degenerates into a trivial representation 1 . Norm and inner product are all special cases of this type, so equivariance is established. From this perspective, EGNN and HEGNN are equivalent to using only the weight coefficients of ( l, l ) → 0 and (0 , l ) → l . Similar ideas include the steerable MLPs in [33] .

A.2 Other Proofs

Theorem A.2 (Theorem 3.4) . Suppose that f ( l ) is an O(3) -equivariant function on geometric graphs, regarding the group representation ρ ( l ) defined in Eq. (2) . Then, for any symmetric graph G induced by the group H ≤ O(3) , namely, ∀G ∈ G ( H ) , we always have

Here we have defined group average as ρ ( l ) ( H ) := 1 | H | ∑ h ∈ H ρ ( l ) ( h ) .

Proof. If f ( l ) is O(3) -equivariant, then it is also a H -equivariant, thus

Theorem A.3 (Theorem 3.5) . If and only if the matrix I 2 l +1 -ρ ( l ) ( H ) is non-singular, the O(3) -equivariant function f ( l ) is always a zero function on G , namely,

Proof. From Theorem 3.4, we know that

and the theorem holds for basic knowledge of linear algebra.

Theorem A.4 (Theorem 3.6) . For a finite group H with its representation ρ ( l ) , ρ ( l ) ( H ) is a zero matrix ( i.e. , ρ ( l ) ( H ) = 0 ) if and only if tr( ρ ( l ) ( H )) = 0 . In this case, f ( l ) ( G ) ≡ 0 , ∀G ∈ G ( H ) .

Proof. It is obvious that ρ ( l ) ( H ) = 0 = ⇒ tr( ρ ( l ) ( H )) = 0 since all elements are zero not to mention the main diagonal, and we only to prove tr( ρ ( l ) ( H )) = 0 = ⇒ ρ ( l ) ( H ) = 0 .

A basic fact is h · H = H , thereby

Now we can use group average and get

Such operation can be repeated many times, so that

Now we calculate the trace for each matrix and find

By Newton's identity [51], all eigenvalues of the matrix ρ ( H ) are 0, that is, the matrix is a zero matrix.

Table 7: The traces of symmetric groups based on [52]. Trace for polyhedral groups can be calculated by D ( l ) ( H ) = ⌊ l/r ⌋ + b [ l mod r ] with repeat length r , where b is a string only with 0 or 1. For exmaple, for Tetrahedral group T , D (5) ( T ) = ⌊ 5 / 6 ⌋ + b [5 mod 6] = 0 + 0 = 0 .

Theorem A.5 (Theorem 4.1) . For any geometric graph, there exists a bijection between the set of inner products { z ( l ) ij } | A ij | l =1 given by Eq. (10) and the set of edge angles A ij = { θ is,jt := ⟨ ⃗ x is , ⃗ x jt ⟩} s ∈N ( i ) ,t ∈N ( j ) .

Proof. Note that the Legendre polynomial is a set of orthogonal polynomial bases, and there is a bijection to the power function polynomial space, that is

where M is any non-negative integer represents the degree of the polynomial space. Moreover, from the knowledge of Newton's identities, the space of power sums can be converted to space of elementary symmetric polynomials as

From Vieta's formulas, when M = | A ij | , with the M +1 polynomial in the space of elementary symmetric polynomials being coefficients, we can build a | A ij | -degree polynomial with { cos θ | θ ∈ A ij } as its all roots 7 . Since all angles are in [0 , π ) , the cosine uniquely determines the angle value, and the proposition is established.

A.3 Further Discussion

Our theory in fact shows that the degeneration of global features of a certain degree (in Table 1) are inevitable on symmetric geometric graphs. This raises two points worth discussing:

The degree not indicated to degenerate not necessarily produce a non-zero representation, which may still be affected by the model form and the edge situation.

There are some tricks to get around this degeneration: for example, making the output a set or relaxing equivariance constraints ( e.g. probabilistic symmetry breaking [54]).

It is worth mentioning that outputting a set can solve most problems, although such operators may be quite intractable to implement in computer systems. The failure cases of frame averaging [26; 27] and Neural P 3 M[30] which depend on singular value decomposition or eigenvalue decomposition, is caused by the non-unique matrix decomposition. Some other examples include ComENet [55], which uses the scatter_min() operator in PyTorch to extract the nearest neighbors, making it intractable to handle the situation where multiple neighbors are simultaneously closest, which is quite common in chemical molecules ( e.g. -CH 3 , -NH 2 ).

Moreover, from Theorem 4.1, we show the expressivity of our HEGNN, which is able to recover the information of all angles between each pair of edge. However, it should be noted that | A ij | may be an extremely large number, which is unacceptable in practical applications to achieve completeness. The same problem also arises in discussions based on CG tensor product models ( e.g. TFN [19]), such as discussions based on D -spanning [23], because a sufficiently high-degree D is unacceptable. However, in terms of actual results, the performance of both our HEGNN and TFN is remarkable. From this perspective, how to bridge the gap between completeness and actual performance with features of limited channels and degrees is a question worth considering.

To get the ball rolling, we raise an interesting question here. Is there a performance gap between this type of purely mathematical representation and other features based on physical and biochemical prior knowledge?

Purely mathematical representation: topological characteristics [56-58], cluster expansion basis [59; 60], subgraph blocks [61-63], frames [26; 27; 64], normalization operators [65; 66];

Features based on physical and biochemical prior knowledge: distance matrix [67-72], chemical bond length, angle and dihedral angle [55; 73-75], force [76; 77], fractional coordinates [78-80], canonical ordering [81; 82].

In fact, some of these features are directly related ( e.g. frames and fractional coordinates). How to construct effective and interpretable pure mathematical features based on those with prior knowledge will become a key point in network design, and we will consider further exploration in future work.

B More Experimental Details and Results

B.1 Comparison of parameters between models

Like EGNN [1], different features use different numbers of channels, so the inference time does not obviously reflect the time complexity of O ( L 2 ) . We list the details of our HEGNN of different degree in Table 8. Intuitively, we add one of each steerable feature on the basis of EGNN (using 64 invariant scalars and 2 Cartesian vectors, i.e. coordinate and velocity).

Table 9 shows the number of parameters and inference time of different models.

B.2 Experiment on k -fold Structure.

The results on the k -fold structure are completely consistent with the conclusion of Table 1. The few results that cannot reach 100% are due to the small number of training epochs, so the classifier fails to perform perfect classification. In fact, the accuracy of models can achieve 100.00± 0.0 after increasing the number of training rounds of the model like 500 epochs. Since from 2nd-degree steerable features can distinguish G 0 and G 1 , and HEGNN/TFN/MACE l ≤ L contain 2nd-degree steerable features when l ≥ 3 , the extra results are hidden but all their values are 100.00± 0.0.

B.3 Further Verification on Regular Polyhedra

Implementation details: Since the e3nn [39] library only implements spherical harmonics up to the 11 th-degree, we verified the expressivity of our HEGNN in disguise. In this experiment, we measure the expressivity of our HEGNN by calculating whether the high-degree feature ˜ v ( l ) i updates on regular polyhedra. Namely, if ∆˜ v ( L ) i = 0 , then HEGNN l = L loses the expressivity. This judgment is necessary and sufficient. We calculated the sum of all degree vectors from L = 1 to 30 , which

covers the maximum repeat length r in Table 7. We implemented the calculation of the high-degree features based on scipy [83] library.

Results: Our calculation results are shown in Table 11 which is completely consistent with the theoretical results in Table 7. The simple experiment shows that the sum of spherical harmonics ∑ i Y ( l ) ( ⃗ x i -⃗ x c ) , as a function on graph, will actually vanish on regular polyhedra for some integer degree l .

B.4 Results on other dataset settings

The N -body dataset setting of this paper refers to FastEGNN [28] 8 , that is, 5,000 samples are used as the training set (instead of 3,000). We tested the situation of using different data segmentation in Table 12. We also add ClofNet [50], a local frame based scalarization method and SEGNN [33] (select l = 1 according to the optimal situation in the paper) and MACE [37] ( l = 2 ), two classic high-degree steerable models for comparison.

Note that SEGNN in Table 12 does not show the effect in the original paper. We also tried to reproduce the dataset in the original paper [33], and the effect of SEGNN on 5-body dataset can be reproduced. However, see Table 13, it is also shown that SEGNN performs poorly on larger datasets.

We found that the GMN-L method proposed in [29] generally performs the best on MD17 dataset, but our HEGNN-6 also achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance.

B.5 Expressiveness of initialization layer

Note that both EGNN [1] and HEGNN l ≤ 1 only use Cartesian vectors. However, in Table 3, the effect of the latter is greatly improved. We speculate that there are two possible factors for this improvement: 1) the extra layer of message passing brought in by Eq. (5); 2) the multi-channel of Cartesian vectors (see Table 8). Therefore, we tested the effect of HEGNN l ≤ 1 -3 layers, and the results are shown in Table 15. From the improvement, the first factor contributes more.

C Limitation

Our current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. Whether our HEGNN is effective on large-scale geometric graph data sets remains to be verified.

D Broader Impact

Our research belongs to the field of AI for Science. The HEGNN proposed in this article is expected to better model scientific problems, thereby promoting the development of higher-precision and efficient AI scientific models.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction nicely summarize the theoretical contributions mentioned in § 3, the model design mentioned in § 4, and the experimental results mentioned in § 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. Whether our HEGNN is effective on large-scale geometric graph data sets remains to be verified.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our main theoretical results are given in § 3 and the proofs are given in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details of our experimental results are in "Design of experiments" in § 5.1 and "Datasets" in § 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: https://github.com/GLAD-RUC/HEGNN .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details have listed in "Design of experiments" in § 5.1 and "Datasets" in § 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the variance of the model runs in Table 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use NVIDIA A100-80G, which has written in § 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our research belongs to the field of AI for Science. The HEGNN proposed in this article is expected to better model scientific problems, thereby promoting the development of higher-precision and efficient AI scientific models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used in our paper have used with citation.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: https://github.com/GLAD-RUC/HEGNN .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present work pertains to analysing and improving Graph Neural Networks operating on geometric graphs that possess E(3) (Euclidean) symmetries, i.e. the tasks of interest are invariant/equivariant to rotations, translations and reflections. The paper challenges the assumption that it is sufficient to use E(3)-equivariant features of 1st degree (the term “degree” is related to the dimensionality of the group representation to which the features are equivariant - 1st-degree features are 3-dimensional and are equivariant to 3D rotations/reflections), as done in EGNN, Satorras et al., ICML’21, which works quite well in practice and is computationally efficient. The authors construct a series of counterexample symmetric geometric graphs (i.e. graphs which, when rotated by certain angles, remain intact up to permutations of their vertices) which are shown to be challenging for geometric GNNs. It is shown that, for certain degrees $l$, a GNN can only produce output features equal to zero, therefore showing that the degree is crucial for expressivity and that $l=1$ is insufficient for some cases. To address this limitation, they combine ideas from EGNN and TFN, Thomas et al., arxiv’18: They create a GNN, similar to EGNN, enriched with auxiliary features of higher degree, using spherical harmonics (similar to TFN). To keep the computational complexity low, they produce the messages used for the updates of node features/positions/auxiliary features via scalarisation using inner products, simplifying the messages used in TFN. The expressivity of the method is theoretically analysed, showing the ability to recover angles, and empirically tested, showing improved performance in several synthetic and real-world benchmarks.\n\nSTRENGTHS:\n**Significance**. The studied problem is important for various physical sciences using data-driven methods and finding better trade-offs between performance and computational complexity is key to their progress. The present paper improves this trade-off both theoretically and empirically. \n\n**Novelty**.\n  - *Theoretical results*. The underlying theory illustrating the expressivity barriers of low-degree representations (theorems 3.4-3.6) is simple and concise, yet insightful and of independent technical interest (can be used for other types of symmetric objects to find out properties of neural representations and illustrate limitations of modelling choices). Additionally, theorem 4.1. backs the proposed method with theoretical arguments, hinting that is probably a good trade-off between expressivity and computational complexity (although the former is not completely characterised).\n   - *Method*. As far as I know, the proposed approach to incorporate higher-degree representations into message-passing is original. Additionally, it is easy to understand and implement, which may facilitate reproducibility.\n\n**Presentation**. Apart from a few exceptions (see weaknesses), the paper is well-explained, with carefully chosen notation and progressive illustration of the different results (from the theoretical motivation to the model improvement and empirical evaluation).\n\n**Empirical results & Execution**.  The experimental outcomes are in agreement with the theory on synthetic data, while the method is shown to generalise better than its competitors in real-world data, with additional computational efficiency advantages. Overall, this work is well-executed: starting from a characterisation of the limitations of existing works, then proposing a remedy (that gets the best of both worlds from two different families of methods) and finally testing it both on the counterexamples, as well as in real-world scenarios, validating the theory.\n\nWEAKNESSES:\n**Clarity: limits audience and creates some issues with contextualization to related work**. Although not a major weakness, there are some issues with the clarity of the presentation, which I think should be addressed to improve the exposition of the paper to a wider audience. \n  - In the introduction and the related work section, the authors mention several concepts without providing explanations which might make the text hard to follow, especially for readers not experts in geometric GNNs and Euclidean symmetries. \n    - For instance: *x-degree steerable representations* - this is a notion at the heart of the paper, so it should probably be intuitively explained early on, e.g. by summarising the formal definitions at the end of section 3.1.\n    - *Clebsch-Gordan tensor product* - this concept is not explained in the paper. However, I do not think that the authors should assume familiarity with it. The reader needs to obtain a clear understanding due to the following: the main methodological innovation of the present work is to replace this operation with inner products to improve computational efficiency. I think that the authors should explain in a separate subsection the differences between these two operators and discuss what they sacrifice by not incorporating CG products in their architecture.\n-\tExamples 3.2. and 3.3. need further elaboration as they are the main motivation behind the proposed methodology. E.g. why do even folds are not symmetric w.r.t. the cyclic group? Perhaps explain what the dihedral group is. Why are regular polygons symmetric w.r.t. the dihedral group?\n\nQUESTIONS:\n- Section 3.1/Section 4: It is unclear to me why the authors introduced the modulation of the spherical harmonics (re-scaling). How does this affect expressivity?\n- Section 3.1: Is the representation of O(3) – Eq. (2) – conventional or introduced here for the first time? In the first case, could the authors provide a reference? In the second case, it would be more appropriate to precisely show that this is indeed a representation for completeness.\n- As far as I understand, Eq. (3) refers to graph-level features. Perhaps this should be explicitly mentioned to improve readability.\n- What information is lost from the scalarization trick of Eq. (6) compared to CG tensor products? To what extent does this limit expressivity?\n- How do the authors explain that in certain cases in Table 2, HEGNN does not achieve 100% performance (although it should according to the theory)? Additionally, why is the std so high (15% and 30%)?\n- In table 3, it seems that HEGNN with $l\\leq 1$ always outperforms EGNN, which is also 1st-degree. How do the authors explain this?\n- It is unclear to me how equivariance to reflections is handled by this method. How does the discussion at the end of section 3.1 affects the operators used?\n- Table 5 is quite helpful, perhaps consider moving it to the main text. \n\n**Minor**: I have the impression that there were a few typos throughout the text. I suggest performing a proof-reading (examples: L78 typo: university --> universality, L266 type: donated --> denoted).\n\nLIMITATIONS:\nI recommend that the authors devote a separate section to discuss the limitations of their work. Currently, I could locate only a limited discussion in the appendix with regards to not testing in large-scale systems.\n\nI do not foresee any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper challenges the prevailing notion that high-degree steerable representations are unnecessary in equivariant Graph Neural Networks (GNNs). The authors provide theoretical analysis showing that equivariant GNNs constrained to 1st-degree representations degenerate to zero functions when applied to symmetric structures like k-fold rotations and regular polyhedra. To address this limitation, they propose HEGNN, a high-degree extension of the EGNN model that incorporates higher-degree steerable vectors while maintaining efficiency through a scalarization technique. The authors evaluate HEGNN on symmetric toy datasets, N-body systems, and MD17 molecular dynamics datasets, demonstrating improved performance over existing models.\n\nSTRENGTHS:\n1. The paper presents a clear and well-motivated research question, challenging an existing assumption in the field of equivariant GNNs.\n2. The theoretical analysis is thorough and provides valuable insights into the limitations of low-degree representations for symmetric structures.\n3. The proposed HEGNN model offers a promising approach to incorporating high-degree representations while maintaining computational efficiency.\n4. The experimental results on symmetric toy datasets align well with the theoretical predictions, providing empirical support for the main claims.\n5. The paper includes a good discussion of the trade-offs between expressivity and efficiency in equivariant GNN models.\n\nWEAKNESSES:\n1. The experimental comparisons on the N-body system are limited and use different splits and variations compared to existing literature. This makes it difficult to directly compare HEGNN's performance to state-of-the-art methods. Including comparisons with more recent baselines (e.g., ClofNet, GCPNet, SaVeNet) and using standardized benchmarks would strengthen the empirical evaluation.\n2. The experiments are primarily focused on predicting future positions of particles/atoms. While this is a relevant task, it may not fully demonstrate the necessity or advantages of high-degree representations across a wider range of equivariant GNN applications. Additional experiments on different tasks or domains could provide more comprehensive evidence to support the paper's claims.\n3. Some relevant baselines are missing from the comparisons. For example, GMN-L is not included in the MD17 dataset experiments, despite outperforming the proposed method on several targets. Similarly, GMN is missing from the N-body systems experiment. Including these baselines would provide a more complete picture of HEGNN's performance relative to existing methods.\n4. The paper could benefit from a more extensive discussion of the computational trade-offs involved in using high-degree representations. While the authors mention that HEGNN is more efficient than traditional high-degree models, a more detailed analysis of the computational costs and scaling behavior would be valuable.\n\nQUESTIONS:\n1. Can the authors conduct additional experiments with different tasks or application domains to further support the claims presented in the paper? This could help demonstrate the broader applicability and necessity of high-degree representations in equivariant GNNs.\n2. How does HEGNN compare to more recent state-of-the-art methods on standardized N-body system benchmarks, such as those used in the ClofNet paper and subsequent works?\n3. Can the authors provide a more detailed analysis of the computational efficiency of HEGNN compared to other high-degree models and EGNN? This could include training times, memory usage, and scaling behavior with respect to the number of particles and degree of representations.\n4. Have the authors explored the performance of HEGNN on tasks beyond position prediction, such as force field prediction or other molecular property predictions? This could help strengthen the argument for the necessity of high-degree representations.\n\nLIMITATIONS:\nThe authors acknowledge that their current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. It remains to be verified whether HEGNN is effective on large-scale geometric graph datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the necessity of higher-degree features in geometric graph neural networks (i.e. graph neural networks processing data embedded in three-dimensional space), focusing on the ability to recognize data with non-trivial rotational inner-symmetry such as k-folds and regular polygons, and based on the findings, proposes and validates an extension of the E(n)-GNN architecture. Theoretically, the authors first state in Theorem 3.4 that any O(3)-equivariant function on an inner-symmetric graph must produce an output which has an identical inner-symmetry. This immediately puts a restriction on the possible space of outputs an equivariant function can produce, more specifically as the invariant subspace under the action of the inner-symmetry (Theorem 3.5). It follows that certain inner-symmetric inputs and certain choices of output spaces (i.e. degrees $l$) leads to degenerate output space {0} (Theorem 3.6 and Table 1). Consequently, it follows that using a sufficiently high-degree (even) features leads to non-degenerate output spaces of layers, hence be able to encode e.g. orientations of the input. The authors empirically validate the findings on expressive power in Section 5.1, and then demonstrate good performance of the proposed model based on high-degree features and invariant scalarization on n-body for n <= 100 and MD17 datasets.\n\nSTRENGTHS:\n- S1. The paper is well-written and easy to follow.\n- S2. The claims on expressive power regarding symmetric inputs in Section 3.3 is correct, and is original as far as I can confirm; the closest related work is [1], and the authors have clarified the differences of their approach which seems technically correct. This claim is verified by the synthetic experiments in Section 5.1.\n- S3. The model proposed by the authors based on the theoretical claims is original as far as I am aware, and it seems useful in practice (Section 5.2-5.3), both in terms of performance and efficiency (slightly more costly than EGNN, but still cheaper than other networks involving higher-degree tensors such as TFN; but it is unclear how the approach compares to MACE, see W4 in the Weaknesses section). In particular, the fact that the model performs well on 100-body simulation task is interesting.\n\n[1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024)\n\nWEAKNESSES:\n- W1. While technically different from [1], this paper conveys a similar message: the use of higher-degree tensors in geometric neural networks is necessary to obtain higher expressive powers which is necessary for certain problems. Given this, one may argue that the GWL hierarchy in [1] is more general (as it handles general non-symmetric inputs as well). Furthermore, inputs with rotational inner-symmetry have been investigated in Section 5.2 of [1] to argue in favor of the higher-degree tensor features, precisely using the task of encoding rotations of the input; while the types of inner-symmetries considered in this work is more general (Table 1), the finding is fundamentally not very different, which can be understood as a weakness of the paper.\n- W2. It was not entirely clear to me why the capability of encoding rotations of inner-symmetric inputs would be important for practical learning tasks, although the experiments seem to imply so. For example, on n-body systems, I believe it is very unlikely that one would encounter symmetric inputs including the ones given in Table 1, due to symmetry-breaking factors such as numerical imprecision, simulation errors, and noise. I have similar concerns regarding chemical structures (e.g. MD17), as slight perturbations to the atomic coordinates are sufficient to eliminate the inner-symmetries (if they exist) and hence not experience the problems given in Theorem 3.5 and 3.6. This implies a gap between the theoretical arguments on the expressive power given in Section 3, and empirical results given in Sections 5.2 and 5.3.\n- W3. While Theorem 4.1 says that the proposed architecture can recover the information of all angles between each pair of edges, it does not directly address the original problem given in Theorem 3.5 and 3.6.\n- W4. I am not sure why MACE [2], while also using higher-degree features and already used for experiments in Tables 1 and 2, is not included for comparison in Tables 3 and 4.\n\n[1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024)\n\n[2] Batatia et al. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2023)\n\nQUESTIONS:\nPlease see the Weaknesses section.\n\nLIMITATIONS:\nThe authors have discussed the limitations in the checklist, but not in the main text. I encourage the reader to move the discussions to a separate section in the main text or Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the benefit of using higher order steerable features in geometric GNNs and theoretically identifies classes of symmetric geometric graphs where methods using only order-1 (or low order) features are guaranteed to fail.\nWith this in mind, the authors propose a simple and efficient way to integrate higher order features in the EGNN architecture.\n\nSTRENGTHS:\nClearly written and well motivated paper.\n\nThe proposed architecture seems a valid alternative to existing steerable architectures, enabling the use of higher order features with reduced computational complexity.\n\nThe paper also includes a few interesting theoretical insights about the failure cases of EGNN.\n\nWEAKNESSES:\nWhile the overall novelty is a bit limited (the message that higher order features are necessary for expressiveness has been theoretically and empirically explored in a number of previous works in the literature), the theoretical derivation of the failure cases and the proposed architecture can be useful contributions to the community. \n\nSee questions below.\n\nQUESTIONS:\nEq 7, 8: do I understand correctly that this kind of model can not pass any information between features $\\tilde{v}^{(l)}$ of different orders $(l)$ beyond the invariant quantities used to compute $m_{ij}$? Then, that essentially means that, after the initial features are extracted via the first convolution with spherical harmonics in Eq. 5, *the rest of model* is invariant to independent rotations by O(3) of each feature $\\tilde{v}^{(l)}$, right?\nThis seems to suggest there should be a gap in performance with models like TFN or SEGNN, which instead allow for different frequencies (features of different orders) to interact inside the model. Can you elaborate on this?\n\n\nSec 5.2: it seems important to compare with SEGNN [21] since that work was also motivated by the idea of adapting EGNN to support higher order steerable features, if I remember correctly.\n\n\nTable 3: I would expect the model size to scale quadratically with the maximum frequency $L$ since $\\sum_{l=0}^L (2l+1) = (L+1)^2$, but the runtimes in table 3 do not show this pattern. Why is the case?\n\n\nTable 4: what are the SOTA on these datasets? It would be good to include the performance achieved by some previous works to give an idea of whether these are competitive results or not\n\n\nWhy introducing Theorem 3.5 when Theorem 3.6 seems more general and practical? Also, I think these theorems hold for any compact (not necessarily finite) subgroups by using the Peter-Weyl theorem.\n\n\nLine 226: is this a more efficient way to implement this fairly sparse operation?\n\nLIMITATIONS:\nSome limitations are addressed in the supplementary material.\n\nFor completeness, I think the authors could comment on the possible gap in expressiveness with other steerable methods like TFN and SEGNN which allow for features of different orders to interact (see my first question above).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}