A new perspective on building efficient and expressive 3D equivariant graph neural networks
Authors: weitao Du
Yuanqi Du
Limei Wang
Dieqiao Feng
Guifeng Wang
Shuiwang Ji
Carla P Gomes
Zhi-Ming Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Neural Networks
Geometric Deep Learning
Equivariance
Symmetry

Abstract: Geometric deep learning enables the encoding of physical symmetries in modeling 3D objects. Despite rapid progress in encoding 3D symmetries into Graph Neural Networks (GNNs), a comprehensive evaluation of the expressiveness of these network architectures through a local-to-global analysis lacks today. In this paper, we propose a local hierarchy of 3D isomorphism to evaluate the expressive power of equivariant GNNs and investigate the process of representing global geometric information from local patches. Our work leads to two crucial modules for designing expressive and efficient geometric GNNs; namely local substructure encoding (\textbf{LSE}) and frame transition encoding (\textbf{FTE}). To demonstrate the applicability of our theory, we propose LEFTNet which effectively implements these modules and achieves state-of-the-art performance on both scalar-valued and vector-valued molecular property prediction tasks. We further point out future design space for 3D equivariant graph neural networks. Our codes are available at \url{https://github.com/yuanqidu/LeftNet}.

A new perspective on building efficient and expressive 3D equivariant graph neural networks
Weitao Du Chinese Academy of Sciences duweitao@mass.c ac en
Yuanqi Du Cornell University yd3928cormel1.ce edu
Limei Wang" Texas A&M University limei@tamu, edu
Dieqiao Feng Cornell University
Guifeng Wang Zhejiang University
Shuiwang Ji Texas A&M University
Carla P Gomes Cornell University
Zhi-Ming Ma Chinese Academy of Sciences
Abstract
Geometric deep learning enables the encoding of physical symmetries in model- ing 3D objects. Despite rapid progress in encoding 3D symmetries into Graph Neural Networks (GNNs), comprehensive evaluation of the expressiveness of these network architectures through a local-to-glon to-global analysis lacks today. In this paper, propose a local hierarchy of 3D isomorphism to evaluate the expressive power of equivariant GNNs and investigate the process of representing global geometric information from local patches. Our work leads to two crucial modules for designing expressive and efficient geometric GNNs; namely local substrue- ture encoding (LSE) and frame transition encoding (FTE). To demonstrate the applicability of our theory, we propose LEFTNet which effectively implements these modules and achieves state-of-the-art performance on both scalar-valued and vector-valued molecular property prediction tasks. We further point out future design space for 3D equivariant graph neural networks. Our codes available at com/ com./yaagqidu/Leffuuao
Introduction
The success of many deep neural networks can be attributed to their ability to respect physical sym- metry, such Convolutional Neural Networks (CNNs) []] and Graph Neural Networks (GNNs) [2]. Specifically, CNNs encode translation equivariance, which is essential for tasks such as object de- tection. Similarly, GNNs encode permutation equivariance, which ensures that the node ordering does not affect the output node representations, by aggregating neighboring messages. Modeling 3D objects, such point clouds and molecules. is fundamental problem with numerous applica- tions B1, including robotics 44, molecular simulation 61, and drug discovery 77.1.. Different from 2D pictures and graphs that only possess the translation [T] and permutation symmetry, 3D objects intrinsically encode the complex S F3) F(3) symmetry 12 which makes their modeling a nontrivial task in the machine learning community.
To tackle this challenge, several approaches have been proposed to effectively encode 3D rotation and translation equivariance in the deep neural network architectures, such TFN [13, EGNN [4), and SphereNet [13]. TFN leverages spherical harmonics to represent and update tensors equivariantly,
Equual contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

while EGNN processes geometric information through vector update. On the other hand, SphereNet is invariant by encoding scalars like distances and angles. Despite rapid progress has been made on the empirical side, it's still unclear what 3D geometrie information can equívariant graph neural networks capture and how the geometric information is integrated during the message passing process [16-18]. This type of analysis is crucial in designing expressive and efficient 3D GNNs, as it's usually a trade-off between encoding enough geometric information and preserving relatively low computation complexity. Put aside the /33) (3) symmetry, this problem is also crucial in analysing ordinary GNNs. For example, 1-hop based message passing graph neural networks [19] are computationally efficient while suffering from expressiveness bottlenecks (comparing with subgraph GNNs [20 21)..
On the other hand, finding a better trade-off for 3D GNNs is more challenging, since we must ensure that the message updating and aggregating process respects the SE(3) E3) symmetry.
In this paper, we attempt to discover better trade-offs between computational efficiency and expressive power for 3D GNNs by studying two specific questions: 1. What is the geometric expressiveness of 3D GNNs through a local 3D) graph isomorphism lens? 2. What is expressiveness of 3D GNNs in representing global geometric information through local patches? The first question relates to the design of node-wise geometric messages, and the second question relates to the design of equivariant (or invariant) aggregation. To tackle these two problems, we take a local-to-global approach. More precisely, we first define three types of 3D graph isomorphism to characterize local 3D structures: tree, triangular, and subgraph isomorphism, following a local hierarchy. Our local hierarchy lies between the I-hop and 2-hop geometric isomorphism defined in Joshi et al. [22), detailed in Appendix G.2] thus, it can be used to measure the expressive power of 3D GNNs by their ability of differentiating non-isomorphic 3I) structures similar to the geometric WI. test proposed summarize one essential ingredient for
in Joshi et al. [22]. Under this theoretical framework, building expressive geometric messages on each node: local 3D substructure encoding (LSE), which allows an invariant realization. To answer the second question, we analyze whether local invariant features are sufficient for expressing global geometries by message aggregation, and it turns out that frame transition encoding (FTE) is crucial during the local to global process. Although FTE can be realized by invariant scalars, we further demonstrate that introducing equivariant messaging passing more efficient. After presenting LSE and FTE modules, we are able present a modular overview of 3D GNNs designs. In realization of our theoretical findings, we propose LEFTNet that efficiently implements LSE and FTE without sacrificing expressiveness. Empirical experiments on real-world scenarios, predicting scalar-valued property (e.g. energy) and vector-valued property (e.g. force) for molecules, demonstrate the effectiveness of LEFTNet.
2 Preliminary
In this section, we provide an overview of the mathematical foundations of F(()) and (3) symmetry, which is essential in modeling 3D data. We also summarize the message passing graph neural network framework, which enables the realization of E333/SE(3) equivariant models.
Euclidean Symmetry. Our target is to incorporate Euclidean symmetry to ordinary permutation- invariant graph neural networks. The formal way of describing Euclidean symmetry is the group E(3) (3) x T(3), where 0(3) corresponds to reflections (parity transformations) and rotations. For tasks that are anti-symmetric under reflections (e.g. chirality). we consider the subgroup SE(3) = SO(3) T(3), where SO(3) is the group of rotations. We will use SF(3) in the rest of the paper for brevity except when it's necessary to emphasize reflections.
Equivariance. A tensor-valued function (x() is said to be equivariant with respect to SFF(3) if for any translation or rotation g E (E))) acting on x E R², we have
x(gx)= = (x))
where M(-) is matrix representation of SF((3) acting on tensors. See Appendix A for general definition of tensor fields. In this paper, we will use bold letters to represent an cquivariant tensor, e.g., X as a position vector. It is worth noting that when cp(x) E R and M(g) group representation), the equivariant function (xx) is also called an invariant scalar function.
(the constant
Message Passing Scheme for Geometric Graphs. A geometric graph G' is represented by G (V, E). Here, U/ E V denotes the set of nodes (vertices, atoms), and Eij E E denotes the set of edges. For brevity, the edge feature attached on esg is also denoted by eij. X (X),..., x,) E R denotes the node positions which determine the geometric structure of G.

---

commion free structure
common trianguior structure
Sj
3V
44
Zin
S
2
(a)
(b)
Figure 1:
St and S, share the same tree structure (edge lengths are identical), but they not triangular isomorphic (different dihedral angles); (b) S, and Sj are triangular isomorphic but not subgraph isomorphic (the relative distance between the two triangles is different).
common machine learning tool for modeling
data the Message Passing Neural
Network (MPNN) [16]. A typical 1-hop MPNN framework consists of two phases: (1) message passing; (2) readout. Let h4, hy denote the 1-th layer's node features of source and target j that also depend on the 3D positions (X,; xj), then the aggregated message
m
miy(h (x:), (x;)), (x)),
JEN(()
and Q (EN(() is any permutation invarriant pooling operation between the l-hop neighbors of i. We also include the edge features ety into the message passing phase for completeness. 3D equivariant GNNs (3D GNNs for short) require the message m, to be equivariant with respect to the geometric graph. That is, for an arbitrary edge eij:
mig(h' (gx:), lt' (gx,)) = M(g)ma (x:), (x))),
(2)
where 9 SE(3) is acting on the whole geometric graph simultaneously: .i....... x.,)
(9X1..... gX,)). ). For example, the invariant model ComENet (23] satisfies Eq. b setting .M(g) 1,
and MACE (24 realized Eq. [ for nonconstant irreducible group representations M(g) through spherical harmonics and Ciebsch-Gordae coefficients.
3 A local hierarchy of 3D graph isomorphism
As presented in Section 2 defining expressive messages is an essential component for building powerful 3D GNNs. In this section, develop fine-grat grained characterization of local 3D structures and build its connection with the expressiveness 3D GNNs.
Since the celebrated work [25]. a popular expressiveness test for permutation invariant graph neural networks is the 1-WL graph isomorphism test [26], and Wijesinghe and Wang (7 has shown that the 1-WL test is equivalent to the ability to discriminate the local subtree-isone iittroeisomm..hism.. It motivates us to develop novel (local) 3D isomorphism for testing the expressive power of 3D GNNs. However, this task is nontrivial, since most of the previous settings for graph isomorphism are only applicable to 2D topological features. For 3D geometric shapes, we should take the SFF() symmetry into account. Formally, two 3D geometric graphs X, Y are defined to be globally isomorphic, if there exists E SE(3) such that
Y y
In other words, X and are essentially the same, if they can be transformed into each other through series of rotations and translations. Not that Eq. 3 is up to the permutation of nodes. Inspired by Wijesinghe and Wang [27). now we introduce a novel hierarchy of SE(3) equivariant local isomorphism measure the local similarity of 3D structures.
Let S; represent the 3D subgraph associated with node i. This subgraph contains all the 1-hop neighbors of i its node set, along with all edges in E where both end points one-hop neighbors of i. For each edge eij E E, the mutual 3D substructure S is defined by the intersection of S, and Sy: S,-j S, ns
Given two local subgraphs S; and Sy that correspond to two nodes and j (not necessarily adjacent), say S; is (tree-, triangular-, subgraph-} isometric to Sy. if there exists bijective function
3

---

f:S; S; - S such that hy(w)) hy for every node u E Sü, and the following conditions hold respectively:
Tree Isometric: there exists a collection of group elements Giu E SE(3), such that
[x)) X)() = (FuxNu for each edge eiu E S
Triangular Isometric: If there exists collection of group elements Giu SE()), such that the corresponding mutual 3D substructures satisfy: f(1)) = SiuSu-i for each edge Cju E S,;
* Subgraph Isometrie: for any two adjacent nodes u. V. E St (u) and f(o) are also adjacent in Sy. and there exist single group element gi E SE(3) such that g.S, = S.
Note that tree isomorphism only considers edges around a central node, which is of a tree shape. On the other hand, the mutual 3D substructure can be decomposed into a bunch of triangles (since it's contained in adjacent node triplets), which explains the name of triangular isomorphism.
In fact, the three isomorphisms form hierarchy from micro to macro, in the sense that the following implication relation holds:
Subgraph Isometric = Triangular Isometric = Tree Isometric
This is an obvious fact from the above definitions. To deduce the reverse implication relation, we provide visualized example. Fig. shows two examples of local 3D structures: 1. the first one shares the same tree structure, but is not trangular-isomorphic 2. the second one riingular-isomorphic but not subgraph-isomorphic In conclusion, the following diagram holds:
Tree Isometric / Triangular Isometric / Subgraph Isometric
One way to formally connect the expressiveness power of a geometric GNN with their ability of differentiating geometric subgraphs is to define geometric WL tests, the reader can consult [22]. In this paper, we take intuitive approach based our nested 3D hierarchy. That is, if two 3D GNN algorithms and B can differentiate all non-isomorphic local 3D shapes of tree (triangular) level, while A can differentiate at least two more 3D geometries which are non-isomorphic at triangular (sbgrraph)))) level than B, then we claim that algorithm A's expressiveness power is more powerful than B.
Since tree isomorphism is determined by the one-hop Euclidean distance between neighbors, distin- guishing local tree structures is relatively simple for ordinary 3D equivariant GNNs. For example, the standard baseline SchNet (28] one instance of Eq. 1 by setting ety (BF(d(x;; x,)), where RBF(.) set of radial basis functions. Although is powerful enough for testing tree non- isomorphism (assuming that RBF(.) is injective), we prove in Appendix D that SchNet cannot distinguish non-isomorphic structures at the triangular level.
On the other hand, Wijesinghe and Wang [27 has shown that by leveraging the topological informa- tion extracted from local overlapping subgraphs, we can enhance the expressive power of GNNs to go beyond 2D subtree isomorphism. In our setting, the natural analogue of the overlapping subgraphs exactly the mutual 3D substructures. Now we demonstrate how to merge the information from 3D substructures to the message passing framework (I). Given an
encoder é, define the 3D structure weights Aij == ((s,-)) for each edge Ciy E E. Then, the message passing framework ①I) is generalized to:
mf = miy (x)), (x))).))
jEN(
Formula 4 is an efficient realization of enhancing 3D GNNs by injecting the mutual 3D substructures. However, a crucial question remains to be answered: Can the generalized message passing framework boost the expressive power of 3D GNNs? Under certain conditions, the following theorem provides an affirmative answer:
Theorem 3.1. Suppose & is universal SE(3) invariann approximator of functions with respect to the mutual 3d structures Si-j. Then, the collection of weights {{A., ((Sssj))te,ee enables the differentiation of local structures beyond tree isomorphism. Furthermore, under additional injectivity assumptions (us described in Eq. 16 3D GNNs based the enhanced message passing framework (see Section F) map at least two distinct local 3D subgraphs with isometric local tree structures to different embeddings.

---

Proof. The detailed proof is provided in Appendix D
This theorem confirms that the enhanced 3D GNN (formula 4 is more expressive than the SchNet baseline, at least in lesting local non-isomorphic geometric graphs. The existence of such local encoder á is proved in two ways: 1. Equivariant construction by the Atomic Cluster Expansion (ACE) [9 ; Invariant construction under scalarization by edge-wise equivariant frames (30]. Note that there are other different perspectives on characterizing 3D structures, we will also briefly discuss them in Appendix
4 From local to global: the missing pieces
In the last section, we introduced local 3D graph isomorphism hierarchy for testing the expressive power of 3D GNNs. Furthermore, we motivated adding SE(3))invariant encoder to improve the expressive power of one-hop 3D GNNs by scalarizing not only pairwise distances but also their mutual 3D structures in Theorem 3.1 However, build a powerful 3D GNN, it remains be analyzed how 3D GNN acquires higher order (beyond I hhhop neighbors) information by accumulating local messages. A natural question arises: Are local invariant messages enough for representing global geometric information?
To formally formulate this problem, we consider a two-hop aggregation case. From fig. 2 the central atom is connected with atoms b and c. Except for the common neighbor a, other atoms that connect 6 and e form two 3D clus- ters, denoted by B, C. Suppose the ground-truth interaction potential of and C imposed on atom is described by a tensor-valued function (a C). Since B and C are both beyond the 1 1-hop neighborhood of a, the information of fa (B, C) can only be acquired after two steps of message passing: 1. atoms l and aggre- gate message separately from and C; the central atom a receives the aggregated message (which contains information of B and C) from its neighbors and
RØD: Frame Transition between and
hg
h
2b
ybb
a
y
fa
h
he
Je
2bb
cluster
cluster
Figure Illustrations of different local frames and their transition.
Let SB (Sc) denote the collection of all invariant scalars created by B (C). For example, Sp contains all relative distances and angles within the 3D structure B. Then, the following theorem holds:
Theorem 4.1. Not all types of invariant interaction (B C) can be expressed solely as functions of the union of two sets SB and Sc. In other words, there exists E(3) invariant function (B, C). such that it cannot be expressed as functions of Sn and Sc: fa B C) + p(Sn, Sc) for arbitrary invariant functions p.
Proof. The detailed proof is provided in Appendix
This theorem essentially demonstrates that simply aggregating "local" scalar information from different clusters is insufficient for approximating "global" interactions, even in the case of simple invariant potential learning tasks. In contrast the previous section, where the local expressiveness was evaluated based on the ability to classify geometric shapes, in this theorem, we constructed counterexamples using continuous regression functions that depend strictly on information beyond the combination of local invariant scalars. Intuitively, the theorem is proved by introducing two local equivariant frames (three independent equivariant vectors, see Appendix for the definition) determined by B (C) separately, through which all scalars in SB (Sc) can be expressed. However, the transition matrix between these frames is not encoded the aggregation, resulting in information loss when aggregating geometric features from two sub-clusters. In other words, local frames provide local observations of geometric quantities, while the transition matrix reveals the global changes between local geometries. Importantly, the proof also highlights that the missing information that causes the expressiveness gap solely the Frame Transition (FT) information, which we will define immediately.
5

---

Frame Transition (FT), Let (ei, cs ej)) and (e). e2. c3) be two orthonormal frames in R These frames are connected by an orthogonal matrix Rig E SO(3):
(e c2 2g) Re c2,
Furthermore, when (ei, ez2 eÿ) and (ef eg. eq) equivariant frames, all elements of R& invariant scalars. For instance, in the case of a geometric graph where i and j represent indexes of two connected atoms, the fundamental torsion angle Tij in ComeNet 23 corresponds to one element of Rij (see Appendix
To address this expressiveness gap, one approach is to directly incorporate all invariant edge-wise frame transition matrices (FT) into the model. This can be achieved using a geometric formulation based on neural sheaf diffusion, as described in Appendix H However, it is worth noting that this method becomes computationally expensive when dealing with large number of local clusters, as it requires (k2)) pairs of FT for each node. Instead, we propose more efficient approach by introducing equivariant tensor features for each node i, denoted as m, These tensor features allow us to maintain equivariant frames directly within each m, We prove in Appendix E that FT can be casily derived through equivariant message passing and updating when utilizing these equivariant tensor features.
Equivariant Message Passing. Similarly with the standard one-hop message passing scheme the aggregated tensor message m, from the layer to the layer can be written as: m EjE(() m Since summation does not break the symmetry rule, it obvious that m -1 are still equivariant tensors. However, the nontrivial part lies in the design of the equivariant update function Φ
m (mm!
To fully capture the information of FT, it necessary for á to possess sufficient expressive power while maintaining SE(3) equivariance. Here, we propose a novel way of updating scalar and tensor messages by performing node-wise scalarization and tensorization blocks (the FTE module of Figure 3.. From the perspective of Eq. 2 m(x,) is transformed equivariantly as:
m(xx.) Σ M' (x))) E SE(3).
1=0
Here, mxx.) is decomposed (X.)) (x.)) according to different tensor types, and (M T is collection of different SE(3) tensor representations (see the precise definition in
Appendix
To illustrate the benefit of aggregating equivariant messages from local patches, we study simple case. Let (B, C) = hg hC be an invariant function B and C (see Fig. 2 then fa can be calculated by direction composition of scalar messages and equivariant vector messages: fa (B, C)
||bll' where m. hg + he is equivariant vector. Note that m. follows the local equivariant aggregation formula 6 and the other vectors' norm || and |lhe are obtained
_[[[m.11 1||mm.12~1100 - 1
through local scalarization atoms b and c. As comparison, it's worth mentioning that [a C) can also be expressed by local scalarization with the additional transition matrix data REC defined by Eq. Let ha and he be the scalarized coordinates with respect to two local equivariant frames
Fn and Fc. Then Sa (a C) = & R + Tic 7.m - Te However, it requires
adding the transition matrix pC for each (B, C) pair into the aggregation procedure, which is computationally expensive compared to directly implementing equivariant tensor updates.
5 Building an efficient and expressive equivariant 3D GNN
We propose to leverage the full power of LSE and FTE modules (along with necessary tensor update module) to push the limit of efficient and expressive 3D equivariant GNNs design. Specifi- cally, we improve recently proposed method based on constructing local frames but without the implementation of LSE and FTE 30
We also analyze related works following this framework
detailed in Appendix G

---

Input 3D graph
Interaction block
LSE
FTE
DEM
S:
Subgraph 5/-]
Embedding
5
LSE: to encude
Au
Equivariant message passing
Trame Fi
Nr
local structures
Fi
Scalarizution
Scalarization
F2
Interaction
mi FIR to encode
RBF
@
Interaction
MLP
MLP
+ Output block
trancition
Average
Ten:sorication
Ai
A/
Ourput
Invariant
Equivariant
Figure 3: Illustrations of our modular framework for building equivariant GNNs and the realization of LEFTNet. Each interaction block contains LSE to encode local 3D structures, equivariant message passing to update both invariant (unbold letters, e-g. h2) and equivariant (bold letter, e.g. h2) features, and FTE to encode frame transition. S is the local 3D structure of each edge eij. Fij and F; are the equivariant frames for each edge Egj and node @ indicates element-wise multiplication, and indicates concatenation. Note that we do not include Ciy in the figure since, practically, they are generated based on h; and h,.
LSE Instantiation. We propose to apply edge-wise equivariant frames (following 30 to encode the local 3D structures S By definition, S,-, contains edge Eij. nodes and j. and their common neighbors. We use the equivariant frame Fij built on eij (see the precise formula in Appendix (F) to scalarize S.j. After scalarization (). the equivariant coordinates of all nodes in S,-j are transformed into invariant coordinates: (x, - Fy for X S.-j). To encode these scalars sufficienlly, we first weight each d by the RBF distance embedding: 7g -> RBF Xx Il) @ MLP((*)) for each X; E S, Note that to preserve the permutation symmetry, the MLP is shared among the nodes. Finally, the 3D structure weight Aij is obtained by the average pooling of all node features.
FTE Instantiation. We propose to introduce equivariant tensor message passing and update function for encoding local FT information. At initialization, let NF' NF'(X,,,, x,) denote the embedded tensor- valued edge feature between and j. We split it into two parts: 1. the scalar part SF (xx))) for aggregating invariant messages; 2. the higher order tensor part TF' TF',,; (x;;x)) for aggregating tensor messages. To transform TF' (x,, Xj), we turn to the equivariant frame Fij once again. After
scalarization by Fiz. TF' (X,, (X,,x, x,) becomes tuple of scalars TF (X,: X;), which is then transformed by MLP. Finally, we output arbitrary tensor messages through equivariant tensorization
TF' (x, x,)
Tensorize
NF Xx),
Further details are provided in Appendix F As we have discussed earlier, the node-wise update function ó in Eq. i also one of the guarantees for powerful FTE. As a comparison, á is usually standard MLP for updating node features 2D GNNs, which is a universal approximator of invariant functions. Previous works [14 31 updated equivariant features by taking linear combina- tions and calculating the invariant norm of tensors, which may suffer from information loss. Then a natural question arises: Can we design an equivariant universal approximator for tensor update? We answer this question by introducing novel node-wise frame. Consider node i with its position x,, let x; : N E Ex)eN(x.)) Xj be the center of mass around x xis neighborhood. Then the orthonormal equivariant frame J, = (ci, ch, ei)) with respect to x; is defined similar to the edge-wise frame between x, and x; in [30, detailed in appendix B Finally, we realize powerful by the following theorem:
Theorem 5.1. Equipped with an equivariant frame for each node the equivariant function defined by the following composition is a universal approximator of tensor transformations: Scalarization - MLP Tensorization.
Proof. The detailed proof is provided in Appendix
F
LEFTNet. An overview of our (LSE, FTE) enhanced efficient graph neural network (LEFTNet) is depicted in Fig- 3 The detailed algorithm for LEFTNet is shown in Algorithm| Df Appendix C LEFTNet receives as input a collection of node embeddings (uf..... J which contain the atom types and 3D positions for each node: of x:), where i E {1,.... ..) N).
For each edge eij E,
7

---

we denote the associated equivariant features consisting of tensors by ety. During each messaging passing layer, the LSE module outputs the scalar weight coefficients A.j as enhanced invariant edge feature and feed into the interaction module. Moreover, scalarization and tensorization as two essential blocks are used the equivariant update module that fulfills the function of FTE. The permutation equivariance of geometric graph is automatically guaranteed for any message passing architecture, we provide a complete proof of -quivvaquve for LEFTNet in Appendix
SE(3) vs E(3) Equivariance. Besides explicitly fitting the SE invariant molecular geometry probability distribution, modeling the energy surface of molecule system is also crucial task for molecule property prediction. However, the Hamiltonian energy function E of molecule is invariant under refection transformation: Energy (X) Energy (RX), for arbitrary reflection transformation R E E3).. In summary, there exist two different inductive biases for modeling 3D data: (1) SE(3) equivariance, e-g. chirality could energy remains the same under reflections.
therapeutic drug to a killing toxin; (2) E(3) equivariance, e.g.
Since we implement (3) equivariant frames in LEFTNet, our algorithm is naturally SE(3) cquivariant. However, our method is flexible to implement ((3) equivariant tasks as well. For E33 equivariance, we either replace our frames E33 equivariant frames, or modify the scalarization
block by taking the absolute value: x T = (x G1, X e2, Ez) + (x-61, eg). es)
SE(3)
E(3)
Intuitively, since the second vector e2 is pseudo-vector, projections of any equivariant vectors along the direction are not E(3) invariant until taking the absolute value.
Efficiency. To analyze the efficiency of LEFTNet, suppose 3D graph G has vertices, and its average node degree is Our algorithm consists three phases: 1. Building equivariant frames and performing local scalarization; 2. Equivariant message passing; 3. Updating node-wise tensor features through scalarization and tensorization. Let be the number layers, then the computational complexity for each of our three phases are: 1. ((nk) for computing the frame and local (1-hop) 3D features; 2. (nkkl) for I-hop neighborhood message aggregation; 3. O(nl) for node-wise tensorization and feature update.
Experiments
We evaluate LEFTNet both scalar value (e.g. energy) and vector value (e.g. forces) prediction tasks. The scalar value prediction experiment is conducted on the QM9 dataset (32) which includes
134k small molecules with quantum property annotations: the vector value prediction experiment
conducted on the MD17 dataset [33] and the Revised MD17(cMD177 17) dataset (34] which includes the
energies and forces of molecules. We compare LEFTNet with list of state-of-the-ar equivariant (invariant) GNNs including SphereNet (15), PaiNN 335 Equiformer (36), GemNet (37]. etc [28 38
13 39
16
48
The training details, results on rMD17, etc. are listed Appendix
Table 1: Mean Absolute Error for the molecular property prediction benchmark QM9 dataset. (The best results are bolded and the second best are underlined.)
Task Units
A=
Ehомo meV
ELUMO meV
R bohr
Uo mcV
bohr
meV
cal/mol
meV
meV
meV
meV
NMP
.092
69
.030
.040
19
17
.180
20
1.50
Cormorant
.085
61
34
38
.038
.126
20
21
.611
21
22
2.03
LieConv
.844
49
30
25
.032
.038
22
24
.800
19
19
2.28
TFN
.223
58
40
SE(3)-T:.
.142
53
35
.051
.054
EGNN
.071
48
29
25
.29
031
12
12
.106
12
11
1,55
SEGNN
.660
42
24
21
003
.03
15
16
.660
13
15
1.62
ClofNet
53
9
9
EQGAT
.063
44
26
.044
.027
12
13
.257
1,50
Equiformer
.056
33
17
16
.014
022
10
TO
.227
11
10
1.32
LEFTNet (ours)
.448
40
24
18
,012
.023
7
.109
7
1,33
Schnet
.235
63
41
34
.033
.333
14
14
.073
19
1.70
DimeNei++
33
25
.331
SphereNet
.046
32
23
18
.026
.021
8
6
.292
1.12
ClofNer
.053
49
33
25
.038
(126
9
8
.255
8
1,59
PaiNN
.045
46
28
.012
.024
7
6
.066
1.28
LEFTNet (ours)
39
.094

---

Table 2: Mean Absolute Error for per-atom forces prediction (kcal/mol Å) on MD17 dataset. The best results are bolded.
WoPF-100
1000
Others
Molccule
xGDMI
SchNer
DimeNet
SphereNer
SponkyNet
L.FFNNet
SphereNet
GommNer
LRFTNN
PaiNN
NewtonNet
Aspirin
0.68
1.35
0.429
0.430
0.258
0.300
0.209
0.217
0.196
0.371
D.348
Benoene
0.20
0.31
0.187
0.178
0.145
0.147
0.145
0.142
Eihannl
0.33
0.230
0,094
0.138
0.091
D.264
Malonaldehyde
0.41
0.66
03383
0.167
0.310
0.155
Naghthalene
0.01
0.58
0.215
0.178
0.089
0.073
0.048
0.051
0.044
0.083
0.084
Salicylic acid
0.28
0.85
0.374
0.360
0.180
0.167
0.113
0.125
0.117
0.209
0.197
Toluene
0.14
0.57
0,210
0,087
0.084
0.054
0.060
0.102
0
Uracil
0.24
0.56
0.301
0.119
0.116
0.106
0.097
6.1 QM9 scalar-valued property prediction
The QM9 dataset is widely used dataset for predicting molecular properties. However, existing models are trained on different data splits. Specifically, Cormorant [40], EGNN [14), etc., use 100k, 18k, and 13k molecules for training, validation, and testing, while DimeNet [3). SphereNet [3], etc., split the data into 1 110k, 10k, and 11k. For fair comparison with all methods,
SerWeet
Clorttet
ComEENt
Dimchicz+
SphoneNet
Thic
conduct experiments using both data splits. Results are listed in Table [ For the first data split, LEFTNet is the best on 7 out of the 12 properties and improves previous SOTA results by 20% on average. Consistently, LEFTNet is the best second best 10 out of the 12 properties for the second split. These results validate the effectiveness of LEFTNet. Detailed ablation in Appendix Appendix[showss shows that both LSE and FTE contribute the final performance. In addition, we compare the forward time of existing methods to show the efficiency of LEFTNet. Specifically, we use the same batch size for all methods and report the one epoch training time. Fig. A that LEFTNet can achieve the best performance using similar forward time SchNet, ClofNet, and ComENet and is much faster than DimeNet and SphereNet. This is consistent with our efficiency analysis in Sec. []and Table
Training per epach Is)
Figure 4: Comparisons of ex- isting methods in terms of the training time and the MAE of the property Uu of QM9.
6.2 MD17 vector-valued property prediction
We evaluate LEFTNet to predict forces on the MD17 dataset. Following existing studies (28 38 [S]. we train a separate model for each of the 8 molecules. Both training and validation sets contain 1000 samples, and the rest are used for testing. Note that all baseline methods are trained
Table 3: Ablation study on Aspirin of the MD17 dataset. Detailed abla- tions on other molecules
on a joint loss of energies and forces, but different methods use different weights of force over energy (WoFE). For example, SchNet (28) sets WoEF as 100, while GemNet 37 uses a weight of 1000. For a fair comparison with existing studies, conduct experiments on two widely used weights of 100 and 1000 following Liu et al. [1). The results are
are listed in Appendix [
Method
MAE
w/o LSE and FTE
1.083
LSE only
0.451
LSE FTE
0.300
summarized in Table 2 Results show that when WoFE is 100, LEFTNet outperforms all baseline methods on 5 of the 8 molecules. In addition, LEFTNet can outperform all baseline methods on 6 of the 8 molecules when WoFE is 1000. These experimental results on MD17 demonstrate the performance of LEFTNet on vector-valued property prediction tasks. The ablation results in Table 3 demonstrate that both LSE and FTE are important the final results. Specifically. the algorithm for LSE only is in Algorithm I of Appendix MD17 dataset we used has numerical noise (34], and recomputed version of MD17, called Revised MD17 (rMD17) 34] is proposed to reduce the numerical noise. In addition to the original MD17, we also conduct experiments on the rMD17, and the results are listed in Appendix !
C Note that the original
7 Limitation and future work
In this paper, we seek a general recipe for building 3D geometric graph deep learning algorithms. Considering common prior of 2D graphs, such as permutation symmetry, has been incorporated in off-the-shelf graph neural networks, we mainly focus on the E3) and SE(3) symmetry specific to 3D geometric graphs. Despite our framework being general for modeling geometric objects, we only conducted experiments on commonly used molecular datasets. It's worth exploring datasets in other domains in the future.

---

To elucidate the future design space of equivariant GNNs, we propose two directions that are worth exploring. Firstly, our current algorithms consider fixed equivariant frames for performing aggregation and node updates. Inspired by the high body-order ACE approach [99 (for modeling atom cen-ceetereed potentials), is worth investigating in the future if equivariant frames that relate to many body (e.g.. the PCA frame in [50] 5T) can boost the performance of our algorithm. For example, to build the A-basis proposed
[49], we can replace our message aggregation Eq. 6 from summation to tensor product, which is also a valid pooling operation. Another direction is to explore geometric mesh graphs on manifolds M, where the local frame is defined on the tangent space of each point: F(x) T.M. Since scalarization technique (crucial for realizing LSE in LEFTNet) originates from differential geometry on frame bundles [52]. it is reasonable to expect that our framework also works for manifold data [53 54
Acknowledgments and Disclosure of Funding
Y.D., D.F. and C.P.G. are supported by the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program; the National Science Foundation (NSF), the Air Force Office of Scientific Research (AFOSR). L. W. and S.J. supported by NSF IIS-2243850.
References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[2] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 169.v.160, 2016.
[3] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307. wtt.:2337..8223.. 2023.
[4] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se equivaarai object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[5] Frank Noé, Alexandre Tkatchenko, Klaus-Robert Müller, and Cecilia Clementi. Machine learning for molecular simulation. Annual review of physicul chemistry, 71:361-390, 2020.
[6] Lars Holdijk, Yuanqi Du, Priyank Jaini, Ferry Hooft, Bernd Ensing, and Max Welling, Path integral stochastic optimal control for sampling transition paths. In ICML 2022 2nd Al for Science Workshop.
[7] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 (4):18-42, 2017.
[8] Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence, 3(12): 2021.
[9] Zhengyang Wang, Meng Liu, Youzhi Luo, Zhao Xu, Yaochen Xie, Limei Wang, Lei Cai, Qi Qi, Zhuoning Yuan, Tianbao Yang, et al. Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics, 38(9):2579-2586 2022.
[10] Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Li6, Carla Gomes, Max Welling, et al. Structure-based drug design with cquivariant diffusion models. arXiv preprint 22X.:22200.1699 2022.
[11] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengehao Liu. Molgensurvey: systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203. 2022.
10

---

[12] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, arXiv preprint arXiv:2104.13472 2021.
[13] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equiiariann neural networks for 3D point clouds, arXiv preprint arXiv: 08219, 2018.
[14] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323 -332.. PMLR, 2021.
[15] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D molecular graphs. In International Conference on Learning Representations, 2022. URL htttps::/ http:p///.ppppeeeeeeeeeer/rrrrra net//foorrmtddbra...axa...
[16] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pages 1263-1272. PMLR, 2017.
[17] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez- Gonzalz- Vinicius Zambaldi. Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806.0126111111 2018.
[18] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling, 59 (8):3370-3388, 2019.
[19] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations.
[20] Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. In International Conference on Learning Representations.
[21] Fabrizio Frasca, Beatrice Bevilacqua, Michael Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. Advances in Neural Information Processing Systems, 35:31376-31394 2022.
[22] Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks. In The First Learning on Graphs Conference, 2022.
[23] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: /ttp:////peeeeeeeeee wet wtt./orrrnnnddddeaiiiii
[24] Ilyes Batatia, David Peter Kovacs, Gregor NC Simm, Christoph Ortner, and Gabor Csanyi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems.
[25] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint
2018
[26] Christopher Morris, Yaron Lipman, Haggai Maron, Bastian Rieck, Nils M Kriege, Martin Grohe, Matthias Fey, and Karsten Borgwardt. Weisfeiler and leman go machine learning: The story so far. arXiv preprint 1xXx::2112.09992 2021.
[27] Asiri Wijesinghe and Qing Wang. A new perspective on" how graph neural networks go beyond weisfeiler-lehman!" In International Conference on Learning Representations, 2021.
[28] Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018
11

---

[29] Ralf Drautz. Atomic cluster expansion for accurate and transferable interatomic potentials. Physical Review 99(1)014404, 2019.
[30] Weitao Du, He Zhang. Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie- Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference Machine Learning, pages 5583-5608. PMLR, 2022.
[31] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
[32] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
[33] Stefan Chmicla, Alexandre Tkatchenko, Huziel Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Rober Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3()):1603315, 2017.
[34] Anders S Christensen and O Anatole Von Lilienfeld. On the role of gradients for machine learning of molecular energies and forces. Machine Learning: Science and Technology, 1(4): 045018, 2020.
[35] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pages 7388.9388. PMLR, 2021.
[36] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11999, 2022.
[37] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Universal directional graph neural networks for molecules. Advances Neural Information Processing Systems, 34: 6790-6802, 2021.
[38] Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123, 2020.
[39] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
[40] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural
networks. Advances in neural information processing systems, 32, 2019.
[41] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165-31766 PMLR, 2020.
[42] Tuan Le, Frank Noé, and Djork-Amé Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprint /xX/v:2202.0989). 2022.
[43] Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications, 14(1):579, 2023.
[44] Dávid Péter Kovács, Cas van der Oord, Jiri Kucera, Alice EA Allen, Daniel Cole, Christoph Ortner, and Gábor Csányi. Linear atomic cluster expansion force fields for organic molecules: beyond rmse. Journal of chemical theory and computation, 17(12):669--711, 2021.
[45] Felix A Faber, Anders S Christensen, Bing Huang, and O Anatole Von Lilienfeld. Alchemical and structural distribution based representation for universal quantum machine learning. The Journal of chemical physics, 148(24):241117,22 2018.
12

---

[46] Albert P Bartók, Mike C Payne, Risi Kondor, and Gábor Csányi Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Physical review letters, 104(13):136403, 2010.
[47] Xiang Gao, Farhad Ramezanghorbani, Olexandr Isayev, Justin Smith, and Adrian Roitberg. Torchani: a free and open source pytorch-based deep learning implementation of the ani neural network potentials. Journal of chemical information and modeling, 60(7):3408-34)) 2020.
[48] Oliver T Unke, Stefan Chmiela, Michael Gastegger, Kristof T Schütt, Huziel E Sauceda, and Klaus-Robert Müller. Spookynet: Learning foree fields with electronic degrees of freedom and nonlocal effects. Nature communicationss 12(1):7273, 2021.
[49] Ilyes Batatia, Simon Batzner, Dávid Péter Kovács, Albert Musaclian, Gregor NC Simm, Ralf Drautz, Christoph Ortner, Boris Kozinsky, and Gábor Csányi The design space of e (3)- cquivariant atom-centered interatomic potentials. arXiv preprint arXiv:2205.0666 06643, 2022.
[50] Omri Puny, Matan Atzmon, Edward J Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference Learning Representations, 2021.
[51] Alexandre Duval, Victor Schmidt, Alex Hernandez Garcia, Santiago Miret, Fragkiskos D Malliaros, Yoshua Bengio, and David Rolnick. Faenet: Frame averaging equivariant gnn for materials modeling. arXiv preprint arXivv2305..5557, 2023.
[52] Elton P Hsu. Stochastic analysis on manifolds. Number 38. American Mathematical Soc., 2002.
[53] Lingshen He, Yiming Dong, Yisen Wang, Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. Advances in Neural Information Processing Systems, 34:27331-27343, 2021.
[54] Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equiv- ariant graph mechanics networks with constraints. arXiv preprint arXiv:2203.0644 2022.
[55] Nadav Dym and Haggai Maron, On the universality of rotation equivariant point cloud networks. arXiv preprint arXiv:2010.02449, 2020.
[56] Sheng Gong, Tian Xie, Yang Shao-Horn, Rafael Gomez-Bombarelli, and Jeffrey C Grossman. Examining graph neural networks for crystal structures: limitations and opportunities for capturing periodicity. arXiv preprint arXiv: 05039, 2022.
[57] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[58] Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In The Eleventh International Con- ference on Learning Representations, 2023. URL https: /ttts:://pperevvine net/ wwrommommm..
9X 9X-bgLDLYKQ
[59] Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference on machine learning, pages 5361-5370. PMLR, 2020.
[60] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867-8887. PMLR, 2022.
[61] Xiyuan Wang and Muhan Zhang. Graph neural network with local frame for molecular potential energy surface. arXiv preprint arXiv:2208.00 00. 2022.
[62] Cristian Bodnar, Francesco Di Giovanni, Benjamin Chamberlain, Pietro Liò, and Michael Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns, Advances in Neural Information Processing Systems, 15:18527-18541, 2022.
13

---

[63] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[64] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning Graphs and Manifolds, 2019.
[65] Meng Liu, Youzhi Luo, Limei Wang. Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. Dig: A turnkey library for diving into graph deep learning research. The Journal of Machine Learning Research, 22(():10873-106 2021.
14

---

Appendix for "A new perspective on building efficient and expressive 3D equivariant graph neural networks"
Contents
Supplementary background
15
Equivariant frame construction
16
Algorithm for the proposed LEFTNe
17
D Related proofs and discussions of Section 3
18
Related proofs and discussions of Section
20
Related proofs and discussions of Section 5
23
G
Extended Related Work
24
Modular overview 3D GNN
24
G.2
Relationship to Geometric WL test
25
H
Neural sheaf interpretation
26
Additional experiments
27
A
Supplementary background
In this section, we first detail the concept of scalarization and tensorization and then review the concept of (contravariant) tensor fields and their associated equivariant group representations.
Scalarization. Scalarization is a general technique that originated from differential geometry for realizing covariant operations tensors [52]. Our method will apply simple version of scalarization
in Rd transform equivariant quantities. At the heart of its realization is the notion of equivariant orthonormal frames, which consist of three orthonormal equivariant vectors:
== (e1, e22 e3).
Based on F. we can build orthonormal equivariant frames for higher order tensors by taking tensor products X, Eq. in Appendix. By taking the inner product between F and given equivariant vector (tensor) x, we get a tuple of invariant scalars (see (30) for proof):
x + क == (x e1 x e2, X es):
(8)
and 7 can be seen as the 'scalarized' coordinates of Tensorization. Tensorization, on the other hand, is the 'reverse' process of scalarization. Given tuple of scalars: (x1, 22, Is),, tensorization creates an equivariant vector (tensor) of
Pairing,
(11, L2, z3)
x = Irei + Z262 + Zze3.
(9)
The same procedure is extended to higher order cases, see Eq. 22 in Appendix.
A order (contravariant) tensor T on vector space V is multilinear map:
T V X V + R!
15

---

where V denotes the dual space of V. In fact, there is canonical 'multiplication' operation between two tensors. Define the tensor product S @ T of two tensors S and T to be tensor of order
S @ T(
te S(v' . TT t)
(10)
where v¹ E V
From now on, we assume V = V* = R Note that when s s= T is exactly an equivariant vector. In practice, the tensor data in R is usually given by its coefficients under Cartesian coordinate system. Take second-order tensor as an example, assume we are given an orthonormal frame (basis) (e1; e2, ea) and its dual frame (e", e )) then the nine coefficients of T are given by
Ti T(e', e'), $
In other words, we say the collection {T, heases is a faithful representation of T in a fixed coordinate system:
Σ The 4
(11)
Once defined tensor R it's easy extend it to continuous manifold discrete graph. A tensor field of order s on a 3D graph G = (V, E) is a tensor-valued function f which assigns to each 3D node x; order tensor, denoted by fx,)..
SE(3) Tensor Representations. Let V be a vector space, then the group SE(3) said act on there is mapping Φ SF(3) x V + V/ satisfying the following two conditions:
if E SF((3 is the identity element, then
Φ(e,x)= = for Va V.
2. if 91,92 E SE(3), then
(02, x)) (09192 x) for Vr V.
If we further require (gg,) )) is a linear map for all g E S3F(3), then e becomes group representation of S (3). From now on, we only consider the rotation subgroup SO(3) and group representations. When V R there is natural representation of SO(3) by rotating vectors in In this way, an element g E SO(3) is identified with Rotation matrix, denoted by {9 higsss.
From the tensor definition 10). this natural representation on R induces a tensor representation on T. Still take T (Tihisa as an example, we have
Th >> ThoT 1skls
(12)
for Ag SO(3). It's easy to check that spanned by second-order tensors.
12 is indeed a SO(3) representation the vector space
Relation with Spherical Harmonics. For the SO(3) group, all representations (including the tensor representations) can decomposed as direct sum of irreducible representations. For each type of irreducible representations, there is a subset of spherical harmonics formulating a basis for this specific representation. However, terms of representing equivariant geometric quantities, the theorem in (55) claims that tensor representations and irreducible representations are equally powerful: They all form a complete basis the space of continuous E3) equivariant functions.
B Equivariant frame construction
Equivariant Frame Definition. An equivariant frame 3D graph G consists of three orthonormal vectors that transform equivariantly under (3)3). In the following, we present specific constructions of equivariant frames. However, it is important note that the proof of Theorem 4.1 does not depend on our particular choice of local equivariant frames.
It is worth noting that the number of vectors required in the equivariant frame is equal to the dimension of the space. For example, in the case of the four-dimensional Lorentz space, where the symmetry group is replaced by the Lorentz group, the equivariant frame consists of four independent vectors.
16

---

Edge-wise frame. From this point onward, we assume that the 3D graph's 's mass has been translated zero. This translation ensures that the center of mass of the graph located the origin, ensuring the translation invariance of the system. Consider node i and one of its neighbors with positions x; and x,, respectively. The orthonormal equivariant frame Fij = (c) e² e3) is defined with respect to x, and X, as follows:
J Xj)
Xj X; X,)
X jj
(13)
X;
X;
X x
Node-wise frame. Consider node i with 3D position x,, and let x; N/4 ExjeN(x.) X; be the center of mass around the I-hop neighborhood of x,. The orthonormal equivariant frame F; == (ej e eqa) defined with respect to X; as follows:
X
XX
X
X; X
(14)
X; x
&
x
|
X;
x;
X x;|
Note that node frames can also be obtained by averaging the edge-wise frames and applying the Gram- Schmidt orthogonalization process. This approach provides an alternative method for constructing equivariant frames at the node level.
C Algorithm for the proposed LEFTNet
The detailed LEFTNet shown in Algorithm 1
Note that for simplicity, the message m in this algorithm could include both invariant and equivariant terms in Fig. 3
The design of LEFTNet (LSE only) used the ablation study is shown Algorithm
Algorithm LEFTNet
Input: 3D graph with equivariant positions X , x,) E R invariant node features
h; E IRd, invariant relative distances dij E IR. equivariant edge features eiy RR
2: Centralize the positions: X < X - Cx.(
3: for (i = 1; <= n; +) do
4: Compute node-wise equivariant frames F; via Eq
14
5: for j E (i) do
6: Compute edge-wise equivariant frames Fij via Eq. 13
7:
Get the mutual 3D structure S, perform local scalarization through Fly:
kij (Scalarize(S...
8:
Calculate the (E(3)-invariant structural coefficients: Ar g(tig, diÿ) Perform equivariant message passing as in Eq.
9:
m. 1n (h,, Arj @ hj, diy) on (hi, As O diy) Bij + (hs Ais hg, Fi
10:
end for
11:
Equivariant message aggregation: m; = EjEN(O) mij:
12:
Transform equivariant node features through F;:
4 = Scalarize F.)
Update invariant node features:
4 (hi, 1i)
14:
Equivariant Output: Perform tensorization through Fi:
h Tensorize(h;; F.).
15:
end for 16: Output: for invariant properties (e.g. energy): AvgPooling(h1.... hn)
for equivariant properties (e.g. force): (hi,..., h,).
17

---

Algorithm 2 LEFTNet (LSE only)
1: Input: 3D graph with equivariant positions X (N1y...,XA)) Xn) E R invariant node features
hú E Rdd invariant relative distances dij E IR.
2: Centralize the positions: X <- - (X).
3: for (i = 1; <= n; +) do
4: for j (i) do Compute edge-wiss wise equivariant frames Fiy via Eq.
5:
13
Fi = EquiFrame xy).
6:
Get the mutual 3D structure Si perform local scalarization:
bi (Scalarize(S..::-3
7:
Calculate the SE(3)-invrriant structural coefficients: Aij - g(Lig, dis)
8:
Perform invariant message passing:
mij = 2m (hs, Ais @ dis)
9:
end for
10:
Update invariant node features:
h4 = Sn Σ mis)
jEN(()
11:
end for
12: Output: AvgPooling (hi...., hn).
D Related proofs and discussions of Section
In section B we proposed a novel hierarchy of local geometric isomorphisms that further motivates the design of incorporating the mutual 3D substructure's 's information into equivariant GNNs. Different from our fine-gra tne-ggained local characterization, cocurrent work GWL (22) proposes to measure geometric isomorphism from local to global by the k-hop partition.
From another point of view, we essentially demonstrated that encoding mutual 3D substructures expands the capacity of the transformation function class with respect equivariant GNN. [23 put forward the Completeness concept for characterizing these transformation functions. However, it mainly concentrates on testing whether function can discriminate global geometric isomorphism (in
the sense of Eq. 3
Discussion the Completeness Concept. Following our terminology in the preliminary section, completeness of a transformation f can be translated into claiming that f is invariant among 3D graphs if and only if they are globally isomorphic (see definition (3). Therefore, it's easy to refine the notion of completeness that adapts to our local version by replacing the global isomorphism to local isomorphism:
((x = S(Y),
if and only if X and Y are local [tree-, triangular- subgraph-| isomorphic. Then, in terms of function class capacities, the following relation holds:
Global complete C Subgraph complete C Triangular complete C Tree complete.
Note that our equivalent description of complete transformation reveals the fact that the completeness concept in 233 is defined from the global 3D isomorphism point of view. Therefore, that the above series of completeness notions belong to the structure completeness. Indeed, the theory developed in section 3 indicates that GNN which can express structure complete functions may not be sufficient expressing general tensor potential functions on a 3D graph.
shall claim
On the other hand, a non-negligible proportion of 3D graph tasks may not be sensitive to the global 3D non-isomorphism. For example, some chemical properties (formulated as a function defined on molecular graphs) are characterized by local substructures (56). In these scenarios, we are looking for geometric transformation f that global non-complete, but (tree-, triangular-, subgraph-) local complete.
18

---

Proof of Theorem 3.1
Remark.
1. When compared to the construction of topological structural coefficients that take the discrete adjaceney matrix of the graph as input, as proposed in [27]. the input 3D structural coefficients we seek are of a continuous nature. For instance, as illustrated in Fig. [ the key features that differentiate these 3D local shapes are the dihedral angle and relative distance, both of which are continuous functions of the 3D positions. Therefore, we require the existence of universal approximators for continuous functions to ensure that the parametrization of As, has sufficient expressiveness.
2. The reason desire the LSE module be invariant is because it ensures the local injectivity of message passing, following the standard argument proposed in Xu et al. [25]. This local injectivity is crucial for proving the second part of Theorem 3.1
3. From a mathematical perspective, the common approach to proving the universal approx- imator property is by demonstrating that neural network can express a complete basis in the function space, typically leveraging the Stone Weierstrass theorem. Two successful examples are: Piecewise linear function bases achieved by wide Multi-layer perceptrons (MLP); b. Polynomial bases. In the 3D GNN setting, equivariant polynomials that are also permutation-invartie through local scalarization and the Kolmogorov representation theorem [57], the universal
can be rigorously constructed using ACE [29]. Additionally,
approximator property of 3D GNNs reduces to the universal approximator property of MLP, which has been proven for broad range non-linear activation functions.
Proof. The first part of the theorem is proved by providing an explicit example. Consider the first pair of 3D shapes in Figure [ where the mutual 3D substructures S,-g and Sj-g' both contain two triangles: (I,, Tp. Fiy), (Ie, Tm, Fig) and (Igz Fyp, Fip)) (Iz, T',, Fgr) respectively.
The difference between these two triangular non-isomorphic shapes can be captured by the distance function:
d(xp, Xm) = ||xp Xm 1|
and similarly, d(xp', xmn' for the second shape. Note that this function cannot be expressed solely by tree-level features since there is no edge connecting Xp (Xpr) and Xm (xmr). The fact that this function produces different output values for the two tree isometric but triangular non-isometric 3D shapes implies that & is capable of distinguishing 3D shapes beyond tree isomorphism.
Therefore, if the structural coefficients Aip (Ajp') take the value of d(xp: X.,) (dl(x,r, X., )), then know that it can differentiate these two triangular non-isomorphic shapes. Thus, it suffices to prove that the encoder á is universal approximator of continuous functions (which obviously contain the distance functions) that takes the mutual 3D substructures S (Sj-q' ) as input.
According to the Stone- Weierstasss theorem, it is sufficient to demonstrate that 6 can express complete basis within the continuous function space. the 3D GNN setting, this basis should satisfy two additional requirements: 1) permutation equivariance, and 2) SE(3) equivariance. An example of such basis is the set of E3) equivariant polynomials that are also permutation equivariant, as demonstrated in Drautz [29]. To obtain invariant coefficients from the combination of equivariant basis functions, we can employ a projection layer, similar to the one used in Dym and Maron [55].
Alternatively, a simpler approach to achieve the universal approximator property is by leveraging local scalarization. Specifically, we use an equivariant frame on the edge eig (as described in the previous section) to scalarize the mutual 3D structure S._, (S)-q'), which includes x, (x,)) and xm
(Xmr). Let 5,-g denote the scalarized 3D positions of Si-q. By quoting the information lossless theorem and the Kolmogorov representation theorem for permutation-invarian functions from Zaheer al. [57] Du al. (30), we know that any y that is a S FF3) and permutation invariant function of S, S.-9 can be expressed as:
( (SS =(
Σ g(z)).
(15)
19

---

where f and 9 are certain continuous functions that can be approximated by wide MLP, following the universal approximator property of MLP 25
The second part of the proof demonstrates that there exists an enhanced message passing framework (under injectivity assumptions, which be realized similarly 25 27 that incorporates the structural coefficients {(ds := o(S,-;) te,,ee) to map two distinct local 3D subgraphs with isometric local tree structures in Figure I different embeddings. Since specifically construct e as an invariant encoder, the collection {{Ais- := ((S_,)) Je, E consists of invariant numbers, and the construction of the message passing framework follows a similar approach to 27 For completeness, we provide a brief review of the injectivity condition and the message passing construction.
To establish the injectivity condition and prove the second part, we introduce the multi-set notation {.) J. following Xu et al. [25]. basic equivariant GNN within our enhanced framework consists of least steps:
Message passing, defined by 4 2. Node-wise update:
h (m,, kf).
For simplicity, we denote the composition of these two steps as Then, the additional injectivity condition requires that V satisfies the following:
T (H Aytht. hgll E E.B. (A,AAA E NGD)
(16)
is injective for each layer and each node i. It is worth noting that this condition can be trivially achieved by incorporating weighted residue terms, similar formula (5) Wijesinghe and Wang [27]. Consequently, based on the injectivity condition, it is evident that two non-identical collections of ((Ai)eaye would yield two different embedded feature vectors.
Furthermore, in the first part, we have proven the existence of at least two distinct local 3D subgraphs with isometric local tree structures such that the corresponding geometric weights {{As) ) e generated by the encoder á are different. Combining both parts, we have successfully demonstrated the theorem.
E Related proofs and discussions of Section 4
Torsion Angle is Secretly Hidden in FT
Recall the edge-wise (signed) torsion angle Tiy [58 involves the 1 I-hop atom pairs i and j and two 2-hop atoms k and then Try is and plane I j-j- - i. Although exhausting all torsion angles requires (kk) complexity, Wang al. (23] reduces the computation (I)) order by selecting a canonical 2-hop atom & and which is enough for detecting the relative orientations between atoms (insufficient for general tasks like many body interactions).
TtJ
defined to be the dihedral angle between plane k -
Yi
X, Figure 5: Tr indicates the rel- ative rotation of two frames along the z-axis.
Now we show how Tiy naturally appears as one of the derivatives from frame transition functions. For node i, define the equivariant frame F, by
(ef, e e²)) = (x; - X,. X; - Xs: eî x ea). Fi is normalized through the Sraum-Schmidt algorithm. For node j. F, is defined similarly by
(ef, e e (x, X, x, of x e
Then following the transition formula
Raj (el, e eg)) (c) e)
Note that for an orthonormal matrix, its inverse is equal to its transpose. Therefore, based on the
standard definition of dihedral angle, have:
Tij e/ ef Ray (3,3).
In conclusion, T. represents only one component of the transition matrix Rij. To fully determine Ryj. we still require two additional angles, as a transition matrix is uniquely determined by three Euler angles.
Proof of Theorem 4.1
20

---

Remark.
1. The crucial property we leverage to prove this theorem is that all local scalars can be expressed through local scalarization using one local frame and a nonlinear transformation. However, it is important to note that the transition between local frames reveals the relative change of local geometries, which is inherently non-local.
2. Interestingly, stemming from the fact that invariant scalars can be expressed using scalariza- tion and a universal approximator, we can deduce that by incorporating the frame transition matrix into the union set of S3 and Sc, all (continuous) invariant interactions fa (B, C) can be expressed.
Proof. This theorem is proved in two steps:
1. The first step characterizes all scalars SB (Sc) determined by isolated local clusters B and C through equivariant frames and local scalarization;
2. The second step constructs specific invariant function fo C) that cannot be expressed by taking the union of Sg and Sc: {SB || Sc).
Let G denote a 3D point cloud. As proven in Du et al. 30 when equipped with an equivariant frame Fc, all equivariant features of G can be transformed into scalar features through scalarization without information loss. More precisely, follow the convention in the main text, let G be the output of performing scalarization (using any equivariant frames purely determined by G. see Section B for examples) on the 3D point cloud G. For any invariant function f(G), there exists a corresponding function f such that:
(17)
Since SG represents the collection of all invariant scalars produced by G, which is equivalent the collection of all invariant functions that depend on the 3D point cloud G, Eq. 17 implies that Sa can be generated by a finite set of invariant scalars G. To apply this insight to our current theorem, need build two local equivariant frames B B and Fc = (ef, e Ee.))) Importantly, the frame FB itself doesn't 't depend on C, and scalarization through FB is only performed the local 3D point cloud B. Therefore, performing operations like scalarizing equivariant information of C through FB would violate the assumptions of the theorem.
we have two different 3D clouds B and C. Therefore,
FB = (cf 2, 3
We are now ready to construct an explicit counterexample fa C) using FB and Fc:
(B, :== of.
Since fa is the inner product of two equivariant vectors, it automatically becomes an invariant function. Next, we check whether fa (B, C) can be expressed as a function of (( || ScJ). The equivariant component of fo related B is precisely ef. Applying the above local scalarization principle, we scalarize B through F3 and obtain:
B + =
Similarly, ef is also transformed into a constant scalar tuple ét = (1, through Fc. As constant inputs generate constant outputs, conclude that the derived local scalars can only approximate constant functions. However, since the local frames change as we vary the 3D structure of and C, it is evident that of ef is not a constant function of (B, C). Thus, we complete the proof by contradiction.
Similarly, of is also transformed to a constant scalar tuple of = 0, through Fc. As constant inputs generate constant outputs, we conclude that the deduced local scalars can only approximate constant functions. However, since the local frames are changing we vary the 3D structure of B and C, it's obvious that et Ef not a constant function (B, C). Therefore, we finish the proof by contradiction.
L
21

---

Further Comments.
1. is important to note that the counterexample we constructed is not the only valid counterex- ample. For instance, the norm of any tensors that depend on both and C (e.g.. ||f + ffl:
2 & Eg can also serve as valid counterexamples.
2. Let RBC denote the frame transition function between F' and Fc, defined in the same way as in Equation 5 The proof of Theorem 4.1 implies that all types of invariant interactions fa C) can expressed solely as functions of the union of three sets SB, Sc, and RBC {SB || Sc RBC). This is because, through RBC. the 3D point cloud C can be scalarized by the local frame FB. Specifically, let SC. Se denote the scalarized 3D point cloud of Sc with respect to FB and Fc, respectively. We have the following diagram:
Sc
Fc
SC
Rac, Sc
Utilizing formula 15 on the union of the two 3D point clouds, (), leads us to this conclusion.
Realizing FT by Equivariant Messages: From the FT definition 5 each element of the 3 x 3 matrix Rig is calculated by
(ky - e e/-
(18)
Now we show how reproduce R,y(k, 1) through equivariant messages. Let the equivariant message m, be the following:
1 0
0
0
0
0
== (ef, e e//)
0 0
0 0
0
0
0
0
0
1
It's easy to check that m, E R 3x9 consists of 9 equivariant vectors (multi-channels), For atom j. m is defined symmetrically. For each node, we also store the scalar messages, e.g., ||| for < k < 3. Flattening the whole matrix Rij into R 1x9 array, then Rij is obtained by simple summation and taking the vector norm:
[ + myll +
{ e ef ighig3"
where the norm is taken for each column of m, + my. such that |m + m,I E R1x9, Then,
1 R3j 3 e + e!ll ||
|
Our illustration also demonstrates the importance of keeping multi-channel tensor messages.
Relation with Previous Equivariant Update Methods. Following the efficiency principle tablished in section 4 we don't encode the data of the transition matrices explicitly. Instead, we implement tensor messages fill the expressiveness gap. Among the tremendously different designs of equivariant graph neural networks, Schütt et al. [35] is closely related to our equivariant updating method. By the above argument, the inner product operation for node (see (9) of Schütt et al. 355
Uv Vv:
can also be reinterpreted as a realization of the (aggregated) frame transition matrix (5)
Moreover, since the equivariant vectors Uy, and Vy; are both aggregated vector features that belong to the same node i and the inner product operation between them is performed in the wise-wise updating phase, Schütt et al. [35] actually avoids the 2-hop O(k2) complexity of computing Ray for all neighborhood node pairs (x,y) y) (while able to express the torsion angle implicitly). For our algorithm, we utilize the scalarization and tensorization in the node-wise updating phase. By the universal approximation theorem 5.1 method can approximate any inner product operations.
22

---

Related proofs and discussions of Section 5
Equivariant Frames and Higher Order Scalarization and Tensorization. Given an edge eij with two atom positions (xi, X2), we define our edge-wise SB(3) equivariant frames Fig as follows:
x
x; Xj
X x Xj X; Xj
(e1, e2, 03) =
(19)
X X3
To ensure frame translation invariance, we adopt the approach followed by previous works [59] 60 by restricting the entire 3D conformer space to linear subspace where the center of mass (CoM) of the system (either the entire system or the sub-cluster to which i and j belong) is set to zero. Alternatively, constructing an E(3) frame is also possible but requires an additional atom position X which be selected using the K-Nearest Neighbor algorithm. If Xj, Xx) spans the 3D space, we obtain an (33) equivariant frame by performing Gram-Schmin orthogonalization. For different constructions of E3) frames, readers refer to Wang and Zhang [6]]. However, in this work, focus on SE(3) frames since molecular 3D conformers exhibit S (E symmetry but not possess reflection symmetry.
Once we have an equivariant frame, every vector linear combination of the three orthogonal vectors in the frame. Moreover, the unique combination coefficients are exactly the 'scalarized" coordinates in (8. A similar procedure also applies to higher order tensors. Indeed, the vector frame F extends to a tensor frame F of arbitrary order r > 1:
F
fei,
@ e1, hish.....
(20)
Since the orthonormal frame F is complete in the sense that spans the whole tensor space of order r, every r-th order tensor admits a unique decomposition:
Σ
Thrme
e,
....
&e, e,,-
(21)
1S
It's easy to prove that the collection (Thimntte hs consists of invariant scalars. We call the process from T to (Th.... |ss scalaraization.
Tensorization the inverse of scalarization, in the sense that it sends scalars (7 ) Ii 1, to tensor T. Under the same frames we use during scalarization, the following diagram demonstrates the pipeline of producing I second-order tensors out of (Tr hssii
T
T12
7197
THL T
T12
T
el.
e2
(T....., Tx =
T
222
2
2
T2
O
e2
e
ea
e2
e2
C3
T
T32
3
T7
T2
T7.
ey
X e1
es
X e2
es
e3
1. channelss
(22)
where denotes the element- product.
Proof of Theorem 5.1
Proof. The proof relies on the invertibility of Scalarization and Tensorization operations, as demon-
strated in Appendix A.5 of Du et al. [30]. This invertibility allows establish commutative
T'-1
T
diagram as follows:
Scalarize
Tensorize
MLP
71-1
This diagram illustrates that for each mapping p. there exists corresponding "scalarized" mapping ò given by:
e = Tensorize p Scalarize.
By applying Tensorize, followed by and then Scalarize, we obtain an invariant representation p. Since MIP serves as universal approximator of invariant functions, we can always find an MI.P that expresses p. By reversing the arrows, conclude the proof.
23

---

Proof of Equivariance for LEFTNet LEFTNet consists of multiple layers of LSE and FTE. LSE realized by scalarization, and FTE realized by scalarization and tensorization. Since the invariance of scalarization and the equivariance of tensorization have been proved, we finish the proof.
G Extended Related Work
Table 4: Categorization of representative geometric GNN algorithms. denotes partially satisfying
the requirement.
Method
Symmetry
LSE
FTE
Complexity
SchNet
B(3)-invariant
(
EGNN
-qquivarra
/
((nk)
GVP-GNN
31
X
/
O(nk)
ClofNet 30
SE(3)
eqqiivaiuui
X
X
O(nk]
PaiNN
35
E())-qqiivariant
X
V
O(nk)
ComENet
23
SE(3)
V
(
TFN 13
SE(3)E(()eqquivarra
O(nk)
Equiformer
SE(3VE())equiivaiian
O(nk)
SphereNet 15
SE(3)-invariant
O(h
GemNet 37
SE(3)-invariant
O(nk-))
LEFTNet (Ours) SECG/EE()-cquivariint
(nk)
Based on the discussions in Section g and 4 we identify two essential components for constructing
expressive equivariant 3D Graph Neural Networks (GNNs): (1) Local 31) Substructure Encodings
(LSE), which enable the local message passing to capture diverse local 3D structures; and (2) Frame Transition Encodings (FTE), which incorporate equivariant coordinate transformations between different local patches into the 3D GNN.
G.1 Modular overview of 3D GNN
Based on the general notion of Local 3D Substructure Encodings (LSE) and Frame Transition Encodings (FTE), we have provided concrete constructions to realize them. However, it is important note that there are also implicit methods to encode the information of LSE and FTE. In order to provide a comprehensive overview, we review previous 3D Graph Neural Networks (GNNs) that follow this framework and summarize the findings Table To ensure a fair comparison, we include the computational complexity as it is often trade-off with expressiveness. A detailed analysis of this trade-off provided at the end Section
Regarding Local 3D Substructure Encodings (LSE), SphereNet [15] and GemNet 37 (implicitly)
encode local 3D substructures through computationnintensiiv edge-based update. We indicate these two architectures with a in Table due to the following reasons: In the case of GemNet [37], the message passing framework considers the 1-hop neighborhood with respect to both end points of an edge Cij, which inherently contains all the nodes of the mutual 3D subgraph S; However, when compared to SphereNet (15), which encodes all node positions under an angular coordinate system, GemNet [37 only captures partial geometric information. On the other hand, SphereNet (15) utilizes an edge-based update that aggregates 1-hop neighbors with respect to source node. While this approach considers the immediate neighbors, it may not include all the nodes inside the subgraph S,-j (c.g., the neibors of the target node).
Regarding Frame Transition Encodings (FTE), we have demonstrated (see the proof of Theorem 4.1 that the essential information of FTE does not rely on the specific choices local equivariant frames. Therefore, most 3D GNNs with equivariant vector updates that can incorporate at least three (multi-channel) independent equivariant vectors as edge features are capable of expressing local frame transitions (FT). However, EGNN [14] is an exception as it only updates the position vector (i.e., one channel), which is insufficient for capturing
entire FT. In other words, the power of the update function @ in Equation (6) also influences the encoding of FT. Additionally, models that encode torsion angle information partially express FTE, as illustrated in Appendix F While there are multiple ways realize LSE and FTE, there is trade-off between efficiency and expressiveness in

---

terms of the number of hops considered for message passing, as indicated in the last column of Table
4
In contrast to our invariant realization of LSE, Batatia et al. 24 constructs their framework by building a complete (F((3 permutation) equívariant polynomial basis using spherical harmonics and tensor product, where the monomial variables are a combination of 3D features from different nodes (bodies). On the other hand, we achieve the function of LSE and FTE through edgewise scalarization Aij and equivariant message passing (see Fig. 3p. While Batatia et al. 24 utilize the atomic cluster expansion (ACE) mechanism, we provide an illustration of how to equivariantly realize LSE based on local ACE in the proof of Theorem 3.1 Another work Wang and Zhang [6]] uses an ensemble of frames to model the local environment defined through a distance cutoff. which is suitable for pure 3D geometry. However, our local 3D hierarchy incorporates well with the 2D topology. Therefore, whether [61] encodes LSE depends on the radius of the local environment and whether the ensemble of 'frames' purely built combination of radical directions (see Equation (6) in [61]) spans R In comparison to geometric GNNs like SphereNet [15] and GemNet [37], which encode rellectionannisymmeete torsion angles, and LeftNet, which implements (3) equivariant, reflection- antisymmetric frames, the ensemble of frames in [61] is reflection symmetric and therefore cannot differentiate local chemical isomers. Additionally, Wang and Zhang [6 defines the "(ensembled) frame to frame' projection, which is somehow equivalent to our FTE, and utilizes it to detect the global isomorphism from local observations. In contrast, our FTE is employed to bridge the expressiveness gap of global continuous functions, such as regression tasks on graphs (see the comments in Section E}, rather than focusing on detecting global isomorphisms, as in classification tasks on graphs. Moreover, we have demonstrated that using one SF((3 equivariant frame for each local patch is sufficient for achieving local expressiveness, considering efficiency as well. Finally, we provide another invariant realization of FTE, inspired by [62]. which is presented at the end of Section H
G.2 Relationship to Geometric WL test
Recently, Joshi et al. 22 propose a geometric k-WL test (GWL.) to assess the expressiveness of geometric GNN algorithms. In essence, our tree isomorphism aligns with the I-hop geometric isomorphism introduced in GWL, while the fine-grained triangular isomorphism falls between the 1-hop and 2-hop geometric isomorphism described in GWL. From model design perspective, we achieve the realization of LSE through local scalarization, which guarantees expressiveness through the Kolmogorov representation theorem (Zaheer al. (57)) and the universal approximator property of MLP. Different from our geometry perspective, the fundamental concepts of body order and tensor order, which stem from classical inter-atomic potential theories and exhibit equivariance, play a crucial role in measuring expressive power in Joshi et al. [22] In addition to the local geometric isomorphism hierarchy, we discover the significance of FTE as the connecting bridge between local invariant scalars and global geometric expressiveness. This realization, along with LSE on mutual 3D substructures, sheds light on the insufficiency of the I-hop local scalarization implemented in CloiNet ( Du et al. (30). We further explore the connection between FTE and the neural sheaf Laplacian in Section H
Computational efficiency of LEFTNet In the analysis of a graph with nodes and an average of l edges per node, the computational efficiency of a message passing-based based graph neural network (MPNN) is typically described by the following form:
O(fn + gnn .nnggn))
(23)
where f represents the computational cost of the node-level updating function (usually an MLP), and g represents the computational cost of calculating the message for each edge. The power i crucial factor that determines the computational cost of performing each message passing operation. For instance, SphereNet (Liu et al., 2021) aggregates all hhhop nodes with respect to the source node of an edge to construct a message, resulting in a complexity of O(nk2) for message passing.
To analyze the computational complexity of LEFTNet, we can divide it into the following computa- tional steps:
Building the Si-, and scalarizing with respect to edge frame Fij. The 3D subgraph Siyj is constructed by intersecting the hhhop neighbors Nr and Ng. which requires O(nk)
25

---

computations. Additionally, collecting the I-hop neighbors for each node has complexity of (nk).. Finally, scalarizing each Si-j with the edge-wise frame F,, (which involves taking inner products) and extending the result to channels has complexity of (mmk). Therefore, the total complexity for this step remains at (mmn)).
2. Equivariant message passing. From the first step, we get the scalarized S from the 3D subgraph Si-j. Then, the edge-wise message is obtained by transforming the scalarized S,-j with MLP, and the output invariant scalars is multiplied with both the invariant and equivariant node features (let d denotes the dimension of the node feature, then the multiplication's complexity is O(dnk). Let denotes the complexity of MLP (depends on the depth and hidden dimension of MLP), then the message passing takes O(d 1)nk) computation.
3. Node level updating. This step involves nodewise scalarization, applying an MLP, and performing tensorization. Suppose the input and output channel numbers for equivariant tensor features are both d, then the scalarization using the node frame (similar to step 2) has a complexity of ((d)) computations. Similarly, tensorization, which is the inverse operation of scalarization, also has a complexity of (dd). Therefore, the total complexity for this step is ( + d)n).
In conclusion, we have analyzed the three steps required for performing one layer of LEFTNet. It is worth noting that the efficiency of LEFTNet comparable to other I-hop based GNN algorithms like SchNet [28] (c = in Eq. 23 The final complexity is Oldnn + (d + + 1)mk),, which is
summarized in Table 4
H
Neural sheaf interpretation
Since our invariant scalarization, utilized in the LSE module, shares similarities with the scalarization
technique on vector sheaves (see Hsu (52).
natural to explore potential interpretation of LEFTNET within the framework of neural sheaf diffusion proposed by Bodnar et al. [62]. In this section, we aim to modify the neural sheaf diffusion architecture to preserve SE(3) equivariance by incorporating our node and edge frames.
We begin by revisiting the concept of neural sheaf and its associated sheaf Laplacian operator. cellular sheaf over a discrete graph is a mathematical object that assigns vector space to each node and edge in the graph and specifies linear map between these spaces for each incident node-edge pair:
Definition H.1. A cellular sheaf FG on a graph G = (V, E) consists of:
A vector space F(V) for each E V.
A vector space F(e) for each E F..
A linear map Fuge: F(v) F(e) for each incident u $ node-edge pair.
In the case of a 3D graph neural network (GNN), the node-wise features typically consist of tensors (as defined in TO), which naturally form vector spaces. The key aspect of defining a sheaf lies in determining the specific assignments of the linear maps Furse for each node v and each edge e. For non-geometric GNNs, Fust captures how the opinions (represented as vectors in F(v)) manifest in "discourse space" formed by F(e). In our approach, denoted by F, and assign an equivariant frame to each edge denoted by F.. By defimition, F. and Fe capture the local geometry around the node and the edge, respectively. Thus, natural choice for Fuge is given by:
equip each node with equivariant frame
Fuge = F. E 0()).
To ensure SE(3) equivariance in the sheaf structure, we define F(U) as the set of scalarized vectors obtained by applying scalarization using F... It is important to note that the linear map F, between vectors can be extended to linear map between higher-order tensors through tensor product operations (see Section FD. This construction introduce the sheaf Laplacian operator that defines
With well-defined sheaf structure, Baaooongaraaa guarantees entire structure to
26

---

transformation between node features:
Lr(x) = Σ (Fgele - Fugettu)
In our case, the sheaf Laplacian can be expressed as:
(x(
= Σ
Do F Fu
(24)
ve
using the properties of orthogonal transformations. Thus, this operator quantifies the collective "disagreement of local geometries" at each node. Neural sheaf diffusion propagates information through nodes using the following partial differential equation (PDE):
( X,
(25)
Here, X represents the node feature matrix. In accordance with the nonlinear parametrization proposed in [62], we have the following model:
+) = o -IIn Lz) @ Wi)x((WW.).
(26)
where any applicable nonlinear activation function can be utilized since our cellular sheaf and its Laplacian are invariant. It is important to note that the key component of L> (denoted by F F, in
Wq. 24 corresponds to the frame transition (FTE) module between node frames. Hence, Equation 26 provides invariant realization of the FTE module based on equivariant node frames. For
equivariant outputs, tensorization can be applied after the final layer Equation (26). To incorporate the LSE module,
include the LSE information part of invariant node features to Fu prior to the message aggregation step in Equation (26).
I Additional experiments
Experiment Detail for Table I and 2 For QM9, baseline results are taken from Liao and Smidt 36 For MD17, baseline results are taken from the original papers (with unit conversions needed). All models are trained on energies and forces, and WoFE the weight of force over energy in loss functions. All experiments are conducted on a single NVIDIA GeForce RTX 2080 Ti 1 IIGB GPU,
Our implementation is based on the libraries including PyTorch 63
PyG
64
and DIG 65]
Ablation Study. As discussed in Section 5 there are two main modules in LEFTNet, namely LSE and FTE. We conduct experiments on QM9 and MD17 to show the importance of each component. Experimental results are summarized in Table and Table 6 The results show that using LSE can outperform the model without both LSE and FTE all tasks. Adding FTE can further improve the performance. The results demonstrate the importance of LSE and FTE modules.
Table 5: Ablation study on QM9 dataset. The evaluation metric is MAE for each property. The best performances are bolded and the second best are underlined. Detailed LEFTNet (LSE only) is shown in Algorithm 2 LEFTNet (LSE vector FTE) is our LEFTNet introduced in the main paper, and the detailed algorithm in Algorithm LEFTNet (LSE + tensor FTE) means the message passing and updating contains higher order tensors, built by Eq. 22
Task
As
C, calémol
G
H!
Rd
U
Un
7PVE
LEFTNet (wlo LSE and FTE)
.053
49
33
25
.038
.026
9
A255
1.59
LEFTNNe (LSS only)
(143
.031
.0.5
.156
1.34
LEFINet (LSE vector FTE)
039
39
.011
.022
.094
1.19
LEFTNet (LSE tensor FTE)
.038
38
22
17
.011
.022
2
.096
120
Results on rMD17. Following Batatia et al. 24 we conduct experiments on rMD17 to compare
with recent studies. Results show that our LEFTNet achieve comparable performance to state- of-the-art methods such as MACE and NequlP, while outperforming other baseline methods like GemNet and PaiNN.
Model and training hyperparameters. Model and training hyperparameters for our method on different datasets are listed in Table
27

---

Table 6: Abalation Study on MD17 dataset. The evaluation metric is MAE for per-atom forces prediction (kcal/mol A). The best performances are bolded and the second best are underlined. Detailed LEFTNet (LSE only) is shown in Algorithm LEFTNet (LSE + vector FTE) is our LEFTNet introduced the main paper, and detailed algorithm is in Algorithm LEFTNet (LSE +
tensor FTE) means the message passing and updating contains higher order tensors, built by Eq. 22
Molecale
LEFTNEt (wio LSE and FT) LEPTN (LSE only)
LEFTNer (LSSE voctor FTE)
LEFINet tensor FTE)
0.451
0.210
Ethunol
0.341
0.149
0.138
0.118
Malonaldehyde
0.594
0.276
0.309
0.159
Naphthalene
0.175
0.063
Salicylic acid
0.828
0.313
0.167
0.141
Toluene
0.625
0.166
0.084
0.070
Uracil
0.581
0.206
0.116
0.112
Table 7: Mean Absolute Error for energy(meV) per-atom forces prediction (meV Á) on rMD17 dataset. Baseline results are taken from Batatia et [24]. The best results are bolded.
LEFINet
MACE
Allegro
BOTNet
NequlP
GemNet (T/Q)
ACE
FCHL
PaIN
2.1
2.2
2.3
2.3
2.3
6.2
17.7
16.6
6.9
Aspirin
6.4
7.3
8.5
8.2
9.5
17.9
20.9
44.9
40.6
16.1
0.7
1.2
1.2
0.7
0.7
3.6
2.8
8.5
15.9
3.3
2.6
3.3
2.9
10.9
10.8
24.5
35.4
0.05
0.3
0,03
0,04
0.04
0.35
0.75
0.3
0.2
0,3
0,3
0.5
Ethanel
0.4
0.4
0.4
0.4
1.2
2.5
2.7
3.6
2.1
3.2
2.8
3.6
7,3
18.1
13.4
Malonaldehyde
0.8
0.8
0.6
0.8
0.8
1.7
1.5
4.8
4.6
3.9
5.4
3.6
5.8
5.1
6.6
11.1
10.3
26.4
24.5
13.8
Naphthalene
0.8
0.5
0.2
0.2
0,9
0.9
1.2
3.8
11.3
5.1
1.9
1.6
0.9
L.8
1.3
1.9
5.1
29.2
3.6
Paracetamol
1.3
1,3
49
1.5
5,8
L33
5.9
1.4
12.7
2.9
8.5
11.5
Salicylic acid
0.9
0.9
0.9
0.8
0.7
1.8
1.8
5.6
9.2
4.9
4.1
2.9
4.3
5.3
Toluene
E
0.3
0.5
D.4
0.3
0,3
1.1
1.7
7.7
4.2
1.8
1.9
2.2
Unaiil
E
0.4
0.5
0.6
0.4
0.4
1.1
D.6
5.1
4.5
3.2
3.8
Table 8: Model and training hyperparameters for our method on different tasks.
Values/Search Space
Hyperparameter
QM9
MD17
rMD17
Number of layers
4, 6
4,
4,
Hidden channels
128, 192, 256
256
Number radial basis
24, 32, 96
16, 32,
16, 32, 64
Cutoff
6.5,
8, 10
8, 10
Epochs
800
1000
1000
Batch size
32
1.4
1.4
Learning rate
le-4, Se-4
5e-4
Se-4
Learning rate scheduler
steplr
stepir
steplr
Learning rate decay factor
0.5
0.5
Learning rate decay epochs
100
200
28


Reviews:
[{"review_id": "PqSZC3PB6s", "reviewer": "Reviewer_Wt9q", "summary": "This paper investigates expressive message passing architectures for processing 3D geometric graphs with permutation and Euclidean symmetries. The authors first provide an investigation of a local hierarchy of isomorphism separability. Then, the authors show cases where powerful local invariant models may fall short of encoding global structures due to information loss during propagation, and aim to reduce the information loss by turning to equivariant message passing incorporating frame transition matrices that are related to neural sheaf diffusion. Based on the theoretical framework, the authors propose a message passing architecture that is expressive in terms of the local isomorphism separability as well as incorporating frame transition. Experimental results on invariant molecular property prediction on QM9 and equivariant force prediction on MD17 shows that the proposed approach outperforms previous equivariant neural networks.", "questions": "The questions are included in the weakness section.", "limitations": "The authors have addressed the limitations of their work in Section 7.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. This paper addresses an important problem of understanding the theoretical expressiveness of geometric graph neural networks and developing architectures that are provably expressive.\n\nS2. The local isomorphism hierarchy proposed in Section 3 is novel as far as I can tell (in context of 3D geometric graphs) and correct.\n\nS3. The visual example shown in Figure 1 was helpful in understanding the proposed hierarchy of local isomorphism.\n\nS4. The connection of the proposed approach to sheaf diffusion mentioned in Appendix I was interesting.\n\nS5. The experimental results and ablation study on QM9 and MD17 seems to support the main arguments.", "weaknesses": "W1. The equality in Eq. 3 seems not correct as it neglects permutation symmetry. If we consider positional features represented as matrices $\\mathbf{X}\\in\\mathbb{R}^{n\\times 3}$ (Line 77), the precise symmetry under consideration is a direct product of permutation group and Euclidean group $S_n \\times SE(3)$ (Dym et al., 2020). I think it should be clarified that the equality in Eq. 3 is up to permutation of nodes.\n\nW2. In Line 111, I think  {-tree, -triangular, -subgraph} should be {tree-, triangular-, subgraph-}. Same applies to Line 603-604 and Line 612.\n\nW3. In the definition of tree and triangular isometries (Line 114-118), is there any reason to use the term tree and triangular? I think the definitions are a straightforward extension of the subtree and overlap isomorphism proposed in Wijesinghe et al., 2022, respectively, and I see no reason to use different name. Especially, for the triangular isometry, I found the name misleading since the isometry condition is imposed on the entire set of triangles that share $e_{ui}$, rather than individual triangle.\n\nW4. In the definition of triangular isometry (Line 116), the condition is specified for each edge in the intersection of the local subgraphs $e_{iu}\\in\\mathbf{S}\\_{i-j}$. Please correct me if I am wrong, but to make sense, I think the condition should be for each edge in one of the local subgraphs, $e_{iu}\\in\\mathbf{S}\\_{i}$.\n\nW5. In Line 606, the term structure completeness is used without proper definition (although it is emphasized by boldface). Without clear definition, I was not able to properly understand and correctness of the argument in Line 608.\n\nW6. The argument in Line 162-164 is ambiguous; do you mean a composition of Atoministc Cluster Expansion and invariant map constructed using scalarization by edge-wise equivariant frames is a universal approximator of SE(3) invariant functions, so it can serve as $\\phi$ in Theorem 3.1? Is the proof of universality available in literature?\n\nW7. For LSE, based on Appendix H, it seems that the hierarchy of local isomorphism proposed in this paper is a subcase of the geometric WL test proposed in Joshi et al., 2023. Since this paper provides 3-way classification while GWL provides a more sensitive separation of expressive power of geometric GNNs based on the orders of node tuple and feature tensors, one can argue that the local isomorphism hierarchy proposed in this paper is less of practical interest compared to GWL.\n\nW8. For FTE, in Section 4, frame transition is introduced to solve the limitation of SE(3) invariant messages as outlined in Theorem 4.1. But as far as I know, a majority of practical equivariant geometric GNNs in literature are already using equivariant (tensor) messages (Thomas et al., 2018; Fuchs et al., 2020; Satorras et al., 2022; Geiger et al., 2022; Kohler et al., 2019), and the expressive power of some of them (TFN (Thomas et al., 2018) and SE(3)-Transformers (Fuchs et al., 2020)) are shown to be universal i.e., maximally expressive (Dym et al., 2020). Is the motivation of frame transition also relevant for these architectures? Also, is there specific reasons to favor the proposed frame based propagation over these already equivariant message passing algorithms, despite the added complexity of incorporating node- and edge-wise frames?\n\nW9. In Line 214, does the frame here refer to frames in Riemannian manifold theory? Since there is no proper definition (Eq. 5 provides a description of key property but is not a definition), I was confused on what exactly are the frames mentioned here and was unable to verify whether Line 216-221 is correct. It seems the precise definition and descriptions of equivariant frame are completely deferred to Appendix B and F although it serves as a main component of the proposed architecture, which makes the paper hard to read and understand; I think key parts of the proposed algorithm should be described in the main text in a self-contained manner.\n\nW10. Overall, I find that the writing of the paper has room for improvement in terms of organization and readability. There are many cross-references back and forth across the paper, so I couldn't serially read through each section without going through a large portion of the entire text multiple text. For example, Line 236-237 (Section 4) refers to Figure 3 (Section 5), Line 555 (Appendix A) refers to Eq. 21 (Appendix F), Line 592 (Appendix C) refers to Figure 3 (Section 5), Algorithm 1 (Appendix C) refers to Eq. 4 (Section 3), and so on.\n\nW11. In Eq. 7, I don't see how the equation describes equivariance. I think it should be something like $\\mathbf{m}(g\\mathbf{x}\\_u) = \\sum\\_{i=0}\\^l\\mathcal{M}\\^i(g)\\mathbf{m}\\_i(\\mathbf{x}\\_u)$, is this a typo?\n\nW12. Typo in Line 1 of Algorithm 1, gragh -> graph\n\nW13. Specification of the output is missing in Algorithm 1, and the output in Algorithm 2 is not clear since normal and boldfaced h are mixed in the algorithm. Furthermore, In Line 7 of Algorithm 1, how do you scalarize the mutual 3D structure, which is a subgraph, based on edge frame $F_{ij}$? I don't see how this can be done in Eq. 8 which defines scalarization. Similarly, in Line 6 of Algorithm 2, how do you scalarize the mutual 3D structure without frame? Also, if the algorithm 2 is LSE only, why should one compute edge-wise frames in Line 5?\n\nW14. In the proof of Theorem 5.1, I can see that the proposed parameterization is universal, but does it correctly guarantee equivariance as well (as specified in Theorem 5.1)? The current proof seems to clearly show universality, but I am not sure about equivariance.\n\nDym et al., On the universality of rotation equivariant point cloud networks (2020)\n\nWijesinghe et al., A new perspective on \"how graph neural networks go beyond weisfeiler-lehman\" (2022)\n\nJoshi et al., On the Expressive Power of Geometric Graph Neural Networks (2023)\n\nThomas et al., Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds (2018)\n\nFuchs et al., SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020)\n\nSatorras et al., E(n) Equivariant Graph Neural Networks (2022)\n\nGeiger et al., e3nn: Euclidean Neural Networks (2022)\n\nKohler et al., Equivariant Flows: sampling configurations for multi-body systems with symmetric energies (2019)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer Wt9q\n\n> typos (W2, W4, W11, W12, W13 (Alg.1 and 2))\n\nThanks for your meticulous examination and useful feedback. We will fix them in the revised version.\n\n> Other questions\n\n**Due to 6k characters' limitation, we only provide short answers. We are happy to explain more if you have further questions. In addition, discussions about frame transition and its realizations (W8 and W9) are provided in the [general response](https://openreview.net/forum?id=hWPNYWkYPN&noteId=1Xmyobc4x8).**\n\n- W1: Eq.3 permutation symmetry: Will clarify that Eq. 3 is up to the permutation of nodes\n\n- W3: name of the isomorphisms\n\nThe sub-tree isometry corresponds to our tree isomorphism as the name suggests. However, the overlap isomorphism is subtle, in the sense that overlap is an obscure terminology for formal definitions. This obscurity becomes more severe when both 2D and 3D structures are present. **Therefore, we propose a more intuitive “ triangular” as an alternative.** But, you are right that triangular isomorphism is based on the entire set of triangles that share (a bundle of triangles that share a common edge). We will add a remark to directly point out that the triangular isomorphism corresponds to 3D overlap isomorphism.\n\n- W5: definition of structure completeness\n\nSorry for the ambiguity. The structure completeness in Appendix refers to the ability to classify non-isomorphic 3D structures (the formal definition is in [1]). In other words, if a neural network can classify non-isomorphic 3D graphs into different categories, we say that this neural network is structure complete. The reason we add the restrictive prefix 'structure' is to emphasize that this completeness doesn't imply a neural network has the universality of approximating any continuous functions as Sec.4 discussed.\n\n- W6: about Line 162-164\n\n1 and 2 are two parallel ways to achieve local universal invariant approximator: 1. Applying local frames will first transform all equivariant quantities to invariant scalars (this procedure doesn’t lose information, as this transform is invertible, see [2], then the rest of the argument follows the universality of MLPs. 2. Atoministc Cluster Expansion can express equivariant functions universally, then the invariant encoder can be built by adding another projection layer (that transforms equivariant output to invariant scalars). This layer can be found in [3] and since ACE is composed of taking the tensor product of equivariant vectors for all orders, and the tensor product type of universality has also been proved in [3], and therefore we neglect some details.\n\n- W7: compare to GWL\n\nThanks for asking this insightful question. First of all, our setup is different from [4] but highly related. We seek to implement expressive (and efficient) equivariant graph neural network that avoids higher-order message passing scheme. Thus, we only considered one-hop message passing. The high level philosophy of our local approach is similar with [5], which also goes beyond WL by encoding local structural coefficients. On the other hand, [4]’s theory is inspired by classical potential theory of many body interaction systems, which is complementary to our geometry view.\nSecondly, we are inspired by [5] such that we aim to develop a local hierarchy of isomorphism that could characterize the expressiveness of 3D equivariant GNNs in a finer scale. We actually believe our local isomorphism hierarchy would be of practical interest to the community in the sense that we can quickly design efficient algorithms (e.g., LEFTNet) to improve expressiveness within one-hop region. We do agree a concrete technical connection between the local isomorphisms and GWL would be interesting (such as Theorem 2 in [5] as a 2D analogy) for future works as GWL is also a recent work.\n\n- W13:\n\nAlg.1: We will add the detailed output, which should be the output block(see the framework in fig.3) which takes all node features (invariant & equivariant) for final prediction.\n\nAlg. 2: Typo here. All features should be invariant.\n\nLine 6 of Alg.2: Typo here. It should be the same as line 7 of Alg.1.\n\nThe scalarization step (details in Fig.3 and line 258-265):\nThe scalarization of a subgraph is realized by scalarizing all the 3D position vectors within this subgraph. In summary, the mutual 3D structure $S_{i-j}$ is associated with edge $e_{i-j}$, and we build an equivariant frame for each edge. Then, we scalarize all the 3D particles inside $S_{i-j}$ through this edge frame by Eq. 8. In summary, LSE requires edge equivariant frames, and FTE requires node frames in our implementation.\n\nW14: equivariance of Theorem 5.1\n\nSince the key ingredient of proving equivariance is similar with previous [2]'s appendix (also by chasing the lower part of the commutative diagram in Line 786), we neglect some details on the equivariance side of the proof. Now we are happy to fill in the details. Roughly speaking, the equivariance is proved by showing that the composition of scalarization, the MLP, and the tensorization is equivariant. The equivariance of the composition of scalarization and MLP is obvious since scalarization turns equivariance into invariance, and then the MLP is a transformation of invariant scalars. On the other hand, the tensorization defined in Line 779-782 is a pairing of equivariant tensors and invariant scalars, which is equivariant by definition. Combining the two steps, we have proved that our implementation of FTE is equivariant.\n\n**Ref**\n\n[1] ComENet: Towards complete and efficient message passing for 3D molecular graphs.\n\n[2] SE (3) equivariant graph neural networks with complete local frames\n\n[3] On the Universality of Rotation Equivariant Point Cloud Networks\n\n[4] On the expressive power of geometric graph neural networks\n\n[5] A new perspective on\" how graph neural networks go beyond weisfeiler-lehman?\"\n\n**Note that we provide the response of W8-W10 in the general response.**"}, {"review_id": "LFp9FtDf9g", "reviewer": "Reviewer_7xD9", "summary": "Authors analize the expressive power of 3D equivariant GNNs and introduce a new expressive equivariant GNN architecture based on local node and edge wise frames.", "questions": "The local frame construction reminded me a bit of Frame Averaging by Puny et al. (ICLR 2022). It would be interesting to see how they stack up.", "limitations": "Authors have addressed the limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed way to achieve equivariance is well grounded and conceptually very interesting and novel.\nThe construction is also based on solid theoretical motivation and achieved good results in practical experiments.\nThe paper is well written", "weaknesses": "The theory about the local hierarchy of 3D Graph Isomorphism (section 3) relatively straightforwardly follows from existing works on equivariant GNN expressive power and subgraph GNN expressive power. That being said, the overall package is still very solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 7xD9\n\nThanks for your meticulous examination and insightfull feedback.\n\n**Response to the Weakness**:  \n\nWe appreciate your discerning observation and insightful feedback regarding the theoretical aspect of the local hierarchy of 3D Graph Isomorphism in Section 3. Indeed, the foundation of this theory draws upon existing works on 2D GNN expressive power and subgraph GNN expressive power.\nHowever, it's important to note that even within the context of established theories, the synthesis and customization of these ideas to our specific problem domain contribute to the overarching value of our work. \n\nWe would like to emphasize the coherent theoretical logic that emerges from the synergy between Section 3, Section 4, and the neural sheaf interpretation in appendix. To enhance the clarity of this logical progression, we will incorporate a roadmap at the conclusion of the introduction, highlighting the interconnectedness of these sections.\n\n**Q1: Frame Averaging by Puny et al. (ICLR 2022)**\n\nA1: Thanks. After carefully checking, we believe Puny et al. (ICLR 2022) [1] provided another way of building equivariant frames.  However, their specific frames come out of the PCA decomposition, which are essentially global. Then, the averaging is a weighted sum of these global frames (a finite approximation of integration on the Lie group).\nWe will add the discussion of this work into our related work section. \n\n[1] Frame Averaging for Invariant and Equivariant Network Design\n\n"}, {"review_id": "2PImYOSTK2", "reviewer": "Reviewer_gxFS", "summary": "This paper introduces local structure encoding and frame transition encoding for more expressive representation learning in 3D graph neural networks. The local structure encoding is inspired by observations in the proposed local hierarchy of 3D graph isomorphisms; specifically the observation that subgraph isomorphism has greater discriminative power than triangle or tree isomorphism. The frame transition encoding is inspired by the observation that not all types of invariant interactions between disjoint clusters can be expressed as functions of the union of those clusters. The authors show that the proposed 3D GNN exceeds the performance of or performs on par with a multitude of existing models.", "questions": "* SpookyNet is not cited or discussed although it shows very strong performance in the analysis. This method appears to have local and global features, it would be interesting to see where it lands in the proposed classification.", "limitations": "The authors discuss some limitations of the method", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Originality: The proposed work appears original. The authors introduce a classification of local and global structure, and use it to classify existing 3D GNNs. They also show how to incorporate the most expressive types of local and global structural elements into a new 3D GNN and show that it outperforms/performs on par with existing methods.\n* Quality/Clarity: The proposed work is of good quality. The paper is well written, preliminary ideas are presented in an accessible way; the local hierarchy of 3D graph isomorphisms is paired with a very nice figure; the work is well motivated; the proposed model outperforms/performs on par with existing methods.\n* Significance: The problem of building expressive representations in GNNs is challenging and of significance to the community.\n", "weaknesses": "* Quality: see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer gxFS\n\nThanks for your meticulous examination and insightfull feedback.\n\n**Q1**: SpookyNet is not cited or discussed although it shows very strong performance in the analysis. This method appears to have local and global features, it would be interesting to see where it lands in the proposed classification.\n\n**Response**: \nThank you for bringing this to our attention. We apologize for the oversight. In Table 2, we have compared our method with SpookyNet, but regrettably, we neglected to cite it in the main text. We will correct it in the revised version.\n\nSpookyNet is a strong baseline with both local and nonlocal interaction blocks. But **the meaning of “local” and “nonlocal” in SpookyNet are different from the local-to-global analysis in our paper.** \n\n  - Specifically, we focus on the expressive power of 3D GNNs, in other words, the ability to distinguish different 3D structures. Here local means local 3D structures, and global means the whole input 3D structure, as used in [1].\n\n  - While in SpookyNet, **the local interaction aims to incorporate neighboring information into central nodes**, which is essentially what an MPNN layer can do. The nonlocal interaction aims to capture long-range interactions between nodes which can not be captured by an L-layer GNN, therefore, they **use attention to consider interactions between all pair of nodes**. This local and nonlocal idea is similar to the method in this paper [2].\n\nIn short, we focus on the ability to distinguish different local and global structures, while SpookyNet focuses on incorporating local and global information. These two are different. \nFor example, a SchNet-like model which considers edge distances during message passing indeed can capture local interactions, but it cannot distinguish different local structures.\n\nIt is worth mentioning that instead of simply using a SchNet-like model to capture local interactions, SpookyNet uses a more powerful model, as shown in Figure 3 of their paper. Specifically, it first constructs some basis functions based on Bernstein polynomials and spherical harmonics. These basis functions are then used to update node features in the local interaction block. Putting SpookyNet in Table 4, we can say that it satisfies LSE, but only partially satisfies FTE by using the nonlocal interaction. Note that it doesn't update equivariant features. The final outputs of the local interaction block are only invariant features.\n\n\n\n[1] ComENet: Towards complete and efficient message passing for 3D molecular graphs\n\n[2]. Recipe for a general, powerful, scalable graph transformer"}, {"review_id": "8GNio5iTPK", "reviewer": "Reviewer_b3yr", "summary": "This paper presents the expressive power of equivariant graph neural networks in the context of 3D local isomorphism from a 2D local isomorphism perspective. Similar to the subgraph isomorphism of GNNs, the paper proposes three types of isomorphism in 3D space: Tree Isometric, Triangular Isometric, Subgraph Isometric. Based on the above definitions, the paper defines geometric Weisfeiler-Lehman (WL) tests. Based on the conclusions in paper [1], this paper enhances the expressive power of equivariant GNN by incorporating mutual 3D substructures. On the other hand, to further enhance the expressive power of the equivariant graph neural network, this paper introduces an SE(3)-invariant encoder to exploit mutual 3D structures. Through a theoretical analysis (theoreom 4.1), the paper demonstrates that simple aggregation of local messages cannot approximate global interactions, thus the authors further enhance the model's expressive power through the frame transition matrix. The effectiveness of the proposed model is demonstrated by experimental results on scalar and vector properties of molecules on QM9.\n\n[1] A New Perspective on \"How Graph Neural Networks Go Beyond Weisfeiler-Lehman?\" ", "questions": "See my comments on weakness.", "limitations": "In my personal opinion, one of the limitations of this paper is that the notation is a bit confusing, and the paper's layout is somewhat disorganized, making it hard to follow. I would suggest that the authors provide a guideline at the end of the introduction to help readers understand the role of each section.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. To the best of my knowledge, this is the first paper to analyze the expressive power of equivariant neural networks from the 3D isomorphism perspective. This could inspire future work to analyze the expressive power of current equivariant neural networks from this perspective.\n\n2. Based on the analysis proposed in the paper on the expressivity of equivariant neural networks, the paper incorporates mutual 3D substructures into 3D GNNs (LSE in this paper) to further enhance their expressive power. The effectiveness of the proposed analysis and methods has been demonstrated experimentally. \n\n3. To reduce the computational cost of neural sheaf diffusion, the paper proposes a novel frame transition matrix to express global geometric information. The computational complexity analysis in line 304 validates the effectiveness of the proposed method.", "weaknesses": "1. I have to say that the notation in this paper is confusing, which imposes a burden on me. For instance, h is defined as a node feature on line 81, and f is a tensor-valued function on line 70. However, f is then used as a bijection on line 112. Does this mean the definition of h on line 112 is different from that on line 81? \n\n2. Although the analysis of the expressivity of 3D GNNs is the first from the 3D isomorphism perspective, this paper seems to be based on the analysis in [1]. Moreover, the method proposed by the analysis in [1] can incorporate structural properties into many GNNs, such as GCN, GIN, and GAT. Can this paper similarly incorporate mutual 3D substructures into other 3D GNNs?\n\n3. Recently, some work [2] has analyzed the expressive power of GNNs from the perspective of the attention mechanism, so how can we analyze the expressive power of 3D GNNs from the attention mechanism? \n\n\n\n[1] A New Perspective on \"How Graph Neural Networks Go Beyond Weisfeiler-Lehman?\"  \n\n[2] Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer b3yr\n\n- **W1**: I have to say that the notation in this paper is confusing, which imposes a burden on me. For instance, h is defined as a node feature on line 81, and f is a tensor-valued function on line 70. However, f is then used as a bijection on line 112. Does this mean the definition of h on line 112 is different from that on line 81?\n\nSorry for the heavy notation. The $h$ on line 112 and line 81 denotes the node features. However, the repeated use of $f$ in line 70 and 112 may cause a burden for readers. Do you think it’s better if we use $\\varphi$ to denote the bijection map rather than $f$?\n\n- **W2**: Although the analysis of the expressivity of 3D GNNs is a first, this paper seems to be based on the analysis in [1]. Moreover, the method proposed by the analysis in [1] can incorporate structural properties into many GNNs, such as GCN, GIN, and GAT. Can this paper similarly incorporate mutual 3D substructures into other 3D GNNs?\n\nThank you for your insightful question. Indeed, [1] has laid the groundwork with a novel perspective for crafting potent GNNs on 2D graphs. Building upon this inspiration, we extend our analysis to the realm of 3D graphs. The primary distinction between 2D and 3D graphs lies in the inclusion of not only nodes (with features) and edges (with or without features), but also 3D coordinates for each node. **It's exactly the 3D symmetry that makes our extention nontrivial.**\n\nRegarding various 3D GNNs: Yes, our proposed method can seamlessly integrate 3D geometric information into diverse 3D GNNs, provided that geometric features such as distances, angles, or equivariant steerable features are integrated. Specifically, the message passing block illustrated in Figure 3 (between LSE and FTE) can be applied to any equivariant 3D GNN (e.g., Schnet, GVP-GNN, EGNN, or invariant graph attention messgae passing).  The equivariance of our method is guranteed by theorem 5.1. Furthermore, \nIn our experimental evaluation, we employ a previous sota method (PaiNN) as the backbone message passing scheme and enhance its capabilities by incorporating our LSE and FTE blocks.\n\nThe adaptability of our approach across different 3D GNN architectures underscores its versatility and potential impact on advancing the field.\n\n- **W3**: Recently, some work [2] has analyzed the expressive power of GNNs from the perspective of the attention mechanism, so how can we analyze the expressive power of 3D GNNs from the attention mechanism?\n\nThanks for bringing up this nice work.  In fact, LEFTNet also implement the structural coefficients in theorem 3.1 as a set of weights that are multiplied to the node features. Note that attention mechanism can be also seen as defining a set of weight, however the difference is that the attention coefficients go through the softmax normalization. Even though we do not see an immediate connection to transfer in [2] to 3D Equivariant GNNs, we do see it is a promising direction to enhance our understanding of 3D Equivariant GNNs and improve interpretability. We will leave this as future work.\n\n- **W4**: In my personal opinion, one of the limitations of this paper is that the notation is a bit confusing, and the paper's layout is somewhat disorganized, making it hard to follow. I would suggest that the authors provide a guideline at the end of the introduction to help readers understand the role of each section.\n\nThanks for the great suggestion! We will reorganize the structure with a roadmap at the end of the introduction.\n\n[1] A new perspective on\" how graph neural networks go beyond weisfeiler-lehman?\"\n\n[2] Interpretable and generalizable graph learning via stochastic attention mechanism."}]

Author rebuttal:
## General Response

We thank all the reviewers for their valuable comments and appreciate that all the reviewers find our study novel and interesting for 3D Equivariant GNNs. As mentioned by the reviewers, 
- the problem we are focusing on is challenging and of significance to the community (Reviewer Wt9q and gxFS),
- our work is novel and original (all reviewers),
- we have solid theoretical analysis (Reviewer 7xD9),
- our paper is well-written with nice figures (Reviewer Wt9q, 7xD9, and gxFS),
- our method achieves good results, and the ablation study can support the main arguments (all reviewers). 

We also would like to thank Reviewer Wt9q and Reviewer b3yr for pointing out some typos and repeated notation that may cause misunderstandings for general ML audiences. We will fix them in the revised version.

In addition, we will add a roadmap at the end of the introduction, and rearrange the order of the appendix to avoid some cross-references, following reviewer Wt9q and b3yr’s suggestion.

Here we add additional explanation about frame transition (FT) (**suggested by Reviewer Wt9q**),  since FT is also one of our main contributions.

W8: about frame transition: why do we need FT?

Our introduction of frame transition (FT) addresses a universal local-to-global phenomenon intrinsic to equivariant neural networks. In this sense, FT is a concept  designed for almost **all equivariant graph neural networks that implement local message passing as backbone models**. On the other hand, this concept assumes significance in the context of symmetry-aware processing, and there won’t be any frame (or coordinate) ‘transitions’  in non-equivariant neural networks used for 3D point clouds, which operate without considering symmetry.

We acknowledge your observation regarding the prevalent use of equivariant architectures that implicitly incorporate aspects of frame transition. In fact, Table 4 of our work elucidates how these architectures partly encode frame transition information, thus **offering a principled basis for their empirical success**. Two of the relevant equivariant models you mentioned have inadvertently been omitted. We will promptly rectify this oversight by including them in Table 4.

To complete our logical framework, we emphasize that invariant architectures can fully capture frame transition information as well. You will find a computationally efficient realization of this principle in Section H of the supplementary materials, where we explore the incorporation of neural sheaf and combined node-edge equivariant frames for a pure invariant encoding of FT.

Regarding the **preference for our proposed frame-based propagation over existing equivariant message passing algorithms**, we underscore three key practical advantages: **1**. Our approach explicitly encodes Frame Transition (FT), offering **both equivariant and invariant methods** for FT encoding, as mentioned earlier. Furthermore, the expressiveness of our FT-encoding is guaranteed by theorem 5.1;
**2**. Optimization Benefits: Our empirical findings demonstrate enhanced stability and faster training for our LSE + FTE modules. While expressiveness gauges approximation errors, optimization challenges persist in neural networks [2]. Hence, we advocate exploring diverse FT realizations to navigate optimization complexities;
**3**. Multi-Modality Pretraining Ease: We observe that incorporating invariant architectures with other neural networks is seamless due to invariant scalar hidden layers. These layers can seamlessly integrate with invariant representations from other modalities [1]. Consequently, the invariant encoding of FT could confer advantages in such multi-modal pretraining scenarios.

[1] Learn to Combine Modalities in Multimodal Deep Learning

[2] DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators

W9: about frame transition: definition of frames

We will move the definition of equivariant frames to the main text. In line 214, the orthonormal frames and especially the orthonormal property are defined in terms of the ordinary Euclidean metric. Therefore, no Riemannian metric is needed. We only mention Riemannian geometry and the frame bundle in the limitations and future work section, as our local frame has a root in the Riemannian context as a basis of the tangent space attached to spatial points.

W10: paper organization

Apologies for the recurrent cross-references throughout the paper. We have taken steps to enhance this aspect. Given that our paper encompasses both theoretical groundwork and theory-inspired algorithm design, and the theory itself encompasses two intricately linked components—1. Local Analysis and 2. Local-to-Global Intermediate Analysis—cross-references become unavoidable and imperative to ensure the rigor of the article.

To mitigate potential confusion, our approach involves furnishing intuitive explanations for each mentioned concept while reserving detailed arguments for the appendix (please consult the supplementary materials' version of the appendix, which is further refined). As part of our improvement strategy, we intend to: 1. Enhance the paper's organization by providing a more structured "roadmap" at the conclusion of the introduction section.
2. Relocate the **initial portion of Appendix F (as featured in the supplementary materials) to Line 556** to eliminate the need for a cross-reference.

Meta-review: This paper suggests a novel approach for building 3D equivariant graph neural networks. First, it develops hierarchy of local 1-hop subgraph isomorphisms to test local expressiveness; and second, it discusses how to aggregate local information globally with minimal loss of information leading to the incorporation of frame transition matrices. On the downside, the main theoretical contributions of this paper closely follow previous works, and the benefit over alternative 3D equivariant architectures does not seem very large.
Decision: Accept (poster)