Hybrid Directional Graph Neural Network for Molecules
Authors: Junyi An
Chao Qu
Zhipeng Zhou
Fenglei Cao
Xu Yinghui
Yuan Qi
Furao Shen
Venue: ICLR 2024 spotlight (2024)
Keywords: Graph Neural Networks; Equivariance; Molecular model
Primary area: learning on graphs and other geometries & topologies

Abstract: Equivariant message passing neural networks have emerged as the prevailing approach for predicting chemical properties of molecules due to their ability to leverage translation and rotation symmetries, resulting in a strong inductive bias. However, the equivariant operations in each layer can impose excessive constraints on the function form and network flexibility. To address these challenges, we introduce a novel network called the Hybrid Directional Graph Neural Network (HDGNN), which effectively combines strictly equivariant operations with learnable modules. We evaluate the performance of HDGNN on the QM9 dataset and the IS2RE dataset of OC20, demonstrating its state-of-the-art performance on several tasks and competitive performance on others. Our code is anonymously released on https://github.com/ajy112/HDGNN.

Published as a conference paper at ICLR 2024
HYBRID DIRECTIONAL GRAPH NEURAL NETWORK FOR MOLECULES
Junyi An': Chao Qu²i, Zhipeng Zhou', Fenglei Cao², Yinghui Xu", Yuan Qi', Furao Shen' State Key Laboratory for Novel Software Technology, Nanjing University
PINFYY TECH (Shanghai) Co., Ltd.
"Shanghai Academy of Artificial Intelligence for Science
Artificial Intelligence Innovation and Incubation (AI²) Institute, Fudan University
ABSTRACT
Equivariant message passing neural networks have emerged as the prevailing approach for predicting chemical properties of molecules due to their ability to leverage translation and rotation symmetries, resulting in a strong inductive bias. However, the equívariant operations in each layer can impose excessive constraints on the function form and network flexibility. To address these challenges, we introduce a novel network called the Hybrid Directional Graph Neural Network (HDGNN), which effectively combines strictly equivariant operations with learn- able modules. We evaluate the performance of HDGNN the QM9 dataset and the IS2RE dataset of OC20, demonstrating its state-of-the-art performance on several tasks and competitive performance others. Our code is anonymously released on https///gtthb..co//ajy 12/HDGNN.
INTRODUCTION
In recent years. Graph Neural Networks (GNNs) have experienced remarkable success across various domains, including social networks (Kipl' & Welling 2017), physical systems (Battaglia et al. 2018). computational biology (Townshend et al. 2019 Chami et al., 2019), and many others. Among these domains, the application of GNNs in computational chemistry has garnered significant interest due to its promising ability to predict energy and other chemical properties with comparable accuracy to quantum mechanical simulation methods (such DFT), while achieving speedups of 4-5 orders of magnitude (Behler & Parrinello 2007: Gilmer et al. 2017).
Nonetheless, regular GNNs alone are insufficient for accurate modeling of molecules, as they overlook crucial chemical constraints such as invariance and equivariance (Fuchs et al. (2000). For instance, both molecular and atomic energies remain invariant under rotation transformations, whereas forces exhibit equivariance. To guarantee the rotational invariance of predicted property, Schnet (Schütt) et al. 2018) and HIP-NN (Lubbers et al. 2018) restrict the network's 's inputs to only depending on interatomic distance, while this leads to a loss of directional and equivariant information (Miller cet al. 2020). Directional GNNs go a step further by explicitly incorporating bond angles and dihedral angles information in GNNs (Gasteiger et al.) 2020 2021).
To capture deep directional features for both invariant and equivariant interactions, equivariant neural networks have been proposed (Brandstetter et al., [22T]] [iao Smidi 2023). These networks utilize
group representations to construct steerable group convolutions message passing blocks, ensuring equivariance throughout the entire network. While equivariant neural networks exhibit appropriate inductive biases for molecular systems, the operations under strictly equivariant constraints will limit the expressiveness of the network. This loss of expressiveness impedes the model's ability to effectively learn the intricate interactions between atoms. An analogous situation can be observed in convolutional neural networks, where imposing cquivariance results in overly constrained net- works and suboptimal performance (Romero Lohit 2022). In contrast, relaxing equivariance approaches (Romero & Lohit 2022, Wang et al. allow for learning approximate equivariance while capturing the nuances of reality, thereby enabling strong generalization. Recently, Zitnick et al.)
"c-mail: junyian @ smail.nju.cde nju.ceu..n
'Corresponding authors: Chao Qu is the corresponding author in INFLY TECH, and Furao Shen is the corresponding author in Nanjing University.

---

Published as a conference paper at ICLR 2024
(2022) relaxes the equivariance constraint in GNNs and accomplishes better performance by using additional coefficients. This work has demonstrated leading performance on challenging tasks such effective molecular model, we investigate the impact of relaxing equivariance based on theories of expressive power (Dym & Maron 2021 Joshi al. (023). It is worth noting that while certain equivariant GNNs with infinite degree of group representation I have been proven to possess universal approximation capabilities for any equivariant function and the expressive power of equivariant GNNs can be enhanced by increasing degree number during conducting (jebsch-Gordan (CG) tensor product (Dym & Maron 202I). it is impractical to employ the high-degree CG product due to the substantial computational costs. Given the aforementioned dilemma, we observe that certain specialized operations can achieve the expressive power of high-degree CG product with significantly lower computational complexity, albeit introducing non-equivariance. Detailed introductions can be found in Section 2.4 Building upon our observation, we aim to learn appropriate relaxed equivariant features, which can achieve expressive power of high-degree representation and ensure approximate equivariance.
as OC20 (Chanussot et al., (02II. To further explore
In this paper, we propose the Hybrid Directional Graph Neural Network (HDGNN), a message- passing GNN designed to enhance expressiveness by relaxing equivariance. To accomplish this, we replace strictly equivariant building blocks, such as the CG tensor product, commonly used in equivariant GNNs (Griffiths & Schroeter 2018 Kondor et al. 2018), with flexible neural networks. However, naively applying neural networks to construct these modules undermines equivariance and degrades the overall network performance, as evidenced by our ablation study in Section [5.2 To this end, we propose a neural structure that incorporates subtle designs for learning equivariant properties. Furthermore, this structure is combined with strictly equivariant operations through a dynamic module. With this design, HDGNN can achieve an automatic balance between equivariance and expressiveness. If we discard certain operations in our building blocks or replace the learned messages with fixed spherical harmonic representations, our networks may degenerate to the vanilla equivariant GNNs which may subsume the existing works (Kondor et al. 2018)] Thomas et al. 2018). We evaluate our HDGNN on the QM9 benchmark (Ramakrishnan et al. 2014) and the IS2RE dataset of OC20 [Chanussot et al. 202T). On the OC20 dataset, HDGNNN outperforms state-of-ther methods and leads a significant improvement on unseen samples. For the QM9 dataset, HDGNN achieves the best results on several tasks and comparable results on the remaining ones. We provide extensive ablation studies on non-equivariant modules and the network structure.
(SOTA)
We summarize our main contributions as follows: (i) A neural architecture for learning relaxed equivariant representations, which can enhance the expressive power suffering from the limitations of finite low-degree group representation and ensure approximate equivariance; (ii) The ablation study sheds light on our building blocks and shows their necessities. It may pave the way for the future study on the approximately equivariant neural network.
2 PRELIMINARY
In this section, we make a brief review of the necessary mathematical background to depict our model, which includes equivariance, spherical harmonics, the Clebsch-Gordee Gordan (CG) tensor product, and so on. Additionally, we briefly describe the universal approximation of equivariant function. More detailed introductions are deferred to Appendix A. We list the notations frequently used in the following. We denote unit sphere 52, where spherical coordinates (0, sp) polar angle and azimuth angle, respectively. The symbol TR stands for the set of real numbers while R represents the rotation matrix for 3D vectors. We use G denote the group and SO(3) to denote the special orthogonal group, i.e., the 3D rotation group. We use o and & to represent the element-wise product and the CG tensor product, respectively.
2.1 EQUIVARIANCE AND INVARIANCE
Given group G and transformation parameter E G. function Φ X' - y is called equivariant to g if it satisfies:
(1) where (g) y + and T(g): X + X' denote the corresponding transformations over y and X. respectively. Invariance is a special case of equivariance where ("(g) is an identity transformation. says that the output of Φ is unaffected by the transformation applied to the input. In this paper, mainly focus on the S0(3) equivariance and invariance, since it is closely related to the interactions between atoms in molecule!
T'()] +(T())]=)
Invariance of translation is trivially satisfied by taking the relative positions as inputs.
2

---

Published as a conference paper at ICLR 2024
2.2 SPHERICAL HARMONICS AND STEERABLE VECTOR
Spherical harmonics, a class of functions defined over the sphere S form an orthonormal basis and have some special algebraic properties widely used in equivariant models (Kondor et al. 2018 Cohen et al. 2018). 8). In this paper, we use the real-valued spherical harmonics denoted as ( : 5 - R where and denote degree and order, respectively. It is known that any square-integrable function defined over 5 can be expressed in a spherical harmonic basis via
f(0.4)= - - Σ
f Y (0. (0,s))
(2)
1=0 m=-!
where f is the Fourier coefficient. For any vector FF with orientation (0, 4),, we define (¹, v/)
= (0,, 3): Y 1=141 v)); ...; Y/ vb)]" a vector with 21 + elements. See details of spherical harmonics in Appendix A.1. Spherical harmonics can be used to encode orientation (Gasteiger et al. 2021 2022 and map the representation the frequency domain to the signal over the spatial domain
(Cohen et al.
2018 Zitnick et al. 2022)..
commonly used property of the spherical harmonics is that for any R E SO((),, we have
Y ( = D' (
(3)
where D' (21+1) x (2(+1) matrix known as a Wigner-D matrix with the degree 1. Therefore, R. and D R correspond to T(2) and (q) in equation | respectively. Following the convention in CChami et al. 2019 Brandstetter et al. 2021). b. we say that (F) is steerable by the Wigner-D matrix of the same degree (. In addition, the (2l + --iimensional vector space which Wigner-D matrix of degree act is termed a type- steerable vector space. In fact, the Wigner-D matrix is an irreducible representation of the group 5O(3) and we refer interested readers to (Kondor et al. 2018).
2.3 EQUIVARIANT OPERATIONS
The key of the equivariant model is to design o(-) in equation I which preserves the equivariance and at the same time enriches the abstract directional information. simple approach is to encode the directional information by ('(.) over each node or edge, and then do certain operations preserving equivariance, such as aggregation or scaling with an invariant variable. However, if we depend solely on such operations, it would limit the learning for deep directional features.
The CG tensor product (Griffiths & Schroeter 2018), originally describing the angular momentum coupling in quantum mechanics, provides another option and becomes the workhorse to design the equivariant networks (Thomas et al.) 20188: Kondor et al. 2018 Brandstetter et al. 20211) The CG tensor product @ is defined follows,
12
(u X v)
Σ mi
(l.m) (1imm)((1a mg) u vha
(4)
1 mg 12
where u and v denote two steerable representations with degree I and l2. On the right-hand side of equation C denotes the CG coefficients, sparse tensor, which produces non-zero terms when
1/1 l2 < (4 12).
(5)
Based on the CG tensor product, the output is type-l vector when and are type-l1 and type-l2 vectors. Besides, there are several equivariant operations in our method, including gate (Weiler et al..
2018 Fuchs et al. 2020) and normalization (Geiger al. 2022). Their details can be found in Appendix A.2.
2.4 ANALYSIS OF EXPRESSIVE POWER
Several studies have indicated that applying the CG tensor product to high-degree representations can push the upper boundary of GNN expressive power. For instance, Theorem 2 in (Dym Maron 2021) has demonstrated that any continuous G-equivariant function can be effectively approximated using TFN structures (Thomas et al. 2018) with infinite degrees. Additionally, Joshi et al. (2023) has revealed that equivariant models with maximum degree L struggle to distinguish n-fold symmetric structures when T, is less than n. In this case, more flexible structures like Multi-Layer Perceptron (MIP) may enhance expressive power, since the functions that MI. approximates encompass universal equivariant functions. We substantiated this perspective through GWL graph isomorphism test (Joshi et al.) 2002), detailed Appendix A.3. Note that these flexible structures introduce non- cquivariance. The expected generalization intricately tied to both equivariance and expressiveness. Consequently, our work also focuses on mitigating the loss of equivariance.
3

---

Published as a conference paper at ICLR 2024
Model:
Message Block
Update Block:
X1
Fy
fg
cU
Concat
Muti-T TP
mij
Embedding
MLP
ft
Norm
MLP
IFFT F
IFFT (-F
M P
Concat
Vij
m
M essage Block
Muti-TP:
X
X
x;
MLP
Update Block
SLU
FFT (
R11
K
ci
МETP
Output Block
Gate
0-1 D-1
Figure 1: Left: The overall framework our model; Middle: message block and structure of Muti-TP; Right: update block. Dotted lines with represent the original value of Cij.
OUR METHOD
The architecture of HDGNN is illustrated Figure | We first introduce the framework of HDGNN. Then show how design each ingredient of the neural network.
3.1 MESSAGE PASSING FOR MODELING ATOMIC MOLECULAR SYSTEM
Message passing GNNs is particularly effective tool to encode the atomic property and model the interactions between the atoms. In our paper, we use the F.j to represent the vector from node to node j. The local neighborhood. (i) defined by a cutoff radius (i) = ( |[ S reat) or the 2D molecular graphs induced by SMILES
HDGNN built upon the message passing propagation mechanism. To enable steerable node embeddings, we define them combination of type-0, type-1, .... type-L vectors, resulting in size of (L+ + 1)a. Here, L represents the maximum degree and can be considered a hyper-parameter which usually set to 14. 6). Similarly, we define spherical harmonic representation of any 3D (F)), which also has a size (L + 1)2. Furthermore, each node embedding is expanded to C' channels. In the first round, the invariant atomic feature 2; is passed through an embedding layer to obtain a representation for the i-th node. As this representation invariant, assign it the type-O vector, while the other type-l vectors
vector F as (F) = (F); Y (F):
set to The overall node embedding is denoted as x!. Then x² goes through norm layer, message block and update block incorporate positional information and neighborhood information, which yields new embedding x Additionally, we construct an invariant branch, with its node embedding denoted as f, This branch assists in the calculation of messages and the final predictions. The invariant branch. depicted the left side of the model in Figure [ is lightweight structure that ensures invariance of embedding f. For further implementation details, please refer to Appendix B.
By repeating this procedure K times, we obtain the final embeddings x and P.K. These embeddings are then processed through specific layers to generate the final results, such as energy, force, or other properties, depending the specific task at hand. In the following section, we detail each ingredient.
3.2 MESSAGE BLOOCK
In this section, we will elaborate on the design the message passing block. Our objective is to compensate for the loss of expressive power resulting from the limitations of finite low-degree terms. To accomplish this, we introduce the message block, depicted Figure which incorporates both equivariant operations and non-equivariant terms. Formally, the message calculation is defined in
4

---

Published as a conference paper at ICLR 2024
To provide insights into the structure of this block, we first present the construction of strictly equivariant counterpart of equation In this counterpart, we define the message mij as
equation II
follows:
(6) where @ denotes the CG tensor product. In equation 6 we do the product between the node embedding (or messages) and fixed spherical harmonic representations of the edge direction F,j, which is the conventional wisdom to implement the CG tensor product in equivariant networks such as (Thomas)
m (x, + x,) S (F,z);
al. 2018, Batzner et al., 2022).
We aim to incorporate the higher-degree nonlinearity beyond the bilinear operation of the CG tensor product. To that end, develop learnable counterpart of equation 6 to reach this goal. We termed it multiple tensor product (Muti- (P).i Each pathway of Muti- TP represents an analogue of CG tensor product, which shown in Muti-TP module of Figure ! In the following, we use D(R.y) D (R.g) + D (R.)) ........ D matrices from degree orientation [0, 0, I]. redNote that there are multiple rotations Raj that can rotate Fi to [0, 0, 1J. We
.... + D L (R.y) to represent block diagonal matrix with Wigner- L. R, denotes the 3D rotation matrix that transforms F.j to a fixed
randomly select one of them and apply it to all message blocks.
To begin with, we introduce the -xtan-xxtension Tensor Product (METP) as one of the pathways in Multi-TP. The METP is designed to capture equivariant interactions in low-degree representations. For representations in low degrees (1 S. L'), where L' is a hyper-parameter typically set to range of [1, 4]. The METP is defined follows:
4
l2
(u METT Vine = we
Σ
Σ
cillm))
vla mn mg?
(12.ma)(
172 10
m22--2
where denotes the learnable weights. ū denotes the mean of the whole channels. It is worth noting that the fully-connected CG tensor product (Fully-TP) (Geiger et al., 2022) is effective but computationally expensive. In contrast, complexity while preserving the property that each output channel depends on all input channels. Furthermore, we apply a gate activation after the METP operation. As previously mentioned, the use of equivariant operations with finite low-degree representations will limit expressiveness. To address this issue, we incorporate flexible neural structure mitigate the loss of expressive power. This structure serves as an additional pathway within Multi-TP and is designed to capture approximately equivariant interactions across both low-degree and high-degree representations < L). The structure is illustrated in the Muti-TP module of Figure I To understand the principle of this structure, we
METP considerably reduces the computational
expand equation 6 to x; X (TT) Xj S
(T:y) and each term can be transformed to
x, 2 S (T D (R.;) S(C)),
(8)
where x-- = D(R,, )x, Č= 10.0. 0. 1]. Hence, the tensor product is simplified to a sparse matrix multiplication, as showcased in (Passaro & Zitmick| (023). This approach reduces complexity and offers a feasible way for neural networks to learn the equivariance of CG tensor product. Kofinas et al. (202T) and et al. (2022) have shown that rotation local coordinate frames can facilitate the learning of filter in GNNs. To learn the equivariance in equation 8, an intuitive approach is to replace the term (x/ @ S(C)) with MLP, denoted N (x()) However, equation 8 introduces randomness in rotation Rej. and the general form of MLP is non-equivariant, rendering it unable to compensate for the introduced randomness (see details in Appendix B.2). Therefore, replacing equation & with an MLP will break equivariance. In order to mitigate the loss of equivariance, we propose an alternative approach, which first transform the equation 8 to
X & sh (T.)) = D (R,))xx @ S(C) + x, & S(C)),
to where x (D(R.;) I)x,, and I denotes an identity matrix. Then, we use MLP N approximate the equivariant function (x @ S(C,))). Thus, equation is approximated by
2 (F,)) ~ D D- (R:,) + (x:)).
(10)
The weights of (x"" and N (xx)) are shared. If the neural network N-) can learn the complete equivariance by training, the right side of equation 10 equivariant. We adopt this structure for two objectives: (a) The neural network learns the transforroooonnaonon pattern during training to acquire learned equivariance, We introduce an additional pattern in equation 9 to enhance the quality
5

---

Published as a conference paper at ICLR 2024
of learned equivariance. (b) By utilizing x; and x as independent inputs for MLP, we anticipate that MLP will effectively extract directional information embedded in the Wigner-D matrix.
Based on structures in equation 7 and equation I0] we incorporate an attention module to regulate the contribution of different terms. Combining all pieces together, we have
myy = abj, @ D (R.)) + (x;)) + 2,2 @ D (NN())))) + (x)))x)))
+ +aja O D (N + xx)) + (xx
+ +ag.4 (x, + +x)) SMETP S (T.)),
(1))
where all the attention coefficients aij produced by applying MLP invariant branch. The addition in equation equation| I1 corresponds to an degree-wise addition. In equation [1 the first three terms, designed to enhance expressive power, are initially non-equivariant. However, through effective training, they tend to achieve approximate equivariance. The last term is inherently equivariant, ensuring that global equivariance is not excessively compromised during training. In our implementation,
utilize 2-layer MLP with the SiLU activation function. Additionally, we incorporate two strategies into the MLP structure. First, we introduce built-in molecular properties by multiplying invariant features with the output of the first MLP layer. These features, represented as Cij in Figure [ include atomic type embeddings (2; and 2j) and edge attributes obtained from a distance block using a Gaussian basis. For more detailed discussions on the implementation, please refer to Appendix B.2.
We conduct an ablation study on the structures of non-equivariant terms in equation I1 in which we compare two extreme cases with our design. In the first case, we replace the MLP with the (equivariant) linear layer and therefore equation reduces to an equivariant term. In the second case, we naively create a high-degree message mij by NN((X; x,; ((F,)))) which violates the equivariance lot and hurts the performance.
3.3 UPDATE BLOCK
After aggregating the messages m, for each node to gain m,, our focus shifts to learning the interactions between the channels within each m,- Using CG tensor products between different channels is a naive approach to ensure equivariance. However, this approach is computationally expensive and falls short in effectively capturing interactions across multiple channels ( 2). Here, we apply a more efficient method inspired by the observation that certain paths of the CG tensor product resemble convolution or correlation operations in the frequency domain, with the degree I serving as an analog to the frequency w. To formalize this, let m; and my represent any two channels within m. The correlation * are represented by (m) m2)² = = mn m where (dl, dl - 1) corresponds to no-zero paths 14 |2| = I in equation 55 Similarly, convolution corresponds to paths 1, + 122 1. Inspired by the convolution theorem of the Fourier Transform (FT), we can approximate the tensor product in the frequency domain by performing point-wise operations in the time domain. To this end, we apply the spherical harmonics expansion, which is analogous to the inverse fast Fourier Transform (IFFT), each channel of m
(F) Σ m Y (T).
(12)
1=0
Next, we concatenate all the elements in the points F and -r and obtain P(F) [pi (f")...,, (F). Pi .p((i)).....(r) Here, we extend the point-wise operation to two points by incorporating both convolution and correlation. Then we apply a MLP (shown in Appendix B.3) with an output size of C' to P(F) and gain PF(F). Finally, we transform the results back to the frequency domain:
m cm =ʃ
(13)
where Y denotes the complex conjugation. In our implementation, we use the Fast Fourier Transform (FFT) on 52 to represent the integral. Additionally, we incorporate a shortcut where we transform the input of the message block, x*_ to the time domain and concatenate the results with (T)) before passing them through the MLP. In this case. the output of equation 13 is the new embedding *+1 Our approach shares similarities with (Cohen al. 2018 Zitnick et al. 2022). where complex operations are transformed to other domains based on the convolution theorem. They apply single transformation, whereas we combine both correlation and convolution. Our approach

---

Published as a conference paper at ICLR 2024
enhances the paths of the CG tensor product embedded in time domain signals, thereby facilitating effective learning of interactions between channels. Our ablation experiments provide support for the efficacy of our approach. These experiments and theoretical details can be found Appendix B.3.
3.4 OUTPUT BLOCK
The final message passing block generates per-atom features x To predict chemical properties, we adopt the approach outlined in (Zitnick et al. 2022). Optionally, we include the output of the invariant branch, f* in the prediction process, For some invariant properties, we add X NN to the prediction produced by x" Here, À is fixed weight.
RELATED WORKS
We focus on equivariant models closely related to ours here. We provide a detailed discussion of other molecular models and works on relaxing equivariance in Appendix C.
Recent research showed that equivariant message passing neural network have achieved remarkable
results in predicting invariant or equivariant molecular properties (Thomas et al.
2018
Fuchs et al.
2020; Batzner et al.) 2022; Brandstetter et al. 202T). TFN (Thomas et al.
2018) and PequlP
(11 Y is learnable radial function. They define the messages mij = F (E.y) & x where xf
[Batzner et al. 2022) define the convolution filters Fl (F,y) R
(FF where R!. (+)
Fia
type 1' steerable vector of node i. They provide a general framework to combine spherical harmonics basis and embedding. Utilizing a spherical harmonics basis enables the learning of intricate equivariant functions (Weiler et al., 2018 Dym & Maron, 2021l) However, the expressiveness of these methods is constrained by band limited degree (Cesa et al. 202I).
(Fuchs et al. 2020) introduces an attention weight aij to enhance the expressiveness of the network.
Specifically, the message can be abstracted to m,j d:jy a.j' (F,) & x Yet, the attention weight a scalar or an invariant term independent of the CG tensor product. SEGNN (Brandstetter et al.. 2021) and Equiformer (Liao & Smidt 2023) extend the function form of abboe-mentionss works by incorporating the message passing neural networks and transformer- based networks, while its workhorse is still convolution filter with the CG tensor product, causing the similar limitation of TFN and transserrasformmrr.. In our work, we do not overconstrain the function form of the convolution filter. Notice that Wigner-D matrices in the message block contain the information of spherical harmonics. We hope that the network can learn interactions between x; and Y!! (Ft). which may replace the CG tensor product. SCN (Zitnick et al. 2022) employs non-equivariant operations in molecular models. Despite sharing the concept of relaxing equivariance, our structures that used to calculate messages are entirely different. Moreover, the equivariance of SCN heavily relies on data, which may be problematic small datasets. In contrast, our method addresses the loss of equivariance through strict equivariant operations.
(Fe5)
EXPERIMENTS
In this section, we conduct experiments to investigate the effectiveness of proposed method over Quantum Machines 9 (QM9) (Ramakrishnan et al., 2014) and IS2RE task in Open Catalyst 2020
(OC20) (Chauusoot al. (021)) benchmarks. In both experiments, we include Equiformer (Liao &
Smidi 2023). SEGNN (Brandstetter et al., )02T) and TFN (Thomas et al. 2018), strong baselines of equivariant neural network: Dimenet++ (Klicpera al. 2020). strong baselines of directional GNNs; Schnet (Schütt et al. 2018) and PaiNN (Schütt et al. (021), classical networks for modeling quantum interactions. the task of IS2RE, we include an additional baseline GemNet (Gasteiger et al.. 2021 1 2022). SphereNet (Liu et al. (2021) and SCN (Zitnick et al. (202)). In the task of QM9, LINet (Miller ct al. 2020), Cormorant (Anderson al.
2019
LieConv (Finzi et al. (020),, TorchMD-NET (Thökke 2022 are also compared with our work. We defer details of
& Fabritiis 2022) and EQGAT (Le et al.
configurations and pappee-ersmeterss of baselines to Appendix D.
5.1 RESULTS
The OC20 dataset contains over 130 million structures used to train models for predicting forces and energies during structure relaxations with a CC Attribution 4.0 License. The goal of a structure relaxation is to find local energy minimum. We report results of Initial Structure to Relaxed Energy task (IS2RE). The IS2RE can be solved by two approaches: 1. I.irretly predict relaxed energy from initial atomic structure; 2.u the relaxed structure computed by predicted forces predict energy. The second approach tends to be more accurate, while needs an efficient force model trained from other bigger dataset. We focus the first approach without pre-trained models that in (Brandstetter
7

---

Published as a conference paper at ICLR 2024
Table 1: Results on IS2RE OC20 Test for approaches that directly predict the relaxed energies.
Energy MAE (mcV)
L
EWT (%)
Model
ID
OOD Ads
OOD Cat
OOD Both
Average
ID
OOD
OOD
OOD Both
Median baseline
1750
1664
1750
0.71
0.72
0.89
0.74
CGCNN
615
916
622
851
751
3,40
3.10
2.00
SchNet
639
734
666
704
685
2.96
2.33
2.94
2.21
PaiNN
575
783
604
743
676
3.46
1.97
3.46
2.28
TEN (SElin)
584
766
636
700
672
4.32
2,51
4,55
2,66
Ge
702
634
4.59
2.09
4.47
DimeNet++
562
661
4.25
4.10
2.41
GemNet-OC
560
711
576
671
630
4.15
2.29
3.85
2.28
SphereNet
563
703
571
638
619
4.47
2.29
4.09
2.41
SEGNN
533
692
537
679
610
5.37
2,46
4.91
2.63
Equiformer
504
5.14
SCN
516
64.3
530
604
573
4.92
2.71
4.42
2.76
HDGNN
510
618
523
550
548
5.08
2.79
4.58
2.82
et al. 2021). The test set is splited into four subsets, including sampling from the same distribution
training (ID), unseen adsorbates (OOD Ads), unseen element compositions for catalysts (OOD
Cat), and unseen adsorbates and catalysts (OOD Both). Their sizes are similar. As shown in Table [
HDGNN outperforms all previous upproaches in terms of the average energy MAE. Furthermore,
notable observation is that HDGNN obviously outperforms other methods on unseen distribution, as
evidenced by the significantly lower average MAE on OOD tasks (HDGNN: 564 vs. SCN/Equiformer. 592/613). We hypothesize that the enhanced performance of HDGNN on unseen distributions may stem from learned features that closely align with actual physical properties. Additionally, our model achieves the best results in terms of the EwT metric (percentage of predictions within 0.02 eV of ground truth) among OOD Ads and OOD Both.
Table 2: Results on QM9 dataset for various chemical properties. t denotes using different data partitions. Bold and underline indicate the best result, and the second best result, respectively.
Task Units
A
Рномо meV
ÉgUMO meV
C2
G
H
R
U
Un
ZPVE
bohr
meV
D
cul/(mol K)
meV
meV
bohr
meV
meV
meV
SchNet
.35
63
41
34
.033
033
14
14
.073
19
14
1.70
Cormorant
.885
38
.038
.266
20
.961
21
2.02
LINet
.088
68
46
35
.043
.031
14
14
.354
14
1.56
LieCony
084
49
30
25
032
0338
22
24
800
19
19
2.28
TEN
.223
58
40
38
.64
.101
DimeNet++
.044
33
25
20
030
.023
8
7
.331
6
6
1.21
PaiNN
20
7.35
.066
5.83
TorchMD-NET
.059
36
20
18
.011
026
7.62
6.16
.033
6.38
6.15
1.84
SEGNN"
.60
42
24
21
.023
.031
15
16
.6660
13
15
1.62
EQGAT
.053
32
20
16
.011
024
23
24
.382
25
25
2.00
Equiformer
.046
.011
.023
7.63
6.63
.251
6.74
6.59
HDGNN
.046
.017
.023
.342
8.12
The QM9 benchmark provides quantum chemical properties for a relevant, consistent, and com- prehensive chemical space of 134K stable small organic molecules up 29 atoms. Each atom is described with 3D position coordinates and embedding of its atomic type (H, C, O, F). The QM9 benchmark is regression task where we optimize the MAE between multiple chemical properties and their ground truths. Here, we replace МETP with Fully-TP because the low-order Fully-TP on small molecular system can achieves the best results on two tasks and the second best results on three tasks. Note that employ the same architecture for all tasks, while TorThMD-NET and PaiNN utilize different architectures for the y and I2 tasks, potentially leading to more accurate inductive basis. Additionally, HDGNN falls behind the SOTA methods on the energy variables (G. H, U. Ug) tasks. This may be attributed to the fact that such targets may benefit from more architectures including attention mechanisms, meighbor-neighbee
cause much computational burden. From Table 2 our model
interactions and mrolemm-tailored architectures (Brandstetter et al. 2022) used in the comparative approaches.
The QM9 dataset is considerably smaller than OC20, rendering models susceptible to overfitting during learning equivariance. For instance, SCN struggles to achieve SOTA results on QM9 tasks learning equivariance, unlike other QM9 leading methods that possess inherent
due to its reliance equivariance. In contrast, our approach merges inherent equivariance with learned equivariance, empowering it to excel in QM9 tasks and achieve SOTA results.
8

---

Published as a conference paper at ICLR 2024
5.2 ABLATION STUDY
In this section, we explore several pivotal questions: 1. Can approximatively equivariant operations achieve improved performance compared to equivariant operations? 2. Does the attention module contribute to regulating both expressiveness and equivariance? 3. Are sub-structures or design in HDGNN valid? The rest ablation experiments can be found in Appendix E, which investigate the following aspects: 1) the effectiveness of normalization and the invariant branch, 2) the structure of each MLP in HDGNN, 3) the effectiveness of each component in the update block, and 4) the impact of hyperparameters. Furthermore, we analyze the training and inference times.
To begin with, we compare the strict equivariance and approximate equivariance. While HDGNN has demonstrated competitive results compared to strictly equivariant models in certain molecular tasks, is important to determine whether these improvements are attributed to the non-equivariant design. To address this, we construct two analogue equivariant models of HDGNN ("Linear" and "Fully-TP" in Table 3). One model replaces the neural networks with equivariant linear layers, while the other only uses fully-connected tensor product operations instead of equation [T] Besides, we construct a model that uses N N (x; ((F,)) )I to calculate messages ("Unrestricted"), The baseline denotes a HDGNN where K 8, C' 64 and L=6. We have three observations from Table L.although the linear structure is equivariant, it fails to capture the fine- fine-grained interaction between atoms; 2.approximately equivariant model can achieve the better generalization compared to equivariant model; 3. Designing approximately equivariant models requires careful consideration, as unconstrained learning modules can significantly undermine the generalization abilities of the models.
Table 3: Ablation studies for equivariance.
Table Ablation studies for attention module.
Energy (meV)
Model
Thain MAE
MAD
Test MAE
Model
ID)
00D Ads
0OD Cat
OOD Both
Average
(Sigmoid)
513
19.9
554
Baseline
554
701
566
642
616
(SonMax)
520
20.8
562
(Linear)
632
689
(0.25, 0.25. 0.25 0.25)
510
21.5
564
(Fully-TP))
591
752
646
704
673
0.3, 0.3, 0.1)
511
21.9
567
(Unrestricted)
883
1052
962
973
The attention module in HDGNN crucial since can adaptively adjust the equivariance of model. To evaluate it, use mean absolute difference (MAD), training MAE and test MAE to approximately denote equivariance, expressiveness and generalization, respectively. Note that MAD we used is equal to E - E"). where E is the predicted energy and E' is the predicted energy based on random SO(3) transformation. We remove the attention module of message block or replace the output activation of attention module to SoftMax. As shown in Table 4 the attention module with Sigmoid achieves the best result. Note that the last two rows in Table 4 represent using fixed weights to replace a ag, as. ay in equation [] Additionally, we manually adjust the attention coefficients to find the trade-off between cquivariance and expressiveness on QM9 task. The detail can be found in Appendix E.5.
Table 5: Ablation studies for non-equivariant module in HDGNN.
Model
Layer Norm
No Linear
Shared MLP
NN(x")
MAE (me 4
554
577
HDGNN
568
555
572
At last, we investigate our approximate equivariant module by several experiments: 1.removing layer
norm for x;; 2.introducing (equivariant) linear layer for x; and x, in message block; 3.using different neural networks in equation 11 4. transfforming NN("") + N to N NN(xx) in equation 11 We have two main observations from Table 5 one of which is that redundant linear layers do not lead to improved performance. More importantly, replacing our operation in learnable module will cause performance degradation.
CONCLUSION AND FUTURE WORK
In this paper, we offer new possible solution to predict the properties of molecules beyond the strictly equivariant neural networks and demonstrate its superiority over the QM9 benchmark and IS2RE dataset of OC20. One limitation is related to the hyper-parameter L'. Increasing L' to the higher order may bring extra benefit but leads to more computations. Hence, one direction of the future work is systematically analyze the effect of T''

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We are thankful to the anonymous reviewers for their helpful comments. This work was supported in part by the STI 2030-Major Projects of China under Grant 2021ZD0201 300, and by the National Science Foundation of China under Grant 62276127. The corresponding authors are Chao Qu and Furao Shen.
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalee Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1eXiv:806.00126) 806.01261, 2018.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Korblluhh, Nicola Molinari, Tess Smidt, and Boris Kozinsky. E (3)-cquivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1
Jörg Behler and Michele Parrinello. Generalized neural- netwolk representation of nigh-dimensional potential-energy surfaces. Physical review letters, (8(14): 146401, 2007.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve (3) equivariant message passing. iclr2022, 2021.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J. Bekkers, and Max Welling. Geomet- ric and physical quantities improve B(3) equivariant message passing. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRe- view.net, 2022.
Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build e (n)-equivariant steerable cnns. In International Conference Learning Representations, 2021.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32, 2019.
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Doming Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 11(10):059-6672, 2021.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical enns. arXiv: 1.Xi:18001.100 10130, 2018
arXiv preprint
Nadav Dym and Haggai Maron. the universality of rotation equivariant point cloud networks. In International Conference on Learning Representations, 2021.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups arbitrary continuous data. In International Conference on Machine Learning, pp. 3165-3176, PMLR, 2020.
Mare Finzi, Gregory Benton, and Andrew G Wilson. Residual pathway priors for soft equivariance constraints. Advances in Neural Information Processing Systems, 34:30037-30049, 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se transffooo:: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33: 33:1970-1991, 2020.
10

---

Published as a conference paper at ICLR 2024
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. ICLR2020, 2020.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34: 6790-6802, 2021.
Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Günnemann, Zachary Ward Ulissi, C. Lawrence Zitmick, and Abhishek Das. Gemnet-OC; Developing graph neural networks for large and diverse molecular simulation datasets. Transactions on Machine Learning Research, 2022.
Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kostiantyn Lapchevskyi, Maurice Weiler, Michat Tyszkiewicz. Simon Batzner, Dylan Madis- etti, Martin Uhrin, Jes Frellsen, Nuri Jung, Sophia Sanborn, Mingjian Wen, Josh Rackers, Marcel Rød, and Michael Bailey. Euclidean neural networks: c3nn, April 2022. URL
attps://abi.org/111 52111..51122eeeee
6459381
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference machine learning, pp. 1263-1272. PMLR, 2017.
David J Griffiths and Darrell F Schroeter. Introduction to quantum mechanics. Cambridge university press, 2018
Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks. arXiv preprint arXiv:2301.0933 2023.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR 2017, 2017.
Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty- directional message passing for non-equilibrium molecules. arXiv preprint stXiv:2011.14113 2020.
Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. Roto-translated local coordinate frames for interacting dynamical systems. Advances Neural Information Processing Systems, 34: 6417-6429, 2021.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordan nets: fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 31, 2018.
Tuan Le, Frank Noé, and Djork-Amt Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprint arXiv:2202.09891, 2022.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. The Eleventh International Conference on Learning Representations, 2023.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. ICLR2022, 2021.
Nicholas Lubbers, Justin S Smith, and Kipton Barros. Hierarchical modeling of molecular energies using deep neural network. The Journal of chemical physics, 148(2))2241755, 2018.
Benjamin Kurt Miller, Mario Geiger, Tess Smidt, and Frank Noé. Relevance of rotationally equivariant convolutions for predicting molecular properties. arXiv preprint arXiv:2008.0844 2020.
Saro Passaro and C Lawrence Zitnick. Reducing SO(3) convolutions to SO(2) for efficient equivariant gnns. volume 202 of Proceedings of Machine Learning Research, pp. 27420-27438. PMLR, 2023.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.

---

Published as a conference paper at ICLR 2024
David W. Romero and Suhas Lohit. Learning partial equivariances from data. In Advances in Neural Information Processing Systems, 2022.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning. pp. 9377-9388, 2021.
Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics. 148(24):241722, 2018.
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. International Conference on Learning Representations, 2022.
Nathaniel
Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and cranlltion-eqquivaiian neural networks for 3d point clouds. Neurips 2018, 2018.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3d protein structure for interface prediction. Advances in Neural Information Processing Systems, 32, 2019.
Tycho van der Ouderaa, David W Romero, and Mark van der Wilk. Relaxing equivariance constraints with non-stationary continuous filters. Advances in Neural Information Processing Systems, 35: 53818-33830, 2022.
Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. International Conference on Machine Learning, pp. 23078 -27091, 2022.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
Lawrence Zitnick, Abhishek Das, Adeesh Kolluru, Janice Lan, Muhammed Shuaibi, Anuroop Sriram, Zachary Ulissi, and Brandon Wood. Spherical channels for modeling atomic interactions. Neurips 2022, 2022.
12


Reviews:
[{"review_id": "BUs0NfiGWS", "reviewer": "Reviewer_rs3C", "summary": "The paper describes an hybrid approach to equivariant graph neural networks. It is hybrid in the sense that the networks contain operations that are not fully equivariant together with those that are. The motivation is that equivariance could be too strong of a constraint in many applications. Another motivation is that the Clebsch-Gordan tensor product framework is demanding, and one could by a trick one could convert the steerable features to a global basis via the Wigner-D matrices, apply an MLP, than map back via the Wigner-D matrices. The authors achieve strong results on QM9, state of the art on various metrics for the open catalyst challenge, and perform ablation studies to investigate the effect of the breaking full equivariance.\n\n*[update after the firs response I increased my score from 3 to 5]*\n\n*[update after a second round of discussion I raised my score from 5 to 6]*\n\n*[given the final modifications/clarifications I raised my score from 6 to 8, the authors effectively addressed all my initial concerns and I see no reason to reject the paper]*", "questions": "See questions in comments above. Thank you for your great efforts, it is an interesting paper, and thanks for considering my comments.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper obtains excellent results on various benchmarks\n2. The theoretical preliminaries part, and most of the rest of the paper, is well written\n3. The paper addresses a relevant topic; it investigating whether the sometimes restrictive inductive bias of equivariance can be relaxed", "weaknesses": "1. A lot of focus is on breaking of equivariance, or relaxing this constraint. There is one element that I find very important which is not discussed however:\n\nWhen converting the features to a global reference frame via the Wigner-D matrix (Equation 9), the rotation is computed from the direction: \"$R_{ij}$ denotes the 3D rotation matrix that transforms  $\\vec{r}_{ij}$ to $[0,0,1]$\". \n\nHowever, this is ill-defined as there is one degree of freedom left in the rotation matrix as any $R_{ij} R_{\\alpha}$ with $R_{\\alpha}$ any rotation around the z-axis results in the same mapping from $\\vec{r}_{ij}$ to $[0,0,1]$.\n\nThus, the equality in equation 9 is not true, and this is an error in the paper. Iis this is a dellibarate error, because you want to break equivariance? Eitherway, it should fixed or commented on.\n\nIt seems that this type of trick as seems to be adopted from [Zitnick et al. 2022], although I haven't check if the same type of error is present in that paper. However, working with local reference frames was also presented in [Kofinas et al. 2021]. See section 3.2 for their discussion on the ill-defined rotation. The claim \"... was first presented in Zitnick et al\" might therefore have to be also updated/nuanced.\n\n[Kofinas et al. 2021] Kofinas, M., Nagaraja, N., & Gavves, E. (2021). Roto-translated local coordinate frames for interacting dynamical systems. Advances in Neural Information Processing Systems, 34, 6417-6429.\n\n2. A second weakness of the paper is the complexity of the model (figure 1) with all sorts of interactions and blocks. It is great that excellent results are obtained with the model, but is it really worth the engineering effort? In my opinion, the contribution of great results is greatly diminished if it comes at the cost of such an intricate (over engineered?) network design. Then, what remains is the ablation that shows that relaxing equivariance could be benefficial. This is a message that contains scientific value, in my opinion, however, this point I think is not thoroughly discussed/analyzed.\n\n3. The main innovation of this paper, which is the partial equivariance is not clearly explained, in my opinion. Apart from the error with respect to ill-defined rotation matrix estimation, the explanaition of why one type of processes breaks equivariance and the other doesn't (Eq 11) could be better explained.\n\n4. In general, I think the paper could still benefit form being more precise on the the purpose of all included experiments (what precisely do they test for?)\n\nDetailed comments:\n1. In the introduction it is mentioned \"While equivariant neural networks exhibit appropriate inductive biases ... leading to inefficient learning of interactions between atoms\" I do not agree with (or understand) this claim at all. Precisely these domains require equivariance: it is demanded by the physics. Any operation that is non-equivariant is bound to fail, unless it *learns to be equivariant*. The property of equivariance is either way essential and thus imposing equivariance should improve efficiency in learning. Perhaps the authors mean something like that it might be more efficient to start-off unconstraint and let the network *learn equivariance* (though the task or augmentation). The paper is not clear on this, it is often suggested that one wants to break equivariance because it might not be the right inductive bias. Eitherway, I really do not understand what is meant with the statement, and I think it is wrong. Please give a counter example where the predicted properties do not require equivariance/invariance, or update the sentence.\n\n2. In 2.4, the analysis of expressive power is I think more important than the benchmark results but hidden in the appendix. Could the results be somehow briefly discussed in the main body, instead of an appendix that is not included in the main pdf?\n\n3. Above equiation 6 in the neighborhood definition please use $\\mid$ (\\mid) to leave some space between the left and right of | . Also equation 6 reads better if you include some white space after the comma. \n\n4. Please include a discussion or more details on the estimation of $R_{ij}$ from $\\vec{r}_{ij}$, as mentioned in my main comments.\n\n5. Above equation 10: \"However, the MLP structure introduces non-equivariance\" Please explain this. I do not see why it would break equivariance. In other words, if equation 9 were true (if the rotation estimate were to be well-defined), then applying an MLP on $x_i'$ and then rotate it back afterwards would be fine, right? It would only break equivariance because of the ill-defined rotation estimation, but that is separate from the use of an MLP. So I wonder what is the issue here really? Could you explain this?\n\n6. Same sentence continues with \"... rotation matrix introduces randomness\". Is this because of the free rotation angle, or what is meant here?\n\n7. Equations 10 and 11 could benefit from more explanation. Why this decomposition? What problem does it solve? Is it to split the processing into an equivariant and non-equivariant part, through NN(x_i) or NN(x_i'')? Or what is the idea behind it.\n\n8. In equation 12, what are the $a_{ij}$, and how are they computed?\n\n9. Equations 13 and 14 describe a Fourier based activation function right? Where as \"activation function\" an MLP is used. I do like this approach but I think the presentation could be simplified a bit.\n\n10. After equation 13: \"we concatenate all the elements...\" why? It seems very ad-hoc, but perhaps there is a good explanation for doing this. Could you explain why this is done?\n\n11. Next, \"Here, we extend the point-wise operation to two points by incorporating both convolution and correlation\" What does that mean? Convolution and correlation are one and the same thing, up to a flip of the kernel. This sentence makes no sense to me, please clarify what is meant by it.\n\n12. In related works: \"Such structure is limited to capture fine-grained geometric features since the learning weights are only in the radial representation which is a scalar in each filter\". *This statement overlooks the fact that also the spherical harmonics are used*. The radial basis, in combination with the spherical harmonic basis makes that the CG operations describe full group convolutions with kernels that can in principle approximate any pattern. They are only limited by the band-limit L. **Saying that the use of the radial scalar is limiting is wrong and misleading because it overlooks the use of Y**. See e.g. for more details on the expresiveness of CG based networks and the band-limit L:\n\nWeiler, M., Geiger, M., Welling, M., Boomsma, W., & Cohen, T. S. (2018). 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31.\n\nor\n\nCesa, G., Lang, L., & Weiler, M. (2021, October). A program to build E (N)-equivariant steerable CNNs. In International Conference on Learning Representations.\n\n13. \"In contrast, our method ... striking a balance between equivariance and expressiveness\". Again, I find a critical discussion on this balance missing. *All the addressed tasks demand equivariance*. The discussion should focus therefore on how certain types of equivariant networks limit expresivity. In CG approach this limit is I suppose the band-limit L. In your approach there is no inherent band-limit because of the non-equivariant MLP approach (though I'm not sure how this is proved), but equivariance should be learned somehow. Do I understand this correctly?\n\n14. Below table 1: \" This highlights the ability of HDGNN to capture more general physical properties in real world scenarios\" I do not see this claim substantiated. The only thing I see from the table is that the method outperforms several others (not always), but this does not imply a better capability to \"capture more general physical properties\". What are these \"more general properties\". There are so many differences between your architecture and those in the table. In my interpretation, HDGNN might just be better engineered to solve the task but does not give any more insights than that.\n\n15. What are the ODD average scores between parenthesis in text (GDGNN: 564 vs SCN/Equiformer: 592/613). I cannot find these in the table (sorry if I overlook them somehow)\n\n16. \"The QM9 dataset ... susceptible to overfitting\" I do not see why this is an issue. Did you indeed observe overfitting in your learning curves? If you want to test these kind of overfitting issues, perhaps the revised rMD17 would be more appropriate. In fact, I think rMD17 might be the best dataset to test your method, because the task in rRMD17 (interatomic potential energy prediction, for a single molecule given different conformations) is purely driven by geometry and should be purely invariant.\n\n17. \"In contrast, our approach merges inherent equivariance with learned equivariance,\" What mechanism induces learned equivariance? This is not discussed in the paper.\n\n18. \"While HDGNN has shown superior performance compared to strictly equivariant models\" did it? It seems sometimes it does, sometimes it doesn't. The claim should be nuanced.\n\n19. To be honest, I do not follow the paragraphs after tables 3,4. They should be improved in clarity. What research questions are being addressed here? Please be precise on the purpose of all experiments in the paper (what precisely do they test for)\n\n20. On several occasions the paper motivates the equivariance relaxation form the efficiency perspective. But what about actual speed efficiency? I see no comparison in the paper and it would be nice to see if something is gained on that level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9oTEbKGNju", "reviewer": "Reviewer_yQB9", "summary": "The paper proposes the Hybrid Directional Graph Neural Network (HDGNN), a message passing graph neural network designed to enhance expression power by relaxing equivariance. the experiments demonstrate promising result, and the ablation study shows the relevant module is useful.", "questions": "No", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the paper is well written and easy to follow\n2. experiment looks good", "weaknesses": "No", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7uKu7eaLp7", "reviewer": "Reviewer_Liwe", "summary": "– This paper presents a novel approach to an equivariant graph encoder for classification. The goal of the paper is to enhance expressiveness by relaxing the equivariance constraint, following recent success of works including (Dym and Maron 2021). \n\n– The main issue is that the representation group degree $l$, which gives higher expressive power, leads to increased computation costs in the CG product.", "questions": "– Since you are claiming that your architecture: “can enhance the expressive power suffering from the limitations of finite low-degree group representation” should this be also shown in a proposition or theorem, to complement your ablation study?\n\n– Could you expand experiments on other recent GNN benchmarks, follow Equiformer or the other works that you are comparing to.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "– Investigating an hybrid approach to improve the tradeoff between equivariance and expressive power seems a fruitful direction.", "weaknesses": "In the abstract you claim to: demonstrating its state-of-the-art performance, however,\n\n– Table 1: Your method does improve the results only in 2 out of 4 metrics.\n\n– Table 2 : Your method improves previous work on 2 out of 11 tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)