EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations
Authors: Yi-Lun Liao
Brandon M Wood
Abhishek Das
Tess Smidt
Venue: ICLR 2024 poster (2024)
Keywords: equivariant neural networks
graph neural networks
computational physics
transformer networks
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a better equivariant Transformer architecture that better leverage higher degrees of representations and the proposed method achieves state-of-the-art results on OC20 and OC22 datasets.

Abstract: Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace $SO(3)$ convolutions
with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements – attention re-normalization, separable $S^2$ activation and separable layer normalization. Putting these all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to 9% on forces, 4% on energies, offers better speed-accuracy trade-offs, and 2$\times$ reduction in DFT calculations needed for computing adsorption energies. Additionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M
datasets to better understand the performance gain brought by higher degrees.

Published as a conference paper at ICLR 2024
IMPROVED EQUIVARIANT TRANSFORMER FOR SCALING TO HIGHER-DEGREE REPRESENTATIONS
EQUIFORMERV2:
Yi-Lun Liao'
Brandon Wood
Abhishek Das
Tess Smidt
Massachusetts Institute of Technology
FAIR, Meta
Equua contribution
(ylliao, tsmidt)@mitteette edu {bmwood, www.hhhkkdweeett.ccmm
https://githan http:////..aotoooorieeeeee//eeee
ABSTRACT
Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures to higher degrees. Starting from Equiformer, we first replace SO(3) convolutions with eSCN convolutions efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements attention re- ne-ormalization, separable S activation and separable layer normalization. Putting this all together, we propose EquiformerV2 which outperforms previous state-of state-of-the-e th-the-art methods on large-scale OC20 dataset by up to 9% on forces, 4% on energies, offers better speed-accuracy trade-offs, and 2x reduction DFT calculations needed for computing adsorption energies. Addi- tionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.
scale well
INTRODUCTION
In recent years, machine learning (ML) models have shown promising results in accelerating and scaling high-accuracyy but compute-intensive counting for key features of atomic systems, such as the discrete nature of atoms, and Euclidean and permutation symmetries (Gilmer et al., 2017; Zhang et al., 201 8; Jia et al., 2020; Gasteiger et al., 2020a; Batzner al., 2022; Lu et al., 2021; Unke al., 2021; Sriram al., 2022; Rackers et al., 2023; Lan et al., 2022). By bringing down computational costs from hours or days to fractions of seconds, these methods enable new insights in many applications such as molecular simulations, material design and drug discovery. A promising class of ML models that have enabled this progress equivariant graph neural networks (GNNs) (Thomas et al., 2018; Weiler et al., 2018; Kondor et al., 2018; Fuchs et al., 2020; Batzner et al., 2022; Brandstetter al., 2022: Musaelian et al., 2022; Liao & Smidt, 2023; Passaro & Zitnick, 2023).
intensive quantum mechanical calculations by effectively ac-
Equivariant GNNs treat 3D atomistic systems graphs, and incorporate inductive biases such that their internal representations and predictions are equivariant to 3D translations, rotations and optionally inversions. Specifically, they build up equivariant features of each node as vector spaces of irreducible representations (or irreps) and have interactions or message passing between nodes based on equivariant operations such as tensor products. Recent works on equivariant Transformers, specifi- cally Equiformer (Liao & Smidt, 2023), have shown the efficacy of applying Transformers ( (Vaswani al., 2017), which have previously enjoyed widespread success in computer vision (Carion et al., 2020; Dosovitskiy et al., 2021; Touvron et al., 2020), language (Devlin et al., 2019; Brown et al., 2020), and graphs (Dwivedi Bresson, 2020; Kreuzer et al., 2021; Ying et al., 2021; Shi et al., 2022), to this domain of 3D atomistic systems.
bottleneck in scaling Equiformer as well as other equivariant GNNs is the computational complexity of tensor products, especially when we increase the maximum degree of irreps Lmea- This limits these models to use small values of Lmax (e.g., Lmaa & 3), which consequently limits their performance. Higher degrees can better capture angular resolution and directional information, which is critical to accurate prediction of atomic energies and forces (Batzner et al., 2022; Zitnick et al., 2022; Passaro Zitnick, 2023). To this end, eSCN (Passaro & Zitnick, 2023) recently proposes efficient convolutions

---

Published as a conference paper at ICLR 2024
Inpar SD Craph
Grogh
Aoee
Crestan
Akonn
Countera
Type
Drdediey
Layn howes
100 F4
Schu
(di Enbledinng Ubben Edge Degret)
Live bar
Netwel
Yow d
du Equiieeeer33
(N6 Equicarinn longh Atsedinn
Feed Fienard Nateork
Arciteetur
Figure 1: Overview of EquiformerV2. We highlight the differences from Equiformer (Liao & Smidt, 2023) in red. For (b), (c), and (d), the left figure is the original module in Equiformer. and the right figure is the revised module in EquiformerV2. Input 3D graphs are emhedded with atom and edge-degree embeddings and processed with Transformer blocks, which consist of equivariant graph attention and feed forward networks. "@" denotes multiplication, "@" denotes addition, and Σ within circle denotes summation over all neighbors. "DTP" denotes depth-wise tensor products used in Equiformer. Gray cells indicate intermediate irreps features.
to reduce SO(3) tensor products to SO(2) linear operations, bringing down the computational cost from O D(Laaw) to O(L and enabling scaling to larger values of Lmaz (e.g., Tamas up to 8). However, except using efficient convolutions for higher T.max. eSCN still follows SEGNN-like message passing network (Brandstetter et al., 2022) design, and Equiformer has been shown to improve upon SEGNN. More importantly, this ability to use higher Limax challenges whether the previous design of equivariant Transformers can scale well to higher-degree representations.
In this paper, we are interested adapting eSCN convolutions for eigher-degre representations to equivariant Transformers. We start with Equiformer (Liao & Smidt, 2023) and replace SO(3) convolutions with eSCN convolutions. We find that naively incorporating eSCN convolutions does not result in better performance than the original eSCN model. Therefore, to better leverage the power of higher degrees, we propose three architectural improvements attention re-normalization. separable 8 activation and separable layer normalization. Putting these all together, we propose EquiformerV2. which is developed on large and diverse OC20 dataset (Chanussot et al., 2021). Experiments on OC20 show that EquiformerV2 outperforms previous wate-of-the-and methods with improvements of up to 9% on forces and 4% on energies, and offers better speed- -ccuuacc trade-ofii compared existing invariant and equivariant GNNs. Additionally, when used in the AdsorbML algorithm (Lan et al., 2022) for performing adsorption energy calculations, EquiformerV2 V2 achieves the highest success rate and 2x reduction in DFT calculations to achieve comparable adsorption energy accuracies as previous methods. Furthermore, EquiformerV? V2 trained on only OC22 (Tran* al.. 2022) dataset outperforms GemNet-OC (Gasteiger et al., 2022) trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 dataset (Ramakrishnan et al., 2014; Ruddigkeit et al., 2012) and OC20 S2EF-2M dataset to better understand the performance gain of higher degrees and the improved architecture.
BACKGROUND
We present background relevant to this work here and discuss related works in Sec. B.
2.1 ( EQUIVARIANT NEURAL NETWORKS
We discuss the relevant background of S F3) -qu)vaqauva neural networks here. Please refer to Sec. A in appendix for more details of equivariance and group theory.
Including equivariance in neural networks can serve as a strong prior knowledge, which can therefore improve data efficiency and generalization. Equivariant neural networks use equivariant irreps features built from vector spaces of irreducible representations (irreps) to achieve equivariance to 3D rotation. Specifically, the vector spaces are (2L dimenniional where degree L is non-negative integer. L can be intuitively interpreted as the angular frequency the vectors, i.c., how fast the vectors rotate with respect to rotation of the coordinate system. Higher I. is critical to tasks sensitive angular
2

---

Published as a conference paper at ICLR 2024
information like predicting forces (Batzner et al., 2022; Zitnick et al., 2022; Passaro & Zitnick, 2023). Vectors of degree L are referred to as type-L vectors, and they are rotated with Wigner-D matrices D(L) when rotating coordinate systems. Euclidean vectors in E can be projected into type-L vectors by using spherical harmonics (1F)) We use order m to index the elements of Lype-L vectors, where - & m < L. We concatenate multiple type-L vectors to form an equivariant irreps feature f. Concretely, f has Ct. type-I. vectors, where I & Lmaxx and CL is the number of channels for type-I. vectors. In this work, we mainly consider Cl C, and the size of f is (Lmaz + 1)² C. We index by channel i, degree L, and order and denote as fm.)
Equivariant GNNs update irreps features by passing messages of transformed irreps features between nodes. To interact different type-L vectors during message passing, we use tensor products, which generalize multiplication to equivariant irreps features. Denoted L,,,,,- the tensor product uses Clebsch-Gordan coefficients to combine type-I1 vector f(l)) and type-L2 vector g(اa) and produces Lype-Ly vector ((aa)
L2 Σ
(Ta)) (f(h)) Lt., g (122)) mg
Σ
ciLa.ma) (Lammi)
Smi Juhi) (
L1
where my denotes order and refers to the m1-th element of f(1)), KebsschGGordan coefficients C(Laa.a))
(LL..m)(LLgmmo) are non-zero only when م - L2l & L3 & |L1 + Lzl and thus restrict output vectors to be of certain degrees. We typically discard vectors with L > Lmaz. where Liaa is hyper-parameter, to prevent vectors of increasingly higher dimensions. In many works, message passing is implemented as equivariant convolutions, which perform tensor products between input irreps features ((aa) and spherical harmonies of relative position vectors y(la)( (
2.2 EQUIFORMER
Equiformer (Liao Smidt, 2023) is an (E(3)/E(())equuuu equivariant GNN that combines the inductive bi- ases of equivariance with the strength of Transformers. First, Equiformer replaces scalar node features with equivariant irreps features to incorporate equivariance. Next, it performs equivariant operations on these irreps features and equivariant graph attention for message passing. These operations include tensor products and equivariant linear operations, equivariant layer normalization (Ba et al., 2016) and gate activation (Weiler et al., 2018). For stronger expressivity in the attention compared to typical Transformers, Equiformer uses non-lineal functions for both attention weights and message passing. Additionally, Equiformer incorporates regularization techniques commonly used by Transformers, e.g., dropout (Srivastava et al., 2014) to attention weights (Veličković et al., 2018) and stochastic depth (Huang et al., 2016) the outputs of equivariant graph attention and feed forward networks.
2.3 ESCN CONVOLUTION
eSCN convolutions (Passaro & Zitnick, 2023) are proposed to use SO(2) linear operations for efficient tensor products. We provide an outline and intuition for their method here, please refer to Sec. A.3 and their work (Passaro & Zitnick, 2023) for mathematical details.
(T.)) and spherical harmonic projec- traditional SO(3) convolution interacts input irreps features Im,
tions of relative positions Y (17)) with an SO(3) tensor product with Clebsch-Gordan coefficients (L,,)))(4,,,.)>>) The projection Yu(L)) (rij) becomes sparse if we rotate the relative position vector Fig with rotation matrix Dij to align with the direction of L 0 and to the z axis traditionally but the y axis in the conventions of e3nn (Geiger et al., 2022). Concretely,
0. which corresponds
given DifTey aligned with the y axis, (DeTT)) * only for my 0. If consider only (L.mm)).(Ly.)) * 0 only when m, = + There- fore, the original expression depending on m, My, and m, is now reduced to only depend m.- This means we are no longer mixing all integer values of m, and my, and outputs order m. are linear combinations of inputs of order -m. eSCN convolutions go one step further and replace the remaining non- non-trivial paths of the SO(3) tensor product with an S0(2) linear operation to allow for additional parameters of interaction between tm, without breaking equivariance.
my = O. m.L.
(L,,,))(L,,,m,)) can be simplified, and ((Lomma))
3

---

Published as a conference paper at ICLR 2024
Sigunai
Diideety the
Oxpat
Gate Activation
(b) Activation
Separable S Activation
Equivarionr Normalization
(b) Seporable Layer Normolization
Figure 2: Illustration of different activation functions. G denotes conversion from vectors point samples on a sphere, F can typically be a SiLU activation or MLPs, and C is the inverse of C.
Figure 3: Illustration of how statistics calculated in different normalizations. "std" denotes standard deviation, and "RMS" denotes root mean square.
3 EQUIFORMERV2
Starting from Equiformer (Liao & Smidt, 2023), we first use eSCN convolutions to scale higher- degree representations (Sec. 3.1). Then, we propose three architectural improvements, which yield further performance gain when using higher degrees: attention re-normalization (Sec. 3.2), separable S activation (Sec. 3.3) and separable layer normalization (Sec. 3.4). Figure illustrates the overall architecture of EquiformerV2 and the differences from Equiformer.
3.1 INCORPORATING ESCN CONVOLUTIONS FOR HIGHER DEGREES
The computational complexity of SO(3) tensor products used in traditional SO(3) convolutions scale unfavorably with Lmaz. Because of this, it is impractical for Equiformer to use Lmaz > 2 for large-scale datasets like OC20. Since higher Lmaa can better capture angular information and are correlated with model expressivity (Batzner et al., 2022), low values of Lmoz can lead to limited performance on certain tasks such as predicting forces. Therefore, we replace original tensor products with eSCN convolutions for efficient tensor products, enabling Equiformer to scale up Lmazz to 6 or 8 on OC20 dataset. Equiformer uses equivariant graph attention for message passing. The attention consists of depth-wise tensor products, which mix information across different degrees, and linear layers, which mix information between channels of the same degree. Since eSCN convolutions mix information across both degrees and channels, we replace the SO(3) convolution, which involves one deit tensor product layer and one linear layer, with a single eSCN convolutional layer, which consists of rotation matrix Dij and SO(2) linear layer as shown in Figure lb.
3.2 ATTENTION RE-NORMALIZATION
Equivariant graph attention in Equiformer uses tensor products to project node embeddings x; and Ij. which contain vectors of degrees from to Immer to scalar features f and applies non-linear functions to _( for attention weights aij- We propose attention re-normalization and introduce one additional layer normalization (LN) (Ba et al., 2016) before non-linear functions. Specifically, given f()) we first apply LN and then use one leaky ReLU layer and one linear layer to exp(s.y) calculate Zij wi LeakyReL.((LN( and aiÿ- = s0ft Ekext exp(=sx] where wn is
a learnable vector of the same dimension as p(0) The motivation is similar to ViT-22B (Dehghani et al., 2023), where they find that they need an additional layer normalization to stabilize training when increasing widths. When we scale up Lmazz we effectively increase the number of input channels for calculating fij By normalizing f! the additional LN make sure the inputs to subsequent non-linear functions and softmax operations still lie within the same range as lower Lmaa is used. This empirically improves the performance as shown in Table la.
3.3 SEPARABLE S? ACTIVATION
The gate activation (Weiler al., 2018) used by Equiformer applies sigmoid activation scalar features to obtain non-linear weights and then multiply irreps features of degree > 0 with non- linear weights add non-line -on-aiicyr to equivariant features. The activation only accounts for the interaction from vectors of degree ( to those of degree > 0 and can be sub-optimal when we scale up Tinaz- To better mix the information across degrees, SCN (Zitnick et al., 2022) and eSCN adopt 5 activation (Cohen et al., 2018). The activation first converts vectors of all degrees to point samples on a sphere for each channel, applies unconstrained functions F to those samples, and finally convert them back to vectors. Specifically, given an input irreps feature x E ( +1)2xC the output is y G (F((()))) where G denotes the conversion from vectors to point samples on sphere, F
4

---

Published as a conference paper at ICLR 2024
can be typical SiLU activation (Elfwing et al., 2017; Ramachandran et al., 2017) or typical MLPs, and G 1 is the inverse of G. We provide more details of S activation in Sec. A.4.
However, find that directly replacing the gate activation with S activation in Equiformer results in large gradients and training instability (Index 3 in Table Ia). To address the issue, we propose separable 8 activation, which separates activation for vectors of degree and those of degree >> O. Similar to gate activation, we have more channels for vectors of degree O. As shown in Figure 2c, we apply SiLU activation to the first part of vectors of degree 0, and the second part of vectors of degree O are used for 5 activation along with vectors of higher degrees. After S activation, we concatenate the first part of vectors of degree 0 with vectors of degrees > 0 as the final output and ignore the second part of vectors of degree O. Separating the activation for vectors of degree O and those of degree > 0 prevents large gradients, enabling using more expressive 5 activation for better performance. Additionally, we use separable 5 activation in feed forward networks (FFNs). Figure 2 illustrates the differences between gate activation, S activation and separable 5 activation.
3.4 SEPARABLE. LAYER NORMALIZATION
Equivariant layer normalization used by Equiformer normalizes vectors of different degrees inde- pendently. However, it potentially ignores the relative importance of different degrees since the relative magnitudes between different degrees become the same after the normalization. Therefore, instead of performing normalization to each degree independently, motivated by the separable S activation mentioned above, we propose separable layer normalization (SLN), which separates nor- malization for vectors of degree 0 and those of degrees > O. Mathematically. let E ( +C)xxC denote input irreps feature of maximum degree Lumax and C channels, and (i denote the L-th degree, m-th order and i-th channel of x. SIN calculates the output y as follows. For L = 0,
(())) 700
(210) o(0)
(0)
+ 1,(00,
where (0 č Ei
(0) #0,s and a (0) V 2i=1 (M0. 1000)22
For O, y(L) 7(0) o
(2) o.Looy where a (LL=0)
Limas Eh (o(4)) and (L)
V -i=1 2T+ -
(L) (mm.)
(1) (2), 3(0) E RCC are learnable parameters, f (0) and o
are mean and standard deviation of vectors of degree 0, (1) and (l.=00) are root mean square values (RMS), and o denotes clement-wise product. Figure 3 compares how μ!(), _(0), g(L) and ((L>0) are calculated in equivariant layer normalization and SLN. Preserving the relative magnitudes between degrees > 0 improves performance as shown in Table la.
3.5 OVERALL ARCHITECTURE
Equivariant Graph Attention. Figure 1 illustrates equivariant graph attention after the above modifications. Given node embeddings 24 and zy., first concatenate them along the channel dimension and then rotate them with rotation matrices Di, based on their relative positions or edge directions Fiy. We replace depth-wise tensor products and linear layers between Fi, Ty and fis with a single SO(2) linear layer. To consider the information of relative distances [|Fall, we transform || |I with a radial function to obtain edge distance embeddings and then multiply the edge distance embeddings with concatenated node embeddings before the first SO(2) linear layer. We split the outputs fij of the first S0(2) linear layer into two parts. The first part scalar features f(0). which only contains vectors of degree 0, and the second part is irreps features f.g and includes vectors
of all degrees up to Lmaa- As mentioned in Sec. 3.2, we first apply an additional LN to f and then follow the design of Equiformer by applying one leaky ReLU layer, one linear layer and a final softmax layer to obtain attention weights azy. As for value Usy. we replace the gate activation with separable 5 activation with being single SiLU activation and then apply the second S0(2) linear layer. While in Equiformer, the message mij sent from node j to node is miy ag x Vij. here we need to rotate aij x Vij back to original coordinate frames and the message my becomes Dij' (ai x Uty). Finally, we can perform parallel equivariant graph attention functions given fiz- The h different outputs are concatenated and projected with a linear layer to become the final output Yt. Parallelizing attention functions and concatenating can be implemented with "Reshape".
Feed Forward Network. As shown in Figure 1d, we replace the gate activation with separable 52 activation. The function F consists of a two-layer MLP, with each linear layer followed by SiLU, and final linear layer.
5

---

Published as a conference paper at ICLR 2024
Atlention
eSCN
EquiformerV2
Index te normalizaaion Activation
Nommalization
Epochs
Forces
undrgy
Lunca
Epachs
forces
energy
forces.
energy
Gate Gate
LN
12
21.85
286
6
12
21.3
294
20.46
285
LN
12
21.86
279
19.78
780
5
LN
12
converge
6
20.1
285
19:42
278
Sep. 8
LN
12
20.77
285
Sep. g2
SINN
12
20,46
285
8
21.3
296
20.46
279
6
Sep. g
LN
20
20.02
276
19.95
273
Sep. S
SLN
20
19.72
278
8 eSCN haseline
12
21.3
204
(a) Architectural improvements. Attention re-normalization improves energies, and separable S activation ("Sep. 5²x)) and separable layer normalization ("SLN") improve forces.
(b) Training epochs. Training for more epochs consistently leads to better results.
eSUN
EquiformerV2
eSCN
EquiformerV2
eSCN
EquiformerV2
frose
Forees:
energy
Torces
energy
Misoe
Toroes
energy
forces
energy
Layers
Toroes
enengy
forces
energy
22.2
21.37
284
285
22.4
306
21.3
294
20.46
285
3
21.2
295
20.24
284
12
21.3
291
20:46
265
20.46
279
282
16
20.5
20.26
278
(c) Degrees Lmuz. Higher degrees are consistently helpful.
(d) Orders Mmaz. Higher orders mainly improve energy predictions.
(e) Number Transformer blocks. Adding more blocks can help both
force and energy predictions.
Table I: Ablation studies of EquiformerV2. We report mean absolute errors (MAE) for forces in meA and energy in meV, and lower better. All models are trained on the 2M split of the OC20 S2EF dataset, and errors are averaged over the four validation sub-splits. The base model setting is marked in gray
Embedding. This module consists of atom embedding and edge-degree embedding. The former is the same as that in Equiformer. For the latter, as depicted in the right branch in Figure Ic, we replace original linear layers and depth-wise tensor products with a single SO(2) linear layer followed by rotation matrix D.j'' - Similar to equivariant graph attention, consider the information of relative distances by multiplying the outputs of the (0) linear layer with edge distance embeddings.
Radial Basis and Radial Function. We represent relative distances ||Fi with finite radial basis like Gaussian radial basis functions (Schütt et al., 2017) to capture their subtle changes. We transform radial basis with a learnable radial function to generate edge distance embeddings. The function consists of a two-layer MLP, with each linear layer followed by LN and SiLU, and a final linear layer.
Output Head. To predict scalar quantities like energy, we use a feed forward network to transform irreps features each node into scalar and then sum over all nodes. For predicting atom-wiss forces, we use a block of equivariant graph attention and treat the output of degree as our predictions.
4 EXPERIMENTS
4.1 OC20 DATASET
Our main experiments focus on the large and diverse OC20 dataset (Chanussot* et al., 2021). Please refer Sec. D.I for details on the dataset. We first conduct ablation studies on EquiformerV2 trained on the 2M split of OC20 S2EF dataset (Sec. 1... Then, we report the results of training All and All+MD splits (Sec. 4. 1.2). Additionally, we investigate the performance of EquiformerV? when used in the AdsorbML algorithm (Lan et al., 2022) (Sec. 4.1.3). Please refer to Sec. C Cnd D for details of models and training.
4.1.1 ABLATION STUDIES
Architectural Improvements. In Table 1a, we start with incorporating eSCN convolutions into Equiformer for higher-degree representations (Index 1) and then ablate the three proposed architectural improvements. First, with attention re-normalization energy mean absolute errors (MAE) improve by 2. 2.4%, while force MAE are about the same (Index 1 and 2). Second, we replace the gate activation with S2 activation, but that does not converge (Index 3). With the proposed S activation, we stabilize training and successfully leverage more expressive activation to improve force MAE (Index 4). Third, replacing equivariant layer normalization with separable layer normalization further improves force MAE (Index 5). We note that simply incorporating eSCN convolutions into Equiformer and using higher degrees (Index 1) do not result in better performance than the original eSCN baseline (Index 8), and that the proposed architectural changes are necessary. We additionally compare the performance gain of architectural improvements with that of training for longer. Attention re- re-nomalizzttonn (Index I and 2) improves energy MAE by the same amount increasing training epochs from 12 to 20

---

Published as a conference paper at ICLR 2024
Thoggge
SI8P wdifutor
STAP
1STRS
Trunteg
ou
Trinirgrim (GPUidk)))
Samppter/ vPU
Ebergy MAU
Foeer MAI
Exrry MAIC
Fhens MAI
AUST
ATICT
Energy MAI
Moskl
(reeV/A)
(nw/WA1
(9911
(90
Sebheet (icinaaetl....... 2017)
0,33
194
549
56.8
540
54.2
14.4
T4P
31-410000 0030 01305119-45
BieadNos-L--- (Qaasge cial, 20ON
10.34
1889
4.6
$1.3
11.8
460
11.3
23.2
53-3
559
CemNen-f (Klicpers 9077)
483
75
11.3
-8.3
GraNNes0 (Gadeger a-. 2022
394
315
18.3
244
21.7
255
26.7
15.3
14.3
555
k-20-22tzsk ai 2002)
eSEN Kult (Paamm.
N011)
MBM
131
06.3
541
Rysstomerr (ke 153M)
197
1344
18
236
150
220
1436
$3.9
068
316
Littaane 2002)
2156
765
2012
1941
N64
GeaNeesOr L.FE (Cuskdgo clal. 3002)
Laid Ka.locomkk 20023
27132
1280
12.2
672.5
321
esEN Los ku (taaar x ciris 2023)
35384
306
20
114
15.0
53.3
16.3
Bysitonerr2
(56M)
153M
1368
146
08.5
Liutmeen22 lAg
13304)
13330
1373
1.6
15.0
110
142
54,
66.4
300
Table 2: OC20 results on S2EF validation and test splits, and IS2RS and IS2RE test splits when trained on
OC20 S2EF-AIl or S2EF-AIl+MD splits. Throughput is reported as the number of structures processed per GPU-second during training and measured on V100 GPUs. A the coefficient of the energy loss.
30
Gemfett-dT dT (2021)
Force MAE (mev (
20-
Farce MANE (mee (A)
18
SCN (2022)
26-
GemNett-C (2022)
eSCN (2023)
24-
24
EquiformerV2
22-
22
30-
18-
18 500
10
20
40
60
1000
2000
6000
8000
Samples per second
Training GPU hours
(a) Trade-offs between inference speed
(b) Trade-off's between training cost and validation force MAE
and validation force MAE
Figure 4: EquiformerV2 achieves better accuracy trade-off's both in terms of inference speed well training cost. All models are trained on the S2EF-2M split and measured on V100 GPUs.
(Index and 7). The improvement of SIN (Index 4 and 5) in force MAE is about 40% of that of increasing training epochs from 12 to 20 (Index 5 and 7), and SLN is about 6% faster in training. We also conduct the ablation studies on the QM9 dataset and summarize the results in Sec. F3.
Scaling of Parameters. In Tables 1c, 1d, le, we vary the maximum degree Lmaar the maximum order Miaaz and the number of Transformer blocks and compare with equivalent eSCN variants. Across all experiments, EquiformerV2 performs might intuitively expect higher resolutions and larger models to perform better, this is only true for EquiformerV2, not eSCN. For example, increasing Lmaz from 6 to 8 or Mmar from 3 to 4 degrades the performance of eSCN on energy predictions but helps that of EquiformerV2. In Table 1b, we show that increasing the training epochs from 12 to 30 epochs can consistently improve performance.
perforns better than its eSCN counterparts. Besides, while one
Speed-Accurace Trade-offs. We compare trade-offs between inference speed or training time and force MAE among prior works and EquiformerV? V2 and summarize the results in Figure 4a and Figure 4b. EquifomerVV2 achieves the lowest force MAE across a wide range of inference speed and training cost.
4.1.2 MAIN RESULTS
Table 2 reports results on the test splits for all the three tasks of OC20 averaged over all four sub-splits. Models are trained on either OC20 S2EF-All A2DD splits. All test results are computed via the EvalAl evaluation server'. We train EquiformerV2 of two sizes, one with 153M parameters and the other with 31M parameters. When trained on the S2EF-All+MD split, EquiformerV2 (Ag = 4, 153M) improves previous state-of-the-art S2EF energy MAE by 4%, S2EF force MAE by 9%, IS2RS Average Forces below Threshold (AFbT) by absolute 4% and IS2RE energy MAE by 4%. In particular, the improvement in force predictions is significant. Going from SCN to eSCN, S2EF test force MAE improves from 17.2 me V/À to 15.6 V/À largely due to replacing approximate equivariance in SCN with strict equivariance in eSCN during message passing. Similarly, by scaling up the degrees of representations in Equiformer, EquiformerV2 (Ag 4, 153M) further improves force MAE to 14.2 me V/Á, which is similar to the gain of going from SCN to eSCN, Additionally, the smaller EquiformerV2 (Ag = 4, 31M) improves upon previously best results for all metrics except IS2RS AFbT and achieves comparable training throughput to the fastest GemNet-OC-L-EE Although the training time of EquiformerV2 is higher here, we note that this because training EquiformerV2
eval. htt.a///eeee//allllmee//eeelleee

---

Published as a conference paper at ICLR 2024
Medel
relutation
Snocess
Speedsp
Speedup
Seccess
Speedop
Spectup
Seccess
Speedup
SchNar (Schatia 2017)
2.77%
43666.1
3.91%
2155.36
4.325
1458.77
4.73%
1104.55
5.045
892.77 79
DineNetsa (Gaskige et a.all
5.34%
4271.23
7.61%
2149.78
8.849
1435.21
10,07%
1081.96
10.79%
865.20
PaNN (Schir ot 2021)
20213
27.4415
4089.77 77
13619
307765
38,0%%
1995
18.69%
1088.5
39,57%
840.45
GenNe-OC: (Castrigm et 2022)
9.4
68, 7616
4185 16
77.29%
2067 11
80. 79%
1992.1
51 5.5
1946. 85
82.9416
240
GeaNer OC MD cCasseger al.,, 2002)
9.4
65.78%
4182.04
78.21%
2090.27
81.815
1404_11
53.25%
1053.36
84.38%
841.64
CeutNet OC MD Large (Gistelger et al. 2022)
43.0
78.18%
4078.76
79.65%
2065.15
83.2516
1391,39
55.41%
1041.50
86.02%
K34.46
SCN-MD4 angs (Zitmck ct 2022)
120.0
77.80%
3974,32
86.
1331.43
87:33
87.775
Equifemm
4.81M)
17%17%
3983.4]
87.15%
1992.64
87.87%
1331.35
53.69%
1000.62
Iiqifoemer 4.55M)
85.41%
40001/71
88.90%
2012.47
1352.05
1016.31
91.57%
Table 3: AdsorbML results with EquiformerV2 (Ag = 1. 31M) and (Ag 4, 153M) trained on S2EF-A1I+MD from Table 2. We visualize the speed accuuray trade-off's of different models in Figure 5,
SIBR Total valiaiiie
SIER Total
[SGRE- Tind toi
Nanderef
Exarg Mal (uzh)) 1 Ferze taadkik
Exccar MAA
Forro
Excra MAE (rac))
Madel
Tiiking Sat
Liear oefunce
0200
Gi (Casnpper d. 3007)
1081
30
24.4
24.5
1726
0022
Gat cit (iuutipe x. 3420
0022-0023
36A3
466
359
27
311
686
26.9
58.3
1230
1534
dSCN (Phssar & Zinnes, 3025)
2008
350
23.8
TypiforeWW 1.Ar -
OC73
120M
343
25.33
14:1
1828
637
23:46
15:57
1884
LigttomerV 4.3g 1000
12233
027
22.88
38.70
253.7
659.8
33.88
1139
Table 4: OC22 results on S2EF- Total validation and test splits and IS2RE-Total test split.
for longer keeps improving performance and that we already demonstrate EquiformerV2 achieves better trade-off's between force MAE and speed.
4.1.3 ADSORBML RESULTS
Lan al. (2022) recently proposes the AdsorbML algorithm, wherein they show that recent state-of- the-art GNNs can achieve more than 1000 speedup over DFT relaxations at computing adsorption energies within V. margin of DFT results with an 87% success rate. This done by using oc2o-trained models to perform structure relaxations for average 90 configurations of an adsorbate placed on a catalyst surface, followed by DFT single-poin calculations for the top-k structures with lowest predicted relaxed energies, as a proxy for calculating the global energy minimum or adsorption energy. We refer readers to Sec. D.4 and the work (Lan et al., 2022) for more details. We benchmark AdsorbML with EquiformerV2, and Table shows that EquiformerV2 (Ag = 4, 153M) improves over SCN by significant margin, with 8% and 5% absolute improvements at k = 1 and k = 2, respectively. Moreover, EquiformerV2 (Ag 4, 153M) at k- 2 is more accurate adsorption energy calculations than all the other models even at I- = 5. thus requiring at least 2x fewer DFT calculations. Since the speedup with respect to using DFT for structure relaxations and that ML models are much faster than DFT, the speedup is dominated by the final DFT single-point calculations and MI. models with the same value of l: have roughly the same speedup. To better understand the speed-accuracy trade-off's of different models, we also report average GPU-seconds of running one structure relaxation in Table 3. Particularly, EquiformeVV2 (Ag = 4, 31M) improves upon previous methods while being 3.7x to 9.8x faster than GemNet-OO--MD-Larg and SCN, respectively.
4.2 OC22 DATASET
Dataset. The OC22 dataset (Tran* et al., 2022) focuses oxide electrocatalysis. One crucial difference between OC22 and OC20 that the energies in OC22 are DFT total energies. DFT total energies are harder to predict but are the most general and closest to DFT surrogate, offering the flexibility to study property prediction beyond adsorption energies. Similar to OC20, the tasks in OC22 are S2EF-Total and IS2RE- -otal. We train models on the OC22 S2EF-Total dataset and evaluate them on energy and force MAE on the S2EF- Total validation and test splits. We use the trained models perform structural relaxations and predict relaxed energy. Relaxed energy predictions are evaluated on the IS2RE-Tota test split.
Training Details. Please refer to Section E. for details on architectures, hyper-parameters and training time.
Results. We train two EquiformerV2 models with different energy coefficients Ag and force coefficients Ay. We follow the practice of OC22 models and use linear reference (Tran* et al., 2022). The results are summarized in Table 4. EquiformerV2 improves upon previous models on all the tasks. EquiformerV2 (Ag = 4, Ap 100) trained on only OC22 achieves better results on all the tasks than GemNet-OC trained on both OC20 and OC22. We note that OC22 contains about 8. 3.4M structures and OC20 contains about 130M structures, and therefore EquiformerV2 demonstrates significatly better data efficiency. Additionally, the performance gap between eSCN and Equiformer V2 is larger than that on OC20, suggesting that more complicated structures can benefit more from the proposed architecture. When trained on OC20 S2HF-AII+MD, EquiformerV2 (A 4, 153M) improves upon eSCN by 4% on energy MAE and 9% on force MAE. For OC22, EquiformerV2 (Ag = 4, Xr 100) improves upon eSCN by 18.9% on average energy MAE and 8.9% on average force MAE.
8

---

Published as a conference paper at ICLR 2024
Task
a
Ac
FLUMD meV
C2
G
ne
Un mnc
ZPVE
Model
Units
meV
meV
D
calimol K
meVV
meV
meVV
meVV
DimeNetee (Gasteiger et aall..200 2020a)
.044
25
.023
1.21
EGNN (Satorms et al., 2021)
.07
48
29
25
(029
.911
12
12.
.106
12
11
1:55
PaiNN (Schint et al., 202.)
1445
46
28
20
.012
.044
7.35
5.98
.066
5.83
5.85
1.28
ToT (Thitike & Fabritis, 2022)
.059
36
20
.026
7.62
6.16
.033
6.38
6,15
1.84
SphereNet (1 et 2022)
046
23
.011
.292
1.12
SEGNN (Brandteeller et ol., 2022)'
060
42
24
21
023
D31
15
16
.660
13
1.5
1.62
EQGAT (Le al., 2022)
.03
32
20
16
.011
1724
23
24
.382
25
25
2,00
Equiformer (Lian Smidk, 2023)
.446
7.63
6.74
EquiformerV2
.050
29
14
.010
.03
7.57
6.22
.186
6.49
6.17
1.47
Table 5: Mean absolute results on QM9 test set. f denotes using differrnt data partitions.
Aesos
Energy MAE (meV))
Focce MAR (meV)A1)
Treating time (GPU--bour))
Equiforner (Lito & Suniet, 2023)
297
27.57
1365
Equiformer (Line & Smidi 2023)
00M
0OM
DOM
Egusformer22
298
26.24
600
EquiforeeV22
284
21:37
966
Eguiforner2
20.46
Table 6: Comparison OC20 S2EF-2M dataset. Errors are averaged over the four validation sub-splits. "OOM" denotes out-of-memory error, and we cannot use T-maz 3 for Equiformer.
4.3 COMPARISON WITH EQUIFORMER
QM99 Dataset. We follow the setting of Equiformer and train EquiformerV2 on the QM9 dataset (Ra- makrishnan et al., 2014; Ruddigkeit et al., 2012) and summarize the results in Table 5. The per- formance gain of using higher degrees and the improved architecture is not as significant as that on OC20 and OC22 datasets. This, however, is not surprising. The training set of QM9 contains only 110k examples, which is much less than OC20 S2EF-2M with 2M examples and OC22 with 8.4M examples. Moreover, QM9 has much less numbers of atoms in each example and much less diverse atom types, and each example has less angular variations. Nevertheless, EquiformerV? achieves better results than Equiformer on out of the 12 tasks and is therefore the overall best performing model. We additionally train EquiformerV V2 with Noisy Nodes (Godwin et al., 2022) to better understand the gain of higher degrees in Sec. F.1.
OC20 S2EF-2M Dataset. We use similar configurations (e.g., numbers of blocks and numbers of channels) and train Equiformer on OC20 S2EF-2M dataset for the same number of epochs as training EquiformerV2. We vary Tanax: for both Equiformer and EquiformerV2 and compare the results in Table 6. For Immax = 2, EquiformerV2 is 2.3x faster than Equiformer since EquiformerV2 uses eSCN convolutions for efficient (3))) convolutions. Additionally, EquiformerV2 achieves better force MAE and similar energy MAE, demonstrating the effectiveness of the proposed improved architecture. For Lina > 2, we encounter out-of-memory errors when training Equiformer even after we reduce the number of blocks and use the batch size = 1. In contrast, We can easily train EquiformerV2 with Lmaz up to When increasing Linar from 2 4, EquiformerV2 achieves lower energy MAE and significantly lower force MAE than Equiformer and requires 1.4x less training time. The comparison suggests that complicated datasets have more performance to gain from using more expressive models, enabling better performance and lower computational cost.
Discussion. One limitation of EquiformerV V2 is that the performance gains brought by scaling to higher degrees and the proposed architectural improvements can depend on datasets and tasks. For small datasets like QM9, the performance gain is not significant. We additionally compare Equiformer and EquiformerV2 on OC20 IS2RE dataset in Sec. D.5. For different tasks, the improvements are also different, and force predictions benefit more from better expressivity than energy predictions. We note that the first issue can be mitigated by first pre- pre-tnain on large datasets like OC20 and PCQM4Mv2 (Nakata & Shimazaki, 2017) optionally via denoising (Godwin et al., 2022: Zaidi et al., 2023) and then fine-tuning on smaller datasets. The second issue might be mitigated by combining DFT with MI. models. For example, AdsorbML uses MI. forces for structural relaxations and single-point DFT for calculating the final relaxed energies.
CONCLUSION
In this work, we investigate how equivariant Transformers can be scaled up to higher degrees of equivariant representations. We start by replacing SO(3) convolutions in Equiformer with eSCN convolutions, and then we propose three architectural improvements to better leverage the power of higher degrees- attention re-normalization. separable S activation and separable layer normalization. With these modifications, we propose EquifornerV2, which outperforms state-of-thee fh-the-rrt art methods on all the tasks the OC20 and OC22 datasets. improves speed-accuracy trade-offs, and achieves the best success rate when used in AdsorbML. We also compare EquiformerV? with Equiformer to better understand the performance gain brought by higher degrees and the improved architecture.

---

Published as a conference paper at ICLR 2024
6 ETHICS STATEMENT
EquiformerV2 achieves more accurate approximation of quantum mechanical calculations and demonstrates one further step toward being able replace DFT compute force fields with machine leamned ones for practical applications in chemistry and material science. We hope these promising results will encourage the community make further progress in applications like material design and drug discovery, rather than use these methods for adversarial purposes. We note that these methods only facilitate the identification of molecules or materials with specific properties; there remain substantial hurdles to synthesize and deploy such molecules materials at scale. Finally, we note that the proposed method is general and can be applied to different problems like protein structure prediction (Lee et al., 2022) long inputs can be modeled as 3D graphs.
REPRODUCIBILITY STATEMENT
We include details on architectures, hyper-parameters and training time in Sec, D.2 (OC20), Sec. E.1 (OC22) and Sec. F.2 (QM9).
The code for reproducing the results of EquiformerV2 trained on OC20 S2EF-2M and QM9 datasets is available at https: //g //ggtgttthuuhu thub. com/a http://ttnicccciieeeeeeeuoooooot
ACKNOWLEDGEMENT
We thank Larry Zitnick and Saro Passaro for helpful discussions. We also thank Muhammed Shuaibi for helping with the DFT evaluations for AdsorbML (Lan et al., 2022). We acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther al., 2018) for providing high performance computing and consultation resources that have contributed to the research results reported within this paper.
Yi-Lun Liao and Tess Smidt were supported by DOE ICDI grant DE-SC0022215.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arxiv preprint 1607v:600.0064 2016.
Ilyes Batatia, David Peter Kovacs, Gregor N. C. Simm, Christoph Ortner, and Gabor Csanyi. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Korbluthh Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1), May 2022. doi: 10. 1111038/4414677-002299999999 URL https: //toi org/10 10. 1038/s41467-022 020-777--22222292-99922
Johannes Brandstetter, Rob Hesselink, Elise der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve e(3) equivariant message passing. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview httt/foormmidd== Xww_gBBe1BBBB dorrmmiiidswwwraddeeee
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
10

---

Published as a conference paper at ICLR 2024
Lowik Chamussot*, Abhishek Das* Siddharth Goyal*, Thibaut Lavril Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier MerassDominge Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi:
11100///cccttaal00005555
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference on Learning Representations (ICLR), 2018. URL http:://////oeeeeeiiee. wwwp://rufffffd Hkbd5xZRb.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang. Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf. Mario Lučié, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.088 05442, 2023.
Jacob Devlin, Ming-Weee-ee Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. arxiv preprint 1r1.v:1810.04480000 2019.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021. URL https: //openreview.net/eeeeeeeeeeeeeee www.net/forrmmiccciccdaanyy
Mildred S Dresselhaus, Gene Dresselhaus, and Ado Jorio. Group theory. Springer, Berlin, Germany, 2008 edition, March 2007.
Vijay Prakash Dwivedi and Xavier Bresson. generalization of transformer networks graphs. arxiv preprint arxiv:20/2.09699 2020.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. arXiv preprint 1rXiv:1702.03118 2017.
Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Connor Coley, and Vijay Gadepally. Neural scaling of deep chemical models. ChemRxiv, 2022. doi: 10.26434/chemrrxx-20000 2012iiv-222235552222
Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. transfrrrnsss 3d roto- translation equivariant attention networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. In Machine Learn- ing for Molecules Workshop, NeurlPS, 2020a.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. International Conference on Learning Representations (ICLR), 2020b.
Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Günnemann, Zachary Ulissi, C Lawrence Zimick, and Abhishek Das. GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets. Trunsactions on Machine Leurning Research (TMLR), 2022.
Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kos- tiantyn Lapchevskyi, Maurice Weiler, Michal Tyszkiewicz, Simon Batzner, Dylan Madisetti, Martin Uhrin, Jes Frellsen, Nuri Jung, Sophia Sanbom, Mingjian Wen, Josh Rackers, Marcel Rød, and Michael Bailey. c3mn/e3mn: 2022-04- 04- April 2022. URL https: //oi org/10. 5281/zenod 6459381.
11

---

Published as a conference paper at ICLR 2024
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley. Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
Jonathan Godwin, Michael Schaarschmidt, Alexander L Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. International Conference on Learning Representations (ICI.), 2022. URL https: http::///pppeeeeeeeeeeeeeeermmmp wtw.://e//oorummtttw/wwwwweobbb
B. Hammer, L. Hansen, and J. K. Nørskov. Improved adsorption energetics within density- functional theory using revised sarddew-burkeeerzzeee functionals. Phys. Rev. B, 1999.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016.
Weile Jia, Han Wang, Mohan Chen, Denghui Lu, Lin Lin, Roberto Car, Weinan E, and Linfeng Zhang. Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations (ICLR), 2021. URL httpss///pperevvew wwwpeeeeeefforreeeeeeeeee mett/forummxd///LDDSs56663
Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordee nets: fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems 32, pp. 10117-10126, 2018
Devin Kreuzer, Dominique Beaini, William I. Hamilton, Vincent I Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In Advances in Neural Information Process- ing Systems (NeurIPS), 2021. URL https://openrevieeet wttp:/rwwwwwwwweeeeeeeeeeeeeeeee www./eett/ooineee met//formmid=thuddB--
Janice Lan, Aini Palizhati, Muhammed Shuaibi, Brandon M Wood, Brook Wander, Abhishek Das, Matt Uyttendacle, C Lawrence Zitnick, and Zachary W Ulissi. AdsorbML: Accelerating adsorption energy calculations with machine learning. arXiv preprint arXiv:2211.16480 2022.
Tuan Le, Frank Noé, and Drnrk-Arné Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprint arXiv:2202.09891, 2022.
Jae Hyeon Lee, Payman Yadollahpour, Andrew Watkins, Nathan C. Frey, Andrew Leaver-Fatt Fayer-Fay. Stephen Ra, Kyunghyun Cho, Vladimir Gllggrijjvvic, Aviv Regev, and Richard Bonneau. Equifold: Protein structure prediction with a novel coarse-grained structure representation. bioRxiv, 2022. doi: 10. 101.2022.00.00.511322... URL httpp::/www.birrrtt biorxiv. htt.://www..o..ooooe/////e//////
2022. 10.1000000 511322.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations (ICLR), 2023. URL https:
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference Learning Representations (ICLR), 2022. URL /ttt:://openreviee ne //tt//rrmmm d=ggssssssssssstttrrr
Denghui Lu, Han Wang, Mohan Chen, Lin Lin, Roberto Car, Weinan E, Weile Jia, and Linfeng Zhang. 86 pflops deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy. Computer Physics Communications, 259:107624, 2021. ISSN 0010-4655. doi: atticlle/pi//00018455228222
URL https:/ /www. htttp::/ww..ccceenedireee.. wtt.//ccccccc//c//oe/e...ccccece
12

---

Published as a conference paper at ICLR 2024
Benjamin Kurt Miller, Mario Geiger, Tess E. Smidt, and Frank Noé. Relevance of rotationally equivariant convolutions for predicting molecular properties. arxiv preprint arxiv: 08461, 2020.
Albert Musaclian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J. Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. arxiv preprini arxiv:2204.05224 2022.
Maho Nakata and Tomomi Shimazaki. Pubchemqe project: A large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57 6: 1300-1308, 2017.
Saro Passaro and C Lawrence Zitnick. Reducing SO(3) Convolutions to SO(2) for Efficient Equivari- ant GNNs. In International Conference on Machine Learning (ICML), 2023.
Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R. Manby, and Thomas F. Miller. OrbNet: Deep learning for quantum chemistry using symmetry-adapteer atomic-orbital features. The Journal of Chemical Physics, 2020.
Joshua A Rackers, Lucas Tecot, Mario Geiger, and Tess E Smidt. A recipe for cracking the quantum scaling limit with machine learned electron densities. Machine Learning: Science and Technology, 4(1):015027, feb 2023. doi: 10. 10.1088/2632-21555achb333 2632-2153/acb314
URL https://dx.doo www..dooo.og///ooo 10. 1088/
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018.
Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Lean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, 52 52(11):2866 010:286442775. 2012. doi: 10.1021/c3300415d URL https://doi. org/10. / PMID: 23088335.
Alvaro Sanchez-Gonzaler Jonathan Godwin, Tobias Pfaff, Rex Ying. Jure Leskovec, and Peter W.
Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning (ICML), 2020.
Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning (ICML), 2021.
K. Schütt, P.-J. Kindermans, H. Sauceda, S. Chmicla, A. Tkatchenko, and K.-R. Müller. Schnet: A continuous-fii filter convolutional neural network for modeling quantum interactions, In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Kristof T. Schütt, Oliver T. Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning (ICML), 2021.
Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You, Jiyan He, Shengjie Luo, Chang Liu, Di He, and Tie- Yan Liu. Benchmarking graphormer on large-scale molecular modeling datasets. arxiv preprint arxiv:2003.048// 2022.
Muhammed Shuaibi, Adcesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. Rotation invariant graph neural networks using spin convolutions. arxiv preprint arxiv: 206.0/5..0. 2021.
13

---

Published as a conference paper at ICLR 2024
Anuroop Sriram, Abhishek Das, Brandon M Wood, and C. Lawrence Zitnick. Towards training billion parameter graph neural networks for atomic simulations. In International Conference on Learning Representations (ICLR), 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):192--1995 1958, 2014. URL http://jmlr org http//ppps//vvv/// Lvastaval4a, htmml....
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials, In International Conference on Learning Representations (ICLR), 2022. URL https:
//openreview.rr
wttpiiiwwwweeeeeeeeeeeeeeeeeeee net/ (tt.fforrmm/dd222222255WWf
Nathaniel Thomas, Tess E. Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translationeequivariant neural networks for 3d point clouds. arxiv preprint arXiv:1802.082211 2018.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint vrXiv:2012.12877, 2020.
Raphael J. L. Townshend, Brent Townshend, Stephan Eismann, and Ron O. Dror. Geometric prediction: Moving beyond scalars. arXiv preprint arXiv:2006. /4163, 2020.
Richard Tran*, Janice Lan*, Muhammed Shuaibi Brandon Wood*, Siddharth Goyal*, Abhishek Das, Javier Heras- Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysis. arXiv preprint arXiv:2206.08911 2022.
Oliver T. Unke and Markus Meuwly. PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of Chemical Theory and Computation, 15(6): 3678-3693 may 2019.
Oliver Thorsten Unke, Mihail Bogojeski, Michael Gastegger, Mario Geiger, Tess Smidt, and Klaus Robert Muller. -quiivarana prediction of molecular wavefunctions and electronic densities. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https:///operreview wwtprreiiiiieeeeefrrmmme.. net/forum? id=auGY2UQFhSu.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Information Processing Systems (NeurIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018. URL https: http://wwww.eeeeorrrr//rredee/e
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neural Information Processing Systems 32, pp. 10402-10413, 2018.
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. arxiv preprint 1tte::1622.04642. 2016.
Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical Review Letters, 2018
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https://openreeiiw net/forum? id=DeWooOxFWDa.
14

---

Published as a conference paper at ICLR 2024
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Tch, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denois- ing for molecular property prediction. In International Conference Learning Representations (ICIR), 2023.
A. Zee. Group Theory in a Nutshell for Physicists. Princeton University Press, USA, 2016.
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and Weinan E. Deep potential molecular dynamics: scalable model with the accuracy of quantum mechanics. Phys. Rev. Lett., 120: 143001, Apr 2018. doi: 10.1:10.110000000 10 120. 130.000 URL 1183/PhysRevLett. 120. 143001.
Larry Zitnick, Abhishek Das, Adeesh Kolluru, Janice Lan, Muhammed Shuaibi, Anuroop Sriram, Zachary Ulissi, and Brandon Wood. Spherical channels for modeling atomic interactions. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
15

---

Published as a conference paper at ICLR 2024
APPENDIX
A Additional background
A.1 Group theory
A.2 Equivariance
A.3 eSCN convolution
A.4
S activation
B Related works
B.1 SE(3)/E3)eeuiivarian GNNs
B.2 Invariant GNNs
C Details of architecture
D Details of experiments on OC20
D.I Detailed description of OC20 dataset
D.2 Training details
D.3 Details of running relaxations
D.4 Details of AdsorbMI.
D.5 Additional comparison with Equiformer OC20 IS2RE
E Details of experiments on OC22
E.1 Training details
F Details of experiments QM9
F.1
Additional results of training with Noisy Nodes
F2
Training details
E3 Ablation study on architectural improvements
A ADDITIONAL BACKGROUND
We first provide relevant mathematical background on group theory and equivariance. We note that most of the content is adapted from Equiformer (Liao & Smidt, 2023) and that these works (Zee, 2016; Dresselhaus et al., 2007) have more in-depth and pedagogical discussions. Then, we provide mathematical details of eSCN convolutions.
A.1 GROUP THEORY
Definition of Groups. A group is an algebraic structure that consists of set G and a binary operator o G G. G. Typically denoted as G, groups satisfy the following four axioms:
1. Closure: go h G for all E G.
2. Identity: There exists an identity element E G such that g e e = g for all g E G.
3. Inverse: For each g G, there exists inverse element g- G such that g g
g og =
4. Associativity: g h (g o h) i g (h a i) for all g. h. G.
In this work, we consider 3D Euclidean symmetry, and relevant groups are:
1. The Euclidean group in three dimensions E(3): 3D rotation, translation and inversion.
2. The special Euclidean group in three dimensions SE(3): 3D rotation and translation.
3. The orthogonal group in three dimensions 033): 3D rotation and inversion.
4. The special orthogonal group in three dimensions SO(3): 3D rotation.
Since eSCN (Passaro & Zitnick, 2023) and this work only consider equivariance to 3D rotation and invariance to 3D translation but not inversion, mainly diseuss SE(3) equiivaraae in the main text and in appendix and note that more details (33iiqquuvere can be found in the work of Equiformer (Liao& & Smidt, 2023).
16

---

Published as a conference paper at ICLR 2024
Group Representations. Given vector space X, the way group G acts X is given by the group representation Dx- Dx is parameterized by g E G. with Dx X - X. Group group actions, take the form of matrix multiplications. This definition of group representations satisfies the requirements of groups, including associativity, P(g) ((h) Dig h) for all g, h G. We say that the two group representations D(g) and (g) are equivalent if there exists a change-of-basis N x N matrix P such that p-1 "Dg)) P = D'(g) for all g G. D(g) is reducible if D'(g) is block diagonal for all G, meaning that (g) acts multiple independent subspaces of the vector space. Otherwise, the representation ((g) said to be irreducible. Irreducible representations, or irreps, are class of representations that are convenient for composing different group representations. Specifically, for the case of 5O(3), Wigner-D matrices are irreducible representations, and we can express any group representation of SO(3) as direct sum (concatentation) of Wigner-D matrices (Zee, 2016; Dresselhaus et al., 2007; Geiger et al., 2022):
representations Dx invertible matrices, and group transformations,
((Lo)
D(g) p-1 ( p p-1
(1)
P
(2)
where (g) are Wigner-D matrices of degree Li.
A.2 EQUIVARTANCE
function I mapping between vector spaces and Y is equivariant to a group of transformations for any input X, output y e Y and group element g E G, we have fg)x Dy (g)y, where (g)g) and (g) are transformation matrices or group representations parametrized by g in X and Y. Additionally, is invariant when Dy is an identity matrix for any g e G.
f))
As neural networks comprise many composable operations, equivariant neural networks comprise many equivariant operations maintain the equivariance of imput, intermediate, and output features. Incorporating equivariance as strong prior knowledge can help improve data efficiency and general- ization of neural networks (Batzner al., 2022; Rackers al., 2023; Frey et al., 2022). In this work, we achieve equivariance to 3D rotation by operating on vector spaces of SO(3) imress, incorporate invariance to 3D translation by acting on relative positions, but do not consider inversion.
A.3 ESCN CONVOLUTION
Message passing is used to update equivariant irreps features and is typically implemented as S0(3) convolutions. A traditional SO(3) convolution interacts input irrep features x!!.)) and spherical harmonic projections of relative positions y<(Lz) (Fr.) with an SO(3) tensor product with Clebsch- C.fo..ma) Gordan coefficients (L,,m.))(L,.m)) Since tensor products are compute-intensive, eSCN convolu- tions (Passaro & Zitnick, 2023) are proposed to reduce the complexity of tensor products when they used in SO(3) convolutions. Rotating the input irreps features Im;
(L:) based on the relative position vectors Fts simplifies the tensor products and enables reducing SO(3) convolutions to SO(2) linear operations. Below we provide the mathematical details of SO(3) convolutions built from tensor products and how rotation can reduce their computational complexity.
Tensor products interact type-L; vector (L.)) and type-L, vector f(br)) to produce type-L. vector (Lo)) with Clebsch-Gordar coefficients c.Laaimo) (,,, Clebsch-Gordan coefficients c.Lommo)
(L...).(L... are non-zero only when [1 Lol & Lg < LL + Lal. Each non-trivial com- bination of L; ② Ly L. is called path, and each path is independently equivariant and can be assigned a learnable weight WL,.Ly.Le'
We consider the message Mis sent from source node to target node in an SO(3) convolution. The L,-th th degree of m,s can be expressed as:
(T.o) mis
Σ L.bk
WLi.Lyy (Wlte ohr (f2s)
(3)
17

---

Published as a conference paper at ICLR 2024
(2.) denotes the L--th degree of xs, and ft. where x, is the irreps feature source node 8, _g
[eaal The spherical harmonic projection of relative positions (ft.)) becomes sparse if rotate fts with a rotation matrix Rt to align with the direction of = 0 and m O, which corresponds to the z axis traditionally but the axis in the conventions of eănn (Geiger et al., 2022). Concretely, given Re aligned with the axis, Y (Risfts) * 0 only for m = 0. Without loss of equivariance, we re-scale y) (ResTes) to one. Besides, we denote (Res) = D and (Ree) = D( as Wigner-D matrices of degrees L, and Lo based on rotation matrix Ree. respectively, and we define D(L)) (Lx)) ((L1), Therefore, by rotating x.
(Lc) and y!h,) based on
fee, we can simplify Eq. 3 as follows:
(La) (I(Lo)
Σ Links
(Res) of y(4)) (Reofta))
mis
=
=(Dloo))
Σ Lindes
Whi
+
Σ
me (Dlli)a(he)
(tamm))(sssm))) (yess)(Bestin) my
me
mmi.m
-1 - -(D(Lo))
Σ
+
(Lesmi)
Lightss
(Dlla))
Σ Lighy
+ my
Σ m2
(((()))
Who
m:
(4)
where + denotes concatenation. Additionally, given my = 0, Clebsch-Gordan coefficients m.)
(L,. sparse and non-zero only when m; +m. which further simplifies Eq. 4:
(Lo)
(Dlnn))
Σ
Lhe Lo +
(s((hi)
mlammo) (L4.
+ (#(lw))
(laimmo) mm
mis
(L).
La,yy
mo
m.
(5)
By re-ordering the summations and concatenation in Eq. 5, we have:
(D(Lo)
ΣS
+
(((L))
Σ
((WLL,L W.ILLh (.)o (Lnmo))(Es,0) + (a!Li))
Σ
( m.o
(L.-mo))((a.))
L, me
me
Lj
mm
Lf
Instead of using learnable parameters for Who cSNN proposes to parametrize ums
and
wi (Li,bo)) as below:
-m
Loo) mo
Σ
WL LL moo. (Lr.0))
(hhoo WT WL,, (Ta, (In--mo) Σ
for m >=
Lj
L,
( (Lrito) -m
Σ
Cho mb)
-
mon WLi.ky (L)--m)) for m > 0
Whiky. doo (L.ma))(L,.)))
(L.,t.)) and (ho,to) enables removing the summation over Ly and further simplifies the computation. By combining Eq. and Eq. 7, we have:
The parametrization Um.
(La)
= (D((.))
Σ
Vi (L..bo))
mis
T.
ma
m.
Vis ((bistto)
uim
(oriko))
(g(L))
wmm
to (Li,ko))
((L))
for m. 0
mm
mh
-mo
(8)
Vis (Caitioo)
w w
((L1)
(g(Ls)
for m, > 0
my
Wms
Vis ((lsbbo))
(Lubka)
((L))
for m. 0
me
mo
18

---

Published as a conference paper at ICLR 2024
The formulation of Via
(يم) coincides with performing SO(2) linear operations (Worrall et al., 2016;
Passaro & Zitnick, 2023). Additionally, eSCN convolutions can further simplify the computation by considering only subset of m, components in Eq. i... |mol & Mmax-
In summary, efficient S0(3) convolutions can be achieved by first rotating irreps features 2 (())) based on relative position vectors fis and then performing SO(2) linear operations on rotated features. The key idea is that rotation simplifies the computation as in Eq. 4, 5, 7, and 8. Please refer to their work (Passaro & Zitnick, 2023) for more details. We note that eSCN convolutions consider only simplifying the case of taking tensor products between input irreps features and spherical harmonic projections of relative position vectors. eSCN convolutions do not simplify general cases such as taking tensor products between input irreps features and themselves (Batatia et al., 2022) since the relative position vectors used to rotate irreps features are not clearly defined.
A.4 Sa ACTIVATION
S activation was first proposed in Spherical CNNs (Cohen et al., 2018). Our implementation of 5 activation is the same as that eănn (Geiger et al., 2022). SCN (Zitnick et al., 2022) and eSCN (Passaro & Zitnick, 2023). Basically, we uniformly sample a fixed set of points on unit sphere along the dimensions of longitude, parametrized by a E [0, 2T), and latitude, parametrized by 3 E [0, T). We set the resolutions R of and 3 to be 18 when Liaaz
6, meaning that we will have 324 (= 18 x 18) points. Once the points are sampled, they are kept the same during training and inference, and therefore there is randomness. For each point the unit sphere, compute the spherical harmonics projection of degrees up to Lmaa- We consider an equivariant feature of C channels and each channel contains Vectors of all degrees from 0 to Imaz- When performing 5a activation, for each channel and for each sampled point, we first compute the inner product between the vectors of all degrees contained in one channel of the equivariant feature and the spherical harmonics projections of sampled point. This results in R x R x C' values, where the first two dimensions, R x R. correspond to grid resolutions and the last dimension corresponds to channels. They can be viewed as 2D grid feature maps and treated as scalars, and we can apply any standard or typical activation functions like SiLU or use standard linear layers performing feature aggregation along the channel dimension. After applying these functions, we project back to vectors of all degrees by multiplying those values with their corresponding spherical harmonics projections of sampled points. The process is the same performing Fourier transform, applying some functions and then performing an inverse Fourier transform.
Moreover, since the inner products between one channel of vectors of all degrees and the spherical harmonics projections of sampled points sum over all degrees, the conversion to 2D grid feature maps implicitly considers the information of all degrees. Therefore, 5 activation, which converts cquivariant features into 2D grid feature maps, uses the information of all degrees to determine the non-linearity. In contrast, gate activation only uses vectors of degree 0 to determine the non-linearity of vectors of higher degrees. For tasks such as force predictions, where the information of degrees is critical, S2 activation can be better than gate activation since S2 activation uses all degrees to determine non-linearity.
Although there is sampling on a sphere, the works (Cohen et al., 2018; Passaro Zitnick, 2023) mention that as long as the number of samples, or resolution R, is high enough, the equivariance error can be close to zero. Furthermore, eSCN (Passaro & Zitnick, 2023) empirically computes such errors in Figure 9 in their latest manuscript and shows that the errors of using Limaz 6 and R = 18 are close to 0. 0.,%, which similar to the equivariance errors of tensor products in e3nn (Geiger et al., 2022). We note that the equivariance errors in e3nn are due to numerical precision.
B RELATED WORKS
B. SEG3VEE3) GNNS
Equivariant neural networks (Thomas et al., 2018; Kondor et al., 2018; Weiler et al., 2018: Fuchs et al.,
2020; Miller et al., 2020; Townshend et al., 2020; Batzner et al., 2022; Jing et al., 2021; Schütt et al., 2021; Satorras et al., 2021; Unke et al., 2021; Brandstetter et al., 2022; Thölke & Fabritis,, 2022; Le al., 2022; Musaelian et al., 2022; Batatia et al., 2022; Liao & Smidt, 2023; Passaro & Zitnick, 2023)
19

---

Published as a conference paper at ICLR 2024
equivariant irreps features built from vector spaces of irreducible representations (irreps) to achieve equivariance to 3D rotation (Thomas et al., 2018; Weiler et al., 2018; Kondor et al., 2018). They operate irreps features with equivariant operations like tensor products. Previous works differ in equivariant operations used in their networks and how they combine those operations. TEN (Thomas al., 2018) and NequlP (Batzner al., 2022) use equivariant graph convolution with linear messages built from tensor products, with the latter utilizing extra equivariant gate activation (Weiler et al., 2018). SEGNN (Brandstetter et al., 2022) introduces non-linearity messages passing (Gilmer et al., 2017; Gonealez et al., 2020) with equivariant gate activation, and the non- non-linear messages improve upon linear messages. SE(3)-Transformer (Fuchs et al., 2020) adopts equivariam dot product attention (Vaswani et al., 2017) with linear messages. Equiformer (Liao & Smidt, 2023) improves upon previously mentioned equivariant GNNs by combining MLP attention and non-linear messages. Equiformer additionally introduces equívariant layer normalization and regularizations like dropout (Srivastava et al., 2014) and stochastic depth (Huang et al., 2016). However, the networks mentioned above rely on compute-inte inttessiiiteesive (3)3) tensor products to mix the information of vectors of different degrees during message passing, and therefore they are limited to small values for maximum degrees Lmaz of equivariant representations. SCN (Zitnick al., 2022) proposes rotating irreps features based on relative position vectors and identifies a subset of spherical harmonics coefficients, on which they can apply unconstrained functions. They further propose relaxing the requirement for strict equivariance and apply typical functions to rotated features during message passing, which trades strict equivariance for computational efficiency and enables using higher values of Lmaz- eSCN (Passaro & Zitnick, 2023) further improves upon SCN by replacing typical functions with S0(2) linear layers for rotated features and imposing strict equivariance during message passing. However, except using more efficient operations for higher Lmaas SCN and eSCN mainly adopt the same network design as SEGNN, which is less performant than Equiformer. In this work, we propose EquiformerV2, V2, which includes all the benefits of the above networks by incorporating eSCN convolutions into Equiformer and adopts three additional architectural improvements.
B.2 INVARIANT GNNS
Prior works (Schütt et al., 2017; Xie & Grossman, 2018; Unke & Meuwly, 2019; Gasteiger et al., 2020b;a; Qiao et al., 2020; Liu et al., 2022; Shuaibi et al., 2021; Klicpera et al., 2021; Sriram et al., 2022: Gasteiger et al., 2022) extract invariant information from 3D atomistic graphs and operate on the resulting graphs augmented with invariant features. Their differences lie in leveraging different geometric features such as distances, bond angles (3 atom features) or dihedral angles (4 atom features). SchNet (Schütt et al., 2017) models interaction between atoms with only relative distances. DimeNet series (Gasteiger et al., 2020b;a) use triplet representations of atoms to incorporate bond angles. SphereNet (Liu et al., 2022) and GemNet (Klicpera et al., 2021: Gasteiger et al., 2022) further include dihedral angles by considering quadruplet representations. However, the memory complexity of triplet and quadruplet representations of atoms do not scale well with the number of atoms, and this requires additional modifications like interaction hierarchy used by GemNet-OC (Gasteiger et al., 2022) for large datasets like OC20 (Chanussot* et al., 2021). Additionally, for the task of predicting DFT calculations of energies and forces on the large-scale OC20 dataset, invariant GNNs have been surpassed by equivariant GNNs recently.
C DETAILS OF ARCHITECTURE
In this section, we define architectural hyper- parameters like maximum degrees and numbers of channels in certain layers in EquiformerV2, which are used to specify the detailed architectures in Sec. D.2, Sec. D.5, Sec. F. and Sec. F.2. Besides, we note that eSCN (Passaro & Zitnick, 2023) and this work mainly consider equiiaariiv
We denote embedding dimensions as dembed, which defines the dimensions of most irreps features. Specifically, the output irreps features of all modules except the output head in Figure la have dimension dembed- For separable S² activation as illustrated in Figure 2c, we denote the resolution of point samples on a sphere as R, which can depend on maximum degree Lmazs and denote the unconstrained functions after projecting to point samples as F.
For equivariant graph attention in Figure Ib, the input irreps features 2; and øj have dimension
dembed- The dimension of the irreps feature (LL) is denoted as hidden- Equivariant graph
20

---

Published as a conference paper at ICLR 2024
attention can have h parallel attention functions. For each attention function, we denote the dimension of the scalar feature f as dattn_alpha and denote the dimension of the value vector, which is in the form of irreps features, as datto_ value- For the separable 5 activation used in equivariant graph attention, the resolution of point samples is R. and we use single SiLU activation for F. We share the layer normalization in attention re-normalization across all / attention functions but have different linear layers after that. The last linear layer projects the dimension back dembed. The two intermediate 50(2) linear layers operate with maximum degree Lmez and maximum order Maaz.
For feed forward networks (FFNs) in Figure 1d, we denote the dimension of the output irreps features of the first linear layer as fn For the separable Sa activation used in FFNs, the resolution of point samples is R. and F consists of a two-layer MIP, with each linear layer followed by SiLU,, and a final linear layer. The linear layers have the same number of channels as dff
For radial functions, we denote the dimension of hidden scalar features as dedge- For experiments on OC20, same as eSCN (Passaro & Zitnick, 2023), we use Gaussian radial basis to represent relative distances and additionally embed the atomic numbers at source nodes and target nodes with two scalar features of dimension dedge. The radial basis and the two embeddings of atomic numbers fed to the radial function to generate edge distance embeddings.
The maximum degree of irreps features is denoted as Lmaz. All irreps features have degrees from 0 to Timaz and have C' channels for each degree. We denote the dimension as (Imarr, C). For example, irreps feature Tirreps of dimension (6, 128) has maximum degree 6 and 128 channels for each degree. The dimension of scalar feature Escolor can be expressed as (0, Cacatar)
Following Equiformer (Liao & Smidt, 2023), we apply dropout (Srivastava et al., 2014) to attention weights and stochastic depth (Huang et al., 2016) to outputs of equivariant graph attention and feed forward networks. However, we do not apply dropout or stochastic depth the output head.
D DETAILS OF EXPERIMENTS ON OC20
D.1 DETAILED DESCRIPTION OF OC20 DATASET
The large and diverse OC20 dataset (Chamussot* et al., 2021) (Creative Commons Attribution 4.0 License) consists of 1.2M DFT relaxations for training and evaluation, computed with the revised Berdew-Burke-Ernzerth
(RPBE) functional (Hammer et al., 1999). Each structure in OC20 has an adsorbate molecule placed on catalyst surface, and the core task is Structure Energy Forces (S2EF), which is to predict the energy of the structure and per-atom forces. Models trained for the S2EF task are evaluated on energy and force mean absolute error (MAE). These models can in turn be used for performing structure relaxations by using the model's 's force predictions to iteratively update the atomic positions until relaxed structure corresponding to a local energy minimum is found. These relaxed structure and energy predictions are evaluated on the Initial Structure to Relaxed Structure (IS2RS) and Initial Structure to Relaxed Energy (IS2RE) tasks. The "All" split of OC20 contains 134M training structures spanning 56 elements, the "MD" split consists of 38M structures, and the "2M" split has 2M structures. For validation and test splits, there are four sub-splits containing in nn-distribution adsorbates and catalysts (ID), out-of-distribuue adsorbates (OOD Ads), out-of-distribution catalysts (OOD Cat), and distrr-distiibution adsorbates and catalysts (OOD Both).
D.2 TRAINING DETAILS
Hyper-Parameters. We summarize the ryper-parametery for the base model setting on OC20
S2EF-2M dataset and the main results on OC20 S2EF-AII and S2EF-AII+MD datasets in Table 7. For the ablation studies on OC20 22EF-2M dataset, when trained for 20 or 30 epochs as in Table lb, we increase the learning rate from 2 x 10 to 4 x 10 When using Lmaz 8 as in Table Ic, we increase the resolution of point samples R from 18 to 20. We vary Lmaz and the widths for speed-accurace trade-offt in Figure 4. Specifically,
first decrease Lmaz from 6 to 4. Then, multiply l and the number of channels of (dembeds datta hidden dn by 0.75 and 0.5. We train all models for 30 epochs. The same strategy to scale down eSCN models is adopted for fair comparisons.
21

---

Published as a conference paper at ICLR 2024
Base model setting on S2EF-2M 2M
Equiformer22 (31M) on S2EF-AIl-MI
Equiformet (153M) on
CDEF-ADS2EF--AN AIDD-D
Optimizer
AdamW
AdamW
Adam'W
Learning rate scheduling
Cosine learning nie with
Casine learning rate with
Cosine leamnnng nite with
lincar warmup
lincar warmup
lincar wammap
Warmup epochs
Maximum leamning rate
2 10
x 10
4 10-d
Batch size
64
012
256 for S2EF-AII,
512 S2EF-AIl-MD
Number of epochs
Weight decay
1 10
10
1 x 10
Dropout rate
Stochastic depth
0.05
0.1
0.1
Energy onefficient Ax
2
2 for S2EF-AII,
2.4 for MDD
Foree coefficient Ay
100
100
Gradient clipping norm threshold
100
100
100
Model EMA decny
0.909
0.980
0.999
Curoff miiius (A)
12
12
12
Maximum number of neighbers
20
20
Number radial bases
600
600
Dimeusion hidden scalar features in radial functions didge
128)
(0,)
(0, 126)
Maximum degree Louax
6
6
Maximum order
3
Number of Transfrmmrr blocks
12
Embbediing dimension dannded
(6. 126)
(4. 1228)
(6, 128)
fi dimension datto hudder
(6, 64)
(4,64) 64)
[6, 64)
Number of attention heads
8
10 dimension aishs
(0, 64)
(0,64)
[0, G4)
Value dimension dator caloe
(6. 16)
(4.16)
[6. 16)
Hidden dimsension food foeward networks dyyn Resolution point samples I?
[6. 128)
(1.128)
[6. 128)
18
Table Hyper- parameters the base model setting on OC20 S2EF-2M dataset and the main results on OC20 S2FF-AII and S2FF-All+MD datasets.
Attention commlizatios
Number of Trassfomerr bocks
Training time (GPU houns)
Inference speed (Sumples/ sec.)
Namber of parantetens
Trainin set
Activation
Normalizstion
lose
Moss
91.00M
1476
12.54
10M
S2LF 2M
1412
18.22
81,16M
19.56
44.8M
Stp.
2709
7,86
234
12
1625
11.92
M5
Scp.
SLN
12
2006
7.58
14M
Sep
SLN
12
3052
7.13
100, GUM
Scp.
SLN
6.08
AIH-D (Ag
Sep.
32534
AlHMD (Ag
7
Sep. 57
37002
150. 0..
Table 8: Training time, inference speed and numbers of parameters of different models trained on OC20 S2EF-2M, S2EF-All and S2EF-AII+MD datascts. All numbers are measured on V100 GPUs with 32GB.
Training Time, Inference Speed and Numbers of Parameters. Table 8 summarizes the training time, inference speed and numbers of parameters of models in Tables la (Index 1, 3, 4, 5), Ic, Id and 2. V100 GPUs with 32GB are used to train all models. We use 16 GPUs to train each individual model on S2EF-2M dataset, 64 GPUs for S2EF-AII, 64 GPUs for Equiformer V2 (31M) on S2EF. All+MD, and 128 GPUs for EquiformerV2 (153M) on S2EF-AILMD..
D.3 DETAILS OF RUNNING RELAXATIONS
structural relaxation is a local optimization where atom positions are iteratively updated based on forces to minimize the energy of the structure. We perform ML relaxations using the LBFGS optimizer (quasi-Newton) implemented in the Open Catalyst Github repository (Chanussot* et al., 2021). The structural relaxations for OC20 IS2RE and IS2RS tasks are allowed to for 200 steps or until the maximum predicted force per atom Fmax & 0.02 eV/,, and the relaxations for AdsorbML are allowed to run for 300 steps or until Fmaz & 0.02 eVIÀ. These settings are chosen to be consistent with prior works. We run relaxations on V100 GPUs with 32GB. The computational cost of running relaxations with EquiformerV2 (153M) for OC20 IS2RE and IS2RS tasks is 1011 GPU-hours, and that of running ML relaxations for AdsorbML. is 1075 GPU-hours. The time for
22

---

Published as a conference paper at ICLR 2024
100
(averaged over k ffrmm 1 to 5)
AdsorbML success rate
90
70
GemNee-OCCMD
60
Larg (218M)
SCN MD Large (271M)
Equ Equme22 (e 153M)
foumoree 31M)
25
50
75
100 125
GPU-seconds per relaxation
Figure 5: Speed-accuracy trade-offs of different models when used in the AdsorbML. algorithm.
running relaxations with EquiformerV2 (31M) is 240 GPU-hours for OC20 IS2RE and IS2RS and 298 GPU-hours for AdsorbML.
D.4 DETAILS OF ADSORBML
We run the AdsorbML algorithm on the OC20-Dense dataset in accordance with the procedure laid out in the paper (Lan et al., 2022), which is summarized here:
1. Run ML relaxations all initial structures in the OC20-Dense dataset. There around 1000 different adsorbate-surface combinations with about 90 adsorbate placements per combination, and therefore have roughly 90k structures in total.
2. Remove invalid MI. relaxed structures based physical constraints and rank the other MIL relaxed structures in order of lowest to highest MI. predicted energy.
3. Take the top & MI. relaxed structures with the lowest MI. predicted energies for each adsorbate-surface combination and DFT single-poin calculations. The single-point calculations are performed on the ML relaxed structures to improve the energy predictions without running a full DFT relaxation and are run with VASP using the same setting as the original AdsorbML experiments. As shown in Table 3, we vary ic from 1 to 5.
4. Compute success and speedup metrics based on our lowest DFT single-point energy per adsorbate-surface combination and the DFT labels provided in the OC20- OC20-Dense dataset.
To better understand the speed-accuracy trade-offs of different models, we compare the AdsorbMI. success rate averaged over & from to 5 and average GPU-seconds of running structure relaxation in Figure 5. We visualize some examples relaxed structures from eSCN (Passaro & Zitnick, 2023), EquiformerV2 and DFT in Figure 6.
D.5 ADDITIONAL COMPARISON WITH EQUIFORMER ON OC20 IS2RE
Training Details. We follow the same setting Equiformer (Liao & Smidt, 2023) and train two EquiformerV2 models on OC20 IS2RE dataset without and with IS2RS auxiliary task. We use the same radial basis function Equiformer. When IS2RS auxiliary task is adopted, use a linearly decayed weight for loss associated with IS2RS, which starts at 15 and decays to and adopt Noisy Nodes data augmentation (Godwin et al., 2022). The hyper-parameten We train EquiformerV? on OC20 IS2RE with 16 V100 GPUs with 32GB and train on OC20 IS2RE with IS2RS auxiliary task and Noisy Nodes data augmentation with 32 V100 GPUs. The training costs are 574 and 2075 GPU-hours, and the numbers of parameters are 36.03M and 95.24M.
summarized in Table 9.
Results. The comparison is shown in Table 10. Without IS2RS auxiliary task, Equiformer V2 overfits the training set due higher degrees and achieves worse results than Equiformer. However, with IS2RS auxiliary task and Noisy Nodes data augmentation, EquiformerV2 achieves better energy MAE. The different rankings of models under different settings also found in Noisy Nodes (Godwin
23

---

Published as a conference paper at ICLR 2024
Equiformer22 on OC20 IS2RE
EquiformerV2 on OC20 IS2RE
with 1S2RS auxiliary task and Noisy Nodes
Optimizer
Learning rate schedaaiing
Cosine learning nite with
Cosine leamning rite with lincar wartuup
lincar wanmup
Warmup epochs
Maximum learning rale
4.x 10
x 10-4
Batch size
32
64
epochs
20
Weight decay
x 10
x 10-3
Dropout rate
01:2
0.2
Stochastic depth
Gradient elipping norm thresisold
100
100
Model EMA decny
0.550
0.959
Cutoll radios
Maximum number of neighbors
50
50
Number of radial basis
128
128
Dimension hidden scalar
radial functions drdge 64)
(0, 64)
Moximum degree Fauax
Maximum onder Maor
Number Transformer blocks
15
Embedding dimension daeatedd
(6, 128)
[0, 128)
( dimension buttn hibdes
(6, 64)
[6,64] 64)
Number of attention heads i
fi dimcasioo sithe
(0.
(0,61)
dimension deteo value
(6.
18)
Hidden dimension in feed forward netwerks dyfe
128)
(28)
Resolution of point samples I
18
18
Table 9: Hyper-paramettess for OC20 IS2RE dataset.
IS2RE
IS2RE IS2RS auxiliary task Noisy
Energy MAE (me)))
Energy MAE (meV)]
Model
ID
OOD Ads
OOD Cat
OOD Both
Average
ID
00D Ads
ODD Cat
0OD Both
Average
Equiformer (Lino Smidt, 2023) EquiformerV2
508.8
627.1
416.5
516.1
704.1
521.5
636.5
595.3
400.4
459.0
406.2
401.8
416.9
Table 10: OC20 IS2RE results on the validation split. With IS2RS auxiliary task and Noisy Nodes (Godwin al., 2022) data augmentation, EquiformerV2 achieves better energy MAE.
ct al., 2022), where they mention a node-level auxiliary task can prevent overfitting and enable more expressive models to perform better.
E DETAILS OF EXPERIMENTS ON OC22
E. TRAINING DETAILS
The syper-parameter for OC22 dataset is summarized in Table 11. We use 32 V100 GPUs with 32GB and train two EquiformerV2 models with different energy coefficients X and force coefficeints AP. The number of parameters is 121.53M, and the training cost of each model 4552 GPU-hours. The time for running relaxations for OC22 IS2RE is 38 GPU-hours.
F DETAILS OF EXPERIMENTS ON QM9
F.1 ADDITIONAL RESULTS OF TRAINING WITH NOISY NODES
Similar Sec. D.5, we train Equiform V2 on QM9 with Noisy Nodes (Godwin al., 2022) to show that the performance gain brought by using higher degrees can be larger when trained with a node-level auxiliary task and data augmentation. Table 12, we summarize the results and compare with previous works using Noisy Nodes and pre-training via denoising (Zaidi et al., 2023). When trained with Noisy Nodes, EquiformerV2 performs better than Equiformer on more tasks. Specifically, without Noisy Nodes, EquiformerV2 V2 is better than Equiformer on 9 out of the 12 tasks, similar on 1 task, and worse on the other 2 tasks. With Noisy Nodes, Equiformer V2 achieves better MAE on 10 tasks, similar on 1 task, and worse 1 task. Additionally, Noisy Nodes is overall better than GNS-TAT with both pre-training and Noisy Nodes (Zaidi et al., 2023) out of the 12 tasks even though EquiformerV2 is not pre-trained on PCQM4My2 dataset,
note that EquiformerV2 with
24

---

Published as a conference paper at ICLR 2024
Hyper-panee paramiceess
Valuc or description
Optimizer
AdamW
Leaming rate scheduling
Cosine learning rate with linear warmup
Warmup epochs
D.1
Maximum learning
10
Batch
Numher of epochs
Weight decay
10
Dropout
Stochastic depth
D.1
Energy coefficiont
1,4
Force cocfficient Ay:
Gradient clipping norm threshold
50
Model EMA decay
0.3019
Cuteff radius
12
Maximum number of neighbors
20
Number of radial bases
600
Dimension of hidden scalar
rudial functions dudge
(08)
Maximum degree lonsz
Maximum order Misae
Number Transforer blocks
18
Embeeking dimension dunated
128)
(15) dimension datto
(6,64) 64)
Numher of attention heads h
dimension daann
(0, 04)
dimension duetn_sstue
Hidden dimension in feed forward networkss drys.
128)
Resolution of point samples it
18
Table 11: Hyper-paramettes for OC22 dataset.
Cusso mcV
C₁
G
If
n
Uniss
%
cabinol
meV
meV
%6
mcV
Equifommer (Liao & Smidt, 2023)
346
30
15
14
.011
023
7.63
6.63
.251
6.74
6.39
1,26
Equilommer (Lin & Smill, 2023) + NN
040
26.4
13.7
13.0
001
.020
5.49
4.61
.235
4.81
4.61
1.18
GNS-TAT
2023)!
GNS-TAY + NN peetmiiing (Zaidi et al., 2003y4
000
22.0
14.7
016
.020
6.90
5.79
0.44
5.76
5.76
Equifomee/22 EquifomeeV2 NN
A50
13
.110
023
7.57
6.22
.186
6.49
6.17
1.47
039
24.2
12.2
11.4
.009
.020
5.34
4.24
1882
4.28
4.34
1.21
Table 12: Mean absolute error results QM9 test when trained with Noisy Nodes (Godwin et al., 2022).
t denotes results are produced by this work. $ denotes using different data partitions. "NN" denotes Noisy Nodes, and "pretraining" denotes pretraining on PCQM4Mv2 dataset (Nakata & Shimazaki, 2017). The performance gain from Equiformer to EquiformerV2 becomes larger when trained with Noisy Nodes.
which is than 30x larger than QM9. This shows that more expressive model match the performance with significantly less data.
F.2 TRAINING DETAILS
We follow the data partition Equiformer. For the tasks of μ4, a, SHOMO. ELUMO. Ae, and Co, we use batch size 64, the number of epochs 300, learning rate 5 10 14-4, Gaussian radial basis functions with the number bases = 128, the number of Transformer blocks = 6, weight decay 5 x 10-3 and dropout rate 0.2 and use mixed precision for training. For the task of R2, use batch size = 48, the number of epochs 300, learning rate 1.5 x 10-1 Gaussian radial basis functions with the number of bases 128. the number of Transformer blocks 5, weight decay = =5 x 10-3, and dropout rate = 0.1 and use single precision for training. For the task of ZPVE, we use batch size 48, the number of epochs 300, learning rate 1.5 10-4 Gaussian radial basis functions with the number of bases = 128, the number of Transformer blocks 5, weight decay 10- 3 and dropout rate 0.2 and use single precision for training. For the task of G. H, U. and Uo. we use batch size 48, the number of epochs 300, learning rate 1.5 10-1 Gaussian radial basis functions with the number bases 128, the number of Transformer blocks 5, weight decay- 0.0, and dropout rate 0.0 and single precision for training. Other hyper- parem-errmeterss the same across all the tasks, and we summarize them in Table 13. We use a single A6000 GPU and train different models for different tasks. The training costs 72 GPU-hours for mixed precision training and 137 GPU-hours for single precision training. The number of parameters are 11.20M for 6 blocks and 9.35M for 5 blocks.
As for training with Noisy Nodes as mentioned in Sec. F.I. we add noise to atomic coordinates and incorporate node-level auxiliary task of denoising atomic coordinates. We thus introduce
25

---

Published as a conference paper at ICLR 2024
Hyper-panne paramicesss
Valuc or description
Optimizer
AdamW
Leaminng rate scheduling
Cosine learning rate with linear warmup
Warmup epochs
Maximum learning
1.5 10 5 10-
Batch
48, 64
Number of epochs
300
Weight decay
0.0, 10-
Drapout
0.0.0.1
Stochastic depch
D.0, 0.05
Cuteff rudius (A)
Maximum number neighibors
500
Number of radial bases
128
Dimension of hicken scalor fentures in mdiia functions fuige
(0, (4)
Maximum degree Loose
Maximum order Mosz
Number of Transforterr blocks
5.0
Embedding dimension deauted
(4,96)
(10)) dimcusion datto_Aidden
(1, 18)
Number of attention heads h
(10) dimension duat olyhe
(0, 64)
Value dimension dueets ssise
(1,24)
Hidden dimension in forward networkss dzjn
(4.96)
Resolution of point samples R
18
Noise standard deviation Fanciv
0.02
Denoising coefficient Akerie
Denoising probubility Nicesie
0.5
Corrupt ratio Tocasac
0.125, 0.25
Table 13: Hyper-parameters for QM9 dataset.
four additional hyper- paeer-parametess, which noise standard deviation dennisee denoising coefficient Adenoise. denoising probability Pdenoise and corrupt ratio deemoise- The noise standard deviation o denoise denotes the standard deviation of Gaussian noise added to each xyz component of atomic coordinates. The denoising coefficient Adenoise controls the relative importance of the auxiliary task compared to the original task. The denoising probability Pdenoise denotes the probability of adding noise to atomic coordinates and optimizing for both the auxiliary task and the original task. Using Pienoise < 1 enables taking original atomistic structures without any noise as inputs and optimizing for only the original task for some training iterations. The corrupt ratio Idenoise denotes the ratio of the number of atoms, which we add noise to and denoise, to the total number of atoms. Using deenoiie allows only adding noise to and denoising subset of atoms within structure. For the task of R2, we use denoise 0.02, Adenoise 0.1, Pdenoise = 0.5 and deeeoisee = 0.125. For other tasks, use denniise 0.02, Adenoise 0.1, Pdenoise 0.5 and Tdenoise 0.25. We share the above ppper-parameters for training EquiformerV2 and Equiformer. and we add one additional block of equivariant graph attention for the auxiliary task. We slightly tune other hyper-parameters when trained with Noisy Nodes. For Equiformer, we additionally use stochastic depth = 0.05 for the tasks of a, Ae, HHOMO ELUMO, and Cy. As for EquiformerV2, we additionally use stochastic depth 0.05 for the tasks of μu Ae, EHOMO. SLUMO. and Cy. We increase the number of blocks from 5 to 6 and increase the batch size from 48 64 for the tasks of G, H1 U, and Ug. We increase the learning rate from 1.5 x 10-4 to 5 x 10-4 and increase the number of blocks from 5 to 6 for the task of R².
F.3 ABLATION STUDY ON ARCHITECTURAL IMPROVEMENTS
We conduct ablation studies on the proposed architectural improvements using the task of Ae on QM9 and compare with Equiformer baseline (Liao & Smidt, 2023). The reults are summarized in Table 14. The comparison between Index 0 and Index 1 shows that directly increasing Lmaa from 2 and using eSCN convolutions degrade the performance. This is due to overfitting since the QM9 dataset is smaller, and each structure in QM9 has fewer atoms, less diverse atom types and much less angular variations than OC20 and OC22. Comparing Index and Index 2, attention re-nommalization clearly improves the MAE result. Although using S² activation is stable here (Index 3) unlike OC20, it results in higher error than using gate activation (Index 2) and the Equiformer baseline (Index 0). When using the proposed separable S activation (Index 4), we achieve lower error than using gate activation (Index 2). We can further reduce the error by using the proposed separable layer normalization (Index 5). Comparing Index 0 and Index 5, we note that the proposed architectural improvements are necessary to achieve better results than the baseline when using higher degrees on QM9. Overall, these ablation results follow the same trends OC20.
26

---

Published as a conference paper at ICLR 2024
Alication re normalizatiion
Index
Activation
Normalization
Lear
A= MAE (me
Equiformer
1
Gate
LN
30.46
2
Gate
LN
29.51
30.23
Sep. 8
I.N
29.31
Sep. 8
Table 14: Ablation studies on the proposed architoctural improvements using the task Ae of the QM9 dataset.
27

---

Published as a conference paper at ICLR 2024
Initial Structure
DFT
eSCN
EquiformerV2 4)
Figure Qualitative examples of initial configuration of an adsorbate on a catalyst surface (column 1), and corresponding relaxed configurations obtained from eSCN (Passaro & Zitnick, 2023) (column 2), EquiformerV2 (column 3). and DFT (column 4). All examples selected from the OC20-Dense dataset (Lan al., 2022). We show top-down views of each structure, with dashed lines showing the boundary of the unit cell repeating in the x and y directions.
28


Reviews:
[{"review_id": "u52mzorhuD", "reviewer": "Reviewer_Joeu", "summary": "The authors propose EquiformerV2, which incorporates eSCN convolutions to efficiently include higher-degree tensors and introduces three architectural improvements: attention re-normalization, separable $S^2$ activation, and separable layer normalization. These enhancements allow EquiformerV2 to outperform state-of-the-art methods across OC20 and OC22.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- One of the significant contributions of this paper is the comprehensive experiments across OC20, OC22, and QM9. And EquiformerV2 achieves the state-of-the-art result over OC20 and OC22. The authors deserve commendation for their efforts in this aspect.\n- The use of attention re-normalization, separable $S^2$ activation, and separable layer normalization is novel.", "weaknesses": "Major:\n- Although the authors did a fantastic job on the experiments, EquiformerV2 is an incremental improvement over existing methods of both eSCN and Equiformer w.r.t. theory. And the novelty lies in those three specific techniques and enhancements. To see if these techniques are generalizable, I would like to see the ablation study of attention re-normalization, separable $S^2$ activation, and separable layer normalization, respectively, on the QM9 dataset like what the authors did in Table (a) for OC20.\n\nMinors:\n- Equation (2) in Appendix A.1: Use $\\ddots$ instead of $\\dots$\n- Equation (4) in Appendix A.3: Commonly, the left side of an equation is used for assigning new notation. I recommend write $D^{(L)} = D^{(L)}(R_{ts})$ and $\\tilde{x}_s^{(L)} = D^{(L)} x_s^{(L)}$ for a degree $L$ before Equation (4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6lghXkW0AA", "reviewer": "Reviewer_3vjk", "summary": "EquiformerV2 is proposed to improve the efficiency of Equiformer on higher-degree tensors. \nTo achieve this, original tensor product (TP) with spherical harmonics is changed to eSCN convolution which can reduce the complexity from $O(L^6)$ to $O(L^3)$.\nBesides, three archtecture module is replaced to improve the performance in attention normalization, nonlinear activation and layer normalization.", "questions": "Minor issue:\nThere is a double citation. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016a.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The empirical results of EquiformerV2 is great. It achieves SOTA performance on OC20 and OC22, where higher-degree tensor shows great improvement. Meanwhile, the efficiency is denoted in Figure 4 showing that EquiformerV2 can has better efficient ability than eSCN.", "weaknesses": "The modification of proposed architecture is similar to the previous Equiformer. Although the ablation studies show the improvement of proposed modules, the results on QM9 is similar compared to Equiformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zhIGnYcl9h", "reviewer": "Reviewer_c4fs", "summary": "In this paper, the authors proposed EquiformerV2, which is a newly developed equivariant network for 3D molecular modeling built on the Equiformer. From the experimental evaluation, the EquiformerV2 model achieved strong performance on the large-scale OC20/OC22 benchmark, QM9 dataset and also the new AdsorbML dataset. Such performance improvement upon Equiformer is achieved via several architectural modifications: (1) levering eSCN's efficient SO(2) convolution implementation for SO(3) convolutions (tensor product operations); (2) Attention Re-normalization for stabilizing training; (3) Separable S2 activation for mixing representations with different degrees; (4) separate Layer Normalization.", "questions": "See the comments in the Weaknesses section", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Regarding the problem studied in this paper**. By leveraging the key techniques from eSCN, the EquiformerV2 also achieves learning irreducible representations with larger maximum degrees, which has been verified again to be useful for large-scale DFT benchmarks.\n\n2. **Regarding the empirical performance**. In the OC20 benchmark, EquiformerV2 sets a new standard by delivering state-of-the-art performance in the Structure-to-Energy-Force task. The model, further trained on this task, effectively serves as a force-field evaluator, demonstrating impressive performance in both IS2RS and IS2RE tasks. EquiformerV2 surpasses the performance of the compared baselines across all tasks, with a notable edge in force prediction. Furthermore, it significantly enhances the success rate on the AdsorbML dataset.", "weaknesses": "The novelty of the proposed architectural modifications is limited. Both the efficient SO(2) convolution and S^2 activation are from eSCN, while the attention re-normalization and layer normalization are more like engineering tricks. Among these differences from Equiformer, the eSCN SO(2) convolution plays an essential role in enabling the use of irreducible representations of higher degrees, and the S^2 activation also replaces all non-linear activations. In fact, these design strategies should be mainly credited to the eSCN work.\n\n*******************************************Post Rebuttal *********************************************\n\nThank the authors for the response. I choose to keep my positive evaluation and hope the authors carefully include the newly added discussion and results in the final version of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)