Spherical Channels for Modeling Atomic Interactions
Authors: C. Lawrence Zitnick
Abhishek Das
Adeesh Kolluru
Janice Lan
Muhammed Shuaibi
Anuroop Sriram
Zachary Ward Ulissi
Brandon M Wood
Venue: NeurIPS 2022 Accept (2022)
Keywords: graph neural networks
equivariance
invariance
materials science
chemistry

Abstract: Modeling the energy and forces of atomic systems is a fundamental problem in computational chemistry with the potential to help address many of the world’s most pressing problems, including those related to energy scarcity and climate change. These calculations are traditionally performed using Density Functional Theory, which is computationally very expensive. Machine learning has the potential to dramatically improve the efficiency of these calculations from days or hours to seconds.

We propose the Spherical Channel Network (SCN) to model atomic energies and forces. The SCN is a graph neural network where nodes represent atoms and edges their neighboring atoms. The atom embeddings are a set of spherical functions, called spherical channels, represented using spherical harmonics. We demonstrate, that by rotating the embeddings based on the 3D edge orientation, more information may be utilized while maintaining the rotational equivariance of the messages. While equivariance is a desirable property, we find that by relaxing this constraint in both message passing and aggregation, improved accuracy may be achieved. We demonstrate state-of-the-art results on the large-scale Open Catalyst 2020 dataset in both energy and force prediction for numerous tasks and metrics.

Spherical Channels for Modeling Atomic Interactions
C. Lawrence Zitnick', Abhishek Das', Adeesh Kolluru², Janice Lan', Muhammed Shuaibi", Anuroop Sriram', Zachary Ulissi?, Brandon Wood'
Colluruu,Janice
Fundamental AI Research Meta AI
Carnegie Mellon University
Abstract
Modeling the energy and forces of atomic systems is fundamental problem in computational chemistry with the potential to help address many of the world's 's most pressing problems, including those related to energy scarcity and climate change. These calculations are traditionally performed using Density Functional Theory, which is computationally very expensive. Machine learning has the potential dramatically improve the efficiency of these calculations from days or hours to seconds.
We propose the Spherical Channel Network (SCN) to model atomic energies and forces. The SCN is graph neural network where nodes represent atoms and edges their neighboring atoms. The atom embeddings are a set of spherical functions, called spherical channels, represented using spherical harmonics. We demonstrate, that by rotating the embeddings based on the 3D edge orientation, more information may be utilized while maintaining the rotational equivariance of the messages. While equivariance a desirable property, we find that by relaxing this constraint in both message passing and aggregation, improved accuracy may be achieved. We demonstrate state-of-the-ar results on the large-scale Open Catalyst 2020 dataset in both energy and force prediction for numerous tasks and metrics.
Introduction
Modeling the properties of atomic systems is foundational challenge in computational chemistry and critical to advancing technologies across numerous application domains. Notable applications include drug discovery 332 39] and the design of new catalysts for renewable energy storage to 333 For catalyst discovery, new materials are currently evaluated using Density Functional Theory (DFT) that estimate atomic energies and forces but is computationally very expensive; taking hours or days to evaluate single material. Machine Learning (ML) has the potential approximate DFT and dramatically speed up these calculations, allowing for high throughput screening of new materials to help address some of the world's 's most pressing challenges.
help addressing climate change [48
Our goal is to approximate DFT calculations using ML. An ML model takes as input a set of atom positions and their atomic numbers. As outputs, model calculates the structure's 's energy (or other properties) and the per-atom forces, i.e., the forces exerted on each atom by the other atoms. A common approach to this problem is to Graph Neural Networks (GNNs) 19 47
where each
node represents an atom and the set of nearby atoms as edges (37 17
35
38
13
A key challenge in network design is balancing the use of model constraints. When modeling atomic systems, a common constraint is SO(3) rotation equivariance 43 system is rotated, the energies should remain constant and the atomic forces should similarly rotate. While this provides strong priors on the model to help in generalization, especially for smaller datasets 7 32] it can result in limiting the expressiveness of the network due to restrictions
3
42
34
36), i.e., if the atomic
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Sphetial Projecion
diPrgpccon
ddegree
R (2
61,
order
(a)
Channel
(b)
Figure 1: (a) Illustration of spherical harmonics up to 4. Note the m = bases are symmetric about the 7-axis (center green). (b) Visualization of 3 spherical channels with / = 8 when varying the number of orders: all, m E [- m E [-1, m O. Note how the resolution decreases with fewer m until values are constant for given with m O. Spherical projections are shown on the left (only half the channel visible) and a 2D grid projection (polar and azimuthal) of the same channels is shown on the right.
non-linear transformations for equivariant models 13
34 36 or limiting interactions to pairs 42
15] for invariant models. Alternatively, non non-equivariant model 23 can provide more freedom to the model, but lead to the model needing to learn approximate equivariance through methods such as data augmentation [40]. To draw an analogy with image detection, the use of a CNN [27] provides translation equivariance and removes the need for the network learn how to detect the same object different locations. However, most CNNs are not equivariant to scale or rotation [4] 43), but are still effective in learning approximate equivariance through rotation and scale diversity in the training data. What is the analogous balance of constraints for modeling atomic systems?
35 38 45), triplets [3 quadruplets of atoms 228
In this paper, propose GNN 19 47 that balances the use of model constraints aid in generalization while providing the network with the flexibility to learn accurate representations. We introduce the Spherical Channel Network (SCN) that explicitly models relative orientations of all neighboring atoms; information that is critical to accurately predicting atomic properties. Each node's embedding is set of functions defined on the surface of sphere (S² RR.. The functions are represented using spherical harmonics, similar to approaches that build strictly equivariant I0 During message passing, angular information between atoms is conveyed by We identify an expanded set of spherical harmonic coefficients that are invariant rotation, which can provide rich information while maintaining message's rotation equivariance. In addition, we demonstrate that if the equivariance constraint relaxed, improved performance can achieved by using additional coefficients. We further improve the expressivity of the network by performing a pointwise non-linear function, which only approximately equivariant, on the embeddings during message aggregation.
models 442
rotating or steering S the embeddings based on each edge's 's orientation 41 28
We demonstrate our Spherical Channel Network on the large-scale Open Catalyst 2020 (OC20) dataset (6) which contains atomic structures useful for numerous applications important to addressing climate change. State-of-the-- results are achieved for atomic force and initial structure to relaxed energy prediction with improvements of 8% 11%. In addition, we demonstrate our model is more sample efficient as compared to other state-of-the models.
2
Approach
Given an atomic structure with n atoms, our goal is predict the structure's energy E and the per-atom forces f: for each atom i E n. These values are estimated using a Graph Neural Network (GNN) 19 where each node represents an atom and edges represent nearby atoms. As input, the network is given the distance dij between atoms and j. and each atom's 's atomic number a. The neighbors N, for an atom are determined using fixed distance threshold, or by picking a fixed number of closest atoms.

---

GemNet-dT
F
Forca MAE (moV AA]
50
GemNettGC
SCN
layers SCN layens, 2-bands
SCN 2-4,
layers, 4-lap, 2-bardde
ld_
m2
D.
D2
10
20
30
40
Training Epochs
(b)
Figure 2: (a) Block diagram of message passing function F. in Equation for source atom to target atom t. The atomic numbers and at, distance between the atoms |dar/. and embeddings æ and 2t are given as imput. (b) Training curves for SCN and GemNet models for force MAEs per epoch evaluated on a 30k subset of the validation ID dataset. Note how the SCN model is significantly more sample efficient during training.
2.1 Node Embeddings
The angular or relative orientation information between atoms critical to accurately modeling atomic properties such energies and forces [13 [4]. Inspired by this, our node embedding models the angular information from all neighboring atoms using spherical functions. Each node i's embedding is a set of C functions argument a 3D unit vector indicating the orientation. That is, (te(F) ) is the value of channel e for node for some orientation f E R Since the spherical channels contain orientation information over the entire sphere, the network may reason about geometric information for all neighboring atoms and not just atom pairs, triplets, etc. Each spherical channel may be represented using several different approaches, such as discrete 2D grids sampled over the sphere, or spherical harmonics. As describe later, rotation equivariant).
channels represented on sphere (S² + IR,, whose
spherical harmonies due to their property of being SO(3)--quivariant (3I)
Using real spherical harmonics, a function sf( defined on the sphere is represented by set of weighted spherical harmonics Yım with s.. = Ei.m Pime (f) where and m are the degree and order the basis functions with m E I- and 1 E L. We refer to the spherical harmonic coefficients, Dime. the coefficients of the spherical channel. For every degree 7 there exists 27 + 1 spherical harmonics (see Figure T [a) for an illustration of functions for I S 4), which results a total of (L 1)2 basis functions and coefficients up degree L. Therefore for maximum degree of L with C channels, æ has size (L+ 1)a x C.
The spherical channels are updated by the GNN through message passing for K layers to obtain the final node embeddings 5(K), g(0) is initialized from embedding based on the atom's 's atomic number a; for 1 coefficients and the 1 o coefficients are set zero. The nodes' embeddings are updated by first calculating set of messages mij for each edge, which are then aggregated at
each node. Finally, the energy and forces are estimated from (KK), We describe each of these steps in turn.
2.2 Message Passing
Given target node l. and its neighbors E N; we want to update the embeddings st at iteration K. The embeddings x²( are a set of spherical harmonic coefficients indexed by their degree 1, order m, and channel c. An important and useful property of spherical harmonics is the function represented by the coefficients is steerable [1 5) i.e., can be rotated in 3D space using linear transformation of the coefficients. Specifically, for 3D rotation matrix R there exists matrix called a Wigner D-matrix D' of size (21 + 1 21 + 1) that rotates the coefficients of degree I by the rotation
R. If aste are all coefficients across orders m for node of degree L and channel c, then for any 3D

---

rotation R. there exists a D' that for all orientations A:
Dite" (R =DD'vae) Yi()).
When calculating the message mat from atom 8 to atom t, we want to use the information contained in both a: and x given the context of the edge's orientation dei. du = dee/ |dall. d We do this by rotating the embeddings by R. for which dg = [0, 0, ",] i.e., the direction of the edge's unit vector dg is aligned with the z-axis. Thus the orientation of the atoms with respect to each other is implicit in the rotated embeddings. This simplifies the task of the network since it only needs to learn the relationship between atoms that are aligned along the 7-axis and not arbitrary rotation. After calculating the messages using a neural network Fe, they are rotated back to the global coordinate frame:
m (1++1) (ar!
(K) Fie (k)
dat; as ae) = FD Datar (R) |dge), asy at):
where the matrices Dst and D perform the rotation on the embeddings' coefficients corresponding to the 3D rotation matrices Rgg and RT respectively. To simplify notation, we assume the coef- ficients in T across degree I and order m are flattened, and D.r E ( x R (I++1) is block diagonal matrix constructed from the set of Wigner D-matrices across degrees, E L. D is the inverse of Das, with D = DT. In addition the rotated embeddings, F. is also provided input edge information that is invariant to rotations; d is the magnitude of the distance between atoms and L, and a, and a their respective atomic numbers. Note that the rotation matrix R., is not unique, since the roll rotation around the vector dua is not specified and is randomly sampled during training. The implications of this are discussed below in Section 2.2.1
The message function Fe is computed using a neural network as illustrated in Figure 2 (]. The atomic numbers a, and a, are used to look up two independent embeddings of size E' T28, and set of ID basis functions are used to represent |d.t| using equally spaced Gaussians every 0.02 Á from 0 to the 8 Å with = 0.04 followed by a linear layer with E outputs. Their values are added together and passed through a neural network to produce a vector of size H. The rotated embeddings a., (R) and D.r2t (K) are concatenated and passed through a single layer neural network to produce another vector of size Н. These two are multiplied together to combine the edge information with the information contained in the rotated embeddings. Two more fully connected layers are performed with each followed by a SiLU non- non-linear activation function [22] Finally, a single linear layer is used to expand the output to the original size of the embeddings.
2.2.1 Message Equivariance
For the network to be equivariant to rotations, message function (Equation (2) needs to be equivariant. In general, this is not the case since the matrix Dac is not unique, i.e., the roll rotation around the vector de is not specified, and in practice it is randomly chosen. The roll rotation corresponds to a rotation about the z-axis after rotating the coefficients =²!²) and , by D Due to this, all of the rotated coefficients will vary based on the random roll rotation chosen, except the m = coefficients that are symmetric about the z-axis; see Appendix D and the m = bases highlighted in green in Figure a)). If rotation equivariance is desired for messages, the input and output coefficients to F. can be restricted to only those for which m = O. Since the other inputs |d ag and a, to F. are also invariant to rotations, F. is invariant to rotations if only are used. The resulting messages m. are equivariant to rotations once they are rotated back to the global coordinate frame using D While equivariance is a desirable property, as we demonstrate later, using m e [-1, 1] coefficients can help improve performance even though equivariance is not strictly enforced.
0 coefficients
If we choose to use coefficients beyond just m = O for F. for message passing in Equation (2), two approaches may be taken. The first to simply add the coefficients to the inputs and outputs of F. and assume the neural network will learn roughly equivariant mapping through the random sampling of the roll rotation. As we demonstrate Appendix A the learned functions are indeed roughly equivariant. The second strategy takes a more direct approach to encouraging the network to learn equivariant mappings by taking advantage of the proprieties of the coefficients as they'n 're rotated by é about the z-axis. If D rotates a set z of coefficients about the z-axis by é, the m 0

---

coefficients are constant as a function of o, while the m É [-1, 1 coefficients are sine and cosine functions of (see Appendix D):
(D) = =Y,
(3)
(D D2z) (-)) = a sin( (o+ + (3 (D ()))()))
(4)
(D (1) = cos + (3 = (D (0+5)))))
(5)
for some set of vectors a, 3 and 7 of size L. T(u) are the m = 0 coefficients and similarly for ( and ()) for the m = and m 1 coefficients respectively. Similar proprieties hold for m
and m < 1. We take advantage of these properties to encourage the output of the message block towards equivariance by computing F. multiple rotations & about the z-axis:
F = F. DD DeD
(k) |datl. 01. 0e);
For m [-1,1], compute four samples or "taps" at á {0, T. m, T and combine them based on m using:
mst
(k+1)
D D=11 1
Fu c(u)
F c())
+ F c()))
+ Fž
<((U)
D -1
st
FO e(-1)
F e(-1) + F
e(l))
+
(7)
D 1 st
e((() +
F2" E(-1)
- e(1) - F
e(-1)
F where Fe "c(u)* e(-1) and F '(1)) are the output coefficients for m 0, m 1 and m respectively, The top line of Equation is simply taking the average, since the m coefficients should be constant regardless of Φ. Similarly for m E [-1, 1), the second and third lines average 4 values that should be equal if Equations and 5 hold, e.g., a sin( + B) aa cos( B + π 22. If higher m desired, similar calculations can be performed. However, larger number of taps will be needed, e.g., m E 2- 2] requires 8 taps separated by द radians.
The number of coefficients used during message passing when calculating F. may be varied based on the edge properties reduce the memory required by the network. For instance, atoms that are far away from each other may not need the same resolution of the spherical functions as those close together, so lower maximum degree may be used. By training several message passing networks with varying degrees for different ranges of edge distances, memory usage can be reduced while not impacting accuracy. Similarly, the range of m can typically be truncated [-1, 1] or [-2, 2] since their observed utility to the network is substantially reduced for values of m greater than 2 (or less than -2). See Figure b for examples of the spherical channels when the range of m is reduced. Note that while the number of coefficients used by F, may be reduced, the node embeddings maintain their original size based on the maximum L, so they may be able to aggregate information from all neighboring atoms at different distances and angles.
2.3 Message Aggregation
At this point in the network, the per edge messages computed and rotated back to the global coordinate frame. Ideally, to allow for complex interactions between the messages, a non-lineat function would be applied after summing all messages directed at target node. If we did this by (k+1) through a fully connected neural naively passing the summed coefficients m (k+1) = E. m st
network, the network may have difficulties learning representations that are approximately rotation equivariant (a fundamental property of atom forces). Another option is to place constraints on the non-linearities performed to enforce equivariance 22 36 3
We propose applying an unconstrained non-linear pointwise function on the sphere, i.e., a function that is applied at every orientation without knowledge of the orientation from which the point was sampled [9]. For our function, we use a non-linear neural network F, (RC -> RRC) that combines information across channels. In practice, F. is applied at discrete number of orientations, which
5

---

m (++1)
F
(4+1)
G-1
(8)
1x1 conv
Figure 3: Illustration of message aggregation. The summed messages m (k+1) Σ. ma and
(k+1)
previous embedding z; (k) converted from spherical harmonic representation spherical grid representation using G. The channels are concatenated and passed through a 3 layer x CNN. Each layer is followed by a SiLU activation. Finally, the channels are converted back spherical harmonic representation using G-1 and added to x; to create zi (k+1)
Figure 4: Illustration of different spherical channels for six structures (top). Blue indicates a positive value and red a negative value on the sphere. The spherical channels were sampled from the last layer before the output blocks using a model with 12 layers and L = 6. Note, how some channels have higher activation for adsorbates (darker colors), certain elements or the bottom top of the surface. Illustrations of the structures are shown on the bottom with different colors indicating different elements.
results in transformation that we numerically demonstrate (see Appendix approximately equivariant to rotations:
(++1) =
(k) G (F- (F((m) G(a()))
(8)
where G is function that converts spherical function represented by spherical harmonic coefficients, one represented by point samples a sphere. For G and azimuthal) to generate a 2D discrete representation of the spherical functions, see Figure 3 A 1 1 convolutional neural network F. is applied each discrete sample containing C' channels. and the result is then converted back to the spherical harmonic representation using G This is analogous to having 2D image represented the frequency domain, converting to the spatial domain using an inverse discrete Fourier transform, applying a transformation and converting back to the frequency domain. Since the same operation applied all orientations on the sphere, transformation that is approximately equivariant (up to discrete sampling limitations) to rotations may be learned, see Appendix AA for more detailed discussion.
spherical coordinate system (polar
diagram of the network is shown in Figure 3 To provide additional information to the network, (k) zt is also converted to spherical grid representation, concatenated with G(m, ((tm))))) and provided as input to the neural network to compute the final updates to the coefficients x; (k+11) at iteration i: + 1. The network can learn circular functions on the sphere if varying resolutions of (k)) are provided to the network, similar to Difference of Gaussian filters for 2D images [30]. The resolution of (k) can be reduced by lowering the degree I. before transforming by G. In our experiments, if multiple resolutions or bands are used, we use resolutions of degree and 1. Three layers of 1 x convolutions 2C' channels (or 4C' if two bands used) are performed with each followed by

---

SiLU non-lineat activation function. To avoid aliasing, the spherical grid is sampled at a resolution of 2 (L + 1). See Figure for several example illustrations of spherical channels for different atomic structures.
2.4 Energy and Forces Estimation
We compute the energy E by first estimating a per-atom energy using pointwise function on the sphere and taking its integral over all possible orientations. The per-atom energy estimates are summed obtain the overall system's energy:
Σ EーΣ
Fenerru (f)) df,
E=
where Fenergy is a three layer fully connected neural network (RC - R!) with SiLU activation functions.
Forces may be calculated using two approaches: First, by calculating the gradients of the energy with respect to the atom positions. This approach enforces energy conservation, but due to the need to back-propagate the gradients computationally much more expensive. The second approach computes the forces in a manner similar to energy prediction, which is computationally more efficient, but does not enforce energy conservation. In this approach, the forces are calculated by estimating a (s. (f)) in every direction Á over the sphere. Integration is performed obtain directional vectors:
force magnitude Ifl (o (K) after multiplying the magnitude by the orientation
Si =
/ FFoorr
8
(F)) df,
(10)
where Fjorce is a three layer fully connected neural network (RC R)). In practice. discrete approximation of the integral is performed for equations 9 and (TO) using set of 128 evenly distributed points on the sphere (see Appendix C.2). Note that the application
pointwise function
of an arbitrary neural network at finite number of discrete orientations is not invariant (energies) or equivariant (forces) to rotations. However in practice, numerically this operation does closely approximate these properties (see Appendix A.2
3 Experiments
We present results on the Open Catalyst 2020 (OC20) dataset [6] that is released under a Creative Commons Attribution 4.0 License. OC20 contains over 130M training examples for the task of predicting atomic energies and forces for catalysts used in renewable energy storage and other important applications [48]. This dataset is a popular benchmark for the ML community. We begin by comparing results across all tasks on the test set. Next, we show numerous ablation studies comparing model variations on the smaller OC20 2M dataset. Finally, since several papers only report results on the IS2RE task using approaches that directly predict relaxed energies, we also train model specifically for this task and approach, and compare Appendix B
3.1 Implementation Details
The implementation of spherical harmonics and their transformations uses the code provided by Euclidean neural networks (e3nn) [16]. For message passing, two resolutions of spherical harmonies are used based on atom's nearest neighbors, see Appendix C for exact parameters. Unless otherwise stated C 128, K 16, H 1024, E' 128, and for F. in message passing only orders m E [- are used. All forces are estimated directly as output of the network, unless stated that the energy conserving gradient- based approach was used. During training, the coefficients for the force and energy losses are 100 and respectively. Training is performed using the Adam W optimizer 29 with a learning rate of 0.0004. The effective batch size increased using data parallelism and Py PoTorh''s 's Automatic Mixed Precision (AMP). All model code will be open sourced with an MIT license in the Open Catalyst Github repo. Please see the supplementary for details on the PaiNN (3 baseline.
7

---

OC20 2M Validation
SZEF Force MAE
IS2RE
Samples GPU
Energy MAE
Force Cos
EFWT
Energy MAE
EWT
Model
|meV]
LmeViAI
1811
ImeV
1981
Median
SchNet
78.3
0.109
DimeNete+
805
65.7
0.217
0.01
SpinConv 44]
406
36.2
0.479
0.13
GemNet-dT 14
25.8
358
29.5
0.557
0.61
438
GemNet-OC 15
18.3
286
25.7
0.598
1.06
407
layers
II
B batch
SCN No rotation
512
8.5
67.7
0.192
SCN No Ixl conv
6
12
1024
128
9.1
313
26.2
0.579
0.74
SCN
6
12
1024
128
10.4
307
26.5
0.588
0.83
SCN m E
1
12
1024
128
8.3
302
24.6
0,601
0.93
SCN E -2 2.2
6
12
1024
96
6.9
301
23.4
0.612
1.01
SCN E -1
512
7.6
24.6
0.595
SCN grad-forces
4
12
512
128
1.2
307
26.2
0.573
0.81
SCN direct-forees
512
128
12.1
25.3
0,592
SCN
2
12
256
128
12.9
312
27.6
0.568
0.70
SCN
12
512
128
12.1
303
25.3
0,592
0.87
SCN
6
12
1024
128
8.3
302
24.6
0.601
0.93
SCN
1024
23.2
0.620
SCN
12
1024
64
7.7
299
24.3
0.605
0.98
SCN 2-hand
0.622
SCN 4-tap
6
12
1024
64
3.7
296
22.7
0.638
1.27
SCN 4-tap 2-band
1024
22.2
0.643
371
11.0
SCN 0
6
16
1024
96
7.4
300
25.7
0.600
0.96
394
9.6
SCN
SCN
6
16
1024
96
5.9
287
22.8
0.623
1.22
371
10.5
SCN
16
1024
96
3.5
283
22.7
0.627
1.22
11.3
SCN t-tup
6
16
1024
2.6
282
22.2
0.648
1.37
378
10.7
SCN 4-tap 2-band
6
16
1024
64
2.3
279
21.9
0,650
1.46
373
11.0
Table 1: Results on the OC20 2M training dataset and ablation studies for SCN model variations.
The validation results are averaged across the four OC20 Validation set splits. All SCN models are trained on 16 GPUs for 12 epochs with the learning rate reduced by 0.3 at 5, 9, and 11 epochs, except SCN with = 8 and 16 layers that used 32 GPUs to obtain a larger batch size. Batch sizes vary based on the number of instances that can be fit in 32GB RAM.
3.2 OC20 Tasks
We compare against numerous models trained on the OC20 2M, All and MD datasets across the Structure to Energy and Forces (S2EF), Initial Structure Relaxed Strueture (IS2RS) and Initial Structure to Relaxed Energy (IS2RE) tasks (6). OC20 2M, OC20 All and OC20 MD have 2, 133, and 38 million training examples respectively. The 12 and layer SCN models significantly outperform state-of the-art GemNet-CC [15 S2EF force prediction (z 14% improvement) and IS2RE energy predictions (+z 10% improvement) when trained on OC20 2M, Table Comparable results are found for energy MAE on S2EF with GemNet-OC. We also report throughput efficiency. Note that while the SCN models process fewer samples per second, they are more sample efficient than GemNet (Figure (b)). Efficiency improvements have also been noted for fully equivariant models 3
We compare our largest SCN models trained on the All and the All+MD datasets (133M 38M examples) across all OC20 tasks in Table 2 For these experiments we use deep 1-tap model, and slightly shallower 4-tap model since the use of 4-taps requires more memory. Our model achieves state-of-the-art results for force MAE (= 9% improvement) and force cosine for the S2EF tasks. The SCN L = model performs similarly to the recently released GemNet-OC model [5] on energy MAE for S2EF. On the challenging relaxation based IS2RE task that requires models to predict both accurate forces and energies, SCN outperforms GemNet-OC by over 7%. Note that we only train single model to predict both energy and forces, where GemNet-OC-L-FFae of two models, one trained specifically for forces and the other for energy.
MD is the combination
3.3 Ablation studies
We explore numerous model variations in Table trained on the OC20 2M dataset and evaluated on the OC20 validation set. The first set of ablation experiments explores reducing the complexity of our model. If messages are not rotated before and after applying F.,. in Equation i.e., A

---

OC20 Test
S2EF MAE
IS2RS
IS2RE
Train time
Energy MAB
Force Cos
EPwT 1811
AFOT
ADWT
Energy MAE
Model
#Farmms
meV
ImeVIAJ3
1811
1817
meV.
Median
2258
84.4
0.016
0.01
Train OC20
SchNer
AS 6
9.1M
194d
540
54.7
0.302
0.00
14.4
764
PaiNN
20.1M
67d
341
33.1
0.491
0.46
11.7
48.5
471
DimeNNtt++LLF44
12 6
10.7M
1600d
480
31.3
0.544
0.00
21.7
51.7
559
SpinCoav (direct-forces)
41
8.5M
275d
336
29.7
0.539
0.45
16.7
53.6
437
GemNettdT
[4
32M
4926
292
24.2
0.616
1.20
27.6
58.7
400
GemNet-OC
15
39M
336d
233
20.7
0.666
2,50
35.3
60.3
355
SCN L=8 K=20
271M
645d
244
17.7
0.687
2.59
40.3
67.1
330
Truin OC20
All MD
GemNet- OC-LL- 15
56M
640d
230
21.0
0.665
2.80
GemnNot-OC-LLF 15
216M
765d
241
19.0
0.691
2.97
40.6
60.4
GemNettOC-LLe
348
SCN 7=6 K=16 4-tap 2-band
168M
414d
228
17.8
0.696
2,95
43,3
64.9
328
SCN 1=8 K==0
271M
1280d
237
17.2
0.698
2.89
43.6
67.5
321
Table Comparison of SCN existing GNN models on the S2EF, IS2RS and IS2RE tasks when trained on the All or All+MDD datasets. Average results across all four test splits are reported. We mark as bold the best performance and close ones, i.e., within 0.5 me MAE, which we found to empirically provide a meaningful performance difference. Training time in GPU days. Median represents the trivial baseline of always predicting the median training energy or force across all the validation atoms. The SCN L 8 model has K 20 layers, C 128 and E 256, while the SCN L = 6 model has K = 16 layers. C 128, E = 128 and an energy loss coefficient of 4.
is an identity matrix, the results are significantly worse for energies and even more so for forces since forces are highly dependent on angular information. If replace the non-linear aggregation (Equation 8) with simple summation (No Ixl conv) in a model with m [-1, 1]. the results also degrade across all metrics. Results improve for m E -,, 1] as compared only using m coefficients, which demonstrates that if the equivariant to rotation constraint is relaxed, improved results may be achieved. For higher m, m [-2, 2] m [-4, 4] diminishing returns are noticed with increased computational cost.
In some applications, such as molecular dynamics, energy conserving models needed. We trained an energy conserving model that estimates forces based on the gradients of the energy with respect to the atom positions. Since this approach is more expensive both in memory and computation, we used a smaller model with L = 4, H = 512 and 12 layers (grad-forces). While the accuracy of the model is good, the results are slightly worse and is 10 times slower than comparable model that directly estimates forces (direct- forces)..
When scaling the network, increasing L has more significant impact force estimation, while depth improves energy estimation. This may be due to higher I. resulting in greater angular resolution, which is important to force estimation. Greater depth, which allows information to travel further, may lead to better energy predictions since energy is function the entire atomic structure.
Sampling multiple z-axis rotations when computing F. with 4-taps (Equation 7 produces significantly improved force predictions, while using two bands and and L-1) when aggregating messages improves both energy and force prediction. The use of 2 bands reduces model throughput small amount, while sampling with 4-taps reduces throughput by over 2x. However, sample efficiency is further increased with 4-taps, resulting in similar training times for the same accuracy, Figure 2 b).
Related work
Molecular and atomic property prediction has made significant recent progress. Models relying on hand-crafted representations like by learned feature representations using GNNs 12 14 36)
4 and sGDML g have recently been surpassed Early GNN developments focused only on invariant representations. CGCNN45 and Sch make energy and force estimations using only atomie species and pair-wise distance information. DimeNet 13 SphereNet (8)) and GemNet[14 14 15 extend this to explicitly capture triplet and quadruplet angles, which are scalars. Utilizing invariant representations offers flexibility in model architecture design since the node or edge features are inherently invariant and model constraints are needed to maintain equivariance. The

---

enumeration of triplets and quadruplets of atoms to calculate functions of relative or dihedral angles requires careful model design to maintain efficiency. More recently, equivariant models, capturing both scalar and equívariant features, have outperformed traditional GNNs on small molecular datasets including MD17[Z] and (M9[32]. While not strictly equivariant, our work falls into this later class of GNNs for atomic property predictions.
Models that strictly enforce equivariance the SO(3) (3D rotations) or E(3) group (3D rigid
transformations) share similarities to our use of spherical channels [6 T 9 S Both represent
atoms by higher-order tensors and not just scalar information. For instance, Tensor Field Networks
42), Cormorant []. NequlP [3], SEGNN [5] and BOTNet [2] use the same spherical harmonic representations as SCNs, while PaiNN 36 is an 1 variation. All models place constraints on the operations that can be performed to ensure equivarance, such as restricting non-lineat operators to invariant scalar inputs, e.g.. pair-wise atom distances. networks like TFN, geometric information of higher degree I are mixed into the I = 0 features via tensor products on which a gated non-linear function is applied. Similar to our approach, the recently proposed SEGNN uses the spherical harmonic representation enable stecrable features based edge orientations. SEGNN introduces steerable MLPs that enforce equivariance by restricting the learnable parameters to those that scale the feessch-Gordal tensor products followed by gated non-linearities 43 In our approach, we increase the expressivity of the model by identifying a set of m = 0 coefficients across degrees that invariant to rotations given an edge's orientation to which unconstrained functions may be applied. We also demonstrate that improved performance can be achieved by relaxing the requirement for strict message equivariance by including a broader set of coefficients beyond = O. Furthermore, we project the spherical channels onto a grid and perform a pointwise 1 x 1 convolution followed by non-linearity. Although not strictly equivariant, the pointwise operation allows for complex mixing of different degrees of spherical harmonics resulting in rich geometric descriptions. Perhaps as result of the network's 's increased expessivity, we see increase in accuracy with higher degrees (L- = 4, 6, 8) unlike previous approaches that typically use L = or L L=2 2.
Discussion
A limitation of SCN is that it is very computationally expensive if energy conservation is enforced through the estimation of forces using energy gradients with respect to atom positions. This may limit its in chemistry applications such molecular dynamics. Non-energy conserving models are still practically useful for applications such as structure relaxations and transition state searches. The SCN model scales On²) with respect to L, which may limit using L larger than 8. Using larger set of m coefficients during message passing also faces challenges, since the network may find it more difficult to learn approximately equivariant mappings. In Table I, saw that increased depth improves results, especially to energy prediction. It remains an open question whether deeper networks will see further improvement.
Given SCN's s increased expressivity due to the relaxation of the equivariance constraint and larger model sizes, it is prone to overfitting on smaller datasets. SCN's results on datasets such as MD17(7) and QM9 are not state-of-the- pre-training on larger datasets, such as OC20, and see if improved results can be achieved on smaller datasets through finetuning (25).
However, in future work would be interesting to explore
While this work is motivated by problems we face in addressing climate change, advances in chemistry may have numerous use cases. Some of these use cases may have unintended consequences, such as the development of the Haber-Bosch process for ammonia fertilizer production that enabled the world to feed a growing population, but over fertilization has recently led to ocean "dead zones" and its production is very carbon intensive. More alarmingly, the knowledge gained from fertilizer production was used to create explosives during wartime 200). We hope to steer research in this area towards beneficial applications by utilizing datasets such OC20.
In conclusion, we propose GNN that uses spherical harmonics to represent channel embeddings that contain explicit orientation information. While the network is encouraged to learn output mappings that are equivariant to rotations, best performance is achieved by relaxing the equivariant constraints and allowing for more expressive non-linear transformations. We demonstrate state- of-the-art results across numerous tasks on the large-scale OC20 dataset for modeling catalyst for applications addressing climate change.
10

---

References
[1] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances neural information processing systems, 32, 2019.
[2] llyes Batatia, Simon Batzner, Dávid Péter Kovács, Albert Musaelian, Gregor NC Simm, Ralf Drautz, Christoph Ortner, Boris Kozinsky, and Gábor Csányi The design space of e (3)- equivariant atom-centered interatomic potentials. arXiv preprint arXiv:2205. 06643, 2022.
[3] Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess Smidt, and Boris Kozinsky. (3)-qqiivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):
1-11, 2022.
[4] Jörg Behler. Perspective: Machine learning potentials for atomistic simulations. The Journal of chemical physics, 145(17):170001, 2016.
[5] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve (3) equivariant message passing. arXiv preprint arXiv:2110.02905, 2021.
[6] Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Mor- gane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 11(10))6059-6072, 2021.
[7] Stefan Chmiela, Alexandre Tkatchenko, Huziel Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5)x16603015, 2017.
[8] Stefan Chmiela, Huziel E Sauceda, Klaus-Robar Müller, and Alexandre Tkatchenko. Towards exact molecular dynamics simulations with machine-leamed force fields. Nature communica- tions, 9()::1-10, 2018.
[9] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In ICLR, 2018.
[10] Carlos Esteves, Christine Blanchel,,,h,tte, Ameesh Makadia, and Kostas Daniilidis. Learn- ing so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68. 2018
[11] William T Freeman, Edward H Adelson, et al. The design and use of steerable filters. IEEE Transactions Pattern analysis and machine intelligence, 13(9):891-906, 1991.
[12] Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. In NeurlPS-Wy - 2020.
[13] Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations (ICLR), 2020.
[14] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Universal directional graph neural networks for molecules. arXiv preprint arXiv:2106.0880 08903, 2021.
[15] Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Günnemann, Zachary Ulissi,
C Lawrence Zitnick, and Abhishek Das. GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets. Transactions on Machine Learning Research (TMLR), 2022.
[16] Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kostiantyn Lapchevskyi. Maurice Weiler, Michal Tyszkiewicz, Simon Batzner, Martin Uhrin, Jes Frellsen, Nuri Jung, Sophia Sanborn, Josh Rackers, and Michael Bailey. Fuclidean neural networks: e3nn, 2020. httphttp:///doo org/ 10.
5292912

---

[17] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International Conference on Machine Learning (ICML.), pages 1273-1272, 2017.
[18] Jonathan Godwin, Michael Schaarschmidt, Alexander L Gaunt, Alvaro Manchez-Gonzalee Yulia Rubanova, Petar Veličkovié, James Kirkpatrick, and Peter Battaglia. Simple gnn regularisation for 3d molecular property prediction and beyond. In International Conference on Learning Representations, 2021.
[19] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729-734. IEEE, 2005.
[20] Thomas Hager. The alchemy of air: Jewish genius, doomed tycoon, and the scientific discovery that fed the world but fueled the rise of Hitler. Broadway Books, 2009.
[21] JH Hannay and JF Nye. Fibonacci numerical integration on a sphere. Journal of Physics A: Mathematical and General, 37(48): 1,91,, 2594,2004.
[22] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1645..8466.......11 2016.
[23] Weihua Hu, Muhammed Shuaibi, Abhishek Das, Siddharth Goyal, Anuroop Sriram, Jure Leskovec, Devi Parikh, and C Lawrence Zitnick. Forcenet: A graph neural network for large-scale quantum calculations. arXiv preprint arXiv:2103.01436, 2021.
[24] Peter Bjørn Jørgensen, Karsten Wedel Jacobsen, and Mikkel N Schmidt. Neural message passing with edge updates for predicting properties of molecules and materials. arXiv preprint arXiv:1806.03144 2018.
[25] Adeesh Kolluru, Nima Shoghi, Muhammed Shuaībi, Siddharth Goyal, Abhishek Das, C Lawrence Zitnick, and Zachary Ulissi. Transfer learning using attentions across atomic systems with graph neural networks (taag). The Journal of Chemical Physics, 156(18)184702, 2022.
[26] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch- gorddn nets: fully fourier space spherical convolutional neural network. Advances Neural Information Processing Systems, 31, 2018.
[27] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11))2278223) 23224, 1998.
[28] Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013. 2021.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[30] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
[31] Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R Manby, and Thomas F Miller III. Orbnet: Deep learning for quantum chemistry using adapted atomic-orbital features. arXiv preprint arXiv:2007.08026, 2020.
[32] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177, 2014.
[33] David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola MlojevvccDupont Natasha Jaques, Anna Waldman- Brown, et al. Tackling climate change with machine learning. arXiv preprint 173443190..05433, 2019.
[34] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. In International conference machine learning, pages 9323-9332. PMLR, 2021.
12

---

[35] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix. Stefan Chmicla, Alexandre Tkatchenko, and Klaus-Rober Robert Müller. Schnet: continuous- convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems (NeurIPS), pages 991-1001, 2017.
[36] Kristof Schüit, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377-9388. PMLR, 2021.
[37] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-cheme insights from deep tensor neural networks. Nature com- munications, 8:13890, 2017.
[38] Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148122) 2018.
[39] Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 5777792))77777777 2020.
[40] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[41] Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. Rotation invariant graph neural networks using spin convolu- tions. arXiv preprint arXiv:2106.0995 2021.
[42] Nathaniel Thomas, Tess Smidt, Steven Kearne,, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivarinn neural networks for 3d point clouds. arXiv preprint /xxiv:/802.002 2018.
[43] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steer- able cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.02547, 2018.
[44] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation cquivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 849-858, 2018.
[45] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 20(14): 145301, 2018.
[46] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tin-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34, 2021.
[47] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
[48] C. L.. Zitnick, L. Chanussot, A. Das, S. Goyal, J. Geraa--ommnnoo C. Ho, W. Hu, T. Lavril, A. Palizhati, M. Riviere, M. Shuaibi, A. Sriram, K. Tran, B. Wood, J. Yoon, D. Parikh, and Z. Ulissi. An introduction to electrocatalyst design using machine learning for renewable energy storage. arXiv preprini arXi::2010.09455 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes], see section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see
section [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes], code is released with MIT license, OCP GitHub.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] experiments are too expensive to run multiple times. However, we provide rough estimates for run variations in Table T
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]. training times and throughput proved.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
14


Reviews:
[{"review_id": "xE3MfwhqA0M", "reviewer": "Reviewer_A9CN", "summary": "The authors present a novel deep learning architecture to predict scalars and non-conservative vector fields for applications in molecular modeling.", "questions": "- How is the orientation $\\hat{r}$ chosen for the node embeddings? Is this choice of $\\hat{r}$ independent for all atoms? How do you compute (4) efficiently?\n- Do you integrate over all rotations in (4) and (5). I believe a $d\\hat{r}$ is missing.\n- Have you considered reporting standard results on datasets like (rev)MD17 or 3BPA?\n- What is the effect of the model's size on its performance?\n- Did you study the effect of the fact that the predicted forces are not conservative? Did you try building a conservative force field from your model for the energy?", "limitations": "The authors have adequately addressed the potential negative societal impact of their work. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The model architecture is simple.\n- The (enormous) model seems to perform better than other (smaller) models on the OC20 task\n- The model seems to be faster than other models (despite its size).\n\n# Weaknesses\n\n- The manuscript fails to relate the presented model to existing ones on an architectural level. While the related work section mentions other models, it is unclear what it is about SCN that other approaches should adopt to improve predictive performance and reduce computational time.\n- It is unclear why the presented model outperforms existing ones on the OC20 dataset. The presented model contains up to 10x more parameters than many other (competitive) models. There is no ablation study investigating the effect of model's size.\n- The model is not run on standard benchmarks such as (rev)MD17 or 3BPA. As a result, it is difficult to compare this model to existing ones and pinpoint the source of its performance.\n- The model is not implementing a conservative force field. As a result, the applicability of the model to problem in computational chemistry mentioned in the abstract is severely limited. For instance, it is unsuitable as a potential in MD simulations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "982GXp3l8ps", "reviewer": "Reviewer_xHpZ", "summary": "The authors develop new kinds of deep learning models for predicting energies and forces from 3D molecular structures. Their core approach is based on representing node embeddings as spherical signals (\"spherical channels\" via coefficients in a spherical harmonic basis) and iteratively updating them in a message-passing process. While this is similar to some prior equivariant networks for 3D molecular structure, they make a few changes: (1) they orient the spherical signals during message passing in a way where the ij direction vector is consistent (non equivariant because the part of the rotation is underdetermined and randomized) and (2) they use more flexible pointwise processing to process the  signals by converting them back and forth to grid representations (approximately equivariant based on sampling resolution). They find that this approach gives strong empirical performance on predicting forces, energies, and relaxed energies on the Open Catalyst dataset.", "questions": "My main request to authors would be that they address the Loss of equivariance issue described in the Weaknesses above. It would be helpful to have further discussion on how the force and energy errors under rotation might be further mitigated and also how much it would affect compounding error for something like molecular relaxation.\n\nUPDATE:\n8/9 - I read the authors rebuttal and appreciate their additional section about message equivariance. I stand by my review and I like the paper. I think it is sufficiently different from prior works for chemistry ML, shows compelling performance, and will challenge the the field to continue developing models that can realize these benefits but with exact equivariance.", "limitations": "The authors adequately discuss both model and societal limitations in the discussion section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nClear and practical ideas. While the author's modifications break rigid equivariance, they do so in an interesting way. Predictably orienting the spherical signals during message passing intuitively makes sense and makes it possible to process the spherical harmonic coefficients in a very flexible and unconstrained way (FC layers) while still having some of the rotational degrees of freedom fixed. I found the paper well-written and easy to follow. The authors adequately position with respect to prior work.\n\nStrong evaluation results. The results are compelling, especially for the relaxed energy tasks (IS2RE). These are interesting because in some ways they would in principle require more high-order geometric reasoning and feature processing than direct energies which can often mostly be predicted from lower-order geometric features (i.e. it would need to be able to predict how energy will change upon coordinate re-configurations).\n\nWeaknesses\n\nLoss of equivariance. As the authors acknowledge, their final best performing models are not equivariant  but they compare to models that are equivariant by construction. I appreciate that the authors did a supplemental analysis on the errors incurred to forces and energies upon random rotation (which were non-trivially large), and even included an analysis based on summing over multiple rotation. It would be helpful to have more discussion on alternative strategies to achieving or improving equivariance, which without summing over multiple rotations seemed to give non-trivially large errors. I am curious if there would be non-randomized, canonicalization approaches to deal with the issue, for example orienting the rotation around the d_st axis in a way that points to the molecular center of mass, or something like that.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6akc3PTiB-i", "reviewer": "Reviewer_RB4y", "summary": "The authors propose a graph neural network (GNN) they name the \"Spherical Channel Network\" (SCN) to map molecular configurations to atomic energies and forces. They model the atom embeddings as a set of spherical channels using spherical harmonics. The authors relax the equivariance constraint in both message passing and aggregation and demonstrate SOTA results on the Open Catalyst dataset for the prediction of energies and forces. The proposed methodology aims to replace the computationally expensive Density Functional Theory (DFT) calculations necessary to compute an estimate of atomic energies and forces.", "questions": "* Can the authors compute the significance of the reported results as compared to the other baseline models reported in the manuscript?\n* For the non-equivariant scenario, how is the variance in the predictions affected by the number of translations in the training set?\n* Have the authors tried evaluating the models on larger molecules than seen in the training set? Is there any reasoning behind outliers or failures seen during testing?", "limitations": "Yes limitations adequately specified.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The use of spherical harmonics to reason about directional information for all neighboring atoms as opposed to atom pairs.\n* Apparent superior out-of-domain generalization compared to baseline models.\n* Model allows for the complex mixing of different degrees of spherical harmonics enabling improvements in representation learning that can improve generalization performance.\n\nWeaknesses:\n* Relaxation of equivariance constraints can make reproducibility difficult and introduce variance in production.\n* Energy is not conserved which hinders applicability to finite chemical systems in which equilibrium dynamics is concerned. \n* Requires arbitrary specification of rotation around vector $d_{st}$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept