ForceNet: A Graph Neural Network for Large-Scale Quantum Chemistry Simulation
Authors: Weihua Hu
Muhammed Shuaibi
Abhishek Das
Siddharth Goyal
Anuroop Sriram
Jure Leskovec
Devi Parikh
Larry Zitnick
Venue:  (2021)
Keywords: Graph Neural Networks
Physical simulation
Quantum chemistry
Catalysis

Abstract: Machine Learning (ML) has a potential to dramatically accelerate large-scale physics-based simulations. However, practical models for real large-scale and complex problems remain out of reach. Here we present ForceNet, a model for accurate and fast quantum chemistry simulations to accelerate catalyst discovery for renewable energy applications. ForceNet is a graph neural network that uses surrounding 3D molecular structure to estimate per-atom forces---a central capability for performing atomic simulations. The key challenge is to accurately capture highly complex and non-linear quantum interactions of atoms in 3D space, on which forces are dependent. To this end, ForceNet adopts (1) expressive message passing architecture, (2) appropriate choice of basis and non-linear activation functions, and (3) model scaling in terms of network depth and width. We show ForceNet reduces the estimation error of atomic forces by 30% compared to existing ML models, and generalizes well to out-of-distribution structures. Finally,  we apply ForceNet to the large-scale catalyst dataset, OC20. We use ForceNet to perform quantum chemistry simulations, where ForceNet is able to achieve 4x higher success rate than existing ML models. Overall, we demonstrate the potential for ML-based simulations to achieve practical usefulness while being orders of magnitude faster than physics-based simulations.

Under review as a conference paper at ICLR 2021
FORCENET: A GRAPH NEURAL NETWORK FOR LARGE-SCALE QUANTUM CHEMISTRY SIMULATION
Anonymous authors Paper under double-blind review
ABSTRACT
Machine Learning (ML) has potential to dramatically accelerate large-scale physics-based simulations. However, practical models for real large-scale and com- plex problems remain out of reach. Here we present ForceNet, a model for accurate and fast quantum chemistry simulations to accelerate catalyst discovery for renew- able energy applications. ForceNet is graph neural network that uses surrounding 3D molecular structure to estimate per-atom forces- a central capability for per- forming atomic simulations. The key challenge is to accurately capture highly complex and non-linear quantum interactions of atoms in 3D space, on which forces dependent. To this end, ForceNet adopts (1) expressive message passing architecture, (2) appropriate choice of basis and non-linear activation functions, and (3) model scaling in terms of network depth and width. We show ForceNet reduces the estimation error of atomic forces by 30% compared to existing ML models, and generalizes well to out-of-distributio structures. Finally, we apply ForceNet to the large-scale catalyst dataset, OC20. We use ForceNet to perform quantum chemistry simulations, where ForceNet is able to achieve 4x higher success rate than existing ML models. Overall, we demonstrate the potential for ML-based simulations to achieve practical usefulness while being orders of magnitude faster than physics- based simulations.
INTRODUCTION
Learning models for simulating complex physical systems has attracted much recent atten- tion (Sanchez-Gonzalez et al., 2020 BBasst et al., 2020 Kipf et al. 2018 Battaglia et al., 2016 Gilmer et all.|2017] Schütt et al. 2017; Klicpera et al., 2020). The premise is that once an accurate ML-based simulator is obtained, it can perform inference ofders-of-magniuee faster than the original underlying physics- simulator. Many existing works have focused on relatively small-scale scale simple domains, where physics-based simulation is cheap (i.e., order of seconds), such as simulating springs and oscillators (Battaglia et al. 2016 Kipf et al. 2018), fluids and rigid solids (Sanchez) Gonzalez et al., (2220) that follow classical Newtonian dynamics, and glassy systems (Bapst et al. 2020) that follow the basic Lennard-Jones potential. Other approaches have explored complex domains with smaller systems, such as small organic molecules (Ramakrishnan et al. 2014 Schlitt al. 2017 Klicpera al. 2020).
An important open question is whether ML-based simulation is effective in larger and more complex quantum chemistry domains. If successful, ML could be applied to problems such as catalyst discovery which is key to solving many societal and energy challenges including solar fuels synthesis, long-term energy storage, and renewable fertilizer production (Seh et al. (2017); Jouny et al. (2018); Whipple & Kenis (2010)). Currently, Density Functional Theory (DFT) Parr 1980) is a popular and reliable, but computationally expensive approach to performing quantum chemistry simulation tasks, such atomic structure relaxation (Figure [[ (left) and molecular dynamics. Approximating DFT with ML-models is an exceptionally challenging task given a simulation's 's sensitivity to the atoms' elements and subtle changes in the atoms' positions. However, if successful, accurate and fast ML-based models may lead to significant practical impact by accelerating simulations from O(hours-days) Oms--)) which in turn accelerates applications such catalyst discovery.
key component of any ML-model used for atomic simulations is predicting the non-linear and complex forces applied atoms by their neighbors. These forces are highly sensitive the element type and small changes in placement of neighboring atoms. One approach that is well-suited to

---

Under review as a conference paper at ICLR 2021
DFT
SchNet
ForceNet-large
Energy (ev)
Energy (ev)
463
664
466
-655
465
866
150 200
0.001
0.01
0.1
10 100
1000 10000
100000
Itoration
Time [seconds]
Figure 1: (left) Illustrative example of atomic structure energies during relaxations performed using DFT and ML-based (SchNet, GNS, ForceNet) approaches. 3D renderings of the structures along the relaxation trajectory are shown, where small atoms are adsorbates and larger atoms are catalysts. The ForceNet (ours) model reaches low energy similar to DFT. (right) Same plot, but changing the x-axis to time rather than iteration steps. All ML models are more than 10 x faster (GPU acceleration of DFT is only 2-5x (Maintz & Wetzstein 2018), while only ForceNet finds similar energy that of DFT this example. Note different optimizers were used for DFT (conjugate gradient) vs. ForceNet (L-BFGS), which may have led to them reaching the same energy by different trajectories.
the modeling of local interactions is Graph Neural Networks (GNNs) (Gilmer et al. 2017) where nodes represent atoms and messages passed along edges represent the atom interactions. In this paper, present ForceNet that demonstrates significant improvement in quantum simulation performance, and provides strong evidence that GNN- -bsedd simulation is effective in practically- relevant and realistic domains. We build on the recent framework of Graph sewwork-based Simulators (GNS) -ooazllzz et al. 2020; Bapst et al., 2020), where node movements (i.e, atomic forces in our application) are predicted from the node embeddings of GNN. We address the key challenge of accurately capturing the local 3D atomic structure using three approaches:
To model the complex atom interactions in 3D space, we extend the continuous filter convolution (Schütt et al. (017)) to make much expressive; condition the convolution filter not only on the Huclidean distance between atoms, but also on the embeddings of source and target nodes well the (x, y, 2) directional differences.
Conditional filter convolution:
Carefully-chosen basis and activation functions: We demonstrate that it is critical to carefully select the appropriate non-linear activation and basis function in the peessage-passing architecture, so that complex non-linear dynamics can be effectively captured. Through comprehensive empirical studies, we identify particularly effective design choice that is based on spherical harmonics and Swish activation (Ramachandran et al. 2017).
Scaling: The scaling of model size is necessary for capturing the complexity of the forces. The model scaled with respect to depth and width significantly improve performance.
We apply our model to the new large-scale quantum chemistry dataset OC20 (Anonymous) 2020) that contains 200+ million samples from atomic relaxations relevant the discovery of new catalysts for renewable energy storage and other energy applications. The result of our work is a simple and scalable ForceNet model that achieves ffte-oofthheart performance in predicting quantum force fields, reducing the MAE force errors by 30% compared to existing GNN models. Similar performance gains are obtained on out-of-distribuutin tasks for which similar atomic structures are not seen during training. Finally, we use ForceNet as surrogate to DFT to perform quantum chemistry simulation; specifically, calculating structure relaxations of complex systems, task of high practical value. We show that ForceNet able to estimate the relaxed structures 4x more accurately than previous state-of-the-arts while being multiple-orders of magnitude faster than DFT, Figure]
2 RELATED WORK
Message-passing GNNs. Our ForceNet is based on message passing GNNs Gilmer et al. 2017) that iteratively update node embeddings based on messages passed from neighboring nodes. In its most general form, the message function depends on the two node embeddings as well as edge

---

Under review as a conference paper at ICLR 2021
features. Many GNN variants fall under this framework (iipf & Welling 2017: Velickovic et al. 2018, Xu et al. 2019). The recently-propose GNN-FiLM (Brockschmidi 2020) uses an embedding of the target node to modulate the message from the source nodes, and is closely related to our use of a conditional filter. However, most existing message-passing GNNs, including GNN-FiL.M, are designed for homogeneous graphs without edge features. Consequently. edge features are often incorporated in an adhoc manner and can be ineffective approximating complex atomic interactions, which are central to this work.
Force-centric Graph Network-based Simulation (GNS). Our ForceNet builds on the force- centric GNS framework (Sanchez- Gonzalez al. 2020 Bapst al. 2020). Here a model's primary output is per-atom forces (thus, force-centric), which are then used to perform simulation, i... updat- ing atom positions based forces. The GNS framework follows the three steps to predict forces: (1) A graph is constructed from 3D points, (2) an encoder GNN is applied to the graph to obtain node embeddings, and (3) a decoder is applied to the node embedding to predict the per-node forces. The GNS framework has demonstrated promising results on relatively simple domains such as fluid dynamics and glassy systems, where ground-truth simulation is already cheap and can be generated on-the-fly during training. Compared to these domains, using MI. to replace expensive quantum mechanical simulation is more practically impactful and challenging. In fact, empirically, the off-the-shell GNS model fails to accurately predict the quantum mechanical forces.
demonstrate
Energy-centric Simulation. The majority of GNN models developed for quantum chemistry simulation fall under the energy-centric simulation framework, in which a model's primary output is the energy of the entire atomic system (hence, energy- centrcc.. The atomic forces are then derived implicitly through negative gradients of energy with respect to the atomic positions, which guarantees the force-field is energy-conserving. Although physically-principled energy-centric models are empirically found to underperform the force-centric models (especially ForceNet and its variants) in our task. This is possibly because the force-centric models capture the dependency of atomic interactions on atomic forces more explicitly than the energy-centri models.
Many sophisticated GNN architectures have been proposed under the energy-centric framework. such as SchNet (Schütt et al. (017) and DimeNet (Klicpera et al.) (020). Our conditional filter convolution is built from SchNet's continuous filter convolution architecture, where make an important extension to resolve a number of critical issues when adopting it to the force-centric GNS framework (see Section 3.1.1 for details).
3 FORCENET
Here we introduce ForceNet by specifying its input and output, the model architecture and training strategy. The goal of the OC20 dataset is to take set of 3D atom positions, referred to as the initial structure, and find its relaxed structure. The relaxed structure is defined as set of 3D atom positions that correspond to local energy minimum for the system. This is typically accomplished by computing the per-atom forces, updating the atom positions based on the forces. and repeating until the forces approach zero. Our goal is compute these forces using ForceNet. Additional details about the OC20 dataset can be found in the supplementary material.
The input to ForceNet is an atomic structure, i.e., set of M atoms and their 3D spatial positions. The output a 3D vector for each node, representing the predicted (xr, y, 2) atomic force.
3.1 MODEL ARCHITECTURE
ForceNet represents atoms as nodes in a GNN and the atomic interactions as edges. The node input features specify the atom's atomic number and other properties (9-dimensional vector adapted from Xie & Grossman (2018). ). Edges in the GNN are constructed from a radius graph of neighboring atoms (Schütt et al. 2017 atoms that are within the cutofi-distance away from the target atom t. On average an atom has 35 neighbors. A directed edge eat from source atom s to target atom t is drawn for s E Nc).). Let du E R be their relative displacement, i.e., a vector pointing from atom s to atom t.
et al. 2020). Let (c) denote a set of neighboring
ForceNet follows the encoder-decoder architecture of the GNS framework
et al.)
2020, |Battaglia et al. 2016 Kipf et al., 2018). The encoder uses iterative message passing compute node embeddings h that capture the 3D structure surrounding each atom, and the decoder uses an

---

Under review as a conference paper at ICLR 2021
h
F
0.04
Swish
0.04
Fc
0.03
a
Average
0.037
est
B
F
F
mst
0.034
(k)
0,33
h
Spharical
Sing
Gaiss
Liwar Art
identity
Nome
Figure 2: Model diagram for messages mst used by ForceNet Eqns. and (3) Two key components (1) the expressive conditional filter Fg that dependent on full edge feature as well source and target node embeddings, and (2) the basis function B over the edge feature that helps the network to accurately capture atomic interactions.
Figure Ablations on basis functions (x-axis) and activation functions (ReLU and Swish) in ForceNet.
MLP to directly predict per-atom forces from these embeddings. The encoder updates h, as:
h
(k++)
Fx
m +
Σ SEN;
myst
+ h (R)
where the messages M&t and m, are summed and passed through the function Fn RD + R that a 1-hidden-layer MLP with batch normalization (loffe & Szegedy 2015). The dimensionality of the node and hidden layer features is D. Equation ① follows standard GNN embedding update formulations Giimer et al. 2017) with the addition of residual connection, h. (He et al. 2016). We define the directional messages m,t and self message m, in the next section.
h The decoder is computed using the last layer K's node embeddings (K) f: = ( (K) where fo the 3D force of atom f, and Fj is 1-hidden-layy -ayer MLP with batch normalization.
The critical aspect of ForceNet is its encoder and specifically the message computation that effectively captures the non-linear and complex atomic interactions predict the atomic forces. To this end, present our approach to message computation and its three key architectural components.
3.1.1 CONDITIONAL FILTER CONVOLUTION
The computation of messages ForceNet utilizes a simple yet effective extension of the continuous filter convolution (Schütt et al.) 2017). The continuous filter convolution of (Schütt et al., 2017) passes the distance between neighboring atoms through a set of radial basis functions to create an embedding used to compute the messages. This approach has the following limitations that arise when transitioning from an energy-centric to force-centric model: (1) edge features are only computed from atom distance information, i.e., 3D angular information is lost. This results in the node embeddings being rotation-invariant, but forces are rotation-covariant (rotate together with a molecular system) (2) the filter does not take into account information about the node or atom properties. Changes in atom properties, such as atomic number and radii, may result in significant force differences for atoms at similar distances.
We resolve these issues by using our E-dimensional edge feature est (described below) that encodes rotation-covariant directional information, and conditioning on both the source h and target h node information:
m&d = Fo est,
eor, 11 (1) Fa(h
(2)
P where F. : RP X R is the conditional filter, and Fg RD + -RD is learnable
linear function. The edge feature ext Concat Pat/c) E R., where n& = d d is a normalized directional vector and Par E I is a list of four atomic distances ||d.rll, d.. /t1 |d.ll a, 0, that take into account the atomic radii a, and a, of atoms and respectively. While the use of est results in the model not being rotation invariant due to the use
ag ||d,r|| 4z
of nst, data augmentation may be used to encourage rotation-covariance of the model's predictions (explored further in Section 3.2. As our results demonstrate, applying the conditional filter F, on the

---

Under review as a conference paper at ICLR 2021
embeddings Fa of neighboring nodes through the element- wise dot product significantly improves performance, even though F. already contains the source node information h
The conditional filter F. combines the raw edge features est with the node embeddings to encode the interactions between atoms $ and (Figure 2 and is defined as
(k), Est: 11 Fo
a(Idsal)) Fe
(n! (R), Fb (es)) hi (k)
(3)
where a(x:) = cos(
cos((Tr:) /T: /T2 22c is scalar that decays to zero as || dgell approaches the distance cutoff c. F. is 2-hidden-layer MLP with the hidden size of D and the three input vectors are concatenated as input. B : _ + R is the basis function we discuss in the next section, and F, R + RD is learnable linear function that maps the B dimensional vector to a D dimensional vector for input F.. The parameters for F, are shared across layers, while no other parameters are shared across layers. Finally, the self message m, is defined by applying an element-wise product between learnalle filter E IR and Fa, ie.,, m. Fa(h.) (k)
3.1.2 BASIS FUNCTIONS
An important aspect of Fe is the choice of basis function B R - RR that transforms the raw distance features Est into ones that are more discriminative. Several choices of basis functions have been proposed, such as a Gaussian over ID distances (Schütt et al. (017) and a spherical Bessel function over the joint 2D space of the edge distance and angle (Kliepera et al. (020). We extend these ideas to capture the full 3D positional differences between atoms, and systematically study the effectiveness of different basis functions in the context of force-centric model.
Each of our basis functions B maps the raw edge distance features estt into dimensional vector, where B varies based on the basis function used. B is typically much larger than E = 7 to aid in capturing subtle differences in atom positions.
Identity: (xx()) = æ. The baseline is to use the edge features e,t directly.
Linear Act: Brinact (a) g( (Wa b), where g(-) is the non-linear activation function, and W and b are the learnable parameters. When followed by the linear layer F, this is equivalent to applying an I-hidden-layer MLP over the edge features est.
Gaussian: B Bgauss(2) (x) = [b]....,by], where by is the output of the j-th basis function (y()) = exp(x-H,)² (2)==)))((00'))) The Gaussian means are evenly distributed on the interval between 0 and 1, i.e., μ1 = j/(J - 1) and the standard deviation o 1/(J - 1). All values of x are normalized to lie between 0 and 1. B gauss is applied to each dimension of est, resulting in a B = J x E vector.
Bain (x) 1bi....,b,, by]. where by is the output of the j-th basis function (x) = sin(1.1'r) The design is based on function approximation using the Fourier series. In our experiments, we find that using only the sinusoidal component of the Fourier series is sufficient. B is applied to each dimension of Est, resulting in an B = J x E vector.
Sine:
Spherical harmonics: Beol ex))) = (0,, σ) R(px)) where YL is the list of Laplace's spherical harmonics (MacRoberi) 1947) used to encode the angular information and R(pst) ) encodes the distance. We use spherical harmonic functions up to degree L., which gives us T?2 orthogonal basis in total. The angles 0 and & can be directly computed from nzt C est- R uses a linear combination of the above sine basis functions computed from pst C est (thus, 4J basis functions in total) to encode distance information. Specifically, R(pst) W Bunu (p.t) + brad E IRS, where Wrad and brad are learnable parameters. Bsph flattened into a vector before being passed into F,, The dimensionality of Baph is B SL2, where set be the dimensionality of Psc.
3.1.3 EXPRESSIVE
IN MLPS
Our final key component is simple: the choice of non-lineat activation function plays a central role in modeling complex non-lineen lineariiiiss. The popular ReLU activation is not ideal, since results in outputs being modeled as piece-wise linear hyper-planes with sharp boundaries. In contrast, atomic forces are locally smooth with continuous curvatures. Furthermore, negative values set ReLUU, reducing its expressivity. Ideally, we desire a smooth and expressive non-linear activation function. Here, we adopt the Swish activation, i.e,, act(x) x sigmoid(x: (Ramachandran et al.) 2017), which provides a smoother output landscape and has non-zero activation for negative inputs.
in

---

Under review as a conference paper at ICLR 2021
As demonstrate in Section 4 the replacement of ReLU with Swish consistently and significantly improves the predictive accuracy while maintaining scalability across all choices basis functions.
3.2 TRAINING STRATEGIES
We present two strategies for training ForceNet. First, we apply random rotation data augmentation to encourage the rotation-covariance of the model. In our application, the z-axis is canonicalized and is always vertical to the material surface. Thus, we only need to rotate the systems and the forces along the z-axis. Second, during the relaxation process, some atoms are kept fixed while others are free to move. At test time, evaluation is only performed on free atoms. However, we train on both free and fixed atoms using different weights for the loss. See Table in the supplementary material.
EXPERIMENTS
In this section, we show that NorceNet significantly improves on predicting forces compared to both energy-centric GNNs (Schüitt et al. 2017: Klicpera et al. 2020) and existing force-centric GNS models (Sanahez-Gonzalez et al. 2020). Through a series of ablation studies, we provide insight into how each building block of ForceNet and model size contributes to its performance. We also use ForceNet to perform structure relaxations and demonstrate its practical effectiveness.
Experimental settings. We train and evaluate our models on a new large-scale quantum chemistry simulation dataset, OC20 (Anonymous] 22020), that is relevant to the discovery of new catalysts for renewable energy storage and other energy applications. The dataset includes 200+ million simulation samples from 1M+ atomic relaxations containing large atomic structures (20-200+ atoms) and corre- sponding per-atom forces. Compared to the existing quantum chemistry QM9 dataset (Ramakrishnan (et al. 2014), OC20 contains three-orders-of ofrrrrrrrrr----mmgnuuuueeee more data points and was constructed from 70 mi million hours of highly accurate Density Functional Theory (DFT) calculations.
Each structure contains atoms corresponding to a catalyst's surface and an adsorbate. A catalyst is material used to increase the reaction rate and efficiency of a chemical reaction, and an adsorbate is a molecule involved in the chemical reaction that interacts with the catalyst. We evaluate models on four validation datasets that test different levels of model generalization: of Domain Adsorbate (OOD Adsorbate), OOD Catalyst, and OOD Both (both the adsorbate and catalyst's ss material are not seen training). Each split contains 1M examples. The Mean Absolute Error (MAE) of per-atom forces on free atoms is evaluated for each validation set. We will evaluate our models on the hidden test sets once that data made available.
Domain (ID), Out
For basis and activation functions, we use the spherical function and Swish activation by default, as this combination consistently provides the best results (see Figure 3 and Table in Appendix DD. For all experiments on force-centric GNNs (including the GNS model), we adopt the two strategies presented in Section 3.2] i.e., the rotation augmentation and loss reweighting. In Appendix D we perform ablation studies and show that both strategics are helpful. Our implementation is based on Pytorch (Paszke al.) 2019) and Pytorch Geometric (Fey & Lenssen 2019), and will be open-sourced All the implementation details and hyper- peperpparameters are provided in Appendix C
Performance comparison with existing models. In Table I. we compare ForceNet with existing GNN models: the energy-centric GNN models, SchNet (Schlitt et al. (017) and DimeNet (Klicpera) reported in (Anonymous) 2020). Note that Net performs message passing over triplets of nodes; thus, its model size was reduced to make training tractable. The GNS models adopted from Sanchez-Gonzatee
et al. 2020),
et al. (2020) based on feedback from the original author (see Appendix B for implementation details), where we set its model size to be comparable to ForceNet. The trivial bascline of always predicting force medians for all atoms is added to the table for reference.
The difficulty of the task is apparent in the errors of existing models. In all validation splits, the models' force MAEs are often far more than even half of that produced by the trivial median baseline. Contrary to the results in Klicpera et al. (2020). we observed that DimeNet performs worse than SchNet on OC20. We believe this is due to limits in considering all atom triplets for angular terms in the node updates, which are harder to fit in memory with growing graph and model sizes.
ForceNet models reduce the error significantly in both ID and OOD scenarios. Compared to the best existing model (the GNS model), our best ForceNet-large reduces the average MAE by 0.0139 ev/À (29.5% relative improvement). Comparing models with similar numbers of parameters we that ForceNet (with 11 ..4M parameters) reduces the average MAE of the GNS model (with 12.5M

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of ForceNet to existing GNN models. We mark as bold the best performance and close ones, i.e., within 0.0005 MAE, which according to our preliminary experiments, is good threshold to meaningfullyy distinguish model performance. The final row represents the relative error reduction of our best model (ForceNet-large) compared to the best existing model (GNS).
Validation Force MAE (eV/A)
Model
Width
Depth
#Params
ID
OOD Ads.
OOD Cat.
OOD Both
Average
Median
0.0757
0.0743
0.0744
0.0868
0.0778
DimeNet Klispera
2000
256
3
4.4M
0.0600
0.0579
0.0575
0.0702
0,0614
SchNet Schin otal D017
1024
4
7.4M
0.0438
0,0511
0.0458
0,0619
0.0507
GNS
768
5
12.5M
0.0419
0.0469
0.0430
0.0562
0.0470
NorceNet
512
5
11.3M
0,0314
0.0348
0.0336
0.0433
0.0358
ForceNet-large
768
7
34.8M
0.0281
0.0316
0.0318
0.0410
0.0331
Relative reduction of ForceNet-large
32.9%
32.7%
26.0%
27.1%
29.5%
Table 2: Ablations on the archi- tecture of ForceNet.
Table 3: The effect of model width D, depth K on ForceNet. All models are trained with a default batch size of 256, except the last row that uses 512.
Avg Force
Ablation
MAE (eV/A)
Validation Force MAE (eV/Á)
ForceNet
0.0358
Width
Depth
OOD Ads.
OOD
OOD Both
Average
(1) Only-dist
0.0698
No-atomie-radii
0.0368
512
5
0.0314
0.0348
0.0336
0,0433
0.0358
(3) No-node-emb
0.0412
768
5
0,0302
0.0338
0,0330
0,0427
0.0349
(4) Only-F;
0.0378
512
7
0.0304
0.0342
0.0330
0.0429
0.0351
(5) Edge-lincar-NN
0.0427
768
7
0.0296
0.0335
0.0325
0.0422
0.0345
Node-linea--NN
0.0357
768
7
0.0281
0.0316
0.0318
0.0410
0.0331
parameters) by 0.0112 eV/À (23.9% relative improvement). We also trained a shallower (depth of 3)
variant of ForceNet with 7.1M parameters for comparison against SchNet (7 (7.4M parameters). The average MAE is reduced by 0.0109 ev/À (21.5% relative improvement) over SchNet.
Effeet of basis and activation functions. We systematically study how choices of different basis and non-linear activation functions affect the model performance. We also compare with the "None" baseline, which does not use a basis function and directly concatenates the input edge feature into node embeddings. In Figure 3 we see that the combination of spherical basis and Swish activation performs the best. In Tables 5 of Appendix D we provide the full results, where we observe the consistent trends across the four validation sets. For comparison, the GNS model uses no basis function ("None") and ReLU, see Appendix B for full GNS model details.
Effect of architecture design. Next, we study the architectural building blocks of our conditional- filter-based message passing with fixed basis and activation functions. We consider six cases: (1) Only-dist: we remove n.tt from the input edge feature, i.e., Cst = Pst. resulting in the edge features being rotation invariant. (2) No-atomic-radii: we set the input edge features to ext =
Concat (n.t: [dyt Il) (atomic radii information is dropped), (3) No-node-emb: filter F. is function
of only ext (conditioning on source and target node embeddings (*) h is dropped), (4) Only-F,: Filter is directly aggregated, i.e., m& Fe, and self-message m, is omitted. (5) Edge-linear-BN: F. is replaced with a linear function followed by batch normalization, (6) Node-linear-BMy F, is replaced with linear function followed by batch normalization. Note that in (5) and especially (6), we find it critical to add batch normalization after the linear function to facilitate training.
Table 2 shows six ablation studies. Most notably, (1) is significantly worse than the rest, because irration-invariann node embeddings are insufficient for predicting ctaationcovvaiiat forces. We also see from (2) and (3) that making the filter less expressive, especially by dropping the dependency on node embeddings, significantly hurts performance. The improvement from element- wise product parameterization F.QFF. F is demonstrated in (4). From (5), we see that it is critical to utilize non-linear models for edge features, as atomic forces are highly dependent linearities are not essential for node features (6). Overall, our analysis suggests that ForceNet benefits most from its expressive edge computation via the conditional filter, which is directly responsible for accurately encoding the 3D neighborhood structure on which the atomic forces depend.
their subtle changes, but non-

---

demonstrates that Under review as a conference paper at ICLR 2021
Table 4: Percentage of relaxations that are successful, as measured by the Average Force below Threshold (AFbT) and Average Distance within Threshold (ADwT) for the same models as in Table The higher, the better. The final row represents the relative performance improvement of our best model compared to the best existing models.
D
AFbT (%)
ADwT (%)
Model
ID
OOD Ads.
OOD Cat.
OOD Both
Average
ID
OOD Ads.
OOD Cat.
OOD Both
Average
DimeNet
0.00
0.00
0.00
0.00
0.00
10.94
9.80
8.36
10.21
9.82
SchNet
6.17
3.38
3,40
2.44
3.84
35.68
34.11
29.42
33,80
33,23
GNS
2.96
1.67
2,01
2.15
2.20
39.65
38.35
35.05
40.37
38.30
ForceNet
12.49
11.21
8.75
8.70
10.34
51.43
49.36
47.66
54.17
50.59
FurceNet-large
19.74
16.06
13.78
13.30
15.82
54.32
52.17
49.68
58.38
53.47
Improveneet
219.9%
375.1%
405.3%
445.1 %
312,0%
37.0%
36.0%
41.7
44.6
39.6%
Effect of model size. We try different model sizes for ForceNet, in terms of both depth K and width D. As shown in Table 3 simply increasing the scale of the model improves performance significantly (0.0314 parameters, 3.5 days to train on 64 GPUs) with a doubled batch size (512), and observe further gains (0.0296 - 0.0281 MAE val ID). While further scaling may lead to further performance gains, it increases inference time significantly. ForceNet-large is ~3 times slower than the base model.
0.0296 MAE on val ID). We also train variant of our largest model (~35M
Are ML models practically useful for approximating DFT structure relaxations? We use ForceNet, as well as GNS, DimeNet, and SchNet to perform ML relaxations (replacing the DFT computed forces with those computed using the ML models). The goal is to find relaxed structure of molecules, i..., 3D structure with minimum energy. Qualitatively. Figure D ForceNet is able to successfully perform relaxations rrders-of-magniuue DFT-based simulation. Individual DFT relaxations in OC20 take numerous hours on multi-core CPUs, while ML models take 1.36 (SchNet), 1.23 (GNS), 1.70 (ForceNet) and 3.81 (ForceNet-large seconds per relaxation to perform on a single GPU (> 10 faster). DFT can be made faster with GPUs, but speed ups are significantly less (2-5x (Maintz & Wetzstein 2018)).
faster than the ground-truth
Quantitatively, we use the same set of evaluation metrics as Anonymous 2020). The ML-relaxed structures are evaluated with DFT to determine whether they represent true relaxed structures. If true relaxed structure is found, the DFT-cooputed forces on it should all be close to zero (local energy minimum). Specifically, we consider the strict metric-the percentage of relaxed structures where all the per-atom DFT computed forces are below a Threshold (FbT). The forces of the ML-predicted relaxed structures (500 from each val split, 2000 total) are evaluated with DFT (see Appendix E for the detailed procedure). We report the average FbT over thresholds ranging from 0.01 to 0.4 eV/À. From Table we see that on average, our best model (ForceNet-large) is roughly 4.1 x more accurate (relative improvement of 312.0%) than SchNet and 7.2x more accurate (relative improvement of 618.1 1 1)) than the GNS model. For an FbT at a strict threshold of 0.05, an indication of practical usefulness, all existing models score 0%, while ForceNet shows small and very modest success (0.4% on validation ID dataset). Following Anooymouus (2020), we also report a surrogate metric that does not require expensive DFT calculations, i.e., the percentage of structures with atom distances within a threshold of the ground truth averaged across thresholds (ADwT, 0.01 to 0.5Å). From Table we see the relative improvements in ADwT is 39.6% on average.
CONCLUSION
Cheap, durable and efficient catalysts are an essential component to addressing the world's rising energy needs while reducing climate change. The efficient approximation of DFT for use in atomic structure relaxations has the potential to significantly increase the rate at which new catalysts can be discovered. Even moderate improvements in accuracy can lead to significant simulation efficiency gains, since ML relaxations are often 1000's of times faster than traditional DFT relaxations.
In conclusion, we demonstrate that force-centric GNN model with carefully designed message passing can model complex atomic interactions. significant improvement in accuracy is achieved over prior results and the results are promising when applied to structure relaxations. Given the size of the OC20 dataset, further model scaling may show further improvement, well as, incorporating more domain specific information in the model architecture.

---

Under review as a conference paper at ICLR 2021
REFERENCES
The atomic simulation environment-a python library for working with atoms. Journal of Physics Condensed Matter, 2017. ISSN 1361648X. doi: 000:1000.88/1133166463A2aaaa 361-648X/aa680e..
Anonymous. The Open Catalyst 2020 (OC20) dataset and community challenges. See supplementary material, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 2016.
arXiv preprint
Victor Bapst, Thomas Keck, A Trabska-Barwinsske Craig Donner, Ekin Dogus Cubuk, SS Schoen- holz, Annette Obika, AWR Nelson, Trevor Back, Demis Hassabis, et al. Unveiling the predictive power of static structure in glassy systems. Nature Physics, 16(4):448-44 454, 2020.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems (NIPS), pp. 4502 4502-4510, 2016.
Marc Brockschmidt. Gnn-film: Graph neural networks with feature-wise linear modulation. In International Conference Machine Learning (ICML), pp. 2014-20233 2020.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.022 2019.
Justin Gilmer, Samuel Schoenholz, Patrick Rilcy, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference Machine Learning (ICML), pp. 1273-1272, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 249-256, 2010.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (7Xxv:/706.02666 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp. 448-456, 2015.
Matthew Jouny, Wesley Luc, and Feng Jiao. High-rate electroreduction of carbon monoxide to multi- carbon products. Nature Catalysis, 2018. ISSN 25801158. doi:
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning (ICML), 2018.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference Learning Representations (ICLR), 2020.
Georg Kresse and Jürgen Furthmüller. Efficiency of ab-initio total energy calculations for metals and semiconductors using plane-wave basis set. Computational Materials Science, 6(1):15-50, 1996a. ISSN 09270256. doi: 10. 1011016099700556660000880

---

Under review as a conference paper at ICLR 2021
Georg Kresse and Jürgen Furthmüller. Efficient iterative schemes for ab initio totalyenergy calculations using a plane-wave basis set. Physical Review B, 54(16):1169-111186 1996b. ISSN 0163-1829. doi: 10. 10.1103/hhyReev.554.111
Georg Kresse and Jürgen Hafner. Ab initio molecularddnammis simulation of the liquid- transition in germanium. Physical Review B, 49(20):14251
14269, 1994. ISSN 0163-1829. doi: 10.1 10.1103/PhysRev..49... ....bBB.9.1425111
Thomas Murray MacRobert. Spherical harmonics: elementary treatise on harmonic functions with applications. 1947.
Stefan Maintz and Markus Wetzstein. Strategies to accelerate vasp with gpus using open acc. Proceedings of the Cray User Group, 2018.
Robert G Parr. Density functional theory of atoms and molecules. In Horizons of quantum chemistry, pp. 5-15. Springer, 1980.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high- begh-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Swish: self-gated activation function. arXiv preprint arXiv:1710.0594 7, 2017.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177, 2014.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W Battaglia. Learning to simulate complex physics with graph networks. International Conference Machine Learning (ICML), 2020.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmicla, Alexandre Tkatchenko, and Roburt Müller. Schnet: continuous- filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems (NIPS), pp. 991-1001, 2017.
Zhi Wei Seh, Jakob Kibsgaard, Colin F Dickens, Ib Chorkendorff, Jens Nørskov, and Thomas Jaramillo. Combining theory and experiment in electrocatalysis: Insights into materials design. Science, 355(6321), 2017. ISSN 0036-8075. doi: 10. 1001112//cience.aaa aad4998. URL https wttg://oneee///////555550 /aad4499/
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
Devin T Whipple and Paul J A Kenis. Prospects of CO2 Utilization via Direct Heterogeneous Electrochemical Reduction. The Journal of Physical Chemistry Letters, 1(24):34513444 2010. doi: 10. 10.1022//210112277. BRLhttps::/ doi. www.10..0..0..........o 1tt//10.1102///111123333
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019,
10

---

Under review as a conference paper at ICLR 2021
Atom neighbors
Center ccll
Neighboring cells
Free cotalyst stoms
63.000
10 d)
0,00,0
Figure 4: 2D Illustration of slab that represents catalyst's surface and adsorbate. The slab is tiled in the x and y directions to create the surface (neighboring cells shown as atoms with dashed outlines). Only the cells to the left ([-1, 0, 0) and right (1, O, 0) are shown. The adsorbate is also assumed to be tiled with the slab (white, red, and grey atoms). Only the top 2 layers the slab are allowed to atoms (black arrows) can be from the same cell or neighboring cells (t and s'). All atoms within radius (dotted circle) are assumed to be neighbors.
during relaxation (dark blue), and the others fixed (light blue). Neighboring
A DESCRIPTION OF OC20 DATASET
The OC20 dataset contains over 130 million atom structures for training. The structures are from the trajectories (the movement of the atoms from the initial state to relaxed state) contained in 650,000 relaxations. In practical scenarios, the relaxed structure is useful in determining a chemical reaction's rate, i.e., whether it will produce enough of the desired product to be useful. When applied to energy applications, such as storage, efficient methods for estimating reaction rates could have significant impact energy scarcity and climate change.
Each structure contains the 3D positions of atoms an adsorbate and catalyst slab, Figure The adsorbate is molecule involved in the chemical reaction that interacts with the catalyst's 's surface. The adsorbate contains to atoms. The catalyst represented as a "slab" that repeats infinitely in the x and directions. The slab structure is repeated in a grid pattern where each repetition is referred as a "cell". The center cell has coordinate (0, 0, 0) with the cell to the left right being (-1. 0, 0) and (1, 0, 0) respectively. The slab is not repeated in the z direction. Instead, the atoms at the bottom of the slab
assumed be fixed and not move during relaxation, which approximates how they would be held in place by the catalyst's s atoms below the slab. Typically, only the top two layers the catalyst's surface assumed to be free and are moved according to their forces during relaxation (see Figure Therefore, forces are only evaluated on free catalyst atoms and the adsorbate.
The forces in the same atom in different cells is assumed to be equal, since the atom neighbors also identical. Thus, we can assume the GNN node hidden states are also identical. When computing the neighborhood of an atom, atoms in neighboring cells need to be taken into consideration (atom t F.gure The node embedding of atoms marked s and in Figure are the same. However, the messages and e,'t are different since the distance between and s is not the same and s'. Computing the distances between atoms from different cells can be done using the supplied information in the OC20 dataset for periodic boundary conditions.
B DETAILS OF GNS MODEL
For the results in this paper, we reimplemented the original GNS model SSanchez-Gonzaler et al.) 2020). Since the public code for the original GNS model (Sanchez-Gonzales et al.) (020) was not available the time of our experiments, we communicated with one the authors to confirm the implementation details.
The message in a GNS model defined as
m(h Cat, h2)) = MLP (Concat (no ezc. 12 hn))

---

Under review as a conference paper at ICLR 2021
Table Ablations on basis and activation functions in the ForceNet architecture.
Validation Force MAE (eV/A)
Basis
Activation
ID
OOD Ads.
OOD Cat.
OOD Both
Average
Spherical Spherical
ReLU
0.0323
0.0366
0.0344
0.0452
0.0371
Swish
0.0314
0.0348
0.0336
0.0433
0.0358
Sine
ReLU
0.0325
0.0369
0.0348
0.0457
0.0375
Sine
Swish
0.0314
0.0355
0.0338
0.0443
0.0363
Gauss
ReLU
0.0334
0.0382
0.0357
0.0472
0,0386
Gauss
Swish
0.0317
0.0360
0.0344
0.0452
0.0368
Linear+Act
ReLU
0.0335
0.0380
0,0350
0,0462
0.0382
Linear+Act
Swish
0.0317
0.0357
0.0337
0,0440
0.0363
Identity
ReLU
0.0337
0.0381
0.0354
0.0465
0.0384
Identity
Swish
0.0322
0.0365
0.0340
0.0446
0.0368
None
ReLU
0.0363
0.0424
0.0380
0.0514
0.0420
None
Swish
0,0330
0.0373
0.0348
0.0457
0.0377
where (LP(.) is a I-hidden-layer MLP with ReLU activation and layer normalization (Ba et al. 2016) applied before the activation. For aggregating the message, GNS used either mean sum, so we tried both in our experiments. We found sum aggregation to perform better, and report results of sum aggregation in Table T After the messages are aggregated, GNS uses a learnable linear function transform the node embeddings. Similar ForceNet, we additionally apply batch normalization on the node embeddings, which alleviates training instability and significantly improves performance. GNS uses residual connection, where the computed node embeddings are added into the node embeddings from the previous layer. For the decoder, the GNS model uses a I-hidden-layer MLP with ReLU activation. All the node embeddings and hidden units the MLPs have the same dimensionality.
C HYPER- PAPER-PARAMETERS
For training, we use the Adam optimizer (Kingma Ba 2015). with an initial learning rate 0.0005. By default, we set the hidden dimensionality D (i.e., width) to 512, number of layers (i.e., depth) to 5, batch size 256. We train all the models for 500K iterations, which is equivalent the entire datase(" and takes 3 to 4 days on 16 Tesla V100 GPUs. All the parameters are initialized with Xavier uniform initialization (Glorot & Bengio 2010). The learning rate kept constant for the first 250K iterations, after which it is halved every 50K iterations. For batch sizes of 512, we double the initial learning rate (Goyal al., 2017) and keep constant for the first 150K steps, after which is halved every 50K steps. We use the checkpoint with the best validation ID performance, and evaluate the saved model over all four validation
epoch for
MAE over forces is used for the training loss. We will evaluate our models on the hidden test sets once the test server ready.
For Gaussian and sine basis functions, B 350. For Act, we set B 350. For spherical basis, we use L=3 3 and = 4, which results in B 36(= 3 4), and we set J 50 for the internally used sine basis function. For encoding the input atomic node features, we first normalize each dimension to between and 1, and adopt the same basis function as used for encoding the edge features. The exception is spherical basis that is specialized for 3D spaces, instead sine basis used. We find that increasing J and L beyond the above values does not improve the performance, while significantly decreasing them worsens the performance.
use J 50, which gives an output dimensionality of
D FULL ABLATION RESULTS
Here we provide detailed results of our ablation studies on ForceNet design as well as its training strategies.
We do not observe much gain by training models longer than epoch. This is probably because of redundancy in data, i.e., out of 130M data points, there are 650k unique atom configurations (ignoring the positional differences).
12

---

Under review as a conference paper at ICLR 2021
Table Ablations on the message passing architecture ForceNet.
Validation Force MAE (eV/A)
Ablation
ID
OOD Ads.
OOD Cat.
OOD Both
Average
ForceNet
0.0314
0.0348
0.0336
0.0433
0.0358
(1) Only-dist
0.0659
0.0670
0.0661
0.0802
0.0698
(2) No-atomic-radii
0.0319
0.0362
0.0342
0.0448
0,0368
(3) No-node-emb
0.0362
0.0413
0.0375
0,0498
0,0412
(4) Only-F.
0.0334
0,0371
0.0351
0.0455
0.0378
Edge-linear-BN
0.0373
0.0427
0.0391
0,0520
0.0427
(6) Mode-lincca--e
0.0315
0.0349
0.0334
0.0431
0.0357
Table 7: The effect of rotation augmentation and different weighting coefficients on fixed atoms while training ForceNet.
Rotation aug.
Weight on fixed atoms
Validation Force MAE (eVIÂ)
Model
ID
OOD Ads.
OOS Cat.
OOD Both
Average
ForceNet 0.05
0.0314
0.0348
0,0336
0,0433
0.0358
ForceNet
0.05
0.0318
0.0359
0.0343
0.0448
0.0367
ForveNet
0.0367
0,0413
0.0388
0.0507
0.0418
ForceNet
<
0.0324
0.0375
0.0340
0.0455
0.0374
Full Results on ForceNet Designs First, we provide the full ablation results on ForceNet designs for each of the four validation sets. Table [5) shows the full ablation results on basis and activation functions, while Table 6 shows the full ablation results on the message passing architectures.
Overall, observe the trends that consistent with the averaged results in Figure and Table 2 Specifically, from Table 5) we see that the combination of spherical basis functions and the Swish activation results in the best performance across the four validation sets. From Table 6 the conditional filter convolution design gives superior performance compared to the more simplified architectures, except for (6) node-linear-BN. in which case the performance is comparable.
that
Effeet of Training Strategies Next, we provide results of ablation studies on our training strategies presented in Section 3.2 for fixed model and optimization procedure. For the model, we ForceNet with depth 5, width 512, with the spherical basis function and the Swish activation function. For optimization, we use the default configuration presented in Section
The results are shown in T First, we see that rotation augmentation helps, especially for the three out- out-of-distribution validation sets. Second, we see that providing small reweighted supervision on fixed atoms is also helpful, significantly improving the validation performance (evaluated on free atoms) compared to the two baseline strategies: (1) uniform loss weighting (equally weighting the losses for fixed and free atoms) and (2) zero-loss weighting (ignoring losses fixed atoms during training).
E DETAILS OF STRUCTURE RELAXATION
Structure relaxations are performed using a PyTorch implementation the Atomic Simulation Environment's (ASE) (Hjo (017g) L-BFGS optimizer. Relaxations were terminated when absolute per-atom force of 0.01 eVi or 200 simulation steps, whichever comes first. All DFT caleulations were performed in the Vienna Ab Initio Simulation Package (VASP) (Kresse Hafner (1994); Kresse & Furthmüller (1996a b)), all of which are popular packages within the computational chemistry and catalysis communities.
13


Reviews:
[{"review_id": "gxlJtUhc46", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper proposes ForceNet which is capable of capturing highly complex and non-linear interaction of atoms in 3D molecular space in order to accurately predict atomic forces. On the outset, the ForceNet framework seems extension of recent work on physics-based simulator for fluid dynamics using deep network called GNS (Sanchez-Gonzalez et al, 2020) but with carefully designed message passing network. Unlike GNS, the ForceNet computes messages using specifically designed continuous filter convolution. Inspired by SchNet, the filter weights in continuous filter are dynamically computed by first lifting edge feature input to new basis space followed by multi layer MLP. However, unlike SchNet, the filter computation is conditioned on the source and target node features as well as the 3D angular information.\n\nApart from this, ForceNet deploys spherical harmonic basis functions and smooth activation function - Swish, to realised better accuracy. Experimental evaluation on large-scale catalyst dataset OC20 shows that ForceNet reduces force estimation error by 30% in comparison to recently popular ML models for quantum chemistry simulation. When employed for quantum chemistry simulation, wherein DFT forces are replaced with ForceNet computed forces, it converges to similar energy structure as DFT in $10^3 \\times$ lesser time.\n\n**Quality**\nThe paper is very well written and easy to follow. The description of their model, experimental setup, comparison to existing baselines and their architectural choices are precisely described (as well as evaluated). Although their largest but best model is $3\\times$ slower than existing work,  it produces highly accurate force prediction.\n\n**Originality**\nAs mentioned above, this work combines best of two world proposed for quantum chemistry simulation in the past - GNN based framework and continuous convolution. From the outset, the architecture seem to me simple but effective extension of SchNet work with modifications in interaction layer such as,\n1. Use of directional vector and atomic radii for feature input to basis function instead of mere scalar distances.\n2. Convolution filter generation conditioned on source and target node features\n3. Inclusion of self-message $m_t$ and scalar multiplier $\\alpha$ and\n4. Different basis and activation function\n\nForceNet achieves this by trading off rotation-invariant property while leveraging random rotation data augmentation to achieve rotation covariant.\n\n**Significance**\nThe carefully designed message passing computation will motivate future work to design better layers for targeted application. Moreover, the speed + accuracy achieved on DFT task is very encouraging for future deployment of ML model for quantum chemistry simulation.\n\n**Clarity**\n1. Like previous works, can you compare errors w.r.t. energy output ?\n2. SchNet uses shifted softplus as smooth activation. Ofcourse the motivation there was twice differentiability. Have you experimented ForceNet with shifted softplus as well as other non-smooth activation such as Leaky ReLU etc. ?\n3. For Table 2, can you share results obtained by only omitting $m_t$.\n4. Have you tried spherical bessel basis as in DimeNet ? Comparison to this was missing under basis function choices.", "extra_scores": {}, "extra_text": {}}, {"review_id": "c3-FpyPD-Lu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents ForceNet, a graph neural network, for estimating per-atom forces based on the 3D molecular structure. The authors argue that force-centric learning could be better in terms of prediction accuracy compared to energy-centric learning. As such, they focused on force-centric simulation and did several simple, but effective optimizations to make their model more accurate and easier to scale to large size. Experiments validate the effectiveness of ForceNet.\n\nOverall, I think it is a nice application paper. It is easy to follow and gives clear reasoning behind their designs. Though many of the design options look simple and may not be strong enough from a machine learning perspective, these designs are tailored towards\nlarge-scale quantum chemistry simulation and would be nice to showcase the potential of machine learning models, especially GNNs, to these application domains. From this perspective, I am happy with its contribution.\n\nQuestions:\n1. Would you make your OC20 dataset open-source upon acceptance? This would be a big plus for this paper, as it will promote more ML experts to pursue this field.\n2. Section 3.1.3 is a little bit vague. Why would Swish be a good choice for the activation function? Little background and insights were given.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z2ewseZK0HR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a neural network force field that predicts atomic forces directly. This has the benefit of not requiring to differentiate an energy model and may be more flexible. The approach is well motivated and the paper is well structured and written. A strong point of the paper is the extensive discussion of model design choices.\n\nA major weakness of the model is its lack of rotational covariance. In the presented application to relaxation of molecules on surfaces, this might be acceptable (since rotation only occurs in 2d here and relaxation path usually do not have a lot of structural variance), but for more flexible systems this will certainly lead to problems. I doubt that data augmentation can make up for this and the deviation of force predictions under rotation should be shown. In particular for MD simulations this might be a deal breaker, since the resulting model might not be energy conserving. For these reasons, the proposed approach might be of limited practical use beyond the demonstrated application. To prove otherwise, additional experiments with more flexible systems would be required.\n\nFinally, the authors state: \"However, practical models for real large-scale and complex problems remain out of reach.\" This is a bit of an overstatement as there exist ML force field approaches for practical problem ranging from nuclear quantum effect over reactions and spectroscopy to thermodynamics with thousands of atoms (e.g. refer to Table 1 in https://arxiv.org/pdf/2010.07067.pdf for an overview). For the particular problem of structure relaxation with ML, there is previous work by Shapeev et al (Phys. Rev. B, 2019) on accelerating crystal structure prediction.\n\nPros\n------\n- better and faster on the task than previous models\n- extensive model selection and ablation studies\n\nCons\n-------\n- not energy conserving / rotational covariant\n- thus probably limited applicability -> evaluate on a dataset with more variance and 3d rotation, e.g. MD trajectories\n\n\nUpdate:\nI read the response of the authors.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KxBPhdn-j08", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\nThis article presents the force fields predictor ForceNet. The authors show that ForceNet reduces the estimation error of atomic forces by 30% compared to existing ML models. Reconstructing molecular force fields is a very active and thriving field, hence this paper is highly relevant. In short, this article presents a very good model based a selection of very good methodologies and combining them in a clever way.\n\n##########################################################################\nReasons for score: \nOverall, I vote for not accepting, but if the authors address most of the comments I don't have a problem if it is accepted. As mentioned before, this problem is of high relevance and is likely to have a considerable impact in the physics and simulations community. Nevertheless, this is a non-energy conserving and non-rotationaly covariant force field, meaning that for most of the applications of a force field, this will provide unphysical results.\n\n##########################################################################\n\nPros:  \n1. ForceNet reduces the estimation error of atomic forces by 30% compared to existing ML models.\n \n2. It is based on strong and reliable previous results, which makes it a robust model.\n \n##########################################################################\nComments: \n\n-The authors state ”However, if successful, accurate and fast ML-based models may lead to significant practical impact by accelerating simulations from O(hours-days) to O(ms-s), which in turn accelerates applications such as catalyst discovery.” And “If successful, ML could be applied to problems such as catalyst discovery which is key to solving many societal…” These statements hint that this has not been done yet, which is a half truth. As exposed in the references cited in the paper, there are already very successful ML-methodologies in this field, but it has to be stressed that the successes come from applications in local domains. This mean that errors of sub-0.1 kcal/mol have been achieved for molecules already couple of years ago (Chmiela et al Nat Commun 9, 3887 (2018), Christensen et al J. Chem. Phys. 152, 044107 (2020), and Unke et al J. Chem. Theory Comput. 2019, 15, 6, 3678–3693). Hence, phrasing the manuscript like this, it could be misleading for the reader.\n-The authors state “This is possibly because the force-centric models capture the dependency of atomic interactions on atomic forces more explicitly than the energy-centric models.” This topic has been extensively analysed by Prof. Müller’s group (Sci. Adv. 3 (5), e1603015, 2017; Nat Commun 9, 3887 (2018)) and recently by Prof. von Lilienfeld (Mach. Learn.: Sci. Technol. 1 045018, 2020). In short, learning forces is equivalent to learn linearisation of the energy surface which is much more informative.\n-MLP is not defined prior usage.\n-The directed message e_{st} is a vector or scalar? In some parts appeared just as e_{st} and on other as \\textbf{e}_{st}. \n-The atomic radii is a given physical value or is it a learning parameter?\n-The authors state: “As we demonstrate in Section 4, the replacement of ReLU with Swish consistently and significantly improves the predictive accuracy while maintaining scalability across all choices of basis functions.” This is not relevant, since it is a direct statement. A more interesting comparison would be to shifted tanh function.\n\n-From my point of view, the main downside of this article is the fact that the ForceNet is neither exactly covariant not energy conserving.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The model presented here may be of use to others in running quantum chemistry simulations, and it may well lead to new advances, but the authors did not sufficiently address the key concerns around the model not being energy conserving and rotation covariant. The approach proposed could be learning such physical rules, and the authors in their general response provide some very preliminary evidence for this, but a much more thorough discussion with a full range of experiments is needed. ICLR is a broad conference where non-experts who may have never heard of DFT simulations must parse this work and decide on if/how to follow up. This is a critical missing piece for anyone that wants to do so.

The above is exacerbated by the fact that the work is not well-situated against prior work as pointed out by two reviewers. Together these two issues conspire to make understanding this model, the contribution of the work, and what followup is possible, untenable for an ICLR reader. For example, one would not be able to surmise from the manuscript and its brief discussion of rotation covariance that this is likely to result in ForceNet having limited applicability to other DM problems; which one reviewer pointed out and the authors generally agreed with. While the authors respond that perhaps the architecture itself may be useful for other applications, why this would be and what the specific advantage of the current model relative to the state of the art in those fields is unclear.

Decision: Reject