GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets
Authors: Johannes Gasteiger
Muhammed Shuaibi
Anuroop Sriram
Stephan Günnemann
Zachary Ward Ulissi
C. Lawrence Zitnick
Abhishek Das
Venue: Accepted by TMLR ()

Abstract: Recent years have seen the advent of molecular simulation datasets that are orders of magnitude larger and more diverse. These new datasets differ substantially in four aspects of complexity: 1. Chemical diversity (number of different elements), 2. system size (number of atoms per sample), 3. dataset size (number of data samples), and 4. domain shift (similarity of the training and test set). Despite these large differences, benchmarks on small and narrow datasets remain the predominant method of demonstrating progress in graph neural networks (GNNs) for molecular simulation, likely due to cheaper training compute requirements. This raises the question -- does GNN progress on small and narrow datasets translate to these more complex datasets? This work investigates this question by first developing the GemNet-OC model based on the large Open Catalyst 2020 (OC20) dataset. GemNet-OC outperforms the previous state-of-the-art on OC20 by 16% while reducing training time by a factor of 10. We then compare the impact of 18 model components and hyperparameter choices on performance in multiple datasets. We find that the resulting model would be drastically different depending on the dataset used for making model choices. To isolate the source of this discrepancy we study six subsets of the OC20 dataset that individually test each of the above-mentioned four dataset aspects. We find that results on the OC-2M subset correlate well with the full OC20 dataset while being substantially cheaper to train on. Our findings challenge the common practice of developing GNNs solely on small datasets, but highlight ways of achieving fast development cycles and generalizable results via moderately-sized, representative datasets such as OC-2M and efficient models such as GemNet-OC. Our code and pretrained model weights are open-sourced.

Published in Transactions on Machine Learning Research (09/2022)
GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets
Johannes Gasteiger
johannex.gasteiger@tmm.
Technical University of Munich Work partially done during on internship FAIR, Mela AL.
Muhammed Shuaibi
mshuaaiilif.. com
Carnegie Mellon University FAIR, Meta AI
Anuroop Sriram FAIR, Meta Al
anuroops@ft.ccom com
Stephan Günnemann
stephann.uuerneeeoonn@tmm..
Technical University of Munich
Zachary Ulissi Cornegie Mellon University
zalissi@endrew.com
C. Lawrence Zitnick FAIR, Meta AI
zitmickkffb.co com
Abhishek Das FAIR, Meta Al
abhshkrd@@f..com
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= u8tuSon4Bs
Abstract
Recent years have the advent of molecular simulation datasets that are orders of magnitude larger and more diverse. These new datasets differ substantially in four aspects of complexity: 1. Chemical diversity (number of different clements), 2. system size (number of atoms per sample), 3. dataset size (number of data samples), and 4. domain shift (similarity of the training and test. set). Despite these large differences, benchmarks on small and narrow datasets remain the predominant method of demonstrating progress in graph neural networks (GNNs) for molecular simulation, likely due to cheaper training compute requirements. This raises the question does GNN progress on small and narrow dalasels translate to these more compler datasets? This work investigates this question by first developing the GemNet-OC model based on the large Open Catalyst 2020 (OC20) dataset Chanussou et al. 2021). GemNet-OC outperforms the previous state-of-the-art on OC20 by 16% while reducing training time by a factor of 10. We then compare the impact of 18 model components and hyperparameter choices on performance in multiple datasets. We find that the resulting model would be drastically different depending on the dataset used for making model choices. To isolate the source of this discrepancy we study six subsets of the OC20 datasct that individually test cach of the abovo-mentioned four datasct aspects. We find that results on the OC-2M subset correlate well with the full OC20 dataset while being substantially cheaper to train on. Our findings challenge the common practice of developing GNNs solely on small datasets, but highlight ways of achieving fast development cycles and generalizable results via moderately-sized, representative datascts such as OC-2M and efficient models such as GemNet-OC. Our code and pretrained model weights are open-sourced at github. cwt.:/ppnnnattalaass-ooopeee/o/e

---

Published in Transactions on Machine Learning Research (09/2022)
1 Introduction
Machine learning models for molecular systems have recently experienced leap in accuracy with the advent of graph neural networks (GNNs) (Gilmer et al. 2017 Schütt et al. 2017 Gasteiger et al. 2020b Batzner
ct al.
2021 Ying et al. 2021). However, this progress has prodominantly been demonstrated on datascts covering a limited set. of systems (Ramakrishaan et al. 2014) sometimes even just single molecules (Chmiela iet al. 2017]. To scale this success to large and diverse atomic datasets and woal-world chemical experiments, models need to demonstrate their abilitics along four orthogonal aspects: chemical diversity, system size, dataset size, and test set diffieulty. The critical question then becomes: Do model improvements demonstraled on small and limited datasets generalize to large and diverse molecular systems?
Some results suggest that they should indeod generalize. GNN architectures for molccular simulations usually generalize between systems and datasets (Unke & Meuwly 2019 Moreover, models have been found to scale well with training set size (Bornschein et al., 2020). However, other works found that model changes can indeed break this behavior, albeit positing that this is limited to singular model properties (Batzner et al,) 2021). It is generally accepted that certain hyperparameters should be adapted to each dataset, e.g. the learning rate, batch size, embedding size. However, there is qualitative reason why only these "special" properties would be affected by the underlying datasct, while other aspects of model components would be unaffected. Changes in model trends wetween datasets have already been seen in other fields, e.g. in computer vision (Kornblihh et al. 2019).
In this work, we set out to directly test the performance of model components and hyperparametors between datasets. We focus on changes around one baseline model instead of separate GNN architect since this most closely matches the fine-graid ane-grained changes that are typical for model development. We develop the GemNet-OC bascline model based the Open Catalyst 2020 dataset (OC20) Chanussot et al., 2021. which is the largest molecular dataset to date. The resulting GemNet-OC model is 10 times cheaper to train than previous models and outperforms the previous state-of-the-art by 16 %. Tt incorporates optimizations that enable the calculation of quadruplet interactions and dihedral angles even in large systems, and introduces an interaction hierarchy to better model long-range interactions.
We analyze these new model components, as well as previously proposed components and hyperparameters, on two narrow datasets (MD17 (Chmiela et al. 2017) and COLL (Gasteiger et al. 2020a)), ), the large-scale OC20 dataset, and six OC20 subsets that isolate the effects of the aforementioned four dataset properties. We find that model components can indeed have substantially different effects on different datasets, and that all four dataset propertios can cause such differences. Large and diverse datasets like OC20 thus posc a learning challenge that is qualitatively different from the smaller chemical spaces and structures represented in most prior molecular datasets.
Testing a model that can generalize well to the large diversity of chemistry thus requires a sufficiently large and diverse dataset. However, model development on large datasets like OC20 is excessively expensive due to long training times. As case in point, the analyses and models presented in this work required more than 16 000 GPU days of training overall. To solve this issue, we identify small data subset on which model trends correlate well with OC20: OC-2M. We provide multiple baseline results for OC-2M to enable its use in future work. Torether, GemNet-OC and OC-2M enable state-of-the-art model development. even on a single GPU. Our code and pretrained model weights are open-sourced at gwithb..co//ppeeeeeee http:DDpe--aaaayyytt-roooeeddeee Concretely, our contributions are:
We propose GemNet-OC, GNN that achieves state-of-the-art results all OC20 tasks while training is 10 times faster than previous large models.
Through carefully-controlled experiments
variety of datasets, we demonstrate the discrepancy between modeling decisions on small and limited vs. large and diverse datasets.
We identify OC-2M, a small data, subset that provides generalizable model insights, and publish a range of baseline results to enable its use in future work.
2

---

Published in Transactions on Machine Learning Research (09/2022)
2 Background and related work
Learning atomic interactions. A model for molecular simulations takes N atoms, their atomic numbers = (E1....,,N]] and positions X RNx3 and predicts the system's energy E and the forces F E NR acting on each atom. Note that these models usually do not use further auxiliary imput information such as bond types, since these might be ill-defined in certain states. The force predictions are then used in a simulation to calculate the atoms' accelerations during a time step. Alternatively, we can find low-energy or relaxed states of the system by performing geometric optimization based on the atomic forces. More concretely, the atomic forces F are the energy's 8 gradient, ie. F; = -- E. The forces are thus conservative, which is important for the stability of long molecular dynamics simulations since ensures energy conservation and path independence. However, in some cases we can ignore this requirement and predict forces directly, which considerably speeds up the model (Park et al. 2021 Classical approaches for molecular machine learning models use hand-crafted representations of the alomic neighborhood (Bartók et al. 2013) with Gaussian 2017) or neural networks (Behler & Parrinello 2007 Smith et al. 2017). However, these approaches have recently been surpassed consistently by graph neural networks (GNNs), in both low-data and large-data regimes [Batzner et al., 2021 Gasteiger et al. 2021).
processes [Bartók et al.) 2010 2017 Chmieea et al.
Graph neural networks. GNNs represent atomic systems as graphs g (V,E), with atoms defining the node set V. The edge set is typically defined as all atoms pairs within a certain cutoff distance, e.g. 5A. The first models resembling modern GNNs were proposed by (Speruuii & Starita (1997 Baskin et al. (1997). However, they only became popular after multiple works demonstrated their potential for wide
range of graph-related tasks (Bruna et al.) 2014 Kipf & Welling, 2017 Gasteiger et al. 2019 Molecules have always been a major application for GNNs (Baskin et al., 1997 Duvenaud el al. 2015) 5), and molecular
simulation is no exception. Modern examples include SchNet (Schütt et al. 2017). PhyssNet (Unke & Meuwly
2019), Cormorant (Anderson et al. 2019), DimeNet (Gasteiger ct al. 2020b). PaiNN (Schiitt et al.] 2021) ),
SpookyNet. (Unke et al.] 2021), and SpinConv (Shuaibi et al.] 2021). Notably, MXMNet (Zhang et al.) 2020)
proposes two-level message passing scheme. Similarly, Alon & Yahav (2021) propose global aggregation
mechanism to fix bottleneck caused by GNN aggregation. GemNet- OC extends these two-level schemes to multi-level interaction hicrarchy, and leverages both edge and atom embeddings.
Model trends between datasets. Previous work often found that models scale equally well with training set, size (Hestness et al. 2017/. This lead to the hypothesis that model choices on subsets of a dataset translate well to the full dataset [Bornschein et al. 2020). In contrast, we find that model choices can have different effects small and large dalasets, which implies different scaling for different model variants. Similarly, [Biiato & locchi (2020) found that simple models work better than caae-offthhearte architectures in the extremely small dataset regime, and Kornblith et al. (2019) observed differences in model trends between datasets for computer vision. Note that we observe differences for subsets of the same dataset, and our findings do not require a dataset reduction to a few dozen samples - we observe qualitative differences even beyond 200 000 samples.
3 Datasets
Datasets for molecular simulation. Training a model for molecular simulation requires a dataset with the positions and atomic numbers of all atoms for the model input and the forces and energy as targets for its output. The energy is either the total inner energy, i.c the energy needed to separate all electrons and muclei, or the atomization energy, iee. the energy needed to separate all atoms. Note that the energy is actually not necessary for lcarning a consistent force field. Most works weight the forces much higher than the energy or even train only on the forces.
Many molecular datasets have been proposed in recent years, often with the goal of supporting model develop- ment for specific use cases. Arguably the most prominent, publicly available datascts are MD17 (Chmicla et al. 2017), IS017 (Schütt et al. 2017), Sx2 (Unke & Meuwly 2019), ANII-x (Smith et al. 2020), QM7-X (Hoja et al. 2020 COLL [aatteiger et al. 2020a), and OC20 (Chanussot et al. 2021). Table gives an overview of these datascts. Note that datasots such as QM9 (Ramakrishnan et al. (014)) or OQMD (Saal et al. 2013) can-
not be used for learning molecular simulations since they only contain systems at equilibrium, where all forces
are zero. Other datasets such as DFS370k (Donchev et al. 2021 and OrbNet Denali (Christensen al. 2021
3

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: Common molecular simulation datasets. Datasets prior to OC20 only cover (1) a narrow range of elements (and molccules), and consequently a low number of neighbor pairs (defined as distinct element pairs within A), and (2) small systems. Furthermore, they (3) provide far fewer samples and (4) often use a test set that is correlated with the training set since they consist of data from the same simulation trajoctories. We investigate six OC20 subsets to isolate these effects.
Dataset
Description
Elements
Neighbor pairs
Avg. size
Train set size
Test set(s)
MD17
Fight separate molecules
H, N.
3-10
12.5
(9-21)
8
1000
Same, single trajectory
IS017
C,O,Hı0 isomers
H. C,
19
1040000
Same traj. / OOD systems
S,2
Methyl halides, halide ions
H, C, F, CI, Br, I
20
5.4 (2-6)
400 400.00
Same trajectories
ANI-Ix
Selected MD sumples
H. C, N, 0
10
15.3 (2-63)
4 1956 005
OOD systems (COMP6)
QM7-X
Small molecules
H, C. N, O, S, CI
20
16.7 (4-23)
4 1175 037
Same traj. / OOD systems
COLL
Molecule collisions
H, C, 0
10.2 (2-26)
120 000
Same trajectorics
OC20
Relaxations of catalysts
56
1454
73.3 (7-225)
133934018
Sep. traj. / OOD ads.
OC-Rb
Only H. C, N. O, Rb
H. C, N. O, Rbb
15
39.1 (7-220)
534
Sep. traj. 0ODD adsorbates
0C-5n
Only H. C, N, O, Sn
H, C, N, O, So
15
59.5 (22-220)
257 757
Sep. traj. / /0O1 adsorbates
OC-sub3U At most 30 atoms
55
881
24.6 (7-30)
4020 568
Sep. traj. / /OOD +ct.
OC-200k
Random subset
56
1454
73.2 (7-225)
000
Sep. traj. / /ODD tct.
OC-2M
Random subset
56
1454
73.3 (7-225)
000
Sep. traj. / OOD cct.
OC-XS
< 30 H, C, N, O, Rb atoms H, C, N, O, Rb
15
19.7 (7-30)
298 797
Same / sep. traj. / OOD ads.
contain out-of-equilibrimm systems but do not provide force labels. This makes them ill-suited for this task as well, since energies only provide one label per sample while forces provide 3N labels (Christensen & Lilienfeld 2020). In this work we focus on the OC20 dataset, which consists of single adsorbates (small molccules) physically binding to the surfaces of catalysts. The simulated cell of N atoms uses periodic boundary conditions to emulate the behavior of a crystal surface.
Dataset aspects. The difficulty and complexity of molecular datasets can largely be divided into four aspects: chemical diversity, system size, dataset size, and domain shift. Chemical diversity (number of atom types or interacting pairs) and system size (number of independent. atoms) determine the data's complexity and thus the expressive power a model needs to fit this data. The dataset size determines the amount of data from which the model can learn. Note that a large dataset might still need a very sample-efficient model if the underlying data very complex and diverse. Finally, we look at the test set's domain shift to determine its difficulty. The learning task might be significantly easier than expected if the test set is very similar to the training set. Note that many dataset aspects are not covered by these four properties, such as the method used for calculating the molecular data or the types of systems. OC20 has multiple details that fall into this category: It consists of periodic systems of a bulk material with an adsorbate on its surface, and has a canonical z-axis and orientation. These dotails also affect the importance of modeling choices, such as rolation invariance (Hu et al., 202T]. We chose to focus on the above four properties since they are easy to quantify, apply to every dataset, and have a large effect by themselves.
Data complexity. Chemical diversity and system size determine the complexity of the underlying data: How many different atomic environments does the model need to fit? Are there long-range interactions collective many-body effects caused by large systems? To quantify dataset's chemical diversity we count the number of elements and the element pairs that occur within a range of 5 Á ("neighbor pairs"). However, note that these proxies are not perfect. For example, COLL only contains three elements but samples a significantly larger region of conformational space than QM7-X. Subsequently, SchNet only achieves a force MAE of 172 cV/A on COLL, but 53.7 on QM7-X (same trajoctory). Still, these proxics do illustrate the stark difference between OC20 and other datasets: OC20 has 70x more neighbor pairs than other dalasets Table 1).
Dataset size. Datasct size determines how much information is available for a model to learn from. Note that larger systems also increase the effective dataset size since each atom provides one force label. Usually, the dataset size should just be appropriately chosen to reach good performance for a given dataset complexity. Table thus lists the official or a typical training set size for cach datasct, and not the total datasct size. An extreme outlier in this respect is MD17. In principle it has 3611 115 samples, i.c. 450 000 samples per each of the 8 molecule trajectories on average. This is extremely large for the simple task of fitting single small
4

---

Published in Transactions on Machine Learning Research (09/2022)
molecules close to their equilibrium. Most recent works thus only train on 1000 MD17 samples per molecule, T.e. less than 1 % of the dataset. Even in this setting, modern models fit the DFT data more closely than the DFT method fits the ground truth (Gasteiger et al. 2020b).
Domain shift. A dataset's test set is another important aspect that determines a dataset's difficulty and how well it maps to real-world problems. In practice we apply a trained model to a new simulation, Le. a different simulation trajectory than was trained on. We might even want to apply it to a different, out-of-distribution (OOD) system. For example, the OC20 dataset contains OOD test sets with unseen adsorbates (ads.), catalysts (cat.) and both (ads.+cat.)) However, many datasets a set of simulations and then randomly splitting up the data into the training and test scts. The data is thus taken from the same trajectories, which leads to strongly correlated training and test data: the test samples will never be significantly far away from the training samples. To properly decorrelate test samples, we have to either let enough simulation time pass between the training and test samples or use a separate simulation trajectory (sep. traj.). An extreme case in this aspoct again MD17: its test samples are taken from the same, single trajectory and the same, single molecule as the training set. MD17 thus only measures minor generalization across conformational space, not chemical space. This severly limits the significance of results on MD17. Morcover, improvements demonstrated on MD17 might not even be useful due to the data's limited accuracy. The most likely reason for MD17's pervasiveness is that its small size makes model development fast and efficient, coupled with the hypothesis that discovered model improvements would work equally well on more complex datasets. Unfortunately, we found this hypothesis to be incorrect, as we show in more detail below. This raises the question: Which dataset aspect changes model performance between benchmarks? Answering this would allow us to create a dataset. that combines the best of both worlds: fast. development cycles and results that generalize to realistic challenges like OC20.
created by running
OC20 subsets. We investigate six subsets of the OC20 dataset to isolate the effects of each dataset aspect. Comparing subsets of the same dataset ensures that the observed differences are only due to these specific aspects, and not duc to other dataset propertics. OC-Rb and OC-Sn reduce chemical diversity by restricting the datasel to a limited set of catalyst materials. We chose rubidium and lin since they are among the most frequent catalyst elements in OC20. We would expect similar results for other catalyst elements. OC-sub30 isolates the effect of small systems by only allowing up to 30 atoms per system. Note that this does not influence the number of neighbors per atom, since OC20 uses periodic boundary conditions. It does, however, impact its effective training size and might skew the selection of catalysts. OC-200k and OC-2M use random subsets of OC20, thus isolating the effect of dataset size. Finally, OC-XS investigates the combined effect of restricting the system size to 30 indopendent atoms and only using Rubidium catalysts. This also naturally decreases the dataset size. We apply these subsets to both the OC20 training and validation set in the same way. To investigate the effect of test set choice we additionally investigate OOD validation splits by taking the respective subsets of the OC20 OOD dataset as well. We use OOD catalysts and adsorbates ("OOD both"") for OC-sub30, OC-200k, and OC-2M, and only OOD adsorbates for OC-Rb, OC-Sn, and OC-XS, since no OOD catalysts are available for these subsets. Additionally, we introduce third validation set for OC-XS by splitting out random samples of the training trajectorics. This selection mimicks the casier "same trajectory" test sets used e.g. by MD17 (Chmiela el al. 2017) and COLL (Gasteiger et al. 2020a).
4 GemNet-OC
We investigate the effect of datasets on model choices by first developing a GNN on OC20, starting from the geometric message passing neural network (GemNet) (Gasteiger et al., 2021 While regular message passing neural networks (MPNNs) only embed each atom a as h, 2 Gilmerr et al., 2017). GemNet additionally embeds the directed edges between atoms (ba) E IR "m, Both embeddings are then updated in multiple learnable layers using neighboring atom and edge embeddings and the full geometric information the distances between atoms Tba the angles between neighboring edges Seab, and the dihedral angles defined via triplets of edges Oonbe- We call our new model Ne-NOOOOC. Gem is well representative of previous models. It uses a similar architecture and the atomwise force errors are well correlated: The Spearman rank correlation to SchNet's force errors is 0.44, to +imeNet++ 0.50, to SpinConv 0.46, and to GemNet 0.59. It is just slightly higher for a separately trained OC model, at 0.66. We next describe the components we propose and investigate in GemNet-OC.
5

---

Published in Transactions on Machine Learning Research (09/2022)
Model:
Interaction:
Message passing:
A/T/Q-MP:
Foe Footi
Peab
CRBF
ha
(-1)
(1-1)
h
(1-1)
RBB
db/00
(1-1)
ho.
RBF
CBF
SBF gesel oe
Message passing
(WW)
W
+512
(WWD)
(cso) CRBP
(rau)) BCPP
Faoted SBP
esRP
Residual
Residual
EA-MP
AE-MP
FCBF
W
7512
Residual
Atom emb
EE-MP
((WO
Embedding
W
32/60
Interaction
ha
Q-MP
AA-MP
Interaction
The
W
WW
(WD)
32/64
Interaction
Residual
5.6
Residual
Interaction
((WA
(WAO)
Aoct.
Meussoo
Atom emb
512 mioo
(W
o(W
Residual
Resicual
Residual
Mea he
Residual
Residual
Residual
Bost.
o(W
F.
27)
haa
out.co
Figure 1: Main parts of the GemNet-OC architecture. Changes are highlighted in orange D denotes the layer's input. || concatenation, a a non-linearity, and yellow a layer with weights shared across interaction blocks. Numbers denote embedding sizes. Whether the input or output of the MP-block are atom or edge embeddings depends the lype of message passing. Difference between different variants (AA, AE, EA, AA, Q-MP) are shown via colors and dashed lines. Note that triplet components are also included in quadruplet interactions.
Neighbors instead of cutoffs. GNNs for molecules typically construct an interatomic graph by connecting all atoms within a cutoff of 5 A to 6 Å. Distances are then usually represented using radial basis, which is multiplied with an envelope function to ensure twice continuously differentiable function at the cutoff Gasteiger et al.) 2020b). This is required for well-defined force training if the forces predicted via backpropagation. However, when we face the large chemical diversity of a dataset like OC20 there are systems where atoms are typically farther apart than 6A, and other systems where all neighboring atoms are closer than 3 A. Using one fixed cutoff can thus cause a disconnected graph in some cases, which is detrimental for energy predictions, and an excessively dense graph in others, which is computationally expensive. To solve this dilemma, we propose to construct a graph from a fixed number of nearest neighbors instead of using a distance cutoff. This might initially seem problematic since it breaks differentiability for forces two atoms switch order by moving some small 8. However, we found that this is not issue in practice. The nearest neighbor graph leads to the same or better accuracy, triples GemNet-OC's throughput, provides easier control of computational and memory requiremeuls, and ensures consistent, fixed neighborhood size.
8x E if
Simplified basis functions. As proposed by Gasteiger et al. (200bb), GemNet represents distances Tia using spherical Bessel functions ji( ste Iba) order 1, with the interaction cutoff Cint and the Bessel function's 's qui
n-th root Zin, and angular information using real spherical harmonics Y!!!) (Peat, Ocata) of degree / and order m. Note that the radial basis order / is coupled to the angular basis degree I. This basis thus requires calculating n/ radial and Im angular functions. This becomes expensive for large systems with a large number of neighbors. If we embed kemb edges per atom and compute dihedral angles for kigitt neighbors, to caleulate O( basis functions. To reduce the basis's computational cost, first decouple the radial basis from the angular basis by using Gaussian 0-order Bessel functions, independent of the spherical harmonics. We then streamline the spherical harmonics by instcad using an outer product of order 0 spherical harmonics, which simplify to Legendre polynomials y(()) (featt) ) (Deate) P((cos (eat)) P.,. (cos Ocabe)- This
need

---

Published in Transactions on Machine Learning Research (09/2022)
only requires the normalized inner product of edge directions, not the angle. These simplified basis functions increase throughput by 29 %, without hurting accuracy.
Tractable quadruplet interactions. Next, we tackle GemNet's dihedral angle-based interactions for large systems. These 'quadruplet interactions' are notoriously expensive, since they scale as (aght However, we observed that quadruplet interactions are mostly relevant for an atom's 's closest neighbors. Their benefits quickly flatten out as we increase Agint- We thus choose a low kgint = 8. This is substantially lower than our kemb 30, since we found model performance to be rather sensitive to kemb- This is opposite to the original GemNett where the quadruplet cutoff was larger than the embedding cutoff. Quadruplet interactions only cause an overhead of 31 % thanks to these optimizations, instcad of the original 330 % (Gasteiger et al. 2021
Interaction hierarchy. Using a low number of quadruplet interactions essentially introduces a hierarchy of expensive short-distance quadruplct interactions and cheaper modium-distance edge-to-odge interactions. We propose to exlend this interaction hierarchy further by passing messages between the alom embeddings ha directly. This update has the same structure as GemNet's message passing, but only uses the distance between atoms. This atom-to-atom interaction is very cheap due to its complexity of ((Nkemb) We can thus use a long cutoff of 12 without any neighbor restrictions. To complement this interaction, we also int atom-to-edge and edge-to-atom message passing. These interactions also follow the structure of GemNet's original message passing. Each adds an overhead of roughly 10 %.
Further architectural improvements. We make three architectural changes to further improve GemNet- OC. First., we output an embedding instead of directly generating a separate prediction per interaction block. These embeddings are then concatenated and transformed using multiple learnable layers (denoted MLP) to generate the overall prediction. This allows the model to better leverage and combine the different pieces of information after each interaction block. Second, we improve atom embeddings by adding a learnable MLP in each interaction block. This block is beneficial due to the new atom-to-atom, edge-to-atom, and atom-to-edge interactions, which directly use the atom embeddings. Third, we improve the usage of atom embeddings further by adding them to the embedding in each energy output block. This creates a direct. path from atom embeddings to the energy prediction, which improves the information flow. These model changes add less than 2 % computational overhead. See Fig. for an overview of the GemNet-OC model.
5 Results on OC20
OC20 tasks. In the OC20 benchmark, predicting energies and forces is the first of three tasks, denoted as structure to energy and forces (S2EF). There are two important additional scores for this task besides the energy and force MAEs. The force cosine determines how correct the predicted force directions are and energy and forces within threshold (EFwT) measures how many structures are predicted "correctly", i.c. have predicted energies and forces sulficiently close to the ground-truth. The second task is initial structure to relaxed structure (IS2RS). In IS2RS we perform energy optimization from an initial structure using our modcl's force predictions gradients. After the optimization process we mcasure whether the distance between our final relaxed structure and the true relaxed structure is below variety of thresholds (average distance within threshold, ADwT), and whether additionally the structure's true forces are close to zero (average force below threshold, AFbT). The third task, initial structure to relaxed energy (IS2RE), is to predict the energy of this relaxed structure. An alternative to this relaxation-basee approach is the so-called direct TS2RE approach. Direct models learn and predict energies directly using only the initial structures. This approach much cheaper, but also less accurate. Our work focuses on the regular relaxation-base approach, which is centered around S2EF data and models.
Results. In Table 2 we provide test results for all three OC20 tasks, as well as S2EF validation results, averaged over the in-distribution (ID), O0D-adsorbate, OOD-catalyst, and OOD-both datascts. To facilitate and accelerate future research, we provide multiple new baseline results on the smaller OC-2M subset for SchNet (Schütt et al.) 2017), ++ (Gasteiger et al., 2020a), SpinConv (Shuaibi et al. 2021), and GemNet-dT [Gasteiger et al. 2021). OC-2M is smaller but still well-reppesentative of model choices on full OC20, as we will discuss in Sec. 6 We use the same OC20 test set for both the OC-2M and OC20 training
sets. On full OC20 we additionally compare to CGCNN (iie & Grossman 2018), ForceNet (Hu et al. 2021
7

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2: Training throughput and results for the validation set and the three test tasks of OC20, averaged across all four splits. GemNet-OCa OC-I outperforms prior models by 16 %. GemNet-OO trained on OC-2M outperforms all pre- GemNet models trained on the full OC20 dataset (~134M samples).
Throughpus
S2EF validation
S2EF test
IS2RS
IS2RE
Train
Sumples/ /
Energy MAE Foree MAE
Force cos
EFwT Energy MAE
Force MAE
Force cos
ETWT AFET AD Energy MAE
set
Model
CPU sec. 1
meV 1
meV/A
f
%
meV 4
mcV/A J
t
% 1
%1
%1
mcV 1
SchNet
14000
78.3
0.109
0.00
1370
17.1
0.116
0.00
OC-2M
DimeNet**
805
65.7
0.217
0.01
761
63.0
0.222
0.01
SpinCouv
406
36.2
0.479
0.13
401
35.5
0.475
0.13
GemNettdTT
358
29.5
0.537
0.61
323
28.1
0.539
0.69
16.7
51.8
438
GemNet-OO
286
25.7
0.598
1.06
274
24.3
0.603
1.25
19.6
56.4
407
CGCNN
590
710
0.142
0.01
608
73.3
0.144
0.01
SchNNet
549
56.8
0,297
0.06
540
64.7
0.302
0.06
14.4
764
ForceNet-large
15.3
33.5
0.515
32.0
0.516
0.01
12.7
19.6
OC22
DimeNet --
4.6
515
32.8
0.541
0.00
450
31.3
0.544
0.00
21.7
51.7
559
PaiNN
60.0
341
351.1
0.491
0.46
11.7
48.5
471
SpinConv
6.0
371
11.2
0.173
0.05
336
29.7
0.539
0.15
16.7
53.6
437
GemNet-dTT
25.8
315
27.2
0.594
0.54
202
24.2
0.616
27.6
58.7
GemNeetXL.
1.5
270
20.5
0.660
1.81
30.8
62.7
371
GemNNet-OC
18.3
244
21.7
0.662
2.07
233
20.7
0.666
2.50
35.3
60.3
355
0C20+ OC-MD
GemNet-OO--LE
7.5
239
22.1
0,682
2.30
230
21.0
0.665
2,80
GemNet-OC-I-F
3.2
252
20.0
0.687
2.51
241
19.0
0.691
2.97
40.6
60.4
348
PaiNN Schütt et al. 2021), and GemNet-XT, (Sriram et al. 2022 Note that PaiNN uses our independent. reimplementation of the original PaiNN architecture with the difference that forces are predicted directly from vectorial features via a gated oquivariant block instead of gradients of the energy output. This breaks energy conservation but is essential for good performance on OC20. GemNet-Q Gastteigrr et al.] 2021 runs out of memory when training on OC20, GemNet-OC outperforms all previous models on both the OC-2M and OC20 training sets. GemNet-OOC on OC-2M even performs on par with GemNet-dTT on OC20, while using 70 times less training data. It also performs better than all previously proposed direct IS2RE models, such as 3D-Graphormer (Ying et al. 2021 which achieves an IS2RF MAF of 472.2 meV. Direct models have seen a lot of interest due to their fast training and development. The OC-2M S2EF dataset provides a similarly fast development method for relaxation based models. After model selection, we can scale the model up to larger training sets and model sizes. We demonstrate this with the GemNet-OC-L model, which was trained both the full OC20 and the OC-MD datasets. OC-MD complements the regular OC20 dataset with data points from molecular dynamics. We trained one model focussing on energy predictions GGem-eet-O---BB and a second one for force predictions (GemNet-OC-L-F)), For IS2RE we then combined both in the CemnNetOC---F-HE model. These models set the state of the art for all OC20 tasks, GemNet-OC even outperforms GemNet-XL, despite GemNet-XL. being substantially larger and slower to train.
Training time. Fig. shows that Ge surpasses the accuracy of GemNet-dTT after 600 GPU hours. It is 40% slower per sample than performance--otinnvee version of GemNet-dT. How- ever, if we consider that GemNet-OC uses four inter- action blocks instead of three as in GemNet-dT, we see that the GemNet-OC architecture is roughly as fast as GemNet-dT overall. GemNet-OC--L roughly 4.5 times slower per sample but still converges faster, surpassing regular Ge after 2800 GPU hours. GemNet-OC-LL reaches the final accuracy of GemNet- XL in 10 times fewer GPU hours, and continues to improve further. Combined with GemNet-OCCss good results on the OC-2M dataset, this fast convergence allows for much faster model development. Overall, OC shows that we can substantially improve model like GemNet if we focus development the
Force MAE (meV (me)/A) /A)
DimeNet -L
30
GemNet-dTT
GemNet-XT
GemNet-OC
25
GenNet-OC-L. (with OC-MD)
20
10x cheaper
3. 3..m/// / better
10a
10
10a
GPU hours
Figure 2: Convergence of GemNet-OO OC and previous models. GemNet-OC surpasses GemNet-dTT after 600 GPU hours, and is 12x faster than GemNet-XL.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Rel. improvement (%)
(%) anddsnooue Rol
10
50
10
MD17
COLL
0
OC-XS
0
OC-Rb
OC-Sn
-10
50 -10
OC-sub30
-20
50
OC-200k
OC-2M
10a
10a
2
10
10a
10a
10a
OC20
Edge embedding size
Number of blocks
Batch size
Throughput
Figure 3: Effect of batch size, edge embodding size, and number of blocks on force MAE and throughput. Most datasets are insensitive to the batch size above a certain size. Large datasets strongly benefit. from large models, while small datasets exhibit optima at small sizes.
target dataset. However, this does not yet answer the question of how its model choices would change if we focused on another dataset. We will investigate this question in the next section.
6 Model trends across datasets
For the purpose of comparing model choices between datasets we run all ablations and senstivity analyses on the Aspirin molecule of MD17, the COLL dataset, the full OC20 dataset, and the OC-Rb, OC-Sn, OC-sub30, OC-200k, OC-2M, and OC-XS data subsets. For consistency we use the same training procodure all datasets, varying only the batch size. If not noted otherwise, we present results on the separate validation trajectories for Open Catalyst datasets. To simplify the discussion primarily focus force mean absolute error (MAE), and show changes as relative improvements compared to the GemNet-OO baseline MAFGeuNa-ooc - 1. We additionally report median relative improvements in throughput (samples per GPU second). We train the baseline model five times for every dataset, but train each ablation only once due to computational restrictions. We assume that the ablations have the same standard deviation as the baseline, and show uncortainties based on the 68 % confidence intervals, calculated using the student's t-distribution for the baseline mean, the standard deviation for the ablations, and appropriate propagation of uncertainty. For further details see App. A
MAE
Model width and depth. In general, would expect larger models to perform better since all investigated settings lie well inside the parammtrmiriiioonn regime (Belkin et al., (201)). However, the edge embedding size (model width) and the number of blocks (depth) in Fig. 3 actually exhibit optima for most datascts instead of strictly increasing trend. Importantly and perhaps unsurprisingly, these optima differ between datasets. MD17 exhibits clear optimum at a low width and depth, and also COLLL, OC-200k and OC-sub30 exhibit shallow optima at low width. These optima appear to be mostly present for datasets with a low number of samples compared to its high data complexity and dissimilar validation set (see Fig. 11 for results on the OOD validation set). Accordingly, we observe the greatest benefit from model size for the largest datasets and in-distribution molecules. We observe a similar effect for other embedding sizes such as the projection size of basis functions. Increasing the projection size yields minor improvements only for OC20 (see Fig. 10.. Overall, we notice that increasing the model size further only leads to modest accuracy gains, especially for OOD data. This suggest that molecular machine learning cannot be solved by scaling models alone. Note that this result might be impacted by model initialization (ang al. 2021].
Batch size. In Fig. 3 we see that most datasets exhibit optima around batch size of 32 to 128. However, there are substantial differences between datasets. MD17 has a particularly low optimal batch size, as do OOD validation sets (see Fig. 11I. MD17 and OOD sets might thus benefit from the regularization effect caused by a small batch size. We also observed that convergence speed is fairly consistent across batch sizes if we focus on the number of samples seen during training, not the number of gradient steps.
Neighbors and cutoffs. The first two sections of Fig. shows the impact of the cutoff, the number of neighbors and the number of quadruplets. Their trends mostly consistent between datasets, albeit with
9

---

Published in Transactions on Machine Learning Research (09/2022)
Rel. improvement (%
indy Rell
MD17
20
50
COLL
OC-XS
0
OC-Rh
OC-8n
OC-sub30
20
50
OC-200k
196
OC-2M
OC20
6 A. nghbra.
6 6 cutoff
quadruplets
quadruplets
20 quadruplets
Bessel
32 Bessel
multi-order Bessel
Sph. harmonics SBF
Throughput
Figure 4: Impact of cutoffs, neighbors, and basis functions on force MAE and training throughput compared to the baseline using& a cutoff of 12 A, 30 neighbors, 8 quadruplets, Legendre polynomials as angular, and
Gaussians as radial basis. Notably, the radial Bessel basis performs significantly better on datasets with small chemical diversity, but worse on full OC20.
1% [uduanomu 18e1
MD17
10
20
COLL
Rel. improvement
OC-XS
php t
OC-Rb
0
OC-Sn
OC-sub30
10
20
OC-200k
OC-2M
No scaling factors
No symmetric MP
No quadruplet int.
No atom-atom int.
No atom- int.
No edge-atom int.
No global output MLP
No atom emb. in out
No atom MLP
OC20
Throughput
Figure 5: Ablation studies of various components proposed by GemNet and GenNet-OC for force. MAE and throughput. Some model changes show consistent improvements (e.g. the quadruplet interaction), while others have very different effects for different datasets.
wildly different impact strengths. For the small molecules in MD17 and COLL the large default cutoff causes a fully-connected interatomic graph, which performs substantially better. Importantly, the default nearest neighbor-based graph performs better than cutofls on OC20. The number of quadruplets has a surprisingly small impact on model performance on all datasets. We observe a small improvement only for 20 quadruplets, which is likely not worth the associated 20 % runtime overhead.
Basis functions. An intruiging trend emerges when comparing and 32 Bessel functions with the default 128 Gaussian basis functions (third section of Fig. 4): Bessel functions perform substantially better on datasets with small chemical diversity, such as MD17, OC-XS, OC-Rb, and OC-Sn. However, this trend completely reverses on large datasets such as OC20. Multi-order Bessel functions also fit within this trend, as shown in the fourth section of Fig. 4. 6 multi-order Bessel functions perform somewhere between 6 and 32 Bessel functions, since they provide more information than regular 0-order Bessel functions. However, they are significantly slower and thus recommended. Simplifying the spherical basis part has similar effect: The outer product of Legendre polynomials is faster and works well as spherical Harmonics across all datasets.
A Abltioon studies. In Fig. 5. we ablate various components from GemNet-OC to show their effect on different datasets, based both on previous and newly proposed components. This is perhaps the most striking result: Many model components have opposite effects on different datasets. The empirical scaling factors proposed by Gasttigger et al. (2021) have a very inconsistent effect on different datascts. Note that we found that they stabilize early training, which can be essential for training with mixed precision even if they do not improve final performance. Symmetric message passing quadruplet interactions have the most.
10

---

Published in Transactions on Machine Learning Research (09/2022)
0.6
OC-XS
Force MAE (meV (A)
No atom-atom int,
OC-Sn
23
No atom-edge
No edge-atom
MD17
0.4
22
No atom emb, in out
OC-200k
No atom MLP
OC-Rb
No global output MLP
20
GemNet- OCC median
OC-2M
OC20
0.0
19
COLL
18
OC-sub30
0.2
0
50
100
150
200
250
300
Million samples seen
OC-xS
OC-Sn
MD17
OC-200k
OC-Rb
OC-2M
OC20
COLL
OC-sub80
Figure 6: Convergence model ablations for OC20. Atom-cdge and edge-atom interactions, and the global output MLP improve convergence, while atom-atom interactions, the atom MLP, and atom embeddings in the output have no positive effect. Note that step-like improvements are due to adaptive learning rate steps.
Figure 7: Kendall rank correlation of model force MAEs between datasets and resulting hierarchical clustering. Results vary strongly between datascts. OC-2M gives the most similar results to OC20.
consistent impact across datasets. The additional interactions in GemNet-OC show little to no impact on
in-distribution OC20, but are beneficial for OC20 distribtitiii data (see Fig. 13 The tddgeto-edge and edge-to-atom interactions furthermore lead to faster convergence, as shown in Fig. 6 Importantly, the impact of these interactions is different on each dataset, varying between significantly negative to significantly positive results. Similarly, the proposed architectural improvements usually have either no or a small positive effect at convergence, but drastically hurt performance on COLL. Predicting forces via backpropagation is another model choice that works well on small datasets (MD17, COLL), but does not benefit the full OC20 dataset. Unfortunately, training GemNet-OC on OC20 with backpropagated forces too unstable to train reliably and report here.
Correlation between datasets. To quantify the overall similarity of model performance bet weeween datasets we calculate the Kendall rank correlation coefficient of the force MAEs on one dataset to each other dataset. Each model variant represents one data point in this calculation. We exclude the batch size experiments since this result seems too obvious. We additionally show a hierarchical clustering based on the nearest point algorithm. Fig. 7 shows that OC-2M provides the most similar results to OC20. Interestingly, OC-2M is roughly as similar to OC20 as early stopping or the OOD test set (see Fig. 9p. This good correlation is due to OC-2M capturing the same chemical diversity and test difficulty thanks to uniform sampling. It is not outlier in this respect: We independently sampled five 2M subsets and observed a standard deviation in GemNet-OC force MAE of merely 0.9 %. We furthermore observe that MD17 is especially different from OC20. The OOD validation set (Fig. 14h shows similar results. Note that these aggregate results hide the fact that some model choices, such as using quadruplets, are very consistent between datasets, while others are very different, such as the choice of radial basis.
Based on these results we conclude that a dataset should cover the same chemical diversity be reflective of large and diverse dataset. Both the less diverse systems in OC-Rb and OC-Sn and the smaller systems in OC-sub30 show a larger difference in model choices. And these differences add up: Results OC-XS are even more different. We can preserve this diversity by uniformly subsampling the datasct (OC-2M), but this still breaks down if we reduce the dataset too much (OC-200k). The choice of test. set also seemss to play a critical role. For example, the OC20 OOD validation set is only slightly closer to in-distribution than OC-2M, and the OC-XS "same trajectory" validation set is very different to regular OC-XS. Note that domain shift. appears make tasks particularly difficult. This is indicated by the (absolute) force MAE; 99.1 mcV for OOD on OC-XS, 16.0 meV/ / for separate trajectories (regular validation set), and 3.2 meV/A / for same
11

---

Published in Transactions on Machine Learning Research (09/2022)
trajectories. Overall, it appears that correlated model choices require datasets to have comparable difficulty and complexity, given by chemical diversity, domain shift, and sufficient training set size.
7 Conclusion
In this work we studiod the effect of developing GNNs on a large dataset instead of small benchmarks. We proposed the GemNet-OC model, which is substantially faster to train than previous models and sets the state of the art on all OC20 tasks. We then investigated how model choices GemNet-OC differ between datasots. We found that model components and hyperparameters can have disparate even opposite effects between datasets, even within the single task of force predictions. We studied the source of this effect by selecting data subsets that individually change four main data complexity aspects. This resulted in two insights: First, consistent model choices require datascts with comparable difficulty and complexity, as given by their chemical diversity, domain shift, and & sufficiently large training set. Second, we can create a well-correlated proxy dataset by uniformly sampling a sufficienlly large data subset. The resulting OC-2M dataset allows model development for the massive OC20 dataset at fraction of the computational cost. To support future model research we provide a range of baseline results for this dataset. Overall, this case study highlights that researchers should exercise caution during model development, since model choices can vary drastically when changing relevant dataset properties.
Acknowledgments
We thank Brandon Wood for helpful discussions on performance and scaling aspects and the Open Catalyst team for their support, foedback, and discussions and for providing the foundational codebase for this project Chanussot et al. 2022)). We furthermore acknowledge PyTorch (Paszke al. 2019) and PyTorch Geometric Fey & Lenssen
2019
References
Uri Alon and Eran Yahav. On the Bottleneck of Graph Neural Networks and its Practical Implications. In ICLR, 2021.
2
Brandon M. Anderson, Truong-Son Hy, and Risi Kondor. Cormorant: Covariant Molecular Neural Networks. In NewrIPS, 2019. 2
Albert P. Bartók, Mike C. Payne, Risi Kondor, and Gábor Csányi. Gaussian Approximation Potentials: The
Accuracy of Quantum Mechanics, without the Electrons. Physical Review Lellers, 104(13):136403, 2010.
Albert P. Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Review
B, 87(18)1184115 2013. 2
Albert P. Bartók, Sandip De, Carl Poelking, Noam Bernstein, James R. Kermode, Gábor Csányi, and Michele Ceriotti. Machine learning unifies the modeling of materials and molecules. Science Advances, 3
e1701816, 2017. 2
Igor I. Baskin, Vladimir A. Palyulin, and Nikolai Zefirov. A Neural Device for Searching Direct Correlations between Structures and Properties of Chemical Compounds. Journal of Chemical Information and Computer Sciences, 37(4):115-721, 1997.
Simon Batzner, Tess E. Smidt, Lixin Sun, Jonathan Mailoa, Mordechai Kornbluth, Nicola Molinari, and Boris Kozinsky. Fquivv-qui Graph Neural Networks for Data-Ffficient and Accurate Interatomic Potentials. arXiv, 2101.03164, 2021.
Jörg Behler and Michele Parrinello. Generalized Neural-Network Representation of High-Dimension Potential-Fanery Surfaces. Physical Review Letters, 98(14):146401, 2007. 2
12

---

Published in Transactions on Machine Learning Research (09/2022)
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Acodemyy of Sciences, 116(32): 15849 15854, 2019. D
Jorg Bornschein, Francesco Visin, and Simon Osindero. Small Data, Big Decisions: Model Selection in the Small-Data Regime. In ICML, 2020.
D 2
Lorenzo Brigato and Luca Tocchi. Close Look at Deep Learning with Small Data. In ICPR, 2020. 2
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral Networks and Deep Locally Connected Networks on Graphs. In ICLR, 2014. 2
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaībi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Amuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open Catalyst 2020 (OC20)
Dataset and Community Challenges. ACS Catalysis, 11(10):6059 6072, 2021. [document)
3
Stefan Chmiela, Alexandre Tkatchenko, Huziel E. Sauceda, Igor Poltavsky, Kristof T. Schütt, and Klaus- Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science Advances, 3 (5):c1603015, 2017.
3
Anders S. Christensen and O. Anatole von Lilienfeld. On the role of gradients for machine learning of molecular energies and forces. Machine Learning: Science and Technology, 144):045018, 2020. 3
Anders S. Christensen, Sai Krishna Sirumalla, Zhuoran Qiao, Michael B. O'Connor, Daniel G. A. Smith, Feizhi Ding, Peter J. Bygrave, Animashree Anandkumar, Matthew Welborn, Frederick R. Manby, and Thomas F. Miller. OrbNet. Denali: A machine learning potential for biological and organic chemistry with semi-empirical cost and DFT accuracy. The Journal of Chemical Physics, 155(20):204103, 2021.
Alexander G. Donchev, Andrew G. Taube, Flizabeth Decolvenaere, Cory Hargus, Robert T. McGibbon, Ka-Hei Law, Brent A. Gregersen, Je-Luen Li, Kim Palmo, Karthik Siva, Michael Bergdorf, John L. Klepeis, and David E. Shaw. Quantum chemical benchmark databases of gold-standard dimer interaction energies. Scientific Data, 8(1):55, 2021. 3
David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraauirre, Rafael Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional Networks on Graphs for Learning Molecular
Timothy
Fingerprints. In NeurIPS, 2015. 2
Matthias Fey and Jan E. Lenssen. Fast Graph Representation Learning with PyTorch Geometric. In Workshop Representation Learning on Graphs and Manifolds, ICLR, 2019. 7
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then Propagate: Graph
Neural Networks Meet Personalized PageRank. In ICLR, 2019. 2
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and Uncertainty- Aware Directional Message Passing for Non-Equilibrium Molecules. In Machine Learning for Molecules Workshop, NewrIPS, 2020.
3
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional Message Passing for Molecular Graphs. In ICLR, 2020.
1
2
3
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Universal Directional Graph Neural Networks for Molccules. In NewrIPS, 2021. 2 4
5
6
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In ICML, 2017.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Hcewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yançi Zhou. Deep Learning Scaling is Predictable, Empirically. arv, 1712.00409, 2017.
13

---

Published in Transactions on Machine Learning Research (09/2022)
Johannes Hoja, Leonardo Medrano Sandonas, Brian G. Ernst, Alvaro Vazquez-Mayagoitta, Robert A. DiStasio Jr., and Alexandre Tkatchenko. QM7-X: A comprehensive dataset of quantum-mecharicall
properties spanning the chemical space of small organic molecules. arXiv, 2006.15139. 2020. 3
Weihua Hu, Muhammed Shuaibi, Abhishek Das, Siddharth Goyal, Amuroop Sriram, Jure Leskovec, Devi Parikh, and C. Lawrence Zitnick. ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations. arXiv, 2103.01436, 2021. 3 5
Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. ICLR, 2017. 2
Simon Korublith, Jonathon Shlens, and Quoc V. Le. Do Better ImageNet Models Transfer Better? In CVPR, 2019.
Cheol Woo Park, Mordechai Kornbluth, Jonathan Vandermause, Chris Wolverton, Boris Kozinsky, and Jonathan P. Mailoa. Accurate and scalable graph neural network force field and molecular dynamics with direct force architecture. upj Compulational Materials, 7(1):1-9, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High- Pirf-rmerforacce Deep Learning Library. In NewrIPS, 2019.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1):1-7, 2014.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. In ICLR, 2018. A
James E. Saal, Scott Kirklin, Muratahan Aykol, Bryce Meredig, and C. Wolverton. Materials Design and Discovery with ThrouThhuuhhut Density Functional Theory: The Open Quantum Materials Database (OQMD). JOM, 65(11):1501 1509, 2013.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoe Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. SchNet: A contimuous-ilter convolutional neural network for modeling quantum interactions. In NewrIPS, 2017.
Kristof T. Schütt, Oliver T. Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In ICML, 2021. 5
Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. Rotation Invariant Graph Neural Networks using Spin Convolutions. arXiv, 2106.09577
2021. 2 5
J.S. Smith, O. Isayev, and A.E. Roitberg. ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost. Chemical Science, 8(4):3192 11033333--, 2017. 2
Justin S. Smith, Roman Zubatyuk, Benjamin Nebgen, Nicholas Lubbers, Kipton Barros, Adrian E. Roitberg, Olexandr Isayev, and Sergei Tretiak. The ANI-lccx and ANI-Ix data sets, coupled-cluster and density functional theory properties for molecules. Scientific Data, 7(1):134, 2020.
Sperduti and A. Starita. Supervised neural networks for the classification of structures. JEEF Transactions on Neural Networks, 8(3):714 735, 1997. 2
Anuroop Sriram, Abhishek Das, Brandon M. Wood, Siddharth Goyal, and C. Lawrence Zitnick. Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations. In ICLR, 2022. 5
14

---

Published in Transactions on Machine Learning Research (09/2022)
Oliver T. Unke and Markus Meuwly. PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges. Journal of Chemical Theory and Compulation, 15(6):3678-30%) 2019.
2
Oliver T. Unke, Stefan Chmiela, Michael Gastegger, Kristof T. Schütt, Huziel E. Sauceda, and Klaus-Robert Müller. SpookyNet: Learning Force Ficlds with Electronic Degrecs of Freedom and Nonlocal Effects. arXiu, 2105.00304, 2021. 2
Tian Xic and Jeffrey C. Grossman. Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. Physical Review Letters, 120(14):145301, 2018. 5
Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. In NeurIPS, 2021.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do Transformers Really Perform Badly for Graph Representation? In NewrIPS, 2021. 5
Shuo Zhang, Yang Liu, and Lei Xic. Molccular Mechanics-Driven Graph Neural Network with Multiplex Graph for Molecular Structures. In Machine Learaing for Molecules Workshop, NewrIPS, 2020. 2
15

---

Published in Transactions on Machine Learning Research (09/2022)
A Training and hyperparameters
All dataset and model ablations used the same basc model hyperparameters detailed in Table 3 Basc effective batch sizes (see Table 4 varied across dataset ablations in order to reach convergence in à reasonable amount of time across the large dataset size differences. MD17 and COLL ablations used a learning rate scheduler consistent with that of Gasteiger et al. (2021) linear warmup, exponential decay, and reduce on plateau. OC ablations only used reduce on platcau, with evaluations performed after a fixed number of steps (1k 5k) instead of after each epoch. MD17, COLL, OC-Rb, OC-XS, and OC-200k were trained on 40GB NVIDIA A100 cards with the remaining datasets trained 32GB NVIDIA V100 cards. We provide the hyperparaneterss for our best performing model variant, GemNNe-OOC-Large in Table 3
OC energies were standardized by subtracting the mean energy of the dataset and dividing by the standard deviation. Forces were divided by the same energy standard deviation in order to ensure consistency with their physical relationship:
=dF For MD17 and COLL, only the mean energy was subtracted Gasteiger de
et al. (2021) All models were trained within the ocp repository and optimized to the following loss function
T(X,2) 1 -x)(x))
N N
(1)
0
ge (X,, - (X, (x),))
+
a=1
where À corresponds to the energy coefficient, corresponds to the force coefficient, X are the atom coordinates, are the atomic numbers, So and go are learnable functions with shared parameters 0, E is the ground-truth energy, and F are the ground-truth forces. All model ablations make direct force predictions. Although models on the MD17 and COLL datasets generally perform better with gradient-based force predictions De,na (X, x) afo(X.x)
Sanp the same is not true for OC20 datasets. Cradient-based models on this dataset often run into numerical instabilities and hit NaNs vory carly in training. All models were optimizod using AMSGrad Reddi et al. (2018) and trained for a max number epochs 4 or until the learning rate has been exhaustively stepped, whichever comes first.
Additional experimental results
Early stopping. To obtain model results faster one might be tempted to stop model training early and draw conclusions from carly model performance. However, as shown in Fig. 6 carly results can be misleading and often do not reflect final performance. Fig. 8 shows that early stopping also has strong effect the choice of basis function, similar to chemical diversity. While large Bessel basis works substantially better early in training, this trend reverses when the model approaches convergence. Overall, results only become well-correlated with final OC20 accuracy late in training (see Fig. 9 Stopping training early should thus not be considered as a way of accelerating model development).
Complementary results. Complementary to the results the main paper, Figs. visualize similar plots across the out-of-das distribiinnt splits of the proposed datasets. Tables to separately show the test results for each OC20 test split.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: Model hyperparamcters for the bascline configuration and our GemNet--OC-Large model.
Hyperparameters
Base
GemNet-OCC.n OCrgge
No. spherical basis
7
No. radial basis
128
128
No. blocks
4
6
Atom embedding size
256
256
Edge embedding size
512
1024
Triplet edge embedding input size Triplct edge embedding output size Quadruplet edge embedding input size Quadruplet edge embedding output size Atom interaction embedding input size Atom interaction embedding output size Radial basis embedding size
64
64
64
128
32
64
32
32
64
64
64
64
16
32
Circular basis embedding size
16
16
Spherical basis embedding size
32
64
No. residual blocks before skip connection
2
2
No. residual blocks after skip connection
2
2
No. residual blocks after concatenation
1
4
No. residual blocks in atom embedding blocks
3
No. atom embedding output layers
3
Cutoff
12.0
12.0
Quadruplet cutoff
12.0
12.0
Atom edge interaction cutoff
12.0
12.0
Atom interaction cutoff
12.0
12.0
Max interaction neighbors
30
30
Max quadruplet interaction neighbors Max atom edge interaction neighbors
8
20
Max atom interaction neighbors
1000
1000
Radial basis function Circular basis function Spherical basis function Quadruplet interaction Atom edge interaction Edge atom interaction Atom interaction
Gaussian
Gaussian
Spherical harmonics Spherical harmonics
Legendre Outer
Legendre Outer
True
True
True
True
True
Truc
True
True
Direct forces
True
True
Activation
Silu
Silu
Optimizer
AdamW
AdamW
Force coefficient
100
100
Energy coefficient
1
EMA decay
0.999
0.999
Gradient clip norm threshold
10
10
Learning rate
0.001
0.0005
17

---

Published in Transactions on Machine Learning Research (09/2022)
Table 4: Training hyperparameters across all proposed datasets.
OC-20
OC-2M
OC-200k
OC-sub30
OC-XS
OC-Rb
OC-Sn
MD17
COLL
Effective batch size
128
64
16
32
No. GPUs
16
Max epochs
80
80
50
50
100
100
100
10000
1000
Rel. improvement (%)
60
OC20, Bessel
Kendall rank correlation
1.0
OC-2M to OC20
OC20, 32 Bessel
OC20 to OC20
OC-2M, Bessel
0.8
OC-2M to OC20, OOD)
OC-2M, 32 Bessel
OC20 to OC20, OOD
20
0.6
0.4
20
0.2
-40
10 Seen training samples
10
107
10a
Converged
Training samples seen
Converged
Figure 8: Effect of radial basis functions during train- ing force MAE. Bessel functions work best OC20 early in training, but this trend reverses at conver- gence. OC-2M exhibits a more consistent behavior.
Figure 9: Kendall rank correlation of model force MAEs during training to the final result. The correla- lion shows a drop early in training, which is likely due to the variance caused by the learning rate (LR) decay on platcau schodule. Correlation then increases again during late training. We found that LR variance has no impact on final converged results. Note that the converged OC-2M points have seen 56 M samples on average, putting it roughly par with early stopping.
Rel. improvement (%)
5.0
RBF projection 64
CRF projection 64
2.5
SRP projection
0.0
-2.5
-5.0
-7.5
MD17
COLL
OCXXS
OC-Rb
OC-Sh
OC-mb30
OC-200k
OC-2M
OC20
Figure 10: Impact of basis projection sizes on force MAE, Large projection sizes have a minor beneficial effect on OC20, while being detrimental on almost all smaller datasets.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Rel. improvement (%
OC-XS
OC-Rb
OC-Sn
OC-sub30
OC-200k
OC-2M
OC20
10a
4
10a
10
10a
10a
Edge embedding size
Number blocks
Batch size
Figure 11: Effect of batch size, edge embedding size, and number of blocks on force MAE for the OOD
validation set. Model size optima emerge on most datasets as we move to the OOD dataset. Note that OC-XS, OC-Rb, and OC-Sn use in-distribution catalysts.
Rel. improvement (%)
20
OC-XS
OC-Rb
OC-Sn
OC-sub30
OC-200k
OC-2M
OC20
20
6 6.A, 30 nghbrs.
6. cutoff
4 quadruplets
12 quadruplets
20 quadruplets
6 Bessel
32 Bessel
multi-order Bessel
Sph. harmonics SBF
Figure 12: Impact of cutoffs, neighbors, and basis functions on force MAE for the OOD validation set. The Bessel basis performs signficantly better on datasets with small chemical diversity, but worse on OC20 consistent with the in-distribution trends.
Rel. improvement (%)
OC-XS
t4 t
Apntuul
OC-Rb
OC-Sn
OC-sub30
OC-200k
OC-2M
OC20
No scaling factors
No symmetric MP
No quadruplet int.
No utom-atom int.
No aton-edge int.
No edge-atom int.
No global output MLP
No atom emb. in out
No atom MLP
Figure 13: Ablation studies for various components proposed by GemNet and GemNet-OC, based force MAE the OOD validation set. Some model changes are fairly consistent across OC20 datscts (c.g. symmetric message passing, quadruplet interaction), while others have varying effects across datasets.
19

---

Published in Transactions on Machine Learning Research (09/2022)
OC-XS
.52
.39
.11
.04
.07
.00
.08
.66
.14
114
.09
.07
OC-Sn
.52
.4
.24
.03
.16
.14
.14
OC-Sn, 0OD
.39
74
.26
.00
.08
.10
.14
.24
.14
.00
.07
.02
OC-XS, OOD
.11
24
26
.13
.20
.38
.02
.11
.09
.11
OC-Rb, OOD
.04
.00
.13
.22
.40
.31
.17
.07
.03
.06
MD17
.20
.52
.59
.29
.15
.04
004
OC-sub30, OOD
.40
.59
.18
.32
COLL
.07
.08
.02
.31
.29
.18
.33
.26
.31
.14
22
.17
.36
.30
.35
OC-sub30
.00
.03
10
.11
.17
.15
.32
.33
.17
.17
.09
26
22
.32
38
.32
-0.2
OC-200k
.09
.26
.17
.70
.16
.20
.33
.48
.30
.33
OC-200k, OOD
.08
.11
.07
.04
.31
.17
70
.21
.18
.20
.52
.30
.32
OC-XS, same trj
.16
.14
.16
.11
.39
.46
.40
.35
-0.0
OC-Rb
.14
.14
24
.03
.22
.66
.20
.18
.39
.54
.35
.46
.28
OC-2M
.14
14
.44
.17
.22
.23
-20
.37
.54
.57
60
.38
OC-2M, OOD
.00
.06
.36
.48
.52
.35
.57
.60
0.2
OC20
.07
.07
.30
.38
.30
.30
.40
46
.60
.60
.66
OC20, OOD
.02
.35
.32
.33
.32
.35
.28
.38
.58
.66
OC-XS
OC-Sn
OC-Sa, OOD
OC-XS, 00D
OC-Rb. 00D
MD17
OC-sub30, OOD
COLL
OC-sub30
OC-200k
OC-200k, OOD
OC-Rb
OC-2M
OC-2M, 0OD
0020
OC20, O0D
OC-XS, same
Figure 14: Kendall rank correlation of model force MAFs and resulting hierarchical clustering of datasets and different validation sets. Positive correlations are annotated. OC-2M is the most correlated to OC20, both in- and aut-of-distributiin
20

---

Published in Transactions on Machine Learning Research (09/2022)
Table 5: Results for the OC20 in-distribution (ID, also called "soparate trajectories") validation set and the three test lasks.
S2EF validation
S2EF test
IS2RS
IS2RE
Train
Energy MAE
Furce MAE Force cos EFwT
Energy MAE
Force MAE Force cos
EFwT AFFT
A Energy MAE
set
Model
meV. 4
meV/À
t
% f
meV 4
meV /À 4
% f
%
%
meV
SchNet
1360
73.7
0.112
0.00
1370
73.6
0.117
0.00
OC-2M
DimeNet
737
59.2
0.229
0.02
738
59.2
0.227
0.01
SpinConv
360
32.8
0.485
0.23
357
32.7
0.479
0.22
G'
275
25.5
0.574
1.15
271
25.5
0.571
1.19
20.0
55.1
435
GemNet-OO
226
22.5
0.610
1.89
226
22.5
0.610
1.94
22.4
56.5
405
CCONN
504
68.4
0,155
0.01
511
68.3
0.154
0.01
SchNet.
447
49.3
0.319
0.13
442
49.3
0.318
0.11
15.2
709
LoroeNet-
28.1
0,534
31.2
0.520
0.01
14.8
50.6
OC20
DimeNet*+-L-PP
360
28.1
0.564
0.00
350
28.0
0.564
0.00
25.6
52.4
502
PaiNN
248
29.3
0.511
0.88
15.6
49.5
442
SpinConv
287
37.0
0.479
0.09
261
26.9
0.548
0.82
21.1
53.7
424
GemNet-dTT
242
22.8
0.613
1.13
21.0
0.637
2. 2,40
33.8
59.2
GemNet-XL
18.1
3.30
34.6
62.7
GemNet-OC
172
17.9
4.59
17.9
40.7
OC20+
OC-MD
GemNee--OC-LB
153
17.8
0.688
5.30
150
17.8
0.688
5.44
GemNet--C--LPP
170
16.3
0.711
5.35
16.3
0.711
5.47
47.4
60.8
GemnNer-OC-I-Pi
331
Table 6: Results for the OC20
adsorbates validation set and the three test tasks.
S2EF validation
S2EF test
IS2RS
IS2RE
Train
Energy MAE Foree MAE Foree cos EFwT
Energy MAE Force MAE Force cos EFwT AFET ADwT Energy MAE
set
Model
meV L
meV/À
f
% f
meV
meV/A //A +
% f
%
%
meV
SchNet
1410
77.3
0.108
0.00
1340
71.1
0.111
0.00
OC-2M
DimeNet**
806
67.0
0.203
0.00
694
61.0
0.215
0.01
SpinCouv
375
35.6
0.479
0.05
350
33.7
0.466
0.09
Gem
309
29.3
0.560
0.21
269
26.5
0.557
0.59
15.9
50.5
442
GemNet-OC
258
25.2
0.600
0.45
235
22.9
0.597
1.09
19.5
52.2
416
CGCNN
300
74.6
0.132
0.00
632
72.8
0.137
0.00
SchNet.
497
57.4
0.286
0.00
486
52.9
0.295
0.04
12.8
774
ForceNet-large
32.0
0.520
28.3
0.521
0.01
12.2
45.2
OC20
DimeNett* LL-F-E
450
31.8
0.550
0.00
402
28.9
0.550
0.00
20.7
48.5
543
PaiNN
280
30.0
0.499
0.43
12.2
44.3
480
SpinConv
314
40.0
0.471
0.03
275
27.7
0.535
0.38
15.7
48.9
442
dTT
247
25.4
0.605
0.30
210
21.9
0.624
1.15
26.8
54.6
391
GemNet-XL
18.6
0.664
1.62
30.3
58.6
GemNet-OC
1.27
18.2
36.1
+
OC-MD
F..L-.
178
19.5
0.685
1.66
152
18.1
0.678
3.21
GemNNet-OC-LPP
195
17.9
0.707
1.70
16.6
0.700
3.18
40.4
56.4
GenNNes-CC-L-F-
336
21

---

Published in Transactions on Machine Learning Research (09/2022)
Table 7: Results for the OC20 out-of-distribution catalysts validation set and the three test tasks.
S2EF validation
S2EF test
IS2RS
IS2RE
Train
Energy MAE
Foree MAE Foree cos
EFwT
Energy MAE
Force MAE Force cos EFwT AFFT ADwT Energy MAE
Model
meV 4
mcV/A
f
%
meV
meV/A
% f
% f
%
meV
SchNet
1330
72.7
0.105
0.00
1340
71.7
0.114
0.00
OC-2M
+imeNet**
738
58.9
0.223
0.02
745
58.0
0.219
0.01
SpinConv
399
33.6
0,461
0.20
398
33.5
0.460
0.16
GemNet-dTT
374
27.3
0.533
0.97
341
26.8
0.536
0.76
15.3
54.8
454
GemNet-OC
288
24.0
0.576
1.68
279
23.3
0.584
1.44
19.8
56.7
416
CGONN
325
67.9
0.146
0.00
520
67.0
0.1
0.01
SchNet
515
02.0
0.297
0.10
528
50.9
0,296
0.06
14.6
767
ForoeNet-large
32.7
0.491
30,9
0.494
0.01
12.2
49.8
OC20
DimeNet*+ +L--F-
541
31.5
0.511
0.00
504
31.2
0.512
0.00
20.1
50.9
578
PaiNN
366
32.7
0.460
0.40
8.93
47.8
486
SpinCoav
397
39.7
0.454
0.03
350
28.5
0.519
0.46
15.9
53.9
457
GemNer-d'T
357
26.9
0.561
0.64
340
24.3
0.581
0.93
24.7
58.7
434
GemNet- XL
308
20.6
0.631
1.72
29.3
62.6
402
GemNet-OO
272
22.3
0.621
2.08
266
21.4
0.632
1.95
33.0
60.6
377
OC20+
OC-MD
GemNer-OOC-LE
278
23.1
0.617
1.86
282
22.1
0.628
1.82
GemNNet-OC-L-P
287
20.7
0.646
2.51
284
20.0
0.656
2.29
37.4
60.8
GemNet-OC-L.PP.
379
Table 8: Results for the OC20 out-of-distribution both (adsorbate and catalyst) validation set and the three test tasks.
S2EF validation
S2EF test
IS2RS
IS2RE
Train
Energy MAE
Foree MAE Foree cos EFwT
Energy MAE Force MAE Force cos EFwT AFET
ADwT
Energy MAE
set
Model
meV L
meV/À
f
% f
meV
meV/A /A +
% f
%
%
meV
SchNet
1490
89.6
0.110
0.00
1440
89.1
0.117
0.00
OC-2M
DimeNet**
940
77.5
0.214
0.00
867
73.7
0.226
0.00
SpinCouv
492
42.8
0.492
0.03
500
42.1
0.494
0.05
GemNet-dTT
475
36.0
0.559
0.11
413
33.5
0.573
0.23
15.7
58.8
420
GemNet-OC
370
31.0
0.606
0.23
355
28.5
0.620
0.52
16.6
60.3
391
CGCNN
731
85.2
0.134
0.01
768
85.1
0.144
0.00
SchNet.
705
68.5
0.285
0.00
706
65.5
0.299
0.01
14.8
806
ForceNet-large
41.2
0.516
37.5
0.530
0.00
11.5
52.9
OC20
DimeNet** -LFF
711
39.6
0.539
0.00
655
37.1
0.552
0.00
20.6
54.9
G12
PaiNN
470
40.5
0.493
0.13
10.1
52.2
474
SpinConv
487
48.2
0.486
0.01
459
35.6
0.555
0.14
14.0
58.0
425
dTT
415
33.5
0.596
0.10
394
29.6
0.622
0.30
25.1
62.2
384
GemNet-XL
24.5
0.670
29.0
66.7
338
GemNet-OC
344
27.1
0.36
25.2
31.3
342
+
OC-MD
CemmNet-OC-ILF
347
27.9
0.658
0.38
336
25.9
0.669
0.74
GemNNet-OC-L-P
357
25.1
0.685
0.50
23.1
0.696
0.93
37.1
63.7
GenNNes-C--L-F-
344
22


Reviews:
[{"review_id": "QVmotRTQqL", "reviewer": "Reviewer_TH9Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The proposed GemNet variant significantly reduce the training time in terms of GPU hours, without performance loss in general. \n- Experimental results are good, indicating a significant improvement from baseline methods including the existing version of GemNet.\n- The faster training also enables thorough empirical and ablation study on the impact of individual model components, which provide insights to the model designing of the the problem.\n- The constructed subset OC-2M for training can largely reflex the trend of model performance on variety of test sets and metrics. It can be used for development purposes with a much shorter turnaround time, which benefits future studies on the molecular simulation problem.\n\nWeaknesses\n\n- The method novelty and technical contribution are limited. Compared to the original GemNet, the proposed variant only differs in the structures by modifying blocks for efficiency purpose.\n- The results are mostly empirical. It is not clear whether the major performance boost comes from the changes in structure or more exhaustive tuning enabled by the faster training (e.g., as shown in Fig 3). It would be better to including more discussions or justifications on the changes.\n- It would make more impact if the propose method and empirical results can be generalize to other problems or tasks related to molecule geometry using the 3D GNNs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work focuses on the molecular simulation problem with the OC20 dataset. The authors propose a variant of GemNet, namely GemNet-OC to improve both efficiency and effectiveness on the OC20 tasks. The authors also conduct comprehensive empirical studies on models’ behaviors and different datasets, and construct a distilled data OC-2M that can further reduce training time and achieve a similar level of performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- The title and paper use the term GNN. In fact the proposed methods as well as baselines are not exactly typical GNNs. I suggest use 3D GNNs or geometric GNNs specifically to avoids confusions with general GNN structures like GCN, GIN and GAT.\n- Some recent related 3D GNNs studies are not discussed or compared in the results, e.g., [1], Graphormer [2], SphereNet [3].\n\n[1] Schütt et al., Equivariant message passing for the prediction of tensorial properties and molecular spectra. ICML 2021\n\n[2] Ying et al., Do Transformers Really Perform Badly for Graph Representation? NeurIPS 2021.\n\n[3] Liu et al., Spherical Message Passing for 3D Molecular Graphs. ICLR 2022.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1zxaeHZEWU", "reviewer": "Reviewer_AHac", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths === \n\n- (S1): The paper takes a very scientific approach to comparing different benchmarking datasets, taking them apart from multiple angles, and running many comparisons. While many of the final results aren't wildly surprising (e.g. a subsample of a large and diverse dataset is a better proxy for it than a different small and less diverse dataset), they are backed by a rigorous quantitative study. \n\n- (S2): Through careful design decisions, the authors push SotA on OC through their GemNet-OC model. The model modifications are often relatively small details, but again, the choices are backed by good experiments and analysis. \n\n- (S3): The paper is very well written, with practically zero typos, grammar errors or confusing sentences. The text is easy to read, and it gets the main points across rather precisely. Writing quality is better than the majority of papers accepted at NeurIPS/ICML/ICLR. \n\n \n\n=== Weaknesses === \n\n- (W1): The paper presents a large-scale but purely empirical study, and one could say the scientific novelty is limited. However, I would say this is not an important weakness; I much prefer a paper presenting a good deal of quantitative insights (as this one), instead of novel ideas with unclear applicability or significance. \n\n \n\n=== Questions === \n\n- (Q1): I haven't been closely following the recent SotA improvement on OC20, and thus I wonder how to contextualize Table 2 with regards to the original challenge. Am I correct in thinking that the OC20 challenge winner (Graphormer model from MSR Asia) is not in the table? I understand the recent models there surpassed the original Graphormer result significantly, but maybe it would still make sense to have it there to connect to the original challenge? \n\n- (Q2): The choice to use a fixed number of neighbors per atom instead of a fixed cutoff is interesting. Did the authors also try using a dynamically chosen cutoff, by computing all atom-atom distances, sorting them, and taking a cutoff such that a constant *average* degree is obtained? \n\n- (Q3): The paper shows that the optimal choice of the basis functions is wildly different depending on the level of chemical diversity in the dataset. Do the authors have any intuition for why this is the case? \n\n \n\n=== Nitpicks === \n\nHere I add some final nitpicks (some are incredibly nitpicky, but as I said the paper is already looking good): \n\n- \"In practise we apply a trained model\" - as far as I understand, \"practise\" is only used in British English and only as a verb. Since this here is a noun, perhaps \"practice\" would be more appropriate. \n\n- Maybe make the color of the word \"pink\" in the caption of Figure 1 exactly match the color of the pink boxes in the figure? \n\n- Figure 3 appears before Figure 2, maybe swap their numbers. \n\n- \"if we focussed on another dataset\" - I understand \"focussed\" is correct albeit uncommon, I wonder if the authors wanted to use \"focused\" instead. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the GemNet model developed for the Open Catalyst benchmark. It looks at various aspects of the dataset in conjunction with details of the model, coming up with several findings: (1) smaller and less diverse datasets than OC are a bad proxy for model development for OC, (2) a random subsample of OC is a good proxy, and (3) GemNet's performance can be improved significantly if one is directly optimizing for OC. Overall, the paper pushes SotA on OC20, as well as provides many guidelines and interesting observations for developing models on molecular datasets. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I think the paper is in good shape for publication, and I don't have significant changes to request. If the authors have any interesting thoughts regarding my questions, it would be nice to include these in the manuscript, otherwise it's fine to skip them. It would also be helpful to address the nitpicks, but this is just to slightly polish the paper and not a critical change.", "broader_impact_concerns": "I have no ethical concerns about the impact of this work. Better models can of course be used for good or bad, but this is expected and not specific to this paper. If anything, this work may reduce the biases in modelling choices (by discouraging transferring from small toy-ish datasets to challenging ones), which should have a good influence on the community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oC7LzIQgNX", "reviewer": "Reviewer_PhJC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n(1) Most studies and observations of this work are based on empirical results. These comprehensive results could be useful for the community.\n\n(2) The presentation of this work is great. The observations are clearly summarized and explained, with well-organized empirical results.\n\n(3) The improvements of GemNet-OC over the original GemNet are obvious. These designs are helpful for the community.\n\nWeaknesses:\n\n(1) Since this work is mainly empirical studies, the methodology-wise novelty is not strong. Having said this, the value of these empirical observations is still great.\n\n(2) To derive the comprehensive experiments in this work, I expect that the computational cost is quite high. It would be better to summarize how much computational cost have been used to obtain these results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work performs a comprehensive (empirical) study for GNNs over molecular simulation datasets. It first proposes GemNet-OC by improving the previous work GemNet. Then, comprehensive empirical analyses have been conducted to show that model components and hyperparameters can have disparate or even opposite effects between datasets. To alleviate this issue and facilitate future research, a refined dataset OC-2M has been developed and empirically verified as a convincing benchmark.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "(1) It would be clearer to the community if the consumed computational cost is provided. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the work is of limited technical novelty, all the reviewers appreciate its solid empirical study (e.g., reduced training cost, sota results with significant improvements over baselines), which is very helpful to the community. Therefore, I recommend "accept as is". 
Decision: Accept (poster)