Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations
Authors: Xiang Fu
Zhenghao Wu
Wujie Wang
Tian Xie
Sinan Keten
Rafael Gomez-Bombarelli
Tommi S. Jaakkola
Venue: Submitted to ICLR 2023 (2023)
Keywords: molecular dynamics
machine learning force field
benchmark
simulation-based evaluation
ML for Sciences

Abstract: Molecular dynamics (MD) simulation techniques are widely used for various natural science applications. Increasingly, machine learning (ML) force field (FF) models begin to replace ab-initio simulations by predicting forces directly from atomic structures. Despite significant progress in this area, such techniques are primarily benchmarked by their force/energy prediction errors, even though the practical use case would be to produce realistic MD trajectories. We aim to fill this gap by introducing a novel benchmark suite for ML MD simulation. We curate representative MD systems, including water, organic molecules, peptide, and materials, and design evaluation metrics corresponding to the scientific objectives of respective systems. We benchmark a collection of state-of-the-art (SOTA) ML FF models and illustrate, in particular, how the commonly benchmarked force accuracy is not well aligned with relevant simulation metrics. We demonstrate when and how selected SOTA methods fail, along with offering directions for further improvement. Specifically, we identify stability as a key metric for ML models to improve. Our benchmark suite comes with a comprehensive open source codebase for training and simulation with ML FFs to facilitate further work.

Under review as a conference paper at ICLR 2023
FORCES ARE NOT ENOUGH: BENCHMARK AND CRIT- ICAL EVALUATION FOR MACHINE LEARNING FORCE FIELDS WITH MOLECULAR SIMULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Molecular dynamics (MD) simulation techniques are widely used for various natural science applications. Increasingly. machine learning (ML) force field (FF) models begin to replace ab-initio simulations by predicting forces directly from atomic structures. Despite significant progress in this area, such techniques are primarily benchmarked by their force/energy prediction errors, even though the practical case would be produce realistic MD trajectories. We aim to fill this gap by introducing a novel benchmark suite for ML MD simulation. We curate representative MD systems, including water, organic molecules, peptide, and materials, and design evaluation metrics corresponding to the scientific objectives of respective systems. We benchmark collection of state-of-the-an (SOTA) ML FF models and illustrate, in particular, how the commonly benchmarked force accuracy is not well aligned with relevant simulation metrics. We demonstrate when and how selected SOTA methods fail, along with offering directions for further improvement. Specifically, we identify stability as key metric for MI. models to improve. Our benchmark suite comes with comprehensive open source codebase for training and simulation with ML FFs to facilitate further work.
INTRODUCTION
Molecular Dynamics (MD) simulations provide atomistic insights into physical phenomena in materials and biological systems. Such simulations are typically based on force fields (FFs) that character- ize the underlying potential energy surface (PES) of the system and then use Newtonian forces simulate long trajectories (Frenkel & Smit, 2001). The PES itself is challenging to compute and would ideally done through quantum chemistry which is computation- ally expensive. Traditionally, the alternative has been parameterized force fields that are surrogate models built from empirically chosen functional forms (Halgren, 1996). Recently, machine learning (MI.) force fields (Unke et al., 2021b) have shown promise to acceler- MD simulations by orders of magnitude while being quantum chemically accurate. The evidence supporting the utility of ML FFs often based their accuracy in reconstituting forces across cases (Faber et al., 2017). The evaluations invariably do not involve simulations. However, we show that force accuracy alone does not suffice for effective simulation (Figure 1).
Sebility
Sylver recorte
RDF MAE
Demple Party
Cin Netta
Force MAE
Figure 1: Results water- 10k. Models sorted by force mean absolute eπror (MAE) in descending order. High stabil- ity and low RDF MAE are bet- ter. Performance in force error does not align with simulation- based metrics.
MD simulation not only describes microscopic details on how the system evolves, but also entails macroscopic observables that char- acterize system propertics. Calculating meaningful observables often requires long simulations sample the underlying equilib- rium distribution. These observables are designed to be predictive of material properties such as diffusivity in electrolyte materials (Webb et al., 2015), and reveal detailed physical mechanisms, such as the folding kinetics of protein dynamics (Lane et al., 2011). Although these observables are critical products of MD simulations, systematic evaluations have not been sufficiently studied in existing

---

Under review as a conference paper at ICLR 2023
literature. To gain insight into the performance of existing models in simulation setting, we propose a series of simulation-base benchmark protocols. Compared to the popular multistep prediction task in the learned simulator community "Sanchez-Gonzalee et al., 2020), MD observables focus on distributional quantities. The exact recovery of the trajectories given the initial conditions is not the ultimate goal.
Evaluating learned models through MD simulations requires careful design over the selection of systems, the simulation protocol, and the evaluation metrics: (1) A benchmark suite should cover diverse and representative systems to reflect the various challenges in different MD applications. (2) Simulations can be computationally expensive. An ideal benchmark needs to balance the cost of evaluation and the complexity of the system so that meaningful metrics can be obtained with reasonable time and computation. (3) Selected systems should be well studied in the simulation domain, and chosen metrics should characterize the system's important degrees of freedom or geometric features.
Are current state-of-the-art (SOTA) ML FF's capable of simulating a variety of MD systems? What might cause a model to fail in simulations? this paper, our aim is to answer these questions with a novel benchmark study. The contributions of this paper include:
We introduce a novel benchmark suite for ML MD simulation with simulation protocols and quantitative metrics. We perform extensive experiments to benchmark collection of SOTA ML models. We provide complete codebase for training and simulating MD with ML FFs to lower the barrier to entry and facilitate future work.
We show that many existing models are inadequate when they evaluated on simulation- based benchmarks, even when they show accurate force prediction (as shown in Figure 1).
By performing and analyzing MD simulations, we summarize common failure modes and discuss the causes and potential solutions to motivate future research.
RELATED WORK
ML force fields learn the potential energy surface (PES) from the data by applying expressive
regressors such as kernel methods ( (Chmiela et al., 2017) and neural networks on symmetry-presernee
representations of atomic environments (Behler Parrinello, 2007; Khorshidi & Peterson, 2016; Smith et al., 2017: Artrith et al., 2017: Unke & Meuwly, 2018; Zhang et al., 201 8b;a; Kovács et al., 2021; Thölke & De Fabritiis, 2021; Takamoto et al., 2022). Recently, graph neural network architectures (Gilmer et al., 2017; Schütt et al., 2017; Gasteiger al., 2020; Liu et al., 2021) have gained popularity as they provide a systematic strategy for building many-body correlation functions capture highly complex PES. In partieular, equivariant representations have been shown powerful in representing atomic environments (Satorras et al., 2021; Thomas et al., 2018; Qiao et al., 2021; ;; Schütt al., 2021; Batzner al., 2022: Gasteiger et al., 2021: Liao & Smidt, 2022). leading to significant improvements in benchmarks such as MD17 and OC22/20. Some works presented simulation-basse results (Unke et al., 2021a; Park et al., 2021; Batzner et al., 2022; Musaelian al., 2022) but do not compare different models with simulation-baat based metrics.
Existing benchmarks for ML force fields (Ramakrishnan al., 2014; Chmiela et al., 2017) mostly focus on force/energy prediction, with small molecules being the most typical systems. The catalyst- focused OC20 (Chanussot et al., 2021) and OC22 (Tran et al., 2022) benchmarks focus on structural relaxation with force computations, where force prediction is part of the evaluation metrics. The structural relaxation benchmark involves relaxation process of around hundreds of steps, and the goal is to predict the final relaxed structure/energy. These tasks do not characterize system properties under a structural ensemble, which requires simulations that are millions of steps long. Several recent works (Rosenberger et al., 2021) have also studied the utility of certain ML FFs in MD simulations. In particular, Stocker et al. 2022 uses GemNet (Gasteiger et al., 2021) to simulate small molecules in the QM7-x (Hoja et al., 2021) dataset, with a focus on simulation stability. Zhai et al. 2022 applies the DecpMD (Zhang et al., 2018a) architecture to simulate water and demonstrates its shortcoming in generalization across different phases. However, existing works focus on a single system and model, without proposing evaluation protocols and quantitative metrics for model comparison. Systematic benchmarks for simulation- based metries are lacking in the existing literature, which obscures the challenges in applying ML FF for MD applications.
2

---

Under review as a conference paper at ICLR 2023
(
(b)
(C)
(d)
Figure Visualization of the benchmarked systems. (a) MD17 molecules: Aspirin, Ethanol,
Naphthalene, and Salicylic acid. (b) 64 water molecules. (c) 512 water molecules. (d) Alanine dipeptide. (e) LiPS.
Table 1: Dataset summary. PBC stands for periodic boundary conditions. dipeptide uses Metadynamics with implicit solvation.
"Simulation of alanine
Dataset
System Type
PBC
#Atoms
Simulation Length
Objective
MD17
Small molecule
X
9-21
300 ps (600k steps)
Interatomic distances
Water
liquid
v
192
500 ps (500k steps)
RDF, Diffusivity
Water-large
liquid
V
1536
150 ps (150k steps)
RDF, Diffusivity
Alanine dipeptide
Peptide
X
22
5 ns (2.5M steps)
Dihedral angle amalysis
LiPS
solid-state state materials
83
50 ps (200k steps)
RDF, Diffusivity
3 PRELIMINARIES
Training, An ML FF aims to learn the potential energy surface ((x) E R as function of atomic coordinates x E NR (N is the number of atoms), by fitting atom-wise forces ((r) and energies from training dataset(<x;; F,, J Mi where æ; E RNx3 F E g E R. For evaluation, the test force prediction accuracy is used as a proxy to quantify the quality of the learned PES. The force field learning protocol has been well established (Unke et al., 2021b).
MD simulation. Simulating molecular behaviors requires integrating a Newtonian equation of motion with forces obtained by differentiating É(x): F(x) = (x) To mimic desired thermodynamic conditions, an appropriate thermostat and barostat are chosen augment the equation of motion with extended variables. The simulation produces a time series of positions: {Iz E Nx317 J==0- where is the temporal order index, and T is the total simulation steps. To evaluate the simulation quality, we propose metrics based-on on well-established observables in the respective types of systems. Definitions of benchmarked observables are Appendix A.
DATASETS AND METRICS
Popular benchmark datasets, such MD17, focus the force prediction task for gas-phase small molecules. However, successes in these tasks are not sufficient evidence for (1) capturing complex interatomic interactions that are critical for condensed phase systems; and (2) recovery of critical simulation observables that cannot be directly indicated by force prediction accuracy. This work focuses on atomic-level MD simulations that manifest complex intermolecular interactions at multiple scales. We choose systems that (1) have been frequently used in force field development (Henderson, 1974); (2) cover diverse MD applications such materials and biology; and (3) can be simulated within resonable time and compute. Beyond force predictions, we conduct simulations and benchmark observables that reflect the actual simulation quality, along with stability and computational efficiency. The selected systems are summarized in Table 1.
Quantifying simulation stability. ML FFs can produce unstable dynamics, the learned force field may not extrapolate robustly to the undersampled configuration space. As result, the trajectories can enter nonphysical states that are not meaningful for observable calculations. Therefore, we closely monitor how much the simulated structure deviates from the physical configurations, with the radial distribution function (RDF) for condensed phase systems and bond lengths for flexible molecules. We say that simulation becomes "unstable" when the deviation exceeds threshold, which implies sampling of highly nonphysical structures. We then use the time duration for which a model remains stable in simulations to measure its stability. The ensemble statistics are only computed over the stable part of the simulated trajectories: the simulation trajectory before the first occurrence of instability. Details on the stability criterion are included in Appendix A.
3

---

Under review as a conference paper at ICLR 2023
Table 2: Models benchmarked in this work. The translation/rotationn symmetries are respected by the
feature representation at every layer. Number of parameters on the MD17 dataset are reported.
Model
Symmetry Principle of Geometric Features
Energy Conservation #Parameters
DcepPot-SE (Zhang et al., 2018b)
B(3)-invariant
104M
SchNet (Schüitt et al., 2017)
E(3)-invariant
0.12M
DimeNet (Gasteiger al., 2020)
M(3)-invariant
2.1M
PaiNN (Schütt et al., 2021)
SE(3)-qquivariant
V
0.59M
SphereNet (Liu et al., 2021)
B(3)-invariant
V
1.8M
ForceNet (Hu al., 2021)
Translatooniinariia
X 11.37M
GemNet-T (Gusteiger et al., 2021)
B(3)-in-ariant
V
1.89M
GemNet-dT (Giatteiger et al., 2021)
SH(3) equiivarit
X 2.31M
NequlP (Batzner et al., 2022)
EQyequivvriant
1.05M
MD17 (Chmiela et al., 2017) dataset contains AIMD calculations for eight small organic molecules and is widely used a force prediction benchmark for ML FFs. We adopt four molecules from MD17 and benchmark the simulation performance. In addition to force error, we evaluate the stability and the distribution of interatomic distances h(r). For each molecule, we randomly sample 9,500 configurations for training and 500 for validation from the MD17 database. We randomly sample 10,000 configurations from the rest of the data for force error evaluation. We perform five simulations of 300 ps for each model/molecule by initializing from 5 randomly sampled testing configurations, with a time step of 0.5 fs, at 500 K temperature, under a Nosé-Hoover Hoover thermostat.
Water is arguably the most important molecular fluid in biological and chemical processes. Due to its complex thermodynamic and phase behavior, it poses great challenges for molecular simulations. In addition force error, evaluate simulation stability and recovery of both equilibrium statistics and dynamical statistics, namely the glement-conditionel RDFs and liquid diffusion coefficient. Our dataset consists of 100,000 structures collected every 10 fs from a ns trajectory sampled at equilibrium and a temperature of 300 K. We benchmarked all models with various training+validation dataset sizes (1k/10k/90k randomly sampled structures) and used the remaining 10,000 structures for testing. We performed 5 simulations of 500 ps by initializing from 5 randomly sampled testing configurations, with a time step of 1 fs. at 300 K temperature, under Nosé Hoover thermostat. We additionally evaluate model generalization to larger system of 512 water molecules with 5 simulations of 150-ps.
Alanine dipeptide features multiple metastable states, making it classic benchmark for the develop- ment of MD sampling methods and force field development (Head-Gordon et al., 1989; Kaminski et al., 2001). Its geometric flexibility is well represented by the central dihedral (torsional) angles and vb. Our reference data are obtained from simulations with explicit water molecules, with detailed protocols described in Appendix B. For faster simulation, we learn implicitly solvated FF following a protocol similar to Chen et al. (2021). Our task is more challenging in that it aims to learn the implicitly solvated atomistic FF rather than the implicit solvation correction in Chen et al. (2021). To facilitate accelerated sampling, we apply metadynamics with ø and t/ as the collective variables. We evaluate force prediction, simulation stability, and free energy surface (FES) reconstruction F(ò,, v)). Our dataset consists of 50,000 structures dumped every 2 ps from 100 ns trajectory at a temperature of 300 K. We used 38,000 randomly sampled structures for training, 2,000 for validation, and the rest as a test set. We performed 6 simulations of 5 ns by initializing from six local minima on the FES (Figure 5) with a time step of 2 fs at 300 K, and under Langevin thermostat to mimic random noise from solvation effects. More information on our simulation protocols can be found in Appendix B.
LiPS is a crystalline superionic lithium conductor relevant to battery development and representative system for MD simulation usage in studying kinetic properties in materials. We adopt this dataset from Batzner et al. 2022, and benchmark all models on their force error, stability, RDF recovery, and Li-ion diffusivity coefficient. The dataset has 25,000 structures in total, from which we use 19,000 randomly sampled structures for training, 1,000 structures for validation, and the rest for computing force error. We conduct 5 simulations of 50 ps by initializing from 5 randomly sampled testing configurations, with a time step of 0.25 fs, at 520 K temperature, under a Nosé-Hoover thermostat.
5 EXPERIMENTS
Benchmarked models. We adopt the Open Catalyst Project implementation of SchNet (Schütt et al., 2017), DimeNet (Gasteiger et al., 2020), ForceNet (Hu et al., 2021), PaiNN (Schütt et al.,
4

---

Under review as a conference paper at ICLR 2023
Table 3: Results on MD17. Darker green color indicates better performance. For all results, force MAE is reported in the unit of [me V/ and stability is reported in the unit of [ps]. The distribution of interatomic distances hr) MAE is unitless. FPS stands for frames per second. For all metrics indicates the lower the better, and (f) indicates the higher the better. Standard deviation from
(L)
simulations is in subscript for applicable metrics.
Model
SE
SchNet
DimeNet
PaiNN
SphereNet
ForceNet
GemNes-T
GemNer-dT
NequIP
Molccue
Aspirin
Force (1)
21.0
35.6
10.0
8.2
3-4
22.1
3.3
5.1
2.3
Stubility
9rss
26gzz9
12)
150
p41
182
72:w0)
192
300
Mri (L)
0.63
0.36 57)
0.044...00
8.04
0.88io.cos
0.56
0.04jmm3
04220000
PPS (f)
88.0
108.9
20.6
85.8
17.5
137.3
28.2
56.8
84
Bisanol
Force
8.9
16.8
4.2
5.0
1.7
10.9
2.1
1.7
1.3
Stability
W000))
(47/2001
28:
33
Willoo
c69
300gg
800gg)
Mrr)
0.21 11)
0.15
0.15
0.15
0.56
m.10
0.08
0.0%
FPS
101 O
112.6 6
21.4
87.3
30.5
191.1
27.1
54.3
8.9
Naphthalene
Force
13.4
22.5
5.7
9.8
1.9
Stability
2000
18,3)
85
300,g)
6:31
20
300gg)
hir)
0. 0.11
BL08
0,100
D. 1.13
0.14
1.02
0
D. 0.12mmm)
FPS
109.3
110.9
19.1
92.8
18.3
120.2
27.7
53.5
8.2
Salicylic Acid
Force
14.9
9.6
6.5
4.0
Stability
300gg
300gg)
73
281
36 no:
Lon
26
(4)
300gg)
hir)
0.033
0 0.06
(.03)
0..66 21.4
0.36 12g/2
0.088
0.07 (uus))
FPS
94.6
111.7
19.4
90.5
28.5
52.4
8,4
(4)
Aupiri
(b)
(c)
Noat Baloro
(d)
Bally Add
Thanlity
10%
Fore MAD
Form MAD
Der
Deggal
Fore MAG
Vame MAG
Figure 3: Hcad-to-heer comparison of force MAE vs. Stability and h(r) MAE on MD17 molecules. Models are on the x-axis and are sorted according to force error in descending order. High stability and low h(r) MAE mean better performance. Error bars indicate 95% confidence intervals.
2021), GemNet- T/dT (Gasteiger et al., 2021), and the official implementation of DeepPot-SE (Zhang et al., 2018b), SphereNet (Liu et al., 2021), ) and NequlP (Batzner et al., 2022). A summary of all benchmarked models are in Table 2. These models have been popular in previous benchmark studies for force/energy prediction. They use different representations for atomistic interactions and respect different levels of euclidean symmetry. We follow all original hyperparameters introduced in the respective papers and only make minimal adjustments when the training is unstable. More details on the hyperparameters can be found in Appendix C.
Key observations. We make two key observations as evidenced in the experimental results:
1. Despite being widely used, force prediction is not sufficient for evaluating ML FFs. It generally does not align with simulation stability and performance in estimating ensemble properties.
2. While often neglected, stability can be a major bottleneck for ML FFs. Lower force error and more training data does not necessarily give rise to more stable simulations, suggesting stability as a fundamental consideration for comparison and model design.
We next go through experimental results of all four datasets in detail to demonstrate the key observa- tions while making other observations.
MD17. As shown in Table 3, more recent models that lie on the right side of the table generally achieve a lower force error, but may lack stability. Figure 3 selects results from Table to demonstrate the non-aligned trends of force prediction performance vs. simulation performance, which supports key observation 1. SphereNet and GemNet-TTdTT can attain a very low force error for all four molecules, but often collapse before the simulation finishes. This observation constitutes key observation 2. We note that although the stable portion of simulated trajectories produced by SphereNet and GemNet- T/dT can recover the hr)) relatively accurately, stability will become bigger issue when the statistics
5

---

Under review as a conference paper at ICLR 2023
Table 4: Results Water-1Ok. RDF MAE is unit-less, Diffusivity is computed by averaging runs from 5 random imitial configurations and its MAE is reported in the unit of [10 m² ns].. The reference diffusivity coefficient 2.3 10 m² /s.
DecpPot-SB
SchNet
DimeNet
PaiNN
SphereNet
ForceNer
GemNer-T
GomNet-dT
NoqulP
Forze (4)
5.8
9.5
1.4
5.1
16.1
10.9
0.7
133
1.5
Suubility (1)
247(147)
232,50)
30
12
500ga
Tes:
26,7)
Pray
500,0)
RDF (4)
0.0f
0.6
0.90
pu89;
0.79
0.22
(.42
0.06
aD. (3)
0.05
0.30gg.02)
(.
(o.o
q.40
0.55
16.0..)
0.
0.05
RDF (2r.0 (U)
0.
0.67.0.04)
0.21
1.14
1.34
0.20
0.42
0.27
Diffusivity
0.04
1.90
FPS
91.0
78.9
17.9
71.8
3.1
67.6
11.3
33.7
3,9
Diffusiivtt MAE [1.0 1m
2.0
1.2
400
RDF 0: MAE
1.5
300
0.8
Stabilty
RDF
200
0.2
10
20
2.5
7.5
10.0
12.5
0.0
25
5.0
10.0
12.5
Force MAE [meViA]
Force MAE [meViA]
Force MAE [meViA]
Model
DeepPot-SE
SchNet
DimeNet
ForceNet
GemNet-T
GemNet-dT
NequiP
Figure 4: Comparison of force MAE vs. stability (Left), force MAE vs. RDF MAE (Middle). and force MAE vs. Diffusivity MAE (Right) on the water benchmark. Each model trained with three dataset sizes. The color of point indicates the model identity, while the point size indicates the training dataset size (small: 1k, medium: 10k, large: 90k). Metries infeasible to extract from certain model/dataset size (e.g.. Diffusivity for unstable models) are not included.
of interest require long simulations, demonstrated in other experiments. On the other hand, despite having a relatively high foree error, DeepPot-SE performs very well on simulation- based metrics on all molecules except for Aspir r (Figure 3). With the highest molecular weight, Aspir rån is indeed the hardest task in MD17 in the sense that all models attain high force prediction errors on PaiNN also attains competitive simulation performance while its foree error is not among the best.
We further observe that good stability does not imply accurate recovery of trajectory statistics. Although ForceNet remains stable for Ethanol and Naphthalene, the extracted l(r)) deviates lot from the reference (Table 3), indicating that ForceNet does not learn the underlying PES correctly, possibly due to its lack of energy conservation and rotational equivariance. Overall, NequlP is the best-performing model on MD17. It achieves the best performance in both force prediction and simulation-basee metrics for all molecules while requiring the highest computational cost. More detailed results on MD17 including a study on stability's 's relation with training epochs and individual f((r) are included in Appendix C.
Water. Under different challenges posed by a condensed phase system, key observation 1 and 2 still evident according to Table 4: GemNet-T/dT and DimeNet are the top-3 models in terms of force prediction, but all lack stability. The water diffusivity coefficient requires long (100 in our experiments) trajectories estimate and thus cannot be extracted for unstable models. Like MD17, DeepPot-SE. does not achieve the best force prediction performance but demonstrates decent stability and highly accurate recovery of simulation statistics. Interestingly, SphereNet has high force error but is highly stable. However, the properties are not accurately recovered.
Figure 4 further compares model performance with different training dataset sizes. Key observation I and 2 are clearly shown: Models located on the left of each scatter plot have very low force error but may have poor stability or high error in simulation statistics. More specifically, although more training data almost always improve force prediction performance, its effect on simulation performance is not entirely clear. On the one hand, GemNet- T/dd,, DimeNet, and ForceNet are not stable even when under the highest training data budget. On the other hand, observe clear

---

Under review as a conference paper at ICLR 2023
Table 5: Results alanine dipeptide. #Finished is the number of simulations stable for 5 ns. MAE of Fo) and (()) are reported in the unit of [kJ/mol]]
DeepPot-SE
SchNet
DimeNet
ForceNet
GemNet-T
GemNet-dTT
NequlP
Foree (L)
272.1
217.0
239.0
284.7
233.5
219.7
215.6
#Finished (1)
0/6
0/6
0/6
0/6
0/6
0/6
S/6
Stability (1)
((0)
030)
0(0)
0,0)
18(27)
0(0)
(168
F(6) (L)
108
F(V) (L)
126
FPS
51.3
42.4
12.1
99.1
15.0
36.5
8.3
Table 6: Results on LiPS. Li-ion Diffusivity coefficient is computed by averaging 5 runs from
random initial configurations. The reference Li-ion diffusivity coefficient 1.35 x 10-9 m²/s.
DeepPot-SH
SchNet
DimeNet
ForceNet
GemNet-T
GemNet-dT
NequlP
Force (L)
40.5
28.8
3.2
12.8
1.3
1.4
3.7
Stability (1)
4(3)
50(m)
48(4)
26(5)
50gm)
50(0)
50(0)
RDF (L)
(.27
0.04g0.00
0.05
0.51(a.08)
0.04
0.04
0.04.a.01)
Diffusivity (L)
0.38
0.30
0.24
0.28
0.34
FPS (1)
66.1
35.2
14.8
72.1
16.9
43.5
8.2
improvement of DeepPot-SE when more training data is used. NequlP again the best performing model, achieving very low force error, excellent stability, and accurate recovery of ensemble statistics, even under the lowest data budget of 1,000 training+validation structures. However, when the training dataset is sufficiently large (90k), DeepPot-SE has equally good results as NequlP while being more than 20 times faster - dataset size also influences the model choice for a certain dataset. More results, including tables for water-1k/90k, study on model size, and generalization to Water-large are included in Appendix C.
Alanine dipeptide poses unique challenges in sampling different metastable states separated by high free energy barriers. Table shows all models have high force errors due to the random forces introduced by the lack of explicit account of water molecules. Although the force errors in the same order of magnitude, all models except NequlP fail to simulate stably. The FES reconstruction task requires stable simulation for the entire ns. NequlP the only model that manages to finish five simulations out of six but produces inaccurate statistics. All other models are not stable enough produce meaningful results. We further analyze the results on this task in Section 6.
LiPS. Compared to flexible molecules and liquid water, this solid material system features slower kinetics. From Table we observe that most models are capable of finishing 50-ns simulations stably. In this dataset, the performance on diffusivity estimation and force prediction align well. observe that both GemNet-T and GemNet-dT show excellent force prediction, stability, and recovery of observables, while GemNet-dT is 2.6 times faster. The better efficiency comes from the direct prediction atomic forces F instead of taking the derivative = 0E/Ox, which also makes GemNet-dT not energy-conserving potential issuc we further discuss in Section 6.
Implications on model architecture. More recent models utilizing (E())E(3)equivaria) rep- resentations and operations such GemNet-dT and NequlP are more expressive and capture interatomic interactions more accurately. This is reflected by their very low force eтror and accurate recovery of ensemble statistics when not bottlenecked by stability. Moreover, NequlP shows that excellent accuracy and stability can be simultaneously achieved. The stability may come from errit--equvvariance and the explicit architecture in manipulating higher-order geometric tensors. We believe further investigations into the extrapolation behavior induced by different equivariant geometric representations and operations (Batatia et al., 2022) is fruitful direction in designing more powerful ML FFs.
6 FAILURE MODES: CAUSES AND FUTURE DIRECTIONS
A case study alanine dipeptide simulation. NequlP achieves decent performance on all our tasks but fails on alanine dipeptide. It is also the only model that can simulate stably for 5 ns. Figure (a) demonstrates how NequlP fails reconstruct the FES: it does not manage sample much of the
7

---

Under review as a conference paper at ICLR 2023
Reference
BkUmol
NequlP
[KJmml]] 40
30
F[JImol]]
1 (dogroo)
30
20
10
-00
NequiP
[dogroc]
107
0 [degrae)
100
(degroo)
-180
V Idegree]
(c)
140
Training Data
-346
LPSS, NVE ensemble
GamNett Water
v (degree)
50
to"
# Dela point
Prediceed Eusa (el)
-348
GemNec-dT
GemNet
Slability [ps]
-300
-90
30
150
-800
20
$ [degree]
0
90
160
10'
Simu abon Time (fs)
Time Step [1s]
0,5
0.25
Figure 5: (a) Ramachandran plots of the alanine dipeptide FES reconstructed from 5-ns reference vs. 5-ns NequlP simulation, both using MetaDynamics. (b) ())) and F(t) of alanine dipeptide extracted from reference simulation vs. from NequlP simulation. (c) (á, tb) distribution of the alanine dipeptide training dataset. The six initialization points are marked with stars. NequlP fails to remain stable when the simulation starts from the point marked with black color. (d) Model-predicter total energy as a function of simulation time when simulating the LiPS system using the NVE ensemble. (e) On woker-1Ok, stability does not improve when the time step is reduced for GemNet-T.
transition regions and the configuration space with ø E [0, 180%). Figure (b) demonstrates the reconstructed FES, which significantly deviates from the reference. This failure can be partially explained by Figure (c), the training data distribution produced by the reference potential. The relatively high-energy (low-density) regions are exactly those that are not reachable by NequlP. Even though our MD trajectory is well-equilibrated, the relative difference in populations of different meta-stable states creates data imbalance, making it more challenging for the model to learn PES for higher-energe energy configurations where density relatively low. our experiments, we observe that simulations starting from the low-density meta-stable state (e.g., black star marked in Figure 5 (c)) tend fail. This implies that generalization across different regions in the conformational space is an important challenge for MI. FFs. To prevent MI. FFs from sampling nonphysical regions, which is a common precursor to failed simulation (Figure 6), one can deliberately include distorted and off-equilibruum geometries in the training data to improve model robustress(Stocker al., 2022). Alternatively, one can resort to active learning (Wang et al., 2020b; Vandermause et al., 2020; Schwalbe-Kond et al., 2021) to acquire new data points based on model uncertainty.
Energy conservation. Models that directly predict forces may not conserve energy. Figure 5 (d) demonstrates the evolution of model-predicted total energy for selected models LiPS, in miero- canonical (NVE) ensemble. The energy of an isolated system in the NVE ensemble in principle conserved. We observe that GemNet-T conserves energy, whereas GemNet-dT fails conserve the predicted total energy. The existence of non-conservative forces breaks the time reversal symmetry and, therefore, may be able to reach equilibrium for observable calculation. However, in our experiment, GemNet-d' performs well on the LiPS dataset when coupled with a thermostat. Previous works (Kolluru al., 2022) also found that energy conservation is not required for SOTA performance on OC20. The usability of non-conservative FF' in simulations requires further careful investigations.
Simulation instability a major bottleneck for highly accurate models such as GemNet-T to fail on several simulation tasks. Moreover, in our water experiments, we find a larger amount of training data does not resolve this issue for GemNet-T/dT and DimeNet (Figure 4). We further experiment with smaller simulation time steps for GemNet-T on water (Figure 5 (e)), but stability still does not improve. On the other hand, Stocker et al. (2022) demonstrates that the stability of GemNet improves with larger training sets QM7-x, which includes high-energy off- off-equilibrium geometries obtained from normal mode sampling. We hypothesize that including these distorted geometries may improve the model's robustness against going into nonphysical configurations. We also observe that
8

---

2019; Fu Under review as a conference paper at ICLR 2023
(a)
NequiP, Alanine dipeptide
Max Force (eViA)
1d"
10"
10'
10"1
3622 fs
3702 is
== 3704 fs
3706 fs
t= 3716 fs
3050
3608
3e50
3/00
Simulation Time [1s]
(b)
GemNel-T, Water
Max Force (eV/AA
10
10'
10'
20000
20360 Simulation Time [[s]
28320
25340
20000
28330 Is
28370 is
= 28372 Is
28373 fs
28376 fs
Figure 6: Examples of simulation collapse when applying (a) NequlP to alanine dipeptide and (b) GemNet-T to water. The y-axis shows the maximum force observed on any atom in the system at a certain time step. An orange cross indicates visualized time steps. Notable nonphysical regions circled. The collapse usually happens within a very short period of time after the initial local errors.
the simulation can collapse within a short time window after a long period of stable simulation, as visualized in Figure 6. In both cases, we observe that the nonphysical configurations first emerge at local regions (circled), which cascade to the entire system very quickly as extremely large forces being produced and subsequently integrated into the dynamies. At the end of the visualization, the bonds in the alanine dipeptide system are broken. Therefore, the oocal-deseriptor-base model predicts very small forces. For the water system, the particles are packed in finite periodic box. The nonphysical configurations exhibit incorrect coordination structures and extremely large forces. Regarding stability, past works found adding noise paired with denoising objective during training helpful in improving out-of-distributie test performance on OC20 (Godwin et al., 2021] 1), and in stabilizing learned simulations SaaeheezGGnzzlezzzzz et al., 2020). Another relevant line of work in coarse- -grained MD) simulation has studied regularization with empirical "prior energy" (Wang al., 2019b) and post-prediction refinement (Fu et al., 2022) to battle simulation instability.
NequlP
CONCLUSION AND OUTLOOK
We have introduced a diverse suite of MD simulation tasks and conducted a thorough comparison of SOTA ML FFs to reveal novel insights into ML for MD simulation. As shown in our experiments, benchmarking only force error is not sufficient, and simulation-based metrics should be used to reflect the practical utility a model. We demonstrate case studies on the failure of existing training schemes/models to better understand their limitations. The performance of model can be highly case-dependent. For more challenging MD systems, more expressive atomistic representations may be required. For example, recent work has explored non-local descriptors (Kabylda al., 2022) aiming at capturing long-range interactions large molecules. Strietly local equivariant representations (Musaelian al., 2022) are studied for very large systems where computational scalability is critical. New datasets (Eastman et al., 2022) and benchmarks have been playing an important role in inspiring future work. This work focuses on atomic-level simulations. Simulating large systems such as polymers and proteins at atomic resolution would be too expensive for existing models. Learning coarse coarse-grained MD (Wang et al., 2019b; Wang & et al., 2022) is another avenue to accelerate MD at larger length/time scales.
The possibility of MI. in advancing MD simulation is not limited to MI. force fields. Enhanced sampling methods enable fast sampling of rare events and have been augmented with ML tech- niques (Schneider et al., 2017: Sultan et al., 2018; Holdijk et al., 2022). Differentiable simula- tions (Schoenholz & Cubuk, 2020; Wang et al., 2020a; Doerr et al., 2021; Ingraham et al., 2018; Greener & Jones, 2021) offer a principled way of learning the force field by directly training the simulation process to reproduce experimental observables (Wang et al.. 2020a; 2022; Thaler & Zavadlav, 2021). We hope our datasets and benchmarks will encourage future developments in all related aspects push the frontier in MI. for MD simulations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mark James Abraham, Teemu Murtola, Roland Schulz, Szilárd Páll, Jeremy C. Smith, Berk Hess, and Erik Lindahl. GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers. SoftwareX, 1-2:19-25, September 2015. ISSN 23527110. doi:
URL
com/ comm//eerrieee/pi///331111111110
Nongmuch Artrith, Alexander Urban, and Gerbrand Ceder. Efficient and accurate machine-learning interpolation of atomic energies in compositions with many species. Physical Review B, 96(1); 014112, 2017.
Ilyes Batatia, Simon Batzner, Dávid Péter Kovács, Albert Musaelian, Gregor NC Simm, Ralf Drautz, Christoph Ortner, Boris Kozinsky, and Gábor Csányi The design space of e (3)-equivariant atom-centered interatomic potentials. arXiv preprint arXiv:2205.0666 2022.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornlluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-qquivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-11, 2022.
Jörg Behler and Michele Parrinello. Generalized neural-network representation of high-dimensional potential-energy surfaces. Physicul review letters, 98( 1)): 46401,, 2007.
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras- Derras-Dmmingo, Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 1 1110))6059-6072 2021.
Yaoyi Chen, Andreas Krämer, Nicholas E Charron, Brooke Husic, Cecilia Clementi, and Frank Noé. Machine learning implicit solvation for molecular dynamics. The Journal of Chemical Physics, 155(8):084101, 2021.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 365)ee6 2017.
Daniel A. Colón-Ramos Patrick Riviere, Hari Shroff, and Rudolf Oldenbourg. Transforming the development and dissemination of cutting-edge microscopy and computation. Nat Methods, 16(8): 667-669, August 2019. ISSN 1548-7091, 1548-7105. doi: 10003///41112--1--44775554 URL hetp::/wwit httt:://wwwnatureee. nwwuratuuee.... htt.tture..oom/artaaeee////a333/
Wendy D Cornell, Piotr Cieplak, Christopher Bayly, Ian Gould, Kenneth M Merz, David M Ferguson, David C Spellmeyer, Thomas Fox, James W Caldwell, and Peter A Kollman. A second generation force field for the simulation of proteins, nucleic acids, and organic molecules j. am. chem. 1995, 117, 5179- 5197. Journal of the American Chemical Society, 11809):309-23000 1996.
Stefan Doerr, Maciej Majewski, Adrià Pérez, Andreas Kramer, Cecilia Clementi, Frank Noe, Toni Giorgino, and Gianni De Fabritiis. Torchmd: A deep learning framework for molecular simulations. Journal of chemical theory and computation, 17(4):2355 222223333333,,, 2021.
Peter Eastman, Pavan Kumar Behara, David L. Dotson, Raimondas Galvelis, John E. Herr, Josh Horton, Yuezhi Mao, John D. Chodera, Benjamin P. Pritchard, Yuanqing Wang, Gianni De Fabritiis, and Thomas E. Markland. Spice, dataset of drug-like molecules and peptides for training machine learning potentials, 2022.
Felix A Faber, Luke Hutchison, Bing Huang, Justin Gilmer, Samuel S Schoenholz, George E Dahl, Oriol Vinyals, Steven Kearnes, Patrick F Riley, and O Anatole Von Lilienfeld. Prediction errors of molecular machine learning models lower than hybrid dft error. Journal of chemical theory and computation, 13(11):52555 (::5555555-524 2017.
Michael Feig. Kinetics from implicit solvent simulations of biomolecules as a function of viscosity. Journal of chemical theory and computation, 3(5):1734-1744 2007.
10

---

Under review as a conference paper at ICLR 2023
Daan Frenkel and Berend Smit. Understanding molecular simulation: from algorithms applications, volume 1. Elsevier, 2001.
Xiang Fu, Tian Xie, Nathan J Rebello, Bradley D Olsen, and Tommi Jaakkola. Simulate time- integrated coarse- coarse-grained molecular dynamics with geometric machine learning. arXiv preprint arXiv:2204. 10348, 2022.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference Learning Representations, 2020.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemmet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34: 6790-6802, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Jonathan Godwin, Michael Schaarschmidt, Alexander I. Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple gnn regularisa- tion for 3d molecular property prediction and beyond. In International conference on learning representations, 2021.
Joe G Greener and David T Jones. Differentiable molecular simulation can learn all the parameters in coarse-grained force field for proteins. PloS one, 16(9))e0,56990, 2021.
Thomas A Halgren. Merck molecular force field. i. basis, form, scope, parameterization, and performance of mmff94. Journal of computational chemistry, 1765-6):40-551, 1996.
Teresa Hcad-Gordon, Martin Head-Gorde Gordon, Michael Frisch, Charles Brooks III,, and John Pople. A theoretical study of alanine dipeptide and analogs. International Journal of Quantum Chemistry, 36($16):311-321) 1989.
RL Henderson. uniqueness theorem for fluid pair correlation functions. Physics Letters A, 49(3): 197-198, 1974.
Johannes Hoja, Leonardo Medrano Sandonas, Brian G Ernst, Alvaro Vazquez-Mayagoitta. Robert DiStasio Jr, and Alexandre Tkatchenko. Qm7-x, a comprehensive dataset of mechttnccam properties spanning the chemical space of small organic molecules. Scientific data, 8(1):1-11, 2021.
Lars Holdijk, Yuanqi Du, Ferry Hooft, Priyank Jaini, Bernd Ensing, and Max Welling, Path integral stochastic optimal control for sampling transition paths. arXiv preprint arXiv:2207.02149, 2022.
Weihua Hu, Muhammed Shuaibi, Abhishek Das, Siddharth Goyal, Anuroop Sriram, Jure Leskovec, Devi Parikh, and C Lawrence Zitnick. Forcenet: A graph neural network for large-scale quantum calculations. arXiv preprint arXiv:2103.0144 2021.
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with differentiable simulator. In International Conference on Learning Representations, 2018.
Adil Kabylda, Valentin Vassilev-Galindo, Stefan Chmiela, Igor Poltavsky, and Alexandre Tkatchenko. Towards linearly scaling and chemically accurate global machine learning force fields for large molecules. arXiv preprint ar arXiv:2209 039955, 2022.
George A Kaminski, Richard A Friesner. Julian Tirado- Rivee, and William L Jorgensen. Evaluation and reparametrization of the opls-aa force field for proteins via comparison with accurate quantum chemical calculations on peptides. The Journal of Physical Chemistry B, (66(2)):6774648)) 2001.
Alireza Khorshidi and Andrew A Peterson. Amp: A modular approach to machine learning in atomistic simulations. Computer Physics Communications, 207:310-322 2016.

---

Under review as a conference paper at ICLR 2023
Adeesh Kolluru, Muhammed Shuaibi, Aini Palizhati, Nima Shoghi, Abhishek Das, Brandon Wood, C Lawrence Zitmick, John R Kitchin, and Zachary W Ulissi. Open challenges in developing general- izable large scale machine learning models for catalyst discovery. arXiv preprint arXfv:2206.02005, 2022.
Dávid Péter Kovács, Cas van der Oord, Jiri Kucera, Alice EA Allen, Daniel J Cole, Christoph Ortner, and Gábor Csányi. Linear atomic cluster expansion force fields for organic molecules: beyond rmse. Journal of chemical theory and computation, 17(12)7696-7711, 2021.
Alessandro Laio and Michele Parrinello. Escaping free-energy minima. Proceedings of the National Academy of Sciences, 19(20):12562-122555 2002.
Thomas Lane, Gregory Bowman, Kyle Beauchamp, Vincent A Voelz, and Vijay S Pande. Markov state model reveals folding and functional dynamics in ultra-long md trajectories. Journal of the 2011.
American Chemical Society, 133(45):18413-18)
Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Christensen, Marcin Dulak, Jesper Friis, Michael N Groves, Bjørk Hammer, Cory Hargus, Eric Hermes, Paul C Jennings, Peter Bjerre Jensen, James Kermode, John R Kitchin, Esben Leonhard Kolsbjerg, Joseph Kubal, Kristen Kaasbjerg, Steen Lysgaard, Jón Bergmann Maronsson, Tristan Maxson, Thomas Olsen, Lars Pastewka, Andrew Peterson, Carsten Rostgaard, Jakob Schiøtz, Ole Schütt, Mikkel Strange, Kristian S Thygesen, Tejs Vegge, Lasse Vilhelmsen, Michael Walter, Zhenhua Zeng, and Karsten W Jacobsen. The atomic simulation environment python library for working with atoms. Jourmal of Physics: Condensed Matter, 29(27):73002,, 2017. // //stacks. www.org//o........o....o........ 095 8984 /11/////////////////////11/111// 2===222222300222
URL http:
Jonas Lederer, Michael Gastegger, Kristof T Schütt, Michael Kampffmeyer, Klaus-Robert Müller, and Oliver T Unke. Automatic identification of chemical moieties. arXiv preprint arXiv:2203.1620. 2022.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11990, 2022.
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations, 2021.
Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. arXiv preprint arXiv:2204.05244 2022.
Cheol Woo Park, Mordechai Kornbluth, Jonathan Vandermause, Chris Wolverton, Boris Kozinsky, and Jonathan P Mailoa. Accurate and scalable graph neural network force field and molecular dynamics with direct force architecture. npj Computational Materials, 7(1):19,, 2021.
Jay W Ponder and David A Case. Force fields for protein simulations. Advances in protein chemistry, 66:27-85, 2003a.
Jay W. Ponder and David A. Case. Force Fields for Protein Simulations. In Advances in Protein Chemistry, volume 66. pp. 27-85, Elsevier, 2003b. ISBN 978-0-12-034266-2 com/
doi: 10.1016600065-323300366666060000
URL
Setrievve/p11/800553333333
Zhuoran Qiao, Anders S Christensen, Matthew Welborn, Frederick R Manby, Anima Anandkumar, and Thomas F Millrr III. Unite: Unitary n-body tensor equivariant network with applications to quantum chemistry. arXiv preprint arXiv:2105.14655. 2021.
Raghunathan Ramakrishnan, Pavlo O Dral, Mathiis Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
David Rosenberger, Justin S Smith, and Angel E Garcia. Modeling of peptides with classical and novel machine learning force fields: A comparison. The Journal of Physical Chemistry B, 125(14): 2598-3612, 2021.
12

---

Under review as a conference paper at ICLR 2023
Alvaro Baachez-Gonzales Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec. and Peter Battaglia. Learning to simulate complex physics with graph networks. International Conference on Machine Learning. pp. 8459- 1459-8468. PMLR, 2020.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Elia Schneider, Luke Dai, Robert Q Topper, Christof Drechsel-Grau, and Mark E Tuckerman. Stochastic neural network approach for learning high-dimensional free energy surfaces. Physical review letters, (1(11))150601 2017.
Samuel Schoenholz and Ekin Dogus Cubuk. Jax md: a framework for differentiable physics. Advances in Neural Information Processing Systems, 33:1 00:111428-11 11441, 2020.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus- Müller. Schnet: A continuous-filee convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pp. 9177-9388. PMLR, 2021.
Daniel Schwalbe-Kodee Aik Rui Tan, and Rafael molecular geometries with uncertainty-but based adversarial attacks. Nature communications, 12(1): 1-12, 2021.
Differentiable sampling of
Hythem Sidky, Wei Chen, and Andrew L. Ferguson. Machine learning for collective variable discovery and enhanced sampling in biomolecular simulation. Molecular Physics, 118(5):e1737742, 2020.
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. Ani-1: an extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 8(4):3192-3203, 2017.
Sina Stocker, Johannes Gasteiger, Florian Becker, Stephan Günnemann, and Johannes Margraf. How robust are modern graph neural network potentials in long and hot molecular dynamics sinulations? 2022.
Mohammad M Sultan, Hannah K Wayment-Steele, and Vijay S Pande, Transferable neural networks for enhanced sampling of protein dynamics. Journal of chemical theory and computation, 14(4): 1887-1894, 2018
So Takamoto, Chikashi Shinagawa, Daisuke Motoki, Kosuke Nakago, Wenwen Li, Iori Kurata, Taku Watanabe, Yoshihiro Yayama, Hiroki Iriguchi, Yusuke Asano, et al. Towards universal neural network potential for material discovery applicable to arbitrary combination of 45 elements. Nature Communications, 13(1):1-11, 2022.
Stephan Thaler and Julija Zavadlav, Learning neural network potentials from experimental data via differentiable trajectory reweighting. Nature Communications, 12(1):1-10, 2021.
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. International Conference on Learning Representations, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation -qquiavainnnni neural networks for 3d point clouds. arXiv preprint arXiv: 0802.0222002228 2018.
Richard Tran, Janice Lan, Muhammed Shuaibi, Siddharth Goyal, Brandon Wood, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, et al. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysis. arXiv preprint arXiv:2206.08917 2022.
Gareth A. Tribello, Massimiliano Bonomi, Davide Branduardi, Carlo Camilloni, and Giovanni Bussi. PLUMED 2: New feathers for old bird. Computer Physics Communications, 185 (2):604-613, February 2014. ISSN 00104655. doi: 10.10166j..p..2013.000. URL https: 05551/550000000550000011
//linkinghub. e sevier.
13

---

Under review as a conference paper at ICLR 2023
Mark Tuckerman. Statistical mechanics: theory and molecular simulation. Oxford university press, 2010.
Oliver T Unke and Markus Meuwly. A reactive, scalable, and transferable model for molecular energies from neural network approach based on local information. The Journal of chemical physics, 148(24):241708, 2018.
Oliver Unke, Stefan Chmicla, Michael Gastegger, Kristof T Schütt, Huziel E Sauceda, and Klaus- Robert Müller. Spookynet: Learning force fields with electronic degrees of freedom and nonlocal effects. Nature communications, 1-144 2021a.
Oliver Unke, Stefan Chmicla, Huziel E Sauceda, Michael Gastegger, Igor Poltavsky, Kristof Sch utt, Alexandre Tkatchenko, and Klaus-Robert Müller, Machine learning force fields. Chemical Reviews, 121(16):00122-100
2021
Jonathan Vandermause, Steven B Torrisi, Simon Batzner, Yu Xie, Lixin Sun, Alexie Kolpak, and Boris Kozinsky. On-the-fly active learning of interpretable bayesian force fields for atomistic rare events. npj Computational Materials, 6(1):1-11, 2020.
Ercheng Wang, Huiyong Sun, Junmei Wang, Zhe Wang, Hui Liu, John Z. H. Zhang, and Tingjun Hou. End-Point Binding Free Energy Calculation with MM/PBSA and MM/GBSA: Strategies and Applications in Drug Design. Chem. Rev. (9(16))9488-5088 August 2019a. ISSN 0009-2665, www..eee.eeo0000...... URL https:tt/////
1520-6890. doi: 10.102 1021/acs. chemrev. 9b00055.
ptttps://pubs.. pubs. acs org/ httg///oi/1
Jiang Wang, Simon Olsson, Christoph Wehmeyer, Adrià Pérez, Nicholas E Charron, Gianni De Fabri- tiss, Frank Noé, and Cecilia Clementi. Machine learning of cearse-grained molecular dynamics force fields. ACS central science, 5(5):755-767, 2019b.
Wujie Wang and Rafael Gómez-Bombarelli, Coarse-graining auto-encoders for molecular dynamics. npj Computational Materials, 5(1):1 2019.
Wujie Wang. Simon Axelrod, and Rafael Gómez-Bombarde Bombarelli. Differentiable molecular simulations for control and learning. arXiv preprint arXiv:2003. 000668 2020a.
Wujie Wang, Tzuhsiung Yang. William H Harris, and Rafael Gómez-Bombarde Bombarelli. Active learning and neural network potentials accelerate molecular screening of ether-based solvate ionic liquids. Chemical Communications, 36(63)):920-8922 2020b.
Wujie Wang, Zhenghao Wu, and Rafael tiable simulations. arXiv preprint (rXii:2209.07679 2022.
Learning pair potentials using differen-
Michael A Webb, Yukyung Jung, Danielle M Pesko, Brett M Savoie, Umi Yamamoto, Geoffrey W Coates, Nitash P Balsara, then-Gang Wang, and Thomas F Miller III. Systematic computational and experimental investigation of lithium-ion transport mechanisms in polyester- based polymer electrolytes. ACS central science, 148):1982055 2015.
Yujie Wu, Harald L. Tepper, and Gregory A. Voth. Flexible simple point-charge water model with improved liquid-state properties. The Journal of Chemical Physics, 124()):024503, January 2006. ISSN 0021 1089-7690, doi: 10.1063/1.2136877 URL http: //aip httt:://aip.scitatiion scitation.
org/doi/100 10w3/10631110 000.006311..1..1... 2136877.
Shuwen Yue, Maria Carolina Muniz, Marcos Calegari Andrade, Linfeng Zhang, Roberto Car, and Athanassios Z. Panagiotopoulos. When do short-range atomistic machine-lcarnigg models fall short? J. Chem. Phys., 154(3)033411, January 2021. ISSN 0021-9606, 1089-7690, doi: 10.1063/5.0031211 URL http: //tpp/appp///// scitation org/ /t1/10.1111111//// 1633/.0001215.
Yaoguang Zhai, Alessandro Caruso, Sigbjöm L Bore, and Francesco Pacsani. A "short blanket" dilemma for a statc-of-the-as neural network potential for water: Reproducing properties or learning the underlying physics? 2022.
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and EJPRL Weinan. Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Physical review letters, 120 (14):14300) 2018a.
14

---

Under review as a conference paper at ICLR 2023
Linfeng Zhang, Jiequn Han, Han Wang. Wissam Saidi, Roberto Car, et al. End-to-end symmetry preserving inter-atomic potential energy model for finite and extended systems. Advances in Neural Information Processing Systems, 31, 2018b.
15

---

Under review as a conference paper at ICLR 2023
A EVALUATION METRICS
Observables. From the time series observations of positions and velocities, observables ((+,) can be computed to characterize the state of the system at different granularities. Under the ergodic hypothesis, the time averages of the simulation observables converge to distributional averages under B(x))
the Gibbs measure: (0) 1 limy->oo Σ. O(x,) I dx p(æ) Ozx), where p(a)) x exp(- kn with 7 as the bath temperature and k as the Boltzmann constant. Calculations of such observables require the system to reach equilibrium. Simulation observables connect simulations to experimental measurements and are predictive of macroscopic properties of matter. Common observables include radial distribution functions (RDFs), virial stress tensor, meaansquared displacement (MSD), etc.
T
Distribution of interatomic distances a low-dimensional description of the 3D structure and has been studied in previous work (Zhang et al., 2018a). For given configuration æ, the distribution of interatomic distances h(r) is computed with:
M()) N(N 1)
Σ
8(r x2 æ;(l)
where r is the distance from a reference particle; is the total number of particles; i, indicates the pairs of atoms that contribute to the distance statisties; ō is the Dirac Delta function to extract value distributions. To calculate the ensemble average, h(r) is calculated and averaged over frames from cquilibrated trajectories. For learned simulations, compute h(r) using only the stable part of the simulation.
RDF. As one of the most informative simulation observables, the radial distribution function (RDF) describes the structuraltthermodynantt properties of the system and is also experimentally measur- able. It has been widely used in force field development (Henderson, 1974). By definition, the RDF describes how density varies as function of distance from a particle (illustrated in Figure 7 (a)). For given configuration æ, the RDF can be computed with the following formula:
N
N Σ
RDF()) 4mT Np
8(r - [z æ;11)
where is the distance from reference particle; N is the total number of particles; i, indicates the pairs of atoms that contribute to the distance statisties; is the density of the system; 5 the Dirac Delta function extract value distributions. To caleulate the ensemble average, RDF(() is calculated and averaged over frames from equilibrated trajectories. The final RDF MAE is then calculated by integrating r:
MAERDF
(RDF(r)) - (RDF()))dr
(3)
where (.) is the averaging operator, (RDF(r)) is the reference equilibrium RDF, and (RDF(r)) is the model-predicted RDF.
Diffusivity coefficient is major property of study in many previous work (Batzner et al., 2022). The diffusivity coeffcient quantifies the time-correlation of the translational displacement (illustrated in Figure 7 (b)), and can be computed from the mean square displacement:
D lim
Σ (t,(t) (0)
N
where zz(t) is the coordinate of particle at time f, and N' is the number of particles being tracked in the system. For the water system we monitor the liquid diffusivity coefficient and track all 64 oxygen atoms. For the LiPS system, monitor Li-ion Diffusivity and track all 27 Li-ions. As the definition implics, D is a quantity that converges with longer simulation time. Accurate recovery of D requires sufficient long trajectories sampled from the Hamiltonian with MI. FFs. In this paper,
16

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
t+1
W
Radial Distribution
Diffusion
Dihedral Angles
Figure 7: Illustrations regarding benchmarked metrics.
we only compute diffusivity for stable trajectories of at least 100 ps for water and ps for LiPS. As we simulate multiple random runs for each system/model, we average the diffusivity coefficient extracted from each valid trajectory to obtain the final prediction.
Free energy surface. Given the probability distributions over configurations p(æ) and chosen geometric coordinate & transformed from æ. Based on the marginalized density p(E), the free energy (Tuckerman, 2010) can be calculated from the following:
F(E) = -n
In the specific case of alanine dipeptide, there are two main conformational DOF: dihedral angle o of C - N - Ca C and dihedral angle 1o of N - C. - C N (illustrated in Figure 7 (c)). Therefore, the FES w.r.t & and is the most physically informative. We propose our quantitative metric (MEF(())F(C based on the absolute enror in reconstructing the FES along the & and vé coordinates. We integrate the absolute difference between the reference free energy F and the model predicted F from [- n):
S MAE MAEF(()
F(Ø) - ((
(6)
$=-ズ FF()) - (xb)ldd
(7)
Stability eriterion. Abstractly speaking, stability a notion of staying within physical (low-energy) configuration spaces. Since all MD systems studied in this paper are in equilibrium, practically we keep track of stability by closely monitoring equilibrium statistics. For systems with periodic boundary conditions, monitor the RDF and say simulation becomes "unstable" at time T when
(IDF(())
|rr A
where (.) is the averaging operator, T is short time window, and A is the stability threshold. In this paper we use
ps, = 3.0 for water, and 5 ps, A 1.0 for LiPS. For water, we assert unstable if any of the three element-conditiones RDFs: RDF (0,)). the threshold. For flexible molecules, we keep track of stability through the bond lengths and say simulation becomes "unstable" time T when:
.) RDF (H,H), RDF (H,O) exceeds
[[]xx xT)]l b(,)) A (L,))
max
where B is the set of all bonds, i, are the two endpoint atoms of the bond, and by, is the equilibrium bond length. We use A 0.5 À for both MD17 molecules and alanine dipeptide.
We measure the stability of a leamed model through the simulation time it remains stable. For each dataset, the threshold we adopt rather relaxed that an "unstable" equilibrium statistic usually indicates the system is already in highly nonphysical configuration.
17

---

Under review as a conference paper at ICLR 2023
FPS. All frames per second (FPS) metrics are measured with an NVIDIA Tesla V100-PCIe GPU. We present FPS as a reference for models' computational efficiency but also note that code speed can be affected by many factors, and likely has room for improvement.
Experimental procedures. Baseline models are trained on the datasets described in Section according to the experimental settings described in Appendix C. At evaluation time, we simulate MD trajectories using the learned models, with thermostats and simulation length described in Section 4. The simulated trajectories are recorded as time series of atom positions, along with other information including atom types, temperature, total energy, potential energy, and kinetic energy. All observables described in this section can be computed from recorded trajectories. We use the stability criterion described above to find the time step when the systems become "unstable", and only use the trajectory before that time step for the computation of observables. Among the observables, the distribution of interatomic distances and RDF are computed for each frame, and averaged over the entire trajectory. Diffusivity coefficients are computed by averaging over the diffusivity coefficient computed from all applicable time windows where the time window length is predefined to be long enough. For example, for a trajectory Tf steps and time window of K steps, we average over the diffusivity computed from the time windows: [1, K], [2, + 11, .I, [T - K + 1, TJ. We use 100 ps as the time window size for water and 35 ps as the time window size for LiPS. We also remove the first 5 ps of the simulated trajectories of LiPS for equilibrium. As we do multiple simulations per model and dataset, we compute the metrics for each trajectory and report the mean and standard deviation. Further details on observable computation can be found in our code submission: observable ipynb.
B DATASET DETAILS
The MD17 'ataset' (Chmiela et al., 2017) and the LiPS dataset? (Batzner et al., 2022) are adapted from previous works and are publicly available. The MD17 dataset is generated from path-integral molecular dynamics simulations that incorporate quantum mechanies into the classic molecular dynamics simulations using Feynman path integrals. The LiPS datasets are generated by ab-initio molecular dynamics simulations with a generalized gradient PBE functional and projector augmented wave pseudopotentials. We refer interested readers to the respective papers for more details on the data generation process. The water dataset and alanine dipeptide dataset are generated by ourselves. process.
Water. Our water dataset is generated from molecular dynamics simulations of a simple classical water model, namely, the flexible version of the Extended Simple Point Charge water model (SPC/E- fw) (Wu et al., 2006) at temperature T = 300 K and pressure P = atm. For this model, the interaction parameters (e.g., O-H bond stretch and H-O-H bond angles), are parameterized match extensive experimental properties such as the self-diffusion and dielectric constants at bulk phase. This classical model has been well-studied in previous work (Wu al., 2006; Yue al., 2021) and has shown reasonable predictions of the physical properties of liquid water. It provides computationally inexpensive way to generate a large amount of training data. The experience and knowledge gained from the benchmark based on the simple model can be readily extended to systems with higher accuracy, such as the ab-initio models.
Alanine dipeptide. Our dataset is generated from the MD simulation of an alanine dipeptide molecule solvated explicit water (1164 water molecules) performed in GROMACS (Abraham et al., 2015) using the AMBER-03 (Ponder & Case, 2003b) force-field. In the AMBER-03 force field, the potential energy parameters such as van der Waals and electrostatics are mostly derived from quantum mechanical methods with minor optimization on the bonded parameters reproduce the experimental vibrational frequencies and structures (Cornell et al., 1996; Ponder & Case, 2003a). The NPT ensemble is applied in simulations, with hydrogen bond length constraints using LINear Constraint Solver (LINCS) and a time step of 2 fs. The temperature and pressure of the system are controlled at T 300 K and P damping frequency t. = 0.1 ps and Parrinello- Rahmann barostat with coupling frequency tp = 2.0 ps, respectively. The Particle Mesh Ewald approach is used to compute long-range electrostatics with periodic boundary conditions applied to the x, y, and z directions. The conformational modes can be characterized by six free energy local minimas, which have been used in previous work (Lederer
bar using a stochastic velocity rescaling thermostat with
'http://www.sge .org/ hhttps://archiss materia al scll cud. wtw.derg/rreoor/2222...
18

---

Under review as a conference paper at ICLR 2023
(8)
(b)
15
40
5
10
30
[ulmo
20
10
[[Jimml)
20
10
10
n n5
0-
0
-180
-90
90
180
-180
-90
90
180
-180
-90
0
90
160
[deggees))
[degrees)]
[degroes]
[dogrees)
Figure 8: F& and F& have converged for the reference force field (a) and NequlP (b) at time ns under Metadynamies.
et al.. 2022). We initialize six simulations for each model from each of the six free energy local minimas.
Implicit solvation. The explicit solvent of 1164 water molecules is not the subject of study but adds a significant computational burden. In this task, we attempt to learn an implicit solvent model (ISM) of the alanine dipeptide, in which the explicit solvent environment is incorporated in the learned FF. The ISM is commonly used in drug design (Wang al., 2019a) because it can speed up the computation by dramatically decreasing the number of particles required for simulation. In general, the mean-field estimation ISM ignores the effect of solvent, thermal fluctuations, and solvent friction (Feig, 2007). Thus, molecular kinetics is not directly comparable to the explicit solvation simulation. However, the cquilibrium configurations can be explicitly compared, as conducted in Chen et al. (2021).
Metadynamics simulation. Simulating energy barrier jump usually requires long sampling of the trajectory in MD simulations. The conformational change of alanine dipeptide in water involves such process, making it difficult extract the complete free energy surface, i.e., the Ramachandran plot, in normal MD. In order to examine the learned ML FFs within a reasonable time limit, metadynamies (Laio & Parrinello, 2002) is employed explore the learned FES of the solvated alanine dipeptide. Metadynamics is a widely used technique in atomistic simulations to accelerate the sampling of rare events and estimate the FES of certain set of degrees of freedom. is based on iteratively "filling" the potential energy using sum of Gaussians deposited on a set of suitable collective variables (CVs) along the trajectory. At evaluation time, we perform metadynamics with dihedral angles & and 1/ as C starting from the configuration energy minimums in the free energy surface indicated Figure 5 (c). The Gaussians with height h = 1.2 and sigma estimated FES of both o and z0 do not significantly change after 5 ns. In addition, the height of the bias gaussian potential smoothly converges to in the time limit of 5 ns. Therefore, simulation time of 5 ns is sufficient for the convergence of the metadynamics. This metadynamics simulation of alanine dipeptide with AMBER force-fields is carried out using GROMACS (Abraham al., 2015) integrated with the PLUMED library (Tribello et al., 2014; Colón- Ramos et al., 2019) of version 2.8.
located at one of the six
0.35 are deposited every 1 centered on te and Φ. As shown in Figure 8, the
C EXPERIMENTAL DETAILS
The Open Catalyst Project' codebase and the official codebases of DeepPot-SE and NequlP' all publicly available. We build our MD simulation framework based on the Atomic Simulation Environment (ASE) library? (Larsen al., 2017).
plpeepaaaaeeeer. We adopt the original model hyperparameters in the respective papers and find they can produce good force prediction results that match the trend and numbers for MD17 reported in previous work. As we introduce new datasets, we set training hyperparameters such as the batch size and summarize them Table 7. For water and LiPS, we use a batch size of like in previous
'In practice, the selection of suitable collective variables can be a bysee challenge (Sidky et al., 2020). 'https: / httgtttttuuuttt/////////t phttps://gh http:ssb///tttbb.../. wttp.b///cmeeeeeeeeeeeeeeeeee hewp:inn//dddddeeeeeeeeee detpmdekppmdddeeeee kit http:://ggthhbbbb.oiiiooiiaaae "https: /attps//gitl/e com.a./a/m////aaeeeeeeeeaa
/
19

---

Under review as a conference paper at ICLR 2023
Table 7: Default training-related hyperparameters for each dataset. size from respective papers when available for MD17. DeepPot-SE: SchNet:
"We adopt the original batch
100; DimeNet: 32;
GemNet-T/dT: 1; NequlP: 5. We use batch size of 8 for ForceNet.
Dataset
Training dataset size
Batch size
Max epoch
LR patience
Longest simulation time
MD17
9,500
1-100"
2,000
5 epochs
20 hours
Water- 1k
950
10,000
50 epochs
28 hours
Water-10k
9,500
2,000
5 epochs
28 hours
Water-90k
85,500
400
3 epochs
28 hours
Alanine dipeptide LiPS
38,000
2,000
5 epochs
75 hours
19,000
2,000
epochs
hours
Table 8: Results on Water- 1k. Force MAE is reported in the unit of [meV/A], Stability reported in the unit of [ps]; Diffusivity MAE is reported in the unit of [10 "m² /s]; RDF MAE and FPS are unitless.
DeepPot-SE
SchNet
DimeNet
ForceNet
GemNet-T
GemNet-dT
NequIP
Force
6.7
13.1
3.5
13.6
5.0
29.2
1.4
Stability
117;
175
4(4)
13(7)
6(7)
0(o)
500ga)
(D.()
0.17
(.52().05)
0.46(0.22)
0.86
(.62
0.07
RDF (n.m))
0.13
0.24(0.02)
(.33
0.56
0.35
(a.7().00)
RDF
(.28
(.54(0.01)
(.4
1 1.44 (0.09)
0.71
0.26
Diffusivity
0.24
1.79
0.37
work (Batzner al., 2022) each structure already contains reasonable number of atoms and interactions. Following previous work, we use an initial learning rate of 0.001 for all experiments except for NequIP, which uses 0.005 as the initial learning rate in the original paper. For models that minimize mixture of force loss and energy loss, we set the force loss coefficient Ap to be 1000 and the energy loss coefficient Ag be 1, if not specified in the original paper. A higher force loss coefficient is common in previous work (Zhang et al., 2018a; Batzner et al., 2022) as simulations do not directly rely on the energy.
Notably, NequlP proposed several sets of hyperparameters for different datasets, including MD17, a water+ice dataset from Zhang et al. 2018a, LiPS, etc. We follow the MD17 hyperparameters of NequlP for our MD17 and alanine dipeptide datasets; the watertice hyperparameters of NequlP for our water dataset; and the LiPS hyperparameters of NequlP for the same LiPS dataset. For DeepPot-SE, we adopted hyperparameters introduced in Zhang et al. 2018a. The only architectural adjustment we made is because we observed training instability for ForceNet on water using the original hyperparameters. We resolve this issue by reducing the network width from 512 to 128 for ForceNet in our water experiments.
To facilitate benchmarking with a reasonable computational budget, we stop the training of model either of the following conditions is met: (1) a maximum training time of 7 days is reached on an NVIDIA Tesla V100-PCIe GPU: (2) maximum number of epochs specified in Table 7 is reached; (3) The learning rate drops below 10-6 with a ReduceLROne lat lateau scheduler with factor 0.8 and learning rate (LR) patience specified in Table 7. We also report the longest time for an ML model to finish our benchmark simulation in Table 7. All numbers are results of NequlP. The high computational cost for evaluating MD simulations has been a major consideration in designing our benchmark datasets and metrics. Training of DeepPot-SB SE is efficient and we follow the training setup specified in Zhang et 2018a.
Complete water results. We present results water- 1k in Table 8 and results on water-90k in Table 9. Results on water-10k is presented in Table the main text. All models generally achieve lower force error when trained with more data, but stability and estimation of ensemble statistics don't necessarily improve. In particular, DeepPot-SE shows clear improvement with more training data and becomes as good NequlP stability. but the estimation ensemble statistics does not improve. This may be due to the limited accuracy of SchNet coming from the limited expressiveness of the invariant atomic representation.
v0ter-90k. SchNet demonstrates significant improvement in
20

---

Under review as a conference paper at ICLR 2023
Table 9: Results on Water-90k, Force MAE is reported in the unit of [meV/A]; Stability is reported the unit of [ps]; Diffusivity MAE reported in the unit of [10-9 m² m//s]; RDF MAE and FPS are unitless.
DeepPot-SE
SchNet
DimeNet
ForceNet
TemNet-T
GemNet-dT
NequlP
Force
5.9
8.4
1.7
8.6
1.4
Stability
(0)
299/70)
36(9)
9(12)
20(a)
S(1;
500
(DF
0.0T 02)
0.67 (0.03)
(0.00)
(0.49)
(.
0.20
0.06
RDF
0.05
0.31 (0.02)
0.14(..01)
(..2
(.
0.16ra.01)
(.01(a.o1)
RDF (1)
(.29().08)
0.67 (0.04)
0.18
2.05
0.21 (0.06)
0.26
0.25ço.08)
Diffusivity
0.35
1.97
0.18
Table 10: Water- l results on NequlP with various model sizes and radius cutoffs.
Force
Stability
RDF (0,0)
RDF RDF (u..r)
RDF (1,o)
Diffusivity
FPS
Width=64, r=4
3.5
(0)
(0.02)
0.05/a.01)
(0.06)
0.38
8.2
Width=32, r=6
1.5
500(0)
(.06(a.01)
0.05
0.26(0.06)
0.25
5.2
Width=64, r=5
1.6
500(0)
0.07 (0.02)
0.05
(0.06)
0.31
4.9
Width=64, r=6
1.4
500(0)
(.07
(.07()
0.26(0.07)
0.37
3.9
Width=128.
1.5
500
0.07 (0.02) 0.05
0.29(0.07)
0.37
2.5
Influence of model size. Table 10 shows ablation study over the model size and radius cutoff of NequlP over water-Ik. lk. We observe that all models are highly stable and attain equally good performance in simulation-based metries. Although small radius cutoff of 4 leads to worse performance in force prediction, it is more computationally efficient and preserves the trajectory statistics. These results show that there exists a trade-off between accuracy and efficiency when choosing the hyperparameters of an ML force field, and force error may not be the preferred criterion for model selection.
Large water system of 512 molecules. To study model performance in generalizing to larger system and model scalability, we evaluate models trained on the water- dataset on a dataset with 512 water molecules simulated for 1 ns, using the same reference force field. Given the high cost of simulating a large system, we simulate 5 trajectories of 150 ps long for each model. The results are shown in Table 11. We observe that all models suffer slightly higher force errors compared to evaluation over the 64-molecule water system. In terms of stability, NequlP and SphereNet always remains stable for the entire 150 ps. However, SphereNet does not produce correct ensemble properties. SchNet the third stable model, while all other models are not stable enough for diffusivity computation. DimeNet, GemNet-T, and GemNet-dT are not stable through the entire simulation, but can produce decent RDF results. Noticeably, the stability of DeepPot-SE drops significantly. We hypothesize that the lack of message passing limits its capability in capturing long-range interactions and thus limits the performance in a larger system.
Stability's relation with dataset size. We extract the force and stability results for each model from Figure 4 to make Figure 9 to better illustrate the relation between stability and dataset size for each model. We observe that while more data almost always reduce force error, stability does not necessarily improve. In particular, NequlP is highly stable across all dataset size. DeepPot-SE and SchNet has significant improvement stability with more data. While for DimeNet ForceNet, and GemNet, more training data does not bring significant stability improvement. Section 6 contains detailed discussions on the causes of instability and potential solutions to improve stability.
Stability's 's relation with training epochs. We study the evolution of simulation stability in the training process an ML force field. We take the SchNet model on the MD17 molecule salicylic acid, and save the checkpoint at 100, 200, 300, 400, and 500 epochs. We conduct 5 simulations of 300 ps using each checkpoint. Figure 10 shows the force error and stability of the model at different stages of training. We observe that the force error decreases training progress, and the stability improves to be stable across the entire 300 ps simulation and training epoch 300. This result reveals that thorough training is important to both accuracy and stability of MI. force fields.
21

---

Under review as a conference paper at ICLR 2023
Table 11: Results the large water system with 512 molecules, with models trained the water- dataset (64-molecule water system), "SphereNet on Water-large requires more memory than Tesla V100 supports. We run its simulations on faster NVIDIA A100 cards so the FPS is not entirely comparable to other models.
DeepPu SE
SchNet
DimeNel
PaiNN
SphereNet'
FonceNet
GemNNet-T
Gfl
NequIP
Ferce
10.6
12.1
5.1
9.7
18.4
13:2
4.2
7.7
Suuiiitty
19:2z)
118:as
18
16
150ke:
45y20)
50rg)
150ha)
(D.,
0.23
(.62
(.17
0.3 (0)
0.33
0.74(u.0))
(.23
0.16
0. 0.10
RDF m..0)
0.24
0.30
0. (.12
0,210.000
0.12...01
0.51
0.15
(.11
0.07 (o
RDF uz
0.67 (..77)
0.55
0.17
0. 0.29
0.97
1.38(
0.23
0.16
0.12 (..2:
Diffusivity
2.54
FPS
80.7
23.1
3.5
17.4
0.8
11.9
2.2
5.3
0.7
Water: DecpPot SE
Water: SchWet
Water, DimeNet
Water, Watar, ForceNet
3.4
Stab
100
Training Data Size
rmnnng Data Size
Training Data Size
Traning Data Size
NAE (mw))AA)
126
Stuni Drl
30
Size
Trairing
Training
Figure The force error and stability of all models on the water dataset, with varying training dataset size: 1k, 10k, and 90k.
Distribution of interatomic distances for MD17. Figure 11 shows the h(r) curves for all models and molecules benchmarked. We randomly selected one simulation out of the five simulations conducted for each model and molecule. We observe that due lack of stability, DeepPot- SE produces noisy h(r) on Aspirin. ForceNet does not manage to learn the correct interatomic interactions and produces incorrect (((r) curves. Most models are able produce lu(r) that match well with the reference, with SchNet being less accurate on Asp rin and Et Ethano.
RDFs for water. Selected RDF 1k/10k/90k are in Figure 12, Figure 13, and Figure 14. Most noisy curves are due insufficient sampling time, which results in a small number of frames to be averaged in obtaining the RDF curves. We observe that SchNet and ForceNet produce inaccurate curves that are not very noisy, showing that their failure not entirely due lack of stability but because of inaccurate modeling of interactions caused by limited expressiveness and lower sample efficiency. Further, curves have zero values below a certain threshold, as any pair of atoms cannot get close each other. However, DimeNet and GemNet-T exhibit abnormal high values for very small distances, indicating the simulations have gone into nonphysical configurations and collapsed.
for water-
35
Foroa (mavi4
25:
Stab lyy (9
20
16.
note that the reference
100
200
3.0
400
500
Training Epoch
Figure 10: The force error and stability of SchNet for simulating Salicylic acid, traing progress.
RDFs for LiPS. As shown in Figure 15, DeepPot-SE does not manage stay stable on LiPS. ForceNet learns inaccurate interactions and produces inaccurate RDFs. All other models can produce highly accurate RDF and can reproduce Li-ion diffusivity relatively accurately, as demonstrated in Table 6.
22

---

Under review as a conference paper at ICLR 2023
DespPot-SE, Aspirin
DeepPot-SE. Ethanol
DeepPot-SE, Naphthalene
DeepPot-SE. Salicylic Adid
Referance Prededin
Reference Presidion
Reloeence Pred clor
Raference
1.0
0.6
0.6
Prectoor
h(g) (4
0.4
0.4
0.5
0.2
0.2
0.0
SchNet, Aspirin
SchNet, Ethanol
SchNet, Naphthalene
SchNet, Salicylic Acid
iteference Predectin
Helerenca Prediction
0.8
Relermnce Preductor
Haterence
htg [4 (A-1-
0.4
0.0
Predktior
0.4
0.2
0.2
0.0
0.0
0.0
0.0
10
DimeNat, Aspirin
DimeNet, Ethanol
DimeNet, Naphthalene
DimeNet, Salicyfic Acid
Relosunso
2.0
Rofarnnc Predittn
0.8
Rolorono Predicdor
0.8
Reforeros
m014 -1
Prodiction
0.6
Presktor
0.4
0.4
0.5
022
0.
0.0
0.0
10
10
PaINN, Aspirin
PaiNN, Ethanol
PaiNN, Naphthalene
PaiNN, Salicylic Acid
0.8
Reforuncs
15
Refurance Prodiciin
Roler Predicdor
Rafurencs Protitorr
-1]
Prodicdon
0.6
0.5
h00 14
0.4
0.4
0.5
0.2
02
0.0
SphereNet, Asplrin
SpheroNet Ethanol
SphereNot Naphthalene
SphersNet. Sallcylic Acid
Heforence Prwdeeson
Refersnce Predicioo
0.8
Rekernnc Predector
Reference
h(1 (A -1
044
Preditior
0.6
02
0.5
0.2
0.0
0.0
0.0
0.0
ForceNet, Aspirin
ForcaNet, Ethanol
ForceNet Naphthalene
ForceNet, Salicylic Acid
1.5
Reforence
Reference
[A-1
0.75
1.0
0.50
0.0
0.00
10
2
4
0
10
GamNattT, Aspirin
GemNet-T, Ethanol
GemNet-T, Naphthalene
GemNet-T, Salicylic Acid
Rcforunse Prodicdon
Refurnnc Prediction
0.8
Robooo you Predidoor
0.6
Rafurenos
0.4
Prokioor
h00 (4
1.0
0.4
02
0.2
0.2
0.0
GomNet-dT, Aspirin
GemNettdTT Ethanol
GemNastdT, Naphthalens
GemNastdTT Salicylic Acid
Reforunca Predecson
15
Refulsnce Predition
0.8
Rolornno
Reference
31
0.4
Fredector
Precetior
M1011 LA
1.0
0.4
02
0.5
0.2
n2
0.6
0.0
0.0
10
NequIP,, Aspltin
NaqulP. Ethanol
NequiP. Naphth alens
NequiP. Sallcylic Acid
1.5
0.8
84
0.6
1101 14
1.0
0.4
0.5
0.
0.0
0.6
0.0
10
2
0
10
10
[A]
[A]
[A]
[A]
Figure 11: h(r) of md17
23

---

Under review as a conference paper at ICLR 2023
DeepPot-SE. 0-0
DeepPol-SE. H-H
DeepPot-SE, H-O
Reference Prediction
Reference
Reference
Prediction
Prediction
RDF(0 (A
30
20
SchNet, 0-O
SchNet, H-H
SchNet, H-O
Reference
Reference Prediction
Reference
Prediction
Prediction
RDF(0)
50
DimeNat. 0-O
DimeNst. H-H
DimeNet,
Reference Prediction
Reference Prediction
Reference
RDF(0 (4 1
40
Prediction
30
ForceNat, 0-O
ForceNat. H-H
ForcaNet, H-O
Reference
Reference
Reference
RDF( (A --1
Prediction
Prediction
Prediction
20
GemNet-T,
GemNettT,
GemNet-T,
Reference Prediction
Reference
Reference
ROF(GIA-1)
Prediction
Prediction
NequiP: O-0
NequiP. H-H
NequiP, H-O
Reference Prediction
Reference Prediction
Reference
Prediction
RDF() [A
90
20
[A]
[A]
[[]]
Figure 12: RDFs of Water-Ik. GemNet-dT does not remain stable for more than ps and is therefore not feasible for RDF computation.
24

---

Under review as a conference paper at ICLR 2023
DeepPot-SE, 0-0
DeepPot-SE. H+H
DeepPot-SE H-O
Reference Prediction
Reference
Reference
RDF (A
Prediction
Prediction
O-O
SchNet. H-O
Reference Prediction
Reference Prediction
Reference
1 DDFOI 10
Prediction
DimeNet, O-O
DimeNet, H-H
DimeNet. H-O
Reference Prediction
Reference
Reference
Prediction
90
Prediction
PaiNN,
PaINN,
Reference Prediction
Reference
Reference
RDF (4
Prediction
Prediction
20
SphereNet, O-O
SphereNet H-H
SphereNel, H-O
Reference Prediction
Reference
Reference Prediction
RDFMn (4
Prediction
ForceNet, O-O
ForceNet,
ForceNet, H-D
Reference Prediction
Referencs Prediction
Reference
14
40
Prediction
GemNet-T, O-O
GerNNet T, H-H
GomNet-T, H-O
Reference Prediction
Reference
Reference
Prediction
Prediction
2
GemNet-dT, 0-O
GomNettdT,
GomNat-dT, H-O
Reference Prediction
Reference Prediction
Reference
Prediction
20
NequiP, 0-O
NequiP, H+H
NequlP, H-O
Reference Prediction
Reference Prediction
Reference
Prediction
RDF(11)
20
[A]
[A]
Figure 13; RDFs of Water-10k
25

---

Under review as a conference paper at ICLR 2023
DeepPal- SE. 0-O
DeepPat SE. H-H
DeepPot SS, H-O
90
Reference Prediction
Reference
Reference
40
Prediction
Prediction
RDF() [A
96
20
SchNel, O-O
SchNel. H-H
SchNel, H-O
Reference
Reference
Reference
Prediction
Prediction
40
Prediction
RDF (A
10
DimeNet. 0-O
DimeNet. H-H
DimeNet, H-O
Reference Prediction
Reference Prediction
40
Reference
RDF(1) 14
90
22
ForceNet, O-O
ForceNat, H-H
ForcaNet, H-O
Reference
Reference Prediction
60
Reference
RDF( [A (1]
Prediction
Prediction
22
GemNet-T,
GemNet-T,
GemNet-T,
Reference Prediction
Reference
RDFI0 (4 -1
Prediction
Prediction
20
50
GemNet-dT, O-0
GemNet-dT, H-H
GemNet-dT, H-O
Reference Prediction
Reference Prediction
Reference
RDF()) [A --1
Prediction
NequiP, O-0
NequiP, H-H
NequiP, H-O
3.2
Reference Prediction
Reference Prediction
Reference
Prediction
RDFM1 [A
1.5
0.5
50
0.
[4]
Figure 14: RDFs of Water-90k.
26

---

Under review as a conference paper at ICLR 2023
DeepPot-SE
SchNet
DimeNet
ForceNet
RDF(r) [A -1
Reference Predistion
Reforance Prediction
Reference Prediction
Reference Prediction
[A]
[A]
r [A]
[A]
GemNet-T
GemNet-dT
NequlP
RDF(r) 11
Reference Prediction
Reference
Reference
Prodiction
[A]
[A]
[A]
Figure 15: RDFs of LiPS.
27


Reviews:
[{"review_id": "zSHgBCc2wU0", "reviewer": "Reviewer_NbtG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on molecular dynamics (MD) simulations. Previous methods use force/energy errors as the evaluation, but based on the authors' analysis, these metrics are not enough.\n\nSpecifically, in this paper, the authors first curate four MD datasets based on previous work and design specific evaluation metrics for each dataset. They then run experiments with seven baseline methods. Finally, they do a lot of analysis on the results, showing forces are not enough, and providing potential solutions.\n", "summary_of_the_review": "This paper is well-written and easy to follow. The motivation, contribution, etc are clear. The experimental analysis looks good. \n\nBut there exist several weaknesses.\n\n- My main concerns are the used **datasets and baseline methods**. OC20 dataset and more equivariant baseline methods should be considered. \n\n- There are some errors in Table 2 (**baseline methods**).\n\n- For **the evaluation metrics**, it is unclear to me which is used in previous work and which is newly proposed in this paper. \n\nIn addition, the main contributions are the benchmark, experimental analysis, and some insights. I think it might be more suitable for a benchmark track.", "strength_and_weaknesses": "> Strengths\n\n1. This paper is well-written and easy to follow. The motivation, contribution, etc. are clear.\n2. The authors point out that forces are not enough to evaluate MD and propose several new metrics. Based on my understanding, the evaluation metrics are new, and the inconsistent performance on different metrics may inspire more future work.\n\n> Weaknesses\n\n1. About **the datasets** in this benchmark: \n\n     - Based on my understanding, OC20 can also be included in this benchmark since it also includes complete relaxation trajectories  (https://github.com/Open-Catalyst-Project/ocp/blob/main/DATASET.md) and is widely used recently. The authors should justify why they don't include this dataset. In addition, if considering OC20, which metric should we use to evaluate the model?\n    - The size of the systems is limited to less than 200 atoms. This may be not suitable for many real-world datasets like water, goop, and sand in [1] and protein data.\n\n\n2. I have several questions about **the baseline methods** in Table 2.\n\n    - I think there is an issue with GemNet-T and GemNet-dT. GemNet-T should be energy-conserving, while GemNet-dT is not.\n    - Based on my knowledge, except NequIP, all the other baseline methods in Table 2 are invariant to rotation, and/or translation, and/or reflection. Different from GemNet-T, GemNet-dT uses a different way to output forces. \n    - Since the task is to predict forces (should be equivariant), I suggest the authors include more equivariant baselines like [2] and [3].\n\n3. For **the detailed evaluation metrics** in Appendix A, it is unclear to me which is used in previous work and which is newly proposed in this paper. This is because some metrics have corresponding references, and some do not.\n\n[1] Learning to Simulate Complex Physics with Graph Networks\n\n[2] Equivariant message passing for the prediction of tensorial properties and molecular spectra\n\n[3] E(n) Equivariant Graph Neural Networks\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "yuM83Jar0X", "reviewer": "Reviewer_j9ZS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel benchmark for machine learning-based molecular dynamic simulations. The proposed benchmark includes four datasets and several quantitative evaluation metrics. Experiments and analysis with a collection of SOTA ML models are performed, and an open-source codebase is provided.", "summary_of_the_review": "The proposed evaluation metrics are meaningful, the analysis of SOTA methods is insightful, and the open-source codebase can benefit future works. However, several important related works are missing. And I’m concerned about the novelty. Therefore, I’m inclined to reject.   ", "strength_and_weaknesses": "**Strength**\n\n1. This benchmark provides comprehensive datasets for four different MD systems, including small molecules, liquid water, peptide, and materials, which is convenient for users who are interested in MD simulation. Moreover, the provided open-source codebase for training and simulation would benefit future works. \n\n2. This work proposes a series of evaluation metrics for MD simulation, which haven’t been well studied in existing works. The authors conduct a thorough analysis with SOTA methods and show that the widely used force prediction is not sufficient for the evaluation and suggest stability should be a key metric to improve. I believe the proposed metrics are very meaningful, and these observations reveal insights for MD simulation.\n\n**Weaknesses**\n\n1. Missing related works. For example, SphereNet[1] is a recent work that has good performance for PES prediction, and TorchMD[2] is a recent DL framework for MD simulation. The authors need to discuss missing related works in Section 2 and add experimental comparison.\n \n2. More data are encouraged to include in this benchmark. For example, include all eight molecules from MD17. In addition, chignolin used in [2] is a peptide that is extensively studied with MD. It can be added to your peptide track. Moreover, polymers used in [3] could also be added to your material track.\n\n 3. Since this is a benchmark work without proposing any novel technical method, I would suggest submitting the paper to a benchmark track rather than ICLR main track.\n\nReference:\n\n[1]. Liu, Yi, et al. \"Spherical message passing for 3d molecular graphs.\" International Conference on Learning Representations. 2021.\n\n[2]. Doerr, Stefan, et al. \"Torchmd: A deep learning framework for molecular simulations.\" Journal of chemical theory and computation 17.4 (2021): 2355-2363.\n\n[3]. Fu, Xiang, et al. \"Simulate Time-integrated Coarse-grained Molecular Dynamics with Geometric Machine Learning.\" arXiv preprint arXiv:2204.10348 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DXlzuAQMsv", "reviewer": "Reviewer_UiVb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper propose a benchmark suite for ML MD. Within the suite, 7 deep learning based methods are included as base models and 4 datasets of different MD systems were tailored for simulation. From intensive comparison and analysis, the authors claim that force error is not a sufficient measure to evaluate ML MD performance, and stability should be considered in benchmarking.   ", "summary_of_the_review": "Overall, I think this work is very informative and useful for researchers in the area of AI for chemoinformatics. ", "strength_and_weaknesses": "Strengths:\n1. The comparison and analysis in this paper is through. Insights into failure cases are provided.\n2. The efforts of providing a benchmarking suite should be praised, considering that ML MD is a new area in AI for chemoinformatics and this is a rather challenging task.\n\nWeaknesses: \n1. The size of each simulation systems is restricted to have less than 200 atoms. This restriction may limit future scalability studies.\n2. The origins of these four datasets are unclear. The authors should explicitly mention whether each data set was generated from experiments or in silico simulation. \n3. Mathematical MD simulation tools such as GROMACS and Amber may be included in comparison if a data set was generated from experiments.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A4fuYwQxSv", "reviewer": "Reviewer_t1Na", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focus on examining the current state of force field machine learning model in molecule dynamics. Specifically, it proposes several simulation based metrics to evaluate the stability of existing models. For the SoTA model, it provides further analysis on simulation failures and reveals underlying reasons.", "summary_of_the_review": "Based on my understanding, this work states the challenge of force field simulation stability, which could possibly inspire more research effort on this topic. The relation of stability and model training is not justified in the current version and novelty of the paper is somehow limited.", "strength_and_weaknesses": "Strength: \n1. The results of the paper is very convincing given the fact that it utilizes more training configurations than existing work such as GemNet.\n2. The illustration of this paper is extraordinary such as Figure 5 and Figure 6.\n\nWeakness & Questions:\n1. One of my major concern is the paper seems more like a benchmarking paper. All of the models and datasets used in the paper already exist and the instability of force field simulation has been discussed by previous work. \n2. I notice the result on MD17 is different from references for method like GemNet, I notice it's mainly because you are using more training data and in their paper GemNet is better than NequIP.\n3. How is the stability related with training epoch and amount of training samples?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5jOjv4g-my", "reviewer": "Reviewer_CmqU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the evaluation of force fields (FF) for atomistic simulations.\nThe authors propose benchmark datasets and metrics based on molecular dynamics(MD) simulation, a major use case of FFs.  \nWhen state-of-the-art machine learning(ML)-based FFs are evaluated on the benchmark datasets, conventional force accuracy is not always related to the performance of MD simulation.\nThe authors argue that there is room for improvement in the stability of ML-based FFs.", "summary_of_the_review": "Although this is not the first paper addressing the problem, I would like to appreciate someone's contribution to standardizing the benchmarking of ML FFs.\nI hope a more detailed description of the evaluation procedures is provided in the final version.", "strength_and_weaknesses": "Strength\n* Although it has been known that force accuracy is not enough metric for ML FFs in the community, the standardization of MD-based evaluation procedure is valuable for developing ML FFs.\n* State-of-the-art ML FF models are evaluated in the experiments.\n* Section 6 is a good introduction to the issues in the current ML FFs in ML community.\n\nWeaknesses\n* Reason for choosing the applications of MD simulations is not provided.  There is other popular applications, such as viscosity (Zhang et al., 2015) or thermal conductivity (Müller-Plathe, 1997).\n* The procedural details of computing physical properties are not provided.   Since one of the promising directions for stabilizing dynamics is an extension of training data (Stocker et al., 2022; Takamoto et al., 2022), extended training datasets will be used in future work.  Therefore, standardizing the evaluation procedures is more important than fixing the dataset set.\n\nQuestions:\n* Why did you perform six simulations only for Alanine dipeptide and five for others?   How did you find the number of simulations?\n* How is \"the stable part of the simulation\" determined?\n\n- Zhang et al.  \"Reliable Viscosity Calculation from Equilibrium Molecular Dynamics Simulations: A Time Decomposition Method\".  Journal of Chemical Theory and Computation (2015), 3537-3546.\n- Müller-Plathe, Florian. \"A simple nonequilibrium molecular dynamics method for calculating the thermal conductivity.\" The Journal of chemical physics (1997), 6082-6085.\n- Takamoto et al., \"Towards universal neural network potential for material discovery applicable to arbitrary combination of 45 elements\".  Nature Communications (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject