Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper propose a benchmark suite for ML MD. Within the suite, 7 deep learning based methods are included as base models and 4 datasets of different MD systems were tailored for simulation. From intensive comparison and analysis, the authors claim that force error is not a sufficient measure to evaluate ML MD performance, and stability should be considered in benchmarking.   
strength_and_weaknesses: Strengths:
1. The comparison and analysis in this paper is through. Insights into failure cases are provided.
2. The efforts of providing a benchmarking suite should be praised, considering that ML MD is a new area in AI for chemoinformatics and this is a rather challenging task.

Weaknesses: 
1. The size of each simulation systems is restricted to have less than 200 atoms. This restriction may limit future scalability studies.
2. The origins of these four datasets are unclear. The authors should explicitly mention whether each data set was generated from experiments or in silico simulation. 
3. Mathematical MD simulation tools such as GROMACS and Amber may be included in comparison if a data set was generated from experiments.  
clarity,_quality,_novelty_and_reproducibility: Clearly, it is a benchmarking and analysis paper in AIMD. The major novelty of this paper is the provision of a suite with baseline models, datasets, and evaluation metrics. 
summary_of_the_review: Overall, I think this work is very informative and useful for researchers in the area of AI for chemoinformatics.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on molecular dynamics (MD) simulations. Previous methods use force/energy errors as the evaluation, but based on the authors' analysis, these metrics are not enough.

Specifically, in this paper, the authors first curate four MD datasets based on previous work and design specific evaluation metrics for each dataset. They then run experiments with seven baseline methods. Finally, they do a lot of analysis on the results, showing forces are not enough, and providing potential solutions.

strength_and_weaknesses: > Strengths

1. This paper is well-written and easy to follow. The motivation, contribution, etc. are clear.
2. The authors point out that forces are not enough to evaluate MD and propose several new metrics. Based on my understanding, the evaluation metrics are new, and the inconsistent performance on different metrics may inspire more future work.

> Weaknesses

1. About **the datasets** in this benchmark: 

     - Based on my understanding, OC20 can also be included in this benchmark since it also includes complete relaxation trajectories  (https://github.com/Open-Catalyst-Project/ocp/blob/main/DATASET.md) and is widely used recently. The authors should justify why they don't include this dataset. In addition, if considering OC20, which metric should we use to evaluate the model?
    - The size of the systems is limited to less than 200 atoms. This may be not suitable for many real-world datasets like water, goop, and sand in [1] and protein data.


2. I have several questions about **the baseline methods** in Table 2.

    - I think there is an issue with GemNet-T and GemNet-dT. GemNet-T should be energy-conserving, while GemNet-dT is not.
    - Based on my knowledge, except NequIP, all the other baseline methods in Table 2 are invariant to rotation, and/or translation, and/or reflection. Different from GemNet-T, GemNet-dT uses a different way to output forces. 
    - Since the task is to predict forces (should be equivariant), I suggest the authors include more equivariant baselines like [2] and [3].

3. For **the detailed evaluation metrics** in Appendix A, it is unclear to me which is used in previous work and which is newly proposed in this paper. This is because some metrics have corresponding references, and some do not.

[1] Learning to Simulate Complex Physics with Graph Networks

[2] Equivariant message passing for the prediction of tensorial properties and molecular spectra

[3] E(n) Equivariant Graph Neural Networks

clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the clarity is good.

About originality, the main original parts are the new evaluation metrics and the experimental analysis. But the used datasets and methods are mainly proposed by previous work.

summary_of_the_review: This paper is well-written and easy to follow. The motivation, contribution, etc are clear. The experimental analysis looks good. 

But there exist several weaknesses.

- My main concerns are the used **datasets and baseline methods**. OC20 dataset and more equivariant baseline methods should be considered. 

- There are some errors in Table 2 (**baseline methods**).

- For **the evaluation metrics**, it is unclear to me which is used in previous work and which is newly proposed in this paper. 

In addition, the main contributions are the benchmark, experimental analysis, and some insights. I think it might be more suitable for a benchmark track.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a novel benchmark for machine learning-based molecular dynamic simulations. The proposed benchmark includes four datasets and several quantitative evaluation metrics. Experiments and analysis with a collection of SOTA ML models are performed, and an open-source codebase is provided.
strength_and_weaknesses: **Strength**

1. This benchmark provides comprehensive datasets for four different MD systems, including small molecules, liquid water, peptide, and materials, which is convenient for users who are interested in MD simulation. Moreover, the provided open-source codebase for training and simulation would benefit future works. 

2. This work proposes a series of evaluation metrics for MD simulation, which haven’t been well studied in existing works. The authors conduct a thorough analysis with SOTA methods and show that the widely used force prediction is not sufficient for the evaluation and suggest stability should be a key metric to improve. I believe the proposed metrics are very meaningful, and these observations reveal insights for MD simulation.

**Weaknesses**

1. Missing related works. For example, SphereNet[1] is a recent work that has good performance for PES prediction, and TorchMD[2] is a recent DL framework for MD simulation. The authors need to discuss missing related works in Section 2 and add experimental comparison.
 
2. More data are encouraged to include in this benchmark. For example, include all eight molecules from MD17. In addition, chignolin used in [2] is a peptide that is extensively studied with MD. It can be added to your peptide track. Moreover, polymers used in [3] could also be added to your material track.

 3. Since this is a benchmark work without proposing any novel technical method, I would suggest submitting the paper to a benchmark track rather than ICLR main track.

Reference:

[1]. Liu, Yi, et al. "Spherical message passing for 3d molecular graphs." International Conference on Learning Representations. 2021.

[2]. Doerr, Stefan, et al. "Torchmd: A deep learning framework for molecular simulations." Journal of chemical theory and computation 17.4 (2021): 2355-2363.

[3]. Fu, Xiang, et al. "Simulate Time-integrated Coarse-grained Molecular Dynamics with Geometric Machine Learning." arXiv preprint arXiv:2204.10348 (2022).

clarity,_quality,_novelty_and_reproducibility: Clarity: good

Quality: ok

Novelty: The proposed evaluation metrics and analysis contribute to most of the novelty. Datasets are all collected from existing ML MD works, and no novel method is proposed in this work.

Reproducibility: should be good

summary_of_the_review: The proposed evaluation metrics are meaningful, the analysis of SOTA methods is insightful, and the open-source codebase can benefit future works. However, several important related works are missing. And I’m concerned about the novelty. Therefore, I’m inclined to reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the evaluation of force fields (FF) for atomistic simulations.
The authors propose benchmark datasets and metrics based on molecular dynamics(MD) simulation, a major use case of FFs.  
When state-of-the-art machine learning(ML)-based FFs are evaluated on the benchmark datasets, conventional force accuracy is not always related to the performance of MD simulation.
The authors argue that there is room for improvement in the stability of ML-based FFs.
strength_and_weaknesses: Strength
* Although it has been known that force accuracy is not enough metric for ML FFs in the community, the standardization of MD-based evaluation procedure is valuable for developing ML FFs.
* State-of-the-art ML FF models are evaluated in the experiments.
* Section 6 is a good introduction to the issues in the current ML FFs in ML community.

Weaknesses
* Reason for choosing the applications of MD simulations is not provided.  There is other popular applications, such as viscosity (Zhang et al., 2015) or thermal conductivity (Müller-Plathe, 1997).
* The procedural details of computing physical properties are not provided.   Since one of the promising directions for stabilizing dynamics is an extension of training data (Stocker et al., 2022; Takamoto et al., 2022), extended training datasets will be used in future work.  Therefore, standardizing the evaluation procedures is more important than fixing the dataset set.

Questions:
* Why did you perform six simulations only for Alanine dipeptide and five for others?   How did you find the number of simulations?
* How is "the stable part of the simulation" determined?

- Zhang et al.  "Reliable Viscosity Calculation from Equilibrium Molecular Dynamics Simulations: A Time Decomposition Method".  Journal of Chemical Theory and Computation (2015), 3537-3546.
- Müller-Plathe, Florian. "A simple nonequilibrium molecular dynamics method for calculating the thermal conductivity." The Journal of chemical physics (1997), 6082-6085.
- Takamoto et al., "Towards universal neural network potential for material discovery applicable to arbitrary combination of 45 elements".  Nature Communications (2022).
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well-organized and well-written.

Quality:
The claims are well-supported by the experiments.

Novelty:
Although this is not the first paper addressing the problem, no standard benchmark set is not available in the community before.

Reproducibility:
There is room for improvement in the description of evaluation procedures above.


Minor comments:
Page 15:  h(r) should be D in the Diffusivity coefficient paragraph.

summary_of_the_review: Although this is not the first paper addressing the problem, I would like to appreciate someone's contribution to standardizing the benchmarking of ML FFs.
I hope a more detailed description of the evaluation procedures is provided in the final version.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focus on examining the current state of force field machine learning model in molecule dynamics. Specifically, it proposes several simulation based metrics to evaluate the stability of existing models. For the SoTA model, it provides further analysis on simulation failures and reveals underlying reasons.
strength_and_weaknesses: Strength: 
1. The results of the paper is very convincing given the fact that it utilizes more training configurations than existing work such as GemNet.
2. The illustration of this paper is extraordinary such as Figure 5 and Figure 6.

Weakness & Questions:
1. One of my major concern is the paper seems more like a benchmarking paper. All of the models and datasets used in the paper already exist and the instability of force field simulation has been discussed by previous work. 
2. I notice the result on MD17 is different from references for method like GemNet, I notice it's mainly because you are using more training data and in their paper GemNet is better than NequIP.
3. How is the stability related with training epoch and amount of training samples?
clarity,_quality,_novelty_and_reproducibility: I think the paper is written clearly and supplementary materials provide code to reproduce result. However, the novelty seems to be rather limited except several stability metrics for the ML-FF.
summary_of_the_review: Based on my understanding, this work states the challenge of force field simulation stability, which could possibly inspire more research effort on this topic. The relation of stability and model training is not justified in the current version and novelty of the paper is somehow limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work provides a benchmark evaluation for machine learning methods in force field prediction in molecular simulation. After review and quite intensive discussions, most reviewers are only moderately supportive to this work, and one reviewer is still not convinced with the significance. We believe this work has some level of values, but might not be a good fit to ICLR. We are recommending to reject this work and encourage the authors to consider NeurIPS datasets and benchmark track, which would be a better fit.
justification_for_why_not_higher_score: Some reviewers are only moderately supportive to this work, and one reviewer is still not convinced with the significance.
justification_for_why_not_lower_score: NA
Decision: Reject