Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper present a training strategy to improve performance in ML MD. In this strategy, first bias of inaccurate data is calculated using the model trained on accurate, then the bias-aware inaccurate data is used to train an ML MD model from scratch, and finally, the model is fine-tuned using accurate data. Pretraining using unlabeled data is also explored in this work.  
strength_and_weaknesses: Strengths:
1. The training strategy proposed in this paper makes sense for ML MD. The authors also investigated data weighting based on quality and explored the use of unlabelled data for pretraining.
2. The experiments conducted in this research are thorough and their results support the main claim.

Weaknesses:
1. The authors should provide more details about the derivation of Eq. 3. 
2. ML FF methods mentioned in the related work were not compared with the proposed method. 

clarity,_quality,_novelty_and_reproducibility: The idea of this paper is presented clearly. The major novelty of this paper lies in the training strategy: pretraining using inaccurate and cheep data and then fine-tuning using accurate but expensive data. Considering DL model training in a broader scope, similar training schemes are seen in many domains. Thus, I rate the novelty of this paper's idea intermediate. 
summary_of_the_review: Overall, this paper shows a carefully design pretraining-finetuning training strategy for ML MD, and their results support the claim that inaccurate data can help boost the performance of ML MD models as accurate data are very expensive to generate. 

Update: I have read through all the reviewers' comments and the authors' response. I think the authors addressed most of the concerns. Even though I still have concerns about the novelty of this work, it may be an informative paper for researchers in the MLFF field. Thus, I increased my rating to 6: marginally above the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In the paper, the authors aim to improve methods to train machine learning force fields for MD simulations. These methods rely on accurate data, which is expensive to generate. They augment the training procedure with less accurate data being cheaper to generate. This is achieved by first training a model with a limited amount of accurate data, which is then used to estimate the bias of the less accurate data. The data is reweighted according to this bias, followed by training a model on those samples. In a third step, the model is fine-tuned on the accurate data.

The authors evaluate their method on five molecules. The learned force field seems more accurate than when using standard training, especially when using a small amount of accurate training data and performs similar or better than a related approach.
strength_and_weaknesses: ### Strengths

The method is simple to implement. It allows using various kinds of additional data to improve the model's performance. The authors test their procedure thoroughly on five molecules and use different types of auxiliary data.

### Weaknesses

The procedure is basically limited to adding additional data to the training procedure, which is in itself not new. The novel idea is training a model to estimate the bias and reweight the data accordingly. However, this bias-aware training does not improve the model significantly, see Table 3. The weights based on the estimated bias are apparently very noisy, but they could be improved by updating them during training in a bootstrapping fashion.

The method has been tested empirically on several molecules, but the paper lacks theoretical validation that the method is sound. Moreover, it should be tested in a toy setting, such as doing regression with a few variables on generated data with varying noise levels.

Most importantly, the authors did only compare the performance of their models with respect to the number of accurate training samples. However, they should do so with respect to the overall runtime used to generate both the accurate and the inaccurate training data. Since the relative improvement decreases rapidly when increasing the number of accurate samples, it is not obvious whether it would not be better to just generate more accurate data.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. All experiments are documented in detail.

As stated above, there is a lack of novelty and empirical evaluation of the method.
summary_of_the_review: In conclusion, I favor to reject this paper. However, I am open for a discussion with the authors and the other reviewers.


Edit after rebuttal:

I appreciate the author's effort to improve their paper. The toy example they added is interesting. However, the setup seems counterintuitive to me. First, there should be noise being added to the data and its standard deviation could depend on the accuracy of the simulated data generation process. Second, I like the idea of adding a sampled bias, but why is it sampled uniformly from the values {0, 2, 4, 8, 16}? Thereby, 20% of the "biased" data is actually unbiased. I suggest drawing the bias from an interval such as [2, 6] or perhaps [0, 8] instead, limiting the number of accurate samples, or even using a constant bias with additional noise centered around 0.
Given that improved runtime is one of the major selling points of the paper, I think a runtime analysis is vital for the paper.

Therefore, I will not change my score, but encourage the authors to improve their draft and submit it to another venue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors in this paper, propose a framework which enables training data generation with is computationally inexpensive while being accurate for molecular dynamics simulation. The authors argue that the current state of the art for machine learning force fields are computationally very expensive when it comes to achieving high accuracy, including generating a large amount of costly training data. While there are options to reduce computational complexity by training on fewer data points or cheaper reference forces, these options come with a tradeoff of loss in accuracy/poor performances. 

The authors propose that ASTEROID works in a multi stage training framework where step 1: small amounts of accurate data is first identified in a large inaccurate dataset, step 2: train the models on large amounts of training data which is inaccurate and cheaply available, followed by step 3: where small amounts of accurate training data which is more expensive to obtain. step 1 is achieved via a bias aware loss function to prevent overfitting on inaccurate training data. The authors demonstrate that such a method significantly outperforms conventional methods such as standard training and sGDML on networks such as GEMNet and EGNN

strength_and_weaknesses: 
Strength:
1. Several performance comparisons around accuracy are method with different methods (state of the art) under various conditions to really drive home the point around performance improvement and superiority over existing methods
2. Sufficient background provided to identify gaps and opportunities for development, although there could be more clarity around datasets and overall learning workflows. 


Weaknesses: 
1. Description of datasets is complex, and not simplified eg. what does accurate, inaccurate data look like along with labels. 
2. Method to estimate bias, metrics around bias measurement/identification could be clearer, eg. why is there a surrogate for bias measurement and how would ideal bias measurement look like prior to introducing surrogate method. 
3. Need additional commentary on any limitations of this method eg. cases where such an approach would not work and what could be alternatives. 
4. Calculations/stats/graphs around time comparison with other methods is not available which could be a key result. 
5. No clear explanation why the 5 particular chemicals were chosen. 
6. Also why test size is different for ethanol vs others? Do other molecules also need to be compared with the same test set size? 
7. For the hyper parameter tuning setting, I dont see the comparison between ASTEROID and standard training, even though its mentioned that way. 
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: Overall the clarity and quality of the work is high in terms of outline but description of datasets is limited and need more explanation. Some graphing could be better eg. Ethanol plotting for GemNet is out of the charts. Also metrics in some tables is missing (eg. what exactly are we measuring in table 1? The paper says accuracy but needs to be part of the table legend for additional clarity). Given this a complex area, a workflow diagram in introduction or shortly after that walks through the key steps from end to end (similar to an architecture diagram) can help simplify understanding. 


Novelty: Overall the method is reasonably novel in terms of an alternative mixture of training strategies although the individual steps themselves aren't novel. Key comparisons in terms of time effort over other methods can demonstrate additional novelty.  


Reproducibility: 
1. ASTEROID implementation or algorithm isn't provided under references. 
2. Calculation and analysis of experimental procedures is unavailable for reproducibility. 
summary_of_the_review: I propose this work is conditionally accepted with few more experiments such as time component comparison with other methods to demonstrate cost awareness and efficiencies to drive home the point of why the method is superior over prior work of art. While the authors argue that the method is superior in terms of accuracy over other current work, additional metrics are needed to really drive home the point without which the work seems less convincing. Also even though there is no availability of materials to test reproducibility, it is expected that a workflow to guide experimental setup and analysis would be necessary to validate some of the results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper address NNP training using an expensive data generation method, CCSD(T).
The proposed method is based on the pretraining of NNP using the training data collected by relatively low-cost data generation methods, such as DFT.
The authors extended it to a three-stage method, in which an importance-weighting method is used for the pretraining of NNP.
An NNP is trained using CCSD(T) at the first stage, and the instance weights for the pretraining are estimated by the discrepancy between the force of low-cost data generation methods and the force of the NNP trained using CCSD(T).   In addition, the authors propose unsupervised pretraining using score matching for the pretraining.
The proposed method outperforms the baseline methods on the MD17 dataset.
strength_and_weaknesses: Strength
* It is interesting to see the pretraining results using a dataset collected by not only DFT but also the empirical force field.
* The use of score-matching for unsupervised pretraining is reasonable approach.
* Unsupervised pretraining of NNP might be useful since it might be possible to aggregate multiple data sets collected by different methods.
The proposed method down-weight structures with higher energy, but we also need the energy/force estimation for disordered structures, i.e., structures of higher energy, in real-world applications, such as MD simulations or structural optimization.  

Weaknesses
* There is no evaluation result on its applications, such as MD simulations or structural optimization.  Since force accuracy is not sufficient for obtaining stable dynamics [1], I am not confident about the effectiveness of the proposed approach for its applications.
* I think ANI- 1ccx  (Smith et al., 2019), a simple pretraining-finetuning method, should be a baseline method.  In my understanding, ASTEROID w/o BAT is the proposed transfer learning in Smith et al., 2019.  However, the results of it are not available in Figures 2, 3, 4, and Table 2. 
* If the unsupervised learning method is used for the second stage, the first stage is meaningless.  Thus, the unsupervised learning method does not extend ASTEROID in this case, and this contradicts the section title of Section 4.   In addition, the experiments conducted in this case.


Questions:
1.  Can you add the simple pretraining-finetuning results, i.e., ASTEROID w/o BAT, in Figures 2, 3, 4, and Table 2?   Since ANI- 1ccx  (Smith et al., 2019) is a suitable baseline method, I think you can omit "Standard Training" results if there is no space available.

2.  I miss the meaning of "Next we extend ASTEROID to settings where unlabelled data is available by fine-tuning the model obtained from score matching".
Is this mean that the unlabeled data is collected by MD simulations using a fine-tuned model?  Or does it simply mean you pre-trained an NNP using unlabelled data and fine-tuned it using CCSD(T) data?

3.  Can you provide any results of applications for which DFT-based NNP fails, as Smith et al., 2019 did?


[1] Sina Stocker, Johannes Gasteiger, Florian Becker, Stephan Günnemann, and Johannes Margraf. How robust are modern graph neural network potentials in long and hot molecular dynamics simulations?  2022.

clarity,_quality,_novelty_and_reproducibility: Clarity:
Although there is an exception, as I asked above, most of the paper is well-written.

Quality:
My main concern is that the evaluation is only about the accuracy of its force field using a small dataset.  However, it is important to convince readers about the performance of the proposed method in real-world applications.

Novelty:
The proposed method is a strait-forward extension of previous work, so the technical contribution is limited.

Reproducibility:
Although there is an exception, as I asked above, most of the experiment settings are clear for reproduction.

summary_of_the_review: The fatal problem of the paper is no evaluation result on its applications, such as MD simulations or structural optimization.  
In addition, the baseline algorithm in the experiments is also not appropriate.
I think a major revision is required for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to improve the training of machine learning force fields for MD simulations. ML force fields usually rely on accurate data, which is expensive to generate. The authors augment the training procedure with less accurate but cheaper data. This is achieved by first training a model with a limited amount of accurate data, which is then used to estimate the bias of the less accurate data. The data is reweighted according to this bias, followed by training a model on those samples. 

Overall speaking, this paper investigates an interesting idea. However, according to the reviewers, the technical novelty is limited. Such a data augmentation method is quite commonly used in other fields (and there are many other even more fancy and effective methods to clean or debias the training data). The reviewers also raised their concerns on many other details (including the theoretical justification, experimental settings, complexity analysis, etc.) The authors did a good job in addressing some of them (and some reviewers updated their scores), however, the eventual consensus is that the paper is not that exciting yet, and is below the bar of ICLR.

justification_for_why_not_higher_score: The technical novelty is limited. Such a data augmentation method is quite commonly used in other fields (and there are many other even more fancy and effective methods to clean or debias the training data). The reviewers also raised their concerns on many other details (including the theoretical justification, experimental settings, complexity analysis, etc.).
Decision: Reject