Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors introduce two strategies for predicting interatomic potentials. The first one is based on label augmentation. In this case, an auxiliary training is performed to classify the best-performing physics-based empirical interatomic potentials (EIP) for a given atomic configuration. If a given configuration (C) results in an a reliable energy E (there is also a label for no good classification), then the pair C-E is appended to the DFT-E training set. This strategy yields a performance boost from 18% to 51 %. The second method is based on transfer learning, where a NN is trained using EIP alone and then fine-tuned based on DFT energies. In this case, the improvements are from 18% to 26%. 

strengths_and_weaknesses: Strengths:

The paper proposes a neural network potential based on physics-based EIP and DFT-energy labelled dataset. The novelty of the manuscript is exploiting this multifidelity data using label augmentation, a novel approach for neural-network potentials. Also, the increase in the performance is obtained with little cost in the computational load (mainly the auxiliary training).

Weaknesses:

1. There is no comment on how this potential generalizes outside the selected species. 

2. The methodology is limited to only two single-specie materials.

2. The paper is overall well-presented, but it lacks clarity is several aspects (see below for more details).
questions: 1. Is the proposed method potentially applicable to NequIP? If so, what would the expected outcome be? 

2. The paper would be stronger if at least another dataset is considered.

3. It is not always clear if EIP is machine-learned or physics based. I suggest using two different nomenclatures for the two cases. 

4. A figure in the main text would help understand the paper. The figure should show how the two NNs are connected to each other (e.g. for the first approach through label augmentation)



limitations: The authors state that their work is limited to single-specie materials (Si and Al in this case) and that future work will possible include multi-species ones. I believe limitations are adequately addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper propose to "inject" the domain knowledge in empirical interatomic potentials (EIPs) into neural networks by using the data generated by EIPs. EIPs is much faster than DFT, and reasonably accurate. However,
* multiple EIPs may be applicable (which EIP to trust?) 
* their accuracy varies in the configuration space (when should we trust EIPs?). 

Two strategies are presented:

**LA**: label augmentation (semi-supervised learning). 

A classifier is trained to jointly handle the two issues above. 
* it predicts the best-performing EIP for a given configuration. (which EIP to trust?) 
* If none is sufficiently accurate, outputs a dummy indicator. (when should we trust EIPs?). 

This classifier is trained on configurations where DFT data is available (how much can it be generalized to unseen configurations?). This classifier is then used to augment data using the predicted best-performing EIP on each configuration. This builds an augmented dataset consisting of both EIP samples and DFT samples. As EIP labels are less accurate, a Tukey loss is used as it's less sensitive to outliers (as it's capped). MSE is used if the label comes from DFT.

**MP**: multi-task pretraining (transfer learning)

instead of only using the label from the predicted best-performing EIP, this strategy use all EIPs labels in pretraining in a multi-task way. Then DFT data is used to finetune the model. 

The strategies are then demonstrated with two typical model backbones (SOAPNet as descriptors + MLP, and SchNet as xyz + GNN, ) on two material datasets (KIM-Si for silicon, and ANI-Al for aluminum). Both datasets are energy prediction tasks. Both strategies are shown to be able to reduce MAE and combining MP and LA can achieve even more improvement.

The main contribution is the two strategies proposed and demonstrated to help neural networks with EIPs data, which is usually much cheaper than DFT.

strengths_and_weaknesses: **Strengths**:

* They propose two strategies shown to improve neural networks with EIPs data, which is usually much cheaper than DFT.
* They demonstration with both "descriptors + MLP" and "xyz + GNN" models. These two cases are very representative and therefore support the significance of their results.
* Several designs are reasonable: (e.g., dummy EIP, Tukey loss)

**Weaknesses**:

* The sensitivity of the performance to the selected set of EIPs (8 for silicon and 10 for aluminum as in the paper) is not clearly addressed. How will the performance change if only 2 or 3 EIPs are selected? What's the trend of model performance as the number of EIPs increases?
* can be more sound if the author compare the performance between the learned models vs. the EIPs used to build dataset. This can be evaluated for configurations with DFT labels. 

some typos:
* line15, "bolster" => "booster"
* line165, "DFT and EIP energies" => "DFT energies"
questions: * how exactly are MP and LA combined? firstly train on MP and then train with LA?
* how will performance change if a different/smaller set of EIPs are used.
* is the learned model more accurate than EIPs? (e.g. for configurations with DFT labels)
limitations: * see weakness. The sensitivity of the performance to the selected set of EIPs is not clearly addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this manuscript, the authors proposed to incorporate domain knowledge into machine learning empirical interatomic potentials with two techniques, a weakly supervised learning based on auxiliary classifiers and a pretraining/fine-tuning mode based on transfer learning. Their experiment results have shown a comprehensive outperformance over the baseline methods on systems with a single atomic species.
strengths_and_weaknesses: On the strengths side, the presented method attempts to solve an import problem by leveraging unlabeled training instances generated from EIPs. The paper is well written and easy to follow. The main ideas are clearly explained and the empirical evaluation protocols and results are well presented.

However, both technical and theoretical contributions are inadequate. The auxiliary task modules and pre-training strategies have been widely applied to the GNNs. Albeit effective, these techniques are quite straightforward and lack technical motivation and insights.
Furthermore, the evaluation of the experiment is weak. The authors chose two old methods (SOAPNet in 2013 and SchNet in 2017) for validation and cannot show that these strategies are still valid in the current SOTA frameworks (e.g., DimNet++, arXiv:2003.03123 and GemNet arXiv:2106.08903). The evaluation conducted only on the single atom system also raises serious concerns about its generalization ability.

Overall, I think this work may be more appropriate for a journal in computational chemistry rather than a machine learning conference.
questions: 1. Please use some of the SOTA neural architectures as the base models.
2. Multiple species systems have been included in ANI and some new datasets such as OC20, and methods such as SchNet, DimNet, ForceNet, etc. have been evaluated on these systems. Why do the authors need to leave this as a future direction?

limitations: I did not find any potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to improve the prediction performance of the expensively computed material energy from density functional theory (DFT) by making use of the cheaply computed material energy from empirical interatomic potentials (EIP). Two strategies are proposed to use EIP based labels, including label augmentation and multi-task pretraining. Experimental results show that these two strategies can both improve the prediction performance of two models.
strengths_and_weaknesses: Strengths:  
(+) This work gives a very meaningful exploration of improving the neural network model for DFT computation prediction with large amounts of EIP labeled data given that DFT labeled data is limited. The success of the proposed method can motivate researchers working on developing machine learning models for DFT prediction to consider using cheap data source to improve their models.  
(+) The writing and organization of the paper is clear and easy to follow.  

Weaknesses:
(-) In experiments, SOAPNet and SchNet are used as prediction models. However, they are designed for molecules, not the best models for material data. It is better to use existing material property prediction models, such as CGCNN [1], in experiments.  
(-) Necessary description about how label augmentation and multi-task pretraining strategies are combined in experiments is lacking.  


[1] Xie, Tian, and Jeffrey C. Grossman. "Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties." Physical review letters 120.14 (2018): 145301.
questions: (1) In label augmentation strategy, what is the prediction accuracy of the auxiliary classification model? Since the auxiliary classification model will determine which EIP-approximated energies will be treated as DFT labeled energies, it is important for it to have a high accuracy.  
(2) In experiments, how label augmentation and multi-task pretraining strategies are combined? Are "+MP+LA" results obtained by training two models following the two strategies and ensembling their results, or by some other combining methods?
limitations: N/A
review_title: null

Meta-review: metareview: This paper proposes two strategies for injecting domain knowledge into neural networks for predicting material propreties, these strategies lead to substantial accuracy gains. All reviewers had positive feedback on the paper and their suggestions helped improving the paper and the experiments. Accept 
confidence: Certain
Decision: Accept