Injecting Domain Knowledge from Empirical Interatomic Potentials to Neural Networks for Predicting Material Properties
Authors: Zeren Shui
Daniel S. Karls
Mingjian Wen
ilia Andreyevich Nikiforov
Ellad Tadmor
George Karypis
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain Knowledge
Neural Networks
Empirical Interatomic Potentials
Material Property Prediction

Abstract: For decades, atomistic modeling has played a crucial role in predicting the behavior of materials in numerous fields ranging from nanotechnology to drug discovery. The most accurate methods in this domain are rooted in first-principles quantum mechanical calculations such as density functional theory (DFT). Because these methods have remained computationally prohibitive, practitioners have traditionally focused on defining physically motivated closed-form expressions known as empirical interatomic potentials (EIPs) that approximately model the interactions between atoms in materials. In recent years, neural network (NN)-based potentials trained on quantum mechanical (DFT-labeled) data have emerged as a more accurate alternative to conventional EIPs. However, the generalizability of these models relies heavily on the amount of labeled training data, which is often still insufficient to generate models suitable for general-purpose applications. In this paper, we propose two generic strategies that take advantage of unlabeled training instances to inject domain knowledge from conventional EIPs to NNs in order to increase their generalizability. The first strategy, based on weakly supervised learning, trains an auxiliary classifier on EIPs and selects the best-performing EIP to generate energies to supplement the ground-truth DFT energies in training the NN. The second strategy, based on transfer learning, first pretrains the NN on a large set of easily obtainable EIP energies, and then fine-tunes it on ground-truth DFT energies. Experimental results on three benchmark datasets demonstrate that the first strategy improves baseline NN performance by 5% to 51% while the second improves baseline performance by up to 55%. Combining them further boosts performance.

Injecting Domain Knowledge from Empirical Interatomic Potentials to Neural Networks for Predicting Material Properties
Zeren Shui University of Minnesota shuix007@umn. edu
Daniel S. Karls*
Mingjian Wen" University of Houston mjwen@uh. edu
University of Minnesota kar10100@umn, edu
Ilia A. Nikiforov University of Minnesota nikif002eumn. edu
Ellad B. Tadmor University of Minnesota tadmor@umn, edu
George Karypis University of Minnesota AWS karypis@unn. edu
Abstract
For decades, atomistic modeling has played crucial role in predicting the behavior of materials in numerous fields ranging from nanotechnology to drug discovery. The most accurate methods in this domain are rooted in first-principles quantum mechanical calculations such density functional theory (DFT). Because these methods have remained computationally prohibitive, practitioners have tradition- ally focused on defining physically motivated closed-fore expressions known empirical interatomic potentials (EIPs) that approximately model the interactions between atoms in materials. In recent years, neural network (NN)-based poten- tials trained on quantum mechanical (DFT-labeled) data have emerged as more accurate alternative to conventional EIPs. However, generalizability of these models relies heavily the amount of labeled training data, which is often still insufficient to generate models suitable for general -prrpose applications. In this paper, propose two generic strategies that take advantage of unlabeled training instances to inject domain knowledge from conventional EIPs to NNs order increase their generalizability. The first strategy, based weakly supervised learning, trains an auxiliary classifier on EIPs and selects the best-performing EIP generate energies to supplement the ground-truth DFT energies in training the NN. The second strategy, based on transfer learning, first pretrains the NN on a large set of easily obtainable EIP energies, and then fine-tunes DFT energies. Experimental results on three benchmark datasets demonstrate that the first strategy improves baseline NN performance by 5% to 51% while the second improves baseline performance by up to 55%. Combining them further boosts performance.
ground-truth
Introduction
Predictive modeling of materials is field with manifold applications that has been the subject of many cross-disciplinary studies. While modeling is conducted at different length scales, all material behavior ultimately has its origins the scale, where the interactions between individual atoms must be understood. The most accurate methods at this scale are based on quantum mechanics theory, requiring explicit consideration of the electronic degrees of freedom described by the Schrödinger equation. However, these methods are presently limited to systems containing at most several
"Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

thousand atoms, precluding their use in investigation of important microstructural phenomena such as crack propagation. To overcome this limitation, practitioners have long relied upon heuristic models known as empirical interatomic potentials (EIPs), which consist of physically motivated analytical functional forms that strive to model the complex electronic interactions between atoms using only the nuclear coordinates of the atoms and their elemental species as input [4). In the past several years, there has been a surge of interest in the development of machine learning EIPs, particularly those based on neural networks (NNs), more accurate alternative to traditional EIPs 4. In contrast to traditional EIPs, NN potentials contain little inductive bias and, accordingly, require large volumes of training data labeled using first-pricippes quantum mechanical methods. The first- principles method most commonly used is density functional theory (DFT), which scales as ( with the number of
valence electrons ne in a given configuration of atoms 23 As result of this high computational cost, it is difficult to acquire a sufficient number of labeled training instances to create an NN potential that performs accurately over wide range of applications.
One potential solution to this problem is to seek additional supervision signals. Traditional EIPs attractive sources of such additional supervision for two reasons. First, their functional forms incorporate prior physical information that allows them to correlate with DFT, and, in regions of relevance to common applications, are often quite accurate. That is, they contain domain knowledge that could benefit the training of NN potentials. Second, EIPs scale linearly with the number of atoms and are thus orders of magnitude faster than DFT, permitting the labeling of massive datasets. Despite the advantages, no research has focused on using EIP supervision signals in training NN potentials.
In this paper, we leverage physically motivated EIPs and unlabeled configurations to tackle the label scarcity challenge for training NN potentials. We propose two generic strategies, weakly supervised learning and transfer learning, for exploiting this additional source of information. In the first strategy, we expand the DFT-labeled training set with unlabeled configurations and their EIP energies. To the original DFT-labeled training configurations
achieve this goal, we train an auxiliary classifier that predicts which one of a selected set of EIPs is likely to produce the most accurate estimate of the DFT energy for each of large set of unlabeled configurations. The unlabeled configurations are then labeled by their corresponding predicted best-performing EIPs and appended to the training set. We train NN potentials on the expanded training set by optimizing robust regression loss to mitigate the influence of noise and outliers introduced by the EIP energies. In the second strategy, we adopt transfer learning approach by way of multi-task pretraining. We first pretrain the representation module of an NN potential to reproduce the energies predicted by the EIPs. During the subsequent fine-tuning stage, the representation module of the NN is paired with prediction head and trained on the DFT-labeled configurations. These two strategies can be flexibly used and coupled to train any NN potential.
The contributions of this work are three-fold: 1) We demonstrate that EIPs are capable of providing high quality supervision signals for training NN potentials, which opens a new direction for future development of NN potentials; 2) We propose two effective and generic strategies that take advantage of EIPs and unlabeled configurations to tackle the label scarcity challenge for training NN potentials; 3) We conduct comprehensive experiments on three benchmark datasets and four representative NN potentials that cover most of the NN potential forms currently in use. Experimental results show that the proposed strategies successfully inject domain knowledge from EIPs to NN potentials and improve the performance of the NN potentials by up to 55%.
2 Preliminaries
2.1 Atomic Configurations
The fundamental input in atomistic modeling an atomic configuration. An atomic configuration is spatial arrangement of atoms C = {(Z,, H where Z; and r; are the atomic number and the dire-dimmnsional Euclidean coordinates of atom i, respectively. In the simplest scenario, an atomic configuration corresponds to an isolated cluster of atoms that comprise a molecule. However, may more generally describe a bulk system such as a crystal, which contains an infinite number of atoms distributed over space. These systems are modeled using small collection of atoms in finite simulation cell that is effectively repeated across all of space with the aid of periodic boundary (see Appendix A.I).
conditions (PBCs) (34
2

---

2.2 Physics-based Potentials
Mathematically, an EIP a function, = V(C':), that takes an atomic configuration C' input and returns its total potential energy E; here, 8 denotes set of fitting parameters to be determined. The functional form V of a physics-based EIP is made up of carefully designed analytic expressions that strive to capture the underlying physics in the material it models [43 44]. Because the functional form itself intended parameters (typically on the order of ten) and are usually fitted to set of material properties deemed most relevant to real-world applications. It is expected that physics- EIPs will approximate the first-principles energy surface well in the vicinity of atomic configurations corresponding to the material properties to which they were fit. However, their generalizability can be inconsistent, as shown in Fig.
capture most of the relevant physics, such models need relatively few
2.3 Machine Learning Potentials
In contrast to physics-based EIPs, machine learn- ing EIPs employ general-purpose regression al- gorithms as the functional form V that do not encode any knowledge of the material it models. Therefore, machine learning EIPs are almost ex- clusively trained on large sets of DFT data as to include as much physical knowledge as possible.
BP2
Energy
One important class of machine learning EIPs neural network (NN)-based potentials. In this paper, we define an NN potential M C + E as
C
Configuration Space
(C) = fpred o frep (C),
Figure 1: Schematic illustration the energy land- scape defined over atomic configuration space by two physics-based EIPs and by DFT. Both EIPs are fitted reproduce properties of the DFT energy landscape near atomic configuration A. Away from configuration A, the relative accuracy of the EIPs compared to DFT varies: at point B, EIP 1 is fair approximation while EIP is less accurate; at point C, neither EIP is accurate.
where feep + Roxd is the representation learning module that maps each of the n atoms of C' feature vector of length based on its local environment, and Spedd Rnnd + E the prediction module that maps the feature vectors of the atoms to the total potential energy.
A concrete instance of this type of model are those that use a graph neural network (GNN) 24
40 31, 21 as the representation learning module and multilayer perceptron (MLP) as the prediction module. Before being passed to the representation learning module frep- an atomic configuration C' is first converted to a graph G (V, E)) with nodes V and edges E. One node is defined for each atom the simulation cell, as well as for additional padding atoms representing PBCs if present. An edge is created between any nodes with distance smaller than prescribed cutoff radius. Next, for each atom i, its atomic number Z₁ is one-hot encoded into an initial feature vector (0) of the node corresponding to the atom. The GNN representation learning module fee then updates the feature vectors using a message passing paradigm [15), i.e., the nodes iteratively aggregate information from their neighbors. Formally, the feature vector of node at the + 1-th
16
layer h (((+1) is updated as a function of the feature vectors of its neighbors (ii) and itself the previous layer,
m = Ag (f50) (n .bah .e lie 1sEN(()) h (41) = Update ( (n."mm(")
(1)
where (() and (") are learnable functions, Agg permuration-innarian function that operates on sets of feature vectors, and e/ connecting node i and node j at the 1-th layer. Finally, the prediction module fpre maps the feature vector of each atom at the last layer h; to a corresponding energy contribution and sums them to arrive at the total energy of the configuration, i.c.. E Σ; MLP(h,).
(1) denotes the feature vector associated with the edge
3

---

3 Related Works
3.1 Neural Network Potentials
The first modern NN potential was proposed by Behler and Parrinello [3] In their formulation, an atomic descriptor (i.e.. basis functions that transform an atomic configuration into a fixed-length fingerprint vector) based on the bond lengths and bond angles is passed to an MLP. On top of this formulation, a Monte Carlo dropout technique can be applied to the MLP to equip the potential with the ability quantify its predictive uncertainty [42]. The DeePMD method is a similar approach to that of Behler and Parrinello, except that a novel atomic environment descriptor used [48]. More recently, researchers have developed GNN potentials based on message passing paradigm [30 46].
Another class of GNN potentials such as NequlP 2 and GemNet [14] pass equivariant messages rather than invariant of-the-art performance. All of these models are trained with supervised learning, without exploring the possibilities of leveraging weakly supervised learning or transfer learning take advantage of unlabeled data.
based the formulation of tensor field networks (37] and achieve state-
3.2 Weakly Supervised Learning
Weakly supervised learning refers to techniques that attempt to train machine learning models from incomplete (only portion of training instances are labeled) or inaccurate (noisy labels) supervision signals [50]. Solutions for incomplete supervision usually fall into the category of semi-supervised
learning, which assumes that nearby instances have similar labels [25] 33 29]. For inaccurate
supervision, a model either learns directly from noisy labels with noise-robust algorithms [12 49
or resorts small portion of clean labeled data to reduce the noise 39 45
3.3 Transfer Learning
Transfer learning [1 refers to a machine learning paradigm that transfers knowledge a model learns from one more relevant tasks to benefit target task. Transfer learning has enjoyed great success, especially in the low-data regime, as demonstrated by the rise of pretrained neural networks. This recent trend began with natural language processing when BERT
(9] and successive large pretrained were released and quickly gained popularity in other domains such as
language models [26]
7
computer vision 117 8 10 and graph learning 47 19 28 These methods pretrain large neural networks on self-supervisee
tasks in order to encode common contextual knowledge in the structured input. Another approach imparts domain-specite sppecfcc knowledge by pretraining models on tasks related to the target task but for which abundant labeled data is available [8] 27 The pretrained model is then fine-tuned on the limited training data of the target task.
Methods
4.1 Problem Definition
Let P be set of physics-based EIPs, C be the space of all possible atomic configurations, CDFT Jimi be set of m configurations with corresponding DFT and EIP energies, and CEIP [Ce, [E } be set of n configurations with only physics-based EIP energies. Here, DPTT and E.T denote the energies predicted for configuration C, by DFT and the p-th physics- based EIP, respectively. In practice, physics-based EIPs are much less expensive than DFT, and so the size of CFP is much larger than CDFT, i.e., n >> m. Our goal to use this data from physics-based EIPs to train an NN-based EIP M to closely approximate the DFT energy surface over the space of all atomic configurations, i.c.,
{C,, (E.)) Ipep, m
M(C) ve DFT AA C.
4.2 Label Augmentation
Because physics-based EIPs are developed as approximations to DFT, it is intuitive to use their predictions as surrogate labels for configurations without DFT energies in order to resolve the label scarcity issue of training neural networks. However, there two fundamental challenges. First, as

---

discussed in Sec. while physics physics-based EIPs are designed to be accurate in specific regions of the configuration space and generalize better than those based on machine learning, they may still be inaccurate in other regions. Given an arbitrary configuration for which no DFT energy is available, is unknown which EIP from a given set will yield the most accurate prediction and how large its error will be. Second, surrogating DFT energies with EIP energies inevitably introduces noise, and potentially outliers, into the training set that may have pathological effects.
EIP prediction using an auxiliary classification model. In order to augment training sets with unlabeled configurations and their corresponding ElP-approximated energies, we use an auxiliary classification model to predict the best-performing EIP for a given configuration. For a configuration
C, the classification model predicts a discrete distribution over the EIP set P = - PUps) that indicates their probability of being the most accurate EIP for C, i.e., P (P' C.. We introduce dummy EIP, Pa. to represent the case where none of the physics-based EIPs in P is predicted to approximate DFT to accuracy level c, i... , I 11 > c, Vp E P, where N; is the number of atoms in configuration i; throughout this work, is set to 0.1. By excluding configurations that are labeled with the dummy class from the training set, c acts as a confidence threshold to control the noise and outliers introduced by using the surrogate EIP energies.
The classification model consists of a representation learning module that converts an atomic configu- ration to fixed-length feature vectors (one for each atom in the configuration), readout function that aggregates them to form feature vector describing the entire configuration, and a prediction module that maps the configuration representation to set of probabilities. We train the classification model on CDFT by optimizing a cross-entropy loss and apply it to CET. Configurations with predicted EIP other than pa are assigned the corresponding EIP energy and are merged with the configurations that have DFT energies to arrive at the final training set. We denote the set of EIP-labeled configurations as ČLP = (Cs, (F.) IEP, E.. 1m++
cormuuatiooninvariinn
li=m+1 where 1 I 8 S n is the number of selected configurations not labelled by DFT, and pr and E. are the predicted best-performing EIP and its prediction on C..
Regression with robust loss functions. We train the NN potential using configurations with ground-truth DFT energies and configurations with EIP energies selected by the classification model. In regression problems, models usually trained by optimizing the mean square error (MSE) loss. The MSE loss is sensitive to outliers, as the magnitude of its gradient is linearly proportional to the difference between the predicted value and the ground truth value. To lessen the impact of outliers. we optimize the MSE loss on DFT-labeled configurations, while on EIP labeled poooeld-confiuuraiions. we optimize the Tukey biweight (bisquare) loss
61 51, i.c.,
m++
1 m
L
DFT E
E₂)²
TTkee (E.) E,)
where ES is the model prediction of the energy for configuration C₁, and a > O is a hyper-paramedt that controls the contribution of labeled configurations the loss and its gradient.
The Tukey biweight loss falls under the M-estimation method (00 and is intended to screen outliers using robust statistics of the regression residuals such as median absolute residuals (MAR) and suppress their influence on the gradient. The Tukey loss function is computed as
(1 (+)
Inal S
k
Takee (r:)
(4)
42
|ral
k
6
where r; = -... É, is the residual and is a tuning constant that is commonly set 4.685 to produce 95% efficiency when the errors are normally distributed with standard deviation a. To set the value of kc, we estimate the standard deviation as à MAR/ MAR/0.6745, 0.6675.. From Eq. 4 residuals with absolute values greater than lc are considered outliers and are rejected for gradient computation. In each gradient step, we sample from both DFT- and labeled configurations to form a batch for gradient propppprpag,tion, and the MAR is estimated on residuals of both DFT- and Elp-labeled configurations the batch. Since the MAR and are dynamically estimated during training, the Tukey loss does not introduce additional Sectonaallyyrrrg hyperparameters.
5

---

4.3 Multi-task Pretraining
We propose a multi-task pretraining strategy to encode the domain knowledge in physics- based EIPs to the parameters of the representation learning module by jointly predicting the set of EIP calculations for configurations with only EIP predictions, CrP, and optimizing the following multi-task regression loss:
m
L
Σ Σ PEP i=m-1
ET)
P
During pretraining, couple the representation learning module with (Pl prediction modules (MLPs) generate predictions corresponding different physics-based EIPs, i.e., D - fireed Srep (C:). The pretrained representation module is then fine-tuned with randomly initialized prediction module for the downstream DFT prediction task. Note that the representation module could either be naively fine-tuned configurations with DFT energies by optimizing MSE loss, generated by our proposed label augmentation method by optimizing Eq. 3
the training set
Although transfer learning has been successful in various application domains, it could easily hinder model performance on the target task the pretraining tasks are unrelated to the target task (negative transfer). We argue that predicting the output of physics-based EIPs is relevant and beneficial to the target task of predicting DFT energies. Although physics- across the space of all possible spatial arrangements of atoms. their functional forms incorporate prior physical information that allow them to correlate with DFT over this space. In Sec. []] we empirically demonstrate that the multi-task pretraining strategy successfully encodes domain knowledge into configuration representations and creates a smoother DFT energy surface.
EIPs are not perfectly accurate
4.4 Combining Label Augmentation and Multi-task Pretraining
The label augmentation and multi-task pretraining methods outlined above can be combined with relative ease. The procedure similar to the ordinary label augmentation strategy, but rather than using a randomly initialized representation module frep for the final NN training, the representation module produced by the multi-task pretraining method is used during fine-tuning. Fig. provviees a schematic overview of both strategies, how they relate to one another, and how they can combined.
Experiments
32 dataset and the KIM-Si To test methodology, experiment with three datasets: the ANI-AI
[22 dataset each with a single species, as well as a multispecies AgAu dataset [41]. Detailed descriptions of the three datasets are provided in Appendix A.2. For each dataset, we generate three splits by randomly assigning 20% of the DFT-labeled configurations as test sets and the other 80% as training sets. During training, use 20% of the training set validation All of the reported experimental results are averaged over three different splits to avoid over- over-fitting to specific split. We release our code and the KIM-Si dataset ( for reproducing our experimental results and continuous works.
for model selection.
5.1 Experimental Setting
5.1.1 Neural Network-based Potentials
We evaluate our proposed strategies on two classes of neural network potentials that reflect the majority of machine learning potentials currently in use. The first represents atomic environments using pre-computed descriptors and learns non-linear transformations (MLPs) to map the descriptors to atomic embeddings, while the second uses GNNs to learn atomic representations from configuration graphs. We select one representative potential from each class. For our use the Smooth Overlap of Atomic Positions (SOAP) [I] atomic environment descriptor together with a representation module and prediction module consisting of MLPs; we term this potential SOAPNet in later discussions. For our GNN-based potential, select SchNet (30] CGCNN 46
potential, we
htttppr//gicaaaaccccco/mmsooo/st http::///oo..org1110666666 6084/99 figshare...... 21266064
figshare..12666444

---

Configurations with DFT and
EIP energies, CDPT
Configurations with only EIP energies, CEP
C1
C2
C2
C2
Cs
C₁
C7
Bg
Ey
E
Eg
Eg
Ey
E
E
Eg
E7
EDFT
DDPT
Eg
Label Augmentation
NN Training
Multi-task Pretraining
Training Aux CLS Model
Label Generation
Seep
Seep
fee
ADPT
fing
Fing
C Fi
C 1
f
fa
cbc
E
ITukey Iman
Configurations with predicted best energies, ČRI
DFT energies from CDPT
P(P)(C) P(pz(C)
P(
C₁
C2
C2 DFFT
E DPT
EDFT
CRP
Figure 2: Illustration of the Label Augmentation (LA) and Multi-task Pretraining (MP) strategies and their usage in the training of NN-based potentials for the case of two physics-based EIPs, p and p2- In LA, classifier is trained predict the most accurate EIP energies CEUP for the unlabeled training instances, which are combined with the DFT energies from CDFT in the loss function when training the final NN (blue arrow + green arrow). In MP, the representation module fiep pretrained by simultancously fitting the energies of each physics-based EIP before it is inherited as the initial state for the representation module in the final NN training, where the DFT-labeled instances are again used (purple arrow + green arrow). The two strategies can be combined by incorporating ČAP into the loss while also using multi-task pretraining to initialize frep (blue arrow purple arrow + green arrow).
and GemNet [4]. In our label augmentation experiments, we set the representation module of the auxiliary classification model to be the same kind as the corresponding NN potential, c.g., the classification model used for training the SchNet potential has a SchNet GNN as representation module. Details about the model and training hyper-parameter parameters can be found in Appendix A.4.
5.1.2 Selection of Physics-based EIPs
The phssics-based EIPs used in experiments functional complexity. Because physics-based EIPs are designed for specific elemental species (in this case, aluminum, silicon, and gold and silver systems), different set of physics-based EIPs had to be chosen for each dataset. A total of ten physics-based EIPs were used for aluminum, eight for silicon, and for the gold-silver system (see Tab. 2 in Appendix). They were taken mainly from the Open Knowledgebase of Interatomic Models (OpenKIM))) repository. [35] 36). For detailed information, see Appendix A.3.
selected to encompass differing levels of
Experimental Results
5.2.1 Performance of Label Augmentation and Multi-task Pretraining
As shown in Tab. T our proposed strategies improve the performance of the four baseline NNs on the three benchmark datasets. In particular, the label augmentation strategy improves the baseline NNs by 5% to 51 %, while the multi-task pretraining strategy improves the baselines by 2% to 55%. Combining the two strategies gives further improvement.
httpe://openkin..ogg

---

Table 1: Performance of the two proposed strategies on DFT energy prediction tasks. We report the configuration levell and atom-level mean absolute eπror (MAE, lower is better) in eV and eV/atom, respectively. We denote the label augmentation strategy by LA and the multi-task pretraining strategy by MP. Best performance is shown in bold. Cases where the training procedure failed due to running out of memory are marked OOM.
KIM-Si Atom
ANI-AI Atom
AgAu Atom
Config
Improv.
Config
Improv.
Config
Improv.
Best EIP
1.6326
0.2524
46,4869
0.3561
4.4587
0.2063
SOAPNet
0.7706
0.0975
0.2153
0.0017
0,5422
0.0226
+LA
0,5595
0.0704
27.58%
0,1786
0.0014
18.14%
0.5067
0.0205
07.92%
+MP
0.5717
0.0733
25.28%
0.1744
0.0014
19.12%
0.3962
0.0154
29.34%
+MP+LA
0.5307
0.0657
31.88%
0.1697
0.0013
22.13%
0.3858
0.0154
30.23% %
SchNet
0,4805
0.0718
0.1693
0.0014
0.7290
0.0290
+LA
0,4015
0,0549
19.99%
0,0845
0.0007
51.24%
0,6815
0.0266
07.33%
+MP
0,4034
0.0569
18.40%
0.1296
0.0010
26.00%
0.3353
0.0130
54.65 %
+MP+LA
0.3719
0.0490
27.17%
0.0816
0.0006
53.27%
0.3496
0.0135
52.80%
CGCNN
0,9314
0.1410
0,2410
0.0019
1.6683
0.0625
+LA
0.7476
0.1050
22.61%
0.1786
0.0014
25.44%
1.6065
0.0589
04.71%
+MP
0.8457
0.1253
10.16%
0,2206
0.0017
07.80%
1.4377
0.0532
14.38%
+MP+LA
0.7435
0.1005
%
0.1392
0.0011
41.65% %
1.3857
0.0499
18.55 %
GemNet
0,5138
0.0546
OOM
OOM
OOM
0.9257
0.0342
+LA
0,4691
0.0511
07.55%
OOM
OOM
OOM
0.8381
0.0300
10.87%
+MP
0.5024
0.0531
02.48%
OOM
OOM
OOM
0,5057
0.0185
45.71% 1%
+MP+LA
0,4651
0.0476
11.12%
OOM
OOM
OOM
0.6074
0.0218
35.30%
5.2.2
EIP Energies as High-Quality Supervision Signals for Training NN Potentials
Recall that in the label augmentation strategy, an auxiliary classification model selects unlabeled configurations and predicts their corresponding best-performing physics-based EIPs, which are subsequently used to label them for the training of the NN potential. These configurations are then labeled by the predicted best-performing physics-based EIPs for NN potential training. We investigate the quality of EIP labels and the auxiliary classification model by training NN potentials on three augmented training sets in which the selected unlabeled configurations are labeled by three different sources: DFT-based energies, ground-truth best-performing-11 energies, and predicted best-performing EIP energies. The DFT-labeled configurations (0.8K) are the same for the three training sets. We only conduct this experiment the ANI-AI dataset, all of its configurations have DFT energies available. The number of selected unlabeled configurations is shown in Tab. 6 (in Appendix).
Fig. shows the performance of the NN potentials trained on the three augmented training sets and the original training set (where only DFT-labeled configurations are used). As shown in the figure, expanding the training set with ground truth DFT calculations (bluc) greatly improves the baseline MAE (yellow bar, model trained on the original training set). Labeling configurations with the ground- truth best-performing physics-based EIPs (red bar) performs slightly worse than with DFT energies (blue bar) but still much better than the baseline (yellow bar). This demonstrates that physics-based EIPs are valuable sources of supervision signals for training NN-based potentials. Using the predicted best perffrrrformng physics-based EIPs for labeling (green bar) performs par with the ground-truth best-performing physics-base based EIP labeling (red), revealing the utility of the auxiliary classification model.
5.2.3 Importance of the Robust Tukey Loss
We next conduct experiments to investigate the importance of the Tukey loss and its ability to reject outliers during training. As before, we only conduct this experiment on the ANI-AI dataset, using DFT energies for unlabeled configurations to determine noise and outliers introduced by the predicted best-performing physics-basee based EIPs. We define an unlabeled configuration with a predicted best- performing physics-based EIP to be outlier if the absolute difference between its physics-
8

---

0.23
1.0
elisiam eVibtom cV/ssam
0.30
OFT-OFF
OFTeBust
of sinadeeeen putteettte
1o way Wo
of resous
13
-66
0.29
0.15
10
0.19
-0
0.05
0.00
100
Wh- basddd Potentaas
Traiiing spochs
Trakning epochs
(a)
(b)
(c)
Figure 3: (a) Performance of NN potentials trained on the original training sets (yellow bar, DFT- labeled configurations only) and three augmented training sets whose unlabeled configurations are labeled by DFT energies (blue bar), ground-trute best-performing physics-based EIP energies (red bar), and predicted best-performing physics-based EIP energies (green bar). (b) Percentage of outliers used for computing gradient during training. (c) Standard deviation of residuals during training.
EIP energy and its DFT energy is larger than a threshold, i.e., |r:|:|:::00 N. I E.1 > where N; is the number of atoms in configuration i. We categorize the outliers as mild, normal, and severe by setting e to {0.1, 1 0.2, 0.3). The initial number of outliers introduced by the unlabeled configurations can be found in Tab. 6 (in Appendix). Figs. BD and 3c show that the number of outliers of all kinds included for computing gradients decreases as the training proceeds, demonstrating that as the NN potential gets progressively more accurate, the Tukey loss can effectively eliminate outliers from the training set. We also conduct an ablation study by replacing the Tukey loss in Eq. with the MSE loss. The results in Tab. 4 (in Appendix) show that the models' performance degrades without the Tukey loss.
5.2.4 Visualization of Pretrained Configuration Representations
Fig. Alots the t-SNE 38 2D projections of the training silicon configuration representations colored by their per-atom DFT energies. The left-hand figure plots representations generated by SchNet with random weights and the right-hand figure plots representations generated by a SchNet pretrained by the multi-task strategy. The representations generated by the randomly initialized SchNet do not exhibit any clear patterns and the energy surface rough. In the right-hand figure, representations of the atomic cluster configurations (i.e., isolated groups of atoms) and the bulk configurations (crystals) are clearly separated and form clusters in the t-SNE 2D space. The per-atom DFT energy surface of the right-hand figure is much smoother, i.e., configurations with similar energies are close one another after pretraining. This verifies our previous statement that, although physics-based EIPs lack complete generalizability, they nonetheless correlate reasonably well with DFT over atomic configuration space, and demonstrates that our proposed multi-task pretraining strategy successfully encodes domain knowledge into the NN-based potentials. T-SNE plots for SOAPNet on both datasets and SchNet on the ANI-Al dataset show similar patterns. These figures may be found in Appendix.
6 Conclusion
In this paper, we show that physics-based EIPs can be used to incorporate domain knowledge into machine learning EIPs by providing additional supervision signals on a large body of unlabeled training instances. Two generic strategies are formulated. The first, label augmentation, is based on weakly supervised learning and uses a classifier to select unlabeled training instances to supplement the original labeled training set. The second, multi-task pretraining, is based on transfer learning and uses the predictions of set of physics-basee EIPs on unlabeled training instances to pretrain a machine learning EIP that is subsequently fine-tuned on the original training set. Our methodology is proven using several experiments, including a cross-val validaaaliot study that demonstrates significant performance gains for either strategy individually and further improvement when they are combined, a label augmentation study that shows that using EIP energies as surrogate labels provides nearly as much robustness as using additional DFT-labeled training instances, and a statistical experiment that illustrates the utility of the Tukey loss in diminishing noise and excluding outliers. Finally, we present

---

Before Pretraining
After Pretraining
40
Clusters
Bulks
20
20
0
-20
-20
-20 0
20
-20 0 20
40
4
-2 -1
1
2
Per-atom DFT energy (eV/atom)
Figure 4: T-SNE plots of silicon configuration representations generated by randomly initialized SchNet (left) and SchNet pretrained with our proposed multi-task pretraining strategy (right). Configurations are colored by their per-atom DFT energies. Representations generated by the pretrained SchNet naturally form two clusters that correspond to the atomic cluster configurations (i.e., isolated groups of atoms) and the bulk configurations (i.e., crystals).
reduced-dimension visualizations that indicate the multi-tast pretraining strategy yields training set feature vectors over which the energy varies smoothly.
One direction for future work is to replace the heuristic EIPs by more accurate but still computationally cheap methods, such as the semi-empirical tight binding method. These methods better approximate DFT and thus can lead to further improvement when combined with our proposed approaches.
Acknowledgement
This work was supported in part by NSF (1447788, 1704074, 1757916, 1834251, 1834332), Army Research Office at the University of Houston, Intel Corp, and Amazon Web Services. Access to research and comput- ing facilities was provided by the Minnesota Supercomputing Institute. We thank the anonymous reviewers for their feedback during NeurIPS 2022 review process. We are grateful to Sijie He and Yingxue Zhou for their insightful discussion and inspiration.
the startup funds from the Presidential Frontier Faculty Program
References
[1] Albert P. Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Phys. Rev. B, 87:184115, May 2013. doi: 10. 11111///yyeee......8...... htRp:t/s:://iin aps org/doi/100 100 1108/PhysRevB..o.mm 87. 184115
[2] Simon Batzner, Albert Musaelian, Lixin Sun. Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1): 1-11, 2022.
[3] Jörg Behler and Michele Parrinello. Generalized neural-network representation of high- dimensional potential-energy surfaces. Physical review letters, 98( 98414))4 2007.
[4] Jörg Behler. Four generations of high-dimensional neural network potentials. Chemical Reviews, 121016):10057-10022,,222 20.1.
10

---

[5] Vasileios Belagiannis, Christian Rupprecht, Gustavo Carneiro, and Nassir Navab. Robust optimization for deep regression. In Proceedings of the IEEE international conference on computer vision, pages 2830-2833 2015.
[6] Michael J Black and Anand Rangarajan. On the unification of line processes. outlier rejection, and robust statistics with applications in early vision. International journal of computer vision, 19(1):57-91, 1996.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[9] Jacob Devlin, Ming-War Weig-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota. June 2019. Association for Computational Linguisties. doi: 11863// 1423.
URL
oorpeelooooooooooooooooooo/ NN19/ 1423
[10]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Reprexentations, 2020.
[11] John Fox and Sanford Weisberg. Robust regression. An R and S-Plus companion to applied regression, 91, 2002.
[12] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey.
IEEE transactions on neural networks and learning systems, 25(5))845 869, 2013.
[13] Wei Gao, Lu Wang, Zhi-Hua Zhou, et al. Risk minimization in the presence of label noise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[14] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, vol- ume 34, pages 6790 6802. Curran Associates, Inc., 2021. URL httpp:://proceedingg pdf
neurips httppp/cc..acoeeoe///c////222222
[15] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[16] Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2 11.0677 2021.
[18) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2019.
[19] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
[20] Peter Huber. Robust statistics. In International encyclopedia of statistical science, pages 1. Springer, 2011.

---

[21] Maria Kalantzi and George Karypis. Position fosition-based hash embeddings for scaling graph neural networks. In 2021 IEEE International Conference on Big Data (Big Data), pages 779-789. IEEE, 2021.
[22] Daniel S Karls. Transferability of empirical potentials and the Knowledgebase of Interatomic Models (KIM). PhD thesis, University of Minnesota, 2016.
[23] Efthimios Kaxiras. Atomic and Electronic Structure of Solids. Cambridge University Press, 2003. doi: 10. 10.1017/CB0978051 1 1755545.
[24] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: (609.669999277 2016.
[25] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in represenlation learning, ICML, volume 3. page 896, 2013.
[26] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint urXiv: 1692, 2019.
[27] Saurav Manchanda, Mohit Sharma, and George Karypis. Distant-supervised slot-filling for e-commerce queries. In 2021 IEEE International Conference on Big Data (Big Data), pages 677-686, 2021. doi: 10. 1t9///i 2021.9671825.
[28] Costas Mavromatis and George Karypis. Graph infoclust: Maximizing coarse-grain mutual information in graphs. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 541-553. Springer, 2021.
[29] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An eeverttiityywwarr pseudo-labed selection framework for semi-supervised learning. In International Conference on Learning Representations, 2020.
[30] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and KlbertRobert Müller. Schnet: A continuous-filee convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
[31] Zeren Shui and George Karypis. Heterogeneous molecular graph neural networks for predicting molecule properties. In 2020 IEEE International Conference on Data Mining (ICDM), pages 492-500. IEEE, 2020.
[32] Justin S Smith, Benjamin Nebgen, Nithin Mathew, Jie Chen, Nicholas Lubbers, Leonid Bu- rakovsky, Sergei Tretiak, Hai Ah Nam, Timothy Germann, Saryu Fensin, et al. Automated discovery of robust interatomic potential for aluminum. Nature communications, 12(1):1-13, 2021. doi: 10. 10.10038/441447-011-2177777
[33] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems, 33:596-66 608, 2020.
[34] Ellad B Tadmor and Ronald Miller. Modeling materials: continuum, atomistic and multiscale techniques. Cambridge University Press, 2011.
[35] Ellad B. Tadmor, Ryan S. Elliott, James P. Sethna, Ronald E. Miller, and Chandler A. Becker. The potential of atomistic simulations and the Knowledgebase Interatomic Models. JOM, 63 July 2011. doi: 10.1007 (1077//11137011111 1-0102-- http: //xx. doi. 10.
011837-01--0102-6
[36] Ellad B. Tadmor, Ryan S. Elliott, Simon R. Phillpot, and Susan B. Sinnott. NSF cyberin- frastructures: A new paradigm for advancing materials simulation. COSSMS, 17(6):298-30 3044, December 2013. doi: 10.10166/.coomm 1010666.csssss220000 10.004. URL http:ttt:://a doi. org/ 1016/
j cosssms. 2013. 10.004
12

---

[37] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an and aunslation-eqivvariann neural networks for 3d point clouds. arXiv preprint arXiv:/802.088 19, 2018.
[38] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[39] Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 839-847, 2017.
[40] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 2017.
[41] YiNan Wang, LinFeng Zhang, Ben Xu, Xiao Yang Wang, and Han Wang. generalizable machine learning potential of ag-au nanoalloys and its application to surface reconstruction, segregation and diffusion. Modelling and Simulation Materials Science and Engineering, 30 (2):025003, 2021.
[42] Mingjian Wen and Ellad B Tadmor. Uncertainty quantification in molecular simulations with dropout neural network potentials. upj Comput. Mater, 6(1):124, 2020. doi: 10.1038/ 441524-020-003900
[43] Mingjian Wen, Stephen Carr, Shiang Fang, Efthimios Kaxiras, and Ellad B. Tadmor. Dihedral- angle-corrected registry-dependent interlayer potential for multilayer graphene structures. Phys. Rev. B, 98(23):235404, dec 2018. doi: 10.1 10.000//yyys 0tt.:pyyyrreeo...255440...
[44] Mingjian Wen, Yaser Afshar, Ryan S Elliott, and Ellad B Tadmor. KLIFF; A framework to develop physics-based and machine learning interatomic potentials. Computer Physics Communications, 272: 108218, 2022.
[45] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
[46] Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an ac- curate and interpretable prediction of material properties. Phys. Rev. Lett. 120:145301, Apr 2018, doi: 10.1 10 wet.ett......... PhysRevLett. 120. 145301
httpst:s: aps. wwg//oo/o//oooooo//oooooooooo 1103/
[47] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823, 5823, 2020.
[48] Linfeng Zhang. Jiequn Han, Han Wang. Roberto Car, and Weinan E. Deep potential molecular dynamics: scalable model with the accuracy of quantum mechanics. Physical review letters, I 120(14):14301, 2018
[49] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[50] Zhi-Hua Zhou. A brief introduction weakly supervised learning. National science review, (3):44-53, 2018.
[51] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
13


Reviews:
[{"review_id": "sPOmbg3W8hM", "reviewer": "Reviewer_DKkp", "summary": "This paper proposes to improve the prediction performance of the expensively computed material energy from density functional theory (DFT) by making use of the cheaply computed material energy from empirical interatomic potentials (EIP). Two strategies are proposed to use EIP based labels, including label augmentation and multi-task pretraining. Experimental results show that these two strategies can both improve the prediction performance of two models.", "questions": "(1) In label augmentation strategy, what is the prediction accuracy of the auxiliary classification model? Since the auxiliary classification model will determine which EIP-approximated energies will be treated as DFT labeled energies, it is important for it to have a high accuracy.  \n(2) In experiments, how label augmentation and multi-task pretraining strategies are combined? Are \"+MP+LA\" results obtained by training two models following the two strategies and ensembling their results, or by some other combining methods?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n(+) This work gives a very meaningful exploration of improving the neural network model for DFT computation prediction with large amounts of EIP labeled data given that DFT labeled data is limited. The success of the proposed method can motivate researchers working on developing machine learning models for DFT prediction to consider using cheap data source to improve their models.  \n(+) The writing and organization of the paper is clear and easy to follow.  \n\nWeaknesses:\n(-) In experiments, SOAPNet and SchNet are used as prediction models. However, they are designed for molecules, not the best models for material data. It is better to use existing material property prediction models, such as CGCNN [1], in experiments.  \n(-) Necessary description about how label augmentation and multi-task pretraining strategies are combined in experiments is lacking.  \n\n\n[1] Xie, Tian, and Jeffrey C. Grossman. \"Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.\" Physical review letters 120.14 (2018): 145301.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WQYNx2xNIsY", "reviewer": "Reviewer_29j7", "summary": "In this manuscript, the authors proposed to incorporate domain knowledge into machine learning empirical interatomic potentials with two techniques, a weakly supervised learning based on auxiliary classifiers and a pretraining/fine-tuning mode based on transfer learning. Their experiment results have shown a comprehensive outperformance over the baseline methods on systems with a single atomic species.", "questions": "1. Please use some of the SOTA neural architectures as the base models.\n2. Multiple species systems have been included in ANI and some new datasets such as OC20, and methods such as SchNet, DimNet, ForceNet, etc. have been evaluated on these systems. Why do the authors need to leave this as a future direction?\n", "limitations": "I did not find any potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the strengths side, the presented method attempts to solve an import problem by leveraging unlabeled training instances generated from EIPs. The paper is well written and easy to follow. The main ideas are clearly explained and the empirical evaluation protocols and results are well presented.\n\nHowever, both technical and theoretical contributions are inadequate. The auxiliary task modules and pre-training strategies have been widely applied to the GNNs. Albeit effective, these techniques are quite straightforward and lack technical motivation and insights.\nFurthermore, the evaluation of the experiment is weak. The authors chose two old methods (SOAPNet in 2013 and SchNet in 2017) for validation and cannot show that these strategies are still valid in the current SOTA frameworks (e.g., DimNet++, arXiv:2003.03123 and GemNet arXiv:2106.08903). The evaluation conducted only on the single atom system also raises serious concerns about its generalization ability.\n\nOverall, I think this work may be more appropriate for a journal in computational chemistry rather than a machine learning conference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "L8wZZYy0JLh", "reviewer": "Reviewer_ZTGs", "summary": "This paper propose to \"inject\" the domain knowledge in empirical interatomic potentials (EIPs) into neural networks by using the data generated by EIPs. EIPs is much faster than DFT, and reasonably accurate. However,\n* multiple EIPs may be applicable (which EIP to trust?) \n* their accuracy varies in the configuration space (when should we trust EIPs?). \n\nTwo strategies are presented:\n\n**LA**: label augmentation (semi-supervised learning). \n\nA classifier is trained to jointly handle the two issues above. \n* it predicts the best-performing EIP for a given configuration. (which EIP to trust?) \n* If none is sufficiently accurate, outputs a dummy indicator. (when should we trust EIPs?). \n\nThis classifier is trained on configurations where DFT data is available (how much can it be generalized to unseen configurations?). This classifier is then used to augment data using the predicted best-performing EIP on each configuration. This builds an augmented dataset consisting of both EIP samples and DFT samples. As EIP labels are less accurate, a Tukey loss is used as it's less sensitive to outliers (as it's capped). MSE is used if the label comes from DFT.\n\n**MP**: multi-task pretraining (transfer learning)\n\ninstead of only using the label from the predicted best-performing EIP, this strategy use all EIPs labels in pretraining in a multi-task way. Then DFT data is used to finetune the model. \n\nThe strategies are then demonstrated with two typical model backbones (SOAPNet as descriptors + MLP, and SchNet as xyz + GNN, ) on two material datasets (KIM-Si for silicon, and ANI-Al for aluminum). Both datasets are energy prediction tasks. Both strategies are shown to be able to reduce MAE and combining MP and LA can achieve even more improvement.\n\nThe main contribution is the two strategies proposed and demonstrated to help neural networks with EIPs data, which is usually much cheaper than DFT.\n", "questions": "* how exactly are MP and LA combined? firstly train on MP and then train with LA?\n* how will performance change if a different/smaller set of EIPs are used.\n* is the learned model more accurate than EIPs? (e.g. for configurations with DFT labels)", "limitations": "* see weakness. The sensitivity of the performance to the selected set of EIPs is not clearly addressed. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n* They propose two strategies shown to improve neural networks with EIPs data, which is usually much cheaper than DFT.\n* They demonstration with both \"descriptors + MLP\" and \"xyz + GNN\" models. These two cases are very representative and therefore support the significance of their results.\n* Several designs are reasonable: (e.g., dummy EIP, Tukey loss)\n\n**Weaknesses**:\n\n* The sensitivity of the performance to the selected set of EIPs (8 for silicon and 10 for aluminum as in the paper) is not clearly addressed. How will the performance change if only 2 or 3 EIPs are selected? What's the trend of model performance as the number of EIPs increases?\n* can be more sound if the author compare the performance between the learned models vs. the EIPs used to build dataset. This can be evaluated for configurations with DFT labels. \n\nsome typos:\n* line15, \"bolster\" => \"booster\"\n* line165, \"DFT and EIP energies\" => \"DFT energies\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9hlZ3GVBX3W", "reviewer": "Reviewer_znL5", "summary": "The authors introduce two strategies for predicting interatomic potentials. The first one is based on label augmentation. In this case, an auxiliary training is performed to classify the best-performing physics-based empirical interatomic potentials (EIP) for a given atomic configuration. If a given configuration (C) results in an a reliable energy E (there is also a label for no good classification), then the pair C-E is appended to the DFT-E training set. This strategy yields a performance boost from 18% to 51 %. The second method is based on transfer learning, where a NN is trained using EIP alone and then fine-tuned based on DFT energies. In this case, the improvements are from 18% to 26%. \n", "questions": "1. Is the proposed method potentially applicable to NequIP? If so, what would the expected outcome be? \n\n2. The paper would be stronger if at least another dataset is considered.\n\n3. It is not always clear if EIP is machine-learned or physics based. I suggest using two different nomenclatures for the two cases. \n\n4. A figure in the main text would help understand the paper. The figure should show how the two NNs are connected to each other (e.g. for the first approach through label augmentation)\n\n\n", "limitations": "The authors state that their work is limited to single-specie materials (Si and Al in this case) and that future work will possible include multi-species ones. I believe limitations are adequately addressed.  ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes a neural network potential based on physics-based EIP and DFT-energy labelled dataset. The novelty of the manuscript is exploiting this multifidelity data using label augmentation, a novel approach for neural-network potentials. Also, the increase in the performance is obtained with little cost in the computational load (mainly the auxiliary training).\n\nWeaknesses:\n\n1. There is no comment on how this potential generalizes outside the selected species. \n\n2. The methodology is limited to only two single-specie materials.\n\n2. The paper is overall well-presented, but it lacks clarity is several aspects (see below for more details).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept