SlowGuess/NeurIPS_2023_OCR huggingface dataset:
An important validation of the MLFF model's effectiveness is its alignment with DFT results. If our MLFF model's predictions are consistent with DFT-calculated metrics, it indicates that our model has a good grasp on the intricate interactions between atoms. For ML experts, the EoS metrics serve a dual purpose. Not only do they offer a straightforward and clear measure of the model's performance, but they also bridge the gap between intricate atomic simulations and tangible material properties, making the underlying science more accessible and understandable.
In the real experiment, when the mismatch of lattice constant  $(\mathrm{V}_0^{1/3})$  exceeds over  $5\%$  between materials, it becomes very difficult to synthesize the materials. Thus, a reasonable MLFF model should predict volume accurately within this range. Since the GGA-level of DFT calculation already contains volume error of  $2 - 3\%$  [58], the error on the lattice constant should be less than  $3\%$ , which suggest that error on  $\mathrm{V}_0(|\mathrm{V}_{0,\mathrm{DFT}} - \mathrm{V}_{0,\mathrm{MLFF}}| / \mathrm{V}_{0,\mathrm{DFT}})$  be less than 0.1.
Our framework offers an easy-to-use evaluation and reference data for a diverse range of solid structures (Table D.2). It not only automates the calculation of the bulk modulus and the equilibrium volume but also provides comparative DFT results as a gold standard. As depicted in these Figures D.4 and D.5, less effective MLFF models may struggle in accurately predicting atomic interactions when there's a change in volume due to varying pressures. This misalignment in prediction can lead to deviations in the energy shift graph from the ideal DFT profile. Such discrepancies indicate the model's challenges in capturing energy variations associated with changes in the internal atomic configurations, highlighting potential inaccuracies in forecasting the material's mechanical properties. This approach gives us a broader perspective, enabling an assessment of MLFF's capabilities beyond merely energy and force predictions, and demonstrates its application in real-world simulation scenarios.
D.3 Potential Energy Curves
Potential energy curves for two-body interatomic interactions were computed by manipulating the distance between a pair of atoms. Additionally, specific MLFF models, which incorporate many-body terms, were assessed with an array of molecular structures that encompassed more than just two atoms. These many-body potential energy curves were derived by adjusting particular distances or angles within these molecular structures. In these evaluations, the ground truth obtained from DFT
(a) MAE-trained models on P31c (ID)
(c) MAE-trained models on Fd3m (OOD)
Figure D.4: EoS comparisons: ground truth (DFT) vs. model predictions on SiN P31c (ID) and Fd3m (OOD) structures. Each curve is shifted to reflect the energy difference relative to its minimum value.
(b) MSE-trained models on P31c (ID)
(d) MSE-trained models on Fd3m (OOD)
(a) MAE-trained models on  $\mathrm{P2}_1 / \mathrm{c}$  (ID)
(c) MAE-trained models on Fd3m (OOD)
Figure D.5: EoS comparisons: ground truth (DFT) vs. model predictions on HfO  $\mathrm{P2}_1 / \mathrm{c}$  (ID) and Fd3m (OOD) structures. Each curve is shifted to reflect the energy difference relative to its minimum value.
(b) MSE-trained models on  $\mathrm{P2}_1 / \mathrm{c}$  (ID)
(d) MSE-trained models on Fd3m (OOD)
Table D.3: Details of PEC evaluations.
Formula	Variable*	Range
SiN	two-body	Si-Si	distance (r)	1.000 ~ 6.000 Å
Si-N	distance (r)	1.000 ~ 6.000 Å
N-N	distance (r)	0.500 ~ 6.000 Å
many-body	Si3N	distance (r)	2.000 ~ 6.000 Å
SiN4	distance (r)	2.000 ~ 6.000 Å
HfO	two-body	Hf-Hf	distance (r)	1.000 ~ 6.000 Å
Hf-O	distance (r)	1.000 ~ 6.000 Å
O-O	distance (r)	0.500 ~ 6.000 Å
many-body	HfO2	distance (r)	0.986 ~ 6.786 Å
HfO3	angle (θ)	0 ~ 140°
HfO4	angle (θ)	0 ~ 180°
*Please refer Figure D.6
calculations was used as a reference. For detailed information on these evaluations, please refer to Table D.3 and Figure D.6. In the case of SiN many-body structures, we employed equilateral triangles  $(\mathrm{Si}_3\mathrm{N})$  and tetrahedron  $(\mathrm{SiN}_4)$  as they represent the atomic environment of each element with their first nearest neighbors in crystals. Additionally, the many-body structures of HfO were prepared by referencing a study on monohafnium oxide clusters, which is relevant for understanding defect sites in HfO thin films [59]. PEC evaluation has the potential to be a valuable tool in the solid-state domain, particularly in scenarios where certain chemical reactions or electrochemical properties are expected to take place in rare atomic environments.
D.3.1 Understanding PEC: Insight, Motive, and Explanation
MLFFs have emerged as a significant tool in computational materials science. Through data-driven processes, they can precisely predict energies and forces within molecular configurations. Their performance is considered reliable when they align well with PECs from high-fidelity DFT calculations.
PECs have historically been essential for empirical interatomic potential model fitting. They have served as the cornerstone of empirical methodologies, used to understand energy changes based on varying atomic configurations. Given this foundational role of PECs, their alignment with modern MLFFs becomes crucial.
The alignment of an MLFF with high-fidelity DFT-derived PECs verifies its ability to capture complex interactions and its competence in modeling forces. Notably, forces are derived from the gradients of the molecular energy landscape. The ability to predict energy landscapes, as validated by its congruence with DFT-derived PECs, ensures that the resulting molecular simulations are trustworthy. In summation, the alignment between MLFFs and DFT-derived PECs, especially across diverse molecular configurations, demonstrates the adaptability and effectiveness of the MLFF model.
A distinguishing feature of our framework is its user-friendly evaluation capability across a broad spectrum of molecular structures (Figure D.6) for both systems, especially concerning PECs. While our main focus has been on condensed-phase systems, handling sparse molecular systems poses inherent challenges. These sparse systems act as outliers for models predominantly trained on condensed-phase datasets. Successfully obtaining accurate PECs for such outliers underlines the robustness and versatility of the MLFF approach.
D.3.2 PEC Evaluation
Figures D.9 and D.10 visualize the PEC evaluation results of models for SiN data, while Figures D.11 and D.12 depict the results for HfO. PEC evaluation poses a significant challenge as it involves assessing sparse atomic environments that are rarely encountered by models trained on condensed-phase datasets. While it is not essential for models used in condensed-phase material research to
Figure D.6: Molecules used for PEC evaluations.
excel in PEC evaluation, the evaluation results provided valuable insights into the disparities between the two datasets. The SiN data included a wide range of atomic compositions and environments, including Si-only and N-only structures. In contrast, the HfO data exhibited a less diverse range. Consequently, the PEC evaluation results for SiN showed a closer resemblance to the DFT ground truth curve compared to the results for HfO.
To explain in a slightly more physics-oriented manner, given that most models predominantly train on low-energy data, predicting the high-energy PE surface becomes a challenging extrapolation task for these models. However, predicting this high-energy PE regime, usually occurring when the atoms get close, is important for stable MD simulation.
Two distinct methods can address this discrepancy in PE surface prediction. The first, termed as a data-centric approach, emphasizes the incorporation of more data from the high-energy regime, necessitating careful DFT calculations. The second approach, so called  $\delta$ -learning, tries to combine classical force-field model and MLFF models [60, 61, 62]. With this method, an initial approximation of the PE surface is derived using the classical force-field model, followed by the refinement of the residual portion using an MLFF model. Both strategies present potential solutions when confronting inconsistencies in the PE surface predictions.
D.4 Calculating Indicator Score: Formula and Details
For all simulation indicators and numerical metrics, lower values indicate better results than higher values. To draw radar plots for model comparison, an inverse linear transformation were applied to map them between 0 and 1, where a score of 1 incidates the best performance. Minimum observed error was considered as the perfect score of 1, serving as the ideal benchmark for each metric. In order to maintain reasonable values, we meticulously selected maximum thresholds for each metric. For detailed information regarding the transformation rule and maximum thresholds, please refer to
(b) HfO
Figure D.7: Pearson correlation matrix illustrating the relationships between "EF Metric (ID)", "EF Metric (OOD)", "Static Inds.", "Dynamic Inds.", and "Total Inds." across two datasets: (a) SiN and (b) HfO. The "Static Inds." and "Dynamic Inds." for each model were computed by summing the indicator scores from both ID and OOD cases. The "Total Inds." represents the combined sum of the "Static Inds." and "Dynamic Inds." scores.
Table D.4: Raw scores of models for indicators using SiN dataset and trained with MAE-based loss. Lower values indicate better performance.
EF mtr. ID	EF mtr. OOD	RDFID	RDFOOD	ADFID	ADFOOD	VID0	VOOD0	BID0	BOOD
BPNN	3.2e+2	4.6e+2	4.7e-2	1.5e-1	3.5e-2	1.1e-1	5.4e-1	1.9e-1	2.1e+1	5.1e+1
DPA-1	3.3e+2	4.7e+2	5.0e-2	1.9e-1	3.4e-2	2.2e-1	1.6e-1	1.5e+0	6.0e+0	2.5e+1
SchNet	4.5e+2	8.5e+2	6.0e-2	1.1e+0	3.9e-2	4.3e-1	8.1e-1	1.9e+0	4.8e+1	9.3e+1
DimeNet++	1.2e+2	2.1e+2	4.2e-2	1.1e-1	3.2e-2	9.2e-2	2.8e-1	5.7e-1	1.4e+0	2.7e+0
GemNet-T	1.2e+2	1.4e+2	4.0e-2	3.3e-2	3.0e-2	2.1e-2	1.5e-1	1.1e-1	3.3e+0	1.2e+0
GemNet-dT	1.2e+2	1.7e+2	4.1e-2	6.9e-2	3.1e-2	5.9e-2	1.2e+0	5.8e-1	1.7e+0	3.2e+0
NequIP	1.7e+2	2.5e+2	4.1e-2	7.0e-2	3.1e-2	4.1e-2	8.3e-2	1.4e-1	1.4e+0	5.2e+0
Allegro	2.3e+2	3.5e+2	4.4e-2	4.2e-1	3.3e-2	2.3e-1	2.5e-2	8.2e-2	2.7e+0	2.8e+1
MACE	1.6e+2	2.2e+2	4.0e-2	4.4e-2	3.0e-2	3.5e-2	7.1e-2	1.2e-1	4.5e-1	6.9e+0
SCN	9.8e+1	1.7e+2	4.1e-2	5.7e-2	3.0e-2	4.4e-2	4.5e-1	1.3e+0	5.7e+0	7.9e+0
Table D.8. The raw scores of the models prior to this transformation are summarized in Tables D.4 to D.7. The details of calculating the raw score for each indicator are explained in Sections D.1 to D.3, along with evaluated structures and a description of simulation environments. The performance comparison among models based on indicators after the reverse transformation to scores ranging from 0 to 1 is visualized using radar plots in Figures D.13 and D.14.
D.5 Empirical Model Analysis
BPNN requires hand-crafted features but has simplest overall structure, which makes the model relatively fast and less accurate than recent SOTA models.
DPA-1 has relatively low accuracy on the EF metric, but shows better performance on simulation indicators.
SchNet lies on the pareto-frontier as the fastest model, but the speed gain may not be enough to compensate overall accuracy drop compared to recent fast models such as Allegro or GemNet-dT.
DimeNet++ and GemNet-T are models with similar base structure. They show a similar overall tendency: generally high accuracy but less accurate on the  $\mathrm{V}_0$  indicator, and having relatively slow inference speed. Overall, GemNet-T, which lies on the most accurate side of pareto-frontier, seems slightly better predictive performance than DimeNet++.
Table D.5: Raw scores of models for indicators using SiN dataset and trained with MSE-based loss. Lower values indicate better performance.
EF mtr. ID	EF mtr. OOD	RDFID	RDFOOD	ADFID	ADFOOD	VID0	VOOD0	BID0	BOOD
BPNN	4.7e+2	5.7e+2	5.0e-2	1.7e-1	3.3e-2	1.4e-1	4.7e-1	7.5e-1	1.2e+1	3.2e+1
DPA-1	3.5e+2	4.7e+2	5.0e-2	1.2e-1	3.5e-2	1.5e-1	3.6e-1	2.5e+0	2.0e+0	3.6e+1
SchNet	4.7e+2	8.2e+2	5.7e-2	1.1e+0	3.5e-2	4.9e-1	6.8e-1	1.8e+0	5.1e+1	1.5e+2
DimeNet++	1.2e+2	1.9e+2	4.5e-2	2.1e-1	3.1e-2	7.0e-2	1.9e-1	4.8e-1	5.0e-1	3.0e+0
GemNet-T	1.6e+2	1.8e+2	4.5e-2	3.6e-2	3.0e-2	2.7e-2	3.3e-1	5.3e-1	1.9e+0	1.2e+0
GemNet-dT	1.6e+2	2.0e+2	5.0e-2	1.1e-1	3.1e-2	7.2e-2	7.4e-1	8.0e-1	1.1e+1	2.3e+1
NequIP	1.7e+2	2.5e+2	4.5e-2	8.4e-2	3.1e-2	5.1e-2	1.6e-1	2.7e-1	2.4e+0	6.5e+0
Allegro	2.5e+2	3.4e+2	4.9e-2	1.4e-1	3.2e-2	1.1e-1	2.3e-1	7.2e-1	3.9e-1	9.8e+0
MACE	2.0e+2	2.6e+2	4.6e-2	3.7e-2	3.2e-2	2.4e-2	1.9e-1	6.3e-2	3.2e+0	6.0e+0
SCN	1.5e+2	2.3e+2	6.7e-2	1.2e-1	3.5e-2	1.3e-1	7.7e-1	1.3e+0	7.4e+0	5.8e+0
Table D.6: Raw scores of models for indicators using HfO dataset and trained with MAE-based loss. Lower values indicate better performance. N/A indicates an interrupted simulation for the dynamic indicator due to abnormal energy changes.
EF mtr. ID	EF mtr. OOD	RDFID	RDFOOD	ADFID	ADFOOD	VID0	VOOD	BID0	BOOD
BPNN	2.1e+2	2.3e+2	4.9e-2	1.3e-1	4.3e-2	1.1e-1	3.4e-1	1.1e+0	7.5e+0	1.0e+1
DPA-1	2.5e+2	2.7e+2	7.7e-2	1.3e-1	8.1e-2	9.9e-2	7.7e-1	6.0e-2	8.2e+0	7.5e+0
SchNet	4.1e+2	5.0e+2	5.1e-2	2.4e-1	4.5e-2	1.5e-1	8.0e-1	3.8e+0	7.2e+1	3.8e+2
DimeNet++	5.2e+1	7.7e+1	2.9e-2	6.7e-2	2.2e-2	4.3e-2	9.5e-2	2.2e+0	9.2e-1	3.6e+0
GemNet-T	3.3e+1	5.4e+1	2.9e-2	1.2e-1	2.2e-2	8.5e-2	1.6e-2	9.2e-1	3.0e-1	7.0e+0
GemNet-dT	4.2e+1	6.7e+1	2.9e-2	6.7e-2	2.1e-2	8.5e-2	6.8e-2	1.6e+0	6.5e-1	2.0e+1
NequIP	8.3e+1	1.2e+2	3.3e-2	1.9e-1	3.1e-2	1.5e-1	1.4e-1	9.0e-1	1.9e+0	3.5e+0
Allegro	1.6e+2	3.6e+2	4.7e-2	N/A	4.7e-2	N/A	3.0e-1	1.7e-1	2.6e+0	8.3e+0
MACE	6.6e+1	9.2e+1	3.2e-2	N/A	2.6e-2	N/A	5.9e-2	1.6e-1	5.8e-1	8.7e+0
SCN	4.2e+1	6.7e+1	2.8e-2	6.4e-2	2.1e-2	5.6e-2	4.6e-1	8.6e-1	5.3e+0	5.2e+0
Table D.7: Raw scores of models for indicators using HfO dataset and trained with MSE-based loss. Lower values indicate better performance. N/A indicates an interrupted simulation for the dynamic indicator due to abnormal energy changes.
EF mtr. ID	EF mtr. OOD	RDFID	RDFOOD	ADFID	ADFOOD	VID0	VOOD0	BID0	BOOD
BPNN	2.2e+2	2.3e+2	4.5e-2	9.1e-2	4.5e-2	7.6e-2	6.3e-1	1.2e+0	3.6e+0	6.3e+0
DPA-1	2.3e+2	2.6e+2	7.1e-2	1.3e-1	7.4e-2	8.9e-2	6.3e-1	6.4e-1	5.9e+0	2.8e+0
SchNet	4.2e+2	5.1e+2	9.0e-2	4.1e-1	9.2e-2	1.5e-1	1.1e+0	6.6e+0	1.2e+2	7.7e+2
DimeNet++	4.8e+1	7.0e+1	3.2e-2	1.3e-1	2.5e-2	1.4e-1	6.3e-2	1.9e+0	1.0e+0	3.9e+1
GemNet-T	4.6e+1	6.3e+1	3.4e-2	6.5e-2	2.6e-2	5.1e-2	4.4e-2	9.5e-1	2.9e-1	1.5e+0
GemNet-dT	5.7e+1	7.5e+1	3.2e-2	1.6e-1	2.5e-2	1.3e-1	1.7e-1	1.1e+0	1.5e+0	1.3e+1
NequIP	8.0e+1	1.0e+2	3.8e-2	1.4e-1	3.5e-2	8.5e-2	5.8e-2	2.7e-1	1.1e+0	8.1e-1
Allegro	1.6e+2	2.3e+2	4.8e-2	N/A	4.7e-2	N/A	3.3e-1	1.9e-1	9.4e-1	1.4e+1
MACE	6.5e+1	1.4e+2	3.3e-2	1.0e-1	2.6e-2	7.0e-2	3.1e-2	5.4e-1	3.7e-1	3.1e+0
SCN	5.7e+1	8.1e+1	2.9e-2	8.0e-2	2.2e-2	7.1e-2	3.2e-1	1.5e+0	2.2e+0	3.7e+0
Figure D.8: Hf collision during the evaluation of Allegro in dynamic indicators. (a), (b), (c), and (d) are presented in chronological order, with each atom's relative force magnitude and direction indicated by a yellow arrow.
Table D.8: The maximum threshold  $(\mathrm{TH}_{\max})$  used to transform metrics for the numerical comparison. The transformation rule is as follows:  $x' = \frac{\mathrm{TH}_{\max} - x}{\mathrm{TH}_{\max} - x_{\min}}$ , where  $x$  and  $x'$  represent the original and transformed scores, respectively.  $\mathrm{TH}_{\max}$  is the maximum threshold, and  $x_{\min}$  represents the minimum value among the model evaluation results, where achieving the minimum signifies the most desirable outcome for each metric.
Metric	xminHfO	SiN	THmax
Energy and force erro metric (EF mtr.)	3.3e+1	9.8e+1	8.5e+2
Radial distribution function (RDF)	2.8e-2	3.3e-2	4.5e-1
Angular distribution function (ADF)	2.1e-2	2.1e-2	4.5e-1
Bulk modulus (B0)	1.9e-1	3.9e-1	5.0e+1
Equilibrium volume (V0)	1.6e-2	2.5e-2	3.0e+0
GemNet-dT employs direct force prediction, which does not require backpropagation to compute forces from energy. As a result, it provides a high inference speed with low accuracy drop on ID samples, but not on OOD samples.
NequIP and MACE show high accuracy on both EF metrics and simulation indicators. However, MACE is 1.7x is faster than NequIP with similar accuracy. Compared to GemNet-dT, MACE has equivalent EF accuracy similar prediction results on energy and force with a slightly slower inference speed, but it works better on simulation indicators, and on OOD samples.
Allegro has a high inference speed and decent EF accuracy, however, it show perform unstable on MD simulations with OOD samples.
SCN achieves high overall accuracy, but has a significantly slow inference speed which is 5x slower than GemNet-T, and 9x slower than MACE.
Figure D.9: Comparison of two-body PECs for SiN using MLFF models: models trained with MAE-based loss (left) and MSE-based loss (right).
Figure D.10: Comparison of many-body PECs for SiN using MLFF models: models trained with MAE-based loss (left) and MSE-based loss (right).
Figure D.11: Comparison of two-body PECs for HfO using MLFF models: models trained with MAE-based loss (left) and MSE-based loss (right).
Figure D.12: Comparison of many-body PECs for HfO using MLFF models: models trained with MAE-based loss (left) and MSE-based loss (right).
(a) MLFFs trained with MAE-based loss.
(b) MLFFs trained with MSE-based loss.
Figure D.13: Comprehensive comparison of models on the SiN dataset, based on EF metric and simulation metrics. Higher values indicate better performance. The red and blue plots represent the results for ID and OOD, respectively.
(a) MLFFs trained with MAE-based loss.
(b) MLFFs trained with MSE-based loss.
Figure D.14: Comprehensive comparison of models on the HfO dataset, based on EF metric and simulation metrics. Higher values indicate better performance. The red and blue plots represent the results for ID and OOD, respectively.