SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(f) MACE
Figure 4: Data scaling effect. The EF metrics are obtained from six MLFF models trained by using  $20\%$ ,  $40\%$ ,  $60\%$ ,  $80\%$ , and  $100\%$  snapshots of the HfO training set. For all the points of Test (red) and OOD (yellow) curves, the test and OOD sets are identically used to obtain EF metrics. At each point of Train (blue) curves, the correspondingly sampled training set is used.
the base architecture used in Section 4.2.1. The relative variation scale was selected from  $\{0.25\mathrm{x},$ $0.5\mathrm{x},2\mathrm{x}\}$ ; the variation choices for each model are presented in Section C.4.2. To ensure clarity, no variant models within each model that fell outside the trade-off were included.
All GemNet-dT models are located on the Pareto-frontier curve. Similar to energy, GemNet-dT predicts forces only by the forward process in GNNs, instead of the conventional calculation in MLFFs, where the forces are calculated by the derivatives of energy with respect to the atom positions to satisfy the energy conservation law. The direct force prediction of GemNet-dT does not satisfy the energy conservation law, leading that the property benchmark results of GemNet-dT fall significantly short of those of GemNet-T (see Section 5.3). However, GemNet-T required a 2.2 times longer inference time and a much larger GPU memory than GemNet-dT, implying that GemNet-dT is more suitable for large-scale simulations by handling more atoms in less time. Therefore, when considering the aforementioned trade-off, a model development approach based on direct force prediction is a promising direction.
Although SchNet is the fastest, it yields the highest error, and thus, may generate unreliable simulations. SCN, which also adopts direct force prediction, achieves an EF metric comparable to that of GemNet-dT; however, its computational requirement is over 10 times higher, making it unsuitable for simulations. Except for GemNet-dT, Allegro and MACE show the Pareto-frontier results indicated by the dashed line. If the significant discrepancy in the EF metrics between the test and OOD sets is resolved, they can be employed in simulations where the energy conservation law should be satisfied.
4.4 Data Scaling Effect
We also present the data-scaling effect, which may be helpful in adopting a training strategy for MLFF researchers. We randomly sample  $20\%$ ,  $40\%$ ,  $60\%$ , and  $80\%$  snapshots from the HfO training
set (i.e., 5.6k, 11.2k, 16.8k, and 22.4k), and trained models by 1000, 500, 334, and 250 epochs, respectively, to train models by the same iterations; the detailed setting is described in Section C.5. We selected six models and evaluated them on the HfO dataset.
As shown in Figure 4, for all models, as more training data are used, the training error increases while the test error decreases, and the gap between these two errors also decreases. In most cases, the OOD error decreases. These results suggest that training with fewer epochs and more data can improve the generalization performance of MLFF models. Meanwhile, the difference between the EF metrics of the models trained using  $80\%$  and  $100\%$  snapshots of the training set is marginal, indicating that performance cannot be improved using more data. This observation implies that the training set sampled from the raw dataset is sufficient for our semiconductor MLFF benchmark.
5 Property Benchmark: Simulation Indicators
To evaluate the accuracy and reliability of the models in predicting material properties, we introduce a set of five indicators: radial distribution function (RDF), angular distribution function (ADF), bulk modulus  $(\mathrm{B_0})$ , equilibrium volume  $(\mathrm{V_0})$ , and various potential energy curves (PECs). In this study, the indicators are categorized into two types depending on the necessity of the MD simulation during evaluation: RDF and ADF are classified as dynamic indicators, whereas the remaining indicators are classified as static indicators. An evaluation of any model using these indicators can be easily performed within our framework. By utilizing these indicators, researchers can enhance their understanding of the MLFF model behavior and performance.
5.1 Dynamic Indicators
The dynamic indicators, RDF and ADF, are derived from high-temperature MD trajectories, allowing us to assess the stability of the models in different atomic environments affected by turbulence or active movement induced by high thermal energy. For SiN (1200 K) and HfO (1200 K and 1800 K), high temperatures were carefully selected to investigate the model stability. On the one hand, RDF, also known as the pair correlation function, captures density changes relative to the distance from a chosen reference particle. On the other hand, ADF expands the analysis beyond radial distances by characterizing the angular distribution of particles surrounding a reference particle. The analysis of RDF and ADF is fundamental in simulations as it aids in identifying structures, phases, and interactions while providing a deeper understanding of the behaviors and reactions occurring within solids. We conducted evaluations using various structures and a wide range of supercells, consisting of 1,296-2,835 atoms for SiN and 2,592-3,456 atoms for HfO, highlighting the significant number of atoms involved (Table D.1). We have calculated representative values (RDF $^{\text{ID}}$ , RDF $^{\text{OOD}}$ , ADF $^{\text{ID}}$ , and ADF $^{\text{OOD}}$ ) for the evaluation results of RDF and ADF; the details are described in Section D.1.
5.2 Static Indicators
The static indicators,  $\mathbf{B}_0$ ,  $\mathbf{V}_0$ , and PECs, are derived from the relationships between input structure and energy, without requiring MD trajectories, offering valuable insights into the atomic and structural characteristics of materials.
Bulk Modulus and Equilibrium Volume. The Birch-Murnaghan equation of state (EoS) provides crucial information about thin-film materials, such as SiN and HfO, through two parameters:  $\mathrm{B_0}$  and  $\mathrm{V_0}$  [50]. These parameters are crucial for studying film stability, elasticity, and interfacial properties.  $\mathrm{B_0}$  signifies the response of films to pressure and external stress and indicates their flexibility, hardness, and resistance to volume changes.  $\mathrm{V_0}$  helps determine the films' thickness-residual stress relationship, enabling evaluation of their state (strained or relaxed) and refinement of the growth procedure. Understanding these parameters is vital for engineering thin-films with desired properties for potential applications in electronics, optoelectronics, and micro-electromechanical systems. To compare the models, we obtained the final scores for each model (i.e.,  $\mathrm{B_0^{ID}}$ ,  $\mathrm{B_0^{OOD}}$ ,  $\mathrm{V_0^{ID}}$ , and  $\mathrm{V_0^{OOD}}$ ) through the EoS evaluation for different structures. For further information regarding the calculation, please refer to Section D.2.
Potential Energy Curves. The performance of the MLFF models at the atomic level was assessed using PECs. This indicator focuses on evaluating systems that have a sparse atomic environment with a small number of atoms, unlike the dense periodic boundary conditions encountered in our
Figure 5: Comprehensive comparison of models based on EF metric and simulation indicators, where red and blue plots visualize the results of ID and OOD, respectively. Model names colored by black and red represent the models trained using MSE- and MAE-based loss, respectively.
solid-state structure dataset. By assessing the model's ability to accurately represent the interactions and dynamics in sparse environments, this evaluation serves as an OOD assessment. This is particularly relevant in scenarios involving molecular reactions, surface interactions, or nanoscale systems. Understanding the model's performance in these contexts provides valuable insights into its generalization capabilities beyond the bulk behavior observed in solid-state structures. For detailed information on PEC evaluations, please refer to Section D.3.
5.3 Results and Discussion
Figure 5 shows a comparison of the models based on the EF metric, incorporating dynamic and static indicators. For detailed information on the calculation of scores, please refer to Section D.4. Each model was trained with either MAE-based or MSE-based loss. Among them, the model with the highest sum of all indicator scores is shown in Figure 5.
Model Comparison. Overall, for both SiN and HfO, GemNet-T, NequIP, MACE, and SCN achieve prominent results; however for SiN,  $\mathrm{V}_0$  of SCN exhibits modest performance. The descriptor-based models, BPNN and DPA-1, show intermediate performance on both datasets. SchNet shows a significant mismatch in  $\mathrm{B}_0$  for both datasets and have the poorest generalization performance.
Correlations of Metrics. In the case of HfO, DimeNet++ and GemNet-dT achieve excellent results in terms of their EF metric scores for the ID and OOD sets, as shown in Figure 1. However, the performances of the two models in terms of the simulation indicators for OOD are not consistent. DimeNet++ shows a significant decrease in OOD performance for  $\mathrm{V_0}$  prediction, while GemNet-dT exhibits a substantial decrease for both  $\mathrm{V_0}$  and  $\mathbf{B}_0$  predictions. This indicates the limitations of the existing evaluation approach, which is primarily focused on energy and force, emphasizing the need for evaluation metrics related to material simulations and properties. For SiN, DimeNet++ and GemNet-dT do not show such a pronounced trend, which could be attributed to the characteristics of the dataset.
Unlike SiN, HfO was generated using the m-MQA method at only one composition ratio (1:2), thus making it relatively challenging to evaluate OOD performance compared to SiN, which was created by considering various defects, compositions, structures, and environments. Although evaluating the EF metric for OOD alone cannot guarantee the performance of the simulation indicators, the EF metric for OOD exhibited a higher Pearson correlation with the simulation indicators than the EF ID evaluation (see Figure D.7). This suggests that the simulation indicators can serve as a guide for evaluating the extrapolation performance of MLFF models.
Simulation Failure. In terms of the OOD evaluation of the dynamic indicators, Allegro performs moderately on SiN but scores 0 on HfO. This difference is due to the interrupted HfO simulations caused by unrealistic atomic collisions during high-temperature simulations (Figure D.8). These interrupted simulations show abnormal atomic movements, with two Hf atoms attracting each other despite being too close, violating the expected repulsion behavior and terminating the simulation. These issues in HfO can be attributed to the differences in the datasets. In the SiN dataset, simulations of single element systems, such as dense structures consisting solely of Si and N elements, were included, whereas the HfO dataset lacked such simulations. Consequently, relatively more datapoints in the SiN training set encompassed situations with short interatomic distances, allowing the model to better learn and handle these scenarios. The variation in dataset characteristics is also evident in the PEC evaluation, which is discussed in further detail in Section D.3.2.
We introduced an MLFF benchmark suite consisting of SAMD23 datasets with two semiconductor materials and ten state-of-the-art models with six evaluation metrics. The experimental results show that benchmarking only the energy and force prediction errors is insufficient, and simulation-based metrics should be used to reflect the practical utility of a model. In addition, evaluation using the OOD sets is important to correctly understand the prediction performance of models. Moreover, although recently developed GNN-based models such as GemNet-T, NequIP, and MACE have shown relatively better simulation quality, no clearly superior models emerged. If we consider the predictive performance of energy and forces and inference time for large-scale simulations, GemNet-dT becomes a viable option; however, it should be improved to operate properly in simulations. We anticipate that our benchmarks will encourage future developments in the field, resulting in MLFF models that satisfy the accuracy, speed, and scalability requirements. We hope that our benchmark will provide valuable insights into the field of electronics.
Limitations. The model that was shown to perform well in other fields (e.g., molecules) may have been underestimated in this benchmark. Unlike datasets focused on small molecule structures, this study considered condensed-phase systems containing many atoms, making it practically impossible to include all possible atomic environments in the dataset. In addition, simulations handling long-range interactions beyond a few hundred nanometers were not included in the training data, because the high computational cost of ab initio simulations limits the number of atoms. Despite using periodic boundary conditions and including larger structures, our DFT-based data generation scheme cannot fully account for long-range interactions.
Future Plans. In future works, we intend to continuously incorporate state-of-the-art MLFF models and expand and share datasets with various compositions openly. Specifically, in the immediate future, we will introduce datasets on the Hf-Si-O and Hf-Zr-O systems, incorporating diverse chemical stoichiometries to ensure their broad applicability across a wide spectrum of compositions. In addition, plans are in motion to sequentially release diverse semiconductor datasets, including ternary systems or higher, with constituents such as Si, Ge, C, N, O, H, Hf, Zr, B, P, Ga, and As. This extension will guarantee the exploration of various atomic ratios, allowing for a more comprehensive evaluation of model performance. As training models from scratch for every appending dataset is inefficient, we are planning to adopt efficient training schemes such as transfer learning, which we expect to reduce the cost of model training for growing datasets. It is also important to subjoin additional metrics and loss factors such as stress to evaluate the compatibility with various simulation conditions (e.g., NPT), and to reflect indicators during training to provide better prediction performance in practice. Finally, since the main purpose of MLFFs with respect to semiconductors is to enable large-scale simulations that traditional ab initio methods cannot perform, we will extend our benchmark to include the compression and parallelization performance of the models.
We express our gratitude to Albert Musaelian, the lead researcher of the NequIP and Allegro in Materials Intelligence Research group at Harvard University, for providing us with an opportunity to enhance the fairness of our benchmark results. In January 2024, he informed us that an updated configuration of Allegro succeeded in achieving a lower EF metric, surpassing the performance of the configuration we utilized in our study before February 2023. Our forthcoming arXiv manuscript will present additional results based on this updated Allegro configuration.
[1] Seung Soo Kim, Soo Kyeom Yong, Whayoung Kim, Sukin Kang, Hyeon Woo Park, Kyung Jean Yoon, Dong Sun Sheen, Seho Lee, and Cheol Seong Hwang. Review of semiconductor flash memory devices for material and process issues. Advanced Materials, 2022.
[2] Roberto Bez, Paolo Fantini, and Agostino Pirovano. Historical review of semiconductor memories. In Semiconductor Memories and Systems, pages 1-26. Elsevier, 2022.
[3] Ndubuisi G Orji, Mustafa Badaroglu, Bryan M Barnes, Carlos Beitia, Benjamin D Bunday, Umberto Celano, Regis J Kline, Mark Neisser, Yaw Obeng, and AE Vladar. Metrology for the next generation of semiconductor devices. Nature Electronics, 1(10):532-547, 2018.
[4] Koji Nakamae. Electron microscopy in semiconductor inspection. Measurement Science and Technology, 32(5):052003, 2021.
[5] Dylan M Anstine and Alexandr Isayev. Machine learning interatomic potentials and long-range physics. The Journal of Physical Chemistry A, 127(11):2417-2431, 2023.
[6] Kyuhyun Lee, Dongsun Yoo, Wonseok Jeong, and Seungwu Han. Simple-nn: An efficient package for training and executing neural-network interatomic potentials. Computer Physics Communications, 242:95–103, 2019.
[7] Emir Kocer, Tsz Wai Ko, and Jörg Behler. Neural network potentials: A concise overview of methods. Annual Review of Physical Chemistry, 73:163-186, 2022.
[8] Sina Stocker, Johannes Gasteiger, Florian Becker, Stephan Gunnemann, and Johannes T Marggraf. How robust are modern graph neural network potentials in long and hot molecular dynamics simulations? Machine Learning: Science and Technology, 3(4):045010, 2022.
[9] Xiang Fu, Zhenghao Wu, Wujie Wang, Tian Xie, Sinan Keten, Rafael Gomez-Bombarelli, and Tommi S. Jaakkola. Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations. Transactions on Machine Learning Research, 2023.
[10] Aidan P Thompson, Laura P Swiler, Christian R Trott, Stephen M Foiles, and Garritt J Tucker. Spectral neighbor analysis method for automated generation of quantum-accurate interatomic potentials. Journal of Computational Physics, 285:316-330, 2015.
[11] Albert P Bartók, Mike C Payne, Risi Kondor, and Gábor Csányi. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Physical Review Letters, 104(13):136403, 2010.
[12] Albert P Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Review B, 87(18):184115, 2013.
[13] Jörg Behler and Michele Parrinello. Generalized neural-network representation of high-dimensional potential-energy surfaces. Physical Review Letters, 98(14):146401, 2007.
[14] Yaolong Zhang, Ce Hu, and Bin Jiang. Embedded atom neural network potentials: Efficient and accurate machine learning with a physically inspired representation. The Journal of Physical Chemistry Letters, 10(17):4962-4967, 2019.
A Details of Datasets
Table A.1: Overview of total simulations performed for the HfO dataset.  $N_{\mathrm{cond}}$  represents the total number of simulation conditions,  $N_{\mathrm{str}}$  is the total number of MD snapshots, and  $N_{\mathrm{f}}$  indicates the total number of atomic force data. Structures with index (idx.) indicate the same crystal family but have different lattice parameters.
Element	Structure (idx.)	Simulation Condition	\(N_{\text{cond}}\)	\(N_{\text{str}}\)	\(N_{\text{f}}\)
Hf & O	Monoclinic	m-MQA method	6	16,000	1,536,000
Tetragonal	m-MQA method	6	16,000	1,536,000
Cubic	m-MQA method	6	16,000	1,536,000
Orthorhombic (1)	m-MQA method	6	16,000	1,536,000
Orthorhombic (2)	m-MQA method	6	16,000	1,536,000
Randomized Structures (ID)	m-MQA method	30	80,000	7,680,000
Hf & O	Randomized Structures (OOD)	m-MQA method	12	32,000	3,072,000
Table A.2: Overview of total simulations performed for the SiN dataset.  $N_{\mathrm{cond}}$  represents the total number of simulation conditions,  $N_{\mathrm{str}}$  is the total number of MD snapshots, and  $N_{\mathrm{f}}$  indicates the total number of atomic force data.
Element	Structure	Simulation Condition	\(N_{\text{cond}}\)	\(N_{\text{str}}\)	\(N_{\text{f}}\)
Si & N	Amorphous	High T MD (2000, 4000 K; w/o strain)	11	13,650	1,006,500
High T MD at 1500K (w/ strain ±7%)	4	4,000	306,000
Quenching (4000 to 300 K; w/o strain)	8	26,150	1,305,600
Quenching (4000 to 300 K; w/ strain ±7%)	4	2,000	153,000
Structure Relaxation (w/o strain)	11	1,274	77,952
Structure Relaxation (w/ strain ±7%)	4	570	38,760
Crystals (\( \alpha \), \( \beta \), \( \gamma \))	MD (300, 1200, 2100 K)	9	18,000	504,000
Structure Relaxation	9	569	15,932
Defect Structures	MD with Divacancy (2000 K)	3	300	87,600
MD with Grain-Boundary (2000 K)	5	500	103,600
Surfaces	MD (300, 1200 K)	6	6,000	480,000
Structure Relaxation	6	1,400	105,600
Isolated Clusters	MD (300, 1200 K)	6	1,200	142,400
Structure Relaxation	6	600	70,800
Si	Crystals (Diamond, SC, BCC, FCC)	MD (2100 K; w/o strain)	4	2,000	69,500
MD (2100 K; w/ strain ±7%)	8	4,000	139,000
Defect Structures	MD with Divacancy (2100 K)	1	150	32,100
MD with Two Vacancies (2100 K)	1	100	51,000
N	\( N_2 \)Molecules	MD (2100, 4000 K)	4	2,000	128,000
Si & N	Amorphous (OOD)	Melt, Quench, and Relaxation	3	3,700	388,500
A.1 Guidance
Simulation data can be flexibly utilized by researchers to study interatomic potential models according to their specific preferences. For example, the HfO dataset was generated using the m-MQA method, which involved initial structures encompassing various crystals as well as randomized structures. In our HfO dataset, we included both types of structures to provide researchers with a comprehensive dataset. Researchers have the flexibility to train their models using a subset of the dataset consisting of randomized structures and evaluate their performance on the remaining portion of the dataset comprising crystal structures, and vice versa. This approach enables thorough testing and assessment of model performance across different structural configurations within the HfO dataset.
(f) SC
(a) Amorphous
(b) Crystals  $(\alpha, \beta, \gamma)$
(h) FCC
(i) Diamond
(j) 1D defect
(c) Defects (1D & 2D)
(d) Isolated cluster
Figure A.1: Reference atomic structures employed in the SiN dataset.
(e) Surface
(k)  $\mathbf{N}_2$  molecules
Researchers can modify the SiN simulation data to investigate the generalization performance of MLFF models. Despite the integrated SiN dataset containing not only SiN compounds but also Si-only structures and N-only structures, researchers have the choice to exclusively use SiN compounds for model training. Subsequently, they can evaluate the generalization performance of these models on Si- and N-only structures. However, it is crucial to recognize the significant challenge arising from the substantial disparities in sample distributions between SiN compounds and the Si- and N-only structures.
The generation of specialized datasets for silicon nitride necessitates a high degree of domain expertise. However, our proposed m-MQA scheme represents a notable advancement by presenting a streamlined approach for generating the HfO dataset. In contrast to conventional methods reliant on specialized knowledge, this scheme empowers researchers familiar with DFT to effortlessly generate the dataset. By simplifying the dataset generation process and relieving the requirement for extensive domain expertise, our m-MQA scheme enhances accessibility for researchers, facilitating advancements in MLFF research.
Figure A.2: Distribution of energy per atom and forces for SiN datasets: (a) energy per atom and (b) force per atom.
(a)
(b)
Figure A.3: Distribution of energy per atom and forces for selected MD simulations for HfO: (a) energy per atom and (b) force per atom.
A.2 Dataset Characteristics
To illustrate the diversity of datasets of our DFT calculations, we analyze the energy and force distribution for each DFT calculation. Figure A.2 shows the energy and force distributions of the representative crystal structures in the SiN datasets. For brevity, we only plot the energy and force distribution of single MD scenario for each structure. The SiN dataset contains numerous phases, showing well-separated variations for both energy and forces. The force distribution is particularly interesting since it reflects the structural properties of each MD simulation. The force distribution shows a considerable variation ranging from 0 to  $6\mathrm{eV} / \AA$  for amorphous phases. However, the highly symmetric crystalline phases  $(\alpha ,\beta ,$  and  $\gamma$  -SiN) show that most of the forces on atoms are zero. MD simulations on face-centered cubic (FCC), body-centered cubic (BCC), and simple cubic (SC) also show relatively large variations of forces, but their distributions are narrower than amorphous cases. The N-only cases show similar distribution as the SiN-crystalline cases.