SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Geonu  $\mathbf{K}\mathbf{m}^{1*}$  Byunggook Na $^{1*}$  Gunhee  $\mathbf{K}\mathbf{m}^{1\dagger}$  Hyuntae Cho $^{1}$  Seung-Jin Kang $^{1}$  Hee Sun Lee $^{1}$  Saerom Choi $^{1}$  Heejae Kim $^{1}$  Seungwon Lee $^{1}$  Yongdeok Kim $^{1\dagger}$  Samsung Advanced Institute of Technology (SAIT)
{geonu.kim, byunggook.na, ghij.kim, robert.cho, sj1222.kang}@samsung.com {heesun88.lee, sincere.choi, heejaeee.kim, seungw.lee, yd.mlg.kim}@samsung.com
As semiconductor devices become miniaturized and their structures become more complex, there is a growing need for large-scale atomic-level simulations as a less costly alternative to the trial-and-error approach during development. Although machine learning force fields (MLFFs) can meet the accuracy and scale requirements for such simulations, there are no open-access benchmarks for semiconductor materials. Hence, this study presents a comprehensive benchmark suite that consists of two semiconductor material datasets and ten MLFF models with six evaluation metrics. We select two important semiconductor thin-film materials silicon nitride and hafnium oxide, and generate their datasets using computationally expensive density functional theory simulations under various scenarios at a cost of 2.6k GPU days. Additionally, we provide a variety of architectures as baselines: descriptor-based fully connected neural networks and graph neural networks with rotational invariant or equivariant features. We assess not only the accuracy of energy and force predictions but also five additional simulation indicators to determine the practical applicability of MLFF models in molecular dynamics simulations. To facilitate further research, our benchmark suite is available at https://github.com/SAITPublic/MLFF-Framework.
The evolution of semiconductor devices has been guided by two main factors: miniaturization and sophisticated design implementation [1, 2, 3]. Understanding complex device structures and fabrication processes requires high-resolution metrology methods, such as scanning electron microscopy and transmission electron microscopy [3, 4]. However, these instruments have limitations in terms of providing a restricted view, requiring substantial time and cost for analysis. In this context, large-scale atomic simulations are becoming increasingly important as a viable alternative approach that is capable of accurately capturing device-level dynamics. Although ab initio simulations based on the density functional theory (DFT) can provide accurate atomic simulations, substantial computational demands make them unsuitable for large-scale simulations [5, 6, 7]. As a practical substitute, molecular dynamics (MD) simulations combined with machine learning force fields (MLFFs) have gained significant attention. MLFFs aim to achieve a precision comparable to that of DFT-based simulations and relieve their computational costs.
† Co-corresponding Author.
Obtaining reliable datasets and benchmarks for studying semiconductors in the condensed phase presents significant challenges, which hinders the development of accurate MLFF models. Generating precise datasets for condensed-phase materials is challenging because of the complex structures, dynamics, and large number of atoms involved. To foster the development of MLFFs in the field, we introduce two new datasets, which are specifically designed for semiconductor advanced materials discovery, called SAMD23 datasets: silicon nitride (SiN) and hafnium oxide (HfO). We conducted DFT simulations under various conditions, including initial structures, stoichiometry, temperature, strain, and defects, resulting in a cost of 2.6k GPU days.
Although we built datasets with broad coverage, dynamic simulations exhibit an enormously wide range of atomic configurations with high degrees of freedom, indicating that attempting to collect all these configurations through computationally expensive ab initio simulations is not feasible in practice. Hence, MLFF models must be capable of extrapolation, which enables them to yield reliable predictions of configurations that are absent from the training dataset. Generally, to assess the extrapolation capability, an evaluation of the energy and force on out-of-distribution (OOD) test datasets, in addition to in-distribution (ID) sets, is employed. However, the energy and force errors may not be sufficient to account for the simulation behavior [8, 9]. Thus, we additionally provide five simulation indicators and correspondingly prepare material structures for both ID and OOD sets, facilitating a comprehensive comparison of the MLFF models.
Moreover, we offer a consolidated framework that streamlines model development, training, and evaluation processes into a unified platform. We curated diverse models that utilize hand-crafted features as atomic representations or employ graph neural networks as feature extractors. For the benchmark, 10 MLFF models were trained and evaluated using the framework, suggesting a reliable model selection policy based on the simulation indicators. Based on a comparative analysis of the training models with various hyperparameters, we suggest a baseline training recipe.
This paper makes the following contributions:
- We introduce SAMD23, two new MLFF benchmark datasets that reflect semiconductor simulations of SiN and HfO under various scenarios.
- We provide a framework to facilitate model development, and present benchmarks for SiN and HfO, along with five simulation indicators to assess the prediction performance in simulations and the extrapolation capability.
- We suggest a baseline training recipe and model selection policy to employ the model for simulations by performing a comparative analysis of 10 MLFF models.
Machine Learning Force Fields. Given a snapshot of  $n$  atoms, consisting of atomic numbers  $\mathbf{Z} \in \mathbb{Z}^n$  and atom positions  $\mathbf{r} \in \mathbb{R}^{n \times 3}$ , an MLFF model approximates the potential energy  $E \in \mathbb{R}$  and forces  $\mathbf{F} \in \mathbb{R}^{n \times 3}$  (see Figure B.1). MLFF models can utilize hand-crafted representations of local atomic environments, called descriptors obtained by experts in physics and material science research fields, or automatically represent the local atomic environments by adopting deep neural networks. The former method combines descriptors with different learning algorithms, such as linear regression [10], kernel methods [11, 12], and neural networks [13, 14, 15]. Recently, the latter method, that is, graph neural networks (GNNs) have shown remarkable results in predicting energy and forces by learning the features of atoms and their connectivity in an end-to-end manner instead of using descriptors. While some GNNs [16, 17, 18, 19, 20] rely on rotational invariant features such as interatomic distances and angles, others have rotational equivariant features derived from spherical harmonics [21, 22, 23, 24, 25].
MLFF Benchmark. Numerous MLFF benchmarks, including ANI [26], rMD17 [27], COLL [18], and 3BPA [28], have been publicly released for the application of MLFF to molecular structures. These datasets, which mainly comprise molecular structures with a small number of atoms, have been extensively utilized in the development of various MLFF models. The open catalyst 20 (OC20) [29] and OC22 [30] datasets were generated through simulations of the relaxation of a molecule on bulk surfaces. OC20 primarily investigates the relaxation behavior of diverse adsorbates on various catalyst surfaces. OC22, an enhancement of OC20, focuses oxygen evolution reactions on oxide electrocatalysts. While both OC datasets predominantly focus on surface reactions, they are less
Table 1: Summary of the semiconductor datasets.  $N_{\mathrm{cond}}$  represents the total number of simulation conditions.  $N_{\mathrm{train}}$ ,  $N_{\mathrm{valid}}$ , and  $N_{\mathrm{test}}$  indicate the number of snapshots contained in train, valid, and test sets, respectively. The generation cost is measured by GPU hours using Nvidia V100.
Element	Ncond	Snapshots	Datapoints	Ntrain	Nvalid	Ntest	Cost (h)
SiN	Si m Nn	92	76,213	4,397,744	20,315	2,542	2,585	29,824
Si	14	6,250	291,600	1,663	212	213	2,963
N	4	2,000	128,000	532	68	68	3,769
Total	110	84,463	4,817,344	22,510	2,822	2,866	36,556
SiNood	Si m Nn	3	3,700	388,500	-	-	1,235	1,166
HfO	Hf O2	60	160,000	15,360,000	27,960	3,510	3,510	19,341
HfOOD	Hf m On	12	32,000	3,072,000	-	-	6,996	4,182
optimal for semiconductor studies, which necessitate an understanding of the phenomena inside solids or at the interface between two solids. In 2023, Fu et al. [9] introduced a benchmark for simulation stability using the time evolution of radial distribution functions and bond lengths. Their framework assessed model stability across small molecules to solids in equilibrium MD systems. They highlighted the risk of relying solely on force accuracy for MLFF evaluation, offering new insights for the MLFF community. However, in semiconductor research, where phase transitions, interphase reactions, and defect dynamics are pivotal, systems often deviate from equilibrium. This points out the need for adapted metrics that are more aligned with semiconductor-specific material properties.
In addition, various condensed-phase datasets [31, 32, 33, 34, 35] have been shared in the existing literature and utilized for model evaluation. However, for the general ML community without prior experience in computational materials modeling, evaluating various material properties beyond energy and force using publicly available benchmarks is difficult. Thus, we offer not only an essential dataset for thin-film research, but also material property benchmarks for comprehensive model evaluation, allowing users with no prior experience in the field to evaluate models through actual simulations.
3 SAMD23 Datasets
SAMD23 includes two semiconductor material datasets: SiN and HfO. SiN is increasingly used in various semiconductor applications, particularly as a storage medium in advanced next-generation flash memory devices [36]. HfO is typically used as a high-k material and a crucial ferroelectric material in complementary metal-oxide-semiconductor technology, showing great potential for emerging electronics applications [37]. These two datasets are presented here to investigate the properties and behavior of SiN and HfO thin-film materials and provide a valuable resource for the development and evaluation of MLFF models. For details on dataset usage and generation methods, please refer to Section A.1.
Composition. The composition of the datasets is summarized in Table 1, and the details are listed in Tables A.1 and A.2. All the datasets were released in extended-xyz format (see Section A.5 for details). The SiN dataset encompasses various stoichiometric ratios, with atom counts ranging from 16 to 510. Meanwhile, the HfO dataset comprises 96 atoms, exhibiting a 1:2 ratio of  $32\mathrm{Hf}$  atoms to  $64\mathrm{O}$  atoms. More detailed information on characteristics of the datasets is provided in Section A.2.
3.1 Generation Methodology
To prepare the raw data, ab initio simulations were performed using Vienna ab initio simulation package (VASP) [38, 39, 40] with the Perdew-Burke-Ernzerhof generalized-gradient approximation exchange-correlation functional [41]. Convergence tests were conducted to determine the k-point meshes and cutoff energies for accurate DFT calculations, with a maximum energy convergence of  $2\mathrm{meV}/\mathrm{atom}$  and a maximum force error convergence of  $50\mathrm{meV}/\mathring{\mathrm{A}}$ . The crystal structures of SiN and HfO were sourced from the Materials Project [42] and AFLOW databases [43] (see Section A.3 for details). For both SiN and HfO, we prepared ID and OOD datasets, which are exclusively sampled
from the raw data. The ID datasets were randomly divided into three groups of a ratio of 8:1:1, referred to as train, validation, and test sets, respectively.
Silicon Nitride. The SiN dataset was derived from 110 independent DFT simulations, encompassing diverse structures including amorphous and crystals  $(\alpha, \beta, \gamma)$ , one- or two-dimensional defects, surfaces, and isolated nano-clusters, which are illustrated in Figure A.1. Simulations were performed at various temperatures, strains, and ensembles, with the number of atoms per unit cell ranging from 16 to 510. The SiN simulations encompassed various stoichiometries of SiN as well as Si-only and N-only structures. The ID dataset consists of samples obtained from these simulations with a sampling interval of 9 fs. To generate the OOD dataset, we employed amorphous SiN as the initial structure, which was used to study the synthesizeability of inorganic materials [44], and conducted a melt-quench-relaxation simulation. For additional details, please refer to Table A.2.
Hafnium Oxide. The HfO dataset was created using a modified melt-quench-annealing (m-MQA) method, in which temperature varied, as illustrated in Figure A.6. The MQA method, which inspired our m-MQA method, has been frequently used to create datasets [45, 46, 33]. To obtain high entropy structures, we incorporated three pre-melting stages at an extremely high temperature of  $5000\mathrm{K}$  in the m-MQA method. For full method details, refer to Section A.4. We conducted 12 MD scenarios (yielding 60 simulation conditions) using the m-MQA method on five types of  $\mathrm{HfO_2}$  crystals (i.e., monoclinic, tetragonal, cubic, and two orthorhombic structures) along with seven randomly generated structures. Each structure had 96 atoms (32 Hf and 64 O) to maintain the  $\mathrm{HfO_2}$  stoichiometry. Randomized structures were generated by randomly distributing 32 Hf and 64 O atoms within the unit cells of the  $\mathrm{HfO_2}$  crystals. Of the 12 simulations performed, simulations for 10 structures (five crystals and five random structures) and the remaining two random structures were utilized as the sources of the ID and OOD datasets, respectively. We sampled snapshots from each source with a sampling interval of 9 fs in the three pre-melting and melting stages and an interval of 12 fs in the quenching and annealing stages.
SiN vs. HfO. SiN data generation demonstrates a wider range of atomic configurations, encompassing various experimentally validated phases and defect structures. Conversely, HfO data generation employs the efficient m-MQA method, which is introduced in this report as a user-friendly approach for researchers proficient in DFT. Further, SiN data generation covers a wide range of stoichiometries and numbers of atoms (16-510 atoms), whereas HfO data generation focuses solely on a composition of  $\mathrm{Hf:O} = 1:2$ , involving 96 atoms. Therefore, the model training results with the two datasets exhibited noticeably different aspects (see Section 4.2).
4 Numerical Benchmark: Energy and Force
MLFF models have their own training schemes on a target dataset, involving data preprocessing methods, loss functions, and hyperparameter settings, all of which can significantly influence the training results. To ensure a fair benchmark, we have empirically but carefully selected a common training scheme tailored for our datasets, as discussed in Section 4.1; the details are explained in Section C. Sections 4.2-4.4 provide the benchmark results on SiN and HfO, including an ablation study and discussions on the model inference speed and data scaling effect.
4.1 Benchmark Setting
Error Metric. Table C.1 lists the errors and loss functions used in previous MLFF studies. In this benchmark, we selected the sum of the root mean square errors (RMSEs) of the per-atom energy and force, denoted by the EF metric, as the metric for evaluating energy and force predictions for the following reasons.
For the energy error metric, two types of energy errors for a snapshot can be used: the total energy error and the per-atom energy error computed by dividing the total energy error by the number of atoms in a snapshot. However, the total energy error depends on the number of atoms in a snapshot, whereas the per-atom energy error remains invariant. Since our SiN dataset consists of snapshots of various sizes, we use the per-atom energy error. Subsequently, among the two per-atom energy errors, the mean absolute error (MAE) and RMSE, we employ the RMSE of per-atom energy as the energy error metric. For stable MD simulations, it is crucial to recognize the potential risks stemming from significant errors that can result in catastrophic failure. The RMSE can capture such risks when
assessing the extrapolation capability of MLFF models using OOD setsm rather than the MAE; this is evident in Section 4.2.
In the case of force error metrics, three error types are widely used in ML research: component-wise MAE, RMSE, and MAE with the size of force vectors (L2MAE). However, component-wise MAE has a critical flaw: the absence of rotational invariance (i.e., if a snapshot is rotated, the component-wise absolute error of forces is changed), which is a highly recommended feature for a force field. Among the remaining two candidates, L2MAE and RMSE, which support rotational invariance, we employ the RMSE of force for the same reason as with the energy metric.
Loss Function. A loss function integrating the errors of energy and force is commonly used to train MLFF models. Since we employ the per-atom energy, the total loss function has the form (1),
$$
\mathcal {L} _ {t o t} = \frac {\lambda_ {e}}{| B |} \sum_ {i \in B} \mathcal {L} _ {E} \left(\frac {E _ {i}}{n _ {i}}, \frac {\hat {E} _ {i}}{n _ {i}}\right) + \frac {\lambda_ {f}}{| B |} \sum_ {i \in B} \mathcal {L} _ {\mathbf {F}} \left(\mathbf {F} _ {i}, \hat {\mathbf {F}} _ {i}\right), \tag {1}
$$
where  $B$  indicates a batch of snapshots,  $\mathcal{L}_E$  (or  $\mathcal{L}_{\mathbf{F}}$ ) indicates energy (or force, respectively) loss, and  $\lambda_{e}$  (or  $\lambda_{f}$ ) is a weight of energy (or force, respectively) loss. In this benchmark, aligning the loss function to the EF metrics, we set  $\mathcal{L}_E$  by the MSE of per-atom energy and  $\mathcal{L}_{\mathbf{F}}$  by the MSE of forces; thus, we refer to it as MSE-based loss. Moreover, to secure generalization performance, we also tested setting  $\mathcal{L}_E$  by the MAE of per-atom energy, and  $\mathcal{L}_{\mathbf{F}}$  by the L2MAE of forces, adopted by some MLFF models that showed promising results [19, 47, 25]; the loss is called MAE-based loss in this paper. The exact formulations of the losses are shown in Table C.1. Generally,  $\lambda_{e}$  is set by 1, and  $\lambda_{f}$  are determined in various ways, such as manually setting it as a constant [16, 17, 19, 25], adjusting it based on the snapshot size [22, 23, 24], or gradually modifying it during the training [48]. In this benchmark, we set default  $\lambda_{f}$  by 1, and performed an exploration study.
Graph Generation. To enable models to effectively capture atomic interactions, it is necessary to generate graphs from the snapshot, which represents point cloud data of atom coordinates. The most straightforward approach to convert snapshots to graphs is to build edges between nearby atoms using a cutoff radius; we set the cutoff radius to  $6\mathring{\mathrm{A}}$ . Further, owing to high training cost and insufficient GPU memory for some models, we restricted the maximum number of neighborhood atoms as 50, from the nearest. The effect of the restriction is discussed in Section C.3.1.
We performed benchmarks using 10 models on our datasets: BPNN [13], DPA-1 [49], SchNet [16], DimeNet++ [18], GemNet-T/-dT [19], NequIP [22], Allegro [23], MACE [24], and SCN [25], all of which are described in Section B. The detailed training recipes are presented in Section C.2.
4.2.1 Model Comparison
Overall, as depicted in Figures 1 (a) and (b), the GNN-based models, except for SchNet, are superior to the two descriptor-based models, BPNN and DPA-1. Surprisingly, across all sets, DimeNet++ outperforms recently proposed GNN-based models. Nevertheless, as shown in Section 5.3, DimeNet++ exhibits a significantly low performance for one of the simulation indicators for HfO. This phenomenon is consistent with the results reported in a recent study [9] arguing that accurate force predictions do not fully represent the accuracy and applicability of MLFF models in simulations.
For every model, the EF metrics of the OOD set are larger than those of the test set, aligned with our intention. In both datasets, SchNet exhibits the largest discrepancy between the EF metrics of the train and test sets as well as the highest EF metric of the OOD set, indicating that SchNet has a potential risk of significant overfitting. BPNN, DPA-1, and Allegro also have relatively high EF metrics for the OOD set. Consistent with the results of the OOD set, these four models display unfavorable performance, as measured by the simulation indicators (see Section 5.3).
GNNs incorporating rotational equivariance (e.g., NequIP, Allegro, and MACE) show relatively smaller differences between the EF metrics of the train and test sets compared with the other models, highlighting the advantageous aspects of rotational equivariance. However, in HfO, a substantial discrepancy between the EF metrics of the test and OOD sets is observed for Allegro and MACE. Considering that both models are based on the atomic cluster expansion (ACE) theory [28], searching for factors that contribute to such discrepancies would provide key insights for improving ACE-based GNNs.
(a) SiN
(c) SiN
(b) HfO
(d) HfO
Figure 1: (a, b) EF metrics of the models trained with MSE-based loss. (c, d) Difference between EF metrics of models trained with MAE- and MSE-based losses, where the positive value infers training with MSE-based loss is superior.
(b) HfO
Figure 2: EF metrics obtained by models trained with varying  $\lambda_{f}$  in training loss
4.2.2 Training Factor Exploration Study
MSE- vs. MAE-based Losses. Additionally, we trained the models using the MAE-based loss. Figures 1 (c) and (d) present an EF metric comparison between the models trained using the MAE-based loss and those trained using the MSE-based loss, respectively. The results obtained for SiN clearly indicate the substantial superiority of the MAE-based loss. However, for HfO, the superiority cannot be easily determined; in most cases, the differences of EF metrics are relatively small. Specifically, while Allegro shows a significant performance drop on the HfO OOD set, the accuracy of MACE is improved.
**Coefficient of Force Loss ( $\lambda_f$ ).** For all models, force errors are dominant in the terms of EF metrics. Thus, if we design a training loss to reduce force loss by varying the force coefficient  $\lambda_f$ , the test force errors can be also reduced accordingly. To investigate the effect of  $\lambda_f$ , we trained GemNet-dT and MACE using the MAE-based loss with  $\lambda_f$  ranged from 0.1 to 10; Figure 2 shows the results. Consistent with expectations, the models trained with  $\lambda_f < 1$  show increased errors compared to  $\lambda_f = 1$ . However, the HfO results of MACE with  $\lambda_f = 10$  reveal that increasing  $\lambda_f$  does not always lead to a decrease in the force error and results with  $\lambda_f > 10$  may be worse. Consequently, when exploring a proper  $\lambda_f$  as a hyperparameter tuning, it is recommended to train MLFF models with  $\lambda_f$  in the range of  $1 \leq \lambda_f < 10$ .
4.3 Model Exploration Study
As a goal of MLFF models, it is important to not only achieve the precision of DFT-based simulations, which provide reference data, but also make MD simulations as fast as possible. Thus, Figure 3 illustrates a Pareto plot between the inference time per snapshot and the EF metric of each model, where these two factors show a trade-off relationship. We performed the inference five times on the HfO test set with a batch size of 1, which mimics a step-by-step process in the simulations, using a V100 GPU. To further explore the tradeoff, we trained models whose feature dimensions varied from
Figure 3: Pareto plot for inference time per snapshot vs. EF metric on HfO test set.
(d) NequIP
(e) Allegro