SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Unlike SiN, the HfO datasets consist of 96 atoms with a fixed Hf:O stoichiometry of 1:2. Each MD scenario shows variations in energy and force, as demonstrated in Figure A.3 (a). The high-temperature pre-melting stage at  $5000\mathrm{K}$  shows the largest energy distribution. Depending on the compressive (HfO-premelting-c-5000K) or expansive (HfO-premelting-e-5000K) strain, they exhibit well-separated energy distribution. The average energy also lowers as the temperature decreases through the melting and quenching process. For the annealing process, they show the narrowest energy distribution. The force distributions of these MD scenarios also follow the typical patterns shown in MD simulations. For liquid-like phases that can be observed in pre-melting and melting
Figure A.4: Distribution of the averaged bond order parameter  $\bar{q}_4$  and  $\bar{q}_6$  for SiN.
Figure A.5: Distribution of the averaged bond order parameter  $\bar{q}_4$  and  $\bar{q}_6$  for HfO.
stages (Figure A.3 (b)), we see relatively broader force distributions. As the temperature decreases through the quenching and annealing process, the average force on each atom decreases.
Another essential aspect of the MLFF dataset is the inclusion of diverse atomic environments. Since the local atomic environment determines the atomic energy and forces, having a diverse atomic environment in the dataset is crucial. One way of quantifying the local environment is the bond order parameter (BOA) [51]. We employ the averaged BOA, usually denoted by two-dimension vectors  $\bar{q}_4$  and  $\bar{q}_6$  [52]. This parameter can be a simple but good indicator for quantifying the diversity of the local atomic environment.
Figures A.4 and A.5 show the averaged BOA of SiN and HfO, respectively. SiN shows distinct phase separation in the  $\bar{q}_4 - \bar{q}_6$  plane, with most BOA values falling between  $0 \leq \bar{q}_4 \leq 0.3$  and  $0 \leq \bar{q}_6 \leq 0.3$ . HfO also exhibits similar phase separation, with unique patterns for each MD scenario. For the high-temperature pre-melting stage, the phase boundary is separated by a change in volume. For the melting and quenching stages, their coverage overlaps. However, the energy and force distribution among these MD scenarios are different as shown in Figure A.3. This means that the local atomic environment also differs even though  $\bar{q}_4$  and  $\bar{q}_6$  shows similar distribution. It is important to acknowledge that the BOA does not take into account atomic species and distance information, which can limit its reliability as a comprehensive descriptor. However, this analysis can provide an initial insight into the dataset's quality, serving as a starting point for further investigations.
Figure A.6: A modified Melting-Quenching-Annealing scheme used for generating HfO dataset.
A.3 Utilization of Large-Scale Structural Databases
The Materials Project [42] and AFLOW [43] are comprehensive databases that encompass a wide range of atomic structures, spanning various elements and structural motifs. These structures are predominantly derived from precise quantum mechanical calculations and represent materials in their energetically stable states, where the forces on most atoms approach zero.
For those diving into MD simulations via MLFF models, understanding the nature of these databases is crucial. They primarily present configurations where atoms are at their stable states with minimum-energy. The MD simulations, on the other hand, often explore high-energy conditions and a variety of force environments. Such scenarios are underrepresented in these databases, and thus the databases are less suitable for training MLFF models directly. However, their extensive collection of stable crystal structures can be initial configurations when curating or generating an MLFF training dataset.
A.4 Details of the Modified Melt-Quench-Annealing Method
The generation of the HfO dataset involved the use of a modified melt-quench-annealing (m-MQA) method, where the temperature was scheduled as depicted in Figure A.6. In the m-MQA method, aimed at obtaining high entropy structures, we incorporated three pre-melting stages at an exceedingly high temperature of  $5000\mathrm{K}$ . These stages involved pre-melting at crystal's original volume (premelting 1), followed by  $10\%$  isotropic compression (pre-melting 2) and  $10\%$  expansion (pre-melting 3) relative to the original volume. These stages were taken to ensure the presence of both dense and sparse atomic environments in the resulting dataset. Initial structures underwent three pre-melting stages at  $5000\mathrm{K}$  for 6 ps, melting at  $3500\mathrm{K}$  for 9 ps, quenching from 3500 to  $300\mathrm{K}$  for 16 ps. Finally, the structures were restored to their original volume and annealed at  $600\mathrm{K}$  for 6 ps. All simulations were performed under the NVT ensemble with the Nose-Hoover thermostat.
A.5 Dataset Instruction
Our datasets can be downloaded when following the instructions in the our github url (https://github.com/SAITPublic/MLFF-Framework). There are four datasets of the extended-xyz format: raw and sampled datasets of SiN, and raw and sampled datasets of HfO. The sampled datasets are a portion of the raw datasets, and used in the experiments of this benchmark.
The extended-xyz format begins with a line declaring the total number of atoms. The subsequent line typically encompasses lattice information, energy, and other relevant details associated with the atomic structure for that specific simulation step. From the third line, each line represents an individual atom, where the first column designates the atomic type, followed by its Cartesian coordinates and force components. This structured format is interspersed throughout the file, capturing snapshots from various points in the simulation, providing a comprehensive record that offers insights into the dynamic behavior of atoms at different stages.
While the extended-xyz format can be handled with conventional text-processing utilities, employing specialized computational toolkits considerably streamlines the process. The Atomic Simulation Environment (ASE) [53], renowned for its versatility, effortlessly accommodates a wide range of
(a) [Input] An atomic structure
(b) [Preprocessed] Graph (Optional)
(c) [Output] Energy and Forces
Figure B.1: General flow of MLFF models.
atomistic formats, including the extended-xyz. From intuitive data visualization to comprehensive atomic analysis, its functionality significantly enhances user efficiency. Similarly, the Python Materials Genomics (pymatgen) [54] presents itself as an invaluable tool for researchers focused on atomic structural analysis. It offers deep insights into atomic structural data, enabling seamless structural analysis and transformations. Both ASE and pymatgen serve as commendable aids, optimizing the user experience in managing atomic structural databases.
B Details of Benchmark Models
In this Section, we briefly review the MLFF models used in this benchmark. Figure B.1 describes general flow of MLFF models for a snapshot. Figure B.1a illustrates a structure of input data, which is an 3D point cloud data consists of coordinates  $r_i = (x_i, y_i, z_i) \in \mathbb{R}^3$ , and atom numbers  $Z_i \in \mathbb{Z}$  for  $1 \leq i \leq n$  where  $n$  represents the number of atoms. Figure B.1b describes the preprocessed data with 3D graph form, by constructing edges between points within a cutoff radius. As shown in Figure B.1c, model output consists of total system energy  $E \in \mathbb{R}$ , which is the sum of atomic features, and force  $F_i = (F_i^x, F_i^y, F_i^z) \in \mathbb{R}^3$  for all atoms in the snapshot.
BPNN [13] As a pioneer study for MLFF, BPNN introduced a base idea for the MLFF field: the total energy is represented by a sum of atomic features. For a regression model, a neural network consisting of fully-connected layers is employed. To train the neural network, the coordinates of atoms are converted into hand-crafted features by using symmetry functions to describe local environment of atoms; the hand-crafted features are generally referred to as descriptors.
DPA-1 [49] DPA-1, which was specifically designed to be operated on DeePMD-kit [48], is an end-to-end deep potential energy model that includes trainable descriptors. These descriptors are invariant under translation, rotation, and permutation. It also employs a self-attention mechanism to effectively incorporate angular information for improved performance.
SchNet [16] SchNet is a deep neural network to adaptively learn atomic features representing local atomic environment from atom-centered symmetry functions without relying on pre-defined descriptors. It combines atomic intermediate features using continuous-filter convolutions to mimic atomic interactions.
DimeNet++ [18] DimeNet [17] is an MLFF model based on message passing GNNs, where directional information of atomic environment is embedded into directional messages. It utilizes distance features and dihedral angle features based on radial basis function and spherical basis function, respectively. As an improved version of DimeNet, DimeNet++ introduce an efficient implementation of the directional message passing layer of DimeNet, and its neural architecture is modified from that of DimeNet, enhancing predictive capabilities.
GemNet (GemNet-T and -dT) [19] GemNet is a GNN-based model that incorporates directed edge embeddings and two-hop message passing, allowing the model to capture complex directional
information. Quadruplet, a tuple with four atoms, can be used for these schemes; GemNet using the quadruplet is called GemNet-Q. In GemNet-T, the message and intermediate features are described by using atom pairs and triplets, which is a tuple with three atoms. As an alternative to calculate forces, a direct force prediction method is proposed and applied into GemNet-T architecture, which is GemNet-dT. GemNet achieves universal approximation capabilities for translation-invariant and permutation/rotation-equivariant predictions.
NequIP [22] NequIP utilizes E(3)-equivariant convolutions to capture interactions of geometric tensors, including vectors and higher-order tensors, of atom embeddings. NequIP can learn a representation of atomic environments more comprehensively, compared to models that use invariant convolutions and operate solely on scalars. Hence, it yields prominent accuracy on a diverse range of molecules and materials while requiring significantly fewer training data.
Allegro [23] In GNN-based MLFF models, message passing is considered as a necessary technique to learn many-body interactions. However, the message passing needs information exchange among atoms, leading a large communication overhead on distributed computing system, which is unsuitable for large-scale simulation. To address this issue, Allegro combines equivariant features and tensor products to represent a strictly local equivariant atomic features without relying on message passing, achieving both accuracy and scalability.
MACE [24] MACE is theoretically based on multi-ACE framework [55] that was extended from atomic cluster expansion (ACE) framework by introducing GNNs equipped with high body-order messages. Due to the usage of high body-order messages, MACE can reduce message passing layers to learn the atomic potentials, enabling fast and parallelizable computations and enhancing the training efficiency.
SCN [25] SCN is a GNN-based model where atom features are a set of spherical channels represented by spherical harmonics, enabling it to satisfy rotation equivariance. By relaxing the constraint of rotational equivariance in message passing and aggregation, SCN demonstrated that the performance of energy and force prediction can be improved. SCN is specifically designed for OC20 and requires heavy computation and memory usage. To deal with this issue, we reduced its model size by using 4 interaction (message passing) layers and 64 spherical channels; the small SCN was used for the all of our experiments.
C Details of Numerical Benchmark
C.1 Error Metrics and Losses
Errors of energy and forces are commonly employed as evaluation metrics to assess the predictive performance of MLFF models, primarily due to their ease of use and straightforward interpretability. The errors and loss functions can be computed in several ways, which are listed in Table C.1. Among the metrics, as in discussed in Section 4.1, we chose the RMSE of per-atom energy, which can be considered invariant among various snapshot sizes, and the component-wise RMSE of forces (briefly called the RMSE of forces). In addition, we also express the MSE-based loss and MAE-based loss.
C.2 Training Hyperparameters
In the ML community, where the reproducibility of experiments is highly valued, sharing training recipes that include hyperparameters for training MLFF models is considered a valuable endeavor. In alignment with this attitude, we present the hyperparameters of the models, which we trained for this benchmark study. The configuration files can be also accessed in our open-sourced code. By doing so, we aim to promote transparency and enable others to replicate and build upon our work in a meaningful way.
The all of configurations including neural architecture hyperparameters and optimization hyperparameters are prepared into our codes in Supplementary materials: codes.zip (in config/sim/ and config/train/HfO/). Also, the training factors are listed in Table C.2.
Table C.1: Error metrics and losses used in MLFF research and this benchmark.  $N$  is the number of snapshots,  $n_i$  is the number of atoms in snapshot with index  $i$ . The normal characters such as  $E_i$ ,  $f_x^{i,k}$  indicate reference data (i.e., ground truth). The characters with a hat such as  $\hat{E}_i$ ,  $\hat{f}_x^{i,k}$  indicate predicted values by MLFF models.  $\lambda_f$  is a coefficient of force loss.
Type	Name	Formulation
Energy error metric	MAE of (total) energy	∑i=0N|Ei-ˆi|/N
RMSE of (total) energy	[∑i=0N|Ei-ˆi|2/N]1/2
MAE of per-atom energy	∑i=0N|Ei-ˆi|/N
RMSE of per-atom energy	[∑i=0N|Ei-ˆi|2/N]1/2
Energy loss	MAE of per-atom energy	|Ei-ˆi|/n_i
MSE of per-atom energy	|Ei-ˆi|2
Force error metric	Axis-wise MAE of force	∑i=0N∑k=0ni(|f^i,k - \hat{f}^i,k| + |f^i,k - \hat{f}^i,k| + |f^i,k - \hat{f}^i,k|) / ∑i=0N3ni
Axis-wise RMSE of force	[∑i=0N∑k=0ni(|f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2) / ∑i=0N3ni]1/2
Force loss	Axis-wise MAE of force	∑k=0ni(|f^i,k - \hat{f}^i,k| + |f^i,k - \hat{f}^i,k| + |f^i,k - \hat{f}^i,k|) / 3ni
Axis-wise MSE of force	∑k=0ni(|f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2) / 3ni
L2-MAE of force	∑i=0Ni(|f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2)1/2 / ni
L2-MSE of force	∑i=0Ni(|f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2 + |f^i,k - \hat{f}^i,k|2) / ni
EF metric = RMSE of per-atom energy + Axis-wise RMSE of force
MAE-based loss = MAE of per-atom energy + λf × L2-MAE of force
MSE-based loss = MSE of per-atom energy + λf × Axis-wise MSE of force
Table C.2: Details of training factors for 10 MLFF models.
Model	Optimizer	Initial learning rate	Batch size	etc.
SiN	HfO
BPNN	Adam	5.e-3	16	16	weight decay: 1.e-6
DPA-1	Adam	1.e-3	16	16
SchNet	Adam	1.e-4	16	16
DimeNet++	Adam	1.e-4	3	8
GemNet-T	AdamW	5.e-4	4	8	AMSGrad, ema decay: 0.999, gradient clipping: 10
GemNet-dT	AdamW	5.e-4	4	8	AMSGrad, ema decay: 0.999, gradient clipping: 10
NequIP	Adam	5.e-3	16	16	ema decay: 0.99
Allegro	Adam	5.e-3	16	16	ema decay: 0.99
MACE	Adam	1.e-2	16	16	weight decay: 5.e-7, ema decay: 0.99
SCN	AdamW	4.e-4	6	8	AMSGrad, ema decay: 0.999
As a variant of Adam [56], AMSGrad [57] maintains the maximum of the past squared gradients instead of using exponential moving average, leading to more stable convergence. However, we empirically observed that training losses of MLFF models are often skyrocketed, resulting in relatively slow convergence. Thus, for models that used AMSGrad in the original papers, we tried to train models with or without AMSGrad, and then the better results are reported in this paper. Recently-proposed models tend to employ exponential moving average (EMA), which may also help to obtain stable training results.
While most of the hyperparameters of each model including the neural architecture introduced in the original paper (except SCN) were preserved, we established a common rule of some hyperparameters which can affect the training cost: epochs, batch size, and learning rate schedule. For both HfO and SiN datasets, MLFF models were trained during 200 epochs. The training batch size for each model was selected from the options of 3, 4, 8, 16 to ensure compatibility with the memory capacity of the V100 GPU.
Figure C.1: EF metrics and training times of GemNet-dT and MACE trained with or without restricting the number of neighborhood atoms.
We trained models using a linear decaying schedule (LinearLR) that an learning rate is linearly decayed every step until the training is completed, i.e., 200 epochs. The most existing MLFF models primarily employed one of the two types of learning rate schedules: ReduceLROnPlateau [19, 22, 23, 24] and StepLR [17, 18, 25] implemented in PyTorch. The former stably gives decent training results but makes it difficult to estimate when the training will be completed. The latter allows for predicting the training completion point by specifying the training cost. However, it requires careful consideration of factors such as the decaying points, the decay factor, and the number of decay steps. Therefore, to control the training cost, we opted for a linear decaying schedule with solely one required hyperparameter, an initial learning rate, and thus we can control the training budget for fair benchmark. During training, the learning rate linearly decreases from the initial learning rate to 0 at each iteration.
C.3 Data Preprocessing
C.3.1 Graph Generation
For the fairness comparison among MLFF models, we trained MLFF models with a radius cutoff of 6.0 and a maximum limit of 50 neighborhood atoms. As illustrated in Figure C.1, despite the  $1.5\mathrm{x}$  and  $1.3\mathrm{x}$  increase in training costs for GemNet-dT and MACE, respectively, the test EF metrics of the two models are hardly improved. This implies that the atomic features, derived by message passing in GNNs, may be more strongly influenced by the closer neighborhood atoms. In MACE, utilizing more neighborhood atoms does not lead to improvements on the OOD set, but instead results in higher EF metric. Thus, analyzing the effect of graph generation conditions can be important to obtain models that have lower OOD errors and are more suitable for simulations.
C.3.2 Normalization for Energy and Forces
In this benchmark, when training MLFF models, we adopt a common normalization strategy for energy and forces. Here, we introduce normalization for energy and forces, and show our empirical results that present little difference regardless of a normalization strategy.
Normalizing energy and forces is commonly utilized in MLFF research, either through explicit normalization of energy and forces [29] or by incorporating scale and shift factors within the model to fulfill the normalization role [22, 23, 24]. In this Section, after defining the normalization of energy and forces, we analyze the effect of the normalization to training results.
First, to introduce the normalization technique for energy and forces, we have to be aware that the force of an atom is equal to the derivative of total energy with respect to the atom position in various MLFF models. Calculating forces via taking the derivatives naturally induces two properties: sum of all forces in a single snapshot is equal to  $\mathbf{0}$ , and multiplying a constant to an energy affects exactly same to the forces. Fortunately, the first property implies that the forces does not require centralization; i.e., the forces are not shifted because their mean is  $\mathbf{0}$ . Thus, it suffices to scale the forces by dividing their standard deviation  $\sigma_f$  [22, 23, 24, 29]. Moreover, by the second property, the scaling factor of the energy should be identical to the scaling factor of the forces, i.e.,  $\sigma_f$ .
The only remaining part in defining the normalization is how to centralize the energy. Several methods for the centralization have been proposed. As discussed about error metrics and losses, the energy types to be targeted for obtaining the shift factor are two: total energy and per-atom energy.
(b) HfO
Figure C.2: EF metrics of GemNet-dT and MACE trained with different normalization strategies.
The total energy can be centralized as  $E_{i} - \bar{E}$ , where  $E_{i}$  is the total energy of snapshot  $i$  and  $\bar{E}$  is the mean of total energies [29]. However, for MLFF models that calculate the total energy by predicting and summing up individual atomic energies, the centralization of total energy only works properly for the datasets, which consist of snapshots with the identical size.
As an alternative, some researches [22, 23, 24] centralize per-atom energy instead of the total energy, which is defined as
$$
E _ {i} - \frac {n _ {i}}{N} \sum_ {i} \frac {E _ {i}}{n _ {i}}, \tag {2}
$$
where  $n_i$  is the size of snapshot  $i$  and thus  $\frac{1}{N} \sum_{i} \frac{E_i}{n_i}$  is the mean of per-atom energies. When shifting the total energy, the mean of per-atom energies is compensated by multiplying the snapshot size. Since the SiN dataset consists of various snapshots with different number of atoms, Eq. 2 is employed for the centralization. Finally, the normalization of energy and forces for this benchmark is formulated as
$$
\tilde {E} _ {i} = \left(E _ {i} - \frac {n _ {i}}{N} \sum_ {i} \frac {E _ {i}}{n _ {i}}\right) / \sigma_ {f}, \quad \tilde {\mathbf {F}} _ {i} = \mathbf {F} _ {i} / \sigma_ {f} \tag {3}
$$
MACE [24] can use per-species normalization, which is an extension of the per-atom normalization, by computing per-species energy for each chemical species by solving linear system set by the datasets. However, such method is not appropriate to the datasets which only contain snapshots with uniform stoichiometric ratio, such as the HfO dataset; the linear system becomes singular, and thus the per-species energy is not determined.
To explore the influence of the normalization strategies on the training results, we trained models using three different strategies: per-atom normalization, per-atom centralization, and no normalization. As presented in Figure C.2, the differences in errors obtained from models trained using these three methodologies are marginal, even in OOD errors. The energy prediction method employed by the MLFF models in this benchmark entails obtaining per-atom values within the model, which are then summed up to calculate the total energy of a snapshot. Thus, even not employing normalization, internal representations in MLFF models can inherently aligns with the per-atom values and be used in simulations for structures whose sizes are different from those in the training dataset. This is empirically demonstrated by the small error differences between test and OOD data.
C.4 Supplementary Results
C.4.1 Results of Benchmark for Energy and Force
Table C.3 presents numerical results, which are visualized in Figure 1. As in discussed in Section 4.1, we trained MLFF models with two loss functions. For SiN and HfO datasets, we illustrate the training results of these two loss functions in Figure C.3 (a) and (b), respectively; blue and red bars represent the training results of MSE- and MAE-based loss functions, respectively. Section 4.2 covers the analysis about these results.