SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After obtaining the final material-specific energy embedding matrix  $\mathbf{E}^{L_3,i}$  of a crystalline material  $\mathcal{G}_i$ , the DOS value at each energy level  $\mathcal{E}_j$ , i.e.,  $\hat{\mathbf{Y}}_j^i$ , is given as follows:  $\hat{\mathbf{Y}}_j^i = \phi_{pred}(\mathbf{E}_j^{L_3,i})$ , where  $\phi_{pred}:\mathbb{R}^d\to \mathbb{R}^1$  is a parameterized MLP for predicting DOS. While previous approaches directly predict DOS from the material representation  $\mathbf{g}_i$  using a function  $\phi_{pred}:\mathbb{R}^d\to \mathbb{R}^M$ , DOSTransformer takes a different approach by making pointwise predictions that align with the nature of DOS calculation, i.e., DOS determines the general distribution of states as a function of energy. To the best of our knowledge, DOSTransformer is the first work that predicts DOS values in a pointwise manner at each energy level, and we later show in Section 5 that such an approach further enhances the performance of not only DOSTransformer but also existing models.
4.4 Model Training and Inference
DOSTransformer is trained to minimize the root mean squared error (RMSE) loss  $\mathcal{L}$  between the predicted DOS value  $\hat{\mathbf{Y}}_j^i$  and the ground truth DOS value  $\mathbf{Y}_j^i$ , i.e.,  $\mathcal{L} = \frac{1}{N \cdot M} \sum_{i=1}^{N} \sum_{j=1}^{M} \sqrt{(\hat{\mathbf{Y}}_j^i - \mathbf{Y}_j^i)^2}$ . More specifically, DOSTransformer is trained by combining two distinct RMSE losses with balancing term  $\beta$ , i.e.,  $\mathcal{L}^{total} = \mathcal{L}^{glob} + \beta \cdot \mathcal{L}^{sys}$ , where  $\mathcal{L}^{glob}$  and  $\mathcal{L}^{sys}$  are obtained by utilizing  $\mathbf{E}^{L_3, glob}$  and  $\mathbf{E}^{L_3, sys}$ , respectively. By doing so, DOSTransformer extracts the relationship information that is shared among the distinct crystal systems and within a crystal system, respectively. For inference, we utilize the model prediction obtained based on  $\mathbf{E}^{L_3, sys}$ . The overall model architecture is depicted in Figure 2.
Datasets. We use two datasets to comprehensively evaluate the performance of DOSTransformer, i.e., Phonon DOS and Electron DOS. We use the Phonon DOS dataset following the instructions of the official Github repository  $^{3}$  of a previous work [9]. For Electron DOS dataset, we collect crystalline materials and their electron DOS data from Materials Project (MP) website  $^{4}$ . We provide further detailed preprocessing procedures and statistics on each dataset in Appendix A.
Methods Compared. We mainly compare DOSTransformer to recently proposed state-of-the-art method, i.e., E3NN [9], which utilizes the Euclidean network for encoding material representation. We also compare DOSTransformer to simple baseline methods, i.e., MLP and Graph Network [4], which predicts the entire DOS sequence directly from the learned representation of the materials without regarding the energies during training. Moreover, to evaluate the effectiveness of the transformer layer that considers the relationship between the atoms and various energy levels, we integrate energy embeddings into baseline methods for DOS prediction by concatenating the energy embeddings to the material representation. We provide more details on the implementation of DOSTransformer and compared methods in Appendix C and D, respectively.
<sup>3</sup>https://github.com/zhantaochen/phonondos_e3nn
<sup>4</sup>https://materialsproject.org/
Table 1: Overall model performance under in-distribution scenarios (Bulk M.: Bulk Modulus / Band G.: Band Gap / Ferm. E.: Fermi Energy).
Model	Phonon DOS	Electron DOS	Physical Properties (MSE)
MSE	MAE	R2	MSE	MAE	R2	Bulk M.	Band G.	Ferm. E.
Energy X
MLP	0.309	0.106	0.576	0.347	0.130	0.487	0.695	1.597	4.650
(0.016)	(0.003)	(0.016)	(0.015)	(0.003)	(0.029)	(0.031)	(0.171)	(0.202)
Graph Network	0.259	0.099	0.638	0.264	0.103	0.613	0.688	1.457	3.362
(0.009)	(0.001)	(0.013)	(0.005)	(0.000)	(0.008)	(0.060)	(0.051)	(0.189)
E3NN	0.210	0.077	0.705	0.296	0.109	0.552	0.504	0.896	2.925
(0.004)	(0.001)	(0.007)	(0.005)	(0.001)	(0.013)	(0.033)	(0.093)	(0.111)
Energy √
MLP	0.251	0.099	0.652	0.341	0.128	0.499	0.521	1.409	4.372
(0.004)	(0.001)	(0.006)	(0.011)	(0.002)	(0.012)	(0.011)	(0.182)	(0.067)
Graph Network	0.226	0.092	0.685	0.240	0.099	0.650	0.543	1.263	3.080
(0.007)	(0.002)	(0.011)	(0.005)	(0.001)	(0.005)	(0.085)	(0.107)	(0.100)
E3NN	0.200	0.074	0.724	0.291	0.114	0.564	0.451	1.605	3.777
(0.001)	(0.001)	(0.002)	(0.000)	(0.000)	(0.008)	(0.023)	(0.231)	(0.175)
DOSTransformer	0.191	0.071	0.733	0.221	0.089	0.679	0.427	0.461	2.337
(0.003)	(0.002)	(0.004)	(0.006)	(0.001)	(0.006)	(0.024)	(0.019)	(0.094)
5.2 In-Distribution Evaluation
Evaluation Protocol. In this section, we evaluate the model performances under in-distribution scenarios. While we evaluate DOSTransformer with given data splits in a previous work [9] for Phonon DOS dataset, we randomly split the Electron DOS dataset into train-valid/test of  $80/10/10\%$ . Note that while all measures are reported in the original scale, we report MSE values for DOS prediction multiplied by a factor of 10 for clear interpretation during all experiments.
Experimental Results. In Table 1, we have the following observations: 1) Comparing the baseline methods that overlook the energy levels for DOS prediction (i.e., Energy  $\mathcal{X}$ ) with their counterparts that incorporate both the energy levels and the crystal structure as heterogeneous input modalities through the energy embeddings (i.e., Energy  $\checkmark$ ), we find out that using the energy embeddings consistently enhances the model performance. This indicates that making pointwise predictions on each energy level is crucial for DOS prediction as discussed in Section 4.3, which also aligns with the domain knowledge of materials science, i.e., DOS determines the general distribution of states as a function of energy. 2) However, we observe that E3NN shows a relatively small performance gain compared with other methods after incorporating the energy information. This is because the integration of the energy embedding may confound the model to learn the proper equivariance of the materials. 3) On the other hand, DOSTransformer outperforms previous methods that do not consider the complex relationships between the atoms in materials and various energy levels via the various attention mechanisms. This again implies that a naive integration of the energy information into previous models cannot fully benefit from the energy information. We also provide qualitative analysis on the obtained DOS in Section 5.5.
Physical Validity of DOS. In addition to the accuracy of DOS prediction, we assess the physical validity of the model-predicted DOS by deriving various physical properties, i.e., bulk modulus, band gap, and Fermi energy, of materials, based on the model-predicted DOS. To do so, given the DFT-calculated DOS, which is considered the ground-truth DOS based on which accurate derivation of various physical properties is possible, we train an MLP with a non-linearity in each layer to predict the properties of a crystal structure. Then, based on the obtained MLP weights, we predict the material properties given the model-predicted DOS as the input and calculate the MSE for each material property. We provide further details on the experimental setting in Appendix B. In Table 1, we observe that DOS predicted by DOSTransformer shows the superiority of predicting physical properties of materials, indicating that DOSTransformer not only achieves high accuracy but also produces physically valid predictions.
5.3 Out-of-Distribution Evaluation
In this section, we evaluate the model performances under out-of-distribution (OOD) scenarios. It is well recognized that existing DFT calculation-based databases have limitations in terms of
Table 2: Overall model performance in OOD scenarios.
Model	Out-of-Distribution
# Atom Species	Crystal System
MSE	MAE	R2	MSE	MAE	R2
Energy X
MLP	0.545	0.161	0.250	0.455	0.149	0.437
(0.005)	(0.0001)	(0.003)	(0.006)	(0.002)	(0.007)
Graph Network	0.460	0.144	0.381	0.407	0.132	0.505
(0.015)	(0.003)	(0.028)	(0.004)	(0.001)	(0.006)
E3NN	0.541	0.154	0.231	0.421	0.134	0.483
(0.007)	(0.001)	(0.003)	(0.004)	(0.001)	(0.007)
Energy √
MLP	0.545	0.159	0.260	0.455	0.149	0.445
(0.005)	(0.001)	(0.016)	(0.004)	(0.001)	(0.007)
Graph Network	0.455	0.144	0.388	0.384	0.129	0.534
(0.002)	(0.002)	(0.010)	(0.012)	(0.004)	(0.014)
E3NN	0.534	0.152	0.237	0.420	0.133	0.483
(0.013)	(0.001)	(0.017)	(0.007)	(0.001)	(0.008)
DOSTransformer	0.454	0.136	0.399	0.373	0.122	0.552
(0.008)	(0.001)	(0.017)	(0.006)	(0.001)	(0.008)
Table 3: Fine-tuning on OOD systems.
Model	MSE	MAE	R2
Energy ×
MLP	0.401	0.137	0.510
(0.017)	(0.003)	(0.021)
Graph Network	0.394	0.131	0.519
(0.007)	(0.002)	(0.010)
E3NN	0.414	0.133	0.490
(0.006)	(0.001)	(0.007)
Energy √
MLP	0.394	0.136	0.519
(0.008)	(0.002)	(0.009)
Graph Network	0.382	0.130	0.533
(0.003)	(0.000)	(0.002)
E3NN	0.417	0.133	0.487
(0.004)	(0.001)	(0.007)
DOSTransformer
All	0.365	0.122	0.558
(0.005)	(0.001)	(0.008)
Only Prompt	0.355	0.122	0.570
(0.008)	(0.001)	(0.010)
coverage, as they often focus on specific types of materials or structural archetypes, resulting in biased distributions [13, 30, 20]. Therefore, it is essential to evaluate the model's performance in OOD scenarios to assess its real-world applicability and generalizability beyond the limitations of the available databases [35].
Evaluation Protocol. To do so, we evaluate the model performance on the crystal structures that 1) contain a different number of atom species with the training set (i.e., # Atom species), and 2) belong to different crystal systems that were not included in the training set (i.e., Crystal System). In both scenarios, training data primarily consist of relatively simple structures compared to those in the test data. We provide further details on data split and evaluation in Appendix B.
Experimental Results. In Table 2, we again observe that utilizing energy embeddings consistently brings performance gain to all baseline models. Especially, Graph Network significantly benefits from energy embeddings compared to in-distribution scenarios (See Table 1). We attribute this to the inherent restrictive inductive biases in Graph Network [4], i.e., Graph Network heavily relies on the structural information of materials. Specifically, as the structure of the materials totally varies in the OOD scenarios, which makes the prediction of the whole DOS sequence challenging, incorporating the energy embeddings becomes especially helpful in the OOD scenarios for Graph Network. Moreover, DOSTransformer also alleviates the restrictive inductive bias of Graph Networks by elaborately modeling the complex relationships between energies and materials. It is worth noting that similar observations have been made by comparing Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in the computer vision domain [1, 66]. In conclusion, we argue that incorporating energy information during model training is crucial not only for in-distribution scenarios but also for OOD scenarios, demonstrating the applicability of DOSTransformer in real-world applications. For a more detailed analysis, please refer to Appendix E.1.
Fine-tuning on Few Materials from Complex Crystal Systems. In addition to evaluating the performance under the OOD scenarios, we verify how well the models can make predictions for materials from complex crystal systems with only few training samples, which is a practical situation in reality. For this, we sampled a small subset of materials (i.e.,  $10\%$ ) from the test set used in OOD scenarios of crystal systems, and fine-tune the models that are already trained on the training set. As for DOSTransformer, we fine-tune the model using two different approaches: i) fine-tuning all model parameters (referred to as "All"), and ii) tuning only the parameters of the prompts and the energy decoder while keeping all other model parameters frozen (referred to as "Only Prompt"). In Table 3, we have the following observations: 1) As we expected, the additional fine-tuning step achieves performance gain for all models, while it was marginal due to a limited number of materials used for fine-tuning. 2) Only fine-tuning the prompts of DOSTransformer achieved more performance gain compared to fine-tuning the whole model. This is because while fine-tuning the whole model on a small subset of materials may easily incur overfitting, fine-tuning only prompts enables the model to additionally learn from few new samples while maintaining the knowledge obtained previously. In conclusion, we believe that the proposed prompt tuning approach can have a significant impact in the
field of materials science beyond simple DOS prediction, where the existing databases exhibit a bias towards dominant types of materials [13, 30, 20]. We further explore different proportions beyond the  $10\%$  of the materials used for fine-tuning, as detailed in the Appendix E.3.
5.4 Model Analysis
Ablation Studies. To verify the benefit of each component of DOS-Transformer, we conduct ablation studies under in-distribution scenarios in Figure 3. We observe that utilizing only either one of the global and system losses deteriorates the model performance. This is because jointly optimizing the losses incentivizes the model to extract the relational information that is shared across the crystal systems and within a crystal system. On the other hand, it is worth noting that DOSTransformer with only one of the losses still outperforms baseline models shown in Table 1, indicating that modeling complex relationships between materials and various energies is crucial in DOS prediction.
Sensitivity Analysis. To verify the robustness of DOSTransformer, we measure the model performance by varying  $\beta$ , which is introduced to control the effect of the crystal system loss, i.e.  $\mathcal{L}^{sys}$ , in Section 4.4. In Figure 4, DOSTransformer shows robustness over various levels of  $\beta$ , consistently outperforming baseline models in Table 1. This verifies that DOSTransformer can be easily trained without an expensive hyperparameter tuning, further demonstrating the practicality of DOSTransformer.
Figure 3: Ablation studies.
Figure 4: Sensitivity analysis.
Crystal System Prompts. As discussed in Section 4.2, it is not trivial to effectively incorporate the crystal system information into the model. To evaluate the effectiveness of our proposed prompt-based approach, we explore different approaches for incorporating the crystal system information into the model. We make modifications to determine where and how to inject this information. In terms of the location, we consider two options: injecting the information into the input atoms (i.e., Atom Feat.) and injecting it before the self-attention layers (i.e., Before SA). Regarding the method
of injection, we explore two strategies: one-hot encoding of crystal systems and the use of learnable crystal system prompts. In Table 4, we have following observations: 1) Naively incorporating prompts into atom feature even perform worse than the model without using crystal system (see "w/o  $\mathcal{L}^{sys}$ " in Figure 3), demonstrating the importance of an elaborate design choice for injecting crystal system information. 2) We observe that our approach, i.e. Before SA and prompt, outperforms all other possible choices, indicating that DOSTransformer successfully incorporates crystal system information during training. We also examine the benefits of injecting crystal system information into baseline methods in Appendix E.2.
Table 4: Comparing various crystal system information injection approaches.
Method	Electron DOS
One-hot	Prompt	MSE	MAE
Location	AtomFeat.	✓	0.222(0.004)	0.089(0.001)
✓	0.227(0.005)	0.090(0.001)
BeforeSA	✓	0.226(0.005)	0.090(0.001)
✓	0.221(0.006)	0.089(0.001)
5.5 Qualitative Analysis
In this section, we provide a qualitative analysis of the predicted DOS by mainly comparing it to the DFT-calculated (i.e., Ground Truth) DOS and our main baseline (i.e., E3NN). In Figure 5 (a), which represents the predicted DOS of materials not containing transitional metals, both E3NN and DOSTransformer successfully capture the overall trend of the DOS for several materials (e.g., mp-982366, mp-1009129, and mp-16378). However, DOSTransformer shows a much more precise prediction that closely aligns with the ground truth DOS, providing even more useful information beyond the shape of DOS. For example, peak points represent regions of high density and are likely to be strongly influenced when materials undergo changes in property, and thus represents the probabilistically important energy regions of the materials in the process of material discovery. Notably, our model better captures the peak points in the ground truth DOS compared to E3NN, demonstrating the applicability of DOSTransformer-predicted DOS for material discovery process.
Figure 5: Qualitative Analysis.
On the other hand, Figure 5 (b) shows the DOS prediction for materials containing transition materials. Although DOSTransformer provides more reliable prediction, we observe that the prediction errors of both models get larger compared to the materials that do not contain transition metals shown in Figure 5 (a). This can be attributed to the inherent complexity of physical properties in materials containing transition metals, as discussed in Section 6. Therefore, for our future work, we plan to design expert models in which each expert is responsible for materials with and without transition metals, to achieve more refined and accurate predictions of the DOS. This approach would enable a more comprehensive and elaborate analysis of the DOS in different material compositions.
6 Limitations & Future Work
It is widely known that materials containing transitional metals inherently have complex physical properties compared to other materials. As a result, the density functionals that yield optimal performance in DFT calculations differ between materials with and without transitional metals [12, 46]. However, despite the distinction, most ML-based DOS prediction research, including DOSTransformer, have attempted to encode the properties of both material types into a single model [9, 17], which may provide conflicting signals to the model. Therefore, as a future work, we plan to address this issue by developing specialized approaches that work well on both material types, e.g., utilizing expert models [41] in which an expert is assigned to each material type and later combined to perform well on both material types. We expect this approach to result in the robust and reliable prediction of DOS for both types of materials, enhancing the potential impact in the materials science field and broadening our understanding of material properties.
In this paper, we propose DOSTransformer, which predicts DOS of crystalline materials with various energy levels by following the nature of DOS calculation. Specifically, DOSTransformer associates the heterogeneous information from materials and energy levels with cross-attention and self-attention layers. Moreover, the crystal system prompts are introduced to effectively train the model to learn the relational information that is shared across all the crystal systems and within a crystal system. By doing so, DOSTransformer outperforms previous works in predicting two types of DOS, i.e., phonon DOS and electron DOS, in various scenarios, i.e., in-distribution scenarios and out-of-distribution scenarios. Extensive experiments verify that incorporating energy information is crucial in predicting the DOS of a crystal structure for real-world application and DOSTransformer effectively utilizes the crystal structural information with crystal system prompts.
This work was supported by the core KRICT project from the Korea Research Institute of Chemical Technology (SI2051-10).
Supplementary Material for Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer
A Datasets
In this section, we provide further details on the dataset used for experiments.
A.1 Phonon DOS
We use the Phonon DOS dataset following the instructions of the official Github repository<sup>5</sup> of a previous work [9]. This dataset contains 1,522 crystalline materials whose phonon DOS is calculated from density functional perturbation theory (DFPT) by a previous work [42]. Since the provided dataset does not contain crystal system information, we additionally collect the information based on the Materials Project (MP) website<sup>4</sup> on the given each material's unique ID (MP-id).
A.2 Electron DOS
We also use Electron DOS dataset that contains 38,889 crystalline materials. The Electron DOS dataset consists of the materials and their electron DOS information that is collected from the MP website  $^{4}$ . Among the collected data, we exclude the materials that are tagged to include magnetism because the DOS of magnetism materials is not accurate to be directly used for training machine learning models [23]. We consider an energy grid of 201 points ranging from  $-5$  to  $5\mathrm{eV}$  with respect to the band edges with  $50\mathrm{meV}$  intervals and the Fermi energy is all set to  $0\mathrm{eV}$  on this energy grid. Moreover, we normalize the DOS of each material to be in the range between 0 and 1. That is, the maximum and minimum value for each DOS is 1 and 0, respectively, for all materials. Moreover, we smooth the DOS values with the Savitzky-Golay filter with the window size of 17 and polyorder of 1 using scipy library following a previous work [9].
A.3 Data Statistics of Electron DOS dataset in OOD scenarios
As described in the main manuscript, we further evaluate the model performance in two out-of-distribution scenarios: Scenario 1: regarding the number of atom species, and Scenario 2: regarding the crystal systems. We provide detailed statistics of the number of crystalline materials for each scenario in Table 5 and Table 6.
Table 5: The number of crystals according to the number of atom species (Scenario 1).
Unary (1)	Binary (2)	Ternary (3)	Quaternary (4)	Quinary (5)	Senary (6)	Septenary (7)	Total
# Materials	386	9,034	21,794	5,612	1,750	279	34	38,889
Table 6: The number of crystals according to different crystal systems (Scenario 2).
Cubic	Hexagonal	Tetragonal	Trigonal	Orthorhombic	Monoclinic	Triclinic	Total
# Materials	8,385	3,983	5,772	3,964	8,108	6,576	2,101	38,889
<sup>5</sup>https://github.com/zhantaochen/phonondos_e3nn
B Evaluation Protocol
Phonon DOS. As described in the main manuscript, we evaluate the model performance based on the data splits given in a previous work [9].
Electron DOS. On the other hand, for the Electron DOS dataset, we use different dataset split strategies for each scenario. For the in-distribution setting, we randomly split the dataset into train-valid/test of  $80/10/10\%$ . On the other hand, for the out-of-distribution setting, we split the dataset regarding the structure of the crystals. For both scenarios, we generate training sets with simple crystal structures and a valid/test set with more complex crystal structures, because it is crucial to transfer the knowledge obtained from simple crystal structures to that from complex structures in real-world materials science. More specifically, in the scenario 1 (different number of atom species, i.e., # Atom species in Table 2), we use Binary and Ternary materials as training data and Unary, Quaternary, and Quinary materials as valid and test data. In the case of Unary, we exclude it from training data despite its simplicity due to the observed difficulty of the structure, as will be discussed in Section E.1. In the scenario 2 (different crystal systems, i.e., Crystal System in Table 2), we use Cubic, Hexagonal, Tetragonal, Trigonal, and Orthorhombic crystal systems as training set and Monoclinic and Triclinic as valid and test set. In this scenario, where no prompt is available for unseen crystal systems, we employ the mean-pooled representations of the trained prompts during testing, i.e., for the Monoclinic and Triclinic crystal systems. Please refer to Table 5 and Table 6 for detailed statistics of crystals in each scenario.