SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Physical Properties. In addition to evaluating the accuracy of the model's predictions of the DOS, it is crucial to assess the physical meaningfulness of the predicted DOS for real-world applications. To assess the physical meaningfulness of the predicted DOS, we utilize the predicted DOS to estimate a range of important material properties. Specifically, we evaluate three materials' properties: the bulk modulus for phonon DOS, and the band gap and Fermi energy for electron DOS (Table 1).
Bulk Modulus  $^6$  is a thermodynamic quantity measuring the resistance of a substance to compression. It provides a measure of the material's ability to withstand changes in volume under applied pressure. In the context of elastic properties, the bulk modulus serves as a descriptor, as it indicates how well a material can recover its original volume after being subjected to compression.
Another property we focus on is the Band Gap<sup>7</sup>, which refers to the energy range in a material where no electronic states exist. It represents the energy difference between the top of the valence band and the bottom of the conduction band in insulators and semiconductors. Functional inorganic materials, such as those used in applications like LEDs, transistors, photovoltaics, or scintillators, require a comprehensive understanding of their band gap [69]. By accurately predicting the band gap based on DOS, we can accelerate the development of new materials for a wide range of applications.
Additionally, we predict the Fermi Energy  $^8$ , which represents the highest energy level occupied by electrons at absolute zero temperature (0K). It can be used to determine the electrical and thermal characteristics of materials.
In this section, we provide implementation details of DOSTransformer.
Graph Neural Networks. Our graph neural networks consist of two parts, i.e., encoder and processor. Encoder learns the initial representation of atoms and bonds, while the processor learns to pass the messages across the crystal structure. More formally, given an atom  $v_{i}$  and the bond  $e_{ij}$  between atom  $v_{i}$  and  $v_{j}$ , node encoder  $\phi_{node}$  and edge encoder  $\phi_{edge}$  outputs initial representations of atom  $v_{i}$  and bond  $e_{ij}$  as follows:
$$
\mathbf {h} _ {i} ^ {0} = \phi_ {\text {n o d e}} (\mathbf {X} _ {i}), \quad \mathbf {b} _ {i j} ^ {0} = \phi_ {\text {e d g e}} (\mathbf {B} _ {i j}), \tag {4}
$$
where  $\mathbf{X}$  is the atom feature matrix whose  $i$ -th row indicates the input feature of atom  $v_{i}$ ,  $\mathbf{B} \in \mathbb{R}^{n \times n \times F_e}$  is the bond feature tensor with  $F_{e}$  features for each bond. With the initial representations
$^{6}$ https://en.wikipedia.org/wiki/Bulk_modulus
<sup>7</sup>https://en.wikipedia.org/wiki/Band_gap
<sup>8</sup>https://en.wikipedia.org/wiki/Fermi_energy
of atoms and bonds, the processor learns to pass messages across the crystal structure and update atoms and bonds representations as follows:
$$
\mathbf {b} _ {i j} ^ {l + 1} = \psi_ {e d g e} ^ {l} \left(\mathbf {h} _ {i} ^ {l}, \mathbf {h} _ {j} ^ {l}, \mathbf {b} _ {i j} ^ {l}\right), \quad \mathbf {h} _ {i} ^ {l + 1} = \psi_ {n o d e} ^ {l} \left(\mathbf {h} _ {i} ^ {l}, \sum_ {j \in \mathcal {N} (i)} \mathbf {b} _ {i j} ^ {l + 1}\right), \tag {5}
$$
where  $\mathcal{N}(i)$  is the neighboring atoms of atom  $v_{i}$ ,  $\psi$  is a two-layer MLP with non-linearity, and  $l = 0,\ldots ,L^{\prime}$ . Note that  $\mathbf{h}_i^{L'}$  is equivalent to the  $i$ -th row of the atom embedding matrix  $\mathbf{H}$  in Equation 1.
Model Training. In all our experiments, we use the AdamW optimizer for model optimization. For all the tasks, we train the model for 1,000 epochs with early stopping applied if the best validation loss does not change for 50 consecutive epochs.
Hyperparameter Tuning. Detailed hyperparameter specifications are given in Table 7. For the hyperparameters in DOSTransformer, we tune them in certain ranges as follows: number of message passing layers in GNN  $L'$  in  $\{2,3,4\}$ , number of cross-attention layers  $L_1$ ,  $L_3$  in  $\{2,3,4\}$ , number of self-attention layers  $L_2$  in  $\{2,3,4\}$ , hidden dimension  $d$  in  $\{64,128,256\}$ , learning rate  $\eta$  in  $\{0.0001, 0.0005, 0.001\}$ , and batch size  $B$  in  $\{1,4,8\}$ . We use the sum pooling to obtain the crystalline material  $i$ 's representation, i.e.,  $\mathbf{g}_i$ . We report the test performance when the performance on the validation set gives the best result.
Table 7: Hyperparameter specifications of DOSTransformer.
Hyperparameters	In-Distribution	Out-of-Distribution
Phonon DOS	Electron DOS	# Atom Species	Crystal Systems
# Message Passing Layers (L')	3	3	3	3
# Cross-Attention Layers (L1)	2	2	2	2
# Self-Attention Layers (L2)	2	2	2	2
# Cross-Attention Layers (L3)	2	2	2	2
Hidden Dim. (d)	256	256	256	256
Learning Rate (η)	0.0001	0.0001	0.0001	0.0001
Batch Size (B)	1	8	8	8
D Methods Compared
In this section, we provide further details on the methods that are compared with DOSTransformer in our experiments.
MLP. We first encode the atoms in a crystalline material with an MLP. Then, we obtain the representation of material  $i$ , i.e.,  $\mathbf{g}_i$ , by sum pooling the representations of its constituent atoms. With the material representation, we predict DOS with an MLP predictor  $\phi'$ , i.e.,  $\hat{\mathbf{Y}}^i = \phi'(\mathbf{g}_i)$ , where  $\phi': \mathbb{R}^d \to \mathbb{R}^{201}$ .
On the other hand, when we incorporate energy embeddings into the MLP, we predict DOS for each energy  $j$  with a learnable energy embedding  $\mathbf{E}_j^0$  and obtained material representation  $\mathbf{g}_i$ , i.e.,  $\hat{\mathbf{Y}}_j^i = \phi (\mathbf{E}_j^0 ||\mathbf{g}_i)$ , where  $\phi :\mathbb{R}^{2d}\to \mathbb{R}^1$  is a parameterized MLP.
Graph Network. We first encode the atoms in a crystalline material with a graph network [4]. As done for MLP, we obtain the representation of material  $i$ , i.e.,  $\mathbf{g}_i$ , by sum pooling the representations of its constituent atoms. With the material representation, we predict the DOS with an MLP predictor,
i.e.,  $\hat{\mathbf{Y}}^i = \phi'(\mathbf{g}_i)$ , where  $\phi': \mathbb{R}^d \to \mathbb{R}^{201}$ . Note that the only difference with MLP is that the atom representations are obtained through the message passing scheme. We also compare the vanilla graph network that incorporates the energy information as we have done in MLP.
E3NN. For E3NN [9], we use the official code published by the authors<sup>9</sup>, which implements equivariant neural networks with E3NN python library<sup>10</sup>. By learning the equivariance, the model can generate high-quality representations with a small number of training materials. After obtaining the crystalline material representation  $\mathbf{g}_i$ , all other procedures have been done in the same manner with other baseline models, i.e., MLP and Graph Network.
E.1 Model Performance Analysis on Out-of-Distribution Scenarios
In this section, we conduct a comprehensive analysis of the model's predictions in the out-of-distribution scenarios presented in Table 2. In Table 8, we evaluate the performance of the model for each type of material, providing detailed insights into its predictive capabilities. We have following observations: 1) We observe that DOSTransformer consistently outperforms in both out-of-distribution scenarios, which demonstrates the superiority of DOSTransformer. 2) The performance of all the compared models generally degrades as the crystal structure gets more complex. That is, models perform worse in Quinary crystals than in Quaternary crystals, and worse in Triclinic crystals than in Monoclinic crystals. 3) On the other hand, it is not the case in Unary crystal. This is because in Unary crystal only one type of atom repeatedly appears in the crystal structure, which cannot give enough information to the model. However, DOSTransformer also makes comparably accurate predictions in the Unary materials by modeling the complex relationship between the atoms and various energy levels.
Table 8: Model performance in Out-of-Distribution scenarios.
Model	# Atom Species	Crystal System
Unary	Quartenary	Quinary	Monoclinic	Triclinic
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Energy ×
MLP	0.578(0.005)	0.180(0.001)	0.500(0.003)	0.153(0.001)	0.673(0.004)	0.182(0.002)	0.370(0.017)	0.132(0.003)	0.470(0.021)	0.146(0.003)
Graph Network	0.485(0.013)	0.165(0.003)	0.443(0.003)	0.141(0.001)	0.620(0.007)	0.170(0.003)	0.376(0.003)	0.127(0.001)	0.504(0.007)	0.147(0.001)
E3NN	0.565(0.025)	0.167(0.002)	0.486(0.001)	0.145(0.000)	0.708(0.013)	0.179(0.002)	0.393(0.003)	0.129(0.001)	0.510(0.006)	0.148(0.002)
Energy √
MLP	0.597(0.034)	0.183(0.005)	0.498(0.003)	0.151(0.002)	0.684(0.015)	0.178(0.002)	0.367(0.012)	0.130(0.002)	0.468(0.020)	0.144(0.003)
Graph Network	0.471(0.028)	0.160(0.005)	0.416(0.002)	0.137(0.002)	0.571(0.005)	0.165(0.002)	0.359(0.010)	0.125(0.004)	0.476(0.008)	0.144(0.004)
E3NN	0.567(0.021)	0.166(0.003)	0.481(0.008)	0.143(0.000)	0.689(0.020)	0.175(0.002)	0.393(0.006)	0.128(0.001)	0.516(0.006)	0.147(0.001)
DOSTransformer	0.417(0.012)	0.145(0.003)	0.413(0.010)	0.128(0.002)	0.570(0.010)	0.157(0.001)	0.343(0.003)	0.117(0.001)	0.466(0.007)	0.137(0.001)
E.2 Injecting Crystal System Information to Baseline Methods
In this section, we adopt our prompt-based crystal system information injection procedure to the baseline methods. We examine two approaches for injecting the information: 1) injecting the information into the input atoms (i.e., Position 1), and 2) injecting it before the DOS prediction layer (i.e., Position 2). In Table 9, we have the following observations: 1) Compared to Table 1, all baseline models benefit from using crystal system information. This demonstrates the importance of
$^{9}$ https://github.com/ninarina12/phononDoS_tutorial
10 https://docs.e3nn.org/en/latest/index.html
utilizing crystal structural systems information, which has been overlooked in previous works. 2) However, DOSTransformer still outperforms all baseline methods with crystal system information (See DOSTransformer in Table 1), verifying the importance of an elaborate design of crystal system injection procedure. To be more specific, we notice a relatively significant performance gap between DOSTransformer and the best baseline model in Electron DOS, which comprises a broader range of crystalline materials than Phonon DOS. This finding highlights the importance of an intricate crystal system injection procedure when striving to learn the DOS of diverse crystalline materials.
Table 9: Baseline model performance with crystal structural system prompts.
Model	Phonon DOS	Electron DOS
Position 1	Position 2	Position 1	Position 2
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Energy ×
MLP	0.308(0.013)	0.105(0.001)	0.324(0.012)	0.109(0.002)	0.343(0.008)	0.128(0.001)	0.340(0.009)	0.127(0.0031)
Graph Network	0.260(0.008)	0.097(0.000)	0.244(0.023)	0.095(0.001)	0.260(0.011)	0.102(0.001)	0.260(0.006)	0.103(0.002)
E3NN	0.210(0.007)	0.077(0.002)	0.209(0.010)	0.079(0.001)	0.292(0.007)	0.108(0.001)	0.299(0.002)	0.110(0.001)
Energy √
MLP	0.251(0.000)	0.100(0.000)	0.245(0.004)	0.098(0.001)	0.332(0.003)	0.125(0.001)	0.336(0.007)	0.127(0.002)
Graph Network	0.230(0.009)	0.094(0.002)	0.224(0.009)	0.091(0.002)	0.234(0.001)	0.097(0.004)	0.230(0.009)	0.093(0.002)
E3NN	0.194(0.004)	0.073(0.000)	0.190(0.000)	0.073(0.001)	0.286(0.002)	0.108(0.000)	0.290(0.003)	0.109(0.000)
E.3 Various Training Data Ratio for Fine-Tuning
In this section, we additionally provide experimental results on various ratios of training data for fine-tuning in Table 3. That is, instead of sampling  $10\%$  of training data from the test set used in OOD scenarios in Section 5.3, we try various sampling ratios, i.e.,  $5\%$ ,  $10\%$ ,  $15\%$ , and  $20\%$ , from the test set. We have the following observations: 1) We notice a significant performance disparity between the "Only Prompt" and "All" approaches, particularly when the training dataset is limited. This phenomenon can be attributed to the challenge of overfitting when fine-tuning the entire model on a small subset of materials, as discussed in Section 5.3. 2) Conversely, as the training data increases, we observe that the difference in performance between the "Only Prompt" and "All" methods diminishes. This is due to the enough amount of training data allowing for the adjustment of
all model parameters, consistent with the analysis discussed in Section 5.3. However, it is important to note that the existing DFT calculation-based databases suffer from a highly biased distribution, which limits their coverage of different materials. This limitation emphasizes the significance of achieving good performance even with a small subset of training data. Therefore, we argue that the application of prompt tuning enhances the real-world applicability of DOSTransformer.
Figure 6: Various training data ratios for fine-tuning.
E.4 Comparison with Crystal Neural Network
We conducted additional experiments on the state-of-the-art crystal neural networks, i.e., ALIGNNN [10] and Matformer [63], by adjusting the output dimensions of these models (51 for Phonon DOS and 201 for Electron DOS) in Table 10. Although both ALIGNNN and Matformer exhibited commendable performance, we observe that DOSTransformer consistently surpassed them significantly. These observations again demonstrate the importance of considering energy levels for accurate DOS prediction.
Table 10: Comparison to crystal neural networks.
Model	Phonon DOS	Electron DOS
MSE	MAE	R2	MSE	MAE	R2
ALIGNN	0.204	0.085	0.717	0.270	0.108	0.600
(0.005)	(0.001)	(0.007)	(0.004)	(0.001)	(0.001)
Matformer	0.198	0.084	0.724	0.268	0.104	0.601
(0.008)	(0.002)	(0.013)	(0.004)	(0.001)	(0.002)
DOSTransformer	0.191	0.071	0.733	0.221	0.089	0.679
(0.003)	(0.002)	(0.004)	(0.006)	(0.001)	(0.006)
E.5 Model Training and Inference Time
In this section, to verify the efficiency of DOSTransformer, we compare the training and inference time of the baseline methods in Table 11. We observe that DOSTransformer requires a longer training time per epoch on the Phonon DOS dataset compared to E3NN, which can be attributed to the two forward passes (i.e., system and global energy embeddings) during the training procedure. However, when it comes to the Electron DOS dataset, DOSTransformer demonstrates a shorter training time per epoch compared to E3NN. This is because the Electron DOS dataset has complex crystal structures, requiring more time for E3NN to learn equivariant representations. Furthermore, in terms of inference time, DOSTransformer demonstrates significantly faster computation per epoch compared to E3NN, particularly on the Electron DOS dataset. This is because we only utilize system prediction without global prediction during inference. As many predictive ML models are used for high-throughput screening in material discovery, inference time is a critical factor for ML models in materials science, demonstrating the practicality of DOSTransformer in real-world applications.
Table 11: Training and inference time per epoch for each dataset (sec/epoch).
Model	Training	Inference
Phonon DOS	Electron DOS	Phonon DOS	Electron DOS
Energy ×
MLP	4.10	23.52	1.51	3.00
Graph Network	16.17	59.74	1.95	3.88
E3NN	21.21	141.02	3.72	9.49
Energy √
MLP	4.67	27.88	1.66	3.10
Graph Network	17.45	66.83	2.16	4.28
E3NN	24.12	152.80	3.92	10.21
DOSTransformer	39.17	145.85	2.98	5.99
F Broader Impacts
Potential Positive Scientific Impacts. In this work, we propose DOSTransformer, which is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural system, demonstrating its applicability in real-world scenarios. For example, transferring the knowledge obtained from simple structured materials to complex structured materials is crucial because DFT calculation-based databases cover limited types of materials or structural archetypes. Therefore, we believe DOSTransformer has broad impacts on various fields of materials science.
Potential Negative Societal Impacts. This work explores the automation process for materials science without wet lab experiments. However, it is important to acknowledge that in the industry, there are skilled professionals dedicated to conducting such experiments for materials science.
Therefore, it is important to proactively address these concerns by encouraging collaboration between automated methods and human experts.
G Pseudo Code
Algorithm 1 shows the pseudocode of DOSTransformer.
Algorithm 1: Pseudocode of DOSTransformer.
Input: An input crystalline material  $\mathcal{G} = (\mathbf{X},\mathbf{A})$  , Ground truth DOS Y, Number of attention layers  $L_{1},L_{2},L_{3}$  , Initialized energy embeddings E, Initialized crystal system prompts P.   
1 H  $\leftarrow$  GNN(X,A)   
2  $\mathbf{E}^{L_1}\gets$  Cross-Attention(H,E,L1)   
3 g  $\leftarrow$  Sum Pooling(H)   
4  $\mathbf{E}^{glob}\gets (\mathbf{E}^{L_1}\| \mathbf{g})$    
5  $\tilde{\mathbf{E}}^{0,glob}\gets \phi_1(\mathbf{E}^{glob})$    
6  $\tilde{\mathbf{E}}^{L_2,glob}\gets$  Self-Attention(  $\tilde{\mathbf{E}}^{0,glob},L_2)$  // Global Self-Attention   
7  $\mathbf{E}^{L_3,glob}\gets$  Cross-Attention(H,  $\tilde{\mathbf{E}}^{L_2,glob},L_3)$    
8  $\hat{\mathbf{Y}}\gets \phi_{pred}(\mathbf{E}^{L_3,glob})$    
9  $\mathcal{L}^{glob}\gets$  RMSE(  $\hat{\mathbf{Y}},\mathbf{Y})$    
10  $\mathbf{E}^{sys}\gets (\mathbf{E}^{L_1}\| \mathbf{g}\| \mathbf{P})$    
11  $\tilde{\mathbf{E}}^{0,sys}\gets \phi_2(\mathbf{E}^{sys})$    
12  $\tilde{\mathbf{E}}^{L_2,sys}\gets$  Self-Attention(  $\tilde{\mathbf{E}}^{0,sys},L_2)$  // System Self-Attention   
13  $\mathbf{E}^{L_3,sys}\gets$  Cross-Attention(H,  $\tilde{\mathbf{E}}^{L_2,sys},L_3)$    
14  $\hat{\mathbf{Y}}\gets \phi_{pred}(\mathbf{E}^{L_3,sys})$    
15  $\mathcal{L}^{sys}\gets$  RMSE(  $\hat{\mathbf{Y}},\mathbf{Y})$    
16  $\mathcal{L}^{\text {total }}\gets \mathcal{L}^{\text {glob }} + \beta \cdot \mathcal{L}^{\text {sys }}//$  Calculate total loss   
17 Function Cross-Attention (H,  $\mathbf{E}^0,L$  ..   
18 for  $l = 1,2,\dots ,L$  do   
19 |  $\mathbf{E}^l\gets$  Softmax(  $\frac{\mathbf{E}^{l - 1}\mathbf{H}^{\top}}{\mathbf{H}}$    
20 end   
21 return  $\mathbf{E}^L$    
22 Function Self-Attention(  $\tilde{\mathbf{E}}^0,L$  ..   
23 for  $p = 1,2,\dots ,L$  do   
24 |  $\tilde{\mathbf{E}}^p\gets$  Softmax(  $\frac{\tilde{\mathbf{E}}^{p - 1}\tilde{\mathbf{E}}^{p - 1}\tilde{\mathbf{T}}}{\tilde{\mathbf{E}}^{p - 1}}$    
25 end   
26 return  $\tilde{\mathbf{E}}^L$