Title: Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer
Venue: NeurIPS 2023
Accepted: yes
Keywords: ML4Materials

AI4Science

Graph Neural Networks
Abstract: The density of states (DOS) is a spectral property of crystalline materials, which provides fundamental insights into various characteristics of the materials.
While previous works mainly focus on obtaining high-quality representations of crystalline materials for DOS prediction, we focus on predicting the DOS from the obtained representations by reflecting the nature of DOS: DOS determines the general distribution of states as a function of energy.
That is, DOS is not solely determined by the crystalline material but also by the energy levels, which has been neglected in previous works.
In this paper, we propose to integrate heterogeneous information obtained from the crystalline materials and the energies via a multi-modal transformer, thereby modeling the complex relationships between the atoms in the crystalline materials and various energy levels for DOS prediction.
Moreover, we propose to utilize prompts to guide the model to learn the crystal structural system-specific interactions between crystalline materials and energies.
Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superiority of DOSTransformer.
The source code for DOSTransformer is available at https://github.com/HeewoongNoh/DOSTransformer.

Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer

Abstract

The density of states (DOS) is a spectral property of crystalline materials, which provides fundamental insights into various characteristics of the materials. While previous works mainly focus on obtaining high-quality representations of crystalline materials for DOS prediction, we focus on predicting the DOS from the obtained representations by reflecting the nature of DOS: DOS determines the general distribution of states as a function of energy. That is, DOS is not solely determined by the crystalline material but also by the energy levels, which has been neglected in previous works. In this paper, we propose to integrate heterogeneous information obtained from the crystalline materials and the energies via a multi-modal transformer, thereby modeling the complex relationships between the atoms in the crystalline materials and various energy levels for DOS prediction. Moreover, we propose to utilize prompts to guide the model to learn the crystal structural system-specific interactions between crystalline materials and energies. Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superiority of DOSTransformer . The source code for DOSTransformer is available at https://github.com/HeewoongNoh/DOSTransformer .

1 Introduction

Along with recent advances in machine learning (ML) for scientific discovery, ML models have rapidly been adopted in computational materials science thanks to the abundant experimental and computational data in the field. Most ML models developed in computational materials science to date have been focused on crystalline materials property consisting of single-valued quantities [29], e.g., band gap energy [31, 69], formation energy [3, 56], and Fermi energy [61].

On the other hand, spectral properties are ubiquitous in materials science, characterizing various properties of crystalline materials, e.g., X-ray absorption, dielectric function, and electronic density of states [29] (Figure 1).

The density of states (DOS), which is the main focus of this paper, is a spectral property that provides fundamental insights on various characteristics of crystalline materials, even enabling direct computation of single-valued properties [17]. For example, DOS is utilized as a feature of materials for analyzing the underlying reasons for changes

𝐒𝐛𝐏𝐎

𝟒

DOS

Energy (eV)

Single-valued Property

3.34 eV

-2.299 eV/atom

0.057 eV/atom

in electrical conductivity [15]. Moreover, band gaps and edge positions, which can be directly derived from DOS, are utilized to discover new photoanodes for solar fuel generation [47, 64]. As a result, the exploration of ML capabilities for predicting DOS takes us one step further toward the core principles of materials science, thereby expediting the process of materials discovery. However, credible computation of DOS with traditional density functional theory (DFT) requires expensive time/financial costs of exhaustively conducting experiments with expertise knowledge [7, 14]. Therefore, alternative approaches for DOS calculation are necessary, whereas demonstrating ML capabilities for learning such spectral properties of the crystalline materials is relatively under-explored.

In this paper, we focus on predicting DOS by following the nature of DOS calculation: DOS determines the general distribution of states as a function of energy . That is, while single-valued properties can be described solely by the crystalline material, DOS is determined by not only the material itself but also the energy levels at which DOS is calculated. Therefore, integrating the heterogeneous signals from both the crystalline material and the various energy levels is crucial for DOS prediction. However, previous works for DOS prediction focus on obtaining a qualified representation of material [7, 14, 17, 9], thereby overlooking the unique characteristics of DOS.

On the other hand, learning from multiple input types, i.e., crystalline material and various energy levels, is not trivial due to their heterogeneity. To this end, we formulate the DOS prediction problem into a multi-modal learning problem, which has recently gained significant attention from ML researchers in various domains thanks to its capability of extracting and relating information from heterogeneous data types [6, 38, 54, 5, 2]. For example, VisualBERT proposes to align elements of input modalities, i.e., images and text, thereby achieving SOTA performance in various visionand-language tasks [36]. Moreover, Dall-E shows its power in generating images based on text data by encoding images and texts together [43]. Among various multi-modal learners, the multi-modal transformer demonstrates its intrinsic advantages and scalability in modeling multiple modalities [62], by associating heterogeneous modalities with the cross-attention mechanism [65, 50].

Inspired by the recent success of the multi-modal transformer, we propose a multi-modal transformer model for DOS prediction, named DOSTransformer , which incorporates crystalline material and energies as heterogeneous input modalities. Unlike previous works that overlook the energy levels for predicting DOS [7, 14, 17, 9], DOSTransformer learns embeddings of energy levels which are used for modeling complex relationships between the energy levels and crystalline material through a cross-attention mechanism. By doing so, DOSTransformer obtains multiple representations for a single crystalline material according to various energy levels, enabling the prediction of a single DOS value on each energy level. However, simply combining a crystalline material with energy levels may fail to consider the significant impact of the crystal's structure, which sometimes results in distinct material properties even when it is composed of identical elements (e.g., carbon). For example, although graphite and diamond are made entirely out of carbon, their arrangement (or structure) of carbon atoms differs significantly, resulting in distinct material properties. Therefore, we utilize learnable prompts to inform DOSTransformer of the input crystal system (among the seven crystal systems 2 ), which navigates the model to capture structure-specific interactions between a crystalline material and energy levels. By doing so, DOSTransformer learns to extract not only the relational information that is shared with a crystal system, but also across the crystal systems. In this work, we make the following contributions:

Inspired by the fact that DOS is determined by not only the materials themselves but also the energy levels at which DOS is calculated, we propose a multi-modal transformer model for DOS prediction, called DOSTransformer , which incorporates crystalline materials and energy as heterogeneous input modalities.

To capture structure-specific interactions between a crystalline material and energy levels, thereby being able to predict DOS that significantly differs according to the crystal's structure, we utilize learnable prompts to inform DOSTransformer of the input crystal system.

Our extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, demonstrate that DOSTransformer outperforms a wide range of state-of-the-art methods in various real-world scenarios, i.e., in-distribution scenarios and out-of-distribution scenarios.

To the best of our knowledge, this is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural systems.

2 Related Works

2.1 Machine Learning for Materials and Density of States

Traditional materials science research heavily relies on theory, experimentation, and computer simulations, but these approaches are time-consuming, costly, and inefficient, hindering their ability to keep up with the field's development [57]. As an alternative research tool, ML, which requires neither expensive trial and error experiments nor domain expert knowledge, has been attracting a surge of interest from materials scientists [67]. Consequently, various ML methods have been established to learn the relationship between materials and their properties which are usually obtained from ab-initio calculations [11]. Inspired by the recent success of GNNs in biochemistry [19, 48, 25, 33, 34, 32], CGCNN proposes a message-passing framework for crystal structure providing highaccuracy prediction for 8 different materials properties [61]. However, most studies have focused on single-valued properties [29], whereas various spectral properties, such as DOS, are also crucial.

The DOS describes the number of different states at a particular energy level that particles are allowed to occupy, determining various properties of crystalline materials. For example, phonon DOS has a crucial influence in determining the specific heat and vibrational entropy of crystalline materials as well as the interfacial thermal resistance [60], while electron DOS is used to derive the electronic contribution to heat capacity in metals [21] and the effective mass of electrons in charge carriers [18].

Despite the importance of DOS, how to incorporate ML capabilities for decoding such spectral properties is under-explored since the vast majority of methods concentrate only on encoding crystalline materials. Specifically, Chandrasekaran et al. [7] proposes to predict electron DOS based on the hand-crafted fingerprint of each grid point in the crystalline materials, while Del Rio et al. [14] leverages that of each atom in the crystalline materials. Chen et al. [9] predicts phonon DOS by learning the representations of the crystalline materials that are equivariant to 3D rotations, translations, and inversion with Euclidean neural networks [49, 28, 58], achieving high-quality prediction with a small number of training materials with over 64 atom types. In this work, we concentrate on decoding spectral properties regarding the complex relationship between the various energy levels and crystalline materials.

2.2 Multi-modal Transformer, Prompt Tuning, and Positional Encoding

Multi-modal Transformer. In recent years, much progress in multi-modal learning has grown rapidly by extracting and aligning the rich information from heterogeneous modalities [2, 6]. Among the various multi-modal learning methods, the multi-modal transformer demonstrates its superior capability in learning multiple modalities [62], by associating the heterogeneous modalities with a cross-attention mechanism. Specifically, MulT [50] learns the representations directly from unaligned multi-modal streams with multi-modal transformer, while Yao & Wan [65] improves machine translation quality by learning the image-aware text representations. In this paper, we investigate how to decode DOS of a crystalline material by learning the energy-aware crystal representations via a multi-modal transformer.

Prompt Tuning. On the other hand, prompt designing has gained significant attention from NLP researchers as an efficient method for fine-tuning large language models (LLMs) for various tasks [39]. There are two main approaches to prompt designing: discrete prompt designing [44] and continuous prompt tuning [37]. While discrete prompt designing involves manually creating task description tokens for LLMs through trial and error, continuous prompt tuning trains learnable prompts in a continuous latent space without requiring expert knowledge. Inspired by the success of continuous prompt tuning in NLP domain, it has been widely adopted in various domains such as computer vision [24] and vision-language [68]. While previous works utilize prompts for efficient fine-tuning of the models, we employ prompts to capture the heterogeneity of various crystal structures.

Positional Encoding. In the original proposal of the Transformer architecture, a sinusoidal function was suggested as a method for positional encoding. However, the sinusoidal function may have limitations in terms of learnability and flexibility, which can affect its effectiveness [53]. To address this issue, most pre-trained language models [16, 40] employ learnable vector embeddings as positional representations. By conceptualizing each energy value as the position of a word in a sentence, various energy embeddings in Section 4.2 can be analogous to the positional encoding used in traditional transformers.

3 Preliminaries

Notations. Let G = ( V , A ) denote a crystalline material, where V = { v 1 , . . . , v n } represents the set of atoms, and A ⊆ V × V represents the set of edges connecting the atoms in the crystalline material G . Moreover, G is associated with a feature matrix X ∈ R n × F and an adjacency matrix A ∈ R n × n where A ij = 1 if and only if ( v i , v j ) ∈ A and A ij = 0 otherwise.

Task: Density of States Prediction. Given a set of crystalline materials D G = {G 1 , G 2 , . . . , G N } and a set of energies D E = {E 1 , E 2 , . . . , E M } , our goal is to train a model M that predicts the DOS of a crystalline material given a set of energies, i.e., Y i = M ( G i , D E ) , where Y i ∈ R M is an M dimensional vector containing the DOS values of a crystalline material G i at each energy E 1 , . . . , E M , and Y i j ∈ R is the DOS value of G i at energy level E j .

4 Methodology: DOSTransformer

In this section, we introduce our proposed method, named DOSTransformer , a novel DOS prediction framework that learns the complex relationship between the atoms in the crystalline material and various energy levels by utilizing a cross-attention mechanism of the multi-modal transformer.

4.1 Crystalline Material Encoder

Before modeling the pairwise interaction between the crystalline material and the energies, we first encode the crystalline material with GNNs to learn the representation of each atom, which contains not only the feature information but also the structural information. Formally, given a crystalline material G = ( X , A ) , we generate an atom embedding matrix for the crystalline material as follows:

where H ∈ R n × d is an atom embedding matrix for G , whose i -th row indicates the representation of atom v i , and we stack L ′ layers of GNNs. Among various GNNs, we adopt graph networks [4] as our crystalline material encoder, which is a generalized and extended version of various GNNs. We provide further details on the GNNs in Appendix C.

4.2 Prompt-based Multi-modal Transformer

After obtaining the atom embedding matrix H , we aim to capture the relationship between a crystalline material and various energy levels by utilizing the cross-attention layers of a multi-modal transformer [62] with self-attention layers [52]. In a nutshell, we expect the cross-attention layers to integrate the heterogeneous signals from a crystalline material and various energy levels, while self-attention layers aim to integrate the information of material-specific energy representations.

Cross-Attention Layers. Specifically, we expect the cross-attention layers to generate the materialspecific representation of the energies by repeatedly reinforcing the energy representation with the crystalline materials. To do so, we first introduce a learnable energy embedding matrix E 0 ∈ R M × d , whose j -th row, i.e., E 0 j , indicates the embedding of energy E j ∈ D E . Then, we present cross-modal attention for fusing the information from the crystal structure into various energy levels as follows:

where l = 1 , . . . , L 1 indicates the index of the cross-attention layers. In contrast to the conventional Transformer, which introduces learnable weight matrices for query Q , key K , and value V , we directly employ the previously obtained energy embedding matrix E l -1 as the query matrix, and the atom embedding matrix H as the key and value matrices. Based on the above cross-attention mechanism, we obtain the material-specific energy embedding E l ∈ R M × d by aggregating the information regarding the atoms in the crystalline material that were important at each energy level. The final material-specific energy embedding matrix E L 1 ∈ R M × d generated by the cross-attention layer reflects the relationship between the atoms in the crystalline material and various energy levels.

Query

Key

! 𝐄

!"#

! 𝐄

!"#

∈ ℝ

$×&

! 𝐄

!"#

Softmax(

! 𝐄

!"#

! 𝐄

!"# '

𝑑

) ∈ ℝ

$×(

Encoder

Energy

Embedding

Model architecture

Energy embedding

Atom embedding

𝐄

!"#

∈ ℝ

$×&

Query

𝐇 ∈ ℝ

'×&

𝐇

Softmax(

𝐄

!"#

𝐇

(

𝑑

) ∈ ℝ

$×'

𝐄

!

∈ ℝ

$×&

Key

Value

Cross-Attention Layer

Prompts

Multi-modal Transformer

Cross-A>en?on Layer

Self-Attention Layer

Cross-Attention Layer

𝐿

#

×

𝐿

)

×

𝐿

*

×

Energy Decoder

Atom

Embedding

Material

Embedding

𝐄

!"#$

ℒ

+!,-

ℒ

./.

𝐄

%&%

Self-Attention Layer

Query

Key

Value

! 𝐄

!"#

! 𝐄

!

∈ ℝ

$×&

! 𝐄

!"#

∈ ℝ

$×&

! 𝐄

!"#

! 𝐄

!"#

Softmax(

! 𝐄

!"#

! 𝐄

!"# '

𝑑

) ∈ ℝ

$×(

Sys

Glob

Monoclinic

𝐒𝐛𝐏𝐎

𝟒

Global Self-Attention

System Self-Attention

𝐄

𝑳

𝟏

! 𝐄

𝑳

𝟐

,,-,

! 𝐄

𝑳

𝟐

,./01

𝐄

𝑳

𝟑

,,-,

𝐄

𝑳

𝟑

,./01

Global Self-Attention. In addition to cross-attention layers, we propose to enhance the materialspecific energy embedding matrix E L 1 by aggregating the information from other energy levels. To do so, we employ variants of self-attention layers in conventional Transformer [52] as follows:

where p = 1 , . . . , L 2 indicates the index of the self-attention layer. We use an enhanced materialspecific energy embedding ˜ E 0 as an input of the self-attention, which is obtained by concatenating the material-specific energy embedding E L 1 with the material representation g i ∈ R d , i.e., ˜ E 0 j = ϕ 1 ( E glob j ) , where E glob j = ( E L 1 j || g i ) , ϕ 1 : R 2 d → R d , and || indicates the concatenation operation. Note that g i is a sum pooled representation of the material G i , and E L 1 j indicates the j -th row of the energy embedding matrix E L 1 computed by Equation 2 where j = 0 , . . . , M . Although the output of cross-attention layers, i.e., E L 1 ∈ R M × d , captures the local atom-level information regarding the material, we also recognize the importance of incorporating the global material-level information g i that may not have been fully encoded during the cross-attention phase. Note that the importance of considering the global contextual information for enhancing the learning of dependencies among neural representations has been widely studied in various domains such as computer vision and natural language processing [51, 55, 22]. By additionally considering the global material-level information, we aim to enhance the material-specific representation of the energies more comprehensively.

System Self-Attention with Crystal System Prompts. However, solely aggregating the information of material-specific energy embeddings may neglect the crucial influence of the structural properties in crystalline materials, whereas the structural properties play a significant role in determining the unique characteristics and properties of these materials [45]. To illustrate this point, let's examine the case of graphite and diamond, both composed entirely of carbon atoms. In graphite, carbon atoms are arranged in stacked layers, where each carbon atom is bonded to three neighboring carbon atoms in a hexagonal lattice. In contrast, the carbon atom in diamond forms strong covalent bonds with four neighboring carbon atoms, resulting in a tetrahedral arrangement. Even though graphite and diamond are made entirely out of carbon, due to the structural difference, graphite exhibits properties such as electrical conductivity and lubricity due to its layered structure [59, 26], while diamond is renowned for its hardness and thermal conductivity owing to its tightly bonded tetrahedral network [8, 27].

Despite the importance of structural information, effectively incorporating it into the model is not trivial. Naively concatenating the structural information as an input feature of materials is straightforward but may lead to interference, hindering the model's ability to learn and generalize knowledge across different crystal structures. To this end, we propose to provide additional information to the selfattention layers with learnable prompts, which indicate the structural information of the crystalline materials. Specifically, we adopt learnable prompts P ∈ R 7 × d p , whose k -th row P k represents one of the seven widely known crystal systems: Cubic, Hexagonal, Tetragonal, Trigonal, Orthorhombic, Monoclinic, and Triclinic. Then, given a crystalline material G i , whose crystal system is given as k , we incorporate the learnable prompts into the material-specific energy embedding ˜ E 0 as an input

of the self-attention, which is obtained as follows: ˜ E 0 j = ϕ 2 ( E sys j ) , where E sys j = ( E L 1 j || g i || P k ) and ϕ 2 : R 2 d + d p → R d . By doing so, we obtain high-level energy features that contain the crystal structural system information (i.e., ˜ E L 2 ), while keeping the low-level features to share the knowledge across all the materials (i.e., E L 1 ). In Section 5.4, we demonstrate the effectiveness of our prompt-based approach compared to other variants.

Besides the self-attention layers, we use additional cross-attention layers on top of the self-attention layers, whose input energy embedding E 0 in Equation 2 is given as ˜ E L 2 , and output the final materialspecific energy embeddings E L 3 , where L 3 is the number of additional cross-attention layers. By doing so, the model extracts the enhanced relationships between the crystalline materials and energy levels regarding the crystal structural system information.

4.3 Energy Decoder

After obtaining the final material-specific energy embedding matrix E L 3 ,i of a crystalline material G i , the DOS value at each energy level E j , i,e., ˆ Y i j , is given as follows: ˆ Y i j = ϕ pred ( E L 3 ,i j ) , where ϕ pred : R d → R 1 is a parameterized MLP for predicting DOS. While previous approaches directly predict DOS from the material representation g i using a function ϕ pred : R d → R M , DOSTransformer takes a different approach by making pointwise predictions that align with the nature of DOS calculation, i.e., DOS determines the general distribution of states as a function of energy To the best of our knowledge, DOSTransformer is the first work that predicts DOS values in a pointwise manner at each energy level, and we later show in Section 5 that such an approach further enhances the performance of not only DOSTransformer but also existing models.

4.4 Model Training and Inference

DOSTransformer is trained to minimize the root mean squared error (RMSE) loss L between the predicted DOS value ˆ Y i j and the ground truth DOS value Y i j , i.e., L = 1 N · M ∑ N i =1 ∑ M j =1 √ ( ˆ Y i j -Y i j ) 2 . More specifically, DOSTransformer is trained by combining two distinct RMSE losses with balancing term β , i.e., L total = L glob + β · L sys , where L glob and L sys are obtained by utilizing E L 3 ,glob and E L 3 ,sys , respectively. By doing so, DOSTransformer extracts the relationship information that is shared among the distinct crystal systems and within a crystal system, respectively. For inference, we utilize the model prediction obtained based on E L 3 ,sys . The overall model architecture is depicted in Figure 2.

5 Experiments

5.1 Experimental Setup

Datasets. We use two datasets to comprehensively evaluate the performance of DOSTransformer , i.e., Phonon DOS and Electron DOS. We use the Phonon DOS dataset following the instructions of the official Github repository 3 of a previous work [9]. For Electron DOS dataset, we collect crystalline materials and their electron DOS data from Materials Project (MP) website 4 . We provide further detailed preprocessing procedures and statistics on each dataset in Appendix A.

Methods Compared. We mainly compare DOSTransformer to recently proposed state-of-the-art method, i.e., E3NN [9], which utilizes the Euclidean network for encoding material representation. We also compare DOSTransformer to simple baseline methods, i.e., MLP and Graph Network [4], which predicts the entire DOS sequence directly from the learned representation of the materials without regarding the energies during training. Moreover, to evaluate the effectiveness of the transformer layer that considers the relationship between the atoms and various energy levels, we integrate energy embeddings into baseline methods for DOS prediction by concatenating the energy embeddings to the material representation. We provide more details on the implementation of DOSTransformer and compared methods in Appendix C and D, respectively.

5.2 In-Distribution Evaluation

Evaluation Protocol. In this section, we evaluate the model performances under in-distribution scenarios. While we evaluate DOSTransformer with given data splits in a previous work [9] for Phonon DOS dataset, we randomly split the Electron DOS dataset into train/valid/test of 80/10/10%. Note that while all measures are reported in the original scale, we report MSE values for DOS prediction multiplied by a factor of 10 for clear interpretation during all experiments.

Experimental Results. In Table 1, we have the following observations: 1) Comparing the baseline methods that overlook the energy levels for DOS prediction (i.e., Energy ✗ ) with their counterparts that incorporate both the energy levels and the crystal structure as heterogeneous input modalities through the energy embeddings (i.e., Energy ✓ ), we find out that using the energy embeddings consistently enhances the model performance. This indicates that making pointwise predictions on each energy level is crucial for DOS prediction as discussed in Section 4.3, which also aligns with the domain knowledge of materials science, i.e., DOS determines the general distribution of states as a function of energy. 2) However, we observe that E3NN shows a relatively small performance gain compared with other methods after incorporating the energy information. This is because the integration of the energy embedding may confound the model to learn the proper equivariance of the materials. 3) On the other hand, DOSTransformer outperforms previous methods that do not consider the complex relationships between the atoms in materials and various energy levels via the various attention mechanisms. This again implies that a naive integration of the energy information into previous models cannot fully benefit from the energy information. We also provide qualitative analysis on the obtained DOS in Section 5.5.

Physical Validity of DOS. In addition to the accuracy of DOS prediction, we assess the physical validity of the model-predicted DOS by deriving various physical properties, i.e., bulk modulus, band gap, and Fermi energy, of materials, based on the model-predicted DOS. To do so, given the DFT-calculated DOS, which is considered the ground-truth DOS based on which accurate derivation of various physical properties is possible, we train an MLP with a non-linearity in each layer to predict the properties of a crystal structure. Then, based on the obtained MLP weights, we predict the material properties given the model-predicted DOS as the input and calculate the MSE for each material property. We provide further details on the experimental setting in Appendix B. In Table 1, we observe that DOS predicted by DOSTransformer shows the superiority of predicting physical properties of materials, indicating that DOSTransformer not only achieves high accuracy but also produces physically valid predictions.

5.3 Out-of-Distribution Evaluation

In this section, we evaluate the model performances under out-of-distribution (OOD) scenarios. It is well recognized that existing DFT calculation-based databases have limitations in terms of

coverage, as they often focus on specific types of materials or structural archetypes, resulting in biased distributions [13, 30, 20]. Therefore, it is essential to evaluate the model's performance in OOD scenarios to assess its real-world applicability and generalizability beyond the limitations of the available databases [35].

Evaluation Protocol. To do so, we evaluate the model performance on the crystal structures that 1) contain a different number of atom species with the training set (i.e., # Atom species), and 2) belong to different crystal systems that were not included in the training set (i.e., Crystal System). In both scenarios, training data primarily consist of relatively simple structures compared to those in the test data. We provide further details on data split and evaluation in Appendix B.

Experimental Results. In Table 2, we again observe that utilizing energy embeddings consistently brings performance gain to all baseline models. Especially, Graph Network significantly benefits from energy embeddings compared to in-distribution scenarios (See Table 1). We attribute this to the inherent restrictive inductive biases in Graph Network [4], i.e., Graph Network heavily relies on the structural information of materials. Specifically, as the structure of the materials totally varies in the OOD scenarios, which makes the prediction of the whole DOS sequence challenging, incorporating the energy embeddings becomes especially helpful in the OOD scenarios for Graph Network. Moreover, DOSTransformer also alleviates the restrictive inductive bias of Graph Networks by elaborately modeling the complex relationships between energies and materials. It is worth noting that similar observations have been made by comparing Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in the computer vision domain [1, 66]. In conclusion, we argue that incorporating energy information during model training is crucial not only for in-distribution scenarios but also for OOD scenarios, demonstrating the applicability of DOSTransformer in real-world applications. For a more detailed analysis, please refer to Appendix E.1.

Fine-tuning on Few Materials from Complex Crystal Systems. In addition to evaluating the performance under the OOD scenarios, we verify how well the models can make predictions for materials from complex crystal systems with only few training samples, which is a practical situation in reality. For this, we sampled a small subset of of materials (i.e., 10%) from the test set used in OOD scenarios of crystal systems, and fine-tune the models that are already trained on the training set. As for DOSTransformer , we fine-tune the model using two different approaches: i) fine-tuning all model parameters (referred to as "All"), and ii) tuning only the parameters of the prompts and the energy decoder while keeping all other model parameters frozen (referred to as "Only Prompt"). In Table 3, we have the following observations: 1) As we expected, the additional fine-tuning step achieves performance gain for all models, while it was marginal due to a limited number of materials used for fine-tuning. 2) Only fine-tuning the prompts of DOSTransformer achieved more performance gain compared to fine-tuning the whole model. This is because while fine-tuning the whole model on a small subset of materials may easily incur overfitting, fine-tuning only prompts enables the model to additionally learn from few new samples while maintaining the knowledge obtained previously. In conclusion, we believe that the proposed prompt tuning approach can have a significant impact in the

field of materials science beyond simple DOS prediction, where the existing databases exhibit a bias towards dominant types of materials [13, 30, 20]. We further explore different proportions beyond the 10% of the materials used for fine-tuning, as detailed in the Appendix E.3.

5.4 Model Analysis

Ablation Studies. To verify the benefit of each component of DOSTransformer , we conduct ablation studies under in-distribution scenarios in Figure 3. We observe that utilizing only either one of the global and system losses deteriorates the model performance. This is because jointly optimizing the losses incentivizes the model to extract the relational information that is shared across the crystal systems and within a crystal system. On the other hand, it is worth noting that DOSTransformer with only one of the losses still outperforms baseline models shown in Table 1, indicating that modeling complex relationships between materials and various energies is crucial in DOS prediction.

Sensitivity Analysis. To verify the robustness of DOSTransformer , we measure the model performance by varying β , which is introduced to control the effect of the crystal system loss, i.e. L sys , in Section 4.4. In Figure 4, DOSTransformer shows robustness over various levels of β , consistently outperforming baseline models in Table 1. This verifies that DOSTransformer can be easily trained without an expensive hyperparameter tuning, further demonstrating the practicality of DOSTransformer .

Crystal System Prompts. As discussed in Section 4.2, it is not trivial to effectively incorporate the crystal system information into the model. To evaluate the effectiveness of our proposed prompt-based approach, we explore different approaches for incorporating the crystal system information into the model. We make modifications to determine where and how to inject this information. In terms of the location, we consider two options: injecting the information into the input atoms (i.e., Atom Feat.) and injecting it before the selfattention layers (i.e., Before SA). Regarding the method

DOSTran

-sformer

w/o

ℒ

w/o

ℒ

𝒈𝒍𝒐𝒃

MSE

1.0

1.2

1.5

𝜷

0.8

0.5

MSE

of injection, we explore two strategies: one-hot encoding of crystal systems and the use of learnable crystal system prompts. In Table 4, we have following observations: 1) Naively incorporating prompts into atom feature even perform worse than the model without using crystal system (see " w/o L sys " in Figure 3), demonstrating the importance of an elaborate design choice for injecting crystal system information. 2) We observe that our approach, i.e. Before SA and prompt, outperforms all other possible choices, indicating that DOSTransformer successfully incorporates crystal system information during training. We also examine the benefits of injecting crystal system information into baseline methods in Appendix E.2.

5.5 Qualitative Analysis

In this section, we provide a qualitative analysis of the predicted DOS by mainly comparing it to the DFT-calculated (i.e., Ground Truth) DOS and our main baseline (i.e., E3NN). In Figure 5 (a), which represents the predicted DOS of materials not containing transitional metals, both E3NN and DOSTransformer successfully capture the overall trend of the DOS for several materials (e.g., mp-982366, mp-1009129, and mp-16378). However, DOSTransformer shows a much more precise prediction that closely aligns with the ground truth DOS, providing even more useful information beyond the shape of DOS. For example, peak points represent regions of high density and are likely to be strongly influenced when materials undergo changes in property, and thus represents the probabilistically important energy regions of the materials in the process of material discovery. Notably, our model better captures the peak points in the ground truth DOS compared to E3NN, demonstrating the applicability of DOSTransformer -predicted DOS for material discovery process.

eDOS

Energy (eV)

(a) w/o

Trans. metal

(b) w/

Trans. metal

eDOS

Ground Truth

E3NN

DOSTransformer

[mp-982366] Tb2S20

[mp-10931] HoSF

[mp-1009129] MgO

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

[mp-768368]K4120

[mp-16378] HoS2

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

[mp-708035] NaZn(H3O2)3

[mp-1080023] MoOs(Pb03)2

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

[mp-17048]Rb2WS4

[mp-288]PtS

[mp-1018851]OsN2

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

5

5

5

0

5

0

5

On the other hand, Figure 5 (b) shows the DOS prediction for materials containing transition materials. Although DOSTransformer provides more reliable prediction, we observe that the prediction errors of both models get larger compared to the materials that do not contain transition metals shown in Figure 5 (a). This can be attributed to the inherent complexity of physical properties in materials containing transition metals, as discussed in Section 6. Therefore, for our future work, we plan to design expert models in which each expert is responsible for materials with and without transition metals, to achieve more refined and accurate predictions of the DOS. This approach would enable a more comprehensive and elaborate analysis of the DOS in different material compositions.

6 Limitations & Future Work

It is widely known that materials containing transitional metals inherently have complex physical properties compared to other materials. As a result, the density functionals that yield optimal performance in DFT calculations differ between materials with and without transitional metals [12, 46]. However, despite the distinction, most ML-based DOS prediction research, including DOSTransformer , have attempted to encode the properties of both material types into a single model [9, 17], which may provide conflicting signals to the model. Therefore, as a future work, we plan to address this issue by developing specialized approaches that work well on both material types, e.g., utilizing expert models [41] in which an expert is assigned to each material type and later combined to perform well on both material types. We expect this approach to result in the robust and reliable prediction of DOS for both types of materials, enhancing the potential impact in the materials science field and broadening our understanding of material properties.

7 Conclusion

In this paper, we propose DOSTransformer , which predicts DOS of crystalline materials with various energy levels by following the nature of DOS calculation. Specifically, DOSTransformer associates the heterogeneous information from materials and energy levels with cross-attention and self-attention layers. Moreover, the crystal system prompts are introduced to effectively train the model to learn the relational information that is shared across all the crystal systems and within a crystal system. By doing so, DOSTransformer outperforms previous works in predicting two types of DOS, i.e., phonon DOS and electron DOS, in various scenarios, i.e., in-distribution scenarios and out-of-distribution scenarios. Extensive experiments verify that incorporating energy information is crucial in predicting the DOS of a crystal structure for real-world application and DOSTransformer effectively utilizes the crystal structural information with crystal system prompts.

References

Bai, Y., Mei, J., Yuille, A. L., and Xie, C. Are transformers more robust than cnns? Advances in Neural Information Processing Systems , 34:26831-26843, 2021.

Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence , 41(2):423-443, 2018.

Banjade, H. R., Hauri, S., Zhang, S., Ricci, F., Gong, W., Hautier, G., Vucetic, S., and Yan, Q. Structure motif-centric learning framework for inorganic crystalline systems. Science advances , 7(17):eabf1754, 2021.

Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Bayoudh, K., Hamdaoui, F., and Mtibaa, A. Hybrid-covid: a novel hybrid 2d/3d cnn based on cross-domain adaptation approach for covid-19 screening from chest x-ray images. Physical and engineering sciences in medicine , 43(4):1415-1431, 2020.

Bayoudh, K., Knani, R., Hamdaoui, F., and Mtibaa, A. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer , 38(8): 2939-2970, 2022.

Chandrasekaran, A., Kamal, D., Batra, R., Kim, C., Chen, L., and Ramprasad, R. Solving the electronic structure problem with machine learning. npj Computational Materials , 5(1):1-7, 2019.

Che, J., Ça˘ gın, T., Deng, W., and Goddard III, W. A. Thermal conductivity of diamond and related materials from molecular dynamics simulations. The Journal of Chemical Physics , 113 (16):6888-6900, 2000.

Chen, Z., Andrejevic, N., Smidt, T., Ding, Z., Xu, Q., Chi, Y.-T., Nguyen, Q. T., Alatas, A., Kong, J., and Li, M. Direct prediction of phonon density of states with euclidean neural networks. Advanced Science , 8(12):2004214, 2021.

Choudhary, K. and DeCost, B. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials , 7(1):185, 2021.

Choudhary, K., DeCost, B., Chen, C., Jain, A., Tavazza, F., Cohn, R., Park, C. W., Choudhary, A., Agrawal, A., Billinge, S. J., et al. Recent advances and applications of deep learning methods in materials science. npj Computational Materials , 8(1):1-26, 2022.

Cramer, C. J. and Truhlar, D. G. Density functional theory for transition metals and transition metal chemistry. Physical Chemistry Chemical Physics , 11(46):10757-10816, 2009.

De Breuck, P.-P., Evans, M. L., and Rignanese, G.-M. Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on modnet. Journal of Physics: Condensed Matter , 33(40):404002, 2021.

Del Rio, B. G., Kuenneth, C., Tran, H. D., and Ramprasad, R. An efficient deep learning scheme to predict the electronic structure of materials and molecules: The example of graphene-derived allotropes. The Journal of Physical Chemistry A , 124(45):9496-9502, 2020.

Deringer, V. L., Bernstein, N., Csányi, G., Ben Mahmoud, C., Ceriotti, M., Wilson, M., Drabold, D. A., and Elliott, S. R. Origins of structural and electronic transitions in disordered silicon. Nature , 589(7840):59-64, 2021.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Fung, V., Ganesh, P., and Sumpter, B. G. Physically informed machine learning prediction of electronic density of states. Chemistry of Materials , 2022.

Garrity, K. F. First-principles search for n-type oxide, nitride, and sulfide thermoelectrics. Physical Review B , 94(4):045122, 2016.

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chemistry. In International conference on machine learning , pp. 1263-1272. PMLR, 2017.

Griffiths, R.-R., Schwaller, P., and Lee, A. A. Dataset bias in the natural sciences: a case study in chemical reaction prediction and synthesis design. arXiv preprint arXiv:2105.02637 , 2021.

Hamaoui, G., Horny, N., Hua, Z., Zhu, T., Robillard, J.-F., Fleming, A., Ban, H., and Chirtoc, M. Electronic contribution in heat transfer at metal-semiconductor and metal silicide-semiconductor interfaces. Scientific reports , 8(1):1-9, 2018.

Hatamizadeh, A., Yin, H., Kautz, J., and Molchanov, P. Global context vision transformers. arXiv preprint arXiv:2206.09959 , 2022.

Illas, F., de PR Moreira, I., Bofill, J., and Filatov, M. Extent and limitations of density-functional theory in describing magnetic systems. Physical Review B , 70(13):132414, 2004.

Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII , pp. 709-727. Springer, 2022.

Jiang, D., Wu, Z., Hsieh, C.-Y., Chen, G., Liao, B., Wang, Z., Shen, C., Cao, D., Wu, J., and Hou, T. Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models. Journal of cheminformatics , 13(1):1-23, 2021.

Jorio, A., Dresselhaus, G., and Dresselhaus, M. S. Carbon nanotubes: advanced topics in the synthesis, structure, properties and applications , volume 111. Springer, 2008.

Kidalov, S. V. and Shakhov, F. M. Thermal conductivity of diamond composites. Materials , 2 (4):2467-2495, 2009.

Kondor, R., Lin, Z., and Trivedi, S. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems , 31, 2018.

Kong, S., Ricci, F., Guevarra, D., Neaton, J. B., Gomes, C. P., and Gregoire, J. M. Density of states prediction for materials discovery via contrastive learning from probabilistic embeddings. Nature communications , 13(1):1-12, 2022.

Kumagai, M., Ando, Y., Tanaka, A., Tsuda, K., Katsura, Y., and Kurosaki, K. Effects of data bias on machine-learning-based material discovery using experimental property data. Science and Technology of Advanced Materials: Methods , 2(1):302-309, 2022.

Lee, J., Seko, A., Shitara, K., Nakayama, K., and Tanaka, I. Prediction model of band gap for inorganic compounds by combination of density functional theory calculations and machine learning techniques. Physical Review B , 93(11):115104, 2016.

Lee, J., Kim, S., Hyun, D., Lee, N., Kim, Y., and Park, C. Deep single-cell rna-seq data clustering with graph prototypical contrastive learning. Bioinformatics , 39(6):btad342, 2023.

Lee, N., Hyun, D., Na, G. S., Kim, S., Lee, J., and Park, C. Conditional graph information bottleneck for molecular relational learning. arXiv preprint arXiv:2305.01520 , 2023.

Lee, N., Yoon, K., Na, G. S., Kim, S., and Park, C. Shift-robust molecular relational learning with causal substructure. arXiv preprint arXiv:2305.18451 , 2023.

Li, K., DeCost, B., Choudhary, K., Greenwood, M., and Hattrick-Simpers, J. A critical examination of robustness and generalizability of machine learning prediction of materials properties. npj Computational Materials , 9(1):55, 2023.

Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Lin, T.-Y., Cui, Y., Belongie, S., and Hays, J. Learning deep representations for ground-toaerial geolocalization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5007-5015, 2015.

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Masoudnia, S. and Ebrahimpour, R. Mixture of experts: a literature survey. The Artificial Intelligence Review , 42(2):275, 2014.

Petretto, G., Dwaraknath, S., PC Miranda, H., Winston, D., Giantomassi, M., Van Setten, M. J., Gonze, X., Persson, K. A., Hautier, G., and Rignanese, G.-M. High-throughput densityfunctional perturbation theory phonons for inorganic materials. Scientific data , 5(1):1-12, 2018.

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International Conference on Machine Learning , pp. 8821-8831. PMLR, 2021.

Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 , 2020.

Shandiz, M. A. and Gauvin, R. Application of machine learning methods for the prediction of crystal system of cathode materials in lithium-ion batteries. Computational Materials Science , 117:270-278, 2016.

Sim, E., Song, S., and Burke, K. Quantifying density errors in dft. The journal of physical chemistry letters , 9(22):6385-6392, 2018.

Singh, A. K., Montoya, J. H., Gregoire, J. M., and Persson, K. A. Robust and synthesizable photocatalysts for co2 reduction: a data-driven materials discovery. Nature communications , 10 (1):1-9, 2019.

Stokes, J. M., Yang, K., Swanson, K., Jin, W., Cubillos-Ruiz, A., Donghia, N. M., MacNair, C. R., French, S., Carfrae, L. A., Bloom-Ackermann, Z., et al. A deep learning approach to antibiotic discovery. Cell , 180(4):688-702, 2020.

Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting , volume 2019, pp. 6558. NIH Public Access, 2019.

Tu, Z., Liu, Y., Lu, Z., Liu, X., and Li, H. Context gates for neural machine translation. Transactions of the Association for Computational Linguistics , 5:87-99, 2017.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Wang, G., Lu, Y., Cui, L., Lv, T., Florencio, D., and Zhang, C. A simple yet effective learnable positional encoding method for improving document transformer model. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 , pp. 453-463, 2022.

Wang, L., Li, Y., and Lazebnik, S. Learning deep structure-preserving image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 50055013, 2016.

Wang, L., Tu, Z., Way, A., and Liu, Q. Exploiting cross-sentence context for neural machine translation. arXiv preprint arXiv:1704.04347 , 2017.

Ward, L., Agrawal, A., Choudhary, A., and Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials. npj Computational Materials , 2(1): 1-7, 2016.

Wei, J., Chu, X., Sun, X.-Y., Xu, K., Deng, H.-X., Chen, J., Wei, Z., and Lei, M. Machine learning in materials science. InfoMat , 1(3):338-358, 2019.

Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. S. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems , 31, 2018.

Wissler, M. Graphite and carbon powders for electrochemical applications. Journal of power sources , 156(2):142-150, 2006.

Wu, Y.-J., Fang, L., and Xu, Y. Predicting interfacial thermal resistance by machine learning. npj Computational Materials , 5(1):1-8, 2019.

Xie, T. and Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters , 120(14):145301, 2018.

Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: a survey. arXiv preprint arXiv:2206.06488 , 2022.

Yan, K., Liu, Y., Lin, Y., and Ji, S. Periodic graph transformers for crystal material property prediction. Advances in Neural Information Processing Systems , 35:15066-15080, 2022.

Yan, Q., Yu, J., Suram, S. K., Zhou, L., Shinde, A., Newhouse, P. F., Chen, W., Li, G., Persson, K. A., Gregoire, J. M., et al. Solar fuels photoanode materials discovery by integrating highthroughput theory and experiment. Proceedings of the National Academy of Sciences , 114(12): 3040-3043, 2017.

Yao, S. and Wan, X. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th annual meeting of the association for computational linguistics , pp. 4346-4350, 2020.

Zhang, C., Zhang, M., Zhang, S., Jin, D., Zhou, Q., Cai, Z., Zhao, H., Liu, X., and Liu, Z. Delving deep into the generalization of vision transformers under distribution shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7277-7286, 2022.

Zhong, X., Gallagher, B., Liu, S., Kailkhura, B., Hiszpanski, A., and Han, T. Explainable machine learning in materials science. npj Computational Materials , 8(1):1-19, 2022.

Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Zhuo, Y., Mansouri Tehrani, A., and Brgoch, J. Predicting the band gaps of inorganic solids by machine learning. The journal of physical chemistry letters , 9(7):1668-1673, 2018.

Supplementary Material for Density of States Prediction of Crystalline Mateirals via Prompt-guided Multi-Modal Transformer

A Datasets

In this section, we provide further details on the dataset used for experiments.

A.1 Phonon DOS

We use the Phonon DOS dataset following the instructions of the official Github repository 5 of a previous work [9]. This dataset contains 1,522 crystalline materials whose phonon DOS is calculated from density functional perturbation theory (DFPT) by a previous work [42]. Since the provided dataset does not contain crystal system information, we additionally collect the information based on the Materials Project (MP) website 4 on the given each material's unique ID (MP-id).

A.2 Electron DOS

We also use Electron DOS dataset that contains 38,889 crystalline materials. The Electron DOS dataset consists of the materials and their electron DOS information that is collected from the MP website 4 . Among the collected data, we exclude the materials that are tagged to include magnetism because the DOS of magnetism materials is not accurate to be directly used for training machine learning models [23]. We consider an energy grid of 201 points ranging from -5 to 5 eV with respect to the band edges with 50 meV intervals and the Fermi energy is all set to 0 eV on this energy grid. Moreover, we normalize the DOS of each material to be in the range between 0 and 1. That is, the maximum and minimum value for each DOS is 1 and 0, respectively, for all materials. Moreover, we smooth the DOS values with the Savitzky-Golay filter with the window size of 17 and polyorder of 1 using scipy library following a previous work [9].

A.3 Data Statistics of Electron DOS dataset in OOD scenarios

As described in the main manuscript, we further evaluate the model performance in two out-ofdistribution scenarios: Scenario 1 : regarding the number of atom species, and Scenario 2 : regarding the crystal systems. We provide detailed statistics of the number of crystalline materials for each scenario in Table 5 and Table 6.

B Evaluation Protocol

Phonon DOS. As described in the main manuscript, we evaluate the model performance based on the data splits given in a previous work [9].

Electron DOS. On the other hand, for the Electron DOS dataset, we use different dataset split strategies for each scenario. For the in-distribution setting, we randomly split the dataset into train/valid/test of 80/10/10%. On the other hand, for the out-of-distribution setting, we split the dataset regarding the structure of the crystals. For both scenarios, we generate training sets with simple crystal structures and a valid/test set with more complex crystal structures, because it is crucial to transfer the knowledge obtained from simple crystal structures to that from complex structures in real-world materials science. More specifically, in the scenario 1 (different number of atom species, i.e., # Atom species in Table 2), we use Binary and Ternary materials as training data and Unary, Quaternary, and Quinary materials as valid and test data. In the case of Unary, we exclude it from training data despite its simplicity due to the observed difficulty of the structure, as will be discussed in Section E.1. In the scenario 2 (different crystal systems, i.e., Crystal System in Table 2), we use Cubic, Hexagonal, Tetragonal, Trigonal, and Orthorhombic crystal systems as training set and Monoclinic and Triclinic as valid and test set. In this scenario, where no prompt is available for unseen crystal systems, we employ the mean-pooled representations of the trained prompts during testing, i.e., for the Monoclinic and Triclinic crystal systems. Please refer to Table 5 and Table 6 for detailed statistics of crystals in each scenario.

Physical Properties. In addition to evaluating the accuracy of the model's predictions of the DOS, it is crucial to assess the physical meaningfulness of the predicted DOS for real-world applications. To assess the physical meaningfulness of the predicted DOS, we utilize the predicted DOS to estimate a range of important material properties. Specifically, we evaluate three materials' properties: the bulk modulus for phonon DOS, and the band gap and Fermi energy for electron DOS (Table 1).

Bulk Modulus 6 is a thermodynamic quantity measuring the resistance of a substance to compression. It provides a measure of the material's ability to withstand changes in volume under applied pressure. In the context of elastic properties, the bulk modulus serves as a descriptor, as it indicates how well a material can recover its original volume after being subjected to compression.

Another property we focus on is the Band Gap 7 , which refers to the energy range in a material where no electronic states exist. It represents the energy difference between the top of the valence band and the bottom of the conduction band in insulators and semiconductors. Functional inorganic materials, such as those used in applications like LEDs, transistors, photovoltaics, or scintillators, require a comprehensive understanding of their band gap [69]. By accurately predicting the band gap based on DOS, we can accelerate the development of new materials for a wide range of applications.

Additionally, we predict the Fermi Energy 8 , which represents the highest energy level occupied by electrons at absolute zero temperature (0K). It can be used to determine the electrical and thermal characteristics of materials.

C Implementation Details

In this section, we provide implementation details of DOSTransformer .

Graph Neural Networks. Our graph neural networks consist of two parts, i.e., encoder and processor. Encoder learns the initial representation of atoms and bonds, while the processor learns to pass the messages across the crystal structure. More formally, given an atom v i and the bond e ij between atom v i and v j , node encoder ϕ node and edge encoder ϕ edge outputs initial representations of atom v i and bond e ij as follows:

where X is the atom feature matrix whose i -th row indicates the input feature of atom v i , B ∈ R n × n × F e is the bond feature tensor with F e features for each bond. With the initial representations

of atoms and bonds, the processor learns to pass messages across the crystal structure and update atoms and bonds representations as follows:

where N ( i ) is the neighboring atoms of atom v i , ψ is a two-layer MLP with non-linearity, and l = 0 , . . . , L ′ . Note that h L ′ i is equivalent to the i -th row of the atom embedding matrix H in Equation 1.

Model Training. In all our experiments, we use the AdamW optimizer for model optimization. For all the tasks, we train the model for 1,000 epochs with early stopping applied if the best validation loss does not change for 50 consecutive epochs.

Hyperparameter Tuning. Detailed hyperparameter specifications are given in Table 7. For the hyperparameters in DOSTransformer , we tune them in certain ranges as follows: number of message passing layers in GNN L ′ in {2, 3, 4}, number of cross-attention layers L 1 , L 3 in {2, 3, 4}, number of self-attention layers L 2 in {2, 3, 4}, hidden dimension d in {64, 128, 256}, learning rate η in {0.0001, 0.0005, 0.001}, and batch size B in {1, 4, 8}. We use the sum pooling to obtain the crystalline material i 's representation, i.e., g i . We report the test performance when the performance on the validation set gives the best result.

D Methods Compared

In this section, we provide further details on the methods that are compared with DOSTransformer in our experiments.

MLP. We first encode the atoms in a crystalline material with an MLP. Then, we obtain the representation of material i , i.e., g i , by sum pooling the representations of its constituent atoms. With the material representation, we predict DOS with an MLP predictor ϕ ′ , i.e., ˆ Y i = ϕ ′ ( g i ) , where ϕ ′ : R d → R 201 .

On the other hand, when we incorporate energy embeddings into the MLP, we predict DOS for each energy j with a learnable energy embedding E 0 j and obtained material representation g i , i.e., ˆ Y i j = ϕ ( E 0 j || g i ) , where ϕ : R 2 d → R 1 is a parameterized MLP.

Graph Network. We first encode the atoms in a crystalline material with a graph network [4]. As done for MLP, we obtain the representation of material i , i.e., g i , by sum pooling the representations of its constituent atoms. With the material representation, we predict the DOS with an MLP predictor,

i.e., ˆ Y i = ϕ ′ ( g i ) , where ϕ ′ : R d → R 201 . Note that the only difference with MLP is that the atom representations are obtained through the message passing scheme. We also compare the vanilla graph network that incorporates the energy information as we have done in MLP.

E3NN. For E3NN [9], we use the official code published by the authors 9 , which implements equivariant neural networks with E3NN python library 10 . By learning the equivariance, the model can generate high-quality representations with a small number of training materials. After obtaining the crystalline material representation g i , all other procedures have been done in the same manner with other baseline models, i.e., MLP and Graph Network.

E Additional Experiments

E.1 Model Performance Analysis on Out-of-Distribution Scenarios

In this section, we conduct a comprehensive analysis of the model's predictions in the out-ofdistribution scenarios presented in Table 2. In Table 8, we evaluate the performance of the model for each type of material, providing detailed insights into its predictive capabilities. We have following observations: 1) We observe that DOSTransformer consistently outperforms in both out-of-distribution scenarios, which demonstrates the superiority of DOSTransformer . 2) The performance of all the compared models generally degrades as the crystal structure gets more complex. That is, models perform worse in Quinary crystals than in Quarternary crystals, and worse in Triclinic crystals than in Monoclinic crystals. 3) On the other hand, it is not the case in Unary crystal. This is because in Unary crystal only one type of atom repeatedly appears in the crystal structure, which cannot give enough information to the model. However, DOSTransformer also makes comparably accurate predictions in the Unary materials by modeling the complex relationship between the atoms and various energy levels.

E.2 Injecting Crystal System Information to Baseline Methods

In this section, we adopt our prompt-based crystal system information injection procedure to the baseline methods. We examine two approaches for injecting the information: 1) injecting the information into the input atoms (i.e., Position 1), and 2) injecting it before the DOS prediction layer (i.e., Position 2). In Table 9, we have the following observations: 1) Compared to Table 1, all baseline models benefit from using crystal system information. This demonstrates the importance of

utilizing crystal structural systems information, which has been overlooked in previous works. 2) However, DOSTransformer still outperforms all baseline methods with crystal system information (See DOSTransformer in Table 1), verifying the importance of an elaborate design of crystal system injection procedure. To be more specific, we notice a relatively significant performance gap between DOSTransformer and the best baseline model in Electron DOS, which comprises a broader range of crystalline materials than Phonon DOS. This finding highlights the importance of an intricate crystal system injection procedure when striving to learn the DOS of diverse crystalline materials.

E.3 Various Training Data Ratio for Fine-Tuning

In this section, we additionally provide experimental results on various ratios of training data for fine-tuning in Table 3. That is, instead of sampling 10% of training data from the test set used in OOD scenarios in Section 5.3, we try various sampling ratios, i.e., 5%, 10%, 15%, and 20%, from the test set. We have the following observations: 1) We notice a significant performance disparity between the 'Only Prompt" and 'All" approaches, particularly when the training dataset is limited. This phenomenon can be attributed to the challenge of overfitting when fine-tuning the entire model on a small subset of materials, as discussed in Section 5.3. 2) Conversely, as the training data increases, we observe that the difference in performance between the 'Only Prompt" and 'All" methods diminishes. This is due to the enough amount of training data allowing for the adjustment of

15

20

Ratio (%)

10

5

MSE

(Table3)

all model parameters, consistent with the analysis discussed in Section 5.3. However, it is important to note that the existing DFT calculation-based databases suffer from a highly biased distribution, which limits their coverage of different materials. This limitation emphasizes the significance of achieving good performance even with a small subset of training data. Therefore, we argue that the application of prompt tuning enhances the real-world applicability of DOSTransformer .

E.4 Comparison with Crystal Neural Network

We conducted additional experiments on the state-of-the-art crystal neural networks, i.e., ALIGNN [10] and Matformer [63], by adjusting the output dimensions of these models (51 for Phonon DOS and 201 for Electron DOS) in Table 10. Although both ALIGNN and Matformer exhibited commendable performance, we observe that DOSTransformer consistently surpassed them significantly. These observations again demonstrate the importance of considering energy levels for accurate DOS prediction.

E.5 Model Training and Inference Time

In this section, to verify the efficiency of DOSTransformer , we compare the training and inference time of the baseline methods in Table 11. We observe that DOSTransformer requires a longer training time per epoch on the Phonon DOS dataset compared to E3NN, which can be attributed to the two forward passes (i.e., system and global energy embeddings) during the training procedure. However, when it comes to the Electron DOS dataset, DOSTransformer demonstrates a shorter training time per epoch compared to E3NN. This is because the Electron DOS dataset has complex crystal structures, requiring more time for E3NN to learn equivariant representations. Furthermore, in terms of inference time, DOSTransformer demonstrates significantly faster computation per epoch compared to E3NN, particularly on the Electron DOS dataset. This is because we only utilize system prediction without global prediction during inference. As many predictive ML models are used for high-throughput screening in material discovery, inference time is a critical factor for ML models in materials science, demonstrating the practicality of DOSTransformer in real-world applications.

F Broader Impacts

Potential Positive Scientific Impacts. In this work, we propose DOSTransformer , which is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural system, demonstrating its applicability in real-world scenarios. For example, transferring the knowledge obtained from simple structured materials to complex structured materials is crucial because DFT calculation-based databases cover limited types of materials or structural archetypes. Therefore, we believe DOSTransformer has broad impacts on various fields of materials science.

Potential Negative Societal Impacts. This work explores the automation process for materials science without wet lab experiments. However, it is important to acknowledge that in the industry, there are skilled professionals dedicated to conducting such experiments for materials science.

Therefore, it is important to proactively address these concerns by encouraging collaboration between automated methods and human experts.

G Pseudo Code

Algorithm 1 shows the pseudocode of DOSTransformer .

Algorithm 1: Pseudocode of DOSTransformer .

Input : An input crystalline material G = ( X , A ) , Ground truth DOS Y , Number of attention layers L 1 , L 2 , L 3 , Initialized energy embeddings E , Initialized crystal system prompts P .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose DOSTransformer, a transformer architecture for the task of DOS prediction, that takes energy levels as input instead of predicting a list of energies for different energy levels. The performances of this proposed DOSTransformer is better than MLP, GNN, and E3NN instances they implemented.\n\nSTRENGTHS:\n- The proposed method is the first one to take energy levels as input instead of directly predict a list of energies for different energy levels.\n- Writing: clear and easy to follow. Figures are informative.\n\nWEAKNESSES:\n- Novelty is limited, core ablation study is missing: the major novelty beyond previous works is taking energy levels as input instead of directly predict a list of energies for different energy levels. To do this, they proposed the multi-modal module that considers energy levels. But this point is not well supported by the performance ablation studies of E3NN. Also, the ablation study for their proposed model about this point (just directly predict a list of energies) is missing.\n- Missing comparisons with previous crystal neural networks. Previous crystal neural networks including CGCNN, MEGNET, Matformer, ALIGNN, M3GNet can all directly be used for this DOS task, by just changing the output dimensionality from 1 (single scalar value prediction) to M. Just ignoring all these methods is not convincing. Comparing with these methods (at least one or two current SOTA methods) can better demonstrate the importance of taking energy levels as conditions instead of predicting a list of energies. \n- Missing implementation details and what level of irreducible representations is used in E3NN. Simple E3NN with only irreducible representations of rotation order zero or 1 is not powerful enough. Using representations with rotation order of 2 will boost the performances significantly. Given the not significant performance gains beyond e3nn, further implementation details is needed.\n\nQUESTIONS:\nAs listed in above weaknesses.\n\nLIMITATIONS:\nThey listed the limitations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a prompt-based Transformer network for predicting the density of states of crystalline materials. The prompts are used to represent and control the energy and the additional structure information of the materials. Experiments show the proposed method can perform very well on two datasets under different settings.\n\nSTRENGTHS:\n- The whole system and method are motivated well and designed carefully. The model design considers the characteristics of both the task and the deep learning methods, e.g., prompt-based Transformer. \n- The experiments are carefully designed to show the benefits of the proposed methods. \n- The paper is well-written and easy to read.\n\nWEAKNESSES:\n- The datasets and experiments are a bit simple. The variations of the material type and structure type in the datasets are restricted. In this sense, although prompts are used to formulate the general variation of the energy and structure, the datasets contain very simple and limited variations. It is a bit limited in evaluating the potential of the proposed method and the techniques on more realistic datasets and applications. \n    - For example, the prompt is used to represent the structure. There are only seven discrete options that will be given during testing. May the model handle the structure variation without relying on the input during testing?\n- In Table 4, the prompt based method is compared with the one-hot based input. Is the one-hot binary encoding directly used to replace the embedding of the prompt? If so, it might be unfair. How does the one-hot encoding work with a fully connected linear layer (i.e., feeding the one-hot encoding as the input to the FC layer), where the weights in the FC layer are corresponding to the embeddings for each type of the system? \n    - In the worst case, the proposed prompt based method may work similarly to the linear embedding based approaches, on the simple datasets.  \n\n- The authors may analyze the learned prompts to indicate whether the leaned embeddings can reflect and are consistent with the physical characteristics of the system types or structures indicated by the prompts, such as whether some similarity and relationship between these characteristics can be reflected in the learned prompts. \n\n- It seems that fine-tuning do not improves the performance significantly, especially given the identical MAE performance comparing Table 2 and Table 3.\n\nQUESTIONS:\nQuestions are left with the weakness points.\n\nLIMITATIONS:\nThe limitations are discussed and can reflect the main limitations of the proposed method –  the proposed method cannot encode the properties of both materials into a single model.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new transformer-based method DOSTransformer for predicting density of states of crystalline materials. Different from previous methods, the energy level is additionally modeled as an input modality, i.e. the model takes in material configuration and energy level as input to predict DOS(material, energy). DOSTransformer uses cross-attention layers to relate the material and energy representations. Self-attention layers are also used to integrate information across different energy levels. Learnable prompts are introduced to provide information about the crystal system type, which helps capture structure-specific interactions between material and energy. Experiments on phonon and electron DOS datasets show DOSTransformer outperforms existing methods like graph networks and E3NN, in both in-distribution and out-of-distribution scenarios. Ablations and sensitivity analysis demonstrate the benefits of the different components like energy embeddings, global/system losses and crystal system prompts.\n\nSTRENGTHS:\n- Novel problem formulation: The authors provide a new perspective on DOS prediction by treating it as a multi-modal learning problem with material and energy as inputs. This proves to work better for DOS prediction compared to prior material-only models.\n- Neural net design: The multi-modal transformer architecture with cross-attention and prompt-guided self-attention is well-suited for the problem. It allows capturing complex relationships between material and energies.\n- Empirical results: The model is evaluated extensively on two datasets — Phonon DOS and Electron DOS — for both in-distribution and out-of-distribution scenarios. The physical validity of predictions is also analyzed through derived material properties. In particular, the introduced energy embeddings are shown to be useful in reducing prediction errors.\n- Ablation studies: The contributions of different components like the dual losses and crystal system prompts are quantitatively demonstrated through ablation studies.\n\nWEAKNESSES:\n- Simple prompts: The crystal system prompts are relatively simple vector embeddings. More sophisticated prompt learning methods could be explored to reason about structural information of different systems.\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
This work proposes a transformer architecture for predicting density of states of crystalline materials for different energy levels.
The distribution of states is spectral property that is approximated as a function of both the material and energy levels.
The architectures proposed consists of a multi-modal transformer with self and cross attention layers, and a GNN for embedding.
Experiments compare the proposed architecture with state-of-the-art E3NNs, demonstrating a decrease in MSE.

STRENGTHS:
The main strengths of this work include:
1. The approach considers the crystalline material, energy levels, and structural properties.
2. The problem and methods are well-defined, and the architectures are implemented using state-of-the-art libraries.
3. The results include out-of-distribution experiments, ablation studies, and sensitivity analysis.

WEAKNESSES:
The main weaknesses of this work are that:
1. It needs to be clarified what MSE's are required for the approach to be useful in the real world.
For example, in proteomics, and RMSD of less than 2-3Angstrom is considered valid compared to experimental results.
What would be the MSE's required to achieve such a validation in this domain.
2. The terminology and definition used for fine-tuning and especially prompting in this work differs significantly from the standard definition
and should be clarified.

QUESTIONS:
What are the MSE values of bulk, band, and ferm. are considered useful for real-world applications?
Are there 38,889 crystalline materials after or before filtering for non-magnetism?

LIMITATIONS:
The limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a transformer model to predict the density of states of given materials. The model takes in all energy levels that are desired for the Density of States calculation as a 'prompt' and information about the materials structure, the output is a prediction of the DOS at the given energy levels.\n\nSTRENGTHS:\n+ The DOSTransformer is evaluated on both in-domain structures and out of domain structures against relevant baseline models and previously reported models. \n+ The DOSTransformer model design makes use of domain specific knowledge, such as the crystal structure type and the energy levels.\n+ Authors demonstrate that the task of predicting DOS is heavily added by providing energy levels for targeted prediction via relevant ablation studies\n+ The model is clearly described. I was able to understand all the components of the proposed model.\n\nWEAKNESSES:\n- More could be done to describe the task of predicting DOS for downstream applications, especially for a non-materials science audience. How are the DOS outputs for predicting bandgap energy / electrical conductivity? How many grid points are needed to gain an 'accurate enough' prediction of these properties? What is kind of error in the DOS prediction is reasonable? Are there certain energy levels where it is more critical to predict the DOS correctly?\n\n- I would appreciate some more statistics of the datasets used (PhononDOS and ElectronDOS) in the main text (e.g. # atom types, # crystal types).\n\nQUESTIONS:\n- As I understand it one of the tasks in this work is to predict the output of a function (Density of States) on a fixed grid of energy levels. Are there other examples in the literature where transformers are used to predict the outputs of a mathematical function, and the fixed grid of inputs is provided as a prompt for the model? If there are, it would be great to learn about them in the background section.\n\n- Because the DOS is predicted over a finite grid, I am curious about the smoothness of the DOSTransformer model between finite grid points. If the DOSTransformer model was to be evaluated at energy levels between the provided finite points in the prompt, are the predictions smooth where you'd expect the function to be smooth?\n\nLIMITATIONS:\n- More description should be provided about the datasets used. How many different elements/ types of materials were included int he PhononDOS and ElectronDOS provided?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}