Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer
Authors: Namkyeong Lee
Heewoong Noh
Sungwon Kim
Dongmin Hyun
Gyoung S. Na
Chanyoung Park
Venue: NeurIPS 2023 poster (2023)
Keywords: ML4Materials
AI4Science
Graph Neural Networks

Abstract: The density of states (DOS) is a spectral property of crystalline materials, which provides fundamental insights into various characteristics of the materials.
While previous works mainly focus on obtaining high-quality representations of crystalline materials for DOS prediction, we focus on predicting the DOS from the obtained representations by reflecting the nature of DOS: DOS determines the general distribution of states as a function of energy.
That is, DOS is not solely determined by the crystalline material but also by the energy levels, which has been neglected in previous works.
In this paper, we propose to integrate heterogeneous information obtained from the crystalline materials and the energies via a multi-modal transformer, thereby modeling the complex relationships between the atoms in the crystalline materials and various energy levels for DOS prediction.
Moreover, we propose to utilize prompts to guide the model to learn the crystal structural system-specific interactions between crystalline materials and energies.
Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superiority of DOSTransformer.
The source code for DOSTransformer is available at https://github.com/HeewoongNoh/DOSTransformer.

Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer
Namkyeong Leelt, Heewoong Noh Sungwon Kim', Dongmin Hyun", Gyoung S. Na², Chanyoung Park
KAIST
Yahoo Research 3 KRICT
(namkyeong96, sww..o.nnonnnoi.inmiim.mm swkim, cy parrkaaksstss..o ac dhyun@yahooinc. com, ngs0Okrict.re. re.kr
Abstract
The density of states (DOS) is a spectral property of crystalline materials, which provides fundamental insights into various characteristics of the materials. While previous works mainly focus on obtaining high-quality representations of crys- talline materials for DOS prediction, we focus on predicting the DOS from the obtained representations by reflecting the nature of DOS: DOS determines the general distribution of states as a function of energy. That is, DOS is not solely determined by the crystalline material but also by the energy levels, which has been neglected in previous works. this paper. we propose to integrate het- erogeneous information obtained from the crystalline materials and the energies via multi-modal transformer, thereby modeling the complex relationships be- tween the atoms in the crystalline materials and various energy levels for DOS prediction. Moreover, we propose to utilize prompts to guide the model to learn the crystal structural system-specific interactions between crystalline materials and energies. Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superior- ity of DOSTransformer. The source code for DOSTransformer is available at httttpss//gtthni http://gtttuub.cooo/e/oooooottee http://........eoooooooooooooaaa ormer
1 Introduction
Along with recent advances machine learning (ML) for scientific discovery, ML models have rapidly been adopted in computational materials science thanks to the abundant experimental and computational data in the field. Most ML models developed
computational materials
science date have been focused crystalline materials property consisting of kngle-valued
quantities [29), e.g., band gap energy [1 69]. formation energy B 56). and Fermi energy 661
On the other hand, spectral properties ubiquitous in materials science, characterizing various properties of crys- talline materials, e.g., X-ray absorption, dielectric function. and electronic density of states (2 (Figure
Singleoptued
Crystalline Material
Escmitio Eosrgy
Rass-Gap o.
1.229 aWtatom
0,577 el/ooo
The density of states (DOS), which is the main focus of
this paper, is a spectral property that provides fundamental insights various characteristics of crystalline materials, even enabling direct computation of single-valued prop- erties [17]. For example, DOS is utilized feature of materials for analyzing the underlying reasons for changes
ShPO,
Figure A crystal structure and various types of its properties.
*Coresponding author.
'Theee authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in electrical conductivity [[5]. Moreover, band gaps and edge positions, which can be directly derived
from DOS, are utilized to discover new photoanodes for solar fuel generation 47 64] As result, the exploration of MI. capabilities for predicting DOS takes us one step further toward the core principles of materials science, thereby expediting the process of materials discovery. However, credible compu- tation of DOS with traditional density functional theory (DFT) requires expensive time/financial costs of exhaustively conducting experiments with expertise knowledge [4]. Therefore, alternative approaches for DOS calculation are necessary, whereas demonstrating ML capabilities for learning such spectral properties of the crystalline materials is relatively under-explored.
In this paper, we focus on predicting DOS by following the nature of DOS calculation: DOS determines the general distribution of states as a function of energy. That is, while single-valued properties can be described solely by the crystalline material. DOS is determined by not only the material itself but also the energy levels at which DOS is calculated. Therefore, integrating the heterogeneous signals from both the crystalline material and the various energy levels is crucial for DOS prediction. However, previous works for DOS prediction focus on obtaining a qualified representation of material 7
91, thereby overlooking the unique characteristics of DOS.
On the other hand, learning from multiple input types, i.e., crystalline material and various energy levels, is not trivial due to their heterogeneity. To this end, we formulate the DOS prediction problem into multi-modal learning problem, which has recently gained significant attention from ML researchers in various domains thanks to its capability of extracting and relating information from heterogeneous data types
2]. For example, VisualBERT proposes to align elements of input modalities, i.e., images and text, thereby achieving SOTA performance in various vision- and-language tasks [36]. Moreover, Dall-E shows its power in generating images based on text data by encoding images and texts together 43 Among various multi-modal learners, the multi-modal transformer demonstrates its intrinsic advantages and scalability in modeling multiple modalities 62]. by associating heterogeneous modalities with the cross-altention mechanism 665 50)
Inspired by the recent success of the multi-modal transformer, we propose a multi-modal transformer model for DOS prediction, named DOSTransforme
which incorporates crystalline material and energies as heterogeneous input modalities. Unlike previous works that overlook the energy levels for predicting DOS 77 [14] I7 91 DOSTransformer learns embeddings of energy levels which are used for modeling complex relationships between the energy levels and crystalline material through a cross-attention mechanism. By doing so, DOS ToSTransformer obtains multiple representations for a single crystalline material according to various energy levels, enabling the prediction of a single DOS value on each energy level. However, simply combining a crystalline material with energy levels may fail to consider the significant impact of the crystal's structure, which sometimes results in distinct material properties even when it is composed of identical elements (e.g.. carbon). For example, although graphite and diamond are made entirely out of carbon, their arrangement (or structure) of carbon atoms differs significantly, resulting in distinct material properties. Therefore, we utilize learnable prompts to inform DOSTransforme P) which navigates the model to capture structure-specifi interactions between a crystalline material and energy levels. By doing so, DOS FOSTransformer learns to extract not only the relational information that is shared with crystal system, but also across the crystal systems. In this work, we make the following contributions:
of the input crystal system (among the
seven crystal systems
Inspired by the fact that DOS is determined by not only the materials themselves but also the energy levels at which DOS is calculated, we propose a multi-modal transformer model for DOS prediction, called POSTransformer which incorporates crystalline materials and energy as heterogeneous input modalities.
To capture structure-specific interactions between a crystalline material and energy levels, thereby being able to predict DOS that significantly differs according to the crystal's 's structure, we utilize learnable prompts to inform DOSTransformer of the input crystal system.
Our extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, demonstrate that DOSTransformer outperforms a wide range of state-of-the-an art methods in various real-world scenarios, i.e., in in-disttribuion scenarios and out-of-distributin scenarios.
To the best of our knowledge, this is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural systems.
atttps://en.. wikipedia. baappedaa.or///iee orgg/wildd/rystall.yy

---

2 Related Works
2.1 Machine Learning for Materials and Density of States
Traditional materials science research heavily relies on theory, experimentation, and computer simulations, but these approaches are time-consuming, costly, and inefficient, hindering their ability to keep up with the field's development [57]. As an alternative research tool, ML, which requires neither expensive trial and error experiments nor domain expert knowledge, has been attracting a surge of interest from materials scientists [67]. Consequently, various ML methods have been established to learn the relationship between materials and their properties which are usually obtained from ab-initio calculations [TT] Inspired by the recent success of GNNs in biochemistry (99 48 25]
33 34 82.
CGCNN proposes a message-passingg passeeggpnnnnggg framework for crystal structure providing high- accuracy prediction for 8 different materials properties [6]] However, most studies have focused on single-valued properties [29], whereas various spectral properties, such as DOS, are also erucial.
The DOS describes the number of different states at a particular energy level that particles are allowed to occupy, determining various properties of crystalline materials. For example, phonon DOS has crucial influence in determining the specific heat and vibrational entropy of crystalline materials as well as the interfacial thermal resistance [60]. while electron DOS is used to derive the electronic contribution to heat capacity in metals (21) and the effective mass of electrons in charge carriers [18)
Despite the importance of DOS, how to incorporate MI. capabilities for decoding such spectral properties is under-explored since the vast majority of methods concentrate only on encoding crystalline materials. Specifically, Chandrasekaran et al. ( proposes to predict electron DOS based on the hand-crafted fingerprint of each grid point the crystalline materials, while Del Rio et al. [4] leverages that of each atom in the crystalline materials. Chen et al. [9] predicts phonon DOS by learning the representations of the crystalline materials that are equivariant to 3D rotations, translations, and inversion with Euclidean neural networks 49 28
58) achieving high-quality
prediction with a small number of training materials with over 64 atom types. In this work, we concentrate decoding spectral properties regarding the complex relationship between the various energy levels and crystalline materials.
2.2 Multi-modal Transformer, Prompt Tuning, and Positional Encoding
Multi-modal Transformer. In recent years, much progress in multi-modal learning has grown rapidly by extracting and aligning the rich information from heterogeneous modalities (2 o Among the various multi-mode learning methods, the multi-modal transformer demonstrates. its superior capability in learning multiple modalities [62], by associating the heterogeneous modalities with a cross-attention mechanism. Specifically, MulT (50) learns the representations directly from unaligned multi-modal streams with multi-modal transformer, while Yao Wan [5] improves machine translation quality by learning the image-aware text representations. In this paper, we investigate how to decode DOS of a crystalline matcrial by learning the energy-aware crystal representations via a multi-modal transformer.
Prompt Tuning, On the other hand, prompt designing has gained significant attention from NLP researchers as an efficient method for fine-tuning large language models (LLMs) for various tasks [39].
There are two main approaches to prompt designing: discrete prompt designing 44] and continuous
prompt tuning [37] While discrete prompt designing involves manually creating task description tokens for LLMS through trial and error, continuous prompt tuning trains learnable prompts a continuous latent space without requiring expert knowledge. Inspired by the success of continuous prompt tuning in NLP domain, has been widely adopted in various domains such as computer vision [24] and vision- Languaee (68]. While previous works utilize prompts for efficient fine-tuning of the models, we employ prompts to capture the heterogeneity of various crystal structures.
Positional Encoding. In the original proposal of the Transformer architecture, sinusoidal function was suggested as a method for positional encoding. However, the sinusoidal function may have limi- tations in terms of learnability and flexibility, which can affect its effectiveness (53). To address this
issue, most pre-trained language models [66 40 employ learnable vector embeddings as positional representations. By conceptualizing each energy value as the position of a word in sentence, various energy embeddings in Section 4.2 can be analogous to the positional encoding used in traditional transformers.
3

---

3 Preliminaries
Notations. Let G = A)) denote a crystalline material, where V= {v1,... .u represents the set of atoms, and A VVV V represents the set of edges connecting the atoms in the crystalline material g. Moreover, 9 is associated with a feature matrix X E Rn and an adjacency matrix A E R where Aij 1 if and only if (vi; 0y) E A and Aij = 0 otherwise.
Task: Density of States Prediction. Given set of crystalline materials Dg = { ,..,.,,,.99. ................................ 96v) and set of energies D3- = (E1,E2
EM J. our goal is to train model M that predicts the DOS of a crystalline material given a set of energies, i.c., Y M(G, Dg), where Y E RM is an M dimensional vector containing the DOS values of a crystalline material G, at each energy E EM. and Y E IR is the DOS value of G, at energy level Ej.
4 Methodology: DOSTransformer
In this section, we introduce our proposed method, named DOSTransformer, novel DOS prediction framework that learns the complex relationship between the atoms the crystalline material and various energy levels by utilizing cross-attention mechanism of the multi-modal transformer.
4.1 Crystalline Material Encoder
Before modeling the pairwise interaction between the crystalline material and the energies, we first encode the crystalline material with GNNs to learn the representation of each atom, which contains not only the feature information but also the structural information. Formally, given a crystalline material G = (X, A). we generate an atom embedding matrix for the crystalline material as follows:
= GNN(N, A),
where H E R is atom embedding matrix for G, whose i-th row indicates the representation of atom and we stack L' layers of GNNs. Among various GNNs, we adopt graph networks 4 as our crystalline material encoder, which a generalized and extended version of various GNNs. We provide further details on the GNNs in Appendix
4.2 Prompt-based Multi-modal Transformer
After obtaining the atom embedding matrix H. we aim to capture the relationship between a crystalline material and various energy levels by utilizing the cross-attention layers of a multi-modal transformer [62] with oeff-attention layers [52] In a nutshell, we expect the cross- aatsentttntion layers to integrate the heterogeneous signals from a crystalline material and various energy levels, while self-attention layers aim to integrate the information of material-specifie energy representations.
Cross-A Attentionn Layers. Specifically, we expect the cross-altention layers to generate the material- specific representation of the energies by repeatedly reinforcing the energy representation with the crystalline materials. To do so, first introduce learnable energy embedding matrix E R xd whose j-th row, i.e., DJ. indicates the embedding of energy &, e Dg. Then, we present cross-modnt attention for fusing the information from the crystal structure into various energy levels as follows:
El
Qg KH. VH) Mxd
E Vd
E
(2)
Softmax(
H
where/ / 1, L1 indicates the index of the cross-attention layers. In contrast to the conventional Transformer, which introduces learnable weight matrices for query Q, key K, and value V, we directly employ the previously obtained energy embedding matrix E as the query matrix, and the atom embedding matrix H as the key and value matrices. Based on the above cross -otss-attention mechanism, we obtain the material-specific energy embedding E' Md by aggregating the information regarding the atoms in the crystalline material that were important at each energy level. The final material-specific energy embedding matrix E.4 E Rd generated by the cross-attention layer reflects the relationship between the atoms in the crystalline material and various energy levels.

---

1' gM
2
Energy Occoder
Scttroa
1elvov Petusko
Softmuc
Prompts
Cross Arsss-Attttioonn
Monedaac
Esum glraso
Lie Glbbll Self mncmiin
1
Self Amtention Layer
gits e gend
It gene
H
System AtPeettttoo
x
Cress Attention Layer
Maxia kakocing
Query
Key
Apa
Query |
Key
Value
Enrrgr Embedding
Crytalline
SNPO,
embedding
Model architecture
Cross-Attention Layer
Self-Attention Layer
Figure 2: Overall model architecture and attention layers in prompt-based multi-modal Transformer.
Global Self-Attention, In addition to cross-attention layers, we propose to enhance the material- specific energy embedding matrix E by aggregating the information from other energy levels. To do so, we employ variants of self-attention layers in conventional Transformer (52] as follows:
E Self-Attentioni
Qga-n. K
g--,, V B E Rdxdd
EP EP-TT
Softmax
Vd
where p I,..., L2 indicates the index of the self-attention layer. We use enhanced material- specific energy embedding E as input of the self-attention, which is obtained by concatenating the material-specifie energy embedding E with the material representation gi Rd i.e., EJ: 61 (Egioh), where Eo = (E/" d1 Rdd + RR and || indicates the concatenation operation. Note that gi is sum pooled representation of the material Gi, and E indicates the j-th row of the energy embedding matrix EL1 computed by Equation 2 where j = O..... .M. Although the output of cross-attention layers, i.e., ' E R. captures the local level information regarding the material, we also recognize the importance of incorporating the global material-leved information gi that may not have been fully encoded during the cross-attention phase. Note that the importance of considering the global contextual information for enhancing the learning of dependencies among neural representations has been widely studied various domains such as computer vision and natural language processing [51 55 22 By additionally considering the global material-level information, we aim to enhance the material-specific representation the energies more comprehensively.
System Self-Attention with Crystal System Prompts. However, solely aggregating the information of material-specific energy embeddings may neglect the crucial influence of the structural properties in crystalline materials, whereas the structural properties play significant role in determining the unique characteristies and properties of these materials [45]. To illustrate this point, let's examine the case of graphite and diamond, both composed entirely of carbon atoms. In graphite, carbon atoms are arranged in stacked layers, where each carbon atom bonded to three neighboring carbon atoms in hexagonal lattice. In contrast, the carbon atom in diamond forms strong covalent bonds with four neighboring carbon atoms, resulting in a tetrahedral arrangement. Even though graphite and diamond made entirely out of carbon, due to the structural difference, graphite exhibits properties such as electrical conductivity and lubricity due to its layered structure [59] 26), while diamond is renowned for its hardness and thermal conductivity owing to its tightly bonded tetrahedral network [8] 27
Despite the importance of structural information, effectively incorporating into the model is not triv- ial. Naively concatenating the structural information as an input feature of materials is straightforward but may lead to interference, hindering the model's 's ability to learn and generalize knowledge across different crystal structures. To this end, we propose to provide additional information to the self- attention layers with learnable prompts, which indicate the structural information of the crystalline materials. Specifically, we adopt learnable prompts P E R 7xd,, whose k-th row P& represents one of the seven widely known crystal systems: Cubic, Hexagonal, Tetragonal, Trigonal, Orthorhombic, Monoclinic, and Triclinic. Then, given crystalline material G. whose crystal system is given as k, we incorporate the learnable prompts into the material-specifer energy embedding E as input
5

---

of the self-attention, which is obtained as follows: E dz (E,"",, where E (E/ PA)
and Ø2 _22d+,, + Rd. By doing so, we obtain high-level energy features that contain the crystal structural system information (i.e., E2 while keeping the low-level features to share the knowl- edge across all the materials (i.e., E.a), In Section 5.4 we demonstrate the effectiveness of our
prompt-based approach compared to other variants.
Besides the self-attention layers, we use additional cross-attention layers on top of the self-attention layers, whose input energy embedding E in Equation 2 given as E and output the final material- specific energy embeddings ELs, where L3 the number of additional cross-attention layers. By doing so, the model extracts the enhanced relationships between the crystalline materials and energy levels regarding the crystal structural system information.
4.3 Energy Decoder
After obtaining the final material-specific energy embedding matrix E of a crystalline material Go the DOS value at each energy level E,. i.e., Yj. is given as follows: Y; = Opred (D)a.t) where Upred Rdd - RR is a parameterized MI.P for predicting DOS. While previous approaches directly predict DOS from the material representation using a function Spred R + RM,, DOSTransformer takes a different approach by making pointwise predictions that align with the nature of DOS calculation, i.e., DOS determines the general distribution of states as a function of energy To the best of our knowledge, DOSTransformer is the first work that predicts DOS values in a pointwise manner at each energy level, and we later show in Section that such an approach further enhances the performance of not only DOSTransTon Transfforer but also existing models.
4.4 Model Training and Inference
DOSTransformer is trained minimize the root mean squared error (RMSE) loss L be- tween the predicted DOS value Y; and the ground truth DOS value Yj, i.e.,
N M V Yj)², More specifically, DOSTransformer is trained by combining two N.M Ei=1
distinct RMSE losses with balancing term 8, i.e., Ltotal = Lolob + B Laws where Calob and Lays are obtained by utilizing Ega.o and E4a.sys, respectively. By doing so, DOSTransformer extracts the relationship information that is shared among the distinct crystal systems and within a crystal system, respectively. For inference, we utilize the model prediction obtained based on Eya.eys The overall model architecture is depicted in Figure
Experiments
5.1 Experimental Setup
Datasets. We use two datasets to comprehensively evaluate the performance of DOSTransformer, ie.. Phonon DOS and Electron DOS. We use the Phonon DOS dataset following the instructions
of the official Github repository of previous work [9] For Electron DOS dataset, we collect crystalline materials and their electron DOS data from Materials Project (MP) website We provide
further detailed preprocessing procedures and statistics each dataset in Appendix A
Methods Compared. We mainly compare DoSTransformer to recently proposed state-of-the-ar method, i.e., E3NN (9) which utilizes the Fuclidean network for encoding material representation. We also compare DOS Transforrmrr to simple baseline methods, i.e., MLP and Graph Network 4. which predicts the entire DOS sequence directly from the learned representation of the materials without regarding the energies during training. Moreover, to evaluate the effectiveness of the transformer layer that considers the relationship between the atoms and various energy levels, we integrate energy embeddings into baseline methods for DOS prediction by concatenating the energy embeddings to the material representation. We provide more details on the implementation of DOSTransformer and compared methods in Appendix C and D respectively.
httppp///gtta...cc/////atttaaaaa httr:::/materrialgpppjeettttt

---

Table Overall model performance under in nn-distribution scenarios (Bulk M. G. Band Gap Ferm. E. Fermi Energy).
Bulk Modulus Band
Phonon DOS
Electron DOS
Physical Properties (MSE)
Model
MSE
MAE
12a
MSE
MAE
R²
Bulk M.
Band G.
Ferm,
Energy X
0.309
0.106
0.576
0.347
0.130
0.487
0.695
1.597
4,650
MLP
(0.016)
(0.003)
(0.029)
(0.171)
(0.202)
Graph Network
0.259
0.099
0.638
0.264
0.103
0.613
0.688
1.457
3.362
(0.13))
(0.051)
E3NN
0.210
0.077
0.705
0.296
0.109
0.552
0,504
0.896
2.925
(0.004)
(0.001)
(0,00)
(0,005)
(0.001)
(0.013)
(0.033)
(0.095)
(0.111)
Energy /
0
0.099
0.652
0.341
0.128
0.499
0.521
1,409
4.372
MLP
(0.004)
(0.301)
(0.006)
(0.011)
(0.002)
(0.012)
(0.011)
(0.182)
(0.067)
Graph Network
0,226
0.092
0.685
0.240
0.099
0.650
0.543
1,263
3,080
(0.007)
(0.002)
(0.011)
(0.005)
(0.001)
(0.005)
(0.085)
(0.107)
(0.100)
0,200
0.074
0.724
0.291
0.114
0.564
0.451
1.605
3.777
E3NN
(0.001)
(0.001)
(0.002)
(0.000)
(0.000)
(0.008)
(0.023)
(0.231)
(0.175)
DOS Transfsforrrr
0,191
0.071
0.733
0.221
0.089
0.679
0,427
0.461
2.337
(0.003)
(0.002)
(0.004)
(0.006)
(0.001)
(0.006)
(0.024)
(0.019)
(0.094)
5.2 In-Distribution Evaluation
Evaluation Protocol. In this section, we evaluate the model performances under in-distribution scenarios. While we evaluate DOSTransformer with given data splits in previous work [ for Phonon DOS dataset, we randomly split the Electron DOS dataset into train/valid/test of 80/10/10% Note that while all measures are reported in the original scale, we report MSE values for DOS prediction multiplied by a factor of 10 for clear interpretation during all experiments.
Experimental Results. In Table D we have the following observations: Comparing the baseline methods that overlook the energy levels for DOS prediction (i.e., Energy X) with their counterparts that incorporate both the energy levels and the crystal structure as heterogeneous input modalities through the energy embeddings (i.e., Energy v), we find out that using the energy embeddings consistently enhances the model performance. This indicates that making pointwise predictions on each energy level is crucial for DOS prediction as discussed in Section 4.3) which also aligns with the domain knowledge of materials science, i.e., DOS determines the general distribution of states as function of energy. 2) However, we observe that E3NN shows relatively small performance gain compared with other methods after incorporating the energy information. This is because the integration of the energy embedding may confound the model to learn the proper equivariance of materials. 3) On the other hand, DOSTransformer outperforms previous methods that do not consider the complex relationships between the atoms in materials and various energy levels via the various attention mechanisms. This again implics that naive integration of the energy information into previous models cannot fully benefit from the energy information. We also provide qualitative analysis on the obtained DOS in Section 5.3
Physical Validity of DOS. In addition to the accuracy of DOS prediction, we assess the physical validity of the moodel-predicted DOS by deriving various physical properties, i.e., bulk modulus, band gap, and Fermi energy, of materials, based on the model-predicted DOS. To do so, given the DFT-calculated DOS, which considered the ground-truth DOS based on which accurate derivation of various physical properties is possible, we train an MLP with a non-linearity in each layer to predict the properties
crystal structure. Then, based on the obtained MLP weights, we predict the material properties given the material property. We provide further details on the experimental setting in Appendix B In Table [ observe that DOS predicted by DOST Transformer shows the superiority of predicting physical properties of materials, indicating that DOS Transformer not only achieves high accuracy but also produces physically valid predictions.
DOS the input and calculate the MSE for each
5.3 Out-of-Distributie Evaluation
In this section, we evaluate the model performances under out-of-dist (OOD) scenarios. It is well recognized that existing DFT calculation-based databases have limitations in terms of
7

---

Table 2: Overall model performance in OOD scenarios.
Table 3: Fine-tuning on OOD systems.
Out-of-Distribution
Model
MSE
MAE
R
Model
Atom Species
Crystal System
Energy X
MSE
MAB
R²
MSE
MAE
R
MLP
0.401
0.137
0.510
Energy x
(0.017)
(0.003)
(0.021)
Graph Network
0.394
0.131
0.519
MLP
0.545
(0.00;:
(0,0001)
0.161
(6.003)
0.250
(0.06)
0.455
0.149
(0.002)
(0.007)
0.437
E3NN
0.414
(0.07)
0.133
(0.002)
0.490
(0.10)
Graph Network
0.460
0.144
0.381
0.407
0.132
0.505
(0.06)
(0.001)
(0.007)
(0015)
(6,025)
(0.04)
(0.001)
Energy /
E3NN
(0.07)
0.541
0.154
0.231
(6.003)
(0.04)
0.421
0.134
(0.007)
0.483
MLP
0.394
0.136
0.519
(0.009)
Energy /
Graph Network
0.382
0.130
0.533
0.545
0.159
0,260
0.455
0.149
0.445
(0.003)
(0.000)
(0.00))
MLP
0.455
(0006)
0.144
0,388
(0,016)
(U000)
0.384
0.129
(0001)
(0.037
0.534
E3NN
(0.004)
(0.001)
0.487
(0.007)
Graph Network
(0002)
(0,010)
(0012)
(0/04)
(0.16)
DOS Transforrer
0.534
0.152
0,237
0.420
0.133
0.483
E3NN
(0013)
(0,001)
(0.017)
(00077
(0/001)
(0.03)
All
0.365
0.122
0.558
(0.005)
(0.001)
(0.008)
DOS ToSTransformer
0,454
0.136
0.399
0.373
0.122
0.552
Only Prompt
0.355
0.122
0.570
(0 (0.008)
(0.001)
(0.017)
(0.006)
(0.001)
(0.008)
(0.008)
(0.001)
(0.010)
coverage, as they often focus on specific types of materials or structural archetypes. resulting in biased distributions [13] 30 20]. Therefore, it is essential to evaluate the model's performance in OOD scenarios to assess its real-world applicability and generalizability beyond the limitations of the available databases 35
Evaluation Protocol.
To do so, we evaluate the model performance the crystal structures that 1) contain a different number of atom species with the training set (i.e., # Atom species), and 2) belong different crystal systems that were not included in the training set (i.e., Crystal System). In both scenarios, training data primarily consist of relatively simple structures compared to those in the test data. We provide further details data split and evaluation in Appendix B
Experimental Results. In Table we again observe that utilizing energy embeddings consistently brings performance gain to all baseline models. Especially, Graph Network significantly benefits from energy embeddings compared to in-distribution scenarios (See Table We attribute this to the inherent restrictive inductive biases in Graph Network (4) i.e., Graph Network heavily relies on the structural information of materials. Specifically, as the structure of the materials totally varies in the OOD scenarios, which makes the prediction of the whole DOS sequence challenging, incorporat- ing the energy embeddings becomes especially helpful in the OOD scenarios for Graph Network. Moreover, DOSTransformer also alleviates the restrictive inductive bias of Graph Networks by elaborately modeling the complex relationships between energies and materials. It is worth noting that similar observations have been made by comparing Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in the computer vision domain [1] 66). In conclusion, we argue that incorporating energy information during model training crucial not only for in-distribution scenar- ios but also for OOD scenarios, demonstrating the applicability of DOSTransformen in real-world applications. For a more detailed analysis, please refer to Appendix E.1
Fine-tuning on Few Materials from Complex Crystal Systems. In addition to evaluating the performance under the OOD scenarios, we verify how well the models can make predictions for materials from complex crystal systems with only few training samples, which is a practical situation in reality. For this, we sampled small subset of of materials (i.e., 10%) from the test used in OOD scenarios of crystal systems, and fine-tune the models that are already trained on the training set. As for DOSTransformer, we fine-tune the model using two different approaches: parameters (referred to as "All"), and ii) tuning only the parameters of the prompts and the energy decoder while keeping all other model parameters frozen (referred as "Only Prompt"). In Table 3. we have the following observations: 1) As we expected, the additional fine-tuning step achieves performance gain for all models, while was marginal due limited number of materials used for fine-tuning. 2) Only fine-tuning the prompts of DOSTransformer achieved more performance gain compared to fine-tuning the whole model. This is because while fine-tuning the whole model on a small subset of materials may easily incur overfitting, fine-tuning only prompts enables the model to additionally learn from few new samples while maintaining the knowledge obtained previously. In conclusion, we believe that the proposed prompt tuning approach can have significant impact in the
fine-tuning all model
8

---

field of materials science beyond simple DOS prediction, where the existing databases exhibit a bias
towards dominant types of materials [ 30 20]. We further explore different proportions beyond the 10% of the materials used for fine-tuning, as detailed in the Appendix
5.4 Model Analysis
Ablation Studies. To verify the benefit of each component of DOS- Transformer, we conduct ablation studies under in in-distribution scenarios in Figure We observe that utilizing only either one of the global and system losses deteriorates the model performance. This is because jointly optimizing the losses incentivizes the model to extract the relational information that is shared across the crys- tal systems and within a crystal system. On the other hand, it is worth noting that DOSTransformer with only one of the losses still outperforms baseline models shown in Table I indicating that mod- cling complex relationships between materials and various energies is crucial in DOS prediction.
0.230
0.255
Phonon D05
Bectron DOS
MSE
0.700
0.255
0.185
0.230
0.190
0.215
w/o
w/o
-former
Figure 3: Ablation studies.
0.210
0.235
0.205
Flectis nos
0.230
Sensitivity Analysis. To verify the robustness of DOSTransformer, we measure the model performance by varying 3, which is intro- duced control the effect of the crystal system loss, i.e. Leve, in Section In Figure DOS Transforrrr shows robustness over various levels of 3, consistently outperforming baseline models Table | This verifies that DOS Transfformrr can be easily trained without an expensive hyperparameter tuning, further demonstrating the practicality of DOSTransformer.
0.395
0.20
0.390
0.15
0:385
0.20
1.0
1.2
1.5
B
Figure 4: Sensitivity analysis.
Crystal System Prompts. As discussed in Section 4.2
Table 4: Comparing various crystal system information injection approaches.
it not trivial to effectively incorporate the crystal system information into the model. To evaluate the effectiveness of our proposed prompt-based approach, we explore different approaches for incorporating the crystal system information into the model. We make
Method
Electron DOS
One-hot
Prompt
MSE
MAE
0,222
0.089
Atom Fear.
(0,004)
(0,001)
modifications to determine where and how to inject
Location
0.227
0.090
this information. In terms of the location, two options: injecting the information into the input atoms (i.e., Atom Feat.) and injecting it before the self- attention layers (i.e., Before SA). Regarding the method of injection, explore two strategies: one-hot encoding of crystal systems and the use of learnable crystal system prompts. In Table we have following observations: 1) Naively incorporating prompts into atom feature even perform worse than the model without using crystal system (see "w/o Laye" in Figure 3). demonstrating the importance of an elaborate design choice for injecting crystal system information. 2) We observe that our approach, i.e. Before SA and prompt, outperforms all other possible choices, indicating that DOS DOSTransformer successfully incorporates crystal system information during training. We also examine the benefits of injecting crystal system information into baseline methods in Appendix E.2
consider
0,226
0,090
Before
(0,005)
(0,201)
SA
0.221
0.089
(0,006)
(0.001)
5.5 Qualitative Analysis
In this section, we provide a qualitative analysis of the predicted DOS by mainly comparing it to the DFT-calculated (i.e., Ground Truth) DOS and our main baseline (i.c., E3NN). In Figure 5 (a).
which represents the predicted DOS of materials not containing transitional metals, both E3NN and DOSTransformer successfully capture the overall trend of the DOS for several materials (c.g.. mp- 9823683 mp- 1pp1000912 129, and mp-16378). However, TDSTransformer shows much more precise prediction that closely aligns with the ground truth DOS, providing even more useful information beyond the shape of DOS. For example, peak points represent regions of high density and are likely to be strongly influenced when materials undergo changes in property, and thus represents the probabilistically important energy regions of the materials in the process of material discovery. Notably, our model better captures the peak points in the ground truth DOS compared to E3NN, demonstrating the applicability of DOS PoSTransformer-predii
DOS for material discovery process.

---

Ground Truth
E3NN
DOSTranstormer
902366
Imp-109311 HoSF
[mp-1009129] Mgo
1.
(a) w/o Trans. metal
eDOS
[mp-13898] Ga2NatnFS
[mp-768368] K412
(mp-16378] Ho52
000
Imp 760381] CUS
Imp 708035) NaZniH30213
(mp-1080023) MoOS1
1.4
3.0
1.0
(b) w/ Trans. metal
eDDS
Rb2054
Imp-288]
Imp-10100511
Energy (eV)
Figure 5: Qualitative Analysis.
On the other hand, (b) shows the DOS prediction for materials containing transition materials. Although DOSTransformer provides more reliable prediction, we observe that the prediction errors of both models get larger compared to the materials that do not contain transition metals shown in Figure 5 (a). This can be attributed to the inherent complexity of physical properties in materials containing transition metals, as discussed in Section Therefore, for our future work, we plan to design expert models in which each expert is responsible for materials with and without transition metals, to achieve more refined and accurate predictions of the DOS. This approach would enable a more comprehensive and elaborate analysis of the DOS in different material compositions.
6 Limitations & Future Work
It is widely known that materials containing transitional metals inherently have complex physical properties compared other materials. As result, the density functionals that yield optimal performance in DFT calculations differ between materials with and without transitional metals 12 46). However. despite the distinction, most ML-based DOS prediction research. including Tranfoosfrrrrr, have attempted to encode the properties of both material types into a single model 17 which may provide conflicting signals the model. Therefore, as a future work, we plan to address this issue by developing specialized approaches that work well on both material types, e.g., utilizing expert models [41 in which an expert assigned each material type and later combined to perform well on both material types. We expect this approach to result in the robust and reliable prediction DOS for both types of materials, enhancing the potential impact in the materials science field and broadening our understanding of material properties.
7 Conclusion
In this paper, we propose DOSTransformer, which predicts DOS of crystalline materials with various energy levels by following the nature of DOS calculation. Specifically, DOSTransformer associates the heterogeneous information from materials and energy levels with layers. Moreover, the crystal system prompts are introduced effectively train the model learn the relational information that is shared across all the crystal systems and within crystal system. By doing so, DOSTransformer outperforms previous works in predicting two types of DOS, i.e., phonon DOS and electron DOS, in various scenarios, i.e., in-distribution scenarios and out-of-distribuiee scenarios. Extensive experiments verify that incorporating energy information is crucial predicting the DOS of a crystal structure for real-world application and DOSTrannformm effectively utilizes the crystal structural information with crystal system prompts.
and self-attention
10

---

Acknowledgements
This work was supported by the core KRICT project from the Korea Research Institute of Chemical Technology (S12051-10).
References
[1] Bai, Y., Je, Yuille, A. L.,, and Xie, C. Are transformers more robust than cnns? Advances in Neural Information Processing Systems, 34:26831-26643, 2021.
[2] Baltrušaitis, T., Ahuja, C., and Morency, L.-. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-443, 2018.
[3] Banjade, H. R., Hauri, S., Zhang, S., Ricci, F., Gong, W., Hautier, G., Vucetic, S., and Yan, Q- Structure motif-centric learning framework for inorganic crystalline systems. Science advances, 7(17)ceabf175 754, 2021.
[4] Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzaler A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01200 2018.
[5] Bayoudh, K., Hamdaoui, F. and Mtibaa, A. Hybrid-covid: a novel hybrid 2d/3d cnn based on cross-domaii adaptation approach for covid-19 screening from chest x-ray images. Physical and engineering sciences in medicine, 43(4):1415-1441, 2020.
[6] Bayoudh, K., Knani, R., Hamdaoui, F,, and Mtibaa, A. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 38(8): 2939 2970, 2022.
[7] Chandrasekaran, A., Kamal, D., Batra, R., Kim, C., Chen, L.,, and Ramprasad, R. Solving the electronic structure problem with machine learning. npj Computational Materials, 5(1):1-7, 2019.
[8] Che, J., Çagın, T., Deng, W., and Goddard III, W. A. Thermal conductivity of diamond and related materials from molecular dynamics simulations. The Journal of Chemical Physics, 113 (16):6888 (9008888-000, 2000.
[9] Chen, 7., Andrejevic, N., Smidt, T., Ding. 7.,, Xu, Q., Chi, Y.-T., Nguyen, Q. T., Alatas, A., Kong, J., and Li, M. Direct prediction of phonon density states with euclidean neural networks. Advanced Science, 8(12):2004214, 2021.
[10] Choudhary, K. and DeCost, Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1): 2021.
[11] Choudhary, K., DeCost, B., Chen, C., Jain, A., Tavazza, F., Cohn, R., Park, C. W., Choudhary, A., Agrawal, A., Billinge, S. J., et al. Recent advances and applications of deep learning methods in materials science. npj Computational Materials, 8(1):1-26, 2022.
[12] Cramer, C. J. and Truhlar, D. G. Density functional theory for transition metals and transition metal chemistry. Physical Chemistry Chemical Physics, 11444) 2009.
[13] De Breuck, P.P.,, Evans, M. L.,, and Rignanese, G.-M. Robust model benchmarking and bias-imbalance in data-driven materials science: case study modnet. Journal of Physics: Condensed Matter, 33440044 2021.
[14] Del Rio, B. G., Kuenneth, C., Tran, H. D., and Ramprasad. R. predict the electronic structure of materials and molecules: The example of graphene-derived allotropes. The Journal of Physical Chemistry A, 124(45)9449-9955) 2020.
An efficient deep learning scheme
[15] Deringer, V. I.., Bernstein, N., Csányi, G., Ben Mahmoud, C., Ceriotti, M., Wilson, M., Drabold, D. A., and Elliott, S. R. Origins of structural and electronic transitions disordered silicon. Nature, 689 2021.

---

[16] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint #rXi::8100048055 2018.
[17] Fung, V., Ganesh, P., and Sumpter, B. G. Physically informed machine learning prediction of electronic density of states. Chemistry of Materials, 2022.
[18] Garrity, K. F. First-principles search for n-type oxide, nitride, and sulfide thermoelectrics. Physical Review B, 94(4):045122, 2016.
[19] Gilmer, J., Schoenholz, S. S., Riley, P. P. Vinyals, O., and Dahl, G. E. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1273-1272
PMLR, 2017.
[20]
Griffiths, R.-R., Schwaller, P., and Lee, A. A. Dataset bias in the natural sciences: case study in chemical reaction prediction and synthesis design. arXiv preprint arXiv:2/05.022 2021.
[21] Hamaoui, G., Horny, N., Hua, Z., Zhu, T., Robillard, J.-F., Fleming, A., Ban, H.. and Chirtoc, M. Electronic contribution in heat transfer metal semiconductor and metal seiiidde-semiconduetor interfaces. Scientific reports, 8(1):1-9, 2018.
[22] Hatamizadeh, A., Yin. H., Kautz, J., and Molchanov, P. Global context vision transformers. arXiv preprint arXiv:2206.0900 209929990009.......92999999999 2022.
[23] Illas, F., de PR Moreira, I., Bofill, J., and Filatov, M. Extent and limitations of density-functional theory in describing magnetic systems. Physical Review B, 70(13):132414, 2004.
[24] Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pp. 709-727. Springer, 2022.
[25] Jiang, D., Wu, Z., Hsieh, C.-Y., Chen, G., Liao, B., Wang, Z., Shen, C.,, Cao, D., Wu, and Hou, T. Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor- based and graph- based models. Journal of cheminformatics, 13(1):1-23, 2021.
[26] Jorio, A., Dresselhaus, G., and Dresselhaus, M. S. Carbon nanotubes: advanced topics the synthesis, structure, properties and applications, volume 111. Springer, 2008.
[27] Kidalov, S. V. and Shakhov, F. M. Thermal conductivity of diamond composites. Materials, 2 (4):2467-2495, 2009.
[28] Kondor, R., Lin, Z., and Trivedi, S. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 31, 2018.
[29] Kong, S., Ricci, F., Guevarra, D., Neaton, J. B., Gomes, C. P., and Gregoire, J. M. Density of states prediction for materials discovery via contrastive learning from probabilistic embeddings. Nature communications, 13(1):1-12, 2022.
[30] Kumagai, M., Ando, Y., Tanaka, A., Tsuda, K., Katsura, Y., and Kurosaki, K. Effects of data bias on machine-learning- beannn---eaaneeaabeeeddddee based material discovery using experimental property data. Science and Technology of Advanced Materials: Methods, 2(1):302-309 2022.
[31] Lee, J., Seko, A., Shitara, K., Nakayama, K., and Tanaka, I. Prediction model of band gap for inorganic compounds by combination of density functional theory calculations and machine learning techniques. Physicul Review B, 93(11):115144 2016.
[32] Lee, J., Kim, S., Hyun, D., Lee, N., Kim, Y., and Park, C. Deep single-cell ma-seq data clustering with graph prototypical contrastive learning. Bioinformatics, btad)3, 2023.
[33] Lee, N., Hyun, D., Na, G. S., Kim, S., Lee, J., and Park, C. Conditional graph information bottleneck for molecular relational learning. arXiv preprint arXiv:2305.01520, 2023.
[34] Lee, N., Yoon, K., Na, G. S., Kim, S., and Park, C. Shift-robust molecular relational learning with causal substructure. arXiv preprint arXiv:2305.1845/ 2023.
12

---

[48] Stokes, J. M., Yang, K., Swanson, K., Jin, W., Cubillos-Ruiz, A., Donghia, N. M., MacNair, Z., et al. A deep learning approach to [35] Li, K. DeCost, B., Choudhary, K. Greenwood, M., and Hattrick- Simpers, J. A critical examination of robustness and generalizability of machine learning prediction of materials properties. npj Computational Materials, 9(1): 2023.
[36] Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J,, and Chang, K.-W Visualbert: simple and performant baseline for vision and language. arXiv preprini
2019.
[37] Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
[38] Lin, T.-Y., Cui, Y., Belongie, S., and Hays, J. Learning deep representations for ground-to- aerial geolocalization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5007-5015, 2015.
[39] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H.,, and Neubig, G. Pre-train, prompt, and predict: systematic survey of prompting methods natural language processing, ACM Computing Surveys, 55(9):1-35, 2023.
[40] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1977 2019.
[41] Masoudnia, S. and Ebrahimpour, R. Mixture of experts: a literature survey, The Artificial Intelligence Review, 42(2):275, 2014.
[42] Petretto, G., Dwaraknath, S., PC Miranda, H., Winston, D., Giantomassi, M., Van Setten, M. J.. Gonze, X., Persson, K. A., Hautier, G., and Rignanese, G.-M. High-throughput density- functional perturbation theory phonons for inorganic materials. Scientific data, 5(1):1-12, 2018.
[43] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C.,, Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
[44] Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.0766 007676, 2020.
[45] Shandiz, M. A. and Gauvin, R. Application of machine learning methods for the prediction of crystal system of cathode materials in lithium-ion batteries. Computational Materials Science, 117:270-278, 2016.
[46] Sim, E., Song, S., and Burke, K. Quantifying density errors in dft. The journal of physical chemistry letters, 9(22):6385-639 2018.
[47] Singh, A. K., Montoya, J. H., Gregoire, J. M., and Persson, K. A. Robust and synthesizable photocatalysts for co2 reduction: data-driven materials discovery. Nature communications, 10 (1):1-9, 2019.
C. R., French, S., Carfrae, L. A., antibiotic discovery. Cell, 180(4):688-70)0 2020.
[49] Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, Tensor field networks: Rotation-ant translation 08219, 2018
translation-cquiiarian equivariant neural networks for 3d point clouds. arXiv preprint arXiv: 11..11822.88 1802.0821,
[50] Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceedings the conference. Association for Computational Linguistics. Meeting, volume 2019, pp. 6558. NIH Public Access, 2019.
[51] Tu, Z.,, Liu, Y., Lu, 7., Liu, X., and Li, H. Context gates for neural machine translation. Transactions of the Association for Computational Linguistics, 5:87-99, 2017.
13

---

[52] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N.. Kaiser, L., and Polosukhin, Attention is all you need. Advances neural information processing systems, 30, 2017.
[53] Wang, G., Lu, Y., Cui, L., T., Florencio. D., and Zhang, C. A simple yet effective learnable positional encoding method for improving document transformer model. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pp. 453-463, 2022.
[54] Wang, L.., Li, Y., and Lazebnik, S. Learning deep structure- preeturviin image-ter embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5005- 5013, 2016.
[55] Wang, L., Tu, Z., Way, A., and Liu, Q. Exploiting cross-sentence context for neural machine translation. arXiv preprint arXiv: trXiv::704.04347 2017.
[56) Ward, L.,, Agrawal, A., Choudhary, A., and Wolverton, C. A general-purposs machine learning framework for predicting properties of inorganic materials. npj Computational Materials, 2(1):
1-7, 2016.
[57] Wei, J., Chu, X., Sun, X.-Y,, Xu, K., Deng, H.-X., Chen, J., Wei, Z., and Lei, M. Machine learning in matcrials science. InfoMat, 1(3):388-333 377338885,,000 2019.
[58] Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. S. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
[59] Wissler, M. Graphite and carbon powders for electrochemical applications. Journal of power sources, 156(2):142-155)) 2006.
[60] Wu, Y.-J., Fang, L., and Xu, Y. Predicting interfacial thermal resistance by machine learning. npj Computational Materials, 5(1):1-8, 2019.
[61] Xie, T. and Grossman, J. C. Crystal graph convolutional neural networks for accurate and interpretable prediction of material properties. Physical review letters, 120(14): (10(:1):44)) 145301, 2018.
[62] Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: a survey. arXiv preprint arXiv:2206606440 2022.
[63] Yan, K., Liu, Y., Lin, Y., and S. Periodic graph transformers for crystal material property prediction. Advances in Neural Information Processing Systems, 115506666000,, 2022.
[64] Yan, Q., Yu, J.. Suram, S. K., Zhou, L., Shinde, A., Newhouse, P. F., Chen, W., Li, G., Persson, K. A., Gregoire, M., et al. Solar fuels photoanode materials discovery by integrating high- throughput theory and experiment. Proceedings of the National Academy of Sciences, 114(12); 2017.
[65] Yao, S. and Wan, X. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 4346 4350, 2020.
[66] Zhang, C., Zhang, M., Zhang, S., Jin, D., Zhou, Q. Cai, Z., Zhao, H., Liu, X., and Liu, Z. Delving deep into the generalization of vision transformers under distribution shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7277-7286, 2022.
[67] Zhong, X., Gallagher, B., Liu, S., Kailkhura, B., Hiszpanski, A., and Han, Explainable machine learning in materials science. npj Computational Materials, 8(1):1-19, 2022.
[68] Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for laniunnl models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[69] Zhuo, Y., Mansouri Tehrani, A., and Brgoch, J. Predicting the band gaps of inorganic solids by machine learning. The journal of physical chemistry letters, 167):166-11673. 2018.
14

---

Supplementary Material for Density of States Prediction of Crystalline Mateirals via Prompt-guided Multi-Modal Transformer
A
Datasets
this section, we provide further details on the dataset used for experiments.
A.1 Phonon DOS
We use the Phonon DOS dataset following the instructions of the official Github repository of previous work (9) This dataset contains 1,522 crystalline materials whose phonon DOS is calculated from density functional perturbation theory (DFPT) by a previous work [42]. Since the provided dataset does not contain crystal system information, additionally collect the information based on the Materials Project (MP) website on the given each material's unique ID (MP-id).
A.2 Electron DOS
We also use Electron DOS dataset that contains 38,889 crystalline materials. The Electron DOS dataset consists of the materials and their electron DOS information that is collected from the MP website 4 Among the collected data, we exclude the materials that are tagged to include magnetism because the DOS of magnetism materials is not accurate be directly used for training machine learning models P3] We consider an energy grid of 201 points ranging from -5 to 5 eV with respect the band edges with 50 meV intervals and the Fermi energy is all set 0 eV this energy grid. Moreover, we normalize the DOS of each material to be in the range between 0 and 1. That is, the maximum and minimum value for each DOS and 0, respectively, for all materials. Moreover, we smooth the DOS values with the Savitzky-Golay filter with the window size of 17 and polyorder of using scipy library following a previous work
A.3 Data Statistics Electron DOS dataset OOD scenarios
As described in the main manuscript, we further evaluate the model performance in two out-of- distribution scenarios: Scenario 1: regarding the number of atom species, and Scenario 2: regarding the crystal systems. We provide detailed statistics the number of crystalline materials for each scenario in Table and T
Table 5: The number of crystals according to the number of atom species (Scenario 1).
Unary
Binary
Ternary
Quaternary
Quinary
Senary
Septenary
Total
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Materials
386
9,034
21,794
5,612
1.750
279
34
38,889
Table 6: The number of crystals according different crystal systems (Scenario 2).
Cubic
Hexagonal
Tetragonal
Trigonal
Orthrroobiic
Monoelinie
Trielinic
Total
M Materials
8,385
3,983
5,772
3,964
8,108
6,576
2,101
38,889
http:://ggtaaa.co///aatttttttttt
15

---

B Evaluation Protocol
Phonon DOS. As described in the main manuscript, we evaluate the model performance based on the data splits given in a previous work [9].
On the other hand, for the Electron DOS dataset, we use different dataset split strategies for each scenario. For the in-distribution setting, we randomly split the dataset into train/valid/test of 80/10/10% On the other hand, for the out-of-distribution setting, we split the dataset regarding the structure of the crystals. For both scenarios, we generate training sets with simple crystal structures and a valid/test set with more complex erystal structures, because is crucial to transfer the knowledge obtained from simple crystal structures to that from complex structures in real-world materials science, More specifically, in the scenario 1 (different number of atom species, i..., # Atom species in Table 2). we use Binary and Ternary materials training data and Unary, Quaternary, and Quinary materials as valid and test data. In the case of Unary, we exclude it from training data despite its simplicity due to the observed difficulty of the structure, will be discussed in Section E. In the scenario (different crystal systems. i.e., Crystal System in Table 2). we use Cubic. Hexagonal, Tetragonal, Trigonal, and Orthorhombic crystal systems as training set and Monoclinic and Triclinic as valid and test set. In this scenario, where no prompt is available for unseen crystal systems, we employ the mean-pooled representations of the trained prompts during testing, i.e., for the Monoclinic and Trielinic crystal systems. Please refer to Table 5 and Table for for detailed statisties of crystals in each scenario.
Electron DOS.
Physical Properties. addition to evaluating the accuracy of the model's predictions of the DOS, is crucial to assess the physical meaningfulness of the predicted DOS for real-world applications. To assess the physical meaningfulness of the predicted DOS, we utilize the predicted DOS to estimate a range of important material properties. Specifically, we evaluate three materials' properties: the bulk modulus for phonon DOS, and the band gap and Fermi energy for electron DOS (Table (.).
Bulk Modulus is thermody namic quantity measuring the resistance of substance to compression. provides measure of the material's ability to withstand changes in volume under applied pressure. In the context of elastic properties, the bulk modulus serves as a descriptor, as it indicates how well a material can recover its original volume after being subjected to compression.
Another property we focus on is the Band G which refers to the energy range in a material where no electronic states exist. It represents the energy difference between the top of the valence band and the bottom of the conduction band in insulators and semiconductors. Functional inorganic materials, such as those used in applications like LEDs, transistors, photovoltaics, or scintillators, require comprehensive understanding of their band gap [69]. By accurately predicting the band gap based on DOS, we can accelerate the development new materials for wide range of applications.
Additionally, we predict the Fermi Energy which represents the highest energy level occupied by electrons absolute zero temperature (OK). It can be used to determine the electrical and thermal characteristics of materials.
C Implementation Details
In this section, we provide implementation details of DOSTransformer.
Graph Neural Networks. Our graph neural networks consist of two parts, i.e., encoder and processor. Encoder learns the initial representation of atoms and bonds, while the processor learns to pass the messages across the crystal structure. More formally, given atom i; and the bond ezy between atom i and vy. node encoder énode and edge encoder Pedge outputs initial representations of atom i and bond Eij as follows:
h Ónode (Xi), by = dedge (Bi)),
where X is the atom feature matrix whose i-th row indicates the input feature of atom B g is the bond feature tensor with F, features for each bond. With the initial representations
http://wwwwieeeeae.ooaoeiiiieeie wtw.a///...rrr//ii/////ue http:///aawwwikiaaaaaooa/asaaa/u modulus
16

---

of atoms and bonds, the processor learns to pass messages across the crystal structure and update atoms and bonds representations follows:
bij Vedge (h/, hy. bly); hot hode (h/.
JEN(()
where (i) is the neighboring atoms of atom ve is two-layer MI.P with non-linearity, and 1 O..... L'. Note that h." is equivalent the i-th row of the atom embedding matrix H in Equation
Model Training. In all our experiments, all the tasks, we train the model for 1,000 epochs with early stopping applied if the best validation loss does not change for 50 consecutive epochs.
the AdamW optimizer for model optimization. For
Hyperparameter Tuning, Detailed hyperparameter specifications are given in Table For the hyperparameters in DOS DOSTransformer,
tune them in certain ranges follows: number of message passing layers in GNN L' in (2, 3, 4), number of cross-attention layers L1, L3 in (2, 3, 4}. number of self-attention layers L2 in (2, 4), hidden dimension d in (64, 128, 256), learning rate 1) in (0.0001, 0.0005, 0.001 ) and batch size in { 1, 4, 8). We use the sum pooling to obtain the crystalline material i's representation, i.e., g.. We report the test performance when the performance the validation set gives the best result.
Table 7: Hyperparameter specifiiatoo of DOSTranstormer.
In-Distribution
Duttof-Distribution
Hyperparameters
Phonon DOS
Electron DOS
Atom Species
Crystal Systems
# Message Passing Layers (L')
3
3
3
# Cross-Attention Layers (Ta))
2
# Self-Attention
2
Layers (L2)
Cross-AAtention Layers (Ls)
2
2
2
Hidden Dim. (d)
256
256
256
256
Learning Rate (r)
0.0001
0.0001
0.0001
0.0001
Batch Size (B)
8
D Methods Compared
In this section, we provide further details on the methods that are compared with DOSTransformeri in our experiments.
MLP. We first encode the atoms in a crystalline material with an MLP. Then, we obtain the repre- sentation of material i.e., ga. by pooling the representations of its constituent atoms. With the material representation, predict DOS with MLP predictor d' i.e., Y d' d'(g.) where o' Rd 201
On the other hand, when we incorporate energy embeddings into the MLP, we predict DOS for each energy j with learnable energy embedding E and obtained material representation g, i.e.,
Y = 2 (E)) g,), where á 2d R a parameterized MLP.
Graph Network. We first encode the atoms in a crystalline material with graph network 4 As done for MIP, we obtain the representation of material i, i.e., gi, by sum pooling the representations of its constituent atoms. With the material representation, predict the DOS with an MLP predictor,
17

---

i... d'=)( where d' R + R2 Note that the only difference with MLP is that the atom representations are obtained through the message passing scheme. We also compare the vanilla graph network that incorporates the energy information as we have done in ML..
E3NN. For E3NN 19, we use the official code published by the author which implements equivariant neural networks with E3NN python library 10 By learning the equivariance, the model can generate high-quality representations with a small number of training materials. After obtaining the crystalline material representation gi, all other procedures have been done in the same manner with other baseline models, i.e, MLP and Graph Network.
E
Additional Experiments
E.1
Model Performance Analysis on Out-of-Distribution Scenarios
In this section, we conduct comprehensive analysis of the model's predictions in the out-of- distribution scenarios presented in Table 2 In T we evaluate the performance of the model for each type of material, providing detailed insights into its predictive capabilities. We have following observations: 1) We observe that DOSTransformer consistently outperforms in both out-of-distribution scenarios, which demonstrates the superiority of DOSTransformer. 2) The performance of all the compared models generally degrades as the crystal structure gets more complex. That is, models perform worse in Quinary crystals than in Quarternary crystals, and worse in Triclinic crystals than in Monoclinic crystals. 3) On the other hand, it is not the case in Unary crystal. This is because in Unary crystal only one type of atom repeatedly appears in the crystal structure, which cannot give enough information to the model. However, makes comparably accurate predictions the Unary materials by modeling the complex relationship between the atoms and various energy levels.
also
Table 8: Model performance in
scenarios.
Atom Species
Crystal System
Model
Unary
Quarternary
Quinary
Monoclinic
Trielinic
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Energy X
0.578
0.180
0.500
0.153
0.673
0.182
0.370
0.132
0.470
0.146
MLP
(0.05)
(0.001)
(0.001)
(0.004)
(0.02)
(0.017)
(0.003)
(0,021)
(0:005)
Graph Network
0.485
0.165
0.443
0.141
0.620
0.170
0.376
0.127
0.504
0.147
(0001)
(0.003)
(0.03)
(0.00))
(0.007)
(0.003)
(0.005)
(0.001)
(0.007)
(0.001)
0.565
0.167
0.486
0.145
0.708
0.179
0.393
0.129
0,510
0.148
E3NN
(0.05)
(0.002)
(0.001)
(0.000)
(0.013)
(0.002)
(0.003)
(0.001)
(0.006)
(0.002)
Energy
0.597
0.183
0.498
0.151
0,684
0.178
0.367
0.130
0.468
0.144
ML.P
(0.024)
(0.005)
(0.003)
(0.002)
(0.015)
(0.002)
(0.012)
(0.002)
(0.020)
(0.003)
0.471
0.160
0.416
0.137
0.571
0.165
0.359
0.125
0.476
0.144
Graph Network
(0.08)
(0.005)
(0.02)
(0.002)
(0.005)
(0.00))
(0/01)
(0.004)
(0.008)
(0:004)
0.567
0.166
0.481
0.143
0.689
0.175
0.393
0.128
0.516
0.147
E3NN
(0.021)
(0.003)
(0.008)
(0.000)
(0.020)
(0.002)
(0.006)
(0.001)
(0.006)
(0.001)
DOSTransformer
0.417
0.145
0.413
0.128
0.570
0.157
0.343
0.117
0.466
0.137
(0.02))
(0.003)
(0.010)
(0.002)
(0.010)
(0.001)
(0.005)
(0.001)
(0.007)
(0.001)
E.2
Injecting Crystal System Information to Baseline Methods
In this section, we adopt our prompt-based crystal system information injection procedure to the baseline methods. We examine two approaches for injecting the information: 1) injecting the information into the input atoms (i.e., Position 1), and 2) injecting it before the DOS prediction layer (i.e., Position 2). In Table we have the following observations: 1) Compared to Table all bascline models benefit from using crystal system information. This demonstrates the importance of
htttpe///gtaab.cc/////aaaaaaaa/p http:r/iiiiititcccoooooosstraaaa tutorial
e3nn httpp//////oooooooooa//////////a html
18

---

utiliziig crystal structural systems information, which has been overlooked in previous works. 2) However, DOSTransformen still outperforms all baseline methods with crystal system information (See DOS ToSTransformer in Table D verifying the importance of an elaborate design of crystal system injection procedure. To more specific, we notice a relatively significant performance gap between DOSTransformer and the best baseline model in Electron DOS, which comprises a broader range of crystalline materials than Phonon DOS. This finding highlights the importance of an intricate crystal system injection procedure when striving learn the DOS of diverse crystalline materials.
Table 9: Baseline model performance with crystal structural system prompts.
Phonon DOS
Electron DOS
Model
Position
Position
Position 1
Position
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Energy X
0.308
0.105
0.324
0.109
0.343
0.128
0.340
0.127
MLP
(0.013)
(0,001)
(0.01))
(0.002)
(0.008)
(0.001)
(0.009)
00.0031
0.260
0.097
0.244
0.095
0,260
0.102
0.260
0.103
Graph Network
(0.008)
(0,000)
(0.023)
(0.001)
(0.011)
(0.001)
(0.006)
(0.002)
0.210
0.077
0.209
0.079
0.292
0.108
0.299
0.110
E3NN
(0.007)
(0,007)
(0.000)
(0.001)
(0.007)
(0.001)
(0.007)
(0.001)
Energy
0.251
0.100
0.245
0.098
0.332
0.125
0.336
0.127
MLP
(0.000)
(0.000)
(0.004)
(0.001)
(0.003)
(0.001)
(0.007)
(0.002)
Graph Network
0.230
0.094
0.224
0.091
0.234
0.097
0.230
0.093
(0,009)
(0,002)
(0.009)
(0.002)
(0.001)
(0.004)
(0.009)
0.194
0.073
0.190
0.073
0,286
0. 0.108
0.290
0.109
E3NN
(0.004)
(0.000)
(0.000)
(0.001)
(0.002)
(0.000
(0.003)
(0.000)
E.3 Various Training Data Ratio for Fine-Tuning
In this section, we additionally provide experimental results on var- ious ratios of training data for fine-tuning in 3 That is, instead of sampling 10% of training data from the test set used in OOD sce- narios in Section [.3 we try various sampling ratios. i.c., 5%, 10%, 15%, and 20%. from the test set. We have the following observations: 1) We notice a significant performance disparity between the "Only Prompt" and "All" approaches, particularly when the training dataset is limited. This phenomenon can be attributed to the challenge of overfitting when fine-tuning the entire model on a small subset of materials, as discussed in Section 5.3 2) Conversely, as the training data increases, we observe that the difference in performance be- tween the "Only Prompt" and "All" methods diminishes. This is due to the enough amount of training data allowing for the adjustment all model parameters, consistent with the analysis discussed in Section to note that the existing DFT calculation-basse databases suffer from a highly biased distribution, which limits their coverage of different materials. This limitation emphasizes the significance of achieving good performance even with a small subset of training data. Therefore, we argue that the application of prompt tuning enhances the real-world applicability of DOSTransformer.
0:38
All
Only Prompt
0.37
MSE
0.36
0.35
0.34
5
10
15
20
Ratio (%)
Figure 6: Various training data ratios for fine-tuning.
5.3 However, it is important
E,4 Comparison with Crystal Neural Network
We conducted additional experiments on the state-of-the-arr crystal neural networks, i.e., ALIGNN [10 and Matformer 633. by adjusting the output dimensions of these models (51 for Phonon DOS and 201 for Electron DOS) in Table Although both ALIGNN and Matformer exhibited commendable performance, we observe that DOSTransformer consistently surpassed them significantly. These observations again demonstrate the importance of considering energy levels for accurate DOS prediction.
19

---

Table 10: Comparison to crystal neural networks.
Phonon DOS
Electron DOS
Model
MSE
MAE
R²
MSE
MAE
R²
0.204
0.085
0.717
0.270
0.108
0.600
ALIGNN
(0005)
(0.001)
(0.007)
(0.04)
(0001)
(0.001)
0.198
0,084
0.724
0.268
0.104
0.601
Matformer
(0.08)
(0,002)
(0.013)
(0.04)
(0.00)
(0.002)
DOS DOTrransformer
0.191
0.071
0.733
0.221
0.089
0.679
(0.003)
(0.002)
(0.004
(0.006)
(0001)
(0.000
E.5 Model Training and Inference Time
In this section, to verify the efficiency of DOS Trannssforrrr we compare the training and inference time of the baseline methods in Table I1 We observe that DOSTransforman requires longer training time per epoch on the Phonon DOS dataset compared to E3NN, which can be attributed to the two forward passes (i.e., system and global energy embeddings) during the training procedure. However, when it comes to the Electron DOS dataset,, DOSTransformer demonstrates shorter training time per epoch compared to E3NN. This is because the Electron DOS dataset has complex crystal structures, requiring more time for E3NN to learn equivariant representations. Furthermore, in terms of inference time, DoSTransformer demonstrates significantly faster computation per epoch compared to E3NN, particularly the Electron DOS dataset. This is because we only utilize system prediction without global prediction during inference. As many predictive ML models are used for high-throughi screening material discovery, inference time is critical factor for ML models in materials science, demonstrating the practicality of DOS Trannfsffrrerr in real-world applications.
Table Training and inference time per epoch for each dataset (sec/epoch).
Training
Inference
Model
Phonon DOS
Electron DOS
Phonon DOS
Electron DOS
Energy X
MLP
4.10
23.52
1.51
3,00
Graph Network
16.17
59.74
1.95
3.88
E3NN
21.21
141.02
3.72
9.49
Energy /
MLP
4.67
27.88
3.10
Graph Network
17.45
66.83
2.16
4.28
ESNN
24.12
152.80
3.92
10.21
Trr
39.17
F
Broader Impacts
Potential Positive Scientific Impacts. In this work, propose DOSTransformme which is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural system, demonstrating its applicability in real-world scenarios. For example, transferring the knowledge obtained from simple structured materials to complex structured materials is crucial because DFT calculation- based databases cover limited types of materials or structural archetypes. Therefore, we believe DOSTransformer has broad impacts on various fields materials science.
Potential Negative Societal Impacts. This work explores the automation process for materials science without wet lab experiments. However, it is important acknowledge that the industry, there are skilled professionals dedicated to conducting such experiments for materials science.
20

---

Therefore, it is important to proactively address these concerns by encouraging collaboration between automated methods and human experts.
G
Pseudo Code
Algorithm
shows the pseudocode of DOS TrSnrfssformer.
Algorithm 1: Pscudocode of DOSTranstormer.
Input: An input erystalline material 9 (X, A), Ground truth DOS Y. Number of attention layers
L1, L2, La. Initialized energy embeddings Initialized crystal system prompts P.
GNN(X, A)
E +- Dposs-Attention(H. H, E, L1)
g Sum (ooling(H)
Eglab (E'1
E.ogiob
Φ1 (Ealal)
El2-slob Seee-AAteenionnEE-aaa L2) // Global Self-Attention
E Llagoo Pross-AttentionEEE E/2-sloh L3)
Y Opred (E sa.gla))))
Calab RMSE(Ý, Y)
g (E [|
Eo.ay* &z (EE"")
12 ELs.oys + Self Seef-AttentionEEe.ovv I2) // System Self-Attentin
E Ela Cross-Attention(H., Blaeev (aoe),,L)) La)
Y Opred (BL3.
La RMSE(Y, Y)
Ctotat +- Lolob + 8. L // Calculate total loss
17 Function Cross-Attention (H, L):
18
for = 1,2, L do
E' < Softmax E'''H"
19
20
return E
21
Function Selt-Attention E L):
22
23
for = 1,2, ...,. L do
E" + Soltmax( B
24
EP
25
end
return BL
26
21


Reviews:
[{"review_id": "cEKCpyqelH", "reviewer": "Reviewer_9USD", "summary": "This work proposes a transformer model to predict the density of states of given materials. The model takes in all energy levels that are desired for the Density of States calculation as a 'prompt' and information about the materials structure, the output is a prediction of the DOS at the given energy levels.", "questions": "- As I understand it one of the tasks in this work is to predict the output of a function (Density of States) on a fixed grid of energy levels. Are there other examples in the literature where transformers are used to predict the outputs of a mathematical function, and the fixed grid of inputs is provided as a prompt for the model? If there are, it would be great to learn about them in the background section.\n\n- Because the DOS is predicted over a finite grid, I am curious about the smoothness of the DOSTransformer model between finite grid points. If the DOSTransformer model was to be evaluated at energy levels between the provided finite points in the prompt, are the predictions smooth where you'd expect the function to be smooth? ", "limitations": "- More description should be provided about the datasets used. How many different elements/ types of materials were included int he PhononDOS and ElectronDOS provided?", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The DOSTransformer is evaluated on both in-domain structures and out of domain structures against relevant baseline models and previously reported models. \n+ The DOSTransformer model design makes use of domain specific knowledge, such as the crystal structure type and the energy levels.\n+ Authors demonstrate that the task of predicting DOS is heavily added by providing energy levels for targeted prediction via relevant ablation studies\n+ The model is clearly described. I was able to understand all the components of the proposed model.", "weaknesses": "- More could be done to describe the task of predicting DOS for downstream applications, especially for a non-materials science audience. How are the DOS outputs for predicting bandgap energy / electrical conductivity? How many grid points are needed to gain an 'accurate enough' prediction of these properties? What is kind of error in the DOS prediction is reasonable? Are there certain energy levels where it is more critical to predict the DOS correctly?\n\n- I would appreciate some more statistics of the datasets used (PhononDOS and ElectronDOS) in the main text (e.g. # atom types, # crystal types).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments on our work. We are more than willing to address each of the specific weaknesses and questions in a detailed manner.\n\n**W1 (Downstream tasks of DOS prediction).**\n- How are the DOS outputs for predicting bandgap energy / electrical conductivity?\n\nDOS serves as a fundamental representation of the electronic density in an atomic system across various energy states. As the electronic density directly influences the electric charge and physical energy of the atoms, DOS provides valuable insights into material properties, including band gap and electrical conductivity. For instance, band gap, a crucial factor in determining the material's chemical applications, can be derived by examining the highest and lowest electronic densities in the DOS.\n\n- How many grid points are needed to gain an 'accurate enough' prediction of these properties?\n\nThe electron distributions vary for each material, and the number of required grid points depends on these differences. Due to the variations, we adopted the DFT configurations from the Materials Project database, one of the most extensive public databases in the field of materials science.\n\n- What kind of error in the DOS prediction is reasonable?\n\nWe mainly compare DOSTransformer to baselines in terms of MSE and MAE as have done in previous works [1]. In addition to the metrics, we further introduced R2, which is crucial in evaluating the regression models, by indicating the proportion of the variance in the dependent variable (i.e., DOS) that is predictable from independent variables (i.e., crystalline materials and energies).\n\nMoreover, in terms of physics, the density peaks within the DOS carry significance beyond the entirety of the density sequence across energy states. In other words, it's important to predict the energy level at which the electron density is the highest. To address this, we introduce the absolute error of peak positions as an extra measure to evaluate the model's prediction accuracy. You can find this in Table 1 in the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv) .\n\nIn the table, we observe that DOSTransformer provides more precise predictions for the peak points in the DOS, showcasing the practical value of the predicted DOS.\n\n[1] CHEN, Zhantao, et al. Direct prediction of phonon density of states with Euclidean neural networks. Advanced Science, 2021, 8.12: 2004214.\n\n- Are there certain energy levels where it is more critical to predict the DOS correctly?\n\nImportant energy levels are different for each material. However, the energy levels of the highest electronic densities are important because they determine many energy-related properties of the materials. For this reason, we evaluated how accurately the predicted peak points in the DOS match the actual peak points in the real DOS by comparing them in terms of MSE in Table 1 in the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv) .\n\n\n**W2 (Dataset statistics).**\n\nIn addition to Electron DOS data statistics provided in Appendix A.3., we provide more details in the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv) .\n\n\n**Q1 (Literature survey).**\n\nFirstly, we would like to provide clear definitions of the terminology used in this paper. We refer to the embeddings for each fixed grid of energy as \"energy embeddings,\" and the prompts for each crystal system as \"crystal system prompts.\" When conceptualizing each energy value as the position of a word in a sentence, energy embeddings can be analogous to the positional encoding used in traditional transformers.\n\nIn the original proposal of the Transformer architecture, a sinusoidal function was suggested as a method for positional encoding. However, the sinusoidal function may have limitations in terms of learnability and flexibility, which can affect its effectiveness [1].\n\nTo address this issue, most pre-trained language models [2, 3] employ learnable vector embeddings as positional representations. Additionally, [1] enhances positional encoding by constructing a learnable fully-connected feed-forward sinusoidal positional encoding network.\n\nWe apologize for any confusion that may have arisen as a result of the missing background section. To address this, we will make comprehensive revisions to the section by incorporating the background about positional features.\n\n[1] Guoxin Wang, et al. A Simple yet Effective Learnable Positional Encoding Method for Improving Document Transformer Model. ACL Findings 2022.\n[2] DEVLIN, Jacob, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. ACL 2019.\n[3] LIU, Yinhan, et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.\n\n\n**Q2 (Smoothness of DOS prediction).**\n\nAs pointed out by the reviewer, we transformed the continuous energy values into finite grids of energy levels through binning. This discretization process prevents the evaluation of DOS prediction models, including DOSTransformer, at energy levels between the provided finite points. However, for specific ranges of energies with desired resolutions, we have the flexibility to train the model with the newly processed data. As a result, we believe that DOSTransformer can be applied universally to accommodate different energy levels and resolutions according to specific research needs.\n\nOn the other hand, a promising avenue for DOS prediction, considering the continuous nature of energies and DOS, could be the incorporation of neural ordinary differential equations (Neural ODEs) [1]. Neural ODEs have shown effectiveness in capturing the continuous nature of sequential data, making them a potential candidate for this task. We are grateful for the insightful feedback, which illuminates future research directions!\n\n[1] Chen, Ricky TQ, et al. \"Neural ordinary differential equations.\" NeurIPS 2018\n"}, {"review_id": "IqrYIJVhcj", "reviewer": "Reviewer_xyRJ", "summary": "This work proposes a transformer architecture for predicting density of states of crystalline materials for different energy levels.\nThe distribution of states is spectral property that is approximated as a function of both the material and energy levels.\nThe architectures proposed consists of a multi-modal transformer with self and cross attention layers, and a GNN for embedding.\nExperiments compare the proposed architecture with state-of-the-art E3NNs, demonstrating a decrease in MSE.", "questions": "What are the MSE values of bulk, band, and ferm. are considered useful for real-world applications?\nAre there 38,889 crystalline materials after or before filtering for non-magnetism?", "limitations": "The limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of this work include:\n1. The approach considers the crystalline material, energy levels, and structural properties.\n2. The problem and methods are well-defined, and the architectures are implemented using state-of-the-art libraries.\n3. The results include out-of-distribution experiments, ablation studies, and sensitivity analysis.", "weaknesses": "The main weaknesses of this work are that:\n1. It needs to be clarified what MSE's are required for the approach to be useful in the real world.\nFor example, in proteomics, and RMSD of less than 2-3Angstrom is considered valid compared to experimental results.\nWhat would be the MSE's required to achieve such a validation in this domain.\n2. The terminology and definition used for fine-tuning and especially prompting in this work differs significantly from the standard definition\nand should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments on our work and for acknowledging the efforts in experiments, including out-of-distribution scenarios! We are more than willing to address each of the specific weaknesses and questions in a detailed manner.\n\n**W1, Q1 (Appropriate level of MSE).**\n\nTo the best of our knowledge, in the literature of DOS prediction, deriving various physical properties based on the model-predicted DOS is a novel endeavor, making it challenging to determine a definitive threshold for the MSE value that is practically applicable in the real world.\n\nFor band gap prediction, a commonly accepted MSE value is around 0.3 ~ 0.4 [1]. However, we would like to emphasize that a direct comparison of our results with those presented in [1] is not appropriate considering our experimental setup. \n\n- **Experimental setup:** As described in Line 277-282, we used the DFT-calculated DOS as the ground-truth DOS, and trained a simple 2-layer MLP with non-linearity to predict the properties of a crystal structure. Then, we use the learned MLP weights to simply predict various physical properties given the model-predicted DOS as input. \n\nIt is important to note that the learned MLP weights themselves are not perfect, meaning that the properties derived from the model-predicted DOS using the learned MLP weights inevitably exhibit error that is accumulated due to the error in the learned MLP weights. For this reason, rather than directly comparing our results with the results reported in [1], our intention was to compare among our baselines, and see whether the various properties predicted by DOS predicted by our proposed method are relatively more accurate. Thus, in the current context, the MSE value itself might hold less significance, and it is more appropriate to consider it as an evaluation metric to compare among baselines. However, we could reduce the accumulated error by training a more advanced neural network instead of a simple MLP. We appreciate the valuable feedback that sheds light on potential paths for future research!\n\n[1] LEE, Joohwi, et al. Prediction model of band gap for inorganic compounds by combination of density functional theory calculations and machine learning techniques. Physical Review B, 2016, 93.11: 115104.\n\n**W2 (Terminology of fine-tuning and prompt-tuning).**\n\nAlthough we made efforts to clarify the concepts of fine-tuning and prompt tuning in Section 2.2, we apologize for any confusion that may have arisen. To provide further clarification, we briefly outline the distinctions between fine-tuning, discrete prompt designing, and continuous prompt tuning.\n\nIn the field of Natural Language Processing (NLP), fine-tuning was introduced to adapt pre-trained language models (LM) for specific downstream tasks. Conversely, discrete prompt designing aims to reformulate downstream tasks to resemble those encountered during the original LM training, achieved by using a textual prompt. Recently, continuous prompt tuning has been proposed, which involves prompting directly in the embedding space of the model. These prompts have their own parameters, which can be adjusted based on training data from the downstream task. [1]\n\nIn this paper, we adopt the concept of continuous prompt tuning to guide the model in understanding the structural types of crystalline materials. While a single prompt was employed for each downstream task in NLP, our approach utilizes a single prompt for each of the seven structural types of materials discussed in the paper. It is important to note that continuous prompts are utilized due to the non-trivial nature of modeling the structural type of crystalline material using human-engineered prompts.\n\nAdditionally, concerning the use of \"fine-tuning\" in Section 5.3 and Table 3, we intended it to refer to tuning on various downstream tasks. In Table 3, \"All\" denotes the model that updates all parameters for the downstream task, whereas \"Only Prompt\" refers to the model that exclusively updates the parameters of continuous prompts, which is the same as \"Continuous prompt tuning.\" We will thoroughly revise the manuscript to ensure clarity on these matters.\n\n[1] LIU, Pengfei, et al. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 2023, 55.9: 1-35.\n\n\n**Q2 (Non-magnetism?).**\n\nYes, 38,889 crystalline materials are non-magnetic materials obtained after filtering out magnetic materials. This choice was influenced by the well-known issue of calculation accuracy in the DFT-generated DOS of magnetic materials, which is considered unreliable [1, 2]. In short, we only used non-magnetic materials for training our model, since using magnetism materials for training may interfere with the model's ability to learn the DOS of non-magnetic materials.\n\n[1] Zeller, R. (2006). Spin-polarized dft calculations and magnetism. Computational Nanoscience: Do It Yourself, 31, 419-445.\n[2] Poblet, J. M., López, X., & Bo, C. (2003). Ab initio and DFT modelling of complex materials: towards the understanding of electronic and magnetic properties of polyoxometalates. Chemical Society Reviews, 32(5), 297-308.\n"}, {"review_id": "rQgRUN79rF", "reviewer": "Reviewer_pHUW", "summary": "The paper proposes a new transformer-based method DOSTransformer for predicting density of states of crystalline materials. Different from previous methods, the energy level is additionally modeled as an input modality, i.e. the model takes in material configuration and energy level as input to predict DOS(material, energy). DOSTransformer uses cross-attention layers to relate the material and energy representations. Self-attention layers are also used to integrate information across different energy levels. Learnable prompts are introduced to provide information about the crystal system type, which helps capture structure-specific interactions between material and energy. Experiments on phonon and electron DOS datasets show DOSTransformer outperforms existing methods like graph networks and E3NN, in both in-distribution and out-of-distribution scenarios. Ablations and sensitivity analysis demonstrate the benefits of the different components like energy embeddings, global/system losses and crystal system prompts.", "questions": "N.A.", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Novel problem formulation: The authors provide a new perspective on DOS prediction by treating it as a multi-modal learning problem with material and energy as inputs. This proves to work better for DOS prediction compared to prior material-only models.\n- Neural net design: The multi-modal transformer architecture with cross-attention and prompt-guided self-attention is well-suited for the problem. It allows capturing complex relationships between material and energies.\n- Empirical results: The model is evaluated extensively on two datasets — Phonon DOS and Electron DOS — for both in-distribution and out-of-distribution scenarios. The physical validity of predictions is also analyzed through derived material properties. In particular, the introduced energy embeddings are shown to be useful in reducing prediction errors.\n- Ablation studies: The contributions of different components like the dual losses and crystal system prompts are quantitatively demonstrated through ablation studies.", "weaknesses": "- Simple prompts: The crystal system prompts are relatively simple vector embeddings. More sophisticated prompt learning methods could be explored to reason about structural information of different systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments on our work and for acknowledging the novelty of our work in DOS prediction! We are more than willing to address each of the specific weaknesses and questions in a detailed manner.\n\n**W1 (Simple prompts).**\n\nAs pointed out by the reviewer, our crystal system prompts are implemented using relatively straightforward methods. We fully agree with the reviewers that we can enhance crystal system prompts in various ways. One possible direction would be integrating them with large language models (LLMs). For instance, an approach could involve initializing the system prompts using the embeddings derived from the descriptions of each crystal system.  We appreciate the valuable feedback that sheds light on potential paths for future research!\n"}, {"review_id": "nMQWKdhFwB", "reviewer": "Reviewer_E5Yg", "summary": "This paper proposes a prompt-based Transformer network for predicting the density of states of crystalline materials. The prompts are used to represent and control the energy and the additional structure information of the materials. Experiments show the proposed method can perform very well on two datasets under different settings.", "questions": "Questions are left with the weakness points.", "limitations": "The limitations are discussed and can reflect the main limitations of the proposed method –  the proposed method cannot encode the properties of both materials into a single model.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The whole system and method are motivated well and designed carefully. The model design considers the characteristics of both the task and the deep learning methods, e.g., prompt-based Transformer. \n- The experiments are carefully designed to show the benefits of the proposed methods. \n- The paper is well-written and easy to read.", "weaknesses": "- The datasets and experiments are a bit simple. The variations of the material type and structure type in the datasets are restricted. In this sense, although prompts are used to formulate the general variation of the energy and structure, the datasets contain very simple and limited variations. It is a bit limited in evaluating the potential of the proposed method and the techniques on more realistic datasets and applications. \n    - For example, the prompt is used to represent the structure. There are only seven discrete options that will be given during testing. May the model handle the structure variation without relying on the input during testing?\n- In Table 4, the prompt based method is compared with the one-hot based input. Is the one-hot binary encoding directly used to replace the embedding of the prompt? If so, it might be unfair. How does the one-hot encoding work with a fully connected linear layer (i.e., feeding the one-hot encoding as the input to the FC layer), where the weights in the FC layer are corresponding to the embeddings for each type of the system? \n    - In the worst case, the proposed prompt based method may work similarly to the linear embedding based approaches, on the simple datasets.  \n\n- The authors may analyze the learned prompts to indicate whether the leaned embeddings can reflect and are consistent with the physical characteristics of the system types or structures indicated by the prompts, such as whether some similarity and relationship between these characteristics can be reflected in the learned prompts. \n\n- It seems that fine-tuning do not improves the performance significantly, especially given the identical MAE performance comparing Table 2 and Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1.**\nAs the reviewer pointed out, using only 7 structural systems may make the dataset seem limited. However, the reason for having 7 structural systems is not a limitation of the dataset but rather a reflection of the knowledge used in crystallography when classifying the structures of materials [1, 2]. In other words, regardless of which dataset is used, there are only 7 crystal systems in the real-world, implying that we are bound to use the 7 crystal system prompts. Also, it is worth noting that the dataset used in the paper, i.e., the Materials Project database, is one of the most extensive public databases in the field of materials science.\n\nOn the other hand, in real-world scenarios, it is widely acknowledged that databases based on DFT calculations have certain limitations in coverage. These databases often concentrate on particular material types or structural archetypes, leading to biased distributions, as discussed in lines 288-290. To address this concern, we performed experiments on out-of-distribution data in Section 5.3, where both the training and test datasets consisted of materials with different crystal systems. These experiments effectively showcase the adaptability of DOSTransformer to various dataset variations, successfully alleviating the reviewer's concerns.\n\n[1] PARK, Woon Bae, et al. Classification of crystal structure using a convolutional neural network. IUCrJ, 2017.\n\n[2] YU, Rong, et al. Calculations of single-crystal elastic constants made simple. Computer physics communications, 2010.\n\n**W2.**\nYes, one-hot encoding is directly used to replace the embedding of prompts. As described in lines 207-210, crystal system information $P_k$ is concatenated with material embeddings $g_i$ and material-specific energy embedding $E_j^L$ and linearly transformed by the function $\\phi_2$.\nTherefore, the main distinction between using Prompts and One-hot vectors lies in how the information about the crystal system is incorporated into the model, either in a soft or hard manner. When using a one-hot vector to represent the crystal system, the model integrates the crystal system information in a hard manner through matrix multiplication.\n\nIn Table 4, we demonstrated that when incorporating crystal system information into “atom features”, the one-hot encoding of the system performs better than soft prompts. This is because soft prompts may hinder the learning of shared information among various crystalline materials due to their higher number of parameters. However, when incorporating the crystal system information “before the self-attention layers”, the model benefits from soft prompts. This allows the model to learn more flexible information that is advantageous for the system-specific integration of energies and crystalline materials. In conclusion, by incorporating soft prompts in appropriate locations, we were able to complete the optimal model.\n\n**W3.**\nTo begin with, our objective in employing crystal prompts is not to comprehend the correlations among crystal systems, but instead to acquire crystal system-specific interactions between a crystalline material and energy levels, aiming to enrich the expert knowledge associated with each distinct crystal system like expert models.\n\nFurthermore, within the domain of crystallography, due to the unique and distinct characteristics of each crystal system in terms of symmetry, establishing a straightforward relationship between different crystal systems is not trivial [1]. However, as per the reviewer's suggestion, we conducted specific case studies by assessing the cosine similarity among the crystal system prompts based on Figure 2 in the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv).\n\nCase 1) The correlation between the Cubic and Trigonal systems is approximately nine times stronger compared to the correlation between the Cubic and Triclinic systems. For example, consider the case of Bismuth (Bi). In ambient conditions, Bismuth exists in the Trigonal crystal system. However, when subjected to high-pressure conditions, it transforms into the (body-centered) Cubic crystal system. However, the Cubic and Triclinic systems exhibit the most significant dissimilarities in terms of symmetry, including axial lengths and angles, which makes crystal system transitions to be hard.\n\nCase 2) Furthermore, it is noticeable that the correlation between the triclinic and monoclinic systems is roughly ten times more pronounced than the correlation between the triclinic and cubic systems. Albite, a plagioclase feldspar mineral, exemplifies this scenario as its crystal symmetry can shift from triclinic to monoclinic contingent upon temperature changes. Nevertheless, due to the considerable discrepancy in symmetry between the triclinic and cubic systems compared to that between the triclinic and monoclinic systems, the likelihood of materials demonstrating both symmetries is quite low.\n\nTo sum up, despite the complexity of establishing a direct correlation between crystal systems, our observations indicate that the alignment between prompt relationships and crystallography domain knowledge holds true.\n\n[1] Thomas, J.C., et al., A. Comparing crystal structures with symmetry and geometry. npj Comput Mater 7, 164 (2021). \n\n**W4.**\nAs mentioned in lines 320-322, the performance improvement during fine-tuning is constrained due to the scarcity of data used in the process. This situation reflects real-world material sciences where existing databases relying on DFT calculations have limited coverage, often emphasizing specific material types or structural archetypes, resulting in a biased distribution. And indeed, we have conducted experiments on various training ratios for fine-tuning in Appendix E.4, revealing that as the volume of training data increases, there is a noticeable enhancement in performance during fine-tuning.\n"}, {"review_id": "TCBsI7CMAy", "reviewer": "Reviewer_pieM", "summary": "This paper propose DOSTransformer, a transformer architecture for the task of DOS prediction, that takes energy levels as input instead of predicting a list of energies for different energy levels. The performances of this proposed DOSTransformer is better than MLP, GNN, and E3NN instances they implemented. ", "questions": "As listed in above weaknesses.", "limitations": "They listed the limitations in the main paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is the first one to take energy levels as input instead of directly predict a list of energies for different energy levels.\n- Writing: clear and easy to follow. Figures are informative.", "weaknesses": "- Novelty is limited, core ablation study is missing: the major novelty beyond previous works is taking energy levels as input instead of directly predict a list of energies for different energy levels. To do this, they proposed the multi-modal module that considers energy levels. But this point is not well supported by the performance ablation studies of E3NN. Also, the ablation study for their proposed model about this point (just directly predict a list of energies) is missing.\n- Missing comparisons with previous crystal neural networks. Previous crystal neural networks including CGCNN, MEGNET, Matformer, ALIGNN, M3GNet can all directly be used for this DOS task, by just changing the output dimensionality from 1 (single scalar value prediction) to M. Just ignoring all these methods is not convincing. Comparing with these methods (at least one or two current SOTA methods) can better demonstrate the importance of taking energy levels as conditions instead of predicting a list of energies. \n- Missing implementation details and what level of irreducible representations is used in E3NN. Simple E3NN with only irreducible representations of rotation order zero or 1 is not powerful enough. Using representations with rotation order of 2 will boost the performances significantly. Given the not significant performance gains beyond e3nn, further implementation details is needed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments on our work and for recognizing that our work is the first work that integrates energy level as input! We are more than willing to address each of the specific weaknesses and questions in a detailed manner.\n\n**W1 (Limited novelty & ablation studies).**\n\n**[Regarding the novelty of the paper]**\nThis paper makes two contributions: firstly, as acknowledged by the reviewer, we adopt energy embeddings as the model input instead of directly predicting densities for different energy levels; secondly, we employ prompts to capture structural-specific interactions between crystalline materials and energy levels.\n\nAs we noted in lines 267-270, the integration of energy embeddings in E3NN, the best baseline method, does not yield significant benefits. This could be attributed to the potential mismatch between the intended purpose of E3NN and the use of energy embeddings. That is, the inclusion of energy embeddings may interfere with the proper learning of material “equivariance” in E3NN, leading to limited advantages in its case. However, it is important to highlight that energy embeddings continue to deliver consistent performance improvements for other models, such as MLP and Graph Networks, showcasing the effectiveness of considering energy levels as input, and applicability in a broader context.\n\n**[Regarding the ablation study of the proposed method]**\nUpon the reviewer's request, we performed ablation studies to explore the benefit of point-wise prediction of energies as in our proposed method compared with directly predicting a list of energies. To do so, we made 2 ablation models as follows (The experimental results are presented in Table 2 in the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv)):\n\n- Ablation 1: We removed the Transformer architecture and utilized only energy embeddings $\\mathbf{E}^{0}$, crystal representation $\\mathbf{g}_i$, and crystal system prompts $\\mathbf{P}$ as input for an MLP to predict the list of densities across different energy levels.\n\n- Ablation 2: While keeping the transformer architecture, we obtained crystal representation by pooling energy representations $\\mathbf{E}^{L_3, i}$, and then making a direct list-wise prediction with the pooled representation.\n\nWe have observed a significant drop in the model performance when excluding the transformer layers (Ablation 1), which highlights the crucial role of these layers in enabling effective point-wise interactions between atomic and energy embeddings. Furthermore, the direct list-wise prediction through pooled representation (Ablation 2) yields inferior results compared to DOSTransformer. This underscores the significance of the Transformer layer and reinforces the need to maintain its point-wise prediction mechanism for the DOS prediction task. We sincerely appreciate your insightful feedback and will make sure to incorporate these supplementary experiments into the ablation study section.\n\n**W2 (Comparison with previous crystal neural networks).**\n\nWe fully agree with the reviewer that comparing DOSTransformer to recent Crystal Neural Networks can further enhance the quality of the paper. Therefore, we conducted additional experiments on Matformer, ALIGNN, and CGCNN, by adjusting the output dimensions of these models (51 for Phonon DOS and 201 for Electron DOS), in Table 3 in the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv).\n\nAlthough both ALIGNN and Matformer exhibited commendable performance, we observe that DOSTransformer consistently surpassed them by a significant margin. These observations again demonstrate the importance of considering energy levels for accurate DOS prediction. We will make sure to include the results in the revised paper, which would undoubtedly elevate the quality of our work. We sincerely thank the reviewer for your valuable insights.\n\n**W3 (Missing implementation details of E3NN).**\n\nFirst of all, we apologize for any confusion caused by not elaborating on the implementation details on E3NN. We utilize the most recent code of E3NN [1] provided by the authors in the GitHub repository, and this implementation uses representation with a rotation order of 1. On the other hand, upon the reviewer’s request, we conducted experiments with the E3NN using a representation with a rotation order of 2 in Table 4 of the attached [PDF](https://openreview.net/forum?id=2lWh1G1W1I&noteId=YWxXBLLMtv).\nIn the experiment, we did not observe substantial performance improvement with a rotation order of 2. The limited expressiveness of its radial functions and spherical harmonics may have contributed to this result. Considering that DOS prediction is inherently more complex than other material properties due to its sequential nature, we conjecture that this inherent complexity might have hindered potential performance improvements.\n\n[1] Chen, Zhantao, et al. \"Direct prediction of phonon density of states with Euclidean neural networks.\" Advanced Science 8.12 (2021): 2004214.\n"}]

Author rebuttal:
Dear reviewers, thank you for your valuable comments on our work. We are more than willing to address each of the weaknesses and questions in detail. We also attach a PDF file that contains Figures and Tables for rebuttal.

Meta-review: The paper proposes a multi-model transformer to estimate the density of states of crystalline materials.

The paper received five reviews and all reviewers are in favor of accepting the paper. 

The reviewers acknowledge that the problem is novel and interesting, the design of the network is well-suited for the problem, and that the method is evaluated well (both in- and out-of-domain), and that it includes ablation studies and sensitivity analysis. Moreover, the paper is well written.

The reviewers also find a few minor weakness that could for the most part clear up in the rebuttal and/or can be incorporated in the final version.
Decision: Accept (poster)