Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation
Authors: Keqiang Yan
Xiner Li
Hongyi Ling
Kenna Ashen
Carl Edwards
Raymundo Arroyave
Marinka Zitnik
Heng Ji
Xiaofeng Qian
Xiaoning Qian
Shuiwang Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: tokenization of crystals
language models
materials generation
Primary area: machine_learning_for_physical_sciences

Abstract: We consider the problem of crystal materials generation using language models (LMs). A key step is to convert 3D crystal structures into 1D sequences to be processed by LMs. Prior studies used the crystallographic information framework (CIF) file stream, which fails to ensure SE(3) and periodic invariance and may not lead to unique sequence representations for a given crystal structure. Here, we propose a novel method, known as Mat2Seq, to tackle this challenge. Mat2Seq converts 3D crystal structures into 1D sequences and ensures that different mathematical descriptions of the same crystal are represented in a single unique sequence, thereby provably achieving SE(3) and periodic invariance. Experimental results show that, with language models, Mat2Seq achieves promising performance in crystal structure generation as compared with prior methods.

Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation
Keqiang Yan', Xiner Li', Hongyi Ling', Kenna Ashen', Carl Edwards', Raymundo Arróyave', Marinka Zitnik", Heng Ji² Xiaofeng Qian', Xiaoning Qian', Shuiwang Ji
Texas A&M University, College Station, TX 77843, USA
University of Illinois Urbana-Champaign. Champaign, IL 61820, USA
HHarvard University, Boston, MA 02115, USA
anggia hrreeyann,xe,,nneeeee aeeg,hongylingg kww.kaashhnnnnc rarroyave rraro www.eenngiiiiee sasiaannsjstaaaa sjie@tamu.eeeeeeeeeeeeeeeaa edu {cne2, hengj ieliiiiiiiiiiiii edu marinkaChms harvard. edu
Abstract
We consider the problem of crystal materials generation using language mod- els (L.ss). key step is to convert 3D crystal structures into ID sequences be processed by LMs. Prior studies used the crystallographic information frame- work (CIF) file stream, which fails to ensure SE() and periodic invariance and may not lead to unique sequence representations for given crystal structure. Here, we propose a novel method, known as Mat2Seq, to tackle this challenge. Mat2Seq converts 3D crystal structures into ID sequences and ensures that different mathe- matical descriptions of the same crystal are represented in a single unique sequence, thereby provably achieving SE(3) and periodic invariance. Experimental results show that, with language models, Mar2Seq achieves promising performance crystal structure generation compared with prior methods.
Introduction
Discovering crystalline materials with desired properties is valuable for advancing variety of technological sectors 2
Р. However, current materials discovery relies on trial- and-error wet-lab experimental methods that are time-consuming and expensive. Computational methods based on advanced quantum mechanical approaches such as first-principles density functional theory (DFT) have sped up this process in the last two decades. Nevertheless, the high computational cost of these methods ranging from (nn)) to O(n²), where n is number of electrons in a material system, poses significant challenge for high- throughuu screening of the infinite materials space.
Recent advances in large language models (LLMs) have showcased substantial capabilities in real- time question answering and image generation. However, the application of these models to atomistic systems, including molecular and material structures, remains relatively underexplored. Specifically, while recent studies have attempted to represent molecules and crystals using XYZ and CIF structure file formats [T] II [2], they have not adequately addressed the critical issues of uniqueness and invariance in sequence representations of 3D crystal structures, which play key roles for successful LL.M-based discovery of novel crystalline materials with discussions provided in Sec. 2.2
The concept of uniqueness [13 [4] in crystal sequences mandates a bijective relationship between crystal structures and their sequence representations. Current approaches using CIF files directly in crystal LLMs violate this uniqueness criterion, leading to significantly varied sequences for identical crystals, particularly in terms of atom ordering and fractional coordinates within the unit cell as illustrated in Figure I] This lack of unique sequence representations means that an infinite number of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CIF files can describe the same crystal, necessitating extensive augmentation during LLM training to recognize these equivalences. Without such augmentation, previous CIF file based methods [TO] essentially train crystal L.MSs using only a limited set of crystal structure snapshots.
In this work, develop framework for creating unique and complete erystal sequence representa- tions, followed by the construction of a material LLM capable of generating novel crystal structures with desired properties of interest. To accomplish this, several challenges must be addressed. First, unlike molecular structures that contain a finite number of atoms, each crystal structure consists of an infinite number of atoms through a periodic translation of a finite set of atoms in a unit cell along the direction of three lattice vectors in piree-dimensional (3D) space. Consequently, there are numerous different unit cells for the same crystal as shown in Figure [ A unique and invariant unit cell must therefore be selected for each crystal. Second, it is crucial that this unit cell can be represented in a one-dimensional (1D) sequence that maintains invariance under arbitrary rotations and ensures completeness, allowing the full reconstruction of the crystal structure from its sequence representation. To tackle these complexities, we introduce Mat2Seq, a method that systematically transforms 31) crystal structures into ID sequences. This is achieved by first identifying SO(3) equivariant unit cells and subsequently converting these into S (3) invariant sequences. By integrating these unique and complete sequences into LLMs, we develop conditional generation capabilities for novel crystal structures. Our experimental results in crystal structure prediction and crystal discovery with desired properties validate the efficacy of Mat2Seq.
2 Preliminaries and related work
2.1 Crystal structure generation
In this work, we study the problem of generating 3D crystals from scratch. Unlike small molecules, crystalline materials are defined by a unit cell which contains a set atoms repeated infinitely across three-dimensional space along three periodic lattice vectors. Following notations of ComFormer [14) each crystal structure is represented by M (A, P, L), where A = [a1, 02 ... aa, E R XR represents the da da-dimensionnl feature vectors of atoms within the unit cell, P Pp..a.... .P.l Ráxn represents the 3D Euclidean positions of these n atoms, and L = [L, L2, Ls. E 33x3 specifies three periodic lattice vectors, representing the repeating patterns of the unit cell in 3D space. The infinite structure of given crystal M (A,P, L) is formalized as P {
P, t2 tkgl2
kzl3, k1 k2, k3 E Z E Z, < S nJ, {ái/á; a,, E Z, < i S n), where P denotes the positions of atoms and their infinite repeats in the 3D space, and Å denotes the corresponding feature vectors. This work targets two primary tasks:
Crystal structure generation: Using dataset of 3D crystals {M, we aim to develop structure generative model pe to synthesize valid and stable 3D crystal structures for input compositions.
Conditional generation:
With a dataset ()M,, where sy denotes specific property of M,. we aim to establish conditional generative model po(-)s) to generate 3D crystal structures possessing the property s.
Generative models for crystal structures must address complex geometric requirements, such as ensuring periodic and unit cell 53(3) invariance [5 14 [6]. The ideal models should consider M MA(A, P. L) and its equivalents under periodic transformations or rotations and translations as identically probable [4]. CDVAE [15] achieves periodic and SE(3) invariance by encoding crystal structures into S3()) invariant latent features but faces limitations in restricting the loss function to be periodic invariant. This limitation is later rectified in SyMat [6). Beyond this, DiffCSP [17] is specifically optimized for crystal structure prediction that generates stable crystal structures for given compositions. In contrast, MatterGen [18] models (A, L. in an equivariant manner by gradually coruppiigg them into known distributions.
While the generation 3D crystal structures as 3D point clouds is well-established, applying powerful language models to this task is novel and nontrivial as discussed in Sec. 2.2 Our approach uniquely transforms infinite 3D crystal structures into SF(3) and periodic invariant sequences, demonstrating that language models can effectively generate crystal structures with high performance.

---

The original crysial siructure
Periodic transformationn
114
Lolk
Lapls
11:00
Lnd
2010 0300 10
11004 0 3224
M0 2010 3000 11001
0000 001 KM0
-14
Original
Clarge
pcriodic beurdary
Figure I: Limitations of directly using CIF files in achieving unique crystal sequence representations. This figure demonstrates variations in CIF files, either with or without symmetry control command denoted as "symprec", for the same crystal structure subjected to periodic transformations. Changes in the CIF contents highlighted in red. Periodic transformations cell structures, resulting in distinct CIF files differed by fractional coordinates, atom ordering, and lattice parameters for the same underlying crystal.
significantly alter the unit
2.2 Language model for 3D crystal structures
Enabling language models to generate 3D crystal structures necessitates transforming these struc- tures into sequence representations. Several studies have employed CIF files as sequence inputs
for this purpose 110 11 12). These approaches convert a set of 3D crystals {Mj} into a dataset of sequences, C = (C1,C2.- C...... C. J. where each C3 represents the CIF text of M,. The sequence Cj- = 02. , of length n, are composed of tokens c₁ from a predefined vocab- ulary V. Autoregressive language models are either trained [10 [2] or fine-tuned [ to encode these sequences by maximizing the conditional probabilities of each token given its predecessors, P(C] = I p(cyle1 C)- with the objective to maximize the probabilities of the dataset p(C IT == p(C;;)) 0). During generation, novel crystal sequences are produced from p(0) in an autoregressive manner.
However, ideal crystal generative models should consider M = (A, L, and its equivalents under periodic transformations or rotations and translations as identically probable, a criterion unmet by current models due to the variability in CIF text descriptions for the same crystal. For example, there are different sequence representations C. = {c1, C2 Cn;) and C² = d6. ... Cx:) for the
same crystal, yet ideally, I,;= pícyle : 1; = II[ p(C/lc () should hold. Despite being highly non-trivial, this is overlooked by previous studies. This issue is illustrated through the failure cases of baF-based crystal language models [10] [T] [2] in Figure [
using CIF files as input violate invariance and generate multitude of different sequence representations for the same crystal. Consequently, these models fail to consistently recognize M = (A, P. L) and its equivalents under periodic transformations including shifting periodic boundaries and changing lattice vectors identically probable. While intensive data augmentation can mitigate this issue, significantly burdens the training process of language models. In contrast, our Mat2Seq approach maps all equivalent crystals to unique sequence, naturally ensuring that all equivalents are considered equally probable without the need for data augmentation.
Differences with previous works. It is crucial to recognize that prior works 10
3 Tokenization of 3D crystal materials
In this section, we demonstrate Mat2Seq for transforming 3D crystal structures into 1D sequences that adhere to the principles of SE))) invariance, periodic invariance, and completeness. We first discuss requirements for ideal crystal sequence representations in Sec. 3.1 and provide formal definitions of unit cell F3)) invariance, periodic invariance, and completeness. We then show how to design
crystal sequence representations to achieve these requirements in Sec. 3.2 and 3.3 with the pipeline
3

---

3D crysial strccure
Deternine lariic vectors
Detennine the primitive unit cell
Mat2Seq sequence
211
Mags
Laac 6-50 1 993)
I 145 IKGl 7. 90
S 24
Ag 1228346 EIKE BURE Hy. 6 1812H66799/0000
(0,0)
Figure 2: The pipeline of Mat2Seq that converts 3D erystal structures into unique erystal sequences. Mat2Seq first determines SO(3) equivariant and periodic invariant lattice vectors using Niggli cell reduction [1). then determines the primitive unit cell. After that, Mat2Seq converts the determined (O(3) equivariant and periodic invariant primitive cells into SE(3) and periodic invariant sequences.
shown in Figure 2 We then provide property proofs of Mat2Seq in Sec. 3.4 and demonstrate the
combination of Mat2Seq and language models for crystal structure generation Sec. 3.5
3.1 Requirements for ideal crystal sequence representations
Uniqueness. Ideal crystal sequences need satisfy uniqueness which ensures deterministic mapping of a crystal structure M = (A,P,L L) and its equivalents, as transformed by periodic and SE(3) operations.
singular, unique sequence. However, to the best of our knowledge, none of achieves uniqueness for crystal sequence representations.
previous studies
I0
When sequence inherently satisfies this criterion, it guarantees that all structural equivalents recognized as equally probable by language models, thus eliminating the need for any manual adjustments for language models extensive augmentation during the training phase. Following ComFormer [4], we further define the necessary invariances for ideal sequence representations.
Definition (Unit Cell SF((3 Invariance). A function f (A,P,L) + X' is unit cell SF(3)
invariant if. for any rotation transformation R R3x3 R and translation transformation
DER R3, we have (A,,P,L) = f(A,RD RP + b, RL).
Definition 2 (Periodic Invariance). A function f f J.P.LL) - X is periodic invariant if. for any possible minimum unit cell representations M' A.P.' L') representing given infinite crystal structure A), we have f(A. P. ('A,P'',').
This definition captures two periodic transformations that result in distinct minimum unit cell representations of the same crystal structure: (1) shifting periodic boundaries, and (2) altering periodic patterns while maintaining the same unit cell volume as shown in Figure
Completeness. Ideal crystal sequences need to be complete, allowing the full reconstruction of 3D crystal structures from their 1D sequences.
3.2 Determination of SO(3) equivariant unit cells
We propose Mat2Seq that converts 3D crystal materials into 1D sequences by (1) determining the SO(3) equivariant unit cells and (2) converting SO(3) equivariant unit cells into SE(3) invariant sequences. In this section, we demonstrate how to determine the S0(3) equivariant unit cell of given crystal structure M L.)
Let's start with an arbitrary minimum (primitive) cell with vectors L1, b, Lz. A set of SO(3) equivariant lattice vectors can be determined by Niggli cell reduction [19] as follows. The three lattice vectors E, Ez Eg of a new primitive cell can be represented terms of the original ones E. = k + kz + ki,3k3; ki,,, ki,,, ki,3 Z. Then, unique and SO(3) cquivariant cell can be determined by choosing three shortest non-planar vectors E, Ly L's that form a right-hand system using three Euclidean vector lengths and three relative angles (i.e. six lattice parameters) [14 19].
Determination of lattice vectors.
Determination primitive unit cells. After lattice vectors L,, = [e,, Eg Eg) have been determined, a unique unit cell is determined as follows. For different unit cells with the same lattice vectors G. by Eg they differ by the origin the cell. We determine the origin of unit cell by: smallest atomic number; 2. local densities of atoms in the cell that are E(3) invariant measurements defined

---

by Psrr = EJEN(. Zj, where (i,r)) denotes the neighbors of atom within the radius r, and Zj denotes the atomic number of neighboring atom j: and 3. densities along three lattice vectors f l, Eg that are SE(3) invariant measurements defined as the following.
Per
E (EN)
6
E
Zz Ptr
Σ
Z33:
(
(E.
where (i,r), (,, denote the neighbors of atom within the radius r along
E lg Eg that satisfy Pinuc,j > Pirac it Pirac.j >> Pirue, and Piinc.j > Pirac correspondingly. Here,
Pirecj denotes the fractional coordinate of atom j as follows,
Pj - Pfrac.. Lj + Pirme L2 Pinc.
(2)
After the unique atom chosen by 1., 2., and 3., the primitive unit cell is determined by moving this atom to the origin through the following transformation applied to all other fractional coordinates,
Piac. (Five.s pirae,) mod 1,
Pirme = y - ;i mod 1,
(4)
Pirac.j (Pirae. .i mod 1,
where the resultant P Pirac,u [Prrae1 1: Piine Pinne.n Rax3 and P. L,, P Pirac,u-
Remarks. By determining lattice vectors and primitive unit cells, SO(3) equivariant and periodic invariant unit cell (A,, P,, L.,) determined for a given crystal material, which means after applying arbitrary SO(3) transformations R E R3x3 JR| = with b E R3 and periodic transformations, the resultant unit cell will transform to (A... R.P. RL,). We then demonstrate how to construct the SE(3) invariant crystal sequence representation based on (A,, P L.,) in the following section.
3.3
SE(3) invariant crystal sequence representations
Given determined SO(3) equivariant and peri- odic invariant unit cells M = (A,, ; Pa. L.). aim represent them by SE(3) and peri- odic invariant sequences that are complete to guarantee the full reconstruction of crystal strue- tures. As discussed in Sec. 2.2] a fundamental requirement for crystal sequence representations is uniqueness which stipulates that (1) different crystal structures must correspond to distinct se- quences and (2) given crystal should yield the same sequence representation across all possible structural descriptions M = (A, P. L). Previ- ous works 10 12) directly use CIF files that
Detennined Unit Cells
Invariast Sequences
Compositional Information
Space Gruup Information
Invoriant Lattior Parmetters
Innoluiible Each
[Aoom Type
Coudisates)
Figure Converting determined unit cells into invariant crystal sequences.
breaks the uniqueness as shown in Figure [ the input sequence for LMs. Therefore, these methods require tremendous augmentation effort during the training process to make LMS aware that these different sequences refer to the same crystal structure. Additionally, due to the ((n²) computational cost of decoder-only language models where n is the window length, the sequence representations of crystal structures need to be efficient to support shorter window length and ease the burden of training and inference processes.
Rather than relying on CIF files, our approach extracts complete geometric information from the uniquely determined SO(3) equivariant unit cells M (A,, P L.). We first follow CIF files and use six invariant lattice parameters including = ||&, 12,1 = ||E2||,, ||al|| and a, 3, that are three bond angles between é,, l2, l3 to represent L,, as invariant sequences. Subsequently, we incorporate space group information of the crystal structures to reduce sequence length, documenting the irreducible atom sets alongside the corresponding symmetry transformations to facilitate the recovery of inner cell structures (A,,, P... We then represent atom i in the irreducible atom sets by the atom type and fractional coordinates [Z,, Pirm,, Pfrac, it :i J. It is worth noting that fractional coordinates of atom bases are invariant instead of equivariant, due to that P = L Pranc where denotes matrix product and RP = RL- Pime- The unique ordering of atoms within the irreducible set is determined by their atomic numbers, the number of duplicates recoverable from each atom, and their fractional coordinates, thereby ensuring consistency in the Mat2Seq sequence representations.
5

---

Table 1: Uniqueness verification compared with previous crystal language models
10
12
Uniqueness - MP20 15
CIF
TO 11
CIF with symmetry
12
Mat2Seq
Success rate f
0%
30%
100%
3.4 Properties and proofs
Uniqueness. Different from previous crystal sequence representations 11 12). Mat2Seq sequence
representations guarantee that all possible mathematical descriptions M = (A, P. for given crystal will have unique sequence representation and the same probability in the follow-up generative modeling process. We prove the uniqueness of Mat2Seq sequences as follows.
Lemma 1. A sequence mapping function f (A, + X unique, if function f periodic invariant and unit cell (E) invariant.
Proof. For a given crystal structure, a variety of structural descriptions (A, P. L) can be obtained by applying SO(3) rotations (A, R.P b, RL) and periodic transformations (A', P L') [14 [S] including (1) shifting periodic boundaries, and (2) altering periodic patterns while maintaining the same unit cell volume.
For sequence mapping function [A L) - X' that satisfies periodic invariance and unit cell SF(3) invariance, we have (AA,P,L) L) SAA, RP + b, RL) = f(A', P',L') according to the definition provided in Sec. 3.1 Hence, periodic invariant and unit cell (E)3) invariant function is naturally unique for crystal structures.
Lemma 2. Mat2Seq: (,, - X' is periodic invariant and unit cell SE(3) invariant.
Proof. To prove that Mat2Seq is periodic invariant and unit cell SE(3) invariant, we first show that the unit cell (Aa. Pw, L.) determined in Sec. 3.2 is (0(3) equivariant and periodic invariant. In other words, after applying arbitrary (3) transformations R IR 3x3 | with b R and periodic transformations, the resultant unit cell should change from (A., Pa. L,) to (Aw, RP... RL.), with detailed proof provided in Appendix A. We then prove that the obtained sequence representation S (3) and periodic invariant in Appendix A.1
Completeness. Crystal sequence representations need to be complete to guarantee the full recon- struction of crystal structures from sequences.
Lemma 3. Crystal structure (A, L) can be fully recovered from Mat2Seq sequence X.
Proof. can be directly recovered from atom types in Mat2Seq sequences, and L can be recovered by three lattice lengths a, b, and three lattice angles a, 3, 7. Then, fractional coordinates of all atoms the unit cell can be fully recovered by applying recorded space group transformations [RR, by), (R,, b,)] to fractional coordinates of the irreducible atom set P Fra follows,
P Trac (P fra,iir Oigigs (R,P frac,ir b;) mod 1)set,
where represents concatenation and ()se: represents the operation of removing duplicate entries. Pirac can then be fully recovered.
Remarks, Based Lemma 1, and 3, Mat2Seq sequences satisfy uniqueness for crystal structures and can fully reconstruct corresponding 3D crystal structures. We also provide experimental verifi- cation on the MP20 [5] training set for the uniqueness of Mat2Seg and previous crystal language models [10
III I2 in Table I by applying periodic transformation that shifts the periodic boundaries.
The evaluation conducted follows: (1) for each crystal structure in the MP20 dataset, we apply a transformation that shifts the periodic boundaries, generating different unit cell representation while keeping the crystal structure unchanged; (2) we then encode both the original and transformed structures into 1D sequence representations using different methods and compare the resulting se- quences. there a mismatch (e.g., differences in the coordinates or atom type for the first atom), is considered failure; (3) finally, compute the success rate across the entire datasct.

---

Table 2: Comparison of Match Rate (%) and RMSE across different models and crystal datasets.
Model
Perov-5
Carbon-24
MP20
MPTS-52
Match
RMSE
Match
RMSE
Match
RMSE
Match
RMSE
One Shot
LMystaLLM
46.1% 1%
0.095
20.3%
0.176
58. 78.7%
0.041
19.2%
0.111
CDVAE
45.3%
0.114
17.1%
0.297
33.9%
0.105
5.34%
0.211
DifICSP
52.0%
0.076
17.5%
0.276
51.5%
0.063
12.2%
0.179
Mat2Seg
50.0%
0.099
23.7% %
0.169
61.3%
0.040
23.1
0.109
20 Shots
LrystaLLM
97.6%
0,025
85.2
0.151
74.0%
0.035
33.8%
0.106
CDVAE
88.5%
0,046
88.4%
0.229
67.0%
0.103
20.8%
0.209
DifCCSP
98.6%
0.013
88.5% %
0.219
77.9%
0.049
34.0%
0.175
Mat2Seq
98.5%
0.023
86.0%
0.148
75.3%
0.037
36.4%
0.088
3.5 Language model enabled 3D crystal structure generation
Discretization. To train language model, crystal sequence representations obtained in Sec. 3.3 need to be discretized. Compositional and space group information including atom types, number of atoms in the cell, and space group symbol are discrete. For lattice parameters and fractional coordinates, we round real numbers to four decimal places to ensure consistency and efficiency following CrystaLLM [12]. We also include keywords and symbols like space space_group ltwgggrrouuumooop symbol, formula, lattice_ paraaeeeasseessssssee and so on in our Mat2Seq dictionary, with the full list provided in Appendix A.2
Training and sampling. We enable language models to generate novel crystals using the proposed unique and complete sequence representations for 3D crystal structures. Specifically, follow the previous work [12] and train a GPT (20 model M with parameters to capture the distribution of Mat2Seq crystal sequences, U, constructed from corresponding crystal datasets with standard next-token prediction cross-entropy loss for all elements in the sequence:
|u|-1 Σ
min EUEU
((Me(u1, U2, U,)) ui+1),
where represents a crystal structure within the dataset. During the sampling phase, we initi- ate the sequence with predetermined starting patterns, applying an autoregressive sampling tech- nique to iteratively predict the next sequence component from the conditional model distribution po(us+1 1u1, u22, ... ..) until a stop pattern or a maximum length is achieved. Consistent with the referenced work [12], our sampling strategy incorporates top-k [21] and temperature control [22] techniques. Importantly, the choice of language model M flexible and can be seamlessly substituted with more advanced or alternative powerful language models.
Conditional generation. For conditional generation, compositions and properties. Specifically, for Mat2Seq crystal sequence representations, the com- positional information is placed ahead of crystal structure information. Hence, by simply initialize sequences with desired compositions, the language model M can generate corresponding crystal structures. To target specific properties, introduce property token to denote the desired attribute and place these tokens at the beginning of the Mat2Seq sequences. The training and sampling processes mirror those used in unconditional generation, with the initial input being the property token that corresponds to the desired attribute.
conditional tokens represent desired
4 Experimental results
In this section, we evaluate the ability of Mat2Seq to discover stable crystal structure for interested crystal systems in Sec.
4.1 and 4.22 generate novel crystal structures satisfying desired physical properties in Sec. 4.3 We also include experimental results for random crystal generation assessing validity, stability (measured by DFT), and the ratios of stability, unique, and novel (S.U.N.) crystals in Appendix A.3
7

---

Table 3: Efficiency and model complexity comparisons.
MP-20 test set
number of parameters
RMSE
20 shots generation speed (sec./crystal)
CDVAE DifCCSP Mat2Seq-small Mat2Seq-large
4.5 M
0.103
37.9 s
Similar to CDVAE
0,049
7.3
25 M
0,039
2.1
200 M
0.037
5.7
Table 4: Mat2Seq match rate (%) and RMSE for experimentally observed crystal structures MP-20
test set.
MP-20
number of crystals
Match Rate (%)
RMSE
Mat2Seq exp observed Mat2Seq the whole test set
3,819
65.2 %
0,042
9,046
61.3 %
0.040
4.1 Generating stable crystal structures for given compositions
Baselines.
We first evaluate the effectiveness of Mat2Seq on the widely used crystal structure
prediction benchmark, comparing against diffusion-based methods including CDVAE [55 and DiffCSP 117, as well as recent language model based CrystaLLM [12]
Datasets. We conduct experiments on four datasets following DiffCSP and CrystaLLM [12] including Perov-5, Carbon-24, MP-20, and MPTS-52. Perov-5 [3 24] contains 18,928 perovskite
materials with similar structures and 5 atoms unit cells. Carbon-24 25 contains 10,153 carbon materials with at most 24 atoms in the unit cell. MP-20 [26] includes 45,231 stable inorgainic
materials from the Materials Project, covering the majority of experimentally -eeellr-eeeeedd materials with at most 20 atoms in the unit cell. MPTS-52 [17] and at most 52 atoms in the unit cell.
is the most challenging one with 40,476 crystals
Experimental setup. For crystal structure prediction, machine learning methods are trained using crystal structures and corresponding compositions in the training set. During the inference phase, compositions of crystals in the test set are provided as input to methods, and the goal is to generate stable crystal structures that match the ground truth crystal structures. To assess the quality of generated crystal structures, two metrics are used, including match rate which measures the ratio of the generated structures that match with the ground truth structure determined by Pymatgen structure matcher [27]. and RMSE (27] which measures the structural differences between the ground truth and matched generated structures. A single NVIDIA A100 GPU used for computing for this task. We directly follow DiffCSP [17 to split corresponding datasets into training, evaluation, and test sets. We use the same language model settings following CrystaLLM 122 to make sure the comparison is fair. Detailed training and inference parameters are provided in Appendix A.2] We also provide additional experimental evaluations in Appendix AA. for random crystal generation on the MP20 dataset, assessing validity, stability (measured by DFT), and the ratios of stable. unique, and novel crystals, compared with statc-of-the-an methods, including FlowMM (28 DiffCSP [17]
and CDVAE 15.
Results. There are several observations from the experimental results shown in Table. (1) Compared with CrystalLM [12 that directly uses CIF files to describe 3D crystal structures, Mat2Seq achieves better match rate for all eight tasks, demonstrating the effectiveness of Mat2Seq crystal sequence representations beyond CIF files. (2) Compared with DiffCSP 17 that is specifically designed for this task, Mat2Seq achieves comparable results, and achieves significantly better match rate and smaller RMSE for the most challenging MPTS-52 dataset, with a 89% improvement in match rate for one shot generation, and a 50% % decrease in RMSE for 20 shots generation.
(3) Mat2Seq demonstrates excellent one shot generation performances across datasets with various difficulty levels and data scales.
Results on experimentally observed crystal structures. Comparing with experimentally observed crystal structures is essential demonstrate the real-worid applicability of the proposed method, beyond synthetic structures generated purely from DFT calculations. The MP20 dataset test set includes 3,819 crystal structures that have been experimentally observed (match entries in the ICSD 2. To assess the accuracy of Mat2Seq on this data, we conducted additional evaluations on these 3,819 structures, with the results presented in Table 4 Notably, Mat2Seq achieved 65.2% match rate with an RMSE of 0.042 for these experimentally observed structures.
8

---

Ground Truth
Mat2Seq Generated
Ground Truth
Mat2Seq Generated
Figure 4:
Mat2Seq can generate recently discovered novel crystals from literature. Eug on the left, and reflected version of the ground truth.
Te on the right. The structure generated by Mat2Seq for Eu is the
Table 5:
Ability to generalize to recently discovered crystals from literature, measured by three seperate generation runs.
CrystaLLM
Ours
Validity 1 + std 23.7% + 2.5% 27.3% + 1.4%
Efficiency. It is well known that language models typically have higher parameter counts and greater computational demands. To address this, we provide a comparison of efficiency in terms of parameter count and generation speed, alongside specialized crystal models such CDVAE and DiffCSP, in T While Mat2Seq, based on large language models, does have more parameters, using similar GPU resources (as done in DiffCSP (7]). Mat2Seq demonstrates significantly faster generation speed. Furthermore, we show that reducing the model size by a factor of eight does not significantly affect Mat2Seq's performance and improves efficiency. In this case, the smaller model achieved an RMSE of 0.039, which is notably better than DiffCSP's RMSE of 0.049, while also being more than three times faster the generation process.
4.2 Ability to generalize to recently discovered crystals from literature
The promising results in crystal structure prediction shown in Sec. have demonstrated the effectiveness of Mat2Seq crystal sequence representations. We then examine the ability of Mat2Seq with language model to discover novel crystal structures that are obtained from recent literature and are not seen during training. Specifically, we randomly choose 10 crystals from the challenge set [12] to evaluate Mat2Seq's is ability discover novel crystal structures that originate from recent literature, and compare with previous CrystalLMM 12].
Dataset. For this task, we follow CrystaLLM [2] and train the model using crystal structures from the Materials Project [26). OQMD 33 and NOMAD [31), which has been optimized using DFT calculations, containing 2,047, (49, crystal structures for training in total. This dataset contains
around 800K unique formulas and 94 chemical element types. The evaluation conducted with 10
recently discovered crystal structures from literature, including AlCuzAs(OH)) (32. BayFezFg (33
Ca
(34] Cdy Te (35). CiyC Oss
(36), Cu,FeGe,OS, 37, Eu,FeGe, OSg 38
KzSta(PO 1o L Laz 40 and Lig Sn, 41
Experimental setup. We first train model using around 2M crystal structures in the training set following Crystal.LM with broken files removed. After training, we evaluate the model's ability generate valid crystal structures for 10 recently discovered crystal structures from literature by taking the compositional information as conditions and generating 20 structures for each composition. To compare with CrystaLLM, we directly use their pre-trained model for the same training set and generate 20 structures for each composition. We follow CrystaLLM and evaluate the validity of generated structures by whether the generated formula consistent with the given composition, whether the space group is consistent with the structure, and whether the generated crystal has reasonable bond lengths. A100 GPUs are used for computing for this task. We also provide additional experimental results in Appendix A.3 evaluating the Hit rate (i.e., whether the generated structure matches recently observed experimental crystals from the literature) and RMSE across 10 challenge crystal systems, in comparison with the previous fate-of-theean method, CrystaLLM.

---

Table 6: Ability to generate crystal structures with desired band gap properties. We measure the success rate of generating crystal structures with band gap value < 0.5 eV and band gap value > 3.0 eV, measured by the
band gap predictor ComFormer [4]. validity, uniqueness, and novelty of the generated erystals.
We also evaluate the
Generation Codition
band gap < 0.5 band gap > 3.0
Validity
Uniqueness
Novelty
Towards band gap + Towards band gap f
83.6%
12.0%
88.0%
98.0%
86.2%
6.4%
90.7%
89.8%
92.2%
98.6%
Results. As shown in 5 for the 20 10 generated crystals, our method achieved 27.3% validity rate for novel crystal structures from recent literature, better than CrystaLLM with 23.7%. We visualize the generation results for EuzFeGezOS, and Ceg CcgCd23 Te in Figure 4 which shows Mat2Seq actually can discover novel crystals reported in recent literature with nearly identical 3D structures. is worth noting that the structure generated by Mat2Seq for EuyFeGe,OS$ is the reflected version of the founded one, and reflected crystal structure will have the identical energy, which on the other hand demonstrating the reflection sensitivity of Mat2Seq.
4.3 Discovering crystal structures with desired properties
Experimental details. We further evaluate Mat2Seq's 's ability to discover crystals with desired properties. Specifically, we have put 10 placeholders marked as unknown _rop before Mat2Seq crystal sequences when training our model on around 2M crystal structures. After pre-training, we fine-tune the model on JARVIS-DFT dataset 42 with 61. 1,,541 crystal structures and corresponding band gap values in the training set. We use intervals to tokenize band gap values, where values from 0 to 0.5 are marked as O, values from 0.5 to 1 are marked as 1, and so and replace one unknown__pov token in the sequence. For future users, this design can enable the discovery of physical properties of their interest by simply replacing the unknown propp token with corresponding task.
properties and fine-tuning the pre-trained model from Sec. 4.2 A single A100 GPU used for this
Evaluation metries. We evaluate our model in two settings, including (1) generating crystals with low band gap (< 0.5 eV) that could potentially be used for thermoelectrics, thermophotooolltiic, infrared sensing and vision, cryogenic cooling, as well as discovering novel catalysts and topological quantum materials, and (2) generating crystals with high band gap (> 3.0 eV) that could potentially be used for solid-state LEDs and lasers, nonlinear optics, ferroelectrics, multiferroics, power electronics, and high-temperature applications. We measure the success rate of 500 randomly generated crystals that satisfy (1) and (2) when our model is conditioned towards generating crystals with small band gap values and large band gap values. We measure the band gap value of generated crystals using the state-of-ther -ort---rr- method ComFormer [4] for band gap prediction with mean absolute error of 0. eV. We also provide measurements of validity, uniqueness, and novelty for the generated crystals.
Results. Table@jdemnnstratee Mat2Seq's 's ability to significantly alter the original property distribution and discover crystals towards large or small band gap values. Specifically, when conditioned towards small band gap values with interval to 0.5 eV, 83.6% % generated crystals have band gap values < 0.5 eV. and when conditioned towards large band gap values with interval 3.5 to 4.0 eV, 90.7% generated crystals have band gap values 3.0 eV.
5 Conclusion, limitation, and discussion
In this work, we propose Mat2Seq, which converts 3D crystals into unique and complete ID sequences for language model-enabled crystal generation, naturally ensuring that all equivalent crystals are considered identically probable without the need for data augmentation. Mat2Seq demonstrates promising performance in crystal structure generation and discovery, and the trained model can be used discover crystal structures with desired properties of interest to users. The limitations of the current Mat2Seq include: (1) it cannot be directly used for other atomic systems, like molecules and proteins: (2) the extension to model disordered materials remains a challenging frontier; and (3) large-scale training with more stable crystal structures can potentially enhance the robustness and performance when more computational resources are available. Positive and negative societal impacts of discovering novel materials with desired properties may apply to this work.
10

---

Acknowledgments and Disclosure of Funding
K.Y. and S.J. acknowledge the support from U.S. National Science Foundation (NSF) grant MOMS- 2331036. First-principles calculations and structure optimization by K.A. were supported by the Center for Reconfigurable Electronic Materials Inspired by Nonlinear Dynamics (reMIND), an Energy Frontier Research Center funded by the Department of Energy under award DE-SC0023353. X.F.Q. acknowledges the support from the Air Force Office of Scientific Research (AFOSR) under Grant No. 109550-24-1-022 and NSF CMMI-2226908. This work was partially supported by the donors of ACS Petroleum Research Fund under Grant 65502-NDIO. X.N.Q. and R.A. acknowledge the support from U.S. National Science Foundation (NSF) grant DMREF-21 211900 and X.N.Q acknowledges the support from NSF through grants SHF-2215573, and IIS IIS-2212419. Portions this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing. M.Z. gratefully acknowledges the support of NIH 101-HD108794 NSF CAREER 2339524, US DoD FA8702-15-D-00W ARPA-H BDF Toolbox, awards from Pfizer Research, Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Sanofi iDEA-iTECH Award, Chan Zuckerberg Initiative, John and Virginia Kaneb Fellowship award at Harvard Medical School, Biswas Computational Biology Initiative in partnership with the Milken Institute, Kempner Institute for the Study of Natural and Artificial Intelligence, and Harvard Medical School Dean's Innovation Fund. C.E. and HJJ. acknowledge the support from the Molecule Maker Lab Institute: an AI research institute program supported by NSF under award No. 2019897 and No. 2034562. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
[1] loannis Petousis, Wei Chen, Geoffroy Hautier, Tanja Graf, Thomas Schladt, Kristin A Persson, and Fritz B Prinz. Benchmarking density functional perturbation theory enable high-throughput screening of materials for dielectric constant and refractive index. Physical Review B, 1311111 2016.
[2] Susmriti Das Mahapatra, Preetam Chandan Mohapatra, Adrianus Indrat Aria, Graham Christie, Yogendra Kumar Mishra, Stephan Hofmann, and Vijay Kumar Thakur. Piezoelectric Mate- rials for Energy Harvesting and Sensing Applications: Roadmap for Future Smart Materials. Advanced Science, 8(1 8(17):2100864, 2021.
[3] Hua Wang and Xiaofeng Qian. Ferroelectric nonlinear anomalous Hall effect in few-layer WTez- npj Computational Materials, 5(1):119, 2019.
[4] Jun Xiao, Ying Wang, Hua Wang, C. D. Pemmaraju, Siqi Wang, Philipp Muscher, Edhert J. Sie, Clara M. Nyby, Thomas P. Devereaux, Xiaofeng Qian, Xiang Zhang, and Aaron M. Lindenberg. Berry curvature memory through electrically driven stacking transitions. Nature Physics, 16(10):1028-1033 2020.
[5] Hua Wang and Xiaofeng Qian. Electrically and magnetically switchable nonlinear photocurrent in PT -yymmertt magnetic topological quantum materials. npj Computational Materials, 6:199, 2020.
[6] Kamal Choudhary, Daniel Wines. Kangming Li, Kevin F Garrity, Vishu Gupta, Aldo H Romero, Jaron' T Krogel, Kayahan Saritas, Addis Fuhr, Panchapakesan Ganesh, al. JARVIS- Leaderboard: a large scale benchmark of materials design methods. npj Computational Materi- als, 10(1) ):93, 2024.
[7] Xuan Zhang, Limei Wang. Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yucheng Wang. Haiyang Yu, YuQing Xie, Xiang Fu, Alex Strasser, Shenglong Xu, Yi Liu, Yuanqi Du, Alexandra Saxton, Hongyì Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao, Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang. Ameya Daigavane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung,
11

---

Minkai Xu, Chaitanya K. Joshi, Simon V. Mathis, Kamyar Azizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Guzik, Erik Bekkers, Michael Bronstein, Marinka Zitnik, Anima Anandkumar, Stefano Ermon, Pietro Liò, Rose Yu, Stephan Günnemann, Jure Leskovec, Heng Ji, Jimeng Sun, Regina Barzilay, Tommi Jaakkola, Connor W. Coley, Xiaoning Qian, Xiaofeng Qian, Tess Smidt, and Shuiwang Ji. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.0844 07840007004 2023.
[8] Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In The 36th Anmual Conference on Neural Information Processing Systems, pages 15066-15080, 2022.
[9] Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient approximations of complete interatomic potentials for crystal property prediction. In ICML, pages 21 21260-21287, 2023.
[10] Daniel Flam- Shanhhhe and Alán Auzik. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files. arXiv preprint N.Ni::2005.05008 2023.
[11) Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint arXiv:2402.00439, 2024.
[12] Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespa Crystal structure generation with autoregressive large language modeling. arXiv preprint arXiv:2307.04.. 0400004444 2023.
[13] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. In The 36th Annual Conference on Neural Information Processing Systems, pages 650-664, 2022.
[14] Keqiang Yan, Cong Fu, Xiaofeng Qian, Xiaoning Qian, and Shuiwang Ji. Complete and efficient graph transformers for crystal material property prediction. In The Twelfth International Conference Learning Representations, 2024.
[15] Tian Xie, Xiang Fu, Octavian- Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal Dif- fusion Variational Autoencoder for Periodic Material Generation. In International Conference on Learning Representations, 2022.
[16] Youzhi Luo, Chengkai Liu, and Shuiwang Ji. Towards symmetry-awand generation of periodic materials. Advances Neural Information Processing Systems, 36, 2024.
[17] Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[18] Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabbé, Lixin Sun, Jake Smith, et al. MatterGen: a generative model for inorganic materials design. arXiv preprint arXiv:2312 03.06677 2023.
[19] Hong-Long Shi and Zi-An Li. Niggli reduction and bravais lattice determination. Journal of Applied Crystallography, 55(1):204-210, 2022.
[20] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[21] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: 1005::1800.0444 04833, 2018
[22] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 91):1177-1,, 1985.
[23] Ivano E Castelli, David D Landis, Kristian Thygesen, Søren Dahl, Ib Chorkendorff, Thomas F Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, (000099334-433333 2012.
12

---

[24] Ivano E Castelli. Thomas Olsen, Soumendu Datta, David D Landis, Søren Dahl, Kristian S Thygesen, and Karsten W Jacobsen. Computational screening of perovskite metal oxides for optimal solar light capture. Energy & Environmental Science, (12):5814558)) 2012.
[25] Chris J Pickard, Ashkan Salamat, Michael Bojdys, Richard J Needs, and Paul F McMillam. Carbon nitride frameworks and dense erystalline polymorphs. Physical Review B, 94(9):094104, 2016.
[26] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):01 1002, 2013.
[27] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin Persson, and Gerbrand Ceder. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314-319, 2013.
[28] Benjamin Kurt Miller, Ricky TQ Chen, Anuroop Sriram, and Brandon M Wood. FlowMM: Generating Materials with Riemannian Flow Matching. In Forty-first International Conference on Machine Learning, 2024.
[29] Dejan Zagorac, H Müller, S Ruehl, Zagorac, and Silke Rehme. Recent developments in the Inorganic Crystal Structure Database: theoretical crystal structure data and related features. Journal of Applied Crystallography, 52(5):918 1099188-222 000922,,2222 2019.
[30] James E Saal, Scott Kirklin. Muratahan Aykol, Bryce Meredig, and Christopher Wolverton. Materials Design and Discovery with Throghhhruuhput Density Functional Theory: The Open Quantum Materials Database (OQMD). JOM, 65:1501-1509, 2013.
[31] Markus Scheidgen, Lauri Himanen, Alvin Noe Ladines, David Sikter, Mohammad Nakhaee, Ádám Fekete, Theodore Chang, Amir Golparvar, José A Márquez, Sandor Brockhauser, et al. NOMAD: A distributed web- beb-based platform for managing materials science research data. Journal of Open Source Software, 8(90))5388, 2023.
[32] Alexandra Plumhoff. Thermodynamic properties, crystal structures, phase relations and isotopic studies of selected copper oxysalts. PhD thesis, 2020.
[33] Elizabeth A Pogue, Jack Bond, Cassandra Imperato, John BS Abraham, Natalia Drichko, and Tyrel M McQueen. A Gold(D) Oxide Double Perovskite: BazAulO,- Journal of the American Chemical Society,
19033-194499 2021.
[34] Matthias Weil. Os, new phase in the CaO-TeOp system. Acta Crystallographica Section E: Crystallographic Communications, 75(1):26-29, 2019.
[35] Griffen Desroches and Svilen Bobev. Synthesis and structure determination of CegCdas Te: a new chalcogen-containie member of the C family (RE is a rare-earth metal and is a late group 14. 15 and 16 element), Acta Crystallographica Section C: Structural Chemistry. 73(2):121-125, 2017.
[36] Gregory Morrison, Virginia G Jones, K Pilar Zamorano, John Greedan, and Cons-C Zur Loye. Flux Synthesis, UV-vis Absorbance, and Magnetism of Cesium Copper Silicates with an Isolated Super-Supe Exchange Spin Dimer in CsgCuuSigg2ss Inorganic Chemistry. 62(29): 1688 2023.
[37] Andrew Craig, Stanislav S Stoyko, Allyson Bonnoni, and Jennifer A Aitken. Syntheses and crystal structures of the quaternary thiogermanates Cu, and Cu,CoGezSr. Acta Crystallographica Section E: Crystallographic Communications, 16(7)1117-1121, 2020.
[38] Nan Zhang, Xiao Huang, Wen-Dong Yao, Yao Chen, Zheng- Rui Pan, Bingxuan Li, Wenlong Liu, and Sheng-Ping Guo. Eu, Ou (M= Mn, Fe, Co): Three Melilite Rare-Farth Oxythiogermanates Exhibiting Balanced Nonlinear-Optice Behaviors. Inorganic Chemistry, 62249): 2023.
13

---

[39] Weimin Dong, Yingjie Sun, Henghao Feng, Dazheng Deng. Jun Jiang, Jin Yang, Wei Guo, Libin Tang, Jincheng Kong, and Jun Zhao. (=Sr((PO3)1o= A Polyphosphate with Deep-UV Cutoff Edge and Enlarged Birefringence. Inorganic Chemistry, 62(39):16215--6222) 2023.
[40] Hong Yan, Kotaro Fujii, Houria Kabbour, Akira Chikamatsu, Yu Meng, Yoshitaka Matsushita, Masatomo Yashima, Kazunari Yamaura, and Yoshihiro Tsujimoto. GayGasSs3 A Rare-Earth Gallium Oxysulfide with Disulfide lons. Inorganic Chemistry, 62126):10411-0444 2023.
[41] Volodymyr Pavlyuk, Grygoriy Dmytriv, Ivan Tarasiuk, and Helmut Ehrenberg. Lig new ordered superstructure of the Li13Sn; type. Acta Crystallographica Section C: Structural Chemistry, 73(4):337-342, 2017.
[42] Kamal Choudhary, Kevin F Garrity, Andrew CE Reid, Brian DeCost, Adam J Biacchi, Angela R Hight Walker, Zachary Trautt, Jason Battrick-Simpent A Gilad Kusne, Andrea Centrone, et al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. upj Computational Materials, 6:173, 2020.
14

---

A Appendix
A.1 Proofs
Lemma 4. Mat2Seq: (A, - X is periodic invariant and unit cell SE(3) invariant.
Proof. To prove that Mat2Seq is periodic invariant and unit cell SE(3) invariant, we first show that the unit (Aa; Pu: L.) determined in Sec. words, after applying arbitrary 50(3) transformations R E 3x33 |R = with b E RR3 and periodic transformations, the resultant unit cell should change from (A., P. L.) (Ag: RP u. RL,). To begin with, because Euclidean distances a, b, c, bond angles a, 3, 7, and whether three vectors form right-hand system are SE(33 and periodic invariant, which means after arbitrary SO(3) transformation R E IR 3x3 |R| 1 with b E I& and periodic transformation, the same but rotated lattice matrix RL, will be determined. Additionally, because local density Pt.r. atom
3.2 Sec.
SO(3) equivariant and periodic invariant. In other
ei Eg 8 type Zi and densities along three lattice vectors Pi.r: Pt.r Pi. are SE(3) and periodic invariant, the same atom will be determined to serve as the origin of the unit cell after arbitrary SO(3) transformation and periodic transformation. Hence, when applying arbitrary SO(3) transformation R _3x3 |R| I. b E R3 and periodic transformation to (A, P,L., the resultant primitive unit cell will be (Aa, R.P... u. RL,)) Hence, the unit cell (A P. L.) determined by Mat2Seq is SO(3) equivariant and periodic invariant.
Based on the S0(3) equivariant and periodic invariant primitive unit cell (A., Pu, Lu). we then prove that the obtained sequence representation is SE(3 and periodic invariant. To begin with, it can be seen that lattice lengths, lattice angles, and atomic numbers are naturally invariant. Additionally, due to that P., L,, in (A,, P., u, L.) are 50(3) equivariant, periodic invariant, and are connected by fractional coordinates p; = Pirec. = x. * l1 + y le z lz. when the crystal is rotated by R we can have
Rp: R(x l1 y 22 Ls) r' R& + 3/ Rlg + 2' RRs.
which means "',,"""'''====>>-->" because Pw; L,, are periodic invariant, the corresponding fractional coordinates are periodic invariant. To summarize, it can be seen that all components in Mat2Seq sequence, including (1) compositional information, (2) space group information, (3) lattice parameters obtained by lattice lengths and angles, and (4) atom types and fractional coordinates SE(3) and periodic invariant.
Hence fractional coordinates are SF(3) invariant. Additionally,
A.2 Experimental details
Following CrystaLLM [1), we use GPT-2 with 16 layers, 16 heads, and embedding size 1024 for all tasks. We show the detailed training parameters including window size, batch size, learning rate, drop out ratio, number of training iterations for different tasks in Table.
Table 7: Training parameters of Mat2Seq for crystal structure prediction benchmark.
Task
Window size
Batch size
Learning rate
Num. iterations
Drop out
Perov-5
768
32
0.0005
40k
0.1
Carbon-24
768
48
0.0003
40k
0.1
MP20
768
32
0.0003
50k
0.1
MPTS52
1,385
24
0.0003
40k
0.1
Crystal.LM 2.3
1,500
128
0.0005
500k
0.1
JARVIS bandgap 1,500
20
0.0001
10k
0.1
During sampling phase, for Perov-5 dataset, we use temperature=0.7 and top-k=10 one shot gen- eration, and temperature=1.0, top-k=10 for 20 shots generation. For Carbon-24 dataset, we use temperature=0.7 and top-k=10 one shot generation, and temperature=1.0, top-k=10 for 20 shots generation. For MP20 dataset, we use temperature=0.4 and top-k=5 one shot generation, and 1emperature=115, top-k=10 for 20 shots generation. for MPTS52 dataset, we use temperature=1.0 and top-k=10 one shot generation, and temperature=1.00, top-k=10 for 20 shots generation. For discovering novel crystal structures obtained from recent literature, use temperature=0.77 and
15

---

top-k=10 during sampling. For discovering crystal structures with desired band gap properties, we use temperature=0.6 and top-k=10 during sampling phases.
Mat2Seq tokens. We provide comprehensive list of tokens used by Mat2Seq, including atom types listed in Table | integer numbers from 0 to 300, digits including 0 to 9 with "." represent real values, special tokens listed in Table 9 and 227 space group symbols [12].
Table 8: List of atom types.
Ac
Ag
A1
Ar
As
Au
B
Ba
Be
Bi
Br
C
Ca
Cd
Ce
CI
Co
Cr
Cs
Cu
Dy
Er
Bu
F
Fe
Ga
Gd
Ge
H
He
Hf
Hg
Ho
I
In
Ir
K
Kr
La
Li
Lu
Mg
Mn
Mo
N
Na
Nb
Nd
Ne
Np
O
Os
P
Pa
Pb
Pd
Pm
Pr
Pt
Pu
Rb
Re
Rh
Ru
Sb
Sc
Se
Si
Sm
Sn
Sr
Ta
Tb
Tc
Te
Th
Ti
T1
Tm
U
V
W
Xe
Y
Yb
Zn
Zr
Table 9: List of special tokens.
space_ Lecee_rruup_SS__ symbol
group_symbol
formula beta <pad>
atoms gamma
lattice parammeerss
b
alpha
unknown prop
A.3 Additional Experimental Results
We provide additional experimental results of Mat2Seq in this section.
In Table 10, we compare Mat2Seq with the recent state-of-the-a method FlowMM [28]. assessing Validity. Stability, and S.U.N. (stable, unique, and novel) metrics. To ensure fair comparison, follow the FlowMM pipeline and generate 1000 materials to obtain these results.
Table 10: Comparison of Validity (%), Stability (%). and Stable, Unique, and Novel (S.U.N) (%) on MP20 dataset. The unit of Epoal is eV/atom.
Model
Validity (%)
Stability Rate DFT (%) f Exait < 0.0
S.U.N. Rate (%)
Composition
Structural
Excu < 0.1
MP
CDVAE
86.7 %
100 %
1.57%
1.4%
DifICSP
83.3
100 %
5.06%
3.3%
FlowMM (ICML 24)
83.2
96.9
4.19%
2.5%
Mat2Seq (lemp=1.35)
88.5
94.2
4.10%
49.2%
2.0%
Mat2Seq (temp=1.65)
81.7
88.6
4.50%
46.6%
3.2%
In Table 11
we present the Hit Rate (whether the generated structures match the ground truth structure) and RMSE the challenge crystal systems discussed Section 4.2 compared previous state-of-the-art method, CrystaLLM.
Table 11: Comparison of Hit Rate (%) and RMSE for ten recently discovered crystal structures from literature.
Model
Hit Rate (%) f
RMSE L
CrystaLLM Mat2Seq
nan
10.0 %
0.0388
A.4 Licenses for existing assets
We have used datasets including Perov-5, Carbon-24, and MP20 curated by CDVAE [.5] with MIT License, MPTS-52 curated by DifICSP [17] with MIT License, JARVIS-DFT 42 with NIST
16

---

License, CrystaLLM 12 with MIT License, the Materials Project 226 with Creative Commons Auribution 4.0 License, OQMD 30 with Creative Commons Attribution 4.0 International License, and NOMAD 31 with Apache License Version 2.0, January 2004. We have used Crystal.LMM [2] model with MIT License.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As clearly discussed in Sec. and Sec. the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations this work discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
Detailed proofs are provided in Sec. 3.4 and Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed experimental settings, datasets, and computations needed are shared in Sec. and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
The code will be release after the paper is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpss//mips.cc/pubbic guides/Cooeeuubsssioooooool
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (htips) wwwabl/ccguudee/CeeeSuuiiiiieeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Detailed experimental settings are provided Sec. [ and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the validity of randomly generated crystal structures for interested crystal compositions from recent literature, including 20 generation runs for each composi- tion. We also repeat this generation process for our method and the baseline method three times and show the std in Sec. 4 For crystal structure prediction tasks, we follow previous benchmark settings and include 20 shots generation results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed compute resources needed are provided in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform that we follow NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide Broader Impacts in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We will consider this when we release the model in the future.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the existing assets in the main paper and provide licenses of them in Appendix A.4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssitheode. seeerssithhhddeccm/aaaaattt has curated licenses for some datasets. Their licensing guide can help determine the license of
dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We discuss details of the dataset and model in Sec. and Appendix A.2
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "k70p7DRzP7", "reviewer": "Reviewer_JJwt", "summary": "This article propose a novel method, known as Mat2Seq, to tackle this challenge. Mat2Seq to converts 3D crystal structures into 1D sequences and ensures that different mathematical descriptions of the same crystal are represented in a single unique sequence, thereby probably achieving SE(3) and periodic invariance. Experimental results show that, with language models, Mat2Seq achieves promising performance in crystal structure generation as compared with prior methods. Overall this work gives new insights and should be accepted for the conference after minor revisions.", "questions": "1. Author needs to verify the generated data with more experimental results as currently it have multiple limitations. 2. Author needs to compare the accuracy with already existing CIF data of crystals.", "limitations": "The limitations of the  current Mat2Seq include: (1) it cannot be directly used for other atomic systems, like molecules and proteins; (2) the extension to model disordered materials remains a challenging frontier; and (3) large-scale training with more stable crystal structures can potentially enhance the robustness and performance when more computational resources are available.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Mat2Seq to converts 3D crystal structures into 1D sequences\n2. it reports the framework for creating unique and complete crystal sequence representations, followed by the construction of a material LLM capable of generating novel crystal structure with desired properties of interest.", "weaknesses": "1. Lack of experimental data validation for the generated data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer JJwt,\n\nThank you very much for your recognition of our work in terms of insights and contributions. For your questions and concerns, we provide point-to-point responses as follows.\n\n1. Lack of experimental data validation for the generated data. Need to verify the generated data with more experimental results as currently it have multiple limitations. Need to compare the accuracy with already existing CIF data of crystals.\n\nThank you for your insightful comments. We totally agree that the comparison with experimental data (or in other words, crystal structures that have been experimentally observed) is important to demonstrate the ability of the proposed method for real world applications, beyond synthetic crystal structures obtained from pure DFT calculations. \n\nActually, to demonstrate this, we conduct experiments by using Mat2Seq to discover recently experimentally discovered crystal structures from literature in Section 4.2. The visual comparison between the Mat2Seq generated structure with the experimentally observed crystal structure is shown in Figure 4, which shows our proposed method's ability for experimentally observed crystal structures. We also add one experiments to show the match rate and RMSE in Table 12 attached above in the general response.\n\nAdditionally, the datasets used in Section 4.1 also contain a large number of crystal structures that are experimentally observed. For example, in MP20 dataset test set, there are 3819 crystal structures that are experimentally observed. To show the accuracy of Mat2Seq for experimental data, we additionally conduct evaluation experiments on these 3819 crystal structures with results shown in Table 10 attached above in general response. It is worth noting that Mat2Seq achieves 65.2% match rate with 0.042 RMSE on these 3819 crystal structures.\n\n2. For current limitations listed in the paper.\n\nThank you for your valuable question. \n\n(1) Designing a LLM-based generative method for all atomic systems while satisfying uniqueness and completeness is highly nontrivial and a open question. We tend to address this problem in future works. \n\n(2) Different from other generative methods like diffusion based or flow based methods, LLM based methods generate crystal structures in an autoregressive manner. This difference makes a lot of techniques used by previous diffusion or flow based methods invalid for LLM-based methods. And this is an open frontier to be explored as also discussed by previous LLM-based methods. In this paper, we proposed a potential solution for this problem, and we tend to believe there are rooms for future works to improve. \n\n(3) As discussed by the ML community, usually more data will result in more powerful ML models. However, generating large scale crystal structure dataset is very expensive and currently out of our scope, so we list this as a potential direction to further improve the power of ML methods for crystal structure generation. \n\nWe hope we addressed your concerns. If you have additional questions or concerns, we are more than willing to answer. \n\nYours sincerely, Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4tzmgGf7T", "reviewer": "Reviewer_5Kst", "summary": "There are several challenges when developing LMs for materials: 1) each crystal structure consists of an infinite number of atoms and a unique and invariant unit cell must therefore be selected for each crystal 2) the unit cell can be represented in a one-dimensional (1D) sequence that maintains invariance under arbitrary rotations and ensures completeness.\nTo tackle these challenges, the paper proposed a method called Mat2Seq that systematically transforms 3D crystal structures into 1D sequences. This is achieved by first identifying SO(3) equivariant unit cells and subsequently converting these into SE(3) invariant sequences. The experimental results in crystal structure prediction and crystal discovery with desired properties validate the efficacy of Mat2Seq.", "questions": "- Could you provide a more detailed explanation with examples regarding the input for the language model (LM)?\n- Why is the target property value inserted at the beginning of the text input when generating materials for a specific target property?\n- Does “irreducible atom sets” refer to the atom sets contained within the minimum unit cell?", "limitations": "Please refer to the Weakness section.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- I understand why it is important for the inherent symmetry of materials to be reflected in the text when designing a language model for material generation.\n- The importance of unique representation and completeness from the structure of the material to the sequence is emphasized.", "weaknesses": "- Overall, the novelty and contribution of the paper are low. Instead of proposing new algorithms, they simply apply technologies that are already widely used in materials science to the process of representing materials as 1d sequence text. More specifically, the method for determining the equivariant unit cell is derived from Niggli cell reduction rather than being an original contribution of this paper. This concept is already widely used in the field of materials science, and the authors’ application of it to the text representation of materials does not seem like a significant contribution.\n- It is unclear what Section 3.3 is trying to convey. The authors claim to achieve text representation reflecting SE(3) symmetry following SO(3) symmetry, but it is doubtful whether this is well conveyed in this section.\n- The evaluation metrics used are limited. To demonstrate efficiency more effectively, the paper should present values for various metrics such as Validity, Stability, and S.U.N., which are commonly addressed in other studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5Kst,\n\nThank you for your time invested in reviewing this work. We provide point-to-point responses to your questions and concerns.\n\n1. Weakness 1\n\nThank you very much for raising this question. However, we kindly disagree with this and we feel there might be a potential misunderstanding of our proposed method. \n\nLet's begin from a question. Could Niggli cell reduction algorithms or other widely used 1D crystal representations including CIF files achieve uniqueness or SE(3) invariance? The answer to this question is **No**. More specifically, **Niggli cell reduction can only give you a set of lattice vectors and cannot determine a unique crystal unit cell**. Additionally, there is a previous work **CrystaLLM that has been using Niggli cell reduction** when converting the crystal structures into 1D sequences. However, as we show in Table 1, **CrystaLLM cannot achieve unique** crystal 1D sequence representations. \n\nFurthermore, let's look into our proposed method Mat2Seq and demonstrate how it achieves the desired properties including uniqueness, completeness, and SE(3) invariance. **The niggli cell reduction is only used in the first step when we want to uniquely determine a set of lattice vectors**. After that, we need to uniquely determine a corresponding unit cell. And then, we need to determine a unique ordering of atoms in the cell, and features used to completely represent the crystal 3D unit cell structures. The Niggli cell reduction is only an initial step of our proposed method, and to the best of our knowledge, Mat2Seq is the first work that can achieve uniqueness, completeness, and SE(3) invariance when converting 3D crystal structures into 1D sequences in the field of materials science. Thus, we kindly disagree that this problem is solved by previous Niggli cell reduction algorithms or any other widely used methods in materials science. \n\nLast but not least, we want to point out that all previous LLM based crystal generation methods fail to achieve uniqueness and SE(3) invariance. It is valuable to propose a method to address this limitation of the current process in this direction.\n\n2. Weakness 2\n\nWe appreciate your question. To clarify, as we mentioned in the beginning of section 3, we begin with Requirements for ideal crystal sequence representations in Section 3.1, and then move forward to how to determine a SO(3) equivariant unit cell in Section 3.2, then, Section 3.3 is used to demonstrate how to convert a SO(3) equivariant unit cell into SE(3) invariant 1D sequence. \n\nSpecifically, in Section 3.3, we show that for given determined SO(3) equivariant and periodic invariant unit cells $\\mathbf{M}=(\\mathbf{A}_u, \\mathbf{P}_u, \\mathbf{L}_u)$, we represent them by SE(3) and periodic invariant sequences that are complete to guarantee the full reconstruction of crystal structures.\n\nMore specifically, the SO(3) equivariant unit cell obtained in Section 3.2 cannot be directly used as input for LLMs, because a given crystal structure can have infinite number of SO(3) equivariant unit cells that differ by a rotation transformation.\n\nAdditionally, to show that the converted 1D sequence satisfies all the requirements, we provide detailed proofs in Section 3.4. \n\nWith these being said, we will appreciate any additional specific suggestions you have to improve this section, and we will revise the main paper accordingly once we have the chance.\n\n3. Weakness 3\n\nThank you very much for your valuable question.\n\nFollowing your suggestion, we further followed a very recent work FlowMM in ICML 24 published after the NeurIPS submission deadline to establish a fair comparison in terms of Validity, Stability, and S.U.N. It is worth noting that the calculation of stability and S.U.N. is very expensive (requires extensive DFT calculations) and only has been used by very recent works even published after NeurIPS deadline. We show the results in Table 8 attached above in general responses. It can be seen in the table that Mat2Seq achieves competitive results in terms of validity, stability, and S.U.N., even 28\\% better beyond FlowMM in terms of S.U.N. (stable, unique, and novel) that published in ICML 24 after NeurIPS deadline.\n\n4. Question 1\n\nSure, we'd like to mention that a concrete example of the Mat2Seq input for a crystal structure is shown in Figure 2 in the main paper. The whole Mat2Seq sequence for that crystal structure is \"formula Ag 4 Hg 2 I 8 \\n space_group_symbol I-4 \\n lattice_parameters \\n a 6.5361 b 6.5361 c 13.1629 \\n alpha 90.0000 beta 90.0000 gamma 90.0000 \\n Ag 2 0.0000 0.0000 0.0000 \\n Ag 2 0.0000 0.5000 0.7500 \\n Hg 2 0.0000 0.5000 0.2500 \\n I 8 0.2400 0.7596 0.6200 \\n\\n\", where all these symbols and numbers are mapped to int values by a mapping dictionary. \n\n5. Question 2\n\nThank you for your valuable question. The nature of Autoregressive language models is the conditional probabilities of the next token given its predecessors, $p(C_i|\\theta)=\\prod_{j=1}^{n_i}p(c_j | c_1:c_{j-1}; \\theta)$. Thus, if you want to establish a conditional distribution $p_\\theta (\\cdot|s)$ to generate  3D crystal structures possessing the property $s$, it is natural to place the target property values at the beginning of the text other than anywhere else.\n\n6. Question 3\n\nNo, the irreducible atom sets refer to the subset of the atoms in the minimum unit cell. For example, as you can see in the structure of Ag4Hg2I8 in Figure 2 in the main paper, there are 14 atoms in the minimum unit cell, however, the irreducible atom set only contains 2 Ag, 1 Hg, and 1 I. This is because the positions of other seven I atoms can be fully recovered by using I 0.2400 0.7596 0.6200 and the space group transformations for I-4 group, and that's exactly why there is a number 8 in \"I 8 0.2400 0.7596 0.6200\" after symbol I. \n\nWith these clarifications provided, we hope we addressed your questions and concerns. And if you have any other questions, we are more than willing to answer.\n\nYours sincerely, Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uhYk6mrlVo", "reviewer": "Reviewer_Sr8M", "summary": "In this paper, the authors focus on the application of language models in the field of material generation. Starting from CIF files that represent crystal unit cell structures, they primarily utilize the Niggli reduction method to organize structures under different translations, rotations, and unit cell expansions into a unique representation. This representation is then used for crystal generation and conditional generation tasks. Compared to CrystaLLM, the proposed method demonstrates better generalization capabilities across multiple tasks.", "questions": "1. In Section 3.1, the authors list two conditions for uniqueness and three types of transformations: translation, rotation, and unit cell expansion. It is suggested that Figure 1 should clearly illustrate all these transformations. Additionally, the \"Change Lattice\" operation in Figure 1 is confusing. Which type of transformation does this correspond to? Simply changing the length in the a-direction should not alter the atomic positions. Clarification is needed on whether the authors intended to illustrate unit cell expansion or some other operation.\n\n2. In Table 1, the authors demonstrate Mat2Seq's ability to recognize crystal uniqueness, but the specific evaluation details are not provided. How were the values obtained, especially the 30% proportion? The methodology and criteria for this evaluation should be clearly explained.\n\n3. In Section 4.3, while band gap is indeed an important property for semiconductors, it should be noted that whether the band gap is zero is a classification task distinguishing metals from non-metals. The authors' use of <0.5 as a threshold is inappropriate, especially when they mention that \"values from 0 to 0.5 are marked as 0,\" which exacerbates the issue. This oversimplification could lead to misleading results.\n\n4. In Section 4.2, in evaluating the ability to generate structures of new materials, the authors only indicate whether the generated structures match the given chemical formulas. It is recommended to include an evaluation of the atomic distance differences between the real and generated structures using RMSE to better assess the generation accuracy.\n\n5. In Section 4.3, compared to commonly used machine learning models for material screening, generative models have the potential to produce out-of-distribution (OOD) structures, aiding researchers in discovering new structures with desired properties. However, the authors do not inform readers about the chemical formula repetitions, structure repetitions, or generation errors in the generated structures. A standard should be defined to evaluate the generated structures based on validity, uniqueness, and novelty.", "limitations": "The authors utilize an existing unique unit cell representation method in the data preprocessing stage for language models. This method holds promise as a necessary step for future work. However, the evaluation does not provide sufficient evidence to demonstrate its application value. Additionally, the conditional generation applications mentioned by the authors are also quite limited, based on the evaluations provided in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors introduce a new process, Mat2Seq, which uniquely represents crystal structures using Niggli reduction. This approach ensures SE(3) invariance, periodic invariance, and completeness, avoiding data augmentation problem and shortening token length in language model.\n2. The paper showcases Mat2Seq’s ability to generalize to novel crystal structures that were not seen during training. This is a crucial aspect for practical applications and demonstrates the robustness of the method.", "weaknesses": "1. The authors utilized a unique representation method for crystal structures, which theoretically promises to become an essential step in data preprocessing. This representation method primarily aims to improve the model's ability to recognize equivariant transformations in materials. However, the evaluation does not demonstrate this method's generalization capability to equivariant structures. It remains unclear from the authors' validation whether this method help language model learns from SE(3) equivariance or if the improved performance is due to the new token representation.\n\n2. In the experimental validation in Section 4.2 and the conditional generation in Section 4.3, the evaluation of capabilities is very limited. The authors only provided the proportion of generated structures without detailing whether new structures can be generated, how many of the generated structures are valid, and accuracy of these generated structures. The generated structures need detailed explanation and evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Sr8M,\n\nThank you very much for your time invested in reviewing our work. For your concerns and questions, we provide point-to-point responses below.\n\n1. About authors' validation whether Mat2Seq helps LM learns from SE(3) equivariance or if the improved performance is due to the new token representation.\n\nThank you for this question. Actually, we'd like to point out that the only major difference between Mat2Seq and CrystaLLM is the SE(3) invariance 1D crystal sequence used, while the tokenization process is similar. To be specific, we used the exactly same model settings and very similar tokenization process following CrystaLLM, with the aim of showing the importance of SE(3) invariance and periodic invariance sequence representations. With CrystaLLM naturally served as the ablation of the inclusion of SE(3) invariance and periodic invariance sequence representations implemented by Mat2Seq, we show by comprehensive experimental results in the original manuscript (Tables 2 and 3) and in the attachment of the global rebuttal (Table 12) that Mat2Seq sequences will yield better performances.\n\nThus, by using CrystaLLM as a solid baseline and ablation, we believe the importance of SE(3) invariance and periodic invariance sequence representations for materials is clearly demonstrated. \n\n2. We slightly organized your questions and concerns for the evaluation of Section 4.2 (Weakness 2, Question 4) and 4.3 (Weakness 2, Question 5), and provided responses below.\n\nFirst of all, we agree with your point that for section 4.2 it is better to further include evaluation of the atomic distance differences between the real and generated structures using RMSE to better assess the generation accuracy. We provide detailed hit rate and RMSE in the new Table 12 attached above in general responses. It can be seen in the table that Mat2Seq significantly performs better than CrystaLLM in terms of Hit Rate (whether the generated structure matches with the literature structure), and RMSE.\n\nAdditionally, your suggestion for section 4.3 to include metrics like uniqueness, validity, and novelty is indeed very important for aiding researchers in discovering new structures with desired properties. Thus, we follow your suggestion and conduct evaluations for the generated crystals when conditioned towards lower or higher band gap values, with results shown in Table 9 attached above in general responses. It can be seen in Table 9 that Mat2Seq achieves remarkable validity with 88\\% for lower band gap condition and 90\\% for higher band gap condition, with good uniqueness of 98\\% and 92\\%, and good novelty of 86\\% and 99\\%.\n\n3. About figure 1 and \"Change Lattice\" operation.\n\nThank you for your suggestion. Currently we show two types of periodic transformations that will alter unit cell structures significantly to demonstrate the failures of previous methods. For sure, we can update Figure 1 to include more demonstrative examples of all transformations that can result in different unit cell structures once we can update the paper.\n\nFor your second question, the \"Change Lattice\" operation does not correspond to cell expansion. Let's use a simple example to better demonstrate this. For example, you have a 2D material with atom at the origin of a cell, with cell lattices (0, 1) and (1, 0). A change lattice operation means you can actually change the lattice vectors to (1, 0) and (1, 1) without changing the area (or volume for 3D structures) of the cell at all. It is still a minimum unit cell, but with different lattice structures. We will also add these texts in the appendix once we have the chance, to enhance the clarity.  \n\n4. How were the values obtained, especially the 30% proportion? \n\nThank you for your question. The evaluation is conducted as follows: (1) for all crystal structure in MP 20 dataset, we transform it to a different unit cell representation without changing the crystal structure at all, e.g., by shifting the periodic boundaries as shown in Figure 1, to obtain a second representation of the same crystal. (2) We feed the original and the second crystal structure into 1D sequence representations of different methods, and then compare the differences between the output of these two different inputs for the same crystal structure. If the sequence mismatches, e.g., the resultant coordinates or type for the first atom is different, then it is a failure. (3) We go through the whole dataset and calculate the success rate. 30% means only 30% of the structures before and after the transformation will have the same sequence representation when using CrystaLLM.\n\n5. The band gap zero is a classification task distinguishing metals from non-metals. This oversimplification could lead to misleading results.\n\nThank you for your question. However, we kindly disagree with you for this point. We want to mention that the band gap of a material is the energy difference between the top of the valence band and the bottom of the conduction band. It is calculated by subtracting the energy of the valence band maximum (E_v) from the energy of the conduction band minimum (E_c). We treat materials with Eg = E_c - E_v < 0.5 eV as a separate group, which will indeed include both true metals/semimetals with (Eg = 0) and small gap materials 0< Eg <0.5 eV), but we do not regard this group as pure \"metal\" group. Similarly, we separate materials with 0.5 <= Eg < 1.0 eV, 1.0 <= Eg < 1.5 eV, ... etc. as individual groups. This is just our current grouping protocol. For your interested applications, you can easily divide materials into different groups, e.g. separating Eg=0 as an individual group for metal electrode applications, and 0<Eg<0.5 as another group for mid-infared and thermoelectric applications.\n\nWith extensive additional experimental results and further clarifications, we hope we addressed your concerns and questions. If you have any other questions, we are more than willing to answer. \n\nYours sincerely, Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Th9rMwfewW", "reviewer": "Reviewer_vCYe", "summary": "This paper presents a novel approach for generating crystal materials using language models. The key innovation lies in the Mat2Seq method, which converts 3D crystal structures into 1D sequences while ensuring SE(3) and periodic invariance. This approach addresses the challenge of representing crystal structures in a way that is unique and invariant under different mathematical descriptions.", "questions": "1. For crystalline materials, isn't the original data already the most primitive unit? If we first apply Niggli cell reduction before using CIF files to describe unit cells, can we ensure a consistent representation?\n2. While generating continuous crystal structures in 3D space seems natural, I am curious whether this token generation method can ensure that the reconstructed 3D crystal structure is continuous. Specifically, can the generated sequences always be converted back into a meaningful crystal representation?", "limitations": "1. When performing conditional generation, it appears that generating structures for each specific property requires fine-tuning under specific conditions. This could result in high complexity in practical applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written, providing clear descriptions of the problem background and proposed methods.\n2. It tackles an interesting problem by using language models to generate crystal structures and introduces the consideration of SE(3) and periodic invariance for the first time.\n3. The experiments are comprehensive, covering standard benchmarks and attempting to discover crystals with specific properties, demonstrating the robustness and versatility of the method.\n4. Although the paper ensures the invariance of the generated structures, enforcing this invariance at the tokenizer level might limit the diversity of the generated structures to a small subspace. To obtain structures with different rotations, additional post-processing may be required, unlike models based directly in the spatial domain which can generate various rotational variants naturally.", "weaknesses": "1. The paper should discuss the computational complexity differences between language models and specialized crystal models, as language models generally have higher parameter counts and computational demands.\n2. The comparison of language models is limited. The experiments only compare with CrystalLLM, despite mentioning other methods.\n3. The rationale for comparing only with CrystalLLM in Section 4.2 and not with other methods from Section 4.1 should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vCYe,\n\nThank you for your recognition that our approach addresses the challenge of representing crystal structures in a unique and invariant way under different mathematical descriptions. For your concerns and questions, we provide point-to-point responses below.\n\n1. The paper should discuss the computational complexity differences between language models and specialized crystal models, as language models generally have higher parameter counts and computational demands.\n\nThank you very much for your suggestion, and we do think this is important to point out. Thus, we include the computational complexity and efficiency in generation comparisons with specialized crystal models like CDVAE and DiffCSP in Table 11 attached in the above general responses. As we can see in the table, LLMs based Mat2Seq indeed has more parameters, but with similar amount of GPU resources following DiffCSP running efficiency comparisons, Mat2Seq is significantly faster in generation. \n\nWe additionally show that decreasing the model size by 8 times will not influence the RMSE a lot and can further increase the running efficiency. For the smaller model, it achieved RMSE of 0.039 which is significantly better than DiffCSP with RMSE of 0.049, and more than 3 times faster in generation process.\n\n2. The comparison of language models is limited. The experiments only compare with CrystalLLM, despite mentioning other methods.\n\nThank you for your question. We'd like to mention that we use Table 1 to directly show the failures of other two LLMs for crystal generation, and crystalLLM from Meta team that are finetuned instead of training from scratch is not directly comparable due to unfair training logic (they are using pre-trained large language models). We tend to believe the crystal structure prediction task established by DiffCSP and CrystaLLM is a reasonable and powerful benchmark to compare with, but other LLMs based crystal generation methods are either not directly comparable, or provide no evaluations for this task.\n\n3. The rationale for comparing only with CrystalLLM in Section 4.2 and not with other methods from Section 4.1 should be clarified.\n\nThank you very much for this question. The reason for not including CDVAE and DiffCSP in Section 4.2 is also simple: we just want to compare with baselines in a fair way. For the tasks defined in Section 4.2, it requires the model to be trained on CrystaLLM dataset with 2.3 million crystal structures, while CDVAE and DiffCSP have only been trained on much smaller dataset like MP20 and MPTS52 with less than 50k training samples. It is unfair to compare with them, so we do not include them in the comparison in Section 4.2.\n\nAdditionally, the major contribution of this work, just as you mentioned before, is addressing the challenge of representing crystal structures in a unique and invariant way under different mathematical descriptions. To show this, we comprehensively compare our Mat2Seq with CrystaLLM that does not address this.\n\nFurthermore, we tend to believe the performance gains beyond CDVAE and DiffCSP are already clearly shown by comparisons in Section 4.1.\n\n4. For crystalline materials, isn't the original data already the most primitive unit? If we first apply Niggli cell reduction before using CIF files to describe unit cells, can we ensure a consistent representation?\n\nUnfortunately no. Although usually the original data is the most primitive unit cells, there are a lot of different most primitive unit cells for the same crystal structure, as shown in Figure 1, like shifting periodic boundaries. \n\nFor your second question, actually the baseline method CrystaLLM uses Niggli cell reduction before using CIF files, but as we can see in Table 1, it fails to maintain a consistent representation for the same crystal structure before and after periodic transformations that will not change the crystal structure at all.\n\n5. While generating continuous crystal structures in 3D space seems natural, I am curious whether this token generation method can ensure that the reconstructed 3D crystal structure is continuous. Specifically, can the generated sequences always be converted back into a meaningful crystal representation?\n\nTo show this, it is better to demonstrate the validity and stability of generated crystals from Mat2Seq. We have provided additional experiments for validity evaluations and stability evaluations in Table 8 attached above in the general responses. It can be seen that Mat2Seq achieves competitive validity ratio and remarkable stability ratios (around 50\\% of the generated structures can fall into the energy hull region of E_{hull} < 0.1 eV following FlowMM pipeline).\n\nFor your second question, as we can see from Table 8, CDVAE, DiffCSP, and the most recent FlowMM published after the NeurIPS deadline, none of them can guarantee a 100\\% validity rate, which means the generated crystals by these SOTA methods are not always meaningful. \n\n6. When performing conditional generation, it appears that generating structures for each specific property requires fine-tuning under specific conditions. This could result in high complexity in practical applications.\n\nYes, currently we tend to utilize the pretrained model on that 2.3 M dataset because this pretrained model is available and potentially powerful beyond training from scratch. This kind of finetuning will potentially introduce less complexity compared with training from scratch. We will release the pretrained model for others to use.\n\nAdditionally, a foundation model covering various properties require a huge amount of crystal structures with corresponding properties. This kind of dataset is very expensive to establish and currently out of our scope. And this is a promising direction to move forward.\n\nThank you again for your questions. If you have any other questions, we are more than willing to answer.\n\nYours sincerely, Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, ACs, and PCs,

We thank all reviewers for your time invested in reviewing our work, and appreciate your valuable suggestions. 

For all your (reviewers') questions and concerns, we provide detailed clarifications with additional experimental results, and some of these experiments are quite expensive to run like S.U.N. and stability (which require extensive DFT calculations). We attached detailed experimental results in the pdf file here for you to view.

To be specific, in Table 8, we add the comparison with a very recent SOTA method FlowMM that is published in ICML 24 after the NeurIPS submission deadline, to establish a fair comparison in terms of **Validity, Stability, and S.U.N.** (stable, unique, and novel). We follow the FlowMM pipeline and generate 1k materials to obtain these results.

In Table 9, we add the **unique ratio, validity ratio, and novelty ratio of generated materials conditioned towards low and high band gap values** detailed in Section 4.3.

In Table 10, we add the match rate (%) and RMSE for **experimentally observed** crystal structures in MP-20 test set.

In Table 11, we add the **efficiency in generation speed, and model complexity** comparisons with CDVAE and DiffCSP.

In Table 12, we add the **Hit rate (whether the generated structure matches with recently experimentally observed crystals from literature) and RMSE** for the 10 challenge crystal systems detailed in Section 4.2, compared with previous SOTA  method CrystaLLM.

With these extensive experiments and detailed further clarifications, we hope we addressed all your concerns and questions thoroughly. If you have any other questions or concerns, we are more than willing to answer. 

Yours sincerely, Authors

Meta-review: The paper presents a new representation for crystalline materials as 1D sequences, that can then be trained using an LLM. While prior works have already demonstrated methods to do this, the novelty of the current paper is in designing a unique & invariant representation for crystals. Through experiments on commonly used datasets, the authors show this new method, called Mat2Seq, is competitive with other methods in this space.

Strengths:
1. New approach for representing crystals in a unique and invariant manner.
2. Competitive results on benchmark datasets.

Weaknesses:
1. Limited novelty, compared to the well-known Niggli reduction.
2. The authors do not compare the method to state-of-the-art LLM approaches to this problem like Gruver et al. 2024.

Recommendation:
Despite the limitations, I vote to accept the paper as it presents a new and interesting method for representing crystal structures. The use of LLMs for material science has been growing rapidly in recent times, and the Mat2Seq representation could be valuable for this line of work.
Decision: Accept (poster)