Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Authors: Nate Gruver
Anuroop Sriram
Andrea Madotto
Andrew Gordon Wilson
C. Lawrence Zitnick
Zachary Ward Ulissi
Venue: ICLR 2024 poster (2024)
Keywords: generative model
large language model
stable materials
AI for science
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We fine-tune LLMs on string representations of crystals and generate new stable materials.

Abstract: We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90\% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned  LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49\% vs 28\%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.

Published as a conference paper at ICLR 2024
FINE-TUNER LANGUAGE MODELS GENERATE STABLE INORGANIC MATERIALS AS TEXT
Nate Gruver' Anuroop Sriram?
Andrea Madotto
Andrew Gordon Wilson C. Lawrence Zitmick?
Zachary Ulissi?
NYU
Meta FAIR
ABSTRACT
We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calcula- tions, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CD- VAE, a competing diffusion model. Because of text prompting's 's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.
INTRODUCTION
Large language models (LLMs) are trained to compress large text datasets, but can also act as strong foundations for non-text data (Delétang et al., 2023). As compressors, LLMs extract common patterns and find simple programs that can produce them (Goldblum et al., 2023; Sutskever, 2023), regardless of the data's origin. From text pretraining alone. LLMs can compress or extrapolate data as diverse as images (Delétang et al., 2023), tabular data (Goldblum et al., 2023), time series (Gruver et al., 2023a), or robotic trajectories (Mirchandani et al., 2023). Alongside generality, LLLM pre-training also gives rise to sample efficiency, as in-context learning and fine-tuning require far fewer training examples to identify salient patterns than training a model from scratch (Brown et al., 2020).
The generality and sample efficiency of LLMs make them particular promising for scientific problems, where data are often limited, collected from diverse sources, or challenging for non-experts to interpret. In materials science, for example, the number of known stable materials is relatively small, and the data describing each material are diverse, including composition, structure, and complex properties. LLMS can learn generalizable rules from small number of examples (Zhu et al., 2023), combine modalities into a single model (Moon et al., 2023), and provide users with a text-based interface. A text interface, in particular, has the potential to improve access scientific discovery (White, 2023); LLMs can use text to describe new observations, or, in design applications (e.g. materials design, drug discovery), LLMs can ingest text that specifies desired properties or constraints (Bran al., 2023).
In this work, we show that fine-tuned LLMs can generate the diree-dimensional structure of stable crystals as text (Figure 1). Our method is simple: first, encode crystals as new-line separated strings and combine with text instructions, then perform parameter efficient fine tuning (PEFT) on a base LLM (LLaMA-2) with multitask curriculum and translation augmentations (Section 4). We evaluate our method with Materials Project data (Jain et al., 2013), comparing against an invariant diffusion model and sequence model trained from scratch. Using both learned ML potentials and gold- standard DFT calculations, we show that our method can generate materials predicted to be stable at higher rates than baseline methods. To understand the success of our fine-tuning approach, probe the learned symmetry properties of our model, proposing a new metric for language models trained atomistic data and examining the effect of model scale learned invariance. Going beyond

---

Published as a conference paper at ICLR 2024
Distribution of Known Stable and Metastable Inorganic Materials
Augmentations
Pretrained LLaMA-2 7B/138/708
Unconditional Stable Materials Generation
LoRA
String Representation
Text-Conditional Materials Generation
CrystalLLM
Infilling for Pointwise Mutations
Prompting
Figure I: Overview of our approach to materials generation with large language models. Using string formatted crystals and task-specific prompting, we enable unconditional stable materials generation, text-condition materials generation, and structural infilling. Base LLaMA-2 models are fine-tuned on a database of known inorganic materials (Liu al., 2020) using low-rank adapters.
unconditional generation, we also show that our LLMs have other useful abilities within materials design, such as text-conditional generation and infilling, which can be used to optimize the properties of existing materials.'
2 RELATED WORK
There are two central challenges in applying generative models to crystals and related atomistic data. The first challenge is that atoms are intrinsically both discrete and continuous objects, as each atom has both an element identity and position in three dimensional space. Approaches to generative modeling often differ between for discrete and continuous data, and modeling both simultaneously can be significantly more complex than modeling either individually. The second key challenge is the prevalence of symmetries atomistic data. The unit cell, a repeated pattern tiled infinitely in every direction, is the common representation for crystals because it easily captures translation invariance, the fact that atoms can be shifted and wrapped around the unit cell while still representing the same underlying structure. Symmetries can pose challenges to deep learning models because they entail constraints on the functions that neural networks can learn.
Diffusion models Xie et al. (2021) introduced crystal diffusion variational autoencoder (CDVAE) to directly deal with both of these challenges. CDVAE uses several individual generative models for discrete and continuous components that share continuous (VAE) latent space. The chemical composition is reconstructed from this latent space using a language modeling head, while atom positions are generated with a denoising diffusion model (Ho et al., 2020). Since CDVAE, several works have extended diffusion processes to capture all parameters of the crystal, not just the atomic coordinates. Both Jiao et al. (2023) and Zeni et al. (2023) accomplish this by creating diffusions for the lattice parameters and atom identities, while Yang et al. (2023) design a new continuous representation that unifies atom identities and positions in a single high-dimensional tensor. In most cases, these diffusion models were designed with a careful eye towards symmetries and are built on top of graph neural networks with strict invariance/equivriian et al., 2023; Zeni et al., 2023). The approach of Yang et al. (2023) is more similar to ours, as they apply general-purye purposs architecture (3D U-net) and modeling approach (Gaussian diffusion) to new representation, without guaranteeing symmetries. Discrete atom identities and variable length (number of atoms), however, require special considerations in diffusion models, unlike standard language models, which were originally designed for modeling discrete sequences.
properties (Xie et al., 2021; Jiao
Language models Flam-Shepherd & Aspuru-Guziz (2023) demonstrate an alternative to continuous denoising models and architectural invariances. Instead of treating discrete and continuous modalities separately, as in CDVAE, "lam-Shephed & Aspuru- -uzik (2023) uses sequences of discrete tokens to represent everything, including the digits of atomic coordinates. With all data encoded as tokens, standard language modeling methods designed for text can be applied with little to modification. The simplicity of this method also makes it simple to adapt to many different kinds of molecular
'https:// thub. com/ http///aeooookeeeeeeeoo
2

---

Published as a conference paper at ICLR 2024
structures, including small molecules, protein binding pockets, and, of course, crystals. In lieu of architectural symmetries, augmentations of the training data are used to encourage learning known invariances. Flam-Shephers Shepherd & Aspuru-Guzik (2023) demonstrates that language models trained from scratch on many common molecular datasets actually outperform popular domain-specifie models, including CDVAE, in their ability capture valid element compositions and high-level statistics of the training data. Similarly, Antunes et al. (2023) also use language models to generate crystal structures as discrete sequences by training from scratch on millions of CIF strings.
Our work In this work, we show that pretrained LLMs are also useful for understanding and generating 3-dimensional atomic structures. By using a pre-trained LLM, we can achieve high rates of validity without crystal-specific tokenization -Flam-Shepher & Aspuru-Guzzi 2023) or millions of auxiliary structures (Antunes et al., 2023). Unlike many methods designed specifically for crystal structures and symmetries, our method can also be easily extended to multiple crystal generation tasks and, in the future, to other atomistic modalities without any changes to the underlying model or training procedure. Building on the basic observations made by (2023), we show that larger models, which are often more effective compressors of data, demonstrate improved ability to learn symmetries from the training data and augmentation.
& Espuru-Guzii
3 BACKGROUND
Language Modeling LLMs perform next-token prediction over sequences. The model is a cate- gorical distribution, (//w-1/Woo=). where Wo: is the prompt, a sequence of input tokens, and W++1 the predicted next token. To generate sequences from the model, the conditional distribution is sampled sequentially, but samples are rarely drawn from the original, unmodified categorical distributions. Instead the sampling procedure is typically modulated with temperature (r) and nucleus size (p) hyperparameters. Temperature serves to flatten the conditional distributions to uniform (high temperature) or collapse them around their maximal probabilities (low temperature). Nucleus size limits which tokens can be sampled based on the cumulative distribution function, clipping out values that contribute very little mass. A nucleus of p (0 < p S 1) corresponds to keeping tokens to cumulatively contribute p% of the total probability, and discarding the rest.
Tokenization To train language models on text datasets, strings are converted into sequences of tokens. Most modern LLMs rely on byte pair encoding (BPE) (Gage, 1994), compression method that assigns tokens to common substrings, making overall sequence lengths shorter. One downside of BPE tokenization is the default tokenization of numbers. BPE typically breaks numbers into irregular substrings instead of individual digits. While breaking numbers into multi-digit tokens creates shorter sequences, it also complicates learning basic arithmetic operations, which typically operate at the level of individual digits. Luckily, Touvron et al. (2023b) introduce tokenizers for LLaMA-2 models that break numbers into sequence of digits, which has been shown to dramatically improve performance on arithmetic tasks (Liu & Low, 2023). We use LLaMA models in our work because they have a natural representation of 3D coordinates and can therefore learn simple functions over those coordinates that obey domain-specifte symmetries (Section 5).
Crystal structures and energy prediction Periodic materials are defined by unit cell repeated infinitely along all three dimensions (Figure 2). The unit cell comprises a lattice (parallelepiped) with side lengths (l,, 12, 13) and angles (01, 02. 0z). Within the lattice, there are N atoms, each specified by an element identity, ei, and set of 3d coordinates (I;, Ur. zi) which can be absolute or fractional (specified as percentage of the unit cell side lengths). Therefore bulk material can be fully described by the tuple
C' = (1,, 12, la, 0,, 02, 0z, e1, 31, MI, Z1, CN, UN UN, ZN).
For given set of environmental conditions, every crystal has a corresponding energy that describes how likely it will occur in a particular configuration. Configuration with unfavorable electrostatic interactions from unlike atomic positions, such as highly overlapping atoms, are typically high energy. The gold standard for energy prediction is density functional theory (DFT), which provides tractable approximations to governing quantum mechanical equations that describe the energy and time evolution of a system. DFT, however, can be prohibitively expensive, often scaling O(z') with the system size, which has motivated development of deep learning potentials to approximate DFT solutions (Lan et al., 2022).
3

---

Published as a conference paper at ICLR 2024
Proposed (Unstable!)
Pure Component
Formation Energy [eV/atom)
Rofaronca States
5.1
Pm22nRh
60 60
Known
String Encoding
Pm
0.25 0.25
Materials
Unstable
Proposed (Metastable!)
null
Stable Materials
Proposed
(Stablel)
0.000.000000
oNotoo
Rh 0.50 0.50
Known Potentially Metastable Materiols
Stability
Pure A
Binary Material Composition
Pure
Figure 2: (left) We convert the crystal lattice, atom identities, and atom positions into strings. The model is trained generate structures conditioned on the text prompt, which might contain additional information about the composition, properties, or starting structure to modify. (right) Energy above hull (Emit) quantifies the stability of a material. A crystal with Enall < 0.1 will be energetically favorable both in its structure and composition.
Stability of hypothetical materials (Fhut) The composition of a crystal also impacts its energy. different elements have different geometries and charge properties. Certain stoichiometries, or ratios of elements, are naturally favored, and composition of elements A and B with constituent parts A, can dissociate into the composition A.Ba it energetically favorable. Because of the effect of composition, the energy of a crystal is typically two dimensional concept captured by the energy hull, which is the minimum observed configuration energy for given composition. For crystal to be low-energy and stable, and therefore give rise to a practically useful material, it must have a small energy above hull (Faui)), the distance from the energy hull for the erystals elemental composition (Figure 2). Crystals with Emall are considered stable and by definition have lower energy than the known minimum (which has Ebull = 0). Crystals with Emoll < 0.1 eV/atom are often metastable and likely to be practical useful (Sun al., 2016).
METHOD
Our approach to generating stable materials is pleasingly simple. We take a pre-trained LLM, which has useful biases towards generalizable patterns, and fine-tune it on crystal string representations. Because language models can also ingest text, we can condition the model's generations descriptions. The flexibility of language models also allows us to solve other tasks, such as infilling, through small modifications the input formatting. Though in this work, our method itself is general purpose and could be easily extended to proteins, nucleic acids, or small molecules. We include more detailed discussion of how general (ext-pretraining impacts our method in Appendix A.5.
text
focus solely crystal structures
String formatting and tokenization precision numbers. An example of crystal string formatting is shown in Figure 2. We represent lattice lengths with one decimal place (2-3 digits) and lattice angles as integers (1-3 digits). Fractional coordinates always represented with two digits. 3DD coordinates all other crystal components are combined with newlines. We deliberately chose LLaMA-2 models because they both state-of-the-arr in overall performance among open-source models and because they tokenize numbers as individual digits by default. Notably, it is therefore impossible to create one token per full number, as Flam Shepherd & Aspuru-Guzz -uzik (2023) do their best performing model (further discussion in Appendix A.1). Instead, we rely on the extensive pretraining of LLaMA-2 models instill useful biases over numerical operations (Liu & Low, 2023).
convert the crystal tuple (Equation 1) using fixed
combined with spaces and
To train model that can be used for many tasks, including unconditional generation, text-conditional generation, and infilling, we use task-specific prompts. The input to the model is prompt followed by the string- formatted crystal (Figure 2). In the most basic case, the prompt indicates that the model should generate bulk materials represented as lattice and atoms. The prompt also be expanded include desired composition or material properties, include starting
Prompt design
4

---

Published as a conference paper at ICLR 2024
structure, in the case of infilling. For infilling, the prompt includes the ttring-formater crystal with every instance of a randomly chosen element replaced with [MASK], and the model is trained to generate the identity of the masked element at the end of the sequence. During training all three tasks are included through random sampling, with two thirds generation and one third infilling (details in Appendix A.2). As in instruction tuning, the prompt is given input the model but does not contribute to the generative loss function. The model is only penalized for its predictions on the crystal string or masked element.
Generation Prompt
Infill Prompt

Below is a description of a bulk material. [The chemical formula is Pm2ZnRhj. Generate description of the lengths and angles of the lattice vectors then element type and coordinates for each atom within the lattice:

 Below is a partial description of a bulk material where one element has been replaced with the string "[MASK]":
I Crystal string with [MASK]s]
Generate an element that could replace [MASK] in the bulk material:
Crystal string |

[ Masked element <

Blue text is optional and included enable conditional generation. Purple text stands in for string encodings of atoms.
Augmentations Crystals structures are symmetric under translational. All atomic coordinates can be shifted modulo the lattice boundaries without changing the resulting material structure. Similarly, the ordering of atoms within the lattice is irrelevant to the underlying material (permutation invariance). Prior work on diffusion generative models guarantee these symmetries as invariance or cquivariance constraints on the model architecture (Xie et al., 2021; Jiao et al., 2023). To encourage translation invariance in our language models, apply random uniform translations to the fractional coordinates. We chose not to augment the ordering of atoms because these variables often contained valuable information, for example grouping of elements together for placement in the lattice (discussion in Appendix 1)
5 EXPERIMENTS
We explore several uses of language models in crystal generative modeling. First, in order to compare with prior work, we show that fine-tuned LLMs can be used for unconditional generation of novel materials and that the resulting materials correspond to stable relaxed structures under the predictions of an ML potential and DFT. We then show that LLMs can also be used for text-conditional generation and to propose small changes to existing materials.
Datasets and models For consistency with prior work (Xie et al., 2021; Flam-Shepherd et al., 2023) we used MP-20 (Jain et al., 2013), a dataset of 45231 materials, when training for unconditional generation. All structures in MP-20 are stable, and therefore an effective generative model trained MP-20 should tend to propose new crystals that at least metastable. For lext-conditioned generation, we train with all forms of prompting (Section 4) on collection 120,000 crystals from Materials Project (Appendix A.3). The collection includes basic property information, such as the space group number, band gap, Emuil and the chemical formula. All of our experiments were conducted with LLaMA-2 models (7B 13B, and 70B) (Touvron et al., 2023a;b) through the Transformers library (Wolf et al., 2020) and PyTorch (Paszke et al., 2019). In order to train on small number of GPUs we use 4-bit quantization (Dettmers et al., 2022) and Low-Rank Adapters (LoRA) (Hu et al., 2021). We provide the full hyperparameters and training details in Appendix A.4.
Evaluation For basic evaluation of the LLM samples, we use the validity and diversity metrics introduced by Xie et al. (2021). Structural validity is determined by non-overlapping atomic radii (overlapping taken to be both atoms within half radius of each other), while compositional validity captures the net charge of the structure (only structures with net neutral total charge valid). Diversity is computed as pairwise distance between samples under featurizations of the structure and composition from Matminer (Ward et al., 2018; Xie et al., 2021).
5

---

Published as a conference paper at ICLR 2024
Table 1: Following prior work (Xie al., 2021), we evaluate fine-tuned LLaMA-2 models using validity, which captures physical constraints, as well as coverage and property metrics, which capture alignment between the ground truth and sampling distribution. We add stability checks, which count the percentage of samples estimated to be stable by M3GNet (Chen & Ong, 2022) and DFT (Hafner, 2008) (details in Appendix B.2). LLaMA models generate a high percentage of both valid and stable materials.
Method
Validity Check
Coverage
Property Distribution wdist (Na))
Metastahle M3GNet
Stable DFT
Structuralf
Composition
Recallf
Precision
wdist (0))
CDVAE
1.00
0.867
0.991
0.995
0.688
1.43
28.8%
5.4% n/a
LM-CH
0.848
0.835
0.9925
0,9789
0.864
0.13
n/a
LM-AC
0.958
0.889
0.996
0,9855
0.696
0.09
n/a
n/a
LLaMA-2
(T 1.0)
0.879
0.969
0.960
3.85
35.1%
6.7%
7B 00.7)
0.964
0.933
0.911
0.949
3.61
1,06
35.0%
6.2%
13B 110
0,933
0,900
0,946
0.988
2.20
0.05
33.4%
8.7%
138 (.7)
0.955
0.924
0.889
0.979
2.13
0.10
38.0%
14.4%
70B (00)
0.965
0.863
0,968
0.983
1.72
0.55
35.4% %
10.0%
70B (.7)
0.996
0.954
0.858
0.989
0.81
0.44
49.8 %
10.6%
Fraction of stractures that are first prodicted by MBtiNer to hove EA < 0.3 eV/aton,, and then verified with DFT to have S
e.D.e.
While useful for sanity checking models, simple validity metrics only reflect subset of our real-world priorities in generating novel materials. Arguably the most important property that we hope to assess in samples is their predicted stability, which we can approximate by predicting the energy of relaxed structures. Using known materials and energy calculations from Materials Project we construct the ground truth energy convex hull and then calculate the approximate energy above hull, Emun- We chose two methods to estimate material stability:
ML potential: M3GNet (Chen & Ong, 2022) provides energy, force, and stress approxima- tions for crystal unit cells. For each sample we first run a relaxation using force and stress approximations then use the energy of the final structure.
DFT: We run a relaxation using the Density Functional Theory code VASP (Hafner, 2008) with INCAR settings chosen by Pymatgen (Ong et al., 2013). DFT is the more accurate, but also much more computationally intense, of the two options.
In both cases, results are compatible with Materials Project values (Jain et al., 2013) (Appendix B.1). Because DFT is prohibitively expensive for many use cases (often hours per calculation), we only use to double-check results obtained with ML potentials, and only run VASP calculations materials that have already been predicted metastable by M3GNet (<0.11 eV/atom Enat). The use of a M3GNet surrogate model is not perfect as many structures in Figure 4 (right) have energies above the expected 0.1 eV/atom threshold, but the structures largely close the hull compared to the broader distribution of materials generated.
Unconditional generation We sample 10,000 structures from each fine- tuned LLaMA model, parsing CIF from the generated string. We reject the sample and draw another if a CIF cannot be parsed from the sampled string, which guarantees all samples can be interpreted crystals but does not guarantee validity of the resulting crystal. We show the validity and predicted stability (Xie et al., 2021) of the resulting structures Table which shows that LLMs can achieve near- sear-perfect rates of structural and compositional validity. flyper-paramenn
6.7 6.76.
138.0
Ln
0.88 0.05 0.96 Ln
0.63 0.30 0.46
like temperature and nucleus size can be used to trade-off validity and stability of samples with their coverage (Appendix B.3). LLaMA-2 70B strikes an effective balance, generating high rates of stable materials with good coverage and diversity (Figure 4). By default, generation is completely unconstrained and therefore the model can hallucinates imaginary elements, for example "Ln," a common abbreviation for Lanthanide (Figure 3), but the problem can be easily avoided by constraining the tokens for element identities (Wolf et al., 2020).
In 0.38 0.55 0.96 In 0.13 0.80 0.46
Figure sample with "hallucinated" el- ement identities (Ln).

---

Published as a conference paper at ICLR 2024
M3GNet
VASP
SmCdPd2
ZrScPd2
# structures
CDVAE
LLAMA-2 70B
200
Zr6Ai2Co
LiMnHO2
0.25
0.00
0.25
0.50
0.75
1.00
0.0
0.5
Enui (eV/atom)
Enwil (eV/atom)
Figure 4: Stability of LLaMA samples compared CDVAE (Xie et al., 2021). Fine-tuned LLaMA-2 70B generates a higher rate of metastable (Fhall < 0.1) and stable materials than CDVAE, using estimates of Enan from both M3GNet (Chen & Ong, 2022) and VASP (Hafner, 2008). Because of computational cost, only run VASP structures predicted to be stable by M3GNet. Stable materials generated by LLaMA are also more diverse (as quantified by Matminer featurization (Ward et al., 2018)) than stable samples from CDVAE. We include sampled stable structures, shown as (2,2,2) supercells, which display high-degree of regularity and understanding of three-dimensional space.
Symmetry learning As crystal structures have translational symmetry, ideally our model's likeli- hood should be invariant to translations. We propose Increase in Perplexity under Transformation (IPT) as metric for assessing the invariance language models to continuous group transformations. For a transformation group G with group elements g and group action t, we define IPT for an input s,
(s) = (PPL((((())) (PL (s))
7B
13B
70B
where
0.02
g = min (f (s))
40
and PPL is the perplexity of the sequence. the exponent of the length-normalized cross entropy loss, PPL(ss = CB())/n In our case G the group of translation, where each g is distance to translate by, and bg is the mapping that decode the string, translates the coordinates (wrapping them around the boundary), and re-encodes the string. IPT captures the degree to which transformations change a language model's compression ability. Good understanding of group transformations and invariance in the data should lead to minimal change in the perplexity of transformed sequence. We approximate IPT by sampling many values of g (e.g. 20), picking g as the minimum among those values, and computing sample mean. Figure 5 shows the mean IPT of 500 random crystals from the test set, for each of the three LLaMA model sizes. We include additional details about our IPT calculation in Appendix B.5.
0.01
20
0.00
Invariance Error (IPT)
Percent Metastable
Translation invariance on test data and ability to generate stable materials increase in propor- tion. Larger models learn invari- ances from augmentations more ef- fectively during training, likely as result of their preference for ab- stract and compressible patterns.
Figure 5:
Diversity, novelty, and sampling speed When using generative models to discover new stable materials, there several properties beyond the rate of stability that are practically significant. Novel and diverse samples encourage sufficient exploration of unknown material space, and sampling speed dictates how expensive it to search within that space. We compare these properties for LLaMA-2 models and CDVAE in Figure 6. To calculate diversity and novelty, we use the same featurizations as in Table 1, calculating pairwise distances for diversity and distance the closest neighbor in the training set for novelty (details in Appendix B.6). All metrics are computed over crystals judged metastable by M3GNet, so that all novelty and diversity are relevant and not an artifact of invalid generations. LLaMA-2 samples match or exceed the diversity of CDVAE samples and also obtain high rates of novelty when we consider both composition and structure. Interestingly, larger LLaMA models display less novel structures but more novel compositions. It's worth noting, however, that both CDVAE and LLaMA-2 7B far exceed the structural novelty of a held out test set, while 13B and 70B are just slightly lower. To judge sampling speed, calculate the time required for 10,000
7

---

Published as a conference paper at ICLR 2024
7B 1.0)
138 (T 1.0)
70B (T 1.0)
CDVAE
7B 0.7)
13B (T 0.7)
70B 0.7)
(Normed to Test Set)
1.5
Score
1.0
Hours
2-
0.5
1
0.0
0
Structure Composition Structure Composition
Either
10,000
10,000 Metastable
Diversity
Diversity Novelty
Novelty
Novelty
Samples
Figure 6: We compare LLaMA-2 models with CDVAE in their ability to generate novel and diverse samples as well as their overall speed. (left) We calculate diversity and novelty using a featurization of structure and composition (as in Table 1). Diversity is calculated as pairwise distance in feature space, while novelty quantifies the percentage of inputs that are far from the training set (Appendix B.6), All metrics are calculated only for samples that were already judged to be metastable. LLaMA-2 models often generate more diverse samples than CDVAE, and achieve similar overall rates of novelty, Interestingly, structural novelty is lower in larger models, while compositional novelty is higher. (right) We compare the time required to generate 10,000 samples from each model. run LLaMA-2 models with the largest feasible batch size on one A 100 GPU (Appendix B.7). While the largest LLaMA model is computationally expensive, smaller language models are very fast, especially when we consider both sampling speed and rate of stability.
samples, using the largest possible batch size on one A100 GPU (Appendix B.7). In Figure 6, we compare the sampling speed with CDVAE and find that smaller models are often significantly faster when generating metastable samples.
Text-conditioned generation Extending method to text-conditional generation is simple as including additional information in the prompt, with a small amount of additional text (Figure 4). We explore conditioning on spacegroup number, composition, and Enuil+ as these properties easy to verify (at least approximately) in silico. We assess the model's ability to perform conditional generation by comparing the intended condition with labels obtained from in-silico oracle for the constraint. For the chemical formula, we simply parse the composition from the generated CIF. For space group determination, we use pymatgen's 's SpacegroupAnalyyee angstroms (Ong et al., 2013). For stability, we use M3GNet to estimate Emoll as before. Using the oracle's 's labels, we then compute the percentage cases which the condition was properly met (Figure 7). The model is able to generate material with the correct composition the majority of the time but becomes less reliable as the number of atoms in the chemical formula increases. Space group conditioning is more challenging, as it requires precise control and understanding of 3D structure, but the observed 24% is impressive when considering the 230 possible space groups. Generating stable/unstable structures as binary task is the most challenging. likely because the training dataset is predominantly stable compounds and stability is defined only in reference to existing compounds. Stability is most easily controlled by modulating sampling hyperparameters.
with precision of 0.2
Infilling Existing Materials In many practical settings, sampling and filtering materials from scratch is unnecessary. Good starting materials are often known, and manufacturing processes are easier to adapt to related compositions than develop completely from scratch by making small edits to their composition -ofeen referred to as template methods (Kirklin et al., 2015; Saal et al., 2013). To emulate a typical template method, we construct lookup table that maps each element to elements that have similar atom radius when in the same oxidation state (code in Appendix C). We choose an element uniformly at random and swap with a random element chosen from the table. The resulting structure is then relaxed using M3GNet. To improve this strategy using fine-tuned LLM, we used the infilling prompt (Section 4) to obtain a distribution over elements (modulated with temperature 7) which we use instead of a uniform distribution over swaps. To evaluate mutation procedure, we sample 3000 structures randomly from the test set and generate perform one mutation mutation-relaxatteo relaxatiien step for each, using both uniform and language model-guided sampling. In Figure, we show the
8

---

Published as a conference paper at ICLR 2024
Condition Satisfied
of template method]
uniform
Below is description ...
Random
Below is partial description
Normalized vallue
1.5
T=
6/2 % Stable
T-0.5
formula PrAIO3
61%
57%
52%
56%
[MASK]
1.0
T=0.1
space group 221
2000
$10 alon
222 atart
0.92 0.34 0.75
constrain
E above hull 0.011
24%
5.0
Generate ... P(element)
Generate
Formula
Space Ess
S Se
Comp. Struc. Perc. Div.
Group
Div.
stable
Figure 7: Text-coonitional generation and infilling of existing structures with fine-tuned LLMs. (left) Including composition or property information (sampled from hold-out set) in the text prompt leads to a high rate of samples with the desired composition/property (space group or stability). We bin stability as Emoll < 0.1 (metastable) and Émull > 0.1 (unstable) for simplicity. Complex formulas and space groups challenge the model, but the samples are correct at rate that facilitates practical use. We also show the rate of samples that both satisfy the condition and are predicted to be metastable by M3GNet. (right) Using the infilling prompt we can select mutations to existing materials. LLaMA-2 70B proposes a distribution over elements, which we constrain using knowledge of atom radii and charge interactions. We sample mutations with temperature T and relax the results structure with M3GNet. When we apply this mutation procedure, obtain more stable materials per mutation, with negligible changes to the overall diversity of the stable materials.
percentage of stable compounds and diversity in the stable compounds for the uniform baseline and LLaMA-2 70B with different temperature values. LLaMA-2 70B proposes elements that lead to stable structures at a higher rate than the baseline template method without sacrificing diversity.
6 DISCUSSION
By generating a high rate of plausible stable materials (verified by DFT), we have demonstrated LLMs can be tfate-of-the-art generative models for atomistic domains with direct application of parameter-effiiien instruction tuning and minimal task-specific modeling choices. This approach to generative modeling opens the door to multitask capabilities within single sampling paradigm and multimodal training on atoms and text (e.g. to extract knowledge from large corpus of scientific papers). We also advocate for the use of evaluation metrics (e.g. Fnan) for generative models that are more closely tied to the downstream task of generating stable or metastable materials. The space of all hypothetical materials is combinatorially large (consider all the ways to pack 20 arbitrary elements into a box), but only a small subset of materials will actually be stable or metastable. Models that can directly generate near-stable structures make all downstream tasks far easier, and increases the likelihood the generative models may be useful for day-to-day tasks in materials discovery.
Limitations Our method shares the limitations of the underlying generative models. LLMs can be sensitive to precise details of the chosen prompt and the tokenization strategies, particularly in how tokenization effects processing of numbers. Hallucination of unphysical chemical elements or structures has been observed, though fortunately is easy to check and filter. Text-conditioning has the potential tap latent conceptual understanding in the underlying LLM, but training LLMs that successfully leverage scientific and chemistry literature is major outstanding challenge. Lastly, training the largest of our LLMs can be prohibitively expensive for some computational budgets. Despite this, inference from all LLMs is often highly tractable when compared to baseline methods (Appendix B.7)
Future directions There is substantial room for improvement in conditional generation, which could be used to directly generate materials with desired properties. While we did not pursue alternative sampling strategies in depth, approaches like classifier-free guidance (Sanchez et al., 2023) or variants of PPLM (Dathathri et al., 2019) might be useful in combination with fine-tuned LLMs to improve conditional generation. These methods could also be combined with primitives from Bayesian optimization for sample-efficien and uncertainty- design (Stanton et al., 2022; Gruver al., 2023b).

---

Published as a conference paper at ICLR 2024
REFERENCES
Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespo Crystal structure generation with autoregressive large language modeling. arXiv preprint arXiv:2307.043300 2023.
Stephen Balaban.
NVIDIA A100
GPU
Benchmarks for Deep Learning.
Lambda Labs
Blog, 2020.
URL https::/Lembdalabbs wtw.bllg..ccoooooo///oo///
Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller, Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXivv2304.053776 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Compulationnl Science, 2(11):718-77) 2022.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv: 19/2.02 164, 2019.
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10660 2023.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. 9th International Conference on Learning Representations, ICLR, 2022.
Daniel Flam-Shepherd and Alán Aspuru-Guzik Language models can generate molecules, materials, and protein binding sites directly three dimensions as xyz, cif, and pdb files. arXiv preprint arXiv:2305.05708, 2023.
Daniel Flam-Shepherd, Kevin Zhu, and Alán Aspuru-Guzik, Atom-by-atom protein generation and beyond with language models. arXiv preprint wwXiv:2308.09482 2023.
Philip Gage, A new algorithm for data compression. The C Users Journal archive, 12:23-38, 1994.
Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05360 2023.
Nate Gruver, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie derivative for measuring learned equivariance. arXiv preprint arXiv:2210002984, 2022.
Nate Gruver, Mare Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models zero-shot time series forecasters. In shirty-evennth Conference on Neural Information Processing Systems, 2023a.
Nate Gruver, Samuel Stanton, Nathan C Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. arXiv preprint 2rXin:2305.20009 2023b.
Jürgen Hafner. Ab-initio simulations of materials using vasp: Density-functional theory and beyond. Journal of computational chemistry, 29(1)):0044-207 2008.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 68510 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685. 2021.
10

---

Published as a conference paper at ICLR 2024
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013.
Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion on lattices and fractional coordinates. In Workshop on" Machinee Learning for Materials" Materials"ICIA "ICLR 2023, 2023.
Scott Kirklin, James E Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The open quantum materials database (oqmd): assessing the accuracy of dft formation energies. npj Computational Materials, 1(I):1-15, 2015.
Janice Lan, Aini Palizhati, Muhammed Shuaibi, Brandon M Wood, Brook Wander, Abhishek Das, Matt Uyttendacle, Lawrence Zitnick, and Zachary W Ulissi. Adsorbml: Accelcrating adsorption energy calculations with machine learning. arXiv preprint arXiv:2211.1664 2022.
Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned Ilama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305. 14201, 2023.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models general pattern machines. arXiv preprint 2077022307.0472,, 2023.
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Ych, Prakash Murugesan, Peyman Heidari, Yue Liu, et al. Anymal: An efficient and scalable any-modality augmented language model. arXiv preprint arXiv:2309. 16058, 2023.
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 18:314-3194 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performncee deep learning library. In Neural Information Processing Systems, 2019.
James E Saal, Scott Kirklin, Muratahan Aykol, Bryce Meredig, and Christopher Wolverton. Materials design and discovery with high- throughput density functional theory: the open quantum materials database (oqmd). Jom, 65:1501-1500 1509. 2013.
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint (xXiv:2306.1786 2023.
Philipp Schmid. LLaMA Amazon Sagemaker, Benchmark. blog/1les aama saama--sagmman www.saaee-eaeaeaeeeeeeaeee wwwaaaaeeerbeeenbmbreeeeea benchmark, 2023.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning, pp. 20459-20478. PMLR, 2022.
Wenhao Sun, Stephen T Dacek, Shyue Ping Ong, Geoffroy Hautier, Anubhav Jain, William D Richards, Anthony C Gamst, Kristin A Persson, and Gerbrand Ceder. The thermodynamic scale of inorganic crystalline metastability. Science advances, 2(1 1)e 6002255, 2016.

---

Published as a conference paper at ICLR 2024
Ilya Sutskever. An observation generalization. Workshop Large Language Models and Transformers, 2023. URL htt /www. www.byoutbe... www.uuube...owaaaeeeeeooeooooeee wtt.:////wwttttttttttetc TVV
chamee=SSionsImstiiiie
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13371, 2023a.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan. Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09228, 2023b.
Logan Ward, Alexander Dunn, Alireza Faghaninia, Nils ER Zimmermann, Saurabh Bajaj. Qi Wang, Joseph Montoya, Jiming Chen, Kyle Bystrom, Maxwell Dylla, et al. Matminer: An open source toolkit for materials data mining. Computational Materials Science, 152:60-69, 2018.
Andrew D White. The future of chemistry is language. Nature Reviews Chemistry, pp. 1-2, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL hwwps::tp/.... : www. . acl web. . org/ an// hol ogy 2020 0. emn pp-dems.... 6.
Tian Xie, Xiang Fu, Octavian Brniinn-Euggn&&n Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint 1rX0v:2110.06197, 2021.
Mengjiao Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mor- datch, and Ekin Dogus Cubuk. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235, 2023.
Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabbé, Lixin Sun, Jake Smith, et al. Mattergen: a generative model for inorganic materials design. arXiv preprint arXiv:2312.036677 2023.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules. arXiv preprint arXiv:2310.0708 2023.
12

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Training Details
13
A.1 Numerical Formatting
13
A.2 Training with Stochastic Prompts
13
A.3 Extended Materials Project Dataset
14
A.4 Training Hyperparameters Details
A.5 Role of Text Pretraining
14
B Model Evaluation
15
B.
Evaluation with ML potentials and DFT
15
B.2
Stability Checks and Percentages
15
B.3
Trade-Off's in Sampling
15
B.4 "Hallucination" Examples
16
B.5 Increase in Perplexity under Transformation (IPT)
18
B.6 Diversity and Novelty Calculation
18
Sampling Speed
C Template Method Baseline
19
A TRAINING DETAILS
A.1 NUMERICAL FORMATTING
Notably, approach to tokenization distinctly different from prior work modeling atomic structures with language models. Instead of using a special vocabulary and training models from scratch, we use LLaMA-2's existing tokenizer. This choice allows us casily process both encoded crystals and text data. In early experiments, we tried out many other approaches, including fine- tuning LLaMA-2 models with additional tokens specific crystal data. These methods were more challenging to train and didn't 't lead to any improvements over using a shared tokenizer. We include set of example training losses below:
Epoch
Epoch 2
Epoch
Epoch 4
Epoch 5
Special Crystal Tokens
0.783
0.693
0.623
0.611
0.588
Shared Tokenization
0.457
0.432
0.424
0.401
0.385
There are many important decisions involved both in text formatting (c.g the choice of fractional or ab- solute coordinates) and augmentation of the input data (e.g. translation or permutation augmentations on coordinates). As simple example, we provide average validity numbers (using low temperature sampling) from earlier experiments on LLaMA-2 7B models trained with different formatting styles
Setting
Structural Validity
Compositional Validity
Fractional coords Absolute coords
91.4%
83.2%
90.8%
80.5%
No permutations
92.5%
82.9%
With permutations
89.2%
81.7%
A.2 TRAINING WITH STOCHASTIC PROMPTS
In order to enable multi-task use the fine-tuned LLMs, we train on a stochastically generated prompt. Two thirds of the time provide the model with generation task, in which the prompt
13

---

Published as a conference paper at ICLR 2024
consists of basic instruction to generate bulk material lattice and atom positions. We randomly sample a set of properties from the available descriptors of given crystal and add any chosen ones (if any) to the prompt, using small amount of wrapper text. The remaining provide use the sampled crystal to construct and infilling task. We choose on element randomly from the set of elements in the composition and we construct a prompt that contain the string encoding of the crystal with this element replaced with [MASK]. The model then generates the replaced element as text following the prompt.
third of the time,
A.3 EXTENDED MATERIALS PROJECT DATASET
To facilitate condiioniiional generation, we extend the original CDVAE training dataset with materials from Materials Project (Jain et al., 2013) as of April 2023. We filter out crystal with more than 30 atoms in the unit cell, which slow down training with minimal benefit to model performance, leaving training that contains 127609 crystal structures. The original validation and test splits are left unchanged and all test/validation points are removed from the new training set.
A.4 TRAINING HYPERPARAMETEN AND DETAILS
We provide the training details per model:
LLaMA-2 7B: Batch size of 256 for 65 epochs with cosine annealed learning rate of 0.0005. LoRA rank 8 and alpha 32.
LLaMA-2 13B: Batch size of 256 for 44 epochs with a cosine annealed learning rate of 0.0005. LoRA rank 8 and alpha 32.
LLLMA-2 70B: Batch size of 32 for 21 epochs with a cosine annealed learning rate of 0.0005. LoRA rank 8 and alpha 32.
Limitations around available compute lead to our use of differing batch sizes and total number of epochs for each model. Ideally, we would train all models with the largest batch sized used among all models and would train all models for the same number of epochs (the maximum used by any model). At the same time, we wanted to properly demonstrate the full potential of all model sizes and therefore chose to present results for the best model were able to train at each model size.
A.5 ROLE OF TEXT PRETRAINING
Text pretraining is essential to our method for two reasons.
1. It would be impractically expensive or computationally infeasible train models with up to 70B parameters from scratch on our data. Using a pretrained model with LoRA (Hu et al., 2021) offers the benefits of model scale while maintaining tractability and limiting overfitting, as the actual number of trainable parameters can be relatively small.
2. Pretraining text data yields model that be conditioned text for free, and text conditioning opens up a huge new realm of exciting possibilities, like conditioning samples desired properties. It would challenging achieve a similar result from scratch without significantly expanding the size of the dataset (to improve general text understanding) and without essentially training general- purposs language model in the process.
To better understand the first point, let's quickly review the exact details of the finetuning procedure. We are using low-rank adapters (LoRA), as opposed to end-to-end finetuning, and this means we are adding a small number of additional parameters to an existing. frozen model. The easiest way to see the difference between this approach and training a model from scratch-as in (Flam-Shepherd & Aspuru-Guzik, 2023) to compare the training loss over the first few epochs of training.
Model
Epoch
Epoch 2
Epoch 3
Epoch 4
Epoch 5
GPT-2 (from scratch)
0.946
0.878
0.807
0.757
0,740
LLaMA- 13B (LoRA)
0.457
0,432
0.424
0.401
0.385
LLaMA-700 (LoRA)
0.402
0.344
0.325
0.305
0.296
14

---

Published as a conference paper at ICLR 2024
If attempt to LoRA finetuning with randomly initialized parameters for the LLaMA-2 7B model we observe an immediate and significant difference the training losses:
Model
Iter
0.33 Epochs
0.66 Epochs
Epoch
Random
13.46
1.53
0.81
0.78
Pre-trained 1.57
0.47
0.41
0.39
While LoRA finetuning is tractable because 99.95% of the model is frozen, finetuning LLaMA-2 model end-to-end in half-precision would require at least 4 times as many GPUs, making it infeasible for all but handful of researchers. When using LoRA, even though the base models are large the number of trainable parameters is very small. In fact, the LLamA-2 7B model has less trainable
parameters than one of the baseline methods we compared (CDVAE) (Xie et al., 2021). The number of trainable parameters for each of our models and the baseline models is shown below:
Model CDVAE LM-CH/AC
Trainable parameters (millions)
Percentage of total
4.5
100%
1-100
100%
LLaMA-2 7B
3,5
0.05%
LLaMA-2 13B
6.5
0.05%
LLaMA-2 70B
35
0.05%
B MODEL EVALUATION
B.I EVALUATION WITH ML POTENTIALS AND DFT
Approximating Fhull from the energies of known materials in Materials Project requires a consistent correction scheme. We touch on some of the details here.
M3GNet Importantly, M3GNet was trained on the total energy of VASP calculations in the Mate- rials Project dataset, so the results were expected to be consistent with the correction schemes and absolute energies Section
VASP To be consistent with the Materials Project settings (e.g. the PBE functional, DFT/DFT+U as appropriate, consistent pseudopotentials, etc). We did a single relaxation for every candidate structure using the default parameters in MPRelaxSet (Ong et al., 2013). VASP relaxations were run using the GPU-accelerated VASP6 code.
In both situations, the total energies were corrected using the MP2020 compatibility scheme, which was important to maintain consistency when calculating formation energies, and allow the use of varying functionals (DFT/DFT+U) for different materials.
B.2 STABILITY CHECKS AND PERCENTAGES
To calculate the percentage of metastable compounds, take all samples and remove samples that are invalid under the basic structure and composition checks. We then run relaxations with M3GNet and obtain the final relaxation energies. The final percentage takes into account both the rate of validity (used to perform the initial filtering), and the rate of compounds with Fmuil 0.1, as determined by the convex hull calculation using the M3GNet relaxation energy. To calculate the VASP percentage, we select materials determined be metastable M3GNet and run VASP with default setting. We then report the percentage of the materials with EpTil 0.0.
B.3 TRADE-OFFE IN SAMPLING
We note that modulating stability with sampling parameters like temperature and nucleus size has a significant effect on the coverage properties of the resulting samples. We illustrate the trade-offs between stability and coverage in Figure 8. Coverage most likely decreases because nucleus size and temperature collapse the distribution around samples with high likelihood, which are also more likely to be valid stable. Notably, I.L.MA-2 70B appears to demonstrate the best trade-offs, possibly
15

---

Published as a conference paper at ICLR 2024
LLaMA-2 7B
LLaMA-2 13B
LLaMA-2 70B
CDVAE
Recall (stable)
Recall (all)
0.95
0.95
0.90
0.90
0.85
0.7
0.8
0.9
0.3
0.4 0.5
Validity
Stability
Figure 8: Validity and rate of stability depend on sampling sapee-parameters.. Lowering the tempera- ture or restricting the nucleus size leads to significant improvements in validity/stability but incurs cost coverage held-out between coverage and stability, generating materials that are both stable and diverse.
(recall). Fine-tuned LLaMA-2 70B displays the best trade-off'
indicating a likelihood model that corresponds better to both the underlying properties of stability and the full, diverse distribution of structures.
B.4 "HALLUCINATION" EXAMPLES
LLaMA-278 7B:
generated using Pymargen
dato_Mert8 (Ca2N)
symmetry. space_gronp, group. name, name, H-M "P
_cell_length_a 5.0000
_celi__ength_b 5.0000
data_L3Li
_cell_length_e
symmmtry
www.c_pppapppp.peppppppppppp group
celli_aggle_app alpha 90.0000
_eell _cell_lengt___ 5.1000
_cell__nglee_bet 90.0000
_cell sell_length_b 3.1000
_cell_angle__e _ammle__ammama... 90.0000
sell_length___ 7,4000
_symmetry_ Lat___ables_ummaa
aell_angie_alphe
84.0000
chemical formula structural
MeI8(Gi2N)
cell angle heta 68.0000
_chemical__ee forlll_oommla_aaaa sum
MMt8 Culo N5'
cetl__ngle__ppppeeeee palmaa 68.0000
_cell_volume 125.0000
_yymettry_nnt haayy__nn__aale__mmeeeee
eell formula units loop_
chemical formula structural 131.
eehemical_fateeee formuln sum "L12 L14
_ymmetry-_ _ew.ie_pps_________ site_id
_cell_ volume 230. 130.15214369
symmotry equiv pas xyz
formala loop
x. y.
toop_
_symmetry_equue _euily_eeuu__p_eeeee _oss-__o___eeeee
arom. site symhol
symmetry equiv
_atom_site_labes
.atom site wite..yymttrymmmllillll multippiilyyl
loop
.atom aatom_stts site. frnct_x
atom site type. symbol
siom_sits_ffactt
_atam_ site label
symmeiry
mammityymullippliin
site. occupancy
atom site. fracr_
Ca Cull
1.5300
0.3900
1.0000
_atom_stte [oom_stte_faiie srae_rraett_
Cn Cal 0.8300
0.4900
1,0000
atom site fract.
0.5300
occupancy
Cu Cu3
0.6300
0.1900
0,2000
0.7100
0.4000
0.8300
Co Ca4
0.2300
0.7900
D. 2000
L5
Lil
0.2200
0. 0.3700
0.3600
Co Ca5
0.6300
0.7000
0.3100
La
L12
0.7100
0.8930
0.3300
0.2300
Ca
1.0000
0,7000
14
1,0000
0.6300
0.6900
Ca Ca8
1.0000
0.5900
2,2000
L0-
L.5
0.5100
0.1400
0.6600
Cu CuP
0.4900
0,8900
0,7000
LO- L6
0.9600
0.5700
0.1700
Mat0=
Me1lO
0.6300
0.1700
Met0-
Meill
0,5000
0.4700
0.4700
L0-
L8
0.9800
0.6100
0.1400
Mat-
Meil2
0,4000
1.0000
0, 9800
L0-
1,9
0.4900
0.00
0.0400
Math=
Meil3
1,0000
0.2200
0.9700
1.--
1,10
1,0000
0.1000
0.6800
Mat0-
MellS
0,2300
0,2200
0.6000
L0- L,12
0,4700
0,5500
0.1800
Mamt
Mell6
1,0000
0.0000
0.6100
1.0-
1.13
1.0000
0.5800
0.6800
Met0-
Mell7
0.6300
0.1000
0,5000
L0- L14
0.4700
0.0600
0.6700
0.7000
0300
N19
0.2300
0.5900
0.2000
N20
0.2300
0.1900
0.7000
N21
0.4900
0.2100
0.00
LLaMA-2 13B:
16

---

Published as a conference paper at ICLR 2024
Leta_ (N0312
"P
_cell_ieagth_a
_celi_length_b
5.9000
_cell_length_c
5.9000
data MandeGi200
_ell__angle_alphe
hell_aagle__bet
$9.0000
ccel longth 3.6000
Srametty_sspce__et prrpp_ namep_nme___e
cell_angle_gamal
59.0000
_cell Sell_length__ 3.6000
Cthemica_.formula_cco
(N01)
cell lelghl__ 3,9000
_cell_volume 141.91223582
_chemical_formnll
"Lel
cell_angle_ _ammaa.mmmm..mmmmmm 90,0000
_ell__ngzle_beta 90.0000
angle alpha
cell_foormula_aitte
symmetry Int Tables number
loap-
_symmetry_
eewprisss__s____________ pos_site_le
[eemical_formula_tteeee chemical chemiccal_ormuuly Jormula, sum Mandel G82 Of
MamdeGd204
syymmtry wew.i__pss_ass.s ao__as__xz
_cell_ volume 76.46400800
x.
_ell__oormula_uiic loop
loop
_Atom_sit "roo__site__ppp htt.__ppe__yyolllle__lppo
_atom_site site_label
_symeetry_cquu _et_iii_pss_steeeee_
ntom site symmerry multiplicity
_symmetry__quue poo_p___s_v_____________
fract_x
stom_site_frract
loop
atom site. Fracr fite_rrae___
_atum_ site www.yyppppyyooooo
_atom_site _ib__sttt_abellllleee___________
atom, site__tteee sec_poccyynnyy
Len+
0,7100
0,5000
0.1700
_atom_sile__ine _it___it__rre_______
B2
0.4680
0.1600
0.5700
_com_site_fract__
B3
0.4600
0.7980
0.7300
0.6500
0.5700
_atom_site wwwrssitt___ceee oiteoppscc
_tomm_ @erm_site__aeee
B6
1.0000
0.0000
0.7900
0.2500
0.0900
MandeO+
Gdl
Monde2
0.8200
0.3200
0.2300
0.7300
0.1500
0.6300
0.8900
0. 0.0080
0.1600
0.6500
03
0.8200
0.7300
0,4100
O
OID
OIL
0.0680
0.0000
0.5500
0.3300
0.5400
0.7600
0.2500
0.7100
0.6100
06
0s 0.5200
0.8200
0.7300
0.2300
0,7900
0,3900
0,4300
O13
0.9400
96900
014
0.6400
0.7700
0.2900
015
0,3000
0.3800
0. 1300
LLaMA-2 70B:
Gro__Gro15dd
symmetry space group name -1-
.cel length_ 7.0000
.cell Jength_b 7,0000
longth 6.9000
LatB_L
_cell_leagth_a 5.5000
_symmetry
symmetey_space_ggnee nameeppaaameeeeeeee name_HIM 'P
_symmetry__nm Tablee numberr
cell angle Tables_
_cell cell_angle_Be
selllangie.. alpha 71.0000
71.0000
69.0000
_celi_length_b
_cell_length_e 5.3000
chemical formula sum "Gol5 Nd4
-chemical furmula, cermullasstrutural
Gral5Nd4
_celi_angle_beta 90.0000
_cell_angle_aappa 62.0000
_cell_ volume 289. 889.96945358
_eelL_agge__aama 90.0000
_eell_ formoloa unittollauuu____ loop
syametio__in_THbleesuumas
Alkemical_formulaasstreeeee Ln3B04
_symmetty__eee squiv_pos__tte site_id
shemiccllformmlt
"La3 04"
_symeerry pquiv_uuvvoo pos_ y, loop_
_cell_volume 146.33178751
_cel__ooruia__sitt__Z
.atum site lype symbul
loep
_symmetry_opil__oos_tle
@com_site_labed
symmotry equiv pas_as 1y7
_atom_sit
leap_
x.
_atom_sit samm_site_fraie
aatom site fract_x
simmsstee__fie Site_ffact___
site__faat__y
_atom_site_ sitm_siit_symmetry
_atom_site_lebbe
_atom. vit__sittt__f__
Nd
Nd Nd0
Ndi I 0.7500
occupancy
0.5600
0.5700
0.7500
0.7800
0.5600
_atom__ite_ffacc
0.1700
0.1700
0.1400
0.3800
atom fract
GroO+
Gro4
0.7600
0.2300
0.3000
Latom site occupancy
Lail
LoDa Lail 1 0,1800
0.6800
0.0600
0.
0.9900
0.9900
Groß+
Groß+
Gras
Gru6
0.3800
0.0300
0.1200
0.8700
0.4500
0.1000
1.0000
Labls
07
04
B3
0.1800
0.6800
0.6800
0.1800
0.6800
0.2800
0.0600
0.8300
0.7800
0.3300
0.1800
0.1500
0.8000
0.4900
0.5500
0.4900
Großle
Großle
Groß+
Groß+
GroO+
Gru9
Gre13
Groll
Gro10
Gre8
Gro12
1
0.6500
0.5600
0.2400
0.9100
0,0600
0.1700
0.2700
0.1000
0.5400
0,7400
0.6200
0.5800
Groß+
Gro14
0.4000
0.4600
Großt
Gro16
0.4500
0.9200
0.9400
Groß+ Gro17
0.9900
0.1300
0.0200
Groß+ Gro18
0.8400
0.5100
0.5200
17

---

Published as a conference paper at ICLR 2024
B.5 INCREASE IN PERPLEXITY UNDER TRANSFORMATION (IPT)
Although there are existing metries for invariance and equívariance neural networks, language models pose unique challenges because of their discrete tokens, which do not change smoothly under continuous transformations. Though might be possible to compute a meaningful analogue of the Lie derivative (Gruver et al., 2022), similar metrics, through interpolation of word embeddings, decide to adopt a simpler metric (IPT), which still highlights significant differences between base models. We calculate IPT for each model using 500 test datapoints and 20 randomly translation sampled as fraction coordinates from uniform distribution per dimension. The translations them- selves are implemented in PyMatgen and respect periodic boundary conditions (Ong et al., 2013). In order to combine the IPT values in meaningful way across different datapoints, we normalize their values by the mean perplexity over transformations. Thus datapoints which happen to have large perplexity, and therefore naturally large potential changes in perplexity, do not drown out points with small perplexity.
B.6 DIVERSITY AND NOVELTY CALCULATION
Following (Xie et al., 2021), we calculate diversity as the pairwise distance between samples using a featurization of structure and composition. To calculate novelty, we also featurize the training dataset and calculate the distance to the nearest element of the training set for each sample. A sample is considered novel if the nearest element in the training set is above a threshold. We use structural distance cutoff of 0.1 and composition distance cutoff of 2. In addition to novelty of structure and composition individual,
also consider the overall novelty of a crystal, where overall novelty is determined by having either a new structure or a new composition. All metrics are calculated on filtered samples that M3GNet qualifies metastable. We report metrics on metastable samples because these numbers are more practically relevant and because the samples are more likely to contribute meaningful variation, instead of being different from the training set and each other simply because they are wildly invalid. We normalize all diversity and novelty values by corresponding value for the test set provide a sense for the underlying data distribution.
B.7 SAMPLING SPEED
Although LLMs might seem like computational overkill at face value, batching for large-scale sam- pling allows LLaMA models to have comparable computational overhead to competing approaches. Making exact comparisons between LLaMA models and CDVAE are slightly challenging because of available hardware and differences in compatibility. We ran experiments primarily on A100 GPUs, while the publicly available code for CDVAE cannot be run on an A100 and reports results on a RTX2080 Ti.
We provide two analyses for the sampling rate of LLaMA models, one from experiments we ran on single A100 and alternative using third-party partyparty numbers for LLaMA models deployed on AWS instances.
Local analysis We obtain benehmark LLaMA-2 sampling times by running 5 batched generations and computingn the average time to completion. We then use these numbers to calculate the equivalent time to sample 10,000 structures. In practice, we used distributed sampling on cluster, so reporting our direct times to compute 10,000 samples would be less informative. We use the maximum batch size that we can fit on an A100 GPU with each model without causing out-of-memory (OOM) errors during sampling. The batch sizes were (7B: 512, 13B: 256, 70B: 128). To compare CDVAE with our results we perform rough, but generous, conversion of their results to an A GPU. We multiply their rate of sampling by 16, to account for the 2x faster rate of operations (Balaban, 2020) and approximately times larger GPU memory (allowing for large batch sizes and utilization rates). We report the intermediate numbers and calculations below. The final rates for metastable samples are shown in Figure 6.
18

---

Published as a conference paper at ICLR 2024
Model
Batch size
Seconds batch
Samples hour
Hours 10,000 crystals
CDVAE
512
n/a
n/a
1.260
LLaMA-2 7B
512
27.18
67814
0.147
LLaMA-2 13B
256
38.24
24100
0.414
LLaMA-2 70B
128
52.52
8774
1.139
AWS analysis Considering AWS as the deployment environment, we can build on a recent bench- 12xlarge) (Schmid, 2023), which found that LLaMA-2 13B achieved 0.416 hr/1M tokens and I.LaMA-2 70B achieved 0.864 hr/1M tokens. One crystal is around 100 tokens on average, so throughput for 10,000 crystals is the same as for 1M tokens. For comparison, we use CDVAE and its recorded runtimes for generating 10,000 crystals on a single RTX2080 Ti GPU (Xie et al., 2021). 1). To obtain the final numbers, we adjust for the number of GPUs (8) and 2x improvement from RTX2080 Ti to A100 GPUs (Balaban, 2020).
mark on cloud instance with A100 GPUs (ml.p4d.12xlarge)
Model CDVAE
Hours 10,000 crystals
Hours 10,000 metastable (M3GNet) crystals
0.363
1.260
LLaMA-2 13B
0.416
1.094
LLaMA-2 70B
0.864
1.728
see that LLaMA-2 13B actually has a comparable computational overhead to prior work, and LLaMA-2 70B is only slightly higher. When considering the rate of stable materials generated by each method, we see that LLaMA-2 13B actually has higher throughput than CDVAE.
C TEMPLATE METHOD BASELINE
We provide code in Listing implementing construction of the physically-inspinn element swap table. This table used by both the template method and the LLM-guided sampling method to constrain search to elements that physically plausible. Listing 2 shows our implementation of a basic template method with uniform sampling. The LLM-guided procedure is mostly identical, except with uniform sampling of the swap element changed for sampling from distribution obtained from the LLM with an infilling prompt (and modulated with temperature parameter T)
import
import randem
import pandas as pd
import numpy as np
from pymatgen.core import Element
from crrmaggen.cor.sstuuuee
import Structure
from m3gnet.moden import Relaxer
def moo__immlar_eeemmne (tanraaaaaoommmtttttttgttttoooe (target_clement, elements, tclerance=0.1) similar_clements []
11
for state, radius target_clement www.:eeentiiiiinrraaeeeeeeee ronie
12
for el elements:
13
if state in el. el.ionic_radii:
addius_iff - abs (radius 21.ionic_ran [state]) if sedius_ditf telerance and el.symbol !=
16
www.eees.eeeeeeeeeeeeeeeee.eees. append (el. symbol, state, similar
radius_diff))
return sorted (similar_ei
key-lambda x(21)
18
19
def make_ make_swap_table()>
20
elements [Element (el) for el in Element]
swap_table
23
24
for el in elements:
25
swap_tae tablettblelleee lel. symooll
x[0] for x in find_sim 1ar_element (e1, elements)
19

---

Published as a conference paper at ICLR 2024
return swap_table
Listing 1: Self contained code to construct the template method table which can be used to proposed mutations for local optimization around an existing material. The same table can used in tandem with a language model to provide sampling constraints (i.e. eliminate elements which very physically unlikely).
def propose new structures (cif str, swap_ wabletale,, swaps=l) struct Structure. From str str, fmt="cif")
elements = [el. symbol for el in struct. species] wwappable_elemette el for el elements if in swap_table and len (swap_tabbl [el1) 0
num_possible swapss sum ([1en [[Lentsswapttbbeeeee for el in swappable_eleee elements]] num_swaps min (nwn_possible_swppp max_swaps)
10
11
relaxer Relaxer()
bulks 11
14
for range (num_ swap_swaps)
15
old el xandom.choice (swappable _elleent_ee
16
possible_new = swap_table [cld_el]
new_cl random.chole (pcssible_new)
18
19
new_bulk = struct.copy ()
20
new_bulk. btt.bllk.repllc_ppeeeeeee... species ((cld_el; nex_el))
relax_results relaxer.relax (new__bxxxe_ewulll
23
tinal_structure relax relax ltt.ouuttssiiiaassssstttteeeeeee
24
= relax relaxt_esults trajectory' ennrrgies -1]
new hew_bulks.appendddee (fmt="ciii") (fn
27
cif=final_structuree
28
29
))
new bulks pd. DataFrarrn (new_bukks) return new_bulks
32
Self contained code implementing template method with uniform sampling. Our language model procedure is essentially the same but replaces uniform sampling with logits from prompted language model. This language model can the context from the rest of the crystal structure to propose a mutation instead of choosing mutation completely at random.
Listing 2:
20


Reviews:
[{"review_id": "us6gszwH54", "reviewer": "Reviewer_9VCZ", "summary": "This paper proposed to use LLMs to model the materials. The main technical contribution comes from 1) a modified tokenization mechanism that is suitable for material data; 2) a data augmentation method that leverages the property of the materials; 3) the prompt designs. Experiments on benchmark datasets show that when tuned on LLAMA2, it is able to improve the performance compared to other baselines.", "questions": "I’d like to see the author’s response on the questions I listed in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Application on the material science domain is interesting\n- The adaptation of LLMs for the material science is reasonable\n- Empirical results are promising", "weaknesses": "- The technical contribution is relatively limited, where the tokenization, the prompt design and the objectives for pertaining are all well studied in the past.\n- Since the nature of the material data is quite different compared to the natural language data, I’m wondering whether the pretrained LLAMA2 is offering any additional value. It would be helpful if one can show the performance with and without loading the pretrained checkpoint from LLAMA2 tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "arOLl3hw1p", "reviewer": "Reviewer_iXqY", "summary": "This paper propose a new application area for large language models (LLMs), i.e. LLaMA-2. The author leverages parameter-efficient fine-tuning to use LLMs. Based on domain expertise in material sciences, the author evaluated the proper tokenization methods for crystal structures and developed new metric to further include symmetric information into fine-tuning process.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. the author got good performance in the shown benchmarks.\n2. the method seems solid since it has been widely used in many other domains.", "weaknesses": "1. While one anticipates good performance from LLMs on standard evaluation metrics, especially with the likes of LLaMA-70B, the critical matter lies in the practical application in experiments.\n\n2. Given that LLMs can sometimes produce hallucinations, it would be beneficial to comprehensively evaluate this behavior in large models, rather than merely touching upon it in the limitations section. Presenting failure examples could offer valuable insights.\n\n3. Regarding Figure 2, could the authors elucidate why larger models result in poorer coverage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4sXeMoNvDM", "reviewer": "Reviewer_1Fky", "summary": "The paper proposes an approach to materials discovery by fine-tuning large language models (LLMs) on text-encoded atomistic data. The authors claim that this method can generate materials predicted to be metastable at a higher rate than competing diffusion models.", "questions": "* Can the authors provide a more detailed comparison with existing generative models in materials discovery, discussing the advantages and limitations of the proposed LLM-based approach compared to other state-of-the-art methods?\n* Are there any potential drawbacks or limitations of using LLMs for materials discovery, such as computational complexity or interpretability of the generated structures?\n* In the paper, the authors mention that \"We chose not to augment the ordering of atoms because these variables often contained valuable information\". How to ensure that the atomic order will not change the results?\n* In the paper, the authors mention that \"we only run VASP calculations on materials that have already been predicted as metastable by M3GNet\", while in the caption of figure 3, \"we only run VASP on structures predicted to be stable by M3GNet.\" What are the differences between the \"stable\" and \"metastable\"?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is well-motivated, addressing the limitations of existing computational materials databases and the potential of generative models for materials discovery.\n* The proposed approach of fine-tuning large language models on text-encoded atomistic data is novel and unorthodox, offering a new perspective on materials generation.\n* The paper is well-written, providing clear background information and a thorough explanation of the proposed method.", "weaknesses": "Related concerns are discussed in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)