FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions
Authors: Anuroop Sriram
Benjamin Kurt Miller
Ricky T. Q. Chen
Brandon M Wood
Venue: NeurIPS 2024 poster (2024)
Keywords: generative models
material generation
chemistry
flow matching
large language models
Primary area: generative_models
TL;DR: New generative model for materials combining Large Language Models and Riemannian Flow Matching, that significantly outperforms prior methods.

Abstract: Material discovery is a critical area of research with the potential to revolutionize various fields, including carbon capture, renewable energy, and electronics. However, the immense scale of the chemical space makes it challenging to explore all possible materials experimentally. In this paper, we introduce FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. FlowLLM first fine-tunes an LLM to learn an effective base distribution of meta-stable crystals in a text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. Our approach significantly outperforms state-of-the-art methods, increasing the generation rate of stable materials by over three times and increasing the rate for stable, unique, and novel crystals by $\sim50$% – a huge improvement on a difficult problem. Additionally, the crystals generated by FlowLLM are much closer to their relaxed state when compared with another leading model, significantly reducing post-hoc computational cost.

FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions
Anuroop Sriram FAIR, Meta anuroops@meta.com con
Benjamin Kurt Miller University of Amsterdam b.k. miller@uva..... nl
Ricky T. Q. Chen
Brandon M. Wood FAIR, Meta bmwood@meta. com
FAIR, Meta
rtqichen@@eeta.co ta. com
Abstract
Material discovery is critical area of research with the potential to revolutionize various fields, including carbon capture, renewable energy, and electronics. How- ever, the immense scale of the chemical space makes it challenging to explore all possible materials experimentally. In this paper, we introduce FlowLLM a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. FlowLLMM first fine- tunes an LLM to learn an effective base distribution of meta-stable crystals in text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. Our approach significantly outperforms state-of-the-aa methods, increasing the generation rate of stable materials by over three times and increasing the rate for stable, unique, and novel crystals by 50% huge improvement on a difficult problem. Additionally, the crystals generated by FlowLLM are much closer to their relaxed state when compared with another leading model, significantly reducing post-hoe computational cost.
Introduction
Material discovery holds transformative potential across numerous industries including carbon capture(38) batteries(28), photovoltaics(() (9], and energy storage However, the vastness of the chemical space has hindered experimental synthesis of the majority of possible materials. Generative models offer a promising avenue for exploring this untapped potential.
Generating crystalline materials is particularly challenging as it involves simultancously generating both discrete (atomic types) and continuous values (atomic positions and lattice geometry). While existing approaches, namely autoregressive large language models (LLMs) () and denoising models, e.g., denoising diffusion and flow matching [47 success, they exhibit complementary strengths and weaknesses. LLMs excel at modeling discrete values, but they can struggle with continuous values due to their reliance on finite precision representations. Conversely, denoising models more effectively handle continuous values and can easily ensure equivariances, but they face challenges with discrete elements.
16 49
48
30
26 17]. have demonstrated
LLMs also offer the distinct advantage of natural language prompting, enabling versatile and intuitive conditional generation. This capability is further enhanced by training LLMs on vast corpora of chemistry text, equipping them with valuable prior knowledge to generate chemically valid outputs. Queries like "Generate materials with a high bandgap and thermal stability" or "Propose novel perovskite structure for efficient solar energy conversion" can be directly integrated into the LLM
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Unconditional Prompt
Noisy Material
Co PrlM(()
5.4 5.6
Below is a description a bulk material.
76 91 Ga
LLM Inference
Generate a description of the lengths anglee....
0.84 0.94 0.25
As
0.63 0.31 0.75
Integration with Riemannian Flow Matching for Crystals
Generated Material
Unit Cell
ParMic c co)
Coordinates
fas
Figure 1: FlowLLM generative process: the fine-tuned LLM is first prompted with unconditional query to generate an initial material representation. This material is then iteratively transformed by the RFM model to update its atom positions and lattice parameters. The atom types static in RFM.
prompt, while denoising models typically require bespoke changes the architecture and training procedure to handle conditional generation.
To harness the strengths of both paradigms, introduce FlowLLM, novel hybrid approach that uses an LLM to generate an initial material representation, which is iteratively refined with a Riemannian Flow Matching (RFM; (2 model. This synergistic approach allows us effectively bridge the gap between discrete and continuous modeling, resulting in a significant improvement in the rate of generation of stable, unique, and novel (S.U.N.) materials. Such materials expand the limited knowledge we have of "material space" and are much more likely to be synthesizable than unstable generations. Our experiments demonstrate that FlowLLM generates stable materials at over 300% higher rate, and S.U.N. materials at ~ 50% higher rate compared to prior models, while retaining the LLM's ability be prompted with natural language instructions.
We offer two interpretations for the effectiveness of our approach. 1) The LLM learns a good base distribution for RFM: the LLM's output distribution serves learned base distribution for RFM, replacing the common practice of using the uniform base distribution. Since the LLM has been trained material data, this learned base distribution is closer to the target distribution, greatly simplifying integration with RFM. 2) RFM refines the output of the LLM: The LLM generates an approximate material representation due its finite precision when handling continuous values. The RFM then re- fines this approximation through iterative denoising, to generate much more accurate representation.
Our contributions are as follows:
We introduce FlowLLM, a novel hybrid approach for materials generation that combines LLMs and RFM, effectively leveraging their complementary strengths.
We demonstrate that FlowLLM significantly outperforms existing state-of-the-an generative models in generating novel and stable materials.
We show through ablation experiments that our method of combining LLM and RFM models through FlowLLM significantly outperform simpler combination approaches.
Code for training the FlowLLM model is available http::///cchtaaccom/cccccoerreee

---

2
Related Work
In the past, computational materials discovery relied on generating numerous candidate materials through random atomic substitutions in known materials(42 followed by computationally expensive quantum mechanical screening[ [9] to assess stability. Genetic algorithms 88 [33, and machine learning models trained to predict energies 37 25] have accelerated this process, but the fundamental bottleneck of brute force search remains.
Recent research has focused on generative models that directly produce stable materials, bypassing brute-force search. Diffusion models, either combined with Variational Autoencoders (VAEs) for partial variable prediction 47) or jointly diffusing all variables 16 48) 49) have shown promise. Addi-
tionally, Riemannian Flow Matching[26, Normalizing Flows 45 and Variational Autoencoders have also been adapted for material generation.
A parallel line of work utilizes autoregressive Large Language Models (LLMs) for material generation 66 [1], representing materials as a sequence discretized tokens. Pretraining these models on natural language imbues them with powerful prior knowledge not attainable by other approaches.
3 Preliminaries
Our approach models probability distributions over crystal lattices, defined as periodic arrangements of atoms in three-dimensional space. crystal lattice is created by tiling fundamental unit cell, where the unit cell contains a specific atomic configuration, forming the entire lattice when repeated. In this section, we present a high-level overview of crystal representations, building up to explain our model in section 4 Background details for the crystal representation are in appendix A
Crystal representation In the paper, we represent an E N atom crystal in product space: e = (a,f.1) E C, indicating the atom types, positions and unit cell geometry, respectively (47) 26). The atom types are represented by a matrix of categorical vectors: a = [a'......'' a=]. where a' E A. The atomic coordinates are represented using fractional coordinates within the unit cell, f= [f-..... ,f' where f' F =T with denoting the unitary length, flat torus manifold, i.e. the fractional coordinates satisfy periodic boundary conditions; that is, the atoms "wrap around" the unit cell. The unit cell geometry is defined using lattice parameters 1 L. where L is the space formed by 6-tuple of three side lengths (a, b, c) E RR+ (A, i.e. Angstrom) and three internal angles (o.337) E [60", 120%). This representation is not unique as the same crystal can be produced by different choices of unit cell. To make the representation unique, we select the minimum-volume unit cell and employ Niggli reduction (10) that uniquely determines the unit cell parameters.
Equivariance & Invariance Given group G with g- denoting a group action for some g G, a function f: X - y is called G-equivariant if Vo E X, Vy E G, f(g x) = 9 (xx), while is called G-invariant if Vvr E X, Vg G. fgg x) f(x) Since crystal not uniquely defined by any particular representation e but an infinite set, we know that the data distribution has G-invariant density, where G represents symmetries of a crystal.
Symmetries of crystals Concretely, our crystal representation exhibits multiple symmetries that detail here. The symmetric group S, on n atoms permutes the atom indices: ([ar()) ( [
...,, ................................ fr(t)] .).. The special Euclidean group SE(3) consists of orien- tation preserving rigid rotations and translations: (Q, where Q E SO(3) and T 1 1 1xx1 denote 3D rotations and translations respectively. This element transforms the crystal as: (Q. T) c (c= f T1 [ff T1).1). We emphasize that the representation is completely invariant w.r.t. Q because lattice parameters do not contain orientation information. Since these represent symmetries fundamental to crystals, the data distribution ((c) is invariant to these group operations.
Method
Our goal is to fit a parametric generative model p(c; 0) to approximate the distribution of known meta-stable materials ((c) using dataset of samples. The distributions p and are defined on the Riemannian manifold C. Our FlowLLM model generates samples from the parametric distribution
3

---

5.4 5.4
Example generation prompt:
Encoding
String
8.14 6.23 9.16
0.04 t.79 0.59
cherkial formddaa Cukzl
deserpttonn of lenghs angles the letice vectors and then the elamant typs and coo cooiitaass dinates stom withi artice (Crystal String/sts
0.23 6.4a 2.40
Hg
Pm,CdHg
Figure 2: Left: String encoding of materials used to train the LLM based on Gruver et al. al.[11]. Right: An example prompt used during training. The conditioning information in blue is optional, and can be replaced with conditioning on other properties well. The text in red is replaced with the crystal string representation shown on the left.
using a two-step procedure (see figure Tp. First it samples the LLM, then refines the LLM output using RFM, like so:
co PL 8o):
(erm(c]ca;00
where PLLM is modeled using large language model IT]. and PREM is modeled using Riemannian Flow Matching (RFM) (3 26), and
(0u. 81). Both the LLM and RFM frameworks are trained to estimate the data distribution over meta-stable crystals on samples from the Materials Project [T]]
Overview of training First, we fine-tune an LLM generate string representations of meta-stable materials [T]. Once trained, we can sample the LLM distribution using next token prediction, optionally conditioning on prompt (see figure 2}. Next, we train the RFM model using the FlowMM objective (26) where, conditioned on the chemical formula, will learn to transport between the LLM's model distribution and the data distribution. The full training process is described Algorithm
Overview of sampling We give the standard prompt to the LLM and allow it to do next token prediction until produces stop token. long as all atom types are actual elements and the lattice parameters are physical, we move forward. Otherwise we reject the sample. Then, we convert the text a crystal representation that serves as the initial sample. This sample's s fractional coordinates and lattice parameters are iteratively refined by the RFM model to produce the final sample of LLM. This sampling process is illustrated in figure 1
4.1 Large Language Model (PLLM) for Crystals
LLMs define a distribution over sequences through an autoregressive decomposition, II /woo), where each (Wo.e) follows categorical distribution conditioned on all previous tokens (Wo:)) in the sequence. Our LLM model closely follows Gruver et [TT.
Tokenization Language models interact with strings in text datasets after the string is converted into a sequence of tokens. The choice of tokenizer can have large impact on the performance of the language model. In terms of tokens, represent crystal e using fixed precision numbers two decimal places for fractional coordinates, and one for lattice lengths. Angles are represented as integers. Atom types represented discrete tokens. We use LLaMA-2 models [41) for our LLM architecture since these models break numbers into a sequence digits, which has been shown to dramatically improve performance arithmetic tasks (33
Training We rely on the extensive pretraining of LLaMA-2 models to instill useful biases over numerical operations. To train PLLM, we fine-tune pre-trained LLaMA-2 model dataset of crystal structures represented as strings along with a prompt indicating that the model should generate bulk materials by writing the lattice in lengths and angles along with atom types and coordinates. An example such a representation along with a prompt shown in figure 2
The flexibility of LLMs allows us to optionally include different kinds of conditional information in the prompt such as the chemical formula. We can also solve other tasks such as infilling by making changes to the prompt. For this hypothetical conditional generation, the prompt could include a desired

---

Algorithm FlowLLM training
1: Input: Training dataset of materials: D {e'), Pre-trained LLM:: PLLM, RFM velocity network:
0t Number of RFM training samples: Nir.
// Step 1: Fine-tune the LLM
2: Fine-tune PLIM on D following the procedure from Gruver et al. 12
// Step 2: Sample the LLM to generate training data for the RFM model
3: Initialize D <- Ø
4: fori= 1: N do
5: Sample cj ~ DD with replacement
6: Sample cj ~ (PLLM(110) using a prompt conditioned on the formula of cj
7: D= DU {(c), {()
8: end for
// Step 3: Train the RFM model on D
9: while not converged do
10: Sample (Co, C1) ~ D, t ~ ((0,, 1])
Ct = exPr, (t (olo (c1))
12: 2(01) = ||o (c,) (e(celc )1]
Take gradient descent step on Vo, 2(0))
14: end while
chemical formula, material properties, a combination of such information. this work, we used the same conditioning used in Gruver et al. [11]. and we leave more detailed study of this to future work.
Sampling To generate sequences from the model, the conditional distribution is sampled sequentially. The sampling procedure is modulated control the diversity and sampling speed using the temperature (r) and nucleus size (P) hyperparameters of nucleus sampling [13]. Temperature con- trols the entropy the conditional distributions, introducing a trade-off between diversity and mode sampling. The nucleus size limits the number of tokens that can be sampled. Given a nucleus size P with 0 < < 1, sampling restricted to the most probable tokens with cumulative probability P.
Symmetries LLMs The LLM architecture does not inherently produce a symmetric density, i.e., the distribution of meta-stable erystals that the LLM learss is not symmetric according to the funda- mental properties of crystals. We perform no fractional coordinate data augmentation via translation, and token permutation data augmentation. Unlike the other symmetries, rotation invariance holds for the learned LLM distribution due to our choice of representing the unit cell with lattice parameters.
4.2 Riemannian Flow Matching (PREM) for Crystals
Riemannian Flow Matching RFM produces a Continuous Normalizing Flow B i.e., a continuous,
parametric, diffeomorphism between the L.M base distribution Po == PILM and an approximation to our target distribution P1 Rs q. To model Prem == P1, we fit a time-dependent vector field of that has been adapted to crystals and is implemented using neural network with parameters 6,- Continuous Normalizing Flows are computationally expensive to train using maximum likelihood, but an alternative objective called Conditional Flow Matching [22] is more stable and scales better. The objective was generalized to Riemannian manifolds (2). and specifically labeled point clouds with periodic boundary conditions, i.e. crystals, by Miller al. (26).
Concretely, each point C has an associated tangent space TcC with an inner product (u, v) for E TcC, enabling the definition of distances, volumes, angles, and minimum length curves (geodesics). The geodesies for any C that we consider can be written in closed form using the exponential and logarithmic maps. The geodesic connecting Cu, E C at time t E [0.
C exPo, loge, (c1));
where exp= and logn are the exponential and logarithm maps for the manifold C. These geodesics help define the supervision signal used to train RFM.
5

---

Our RFM generative model et [0.1] x C - TC is parameterized as a time- smooth vector field. Training proceeds by regressing onto conditional vector fields u,(c/ci) that generate single data points For the geodesic path, this corresponds to u,(cle) --? loge (c). The general RFM training objective is then:
2(0))= ((,p(c)))(er) [1+2 (e [
(4)
Since we only use flat manifolds, ||-Il the Euclidean norm. At the optimal solution, vgt generates Pt with endpoints Po p. P1 = 4. At sampling time, we draw sample from Co and solve the ordinary differential equation c+ v (c+) with initial value co at t O; the solution at t 1 is then the sample from our RFM model.
Geometry of F We apply the conditional vector field for point cloud living on a x 3-dimensional product of flat tori invariant to global translations, i.e. fractional coordinates with periodic boundary conditions [26). This is a geodesic path, which may cross the periodic boundary:
expy cxp ff
1f +l,.
log (fi) = 2x
atan2 ssin(w"), cos(w"))
(5)
where w² == 2T ( f8). and f E Ty.Fi for i = 1, ,,..., n. Computing the geodesic of atoms amounts to an atom-wise application of log,, on f1 and expy on J E TyF respectively. Additionally,
following Miller al. [26] we address canslation-invariance by removing the mean torus translation:
uf (f fi) = log,, (f)
log (f)).
(6)
Geometry of L The space of lattice parameters, C +3 [60, 120) is Euclidean space with boundaries. We can ignore these boundaries for the lattice lengths in TR+ since (i) the data does not lie on the boundary (a, c >> 0) and (ii) we can clamp our base distribution be positive with rejection. The boundary issue for the lattice angles a, 0,3,7 can be addressed (26) using diffeomorphism yp: [60%, 120%] I to unconstrained space, applied element-wise each angle:
11 60
(n) = logit
1 120 (17) 60,
120
where (.) and logit are the sigmoid and the log-odds functions, respectively. We directly apply RFM in the unconstrained space, and for sampling, we map the angles back into [60", 120%] using (p -1
The RFM training objective
With this formulation, training objective based on 44
becomes:
2
E P (o
>
F1
F (ct)
logg, (fo)
1
n
log (56)
(8)
3n
n2
A
2,9 1 (ce)
1, -111..
where we now use PLLM the base distribution, and c (f,,4,,a) The loss coefficients > A R+ are hyperparameters. We use graph neural network (GNN) inspired by (36] 16 26 for of (c). This GNN enforces equivariance to atom permutations via message passing, invariance to atom translation by featurizing graph edges as relative displacements nodes, and invariance to rotations by our choice of lattice representation. See appendix B for more details about the GNN architecture.
4.3 Consequences of using an LLM as the base distribution
Model symmetries Just like the LLM, the orientation- invvriiant representation of the unit cell leads to global rotation invariance. However, permutation and translation symmetries are not so simple. If the parameterization of the RFM velocity field is G-equivariant, and the base distribution is G-invariont, then the model density is G-invariant [8]. We use graph neural networks (36 40 27
44 7 [1] 31, 51), and additional projections (6),, to ensure that the RFM velocity predictions are G- equivariant to both permutation and translation. However, will generally not recover translation

---

invariant density because the base distribution defined by the LLM is not invariant to translation. The density will be permutation invariant our RFM representation because the each atom is a node in an unordered point cloud and the L.M ordering is ignored by the RFM, but the density will not be per- mutation invariant in the text representation, due to the LLM's lack of token permutation invariance.
Empirically, we do not find the lack of exact invariance to be problem, and FlowLLM outperforms methods with exact invariance (section (). This is because an LLM trained to generate crystals is approximately invariant to crystal symmetries. This was verified by Gruver et al.([T] who proposed a new metric, Increase Perplexity under Transformation (IPT), to quantify this approximation:
(s) (() PPL(((g-(s)) (s))
where g" arg min PPL(t,, (s)), and PPL is the perplexity of the sequence, the exponent of the cengthnnomaliize cross entropy loss, PPL((s) 2 CE(o)/n They find that well-trained L.M obtains a small IPT value, implying that it is approximately invariant.
Invalid erystals The LLM base distribution is constrained to C, i.e. the LLM can generate invalid crystals. We find that this is extremely rare and easy to detect. In such cases, we simply reject that sample, and draw a new sample until we get a valid crystal. Empirically, we found this rejection rate to be ~ 0.5% with softmax temperature of 0.7.
Text not continuous in L or F The LLM base distribution only takes non-zero values over a small number of discrete points due to the use of finite precision representations. For example, we represent fractional coordinates with only 2 decimal places, so they can only take one of 100 distinct values. We can mitigate this problem by adding small amount of random zero-mean gaussian noise to all continuous values predicted by the LLM. Empirically, we do not observe any noticeable difference in performance due to this added noise (see appendix F)
5
Experiments
5.1 Setup
We trained our model on the widely used MP-20 datase of inorganic crystalline materials (47. MP-20 comprises 45,231 materials, a subset of the Materials Project 15 containing up to 20 atoms known to be metastable (see section 5.2)
I
We first train our LLM independently using the various prompting strategies described in Section 4 Unless otherwise specified, we employed pretrained LLaMA-2 70B model [41] for all experiments,
that was fine-tuned with the Low-Rank Adapters (LoRA) method [4] using PyTorch] 32) and Transformers (6)..
Next, we trained the RFM model using the fine-tuned LLM (with frozen weights) as the base distribution and the MP-20 dataset the target distribution. For computational efficiency, sampled a large number (Nrr) of examples from the base distribution in advance, and used the same set for all of our training runs. To create this set, sampled Nor materials, with replacement from MP-20, and queried the LLM with a prompt conditioned on the chemical formula of each of these materials. This results in a set of Ntr pairs, ((c), cf) ) 1en of LLM generated materials and ground truth materials that constitutes the training set for the RFM model. We list the hyperparameter values used in our experiments in appendix C
To generate new samples, we first generate material from the LLM using unconditional query. We then perform an integration with the RFM model, starting from this LLM-generated material. During sampling,
adjust hyperparameters such as temperature т, nucleus probability P, and the number of integration steps to achieve different trade-offs between diversity, accuracy, and efficiency.
5.2 Metrics
Our primary metrics are Stability Rate, the percentage of generated materials that thermodynami- cally stable, a key indicator of synthesizabiliity and the S.U.N. rate, the percentage of materials that
Publicly available at http://///ttttccocccccee///////e

---

Mathed
11M Params
lsgg Stepe
Validyy (9)
Courage
Property
Stdvilty Rate (53
SUN Rono(R3)
Strecurii
Composition
Recall
Precsion
wdiss (p)
(..st(V..)
MP 2023
CINAR DIRCSP
A
S000
100,00
86-10
99.15
99 49
0.086
0.278
1:57
1000
100.00
K3.25
99.71
99.76
0.353
0.125
5,06
3.34
FlowMM
Mg
1000
96.85
83.19
99.49
99.55
0,239
0.883
4.65
2334
CrysalLLM L.M (708)
T- 0.7
99.5
95.4
85.8
98.9
0.51
0.64
5.28
1towLLM typs
1S, 0.9
D5. 0.
97.18
99.78
99.76
0.20
8.55
Flod1M
87. 1.0
10. 0.9
99.85
99.81
99.71
99.65
0.14
0.09
13003
10.07
4.58
4:89
07, 0.
250
99.94
90384
96.95
9982
1.34
0.15
17.82
4.92
Table 1: Results for material generation on the MP-20 dataset. Stability rate is the percentage of
generated materials with E!mll 0.0 N-ary 2 2.
are stable, unique and novel. Since computing stability is computationally expense, Xie et al. [47 proposed number of proxy metrics. We explain these metrics in more detail in appendix D
One key difference in evaluation between the proxy metries and the stability metries is the use of pre-relaxation and relaxation techniques. Proxy metrics are computed on raw samples without any further processing. Stability metrics are computed on structures that are first pre-relaxed using CHGNet[5] then relaxed using Density Functional Theory.
Density Functional Theory is extremely expensive, even with speedups using pseudo-potentials Ideally, the generative model can generate many S.U.N. structures that are already close to their relaxed ground state. Generating structures close to ground state may also indicate that the model has done a better job capturing the data distribution. It can also speed up or obviate the need for relaxing the generated structures, which has huge computational benefits. We include several additional metrics to measure the closeness of generated and corresponding ground state structures, that are described in appendix E
5.3 Results
We compare our model to four prior methods: CD-VAE 47), a hybrid Variational Autoencoder diffusion model; Dif diffusion model; FlowMM [26]. Riemannian Flow Matching model; and [rystall.LM[[T]] which fine-tunes a LLaMA-2 model on materials represented as sequences. The LLM and RFM components of FlowLLM closely resemble the formulations in CrystalLLLM and FlowMM, respectively. To compare different models, we generate 10,000 new structures from each model and compare the metrics described in section 5.2
Our main results are presented in table On the most important metries, namely the Stability & S.U.N. rates, FlowLLM significantly outperforms all prior methods across various LLM sampling parameters. For our best FlowLLM model (r 0.7, P = 0.9), 17.82% of the generated structures are stable, out of which 48% are novel (not similar to any training or validation structure). Of the remaining structures, 58% are unique, leading to a S.U.N. rate of 4.92%. FlowLIM obtains 300% higher stability rate and ~ 50% higher S.U.N. rate than the best prior model!
Figure 3a shows histograms comparing the Ebull values of generated materials from FlowLLM com- pared to prior models. Clearly, FlowLLM generates many more materials with lower ghull values than the other models.
The results on proxy metrics, on the other hand, remain mixed. Diffusion and flow matching methods excel Coverage Recall, while CrystalLLM has the best Composition Validity. FlowLLM achieves the best compromise between coverage and validity, potentially explaining its superior Stability & S.U.N. rates. It is important note that many of these metries have become saturated, offering limited discriminatory power for evaluating state-of-the-art models. As a result, we anticipate a decreased reliance these metrics in future research.
Comparison of generated and relaxed structures While the stability rate and S.U.N metrics capture whether the generated structures can be relaxed to stable S.U.N. states, they do not address the question: How close are the generated structures to their relaxed state? To answer this question, we compared generated structures to those same generated structures after relaxation using CHGNet, computing the following metrics between generated and CHGNet relaxed states: Match Rate and RMSD, as defined by StructureMatcherr. along with the A- A-Energy and the average Num steps between the states. Definitions for these metrics can be found in appendix
8

---

Method
Match Rate (%)
RMSD (Å)
A-Energy (eV/atom) + Num Steps
FlowMM
74.3
0.096
0.3031
191.98
FlowLLL
94.9
0.023
0.0898
37.97
Table 2: Comparison of generated and corresponding ground state structures from the CHGNet relaxation. Compared to FlowMM, FlowLLM generates structures much closer to the ground state,
Powdll
HowEL (ye p-0.3
HovELM pye.7, p-0.3)
Flod UM 2,007,
1590
1230
1999
730
500
230
-22
0.2
0.4
22
60000
Number matchoo
Floill 1ym1 1.0, p-1..)
3000
0:92
2000
2.90
0.53
1000
5.85
ToowM
Number elements material
Number integration
(b)
Figure 3: (a) Histogram of Ebull values comparing FlowLLM with prior models. The dashed line shows thermodynamic stability threshold (Ehull = =0). (b) Histogram of N-ary compared to the data distribution. (c) Structural validity as function of number of integration steps.
Table 2 shows comparison of FlowMM and FlowLLM. The samples generated by FlowLLM significantly closer to ground state compared to FlowMM, according to our metrics.
Importance of learned base distribution One motivation for hybrid LLM-RFM model is to leverage the LLM's superior ability to generate accurate atom types compared to denoising models. To isolate this effect, trained the FlowELM-Tyypes model, following similar procedure as FlowLLM but using simple base distributions for lattice parameters and fractional coordinates identical those used in FlowMM (6).. Thus, the L.LM only contributes atom type prediction in this model. Despite this simplification. FlowLLM-Type still surpasses prior models on the Stability Rate metric (table l). highlighting the benefits of employing an LLM for atom type prediction. The stability rate of FlowLLM-Tye Types remains considerably lower than that of FlowLLM, underscoring the substantial value of using learned base distributions.
N-ary analysis The number of distinct element types in a material is called the N-ary value of that material. Bggure33jcooppan the distribution N-ary values for different models with the target data distribution. FlowMM and FlowLLM match the data distribution better than the diffusion models, which tend to generate too many materials with high n-ary.
Number of RFM integration steps Compared to diffusion and flow matching models which require hundreds or thousands integration steps, FlowLLM able to converge in little as 50 steps (figure 3ch. This not surprising given our use of learned base distribution.

---

Discussion
The discovery novel, stable materials holds the potential to help revolutionize numerous industries, but progress has been slow due to the high computational costs involved. Widely used random structure search methods(33) yield less than a 1% success rate in identifying stable materials. Given the substantial cost of validating generated structures using density functional theory, improving this rate is paramount importance.
Recent breakthroughs with denoising models 16 26 and large language models] 11] have increased traditional approaches. In this work,
the stability rate 5%, significant improvement propose a novel generative model which harnesses the strengths of both paradigms to further increase this number by 3x, representing a major advancement in the field.
Limitations While FlowLLM excels at generating stable materials, a key limitation is its lack of end-to-end differentiability. This hinders its direct application inverse design, where generative models are optimized to generate material with specific properties, as explored in prior work using denoising models 49 477. Future research could investigate extending FlowLLM for inverse design.
This work can accelerate the discovery of new materials for renewable energy, electronics, and carbon capture, ultimately benefiting society by enabling more efficient and sus- tainable technologies. However, the adoption of generative models also raises concerns, such as the creation of harmful substances and access inequalities.
Broader impact
References
[1] Lowik Chanussot et al. "Open Catalyst 2020 (OC20) Dataset and Community Challenges", In: ACS Catalysis (2021). DOI: 10. 1021/ 0c04525.
[2] Ricky T. Q. Chen and Yaron Lipman. "Riemannian flow matching on general geometries". In: The Twelfth International Conference on Learning Representations. 2024. URL: https
//openreview.
[3] Ricky T. Q. Chen al. "Neural ordinary differential equations". In: Advances in neural
information processing systems 31 (2018).
[4] Daniel W Davies et al. "SMACT: Semiconducting materials by analogy and chemical theory". In: Journal of Open Source Software 4.38 (2019). p. 1361.
[5] Bowen Deng et al. "CHGNet as a pretrained universal neural network potential for charge- informed atomistic modelling". In: Nature Machine Intelligence 5.9 (2023), pp. 1031-1041.
[6] Daniel Flam-Shepherd and Alán Aspuru-Guzik. "Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files". In: arXiv preprint arXiv:2305.05770 (2023).
[7] Mario Geiger and Tess Smidt. "e3nn: Euclidean neural networks". In: arXiv preprint arXiv:2207. 0007009453 (2022).
[8] Colin W Glass, Artem R Oganov, and Nikolaus Hansen. "USPEX Evolutionary crystal structure prediction". In: Computer physics communications 175.11-12 (2006), pp. 713-720.
[9] Martin Green, Anita Ho-Baillie, and Henry Snaith. "The emergence of perovskite solar cells".
In: Nature Photonics (July 2014). DO1: 10. 1038/ /NBH/PTTOOO... 2014. 134
[10] Ralf W Grosse- Kunsttev, Nicholas K Sauter, and Paul D Adams. "Numerically stable al- gorithms for the computation of reduced unit cells". In: Acta Crystallographica Section A: Foundations of Crystallography 60.1 (2004), pp. 1-6.
[11] Nate Gruver et al. "Fine-Tuneed Language Models Generate Stable Inorganic Materials as Text". In: arXiv preprint arXiv:2402.04377 (2024).
[12] Nate Gruver et al. "Large Language Models Are Zero-Shot Time Series Forecasters". In: Thirty-seventh Conference on Neural Information Processing Systems. 2023.
[13] Ari Holtzman et al. "The Curious Case of Neural Text Degeneration". In: International Conference on Learning Representations. 2020. URL:
http:/ppperreiiee.enttffoormmmm net/forum?
id=rygGQyrFvit
[14] J. Edward Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models". In: ArXiv 0bs/2206.0960 (2021).
10

---

[15] Anubhav Jain et al. "The Materials Project: A materials genome approach to accelerating materials innovation". In: APL Materials 1.1 (July 2013). p. 01 ISSN: 2166-532X, DOI:
10. 1063/1 4312323. eprint: https: : //pbs. aip. org/ hwtppppp/atticlleeeeeeeapa//pppe pttpleddpddoo////////// /o0//0
1063/1. 48122233113163888 0111331116666101100 100001221111oiieeeeee pdf
CRI.: https: http:://///o..oo/1.....
1.4812323
[16]
Rui Jiao et al. "Crystal Structure Prediction by Joint Equivariant Diffusion". In: arXiv preprint
arXiv:2309.044 04445 (2023).
[17] Rui Jiao et al. "Space Group Constrained Crystal Generation". In: The Twelfth International
Conference on Learning Representations. 2024. URL: https:: wwt.:./.ee/fforrmm
1d=jkvZ7v40mPP
[18] Jonas Köhler, Leon Klein, and Frank Noé. "Equivariant flows: exact likelihood generative learning for symmetric densities", In: International conference on machine learning. PMLR. 2020. pp. 5361-5370.
[19] Walter Kohn and Lu Jeu Sham. "Self-consistent equations including exchange and correlation effects". In: Physical review 140.4A (1965), A1133.
[20] Georg Kresse and Jürgen Furthmüller. "Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set". In: Computational materials science
6.1 (1996), pp. 15-50.
[21] Yi-Luu Liao et al. "FquiformerV2: Improved Equivariant Transformer for Scaling to Higher- Degree Representations". In: arXiv preprint arXiv:2306. /2059 (2023).
[22] Yaron Lipman et al. "Flow Matching for Generative Modeling". In: The Eleventh International Conference on Learning Representations. 2022.
[23] Tiedong Liu and Bryan Kian Hsiang Low. "Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks". In: arXiv preprint 23Xiv:2305.14201 (2023).
[24] Ilya Loshchilov and Frank Hutter. "Decoupled Weight Decay Regularization" In: International Conference on Learning Representations. 2018.
[25] Amil Merchant et al. "Scaling deep learning for materials discovery". In: Nature (2023),
pp. 1-6.
[26] Benjamin Kurt Miller et al. "FlowMM: Generating Materials with Riemannian Flow Match- ing". In: Forty-first International Conference on Machine Learning. 2024. URL: https //openreview.
[27] Benjamin Kurt Miller et al. "Relevance of rotationally equivariant convolutions for predicting molecular properties". In: arXiv preprint arXiv:2008. 08461 (2020).
[28] K. Mizushima et al. "LixCoO2 (0<x<-1): A new cathode material for batteries of high energy density". In: Materials Research Bulletin 15.6 (1980), pp. 783-789, ISSN: 0025- URL: hhtpps: www
5408. DOL: https: /ttp::///ooooo///////// 10 1016 0025- (89) 00012-4
sciencedirect.co httpa///eee.coeeeeeeeeeeeeee///s
[29] Shyue Ping Ong et al. "Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis". In: Computational Materials Science 68 (2013), pp. 314-319,
[30] Teerachote Pakornchote et al. "Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling". In: Scientific Reports 1 14.1 (2024), p. 1275.
[31] Saro Passaro and C Lawrence Zitnick. "Reducing SO (3) Convolutions to SO (2) for Efficient Equivariant GNNs". In: arXiv preprint arXiv:2302.03655. (2023).
[32] Adam Paszke et al. "PyTorch: An Imperative Style, High-Performance Deep Learning Library". In: Neural Information Processing Systems. 2019.
[33] Chris J Pickard and RJ Needs. "Ab initio random structure searching". In: Journal of Physics:
Condensed Matter 23.5 (2011) p. 053201.
[34] Zekun Ren et al. "An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties". In: Matter (2021). ISSN: 2590-2385. DOI:
https://dol. www.org///0.. 1016/ www.mattt....mmm 2021. 11.032
[35] Janosh Riebesell. "Matbench Discovery v1.0.0". In: figshare (Jan. 2024). DOT: 6084/m9 figshare. 22715118. v12 URL.: https:
Discovery_vl_@.. S_teerryyv______220_ 00_0022715155
[36] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. "E (n) equivariant graph neural networks". In: International conference on machine learning. PMLR. 2021, pp. 9323-9332,
11

---

[37] Jonathan Schmidt et al. "Large-scale machine-learning learning-assistedd assisted exploration of the whole materials space". In: arXiv preprint arXiv:2210.000 0.509 (2022).
[38] Anuroop Sriram et al. "The Open DAC 2023 Dataset and Challenges for Sorbent Discovery
in Direct Air Capture". In: ACS Central Science 0.0 (0), null. DOL: 10. 1021 a21/aascentsci
3c01 eprint: https: //oii. org/ 10. 1ell///cceents//eeoo 3c01629
URL: https::/d/e
org/10. 100.0//////ccccttcccccccc 3c01629
[39] Wenhao Sun al. "The thermodynamic scale of inorganic crystalline metastability". In: Science advances 2. (2016), e1600225.
[40] Nathaniel Thomas et al. "Tensor field networks: Rotation-and translation-equivarinn neural
networks for 3d point clouds". In: arXiv preprint arXiv: 29229 (2018).
[41] Hugo Touvron et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models". In: ArXiv
abs/2307. 0022000009288 (2023).
[42] Hai-Chen Wang, Silvana Botti, and Miguel AL Marques. "Predicting stable crystalline com- pounds using chemical similarity". In: upj Computational Materials 7.1 (2021), 12.
[43] Logan Ward et al. "A general-purpose machine learning framework for predicting properties of inorganic materials" " In: npj Computational Materials 2.1 (2016), pp. 1-7.
[44] Maurice Weiler et al. "Coordinate Independent Convolutional Networks-Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds". In: arXiv preprint arXiv:2106. 06020 (2021).
[45] Peter Wirnsberger et al. "Normalizing flows for atomic solids". In: Machine Learning: Science and Technology 3.2 (2022). p. 025009.
[46] Thomas Wolf et al. "Transformers: State-of-the-AA Natural Language Processing". In: Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online: Association for Computational Linguistics, Oct. 2020, pp. 38-
45. URL: https:://www //www. wwwweb. org wwwhoaootoooyyoaaa wwtp::000..0meeeeeeem memlp-demos..
[47] Tian Xie et al. "Crystal Diffusion Variational Autoencoder for Periodic Material Generation".
In: International Conference on Learning Representations. 2021.
[48] Mengjiao Yang al. "Scalable diffusion for materials generation". In: arXiv preprint
arXiv:231 1.09233 (2023).
[49] Claudio Zeni et al. "MatterGen: a generative model for inorganic materials design". In: arXiv preprint arXiv:2312.0336. (2023).
[50] Nils ER Zimmermann and Anubhav Jain. "Local structure order parameters and site fingerprints
for quantification of coordination environment and crystal structure similarity". In: RSC advances 10.10 (2020), pp. 6063-6081.
[51] Larry Zitnick et al. "Spherical Channels for Modeling Atomic Interactions".. In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, 8054-8067. URL: h neurips. cc paper_ _files/ paper/2022/fille 3501 Foaar//eeooooeooooooooaarrrararr oww.a//eeee.eeeaaeeeeeeaaeaeoeee
12

---

A Crystal Representations Details
Atomic types The representation of atomic number is dependent the model processing the data. In the LLM, the name of the element can be written into the text representation directly. This can be string or single token, depending on LLaMA-2's tokenization. In the RFM framework, applied a one-hot representation.
Unit cell geometry Throughout the paper and in our implementation, represent the unit cell using lengths and angles; however, there is another representation relevant for defining the fractional coordinates and better expressing crystal symmetries. The unit cell can be defined by matrix of Cartesian column vectors == P, E R3x3 This representation has strictly more information than I, since it also defines the orientation of the unit cell. This orientation is irrelevant in paper, since want rotation invariance. That's why we choose I in the first place.
Fractional coordinates Now that we have the representation 1, we can define fractional coordinates. Recall, atomic positions are typically represented using Cartesian coordinates a == [ a"] E X' = g3xn with coordinates in the rows and atoms the columns. The Fractional coordinate representation defined f x = f E F = [0, 1)sxn.
B Graph Neural network in the RFM Model
In this section, we describe the graph neural network used in our RFM model. Our GNN model is inspired by the GNNs used in FlowMM [26] and Dif which in turn adapted the EGNN (36) model for fractional coordinates,
(o 0h (a')
(10)
(x)
1f = Pm (h (h(s-1) 1): h/ ()--1):4 SinusoidalEmbeedingg( ff'),,
(11)
N
m(.)
m
(s)
(12)
j=1
h(s)
((s1)
+ yh *h (h m)))
(13)
fr f/
(bj (himaxx)) s)
(14)
i = Pi
h (maxx)) s)
(15)
ij where (i)). mla) represent messages at layer between nodes and j. hi represents hidden
representation of node j at layer s; (mms (Ph, Oluo f 4 Pi represent parametric functions with all parameters noted together as 0. Finally, we define
SimusoidalEmbeeddinggrr = (sin(2 cos(2
T k=
(16)
where freg is a hyperparameter. We standardized the input to the network with z-scoring. We also standardized the outputs for predicted tangent vectors f. i. Models were trained using the Adaml optimizer 24
C Hyperparameters
We used N₂r = 3.3 x 10th and trained the model for 20 epochs with early stopping. To generate the Nir training pairs, we used temperature 7 0.9 and nucleus probability P 0.99. While other values might be explored, the high computational cost of experimentation limited our exploration of these parameters. For training the RFM model, we swept over a few values of learning rates: (le-3, 7e-4, 5e-4, 3e-4, le-4). To compute the loss function, we used loss weights Xy 200, and X = 1 in the training objective (equation (8). These values were chosen by running grid search over og E {100, 200, 300, 400), X {1 Additional hyperparameter settings are given in table
13

---

The LLM was trained for 10 epochs with batch size of 16, and cosine annealed learning rate of 0.0005, with LoRA rank 8 and 32.
Compute resources We trained our LLM model on 80GB A100 GPUs for roughly day. We used 4-bit quantization and LoRA to optimize training. Sampling from the trained LLM required total of 250 A A100 GPU days, that were parallelized over 300 A100 GPUs.
Each of our RFM models were trained for 2 days on single 32GB V100 GPU. All experiments were performed on an internal GPU cluster.
Evaluations required running DFT computations that were run on a large internal CPU cluster with 5000 nodes, each equipped with 26-core Intel Cooper Lake-SP CPU, and 64GB memory. Each DFT computation took about hour compute on a single node, and we ran nearly 50,000 such computations to evaluate all of our models.
Table 3: RFM model hyperparameters
Value
Hidden Dimension
512
Time Embedding Dimension
256
Number of Layers
6
Activation Function
silu
Layer Norm
True
Batch Size
256
Max Epochs
20
Inference anti-annealing scale
D Metrics
Thermodynamic stability is a key indicator of synthesizability, and generating novel stable materials of keen interest material science. Stability is determined by comparing a material's energy to those of competing crystals with the same elements. Formally, stability is measured by constructing convex hull of all competing materials from a reference set and computing the distance from this hull (called Energy above the Hull, or Elmll),, Stable materials have Ehuil < O, while materials with Ehull 0.08 eV/atom are called metastable (39] With this defintion of stability, we define our Stability Rate metric as the percentage of generated materials that are stable (Ehulf < O, and n-ary 2 2). For our reference set of materials, we use the Materials Project database recorded by (35 in February 2023.
Following Miller al. 266.. we compute Ehhlll values by running structure relaxations the generated structures with the CHGNet model () followed by density functional theory (DFT) 19 calculations.
While stability rate is an important metric, it does not capture novelty. Therefore, we define a second metric, the S. S.U.N. rate which measures the percentage of generated structures which are Stable, Unique, and Novel. To determine novelty, structure in the training dataset. Similarity is measured using pymatgen's 's StructureMatcher (29 with default settings. A generated structure that is not similar to any training data structure is considered novel.
exclude generated structures that similar any
To compute uniqueness, we use StructureMatthhr to do pairwise comparisons between all generated structures, and group similar structures into equivalence classes. Each group is only counted as a single unique structure for the purpose of computing the S.U.N. rate. Formally,
Netable
Stability Rate
(17)
Ngen
Ns..u.
S.U.N. Rate
(18)
N gen
(19)
14

---

Method
Noise Std
Integ, Steps
Validity (%) t Composition
Coverage (%)
Structural
Recall
Precision
99.64
91.99
94.36
94.38
0.01
99.85
91.99
94.74
94,50
FlowLLM
0.02
99.99
91.99
93.41
94.58
0.04
250
99.97
91.99
93.24
94.54
Table 4: Proxy metries for a FlowLLM trained with different levels of random gaussian noise added to continuous values predicted by the LLM. Added noise increases the support of the base distribution, but we do not see an appreciable difference in the metrics.
where Ngen is the number of generated samples, Nstable the number of generated samples which are stable, and Ns.u.N is the number of generated samples which are stable, unique, and novel.
Due the computational expense of DFT needed compute stability and S.U.N. rates, number of proxy metrics have been proposed by Xie et al. [4]] to benchmark model performance:
1. Structural Validity: Percentage of structures with valid atomic arrangements, where all pairwise interatomic distances exceed 0.5 Å.
2. Compositional Validity: Percentage of charge-neutral crystals, as determined by the SMACT
heuristic system [4).
3. Coverage Recall Precision: Standard recall and precision metrics assessing the model's
ability generate structures close to those in the test dataset. Closeness is evaluated using structural and compositional fingerprints [50] 43).
4. Wasserstein Distances of Property Distributions: Wasserstein distances between the distri- butions of computed properties (density, and Na- - the number of unique atoms) for crystal samples from the test set and generated structures.
E Comparison of generated structures to ground state structures
For many practical applications in chemistry, it is important find the local energy minimum of a generated structure. This is done by performing computationally expensive structure relaxations. Thus, beneficial generate structures close their ground state. To compare how close the generated structures are to their ground state (i.e. local energy minimum), we define additional metries (shown in table 2
1. Match Rate: What fraction of generated structures and corresponding ground state structures are similar (where similarity computed using pymatgen's StrcheureMatcher with default settings).
2. RMSD: Average RMS distance between generated structures and corresponding ground state structures computed using pymatgen's StructureMatcher whenever there is match.
3. A A-Energy: Difference in energy between the generated structure and ground state structure of the DFT relaxation. This measures the reduction in energy during the structure relaxation process.
4. Num Steps: Number of optimizer steps needed to pre-relax the generated structure using CHGNet.
F Adding noise to the base distribution
Table shows the effect of adding noise to the base distribution. We do not see significant impact from the added noise.
G Material Generation Time
We compare the time to generate 10,000 materials between FlowLLM with FlowMM. Inference for both models was run on machine with a 32 core Intel(R) Xeon(R) Platinum 8488C CPU, and single 80GB A100 GPU. FlowMM used 750 integration steps, and the RFM step of FlowLLM used
15

---

250 integration steps. With this setup, the FlowMM model takes 65.1 minutes to generate 10,000 materials, while FlowLLM takes 89.6 minutes, which comparable FlowMM.
more useful metric is the time to generate a S.U.N. material, which is computed by dividing the inference time by the number of generated S.U.N. materials. With this metrie, FMowMM takes 16. seconds to generate S, S.U.N. material, while FlowLLM takes only 10.9 seconds.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research. addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No] or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes]" " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: The claims made are about designing a hybrid model (described in section and 3x higher stability rate (experimental results in section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings. is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
much the results
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Lack of exact invariance is discussed section Lack of end-to-ent differ-
entiability, and the difficulty in applying inverse design as result are explained in section
6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
17

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the
implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We do not include any theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our method has been described in detail (section 4 with all relevant hyperpa- rameters (appendix C). The MP-20 dataset we use is publicly available. We have made our code available on Github.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken
make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same
18

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released our code on Github. The dataset we use is already publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines httttp::/nipss.oo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
/ //nips. cc/publ htc//ublc/gguieeseeeeeeseess Sttps//oooSSuiiiiiiiiii
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The dataset with train/val/test splits is available publicly (url provided in section 5 All hyperparameters and training details are described in section 5 and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars due high computational costs in training and evaluating our model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Compute resources are mentioned in appendix Guidelines:
C
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httph::t://///
Answer:
[Yes]
Justification: To the best of our knowledge, conform to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
20

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss societal impact in section Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Question:
Justification: We do not plan to release trained models or any new datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use a publicly available dataset and we cite the appropriate source for the data (section 5)
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release any assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "sguaZPF58D", "reviewer": "Reviewer_sJw2", "summary": "This paper presents a generative model for crystals that uses a LLM as a base distribution and a flow matching model to refine the 3D structure. Both the LLM and flow matching parts are mostly borrowed from prior studies. However, the authors demonstrate a significant improvement in the stability of generated materials by using LLM generated crystal as the prior for the flow matching model.", "questions": "Have the authors looked at the RMSD of the generated crystals to their equilibrium structures? The wdist for density is high for FlowLLM. It might indicate that the model is generating structures far away from equilibrium", "limitations": "The authors noted the key limitation of FlowLLM is that it lacks end-to-end differentiability. However, the authors didn’t discuss potential negative societal impact of the model.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The main strength of the paper is the strong empirical result that shows FlowLLM significantly improves the percentage of stable crystals by replacing the prior distribution used in FlowMM with a LLM prior. The results are validated with DFT calculations and thus convincing. \n\nAblation study is conducted to isolate the contribution of LLM prior. They show that the atom type is the most important factor that contributes to higher stability.", "weaknesses": "Despite the significant improvement in empirical results, the method presented is a straightforward combination of 2 prior works. The method is also unprincipled in some cases. For example, the LLM prior doesn’t always generate valid crystals in the defined domain. It may also generate lattices with angles outside the 60-120 range. The authors say that they simply reject invalid samples from LLM. It would be nice to report the rejection rate. \n\nIn Table 1, the SUN rate is significantly lower than the stability rate for FlowLLM. It means either the model is generating non-novel or non-unique crystals. Can the authors also report the novel and unique rates?\n\nThe wdist for density is significantly higher than DiffCSP and FlowMM. It indicates that the density of the generated crystals is less accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Novelty\nAddressed in the global rebuttal.\n## Rejection Rate of the LLM\nAddressed in the global rebuttal.\n## Novel and Unique Rates\nThe reviewer's suggestion to report the novel and unique rates separately is well-taken. For our best model, 48% of the generated structures are stable and novel, of which 58% are also unique. Novel structures are defined as those that are not close (as determined by StructureMatcher) to any structure in the training or validation sets. We have added these novel and unique rates to the revised manuscript (section 5.3).\nNote that it is possible to obtain more diverse outputs by increasing the softmax temperature of the LLM. The softmax temperature controls the randomness of the sampling process – a higher temperature leads to more diverse and creative outputs increasing the novel and unique rates, but hurts stability rates. This can be seen from our results (Table 1) where different sampling parameters lead to different stability rates, but roughly similar SUN rates. Because we can select different trade-offs between stability, novelty, and uniqueness by tweaking this parameter, we chose to focus on the SUN rate which combines all of them.\n## wdist for density\nThe higher wdist for density in FlowLLM compared to FlowMM suggests that FlowLLM's generated structures might deviate more from the test set distribution in terms of density. However, this doesn't necessarily imply less accurate crystal structures, as FlowLLM could be learning a slightly different distribution of densities that is still physically valid. The fact that FlowLLM produces significantly more stable, unique, and novel materials than FlowMM after relaxation further supports this notion. The difference in density distribution could stem from the distinct base distributions used in each model. FlowMM employs a carefully designed base distribution for the unit cell, which has been shown to improve performance. It's possible that the LLM-learned base distribution in FlowLLM is inherently more challenging to transform to the target distribution, potentially due to multimodality or disconnectedness. Investigating the impact of different base distributions and their interaction with the flow matching process is an interesting avenue for future research.\nGiven that FlowLLM produces many more SUN materials after relaxation, while maintaining a high coverage, this may not be a major concern.\n## RMSD to ground state structure\nChecking the RMSD to the equilibrium (relaxed) structure is a nice suggestion. Using methods in `pymatgen`, this requires that the relaxed structure and the generated structure “match” according to `StructureMatcher`. In addition, we also compute the average difference in energy between the initial and relaxed structures, and the average number of steps to relax the structures. All of these were computed for the CHGNet relaxation.\nThe results are as follows:\n* FlowLLM: RMSD = 0.023, Match Rate = 94.9%, Delta Energy = 0.0898 eV/atom, Num steps = 37.97\n* FlowMM: RMSD = 0.096, Match Rate = 74.3%, Delta Energy = 0.3031 eV/atom, Num steps = 191.98\n\nHere, match rate is the fraction of cases where the generated and relaxed structures match. **These results show that FlowLLM actually produces structures much closer to the ground state.**\nWe mentioned in the global rebuttal that inference time depends on training, generation, and (pre-)relaxation. This result shows that FlowLLM is making big improvements by reducing the number of pre-relaxation steps to find the local minimum for an atomic arrangement. We will include this in the paper as another strong result of our method.\n## Potential negative societal impact\nWe thank the reviewer for pointing this out. We have updated the broader impact section to include the following:\n\n*“The quality of technologies, e.g. batteries, depend directly on the materials that comprise them. By upgrading the underlying materials, we could improve the technology by making it more effective, cheaper, or more biodegradable. This is a double edge sword because advanced, cheap technology can both provide strong benefits to society, but can also lead to more waste or other problems. An example case is plastic. Plastic provides untold benefits to humans in convenience and preventing the spread of disease; however, it also produces a huge amount of waste that has a significant environmental impact.*\n\n*One strong point of consideration is that the materials generated by FlowLLM only exist in a computational form. It is expensive and complicated to go from a computational representation of a material with useful predicted properties to an actual technology available to the public. For this reason, we believe that the potential negative impact from computational tools for material discovery to be quite limited.”*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eM9tZiG8ZP", "reviewer": "Reviewer_1vvS", "summary": "The paper proposed the FlowLLM, a hybrid approach for material generation that combines LLMs and RFMs, effectively leveraging their complementary strengths. Namely, this method generates samples from the parametric distribution using a two-step procedure, using 1) LLM and 2) RFM. By adopting this hybrid approach, experiments demonstrated that it exhibited better generation capabilities especially in terms of SUN and stability compared to the existing baseline.", "questions": "1. Are there any experiments supporting the claim that LLMs are weak with continuous type data and denoising models are weak with discrete type data?\n2. In FlowMM, bit representation was applied to atom types. Is there a specific reason for not using this approach?\n3. Approximately what is the rate of generating invalid crystals?\n4. Could you explain the process of how $\\lambda_f$ and $\\lambda_s$ were determined?\n5. Could you please explain in more detail about the invariance of flat tori and fractional coordinates?", "limitations": "Please refer to the Weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Since the LLM has been trained on material data, the proposed method can greatly simplify the denoising process.\n2. The approach of utilizing both LLM and RFM, two generative model approaches, to complement each other’s weaknesses is intriguing.", "weaknesses": "1. Since the output of the LLM is used as the starting point for the RFM denoising process, the generation time is longer compared to using a simple data distribution if there is no prior sampling of a large number ($N_{tr}$) of examples from the base distribution in advance. I would appreciate a comparison of the generation times between using 1) RFM with a simple base distribution and 2) FlowLLM without prior sampling from the LLM. Additionally, since the fewer required integration steps in the structural validity-number of integration steps plot could be attributed to the benefits of RFM, it would enhance credibility to compare this with FlowMM.\n2. Using the trained LLM as the base distribution for the RFM necessitates additional LLM training costs.\n3. The Preliminaries section of the paper overlaps significantly with that of the FlowMM paper. The structure and content of the Preliminaries section are very similar, which raises some questions.\n4. It is insufficient to verify the performance of the proposed method on only the MP-20 dataset. I believe it would be fairer to cover all the datasets addressed in other baselines.\n5. This appears to simply combine the models of CrystalLLM and FlowMM, making the novelty seem somewhat weak. If there are contributions beyond this combination, please let me know.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Generation Time\nConsidering the overall generation “cost” is a good point. Could you please clarify the following?\n> I would appreciate a comparison of the generation times between using 1) RFM with a simple base distribution and 2) FlowLLM without prior sampling from the LLM.\n\nWe interpret this to mean: What is the time comparison between FlowLLM and FlowMM to generate a sample? If so, we have added these to the global rebuttal.\n## Training Time of LLMs\nWhile we acknowledge the additional cost of training the LLM, we believe that the significant  performance improvements justify this investment. Additionally, since we fine-tune LLAMA-2 models, the actual cost of fine-tuning these LLMs is comparable to that of DiffCSP. An additional benefit of fine-tuning LLMs is that as better pre-trained LLMs become available, the fine-tuning time and generation quality of these models will keep improving.\n## Comparing structural validity vs NFE\nWe have updated the plot comparing structural validity versus number of integration steps for FlowMM and FlowLLM (see attached fig). FlowLLM converges significantly faster than FlowMM.\n## The Preliminaries section of the paper overlaps significantly with that of the FlowMM paper.\nWe have made some revisions to the Preliminaries section to reduce the overlap, and plan to make a larger revision in the final version. Here we will outline the intended format of our preliminaries section so that you can consider it:\n1. Introduction of crystals\n\n    a. the representation in math (similar)\n\n    b. the representation in an LLM (new)\n2. Symmetry of crystals (shorter, focusing merely on what symmetries exist)\n3. FlowMM (one or two paragraph summary about how FlowMM works)\n4.CrystalLLM (one or two paragraph summary about how CrystalLLM works)\n\nThe format is similar to the current paper, but we will reduce emphasis on the sections that build up flow matching for materials and instead focus on summarizing the contributions from FlowMM and CrystalLLM. We will expand the representation section so the reader understands exactly what goes into the algorithm.\nWe will use the extra space to showcase new results requested by reviewers (e.g. energy/rmsd from predicted to ground state, rejection rates, more datasets) and motivate the ways in which LLMs and Flow Matching methods synergize well.\n## More datasets\nIncluding other datasets is certainly a good idea, but it is unfortunately not possible to do so within the rebuttal period given the time constraints.\nAs a reminder, typically used datasets are Perov, Carbon and MPTS-52. Perov and Carbon are not stable structures and therefore not relevant to unconditional generation. MPTS-52 is indeed interesting due to its chronological data split and we will focus on that for the final version of the paper.\nWe want to emphasize that CrystalLLM was only tested on MP-20, and FlowMM and DiffCSP were only tested on MP-20 for De Novo Generation, which is the only task we focus on. While we agree that more datasets are better, there is significant precedent for using only MP-20.\n## “LLMs are weak with continuous data and denoising models are weak with discrete data”\n*Denoising models are weak with discrete data:* This can be seen from the compositional validity metric, where denoising models significantly lag language models. Compositional validity measures the fraction of crystals that are charge-neutral (which is a function of the atom types). This implies that the denoising models are generating many structures with invalid atom type combinations.\n\n*LLMs are weak with continuous data:* Effectively training LLMs on materials data requires using low precision representations of atom positions [Gruver et al. 2024], which limits the range of structures that can be generated by LLMs compared to denoising models. \nThis can be seen from the coverage recall metric, which measures the portion of the test distribution that is “covered” by generated structures. Crystal LLM obtains a lower recall than the denoising models (Table 1).\n## Bit representation for atom types from FlowMM\nOur motivation was based on the following observations: 1) the atom types generated by the LLM are superior to those learned by the FlowMM using bit representations; 2) the FlowMM excelled at generating structures conditioned on the atom types (“CSP” task), significantly outperforming prior methods. \nThese observations suggested to us that we should fix the atom types generated by the LLM and only use FlowMM for updating the atom positions and lattice parameters (similar to the CSP task). This allowed us to leverage the LLM’s strong comprehension of discrete data types AND FlowMM’s strong “conditional” generation capabilities.\nIt would, however, be interesting to try updating the atom types during denoising in future work.\n## How were 𝜆𝑓 and 𝜆𝑠 obtained?\nWe fixed $\\lambda_l = 1$, and swept over $\\lambda_f \\in $ {100, 200, 300, 400} (added to the manuscript).\n## Invariance of flat tori and fractional coordinates\nCould you please clarify this question? There are a number of invariances that could be relevant.\n\nThe idea of using fractional coordinates is to represent cartesian positions $x$ decomposed into a natural representation of the unit cell $l$ and the fractional coordinates $f$. Due to the fact that there are symmetries of the distribution of materials in cartesian space, some of those symmetries are inherited by the fractional coordinates. An important one is: translating all atoms by a fixed vector does not change the energy of a crystal, and therefore does not change its position in the thermodynamic competition for stability. That means we want to represent a distribution that is invariant to translation in fractional coordinates. We do this using our neural network for flow matching, like in FlowMM. In principle, the LLM part is not translation invariant, although the work by [Gruver, et. al. 24] shows that the LLM is approximately translation invariant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZEPgRwRV6", "reviewer": "Reviewer_ixaa", "summary": "This paper proposes using a second-step Riemannian flow matching (RFM) model, separately trained, to improve the quality of crystal structures generated by a pre-trained large language model (LLM), closely following Gruver et al. [1].\n\nSpecifically, the authors first follow Gruver et al. to fine-tune an LLM to enable the generation of crystal structures, and then train an RFM model from scratch to improve the stability, or in other words, quality, of the generated crystals.\n\nThe motivation of this work is clear. The reason for introducing this kind of post-processing machine learning model is to address the issue that LLMs cannot deal with real values, while atom positions and lattice parameters in crystals are usually real values.\n\nThe experimental results show that introducing this second-step RFM model can improve the stability of generated crystals. However, it faces limitations, such as the inability to generate materials with specific properties, as mentioned in the limitations section. Additionally, there are other well-established pipelines to refine the generated crystal structures that may need to be compared and discussed.\n\n\nReferences:\n\n[1] Nate Gruver et al. “Fine-Tuned Language Models Generate Stable Inorganic Materials as Text”. In: arXiv preprint arXiv:2402.04379 (2024).", "questions": "## Questions:\nMost of my concerns are listed above in the weaknesses. Addressing the weaknesses points and the following questions may impact the final score.\n\n1. Why not just use MLFFs like CHGNet to increase the stability of generated crystals from LLMs?\n\n2. Are a majority of the generated stable crystals similar or the same as each other? If not, why does the stability rate drop significantly when removing the duplicates (to SUN rate), up to more than three times (from 17.8 to 4.92)?\n\n3. Why only compare with methods without this kind of second-step refinement module?", "limitations": "Limitations are discussed in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "## Strengths:\n\n1. Clear motivation: LLMs face challenges when dealing with real values, and crystal structures are usually represented by lattice parameters and atom fractional coordinates that are real values. Introducing a post-process module to refine the generated structures of LLMs is reasonable.\n\n2. Good performance when compared with models without such second-step refinement modules. The experimental results show that introducing such a second-step RFM model will increase the overall stability of generated crystals.", "weaknesses": "## Weaknesses:\n\n1. Missing discussions with other second-stage refinement strategies that are well-established. The generated crystal structures from LLMs can also be refined by machine learning force fields (MLFF), such as M3GNet, CHGNet (which the authors have used when calculating stability), or MACE-MP. For these MLFFs, one can directly use them without training a separate RFM model from scratch to refine the generated crystal structures.\n\n2. The contribution may be a little limited. Given that there are well-established methods for the proposed issue in this paper, the contribution of introducing an RFM model to increase stability is somewhat limited.\n\n3. The stability rate drops significantly when removing the duplicates (to SUN rate), up to more than three times (from 17.8 to 4.92). This means a majority of the generated stable crystals are similar or the same as each other.\n\n4. Inability to generate materials with specific properties as mentioned in the limitations section. If the ability of this model is limited to generating stable crystal structures, there are other computationally cheaper models like DiffCSP that the authors have compared with for this task. Thus, it would be beneficial to add more discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## > “Why not just use MLFFs like CHGNet to increase the stability of generated crystals from LLMs?”, “Why only compare with methods without this kind of second-step refinement module?”\n\nWe appreciate the reviewer’s thorough knowledge on the matter of refinement, but believe a clarification is necessary. All models evaluated in our paper, including the baselines, utilize a second-step refinement process involving relaxation with a machine learning force field (specifically, CHGNet) followed by DFT relaxation. Therefore, the comparison presented in the paper is fair and focuses on the effectiveness of different generative models in producing high-quality structures that lead to stable materials after refinement.\n\nThe superior stability and SUN rates we obtained for FlowLLM suggest that **our flow matching based refinement provides significant benefits BEYOND those provided by second-stage refinement with an MLFF.** \n\nOne explanation for this superior performance is that MLFF relaxations only find a local energy minimum close to the generated structure, while conditional generation using riemannian flow matching does not have such a limitation. Indeed, on visual inspection of the generated structures, we found many cases where the flow matching refinement produced significantly different structures than what they started with.\n## > The contribution may be a little limited. Given that there are well-established methods for the proposed issue in this paper, the contribution of introducing an RFM model to increase stability is somewhat limited.\n\nAs mentioned above, the RFM based refinement goes beyond what an MLFF provides, and our superior results compared to other models + MLFF relaxation demonstrates this. We also address novelty in the global rebuttal above.\n## Inability to generate materials with specific properties\nThe reviewer points out the limitation of FlowLLM in generating materials with specific properties. We acknowledge this limitation in the paper, and leave extending our model to property optimization for future work. \n\nHowever, it's crucial to recognize that the generation of stable materials itself is a significant and challenging problem in materials science. The vast majority of theoretically possible materials are unstable and therefore not synthesizable. The ability to efficiently generate a high proportion of stable materials, as demonstrated by FlowLLM, is a substantial contribution in its own right. It dramatically reduces the search space and computational cost associated with identifying promising candidates for further investigation and potential synthesis. The focus on stability serves as a crucial filter, ensuring that the generated materials have a higher likelihood of practical relevance and applicability.\n\nWe also want to point out that the LLM aspect of our proposed model leaves open the possibility of conditioning to produce specific properties in a way that is unheard of for other diffusion-based approaches: asking the LLM to produce materials with those properties. Trying to generate materials with specific properties using LLM prompting is a very interesting, albeit distinct and significant *further* contribution that we leave for future work.\n## Stability vs SUN rate\nTo compute SUN rates, we remove duplicates, and also any generated structures that are close to a structure in the training or validation data.\n\nFor our best model (τ=0.7, p=0.9), 1,782 of the 10,000 generated structures are stable, out of which 926 were close to training data samples. Out of the remaining 856 structures, 492 were unique structures. Therefore, the majority of the difference between stability in SUN rates is explained by the fact that our model generates many structures close to structures in the training data. This is expected behavior for a generative model that has been trained to accurately capture the distribution of the training data. Note that the most important aspect is that the SUN rate is high, not whether the non-SUN materials are in the training data or not.\n\nWe believe this offers great opportunity for future work where we can use the flexibility of LLMs for complex conditioning, while retraining the ability of our network to accurately predict stable structures (Crystal Structure Prediction) due to FlowMM’s strong performance.\n## Cheaper methods\nWhile methods such as FlowMM or DiffCSP may not require tuning a LLM, they also produce far fewer SUN structures. In this dimension, FlowLLM is unequivocally better as its SUN rate is ~50% higher than the runner up.\n\nYou raise an important point that one should carefully consider cost tradeoffs in this process, but they are difficult to quantify. \n1. Inference costs have been addressed in the global rebuttal. The generation time for FlowLLM is comparable to FlowMM, which is significantly cheaper than DiffCSP, as it requires  fewer integration steps.\n2. There is no doubt that an LLM with FlowMM will have a higher training cost, but it results in much stronger performance that cannot be replicated by merely applying existing MLFF (or DFT) relaxations to generations from the LLM. The two-step procedure makes a quantitative difference in SUN materials with higher training costs, but still low inference costs compared to DiffCSP.\n3. Finally, the majority of the cost in a material discovery pipeline lies in the DFT relaxation that is run to check if the generated material is stable or not. Since FlowLLM generates more SUN structures, and these are generally closer to their ground state structures (see “RMSD to ground truth structures” section in the rebuttal to Reviewer sJw2), FlowLLM requires much less compute for DFT relaxations than competing methods like FlowMM. On average, structures generated by FlowLLM only need 7.08 ionic steps of DFT relaxation, while those generated by FlowMM require 14.35 steps. Therefore, **FlowLLM is able to cut the DFT cost by a factor of 2.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed reviews and insightful comments. We are proud to say that all reviewers are recommending acceptance at this time. However, we aim to address any remaining critiques to improve the paper even further!

We summarize the perspective of the reviewers as follows: *Our method, FlowLLM, represents a significant improvement in empirical results when compared to existing work such as FlowMM and CrystalLLM. These results are achieved by proposing a novel and intriguing combination of LLM and Flow Matching in a way that the two methods complement and support one another. The motivation is therefore clear: take the best of the two methods and combine them. At the same time, this combination is somehow simple and therefore may have limited novelty.*

We appreciate the acknowledgement from reviewers that our method is a significant empirical improvement from previous methods. We address the critique of limited novelty below, but want to emphasize here that simplicity and novelty are not the same thing. We believe our proposal may be simple, but one that has heretofore not been explored in the literature and is therefore novel. Also, we view the simplicity of the approach as an asset rather than a hindrance.
## RMSD to ground state
Based on suggestions from one of the reviewers, we added a new set of results to the paper (Appendix D), comparing the generated structures to their ground state. We found that **FlowLLM was generating structures much closer to the ground state** than FlowMM. We believe this to be a very strong result, and re-emphasizes the efficacy of the FlowLLM method. See RMSD to ground state structure section in rebuttal to Reviewer sJw2.
## Novelty
We agree that our method combines two existing techniques. However, we respectfully disagree with the assessment that the combination is straightforward. The novelty of our contribution lies in the innovative combination of LLMs and Flow Matching:
1. **Unique Combination of LLMs & RFM:** To the best of our knowledge, no prior work has integrated the strengths of LLMs and Flow Matching together in one model for materials generation. This unique combination allows us to leverage the strengths of both methods, resulting in a substantial improvement in the rate and cost of generating stable materials.
2. **Exploiting Flow Matching's Flexibility:** FlowMM is the only prior work that uses flow matching for generating materials. The distinctive feature of Flow Matching, its ability to map to arbitrary base distributions, sets it apart from diffusion models. We are the first to recognize and leverage this property to take the output of an LLM and transform it with Flow Matching. It also opens the possibility of future work that is only possible with the flexibility of an LLM such as prompting for certain material properties.
3. **Bridging Disparate Philosophies:** The conventional wisdom in the field often views methods with inductive biases, like FlowMM, as an exclusive alternative to scalable methods like LLMs. Our work challenges this notion by successfully integrating these two seemingly disparate philosophies. The impact of such integration may be applicable beyond material generation to other domains such as proteins or small molecules. Those domains also apply strong, domain-specific inductive biases.
The substantial improvement in performance achieved by FlowLLM, as evidenced by our empirical results and ablation study, is very novel. Furthermore, the LLM's contribution extends beyond atom type prediction, serving as a learned base distribution that significantly enhances the learned target distribution. This combination and the resulting performance gains highlight the innovative nature of our work.
## Rejection Rate of the LLM
We acknowledge the limitations of the LLM prior in generating invalid crystals or lattices with angles outside the defined range. In our experiments, we observed a rejection rate of less than 0.5% due to invalid samples from the LLM (with a softmax temperature of 0.7). Note that since we can identify and reject these invalid cases, the inputs to flow matching are always valid materials. We have included this rejection rate in the revised manuscript (section 4.2).
In future work, we can explore strategies to address these limitations, such as incorporating constraints within the LLM.
## Inference Time
To compare generation speed between FlowLLM and FlowMM, we ran inference with both methods on an A100 GPU to generate 10000 samples. The time to generate 10000 samples breaks down as follows:

`InferenceTime(FlowLLM) = generate_text_time + rfm_time(250 steps) = 89.6 min`

`InferenceTime(FlowMM)  =                                    rfm_time(750 steps) = 65.1 min`

Since what we really care about is generating novel materials, a better metric is to compute the time required to generate a SUN material, which is:

`TimePerSUN(FlowLLM) = 10.9 sec / sun material`

`TimePerSUN(FlowMM) = 16.14 sec / sun material`

These results show that the inference times for the two methods are comparable, while FlowLLM generates new SUN materials faster than FlowMM.

Meta-review: The paper presents a hybrid approach that combines LLMs and conditional flow models for material design. LLMs are used to generate a noisy version of the material and flow models are used to refine the initial generation. The reviewers marked the strong empirical results as one of the main strengths of the work and found the idea of combining LLMs and Flows interesting. However, they also raised concerns regarding the novelty, computational cost, and lack of comparisons against force field-based approaches which were mostly addressed in the rebuttal. Given these we are happy to recommend the paper for acceptance.
Decision: Accept (poster)