LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation
Authors: Subhojyoti Khastagir
KISHALAY DAS
Pawan Goyal
Seung-Cheol Lee
Satadeep Bhattacharjee
Niloy Ganguly
Venue: NeurIPS 2025 poster ()
Keywords: ML4Materials
Diffusion Models
Crystal Materials
Material Generation
AI4Science
Primary area: machine_learning_for_sciences
TL;DR: we propose CrysLLMGen, a hybrid framework that combines the strengths of both LLMs and diffusion models for periodic material generation.

Abstract: Recent advances in generative modeling have shown significant promise in designing novel periodic crystal structures. Existing approaches typically rely on either large language models (LLMs) or equivariant denoising models, each with complementary strengths: LLMs excel at handling discrete atomic types but often struggle with continuous features such as atomic positions and lattice parameters, while denoising models are effective at modeling continuous variables but encounter difficulties in generating accurate atomic compositions. To bridge this gap, we propose CrysLLMGen, a hybrid framework that integrates an LLM with a diffusion model to leverage their complementary strengths for crystal material generation. During sampling, CrysLLMGen first employs a fine-tuned LLM to produce an intermediate representation of atom types, atomic coordinates, and lattice structure. While retaining the predicted atom types, it passes the atomic coordinates and lattice structure to a pre-trained equivariant diffusion model for refinement. Our framework outperforms state-of-the-art generative models across several benchmark tasks and datasets. Specifically, CrysLLMGen not only achieves a balanced performance in terms of structural and compositional validity but also generates more stable and novel materials compared to LLM-based and denoising-based models Furthermore, CrysLLMGen exhibits strong conditional generation capabilities, effectively producing materials that satisfy user-defined constraints. Code is available at \url{https://github.com/kdmsit/crysllmgen}

LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation
Subhojyoti Khastagir Kishalay Das Pawan Goyal', Seung-Cheol Lee",
Satadeep Bhattacharjee", Niloy Ganguly²
1 Indian Institute of Technology, Kharagpur, India
Indo Korea Science and Technology Center, Bangalore, India Equal Contribution
Abstract
Recent advances in generative modeling have shown significant promise in de- signing novel periodic crystal structures. Existing approaches typically rely on either large language models (LLMs) or equivariant denoising models, each with complementary strengths: LLMs excel at handling discrete atomic types but often struggle with continuous features such as atomic positions and lattice parame- ters, while denoising models are effective at modeling continuous variables but encounter difficulties in generating accurate atomic compositions. To bridge this gap, propose CrysLLLMGen, hybrid framework that integrates an LLM with a diffusion model to leverage their complementary strengths for crystal material generation. During sampling, CrysLLMGen first employs fine-tuned LLM produce an intermediate representation of atom types, atomic coordinates, and lattice structure. While retaining the predicted atom types, it passes the atomic coordinates and lattice structure to a pre-trained equivariant diffusion model for refinement. Our framework outperforms state-of of-the-art generative models across several benchmark tasks and datasets. Specifically, CrysLLMGen not only achieves balanced performance in terms of structural and compositional validity but also generates more stable and novel materials compared to LLM-based and denoising- based models Furthermore, CrysLLMGen exhibits strong conditional generation capabilities, effectively producing materials that satisfy user-defined constraints. Code is available at fictps://githan com/ kwkaitditttt tt crysllmgen
Introduction
Discovery of novel direesiiiniionall crystal materials with desired chemical properties remains fundamental challenge the field of materials design. These materials play a critical role in driving innovations such as development of batteries, solar cells, and semiconductors [] 2 Generating crystal materials presents a significant challenge due to its intrinsic complex structure. Unlike molecules, which are typically represented as regular graphs, crystal materials are typically modeled by a minimal unit cell containing all the constituent atoms in different coordinates, repeated infinite times in 3D space generating stable crystal material requires the simultaneous prediction of both discrete (atomic types) and continuous components (atomic coordinates and lattice structures).
regular lattice which makes material structures periodic nature (3). Hence,
Traditionally, material discovery has relied on either computationally expensive density functional the- ory (DFT) simulations 5 6 labor-intensive experimental procedures. However, recent advances in generative modeling have opened up promising directions for designing novel crystal structures with greater efficiency and scalability. Current generative approaches for material generation be
+Correspondence to Kishalay: kishalaydas @ kepian www.iaa.iiggp.aca.....mo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Denoising Models
LLMs LLaMA-2(7B)
LLM+Diffusiun
Validity Metric
CDVAE 8
DifICSP TO
FlowMM 14
7
LMGen
Structural Validity
100
100
96.85
96.40
99.94
Compositional Validity
86.70
83.25
83.19
93.30
93.55
Table 1: Validity Evaluation of Crystal Generation across model classes: Denoising models tend to achieve higher structural validity, while LLMs excel in compositional validity, LrysLLMGen demonstrates a balanced performance, delivering both high structural and compositional validity.
broadly classified into two main categories: (I) autoregressive large language models (LLMs) 77. and (2) renoising-bbsee frameworks, including diffusion models 8 10 13] and flow matching techniques [5). Each of these generative models has distinct ad advantages and drawbacks. LLMs excel capturing discrete information, making them particularly effective at predicting atomic types and achieving high compositional validity. However, they often face challenges in accurately generating continuous data such as atomic positions and lattice parameters due to limitations in finite precision encoding, leading to reduced structural validity. In contrast, denoising based models are inherently better handling continuous variables and preserving geometric equivariances, resulting in higher structural validity. Nevertheless, these models struggle with discrete components, such as correctly identifying atomic types, which leads to lower compositional validity (Table-[].
In this work, we aim to harness the complementary strengths of both LLMs and denoising models to mitigate their individual limitations. To this end, we introduce CrysLLMGen. a novel hybrid framework for periodic material generation, which consists of two modules: fine-tuned LLM and a pre-trained Diffusion Model. Our sampling process begins with the LLM that generates intermediate predictions for atom types, coordinates, and lattice. Given that LLMs excel at modeling discrete information, the generated atom types are well-aligned with the true atomic distributions observed in material datasets. Hence, we retain atomic types and then pass the predicted atomic coordinates and lattice structures pre-trained equivatiant diffusion model, which refines and adjusts these continuous components to produce more stable and structurally valid crystal materials. Our simple yet effective hybrid approach enables the joint modeling of multimodal features, both discrete and continuous, of crystal materials, significantly enhancing the validity, stability, and novelty of the generated structures. A major advantage of incorporating large language models (LLMs) is their ability to process natural language prompts, allowing for conditional generation. Additionally, denoising models handle crystal symmetrics and the equivariant nature of material data distributions. Importantly, our proposed framework is architecture aggnssiic, meaning it can readily accommodate future advancements in both LLMSs and denoising networks without requiring substantial modifications.
Extensive experiments on benchmark datasets demonstrate that hybrid models can effectively capture both the discrete and continuous aspects of crystal structures, resulting in improvements in both compositional and structural validity. Furthermore, our results show that CrysLL.MGes generates 32 % and 68% more stable materials compared state-of-the-art LLM-basedd models and best-performing denoising model. To sum up, our novel contributions in this work are as follows:
To the best of our knowledge, CrynLLMGen is the first hybrid model that combines large language models (LLMs) with diffusion models for crystal material generation.
Extensive experiments benchmark datasets demonstrate that CrysLLMGen consistently outperforms state-of-the-arr models in both structural and compositional validity. It also generates more stable, unique, and novel crystal structures compared to existing approaches.
CrynI.LMGen shows strong generative capability under conditional prompts, effectively pro- ducing materials aligning with specified atomic compositions and space group constraints.
2
Related Work: Crystal Material Generation
Earlier works on periodic material generation primarily focused on atomic composition. often overlooking 3D structural details. With the rise of generative models, approaches using VAEs or GANs have aimed to generate 3D periodic structures by representing materials as voxel images [16
17 18 [9] or embedding vectors 20 21 22]; however, these methods neither ensure structural stability nor maintain invariance to Euclidean or periodic transformations.

---

Diffusion Models. Recently, equivariant diffusion models have become the leading approach for stable crystal material generation due to their ability utilize the physical symmetries of periodic structures. Models like CDVAE S and SyMat [9] combine VAES with score-based denoising networks operating on atomic coordinates through equivariant GNNs, ensuring Fuclidean and periodic
invariance. Subsequent models, including DiffCSP [TO] and MatterGen jointly learn atomic
composition, coordinates, and lattice parameters via diffusion frameworks. UniMat [13] extends this idea with a unified 4D tensor representation that models discrete atom types and continuous coordinates using a probabilistic diffusion model. More recently, text-guided diffusion models such as TGDMat [23] and Chemeleon (44 integrate contextual embeddings from pretrained models like MatSciBERT or CLIP into GNN-based denoising networks to generate stable periodic materials aligned with textual descriptions.
Symmetry-Awane Generation. While conventional diffusion models learn atomic positions inde- pendently, incorporating space group symmetry greatly reduces model complexity. DifCCSP++ [12 the first to extend DifCCSP by enforcing symmetry constraints on lattice parameters and atomic coordinates, ensuring correct lattice systems and restricting atoms to Wyckoff positions from training templates. SymmCD (25 further advances this by jointly learning fractional coordinates and Wyckoff positions using a sitessymeerry-wwar representation. By modeling only one representative atom per crystallographic orbit, both methods substantially reduce generative complexity.
Latent Diffusion Models. A major limitation of current diffusion based methods is their reliance on high-ding sigh-dimensional feature spaces that jointly model atom types, fractional coordinates, and lattice structures: an inherently multimodal distribution with distinct statistical characteristics for each component. This leads high computational costs during training and inference, restricting their use in resource- limited settings. Recent works such as CrysLDM (26 and ADIT (27) address these challenges through latent diffusion models, which operate in a compact latent space to significantly reduce sampling time and computational overhead, offering improved efficiency over conventional
feature-space diffusion approaches.
29 30 has recently emerged as strong alternative to diffusion-based methods for crystalline material generation. Unlike diffusion models that iteratively denoise samples from Gaussian prior, FM directly learns time-dependent velocity field that continuously transports an arbitrary base distribution toward the target distribution of stable crystals. The first such application, FlowMM [14), introduced a representation that preserves global rotational and translational symmetries and enforces periodic boundary conditions through equivariant flows to ensure symmetry invariance. Building on this, FlowLLM [15] combines geometric inductive biases with base distributions guided by large language models 7. Additionally, CrysBEN (31) employs periodic Bayesian Flow Networks with entropy conditioning and non-monotonic dynamics to better capture periodicity in non-Euclidean space, enhancing both sampling efficiency and generation quality.
Flow matching. Flow matching (FM) 28
Large Language Models. Large language models (LLMs) are increasingly applied in the natural sciences as versatile priors for reasoning over sequences, graphs, and spatial data, a trend that has recently extended to materials generation. By representing crystal structures as textual descriptions of unit cells and atomic positions, token-based language models 32 have shown strong performance in generating stable and valid materials.
Key Differences with FlowLLM [15]. By design, our proposed CrysLLMGen comes close to FlowLLM [3), however here are key distinctions in design and methodology. Our approach differs from FlowLLM in three key aspects. (1) Generative Module: While FlowLLM employs a flow-matching framework as its generative component, our method, CrysLLMGen, utilizes a diffusion- based model. Through comparative analysis on unconditional material generation (Table ( we observed that the diffusion-basee based DiffCSP consistently outperforms the flow-matching model FlowMM across most benchmark metrics. (2) Parallel Training: Unlike FlowLLM's 's sequential training paradigm, where a large language model (LLaMA-2)) first fine-tuned on the dataset and its generated samples are subsequently used to train the flow fow-matching model, our approach trains both the LLM and the diffusion module in parallel on the same training dataset. (3) Integration Strategy: In Fl,wLLM, the material structures generated by the LLM are directly refined by the flow-matching module. In contrast, we treat the -enerrree material structures as intermediate representations, which are injected into our diffusion model at intermediate timestep 7 (where 0 I T S T to
3

---

initiate denoising from that point. This integration strategy allows more effective refinement and generation, harnessing the complementary strengths of both the LLM and the diffusion model.
3 Preliminaries
3.1 Crystal Structure Representation
The crystal structure can be visualized three-dimensional point cloud, where atoms are positioned at specific coordinates within minimal unit cell that repeats periodically in space to form the infinite, periodic crystal lattice. Given a material with N number of atoms in its unit cell, we can describe the unit cell by two matrices: Atom Type Matrix (A) and Coordinate Matrix (X). Atom Type Matrix A (a a2 N\" E RN denotes set of atomic type in one hot representation (k: maximum possible atom types). On the other hand, Coordinate Matrix X = atomic coordinate positions, where x; E I corresponds to coordinates of 7th atom in the unit cell. Further, there is an additional Lattice Matrix L 11, E IR3x3, which describes how a unit cell repeats itself in the 3D space towards I lz and 13 direction to form the periodic 3D structure of the material. Formally, given material can be defined as M (A, L) and we can represent its infinite periodic structure as X = { = x, + Σ3= 1j=1 k A {ā. a_ = a,} where ki, k2, ka, E Z, SisN.
E RRNx3 denotes
3.2 Symmetry in Crystal Structure
Crystal structures inherently exhibit range of physical symmetries, which are fundamental to their characterization and physical properties. Consequently, major challenge for any generative model designed for crystal generation is to ensure that the learned distribution satisfies periodic E(3) invariance, meaning invariance permutation, translation, rotation, and periodic transformations. Permutation invariance implies that reordering the indices of the constituent atoms does not alter the identity of the material, whereas translation invariance means that shifting all atomic coordinates by constant vector leaves the material structure unchanged. Rotational invariance indicates that rotating both the atomic coordinates and the lattice matrix does not affect the material's identity. Periodic invariance arises from the fact that atoms in a unit cell repeat infinitely along the lattice vectors, allowing multiple valid representations through different choices of unit cells and coordinate matrices for the same material.
4 Methodology
4.1 Problem Formulation
In this work, we consider generative modeling of 3D crystal structures from scratch, to discover new stable materials. Formally, given dataset M = [M,) containing crystal structure M, = (A;, L,), the goal is to capture the underlying data distribution p(M) via learning generative model (M), where a set of learnable parameters. While training. we need fo to ensure that the learned distribution is invariant to different symmetry transformations mentioned in fection[3.2 Once trained, the generative model can sample valid and stable material structures that remain invariant under various symmetry transformations (unconditional generation). Additionally, by providing specific constraints during sampling, the model can generate structures that meet desired criteria (conditional generation).
4.2 Proposed Framework: CrysLLMGen
Our proposed framework, CrysLLMGen, is a hybrid model that combines large language model (LLM) with a diffusion model (DM) for crystal material generation. We fine-tune the LLM and train the diffusion model independently as standalone components using the training dataset. The sampling process begins by prompting the LLM to generate initial predictions for A, X, and L. Given the LLM's strong capability modeling discrete information, we assume that the predicted atom types closely align with the true atomic distributions found in material datasets. Therefore, we retain these predicted atom types as the final atomic composition of the generated material. The atomic coordinates and lattice parameters are then passed to diffusion model, which refines these

---

Below a description of bulk material.
<The number is 164, The chemical formulo Mg(AiSi)2. The crystal trigonal>> Generate description the lengths and the lattice vectors and then element coordinates...
X
A.X
Ao Lo Generated Material
LLM Inference
L
Prompt (Uncondiional/Condttionn))
Intermediate Representation
Denoising Process
Figure 1: Model Architecture of our proposed MMen. The sampling process begins with prompting the LLM to generate intermediate representations for A, X, and L. The predicted atom types (A) are retained as-is, while the atomic coordinates (X) and lattice parameters (L) are further refined through a denoising diffusion network.
continuous components ensure structural validity and enhance the stability of the generated crystal materials. Next, we will explain in details both the components of our proposed framework.
4.3 Large Language Model (LLM) fo
FLLMM for Crystal Materials
We build upon prior work 7 to develop our LLM-based component, in which crystal structures are converted into sequential representations and fed into the LLM for further processing. Fundamentally, I.MMs model a distribution over sequences using autoregressive next-token prediction task, where each token is generated based on a categorical distribution conditioned on all preceding tokens in the input sequence. We transform dataset of crystal structures M = {M; ! into a corresponding dataset of sequences W {W1, W2, .... W J. where each sequence W; represents the CIF text format of the crystal structure M, Next, we tokenize each text sequence representing a crystal structure. We represent each crystal material M, using fixed- exed-precision values: fractional 3D coordinates to two decimal places, lattice lengths to one decimal place, and angles are encoded as integers. Atom types are treated discrete tokens. For our LLM backbone, adopt the ffate-of-th--ee ahe LLaMA-2 models, which have demonstrated strong performance in material generation tasks across several recent studies. Additionally. ILLMM-22 tokenizes numbers individual digits by default, feature shown to significantly enhance performance on arithmetic-related tasks.
rounded
We use the pre-trained ILLMMA-277B base model our starting point. It fine-tuned dataset of crystal structures represented as text sequences. We use task-specific prompts to fine-tune the model on different tasks like unconditional or texl-conditional generation. Although larger LLaMA variants such as -LaMA-2-13BB LLaMA-2-70B, or the newer LLaMA-3 models may offer enhanced performance, we opted for the LLaMA-2--1 2-7B model balance computational efficiency, the scale of available pretraining data, and practical deployment considerations for the broader materials science community. Exploring higher-end models is left direction for future work.
Crystal structures exhibit translational and rotational symmetries, which standard LIM architectures are not inherently equipped to capture. To address this, we adopt the data augmentation strategy proposed by (7). Once trained, the LLM can generate sequences by sampling tokens sequentially from the learned categorical distribution.
4.4 Diffusion Model /Diff for Crystal Materials [Atom Coordinates, Lattice Structures]
Prior works leverage equivariant diffusion models to jointly learn atom types A, atomic fractional coordinates X, and lattice structure L. However, in our setup, we retain the atom types predicted by the LLM as the final atomic composition the generated material, while the atomic coordinates and lattice structure refined using diffusion model. To enable this, train the diffusion model on a structure prediction task, where, given the atom types of a crystal material, it learns the joint distribution of atomic fractional coordinates X and lattice structure L.
Diffusion on Atom Coordinates (X). Atom coordinates can be diffused in two ways: by apply- ing noise to either the cartesian coordinates or the fractional coordinates. Prior works such as
5

---

CDVAE 8 adopt cartesian coordinate diffusion, while DiffCSP [10] operates on fractional coordi- nates. In our approach, since we jointly learn atomic positions and the lattice matrix, we follow the methodology of DiffCSP and perform diffusion on the fractional coordinates. Coordinate Matrix E T Nx3 contains fractional coordinates of constituent atoms, that resides in xx / x x3 induced by the crystal periodicity. Since the Gaussian distribution used in DDPM is unable to model the cyclical and bounded domain X, it is not suitable to apply DDPM to model X. Hence at each step of forward diffusion, we add noise sample from Wrapped Normal (WN) distribution 33 to X and during backward diffusion leverage Score Matching Diffusson Networks 34 B3 to model underlying transition probability q(x, Xo) Nyw (X, Xo. of
xN x x= x2, quotient space space TR. x33
In specific, at each 1th step of diffusion, we derive X; as Xx = Las (Xo + oTE*)) where, 6* is noise, sampled from (O, tt. the noise scale following exponential scheduler and fz is truncation function. Given fractional coordinate matrix X, truncation function (x) -X- [X]) returns the fractional part of each element of X. As argued in [10), q(X,) (Xo) is periodic translation equivariant, and approaches uniform distribution (0, 1) for sufficiently large values of Ør. We use
denoising network D (Ar, Xt, L,, 1) to model the backward diffusion process, which is trained using the following acore-matching objective function:
Looord = Ex,xg(X,1xx)) Vxx (ogo(X - -è" (M,,, 1)12 t0((1)
where Vx, (o,(g(((X((Xo) & ExeeNaS exp(-
||x Xo+ is the score function of transitional distri- 20?
bution and é²t (M;, denoising term.
Diffusion on Lattice Structure (L). Lattice Matrix L = [41 12.12, E R is global feature of the material which determines the shape and symmetry of the unit cell structure. Since L is in continuous space, we leverage the idea of the Denoising Diffusion Probabilistic Model (DDPM) [36 for diffusion on L. Specifically, given an initial lattice matrix Lo p(L), the forward diffusion process gradually corrupts it over 7 timesteps, resulting in noisy lattice matrix Ly. At each timestep t, the transition is governed by a conditional probability distribution g(L, Lo). which can formally
expressed as: qL, Lo) = (L, vāzLo. (1 - T)) where, ā, = II ak, a = 1 - Bg and {3e E 1) controlling the level of noise added at each step. By reparameterization, we can rewrite as: L, = vozLo + -I- ā,ch where, e¹ is a noise, sampled from N( I), added with original input sample Lo at 4th step to generate L,, After T such diffusion steps, noisy lattice matrix L1 is generated from prior noise distribution ~ (O, I). During denoising, the reverse conditional distribution can be expressed as follows p(L,- MM,) WV 1d (M,,, B, Q- where (M, = -aa (L. VI-a. (M,, t)). Intuitively, is the denoising term that needs to be subtracted from Lg to generate L--1- We use a denoising network (A, X,, 1) to model the noise term (', t). Following the simplified training objective proposed by [36], train the aforementioned denoising network using l2 loss between é' and ex
(1-6.)
1-a,
-az
Liattice = Ee Be,
-/-112
Denoising Network. For the denoising network fo (A Xx, Lr, in the reverse diffusion process, we extend the CSPNet architecture [IO], which is built upon the Equivariant Graph Neural Network (EGNN) (37]. This architecture is specifically designed conditions inherent in periodic crystal structures. At the t layer message passing, the Equivariant Graph Convolutional Layer (EGCL) takes as input the set of atom embeddings h² = [ h ...,,% atom coordinates x² +,.,xx
satisfy the periodic E(3) invariance
x.. and Lattice Matrix I. and outputs a transformation on h k+1 Formally, we can define the h layer message passing operation as follows:
m = (m h.. L' (rr xf) m Σ mayii h
h
Ph (h (m. m:}
where, Pm and Ps are multi-layer perceptrons (MLPs), and ViFT denotes a Fourier Transformation function applied to the relative difference between fractional coordinates x² andx/% The use of Fourier Transformation ensures invariance to periodic translations and captures a spectrum of frequencies from the relative fractional distances, which is beneficial for modeling the periodic nature of crystal structures. Input atom features h" and coordinates x² are fed through XC layers of EGCL to produce

---

é" and ex as :
1=1 Σ )²);; === px =xxxx(h
= Lpr.
where PL. Px are multi-layer perceptrons the final layer embeddings.
Algorithm 1 Sampling Process of CrysLLMGen
1: Input: A pretrained LLM FLLM, A pretrained Diffusion Model f pDir (Condi-
tional/Unnoonitional) Prompt for Material P, Intermediate step T
2: Step 1: Sample from LLM FLLM
3: Sample A, X and L from LLM given prompt P: LA - PLLM(P)
4: Step 2: Refinement using Diffusion Models fg fD1ff
5: Ap := -A/ Retain *
6: L, := +, =
7: for 4- 1 to do
8:
xx,ch + SooA,, X,, Lt,t)
9:
L- _ (L V1-ai
8. _ +
1B1 1-
T-a e~ N00,1)
Var,
02-1Voi-0
10:
X,_ t w + (07 - -0²-1)*
E; N(O,T)
11:
_x + fel (A, N.,, 1c-
hc-1
12: The - slep_size size
a1
13:
X,-1 t w (X + The + v 22
14: end for
15: Output: New Generated Crystal Material: M Mnew = (A,Xo: ko)
4.5 Training and Sampling
We train both components of proposed framework, CrysLLMGen, the LLM and the diffusion model, independently using the training dataset. For the LLM component, we start with the pre- trained LLaMA-2-7I base model and fine-tune with the Low-Rank Adapters (LoRA) method on crystal structures represented as text sequences, guided by task-specific prompts for either conditional or unconditional generation. this work, we adopt the prompt formats proposed in [7] for both lasks. The diffusion model is trained for a structure prediction task using a combined loss function: L Llattice Leoord.
Once trained, the sampling process begins with the LLM component. A task-specific prompt is provided input, and the LLM generates sequence by sampling tokens sequentially from its learned distribution []] This yields an intermediate representation consisting of predicted atom types A, atomic coordinates X, and lattice structure É. Given the LLM's strong capacity for modeling discrete information, we hypothesize that À closely aligns with the true atomic distributions observed in material datasets. Therefore, we retain Á the final atomic composition of the generated material. But, the continuous variables X and i. require further refinement and hence we need to feed them to the diffusion model. Typically, diffusion models begin the sampling process with fully noisy inputs, i.e., XT, Lr (0, I) and iteratively denoise them over T steps to generate X, and Lo- approximating the target data distribution. However, in case, X and i. produced by the LLM are not pure noise but meaningful intermediate representations. Therefore, instead of starting the denoising process at step T, inject these representations an intermediate timestep 7, where
'LLMs sometimes hallucinate and generate invalid or unphysical chemical elements. address this, we employ a simple validation strategy that checks and filters compositions during sampling, immediately discarding any invalid ones. On average, about 2 2-5% of the sampled structures are removed due to invalid atom types or compositions.
7

---

Dutsset
Casegory
Model
Structural
Validity
Compusitional
Precision
Coverage
Recall
Demsiyy
Property
* Elerment
CDVAR
100
99.49 49
99.15
1,452
DifTCSP
100
83.25
09.76
99.71
0.252
0.339
Dillusion Modells
99.94
99.25 99.59
99.73
0.235
0.375
MatterGeen
100
86.34 34
99.45
99 39
0.459
0.254
UniMat
17.20
89.40)
99. 70
99.80
0
BJ656
SymmCD
92.30
87.13
98.78
97.33
0.531
0.210
MP-20
Flow Matcching
FloudJ.M
HowMM
99.94
85
90.84
83.19
99.82
99.58
96.95
99.49
0.239
1,142
0AIK
0,150
Buyesian Flow Networks
CryBFN
100
87.51
99.79
99.00
0.306
0.163
LLMs
Lama-2 (7B)
97.70
93.55
99.32
96.95
1,575
0.272
LLM Diffusion
LMGnn (7B)
99.94
93.55
99.84
98.55
0.972
0.272
CDVAE
100
99:45
DiTTCSP
Diffusice Models
100
MatierGen
99.84
96.21
98.17
98.96
0.109
0.036
UniMat
100
96.80
98.20
99.20
0.076
0.025
Pero-5
Flow Matching
FlowLLM
FloMMM
99.70
100
96.06
97.91
90.27
88.91
99.40
99.31
0.892
1,210
0
0.060
Ruyesian Flow Networks
CrysRPN
100
99.52
LLMs
Liena22 (7B)
46
Dillusion
Lr
Table 2: Summary of results on De Novo Generation Task of Different Class of Generative Models.
0 T S T, and initiate the denoising process from there to refine both the coordinates and the lattice structure. 7 is a hyper-parametee which we choose based on the validation set for each dataset. The final atomic coordinates Xn and lattice structure Lg, combined with the retained atom types A, constitute the generated crystal material, denoted as: M = (A, (o, Lo). The full sampling process described in Algorithm
Experiments
In this section, present comprehensive evaluation of method against several baselines on three benchmark tasks: De Novo Material Generation (Section [.T]. Stable, Unique, and Novel (S.U.N.) Materials Generation (Section 5.2) and Text-Conditioned Generation (Section 5.3
5.1 De Novo Material Generation (Gen)
Setup. First, we focus on De Novo Material Generation (Gen), an unconditional generation task aimed at producing novel, stable crystal materials that are distributionally similar to those in the test dataset. To evaluate the effectiveness of CrysLLMGen in this task, we compare it against seven state-of-the-arr generative models across different categories: CDVAE (8. DiffCSP (TO), DiffCSP++ [12). MatterGen [II]. UniMat [13] and SymmCD (25 from the class of denoising models; FlowMM [14) and FlowLLM [15] from the flow matching paradigm; CrysBFN [31] from the Bayesian Flow Networks and LLaMA-2(7B)
7 representing the LLM-based approach. We use two popular material datasets for this task: Perov-5 [38] 39 and MP-20 [4).. While training all competitive models, followed the standard dataset split of 60% for training, 20% for validation, and 20% for testing. Following 8 we assess all models using seven evaluation metrics grouped under three broad categories: validity, coverage, and property statistics. (More details in Appendix C Results. We present the results in Table 2 We observe that across both datasets, denoising models such diffusion and flow matching frameworks excel in structural validity, while LLM models perform better in compositional validity. In contrast, CrysI. surpasses all baseline models both structural and compositional validity. Specifically, on MP-20 dataset, CrysLLMGen demonstrates a 4.64 % improvement in compositional validity over leading denoising models and a 2.29 % gain in structural validity over LLM-based models. Furthermore. in coverage metrics, CrysLLMGen achieves the highest performance in COV-Precision, however in COv-Recall, it outperforms LLMs and delivers competitive results against denoising models. Lastly, in terms of property statistics, recent denoising models such as UniMat and FlowMM demonstrate strong performance, while approaches incorporating LLMs (e.g.. LLaMA or FlowLLM) tend to underperform. We observe that CrysLLMGen consistently surpasses LLM-based baselines and performs comparably to SOTA denoising models. Overall, CrysLLMGeen exhibits promising performance in the material generation task, effectively leveraging the strengths of both LLMs
as a unified model,
8

---

and diffusion models to enhance generative capability and produce more valid, periodic 3D crystal structures.
5.2 Stable, Unique and Novel (S.U.N.) Materials
The ultimate objective of materials discovery is to efficiently screen for stable, unique, and innovative materials. Thermodynamic stability provides strong indication of a material's synthesizability. A crystal is considered stable when it is energetically more favorable than any alternative structures composed of the same atomic elements, but in different proportions arrangements. The stability gap can be rationalised by considering the factors contributing to the energy above the convex hull for given material structure (A, X, L.. Fnull(A (A, X, I)) can be conceptually decomposed into contributions related to the intrinsic stability of the composition A itself, and the relative stability of the specific structure (X, L) for that composition,
(A, L))
A Fsstruct (
A Eichem (A)
(5)
Polymorph
Compositional
energy split
instability
Such stable structures lie directly on or below the convex hull obtained after structural relaxation, the next objective is
Once a
set of stable materials is to identify those that are In this section, we evalu- ate the effectiveness our proposed framework in generating stable, unique, and novel materials. To this end, we adopt the S.U.N. (Stability, Unique- ness, and Novelty) met- rics introduced in prior work [ and compare our model's performance against several baseline approaches. Specifically, we follow the evaluation protocol used in 14 [.5]
structurally unique and truly novel within that stable set.
Method
& Meta-Stable 1 & M.S.U.N.
% Stable t
% S.U.N
CDVAE
23.58
21.99
3.08
DifICSP
35.04
32.19
7.36
5.61
DiflCSP++
42.39
30.56
8.58
6.55
FlowMM
31.64
22.46
4.76
3.06
SymmCD
40.01
31.69
9.99
6.76
Llama-2 (7B)
56.60
26.66
12.67
CrysLLMGen (7B)
62.02
35.94
16.79
9.21
Table 3: Results on S.U.N. metrics for different SOTA models.
wherein 10,000 candidate structures generated for each state-of-the-art model and subsequently relaxed using a pretrained CHGNet [41] model to estimate their formation energies. These energies are then compared against convex hull constructed from the Materials Project database. Relaxed structures are labeled as stable if Ehull < 0.0 eV/atom and as metastable Ehull < 0.1 eV/atom. Finally, we assess whether the relaxed stable and metastable structures also unique and novel, reporting them as % S.U.N. and % M.S.U.N., respectively.
We present the results of all competing baseline models along with CrysLL on the MP-20 dataset in Table 3 Overall, CrysLLLMGen demonstrates substantial improvements in generating stable, unique, and novel structures. In specific, compared to the best-performing denoising model, CrysLLMGen achieves relative gains of 46.29 % metastable rate, 11.65% % in MSUN rate, 68.07 % in stability rate, and 36.27 % in SUN rate. Furthermore, when compared to the LLM-based baseline, the improvements are 9.56 in metastable rate, 34.80% in MSUN rate, 32.53 % stability rate, and 90.29% % in SUN rate. Finally, to further assess how effectively CrysLLMGen generates low-energy structures compared to baseline models, we plotted the histogram of the computed E Ehwll distribution for relaxed structures across different methods, as shown in Figure 2 a). The results indicate that CrynLLMGen produces a greater proportion of low-energy structures than all other baseline models, confirming its ability generate more stable materials. As mentioned, Crys CrysI.LMGen introduces a hybrid approach where the L.MM predicts chemically valid compositions A, which are then fixed while diffusion model refines lattice and atomic coordinates. This separation leverages the L.M''s chemical prior to prioritize stable compositions, resulting in lower formation energies compared to fully joint diffusion models like DifrCSP.(Check detailed analysis in Appendix B
5.3 Text Conditioned Generation
Next, we evaluate CrysLLMMGen on conditional crystal generation task. Following the experimental setup proposed by (7), we extend the idea of text-conditional material generation by providing

---

1800
Conditional Prompt
Conditiions Satiified
Counts
600
600
deactiynan rateria The chenace formale Sed, spor gropp
=CryilLMGGnn
Ooneri Masen Ruth
10.1
Demonoe demcpppionn
ingtte ong or thes
E: as Ehull (eV/Atom)
ovend type
25 storta
Group
(a) Histogram of Ehull,
(b) Results of Conditional Generation.
Figure 2: (a) Histogram of (Frhalt) distribution computed after relaxation with CHGNet.(b) Results of Text-Conditional Generation atomic compositions and space group.
the LLM component with user-defined prompts that specify desired conditions for the generated materials. Specifically, we explore conditioning atomic composition and space group number by incorporating the target chemical formula and space group into the prompt. This prompt is then passed to the LLM component of our model, which generates materials that aim to satisfy the given constraints. To assess the model's ability to generate crystal materials that satisfy specified atom types and space group constraints, we compare the generated outputs with available ground-truth labels. For chemical composition, we parse the atomic formula from each generated CIF file and check for match with the target composition. Further, with tolerance of 0.2 Å to determine the space group of both generated and ground truth materials and compute the match rate. These experiments are conducted on the MP-20 test dataset, and the results are presented in Figure bb), where we compare the performance of CrysLLMGen against LLM baseline proposed by [7]. For atom types, we report match rates across three classes of materials based on the number of atoms: $5, < 10, and <25. We observe that CrysLL performs comparably to the LLM bascline across all three categories. Specifically, the model is able to generate materials with the correct composition in most cases, although accuracy declines as the number of atoms in the chemical formula increases. This similar performance is primarily due to CrysLLMGen retaining the atomic compositions provided by the LLM component. However, in the case of space group prediction, CrysLLMGen outperforms the LLM baseline by 42%. This improvement can be attributed to the diffusion model's ' structural refinement, which enables CrysLLMGen to better align the generated structures with the desired space group.
use Pymatgen's SpacegroupAnall
[4]
Conclusion
In this work, we explore hybrid approach that integrates large language models (LLMs) with diffusion models for crystal material generation. We propose CrysLLMGen, a two-stage framework that first uses fine-tuned LLM generate intermediate representation of atom types, atomic coordinates, and lattice parameters, and then refines the coordinates and lattice using pre-trained diffusion model. Extensive experiments popular material generation tasks demonstrate that CrysLLMGen maintains both structural and compositional validity. outperforming existing bascline models by a good margin. Moreover, CrysLLMGen generates crystal structures that are more stable, unique, and novel than those produced by prior methods. Additionally, it exhibits strong conditional generation capabilities, effectively synthesizing materials that meet user-defined constraints.
7 Limitations and Future Work
In our current proposed framework, there is no interaction between the LLM and the diffusion model; both components are trained independently as standalone modules. During sampling, the LLM first generates predictions, which are then passed to the diffusion model for further processing. This separation suggests a potential avenue for future research: exploring whether these two components could benefit from mutual guidance or feedback mechanisms to enhance the overall generation quality, Also, At present, we use a pre-trained I.LMA-2-77 base model as the LLM component, coupled with vanilla diffusion model that extends DiffCSP. Nonetheless, our framework is designed to be flexible, allowing seamless integration of more advanced L.LM variants, such as I.LaMA33, or more fine-grat fine-grained diffusion models for crystal material generation in future work.
10

---

8 Acknowledgment
This work was funded by Indo Korea Science and Technology Center, Bangalore, India, under the project name "Generating Stable Periodic Materials using Diffusion Models". We thank the Ministry of Education, Govt. of India, for supporting Kishalay with Prime Minister Research Fellowship (PMRF) during his Ph.D. tenure.
References
[1] Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. Machine learning for molecular and materials science. Nature, 559(7715):547-5555 2018.
[2] Gautam R Desiraju. Cryptic crystallography. Nature materials, 1(2):77-79, 2002.
[3] Kristof T Schütt, Henning Glawe, Felix Brockherde, Antonio Sanna, Robert Müller, and Eberhard KU Gross. How to represent crystal structures for machine learning: Towards fast prediction of electronic properties. Physical Review B, 89(20)2005118 2014.
[4] R. O. Jones and O. Gunnarsson. The density functional formalism, its applications and prospects. Reviews of Modern Physics, 61(3):889-746, July 1989.
[5] Robert G. Parr and Yang Weitao. Densitt-Functional Theory Atoms and Molecules. Oxford University Press, January 1995.
[6] David Sholl and Janice A. Steckel. Density Functional Theory: A Practical Introduction. Wiley, March 2009.
[7] Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint arXiv:2402.0433 04379, 2024.
[8] Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crys- tal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:21 2021.
[9] Youzhi Luo, Chengkai Liu, and Shuiwang Ji. Towards symmetry-aware generation of periodic materials. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[10] Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion. arXiv preprint arXiv:2309.044 04475, 2023.
[11] Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabbé, Lixin Sun, Jake Smith, et al. Mattergen: generative model for inorganic materials design. arXiv preprint arXiv:2312.0300 2023.
[12] Rui Jiao, Wenbing Huang, Yu Liu, Deli Zhao, and Yang Liu. Space group constrained crystal generation. arXiv preprint arXiv:2402.033992, 2024.
[13] Mengjiao Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Dogus Cubuk. Scalable diffusion for materials generation. arXiv preprint drXiv:2311.09235. 2023.
[14] Benjamin Kurt Miller, Ricky TQ Chen, Anuroop Sriram, and Brandon M Wood. Flowmm: Generating materials with riemannian flow matching. arXiv preprint arXiv:2406.04713, 2024.
[15] Anuroop Sriram, Benjamin Miller, Ricky TQ Chen, and Brandon Wood. Flowllm: Flow matching for material generation with large language models as base distributions. Advances in Neural Information Processing Systems, 37:46025-46008 2024.
[16] Callum J Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of Chemical Information and Modeling, 60(10):4518-4533 2020.

---

[17] Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-d crystal structures, arXiv preprini arXiv:1909.000 19990009494 2019.
[18] Teng Long, Nuno M Fortunato, Ingo Opahle, Yixuan Zhang, Ilaas Samathrakis, Chen Shen, Oliver Gutfleisch, and Hongbin Zhang. Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures. upj Computational Materials, 7(1):66, 2021.
[19] Juhwan Noh, Jaehoon Kim, Helge Stein, Benjamin Sanchez-l.nggeling, John M Gregoire, Alan Aspuru-Guzik, and Yousung Jung. Inverse design of solid-state materials via a continuous representation. Matter, 1(5):1370-138) 2019.
[20] Sungwon Kim, Juhwan Noh, Geun Ho Gu, Alan Aspuru-Guzik, and Yousung Jung. Generative adversarial networks for crystal structure prediction. ACS central science, 668):1412-1420, 2020.
[21] Zekun Ren, Juhwan Noh, Siyu Tian, Felipe Oviedo, Guangzong Xing. Qiaohao Liang, Armin Aberle, Yi Liu, Qianxiao Li, Senthilnath Jayavelu, et al. Inverse design of crystals using generalized invertible crystallographic representation. arXiv preprint arXiv:2005.07609 3(6):7, 2020.
[22] Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya MD Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. Advanced Science, 8(20):2 100566, 2021.
[23] Kishalay Das, Subhojyoti Khastagir, Pawan Goyal, Seung- Soung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Periodic materials generation using text-guided joint diffusion model. In The Thirleenth International Conference on Learning Representations, 2025.
[24] Hyunsoo Park, Anthony Onwuli, and Aron Walsh. Exploration of crystal chemical space using text-guided generative artificial intelligence. Nature Communications, 16(1):4379, 2025.
[25] Daniel Levy, Siba Smarak Panigrahi, Sékou-Oumar Kaba, Qiang Zhu, Kin Long Kelvin Lee, Mikhail Galkin, Santiago Miret, and Siamak Ravanbakhsh. Symmed: symmetry-preservingg crystal generation with diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
[26] Subhojyoti Khastagir, Kishalay Das, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysldm: Latent diffusion model for crystal material generation. In Al for Accelerated Materials Design-ICLR 2025, 2025.
[27] Chaitanya K Joshi, Xiang Fu, Yi-Lun Liao, Vahe Gharakhanyan, Benjamin Kurt Miller, Anuroop Sriram, and Zachary W Ulissi. All-atom diffusion transformers: Unified generative modelling of molecules and materials. In International Conference Learning Representations, 2025.
[28] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2023.
[29] Michael Samuel Albergo and Eric Vanden-Fijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023.
[30] Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
[31] Wu, Y Song, J Gong, Cao, Ouyang, Zhang, Zhou, WY Ma, and Liu. periodic bayesian flow for material generation (2025). arXiv preprint 2502::22000001666. 2025.
[32] Luis Antunes, Keith T Butler, and Ricardo Grau-Crespo Crystal structure generation with autoregressive large language modeling. Nature Communications, 15(1):00570, 2024.
12

---

[33] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thortoo,, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling, Advances in Neural Information Processing Systems, 35:2406-2422, 2422,, 2022.
[34] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[35] Yang Song and Stefano Ermon. Improved techniques for training basee-baeed generative models. Advances in neural information processing systems, 1111243812444 2020.
[36] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020.
[37] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, E (n) equivariant graph neural networks. International conference on machine learning, pages 9323 9332. PMLR, 2021.
[38] Ivano E Castelli, David D Landis, Kristian Thygesen, Søren Dahl, Ib Chorkendorff, Thomas F Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Enviroumental Science, 5(10):9034-9433 2012.
[39] Ivano E Castelli, Thomas Olsen, Soumendu Datta, David Landis, Søren Dahl, Kristian Thygesen, and Karsten W Jacobsen. Computational screening of perovskite metal oxides for optimal solar light capture. Energy Environmental Science, 5(2):5814-581 2012.
[40] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. The materials project: A materials genome approach to accelerating materials innovation, apl mater. APL Mater, 2013.
2013.
[41] Bowen Deng, Peichen Zhong, KyuJung Jun, Janosh Riebesell, Kevin Han, Christopher J Bartel, and Gerbrand Ceder. Chgnet as a pretrained universal neural network potential for charge-informed atomistic modelling. Nature Machine Intelligence, 5(9): 2023.
[42] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314-319, 2013.
[43] Kishalay Das. Ai for scalable material design and property prediction. github, 2025.
[44] Kishalay Das. Learning robust representation of crystal materials for property prediction. KDD, 2024.
[45] Ankan Mullick, Akash Ghosh, G Sai Chaitanya, Samir Ghui, Tapas Nayak, Seung- Seung-Cheol Lee, Satadeep Bhattacharjee, and Pawan Goyal. Matscire: Leveraging pointer networks to automate entity and relation extraction for material science knowledge-han construction. Computational Materials Science, 233:1 12659, 2024.
[46] Souradip Guha, Ankan Mullick, Jatin Agrawal, Swetarekha Ram, Samir Ghui, Seung-Cheol Lee, Satadeep Bhattacharjee, and Pawan Goyal. Matscie: An automated tool for the generation of databases of methods and parameters used in the computational materials science literature. Computational Materials Science, 192:110325 2021.
[47] Ankan Mullick, Shubhraneel Pal, Tapas Nayak, Seung-Cheol Lee, Satadeep Bhattacharjee, and Pawan Goyal. Using sentence-level classification helps entity extraction from material science literature. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4540-4545, 2022.
[48] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett., 120(14): 145301, 2018.
[49] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as universal machine learning framework for molecules and crystals. Chem. Mater, 31(9):5644 3572, 2019.
13

---

[50] Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):1-8, 2021.
[51] Kishalay Das, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysmmnet: multimodal representation for crystal property prediction. In Uncertainty in Artificial Intelligence, pages 507-517. PMLR, 2023.
[52] Steph-Yve Yves Louis, Yong Zhao, Alireza Nasiri, Xiran Wang. Yuqi Song, Fei Liu, and Jianjun Hu. Graph convolutional neural networks with global attention for improved materials property prediction. Physical Chemistry Chemical Physics,
2020.
[53] Cheol Woo Park and Chris Wolverton, Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery. Physical Review Materials, 4(6), Jun 2020.
[54] Jonathan Schmidt, Love Pettersson, Claudio Verdozzi, Silvana Botti, and Miguel AL Marques. Crystal graph attention networks for the prediction of stable materials. Science Advances, 7(49):cabi7948, 2021.
[55] Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In The 36th Annual Conference on Neural Information Processing Systems, 2022.
[56] Kishalay Das. Bidisha Samanta, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysxpp: An explainable property predictor for crystalline materials. upj Computational Materials, 8(1):43, 2022.
[57] Kishalay Das, Bidisha Samanta, Pawan Goyal, Seung Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysgnn: Distilling pre-trained knowledge enhance property prediction for crystalline materials. arXiv preprint arXiv:2301.0554 2..5522,, 2023.
[58] Chris Pickard. Airss data for carbon at 10gpa and the c+n+h+o system at 1gpa. materi- 2020.
[59] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The Materials Project: A materials genome approach accelerating materials innovation. APL Materials, 1(1):01 1002, 07 2013.
[60] Daniel W Davies, Keith T Butler, Adam Jackson, Jonathan M Skelton, Kazuki Morita, and Aron Walsh. Smact: Semiconducting materials by analogy and chemical theory. Journal of Open Source Software, 4(38):1361, 2019.
[61] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013.
[62] Scott Kirklin, James Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The open quantum materials database (oqmd): assessing the accuracy of dft formation energies. npj Computational Materials, 1(1):1-15, 2015.
14

---

A Technical Appendices and Supplementary Material
B Related Work(In Details)
Deep learning models are increasingly being applied across various downstream tasks in mate- rials science, including scalable materials design (43, property prediction [44], knowledge-bass construction [45], database generation [46), and entity extraction [47]. In this work, we present a
comprehensive survey of state-of-the-art methods for crystal material representation learning and generation.
B.1 Crystal Representation Learning
In recent times, graph neural network (GNN) based approaches have emerged as powerful model in learning robust representation of crystal materials, which enhance fast and accurate property prediction. CGCNN 48) is the first proposed model, which represents a 3D crystal structure as an undirected weighted multi-edge graph and builds a graph convolution neural network directly on the graph. Following CGCNN, there are a lot of subsequent studies [49. 50
51 52
53 54,, where
authors proposed different variants of GNN architectures for effective crystal representation learning. Recently, graph transformer-batt graph representation of the material, which marginally improves the performance, however, is much faster than the prior SOTA model. Moreover, scarcity of labeled data makes these models difficult to train for all the properties, and recently, some key studies [56 57 have shown promising results to mitigate this issue using transfer learning, pre-training, and knowledge distillation respectively.
architecture Matformer [55] is proposed to learn the periodic
B.2 Crystal Material Generation
In the past, there were limited efforts in creating novel periodic materials, with researchers concen- trating on generating the atomic composition of periodic materials while largely neglecting the 3D structure. With the advancement of generative models, the majority of the research focuses on using popular generative models like VAEs or GANs to generate 3D periodic structures of materials, how- ever, they either represent materials as stree-dimensional voxel images [6 images to depict material structures (atom types, coordinates, and lattices), or they directly encode material structures as embedding vectors 20 21 22] However, these models neither incorporate stability in the generated structure nor are invariant to any euclidean and periodic transformations.
17
18 [9 and generate
Diffusion Models. In recent times equivariant diffusion models have become the leading method for generating stable crystal materials, thanks to their capability to utilize the physical symmetries of periodic material structures. In specific, state-of-the-art models like CDVAE g and SyMat [ integrate a variational autoencoder (VAE) and powerful score-based denoising network, work directly with the atomic coordinates of the structures and uses an equivariant graph neural network to ensure euclidean and periodic invariance. Subsequent models, such as DiffCSP [10] and MatterGen [II]. adopt a joint diffusion framework to simultaneously learn atomic composition, fractional coordinates. and lattice parameters. In contrast, UniMat [13] introduces a unified 4D tensor-based crystal represen- tation that jointly models discrete atom types and continuous atomic coordinates using a probabilistic diffusion model with interleaved attention and convolution layers. Lately, there have been a few efforts to develop text-guided diffusion models for condiiiion material generation. Models such as TGDMat [23] and Chemeleon [24] incorporate contextual representations, leveraging pretrained models like MatSciBERT or CLIP, into a GNN-based denoising network. This enables the generation of valid and stable periodic materials that align with the conditions specified in textual descriptions.
Symmetry-Awarr Generation. While conventional diffusion models independently learn atomic positions within a unit cell, incorporating space group symmetry significantly reduces the model's degrees of freedom. DiffCSS++ [12 is the first approach which builds upon DiffCSP by enforcing space group constraints on both lattice parameters and atomic coordinates. In this approach, lattices are parameterized to ensure the generated structures conform to the correct lattice system, while atomic fractional coordinates are restricted to Wyckoff positions derived from templates in the training data. SymmCD (25 advances this idea by jointly learning the fractional coordinates and corresponding Wyckoff positions of atoms through site-symmetry-ant modeling only one representative atom per crystallographic orbit, both SymmCD and DiffCSP+ achieve substantial reduction in generative complexity.
representation. By
15

---

Latent Difffusion Models. A major limitation of current diffusion based methods lies in their operation within high-dimensionas feature space. where they model the joint distribution of atom types, fractional coordinates, and lattice structures: an inherently multimodal distribution with distinct statistical properties for each component. As result, these models entail substantial computational cost for both training and inference, limiting their applicability in resource-constrained settings. Few of the recent works like CrysLDM [26], ADiT (27) utilize a latent diffusion model to address the above limitations, reducing sampling time for crystal material generation. Operating in the latent space, these offers unique advantages in generative modeling complexity over existing feature-domain diffusion models, making it more efficient in terms of time and resource consumption.
Flow matching. Flow matching (FM) [28 29 30 has recently emerged as a promising alternative to diffusion-brit based methods for crystalline material generation. Unlike diffusion models that iteratively denoise samples from Gaussian prior, FM directly learns time-dependent velocity field that continuously transports an arbitrary base distribution toward the target distribution of stable crystals. The first application of this framework, FlowMM [4,, introduced a representation that enforces global rotational and translational symmetries along with periodic boundary conditions, while defining base distributions with equivariant flows to ensure symmetry invariance. Building on this, [5] integrates geometric inductive biases with base distributions informed by large language models 77 Moreover, CrysBFN [11] employs periodic Bayesian Flow Networks (BFNs) with entropy conditioning and monnoonot dynamics to more effectively capture periodicity in non-cuclidean space, thereby improving both sampling efficiency and generation quality.
FlowLLM
Large Language Models. Large language models (LLMs) are increasingly adapted in the natural sciences as versatile priors for reasoning over sequences, graphs, and spatial data, and this trend has recently extended to materials generation. By encoding crystal structures as textual descriptions of unit cells and atomic positions, token-based language models [32 have demonstrated good performances in generating stable and valid materials.
C Experimental Results
C.1 Benchmark Task
De Novo Material Generation (Gen): In Gen task, the goal of the generative model is to generate novel stable materials (atom types, fractional coordinates, and lattice structure).
Conditional Material Generation: In this task, specific criteria are provided, and the objective is to generate stable crystal materials that satisfy those given conditions.
Crystal Structure Prediction (CSP): Atom types of the materials are given and the goal is to predict/match the crystal structure (atom coordinates and lattice).
C.2 Datasets
Following prior works we evaluate our model on three baseline datasets: Perov-5, Carbon-24 and MP-20.
Perov-5
dataset consists of 18,928 perovskite materials, each with 5 atoms in a 38
39
cell. They generally be denoted by ABX; indicating the three different types of atoms usually observed in such materials.
Carbon-24
[58
dataset has 10,153 materials with 6 to 24 atoms of carbon in the crystal
lattice.
MP-20 [40] dataset has 45,231 materials curated from the Materials Project library where each material has at most 20 atoms in the lattice.
59
MPTS-52 is a more challenging extension of MP-20, consisting of 40,476 structures up to 52 atoms per cell, sorted according to the earliest published year in literature.
Crystals from Perov-5 dataset share the same structure but differ in composition, whereas Crystals from Carbon-24 share the same composition but differ in structure. Crystals from MP-20 and MPTS-52 differs in both structure and composition.
16

---

C.3 Hyper-parameter Details
LLM Component We finetune the LLaMA-2 7B model for epoch using the Adam W optimizer implemented via the 'transformers. Traann interface. The learning rate is set to 0.0001.
For the diffusion component, we batch size of 256 and adopt cosine noise schedule. The model is trained for 1000 diffusion steps and inference is performed using 900 steps. The denoising network is implemented using 6-layer CSPNet. Optimization is done using the Adam optimizer with a learning rate of 0.001.
Diffusion Model:
C.4 Evaluation Metrics for CSP Task
We evaluate the performance of CrysLL and baseline models on stable structure prediction using standard metrics proposed in prior works [TO] 8), by assessing how well the generated structures match the corresponding ground truth structures in the test set. We compute the Match Rate and RMSE metrics using the StructureMatcher class from Pymatgen, which identifies the best correspondence between two structures while accounting for all material invariances. The Match Rate measures the percentage of generated structures in the test set that successfully match the ground truth structures under the thresholds: stol = 0.5, angle_to __tle_ttlllloo = 10, and 1tol = 0.3. The RMSE is calculated between atomic positions of the ground truth and the best-matching generated structure, normalized by \/V/N, where V is the lattice volume and N is the number of atoms, and averaged over all matched structures.
C.5 Validity Metrics for Gen Task
assess both structural and compositional validity. Structural validity represents the percentage of generated crystals with valid periodic structures, while compositional validity refers to the percentage of structures with correct atom types. A structure is considered valid if the shortest distance between any pair of atoms exceeds 0.5 Å, and its composition is deemed valid if the overall charge remains neutral, as determined by SMACT [60]
Validity: In line with previous studies [6 8
Coverage We consider two coverage metrics, COV-R (Recall) and COV-P (Precision). COV-R measures the percentage of the test set materials being correctly predicted, whereas COV-P measures the percentage of generated materials that cover least one of the test set materials.
Property Statistics We evaluate the similarity between the generated materials and those in the test set using various property statistics, where we compute the earth mover's distance (EMD) between the distributions in element number (# Elem) and density (p, unit g/cm3).
C.6 Stability Metrics for Gen Task
% Meta-Stable We report the percentage of structures, among 1,000 generated materials, that achieve Ehull < 0.1 eV/atom after relaxation using CHGNet.
% M.S.U.N. Among the metastable structures, report the percentage of crystal materials that are unique and novel.
Stable Out 1,000 generated materials, we report the percentage structures that achieve Ehull < 0.0 eV/atom after relaxation using CHGNet. This metric reflects the thermodynamic stability of the generated materials.
% S.U.N: Out of all stable structures, we report the percentage of crystal materials that are both unique and novel.
D Crystal Structure Prediction (CSP)
We also evaluate our proposed model on the Crystal Structure Prediction (CSP) task, where the goal to generate complete crystal structure, including atomic coordinates and lattice parameters, given only the atomic composition.
17

---

Category
Method
Match Rote 1
Perou-5
RMSF.
March Rate T
Carben-34
RMSF
Match Rare T
MP-30
RMSF.
March Rate
MPTS-52
RMSR 1
Diffasion Models
COVAR DIEUSP
45.31
52.02
0,1138
00766
17.54
17.99
0,248
0.2759
51.49
33,90
01631
0.1055
12.19
5.34
0.2100
0.1736
Flow Masching
HowMM
53.15
019922
23.47
0.4122
61.39
0.566
17.54
0.1726
14
1lams-2-78
97.60
0.09
97.21
0.0255
73.22
0.154
39.96
0.10
11M4-iifision
Crysl MGen (7B)
98.30
0.0066
98.81
0.0136
74.24
0.1276
42.16
0,0977
Table 4: Summary of results CSP Task of Different Classes of Generative Models.
Setup. In this setting, we incorporate the atom types directly into the input prompt and use Crys- LLMGen to generate the corresponding fractional coordinates and lattice structure. We conduct experiments on three benchmark datasets: Perov-5, Carbon-24, MP-20 & MPTS-52 and compare the performance against four strong baselines: CDVAE [8) DiffCSP [O], FlowMM [14. and LLaMA- 2(7B) 77. To assess the effectiveness of our model and baselines in generating stable structures, adopt two widely used evaluation metrics, Match Rate and RMSE, as defined in prior works [8, by comparing the generated structures to ground truth structures from the test set.
Results. We report the results in Table We observe that CrysL.LMGen consistently outperforms competing baseline models across all benchmark datasets, including challenging and realistic ones like MP-20. In particular, it shows significant gains over denoising- approaches such as diffusion and flow matching models, and achieves notable improvements over standalone LLM-based methods.
E Comparison of CrysLLMGen and other Joint Diffusion Frameworks in terms of stability
CrysLLMGen adopts hybrid strategy that combines large language model (LLM) with symmetry- cquivariant diffusion network. First, the LLM proposes a chemically plausible composition A; this stoichiometry is subsequently frozen. A second stage then performs score-based denoising of the lattice vectors L and fractional coordinates X. In contrast, joint diffusion frameworks (for example, DIFFCSP) operate directly on the triplet (A, L, X), requiring single score network to denoise both discrete and continuous variables simultaneously. The latter increases the dimensionality of the sample space and introduces high-variance gradients associated with the one-hot representation of A.
Both pipelines optimise a likelihood diffusion loss, yet the LLM in CrysI.LMGen is pre-trained on extensive corpora that embed chemical knowledge (ICSD entries, Materials Project abstracts, patents, etc.). Consequently, the LLM internalises empirical rules of charge balance, common oxidation states, and frequency-weighter formation energies that are largely invisible to structure- only datasets. This chemical prior endows CrynLLMGen with a selective bias toward compositions empirically known be low-enthalpy, translating into systematically smaller energies above the convex hull, Emull, than those produced by fully coupled diffusion.
The stability gap can be rationalised by considering the factors contributing to the energy above the convex hull for a given material structure (A, X, L). Emaul (A, X, L) can be conceptually decomposed into contributions related to the intrinsic stability of the composition A itself, and the relative stability of the specific structure (X, L.) for that composition,
Ebaul (An L, AEtrrer(X,, (X, (A) + A Echem (A)
(6)
Polymorph energy split
Compositional instability
Here, A Fe (X, T. A) is the energy difference between structure (X, I.) and the most stable known structure for composition A, while E' (A) represents the minimum energy above the convex hull for composition A. Empirical data from high-throughput density functional theory calcu- lations across large materials databases clearly shows that & values (reflecting compositional instability) are typically an order of magnitude or more larger than A EEttruct values (reflecting poly- morph energy differences) informed prior on A via the LLM is the most effective route to generating thermodynamicall stable structures with low Ebon- In fully coupled model, rare yet low-A Echem formulas are under-sampled by the standard likelihood objective, forcing the generator to explore chemically broader and there- fore statistically higher-energy compositional subspace, other words, sampling compositions A that are thermodynamically very stable EAe (A) Ybu rare in existing structural databases poses
[6] 622. Because A is the dominant term, enforcing a chemically
18

---

significant challenge, they correspond to low-density regions in the training data. The diffusion process, driven by gradients of the log- likllkiiiiood, may struggle explore or converge effectively in these sparse regions.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See sections and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See section and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We provide the source code and data, along with complete instructions to reproduce the main experimental results in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See section and appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We followed the experimental setup of earlier works, none of them reported Statistical Significance on experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See section and appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pse//.eeoo.occo/ooo///taa
Answer: [Yes]
Justification: Yes, the research we conducted in the paper conforms, in every respect, with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
The work performed in this paper has no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper does not pose any such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See section 5 and appendix C
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See section and appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not have any such risk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
25

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: See section 4
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /ttparrppp..cc//oooeeeeeee////// for what should or should not be described.
26


Reviews:
[{"review_id": "gSy7M3k1uv", "reviewer": "Reviewer_A8AZ", "summary": "The paper introduces a hybrid approach for crystal generation. Specifically, it first trains a large language model (LLM) to provide an initial structure, represented by A (discrete atomic types), F (fractional coordinates), and L (lattice matrix). This initial structure is then refined using an equivariant diffusion model. In this way, the approach combines the strengths of autoregressive (AR) models for generating atomic compositions and diffusion models for structure generation. The shown benchmarks of Crystal Structure Prediction and Denovo generation has shown the flexibility of the proposed framework.", "questions": "1. Could the authors provide a more in-depth discussion of the performance gap between Llama-2 (7B) and CrysLLMGen (7B) to better illustrate the effectiveness of their approach?\n\n2. I am also curious about the choice of using diffusion in the second phase—are there specific benefits or motivations for selecting this method?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Leveraging the generative capabilities of LLMs for composition generation, combined with fine-tuning via a domain- or geometry-specific model, is a simple yet effective idea. Bridging advanced AI tools with LLMs is an important and promising direction that should be encouraged.\n2. I appreciate the effort in integrating this pipeline to enhance performance across multiple benchmarks. The empirical results and the text-based generation capabilities demonstrated are impressive.\nThe paper is well written, with sufficient introduction and technical details to enable a full understanding of both the method and the results.\n\n**Weaknesses**\n\n1. Although the paper includes a paragraph comparing the proposed approach to FlowLLM, I remain unconvinced by the differences highlighted. The key motivation for FlowLLM's use of sampled data during training is to maintain alignment in the inference process under distribution shift [1]. The minor difference of fixing atom types in the second phase does not, in my view, constitute a significant distinction in design. This raises major concerns regarding the technical novelty and contribution of the proposed approach.\n\n2. Upon closer examination of the results, the base 7B LLM alone already achieves very strong overall performance, making the incremental improvement from the diffusion process seem marginal. I recommend the authors provide a more detailed analysis of the inference efficiency in the two-phase approach, and further clarify the motivation for the framework.\n\n3. The paper lacks comparison with some important baselines and citations of related works, including [2,3,4]. Comparisons with state-of-the-art or advanced models are essential.\n\n[1] FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions\n\n[2] MatterGen: a generative model for inorganic materials design\n\n[3] A Periodic Bayesian Flow for Material Generation\n\n[4] SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models", "was_revised": true, "final_justification": "empirical strong paper with limited novelty", "rebuttal": "We thank the reviewer for the valuable feedback. Below, we provide detailed point-by-point responses to each of the comments and questions:\n\n***Weaknesses :***\n\n - ***W1. Difference with FlowLLM***\n\n> By design, our proposed CrysLLMGen bears some resemblance to FlowLLM; however, it is not merely an adaptation of FlowLLM’s pipeline with the flow-matching module replaced by a diffusion model. The key innovations and novel aspects of CrysLLMGen are as follows:\n> \n> - **Generative Module:** FlowLLM employs a flow-matching model as its generative component, whereas CrysLLMGen uses a diffusion model. We analyzed the performance of both the flow matching and diffusion models in the context of unconditional material generation. Our observations indicate that DiffCSP, the diffusion-based model for material generation, outperforms FlowMM, the flow matching model, across most of the evaluation metrics for this task. Referencing the FlowMM paper [1], the summary of results from de novo generation on the MP-20 dataset is as follows:\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|Stability Rate|SUN Rate|\n> |-|-|-|-|-|-|-|-|-|\n> |DiffCSP(1000 Steps)|**100**|**83.25**|**99.76**|**99.71**|0.35|0.125|**5.06**|**3.34**|\n> |FLowMM(1000 Steps)|96.85|83.19|99.58|99.49|**0.239**|**0.083**|4.65|2.34|\n> \n> These observations indicate that the diffusion model is a more natural choice for material generation compared to flow matching, as it consistently produces more valid and stable materials.\n> \n> - **Integration Strategy:** In FlowLLM, the material structures generated by the LLM are directly passed into the flow-matching module for refinement. In contrast, since LLM generated material structure are intermediate representations, we inject these representations into our diffusion model at an intermediate timestep $\\tau$ , where 0 ≤ $\\tau$ ≤ T, and initiate the denoising process from that point. This design allows for more effective refinement and generation, leveraging the strengths of both LLM and diffusion models.\n> \n> - **Parallel Training:** We did not adopt the sequential training paradigm of FlowLLM, where a large language model (LLaMA-2) is first fine-tuned on the training dataset, and the samples generated from this LLM are then used to train the flow matching framework. Instead, in our approach, both the LLM and the diffusion module are trained in parallel using the same training dataset. \n> \n> - **Experimental Evaluation:** Finally, our experimental results demonstrate that CrysLLMGen outperforms FlowLLM across most standard benchmark metrics for the de novo material generation task (as shown in Table 3 of the paper) across different datasets. These results highlight the overall effectiveness of our proposed approach.\n>\n> [1] Miller, Benjamin Kurt, et al. Flowmm: Generating materials with Riemannian flow matching, ICML-2024\n\n- ***W2. Improvement of CrysLLMGen over LLMs***\n\n>The reviewer’s observation regarding the incremental improvement of CrysLLMGen over the base 7B LLM model may have come from the results of the de novo generation task on the Perov dataset (a simpler benchmark) reported in Table 3. However, if one examine the results on the more challenging MP-20 benchmark dataset, there are significant gains. Here is the summary of the results for the de novo generation task and stability metrics (detailed results are provided in Tables 3 and 4 of the paper):\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|Meta Stability Rate| MSUN Rate | Stability Rate|SUN Rate|\n> |-|-|-|-|-|-|-|-|-|-|-|\n> |LLaMA-2 7B      |96.40|93.30|94.90|91.10|3.610|1.060|36.27|23.92|6.910|5.29|\n> |CrysLLMGen      |99.94|93.78|99.84|98.52|0.664|0.459|45.20|35.94|11.98|9.21|\n> |% Improvements  |3.672|0.514|5.205|8.144|81.60|56.69|24.62|50.25|73.37|74.10|\n> \n> We clearly observe that CrysLLMGen outperforms the LLaMA baseline across most benchmark metrics by a significant margin. Notably, the overall average improvement across all metrics is 37.81%.\n> \n> We observe that CrysLLMGen significantly outperforms the LLaMA-2 7B model on most of the metrics for material generation.\n\n\n- ***W3. Comparing with additional generative models: Mattergen, SymmCD, CrysBFN***\n\n> We thank the reviewer for highlighting the baseline models. All the mentioned baselines have been evaluated on the De Novo Generation Task using the MP-20 dataset, and the results are as follows:\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|\n> |-|-|-|-|-|-|-|\n> *Already reported*\n> |DiffCSP    |100    |83.25|99.76|99.71|0.3502|0.3398| \n> |FlowMM     |96.85  |83.19|99.58|99.49|0.2390|0.083|\n> *New experiments*\n> |Mattergen  |100    |86.34|99.45|99.59|0.4597|0.2538|\n> |SymmCD     |88.10  |85.32|96.82|99.53|0.201|0.3639|\n> |CrysBFN    |100    |87.51|99.79|99.09|0.206|0.1628|\n> \n> Overall, the results follow similar trends as observed with other generative models reported in Table 3. Similar to other diffusion- and flow-based generative models, these models generally perform well in structural validity but struggle with discrete components, such as accurately identifying atomic types, leading to lower compositional validity. While there may be some quantitative differences, the overall observations remain consistent across most of these models. We will include these results in the revised manuscript and appropriately cite these works.\n\n\n***Questions:***\n\n- ***Q1. In-depth discussion of the performance gap between Llama-2 (7B) and CrysLLMGen (7B)***\n\n> Check W2.\n\n- ***Q2. Motivation behind Choosing Diffusion Model***\n\n> In an unconditional setup, DiffCSP performs better than FlowMM. We analyzed the performance of both the flow matching and diffusion models in the context of unconditional material generation. Our observations indicate that DiffCSP, the diffusion-based model for material generation, outperforms FlowMM, the flow matching model, across most benchmark metrics (6 out of 8) for this task. Referencing from the FlowMM paper, the summary of results from de novo generation on the MP-20 dataset is as follows:\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|Stability Rate|SUN Rate|\n> |-|-|-|-|-|-|-|-|-|\n> |DiffCSP(1000 Steps)|**100**|**83.25**|**99.76**|**99.71**|0.350|0.125|**5.06**|**3.34**|\n> |FLowMM(1000 Steps)|96.85|83.19|99.58|99.49|**0.239**|**0.083**|4.65|2.34|\n> \n> \n> These observations indicate that the diffusion model is a more natural choice for material generation compared to flow matching, as it consistently produces more valid and stable materials."}, {"review_id": "5UFBklKEyW", "reviewer": "Reviewer_7S97", "summary": "The authors introduce CrysLLMGen, a novel crystal material generative framework that combines large language and diffusion models to harness their advantages and level out weaknesses. Authors assess the quality of the proposed framework on **Crystal Structure Prediction**, **De Novo Material Generation**, **S.U.N. Materials Generation** and **Text Conditioned Generation** tasks and show competitive performance to pure language models, denoising framework models and LLM + Diffusion competitors.", "questions": "In Table 3, there is a huge gap in #Element metric between Llama-2 (7B) and CrysLLMGen (7B). It’s not clear for me how it’s possible if CrysLLMGen also adopts Llama-2 as a first step and the diffusion model doesn’t change elements. If the reason for such a gap is the stochasticity of the sampled objects, I recommend adding standard deviations to all metrics values. I also recommend adding the visualizations of intermediate and final samples to assess how they are changing after diffusion model applying.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is written in an easy-to-follow way, the quality and clarity of the text is high. \nThe experimental part is strong enough, covering 4 different generative setups. Important to mention, that Text Conditioned Generation section proposes a novel setup that incorporates general textual descriptions as conditions.\n\nDespite the above listed strengths of the work, the novelty of the framework is very limited. The general idea of the framework is very similar to the FlowLLM approach. Authors discuss the key differences between FlowLLM and CrysLLMGen in the Related Work section, still I'm not convinced with this discussion and consider independent training of LLM and diffusion as the only key difference. In the Experiment section, Table 3 authors show that CrysLLMGen outperforms FlowLLM, still it’s not clear if it is due to approach changes or due to different LLM/diffusion architectures and size. I recommend authors conducting Ablation Study to clarify this point, for instance, take FlowLLM language model and apply CrysLLMGen diffusion to fix broken language model samples, and vice versa, take CrysLLMGen language model and apply FlowLLM diffusion afterwards.", "was_revised": true, "final_justification": "I maintain my original assessment and believe my score remains a fair evaluation of this work.", "rebuttal": "Thank you very much for your valuable feedback. We are pleased to know that you appreciated our work and the thoughtful insights you provided. Below, we offer detailed, point-by-point responses to each of your comments.\n\n***Weaknesses :***\n\n - ***W1. Novlety of our proposed framework and difference with FlowLLM***\n\n> By design, our proposed CrysLLMGen bears some resemblance to FlowLLM; however, it is not merely an adaptation of FlowLLM’s pipeline with the flow-matching module replaced by a diffusion model. The key innovations and novel aspects of CrysLLMGen are as follows:\n> \n> - **Generative Module:** FlowLLM employs a flow-matching model as its generative component, whereas CrysLLMGen uses a diffusion model. We analyzed the performance of both the flow matching and diffusion models in the context of unconditional material generation. Our observations indicate that DiffCSP, the diffusion-based model for material generation, outperforms FlowMM, the flow matching model, across most benchmark metrics for this task. Referencing from the FlowMM paper [1], the summary of results from de novo generation on the MP-20 dataset is as follows:\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|Stability Rate|SUN Rate|\n> |-|-|-|-|-|-|-|-|-|\n> |DiffCSP(1000 Steps)|**100**|**83.25**|**99.76**|**99.71**|0.35|0.125|**5.06**|**3.34**|\n> |FLowMM(1000 Steps)|96.85|83.19|99.58|99.49|**0.239**|**0.083**|4.65|2.34|\n> \n> These observations indicate that the diffusion model is a more natural choice for material generation compared to flow matching, as it consistently produces more valid and stable materials.\n> \n> - **Integration Strategy:** In FlowLLM, the material structures generated by the LLM are directly passed into the flow-matching module for refinement. In contrast, since LLM generated material structure are intermediate representations, we inject these representations into our diffusion model at an intermediate timestep $\\tau$ , where 0 ≤ $\\tau$ ≤ T, and initiate the denoising process from that point. This design allows for more effective refinement and generation, leveraging the strengths of both LLM and diffusion models.\n> \n> - **Parallel Training:** We did not adopt the sequential training paradigm of FlowLLM, where a large language model (LLaMA-2) is first fine-tuned on the training dataset, and the samples generated from this LLM are then used to train the flow matching framework. Instead, in our approach, both the LLM and the diffusion module are trained in parallel using the same training dataset. \n> \n> - **Experimental Evaluation:** Finally, our experimental results demonstrate that CrysLLMGen outperforms FlowLLM across most standard benchmark metrics for the de novo material generation task (as shown in Table 3 of the paper) across different datasets. These results highlight the overall effectiveness of our proposed approach.\n>\n> [1] Miller, Benjamin Kurt, et al. \"Flowmm: Generating materials with riemannian flow matching, ICML-2024\n\n- ***W2. Clarification on Architectural Differences and Performance Gains in CrysLLMGen***\n\n> We thank the reviewer for this insightful question. First, we clarify that both CrysLLMGen and FlowLLM use the same underlying LLM architecture (LLaMA-2) in the first stage of their respective hybrid pipelines. The key differences, and our primary innovations, lie in the second stage of the pipeline. Specifically, there are two main distinctions:\n> \n> - **Generative Module:** FlowLLM employs a flow-matching model as its generative component, whereas CrysLLMGen uses a diffusion model. We evaluated the performance of both flow-matching and diffusion models in the context of unconditional material generation and found that diffusion models consistently outperform flow-matching models across the majority of the evaluation metrics.\n> \n> \n>  - **Integration Strategy:** In FlowLLM, the material structures generated by the LLM are directly passed into the flow-matching module for refinement. In contrast, since LLM generated material structure are intermediate representations, we inject these representations into our diffusion model at an intermediate timestep τ, where 0 ≤ τ ≤ T, and initiate the denoising process from that point. This design allows for more effective refinement and generation, leveraging the strengths of both LLM and diffusion models.\n> \n> \n> These two innovations collectively contribute to the superior performance of CrysLLMGen compared to FlowLLM, as shown in Table 3.\n\n- ***W3. Ablation Study***\n\n> As suggested by the reviewer, we conducted an ablation study in which we replaced the FlowMatching module of FlowLLM with a diffusion model (following the DiffCSP framework). In specific, we first used data sampled from LLaMA-2 to train the diffusion model. During sampling, given a prompt, we first generate A, X, L from the LLM, which are then passed to the diffusion model (injecting at the final step T) for further refinement. The results we observed from this ablation are as follows:\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|\n> |-|-|-|-|-|-|-|\n> |FlowLLM w\\ Diffusion|99.88|82.50|98.54|97.55|0.724|0.5388|\n> |CrysLLMGen|99.94|93.78|99.84|98.52|0.664|0.459|\n> \n> We observe performance degradation. In specific, 12.03% performance degradation in compositional validity. These results suggest that training the diffusion model on true ground-truth data (rather than on LLM-sampled data), along with employing intermediate denoising to refine LLM-generated structures, enhances the effectiveness of our proposed hybrid framework.\n\n\n***Questions:***\n\n- ***Q1. Clarification on #Element Metric Discrepancy***\n\n> We thank the reviewer for pointing this out. For consistency, we have reported benchmark results as presented in the respective original papers. The results for LLaMA-2 (7B) are taken from the Crystal-Text-LLM paper [2]. However, we also conducted our own sampling using the fine-tuned LLaMA-2 (7B) models, and the results we obtained for the de novo generation task are as follows:\n> \n> |Models|Struct Validity|Comp Validity|#Elements|\n> |-|-|-|-|\n> |LLaMA-2 7B|96.40|93.78|0.462|\n> |CrysLLMGen|99.94|93.78|0.459|\n> \n> We observe both compositional validity and # Element metrics are similar the two models. We will update it in the revised manuscript.\n>\n> [2] Gruver, Nate, et al. \"Fine-tuned language models generate stable inorganic materials as text.\" ICLR 2024\n\n- ***Q2. Visualizations of intermediate and final samples***\n\n> We thank the reviewer for this valuable suggestion. We will include visualizations of a few generated material samples in the revised manuscript\n\n***Paper Formatting Concerns***\n\n> We thank the reviewer for these thoughtful suggestions and appreciate the attention to detail regarding formatting and presentation.\n> - We will include representative examples of sampled structures in the revised manuscript, enabling experts to visually assess the quality of the generated materials.\n> - Regarding the Denoising Network section (Section 4.4), we agree that the technical depth may be better suited for the Supplementary Material. We will move this section accordingly to maintain the flow and readability of the main paper.\n> - We acknowledge the issue with Figure 1 and will replot it at higher resolution to ensure clarity and better visual quality.\n> - Additionally, we appreciate the recommendation to include examples of intermediate textual representations and details on coordinate tokenization. We will incorporate these elements in the revised manuscript or the camera-ready version to enhance reproducibility of our approach."}, {"review_id": "3qGg7J4Lr5", "reviewer": "Reviewer_JFzg", "summary": "This paper presents CrysLLMGen, a hybrid framework that combines a large language model (LLM) and a diffusion model for generating 3D crystal structures. The LLM is used to generate atom types, coordinates, and lattice parameters in a text-based format, while the diffusion model refines the continuous parts (coordinates and lattice) to improve structural validity. One notable design choice is the use of an intermediate timestep (τ), where the LLM output is injected into the diffusion process rather than starting from pure noise.\n\nThe approach is evaluated on several benchmark datasets and shows strong results in both structural and compositional validity. It also performs well in generating stable and novel materials. Overall, the paper is well-written, the methodology is clearly explained, and the experimental results support the main claims. The hybrid design is modular, and the work is a meaningful contribution to generative modeling for materials science.", "questions": "1. Regarding the manually selected injection timestep (τ):  \n   As discussed, τ plays a critical role in determining how much the diffusion model relies on the LLM's intermediate output. However, τ is currently hand-tuned using the validation set. Have the authors considered making τ a learnable or adaptive parameter? This could allow the model to dynamically adjust its reliance on the LLM's predictions based on input complexity. In the rebuttal, please comment on the feasibility of implementing a learnable τ or using a schedule to make the approach more generalizable across datasets.\n\n2. On prompt flexibility and robustness:  \n   The model relies on structured, template-based prompts for text-conditional generation. This limits usability, especially in practical or user-facing settings. Could the authors evaluate how robust the system is to variations in prompt format (e.g., synonyms, paraphrased inputs, or minor typos)? In the rebuttal, it would be helpful to include any experimental results or observations on how the model handles non-standard prompts.\n\n3. Property-guided generation capability:  \n   While the model supports conditional generation based on atomic composition and space group, it does not currently allow for conditional generation based on desired physical or chemical properties (e.g., target stability or bandgap). Have the authors considered extending the framework to incorporate property-conditioned prompts or downstream property prediction feedback? Please discuss whether this integration is feasible and whether the current architecture could support such conditioning in future work.\n\n4. Prompt robustness evaluation:  \n   In addition to the structured prompting discussed in Section 5.5, it would strengthen the work to explore how model outputs are affected by varying prompt wording. Could the authors test the model using multiple alternative prompts per structure (e.g., different textual descriptions of the same composition and space group)? A prompt-level robustness evaluation would provide useful insight into the generalizability of the LLM component.\n\n5. Possibility of joint training between LLM and diffusion model:  \n   The current framework treats the LLM and diffusion components as independently trained modules, with no mutual feedback or end-to-end optimization. Have the authors considered training both components jointly or using intermediate supervision to align the LLM’s outputs with the diffusion model’s denoising objectives? Such an approach could enable tighter coupling between discrete and continuous generation and potentially improve coherence and stability of the final outputs. Please comment on whether joint training was explored or if it could be feasible for future work.", "limitations": "- The LLM and diffusion components are trained separately. While this modular design is flexible, it prevents the diffusion model from learning how to correct errors that may come from the LLM during training.\n\n- The atom types predicted by the LLM are fixed during generation. This means the system cannot recover from mistakes in composition, such as selecting chemically invalid or unrealistic elements.\n\n- The injection timestep τ is manually chosen based on validation results. Since it’s not learned or adaptive, this setting may not transfer well across different datasets or tasks and adds extra tuning effort.\n\n- The model relies on structured, template-style prompts (e.g., chemical formula and space group). It does not currently support free-form or natural language input, which limits its flexibility and user accessibility.\n\n- The framework does not support generation based on specific material properties (like bandgap or stability). This makes it harder to use the model in property-targeted material discovery tasks.\n\n- The diffusion model is trained from scratch without using any pretrained weights or shared representations from the LLM. This could make training less efficient and limit its ability to use existing chemical knowledge.\n\n- The language model uses LLaMA-2’s default tokenizer, which is not specialized for chemical or scientific text. This may reduce accuracy when handling element symbols, numbers, or other technical inputs.\n\n- The paper only evaluates the model with clean, well-structured prompts. It does not test how the model performs when prompts are varied, paraphrased, or contain small errors, leaving its robustness in real-world use cases uncertain.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper introduces a well-motivated and novel hybrid framework, CrysLLMGen, which effectively combines the strengths of large language models (LLMs) and equivariant diffusion models for periodic crystal material generation. The design cleanly separates the modeling of discrete (atom types) and continuous (atomic coordinates and lattice) components.\n  \n- I particularly appreciate the introduction of the intermediate injection timestep τ. Instead of starting diffusion from full noise, the authors innovatively initialize the refinement process from τ using the LLM’s output. This is a practical and novel mechanism that enhances efficiency and leverages the LLM’s predictions meaningfully.\n\n- The experimental results are strong and consistently reported across multiple datasets and evaluation metrics, including structural and compositional validity, crystal structure prediction accuracy, and stable material discovery (S.U.N.). The hybrid method outperforms both LLM-only and diffusion-only baselines.\n\n- The modular architecture is future-proof and flexible, allowing for improvements in either the LLM or the diffusion component without major changes to the pipeline.\n\n- The writing is generally clear, well-organized, and easy to follow. Key components of the method are explained with appropriate visualizations and equations. The methodology is well-supported by quantitative results, and appendices provide thorough implementation details.\n\n\n**Weaknesses:**\n\n- The LLM and diffusion model are trained independently, with no joint optimization or end-to-end training. This separation prevents the system from learning interactions between discrete (composition) and continuous (structure) variables. As a result, the diffusion model cannot adapt to or correct specific types of errors made by the LLM during training.\n\n- The atom types generated by the LLM are frozen during inference, limiting the system’s ability to recover from incorrect or unrealistic compositions.\n\n- The injection timestep τ is hand-tuned based on validation performance and is treated as a static hyperparameter. While effective, it introduces additional overhead and may not generalize across datasets. It would be valuable to explore whether τ could be learned or adaptively selected during inference.\n\n- The model requires prompts to be in a strict, structured format (e.g., chemical formula and space group). It does not support free-form or natural language input, which limits its accessibility and usability for broader scientific or non-expert users.\n\n- The system does not support conditional generation based on desired physical or chemical properties (e.g., bandgap, density, stability), which limits its utility in property-driven materials discovery workflows.\n\n- The diffusion model is trained entirely from scratch rather than leveraging any pretrained embeddings or shared chemical priors with the LLM. This may reduce training efficiency and downstream performance.\n\n- The default LLaMA-2 tokenizer is not chemistry-specific, which may impair tokenization fidelity for scientific symbols and numerical precision. A domain-adapted tokenizer could potentially improve performance.\n\n- Finally, the prompt robustness evaluation is limited. The paper evaluates conditional generation using only clean, template-style prompts. It does not explore robustness to paraphrased prompts, typos, or noisy inputs, which would be important for practical deployment.", "was_revised": true, "final_justification": "Thank you for the clarification regarding the ablation study — this helps contextualize your design decisions.\n\nI’d like to respectfully note that during the discussion period, reviewers may echo or build upon one another’s concerns to ensure a thorough and fair evaluation. The fact that Reviewer 7S97 did not request further clarification does not imply that the issue is fully resolved for all reviewers.\n\nI also appreciate your confirmation that the validity rate will be included in the revised paper. Given that LLMs are known to hallucinate — especially in scientific applications — transparent reporting of such metrics is crucial. It enables the community to make informed decisions about which models best align with their specific needs and reliability requirements.\n\nAdditionally, I strongly encourage including the ablation study (FlowLLM w/ Diffusion vs. CrysLLMGen) in the main text or supplementary materials. This comparison provides compelling evidence that your choice of the diffusion model, along with the training and integration strategy, is both effective and necessary.\n\nBased on the thorough rebuttal, the additional results provided, and the helpful clarifications — particularly the ablation — I am inclined to recommend acceptance of this work. Nicely done.", "rebuttal": "We thank the reviewer for the valuable feedback and are pleased to know that you appreciated our work and the thoughtful insights provided. Below, we offer detailed point-by-point responses to each of reviewer's comments.\n\n***Question:***\n\n***Q1. Regarding making τ learnable or adaptive for better generalization***\n\n> The suggestion given by the reviewer is appreciated and indeed a good one. However, in this work, we do not treat τ as a learnable or adaptive parameter. Instead, τ is considered a hyperparameter, which we tune using the validation dataset. Making τ learnable or adaptive is an interesting direction and remains a potential avenue for future work. This has been noted in Appendix B: Limitations and Future Work.\n\n\n***Q2. Regarding prompt flexibility and robustness***\n\n> Again, a good suggestion by the reviewer regarding the robustness and flexibility of the prompt. However, we have followed the existing literature in this regard. Most LLM/text-based material generation works to date (e.g., Crystal-Text-LLM [1], FlowLLM [2], TGDMat [3]) rely on template-based prompts while adapting pretrained LLM/BERT models for text-conditional generation. Accordingly, we adopted a similar framework to maintain consistency with prior work. That said, introducing greater flexibility and robustness into prompt design is an interesting direction that warrants further exploration, and we consider this a potential avenue for future work.\n\n\n***Q3. Regarding property-guided generation capability***\n\n> Again, a good suggestion by the reviewer regarding the property-guided generation capability of our proposed framework, enabling conditional generation based on target physical or chemical properties. However, implementing this would require a pretrained property predictor for each target property to assess whether the generated material aligns with the desired property values. The main challenge here lies in selecting an appropriate property predictor, as the literature offers many architectural choices, and currently, there is no standard benchmark for this task. Unfortunately, none of the existing works have addressed this issue too.\n>\n> In our work, we followed the existing literature in this regard. Specifically, we adopted the evaluation methodology used in [1], where the number of atoms and space group are evaluated for text-conditional generation. These are intrinsic material properties that can be validated without requiring any pretrained property predictor. Nonetheless, property-guided generation is an interesting direction that warrants further exploration, and we consider it a promising avenue for future work.\n\n\n***Q4. Regarding  joint training between LLM and diffusion model***\n\n> In our current proposed framework, there is no interaction between the LLM and the diffusion model; both components are trained independently as standalone modules. While jointly training the LLM and diffusion model is an interesting direction, it presents several challenges—such as designing an appropriate optimization framework and establishing an effective feedback mechanism between the two modules. There are many nuances in this design that require further exploration and significant engineering effort. Therefore, we consider this as a potential avenue for future work, which has been noted in Appendix B: Limitations and Future Work.\n\n***Q5. Regarding  invalid or unrealistic atom types***\n\n> We agree with the reviewer’s observation that LLMs can generate invalid or unrealistic atom types. In our experiments, we observed a small percentage of such materials generated by the LLM models, which we filtered out during post-processing using simple validation scripts.\n\n\n***Q6. Regarding  LLaMa's default tokenize***\n\n> We have followed the existing literature in this regard. As LLM backbone, LLaMA-2 have demonstrated strong performance in material generation tasks across several recent studies[1][2]. Additionally, LLaMA-2 tokenizes numbers as individual digits by default, a feature shown to significantly enhance performance on arithmetic-related tasks. However, we have also conducted experiments on the de novo generation task using latest LLaMA model, LLaMA 3.1 8B model. Results for MP-20 are as follows: \n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|F1-Score|Density|#Elements|\n> |-|-|-|-|-|-|-|-|\n> |LLaMa-2 7B  |96.40|93.30|94.90|91.1|92.96|3.61|1.06|\n> |LLaMa-3.1 8B|93.90|87.90|99.40|90.58|94.74|0.8309|0.1713|\n> \n> We observe a similar trend for all the LLaMa baseline models as well.\n\n\nOverall, the suggestions provided by the reviewer are very insightful, and we agree that there is significant potential to enhance our proposed hybrid framework. However, CrysLLMGen, as presented in this paper, represents one of the initial attempts at designing hybrid models for material generation. While there is considerable scope for further development, these directions require deeper exploration and will be pursued in future work. We will include these in \"Appendix B: Limitations and Future Work\" section.\n\n\n\n[1] Gruver, Nate, et al. Fine-tuned language models generate stable inorganic materials as text. ICLR 2024\n\n[2] Sriram, Anuroop, et al. Flowllm: Flow matching for material generation with large language models as base distributions. NeurIPS - 2024\n\n[3] Das, Kishalay, et al. Periodic materials generation using text-guided joint diffusion model. ICLR 2025"}, {"review_id": "VyeapJrVh5", "reviewer": "Reviewer_j4sq", "summary": "The paper introduces a two step approach for crystal structure generation. A LLM that genreates an initial guess, and a diffusion model that improves the positions and lattice vectors of that guess.", "questions": "Why is only one LLM shown in Table 1? There are multipl LLM based crystal structure generation models by now. Do they also fit the trend that you want to support here?\nWhy are Bayesian Flow Networks (e.g. CrysBFN) not discussed, which solve exactly the problem of diffusion models regarding discrete denoising.\n\nWhat are the drawbacks of LLama 3 vs LLama 2 regarding \"balance computational efficiency, the scale of available pretraining data, and practical deployment considerations\"? Or did you just develop and test your model with LLama 2 and did not yet test it with a LLama 3 model of similar size yet?\n\nCan you please add more details on conditional and unconditional fine tuning of the LLMs?\n\nHow do you ensure that the samples encountered during training of the diffusion component come from the same distribution as the samples generated by the LLM? The errors/noise generated by the LLM might be strongly correlated and also biased, and might not share much similarity with the noise in the noisy samples encountered during the training time of the diffusion component.Furthermore, it would be good to give a few more details about the choice of tau.\n\nDid you benchmark on the more challenging MPTS-52 which is now more and more replacing the simpler benchmarks on Perov-5 and Carbon-24?\nWhat reference was used for the LLMs line in Table 2? Or was this model trained by you in the context of this study? How statistically significant is the advantage of LLM+Diffusion over just the LLMs line? One percentage point without any error estimating does not say anything about the significance of the results, especially for the MP-20 dataset. Why is the MP-20 RMSE value of your model bold? The Flow Matching result seems to outperform your model, with an RMSE of 0.0566.\n\nTable 4: How us the SUN rate of DiffCSP++ evaluated as it was not reported in the original paper and as other literature in the past months report a different value for the DiffCSP++ SUN rate?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is very clearly written and easy to follow.\nThe idea is interesting and novel.\nThe experiments are showing a good performance of the model, even though the performance improvements are mostly limited to the CSP task and are less strong for the generation task. The difference between the LLM-only approach and the LLM+diff approach is rather small in some of the benchmarks. Some recent baselines, in particular other LLM-based approaches as well as Bayesian flow networks, are missing.", "was_revised": true, "final_justification": "After the rebuttal and the overall impression of the paper and the reviews/rebuttals, I think this paper is a very valuable contribution to crystal structure generation, and I therefore recommend accepting the paper.", "rebuttal": "We thank the reviewer for the valuable feedback. Below, we provide detailed point-by-point responses to each of the comments and questions:\n\n***Questions:***\n\n***Q1. Limited LLM Baselines Comparison in Table-1***\n\n> To the best of our knowledge, Crystal-Text-LLM is the most widely used LLM-based model for material generation in the literature. However, we have also conducted experiments on the de novo generation task using LLaMA-2 13B and the latest LLaMA model, LLaMA 3.1 8B model. Results for MP-20 are as follows: \n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|F1-Score|Density|#Elements|\n> |-|-|-|-|-|-|-|-|\n> |LLaMa-2 7B  |96.40|93.30|94.90|91.1|92.96|3.61|1.06|\n> |LLaMa-2 13B |95.51|92.41|97.91|88.91|93.19|2.13|**0.1**|\n> |LLaMa-3.1 8B|93.90|87.90|99.40|90.58|94.74|0.8309|0.1713|\n> |CrysLLMGen  |**99.94**|**93.78**|**99.84**|**98.52**|**99.17**|**0.664**|0.459|\n> \n> We observe a similar trend for all the LLaMa baseline models as well. \n\n***Q2. Including Bayesian Flow Networks (e.g. CrysBFN) Baseline***\n\n> We thank the reviewer for the suggestion. The results of CrysBFN for the de novo generation task are reported below.\n> \n> |Models|Struct Validity|Comp Validity|COV-Precision|COV-Recall|Density|#Elements|\n> |-|-|-|-|-|-|-|\n> |DiffCSP  |100 |83.25|99.76|99.71|0.3502|0.3398|\n> |CrysBFN  |100 |87.51|99.79|99.09|0.2070|0.1628|\n> \n> Overall, the results follow similar trends as observed with other generative models reported in Table 3. Similar to other diffusion-based generative models (like DiffCSP), these models generally perform well in structural validity but struggle with discrete components, such as accurately identifying atomic types, leading to lower compositional validity. \\We will include these results in the revised manuscript and appropriately cite CrysBFN.\n\n\n***Q3. Use of Llama-2 instead of Llama-3 — Justification and Practical Considerations***\n\n> We have conducted experiments with LLaMA 3 (LLaMa-3.1 8B) as well, and the results are reported in the table above (See Q1). We observe a similar trend for both LLaMA-2 and LLaMA-3 models. Therefore, we adopted the LLaMA 2 model, as it is more computationally efficient and better suited for our limited fine-tuning data.\n\n***Q4. Details on conditional and unconditional fine tuning of the LLMs***\n\n> For fine-tuning LLMs, we adopted the strategy proposed in [1],[2]. We fine-tune the pre-trained LLaMA-2 model on a dataset of crystal structures represented as strings, accompanied by prompts instructing the model to generate bulk materials by providing lattice parameters (lengths and angles), along with atom types and atomic coordinates.\n> Specifically, we use the following prompt:\n>\n>  `Below is a description of a bulk material. [The chemical formula is Pm₂ZnRh, ... <Other Conditions>]. Generate a description of the lengths and angles of the lattice vectors and then the element type and coordinates for each atom within the lattice: [Crystal string]`\n>\n> The text in [Conditions] enables conditional generation; omitting it allows for unconditional generation. [Crystal string] represents the string encoding of atoms within the lattice.\n\n***Q5. Distribution Alignment Between LLM Outputs and Diffusion Training Samples***\n\n> In our design, both components, LLM and Diffusion, are trained independently on the true data distribution using the available training dataset. However, we acknowledge the reviewer’s concern regarding the potential distribution shift between the real training data and the data generated by the LLM.\n> \n> To address this, during sampling, we do not directly feed the LLM-generated intermediate representation into the diffusion model for the full T-step denoising process. Instead, we inject this intermediate representation at an intermediate step, denoted as $\\tau$, where T > $\\tau$ > 0. The value of $\\tau$ is treated as a tunable hyperparameter.\n> \n> We determine the optimal $\\tau$ using a validation dataset. Specifically, for each material in the validation set, we generate prompts and obtain outputs from CrysLLMgen, where we try different values of $\\tau$ (900, 800, 700, 600, 500, 400, 300, 200, and 100) by injecting the LLM output into the diffusion model at those respective steps. For each $\\tau$, we evaluate the final generated materials using metrics for both CSP and Gen tasks. The $\\tau$ value that yields the best validation performance on these metrics is selected for our final model for final evaluation.\n\n***Q6. CSP Task results on MPTS-52 dataset***\n\n> We thank the reviewer for the suggestion. We ran both LLaMA-2 (7B) and our proposed CrysLLMGen on the MPTS-52 dataset for the CSP task and compared the results with existing baseline generative models. The results are as follows:\n> \n> |Models|Match Rate|RMSE|\n> |-|-|-|\n> |CDVAE      |5.34 |0.2106|\n> |DiffCSP    |12.19|0.1786|\n> |FlowMM     |17.54|0.1726|\n> |CrysBFN    |20.52|0.1038|\n> |LLaMA-2 7B |39.96|0.1092|\n> |CrysLLMGen |**42.16**|**0.0977**|\n> \n> Results for CDVAE, DiffCSP, FlowM, and CrysBFN are taken from the respective papers. We observe that CrysLLMGen outperforms all baseline models in both Match Rate and RMSE. We will include these results in the revised manuscript.\n\n***Q7. Source of LLM Results in Table-2 for CSP Task***\n> We selected LLMs as one of the baselines for our study and trained them for the CSP task. Our goal was to include both generative models and LLMs as baselines for both tasks, allowing for a comprehensive comparison with our proposed CrysLLMGen model.\n\n***Q8. Typo regarding MP-20 RMSE in Table-2***\n> We thank the reviewer for pointing this out. It was an oversight on our part, and we will rectify it in the revised manuscript.\n\n***Q9. Evaluation of SUN Rate for DiffCSP++ in Table 4***\n\n> Since SUN and Stability Rate were not reported in the original DiffCSP++ paper, we generated 10K materials using DiffCSP++ and computed the Stability and SUN scores using CHGNet (as described in Section 5.4). The results are reported accordingly.\n> \n> To further verify the reviewer’s comment, we examined the FlowMM[2] and FlowLLM[3] papers, where Stability Rate and SUN Rate for other baselines such as CDVAE, DiffCSP, and FlowMM, measured using CHGNet, are reported. We verified these values and observed a similar trend in our results. While the absolute values we obtained are slightly higher, the rank order among baselines (CDVAE, DiffCSP, FlowMM) remains consistent with our observations.\n\n\n[1] Gruver, Nate, et al. \"Fine-tuned language models generate stable inorganic materials as text.\" ICLR 2024\n\n[2] Miller, Benjamin Kurt, et al. \"Flowmm: Generating materials with riemannian flow matching, ICML-2024\n\n[3] Sriram, Anuroop, et al. Flowllm: Flow matching for material generation with large language models as base distributions. NeurIPS - 2024"}]

Author rebuttal:

Meta-review: The paper proposes a hybrid LLM–diffusion model for crystal material generation that demonstrates competitive empirical performance. Reviewers agreed the paper is clearly written and that the empirical evaluation is strong, though concerns were raised about limited novelty, lack of joint training between the LLM and diffusion model, evaluation transparency, and missing baselines. The authors addressed most of these convincingly during rebuttal, and with three of four reviewers recommending acceptance, I recommend acceptance as well.
Decision: Accept (poster)