Generative Hierarchical Materials Search
Authors: Sherry Yang
Simon Batzner
Ruiqi Gao
Muratahan Aykol
Alexander L Gaunt
Brendan McMorrow
Danilo Jimenez Rezende
Dale Schuurmans
Igor Mordatch
Ekin Dogus Cubuk
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative models for materials
diffusion models
multimodality
hierarchical search
Primary area: machine_learning_for_physical_sciences
TL;DR: We propose Generative Hierarchical Materials Search, a language-to-structure generative framework that uses LLMs to generate crystal formulas, diffusion models to generate crystal structures, and hierarchical search to find the best structure.

Abstract: Generative models trained at scale can now produce novel text, video, and more recently, scientific data such as crystal structures. The ultimate goal for materials discovery, however, goes beyond generation: we desire a fully automated system that proposes, generates, and verifies crystal structures given a high-level user instruction. In this work, we formulate end-to-end language-to-structure generation as a multi-objective optimization problem, and propose Generative Hierarchical Materials Search (GenMS) for controllable generation of crystal structures. GenMS consists of (1) a language model that takes high-level natural language as input and generates intermediate textual information about a crystal (e.g., chemical formulae), and (2) a diffusion model that takes intermediate information as input and generates low-level continuous value crystal structures. GenMS additionally uses a graph neural network to predict properties (e.g., formation energy) from the generated crystal structures. During inference, GenMS leverages all three components to conduct a forward tree search over the space of possible structures. Experiments show that GenMS outperforms other alternatives both in satisfying user request and in generating low-energy structures. GenMS is able to generate complex structures such as double perovskites (or elpasolites), layered structures, and spinels, solely from natural language input.

Generative Hierarchical Materials Search
Sherry Yang, Simon Batzner, Ruiqi Gao, Muratahan Aykol, Alexander Gaunt, Brendan McMorrow, Danilo Rezende, Dale Schuurmans, Igor Mordatch, Ekin Dogus Cubuk Google DeepMind
Abstract
Generative models trained at scale can now produce text, video, and more recently, scientific data such crystal structures. In applications of generative approaches to materials science, and in particular to crystal structures, the guidance from the domain expert in the form of high-level instructions automated system to output candidate crystals that are viable for downstream research. In this work, formulate end-to-end language-to -t-rruucttt generation as a multi-objective optimization problem, and propose Generative Hierarchical Materials Search (GenMS) for controllable generation of crystal structures. GenMS consists of (1) language model that takes high-level natural language as input and generates intermediate textual information about crystal (e.g., chemical formulae), and (2) diffuuson model that takes intermediate information as input and generates low-level continuous value crystal structures. GenMS additionally uses graph neural network to predict properties (c.g., formation energy) from the generated crystal structures. During inference, GenMS leverages all three components conduct a forward tree search over the space of possible structures. Experiments show that GenMS outperforms other alternatives of directly using language models to generate structures both in satisfying user request and in generating low-energy structures. We confirm that GenMS is able to generate common crystal structures such double perovskites, or spinels, solely from natural language input, and hence can form the foundation for more complex structure generation in near future.
be essential for
Introduction
Modern technologies increasingly rely on the development of materials, such as semiconductors [1], solar cells [2], and lithium batteries [3]. Large-scale generative models, trained on expansive internet data, exhibit intriguing generalization capabilities. For example, these models can synthesize a highly realistic image of "an astronaut riding horse" by merging two distant concepts [4]. This raises compelling question: can the generalization capabilities of large generative models, pretrained on existing materials science knowledge, be harnessed to combine knowledge from existing materials systems to propose candidate crystals?
Previous research has demonstrated that generative models can output crystal structures that not in the the training data [5, 6, 7]. However, these works typically require either a vast number of unconditional samples to generate unknown material [5, 8] or chemical formula provided during inference [6, 9]. It is difficult for end users to come up with new chemical formulae, as is hard to know which compositions will result in what material properties. Therefore, it is highly desirable to develop an interface that allows users to describe the desired characteristics of crystal structures such properties, compositions, space groups, and geometric characteristics For example, a user might specify "a stable chalcogenide with atom ratio 1:1:2 that is not on ICSD." Ideally, model should automatically interpret these high-level language instructions search for,
in natural language.
"Correspondence to sherryy@google. com and cubuk&google com.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Instruction following
Validity
Chulesgerio conranning one or mom chrieggnn ckamores, e. 8.5t,
KAg3Se2
X
Retrieval
W KUSJII
Gencrate stable, chaleagenide with ator 1:1 is not in ICSD
KGdSe2
TTo
KTmTee2
Instruction following
Low
NaHSD4
X
Figure 1: Overview of GenMS. GenMS takes high-level language instruction input, retrieves relevant information from the internet, and samples from a high-level LLM (Tm) to generate candidate formulae that satisfy user requirement. GenMS then samples from low-level diffusion model (ma) to generate structures conditioned on candidate formulae. Sampled structures then go through a property prediction module for selection.
generate, and validate a wide range of potential structures, ultimately producing one that best meets the user's specifications.
However, developing an end-to-end language-fo-struucure generative model presents several chal- lenges, for which we make a few key observations. First, there are no existing labeled datasets that map language descriptions directly to crystal structures. Nevertheless, observe that there is wealth of language-to-fore formulll data available online, including Wikipedia articles, research papers, and textbooks. This data can be complemented by formula-to-structur information from specialized materials databases such as the Materials Project [10], ICSD [11]. OQMD [12]. etc. Second, the task of converting language into structures is inherently multimodal, requiring the transformation of discrete linguistic inputs to continuous structural outputs. Nevertheless, it has been shown that semantic-level autoregressive models combined with low-level (pixel-level) diffusion models effective for cross-modal generation, such as in text-to-video applications [13, 14]. Lastly, user descriptions of desired crystal structures details about the crystal they wish to generate. We observe that one can leverage generative models infer missing information, and rely on additional search and selection mechanisms to identifyy structures that best satisfy a user's requirement.
often be vague- users may not articulate all relevant
Based on these observations, propose Generative Hierarchical Materials Search (GenMS) for end-to-end language-to-stin to-structure generation. GenMS consists of (1) a large language model (LLM) pretrained on high-level materials science knowledge from the internet, (2) a diffusion model trained on low-level crystal structure databases, and (3) graph neural network (GNN) for property prediction. To improve the efficiency of (2), GenMS proposes a compact representation of crystal structures for diffusion models. During inference, GenMS prompts the LLM to generate candidate chemical formulae according to user specification, samples structures from the diffusion model, and uses the GNN to predict the properties of the sampled structures. To sample structures that best satisfy user requirements during inference, we formulate language-fo-structure as multi-objective optimization problem, where user specifications are transformed into objectives that can be optimized at both the formula and structure level.
We first evaluate GenMS's ability to generate crystal structures from language instructions, and find that GenMS of the time for three major families of structures, while proposing structures with low formation energies, verified by DFT calculations. In contrast, using pretrained LLMs to directly generate crystal structures from user instructions in a zero-shot manner often results in close to a 0% success rate. Qualitative evaluations show that GenMS is able generate complex structures, such as layered structures, double perovskites, and spinels, solely from natural language. We next study the effect of each individual component of GenMS. Here we find that language instructions have a significant impact on the structures generated, that the novel compact representation of crystals proposed by GenMS improves the DFT convergence rate of diffusion generated crystal structures by 50% over previous work, and that using a pretrained GNN to select samples leads to lower energy structures more than 80% of the time. Given such experimental evidence, we believe the development of language-to-struue to-tuccuue models are promising for enabling users to find viable crystal structure candidates, complementing existing databases in utility
successfully generate structures that satisfy user requests more than 80%

---

2 Generative Hierarchical Materials Search
We begin by formulating the problem of generating crystal structures from high-level language as a multi-objective optimization task. Given this formulation, we then propose a hierarchical, multi- modal tree search algorithm that leverages language models, diffusion models, and graph neural networks as submodules. Lastly, we discuss the specific design choices for each of the submodules.
2.1 Language to structure as a multi-objectivv optimization
Given some high-level language description g E G of desired structures, we want to learn a conditional crystal structure generator x (-19)) 9 ~> (x that can be used to sample crystal structures x E X conditioned on language. One option is to parametrize with pretrained LLM. However, pretrained LLMs alone are not able to predict sufficiently accurate crystal structures, due to the lack of low-level structural information about crystals (e.g., 3D atom coordinates) in the pretraining data.
If we had access to a paired language-to-strene st-tructure dataset, D 2g π could be trained using a maximum likelihood objective. However, materials data naturally exist at different levels of abstraction and are segregated into different sources: high-level symbolic knowledge is documented in sources like Wikipedia articles, research papers, and textbooks, whereas detailed low-level crystal atom positions, is stored in specialized crystal databases like the Materials Project [10] and ICSD [11]. Even though direct language- eeguuage-tossinn ttrruucture dataset 'D remains unavailable, the pretraining data for LLMs, including Wikipedia articles, research papers, and textbooks, can be viewed as high-level symbolic dataset Dni = (gi, 24 km where E Z denotes symbolic textual information such as chemical formulae. Meanwhile, many crystal databases already feature paired data, Di=- {z,, Tz jim linking chemical formulae to detailed crystal structures.
information, including continuous-value
Given this observation, we propose to factorize the crystal generator as π Thi Tio. where Thi G + A(Z) and To Z + ((X)) so that Thi and can be trained using different datasets Dai and Dio- Furthermore, we consider two heuristic functions, Rui Rgi =) G x + IR and R2)(2,2) 2 x X' + R, where the high-level heuristic function Rui can be used to select formulae that satisfy the language input at high level, and the low-level heuristic function Rs can be used to select structures that are both valid and exhibit desirable properties such as low formation energy. To this end, we propose to search for crystal structure given language input by finding a chemical formula / space group with a corresponding crystal structure that jointly optimize
2" x = arg
max Tai.
E Bg (2) Dm Rmi(g,2) 2) + Aso Rio(2,)),,
(1)
where An and A are hyperparameters to control how much weight to put on high and low-level heuristics. Note that Ru and Rn also be combinations of multiple objectives. For instance, Rai can be a weighted sum of instruction following and simplicity, where Ri, can be a weighted sum of properties such as band gap, conductivity, and formation energy.
2.2 Searching through language and structure
Given the objective in Equation is clear that pretrained LLM (even with finetuning) insufficient to optimize for the best structure x=. Instead, we propose to first sample set of intermediate chemical formulae from pretrained LLM mhi (g) conditioned on language input g. We then use the high-level heuristic function Pui to prune and rank the intermediate formulae. In practice, Rui is a combination of (i) a regular expression checker (to ensure sampled formulae are valid chemical formulae), (ii) uniqueness checker against formulae from existing crystal datasets such as Materials Project and ICSD, and (iii) a formula compliance checker to ensure the sampled formulae are compatible with user request (c.g.. atom ratio 113 for perovskites, 227 for pyrochlore, and 124 for spinel). For formulae that pass these checks, we prompt a pretrained LLM as Rm to rank the formulae by how likely they to comply with the user request g. We then select the top W ranked formulae to generate L crystal structures each using Th parametrized by a diffusion model, and use a graph neural network Rs to rank the W x L structures by their predicted formation energy. Note that additional checkers can be integrated in Rio such as structural and compositional validity defined in [5]. We illustrate the overall search procedure in Algorithm
We use A(-) to denote probability simplex function.
3

---

Algorithm Generative Hierarchical Materials Search
1: Input: Language input
2: Functions: High-level language policy mm(=]g). high-level heuristic function Raig, 2), low-level diffusion
policy m=(x))), low-level heuristic function ())
3: Hyperparameters: High-level language branching factor H, low-level structure branching factor I, max
width for formulae W.
4: plans + [[g] V i E {1...HH
Initialize H different plans starting with language input.
5: for h = 1... H do
6: g plans/h]] [h][-1]
Get the high-level language specification from the trec.
7: {= t (9)
# Generate H different intermediate formulae.
8: 2" argmax(( go, 2 2 Rm)
9: plans(h).append(
# Add formula with the best heuristic value to plan.
10: end for
11: plans sort(plans, Ra)
# Sort formulae based on heuristic.
for L... W do
z + pl-paaassu [w][
# Get the best intermediate formula from the tree.
14: {a.))=1 m=(=)
# Generate L low-level structures.
15: x² = =aggma (z, m. 1 Rw))
16:
plans(w].append(r")
# Add structure with the best heuristic value to plan.
17: end for
return plans[O][O]
Return the best structure.
Alternative search strategies. The search algorithm described above, Algorithm follows the best-first search strategy, i.e., intermediate formulae and final structures are sorted and searched over based on the preference of a heuristic function. Alternative search strategies such as breadth-first or depth-first can also be employed. The most suitable search strategy depends on the downstream application computational resources available. For instance, if large-scale density function theory (DFT) calculations are available downstream, we can employ breadth-first search to devise more diverse composition.
Prevent heuristic exploitation. One concern of using heuristic GNN to select structures with the lowest formation energy is that the GNN might exploit irregularities in the predicted structures, especially when predicted structure lies outside of the training manifold of the energy GNN. To mitigate this issue, we use the GNN pretrained by [15] on DFT energies and forces of unrelaxed structures (hence the GNN has seen more irregular structures prior to relaxation.) Furthermore, we discard sampled structures from To if they result in energy predictions from Rio that lie outside of threshold range.
2.3 Choices of parametrization for the submodules
Since controllable crystal structure generation from language input is multimodal by nature, there are various design choices for the parametrization of the submodules in Equation 1, namely the generators Thi, Fl and the heuristic functions Rai Rio. In this section, discuss the parametrization choices we have found to be the most effective.
Retrieval augmentation and long-context deduplication. One important recent advance in LLMs increased context length [16]. The factorization ग = Thi Tho provides a natural way to integrate additional context in Thi via long-conext generation. Specifically, we further factorize mhi into retuivall (- is deterministic retrieval function that uses the Wikipedia RAG is a retrieval augmented the
retrival RAG where Thi
Thi Thi Thi
API to retrieve textual information related to language input g, while "hi
generative (RAG) model that proposes chemical formulae and space groups conditioned information retrieved from the internet. Another use case for long-context L.MSs to further encourage the generation of new compositions by providing the formulae for all known crystals in the context, then asking to produce a formula that is not in the context. As we will see in Section 3.2, this drastically improves the efficiency of the search, as large subset of the search space with known crystals can be eliminated.
Compact crystal representation. In order to support efficient tree search at inference time, we need to ensure that sampling from both Thi and no efficient. Previous work diffusion models

---

Noisy atoms
Denoised atoms
2, (h. 4)
MLP
D. 1D
D. m.m* MLP
attention
Sclf-
It, mmk
31 1.
MLP
X,
(h, 4)
Figure 2: Diffusion architecture with compact crystal representation. The diffusion model in GenMS represents crystal structures by the y, location of each atom plus the atom number represented as a continuous value. Each atom undergoes blocks consisting of multi-layer perceptrons followed by order-invariant self-attention. The MLP and self-attention blocks are repeated & times where each repetition increases the dimension of the hidden units. The concatenation of skip connections are employed as in other U-Net architectures.
for crystal structure generation has leveraged sparse data structures, such as voxel images [17, 18, 19], graphs [5], and periodic table shaped tensors [6]. These existing representations crystals incur computational overhead due to sparsity (voxel images, padded tensors) or quadratic complexity the number of atoms in the system increases (graphs). Instead, we propose new compact representation of crystal structures, where each crystal being the number of atoms in the crystal, and the inner dimensions representing the æ, y, location of atom along with its atom number. Here directly represent the atom number as continuous value normalized to the range the input in the diffusion model to further improve inference speed, as opposed representing the atom number using one-hot vector. In addition, use another 2 x 3 vector to represent the lattice structure (i.e., angles and lengths of the unit cell). Figure illustrates the architecture for the diffusion model with compact crystal representations, where each atom undergoes multi-layer perceptron (MLP) followed by order-invariant self-attention (without positional encoding) across atoms. Different from typical U-Net architecture for image generation, there is no downsampling or upsampling passes that change the input resolution. Nevertheless, we follow the concatenation of skip connections commonly used in U-Net architectures [20]. Additional details and hyperparameters for the diffusion model can be found in Appendix A.3.
X' R RAx4 is represented by A x 4 tensor, with A
Experimental Evaluation
We now evaluate the ability of GenMS to generate crystal structures from high-level language descriptions. First, we evaluate the success of end-to-end generation Section 3.1. We then investigate the individual components of GenMS in Section 3.2. See details of experimental setups in Appendix A.
3.1 End-to-end evaluation
Baselines and metries. We aim to evaluate GenMS's ability to generate unique, valid, and po- tentially stable crystal structures from well-known crystal families that satisfy high-level language specifications. We consider few-shot prompting of LLMs to generate crystal information files (CIF) as a baseline. Specifically, we give the Gemini long context model [16] a number of CIF files from a particular crystal family, specified by language input prompt, with the number of CIF files rang- ing from 1, 5, 25 to as many as can fit in the context. We ask the LLM to generate 100 samples given each language instruction. See additional details of baselines in Appendix A.2. We do compare finetuning LLMs to generate CIF files in this section, as there are no high-level language to low-level crystal structure datasets available for finetuning such instruction following L.M.. Nevertheless, we will compare the diffusion model GenMS to formula-conditioned structure generation using finetuned LLM in Section 3.2. We consider language input that directs the model to generate unique and stable crystals from a particular crystal family (perovskite, pyrochlore, and spinel). We consider the following metrics for evaluation: (i) CIF validity, which measures whether the generated CIF file can be properly parsed by pymatgen parser [21]. (ii) Structural and composition validity, which verify atom distances and charge balances using SMACT [22], following [5]. (iii) Formation energy per atom (E,) in the unit of VVatom, which measures the stability of predicted structures using pretrained GNN. We further conduct DFT calculations to compute E, (see details Appendix A.4) for structures predicted by GenMS. (iv) Uniqueness, which measures the percentage of generated formulae that do not exist Materials Project [10] or ICSD [11]. Finally, (v) the match rate, which measures the percentage of generated structures that can be matched (according to the pymatgen
5

---

Prompting CIF
Family
Metric
GenMS
shot
shot
25 shot
Max
CIF validity 1
0.94
1.00
0.98
0.88
1.00
Structural validity f
0.04
0.28
0.66
0.22
1.00
Composition validity
0.07
0.17
0.45
0.00
0.85
Perovskites
By (GNN/DFT) & Uniqueness
N/A
-0.19
0.28
0.53
-0.47/-1.32
0.07
0.29
0.68
0.16
0.90
Match rate
0.00
0.09
0.36
0.19
0.93
CIF validity
0.40
0.60
0.64
0.88
1.00
Structural validity?
0.36
0.36
0.28
0.23
0.95
Composition validity?
0.00
0.22
0.18
0.00
0.89
Pyrochlore
By (GNN/DFT)4
1.28
1.19
0.63
-1.22
-1.37/-2.56
Uniquenesst
0.36
0.38
0.45
0.08
0.49
Match rate
0.00
0.00
0.04
0.00
0.86
CIF validity
0.73
0.96
0.97
0.96
Structural validity?
0.47
0.61
0.71
1.00
1.00
Composition validity?
0.29
0.92
0.92
1.00
1.00
Spinel
E, (GNN/DFTJ
1.09
-0.85
-0.97
-1.37
-1.38/-1.77
Uniquenesst
0.48
0.13
0.51
0.08
0.44
Match rate
0.00
0.12
0.18
0.08
0.89
Table I: End-to-end evaluation of generating crystal structure from natural language. GenMS significantly outperforms LLM prompting baselines in producing unique and eowrenergy (predicted by GNN) structures that satisfy user request. We further conduct DFT calculation to compute E, (formation energy e e/atoo) averaged across structures generated by GenMS. Values before "f" in the row "(GNN/DFT)" represent GNN predicted Fyy. and after "f" represent DFT computed F.y. We report Fy from GNN prior to relaxation, and F, from DFT post relaxation. DFT calculations for baselines are eliminated as many structures from the baselines do not follow user instruction. N/A represents F predicted by GNN falling outside of the reasonable range.
structure matcher) to one of the structures of the corresponding family in Materials Project. More details of these metrics can be found Appendix A.1.
Results on specifying crystal family. The evaluation of GenMS and baselines are shown Ta- ble 1. Since GenMS does not rely LLM to directly generate CIF files, the compact crystal representation (described Section 2.3) always results in structures that can parsed by pymatgen (100% CIF validity). In addition, structures generated by GenMS have much higher validity and match rate compared to those generated by the basclines. GenMS struggles slightly with uniqueness, as less than half of the generated formulae for pyrochlore and spinel unique with respect to MP and ICSD. Structures produced by GenMS have lower average Ey. Increasing the number CIF files in the context generally improves the performance of the baselines (1, 5, and 25-shot), but including too many files the context can hurt performance (Prompting CIF Max).
Qualitative evaluation. In addition to the three families of structures evaluated above, we qualita- tively evaluated GenMS's ability to generate structures that satisfy ad hoc user requests, such "a layered material", "an elpasolite", and so on. GenMS can consistently produce structures that satisfy user request as shown in Figure 3, and have plausible initial geometries. Interestingly, that GenMS can understand semantic-level request, suggesting more "fluoride" like chemistries when asked for "elpasolite", which reasonable as elpasolite is associated with the mineral K2NaAIF6.
observe
Effect of search. Next, we aimed understand the effect of search in GenMS, especially in producing low-energy structures. For each of the family of crystals in Table 1, we analyzed the effect of the language and structure branching factors (H and L in Algorithm 1). Only crystals that match input specification were considered for energy computation. We found that increasing the branching factor of both language and structure enables GenMS to generate structures with lower formation energies (at higher inference cost).

---

Cl perovskite
Pryochlore
Double perovskite
Spinel
Figure 3: Qualitative evaluation. We test GenMS of ad hoc language inputs generate plausible examples from well-known crystal families. GenMS is able to search for the corresponding structures that satisfy user requests and have plausible initial geometries. Visualization provided by VESTA [23].
Perovskite
Pyrochlore
Spinel
Language
Structure branch (L)
F
Language branch (H)
Structure branch (L)
Ff
Language branch (H)
Structure branch (L)
Fy
branch (H)
(DFT)
(DFT)
(DFT)
-0.55
1
-2.40
-1.67
1 100
-0.79
100
-2.51
100
-1.74
25
-2.76
25
1
-3.02
25
1
-1.82
25
100
-2.91
25
100
-3.24
25
100
-1.95
Table 2: F. (computed by DFT) vs. branching factor. GenMS can generate structures with lower formation energy (computed by DFT) at the cost slower inference when language and structure branching factors are increased.
3.2 Evaluating individual components of GenMS
Next, evaluate the individual component of GenMS, including the effect of using language to narrow down the search space, the choice of the compact representation of crystal structures, and finally the best-of-N sampling strategy for choosing the crystal structures with low formation energy.
Effect of language. We want to understand whether GenMS can provide effective control over formulae proposed by the LLM the semantic-level through natural language. In Table 3, we first show that requesting a particular element to be in the formula always results in formulas with that particular element being proposed by the pretrained L.M Thi We then show that when user requests for metal, the model is 4 times more likely to generate formulae for metal. The model also respects user's request for the generated formulae be unique (with respect to either a user provided list of known formulac in the context of the LLM, the name of some crystal database).
Next, we study the effect of retrieval augmented generation (RAG). We GenMS with and without RAG to propose 25 formulae for each of the three major crystal families from Section 3.1 and generates structures per formula using the diffusion model. We report the rate of valid formulae proposed by the LLM and the structures that can be matched with existing structures from the corresponding family in Table RAG improves both the rate of valid formulae and matched structures.
Element
Metal only
Unique (custom list)
Unique (Materials Project)
Valid formula
Match rate
constraint
Not asking Asking
N/A
0.25
0.24
0.16
Without RAG
0.97
0.72
1.00
1.00
0.88
0.96
With RAG
1.00
0.89
Table 3: Effect of language. Asking for a specific element from the periodic table results in formulae that always tain that element. Asking for metal and formulae unique with respect to existing formula sets result in formu- lae that are more likely to satisfy user requests.
Table 4: Effect of RAG. Using retrieval augmented generation improves the per- centage valid formulae and matched structures. See details for the structure matcher used in Appendix A.1.

---

UniMat [6]
GenMS
Small LLM [9] Large LIM GenMS
DFT converge
0.62
0.93
Success
0.86
0.87
0.93
Mean Fyy
-0.40 = 0.06
-0.49 + 0.03
Match (unscen)
0.26
0.37
0.48
Table 5: DFT evaluation of GenMS vs Uni- Mat. Structures proposed by GenMS result in much high DFT convergence rate and lower average E than structures proposed by Uni- Mat. Error bars reflect standard error.
Table 6:
Comparison to finetuned LLMs.
GenMS's diffusion model with compact represen- tations achieves high success rate of generating valid crystals, as well as a high matching rate to holdout structures compared to Crystall M [9].
Formation energy difference (GNN)
Formation energy difference (DFT)
bf
Erbest-of-it sisingle
19 single
singee
Composition
Composition
Figure 4: Formation energy between Best-of-N and single sample. Both according to energy predicted by GNN and calculated by DFT, best-of-N with N = 10 leads to improvements energy compared to single samples for 80% of 1,000 compositions considered.
Compact crystal representation. We now evaluate the diffusion model Ma trained using the com- pact representation of crystals structures described in Section 2.3. We compare diffusion model with compact crystal representation against two prior work for generating crystal structures conditioned on composition. UniMat [6] proposed a periodic table representation crystals which requires a large amount of paddings to handle atoms that do not exist in the structure. CrystalLM [9] proposes to finetune an LLM to directly generate CIF files from input compositions. In Table 5, we report the DFT convergence rate and DFT calculated E on set of holdout structures following [6]. We observe that the compact crystal representation results in both higher convergence rate and lower E than the sparse representation in [6]. To compare GenMS's diffusion model against finetuning LLMs to generate CIF files directly, we follow the experimental setting of CrystaL.LM where we train a composition conditioned diffusion model and NOMAD [25], and test the success rate of generating matching structures for unseen composi- tions following [9]. Table 6, we see that GenMS has significantly higher rate in producing valid crystal and a crystal that can be matched to the test set in [9].
combination of Materials Project [10], OQMD [24),
Best-of-N structure sampling. To better understand the effect of high structure branching factor in Algorithm across different compositions, we measure the difference the formation energy, using a holdout test set of 1,000 compositions, between using the energy prediction GNN to select the best of 10 samples compared to only predicting a single structure. The energy difference with and without best-of-N sampling is shown in Figure 4. Using best-of-N with N = 10 results in improved energy for over 80% of structures (as also verified by DFT calculations). We found the energy prediction GNN to be good indicator of the true energy of the crystal structures, i.e., the GNN predicted energy difference (left) and the DFT calculated energy difference (right) are very similar in Figure
4 Related work
Hierarchical and latent image and video generation. Image and video generative models have exhibited an impressive ability synthesize photorealistic images or videos when given text de- scription as input. Many of the state-of-the-an models adopt a hierarchical modeling approach that inspired the design of with GenMS. For example, latent diffusion models [26, 27] contains (1) language model that converts text to high-level text embeddings, (2) a diffusion model takes the text embeddings as input and output latents in a compressed latent space, and (3) feed forward decoder network [26] or diffusion decoder [28, 14] that given the generated latents generates full-resolution signals in the pixel space. Cascaded diffusion models [29, 30, 31] instead proposed to generate signals at the lowest resolution with standard diffusion model, followed by few super-resolution
8

---

models that successively upsample signals and add high-resolution details. Similar to GenMS, by breaking down complicated image or video generation into a hierarchy of less challenging problems, these models can generate high quality samples more efficiently and effectively.
Generative models for crystal structures. A number of works [9, 8, 32] have proposed to train or fine-tune language models to generate output files containing crystal information or low-level atom positions. However, remains expensive and challenging to train and generate detailed structural information with LLMs. On the other hand, diffusion models, as a powerful class of generative model in vision, have been applied to generate crystal structures [5, 7, 6J. However these methods either reply on training with large set of unconditional samples and brute-force sampling for new materials not in the training set, or necessitate predetermined compositions as conditioning information during inference. Handling of candidate structure generation requires a model capable of independent reasoning about chemical compositions based on high-level user specifications and structure optimization, as done in GenMS.
Hierarchical search and planning. The problem of learning to generate low-level continuous output from high-level language instructions, while employing intermediate search and planning steps, has been studied in other domains such as continuous control [33], self-driving [34], and robotics [35]. While some works have focused purely using LLMs to search and plan through complex output spaces [36, 37], other research has shown that solely relying on LLMs to search and plan can fail short due to the lack of low-level information (e.g., locations, precise motions) captured in the model [38]. Recently, video generation models have been applied to provide additional details about the physical world that low-level control actions can be extracted more accurately [39, 40, 41, 42]. GenMS follows a similar approach but focuses on generating crstyal structures, using diffusion models on top of LLMs to provide additional details about crystal structure, enabling high-level plans (i.e., symbolic chemical formulae) to be verified at a low-level (i.e., crystal structures with precise atom locations).
Large language models for science. Recently, there has been a surge of interest in applying large langauge models in domains of science, such as physics [43], biology [44], chemistry [45, 46], and materials science [47]. In these settings, LLMS generally serve as conversational [44] or educational [48] tool, where LLMs output natural language to be consumed by human users (e.g.. an answer to a scientic question asking about the property of some existing crystal structure). On the other hand, we are interested in the ability of a pretrained LLM to propose intermediate textual information such as chemical formulae for interesting crystal structures. Closest to our work are [49, 50] which leverage an L.M to generate SMILES or other chemical strings for molecular design. Nevertheless, we are interested in generating not just the formulae, but the actual crystal structures with continuous-valuee atom locations, as many materials property can only be calculated and verified once the full structure available.
Conclusion and future work
We have introduced GenMS, an initial attempt at enabling end-to-end generation of candidate crystal structures that look physically viable and satisfy instructions expressed in natural language. GenMS can generate examples from families such as pyrochlores and spinels purely from natural language prompts. We hope the design principles of GenMS will initiate broad interest in exploiting language as natural interface for flexible design and generation of crystal structures that meet user-specified criteria, and enable the domain experts to work more efficiently. GenMS has few limitations that call for future work:
Generating complex structures. While GenMS is able generate simple structures such as those shown in Figure 3, we found that GenMS is less effective in generating complex structures such Mxenes and Kagome lattices. Controllable generation of highly complex crystal structures is an interesting area of future work.
Impact on experimental exploration. While we have shown that GenMS is effective in gen- erating crystal structures that are not in public databases and that satisfy user requirements, its effectiveness in suggesting specific materials with target properties (e.g., battery electrodes or electrolytes, semiconductors, superconductors etc.) requires further experimental verification.

---

Synthesizability. While the goal of GenMS is to provide an end-to-end generative framework from natural language instructions to realistic erystal structures, synthesizability of the generated crystals is not currently part of the pipeline. We foresee development in multimodal models and integration of other computational tools from materials science to allow predicted structures be assessed for synthesizability.
Extension to other chemical systems. We have shown that GenMS can effectively generate crystal structures from natural language. We note that GenMS can also potentially be extended to generating molecules and protein structures from natural language (e.g. "generate a protein with an alpha-helix"), We leave these explorations for future work.
References
[1] Lev Berger. Semiconductor materials. CRC press, 2020.
[2] Martin A Green, Anita Ho-Baillie, and Henry Snaith. The emergence perovskite solar cells. Nature photonics, 8(7):506-511 2014.
[3] KJPC Mizushima, PC Jones, PJ Wiseman, and John Goodenough. Lixcoo2 (0< x<-I): A new cathode material for batteries of high energy density. Materials Research Bulletin, 15(6))783-789, 1980.
[4] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International conference machine learning. pages 8821-8831. Pmlr, 2021.
[5] Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crys- tal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197, 2021. 3, 5.
[6] Mengjiao Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Dogus Cubuk. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235 2023. 5, 9.
[7] Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabbé, Lixin Sun, Jake Smith, et al. Mattergen: generative model for inorganic materials design. arXiv preprint vrXiv:2312.03668 2023.
[8] Daniel Flam-Shepherd and Alán Aspuru-Guzik. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files. arXiv preprint arXiv: 2023. 1,9
[9] Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespo. Crystal structure generation with autoregressive large language modeling. arXiv preprint arXiv:2307.043440 2023. 1, 14
[10] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013. 3, 14
[11] Mariette Hellenbrandt. The inorganic crystal structure database (pese)-preeent and future. Crystallography Reviews, 10(1):17-2,, 2004. 2, 3, 5, 14
[12) Scott Kirklin, James E Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The open quantum materials database (oqmd); assessing the accuracy of dft formation energies. npj Computational Materials, 1(1):1-15, 2015. 2
[13] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023. 2
[14] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. 2, 8
10

---

[15] Amil Merchant, Simon Batzner. Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80-83,
2023. 4
[16] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint 20X1v:2403.055330 2024. 4,5
[17] Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-d crystal structures. arXiv preprint arXiv: /9999999999 2019. 5
[18] Juhwan Noh, Jachoon Kim, Helge Stein, Benjamin Sanchez-Lengeling, John M Gregoire, Alan Aspuru-Guzik, and Yousung Jung. Inverse design of solid-state materials via a continuous representation. Malter, 16)):137-1384, 2019.
[19] Callum Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of Chemical Information and Modeling, 60(10):4518 15353345118-55, 2020. 5, 14
[20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiisted Intervention-MICC MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015. proceedings, part III 18. pages 234-241. Springer, 2015.
[21] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314-319 2013. 5, 15
[22] Daniel W Davies, Keith Butler, Adam Jackson, Jonathan M Skelton, Kazuki Morita, and Aron Walsh. Smact: Semiconducting materials by analogy and chemical theory. Journal of Open Source Software, 4(38):1361, 2019. 5, 14
[23] Koichi Momma and Fujio Izumi. Vesta 3 for tiree-dimensiona visualization of crystal, vol- umetric and morphology data. Journal of applied crystallography, 44(6):1272-12777 2011.
7
[24] James E Saal, Scott Kirklin, Muratahan Aykol, Bryce Meredig, and Christopher Wolverton. Materials design and discovery with thghtthroughput density functional theory: the open quantum materials database (oqmd). Jom, 65:1501-1500 2013. 8
[25] Claudia Draxl and Matthias Scheffler. The nomad laboratory: from data sharing artificial intelligence. Journal of Physics: Materials, 2(3):036001, 2019. 8
[26] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022. 8
[27] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems, 34:1 1287-11302 2021.
[28) Aditya Ramesh et al. Hierarchical text-conditional image generation with clip latents, 2022. 8
[29] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022. 8
[30] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 01436647-364999 2022. 8

---

[31] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22100023033 2022. 8
[32] Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint (XX::22422.04379 2024. 9
[33] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500 9500. IEEE, 2023. 9
[34] Xingcheng Zhou, Mingyu Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C Knoll. Vision language models in autonomous driving and intelligent transportation systems. arXiv preprint 2rX14:2310.14414, 2023. 9
[35] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958 919,99992228 2024. 9
[36] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023.
9
[37] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36:75993-76005, 2023." 9
[38] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati, Large language models still can't 't plan (a benchmark for iims on planning and reasoning about change). arXiv preprint arXiv:2206. 10498, 2022. 9
[39] Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint 2023.
[40] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text- exxt-guiedd video generation. Advances in Neural Information Processing Systems, 36, 2024. 9
[41] Yilun Du, Mengjaa Yang, Pete Florence, Fci Xia, Ayzaan Wahid, Brian lchter, Pierre Sermanet, Tianhe Yu, Picter Abbecl, Joshua B Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625. 2023. 9
[42] Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum. Leslic Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36, 2024. 9
[43] Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T Sio, Lisa A McGee, Jonathan B Ashman, Xiang Li, Tianming Liu, Jiajian Shen, ct al. Evaluating large language models highly-specialized topic, radiation oncology physics. Frontiers in Oncology, 13, 2023.
[44] Rachel K Luu and Markus Buehler. Bioinspiredllm: Conversational large language model for the mechanics of biological and bio-inspired materials. Advanced Science, 11(10):230672, 2024. 9
[45] Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelli- gence, pages 1-11, 2024. 9
12

---

[46] Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang. Xiangyu Yue, Dongzhan Zhou, al. Chemllm: chemical large language model. arXiv preprini arXiv:2402.006852 2024. 9
[47] Ge Lei, Ronan Docherty, and Samuel J Cooper. Materials science in the era of large language models: a perspective. arXiv preprint arXiv:2403.06944, 2024. 9
[48] Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053-19061, 2024.
[49] Hisaki Ikebata, Kenta Hongo, Tetsu Isomura, Ryo Maezono, and Ryo Yoshida. Bayesian molecular design with a chemical language model. Journal of computer-aided molecular design, 31:379-300 391, 2017.
[50] Michael Moret, Irene Pachon Angona, Leandro Cotos, Shen Yan, Kenneth Atz, Cyrill Brun- ner, Martin Baumgartner, Francesca Grisoni, and Gisbert Schneider. Leveraging molecular structure and bioactivity with chemical language models for de novo drug design. Nature Communications, 14(1):114, 2023. 9
[51] Özgün Çięek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Conputer-Assisted Intervention-MICCAA 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pages 424-432. Springer, 2016. 14
[52] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022. 14
[53] Georg Kresse and Jürgen Furthmüller. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Physical review B, 54(16):11169 1996. 15
[54] Georg Kresse and Jürgen Furthmüller. Efficiency of ab-initio total energy calculations for metals and semiconductors using plane- wave basis set. Computational materials science, 6(1):15-50, 1996. 15
[55] John P Perdew, Matthias Ernzerhof, and Kieron Burke. Rationale for mixing exact exchange with density functional approximations. The Journal of chemical physics, 105929999822-9985
1996. 15
[56] Peter E Blöchl. Projector augmented-wand wave method. Physical review B, 50(24)) 1994. 15
[57] Georg Kresse and Daniel Joubert. From ultrasoft pseudopotentials to the projector augmented- wave method. Physical review b, 1958),775, 1999. 15
[58] Kiran Mathew, Joseph H Montoya, Alireza Faghaninia, Shyam Dwarakanath, Muratahan Aykol, Hanmei Tang, lek-heng Chu, Tess Smidt, Brandon Bocklund, Matthew Horton, et al. Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows. Computational Materials Science, 139: 339:140-155 152, 2017. 15
13

---

Appendix
A Experiment details
In this section, we provide additional experimental details, including metrics used for evaluation, baselines, architecture and training of the diffusion model with the compact crystal representation, and details of the setup for the DFT calculations.
A.1 Details of evaluation metrics
Structure and composition validity. The structure and composition validity metrics follow [5]. The structure validity determins that structure is valid as long as the shortest distance between any pair of atoms is larger than 0.5 Å [19]. The composition is valid if the overall charge is neutral as computed by SMACT [22].
Uniqueness. We determine generated formula is unique if the reduced form of the formula does not exist in either Materials Project [10] ICSD [11]. For instance, if ICSD contains formula in the form of AB2, we consider A2B4 generated by the model as duplicate (thus not unique) structure.
compute the match rate, we use the StructureMatcher module from pymatgen's 's analysis package. We set the hyperparameters of the matcher following [9], specifically with stol 0.5, ltol = 0.3, angle_tol and spinel, we first curate the reference set by downloading CIF files from Materials Project [10] that likely to belong to each family based on formula and space group. We then use fit_ fit_anonymouu method of the matcher to compare each generated structure the structures in the reference set. A generated structure is considered matched fit f anonymous returns true for at least one reference structure of the corresponding family. Note that this approach might result in false positive matches. For example, when we selected the reference set for pyrochlore, we downloaded CIF files Material Project that have composition A2B207. However, not all A2B207 are pyrochlore, so generated structures may still not be a pyrochlore despite being matched to one of the reference structures.
Match rate.
10. For each family of crystals in perovskite, pyrochlore,
A.2 Details of baselines
We use the following prompts in Table 7 to generate the CIF files for the end-to-end prompting baseline or to generate the chemical formulae for GenMS.
Method
Prompt
Prompt CIF (baseline)
"I want you to generate another crystal information (CIF) files for a stable and potentially realistic material that belongs to (category J. ((Optional) Here are some information about (category) from Wikipedia. Below some examples of CTF files from this category: (examplel, example2, ...) Please generate one more for a crystal that is not existing materials databases like Materials Project and ICSD. Please make sure the CIF file is valid. Just generate the file and do not anything else."
Prompt formula (GenMS)
[(Optional) Here are some information about [category] from Wikipedia.] Please give me a list of chemical formulae for a hypothetical material for [category] ). I want the formula to be stable. and potentially realistic and do not exist in dataset like Materials Project ICSD. Please just give the formula and do say anything else."
Table LLM prompts for baseline and GenMS.
A.3 Compute, architecture, and training
We repurpose the 3D U-Net architecture [51, 52] into modeling atoms within a crystal structure by their z, y, locations concatenated with atom number (number of protons) As result, we can
14

---

represent each crystal structure using an Ar4 matrix where A is the total number of atoms in the structure, and the dimension with size 4 represents the z, y. location and atom number of each atom. We repurpose the spatial downsampling and upsampling passes from typical U-Net for images or videos, and keep the resolution (number points) the same, but still employ residual network with concatenating skip connections (see Figure 2 from the main text). Below we show the architecture and hyperparameters used in the diffusion model for crystals with compact representation.
Hyperparameter
Value
Learning rate
5e-5
Optimizer
Adam (31 = 0.9, 82 0.99)
Base hidden dimension
256
Hidden dimension multipliers
1,2,4 4
Number of mlp and self-attention blocks
9
Batch size
512
EMA
0.9999
Weight decay
0.0
Prediction target
Attention head dimension
64
Dropout
0.1
Training hardware
64 TPU-v4 chips
Diffusion noise schedule
cosine
Noise schedule log SNR range
[-20, 20]
Training steps
200000
Sampling timesteps
256
Sampling log-variance interpolation
7= 0.1
Table 8: Hyperparameters for training the diffusion model in GenMS.
A.4 Details of DFT calculations
In all our density functional theory (DFT) calculations, we employ the Vienna ab initio simulation package (VASP) [53, 54] with the Berdew-Burke-Eminn augmented wave (PAW) potentials [56, 57]. Our computational settings align with those used in the Materials Project, as implemented in pymatgen [21] and atomate [58]. These settings include the application of the Hubbard U parameter to selected transition metals in DFT+U calculations, plane- wave basis cutoff of 520 eV, specific magnetization settings, and the use of PBE pseudopotentials. However, we opt for updated versions of potentials for Li, Na. Mg. Ge. and Ga, maintaining the same valence electron count. For structural optimization, our protocol involves a two-stage relaxation of all geometric parameters, followed by final static computation. We utilize the custodian package [21] to manage any issues with VASP and to make necessary adjustments to the simulations. Additionally, we generate gamma-centered k-points for hexagonal cells, deviating from the conventional Monkhorst- Pack scheme. We initialize our simulations with ferromagnetic spin, observing that attempts to explore alternative spin configurations were computationally too demanding. In our ab initio molecular dynamics (AIMD) simulations, we disable spin polarization and employ the NVT ensemble with 2 fs timestep. For systems containing hydrogen, we reduce the timestep to 0.5 to ensure accuracy.
(PBE) [55] functional and projector-
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provided thorough experiments and explanations of the algorithms jusify the claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discussed that in the conclusion section and we pointed out four major limitations.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
We do not introduce new theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We use public LLMs, and the prompts are all included in the appendix. For our work we have provided enough details for reimplementation, and we will work on open sourcing well.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data and prompts are all public and we have provided the information. We are working on open sourcing the code after going through the internal approval process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
http//guidcccsdeeeeee ceaibbmisssoooooo.ooooooo coooooolioo olicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. ct/pcc//pllc. ctc/public//ee Semnidde//ooeeuue
cessoonoolccnn for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the information in the appendix and main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provided the variance of the results in almost all tables when applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report this in our experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nurrps.....p// cc http:/pllc/tthissuuiii
Answer: [Yes]
Justification:
We have checked.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work is exploratory and is at too early stage to have broad impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No risk as far as we can see.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited the models (Gemini) used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Not involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: no crowdsourcing nor research
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "VXxEDH3nU1", "reviewer": "Reviewer_Xu7Y", "summary": "This paper presents Generative Hierarchical Materials Search (GenMS), a novel framework for material search using LLMs. GenMS consists of a language model that takes high-level natural language as input and generates intermediate textual information about crystals (e.g., chemical formulae). Specifically, it uses a RAG architecture to make an LLM incorporate information from Wikipedia articles and other retrieved articles from the Internet. The generated formulae are filtered and re-ranked by a heuristic function involving domain knowledge, such as a formulae regular expression checker. A diffusion model then generates material structures conditioned on the filtered formulae. Experimental results show that GenMS is a solid framework that can generate structures satisfying users' requirements, demonstrating that using LLMs for structure search is a promising direction.", "questions": "1. Considering the complexity of the whole system, what is the average inference latency for each query?\n2. How does using Wiki-API affect performance? Is there any ablation study on combining Wiki-API and RAG differently?", "limitations": "The design of the heuristic function requires a lot of domain knowledge, making the method difficult to reproduce.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow, even for readers not familiar with materials search.\n2. The proposed method is interesting, particularly in its use of RAG  to enable LLMs to follow the formula in articles. The ablation study shows RAG's effectiveness.\n3. Using LLMs for material search or other scientific problems is an important but underexplored domain. This paper presents a solid framework for using LLMs in material structure search, showing that this is a promising direction for scientific applications.", "weaknesses": "1. The architecture involves several steps requiring ad-hoc decisions, such as using Wiki-API for information retrieval or heuristic operations for selecting formulae. For example, lines 117-120 describe ad-hoc operations to prune intermediate formulae, making the method hard to reproduce and requiring more human effort. There might be more elegant, machine-learning-based ways to incorporate these operations, such as using LLMs for formula checking.\n2. The heuristic function design involves significant domain knowledge, requiring more ablation studies or analysis to justify the heuristic choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the significance of this work! Please see our response to your questions below.\n\n> Ad-hoc decisions in the framework.\n\nWe would like to clarify that the choice of $R_\\text{hi}$ and $R_\\text{lo}$ are not ad-hoc decisions made by the framework, but highlighting what practitioners can enforce in the materials search process. Checks currently implemented in the current framework are rather basic, e.g., formulae containing the desired elements, formulae consist of valid chemical elements, don’t contain duplicates, and have desired element ratios. There are certainly other checks that can be incorporated into the current framework, such as \"must contain element Si\" and \"have less than 5 chemical elements\", which gives a domain expert additional control in the generation process. We have rewritten lines 116-120 to reflect that $R_\\text{hi}$ and $R_\\text{lo}$ are only examples of what can be optimized in GenMS, as opposed to arbitrary choices we made.\n\n> Additional ablations.\n\nAs discussed in the previous paragraph, the goal of GenMS is not to develop the best heuristics but to allow diverse checks and constraints to be incorporated. That being said, the choice of what information to include in RAG is indeed an interesting ablation. We conducted two additional studies. First, we eliminate half of the retrieved Wiki text, and observe the match rate in Table 4 drops from 0.89 to 0.85. Second, we replace the text retrieved from Wikipedia by text retrieved from Materials Project [1] (entries that contain the keyword Perovskite), and observe that the match rate in Table 4 drops to 0.86, which is still better than no retrieval at all. We will include these additional results in the final version of the paper. However, we were unable to conduct more ablations, such as using research articles containing keywords as retrieval text in RAG due to the lack of a dataset with easily retrievable research articles in materials science. We leave this investigation for future work.\n\n[1] Jain, Anubhav, et al. \"Commentary: The Materials Project: A materials genome approach to accelerating materials innovation.\" APL materials 1.1 (2013).\n\n> Incorporating LLMs for formula checking.\n\nWe agree that using LLMs for formula checking is more general than using regular expression checkers. In fact, we are already using LLMs to rank the top W formulas as described on line 122. To investigate the reviewer’s suggestion of LLM feedback, we use Gemini long context [1] to judge whether structures are nonmetals. For 25 known nonmetals, LLM determins 23 are nonmetals, achieving a success rate of 92%. However, using LLM feedback for *novel* structures is more challenging, as we do not have ground truth for whether the LLM feedback is correct.\n \nWe believe that similar to RL with feedback for LLMs, where diverse heuristic and reward functions such as human feedback, AI feedback, and machine feedback have been jointly used to improve system performance, GenMS can also benefit from diverse heuristics and sources of feedback, including feedback from LLMs.\n\n[1] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean365 baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.\n366 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv\n367 preprint arXiv:2403.05530, 2024\n\n> Latency of the system\n\nThank you for bringing up this question. We have measured the latency of the system on TPU-V4 with batching enabled (using batch size 64 for the structure branch and energy GNN), and report the results below.\n\n| Language Branch (H): 1 | Structure Branch (L): 1 (batch size 64) | Energy GNN (batch size 64) | Total |\n| ------------ | ----------- | ----------- | ----------- |\n| 43ms  | 100ms | 800 ms | 943ms |\n\n| Language Branch (H): 25 | Structure Branch (L): 100 | Energy GNN (batch size 64) | Total |\n| ------------ | ----------- | ----------- | ----------- |\n| 1150ms | 4600ms | 36690ms | 42440 ms |\n\nThe latency of the language branch is measured on Gemini long context model and latency of structure branch is measured on the compact diffusion model. The latency of the energy prediction GNN is high due to the paddings applied to the structures to pad structures to the same number of nodes, and due to the breadth of the search (i.e. predicting energies for 2500 structures). We will include this result in the final version of the paper.\n\n> Concerns around the requirement of domain knowledge\n\nWhile it is compelling to eliminate the need for any domain knowledge, we think that some level of domain knowledge is hard to avoid, especially given the end goal is to have materials scientists use the framework to discover and even synthesize novel materials. Therefore, the framework should provide a way for material scientists to specify and verify the generation. Instead of getting rid of domain knowledge, GenMS leverages domain knowledge both through in-context learning and through heuristic/reward design, which gives practitioners the flexibility to generate and verify crystals based on their need.\n\nThat being said, we are not leveraging more domain knowledge to evaluate GenMS than previous work such as [1] and [2]; the validity metrics in Table 1 are standard in previous generative models for materials work. The results of GenMS are easy to reproduce, with the language prompt provided in Appendix A.2 and validity metrics following previous work ([1], [2]).\n\n[1] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" arXiv preprint arXiv:2110.06197 (2021).\n\n[2] Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., & Cubuk, E. D. (2023). Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c40GFA1KD9", "reviewer": "Reviewer_Tzq3", "summary": "The paper presents a language-to-structure generation model for crystal structures. They propose a hierarchical approach that uses a language model to generate intermediate textual crystal information and then generate low-level crystal structures using diffusion models. They demonstrate their ability to generate crystal structures from language instructions and compare it to baselines. Results show that GenMS outperforms the baselines in generating valid and low-energy structures that satisfy user requests.", "questions": "Can you add stronger baselines and show more comprehensive evaluation results?", "limitations": "the authors adequately addressed the limitations", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a technically sound approach that decomposes the problem into two step and uses separate models for each step based on the task characteristics. \n2. The empirical results are good. They demonstrate their effectiveness in generating valid and low-energy structures satisfying user requests and they outperform baselines.", "weaknesses": "1. The baseline is relatively weak as it is only a few-shot prompted LLM. For example, the ablation of each of their proposed modules (e.g., LLMs, retrieval) should be treated as baselines and the results should be put in the main table with more comprehensive evaluation results than the current ones.\n2. The overall workflow is a combination of several widely-used models (e.g., LLMs, diffusion models, retrieval models). The pipeline makes sense but combining these models may not be enough to make significant contributions for a research paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback. Please see our response below.\n\n> More comprehensive baselines\n\nThe reason that we only included few-shot prompting baselines in the end-to-end evaluation of Table 1 is because, to our knowledge, there is no natural-language-to-crystal-structure dataset available for finetuning LLMs to generate crystals from loose natural language descriptions, as this work is among the first to study this problem. Hence, few-shot prompting is the most natural baseline. However, we do evaluate the subcomponents of GenMS to previous baselines, and show that the formula-conditioned generation in GenMS performs favourably compared to diffusion model [1] and finetuned language model baselines [2] in Table 5 and Table 6 of the paper.\n\n[1] Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., & Cubuk, E. D. (2023). Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235.\n\n[2] Antunes, Luis M., Keith T. Butler, and Ricardo Grau-Crespo. \"Crystal structure generation with autoregressive large language modeling.\" arXiv preprint arXiv:2307.04340 (2023).\n\n> Concerns around contribution for a research paper.\n\nWe would like to highlight the nontrivial aspects of this work. Even though the solution of combining LLMs with diffusion models might seem trivial, the reason behind such a design and the process of how GenMS is the first to arrive at this design is nontrivial. Our contributions include recognizing the knowledge gap in high-level textbooks and low-level materials structure datasets, recognizing the gaps between models of different abstraction levels (LLMs for high-level semantics and diffusion models for low-level details), and properly addressing these gaps using both models at the abstraction level with abundant data. We hope that these insights, as well as our novel hierarchical, multimodal search algorithm may inspire solutions to problems in other domains that involves discovery using natural language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ig76mcrW4U", "reviewer": "Reviewer_SSy6", "summary": "This work leverages the powerful capabilities of large language models to propose a generative framework that transitions from natural language to material structures. The framework divides material generation into high-level semantic information, which can be expressed as chemical formulas in text, and low-level structural information, which requires atomic types and coordinates for expression. High-level semantic information is generated by the large language model based on user instructions, while the low-level structural information is completed by a conditional diffusion model. Both high-level and low-level information are optimized and ranked using empirical formulas. Finally, a property-predicting Graph Neural Network (GNN) is used to further optimize the generated material structures. The proposed model accomplishes end-to-end material generation, particularly starting from user instructions, which makes this research highly practical.", "questions": "1. Lacking domain knowledge, I am curious about the guiding capability of the chemical formula in this context. Why is it that a chemical formula generated by an LLM based on user instructions is considered so meaningful in providing guidance?\n2. Using compact crystal representations to accelerate inference is understandable, but I am still curious why the authors chose a U-Net architecture as the denoising model. Typically, in non-image domains, common backbones are GNNs or Transformers.", "limitations": "1. While generating molecules through user natural language instructions is a very exciting prospect, it seems that the role of the language model in this context is merely to generate a chemical formula. This makes it feel like the potential of the LLM is quite limited.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Leveraging the capabilities of large language models to generate materials based on user instructions is a highly imaginative approach, with significant potential for future applications.\n2. The overall structural design is comprehensive, including various component models and empirical functions.\n3. There are technical innovations, such as the use of compact crystal representations to accelerate inference.", "weaknesses": "1. The claim of \"end-to-end\" is confusing. The overall model seems more like a combination of different components (LLM, diffusion model, GNN). These components are not inherently connected and are not trained together, which seems to create a gap with the typical \"end-to-end\" meaning in the machine learning field.\n2. The work seems more like an engineering contribution. While I do not deny the significance of such work in the current LLM era, the extensive discussion on multi-objective optimization (Section 2) feels somewhat redundant. This part appears to involve simple rule-based pruning, making the emphasis on multi-objective optimization as a contribution rather unconvincing.\n3. The work lacks ablation studies. For instance, as mentioned in point 2, many rules are designed based on heuristic experience (including how to filter chemical formulas and molecular structures). I am uncertain about the actual impact of these heuristic rules and their robustness.\n4. The core comparison baseline in this paper is the ability of a generic LLM without fine-tuning to generate materials. This is evidently unfair when compared to the proposed complete framework, which includes the diffusion model. Even adding fine-tuned LLMs does not seem convincing. The authors should at least compare their work with some specialized material generation models based on diffusion models or other methods.\n5. Similar to point 1, although it claims to be end-to-end, it seems that only the diffusion model is trained. I believe there is a lack of detailed explanations regarding the LLM and the property-predicting GNN used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. Please see our response below.\n\n> The \"end-to-end\" terminology.\n\nWe agree that our usage of \"end-to-end\" is not conventional and can be misleading. \"End-to-end\" in this paper’s context refers to end-to-end automation from formulae proposal to structure generation and verification, as opposed to end-to-end modeling. We have clarified the usage of \"end-to-end\" in the paper and reduced its frequency for clarity.\n\n> The multi-objective optimization formulation\n\nThank you for recognizing this work’s contribution despite its simplicity. We believe that formally formulating the materials search problem from natural language input is important, so that many problems in materials discovery can be considered under the same framework. Under the formulation of the multi-objective optimization, rule-based pruning is only one proposed approach to solve this objective. One can also perform constrained optimization, planning, or reinforcement learning with external feedback to solve this multi-objective optimization problem, which we consider to be interesting future work.\n\n> Heuristic rules and their effectiveness\n\nWe would like to clarify that the heuristics considered in this paper are only examples of many constraints and heuristics that practitioners in materials science may want to enforce, as opposed to a set of hand-designed heuristics that work the best for materials discovery. The current heuristics, especially those in $R_\\text{hi}$, are only basic checks on instruction following. As practitioners use this framework, we anticipate additional constraints and heuristics to be enforced depending on the specific types of materials being discovered and the additional domain knowledge of the respective users. We have rewritten line 116-120 to highlight that $R_\\text{hi}$ considered in this paper are only examples of high-level heuristics that can be optimized.\n\n> Comparison to finetuned LLM and diffusion models.\n\nThe reason that we only included few-shot prompting baselines in the end-to-end evaluation of Table 1 is because, to our knowledge, there is no natural-language-to-crystal-structure dataset available for finetuning LLMs to generate crystals from loose natural language descriptions, as this work is among the first to study this problem. Hence, few-shot prompting is the most natural baseline. However, we do evaluate the subcomponents of GenMS to previous baselines, and show that the formula-conditioned generation in GenMS performs favourably compared to diffusion model [1] and finetuned language model baselines [2] in Table 5 and Table 6 of the paper. [1] and additional work on diffusion models has demonstrated better performance than GNNs ([3]), hence we did not include comparison to GNN generation.\n\n[1] Yang, et al. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235 (2023).\n\n[2] Antunes, et al. Crystal structure generation with autoregressive large language modeling. arXiv preprint arXiv:2307.04340 (2023).\n\n[3] Zeni, Claudio, et al. \"Mattergen: a generative model for inorganic materials design.\" arXiv preprint arXiv:2312.03687 (2023).\n\n> Detailed explanation of LLM and property GNN used\n\nThe LLM and GNN used are described in detail in the previous work [1] and [2]. We have included these detailed descriptions in the appendix for the final version of this paper.\n\n[1] Reid et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024\n\n[2] Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80–85, 363 2023.\n\n> Why chemical formula\n\nChemical formulas are both connected to low-level properties of structures (e.g., conductivity) and high-level semantics that a user might specify using natural language (i.e., a crystal with carbon and silicon). Hence, chemical formulas are natural intermediate information. Moreover, chemical formula exists both in high-level datasets such as textbooks, wiki articles, and research papers, as well as in low-level crystal structure databases such as Materials Project [1] and ICSD [2], making them a natural interface as LLMs and diffusion models are trained on high and low level datasets, respectively.\n\n[1] Jain, Anubhav, et al. \"Commentary: The Materials Project: A materials genome approach to accelerating materials innovation.\" APL materials 1.1 (2013).\n\n[2] Hellenbrandt, Mariette. \"The inorganic crystal structure database (ICSD)—present and future.\" Crystallography Reviews 10.1 (2004): 17-22.\n\n> Why UNet\n\nWe have experimented with different architectures including only using transformers with local residual connections as opposed to a UNet-like architecture, but these alternatives did not work as well in practice. We did not experiment with GNNs for generation, because GNNs with graphs represented by adjacency matrices scale quadratically in the number of atoms. On the other hand, the modeling complexity of GenMS only scales linearly with the number of atoms, as the compact crystal representation of GenMS does not consider edges.\n\n> Limitations of only generating chemical formula\n\nGenerating formula is in fact nontrivial, especially for complex structures such as metal-organic frameworks (MOFs), which often have long formulae. Formulae implicitly capture rich human knowledge such as charge balancing, conductivity, etc. This knowledge is implicitly captured by LLMs and elicited during the formula generation process. That being said, we believe that formulae do not have to be a hard interface between high-level semantic and low-level structure generation. Exploring alternatives like including additional textual information such as space groups are interesting future works to explore, as well as directly generating structures from loose natural language, once such a dataset becomes available for finetuning LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AyKkG2GnyJ", "reviewer": "Reviewer_DY2F", "summary": "This paper proposes a hierarchical generative system for material search, which consists of a language model to translate user input into intermediate textual representation, a diffusion model which then generates crystal structure, and finally a property predictor used for sample selection.\n\nThe works leverage data at different levels from different sources: the high-level knowledge in web texts and low-level crystal information in specialized databases. This motivates the factorization of the text-to-crystal generation process into the high-level text-to-symbolic step, and the low-level symbolic-to-crystal step. \n\nThe generation results are filtered/ranked by a heuristic module for each step. \n\nThe approach is supported with experiments on several families of material.", "questions": "it seems the baseline is few-shot, but the GenMS language prompt is zero-shot.\nWhy? it is not easy to apply few-shot on GenMS or did you find it's not helpful compared to zero-shot?", "limitations": "the language model used in this paper, as well as potential ways to improve this module (such as prompting strategies, few-shot, finetuing) are not well explored.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The proposed framework effectively utilize data from multiple data sources.\nThe collaboration of the generation modules and scoring modules in the framework is a reasonable approach and might benefit other works. The discussion on future works are also useful.", "weaknesses": "The baselines seems to be weak. If the LLMs are finetuned on CIF data, probably the baseline performance can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback of our work! Please find our response to your questions below.\n\n> LLM finetunine baseline\n\nThe reason that we only included few-shot prompting baselines in the end-to-end evaluation of Table 1 is because, to our knowledge, there is no natural-language-to-crystal-structure dataset available for finetuning LLMs to generate crystals from loose natural language descriptions, as this work is among the first to study this problem. Hence, few-shot prompting is the most natural baseline. However, we do evaluate the subcomponents of GenMS to previous baselines, and show that the formula-conditioned generation in GenMS performs favourably compared to diffusion model [1] and finetuned language model baselines [2] in Table 5 and Table 6 of the paper.\n\n[1] Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., & Cubuk, E. D. (2023). Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235.\n\n[2] Antunes, Luis M., Keith T. Butler, and Ricardo Grau-Crespo. \"Crystal structure generation with autoregressive large language modeling.\" arXiv preprint arXiv:2307.04340 (2023).\n\n> Why GenMS uses zero-shot prompting but baselines use few-shot prompting\n\nWe consider GenMS in a zero-shot prompting setting because of the practical motivation of materials discovery, i.e., structures to be discovered may not have similar existing structures to use as few-shot prompt. However, when we tried zero-shot prompting of cif files using LLMs directly, we observed close to zero percent valid cif files being generated. Therefore we had to include a few cif file examples as input to the LLMs to specify the format of the cif files.\n\n> Details of the LLM used and potential ways to improve the LLM module\n\nPer the reviewer’s suggestion, in addition to Gemini long context, we have also used GPT4 as the LLM module, and observe that GPT4 performs similarly (and occasionally better) than Gemini long context. We did not further tune the prompts, which may lead to different results. \n\n| Metrics | GEMS-Gemini | GEMS-GPT4 |\n| ----------- | ----------- | ----------- |\n| CIF validity | 1 | 1 |\n| Structural validity | 1 | 1 |\n| Composition validity | 0.85 | 0.97 |\n| Formation energy (GNN) | -0.47 | -0.95 |\n| Formation energy (DFT) | -1.32 | -1.39 |\n| Uniqueness | 0.9 | 0.9 |\n| Match rate | 0.93 | 0.95 |\n\nWe have also conducted two additional studies for RAG. First, we eliminate half of the retrieved Wiki text, and observe the match rate in Table 4 drops from 0.89 to 0.85. Second, we replace the text retrieved from Wikipedia by text retrieved from Materials Project [1] (entries that contain the keyword Perovskite), and observe that the match rate in Table 4 drops to 0.86, which is still better than no retrieval at all. We will include these additional results in the final version of the paper. We think there are many other interesting aspects of the system that can be improved, such as using research articles to further enhance RAG. We see our work as the first that calls for further contribution for language-to-structure generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduced GenMS, a framework that uses LLMs to generate novel material structures based on high-level NL input using chemical formulae. For the inference they combine LLMs with a diffusion model and domain-specific heuristics. They showcase promising results in generating structures that satisfy user requirements. This is an interesting new direction towards materials discovery and design. 

The paper includes ad-hoc decisions in their framework, making the method difficult to reproduce. Authors clarified that the ad-hoc decisions are not arbitrary, but rather examples of what practitioners can enforce in the materials search process. The authors have also conducted additional experiments and demonstrated the effectiveness of the RAG architecture and the impact of using different retrieval texts. The authors acknowledge that some level of domain knowledge is unavoidable, but it would be beneficial to provide more guidance on how to incorporate domain knowledge into the framework.

Another point that was raised is that the authors should include stronger baselines and more comprehensive evaluation results. The authors explained that the few-shot prompting baseline is the most natural choice due to the lack of a natural-language-to-crystal-structure dataset, but also they pointed to additional evaluations in the paper. The reviewer has also raised the issue for the paper that it is a combination of widely-used models, questioning whether this is enough to make significant contributions for a research paper. The authors responded by highlighting that the approach recognizes knowledge gaps and properly addresses them using a novel hierarchical, multimodal search algorithm.

The third reviewer has pointed to the paper's technical novelty and the lack of experimental evidence. The authors have provided additional evidence but the reviewer still thinks that the alignment between text prompts and generated materials still remains particularly with regard to specific properties of characteristics. 

The contributions of this paper is significant, as they pave the way for further progress in the materials discovery problem. Although the evaluations and results may not be on par with other machine learning papers in the natural sciences domain (e.g., biology), this is a novel problem to tackle with ML. 

I suggest the following minor revisions to improve the quality of the paper (which can be done easily):
-- provide more guidance on how to incorporate domain knowledge into the framework.
-- the authors provided latency measurements, but it would be good to discuss potential strategies for reducing latency.
Decision: Accept (poster)