Generative Hierarchical Materials Search

Authors: sherry  yang
simon  batzner
ruiqi  gao
muratahan  aykol
alexander  gaunt
brendan  mcmorrow
danilo  rezende
dale  schuurmans
igor  mordatch
ekin  dogus
cubuk  google deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:42:23

Abstract: Generative models trained at scale can now produce novel text, video, and more recently, scientific data such as crystal structures. The ultimate goal for materials discovery, however, goes beyond generation: we desire a fully automated system that proposes, generates, and verifies crystal structures given a high-level user instruction. In this work, we formulate end-to-end language-to-structure generation as a multi-objective optimization problem, and propose Generative Hierarchical Materials Search (GenMS) for controllable generation of crystal structures. GenMS consists of (1) a language model that takes high-level natural language as input and generates intermediate textual information about a crystal (e.g., chemical formulae), and (2) a diffusion model that takes intermediate information as input and generates low-level continuous value crystal structures. GenMS additionally uses a graph neural network to predict properties (e.g., formation energy) from the generated crystal structures. During inference, GenMS leverages all three components to conduct a forward tree search over the space of possible structures. Experiments show that GenMS outperforms other alternatives both in satisfying user request and in generating low-energy structures. GenMS is able to generate complex structures such as double perovskites (or elpasolites), layered structures, and spinels, solely from natural language input.

Main paper: 
 introduction: 
 Modern technologies increasingly rely on the development of materials, such as semiconductors [1] , solar cells [2] , and lithium batteries [3] . Large-scale generative models, trained on expansive internet data, exhibit intriguing generalization capabilities. For example, these models can synthesize a highly realistic image of "an astronaut riding a horse" by merging two distant concepts [4] . This raises a compelling question: can the generalization capabilities of large generative models, pretrained on existing materials science knowledge, be harnessed to combine knowledge from existing materials systems to propose candidate crystals? Previous research has demonstrated that generative models can output crystal structures that are not in the the training data [5, 6, 7] . However, these works typically require either a vast number of unconditional samples to generate an unknown material [5, 8] or a chemical formula provided during inference [6, 9] . It is difficult for end users to come up with new chemical formulae, as it is hard to know which compositions will result in what material properties. Therefore, it is highly desirable to develop an interface that allows users to describe the desired characteristics of crystal structuressuch as properties, compositions, space groups, and geometric characteristics -in natural language. For example, a user might specify "a stable chalcogenide with atom ratio 1:1:2 that is not on ICSD." Ideally, a model should automatically interpret these high-level language instructions to search for, GenMS takes a high-level language instruction as input, retrieves relevant information from the internet, and samples from a high-level LLM (π hi ) to generate candidate formulae that satisfy user requirement. GenMS then samples from a low-level diffusion model (π lo ) to generate structures conditioned on candidate formulae. Sampled structures then go through a property prediction module for selection. generate, and validate a wide range of potential structures, ultimately producing one that best meets the user's specifications.However, developing an end-to-end language-to-structure generative model presents several challenges, for which we make a few key observations. First, there are no existing labeled datasets that map language descriptions directly to crystal structures. Nevertheless, we observe that there is a wealth of language-to-formula data available online, including Wikipedia articles, research papers, and textbooks. This data can be complemented by formula-to-structure information from specialized materials databases such as the Materials Project [10] , ICSD [11] , OQMD [12] , etc. Second, the task of converting language into structures is inherently multimodal, requiring the transformation of discrete linguistic inputs to continuous structural outputs. Nevertheless, it has been shown that semantic-level autoregressive models combined with low-level (pixel-level) diffusion models are effective for cross-modal generation, such as in text-to-video applications [13, 14] . Lastly, user descriptions of desired crystal structures can often be vague -users may not articulate all relevant details about the crystal they wish to generate. We observe that one can leverage generative models to infer missing information, and rely on additional search and selection mechanisms to identify structures that best satisfy a user's requirement.Based on these observations, we propose Generative Hierarchical Materials Search (GenMS) for end-to-end language-to-structure generation. GenMS consists of (1) a large language model (LLM) pretrained on high-level materials science knowledge from the internet, (2) a diffusion model trained on low-level crystal structure databases, and (3) a graph neural network (GNN) for property prediction. To improve the efficiency of (2), GenMS proposes a compact representation of crystal structures for diffusion models. During inference, GenMS prompts the LLM to generate candidate chemical formulae according to user specification, samples structures from the diffusion model, and uses the GNN to predict the properties of the sampled structures. To sample structures that best satisfy user requirements during inference, we formulate language-to-structure as a multi-objective optimization problem, where user specifications are transformed into objectives that can be optimized at both the formula and structure level.We first evaluate GenMS's ability to generate crystal structures from language instructions, and find that GenMS can successfully generate structures that satisfy user requests more than 80% of the time for three major families of structures, while proposing structures with low formation energies, as verified by DFT calculations. In contrast, using pretrained LLMs to directly generate crystal structures from user instructions in a zero-shot manner often results in close to a 0% success rate. Qualitative evaluations show that GenMS is able to generate complex structures, such as layered structures, double perovskites, and spinels, solely from natural language. We next study the effect of each individual component of GenMS. Here we find that language instructions have a significant impact on the structures generated, that the novel compact representation of crystals proposed by GenMS improves the DFT convergence rate of diffusion generated crystal structures by 50% over previous work, and that using a pretrained GNN to select samples leads to lower energy structures more than 80% of the time. Given such experimental evidence, we believe the development of language-to-structure models are promising for enabling users to find viable crystal structure candidates, complementing existing databases in utilityQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We provided thorough experiments and explanations of the algorithms to jusify the claim.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
generative hierarchical materials search: 
 We begin by formulating the problem of generating crystal structures from high-level language as a multi-objective optimization task. Given this formulation, we then propose a hierarchical, multimodal tree search algorithm that leverages language models, diffusion models, and graph neural networks as submodules. Lastly, we discuss the specific design choices for each of the submodules.Given some high-level language description g ∈ G of desired structures, we want to learn a conditional crystal structure generator π(•|g) : G → ∆(X )2 that can be used to sample crystal structures x ∈ X conditioned on language. One option is to parametrize π with a pretrained LLM. However, pretrained LLMs alone are not able to predict sufficiently accurate crystal structures, due to the lack of low-level structural information about crystals (e.g., 3D atom coordinates) in the pretraining data.If we had access to a paired language-to-structure dataset, D = {g i , x i } N i=1 , π could be trained using a maximum likelihood objective. However, materials data naturally exist at different levels of abstraction and are segregated into different sources: high-level symbolic knowledge is documented in sources like Wikipedia articles, research papers, and textbooks, whereas detailed low-level crystal information, including continuous-valued atom positions, is stored in specialized crystal databases like the Materials Project [10] and ICSD [11] . Even though a direct language-to-structure dataset D remains unavailable, the pretraining data for LLMs, including Wikipedia articles, research papers, and textbooks, can be viewed as a high-level symbolic datasetD hi = {g i , z i } m i=1, where z ∈ Z denotes symbolic textual information such as chemical formulae. Meanwhile, many crystal databases already feature paired data, D lo = {z i , x i } n i=1 , linking chemical formulae to detailed crystal structures. Given this observation, we propose to factorize the crystal generator as π = π hi • π lo , where π hi : G → ∆(Z) and π lo : Z → ∆(X ), so that π hi and π lo can be trained using different datasets D hi and D lo . Furthermore, we consider two heuristic functions, R hi (g, z) : G × Z → R and R lo (z, x) : Z × X → R, where the high-level heuristic function R hi can be used to select formulae that satisfy the language input at a high level, and the low-level heuristic function R lo can be used to select structures that are both valid and exhibit desirable properties such as low formation energy. To this end, we propose to search for crystal structure given language input by finding a chemical formula / space group z with a corresponding crystal structure x that jointly optimizeEQUATIONwhere λ hi and λ lo are hyperparameters to control how much weight to put on high and low-level heuristics. Note that R hi and R lo can also be combinations of multiple objectives. For instance, R hi can be a weighted sum of instruction following and simplicity, where R lo can be a weighted sum of properties such as band gap, conductivity, and formation energy.Given the objective in Equation 1, it is clear that a pretrained LLM (even with finetuning) is insufficient to optimize for the best structure x * . Instead, we propose to first sample a set of intermediate chemical formulae from a pretrained LLM π hi (g) conditioned on language input g. We then use the high-level heuristic function R hi to prune and rank the intermediate formulae. In practice, R hi is a combination of (i) a regular expression checker (to ensure sampled formulae are valid chemical formulae), (ii) a uniqueness checker against formulae from existing crystal datasets such as Materials Project and ICSD, and (iii) a formula compliance checker to ensure the sampled formulae are compatible with user request (e.g., atom ratio 113 for perovskites, 227 for pyrochlore, and 124 for spinel). For formulae that pass these checks, we prompt a pretrained LLM as R hi to rank the formulae by how likely they are to comply with the user request g. We then select the top W ranked formulae to generate L crystal structures each using π lo parametrized by a diffusion model, and use a graph neural network R lo to rank the W × L structures by their predicted formation energy. Note that additional checkers can be integrated in R lo , such as structural and compositional validity defined in [5] . We illustrate the overall search procedure in Algorithm 1. Prevent heuristic exploitation. One concern of using a heuristic GNN to select structures with the lowest formation energy is that the GNN might exploit irregularities in the predicted structures, especially when a predicted structure lies outside of the training manifold of the energy GNN. To mitigate this issue, we use the GNN pretrained by [15] on DFT energies and forces of unrelaxed structures (hence the GNN has seen more irregular structures prior to relaxation.) Furthermore, we discard sampled structures from π lo if they result in energy predictions from R lo that lie outside of a threshold range.Since controllable crystal structure generation from language input is multimodal by nature, there are various design choices for the parametrization of the submodules in Equation 1, namely the generators π hi , π lo and the heuristic functions R hi , R lo . In this section, we discuss the parametrization choices we have found to be the most effective.Retrieval augmentation and long-context deduplication. One important recent advance in LLMs is increased context length [16] . The factorization π = π hi • π lo provides a natural way to integrate additional context in π hi via long-conext generation. Specifically, we further factorize π hi intoπ hi = π retrival hi • π RAG hi , where π retrival hi (•|g) is a deterministic retrieval function that uses the Wikipedia API to retrieve textual information related to language input g, while π RAG hi is a retrieval augmented generative (RAG) model that proposes chemical formulae and space groups conditioned on the information retrieved from the internet. Another use case for long-context LLMs is to further encourage the generation of new compositions by providing the formulae for all known crystals in the context, then asking π hi to produce a formula that is not in the context. As we will see in Section 3.2, this drastically improves the efficiency of the search, as a large subset of the search space with known crystals can be eliminated.In order to support efficient tree search at inference time, we need to ensure that sampling from both π hi and π lo are efficient. Previous work on diffusion models for crystal structure generation has leveraged sparse data structures, such as voxel images [17, 18, 19] , graphs [5] , and periodic table shaped tensors [6] . These existing representations of crystals incur computational overhead due to sparsity (voxel images, padded tensors) or quadratic complexity as the number of atoms in the system increases (graphs). Instead, we propose a new compact representation of crystal structures, where each crystal x ∈ X ⊂ R A×4 is represented by a A × 4 tensor, with A being the number of atoms in the crystal, and the inner 4 dimensions representing the x, y, z location of an atom along with its atom number. Here we directly represent the atom number as a continuous value normalized to the range of the input in the diffusion model to further improve inference speed, as opposed to representing the atom number using a one-hot vector. In addition, we use another 2 × 3 vector to represent the lattice structure (i.e., angles and lengths of the unit cell). Figure 2 illustrates the architecture for the diffusion model with compact crystal representations, where each atom undergoes multi-layer perceptron (MLP) followed by order-invariant self-attention (without positional encoding) across atoms. Different from typical U-Net architecture for image generation, there is no downsampling or upsampling passes that change the input resolution. Nevertheless, we follow the concatenation of skip connections commonly used in U-Net architectures [20] . Additional details and hyperparameters for the diffusion model can be found in Appendix A.3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed that in the conclusion section and we pointed out four major limitations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental evaluation: 
 We now evaluate the ability of GenMS to generate crystal structures from high-level language descriptions. First, we evaluate the success of end-to-end generation in Section 3.1. We then investigate the individual components of GenMS in Section 3.2. See details of experimental setups in Appendix A.Baselines and metrics. We aim to evaluate GenMS's ability to generate unique, valid, and potentially stable crystal structures from well-known crystal families that satisfy high-level language specifications. We consider few-shot prompting of LLMs to generate crystal information files (CIF) as a baseline. Specifically, we give the Gemini long context model [16] a number of CIF files from a particular crystal family, as specified by language input as prompt, with the number of CIF files ranging from 1, 5, 25 to as many as can fit in the context. We ask the LLM to generate 100 samples given each language instruction. See additional details of baselines in Appendix A.2. We do not compare to finetuning LLMs to generate CIF files in this section, as there are no high-level language to low-level crystal structure datasets available for finetuning such an instruction following LLM. Nevertheless, we will compare the diffusion model in GenMS to formula-conditioned structure generation using finetuned LLM in Section 3.2. We consider language input that directs the model to generate unique and stable crystals from a particular crystal family (perovskite, pyrochlore, and spinel). We consider the following metrics for evaluation: (i) CIF validity, which measures whether the generated CIF file can be properly parsed by pymatgen parser [21] . (ii) Structural and composition validity, which verify atom distances and charge balances using SMACT [22] , following [5] . (iii) Formation energy per atom (E f ) in the unit of eV/atom, which measures the stability of predicted structures using a pretrained GNN. We further conduct DFT calculations to compute E f (see details in Appendix A.4) for structures predicted by GenMS. (iv) Uniqueness, which measures the percentage of generated formulae that do not exist in Materials Project [10] or ICSD [11] . Finally, (v) the match rate, which measures the percentage of generated structures that can be matched (according to the pymatgen 1 : End-to-end evaluation of generating crystal structure from natural language. GenMS significantly outperforms LLM prompting baselines in producing unique and low-energy (predicted by GNN) structures that satisfy user request. We further conduct DFT calculation to compute E f (formation energy in eV/atom) averaged across structures generated by GenMS. Values before "/" in the row "(GNN/DFT)" represent GNN predicted E f , and after "/" represent DFT computed E f . We report E f from GNN prior to relaxation, and E f from DFT post relaxation. DFT calculations for baselines are eliminated as many structures from the baselines do not follow user instruction. N/A represents E f predicted by GNN falling outside of the reasonable range.structure matcher) to one of the structures of the corresponding family in Materials Project. More details of these metrics can be found in Appendix A.1.Results on specifying crystal family. The evaluation of GenMS and baselines are shown in Table 1. Since GenMS does not rely on an LLM to directly generate CIF files, the compact crystal representation (described in Section 2.3) always results in structures that can be parsed by pymatgen (100% CIF validity). In addition, structures generated by GenMS have a much higher validity and match rate compared to those generated by the baselines. GenMS struggles slightly with uniqueness, as less than half of the generated formulae for pyrochlore and spinel are unique with respect to MP and ICSD. Structures produced by GenMS have lower average E f . Increasing the number of CIF files in the context generally improves the performance of the baselines (1, 5, and 25-shot), but including too many files in the context can hurt performance (Prompting CIF Max).Qualitative evaluation. In addition to the three families of structures evaluated above, we qualitatively evaluated GenMS's ability to generate structures that satisfy ad hoc user requests, such as "a layered material", "an elpasolite", and so on. GenMS can consistently produce structures that satisfy user request as shown in Figure 3 , and have plausible initial geometries. Interestingly, we observe that GenMS can understand semantic-level request, suggesting more "fluoride" like chemistries when asked for "elpasolite", which is reasonable as elpasolite is associated with the mineral K2NaAlF6.Effect of search. Next, we aimed to understand the effect of search in GenMS, especially in producing low-energy structures. For each of the family of crystals in Table 1 , we analyzed the effect of the language and structure branching factors (H and L in Algorithm 1). Only crystals that match input specification were considered for energy computation. We found that increasing the branching factor of both language and structure enables GenMS to generate structures with lower formation energies (at a higher inference cost).Double perovskite Pryochlore SpinelFigure 3 : Qualitative evaluation. We test GenMS on a set of ad hoc language inputs to generate plausible examples from well-known crystal families. GenMS is able to search for the corresponding structures that satisfy user requests and have plausible initial geometries. Visualization provided by VESTA [23] .Perovskite Pyrochlore SpinelLanguage Structure E f Language Structure E f Language Structure E f branch (H) branch (L) (DFT) branch (H) branch (L) (DFT) branch (H) branch (L) (DFT) 1 1 -0.Next, we evaluate the individual component of GenMS, including the effect of using language to narrow down the search space, the choice of the compact representation of crystal structures, and finally the best-of-N sampling strategy for choosing the crystal structures with low formation energy.Effect of language. We want to understand whether GenMS can provide effective control over formulae proposed by the LLM at the semantic-level through natural language. In Table 3 , we first show that requesting a particular element to be in the formula always results in formulas with that particular element being proposed by the pretrained LLM π hi . We then show that when a user requests for metal, the model is 4 times more likely to generate formulae for metal. The model also respects a user's request for the generated formulae to be unique (with respect to either a user provided list of known formulae in the context of the LLM, or the name of some crystal database).Next, we study the effect of retrieval augmented generation (RAG). We use GenMS with and without RAG to propose 25 formulae for each of the three major crystal families from Section 3.1 and generates 4 structures per formula using the diffusion model. We report the rate of valid formulae proposed by the LLM and the structures that can be matched with existing structures from the corresponding family in Compact crystal representation. We now evaluate the diffusion model π lo trained using the compact representation of crystals structures described in Section 2.3. We compare diffusion model with compact crystal representation against two prior work for generating crystal structures conditioned on composition. UniMat [6] proposed a periodic table representation of crystals which requires a large amount of paddings to handle atoms that do not exist in the structure. CrystalLM [9] proposes to finetune an LLM to directly generate CIF files from input compositions. In Table 5 , we report the DFT convergence rate and DFT calculated E f on a set of holdout structures following [6] . We observe that the compact crystal representation results in both higher convergence rate and lower E f than the sparse representation in [6] . To compare GenMS's diffusion model against finetuning LLMs to generate CIF files directly, we follow the experimental setting of CrystaLLM where we train a composition conditioned diffusion model on a combination of Materials Project [10] , OQMD [24] , and NOMAD [25] , and test the success rate of generating matching structures for unseen compositions following [9] . In Table 6 , we see that GenMS has significantly higher rate in producing a valid crystal and a crystal that can be matched to the test set in [9] .Best-of-N structure sampling. To better understand the effect of high structure branching factor in Algorithm 1 across different compositions, we measure the difference in the formation energy, using a holdout test set of 1,000 compositions, between using the energy prediction GNN to select the best of 10 samples compared to only predicting a single structure. The energy difference with and without best-of-N sampling is shown in Figure 4 . Using best-of-N with N = 10 results in improved energy for over 80% of structures (as also verified by DFT calculations). We found the energy prediction GNN to be a good indicator of the true energy of the crystal structures, i.e., the GNN predicted energy difference (left) and the DFT calculated energy difference (right) are very similar in Figure 4 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The data and prompts are all public and we have provided the information. We are working on open sourcing the code after going through the internal approval process.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Hierarchical and latent image and video generation. Image and video generative models have exhibited an impressive ability to synthesize photorealistic images or videos when given text description as input. Many of the state-of-the-art models adopt a hierarchical modeling approach that inspired the design of with GenMS. For example, latent diffusion models [26, 27] contains (1) a language model that converts text to high-level text embeddings, (2) a diffusion model takes the text embeddings as input and output latents in a compressed latent space, and (3) a feed forward decoder network [26] or a diffusion decoder [28, 14] that given the generated latents generates full-resolution signals in the pixel space. Cascaded diffusion models [29, 30, 31] instead proposed to generate signals at the lowest resolution with a standard diffusion model, followed by a few super-resolution models that successively upsample signals and add high-resolution details. Similar to GenMS, by breaking down complicated image or video generation into a hierarchy of less challenging problems, these models can generate high quality samples more efficiently and effectively.Generative models for crystal structures. A number of works [9, 8, 32] have proposed to train or fine-tune language models to generate output files containing crystal information or low-level atom positions. However, it remains expensive and challenging to train and generate detailed structural information with LLMs. On the other hand, diffusion models, as a powerful class of generative model in vision, have been applied to generate crystal structures [5, 7, 6] . However these methods either reply on training with a large set of unconditional samples and brute-force sampling for new materials not in the training set, or necessitate predetermined compositions as conditioning information during inference. Handling of candidate structure generation requires a model capable of independent reasoning about chemical compositions based on high-level user specifications and structure optimization, as done in GenMS.Hierarchical search and planning. The problem of learning to generate low-level continuous output from high-level language instructions, while employing intermediate search and planning steps, has been studied in other domains such as continuous control [33] , self-driving [34] , and robotics [35] .While some works have focused on purely using LLMs to search and plan through complex output spaces [36, 37] , other research has shown that solely relying on LLMs to search and plan can fail short due to the lack of low-level information (e.g., locations, precise motions) captured in the model [38] .Recently, video generation models have been applied to provide additional details about the physical world so that low-level control actions can be extracted more accurately [39, 40, 41, 42] . GenMS follows a similar approach but focuses on generating crstyal structures, using diffusion models on top of LLMs to provide additional details about crystal structure, enabling high-level plans (i.e., symbolic chemical formulae) to be verified at a low-level (i.e., crystal structures with precise atom locations).Large language models for science. Recently, there has been a surge of interest in applying large langauge models in domains of science, such as physics [43] , biology [44] , chemistry [45, 46] , and materials science [47] . In these settings, LLMs generally serve as a conversational [44] or educational [48] tool, where LLMs output natural language to be consumed by human users (e.g., an answer to a scientic question asking about the property of some existing crystal structure). On the other hand, we are interested in the ability of a pretrained LLM to propose intermediate textual information such as chemical formulae for interesting crystal structures. Closest to our work are [49, 50] which leverage an LLM to generate SMILES or other chemical strings for molecular design. Nevertheless, we are interested in generating not just the formulae, but the actual crystal structures with continuous-valued atom locations, as many materials property can only be calculated and verified once the full structure available. 
conclusion and future work: 
 We have introduced GenMS, an initial attempt at enabling end-to-end generation of candidate crystal structures that look physically viable and satisfy instructions expressed in natural language. GenMS can generate examples from families such as pyrochlores and spinels purely from natural language prompts. We hope the design principles of GenMS will initiate broad interest in exploiting language as a natural interface for flexible design and generation of crystal structures that meet user-specified criteria, and enable the domain experts to work more efficiently. GenMS has a few limitations that call for future work:• Generating complex structures. While GenMS is able to generate simple structures such as those shown in Figure 3 , we found that GenMS is less effective in generating complex structures such as Mxenes and Kagome lattices. Controllable generation of highly complex crystal structures is an interesting area of future work.• Impact on experimental exploration. While we have shown that GenMS is effective in generating crystal structures that are not in public databases and that satisfy user requirements, its effectiveness in suggesting specific materials with target properties (e.g., battery electrodes or electrolytes, semiconductors, superconductors etc.) requires further experimental verification.• Synthesizability. While the goal of GenMS is to provide an end-to-end generative framework from natural language instructions to realistic crystal structures, synthesizability of the generated crystals is not currently part of the pipeline. We foresee development in multimodal models and integration of other computational tools from materials science to allow predicted structures to be assessed for synthesizability. • Extension to other chemical systems. We have shown that GenMS can effectively generate crystal structures from natural language. We note that GenMS can also potentially be extended to generating molecules and protein structures from natural language (e.g. "generate a protein with an alpha-helix"). We leave these explorations for future work.NeurIPS Paper Checklist 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided the information in the appendix and main paper. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We provided the variance of the results in almost all tables when applicable. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No risk as far as we can see.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited the models (Gemini) we used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This work is exploratory and is at a too early of a stage to have broad impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We use ∆(•) to denote a probability simplex function. 


Decision: This paper introduced GenMS, a framework that uses LLMs to generate novel material structures based on high-level NL input using chemical formulae. For the inference they combine LLMs with a diffusion model and domain-specific heuristics. They showcase promising results in generating structures that satisfy user requirements. This is an interesting new direction towards materials discovery and design. 

The paper includes ad-hoc decisions in their framework, making the method difficult to reproduce. Authors clarified that the ad-hoc decisions are not arbitrary, but rather examples of what practitioners can enforce in the materials search process. The authors have also conducted additional experiments and demonstrated the effectiveness of the RAG architecture and the impact of using different retrieval texts. The authors acknowledge that some level of domain knowledge is unavoidable, but it would be beneficial to provide more guidance on how to incorporate domain knowledge into the framework.

Another point that was raised is that the authors should include stronger baselines and more comprehensive evaluation results. The authors explained that the few-shot prompting baseline is the most natural choice due to the lack of a natural-language-to-crystal-structure dataset, but also they pointed to additional evaluations in the paper. The reviewer has also raised the issue for the paper that it is a combination of widely-used models, questioning whether this is enough to make significant contributions for a research paper. The authors responded by highlighting that the approach recognizes knowledge gaps and properly addresses them using a novel hierarchical, multimodal search algorithm.

The third reviewer has pointed to the paper's technical novelty and the lack of experimental evidence. The authors have provided additional evidence but the reviewer still thinks that the alignment between text prompts and generated materials still remains particularly with regard to specific properties of characteristics. 

The contributions of this paper is significant, as they pave the way for further progress in the materials discovery problem. Although the evaluations and results may not be on par with other machine learning papers in the natural sciences domain (e.g., biology), this is a novel problem to tackle with ML. 

I suggest the following minor revisions to improve the quality of the paper (which can be done easily):
-- provide more guidance on how to incorporate domain knowledge into the framework.
-- the authors provided latency measurements, but it would be good to discuss potential strategies for reducing latency.

Reviews: review: paper_summary: paper_summary: This paper presents Generative Hierarchical Materials Search (GenMS), a novel framework for material search using LLMs. GenMS consists of a language model that takes high-level natural language as input and generates intermediate textual information about crystals (e.g., chemical formulae). Specifically, it uses a RAG architecture to make an LLM incorporate information from Wikipedia articles and other retrieved articles from the Internet. The generated formulae are filtered and re-ranked by a heuristic function involving domain knowledge, such as a formulae regular expression checker. A diffusion model then generates material structures conditioned on the filtered formulae. Experimental results show that GenMS is a solid framework that can generate structures satisfying users' requirements, demonstrating that using LLMs for structure search is a promising direction.
strength_weakness: strength_weakness: 1. The architecture involves several steps requiring ad-hoc decisions, such as using Wiki-API for information retrieval or heuristic operations for selecting formulae. For example, lines 117-120 describe ad-hoc operations to prune intermediate formulae, making the method hard to reproduce and requiring more human effort. There might be more elegant, machine-learning-based ways to incorporate these operations, such as using LLMs for formula checking.
2. The heuristic function design involves significant domain knowledge, requiring more ablation studies or analysis to justify the heuristic choices.
questions: questions: 1. Considering the complexity of the whole system, what is the average inference latency for each query?
2. How does using Wiki-API affect performance? Is there any ablation study on combining Wiki-API and RAG differently?
limitations: limitations: The design of the heuristic function requires a lot of domain knowledge, making the method difficult to reproduce.
paper_summary: paper_summary: The paper presents a language-to-structure generation model for crystal structures. They propose a hierarchical approach that uses a language model to generate intermediate textual crystal information and then generate low-level crystal structures using diffusion models. They demonstrate their ability to generate crystal structures from language instructions and compare it to baselines. Results show that GenMS outperforms the baselines in generating valid and low-energy structures that satisfy user requests.
strength_weakness: strength_weakness: 1. The baseline is relatively weak as it is only a few-shot prompted LLM. For example, the ablation of each of their proposed modules (e.g., LLMs, retrieval) should be treated as baselines and the results should be put in the main table with more comprehensive evaluation results than the current ones.
2. The overall workflow is a combination of several widely-used models (e.g., LLMs, diffusion models, retrieval models). The pipeline makes sense but combining these models may not be enough to make significant contributions for a research paper.
questions: questions: Can you add stronger baselines and show more comprehensive evaluation results?
limitations: limitations: the authors adequately addressed the limitations
paper_summary: paper_summary: This work leverages the powerful capabilities of large language models to propose a generative framework that transitions from natural language to material structures. The framework divides material generation into high-level semantic information, which can be expressed as chemical formulas in text, and low-level structural information, which requires atomic types and coordinates for expression. High-level semantic information is generated by the large language model based on user instructions, while the low-level structural information is completed by a conditional diffusion model. Both high-level and low-level information are optimized and ranked using empirical formulas. Finally, a property-predicting Graph Neural Network (GNN) is used to further optimize the generated material structures. The proposed model accomplishes end-to-end material generation, particularly starting from user instructions, which makes this research highly practical.
strength_weakness: strength_weakness: 1. The claim of "end-to-end" is confusing. The overall model seems more like a combination of different components (LLM, diffusion model, GNN). These components are not inherently connected and are not trained together, which seems to create a gap with the typical "end-to-end" meaning in the machine learning field.
2. The work seems more like an engineering contribution. While I do not deny the significance of such work in the current LLM era, the extensive discussion on multi-objective optimization (Section 2) feels somewhat redundant. This part appears to involve simple rule-based pruning, making the emphasis on multi-objective optimization as a contribution rather unconvincing.
3. The work lacks ablation studies. For instance, as mentioned in point 2, many rules are designed based on heuristic experience (including how to filter chemical formulas and molecular structures). I am uncertain about the actual impact of these heuristic rules and their robustness.
4. The core comparison baseline in this paper is the ability of a generic LLM without fine-tuning to generate materials. This is evidently unfair when compared to the proposed complete framework, which includes the diffusion model. Even adding fine-tuned LLMs does not seem convincing. The authors should at least compare their work with some specialized material generation models based on diffusion models or other methods.
5. Similar to point 1, although it claims to be end-to-end, it seems that only the diffusion model is trained. I believe there is a lack of detailed explanations regarding the LLM and the property-predicting GNN used.
questions: questions: 1. Lacking domain knowledge, I am curious about the guiding capability of the chemical formula in this context. Why is it that a chemical formula generated by an LLM based on user instructions is considered so meaningful in providing guidance?
2. Using compact crystal representations to accelerate inference is understandable, but I am still curious why the authors chose a U-Net architecture as the denoising model. Typically, in non-image domains, common backbones are GNNs or Transformers.
limitations: limitations: 1. While generating molecules through user natural language instructions is a very exciting prospect, it seems that the role of the language model in this context is merely to generate a chemical formula. This makes it feel like the potential of the LLM is quite limited.
paper_summary: paper_summary: This paper proposes a hierarchical generative system for material search, which consists of a language model to translate user input into intermediate textual representation, a diffusion model which then generates crystal structure, and finally a property predictor used for sample selection.

The works leverage data at different levels from different sources: the high-level knowledge in web texts and low-level crystal information in specialized databases. This motivates the factorization of the text-to-crystal generation process into the high-level text-to-symbolic step, and the low-level symbolic-to-crystal step. 

The generation results are filtered/ranked by a heuristic module for each step. 

The approach is supported with experiments on several families of material.
strength_weakness: strength_weakness: The baselines seems to be weak. If the LLMs are finetuned on CIF data, probably the baseline performance can be improved.
questions: questions: it seems the baseline is few-shot, but the GenMS language prompt is zero-shot.
Why? it is not easy to apply few-shot on GenMS or did you find it's not helpful compared to zero-shot?
limitations: limitations: the language model used in this paper, as well as potential ways to improve this module (such as prompting strategies, few-shot, finetuing) are not well explored.

Comments: title: Thank you for the response
Author follow-up
Author follow-up
Reply to Reviewer SSy6
comment: Thank you for the response!

I've read all the responses and the other reviews. The response does not address my concern about the technical contributions of the paper, as the overall workflow still seems like a combination of several widely-used models. However, because I am not an expert in this field and this work may be interesting to researchers in this particular domain, I would keep my relatively positive rating as is.
Thanks for the response. After carefully reviewing the detailed rebuttal provided by the authors, I have decided to raise my score to a borderline accept. I acknowledge that this paper may indeed represent the first attempt to accomplish a text-to-material generation task. 

The experimental results, while showing potential, demonstrate the ability to capture higher-level semantics through text and subsequently generate materials that meet certain abstract properties at a lower level. 

However, I must clarify that I lack sufficient domain knowledge in this area, which limits my ability to objectively assess the solidity of the experiments. Additionally, I perceive this work as more of an engineering effort that has constructed a preliminary but promising integrated system, rather than offering significant technical novelty. As a result, I am lowering my confidence score by 1.
Dear Reviewer,

We would like to ask if your questions and concerns have been addressed by our response. Please let us know, and thank you for your time.
Thanks for your response! My concerns/questions have been addressed.
Dear Reviewer,

We would like to ask if your questions and concerns have been addressed by our response, and if there is anything else preventing you from increasing your score. Please let us know, and thank you for your time.
Thank you for engaging in discussion! First, we would like to highlight that we have indeed focused on evaluating the alignment between text prompt and generated materials in Table 1, Table 3, Table 4, and Figure 3, including evaluating alignment of properties and characteristics. Specifically, the “Match rate” metric in Table 1 measures the geometric alignment between generated structures and known structures from the family specified by the language input. In Table 3, we showcase that language prompts can flexibly control elements contained and conductivity, as well as being used to encourage uniqueness. In Table 4, the “Match rate” again measures alignment of geometric properties between language prompt and generated structures. In Figure 3, we showcased the alignment of language and material while considering prompts that imply chemical elements (Elpasolite implies sodium potassium aluminum fluoride) and geometric properties (double perovskite and layered material). We would like to emphasize that measurements of geometric alignment are highly reflective of property alignment, as chemical properties are often determined by geometry of crystals structures.

Second, while chemical properties might seem too low-level to be controlled by language, there are often high-level patterns of properties that can benefit from language control. For instance, conductivity is determined by band gap (a low-level property), but metallic elements are more likely to lead to conductive materials, while semimetals (e.g., boron, silicon, germanium) may conduct depending on what other elements are in the structure. Such high-level knowledge often exists in webpages and research papers, and language models can utilize these knowledge to suggest appropriate formulas.

Third, having domain experts directly design chemical formulas can be difficult, because certain properties and characteristics such as formation energy are often determined by low-level structures, which is exactly where GenMS shines in its ability to integrate high and low level information.
Thanks for the rebuttal. I appreciate the authors' efforts in addressing the concerns raised. However, I still have some reservations about this paper. First and foremost, I would like to acknowledge the contribution this work makes to the field of text-to-material generation. However, given that the core of this paper is about demonstrating a comprehensive system for generating materials from text, I believe more emphasis should have been placed on showcasing the potential of this process.

In particular, the paper seems to lack sufficient experimental evidence that demonstrates the alignment between text prompts and the generated materials, especially with regard to specific properties or characteristics. The only relevant experiment I observed was the one regarding limestone, which primarily demonstrated that the generated materials contained a specific element and metal as specified by the text input.

I am not an expert in the field, so my perspective might be limited, but I am struggling to understand the advantage of using text prompts to control such low-level information, as opposed to having domain experts directly design chemical formulas that can then be used by a diffusion model to generate materials.

In conclusion, I believe that this paper does not make a particularly strong contribution in terms of technical novelty, and it also falls short in showcasing the true potential of text-to-material generation (as most experiments seem to focus on the reasonableness and novelty of the generated materials, with little emphasis on how well they align with the original text prompts).