LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval

## Abstract

Abstract Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences where reproducibility is crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc , and inevitably biased task to fine-tune them on domain-specific literature and data. Here we introduce LLaMP 1 1 1 Code and live demo will be available at https://github.com/chiang-yuan/llamp . , a multimodal retrieval-augmented generation (RAG) framework of multiple data-aware reasoning-and-acting (ReAct) agents that dynamically interact with computational and experimental data on Materials Project (MP) . Without fine-tuning, LLaMP demonstrates an ability to comprehend and integrate various modalities of materials science concepts, fetch relevant data stores on the fly, process higher-order data (such as crystal structures and elastic tensors), and summarize multi-step procedures for solid-state synthesis. We show that LLaMP effectively shifts the predictions or corrects the errors of GPT-3.5 intrinsic knowledge, counteracting a 5.21% MAPE of GPT-3.5 on frequently-documented bandgaps and a significant 1103.54% MAPE on formation energies—errors that GPT-3.5 seems to derive from mixed data sources. Additionally, LLaMP substantially reduces the hallucinated volumetric strain in a diamond cubic silicon structure from 66.3% to 0. The proposed framework offers an intuitive and nearly hallucination-free approach to exploring materials informatics and establishes a pathway for knowledge distillation and fine-tuning other language models. We envision the framework as a valuable component for scientific hypotheses and a foundation for future autonomous laboratories where multiple LLM agents communicate and cooperate with robotics to drive material synthesis and chemical reactions without hard-coded human logic and intervention.

## 1 Introduction

On the recent development of large language models (LLMs), the generation of convincing yet unreliable information poses a pressing challenge. LLMs are prone to hallucination - providing misleading or outright false information with seemingly high confidence (Bang et al., ). This issue is particularly concerning for knowledge-intensive tasks, where users rely on chatbots and other AI systems to provide accurate guidance Lewis et al. (). Recent events have underscored how LLMs often lack up-to-date factual knowledge on topics outside their training data, necessitating vigorous verification against trusted external sources (Mallen et al., ).

The challenge extends to the archive and retrieval of factual knowledge. Much like the alchemists of old, scientists nowadays also grapple with a complex amalgamation of insights and inaccuracies. A notable example is the whirlwind surrounding copper-substituted lead phosphate apatite (LK-99) as a room-temperature ambient superconductor (Laperche et al., ; Lee et al., ; ; Griffin, ). The scenario underscores the pressing need to validate information effectively and efficiently over structured and unstructured data. Furthermore, in the era marked by the escalating capabilities of generative models, the deployment of untestable and irreproducible LLMs for high-stakes tasks, particularly in scientific and engineering applications, is on the rise. This trend accentuates the importance of scrutinizing and ensuring the reliability of information sources.

Many of the current LLM applications for domain-specific knowledge rely on the fine-tuning of pre-trained LLMs (Gupta et al., ; Dunn et al., ) or tailored prompt engineering techniques (Yang et al., ; Zheng et al., ). While these models are easy to deploy, they suffer from diminished reproducibility and data adherence due to the absence of a memory base, untraceable fine-tuning history, or opaque extraction processes. Even though fine-tuning can encode a certain amount of domain-specific knowledge into LLMs, it is constrained by scalability and intrinsic memory capacity limitations. Fine-tuned LLMs struggle to retain in the long term the knowledge they were trained on as the training progresses, nor can they be aware of the up-to-date events and data beyond pretraining. Tailored prompt messages, while effective, also compromise the generalizability, thus limiting the overall power and flexibility of LLMs. Therefore, a more sensible approach involves equipping LLMs with external memory storage and knowledge databases, allowing them to generate holistic responses via few-shot adaptation to the accessible data and factual information (Lewis et al., ).

In this work, we implement a multimodal retrieval-augmented generation (RAG) framework LLaMP to connect LLMs with multiple Materials Project (MP) API endpoints using LangChain (Lewis et al., ; Chase, ). The framework serves as a safeguard against LLM hallucination and grounds them on high-fidelity material informatics derived from various sources, including computational data from quantum-mechanical first-principles calculations and expert-curated material synthesis recipes (Kononova et al., ). Through hierarchical planning of multiple reasoning-and-acting (ReAct) agents (Yao et al., ), we demonstrate that LLaMP not only correctly retrieves higher-order material data such as tensors and crystal structures but also combines different modalities for complex, knowledge-intensive inferences and operations. Additionally, we quantify the deviation of LLM intrinsic knowledge from MP data on scalar materials properties and 3D crystal structures, highlighting the importance of data adherence of LLMs in scientific hypotheses and applications.

The remainder of the paper is organized as follows: [Section 2](#S2) introduces the background of the Materials Project ([materialsproject.org](https://materialsproject.org)) and the utilization of natural language processing in materials informatics; [Section 3](#S3) provides a review of related work, focusing on material applications; [Section 4](#S4) introduces our proposed methodology and the LLaMP architecture; [Section 5](#S5) outlines the experimental setup and results; and finally, in [Section 6](#S6), we discuss the implications of our findings and propose directions for future research.

## 2 Background

Materials Project (MP) The Materials Project is dedicated to advancing the exploration of advanced materials crucial for sustainable energy and advanced technology (Jain et al., ). Leveraging high-throughput computing and the Vienna Ab Initio Simulation Package (VASP) (Kresse & Furthmüller, ), the initiative employs density functional theory (DFT) for precise assessments of material properties using various exchange-correlational functionals including GGA, GGA+U, r2SCAN, and others. These calculations involve spin polarization with high-spin ferromagnetic initialization, potentially allowing low spin state relaxation (Jain et al., ; Ong et al., ). Structures selected from the Inorganic Crystal Structure Database (ICSD) undergo standardized relaxation procedures, followed by post-processing or additional calculations for higher-order material properties such as electron and phonon bandgaps, elastic tensors, dielectric tensors, and more.

Natural language processing (NLP)  NLP has found extensive application in extracting valuable information from scientific publications, with notable instances involving text-to-text or more recent image-to-text summarization techniques (Tshitoyan et al., ; Radford et al., ; Gupta et al., ). For summarizing crystal structures in textual form, Ganose & Jain () introduced the robocrystallographer, a toolkit designed for the analysis and generation of descriptions for crystalline materials. Their method condenses atomic structures into descriptive JSON representations that encompass coordination statistics, connectivity motifs, geometric features, and dimensionality. Multiparagraph human-level descriptions for 130K compounds from the Materials Project were generated and are accessible through the MP website and API (Jain et al., ).

Language models (LMs)  Recent work has explored scaling language models to billions or trillions of parameters, resulting in models such as GPT-3, GPT-4 (OpenAI, ), LLaMA 2 (Touvron et al., ), and PaLM (Chowdhery et al., ). These models are trained in a self-supervised fashion on extensive text corpora, employing objectives such as auto-regressive next-word prediction. Utilizing the transformer architecture (Vaswani et al., ), which incorporates self-attention mechanisms to model long-range dependencies in text, these models learn representations during pre-training that capture semantic and syntactic information about language. The resulting large language models exhibit few-shot generalization abilities on downstream natural language tasks without the need for task-specific fine-tuning. However, when applied to knowledge-intensive tasks, language models show limited capability and often hallucinations (Lewis et al., ; Bang et al., ).

## 3 Related Work

Prompting  Prompt-based methods have emerged as effective tools for automating the data extraction process from the literature (Polak & Morgan, ; Yang et al., ). The common approach typically involves one or multiple of the following workflows: (1) Classification, where language models assess whether a paragraph or sentence contains the relevant information (Polak & Morgan, ). (2) Summarization, where sentences and paragraphs are distilled or transformed into different formats such as table and JSON (Zheng et al., ). (3) Embedding and indexing, where LLMs transform the extracted data into vector embeddings for downstream filtering or top-K retrieval. (4) Examination, where another or the original LLMs are asked to criticize or evaluate the responses. For examples, Polak & Morgan () employ a prompt workflow to extract the cooling rates of metallic glasses and yield strengths of high entropy alloys. Zheng et al. () implement a ChatGPT metal-organic framework (MOF) synthesis assistant through embedding and searching on preselected papers. Yang et al. () use GPT-4 to extract experimentally measured bandgaps to train a graph neural network for accurate bandgap prediction from crystal structures.

Despite the success in the specific data extraction tasks, prompt-based methods face challenges in reproducibility when the used prompts are kept closed-source or are fine-grained to work for specific edge cases. They are also less generalizable to combine different modalities and sources due to the deliberately designed prompt messages. Regarding response examination and evaluation, although LLM-as-a-judge is scalable and explainable, it has been shown subject to position, verbosity, and self-enhancement biases (Zheng et al., ). The limited capability of LLMs in math and reasoning also contributes to the difficulties in properly grading the responses, particularly for knowledge intensive tasks in scientific and engineering workflows.

Fine-tuning  Several knowledge-grounded, domain-specific language models lean on the fine-tuning approach against pre-selected data and literature. For instance, ChemGPT, proposed by Frey et al. (), involves fine-tuning GPT-neo on self-referencing embedded strings (SELFIES) representations of small molecules. Dunn et al. () fine-tuned GPT-3 on $\mathord{\sim}$500 prompt-completion pairs to enhance LLM’s capability to extract useful information on materials chemistry from text paragraphs. However, the fine-tuned models without augmentation inherently lack awareness of recent results and any data only available after their training. Moreover, LLMs suffer from limited memory retention and are prone to forget during continual training (Wang et al., ).

An alternative perspective is utilizing fine-tuned models for dataset augmentation or knowledge distillation rather than aiming for exhaustive generality within the constraints of memory capacity and retention. Xie et al. () tuned GPT-3 on the Perovskite solar cell FAIR dataset, employing the model to expand the existing dataset by extracting data from newly published literature. In a different approach, Semnani et al. () introduced WikiChat, a combination of RAG and fine-tuning to distill an open-domain chatbot, alleviating the hallucination problem and improving latency and affordability.

## 4 LLaMP Implementation

Retrieval-augmented generation (RAG)  While LLMs and generative pre-trained transformers have demonstrated the capability to store factual knowledge in model parameters, their ability to accurately access and manipulate precise information is limited (Lewis et al., ). Deliberate prompting does not entirely resolve this problem and, in fact, can make models overly sensitive to input requests and induces serious hallucinations. Addressing these challenges, Lewis et al. () proposed the RAG framework, which significantly mitigates these problems through the retriever-generator paradigm. The original RAG framework has two components: (i) a retriever $p_{\eta}(z|x)$ with parameters $\eta$ that returns top-K documents given the input text sequence $x$ and (ii) a generator $p_{\theta}(y_{i}|x,z,y_{1:i-1})$ with parameters $\theta$ that generates the current output text sequence $y$ based on the context of input sequence $x$, the retrieved document $z$, and the previous output sequences $y_{1:i-1}$ (Lewis et al., ). In the context of this work, the retriever and generator are replaced by LLMs, and an intermediate data retrieval step is introduced: API function calls to Materials Project, arXiv, and Wikipedia are executed to retrieve consistent and reliable data, which is then parsed or filtered by LLMs to generate text or JSON responses.

Reasoning and acting (ReAct)  One-shot generation often falls short in delivering optimal performance for numerous knowledge-intensive reasoning tasks (e.g. HotpotQA, Fever) (Yao et al., ). In an effort to enhance performance, Wei et al. () introduced the chain-of-thought (CoT) framework, which encourages language models to engage in reasoning before generating a response. While successful in boosting performance, CoT is susceptible to hallucinations arising from the generated thoughts (Yao et al., ). In this work, we adopt ReAct to augment the agent’s action space $\mathcal{A}$ with a language space $\mathcal{L}$ to create an expanded action space $\hat{\mathcal{A}}=\mathcal{A}\cup\mathcal{L}$. This expanded action space empowers the agent to take action $\hat{a}_{t}\in\mathcal{L}$ in language space that allows LLMs to reason over context. The action $\hat{a}_{t}$ functions as a thought, synthesizing valuable information by reasoning over the existing context $c_{t}$ and updating the context $c_{t+1}=(c_{t},\hat{a}_{t})$ for future actions (Yao et al., ). These intermediate reasoning steps play a crucial role in scientific hypotheses and greatly improve the hit rate of LLaMP.

Figure: Figure 1: Hierarchical ReAct agent orchestration in LLaMP. Two levels of agents are deployed using a standardized LangChain interface. Top-level ReAct agent oversees assistant ReAct agents at the bottom-level, each equipped with distinct toolkits and data stores. The figure showcases six of the implemented Materials Project (MP) specialized agents. Components represented by dashed lines are excluded from exploration in this paper due to their significant impact on response latency.
Refer to caption: /html/2401.17244/assets/x1.png

Hierarchical orchestration  To adeptly manage heterogeneous data sources and diverse types of queries, we introduce hierarchical ReAct agent planning, featuring the top-level ReAct agent overseeing specialized ReAct agents ([Figure 1](#S4.F1)). This design offers three major advantages. Firstly, the top-level agent acts as a router and decision-maker, handling abstract logic between user requests and specialized agents, thereby enhancing the robustness and interpretability of the final response. Secondly, top-level agents can integrate different sources of data parsed via specialized agents, enhancing the completeness of the final response. Thirdly, the process of function calling necessitates LLMs to assimilate API schemas, a task that can impede efficiency by consuming valuable space in the context window for each question and accelerating the depletion of API tokens. By delegating multiple functions to specialized agents, we effectively reduce the total number of required tokens effectively and circumvent the need for unnecessary schema parsing. Simultaneously, adherence to API schemas is improved since LLMs increasingly struggle to comply if said schemas approach significant fractions of the context window.

Figure: Figure 2: Multimodal retrieval-augmented generation for materials informatics. (a) User query. (b) Top-level ReAct agent capable of handling multiple specialized agents and high-level reasoning for user queries. (c-d) Specialized ReAct agents executing function calling and summarization. (c) MPThermoExpert and (d) MPElasticityExpert have access to the API schemas of thermo and elasticity endpoints on Materials Project, respectively. The selected details are highlighted in red, demonstrating the capabilities of RAG and ReAct implemented in LLaMP. The blue texts show LLaMP specialized ReAct agent can handle API calling errors and adapt the input query accordingly.
Refer to caption: /html/2401.17244/assets/x2.png

## 5 Experiments

Multimodal augmentation  Condensed matter physics, inorganic chemistry, and materials engineering frequently involve multifaceted, intricate aspects of materials properties. In many cases, these properties interact in complex ways, with optimizing one factor potentially incurring costs or deterioration in others. For example, dielectric constant usually conflicts with bandgap in materials for electronic devices (Riebesell et al., ); and high elastic modulus and low thermal conductivity are hard to achieve simultaneously for thermoelectrics (Musicó et al., ). Hence, thoroughly assessing all competing factors and relevant material properties is crucial. Direct interaction with LLMs is unlikely to systematically process different modalities, as engaging in direct conversation with LLMs proves limited in handling compound logic, arithmetic operations, and multiple information sources (Wei et al., ).

LLaMP solves the limitation naively through the hierarchical orchestration of multiple ReAct agents (Yao et al., ). As illustrated in [Figure 2](#S4.F2), when presented with a query necessitating multimodal information and compound logic, LLaMP’s top-level agent decomposes the query into multiple subtasks and plans to use specialized agents (MPThermoExpert and MPElasticityExpert) for information retrieval. For the example query “*What’s the stiffest material with the lowest formation energy in the \chSi-O chemical space?*,” LLaMP strategically dissects this question into two subtasks during its thought process: (i) using MPThermoExpert to search for \chSi-O
materials and (ii) using MPElasticityExpert to assess the stiffness of the retrieved materials. Following this planning phase, the top-level agent orchestrates the retrieval of formation energies ([Figure 2](#S4.F2)c) and elastic moduli ([Figure 2](#S4.F2)d) from the specialized agents. Subsequently, in the final stage of reasoning (step 14 in [Figure 2](#S4.F2)b), the top-level agent integrates information from both modalities, drawing on the context stored in the memory buffer and responses from the specialized agents.

Higher-order and high-fidelity data retrieval  The challenge for LLMs in excelling at knowledge- and data-intensive tasks is well-documented. Without RAG, vanilla LLMs often succumb to hallucinations, especially when the tasks involve numerical values or operations, or finding supporting references to the given answer. As shown in Table LABEL:ex:2, GPT-3.5 hallucinates the values for the components in the elastic tensor of \chNaCl, with serious erroneous values such as $C_{11}=$289.2\text{\,}\mathrm{G}\mathrm{P}\mathrm{a}$$—a significant deviation from DFT-calculated values ($76\text{\,}\mathrm{G}\mathrm{P}\mathrm{a}$). It also omits the values for $C_{22},C_{33},C_{55},C_{66}$ and fails to represent the full elastic tensor in a matrix format, despite the query explicitly requesting the full elastic tensor.

Figure: Figure 3: LLaMP RAG responses vs GPT-3.5 intrinsic knowledge on material properties. LLaMP values are identical in all cases to the corresponding data in MP which we treat as ground truth. (a) Bulk moduli, $K$, of 3d transition metals. (b) Formation energies, $\Delta H_{f}$, of common compounds. (c) Intrinsic bandgaps, $E_{g}$, of common bulk crystals for semiconductors. (d) Intrinsic bandgaps of multi-element (ternary or quaternary) materials. Missing predictions are marked by cross symbols. Mean absolute percentage errors (MAPE) are reported in [Table 1](#S5.T1).
Refer to caption: /html/2401.17244/assets/x3.png

**Table 1: Mean absolute percentage error (MAPE) of material property extraction from GPT-3.5 intrinsic knowledge in comparison with LLaMP retrieved MP values: (a) bulk moduli $K$, (b) formation energies $\Delta H_{f}$, and (c-d) bandgaps $E_{g}$. Values are computed from the data in Figure [3](#S5.F3).**
| Property | MAPE (%) |
| --- | --- |
| (a) $K$ | 21.65 |
| (b) $\Delta H_{f}$ | 1103.54 |
| (c) $E_{g}$ | 5.21 |
| (d) $E_{g}$ | NA |

To underscore the pivotal role of RAG in material informatics applications, we test vanilla GPT-3.5 (without RAG) and LLaMP (with RAG) on queries related to scalar material properties. [Figure 3](#S5.F3) presents the responses of LLaMP and GPT-3.5 regarding the bulk modulus, formation energy, and bandgap of selected materials. It is noteworthy that GPT-3.5 exhibits surprisingly accurate responses for the bandgaps of common semiconductors(^2^22bandgaps calculated from PBE-GGA functional are known to underestimate the experimental values) but tends to hallucinate when predicting the formation energies of common compounds and the bulk moduli of certain transition metals ([Figure 3](#S5.F3), [Table 1](#S5.T1)). This is likely attributable to the extensive literature corpus on experimental and theoretical semiconductor bandgaps, which have been studied and reported for decades due to intense and enduring interest in these materials for electronic devices. In contrast, reliable mechanical tests of materials are often scarce and not easily searchable via text mining.

Without additional prompting, language models overlook the subtleties of domain knowledge and may display overconfidence in intrinsic knowledge, which could be influenced by misleading data that appear more frequently and are more accessible. This is commonly referred to as the “tail knowledge problem”(Kandpal et al., ; Liu et al., ). To illustrate this, we query LLaMP and GPT-3.5 about the bandgaps of 10 multi-element materials that are less commonly encountered in the literature. As demonstrated in [Figure 3](#S5.F3)d and Table LABEL:ex:bandgap-multiele, GPT-3.5 lacks intrinsic knowledge of the bandgaps for the queried multi-element materials and fails to make predictions, whereas LLaMP could not only correctly retrieve the data but also identify the bandgap of the stable form when multiple polymorphs are present. Additionally, GPT-3.5 severely underestimates formation energies, possibly due to its training corpus being skewed toward molecular rather than crystalline data, as suggested by the use of “kJ/mol”, a more common unit in the molecular chemistry community, in the GPT-3.5 response.

RAG-assisted crystal generation and editing  LLMs fine-tuned for text-encoded atomistic information have shown the capability to generate stable crystals under the constraints of atomic positions and charges (Gruver et al., ). In this context, we delve into the examination and comparison of the crystal generation capabilities between LLaMP and GPT-3.5, without resorting to fine-tuning or tailored prompt messages in previous work. [Figure 4](#S5.F4) showcases the structures generated by LLaMP and vanilla GPT-3.5 without RAG, both instructed to insert one lithium atom at the tetrahedral site of the diamond cubic silicon structure (Table LABEL:ex:Si-Li-interstitial). Notably, both LLaMP and GPT-3.5 place an additional Li atom at fractional coordinate $(0.5,0.5,0.5)$. However, the Si structure retrieved by LLaMP adheres to the MP convention, positioning two \chSi bases at $(0.125,0.125,0.125)$ and $(0.875,0.875,0.875)$. This causes the inserted \chLi atom to be hexagonal interstitial instead of tetrahedral interstitial.

GPT-3.5 locates the Li atom at the tetrahedral site given the “luckily chosen” \chSi bases at $(0,0,0)$ and $(0.25,0.25,0.25)$; however, the resulting cell volume and shape are highly distorted, and the \chSi-Si bond length and \chSi-Si-Si angle deviate significantly from the ground truth ([Table 2](#S5.T2)), highlighting the limitations in the intrinsic encoding of LLMs for atomistic information and the challenges associated with zero-shot generation of crystal structures.

In contrast, the LLaMP-retrieved MP structure serves as a robust prior, anchoring the lattice parameters of the generated structure to the correct values. While one might argue that a fine-tuning campaign could correct the errors of naive generation to some extent, the meaningful applications usually require high-quality, accurate values, and it becomes evident that the performance of vanilla LLMs will degrade sharply when the target composition and structure are out of distribution, as exemplified by the lack of intrinsic knowledge for bandgap for multi-element materials in [Figure 3](#S5.F3)d.

Figure: (a) MP (DC Si, mp-149)
Refer to caption: /html/2401.17244/assets/fig/05-crystal-Si.png

**Table 2: Structural parameters of the generated crystals compared with diamond cubic (DC) silicon. From left to right are fractional coordinates of inserted Li atom $(x,y,z)_{\text{\ch{Li}}}$, total cell volume $V$, average \chSi-Si bond lengths $\ell_{\text{\ch{SiSi}}}$, \chSi-Si-Si angles $\theta_{\text{\ch{SiSiSi}}}$, and \chSi-Li-Si angles $\theta_{\text{\ch{SiLiSi}}}$.**
|  | $(x,y,z)_{\text{\ch{Li}}}$ | $\ell_{\text{\ch{SiSi}}}$ (Å) | Error (%) | $V$ (${\mathrm{\SIUnitSymbolAngstrom}}^{3}$) | Error (%) | $\theta_{\text{\ch{SiSiSi}}}$ ($\mathrm{\SIUnitSymbolDegree}$) | Error (%) | $\theta_{\text{\ch{SiLiSi}}}$ ($\mathrm{\SIUnitSymbolDegree}$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMP | $(0.5,0.5,0.5)$ | 2.36 | 0.0 | 40.33 | 0.0 | 109.47 | 0.0 | 62.96 |
| GPT-3.5 | $(0.5,0.5,0.5)$ | 2.71 | +15.0 | 67.05 | +66.3 | 98.28 | -10.2 | 67.69 |
| DC Si (mp-149) |  | 2.36 |  | 40.33 |  | 109.47 |  |  |

Synthesis recipes and procedures  Empowered by the MP synthesis endpoint, LLaMP can extract synthesis recipes and summarize detailed step-by-step procedures grounded on real experimental papers with associated digital object identifier (DOI), as demonstrated in the example queries (Table LABEL:ex:4 and LABEL:ex:synthesis-LiFePo4). On the other hand, GPT-3.5 gives a seemingly correct and verbose synthesis procedure for \chYMnO3 in Table LABEL:ex:4, inferring possible reaction pathways from two common oxides as precursors (\chY2O3 and \chMnO2). However, it pulls irrelevant lithium compounds (\chLi2CO3 and \chLiOH) into the recipe and overlooks the fact that metathesis reactions (Li et al., ; Todd et al., ) require less applied energy to perform than high-temperature sintering, which relies on solid-state diffusion (Maximenko & Olevsky, ). Vanilla GPT-3.5 also exhibits uncertainty about specific synthesis details, such as heating temperature, duration, cooling rate, etc.

Consider the example of \chLiFePO4 presented in Table LABEL:ex:synthesis-LiFePo4. Explicit instruction is provided: “Please provide a detailed step-by-step procedure and reference”. While GPT-3.5 does offer both a procedure and reference as asked, and the reference is indeed associated with a real paper, the paper itself contains no information about the synthesis procedure of \chLiFePO4. The procedure listed in Table LABEL:ex:synthesis-LiFePo4 is completely hallucinated from the pretraining corpus.

## 6 Discussion

The implementation of ReAct for the bottom-level agents fortifies the robustness of data retrieval. As presented in [Figure 2](#S4.F2)c, MPThermoExpert initially misunderstood the schema at the first trial and filled in the formula field with \chSi-O, an invalid input but valid as a chemical system (chemsys in the API). The observation step (step 4) allows MPThermoExpert to handle exceptions and subsequently generate the correct input fields for MP API endpoints after adaptation (step 6). In future works, storing and retrieving question-answer or query-response pairs from vector databases could reduce the number of trial-and-error steps. Furthermore, the stored pairs can be used to refine foundation LLMs to increase the successful rate of function calling.

LLMs, and recent visual language models (VLMs), have demonstrated the promise to control robotics and guide autonomous chemical experiments (Brohan et al., ; Boiko et al., ). In the context of self-driving or autonomous laboratory (Boiko et al., ; Szymanski et al., ), LLM agents must proficiently fetch and manipulate data to support their decision-making and operational tasks. For example, accurate formation energy data calculated from first principles, coupled with correct reaction energy calculations, can aid in identifying viable and faster reaction pathways. The absence of access to a high-fidelity database through autonomous agents like LLaMP would escalate the number of trial-and-error iterations, making the scaling of self-driving laboratory prohibitively expensive and potentially hazardous.

While MP is one of the most comprehensive materials databases, the available crystal structures on MP are not exhaustive but continuously expanding (Merchant et al., ). Furthermore, Kohn-Sham DFT theory is insufficient in some cases, and a higher level of theory is needed. One important future direction is to connect with automated high-throughput computational workflows to streamline the process from literature exploration, data extraction, diffusion- or language-based crystal generations (Zeni et al., ; Gruver et al., ), machine-learned atomistic simulations (Batatia et al., ), to high-fidelity computed materials properties.

## 7 Conclusion

We present a hierarchical agentic framework, LLaMP, based on RAG and ReAct to extract and manipulate material informatics through few-shot generalization. By grounding thoughts and actions in high-fidelity information, LLaMP showcases the ability to integrate various modalities of material properties and perform logical inferences to accomplish assigned tasks, all without the need for fine-tuning. In essence, the proposed LangChain framework holds the potential to expand its applicability to multiple data sources, encompassing both theoretical computations and experimental data, and real-world laboratories by incorporating additional assistant agents for data retrieval and robot control. LLaMP functions as a knowledge-aware agent, empowering users to navigate and manipulate complex materials informatics. As new tools continue to emerge, there is an exciting avenue for further exploration to ascertain if this framework can effectively facilitate scientific hypotheses and guide data-driven experiments.

### Author Contributions

Yuan Chiang led, conceptualized, and designed the project, coded the Python backend and application frontend, and drafted the initial manuscript. Chia-Hong Chou coded the application’s front end. Janosh Riebesell provided technical support on crystal visualization using [elementari](https://github.com/janosh/elementari) for the front end and revised the manuscript.

### Acknowledgments

We appreciate the helpful discussion with Dr. Anubhav Jain and Dr. Karlo Berket at LBNL, and the early test and feedback from Dr. Matthew McDermott and Dr. Jordan Burns. Yuan Chiang acknowledges the support and advice from Prof. Mark Asta and the financial support from the Berkeley Fellowship from UC Berkeley and Taiwan-UC Berkeley Fellowship from the Ministry of Education in Taiwan. We also thank OpenAI for granting the API credits.

This work was partly supported by the US Department of Energy, Office of Science, Office of Basic Energy Sciences, Materials
Sciences and Engineering Division, under contract no. DE-AC02-05-CH11231 (Materials Project program KC23MP).

## Appendix A Prompt Template

We use the ReAct template hwchase17/react-multi-input-json from LangChain Hub ([https://smith.langchain.com/hub/hwchase17/react-json](https://smith.langchain.com/hub/hwchase17/react-json)) as follows:

.

The above system prompt message is prepended with the following solicitation to guide the top-level agent:

## Appendix B Sample questions

**Table 3: QA comparison between LLaMP and GPT-3.5.**
|  | Question | Answer |  |
| --- | --- | --- | --- |
| LLaMP | GPT-3.5 |  |  |
|  |  |  |  |
| B3.1 | What are the bulk moduli of the following metals: Sc, Ti, V, Cr, Mn, Fe, Co, Ni, Cu, Zn? | The bulk moduli of the metals you asked for are as follows (in GPa):<br>- Scandium (Sc): Voigt = 45.715, Reuss = 45.34, VRH = 45.528<br>- Titanium (Ti): Voigt = 112.955, Reuss = 112.807, VRH = 112.881<br>- Vanadium (V): Voigt = 181.232, Reuss = 181.232, VRH = 181.232<br>- Chromium (Cr): Voigt = 252.155, Reuss = 252.155, VRH = 252.155<br>- Manganese (Mn): Voigt = 280.959, Reuss = 280.959, VRH = 280.959<br>- Iron (Fe): Voigt = 296.003, Reuss = 295.207, VRH = 295.605<br>- Cobalt (Co): Voigt = 200.587, Reuss = 200.587, VRH = 200.587<br>- Nickel (Ni): Voigt = 197.399, Reuss = 197.29, VRH = 197.345<br>- Copper (Cu): Voigt = 151.394, Reuss = 151.394, VRH = 151.394<br>- Zinc (Zn): Voigt = 76.283, Reuss = 95.46, VRH = 85.872 | The bulk modulus (K) is a measure of a material’s resistance to uniform compression. It is defined as the ratio of the change in pressure to the fractional volume compression. The values for the bulk modulus can vary somewhat depending on the specific conditions and crystallographic structures of the materials. The values provided here are approximate and given in gigapascals (GPa):<br>- Scandium (Sc): 55 GPa<br>- Titanium (Ti): 110 GPa<br>- Vanadium (V): 160 GPa<br>- Chromium (Cr): 160 GPa<br>- Manganese (Mn): 120 GPa<br>- Iron (Fe): 170 GPa<br>- Cobalt (Co): 180 GPa<br>- Nickel (Ni): 180 GPa<br>- Copper (Cu): 140 GPa<br>- Zinc (Zn): 70 GPa<br>These values represent typical bulk moduli for these metals, and actual values may vary depending on factors such as temperature and pressure. |
| B3.2 | What is the full elastic tensor of \chNaCl? | The elastic tensor for \chNaCl is available for two different entries on Materials Project. For material_id ’mp-22851’, the elastic tensor in IEEE format is [[76.0, 1.0, 1.0, -0.0, 0.0, -0.0], [1.0, 76.0, 1.0, -0.0, -0.0, 0.0], [1.0, 1.0, 76.0, 0.0, -0.0, -0.0], [-0.0, -0.0, 0.0, -3.0, -0.0, -0.0], [0.0, -0.0, -0.0, -0.0, -3.0, 0.0], [-0.0, 0.0, -0.0, -0.0, 0.0, -3.0]]. For material_id ’mp-22862’, the elastic tensor in IEEE format is [[47.0, 12.0, 12.0, 0.0, -0.0, -0.0], [12.0, 47.0, 12.0, 0.0, -0.0, -0.0], [12.0, 12.0, 47.0, 0.0, 0.0, -0.0], [0.0, 0.0, 0.0, 12.0, -0.0, -0.0], [-0.0, -0.0, 0.0, -0.0, 12.0, 0.0], [-0.0, -0.0, -0.0, -0.0, 0.0, 12.0]] | The elastic tensor describes the relationship between stress and strain in a material. It is a 4th-rank tensor that characterizes the elastic properties of a crystal. In the case of \chNaCl (sodium chloride), the elastic tensor is typically represented by a 6x6 matrix.<br>The elastic constants of \chNaCl can vary slightly depending on the specific conditions and crystallographic orientation. Here is an approximate form of the elastic tensor for \chNaCl:<br>C11 = 289.2 GPa<br>C12 = 97.8 GPa<br>C44 = 81.6 GPa<br>C13 = C14 = C15 = C16 = C23 = C24 = C25 = C26 = 0 GPa<br>Note: The values are given in gigapascals (GPa). The elastic constants Cij represent the components of the elastic tensor. In this case, C11, C12, and C44 are the non-zero elastic constants. The off-diagonal terms C13, C14, C15, C16, C23, C24, C25, and C26 are typically zero due to cubic symmetry.<br>Keep in mind that these values are approximate, and there may be slight variations depending on the source and the specific conditions under which the measurements were made. Experimental techniques, such as Brillouin scattering or ultrasonic measurements, are often used to determine the elastic constants of materials. |
| B3.3 | What are the bandgaps of the following materials: Si, Ge, and GaAs, GaN, SiC, BN, InSe, MoS2, BaTiO3, CsPbI3? Only look for ground-state bandgaps. | The ground state bandgaps for the requested materials are as follows: Si: 1.12 eV, Ge: 0.66 eV, GaAs: 1.42 eV, GaN: 3.28 eV, SiC: 2.36 eV, BN: 6.4 eV, InSe: 1.26 eV, MoS2: 1.29 eV, BaTiO3: 3.2 eV, CsPbI3: 1.73 eV. Please note that these values are typical for bulk materials and can vary depending on the specific structure and calculation method. | As of my last knowledge update in January 2022, I can provide you with approximate values for the ground state bandgaps of the materials you mentioned. Keep in mind that these values can vary depending on factors like temperature and crystal structure. Additionally, more recent research may have refined these values. Here are the approximate ground state bandgaps in electron volts (eV):<br>Silicon (Si): 1.1 eV<br>Germanium (Ge): 0.67 eV<br>Gallium Arsenide (GaAs): 1.42 eV<br>Gallium Nitride (GaN): 3.4 eV<br>Silicon Carbide (SiC): 3.2 eV (for 4H-SiC)<br>Boron Nitride (BN):  6 eV<br>Indium Selenide (InSe):  1.25 eV<br>Molybdenum Disulfide (MoS2):  1.3 eV (indirect bandgap)<br>Barium Titanate (BaTiO3):  3.2 eV<br>Cesium Lead Iodide (CsPbI3):  1.7 eV<br>Please note that these values are general approximations and may not represent the specific conditions or variations in different forms of these materials. For the most accurate and up-to-date information, I recommend checking recent scientific literature or databases. |
| B3.4 | Give me the decomposition enthalpies of the following materials: Ag2O3, MgZn2, GaAs, GaN, SiC, BN, InSe, MoS2, BaTiO3, CsPbI3. If there are multiple values, please give me the largest value. | The maximum decomposition enthalpies for the requested materials are as follows:<br>- Ag2O3: 0.2565836414999989 eV/atom<br>- MgZn2: 0.1356 eV/atom<br>- GaAs: 0.075 eV/atom<br>- GaN: 0.07162 eV/atom<br>- SiC: -0.172 eV/atom<br>- BN: -0.358 eV/atom<br>- InSe: -0.01816 eV/atom<br>- MoS2: 0.267 eV/atom<br>- BaTiO3: 3.055 eV/atom<br>- CsPbI3: 0.017536019000000902 eV/atom<br>Please note that the values for SiC, BN, and InSe are negative, indicating that the decomposition is exothermic. All other values are positive, indicating endothermic decompositions. | The decomposition enthalpies (in kJ/mol) of the following materials are:<br>Ag2O3: -335<br>MgZn2: -82<br>GaAs: -317<br>GaN: -505<br>SiC: -40<br>BN: -248<br>InSe: -125<br>MoS2: -393<br>BaTiO3: -112<br>CsPbI3: -153 |
| B3.5 | Give me the formation energies of the following materials: Ag2O3, MgZn2, GaAs, GaN, SiC, BN, InSe, MoS2, BaTiO3, CsPbI3. If there are multiple values, please give me the smallest value. | The smallest formation energies of the materials you asked for are as follows: Ag2O3: -0.3601 eV/atom, MgZn2: -0.1306 eV/atom, GaAs: -0.446 eV/atom, GaN: -0.052 eV/atom, SiC: -0.204 eV/atom, BN: -1.41 eV/atom, InSe: -0.742582154921875 eV/atom, MoS2: -1.202 eV/atom, BaTiO3: -3.493 eV/atom, CsPbI3: -1.296 eV/atom. | Ag2O3: -2.63 eV/atom<br>MgZn2: -0.27 eV/atom<br>GaAs: -4.13 eV/atom<br>GaN: -3.51 eV/atom<br>SiC: -4.98 eV/atom<br>BN: -3.8 eV/atom<br>InSe: -0.89 eV/atom<br>MoS2: -3.31 eV/atom<br>BaTiO3: -4.45 eV/atom<br>CsPbI3: -2.32 eV/atom |
| B3.6 | Give me the decomposition enthalpies of the following materials: Ag2O3, MgZn2, GaAs, GaN, SiC, BN, InSe, MoS2, BaTiO3, CsPbI3. If there are multiple values, please give me the smallest value. | The smallest decomposition enthalpy values for the requested materials are as follows: Ag2O3: 0.019373648083333528 eV/atom, MgZn2: -0.0197 eV/atom, GaAs: -0.446 eV/atom, GaN: -0.05231 eV/atom, SiC: -0.204 eV/atom, BN: -1.41 eV/atom, InSe: -0.06336 eV/atom, MoS2: -0.669 eV/atom, BaTiO3: -0.063 eV/atom, CsPbI3: -0.018602 eV/atom. | The decomposition enthalpies of the materials are as follows:<br>- Ag2O3: -1138.5 kJ/mol<br>- MgZn2: -54.2 kJ/mol<br>- GaAs: -452.2 kJ/mol<br>- GaN: -510.9 kJ/mol<br>- SiC: -2133.6 kJ/mol<br>- BN: -2551.0 kJ/mol<br>- InSe: -209.3 kJ/mol<br>- MoS2: -739.3 kJ/mol<br>- BaTiO3: -393.5 kJ/mol<br>- CsPbI3: -65.3 kJ/mol |
| B3.7 | Give me the energy above hull values of the following materials: Ba2MnSe3, KAlSiO4, LiCuO, H8C3N2O, Dy2ZnS4, Rb2BeF4, K9Fe2S7, Tb2FeC4, ScAlAu, CaCrO4. If there are multiple values, please give me the lowest one. | Here are the lowest energy above hull values for the requested materials:<br>Ba2MnSe3: 0.0 eV/atom<br>KAlSiO4: 0.0 eV/atom<br>LiCuO: 0.0 eV/atom<br>H8C3N2O: 0.0735 eV/atom<br>Dy2ZnS4: 0.00966 eV/atom<br>Rb2BeF4: 0.0 eV/atom<br>K9Fe2S7: 0.0 eV/atom<br>Tb2FeC4: 0.0 eV/atom<br>ScAlAu: 0.0 eV/atom<br>CaCrO4: 0.0 eV/atom.<br>Please note that these are the lowest values found in the Materials Project database and actual values may vary based on different entries. | The energy above hull values for the given materials are as follows:<br>- Ba2MnSe3: 0.0000 eV/atom<br>- KAlSiO4: 0.0000 eV/atom<br>- LiCuO: 0.0000 eV/atom<br>- H8C3N2O: 0.0000 eV/atom<br>- Dy2ZnS4: 0.0000 eV/atom<br>- Rb2BeF4: 0.0000 eV/atom<br>- K9Fe2S7: 0.0000 eV/atom<br>- Tb2FeC4: 0.0000 eV/atom<br>- ScAlAu: 0.0000 eV/atom<br>- CaCrO4: 0.0000 eV/atom |
| B3.8 | What are the bandgaps of the following materials: Ba(PdS2)2, FePO4, DyBi2IO4, NaFe(SO4)2, Eu2B5BrO9, Sr5As3H, TlGaS2, AgSbS2, Pr3S3N, LiLaO2. If there are polymorphs, give me only the bandgap of the stable material. | The bandgaps for the materials are:<br>- Ba(PdS2)2: 0.779 eV |  |
| B3.9- Most stable FePO4: 2.1932 eV |  |  |  |
| B3.10- DyBi2IO4: 1.367 eV |  |  |  |
| B3.11- Most stable NaFe(SO4)2: 2.094 eV |  |  |  |
| B3.12- Eu2B5BrO9: 0.88 eV |  |  |  |
| B3.13- Sr5As3H: 0.676 eV |  |  |  |
| B3.14- TlGaS2: 1.7885 eV |  |  |  |
| B3.15- Most stable AgSbS2: 1.372 eV |  |  |  |
| B3.16- Pr3S3N: 1.15 eV |  |  |  |
| B3.17- Most stable LiLaO2: 3.7336 eV |  |  |  |
| B3.18Please note that these values are obtained from the Materials Project database and might be different in experimental results due to various factors. | I don’t have real-time data access, and my training only includes information available up to January 2022. As a result, I may not have specific bandgap values for the materials you mentioned. However, I can provide you with general information about some of these materials and their known properties up to my last update:<br>Ba(PdS2)2: I don’t have specific bandgap information for this compound. The bandgap may depend on factors such as crystal structure and synthesis conditions.<br>FePO4: Iron(II) phosphate, known as ferrous phosphate, typically has a bandgap in the range of 2.5-3.0 eV.<br>DyBi2IO4: Dysprosium bismuth iodate. I don’t have specific bandgap information for this compound.<br>NaFe(SO4)2: Sodium iron sulfate. I don’t have specific bandgap information for this compound.<br>Eu2B5BrO9: Europium borate bromate. I don’t have specific bandgap information for this compound.<br>Sr5As3H: Strontium arsenide hydride. I don’t have specific bandgap information for this compound.<br>TlGaS2: Thallium gallium sulfide. I don’t have specific bandgap information for this compound.<br>AgSbS2: Silver antimony sulfide. This material is a semiconductor with a bandgap typically around 1.5-2.0 eV.<br>Pr3S3N: Praseodymium sulfide nitride. I don’t have specific bandgap information for this compound.<br>LiLaO2: Lithium lanthanum oxide. I don’t have specific bandgap information for this compound.<br>The most stable structure for each material can vary based on synthesis conditions and other factors. For accurate and up-to-date bandgap values and structural information, you may want to consult recent scientific literature or databases that specialize in material properties. |  |  |
| B3.19 | How to synthesize YMnO3? | To synthesize YMnO3, a solid-state reaction method can be employed. Here is a summarized procedure based on the synthesis details provided in the literature with DOI: 10.1021/ic502027k:<br>1. Dissolve Y(NO3)3·6H2O, Mn(NO3)2·4H2O, and a source of [OH-] in water to create a homogeneous solution.<br>2. Add citric acid to the solution to act as a mixing media.<br>3. Heat the mixture at 250°C for 2 hours to achieve pre-calcination.<br>4. Finally, heat the pre-calcined mixture at a temperature range of 700°C to 930°C for 10 to 18 hours in an air atmosphere to form YMnO3. | Synthesizing YMnO3 (yttrium manganite) typically involves the solid-state reaction method. Please note that the details may vary depending on the specific conditions and equipment available in your laboratory. Here’s a general procedure:<br>Materials:<br>Yttrium oxide (Y2O3)<br>Manganese oxide (MnO2 or Mn2O3)<br>High-purity manganese carbonate (MnCO3)<br>Lithium carbonate (Li2CO3) or lithium hydroxide (LiOH)<br>Crucible and pestle<br>High-temperature furnace<br>Mortar and pestle<br>Platinum crucible or alumina crucible<br>Procedure:<br>Step 1: Weighing the Precursors<br>Weigh the appropriate amounts of yttrium oxide (Y2O3), manganese oxide (MnO2 or Mn2O3), and lithium carbonate (Li2CO3) or lithium hydroxide (LiOH) according to the stoichiometric ratio of YMnO3.<br>Step 2: Mixing the Powders<br>2.GPT-3.5 starts to fail numbering correctly in this response. Grind the powders thoroughly in a mortar and pestle to ensure a homogenous mixture.<br>Step 3: Heating<br>3. Place the mixed powder in a crucible or alumina crucible.<br>Heat the mixture in a high-temperature furnace. The specific temperature and heating rate depend on the exact synthesis conditions, but a typical range is 800-1100°C.<br>Step 4: Sintering<br>5. Hold the mixture at the target temperature for a specific duration (several hours) to allow for sintering and the formation of YMnO3.<br>Step 5: Cooling<br>6. Slowly cool the sample to room temperature to avoid thermal stress.<br>Step 6: Characterization<br>7. Characterize the synthesized YMnO3 using techniques such as X-ray diffraction (XRD), scanning electron microscopy (SEM), and energy-dispersive X-ray spectroscopy (EDS) to confirm the desired phase and composition.<br>Notes:<br>The choice of manganese precursor (MnO2 or Mn2O3) and lithium source (carbonate or hydroxide) can impact the synthesis process and may require optimization.<br>It’s crucial to use high-purity starting materials to avoid impurities in the final product.<br>The specific temperatures and times mentioned are general guidelines and may need to be adjusted based on the equipment and conditions in your laboratory.<br>Always follow safety protocols when working with high temperatures and potentially hazardous materials.<br>This is a simplified overview, and you should consult relevant literature and adapt the procedure based on your specific requirements and equipment. |
| B3.20 | How to synthesis LiFePO4? Please provide a detailed step-by-step procedure and the reference. | LiFePO4 can be synthesized using various methods. One method involves a high-temperature solid-state synthesis using precursors such as Fe2O3, LiOH, and NH4H2PO4. The precursors are mixed and heated at temperatures up to 850°C for several days. Another method uses a sol-gel combustion approach with precursors like FePO4 and Li2CO3, where the mixture is calcined at 700°C in a nitrogen atmosphere. For detailed procedures, please refer to the following scientific literature:<br>1. High-temperature solid-state synthesis - DOI: 10.1021/jp407510h<br>2. Ion-exchange method - DOI: 10.1021/cm402617b<br>3. Solution method - DOI: 10.1021/jp306936t<br>4. Solid-state reactions - DOI: 10.1021/cm303259j<br>5. Sol-gel combustion method - DOI: 10.1007/s10853-011-6139-7 | Synthesizing \chLiFePO4 (lithium iron phosphate) typically involves a solid-state reaction method. Below is a step-by-step procedure for synthesizing \chLiFePO4:<br>Materials:<br>- Lithium carbonate (Li2CO3)<br>- Iron (II) acetate (\chFe(CH3COO)2)<br>- Ammonium dihydrogen phosphate (NH4H2PO4)<br>- Phosphoric acid (H3PO4)<br>- Deionized water<br>- Ethanol<br>- Mortar and pestle<br>- Crucible<br>- Furnace<br>Procedure:<br>1. Weigh out the appropriate amounts of lithium carbonate (Li2CO3) and iron (II) acetate (\chFe(CH3COO)2) in a 1:1 molar ratio. For example, if you want to synthesize 1 mole of \chLiFePO4, you would weigh out 74.55 grams of Li2CO3 and 173.93 grams of \chFe(CH3COO)2.<br>2. Grind the lithium carbonate and iron (II) acetate together using a mortar and pestle to ensure a homogenous mixture.<br>3. Weigh out the appropriate amount of ammonium dihydrogen phosphate (NH4H2PO4) and add it to the lithium carbonate and iron (II) acetate mixture in a 1:1 molar ratio. For example, if you are synthesizing 1 mole of \chLiFePO4, you would weigh out 122.00 grams of NH4H2PO4.<br>4. Add a few drops of phosphoric acid (H3PO4) to the mixture to adjust the pH to around 7. This will help in the formation of \chLiFePO4.<br>5. Add a small amount of deionized water to the mixture and mix it thoroughly to form a slurry.<br>6. Slowly evaporate the water from the slurry by placing it on a hot plate or in a furnace at a low temperature.<br>7. Once the water has evaporated, heat the mixture in a crucible in a furnace at around 600-700°C for several hours to initiate the solid-state reaction and form \chLiFePO4.<br>8. After the reaction is complete, allow the crucible to cool and collect the \chLiFePO4 powder.<br>9. Wash the \chLiFePO4 powder with ethanol to remove any impurities and then dry it in a desiccator.<br>Reference:<br>P. G. Bruce, B. Scrosati, J.-M. Tarascon, "Nanomaterials for Rechargeable Lithium Batteries," Angew. Chem. Int. Ed. 2008, 47, 2930-2946. DOI: 10.1002/anie.200702505. |
| B3.21 | Could you insert one Li atom at the tetrahedral interstitial site in diamond cubic Si? Pull out the most stable DC Si structure from MP directly in JSON string, modify the retrieved structure directly, and give me the final pymatgen structure as a JSON string. | The final structure after inserting a Li atom at the tetrahedral interstitial site in the diamond cubic Si structure is as follows:<br>[⬇](data:text/plain;base64,eydAbW9kdWxlJzogJ3B5bWF0Z2VuLmNvcmUuc3RydWN0dXJlJywgJ0BjbGFzcyc6ICdTdHJ1Y3R1cmUnLCAnY2hhcmdlJzogMCwgJ2xhdHRpY2UnOiB7J21hdHJpeCc6IFtbMy4zMzM1NzMsIDAuMCwgMS45MjQ2MzldLCBbMS4xMTExOTEsIDMuMTQyOTI0LCAxLjkyNDYzOV0sIFswLjAsIDAuMCwgMy44NDkyNzhdXSwgJ3BiYyc6IFtUcnVlLCBUcnVlLCBUcnVlXSwgJ2EnOiAzLjg0OTI3ODQwMzM2OTkwOTUsICdiJzogMy44NDkyNzk0MTE2MDEzNDU2LCAnYyc6IDMuODQ5Mjc4LCAnYWxwaGEnOiA2MC4wMDAwMTIxMzA5NDQyMSwgJ2JldGEnOiA2MC4wMDAwMDM0NjY0NTk4NCwgJ2dhbW1hJzogNjAuMDAwMDEwOTc1NDU3ODksICd2b2x1bWUnOiA0MC4zMjk1MjY4NDc0MTQwNX0sICdwcm9wZXJ0aWVzJzoge30sICdzaXRlcyc6IFt7J3NwZWNpZXMnOiBbeydlbGVtZW50JzogJ1NpJywgJ29jY3UnOiAxfV0sICdhYmMnOiBbMC44NzUsIDAuODc1LCAwLjg3NV0sICd4eXonOiBbMy44ODkxNjg1LCAyLjc1MDA1ODQ5OTk5OTk5OTcsIDYuNzM2MjM2NV0sICdwcm9wZXJ0aWVzJzogeydtYWdtb20nOiAtMC4wfSwgJ2xhYmVsJzogJ1NpJ30sIHsnc3BlY2llcyc6IFt7J2VsZW1lbnQnOiAnU2knLCAnb2NjdSc6IDF9XSwgJ2FiYyc6IFswLjEyNSwgMC4xMjUsIDAuMTI1XSwgJ3h5eic6IFswLjU1NTU5NTUsIDAuMzkyODY1NSwgMC45NjIzMTk1XSwgJ3Byb3BlcnRpZXMnOiB7J21hZ21vbSc6IC0wLjB9LCAnbGFiZWwnOiAnU2knfSwgeydzcGVjaWVzJzogW3snZWxlbWVudCc6ICdMaScsICdvY2N1JzogMX1dLCAnYWJjJzogWzAuNSwgMC41LCAwLjVdLCAneHl6JzogWzIuMjIyMzgyLCAxLjU3MTQ2MiwgMS45MjQ2MzldLCAncHJvcGVydGllcyc6IHt9LCAnbGFiZWwnOiAnTGknfV19)<br>{’@module’: ’pymatgen.core.structure’, ’@class’: ’Structure’, ’charge’: 0, ’lattice’: {’matrix’: [[3.333573, 0.0, 1.924639], [1.111191, 3.142924, 1.924639], [0.0, 0.0, 3.849278]], ’pbc’: [True, True, True], ’a’: 3.8492784033699095, ’b’: 3.8492794116013456, ’c’: 3.849278, ’alpha’: 60.00001213094421, ’beta’: 60.00000346645984, ’gamma’: 60.00001097545789, ’volume’: 40.32952684741405}, ’properties’: {}, ’sites’: [{’species’: [{’element’: ’Si’, ’occu’: 1}], ’abc’: [0.875, 0.875, 0.875], ’xyz’: [3.8891685, 2.7500584999999997, 6.7362365], ’properties’: {’magmom’: -0.0}, ’label’: ’Si’}, {’species’: [{’element’: ’Si’, ’occu’: 1}], ’abc’: [0.125, 0.125, 0.125], ’xyz’: [0.5555955, 0.3928655, 0.9623195], ’properties’: {’magmom’: -0.0}, ’label’: ’Si’}, {’species’: [{’element’: ’Li’, ’occu’: 1}], ’abc’: [0.5, 0.5, 0.5], ’xyz’: [2.222382, 1.571462, 1.924639], ’properties’: {}, ’label’: ’Li’}]} | Here’s the JSON string for the most stable diamond cubic Si structure retrieved directly from MP:<br>[⬇](data:text/plain;base64,YGBganNvbgp7J3NpdGVzJzogW3snc3BlY2llcyc6IFt7J2VsZW1lbnQnOiAnU2knLCAnb2NjdSc6IDF9XSwKICAgJ2FiYyc6IFswLCAwLCAwXSwgJ3h5eic6IFswLCAwLCAwXSwgJ2xhYmVsJzogJ1NpJywgJ3Byb3BlcnRpZXMnOiB7fX0sCiAgeydzcGVjaWVzJzogW3snZWxlbWVudCc6ICdTaScsICdvY2N1JzogMX1dLAogICAnYWJjJzogWzAuMjUsIDAuMjUsIDAuMjVdLCAneHl6JzogWzAuMTc4LCAwLjE3OCwgMC4xNzhdLCAnbGFiZWwnOiAnU2knLCAncHJvcGVydGllcyc6IHt9fV0sCiAnbGF0dGljZSc6IHsnbWF0cml4JzogW1szLjg2NjAwMiwgMCwgMi4yNDMwNDldLAogICBbMy44NjYwMDIsIDMuODY2MDAyLCAwXSwKICAgWzAsIDMuODY2MDAyLCAyLjI0MzA0OV1dLAogICdhJzogNS40MzEsICdiJzogNS40MzEsICdjJzogNS40MzEsCiAgJ2FscGhhJzogMTA5LjQ3MTIxOSwgJ2JldGEnOiAxMDkuNDcxMjE5LCAnZ2FtbWEnOiAxMDkuNDcxMjE5LAogICd2b2x1bWUnOiAxMjkuNTg0MzU0ODk4fSwKICdwcm9wZXJ0aWVzJzogeydzcGFjZV9ncm91cCc6IHsnbnVtYmVyJzogMjI3LAogICAnc291cmNlJzogJ3NwZ2xpYicsCiAgICdzeW1ib2wnOiAnRmQtM20nfX19CmBgYA==)<br>‘‘‘json<br>{’sites’: [{’species’: [{’element’: ’Si’, ’occu’: 1}],<br>’abc’: [0, 0, 0], ’xyz’: [0, 0, 0], ’label’: ’Si’, ’properties’: {}},<br>{’species’: [{’element’: ’Si’, ’occu’: 1}],<br>’abc’: [0.25, 0.25, 0.25], ’xyz’: [0.178, 0.178, 0.178], ’label’: ’Si’, ’properties’: {}}],<br>’lattice’: {’matrix’: [[3.866002, 0, 2.243049],<br>[3.866002, 3.866002, 0],<br>[0, 3.866002, 2.243049]],<br>’a’: 5.431, ’b’: 5.431, ’c’: 5.431,<br>’alpha’: 109.471219, ’beta’: 109.471219, ’gamma’: 109.471219,<br>’volume’: 129.584354898},<br>’properties’: {’space_group’: {’number’: 227,<br>’source’: ’spglib’,<br>’symbol’: ’Fd-3m’}}}<br>‘‘‘<br>Now, let’s insert one Li atom at the tetrahedral interstitial site in this diamond cubic Si structure and retrieve the final pymatgen structure as a JSON string:<br>[⬇](data:text/plain;base64,YGBganNvbgp7J3NpdGVzJzogW3snc3BlY2llcyc6IFt7J2VsZW1lbnQnOiAnU2knLCAnb2NjdSc6IDF9XSwKICAgJ2FiYyc6IFswLCAwLCAwXSwKICAgJ3h5eic6IFswLCAwLCAwXSwKICAgJ2xhYmVsJzogJ1NpJywKICAgJ3Byb3BlcnRpZXMnOiB7fX0sCiAgeydzcGVjaWVzJzogW3snZWxlbWVudCc6ICdTaScsICdvY2N1JzogMX1dLAogICAnYWJjJzogWzAuMjUsIDAuMjUsIDAuMjVdLAogICAneHl6JzogWzAuMTc4LCAwLjE3OCwgMC4xNzhdLAogICAnbGFiZWwnOiAnU2knLAogICAncHJvcGVydGllcyc6IHt9fSwKICB7J3NwZWNpZXMnOiBbeydlbGVtZW50JzogJ0xpJywgJ29jY3UnOiAxfV0sCiAgICdhYmMnOiBbMC41LCAwLjUsIDAuNV0sCiAgICd4eXonOiBbMC41LCAwLjUsIDAuNV0sCiAgICdsYWJlbCc6ICdMaScsCiAgICdwcm9wZXJ0aWVzJzoge319XSwKICdsYXR0aWNlJzogeydtYXRyaXgnOiBbWzMuODY2MDAyLCAwLCAyLjI0MzA0OV0sCiAgIFszLjg2NjAwMiwgMy44NjYwMDIsIDBdLAogICBbMCwgMy44NjYwMDIsIDIuMjQzMDQ5XV0sCiAgJ2EnOiA1LjQzMSwKICAnYic6IDUuNDMxLAogICdjJzogNS40MzEsCiAgJ2FscGhhJzogMTA5LjQ3MTIxOSwKICAnYmV0YSc6IDEwOS40NzEyMTksCiAgJ2dhbW1hJzogMTA5LjQ3MTIxOSwKICAndm9sdW1lJzogMTI5LjU4NDM1NDg5OH0sCiAncHJvcGVydGllcyc6IHsnc3BhY2VfZ3JvdXAnOiB7J251bWJlcic6IDIyNywKICAgJ3NvdXJjZSc6ICdzcGdsaWInLAogICAnc3ltYm9sJzogJ0ZkLTNtJ319fQpgYGA=)<br>‘‘‘json<br>{’sites’: [{’species’: [{’element’: ’Si’, ’occu’: 1}],<br>’abc’: [0, 0, 0],<br>’xyz’: [0, 0, 0],<br>’label’: ’Si’,<br>’properties’: {}},<br>{’species’: [{’element’: ’Si’, ’occu’: 1}],<br>’abc’: [0.25, 0.25, 0.25],<br>’xyz’: [0.178, 0.178, 0.178],<br>’label’: ’Si’,<br>’properties’: {}},<br>{’species’: [{’element’: ’Li’, ’occu’: 1}],<br>’abc’: [0.5, 0.5, 0.5],<br>’xyz’: [0.5, 0.5, 0.5],<br>’label’: ’Li’,<br>’properties’: {}}],<br>’lattice’: {’matrix’: [[3.866002, 0, 2.243049],<br>[3.866002, 3.866002, 0],<br>[0, 3.866002, 2.243049]],<br>’a’: 5.431,<br>’b’: 5.431,<br>’c’: 5.431,<br>’alpha’: 109.471219,<br>’beta’: 109.471219,<br>’gamma’: 109.471219,<br>’volume’: 129.584354898},<br>’properties’: {’space_group’: {’number’: 227,<br>’source’: ’spglib’,<br>’symbol’: ’Fd-3m’}}}<br>‘‘‘ |

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: - Figure 2, which reports with Table 1 the main results of the manuscript, is barely readable. 
Besides working harder on contrasting colors overall visibility, the authors should explain all symbols' choices and consider either reducing the amount of data shown or make the figure in more panels (possibly reporting some crucial example in the main text and the rest in an appendix).
- in Table 1, confidence scores equal exactly to 1 (at least to the third digit precision), especially if combined with MAE equal to 0.000 (see electronic band gaps of common elements for LLaMP) look unlikely good and suggest the prediction use training data. This aspect would need discussion in presenting the results, and probably a different design of the performance tests.
soundness: 3
strengths: The introduced LLM is very likely of great interest for the materials science community, in particular because it is ready to be used and it is fine tuned on one of the largest and most consistent corpus of materials-science data, especially computational (atomistic-simulation) data. 
Results suggest it would be a useful tool used for materials-science research.
summary: The manuscript describes the introduction and test of LLaMP, an LLM fine tuned to interact with the materials science data from the Materials Project database and run atomistic simulations via a workflow interface.
weaknesses: There is no methodological advance worth highlighting here. The described "hierarchical agent framework" is a simple device to design and implement, while the metric assessing consistency is reasonable for reporting and discussing results but does not strike as a major advance.
In view of these strengths and weaknesses, the paper seems more indicated for a materials-science specialized journal such as npj Computational Materials.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. According to Section 4.2, $CoP = exp(−Precision)$. However, a lot of Precision and CoP results in Table 1 do not match this relationship. For example, The Precision of LLaMP on bulk modulus is $2.698$, then $exp(-2.698) = 0.067$, but the reported value is $0.900$. Can the authors clarify that?

1. Have the authors tried LangGraph instead of LangChain for a more structured agentic workflow instead of a linear one? Langgraph will probably be beneficial as it may enhance the hierarchical planning in LLaMP, potentially improving agent coordination and task prioritization, especially for complex, multi-step processes.

1. In the example in Figure A.1, it seems that the supervisor gives mp-9258 as the answer to the query even though the two assistant agents never return any of the relevant information. So can I interpret that the example ends up with a hallucination? Is there any method to prevent this, and why is there not an observation step to capture the mismatch between the requested and returned MP IDs and to require the refinement of the action?

1. While LLaMP shows strength in typical systems (e.g., Si-O, Li-based compounds), how does it perform on less common or highly complex systems, such as multinary oxides or intermetallic compounds?
soundness: 3
strengths: 1. The paper presents a well-designed hierarchical ReAct agentic framework that dynamically manages multiple agents, each specialized in distinct tasks. This structured approach is highly modular, improving the model's accuracy and efficiency in handling complex workflows.

1. The authors provide a comprehensive discussion of LLaMP's strengths and limitations, offering valuable insights into its potential and constraints in materials science. This balanced analysis enhances the paper's utility for related researchers by clarifying LLaMP's practical applicability and areas for further development. 

1. The self-correcting agent mechanism enables real-time error correction during tool usage and API interactions. This design minimizes the propagation of errors and enhances reliability in complex tasks. 

1. The paper not only presents benchmarking results and analysis but also discusses the real-world applications in materials science. This applied focus, combined with LLaMP's use of high-fidelity data sources, makes it highly relevant and impactful for researchers.
summary: The authors propose LLaMP, a multimodal retrieval-augmented generation (RAG) framework leveraging hierarchical reasoning-and-acting (ReAct) agents to dynamically interact with Materials Project (MP), arXiv, Wikipedia, and atomistic simulation tools. To reduce the hallucination of Large Language Models (LLMs), the framework provides LLMs with high-fidelity material informatics derived from various sources. Consequently, LLaMP leverages hierarchical planning and correctly retrieves higher-order materials data and also combines different modalities to perform complex, knowledge-intensive inferences and operations essential for real-world research applications. LLaMP manifests enhanced performance in predicting key material properties, real-world applications in materials science, and language-driven simulations.
weaknesses: 1. The framework relies heavily on data from the Materials Project, which may restrict its application to new or less-explored materials. Although MP is comprehensive, the paper acknowledges that the database's coverage is not exhaustive, particularly for certain magnetic and bandgap configurations.

1. While the paper demonstrates LLaMP’s applications in materials science, the analysis is primarily based on one or two examples for each application. A more comprehensive experiment or in-depth discussion of success and failure cases should be helpful for exploring real-world applications. 

1. The statistics for precision and CoP in Table 1 seem not consistent with the equation in Section 4.2. See the questions for details.

---

confidence: 4
contribution: 2
initial_rating: 1
presentation: 2
questions: * How are the agents distinguished from each other? Do they receive different prompts, different context? It seems like not all of them are LLM based - it would be good to describe more details on what is considered an agent and what is considered a tool.
* Have you compared to other RAG methods? It seems a bit unfair to have to benchmark LLMs solely on implicit knowledge. It would also be good to get additional context on how your proposed tool calling method compares to other methods [8]
* How limited are the proposed methods to Materials Project? What would be needed to apply it to other materials science knowledge sources?
* Could you add the main conclusions in the captions of figures and tables for greater clarity?

[8] Qin, Yujia, et al. "ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs." The Twelfth International Conference on Learning Representations.
soundness: 1
strengths: * The paper introduces a new framework to relevant challenges in materials science. 
* The tools and knowledge bases included in the LLAMP framework could be useful to build upon.
summary: The paper proposes a new method for LLMs (LLAMP) applied to materials science based on retrieval-augmented generation (RAG) and hierarchical reasoning-and-acting (ReAct) that addresses language-based materials science tasks. The papers starts by motivating LLAMP based on the fact that many LLMs lack access to up-to-data for solving timely materials challenges and are prone to hallucination. As such, LLAMP aims to alleviate those challenges by providing a way to infuse external data sources when solving materials science tasks. The paper claims five contributions: 1. the LLAMP method; 2. metrics for understanding self-consistency for LLMs; 3. a study for materials property prediction; 4. showcasing LLAMP capabilities in synthesis and materials structure generation; 5. demonstrating high-throughput atomistic simulations using LLAMP.

Next, the paper describes related work in Section 2 and Section 3 respectively focused on the materials project, NLP in science as well as the application of prompting and tool use to solve materials science challenges. Section 4 the describes the main pieces of the LLAMP-ReAct framework, including the supervisor agent and assistant agents that help perform relevant tasks. Section 4 also outlines the metrics used in subsequent analysis, including Prediction, Coefficient of Precision, Confidence and the Self-consistency of Response (SCoR). The primary aim of the metric is the understand the consistency and confidence of LLM responses to certain queries where reliability is important. Section 5 provides the main experiments with the main focus on including materials property prediction. The paper then provides an analysis based on the metrics proposed in Section 4. The analysis generally shows that LLAMP provides lower MAE in property prediction and usually second best in SCoR. Section 5.2 also showcases a study on data retrieval for materials property analysis where LLAMP with its ReAct framework generally outperforms vanilla LLMs relying on implicit knowledge. Section 5.3 provides a brief description summarization of synthesis procedures as well as crystal generation and language-drive simulation. The discussion of the experiments in Section 5.3 is brief with much reference to the appendix for supporting information. The general claims in Section 5.3 is that LLAMP outperforms vanilla LLMs.

The paper concludes in Section 6 with a discussion on robustness, which ties into one of the main motivations outlined in the introduction, as well as limitations and a broader summary of the work and potential future work.
weaknesses: In its current form, the paper has substantial weaknesses related to limited machine learning novelty, scant evidence of the initial claims and missing discussion of relevant related work.

* Machine Learning Novelty: the paper does not provide good evidence for its machine learning novelty. The background and related work provided limited discussion of RAG and tool-calling in general and miss relevant work in related domains. As such it is difficult to understand the novelty of the LLAMP. 
* Evidence for claims: In the current form, the main addition novelty related to LLAMP focuses on RAG and ReAct and many of the presented experiments show that a RAG-based performs better a RAG-based tasks (e.g., information retrieval) compared to a non-RAG model. As such, the only claim that is somewhat properly supported is that a RAG-based model is somewhat better and more consistent on tasks where RAG is useful, which is not particularly surprising given prior success of RAG. Additionally the paper provides scant evidence for claims 4 & 5 in the main text with much reference for the appendix. This is not (in my opinion) best practice and weakens the initial claims made in the introduction.
     * The claims related to the ReAct framework could also be strengthened by adding an ablation on tool calling - especially since the paper mentions flat tool calling.
* Related work: The paper provides a limited discussion of relevant work related to machine learning for relevant parts of the proposed method, including RAG and tool-calling. This extends to LLMs for chemistry and materials science where methods, including agent methods, have been proposed [1] [2]. The section on NLP for science could also benefit from a broader inclusion of work on applying LLMs for question-answering tasks in materials science [3] [4] [5] along with additional work on LLMs for materials science data extraction [6] [7 - mara].

[1] Ghafarollahi, Alireza, and Markus J. Buehler. "AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence." arXiv preprint arXiv:2407.10022 (2024).

[2] Zhang, Huan, et al. "Honeycomb: A flexible llm-based agent system for materials science." arXiv preprint arXiv:2409.00135 (2024).

[3] Yu Song, Santiago Miret, and Bang Liu. 2023. MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3621–3639, Toronto, Canada. Association for Computational Linguistics.

[4] Zaki, Mohd, and NM Anoop Krishnan. "MaScQA: investigating materials science knowledge of large language models." Digital Discovery 3.2 (2024): 313-327.

[5] Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. 2023. HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5724–5739, Singapore. Association for Computational Linguistics.

[6] Hira, Kausik, et al. "Reconstructing the materials tetrahedron: challenges in materials information extraction." Digital Discovery 3.5 (2024): 1021-1037.

[7] Schilling-Wilhelmi, Mara, et al. "From text to insight: Large language models for materials science data extraction." arXiv preprint arXiv:2407.16867 (2024).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1) Can you please elaborate on how the standard deviation in SCoR is calculated? I assume that it only works for numerical answers and it's the standard deviation of the population.
2) Can you please clarify Figure 2. What are the ground truth values?
3) Can you provide code to reproduce the framework? This would add value to your submission.
soundness: 3
strengths: **Originality**
LLaMP is original in its use of hierarchical agents and the combination of agents used. The metrics and combination of tasks is also original. 

**Quality**
The hierarchical orchestration is well thought out and seems to perform well. The variety of agents makes the framework useful for materials scientists. The SCoR is mostly well justified. The experiments are convincing. The variety of tasks used is a bonus.

**Clarity**
The paper is easy to read  and well organised. Figures and Tables are informative (except Figure 2, see Weaknesses).

**Significance**
Accuracy of answers and reducing hallucinations is important when using LLMs in a scientific context. LLaMP is a significant step towards addressing these issues and thus is a valuable contribution to the community.
summary: The paper proposes LLaMP, a RAG framework based on hierarchical ReAct agents. The supervisor agent handles high level logic while assistant agents interact with Materials Project, arXiv, Wikipedia and atomistic simulation tools. 

The authors then introduce a metric, the Self-Consistency of Response (SCoR), which aggregates the model's confidence in answering with the variability of the answers. 

Experiments show that LLaMP achieves a low MAE and high SCoR in generating Bulk Modulus, Formation Energy, and Electronic Bandgap while LLMs and other RAG frameworks achieve higher MAEs and comparable or lower SCoR. 

LLaMP is also able to predict the magnetic orderings and total magnetisation, retrieve inorganic synthesis recipes, edit crystal structures and drive simulations.
weaknesses: 1) The name precision in SCoR feels misleading. When SCoR = 1, the model generates the same response to queries, meaning that the standard deviation is zero and also the precision is zero. This is counter-intuitive as I think of precision as True Positives / (True Positives + False Positives).
2) Figure 2 is unclear. How should I read the values produced by each method? What is the ground truth?
3) (Minor) I can't find a citation for LangChain.

Meta-review: In this work, authors present a multimodal retrieval-augmented generation (RAG) framework that uses hierarchical reasoning-and-acting (ReAct) agents to interact with Materials Project database and run atomistic simulations. The paper claims improved accuracy in material property prediction, reduced hallucination through data retrieval, and capabilities in synthesis recommendations and molecular simulations. The framework shows lower MAE and better self-consistency across multiple LLM backends for various materials science tasks. The work's strengths lie in its practical utility, comprehensive evaluation, novel consistency metrics, and strong empirical results, supported by reproducible implementation. However, significant weaknesses undermine its suitability for ICLR. The core methodology primarily combines existing approaches without fundamental theoretical advances in LLM or agent architectures. The heavy dependence on Materials Project database raises concerns about generalizability, while claims about synthesis capabilities lack comprehensive validation. The paper's emphasis on engineering implementation over theoretical innovation, coupled with inadequate comparisons to state-of-the-art RAG/tool-use methods, suggests it would be better suited for a materials science venue. While LLaMP represents a valuable contribution to materials science research tooling, its limited ML methodology advancement and focus on practical implementation over theoretical insights do not meet the standards expected for ICLR. The authors should consider expanding the theoretical framework, broadening experimental validation, and resubmitting to a domain-specific journal where the practical impact would be more appropriately valued.

Decision: Reject