Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design
Authors: Henry W. Sprueill
Carl Edwards
Mariefel V Olarte
Udishnu Sanyal
Heng Ji
Sutanay Choudhury
Venue: EMNLP 2023 Findings (2023)
Keywords: Chain-of-Thought
Large Language Model
Reasoning
Scientific Discovery
Chemistry
Catalysis

Abstract: Discovering novel catalysts requires complex reasoning involving multiple chemical properties and resultant trade-offs, leading to a combinatorial growth in the search space. While large language models (LLM) have demonstrated novel capabilities for chemistry through complex instruction following capabilities and high quality reasoning, a goal-driven combinatorial search using LLMs has not been explored in detail. In this work, we present a Monte Carlo Tree Search-based approach that improves beyond state-of-the-art chain-of-thought prompting variants to augment scientific reasoning. We introduce two new reasoning datasets: 1) a curation of computational chemistry simulations, and 2) diverse questions written by catalysis researchers for reasoning about novel chemical conversion processes. We improve over the best baseline by 25.8% and find that our approach can augment scientist's reasoning and discovery process with novel insights.

Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design
Henry W. Sprueill,, Carl Edwards?, Mariefel V. Olarte', Udishnu Sanyal',
Heng Ji²,, Sutanay Choudhury'
Paciffic Northwest National Laboratory, Richland, Washington, USA Universityy of Illinois Urbana-Champar Urbana, Illinois, USA (henry.sprueill, seweeel.olartee udishnu. sanyal, sutanay. choudhuryy@pnnl (cne2, hengjijeillinois. edu
Abstract
Q: What are the top-3 catalysts water gas
Discovering novel catalysts requires complex reasoning involving multiple chemical proper- ties and resultant trade-offs, leading to a combi- natorial growth in the search space. While large language models (LLM) have demonstrated novel capabilities for chemistry through com- plex instruction following capabilities and high quality reasoning, goal-driven combinatorial search using LLMs has not been explored in detail. In this work, we present Monte Carlo Tree Search- based approach that improves be- yond state-of-the-arr chain-of-thought prompt- ing variants augment scientific reasoning. We introduce two new reasoning datasets: I) a curation of computational chemistry simu- lations, and 2) diverse questions written by catalysis researchers for reasoning about novel chemical conversion processes. We improve over the best baseline by 25.8% and find that our approach can augment scientist's ' reasoning and discovery process with novel insights.
reaction? Let's think step by step.. Pu,
Q: What are the top catalysts like PLPd Ru for the RWGS reaction. Include cata with coct.
A: Ou, Ni, Co. Because..
Q: Provide 3 metal oxidle catalysts that include Cu, for
water gas reaction. Consider catalyets with (low cost, high activity) NEO.O, Co.O.. Because...
Figure I: An example prompt design via tree search, The search begins with generic query at the root node. The answer from each node is passed to the child nodes and additional criterion are added to the prompt. For instance, low cost. Information passed to children nodes color coded to show the reasoning pathway.
expert reasoning with knowledge of the scientific literature that dates back a century. The emerg- ing capability of large language models (LLMs) (Wei et al., 2022: Ouyang et al., 2022; Taylor et al., 2022; Lai et al., 2023; OpenAI, 2023) provides opportunity to automatically reason with large knowledge space in human-interpretable way.
1 Introduction
Scientific discovery thrives on uncovering the opti- mal combinations of factors that maximize prop- erty of interest. For example, to discover new ef- ficient fuels (Yang et al., 2019; Tran et al., 2023; Zitnick et al., 2020) or chemical conversion pro- cesses requiring less energy, a scientist would need to consider the chemical reaction, the reactants that undergo the reaction, the catalysts that improve the rate of reaction, and find the optimal combination of operating conditions (Fig. 2). Mathematically, one could represent this as an optimization prob- lem where we model a chemical process as func- tion and formulate the search problem as finding the optimal combination of all process parameters that minimizes a cost function modeled around en- ergy efficiency. For highly empirical fields such chemistry, these combinatorial searches require
Despite their promise, the brittleness of language models their inputs and hallucination remain ar- eas for concern (Creswell and Shanahan, 2022; Tay- lor et al., 2022). Our initial investigation LLMs revealed that basic prompting (such as "What a good catalyst for reaction X?") leads to basic an- swers that could be found on a Wikipedia page. To improve the quality of answers, one can incorpo- rate desirable properties into the prompt which lead the LLM to produce more specific answers (such as "What good catalyst with low cost for reac- tion X?"). Additionally, L.MMs often hallucinate, producing answers without grounding in scientific fact. Achieving accurate answers with high speci-
'Our codebase and datasets are freely available at https: tiggthuubbcm wtt.tttth..oeoeeeeeeeeeeeeeetttt cwm/pnnl chemreasonner

---

MrSmmeitc:
Owrteer
Reason-via-catalys composition
Q+eLt''s think in terms bulk structural properties"
CO2 CD +H20
Query: What the top-3 catalysts that are cheap, perform reverse water gas shift reaction at lower temperature (<200 C) and demonstrate higher adsorption energy for both CO2 and H,?
148
211
Reason-via wia-electronic structure
Reasoon-via-targen properties
Q++ "Let's think in terms metal- support interaction"
Q- "Let's think in terms adsorption and orystal planes"
Figure 2: Illustration of the combinatorial thinking used by human experts to reason about a catalyst (best viewed in color). They successively "think in terms of"" different constraints and factors, each of which are related via scientific principles, and narrow down the set of possible candidates. Our Monte Carlo Reasoner emulates such cognitive thinking by prompting language model with different combinations, yielding tree-structured space of queries and potential candidates, and returns the optimal answer via efficient exploration of the possible space.
licity and which use key technical terminology (Fig. 2) is essential to earn the scientific community's trust and pave the way for the adoption of machine reasoning systems.
lenge of query decomposition. Considering the practical challenges of learning prompts that are both human comprehensible (a key consideration for scientists) and provide the best performance (Deng et al., 2022), we pursue a stochastic, heuris- tic bearch-based approach that leverages LLMs trained on scientific literature with sophisticated instruction following capabilities (Ouyang et al., 2022).
In this work, we focus the problem of prompt- ing LLM to return the top-k catalysts for given chemical reaction and generating the rea- soning for each candidate. In collaboration with researchers from the catalysis community, we de- velop a new dataset, BioFuels Question Reason- ing (BioFueIQR), consisting of complex reasoning questions and answers. We observe the reason- ing pathways used by domain scientists and con- clude that it is important for LLMs progress from "thinking step-by-step" to "thinking step-by-step in terms of relevant properties". In this setting, we are given a question which has some relevant cata- lyst properties P, |P| = n (e.g. ("crystal planes", "toxicity")) )) and we want to identify the best sub- set R C P, [ = r of properties for the language model to "think" in terms of. Considering that lan- guage models are sensitive to permutations of their inputs, there Pm (n-r)) possible prompts to search through. This goal can be accomplished by learning to prompt the L.LM with the most relevant subset of properties (Deng et al., 2022) or decom- posing the set into a sequence of chained queries (Dohan et al., 2022). In both cases, identification of the prompt-generating property subset becomes the limiting factor.
We formulate the task as a search problem which an agent performs a query in an uncertain environment (represented by the LLM) and de- termines query variant to pursue based on the evaluated reward. Given an initial query, struct a tree structure of these unique query vari- ants in order progressively refine the original query (the root) into property-specific variations (the leaves). Our methodology demonstrates im- provement over basic querying of the LLM without any additional training of the LLM. Instead, we use Monte Carlo Tree Search algorithm (MCTS) to perform stochastic search over the existing knowledge space of an LLM achieve more sci- entifically valuable answers.
n!!
Our second major contribution is demonstrating the efficacy of using scientific domain-specii re- ward function in -LMM-baed computations for our top-k catalyst problem. Estimation of the "adsorp- tion energy" of chemical structure is at the core of developing efficient chemical reactions (see Ap- pendix A.2 for details). Finding catalysts that can enable chemical reactions with the least amount external energy is key to developing environmen-
To solve this problem, we propose the Monte Carlo Reasoner (MCR), a generic heuristic search methodology that addresses the combinatorial chal-

---

tally friendly industrial processes. In this work, we implement such energy function specific consid- erations via LLM-derived reward function. Our experiments (using questions detailed in Table 4) show that even a simplistic reward function dramat- ically improves the specificity of answers and their associated reasoning from the LLM. In summary, make the following contributions:
to the parent node's answer candidates (switching between these possibilities is an action). Finally, we can take an action to change the type of cat- alyst requested (unary. binary, ternary, or -oxide catalysts). Clearly, the number of possible reason- ing pathways increases drastically with tree depth due to the possible combinations of actions. Thus, apply Monte Carlo Tree Search, an efficient method to optimize a sequence of actions with reward function, R.
1. We present Monte Carlo Reasoner (MCR), an al- gorithm prompt LLMS for zero-shot complex reasoning tasks involving combinatorial search.
In MCTS, each prompt P is stored as node in a tree T. where edges are prompt-action pairs (P, az). The search tree decides at each prompt which action to take to obtain the best reward based on previous results. Typically, prompt-action pairs are weighted by policy, which determines a-priori the importance of each action for prompt, given as prior probabilities. Here, we assign equal weight to all possible actions. Impossible actions are as- signed weight (see Appendix C.2).
2. We introduce new chemistry-foeuset dataset, BioRueIQR, that captures key reasoning chal- lenges in hypothesis generation and testing faced daily by scientists. We present in-depth qualitative analysis of MCR on BioFuelQR.
3. We demonstrate that a domain-specific reward function that represents a fundamental scientific concept can lead to dramatic improvement in the quality and specificity of LLM answers.
In MCTS, each edge stores a count N(P. ) weight representing a prior probability p a))), and the total downstream reward ((P.)) where
2 Monte Carlo Reasoner
Problem definition Our goal find the optimal prompt, po, which leads the LLM to return the best candidate catalysts for specific problem. Start- ing with a general initial prompt Po. we use a set of actions to automatically modify the prompt to improve the LLM output with respect to reward function, R.
V(P,a) Pr PeEncceesoo((P)
Σ
7 R(())
(1)
Here, 7 is a discount factor and is the (tree) dis- tance of p' from P. If there are no discovered successors to P, then we set V(P, a) 0. The search determines the next action to take with pol- icy PV),NN,):
For instance, suppose Po is the prompt given in Figure 2(1eft). Each prompt is a template, where we use actions a A to create better prompts, based on how experts might modify their own queries, so that the LLM will suggest superior catalysts. See Appendix C.1 for a more detailed explanation of the actions and prompt. By modifying prompts, we create tree of prompts, answers, and rewards, as demonstrated in Figure 1. We call a path from the root to Icaf node "reasoning pathway". These reasoning pathways can be constructed in several different ways. For instance, we can take an action to introduce additional catalyst properties to con- sider (such as "composition of metals" and "elec- tronic structure" in Fig. 2 (right)) so that the LLM will include or exclude certain catalysts in its an- swer. Also, for each prompt P after Po. we in- clude P's parent node's answer in P provide the LLM with additional context about the previ- ous answer. Further, each node, we prompt the LLM to produce catalysts with either "new elc- ments", "similar elements", or "different elements"
V(Pa)
NIPaJ tpp(P, VE, M(Paj
argmax arE.A
I+N(Pa)
(2)
where e is an exploration-exploitationn trade-off. The simulation starts at the root node each time and traverses the constructed tree until a new state is reached. Then, its answer and reward are calcu- lated, stored, and the upstream values V, are updated. This is repeated to generate the desired number of prompts (in our case 300). MCTS is superior to re-sampling methods because it avoids repeatedly sampling the same prompt and it is su- perior to brute-fore force tree search methods such BFS and DFS because it selects trajectories in the tree that show promising results.
2.1 Reward Function
Our reward function, R, measures the effective- ness of the catalysts proposed by the LLM for a given prompt, P. Here, we measure effectiveness

---

Table I: Final catalyst suggestion results. Np is number of prompts evaluated and dmaz maximum search tree depth. Values are averaged over evaluated examples.
Algorithm 1: Run MCR search. Here, a' indicates 8 action from root.
Require: L.M,, initial prompt Po. number of candidate catalysts k, number of prompts to generate M
Opentatalysis
BioFaclQR
Method
Reward
Np
daass
Reward
Np
dasse
CaT
2.04
NIA
2.77
1
NIA
CoT w/ Self seff-consistency
4.04
NIA
6.38
10
NIA
Initialize tree T. Define nodes P and edges (P.a)), discount y, stored values N a;). V(P, a;), (lp.((PP)))))) (())) and reward function 3 (Tol(T) 4 Po
ToT (breadhh wirs-wwarch)
138
253
5
MCR (ours)
12447
301
9.33
154
301
9.5
for an example). We collected two answers from catalysis researchers for a subset of 51 queries to observe different reasoning patterns and human bi- ases. See section C for details on dataset design. Baselines We benchmark MCR's performance with three recent methods: 1) Chain-OP Sai---O--Thhhh Thought (CoT) prompting (Kojima et al., 2022), 2) Self- consistency- -based CoT (Wang al., 2022), 3) breadth-firste fentth--irrt-seaach (BFS) based Tree-of-Thoughe (ToT) (Yao et al., 2023) (a contemporary work to ours). Experiments are based on GPT-3 text- davinci-003 2. Table shows MCR improves by 25.8% and 13% over the reward obtained by BFS OpenCatalysis and BioFueIQR, respectively, Performance improves by ~600% over CoT.
for -=,..... M do
Current Node Po = root(T)
Current Depth t - 0
while Pt E T do
Select af P(Pr, 0i; N, V,p) Increment N( ((P,, a²)
10
Pt+1 (P:))
Apply
action Increment
11
end
Send P. to LLM
Save Pt, (Pt, a²) T
14
15
+ R(P))
> Calculate reward
using answer from LLM
Query Cost Despite significant effort with the dataset creation, our results in Table are obtained from 1 11/130 queries. MCR and baselines are im- plemented using OpenAI text-davinci-O01 for con- sistency. MCR and ToT method is computation- ally expensive (Table 2), so evaluation of all 130 queries over all methods requires approximately 774,470 API calls, and could not secure com- pute capacity from OpenAT evaluate more than 11 queries with each method. We further discuss the limitations that arose in Limitations (4).
16
for 1' = 1,.....,00
+ Pr
17
end
19 end
20 return arg max (a(P))(R((P))
of catalyst by querying the LLM to produce ad- sorption energies for a given adsorbate electron volts (eV). We describe the prompt used to gener- ate the adsorption energy in Appendix C.1. The significance of adsorption energy for catalysis de- sign is explained in Appendix A.2. The reward is calculated
Key Takeaways We find that MCR's 's use of stochastic search prunes the more uniform explo- ration of search space conducted by ToT (Yao et al., 2023), Table shows given a maximum query limit, MCR was able search significantly deeper (re- ported by dmaa) than ToT. While MCR reached higher reward than ToT, MCR generated more nodes than ToT (see C.4). However, we are not able to definitively declare that both tree-based methods outperformed CoT and CoT w/ Self-consistency. To confirm if the increased reward over CoT in- deed translates into better reasoning quality, two catalysis experts compared the best answer gener- ated by MCR with the GPT-3 CoT implementation. Overall, the experts preferred MCR CoT (Fig-
R(P)
Σ (LM (3)
aE adsorbates
where C(P) the top-k catalysts from prompt P.
3 Experiments
Experimental setup We conduct our experiments on two new chemistry-focusee reasoning query benchmarks containing 130 queries (Table 2). We compile OpenCatalysis from the OC20 (Chanus- sot al., 2010) and OC22 (Tran et al., 2023) catalyst datasets (Zitiick et al., 2020). Second, we develop BioFucIQR query dataset targeting biofuels-focusee catalyst discovery (see Table
OpenAA has now discontinued support for this model. In the future, using open models to avoid this issue would he desirable, hut we found their performance lacking.

---

erties for augmenting a chemist's 's reasoning via use of a domain-specific reward function, enabling gen- eration of relevant scientific explanations with high specificity. MCR is a zero-shot reasoning method- ology that circumvents the need for large-scale, hard-to-obtain, domain-specifte training data. We apply it to catalyst research: a highly empirical, reasoning- Season scientific field dating back a cen- tury. Future work can investigate large-scale evalu- ation of our benchmark, integration with atomistic prediction models trained on quantum chemistry datasets for more trustworthy reward functions, and finetuned language models.
Question: What are the top-3 catalysts that perform the RWGS reaction at lower temperature (4200 C) and demonstrate higher adsorption energy for both CO2 and H2 (or facilitates both CO2 and H2 adsorption)?. Indlude candidate catalysts with following properties: low Provide scientific explanations and return a list of top 3 answers and their explanations as list of pairs. Let's think step- by step.
Bimetallic PiRe ( piatinum-thenium Alloy) - This alloy has been found to be highly effective for both CO2 and H2 adsorption, and it catalyzes the RWGS reaction at temperatures lower Ihan 200 using lower amount of energy. The PtRa alloy also has lower cost compared to other catalysts making a popular choice.
"The answer correct and consistent with Literature. Integrating with Rbantum does bring the cost down. will surely use
Ni-M--Cu: NI-Mn-Cu catalysts demonstrate one the highest known rate constants for the RWGS reaction, and also higher adsorption energy both CO2 and H2 compared to PIRe, Ni-Cu, and Cu-Zn.
Limitations
In this work, we consider applications of large lan- guage models in the scientific domain. In general, this comes with a number of limitations. First, LLMs display largely black box behavior, which is exacerbated by many strong models only being accessible as APIs. Second, generative modeling in the scientific domain is incredibly difficult to evaluate, requiring laboratory verification in many settings. Third, hallucination about factual informa- tion is a concern. One benefit of our method is that it provides reasonings based on refined prompts, which we show can be inspirational to domain ex- perts searching for a solution.
"I am really tmpressed by the spectfiatty of the suggection and use of its rate nonstant based justifloattio.
another aclleague told this, I would be really inteigued by their insight. am inllned to validate this experiment."
Ni-Al-CoMMo; Ni-Al-ooMo low-oost metal oxida catalyst that effective for both CO2 and H2 adsorption.
The catalyst mixure nickel (NI), aluminum (Al), cobalt (Co) and molybdenum (Mc). In addition to being low-oost, Ni-Al-oo-Mo is a highly active and selective presence active
sites and possesses strong acid sltes, which improves the activity of catalyst.
catalyst
"The answer is logically coherent and not incorrect, However, ignores the practical challenge combining four motals. Thia noommendation not actionable."
Our work demonstrates that tree-searcl methods have a strong value proposition over existing meth- ods for LLM reasoning (CoT, self-consistency etc.). Since ToT is contemporary to our methodology, important contribution of this work is demonstrat- ing the merit of tree-based reasoning approaches for complex scientific reasoning tasks; scientific reasoning is not discussed in (Yao et al., 2023). We do not claim that MCR is necessarily superior to ToT in all settings. In fact, further experiments have shown the two methods can be quite compa- rable. However, are limited in this work by the cost of experimentation that we cannot perform ideal comparison of MCR to ToT.
Figure 3: Domain expert evaluation of LLM answers on the reasoning path to the final node with highest reward.
ures 8, 9). Figures 5-9 illustrates the evaluation for one such query. Review from both experts (Fig- ures 8, 9) deemed the CoT response in Figure 6 incorrect and MCR correct (Figure 8).
The experts also evaluated how the prompts and LLM answers evolve as MCR searches deeper in the prompt tree (Figures 3 and 10)-in in many cases they found the LLM answers to be logically co- herent and in some cases even insightful enough for follow-up experimentation (see the second user feedback in Figure 3). Overall, both experts pre- ferred MCR for having higher specificity over CoT and reasoning in terms of correct properties (de- tailed in Figures 8, 9).
In particular, our reward function based on LLM outputs of scientific questions can be considered a limitation. However, it allows for much quicker validation of ideas and we find it to be an effee- tive proxy (which on its own is interesting). In the future, comparatively costly atomistic simula- tions can be used to replace our reward function. These can be quite time-consuming and computa- tionally expensive, so we focus on our algorithmic
4 Conclusion and Future Work
LLMs offer major promise to automate the cycle of scientific hypothesis generation and testing. Our work tackles the challenge of identifying key prop-

---

contribution in this work. Because of the efficacy we demonstrate using LLM rewards, it may also be possible to use a hybrid approach to save on computational chemistry simulations. This could initially leverage L.M embeddings as an initial re- ward to narrow down promising search sub-trees by selecting the most promising nodes in the first few layers of the search tree. Advanced simulations can then be used for searching final answers in these sub-trees. Alternatively, simulations can be used as limited-us use oracle like in active learning. We leave this for future work.
AI research institute program supported by NSF under award No. 2019897 and No. 2034562. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, ei- ther expressed or implied, of the U.S. Government, The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
Our method's improvement comes with higher cost of inference, similar to Tree-of- Thoughtt When doing inference locally, this may not be a problem. However, we utilize third-party APIs which are both expensive and rate-limited. We found existing open-source models trained on chemistry text did not possess sufficient instruction- following capabilities to be reliable or effective here. Thus, we were limited in quantity of exper- iments that could be done, as well as the models which could be accessed. This is because our ap- proach requires an average of 750 API calls per tree search. Although
Jens Artz, Thomas Müller, Katharina Thenert, Jo- hanna Kleinekorte, Raoul Meys. André Sternberg. André Bardow, and Walter Leitner. 2018. Sustain- able conversion of carbon dioxide: an integrated re- view catalysis and life cycle assessment. Chemical reviews, (4:4)44444504
Danii A Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent autonomous scientific research ca- pabilities of large language models. arXiv preprint
DrXiv:2304.05322
Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. 2023. Chemerow: Augmenting large- langelan models with chemistry tools. arXiv preprint 204.5:230..0337.
evaluate relatively few ini- tial questions, our in-depth expert-perfornn anal- ysis is based on ~,,200 queries.
Mustafa Canakci and Jon Van Gerpen. 1999. Biodiesel production viaacid catalysis. Transactions of the ASAE, 42(5))
Ethical Considerations
Cayque Monteiro Castro Nascimento and André Silva Pimentel. 2023. Do large language models un- derstand chemistry? a conversation with chatgpt. Journal of Chemical Information and Modeling, 63(6):1649-163
We propose zero-shot prompting methodology for LLMs that enables reasoning for complex queries in the scientific domain. Like most applications of LLMs, this has similar ethical considerations, espe- cially in regards to implicit biases from large-scale pretraining and the hallucination of false informa- tion. Thus, it is still important for human oversight and careful evaluation of language model output. One consideration of our method is that may en- able discovery of molccules, materials, or other scientific products which can be used for harm- ful applications (Urbina et al., 2022). Overall, we believe these downsides are outweighed by the ben- efits of this work to both the NLP community and other scientific communities which may benefit.
1. Chanussot, A Das, S Goyal, T Lavril, M Shuaibi, M Riviere, K Tran, Hleras-Domingo. C Ho, W Hu, et al. 2010. The open catalyst 2020 (oc20) dataset and community challenges. arxiv. arXiv.
Shuan Chen and Yousung Jung. 2022. A generalized- template- based graph neural network for accurate organic reactivity prediction. Nature Machine Intelli- gence, 4(9):772-780.
Austin H Cheng, Andy Cai, Santiago Miret, Gustavo Malkomes, Mariano Phiclipp. and Alán Aspuru- Guzik. 2023. Group selfies: a robust fragment-based molecular string representation. Digital Discovery.
Seyone Chithrananda, Gabe Grand, and Bharath Ramsundar. 2020. Chemberta: Large-scale self- supervised pretraining for molecular property pre- diction. arXiv preprint arXiv:2010.09883
Acknowledgements
This work is supported in part by the PNNL seed Laboratory Directed Research and Development (LDRD) program, Data-Model Convergence initia- tive. This work is also in part based upon work supported by the Molecule Maker Lab Institute: an
Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Man- ica. 2023. Unifying molecular and textual represen- tations via multi-task language modelling. arXiv preprimt 2rXiv:2301. 12586.

---

Antonia Creswell and Murray Shanahan. 2022. Faith- ful reasoning using large language models. arXiv preprint ar 2rx8 14271.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. 2018. Junction tree variational autoencoder for molecular graph generation. In International confer- ence on machine learning, pages 2323-2332. PMLR.
Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-infrrencc: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.0m 220.002.......
Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205. 18822
Yolanda A Daza and John N Kuhn. 2016. Co 2 con- version by reverse water gas shift catalysis: compari- son of catalysts, mechanisms and their consequences for co 2 conversion to liquid fuels. RSC advances, 6(55):49675-4991..
Shyam Kattel, Ping Liu, and Jingguang G Chen. 2017.
Tuning selectivity of co2 hydrogenation reactions at the metal/oxide interface. Journal of the American Chemical Society, 12 139((9):9739-97)) -9554.
Mingkai Deng. Jianyu Wang, Cheng-Ping Hsich, Yi- han Wang. Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing discrete text prompts with reinforcement learning. Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 3369-3391, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint (tt::22055119166
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pas- cal Friederich, and Alan Aspuru-Guzik. 2020. Self- referencing embedded strings (selfies): A 100% ro- bust molecular string representation. Machine Learn- ing: Science and Technology, 1(4):045024.
David Dohan, Winnie Xu, Aitor Lewkowyez, Ja- cob Austin, David Bieher, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl- Dickstein, et al. 2022. Language model cascades. arXiv preprint 200322...1
Tuan M. Lai. Chengxiang Zhai and Heng Ji. 2023. Knowledge-enhanee biomedical language models. In Journal of Biomedical Informatics.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022. Translation between molecules and natural language. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 3755413, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao. Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar. 2022. Multi-modal malecule structure-tex model for text-based retrieval and editing. arXiv preprint arXiv:2212. 10789.
Shengchao Liu, Jiongxiao Wang. Yijin Yang, Cheng- peng Wang. Ling Liu, Hongyu Guo, and Chaowei Xiao. 2023a. Chatgpt-powert conversational drug editing using retrieval and domain feedback. arXiv preprint arXiv:2305. 18090.
Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021. Text2Mol: Cross-modal molecule retrieval with nat- ural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 595-607, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Shengehao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar, 2023b. A text-guide eext-guided protein design framework. arXiv preprint a3X1v:2302.04671.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Mar- win Segler, Joshua Meyers, Marco Fiscato, and Mo- hamed Ahmed. 2020. Molecular representation learn- ing with language models and domain-relevant auxil- iary tasks. arXiv preprint arXiv:201 1.13230.
repressenatt
Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. 2023c. Molxpt: Wrapping molecules with text for generative pre-training. arXiv preprint arXiv:2305. 10688.
Johannes Gasteiger, Florian Becker, and Stephan Gün- nemann. 2021. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-688 -6002.
Ahmad Mukhtar, Sidra Saqib, Hongfei Lin, Mansoor UI Hassan Shah, Sami Ullah, Muhammad Younas. Mashallah Rezakazemi, Muhammad Ibrahim, Abid Mahmood, Saira Asif, et al. 2022. Current status and challenges in the heterogencous catalysis for biodiesel production. Renewable and Sustainable Energy Reviews, 157:1 12012.
Glen M Hocky and Andrew D White. 2022. Natural language processing models that automate program- ming will transform chemistry research and teaching. Digital discovery, 1(2)799-83.
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, and Berend Smit. 2023. Is gpt- 3 all you need for low-data discovery in chemistry? ChemRxiv preprint.
Jens K Nørskov, Frank Abild-Pedersen, Felix Studt, and Thomas Bligaard. 2011. Density functional theory in surface chemistry and catalysis. Proceedings of the National Academy of Sciences, 108(3):937-943.

---

NVIDIA Corporation, 2022. Megamolbart v0.2.
Richard Tran, Janice Lan, Muhammed Shuaibi, Bran- don M Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, et al. 2023. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocata- lysts. ACS Catalysis, 13(5):3066-3084,
OpenAl. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Emma P Tysinger, Brajesh K Rai, and Anton V Sinitskiy. 2023. Can quickly learn to "translate" bioactive molecules with transformer models? Journal of Chemical Information and Modeling, 63(6):1734 1744.
Mayk Caldas Ramos, Shane S Michtavy, Mare D Porosoff, and Andrew D White. 2023. Bayesian opti- mization of catalysts with in-context learning, arXiv preprint arXiv:2304.05344
Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. 2022. Dual use artificial-intelligee powered drug discovery. Nature Machine Intelli- gence, 4(3):189-19)
Kristof Schüitt, Huziel E Sauceda, P-J Kinder- mans, Alexandre Tkatchenko, and K-R Müller, 2018. Schnet-a a deep learning architecture for molecules and materials. The Journal of Chemical Physics,
Andreu Vall. Sepp Hochreiter, and Günter Klambauer. 2021. Bioassayclr: Prediction of biological activity for novel bioassays based on rich textual descriptions. In ELLIS MLAMolecules workshop.
148(24):241722.
Philippe Schwaller. Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. 2019. Molecular transformer: chemical reaction
Alain C Vaucher, Philippe Schwaller, Joppe Geluykens, Vishnu H Nair, Anna Iuliano, and Teodoro Laino. 1021. Inferring experimental procedures from text- based representations of chemical reactions. Nature communications, 12(1):2573.
model for ancertainty-calibratt prediction. ACS central science, 1725:157--
Philippe Schwaller, Daniel Probst, Alain C Vaucher, Vishnu H Nair, David Kreutter, Teodoro Laino, and Jean-Louis Reymond. 2021. Mapping the space of chemical reactions using attention-based neural net- works. Nature Machine Intelligence, 3(2):144
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 11771.
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. 2023. Enhancing activity pre- diction models in drug discovery with the ability understand human language. arXiv preprint arXiv:2303.03363.
Logan Ward, Jenna A Bilbrey, Sutanay Choudhury, Neeraj Kumar, and Ganesh Sivaraman. 2021. Bench- marking deep graph generative models for optimiz- ing new drug molecules for covid-19 19, arXiv preprint arXiv:2102.044
Jacek K Stolarezyk, Santanu Bhattacharyya, Lakshmi- narayana Polavarapu, and Jochen Feldmann. 2018. Challenges and prospects in solar water splitting and co2 reduction with inorganic and hybrid nanostruc- tures. ACS Catalysis, 8(4):3602-366 363-3635...
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 1rXi:22001.1990
David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical infor- mation and computer sciences, 28(1):31-33
Thomas J Struble, Juan C Alvarez, Scott P Brown, Mi- lan Chytil, Justin Cisar, Renee L DesJarlais, Ola En- gkvist, Scott A Frank. Daniel R Greve, Daniel J Grif- fin, al. 2020. Current and future roles of artificial intelligence medicinal chemistry synthesis. Jour- nal of medicinal chemistry, 63(16)8866-8868
David Weininger, Arthur Weininger, and Joseph L Weininger. 1989. Smiles. 2. algorithm for genera- tion of unique smiles notation. Journal of chemical information and computer sciences. 29(2):97-10
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiang- meng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji- Rong Wen. 2022. A molecular multimodal founda- tion model associating molecule graphs with natural language. arXiv preprint arXiv:2209.0548)
Andrew D White, Glen M Hocky, Heta A Gandhi. Mehrad Ansari, Sam Cox, Geemi P Wellawatte, Sub- arna Sasmal, Ziyue Yang, Kangxin Liu, Yuvraj Singh, et al. 2022. Do large language models know chem- istry? ChemRxiv preprint.
Ross Taylor, Marein Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science, arXiv preprint arXiv:2211.09000
Andrew D White, Glen M Hocky, Heta A Gandhi, Mehrad Ansari, Sam Cox, Geemi P Wellawatte, Sub- arna Sasmal, Ziyue Yang. Kangxin Liu, Yuvraj Singh, et al. 2023. Assessment of chemistry knowledge in

---

large language models that generate code. Digital Discovery, 2(2):668-377 -766.
(Nørskov al., 2011). Currently, there is lack of data and design guidelines for heterogencous catalysis because the computational cost of obtain- ing accurate theoretical models for such complex systems is currently prohibitively high while high- throughput experimental methods that have been applied successfully in related fields have not yet been thoroughly explored (Yang et al., 2019).
Hanwen Xu and Sheng Wang. 2022. Protranslator: zero-shot protein function prediction using textual description. In Research Computational Molecu- lar Biology: 26th Annual International Conference, RECOMB 2022, San Diego, CA, USA, May 22-25, 2022, Proceedings. pages 279 2994. Springer.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023. Protst: Multi-modality learning of pro- tein sequences and biomedical texts. arXiv preprint arXiv:2301. 12000
Experimental validation of a new catalyst and its performance is expensive (Yang et al., 2019). Artifi- cial intelligence-driven computing approaches aims to accelerate such discovery by down-selecting can- didates that are most promising and merit extensive evaluation in a laboratory (Ward al., 2021). The past few years have seen lot of developments for applying Al to chemistry that range from predict- ing properties of atomistic structures, or outcomes of reactions (Schwaller et al., 2019; Chen and Jung, 2022). Generative models (Jin et al., 2018) or deep reinforcement learning methods (You et al., 2018) have demonstrated abilities to propose novel chem- ical compounds that satisfy unique property con- straints, and then suggest synthesis pathways for producing such compounds (Struble et al., 2020). Generally, such models are trained on representa- tions of atomistic structures, or reactions between multiple structures (Struble et al., 2020; Chen and Jung, 2022).
Shenzhen Xu and Emily A Carter. 2018. Theoretical in- sights into heterogeneous (photo) electrochemical co2 reduction. Chemical reviews, (19(11):66) 6669.
Wenhong Yang, Timothy Tizhe Fidelis, and Wen-Hur Sun. 2019. Machine learning in catalysis, from pro- posal to practicing. ACS omega, 5(1))83-88.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovee. 2018. Graph convolutional policy net- work for goal-directed molecular graph generation. Advances in neural information processing systems, 31.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable human professionals. Nature commu- nications, 11):
A.2 Motivation for molecular energy prediction as a reward function
Electronic structure calculations play crucial role in developing atomistic-level understanding of the interaction of liquid or gaseous molecules with solids, functional of the topological property of the solid surface (Nørskov et al., 201 1). Much of the literature from machine-learning for atomistic systems have focused on training system-level prop- erties such as potential energy functions (Schürt et al., 2018; Gasteiger et al., 2021). The follow- ing paragraph explains why estimating the energy functions associated with molecular structure is critical to discovering processes with lower energy requirements.
Wenyu Zhao, Dong Zhou, Buqing Cao, Kai Zhang, and Jinjun Chen. 2023. Adversarial modality alignment network for cross-modal molecule retrieval. IEEE Transactions on Artificial Intelligence.
C Lawrence Zitiick, Lowik Chanussot, Abhishek Das, Siddharth Goyal, Javier Heras-Dominng, Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, al. 2020. An introduction to electrocat- alyst design using machine learning for renewable energy storage. arXiv preprint arX1v:2010.09435.
A Background
A.1 Scientific Drivers from Catalysis
The amount of usable energy for physical sys- tem with constant temperature and pressure is re- ferred as the Gibbs free energy, or Gibbs energy and defined as: G II TS, where H is the energy contained in the bonds between atoms, T is the temperature and is the entropy (Zitnick et al., 2020). The entropy of a system increases when
Discovery of novel catalysts is essential for accel- erating the transition sustainable future. De- spite the significant progress in the development of highly efficient catalysts, heterogeneous cataly- sis remains largely an empirical science owing to the complexity of the underlying surface chemistry

---

molecules break their bonds and decreases when they form new ones. The computation of H in- volves the potential energy between atoms. When Gibbs energy is negative, it means that the energy contained in the bonds is higher, and a system will naturally approach a lower energy state. Thus, a reaction or process will proceed spontaneously. On the contrary, a positive Gibbs energy indicates that the extrinsic energy is required enable target process or reaction. The path decarbonization lies with discovering chemical processes that re- quire lesser amount of extrinsic energy.
tific tools (Bran et al., 2023; Boiko et al., 2023; Liu al., 2023a). In general, these studies assess the inherent chemistry knowledge LLMs and the effect of integrating chemistry data via in-context learning or finetuning. This differs from our contri- bution, where we propose an algorithmic approach for improving model output using domain-specific rewards. A future research direction may be able to incorporate these two approaches together for exciting results.
B.3 Chain-of-Thought (CoT) Variants
Several works have considered improving LLM output on complex reasoning tasks via formulating multiple queries. (Creswell et al., 2022) explored the decomposition of complex queries into smaller, more reliable operators. (Creswell and Shanahan, 2022) presents methodology for generating the answer in a step-by-step fashion and uses another model or function to pick the top-ranked answers, and avoids hallucination by constraining the output to a narrower set. (Jung et al., 2022) proposed an alternate approach to generate a tree of possible explanations (both correct and incorrect), and then analyzes their relationships to infer the correct set of answers. (Wang et al., 2022) improves reliability by sampling multiple explanations and answers from the model and then selecting the final answer that appears most often. Tree-of-Thought (ToT) (Yao et al., 2023) generalizes the CoT approach to enable exploration over coherent units of text (thoughts) to perform deliberate decision making by considering multiple different reasoning paths. We benchmark against (Kojima et al., 2022; Wang et al.. 2022; Yao al., 2023) in our work.
B Related work
We begin with providing an overview of the broader literature around language models and their appli- cations into chemistry, then specifically focus on large-language models. Finally, we finish with an overview of various chain-of -in--off-tte thoutht prompting methods that have been instrumental in improving the reasoning capability of LLMs.
B.1 Multi-modal models for Chemistry
Recently, advances in NLP have found surprising, strong results in the chemistry domain by train- ing LLMs (Fabian et al., 2020; Chithrananda et al., 2020; Vaucher al., 2021; Schwaller et al., 2021; NVIDIA Corporation, 2022: Tysinger et al., 2023) on string representations of molecules (Weininger, 1988; Weininger et al., 1989; Krenn ct al., 2020; Cheng et al., 2023). To enable higher-level control over molecular design, multi-modal models (Ed- wards et al., 2021; Vall et al., 2021; Zeng et al., 2022; Xu and Wang, 2022; Su et al., 2022; Seidl et al., 2023; Xu et al., 2023; Zhao et al., 2023; Liu al., 2023b) have been proposed. Existing work focuses on cross-modal retrieval (Edwards al., 2021; Zeng et al., 2022), translation (Edwards et al., 2022; Liu et al., 2023c; Christofidellis et al., 2023), and editing (Liu et al., 2022).
C Dataset Design
We propose two task datasets related to catalyst design: the first is derived from the Open Catalyst (OC) Project (Zitnick et al., 2020) and the second consists of complex reasoning queries designed by catalysis experts. Our multi-disciplinary team in- volves researchers who actively work on designing new catalysts for bio-fuels development.
B.2 LLMs for Chemistry
Due to recent progress in chat-oriented models such as GPT-4 (OpenAI, 2023), interest has grown in uncovering chemical knowledge and molecu- lar discovery from existing general LLMs (Hocky and White, 2022; White et al., 2022, 2023; Cas- tro Nascimento and Pimentel, 2023). This has been extended work in the few-shot setting (Ramos et al., 2023; Jablonka et al., 2023). In particular, there is an interest in endowing LLMs with scien-
C.1 Action-Driven Prompt Design
To apply MCR catalyst discovery, define a set of prompt templates and a set of actions modify the fields of those templates. The exact structure of the prompt templates varies between

---

task datasets, but there are several common ele- ments. Table 3 lists the action types that we use. Firstly, all prompts query the language model to return "top-k" catalysts as where k is given by the user. Secondly, each template has list of "in- clude properties" and "exclude properties", which specify contexts for the LLM to consider positively when determining catalysts to include and exclude. respectively. Next, each prompt in both ToT (Yao et 2023) breadth-first-serch and MCR after the initial prompt uses the previous list of candidate catalysts. The LLM is prompted to either include elements "similar to" or "different from" the previ- ous list or to "include elements from" or introduce "new elements to" the list. Finally, the template includes field to prompt for certain kind of cata- lyst: unary, binary, trinary, and oxides. Of course, a prompt can have no specification on the catalyst type.
Table 2: Dataset Summary
OpenCatalysis
BioFuelQR
# #Queries
79
51
Adsorbates
Reactions
X
/
Human
X
Answers
Include Property and Add Exclude Property action types Table 3. The 'catalyst label' field de- termines which kind of catalyst the LLM should return. It's value is set by the Change Catalyst Type action and the Toggle Oxide action can set this field to query for oxide catalysts. Finally, the candidate list statement is a phrase built from the list of candidates generated by the parent prompt. Since the candidate list can have an impact on the output of the LLM, we include an action to re-run the previous query with the candidate list from the previous query's output.
The specific template depends the task dataset and the original query.
Possible actions are weighed with equal prior probabilities p (see Section 2) and impossible ac- tions are given prior probability zero. Actions are impossible if they: add property to a list which already has that property, add relationship the previous candidate list when there is no candidate list, or they would allow the next action to not have a relationship to the previous candidate list while the candidate list is not empty.
C.2 Open Catalyst Dataset
The Open Catalyst project (Zitnick al., 2020) is an online repository of datasets intended for train- ing surrogate models for computational chemistry simulations related to catalysis. The dataset con- tains hundreds of thousands of adsorption energies for adsorbate-catalys pairs calculated using den- sity function theory (DFT), an accurate method for computing energies of atomic configurations. We use the Open Catalyst dataset build an eval- uation dataset consisting 79 adsorbates. This dataset targets the LLM's ability to reason about the adsorption of specific adsorbates.
C.3 BioFuelQR Dataset
Our application focus is driven by the design of catalysis for reverse order gas reaction that is key to generation of synthetic biofuels with higher se- lectivity (Canakci and Van Gerpen, 1999; Daza and Kuhn, 2016; Kattel et al., 2017; Artz et al., 2018; 8; Stolarczyk et al., 2018; Xu and Carter, 2018; Mukhtar et al., 2022).
We use the following template for this dataset: Generate a list of candidate (catalyst label} {candidate list statement) for the adsorption of (adsorbate). {include statement) (exclude statement] Let's think step-by-step and return list of top (k) answers and their explanations as a list of pairs.
Questions in the BioFuelQR dataset uses the following template:
What are the top-3 (catalyst label} {candidate list statement) that perform the RWGS reaction at a lower temperature (<200 C) and demonstrate higher adsorption energy for both CO2 and H2 (or facilitates both CO2 and H2 adsorption)?.
Here, } denote fields that need to be filled. The fields provided in the base query are the number of candidate catalysts 'k' (k=5 for the OC dataset) and enters the adsorbate symbols 'adsorbate' from the OC dataset. 'Include statement' and 'exclude statement' * are phrases built from the list of properties to include and exclude, respec- tively. These statements are affected by the Add
(include statement) (exclude statement] Provide scientific explanations and return a list of top 3 answers and their explanations as a list of pairs. Let's think step-by-step.

---

Table 3: List of actions and their possibilitics.
duce computational cost, we restricted the number of actions the top 12 actions with the highest prior probability p P, az). This way, we reduce the number of actions simulated at each step. If there are not 12 actions with nonzero prior probability for node, we generate many children as possible. This happens. for instance, at the second level of the search tree, where the action "change relation to previous answer" must be taken (of which there are 4 possibilitiie). This is because they will pass their candidate catalysts to their successor prompts. The ToT method was run for 5 steps to generate a tree with depth 5. If all actions were possible at every level, we would generate 300 nodes in BFS (not including the root node), but only 252 nodes were generate on average. Still, we were able to select at least 6 best nodes each level. We did not experience similar discrepancy in MCR because MCR has a more flexible branching policy. The average observed number of nodes the final trees is reported in Table
Action
Possible Values
possible
Type
Add Prop- erty to In-
high activity, high selec- tivity, high stability, nov- elty, low cost, low tox- icity, high surface area. high porosity, crystal facet, availability
11
clude
Add Prop- erty to Ex-
low activity, low selec- tivity, low stability, high cost, high toxicity, low dispersion, low porosity, high scarcity
9
clude
Change
unary catalyst, binary cat- alyst, trimary catalyst. cat- alyst
Catalyst
Type
Toggle
on/off
Oxide
Change
including elements that 4 are different from, in- cluding elements similar Prev.
Relation
Answer
to, introducing new elc- ments to, including elc-
ments from
We did not include the depth depth-first- sirss- method from Tree-of-Thought because search does not support a specific ending criterion.
Repeat
N/A
Prompt
MCR For MCR, we set a discount factor,
C.4 Baseline implementations
0.9 and exploration-exploittiio trade-off of 15 control the branching and depth of the search tree. Generally, decreasing y decreases the length of chains the search tree while increasing e increases the branching of the tree. We generated 300 nodes after the root node, meaning 301 nodes were in the final search tree.
Here define the parameters for the evaluations of the Baseline and MCR methods.
Chain-of-Thought (CoT) For the CoT baseline, we generated prompt for each query following the templates described in Appendix C.1. We evaluated 9 adsorbates from the Open Catalysis Dataset and 2 prompts from the BER dataset. For CoT, we simply send one prompt to the LLM to generate a list of candidate catalysts, including the phrases "Provide a scientific explanation" and "Let's 's think step-by-step". The reward of the result is reported. CoT with Coniisssssy For the self consis- tency baseline, the query was evaluated 10 times independently using the same prompt from COT. We checked the answer for consistency. However, there was consistency between the top-k an- swers from the LLM over the 10 trials. Perhaps due to the large diversity in catalyst compositions. Thus, the reward estimate returned in Table is simply the maximum reward over the 10 trials.
MCR utilizes the policy in Equation 2 de- termine which actions to carry out at which step. However, the policy must be modified in two cases. First, if a node a leaf node, the policy is replaced by the prior probability distribution over actions, p( (Prs a4) (see Section 2). Secondly, if a node ac- tion pair has no visits (N(P,, a;) = 0) then the first term of Equation is dropped to avoid dividing by zero.
C.5 Reward Query
To query the language model to return adsorption energies, we use another prompt template:
Generate list of adsorption energies, in eV, for the adsorbate (adsorbate) to the surface of each of the following catalysts: (candidate list]. Return the adsorption energies as a list of only {len(candidate list)) numbers in the order
Tree-of-Thoughe (ToT) For ToT, keeping com- putational cost in mind, we branching factor b = 6. This controls the number of nodes expanded at each point in the search. Thus, at each level the nodes with the top rewards are expanded. To re-

---

specified.
The LLM should return list of numbers which can be averaged to produce a final energy. Since adsorption energies are negative we take the abso- lute value of the numbers listed by the LLM. units are in eV.. If multiple adsorbates are given, as in the BFR examples, multiple prompts are generated and the results are summed over. Occasionally, the LLM does not give output that can be easily parsed into a list of floats. In these cases, the query is re-run maximum of times. Such examples include but are not limited to uncommon delimiters and sporadic phrases in the output.
D Qualitative Analysis

---

Questions
Answers
Reasoning criteria
What are the top catalysts with higher adsorption energy for both CO2 and Ha (or facilitates both CO2 and Hz
Noble metal catalysts such as Pt, Rh, Pd, Ru supported on reversible metal oxide iee., Ce02 (cerium oxide), TIO2 (Titanium dioxide)
Adsorption energy
Electronic structures
adsorption)
Metal-support interaction
While noble metals active for hydrogen adsorption, reversible metal oxide facilitates the CO2 adsorption. The oxygen vacancy present in the reversable metal oxide facilitates C-O bond cleavage of CO2. Generally, interface sites are coined as the active sites. Higher metal-support interaction key for their high activity.
Identify the catalysts that exhibit weak adsorption energy for co (product)
Metal catalysts such Au, Ag, Zn demonstrate adsorption energy corresponding CO
Adsorption energy
What are the top catalysts that perform RWGS reaction at lower temperature (<200 'C)
Atomically dispersed Pt, Rh, Pd and Ru catalysts on CeO2 or TIO2. Atomically dispersed metal sites bind CO very weakly due to their unique electronic structure consequently exhibits high selectivity CO
Electronic structures
Is RWGS reaction structure sensitive?
Yes. Particles that are typically <2 nm more active for the RWGS reaction. smaller particle size enables higher which
Electronic structures
Adsorption on specific crystal planes
metal-support interaction facilitates the CO2 activation and facilitates the reaction. In general step and kink sites i.e., coordinatively unsaturated sites (such as 110 planes) exhibit weaker CO binding energy.
What the crystal planes that most active for the adsorption of CO2 for noble metal catalysts?
Open structure such (100) (110) planes of metal catalyst are more active towards CO2 activation. Binding energy of CO2 is higher these crystal surfaces
Surface structure
Adsorption specific crystal planes
What are the top bimetallic catalysts that are cheap as well as demonstrate higher adsorption energy for both CO2 and (or facilitates both and adsorption)
oxide supported PtRe, PtCo, PtNi bimetallic alloys demonstrate higher adsorption energy for both CO2 and Addition of Re, and other transition metals reduces cost based catalyst
Structural properties
Identify which three catalysts bind phenol during hydrodeoxygenation reaction with High Conversion.
Zeolite supported Pt, Rh and Ru catalyst. Pt, Rh and Ru binds phenol strongly and activate for the hydrogenation. Acidity of zeolites facilitates C-O bond cleavage i.e.,
Structural properties
Adsorption energy
hydrodeoxygenation
Identify which three catalysts can bind acetic acid in hydrogenation reaction with High Conversion.
Ru/C, Ru/AI203 and PVC. Ru/C catalysts. Ru/C exhibited higher conversion compared other support materials
Structural properties
Metal-support interaction
Figure 4: Example queries from the BioFuelQR dataset representing reasoning with different combinations of chemical descriptors.

---

Question: What are the top-3 catalysts that perform the RWGS reaction at a lower temperature (<200 C) and demonstrate higher adsorption energy for both CO2 and H2 (or facilitates both CO2 and H2 adsorption)?. Provide scientific explanations and return list of top answers and their explanations a list pairs. Let's think step-by-step.
Answer by human expert (A1) Noble metal catalysts such as Pt, Rh, Pd, Ru supported on reversible metal oxide ie, CeO2 (cerium oxide), TiO2 (Titanium diaxide). (A2) Pt, Rh, Pd single atom catalyst supported metal oxide.
Figure 5: Example question and human answer from our compiled QA-dataset.
Step 1: Identffy referred catalysts for the RWGS reaction The reverse water-gas shift (RWGS) reaction is used to convert CO2 and H2O into CO and H2. is important reaction the Haber-Bosch process for the manufacturing ammonia. Many catalysts can facilitate the RWGS reaction, including, e.g, transition metal oxides, ionic liquids, zeolite-brr based materials.
Step 2: Identify the catalysts' desired properties The desired properties for catalyst to perform the RWGS reaction at low temperatures (<200'C) and to demonstrate higher adsorption energy for both CO2 and H2 include: high catalytic activity, high selectivity, good thermal stability, and robustness. Additionally, the catalyst should demonstrate good adsorption capability for both CO2 and H2.
Step 3: Identify the top-3 catalysts The top-3 catalysts for the RWGS reaction at a lower temperature (<200 C) and demonstrate higher adsorption energy for both CO2 and H2 include:
Transition metal oxides: Transition metal oxides (eg. iron oxides and copper oxides) demonstrate high adsorption energy both CO2 and H2, and provide high catalytic activity low temperatures (<200 C).
Additionally, they are chemically and thermally robust.
lonic liquids: lonic liquids provide high selectivity and good thermal stability for the RWGS reaction. Due to their high surface area, they can also promote good adsorption capability for both CO2 and H2. Zeolite-based materials: reolite-based materials provide good thermal stability and robustness for the RWGS reaction. Additionally they exhibit higher adsorption energy for both CO2 and H2 when compared other catalysts.
Figure 6: Response to above query returned by Chain-of of promting with GPT-3,
1. Ruthenium Oxides; Ruthenium oxides have been studied as catalysts for a variety of reactions, including the RWGS reaction. These catalysts have very high binding energy and are relatively inexpensive, making them good candidate for this purpose.
Explanation: Ruthenium axides possess a high binding energy which enables them to achieve the RWGS reaction at very low temperatures (<200C) and demonstrate strong adsorption properties both for CO2 and H2, making them well-suited for this purpose.
2. Hafnium Oxides; Hafnium oxides have been studied for their ability catalyze the RWGS reaction with high binding energy and low cost. These catalysts are very stable and have been found to have stronger adsorption for both CO2 and H2 compared to the Ni-Co, Zeolite-Supportee Supported Metal Oxides, and Doped Titanium Oxide catalysts.
Explanation, Hafnium oxides are good choice for the RWGS reaction they have higher binding energy, and can catalyze the reaction lower temperatures (<200C) with higher adsorption rates for both CO2 and H2, as well as being cost effective.
3. Setinumm-utheniumn Alloy; Pt-Ru alloys have been studied for their potential use as catalysts in variety of reactions, including the RWGS reaction. They are able to promote the reaction at low temperatures (=200C), and have been found achieve high binding energy and strong adsorption for both CO2 and H2 due to the combination of the two metals in the alloy. Explanation; auttnnmm-rutheiin
alloys have been found to be effective catalysts for the RWGS reaction at low temperatures (4200C) due to their high binding energy, and also due to their strong adsorption properties for both CO2 and H2, These alloys are also relatively inexpensive, making them good choice for this purpose.
Figure 7: Response to above query returned by MCR.

---

1) Quality: How did the Al methods matched your answer? Answer:
One answer from GPT-3.5 3.5 can be considered partially correct (transition metal) while the Monte Carlo Reasoner partially matched my answers and reasoning for the noble metal catalysts' RWGS activity. Both models were not able to address the requirement of catalyst activity of less than <200C. The Monte Carlo Reasoner identified noble metals, Platinum and Ruthenium. Hafnium something that would not have considered. For catalysts that have high adsorption energies for CO2 and H2, my answers were three Pt-based catalysts (PtRe/SiO, Pt/CeO, and Na-doped Pt/ZrOz). also identified Ni-based (Ni/La- Wi/LLa-dopedCeO, NiCu, Ni/Ce- Zr-O) and Cu-based (4Cu-Al,0, catalysts from my research. My reasoning is that catalysts that would be expected to demonstrate higher adsorption energies for both CO2 and H₂ would contain noble and base metals such as Pt, Ru and Ni supported on oxides with high level of oxygen vacancies facilitate high adsorption energies for both CO2 and Hz. From the manuscripts that reviewed that have tested RWGS at 200C, none resulted in any significant CO2 conversion (> 5%). Lastly, calculated equilibrium constants from another paper reported 0.0043 at 200C and 0.0830 at 400C.
2) Specificity: Which Al method matched the specificity of your explanation?
Answer: Both methods didn't completely match the specificity of my explanation, but would choose the Monte Carlo Reasoner because it identified Pt, even part of bimetallic catalyst. However, even Pt catalysts do not have much activity (> 5% CO2 conversion) for RWGS <200C.
3) Reasoning: Which Al methods used similar chemical descriptors as yours to reason about?
In part, the Monte Carlo Reasoner. It correctly identified strong adsorption properties for both CO, and H, for the noble metal catalysts.
Answer:
4) Did the Al method return any wrong answer?
Answer: Yes, they both did. GPT-3.5's claim that the ionic liquid and zeolite were good catalysts for RWGS was incorrect. They were not identified as RWGS catalysts in my search. Transition metal catalysts, like Ni, Cu, and their alloys, were identified as potential RWGS catalysts but they are not active at <200C. The Monte Carlo Reasoner incorrectly identified Hafnium catalyst. However, conducted a follow-on on search because am not very familiar with its chemistry. Hafnium seems to be able activate CO2 but whether it can produce CO selectively through RWGS was not conclusive.
potential RWGS
Are any of the Al-generated answers novel/superior to the human expert answer?
Answer: The Hafnium suggestion was novel for me, but was not superior to the human expert answer.
Figure Comparison MCR vs standard Chain- Thain-Of-Thouuee Of-Thought prompting (via GPT-3) by domain expert

---

1) Quality: How did the Al methods matched your answer? Answer:
The answer provided by the Monte Carlo Reasoner is much closer my expectation. Traditionally, the catalyst for the RWGS reaction is composed of both noble metal and reversable metal oxide, which activate H2 and CO2 respectively. The catalyst systems provided by the Monte Carlo Reasoner falls into this category and are expected to show activity towards RWGS reaction. However, their performance lower temperature is still debatable and subjected to experiment. On the other hand, none of the catalyst provided by GPT-3.5 known to be a good catalyst for H2 activation. Although their activity towards activation of CO2 has been demonstrated, performance these catalysts at lower temperature is highly unlikely.
2) Specificity: Which Al method matched the specificity of your explanation?
The catalyst system provided by Monte Carlo Reasoner are more relevant to RWGS reaction. PtRu is known to activate both CO2 and H2 at lower temperature and thus, would be an interesting system to consider for RWGS reaction at lower temperature.
Answer:
3) Reasoning: Which Al methods used similar chemical descriptors as yours to reason about?
Answer: The catalyst system provided by Monte Carlo Reasoner considered adsorption energy of both CO2 and H2 the descriptor. This is consistent as adsorption energy has always been used as major descriptor in heterogeneous catalysis. Consequently, the answers provided herein are associated with higher confidence.
4) Did the Al method return any wrong answer?
Answer: Zeolite materials and ionic liquid suggested by GPT-3.5 is not correct as these materials are not active for hydrogen activation and do not expect to show any activity towards RWGS. Although, transition metal oxide activate hydrogen but requires higher temperature. The catalyst systems provided by Monte Carlo reasoner are more relevant to RWGS reaction, however, the choice of the metal oxide catalyst system may not correct as metal oxides are known to perform towards RWGS reaction only at higher temperature.
5) Are any of the Al-generated answers novel/superior to the human expert answer?
Answer: Suggestion for Hafnium oxide materials could be considered as novel as this has not been considered in the literature. but its activity towards RWGS reaction remains questionable.
Figure 9: Comparison of MCR vs standard Chain-Of Of-Thought prompting (via GPT-3) by domain expert 2.

---

Question: What are the top-3 catalysts that perform the RWGS reaction at lower temperature (<200 and demonstrate higher adsorption energy for both CO2 and H2 (or facilitates both CO2 and H2 adsorption)?. Provide scientific explanations and return list top answers and their explanations of pairs. Let's think seep-by-step.
Bimetallic Phosphates, Copper-based catalysts, Zeolite-like materials.
Averaged reward: 2.22
"This search is starting really. broad. While Bimetallic Phosphates is acceptsble direction, would prefer to specific instance suggested than category."
Modification of Copper-based Catalysts, Bimetallic Phosphates, Zeolite-like Materials.
Averaged reward: 5.03
Substitution of alkali and transition metal elements high temperaturos, Addition of protected copper(()-oxo groups, Coordination of protonated phosphines. Averaged reward: 3.16
"The search still lacks specifialty. It is not useful at this point, and see that reflected in the drop in reward."
Ni, Co Cu-based Hydride- Promoted Cation Exchange (HPCX) catalysts, Zeolite- Supported Metal Oxides (ZSMOs), Doped Titanium Oxide (TiO2) Catalysts. Averaged reward: 3.87
Ruthenium Oxides, Hafnium Oxides, Ruthhum-
"I see how the search became progressively focused. Ruthentum axide is good answer. Hafnium oxide is questionsble."
Alloy. Averaged reward: 12.06
Figure 10: Illustration an evaluation by domain expert on the progression of top search results found on the path to the answer with highest reward.


Reviews:
[{"review_id": "Cisk4HoiVl", "reviewer": "Reviewer_LJ9V", "summary": "", "questions": "A. What are the set of possible \"mutations\" of the prompt? Are they listed in Table 3 in Appendix C? This table is fairly difficult to understand.\n\n B. Can the same action be applied multiple times to a prompt? What would this look like?\n\n C. In Algorithm 1, on line 2, is the full tree initialized? Isn't the tree very large? What is P_t?\n\n D. In the comparison with ToT, why stop at depth 5? It seems like the search could have continued further, and the current comparison isn't as fair since ToT evaluates fewer prompts than MCR.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1.  **Reasons To Reject:**\n\n    1.  **The motivation in the Introduction is unclear. What is the\n        search space? How is query decomposition helpful for searching\n        in this space? Perhaps a concrete grounding example can help\n        clarify**\n\nThank you for these thought-provoking questions. They are at the heart\nof our objective to apply an LLM-driven search for novel catalysts, and it is truly important that we\nare able to communicate the potential for impact and key scientific\ninsights to the reviewers and any reader.\n\n> **\"The motivation in the Introduction is unclear.\"**\n\nLet us begin with \"motivation\" with reference to Figure 1. As we mention\nin lines 28-36 (page 1), energy efficiency or low energy (or heat)\nrequirements is a key consideration for discovering the next generation\nof chemical processes. Such consideration is at the core for converting\nwaste to synthetic fuel that would be used to fuel airplanes, or\nproducing steel which consumes an enormous amount of heat and\ncontributes significantly to pollution.\n\nMathematically, one could represent this as an optimization problem\nwhere we model a chemical process as a function and formulate the search\nproblem as finding the optimal combination of all process parameters\nthat minimizes a cost function modeled around energy efficiency.\n\nHowever, fields such as chemistry/catalysis research are highly\nempirical, where experts place a premium on their ability to reason\nbased on scientific literature that dates back a century. This implies\nthat the optimization process needs to be 1) human interpretable, and 2)\nthe reasoning needs to follow key concepts from scientific literature.\nThis opens the door to LLMs trained on scientific literature, and our\nwork demonstrates how an automated search algorithm could mimic that\nmental search-and-elimination process employed by a human expert.\n\n> **\"What is the search space?\"**\n\nThe search space is the collection of possible factors that can\ninfluence the design and performance of a catalyst. We interviewed\ncatalyst researchers to codify their thinking processes and listed\nfactors they themselves use in their mind to search for a catalyst.\nTable 3 in appendix C provides a complete list of these factors.\n\n\"**How is query decomposition helpful for searching in this space?\"**\n\nOur initial investigation of LLMs revealed that basic prompting (such as\n\"what is a good catalyst for reaction X?\") leads to basic answers that\ncould be found on a wikipedia page. To quote our colleagues from\nchemistry, they do not like an answer \"that feels read out of a science\nencyclopedia, not specific to any process or parameters\". They prefer\nanswers \"that are grounded in domain specific concepts, like how they\nwould explain a rationale to another colleague\". In fact, several of our\nMCR guided answers were praised by them for their high specificity over\nthe state of the art.\n\nNow, we employ two key ideas to achieve this high specificity. 1) We use\nthe \"adsorption energy\" as a measure of reward, which is an important\ndomain-specific measurement commonly used by catalyst researchers. 2) we\n\"probe\" the LLM with different combinations of prompt phrases.\nIdentifying and ordering relevant phrases is critical in narrowing down\nthe search for catalysts. For example, we may first prioritize \"low\ncost\", which directs the search towards a specific family of metal\ncatalysts. Next, we may prompt the model to narrow the search based on,\nsay, \"crystal structures\". These different considerations are visually\ndemonstrated in Figure 1. The need for order and identification of\nrelevant feature subspaces for construction of prompts and efficient\nprobing of LLM's internal knowledge makes query decomposition a focal\npoint of our research.\n\n\"**Perhaps a concrete grounding example can help clarify\"**\n\nIn Figure 10 of the appendix, we provided concrete qualitative analysis\nof how the search works; we analyze how the LLM may use elements of the\nprompt to guide the selection of catalysts. With the extra page, we will\nmove this figure into the main text and provide additional concrete\ndetails to clarify.\n\n2.  **It is difficult to understand the Monte Carlo Reasoner as\n    described, and a figure containing an example would certainly\n    help.**\n\nWe included a figure demonstrating the tree search and prompt mutations\nin figure 10 of the appendix. With the extra page, including this in the\nmain text would clarify the confusions around the algorithmic design and\nthe search space and query decomposition of the previous comment.\n\n3.  **Experiments were run only on text-davinci-003, so they are not\n    easily reproducible in the long-term. There were no experiments on\n    open models, which could have alleviated the query cost, and enabled\n    experiments on more queries from the dataset.**\n\nWe believe in the promise of our methodology, but as the reviewer\npointed out, we also recognize the critical importance of\nreproducibility and increased experiments. We fully agree that using\nopen models would have been highly desirable--in fact, we initially\nexperimented with doing so. However, we finally decided to base our\nexperiments on OpenAI models because:\n\n1.  Our overall methodology strongly relies on the robustness of\n    LLM-instruction following. Robustness is critical as both\n    Tree-of-Thoughts and our methods pursue \"deep chains\" in the search\n    tree, where results from one internal node are transferred to the\n    searches spawned over its children. LLMs are an extremely\n    fast-moving field, and despite all the excitement around open source\n    models, we could not find any other model that offers both the\n    robustness of instruction following and coverage of scientific\n    literature. We had performed extensive experiments with Galactica\n    but it was not reliable for deep tree searches. We will include\n    these results in the final version.\n\n2.  Preparing a robust open-source model with coverage of the scientific\n    literature would not be impossible, but it would require a\n    tremendous amount of resources. Any such investment is also risky in\n    a time when new models are released nearly every week.\n\n3.  Our primary goal for this short paper was to demonstrate the\n    possible scientific reasoning capabilities and resulting impact of\n    LLMs. Since our goal is to demonstrate a new method to augment\n    scientific reasoning, we chose to focus on developing methodologies\n    for eliciting these scientific capabilities in lieu of LLM\n    engineering. Therefore, we chose the tool that helped to demonstrate\n    that point best.\n\nBesides the demonstration of scientific impact, we believe our work\nconveys other critical points for LLM research:\n\n1.  Our methodology does not make any assumptions about a specific LLM,\n    and it would be trivial to swap a new LLM into our framework\n    (provided the LLM is sufficiently capable of instruction following).\n\n2.  Our work demonstrates that tree-search methods have a strong value\n    proposition over existing methods for LLM reasoning (CoT,\n    self-consistency etc.). We would like to emphasize that\n    Tree-of-Thoughts (ToT) was released as a preprint in May and is\n    contemporary to our work--our methodology was developed separately.\n    Thus, an important contribution of our work is demonstrating the\n    merit of tree-based reasoning approaches for complex scientific\n    tasks. We note that ToT is primarily tested on toy datasets instead\n    of real-world, impactful problems--we show that tree-based\n    approaches are capable here. Related further experiments conducted\n    after submission strongly confirm the high performance of tree-based\n    search methods over existing approaches.\n\n    1.  **Many parts of the paper are unclear (see below) and difficult\n        to follow.**\n\nThank you for elaborating on these points. Since our work is highly\ninterdisciplinary, crafting a clear message is more challenging but of\nutmost importance. We have attempted to address your questions below and\nwe will implement them into the final version to ensure it is clear and\neasy to follow.\n\n2.  **Questions For The Authors**\n\n    1.  **What are the set of possible \\\"mutations\\\" of the prompt? Are\n        they listed in Table 3 in Appendix C? This table is fairly\n        difficult to understand.**\n\nWe will clarify the technical details of the prompt generation in the\nfinal version. As an example, . Further, we will also include an example\nof a reasoning pathway to clarify what the actions do to the prompt.\n\n2.  **Can the same action be applied multiple times to a prompt? What\n    would this look like?**\n\nThe same action can be applied only if it does not cause a phrase to be\nrepeated. For example, if the algorithm has already taken the action to\nconsider catalysts with \"low cost\" in the prompt, it won't take the\naction to consider \"low cost\" again (we don't consider catalysts with\n\"low cost and low cost\"). We discussed this briefly in Appendix C.2\n(lines 863-871); we will further clarify this explanation along with the\nimprovements for Table 3. We will include a detailed list of which\nspecific actions can be taken multiple times.\n\n3.  **In Algorithm 1, on line 2, is the full tree initialized? Isn\\'t\n    the tree very large? What is P_t?**\n\nInitially, the tree in Algorithm 1 only contains a single node, the root\nnode. As the algorithm runs, new nodes are added to the tree using the\nset of actions. For example, if the phrase \"considering catalyst with\nlow cost\" is added to a prompt (a node), then a new child node is\ncreated containing the new prompt. Once a new node is created, the\nanswer and reward are stored in the node. The number of nodes depends on\nthe use case. For our tests, we used 301 nodes per tree, as shown in\nTable 1. P_t is the node selected by MCR in the tree T at depth t in\neach step. We interpret P_t as the t-th prompt in a reasoning pathway.\n\n4.  **In the comparison with ToT, why stop at depth 5? It seems like the\n    search could have continued further, and the current comparison\n    isn\\'t as fair since ToT evaluates fewer prompts than MCR.**\n\nWe appreciate the Reviewer's comment. While the ToT method had fewer\nnodes, we still demonstrate an advantage. In Table 1, we included the\nmaximum depth of each ToT and MCR. MCR was able to achieve a depth of\n\\~9 in each evaluation with a reasonably few number of extra nodes over\nToT. At each step in the ToT process, 6\\*12=72 prompts are created. So,\nit would take 288 more steps to reach a depth of 9. We set the stopping\ndepth at 5 to generate close to 300 nodes, which is the same number of\nnodes as MCR in this work. However, ToT was not able to generate the\nfull 12 actions for every node at every level. This is because some\nactions cannot be taken multiple times and actions are ignored if the\nadsorption energies cannot be parsed out of a reward answer. This means\nthat some steps may generate less than 6\\*12 nodes.\n\nWe again would like to note that ToT is contemporaneous work which was\nreleased only a month prior to our submission. Hence, our comparisons to\nit were not required, but we included them to further strengthen our\noverall message; tree-based reasoning can enable LLMs to perform well on\ncomplex and impactful scientific reasoning tasks. In additional tests\nthat we have done after submission, we have modified the ToT\nimplementation to traverse more nodes. We find that this increases\nperformance closer to MCR. Regardless, our overall results show that\ntree-based models are quite effective for complex reasoning problems in\nthe scientific domain, which was not shown by ToT.\n\n3.  **Typos Grammar Style And Presentation Improvements:**\n\n    1.  **Lines 65-67: I don\\'t understand the meaning of R here. What\n        would be an example of a possible prompt?**\n\nR is the optimal set of catalyst properties to reason over. Our goal\nwith the tree search methods is to use a reward function to search for\nR. We will replace this sentence with a more thorough explanation of the\ngoal of our prompt search.\n\n2.  **Lines 97-111: \\\"reward function in LLM-based computations\\\" How\n    are reward functions used in LLM computations? Is the reward\n    function for the MCTS?**\n\nBy \"LLM-based computations\", we meant using an LLM to perform scientific\nreasoning. It has been common to apply LLMs with CoT-based prompting to\nmathematical reasoning problems. We adopted this terminology from that\narea of research. MCR and ToT are just two types of LLM-based\ncomputation methods that use a reward function to guide which actions to\napply to a prompt. The LLM-computation is the whole process of taking an\ninitial query and returning an answer.\n\n3.  **Lines 136-139: It would greatly aid the reader\\'s understanding if\n    an example were provided of the original prompt and the new prompts\n    after introducing \\\"additional catalyst properties\\...such as\n    \\'composition of metals\\'\\\". Further, what would a child node of\n    this new prompt look like?**\n\nWe believe our planned inclusion of figure 10 from the appendix will\nhelp explain our prompt creation methodology.\n\n4.  **Line 143: From the perspective of the LM, isn\\'t it the case that\n    each prompt in the search tree is independent of the other prompts?\n    If so, perhaps \\\"reasoning pathway\\\" is not a good term for this, as\n    it suggests something akin to a chain-of-thought, where each step is\n    a deduction of a new fact from premises given by previous steps.\n    Since each node in the tree is not a reasoning step, I\\'m not sure\n    \\\"Monte Carlo Reasoner\\\" is the most appropriate name either.**\n\nWe had two reasons to call these pathways \"reasoning pathways\". 1) the\nsequence of actions taken in the pathways is determined by the reward at\neach step. Thus, the path from the root query to a leaf of the search\ntree is a sequence of decisions based on heuristic feedback. This is\nakin to how scientists would rationally deduce a set of catalysts and\ncatalyst properties by moving from one inquiry to the next. 2) each step\nof the reasoning pathway uses catalysts from the previous prompt to\ncondition the LLM to produce the next catalysts to be either \"similar\nto\", \"different from\", or \"incorporating elements of\" the previous\ncatalysts. So, the reasoning pathways we define are not independent\nprompts--the LLM is reasoning based on intermediate results just like\nCoT. Figure 10 in the appendix, which we will include in the final\nversion of the paper, shows how the catalyst properties and catalysts\nare passed between prompts in a reasoning pathway.\n\nThe strength of our Monte Carlo Reasoner (MCR) method and the\ncontemporaneous ToT method is their ability to search branching\nreasoning pathways. This is the key distinction from CoT; MCR uses a\nreward-function to select the best path in the tree to follow. The MCR\nmethod uses random sampling (MCTS) to determine which branches to use at\neach step, under the assumption that the highest reward branch may not\nalways lead to the best catalysts. Since the MCR combines Monte Carlo\nand reasoning pathways, we decided to call it the Monte Carlo Reasoner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The concept is interesting, with an interesting application of MCTS and LLMs to a real problem in chemistry research.", "reasons_to_reject": "- The motivation in the Introduction is unclear. What is the search space? How is query decomposition helpful for searching in this space? Perhaps a concrete grounding example can help clarify.\n - It is difficult to understand the Monte Carlo Reasoner as described, and a figure containing an example would certainly help.\n - Experiments were run only on text-davinci-003, so they are not easily reproducible in the long-term. There were no experiments on open models, which could have alleviated the query cost, and enabled experiments on more queries from the dataset.\n - Many parts of the paper are unclear (see below) and difficult to follow.", "questions_to_authors": "A. What are the set of possible \"mutations\" of the prompt? Are they listed in Table 3 in Appendix C? This table is fairly difficult to understand.\n\n B. Can the same action be applied multiple times to a prompt? What would this look like?\n\n C. In Algorithm 1, on line 2, is the full tree initialized? Isn't the tree very large? What is P_t?\n\n D. In the comparison with ToT, why stop at depth 5? It seems like the search could have continued further, and the current comparison isn't as fair since ToT evaluates fewer prompts than MCR.", "paper_topic_and_main_contributions": "The authors apply Monte Carlo tree search (MCTS) to the problem of finding a query such that, when given as input to an LM, outputs recommendations for a chemical catalyst. They propose an iterative approach, where at each step, the current prompt is modified using a randomly selected action to produce a modified prompt, with the hope that the new prompt is better at performing the task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 65-67: I don't understand the meaning of R here. What would be an example of a possible prompt?\n\nLines 97-111: \"reward function in LLM-based computations\" How are reward functions used in LLM computations? Is the reward function for the MCTS?\n\nLines 136-139: It would greatly aid the reader's understanding if an example were provided of the original prompt and the new prompts after introducing \"additional catalyst properties...such as 'composition of metals'\". Further, what would a child node of this new prompt look like?\n\nLine 143: From the perspective of the LM, isn't it the case that each prompt in the search tree is independent of the other prompts? If so, perhaps \"reasoning pathway\" is not a good term for this, as it suggests something akin to a chain-of-thought, where each step is a deduction of a new fact from premises given by previous steps. Since each node in the tree is not a reasoning step, I'm not sure \"Monte Carlo Reasoner\" is the most appropriate name either."}}, {"review_id": "HAkzS2iMeB", "reviewer": "Reviewer_etXD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1.  **Reasons to reject**\n\n    1.  **I don\\'t have strong reasons to reject. As a suggestion, I\n        would like to see more experimental results on different\n        datasets.**\n\nWe sincerely thank the reviewer for their assessment of the work. Our\nsubmission provides initial empirical evidence to the merit of\ntree-based heuristic search approaches for LLM-based reasoning, and our\nfurther experiments confirm these findings. However, we fully agree with\nthe reviewer that more experiments are needed to establish such findings\non a rigorous basis. Such experiments should be the subject of future\nwork. Our motivation to pursue a short paper was based on the promise of\nstrong initial findings:\n\n1.  The correlation between the performance of both our Monte Carlo\n    Reasoner and the contemporary Tree-of-Thoughts work is really\n    important. Even on a smaller scale study, both their outperformance\n    of other reasoning approaches such as Chain-of-Thought and\n    Self-Consistency etc. suggests that tree-based methods hold\n    significant promise for complex scientific domains such as\n    chemistry, or medicine.\n\n2.  We were also astonished by the impact of a domain-specific reward\n    function. We believe integration of such reward functions can be key\n    to address model hallucination challenges for scientific domains.\n\n3.  We also observed that the improved algorithmic performance also\n    comes with the price of significant inference cost and processing\n    latency, and that was another motivation to share our findings with\n    the community at the earliest opportunity. Parallelization of\n    tree-search methods is a well-studied topic in high-performance\n    computing literature, and we seek to draw the attention of the ML\n    systems research community to transition tree-based inference\n    methods like ours from promise to practice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors propose an interesting approach for domain adaptation of the Large Language models. It is known that LLMs suffer from high sensitivity with respect to the input prompt: even small variation in the prompt can lead to significant differences in the generations. Authors propose to generate a wide variation of prompts by sampling from the action space and use MCTS with domain-specific reward to improve efficiency of the combinatorial search. The paper can be of interest to the wider research community seeking of ways to efficiently adapt latest SOTA LLM to narrow domains. Paper is well-written and contains relevant background information, detailed description and experiments backing up the proposed approach.", "reasons_to_reject": "I don't have strong reasons to reject. As a suggestion, I would like to see more experimental results on different datasets.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors propose a LLM-based model that solves a problem of suggesting the top-k catalysts for a given chemical reaction and generates the reasoning for each candidate. In particular, authors propose to use a Monte Carlo-based algorithm (\"MCR\") to prompt language model, and develop a domain-specific reward function. MCR algorithm solves the problem of searching over the large space of possible prompts that would result in the best outcome with respect to the reward function. Prompts are generated by recursively modifying a prompt node (starting with a root) with a set of available actions, creating tree-like structure. Then MCTS is applied to select the branches that show the most promising results. Finally, authors use a reward function that for each prompt queries the LLM to evaluate the adsorption energies, and use this feedback to guide generations. Additionally, authors build a new dataset for complex biological reasoning. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "khMIJqor94", "reviewer": "Reviewer_CnCD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their support of the work. We are\npassionate about demonstrating the impact of large-language models to\ncomplex scientific tasks. We specially sought to reverse engineer the\nmental reasoning processes employed by chemistry researchers as they go\nabout designing experiments through long interviews and come up with a\nmethodology that nearly codifies the process.\n\nWe also recognize that even for organizations with nearly infinite\nresources, fine-tuning or engineering LLMs is destined to reach a point\nof diminishing returns, and it is more important to integrate\nwell-established scientific concepts or functions than reinventing\neverything through language-modeling objectives.\n\nWe sincerely hope our work will be recognized by the reviewers and\ncommunity positively, and again thank you for your support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Method: The proposed MCR method delicately reduces the search space for the prompt design, and it works good empirically.\n\nReward function: The proposed reward function is suitable for the specific chemical task.", "reasons_to_reject": "No.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper targets at the scientific reasoning (i.e., catalysts discovering) problem by prompting LLMs. The authors propose a Monte Carlo Tree search method to retrieve the optimal prompt design, and a specially designed reward function is proposed. In experiment, they propose a new dataset and evaluate their method on it and a public dataset. Results show that their method can significantly outperform existing baselines on the catalysts discovering task.\n\nThe main contribution is: proposing an effective prompt searching algorithm as well as a reward function for the catalysts discovering task; introducing a new dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed an approach for domain adaptation of the Large Language models.
The 2 out of 3 reviewers selected highly positive score (one selected "borderline")
Decision: Accept-Findings