ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction
Authors: Yaorui Shi
An Zhang
Enzhi Zhang
Zhiyuan Liu
Xiang Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: graph neural networks
chemical reaction

Abstract: Predicting chemical reactions, a fundamental challenge in chemistry, involves forecasting the resulting products from a given reaction process. Conventional techniques, notably those employing Graph Neural Networks (GNNs), are often limited by insufficient training data and their inability to utilize textual information, undermining their applicability in real-world applications. In this work, we propose **ReLM**, a novel framework that leverages the chemical knowledge encoded in language models (LMs) to assist GNNs, thereby enhancing the accuracy of real-world chemical reaction predictions. To further enhance the model's robustness and interpretability, we incorporate the confidence score strategy, enabling the LMs to self-assess the reliability of their predictions. Our experimental results demonstrate that ReLM improves the performance of state-of-the-art GNN-based methods across various chemical reaction datasets, especially in out-of-distribution settings. Codes are available at https://github.com/syr-cn/ReLM.

ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction
Yaorui Shi
An Zhang Enzhi Zhang" Zhiyuan Liu
Xiang Wang
$ University of Science and Technology of China
National University of Singapore, 'Hokkaido University
yaoruishi@@mail com, anzhang@u. . nus edu, enzhi zhang. n6@elms. hokudai ac jp. acharkq@gmail com, xiangwang (atpwanng22@@@mmmmm 223@gmail. com
Abstract
with the graph structures of molecules in chemistry- related tasks (Bran al., 2023).
Predicting chemical reactions, a fundamental challenge chemistry, involves forecasting the resulting products from given reaction pro- cess. Conventional techniques, notably those employing Graph Neural Networks (GNNs), are often limited by insufficient training data and their inability to utilize textual informa- tion, undermining their upplicability in real- world applications. this work, we propose ReLM, novel framework that leverages the chemical knowledge encoded in language mod- els (LMs) to assist GNNs, thereby enhancing the accuracy of real-world chemical reaction predictions. To further enhance the model's robustness and interpretability, we incorporate the confidence score strategy, enabling the LMs to self-assess the reliability of their predic- tions. Our experimental results demonstrate that ReLM improves the performance of state- of-the-art GNN-based methods across various chemical reaction datasets, especially out- of-distribution settings. Codes are available at https://g/ thub. com/ https/ss-cc/eeeee
Recently, graph neural networks (GNNs) have been widely used to address chemical reaction prob- lems due to their ability to handle complex graph structures inherent in chemical compounds (He et al., 2022; Somnath et al., 2021; Sacha et al., 2021). However, GNN-based methods often suffer from limited and biased training data, resulting in poor performance in real-world applications that involve diverse reaction mechanisms. This is ex- emplified by the unsatisfactory performance of Lo- calRetro (Chen and Jung, 2021) on Imidazo and NiCOlit (see results in Table 1), especially when encountering new reaction types absent in the train- ing set. In addition, it is difficult for GNNs to effectively leverage the textual information (e.g.. reaction conditions) that could be derived from reaction descriptions. Identical reactants gas can yield different products depending on the catalyst used. An illustrative example provided in Ap- pendix C.I.
A crucial question arises: can we develop a chemical reaction framework that synergistically integrates the advantages of both GNNs and LMs? In this work, we propose ReLM, novel frame- work designed to conduct chemical reaction predic- tion by utilizing both pre-trained LMs and GNNs. ReLM enhances the prediction accuracy in out-of- distribution (OOD) scenarios by utilizing graph structure understanding of GNNs and the natural language understanding capabilities of LMs. More specifically, ReLM employs GNNs to generate sev- eral probabibbyyylity candidate products. These products, along with appropriate in-context exam- ples and descriptions of the reaction conditions, are then fed into the LMs facilitate accurate chemi- cal reaction prediction (as depicted in Figure 1). To further improve the robustness and interpretability of ReLM, we propose a prompting technique called the confidence score strategy (CSS). Harnessing
1 Introduction
Pre-trained language models (LMs) possess vast reserve of knowledge, coupled with impressive ca- pabilities for logical inference (Brown et al., 2020; OpenAI, 2023; Taylor al., 2022: Chowdhery
al., 2022; Touvron al., 2023; Chiang al., 2023). These advantages render LMs useful for crostion-answerne tasks, such as scientific inquiry and chemical inference (Bran et al., 2023; Shao et al., 2023). However, due to LMs' black-box natures, distinguishing whether the answers stem from their inherent knowledge or arbitrarily generated poses significant challenge. Further- more, recent studies indicate that LMs struggle
*Corresponding author
Xiang Wang also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center.

---

conficence
6/6/300
Inveotteot somppe !
True Apraent: (3/00)
cortoll commppic Q,
Puloo Answan: Ic/uo
Ammuaage Model
[UMAC name] +SMILES
(prop-2-nnccc add coacquooj
Antwer:
Target Beattion
(1-aroma S-nnon---tthyttnn--ee cCCy-CClrjC> produce when heatsl??
Confidence Scorw 3/30
1pp
jeub---eryl prop- prop-2-ytound Z-ynound]
GNN
(oubyl acatate)
ecooCDCL-OjE>
[3-nethyttu- prap- 2-ynouts]
CCCuc cencyujjeacee oojeacyc>
Figure I: The overall framework of ReLM. The ReLM encompasses three key inputs for the language models: similar reactions (grey box), the target reaction along with its conditions (yellow box), and candidate products generated by GNNs (green box). Specifically, we select in-context examples with various confidence scores from the training samples that are nearest to the target reaction query. Additionally, choose the K-candidate products with the highest likelihood as predicted by the GNN.
the intrinsic self-criticism capacity of LMs, CSS boosts the model's performance without involving significant computational costs (see Table 5). Ex- tensive experiments on real-world Open Reaction Database (ORD) (Kearnes et al., 2021) demonstrate that ReLM achieves significant improvement over statc-of-the-art GNN-based methods.
Given a query with reactants R, the model is required search across the product corpus identify the most probable product. The likelihood between reactants and products is estimated using L2 distance between their corresponding molecular embeddings:
D(R, P') - Σ G(rl0)
Σ G(pl0)ll2 (2) PEP
2 Background
PER
Chemical Reaction Analysis. Recent studies utilize language models (LMs) for molecule anal- ysis, relying solely on Simplified Molecular Input Line Entry System (SMILES) as input for molec- ular representation learning (Irwin et al., 2022; Chithrananda et al., 2020; Fabian et al.. 2020). However, SMILES only provides one-dimensional linearizations of molecular structures, meaning that molecules with identical SMILES may exhibit en- tirely different properties (Wang et al., 2022). In contrast, GNN-based approaches take the structure of molecular graphs into consideration and have shown remarkable performance in in-distribution chemical reaction prediction tasks (Somnath et al., 2021; Dai et al., 2019).
where R is the given set of reactant molecules a reaction, pr E C is a product set from the cor- pus, and G(.10) stands for the GNN model with parameter 6.
Prompting Strategies. Instead of directly ap- plying in-context learning to real-world tasks, some studies use prompting strategies for better perfor- mance (Shao et al., 2023; Bran et al., 2023). A commonly used strategy is instructing the language model to provide responses in a formalized for- mat (e.g., JSON or YAML). By using appropri- ate prompts, the thought process of an LM can be elicited using structured key-value pairs. Yang et al. develop the Multi-Query Ensemble Strat- egy (MES), which enhances the robustness of LMs. This strategy involves iteratively querying an LM with diverse in-context examples, conducting the inference process multiple times, and ultimately using a majority vote to determine the outcome.
A chemical reaction between reactant set {ri, r.,.... and product set P = {P1,p2, ...) can be defined as:
F1, T2,
P1, P1. ....
(1)
Methodology
Following the evaluation protocol established by Wang al., we can consider reaction prediction as task of ranking products within predetermined product corpus = (Pi, .2....). This evaluation protocol ensures that P. E C holds for all reactions R, + P; in the tests.
In this section, we propose ReLM, framework combining graph neural networks (GNNs) and lan- guage models (LMs) for chemical reaction anal- ysis. ReLM employs GNNs to generate answer candidates and in-context examples, then utilizes LMs for analysis in the form of multiple-choice

---

questions. We also propose the confidence score strategy, a generic prompting stralegy to improve the reliability of LMs.
(5) With reaction information {R, C, PR} and in- context examples &, the product prediction prob- lem can be formulated choice question. We input this question into the pre-trained language model, and the answer gener- ated by the model regarded as the output of our approach:
= ( P T3
3.1 Context Generation
Answer Candidates. The prompt for language models can be formulated with input reactants R, reaction condition and set of answer candidates P generated by GNN encoder.
single select multiple
The process of answer candidates generation can be succinetly described as data retrieval directed by a pre-trained GNN model.
p = argmaxx (L,,
(6)
Leveraging insights from the field of chemistry, chemical equations encapsulate the conservation of matter, charge, and various chemical properties. This implies level of correspondence in the latent representations of the molecules both sides of the equation. Therefore, for a specific set of re- actants R, we employ the distance measurement function defined in Equation (2) to filter out those product sets P3 E C that exhibit the highest simi- larity to R in the latent space.
PEP x
here (AIP) is the probability distribution of answer A in language model given prompt P as input. To feed molecules into language models, we use both IUPAC names and SMILES string to represent molecules. In Appendix C.2, we present a detailed, step-by-step example of the context gen- cration process.
3.2 Confidence Score Strategy
Formally, the answer candidates are generated by selecting the top-K products with the highest similarity from the candidate pool C:
Besides prompting language models for chemical reaction prediction, we propose a universal prompt- ing strategy named the confidence score strategy, which can be seamlessly transferred to any prompt- based language model application.
P nt = Topk - DDR, Piee
(3)
During inference, we ask the language model to report its confidence score (an integer number between and 9) based on its understanding and familiarity with the given multi-choice question. To help the language model better understand how this score works, we also introduce the confidence score in the context prompt. For each in-context example, random integer in {8, 9} is chosen as the confidence score, which is then combined with the ground truth answer to form the prompt. Nevertheless, the above random generation be- havior can easily lead a misunderstanding of the meaning of confidence scores by the language models, resulting in a degradation where the model only generates higher scores. Hence, we deliber- ately distort the answer of an in-context example to an incorrect one and assign it to lower confi- dence score (e.g., random integer in {1,2), 2}). Thus, Equation 5 can be rewritten
where PR is the set of answer candidates for reactants R, and D is the GNN-based similarity measurement defined in Equation (2).
In-context Examples. Besides the reaction in- formation, the choice of in-context examples also crucial for LM's feo-shot learning performance. To acquire in-context examples that imply a reaction mechanism similar to the given reaction sample, we use the answer-aware example selection strat- egy proposed by Shao et al. (2023). Specifically, for given test sample, we choose top-N nearest neighbors from the training set based on the simi- larity of their reactants in the latent space, and use the training samples as in-context examples:
T arg aogTopN
(4)
|
iC(1,2....MM)
where M is the size of the training set, hR EPER G(rl0) is the sum of all reactant representa- tions, and T denotes the index set of the selected training samples. With the ground truth products og of each training sample, the in-context examples are defined as:
(7) where a'' is the perturbed answer and 8.' the corresponding confidence score. By asking for the confidence score, we let the language model
غ E
PRO)IET)

---

implicitly self-critique its answers during inference. In Appendix D, we verify that the confidence score strategy can be comprehended by language models.
Results. In Table 1 and Table 5 (see Appendix B.3), we present the average accuracy of RcLM on the ORD dataset. ReLM improves the perfor- mance of GNN backbones across all test datasets. As ReLM utilizes the top-K candidates provided by the GNN, the theoretical upper bound of our approach is constrained by the hit@ K accuracy of the GNN backbones. These upper bounds are also delineated in tables. To assess the stability of ReLM, we examine its accuracy under varying K- --aauuss and degrees of in-context randomness. Detailed results can be found in Appendices B.6 and B.7.
3.3 Fine-grained Confidence Score Strategy
To achieve a more detailed analysis of confidence and enhance the language model's s fine-grained in- terpretability, we also attempt to generate confi- dence scores for each answer candidate instead of a singular overall score. Subsequently, these candidates are re-ranked based on their respective confidence scores, selecting the one with the high- est confidence as the definitive answer. We refer this methodology the fine-grained confidence score strategy. Please see more experimental re- sults and analysis in Section 4.4 and Appendix B.4
Apart from the evaluation on out-of-distribution datasets, we also run experiments under indepen- dent and identically distributed (i.i.d.) conditions. Refer to Appendix B.5 for more results.
4 Experimental Results
4.3 Effectiveness of Confidence Score Strategy (RQ2)
We aim to answer the following research questions: RQI: Can ReLM improve the reaction predic- tion capability of graph neural networks on both out-of-distribution and in-distribution data?
Motivation. The Confidence Score Strategy pro- posed in this paper is a generalizable prompt- ing method. To verify the effectiveness of this strategy, we conduct comparative experiments against other prompting strategies specifically tai- lored for multiple-choice questions. The baseline strategies include the Multiple- Ensemble Strategy (MES) (Yang et al., 2022), the JSON Strategy, and a control group that does not employ any strategy. MES necessitates multiple runs of the inference by the language model (10 times in our experiments). A majority vote is conducted based on these re- sults, and the most frequently occurring answer is selected the final outcome. By JSON strat- egy, we refer to the strategy that asks the language models present their understanding of reaction precursors, reaction mechanisms, and its inferring process. Under this strategy, the language model is required to output all the above information along with its answer in machine readable maaaile-readable format.
RQ2: Does the confidence strategy enhance the accuracy of ReLM?
RQ3: How does confidence score strategy influ- ence the inference process of language models?
4.1 Experiment Setup
Baselines. Two popular GNN chemical reaction analysis methods, MolR (Wang et al., 2022) and LocalRetro (Chen and Jung, 2021), are selected as GNN backbones. For language models, we test GPT-3.5 (Brown et al., 2020) and Vicuna (Chiang al., 2023). See more details in Appendix B. Datasets. We utilize the USPTO dataset train the GNN backbones. For evaluation, four datasets from the Open Reaction Database (ORD) (Kearnes et al., 2021) are utilized as the testbed for our exper- iments. The ORD contains diverse real-world reac- tion records from open-source projects and chem- ical literature. Notably, its utility for GNN-based chemical reaction predictions remains largely derexplored.
Results. In Table 2, we compare the overall accu- racy (Acc), number of input tokens (#Token), and average inference time (Time/s) for each prompt- ing strategy. Clearly, both the MES and JSON strategies enhance the model's performance, albeit with non-negligible time costs. Conversely, our proposed Confidence Score Strategy (CSS) promi- nently improves the accuracy without imposing a noticeable computational burden on the language model. See Appendix B.8 for comparison with more prompting strategies.
4.2 The OOD Capability of ReLM (RQ1)
Motivation. To evaluate the model's generaliza- tion ability, a comparison is made between our method and GNN baselines on the ORD dataset, Natural language descriptions of reactions, sourced from the ORD database, are utilized to enhance the quality of the prompts.

---

Table 1: Accuracy on out-of-distribttinn settings.
Imiiazo
NiCOsi
Resgen-300
Rexgen-4u 40k
K=3
K-4
K-3
K-4
K=3
K-4
Km3
K-4
MolR (Wang 2022)
0.313
0.513
0.437
0:437
0.471
0:171
0.118
0:448
ReLM (MoIR+ Vicuna)
0.878-7140%
0.510 o00
0.525-19.700
0.496 0.498-5.877 +5.07
0 0:199 0.29g-b.500
0.175++5306
0:173+5.318
ReL.M (MoIR+ GPT-3.5)
0.865
0.815 56.355
0.443
0.478
0.486 18r
0.458
2.823
0.450 19308
0.467 1470
Upper Bound
0.945
0.979
0.641
0.679
0.581
0.562
0.586
LocalRetro (Chen and Jung. 2021)
0.023
0.023
0.086
0.088
0.279
0.279
0.245
0.245
ReLM (LocalRetro Vicuna)
0.023
000
0:037 -35.955
0.173 100 one
0.18
0.325 16 4718
0 0.329 17.965
0.295
100 2008
0 0,298 191.095
ReLM (LocalRetro GPT-3,5)
0.031 HaLLRN
0.057
-3535%
0.221+100
0,256-192.228
0.348+20
0.3 -3361-
0.308
0.316-
Upper Bound
0.033
0.046
0.323
0.340
0.400
0.440
0:374
0.406
Table 2: Comparison of different prompting strategies within ReLM.
respectively.
This correlation sheds light on the consistency between these two models in their decision-making processes. The overall rankings of the ReLM ex- hibit a positive correlation with the rankings of the MolR, implying that the ReLM concurs with most
Rougen- SOk
Rexges-40X
Asc
aTokk
Timal
Aoz
aTrikan
Timeh
MoiR Vicuna wio stratogy
0.472
941LG
2.19
0.449
942.3
2:19
MaIR Vicuns ISON
0.456
1087.4
10816
Moil- Vicum MFS
0.400
9402.0
22:2
0.463
0412.5
22.4
MolR + Vicuna CSS
0:107
991.8
1.5
0:473
990.5
1.5
MaIR + GPT-3. wlo stnstgy
0.461
956.6
2,00
0.420
966.4
1.63
MalR+
ISON
of the MolR's 's judgments.
MolR + GPE-3.5 MES
0.476
8366.1
18.9
0.425
9001.
18.3
MotR+ + GPT-3.5 CSS
0.485
1007.5
1.6
0.450
1017.4
2.0
Moreover, the ReLM concurs with the MolR when it's correct, and makes contradictory choices when it errs. This suggests that the ReLM is able to make informed judgments, diverging from the MolR when it believes the MolR is wrong. Ad- ditional results under the fine-grained confidence score strategy are in Appendix B.4. For further statistical analysis pertaining to confidence scores, refer to Appendix D.
Table 3: Rank Correlation between Me fine-grained confidence score and MolR's candidates ranking.
Imidazzo
NiCOlit
rexgen-30k
rexgen-40k
P 0.363
0.359
0.346
0.347
/* 0.469
0.429
0,388
0.397
P-
0.243
0.227
0.199
0.174
It is noteworthy that ReLM achieves competitive even superior results regardless of the prompting strategy employed for the language model. These results suggest that our reaction prediction method maintains a high level of robustness irrespective of the prompting techniques used.
Conclusion
Despite the great success of molecular structure understanding, today's 's GNN-based reaction pre- diction methods are still far from being able do real-world reaction analysis. In this work, we proposed ReLM, an in-context prompting method that utilizes both language models and graph neu- ral networks for chemical reaction prediction. Ex- tensive experiments demonstrate the remarkable improvement of ReLM on various datasets indeed comes from the reasoning ability and self-criticism of LMs.
4.4 Interpretability of Confidence Scores (RQ3)
Motivations. With more detailed confidence infor- mation, we aim to interpretably analyze the differ- ences between language models and graph neural networks during decision-making processes, and investigate the reasons behind the performance en- hancement brought by LMs. We employ Spear- man's 's rank correlation coefficient as the metric to ascertain how the ranking produced by the LM cor- relates with the original rankings from the GNN model.
Limitations
The limitations of ReLM are in two aspects, which will be addressed in future work. Firstly, ReLM focuses solely on chemical reaction prediction, neglecting other essential tasks in chemical reac- tion analysis such as retrosynthesis planning and yield prediction. Secondly, the performance gains achieved by ReLM are hindered by the drawbacks of backbone GNNs. The accuracy improvement is restricted by the hit@ K metric of GNN models, lcading to only marginal advancements on specific datasets, as demonstrated in Table 1. We believe
Results. For the evaluation of the fine fin-innaieed confidence score strategy mentioned in Section 3.3, we use MolR as the GNN backbone. Vicuna as the language model, and K 1. In Table 3, the symbol p denotes the rank correlation derived from ReLM and MolR across all test samples. On the other hand, p+ and pt represent this correlation when MolR's ' predictions are correct and incorrect

---

our ReLM sheds light on chemical reaction pre- diction tasks and will inspire more work in this research line.
Hanjun Dai, Chengtao Li, Connor W. Coley, Bo Dai, and Le Song. 2019. Retrosyuthesis prediction with conditional graph logic network. In NeurIPS.
Jian Du Shanghang Zhang, Guanhang Wu, José
Acknowledgements
M. F. Moura, and Soummya Kar. 2017. Topol- ogy adaptive graph convolutional networks. CoRR, abs/ 10. 10370.
This research is supported by the National Natural Science Foundation of China (92270101 14), the University Synergy Innovation Program of Anhui Province (GXXT-2022-040), and the NExT Re- search Center.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Mar- win Segler, Joshua Meyers, Marco Fiscato, and Mo- hamed Ahmed. 2020. Molecular representation learn- ing with language models and domain-relevant auxil- iary tasks.
Jiyan He, Keyu Tian, Shengjie Luo, Yaosen Min, Shuxin Zheng, Yu Shi, Di He, Haiguang Liu, Nenghai Yu, Liwei Wang. Ji Wu, and Tie-Yan Liu. 2022. Masked molecule modeling: A new paradigm of molecular representation learning for chemistry understanding.
References
Andres M. Bran, Sam Cox, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXtv:2304.05376.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. 2022. Chemformer: pre-trained transformer for computational chem- istry. Machine Learning: Science and Technology, 3(1:0015022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
Steven M Kearnes, Michael R Maser, Michael Wleklin- ski, Anton Kast, Abigail G Doyle, Spencer D Dreher, Joel M Hawkins, Klavs Jensen, and Connor W Co- ley. 2021. The open reaction database. Journal of the American Chemical Society, 188445):18200-1888
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte. Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. 2016. Pub- chem substance and compound databases. Nucleic acids research, 44(D1):D1202 -122111.....
Shuan Chen and Yousung Jung. 2021. Deep retrosyn- thetic reaction prediction using local reactivity and global attention. JACS Au, 1610).662-
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In 5th International Conference on Learn- ing Representations.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing 4 with 90%* chatgpt quality.
Daniel Mark Lowe. 2012. Extraction of chemical struc- tures and reactions from the literature. Ph.D. thesis, University of Cambridge.
Seyone Chichrananda, Gabriel Grand, and Bharath Ramsundar. 2020. Chemberta: Large-scale self- supervised pretraining for molecular property pre- diction.
OpenAL. 2023. Gpt-4 technical report.
Kohulan Rajan, Achim Zielesny, and Christoph Stein- beck. 2021. Stout: Smiles to iupac names using neu- ral machine translation. Journal of Cheminformatiis. 13(1):1-14.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv.2204.02311.
Mikolaj Sacha, Mikolaj Blaz, Piotr Byrski, Pawel Dabrowski-Tumanse Mikolaj Chrominski, Rafal Loska, Pawel Bondarrzyk-PPuszynsss and Stanislaw Jastrzebski. 2021. Molecule edit graph attention net- work: Modeling chemical reactions as sequences of graph edits. J. Chem. Inf. Model., 61(7):327-3228
Connor W Coley, Wengong Jin, Luke Rogers, Timo- thy F Jamison, Tommi S Jaakkola, William H Green, Regina Barzilay, and Klavs Jensen. 2019. A graph-convolutional neural network model for the prediction of chemical reactivity. Chemical science, 10(2):370-377.
Philippe Schwaller, Benjamin Hoover, Jean-Louis Rey- mond, Hendrik Strobelt, and Teodoro Laino. 2021. Extraction of organic chemistry grammar from un- supervised learning of chemical reactions. Science Advances, 7155)eabe4416.

---

Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuris- ties for knowlegge-hasee saagg--aaeed visual question answering. Proceedings of the IEEE/CVF Conference on Com- puler Vision and Puttern Recognition, pages 14974- 14983.
Vignesh Ram Somnath, Charlotte Bunne, Connor W. Coley, Andreas Krause, and Regina Barzilay. 2021. Learning graph models for retrosyuthesis prediction. In NeurlPS.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.00000
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Hongwei Wang. Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin D. Burke. 2022. Chemical-reaction-ann
molecule representation learning. In ICLR.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's ss transformers: State-of-the-an natural language processing. CoRR, abs/1910.03371.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study gpt-3 for few-shot knowledge- based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 308
3089.
Shuangjia Zheng, Jiahua Rao, Zhongyue Zhang, Jun Xu, and Yuedong Yang. 2019. Predicting retrosyn- thetic reactions using self-corrected transforeer neu- ral networks. Journal of chemical information and modeling, 60(1):47-55

---

A Datasets
through the OpenAl API, while for Vicuna, we employ the model released by its authors on Hug- approximately 479k reaction samples, which are collected by Lowe (2012) and cleaned by Zheng et al. (2019).
The USPTO dataset used in Section 4 contains
gingFace (Wolf et al., 2019). It's worth noticing that due to the substantial financial costs associ- ated with accessing GPT-3.5, we only use random subsets of size 500 of the test datasets when using GPT-3.5 as the backbone language model.
We conduct our evaluation on four datasets ex- tracted from Open Reaction Database (Kearnes et al., 2021). Considering there's an overlap between the two Rexgen datasets and USPTO, we exclude all reaction records in Rexgen-30k and Rexgen-40k that resulted in products already present in the training set. This operation ensures that the target products of test reactions have never been encountered during the training phase, impos- ing higher demands on the model's generalization ability.
H Hyperparameters. There are two important hyperparameters in our approach, the number of in-context examples N (see Equation 4) and the number of answer candidates K (see Equation 3), For the number of answer candidates, we try K 3, 4, and 5. For the number of in-context examples, we use constant number N = 3 throughout our experiments.
The datasets, Imidazo and NiCOlit, encompass a wealth of textual descriptions of chemical reac- tions, which can be leveraged by language models for inferential purposes (refer to Section 4.2). The contents of the datasets utilized in this study are summarized in Table 4. Additionally, we provide representation of some examples pertaining to reaction type and reaction condition information contained within these datasets, as depicted in Fig- ure 2.
B.2 Implementation Details
For the speed test demonstrated in Table 2. we test all the involved methods on a single NVIDIA RTX A500 graphic card. For token count computation, we use tiktoken, fast open-source byte pair en- coding tokenizer that can be used with OpenAl's models. For fair comparisons, we also use this tok- enizer to count the token usage of Vicuna (Chiang et al., 2023) in Section 4.3, even though it's not developed by OpenAI.
B Experiments
The training of LocalRetro requires atom-wis matching between reactants and products, but the training dataset used by us does not contain such data. We thus preprocess the training dataset with Rxmmapper (Schwaller et al., 2021) I) to meet this requirement.
B.1 Experiments Settings
GNN models. For MolR (Wang et al., 2022), we the pre-trained checkpoints provided by its authors on their GitHub repository. The check- point model trained on the USPTO dataset for 20 epochs with a batch size of 4096 and a learning rate of 1 x 10 Specifically, we use a 2-layer TAG (Du et al., 2017) as the GNN encoder, as it has demonstrated superior performance in experiments of MolR. For LocalRetro (Chen and Jung, 2021), we train the model the USPTO reaction predic- tion dataset. Following the experimental settings of Chen and Jung, we use a l-yayer GCN (Kipf and Welling, 2017) as the backbone GNN and train for 50 epochs with a batch size of 16. The training process employs an Adam optimizer with learn- ing rate of 1 x 10 and weight decay of 1 x 10 10-6 During the Evaluation phase, we use 256 as batch size for both MolR and LocalRetro.
Besides, shown in Figure we use IUPAC names along with SMILES represent molecules in prompt design. However, the datasets only con- tain the SMILES expression of molecules. We leverage the database of PubChem (Kim et al., 2016) and open-source tool STOUT (Rajan et al., 2021) to perform the conversion from SMILES strings IUPAC names.
B.3 Ablation Study of Reaction Conditions.
Table 5 displays the results of ablation studies deseriptive reaction information. Incorporat- ing reaction conditions (e.g., reaction temperature and catalysts) and reaction type information in the prompts leads ReLM to achieve higher per- formance gains than compared to only reactant information.
Language models. Training is not performed for the language model backbones due to the high training cost of LMs. We interact with GPT-3.5

---

Table 4: Statistics of the Four Datasets in the Open Reaction Database
Description
Reaction
Reaction Type
Condition
Imidaro
chree-coempeeent approach towards diverse imiiaaoppyinn ridinss reactions.
NiCOlt
1762
Nickel-catallzed cross couplings reactions.
Reugen-30k
7700
Test data used by Rexgen (Colcy ci al., 2019).
K
Rexgen-40k
10235
Validition data Resgen (Coley
X
X
Type: Three-ssmpmmet apprtech
towards diverse inklazoyyidiness reactions.
Canalyyt: Trilluoroseetie Aci
Isoprapyl Alcohal
Tempersture: 22 (s
Illumination: DARK
ord @U55B from Imidazo
Reacioon Time: 20 1/
Reaction Type: Nickel-catalyadd www.scoopplligeeee. reactions.
ord 0r-004422cb55deeesbbbbs522222222 from NiCOlit
Figure 2: Examples of reaction type and condition in Imidazo and NiColit.
Table Ablation study of textual reaction information.
Table Evaluation on the test set of USPTO.
Imidazo
NiCOli
MolR
LecalRetro
GPT-3.5
Vicuma
GPT.3.5
Vicuna
MeIR ReL.M
0,882
0.5663
MoIR (Wang et al.,, 2022)
0.513
0.513
0.437
0.437
0.871
0.6273
(MolR)
0.812
0.765
0.412
Upper Bound
0.9527
0,7583
reaction type
0.822
0.903
0.478
0.523
+ reuction ty and condition
0.864
0.914
Table 6: Accuracies with multiple confidence strategy.
Vicuna the language model and MolR as the GNN backbone, with K 4. Due the exten- sive size of the USPTO dataset, we do not conduct experiments under all experimental settings.
Imidazo
NiCOlit
rexgen-30k
rexgen-40k
MolR ReLLM
0,513
0.437
0.471
0.449
0.870
0.507
0.449
0.421
In Table 7, can be observed that GNN meth- ods have solely achieved considerable performance in in mn-disrribution scenarios, thus the performance enhancements provided by ReLM are limited in this circumstance. Furthermore, the lack of reac- tion type and condition information in the USPTO dataset also presents inference difficulties for the language model. Despite these challenges, RcLM still exhibited a certain capacity to select the cor- rect answers from the options, without exhibiting significant performance deterioration.
B.4 Accuracy of fine-grained Confidence Score Strategy.
In this section, test the performance of the fine- grained confidence score strategy (introduced in Section3.3). Table displays the accuracies of both the ReLM and GNN models when using the multiple-CSS strategy. Although ReLM's perfor- mance does not surpass that of the original CSS strategy, it offers a greater degree of interpretabil- ity.
B.6 Effect of Different K K-Values
Given that the accuracy upper bound of ReLM is determined by the number of answer candidates, it is necessary to evaluate the performance of the model under different K levels. In this section, conduct experiments across a wider range of K values. The results presented in Figure 3 using MolR as the GNN backbone and Vicuna as the language model. It can be observed the table that ReLM's performance remains relatively stable
B.5 Evaluation on In-distribution Dataset
The experiments in Section 4.2 demonstrate the powerful out-of-distribution performance of ReLM. We also conducted experiments under the indepen- dently and identically distributed (i.i.d.) setting on the test set of USPTO-479k. The results are shown in Table 7. This part the experiment was conducted on the test set of USP,O-479K, using

---

Table 9: ReLM with other strategies.
Inliniazo
NKOI
Imiidazo
NiCOlir
resgen-30k
rexgen-40k
MolR
0.513
0.437
0.471
0.449
Zero-shot
0.870
0.446
0,301
0,286
Few-shot CoT
0.781
0,305
0,267
0,244
Zero-shot CoT
0,318
0,302
CSS
0.878
0.523
0,499
0,473
Rooger 30k
(di
Upper Bound
0.979
0.611
0.587
In the table, we can observe that using varying confidence scores causes slight fluctuations in ex- perimental outcomes, though the degree varies by dataset. This further demonstrates that the effective- ness of our ReLM lies not in the details of prompt design. Instead, it stems from our main idea of amalgamation of the molccular modeling ability inherent GNNs with the vast reaction knowledge of the language model.
Figure 3: Accuracy under different values
Table 8: Accuracy over fixed and randomized in-context confidence scores.
Imidazo
NiCOli
resgen 30k
resges 40k
B.8 Comparison with More Prompting Strategies
Fived: and (9)
91.93%
53.90%
18.65%
45.627%
Randomizzd:: (1.2) and (8,9)
87.76%
52.32%
49.58%
47.27%
Randomised: (1.2,3) (7,99)
53.78%
In addition to the prompting strategies in Section 4.3, we also included Zero-shot, Few-shot CoT, and Zero-Shot CoT as baseline prompting methods. Table B.8 shows the experimental results of these methods.
K increases across wide range of values (e.g. K 7), and some point, the performance gets better as K increases. However, with very large K values (e.g. K > 10), the accuracy of the model exhibited a noticeable decrease.
We observe that all prompt designs offer some benefits. However, our original confidence score approach still outperforms these baselines. Specifi- cally, as shown in the case studies in Section C.3, the language model's CoT analysis may not pro- vide additional insightful information regarding the reaction mechanism. At this stage, incorporat- ing additional analytical steps may introduce more molecular structures, exacerbating the language model's comprehension difficulties. Further step- by-step elucidation of the reaction process likely necessitates incorporating more domain knowledge of chemical reactions.
This drop in accuracy with extremely large K values is likely because the candidates become overly saturated with implausible options, making it more difficult for the language model discern the correct answer. Since there could be at most one ground truth candidate, increasing K mcans adding more distracting and unlikely candidates. While the model is robust to these distractors up to a point, at some threshold of implausible options the task does become more challenging.
B.7 Influence of In-context Randomness
In Section 3.1 we introduce the generation process of in-context examples. For each in-context exam- ple, we choose random integer within (8, 9} (or (1) its confidence score. This intuitive ran- dom selection aims to faithfully mimic human be- havior when answering the multple-choice chem- istry question. this section, we demonstrate the efficacy of this strategy through comparative exper- iments.
C
Case Studies
C.1 Importance of Reaction Condition
Previous GNN-based reaction analysis models could only process molecular graphs and were un- able to utilize the abundant information contained within chemical reaction conditions. However, many chemical reaction conditions exert profound influences on the reaction mechanisms, determin- ing the direction of synthesis. this section, we illustrate the importance of reaction conditions reaction outcomes through an example from the Imidazo dataset.
In Table we show the performance of ReLM under different levels of in-context randomness. The experiments are carried out using K 4, Vi- cuna as the language model, and MolR as the GNN.

---

In Figure 4, report the answers from GNN and ChatGPT to chemical reaction problems involving the same reactants. The only difference between these two problems is that problem 2 does not include reaction conditions and types. Question is the target reaction given the correct catalyst, while Question 2 omits the important catalyst. For GNN, it incorrectly predicts 'B' as the answer for both questions. In contrast, the language model ac- curately identifies option 'C' for Question which aligns with the ground truth. However, for Ques- tion 2. the language model predict product that was not present in the candidate pool.
assign corresponding scores based their famil- larity with the questions. In this section, we clu- cidate this claim by presenting the distribution of these scores across a large number of samples, that confidence scores can indicate ReLM's degree comprehension of the queries.
We illustrate the distribution of confidence scores using the Rexgen-40 dataset, employing Vi- cuna + MolR backbones. The results are pre- sented Figure 7. Confidence scores of ReLM exhibit significantly different distributions between correct and incorrect answers. For incorrect ReLM choices, the proportion of high confidence scores (7 ~ 9) decreased from 82.3% to 51.2%, while that of low confidence scores (1 ~ 3) increased from 1.1% to 22.7%.
In reality, this organic reaction involves three components an aldehyde, an amine, and an iso- cyanide. Without added catalyst, the combina- tion of these three reactants undergoes Passerini reaction, which is type of multi-component condensation. Nevertheless, in the presence of the trifluoroacetic acid catalyst, the reaction pro- ceeds via an imidazopyridine formation mecha- nism. This leads to the major product being CC', http://cchheellllmoooooooettt/ie a alpyridin---ylJbene
Additionally, designed experiments com- pare the accuracy of the large model at different confidence levels. We divide the datasets into parts based on the model's output confidence and calculate the accuracy separately. The results are shown in Table 10. The "High Conf." column rep- resents the model's accuracy on the subset with higher confidence, and the "Low Conf." column represents the accuracy the part with lower con- fidence. For all control groups, ReLM's accuracy on high-confidence samples significantly higher than those with lower confidence (p-value < 0.05).
1,2-diol.
These case studies clearly demonstrate that iden- tical reactants yield different products under dif- ferent reaction conditions, further reinforcing our driving hypothesis: the incorporation of reaction type and condition boosts the predictive accuracy of ReI.M.
C.2 Illustration of Inferring Process
In Section 4.2, we mentioned that the accuracy up- per limit of ReLM is determined by the hit@ K metric of the backbone GNN model. In this sec- tion, furnish illustrative example elucidate our proposed methodology more comprehensively. Figure provides a step-by-step demonstration of our method when K = 4. The in-context examples have been omitted for brevity.
C.3 Examples of Prompting Strategies
Section 4.3 we compare the confidence score strategy with other prompting strategies. Figure 6, we use example from the NiCOlit dataset to further demonstrate the difference between the involved prompting strategies.
D Statistical Analysis
this paper, we posit that language models possess the capability to comprehend confidence scores and

---

Table 10: Accuracy under high/low confidence levels. The "High Conf." column represents the accuracy of the model on the subset with higher confidence, and the "Low Conf." column represents the lower ones. ReLM's accuracy on high-confidence samples is significantly higher than those with lower confidence (p-value- 0.05).
Rexgen- 30k
Rexgen-40k
k=3
k=4
k=3
k=4
MolR
0.471
0.471
0.448
0.448
High Conf.
0.500#0.235
0.502+0.234
0.477+0.229
0.478+0.229
ReLM (MolR+ Vicuna)
Low Conf.
0.36140.300
0.32920.305
0.330#0.291
0,28840.293
High Conf.
0.525+0.201
0.525+0.257
0.499+0.160
0.510=0.199
ReLM (MolR GPT-3.5)
Low Conf.
0.438=0.268
0.333=0.313
0.400#0.245
0.440=0.254
LocalRetro
0,279
0,279
0.245
0.245
High Conf.
0.330+0.169
0.332+0.190
0.299+0.151
0.302+0.178
ReLM (LocalRetro Vicuna)
Low Conf.
0.235+0.203
0.213=0.220
0.238+0.181
0.0
High Conf.
0.368+0.130
0,405=0. 1132
0.33740. 0.321
ReLM (LocalRetro + GPT-3355
Low Conf.
0.248+0.203
0.270=0.219
0.200+0.118
0.26240. 193

---

Q1: Which of following major product organic reaction between 2,3- be-ddiiddrooybenalddeeyyyy <CIcIccal<10/0)=0: 5-methy/pyridin amine <CC1=CN-CIC-CI)N> and isocyanocyclohexaee <[C- |#N+|CCCCCCCC>, with 2.22--tiillorroacceti acid <C[=D][[[[NFFF)OO as catalyst? The given reaction multicomponent reaction leads to synthesis diverse imidazopyridines.
A. ethyl s(Cj:ttteDl(toopCCCCccceceheCCep
B. methyl 4iiopppnocyycheeeeee////ooo//o/a [CCC///NNACCCccccccoooooooco>>>"
C. 3 B hrrpbbanwsiiiiooo/mmm//mmm/ddddd
<C acccccnceeccclsoooooeeeccccccccc
D. 2-[3 http://wwww..ooooooooom/mmmeeeae e-thoyhhhoo aCCClcacccDDpcccccccccccchccaaaa
GNN: methyl h-tpg//aaccooooooooossssaaaaaaaa
The question describes mutiicompoonntt three reactants:
....
Given the reactants, mote://ooccc/mmppoooooouuussssee be:
Q2:
Which of the following is the major product of the organic reaction between 2,3 ..3-dllydroxxybenzaldeyyy bew.eeebbeenadeyyyeeeeee dehyyde
<C/cIccoo(c10)0)=0>, y hwnpeh/ypriiin222ammm----------- and lscyanocycloheexaee <(C- ||[N=JciCCCCCI>
ethyl h.t.fiocoo..cccgooe.eeeceeeeeeee
methyl 4+jpcypnnoryohheaaeee//aaaaaarrr
GNN: B. methyl witser/rmooooooocceeeetaattorroa
The given reactants are:
Drawing out the structure, the major product is: be000//eeebcccccco00000/e/eeeeee
Figure 4: The answers from GNN and ChatGPT two similar chemical reaction problems. The only difference between these two questions is the existence of reaction conditions. The GNN incorrectly predicts 'B' as the answer for both questions. In contrast, the language model accurately identifies "C" as the answer for the first problem, but fails to predict the second question. These case studies demonstrate that identical reactants yield different products under different reaction conditions.

---

reaction sampled from the test associated reactants, reaction conditions, and other relevant information provided input.
reactants 1: 2.3-dilyydroooybenaddddyyyy
reactants 2: methhyloyrrdnn2-aminee
reactants 3: isacyanocyclohexane
catalyst 1: tralutttiiluoooocctticc acid
HO
Temperature: 22 (+ 1)
reaction type: multicomponent reaction leads to the synthesis imidazopyridines.
Step 2:
The GNN generates ranked the top-4 product candidates (i.e., the four choices A, B, and in step 4) for the above set of reactants.
Step Proper in-context examples chosen from the training using the GNN model.
Step 4: Generate prompts based on Top-K possible answer candidates from the GNN model.
Question: Which following is major product the organic reaction between 2,3- dinydrrawyeeraaddehddd <C/cIcco((11)))==> S-methylpyridin lsccyanocyclohexane <[C-HH[N-]CICCCCC under 'C, with a.t.oottil <<{=O)iC[F)(F)jC>> catalyst?
<CCI=ON=C[C=CI)>>
A. methyl 4 wit.oo/eooccooheaan---aaaaaeeeee 1-carbooyllle
<[C-) <[C:iftweccCccCC(/-ojpoccca
8. ethyl 4,5 3.--iifiuooo isocyano- 1,3 sitiydrriindeeeeeaarrayyy
33 CcttchacccccccbbooneescccCeaeeee
httpp/iddiii-llllleeeeeee-ee- 1,2-dio
Step 5:
The LM delivers its selection accommpanied by an associated confidence score.
Ground Truth: C. "ale:ceesssmoggieiimmmmmallellle
GNN:
A. hethy/--ccoopooosoooeeeeeeeoaoar
Answer:
Confidence Score: 7
Figure The step-by-step illustration of the inferring process. The test case from the Imidazo dataset.

---

Question:
Which the following the major product of the organic reaction between methoxynaphtaalmee <COeler2cecc2e1>>
lithium;1,3, 1,5, t-trimethybbeeeenee
<Celee()<{LL)jc(Cc>>>>> given reaction
couplings reaction, which forms carbon-carte caronn (C-C) and carbon-heteroatom (C-X, where X can be O, N, S, etc. bonds.
Nickel-cane catalyzed
A. 2-12,4,6-trimee S-4immeeli/shhe/hhhhhhhhhhee
8. 1-(0-methoxypheenll--3 http://wwwpeeni...aaaeeeeeeeeeee S-dimethylbenzeee
<CDc2cc(c1cc(C/cccccclccz
1-12,4, Settmmethyjlhheryllaaahhhhb
0. 2 paapillliee-peee//beeeeeee
Ground Truth:
A. htt.s//twtyyhhyyyppaaaaacneeecce
Zero-Shot CoT
Let's think step by step.
The reaction between 2-methhwynaphhhhlleee- COc1cc2ccoc211> and lithium;1,3,5-
trimethylbenzene -e-----iee <<clec(Ce[[U]]eCCCd>>> is Nickel catalyzed cross couplings reaction, which forms carbon carbon (C (C-C) and aurbon-heteronnn (C-X, where can be O, N, S, etc.) bonds. The major product
reaction 2-trrimmtttl-2--hheeeeee phemyleeeeeeeee <[<2a[[[e]c1ccccdl J(C])
other options 1- methoxyphenyll
(2,4,6 rimmettuppher///spphhhheeee 2cceem/eeeccccceeeeeeceeeeeeeeec
dimethylbenzene <CDceeet( <Coeacc(cccSSSSSCCCCccccc and 1- 1-(2,4,6 1,.4-,,,6mmmeeee trimethypherwyaaphhhhleer
<C22cdCDeeecccccc))(d)2.... So, correct answer 1,3,5 3,55-trimthyl-- brimethyl-22pphe 2-hhe-yle-eeeneeeeeee
capital presenting your choice) D. 4.355ttimetlyy222ppeeyybeeeeeeee <cCbc((bcccooccc))(()).
Few-Shot CoT
The reaction between 2-methokyyaphthaleee <Coelcc2ccc2ec>> and 1, 1,sp:l/m.......mmttttteeee S-trimetthlbbeezeee-6 ide <Ccicat[ee(LIIcc(Ckk>>> is Nickel-catalyzed cress-couplings reaction, which forms carbon-carbon (C-C) and carbonn-hetrratoo (C-X, where can O, N, S, etc.) bonds. reaction cycloaddition reaction, where the -OH group lithium replaced the oxygen
from the benzene moiety. The result the atom, yielding the This process involves a substitution reaction, replacing alcohol group. Option duplicated should considered. This prediction based the cycloaddition seaction and the potential rearrangements during the reaction in aqueous conditions. My choice: D
attachment of the naphthalene ring the benzoic acid moiety through
compound 2: 12,4,6 12244.6rrimechy/pheeeee trimethy/pherylnaphhha oxygen atom
Confidence Score
Answer: A
Confidence Score: B
Figure 6: Illustration of different prompting strategies. The in-context examples in the few-shot CoT and Confidence Score strategies have been omitted for brevity.
40.0%
20.0%
10.0%
In Dlsriibution over camples
Distributton iftn over samples where ReLM comeett
Distibutton over samples where ReIM is wronz
Figure 7: Distribution of Confidence Scores on Rexgen-40 dataset.


Reviews:
[{"review_id": "ZFOUl8wTmE", "reviewer": "Reviewer_cAqR", "summary": "", "questions": "A. I understand the experimental setup to utilize an out-of-distribution setting to demonstrate the advantages of LLM's knowledge. However, it's also possible to conduct the same experiments in the in-distribution setup. It would be beneficial to include those results in the appendix as well.\n\nB. How about generating confidence scores for each candidate instead of just an overall confidence score for the answer? It would be interesting to compare how these confidence scores differ from the rank produced by GNN. Particularly, in samples where GNN's predict a low rank, but LLM correctly identified the answer, it would be valuable to analyze if such chemicals are far from the distribution in the USPTO dataset.\n\nC. Also, It would be beneficial to represent examples of the reaction type and reaction condition in the Appendix.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our submission and provide thoughtful feedback. Your valuable insights into experimental design and model enhancements have greatly inspired our research. In response to the suggestions you have raised, we provide a point-to-point response as follows.\n\n> **Comment 1: Cases of Reaction Conditions** - \"The main approach of this work is its use of text information via LLM, which in turn, enables the integration of reaction type and reaction condition. These factors were previously out of reach for traditional GNNs. However, the current experimental setups don't adequately demonstrate how these factors boost the accuracy of chemical reaction predictions. An additional investigation into how the introduction of reaction condition information aids in distinguishing the correct answers from the candidates generated by GNN is needed. It would be illuminating, for instance, to present cases where identical reactants lead to different products under different reaction conditions. Offering such examples could be a valuable way to highlight this.\"\n\nThank you for insightful suggestions regarding the importance of demonstrating the influence of reaction condition. Following your suggestion, we add addtional case studies in the Imidazo dataset to illuminate the pivotal role of reaction conditions of ReLM. \n\nFor clarity, we present two cases, each showcasing a subtly different prompt design, accompanied by ChatGPT's sketched responses. We intentionally exclude in-context examples, allowing the language model to elucidate its answers.\n```\nQuestion1: Which of the following is the major product of the organic reaction between 2,3-dihydroxybenzaldehyde<C(c1cccc(c1O)O)=O>, 5-methylpyridin-2-amine<CC1=CN=C(C=C1)N> and isocyanocyclohexane<[C-]#[N+]C1CCCCC1>, with 2,2,2-trifluoroacetic acid<C(=O)(C(F)(F)F)O> as catalyst? The given reaction is a multicomponent reaction that leads to the synthesis of diverse imidazopyridines.\nA. ethyl 4,5-difluoro-2-isocyano-1,3-dihydroindene-2-carboxylate<[C-]#[N+]C1(C(=O)OCC)Cc2ccc(F)c(F)c2C1>\nB. methyl 4-isocyanocyclohexane-1-carboxylate<[C-]#[N+]C1CCC(C(=O)OC)CC1>\nC. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol<Cc1ccc2nc(c3cccc(c3O)O)c(NC3CCCCC3)n2c1>\nD. 2-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]-6-ethoxyphenol<CCOc1cccc(c1O)c1c(NC2CCCCC2)n2cc(C)ccc2n1>\n\nChatGPT sketched response: \nThe question describes a multi-component reaction between three reactants:\n….\nGiven the reactants, the most logical major product seems to be:\nC. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol.\n```\n```\nQuestion2: Which of the following is the major product of the organic reaction between 2,3-dihydroxybenzaldehyde<C(c1cccc(c1O)O)=O>, 5-methylpyridin-2-amine<CC1=CN=C(C=C1)N> and isocyanocyclohexane<[C-]#[N+]C1CCCCC1>?\nA. ethyl 4,5-difluoro-2-isocyano-1,3-dihydroindene-2-carboxylate<[C-]#[N+]C1(C(=O)OCC)Cc2ccc(F)c(F)c2C1>\nB. methyl 4-isocyanocyclohexane-1-carboxylate<[C-]#[N+]C1CCC(C(=O)OC)CC1>\nC. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol<Cc1ccc2nc(c3cccc(c3O)O)c(NC3CCCCC3)n2c1>\nD. 2-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]-6-ethoxyphenol<CCOc1cccc(c1O)c1c(NC2CCCCC2)n2cc(C)ccc2n1>\n\nChatGPT sketched response:\nThe given reactants are:\n…\n\nDrawing out the structure, the major product is:\nO=C(O)C(NC(=O)c1cccc(c1O)O)N=C(N)c2cccnc2C1CCCCC1.\n\n```\n\nQuestion 1 is the target reaction given correct catalyst. While question2 omits the important catalyst. For GNN, it incorrectly predict B as the answer, which is not even the correct product without the given catalyst. In contrast, language model accurately identify option C for Question 1, which aligns with the ground truth. Suprisingly, for Question 2, the language model predict a product that was not present in the initial candidate pool.\n\nThese case studies clearly demonstrate that identical reactants yield different products under different reaction conditions, further reinforcing our driving hypothesis: the incorporation of reaction type and condition boost the predictive accuracy of ReLM.\n\nWe will incorporate such case studies into the appendix section of our paper to facilitate readers' comprehension of the significance of reaction conditions in chemical reactions. We are grateful for your recommendation to include these illustrative examples.\n\nFYI, the detailed analysis of this reaction is provided below:\nThis organic reaction involves three components - an aldehyde, an amine, and an isocyanide. Without an added catalyst, the combination of these three reactants undergoes a Passerini reaction, which is a type of multi-component condensation. In a Passerini reaction, the aldehyde gets oxidized in situ to a carboxylic acid. Then, the carboxylic acid, amine, and isocyanide condense together to form an α-acyloxy amide product.\n\nSpecifically, in this reaction the aldehyde 2,3-dihydroxybenzaldehyde gets oxidized to 2,3-dihydroxybenzoic acid. This condenses with the amine 5-methylpyridin-2-amine and the isocyanide isocyanocyclohexane. The expected major product is an α-acyloxy amide with the IUPAC name 2-[[amino-(2-cyclohexylpyridin-3-yl)methylidene]amino]-2-[(2,3-dihydroxybenzoyl)amino]acetic acid. The structure contains the 2,3-dihydroxybenzoyl group from the oxidized aldehyde, the 5-methylpyridin-2-amine portion, and the isocyanocyclohexane portion.\n\nHowever, in the presence of the trifluoroacetic acid catalyst, the reaction proceeds via an imidazopyridine formation mechanism. The aldehyde and the amine first react in a condensation reaction, with loss of water, to form an imine intermediate. The imine then cyclizes via intramolecular nucleophilic attack to form a dihydropyridine intermediate. Dehydrogenation of the dihydropyridine gives an imidazopyridine heterocycle. The isocyanocyclohexane then attacks the imine carbon of the imidazopyridine, displacing the hydroxyl group. This leads to the major product being a 2,3-disubstituted imidazopyridine attached to the cyclohexylamine group. The specific major product is C.  3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol.\n\n\n> **Comment 2: Model accuracy across $K$ values** - \"To ensure that the LLM is predicting chemical reactions accurately, the experimental results should incorporate a wider range of values for K, not limited to just 3 and 4. To provide more context, the current framework feeds LLM with answer candidates generated by the GNN. At its core, LLM then selects the correct answer from these candidates. If LLM indeed possesses the capability to accurately pinpoint the correct answer among a variety of candidates, it's important to check whether its performance stays consistent or even gets better as the value of K goes up. Increasing K essentially means adding less plausible options to the candidates, yet it's crucial to see if LLM maintains its accuracy in selecting the correct answer despite the added choices.\"\n\nWe appreciate the reviewer's insightful suggestion to evaluate our model's performance across a wider range of $K$ values. Following your suggestion, we add an additional experiments over different $K$ values on diverse datasets. The findings are presented in Table 1 using MolR as the GNN backbone and Vicuna as the language model.\n\nWe observe that: the ReLM’s performance remains relatively stable as we increase $K$ across a wide range of $K$ values (e.g. $K=2\\sim 7$). However, with very large $K$ values (e.g. $K>=10$), the performance does start to degrade.\n\nThis drop in accuracy with extremely large K values is likely because the candidates become overly saturated with implausible options, making it more difficult for the language model to discern the correct answer. As the reviewer pointed out, increasing K means adding more distracting and unlikely candidates. While the model is robust to these distractors up to a point, at some threshold of implausible options the task does become more challenging.\n\nTable 1. Experiments over different $K$ values.\n\n|           | Imidazo  |           | NiCOlit  |           | rexgen-30k |           | rexgen-40k |           |\n| --------- | -------- | --------- | -------- | --------- | ---------- | --------- | ---------- | --------- |\n| $K$ value | ReLM Acc | GNN hit@K | ReLM Acc | GNN hit@K | ReLM Acc   | GNN hit@K | ReLM Acc   | GNN hit@K |\n| 1         | -        | 51.30%    | -        | 43.70%    | -          | 47.13%    | -          | 44.89%    |\n| 2         | 79.43%   | 81.25%    | 41.92%   | 59.30%    | 48.40%     | 55.57%    | 44.51%     | 51.10%    |\n| 3         | 91.41%   | 94.53%    | 51.00%   | 64.09%    | 49.75%     | 58.46%    | 47.28%     | 56.25%    |\n| 4         | 87.76%   | 97.92%    | 52.32%   | 67.91%    | 49.88%     | 61.10%    | 47.27%     | 58.69%    |\n| 5         | 90.89%   | 98.70%    | 55.32%   | 72.46%    | 46.70%     | 62.91%    | 44.68%     | 61.05%    |\n| 7         | 89.84%   | 99.48%    | 56.86%   | 79.85%    | 46.66%     | 65.18%    | 44.72%     | 65.37%    |\n| 10        | 85.42%   | 99.74%    | 45.82%   | 85.38%    | 45.33%     | 69.84%    | 42.89%     | 68.39%    |\n\nWe will highlight this findings in our revised manuscript to fully characterize the model's prediction capabilities. We appreciate the thoughtful feedback.\n\n> **Comment 3: Analysis of confidence score** - \"Referring to the study on LLM's confidence score [1], unless supervised fine-tuning (Calibration) is conducted on the confidence score, the score itself is not very reliable. This study(ReLM) claims that merely producing the confidence score increases the accuracy of reaction prediction. However, further analysis on this point is needed. \"\n\nWe appreciate the reviewer highlighting the significance of confidence score in our paper, an aspect worthing further exploration.\nAs delineated in Table 5 of our paper, reactions with higher confidence scores are prone to accurate predictions compared to their low-confidence counterparts. This underscores the language model's capability for self-assessment.\n\nTo further address your concerns, we **add additional experiments**  and **statistical significance test** to show that even uncalibrated confidence scores offer valuable insights for ReLM’s introspection.\n\nOur experiments are conducted on Rexgen-40 dataset, using Vicuna + MolR as backbones and our findings are presented in Table 2.\n\n Table 2. The distribution of confidence scores.\n\n|                                          | 1    | 2     | 3    | 4    | 5    | 6     | 7     | 8     | 9     |\n| ---------------------------------------- | ---- | ----- | ---- | ---- | ---- | ----- | ----- | ----- | ----- |\n| Distribution over all samples            | 3.2% | 4.3%  | 0.5% | 1.9% | 1.1% | 12.9% | 15.2% | 25.0% | 35.8% |\n| Distribution over correct ReLM choices   | 1.6% | 2.4%  | 0.4% | 1.5% | 1.0% | 10.8% | 16.0% | 27.3% | 39.0% |\n| Distribution over incorrect ReLM choices | 9.6% | 12.1% | 1.0% | 3.3% | 1.5% | 21.3% | 12.1% | 15.9% | 23.2% |\n\nWe observe that:\n- There is a clear difference between verbalized probability [1] and confidence score of ReLM. The scores of our method are distributed across 1 to 9, while verbalized probability mainly fall within the 80\\% to 100\\% range, as tested by Xiong et al. [3]. This result underscores that verbalized probabilities may not serve as reliable indicators for assessing the outcomes of language models. A promising research direction for improved assessment could be the confidence scores inherently generated by the large language model.\n- Confidence score of ReLM exhibites significantly different distributions between correct and incorrect answers, a trend corroborated by a p-value of less than 5%. For incorrect ReLM choices, the proportion of high confidence scores ($7\\sim 9$) decreased from 82.3% to 51.2%, while that of low confidence scores ($1\\sim 3$) increased from 4.4% to 22.7%.\n\n\nThe reviewer makes a fair critique, we fully agree that more evidence is needed before concluding confidence elicitation directly boosts prediction accuracy. In future revisions, we will thoroughly analyze the relationships between confidence scores and performance through controlled experiments.\n\n\n> **Comment 4: Comparing with more Baselines** - \"The current baseline for the confidence score strategy is not proper, but other forms of prompts related to reasoning could serve as potential baselines (e.g. [2]).\"\n\nWe appreciate the reviewer suggesting the chain-of-thought (CoT) [2] as new baselines.\n\nFollowing your suggestion, we **add additional experiments** using Zero-shot, Few-shot CoT, Zero-Shot CoT as new baselines as shown in Table 3.\n\nTable 3. The comparison with more baselines.\n\n|               | Imidazo | NiCOlit | rexgen-30k | rexgen-40k |\n| ------------- | ------- | ------- | ---------- | ---------- |\n| MolR          | 51.30%  | 43.70%  | 47.13%     | 44.89%     |\n| Zero-shot     | 86.98%  | 44.60%  | 30.05%     | 28.62%     |\n| Few-shot CoT | 78.13%  | 30.54%  | 26.70%     | 24.43%     |\n| Zero-shot CoT | 84.38%  | 41.51%  | 31.75%     | 30.17%     |\n| **ReLM**      | 87.76%  | 52.32%  | 49.88%     | 47.27%     |\n| MolR hit@K    | 97.92%  | 67.91%  | 61.10%     | 58.69%     |\n\n\nWe try our best to implement task-specific versions of these baseline methods for chemical reaction prediction. However, due to time constraints, we do not explore optimal prompt engineering for these baselines, which we leave for future work. \n\nWe observe that all prompt designs offer some benefits. However, our original confidence scoring approach still outperforms these baselines. Specifically, as shown in the case studies below, the language model's CoT analysis may not provide additional insightful information regarding the reaction mechanism. At this stage, incorporating additional analytical steps may introduce more molecular structures, exacerbating the language model's comprehension difficulties. Further step-by-step elucidation of the reaction process likely necessitates incorporating more domain knowledge of chemical reactions.\n\nTo provide more detailed insight, we include several case studies that compare our confidence score strategy against reasoning-based baselines.\n\n```\nQuestion: Which of the following is the major product of the organic reaction between 2-methoxynaphthalene<COc1ccc2ccccc2c1> and lithium;1,3,5-trimethylbenzene-6-ide<Cc1cc(C)c([Li])c(C)c1>? The given reaction is a Nickel-catalyzed cross-couplings reaction, which forms carbon-carbon (C-C) and carbon-heteroatom (C-X, where X can be O, N, S, etc.) bonds.\nA. 2-(2,4,6-trimethylphenyl)naphthalene<Cc3cc(C)c(c2ccc1ccccc1c2)c(C)c3>\nB. 1-(4-methoxyphenyl)-3,5-dimethylbenzene<COc2ccc(c1cc(C)cc(C)c1)cc2>\nC. 1-(2,4,6-trimethylphenyl)naphthalene<Cc3cc(C)c(c1cccc2ccccc12)c(C)c3>\nD. 1,3,5-trimethyl-2-phenylbenzene<Cc2cc(C)c(c1ccccc1)c(C)c2>\n```\nGround truth: A. 2-(2,4,6-trimethylphenyl)naphthalene<Cc3cc(C)c(c2ccc1ccccc1c2)c(C)c3>\n\nAnswer by Vicuna under **confidence score strategy**:\n```\nAnswer: A\nConfidence Score: 8\n```\n\nAnswer by Vicuna under **Zero-shot CoT** strategy [2]:\n```\nAnswer: Let's think step by step.\nThe organic reaction between 2-methoxynaphthalene< COc1ccc2ccccc2c1> and lithium;1,3,5-trimethylbenzene-6-ide< Cc1cc(C)c([Li])c(C)c1> is a Nickel-catalyzed cross-couplings reaction, which forms carbon-carbon (C-C) and carbon-heteroatom (C-X, where X can be O, N, S, etc.) bonds.\n The major product of this reaction is 1,3,5-trimethyl-2-phenylbenzene<Cc2cc(C)c(c1ccccc1)c(C)c2>.\n The other options are 2-(2,4,6-trimethylphenyl)naphthalene<Cc3cc(C)c(c2ccc1ccccc1c2)c(C)c3>, 1-(4-methoxyphenyl)-3,5-dimethylbenzene<COc2ccc(c1cc(C)cc(C)c1)cc2>, and 1-(2,4,6-trimethylphenyl)naphthalene<Cc3cc(C)c(c1cccc2cccc12)c(C)c3>.\n So, the correct answer is D. 1,3,5-trimethyl-2-phenylbenzene< Cc2cc(C)c(c1ccccc1)c(C)c2>.\nTherefore, the answer (one capital letter, presenting your choice) is \nD. 1,3,5-trimethyl-2-phenylbenzene<Cc2cc(C)c(c1ccccc1)c(C)c2>.\n```\n\nAnswer by Vicuna under **Few-shot CoT** strategy:\n```\nThe reaction between 2-methoxynaphthalene< COc1ccc2ccccc2c1> and lithium;1,3,5-trimethylbenzene-6-ide< Cc1cc(C)c([Li])c(C)c1> is a Nickel-catalyzed cross-couplings reaction, which forms carbon-carbon (C-C) and carbon-heteroatom (C-X, where X can be O, N, S, etc.) bonds.\n The reaction is a cycloaddition reaction, where the -OH group of lithium is replaced by the oxygen atom from the benzene moiety. The result is the attachment of the naphthalene ring to the benzoic acid moiety through an oxygen atom, yielding the compound 2-(2,4,6-trimethylphenyl)naphthalene. This process involves a substitution reaction, replacing the iodine atom with the oxygen atom from the alcohol group.\n Option D is duplicated and should not be considered. This prediction is based on the cycloaddition reaction and the potential for rearrangements during the reaction in aqueous conditions.\n My choice: D\n```\nIn the revised manuscript, we will include detailed analyses highlighting the performance comparison between our confidence scoring approach and these reasoning-based baselines.\n\n\n> **Comment 5: In-distribution Experiments** - \"I understand the experimental setup to utilize an out-of-distribution setting to demonstrate the advantages of LLM's knowledge. However, it's also possible to conduct the same experiments in the in-distribution setup. It would be beneficial to include those results in the appendix as well.\"\n\nThank you for raising this insightful point about including in-distribution experiments. We agree that showing performance on both in-distribution and out-of-distribution test sets can provide a more comprehensive view of the model's capabilities. Following your suggestion, we **conduct new in-distribution experiments** on USPTO-479 dataset, results shown in Table 4. \n\n\nTable 4. Accuracy on in-distribution settings.\n\n|               | MolR   | LocalRetro |\n| ------------- | ------ | ---------- |\n| GNN           | 88.20% | 56.63%     |\n| **ReLM (Vicuna)** | 87.09% | 62.73%     |\n| GNN hit@4     | 95.27% | 75.83%     |\n\nWe observe that: due to the GNN itself achieving relatively high accuracy in in-distribution scenarios, the performance gains afforded by ReLM were limited in those cases. Moreover, the absence of reaction type and condition information in the USPTO dataset posed inference challenges for the language model. Nonetheless, ReLM still exhibited a certain capacity to select the correct answers from the options, without exhibiting significant performance deterioration.\n\nThe USPTO dataset comprises numerous chemical reactions. Due to time constraints, we have thus far only conducted experiments under certain settings (Vicuna as the language model, $K=4$). Comprehensive experiments will be completed and reported in future revisions of the paper.\n\n> **Comment 6: Generate Confidence Scores for each candidate** - \"How about generating confidence scores for each candidate instead of just an overall confidence score for the answer? It would be interesting to compare how these confidence scores differ from the rank produced by GNN. Particularly, in samples where GNN's predict a low rank, but LLM correctly identified the answer, it would be valuable to analyze if such chemicals are far from the distribution in the USPTO dataset.\"\n\nThank you for your insightful comments. I have carefully considered your suggestion to generate confidence scores for each candidate instead of just an overall confidence score for the answer. Generating confidence scores for each candidate might be better aligned with the framework of our proposed methodology. Thanks again.\n\nFollowing your suggestion, we conducted **new experiments** using MolR as the GNN backbone, Vicuna as the language model, and $K=4$, and recorded the results in Table 5. In addition to model accuracy, we also reported the correlation between the language model and GNN model answers. We used Spearman's rank correlation coefficient as the metric to measure how the rank generated by the language model (calculated based on the option-wise confidence scores) correlates with the original ranking from the GNN model. \n\nTable 5. Results of 'Confidence scores for each candidate' stratety.\n\n|                                           | Imidazo | NiCOlit | rexgen-30k | rexgen-40k |\n| ----------------------------------------- | ------- | ------- | ---------- | ---------- |\n| GNN Acc                                   | 51.30%  | 43.70%  | 47.13%     | 44.89%     |\n| **ReLLM Acc**                                 | 86.98%  | 50.69%  | 44.91%     | 42.10%     |\n| rank correlation over all samples         | 0.3632  | 0.3593  | 0.3457     | 0.3466     |\n| rank correlation over correct GNN choices | 0.4691  | 0.4292  | 0.3883     | 0.3971     |\n| rank correlation over incorrect GNN choices | 0.2425  | 0.2271  | 0.1993     | 0.1743     |\n\n\nWe obseve that: the language model's overall rankings positively correlate with the GNN model's rankings, meaning the language model agrees with most, but not all, of the GNN's analytical results. Importantly, for questions where the GNN made incorrect choices (last row in table), the language model's rankings showed a significantly lower correlation with the GNN's rankings. On the contrary, For those where the GNN made correct choices, the correlation between the LLM's answers and the GNN's increased. The above observations indicate that the language model forms its own informed opinions on the answers, and differs from the GNN when GNN makes incorrect choices.\n\n\n> **Comment 7: Examples of Reaction Conditions** - \"Also, It would be beneficial to represent examples of the reaction type and reaction condition in the Appendix.\"\n\nThank you for highlighting the omission in our paper regarding the description of reaction information. We will systematically arrange the reaction type and reaction condition data from the downstream datasets, and provide its distribution along with illustrative examples in the forthcoming revisions of the paper.\n\n> **Minor Comment 8: Research Question Clarification** - \"Section 4 Experimental Results, 'RQ1 : Can ReLM improve the reasoning ability of graph neural networks in out-of-distribution scenarios?' (222-223) The term 'reasoning ability' needs to be clarified in this sentence. It might confuse the reader into thinking it refers to the reasoning for the interpretability of predictions.\"\n\nWe appreciate the reviewer noting the ambiguity of \"reasoning ability\". To clarify, we will revise the sentence to: \"RQ1: Can ReLM improve the reaction prediction capability of graph neural networks on out-of-distribution data?\" This refers specifically to prediction accuracy, not interpretability. Please let us know if this revision provides sufficient clarification. Thank you for improving the clarity of our work.\n\n[1] Teaching models to express their uncertainty in words\n\n[2] Large Language Models are Zero-Shot Reasoners\n\n[3] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper provides a detailed analysis of the proposed approach, including experiments, statistical analysis, and ablation studies, which can help researchers better understand the strengths and limitations of ReLM.\n\nB. Considering all GNN-based deep neural networks lack the capability to leverage reaction conditions and types, It is noteworthy that ReLM approach aims to utilize the reaction type and reaction condition information in prediction, which are essential factors in chemical reactions. Therefore, this work certainly has aspects that can captivate the readers' interest.\n\nC. Leveraging the knowledge of LLM in out-of-distribution scenarios to improve prediction performance is both consolidated and has a practical impact.", "reasons_to_reject": "A. The main approach of this work is its use of text information via LLM, which in turn, enables the integration of reaction type and reaction condition. These factors were previously out of reach for traditional GNNs. However, the current experimental setups don't adequately demonstrate how these factors boost the accuracy of chemical reaction predictions. An additional investigation into how the introduction of reaction condition information aids in distinguishing the correct answers from the candidates generated by GNN is needed. It would be illuminating, for instance, to present cases where identical reactants lead to different products under different reaction conditions. Offering such examples could be a valuable way to highlight this.\n\nB. To ensure that the LLM is predicting chemical reactions accurately, the experimental results should incorporate a wider range of values for K, not limited to just 3 and 4. To provide more context, the current framework feeds LLM with answer candidates generated by the GNN. At its core, LLM then selects the correct answer from these candidates. If LLM indeed possesses the capability to accurately pinpoint the correct answer among a variety of candidates, it's important to check whether its performance stays consistent or even gets better as the value of K goes up. Increasing K essentially means adding less plausible options to the candidates, yet it's crucial to see if LLM maintains its accuracy in selecting the correct answer despite the added choices.\n\nC. Referring to the study on LLM's confidence score at **https://arxiv.org/pdf/2205.14334.pdf**, unless supervised fine-tuning (Calibration) is conducted on the confidence score, the score itself is not very reliable. This study(ReLM) claims that merely producing the confidence score increases the accuracy of reaction prediction. However, further analysis on this point is needed. The current baseline for the confidence score strategy is not proper, but other forms of prompts related to reasoning could serve as potential baselines (e.g. **https://arxiv.org/pdf/2205.11916.pdf**).", "questions_to_authors": "A. I understand the experimental setup to utilize an out-of-distribution setting to demonstrate the advantages of LLM's knowledge. However, it's also possible to conduct the same experiments in the in-distribution setup. It would be beneficial to include those results in the appendix as well.\n\nB. How about generating confidence scores for each candidate instead of just an overall confidence score for the answer? It would be interesting to compare how these confidence scores differ from the rank produced by GNN. Particularly, in samples where GNN's predict a low rank, but LLM correctly identified the answer, it would be valuable to analyze if such chemicals are far from the distribution in the USPTO dataset.\n\nC. Also, It would be beneficial to represent examples of the reaction type and reaction condition in the Appendix.", "paper_topic_and_main_contributions": "The paper topic is ReLM, a framework that leverages language models to enhance chemical reaction prediction. The main contributions of the paper are introducing a novel approach that combines language models with graph neural networks to improve the accuracy of chemical reaction prediction, demonstrating the effectiveness of ReLM in out-of-distribution settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 4 Experimental Results, \"RQ1 : Can ReLM improve the reasoning ability of graph neural networks in out-of-distribution scenarios?\" (222-223)\nThe term \"reasoning ability\" needs to be clarified in this sentence. It might confuse the reader into thinking it refers to the reasoning for the interpretability of predictions."}}, {"review_id": "feMmzFbxUJ", "reviewer": "Reviewer_ZhrP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough and valuable feedback. To address your concerns, we present the point-to-point responses as follows.\n\n> **Comment 1: Candidate Selection** — \"In the section 3.1, if we consider all molecules or a very large set of molecules as the candidate pool C, it might be hard to select related answer candidates there.\"\n\nThank you for pointing out the ambiguity in the methodology of this paper. We totally agree with you that it’s hard for language models to tackle a large amount of candidate molecules. \n\n**Additional experiments** were conducted to demonstrate this perspective. By varying the value of $K$ in our method, we test the effect of the number of answer candidates (using Vicuna as the language model and MolR as the GNN), as shown in Table 1. While our model's performance remained relatively stable across different values of $K$, the redundant information introduced by an excessively large $K$ ($K=10$) still led to an obvious accuracy drop. This implies that increasing $K$ to the size of the entire database would result in a significant decrease in model accuracy due to information redundancy.\n\nTable 1. Experiments over different $K$ values.\n\n|           | Imidazo  |             | NiCOlit  |             | rexgen-30k |             | rexgen-40k |             |\n| --------- | -------- | ----------- | -------- | ----------- | ---------- | ----------- | ---------- | ----------- |\n| $K$ value | ReLM Acc | GNN hit@$K$ | ReLM Acc | GNN hit@$K$ | ReLM Acc   | GNN hit@$K$ | ReLM Acc   | GNN hit@$K$ |\n| 1         | -        | 51.30%      | -        | 43.70%      | -          | 47.13%      | -          | 44.89%      |\n| 2         | 79.43%   | 81.25%      | 41.92%   | 59.30%      | 48.40%     | 55.57%      | 44.51%     | 51.10%      |\n| 3         | 91.41%   | 94.53%      | 51.00%   | 64.09%      | 49.75%     | 58.46%      | 47.28%     | 56.25%      |\n| 4         | 87.76%   | 97.92%      | 52.32%   | 67.91%      | 49.88%     | 61.10%      | 47.27%     | 58.69%      |\n| 5         | 90.89%   | 98.70%      | 55.32%   | 72.46%      | 46.70%     | 62.91%      | 44.68%     | 61.05%      |\n| 7         | 89.84%   | 99.48%      | 56.86%   | 79.85%      | 46.66%     | 65.18%      | 44.72%     | 65.37%      |\n| 10        | 85.42%   | 99.74%      | 45.82%   | 85.38%      | 45.33%     | 69.84%      | 42.89%     | 68.39%      |\n\nTo address this issue, graph neural networks are utilized to encode molecular graph embeddings and perform candidate ranking to obtain the Top-$K$ most suitable candidates, effectively reducing the number of answer candidates. Importantly, product candidate features can be pre-computed and cached, independent of the query reactants. This allows the graph network to swiftly retrieve candidates over a large candidate pool.\n\n\n> **Comment 2: Top-$N$ Nearest Neighbor Selection** — \"In addition, it is not clear about Top-$N$ nearest neighbor selection. Please share more information including how to get the embeddings of reaction samples.\"\n\nYour concerns regarding the Top-$N$ nearest neighbor selection are well-founded. During the Top-$N$ nearest neighbor selection (i.e., Equation (3)), the objective is to retrieve the $N$ most similar reactions from the training set, given the input set of reactants $R$. This process is implemented by sequentially comparing the summed input reactant features to the summed reactant features for each reaction in the training set. More precisely,, the embedding of the i-th training reaction sample $h_{R_i’}$ is calculated by:\n$$\nh_{R_i’}=\\sum_{r\\in R_i'} G(r|\\theta)\n$$\nwhere $R_i'$ denotes the reactants of the i-th training sample, and $G(\\cdot|\\theta)$ is a graph neural network parameterized by $\\theta$.\n\nGiven the query and training sample embeddings, this paper implements the k-nearest neighbors (KNN) algorithm with cosine similarity as the similarity metric to retrieve the N training samples most similar to the input data.\n\nThe reviewer rightly notes that the methodology for this section lacks detailed elaboration. In future revisions, we will expand this portion with a more comprehensive delineation of the procedures.\n\n> **Comment 3: Structure Relations between molecules**  — \"This method hasn't considered the structure relations between input molecules and the output products.\"\n\nThank you for your thoughtful feedback. We fully concur on the importance of modeling the structural relations between reactants and products for predicting chemical reactions. To enable this, we put forth graph neural networks to encode reactant-product correlations and integrate them with language models to augment reasoning. Owing to this hybrid approach combining GNN molecular understanding and language model knowledge and inference, ReLM demonstrates remarkable gains on certain downstream tasks (as evidenced in Table 1 of the paper).\n\nTo further illustrate the aforementioned capabilities, we provide a case study from the Imidazo dataset:\n\n```\nQuestion: Which of the following is the major product of the organic reaction between 2,3-dihydroxybenzaldehyde<C(c1cccc(c1O)O)=O>, 5-methylpyridin-2-amine<CC1=CN=C(C=C1)N> and isocyanocyclohexane<[C-]#[N+]C1CCCCC1>, with 2,2,2-trifluoroacetic acid<C(=O)(C(F)(F)F)O> as catalyst?\nA. methyl 4-isocyanocyclohexane-1-carboxylate<[C-]#[N+]C1CCC(C(=O)OC)CC1>\nB. ethyl 4,5-difluoro-2-isocyano-1,3-dihydroindene-2-carboxylate<[C-]#[N+]C1(C(=O)OCC)Cc2ccc(F)c(F)c2C1>\nC. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol<Cc1ccc2nc(c3cccc(c3O)O)c(NC3CCCCC3)n2c1>\nD. 2-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]-6-ethoxyphenol<CCOc1cccc(c1O)c1c(NC2CCCCC2)n2cc(C)ccc2n1>\n```\n\nGround truth: C. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol\n\nAnswer by GPT-3.5 (without the choices):\n```\n2-(5-methylpyridin-2-yl)-3-(2,3-dihydroxyphenyl)-1H-pyrrole-1-carbonitrile.\nConfidence Score: 8\n```\n\nAnswer by Vicuna (without the choices):\n```\n[C-]#[N+]C1CCCCC1>\nConfidence Score: 7\n```\n\nAnswer by Both LM (with the choices):\n```\nC. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol\nConfidence Score: 7\n```\n\n\nThank you again for the valuable feedback. Please let us know if any clarification or additional experiments would be helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1.\tThis paper proposes a new in-context learning framework that combines both the language model and GNN method. \n2.\tThe confidence score strategy is proposed to enhance the robustness and interpretability.", "reasons_to_reject": "Weaknesses:\n1. In the section 3.1, if we consider all molecules or a very large set of molecules as the candidate pool C, it might be hard to select related answer candidates there. In addition, it is not clear about Top-N nearest neighbor selection. Please share more information including how to get the embeddings of reaction samples.\n2. This method hasn't considered the structure relations between input molecules and the output prodcuts.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a in-context learning method that utilized both the language model and GNN methods for the chemical reaction task. First, this method extracts the top-K answer candidates by GNN and similarity measurement. These will be the answer candidates. Second, for the in-context learning, this framework also extracts the top-N nearest neighbors. The similarity is calculated using the embeddings. The proposed method outperforms the SOTA GNN-based method. More information can be added into this framework. For example, you can extract some related text about the molecules or generate the description using the language models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GqR4JUIF3b", "reviewer": "Reviewer_i7LA", "summary": "", "questions": "1.  Eqn (1): What is the size of the reactants R? Is it a list that contains all the reactants or a single reactant?  Is it fixed for a query? If there are multiple reactants, how do you leverage GNN to select top-K candidates? Do you pick top-K from only one of the reactants or all of them?\n\n2. As discussed in weakness, for your confidence score strategy, why pick [1,2] and [8,9]? Why not just [1] and [9] or [1,2,3] and [7,8,9]? Wouldn’t the randomness bias the model?\n\n3. Could you explain the difference between the baseline between the first row (method name) and last row (hit@K), and explain why hit@K is not enough to get results? How do you plan to extend this work to increase the generalizability and adaptability of ReLM? \n\nQuestion importance: 3 > 2 > 1. I’d love to change my score based on the author's response.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and insightful comments. Your primary suggestions regarding our motivation and methodology have helped us improve the clarity of our work. To address your concerns, below we present the point-to-point responses.\n\n> **Comment 1: Significance of proposed method** — \"The choice of baselines is a little bit confusing. If pre-training an in-house GNN is always required, and GNN hit@$K$ is a theoretical upper bound, then why do we still need ReLM to introduce textual knowledge? This question is important as it is closely related to the use case of ReLM.\"\n\nWe appreciate you asking this important question. The hit@$K$ metric measures the performance of a ranking algorithm by checking whether the ground truth is within the Top-$K$ results. For example, hit@3 measures whether the correct result is in the Top-3 ranked items. Previous GNN approaches only considered the top rank 1 prediction as the final output, so model accuracy was solely determined by hit@1.\n\nIn this paper, we propose analyzing the Top-$K$ results of the GNN model and using a language model to select the most plausible option as the final prediction. This enables potentially selecting the correct answer from the GNN's Top-$K$ choices (instead of Top-1). Therefore, we refer to the GNN's hit@$K$ metric as the theoretical upper bound on the accuracy of our model.\n\nTo clarify, we prodive a step-by-step example of our methodology when $K=4$ below:\n\n- Step 1: A reaction is sampled from the test set. Its associated reactants, reaction conditions, and other relevant information are provided as input.\n\t- reactants 1: 2,3-dihydroxybenzaldehyde\n\t- reactants 2: 5-methylpyridin-2-amine\n\t- reactants 3: isocyanocyclohexane\n\t- catalyst 1: 2,2,2-trifluoroacetic acid\n\t- reaction type: a multicomponent reaction that leads to the synthesis of diverse imidazopyridines\n- Step 2: The GNN generates a ranked list of the top 4 product candidates (i.e., the four choices A, B, C, and D in step 4) for the above set of reactants.\n- Step 3: Proper in-context examples are chosen from the training set using the GNN model.\n- Step 4: Generate prompts based on Top-$K$ possible answer candidates from the GNN model. (The in-context examples are omitted for brevity.)\n\n```\nQuestion: Which of the following is the major product of the organic reaction between 2,3-dihydroxybenzaldehyde<C(c1cccc(c1O)O)=O>, 5-methylpyridin-2-amine<CC1=CN=C(C=C1)N> and isocyanocyclohexane<[C-]#[N+]C1CCCCC1>, with 2,2,2-trifluoroacetic acid<C(=O)(C(F)(F)F)O> as catalyst?\nA. methyl 4-isocyanocyclohexane-1-carboxylate<[C-]#[N+]C1CCC(C(=O)OC)CC1>\nB. ethyl 4,5-difluoro-2-isocyano-1,3-dihydroindene-2-carboxylate<[C-]#[N+]C1(C(=O)OCC)Cc2ccc(F)c(F)c2C1>\nC. 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol<Cc1ccc2nc(c3cccc(c3O)O)c(NC3CCCCC3)n2c1>\nD. 2-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]-6-ethoxyphenol<CCOc1cccc(c1O)c1c(NC2CCCCC2)n2cc(C)ccc2n1>\n```\n\n- Step 5: The LM chooses C as the final answer with a confidence level of 7, while the GNN model chooses **A** (the rank 1 among all the molecule corpus). The ground truth is **C**, 3-[3-(cyclohexylamino)-6-methylimidazo[1,2-a]pyridin-2-yl]benzene-1,2-diol.\n\nYou make an excellent point that our paper is currently missing detailed case examples illustrating this process. We will address this oversight by adding case studies to the appendix to demonstrate how our model selects the final prediction from the GNN's Top-K candidates. Thank you again for highlighting this gap - the additional examples will greatly improve the clarity and completeness of our approach description.\n\n> **Comment 2: Novelty and Contribution** — \"The novelty of methodology looks limited. The main idea contribution lies mostly on the prompt design part.\"\n\nWe appreciate you highlighting this context around the goal and scope of our exploration. The additional background helps situate the motivation for our approach. Recent studies have demonstrated that large language models alone struggle with molecular understanding, hampering performance on tasks like reaction prediction [1]. This paper explores new methods to combine the molecular modeling strengths of graph neural networks (GNNs) with the broad chemical knowledge and reasoning ability of language models (LMs).\n\nImportantly, our focus here is investigating techniques for integrating these models, rather than developing cutting-edge prompts. So the prompting methodology prioritizes intuitive chemical reaction questions answering rather than pursuing state-of-the-art engineering solutions for this task. \n\n> **Comment 3: Randomness of In-context Confidence Scores** — \"The confidence score design is also a little bit weird, what’s the intuition to set the score choice as {1,2} and {8,9}? Why is randomly picking up a score reasonable? Why do strict score assignments (e.g., 1 for bad and 9 for good) not work? Missing explanation to this further limits the novelty of the idea itself.\" \"As discussed in weakness, for your confidence score strategy, why pick {1,2} and {8,9}? Why not just {1} and {9} or {1,2,3} and {7,8,9}? Wouldn’t the randomness bias the model?\"\n\nThank you for the valuable comments on our prompt design. Our primary intuition for incorporating randomness is to faithfully mimic human behavior when answering the multiple-choice chemistry question. Typically, humans exhibit varying degrees of confidence, even when answering correctly.\n\nYour comment has been enlightening, suggesting an intriguing question of exploration. What if we design diverse confidence labels for ReLM? In response, we **add additional comparative experiments** with MolR+Vicuna at $K=4$, using both fixed and randomized labels for the in-context examples.\n\nTable 1. Accuracy over fixed and randomized in-context confidence scores.\n\n|                                 | Imidazo | NiCOlit | rexgen-30k | rexgen-40k |\n| ------------------------------- | ------- | ------- | ---------- | ---------- |\n| MolR                            | 51.30%  | 43.70%  | 47.13%     | 44.89%     |\n| Fixed: {1} and {9}              | 91.93%  | 53.94%  | 48.65%     | 45.62%     |\n| Randomized: {1,2} and {8,9}     | 87.76%  | 52.32%  | 49.88%     | 47.27%     |\n| Randomized: {1,2,3} and {7,8,9} | 91.41%  | 53.78%  | 48.51%     | 45.52%     |\n\nWe can observe that: using varying confidence scores causes slight fluctuations in experimental outcomes, though the degree varies by dataset. This further demonstrates that the effectiveness of our ReLM lies not in the details of prompt design. Instead, it stems from our main idea of amalgamation of the molecular modeling ability inherent to GNNs with the vast reaction knowledge of the language model. \n\nWe will expand upon these investigations in the appendix.\n\n> **Comment 4: Top-$K$ Candidates Selection** — \"Eqn (1): What is the size of the reactants R? Is it a list that contains all the reactants or a single reactant? Is it fixed for a query? If there are multiple reactants, how do you leverage GNN to select top-K candidates? Do you pick top-K from only one of the reactants or all of them?\"\n\nThank you for catching the unclear explanation. To clarify how Equations (1) and (2) operate:\n- $R$ refers to the set of multiple query reactants and $r$ denotes an individual reactant\n- $P$ represents multiple product molecules on the right side of the reaction, with $p$ as a single product\n- Equation (1) sums reactant representations to retrieve Top-$K$ product candidates $P_j$\n- Importantly, $P_j$ comes from a predefined corpus so Equation (1) computes the distance between $R$ and all $P_j$\n- The GNN ranks distance $D(R, P_j)$ for all $P_j$ sets and picks the top $K$ closest as the predictions (per equation 2)\n\nIn summary, the reactant representations select the most similar Top-$K$ product sets $P_j$ from the corpus based on distance. Thank you again for identifying this point needing further improvement - please let us know if the revised explanation fails to clarify the process.\n\n> **Comment 5: Experiment Metrics and Future Work** — Could you explain the difference between the baseline between the first row (method name) and last row (hit@$K$), and explain why hit@$K$ is not enough to get results? How do you plan to extend this work to increase the generalizability and adaptability of ReLM?\n\nThank you for your interest in the experimental setup. To summarize:\n- The GNN transforms reaction prediction into a ranking task over a predefined product vocabulary $C$\n- For a given reactant $R$, the GNN ranks all $P\\in C$ candidates by distance to $R$ (Equation 1)\n- Accuracy is if the ground truth product is rank 1, i.e., GNN’s performance.\n- Hit@$K$ is if the ground truth product is in the Top-$K$ ranks.\n\nIf this is not clear enough, you can imagine that each reaction only produces one product (this holds in most cases), and the goal is to find this product out of the predetermined vocabulary $C$. We will revise the related sections to elucidate the ranking process and the scope of the product candidate space more comprehensively.\n\nOur plan to increase the generalizability and adaptability of this work:\n- The current ReLM approach is limited by the hit@K metric of the GNN model. Going forward, we may seek to address this limitation, such as by coupling the language model and graph neural network at the feature level.\n- Additionally, we aim to incorporate experimental procedure data from wet lab chemistry experiments to refine the reaction conditions in a more step-by-step manner, providing the model with richer textual information during reasoning.\n\n\nThank you again for the insightful feedback.\n\n[1] Augmenting large-language models with chemistry tools", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explores a simple way to combine knowledge from both LLM and GNN to enhance chemical prediction without the need to finetune the heavy LLM. \n\nThe improved results are shown on 2 GNN methods {MoIR, LocalRetro} x 2 LLM {GPT 3.5, Vicuna} testbed combinations. \n\nThorough ablation studies are shown to prove the effectiveness of the prompt design and confidence score strategy. I appreciate the writing as well as the experimental presentation of the works done. \n\nCode availability is good.", "reasons_to_reject": "1. The choice of baselines is a little bit confusing. If pre-training an in-house GNN is always required, and GNN hit@K is a theoretical upper bound, then why do we still need ReLM to introduce textual knowledge? This question is important as it is closely related to the use case of ReLM.\n\n2. The novelty of methodology looks limited. The main idea contribution lies mostly on the prompt design part. The confidence score design is also a little bit weird, what’s the intuition to set the score choice as [1,2] and [8,9]? Why is randomly picking up a score reasonable?  Why do strict score assignments (e.g., 1 for bad and 9 for good) not work? Missing explanation to this further limits the novelty of the idea itself.", "questions_to_authors": "1.  Eqn (1): What is the size of the reactants R? Is it a list that contains all the reactants or a single reactant?  Is it fixed for a query? If there are multiple reactants, how do you leverage GNN to select top-K candidates? Do you pick top-K from only one of the reactants or all of them?\n\n2. As discussed in weakness, for your confidence score strategy, why pick [1,2] and [8,9]? Why not just [1] and [9] or [1,2,3] and [7,8,9]? Wouldn’t the randomness bias the model?\n\n3. Could you explain the difference between the baseline between the first row (method name) and last row (hit@K), and explain why hit@K is not enough to get results? How do you plan to extend this work to increase the generalizability and adaptability of ReLM? \n\nQuestion importance: 3 > 2 > 1. I’d love to change my score based on the author's response.", "paper_topic_and_main_contributions": "This paper proposes a framework that tries to predict chemical reactions utilizing both GNN and recently emerged large language models named ReLM. \nGiven a test reaction, ReLM generates questions and answers from similar reactions together with a confidence score as in-context learning prompts and feeds them into LLM. GNN is used to provide candidate target pools. The author proposes a Confidence Score Strategy to generate a confidence score fir each question-answer pair to improve the effectiveness of LLMs. \nIn general, this is a well-written short paper with a simple method, lying more to the engineering side.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes ReLM, a framework for predicting chemical reactions. ReLM combines the capability of language models as well as graph neural networks to provide answer candidates, followed by the confidence score strategy. All reviewers agreed on the well-motivated approach for combining LM and GNN, as well as postiive experiment results.
Decision: Accept-Findings