Benchmarking Retrieval-Augmented Generation for Chemistry
Authors: Xianrui Zhong
Bowen Jin
Siru Ouyang
Yanzhen Shen
Qiao Jin
Yin Fang
Zhiyong Lu
Jiawei Han
Venue: COLM 2025 (2025)
Keywords: Retrieval-Augmented Generation
RAG
Benchmark
AI for Science
LLM
Large Language Model
Chemistry
TL;DR: We construct a comprehensive Retrieval-Augmented Generation benchmark for chemistry.

Abstract: Retrieval-augmented generation (RAG) has emerged as a powerful framework for enhancing large language models (LLMs) with external knowledge, particularly in scientific domains that demand specialized and dynamic information. 
Despite its promise, the application of RAG in the chemistry domain remains underexplored, primarily due to the lack of high-quality, domain-specific corpora and well-curated evaluation benchmarks. 
In this work, we introduce ChemRAG-Bench, a comprehensive benchmark designed to systematically assess the effectiveness of RAG across a diverse set of chemistry-related tasks. 
The accompanying chemistry corpus integrates heterogeneous knowledge sources, including scientific literature, the PubChem database, PubMed abstracts, textbooks, and Wikipedia entries. 
In addition, we present ChemRAG-Toolkit, a modular and extensible RAG toolkit that supports five retrieval algorithms and eight LLMs. 
Using ChemRAG-Toolkit, we demonstrate that RAG yields a substantial performance gain—achieving an average relative improvement of 17.4\% over direct inference methods. 
We further conduct in-depth analyses on retriever architectures, corpus selection, and the number of retrieved passages, culminating in practical recommendations to guide future research and deployment of RAG systems in the chemistry domain.

Published as a conference paper at COLM 2025
Benchmarking Retrieval-Augmenter Generation for Chem- istry
Xianrui Zhong', Bowen Jin', Siru Ouyang', Yanzhen Shen', Qiao Jiin", Yin Fang², Zhiyong Lu² & Jiawei Han
Siebel School of Computing and Data Science, University of Illinois Chaama- National Library of Medicine, National Institutes of Health (xzhong23, bowenj4, siruo2, yanzhen4, hanj eillleois.iii.... edu {qiao. fan.fang, zhiyong. lulluuuen enih.gov
Abstract
Retrieval-augmented generation (RAG) has emerged as powerful frame- work for enhancing large language models (LLMs) with external knowl- edge, particularly in scientific domains that demand specialized and dy- namic information. Despite its promise, the application of RAG in the chemistry domain remains underexplored, primarily due to the lack of high- quality, domain-specific corpora and well-curated evaluation benchmarks. In this work, we introduce CHEMRAG AG-BENCH, comprehensive bench- mark designed to systematically assess the effectiveness of RAG across a diverse set of themistry-relatedr tasks. The curated chemistry corpus inte- grates heterogeneous knowledge sources, including scientific literature, the PubChem database, PubMed abstracts, textbooks, and Wikipedia entries. modular and extensible RAG toolkit that supports five retrieval algorithms and eight LLMs. Us- ing CHEMRAG-TOOLKC performance main-achheving an average relative improvement of 17.4% over direct inference methods. We further conduct in-depth analyses on retriever architectures, corpus selection, and the number of retrieved pas- sages, culminating in practical recommendations to guide future research and deployment of RAG systems in the chemistry domain. The code and data is available at https: / /chemragg io.
In addition, we present CHEMRAG-TOOLAS
we demonstrate that RAG yields substantial
1 Introduction
Retrieval-augmented generation (RAG) (Gao et al., 2023) has emerged as powerful paradigm for enhancing large language models (LLMs) with external knowledge sources. By incorporating retrieval into the generation process, RAG can effectively mitigate hal- lucinations (Zhang et al., 2023b) and inject up-to-date domain-specific information into LLMs (Siriwardhana et al., 2023). These capabilities are particularly valuable in scientific domains, where factual accuracy and timely knowledge are critical. A typical scientific RAG system consists of two components: (1) retriever that selects relevant documents or facts from scientific knowledge base, and (2) a generator, often an LLM, that integrates the retrieved content to produce informed and coherent responses. Such frameworks have shown promising applications in domains like biomedicine (Xiong et al., 2024).
In chemistry, LLMs have shown remarkable potential across various tasks, including molec- ular captioning (Li et al., 2024), chemical reasoning (Tang et al., 2025), and reaction predic- tion (Shi et al., 2023). However, chemistry a highly specialized and dynamic discipline, characterized by complex terminologies, domain-specific conventions, and rapidly-evolvine knowledge. As a result, LLMs trained on general corpora often fail to generate grounded and accurate responses, instead producing hallucinated outdated content (Zhang et al., 2023a; Wang et al., 2024b). RAG presents a natural solution to these limitations, allowing models to retrieve and incorporate trusted chemical knowledge during inference.

---

Published as a conference paper at COLM 2025
Chem-RAG Corpora
Chemistry Understanding
The 13C scectrum which isomer C5H14 has lines with five distinct chemical shift??
Retrievers
Inxtesing
Wikipedia PubChem
Textbook
Collage-loval Chamistry
SPECTER
Contriever
At carticulor temperature. 2 10 mol Mg 2.50 How would you calculate
flask equilirnum contains 10-* mal O2 and 2.00 x 10
BM25
RRF
e.
Retrieval
PubMad
Semantic
Scholar
USPTO
Retrosynthosis
Please suggest potential moctants for the product: Ar
Retrieved Documents
Description Guided Moleculo Dasign
[Documsitt 922. Exercises - 922 Stoichiametly of Gaseous Substtacess, Mixtures, Reactionsinloos, What the dersityy of luggh...
Design molecule that meets the or teriaa outlined in the
following: appeers as
[Rocument Repió-equilbyin nate equations for the enrymati catalysis of Ai-BPPr PG over a range of pH.vii article shows how pks for the enzymatic site and enzyme ausstrate complexes can be comploxes...
DDacument
Yield Prediction
What yield might be expected chemical reaction
LLMs
described by
product:
Product Prediction
Qwen
What is the outcome
and
Liama
Answer
GPT
ChemLLM
Chem-RAG Bench
Mol-Instructions
MMLU--Cham
Chembench4K
SciBench
Figure 1: Overview of the CHEMRAG toolkit. Retrievers are first constructed from CHEM- RÁG corpora. For each question the CHEMR G- BENCH benchmark, we retrieve related documents additional contexts for LLMs to predict the final answer.
Despite the growing interest in applying RAG to the chemistry domain, there remains a lack of standardized benchmarks and curated domain-specific resources to support rigorous evaluation and design of RAG systems. To address this gap, we introduce CHEMRAG- BENCH, novel evaluation benchmark comprising 1,932 expert-curated question-answw answer pairs covering diverse chemistry tasks. These include description-guided sign, retrosynthesis, chemical calculations, molecule captioning, name conversion, and reaction prediction. This benchmark provides a foundation for systematically evaluating the effectiveness of RAG systems in chemistry and guiding future research in this direction.
molecular de-
To facilitate comprehensive and reproducible evaluation on CHEMRA Ma BENCCH, we intro- duce CHEMRAG-TOOLKIT, user-friendly and extensible toolkit that supports chemistry- related corpora, 5 retrieval methods, and 8 LLMs, encompassing both general-purpose and domain-specific LLMs. Based on the proposed CHEMRAG-BENT conduct systematic evaluation of various CHEMRAG solutions and analyze the impact of individual components on overall performance from multiple perspectives. Across range of LLMs, we observe an average relative performance improvement of 17.4% when using CHEMRAG compared to direct inference without retrieval.
benchmark, we
Along the retrieval corpus dimension, we find that different chemistry tasks exhibit distinct preferences for specific corpora. For instance, molecule design and reaction prediction tasks benefit more from literature-deriveel corpora, while nomenclature and conversion tasks favoor structured chemical databases. These observations suggest that task-aware corpus selection is crucial for maximizing RAG performance. Moreover, show that combining all available corpora often yields the most robust results, serving as a comprehensive retrieval base. In terms of the retriever component, Contriever (Izacard et al., 2021) demonstrates consistently strong performance across tasks. We further find that performance can be enhanced by leveraging ensemble retrieval strategies that combine the strengths of multiple retrievers. Beyond standard evaluation metrics, we uncover a log-linear scaling trend between the number of retrieved passages and downstream model performance, indicating that retrieval depth plays key role in generation quality. Additionally, we investigate the proportion of external knowledge utilized per task and provide in-depth analysis on retriever selection for chemistry discovery scenarios. Finally, we distill set of practical recommendations from our findings, offering actionable insights for deploying and advancing RAG systems in the chemistry domain. In summary, our key contributions fourfold:
2

---

Published as a conference paper at COLM 2025
We introduce CHEMRAG-BENCH comprehensive benchmark comprising 1,932 expert- curated question-answer pairs across six chemistry-related knowledge sources, enabling systematic evaluation of RAG methods in the chemistry domain.
We curate CHEMRAG-CORPUS, a large-scale and comprehensive corpora construction.
We develop CHEMRAG-TOOLKIT an easy-to-use and extensible framework that inte- grates five retrieval algorithms and eight large language models, and demonstrate an average relative improvement of 17.4% when applying CHEMRAG over direct inference.
We conduct comprehensive empirical analyses to examine the impact of retrieval corpus selection, retriever architecture, the number of retrieved documents, etc. Based our findings, we provide practical guidelines to inform future research and the real-world deployment of themistry-focused RAG systems.
Related Work
2.1 Retrieval-Augmented Generation
Retrieval-augmented generation (RAG) enhances large language models by incorporating external knowledge sources (Lewis et al., 2020). It has been shown to reduce hallucinations (Ayala & Bechard, 2024) and provide access to up-to-date information (Fan et al., 2024). Recent work has sought to improve RAG performance through various enhancements, including more effective retrieval mechanisms (Glass et al., 2022), iterative retrieval-and- reasoning pipelines (Trivedi et al., 2022; Jin et al., 2025), and the integration of long-context language models to better handle extended inputs (Jin et al., 2024). While substantial RAG benchmarks (Asai et al., 2023; Yu et al., 2024; Kwiatkowski et al., 2019; Yang et al., 2018), relatively little attention has been given to the scientific domain. Although recent efforts, such as Xiong et al. (2024), begin to explore this direction in the medicine domain, the application of RAG to the chemistry domain remains underdeveloped. Notably, TextReact (Qian et al., 2023) applies text retrieval to tasks like reaction condition recommendation and one-step retrosynthesis. ChemLit-QU (Wellawatte et al., 2024) introduces a dataset for chemistry-oriented RAG, but its questions generated from isolated paper excerpts and may lack real-world utility. Importantly, there remains gap in the availability of high-quality, domain-specific corpora and comprehensive RAG benchmarks tailored to chemistry.
progress has been made in general-domat
2.2 Large Language Models for Chemistry
The rapid advancement of large language models (LLMMs) has opened up new opportuni- ties across various scientific domains (Ouyang et al., 2023), spurring the development of numerous benchmarks (Lu et al., 2022; Wang et al., 2023; Zhang et al., 2024). Among these domains, chemistry stands out particularly challenging yet promising area for LLM applications Fang et al. (2023). Recent efforts, such as ChemCrow (Bran et al., 2023), have demonstrated the potential of integrating LLMs with specialized tools to address wide range of downstream tasks. In addition, LLMs have been employed to improve performance on specific chemistry tasks, including reaction prediction (Zhong et al., 2023; 2024), drug discovery (Edwards et al., 2023), and SMILES recognition (Edwards et al., 2021). Despite growing interest, existing benchmarks often fall short in capturing the unique demands of the chemistry domain, which is inherently knowledge-intensivg tasks that frequently involve surface-level reasoning, chemistry requires precise retrieval and synthesis of complex, domain-specific knowledge. These characteristics make it compelling testbed for RAG, where the incorporation of external knowledge sources can substantially enhance LLM reasoning and decision-making
In contrast to general NLP
3
The CHEMRAG-BENCE Benchmark
3.1
Evaluation Settings
The primary goal of this work is to assess RAG systems in a setting that closely mirrors real-world information needs in the chemistry domain while remaining feasible and scalable
3

---

Published as a conference paper at COLM 2025
Dataset
Type
Task
Size
Avg. Length
MMLU-Chem
Multi-Choice
Chemistry Understanding
303
31
SciBench-Chem
Calculation
College-level Examination
229
94
Caption2Mol
100
Mol2Caption
100
Name Conversion
100
Product Prediction
100
ChemBench4K
Multi-Choie
72
RetroSynthesis
100
Solvent Prediction
100
Temperature Prediction
100
Yield Prediction
100
Desc.guuied Molecule Design
100
Forward Reaction Prediction
100
Molecular Desc. Generation
100
Mol-Instructions
Open-Ended
54
Property Prediction
100
Reagent Prediction
100
RetroSynthesis
100
Table 1: Statistics of CHEMRAG-BENCH, including question type, task type, data size, and the average length of each question.
in practice. To this end, the proposed CHEMRAG-BENCH benchmark is designed around four core evaluation scenarios:
Zero-Shot Learning: In real application, demonstrations are hard to find when conducting novel chemistry discovery. Therefore, we do not use any demonstration when evaluating the RAG systems.
Open-ended Evaluation: Most chemistry tasks are open-ended and not have answer options, including description-guide molecule design, retrosynthesis, and reagent pre- diction. To better align with chemists' needs, the RAG system should be evaluated in an open-ended setting. In this setting, no answer options will be provided.
Multi-Choice Evaluation: Multiple choice questions are common in LLM-related system evaluation. We adopt a multiple-choice setting to be consistent with previous work, and to make the evaluation more comprehensive. Many open-ended questions can be converted to multiple-choice questions by adding incorrect options.
Question-Only Retrieval: To mimic real-world usage, for multiple-choice questions, only the question used as the query for RAG.
3.2 Question Datasets
Our CHEMRA G-BENCH contains four datasets that cover wide range of chemistry tasks, including three multi-choice benchmarks, MMLU--hem (Hendrycks al., 2021), SciBench (Wang et al., 2024c), and ChemBench4K (Zhang et al., 2024), and one open-ended benchmark, Mol-Instructions (Fang et al., 2024). MMLU-Chem consists of college chemistry questions collected online. SciBench collects questions from chemistry textbooks. ChemBench4K contains multiple chemical analysis and prediction tasks, but in a multiple-choice fashion. Mol-Instructions is collection of molecule design, retrosynthesis, and prediction tasks. The statistics of the datasets are shown Table
For multi-choice questions, use accuracy as the metric. For open-ended ques- tions, the generated molecule is evaluated by exact match (EM), validity, MACCS FTS, RDK FTS, Morgan FTS, and BLEU. To evaluate the generated text, use BLEU and ROUGE For numerical results, we use accuracy with a 5% relative enror tolerance. Please refer to Appendix A for more details molecule evaluation metrics.
Metric
4

---

Published as a conference paper at COLM 2025
4 The CHEMRAG-TOOLKIT
CHEMRAG-TOOLKIT analyzes how RAG systems perform on CHEMRAG-BENCH, The CHEMRAG-TOOLKIT contains three major components: Corpora, Retrievers, and LLMs.
Corpora By working with researchers in chemistry, we construct a large and comprehen- sive collection of corpora covering wide range of chemistry information from six sources: 'PubChem for molecule information (English name, SMILES, IUPAC name, weight, mocular formula, and synonyms), ?PubMed for biochemistry abstracts, USPTO for chemical patents information, "Semantic Scholar for chemistry full-text papers, and *OpenStax for chemistry textbooks. Data processing is described in Appendix C, and the statistics of the corpora are shown in Table 2.
Retrievers In CHEMRAG- TOOLKIT, we select four representative retrievers for the retrieval process RAG: BM25
Corpus
Snippets
Avg. Length
Domain
PubChem
14.6M
72
Chemistry
PubMed
23.9M
305
Biochemistry
(Robertson & Zaragoza, 2009),
USPTO
143K
140
Chemistry
Contriever (Izacard et al., 2022),
Semantic Scholar
32.7M
403
Chemistry
SPECTER (Cohan et al., 2020),
OpenStax
5521
273
Chemistry
and e5 (Wang et al., 2024a).
Wikipedia
29.9M
163
General
In addition, we implement Reciprocal Rank Fusion (RRF, Cormack et al. (2009)) to combine the results from different retrievers.
Table 2: Statistics of corpora in CHEMRAG-TOOLXIT.
LLMs We choose a few representative LLMs be used in CHEMRAG-TOOLKE Llama- 3.1-8B-Instruct,
1in--311-7OB-Inttrre and Mistral-7B-IInsrrutta models, ChemLLM for chemistry open-source model, GPT-3.5-turbo and GPT-4o for closed- source models, Deepseek-Rl-nne R1-Llama-8B and ol for reasoning models.
for general open-source
Experiment Result
5.1 Comparison of Backbone LLMs
To systematically study how LLMs perform on chemistry tasks and how the proposed CHEMRAG-TOOLKTT affects models, we benchmark various LLMs CHEMRAG-BENCE with the same ChemRAG-Corpora. The top 5 documents retrieved by the RRF retriever are prepended to each question. The results are in Table 3, Appendix E, and Appendix F. More implementation details could be found in Appendix B.
As shown in Table 3, different models behave differently when CHEMRAG-TOOLKE is in use. On average, most models benefit from using CHEMRAG-TOOLKI,, Llama-3.1- 8B-Instruct gains 25.86%, hlama-3.-700-Insttra 36.9%, GPT-3.5-turbo gains 28.43% GP'T-4o gains 20.92%, and ol gains 16.38%. The largest improvement often comes from the one in Mol-Instruuttons and ChemBench4K. Among the backbone LLMs, ol achieves the highest performance in both baseline and RAG settings.
gains 24.5%, Mistral-7B-Instruct gains
Although most models benefit from CHEMRAG-TOOLKI,, the performance of ChemLLM decreases slightly (-12.6%) and Deepseek-R1-Llangg barely improves. They still gain some performance on certain question datasets. Both ChemLLM and BeepSeek-R1-L.ne from RAG on MMLU-Chem (+14.91% and +3.59%). DeepSeek-RI-LLane also performs slightly better on SciBench and Mol-Instructions with the proposed toolkit (+0.78 and notice that ReepSeek-R1-Llanee
benefit
+4.07). In our experiments, instructions and generates its answers in various forms, which poses difficulty in parsing its answers and may lead to poor performance in calculation.
does not follow our
We observe that larger models have consistent gains in -peeiic benchmarks (SciBench-Chem, Chem Bench4,, and Mol-Innttuctiins). This suggests that larger models

---

Published as a conference paper at COLM 2025
LLM
Method
MMLU
SciBench
ChemBenchaK
Mol-Instruct.
Avg.
Llama3.1 (8b)
Baseline
42.90
3.30
27.25
23.99
24.36
Ours
52.15
3.56
25.88
41.05
30.66
Llama3.1 (70b)
Baseline
62.38 61.05
5.99
24.25
28.33
30.24
Ours
13.63
26.25
49.67
37.65
Mistral (7b)
Baseline
45.21
2.09
12.63
4.66
16.15
Ours
42.57
11.13
34.73
22.11
LL (7b)
Baseline
37.62 43.23
8.72
23.5
17.74
21.90
Ours
2.03
16.75
14.56
19.14
Deepseek-rl
Baseline
55.44
3.09
35.38
3.75
24.42
-Slama(Sb)
Ours
57.43
3.87
29.13
7.82
24.56
Baseline
49.17
9.66
30.5
29.00
29.58
GPT3.5
Ours
52.81
8.80
44.5
45.83
37.99
Baseline
74.59
4.97
59.5
28.79
41.96
GPT-4o
Ours
73.92
8.59
67.25
53.18
50.74
Baseline
85.81 85.48
40.82
41.63
31.55
49.95
ol
Ours
43.61
58.38
45.04
58.13
Table 3: Benchmark results of different LLMs on CHEMRAG-BENCH.
have better understanding of the retrieved documents. In MMLU-Chem, most large models (Llama-3.1-70b, GPT-4o, and o1) do benefit from our toolkit. This may be because MMLU is common benchmark when evaluating LLMs, and these models are trained on related knowledge. The toolkit may not be able bring new knowledge to larger models. In SciBench-Chemi many models suffer from using the toolkit, this reflects that these models may not understand the retrieved documents well, since advanced models (Llama-3.1-70b, GPT-4o, and ol) all benefit from the toolkit, and ol even reaches the highest performance when using the toolkit. In ChemBench4K, similar patterns occur: smaller models have worse results, but larger models gain from the toolkit.
Since Mol-Instructions contains multiple sub-tasks, and each tab-tasks require multiple metrics, select description-guidee molecule design representative to analyze in detail how models perform after using our toolkit. The comparison is shown in Figure 2, with more details in the appendix. From Figure 2, observe that with our toolkit, all models improve in all aspects, except ChemLLM.
5.2 Comparison of Retrievers and Corpora
To understand the effect of each component in CHEMRAG-TOOLKIT we benchmark differ- ent retrievers with different corpora on CHEMRAGE BEBCHH. The experiments are conducted with GPT-3.5-turbo since is one of the models that benefit most from our toolkit, and it is also efficient and inexpensive for inference. The results in Table 4.
Comparison between Corpora From Table 4, we observe that the performance of RAG system is correlated to the selected corpus. The model performs the best with OpenStax MMLU-Chem and SciBench-Chem, but OpenStax barely has benefit for Mol-Instruucions, USPTO helps the model to achieve its best on ChemBench4K and Mol-Instructions, but it provides little benefit on MMLU-Chem and SciBench-Chen CHEMRAG Corpus, the model achieves the best on MMLU-Chem and ChemBench4KK surpassing leveraging only one corpus, which demonstrates the significance of combining multiple corpora. The CHEMRAG Corpus also helps the model perform better Mol- Instructions, only not as good as USPTO. Our corpus is also beneficial for SciBench when using Contriever as the retriever.
When using the combined
Comparison between Retrievers The Retriever plays another critical role as decides how the documents rank. From experiments shown in Table 4, all retrievers have their best

---

Published as a conference paper at COLM 2025
Seult
MACOO
RDX
XTK
BL.U
RDK
Vergar
Mergnn
Morgnn
Maggnn
[A)1 larma33 1-38
(i)
(e) Chemi IM-7B-Chat
(6) Mileraal-B-/rvtrrttt---0
MAGOS
CM
MACOS
AM
14000
Vargi
Megan
Vagan
Page
(6) Deepsookr1 (1,
S-Trboo-Tuubb
(g) GPT-4o
du
Model'
Models
Ours
Figure 2: Performance comparison on description-guide molecule design w.r.t evaluation metrics for molecule generation. Ours outperforms the baseline in almost all the scenarios.
11/13
$2.81
9.86
57
48.03
9.7
MMLU-Chem
(b) SciBench
30
42.5
44.5
44.63
43
45.83
46:31
47/12
16
30
29.0
25
10
(c) ChemBench4K
(d) Mol-Instructions
Figure 3: Performance comparison on different numbers of retrieved documents. The red dotted line represents the baseline. The experiments conducted on GPT-3.5-turbo.
performance on specific corpus and task. BM25 shows a very strong performance when using USPTO on Mol-Instructions, and using PubChem on SciBench-Chem. Contriever outperforms other retrievers when incorporating CHEMRAG Corpus on MMLU-Chem, also works well with PubChem and the CHEMRAG Corpus. SPECTER and e5 have mixed performances but still can excel in certain corpora. The RRF retriever, combining the results of the four retrievers, usually improves the performance, best, and sometimes results in the best performance. For instance, RRF helps the model achieve the best MMLU-Chen and ChemBench4K.
though it might not be the
Discussion and Analyses
6.1 Performance Scaling
The number of retrieved documents is an important factor in RAG systems. When k is too small, RAG systems may lack critical information; the other hand, when k is too large,
7

---

Published as a conference paper at COLM 2025
Chem- Bench4K
Mol- Instructions
Corpus
Retriever
MMLU
SciBench
Avg.
None
None
49.17
9.66
30.5
29.00
29.58
BM25
47.19
12.98
36.00
27.73
30.98
Contriever
48.18
10.02
39.50
29.72
31.86
PubChem
SPECTER
49.83
9.98
36.75
26.80
30.84
46.86
8.61
40.50
30.65
31.66
RRF
48.84
9.08
37.38
29.58
31 .2
BM25
46.86
12.02
38.63
28.14
31.41
Contriever
49.17
10.37
37.13
27.51
31.05
PubMed
SPECTER
47.19
9.08
36.63
27.69
30.15
e5
46.53
10.36
39.63
25.07
30.40
RRF
48.18
8.98
37.13
25.70
30.00
BM25
49.50
11.55
44.00
56.17
40.31
Contriever
49.50
10.92
42.25
37.40
35.02
USPTO
SPECTER
47.85
9.44
37.00
31.71
29.00
e5
47.52
11.05
38.13
37.55
33.56
RRF
49.17
10.70
43.38
56.68
39.98
BM25
45.54
7.18
37.25
29.72
29.92
Contriever
47.85
12.65
38.88
31.73
32.78
Semantic Scholar
SPECTER
49.17
10.45
37.00
26.44
30.77
e5
45.21
10.78
38.75
31.52
31.57
RRF
44.55
8.91
39.13
31.76
31.09
BM25
50.17
10.04
37.88
28.34
31.61
Contriever
49.50
12.66
36.38
27.95
31.62
OpenStax
SPECTER
50.50
11.88
37.13
28.20
31.93
49.83
11.57
38.5
29.96
32.47
RRF
52.48
11.55
37 37.25
29.35
32.66
BM25
49.17
8.06
38.75
27.67
30.91
Contriever
48.84
10.33
37.25
29.14
31.39
Wiki
SPECTER
47.52
9.21
39.25
27.22
30.8
e5
50.83
8.93
37.13
29.54
31.61
RRF
50.17
10.70
38.00
27.66
31.63
BM25
49.83
6.51
38.13
34.99
32.37
Chem -RAG
Contriever SPECTER
53.46
12.58
42.63
42.08
37,69
48.18
8.57
41.63
32.35
32.69
Corpus
e5
47.19
7.56
37.13
42.24
33.53
RRF
52.81
8.80
44.5
45.83
37.99
Table 4: Experiment results of various retrievers and corpora on CHEMRAG-BENT Com- pared with the baseline (first row), the intensity of the shade represents the magnitude of the decreases and increases
RAG systems may suffer from too much irrelevant information. To better understand how this factor affects RAG systems, we conduct experiments k 1,5, 10, 15. The results are shown in Figure 3. The phenomenon where performance first increases and then decreases as k increases clearly observed in MMLU-Chem and ChemBench4K. In SciBench-Chen the performance first decreases but then increases. This suggests that better retriever is needed or a reranker should be used. In our opinion, a better retriever should be developed since current retrievers only consider semantic similarity, however, semantic similarity may not be sufficient in reasoning tasks like SciBench. Ovérall, k a good choice since provides sufficient information in most cases.
6.2 Proportion in the CHEMRAG Corpus
We investigate the proportion of different sources used across various tasks. Figure 4 shows the proportions of six sources in CHEMRAG-Corpus and the actual proportions in the top 50 retrieved chunks in CHEMRAG-BENE A task-specific pattern of proportion is observed.
8

---

Published as a conference paper at COLM 2025
OpenStax has larger proportion in SciBench and relatively large proportion in MMLU- Chem. This is natural since the questions in SciBench and MMLU are derived from academic settings. PubChem has the largest proportion in both ChemBencháK and Mol-Instructions, which can be explained by the fact that these two tasks focus on molecule-related questions.
6.3 Retrievers in Chemistry
In our observation from Table 4 and Figure 3, we believe that a better retriever is needed for retrieving documents for chemistry downstream tasks. In Table 4, the model always performs better with USPTO and OpenStax corpora, but it performs worse the combined corpus, which suggests the retriever ranks the helpful snip- pets to lower place. This is also validated by the sudden rise in Figure (b).
Overall
In addition, chemistry retrieval faces 'mmultimodality" issue. One chemical compound may have multiple representations, in- cluding SMILES strings, IUPAC names, and English names, and each of them has variants.
ChemBench4K
MMLU-Chem
Mol-Instruct.
SciBench
Finally, current retrievers only con- sider keyword matching and se- mantic similarities, but chemistry tasks require more. More discus- sion and analysis are in Appendix D.
0.2
0.4
0.6
PubChem
PubMed
Textbook
USPTO
Wikipedia
Semantic Scholar
Figure 4: The overall corpus composition of CHEM- RAG corpora and the actually retrieved proportion in different tasks.
6.4 Practical Recommendations
Based experiments, provide some practical recommendations:
Corpus Selection The proposed CHEMRAG-Corpus is good start and is likely to outperform using only one corpus source. This is confirmed in Table 4, MMLU-Chem and ChemBench4K in particular. When working on molecule-related tasks, one may want to try USPTO since reaches high performance in both ChemBench4K and Mol-Instructions, As for questions in school, OpenStax (textbook) may be preferred, but the performance is still lower than using CHEMRAG-Corpus MMLU-Chem, illustrated in Table 4.
Retriever Selection Contriever is the most stable retriever in the four individual retrievers, but its performance still fluctuates across tasks and corpora. The proposed RRF retriever is recommended since it usually performs close the best individual retriever and sometimes outperforms them.
LLM Selection ol is the best model for all the tasks. Considering the cost and inference speed, GPT-335-turbo and GPTT-4o are good options. For open-source models, Llama-3.1 8B-Instruct is preferred since it achieves the second among the five open-source models and performs similar to the best model, LLamm-3.1170B-insttru performs 24% better, but with 775% more parameters and much higher computation cost.
Llama-3.1-70B only
7 Conclusion
We propose CHEMRAG-BENE and CHEMRAG-TOOLKIT to systematically evaluate RAG systems in chemistry, Based on our extensive experiments, we provide some novel findings, practical recommendations, and future directions for the community to better leverage RAG systems in chemistry in real world.

---

Published as a conference paper at COLM 2025
Acknowledgments
Research was supported in part by US DARPA INCAS Program No. HR0011-21-C0165 and BRIES Program No. HR0011-24-3-0325, National Science Foundation IIS-19-56151, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 2118329, and Apple PhD Fellowship. This research was also supported in part by the Division of Intramural Research (DIR), National Library of Medicine (NLM), National Institutes of Health (NIH). This advanced computing and data resource, a joint effort research has used the Urbana-Chan Shbamaa-hammpign and its National Center for Supercomputing of the University of Illinois Applications, supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. Any opinions, findings, and conclusions recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government.
References
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https: //arxiv.org/ herggabs/22111 http///1...11.1 11511.
Orlando Ayala and Patrice Bechard. Reducing hallucination in structured outputs via retrieval-augmented generation. In Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pp. 228-238. Association for Computational Linguistics, 2024. doi: 10.18653 naacl-industry. 19.
URL htttp://dx: doi org/10rrg110000 18653/v1/2024,
Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. Chemcrow: Aug- menting large-language models with chemistry tools. arXiv preprint arXit:2304.00376, 2023.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL, 2020.
Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '09, pp. 758-759, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 7781660588836. doi: 10.1145/157194111522111 URL https:://di..rr/// 1145/ 1571941. 15.2114.
Joseph L. Durant, Burton A. Leland, Douglas R. Henry, and James G. Nourse, Reoptimization of MDL keys for use in drug discovery. J. Chem. Inf. Comput. Sci., 42(5):1273-1288, 2002. 2002.
doi: 10.1021 CI010132R. URL https://ddo/ org/10. 1021 cci00013r.
Carl Edwards, Xiang Zhai, and Heng Ji. Text2Mol: Cross-modal molecule retrieval lanurala queries. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595-607, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021. emnlp-main.44
with natural language
emnlp-main.47 URL https://aclaantholog org/222l.emllpmmiin.
Carl N Edwards, Aakanksha Naik, Tushar Khot, Martin D Burke, Heng Ji, and Tom Hope. Synergpt: In-context learning for personalized drug synergy prediction and drug design. bioRxiv, pp. 2023-07, 2023.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmented large language
10

---

Published as a conference paper at COLM 2025
models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 6491-6501, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 041114//6667528337717777 URL https://doi.onnn 10. 1e11111136328....1 36771777..6777..
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: large-scale biomolecular instruction dataset for large language models. arXiv preprint arXit:2306.08016, 2023.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. ICLR. OpenReview.net, 2024. URL https://openreviin het/paffid-TTsdsbb6
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmennte generation for large language models: A survey. arXiv preprint arXi::2312.10997, 2, 2023.
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. Re2g: Retrieve, rerank, generate. arXiv preprint ar axXi::2207.06300 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arrxv.or///bb//220 orgg/bs//////// 03300.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- trastive learning. ar Xiv preprint arXio:2112.09118, 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- trastive learning, 2022. URL
091191...
Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context Iims meet rag: Overcoming challenges for long inputs in rag. The Thirteenth International Conference on Learning Representations, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training ilms to reason and leverage search engines with reinforcement learning, ar Xiv preprint ar arXiv:2503.09514 2025.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8:31, 2013.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensiee nlp tasks. Advances in Neural Information Processing Systemss, 33:9459-9474, 2020.
Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. J.EE Transactions on Knowledge and Data Engineering, 2024.
Yujian Li and Bi Liu. A normalized levenshtein distance metric. IEEE Trans. Pattern Anal. Mach. Intell., 29(6):1091-1999 2007. doi: aedoi.org/10.. 1ew./11099PPPP 2007. 1078.
URL https:
11

---

Published as a conference paper at COLM 2025
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- fion branches out, pp. 74-81, 2004.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task-oriented investigation of user-GPT interactions. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
www.:nne/fofumm// perrummii=qS111 ip2dGHo.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meefing the the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002. doi: 10.3115/ 10.3115/10730833 01555/007083.10331333 URL https::/aclaathhlleeeee -rg.
Yujie Qian, Zhening Li, Zhengkai Tu, Connor W. Coley, and Regina Barzilay. Predictive chemistry augmented with text retrieval, 2023. URL https://arxiv.... org abs/2312. 04881.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retriewal, 3(4):333-38) 389,, 2009. ISSN 1554-0669. doi: 0..1561/150000000 URL /ttt:://dx.doe org/ 15611100001111001199
Nadine Schneider, Roger A. Sayle, and Gregory A. Landrum. Get your atoms in order an open-source implementation of novel and robust molecular canonicalization algorithm. J. Chem. Inf. Model., 55(10):2111-2233 2015. doi: 10.101/AASJJCMM.5R0555 URL https://doi, org/ /021/acs. jeim. 5b00543.
Yaorui Shi, An Zhang, Enzhi Zhang, Zhiyuan Liu, and Xiang Wang. Relm: Leverag ing language models for enhanced chemical reaction prediction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguis- fics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 3506-5520 Association for Com- putational Linguistics, 2023. https://doi, org/ro.g////o 10. 18653/v1/2023 2f65:///2223...nnnng-eeeee www.iiinssnnnnnp...... 366.
2006663/VV/022./IIN/2 WWW.IINN---EMNNL..66. URL
Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval aug- mented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics, 11:1-17, 2023.
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchun- shu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein. Chemagent: Self-updating memories in large language models improves chemical rea- soning. In The Thirteenth International Conference on Learning Representations, 2025, URL https://openrevin/ nwwtforumrnn.ee FirFrrrmmii-kkuleeegga
Taffee T Tanimoto. Elementary mathematical theory of classification and prediction. 1958.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. Xiv preprint ar arXiv:2212.10509 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly- contrastive pre-training, 2024a. URL http:::///axiv.or//bbsssss
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowl- edge editing for large language models: A survey. ACM Computing Surveys, -7(3) 2024b.
12

---

Published as a conference paper at COLM 2025
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint ar 7xXt:2307.10635, 2023.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Ar- jun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. In Proceed- ings of the Forty-First International Conference on Machine Learning, 2024c
Geemi Wellawatte, Huixuan Guo, Magdalena Lederbauer, Anna Borisova, Matthew Hart, Marta Brucka, and Philippe Schwaller. Chemlit-QA: A human evaluated dataset for chemistry RAG tasks. In Al Al for Accelerated Materials Design NeurlPS 2024, 2024. URL
https://openreviier
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval- augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024, pp. 6233-6251, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdi- nov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint ar arXio:1809.09000 2018.
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval-augmented
generation for large language models, 2024. URL https::/arxxwwoor//bbb 194433
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, Shufei Zhang, Mao Su, Hansen Zhong, Yuqiang Li, and Wanli Ouyang. Chemllm: A chemical large language model, 2024.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's song in the Al ocean: A survey on hallucination in large language models. CoRR, abs 2309..220000220 2023a. doi: 10.48550/ 10.4850/ARXIV.2000 ARXIV.23090012 012199. URL https://doi. org/10 10. 48550/ aaw.v./aiiii.a////i/ 200..3399.01100
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ocean: a survey hallucination in large language models. ar Xiv preprint ar Kio:2309.01219, 2023b.
Ming Zhong, Siru Ouyang, Minhao Jiang, Vivian Hu, Yizhu Jiao, Xuan Wang, and Jiawei Han. ReactlE: Enhancing chemical reaction extraction with weak supervision. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 12120-12130, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10.18653/ /v1 2021/2222222 findings--ccl.767. URL https://aclanttol..0g org/2023. findings-acl. 767.
Xianrui Zhong, Yufeng Du, Siru Ouyang, Ming Zhong, Tingfeng Luo, Qirong Ho, Hao Peng, Heng Ji, and Jiawei Han. Actionie: Action extraction from scientific literature with programming languages. In the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, August 2024.
13

---

Published as a conference paper at COLM 2025
A Evaluation Metrics for Molecules
To assess the quality of generated molecules, we first employ general text-based generation metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), which compare generated outputs against reference answers.
For molecular generation, we begin by verifying the validity of generated molecules using RDKit (Landrum et al., 2013) and then compute their exact match with reference solutions. However, single textual description can correspond to multiple molecular structures, making exact matching a limited evaluation criterion. Moreover, expecting an L.M-even one fine-tuned with LoRA on specific instructions to consistently generate outputs that perfectly match reference molecules is often unrealistic.
To address these challenges and provide more comprehensive evaluation, we incorporate molecular similarity metrics, including similarity scores based on RDKit, MACCS, and Morgan fingerprints (Tanimoto, 1958; Schneider et al., 2015; Durant al., 2002), alongside Levenshtein (Li & Liu, 2007) and BLEU scores.
For tasks that need compare numbers, following previous work (Wang al., 2024c), we compare the generated output with the ground truth, allowing a 5% relative error. This makes sure that the score within 0 and 1, making more suitable for combining with other scores. These results can be found in Appendix F.
B Implementation Details
Since ChepSeek-Rl-Llame and oll are reasoning models, following their guidelines, we set 0.6 and their temperatures respectively. The temperatures for other models are set to 0 for reproducibility. For each experiment, we run three rounds and report the mean for DeepSeek-R1-Llanee and ol models. We one round for other models. The number of max generation tokens is set to 10,000 for DeepSeek- -a-eeek--1-Llmaa and ol since their reasoning requires more tokens, and the numbers for other models are 512.
C Corpus Data Processing
PubChem For each chemical compound in PubChem, we collect its names, properties, and description and transform them into JSON format.
Semantic Scholar We collect 1,849, 95,9 full-text chemistry papers from Semantic Scholar, then chunk them into chunks of 512 tokens with 50-token overlap.
USPTO USPTO contains a large amount of information about reactions: reactants, reagents, products, and yields. We transform these attributes into JSON format.
We parse the PDF textbooks to texts with Mathpix and then split the parsed textbook into chunks of 512 tokens with 50-token overlap.
OpenStax
PubMed and Wikipedia
The data from Xiong al. (2024).
D Retrieval Error Case Study
We present two representative errors in chemistry retrieval observed in our experiments.
D.1 Prioritize too much molecule matching
For instance, the query "Which ingredients are commonly selected for creating Ccloc(- @ZccchaeCCocccccccccss40000000Cn is asking about reagents information
"nttps://mathpixx.com/
14

---

Published as a conference paper at COLM 2025
Model Size
Method
MMLU
SciBench
ChemBench<K
Mol-Instruct.
Avg.
Bascline
42.90
3.30
27.25
23.99
24.36
8B
CoT
61.38
1.92
32.75
6.67
25.68
Ours
52.15
3.56
25.88
41.05
30.66
Baseline
62.38
5.99
24.25
28.33
30.24
70B
Coll
67.66
4.36
32.13
25.97
32.53
Ours
61.05
13.63
26.25
49.67
37.65
Table 5: Comparison with Chain-of-Thought (CoT) on Llama-3.1.
for generating the mentioned compound as a product. When there is no such information, retrievers usually give high scores to irrelevant documents that contain the same SMILES. This may introduce noise to the retrieved documents and mislead LLMs. A better retrieval system may identify this situation, then search for the synonyms, and search for similar compounds if synonyms still fail.
D.2 Often fail when the document only mention one name
molecule may have many names, including SMILES, IUPAC, and English names. This makes retrieving the right document more difficult as the question may only contain English names, but there may only be SMILES in the relevant documents. For example,
Query: What is the molecular weight of aspirin?
Documentl: The molecular weight of CHВCOОс6НĄCOOH 180.16 g/mol.
Document2: Aspirin can cause developmental toxicity.
In this example, CHВCOOс6Н4COн is the formula of aspirin, but current retrievers don't give Documentl high scores because it doesn't know СИЗСOОС6НИСоOь is aspirin. By training with synonyms, the texts with CИвсOосьыІсOо and aspirin will be closer in the embedding space. Alternatively, when using LLM to generate queries, the LLM may first find the formula of aspirin, and then use the formula to search for the molecular weight.
E
More Experiment Results
E.1
Chain-of-Thought Comparison
We deploy Chain-of-Thought (CoT) on Llama3.1-8D and Llama3,1-70B, the results are in Table 5. CoT does better than baseline and solely using RAG in multiple-choice settings (MMLU and ChemBench4K), but performs worse than baseline and solely using RAG in open QA (SciBench and Mol-Instructions).
Detailed Experiment Results
E1
SciBench-Chemistry
Table shows the performances of different models on SciBench-Chemistyy tasks.
E2
ChemBench4K
Table 7 and Table 8 demonstrate the performances of different models on ChemBench4K tasks.
E3
Mol-Instructions
Table 9, Table 10, Table 11, and Table 12 demonstrate results in Mol-Instructions.
15

---

Published as a conference paper at COLM 2025
LLM
Method
SciBench
Avg.
atkins
chemmec
matter
quan
Llama3.1 (8b)
Baseline
0
10.26
0
2.94
3.30
Ours
3.74
2.56
2.04
5.88
3.56
Llama3.1 (70b)
Baseline
2.56
5.99
0
17.65
5.99
Ours
6.54
15.38
6.12
26.47
13.63
Mistral (7b)
Baseline
3.74
2.56
2.04
2.09
Ours
0
0
LM (7b)
Baseline
11.21
12.82
2.04
8.82
8.72
Ours
0.93
5.13
2.04
2.03
Deepseek-rl
Baseline
4.67
7.69
0
3.09
-llama(8b)
Ours
2.80
7.69
2.04
2.94
3.87
Baseline
5.61
23.08
4.08
5.88
9.66
GPT3.5
Ours
5.61
20.51
6.12
8.80
Baseline Ours
3.74
10.26
0
5.88
4.97
GPT-4o
10.28
10.26
2.04
11.76
8.59
Baseline 38.32
46.15
34.69
44.12
40.82
o1
Ours 44.86
48.72
36.73
44.12
43.61
Table 6: Detailed benchmark results of different LLMs on SciBench-Chemistry. The accuracy is computed by comparing the generated answer with the ground truth, allowing 5% relative error.
LLM
Method
ChemBenchaK
Name Conversion
Product Prediction
Caption2Mol
Mol2Caption
Llama3.1 (8b)
Baseline
0
88
57
13
Ours
7
70
59
15
1 (70b)
Baseline
3
86
68
0
Ours
5
87
64
11
Mistral (7b)
Baseline
3
26
40
15
Ours
7
19
33
11
CM (7b)
Baseline
24
46
48
2
Ours
21
33
38
0
Deepseek-rl -llama(8b)
Baseline Ours
15
81
70
19
18
68
65
12
Baseline Ours
20
89
48
17
GPT3.5
36
87
60
39
Baseline
41
98
79
93
GPT-4o
Ours
61
98
81
83
Baseline Ours
6
99
76
26
o1
27
96
80
59
Table 7: Detailed benchmark results of different LLMs on ChemBench4K, Part 1.
G
Prompt
The prompts used in our experiments can be found in Table 13, 14, 15, 16, 17, 18, 19, 20.
16

---

Published as a conference paper at COLM 2025
LLM
Method
ChemBench4K
Solvent
Temp. Prediction
Yield
Retrosynthesis
Prediction
Prediction
Llama3.1 (8b)
Baseline
0
21
17
22
Ours
6
20
15
15
1 (70b)
Baseline
0
24
9
Ours
2
22
1
18
Mistral (7b)
Baseline
0
9
Ours
2
8
0
9
LM (7b)
Baseline
25
21
21
Ours
0
28
4
10
Deepseek-rl
Baseline
14
36
31
17
-llama(Bb)
Ours
3
33
17
17
Baseline
23
18
25
GIT3.5
Ours
26
41
28
39
Baseline
54
35
33
43
GPT-4o
Ours
76
49
43
47
Baseline Ours
5
42
48
31
o1
50
50
63
42
Table 8: Detailed benchmark results of different LLMs on ChemBench4K, Part 2.
LLM
Method
Description-Guided Molecule Deisgn
MACCS FISt
RDK FISt
Morgan FISt
EM1
Validity
BLEU
Llama3.1 (8b)
Baseline
73
35.71
25.01
13.37
6.02
Ours
60.78
48.85
40.64
10.92
1lama3.1 (70b)
Baseline
95
32.61
21.67
16.72
18.34
Ours
60.35
49.65
40.89
31.56
Mistral (7b)
Baseline
32.74
20.66
10.34
3.61
Ours
68.14
53.26
47.52
10.35
ChemLLM (7b)
Baseline
26.40
10.70
9.41
5.41
Ours
10.48
6.59
5.10
0
Deepseek-rl -lbama(Bb)
Baseline Ours
3.70
0
26,25
Baseline Ours
45.53
26.48
18.08
9.41
GP7
12
95
92.28
49.35
40.45
30.56
Baseline Ours
47.33
28.88
20.32
11.88
GPT--o
14
96
60.84
49.47
42.45
27.92
Baseline
40.12
25.72
17.55
ol
Ours
12
97
57.59
46.01
39.78
Table 9: Detailed benchmark results of different LLMs on Mol-Instruccions Description- guided molecule design.
17

---

Published as a conference paper at COLM 2025
LLM
Method
Forward Reaction Prediction
MACCS FTSt
RDK FISt
Morgan FISt
EM1
Validityt
BLEU
Llama3.1 (8b)
Baseline
60.24
43.19
6.63
Ours
63.29
53.61
45.46
29.50
1lama3.1 (7Ub)
Baseline
63.74
60.68
44.94
17.47
Ours
22
91
72.70
62.74
57.14
43.89
Mistral (7b)
Baseline
0
2
Ours
29
76.68
77.54
63.33
11.31
Ch (7b)
Baseline
45.88
34.61
28.15
3.18
Ours
57
22.55
17.55
12.68
0
Deepseek-rl (8bama(B)
Baseline
1.56
0.39
0.65
12.98
Ours
59
0
1.77
Baseline Ours
58.37
52.03
40.63
23.43
GIT3..5
16
96
72.11
67.80
56.21
39.43
Baseline Ours
96
66.35
62.6
50.84
50.7
GPT-4o
26
89
78.31
73.88
68.3
61.44
Baseline
13
81,95
78.95 82.89
72.06
o1
Ours
30
90
87.35
78.92
Table 10: Detailed benchmark results of different LLMs on Mol-Instruction Forward Reaction Prediction.
LLM
Method
Molecule Description Generation
BLEU
Rouge-L
Llama3.1 (8b)
Baseline
0
8.98
Ours
8.24
32.79
Llama3.1 (70b)
Baseline
0.83
15.25
Ours
4.30
27.6
Mistral (7b)
Baseline Ours
0.63
18.64
4.48
32.09
ChemLLM (7b)
Baseline
5.33
34.04
Ours
0
Deepseek-rl -Sbama(B))
Baseline Ours
0
0
Baseline
3.18
20.58
GPT3.5
Ours
3.75
21.51
Baseline Ours
1.23
18.25
GPT--4o
2.98
30.06
Baseline Ours
0
o1
1.02
14.44
Table 11: Detailed benchmark results of different LLMs on Mol-Instructions Moleclue Description Generation.
18

---

Published as a conference paper at COLM 2025
LLM
Method
Property Prediction
Accuracy
Llama3.1 (8b)
Baseline
0.14
Ours
Llama3.1 (70b)
Baseline
Ours
Mistral (7b)
Baseline
Ours
ChemLLM (7b)
Baseline
Ours
15
Deepseek-rl -Ilama(8b)
Baseline
Ours
Baseline
GPT3.5
Ours
Baseline
GPT-4o
Ours
Baseline
ol
Ours
Table 12: Detailed benchmark results of different LLMs on Mol-Instructions - Property Prediction. The accuracy is computed by comparing the generated answer with the ground truth, allowing a 5% relative error.
Table 13: Baseline prompt template for general open-ended questions.
Open-ended Baseline Prompt
Answer the question directly. Only give me the answer and do not output any other words. Question: Instruction }
Answer:
Table 14: Multi-choice baseline prompt template for general open-ended questions.
Multi-choice Baseline Prompt
Answer the question directly.
Only give me the answer and do not output any other words.
Question: { Instruction }
Choices: Choices)
Make prediction from the given choices.
Answer:
Table 15: Numerical baseline prompt template for general open-ended questions.
Numerical Baseline Prompt
Answer the question directly. Conclude the answer by stating "The answer is therefore [ANSWER]" Only give me the answer and do not output any other words.
Question: (Instruction) }
Answer:
19

---

Published as a conference paper at COLM 2025
Table 16: Generation baseline prompt template for general open-ended questions.
Generation Baseline Prompt
Answer the question directly.
Your answer should be surrounded by [ANSWER] and |/ I/ANSWERJ. When generating molecule, please generate valid SMILES string. Only give me the answer and do not output any other words.
Question: { Instruction Answer:
Table 17: RAG prompt template for general open-ended questions.
Open-ended RAG Prompt
Answer the question based on the given document.
Only give me the answer and do not output any other words. The following are given documents. reference
Question: Instruction) }
Answer:
Table 18: RAG Prompt template for multiple-choice questions.
Multi-choice RAG Prompt
Answer the question based on the given document.
Only give me the answer and do not output any other words. The following are given documents.
reference
Question:
Instruction }
Choices:
{ Choices) }
Make prediction from the given choices.
Answer:
Table 19: Prompt template for numerical questions.
Numerical RAG Prompt
Answer the question based the given document. Conclude the answer by stating "The answer therefore [ANSWER]"
Only give me the answer and do not output any other words. The following are given documents. referrecee
Question: Instruction }
Answer:
20

---

Published as a conference paper at COLM 2025
Table 20: Prompt template for generation questions.
Generation RAG Prompt
Answer the question based on the given document.
Your answer should be surrounded by [ANSWER] and I/ANSWERJ. When generating molecule, please generate a valid SMILES string. The following are given documents.
reference
Only give me the answer and do not output any other words.
Question:
{
Instruction }
Answer:
21


Reviews:
[{"review_id": "eHT7tp3e2E", "reviewer": "Reviewer_vCXR", "summary": "This paper collect a benchmark by selecting useful data from existing benchmarks, build a RAG toolkit using an ensemble of existing retriever & reranker, and conduct lots of experiments on analyzing the effect of corpus to retrieve, retriever selection, and task difference.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Lots of experiments are conducted, and very reasonable analysis on RAG's effect on many meaningful aspects.\n2. RAG's effect indeed need systematic research: this work focus on an interesting and novel research question.", "reasons_to_reject": "1. The contribution seems a bit limited: the constructed benchmark seems is collected from existing benchmarks, and the developed toolkit is an ensemble of existing methods.\n2. It might lack of some insights on how to build a better retriever from the analysis from this paper. Suggestion: maybe build one to verfiy the findings, then it would probably be a (much) more complete submission.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yArFjLm5nu", "reviewer": "Reviewer_JvHc", "summary": "This paper presents benchmarking LLMs with RAG for chemistry tasks. It covers six task categories, six heterogeneous corpora, five off-the-shelf retrievers, and eight LLMs. Through systematic experiments, the authors demonstrate an average 17.4% relative improvement over direct inference, analyze the impact of corpus selection, retriever choice, and retrieval depth.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. RAG for scientific domains is an important and understudied direction.\n\n2. It is a well-organized empirical study on RAG-enabled LLMs on diverse chemistry tasks.", "reasons_to_reject": "1. No new retrieval algorithms or corpora are proposed--both the “benchmark” and “toolkit” are essentially assemblies of existing resources.\n\n2. Shallow findings and analysis. Specifically, the key observations (e.g., corpus choice impacts task performance; retrieval depth influences results) are expected for any RAG system. The paper does not provide deeper analysis such as why certain retrievers outperform others, or how often retrievers actually surface truly relevant chemistry information, or what the common failure modes are.\n\n3. It is not chemistry-specific enough, other than the used datasets. It does not explore the unique characteristics and challenges of chemistry-specific tasks, and does not develop any domain-specific methodologies to address the challenges.\n\n4. The comparative evaluation is not very comprehensive. It does not compare with task SoTA models, as well as existing chemistry agents (such as ChemAgent, ChemCrow, or ChemToolAgent), many of which are similar to RAG in the way that they use tools to obtain domain information and has improved the performance over base LLMs. Without this, it's hard to know how much the RAG system can help. In addition, the experiments omit test-time techniques such as chain-of-thought prompting, which could significantly affect performance.", "questions_to_authors": "**Question**:\n1. You describe your benchmark as “expert-curated.” Could you clarify who the experts were and what their role was in curating the data?\n\n**Suggestion**:\n1. Consider including the names of the LLMs and retrievers used directly in the table and figure captions to improve clarity and readability.", "extra_scores": {}}, {"review_id": "nSKdlVe4Zh", "reviewer": "Reviewer_cNEq", "summary": "The paper explores RAG for domain-specific search, more specifically chemistry-related tasks. Tapping into previously released benchmark datasets the paper brings together a benchmark (ChemRAG-Bench) that combines four existing datasets. In addition to that the authors present a toolkit (ChemRAG-Toolkit) which allows the exploration of RAG performance on the benchmarks by modifying different experimental settings such as the retrieval algorithm and underlying language model. Experimental results based on deploying the toolkit are being reported as well. \n\nTopically this a very good fit for COLM. The paper also provides interesting insights. However, there are a number of concerns that limit the overall impact of the contribution as described below.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper provides a strong motivation and a domain-specific scenario that helps get a better understanding of how large language models affect different downstream use cases. \n* The authors make a practical contribution that may be picked up as a reference point by others in the same research field. \n* The bibliography is comprehensive referencing many papers from key research outlets. There is however a bit of a skew towards papers from the non-peer-reviewed literature.\n* Good use is made of appendices. To further support reproducibility however it would have been beneficial to also include an anonymized GitHub repository.", "reasons_to_reject": "I see a number of shortcomings affecting the overall value of the contribution:\n\n* The work appears a bit incremental as the “comprehensive benchmark” collection the authors present seems to be a compilation of benchmarks published previously (as a side note here: please address any possible copyright issues that might arise when re-distributing such resources). I might be mistaken here but that is how I interpreted the paper. \n* While each dataset looks like a valuable resource it remains unclear how representative the chosen data actually is for a specific practical setting. \n* Another shortcoming I see is the experimental work which does not make use of suitable baselines. More specifically, the results in Table 3 would be a lot stronger if state-of-the-art baselines were to be deployed to demonstrate what a RAG approach offers over alternatives. Such baselines could be implementations of what has been reported in the literature as top-performing approaches for each of the datasets, possibly also a fine-tuned BERT system or something similar as long as non-contamination is being guaranteed (see also comment further down). \n* The discussion of related work does not address the specific challenges (and the possible solutions put forward) that generative approaches such as RAG pose. The evaluation paradigm RAGAs is one such approach [1].\n* No statistical significance testing has been applied to the experimental results (despite the use of terms such as “outperform”). As an example, the results in the last column of Table 4 look more or less on par and this suggests that none of the chosen settings might be any better or worse than any of the others. \n* Given the datasets have been published previously (one of them several years ago) it is reasonable to assume that LLMs will have seen them during training. This means that they may well be contaminated and the results obtained from testing on what has been used as training data will be potentially flawed.\n* There is no limitations section which should critically assess how generalisable the contribution is. \n\n\nMinor points: \n\n* Make all captions self-contained (e.g. what metrics are being used in Table 3?)\n* The bibliographical entries need to be polished (e.g.  incorrect capitalisation, missing details etc.). \n\n\n[1] Es et al. (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. EACL 2024.", "questions_to_authors": "* “We collect data from six sources” —> How exactly is this being done? Are these all results from each of these sources or just a sample (if so, how was this sampled)? \n* I am puzzled by the statement  that “current retrievers only consider semantic similarity”. I think it is more common to simply use bag-of-words approaches (e.g. BM25) as a first step in RAG rather than actually utilizing semantics. This statement therefore needs to be backed up by suitable references.", "extra_scores": {}}, {"review_id": "vOxG5PSvJs", "reviewer": "Reviewer_YkjK", "summary": "The authors take 4 datasets and 6 corporate in the chemistry domain and see how RAG with various retrievers works. There are ablations and analyses.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "CHEMRAG-BENCH and CHEMRAG-TOOLKIT are clearly useful contributions on which to evaluate and improve models.", "reasons_to_reject": "I don't have major reason to reject. Some questions below.", "questions_to_authors": "- Why use SPECTER? SPECTER2 is better and has adapters for various tasks. Also, SPECTER was made to encode titles + abstracts specifically. Many of your datasets are *not* in that format, so it wouldn't make sense to use SPECTER for those datasets.\n\n- For the original 4 datasets (MMLU-Chem etc) -> what was the retrieval intention of the authors? Did these datasets come with corpora? It's unclear how your corpora relate to the original datasets, their construction, etc.\n\n- There's no mention of code release or how to use the benchmark/toolkit. Given the framing, I assume you'll release the code. It would be very nice to have examples of how to run it. Is the benchmark easy to run with another LLM? How easy is it to add another retriever? What about another corpus?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary

This paper presents ChemRAG-Bench, a new benchmark designed to systematically assess RAG for chemistry tasks. It also provides ChemRAG-Toolkit, a modular framework that supports multiple retrievers and LLMs for controlled experimentation. The benchmark integrates various chemistry-specific corpora and evaluates across a variety of tasks. The paper argues that this is the first large-scale benchmark dedicated to RAG in chemistry.
 
Strengths:

1. Reviewers in general agree that this is the first comprehensive benchmarking effort for RAG applied to chemistry, a highly specialized and underexplored domain for LLMs. General-purpose LLMs struggle in chemistry without retrieval, and this work helps fill the gap. 
 
2. ChemRAG-Bench and ChemRAG-Toolkit can be useful to the community (Reviewer YkjK). Lots of experiments are conducted on meaningful aspects, they are well organized, and analysis is reasonable (Reviewer JvHc, vCXR). 

Weaknesses:

1. One weakness mentioned by most reviewers (Reviewer vCXR, JvHc, cNEq) is that the paper mostly combines existing datasets, models, and retrievers. No new retrieval methods are proposed, and the toolkit primarily assembles existing components. Relatedly, the paper does not discuss in depth about chemistry-specific modeling or retrieval challenges, or propose new retrieval or generation methods adapted to the unique challenges of chemistry domain.

2. Reviewer cNEq mentioned about data contamination and copyright issues (very important). The authors should add some discussions on limitations and ethics to clarify them. For example, it might be OK to acknowledge there is a risk of data contamination, but given the low performance in general, the issue might not be as serious as one would imagine.  
 
During the author response, authors added new experiments (e.g., chain-of-thought, ChemCrow agent comparisons), released code, improved analysis, and clarified the corpora as well as the engagement with domain experts, which addressed many concerns.
 
I think the creation of a chemistry-specific RAG benchmark and toolkit fills an important gap and is likely to benefit both the AI-for-chemistry and RAG research communities. Therefore, I would recommend to accept the paper, provided that the authors add the new results and more discussions about current methods’ failure modes and possible future work as well as the missed references mentioned during the response period (such as ChemCrow and ChemToolAgent).
Decision: Accept