Benchmarking Retrieval-Augmented Generation for Medicine (2024)

Abstract: While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrievalaugmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MEDRAG toolkit introduced in this work. Overall, MEDRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the “lostin-the-middle” effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine1.

heading: 1 Introduction
text: Large Language Models (LLMs) have revolutionized the way people seek information online, from searching to directly asking chatbots for answers. Although recent studies have shown their state-ofthe-art capabilities of question answering (QA) in
†Equal contribution. §Co-correspondence. 1Project homepage: https://teddy-xionggz.github.
io/benchmark-medical-rag/
both general and medical domains (OpenAI et al., 2023; Anil et al., 2023; Touvron et al., 2023b; Singhal et al., 2023a; Nori et al., 2023a), LLMs often generate plausible-sounding but factually incorrect responses, commonly known as hallucination (Ji et al., 2023). Also, the training corpora of LLMs might not include the latest knowledge, such as recent updates of clinical guidelines. These issues can be especially dangerous in high-stakes domains such as healthcare (Tian et al., 2024; Hersh, 2024), and will affect the overall performance of LLMs on domain-specific QA tasks.
By providing LLMs with relevant documents retrieved from up-to-date and trustworthy collections, Retrieval-Augmented Generation (RAG) has the potential to address the above challenges (Lewis et al., 2020; Gao et al., 2023). RAG also improves the transparency of LLMs by grounding their reasoning on the retrieved documents. As such, RAG has already been quickly implemented in various scientific and clinical QA systems (Lála et al., 2023; Zakka et al., 2024). However, a complete RAG system contains several flexible modules, such as document collections (corpora), retrieval algorithms (retrievers), and backbone LLMs, but the best practices for tuning these components are still unclear, hindering their optimal adoption in medicine.
To systematically evaluate how different components in RAG affect its performance, we first compile an evaluation benchmark termed MIRAGE, representing Medical Information RetrievalAugmented Generation Evaluation. MIRAGE includes 7,663 questions from five commonly used QA datasets in biomedicine. To evaluate RAG in realistic medical settings, MIRAGE focuses on the zero-shot ability in RAG systems where no demonstrations are provided. We also employ a questiononly setting for the retrieval phase of RAG, as in real-world cases where no options are given. For a comprehensive comparison on MIRAGE, we provide MEDRAG, an easy-to-use toolkit that covers
6233
five corpora, four retrievers, and six LLMs including both general and domain-specific models.
Based on the MIRAGE benchmark, we systematically evaluated different MEDRAG solutions and studied the effects of each component on overall performance from a multidimensional perspective. For various LLMs, there is a 1% to 18% relative performance increase using MEDRAG compared to chain-of-thought prompting (Wei et al., 2022). Notably, with MEDRAG, GPT-3.5 and Mixtral (Jiang et al., 2024) can achieve comparable performance to GPT-4 (OpenAI et al., 2023) on MIRAGE. On the corpus dimension, we found different tasks have a preference over the retrieval corpus. While pointof-care articles and textbooks are solely helpful for examination questions, PubMed is a robust choice for all MIRAGE tasks. Our results also show that a combination of all corpora can be a more comprehensive choice. On the retriever dimension, BM25 (Robertson et al., 2009) and the domain-specific MedCPT (Jin et al., 2023a) retriever display superior performance on our MIRAGE benchmark. The performance can be further enhanced by combining multiple retrievers. Beyond the evaluation results on MIRAGE, we found a log-linear scaling relationship between model performance and the number of retrieved snippets. We also observed a “lost-in-the-middle” phenomenon (Liu et al., 2023) between model performance and the position of the ground-truth snippet. Finally, we provide several practical recommendations based on the results and analyses, which can guide the application and future research of RAG in the biomedical domain.
In summary, our contributions are three-fold:
• We introduce the MIRAGE2, a first-of-its-kind benchmark for systematically comparing different medical RAG systems.
• We provide MEDRAG3, a RAG toolkit for medical QA that incorporates various domainspecific corpora, retrievers, and LLMs. MEDRAG significantly improves the performance of LLMs on MIRAGE.
• We recommend a set of best practices for research and deployments of medical RAG systems based on our comprehensive results and analyses on MIRAGE with MEDRAG.
2https://github.com/Teddy-XiongGZ/MIRAGE 3https://github.com/Teddy-XiongGZ/MedRAG

heading: 2 Related Work

heading: 2.1 Retrieval-augmented Generation
text: Retrieval-Augmented Generation (RAG) was proposed by Lewis et al. (2020) to enhance the generation performance on knowledge-intensive tasks by integrating retrieved relevant information. RAG not only mitigates the problem of hallucinations as LLMs are grounded on given contexts, but can also provide up-to-date knowledge that might not be encoded by the LLMs. Many follow-up studies have been carried out to improve over the vanilla RAG (Borgeaud et al., 2022; Ram et al., 2023; Gao et al., 2023; Jiang et al., 2023; Mialon et al., 2023).
In biomedicine, there have also been various explorations on how LLMs can improve literature information-seeking and clinical decision-making with RAG (Frisoni et al., 2022; Naik et al., 2022; Jin et al., 2023b; Lála et al., 2023; Zakka et al., 2024; Jeong et al., 2024; Wang et al., 2023b), but their evaluations are not comprehensive. Nevertheless, current systematic evaluations in biomedicine typically focus on the vanilla LLMs without RAG (Chen et al., 2023a; Nori et al., 2023a). Our study provides the first systematic evaluations of RAG systems in medicine.

heading: 2.2 Biomedical Question Answering
text: Biomedical or medical question answering (QA) is a widely studied task since various information needs are expressed by natural language questions in biomedicine (Zweigenbaum, 2003; Athenikos and Han, 2010; Jin et al., 2022). While BERTbased (Devlin et al., 2019) models used to be the state-of-the-art methods of medical QA (Abacha et al., 2019; Lee et al., 2020; Soni and Roberts, 2020; Gu et al., 2021; Yasunaga et al., 2022), they are outperformed by LLMs with large margins (Singhal et al., 2023b; Chen et al., 2023b; Nori et al., 2023b). Due to their knowledge-intensive nature, QA datasets are commonly used to evaluate the biomedical capabilities of both general LLMs (Nori et al., 2023a,b) and domain-specific LLMs (Luo et al., 2022; Chen et al., 2023b; Wu et al., 2023; Singhal et al., 2023a,b). Following these studies, we also use medical QA datasets to test if a RAG system can retrieve and leverage relevant contexts. Unlike prior efforts, our evaluation employs both RAG and question-only retrieval settings, a more realistic evaluation for medical QA.

heading: 3 The MIRAGE Benchmark

heading: 3.1 Evaluation Settings
text: The main objective of this work is to evaluate RAG systems in a setting that reflects real-world medical information needs as much as possible while being practically scalable. As such, our MIRAGE benchmark adopts four key evaluation settings:
Zero-Shot Learning (ZSL). As real-world medical questions are often posed without similar exemplars available, in our benchmark, the RAG systems should be evaluated in a zero-shot setting where in-context few-shot learning is not permitted.
Multi-Choice Evaluation (MCE). Evaluating medical QA systems using multi-choice questions is a widely adopted method that can be practically implemented for large-scale evaluation (Nori et al., 2023a,b; Singhal et al., 2023a; Liévin et al., 2022; Lála et al., 2023). To be consistent with existing research, we also use a multi-choice setting in our benchmark to compare different systems.
Retrieval-Augmented Generation (RAG). The medical questions used in MIRAGE are knowledgeintensive, which are difficult to answer without external knowledge. Moreover, due to the problem of hallucination, letting LLMs be reasoning engines instead of knowledge databases could be a better practice in medicine (Truhn et al., 2023). Thus, RAG is needed to collect external information for accurate and reliable answer generation.
Question-Only Retrieval (QOR). To align with real-world cases of medical QA, answer options should not be provided as input during retrieval. This is a more realistic setting for evaluating RAG systems. While Liévin et al. (2022) and Lála et al. (2023) evaluated LLMs with RAG on medical QA, options were used for retrieval in their work, which is not a realistic setting. To the best of our knowledge, we are the first to propose and employ this setting for medical QA evaluation.
Table 1 lists related work on the evaluation settings. Only MIRAGE adopts all four considerations.

heading: 3.2 Component Datasets
text: As shown in Figure 1, MIRAGE contains five commonly used datasets for medical QA for the evaluation of RAG systems (Hendrycks et al., 2020; Jin et al., 2021; Pal et al., 2022; Jin et al., 2019; Tsatsaronis et al., 2015), including three medical examination QA datasets (MMLU-Med, MedQAUS, MedMCQA) and two biomedical research QA datasets (PubMedQA*, BioASQ-Y/N). Specifically, we only include multi-choice questions that are related to biomedicine and exclude all groundtruth supporting contexts for the questions. For example, we remove the contexts of PubMedQA and only use the questions, resulting in PubMedQA*. More details are described in the appendix. Table 2 presents the statistics of the datasets in MIRAGE.
As the tasks in MIRAGE are all composed of multi-choice questions, we evaluate a given RAG system by testing its performance in predicting the correct answer choices. For each specific task, we compute the accuracy of model predictions as the evaluation metric, as well as the standard deviation for the proportion of correctly answered questions, reflecting the error bound of the results. Across
all five tasks in MIRAGE, an average score of the accuracies will be measured to show how a given system performs on medical QA in general.

heading: 4 The MEDRAG Toolkit

heading: Retrievers Copora

heading: GPT-4

heading: Llama-2

heading: MEDITRON
text: To comprehensively evaluate how different RAG systems perform on our MIRAGE benchmark, we propose MEDRAG, a toolkit with systematic implementations of RAG for medical QA. As shown in Figure 2, MEDRAG consists of three major components: Corpora, Retrievers, and LLMs, which are briefly introduced in this section. More details of each component can be found in the appendix.
For corpora used in MEDRAG, we collect raw data from four different sources, including the commonly used PubMed4 for all biomedical abstracts, StatPearls5 for clinical decision support, medical Textbooks (Jin et al., 2021) for domain-specific knowledge, and Wikipedia for general knowledge. To the best of our knowledge, this is the first work that evaluates new corpora like StatPearls. We also provide a MedCorp corpus by combining all four corpora, facilitating cross-source retrieval. Each corpus is chunked into short snippets. Statistics of used corpora are shown in Table 3.
4https://pubmed.ncbi.nlm.nih.gov/ 5https://www.statpearls.com/
For the retrieval algorithms, while many general and domain-specific retrievers have been proposed (Remy et al., 2022; Ostendorff et al., 2022; Karpukhin et al., 2020; Xiong et al., 2020), we only select some representative ones in MEDRAG due to limited resources, including a lexical retriever (BM25, Robertson et al., 2009), a general-domain semantic retriever (Contriever, Izacard et al., 2022), a scientific-domain retriever (SPECTER, Cohan et al., 2020), and a biomedical-domain retriever (MedCPT, Jin et al., 2023a). Their statistics are presented in Table 4. In our experiments, 32 snippets are retrieved by default. Additionally, we utilize Reciprocal Rank Fusion (RRF, Cormack et al., 2009) to combine results from different retrievers, including RRF-2 (fusion of BM25 and MedCPT), and RRF-4 (fusion of all four retrievers).

heading: LLM Size Context Open Domain
text: Similarly, although various LLMs have emerged in recent years (Singhal et al., 2023a,b; Taylor et al., 2022; Luo et al., 2022; Yang et al., 2022), we select several frequently used ones in MEDRAG, including the commercial GPT-3.5 and GPT-4 (OpenAI et al., 2023), the open-source Mixtral (Jiang et al., 2024) and Llama-2 (Touvron et al., 2023b), and the biomedical domain-specific MEDITRON (Chen et al., 2023b) and PMC-LLaMA (Wu et al., 2023). Statistics of the used LLMs are in Table 5. For all LLMs, we concatenate and prepend retrieved snippets to the input, and perform chain-of-thought (CoT) prompting (Wei et al., 2022) in MEDRAG to fully leverage the reasoning capability of the models. Temperatures are set to 0 for deterministic outputs. CoT without RAG is used as the baseline.

heading: 5 Results
text: We systematically evaluate MEDRAG on our MIRAGE benchmark, which provides us with a multidimensional analysis of different components in RAG for medicine6. Section 5.1 presents the results for different LLMs, and Section 5.2 includes the results of different corpora and retrievers. Based on the results, we provide practical recommendations for RAG implementations in Section 6.4.

heading: 5.1 Comparison of Backbone LLMs
text: We first benchmark various LLMs on MIRAGE under both the CoT and the MEDRAG settings. For different LLMs, we use the same MedCorp corpus and the RRF-4 retriever and prepend 32 retrieved snippets for RAG. Results are shown in Table 6.
Under the CoT setting, GPT-4 significantly outperforms other competitors, with an average score of 73.44% on MIRAGE. While the best average score of other backbone LLMs can only achieve about 61% (GPT-3.5 and Mixtral) in the CoT setting, their performance can be significantly improved to around 70% with MEDRAG, which is comparable to GPT-4 (CoT). These results suggest the great potential of RAG as a way to enhance the zero-shot capability of LLMs to answer medical questions, which can be a more efficient choice than performing larger-scale pre-training. On all five tasks in MIRAGE, Mixtral shows an accuracy of 61.42% on average in the CoT setting, which slightly surpasses the performance of GPT-3.5. However, Mixtral is still outperformed
6Leaderboard: https://teddy-xionggz.github.io/ MIRAGE/
by GPT-3.5 with MEDRAG by 3.0%, indicating the advantage of GPT-3.5 in following MEDRAG instructions.
Our results also demonstrate that domainspecific LLMs can exhibit advantages in certain cases. For example, in the CoT setting for PubMedQA*, MEDITRON and PMC-LLaMA present significantly higher accuracies than all other models, including GPT-4 (+34.8% & +40.9%). Additionally, MEDITRON shows a better performance in both CoT (+13.5%) and MEDRAG (+12.7%) than its base Llama-2 model. The comparison of Llama-2 (MEDRAG) and MEDITRON (CoT) reflects the differences between RAG (+6.3%) and supervised fine-tuning (SFT, +13.5%) in improving the performance of LLMs on medical QA. While SFT is better at fusing medical knowledge into LLMs, RAG remains a more flexible and costefficient way to improve medical QA. For questions in PubMedQA* and BioASQ-Y/N where the closely related literature can be found from PubMed, MEDRAG greatly improves the ability of Llama-2 to answer medical questions (+19.4% & +20.9%), leading to a comparable or even better performance than MEDITRON (CoT). However, for examination questions in MIRAGE that are carefully designed to differentiate between medical students, MEDRAG does not always improve over SFT since the helpful snippets might be difficult to retrieve. The performance gap between these two types of questions suggests that there is still much room for improvement.
denote performance decreases and increases compared to CoT (first row). The shade reflects the relative change.

heading: 5.2 Comparison of Corpora and Retrievers
text: We also compare how different corpora and retrievers affect the MIRAGE performance with MEDRAG. Based on the results in Table 6, we conduct the following experiments with GPT-3.5 as it benefits the most from MEDRAG (+17.9%).
As shown in Table 7, the performance of one RAG system is strongly related to the corpus it selects. MEDRAG with Textbooks achieves the highest accuracy on MMLU-Med (76.68%) and the one with StatPearls performs the best on MedQA-US (67.48%). However, these two corpora provide little assistance in answering questions from PubMedQA* and BioASQ-Y/N, which almost solely benefit from the PubMed corpus. This is expected due to the design of these two datasets. Overall, PubMed is the only corpus that provides improvement for all MIRAGE tasks, probably due to its large scale and domain-specificity. Therefore, se-
lecting a suitable corpus for the task should be the first key step in RAG for medicine. While choosing task-specific corpora may require expert knowledge, we find MedCorp, a simple combination of all corpora, that performs robustly across various tasks, to be a satisfactory solution. As for the four tasks mentioned above, MEDRAG can always find useful snippets from the MedCorp corpus. Even on MedMCQA, where MEDRAG does not benefit from any single corpus, MedCorp still improves almost all retrievers (-1.5% ∼ +5.0%).
The selection of retrievers is another flexibility in MEDRAG that affects overall performance, which decides whether relevant information can be found from corpora. Table 7 shows the variable performance of different retrievers, which can be explained by the data and strategy differences in their training. For example, MedCPT is a biomedical retriever that has been trained on PubMed user logs. Thus, compared with other retrievers, it has
a better performance when PubMed is used as the corpus in MEDRAG (+0.2% ∼ +7.7%). Similarly, with Wikipedia as part of the training data, Contriever shows better performance than other retrievers in tasks with the Wikipedia corpus, especially on MMLU-Med and MedQA-US. Moreover, during the training of SPECTER, the retriever is tuned to regularize pairwise article distances rather than query-to-article distances. As such, it has an inferior average performance to other individual retrievers (-7.8% ∼ -6.8%) on MedCorp as its training setting mismatches the cases in medical QA.
Table 7 also shows that the fusion of retrieval results with RRF effectively improves the performance on MIRAGE. Using MedCorp, MEDRAG with RRF-4 have a 1.4% to 10.7% increase in the average performance compared to individual retrievers. However, the fusion of more retrievers may not always lead to a better performance. For example, on Wikipedia where SPECTER has a poor performance across all tasks, RRF-2 shows a better average performance than RRF-4 on MIRAGE (+1.7%). Specifically, for tasks like BioASQY/N where both Contriever and SPECTER perform poorly, RRF-2 can significantly improve the performance of MEDRAG, which is better than RRF-4 (+5.8%) and all other individual retrievers (+3.7% ∼ +14.0%). In contrast, on MedQA-US where Contriever achieves the best score (65.99%), RRF2 underperforms RRF-4 (-0.6%). On the MedCorp corpus where MEDRAG can benefit from all retrievers, RRF-4 brings a larger improvement than RRF2, with a state-of-the-art average score of 71.57% on our MIRAGE benchmark.

heading: 6 Discussions

heading: 6.1 Performance Scaling
text: We explore how the performance of MEDRAG scales with the increase in the number of snippets used for medical QA. To study the scaling properties, we use GPT-3.5 as the backbone LLM, RRF-4
as the retriever, and MedCorp as the corpus. Figure 3 shows the scaling curves of MEDRAG on each task in MIRAGE with different numbers of snippets k ∈ {1, 2, 4, ..., 64}. On MMLU-Med, MedQA-US, and MedMCQA, we see roughly loglinear curves in the scaling plots for k ≤ 32. The results show that when k is small (k ≤ 8 in this case), MEDRAG cannot provide enough useful information, which even hinders the LLM from using its inherent knowledge to derive the correct answer. In general, the RAG performance improves as k increases, indicating the existence of helpful knowledge from the retrieved snippets. However, the RAG performance can drop when k is too large and the signal-noise-ratio begins to decrease.
Compared with the three examination tasks, PubMedQA* and BioASQ-Y/N can be relatively easier for MEDRAG since the ground-truth supporting information can be found in PubMed. Figure 3 reveals that MEDRAG can achieve high accuracy on PubMedQA* with just k = 1, and its performance drops with the increase of k as more irrelevant snippets are entered, which corresponds to the fact that 79.6% ground-truth snippets are successfully identified as the top-1 related context by the retrieval system. MEDRAG also shows a dramatic increase in accuracy on BioASQ-Y/N when k = 1, whose performance continues to grow as k gets larger.

heading: 6.2 Position of Ground-truth Snippet
text: Liu et al. (2023) found the RAG performance is lowest when the relevant information is placed in the middle, a phenomenon known as “lost-in-themiddle”. In our MIRAGE benchmark, PubMedQA* and BioASQ-Y/N are the tasks that have groundtruth labels of the supporting snippets for each question. Here we use PubMed as the corpus, and take GPT-3.5 and RRF-4 as the LLM and retriever, respectively. For each dataset, we group the positions of ground-truth snippets into several bins, on which we evaluate how accurate MEDRAG is in answering questions whose ground-truth snippets are in
corresponding bins. For PubMedQA*, we only show the results of the first 18 positions, since no ground-truth snippets have been placed after it.
Figure 4 shows the changes in model accuracy corresponding to different parts of context locations. From the figure, we can see a clear U-shaped decreasing-then-increasing pattern in the accuracy change concerning the position of ground-truth snippets, which sheds light on the arrangement of snippets for medical RAG in future research.

heading: 6.3 Proportion in the MedCorp Corpus
text: We also examine the proportion of different sources in the retrieved snippets from MedCorp, and explore how this proportion changes across different tasks. Figure 5 displays the proportions of four different sources in MedCorp and the actually retrieved sources in the top 64 retrieved snippets for each task in MIRAGE. It can be observed from the figure that, in general, the proportion of Wikipedia drops in the retrieved snippets for medical questions, which is expected as many snippets in Wikipedia are not related to biomedicine.
Comparing the distributions for different tasks, there is a task-specific preference pattern. Medical examination tasks (MMLU-Med, MedQAUS, and MedMCQA) tend to have a larger proportion of retrieved snippets from Textbooks and StatPearls. PubMedQA* and BioASQ-Y/N with research-related questions have more relevant snippets from PubMed. The Textbooks corpus has a larger proportion in MedQA-US than in other datasets, which can be explained the fact that this corpus is composed of frequently used textbooks for the US medical licensing examination.

heading: 6.4 Practical Recommendations
text: In this section, we discuss the practical indications and recommendations based on our evaluation results of different MEDRAG settings on MIRAGE.
Corpus selection. Results in Table 7 indicate that PubMed and the MedCorp corpus are the only corpora with which MEDRAG can outperform CoT on all tasks in MIRAGE. As a large-scale corpus, PubMed serves as a suitable document collection for various kinds of medical questions. If resources permit, the MedCorp corpus could be a more comprehensive and reliable choice: Nearly all MEDRAG settings using the MedCorp Corpus show improved performance (green-coded cells) compared to the CoT prompting baseline. In general, single corpora other than PubMed are not recommended for medical QA due to their limited volumes of medical knowledge, but they can also be beneficial in specific tasks such as question answering for medical examinations.
Retriever selection. Among the four individual retrievers used in MEDRAG, MedCPT is the most reliable one which constantly outperforms other candidates with a higher average score on MIRAGE. BM25 is a strong retriever as well, which is also supported by other evaluations (Thakur et al., 2021). The fusion of retrievers can provide robust performance but must be utilized with caution for the retrievers included. As for the PubMed corpus recommended above, a RRF-2 retriever that combines the results from BM25 and MedCPT can be a good selection, since they perform better than the other two with snippets from PubMed. For the MedCorp corpus, both RRF-2 and RRF-4 can be reliable choices, as the corpus can benefit all four individual retrievers in MEDRAG.
LLM selection. Currently, GPT-4 is the best model with about 80% accuracy on MIRAGE. However, it is much more expensive than other backbone LLMs. GPT-3.5 can be a more cost-efficient choice than GPT-4, which shows great capabili-
ties of following MEDRAG instructions. For highstakes scenarios such as medical diagnoses where patient privacy should be a key concern, the best open-source Mixtral model, which can be deployed locally and run offline, could be a viable option.

heading: 7 Conclusion
text: To evaluate RAG systems in medicine, we introduced the MIRAGE benchmark and the MEDRAG toolkit. Based on our comprehensive evaluations, we presented many novel observations and practical recommendations to guide the research and real-world deployments of medical RAG systems.

heading: Limitations
text: While our study provides systematic evaluations and practical recommendations for medical RAG systems, there are several limitations that need to be acknowledged. First, there have been novel developments in the architecture of RAG (e.g., active RAG, Jiang et al., 2023). However, we mainly evaluate the vanilla RAG architecture where the retrieved documents are directly prepended in the LLM context because this is the most widely implemented architecture. Evaluating new RAG system designs remains an important direction to explore. Second, while the coverage of corpora, retrievers, and LLMs in MEDRAG is reasonably comprehensive, there are other potentially useful resources that can also be incorporated into MEDRAG in future work, such as the full-text articles from PubMed Central (PMC)7 and Frequently Asked Questions (FAQs) from trustworthy sources (Ben Abacha and Demner-Fushman, 2019). Third, we only evaluate the retrieval component for PubMedQA* and BioASQ-Y/N since the other three examination datasets lack labels of ground-truth supporting documents. Further research should also evaluate whether the retrieved snippets are actually helpful for the examination datasets, and explore the use of cross-encoder re-rankers to improve the retrieval performance for relevant information. Fourth, while QA is the most commonly used task for evaluating biomedical LLMs, there are also other knowledge-intensive tasks that might benefit from MEDRAG, such as claim verification (Wadden et al., 2020; Liu et al., 2024). Following most other studies, we use the format of multichoice questions for large-scale and automatic evaluation of medical QA. Although we restrict the
7https://www.ncbi.nlm.nih.gov/pmc/
retrieval phase to having no access to the choices, LLMs still need to use them as input for the final prediction. The rationales generated by MEDRAG remain to be evaluated as well. As the goal of this study is to systematically benchmark the most commonly used medical RAG settings, we leave the potential solutions of the above-mentioned limitations to future work.

heading: Acknowledgements
text: Guangzhi Xiong and Aidong Zhang are supported by NIH grant 1R01LM014012 and NSF grant 2333740. Qiao Jin and Zhiyong Lu are supported by the NIH Intramural Research Program, National Library of Medicine.

heading: Appendix

heading: A Details of MIRAGE Datasets
text: The selection of datasets in a crucial step for constructing our MIRAGE benchmark. Our study requires large-scale experiments to comprehensively evaluate various RAG settings in medicine. While open-ended questions, as compared to multi-choice questions, offer a closer simulation of real-world settings, their evaluation presents significant challenges. On the one hand, manual evaluation is often considered the gold standard but is prohibitively expensive. On the other hand, although automatic metrics are scalable, they do not correlate well with human judgments (Guan et al., 2021; Wang et al., 2023a; Fabbri et al., 2021), further complicating the evaluation. As such, studies that evaluate open-ended QA in medicine are typically limited in scale, using only hundreds of questions (for example, Singhal et al. (2023a); Zakka et al. (2024)). In addition to the evaluation issue, another practical reason for not including such questions is the lack of commonly used open-ended QA datasets for LLM evaluation in medicine.
Therefore, we adopted a multiple-choice format as it is suitable for scalable evaluation of medical knowledge. This format allows us to assess the medical knowledge of RAG systems directly in an objective manner without introducing additional information, which is commonly used to evaluate the ability of LLMs to solve medical tasks (Singhal et al., 2023a,b; Nori et al., 2023b,a; Liévin et al., 2022). Despite focusing on multi-choice questions, our methodology is designed to mirror real-world applications as closely as possible. By adopting question-only retrieval, we simulate scenarios where answer options are not available for retrieval, thus requiring the system to rely solely on its comprehension of the medical questions. Moreover, our use of a zero-shot learning setting further ensures that our medical RAG system evaluation reflects real-world constraints that no training or demonstration data are provided for user queries.
Here are the descriptions of the five datasets used in MIRAGE.
MMLU-Med. Massive Multitask Language Understanding (MMLU)8 is a benchmark for the evaluation of the multitask learning capability of language models. The benchmark contains a variety of 57 different tasks (Hendrycks et al., 2020). To
8https://github.com/hendrycks/test
measure the performance of medical RAG systems, we select a subset of six tasks that are related to biomedicine following (Singhal et al., 2023a), including anatomy, clinical knowledge, professional medicine, human genetics, college medicine, and college biology. The subset is collectively denoted as MMLU-Med. Only the test set of each task is used in our benchmark, which contains 1089 questions in total.
MedQA-US. MedQA9 (Jin et al., 2021) is a multi-choice QA dataset collected from professional medical board exams. Specifically, we focus on the English part, which includes real-world questions from the US Medical Licensing Examination (MedQA-US). The 1273 four-option test questions are included in our MIRAGE benchmark.
MedMCQA. MedMCQA10 (Pal et al., 2022) contains 194k multi-choice questions collected from Indian medical entrance exams. The questions cover a wide range of 2.4k healthcare topics and 21 medical subjects. Since the ground truth of its test set is not provided, the dev set of the original MedMCQA is chosen for MIRAGE, including 4183 medical questions.
PubMedQA*. PubMedQA11 (Jin et al., 2019) is a biomedical research QA dataset. It has 1k manually annotated questions constructed from PubMed abstracts. Different from the datasets above, PubMedQA also provides a relevant context for each question to evaluate the reasoning ability of language models. To test the capability of RAG systems to find related documents and answer the question accordingly, we build PubMedQA* by removing given contexts in the 500 expert-annotated test samples of PubMedQA following (Lála et al., 2023). The possible answer to a PubMedQA* question can be yes/no/maybe, reflecting the authenticity of the question statement based on scientific literature. As the relevant knowledge of questions in PubMedQA* can often be covered by a small number of PubMed articles, it simplifies the retrieval of sufficient information and makes PubMedQA* a less challenging dataset compared with other tasks, However, PubMedQA* functions as a basic and indispensable evaluation task to test if the given RAG system works in relatively simple cases.
9https://github.com/jind11/MedQA 10https://medmcqa.github.io/ 11https://pubmedqa.github.io/
BioASQ-Y/N. BioASQ12 (Tsatsaronis et al., 2015; Krithara et al., 2023) is an annual competition for biomedical QA, which includes both the information retrieval track (Task A) and machine reading comprehension track (Task B). To leverage the resources of BioASQ for our medical RAG benchmark, we select the Yes/No questions in the ground truth test set of Task B from the most recent five years (2019-2023), including 618 questions in total. In the original task, questions are constructed based on biomedical literature, and the ground truth snippets are provided as a basis for machine reading comprehension. Similar to PubMedQA*, BioASQ-Y/N is also a modified version on which RAG systems are supposed to answer the questions without the ground-truth snippet provided.

heading: B Detailed Descriptions of MEDRAG

heading: B.1 Document Collections
text: PubMed. PubMed13 is the most widely used literature resource (Lu, 2011; Jin et al., 2024), containing over 36 million biomedical articles. Many relevant studies solely use PubMed as the retrieval corpus (Frisoni et al., 2022; Naik et al., 2022). For MEDRAG, we use a PubMed subset of 23.9 million articles with valid titles and abstracts.
StatPearls. StatPearls14 is a point-of-the-care clinical decision support tool similar to UpToDate15. We use the 9,330 publicly available StatPearl articles through NCBI Bookshelf16 to construct the StatPearls corpus. We chunked StatPearls according to the hierarchical structure, treating each paragraph in an article as a snippet and splicing all the relevant hierarchical headings as the corresponding title. To the best of our knowledge, our work presents the first evaluation of StatPearls in the biomedical NLP community.
Textbooks. Textbooks17 (Jin et al., 2021) is a collection of 18 widely used medical textbooks, which are important references for students taking the United States Medical Licensing Examination (USLME). In MEDRAG, the textbooks are processed as chunks with
12http://bioasq.org/ 13https://pubmed.ncbi.nlm.nih.gov/ 14https://www.statpearls.com/ 15https://www.uptodate.com/ 16https://www.ncbi.nlm.nih.gov/books/NBK430685/ 17https://github.com/jind11/MedQA
no more than 1000 characters. We used the RecursiveCharacterTextSplitter from LangChain18 to perform the chunking.
Wikipedia. As a large-scale open-source encyclopedia, Wikipedia is frequently used as a corpus in information retrieval tasks (Thakur et al., 2021). We select Wikipedia as one of the corpora to see if the general domain database can be used to improve the ability of medical QA. We downloaded the processed Wikipedia data from HuggingFace19 and also chunked the text with LangChain.

heading: B.2 Retrieval Systems
text: BM25. BM25 (Robertson et al., 2009) is a commonly used baseline retriever which use bag-ofwords and TF-IDF to perform lexical retrieval. In MEDRAG, BM25 is implemented with Pyserini (Lin et al., 2021)20 using the default hyperparameters to index snippets from all corpora.
Contriever. Contriever21 (Izacard et al., 2022) is a dense retriever pre-trained on Wikipedia and CCNet (Wenzek et al., 2020) with contrastive learning. It is shown to be competitive with BM25 on retrieval tasks in the general domain (Thakur et al., 2021).
SPECTER. SPECTER22 (Cohan et al., 2020) is a document-level scientific dense retriever which was pre-trained on the Semantic Scholar corpus (Ammar et al., 2018) to encode similar documents with close embeddings.
MedCPT. MedCPT (Jin et al., 2023a) is a biomedical embedding model that is contrastively pre-trained by 255 million user clicks from PubMed search logs (Fiorini et al., 2018). It achieved state-of-the-art performance on several biomedical IR tasks. We use the MedCPT Query Encoder23 and Article Encoder24 to encode the questions and corpus snippets, respectively.
RRF. Cormack et al. (2009) proposed to merge results from different retrievers with Reciprocal Rank Fusion (RRF), which effectively fuses the
18https://www.langchain.com/ 19https://huggingface.co/datasets/wikipedia 20https://github.com/castorini/pyserini 21https://huggingface.co/facebook/contriever 22https://huggingface.co/allenai/specter 23https://huggingface.co/ncbi/
MedCPT-Query-Encoder 24https://huggingface.co/ncbi/ MedCPT-Article-Encoder
information from different sources by selecting shared predictions. In MEDRAG, we provide two versions of RRF systems, RRF-2 and RRF-4. RRF2 is the fusion of results from BM25 and MedCPT, which appear to be the optimal lexical and dense retrievers in our experiments. RRF-4 is a more comprehensive system which fuses the information from all individual retrievers used.

heading: B.3 Backbone LLMs
text: GPT-3.5 & GPT-4. GPT-3.5 and GPT-4 (OpenAI et al., 2023) are two popular commercial LLMs developed by OpenAI, which have already shown great capabilities in answering medical questions (Nori et al., 2023b; Liévin et al., 2022). In MEDRAG, we use the specific version of GPT3.5-turbo-16k-0613 and GPT-4-32k-0613 accessed through Microsoft Azure OpenAI Services25.
Mixtral. In MEDRAG, we use Mixtral-7×8B26, which is an open-source sparse mixture of expert models. Compared with existing open-source models, Mixtral-7×8B can achieve both good task performance and fast inference speed (Jiang et al., 2024).
Llama-2. Llama-2 (Touvron et al., 2023b) is a series of open-source models that are pre-trained on large-scale data and fine-tuned with human instructions. In MEDRAG, we use Llama-2-70B27, which is the largest model in the Llama-2 series.
MEDITRON. MEDITRON (Chen et al., 2023b) is a series of biomedical LLMs that are built based on Llama-2 and fine-tuned on open-source biomedical literature. Its 70B28 version model is contained in MEDRAG.
PMC-LLaMA. PMC-LLaMA (Wu et al., 2023) is fine-tuned based on LLaMA (Touvron et al., 2023a) using PubMed Central (PMC) papers. Its largest version, PMC-LLaMA-13B29, is included in MEDRAG.

heading: C Prompt Templates
text: Here are the prompt templates used in our experiments. Figures 6 and 7 show the template for all
25https://oai.azure.com/ 26https://huggingface.co/mistralai/
Mixtral-8x7B-Instruct-v0.1 27https://huggingface.co/meta-llama/ Llama-2-70b-chat-hf 28https://huggingface.co/epfl-llm/meditron-70b 29https://huggingface.co/axiong/PMC_LLaMA_13B
LLMs except MEDITRON. Since the officially released checkpoint of MEDITRON30 is only the pretrained version without any instruction tuning, it cannot follow the given system prompt well. Therefore, we provide a pseudo one-shot demonstration in the prompt for MEDITRON, where the demonstration does not contain any information of real examples. The templates for MEDITRON are provided in Figures 8 and 9.
30https://huggingface.co/epfl-llm/meditron-70b

heading: Future Work
text: Second, while the coverage of corpora, retrievers, and LLMs in MEDRAG is reasonably comprehensive, there are other potentially useful resources that can also be incorporated into MEDRAG in future work, such as the full-text articles from PubMed Central (PMC)7 and Frequently Asked Questions (FAQs) from trustworthy sources (Ben Abacha and Demner-Fushman, 2019). Third, we only evaluate the retrieval component for PubMedQA* and BioASQ-Y/N since the other three examination datasets lack labels of ground-truth supporting documents. Further research should also evaluate whether the retrieved snippets are actually helpful for the examination datasets, and explore the use of cross-encoder re-rankers to improve the retrieval performance for relevant information. Fourth, while QA is the most commonly used task for evaluating biomedical LLMs, there are also other knowledge-intensive tasks that might benefit from MEDRAG, such as claim verification (Wadden et al. , 2020; Liu et al. , 2024). Following most other studies, we use the format of multichoice questions for large-scale and automatic evaluation of medical QA. Although we restrict the
7https://www. ncbi. nlm. nih. gov/pmc/
retrieval phase to having no access to the choices, LLMs still need to use them as input for the final prediction. The rationales generated by MEDRAG remain to be evaluated as well. As the goal of this study is to systematically benchmark the most commonly used medical RAG settings, we leave the potential solutions of the above-mentioned limitations to future work