To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering (2024)

Abstract: Medical open-domain question answering demands substantial access to specialized knowledge. Recent efforts have sought to decouple knowledge from model parameters, counteracting architectural scaling and allowing for training on common low-resource hardware. The retrieve-then-read paradigm has become ubiquitous, with model predictions grounded on relevant knowledge pieces from external repositories such as PubMed, textbooks, and UMLS. An alternative path, still under-explored but made possible by the advent of domain-specific large language models, entails constructing artificial contexts through prompting. As a result, "to generate or to retrieve" is the modern equivalent of Hamlet’s dilemma. This paper presents MEDGENIE, the first generate-thenread framework for multiple-choice question answering in medicine. We conduct extensive experiments on MedQA-USMLE, MedMCQA, and MMLU, incorporating a practical perspective by assuming a maximum of 24GB VRAM. MEDGENIE sets a new state-of-the-art in the open-book setting of each testbed, allowing a small-scale reader to outcompete zero-shot closed-book 175B baselines while using up to 706× fewer parameters. Our findings reveal that generated passages are more effective than retrieved ones in attaining higher accuracy.1

heading: 1 Introduction
text: Question answering is a challenging task that requires complex reasoning on explicit constraints described in the question and unstated domain knowledge. Open-domain question answering (ODQA) aims to tackle natural questions across various topics without predefined evidence (Chen et al., 2017). This setting mirrors real-world scenarios where
* Equal contribution (co-first authorship). 1Our code, fine-tuned models, and generated contexts are publicly available at https://github.com/unibo-nlp/medgenie.
there cannot be a labeled passage for every potential user inquiry. ODQA has particular significance in medicine due to the high-quality standards it demands, including a deep understanding of specialized terminology and background concepts, and an effective recall of expert insight for clinical decision making (Frisoni et al., 2022).
Recent efforts have transitioned from a closedbook strategy, where models rely solely on their opaque parametric knowledge, to an open-book alternative, allowing them to consult external sources for grounding. In particular, the retrieve-thenread framework is a common thread (Zhu et al., 2021; Zhang et al., 2023a), where the input is augmented with relevant knowledge chunks retrieved from an external datastore, which can be unstructured (e.g., PubMed, textbooks) or structured (e.g., UMLS). However, performance is highly depen-
9878
dent on the quality of the retriever. Developing custom retrieval modules generally requires extensive question–context pairs (Karpukhin et al., 2020) or intensive computational resources (Lee et al., 2019), particularly when dealing with massive sources (Gan et al., 2023). Furthermore, the retrieved fragments may be incomplete or not specifically tailored to the query, leading to noise (Oh and Thorne, 2023).
In parallel, medical large language models (LLMs) have gained increasing research interests to aid professionals and improve patient care (Zhou et al., 2023). After pre-training on an extreme-scale collection of specialized text corpora, they implicitly encode an impressive amount of domain knowledge that can be evoked through prompting, akin to summoning a genie from a lamp. This facilitates a paradigm shift towards a generate-then-read approach, wherein contexts are directly generated by an LLM. Despite preliminary work (Yu et al., 2023; Su et al., 2023; Zhang et al., 2023b), there is an ongoing debate on “whether generative augmentation is preferable to retrieval augmentation”.
In this paper, we introduce MEDGENIE, the first generate-then-read framework for multiplechoice medical ODQA. Specifically, we study the effectiveness of grounding generalist LLMs and small language models (SLMs) on multi-view contexts generated by a medical LLM via in-context learning (ICL) and fine-tuning, respectively. To foster accessibility and match prevalent hardware configurations, we assume a low-resource infrastructure with 24GB VRAM.
We evaluate MEDGENIE on three standard ODQA benchmarks designed to quantify professional medical competencies: MedQA-USMLE (Jin et al., 2020), MedMCQA (Pal et al., 2022), and MMLU-Medical (Hendrycks et al., 2021). MEDGENIE demonstrates significant performance gains, improving the accuracy of few-shot LLM readers on all testbeds by up to ≈16 points. By fine-tuning the reader, MEDGENIE allows FlanT5-base to outcompete closed-book zero-shot 175B LLMs and supervised 10B baselines on MedQA (Figure 1), using up to 706× fewer parameters. Furthermore, our research demonstrates a clear inclination of cutting-edge rerankers towards favoring generated contexts over retrieved ones. When treated as knowledge sources or incorporated into humancurated ones, generated passages notably enhance the effectiveness of retrieve-then-read workflows (up to ≈6 extra points). RAGAS evaluation con-
firms the quality of generated contexts, even allowing for more faithful answers from the reader. Finally, we release a comprehensive dataset of ≈1 million artificial contexts in the medical field, adhering to principles of open science and encouraging further research endeavors.

heading: 2 Related Work
text: Medical Language Models Transformer-based language models have become the go-to solution for any task in medical NLP. Early work on domain adaption focused on encoder-only models pretrained on PubMed articles and abstracts, counting 40+ models introduced between 2019 and 2021 (Kalyan et al., 2022). With the recent trend of scaling up pre-training data and model parameters, multiple studies have delved into medical LLMs, transitioning to decoder-only architectures and generative tasks (He et al., 2023; Zhou et al., 2023). Popular open-source milestones include ChatDoctor (Li et al., 2023), PMC-LLaMA (Wu et al., 2023), and MEDITRON (Chen et al., 2023). "Pre-train, prompt, and predict" is today’s prevalent methodology for applying LLMs to new problems, circumventing the need for extensive fine-tuning on human-labeled examples, which are rarely accessible in clinical scenarios. We are the first to examine the prowess of medical LLMs in generating background context for ODQA without supervision.
Open-Book Question Answering Closed-book LLMs, such as GPT-IV and Med-PaLM-2 540B, exhibit impressive results in ODQA with fine-tuning (Singhal et al., 2023) or prompting (Nori et al., 2023a,b) techniques, performing non-trivial reasoning steps (Liévin et al., 2022). As LLMs grow to achieve predictable accuracy gains (Kaplan et al., 2020), their computational demands exceed the resources of most product teams. To counteract the scaling trend, reach satisfactory results with commodity hardware, and ensure greater control over knowledge, various open-book contributions have surfaced in medicine, comprising BioReader (Frisoni et al., 2022), DRAGON (Yasunaga et al., 2022), and VOD (Liévin et al., 2023). Nevertheless, to date, generate-then-read pipelines have predominantly undergone scrutiny solely within the general domain. GenRead (Yu et al., 2023) and CGAP (Zhang et al., 2023b) are among the first to demonstrate the efficacy of LLMs as strong context generators, focusing on datasets like Natural Questions, TriviaQA, and WebQuestions. They utilized
9879
LLMs such as InstructGPT (175B) and Megatron (530B) both as generators and readers. GenRead introduced a clustering-based prompting technique involving context generation, filtering, encoding, kmeans clustering, and per-cluster sampling. While we acknowledge the efficacy of these techniques, as demonstrated in ad-hoc ablations (Appendix D), we caution against their feasibility in real-world scenarios due to associated high costs. Additionally, investigations into the fusion of retrieved and LLM-generated contexts for ODQA, exemplified by COMBO (Su et al., 2023), have shown promising results in enhancing performance across aforementioned benchmarks. However, their approach entails utilizing two discriminators trained on silver labels to compute compatibility scores, with InstructGPT and ChatGPT as context generators and a FiD Flan-T5-large as the reader, scaling up to 10 contexts using four A40 GPUs, each equipped with 46 GB VRAM.

heading: 3 Method
text: In this section, we discuss our MEDGENIE framework, illustrated in Figure 2. First, it prompts a medical LLM to furnish multi-view background contexts for a given question (§3.1). Then, it diverges into two distinct paths depending on the reader type: ICL for LLMs (§3.2), and Fusion-InDecoder (FID) fine-tuning for SLMs (§3.3). Both strategies equip readers with custom background passages, allowing them to tackle medical questions effectively even without prior knowledge.
Problem Statement In multiple-choice ODQA, a dataset record consists of a question q and an answer set A={a1, . . . ,a|A|}, all expressed in text. Our goal is to identify which answer a ∈ A is correct (â). A closed-book solution would estimate pθ(a|q,A) parameterized by θ, leaving considerable domain knowledge unexploited (Levine et al., 2022). Conversely, we assume access to a set of record-specific artificial contexts and add an auxiliary variable C={c1, . . . , c|C|}. We thus move to a marginal likelihood with a reader-generator model: pθ(a|q,A)= ∑ i pθ(a|Ci,q,A)pθ(Ci|q,A). Since we cannot sum over all possible contexts, we approximate the maximum a posteriori estimation with a decoding strategy, using a single value for C.

heading: 3.1 Multi-view artificial contexts
text: In the first stage, we prompt a medical LLM to output C. The primary challenge in generating mul-
tiple synthetic contexts for the same question is to promote diversity and knowledge coverage while avoiding repetitive information. Addressing this concern, Yu et al. (2023) proposed a three-step approach: (i) building a supporting repository with an artificial document for each q of the training set, (ii) encoding artificial documents and clustering them, (iii) given a training or inference query, sampling various question–document pairs from each cluster, and using them as ICL demonstrations to craft a series of topic-guided contexts. Although we acknowledge that clustering-based prompting can amplify the effectiveness of MEDGENIE (Appendix §D), it introduces complexity, largely impacting time and memory costs. To streamline the process and reduce resource constraints, we devise a multi-view prompt technique, dividing C into two segments, Ca and Cb. We first ask the LLM to generate a set of contexts {ca,1, . . . , ca,l} conditioned on q ∪ a ∈ A (option-focused), and then a set of contexts {cb,1, . . . , cb,m} conditioned solely on q (option-free). The former aims to provide information for each candidate answer, whereas the latter targets a broader context, encompassing pertinent domain concepts that could aid reasoning. Figure 3 clarifies their distinction.
We rely on few-shot prompting (Brown et al., 2020) to guide the LLM on two examples Ec meticulously curated by a healthcare professional in distinct and representative domains:
Pra= [|E| ⊕ i [qi ⊕Ai ⊕ ci] ] ⊕ q⊕A (1)
Prb= [|Ec| ⊕ i [qi ⊕ ci] ] ⊕ q (2)
where Pra is the option-focused prompt, Prb the option-free one, and ⊕ denotes concatenation. Full prompt templates are listed in Appendix §K.
To promote intra-context diversity, we (i) use a random sampling decoding strategy with high temperature, ensuring a wider exploration of the model’s output space, and (ii) penalize new tokens based on their frequency in the prompt and the inferred text. Details in Appendix §A.
According to in-depth ablation studies (Appendix §B.3), we set l=3 and m=2, corroborating the importance of each view. Importantly, we preprocess each context by removing any answerguessing phrase that could bias the reader.
9880

heading: 3.2 ICL reader (Unsupervised)
text: For the unsupervised setting, we use ICL which does not need the complete training data but only a few demonstration examples. In particular, we feed an LLM reader with few-shot ODQA demonstrations and the test query preceded by its artificial context. Inferring the prompt continuation indirectly accomplishes the unseen ODQA task. Mechanically, we build the following two-shot prompt:
[|Ea| ⊕ i ci ⊕ qi ⊕Ai ⊕ âi ] ⊕ c⊕ q⊕A (3)
where Ea encloses two train instances of the target benchmark, and ci is an option-focused context from the medical LLM—proved to be more effective than a human sample (Appendix §C). c is the concatenation of option-focused and option-free contexts from §3.1. Consistently, we assign the initial positions within c to option-focused contexts due to their higher probability of containing the correct answer. This conjecture is further validated by
the outcomes of our experiments in §5.2. Our design choice wants to prevent Ca from "getting lost in the middle" (Liu et al., 2023). Indeed, LLMs are prone to difficulty extracting relevant information when it is buried within the central portion of a lengthy context. Contrarily, accuracy improves when pertinent information is situated at either the beginning or end of the input context.

heading: 3.3 Fine-tuned reader (Supervised)
text: While the ICL methodology unlocks the utilization of off-the-shelf LLMs and harnesses their innate language understanding abilities, it mandates loading billions of weights into memory, making inference alone expensive. LLM readers could remarkably benefit from additional finetuning steps, but training generally requires up to 20 extra bytes per parameter, easily evolving prohibitive. Therefore, for the supervised regime, we train a lightweight FID reader (Izacard and Grave, 2021). Each <c,q,A> tuple is joined with special separator tokens, and processed independently by the encoder. The representations produced by the last encoder layer are then concatenated and sent to the decoder layers. In this way, the computation cost grows linearly with |C|. We keep the relative order of contexts fixed across all input pairs, always putting Ca in front of Cb.

heading: 4 Experimental Setup

heading: 4.1 Benchmarks
text: We evaluate MEDGENIE on three multiple-choice ODQA benchmarks (Table 1), requesting multihop reasoning capabilities, and human-level reading comprehension skills.
9881
MedQA-USMLE (Jin et al., 2020) Englishlanguage subset of MedQA tailored for medical license examination in the United States. It amalgamates various knowledge facets, such as patient profiles, disease symptoms, and drug dosage requirements. Each question entails a medical case history, vital signs (e.g., blood pressure, temperature), and eventual diagnostic evaluations (e.g., CT scan). It exists in two versions: 4 and 5 options. Following Chen et al. (2023), in the supervised pipeline, we fine-tune the reader over the original 5-option train set for fair comparisons.
MedMCQA (Pal et al., 2022) Highly comprehensive dataset sourced from Indian medical school entrance exams (AIIMS, NEET-PG). It features mixed question types, such as diagnosis and mathematical problems. Due to the absence of public answer labels for the test set, we employ the eval set as the main reference, adhering to prior literature (Liévin et al., 2022; Singhal et al., 2022; Nori et al., 2023b; Chen et al., 2023; Wu et al., 2023).
MMLU-Medical (Hendrycks et al., 2021) MMLU is a multi-task zero-shot benchmark suite of 57 subsets spanning STEM, humanities, and social sciences. In the footsteps of Singhal et al. (2022); Liévin et al. (2023); Chen et al. (2023), we limit our analysis to 9 medically relevant subsets. As MMLU lacks training data, we fine-tune the FID reader over the MedMCQA train set and gauge outof-domain generalization, securing fairness with Liévin et al. (2023); Chen et al. (2023).
Terminology. Throughout the remainder of this paper, we use the terms "MedQA" and "MMLU" to denote their medical subsets, aiming for succinctness. With regard to MedQA, unless otherwise specified, we refer to the common 4-option variant.

heading: 4.2 Medical-expert generator
text: Following exhaustive preliminary experiments (Appendix §B.1), we select PMC-LLaMA-13B as the
specialized generator model (Wu et al., 2023).2
Technically, PMC-LLaMA performs a two-step training above LLaMA (Touvron et al., 2023). First, it infuses medical knowledge by continuing the pre-training on 4.8M academic papers and 30K medical books. Second, it undergoes medical instruction tuning on question answering, rationale, and conversation, also exploiting the MedQA and MedMCQA train sets. Although the model has been exposed to the training data of some benchmarks, it is imperative to note that it was not explicitly trained to forge question-conditioned contexts. Pre-aligning the selected medical expert generator towards the benchmark data is not strictly necessary. The essential aspect is the latent knowledge encoded within the model parameters, which is frequently correlated with their quantity (Tirumala et al., 2022; Carlini et al., 2023). Nori et al. (2023b) operated CoT and self-consistency prompting in conjunction with major voting to recall knowledge within generalist foundational models and surpass domain-specialized counterparts. However, these methodologies demonstrate efficacy when applied to the largest versions of a foundational model (Singhal et al., 2022; Liévin et al., 2022; Chen et al., 2023), preferring domain-specific solutions at smaller magnitudes. In light of this, we opt for the largest medical LLM runnable on consumer GPU hardware. The average length of a single PMC-LLaMA context is ≈200 words; see Appendix §B.2 for distributional statistics.

heading: 4.3 Readers
text: MEDGENIE is reader-agnostic but, for this study, we adopt the following implementation choices.
• ICL. We test LLaMA-2-chat (7B) (Touvron et al., 2023)3 and Zephyr-β (7B) (Tunstall et al., 2023),4 both supporting a context window length of 4K tokens. Additionally, we examine more recent models with increased context windows: LLaMA-3-instruct (8B)5
with 8K tokens and Phi-3-mini-instruct (3.8B) with 128K tokens.6
• FID. We train a FID Flan-T5-base (250M) (Chung et al., 2022).7
2axiong/PMC_LLaMA_13B 3meta-llama/Llama-2-7b-chat-hf 4HuggingFaceH4/zephyr-7b-beta 5meta-llama/Meta-Llama-3-8B-Instruct 6microsoft/Phi-3-mini-128k-instruct 7google/flan-t5-base
9882
This article has evolved over two distinct versions, each exploring different LLM ICL readers:
v1 - February 2024: Experiments and ablation studies conducted utilizing LLaMA-2 and Zephyr. v2 - June 2024: Integration of the newly released LLaMA-3 and Phi-3-mini in the core experiments, with up-to-date baselines, namely BioMistral, MeditronLlama-Slerp, LLaMA-3-Meditron, and OpenBioLLM.

heading: 4.4 Evaluation
text: Multiple Choice Prompting and Direct Completion In alignment with recent literature (Nori et al., 2023b; Chen et al., 2023; Liévin et al., 2022; Singhal et al., 2022), as anticipated in §3, we utilize multiple-choice prompting (MCP). In MCP, the model is presented with both q and A, where each option is bound to a proxy symbol (A, B, C, D, E), and the model only has to predict a single token.8 This diverges from earlier cloze prompting (CP) strategies (Brown et al., 2020; Lieber et al., 2021; Du et al., 2022; Smith et al., 2022; Chowdhery et al., 2023), where only q is passed to the model and the candidate answers are scored independently. Unlike CP, MCP (i) does not conflate the likelihood of an answer with the likelihood of its text, (ii) does not require normalization procedures, (iii) explicitly contrasts different answer options, and (iv) only requires a single forward pass; see (Robinson and Wingate, 2023). Since we focus on evaluating the pure impact of artificial contexts, we operate under the strict assumption of directly inferring the correct answer with this additional input signal. We do not resort to alternative schemas, such as CoT or self-consistency CoT, beyond the scope of our research work. For reproducibility, we adopt greedy decoding as in Chen et al. (2023).
Metrics We use Accuracy (% of correctly solved questions) as our main metric across all benchmarks. We resort to Recall@K for measuring the % of retrieved generated contexts rather than goldchunked ones after top-K retrieval and reranking.

heading: 4.5 Baselines
text: We juxtapose MEDGENIE with two categories of models. Closed-book. We prioritize mediumsized LLM baselines (≤8B), either pre-trained or fine-tuned on medical data, fostering an equitable and resource-aware comparison with our models.
8The prediction may be longer (e.g., "(A) Vitamin B12"), but only the mentioned symbol is considered as the answer.
Open-book. We incorporate all the contributions to our knowledge, going beyond text-only grounding.
We further incorporate baselines implemented and run by us. For the closed-book category, we conduct few-shot inferences utilizing up-to-date generalist LLM backbones. For open-book, we explore augmenting our backbone models with MedWiki (Liévin et al., 2023), a collection of 293,593 medical articles from English Wikipedia, built by querying each option of MedMCQA and MedQA against the Wikipedia API.

heading: 5 Results
text: Our core findings are delineated in Table 2. For space reasons, we report key baselines with available results for each benchmark. To ensure complete and impartial evaluations, we also document current state-of-the-art (SOTA) models outside the intersection. We refer the reader to Appendix §E for full benchmark-specific baselines.

heading: 5.1 Artificial grounding effect
text: We direct our attention to the accuracy gap between MEDGENIE models and their non-grounded version (cf. teal-colored scores in Table 2).
Impact on ICL Reader Our framework significantly enhances the performance of all the considered reader models. Specifically, LLaMA-2chat, Zephyr-β, and Phi-3-mini show average improvements of +11.7, +7.8, and +3.6 points, respectively. The impact on MedQA is particularly notable, with improvements of +15.7, +9.8, and +9.6 points, respectively. Artificial grounding further elevates Phi-3-mini, establishing a new SOTA in MedQA and delivering the best overall performance. The combination of MEDGENIE with the strong reasoning capabilities and extended context window of Phi-3 fully unlocks the potential of this lightweight model. The results for LLaMA-2-chat and Zephyr-β indicate that the advantages of MEDGENIE are particularly prominent in models without prior domain-specific knowledge. In contrast, newer models, such as LLaMA-3, already demonstrate a robust medical background, likely due to their extensive pre-training phase over 15 trillion tokens. As a result, the impact of MEDGENIE is less significant. Few-shot models struggle against finetuned versions on MedMCQA, highlighting the necessity of dedicated training for achieving high scores. Surprisingly, the ICL MEDGENIE models excel in both MedQA and MMLU, outperform-
9883
Model Ground (Source)* Learning Params MedQA MedMCQA MMLU AVG (↓)
ing fully fine-tuned LLMs. Notably, MedGENIELLaMA-2-chat surpasses the fine-tuned LLaMA2-chat by 3 points in MedQA and 2.5 points in MedMCQA. Similarly, MedGENIE-Zephyr-β outperforms the strongest fine-tuned alternative, MEDITRON, by +7.7 points in MedQA and +10.5 points in MMLU. Our approach eliminates the need for expensive full fine-tuning of 7B models, which requires 56GB VRAM when using a standard AdamW optimizer. Instead, ICL MEDGENIE operates efficiently on a standard consumer GPU with 14GB VRAM in half precision.
Impact on FID Reader MedGenie-FID-FlanT5 achieves 53.1% accuracy on MedQA, with outstanding efficiency. It outshines 7B baselines fine-tuned on the train set of target benchmarks, such as MEDITRON (+1.1), PMC-LLaMA (+3.9), LLaMA-2-chat (+3.5), and even CODEX 175B in zero-shot settings (+0.6)—leveraging up to 706× fewer parameters. In MedMCQA, it significantly exceeds ICL models, such as Zephyrβ (+8.7), Mistral-Instruct (+11.9), and LLaMA-2chat (+17.1). It remains superior to fully fine-tuned PMC-LLaMA (+0.7) and the zero-shot CODEX 175B model (+1.2), but struggles to compete with previous SOTA holders: MEDITRON (-7.1) and VOD (-6.2). We motivate this gap with the huge difference in size and pre-knowledge, other than
the more notional nature of the MedMCQA questions, where MedWiki grounding is highly effective. In MMLU, it again shows exceptional performance, outperforming all fine-tuned baselines on the MedMCQA train set. This highlights the generalization power of MEDGENIE compared to larger and more costly LLMs. Zooming out, our findings corroborate the tiny titan nature of Flan-T5 observed by other researchers (Fu et al., 2024).

heading: 5.2 Generated vs retrieved context
text: MEDGENIE tops all prior retrieve-then-read solutions across all benchmarks except for MedMCQA, where it is only rivalled by VOD. Generated contexts yield higher accuracy compared to Wikipedia and textbook chunks, as well as UMLS subgraphs. Impressively, artificial augmentation applied to LLaMA-2-chat results in +10 points (avg) compared to MedWiki augmentation.
Reranking Preference A crucial aspect of our framework is to ensure top-tier contexts. However, the use of a 13B model as the context generator raises concerns about its ability to deliver adequate quality contexts, especially compared to stronger closed and open-source alternatives such as GPT-4 and LLaMA-70B. These concerns arise from the risk of hallucinations or less accurate text compared to factual information stored in knowledge
9884
bases. Human evaluation is worth exploring, but it would require considerable resources and expertise, which may not be readily accessible in our case. Hence, we propose using a SOTA reranker, BGElarge (Xiao et al., 2023),9 to determine whether our artificial contexts are deemed more relevant than the top contexts (chunks) fetched from MedWiki. Given a question q, we take the 5 multi-view contexts C generated by PMC-LLaMA. Next, we retrieve the 10 MedWiki contexts most similar to q (PubMedBERT (Gu et al., 2022), cosine similarity). Subsequently, we create 15 q–context pairs to feed the reranker, registering their relevance score. Finally, we apply Recall@K to quantify the % of generated (relevant) contexts in the top-K positions according to the reranker. Figure 4 portrays the results. The reranker strongly favors our artificial contexts over the retrieved ones across all three benchmarks. Precisely, we achieve a Recall@1 of 91%, 98%, and 96% on MedQA, MedMCQA, and MMLU, respectively. Option-focused contexts contribute predominantly to this preference. However, as K rises, our recall begins to diminish, and the significance of option-free contexts amplifies, thus advocating for our multi-view approach.
Artificial Contexts for Retrieval Augmented Generation Our analysis delves deeper into assessing the efficacy of synthetic contexts by implementing them within a standard retrieve-thenread pipeline. The objective is to discern whether their combination with factual data can improve overall performance. We analyze two different settings. First, we supplement MedWiki with artificial contexts from the MedQA test set and the
9BAAI/bge-reranker-large
MedMCQA eval set. We assess whether a small portion of synthetic data may impact a much larger body of factual data. Second, we broaden the knowledge base to cover contexts generated from the MedQA and MedMCQA train set questions. Although such contexts may not pertain to the benchmark test sets, they might offer additional insights to improve accuracy. For each setting, we evaluate retrieval-augmented ICL readers on all three benchmarks. We refrain from considering MMLU contexts to ensure a complete out-ofdomain validation, enabling us to understand the validity of artificial passages when obtained from a different domain. The results are pointed out in Table 3. Mixed generation+retrieval knowledge sources consistently improve accuracy on all benchmarks, except Mistral-Instruct on MMLU. Combined with Zephyr-β, we obtain 67.4, pushing the SOTA on MMLU. Despite these positive gains, scores fall short in competing with the generatethen-read paradigm of MEDGENIE. We attribute this discrepancy to the retriever influence and the inevitable degradation deriving from segmentation.
RAGAS Evaluation When tasked with generating extensive medical content, the context genera-
9885
tor may produce significant hallucinations. Demonstrating a preference for generated contexts over those retrieved from MedWiki might not be sufficient, as the preference could be attributed to coherence rather than factual accuracy. To validate our findings further, we evaluate both generated and retrieved contexts using the RAGAS library (Es et al., 2024). RAGAS offers various reference-free metrics for RAG pipelines, following the LLM-asa-judge paradigm. Particularly we focus on:
• Context Recall: Measure the extent to which the retrieved/generated context aligns with the ground-truth (GT) answer. Each sentence in the GT answer (generally one) is analyzed to determine whether it can be attributed to the retrieved context or not.
CR = |GT sentences linked to the context|
|GT sentences| (4)
• Context Precision: For each chunk in retrieved/generated context, check if it is relevant or not relevant to arrive at the GT answer for the given question. Ideally, all relevant chunks should appear in the highest ranks.
CP@K =
∑K k=1 P@k · vk
|Relevant chunks in the top-K| (5)
P@k = true positives@k
(true positives@k + false positives@k) (6)
where K is the total number of chunks in the context and vk ∈ {0, 1} is the relevance indicator at rank k.
• Faithfulness: The factual consistency of the generated answer is evaluated by comparing its claims against the provided context to determine if they can be accurately inferred.
F = |Generated answer claims implied by the context|
|Generated answer claims| (7)
Each metric gives score in a [0, 1] boundary; the higher, the better. We utilize gpt-4-turbo-2024-04-09. Due to the high API costs, we limit our evaluation to a sample of MedQA, the most relevant testbed in our study. Table 4 reports the performance of Zephyr-β on 150 randomly selected instances, where both the generated and retrieved contexts direct the LLM to produce correct answers. We also apply RAGAS to 50 random instances where both sources lead to wrong answers. The results reaffirm the superiority of artificial contexts, achieving up to +27.2 CR, +39.3 CP , +35.9 F .

heading: 6 Conclusion
text: This paper introduces MEDGENIE, a novel fullygenerative framework for medical ODQA, with a particular emphasis on resource-constrained environments. Through comprehensive experimentation with three standard medical benchmarks, MEDGENIE demonstrates substantial performance improvements over the existing closedbook and open-book methodologies. By injecting multi-view contexts from a medical LLM via ICL or lightweight fine-tuning, MEDGENIE achieves new SOTA results, even surpassing expensive finetuned LLMs or the largest zero-shot models with up to 706× fewer parameters. Furthermore, our research highlights the effectiveness of artificial passages in enhancing retrieve-then-read workflows, showcasing the potential of generated contexts to surpass or assist retrieved counterparts. From a wider angle, MEDGENIE represents a promising approach to address the intricate challenges of medical ODQA, laying the foundation for future advancements in the field.
Ethical Statement
We honor and support the ACL Code of Ethics. LLMs offer significant social benefits, but also pose potential risks. Safety and trustability are paramount in the medical domain. This paper uses knowledge embedded within LLM parameters to tackle open-domain questions. Unlike retrievethen-read methods reliant on curated external corpora, using models to generate contextual documents may inadvertently amplify inherent biases and deviate from clinical and societal norms, potentially leading to the dissemination of convincing medical misinformation. Therefore, we advocate for a cautious approach, recommending manual scrutiny of any output by domain experts before real-world utilization. This ethical precaution is vital to avoid disseminating potentially erroneous
9886
or misleading information, especially in the clinical and scientific communities.
Limitations and Future Work
Despite achieving SOTA performance on the benchmarks presented, our MedGENIE pipeline has several limitations that warrant attention. By replacing the retrieval component with a generative one, we lose the ability to update the knowledge state, which remains frozen to that of the LLM training time. This stands in contrast to the standard retrieve-then-read approach, which allows for the incorporation of new documents as new information becomes available. In a rapidly evolving field like medicine, this ability to quickly adapt by adding temporally recent documents or documents from new domains is important to cope with scientific information overload (Landhuis, 2016). In our method, the responsibility of retaining all knowledge rests entirely on the LLM, and incorporating new knowledge would probably require retraining the context generator.
Our method generates context for any given question, even when the medical LLM lacks knowledge. This may produce a noisy context with inaccurate or irrelevant information. We retain all retrieved contexts without relevance filtering, a strategy that could notably enhance overall performance if implemented. Moreover, the efficacy of our ICL pipeline hinges on the reader’s ability to process long prompts within the context window, which may not always be the case for every LLM. This limitation can impact results, particularly when dealing with complex or verbose contexts.
Recent literature suggests solutions for these limitations. Zhang et al. (2024) introduced Retrieval Augmented Fine Tuning (RAFT), a training approach that enhances the model’s ability to answer questions in an open-book in-domain setting. RAFT trains the model to disregard irrelevant documents when given a question and a set of retrieved documents. Labruna et al. (2024) propose ADAPTLLM, teaching an LLM to generate a <RET> token when it does not know the answer, triggering retrieval only when necessary. Interestingly, our work suggests potential improvements for both approaches. For RAFT, it might be interesting to explore whether fine-tuning can benefit from augmentation with artificially generated contexts. For the ADAPT-LLM approach, introducing a second token, <GEN>, could be valuable when the retrieved
context does not contain the answer, prompting the generation of a context from an auxiliary LLM.
Previous work, such as GenRead (Yu et al., 2023), has explored the zero-shot pipeline using the same model for both generator and reader, following a self-distillation approach. This suggests that a model may indeed benefit from directly extracting knowledge embedded within its own parameters. We acknowledge the significance of assessing the performance of our generator, PMC-LLama13B (awq), in the role of a reader. Nonetheless, we chose not to pursue this evaluation due to the model’s restricted context window of merely 2,000 tokens. Expanding the input prompt with artificial contexts would rapidly exhaust this limited context capacity. However, with recent advancements in LLMs that possess a stronger medical background and an extended context window of 8K tokens (e.g., LLaMA 3 10, OpenBioLLM 11), this approach warrants further exploration as future work.
Finally, evaluating the effectiveness of multipleround reasoning coupled with context generation, as opposed to other methodologies such as multiplehop reasoning with retrieval (e.g., DSP Khattab et al. (2022)), could provide valuable insights into optimizing QA systems for the complexities of medical domain tasks.

heading: Acknowledgements
text: This research is partially supported by (i) the Complementary National Plan PNC-I.1 "Research initiatives for innovative technologies and pathways in the health and welfare sector" D.D. 931 of 06/06/2022, DARE—DigitAl lifelong pRevEntion initiative, code PNC0000002, CUP B53C22006450001, (ii) the PNRR—M4C2— Investment 1.3, Extended Partnership PE00000013, FAIR—Future Artificial Intelligence Research, Spoke 8 "Pervasive AI," funded by the European Commission under the NextGeneration EU program, (iii) the European Commission and the Italian MIMIT through the Chips JU TRISTAN project (G.A. 101095947). We thank Andrea Zecca, Antonio Lopez, Matteo Vannucchi, and Stefano Colamonaco for their valuable support in ablation studies with retrieve-then-read pipelines.
10https://www.meditron.io/ 11aaditya/Llama3-OpenBioLLM-8B
9887

heading: B Context Generation Insights
text: B.1 Human Evaluation of Context Generator
We compare three LLMs as candidate generators, two medical (PMC-LLaMA-13B, BioMedGPT-7B) and one generalist (LLaMA-2-Chat-13B), by enlisting the expertise of five external medical professionals with NLP competencies. We exclude MEDITRON due to the release of pre-trained checkpoints only. Motivated by (Fabbri et al., 2019; Huang et al., 2023; Narayan et al., 2018), we use a direct comparison strategy, proven to be more reliable, sensitive, and less labor-intensive than rating scales. We consider a pool of 100 questions sampled in a balanced manner from MedQA, MedMCQA, and MMLU. For each, experts are presented with contexts inferred by 2 out of 3 LLMs. We generate one context for question, allocating 60% of the evaluation to option-focused and 40% to optionfree. In order to have also insights about the quality of the contexts generated, we ask experts to select the best all-around context considering relevance, factuality and succinctness, optionally providing a text comment; a “tie” is declared if an expert perceives the two contexts as of equal quality. This approach inherently assesses context quality, with high-quality contexts being those that fully satisfy our evaluation criteria. Finally, the total number of mandatory preference labels per participant is 300. We randomize the order of pairs and context sources per question to guard the rating against being gamed. The final score of each LLM is the percentage of times that its contexts are selected as the best, minus the percentage of times they are not. The results indicate PMC-LLaMA as the best choice, garnering 41% preference, surpassing LLaMA-2-Chat-13B with 36% and BioMedGPT7B with 33%. The average Kendall’s τ coefficients among all inter-annotator agreements is 0.72. Table 5 provides an annotation example.
B.2 Artificial Context Distributional Statistics
Figure 5 plots the length distribution of the artificial contexts used in this study, focusing on the test sets of our benchmarks (eval set for MedMCQA). We tokenize contexts into words using NLTK (Bird and Loper, 2004). In summary, the average number
9892
of words for MedQA-4opt, MedQA-5opt, MedMCQA, and MMLU is 207, 207, 203, and 202, respectively. The maximum number of words is 442, 452, 460, and 453.
B.3 Number and Type of Generated Contexts
Table 6 displays the tangible impact of incorporating option-free contexts alongside option-focused contexts. Consistently, this integration enhances the performance of every model considered across all benchmarks, with the total number of contexts playing an essential role. We observe that given more contexts, our framework generally achieves greater performance. Particularly noteworthy is the experiment conducted on MedQA with the ICL MEDGENIE models. Here, we demonstrate that employing 3 option-focused + 2 option-free contexts yields superior accuracy compared to using only 5 option-focused contexts. Specifically, there’s an accuracy enhancement of +1.4 and +0.6
for MedGENIE-LLaMA-2-chat and MedGENIEZephyr-β, respectively.
C In-Context Learning Sample Selection
Selecting appropriate samples for the prompts is essential to generate high-quality contexts. Recent studies suggest strategies that aim to uncover more effective contextual demonstrations by retrieving examples that closely match the semantics of each input test (Liu et al., 2022). However, we are constrained by the need to first consider each demonstration as a context–question pair, rather than solely focusing on the question. Secondly, we need to ensure that the context is not too long to avoid (i) saturation of the context window when appending the new test input, which is the result of concatenating five other contexts; (ii) the "lost in the middle" phenomenon; (iii) excessive increase in inference costs. As a result, we employ a strategy similar to that pursued by Chen et al. (2023) for
9893
MEDITRON, adapted based on the characteristics of the reader model considered. For models with lower context window capability, such as Zephyr-β and LLaMA-2-chat, we follow these steps:
• We sample three questions from the training set and generate a context with PMC-LLaMA for each of them.
• We ensure accurate filtering of only relevant content from the generated text.
• Additionally, we involve a medical expert to craft pairs of questions with relevant contexts for each benchmark considered.
• Consequently, we end up with four different pairs of shots for each model in each benchmark.
• We consider two shots as in-context demonstrations.
Tested pairs for MedQA and MedMCQA are documented in Table 7 and Table 8, respectively. Additionaly, Table 9 presents how the performances of each ICL MEDGENIE model are affected by each pair, unveiling the accuracy effectiveness of artificial examples.
Finally, For LLMs with higher context window capacity, such as LLaMA-3-instruct and Phi-3mini-instruct, our approach is slightly different:
• We sample three questions from the training set and generate a context with PMC-LLaMA for each of them.
• We consider only one long-context artificial shot as in-context demonstration.
This refined strategy ensures effective handling of context constraints while generating accurate and informative demonstrations for each model and benchmark scenario.

heading: D Clustering-Based Prompting
text: The cluster-based methodology by Yu et al. (2023) can be detailed as follows:
1. ask an LLM to generate a background context for each question in the training split (alternatively, we can retrieve a context from a knowledge source), obtaining a set of question– context pair;
2. perform inference with the LLM by leveraging the generated context, and maintain from
9894
9895
the initial set of pairs only those for which the LLM answered correctly;
3. encode each question–context pair from the filtered set;
4. use K-means to cluster all the embedding vectors obtained into K classes, where the num-
ber of classes matches the number of documents that need to be generated eventually;
5. randomly select n question–context pairs from each cluster;
6. present each different n question–context pairs to the LLM as in-context demonstra-
9896
tions for generating a context from a given test question.
This approach ensures that the LLM is exposed to different distributions of examples, resulting in generated contexts that cover various (uncontrolled)

heading: E Full Baselines
text: In Table 11, Table 12, and Table 13, we present a fine-grained comparison of MedGENIE models with closed and open-book alternatives known in the literature, both fine-tuned and in zero/few-shot modes. It should be noted that the baseline results provided by Liévin et al. (2022) for MedMCQA should only be regarded as a reference point, as they relate to a subset of 1K samples from the eval set.

heading: F Robustness
text: F.1 Sensitivity to Question Subject
In Figure 6, we present a detailed analysis of the per-subject performance of the MEDGENIE models across MedMCQA (21 medical subjects) and MMLU (9 health and biology subjects). For MedMCQA, the analysis is done in comparison with our baselines: Zephyr-β and LLaMA-2-chat in 2- shot prompting. For MMLU, we extend the analysis to known results from MEDITRON, PMCLLaMA, and VOD, all fine-tuned over the train set of MedMCQA. In this case, we consider Zephyr-β in 3-shot prompting (Chen et al., 2023) due to its
9897
higher accuracy. Please note that the subcategory professional_medicine of MMLU corresponds to the MedQA questions.
F.2 Sensitivity to Option Order
Table 14 illustrates the frequencies of the predicted answers after option shuffling with 10 distinct random seeds, testing MedGENIE-FID-Flan-T5 in all benchmarks. Note that our seed base is 0. The results highlight a classification bias. In MedQA, the model is inclined to default the last option, discouraging the first, while in MedMCQA, it underestimates D. Similarly, Table 15 examines the robustness of the model to shuffled options with an unsupervised pipeline. Shuffling can impact MedGENIE-Zephyr-β up to 4.4 accuracy points, and MedGENIE-LLaMA-2-chat up to 5.9, registering more oscillations in MMLU.

heading: G Artificial Context Augmentation for RAG
text: Table 16 shows the detailed results of our experiments aimed at evaluating the effectiveness of combining artificial data with factual data sourced from MedWiki. We compare two approaches: (i) a standard RAG pipeline, where we retrieve the top k=5 relevant contexts for a given query, and (ii) a modified approach where we retrieve the top k=10 most relevant contexts and rerank them to consider only the top-5 reranked contexts.
Both artificial context and MedWiki data are segmented into chunks using the RecursiveCharacterTextSplitter with parameters chunk_size=1000 and chunk_overlap=200 from LangChain. Each chunk is encoded using pubmedbert-base-embeddings.14 The embeddings of each chunk are stored using Milvus as vectorDB. To determine the similarity between vectors, we employ cosine similarity. Additionally, we use BGE-large15 as reranker.

heading: H Relation between Context Window and Context Length
text: To ensure the feasibility and effectiveness of our approach, we implement measures to maintain manageable length of generated contexts. Primarily, our methodology involves crafting optimized prompts
14https://huggingface.co/NeuML/ pubmedbert-base-embeddings
15https://huggingface.co/BAAI/ bge-reranker-large
and selecting relevant shot examples to guide the LLM, as illustrated in Figure 7 and Figure 8. This approach combines prompt engineering techniques with expert insights from medical professionals. Additionaly, ss detailed in Section A, we establish a maximum generative hyperparameter of 512 new tokens, resulting in a maximum context length of 2560 tokens when concatenated with four other contexts. This deliberate design choice ensures compatibility with the 4096-token context window of both LLaMA-2 and Zephyr-β models. By avoiding longer context lengths, we prevent exceeding the maximum context length in a significant proportion of test samples, particularly considering the question lengths in benchmarks like MedQA. By imposing these constraints, we guarantee that our models can effectively process and generate responses within the specified bounds.

heading: I MedWiki as Knowledge Base for RAG
text: While we acknowledge that PubMed 16 and similar sources offer a more comprehensive repository of medical knowledge, it is important to consider practical constraints, especially in low-resource settings. Hosting such extensive datasets can be prohibitive in terms of infrastructure and cost. While exploring API services is possible, they would lack the control and flexibility needed for an optimized retrieval pipeline. It is worth noting that previous work, notably by Liévin et al. (2023), has demonstrated the quality and effectiveness of MedWiki as a retrieval corpus. As Table 12 shows, MedWiki-augmented VOD achieves SOTA accuracy scores for openbook models on MedMCQA, largely surpassing other small pre-trained language models like PubmedBERT, SciBERT, and BioBERT augmented with the full PubMed dataset. One reason for this lies in the type of knowledge required to achieve high results on MedQA, MedMCQA, and MMLUMedical benchmarks. In fact, the resolution of the targeted questions benefits particularly from the use of broad notional knowledge, rather than chunks of text extrapolated from highly specialized scientific publications, which are not always directly transferable and generalizable to other contexts, risking to introduce noise. To further validate our choice of MedWiki as the knowledge base, we conducted additional experiments using Textbooks (the medical text corpus attached to MedQA) as an alternative knowledge base for RAG. As shown
16https://pubmed.ncbi.nlm.nih.gov/
9898
in Table 11, the results demonstrate that LLMs augmented with Textbooks only exhibit a marginal increase in performance with respect to WedWiki, corroborating the effectiveness of the latter. In light of these findings, we opted to maintain MedWiki as the knowledge base for all benchmarks, thereby providing a common reference point and ensuring consistency across our experiments.

heading: J Qualitative Examples
text: Table 17 and Table 18 showcase qualitative examples of contexts generated by PMC-LLaMA on MedQA and MedMCQA, respectively. Concept links are highlighted with colors, while text spans that are highly effective in deducing the correct answer option are bolded.
9899

heading: K Prompt Templates
text: Figure 7 and Figure 8 report the constructed prompts for option-focused and option-free artificial context generation, with expert-crafted examples. Figure 9 and Figure 10 illustrate the prompt template used to perform 1-shot inference on MedQA with LLaMA-3-instruct and Phi-3-miniinstruct for artificial grounding (MedGENIE). Figure 11 and Figure 12 show the prompt template used to perform 2-shot inference on MedQA with Zephyr-β and LLaMA-2-chat for both artificial grounding and MedWiki grounding. Figure 13, Figure 14 and Figure 15 instead show how Zephyr-β, LLaMA-2-chat and PMC-LLaMA perform 2-shot inference on MedQA without grounding, respectively. These prompt templates are also applicable to MedMCQA and MMLU, with adjustments made by interchanging the provided 2-shot examples. Figure 16 and Figure 17 display the templates adopted with Zephyr-β and LLaMA-2-chat for performing 2-shot inference with either artificial grounding (MedGENIE) or MedWiki grounding on MedMCQA. The templates utilized for MMLU are presented in Figure 18 and Figure 19 instead.
9900
9902
9903
9904
9905
9906
9908
9909
9910
9911
9912
9913
9914
9915
9916
9917
9918
9919

heading: Future Work
text: 3, Extended Partnership PE00000013, FAIR—Future Artificial Intelligence Research, Spoke 8 "Pervasive AI," funded by the European Commission under the NextGeneration EU program, (iii) the European Commission and the Italian MIMIT through the Chips JU TRISTAN project (G. A. 101095947). We thank Andrea Zecca, Antonio Lopez, Matteo Vannucchi, and Stefano Colamonaco for their valuable support in ablation studies with retrieve-then-read pipelines. 10https://www. meditron. io/ 11aaditya/Llama3-OpenBioLLM-8B
9887