Venue: NAACL 2025

# SimRAG: Self-Improving Retrieval-Augmented Generation for

Adapting Large Language Models to Specialized Domains

 Ran Xu\({}^{1,2}\) , Hui Liu\({}^{2}\), Sreyashi Nag\({}^{2}\), Zhenwei Dai\({}^{2}\), Yaochen Xie\({}^{2}\), Xianfeng Tang\({}^{2}\), Chen Luo\({}^{2}\), Yang Li\({}^{2}\), Joyce C. Ho\({}^{1}\), Carl Yang\({}^{1}\), Qi He\({}^{2}\)

\({}^{1}\) Emory University  \({}^{2}\) Amazon

{ran.xu,joyce.c.ho,j.carlyang}@emory.edu, liunhu@amazon.com

Work done during an internship at Amazon.

###### Abstract

Retrieval-augmented generation (RAG) enhances the question answering (QA) abilities of large language models (LLMs) by integrating external knowledge. However, adapting general-purpose RAG systems to specialized fields such as science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To tackle this, we propose SimRAG, a self-training approach that equips the LLM with joint capabilities of question answering and question generation for domain adaptation. Our method first fine-tunes the LLM on instruction-following, question-answering, and search-related data. Then, it prompts the same LLM to generate diverse domain-relevant questions from unlabeled corpora, with an additional filtering strategy to retain high-quality synthetic examples. By leveraging these self-generated synthetic examples, the LLM can improve their performance on domain-specific RAG tasks. Experiments on 11 datasets, spanning two backbone sizes and three domains, demonstrate that SimRAG outperforms baselines by 1.2%-8.6%.

## 1 Introduction

Retrieval-augmented generation (RAG) Lewis et al. (2020); Gao et al. (2023); Gutierrez et al. (2024); Asai et al. (2024) is a powerful technique that enhances large language models (LLMs) for various knowledge-intensive tasks such as question answering (QA) by incorporating external knowledge sources. This method not only customizes responses to handle long-tail knowledge but also avoids the need for costly model retraining Ovadia et al. (2023). Additionally, RAG helps reduce the issue of LLM hallucination by ensuring responses are grounded in relevant evidence Shuster et al. (2021), thereby improving the overall accuracy and reliability of LLM outputs.

While extensive research has focused on developing effective Asai et al. (2024); Lin et al. (2024); Liu et al. (2024) and efficient Xu et al. (2024) RAG systems for general-domain QA tasks, adapting RAG to specialized domains for LLMs poses significant challenges. These models often struggle with distribution shifts and fail to accurately extract information from domain-specific contexts Miller et al. (2020); Liu et al. (2022). Moreover, directly using black-box LLMs OpenAI (2023); Anthropic (2023); Wang et al. (2023b) in specialized domains raises concerns about privacy when dealing with sensitive proprietary data. It is essential to fine-tune LLMs on domain-relevant QA tasks to unlock the full potential of LLM-based RAG systems in specialized domains.

Despite the critical need for domain-specific fine-tuning, the primary challenge lies in the acquisition of high-quality fine-tuning data towards RAG applications. Prior works rely on continuous pretraining Chen et al. (2023); Zhang et al. (2024a) on specialized corpora or fine-tuning on domain-specific instruction-tuning data Wu et al. (2024); Wadden et al. (2024). However, the mismatch between these general-purpose tasks and domain-specific QA hinders their effectiveness. More recently, several approaches Liu et al. (2024); Schimanski et al. (2024); Zhang et al. (2024c) use synthetic data from powerful LLMs (e.g., GPT-4) to create QA finetuning datasets. While promising, these methods are costly, inefficient, and lack explicit quality control over the generated outputs. Additionally, the direct use of proprietary corpora with black-box LLMs introduces privacy concerns, making these methods unsuitable for sensitive domains.

To tackle the data scarcity issue mentioned above, we propose SimRAG1, a self-improving approach to harness the LLMs' own capabilities to generate pseudo-labeled data for domain adap

tative question answering. Our method is inspired by the success of self-training in LLM development, where models are refined using synthetic examples generated from unlabeled corpora Wang et al. (2022); Li et al. (2024). However, for RAG applications, special considerations are needed to adapt LLMs for generating questions that require external context to answer. The core objective of SimRAG is to fine-tune a single LLM to perform two complementary tasks: _question answering with context_ and _question generation from context_. Both tasks involve extracting and summarizing relevant information from the context, allowing them to mutually reinforce each other.

Specifically, we design a two-stage procedure to adapt LLMs for domain QA, we first fine-tune LLMs on _instruction-following_, _question answering_, and _question generation_ data from general-domains. This step equips LLMs with basic instruction-following and context utilization skills. Then, to specialize the model for domain-specific tasks, we then harness unlabeled domain corpora, prompting the same LLM to generate high-quality QA pairs grounded in the context of these specialized domains. To further _enhance the quality_ of synthetic pairs, we incorporate multiple task types to improve the model's generalization capabilities, combined with round-trip consistency filtering technique Bartolo et al. (2021) to preserve generated QA pairs only when the original context is retrieved among top results. With these pseudo-labeled (_question_, _passage_, _answer_) tuples generated by LLMs, we continuously fine-tune the models with those synthetic examples. This pipeline allows the LLM to progressively refine its output on synthetic pairs, thus adapting itself towards domain-specific QA applications.

We conduct experiments on three different domains spanning from biomedical, natural/social sciences, and computer science (CS), where we observe SimRAG consistently achieve better performance than other domain-specific LLMs and general-domain retrieval-augmented LMs. Qualitative studies highlight the benefits of joint training in question answering and generation, along with diverse, denoised QA pairs.

Our contribution can be summarized as follows:

* We propose SimRAG, a RAG framework that enhances LLM's capability for question answering on specialized domains.
* We design a novel instruction fine-tuning approach that enables LLMs to perform both question answering and question generation. This joint capability facilitates self-improvement through self-training on generated synthetic data, leading to enhanced model performance.
* We validate our approach with empirical studies across 11 datasets from three distinct domains, demonstrating that SimRAG outperforms baseline models by 1.2%-8.6%.

## 2 Related Work

**Retrieval-augmented generation.** RAG has emerged as a powerful tool in knowledge-intensive NLP tasks such as language modeling Borgeaud et al. (2022) and question answering Lewis et al. (2020); Shi et al. (2024). The typical approach involves integrating a retriever with the LLM generator and designing a fine-tuning process to align the retriever with LLM capabilities. To further refine RAG, recent research explored various enhancements. These include developing dynamical retrieval processes to refine the relevance of fetched content Jiang et al. (2023); Jeong et al. (2024); Su et al. (2024), and filtering out irrelevant contexts to robustify RAG Yoran et al. (2024); Yu et al. (2024, 2023); Wang et al. (2024). Additionally, several studies have developed instruction-tuning methods aimed specifically at improving search and RAG capabilities of LLMs Liu et al. (2024); Lin et al. (2024); Dong et al. (2024); Wei et al. (2024).

**Self-training.** Self-training (or Pseudo-Labeling) is one of the earliest approaches to semi-supervised learning Rosenberg et al. (2005). The method uses a teacher model to generate new labels on which a student model is fitted. Self-training has been widely adopted for various NLP tasks including text classification Du et al. (2021), natural language understanding Vu et al. (2021) and ranking Wang et al. (2022). Recently, the idea of self-training has also been applied to LLM instruction fine-tuning Yuan et al. (2024); Li et al. (2024), reasoning Pang et al. (2024), and alignment Gulcehre et al. (2023), yet to the best of our knowledge, this pipeline has not been widely explored for RAG applications. The major drawback of self-training is that it is vulnerable to label noise Arazo et al. (2020). There are several approaches to stabilize the self-training, with sample selection Li et al. (2024) and reweighting Wang et al. (2021) strategies.

**Domain-specific LLMs.** Most domain-specific LLMs rely on continuous pretraining Labrak 

et al., 2024; Chen et al., 2023; Xu et al., 2024b) or domain-specific fine-tuning Wu et al. (2024); Zhang et al. (2024a, 2023); Wadden et al. (2024); Shi et al. (2024b), with little focus on adapting models for domain-specific RAG settings. Relevant works Zhang et al. (2024c); Schimanski et al. (2024) use strong GPT models for synthetic data generation in RAG scenarios. In contrast, SimRAG leverages the same LLM for both question generation and answering, enabling self-improvement and offering a more cost-effective approach for adapting LLMs to domain-specific QA tasks.

## 3 Methodology

### Problem Setup

In a RAG problem, we aim to generate answers for queries based on a set of supporting documents or contexts. Specifically, for a query \(q\), an retriever \({\cal R}\) is utilized to retrieve top-\(k\) most relevant contexts \({\cal D}=\{d_{1},d_{2},...,d_{k}\}\) from a large corpus \({\cal C}\). The LLM \({\cal M}_{\theta}\) then generates an answer \(a\) to the query \(q\) based on the retrieved context \({\cal D}\).

In this work, we aim to improve the LLM's QA capability in RAG system towards specialized domains where only unlabeled corpus \({\cal C}\) is available. As shown in Figure 1, our approach first learns from retrieval-oriented instruction data in the general domain in Stage-I and then augments \({\cal T}\) with pseudo-labeled \({\cal T^{\prime}}=(q^{\prime},{\cal D^{\prime}},a^{\prime})\) tuples in Stage-II, where \({\cal D^{\prime}}\) is sampled from the specialized domain \({\cal C}\) for self-training. The overall objective of our study is to adapt the LLM \({\cal M}_{\theta}\) to specialized domains with \({\cal T}\cup{\cal T^{\prime}}\).

### Stage-I: Retrieval-oriented fine-tuning

To start with, we leverage instruction fine-tuned LLMs as the backbone (e.g. meta-llama/Meta-llama-3-8B-Instruct). Although these models have been instruction finetuned, they still exhibit a deficiency in leveraging context information to answer domain-specific questions. To improve their abilities on knowledge-intensive tasks, we fine-tune the LLM with retrieval-oriented tasks. Specifically, we follow Lin et al. (2024); Liu et al. (2024) and leverage the training data blend that consists of the following components:

(1) **General Instruction Fine-tuning (SFT) data.** To help maintain the models' ability to comprehend and follow instructions, we leverage the SFT data including OpenAssistant Kopf et al. (2023), Dolly Conover et al. (2023), SODA Kim et al. (2023), ELI5 Fan et al. (2019), Self-Instruct Wang et al. (2023), and Unnatural Instructions Honovich et al. (2022). Note that we make sure there is no overlap between SFT data and test data from target tasks.

(2) **General domain Context-aware QA data.** To bolster the LLMs' general RAG skills of generating accurate answers grounded in relevant contexts, we fine-tune them on a diverse array of general domain question-answering datasets. This includes DROP Dua et al. (2019), NQ Kwiatkowski et al. (2019), Squad Rajpurkar et al. (2016), NarrativeQA Kocisky et al. (2018), Quoref Dasigi et al. (2019), ROPES Lin et al. (2019), OpenbookQA Mihaylov et al. (2018), LogiQA Liu et al. (2020), TAT-QA Zhu et al. (2021), We

Figure 1: Two-stage fine-tuning framework for our proposed method SimRAG. The model is first fine-tuned on retrieval-related data. Then, it generates pseudo-labeled tuples by first extracting candidate answers from the corpus, and then generating candidate questions conditioned on both document and answer. The LLM is further fine-tuned on pseudo-labeled examples filtered with round-trip consistency.

 

bGLM Liu et al. (2023), StrategyQA Geva et al. (2021), BoolQ Clark et al. (2019), FaVIQ Park et al. (2022) and FEVER Thorne et al. (2018) datasets, where for each sample, a query \(q\) and its relevant context \({\cal D}\) is given, and the LLM is trained to generate answer \(a\) to the query.

(3) **General Retrieval-related Data:** To better generate high-quality pseudo-labeled QA samples in the next stage, we incorporate retrieval-related data to improve two specific skills of LLMs: (a) _Answer Generation_: where a grounding document is given, and the LLMs are trained to generate candidate spans from the context that are likely to be answers to some questions. In this part, we incorporate Squad 1.1 and 2.0 versions Rajpurkar et al. (2016), DROP Dua et al. (2019) and WebQuestions Berant et al. (2013) datasets. (b) _Query Generation_: where an answer and its grounding document are given, and the LLMs are trained to generate a query based on the document and answer. In this part, we leverage NQ Kwiatkowski et al. (2019), Squad 1.1 Rajpurkar et al. (2016), StrategyQA Geva et al. (2021), WebQuestions Berant et al. (2013), FaVIQ Park et al. (2022) and FEVER Thorne et al. (2018) datasets.

The details for each dataset (e.g. the instruct format and the amount of data used) are deferred to Appendix A. For each sample in the fine-tuning dataset, we adopt a standard instruction finetuning objective, computing the loss exclusively on the tokens of the assistant's response.

### Stage-II: Domain Adaptive Fine-tuning

The model after Stage-I is only trained in the general domains. When directly adopting them to specialized applications, the performance can still be suboptimal due to the distribution shift issue Miller et al. (2020). To tailor the LLMs for specialized domains and address the scarcity of labeled data in these areas, we employ a self-training approach leveraging domain-specific unlabeled corpora. This method capitalizes on the model's enhanced capabilities from the previous retrieval-augmented fine-tuning stage. We utilize the fine-tuned LLM to generate pseudo-labeled training samples \({\cal T^{\prime}}=(q^{\prime},{\cal D^{\prime}},a^{\prime})\) by creating queries grounded in the unlabeled text and gathering the corresponding retrieved documents.

Specifically, we conduct a two-step procedure to synthesize additional training data, which corresponds to the two skills learned in Stage-I: (a) _Answer Generation_: for each document \(d_{i}\in{\cal C}\), where \({\cal C}\) is the unlabeled corpus, we prompt our fine-tuned LLMs to generate several candidate spans \(a_{i}^{1},a_{i}^{2},\dots,a_{i}^{m}\) that are likely to be answers to some questions. Formally, the model generates \(a_{i}^{j}\sim p_{\theta}(\cdot|d_{i})\) for \(j=1,\ldots,m\). (b) _Ansverconditioned Query Generation_: for each candidate answer \(a_{i}^{j}\) and its corresponding document \(d_{i}\), we prompt the fine-tuned LLM again to generate candidate questions \(q_{i}^{j}\sim p_{\theta}(\cdot|a_{i}^{j},d_{i})\), with \(a_{i}^{j}\) as the ground truth answer and \(d_{i}\) as the supporting context. This gives us the pseudo-labeled query-answer pair \((q_{i}^{j},a_{i}^{j})\) based on the context \(d_{i}\).

During this process, we adopt two additional strategies, namely _diverse question generation_ and _data filtering_, to further improve the quality of the synthetic pairs. For diverse question generation, we prompt the LLM to create various types of questions, including _short-span question-answering_, _multiple-choice question-answering_, and _claim verification_ tasks. While short-span questions follow the same pipeline as previously described, multiple-choice questions are constructed by using alternative candidate answers from the same unlabeled corpus in step (a) as incorrect options. Claim verification, on the other hand, bypasses the answer generation step; instead, the LLM generates a claim that can be either supported or refuted by the provided document. By injecting different question types, we prevent the LLM from overfitting to a specific output format and improve the model's generalization ability across different QA tasks.

After generating large amounts of candidate QA pairs, we implement a filtering step to keep only high-quality QA pairs. We define high-quality QA pairs as those that are answerable using the top-\(k\) retrieved contexts. Specifically, we retain only those samples where the ground truth answer \(a_{i}^{\prime}\) is present in the top-\(k\) documents retrieved by a strong retriever, such as Dragon Lin et al. (2023), based on the generated query \(q_{i}^{\prime}\). Formally, the sample is retained if \(a_{i}^{\prime}\in{\cal D}_{i}^{h},\) where \({\cal D}_{i}^{h}\) denotes the top-\(k\) documents retrieved for query \(q_{i}^{\prime}\). From these retained samples, we create pseudo-labeled training tuples \({\cal T^{\prime}}=(q_{i}^{\prime},{\cal D}_{i}^{\prime},{a_{i}^{\prime}}_{i})_{ i=1}^{n}\).

With the created synthetic tuples \({\cal T^{\prime}}\), we augment it with the SFT data \({\cal T_{\text{SFT}}}\) and the general domain context-aware QA data from Stage-I \({\cal T_{\text{gen}}}\), to continuously fine-tune our models, enhancing the LLMs' QA abilities within the specific domain. The size and blending ratio of the pseudo-labeled samples can be found in Appendix 

Experimental Setup

### Tasks and Datasets

We evaluate our model across a total of 11 datasets spanning the _medical_, _scientific_ and _computer science_ domains. For the medical domain, we include the five datasets in the MIRAGE benchmark Xiong et al. (2024), including PubMedQA Jin et al. (2019), BioASQ Tsatsaronis et al. (2015), MedQA Jin et al. (2021), MedM-CQA Pal et al. (2022), the medical subsets in MMLU Hendrycks et al. (2021), and two additional open-ended QA datasets LiveQA Abacha et al. (2017), and MedicationQA Abacha et al. (2019). For the scientific domain, we consider ARCH challenge Clark et al. (2018), SciQ Welbl et al. (2017)2, and the scientific subsets (14 subtasks in total) in MMLU Hendrycks et al. (2021). For computer science, we use CS-Bench Song et al. (2024) for evaluation. We distinguish the computer science domain from the broader scientific domain as the scientific domain predominantly covers natural and social sciences, with limited representation of computer science topics. We use accuracy as the evaluation metric for multiple-choice and True-or-False questions, Rouge-L and MAUVE for open-ended questions, Exact Match (EM) and F1 for Fill-in-the-blank questions, with Rouge-L and F1 as the main metrics, respectively. An exception is CS-Bench, where we follow the original paper's evaluation method by using GPT-4 as a judge for fill-in-the-blank and open-ended questions.

Footnote 2: We convert the multiple-choice questions in SciQ into short-phrase answer generation tasks to better assess the model’s generative capabilities.

For the medical domain, we use the corpora from Textbooks Jin et al. (2021), Wikipedia and PubMed articles3 to generate pseudo-labeled samples in Stage-II. For the scientific domain, we leverage Wikipedia. For the CS domain, we use Wikipedia CS Subset4 and arXiv articles5.

Footnote 3: [https://pubmed.ncbi.nlm.nih.gov/](https://pubmed.ncbi.nlm.nih.gov/)

Footnote 4: [https://huggingface.co/datasets/AlaaElInlo/wikipedia_ComuterScience](https://huggingface.co/datasets/AlaaElInlo/wikipedia_ComuterScience)

Footnote 5: [https://huggingface.co/datasets/CCRss/arxiv_papers_cs](https://huggingface.co/datasets/CCRss/arxiv_papers_cs)

### Baselines

We categorize our baselines into four groups: (1) _Off-the-shelf general domain LLMs_, which include GPT-3.5 OpenAI (2022), GPT-4 OpenAI (2023), Llama3-8B-it Meta-AI (2024), and Gemma-27B-it Team et al. (2024). (2) _Off-the-shelf domain-specific LLMs_, including PMC-llama-13B Wu et al. (2024), MEDITRON-70B Chen et al. (2023), AdaptLLM-v2-8B Cheng et al. (2024), BioMistral-7B Labrak et al. (2024) and MedLIama3-8B John Snow Labs (2024) in the medical domain, as well as SciTulu 7B and 70B Wadden et al. (2024) in both the scientific domain and the computer science domain, due to the absence of LLMs specifically fine-tuned for the computer science domain. (3) _General domain retrieval-augmented LLMs_, which include Self-RAG-13B Asai et al. (2024), ChatQA1.5-8B and 70B Liu et al. (2024). (4) _Domain-specific Retrieval-augmented LLMs_, including RAFT Zhang et al. (2024c) and EvidenceRAG6 Schimmski et al. (2024). Since RAFT and EvidenceRAG have not released their checkpoints, we re-implemented their methods using the same backbones as our approach. Note that for all the baseline models, we conduct the zero-shot evaluation and augment the context with retrieval for fair comparison. We also note that we do not compare with several domain-specific baselines such as Zhang et al. (2024b); Nori et al. (2023) which have access to task-specific examples that overlap with our evaluation tasks.

Footnote 6: We named this method ourselves, as the model does not have an officially designated name.

### Implementation Details

We use Llama3-it 8B Meta-AI (2024) and Gemma2-it 27B Team et al. (2024) as our backbones. For the Gemma-2 model, we use LoRA Hu et al. (2022) (\(r=32,\alpha=32\)) during fine-tuning due to resource constraints. For both stages, we set the global batch size to 64, with gradient accumulation as 8 and train the model for 1 epoch. For Stage-I, the learning rate is set to \(5e-7\) and for Stage-II, it is set to \(2e-7\) for the Llama3 backbone and \(5e-7\) for the Gemma backbone. AdamW optimizer Loshchilov and Hutter (2019) is used with \(\beta_{1}=0.9\) and \(\beta_{2}=0.95\). To create context-enhanced examples for our synthetic queries, we use Dragon Lin et al. (2023) to extend context length for SimRAG and baselines, which improves RAG model robustness Yu et al. (2024, 2023). For retrieval during evaluation on medical datasets, we follow the original MIRAGE benchmark by using the top-10 retrieval results as context, ensembled from multiple models. For other datasets, we fetch the top-10 passages by Google Search7. All experiments are conducted on 8 NVIDIA A100 GPUs. The prompt format for answer and question gener 

ation and inference can be found in Appendix E.

## 5 Experimental Results

### Main Results

Table 1, Table 2, and Table 3 present the experimental results for the medical, scientific, and computer science domains, respectively. The results of the 14 tasks in MMLU-sci can be found in Appendix D From the results, we have the following findings:

(1) SimRAG consistently outperforms baselines across these domains and a variety of question-answering formats. In medical, scientific, and computer science domain, the average performance gain is 8.01%, 6.37%, 8.61% over the Llama variant and 1.19%, 3.50%, 3.20% over the Gemma variant, respectively. Besides, SimRAG also achieves

\begin{table}
\begin{tabular}{l|c c c c c c c c} \hline \hline
**Datasets** & **PubMedQA** & **BioaSQ** & **MedQA** & **MedMCQA** & **MMLU-med** & **LiveQA** & **MedicationQA** & **Avg.** \\ \hline
**Metrics** & ACC & ACC & ACC & ACC & Range-L*/ MAUVE & Ronge-L*/ MAUVE & — \\ \hline _Proprietary LIMs, For Reference Only_ & & & & & & & & \\ GPT+3 (OpenAI, 2023) & 67.40 & 90.29 & 66.61 & 58.04 & 75.48 & 42.3 / 62.5 & 36.3 / 46.0 & 62.35 \\ GPT+4 (OpenAI, 2023) & 70.60 & 92.56 & 82.80 & 66.65 & 87.24 & 44.0 / 65.9 & 41.5 / 59.2 & **69.34** \\ \hline _Model LLMs_ & & & & & & & \\ PMCALA [1] & 56.00 & 65.21 & 42.58 & 48.29 & 52.53 & 35.7 / 60.6 & 36.4 / 38.3 & 48.10 \\ MDIPTRON TDR (Chen et al., 2023) & 56.40 & 75.86 & 49.57 & 52.67 & 65.38 & 43.6 / 34.6 & 37.19 \\ AdaptLMv2 8B (Cheng et al., 2024) & 45.00 & 78.80 & 43.13 & 42.74 & 51.24 & 30.2 / 48.0 & 39.2 / 51.4 & 47.19 \\ BioLSTM 78B (Jazeb et al., 2024) & 59.20 & 62.69 & 32.52 & 32.20 & 47.47 & 43.1 / 63.2 & 39.6 / 51.9 & 48.11 \\ MedLlama's 8B (Jollon Snow Labs, 2024) & 74.20 & 35.30 & 61.43 & 61.18 & 77.13 & 27.9 / 45.2 & 29.8 / 35.0 & 59.31 \\ \hline _Revised-Augmented LIMA_ & & & & & & & \\ Self-Add NB [1] & 71.20 & 73.70 & 48.60 & 44.00 & 53.90 & 33.6 / 54.1 & 39.3 / 46.4 & 52.33 \\ ChatDA1S 58B (Liu et al., 2024) & 66.40 & 82.69 & 42.36 & 46.97 & 61.40 & 39.3 / 65.5 & 39.3 / 48.9 & 54.15 \\ ChatDA1S 70B (Liu et al., 2024) & 74.80 & 83.17 & 68.89 & 62.54 & 80.51 & 40.1 / 66.3 & 40.8 / 50.2 & 64.40 \\ \hline \multicolumn{10}{l}{_Backbone: LDAN3-8B-Internet_} \\ \cline{1-1} \cline{2-7}  & Llama’s-8B (Metzian et al., 2024) & 64.60 & 88.51 & 55.30 & 58.91 & 69.79 & 34.1 / 54.1 & 37.2 / 45.6 & 58.34 \\ RATT SB (Zhang et al., 2024) & 73.40 & 88.67 & 54.28 & 60.15 & 70.25 & 36.2 / 55.6 & 38.9 / 56.4 & 60.26 \\ EvidenceRG MB [1] & 75.00 & 90.61 & 57.74 & 61.13 & 72.27 & 36.6 / 57.8 & 34.6 / 53.6 & 61.14 \\ Similar RG MB [1] & **80.00** & 91.75 & 62.92 & **67.51** & 73.57 & **44.46** / 66.69 & 40.1 / 57.5 & **66.84** \\ \hline \multicolumn{10}{l}{_w/o Sage II_} \\ \hline \multicolumn{10}{l}{_Sweene: GENazzi-278-Internet_} \\ \cline{1-1} \cline{2-7}  & Zitnicky [1] & 56.20 & 89.32 & 59.70 & 57.30 & 75.67 & 37.4 / 52.8 & 40.2 / 57.0 & 59.40 \\ BATTER? HBart et al., 2024) & 67.20 & 91.70 & 62.22 & 61.56 & 78.97 & 39.4 / 62.2 & 40.2 / 48.2 & 63.04 \\ EvidenceRG 27B (Schimanski et al., 2024) & 63.00 & 90.61 & 62.14 & 61.80 & 79.43 & 34.5 / 8.6 & 34.5 / 44.6 & 60.85 \\ Simple RG27B & 73.60 & **92.07** & **63.03** & 64.16 & **81.63** & 39.9 / **66.8** & **41.2 / 62.1** & 65.17 \\ Wild Stage III & 66.00 & 91.59 & 62.45 & 58.67 & 72.61 & 37.2 / 61.6 & 40.8 / 58.6 & 62.33 \\ \hline \end{tabular}
\end{table}
Table 1: Results of our proposed method and baselines in the medical domain. All the presented methods use RadG for inference. **Bold** and underline highlight the best and second best performance, respectively. \({}^{*}\): the main metric used for average calculation. \({}^{!}\): models trained using synthetic data generated from GPT-4. \({}^{!}\): our own implementation of the models with the same unlabeled corpora. The notations are the same for the following tables.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline
**Models** & **MC** & **AS** & **FB** & **OE** & **Overall** \\ \hline
**Metrics** & ACC & ACC & Acc & Auto & Auto & — \\ \hline _Proprietary LIMs, For Reference Only_ & & & & & \\ GPT+3 (OpenAI, 2023) & 54.89 & 67.30 & 42.93 & 50.11 & 55.74 \\ GPT+4 (OpenAI, 2023) & 71.48 & 73.62 & 56.87 & 71.43 & **70.34** \\ \hline _Scientific LIMs_ & & & & & \\ Sculletba Wadler et al. (2024) & 38.40 & 56.56 & 27.66 & 32.29 & 40.44 \\ SetTuTba 70B (Widden et al., 2024) & 44.24 & 60.18 & 31.06 & 54.76 & 46.87 \\ \hline _Retension-augmented LIMA_ & & & & & & \\ \hline Sarkd-RGB [1] (Vai et al., 2024) & 29.87 & 54.52 & 30.64 & 24.94 & 34.56 \\ ChiDaN 8B (Liu et al., 2024) & 35.33 & 60.18 & 27.66 & 29.82 & 39.11 \\ ChatDA vno Hau et al. (2024) & 54.94 & 62.67 & 34.89 & 38.53 & 53.07 \\ \hline _Spinkdown Legal-Based-Biformer_ & & & & & & \\ \hline Limasu-SR+(Meta-XL, 2024) & 52.69 & 60.41 & 26.81 & 44.12 & 50.80 \\ **ExtrNT**SB [2] (Zhuang et al., 2024) & 54.57 & 60.86 & 32.76 & 40.23 & 52.38 \\ Extended AG [3] (Shchamanski et al., 2024) & 54.62 & 67.57 & 40.23 & 40.30 & 53.06 \\ BiSoftAQ DB [1] & 60.03 & 64.93 & 38.47 & 47.31 & 57.63 \\ Eui 

comparable performance to strong proprietary models: when using Gemma2-27B as the backbone, we achieve 93.99%, 98.95% and 86.66% of the performance of GPT-4. This demonstrates the effectiveness and robustness of SimRAG in adapting general-domain LLMs to specialized domain knowledge using only unlabeled corpora.

(2) _Domain-specific LLMs_ (e.g. SciTulu and MedLlama), although fine-tuned on relevant data, underperform compared to SimRAG because they are not optimized for RAG tasks, where effectively utilizing retrieved context is crucial. As a result, they struggle to incorporate relevant context into their answers, leading to weaker performance. On the other hand, _general-domain RAG models_ (e.g. ChatQA) face distribution shifts when applied to specialized tasks, as they struggle to integrate the retrieved domain-specific knowledge accurately.

(3) _Domain-specific retrieval-augmented LLMs_ such as RAFT and EvidenceRAG still show sub-optimal performance despite utilizing the powerful (yet expensive) GPT-4 model to generate synthetic training data. In contrast, SimRAG, fine-tuned specifically for the QA generation task, produces more accurate and contextually relevant synthetic QA pairs, leading to better downstream performance across all QA tasks.

(4) Although the CS domain is relatively new and less-studied compared to other natural and social sciences, SimRAG still demonstrates promising performance in this area. This justifies the potential for adapting SimRAG to emerging domains.

### Ablation Studies

**Effect of Stage-I and Stage-II.** Table 1 to 4 show that retrieval-oriented fine-tuning (Stage-I) significantly enhances LLM performance on QA tasks compared to the original backbone, demonstrating its effectiveness. However, further improvements become challenging after this stage. When the LLMs are fine-tuned on self-synthesized training tuples, their performance on target tasks improves even more, with an average increase of 2.21% for Lama and 3.50% for Gemma.. This suggests that, with access to a target domain corpus, LLMs can generate high-quality synthetic data, enabling self-improvement and further boosting performance.

**Effect of Different Retrievers.** We show the performance of SimRAG using Dragon Lin et al. (2023) as the retriever in Table 5. The results show consistent performance improvements of SimRAG over the LLM backbone, demonstrating that SimRAG is robust to different retriever choices and that its self-improvement mechanism consistently enhances performance.

### Study on Pseudo-labeled Tuples

We mainly demonstrate the advantage of SimRAG in generating pseudo-labeled data from the following three perspectives.

**Effect of different question generation models.** To demonstrate the benefit of training on question generation and question-answering data, we compare the performance of Stage-II using different synthetic question-answer pairs. These pairs are generated either directly by Llama-3-8b-it or by an off-the-shelf QG model with T5-Large Rafel et al. (2020) as the backbone. The results demonstrate that our approach achieves better performance on average, demonstrating the clear advantage of leveraging the fine-tuned model itself for pseudo-labeled data generation.

**Effect of question filtering.** We further demonstrate the advantages of question filtering in Figure 3, showing that removing low-quality data not only improves overall model performance but also accelerates the training process. It is also worth noting that even without filtering, SimRAG can achieve strong performance, suggesting that

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Method** & **PubMedQA** & **BioASQ** & **MedQA** & **MedMCQA** & **MLLU-med** & **LiveQA** & **MediationQA** & **Avg.** \\ \hline SimRAG 8B & 80.00 & 91.75 & 62.92 & 67.51 & 75.57 & 44.4 & 40.1 & 66.04 \\ \hline SimRAG w/o general SFT data & 79.60 & 90.78 & 59.47 & 61.92 & 73.09 & 39.9 & 38.9 & 63.38 \\ SimRAG w/o general-domain QA & 80.20 & 91.10 & 61.04 & 65.31 & 72.91 & 42.8 & 39.4 & 64.68 \\ SimRAO w/o general retrieval data & 79.40 & 90.94 & 57.97 & 62.42 & 71.72 & 39.4 & 38.9 & 62.96 \\ SimRAG w/o Stage-I & 76.80 & 89.81 & 57.97 & 60.02 & 70.71 & 39.3 & 38.5 & 61.87 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of SimRAG using ILMA-3-8b-it as the backbone and its variants across medical datasets.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Models** & ** PubMedQA** & **BioASQ** & **MacQA** & **MnDCA** & **NDLL-med** & **Avg.** \\ \hline Llama-38b-4 [20] & 57.00 & 81.55 & 55.30 & 55.16 & 65.53 & 65.07 \\ SimRAG 8B & **79.00** & **84.40** & **60.00** & **63.00** & **74.00** & **73.00** \\ Comma-27b-4 [20] & 58.80 & 89.88 & 57.97 & 55.13 & 26.67 & 67.61 \\ SimRGG-27b-7 & 72.60 & 90.94 & **62.30** & 60.02 & **79.06** & 72.32 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of the 5 datasets from the medical MIRAGE benchmark Xiong et al. (2024), using DRAGON Lin et al. (2023) as an alternative retriever.

 

the synthetic questions generated from fine-tuned LLMs are already highly relevant to the context.

**Effect of diverse question types.** From Figure 3, we observe that SimRAG achieves the best performance when all three different types are included. These results justify the necessity for incorporating different task types into the fine-tuning step in Stage-II. Besides, claim verification benefits PubMedQA and BioASQ more, while multiple-choice questions boost performance on MedQA, MedM-CQA, and MMLU, aligning with the question types in each dataset. Lastly, we observe that removing short-span QA leads to the largest performance drops, indicating its central role in adapting the LLM's performance towards specialized domains.

### Case Studies

To better illustrate the quality of pseudo-labeled samples generated by SimRAG after Stage-I finetuning, we present two case studies in Table 6, comparing the samples produced by SimRAG with those from the baseline model, Llama3-8B-it.

In the first case, where the model is asked to generate a claim supported by the context, Llama3-8B-it simply selects a sentence from the context. This results in relatively simple QA pairs, making the task less challenging for Stage-II training.

In the second case, the model is tasked with generating an answer first, and then formulating a question based on the context and the answer. While Llama3-8B-it does not copy a sentence exactly, it generates a lengthy question that closely paraphrases the context. This makes the question overly dependent on the original text, making it difficult to interpret without it. Additionally, the model misinterprets the context by implying that the research was focused on children when actually adults are the focus. In contrast, after fine-tuning on answer generation and query generation in Stage-I, SimRAG generates higher-quality QA pairs that are self-contained and understandable without relying on the context. These QA pairs also present more challenging tasks, as they require deeper comprehension of the context, providing harder and more effective training data for Stage-II.

 

Conclusion

We introduce SimRAG, an instruction fine-tuning framework designed to enhance LLMs for domain-specific question-answering tasks. By equipping LLMs with joint capabilities for both question answering and question generation, SimRAG enables the generation of diverse, high-quality synthetic questions from unlabeled domain-relevant corpora. This approach facilitates effective adaptation to specialized fields, where distribution shifts and limited domain-specific data typically pose challenges. Extensive experiments across 11 datasets in three domains show that SimRAG consistently outperforms baseline models, demonstrating its effectiveness in tackling the challenges of retrieval-augmented, domain-specific question-answering tasks.

## Limitation

While SimRAG demonstrates notable improvements, there are some limitations to our approach: **Single Round Pseudo-Label Generation**: Our current method relies on a single round of query generation from the corpus, which may restrict the refinement of pseudo label quality. Iterative refinement of generated synthetic queries could potentially lead to better results.

**Additional Training Time**: The incorporation of synthetic query generation and filtering adds time complexity compared to baseline models, which may affect efficiency in environments with limited computational resources. However, we would like to note that our method _will not increase the inference time complexity_ compared to the existing RAG approaches with the same backbone models. **Stronger Query Generation Models**: Although we achieved strong performance with Llama3 8B and Gemma2 27B models, leveraging more powerful query generation models, such as Llama-3.1-70B-it (Meta-AI, 2024), could yield further gains. However, using larger models would incur higher computational costs beyond our current budget.

## Acknowledgement

This research was partially supported by the Emory Global Diabetes Center of the Woodruff Sciences Center, Emory University.

## References

* B. Abacha, E. Agichtein, Y. Pinter, and D. Demner-Fushman (2017)Overview of the medical question answering task at trec 2017 liveqa. In TREC, pp. 1-12. Cited by: SS1.
* A. Ben Abacha, Y. Mrabet, M. Sharp, T. R. Goodwin, S. E. Shooshan, and D. Demner-Fushman (2019)Bridging the gap between consumers medication questions and trusted answers. In MEDINFO 2019: Health and Wellbeing e-Networks for All, IOS Press, External Links: Link Cited by: SS1.
* A. Anthropic (2023)Model card and evaluations for claude models. External Links: Link Cited by: SS1.
* E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. McGuinness (2020)Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint conference on neural networks (IJCNN), pp. 1-8. Cited by: SS1.
* A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi (2024)Self-radio: learning to retrieve, generate, and critique through self-reflection. In ICLR, Cited by: SS1.
* M. Bartolo, T. Thrush, R. Jia, S. Riedel, P. Stenetorp, and D. Kiela (2021)Improving question answering model robustness with synthetic adversarial data generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 8830-8848. External Links: Link Cited by: SS1.
* J. Berant, A. Chou, R. Frostig, and P. Liang (2013)Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, pp. 1533-1544. External Links: ISBN 978-1-649-3861-1, Document, Link Cited by: SS1.
* S. Borgeaud, C. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre (2022)Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pp. 2206-2240. External Links: Link Cited by: SS1.
* Z. Chen, A. Hernandez Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Kopf, A. Mohtashami, et al. (2023)Meditron-70b: scaling medical pretraining for large language models. ArXiv preprintabs/2311.16079.

 

Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024. Instruction pretraining: Language models are supervised multitask learners. _arXiv preprint arXiv:2406.14491_.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2924-2936, Minneapolis. Association for Computational Linguistics.
* Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. _ArXiv preprint_, abs/1803.05457.
* Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the world's first truly open instruction-tuned llm.
* Dasigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5925-5932, Hong Kong, China. Association for Computational Linguistics.
* Dong et al. (2024) Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen. 2024. Understand what llm needs: Dual preference alignment for retrieval-augmented generation. _arXiv preprint arXiv:2406.18676_.
* Du et al. (2021) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021. Self-training improves pre-training for natural language understanding. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5408-5418, Online. Association for Computational Linguistics.
* Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELIS: Long form question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
* Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. _ArXiv preprint_, abs/2312.10997.
* Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. _Transactions of the Association for Computational Linguistics_, 9:346-361.
* Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivarstan Srinivasan, Ksenia Konyshkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. _ArXiv preprint_, abs/2308.08998.
* Gutierrez et al. (2024) Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. _arXiv preprint arXiv:2405.14831_.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. _ArXiv preprint_, abs/2212.09689.
* Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Jeong et al. (2024) Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7036-7050, Mexico City, Mexico. Association for Computational Linguistics.
* Jiang et al. (2023) Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 7969-7992, Singapore. Association for Computational Linguistics 

Di Jin, Eileen Pan, Nassim Oufatrole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. _Applied Sciences_, 11(14):6421.
* Jin et al. [2019] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2576-2577, Hong Kong, China. Association for Computational Linguistics.
* Labs [2024] John Snow Labs. 2024. Jsl-medllama-3-8b-v2.0. [https://huggingface.co/johnsnowlabs/JSL-Medllama-3-8B-v2.0](https://huggingface.co/johnsnowlabs/JSL-Medllama-3-8B-v2.0).
* Kim et al. [2023] Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, et al. 2023. Soda: Million-scale dialogue distillation with social commonsense contextualization. In _EMNLP_.
* Kocisky et al. [2018] Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. _Transactions of the Association for Computational Linguistics_, 6:317-328.
* democratizing large language model alignment. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Kwiatkowski et al. [2019] Tom Kwiatkowski, Jenninaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Labrak et al. [2024] Yanisa Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Goouraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. _ArXiv preprint_, abs/2402.10373.
* Lewis et al. [2020] Patrick S. H. Lewis, Ethan Perez, Aleksandra Pitts, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kutler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Li et al. [2024]Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024. Self-alignment with instruction backtranslation. In _The Twelfth International Conference on Learning Representations_.
* Lin et al. [2023] Kevin Lin, Ovynid Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_, pages 58-62, Hong Kong, China. Association for Computational Linguistics.
* Lin et al. [2023] Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. In _Findings of EMNLP_.
* Lin et al. [2024] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2024. RA-DIT: Retrieval-augmented dual instruction tuning. In _ICLR_.
* Liu et al. [2020] Jian Liu, Leyang Cui, Hannemg Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_, pages 3622-3628. ijcai.org.
* Liu et al. [2022]Lining Liu, Patrick Lewis, Sebastian Riedel, and Ponthus Stenetorp. 2022. Challenges in generalization in open domain question answering. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 2014-2029, Seattle, United States. Association for Computational Linguistics.
* Liu et al. [2023] Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. Webglm: Towards an efficient web-enhanced question answering system with human preferences. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 4549-4560.
* Liu et al. [2024] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Chatqa: Surpassing gpt-4 on conversational qa and rag. In _NeurlPS_.
* Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Meta-AI [2024]Meta-AI. 2024. Llama 3 model card.

 

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
* Miller et al. (2020) John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020. The effect of natural distribution shift on question answering models. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 6905-6916. PMLR.
* Nori et al. (2023) Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishuang Liu, et al. 2023. Can generalist foundation models outcome special-purpose tuning? case study in medicine. _arXiv preprint arXiv:2311.16452_.
* OpenAI (2022) OpenAI. 2022. Introducing ChatGPT.
* OpenAI (2023) OpenAI. 2023. GPT-4.
* Ovadia et al. (2023) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023. Fine-tuning or retrieval? comparing knowledge injection in lms. _ArXiv preprint_, abs/2312.05934.
* Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcaq: A large-scale multi-subject multi-choice dataset for medical domain question answering. In _CHIL_.
* Pang et al. (2024) Richard Yuanze Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sakhbbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. In _NeurIPS_.
* Park et al. (2022) Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. FaVIQ: FAcc verification from information-seeking questions. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5154-5166, Dublin, Ireland. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of machine learning research_, 21(140):1-67.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SOuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Rosenberg et al. (2005) Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005. Semi-supervised self-training of object detection models. In _2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV-MOTION'05)_.
* Schimanski et al. (2024) Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Assh, and Markus Leippold. 2024. Towards faithful and robust LLM specialists for evidence-based question-answering. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1913-1931, Bangkok, Thailand. Association for Computational Linguistics.
* Shi et al. (2024a) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024a. Replug: Retrieval-augmented black-box language models. In _NAACL_.
* Shi et al. (2024b) Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Hang Wu, Carl Yang, and May D Wang. 2024b. Medadapter: Efficient test-time adaptation of large language models towards medical reasoning. In _EMNLP_.
* Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3784-3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Song et al. (2024) Xiaoshai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, et al. 2024. Cs-bench: A comprehensive benchmark for large language models towards computer science mastery. _ArXiv preprint_, abs/2406.08587.
* Su et al. (2024)Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12991-13013, Bangkok, Thailand. Association for Computational Linguistics.
* Team et al. (2024) Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, et al. 2024. Gemma 2: Improving open language models at a practical size. _ArXiv preprint_, abs/2408.00118.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics 

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotiis, Ioannits Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergio Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasa large-scale biomedical semantic indexing and question answering competition. _BMC bioinformatics_.
* Vu et al. (2021) Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit Iyyer. 2021. STraTA: Self-training with task augmentation for better few-shot learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5715-5731, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Wadden et al. (2024) David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, et al. 2024. Sciriff: A resource to enhance language model instruction-following over scientific literature. _ArXiv preprint_, abs/2406.07835.
* Wang et al. (2024) Haoyu Wang, Tuo Zhao, and Jing Gao. 2024. Blendfilter: Advancing retrieval-augmented large language models via query generation blending and knowledge filtering. In _EMNLP_.
* Wang et al. (2022) Kexin Wang, Namdan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2345-2360, Seattle, United States. Association for Computational Linguistics.
* Wang et al. (2021) Yaqing Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Hassan Awadallah. 2021. Meta self-training for few-shot neural sequence labeling. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 1737-1747.
* Wang et al. (2023a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaeh Hajishirzi. 2023a. Self-infunct: Aligning language models with self-generated instructions. In _ACL_.
* Wang et al. (2023b)Yubo Wang, Xueguang Ma, and Wenhu Chen. 2023b. Augmenting black-box llms with medical textbooks for clinical question answering. _ArXiv preprint_, abs/2309.02233.
* Wei et al. (2024) Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2024. Instructrag: Constructing retrieval-augmented generation with explicit denoising. _ArXiv preprint_, abs/2406.13629.
* Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_, pages 94-106, Copenhagen, Denmark. Association for Computational Linguistics.
* Wu et al. (2024) Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. 2024. Pmc-llama: toward building open-source language models for medicine. _Journal of the American Medical Informatics Association_, page ocae045.
* Xiong et al. (2024) GuangZhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 6233-6251, Bangokok, Thailand and virtual meeting. Association for Computational Linguistics.
* Xu et al. (2024a) Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024a. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In _ICLR_.
* Xu et al. (2024b) Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May D Wang, Joyce C Ho, Chao Zhang, and Carl Yang. 2024b. Bmretriever: Tuning large language models as better biomedical text retrievers. In _EMNLP_.
* Moran et al. (2024) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making retrieval-augmented language models robust to irrelevant context. In _ICLR_.
* Yu et al. (2023) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023. Chain-of-note: Enhancing robustness in retrieval-augmented language models. In _EMNLP_.
* Yu et al. (2024) Yue Yu, Wei Ping, Zihan Liu,oxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llvms. In _NeurIPS_.
* Yuan et al. (2024) Weizhe Yuan, Richard Yuanze Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. In _ICML_.
* Zhang et al. (2024a) Dan Zhang, Ziniu Hu, Siining Zoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yunxiao Dong, and Jie Tang. 2024a. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. _ArXiv preprint_, abs/2401.07950.
* Zhang et al. (2024b) Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, et al. 2024b. Ultracmedical: Building specialized generalists in biomedicine. _arXiv preprint arXiv:2406.03949_.
* Zhang et al. (2024c) Tianjun Zhang, Shishifr G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. 2024c. RAFT: Adapting language model to domain specific RAG. In _COLM_.
* Zhang et al. (2023) Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. 2023. Alpacare: Instruction-tuned large language models for medical application. _arXiv preprint arXiv:2310.14558_ 

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3277-3287, Online. Association for Computational Linguistics.

## Appendix A Training Data Details

We include the training dataset, the number of examples used in each stage, as well as the instruction format in Table 7. Note that in our implementation, we use the interleave_datasets() function to generate the final training data by merging multiple sources. The numbers presented in the table represent the sample counts of the training data from their original sources. In Stage-I, we set stopping_strategy="all_exhausted", an oversampling strategy where dataset construction stops only after every sample from all datasets has been included at least once. In Stage-II, we use stopping_strategy="first_exhausted", an undersampling strategy where construction stops as soon as one dataset has ran out of samples. In this case, the pseudo-labeled QA dataset is the first to run out, ensuring that all its samples are fully utilized, while only a small portion of the other datasets is included.

## Appendix B Test Data Details

We evaluate on 11 datasets in total from the medical, scientific and computer science domain. (1) **Medical:**

* MMLU-med Hendrycks et al. (2021) is a subset of six tasks related to biomedicine, including anatomy, clinical knowledge, professional medicine, human genetics, college medicine, and college biology. It contains 1089 questions in total.
* MedMCQA Pal et al. (2022) includes multiple-choice questions derived from Indian medical entrance exams, covering 2400 healthcare topics across 21 medical subjects. We use the 4,183-question development set from MedMCQA, as the test set lacks provided ground truths.
* MedQA Jin et al. (2021) is collected from the US Medical Licensing Examination, containing 1273 four-option multiple-choice questions focused on real-world scenarios from professional medical board exams.
* BioASQ Tsatsaronis et al. (2015) includes 618 questions constructed from biomedical literature without providing the ground truth snippets, challenging RAG systems to infer answers independently 

* PubMedQA Jin et al. (2019) is a biomedical research QA dataset consisting of 1000 manually annotated questions based on PubMed abstracts. Answers in PubMedQA are structured as yes/no/maybe to reflect the validity of the questions.
* LiveQA Abacha et al. (2017) and MedicationQA Abacha et al. (2019) are two QA datasets focusing on answering consumer health questions about medications, including 100 and 674 question-answer pairs, respectively.

(2) **Scientific:**
* SciQ Welbl et al. (2017) is a scientific questionnaire dataset containing 13,679 crowdsourced science exam questions about Physics, Chemistry, and Biology, among others.

\begin{table}
\begin{tabular}{l|c|c c c c} \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Specific Instruction**} & **Stage-I** & **Stage-I** & **Stage-II** & **Stage-II** \\  & & _# Samples_ & **Blending Ratio** & _\# Samples_ & **Blending Ratio** \\ \hline \multicolumn{5}{l}{**Instruction Fine-tuning**} \\ \hline ChatQA SFT Data & — & 60000 & 0.18 & 128000 & 0.12 \\ \hline \multicolumn{5}{l}{**Question Answering**} \\ \hline DROP & & 12000 & 0.034 & 29195 & 0.04 \\ NarrativeQA & & 12000 & 0.034 & 40000 & 0.04 \\ \cline{2-5} Quered & & 4800 & 0.014 & 10996 & 0.015 \\ \cline{2-5} ROPES & & 4800 & 0.014 & 10924 & 0.015 \\ \cline{2-5} Squad1.1 & & 16000 & 0.045 & 40000 & 0.035 \\ \cline{2-5} Squad2.0 & & 16000 & 0.045 & 52474 & 0.05 \\ \hline \multicolumn{5}{l}{OprnbookQA} & Answer the following question by selecting one of the provided options with A, B, C, or D. Please answer with the capitalized alphabet only, without adding any extra phrase or period. & 2000 & 0.006 & 82092 & 0.005 \\ \hline NQ & Answer the following question with a short phrase. & 16000 & 0.045 & 46126 & 0.04 \\ \hline \multicolumn{5}{l}{TatQA-arithmetic} & Answer the following question with a number from context or the math arithmetic using +’,’s,’ or \(J\) & 8325 & 0.045 & 24975 & 0.034 \\ \hline \multicolumn{5}{l}{TatQA-others} & Answer the following question with an soft span, or a full and complete answer. & 3176 & 0.023 & 9528 & 0.013 \\ \hline \multicolumn{5}{l}{WebGLM} & Please give a fall and complete answer for the question using only the provided search results (some of which might be irrelevant) and their them properly. Use an unbiased and journalistic note. When citing several search results, use [1][2][3]. & 12000 & 0.034 & 43579 & 0.023 \\ \hline StrategyQA & \multirow{2}{*}{Answer the following question with Yes or No. Is the statement FEVER} & \multirow{2}{*}{4000} & \multirow{2}{*}{0.012} & \multirow{2}{*}{9427} & \multirow{2}{*}{0.013} \\ \cline{2-5} Strategy & & & & & \\ \cline{2-5} FairIQ & Answer the following question with Yes or No. Is the statement & 2000 & 0.006 & 10906 & 0.01 \\ \cline{2-5} FEVER & & [claim] correct? & 2000 & 0.006 & 10444 & 0.01 \\ \hline \multicolumn{5}{l}{**Pseudo-labeled Question Answering**} \\ \hline Short-span QA & Answer the following question with a short span. & — & — & 150,000 /45,000 & 0.2625 \\ \cline{2-5} Multiple-choice QA & Answer the following question by selecting one of the provided options with A, B, C, or D. Please answer with the capitalized alphabet only, without adding any extra phrase or period. & — & 50,000 / 15,000 & 0.0875 \\ \hline Claim Verification & Answer the following question with Yes or No. Is the statement \{claim\} correct? & — & — & 100,000 / 30,000 & 0.175 \\ \hline \multicolumn{5}{l}{**Answer Generation**} \\ \hline Squad1.1 & \multirow{3}{*}{Based on the context, generate candidate spans within the passage that are likely to be answers to a question. Separate different candidate answers with a semicolon (’;').} & 18877 & 0.063 & — & — \\ \cline{2-5} Quad2.0 & & 18863 & 0.059 & — & — \\ \cline{2-5} DROP & & 4984 & 0.023 & — & — \\ \cline{2-5} WebQuestons & & 1084 & 0.012 & — & — \\ \hline \multicolumn{5}{l}{**Query Generation**} \\ \hline NQ & \multirow{3}{*}{Based on the context, please generate a question. The answer to the question should be \{answer\}.} & 20000 & 0.068 & — & — \\ \cline{2-5} Suqall.1 & & 20000 & 0.068 & — & — \\ \cline{2-5} StrategyQA & & 131 & 0.023 & — & — \\ \cline{2-5} WebQuestons & & 24000 & 0.068 & — & — \\ \hline FaVIQ & \multirow{3}{*}{Based on the context, please generate a claim that can be supported/refuted by the context.} & 10000 & 0.028 & — & — \\ \cline{2-5} FEVER & & 10000 & 0.028 & — & — \\ \hline \end{tabular}
\end{table}
Table 7: The blending ratio of different datasets with their specific prompt format in Stage-I and Stage-II fine-tuning. For Stage-II Pseudo-labeled QA Samples, the two numbers represent the # sample for the Llama and Gamma backbones, respectively.

 

* ARC-easy/challenge Clark et al. (2018) contains 7,787 authentic multiple-choice science questions at the grade-school level, designed to foster advanced question-answering research. The dataset is divided into a Challenge Set, with questions that stumped both a retrieval-based and a word co-occurrence algorithm, and an Easy Set.
* MMLU-Sci Hendrycks et al. (2021) is the Massive Multitask Language Understanding dataset, designed to test a wide range of language understanding abilities across 57 tasks. In this work, we select 14 subjects to ensure the evaluation is not limited to certain fields. (3) **Computer Science:**
* CS-Bench Song et al. (2024) is a recently-proposed benchmark specifically designed to assess the performance of large language models (LLMs) in computer science. It contains around 5,000 carefully selected test samples that span 26 subfields within four major areas of computer science, covering various task forms and divisions of knowledge and reasoning.

## Appendix C Baseline Descriptions

* Self-RAG Asai et al. (2024) utilizes instruction fine-tuning to adaptively retrieve passages based on the question and determine if the passage contains useful information for answering the question.
* ChatQA Liu et al. (2024) is a fine-tuning pipeline tailored for RAG and conversational QA tasks via aggregating multiple QA and dialogue datasets.
* RAFT Zhang et al. (2024c) is a domain-specific fine-tuning approach that incorporates top-\(k\) passages as context during fine-tuning, helping to address discrepancies between training and testing data.
* EvidenceRAG Schimanski et al. (2024) leverage off-the-shelf LLMs (GPT-4) to generate context-aware question answering datasets, which is then used to fine-tune the student model.

## Appendix D Additional Experimental Results

We list the per-task results of MMLU-sci in Table 8.

## Appendix E Prompt Details

### Answer Generation

[System]

[Context] Basedonthecontext,generate several candidates spanswithin thepassagethatarelikelytobe answersotaquestion.Theanswerscanbentities,verbsor evennumbers.Makesurethattheansweraredifferentand diverse.Separatetdifferentcandidateanswewitha semicolon(';').

### Query Generation

[System]

[Context] Basedonthecontext,please generatesquestionsintatis relevanttotheinformation provided.Thequestionshouldstandaloneandnotreferbacktothecontextexplicitly.Thequestionshouldbeclearandunderstandablewithoutneedingthecontext.Theanswerotethequestionshouldbe[Answer].

### Inference

[System]

[Top10Contexts]

[SpecificInstruction]

[Question]

The [Specifc Instruction] for each evaluationdataset depends on their question type and can refer to those in Table 7.

 



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c c} \hline
**Method** & **sublinearity** & **college** & **college** & **college** & **computer** & **high-school** & **high school** & **High school** & **High school** & **High school** & **high school** & **high school** & **lowism** \\  & **biology** & **chemometry** & **physics** & **energy** & **improversymmetries** & **microconnections** & **micconnections** & **psychology** & **UShistory** & **world history** & **available** & **activities** & **activity** & **video** & **Avg.** \\ \hline
**Metrics** & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _ACC_ & _acc_ & _acc_ & _acc_ \\ _primary LEMK, five Reference Only_ & _GPT3:519(A)_ & 64.45 & 64.28 & 35.00 & 46.55 & 65.00 & 77.27 & 96.54 & 64.29 & 80.12 & 76.63 & 72.15 & 70.09 & 66.01 & 47.99 & 66.04 \\ _GPT4_ (Applum)_ & 89.42 & 89.75 & 61.00 & 73.27 & 91.00 & 94.85 & 67.05 & 94.54 & 90.15 & 95.99 & 94.51 & 93.18 & 90.22 & 90.62 & 87.46 \\ _Scientific Link_ & _SciRma_ (Wakata et al., 2012b) & 69.74 & 63.39 & 31.00 & 18.63 & 62.80 & 70.20 & 70.20 & 86.05 & 77.68 & 77.43 & 53.08 & 65.65 & 54.90 & 48.78 & 55.55 \\ _St