InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales

## Abstract

Abstract Retrieval-augmented generation (RAG) has shown promising potential to enhance the accuracy and factuality of language models (LMs).
However, imperfect retrievers or noisy corpora can introduce misleading or even erroneous information to the retrieved contents, posing a significant challenge to the generation quality.
Existing RAG methods typically address this challenge by directly predicting final answers despite potentially noisy inputs, resulting in an implicit denoising process that is difficult to interpret and verify.
On the other hand, the acquisition of explicit denoising supervision is often costly, involving significant human efforts.
In this work, we propose InstructRAG , where LMs explicitly learn the denoising process through self-synthesized rationales — First, we instruct the LM to explain how the ground-truth answer is derived from retrieved documents.
Then, these rationales can be used either as demonstrations for in-context learning of explicit denoising or as supervised fine-tuning data to train the model.
Compared to standard RAG approaches, InstructRAG requires no additional supervision, allows for easier verification of the predicted answers, and effectively improves generation accuracy.
Experiments show InstructRAG consistently outperforms existing RAG methods in both training-free and trainable scenarios, achieving a relative improvement of 8.3% over the best baseline method on average across five knowledge-intensive benchmarks.
Extensive analysis indicates that InstructRAG scales well with increased numbers of retrieved documents and consistently exhibits robust denoising ability even in out-of-domain datasets, demonstrating strong generalizability. 1 1 1 Our code is available at https://github.com/weizhepei/InstructRAG .

## 1 Introduction

While large language models (LMs) have demonstrated remarkable text generation abilities [, , ], they may occasionally produce factually incorrect contents [, , , , , ], particularly when the task at hand requires the most current information or out-of-domain knowledge not adequately represented in the pre-training corpus [, , , ].
This limitation significantly hinders the reliable deployment of LMs in high-stakes domains where factuality is crucial [, , , ].

In light of this, retrieval-augmented generation (RAG) [, , , , ] has been introduced to enhance the generation accuracy of LMs in knowledge-intensive tasks by leveraging the most up-to-date information and specialized knowledge from external sources [, , , ].
However, the retrieved contents are typically mixed with irrelevant or even erroneous information due to the absence of perfect retrieval solutions [, , , , ] and the presence of noisy data in the retrieval corpus [, , ], posing a long-standing challenge to almost all RAG systems.
Typically, vanilla RAG approaches address this issue *implicitly* by training LMs to directly predict correct answers despite potentially noisy inputs.
Such latent processes are not only difficult to interpret and verify but also vulnerable to higher noise ratios, especially when the number of retrieved documents is large [, , , ].
On the other hand, obtaining high-quality explicit denoising supervision often requires substantial human efforts, which is time-consuming and costly.

In this work, we introduce a new RAG framework, InstructRAG, which enables the LM to explicitly denoise retrieved information and justify its predicted final answers by generating denoising responses (i.e., *rationales*), as illustrated in Figure 1.
Compared to vanilla RAG approaches, InstructRAG does not require any additional supervision, while enjoying improved generation accuracy and trustworthiness.
Specifically, our method consists of two steps.
First, given a set of question-answer pairs and potentially noisy retrieved documents, we prompt an instruction-tuned LM to synthesize denoising rationales that analyze the documents and articulate how they lead to the ground-truth answers (§ 2.2).
Then, these synthetic rationales can be utilized as in-context learning examples or as supervised fine-tuning data, allowing the LM to explicitly learn to denoise retrieved contents (§ 2.3).
The effectiveness of InstructRAG can be attributed to the strong instruction-following ability of LMs [, , ], a significant feature that still remains underexplored in the context of RAG.
We show that such self-synthesized rationales not only provide high-quality explicit denoising supervision for in-domain RAG tasks, but also facilitate superior out-of-domain generalization.
This finding underscores how instruction-tuned LMs can synthesize generalizable
supervision to effectively overcome the inevitable noise in RAG.

The main contributions of this work are as follows:
(1) We propose InstructRAG, a simple yet effective RAG framework that allows LMs to explicitly denoise retrieved contents by generating rationales for better verifiability and trustworthiness.
(2) InstructRAG is a self-synthesis method that does not require additional supervision compared to standard RAG methods, and can be seamlessly applied to both in-context learning and supervised fine-tuning settings.
(3) InstructRAG consistently outperforms state-of-the-art RAG approaches, yielding a relative improvement of 8.3% on average compared to the best baseline method across five knowledge-intensive benchmarks.
Extensive analysis and ablation studies further confirm the superiority of self-synthesized denoising rationales, and demonstrate InstructRAG’s robust denoising ability against increased noise ratios and strong task transferability in various training-free and trainable scenarios.

Figure: Figure 1: Comparison between vanilla RAG and our InstructRAG. In vanilla RAG, the model is tasked to directly predict answers given user queries and potentially noisy retrieved documents, without explicit denoising processes or explanations for how the answer is derived. In contrast, our proposed InstructRAG generates rationales that explicitly denoise the retrieved documents and justify the predicted answers, enhancing both the generation accuracy and trustworthiness.
Refer to caption: x1.png

## 2 Our Method: InstructRAG

In this section, we first introduce our problem setting (§ 2.1) and then present the proposed framework InstructRAG that enables LMs to explicitly denoise retrieved contents.
As shown in Figure 2, our method consists of two steps.
First, we prompt an
instruction-tuned LM (i.e., rationale generator $\mathcal{M}_{\phi}$) to synthesize rationales that provide denoising supervisions (§ 2.2).
These rationales aim to explain how to derive the correct answer from potentially noisy retrieved documents for each training sample.
Then, we guide the LM (i.e., rationale learner $\mathcal{M}_{\theta}$) to learn explicit denoising by leveraging these rationales as either in-context learning demonstrations or as supervised fine-tuning data (§ 2.3).

As detailed in Algorithm 1, during the entire process, InstructRAG does not require any additional supervisions beyond standard RAG methods.
By default, we instantiate both $\mathcal{M}_{\phi}$ and $\mathcal{M}_{\theta}$ with the same off-the-shelf instruction-tuned model (i.e., [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)), making InstructRAG a fully *self-synthesis* method.
We also experiment with different instantiations of $\mathcal{M}_{\phi}$ and $\mathcal{M}_{\theta}$ and conduct ablation study in both training-free and trainable settings (§ 3.3).
For simplicity, we use placeholders to represent omitted instructions in the prompts presented in this section, while the full list of complete prompt templates is provided in Appendix C.

Figure: Figure 2: An overview of InstructRAG. In step one, given the question $q$, retrieved documents $\{d_{1},\cdots,d_{K}\}$ and ground-truth answer $a$ from the training set, we prompt an instruction-tuned LM (i.e., rationale generator $\mathcal{M}_{\phi}$) to generate rationale $r$ that explains how the answer can be derived from the potentially noisy input. In step two, we utilize the synthesized rationales from the first step to guide the LM (i.e., rationale learner $\mathcal{M}_{\theta}$) to explicitly learn denoising of the retrieved documents, either through in-context learning or supervised learning. By default, we use the same model for both $\mathcal{M}_{\phi}$ and $\mathcal{M}_{\theta}$, but they can be instantiated with different models as well (see ablation study § 3.3).
Refer to caption: x2.png

Table: Table 1: Rationale generation prompt for the $i$-th training sample.

### 2.1 Problem Setting

We adopt the standard RAG setting where the LM $\mathcal{M}_{\theta}$ has access to annotated datasets of downstream tasks (e.g., question-answering task $\mathcal{T}=\{\mbox{$\langle$}q,a\mbox{$\rangle$}\}$), and an external knowledge base with the off-the-shelf retriever $\mathcal{R}$ for retrieval.
Different from previous works [, ]
which leverage additional supervisions from GPT-3 [] or GPT-4 [], we assume the model has strictly limited access to the above two information sources.
Given a question $q$, the retriever $\mathcal{R}$ returns a set of potentially noisy documents $D=\{d_{1},\cdots,d_{K}\}$ from the external knowledge base.
The model is then tasked to predict the correct answer $a$ to the given question $q$ based on $D$ and its own parametric knowledge, denoted as $p_{\theta}(a|q,D)$.

Our work focuses on investigating the noise robustness of LMs and developing efficient denoising techniques for RAG.
Hence, we directly employ off-the-shelf retrievers instead of training our own, and prepend all retrieved documents to the question as input to the model, without any filtering or re-ranking.
This setting is orthogonal to existing research efforts centered on optimizing the retriever or performing adaptive retrieval [, , ].

### 2.2 Rationale Generation via Instruction-Following

Recent studies [, , ] have made encouraging progress in aligning LMs with human preferences and intentions, enabling the synthesis of high-quality data that closely follows user instructions [].
Inspired by these advances, we propose to leverage the LM’s strong instruction-following ability to generate explicit denoising responses (i.e., *rationales*) for RAG.
As shown in Table 1, given a QA pair $\mbox{$\langle$}q_{i},a_{i}\mbox{$\rangle$}\in\mathcal{T}$ and a set of retrieved documents $\{d^{1}_{i},\cdots,d^{K}_{i}\}$, we prompt an off-the-shelf LM $\mathcal{M}_{\phi}$ (as the rationale generator) with denoising instructions to produce the corresponding rationale $r_{i}$ that distinguishes useful documents from noisy ones and explains how the contexts lead to the ground-truth answer $a_{i}$.
To ensure the synthetic rationales are aligned with the ground-truth answers, we use a simple substring match to assess their consistency.
The consistency ratio on training samples with at least one relevant document containing the ground-truth answer is 98% on average across five benchmarks, supporting the reliability of synthetic rationales as a sanity check.
This allows us to effectively augment the standard dataset $\mathcal{T}=\{\mbox{$\langle$}q,a\mbox{$\rangle$}\}\rightarrow\mathcal{T}^{+}=
\{\mbox{$\langle$}q,r\mbox{$\rangle$}\}$ with self-synthesized denoising rationales solely by instructing the LM, without any additional supervision.

We also validate the necessity of using an LM-based generator (i.e., $\mathcal{M}_{\phi}$) to create the rationales instead of employing simple heuristics — without the generator, rationales can be created in a template-based manner (Table 6), by roughly identifying relevant retrieved documents through simple substring-matching with the ground-truth answer.
However, as demonstrated in our ablation study, this approach suffers from semantically inaccurate matching of relevant documents, leading to significant performance degradation.
Another advantage of the LM-based generator is that it can produce high-quality rationales even *without referring to the ground-truth answer*, which only results in a minor performance drop.
More detailed analyses on rationale generation design can be found in our ablation study (§ 3.3).

### 2.3 Learning Denoising Rationales in RAG

With the rationale-augmented dataset $\mathcal{T}^{+}$, it becomes possible to develop a rationale learner $\mathcal{M}_{\theta}$ that directly learns explicit denoising for RAG tasks with efficient learning strategies.
Next, we introduce two simple yet effective learning methods in the *training-free* and *trainable* RAG settings, namely, InstructRAG-ICL and InstructRAG-FT.

InstructRAG-ICL is a training-free instantiation of InstructRAG where the model learns denoising rationales via in-context learning (ICL).
As shown in Table 10, given a test question $q$ and a set of retrieved documents $D=\{d_{1},\cdots,d_{K}\}$, we first randomly sample $N$ demonstrations $\mbox{$\langle$}q_{i},r_{i}\mbox{$\rangle$}\in\mathcal{T}^{+}$ from the rationale-augmented training dataset, and then prompt the model to follow the exemplars and generate rationale $r$.
To save memory and enhance inference efficiency, we only show exemplary questions and their corresponding rationales in such ICL demonstrations.

InstructRAG-FT is a trainable instantiation of InstructRAG that learns denoising rationales via supervised fine-tuning (FT) with standard language modeling objective.
As defined in Eq. (1), it maximizes the likelihood of rationale $r$ conditioned on question $q$ and retrieved documents $D$.

$$ $\vspace{-.8em}\max_{\mathcal{\theta}}\mathbb{E}_{(q,r)\sim\mathcal{T}^{+}}\log p _{\mathcal{\theta}}(r|q,D).$ (1) $$

where $\theta$ represents the model parameters.
Both the training and inference of InstructRAG-FT share the same data format.
As depicted in Table 11, it takes as input the retrieved documents followed by the question, and outputs the denoising rationale $r$.

Figure: Algorithm 1 InstructRAG

## 3 Experiments

### 3.1 Experimental Setting

**Table 2: Dataset statistics and retrieval setting.**
| Dataset | Train | Test | Retriever | Top-$K$ | Recall@$K$ |
| --- | --- | --- | --- | --- | --- |
| PopQA | 12,868 | 1,399 | Contriever | 5 | 68.7 |
| TriviaQA | 78,785 | 11,313 | Contriever | 5 | 73.5 |
| Natural Questions | 79,168 | 3,610 | DPR | 5 | 68.8 |
| ASQA | 4,353 | 948 | GTR | 5 | 82.2 |
| 2WikiMultiHopQA | 167,454 | 12,576 | BM25 | 10 | 40.7 |

RAG tasks and evaluation metrics. We extensively validate the effectiveness of InstructRAG on five knowledge-intensive benchmarks, including PopQA [], TriviaQA [], Natural Questions [], ASQA [], and 2WikiMultiHopQA [].
We use Wikipedia corpus as the retrieval source, and test our method with both sparse and dense off-the-shelf retrievers, including BM25 [], DPR [], GTR [] and Contriver [].
The retrieval quality is measured by Recall@$K$, indicating whether the retrieved $K$ documents contain the correct answer.
Table 2 shows the detailed dataset statistics.
Following standard evaluation settings [], we adopt the official metric of correctness (*str-em*), citation precision (*pre*) and recall (*rec*) for ASQA [], and use *accuracy* for the other tasks, which measures whether the ground-truth answers are included in the model generations [, ].
Additionally, we also adopt LLM-as-a-judge for further evaluation (§ 3.4), as the above standard metrics are subject to the limitations of pattern-matching, which cannot accurately handle semantic equivalence.

Baselines. We compare our method with a wide range of RAG baselines under both training-free and trainable settings.
Given that state-of-the-art LMs have incorporated a large amount of world-knowledge during the pre-training stage, we also report the performance of a non-retrieval baseline (namely, vanilla zero-shot prompting) for reference.
Specifically, the training-free RAG baselines includes: (1) in-context retrieval-augmented language modeling (RALM) [], a prompting method that extends the non-retrieval baseline by presenting the model with retrieved documents; (2) few-shot demonstration with instruction, an ICL method using ground-truth question-answer pairs sampled from the training set as demonstration exemplars.

The trainable RAG baselines include: (1) vanilla supervised fine-tuning (SFT), a supervised method with the training objective of maximizing the data likelihood of ground-truth answer given potentially noisy input; (2) RetRobust [], which fine-tunes the RAG model on a mixture of relevant and irrelevant contexts to make it robust to irrelevant contexts; (3) Self-RAG [], a strong trainable baseline, focusing on adaptive retrieval controlled by special reflection tokens.
Both RetRobust and Self-RAG were originally built on Llama-2 [] with additional supervisions.
For example, RetRobust augments the training data for multi-hop reasoning tasks (e.g., 2WikiMultiHopQA) by prompting GPT-3 to decompose the original query and generate intermediate subqueries, and Self-RAG requires GPT-4 to generate additional reflective tokens to augment training samples.

For a fair comparison, we re-implement the two methods on Llama-2${}_{\textsc{7B}}$ and/or Llama-2${}_{\textsc{13B}}$ with augmented training data released by their authors, and report their performance as the higher one between the original scores and our reproduced results.
As our method adopts instruction-tuned Llama-3 as the backbone model, we also train RetRobust and Self-RAG with Llama-3-Instruct${}_{\textsc{8B}}$ and optimize their performance through extensive hyper-parameters search.
More details on implementation, including training, inference, and prompt design are available in Appendix A and Appendix C.

### 3.2 Main Result

Table 3 shows the overall experimental results, providing a comprehensive comparison between our InstructRAG and baseline methods in both training-free and trainable RAG settings.

Baselines without retrieval.
As shown in the first block, the basic instruction-tuned models (Llama-3-Instruct${}_{\textsc{8B}}$ and Llama-3-Instruct${{}_{\textsc{70B}}}$) already achieve notable performance across all five benchmarks, with the 70B model exhibiting a surprisingly competitive performance of 80.6% on the TriviaQA.
This observation suggests that the required knowledge for these tasks mostly falls within the LM’s parametric knowledge, probably due to what is known as *data contamination* (i.e., the presence of test data of downstream tasks in the pre-training data of LMs) [, , ].

**Table 3: Overall results of InstructRAG and baselines on five knowledge-intensive benchmarks in training-free and trainable RAG settings. We re-implement baselines and report their performance as the higher one between the original scores and our reproduced results. * indicates the results copied from [] for reference. “–” indicates the results are not reported in the original paper or not applicable (e.g., some methods cannot produce citations). The best performance is highlighted in *bold*.**
|  | PopQA | TriviaQA | NQ | MultiHopQA | ASQA |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | (acc) | (acc) | (acc) | (acc) | (em) | (pre) | (rec) |
| Baselines w/o Retrieval |  |  |  |  |  |  |  |
| Vanilla Zero-shot Prompting |  |  |  |  |  |  |  |
| ChatGPT* | 29.3 | 74.3 | – | – | 35.3 | – | – |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 22.8 | 69.4 | 46.6 | 45.6 | 30.6 | – | – |
| Llama-3-Instruct${}_{\textsc{70b}}$ | 28.9 | 80.6 | 57.9 | 57.5 | 39.1 | – | – |
| RAG w/o Training |  |  |  |  |  |  |  |
| In-Context RALM [] |  |  |  |  |  |  |  |
| ChatGPT* | 50.8 | 65.7 | – | – | 40.7 | 65.1 | 76.6 |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 62.3 | 71.4 | 56.8 | 43.4 | 40.0 | 62.1 | 66.4 |
| Llama-3-Instruct${}_{\textsc{70b}}$ | 63.8 | 76.3 | 60.2 | 51.2 | 43.1 | 62.9 | 67.6 |
| Few-Shot Demo. w/ Instruction |  |  |  |  |  |  |  |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 63.1 | 74.2 | 60.1 | 45.3 | 42.6 | 55.0 | 64.4 |
| Llama-3-Instruct${}_{\textsc{70b}}$ | 63.9 | 79.1 | 62.9 | 53.9 | 45.4 | 49.3 | 57.1 |
| InstructRAG-ICL |  |  |  |  |  |  |  |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 64.2 | 76.8 | 62.1 | 50.4 | 44.7 | 70.9 | 74.1 |
| Llama-3-Instruct${}_{\textsc{70b}}$ | 65.5 | 81.2 | 66.5 | 57.3 | 47.8 | 69.1 | 71.2 |
| RAG w/ Training |  |  |  |  |  |  |  |
| Vanilla Supervised Fine-tuning |  |  |  |  |  |  |  |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 61.0 | 73.9 | 56.6 | 56.1 | 43.8 | – | – |
| Self-RAG [] |  |  |  |  |  |  |  |
| Llama-2${}_{\textsc{7b}}$ | 55.8 | 68.9 | 42.4 | 35.9 | 30.0 | 66.9 | 67.8 |
| Llama-2${}_{\textsc{13b}}$ | 56.3 | 70.4 | 46.4 | 36.0 | 31.4 | 70.3 | 71.3 |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 55.8 | 71.4 | 42.8 | 32.9 | 36.9 | 69.7 | 69.7 |
| RetRobust [] |  |  |  |  |  |  |  |
| Llama-2${}_{\textsc{13b}}$ | – | – | 39.6 | 51.5 | – | – | – |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 56.5 | 71.5 | 54.2 | 54.7 | 40.5 | – | – |
| InstructRAG-FT |  |  |  |  |  |  |  |
| Llama-3-Instruct${}_{\textsc{8b}}$ | 66.2 | 78.5 | 65.7 | 57.2 | 47.6 | 65.7 | 70.5 |

RAG without training.
The second block shows the comparison among training-free RAG methods.
In-context RALM and few-shot demonstration with instruction methods generally achieve higher performance than the non-retrieval baseline, highlighting the importance of retrieval for knowledge-intensive tasks.
Encouragingly, our InstructRAG-ICL consistently outperforms all training-free baselines across various metrics, confirming the effectiveness of self-synthesized denoising rationales.
Moreover, the boost from 8B to 70B model also demonstrates that InstructRAG-ICL scales effectively with larger backbone models, validating the generalizability of our method.

RAG with training. As present in the bottom block of Table 3, our InstructRAG-FT not only surpasses all non-retrieval and training-free baselines across all five benchmarks, but also significantly outperforms trainable RAG baselines on almost every metric.
The only exception is in the ASQA task, where our method slightly underperforms Self-RAG in terms of citation (i.e., *pre* and *rec*).
This is because our work primarily focuses on explicit denoising for RAG to improve the correctness of generations, which is measured by *em*.
Despite not being explicitly optimized for citation metrics, our method still achieves competitive citation performance, significantly enhancing both generation accuracy and trustworthiness.
Note that RetRobust achieves competitive performance on 2WikiMultiHopQA, which involves multi-hop reasoning.
We attribute this to the additional training supervision provided by GPT-3, which enables the model to explicitly generate intermediate sub-queries and sub-answers.
Another interesting finding is that Self-RAG consistently exhibits inferior performance compared to vanilla SFT, and even underperforms the training-free in-context RALM baseline across all benchmarks.
We speculate the reason might be that these RAG tasks favor more domain-specific knowledge than general knowledge.
However, it is challenging for Self-RAG to directly leverage in-domain features from existing training data as it requires GPT-4 to generate reflection tokens on these benchmarks, which is not available in our problem (§ 2.1).

### 3.3 Ablation Study

**Table 4: Ablation study on the impact of ground-truth answer, retrieved documents, and model size on rationale generation, and the use of demonstrations during model inference. The results of our default setting in InstructRAG are underlined.**
|  | Trainable RAG Setting | Training-free RAG Setting |  |  |
| --- | --- | --- | --- | --- |
| Method | PopQA | ASQA | PopQA | ASQA |
| Rationale Generation Design |  |  |  |  |
| with both | 66.2 | 47.6 | 64.2 | 44.7 |
| w/o ground-truth answer | 65.2 ($\downarrow 1.5\%$) | 46.4 ($\downarrow 2.5\%$) | 64.0 ($\downarrow 0.3\%$) | 44.5 ($\downarrow 0.4\%$) |
| w/o retrieved documents | 64.5 ($\downarrow 2.6\%$) | 45.2 ($\downarrow 5.0\%$) | 64.1 ($\downarrow 0.2\%$) | 44.3 ($\downarrow 0.9\%$) |
| Model Size of Rationale Generator |  |  |  |  |
| rationale template (no generator) | 59.6 ($\downarrow 10.0\%$) | 46.3 ($\downarrow 2.7\%$) | 60.0 ($\downarrow 6.5\%$) | 41.4 ($\downarrow 7.4\%$) |
| Llama-3-Instruct (8B) | 66.2 | 47.6 | 64.2 | 44.7 |
| Llama-3-Instruct (70B) | 67.0 ($\uparrow 1.2\%$) | 49.1 ($\uparrow 3.2\%$) | 64.8 ($\uparrow 0.9\%$) | 47.9 ($\uparrow 7.1\%$) |
| Inference Strategy Comparison |  |  |  |  |
| w/o demonstration | 66.2 | 47.6 | 63.0 ($\downarrow 1.9\%$) | 43.1 ($\downarrow 3.6\%$) |
| w/ demonstration | 66.1 ($\downarrow 0.2\%$) | 44.7 ($\downarrow 6.1\%$) | 64.2 | 44.7 |

Providing ground-truth answers and retrieved documents is important for rationale generation.
As depicted in the first block of Table 4, we ablate the rationale generation design from two aspects: (1) *w/o ground-truth answer*, where the model has no access to the ground-truth answer during rational generation and must predict the answer and explain how it is derived solely based on retrieved documents; (2) *w/o retrieved documents*, where the model is not provided with any retrieved documents during rational generation, and in this case, it has to explain the given answer based on its own knowledge.
Although it is not surprising that our default design consistently outperforms the two ablations, it is encouraging to find that our method still works well even without access to the retrieved documents or ground-truth answers.
This finding suggests the great potential of our InstructRAG to operate in a fully unsupervised manner, which we believe is an exciting direction for future work.

Larger rationale generator leads to better results. The middle block shows how different sizes of rationale generators impact the performance of our method.
It is evident that the template-based rationale generation method significantly underperforms our method, highlighting the necessity of rationale generator.
This is because the template-based method relies on pattern matching to identify relevant documents containing the ground-truth answer, which only considers lexical similarity while ignoring semantic meaning.
The neglect of semantics inevitably introduces noise in template-generated rationales, making them less effective compared to rationales generated by LMs.
Moreover, we also compare two variants of InstructRAG using Llama-3-Instruct${}_{\textsc{8B}}$ and Llama-3-Instruct${}_{\textsc{70B}}$ as rationale generators.
The results show that the one with a 70B generator consistently outperforms its 8B counterpart in both training-free and trainable settings, indicating that the self-synthesized denoising rationales can provide better supervision when generated by stronger models.

Inference with demonstrations should only be applied to InstructRAG-ICL.
In the bottom block, we study the use of demonstrations during the model inference. While demonstrations play an important role for InstructRAG-ICL, we find that they actually hurt the performance of InstructRAG-FT.
We attribute this to the fact that InstructRAG-FT is optimized to directly generate denoising rationales given potentially noisy input, without referring to any demonstrations.
Therefore, providing in-context demonstrations for InstructRAG-FT is redundant and may compromise its capability due to the discrepancy between training and inference.

### 3.4 Analysis

InstructRAG-ICL consistently benefits from more demonstrations.
Figure 3(a) shows the demonstration sensitivity of InstructRAG-ICL and the few-shot demonstration with instruction baseline.
It is interesting to find that the baseline method achieves its best performance with only one demonstration, and presenting more demonstrations actually harms its performance.
In contrast, our method consistently improves with the increasing number of demonstrations, confirming the superiority of self-synthesized rationales over plain answers in terms of denoising.

InstructRAG-ICL and InstructRAG-FT are robust to increased noise ratios. Figure 3(b) and Figure 3(c) show the generation accuracy of InstructRAG-ICL and InstructRAG-FT and the corresponding retrieval precision under an increasing number of retrieved documents.
While retrieving more documents provides richer external knowledge to the RAG model, it also introduces more noise and lowers the retrieval precision.
As a result, both the training-free and trainable baselines show diminishing improvements or even degrade as the number of documents increases, reflecting their vulnerability to high noisy ratios.
In contrast, our InstructRAG-ICL and InstructRAG-FT are not negatively affected by this increased noise ratio but rather gain further improvement, demonstrating their robust denoising ability.

Figure: (a) Training-free RAG setting.
Refer to caption: x3.png

Figure: (a) Short-form to long-form QA.
Refer to caption: x6.png

InstructRAG-ICL and InstructRAG-FT generalize well to unseen tasks.
Figure 4 demonstrates the generalization ability of our method in both training-free and trainable settings.
For the in-domain (ID) method, it directly utilizes target domain demonstrations (in training-free settings) or is trained on the target domain task (in trainable settings).
In contrast, the out-of-domain (OOD) method can only learn from demonstrations or training data in the source domain, and have no prior knowledge of the target domain.
In this case, the model must leverage the knowledge learned from the source domain task to solve the unseen target domain task.
The results show that our method (i.e., InstructRAG-ICL and InstructRAG-FT) consistently outperforms the baselines across various scenarios in both in-domain and out-of-domain settings, demonstrating strong task generalizability.
One counter-intuitive finding is that in the scenario of generalizing from long-form to short-form QA task (Figure 4(b)), the training-free OOD method substantially outperforms its in-domain counterpart.
We speculate that the training-free OOD method achieves better performance because it benefits from the demonstrations with long answers from the source domain (ASQA).
The reason is that the questions in ASQA are ambiguous and can have multiple interpretations, and ground-truth long answers often address the questions from various perspectives, which can be regarded as a form of chain-of-thought demonstration.

Furthermore, we also study the generalizability of InstructRAG to a non-QA knowledge-intensive task such as code generation. As presented in Table 5(a), we directly apply InstructRAG-FT trained on the QA task (PopQA) to solve the unseen code generation task (HumanEval []), following the CodeRAG-Bench setup [].
We evaluate the code generation performance
using the standard pass@k metric and compare our method with the off-the-shelf Llama-3-8B-Instruct as the baseline.
It can be observed that our method consistently achieves better generalization performance in the unseen code generation task in both non-retrieval and RAG settings. This finding aligns with our observation that InstructRAG trained on QA tasks tends to generate more text-based comments that articulate the design of coding solutions compared to the off-the-shelf Llama-3-8B-Instruct, thereby leading to more accurate code generation.

Evaluation with LLM-as-a-judge. Despite being standard evaluation metrics for question-answering, accuracy or exact match are known to be imperfect [] as they mainly rely on pattern-matching to judge whether the predicted answer aligns with the ground-truth answer. Such metrics cannot handle cases where the predicted answer and ground-truth answer are synonyms (e.g., “Donald Trump” vs “Donald J. Trump” cannot be correctly recognized as a match), leading to biased evaluation results.

Therefore, we adopt the LLM-as-a-judge [, ] approach to evaluate whether the model response aligns with the ground-truth answer using GPT-4o [], which allows the judge to consider semantic equivalence and is expected to yield a more fair evaluation.
As shown in Table 5(b), we evaluate our method and baseline models on the open-domain Natural Questions benchmark in both training-free and trainable RAG settings.
Compared to pattern-matching based metrics, LLM-as-a-judge generally leads to higher evaluation results, mostly due to its capability to accurately match semantically equivalent phrasings.
Notably, our method consistently outperforms baselines under both pattern-matching based and LLM-based evaluation metrics, further validating the effectiveness of InstructRAG.

**Table 5: (a) Transfer from the QA task (PopQA) to the code generation task (HumanEval). Our method InstructRAG-FT is fine-tuned only on the QA task and is evaluated for solving the unseen code generation task. We compare it with off-the-shelf LLaMA-3-8B-Instruct in both non-retrieval and retrieval-augmented generation settings, and report the standard evaluation metrics pass@k []. (b) Evaluation with GPT-4o as the judge. Compared to pattern-matching based metrics, it allows the judge to consider semantic equivalence and is expected to yield a more fair evaluation.**
| Method | pass@1 | pass@10 |
| --- | --- | --- |
| Without Retrieval |  |  |
| Llama-3-8B-Instruct | 58.5 | 64.6 |
| InstructRAG-FT | 60.4 | 65.2 |
| With Retrieval |  |  |
| Llama-3-8B-Instruct | 59.8 | 69.5 |
| InstructRAG-FT | 64.6 | 71.3 |

## 4 Related Work

### 4.1 Retrieval-augmented Generation

Retrieval-augmented generation (RAG) is a widely adopted approach to enhance large language models (LLMs) with external knowledge [, , , , , , , , ], demonstrating promising potential to reduce hallucinations and enhance the generation accuracy of LLMs across various real-world applications [, , , , , , ].
Recently, a growing research effort has been devoted to enhancing RAG from various aspects, such as improving decoding efficiency [, , ], exploring long-context retrieval [, ], compressing prompts [, ], and addressing practical concerns such as adversarial retrieval [, , ] and privacy leakage [, ].
Despite their advantages, these RAG systems inevitably suffer from irrelevant information introduced by imperfect retrievers or noisy retrieval corpora.
However, most existing works typically address this issue by improving the retrieval quality and reducing noise exposure to the model [, , , , , , , ].
Notable methods include adaptive retrieval [, , ] and query rewriting [, ].
In contrast, our work focuses on an orthogonal direction of developing explicit denoising methods for RAG, thereby enhancing the model’s noise robustness and generation accuracy, even in highly noisy contexts.

### 4.2 Eliciting Reasoning in Large Language Models

Recent studies have extensively explored the reasoning capability of LMs, but typically not in the context of RAG where potentially noisy retrieved contents may mislead the reasoning if not properly addressed.
Chain-of-thought (CoT) prompting [] is an effective method to elicit step-by-step reasoning from LMs by showing exemplars with detailed explanations (i.e., rationales [, , , ]) that lead to the final answer.
However, such works often requires manually crafted demonstrations [, ], which is costly and requires extensive efforts and domain knowledge [].
To mitigate this limitation, automatic chain-of-thought prompting (Auto-CoT) [] is introduced to automatically select instances from the corpus coupled with zero-shot CoT [], where the rationales are generated by LMs.
Furthermore, it has been shown that CoT reasoning can be elicited even without explicit prompting, particularly for instruction-tuned LMs [].
Another related work shows rationales generated by small models can help large models reason better [].
Although rationalization has been extensively investigated in many NLP tasks [, , , ], none of them are designed for RAG, and how to leverage the instruction-following abilities of LMs for explicit denoising in the context of RAG is still underexplored.

## 5 Conclusion

In this work, we presented InstructRAG, a simple retrieval-augmented generation (RAG) approach that explicitly denoises retrieved contents and produces accurate generations.
By leveraging the strong instruction-following abilities of large language models, InstructRAG generates detailed rationales that articulate how the ground-truth answers can be derived from the retrieved documents. These synthetic rationales can serve as either in-context learning examples or supervised fine-tuning data, enabling the model to learn an explicit denoising process.
Experiments on five knowledge-intensive benchmarks show InstructRAG consistently outperforms state-of-the-art RAG approaches with significant improvements in both training-free and trainable settings.
Compared to the best baseline method, InstructRAG achieves an average improvement of 8.3% across all benchmarks, demonstrating its effectiveness in enhancing the noise robustness of retrieval-augmented generation.

#### Limitations.

In this work, we mainly conduct experiments on question answering-type tasks, and it remains unclear how our method may generalize to other scenarios (e.g., open-ended generation).
Moreover, despite being the standard evaluation metrics, both accuracy and exact match are biased and cannot perfectly reflect the quality of the model’s generations.
For instance, such metrics heavily rely on string matching, which assesses correctness at the lexical level rather than the semantic level, thereby failing to recognize different phrasings that convey identical meanings.
The evaluation results also suffer from length bias, as longer generations tend to achieve higher accuracy.
Exploring more advanced metrics like using LLMs as judges would better evaluate RAG model generations [].
Another potential limitation is that our model might be subject to sample bias in the training data. Incorporating bias mitigation methods [, , ] would be helpful for further improving our work.

#### Future Work.

Future research directions include exploring more advanced techniques for generating high-quality rationales, such as incorporating domain-specific knowledge or leveraging multi-task learning to enable better generalization across various tasks.
For instance, although the consistency ratio between synthetic rationales and ground-truth answers on training samples with at least one relevant document achieves 98%, the overall consistency ratio on all training samples is only 89%. This is because for some samples, none of the retrieved documents is relevant to the question, which significantly compromises the quality of the generated rationales.
Therefore, it will be interesting to fully explore the potential of our method by incorporating additional designs such as a filtering mechanism, which we leave as future work.
Additionally, investigating the scalability of our method to larger datasets and models could further demonstrate its generalizability and robustness.
It will also be interesting to evaluate the model performance under long-context settings with a dynamic or extremely large number of retrieved documents.
Finally, integrating our method with other advanced retrieval techniques, such as active retrieval, could potentially lead to even better performance on knowledge-intensive tasks.

## Acknowledgments

The authors would like to thank Xinyu Zhu from University of Virginia, Tianyu Gao and Zexuan Zhong from Princeton NLP group for their valuable feedback and discussions.

## Appendix A Implementation Details

Retrieval setup.
Following [, ], we use the Wikipedia dump from [] as the external retrieval corpus for all five benchmarks studied in this work, where each document is a disjoint text block of up to 100 words extracted from a Wikipedia article.
We compared all RAG methods under a diverse retrieval environment with various sparse and dense retrievers and number of retrieved documents.
Specifically, we use Contriever-MS MARCO as the retriever for PopQA and TriviaQA, DPR for Natural Questions, GTR for ASQA, and BM25 for 2WikiMultiHopQA.
By default, we retrieve the top 5 documents from the retrieval corpus for each query in all tasks except 2WikiMultiHopQA, where the top 10 documents are retrieved.
We use the official weights for all dense retrievers and the implementation from Pyserini [] for the sparse retriever BM25.

Training details.
Our models are trained on 4 Nvidia H100 GPUs with 80GB memory via full-parameter fine-tuning.
We use fully sharded data parallelism (FSDP) for distributed training, along with
FlashAttention [] and bf16 mixed precision training enabled for computation efficiency.
By default, all models are trained using the Adam optimizer [] for 2 epochs, with a batch size of 128, a learning rate of 2.5e-5, and a cosine learning rate schedule with 3% warmup steps.
For the trainable baseline vanilla SFT, we use a slightly different learning rate of 2e-5 based on our hyper-parameter search results.
To fairly compare with Self-RAG and RetRobust, we re-implement them using Llama-3-Instruct-8B.
We also optimize their performance through an extensive hyper-parameter search with learning rates in [8e-6, 1e-5, 2e-5] and training epochs in [1, 2, 3].
For Self-RAG, we use a learning rate of 1e-5 with a single training epoch.
For RetRobust, we use a learning rate of 2e-5 with two training epochs.
The only exception is the training for RetRobust on 2WikiMultiHopQA, where we train the model for 5 epochs on the augmented training set released by the original authors.
The maximum token length for all models is fixed at 4096.

Inference details. By default, the number of demonstrations used in InstructRAG-ICL and the baseline method few-shot demonstration with instruction is set to be 2.
We use vLLM [] to load models for memory-efficient inference and adopt the greedy decoding strategy for model generation.

## Appendix B Case Study

Figure: (a) Vanilla SFT.
Refer to caption: x9.png

Figure: Figure 6: A case study of InstructRAG-FT compared with in-context RALM and vanilla SFT. The red texts denote irrelevant or inaccurate model generations, while the green texts denote contents relevant to the question. This study shows that our model can effectively identify relevant information from noisy input and leverage its own knowledge to correctly answer questions when required.
Refer to caption: x11.png

Attention visualization.
To intuitively understand the denoising process of our InstructRAG, we visualize its attention from the answer to retrieved documents.
As pointed out by a recent work [], only attention distributions from deep layers can accurately reflect the LM’s retrieval behavior and focus on key information, while attention from shallow layers usually do not imply meaningful patterns.
Therefore, we only plot the attention weights of the last 10 layers (Layer 22 to Layer 31).
As presented in Figure 5, our model accurately identifies the only benign document from noisy input, showing a strong denoising signal compared to vanilla SFT.

Generation comparison. Figure 6 compares the generated responses of in-context RALM, vanilla SFT, and our InstructRAG-FT for an actual question from the ASQA task.
Among them, only our method can correctly answer this question while providing comprehensive denoising details.
Specifically, it first identifies potentially relevant documents from noisy inputs, and then lays out the candidate information.
More encouragingly, we find that InstructRAG-FT is able to refer to its own parametric knowledge when no relevant document is present in the context after denoising, demonstrating its superiority over existing RAG approaches.

## Appendix C Prompt Templates

In this work, we instantiate the proposed InstructRAG with off-the-shelf instruction-tuned LMs (i.e., [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)), and apply the official [Meta-Llama-3-Instruct chat template](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3) (marked in gray) in all prompts.

Rationale generation. Below are the prompt templates for rationale generation used in all five tasks.
Table 6 shows the rationale template used in the ablation study.
For simplicity, we use the same prompt structure (Table 7) for all tasks with minor differences in task-specific instructions (Table 8).

Table: Table 6: Rationale template used in ablation study.

Table: Table 7: Rationale generation prompt template.

Table: Table 8: Task-specific instruction used in rationale generation prompt.

Inference prompts. Below we present the inference prompts for both training-free and trainable RAG methods used in this work, including in-context RALM (Table 9), few-shot demonstrations with instruction (Table 10), and vanilla supervised fine-tuning (Table 11).
Note that for a fair comparison, the inference prompt for our InstructRAG-FT is exactly the same as vanilla SFT.
Similarly, the inference prompt for InstructRAG-ICL shares the same inference instruction as the few-shot demonstrations with instruction.
The only difference between the prompts of these two methods lies in the demonstrations where InstructRAG-ICL employs denoising question-rationale ${\mbox{$\langle$}q,r\mbox{$\rangle$}}$ pairs, while few-shot demonstrations with instruction uses plain question-answer ${\mbox{$\langle$}q,a\mbox{$\rangle$}}$ pairs.

Table: Table 9: Inference prompt for In-Context RALM.

Table: Table 10: Inference prompt for InstructRAG-ICL and few-shot demonstrations with instruction.

Table: Table 11: Inference prompt for InstructRAG-FT and vanilla supervised fine-tuning.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The questions can be viewed in the weakness
soundness: 3
strengths: 1. InstructRAG is a simple but effective method to denoise retrvied information and justify its predicted final answers by generating denoising reponse 
2. InstructRAG does not require any additional supervision and can be applied to both in-context learning and supervised fine-tuning settings 
3. good presentation and  detailed experiment
summary: InstructRAG introduct one interesting method to explicitly learn the denoising process through self-synthesized rationales.  InstructRAG requires no additional supervision,  allows for easier verification of the predicted answers, and effectively improves generation accuracy.
weaknesses: 1. The rationale data generatioin is crutial in InstructRAG.  I hope to get more details about how to denoise wrong rationales and how to ensure the generated rationales satisfy the corresponding standard. 
2. I have some questions about how to use rationale in evaluation. From what I understand, InstructRAG needs to generate rationales first,  then concats the retrvied information and generated rationales to LM.  Maybe the added computation cost need to be analysized in the paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: The paper presents an interesting approach to generating rationales for document retrieval tasks. However, there are several aspects that require further clarification and improvement to strengthen the validity and robustness of the proposed method. Below are my detailed comments:
1. It is unclear whether the document collections used in the experiments contain noise samples, and if so, what the concentration of these noise samples is. Understanding the presence and extent of noise is crucial for evaluating the robustness of the proposed method.
2. Beyond validating the effectiveness of the rationales through final experimental outcomes, how was the intrinsic quality of the generated rationales assessed? What measures were taken to ensure that the model-generated rationales are free from noise and that the selected documents are indeed highly relevant? This is important to confirm that the rationales are not only effective but also reliable and interpretable.
soundness: 3
strengths: The paper effectively highlights the importance of denoising in RAG systems. This is a significant contribution to the field, as it addresses a critical issue that can significantly impact the performance of such models. The authors provide a thorough set of experiments that convincingly demonstrate the effectiveness of their proposed method. The results show that INSTRUCTRAG consistently outperforms both training-free baselines and training baselines across various metrics, which strongly supports the claim that self-synthesized denoising rationales are effective. Besides, the paper is well-organized and clearly written, making it easy for readers to follow the methodology, experimental setup, and results. The figures and tables are well-designed and enhance the understanding of the findings.
summary: This paper introduces INSTRUCTRAG, a novel approach to enhance retrieval-augmented generation by explicitly teaching language models to denoise retrieved information. Unlike traditional RAG methods that implicitly handle noisy inputs, INSTRUCTRAG leverages the instruction-following capabilities of LMs to generate synthetic rationales explaining how the correct answer is derived from the retrieved documents. These synthetic rationales serve as explicit denoising supervision, which can be used for in-context learning or supervised fine-tuning. This method not only improves the quality of in-domain RAG tasks but also enhances out-of-domain generalization. Finally, the paper demonstrates that by utilizing self-synthesized rationales, INSTRUCTRAG effectively addresses the challenges posed by noisy retrievals, making it a promising advancement in the field of RAG.
weaknesses: The paper could benefit from a more detailed description of the document retrieval process. Specifically, it is unclear whether the document collections used in the experiments contain noise samples, and if so, what the concentration of these noise samples is. This information is crucial for understanding the robustness of the proposed method and its applicability to real-world scenarios where noise data is often present. In addition, this paper could be strengthened by including a comparative analysis with other denoising techniques that have been proposed in the literature. This would help to contextualize the performance of INSTRUCTRAG and highlight its unique contributions.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: A detailed analysis of the noise present in the dataset would be valuable for better understanding the robustness of the proposed method. More analysis of examples and rationales will be helpful. Testing it across different base models would be beneficial in assessing the robustness and generalizability of the proposed method.
soundness: 3
strengths: 1. The method effectively leverages existing data, reducing the need for costly new annotations, which often involve extensive human effort.

2. Demonstrating consistent outperformance over traditional RAG methods across multiple benchmarks indicates effectiveness.
summary: The paper introduces an instructed RAG method designed to explicitly learn the denoising process in language models using self-synthesized rationales. InstructRAG addresses these issues by instructing LMs to generate explanations on how ground-truth answers are derived from retrieved documents. These explanations are rationales utilized in two ways: in-context learning to teach explicit denoising without additional data and as data for supervised fine-tuning. This approach does not require extra human-labeled supervision, simplifies the verification of predictions, and improves generation accuracy.
weaknesses: 1. The paper introduces a method that may involve noise in extracted knowledge or filtering information. However, the extent and nature of this noise are not adequately discussed. A quantitative measurement of the noise and its impact on the model’s performance would provide valuable insights into the robustness of the proposed method. Analyzing and quantifying the noise would also help clarify how much of the model’s learning is genuinely informative versus potentially spurious.

2. The paper would benefit from a more in-depth qualitative analysis of the extracted knowledge, including illustrative examples and rationales. Conducting a human study to evaluate whether the extracted knowledge aligns with useful, meaningful content (or if it simply reinforces dataset biases) would add a layer of rigor. This evaluation could help establish whether the proposed method provides actionable insights or merely reflects the underlying biases in the dataset. 

3. The proposed method’s restriction on accessing additional information sources (GPTs), relying primarily on self-regularization, may limit its practical utility in real-world applications where additional context and supervision are often available. Furthermore, the filtering and re-ranking mechanisms can be viewed as forms of self-regularization, and evaluating the effects of incorporating or comparing these elements would provide a more balanced assessment. Testing the method with and without these components could illustrate its practical utility in more realistic scenarios.

4. The paper’s approach, which employs instruction tuning, in-context learning, and knowledge extraction, aligns closely with established methodologies in related fields. As a result, the method may lack significant novelty, as it primarily adapts a classical pipeline to the Retrieval-Augmented Generation (RAG) domain. Introducing new techniques or unique modifications tailored to RAG-specific challenges could significantly enhance the paper’s contribution to the field.

5. The paper would be strengthened by testing the proposed rationale learner across various models to assess its robustness. Demonstrating consistent performance improvements with different model architectures would reinforce the generalizability of the method and provide a more comprehensive evaluation of its effectiveness.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. If the content in the documents does not explicitly contain the answer to the question, but the answer can still be inferred from the document content, can the rationale generator correctly reason and identify the relevant content? Did the authors conduct any corresponding ablation experiments or case studies to prove this?
soundness: 3
strengths: 1. This paper proposes a RAG (Retrieval-Augmented Generation) paradigm that is suitable for both fine-tuning and in-context learning (ICL) and demonstrates its effectiveness. The design and validation of the entire approach are interesting.
2. This paper uses ablation experiments to prove that providing true answers and retrieved documents is very important for inference generation.
3. By comparing ICL and FT (fine-tuning) methods, the authors found that example-based reasoning should only be applied to InstructRAG-ICL, as it negatively impacts the performance of InstructRAG-FT. They provide reasoning for this, which is valuable for guiding future research.
4. Through out-of-distribution (OOD) experiments, the authors demonstrate that both InstructRAG-ICL and InstructRAG-FT can generalize well to unseen tasks.
summary: This paper introduces InstructRAG, a model that learns denoising through self-generated rationales, which can be applied as in-context learning (ICL) demonstrations or supervised fine-tuning (SFT) training data to enhance performance. Experimental results across diverse datasets confirm the method’s effectiveness.
weaknesses: 1. If the complete answer to a question is distributed across multiple documents, with each document containing only part of the answer, this approach might mistakenly filter out these useful documents, leading to hallucinations or an inability to correctly answer the question. Could the authors provide more detailed analyses and experiments to clarify or verify the model's robustness?
2. The authors conducted experiments using only LLaMA-3-Instruct as the backbone, with the quality of self-synthesized rationales largely influenced by the underlying model's capability. Low-quality rationales may impair model performance, suggesting that the success of this approach could be attributed to LLaMA-3-Instruct's strong capabilities. Could the authors provide additional experiments with other backbones, such as LLaMA-2, Qwen, and Mistral, to help validate the method’s transferability and generalizability?
3. Although the datasets used in this paper include a variety of question-answering formats, such as short-form, multi-hop, and long-form, they are all based on wiki-style content. To more convincingly demonstrate the method's effectiveness, the authors should further extend their evaluation to more realistic and domain-diverse RAG datasets, like FreshQA and BRIGHT. Could the author also discuss the potential challenges in applying their method to these datasets and propose specific experiments to show the performance?
4. In Section 3.1, the experimental setting is unclear, do the authors utilize all the training samples for each dataset to generate rationales? Why do the authors adopt different retrievers for different datasets? What are the performance differences of the retrievers, such as Contriever, DPR, GTR, and BM25? Why is 2WikiMultiHopQA required to recall 10 documents while other datasets only require 5 documents?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Please refer to the Summary section.
soundness: 4
strengths: Please refer to the Summary section.
summary: The article introduces a simple yet effective RAG method to address noise problems in retrieved text results. Although the technique appears straightforward, I believe its simplicity enhances its versatility. I summarize the advantages as follows:

1. The approach is well-founded, as noise problems are prevalent in real-world RAG scenarios. InstructionRAG effectively addresses this problem.
2. The design is sound. The first phase generates rationales, which mitigate potential noise issues. Furthermore, the authors note, "The consistency ratio on training samples with at least one relevant document containing the ground-truth answer is 98% on average across five benchmarks, supporting the reliability of synthetic rationales as a sanity check." This aspect is crucial to the method's success.
3. The experiments are comprehensive, with validation across five datasets, demonstrating the effectiveness of InstructionRAG.

However, one might question whether using such sampling methods to address noise issues is worthwhile. This approach incurs additional time costs, which I believe cannot be overlooked. Exploring whether enhancing retrieval models might more effectively resolve this problem is a worthwhile consideration.

Overall, I am quite satisfied with the InstructionRAG method, which represents an influential contribution to the RAG field.
weaknesses: Please refer to the Summary section.

Meta-review: This paper introduces InstructRAG, a novel framework for retrieval-augmented generation that leverages self-synthesized rationales to explicitly denoise retrieved content, enhancing both the interpretability and accuracy of generated answers. The method is applicable in both training-free and trainable settings, demonstrating consistent improvements over baselines across five knowledge-intensive benchmarks. Strengths include its minimal reliance on additional supervision, robust generalizability across tasks, and detailed experimental analysis. While some reviewers raised concerns about computational overhead and limited testing on diverse datasets, the rebuttal and additional experiments addressed these issues satisfactorily. This paper is recommended for Accept (spotlight) due to its strong contributions to enhancing RAG systems.

Decision: Accept (Poster)