Title: RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLM

Retrieval-augmented Generation

Ranking
Abstract: Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG).  In this work, we propose a novel method called RankRAG, which instruction-tunes a single LLM for both context ranking and answer generation in RAG.  In particular, the instruction-tuned LLMs work surprisingly well by adding a small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on a large amount of ranking data.  For generation, we compare our model with many strong baselines, including ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks.  Specifically,  our Llama3-RankRAG-8B and Llama3-RankRAG-70B significantly outperform Llama3-ChatQA-1.5-8B and Llama3-ChatQA-1.5-70B, respectively, on nine general knowledge-intensive benchmarks for RAG. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization to new domains.

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

Abstract

Large language models (LLMs) typically utilize the topk contexts from a retriever in retrieval-augmented generation (RAG). In this work, we propose a novel instruction fine-tuning framework RankRAG, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG. In particular, the instruction-tuned LLMs work surprisingly well by adding a small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on a large amount of ranking data. For generation, we compare our model with many strong baselines, including GPT-4-0613, GPT-4-turbo-2024-0409, and ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks. Specifically, our Llama3-RankRAG significantly outperforms Llama3-ChatQA-1.5 and GPT-4 models on nine knowledge-intensive benchmarks. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization to new domains.

1 Introduction

Retrieval-augmented generation (RAG) (Lewis et al., 2020; Izacard & Grave, 2021; Lin et al., 2024; Wang et al., 2024) is a widely used technique for customizing large language models (LLMs) to handle long-tail knowledge (Mallen et al., 2023; Asai et al., 2024b), provide up-to-date information (Kasai et al., 2023), and adapt to specific domains and tasks (Xiong et al., 2024) without modifying the model weights. In general, a dense embedding-based retriever (Karpukhin et al., 2020; Lin et al., 2023; Wang et al., 2022) first retrieves topk chunked contexts from a collection of documents or external database for a given question. Then, LLM reads the topk contexts to generate the answer.

However, the current RAG pipeline has the following limitations: i) LLMs are not good at reading too many chunked contexts (e.g., top-100) even with the long-context window, not only due to efficiency reasons, but also because a shorter list of topk (e.g., 5, 10) contexts usually leads to higher accuracy of generation (e.g., see Table 5 in Xu et al., 2024b). ii) Given a small k , one needs a mechanism to ensure the high recall of relevant contents. Relying solely on a retrieval model may be inadequate due to challenges in learning effective local alignments across the entire embedding space to support accurate matching (Luan et al., 2021). In practice, a separate ranking model (Nogueira et al., 2020; Glass et al., 2022; Ma et al., 2023) that cross-encodes question and candidate context can work better than a dense embedding-based retriever for obtaining the most relevant topk contexts from topN

candidates ( N ≫ k ). iii) However, the zero-shot generalization capability of the expert ranking model can be relatively limited compared to the versatile LLM itself.

Based on the above considerations, our goal is to design an RAG instruction tuning pipeline that uses a single language model to achieve both high-recall context extraction and high-quality content generation. In previous study, instruction-tuned LLMs demonstrate a strong ability to extract answers from relevant context for a given question (e.g., OpenAI, 2023; Liu et al., 2024; Lin et al., 2024). This capability can be viewed as the 'dual capability' of determining whether a chunk of context is relevant to the question thus is useful for generating the answer. We hypothesize that these capabilities mutually enhance each other. Motivated by this insight, we propose RankRAG, which instruction-tunes a single LLM for both context ranking and answer generation in the RAG framework. Furthermore, RankRAG expands upon existing instruction-tuning data by incorporating context-rich QA, retrieval-augmented QA and ranking datasets, enhancing the LLM's ability to filter out irrelevant contexts during both the retrieval and generation phases of RAG.

Our contribution can be summarized as follows:

We propose RankRAG, a novel framework that enhances LLM's RAG capability through simultaneously instructing the LLM on context ranking and answer generation. During training, we design a specialized task focused on identifying relevant contexts or passages for a given question. This task is structured for ranking and framed as regular question answering with instruction, aligning more effectively with retrieval-augmented generation tasks. At inference, the LLM first reranks the retrieved contexts, then generates answer based on the refined topk (e.g., 5). This framework is readily applicable to diverse knowledge-intensive NLP tasks.

Remarkably, we observe that integrating a small fraction of ranking data into the instruction tuning blend of LLM works surprisingly well on the evaluations of ranking associated with the RAG tasks, even surpassing the LLMs fine-tuned with 10 × more ranking data. We attribute this success to the transferable design of RankRAG training.

We extensively compare the proposed RankRAG method with several strong baselines, including the open-sourced ChatQA-1.5. On nine general-domain and five biomedical knowledge-intensive benchmarks for RAG, Llama3-RankRAG-8B and Llama3-RankRAG-70B outperforms Llama3ChatQA-1.5-8B and Llama3-ChatQA-1.5-70B by a margin, respectively.

In the remainder of the paper, we discuss related work in § 2. We introduce problem setup in § 3 and RankRAG method in § 4. We present the experimental setup in § 5, and conclude the paper in § 6.

2 Related Work

Retrieval-augumented generation (RAG) has been established for knowledge-intensive NLP tasks (Lewis et al., 2020; Borgeaud et al., 2022; Izacard et al., 2023; Izacard & Grave, 2021). In the standard process, a standalone dense-embedding-based retriever (e.g., Karpukhin et al., 2020) first retrieves relevant information from an external corpus, which the LLM then utilizes in the generation process. To improve this pipeline, recent research has focused on aligning retrievers to the needs of LLMs for generation (Shi et al., 2024; Lin et al., 2024), designing multi-step retrieval processes (Trivedi et al., 2023; Jiang et al., 2023; Jeong et al., 2024; Shao et al., 2023), or filtering irrelevant contexts (Wang et al., 2023c; Yoran et al., 2024; Xu et al., 2024a). To improve generation, several studies have designed instruction-tuning methods dedicated to enhancing the search (Ma et al., 2023; Zhu et al., 2024; Muennighoff et al., 2024) and RAG capability of LLMs (Liu et al., 2024; Lin et al., 2024; Luo et al., 2023; Asai et al., 2024a; Wang et al., 2024).

Although strong retrievers have been introduced (e.g., Lin et al., 2023; Yu et al., 2022; Wang et al., 2022, 2023a; Lee et al., 2024), one potential approach to improve retriever is optimizing it along with LLM in an end-to-end manner (e.g., Guu et al., 2020; Shi et al., 2024; Sachan et al., 2021; Izacard et al., 2023). However, this requires surrogate loss for optimization and complicates the training pipeline, especially when the embedding database needs to be re-indexed frequently due to the update of the embedding model (i.e., retriever).

Ranking serves as an intermediate step to improve the quality of information retrieval (Mitra et al., 2018), and has been applied to RAG pipeline for improving generation quality (Glass et al., 2022; Ram et al., 2023). However, these methods still rely on an additional moderate-sized model (e.g. BERT, T5) for ranking, which is often insufficient to capture the relevance between query and contexts

5

10

20

40

42

Exact Match

ChatQA-1.5 8B

5

10

20

44

46

48

ChatQA-1.5 70B

Context Size

k

(a) NQ

5

10

20

78

80

82

Exact Match

ChatQA-1.5 8B

5

10

20

82

84

86

ChatQA-1.5 70B

Context Size

k

(b) TriviaQA

5

10

20

46

48

50

52

54

Exact Match

ChatQA-1.5 8B

5

10

20

48

49

50

51

52

ChatQA-1.5 70B

Context Size

k

(c) PopQA

5

10

20

89

90

91

92

Exact Match

ChatQA-1.5 8B

5

10

20

90

91

92

93

ChatQA-1.5 70B

Context Size

k

(d) FEVER

and may lack the zero-shot generalization capability. Although recent studies have demonstrated the strong ability of LLMs at ranking tasks (Khalifa et al., 2023; Qin et al., 2024; Sun et al., 2023), how to harvest this ability for the RAG pipeline remains underexplored.

3 Preliminaries

In this section, we first introduce the preliminaries of retrieval-augmented generation as well as the problem setup. Then we present the limitations in the current RAG pipeline, which motivates the proposed RankRAG method.

3.1 Problem Setup

In retrieval-augmented generation, a collection of documents or contexts (e.g. Wikipedia) is given, providing the grounded knowledge. Given a question q , the retriever R (e.g., a parameterized embedding model) first retrieves topk contexts C = { c 1 , · · · , c k } that are most relevant to the question. Subsequently, the language model produces the final answer where the answer can either be a short phrase or a long sentence, depending on the type of the target task. Our focus is on autoregressive language models (OpenAI, 2022, 2023; Meta-AI, 2024), which is the most common architectures for LLMs.

3.2 Limitation of Current RAG Pipelines

Before formally introducing RankRAG, we would like to first pinpoint several limitations of the current 'retrieve-then-generate' pipeline with large language models.

Limited Capacity of Retriever. Current RAG systems usually employ sparse retrieval (e.g. BM25 (Robertson et al., 2004)) or moderate-size (e.g. BERT-based) embedding models (Karpukhin et al., 2020; Lin et al., 2023; Wang et al., 2022) as the retriever R , mainly due to efficiency consideration as there are often millions of, if not more, documents need to be indexed. These models encode questions and documents independently and calculate the similarity between question and documents using vector similarity metrics. However, the limited capacity of embedding models and independent processing of query and documents constrain the ability to estimate textual relevance between question q and documents d , reducing their effectiveness in new tasks or domains, verified by both theoretical (Menon et al., 2022) and empirical (Luan et al., 2021; Thakur et al., 2021) studies.

Trade-off of Picking Topk Contexts. Although the state-of-the-art long-context LLM can take many retrieved contexts as input for answer generation, the performance quickly saturates with increased k in practice. For example, Xu et al. (2024b) finds the optimal number of chunked context k is around 10 for long document QA tasks. As illustrated in Figure 1, we perform evaluation on ChatQA-1.5 (Liu et al., 2024), one of the strongest RAG models with open weights, and find the saturation of accuracy when k = 10 . In general, a smaller k often fails to capture all relevant information, compromising the recall , given the limited expressibility of retriever. In contrast, a larger k improves recall but at the cost of introducing irrelevant content that hampers the LLM's ability to generate accurate answers (Yoran et al., 2024; Yu et al., 2023b).

Query

Corpus

1.

2.

…

…

𝑁

.

Top-N Docs

0.73

0.46

…

…

0.57

1.

2.

…

𝑘

.

Top-K Docs

0.94

0.73

…

0.63

Relevance scores

1.

2.

…

…

𝑁

.

Top-N Docs

Retriever

(e.g., DPR, Dragon)

RankRAG

Rerank

Generate

Answer

Synthetic Instructions

Self-Instruct,

Unnatural Instruction

Conversation

SODA, Dolly,

OpenAssistant

Long-form QA

ELI5

Chain-of-thought

FLAN

Stage-I: SFT

Conversation

Synthetic Conversation,

HumanAnnotatedConvQA

Retrieval-augmented

QA&Ranking

SQuAD, WebQuestion

Reading Comprehension

NarrativeQA, DROP, Quoref,

NewsQA, TAT-QA, ROPES

Context Ranking

Synthetic Conversation

Stage-II: RankRAG Instruction-Tuning

LLM

Inference

Training

RankRAG

Newly Introduced Tasks

4 RankRAG

To address the limitations mentioned in the previous section, we propose the RankRAG method to enhance the LLM's ability for retrieval-augmented generation. Specifically, we instruction-tune the LLM to simultaneously capture the relevance between the question and context and utilize the retrieved context for answer generation . The details are introduced as follows.

4.1 Stage-I: Supervised Fine-Tuning (SFT)

It is observed that general instruction-tuning or supervised fine-tuning (SFT) often significantly improves the ability of LLMs to follow instructions, thus improving zero-shot results on various downstream tasks (Wei et al., 2022; Ouyang et al., 2022). As such, we follow existing works (Chung et al., 2024; Wang et al., 2024; Liu et al., 2024) to first leverage SFT on a blend of high quality instruction following datasets, including: i) a private crowd-sourced conversational dataset and public conversation datasets : OpenAssistant (Köpf et al., 2023), Dolly (Conover et al., 2023), and SODA (Kim et al., 2023), ii) a long-form QA dataset ELI5 that requires elaborate answers (Fan et al., 2019), iii) LLM-generated instructions : Self-Instruct (Wang et al., 2023b) and Unnatural Instructions (Honovich et al., 2023), iv) FLAN and Chain-of-thought datasets (Chung et al., 2024).

There are overall 128K SFT examples in total. We make sure that there is no overlap between SFT data and data from evaluation tasks. For each sample in the instruction-following dataset, we take the multi-turn conversational format, use the previous turns of conversation between the user and the assistant as the context, and compute the loss only at the last response from the assistant.

4.2 Stage-II: Unified Instruction-Tuning for Ranking and Generation

The Stage-I SFT enpowers the LLMs with basic instruction-following capabilities; however, their performance on RAG tasks often remains suboptimal, as the LLMs are not optimized for extracting answers from retrieved context for a given question. Although recent studies (Lin et al., 2024; Liu et al., 2024; Zhang et al., 2024) enhance the RAG capability of LLM by instruction tuning it on context-rich generation tasks, these approaches can still be ineffective with poor initial retrieval results. RankRAG instruction tunes the LLM for both retrieval-augmented generation and context ranking. In particular, the context ranking capability is crucial to obtain more relevant topk context with imperfect retriever.

To achieve this goal, the instruction tuning blend of Stage-II consists the following five parts:

SFT data from Stage-I. This part is included to maintain LLM's instruction-following capability.

Context-rich QA data. We first follow Liu et al. (2024) to leverage multiple context-rich QA tasks to enhance the LLM's capability of using context for generation. The training blend we use consists of: i ) standard QA and reading comprehension datasets: DROP (Dua et al., 2019), NarrativeQA (Koˇ cisk` y et al., 2018), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), NewsQA (Trischler et al., 2017), TAT-QA (Zhu et al., 2021), which contains a question, a golden context and an answer. ii ) conversational QA datasets: HumanAnnotatedConvQA and SyntheticConvQA open-sourced by Liu et al. (2024), which contains a conversation between user and assistant, as well as one background document. The model needs to generate an answer given the conversation history and document.

Retrieval-augmented QA data. In addition to the above QA datasets used in Liu et al. (2024), we add two datasets with not only gold context but also the top-retrieved context using BM25. Note that it is crucial to improve LLM's robustness over irrelevant context at generation. Being aware of

this, we consider two QA tasks, namely SQuAD (Rajpurkar et al., 2016) and WebQuestions (Berant et al., 2013). For each question with the answer, we combine the gold context with the top-retrieved contexts using BM25, ensuring a total of five contexts. Note that some retrieved contexts may not contain the answer, and could be the 'hard-negative' contexts.

4) Context ranking data. To empower LLMs with ranking capabilities, we use the popular MS MARCO passage (context) ranking dataset (Bajaj et al., 2016). We treat the gold query-passage pairs ( q, c + ) as relevant while using hard negative passages ( q, c -) mined via BM25 as irrelevant pairs. The LLM needs to generate 'True' or 'False' given the corresponding query-passage pair, where the question along with the task-specific instruction is 'For the question {question} , access whether the passage is relevant to the question.'.

We want to handle ranking in conversational scenarios as well. While MS MARCO spans various topics, the questions are only single-turn short sentences. However, ranking data is only available, if any, at a small amount for conversation QA. To overcome this limitation, we repurpose the conversational QA pairs to generate pseudo relevance pairs. As each conversation is only associated with one document d , we cut each document into 150-word chunks ( c 1 , c 2 , . . . , c n ) . We compute the 4-gram recall score between each chunk c i and the ground-truth answer a , considering segments with a recall score above 0.5 as relevant and those below 0.1 as irrelevant for the corresponding conversation. Note that, each sample contains one question-context pair for this ranking dataset. In total, there are around 50k ranking pairs from MS MARCO ranking and synthetic conversations for instruction finetuning.

5) Retrieval-augmented ranking data. We aim to train the LLM with the capability of determining the relevance of multiple contexts simultaneously given a question, which is closer to the test-time behavior of RAG with topk contexts. As before, we use two QA datasets, SQuAD (Rajpurkar et al., 2016) and WebQuestions (Berant et al., 2013). We combine the gold context with the top-retrieved contexts using BM25, ensuring a total of five contexts. The contexts containing the answer are considered relevant, and the LLM is trained to explicitly identify all relevant contexts for the question.

Unifying RAG and ranking with instruction tuning. It is worth noting that, despite the variety of datasets and tasks described, they can all be cast into a standardized QA format ( x, c, y ) , where x is the question, c is the corresponding context, and y is the target output answer. For example, for the retrieval-augmented ranking data, the question is ' For the question <question>, find all the passages from the context that are relevant to the question .' Table 1 exhibits how to cast different tasks into a unified format. Despite its simplicity, this approach has the following advantages: i ) It empowers the LLM with the ranking capability by adding a relatively small amount of ranking data. ii ) By standardizing these tasks into a unified format, they can mutually enhance each other. After that, we obtain the final RankRAG model that can be applied to various knowledge-intensive NLP tasks.

4.3 RankRAG Inference: Retrieve-Rerank-Generate Pipeline

As RankRAG incorporates an additional reranking step, the inference pipeline for each question is modified as a retrieve-rerank-generate pipeline, described as follows: (1) the retriever R first retrieves topN contexts from the corpus. (2) the RankRAG model calculates the relevance score between the question and retrieved N contexts as the probability of generating the answer as True using the prompt in Table 1, then reranks contexts to only retain topk ( k ≪ N ) contexts, which are then used as the input for the generation step. (3) The topk contexts, along with the question, are concatenated and fed back into the RankRAG model to generate the final answer.

Efficiency Discussion. We are aware that the addition of a reranking step introduces extra processing time. In practice, for each question, denote the time for indexing and retrieval as t 1 , the time for using LLM to calculate the relevance score as t 2 and the time for generation as t 3 , then the ratio of added time overhead is N ∗ t 2 t 1 + t 3 . In practice, calculating relevance typically requires generating just one token and involves much shorter inputs compared to the generation step with topk contexts. We provide efficiency study in §5.5.

5 Experiments

In this section, we conduct comprehensive experiments on a variety of knowledge-intensive NLP tasks to demonstrate the zero-shot capabilities of RankRAG.

5.1 Experiment Setup

Tasks and Datasets. We consider 3 types of tasks in experiments: (1) Open-domain QA (OpenQA), which includes NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023), HotpotQA (Yang et al., 2018) and 2WikimQA (Ho et al., 2020). The first three are singlehop QA tasks, while the last two are multi-hop QA datasets. For NQ, TriviaQA, and HotpotQA, we use the split from KILT benchmark (Petroni et al., 2021) 2 . (2) Fact verification , where we use FEVER (Thorne et al., 2018) from KILT benchmark. (3) Conversational QA (ConvQA), we consider three datasets including Doc2Dial (Feng et al., 2020), TopiOCQA (Adlakha et al., 2022) and INSCIT (Wu et al., 2023), which have long documents that cannot be fitted directly into LLMs thus necessitates retrieval and ranking. The detailed dataset information is in Appendix A.1.

Baselines. We consider the following baselines: (1) Baseline LLMs without RAG , where we consider LLMs trained with proprietary data including InstructGPT (Ouyang et al., 2022), PaLM 2 (Anil et al., 2023), FLAN-LaMDA (Longpre et al., 2023), GLaM (Du et al., 2022), Claude 2 (Anthropic, 2023), Mixtral-8x22B-Instruct (Mistral, 2024), DeepSeek-V2 Chat (DeepSeek, 2024) and only use the official reported results. We also consider two ChatGPT-series models, namely GPT-3.5-turbo ( gpt-3.5-turbo-0613 ) (OpenAI, 2022) and GPT-4 ( gpt-4-0613 ) (OpenAI, 2023). (2) Baselines with retrieval , we evaluate models augmented with retrieval. Specifically, we include Atlas (Izacard et al., 2023) and Raven (Huang et al., 2023), two RAG models based on encoderdecoder LMs. For decoder-only models, we consider Self-RAG (Asai et al., 2024a), RECOMP (Xu et al., 2024a), InstructRetro (Wang et al., 2024), RePlug (Shi et al., 2024), RA-DIT (Lin et al., 2024), Llama-3-instruct (Meta-AI, 2024) and ChatQA-1.5 (Liu et al., 2024). We also list the result of RAG pipelines using InstructGPT (175B parameters) as the backbone including GenRead (Yu et al., 2023a), Retrieve-read (Lazaridou et al., 2022) and ReFeed (Yu et al., 2024), but mainly for reference. Other reported numbers are directly comparable if they follow the standard zero-shot settings.

Evaluation Metrics. For OpenQA datasets, we use Exact Match (EM) as the main metric but also report Accuracy for TriviaQA and PopQA and F1 score for HotpotQA and 2WikimQA as it is used in several studies (Asai et al., 2024a; Mallen et al., 2023). For FEVER, we use accuracy as the metric. For ConvQA datasets, we follow (Liu et al., 2024; Wang et al., 2024) to use F1 score as the metric.

Implementation Details. We use Llama3 8B and 70B (Meta-AI, 2024) as the backbone in our main experiments. For the two-stage instruction tuning, we set the batch size to 128 and train the model for 1000 steps with learning rate 5e-6 in Stage-I. Then, we reduce the learning rate to 3e-7 for 8B and 2e-7 for 70B model, set the batch size to 64, and train the model for 3300 steps (around 1 epoch). We use the Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 . 9 and β 2 = 0 . 98 . During the inference stage, we use the December 2018 Wikidump as the corpus index for NQ, TQA, HotpotQA, 2WikimQA, and use the December 2020 Wikidump for PopQA, following (Asai et al., 2024a). By default, we follow (Wang et al., 2024; Lin et al., 2024; Liu et al., 2024) to use the Dragon retriever (Lin et al., 2023) as default and retrieve topN ( 100 for 8B and 30 for 70B) documents for ranking, but RankRAG can be adapted to various retrievers and different N (see § 5.3 and 5.5). To ensure a fair comparison, we test the performance of k ∈ { 5 , 10 , 20 } and report the best performance for baselines. For generation, we keep temperature T = 0 and set the maximum number of generated token to be 32 for OpenQA, 128 for ConvQA and 8 for others. Training RankRAG-8B uses 32 NVIDIA A100 GPUs for 10 hours (4 hours for Stage-I and 6 hours for Stage-II finetuning), while training RankRAG-70B uses 128 NVIDIA A100 GPUs for 16 hours (4 hours for Stage-I and 12 hours for Stage-II Finetuning).

Data Contamination Issues. One possible issue for the zero-shot evaluation is the test set contamination, where some of the task-specific examples overlap with the instruction fine-tuning data (Oren et al., 2024). To address this issue, we have performed a string match-based analysis where we do not observe any overlap between the train data and data from target tasks.

5.2 Main Experiments

Table 2 presents results of RankRAG and baselines. The findings are summarized as follows:

RankRAG outperforms existing RAG methods. With 8B scale, RankRAG consistently outperforms ChatQA-1.5 8B, one of the most recent open-sourced models with state-of-the-art performance on many RAG benchmarks. RankRAG 8B is also competitive when compared with baseline models with many more parameters. For example, it significantly outperforms InstructRetro ( 5 × parameters), RA-DIT 65B ( 8 × parameters), and even outperforms Llama3-instruct 70B ( 8 × parameters) on NQ and TriviaQA tasks. With more parameters, RankRAG 70B outperforms the strong ChatQA-1.5 70B model, and largely outperforms previous RAG baselines with InstructGPT as the underlying LLM.

RankRAG demonstrates larger improvement on more challenging datasets. We observe that the performance gains of RankRAG over baselines are more pronounced for more challenging QA datasets. For example, on long-tailed QA (PopQA) and multi-hop QA (2WikimQA) tasks, we achieve more than 10% improvement over ChatQA-1.5. These findings suggest that in challenging OpenQA datasets where top documents from retrievers are less relevant to the answer, context ranking effectively enhances performance. In this work we focus on improving single-time retrieval for QA tasks. How to effectively combine multi-round RAG pipelines (Jiang et al., 2023; Khattab et al., 2022; Jeong et al., 2024) with RankRAG is an interesting avenue of future work.

5.3 Ablation Studies

Effect of Designed Components. Table 3 shows the ablations of RankRAG with Llama3 8B as the backbone on nine general-domain datasets. Overall, we observe all of the proposed components

contribute to the final performance. Removing context ranking hurts performance on all tasks, justifying its efficacy in selecting the most relevant contexts for the target question. Besides, the retrieval-augmented QA (RQA) and retrieval-augmented ranking (RAR) designed for instruction finetuning improve outcomes on most tasks by helping the model explicitly pinpoint relevant contexts. On the contrary, the RAFT method used in (Lin et al., 2024) treats each retrieved context separately during instruction finetuning, which yields suboptimal results when compared to RankRAG with the same training data.

Performance with Different LLMs. Table 4 reports the performance of RankRAG and the most recent baseline ChatQA using Llama2 with backbone having varying amounts of parameters. Notably, there exist consistent gains in terms of the average performance (7.8%/6.4%/6.3% on 7B/13B/70B variants respectively), justifying the advantage of RankRAG across different LLM types and scales.

Performance with Different Retrievers. Figure 3 exhibits the performance of RankRAG and ChatQA1.5 with different dense retrievers on three representative tasks, where we consider DPR (Karpukhin et al., 2020) and Contriever-MS MARCO (Izacard et al., 2022) as two variants. We note that although the initial retrieved result is not good enough, RankRAG still surpasses ChatQA-1.5 by more than 10% for both retrievers on average. To summarize, RankRAG is robust to the choice of retrievers.

NQ

TQA

PopQA

20

40

60

Exact Match

ChatQA-1.5

RankRAG

(a) DPR

NQ

TQA

PopQA

20

40

60

80

Exact Match

ChatQA-1.5

RankRAG

(b) Contriever

5.4 Experiment on Domain-specific RAG Benchmarks

To demonstrate that RankRAG can adapt to specialized domains, we conduct experiments on Mirage (Xiong et al., 2024), a recently introduced RAG benchmark for the biomedical field. We follow Xiong et al. (2024) to employ MedCPT (Jin et al., 2023) as the retriever R with MedCorp 3 as the corpus D .

The experiment results of RankRAG and baselines are shown in Table 5. From the table, we observe that RankRAG, even without fine-tuning on the biomedical domain, excels at medical QA tasks.

Notably, RankRAG 8B surpasses Meditron 70B-a leading open-source LLM for the medical domain-by 6.3%. Besides, RankRAG 70B attains more than 98% performance of GPT-4. These results justify RankRAG's capacity to be readily applied to new domains without extra post-training.

5.5 A Closer Look at the Ranking Module

As the context ranking serves as a core step in RankRAG, we take a closer look at this component. All the studies are done using Llama3-8B as the backbone.

RankRAG is Data-efficient. Previous approaches that infuse context ranking into the RAG pipeline usually involve a separate reranking model. To compare our model with these baselines, we evaluate four models (BERT (Glass et al., 2022)/T5 (Nogueira et al., 2020)/Llama3 (Ma et al., 2023)) fine-tuned on the full MS MARCO passage ranking dataset, a strong off-the-shelf reranker model BGE-ranker, and two OpenAI GPT-series models. For the GPT-series models, we use the token probability of ' True ' as a proxy for the relevance score 4 . These models are then used to rerank top-retrieved passages by Dragon, similar to our approach. Surprisingly, as shown in Table 6, RankRAG achieves better recall over dedicated ranking models trained on 10 × more ranking data for most cases. Besides, RankRAG can still outperform the BGE-ranker on most tasks, which has been extensively trained on more than 1 million ranking pairs, including some that overlap with our evaluation tasks. This advantage is likely due to the adaptable nature of our model's training, where the ranking data closely resembles the general RAG fine-tuning data. Directly using ChatQA-1.5 to rank passages hurts the performance, indicating the necessity of incorporating ranking data into instruction fine-tuning.

We further study the relation between the number of context ranking data and final performance. As shown in Figure 4, with 5k ranking data only ( ∼ 1% of the MS MARCO dataset), RankRAG can already obtain very compelling results, while further increasing the number of ranking data to 50k yields non-marginal gains. This finding confirms RankRAG's data efficiency - achieving effective performance with a modest amount of ranking data and maintaining adaptability across various tasks.

Performance v.s. Time-efficiency for RankRAG. One specific caveat for scaling up model size is the increment in the latency overhead - as mentioned in §4.3, it requires sample-wise ranking which incurs additional time. To study the relation between the time efficiency and performance, we change the N used in reranking and plot the relation of N and final accuracy in Figure 5, from which we observe that even with N = 20 , RankRAG still improve the baseline model without reranking. While reranking across N = 20 to 100 improves the exact match score by 5.9% to 9.1% across three tasks, it incurs an additional 0 . 9 × to 6 . 0 × increase in time significantly less than the 20 × to 100 × increase one might expect.

5.6 Case Study

Table 7 presents a case study on NQ dataset, where we observe that using retriever only yield noisy contexts, as there are several distractors, and some contexts (e.g. Passage 4/5 for ChatQA-1.5) are

0k

5k

10k

50k

40

45

50

Exact Match

ChatQA-1.5 (NQ)

RankRAG (NQ)

Figure 4: Performance

0

2

4

6

8

10

Inference Time

42

45

48

51

Exact Match

N=30

N=40

N=50

N=100

ChatQA-1.5

RankRAG

(a) NQ

0

2

4

6

8

10

Inference Time

81

82

83

Exact Match

N=30

N=40

N=50

N=100

ChatQA-1.5

RankRAG

(b) TriviaQA

0

2

4

6

8

10

Inference Time

34

35

Exact Match

N=30

N=40

N=50

N=100

ChatQA-1.5

RankRAG

(c) HotpotQA

Figure 5: Performance v.s. Efficiency analysis for RankRAG.

Q : who hosted and won the inagural world cup? A : Uruguay

Passage 1 : FIFA World Cup second round on home soil in 1982. England (1966) won its only title while playing as a host nation. Uruguay (1930), Italy (1934), Argentina (1978) and France (1998) won their first titles as host nations but have gone on to win again, while Germany (1974) won their second title on home soil...

Passage 2 : FIFA World Cup hosts country is now chosen in a vote by FIFA's Congress ... Only Mexico, Italy, France, Germany (West Germany) until shortly after the 1990 World Cup) and Brazil have hosted the event on two occasions.

Passage 3 : CONCACAF hosts, beating the bids of Canada and the United States, and thereby became the first nation to host two World Cups. This second World Cup in Mexico came 16 years after the first one in 1970...

Passage 4 : 1998 FIFA World Cup Africa made their first appearances in the finals. France was awarded the 1998 World Cup on 2 July 1992 by the executive committee of FIFA during a general meeting in Zürich, Switzerland. They defeated Morocco by 12 votes to 7.

Passage 5 : 2026 FIFA World Cup be hosted by one of the remaining four confederations: CONCACAF (North America; last hosted in 1994), CAF (Africa; last hosted in 2010), CONMEBOL (South America; last hosted in 2014), or OFC (Oceania, never hosted before)... Prediction : Germany ( ✘ )

Passage 1 : FIFA World Cup second round on home soil in 1982. England (1966) won its only title while playing as a host nation. Uruguay (1930), Italy (1934), Argentina (1978) and France (1998) won their first titles as host nations but have gone on to win again, while Germany (1974) won their second title on home soil...

Passage 2 : Timeline of association football penalty kicks. Thirteen teams enter the first World Cup, held in Uruguay. The hosts beat Argentina 4-2 in the final. Contested between the top national teams of continental Europe, Dr. Gerö Cup' first edition is won by Italy.

Passage 3 : The Uruguay national football team represents Uruguay in international association football and is controlled by the Uruguayan Football Association. They have won the Copa América 15 times, the most successful national team in the tournament, the most recent title being the 2011 edition. The team has won the FIFA World Cup twice, including the first World Cup in 1930 as hosts, defeating Argentina 4-2 in the final.

Passage 4 : FIFA World Cup hosts country is now chosen in a vote by FIFA's Congress. The decision is currently made roughly seven years in advance of the tournament, though the hosts for the 2022 tournament were chosen at the same time as those for the 2018 tournament.

Passage 5 : CONCACAF hosts, beating the bids of Canada and the United States, and thereby became the first nation to host two World Cups. This second World Cup in Mexico came 16 years after the first one in 1970... Prediction : Uruguay ( ✓ )

unhelpful. However, the utilization of reranking uncovers two additional relevant passages, aiding the model in providing the correct answer. More case studies are provided in Appendix G.

6 Conclusion

In this work, we introduce a new RAG framework, RankRAG, which instruction-tunes a single LLM for both ranking and answer generation. We find that the instruction tuned LLMs can outperform existing expert ranking models by only adding a small fraction of ranking data into the training blend. We compare our RankRAG with the state-of-the-art RAG models on comprehensive knowledgeintensive benchmarks and demonstrate RankRAG significantly outperform all of them on nine general-domain and five biomedical benchmarks for RAG.

References

Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., and Reddy, S. Topiocqa: Open-domain conversational question answering with topic switching. TACL , 2022.

Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Anthropic. Model card and evaluations for claude models. 2023.

Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR , 2024a.

Asai, A., Zhong, Z., Chen, D., Koh, P. W., Zettlemoyer, L., Hajishirzi, H., and Yih, W.-t. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03187 , 2024b.

ChatQA-1.5

RankRAG

Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 , 2016.

Berant, J., Chou, A., Frostig, R., and Liang, P. Semantic parsing on freebase from question-answer pairs. In EMNLP , 2013.

Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In ICML . PMLR, 2022.

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2023a.

Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 , 2023b.

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. JMLR , 25(70), 2024.

Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free Dolly: Introducing the world's first truly open instruction-tuned llm, 2023.

Dasigi, P., Liu, N. F., Marasovi´ c, A., Smith, N. A., and Gardner, M. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In EMNLP , 2019.

DeepSeek. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.

Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of-experts. In ICML , 2022.

Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL , 2019.

Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., and Auli, M. Eli5: Long form question answering. In ACL , 2019.

Feng, S., Wan, H., Gunasekara, C., Patel, S., Joshi, S., and Lastras, L. doc2dial: A goal-oriented document-grounded dialogue dataset. In EMNLP , 2020.

Glass, M., Rossiello, G., Chowdhury, M. F. M., Naik, A., Cai, P., and Gliozzo, A. Re2G: Retrieve, rerank, generate. In NAACL , 2022.

Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In ICML , 2020.

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In ICLR , 2021.

Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING , 2020.

Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. In ACL , 2023.

Huang, J., Ping, W., Xu, P., Shoeybi, M., Chang, K. C.-C., and Catanzaro, B. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.07922 , 2023.

Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In EACL , 2021.

Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense information retrieval with contrastive learning. TMLR , 2022.

Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot learning with retrieval augmented language models. JMLR , 24(251):1-43, 2023.

Jeong, S., Baek, J., Cho, S., Hwang, S. J., and Park, J. C. Adaptive-rag: Learning to adapt retrievalaugmented large language models through question complexity. In NAACL , 2024.

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. In EMNLP , 2023.

Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences , 11(14):6421, 2021.

Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. Pubmedqa: A dataset for biomedical research question answering. In EMNLP , 2019.

Jin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., and Lu, Z. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics , 39(11), 2023.

Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL , 2017.

Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In EMNLP , 2020.

Kasai, J., Sakaguchi, K., yoichi takahashi, Bras, R. L., Asai, A., Yu, X. V ., Radev, D., Smith, N. A., Choi, Y., and Inui, K. Realtime QA: What's the answer right now? In NeurIPS , 2023.

Khalifa, M., Logeswaran, L., Lee, M., Lee, H., and Wang, L. Few-shot reranking for multi-hop QA via language model prompting. In ACL , 2023.

Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. Demonstratesearch-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024 , 2022.

Kim, H., Hessel, J., Jiang, L., Lu, X., Yu, Y., Zhou, P., Bras, R. L., Alikhani, M., Kim, G., Sap, M., et al. Soda: Million-scale dialogue distillation with social commonsense contextualization. In EMNLP , 2023.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Koˇ cisk` y, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The narrativeqa reading comprehension challenge. TACL , 2018.

Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research. TACL , 2019.

Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations - democratizing large language model alignment. arXiv preprint arXiv: 2304.07327 , 2023.

Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 , 2022.

Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catanzaro, B., and Ping, W. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428 , 2024.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS , 33, 2020.

Lin, K., Tafjord, O., Clark, P., and Gardner, M. Reasoning over paragraph effects in situations. In Workshop on Machine Reading for Question Answering , 2019.

Lin, S.-C., Asai, A., Li, M., Oguz, B., Lin, J., Mehdad, Y ., Yih, W.-t., and Chen, X. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. In Findings of EMNLP , 2023.

Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., Zettlemoyer, L., and tau Yih, W. RA-DIT: Retrieval-augmented dual instruction tuning. In ICLR , 2024.

Liu, Z., Ping, W., Roy, R., Xu, P., Shoeybi, M., and Catanzaro, B. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401.10225 , 2024.

Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., et al. The flan collection: Designing data and methods for effective instruction tuning. In ICML , 2023.

Luan, Y., Eisenstein, J., Toutanova, K., and Collins, M. Sparse, dense, and attentional representations for text retrieval. TACL , 2021.

Luo, H., Chuang, Y.-S., Gong, Y., Zhang, T., Kim, Y., Wu, X., Fox, D., Meng, H., and Glass, J. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225 , 2023.

Ma, X., Wang, L., Yang, N., Wei, F., and Lin, J. Fine-tuning llama for multi-stage text retrieval. arXiv preprint arXiv:2310.08319 , 2023.

Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL , 2023.

Menon, A., Jayasumana, S., Rawat, A. S., Kim, S., Reddi, S., and Kumar, S. In defense of dualencoders for neural ranking. In ICML , 2022.

Meta-AI. Llama 3 model card. 2024.

Mistral. Mixtral 8x22b. 2024. URL https://mistral.ai/news/mixtral-8x22b/ .

Mitra, B., Craswell, N., et al. An introduction to neural information retrieval. Foundations and Trends® in Information Retrieval , 2018.

Muennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906 , 2024.

Nogueira, R., Jiang, Z., Pradeep, R., and Lin, J. Document ranking with a pretrained sequence-tosequence model. In Findings of EMNLP , 2020.

OpenAI. Introducing ChatGPT, 2022.

OpenAI. GPT-4, 2023.

Oren, Y., Meister, N., Chatterji, N. S., Ladhak, F., and Hashimoto, T. Proving test set contamination in black-box language models. In ICLR , 2024.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NeurIPS , 35, 2022.

Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In CHIL , 2022.

Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y ., Karpukhin, V., Maillard, J., Plachouras, V ., Rocktäschel, T., and Riedel, S. KILT: a benchmark for knowledge intensive language tasks. In NAACL , 2021.

Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., et al. Large language models are effective text rankers with pairwise ranking prompting. In Findings of NAACL , 2024.

Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In EMNLP , 2016.

Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., and Shoham, Y. In-context retrieval-augmented language models. TACL , 2023.

Robertson, S., Zaragoza, H., and Taylor, M. Simple bm25 extension to multiple weighted fields. In CIKM , 2004.

Sachan, D. S., Reddy, S., Hamilton, W. L., Dyer, C., and Yogatama, D. End-to-end training of multi-document reader and retriever for open-domain question answering. In NeurIPS , 2021.

Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., and Chen, W. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of EMNLP , 2023.

Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-augmented black-box language models. In NAACL , 2024.

Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. Is ChatGPT good at search? investigating large language models as re-ranking agents. In EMNLP , 2023.

Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS , 2021.

Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. Fever: A large-scale dataset for fact extraction and verification. In NAACL , 2018.

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., and Suleman, K. Newsqa: A machine comprehension dataset. In RepL4NLP Workshop at ACL , 2017.

Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Interleaving retrieval with chain-ofthought reasoning for knowledge-intensive multi-step questions. In ACL , 2023.

Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics , 2015.

Wang, B., Ping, W., McAfee, L., Xu, P., Li, B., Shoeybi, M., and Catanzaro, B. Instructretro: Instruction tuning post retrieval-augmented pretraining. In ICML , 2024.

Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 , 2022.

Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368 , 2023a.

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In ACL , 2023b.

Wang, Z., Araki, J., Jiang, Z., Parvez, M. R., and Neubig, G. Learning to filter context for retrievalaugmented generation. arXiv preprint arXiv:2311.08377 , 2023c.

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In ICLR , 2022.

Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., and Wang, Y. Pmc-llama: toward building opensource language models for medicine. JAMIA , 2024.

Wu, Z., Parish, R., Cheng, H., Min, S., Ammanabrolu, P., Ostendorf, M., and Hajishirzi, H. Inscit: Information-seeking conversations with mixed-initiative interactions. TACL , 2023.

Xiong, G., Jin, Q., Lu, Z., and Zhang, A. Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178 , 2024.

Xu, F., Shi, W., and Choi, E. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In ICLR , 2024a.

Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. In ICLR , 2024b.

Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP , 2018.

Yoran, O., Wolfson, T., Ram, O., and Berant, J. Making retrieval-augmented language models robust to irrelevant context. In ICLR , 2024.

Yu, W., Iter, D., Wang, S., Xu, Y., Ju, M., Sanyal, S., Zhu, C., Zeng, M., and Jiang, M. Generate rather than retrieve: Large language models are strong context generators. In ICLR , 2023a.

Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., and Yu, D. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210 , 2023b.

Yu, W., Zhang, Z., Liang, Z., Jiang, M., and Sabharwal, A. Improving language models via plug-andplay retrieval feedback, 2024.

Yu, Y., Xiong, C., Sun, S., Zhang, C., and Overwijk, A. Coco-dr: Combating distribution shift in zero-shot dense retrieval with contrastive and distributionally robust learning. In EMNLP , 2022.

Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., and Gonzalez, J. E. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131 , 2024.

Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., and Chua, T.-S. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In ACL , 2021.

Zhu, Y., Zhang, P., Zhang, C., Chen, Y., Xie, B., Dou, Z., Liu, Z., and Wen, J.-R. Inters: Unlocking the power of large language models in search with instruction tuning. In ACL , 2024.

A Dataset Description

The information for 14 datasets used in RankRAG is listed as follows.

A.1 Main Experiments

NQ (Kwiatkowski et al., 2019) is a widely used question-answering dataset constructed with Wikipedia. The questions are constructed from the Google search engine, and the answers are identified as text spans in the Wikipedia article.

TriviaQA (Joshi et al., 2017) is a challenging QA dataset containing question-answer pairs from trivia enthusiasts and independently gathered evidence documents.

PopQA (Mallen et al., 2023) is an entity-centric QA dataset concentrated on long-tail entities. For PopQA, we follow (Asai et al., 2024a) to use the long-tail subset, consisting of questions on 1399 rare entities whose monthly Wikipedia page views are less than 100.

HotpotQA (Yang et al., 2018) is a multi-hop QA dataset, where the goal is to answer complex questions that require understanding and linking information from multiple documents.

2WikimQA (Ho et al., 2020) is also a multi-hop QA designed to test machine understanding across two different Wikipedia entities, evaluating the ability of systems to handle cross-lingual and cross-cultural retrieval and question answering.

FEVER (Thorne et al., 2018) is a fact verification dataset aimed at supporting research into the automatic verification of factual claims. It consists of claims that are manually verified against evidence from Wikipedia, providing a benchmark for fact-checking systems.

Doc2Dial (Feng et al., 2020) is a document-grounded conversational QA dataset covering four domains: DMV, SSA, VA, and Student Aid. Each sample comprises a dialogue where a user poses queries regarding the document, and an agent responds those questions. The average document length is around 101K words.

TopiOCQA (Adlakha et al., 2022) is grounded on the whole Wikipedia. It incorporates topic switching and requires the agent to search the entire Wikipedia for answers to user questions.

INSCIT (Wu et al., 2023) is also grounded on the whole Wikipedia. It studies the case where user questions are under-specified and require clarification.

A.2 Biomedical Benchmarks

MMLU-med (Hendrycks et al., 2021) is a subset of six tasks related to biomedicine, including anatomy, clinical knowledge, professional medicine, human genetics, college medicine, and college biology. It contains 1089 questions in total.

MedQA (Jin et al., 2021) is collected from the US Medical Licensing Examination, contaiing 1273 four-option multiple-choice questions focused on real-world scenarios from professional medical board exams.

MedMCQA (Pal et al., 2022) includes multiple-choice questions derived from Indian medical entrance exams, covering 2400 healthcare topics across 21 medical subjects. We use the 4,183question development set from MedMCQA, as the test set lacks provided ground truths.

PubmedQA (Jin et al., 2019) is a biomedical research QA dataset consisting of 1000 manually annotated questions based on PubMed abstracts. Answers in PubMedQA are structured as yes/no/maybe to reflect the validity of the questions.

BioASQ (Tsatsaronis et al., 2015) includes 618 questions constructed from biomedical literature without providing the ground truth snippets, challenging RAG systems to infer answers independently.

B Data Blending Details for Ranking-enhanced Instruction Finetuning

The dataset blending ratio for Stage-II is as follows:

Drop: 0.069

narrativeqa: 0.09 · quoref: 0.026 · ropes: 0.026 · Squad (Retrieval-augmented QA): 0.09 · Squad (Retrieval-augmented Ranking): 0.02 · WebQuestions (Retrieval-augmented QA): 0.09 · WebQuestions (Retrieval-augmented Ranking): 0.02 · newsqa: 0.09 · tatqa-arithmetic: 0.15 · tatqa-others: 0.08 · ConvQA: 0.2 · MS MARCO ranking: 0.15 · ConvQA ranking: 0.03

· SFT: 0.2 The ratio for each dataset is further normalized to ensure the total ratio equals to 1. C Prompt Formats of Instruction Tuning C.1 Stage I: Supervised Fine-tuning The format template of LLM inputs in stage-I is as follows: System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context. User: {Question 1} Assistant: {Answer 1} ... User: {Latest Question} Assistant: C.2 Stage-II: Unified Instruction-Tuning for Ranking and Generation be

found in the context. Passage 1: {(Shuffled) Passage 1} Passage 2: {(Shuffled) Passage 2} Passage 3: {(Shuffled) Passage 3} Passage 4: {(Shuffled) Passage 4} Passage 5: {(Shuffled) Passage 5} User: {For the question <question>, access whether the above passages are relevant to the question. Return all the relevant passage id. } Assistant: D Prompt Formats of Target Tasks D.1 Context Ranking NQ/TriviaQA/HotpotQA/PopQA: System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context. Passage: {Passage} User: {For the question <question>, access whether the passage is relevant to the question. Return True if relevant, otherwise False. } Assistant: FEVER: System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context. Passage: {Passage} User: {For the claim <claim>, access whether the passage is relevant to the claim. Return True if relevant, otherwise False. } Assistant: Doc2dial, Inscit, TopiocQA: System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context. Passage: {Passage}

User: {Question 1}

The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context.

E Additional Experiment Results

E.1 Ranking Performance Using DPR and Contriever as Retrievers R

Table 8 shows the ranking performance of RankRAG-8B using DPR (Karpukhin et al., 2020) and Contriever (Izacard et al., 2022) on three datasets. There are consistent performance gains for all tasks, indicating that RankRAG can apply to many popular retrieval models to improve the quality of retrieved contents.

E.2 RAG Performance with Different k

We also show the performance of RankRAG with different context size k in figure 6. From the result, we observe that different from the trend of vanilla RAG approaches (without ranking), k = 5 already works well for most datasets. This effectiveness stems from the reranking step, which prioritizes the most relevant contexts at the top, reducing the necessity to include additional contexts.

5

10

20

46

48

50

52

Exact Match

NQ

5

10

20

80

82

84

TriviaQA

5

10

20

50

55

60

PopQA

5

10

20

90

91

92

93

FEVER

Context Size

k

F Performance of NQ and Trivia QA on DPR Splits

We observe that the NQ and TriviaQA datasets exist in two versions: one used by the DPR (Karpukhin et al., 2020) and FiD (Izacard & Grave, 2021) papers, which include 3610 and 11316 questions for NQ and TriviaQA, respectively. In contrast, the KILT benchmark (Petroni et al., 2021) utilizes only subsets of these, comprising 2837 and 5355 examples for NQ and TriviaQA, respectively. It is noteworthy that many recent studies report performance metrics on these datasets without clarifying which version was employed for evaluation.

To facilitate an honest and fair comparison, we present the performance of RankRAG on both datasets using the DPR splits in Table 9. Notably, regardless of the subset used, RankRAG consistently outperforms both ChatQA and Llama-3-instruct, our direct competitors, as well as other methods utilizing InstructGPT as backbones. We aim for these results to assist the community in making accurate comparisons when referring to the performance of RankRAG.

G Additional Case Studies

Tables 10 and 11 provide additional examples from the PopQA and HotpotQA datasets, which focus on long-tailed and multi-hop QA. These tasks are particularly challenging for retrievers, making it difficult to obtain relevant context from the corpus. Consequently, ChatQA-1.5 often struggles to produce the correct answers. However, the reranking step in RankRAG helps counteract poor initial

Q : Who is the author of The Universe Around Us? A : James Hopwood Jeans / James Jeans / Sir James Jeans / Sir James Hopwood Jeans

ChatQA-1.5 Passage 3 : The Universe" and "The Planets." With Sidney Wolff, Fraknoi founded and was co-editor of the first on-line journal devoted to astronomy education, "Astronomy Education Review". He edited two collections of resources for K-12 teachers, The Universe at Your Fingertips and More Universe at Your Fingertips published through the Astronomical Society of...

Passage 1 : Hooper is the author of two books published by Smithsonian Books/HarperCollins. The first, Dark Cosmos: In Search of our Universe's Missing Mass and Energy (2006) was named a notable book by Seed Magazine. His second book, Nature's Blueprint: Supersymmetry and the Search for a Unified Theory of Matter and Force (2008), was called "essential reading" by New Scientist...

Passage 2 : Fraknoi is the author or co-author of 14 books in the field of astronomy. He was the lead author of Voyages through the Universe, an introductory college astronomy textbook published by Brooks-Cole, which went through three editions. In the 1980s, he co-edited with Byron Preiss two collections of science articles and science fiction stories, "The Universe" and "The Planets." With Sidney Wolff, Fraknoi founded and was co-editor of the first on-line journal devoted to astronomy education, "Astronomy Education Review"...

Passage 4 : Lincoln Kinnear Barnett (1909-1979) was an editor and author, most notably at Life Magazine for many years. Lincoln Barnett wrote a number of books, including "The Universe and Doctor Einstein"

Passage 5 : The Universe Maker is a science fiction novel by American author A.E. van Vogt, published in 1953 by Ace Books as an Ace Double with The World of Null-A. It is based on the author´ s "The Shadow Men" (Startling Stories, 1950). Set 400 years into the future, the main character is Morton Cargill, a U.S. Army officer who served in the Korean War... Prediction : Lincoln Barnett ( ✘ )

RankRAG

Passage 1 : The Universe Around Us is a science book written by English astrophysicist Sir James Jeans, first published in 1929 by the Syndics of the Cambridge University Press....

Passage 2 : These books made Jeans fairly well known as an expositor of the revolutionary scientific discoveries of his day, especially in relativity and physical cosmology. In 1939, the Journal of the British Astronomical Association reported that Jeans was going to stand as a candidate for

Passage 3 : James Jeans books for the lay public, including "The Stars in Their Courses" (1931), "The Universe Around Us", "Through Space and Time" (1934), "The New Background of Science" (1933), and "The Mysterious Universe.

Passage 4 : The Universe Around Us no special scientific knowledge. Parts of the book cover the same ground as various lectures I have recently delivered to University and other audiences, including a course of wireless talks I gave last autumn. It has been found necessary to rewrite these almost in their entirety, so that very few sentences remain in their original form, but those who have asked me to publish my lectures and wireless talks will find the substance of them in the present book.

Passage 5 :Lincoln Barnett Lincoln Kinnear Barnett (1909-1979) was an editor and author, most notably at Life Magazine for many years. Lincoln Barnett wrote a number of books, including "The Universe and Doctor Einstein", "The World We Live In", and "The Treasure of Our Tongue". The Universe and Doctor Einstein is a layman´ s introduction to the theory of relativity. It includes a foreword by Albert Einstein, and has been reprinted several times... Prediction : Sir James Jeans ( ✓ )

Q : Which song did Eminem and Rihanna collaborate on after their other collaboration song in studio album "Unapologetic?" A : The Monster

Passage 1 : Unapologetic is the seventh studio album by Barbadian singer Rihanna. It was released on November 19, 2012, by Def Jam Recordings and SRP Records. It was recorded between June and November 2012, during promotion of her sixth album, "Talk That Talk" (2011). As executive producer, Rihanna enlisted previous collaborators The-Dream, David Guetta, Chase & Status, and StarGate to work alongside new collaborators such as Parker Ighile, Mike Will Made-It, and Labrinth...

Passage 2 : Def Jam France announced via Twitter that Rihanna would release a new single the upcoming week while her seventh studio album was scheduled to be released in November 2012. On October 11, 2012, in one of her tweets revealed that the title of her new album is "Unapologetic" alongside with its cover. "What Now" was written by British singer-songwriter Livvi Franc together with Rihanna, Parker Ighile and Nathan Cassells, while the production of the song was done by the latter two. Ighile and Cassells...

ChatQA-1.5 Passage 3 : Justin then went on to co write "Stay" with Mikky Ekko and recorded by Barbadian singer Rihanna for her seventh studio album, "Unapologetic" (2012). It features guest vocals by Mikky Ekko, and was released as the second single from the album on 7 January 2013. The song reached the top five of twenty-four countries worldwide including number four in the UK and number three on the US Billboard Hot 100, becoming Rihanna's twenty-fourth top ten on the latter chart...

Passage 4 : Via her official Twitter account, Rihanna posted series of "teasing" tweets announcing her seventh studio album. On October 11, 2012, in one of her tweets revealed that the title of her new album is "Unapologetic" alongside with its cover. "Jump" is the overall seventh and final single off Unapologetic. It was written by Kevin Cossom and M. B. Williams together with its producers StarGate (Mikkel S. Eriksen and Tor Erik Hermansen) and Chase & Status (Saul Milton

Passage 5 : copies of the song were sold in the UK, making "Love the Way You Lie" the country's biggest-selling song of 2010. The same year, a sequel to the single, titled "Love the Way You Lie (Part II)" was released as part of Rihanna's fifth studio album "Loud"; it mainly views matters from the female protagonist perspective. In November 2012, Eminem and Rihanna collaborated again on "Numb", which was included on Rihanna's seventh album "Unapologetic"... Prediction : Love the Way You Lie ( ✘ )

Passage 1 : The Monster (song). Copies of the song were sold in the UK, making "Love the Way You Lie" the country's biggest-selling song of 2010. The same year, a sequel to the single, titled "Love the Way You Lie (Part II)" was released as part of Rihanna's fifth studio album "Loud"; it mainly views matters from the female protagonist perspective. In November 2012, Eminem and Rihanna collaborated again on "Numb", which was included on Rihanna's seventh album "Unapologetic"....

Passage 2 : "Numb" is a song by Barbadian singer Rihanna from her seventh studio album "Unapologetic" (2012). It features guest vocals by American rapper Eminem, making it the pair's third collaboration since the two official versions of "Love the Way You Lie". Following the album's release, "Numb" charted on multiple charts worldwide including in Canada, the United Kingdom and the United States. "Numb" lasts for a duration of .

Passage 3 : Eminem also wanted to experiment with "retro, vintage" sounds such as beatbreaks and scratches, and he felt that Rubin could help him "take that to another level." Rihanna, with whom Eminem previously collaborated on "Love the Way You Lie" from Eminem's previous studio effort, "Recovery" (2010), was featured on the song "The Monster". On September 11, 2013, she hinted at the...

Passage 4 : together with Jay-Z, Bono and The Edge for the same campaign to alleviate the 2010 Haiti earthquake. In summer 2010, Rihanna collaborated with rapper Eminem on "Love the Way You Lie", which was a major worldwide success, reaching No. 1 in over 20 countries. Reaching number 2, the song became the biggest-selling song of 2010 in the UK and the first of Rihanna's singles to sell over a million copies in the country. In October 2010, Rihanna switched managers ...

Passage 5 : Eminem asked for more tracks and subsequently heard "Love the Way You Lie". He chose it and told his manager Paul Rosenberg he wanted to collaborate with the Barbadian singer Rihanna. Eminem told Skyrock, "It's one of those tracks that I felt like only she could pull it off." Rosenberg sent the track to Rihanna, who accepted Eminem's request "at the last moment." Eminem then wrote the rapped verses. Prediction : The Monster ( ✓ )

retrieval by finding more pertinent evidence. Coupled with RAG-oriented finetuning, RankRAG effectively filters out distracting entities and pinpoints the correct answers.

RankRAG

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have comprehensive experimental results in § 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of RankRAG in Appendix ?? .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not propose any theory assumptions and does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the details of the data collection for training and evaluations, which can be found in section 4, 5. We will open-source model weights and scripts for reproducing our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We open-source model weights and scripts for reproducing our results. For training data, they are all public available data with open access.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details can be found in section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: All results are zero-shot and deterministic; we use greedy search for generations. Retrieval and re-ranking scores are also deterministic.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the compute resources information in section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research is conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss both potential societal impacts and negative impacts in Appendix ?? .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all data, models used in the paper properly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We did not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a method about how to use a single LLM for both context reranking and answer in RAG tasks. Particularly, they finetuned a LLM with both ranking data and QA data with two stages. For inference, they use the trained llm to sample the top_k contexts at first, and then input them into llm to get the answer. Compared with normal rag workflow, this method choose the contexts by llm itself. Experiments in some QA tasks evaluated the QA effectiveness of this LLM.\n\nSTRENGTHS:\n1. Two stages of training enhance the reranking capacity of llm using fewer ranking data.\n2. Using just a single llm for context ranking and answer at the same time.\n\nWEAKNESSES:\n1. The motivation is unclear. In lines 23 to 31, limitations 1 and 2 precisely explain the need for a reranker, which is also a challenge related to retrieval. Only with limited text in limitation 3  briefly mentions why the LLM is used for reranking contexts due to zero-shot performance, which has been proved with other rerankers in this paper. It is unclear about why using llm reranking itself rather than a separate reranker, maybe including the semantic gap between the reranker and llm or the larger length of reranking input at once.\n2. Insufficient experiments in same reranking setting. It is lack of baselines with RAG methods also using a reranker, which can prove the core reranking effectiveness of RankRAG, such as rankgpt(Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents) and rankvicuna(RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models). And also, the amount of top-k contexts after reranking is also hard to find in Table 2&6. How many contexts for reranking are input into llm at once? In addition, there are some blank spaces in Table 2 that are not convincing. \n3. Incremental techniques. Compared with ChatQA, RankRAG introduced the new training data RQA and RAR in stage 2th and reranking phrase during inference. But as shown in ablation study, each component contribute a little.\n\nQUESTIONS:\n1. Can you analyze the relationship between high retrieval call and QA performance in RankRAG for limitation2 on line 26? Can you provide further analysis to demonstrate that your method can solve this limitation?\n\n2. Have you conducted experiments to test whether RankRAG's reranking performance is robust when using different retrievers on lines 300?\n\nLIMITATIONS:\nNo potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel approach to instruction fine-tuning large language models (LLMs) for ranking and answer generation tasks.\nTheir approach involves two main steps:\n1. Supervised Fine-Tuning: Initially, the LLM is fine-tuned on a general instruction-following dataset.\n2. Ranking Task Fine-Tuning: The LLM is then further fine-tuned using a mix of different instruction ranking datasets that contains multiple ranking-oriented tasks.\n\nIncorporating ranking-based data during fine-tuning enhances the LLM's ability to rank documents retrieved through retrieval-augmented generation (RAG). The fine-tuned LLM is then used to rank the RAG-retrieved documents, and only the top-k ranked documents are added in the context.\n\nThe authors demonstrate that their method significantly improves the LLM's performance for knowledge intensive RAG based tasks.\n\nSTRENGTHS:\n1. The authors introduce a novel method of fine-tuning LLMs for combined ranking and answer generation tasks.\n2. The proposed method outperforms existing approaches on RAG benchmarks, especially on challenging datasets where RAG retrieval is suboptimal, due to the LLM-based re-ranking.\n3. Their model exceeds the performance of re-ranking models trained on much larger datasets.\n4. The fine-tuned LLM demonstrates strong generalization capabilities, showcased by its performance on medical benchmarks.\n5. The paper is well-written and includes exhaustive experiments.\n\nWEAKNESSES:\n1. Scoring each document individually increases the latency significantly. Have you considered scoring multiple documents simultaneously? Similar to retrieval-augmented ranking dataset, you could input a group of documents and score them in a single pass. The group size could be tunable, balancing performance drop against latency improvement.\n2. It would be helpful to include examples where RankRAG fails, particularly in cases where relevant documents are not ranked higher. Providing these examples can help understand scenarios where it might not perform optimally.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe authors have acknowledged the limitations, specifically in terms of latency and its lack of training on code or mathematical data.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an effective approach that enhances existing RAG methods by introducing an additional context refinement step. This step filters out retrieved, but non-relevant contexts prior to including them as context in the input for answer generation.\n\nThe authors train context reranking alongside answer generation using a single LLM. They demonstrate that adding only a fraction of ranking data to the training blend yields superior ranking performance compared to training the same model in isolation on the ranking data, while outperforming other models that have been trained on 10 times more ranking data.\n\nFor evaluation, the authors compare their method on 9 general- and 5 specific-domain datasets, showing consistent improvements across LLM sizes for the Llama model family (Llama 2 and 3) over existing methods.\n\nSTRENGTHS:\n- The paper introduces the novel idea of using the same LLM to first assess the relevance of individual contexts in a cross-encoder ranking style before using them as input for answer generation.\n\n- The proposed RankRAG outperforms existing methods on various general and specific-domain datasets.\n\n- Extensive experimentation and ablations covers a wide range of possible setups including LLM size, retriever, and efficiency and effect of different model components.\n\nWEAKNESSES:\nMain concern: \n- Reranking contributes only around 5% of the overall effectiveness on average (Table 4 RankRAG compared to Llama3-ChatQA-1.5-X), and the 7x computational overhead in inference time raises questions about its realistic application, given the computational demands of large models already without ranking contexts. The ratio of performance gained ( in combination with no sig. testing) and increase in computation is my main issue with this work. I am aware reranking fewer contexts decreases performance, however, similarity decreases performance gain over other methods. \n\n- No significance testing was done (for both generation on ranking ) to strengthen effectiveness claims as differences for most datasets are minor. Authors \njustify in their additionally provided checklist that sig. testing is not needed as generation and ranking is deterministic, this however, touches upon a different aspect and does not remove the need for testing whether the performance is significantly better than previous methods.\nFurther, improvements in Table 2 stem from NQ and PopQA which are relatively simple datasets that can be answered with a single context, therefore it is not apparent why RankRAG would particularly excel at those. Moreover, the ranking performance for these datasets in Table 6 only marginally improves over other rerankers, therefore it is to be expected to obtain similar gains when replacing the RankRAG reranker module  with other strong rerankers. As a side note averaging over different metrics - even though seen in many recent works - should not be done. \n\n\nOther points to improve upon:\n- No information about the crowdsourced dataset is provided.\n- In Section 4.2.3, the claim that LLM needs to be robust against irrelevant context contradicts the proposal to filter out irrelevant context beforehand.\n\nSome experimental setups are unclear:\n- It is not described how true/false tokens from context ranking are translated into a score that can be used for ranking.\n- In Section 5.1, it is not clear if baselines use different retrieval setups; otherwise, effectiveness claims do not seem valid.\n- It is unclear which number k is eventually used in RankRAG. The paper mentions optimizing for\nk=5,10,20, the baselines. \n\n- The related work section could mention GRITLM as the first model jointly training answer generation and ranking for RAG: Muennighoff, Niklas, et al. \"Generative representational instruction tuning.\" arXiv preprint arXiv:2402.09906 (2024).\n\nIssues in Writing:\n\n- Line 332: \"Observe that even with N = 20, We noted that\" – incomplete sentence.\n\n- Line 41: \"in RAG framework\" -> \"in the RAG framework\".\n\n- Caption Fig 1: \"the strongest RAG model,\" -> \"the strongest RAG models,\".\n\n- Line 100: \"embedding model\" -> \"embedding models\".\n\n- Line 111: \"As illurstraded\" -> \"As illustrated\".\n\n- Line 112: \"one of the strongest model.\" -> \"one of the strongest models\".\n\n- Line 157: \"that, it is\" -> \"that it is\".\n\n- Line 165: \"The LLM need\" -> \"The LLM needs\".\n\n- Line 169: \"ranking data are\" -> \"ranking data is\".\n\nQUESTIONS:\n- Why is a fixed number of contexts used? Some questions might need more contexts such as multi-hop datasets while others such as NQ (single context dataset) would need less. Instead of a score cutoff, why not use a dynamic number of contexts that is determined by the binary true/false label the mode already outputs?\n- In Section 5.4, the paper states that it incurs an additional 1.8× to 7.0× increase in time, significantly less than the 20× to 100× increase one might expect. It is not sufficiently explained why one would expect a 100x increase in time.\n- The approach for reranking context looks at passages individually, but retrieval training included listwise ranking. Was listwise ranking also tried for context reranking?\n\nLIMITATIONS:\nThe authors addressed the limitations adequately, however, could emphasize more on the dramatic inference time increase that results from the reranking step.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-19"}