RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
Authors: Yue Yu
Wei Ping
Zihan Liu
Boxin Wang
Jiaxuan You
Chao Zhang
Mohammad Shoeybi
Bryan Catanzaro
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
Retrieval-augmented Generation
Ranking
Primary area: natural_language_processing

Abstract: Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG).  In this work, we propose a novel method called RankRAG, which instruction-tunes a single LLM for both context ranking and answer generation in RAG.  In particular, the instruction-tuned LLMs work surprisingly well by adding a small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on a large amount of ranking data.  For generation, we compare our model with many strong baselines, including ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks.  Specifically,  our Llama3-RankRAG-8B and Llama3-RankRAG-70B significantly outperform Llama3-ChatQA-1.5-8B and Llama3-ChatQA-1.5-70B, respectively, on nine general knowledge-intensive benchmarks for RAG. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization to new domains.

RankRAG: Unifying Context Ranking with
Retrieval-Augmented Generation in LLMs
Yue Yu* *
Wei Ping NVIDIA
Zihan Liu NVIDIA
Boxin Wang NVIDIA
Jiaxuan You NVIDIA
Georgia Tech
Chao Zhang Georgia Tech
Mohammad Shoeybi
Bryan Catanzaro
NVIDIA
NVIDIA
Abstract
Large language models (LLMS) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG). In this work. we propose novel instruc- tion fine-tuning framework RankRAG, which instruction-tume cettiisoottttstseeeee single LLM for the dual purpose of context ranking and answer generation in RAG. In particular, the instruction-tuned LLMs work surprisingly well by adding small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on large amount of ranking data. For generation, we compare our model with many strong baselines, including GPT-4-0613, GPT-4-aurb-2022--0465
and ChatQA-1.5, open-sourced model with the state-of-the-an performance on RAG benchmarks. Specifically, our L.lama3-Rankt AG significantly outperforms Llama3-ChatQAA models on nine knowledge-inten inddge-ineeeive benchmarks. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization new domains.
and GPT-4
1 Introduction
Retrieval-augmmented generation (RAG) (Lewis al., 2020; Izacard & Grave, 2021; Lin al., 2024; Wang et al., 2024) is a widely used technique for customizing large language models (LLMs) to handle long-tail knowledge (Mallen al., 2023; Asai et al., 2024b), provide up-to-date information (Kasai et al.. 2023), and adapt to specific domains and tasks (Xiong et al., 2024) without modifying the model weights. In general, dense embedding based retriever (Karpukhin et al., 2020; Lin al., 2023; Wang et al., 2022) first retrieves top-k chunked contexts from collection of documents or external database for given question. Then, LLM reads the top-k contexts to generate the answer.
However, the current RAG pipeline has the following limitations: i) LLMs are not good at reading too many chunked contexts (e.g.. top- even with the long-context window, not only due to efficiency reasons, but also because a shorter list of top-k (e.g., 5, 10) contexts usually leads to higher accuracy of generation (e.g., see Table in Xu al., 2024b). ii) Given small k, needs mechanism to ensure the high recall of relevant contents. Relying solely on a retrieval model may be inadequate due to challenges learning effective local alignments across the entire embedding space to support accurate matching (Luan et al., 2021). In practice, a separate ranking model (Nogueira et al., 2020; Glass al., 2022; Ma al., 2023) that cross-encodes question and candidate context can work better than dense embedding-baat based retriever for obtaining the most relevant top-k contexts from top-N
Yue did work during an internship at NVIDIA, Correspondence to: Yue Yu <yueyu@ gatech.edu>, Wei Ping srppin@@nvdia.com..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

candidates (N >> k). tii) However, the zero-shot generalization capability of the expert ranking model can be relatively limited compared to the versatile LLM itself.
Based on the above considerations, our goal is to design an RAG instruction tuning pipeline that uses single language model to achieve both high-recall context extraction and high-quality content generation. In previous study, instruction-tune tuneon LLMs demonstrate a strong ability to extract answers from relevant context for a given question (e.g., OpenAI, 2023; Liu al., 2024; Lin al., 2024). This capability can be viewed as the "dual capability" of determining whether a chunk of context relevant to the question thus is useful for generating the answer. We hypothesize that these capabilities mutually enhance each other. Motivated by this insight, we propose RankRAG, which instruction-tunes a single LLM for both context ranking and answer generation in the RAG framework. Furthermore, RankRAG expands upon existing instruction-tune QA, retrieval- -aggented QA and ranking datasets, enhancing the L.M's ability to filter out irrelevant contexts during both the retrieval and generation phases of RAG.
data by incorporating context-rich
Our contribution can be summarized as follows:
We propose RankRAG, a novel framework that enhances LLM's RAG capability through simul- taneously instructing the LLM on context ranking and answer generation. During training, we design specialized task focused This task is structured for ranking and framed as regular question answering with instruction, aligning more effectively with retrieval-augmented generation tasks. At inference, the LLM first reranks the retrieved contexts, then generates answer based on the refined top-k (c.g.. 5). This framework is readily applicable to diverse mnowledge-intensse NLP tasks.
identifying relevant contexts or passages for given question.
Remarkably, we observe that integrating small fraction of ranking data into the instruction tuning blend of LLM works surprisingly well on the evaluations of ranking associated with the RAG tasks, even surpassing the LLMs fine-tuned with 10x more ranking data. We attribute this success the transferable design of RankRAG training.
We extensively compare the proposed RankRAG method with several strong baselines, including the open-sourced ChatQA-1.5, On nine seneral-domalt and five biomedical knowledge-intenee outperforms Llama3-
benchmarks for RAG, Jlama3-RankRAGG-88 and Llama3-RankRAR ChatQA--15588 and Llama3 Lamas-ChatQA-1155-700 by a margin, respectively.
In the remainder of the paper, we discuss related work in $ 2. We introduce problem setup in $ 3 and RankRAG method in Š 4. We present the experimental setup in $ 5, and conclude the paper in $ 6.
2 Related Work
Retrieval-augumented generation (RAG) has been established for knowledge-intensive NLP tasks (Lewis et al., 2020; Borgeaud et al., 2022; Izacard et al., 2023; Izacard & Grave, 2021).
In the standard process, standalone cennse-mbeddinng-- based retriever (c.g.. Karpukhin al., 2020) first retrieves relevant information from an external corpus, which the LLM then utilizes in the generation process. To improve this pipeline, recent research has focused on aligning retrievers to the needs of LLMs for generation (Shi et al., 2024; Lin et al., 2024), designing multi-step retrieval processes (Trivedi al., 2023; Jiang et al., 2023; Jeong et al., 2024; Shao et al., 2023), or filtering irrelevant contexts (Wang et al., 2023c; Yoran et al., 2024; Xu et al., 2024a). To improve generation, several studies have designed instruction -onninttiionnninnnggg methods dedicated to enhancing the search (Ma et al., 2023; Zhu et al., 2024; Muennighoff et al., 2024) and RAG capability of LLMS (Liu et al., 2024; Lin al., 2024; Luo al., 2023; Asai al., 2024a; Wang et al., 2024).
Although strong retrievers have been introduced (c.g., Lin et al., 2023; Yu et al., 2022: Wang et al., 2022, 2023a; Lee et al., 2024), one potential approach to improve retriever is optimizing it along with LLM in an end-to-end manner (c.g., Guu et al., 2020; Shi et al., 2024; Sachan et al., 2021; Izacard et al., 2023). However, this requires surrogate loss for optimization and complicates the training pipeline, especially when the embedding database needs to be re-indexed frequently due to the update of the embedding model (i.e., retriever).
Ranking serves as an intermediate step to improve the quality of information retrieval (Mitra et al., 201 8), and has been applied to RAG pipeline for improving generation quality (Glass et al., 2022; Ram et al., 2023). However, these methods still rely on an additional moderate-sized model (e.g. BERT, T5) for ranking, which is often insufficient to capture the relevance between query and contexts

---

5 1B
(..0441.5 708
ChaiQA-] 5 766
Chvi04-1.588
ChaiQ4-155700 708
ChaiQA-I 5B
Chiiq4-115 7B
Ftae Mach
Etact Marh
Ducc Manc
13
Frast Math
15
50-
10
20
10
20
78
20
10
20
16
26
00
20
16
Couteax Size
Connent Size K
ContextStee k
Coutest Size
(a)
NQ
(b) TriviaQA
(c) PopQA
FEVER
Figure I: Performance of ChaiQA-1:5, one of the strongest RAG models, on different context size k. We observe trade-off of selecting top-k contexts: a smaller compromises the recall, while larger k could introduce irrelevant or noisy context and mislead the LLM generation.
and may lack the zeot generalization capability. Although recent studies have demonstrated the strong ability of LLMs at ranking tasks (Khalifa et al., 2023; Qin et al., 2024; Sun et al., 2023), how harvest this ability for the RAG pipeline remains underexplored.
3 Preliminaries
this section, we first introduce the preliminaries of retrieval-augmented generation as well as problem setup. Then we present the limitations in the current RAG pipeline, which motivates the proposed RankRAG method.
3.1 Problem Setup
In retrieval-augmenteed generation, collection of documents or contexts (e.g. Wikipedia) is given, providing the grounded knowledge. Given a question the retriever R (e.g., a parameterized embedding model) first retrieves top-k contexts C {c , ,Ck that most relevant to the question. Subsequently, the language model produces the final answer where the answer can either be short phrase or a long sentence, depending the type of the target task. Our focus is on autoregressive language models (OpenAl, 2022, 2023; Meta-Al, 2024), which is the most common architectures for LLMs.
3.2 Limitation of Current RAG Pipelines
Before formally introducing RankRAG, we would like to first pinpoint several limitations of the current "retrieve-then-geeetie pipeline with large language models.
Limited Capacity of Retriever. Current RAG systems usually employ sparse retrieval (e.g. BM25 (Robertson et al., 2004)) moderate-size (e.g. BERT-based) embedding models (Karpukhin et al., 2020; Lin et al., 2023; Wang et al., 2022) as the retriever R, mainly due to efficiency con- sideration there are often millions of, if not more, documents need be indexed. These models encode questions and documents independently and calculate the similarity between question and documents using vector similarity metrics. However, the limited capacity of embedding models and independent processing of query and documents constrain the ability to estimate textual relevance between question and documents reducing their effectiveness in new tasks or domains, verified by both theoretical (Menon et al., 2022) and empirical (Luan et al., 2021; Thakur et al., 2021) studies.
Trade-off of Picking Top-k Contexts. Although the state-of state-of-the-ar long-context LLM take many retrieved contexts as input for answer generation, the performance quickly saturates with increased in practice. For example, Xu et al. (2024b) finds the optimal number of chunked context & is around 10 for long document QA tasks. As illustrated in Figure 1, we perform evaluation on ChatQA-1.5 (Liu al., 2024), one of the strongest RAG models with open weights, and find the saturation of accuracy when I = 10. In general, a smaller & often fails to capture all relevant information, compromising the recall, given the limited expressibility of retriever. In contrast, larger k improves recall but at the cost of introducing irrelevant content that hampers the LLM's ability to generate accurate answers (Yoran al., 2024; Yu al., 2023b).

---

Training Inference
Carpun
Stage-I:
LLM
RankRAG
Chale-ofthhught
Longfarm QA
Datly
FLAN
RankRAG
Rerank
Generate Answer
Doos
Top-N
Top-S
Stage-II: II: RankRAG Instruction-Tuniiee Tuning
6:73
1.
0.64
Conversation
Reading Comprohrsiion
Retrimal--nge anggentl-dggeetted
Cantest Ranking
Syutheti Conversation,
NanativOOA DR07: Quoret.
QA A Ranking
NewsQs, IAEQA,
9QwAD, WelQuuatiaz
Synthetiz
Newly Introduced Tocks
0.57
Figure 2: Two-stage instruction tuning framework for RankRAG.
4 RankRAG
To address the limitations mentioned in the previous section, propose the RankRAG method to enhance the LLM's ability for retrieval-augmenned generation. Specifically, we instruction-tung the LLM simultaneously capture the relevance between the question and context and utilize the retrieved context for answer generation. The details are introduced as follows.
4.1 Stage-I: Supervised Fine-Turee Tuning (SFT)
It is observed that general instruction-cuning improves the ability of LLMs to follow instructions. thus improving zero-shot results on various Ouyang al., 2022). As such, we follow existing works (Chung
supervised fine-tuning (SFT) often significantly
downstream tasks (Wei et al., 2022: et al., 2024; Wang et al., 2024; Liu et al., 2024) to first leverage SFT on a blend of high quality instruction following datasets, including: i) a private growd-sourced conversational dataset and public conversation datasets: OpenAssistant (Köpf et al., 2023), Dolly (Conover et al., 2023), and SODA (Kim al., 2023), ii) a long-form QA dataset ELI5 that requires elaborate answers (Fan et al., 2019), lii) LLM -eMeeaeraated instructions:
(Wang et al., 2023b) and Unnatural Instructions (Honovich et al., 2023), iv) FLAN and shain-of-thougll datasets (Chung al., 2024).
There are overall 128K SFT examples in total. We make sure that there is no overlap between SFT data and data from evaluation tasks. For each sample in the instruction-follviii multi-turn conversational format, use the previous turns of conversation between the user and the assistant the context, and compute the loss only at the last response from the assistant.
dataset, we take the
4.2 Stage-II: Unified Instruction- Dnstruction-Tuning Tuning for Ranking and Generation
The Stage-I SFT enpowers the LLMs with basic instruction-followiing capabilities; however, their performance on RAG tasks often remains suboptimal, as the LLMs are not optimized for extracting answers from retrieved context for a given question. Although recent studies (Lin et al., 2024; Liu et al., 2024; Zhang et al., 2024) enhance the RAG capability of L.M by instruction tuning it on context-rich generation tasks, these approaches can still ineffective with poor initial retrieval results. RankRAG instruction tunes the LLM for both retrieval-augnee augmented generation and context ranking. In particular, the context ranking capability crucial to obtain more relevant top-k context with imperfect retriever.
To achieve this goal, the instruction tuning blend of Stage-Il consists the following five parts:
1) SFT data from Stage-I. This part is included to maintain LLM's instruction-following capability.
2) ront QA data. We first follow Liu et al. (2024) to leverage multiple context-rich QA tasks to enhance the LLM's s capability of using context for generation. The training blend we use consists of: standard QA and reading comprehension datasets: DROP (Dua al., 2019), NarrativeQA (Kočiský et al., 2018), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), NewsQA (Trischler et al., 2017), TAT-QA (Zhu et al., 2021), which contains question, golden context and answer. ii) conversational QA datasets: MumannAmotatedCoone
and SynthetieConv QA open-sourced by Liu al. (2024), which contains a conversation between user and assistant, well as background document. The model needs to generate an answer given the conversation history and document.
3) Retrieval- augmented QA data. In addition to the above QA datasets used in Liu et al. (2024), we add two datasets with not only gold context but also the top-retrieved context using BM25. Note that it is crucial to improve LLM's robustness over irrelevant context at generation. Being aware of

---

Table 1: The instruction template for Stage-II. It is worth noting that all the tasks can be unified in the (2c, C, y) format, which is able to facilitate effective knowledge transfer across tasks.
Task
Question x
Context
Answer
Context-rich QA
Answer the following question
[question)
Presage: (Pansage) (1 Pig)
A phraselsentence
Presage (aasaage 1)...
Retrivval-mugmeeetedd QA
Answer the following question from comiest. (quertion)
Passage S: (Pazsage 53 (5 Psg. total)
A phrassssentence
For the question (gseation), access whether the passage is relevant to question.
Context ranking
Passage: (Pansage) (1 Peg.)
True/False
Retrieval-mggeneee ranking
For the question [question), lind all passages from the context that relevant to the question.
Presage (Pannage 5) Psg. tocal)
Paesage I: (Pazsage 13...
Paesage lideses
this, we consider two QA tasks, namely SQuAD (Rajpurkar et al., 2016) and WebQuestions (Berant et al., 2013). For each question with the answer, we combine the gold context with the top-retrieved contexts using BM25, ensuring a total of five contexts. Note that some retrieved contexts may not contain the answer, and could be the haaddnegativee contexts.
4) Context ranking data. To empower LLMs with ranking capabilities, we use the popular MS MARCO passage (context) ranking dataset (Bajaj et al., 2016). We treat the gold quesy-passage pairs (g, et) as relevant while using hard negative passages (g. c²)) mined via BM25 as irrelevant pairs. The L.LM needs to generate "True" or "False" given the corresponding query-passan pair, where the question along with the task-specific instruction is "For the question {question), access whether the passage is relevant to the question.".
We want to handle ranking in conversational scenarios as well. While MS MARCO spans various topics, the questions are only single-turn short sentences. However, ranking data is only available, if any, at a small amount for conversation QA. To overcome this limitation, we repurpsse the conversational QA pairs generate pseudo relevance pairs. As each conversation is only associated with one document d, we cut each document into 150-word chunks (c1, C2, Cn). We compute the 4-gram recall score between each chunk e; and the ground-truth answer a, considering segments with a recall score above 0.5 as relevant and those below 0.1 as irrelevant for the corresponding conversation. Note that, each sample contains one question-contex pair for this ranking dataset. In total, there are around 50k ranking pairs from MS MARCO ranking and synthetic conversations for instruction finetuning.
5) aetrieval-augmented ranking data. We aim to train the LLM with the capability of determining the relevance of multiple contexts simultaneously given question, which is closer to the test-time behavior of RAG with top-k contexts. As before, we use two QA datasets, SQuAD (Rajpurkar et al., 2016) and WebQuestions (Berant et al., 2013). We combine the gold context with the top-retrieved contexts using BM25, ensuring a total of five contexts. The contexts containing the answer are considered relevant, and the LLM is trained to explicitly identify all relevant contexts for the question.
Unifying RAG and ranking with instruction tuning. It is worth noting that, despite the variety of datasets and tasks described, they can all be cast into a standardized QA format (xr, c, yy) where is the question, e is the corresponding context, and is the target output answer. For example, for the sarrivval-augmentee
ranking data, the question is "For the question <question>, find all the passages from the context that are relevant to the question." " Table exhibits how to cast different tasks into a unified format. Despite its simplicity, this approach has the following advantages: i) It empowers the LLM with the ranking capability by adding a relatively small amount of ranking data. ii) By standardizing these tasks into unified format, they can mutually enhance each other. After that, we obtain the final RankRAG model that can be applied to various wwwllede-intensiig NLP tasks.
4.3 RankRAG Inference: Retrieve-Rerant Reraee--eeaaan-eeraatttt Pipeline
As RankRAG incorporates an additional reranking step, the inference pipeline for each question modified as a retrieve-rerank -enerate pipeline, described as follows: (1) the retriever R first retrieves top-N contexts from the corpus. (2) the RankRAG model calculates the relevance score between the question and retrieved N contexts as the probability of generating the answer as True using the prompt in Table 1, then reranks contexts to only retain top-k (k & N) contexts, which are then used as the input for the generation step. (3) The top-k contexts, along with the question, are concatenated and fed back into the RankRAG model to generate the final answer.
5

---

Efficiency Discussion. We are aware that the addition of a reranking step introduces extra processing time. In practice, for each question, denote the time for indexing and retrieval as t,, the time for using L.M to calculate the relevance score as l2 and the time for generation as 43, then the ratio of added time overhead is Notz, In practice, calculating relevance typically requires generating just one token and involves much shorter inputs compared to the generation step with top-k contexts. We provide efficiency study in $5.5.
5
Experiments
In this section, we conduct comprehensive experiments on a variety of knowledge-intensive NLP tasks to demonstrate the zero-shot capabilities of RankRAG.
5.1 Experiment Setup
Tasks and Datasets. We consider 3 types of tasks in experiments: (1) Open-domain QA (OpenQA), which includes NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023), HotpotQA (Yang et al., 2018) and 2WikimQA (Ho et al., 2020). The first three are single- hop QA tasks, while the last two are multi-hop QA datasets. For NQ, TriviaQA, and HotpotQA, we use the split from KILT benchmark (Petroni et al., 2021) 2. (2) Fact verification, where we use FEVER (Thorne et al.. 2018) from KILT benchmark. (3) Conversational QA (ConvQA), consider three datasets including Dic2Dial (Feng et al., 2020), TopiOCQA (Adlakha et al., 2022) and INSCIT (Wu et al., 2023), which have long documents that cannot be fitted directly into LLMs thus necessitates retrieval and ranking. The detailed dataset information is in Appendix A.I.
Baselines. We consider the following baselines: (1) Baseline LLMs without RAG, where we con- sider LLMs trained with proprietary data including InstructGPT (Ouyang et al., 2022), PaLM 2 (Anil et al., 2023), FLAN-LaMDA (Longpre et al., 2023), GLaM (Du et al., 2022), Claude Seetll-8222--Inttrue (Mistral, 2024), DeepSeek- V2 Chat (DeepSeck, 2024)
2 (Anthropic. 2023),
and only use the official reported results. We also consider two ChatGPT-series models, namely GPT-3.5-turbo (gpt-3.5-turbo w-t.urro--0000 -0t03) (OpenAl, 2022) and GPT-4 (gpt 1gpt-4-0613) (OpenAI, 2023). (2) Baselines with retrieval, we evaluate models augmented with retrieval. Specifically, we include Atlas (Izacard et al., 2023) and Raven (Huang et al., 2023), two RAG models based on encoder- decoder LMs. For decoder-only models, we consider Self-RAG (Asai et al., 2024a), RECOMP (Xu et al., 2024a). InstructRetro (Wang et al., 2024), RePlug (Shi et al., 2024), RA-DIT (Lin et al., 2024), Llama-3-instruct (Meta-AI, 2024) and Cha (Liu et al., 2024). We also list the result of RAG pipelines using InstructGPT (175B parameters) as the backbone including GenRead (Yu et al., 2023a), Retrieve-read (Lazaridou et al., 2022) and ReFeed (Yu al., 2024), but mainly for reference. Other reported numbers are directly comparable if they follow the standard zero-shot settings.
Evaluation Metrics. For OpenQA datasets, we Exact Match (EM) as the main metric but also report Accuracy for TriviaQA and PopQA and F1 score for HotpotQA and 2WikimQA as it is used in several studies (Asai et al., 2024a; Mallen al., 2023). For FEVER, use accuracy the metric. For ConvQA datasets, we follow (Liu et al., 2024; Wang et al., 2024) to use F1 score as the metric.
Implementation Details. We use Llama3 8B and 70B (Meta-AI, 2024) as the backbone in our main experiments. For the two-stage instruction tuning, we set the batch size to 128 and train the model for 1000 steps with learning rate 5e-6 in Stage-I. Then, we reduce the learning rate to 3e-7 for 8B and 2e-7 for 70B model, set the batch size to 64, and train the model for 3300 steps (around I epoch). We use the Adam optimizer (Kingma& & Ba, 2014) with 31 = 0.9 and 83 0.98. During the inference stage, we use the December 2018 Wikidump as the corpus index for NQ. TQA, HotpotQA, 2WikimQA, and use the December 2020 Wikidump for PopQA, following (Asai al., 2024a). By default, follow (Wang et al., 2024; Lin et al., 2024; Liu et al., 2024) to use the Dragon retriever (Lin et al., 2023) as default and retrieve top-N (100 for 8B and 30 for 70B) documents for ranking, but RankRAG can be adapted to various retrievers and different N (see $ 5.3 and 5.5). To ensure a fair comparison, we test the performance of & E (5, 10, 20) and report the best performance for baselines. For generation, we keep temperature T = 0 and set the maximum number of generated token to be 32 for OpenQA, 128 for ConvQA and 8 for others. Training RankRAG-8B uses 32 NVIDIA A 1100 GPUs for 10 hours (4 hours for Stage-I and 6 hours for Stage-II II finetuning), while training RankRAG-70B uses 128 NVIDIA A100 GPUs for 16 hours (4 hours for Stage-I and 12 hours for Stage- II Finetuning).
'Th results of NQ and TriviaQA using the split from DPR (Karpukhin et al., 2020) are in Appendix F.

---

Table 2: Results of RankRAG and baselines 9 datasets. Unless specified, all results under zero-shot evaluation without additional demonstrations. Results unavailable in public reports are marked as "_". We use NQ, TriviaQA, and HotpotQA from the KILT benchmark for Llama3-Instruc Llama3-ChatQA-la. and Llama3-RankRAG: Note that GPTT4 and GPT-4-turbo may refuse to answer the question when retrieved passages do not contain relevant information, thus the EM accuracy drops after including RAG on TriviaQA, HotpotQA and 2WikimQA.
Task (Zero-shot)
TrivinQA
PopQA
HolporQA
2WikimQA
FEVER
Doc2Diul
TopiOCQA
Inscit
Avg.
Metric
EM/ACE
EM
EM/F1
EM/FI
FI
F1
Winhou Ketrieval Augmeeted Goneration
PPttrecGPPP (Ouyang
65.8/332
260/382
PaLME 540B
76.91
PaLM2 540B shot, Anil al.)
86.1
GLaM 64B shot,
31.5
11.37
FLAN BLAN-LLMDA 137B (Wei all
20.7
68.17
Chade 2.65 shot, Amthrogie)
87.5/-
Mixiral-&x2BB-- Instract (5 shot, Mistral)
40.1
82.27-
DespSeck-V2. 266 shot. DorpSook)
53.
86.7/-
GIEE3 Satt-3.-tubbo- 106 (QmmAD)
38.6
62.9/91.7
/3.4
20.9/420
23./304
K2.7
20.1
28.5
27.2
38.5
GPT-4 0613 (OpenA))
40.3
$L.89945
94.5
5133/388
34.57 46.9
29.8/36.6
87.7
27.6
30.1
27.0
42.0
GPT-4-urt -rr-4-urbo-222 2004 0409 (OpenA))
41.5
$0.0 94,3
25.0733.5
26.61 43.8
25.1
87.0
27.6
26.4
24.4
38.6
Whih Retieeul- Genrration
Arlas 11B (Iearndd
26.7
56.97
3471-
77.0
Rosen 11B (Husug all)
29.6
65.77-
Selt-RAG 7B (Asai ot al.)
-666.4
1599
RoI-RRAG (Asai et al.
-/89.3
-1558
RECOMP 208 (Xo et
37.0
59.01-
50.47401
InstreiRat 478 (Wing nl.
$8.9
78-31-
-1-
36.0
RePlug 658 (Shi ot all)
28.8
72.6
33.01-
73.3
RA-DIT 658 (1.in al.)
35.2
75-1-
39.71-
80.7
Llama-- Insttit- 88 (MelaAA)
3099
70.71 30.4
34.9
260/338
9.6/25,2
H6.9
33.6
44.9
32.6
40.8
Liumd Instruct 70B (Mata-
42.7
K2.41 39.3
25.3 5664
35.5/433
13.5/279
91.4
37.9
49.7
36.2
47.1
Lium3 CharQA- 8B (Liu
4244
$1.01 57.6
52.6 59.8
44.6
26.8/319
90.9
39.3
29.9
30.1
49.6
Llund CharQA 1.5 700 et
47.0
53.07
91.4
50.958.3
92.21 53.
35,9/374
92.7
41.3
55.6
32.3
53.6
Lluma Kank-MMG 8B shot)
30.6
52.91 89.5
57.6 64.1
35.37 96-7
314736.9
92.0
40.4
50.4
33.3
52.0
Lluma RanLRAGG 708 (0 shot)
54.2
96.5
92.3
65.4
42.7/ 55.4
38.2/433
93.8
41.5
52.8
33.2
56.1
For reference: Using InstrucGPP or CodeX
(-1158) (Chrang al. 2022) as the Borkboon LLM.
GenRead (Yu ct.l.)
32.5
66.27-
46.01-
36.47 39.9
-I-
MU.4
Resteve Road (Lazandouu al.)
31.7
61.41-
35.27 38.0
27.
N2.7
Reloee (Yu all)
39.6
08.91
41.31 45.1
GPT 3.5 tubbo 1106 RAG (OpenAl)
46.7
79.7 $8.0
499/57.0
41.2/41.2
27/23222
90.8
34.8
44.3
35.3
46.8
037-4-0613 RAG (OpenAl)
40.4
75.0 58.5
44.3/61.4
27.6/ 38.1
14./1776
92.6
34,2
45.1
36.4
43.5
GPT turbo
RAG! (OpenAI)
70229111
/9.5
8.7/179
92.2
483
41.6
Data Contamination Issues. One possible issue for the zero-shot evaluation is the test set contami- nation, where some the tuee-specific examples overlap with the instruction fine-tuning data (Oren al., 2024). To address this issue, have performed string match-based analysis where we do not observe any overlap between the train data and data from target tasks.
5.2 Main Experiments
Table 2 presents results of RankRAG and baselines. The findings summarized follows:
RankRAG outperforms existing RAG methods. With 8B scale, RankRAG consistently outperforms ChatQA-1.5 8B, one of the most recent open-sourced models with state-of-the of-the-art performance on many RAG benchmarks. RankRAG 8B also competitive when compared with baseline models with many more parameters. For example, significantly outperforms InstructRetro (5x parameters), RA-DIT 65B (8x parameters), and even outperforms Llama3-instruct 70B (8x parameters) on NQ and TriviaQA tasks. With more parameters, RankRAG 70B outperforms the strong ChatQA-1.5 70B model, and largely outperforms previous RAG baselines with InstructGPT as the underlying LLM.
RankRAG demonstrates larger improvement on more challenging datasets. We observe that the performance gains of RankRAG over baselines are more pronounced for more challenging QA datasets. For example, on long-tailed QA (PopQA) and multi-hop QA (2WikimQA) tasks, we achieve more than 10% improvement over ChatQA-1.5. These findings suggest that in challenging OpenQA datasets where top documents from retrievers are less relevant to the answer, context ranking effectively enhances performance. In this work we focus on improving single-time retrieval for QA tasks. How to effectively combine multi-round RAG pipelines (Jiang al., 2023; Khattab et al., 2022; Jeong et al., 2024) with RankRAG is an interesting avenue of future work.
5.3 Ablation Studies
Effeet of Designed Components. Table 3 shows the ablations of RankRAG with Llama3 8B as the backbone nine general-domain datascts. Overall, observe all of the proposed components
7

---

Table 3: Ablation study of RankRAG. We Llama3-88 8B as the backbone. Where 'QAA' and 'RAR' stands for rerrenee QA and retrievll-uummented ranking data, respectively. For "w/o reranking', we do not perform ranking in the inference stage.
Task (Zero-Shot)
NQ
TriviuQA
PupQA
HotputQQ
2WikimQA
FEVER
Duc2Dial
TopiOCQA
Inscit
Avg.
Metric
EM
EM / Ace.
EM / Aec.
EM/FI
EM/FI
Acc.
F1
Fi
F1
RaakRAG
82./89.5
31.4/669
50.4
w/o reranking
48.0
80.3 186.8
49.37 19.0
31.3 /4.6
26.4/ 30.5
91.1
39.7
49.4
30.9
49.8
wio RQA
49.4
82.0/ 88.9
55.1/62.9
35.6/445.9
31.8/77.5
92.1
39.4
46.8
32.4
51.6
we RAR
48.6
82.2/ 89.1
56.0/62.66
35.1/45.2
35.77
91.4
39.6
48.6
33.5
51.8
w/ (L atl)
43.3
80.8 88.6
30.5/4188
25.2/29.6
91.2
36.8
46.4
30.1
w/ Siage-I SPT Only
38.3
63.7 1..7776.6 76.6
49.8 54.6
/40.3
18.07 25.9
85.7
33.3
33.7
30.5
42.2
Table 4: Zero-shot evaluation using Llama2 (Touvron et al., 2023) model as the backbone.
Task (Zero-Shot)
TrivinQA
PopQA
HelporQA EM/FI
2WikinQA EM/P1
FEVER
TopiOCQA
Inscit
Ang.
Metric
Acc.
Acc.
F1
Llama2 70B (Tuuvrun et all
25.3
82447
Hmma2-ChatQA-110 78 (1.u all)
41.4
77.8 86.5
46.7
280/400.
27.5
85.9
37.9
45.5
31.01
46.6
Lima2 Chaaa-CaiQAO 13D (Liu et al.)
47.9
80.9/876
51.8/56.2
32.94332
27.//1.1
87.6
38.1
48.9
30.8
49.6
Ulmma2- ChaiQA-110 700 (Liu et al.)
49.5
83.2/ 89.7
/5.1
39.0749.
28,91 34.1
91.7
38.9
$1.0
31.9
51.8
Hamaa2-KankRAGG 7B
46.9
84.01/ 89.6
55.976133
32.2/43.2
26.87 30.7
86.6
38.6
49.2
32.3
50.3
RAMA2-RanRRAG 13B
50.5
84.5/910
58.0/63.9
364/47.3
29.5/ 34,3
91.7
39.5
49.2
33.4
52.5
Imma--RankRAGG 708
51.2
85.8/ 92.1
58.7/64.5
41.8/5.11
33.8/ 38.8
91.9
41.2
52.9
35.8
55.0
contribute to the final performance. Removing context ranking hurts performance on all tasks, justifying its efficacy in selecting the most relevant contexts for the target question. Besides, the retrieval-augmented QA (RQA) and retrieval-augmented ranking (RAR) designed for instruction fine- tuning improve outcomes on most tasks by helping the model explicitly pinpoint relevant contexts. On the contrary, the RAFT method used in (Lin et al., 2024) treats each retrieved context separately during instruction finetuning, which yields suboptimal results when compared to RankRAG with the same training data.
Performance with Different LLMs. Table 4 reports the performance of RankRAG and the most recent baseline ChatQA using Llama2 with backbone having varying amounts of parameters. Notably, there exist consistent gains in terms of the average performance 7..8%/..4%/666. /%%6.%% on 7B/13B/70B variants respectively), justifying the advantage of RankRAG across different LLM types and scales.
Performance with Different Retrievers. Figure exhibits the performance of RankRAG and ChatQA- 1.5 with different dense retrievers on three represen- tative tasks, where consider DPR (Karpukhin et al., 2020) and Contriever-MS MARCO (Izac- ard et al., 2022) two variants. We note that al- though the initial retrieved result is not good enough, AG still surpasses Cha:QA-1.5 by more than 10% for both retrievers on average. To summarize, RankRAG is robust to the choice of retrievers.
Chaigh-15
KeAARAG
Exsect Match
TQA
PoogA
NQ
PopgAA
(a) DPR
(b) Contriever
Figure 3: Performance with different retrievers. The performance of Recall is in Appendix 1.
5.4 Experiment on Domain-specifie RAG Benchmarks
To demonstrate that RankRAG can adapt to specialized domains, conduct experiments on Mi- rage (Xiong et al., 2024), a re- cently introduced RAG bench- mark for the biomedical field. We follow Xiong al. (2024) to employ MedCPT (Jin et al., 2023) as the retriever R with MedCorp' as the corpus D.
Table 5: The performance of RankRAG on Mirage, zero-shot biomedical RAG benchmark. RankRAG and basclines use re-
trieval by default. Most of numbers are from (Xiong al., 2024).
MML U-maa
PabmooQ4
BioASQ
MrdQA
GPT44-6 ()penA))
87.34
70:60
93.56
$2.80
66.65
70.97
GPEAS (OpenAl)
75.45
67.40
00:20
68.61
SK
71.56
Mixtral sY78 cling et al.)
75.85
07.50
$7.56
60112
56.42
69 09
Toovmm )
73.05
Muslimm 70B
65.38
40.57
60.15
PMC-lllaa 138 (Wo
52.53
48.29
56,00
S2.1
Lham33-Chuiaa- A- 1.58B clis all
61.40
66.40
82,50
42.35
46.07
$9.96
LJmma -hittt-- 1. 7001
30.51
74,801
81.17
68.99
02.54
73.94
48.85
KunkR.M
81.44
75/05
The experiment results of RankRAG and baselines shown in Table 5. From the table, we observe that RankRAG, even without fine-tuning on the biomedical domain, excels at medical QA tasks.
Link: https://hugginee co co/MedRAG. Detailed dataset information is in Appendix A.2.

---

Table 6: Ranking performance with different ranking models. Unless specified, all baselines are used to rank the top 100 retrieved passages. RankRAG achieves better performance despite using fewer ranking data. NQ. TriviaQA and HotpotQA are used for training the re- re-implementation. & We only rerank top-30 passages for GPT-4 due to budget constraint.
model. t; Our
NO
Trivag
PePUA
Teaci
Recall
R:010
R4Y5
Re000
R.P:3
RG95
RPP00
R45 RV
R.970
R. 10
R4910
Backhene Rebicver
Deigin
Fiactuned Basdine Ranking Model
RackSERT irllss
74.3
86.0
07.0
78.1
82.6
81.5
51.
59.8
63.7
45.0
06.7
BGE Resani- 2end 565M CCaca ctul.)
-1,34
79.07
$2.6"
$5.65
91.67
94.5"
97.1"
70.5
$4.5
$5.9
58.5
61.8"
65.07
51.3
59.5
69.7
Rarki] AM and (Ma al)
17.4
81.1
50.
91-3
M1.1
96-4
B6.11
$5.8
57.1
62.1
64.38
57.8
63.1
71.3
Cher24-111 1 10 di
75.7
82.0
91.1
$5.0
57.3
76.1
81.5
77-
45.0
13.
31.3
47.6
$1.9
Offitheshelf LLM Recanker
LaFESS tte 1M0, OpenAA) cappal top ML. OpenAA))
Lsc
37.
82.5
RS.11
91.1
14.6
16.7
17.4
K2.
$2.3
82.1
36.6
62.4
502
59.1
65.6
USc
19.3
83.2
R3.1
91.3
05.5
56.3
19.3
83.6
95.2
53.2
51.0
61.01
51.3
63.7
7011
RaCkRAG BB top 100)
-50
50.3
$4.0
96.3
93.2
05.
97.3
$1.5
94.9
87.0
97.6
61.8
65.2
600
65.7
73.5
RacARANG 706 Dhy 30
-56
$8.6
$4.0
83.
93.6
95.9
97.1
$1.5
84.6
56.5
565
59.7
02.2
61.3
66.4
74.6
Notably, RankRAG 8B surpasses Meditron 70B leading open-source LLM for the medical domain- by 6.3%. Besides, RankRAG 70B attains more than 98% performance of GPT-4. These results justify RankRAG's capacity to be readily applied to new domains without extra post-training.
5.5 A Closer Look at the Ranking Module
As the context ranking serves as a core step in RankRAG, we take a closer look at this component. All the studies are done using Llama3-8B as the backbone.
RankRAG is Data-efficient. Previous approaches that infuse context ranking into the RAG pipeline usually involve a separate reranking model. To compare our model with these basclines, we evaluate four models (BERT (Glass et al., 2022y/T5 (Nogueira al., 2020)/Llama3 (Ma et al., 2023)) fine-tuned on the full MS MARCO passage ranking dataset, a strong off-the-shelf reranker model BGE-ranker, and two OpenAl GPT-series models. For the GPT-series models, we the token probability of 'True' as a proxy for the relevance score*. These models are then used to rerank top-retrieved passages by Dragon, similar our approach. Surprisingly, as shown in Table 6, RankRAG achieves better recall over dedicated ranking models trained on 10x more ranking data for most cases. Besides, RankRAG can still outperform the BGE-ranker on most tasks, which has been extensively trained on more than million ranking pairs, including some that overlap with our evaluation tasks. This advantage is likely due to the adaptable nature of our model's training, where the ranking data closely resembles the general RAG fine-tuning data. Directly using ChatQA-1.5 to rank passages hurts the performance, indicating the necessity of incorporating ranking data into instruction fine-tuning.
We further study the relation between the number of context ranking data and final performance. As shown in Figure 4, with 5k ranking data only (~ 1% of the MS MARCO dataset), RankRAG can already obtain very compelling results, while further increasing the number of ranking data to 50k yields non-marginal gains. This finding confirms RankRAG's data efficiency achieving effective performance with a modest amount of ranking data and maintaining adaptability across various tasks.
Performance v.s. Time-efficiency for RankRAG. One specific caveat for scaling up model size is the increment in the latency overhead - as mentioned in $4.3, it requires sample-wise ranking which incurs additional time. To study the relation between the time efficiency and performance, we change the N used in reranking and plot the relation of N and final accuracy in Figure 5, from which observe that even with N 20, RankRAG still improve the baseline model without reranking. While reranking across N = 20 to 100 improves the exact match score by 5.9% to 9.1 % across three tasks, it incurs an additional 0.9x to 6.0x increase in time significantly less than the 20x to 100x increase one might expect.
5.6 Case Study
Table 7 presents a case study on NQ dataset, where we observe that using retriever only yicld noisy contexts, as there are several distractors, and some contexts (e.g. Passage 4/5 for ChaiQA-1.5) are

---

x (50g *R (NG
Baage Manca
Exge Muish
Exact Match
Daset Manch
35-
85
Re
ChutgA-155
ChaQAA-15
RonkRAG
50k
.Raking
Inference Time
Inference Time
Inference Time
Figure 4: Performance w.r.t. # Ranking Data
(a) NQ
(b) TriviaQA
(c) HotpoiQA
Figure 5: Performance v.s. Efficiency analysis for RankRAG.
Table 7: A case study on the top-retrieved context and predictions on NQ dataset, illustrating the effectiveness of RankRAG-8B over 11.5-A---5-8B Red text denotes distractors, while green stands for evidences. Ra is able find the correct answer via extract more evidence with reranking.
1 who hostod and won the inagural weeld A: Uraguay
home soil 1982, Linglnnd (1966) only while playing host Uruguay (1930), Italy (1344,, Argenina (1975) France (1998) firs tiles host nations but have gune again, while Germany (1974) their second
Passuge FIR World second
FIFA's Congress Only Mesios, Iolly France, Germany Germanyy until Presage World Cup country have bosted event on two accasions.
shortly the
Cup)
CharQ4-1.55
United Stares, and thereby became the first to two Cups. This second World in Mosico came 16 years the lirst in 1970...
Passage CONCACAF bosts, beating the
Passage 1998 FIFA World Cup Africa made their first appearances in the finals. France was swarded 1998 World Cup on 2 July 1992 by the of HIP during general meeting Zürich, Switzerland They deteated Morocco by 12 voses to
comminiee esecuitve
execuiive
Pussage 2028 FIEA World Cup bosted by onz of the remaining four confederations CONCACAF (North America: lust bosted 1994), CAP hosee beffore.... Prodiction: Germany (8))
(Alrica bosted
CUNMEBOL (South America, last bosted 2014), OKC (Uecania,
Passage FIFA World Cup second round on home scil in 1982. England (1666) won its only title while plaing as host mation Urigury (1930), Italy (1934), Argentin (1975) and France (1998) wan their firs tnlls as bost nations but have gune on to win again, while Germany (1974) won their socond file on home soil..
Phesage 2. Timdlin of avsociation fomball penalty kicks. Thitteen teauss ester the firtt World Cup, held in Unugayy. The hoots best Argestinn 4-2 in the final. Contested between the top national teams of comtinenal Europe, Dr Gern Cup fint edition won by Italy
Passage The Urugusy national football team repeesenss Unuguzy international issociation football and is controlled by the Uruguayan Football Assccittoo. They have wos the Capa Americs 15 times, the most suceessfill nativual toom in the toumoment the mewl recent tille being the 2011 edition. The team
RankRAG
the FIF World twice, including first World Cup 1930 as defeating Argentina 2 the final.
the chosen vote by FIEN's Congress, The decision currently made roughly the and the United second World Cup in Mexico came 16 years after the lirst one in 1970...
Presage FIFA World Cup busts country
advancc
though Paesage CONCACAF hosts, heating the hids
the therehy the first
World Cups. This
Prediction Uragusy (V))
unhelpful. However, the utilization of reranking uncovers two additional relevant passages, aiding the model in providing the correct answer. More case studies are provided in Appendix G.
Conclusion
In this work, we introduce a new RAG framework, RankRAG, which Instruction-tunes a single LLM for both ranking and answer generation. We find that the instruction tuned L.Mss can outperform existing expert ranking models by only adding a small fraction of ranking data into the training blend. We compare our RankR with the state-of-the-art RAG models on comprehensive knowledge- intensive benchmarks and demonstrate RankRAG significantly outperform all of them on nine general-domain and five biomedical benchmarks for RAG.
References
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., and Reddy, S. Topiocqa: Open-domain conversational question answering with topic switching. TACL. 2022.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D.,, Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 2023.
Anthropic. Model card and evaluations for claude models. 2023.
Asai, A., Wu, Z., Wang, Y. Sil, A., and Hajishirzi, H. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR, 2024a.
Asai, A., Zhong, Z., Chen, D., Koh, P. W., Zettlemoyer, L., Hajishirzi, H., and Yih, W.-t. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint 2403v:2403.03188 2024b.
10

---

Bajaj. P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. Ms marco: human generated machine reading comprehension dataset. arXiv preprint arXiv: /61 1.09668, 2016.
Berant, J., Chou, A., Frostig. R., and Liang, P. Semantic parsing on freebase from question-anse -nnser answer pairs. In EMNLP, 2013.
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. JCML. PMLR, 2022.
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2023a.
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. Meditron-7000 Scaling medical pretraining for large language models. arXiv preprint 2. 2023b.
Chung. H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang. X., Dehghani, M., Brahma, S., et al. Scaling inltruuctionfinetunee language models. JMLR, 25(70), 2024.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free Dolly: Introducing the world's first truly open instruction-tunt Ilm, 2023.
Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., and Gardner, M. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. EMNLP, 2019.
DeepSeek. Deepseek-v2 A strong, economical, and efficient mnixture-of-expee language model, 2024.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., al. Glam: Efficient scaling of language models with mixture-of-experrs, ICML, 2022.
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading compre- hension benchmark requiring discrete reasoning over paragraphs. In NAACL,
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., and Auli, M. Eli5: Long form question answering. In ACL, 2019.
Feng, S., Wan, H., Gunasekara, C., Patel, S., Joshi, S., and Lastras, L. doc2dial: A goal-oriented decument-grounder dialogue dataset. In EMNLP, 2020.
Glass, M., Rossiello, G., Chowdhury, M. M., Naik, A., Cai, P., and Gliozzo, A. Re2G: Retrieve, rerank, generate. In NAACL, 2022.
Guu, K., Lee, K., Tung. Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In ICML, 2020.
Hendrycks, D., Burns, C., Basart, S.. Zou, A., Mazeika, M., Song, D., and Steinhardt, Mcasuring massive multitask language understanding. In ICLR, 2021.
Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. COLING, 2020.
Honovich, O., Scialom, T., Levy, O.,, and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. In ACI,, 2023.
Huang, J.. Ping, W. Xu, P., Shoeybi, M., Chang, K. C.-C,, and Catanzaro, B. Raven: In-context learn- ing with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.0797 2023.
Izacard, and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In EACL, 2021.

---

Izacard, G. Caron, M., Hosseini, L.. Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsuper- vised dense information retrieval with contrastive learning. TMLR, 2022.
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi- Yu, J., Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot learning with retrieval augmented language models. JMLR, 24(251):1 143,, 2023.
Jeong, S., Baek, J., Cho, S., Hwang. S. and Park, J. C. Adaptive-rag: Learning to adapt retrieval- augmented large language models through question complexity. In NAACL, 2024.
Jiang, A. Q.. Sablayrolles, A., Roux, A., Mensch, A.. Savary, B., Bamford, C., Chaplot, D. S.. Casas,
D. d. I,, Hanna, B., al. Mixtral of experts. arXiv preprint arXiv:2401.0408.. 2024.
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q.. Dwivedi- Yu, J., Yang. Y., Callan, J., and Neubig, G. Active retrieval augmented generation. In EMNLP, 2023.
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? a large-scat scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Jin, Q., Dhingra, B., Liu, Z.,, Cohen, W., and Lu, X. Pubmedqa: A dataset for biomedical research question answering. In EMNLP, 2019.
Jin, Q. Kim, W. Chen, Q.. Comeau, D. C., Yeganova, L., Wilbur, W. J., and Lu, Z. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(1)). 2023.
Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L.,, Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. EMNLP, 2020.
Kasai, J., Sakaguchi, K., yoichi takahashi, Bras, L., Asai, A., Yu, X. V., Radev, D., Smith, N. A.,
Choi, Y., and Inui, K. Realtime QA: What's the answer right now? In NeurlPS, 2023.
Khalifa, M., Logeswaran, L., Lee, M., Lee, H., and Wang, L. Few-shot reranking for multi-hop QA via language model prompting. In ACL, 2023.
Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. Demonstrate- search-predict: Composing retrieval and language models for knowledge-intensing nlp. arXiv preprint arXiv:2212. /4024, 2022.
Kim, H., Hessel, J., Jiang, I.., Lu, X., Yu, Y., Zhou, P., Bras, I.., Alikhani, M., Kim, G., Sap, M., et al. Soda: Million-scale dialogue distillation with social commonsense contextualization. In EMNLP, 2023.
Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv preprint 1NXiv:4422.6980 2014.
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The narrativeqa reading comprehension challenge. TACL, 2018.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research. TACL, 2019.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, Openassistant conversations democratizing large language model alignment. arXiv preprint arXiv: 2304.07327, 2023.
Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Internetaaummented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2003.05115. 2022.
12

---

Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catanzaro, B., and Ping, W. Nv-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint arXiv:2405.17428. 2024.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. eetrieval-aggmentt generation for knowledge-intensee nlp tasks. NeurIPS, 33, 2020.
Lin, K., Tafjord, O., Clark, P., and Gardner, M. Reasoning over paragraph effects in situations. In Workshop on Machine Reading for Question Answering, 2019.
Lin, S.-C., Asai, A., Li, M., Oguz, B., Lin, J., Mehdad, Y., Yih, W.-t, and Chen, X. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. Findings of EMNLP. 2023.
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., Zettlemoyer, L., and tau Yih, W. RA-DIT: Retrievallaugmmnted dual instruction tuning. In ICLR, 2024.
Liu, 7., Ping, W., Roy, R., Xu, P., Shoeybi, M., and Catanzaro, B. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401. /0225, 2024.
Longpre, S., Hou, L.. Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., al. The flan collection: Designing data and methods for effective instruction tuning. In ICML, 2023.
Luan, Y., Eisenstein, J., Toutanova, K., and Collins, M. Sparse, dense, and attentional representations for text retrieval. TACL, 2021.
Luo, H., Chuang, Y-S., Gong, Y., Zhang, T., Kim, Y., Wu, X., Fox, D., Meng, H., and Glass, J. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305. 15225, 2023.
Ma, X., Wang, L., Yang. N., Wei, and Lin, J. Fine-tuning Ilama for multi-stage text retrieval. arXiv preprint arXiv:2310.08379, 2023.
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. ACI., 2023.
Menon, A., Jayasumana, S., Rawat, A. S., Kim, S., Reddi, S., and Kumar, S. In defense of dual- encoders for neural ranking. In ICML, 2022.
Meta-AI. Llama model card. 2024.
Mistral. Mixtral 8x22b. 2024. URL https://mistral httpp/swww..aaaaeeeee//aaaaaaaaa
Mitra, B., Craswell, N., et al. An introduction to neural information retrieval. Foundations and Trendy in Information Retrieval, 2018.
Muennighoff, N., Su, H., Wang. L., Yang. N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906 2024.
Nogueira, R., Jiang, Z., Pradeep, R., and Lin, J. Document ranking with pretrained sequence-to- sequence model. In Findings of EMNLP, 2020.
OpenAI. Introducing ChatGPT, 2022.
OpenAl. GPT-4, 2023.
Oren, Y., Meister, N., Chatterji, N. S., Ladhak, F., and Hashimoto, T. Proving test set contamination in black-box language models. In ICLR, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.. Mishkin, P.. Zhang, C., Agarwal, S.. Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Neur!PS, 35, 2022.
Pal, A., Umapathi, I.. K., and Sankarasubbu, M. Medmeqa: large-scale multi-subject multi-choice dataset for medical domain question answering. In CHIL, 2022.
13

---

Petroni, F, Piktus, A., Fan, A., Lewis, P.. Yazdani, M.. De Cao, N., Thome. J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V., Rocktäschel, T., and Riedel, S. KILT: benchmark for knowledge intensive language tasks. In NAACI, 2021.
Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., et al. Large language models are effective text rankers with pairwise ranking prompting. In Findings of NAACL, 2024.
Rajpurkar, P., Zhang. J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton- Brown, K., and Shoham, Y. In-context retrieeal-augmented language models. TACL, 2023.
Robertson, S., Zaragoza, H., and Taylor, M. Simple bm25 extension to multiple weighted fields. In CIKM, 2004.
Sachan, D. S., Reddy, S., Hamilton, W. L., Dyer, C., and Yogatama, D. End-to-end training of multi-document reader and retriever for open-domain question answering. In NeurlPS, 2021.
Shao, Z., Gong, Y. Shen, Y.. Huang, M., Duan, N., and Chen, W. Enhancing retrieval- -augeettd large language models with iterative retrieval-genae generation synergy. In Findings of EMNLP. 2023.
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L.,, and Yih, W.-t.
Replug: aetrieval-augmentee black-bos box language models. In NAACL, 2024.
Sun, W. Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. Is ChatGPT good at search? investigating large language models re-ranking agents. EMNLP, 2023.
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, Beir: A heterogencous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS, 2021.
Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. Fever: A large-scale dataset for fact extraction and verification. In NAACL, 2018.
Touvron, H., Martin, L.. Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S.,
Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092:
2023.
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., and Suleman, K. Newsqa: machine comprehension dataset. In RepL4NLP Workshop at ACL, 2017.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Interleaving retrieval with chain-of- thought reasoning for knowledge-intensst intensive multi-step questions. In ACL, 2023.
Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 2015.
Wang. B., Ping, W. McAfee, L., Xu, P., Li, B., Shoeybi, M., and Catanzaro, B. Instructretro: Instruction tuning post retrieval-augmented pretraining. ICML, 2024.
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366 2023a.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In ACL, 2023b.
Wang, 7., Araki, J., Jiang, Z., Parvez, R., and Neubig, G. Learning to filter context for retrieval- augmented generation. arXiv preprint arXiv:2311.08.77 2023c.
14

---

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In ICLR, 2022.
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W. and Wang, Y. Pme-llama: toward building open- source language models for medicine. JAMIA, 2024.
Wu, Z.., Parish, R., Cheng, H., Min, S., Ammanabrolu, P., Ostendorf, M., and Hajishirzi, H. Inscit: Information-sen seeking conversations with mixed-initiative interactions. TACL, 2023.
Xiong, G., Jin, Q. Lu, 7., and Zhang, A. Benchmarking rermentd generation for medicine. arXiv preprint ar arXiv:2402.13178, 2024.
Xu, F., Shi, W., and Choi, RECOMP: Improving etrreval-augmeent LMs with context compres- sion and selective augmentation. In ICLR, 2024a.
Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C.,, Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. ICLR, 2024b.
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D.
HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
Yoran, O., Wolfson, T., Ram, O., and Berant, J. Making retrieval-augmmeted language models robust to irrelevant context. In ICLR, 2024.
Yu, W., Iter, D., Wang, S., Xu, Y., Ju, M., Sanyal. S., Zhu, C.,, Zeng, M., and Jiang, M. Generate rather than retrieve: Large language models are strong context generators. In ICLR, 2023a.
Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., and Yu, D. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.092111 2023b.
Yu, W., Zhang. Z., Liang, Z., Jiang, M., and Sabharwal, A. Improving language models via plug-and play retrieval feedback, 2024.
Yu, Y., Xiong, C., Sun, S., Zhang, C., and Overwijk, A. Coco-dr: Combating distribution shift in zero-shot dense retrieval with contrastive and distributionally robust learning. In EMNLP. 2022.
Zhang, T., Patil, S. G., Jain. N., Shen, S., Zaharia, M., Stoica, I., and Gonzalez. J. language model to domain specific rag. arXiv preprint ar 20112240..1111,, 2024.
Raft: Adapting
Zhu, F. Lei, W., Huang. Y., Wang, C., Zhang, S., Lv, J., Feng. F,, and Chua. T.-S. Tat-qa: A question answering benchmark on hybrid of tabular and textual content in finance. ACL, 2021.
Zhu, Y., Zhang, P., Zhang, C., Chen, Y., Xie, B., Dou, Z., Liu, Z., and Wen, J.-R. Inters: Unlocking the power of large language models in search with instruction tuning. In ACL, 2024.
15

---

A Dataset Description
The information for datasets used in RankRAG is listed as follows.
A.1 Main Experiments
NQ (Kwiatkowski et al., 2019) is a widely used prestion-answering dataset constructed with Wikipedia. The questions are constructed from the Google search engine, and the answers are identified as text spans in the Wikipedia article.
TriviaQA (Joshi et al., 2017) is a challenging QA dataset containing question-answer pairs from trivia enthusiasts and independently gathered evidence documents.
PopQA (Mallen al., 2023) is entity-centric QA dataset concentrated on long-tail entities. For PopQA, we follow (Asai et al., 2024a) to use the long-tail subset, consisting of questions on 1399 rare entities whose monthly Wikipedia page views less than 100.
HotpotQA (Yang al., 2018) is multi-hop QA dataset, where the goal is to answer complex questions that require understanding and linking information from multiple documents.
2WikimQA (Ho al., 2020) is also a multi-hop QA designed to test machine understanding across two different Wikipedia entities, evaluating the ability of systems to handle cross-lingual and cross-cultural retrieval and question answering.
FEVER (Thorne al., 2018) a fact verification dataset aimed supporting research into the automatic verification of factual claims. It consists of claims that are manually verified against evidence from Wikipedia, providing a benchmark for fact-checking systems.
Doc2Dial (Feng et al., 2020) document-grouncee conversational QA dataset covering four domains: DMV, SSA, VA, and Student Aid. Each sample comprises a dialogue where a user poses queries regarding the document, and agent responds those questions. The average document length is around 101K words.
TopiOCQA (Adlakha et al., 2022) is grounded on the whole Wikipedia. It incorporates topic switching and requires the agent search the entire Wikipedia for answers questions.
INSCIT (Wu et al., 2023) is also grounded the whole Wikipedia. It studies the case where user questions are under-specified and require clarification.
A.2 Biomedical Benchmarks
MMLU-med (Hendrycks et al., 2021) is subset of six tasks related to biomedicine, including anatomy, clinical knowledge, professional medicine, human genetics, college medicine, and college biology. It contains 1089 questions in total.
MedQA (Jin et al., 2021) is collected from the US Medical Licensing Examination, contaiing 1273 four-option multiple-choice questions focused on real-world scenarios from professional medical board exams.
MedMCQA (Pal et al., 2022) includes multiple-choice questions derived from Indian medical entrance exams, covering 2400 healthcare topics across medical subjects. We use the 4,183. question development set from MedMCQA, as the test set lacks provided ground truths.
PubmedQA (Jin et al., 2019) is a biomedical research QA dataset consisting of 1000 manually annotated questions based on PubMed abstracts. Answers in PubMedQA are structured as yes/no/maybe to reflect the validity of the questions.
BioASQ (Tsatsaronis et al., 2015) includes 618 questions constructed from biomedical litera- ture without providing the ground truth snippets, challenging RAG systems to infer answers independently.
B
Data Blending Details for Ranking-enhanced Instruction Finetuning
The dataset blending ratio for Stage-Il as follows:
Drop: 0.069
16

---

narrativeqa: 0.09
quoref: 0.026
ropes: 0.026
Squad (Retrieval-augmmeetedd QA): 0.09
Squad (Retrieval-augmeetted Ranking): 0.02
WebQuestions (Retrieval-augmented QA): 0.09
WebQuestions (Regrieval-uugmeeteed Ranking): 0.02
newsqa: 0.09
tatqa-arithmetii: 0.15
tatqa-others: 0.08
ConvQA: 0.2
MS MARCO ranking: 0.15
ConvQA ranking: 0.03
SFT: 0.2
The ratio for each dataset is further normalized to ensure the total ratio equals to 1.
C Prompt Formats of Instruction Tuning
C.1 Stage I: Supervised Fine-tuning
The format template of LLM inputs in stage-I is as follows:
This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found the context.
System:
User: (Question 1}
Assistant: (Answer 1}
User: (Latest Question]
Assistant:
C.2
Stage-IT: Unified Instruction-Tine Tuning for Ranking and Generation
The format template of L.L inputs in stage-Il are as follows:
Context-rich QA data
This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found the context.
System:
Passage: {Goldd Passage containing relevant context QA}
User: (Question 1}
Assistant: {Answer 1}
17

---

User:
(Latest Question}
Assistant:
We tailor specific user instructions for various dataset types. For instance:
For datasets requiring short answers (such as DROP, NarrativeQA, Quoref, ROPES, SQUADI.1, SQuAD2.0, NewsQA), we use: "Answer the following question with short span."
For datasets that necessitate long answers (such as full and complete answer for the question."
we instruct: "Please give
For datasets involving arithmetic calculations or number extraction from the context (such as TAT- QA), we specify: "Answer the following question with a number from the context or through math arithmetic."
For datasets that may require both short and long answers (such as TAT-QA--thhrs), we direct: "Answer the following question with short span, full and complete answer."
2) Retreeeal-augmented QA data
System: This is chat between user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user s questions based the context. The assistant should also indicate when the answer cannot be found in the context.
Passage 1: (Shuffledd Passage 1}
Passage 2: (Shuffled) Passage 2}
Passage 3: ((huffleed Passage 3}
Passage 4: (Shuffledd Passage 4}
Passage 5: (Shuffledd Passage 5}
User:
{Question}
Assistant:
3) Context ranking data
System: This is chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found the context.
Passage: (Passage 1}
User: {For the question <question>, access whether the passage relevant to the question. Return True if relevant, otherwise False. }
Assistant:
4) Retrieeall-ugmmented ranking data
System: This is chat between and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's 's questions based the context. The assistant should also indicate when the answer cannot be
18

---

found in the context.
Passage 1: (Shuffleed Passage 1}
Passage 2: (Shuffleed) Passage 2}
Passage 3: (Shuffleed) Passage 3}
Passage 4: (Shuffleed) Passage 4)
Passage 5: (Shuffledd Passage 5}
User: {For the question <question>, access whether the above passages relevant to the question. Return all the relevant passage id. }
Assistant:
D Prompt Formats of Target Tasks
D.1 Context Ranking
NOTTIiAQQAAAHOPPQQAAAAOOA
System: This is chat between and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's 's questions based the context. The assistant should also indicate when the answer cannot be found in the context.
Passage: (Passage)
User:
(For the question <question>, access whether the passage is relevant to the question. Return True if relevant, otherwise False. }
Assistant:
FEVER:
System: This is chat between user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based the context. The assistant should also indicate when the answer cannot be found the context.
Passage: (Passage)
User: {For the claim <claim>, access whether the passage is relevant to the claim. Return True if relevant, otherwise False. }
Assistant:
Doc2dial, Inscit, TopioeQA:
System:
This is chat between user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found the context.
Passage: (Passage)
User: {Question 1}
19

---

Assistant:
(Answer 1}
User:
(For the question <latest question>, access whether the passage is relevant to the question. Return True if relevant, otherwise False. }
Assistant:
D.2 RAG
NOTIiviQQAHooooQQPPPQQQ
System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context.
Passage (Rerank Top Passage 1}
Passage (Rerank Top Passage
Passage (Rerank Top Passage
Passage (Rerank Top Passage
Passage (Rerank Top Passage
User: (Question). Answer the above question with a short phrase.
Assistant:
Fever:
System: This is chat between user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found the context.
Passage (Rerank Top Passage
Passage (Rerank Top Passage
Passage 3:
(Rerank Top Passage 3}
Passage 4: (Rerank Top Passage 4}
Passage 5: (Rerank Top Passage 5)
User: Answer the following question with True or False. Is the claim <<laim>> correct?
Assistant:
Doc2dial, Inseit, TopiOCQA:
System: This is chat between and an artificial intelligence assistant.
20

---

The assistant gives helpful, detailed, and polite answers to the user's questions based on the context, The assistant should also indicate when the answer cannot be found in the context.
Passage 1: (Rerank Top Passage 1}
Passage 2: (Rerank Top Passage 2}
Passage 3:
(Rerank Top Passage 3}
Passage 4: (Rerank Top Passage 4}
Passage 5: (Rerank Top Passage 5}
User: (Question 1}
Assistant: {Answer 1}
User:
(Latest Question}
Assistant:
E
Additional Experiment Results
E.1
Ranking Performance Using DPR and Contriever Retrievers R
Table shows the ranking performance of RankRAG-8B using DPR (Karpukhin et al., 2020) and Contriever (Izacard et al., 2022) on three datasets. There are consistent performance gains for all tasks, indicating that RankRAG can apply to many popular retrieval models to improve the quality of retrieved contents.
Table Answer Recall Comparison Before and After Ranking on 3 Representative Datasets.
DPR
Contriever
NQ
R@5
R@ 10
R@20
R@5
R@10
R@-20
Before Ranking
69.50%
76.20%
81.00%
67.60%
75.24%
80.67%
w/ RankRAG
77.95%
81.70%
84.56%
75.32%
80.18%
84.70%
DPR
Contriever
TriviaQA
R@5
R@ 10
R@20
R@5
R@ 10
R@20
Before Ranking 67.80%
74.20%
80.30%
81.95% %
86.76%
90.08%
w/ RankRAG
77.73%
79.40%
84.74%
88.71% %
90.05%
92.59%
DPR
Contriever
PopQA
R@5
R@ 10
R@20
R@5
R@ 10
R@200
Before Ranking
43.60%
48.90%
54.25%
60.61 % 65.11%
65.54%
69.90%
w/ RankRAG
50.32%
53.75%
57.76%
68.41%
71.77%
E.2 RAG Performance with Different
We also show the performance RankRAG with different context size k in figure From the result, we observe that different from the trend of vanilla RAG approaches (without ranking), A already works well for most datasets. This effectiveness stems from the reranking step, which prioritizes the most relevant contexts at the top, reducing the necessity to include additional contexts.
21

---

NQ
TriviaQA
PopQA
FEVER
52
84
60
93
Exacl Match
50
92
82-
48
46
80
50
90
Context Size
Figure 6: Performance of RankRAG on different context size k.
F
Performance of NQ and Trivia QA on DPR Splits
We observe that the NQ and TriviaQA datasets exist in two versions: one used by the DPR (Karpukhin et al., 2020) and FiD (Izacard Grave, 2021) papers, which include 3610 and 11316 questions for NQ and TriviaQA, respectively. In contrast, the KILT benchmark (Petroni et al., 2021) utilizes only subsets these, comprising 2837 and 5355 examples for NQ and TriviaQA, respectively. is noteworthy that many recent studies report performance metrics these datasets without clarifying which version was employed for evaluation.
To facilitate honest and fair comparison, present the performance of RankRAG on both datasets using the DPR splits Table 9. Notably, regardless of the subset used, Ra consistently outperforms both ChatQA and Llama-3-instruct, our direct competitors, as well other methods utilizing InstructGPT as backbones. We aim for these results to assist the community in making accurate comparisons when referring to the performance of RankRAG.
Table 9: Performance Across Models.
Model
Model Configuration
NQ EM (%)
TriviaQA EM Acc. (%)
Representative Baselines
GPT-3.5-0613
35.2
70.1 81.3
GPT-3.5-0613 RAG
42.3
65.8 76.7
GPT-4-0613
37.2
72.6 85.1
OpenAl GPT
GPT-4-0613 RAG
36.2
61.2/75,9
GPT-4-turbo-2024 201tbo--22--4099
38.3
68.0/ 84.5
GPT-4-tubbo-2224 RAG
36.3
57.6/79.2
Using Llama-2 (Touvron al., 2023) the backbone LLM
Llama-2 Chat
Llama-2 RAG 70B
37.7
65.6
Llama-2 2 7B
37.0
74.3
ChatQA-1.0
Llama-2 13B
43.9
66.6 76.9
Llama-2 70B
45.0
69.8 80.2
Llama 2 7B
42.4
68.3/78.9
RankRAG
Llama-2 13B
46.2
/ 80.0
Llama-2 70B
48.7
72.3 82.6
Using 3 (Meta-AI, 2024) as the backbone LLM
Llama-3 Instruct RAG 8B
27.6
57.1 74.66
Llama-3- Instruct
Llama-3 Instruct RAG
37.3
67.6
1.aam-3 8B
44.1
65.4/75.. 75.88
CharQA-1.5
Llama-3 70B
46.0
69.0 80.4
Llama-3 8B
46.1
68.8 79.9
RankRAG
Llama-3 70B
50.0
72.6 82,9
G Additional Case Studies
Tables 10 and provide additional examples from the PopQA and HotpotQA datasets, which focus on long-tailed and multi-hop QA. These tasks are particularly challenging for retrievers, making it difficult to obtain relevant context from the corpus. Consequently, ChatQA-1.5 often struggles to produce the correct answers. However, the reranking step in RankRAG helps counteract poor initial
22

---

Table 10: A case study the top-retrieved context and predictions on PopQA dataset, illustrating the effectiveness of RankRAG-8B over ChatQA--.5-8BB. Red text denotes distractors, while green stands for evidences.
Q: Who is the author of The Universe Around Us?
A: James Hopwood Jeans / /James Jeans/ Sir James Jeans/ / Sir James Hopwood Jeans
Passuge I: Hooper is the author of two books published Smithsonian Booko/HaperCollnns The first. Dark Coemos: In Seurch of our Universe's Missing Mass
Energy (2006) was nomed notable book by Seed Magazine. His second book. Nature's Blueprint: Supersyymerry and the Search for a Unitted Theary of Matter and Force (2008). was called "essential reading" by New Scientist...
Passage 2: Fraknoi is the author or co-author of 14 books in the field of astronomy, He was the lead author of Voy Voyages through the jexibook publishaed by Brooks Cole, which went through three editions. In the 1980s, he science liction stories, "The
Universe, introdactory college
co-odited Byron Prciss collections science articks Wolff, Fraknei founded and was co-editor of the first on on-line journal devoted to astronomy education, 'Astrooomy Education Review" ... Passage 3: The Universe' and "I he Planets. With Sidney Wolff, Praknni founded and was co-editor of the first on-line jouraal devoted to astronomyy education. 'Asmooomy Education Review". He edited two collections of resources for K- 12 teachers, The Universe Your Fingertips and More Universe at Your Fingamips published through the Astronomical Society of.. Passage 4: Linooln Kinnear Bament 19899-1979) was editor and author,
and
Plancts." With Sidncy
ChaiQA-1.5
notably Lifec Magazine for many years. Lincoln
Barneil wrote a number of books, including "The Universe and Doctor Einstein"
Passage 5: The Univense Maker is a science fiction novel by American author A.E. van Vogt, published in 1953 by Aze Books as an Ace Double with The World of Null-A. is based on the authors Thhe Shadow Men' (Starting Stories, 1950). Set 400 years into the future.
the main character is Monon Cargill, a U.S. Ammy officer who served in the Korean War...
Prediction: Lincoln Barnett (X)
Passage I:
The Universe Around Us science book writen by English astrophysicist Sir James Jeans, first published 1929 the Syndics Cambridge University Press...
Passage 2: Those books made Jeans faidy well known as an expositor of the revolutiocur scientific discoverics of his day, espocially in relitivity and physical counology. In 1939, the Joumaal of the British Astronomicaa Association reported that Jeans was going to stand as candidate for
Passage 3: James Jeans books for the lay public, including "The Sturs in Their Courses' (1931), "The Universe Around Us", "Thmogh Space and Time"" (1934), 'The Background Science" (1933), 'The Mysterious Universe. Passage 4: The Universe Around Us no special scientillc lodge Purts of book cover the same ground as various lectures have recemlly delivered to University and other sudiences, incloding a course of wircless talks I gave last auturnn It has boen found necessary rewrite these almust their cutirety, that very low sentencces remmin Uheir original form, but those who have asked my lectures wireless Passage S-Lincodn Burnett Lincoln Kinnear Barnett (1909-1979))) 1979) was an editer author, most notably Life Magazine many years. Lincoln Barnett wrote a number of books, including 'The Universe and Doctor Einstein", "The World We Live In", and "The Treasure of Our Tongue". The Universe and Doctor Einstein is laymaná introduction the theory of relazivity It includes foreword by Albert Einstein, and has been reprinted several times...
RaokRAG
publish
will find the substance of in present book.
Prediction: Sir James Jeans (V)
Table 11: A case study on the top-retrieved context and predictions on HotpotQA dataset, illustrating the effectiveness of RankRAG-8B over ChatQA-1.5 Red text denotes distractors, while green stands for evidences.
Q: W Which song did Eminem and Rihanna collaahorate on after their other collaboration soeg in studio almmm "Unapologetic"" A: The Monster
Passage 1: Unapologetic the seventh studoo album by Burbodian singer Rihanna. It was relessed on November 19, 2012. by Def Jam Recordings and SRP Records. was recorded berween June and November 2012, during promotion of (2011). As executive producer, Rihamaa enlisted previous collaboratoes The Dream, David Guetta, Chase & Status, and SturGate to week alongside new collahorraoors such as Parker Ighile, Mike Will Made-It, It, and I Lhhinnhh... Passag2 2: Def Jam France announcd albun was scholule released in November
album. "Talk That
Twitter that Rihama would release
single the upcoming week while her sewett studio of tweets
October
that the
hor album
"Umapologcicc" alongside with its cover. "What Now" was written by British singer songwriter Livvi Franc logother with Rihanna, Parker [ghile and Natlian Cassells, while the production of the song was done by the latter twa. lgbile and Cassell...
Passuge 3: Justin then went on to write "Stay' with Mikky Ekão and recorded by Barbadian singer Rihanna for her seventh studio CharQA-L.5 alhum. "Uiappolgeetc" (2012), features guest vocals by Milky Fikoo and released the second single from the album lanuary 2013.
song reached top five twenty-four countries worldwide including number the US Billboard Hot 100, becoming Rihanna's twenty- fourth top ton on the latter chart...
the and number three
Passage 4: Via her ofiicial Twitter account, Rihanna posted series of "teasing" tweets ancouncing her seventh studio album. On October her
11, 2012, in
revealed that the title of her new album is "Unapologetic" alongside with its cover. "Jump" the overall seventh and final single Unapolegetic. was written by Kevin Cossom and B. Williams together (Mikkel S. Erikssn and Tor Erik Hermansen) and Chase & Status (Saul Milton.
its producers StarGate
Passage S: copies of the song were sold the UK, making "Love the Way You Liee the country's 's higgest-sellin song of 2010. The same year, sequel to the single, titled "Love the Way You Lie (Part II" was released as part of Rihanna is fifth studio album "Loud"; it mainly views
the female protagomist perspective November 2012, Eminem and Rihanna collaborntdd again "Numb", which was included on Rihama''s seventh album "Umppologetie...
Prediction: Love the Way You Lie (X)
Passage 1: The Monster (song). Copies of the song were sold in the UK, making 'Love the Way You Lie" the country's biggest-sell selling song of 2010, The same year, a sequel to the single, titeed "Love the Way You Lie (Pan III was released as par of Rihanna's fifth studio album 'Lond': mainly views marters from the female protagonist perspective. In November 2012, Eminem and Rihanna collaborated again on "Numb", ,
was included Rihanna's seventh album "Unapologetic" ....
Passage 2: "Numhh is a song by Barbadian singer Rihanna from her seventh studio alham "Unapologeric" (2012). fearures guest vocals by American rapper Eminem, making it the pair's mir's third collabortioon since the two ollicial varsios of 'Love the Way You Lic" Kollow ing the album's release, "Numb" charted muliippe charts worldwide including Canada, the United Kingsom and the United States. "Numh" lasts for curation
fohl Rubin could help him "take that to another level." Rihanna, with whom Eminem previously collaborated on "Love the Way You Lie" from Eminem's 's previous studio effert, "Recovary" (2010), was featured on the song "The Monster.. On O September 11. 2013, she hinted at the...
Passage Eminom also wanted experiment with "retro, vintage" sounds such beaibreaks scratches,
RankRAG
Passuge 4: together with Jay- Z. Bono and The Edge for the same campaign to alleviate the 2010 Haiti earthqualie In summer 2010, Rihunna collabornted with rapper Eminem on "Love the Way You Lie", which comnties, Reaching numher 2 song hecame bigges- selling song of 2010 the UK and the finst of Rihanna's is singles to sell over a muilion capies in the country. In October 2010, Rihamna switched managets... Passuge St Eminem asked for more tracks and subsequently heard "Love the Way You Lie". He chose it and told his manager Paul Rosenberg he wanted to collaborate with the Batbodian singer Rihanna. Eminem told Skyrock, "It's only she could pull it Rosenberg sent the track Rihanna, who accepted Eminem's is request "at the moment.' Eminem then Prediction: The Monster (V)
major workdwide success, reaching No. over
those tracks that felt like
wrote the rapped verses.
retrieval by finding more pertinent evidence. Coupled with R RAG-oriented finetuning, RankRAG effectively filters out distracting entities and pinpoints the correct answers.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have comprehensive experimental results in $ 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of RankRAG in Appendix ??.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

This work does not propose any theory assumptions and does not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the details of the data collection for training and evaluations, which can be found in section 4, 5. We will open-source model weights and scripts for reproducing our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
25

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We open-source model weights and scripts for reproducing our results. For training data, they are all public available data with open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
Serriidee/CodeSuumiise
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / ctttpp..cc/ubbiceec c guwdguides/e// CoodeSubmis ssi assoonPoicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details can be found in section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All results are zero-shot and deterministic; we use greedy search for genera- tions. Retrieval and re-ranking scores also deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the compute resources information in section 5,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics https:: //tprrip///uui.....// cc htte//ublcc/ttiicccooee
Answer: [Yes]
Justification: This rescarch conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both potential societal impacts and negative impacts in Appendix
??.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all data, models used in the paper properly.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwoee.hcoo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "Y56I7ITysF", "reviewer": "Reviewer_ze6x", "summary": "This work proposes an effective approach that enhances existing RAG methods by introducing an additional context refinement step. This step filters out retrieved, but non-relevant contexts prior to including them as context in the input for answer generation.\n\nThe authors train context reranking alongside answer generation using a single LLM. They demonstrate that adding only a fraction of ranking data to the training blend yields superior ranking performance compared to training the same model in isolation on the ranking data, while outperforming other models that have been trained on 10 times more ranking data.\n\nFor evaluation, the authors compare their method on 9 general- and 5 specific-domain datasets, showing consistent improvements across LLM sizes for the Llama model family (Llama 2 and 3) over existing methods.", "questions": "- Why is a fixed number of contexts used? Some questions might need more contexts such as multi-hop datasets while others such as NQ (single context dataset) would need less. Instead of a score cutoff, why not use a dynamic number of contexts that is determined by the binary true/false label the mode already outputs?\n- In Section 5.4, the paper states that it incurs an additional 1.8× to 7.0× increase in time, significantly less than the 20× to 100× increase one might expect. It is not sufficiently explained why one would expect a 100x increase in time.\n- The approach for reranking context looks at passages individually, but retrieval training included listwise ranking. Was listwise ranking also tried for context reranking?", "limitations": "The authors addressed the limitations adequately, however, could emphasize more on the dramatic inference time increase that results from the reranking step.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces the novel idea of using the same LLM to first assess the relevance of individual contexts in a cross-encoder ranking style before using them as input for answer generation.\n\n- The proposed RankRAG outperforms existing methods on various general and specific-domain datasets.\n\n- Extensive experimentation and ablations covers a wide range of possible setups including LLM size, retriever, and efficiency and effect of different model components.", "weaknesses": "Main concern: \n- Reranking contributes only around 5% of the overall effectiveness on average (Table 4 RankRAG compared to Llama3-ChatQA-1.5-X), and the 7x computational overhead in inference time raises questions about its realistic application, given the computational demands of large models already without ranking contexts. The ratio of performance gained ( in combination with no sig. testing) and increase in computation is my main issue with this work. I am aware reranking fewer contexts decreases performance, however, similarity decreases performance gain over other methods. \n\n- No significance testing was done (for both generation on ranking ) to strengthen effectiveness claims as differences for most datasets are minor. Authors \njustify in their additionally provided checklist that sig. testing is not needed as generation and ranking is deterministic, this however, touches upon a different aspect and does not remove the need for testing whether the performance is significantly better than previous methods.\nFurther, improvements in Table 2 stem from NQ and PopQA which are relatively simple datasets that can be answered with a single context, therefore it is not apparent why RankRAG would particularly excel at those. Moreover, the ranking performance for these datasets in Table 6 only marginally improves over other rerankers, therefore it is to be expected to obtain similar gains when replacing the RankRAG reranker module  with other strong rerankers. As a side note averaging over different metrics - even though seen in many recent works - should not be done. \n\n\nOther points to improve upon:\n- No information about the crowdsourced dataset is provided.\n- In Section 4.2.3, the claim that LLM needs to be robust against irrelevant context contradicts the proposal to filter out irrelevant context beforehand.\n\nSome experimental setups are unclear:\n- It is not described how true/false tokens from context ranking are translated into a score that can be used for ranking.\n- In Section 5.1, it is not clear if baselines use different retrieval setups; otherwise, effectiveness claims do not seem valid.\n- It is unclear which number k is eventually used in RankRAG. The paper mentions optimizing for\nk=5,10,20, the baselines. \n\n- The related work section could mention GRITLM as the first model jointly training answer generation and ranking for RAG: Muennighoff, Niklas, et al. \"Generative representational instruction tuning.\" arXiv preprint arXiv:2402.09906 (2024).\n\nIssues in Writing:\n\n- Line 332: \"Observe that even with N = 20, We noted that\" – incomplete sentence.\n\n- Line 41: \"in RAG framework\" -> \"in the RAG framework\".\n\n- Caption Fig 1: \"the strongest RAG model,\" -> \"the strongest RAG models,\".\n\n- Line 100: \"embedding model\" -> \"embedding models\".\n\n- Line 111: \"As illurstraded\" -> \"As illustrated\".\n\n- Line 112: \"one of the strongest model.\" -> \"one of the strongest models\".\n\n- Line 157: \"that, it is\" -> \"that it is\".\n\n- Line 165: \"The LLM need\" -> \"The LLM needs\".\n\n- Line 169: \"ranking data are\" -> \"ranking data is\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments and feedback. We discuss your main concerns in the following. \n\n---\n> “Reranking contributes only around 5% of the overall effectiveness on average (Table 4 RankRAG compared to Llama3-ChatQA-1.5-X), and the 7x computational overhead in inference time raises questions about its realistic application\n- First, we would like to clarify that the 7x computational overhead of ChatQA-1.5-8B occurs when RankRAG-8B reranks 100 retrieved contexts. By doing so, the smaller RankRAG-8B outperforms most state-of-the-art large models including Mixtral-8x22B-Instruct and RePlug 65B, except ChatQA-1.5 70B and Claude 2. Notably, RankRAG-8B even outperforms ChatQA-1.5-70B on NQ (50.6 vs. 47.0), PopQA (57.7 vs. 50.9), and Inscit (33.3 vs. 32.3). Its average score across nine datasets (52.6) is 3 points higher than same size ChatQA-1.5-8B (49.6), 5.5 points higher than much larger Llama3-Instruct 70B (47.1), while being just 1 point below SOTA ChatQA-1.5-70B (53.6).  These accuracy gains over frontier-class models are not marginal.\n\n- Second, RankRAG 70b only uses top-30 contexts for reranking (mentioned in Line 242) with around 2.5x overhead when compared to ChatQA-70b. \n\n- Third, the 7x overhead was calculated using a basic PyTorch setup without optimizations. Techniques like prefilling and batching could notably decrease this overhead since the instruction prompt and question are shared for 100 passages, suggesting significant potential for improved deployment efficiency.\n\n- We have also demonstrated a compelling accuracy-efficiency trade-off in Figure 5. For example, when RankRAG-8B reranks 30 contexts, the computational overhead is 2.5x that of ChatQA-1.5-8B, while its accuracy on NQ increases from 42.4 to 49.4, outperforming ChatQA-1.5-70B (NQ: 47.0) by 5%. We show the performance of RankRAG 8B with ranking 30 contexts, which consistently outperforms ChatQA 8B with an average gain of 2.3%. The gain is significant on 7 of 9 datasets.\n\n---\n> “No significance testing was done …“\n- Research on few/zero-shot evaluation of LLMs often lacks statistical significance reporting due to two main challenges: 1) most baselines, such as PaLM 2, RA-DIT, do not release model weights, prompts and prediction results; and 2) the zero-shot performance of LLMs shows significant variance, as evidenced by the Open LLM Leaderboard, where no single model consistently leads across all datasets.\n\n- Per your suggestion, we conducted a paired statistical significance test for RankRAG against ChatQA-1.5, a strong and open-sourced baseline. Results from Fisher's randomization test are in Tables 1 (RAG) and Table 2 (Ranking) of the supplementary PDF. RankRAG significantly outperforms ChatQA-1.5 on 8/7 out of 9 datasets for the 8B/70B variants. In ranking tasks, RankRAG scores 5/4/3 out of 5 datasets for Recall@5/10/20, respectively.\n\n---\n> “Improvements in Table 2 stem from NQ and PopQA which are relatively simple datasets that can be answered with a single context, therefore it is not apparent why RankRAG would particularly excel at those.” \n\n- While NQ and PopQA are single-hop QA datasets, some of the questions require long-tailed knowledge from Wikipedia, leading to subpar initial retrieval with a recall@5 below 75%, which is 14% lower than TriviaQA. RankRAG significantly enhances passage recall by 6%-12% by effectively reranking top passages.\n\n---\n> \"Moreover, the ranking performance for these datasets in Table 6 only marginally improves over other rerankers, therefore it is to be expected to obtain similar gains when replacing the RankRAG reranker module with other strong rerankers.\"\n\n- We have reported the result of RankRAG and two strong baselines using RankLlama 8B as the ranker in Table 4 of the supplementary PDF.  RankRAG outperforms these on 8 of 9 datasets, with significant gains on 6 datasets. Another advantage of RankRAG is its label and memory efficiency; it requires less labeled data for training and uses only one model during inference, unlike baselines that lack these efficiencies, while the baselines using other rerankers does not have the advantage of efficiency.\n\n---\n> \"As a side note averaging over different metrics - even though seen in many recent works - should not be done.”\n\n- Thanks for this advice. We will modify the table.\n\n---\n> \"No information about the crowdsourced dataset is provided.\"\n\n- In this work, we directly use the crowdsourced dataset from ChatQA work without further modification. We will include a reference to  Sec 3.2.1 of the ChatQA paper in the revision.\n\n\n---\n> \"In Section 4.2.3, the claim that LLM needs to be robust against irrelevant context contradicts the proposal to filter out irrelevant context beforehand.\"\n\n- Both filtering out irrelevant context and training LLMs to be robust against irrelevant context are designed to generate accurate answers. Specifically, reranking may not ensure that all top-ranked documents are relevant to the question. Enhancing the LLM's robustness to such irrelevant contexts also contributes to accurate generation. Empirically, as shown in Table 4 of the main paper, incorporating noise-robust training techniques has improved zero-shot RAG performance on these datasets by over 1%.\n\n---\n> \"It is not described how true/false tokens from context ranking are translated into a score that can be used for ranking.\"\n- We use the probability of the <True> token as a proxy of relevance score for ranking.\n\n---\n> \"It is unclear which number k is eventually used in RankRAG.\"\n- We set a fixed k=5 for RankRAG to simplify hyperparameter tuning and expedite inference. The performance of RankRAG across different k is in Figure 6, Appendix G.2.\n\n---\n---\nWe appreciate you taking the time to review our paper again. We have added an additional comment to address the remaining questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L1IKzIbbpC", "reviewer": "Reviewer_LbZs", "summary": "The authors introduce a novel approach to instruction fine-tuning large language models (LLMs) for ranking and answer generation tasks.\nTheir approach involves two main steps:\n1. Supervised Fine-Tuning: Initially, the LLM is fine-tuned on a general instruction-following dataset.\n2. Ranking Task Fine-Tuning: The LLM is then further fine-tuned using a mix of different instruction ranking datasets that contains multiple ranking-oriented tasks.\n\nIncorporating ranking-based data during fine-tuning enhances the LLM's ability to rank documents retrieved through retrieval-augmented generation (RAG). The fine-tuned LLM is then used to rank the RAG-retrieved documents, and only the top-k ranked documents are added in the context.\n\nThe authors demonstrate that their method significantly improves the LLM's performance for knowledge intensive RAG based tasks.", "questions": "N/A", "limitations": "The authors have acknowledged the limitations, specifically in terms of latency and its lack of training on code or mathematical data.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors introduce a novel method of fine-tuning LLMs for combined ranking and answer generation tasks.\n2. The proposed method outperforms existing approaches on RAG benchmarks, especially on challenging datasets where RAG retrieval is suboptimal, due to the LLM-based re-ranking.\n3. Their model exceeds the performance of re-ranking models trained on much larger datasets.\n4. The fine-tuned LLM demonstrates strong generalization capabilities, showcased by its performance on medical benchmarks.\n5. The paper is well-written and includes exhaustive experiments.", "weaknesses": "1. Scoring each document individually increases the latency significantly. Have you considered scoring multiple documents simultaneously? Similar to retrieval-augmented ranking dataset, you could input a group of documents and score them in a single pass. The group size could be tunable, balancing performance drop against latency improvement.\n2. It would be helpful to include examples where RankRAG fails, particularly in cases where relevant documents are not ranked higher. Providing these examples can help understand scenarios where it might not perform optimally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments and feedback. We discuss your raised points in the following. \n\n---\n> \"1. Scoring each document individually increases the latency significantly. Have you considered scoring multiple documents simultaneously? Similar to retrieval-augmented ranking dataset, you could input a group of documents and score them in a single pass. The group size could be tunable, balancing performance drop against latency improvement.\"\n- Thank you for this nice suggestion. In our initial attempt, we tried to use the listwise reranking approach with a similar format to the retrieval-augmented ranking dataset to fulfill the reranking purpose but did not observe the performance gain. We will definitely consider your suggestion to explore how to further improve the efficiency of the reranking step. \n\n---\n> \"2. It would be helpful to include examples where RankRAG fails, particularly in cases where relevant documents are not ranked higher. Providing these examples can help understand scenarios where it might not perform optimally.\"\n\n- Thank you for this excellent suggestion. We will include the examples in the final version of the paper. Specifically, RankRAG may encounter difficulties in the following scenarios:\n  - QA involving long-tailed knowledge, where poor initial retrieval excludes relevant documents from the top-N contexts, preventing further ranking.\n  - Multi-hop QA tasks, where finding relevant documents can be challenging as it requires multi-hop reasoning beyond simple keyword or semantic matching.\nTo alleviate these issues, potential solutions include incorporating more powerful retrieval models [1] or implementing a multi-step reranking strategy [2].\n\n[1] Wang et al. \"Improving text embeddings with large language models.\" arXiv preprint arXiv:2401.00368 (2023). \n\n[2] Khalifa et al. \"Few-shot reranking for multi-hop QA via language model prompting.\" arXiv preprint arXiv:2205.12650 (2022).\n\n---\n---\nThank you once again for your insightful review. We appreciate your feedback on our work. If you have any further questions, please let us know. We would be happy to discuss them further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "73XmgwSF5R", "reviewer": "Reviewer_y5n5", "summary": "This paper proposed a method about how to use a single LLM for both context reranking and answer in RAG tasks. Particularly, they finetuned a LLM with both ranking data and QA data with two stages. For inference, they use the trained llm to sample the top_k contexts at first, and then input them into llm to get the answer. Compared with normal rag workflow, this method choose the contexts by llm itself. Experiments in some QA tasks evaluated the QA effectiveness of this LLM.", "questions": "1. Can you analyze the relationship between high retrieval call and QA performance in RankRAG for limitation2 on line 26? Can you provide further analysis to demonstrate that your method can solve this limitation?\n\n2. Have you conducted experiments to test whether RankRAG's reranking performance is robust when using different retrievers on lines 300?", "limitations": "No potential negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Two stages of training enhance the reranking capacity of llm using fewer ranking data.\n2. Using just a single llm for context ranking and answer at the same time.", "weaknesses": "1. The motivation is unclear. In lines 23 to 31, limitations 1 and 2 precisely explain the need for a reranker, which is also a challenge related to retrieval. Only with limited text in limitation 3  briefly mentions why the LLM is used for reranking contexts due to zero-shot performance, which has been proved with other rerankers in this paper. It is unclear about why using llm reranking itself rather than a separate reranker, maybe including the semantic gap between the reranker and llm or the larger length of reranking input at once.\n2. Insufficient experiments in same reranking setting. It is lack of baselines with RAG methods also using a reranker, which can prove the core reranking effectiveness of RankRAG, such as rankgpt(Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents) and rankvicuna(RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models). And also, the amount of top-k contexts after reranking is also hard to find in Table 2&6. How many contexts for reranking are input into llm at once? In addition, there are some blank spaces in Table 2 that are not convincing. \n3. Incremental techniques. Compared with ChatQA, RankRAG introduced the new training data RQA and RAR in stage 2th and reranking phrase during inference. But as shown in ablation study, each component contribute a little.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments and feedback. We discuss your raised points in the following. \n\n---\n> “1. The motivation is unclear. … It is unclear about why using llm reranking itself rather than a separate reranker, maybe including the semantic gap between the reranker and llm or the larger length of reranking input at once.”\n- Thank you for this insightful comment. The key advantages of using an LLM for both ranking and generation include:\n\n- **Performance**: As shown in Table 6 of the main paper, the reranking performance with LLMs surpasses that of state-of-the-art ranking models, likely due to enhanced transfer learning between RAG and ranking tasks facilitated by similar input formats (Table 1).\n\n- **Label Efficiency**: LLM-based reranking reduces the need for labeled data, as demonstrated by RankRAG's superiority over RankLlama, which requires ten times more data. This is particularly beneficial in resource-scarce settings.\n\n- **Memory Efficiency**: RankRAG operates with a single model during inference, enhancing deployment efficiency.\nTo further illustrate the performance gains, we have exhibited the RAG performance of ChatQA-1.5, Llama-3-instruct, and RankRAG using RankLlama as the Reranker in Table 4 of the supplementary PDF. The results justify that RankRAG outperforms these on 8 of 9 datasets, with significant gains on 6 datasets. \n\n---\n> “2. Insufficient experiments in same reranking setting. It is lack of baselines with RAG methods also using a reranker, which can prove the core reranking effectiveness of RankRAG, such as rankgpt(Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents) and rankvicuna(RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models)”\n\n- Thanks for pointing out these works. We have compared RankRAG with RankVicuna, RankGPT-3.5 and RankGPT-4 in Table 6 of the supplementary PDF.  We observe that RankRAG outperforms all of them on 5 datasets. We will mention these papers in the “related work” section and add these comparison in the next version of the work.\n\n---\n> “And also, the amount of top-k contexts after reranking is also hard to find in Table 2&6. How many contexts for reranking are input into llm at once? ”\n- We set a fixed k=5 for RankRAG to simplify hyperparameter tuning and expedite inference. The performance of RankRAG across different k is in Figure 6, Appendix G.2.\n\n---\n> “In addition, there are some blank spaces in Table 2 that are not convincing.”\n\n- Thanks for pointing this out. In table 2,  most numbers are from existing papers to ensure fair comparisons and many existing models are either not-public (e.g. PaLM2, RA-DIT, RePlug) or only test on a small number of tasks, thus, it is somehow challenging for us to provide additional results for these models. In response to your suggestion, we included results for all 9 datasets using three OpenAI models (GPT-3.5-turbo-1106, GPT-4-0125, GPT-4-turbo-0409), both with and without retrieval, detailed in Table 5 of the supplementary PDF. RankRAG outperforms these models on 7 out of 9 datasets, achieving average gains of 5.8% and 9.3% for the 8B and 70B variants, respectively.\n\n---\n> “3. Incremental techniques. Compared with ChatQA, RankRAG introduced the new training data RQA and RAR in stage 2th and reranking phrase during inference. But as shown in ablation study, each component contribute a little.”\n\n- With all these techniques combined, our RankRAG-8B consistently outperforms ChatQA-1.5-8B on all 9 RAG tasks. RQA and RAR contribute to this performance gain in the majority of these tasks—6 for RQA and 8 for RAR, respectively. Notably, the final RankRAG-8B even outperforms much larger ChatQA-1.5-70B on NQ (50.6 vs. 47.0), PopQA (57.7 vs. 50.9), and Inscit (33.3 vs. 32.3). Its average score across nine datasets (52.6) is 3 points higher than same size ChatQA-1.5-8B (49.6), 5.5 points higher than much larger Llama3-Instruct 70B (47.1), while being just 1 point below SOTA ChatQA-1.5-70B (53.6). It also outperforms other  state-of-the-art large models including Mixtral-8x22B-Instruct and RePlug 65B. \n\n- Our RankRAG-70B outperforms ChatQA-1.5-70B on 8 out of 9 RAG tasks. Its average score 56.1 is also much better than ChatQA-1.5-70B’s 53.6, which already suppasses GPT-4 Turbo on many RAG tasks. In this work, we improve the performance of frontier-class ChatQA-1.5 by a margin, which is non-trivial.  \n\n---\n> “ Questions: 1. Can you analyze the relationship between high retrieval call and QA performance in RankRAG for limitation2 on line 26? Can you provide further analysis to demonstrate that your method can solve this limitation?”\n- Table 6 highlights that the initial recall of Dragon retriever is often inadequate, with Recall@5 below 70% for PopQA and under 50% for HotpotQA. RankRAG significantly boosts the recall of relevant content, improving Recall@5 by 12% and 9% for PopQA and HotpotQA, respectively. This enhancement in recall translates into notable performance improvements—Table 4 shows absolute gains of 8% and 4% on these two datasets. These findings further justify RankRAG's effectiveness in addressing these limitations.\n\n---\n> “ Questions: 2. Have you conducted experiments to test whether RankRAG's reranking performance is robust when using different retrievers on lines 300?”\n- Yes, we detailed the reranking performance in Table 8, Appendix G.1 of our manuscript. There, we find that RankRAG's reranking notably enhances performance, achieving over **8% gains for DPR** and **7% for Contriever** in terms of Recall@5 on average, compared to using only the retriever.\n\n---\n---\nThank you once again for your insightful review. We appreciate your feedback on our work. Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their thoughtful feedback. 

In addition to addressing the detailed questions in each review, we have summarized the new experiments suggested by the reviewers below:

- **(ze6x, Table 1,2)** We have included the statistical significance test for both Generation (Table 1) and Ranking (Table 2) to demonstrate that the gain of RankRAG is significant.

- **(ze6x, Table 3)** We have shown additional experiments on RankRAG 8B that rank 30 contexts only, and it still consistently outperforms ChatQA-1.5 8B by a margin.

- **(ze6x, y5n5, Table 4)** We report the performance of baselines and RankRAG using RankLlama 8B as the reranker, demonstrating the advantage of using RankRAG for reranking compared to other off-the-shelf strong rerankers.

- **(y5n5, Table 5)** We have shown the performance of three OpenAI-GPT series model (GPT-3.5-turbo, GPT-4, GPT-4-turbo) with and without RAG, and demonstrated that RankRAG consistently outperforms them. 

- **(ze6x, y5n5, Table 6)**  We show the performance of RankRAG over three strong ranking models (e.g., RankVicuna, RankGPT-3.5, RankGPT-4), which further justifies the efficacy of RankRAG on passage ranking tasks for RAG applications.

- **(ze6x, Table 7)** We provide an empirical comparison on the static (k=5) and dynamic number of contexts of RankRAG on four RAG tasks.

- **(ze6x, Table 8)**  We compare RankRAG and Self-RAG with the same retrieval setups and find that RankRAG consistently outperforms Self-RAG using both Contriever and DRAGON as the retrievers.


Please refer to the attached PDF for detailed information. We hope these extensive new results adequately address the concerns raised by the reviewers. If you have any further questions, please let us know; we would be happy to discuss them.

Meta-review: This paper proposes RankRAG, a method to enhance Retrieval Augmented Generation by using a single LLM for both context reranking and answer generation. The authors argue that this approach improves performance, label efficiency, and memory efficiency compared to using separate models for ranking and generation. They demonstrate improved performance on 9 question-answering benchmarks, particularly in cases where the initial retrieval is suboptimal. However, the approach requires multiple times more compute than the nearest baseline, but the authors argue that the compute is worth the cost.

The reviews for the paper are positive, however I have concerns about the use of a private crowd-sourced conversational dataset. Is there a way this could not be used and achieve similar results? There are also many important experiments that are needed to contextualize the improvements of this approach over baselines added in the rebuttal (in an attached pdf) and were not in the main text. Additionally, the writing of the paper could be made more clear (as noted in the author reviews). If accepted, I hope all these points are addressed.
Decision: Accept (poster)