Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
Authors: Zilong Wang
Zifeng Wang
Long Le
Steven Zheng
Swaroop Mishra
Vincent Perot
Yuwei Zhang
Anush Mattapalli
Ankur Taly
Jingbo Shang
Chen-Yu Lee
Tomas Pfister
Venue: ICLR 2025 Poster (2025)
Keywords: generative model
retrieval augmented generation
Primary area: generative models

Abstract: Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce Speculative RAG - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that Speculative RAG achieves state-of-the-art performance with reduced latency on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 50.83% compared to conventional RAG systems on PubHealth.

Published as a conference paper at ICLR 2025
SPECULATIVE RAG: ENHANCING RETRIEVAL AUG- MENTED GENERATION THROUGH DRAFTING
Zilong Wang'~ Zifeng Wang Long T. Le
Huaixiu Steven Zheng
Swaroop Mishra
Anush Mattapalli Ankur Taly Jingho Shang' Chen-Yu Lee² Tomas Pfister
Vincent Perot
Yuwei Zhang'
University of California, San Diego ?Google Cloud AI Research
'Google DeepMind
'Google Cloud Al
ABSTRACT
Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more ac- curate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we intro- duce SPECULATIVE RAG a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by smaller, distilled specialist L.M. Each draft is generated from a distinet subset of retrieved docu- ments, offering diverse perspectives counts per draft. This approach enhances comprehension of each subset and mit- igates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM per- forming a single verification pass over the drafts. Extensive experiments demon- strate that SPECULATIVE RAG achieves state-of-the-an performance with re- duced latency on TriviaQA, MuSiQue, PopQA, PubHealth, and IRC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing la- tency by 50.83% compared to conventional RAG systems on PubHealth.
the evidence while reducing input token
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable success in question answering tasks (Brown et al., 2020; Achiam et al., 2023; Team et al., 2023). Trained on massive datasets, LLMs leverage their extensive parametric memory to generate seemingly plausible responses to user queries (Kojima et al., 2022; Kamalloo et al., 2023). However, when faced with knowledge- intensive questions demanding up-to-date information obscure facts (Petroni al., 2021), LLMs struggle with factual inaccuracies and produce hallucinated contents (Huang et al., 2023).
Retrieval Augmented Generation (RAG) has emerged promising solution to mitigate these is- sues. By incorporating information retrieved from an external database into the context (Gao et al., 2023b), RAG effectively reduces factual errors in knowledge-intensive tasks. This approach not only enables easy and efficient access to vast databases but also facilitates timely and accurate knowledge integration. Due the inherent limitations in the precision of current dense retriev- ers and the vastness of knowledge required to answer complex questions (Chen et al., 2022), RAG systems typically retrieve multiple documents to ensure the inclusion of all necessary information in the context (Petroni et al., 2021). This practice inevitably increases the length of the input to the LLMs, presenting significant challenges, particularly since encoding lengthy retrieved documents incurs additional latency and require more complex reasoning. Recent studies have explored ways to extend the context length limit of LLMs (Ding et al., 2023; Reid et al., 2024; Ma et al., 2024), yet achieving well-grounded reasoning over extended contexts remains open question (Liu al., 2024; Li et al., 2024). Consequently, striking a balance between efficiency and effectiveness in RAG has become central research question in the literature. Existing work on RAG systems primarily
Work done while the author student researcher at Google Cloud AT Research. Correspondence to: Zilong Wang <z szlwang@ussd. edu>, Chen-Yu Yu Lee <chenyulee@googgle com>

---

Published as a conference paper at ICLR 2025
Knowerdee-nteneee intenvvveeeeeeeveveee Query
Documents
Evieence supported Answers
2M
SpecialeT..
(a) Standard RAG
(b) Self- Self-Refiective RAG
(c) Corrective RAG
Promot
Cuary
Final
Overy
Answer
Generiest
Dock
proopperate
Gesterii
mho Generulat
48l dics
Speckal lags
reflect over Vic secroved cioot and the
daing
Use on NJ moocl to closelty resitvss docs
Pocopporai rolosone does and the oboo flom weo search into pronpe
geoeented woccaes
(d) Ours: Speculative Retrieval- Generation (Speculative RAG)
End generation or Conthue querying Genoralist LM for othar casts
UEnderstand diccutents paraller Drafter
A=0L
Bi
Evoluste drafts Generalist Accept best draft
fi
No need insmuctiinn
argmax Score( Q fr
Genoralist
+
h
Specialist RAG Dvafter
000
Evaluate each answer draft based the question the rationale parallel using Generalist and Accept the Best Draft.
When Generalist LM encounters knowledge- intensive queries?
Efficientl generate drafts and rationals /
Understand multiple perspectives of the docs Eocs the zere color are hoo the zame top's claster
Query Goneralist LM
Call Specialist RAG Drafter
Evalueto Accept Drafts into Generalist LM
Figure Illustration of different RAG approaches. Given a knowledge-intensive query Q and retrieved documents, (a) Standard RAG incorporates all documents into the prompt, increasing in- put length and slowing inference; (b) celf-Reelecttve RAG (Asai et al., 2023) requires specialized instruction-tuning of the general-purpose language model (LM) to generate specific tags for self- reflection; (c) Corrective RAG (Yan et al., 2024) employs an external retrieval evaluator to refine document quality, focusing solely contextual information without enhancing reasoning capabili- ties; (d) In contrast, our proposed SPECULATIVE RAG leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by smaller, specialized LM. Each draft is gener- ated from a distinct subset of retrieved documents, providing diverse perspectives on the evidence while minimizing the number of input tokens per draft.
concentrates on improving the quality of contextual information in retrieval outcomes, but often ne- glecting the latency issues associated with these systems (Ma al., 2023; Baek et al., 2023; Yan et al., 2024; Xie et al., 2023; Asai et al., 2023; Feng et al., 2023). These methods typically rely on multiple refinement iterations or customized instruction-tuning for self-critique abilities. Integrat- ing such enhancements into generic LMs requires additional training increased latency, posing practical challenges in real-world applications.
To this end, we introduce SPECULATIVE RAG, a RAG framework designed to offload computational burden to a smaller, specialist LM that serves as an efficient and robust RAG module for existing generalist LMs. Inspired by Speculative Decoding (Leviathan et al., 2023; Chen et al., 2023a; Xia et al., 2024a), which accelerates auto-regressive LM inference by concurrently generating multiple draft tokens with a smaller model and verifying them in parallel with the base model, our approach adapts this concept to RAG.
In SPECULATIVE RAG, partition retrieved documents into subsets for drafting answer candi- dates. We cluster the retrieved documents by content similarity and sample one document from each cluster to form subset, minimizing redundancy and maximizing diversity. These document subsets are then fed to multiple instances of the RAG module, which generate draft answers with corresponding rationales in parallel. This smaller, specialized RAG module, excels at reasoning over retrieved documents and can rapidly produce accurate responses. Subsequently, the generalist LM bypasses the detailed review of potentially repetitive documents, focusing instead on validating the drafts against the rationales to determine the most accurate answer. We utilize the strong language modeling capabilities of generalist LMs, calculating the conditional generation probability of the answer drafts and rationales confidence score. Our key contributions are:
We introduce novel RAG framework that employs smaller specialist RAG drafter to generate high-quality draft answers. Each draft is derived from distinct subset of retrieved documents, offering diverse perspectives while reducing input token counts per draft.
The generalist LM, operating with the RAG drafter, requires no additional tuning. It simply verifies and integrates the most promising draft into the final answer. This approach enhances
2

---

Published as a conference paper at ICLR 2025
comprehension of each subset and mitigates potential te--intth-middlee (Liu et al., 2024) phe- nomenon.
Our method significantly accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a singic, unbiased verification pass over the drafts in parallel. Extensive experiments on free-form question-answering and closed-set generation benchmarks demonstrate the superior effectiveness and efficiency of the method.
2 RELATED WORKS
Retrieval Augmented Generation Retrieval Augmented Generation (RAG) enhances LLMs by retrieving relevant documents from external databases and incorporating them into the generation process (Gao et al., 2023b; Lewis al., 2020; Khandelwal et al., 2020; Izacard & Grave, 2021; Luo et al., 2023a; Xia et al., 2024b; Wang et al., 2024). Recent work has primarily focused on enabling LLMs to understand when and what to retrieve (Ma et al., 2023; Chen et al., 2023b; Jiang et al., 2023b; Schick et al., 2024), or designing approaches to better utilize contexts (Yu et al., 2023; Yoran et al., 2023; Wang et al., 2023b; Sarthi et al., 2024; Back et al., 2023; Xu et al., 2023; Kim et al., 2024). Among them, SAIL (Luo et al., 2023a) fine-tunes pre-trained LLM web search data to filter irrelevant contents. Self-Reflective RAG (Asai et al., 2023) introduces reflection tokens to guide retrieval and annotation in instruction-tune tunning datasets. However, both approaches require additional Iestruction-tuiigg of generic LLMs, which is resource-intensive and may lead to forgetting over-fitting (Luo al., 2023b). Furthermore, long context with retrieved documents can suffer from computational inefficiency and position bias (Liu et al., 2024). Corrective RAG (Yan al., 2024) on the other hand proposes lightweight retrieval evaluator. but lacks the capability for high-level reasoning. In contrast, our proposed SPECULATIVE RAG addresses these limitations by leveraging smaller RAG drafter model to efficiently understand diverse perspectives in retrieval results and generate drafts for the generalist LMs to verify and integrate.
Speculative Decoding Speculative decoding (Stern et al., 2018; Xia et al., 2023; Chen et al., 2023a; Leviathan et al., 2023; Xia et al., 2024a) aims to reduce auto-regressive decoding latency through a draft-then-veriff paradigm. This involves drafting multiple future tokens with a small model and verifying them in parallel with the target model (Xia et al., 2024a). The draft model is typically either an independent model from the same series (Leviathan et al., 2023; Chen et al., 2023a) the target model itself (Zhang al., 2023a; Cai et al., 2024). Our approach extends this concept from token-level drafting to answer-level drafting. In contrast to traditional verification criteria (Stern al., 2018; Xia al., 2023; Leviathan al., 2023; Chen et al., 2023a; Miao al., 2024), which accept or reject tokens based on their generation probabilities, we leverage language modeling objectives directly assess the confidence of entire answer drafts.
SPECULATIVE RETRIEVAL AUGMENTED GENERATION THROUGH DRAFTING
Problem Formulation In knowledge intensive tasks, each entry can be represented as (Q, D, A), where Q is a question or statement that requires additional knowledge; D (di,...,dd, d is a set of n documents retrieved from the database; A is the expected answer. Particularly, in question answering tasks, Q and A are the question and the expected answer in natural language form; in the statement verification tasks, Q is statement and A E [True, False) is Boolean value indicating the statement's correctness; in the multiple choice tasks, Q is question with few options and A E (A, B, ...} } is the index of the correct answer. The objective a RAG system is to generate fluent response containing the expected answer select the expected answer from the provided options based on the context provided by the retrieved supporting documents.
3.1 OVERVIEW
We introduce Speculative Retrieval Augmented Generation (SPECULATIVE RAG), as illustrated in Figure 1. We aim at enhancing the reasoning ability of LLMs over retrieved documents without compromising processing speed. Instead of relying on brute-force parameter scaling instruction- tuning an entire LM to handle knowledge-intene proach. We utilize smaller specialist LM,, the RAG drafter, to rapidly generate multiple answer
tasks, we propose a divide-and-conen conquer ap-
3

---

Published as a conference paper at ICLR 2025
drafts based retrieved results. Then, larger generalist LM, the RAG verifier, assesses these drafts, selects the best one based on its rationale, and integrates into the generation results.
Algorithm 1: SPECULATIVE RAG
Data: (Q, D ) is the question and retrieved documents; m subsets, each containing it
documents, are sampled from D; also corresonnds to the number of clusters during clustering.
Result: A is the predicted answer to the question.
Function Speculative RAG (Q, D. m, k):
{ci, (2,,C2.... Ck A
K-Menns C(d1, ..., dn IQ)
p- Cluster the documents into groups using an embodding model
3
4.
repeat
8, <
D Construct subset the retrieved documents 6j
for c. (e,.... Ck do
7
8, 8, U (random. samplecc))
P Sample one document from each cluster es into subset 85-
8
end
A = A U }
10
until A = m
b. Ropeat the sampling until there are m unique subsets total.
11
for
8, E A do in parallel
> Process m subscts in parallel.
aj. Bg + Manite generate(
Generate the draft and rationale / with Maants Compute the confidence p with Myailia.
P3 M score(o,lQ.
14
end
15
Å + arg maxa, P)
D Select the one with the highost score as the final answer.
16 return
Specifically, as shown in Algorithm 1, we first cluster the retrieved documents with regard to their relation the posed question, where each cluster represents one perspective the retrieval results (Line 2). Then we sample one document from each cluster into a subset so the documents in this subset covers the multiple perspectives the retrieval results. We aim at minimizing redundancy and increase the diversity of the documents (Line 5 to 8). We denote one subset as ō C D that contains retrieved documents with diverse contents and multiple perspectives the retrieval results. Then, distribute each subset ô to RAG drafter endpoint MDrater with the posed question Q generate the answer draft a and the rationale 3 in parallel (Line 12). The RAG drafter is instruction tuned to be specialist in understanding the retrieved documents and produce rationales that are faithful to the input documents. It is smaller than generalist LMs, and its parallel processing further ensures high efficiency. For each draft-rationale pair (a, 33 from MDrateer- we compute confidence score with the generalist LM MVeriner based on the question Q and corresponding rationale 3 (Line 13). It is worth mentioning that MVerifier does not need to be instruction-tuned since we leverage its language modeling ability already learned during pre- pre-training. Meanwhile, M can verify the drafts based on the informative rationale provided by MDraterr instead of processing tedious or possibly redundant retrieved documents. Finally, we select the answer draft with the highest confidence score as the final answer and integrate it into the generation results of the generalist LM (Line 15).
3.2 SPECIALIST RAG DRAFTER
Instead of tuning a large generalist LM for the RAG scenario, we leverage a smaller specialist LM, MD to understand retrieved documents. MDrafeer is specialized in answering the given question based on the supporting documents and not expected to cope with general problems. It serves as RAG module for the generalist LMS when solving Knowledge-intensive tasks. We train Mieater to generate both the answer draft and the rationale to better understand the contextual documents.
Instruction Tuning Given a triplet (Q, A, D), where Q is a general query, A is the response, and D is a retrieved supporting document, we augment it with the rationale of the response A based on the document D. We denote the rationale as F. which extracts essential information from the document and explains why the response is reasonable to the query concisely (Hsieh et al., 2023) so it is of shorter length and delivers information coherent with the original document. We leverage relatively strong LMs to automatically synthesize the rationale E for each triplet. Specifically, we directly query the strong LM to understand the knowledge from the document and provide the intermediate rationale between the instruction and response. Refer to Appendix G for detailed prompts. After generating the rationale, we finetune a pre- LM using the standard language modeling objec- tive, maximizing the likelihood: [(Q.A,D,EE) log Mo (A, F. Q, D), where (Q, A, D, F)) is an
4

---

Published as a conference paper at ICLR 2025
augmented entry in the dataset; PMMona (A, Q. D) is the probability of generating the response and rationale based on the query and document. We use this instruction tuned model as the specialist RAG drafter which learns generate well-grounded response and rationale given the query and relevant documents.
Multi- PentiiPPesspective Sampling For each Bhowledgg-intenseee question, retrieve set of docu- ments from the database using the posed question as the retrieval query. These documents may contain diverse content due to the ambiguity inherent in the query. To minimize redundancy and enhance diversity of the document subsets used for generating answer drafts, we employ a multi- perspective sampling strategy. We first cluster the documents into few topics using an instruction- aware embedding model (Peng et al., 2024) and the K-Means clustering (Jin & Han, 2011).
emb (1) ), ... emb (d,,) = E(.,, Q,
{cl, C+ = K K- (emb(d() emb emb(d.))
5 raanomm... sample(c) for c E {c]ff
where & is instruction-aware embedding model which embeds string with regard provided instruction (the posed question Q): emb(d,) is the embedding for the retrieved document dæ сy is cluster of retrieved documents with similar topics and contents; is a hyper-parameter that controls the number of clusters. We sample one document from each cluster into a document subset ò so each subset contains i documents of diverse contents. In total, we construct m subsets for parallel inference with the RAG drafter.
RAG Drafting We run Mpmttr over the m document subsets and produce corresponding answer drafts. Refer to Appendix H for detailed prompt. We incorporate each document subset into the prompt and query MDrater for responses. We obtain m drafts as the answer candidates and each draft is grounded based on the multiple perspectives in the retrieval results. Specifically, given a document subset ò, = {dj...., .., dj. we query MDrafier in parallel with the following prompt for the answer draft and rationale: Q, dj.;...... .... dj. D.j: Sy, where the prompt contains the posed question Q along with the document subset; the generation result contains the answer draft a and the rationale B. We denote the conditional generation probability Prratt P(3,IQ, dj, ..... dj. + P(az IQ, dji:-. dij:-..,dj die (3), which measures the reliability of generating rationales and the confidence in producing answer drafts.
3.3 GENERALIST RAG VERIFIER
After generating drafts and the rationale from the RAG drafter MDmafeers we evaluate them by generalist LM Myerifier filter out the less reliable drafts and select the best answer. The generalist LM can be any off-the-shelf pre-trained LM. We only consider the draft-rationale pair (a, B) and skip the tedious and redundant retrieval results. We resort the language modeling ability of the generalist LM to rank and select the draft-rationale pairs.
Evaluation Scores First, we calculate the self-consistency score by determining the conditional probability of generating a drast-rationale pair given the question, PSelf-contain P(a. ,IQ)). This score helps assess whether the draft and rationale are self-consistent in the context of the question. Given the characteristics of language modeling, a self-consistent draft-rationale pair is expected to yield a higher probability. Furthermore, we incorporate a self-reflection statement R that prompts MVeritier assess the reliability of answer draft (c.g. "Do you think the rationale supports the answer, yes or no?"). We define the self-reflection score as PSelf-reftstt = P("Yes" " [Q, a, B, R) where we compute the conditional probability of the positive answer ("Yes") to the self-reflection statement.
Computation Method We can efficiently compute the self-consistency and self-reflection scores within one forward pass of Merifer- Given a question Q and draft-rationale pair (), we construct a prompt [Q, o., B, R. "Yes"), where Rt is the self-reflection statement. We encode the prompt with M and acquire the probability of each token conditioned on the previous tokens P(t,)tz)). We leverage this auto-regressive feature and aggregate the probability of the relevant tokens to compute the self-consistent score PSeli-comain and self-refiection score Pellet
PSC
PSR
|sc II P(li(ee)) II.,s P(()((e))
Q, R, "Yes" +
|SR IY.e" P(taltee)
5

---

Published as a conference paper at ICLR 2025
Finally, we produce the final score, P) PDraat, Psc.j PsR.j, and then select the most reliable answer as the final answer to the question Á = arg maXa, Pj-
4 EXPERIMENTS
We evaluate our proposed SPECULATIVE RAG on five public retrieval augmented generation bench- marks: TriviaQA (unfiltered) (Joshi et al., 2017), MuSiQue (Trivedi et al., 2022), PopQA (Mallen al., 2023), PubHealth (Zhang et al., 2023b), and ARC-Challenge (Clark et al., 2018). We provide representative examples for case study in Appendix J. TriviaQA, MuSiQue, PopQA are challenging open-domain question answering datasets where RAG systems are required to answer questions on factual knowledge. TriviaQA and PopQA typically require one accurate piece of evidence from the documents, whereas MuSiQue demands multiple documents to construct a multi-hop reasoning chain. More detailed experiments on multi-hop reasoning can be found in Appendix F. Following previous works (Guu et al., 2020; Asai al., 2023; Yan et al., 2024), we evaluate performance the free-form generation based on whether gold answers are contained within the generated response or not. PubHealth and ARC-Challenge are closed-set generation datasets. PubHealth is a dataset of medical claims spanning variety of biomedical subjects and it requires the RAG system to verify a given claim based on the retrieved documents. ARC-Challenge introduces a multi-choice ques- tion answering dataset, composed of science exam questions from grade 3 to grade 9. For closed-set generation tasks, we use accuracy to evaluate whether the generated answers match the ground truth.
4.1 BASELINES
Standard RAG For standard RAG, incorporate all the retrieved documents into the prompt as contextual information. Refer to Appendix for detailed prompts. We run standard RAG experiments on off-the-shelf LLMs including Mistral7n, -istral-nnstrucyys (Jiang et al., 2023a), Mixtralg.7n, Mixtral-Instructssn (Jiang et al., 2024), and Alpacars (Dubois et al., 2024). We also include the performance of Toolformer (Schick et al., 2024) and SAIL (Luo al., 2023a) which to use tools
originally reported from Asai et al. (2023). Toolformeryg is an LM instruction-tune including a search engine, and SAIL/B is an LM instruction-tuned on the Alpaca instruction tuning set augmented with search results from different sources such as DuckDuckGo and Wikipedia.
Seff-Reflective RAG and Corrective RAG Self-Reflective RAG (Self-RAG) (Asai et al., 2023) and Corrective RAG (CRAG) (Yan et al., 2024) are more advanced RAG systems that enhances the quality of contextual information in the retrieval results. CRAG introduces external evaluator to assess the quality of retrieved documents, and to refine them before the response generation. Self- RAG instruction dynamically retrieve documents when necessary, critique the retrieved documents relevance before generating responses. Self-CRAG is apply the Self-RAG approach the refined documents of CRAG. We adopt the same backbone LLMs across all methods as our proposed SPECULATIVE RAG for fair comparisons.
an LM to generate special self-refection tags. These tags guides the LM to
4.2 EXPERIMENT SETTINGS
In our experiments, we utilize Mistraly (v0.1) as our base LM for the RAG drafter. For RAG verifier, we employ either Mistralı (v0.1) or Mixtrals.78 (v0.1) without any fine-tuning, de- noted as MMeniner-BB or M We pre-compute embeddings of retrieved documents using lightweight instruction-awart embedding model InBedderRobena (Peng et al., 2024) as part of the retrieval process. Inference is conducted using the vLLM framework (Kwon et al., 2023) with greedy decoding (temperature = 0). We adopt the same experiment settings from Asai et al. (2023) and include a more challenging benchmark, MuSiQue (Trivedi et al., 2022). Our focus is on RAG reasoning rather than evidence citation, so we omit the other two long-form generation benchmarks, Biography (Min et al., 2023) and ALCE-ASQA (Gao et al., 2023a). On TriviaQA, PopQA, Pub- Health, and ARC- CRe-Challenge, we retrieve top 10 documents and generate 5 drafts per query (m = 5), with each draft based subset of 2 documents (k 2). For MuSiQue, ments and generate 10 drafts for each query (m = 10), each using a subset of 6 documents due to more complex reasoning. Further details regarding instruction-tuning can be found in Appendix A.
retrieve top 15 docu-

---

Published as a conference paper at ICLR 2025
Table 1: Retrieval augmentation generation results TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge (ARC-C). (*We use the RAG drafter's generation generaats probability PDraft as the con- fidence score for selecting drafts when we use alone; indicates numbers reported in Asai et al. (2023); denotes numbers that are not reported by the original papers or are not applicable; 'we use Mistral or Mixtralg.78 as the RAG verifier, and denote them as M or M
Free-form
Closed-set
RAG Method
TriviaQA MuSiQue PopQA
PublIealth ARC-C
Standard RAG
Mistrabs (Jiang et al., 2023a)
54.15
16.71
31.38
34.85
42.75
Mixtralac7 (Jiang al., 2024)
59.85
19.16
34.02
37.08
48.72
Mistral- Mistrral-nstrructn (Jiang al., 2023a)
67.11
17.99
42.17
42.15
47.70
Mixtral- Saatral-nsttructs.r (Jiang al., 2024)
73.91
29.42
$3.68
63.63
78.41
Alpacar (Dubois et al., 2024)
64.1
46.7
40.2
48.1
Toofformeras (Schick al., 2024>
48.8
SAIL (Luo el al., 2023a)
69.2
48.4
Self-Re Reflectee RAG & Corrective RAG
CRAG (Yan al., 2024)
59,03
49.46
59.04
74.87
Self-RAGMinnn AG (Asai al., 2023)
64.84
21.72
$2.68
72.44
74.91
AC (Yan al., 2024)
65.43
56.11
72.85
75.26
Our Speculative RAG
MDafan-B
71.11
27.89
56.40
75.58
74.49
Mventer TB
Manatts 7B
73.91
31.03
56.75
75.79
76.19
74.24
31.57
57.54
76.60
80.55
4.3 MAIN RESULTS
We compare SPECULATIVE RAG with standard RAG approaches. as well as the more advanced Self-Reflective RAG and Corrective RAG on five datasets: TriviaQA, MuSiQue, PopQA, Pub- Health, and ARC-Challenge. We report the performance of M 78 when used alone or paired with the RAG verifier (e.g. Mveniier: 7B, (er1 Following prior work (Asai al., 2023; Yan et al., 2024), we report accuracy as the performance metric.
Superior Performance over Baselines Table demonstrates that our method consistently out- performs all baselines across all five benchmarks. Particularly, M Meritirr-8877 + +MDrrater77 surpasses the most competitive standard RAG model, Mixtral-Instrruteern. by 0.33% on TriviaQA, 2.15% on MuSiQue, 3.86% PopQA, 12.97% PubHealth, and 2. 14% on ARC-Challenge. With comparable number of Instruction-tunar parameters, Mveritier-7 + MDratter-7 outperforms all Self- Reflective and Corrective RAG methods, and MDrafeer alone surpasses the basclines in most settings.
Effective Instruction Tuning for RAG Drafter Our instruction tuning is effective in enhancing observe remarkable perfor-
the reasoning ability of the drafter model (Hsich et al., 2023). mance improvement comparing Mistralyg and MDeateer-B- Additionally, we further investigate the performance of MD when directly feed all documents to the RAG drafter and generate one draft, with detailed results provided in Appendix B. Moreover, the performance significantly improves when paired with the instruction-tuned RAG drafter MDrafter-7B, showing gains of 14.39% on TriviaQA, 12.41% on MuSiQue, 23.52% on PopQA, 39.52% on PubHealth, and 31.83% on ARC-Challenge. Similar improvements are observed with Mistralz well. For Mistral7,, we observed improvements of 19.76% on TriviaQA, 14.32% on MuSiQue, 25.37% on PopQA, 40.94% on PubHealth, and 33.44% on ARC-Challenge. We attribute these improvements to the superior reasoning capabilities of the RAG drafter over the retrieved documents in SPECULA- TIVE RAG. By minimizing the redundancy the sampled documents, the RAG drafter generates higher quality answer drafts based on diverse perspectives from the retrieval results.
Mixtralgan
Reliable Scoring by RAG Verifier The reliable draft verification by the generalist LM also con- tributes to the enhanced performance. The performance improves remarkably comparing MD and Mveniier-- Mprater-7B The instruction-tunal RAG drafter is specialized in generating answer drafts based on the retrieved documents while the language modeling capabilities of generic LMs are leveraged to validate each draft in light its rationale. This method both effective and easy to implement, showcasing the effectiveness of this verification approach.

---

Published as a conference paper at ICLR 2025
4.4 EFFECTS OF GENERATED RATIONALE FOR VERIFICATION
In SPECULATIVE RAG, we utilize the generated rationale 3 from the RAG drafter as an indicator of the trustworthiness of answer drafts a.
Shortened length compared to the retrieved documents. The rationales highlight relevant points, omit redundant information, and bridge logical gaps between drafts and their supporting documents. We compare the number of tokens in the generated rationale and the retrieved docu- ments, and plot them in Figure 2. We find that the generated rationale is significantly shorter than the retrieved documents.
Retrieved Documents
Generated Rationale
Avg. Token Number
800
600
517-87
400
357.04
272.11
268.52
200
41.42
44.47
34.07
34.94
ThivaQA
MUSI Que
Pop OA Dataset
PubHealtth
ARC-Challenge
Figure 2: Average number of tokens in the generated rationale and the retrieved documents in Triv- iaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge. The generated rationale is of much shorter length than the original retrieved documents.
Table 2: Performance and latency analysis of SPECULATIVE RAG TriviaQA and PubHealth using Mventter + MD We add the original document subset ō to the context or re- place the generated rationale 3 with the original retrieved document subset 5 during verifica- tion, i.e. we compute the self-containment score as PSelf-contain = P(a, 8lQ) or PSetf--ontain P(a, 8, BIQ), and compute the self-reflection score PSelf-reftect PSelf-reflect = PP""es"|[Q, a,, 3, R), where Q is the query; is the answer draft; R is the self- reflection statement.
Pe"Yes" Q2,a, Rt) or
TriviaQA
PubHeallh
Accuracy (%)
Latency (S)
Accuracy (%)
Latency (S)
Mummass + Ma
p = Score(a] 8)
74.24
76.60
P Score(a Q
74.08 (0.16)
2.13 (x10.30%)
76.09 (-0.5))
1.31 (611.9756)
Score(a
74.32 (+0.08)
2.17 (+12.445)
76.29 (.331)
1.33 (+13.68%))
Comparable performance with retrieved documents and lower latency. To evaluate the ef- fectiveness of the rationales, we create two alternative scoring methods: (a) replacing rationale with retrieved documents (p- Score(alQ, 8)), or (b) adding retrieved documents rationale (p Score(al 3, 6)). We compare these alternatives to the scoring method used in SPECULA- TIVE RAG (p = Score(al B)) in Table 2. The results show that incorporating longer retrieved documents does not consistently improve performance and tends to increase latency. This suggest that the generated rationale already of high quality and serves as an effective bridge between the supporting documents and the generated answer drafts. By leveraging this rationale, can effi- ciently verify drafts using a generic LM,, leading to accurate final results. We further validate the rationale generation in the instruction-tuning stage. See Appendix D for more details.
4.5 LATENCY ANALYSIS WITH BASELINES
We analyze the latency of Standard RAG, Self-RAG, and our SPECULATIVE RAG on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge. We randomly sample 100 cases from each dataset and report the average time cost for each case, as shown in Figure 3. To simulate real-world application scenarios, we process cases individually without batching. As representative example, we run M 8x7B + .DDrater:77 for SPECULATIVE RAG and Mixtral-Instrrugg..n for Standard RAG, as these demonstrate the highest performance among competitive baselines (see Table 1). We also include the analysis for Standard RAG: Mistral- and Self-RAG: Mistral-Instruutra in this study. For SPECULATIVE RAG, we launch 5 endpoints MDeater- for parallel drafting on TriviaQA, PopQA. PubHealth, and ARC-Challenge. We launch 10 endpoints for MuSiQue due to more drafts. We use tensor parallelism of 4 to fit Instruct-n into the GPU memory. We use the same tensor parallelism setting for the other methods for fair comparison.
8

---

Published as a conference paper at ICLR 2025
Std. RAG: Mixtral inst, Bx78
Std. RAG: Mistral inst, 78
SeffRAG: Nistra inst. 78
Spec. RAG, Hixtrol 8x78 w/ RAG Drafter (78)
1.34% 2.90
Latency
+3.4%
2.24
+1%%
2.22
+18% +7%
2.47
2.31
+80%
1.85%
+611% 2:28
+103%
2.40-74%
2:05
+1.6% 1.37
+2954-441226% 2.02 1.47
1.42
0
MusiQue
PopOA
PubHealth
ARC-Challenge
Dataset
Figure 3: Latency analysis of Standard RAG, Self-RAG, and SPECULATIVE RAG on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge. The latency difference between Standard RAG/Self-RAG and SPECULATIVE RAG is highlighted in red (+r%). The latency varies across different datasets due to different retrieved document lengths. SPECULATIVE RAG encodes the retrieved documents parallel and generates answer drafts with a smaller RAG drafter. This signif- icantly improves the efficiency.
Reducing processing time while maintaining high performance As the results demonstrate, SPECULATIVE RAG consistently achieves the lowest latency compared to all other methods. This advantage comes from its utilization of fewer documents needed per draft and parallel drafting. Par- ticularly, compared with the most competitive baseline, Standard RAG: MM + +Deateer-B our proposed SPECULATIVE RAG reduces latency by up to 11.90% on TriviaQA, 15.07% on MuSiQue, 44.31% % on PopQA, 50.83% on PubHealth, and 22.77% on ARC-Challenge. Further- more, direct comparison between Standard RAG: Wistral-Instruchhy and our method reveals that the higher latency of Standard RAG: Mistral-Instruce is due to its longer context length which con- tains all retrieved documents. Self-RAG: Histral-Instruce also exhibits higher latency due to the generation of longer answers with self-reflection tags and the additional overhead associated with evidence selection. These findings highlight the advantage of our approach in reducing processing time while maintaining high performance.
4.6 ABLATION STUDIES
We conduct ablation studies on the multi-perspeetio sampling (Section 3.2) and the evaluation scores (Section 3.3) SPECULATIVE RAG during the drafting or the verification stages on Trivi- aQA and PubHealth in Table 3. We use M 8x78 .MDmatee:1 as a running configuration. Same as the main results, we report the accuracy as performance metries.
Diversity and reduced redundancy in retrieval improves draft quality significantly. In the first set of experiments, we evaluate the impact of multi-perspective sampling during the drafting- Recall that SPECULATIVE RAG clusters retrieved documents into distinct perspectives and sample one document from each cluster to reduce redundancy for the draft generation. We compare this against two alternative sampling strategies:
(1) Random sampling without multi-perspective clus- tering, where we randomly select a document subset as context, and (2) Always sampling from the cluster, where select all documents from single cluster. Our results indicate that our pro- posed sampling method yields the best performance thanks to its ability to leverage diverse context. Particularly, it improves the accuracy up to 1.88% on TriviaQA and 2.23% on PubHealth. While random sampling without clustering introduces diversity, it is prone to including redundant docu- ments, degrading draft quality. Sampling from the same cluster significantly underperforms due to a lack of diverse perspectives.
Scoring method on self-consistency and self-reflection refines draft quality effectively. In the second set of experiments, we examine the scoring method during verification. We remove each of the specific confidence scores, PDraft contain or reflec in turn. Performance drops are ob- served when any score is removed. Particularly, removing PDealt leads to a minimal decline, 0.19% on TriviaQA and 1.12% on PubHealth, likely due to the limited verification capability of the smaller RAG drafter. Removing either PSelf-contain PSelf-refteer results in similar performance decreases, around 2.0% on TriviaQA and around 0.8% on PubHealth, indicating that both celf-containment and self-reflection capture different key aspects of reasoning and crucial during verification. Ran- dom selection without verification leads to substantial underperformance, resulting in a performance decline of 5.69% TriviaQA and 5.37% PubHealth.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study of SPECULATIVE RAG in the drafting and verification stages TriviaQA and PubHealth.
TriviaQA
PubHealth
My + MD
74.24
76.60
Drafting Stage
Random sampling w/ multi-pen pulti-pprspeective clustering
73.02 (-1.22)
75.38 (-1.22
Always sampling from the same perspective cluster
72.36 (-1.88)
74.37 (-2.3)
Verification Stage
w/o Pomn (p = PSelf-cumain PSetf-reflect)
74.05 (...1)
75.48 (-1.1)
w/o PSelf-oontain (p = PDra (Setf-eoflct)
72.04 (-2.00)
75.89 (.0.71)
w/o Psett-ecttett (p = PDratt PSett-commia)
72.36 (-1.8)
75.68 (-0,92)
Random selection w/o verification
68.55 (-569)
71.23 (-5.37)
76
73.9
74.0
24.4
74.7
27.0
76
75.5
75.8
75.7
75 7466
76.4
Accuracy
76
75.0
76.4
Accuracy
74
74:0
23.2
70
74
70
68.0
75.0
71.1
67.6
68
5
10
15
20
1
5
10
15
20
1 2 5 9100 TrviaQA: subset
1 5 8 9 RubHealth: Dac
TrMaQA: Drafts
PubHealth: Drafts
(a) We include 5, 10, 15, 20 drafts and sample supporting documents for each draft.
(b) We sample 1, 2, 4, 6, 10 supporting documents for each draft and we generate 10 answer drafts.
Figure 4: Performance analysis of SPECULATIVE RAG with (a) different numbers of drafts, and (b) different supporting document subset size on TriviaQA and PubHealth.
4.7 EFFECTS OF DRAFT NUMBER AND DOCUMENT SUBSET SIZE
Increasing the number drafts improves performance without adding latency. We investi- gate the performance of SPECULATIVE RAG under varying numbers of drafts. Using M MDattr-7B with 1, 5, 10, 15, 20 drafts on TriviaQA and PubHealth. We sample two documents as context per draft. The results are illustrated in Figure 4(a). Since we retrieve top 10 documents in total, we sample up to 20 drafts in these experiments. The results indicate that incorporating more drafts can further improve performance, likely thanks higher coverage of diverse perspective of documents. Importantly, in SPECULATIVE RAG, we can launch multiple RAG drafter instances to generate drafts parallel without additional latency.
Increasing the document subset size doesn't always lead to better performance. We also ex- amine the effect of document subset size. By varying the number of documents (1, 2, 4, 6, 10) sampled for draft generation on TriviaQA and PubHealth (Figure 4(b)), we find that including more documents in the context does not always lead consistent performance improvement. While Triv- iaQA queries may benefit from more supporting documents due to their complexity, MVeanerr-BB + MD can surpass Mistral-Instructy, even with a single supporting document per draft. Fur- thermore, with two or more documents per draft, MMveniier:n + MDrater can even surpass Mixtral- Instructs.7B- This further demonstrates the effectiveness of our drafting design.
5 CONCLUSION
Our proposed SPECULATIVE RAG decomposes RAG tasks into two separate steps of drafting fol- lowed by verification. SPECULATIVE RAG delegates the heavy lifting of drafting to a small spe- cialized RAG drafter, while verification is done using large generalist L.M. The parallel generation of multiple drafts from diverse document subsets provides high quality answer candidates while sirerio--iiii-v---oooo------
reducing input token counts and the potential risk substantial improvements in both the quality and speed of the final output generation. We demon- strate the effectiveness SPECULATIVE RAG with accuracy gains up to 12.97% while reducing latency by 50.83% compared conventional RAG systems. SPECULATIVE RAG sheds new light on the potential collaborative architectures for enhancing RAG performance through task decom- position.
resulting in
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11552 2023.
Jinheon Baek, Soyeong Jeong, Minki Kang, Jong C Park, and Sung Hwang. hnwwledge-augmentee language model verification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1720-1736, 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri
Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Caan-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013//. 2023a.
Tong Chen, Hongwei Wang. Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.066-53 2023b.
Wei Chen, Yu Liu, Weiping Wang, Erwin M Bakker, Theodoros Georgiou, Paul Fieguth, Li Liu, and Michael S Lew. Deep learning for instance retrieval: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803..0800.000 05457, 2018
Jiyy Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang. Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.00 02.60.2666, 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling Iims' knowledge gaps with plug-in specialized language models. In The Twelfth International Conference on Learning Representations, 2023.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models generate text with citations. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 6465-6488, 2023a.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. detrieval-augmennt generation for large language models: A survey. arXiv preprint arXiv:2312. 10997, 2023b.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938 PMLR, 2020.
Cheng-Ya Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Rat- ner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In ACI, 2023.
11

---

Published as a conference paper at ICLR 2025
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on ballucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXhv:2311.05222, 2023.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguis- tics. doi: 10. 10.18653//1/2021.coo eacl-mmai..4m444 eacl-ma manmmai.... 74.
URL https: hatp///aalaaani ha//aaaaaaattoooooooooo or org/2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:21 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024.
Mixtral of experts. arXiv preprint arXiv:2401.0400
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- Singapore, December 2023b. Associa-
ods in Natural Language Processing, pp. tion for Computational Linguistics. doi: 10.18653/v1/2033 http://e.222........mm 495. URL https: //ac //aclanthology org/ 2022.223.. emn Lp- main. 495.
Li Jiapeng, Liu Runze, Li Yabo, Zhou Tong. Li Mingling, and Chen Xiang. Tree of reviews: tree- based dynamic iterative retrieval framework for multi-hop question answering. arXiv preprint arXiv:2404.14 144644 2024.
Xin Jin and Jiawei Han. K-means clustering. Encyclopedia of machine learning, pp. 563-564, 2011.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. Evaluating open-domain ques- tion answering the era large language models. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5591-5606, 2023.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through Memorization; Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR), 2020.
Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang- Woo Lee, Minjoon Seo, Jung-Wor Ha, and Jinwoo Shin. Sure: Summarizing retrievals using answer candidates for open-domain qa of llms. arXiv preprint arXiv:2404.13081, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
12

---

Published as a conference paper at ICLR 2025
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, Pp. 19274-19288 PMLR, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel. et al. Retrieval-augment genera- tion for knowledge-inteneeeee intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning. arXiv preprint arXiv:2404.02000 02060, 2024.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. arXiv preprint MrXiv:2305.1522. 2023a.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308. 08747, 2023b.
Xinbei Ma. Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303-5315, 2023.
Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zeuleeoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient Ilm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404. 08801, 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non -ormmarmettiic memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, 2023.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large lan- guage model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 932-949, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381 2391-2391, 2018.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannanch Hajishirzi. Factscore: Fine- mene-grained atomic evaluation of factual pre- cision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100, 2023.
Letian Peng, Yuwei Zhang, Zilong Wang. Jayanth Srinivasa, Gaowen Liu, Zihan Wang, and Jingbo Shang. Answer is all you need: costruction-oolowiinn text embedding via answering the question. arXiv preprint 2:Xxv:2402.09644 2024.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowl- edge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 2523-2544 2021.
13

---

Published as a conference paper at ICLR 2025
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506 2020.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0555 05533330 2024.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Man- ning. Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprini arXiv:2401.18059, 2024.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zeutlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. Advances in Neural Information Processing Systems, 36, 2024.
Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. Generate-then-groundd in retrieval-augmenteed generation for multi-hop question answering. arXiv preprint arXiv:2406. 14891, 2024.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. Asqa: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 8273 8288, 2022.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances Neural Information Processing Systems, 31, 2018.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.1/800 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint 2xXiv:2488.00. 18, 2024.
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics,
2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go. Exploring the state of instruction tuning on open resources, 2023a.
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to 202b.
filter context for retreeeal-augmentedd generation. arXiv preprint TrXiv:2311.08377 2023b.
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts clicit context-aware reasoning in long-horizon generation. arXiv preprint arXiv:2403.053 2024.
Thomas Wolf, Lysandre Debut. Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-an natural language processing. arXiv preprint arXiv: 19Xiv:/910.03771, 2019.
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decod- ing: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909-3925, 2023.
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yonggi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.0785. 2024a.
14

---

Published as a conference paper at ICLR 2025
Sirui Xia, Xintao Wang, Jiaqing Liang. Yifei Zhang, Weikang Zhou, Jiaji Deng, Fei Yu, and Yanghua Xiao. Ground every sentence: Improving retrieval- -uggeeene Ilms with interleaved reference- claim generation. arXiv preprint 240.0:240,.01796 2024b.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving surieval-augmented Ims with compression and selective augmentation. arXiv preprini arXiv:2310.04408 2023.
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and ZZhen-Hua Ling. Corrective retrieval augmented generation.
arXiv preprint arXiv:2401. 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
2018
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making runmened language models robust to irrelevant context. arXiv preprint arXiv:2310.01558, 2023.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210. 2023.
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding, arXiv preprint arXiv:2309.08/ 2023a.
Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Lue Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng. and James Glass. Interpretable unified language checking. arXiv preprint arXiv:230..03726 2023b.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A INSTRUCTION-TUNEE SETTINGS
We construct our training dataset for the RAG drafter from diverse instruction-follovii pairs. We sample instances from Open-Instruct processed data (Wang al., 2023a) and knowledge-intensive datasets (Petroni et al., 2021; Stelmakh et al., 2022; Mihaylov et al., 2018). We augment the instruction- toollowion-ollooinngg pairs with retrieved documents and generated rationale. We use the off-the- shelf dense retriever Contriever-MS MARCO (Izacard et al., 2021) to retrieve up to 10 documents for each pair and use Gemini-Ulte (Team et al., 2023) to generate rationale. In total, we acquire dataset of 40k instances. We use Mistral (v0.1) as our base LM for the RAG drafter. We reproduce the performance of Self-RAG (Asai et al., 2023) and CRAG (Yan et al., 2024) with Mistral (v0.1) for fair comparison. We implement the training scripts using the Transformers library from Hug- ging Face (Wolf et al., 2019). We employ DeepSpeed (Rasley et al., 2020) to accelerate the training process. All experiments are conducted on a Linux server equipped with 16 Nvidia A100-SXM4- 40GB GPUs.
Additionally, we replace Gemini-Ultra (Team et al., 2023) with GPT-4o (Achiam al., 2023) when curating the instruction-tunin data for our RAG drafter to investigate the effects of different LLMs. These results demonstrate that SPECULATIVE RAG maintains its performance advantage, even when trained on data curated by GPT-4o. It consistently outperforms the baselines from Standard RAG, SelfRAG, and CRAG, further validating the effectiveness of our approach.
Table 4: RAG results on TriviaQA, PubHealth, ARC-Challenge with the RAG drafter trained on instruction-tuning data curated by GPT-4o and Gemini-Ulte
Trivial QA
PubHealth
ARC-C
Myeritaa-TB + (iemii-l1
73.91
75.79
76.19
M Mveritaa-o77 Murnie-1
74.24
76.60
80.55
Mveriter TD M
72.24
73.05
76.54
M 8BB GPT-40
73.58
73.35
80.63
Moreover, we use an instruction-tuned Gemma-22s (Team et al., 2024) as the RAG drafter and the frozen Mistraly or Mixtralay78 the RAG verifier. We report the performance analysis in Table 5. These results suggest that Gemma-22n provides a promising avenue for future work of further optimization.
Table 5: RAG results on TriviaQA, PubHealth, ARC-Challenge with an instruction-tur -tnned Gemma- 2nn or Mistral as the RAG drafter and Mixtrals.78 the RAG verifier.
Trivial QA
PubHeallt
ARC-C
M MMerifieSccBB MDrafeer228
67.22
72.14 76.60
67.92
M MD
74.24
80.55
B EFFECTS OF INSTRUCTION TUNING
In SPECULATIVE RAG, we introduce a framework that combines the RAG drafter and the verifier. In this ablation study, we directly feed all documents to the RAG drafter and generate one draft (m = 1, &o = total # of documents). As shown in Table 6, we observe that instruction tuning effectively enhances the document understanding capability of the RAG drafter, it outperforms both Mistralm and Mistral- Instruct- However, there remains gap compared to SPECULATIVE R RAG, showing the effectiveness of the drafting and verification framework.
16

---

Published as a conference paper at ICLR 2025
Table 6; RAG results on TriviaQA and PubHealth (m 1, total # of docs)
TrivialQA
PubHealth
Mistraby
54.15
34.85
Mistral Instrut-nnstructm
67.11
42.15
My (m = 1, 10)
73.23
65.25
My + MDnater.7B (m72 5, k: 2)
73.91
75.79
C EFFECTS OF SELF-REFLECTION STATEMENT
We use "Do you think the explanation supports the answers? (Yes No)" as the self-reflection statement in our main results. In this study, we replace it with other alternatives to see how the self-reflection statement affects the accuracy. The results reported in Table 7, We observe that the performance does not change a lot given different self-reflection statements, which shows the stable verification capability of the generalist LMs by language modeling objective.
Table 7: Performance analysis of SPECULATIVE RAG with different self-reflection statements R when computing the self-reflection score (Self-reflect P("Yes" Q,, a, 8, R),, where is the query, a, 33 are the generated answer draft and rationale.
Reflection Statement
TriviaQA
PubHealth
Do you think the explanation supports the answers? (Yes No)
74.24
76.60
Does the rationale support the answer? (Yes or No) What do you think about the rationale? A good one? (Yes or No)
74.22
76.09
74.25
75.79
Is the rationale good enough to support the answer? (Yes or No)
74.39
76.29
D EFFECTS OF RATIONALE GENERATION
We acknowledge that the generation of rationale potentially increases the inference cost during the drafting stage while this is crucial for the verifier in our method assess the quality and reliability of generated drafts. And, the potential overhead can be mitigated through efficient parallel inference.
To further study the impact of rationale generation, finetune the RAG drafter without rationale. We denote this setting as: without rationale drafting. Similarly, with rationale/doc in verification indicates that we the generated rationale the retrieved documents reference during the verification stage. We use M + MDrater-7B as a running example. The results are shown in Table 8.
Table 8: Ablation study on the draft generation in the drafting and verification stages on TriviaQA and PubHealth.
Drafting
Verification
TrivialQA
PubHealth
ARC-C
w/o rationale
w/ doc
70.86
75.28
78.67
rationale
w/ doe w/ rationale
74.08
76.09
80.46
rationale
74.24
76.60
80.55
Better answer drafting As explored in Hsieh et al. (2023), incorporating rationale generation during instruction-tuning can lead to the RAG drafter producing higher-quality answer drafts. The results in Table clearly demonstrate this. We observe a significant performance drop across all three benchmarks when the RAG drafter is finetuned without the rationale component,
Lower latency and cost in verification We verify each draft against the rationale instead of the retrieved documents. From the ablation results, these generated rationales serve as high-quality grounding facts, improving verification performance compared using the retrieved documents.
17

---

Published as a conference paper at ICLR 2025
E EFFECTS OF DIFFERENT VOLUME OF TRAINING DATA
We acknowledge the importance of evaluating our framework's performance across different train- ing data volumes. Our primary experiment utilized 40,059 instances to train drafter model. To thoroughly assess scaling effects, we conducted additional experiments using incremental subsets of 10,000, 20,000, and 30,000 training instances. The results of these systematic evaluations are detailed Table 9.
Table 9: Performance analysis of SPECULATIVE RAG with different volumn of instruction-tuning data.
M Myeiterar7
MaatersB
TriviaQA
PubHeallh
10,000
71.69
72.34
20,000
72.64
72.44
30,000
73.20
74.37
Total (40,059)
74.24
76.60
From the table, we can conclude that increasing the volume of training data leads to improved per- formance. Specifically, the model's accuracy continues rise instances are included, with the highest performance observed at 40,059 instances. This suggests that larger training datasets contribute positively the performance of our drafter-verifier framework, indicating that scaling up data size could enhance the robustness of the model.
F EFFICACY OF SPECULATIVE RAG IN MULTI-HOP REASONING
We further validate SPECULATIVE RAG the scenario of multi-hop reasoning. One of the key challenges of multi-hop reasoning is to effectively combine multiple pieces of evidence to arrive at final answer. Indeed, ability to verify or contrast information across documents crucial to solve complex questions. We compare the performance of SPECULATIVE RAG with baselines on MuSiQue (Trivedi et al., 2022) and HotpotQA (Yang et al., 2018), two multi-hop reasoning benchmarks. We randomly sample 500 examples from the validation set of HotpotQA as the test set in our experiment. We adopt the same setting as MuSiQue on HotpotQA. The results are in Table 10. We find that our SPECULATIVE RAG achieves the best performance. Specifically, SPECULATIVE RAG improves accuracy by 2. 251 on MuSiQue and by a substantial 5.4% on HotpoiQA.
Table 10: RAG results MuSiQue and HotpotQA
MuSiQue
HotpotQA
Mixtral-IInttrutsssre MiI-RAGGiitta--B8
29.42
43.60
21.72
27.20
M Verifier-7B + MDmater-7B
31.03
47.60
MVerifie-88778 + MDrateer-78
31.57
49.00
Our approach tackles this challenge by multi multi-perspective sampling when selecting documents for each draft (Section 3.2). We cluster the retrieved documents into distinct topics using an instruction- aware embedding model (Peng et al., 2024). Then, we sample one document from each cluster to form diverse document subset, ensuring each drafter receives a variety of perspectives from the retrieval results. To validate the efficacy of this strategy, we further conduct an ablation study on MuSiQue and HotpotQA in Table 11. From the table, sampling strategy effectively guarantees the diversity of information within the supporting document subsets, leading to improved perfor- mance of SPECULATIVE RAG these tasks.
F.1 PERFORMANCE BREAKDOWN ON HOTPOTQA
HotpotQA includes two types of quesitions: bridge-type questions HotpotQA require a two-step reasoning process where the answer to the first step is crucial for answering the second. For example:
18

---

Published as a conference paper at ICLR 2025
Ablation study of multi-perspective sampling multi-hop reasoning benchmarks: MuSiQue, HotpotQA.
Table 11:
MuSiQue
HotpotQA
Random sampling
29.33
48.2
Multi-perspecae sampling
31.57
49.00
"When was the singer and songwriter of Radiohead born?"
Step Who the singer and songwriter of Radiohead? -> Thom Yorke
-Step 2: When was [Thom Yorke] step 1) born? - October 7, 1968
Final answer: October 7, 1968
In contrast, comparison questions also involve two steps, but the answers to each step are independent of each other. For example:
"Who was born first, Morgan Llywelyn or Robert Jordan?"
Step 1: What's Morgan Llywelyn's 's DOB? ~> December 3, 1937
Step 2:
What's Robert Jordan's 's DOB? - October 17, 1948
Final answer:
Morgan Llywelyn
Table 12: Performance of SPECULATIVE RAG for different question types
Question Type
# of Questions SPECULATIVE RAG
Bridge-type Comparison-type
400
41.75
100
78.00
Overall
500
49.00
We report the performance breakdown of SPECULATIVE RAG on HotpotQA in Table 12. The results demonstrate superior performance on comparison-type questions with multi-perspective sampling. This aligns with our expectations, as culti-persppectvee sampling ensures the document subset covers the diverse topics necessary for answering comparison-type questions. Revisiting the example above, "Who was born first, Morgan Llywelyn or Robert Jordan?", with = 4, our approach clusters retrieved documents into four groups. Group 0 and 3 focus Morgan, while group I and focus on Robert. As we sample one document from each group for the drafters, this clustering result ensures each drafter receives documents about both individuals. This balanced information distribution is crucial for the comparison-type questions. In contrast, random sampling risks providing drafter with information about only one person, yielding a suboptimal draft.
On the other hand, we also observe that the perspipperpeectvee sampling is less helpful for bridge-type questions. These questions require the LLM to first identify the "bridge entity" (e.g., Thom Yorke in the earlier example), a task our current work isn' in''t explicitly designed for. While multi- perspective sampling effectively covers different topics in the drafts and the map-reduce approach accelerates inference, they might not directly contribute to pinpointing the "bridge entity" -the key to answering bridge-type questions.
We believe our framework could be effectively combined with other techniques specifically designed for bridge-type questions, such as those proposed Xia et al. (2024b); Jiapeng et al. (2024); Shi et al. (2024). For instance, the Tree-of-Reviews (ToR) framework, introduced in Jiapeng et al. (2024), addresses multi-hop reasoning problems by dynamically initiating new searches based on previously retrieved documents and constructing various reasoning paths. This dynamic searching strategy can be integrated into our SPECULATIVE RAG, enabling each drafter to answer bridge-type questions more effectively.
19

---

Published as a conference paper at ICLR 2025
G PROMPT OF RATIONALE GENERATION
Prompt
Memorize this piece of evidence in mind and use it as if you already know it.
Evidence: State religion
Despite enjoying considerable popular support, Christianity was still not the official religion Rome, although it in neighboring states such Armenia, Iberia, and Aksum. Roman Religion (Neoplatonic Hellenism) was restored for time by the Emperor Julian from 361 to 363. Julian does not appear have reinstated the persecutions earlier Roman emperors. Catholic Christianity, deemed heretical, was declared to be the state religion of the Roman Empire on 27 February 380 the decree "De fide catolica".
the opposed Arlanism and other 1deologies
Instruction: when did christianity become official religion of Rone?
Response: 380 AD
Rationale: (complete the following sentence with details from the evidence; you can only the information from the evidence)
Completion
Catholic Christianity was declared the official state religion of the Roman Empire on February 27th, 388, through the decree "De fide catolica".
Figure 5: Prompt of Rationale Generation for Gemini-Ultra
H
PROMPT OF RAG DRAFTING
Prompt
Response the instruction. Also provide rationale for your response.
## Instruction: In Buddhism, what is the state of blissful repose or absolute existence by someone relieved of the necessity of rebirth?
## Evidence:
[1] Buddhism
Nirvana Literally
"blowing quenching, becoming extinguished". In early Buddhist texts, it is the state of restraint and self-control that leads to the "blowing out" and the ending of the cycles of sufferings associated with rebirths Buddhist texts describe nirvana as identical with "anatta" with complete "emptiness, described
redeaths. Many later
nothingness".
texts, the through the gate of emptiness ("sunyata") realizing that there" [2] Salwation
greater detail, such passing
It includes variety of disciplines such yoga and meditation. Nirvana the profound peace of mind that is acquired with moksha (liberation). . In Buddhism and Jainism, it is the being free from suffering. Supreme Being). The word literally means "blown out" (as in candle) and refers, in the Buddhist context,
Hindu philosophy,
with the Brahman
blowing of the fires of desire, aversion, delusion, the imperturbable stillness of mind acquired thereafter. In Theravada Buddhiss the emphasis on one's
Completion censenencnnceeen
literally means 'blowing out, quenching, becoming extinguished'. is described as a state of "restraint and self-contra that leads to the "blowing out" and ending cycles sufferings associated ## Response: In Buddhism, the state of blissful repose absolute existence by someone relieved of the necessity rebirth is called Nirvana.
Rationale:
rebirths
Figure 6: Prompt of RAG Drafting
20

---

Published as a conference paper at ICLR 2025
1 PROMPT OF STANDARD RAG
Prompt
Below is an instruction that describes task. Write a response that appropriately completes the request.
Evidence:
[1] Britain (place name)
Britain, after which "Britain" becane the more commonplace name the island called Great Britain. After the Anglo-Saxon period, "Britain" was used as historical term only. Geoffrey of Monmouth his pseudohtstorical "Historia Regum Br tanntane""
[2] Great Britain
islands of "Prettanike" Pri Priteni" is the source of the Welsh language term Prydain, "Britain", which has the same source as the Goidelic term Cruithne used to refer to the early Brythonic-speaking inhab tants of Ireland. The latter were later called Picts or Caledonians
peoples
called the "Priteni" or "Pretani",
Albion
Albion is an alternative name far Great Britain. The oldest attestation of the topanym comes from the Greek language. It is sometines used poetically and generally to refer the island, but is less common than 'Britain' today. The name far Scotland in most of the Celtic languages is related to Albion: "Alba" in Scottish Gaelic, "Albain"
### Instruction: What was Britain called before it was Britain?
### Response:
Figure 7: Prompt of Standard RAG for Non-instruction-tuned LM
Prompt
(INST) Below is an instruction that describes task. Write response for and state your explanation supporting your response.
Instruction: What Britain called before Britain?
Aни Evidence:
[1] Britain (place name)
Britain, after which "Britain" became the more commonplace name for the island called Great Britain. After the Anglo-Saxon per period, "Britain" was used as a historical term only. Britannize"
Geoffrey of Monmouth in his psoudohistorical "Historia
[2] Britain
The peoples of these islands of "Prettanike" were called the "Priteni" or "Pretani". language Prydain, "Britain",
Priteni" is the source the Goidelic term Cruithin used to refer to the early Brythonic-speaking inhabitants of Ireland. The latter were later called Picts or Caledonians
the same
[10] Albion
Albion alternative name Great Britain. The oldest attestation the toponym comes from the Greek language. It is sometimes used poetically and generally to refer to the island, but is less common than 'Britain' today. The name for Scotland most of the Celtic languages related to Albion: "Alba" Scottish Gaelic, "Albain" 
 The response is:
Figure Prompt of Standard RAG for Instruction-tune LM
21

---

Published as a conference paper at ICLR 2025
J CASE STUDY
Figure 9 and 10 are two representative cases from TriviaQA and PubHealth. They show the two drafts generated for the same question. We observe that our RAG drafter well understand the multiple perspectives in the retrieval results and generate high-quality drafts. Our RAG verifier can also help filter out the unreliable drafts as we observe a relatively low scores in the first draft in Figure and 10.
Prompt of Drafter
Response to the instruction. Also provide rationale for your response.
#* Instruction: Which actress/singer starred as Doralee Rhodes in the 1988 film, "Nine to Five"?
## Evidence:
Diana DeGarmo
Sheila until the show's closure on June 27, 2010. DeGarmo played the role of Doralee Rhodes of "9 to 5", which launched in Nashville September 21, She ended her run as Doralee after the July 2011 Minneapolis tour stop. DeGarmo appeared on musical theatre composer Scott Alan's third album,
the national
[2] to (musical)
Pippa Winslow as Violet Newstead, Louise Olley as Doralee Rhodes and Leo Sene as Franklin Matthew Chase and Blair Anderson. "g to 5" will play in the West End at the Savoy Theatre from January 29 production
Hart Jr, with Samantha Giffard
and Mark Houston, Rachel Ivy,
August 31,
(Judy),
Completion
## Rationale: Diana DeGarmo played the role of Doralee Rhodes in the national tour of "9 5", which began in September 2018.
## Response: ana DeGarmo
Verification
draft score: 6594
self-contaimment score: 0.3417
self-reflection score: 0.5238
Prompt
Response to the instruction. Also provide rationale for your response.
## Instruction: Which actress/singer starred Doralee Rhodes in the 1988 film, "Nine
Five"?
Evidence:
[1] Cynthia Rhodes
non-dance related thriller "Runaway" with Tom Selleck, Kirstie Alley and Gene Simmons. Her most notable role was as dance instructor Penny Johnson in the hit 1987 notion picture "Dirty Dancing" with Jennifer Grey and Patrick Swayze. Rhodes' final motion picture role the character of Vickie Phillips opposite actor Jameson Parker in the sleeper action-adventure movie Dolly Parton
Rhodes garnered
Crichton's
fiction
Parton's feature film "9 to 5" (1980). The musical ran at the Ahmanson Theatre, Los Angeles late 2008. opened on Broadway at the Marquis Theatre New York City, on April 30, 2009, to mixed reviews. The title track of her 2698 "Backwoods Barbie" album was written for the musical's character Doralee. Although her score (as well as the musical debut Allison Janney)
praised,
Completion ## Rationale: Dolly Parton starred in the 1989 film "g to 5" as Doralee Rhodes. Additionally, Parton's 2098 musical "9 to 5" was based on the film, with Parton reprising her role as Doralee, A# Response: Delly Parton starred as Doralee Rhodes the 1980 film, "Nine to Five".
Verification
draft score: 8.71
self-containment score: 0.4346 self-reflection score: 7449
Figure 9: Case study of SPECULATIVE RAG from TriviaQA where Dolly Parton is the correct answer.
22

---

Published as a conference paper at ICLR 2025
Prompt of Drafter
Response the instruction. Also provide rationale for your response. ## Instruction: A lot of the prablems with forest fires is because of bad policy (not to was put place by environnentalists.
clear out forests)
IN Evidence:
[1] Forest restoration
That be because of provoked deforestation order substitute forests crop areas, or in dry areas, because of wild fires occurring naturally or intentionally, A whole many
section of forest landscape cases, the net lass of ecosystem value is very high and can open the drop to an accelerated further degradation
in linked to this particular problem,
the soit conditions through erosion and desertification. indeed has dire consequences both the quality of the habitats and their related fauna.
[2] Wildfire
Attempts to thin out the fuel load may be met with opposition due to the desirability of forested areas. Wildland goals may further resisted because endangered species protections and habitat preservation. The ecological benefit of fire is often overridden by the economic benefits protecting structures that cover wildland areas usually differ from local and state policies that govern urban lands.
lives. Additionally, federal policies
Completion a*naamanasaaa.
Rationale: Inadequate management can contribute severity of wildfires. ## Response: True
Verification
draft score: 8161 self-containment 1360 self-reflection score: 0.8877
Prompt of Drafter
Response the instruction. provide rationale for your response. #t Instruction: A lot of the prablems with forest fires is because of bad policy (not to clear out forests)
was put in place by environnentalists.
## Evidence:
[1] 1997 Indonesian forest fires
The 1997 Indonesian farest fires
caused by changing land which made the tropical forest vulnerable to fire during a drought associated with that year's El Nino, Indonesian distorically been resistant burning even during long dry seasons despite the use of fire to clear land for swidden agriculture. The land use changes that industrial-seate logging, draining peatlands
fires were combination conversion to oil palm and fast-growing tree plantations, and massive government program to drain swamps and convert them to rice paddies. [2] Wildfire
Attempts to thin out the fuel load may be met with opposition due to the desirability of forested areas. Wildland goals may further resisted
endangered species protections and habitat preservation. The ecological benefit of fire is often overridden by the economic benefits protecting structures that cover wildland areas usually differ from local and state policies that govern urban lands.
lives. Additionally, federal policies
Completion
were caused by including industrial-scale logging, draining peatlands.
Rationale:
changes**,
## Response: False
Verification
draft score: 8.8298
self-contsinment score: 0.1325 self-reflection 0.9533
Figure 10: Case study of SPECULATIVE RAG from PubHealth where False is the correct answer.
23


Reviews:
[{"review_id": "9H94D6eaQI", "reviewer": "Reviewer_GVM9", "summary": "The paper proposed a retrieval-augmented generation framework termed SpeculativeRAG, leveraging high-level concepts analogical to speculative decoding.\nThe framework exhibits better performance with lower latency via lunching a set of smaller model instances in parallel, each processes a subset of retrieved documents, to produce answer drafts and rationales.\nThe answer drafts and rationales are subsequently verified by a larger, strong base LLM to select the final answer.", "questions": "- For the comparison of the average number of tokens in the generated rationale and the retrieved documents in Figure ```2```: What is the number of retrieved documents? Is it before or after subsetting to drafters?\n\n- Is the Multi-Perspective Sampling stage also considered in the latency analysis in Section ```4.5```?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The conceptual extension of the speculative farmwork proposed by the author is both novel and appealing, and supported by strong empirical results.\n\n- The paper targets a timely challenge in RAG and offers promising techniques to improve the efficiency for the system.\n\n- The experiments are comprehensive and thorough, in particular, the extensive ablation study and analysis provide great insights for a better understanding of the proposed approach.", "weaknesses": "- The instruction-tuning of the small drafter LMs requires synthesis rationales generated by Gemini Ultra from an additional 40k instances. This presents an unfair comparison for baselines methods that do not have access to these external resources in the experiments.\n\n- It is unclear whether multiple instances of large verifier LLMs are also lunched in parallel. According to Line ```11~14``` in Algorithm ```1```, this seems to be the case. If so, the large memory overhead might significantly offset the latency gain in practical scenarios.\n\n- A relevant work [1] (first released ~4.5 month ago) is missing in the baseline and not mentioned in the related work Section. To the best of my understanding, [1] also proposes a very similar synthesis rationale-augmented generation approach for RAG, thus, might be a suitable method for baseline comparison.\n\n- The contribution and result of the paper could be further strengthened if newer generation of models (e.g., gemma-2, llama-3) are adopted in the experiments.\n\n[1] Wei et al, *InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales*. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MfcVuLJk1o", "reviewer": "Reviewer_RYjP", "summary": "This work introduces a new speculative decoding method to enhance the retrieval augmented generation. Specifically, a smaller distilled specialist LM is used to generate answer drafts based on randomly selected document subsets. After that, a larger generalist LM is used to verify the answer confidences according to the answer and rationale generation probabilities.", "questions": "1. In Line 207-209, is the answer A obtained by human labeling or generated by LLMs given documents D? What if the D does not provide evidence for generating A?\n2. Except for the inference latency, what are the consuming token numbers of different methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. The authors present detailed descriptions of their methods, including prompts and experimental setups.\n2. The proposed method achieves obvious improvements among the baselines, and the ablation studies can verify the effectiveness of their method.", "weaknesses": "1. The conducted experiments should include more recent RAG baselines, especially the speculative decoding methods (e.g., [1], [2]). \n2. It is a trivial idea and makes limited contributions compared to previous studies. As mentioned in this paper, many recent studies investigate the two-stage RAG, where a drafting stage produces answer candidates and the assembling stage generates final answers based on the drafts. There are only a few differences between them.\n\n\n[1] Ground every sentence: Improving retrieval-augmented llms with interleaved reference claim generation.\n\n[2] RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTeTFNxCtr", "reviewer": "Reviewer_niGd", "summary": "This paper introduces Speculative RAG – a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft.", "questions": "Since this is a paper target for the RAG framework, testing the proposed framework on some RAG benchmarks may be more reliable. Because the QA benchmark is mainly designed for reasoning, not for the RAG framework.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper trained an LLM (Drafter) to generate multiple answers paired with rationale (reference + explains). Then, they use another LLM (Verifier)to confirm the answers, which enhances the robustness.", "weaknesses": "The main contribution of this paper is insufficient. Because the whole framework just involves clustering the documents and fine-tuning small LLM to generate answers with rationale. Then prompt Verifier model confirms the answer based on rationale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CprqfsJWfS", "reviewer": "Reviewer_dkh1", "summary": "This paper leverages the idea of speculative decoding to design a speculative RAG framework, which offloads computational burden to a smaller, specialist LM that serves as the RAG drafter for generalist LMs. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.", "questions": "1. While the author has shown that the proposed method is effective, it is important to note that the generalist LM has not undergone fine-tuning. RAG serves to supply additional information to the LLM to mitigate knowledge gaps. However, if the generalist LM itself lacks relevant knowledge, can it reliably evaluate the rationale and drafts generated by the specialist LM, particularly when the specialist’s output is inaccurate or includes hallucinations? To better understand this limitation, could the author provide boundary cases or conduct an error analysis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The designed specialist LM is lightweight and allows to analyze multiple documents and produce RAG drafts in parallel, which effectively reduces the computation cost and increases the inference efficiency.\n2. The distinction between specialist and generalist LMs clarifies their functional roles, reducing the risk of generalization degradation that may result from supervised fine-tuning (SFT) of a generalist LM. Specialists, optimized through targeted fine-tuning, focus on improving draft generation capabilities, thereby enhancing result accuracy.\n3. Applying a clustering algorithm, i.e. K-means, to pre-group retrieved documents and then uniformly sampling from each group, i.e., multi-perspective sampling, can mitigate analysis inaccuracies caused by incomplete information, enhancing robustness in practical applications.", "weaknesses": "1. The quality of the SFT data constructed by the author remains unclear due to insufficient analysis. Additionally, further ablation studies are needed to evaluate the drafter’s performance across varying data volumes. Acquiring substantial SFT data through a proprietary model is resource-intensive. Therefore, it is essential to investigate whether a smaller dataset could yield satisfactory results or if a large volume of SFT data is indeed required to achieve optimal outcomes.\n2. P(Yes) or SR is commonly applied in hallucination detection. However, [1] highlights that LLMs without target SFT or aligned, while LLMs may differentiate between correct and incorrect results, the probability itself, i.e., P(Yes), is either uncalibrated or demonstrates poor calibration performance. Although Table 3 shows that incorporating SR enhances overall performance, there is insufficient evidence to validate the intrinsic effectiveness of P(Yes). This limitation weakens the robustness of the proposed method. Could the author also add an ablation study to independently assess the effectiveness of P(Yes)?\n3. It seems that results for key baseline models, such as CRAG and Self-CRAG, are missing for the three free-form datasets. This omission is concerning, as it is standard research practice to replicate baseline results when they are not available from prior studies, especially when the authors are comparing their approach with only a few previous works. Could the authors also include the performance metrics of CRAG and Self-CRAG on the free-form datasets for a more comprehensive comparison?\n4. According to the results in Table 1, Mixtral-Instruct-8x7B has already achieved high scores on TriviaQA and ARC-C (73.91% and 78.41%), and the improvement brought by Speculative RAG is limited (74.24% and 80.55%). These results may diminish the contribution of the proposed method, as the performance improvement brought by instruction tuning is more pronounced and stable (Mistral-7B vs. Mistral-Instruct-7B and Mixtral-8x7B vs. Mixtral-Instruct-8x7B). Could the authors provide more explanation and analysis to clarify this result?\n5. Although the datasets used in this paper include a variety of question-answering formats, such as free-form (short-form and multi-hop) and closed-set, they are all based on wiki-style or specific domains. To more convincingly demonstrate the method's effectiveness, the authors should further extend their evaluation to more realistic and domain-diverse RAG datasets, like FreshQA and BRIGHT. Could the author also discuss the potential challenges in applying their method to these datasets and propose specific experiments to show the performance?\n\n[1] Kadavath et al., Language Models (Mostly) Know What They Know, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)