Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting

Authors: Zilong Wang, Zifeng Wang, Long Le, Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen-Yu Lee, Tomas Pfister
Primary area: generative models
Keywords: generative model, retrieval augmented generation

Abstract:
Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce Speculative RAG - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that Speculative RAG achieves state-of-the-art performance with reduced latency on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 50.83% compared to conventional RAG systems on PubHealth.

Decision: accept (poster)
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730271860570

thread_type:
official_review

<thread_object>
creator: reviewer_dkh1

<content>
summary:
This paper leverages the idea of speculative decoding to design a speculative RAG framework, which offloads computational burden to a smaller, specialist LM that serves as the RAG drafter for generalist LMs. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.

strengths:
1. The designed specialist LM is lightweight and allows to analyze multiple documents and produce RAG drafts in parallel, which effectively reduces the computation cost and increases the inference efficiency.
2. The distinction between specialist and generalist LMs clarifies their functional roles, reducing the risk of generalization degradation that may result from supervised fine-tuning (SFT) of a generalist LM. Specialists, optimized through targeted fine-tuning, focus on improving draft generation capabilities, thereby enhancing result accuracy.
3. Applying a clustering algorithm, i.e. K-means, to pre-group retrieved documents and then uniformly sampling from each group, i.e., multi-perspective sampling, can mitigate analysis inaccuracies caused by incomplete information, enhancing robustness in practical applications.

weaknesses:
1. The quality of the SFT data constructed by the author remains unclear due to insufficient analysis. Additionally, further ablation studies are needed to evaluate the drafter’s performance across varying data volumes. Acquiring substantial SFT data through a proprietary model is resource-intensive. Therefore, it is essential to investigate whether a smaller dataset could yield satisfactory results or if a large volume of SFT data is indeed required to achieve optimal outcomes.
2. P(Yes) or SR is commonly applied in hallucination detection. However, [1] highlights that LLMs without target SFT or aligned, while LLMs may differentiate between correct and incorrect results, the probability itself, i.e., P(Yes), is either uncalibrated or demonstrates poor calibration performance. Although Table 3 shows that incorporating SR enhances overall performance, there is insufficient evidence to validate the intrinsic effectiveness of P(Yes). This limitation weakens the robustness of the proposed method. Could the author also add an ablation study to independently assess the effectiveness of P(Yes)?
3. It seems that results for key baseline models, such as CRAG and Self-CRAG, are missing for the three free-form datasets. This omission is concerning, as it is standard research practice to replicate baseline results when they are not available from prior studies, especially when the authors are comparing their approach with only a few previous works. Could the authors also include the performance metrics of CRAG and Self-CRAG on the free-form datasets for a more comprehensive comparison?
4. According to the results in Table 1, Mixtral-Instruct-8x7B has already achieved high scores on TriviaQA and ARC-C (73.91% and 78.41%), and the improvement brought by Speculative RAG is limited (74.24% and 80.55%). These results may diminish the contribution of the proposed method, as the performance improvement brought by instruction tuning is more pronounced and stable (Mistral-7B vs. Mistral-Instruct-7B and Mixtral-8x7B vs. Mixtral-Instruct-8x7B). Could the authors provide more explanation and analysis to clarify this result?
5. Although the datasets used in this paper include a variety of question-answering formats, such as free-form (short-form and multi-hop) and closed-set, they are all based on wiki-style or specific domains. To more convincingly demonstrate the method's effectiveness, the authors should further extend their evaluation to more realistic and domain-diverse RAG datasets, like FreshQA and BRIGHT. Could the author also discuss the potential challenges in applying their method to these datasets and propose specific experiments to show the performance?

[1] Kadavath et al., Language Models (Mostly) Know What They Know, 2022.

questions:
1. While the author has shown that the proposed method is effective, it is important to note that the generalist LM has not undergone fine-tuning. RAG serves to supply additional information to the LLM to mitigate knowledge gaps. However, if the generalist LM itself lacks relevant knowledge, can it reliably evaluate the rationale and drafts generated by the specialist LM, particularly when the specialist’s output is inaccurate or includes hallucinations? To better understand this limitation, could the author provide boundary cases or conduct an error analysis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730449578334

thread_type:
official_review

<thread_object>
creator: reviewer_nigd

<content>
summary:
This paper introduces Speculative RAG – a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft.

strengths:
This paper trained an LLM (Drafter) to generate multiple answers paired with rationale (reference + explains). Then, they use another LLM (Verifier)to confirm the answers, which enhances the robustness.

weaknesses:
The main contribution of this paper is insufficient. Because the whole framework just involves clustering the documents and fine-tuning small LLM to generate answers with rationale. Then prompt Verifier model confirms the answer based on rationale.

questions:
Since this is a paper target for the RAG framework, testing the proposed framework on some RAG benchmarks may be more reliable. Because the QA benchmark is mainly designed for reasoning, not for the RAG framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730738099908

thread_type:
official_review

<thread_object>
creator: reviewer_ryjp

<content>
summary:
This work introduces a new speculative decoding method to enhance the retrieval augmented generation. Specifically, a smaller distilled specialist LM is used to generate answer drafts based on randomly selected document subsets. After that, a larger generalist LM is used to verify the answer confidences according to the answer and rationale generation probabilities.

strengths:
1. The paper is well-written and easy to follow. The authors present detailed descriptions of their methods, including prompts and experimental setups.
2. The proposed method achieves obvious improvements among the baselines, and the ablation studies can verify the effectiveness of their method.

weaknesses:
1. The conducted experiments should include more recent RAG baselines, especially the speculative decoding methods (e.g., [1], [2]). 
2. It is a trivial idea and makes limited contributions compared to previous studies. As mentioned in this paper, many recent studies investigate the two-stage RAG, where a drafting stage produces answer candidates and the assembling stage generates final answers based on the drafts. There are only a few differences between them.


[1] Ground every sentence: Improving retrieval-augmented llms with interleaved reference claim generation.

[2] RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation.

questions:
1. In Line 207-209, is the answer A obtained by human labeling or generated by LLMs given documents D? What if the D does not provide evidence for generating A?
2. Except for the inference latency, what are the consuming token numbers of different methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730795276748

thread_type:
official_review

<thread_object>
creator: reviewer_gvm9

<content>
summary:
The paper proposed a retrieval-augmented generation framework termed SpeculativeRAG, leveraging high-level concepts analogical to speculative decoding.
The framework exhibits better performance with lower latency via lunching a set of smaller model instances in parallel, each processes a subset of retrieved documents, to produce answer drafts and rationales.
The answer drafts and rationales are subsequently verified by a larger, strong base LLM to select the final answer.

strengths:
- The conceptual extension of the speculative farmwork proposed by the author is both novel and appealing, and supported by strong empirical results.

- The paper targets a timely challenge in RAG and offers promising techniques to improve the efficiency for the system.

- The experiments are comprehensive and thorough, in particular, the extensive ablation study and analysis provide great insights for a better understanding of the proposed approach.

weaknesses:
- The instruction-tuning of the small drafter LMs requires synthesis rationales generated by Gemini Ultra from an additional 40k instances. This presents an unfair comparison for baselines methods that do not have access to these external resources in the experiments.

- It is unclear whether multiple instances of large verifier LLMs are also lunched in parallel. According to Line ```11~14``` in Algorithm ```1```, this seems to be the case. If so, the large memory overhead might significantly offset the latency gain in practical scenarios.

- A relevant work [1] (first released ~4.5 month ago) is missing in the baseline and not mentioned in the related work Section. To the best of my understanding, [1] also proposes a very similar synthesis rationale-augmented generation approach for RAG, thus, might be a suitable method for baseline comparison.

- The contribution and result of the paper could be further strengthened if newer generation of models (e.g., gemma-2, llama-3) are adopted in the experiments.

[1] Wei et al, *InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales*. 2024.

questions:
- For the comparison of the average number of tokens in the generated rationale and the retrieved documents in Figure ```2```: What is the number of retrieved documents? Is it before or after subsetting to drafters?

- Is the Multi-Perspective Sampling stage also considered in the latency analysis in Section ```4.5```?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed Speculative RAG, a framework that decomposes retrieval-augmented generation into two stages. First, a specialist draft model produces multiple candidate responses from distinct document subsets. Then, a larger generalist LM verifies these drafts to select a final answer. The authors claim this two-step process (1) improves latency by offloading most of the generation onto a smaller model, and (2) increases accuracy by ensembling generations on diverse retrieval subsets and reducing position bias in lengthy contexts. Experimental evaluations on multiple QA benchmarks show accuracy gains and latency reduction compared to baseline RAG methods. 

Strengths
1. The idea paper of adapting speculative decoding into a RAG setting on different retrieval subsets is reasonable.
2. The proposed method achieves improvement over baseline RAG methods on multiple QA benchmarks, with noticable reductions in inference time.
3. The authors conduct thorough ablations to isolate the contributions of different components

Weaknesses
1. Multiple reviewers noted the lack of direct baselines against newly proposed speculative decoding or multi-stage RAG approaches
2. A stronger examination of more modern LLMs could strengthen the paper’s contributions.
3. Certain reviewers view the work as a relatively modest extension of prior multi-stage RAG methods.

The paper presents a novel and efficient approach to RAG, with sufficient empirical results and detailed ablations to support its claims. While some questions about novelty and evaluation were proposed by the reviewers, I believe the authors addressed most points during the rebuttal. The novelty is still a major concern, but considering its simplicity and efficacy, I recommend a weak acceptance of this paper.