RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
Authors: Guanzheng Chen
Qilong Feng
Jinjie Ni
Xin Li
Michael Qizhe Shieh
Venue: ICML 2025 spotlightposter (2025)
Keywords: Speculative Decoding
Long-Context LLM
RAG
Primary area: deep_learning->large_language_models

Abstract: The emergence of long-context large language models (LLMs) offers a promising alternative to traditional retrieval-augmented generation (RAG) for processing extensive documents. However, the computational overhead of long-context inference presents significant efficiency challenges. While Speculative Decoding (SD) traditionally accelerates inference using smaller draft models, its effectiveness diminishes substantially in long-context scenarios due to memory-bound KV cache operations. We introduce Retrieval-Augmented Speculative Decoding (RAPID), which leverages RAG for both accelerating and enhancing generation quality in long-context inference. RAPID introduces the RAG drafter—a draft LLM operating on shortened retrieval contexts—to speculate on the generation of long-context target LLMs. Our approach enables a new paradigm where same-scale or even larger LLMs can serve as RAG drafters while maintaining computational efficiency. To fully leverage the potentially superior capabilities from stronger RAG drafters, we develop an inference-time knowledge transfer that enriches the target distribution by RAG. Extensive experiments on the LLaMA-3.1 and Qwen2.5 backbones demonstrate that RAPID effectively integrates the strengths of both RAG and long-context LLMs, achieving significant performance improvements (e.g., from 39.33 to 42.83 on InfiniteBench for LLaMA-3.1-8B) with more than 2$\times$ speedups for long-context inference. Our analyses also reveal the robustness of RAPID across various context lengths and retrieval quality.

RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
Guanzheng Chen 123
Qilong Feng"
Jinjie Ni
Xin Li Michael Qizhe Shieh
Code: bttps://githubbee http://tttibb.coom//soutttraraaa TRATT/RAPP
Abstract
1. Introduction
The emergence of long-context large language models (LLMs) offers promising alternative to traditional
Large language models (LLMs) have traditionally relied on retrieval-augmented generation (RAG) to process ex- tensive documents by selectively retrieving relevant text segments. While effective, the performance of RAG is in- herently bounded by the capability of the retriever to extract pertinent information across diverse queries (Gao et al., 2023). The recent emergence of long-context LLMs, capa- ble of directly processing million-wrrd documents (Team et al., 2024), suggests a promising alternative to complex RAG pipelines. However, this breakthrough bottlenecked by the computational efficiency of long-context inference. where processing extensive key-value (KV) caches becomes memory-bound and introduces substantial latency (Pope et al., 2022).
generation (RAG) for processing extensive documents. However, the computational overhead of long-context inference presents significant efficiency challenges. While Speculative Decoding (SD) traditionally acceler- ates inference using smaller draft models, its effec- tiveness diminishes substantially in long-context scenarios due to memory- bound KV cache op- erations. We introduce Aurrivval-Augmnneed SPeculatIve Decoding (RAPID), which leverages RAG for both accelerating and enhancing gener- ation quality in long-context inference. RAPID introduces the RAG drafter-a draft LLM oper- ating on shortened retrieval contexts-to spec- ulate on the generation of long-context target LLMs. Our approach enables a new paradigm where same-scale or even larger LLMs can serve as RAG drafters while maintaining computational efficiency. To fully leverage the potentially supe- rior capabilities from stronger RAG drafters, we develop an enference-time knowledge transfer that enriches the target distribution by RAG. Exten- sive experiments on the LLaMA-3.1 and Qwen2.5 backbones demonstrate that RAPID effectively integrates the strengths of both RAG and long- context L.LMs, achieving significant performance improvements (e.g., from 39.33 42.83 In- finiteBench for LLaMA-3. 3.1-BB with more than 2x speedups for long-context inference. Our anal- yses also reveal the robustness of RAPID across various context lengths and retrieval quality.
Speculative Decoding (SD) (Chen et al., 2023; Leviathan et al., 2023) a prevalent approach to accelerate L.LM in- ference without compromising generation quality. By lever- aging smaller draft model propose multiple candidates for single-pass -pgss validation by the target model, SD achieves significant speedup when candidates are accepted. The ben- efits of SD hinge on two critical factors: the computational efficiency of the draft model in generating candidates, as well as its capability to produce high-quality and acceptable candidates. However, SD will become less effective in long- context scenarios, as memory-bound KV cache operations prevent smaller LLMs from maintaining significant speed benefits over larger models (Pope et al., 2022; Ainslie et al., 2023b). As depicted in Figure 1, the throughput gains of ILLMA-3.1--88 over LLaMA-3.1 1-1- diminish drastically (23.6 + 9.4) with increasing context lengths from IK to 128K tokens.
In this work, SPeculatIve Decoding (RAPID), to bridge the gap of SD for accelerating long-context inference while enhancing generation quality. RAPID employs a RAG drafter-the draft LLM operating on shortened context from RAG-to speculate the generation of long-context LLM following the SD process. We propose that RAG drafter can serve as ideal draft model for long-context target LLM, as
introduce Retrieval-Augmennt
Equal contribution 'National University of Singapore DAMOO Academy, Alibaba Group 'Hupan Lab, 310023, Hangzhou, China. Correspondence to: Guanzheng Chen <ge.chen @u @w..uu.udeeeoooooo edu>, Michael Qizhe Shich <michaclshiche comp.nus.ede edu......s........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Title Suppressed Due to Excessive Size
Perfommace
Thrsughuu
Perfermance
student, derive distilled logits shift towards the RAG drafter during inference. By incorporating the shift into the prediction logits of target LLM, we obtain enriched target distribution that is more receptive to high-quality speculative candidates.
40
18
Thronghpur (tokems/sse)
Perforancc (acc)
36-
34
25
Our RAPID can serve as a drop-in decoding method dur- ing long-context inference. We conduct experiments on (8B, 70B) (Dubey al., 2024) and Qwen2.5 (7B, 72B) (Yang al., 2024) series BeBehch (Zhang et al.) and LongBench v2 (Bai et al., 2024b). The experimen- tal results demonstrate that RAPID successfully integrates the complementary strengths of long-context LLMs and RAG while maintaining significant inference speedups. In self-speculation settings, RAPID achieves consistent perfor- mance improvements (e.g., 42.83 vs 39.33 on InfiniteBench for LLaMA-3.1-8BB with significant speedup (up 2.69x) over the long-context target LLMss The upward-speculation setting further boosts performance through effective knowl- edge transfer from larger RAG drafters (e.g., improving LLaMA-3.1 from 42.83 to 49.98 on InfiniteBench), with comparable efficiency with the smaller long-context target LLMs. With moderate retrieval length (< for RAG drafter, found RAPID consistently achieves speedup when target long-contest length beyond 32K. Our analyses also indicate that RAPID demonstrates robustness to re- trieval quality and potentially superior generation quality in real-world multi-turn dialogue tasks. These results validate RAPID as an effective decoding method for accelerating long-conted inference and, at the same time, enhancing speculation.
12-
20
30-
01
LLaMA-3.1
100
26-
16X
128K
Retrieval Context Length (Tokens)
Figure 1. Performance (accuracy, left axis) and throughput (to-
kens/sec, right axis) of LLaMA-3.1 1-BB (serving on 1 xxA000) and LLaMA-3.1-700 (serving on 8 A800) on LongBench v2 (Long) across different retrieval context lengths.
demonstrates the potential to approach the capabilities of long-context LLM (Li et al., 2024b) while offering superior computational efficiency. As illustrated in Figure I, LLaMA-3.1 1--BBBBBBBBBBBB- with RAG on 4K~ tokens can recover most performance achieved with full 128K tokens. This indicates that the RAG drafter is capable of producing high-quality candidates for long-context target LLM with high acceptance rate, while eliminating the memory-bound KV cache operations over long-context to accelerate the inference process.
generation quality through
In addition, our RAPID opens a new paradigm for SD that leveraging the scae-scale or even larger LLMs as the RAG drafters to accelerate smaller target LLMs. This paradigm shift is possible since RAG drafters, operating on shortened contexts (e.g.. 4K), potentially maintain higher efficiency than target LLMs of the same or even larger scale on long contexts (e.g., 128K) as evidenced in Figure 1. Therefore, our RAPID operates on two settings: (1) self-speculation, where long-context target LLM and RAG drafter are of the same scale; and (2) apward-speculattins where RAG drafter involves larger parameter scale than target L.M.. Moreover, in the both settings, the generation quality of RAG drafter may surpass that of long-context target models in some scenarios (Li et al., 2024a). However, the native SD, utilizing target L.LM prediction as ground-truth distribution to perform rejection sampling, may neglect the candidates of high quality from the stronger RAG drafter. This would result in unnecessary rejection of valid candidates, thereby impeding both efficiency and performance gains.
2. RAPID: Retrieval-Augmented Speculative Decoding
2.1. Background: Speculative Decoding
Autoregressive generation with a LLM ps traditionally re- quires sequential forward passes, where each token 2; is sampled from the distribution P (x.( This sequen- tial nature incurs substantial computational overhead for LLM parameters loading and KV cache manipulation in GPU DRAM. SD accelcrates this process using a smaller draft model d to generate y candidate tokens, which are then validated by the target model P& in single forward pass through rejection sampling. For each speculative token of ~ ( the acceptance criterion is:
S min (1. po(t))le (1<(1//<)))
(1)
To address this limitation, RAPID implements a retrieval- augmented target distribution, which incorporates the native long-context target distribution in SD with an inference-tin knowledge transfer. Specifically, we reversely position the RAG drafter as teacher and long-context target I.LM as the
where r ~ U(O, Upon rejection, a new token is sampled from the residual distribution:
I; ~ no normmmax (2: (1ea)) - (x. 0)),
(2)
where norm is to normalize the distribution by & norm.
2

---

Title Suppressed Due to Excessive Size
This procedure guarantees that the resampled tokens follow the exact distribution direct sampling from the target model Ps. while potentially achieving significant speedup when the speculative tokens are accepted.
Algorithm Retrieval- Speculative Decoding Require: Target LLM ps, RAG drafter 44, context C, re- trieval context CS, number of speculative tokens 7 tem- perature T, transfer strength v
Ensure: Generated sequence Ni:n
2.2. Overview
1: i-1
2: while i S do
While traditional SD offers significant speedups for standard-length contexts, its benefits diminish substantially when handling extensive documents due to memory-bound KV cache operations. We present RAPID, a method that reimagines SD for long-context scenarios while enhanc- ing generation quality, As demonstrated in Alg. I, RAPID comprises two critical components:
3: // Generate speculative tokens using RAG drafter
4: for + to do
- = Io (. (-110 : Fes end for
7: // Validate speculative tokens sequentially
8:
for k: + to do
9:
+ k
10:
// Compute target and draft distributions
RAG Drafter. SD becomes inefficient with long contexts as both draft and target L.MSs must process complete con- text in memory, negating the computational advantages of smaller drafter. To overcome this challenge, RAPID uti- lizes a RAG drafter to generate candidates for long-context LLMs as introduced in $2.3. The RAG drafter operates on selectively retrieved context segments, enabling significant speedups while maintaining access relevant information.
11:
2(2() + (p
12:
p(a,) + softmax(=(x5)/7)
13:
a(x)) + (,]x)|[())
14:
// Compute retrieval-augmented target distribution
15:
$(x() + + (Eq. (8)
16:
p(a)) + softmax(2(x))
17:
r U00(0,
if min(1, p then {(,))
18:
Detrieval-Augmentee Target Distribution. The strict ac- ceptance criterion in SD may reject high-quality candidates,
19:
20:
i+
as it requires strict match to the target LLM distribution for acceptance. This constraint becomes particularly limiting
21:
else
22:
goto line 26
when using RAG drafters, which can potentially generate higher-quality outputs than long-context LLMs in certain scenarios (Li et al., 2024a). To incorporate the benefits from RAG drafters, RAPID steers eetrievalauugment target distribution ($2.4), which enables knowledge transfer from RAG drafter to target model during inference. This mecha- nism allows the target distribution to incorporate valuable information while maintaining theoretical guarantees of the original SD.
23:
end if
24:
end for
25: Sample from residual if rejected
26:
2, norm(max((((e)) - p(x,), p(z,) q(xx). 0))
27: i-i+1
28: end while
29: return 31
enabling efficient identification and extraction of the most relevant context chunks.
2.3. RAG Drafter
When processing queries for extensive context C, the target distribution of naive SD is
After deriving the compress context CS, the draft distribution is formally defined
p(z.) ps (2i
q(x:)) = qw (x
(4)
Even with smaller draft models, the computational bene- fits diminish substantially due to memory-bounn KV cache operations over the complete context C. To overcome this limitation, we propose to leverage RAG as the foundation for our draft model.
where we maintain strict control over the compression ratio by enforcing |C5| < CCI/A with À > 1. This compressed context enables our draft model to maintain significant speed ad advantages while preserving access to relevant information.
Instead of processing the entire context C, our RAG drafter operates on compressed context CS Specifically, CS is constructed through selective retrieval: text segments from are encoded into a dense vector space, where semantic similarity to the query is measured via cosine similarity,
Based on the RAG drafter, the modified speculative decod- ing process proceeds as follows. For each generation step, we sample y speculative tokens from the RAG drafter as <<=)((x)) These candidates are validated against the target
3

---

Title Suppressed Due to Excessive Size
model using modified acceptance criterion:
the student, infuse proportion of knowledge from RAG drafter into naive long-context target distribution.
S min (1, p(xz) =min (I, (a [C; 9i (1r; C3;
g1x))
Specifically, for distillation loss (Hinton et al., 2015) L between RAG draft distribution q(x;) (teacher) and long-context target distribution p(z;) (student), according to Proposition 2.1, we have the distilled logits shift as
where r ~ U(0,1), 1).
The RAG-based drafting mechanism offers two key advan- tages: (1) significant reduction in memory overhead and computational cost through compressed context operations (IC*) < CC), and (2) potentially enhanced speculation qual- ity through selective retrieval of relevant information com- pared to processing diluted full context. Moreover, due to the remarkable efficiency on shorten context, RAPID even enables the use of same-scale or larger models as drafters to accelerate smaller target L.LMs.
ac T (p(a,) g(zx)).
@z(w;)
Now we can derive a "distilled" x(x;) augmented by RAG drafter through
0c
((x;) = =(2;) 1) az(x;;
(8)
2.4. Retrieval-Augmented Target Distribution
= =x4))) + (nI( p(x:)):
The capability of LLMs to effectively utilize context often deteriorates with irrelevant information inclusion. Our em- pirical analysis in Figure shows that LLMs, by focusing on retrieved relevant chunks, can sometimes surpass full- context utilization in generation quality. However, the strict acceptance criterion of traditional SD) may potentially sult in unnecessary rejection for these superior generations when they deviate from the target distribution, leading to both quality degradation and computational inefficiency.
where controls the strength of knowledge transfer. There- fore, the retrieval-augmenntd target distribution in Eq. (6) is equivalent to the normalized [(z4). i.c. p(x;) softmax(3(x:)/T),
The retrieval-augmennter target distribution p(x,) enables flexible knowledge transfer from the RAG drafter while maintaining verification capability. Since the unnormal- ized logits =<x;,) E R have larger magnitude compared to the normalized distributions p(æ;), q(x;) E [0, 1]. the p(x;) preserves the long-context ability of target LLM to verify candidates effectively. We empirically validate the robust- of this distribution in $4.5.
To address this limitation, we introduce retrieval- auumenel- target distribution, which enables knowledge transfer from the RAG drafter the long-context target model during inference. Formally, the eetrieval-augment target distribu- tion in RAPID is defined as:
For inference, we replace p(z;) with p(x;)) in the accep- tance criterion (Eq, (5)). Let p(x;) = [w and p(x;) =
p(x;) = (0ftmax(2(x))TT + (g(x;) - p(x:))), (6)
denote the probability vectors over vocabulary V. Following Li al. (2023), we maintain
W [
where is a hyperparameter controlling the strength of knowledge transfer, z(x;) is the unnormalized logits of tar- get LLM, namely p(x;)) = softmax (x(x:)/T) and T is the temperature.
W& Wk. 4k {u E [[V] w, < <.1 max w (9)
JEl[V]
to prevent distortion in the tail of the distribution.
Proposition 2.1. Let p(z) softmax(2(x)/T) be a student model distribution parameterized by logits z(xr) and temperature T. and q(z) be teacher model distri- bution. The gradient of the knowledge distillation loss
When rejection occurs, we sample from an adjusted residual distribution:
z, ~ norm(max(p(x.) - p(xx),(( g(x:))). (10)
= T (PL(((())))())) with respect to z(x) is:
This sampling strategy maintains theoretical guarantees, where we prove in Appx. $B that the resulting tokens follow the same distribution direct sampling from the original target model p(æ:).
aL = =T (p(x) g(z))
0z(x)
where KI.(-l.) denotes the Kullback- Leibler divergence.
Proof. See Appx. SA.
3. Experimental Setup
3.1. Implementation Details
The design of retrieval- augmented target distribution in Eq. (6) implies knowledge distillation step by posi- tioning the RAG drafter as the teacher and the target model
Target and Draft LLMs. RAPID is evaluated across different model scales using LLaMA-3.1 (8B, 70B) and
4

---

Title Suppressed Due to Excessive Size
Table Comprehensive evaluation of RAPID against haseline methods across different target-draft model configurations. We report performance on oc BeBench and LongBench v2, along with prefill time and throughput speedup on LongBench v2 (Long, CoT) subset. LC and RAG denote evaluating the target model on long and retrieval contexts, respectively. For RAPID, we evaluate both specullennn (using same-size RAG drafter) and upward-speculltton (using larger RAG drafter) settings. Green/red highlighting indicates better/worse performance compared to LC baseline. Bold and underline indicate best and second best metric score.
Target Model
Method
Draft Model
oc Benech
LongBench v2
Efficiency
En. QA En, MC En. Sum AVG. Overall Overall (CoT) Prefill Time (s) Speedup
LC
34.58
53.28
30.14
39.33
28.0
30.4
25.80
x.00x
RAG
31.91
62.01
27.27
40.40
29.2
33.4
0.36
3.35x
SD
32.90
55.90
30.11
39.64
29.4
31.0
26.37
1.63
LJMMA-3.1-8
MagicDee-
29.83
52.03
30.18
37.35
29.2
30.6
26.05
0.71
RAPID
I. (RAG)
34.90
63.32
30.27
42.83
32.4
34.2
26.37
2.10x
RAPID
LLaMA-3.1-708 (RAG)
40.94
79.04
29.96
49.98
38.8
40.2
28.04
1.14x
LC
36.48
68,56
30.18
45.07
31.6
36.2
160.54
x.00x
LLaMA-3.1-70 RAG
38.66
76.86
27.17
47.56
38.0
39.4
2.81
x
RAPID
LLaMA-3.1-70B (RAG)
40.56
81.66
29. 64
50.62
40.2
40.2
163.43
2.69x
LC
16.93
66.81
30.62
38.12
30.2
33.2
20.32
x.00x
RAG
20.28
75.11
25.60
40.33
31.2
33.8
0.34
6.47
Qwen2.5-7B
RAPID
Qwen2.5-7B (RAG)
19.81
75.98
31.64
42.48
32.0
35.4
21.62
x.65x
RAPID>
Qwen2.5-728 (RAG)
30.10
83.84
32,21
48.72
35.6
41.2
23.45
0.93
L.C
39.21
81.66
32.45
51.11
40.0
43.9
162.42
x.00x
Qwen2.5-72B
RAG
30.72
80.22
28.63
46.52
38.8
39.8
3.09
3.60
RAPID
Qwen2.5-72B (RAG)
40.52
85.59
32,94
53.02
42.9
44.1
164.80
1.98
Qwen2.5 (7B, 72B) as target LLMs. We implement two speculation settings: (1) self- specullann,,,nn where the RAG drafter matches the target LLM's scale, and (2) upward-speculation, where a larger RAG drafter assists a smaller target LLM. For smaller models (LLaMA-3.1) 1--8B, Qwen2.5-7B), we evaluate both settings, while larger mod- els (LLaMA-3.1-700) Qwen2.5-72B) use self-speculation only. The RAG drafter generates 7 10 tokens per step for target LLM verification. We search 17 in Eq. (6) between {5, 10, 20} for self-speculation and {40, 50} for upward- speculation, which would be further investigated in $4.5.
cal target and draft LLMs with RAPID but using the naive long-context target distribution; (4) MagicDec (Chen et al., 2024a), which utilizes the StreamingL (Xiao et al., 2023) to compress the KV cache of draft model. We set the KV cache size as 4096 and sink tokens as 4.
Benchmarks. We evaluate our RAPID with baselines on two benchmarks: (1) oeBench. We evaluate our method with baselines on three realistic tasks in this benchmark: long-book question answering (En. LA, metric: F1), multi- choice question-ansvee answering (En.MC, metric: accuracy), and summarization (En.Sum, metric: ROUGE-L-Sum), The context length in these tasks are beyond 100K. (2) Long- Bench v2, which involves multi-choice tasks across various context lengths from 8K to 2M words. We apply middle truncation following benehmark setup to ensure the context length within 128K tokens.
RAG Setup. The long context is segmented into 512- token chunks and embedded using BGE-M3 (Chen et al., 2024b). We retrieve top-k segments based on cosine similar- ity with the query embedding, filtering out segments below 0.3 similarity threshold. The retrieval context length is bounded between 4096 tokens and 1/24 of the input length.
Evaluation Setup
We conduct efficiency evaluations us- ing the LongBench v2 (Long, CoT) subset, where each example involves 120K (tokens) context length after trunca- tion and 1K maximum generation tokens. Efficiency metrics include: (1) prefill time and (2) speedup, computed as the ratio of method decoding throughput to LC throughput, both averaged across the subset. Additional experimental details are provided in Appx. SC.
3.2. Evaluation Protocol
Baselines. We compare our RAPID with baselines includ- ing: (1) long-context target LLM (LCC,, where the target LLM in RAPID directly generates responses upon long con- text; (2) RAG, where the target LLM generates responses upon retrieval context of draft LLM input in RAPID; (3) naive Speculative Decoding (SD), which involves identi-
5

---

Title Suppressed Due to Excessive Size
56
Daal:
Drah:
513 IAU1D
SD
RAIID
Rebatve Sucress (%)
Oaly
Rolative (%)
15.
7.75
12.52
5.17
4.17
1.36
4.57
3.78
2.94
2.19
01
(Thegr)) (00(0
(Tegr)) (Drel) (Tenge)) (Prrd)
(Thrgg)))
(Targe) (Dnc)) (Twgee) (Driit)
Figure 2. Relative performance to larget LLMs across different target-draft model configurations of LLaMA-3.1 series LongBench v2 (Overall). RAPID integrates both benefits from target and draft LLMs, hence achieving higher relative success rate (benefits from draft) without increasing failure rate (benefits from target). Relative success represents correct predictions made by each method but missed by the target L.M.M Relative failure represents correct predictions by the target L.IM missed by each method, "SD Only" and "RAPID Only" indicate correct (or wrong) predictions made exclusively by SD and RAPID where both target and draft models cannot attain.
4. Results and Analyses
drafter achieves 49.98 BeBenc and 40.2% overall ac- curacy on LongBench v2, surpassing not only its self- speculation results but even the LC performance of LLaMA- 3.1-70B (36.2%). Similar patterns emerge for Qwen2.5- 7B with 72B RAG drafter, where the performance gains (48.72 vs 42.48 on ocBench) demonstrate the effectiveness of RAPID in leveraging and integrating knowledge from larger models through the retrieval -augreeela speculation.
4.1. Main Results
We evaluate RAPID against baselines across different model scales and benchmarks. The results in Table demonstrate the effectiveness of RAPID in both improving generation quality and efficiency for long-context inference.
RAPID integrates benefits from both target LLM and RAG drafter through self-speculation. In the self- speculation setting, where RAPID uses same-scale models for target and draft, consistent improvements are observed across model families. For LLaMA-3.1 1-BBB, RAPID with self-speculation achieves superior performance on x. Bench (42.83 vs 39.33 LC, 40.40 RAG) and LongBench v2 (34.2% vs 30.4% LC, 33.4% RAG). Similar improvements are seen for LLaMA-3.1 (50.62 45.07 LC, 47.56 RAG BoBench and Qwen2.5 series. Notably, RAPID effectively integrates the complementary strengths of LC and RAG ap- proaches- while RAG shows superior performance on cer- tain tasks (e.g., Fn.MC: 79.04% vs 53.28% L.C for L-LaMA- 3. 3-B)) LC demonstrates advantages in others (c.g., En. QA: 34.58% vs 31.91% RAG), RAPID successfully captures these complementary benefits during inference, consistently achieving better or comparable performance to the stronger of its two components. Compared to existing speculative decoding approaches including naive SD and MagicDec, RAPID demonstrates superior performance through this effective integration mechanism.
RAPID demonstrates > 2x speedup for long-context inference. In self-speculation settings, RAPID achieves significant speedup over LC baseline (2.10x for LLaMA- 3. 3-BB 2.69x for LLaMA-3. 1-70B), and significantly sur- passes naive SD and MagicDec. When employing upward- speculation with larger drafters, RAPID still maintains com- parable throughput (1.14x for LLaMA-3.1-88 with 70B drafter, 0.93 for Qwen2.5-7B with 72B drafter) while sub- stantially improving generation quality. While pure RAG shows highest throughput (e.g., 3.35x speedup for LLaMA- 3. 3-1. its performance can be significantly compromised in certain scenarios (e.g., En.QA accuracy drops from 39.21 to 30.72 for Qwen2.5-72B). In contrast, RAPID effectively maintains competitive throughput while consistently achiev- ing superior generation quality across different settings.
4.2. Benefits Integration Analysis
RAPID incorporates benefits from RAG drafter while maintaining target model capabilities. To analyze how RAPID integrates the strengths of both RAG drafter and target LLM, we examine the relative success and failure of RAG drafter, SD, and RAPID on LongBench v2. As shown in Figure 2, RAPID successfully handles additional cases where the target LLM fails by incorporating beneficial
Larger RAG drafters further boost performance through effective knowledge transfer. Beyond self- speculation, RAPID enables a unique upward-speculatie mechanism where larger models serve as RAG drafters while maintaining efficiency. This setting yields even more substantial improvements: ILaMA-3.1 1-B--B with 70B RAG
Note that upward-speculation requires extra GPUs the RAG drafter like regular SD.
6

---

Title Suppressed Due to Excessive Size
Table 2. Evaluation on multi-tumn dialogue generation with ex- tended chat history for LLaMA-3. 1-8B as both target and draft LLM. Quality scores (1-10) are rated by GPT-4-Turbo 1106 using
AAceuaayy (%)
LLM-as-a-Judge protocol.
Quality
Acceptance Rate (%) Throughput
Target LLM
2.82
10.64
4X
8K
16K
52K
64K
124K
RAG Drafter
3.95
40.49 10.47
SD RAPID 4.21
2.94
56.34+0.13
14.07
Acceptance Rate (%)
76.9410.13
18. 18.18 1.3.33
80
Table 3. Robustness study of RAPID with different draft influ- ence parameter Results show performance gains (AAccuracy) and speedup ratios on LongBench v2 (Long, CoT) subset using ILaMA-3.1 1-BB8B as target I.M,, with I.LaMA-3.1-88 and LLaMA- 3.1-70B as RAG drafters under unrelated retrieval context.
70
4K
125K
2.0
RAPD (Retiivvl 4K)
SD (Ratrieal 4K)
LLaMA-3.1 (Draft)
LLaMA- 3.1- (Draft)
Speedup Ratio
1.5
RAPID (Retieval
(Rctriooll lok)
(Rotieval
(Rarieed 168)
7
AAccuracy
Speedup
Acccuracy
Speedup
RAPID Betiivdd X2K)
SD (Retrival 32X)
0
1.20
1.62x
-1.30
0.67x
1.0
5
2.80
1.75x
0.40
0.69x
10
1.60
1.77x
1.20
0.72x
20
1.20
1.78x
4.40
0.75x
4K
KK
16K
32K
64K
128K
30
-2.40
2.07x
6.60
0.80
Target Context Length (Tokens)
40
-2.60
2.08
6.60
0.84x
50
-6.30
2.10x
6.00
0.87x
Figure 3. Impact of context and retricval lengths on RAPID (self- sepculation) performance and efficiency based on LLaMA-3.1 8B. RAPID consistently outperforms naive SD and achieves speedup beyond 32K context length with moderate retrieval lengths Top: indicates the accuracy margins on Long- Bench v2 (Long, CoT) subset over target I.I.M. Middle: Ac- ceptance rate indicating the proportion of accepted draft tokens. Bottom: Speedup ratio compared target LLM inference (> indicates acceleration).
where the stronger RAG drafter facilitates more sophisti- cated knowledge transfer during inference.
4.3. Impact of Context and Retrieval Length.
RAPID demonstrates effectiveness across various con- text configurations. We analyze how RAPID performs under varying target context lengths and RAG drafter re- trieval lengths, as shown in Figure 3. The results demon- strate consistent advantages of RAPID over naive SD across all configurations. First, RAPID achieves significantly bet- ter performance gains (2-8% AAccuracy) over the long- context baseline compared to the marginal or negative gains (-5-2%) of naive SD. This superior performance is accompa- nied by consistently higher acceptance rates (75-85% versus 60- 70%) and better speedup ratios across all context and retrieval lengths configurations.
knowledge from the RAG drafter. Meanwhile, RAPID main- tains the capabilities of target LLM,, exhibiting significantly lower failure rates compared to using RAG drafter alone. This combination of gains from RAG drafter with mini- mal degradation of target LLM capabilities enables RAPID to outperform both target and draft models. Furthermore, the gains from RAG drafter in RAPID substantially exceed those in naive SD, demonstrating the effectiveness of our aerieval-augmente target distribution in Eq. (6).
RAPID exhibits capabilities beyond individual tar- get/draft LLMs. Most notably, we observe an "emergent phenomenon" where RAPID successfully handles cases that both the target LLM and RAG drafter fail individually (shown as "RAPID Only" in Figure 2). Specifically, this emergent accuracy mass grows more pronounced as RAG drafters become stronger, from ALaMA-3.1-88 to LLaMA- 3. 1...B...T.. This suggests that RAPID not only combines the strengths of both models but also enables new capabilities through their synergistic interaction. The phenomenon be- comes particularly evident in the
RAPID achieves speedup for long-context inference be- yond 32K. The impact of retrieval length reveals an inter- esting efficiency-effectiveeee trade-off. In terms of compu- tational efficiency, RAPID achieves acceleration (speedup > 1.0x) when the target context length exceeds 32K, while SD requires contexts beyond 64K to demonstrate speedup. For retrieval length, while longer retrieval contexts generally lead to higher acceptance rates (up to 85%), the speedup ratio is not necessarily increasing. Specifically, retrieval lengths of 4K and 8K achieve nearly identical speedup ra- tios, indicating minimal overhead in this scope. However,
setting,
7

---

Title Suppressed Due to Excessive Size
when retrieval length exceeds 16K, the increased computa- tional overhead from longer draft contexts becomes apparent and impacts the overall speedup. These findings suggest that RAPID achieves remarkable efficiency when accelerating long-context inference beyond 32K tokens upon moderate retrieval length within 16K.
retrieval quality.
5. Related Work
Speculative Decoding Speculative Decoding (Chen et al., 2023; Leviathan et al., 2023) accelerates LLM inference by leveraging smaller draft models to propose multiple to- kens for single-pass validation. REST (He et al., 2024b) extends the drafting mechanism by retrieving possible con- tinuation from built corpus rather than generating with draft LLM. Ouroboros (Zhao et al., 2024) proposes pro- ducing longer and more acceptable candidates from draft LLM per step based on draft phrases. Inspired by the spec- ulation mechanism, Speculative RAG (Wang et al., 2024) proposes a parallel draft-then-veriff mechanism to improve RAG quality. Recent works like TriForce (Sun et al., 2024) and MagicDec (Chen et al., 2024a) attempt to extend SD to long-context scenarios through KV cache compression techniques (Xiao et al., 2023). However, such compression approaches often result in weakened draft models with lim- ited speedup in complex applications. In contrast, RAPID adopts RAG drafters that maintain both high-quality specu- lation and substantial speedup in various applications.
4.4. Generation Quality Analysis
RAPID achieves superior generation quality and throughput in real-world application. To evaluate the ef- fectiveness of RAPID in practical long-context applications, we assess its performance on multi-turn dialogue generation. We construct a challenging evaluation dataset by adapting 10-BBecch-101 (Bai et al., 2024a): for each of the first 100 samples, we preserve their last-turn queries while distribut- ing their previous conversation context within a longer chat history comprising additional dialogue turns from another 500 samples in MT-Bench- 101. The resulting chat history of around 122K tokens length. This setup tests the abil- ity of models maintain coherence and relevance while processing extensive dialogue history.
As shown in Table 2. RAPID demonstrates substantial im- provements across all metrics. Using GPT-4-Turbo- 106 as evaluator following LLM Judge (Zheng et al., 2023), RAPID achieves generation quality score of 4.21, signifi- cantly outperforming the target LLM (2.82), RAG drafter (3.95) and naive SD (2.94). This quality improvement comes with a robust acceptance rate of 76.94% (vs. 56.34% for SD) and enhanced throughput of 18.15 18 tokens/second (1.7x x speedup over target LLM), demonstrating practical advan- tages of RAPID in real-world long-context applications.
Long-Conter Inference Speedup Research on acceler- ating long-context inference has primarily focused on two directions: optimizing KV cache operations through selec- tive retention (Xiao al., 2023; Kang et al., 2024; Zhang et al., 2023) or quantization (Sheng et al., 2023; Liu et al., 2024b; He et al., 2024a), and exploring prompt compres- sion methods (Chevalier et al., 2023; Jiang et al., 2023; Pan et al., 2024). While these approaches improve efficiency, they often compromise contextual information without qual- ity guarantees (Zhang et al., 2024). RAPID addresses this limitation by leveraging SD maintain generation qual- ity through explicit verification from long-context LLMs, providing a more reliable balance between efficiency and performance.
4.5. Robustness to Retrieval Quality
RAPID shows robustness to retrieval quality, which is further enhanced by stronger drafter. To assess the robustness of RAPID regarding retrieval quality, we con- duct stress tests by deliberately using unrelated retrieval context (using the context of first sample from LongBench v2 for all samples) while varying the knowledge transfer parameter in Eq. (6). As shown in Table 3, with self- speculation (LLaMA-3.1-88) drafter), RAPID maintains per- formance gains (AAcurayy > 0) and improved efficiency (speedup 1.62x-1.78x) when 17 S 20, even with irrelevant retrieval context. However, when 17 > 20, the RAG drafter may overly impact the target distribution, leading to per- formance degradation. Moreover, upward-speculatton with LLaMA-3.1-70B as drafter demonstrates even better robust- ness, maintaining positive performance gains (up to 6.60%) across all 7 values despite totally unrelated retrieval context. This increased resilience suggests that RAPID effectively leverages the inherent capabilities of stronger RAG drafters, maintaining reliable performance even under suboptimal
RAG and Long-Context LLMs Recent studies have re- vealed complementary strengths between RAG and long- context LLMs, with substantial prediction overlap despite different performance characteristics (Li et al., 2024b;a). While long-context LLMs excel in document-based tasks, RAG shows advantages in scenarios like dialogue-based question-answering Previous attempts to combine these approaches, such as self-refiection routing (Li et al., 2024b) and step-by-step RAG enhancement (Yue et al., 2024), rely heavily on task-specific prompt engineering, RAPID pro- vides a more principled solution by directly integrating RAG benefits into the decoding process, enabling dynamic adap- tation while preserving advantages of both paradigms.
8

---

Title Suppressed Due to Excessive Size
6. Conclusion
Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., and Ouyang, W. MTTbench- 101: A fine-grained benchmark for evaluating large lan- guage models in multi-turn dialogues. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7421-7454, Bangkok, Thailand, August 2024a. Association for Com- putational Linguisties. doi: 10. 10.18663//1/2222..cdl 401. URL a acl scll-oong.401
In this work, we introduce RAPID, a novel decoding method that bridges the efficiency gap of speculative de- coding (SD) in long-context inference while enhancing generation quality through retrieval-augmennte The key of RAPID lies in leveraging RAG drafters to en- able efficient speculation for long-come contectt target LLMs, along with a retrieval-augmented target distribution that effectively integrates knowledge from potentially stronger drafters. Through extensive experiments, we demonstrate that RAPID successfully achieves both computational ef- ficiency and improved generation quality across different model scales and tasks. Specifically, RAPID enables more than 2x speedup while maintaining performance advantages in self-speculation settings, and achieves substantial quality improvements through -peeuulspeellttion with stronger RAG drafters. These results establish RAPID as a prac- tical solution for accelerating long-context inference with improved generation quality.
speculation.
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang. X., Lv, X., Cao, S., Xu, J., Hou, L.. Dong, Y., Tang, J., and Li, J. Longbench v2: To- wards deeper understanding and reasoning on realis- tic long-context multitasks. ArXiv, abs/2412.15204, 2024b. URL https: httpps//aap..smmmtttcccooooo org/ CorpusID:
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling, 2023. URL https: //arxiv. www. abs 2302. 01318.
Acknowledgments
This project was partially supported by the Singapore Min- istry of Education Academic Research Fund Tier I (Award Number: TI 251RES2514) and DAMO Academy Research Intern Program.
Chen, J., Tiwari, V., Sadhukhan, R., Chen, Z., Shi, J., Yen, I. E.-H., and Chen, B. Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding. arXiv preprint arXiv:2408.1104 2024a.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. M3-embedding: Multi-linguality, multi- functionality, multi tratiiirtaullrtyy text embeddings through self-knowledge distillation. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguisties: ACL 2024, pp. 2318- 2335, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024 findings- ac1. URL https: h/tp://hooaa...o....... 2024/2224.22
References
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F,, and Sanghai, S. GQA: Training gen- eralized multi-query transformer models from multi- head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895-4901, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023. emnlp-main.298. URL https: /awalaaannoooooyoo...o org/2022 masppppmmii..9......
Chevalier, A., Wettig, A. Ajith, A., and Chen, D. Adapting language models compress contexts. ArXiv, abs/2305.14788, 2023. URL https:
/ 258865249.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Ainslie, J., Lee- Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head check- points. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b. URL https: //openreview httpe/iwwwwweeeeeeeeeee set/foormm forum d=hmOwoZW2VE..
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y. Sun, J., Guo, Q., Wang, M., and Wang, H. Retrieval-augmented generation for large lan- guage models: A survey. ArXiv, abs/2312.10997,

---

Title Suppressed Due to Excessive Size
2023. URL htcps://ppi.smmanticchhool... oTg/CorpuSTD:22635311
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, I.., and Lewis, M. Con- trastive decoding: Open-ended text generation opti- mization. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 12286-12312, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: anl- long. 687/.
He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. Zipcache: Accurate and efficient KV cache quantization with salient token identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. UtLhttps:/// http:ppss///perevie....o net/ Forum?id=5t 4ZAXPiJs.
doi: 10. 10.186553/1/0033.dlllooo // h/aclannhology.orgggg....
He, Z., Zhong. Z., Cai, T., Lee, J., and He, D. REST:
Retrieval-based speculative decoding. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp- 1582-1595, Mexico City, Mexico, June 2024b. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ 2024.naacl-long.55 88. URL https::////// hwtphhtsss//caaattttteeeeeeee /aclaaananoooo...... org 2024 1oa.el11ooo...00
Li, Z., Li, C., Zhang, M. Mei, Q. and Bendersky, M. Retrieval augmented generation or long-context llms? comprehensive study and hybrid approach. ArXiv, abs/2407.16833, 2024b. URL https:
/ //api.sennt wwr.rccccccoo.ooooooooorot.o 27 271404721.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with blockwise ringattention. ArXiv, abs/2402.08268, 2024a. URL ssttps://apts semanticscholar. org/ http:/CoppIII::277557777
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. ArXiv, 0bs//1503.0231, 2015. URL pettps://ap..stt www.smattiicchhar........ org/CozpusII: 7200347.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. ArXiv, abs/2402.02750, 2024b. URL https: / /api. semanticscholaa org/ /Copp/osI:
Jiang, H. Wu, Q. Lin, C.-Y., Yang, Y., and Qiu, L. Limlingua: Compressing prompts for accelerated in- ference of large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https:/ http::::/ap..sematiccssso. weeeetiiisccooo..e............te
Pan, Z. Wu, Q. Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang. Y.. Lin, C.-Y., Zhao, H. V., Qiu, L., Zhang, D., Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plap- pent M.. Tworek, J., Hilton, J., and Nakano, Limlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Annual Meeting of the Association for Computational Linguistics. 2024. URL https: / //ppp/appa// org Io:
Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Tiihth- Annual Conference Neural Information Processing Systems, 2024. URL https: http:////peeeeeeeeeeeee net./ forum?id:fPBBee.
Kang, H., Zhang, Q. Kundu, S., Jeong, G., Liu, Z. Krishna, T., and Zhao, T. Gear: An efficient kv cache compression recipe for near-lossless gen- erative inference of Ilm. ArXiv, abs/2403.05527, 2024. URL https: http:://pp...mmmtiiccseoee www.stiischoleee.e.... otr//CorpusID::2822222
Pope, R. Douglas, S., Chowdhery, A., Devlin, J. Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scal- ing transformer inference. ArXiv, abs/2211.05102. 2022. URL https: //ats:://ai... semanticscholas org/ /
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding, 2023. URL attts:://arxtr orggabs. / 2211 17192.
Sheng, Y., Zheng, L., Yuan, B.. Li, Z., Ryabinin, M., Fu. D. Y., Xie, Z., Chen, B., Barrett, C. W. Gonzalez, J., Liang, P., Ré, C., Stoica, I., and Zhang, C. High-throughput generative infer- ence of large language models with a single gpu. International Conference on Machine Learning, 2023. URL httttasss/pp..eennttccccoooooll org/CorpusTD:227445533
Li, X., Cao, Y. Ma, Y., and Sun, A. Long con- text vs. rag for llms: An evaluation and revisits. 2024a. URL http:://ppp..mmmniccccooloo// ort//oorpiSTD:22553388
10

---

Title Suppressed Due to Excessive Size
Sun, H., Chen, Z.,, Yang. X., Tian, Y., and Chen, B. Tri- force: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404. 1912, 2024.
Zhang, Z. A., Sheng, Y., Zhou, T., Chen, T., Zheng,
L., Cai, R., Song, Z.,, Tian, Y., Ré, C., Barrett, C. w. Wang. Z., and Chen, B. H2o: Heavy- hitter oracle for efficient generative inference of large language models. ArXiv, abs/2306. 14048, 2023. URL https://api.semat..ccom www.sttesmcnoooooooooeeo orgg/oorpa1D::2595555
Team, G., Georgiev, P. Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Zhao, W., Huang, Y.. Han, X., Xu, W., Xiao, C., Zhang, X., Fang, Y., Zhang, K., Liu, Z., and Sun, M. Ouroboros: Generating longer drafts phrase by phrase for faster speculative decoding. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference Natural Language Processing, pp. 13378-13393, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10. 10.18663/112022.cmmlpp main. 742. URL https:
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E.,, and Lam- ple, G. Llama: Open and efficient foundation language models. 2023. URL https: //rxiiv.// org/ abs/ 2302.13977
Empirical Methods in
2024.
Wang, Z., Wang, 7., Le, I.. T., Zheng, H. S., Mishra, S. Perot, V., Zhang, Y., Mattapalli, A.,
emn munlp mem-mmain.. 00.mn.74//) 742/.
Taly, A., Shang, J., Lee, C.-Y., and Pfister, T.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.. Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, 0lx/2306.05683 2023. URL https: :/op:p:://pp/ //api. www.nticschol.i.......o ozs/CorppusD::25912953
Speculative rag. Enhancing retrieval augmented gen- eration through drafting. ArXiv, abs/2407.08221 2024. URL oxg//oozpusID:2710977
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv, 2023.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 21X1v:2412.15115, 2024.
Yue, Z., Zhuang. H., Bai, A., Hui, K., Jagerman, R., Zeng, H., Qin, Z., Wang, D., Wang, X., and Ben- dersky, M. Inference scaling for long-context re- trieval augmented generation. ArXiv, abs/2410.04443,, 2024. URL https: /api/app.../ http:://ppi.smmmticccc.oo. semanticschola.r. :ttp:Coopps
Zhang, J., Zhu, D., Song, Y., Wu, W., Kuang, C., Li, X., Shang, L., Liu, Q. and Li, S. More tokens, lower precision: Towards the optimal
trade- off in kv cache compression. ArXiv, abs/2412.12706., 2024. URL https::/api.semant... TTr//oorpuSID:224985
Zhang, X., Chen, Y., Hu, S.. Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S. Liu, Z., and Sun, M. ocBench: Extending long context evaluation beyond 100K tokens. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). URL https:/ /aclanthology. org/2022 11224.ccl-110 acl-long. 814.

---

Title Suppressed Due to Excessive Size
A. Proof of Theorem 1
analyze the gradient of the knowledge distillation loss with respect to the target model's logits. The distillation loss with temperature T is defined as:
C C= (L((())llp(x))
= -7² ((2)) log {(1z)) p(z,)
(11)
where the target distribution p(z) is parameterized by logits 2 through softmax:
exp(z,/T)
plæ,) = Σx exp(2x/T)
(12)
Theorem:
The gradient of the distillation loss with respect to logit 2; is:
ac = -T(((z)) - p(2;))
(13)
0z,
Proof: We derive this gradient through the following steps:
1) First, expand the derivative using the chain rule:
OC
T Σ g((z)) Ozi
[log (ogg(a;)) - (x,)(
(14)
0z:
2) Note that q(z)) is independent of z4:
0
- Σ a(z5))
log p(z;)
(15)
3) Expand the log probability:
-- Σ
(3(rg)
θ
2j
log Σ exp(2x/T)
(16)
Oz,
T
4) Apply the derivative using the Kronecker delta ōzy:
ozg exp(2./T)
-2 = g(x,))
(17)
E exp(2x/T)
5) Simplify using the definition of p(x:):
-
-T Σ (555)(555 - p(x:))
(18)
6) The sum over j with diz selects only q(x;):
= =-T(a(xi) - Σ a(x))r(Z.))
(19)
7) Since E, ((1;)) 1, we obtain our final result:
-T(g(x>)) p(x:)]
(20)
This gradient shows that the distillation loss pushes the target distribution p(æ) towards the draft distribution g(Φ) with strength proportional the temperature T.
12

---

Title Suppressed Due to Excessive Size
B. Correctness of RAPID's Residual Distribution
We prove that RAPID's retrieval-augmentiee speculative decoding, when rejection occurs, sampling from the distribution
3i ~ norm(max(p((t) - -((x;),((;)) ((z:)))
(21)
maintains the target distribution p(z;), where:
p(a,) P& (2, [C; aed) (target distribution)
(22)
((z4) (0: x<+)) (RAG drafter distribution)
(23)
p(x;) = (a (retrieval-augmeette target)
(24)
Proof: Let a' be candidate token. Under RAPID's rejection sampling scheme:
1) For token z' proposed by the draft model, the acceptance criterion is:
min(1, p(x')
(25)
((x))
where 1)
2) This leads to an acceptance probability:
P(accept|zz) min(g(="),
(26)
3) The residual probability mass that needs to be redistributed upon rejection is:
p(a') min(g(z"), max(p(x') q(x'), p(x'))
(27)
4) Let (3 be the total acceptance probability:
B Σ min(g(='),
(28)
5) Therefore, upon rejection, must sample from:
p(x') min min(g(2()) p(x'))
p(x') min(g(x'), P(('))
v'(x') En min (g(x') (g(xx))((()))))))))) p(x'))
(29)
This residual distribution ensures that for any token x!:
P( =') min((mmm)))) ((1-)) -
p(x') 1 1-8
=(xx)
(30)
C. Evaluation Setup
We conduct comprehensive evaluations across different model scales and configurations. We use temperature values of 1.0 and 0.1 for BBennh and LongBench v2, respectively. For base-scale models (LLaMA-3.1 1-BB and Qwen2.5-7B), evaluate RAPID's self self-speculation capabilities against multiple baselines including naive Speculative Decoding, MagicDec, Long Context (LC). and RAG implementations, using single NVIDIA A800 80GB GPU.
For large-scale models (LLaMA-3.1-) and Qwen2.5-72B), self-speculation experiments are conducted using distributed setup with 8 A800 80GB GPUs. In (LLaMA-3.. --1 1-8.-wwn-.....55555 to accommodate the larger RAG drafter.
settings, we employ a hybrid configuration where the target models operate on a single A800 80GB GPU, while leveraging an additional 7 A800 80GB GPUs
13

---

Title Suppressed Due to Excessive Size
D. More Efficiency Analyses
D.I. FLOPs Comparison
We present a detailed comparison floating-point operations (FLOPs) per generation step (producing tokens) Table 4, analyzing our RAPID method against baseline approaches. Let T denote the number of parameters in the target model and L represent the long context length. For the draft model, define:
D: Number of parameters
L R Retrieval length for draft LLM input
The key parameters for speculative generation include:
y: Number tokens generated by the draft model per step
gSD: Expected acceptance rate for standard speculative decoding
BRAPID: Expected acceptance rate for RAPID
Our analysis reveals that while all methods scale linearly with the target model size T, RAPID achieves superior efficiency through its higher acceptance rate (3RAPID BSD), which directly reduces the amortized FLOPs per generated token.
Table 4. FLOPs comparison for different methods per step.
Method
FLOPs
Long Context
2/TL + 2T
RAG Drafter
2yDL R. + 7
2rD1 4
SD
3SD
RAPID
BRAPID
D.2. Overhead RAG
Unlike regular RAG pipeline, which builds indexes for a large external corpus (hundreds millions of documents), we only index/retrieve the chunks for the input long context (< 128K) on-the-fly during inference. Therefore, the RAG component latency in method will become marginal compared to the inference latency over long context. Table 5 presents the average latency (in seconds) for each component of RAPID on LongBench v2 (Long, CoT) using LLaMA-3.1-88 and LLaMA-3.1-700 in self-speculative mode.
Table 5. Latency of RAPID Components LongBench v2 (Long, CoT)
Model
RAG Pipeline (s)
Prefill (s)
Generation (s)
LLaMA-3.1-BB-RAPII LLaMA-311-70B-RAPH
1.43
26.37
32.25
1.43
163.43
121.76
E. More Results
E.1. Comparison with TriFForee
TriForce was not included in Table since it is not directly compatible with modern LLMs using Grouped Query Attention (GQA) (Ainslie et al., 2023a). We conducted comparisons on 7B (Touvron et al., 2023)), with a retrieval budget of 4096 tokens, chunk size of 8, and a draft cache budget of 256 for TriForce. Table 6 shows the performance and speedup of the decoding in LongBench (Long, CoT).
(Liu et al., 2024a) (based on LLaMA2-
14

---

Title Suppressed Due to Excessive Size
Table 6. Comparison of RAPID and TriForce -WM-Text-Chat- 128K LongBench v2 (Long, CoT) task.
Model
Accuracy
Speedup
WWWM-Text-Cht---- 128K
18.4
1.00
TriForce
18.0
1.27
RAPID
21.6
2.56
While TriForce achieves modest efficiency gains, RAPID delivers superior speedup and performance. TriForce relies on chunk-wise attention scores for information recall, but high attention scores do not always correlate with semantic relevance, e.g., initial tokens may act as "attention sinks" despite lacking meaningful content (Xiao et al., 2023). contrast, our RAPID drafter prioritizes semantically relevant information, resulting in a higher acceptance rate and greater speedup for complex tasks.
E.2. Comparison with MInference
evaluated MInference (Jiang et al., 2024) against our RAPID using LLaMA-3. 1-BB on the LongBench v2 (Long, CoT) task. Table 7 reports the performance, prefill time (in seconds), and decoding speedup relative the LLaMA-3.1-8B.
Table 7. Comparison of RAPID and MInference LLaMA-3.1-8B in LongBench v2 (Long, CoT) task.
Model
Accuracy
Prefill Time (s)
Speedup
LLaMA-3. 1-8B (Baseline)
30.4
25.89
1.00
MInference
30.9
9.10
0.62
RAPID
34.2
26.37
2.10
MInference significantly reduces prefill time, showcasing its efficiency in the initial processing phase. However, RAPID outperforms MInference in overall performance and decoding throughput, achieving higher speedup. We note that sparse attention, as utilized by MInference, is orthogonal to our approach, suggesting that integrating sparse attention with RAPID could further enhance efficiency.
15


Reviews:
[{"review_id": "ntFkAFr2Pc", "reviewer": "Reviewer_bhTe", "summary": "This paper introduces Retrieval-Augmented Speculative Decoding (RAPID) that aims at both accelerating and enhancing generation quality in long-context inference. SD becomes inefficient with long contexts since both draft and target LLMs need to process complete context in memory. The authors introduce a RAG drafter, a draft LLM that operates on shortened retrieval contexts to speculate on the generation of long-context target LLMs. This method allows same-scale or even larger LLMs to function as drafters while maintaining computational efficiency. Additionally, the approach incorporates inference-time knowledge transfer, enabling stronger RAG drafters to enhance the final output quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bhTe,\n\nWe sincerely appreciate your thorough review of our paper. Your constructive feedback will help us strengthen this work. Below are our responses to your concerns:\n\n---\n\n### 1. Possible explanation for the sudden drop in Figure 1.\n\nA1: This is an observed issue in RAG that performance may drop at certain lengths when more retrieval chunks are included [1]. The explanation from [1] is that the presence of certain hard negatives can mislead the LLMs and hinder their ability to generate accurate answers (even if relevant information is included). This issue cannot be directly addressed by using a stronger retrieval model. We believe the retrieval chunks between 16k and 64k may contain many hard negatives. Note that the drop in Fig 1 of our paper is not as “sudden” as displayed, as the x-axis is log-scaled.\n\n[1]Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG\n\n---\n\n### 2. FLOPs comparison\n\nA2: Thanks for your suggestion. We now list the FLOPs of our RAPID and baselines per step (generating $\\gamma$ tokens) below. As our RAPID consistently demonstrates higher acceptance rates than naive speculative decoding (SD) (shown in Fig. 3 of our paper), the FLOPs order will be Long Context >> SD > RAPID > RAG Drafter.\n\n|  | FLOPs |\n| --- | --- |\n| Long Context Target | $2\\gamma T L + \\gamma^2 T$ |\n| RAG Drafter | $2\\gamma D L^{R} + \\gamma^2 D$ |\n| SD | $\\frac{2\\gamma D L^{R} + \\gamma^2 D+ 2T(L+ \\gamma)} {\\beta^{\\text{SD}}}$ |\n| RAPID | $\\frac{2\\gamma D L^{R} + \\gamma^2 D+ 2T(L+ \\gamma)}{\\beta^{\\text{RAPID}}}$ |\n\n\n\nTarget Model:\n\n- $T$ : Number of parameters in the target model.\n- $L$: Long context length.\n\nDraft Model:\n\n- $D$: Number of parameters in the draft model\n- $L^{R}$: The retrieval length of draft LLM input.\n\nSpeculation:\n\n- $\\gamma$: the generation length of draft model per step.\n- $\\beta^{\\text{SD}}$: the expectation of speculative decoding acceptance rate.\n- $\\beta^{\\text{RAPID}}$: the expectation of our RAPID acceptance rate.\n\n---\n\n### 3. The prefill gets delayed compared to the baseline\n\nA3: Yes, the speculative decoding mechanism does introduce extra prefill latency for the draft model. In our settings, however, the extra latency will be marginal (e.g, 26.37s vs 25.89s) as the draft input length is far shorter than the target one. Moreover, the two prefill stages are possible to be further optimized to overlap at the infrastructure level, which has the potential to totally erase the extra latency.\n\n---\n\n### 4. The generation throughput benefits should be clearly highlighted.\n\nA4:  We primarily list the generation throughput speedup in Table 1 and discuss it in Section 4.1. The efficiency benefits are also concluded in the abstract and introduction sections. We apologize for any confusion and are glad to strengthening the presentation of these efficiency benefits in revised version.\n\n---\n\n### 5. The prefill delay overhead should be discussed associated to each component.\n\nA5: The prefill delay overhead in Table 1 of our paper is only related to the prefill of the draft model upon the retrieval input. In addition, the RAG pipeline will introduce avg 1.43s latency per data sample. We will state the latency more clearly in a revised version.\n\n---\n\n### 6. The decode throughput: when it starts to become beneficial compared to the baseline SD, how it is a function of seq length.\n\nA6: In section 4.3 of our paper, we have discussed the impact of context and retrieval length for both performance and efficiency. The conclusions are (1) Our RAPID consistently improve the performance when the seq length > 8K. (When seq len < 8K, it is not really a long context and doesn’t need to apply RAPID) (2) Our RAPID starts to improve efficiency when the seq length is beyond 32k (with retrieval length < 16k). The longer the seq length, the more significant the speedup.\n\n---\n\n### 7. The memory and compute overhead to support the inference-time knowledge transfer.\n\nA7: As all required variables in Eq. (6) are necessary to compute for naive SD, **our RAPID would not introduce any extra memory overhead for the inference-time knowledge.** While the computation is just a tensor add operation, **the extra latency is also very marginal and can be ignored**.\n\n---\n\nWe hope the responses above can address your concerns and contribute to a reconsideration of review score. Looking forward to discussing more with you.\n\nBest,\n\nAuthors", "claims_and_evidence": "**Claim:**  RAPID as an effective decoding method for accelerating long-context inference and, at the same time, enhancing generation quality through retrieval-augmented speculation\n\n**Evidence:** In self-speculation settings (draft LLM of the same size as target LLM), RAPID achieves consistent performance improvements (e.g., 42.83 vs 39.33 on InfiniteBench for LLaMA-3.1-8B) with significant speedup (up to 2.69×) over the long-context target LLMs. The upward-speculation setting (draft LLM bigger than target LLM) further boosts performance improving LLaMA-3.1-8B from 42.83 to 49.98 on InfiniteBench), with comparable efficiency with the smaller long-context target LLMs.", "methods_and_evaluation_criteria": "RAPID is evaluated using LLaMA-3.1 (8B, 70B) and Qwen2.5 (7B, 72B) as target LLMs. The authors implemented two speculation settings: (1) self-speculation, where the RAG drafter matches the target LLM’s scale, and (2) upward-speculation, where RAG drafter is larger than the target LLM. For smaller models (LLaMA-3.1-8B, Qwen2.5-7B), they evaluate both settings, while larger models (LLaMA-3.1-70B, Qwen2.5-72B) they use self-speculation only.\n\nEvaluation is performed on  $\\inf$Bench and  LongBench v2. Efficiency metrics include: (1) prefill time and (2) speedup.", "theoretical_claims": "1. For rejection, they sample from an adjusted residual distribution. This sampling strategy maintains theoretical guarantees. The authors prove in Appx. §B that the resulting tokens follow the same distribution as direct sampling from the original target model.\n\n2. The gradient of the knowledge distillation loss L = T2 · KL(q(x)∥p(x)) with respect to the target LLM output is derived in Appendix A.", "experimental_designs_or_analyses": "RAPID is compared with four baselines (1) target LLM, (2) RAG, where the target LLM generates responses upon retrieval context of draft LLM input in RAPID, (3) naive Speculative Decoding (SD), which involves identical target and draft LLMs with RAPID but using the naive long-context target distribution, (4) MagicDec, with KV cache compression of draft model.", "relation_to_broader_scientific_literature": "The contributions should directly benefit the parallel verifier for SD.", "essential_references_not_discussed": "Please see weakness", "other_strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-written and includes extensive experimental evaluation.\n\n2. The results demonstrate improved performance with enhanced efficiency for long context LLMs. \n\n**Weakness:**\n1. In Figure 1, there is a sudden drop in accuracy for 32K tokens. Is there any possible explanation for this?\n\n2. Generating the retrieval augmented target distribution requires computing target and draft distribution for each speculative token, which increases computational overhead. A FLOPs comparison with baseline should be provided for an estimate of the increased computation. \n\n3. The prefill gets delayed compared to the baseline. \n\n4. The generation throughput benefits should be clearly highlighted.\n\n5. The prefill delay overhead to be discussed in details stating the delay associated to each component \n\n6. The decode throughput: when it starts to become beneficial compared to the baseline SD, how it is a function of seq length\n\n7. The memory and compute overhead to support the inference-time knowledge transfer should be discussed more.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See before", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N7CZHsZVG5", "reviewer": "Reviewer_s5Em", "summary": "The paper presents a novel decoding method called RAPID, designed to enhance the efficiency and quality of long-context inference in large language models (LLMs). RAPID introduces the RAG drafter—a draft LLM operating on shortened retrieval contexts—to speculate on the generation of long context target LLMs. RAPID operates in two settings: self-speculation, where the RAG drafter matches the target LLM's scale, and upward-speculation, where a larger RAG drafter assists a smaller target LLM. Both settings demonstrate effectiveness in improving performance and efficiency. \n\nMain Results\n1. RAPID achieves consistent performance improvements across different model scales and tasks. For example, LLaMA-3.1-8B with RAPID shows a performance increase from 39.33 to 42.83 on InfiniteBench.\n2. RAPID provides significant speedup over long-context target LLMs, with up to 2.69× speedup for LLaMA-3.1-70B.\n3.  RAPID enables effective knowledge transfer from larger RAG drafters to smaller target LLMs, further boosting performance. For instance, LLaMA-3.1-8B with a 70B RAG drafter achieves a performance of 49.98 on InfiniteBench.\n\n## update after rebuttal\nMy primary concern was regarding the resource consumption and robustness of RAPID. The author's rebuttal has alleviated my concerns, so I will increase my rating from 3 to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer s5Em,\n\nWe sincerely appreciate your thorough review of our paper. Your constructive feedback will help us strengthen this work. Below are our responses to your concerns:\n\n---\n\n### 1. The paper does not introduce entirely new theoretical frameworks but rather combines existing ideas in a way.\n\nA1: We believe big ideas are always composed of many small and existing ideas. While individual components (speculative decoding, RAG, long-context LLMs) may be familiar,  our contribution lies in the novel integration of RAG drafter for long-context target models using speculative decoding, with theoretically guaranteed inference-time transfer to combine both benefits from long-context LLMs and RAG. Our method is not an extension of any previous works, but provides a new perspective and solution for a long-time debate regarding “long-context LLMs or RAG”, which is meaningful for the application of long-context LLMs.\n\n---\n\n### 2. The upward-speculation setting requires additional computational resources (extra GPUs) to serve the larger RAG drafter.\n\nA2: We agree and especially point out that a larger RAG drafter will introduce more latency and require extra GPUs to serve. However, our RAPID can operate on two modes: (1) self-speculation for low-resource scenarios and (2) upward-speculation for high-resource scenarios. For a low-resource scenario, the self-speculation of RAPID can work well with both improved performance and efficiency and without additional resource requirements, which demonstrates great potential in real applications. \n\nWhile we believe **the upward speculation of RAPID can serve as a turbo mode, which maximizes performance with comparable generation speed, but consumes more resources.**\nFor example, our RAPID enables LLaMA-3.1-8B (LC) with LLaMA-3.1-70B drafter to achieve 10 points accuracy gains with a comparable generation throughput to naive LLaMA-3.1-8B (LC), which allows us to explore the higher-bound performance wth a similar level of speed but more GPUs.\n\n---\n\n### 3. Although the paper demonstrates robustness to suboptimal retrieval contexts, the method's performance can still be influenced by the quality of retrieval.\n\nA3: We agree that RAPID’s performance can still be influenced by the quality of retrieval.  We are not only demonstrating the robustness of RAPID to retrieval quality in Section 4.5, but also seek to highlight that **our inference-time knowledge transfer will not hinder the target LLM from utilizing its long-context capabilities to reject low-quality candidates from RAG drafter as long as $\\eta$ is properly set.** Our adjusted resampling distribution in Eq. (10) for rejected candidates guarantees that **the resampled tokens follow the exact distribution as direct sampling from the target model.** In other words, our RAPID can guarantee the lower-bound performance at the level of the target LLM as long as $\\eta$ is properly set, while the retrieval quality only affects the performance “gains” and will not introduce too many drops.\n\n---\n\n### 4. Missed Reference.\n\nA4: Thanks for pointing it out. REST[1] proposed directly selecting possible continuation by retrieving from a built datastore rather than generating with a draft LLM. This work should be conceptually related to our work, but the method is distinct and orthogonal. We will discuss the mentioned works (and further) in a revised version.\n\n[1]  [REST: Retrieval-Based Speculative Decoding](https://aclanthology.org/2024.naacl-long.88/) (He et al., NAACL 2024)\n\n---\n\nWe hope the responses above can address your concerns and contribute to a reconsideration of review score. We also appreciate your careful review to point out missed references, which will be fixed in a revised version. Looking forward to discussing more with you.\n\nBest,\n\nAuthors", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. \n1. Proof of Theorem 1 (Gradient of Distillation Loss)  The application of the chain rule, expansion of the log probability, and simplification using the Kronecker delta appear correct.\n2. Proof of Correctness for RAPID's Residual Distribution. The proof logically demonstrates that the residual distribution ensures the overall sampling process still follows the target distribution.", "experimental_designs_or_analyses": "In conclusion, the experimental designs and analyses in the paper are sound and valid. They appropriately address the research questions, use relevant benchmarks and metrics, and provide comprehensive evaluations that support the claims made about RAPID's effectiveness for long-context inference in LLMs.", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "1. [REST: Retrieval-Based Speculative Decoding](https://aclanthology.org/2024.naacl-long.88/) (He et al., NAACL 2024)", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a novel integration of speculative decoding with retrieval-augmented generation (RAG), creating a new paradigm for efficient long-context inference\n2. The paper demonstrates significant speedups (over 2× in self-speculation settings) while maintaining or improving generation quality. \n3. The method's effectiveness across different model scales (from 8B to 72B parameters) and diverse benchmarks suggests that it can be widely applied to various LLM architectures and tasks, enhancing its practical significance.\n4. The paper is organized logically, with clear explanations of the methodology, experimental setup, and results.\n\nWeaknesses:\n1. The paper does not introduce entirely new theoretical frameworks but rather combines existing ideas in a way.\n2. The upward-speculation setting requires additional computational resources (extra GPUs) to serve the larger RAG drafter. \n3. Although the paper demonstrates robustness to suboptimal retrieval contexts, the method's performance can still be influenced by the quality of retrieval", "other_comments_or_suggestions": "No", "questions_for_authors": "See Other Comments Or Suggestions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XVVkpBnRMU", "reviewer": "Reviewer_SMJY", "summary": "To enhance the efficiency and effectiveness in long-context scenarios, the paper proposes a method called Retrieval-Augmented Predictive Decoding (RAPID), which aims to address the decline in efficiency and quality of traditional speculative decoding due to memory limitations in long-context reasoning. RAPID integrates RAG , selectively retrieving compressed context  from long documents to generate candidate tokens. This approach reduces computational overhead while maintaining information relevance. Additionally, RAPID incorporates a knowledge distillation mechanism, transferring the knowledge of the RAG Drafter to the target model to form an enhanced target distribution. This not only increases the acceptance rate of high-quality candidates but also maintains theoretical reliability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SMJY,\n\nWe sincerely appreciate your thorough review of our paper. Your constructive feedback will help us strengthen this work. Below are our responses to your concerns:\n\n---\n\n### 1. Missed References.\n\nA1: Thanks for pointing it out. REST[1] proposed selecting possible continuation by retrieving from a built datastore rather than generating with a draft LLM. TRIFORCE[2] introduced KV Cache compression for draft LLM based on chunk-wise similarity, which we have discussed in the related work and compared with in the response to Reviewer KbZ7. Ouroboros[3] sought to produce longer and more acceptable candidates from draft LLM per step based on draft phrases. Speculative RAG [4] did not utilize speculative decoding, but proposed a parallel draft-then-verify mechanism to improve RAG quality. **These works may be conceptually related to our work, but our method is quite distinct and orthogonal to them.** We will discuss the mentioned works (and further) in a revised version.\n\n[1] REST\n\n[2] TRIFORCE\n\n[3] Ouroboros\n\n[4] Speculative RAG\n\n---\n\n### 2. Overhead of each component.\n\nA2: Unlike regular RAG pipeline, which builds indexes for a large external corpus (hundreds of millions of documents), we only index/retrieve the chunks for the input long context (<128K) on-the-fly during inference. Therefore, **the RAG component latency in our method will become marginal compared to the inference latency over long context.** We now list the latency of each component of our RAPID on LongBench v2 (Long, CoT) with LLaMA-3.1-8B below:\n\n|  | RAG pipeline time (avg) | Prefix Time (avg) | Generation Time (avg) |\n| --- | --- | --- | --- |\n| LLaMA-3.1-8B-RAPID (Self-Spec) | 1.43 | 26.37 | 32.25 |\n| LLaMA-3.1-70B-RAPID (Self-Spec) | 1.43 | 163.43 | 121.76 |\n\n---\n\n### 3. Potential Bias of the RAG Drafter.\n\nA3: In section 4.5 of our paper, we have discussed the robustness of our RAPID to the retrieval quality. The results indicate that **our RAPID can maintain a stable (or better) performance even when the retrieval text is irrelevant to the long context**. This demonstrates the robustness of RAPID to retrieval quality, which is because the long-context target model in target distribution (Eq. 6) will preserve the long-context ability to verify candidates effectively, while RAG drafter provides benefits rather than reconstruction.\n\n---\n\n### 4. Explanation regarding upward speculation (using larger LLMs as drafter).\n\nA4: **For the rationality**, as the RAG based on compressed context introduces bias, even the superior LLMs may still lose some crucial information. As shown in Fig 2 of our paper, LLaMA-3.1-70B (RAG) on LongBench v2 achieves 23.66 points gain with another 13.72 points drop compared to LLaMA-3.1-8B (LC). **Despite the final gains of LLaMA-3.1-70B (RAG) over LLaMA-3.1-8B (LC) approach 10 points, there is still a large proportion of samples that cannot be solved by the superior LLaMA-3.1-70B (RAG).** However, our RAPID can integrate benefits from both long-context target LLM and stronger RAG drafter, **achieving performance improvements by incorporating the gains from drafter with minimum extra drops**.\n\n**For the efficiency**, larger RAG drafter will introduce more latency and require extra GPUs to serve. However, **our RAPID allows LLaMA-3.1-8B (LC) with LLaMA-3.1-70B drafter to achieve a comparable generation throughput to naive LLaMA-3.1-8B (LC)**. Empirically, the upward speculation of RAPID can serve as a turbo mode, which maximizes performance with comparable generation speed but consumes more resources. For a low-resource scenario, the self-speculation of RAPID can work well with both improved performance and efficiency and without additional resource requirments.\n\n---\n\n### 5. Use models from another family\n\nA5: Thanks for the insightful question. We now conduct a pilot experiment that uses LLaMA-3.1-8B-Instruct (LC) target LLM and Qwen2.5-7B-Instruct RAG drafter for RAPID. We convert the draft logit to target logit space and cut off the probability mass which is mismatched. The implementation is inspired by https://github.com/huggingface/transformers/blob/786d9c5ed920a099573ea7b6dbf265f1aeb32fc0/src/transformers/generation/candidate_generator.py#L783\n\nSurprisingly, using Qwen2.5-7B as RAG drafter can further improve the performance (though lower speedup due to logit alignment overhead). The gains may be due to the better short-context abilities of Qwen2.5-7B compared to LLaMA-3.1-8B upon RAG context. This indicates the essentiality of LLM capability in RAPID and opens more potential exploration of LLM combinations. Thanks for the constructive discussion.\n\n| Target | Draft | Overall | Overall (CoT) | Speedup |\n| --- | --- | --- | --- | --- |\n| LLaMA-3.1-8B | LLaMA-3.1-8B | 32.4 | 34.2 | 2.10 |\n| LLaMA-3.1-8B | Qwen2.5-7B | 34.0 | 34.8 | 1.81 |\n\n---\n\nWe hope the responses above can address your concerns and contribute to a reconsideration of review score.\n\nBest,\n\nAuthors", "claims_and_evidence": "Yes, all claims made in the submission are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are well-aligned with the problem of long-context inference acceleration. And further detailed evaluation is discussed in  Appendix C.", "theoretical_claims": "Theoretical claims are supported by proofs in the appendix.\n1. In Appendix A, the gradient derivation for knowledge distillation loss is correct, showing the logits shift from RAG drafter to target model aligns with distillation principles.\n2. Appendix B discuss the  Correctness of RAPID’s Residual Distribution, proving RAPID’s sampling maintains the target distribution.", "experimental_designs_or_analyses": "In the main experiment 4.1, the author conducted a detailed analysis of the efficiency of RAPID, demonstrating a 2X performance improvement under long context conditions. However, in the analysis section, there is a lack of discussion on the breakdown of time consumption. In particular, it appears that the chunking of context and the construction of indexes are both carried out during the online process, which also consumes a significant amount of time. It is unclear whether this part is included in the overall time calculation and what proportion it accounts for. This issue has a significant impact on the evaluation of RAPID's efficiency.", "relation_to_broader_scientific_literature": "RAPID builds on prior work in speculative decoding, retrieval-augmented generation, and long-context optimization. Key connections include:\n1. SD Limitations: RAPID addresses SD’s inefficiency in long contexts by replacing the draft model with a RAG drafter.\n2. RAG Integration: By leveraging RAG’s context compression, RAPID avoids KV cache bottlenecks .\n3. Knowledge Transfer: The retrieval-augmented target distribution aligns with distillation techniques , enabling upward-speculation.", "essential_references_not_discussed": "The author's core contribution is the RAG+SD collaborative method. However, this is not an entirely new idea. This paper is not the first to propose the RAG+SD method, yet the author fails to discuss related work in the Introduction and the Related Work sections.\n\nFor example:\n\n- REST: Retrieval-Based Speculative Decoding, published in NAACL 2024.\n- TRIFORCE: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding, published in COLM 2024.\n- Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding, published in EMNLP 2024.\n- Speculative RAG: Enhancing Retrieval-Augmented Generation through Drafting, published in ICLR 2025.", "other_strengths_and_weaknesses": "**Strength:**\n\n- The author's choice of research question is highly valuable, and the approach is somewhat innovative. Numerous studies have demonstrated the respective strengths and weaknesses of RAG and direct LLM responses in long-context scenarios. This paper proposes a method to integrate the two in long-context tasks, balancing efficiency and effectiveness.\n- The work extends the research on speculative decoding . In traditional SD studies, a smaller-parameter model is typically used as the Drafter. The author proposes using a model with an equivalent or even larger parameter scale but with a smaller context as the Drafter, and introduces knowledge transfer. To some extent, this integrates the idea of knowledge distillation into SD research.\n\n**Weakness:**\n\n- As mentioned in the section on Essential References Not Discussed, there has already been much work on combining RAG and SD. The author's review of related work in this area is not comprehensive, and there is a lack of targeted comparisons. This makes it difficult to fully assess the novelty and innovation of RAPID.\n- From a performance perspective, the pipeline is relatively long, and efficiency is crucial. The overhead of each component (e.g., Long-context RAG, Drafter, Target LLM) needs to be better demonstrated in terms of practicality, which will determine the overall applicability of this paper.\n- Potential Bias of the RAG Drafter. The candidates generated by the RAG Drafter may be overly reliant on retrieved passages, excessively pruning contextual information and potentially losing too much detail. This could lead to locally optimal rather than globally optimal generated content. The performance of RAPID is highly dependent on retrieval quality, yet the paper does not sufficiently discuss the limitations of the retrieval module and the long-term impact of such biases on long-context reasoning.", "other_comments_or_suggestions": "The overall writing is smooth, but the abbreviations should have their first letters capitalized. For example, the full form of RAG is not consistent throughout the text.", "questions_for_authors": "1. The distinction, rationality, and overhead of using a larger model as the Drafter compared to traditional Speculative Decoding require additional explanation. Using a larger model to enhance and accelerate a smaller model, especially when the smaller model is responsible for validating the speculative tokens generated by the Drafter, raises several questions. If the RAG Drafter is more powerful, would such validation not have a negative impact? On the other hand, when using a larger Drafter for inference, the inference speed itself is not fast. Is the speed gain actually derived from the smaller context? I hope the author can provide a more detailed description of the motivation behind this approach.\n\n\n2. Recent studies[1,2] have shown that in long-context scenarios, RAG does not always improve the performance of LLMs, especially with more powerful models. Similar conclusions are also validated in this paper. Can the author provide further analysis, such as case studies, to explain the reasons for the performance gains in RAPID? Is it because RAG retrieval makes the information in long-context scenarios more focused? Conversely, in which scenarios do the bad cases occur?\n    \n3. When performing inference knowledge transfer, is it necessary to use models from the same family (e.g., Llama series and Qwen series)? If not, that is, if there is a significant difference in token distribution between the teacher model and the student model, will the same effect be achieved or will the gain be weakened? I hope to see a discussion from the author on the generalizability of this approach.\n\nReference:\n\n [1] U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack \n\n [2] LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YYfEYFWTzG", "reviewer": "Reviewer_KbZ7", "summary": "This work proposes RAPID, a variation on the typical speculative decoding framework for long-context tasks by using relatively large draft models that use RAG to compress the context. The quality and performance of RAPID is further boosted by using a “retrieval-augmented target distribution” which modifies the target model’s output distribution by multiplying the original target model logits by the difference of the draft and target logit distributions; the influence of the draft model logits is controlled with a hyperparameter, $\\eta$. The benefits of RAPID for long-contexts are demonstrated with models from two model families and two long-context benchmarks. RAPID achieves the highest accuracy and overall throughput increases compared to the other baselines examined.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KbZ7,\n\nWe sincerely appreciate your thorough review of our paper. Your constructive feedback will help us strengthen this work. Below are our responses to your concerns:\n\n---\n\n## 1. Comparison with LC models like Qwen-1M\n\nWe've evaluated RAPID on Qwen2.5-7B-1M (released post-submission). Results show RAPID on Qwen2.5-7B-Instruct is comparable to Qwen2.5-7B-1M (35.4 vs 35.6), while RAPID on Qwen2.5-7B-1M further improves both efficiency and performance:\n\n|  | Overall (CoT) | Speedup |\n| --- | --- | --- |\n| Qwen2.5-7B-1M | 35.6 | 1 |\n| - RAPID ($\\eta = 5$) | 38.4 | 2.01 |\n\n---\n\n## 2. Comparison with Triforce\n\nTriforce wasn't included because **it can't be directly applied to new LLMs with GQA**. We've now conducted comparisons on LWM-Text-Chat-128K (based on LLaMA2-7B), setting retrieval budget at 4096, chunk size at 8, and draft cache budget at 256 for Triforce:\n\n|  | Overall (CoT) | Speedup |\n| --- | --- | --- |\n| LWM-Text-Chat-128K | 18.4 | 1 |\n| - Triforce | 18.0 | 1.27 |\n| - RAPID | 21.6 | 2.56 |\n\n**While Triforce achieves efficiency gains, RAPID demonstrates more significant speedup and performance improvements**. Triforce recalls information based on chunk-wise attention scores, but higher attention scores don't necessarily indicate greater semantic relevance (e.g., initial tokens often attract high attention as \"sinks\" despite lacking semantic importance [1]). Our RAD drafter better recalls semantically relevant information, hence achieving higher acceptance rate and speedup for more challenging tasks.\n\n[1] Efficient Streaming Language Models with Attention Sinks\n\n---\n\n## 3. Additional Results for MagicDec and SD\n\nWe haven’t included results of MagicDec and SD on more models since they achieve quite similar efficiency gains (significantly below our method) across models, while they aren’t designed for performance improvement. We now include more results below.\n\n|  | Overall (CoT) | Speedup |\n| --- | --- | --- |\n| Qwen2.5-7B-SD | 29.2 | 1.83 |\n| -MagicDec | 30.0 | 0.71 |\n| Qwen2.5-70B-SD | 43.7 | 1.59 |\n| -MagicDec | 43.5 | 0.65 |\n| LLaMA3.1-70B-SD | 35.3 | 1.75 |\n| -MagicDec | 34.8 | 0.73 |\n\n---\n\n## 4. Comparison with Sparse Attention (MInference)\n\nPer your suggestion, we've compared with MInference on LLaMA-3.1-8B. Results show **MInference achieves impressive prefill speedup, while RAPID demonstrates significant performance and decoding throughput advantages**:\n\n|  | Overall (CoT) | Prefill Time | Speedup |\n| --- | --- | --- | --- |\n| LLaMA-3.1-8B | 30.4 | 25.89 | 1 |\n| -MInference | 30.9 | **9.10** | 0.62 |\n| -RAPID | 34.2 | 26.37 | **2.10** |\n\nWe believe sparse attention is orthogonal to our work and combining it with RAPID holds potential for future research.\n\n---\n\n## 5. Retrieval Hyperparameter Settings\n\nWe haven't extensively tuned hyperparameters except for $\\eta$, as our goal is to propose a generally effective method rather than overfitting benchmarks. The RAG hyperparameters only affect retrieval quality, and RAPID has demonstrated such robustness (Section 4.5). To clarify our selection criteria:\n\n- **Cosine similarity threshold (0.3)** was selected from {0.1, 0.2, 0.3, 0.4, 0.5} for RAG on LLaMA-3.1-8B, with overall scores of {28.8, 29.2, 29.2, 29.0, 29.0}.\n- **Compression ratio** of 24 (120K/5K) was chosen as retrieval length beyond 5K showed no significant benefits.\n\nFor draft tokens per step, we set 10 without tuning. Our ablation study below shows RAPID maintains stable performance with <15 draft tokens, though excessive tokens may reduce performance gains despite increasing throughput.\n\n| # candidates | Overall (CoT) | Speedup |\n| --- | --- | --- |\n| 5 | 34.0 | 1.95 |\n| 10 | 34.2 | 2.1 |\n| 15 | 34.4 | 2.24 |\n| 20 | 32.8 | 2.51 |\n\n---\n\n## 6. Robustness Analysis Intuition\n\nThe robustness analysis in Section 4.5 was conducted on LongBench v2 (Long, CoT) subset, which involves generating reasoning paths before providing answers. Given some preliminary chains,  we believe a strong drafter is capable of generating continual CoTs with higher quality to introduce the performance gains. In our LLaMA-3.1-8B target / 70B draft analysis, acceptance rates significantly increased after generating 32 tokens despite irrelevant retrieval context, supporting this intuition.\n\n---\n\n## 7. Generation Quality Analysis\n\nOur generation quality analysis (Section 4.4) is a pilot experiment demonstrating RAPID's potential effectiveness in real-world applications. We acknowledge that real-world long-context conversations typically contain similar themes/topics, and hope for robust benchmarks with better evaluation metrics beyond LLM-as-judge in the future.  (Human evaluation is both expensive and logistically complex at scale.)\n\n---\n\nWe hope the responses above can address your concerns and contribute to a reconsideration of review score. We also appreciate your careful review to point out some typos/missed references, which will be fixed in a revised version.\n\nBest,\n\nAuthors", "claims_and_evidence": "* RAPID claims to outperform both naive SD and MagicDec. However, only Llama-8B is used to establish this relationship. It would be more convincing to include the other models examined. \n* The authors claim that TriForce results in “weakend draft models”. As TriForce is not used as a baseline in the paper, this is an unsubstantiated claim. Notably, TriForce’s second tier retrieval cache based drafter is similar to RAPID’s use of RAG for drafting. \n* Otherwise, the claims made appear to be well founded.", "methods_and_evaluation_criteria": "Generally, the methods and evaluation criteria are sensible for the method proposed.", "theoretical_claims": "I did not review the proofs in the supplementary materials. The claims regarding the proposed retrieval-augmented target distribution appear to be sensible.", "experimental_designs_or_analyses": "* My primary concern with the experimental design and analyses is the lack of baseline comparisons across all models.\n* Further, additional baseline models should be considered such as specialized long-context models such as Qwen-1M and alternative approaches to accelerating long-context generation such as sparse attention (MInference or similar). While I believe the experimental results are still valuable without these direct comparisons, more robust baselines would better convince me that RAPID offers advantages over these competing methods. \n* The ablation studies are somewhat limited. While the authors analyze the impact of context and retrieval length and the effect of retrieval length, I think additional ablations would be helpful. For example, it would be useful to see the impact of different retrieval strategies or different choices of the draft model or how sensitive the cosine similarity threshold is to the overall methods performance and quality. \n* The generation quality experiment relies on LLM-as-a-Judge evaluation using GPT-4 Turbo. This evaluation method is subjective and can be unreliable. An additional judge or human-verification would improve reliability of these results. \n* The generation quality analysis also relies on a synthetic dataset in which unrelated dialogs are inserted into a multi-turn chat context. This seems like a setting in which RAG would be disproportionality well suited to given that the unrelated dialogs would have low similarity to the context of interest. This is distinct from real-world long contexts which typically contain similar themes / topics throughout.", "relation_to_broader_scientific_literature": "* Accelerating long-context generation is a topic that has seen significant interest of late. MagicDec and TriForce are the leading examples which approach the problem from the perspective of speculative decoding. Competing methodologies such as sparse attention for prefill (MInference etc.) and sparse decoding have also seen significant interest. \n* The combination of speculative decoding and RAG appear for long-context generation is related to but distinct from other recent approaches that rely on approximated attention using truncated queries (ie.., the last 64 tokens in a long prompt) or compressed / pooled k/v vectors (RetrievalAttention, Quest). \n* With respect to prior literature, I believe the most novel contribution is the proposed retrieval-augmented target distribution. Combining RAG with the draft model is closely related to TriForce and MagicDec in which the draft model uses a compressed KV cache of some kind. \n* This work shares some similarity with [1] but is motivated specifically for long-context generation and uses a distinctly different approach. \n\n[1] https://arxiv.org/abs/2407.08223", "essential_references_not_discussed": "In my opinion [2] should be included as a seminal work that inspired much of the following speculative decoding literature. \n\n[2] M. Stern, N. Shazeer, and J. Uszkoreit, “Blockwise Parallel Decoding for Deep Autoregressive Models,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2018.", "other_strengths_and_weaknesses": "## Strengths:\n* Important and timely topic\n* Strong empirical results, particularly with upward-speculation in which the Llama-8B target + 70B drafter outperforms the 70B LC model both in terms of quality and performance. \n* The proposed retrieval-augmented target distribution is novel and appears to address one of the main pitfalls of speculative decoding in that the target model is assumed to represent the ground truth for the drafter. \n* The paper provides a good combination of algorithmic development, theoretical analysis, and experimental evaluation.\n\n## Weaknesses:\n* RAPIDs has several hyperparameters: the number of draft tokens per round, $\\eta$, the cosine similarity threshold for retrieval, the retrieval length, and the compression ratio. It’s not clear how each of these values were selected nor how sensitive the method is to each of these. Only $\\eta$ is studied in detail and in a somewhat unrelated setting to the main results since an unrelated context is used. \n* Additional baselines should be considered to compare RAPIDs to other speculative decoding methods like TriForce and other approaches in the long-context literature such as sparse attention.", "other_comments_or_suggestions": "* L065: I believe this cross reference should be for Figure 3 not 1? \n* L074: Self speculation is not a novel contribution of this work and should be removed from the claim that the work introduces it as a “new paradigm”. \n* L086: I believe this should be DRAM not SRAM. The memory i/o latency bottleneck is typically from DDR memory to the streaming multiprocessor rather than from shared memory. \n* Table 1 first row of Qwen-72B results should have no shading, appears light pink. \n* L328: “Infernce”", "questions_for_authors": "1. How does RAPID compare with specialized LC models such as Qwen-1M? How does RAPID compare with TriForce? What are the results for SD and MagicDec on the other models not reported? This is the key question to answer to improve my score. \n2. How were the following hyperparameters selected: the number of draft tokens per round, the cosine similarity threshold for retrieval, retrieval length, and the compression ratio. How sensitive is RAPID to changing these parameters? \n3. The robustness analysis in Section 4.5 is surprising. I would have expected a larger degradation in quality given that completely unrelated context is used. Could the authors speculate on where the latency / accuracy gains are coming from for these results given that the draft model is preventing from attending the correct context?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces RAPID, a novel integration of Retrieval-Augmented Generation (RAG) with Speculative Decoding (SD) for efficient long-context inference in LLMs. All reviewers found the problem timely and the method promising. The authors provided extensive empirical evidence on multiple model families and benchmarks, demonstrating improvements in both performance and efficiency. The theoretical justifications were also sound.

Several reviewers initially raised concerns about baselines, overhead, and novelty relative to prior work like TRIFORCE and REST. The authors provided a very thorough rebuttal, which addressed these concerns convincingly. All reviewers acknowledged the responses, and all four increased their scores, resulting in a consensus recommendation of "Accept".

Given the strong empirical results, clear methodological contributions, and effective rebuttal engagement, I believe this paper will be a valuable contribution to the conference and the broader community interested in long-context LLM efficiency.
Decision: Accept (spotlight poster)