Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024)

Abstract: To mitigate the high inference latency stemming from autoregressive decoding in Large Language Models (LLMs), Speculative Decoding has emerged as a novel decoding paradigm for LLM inference. In each decoding step, this method first drafts several future tokens efficiently and then verifies them in parallel. Unlike autoregressive decoding, Speculative Decoding facilitates the simultaneous decoding of multiple tokens per step, thereby accelerating inference. This paper presents a comprehensive overview and analysis of this promising decoding paradigm. We begin by providing a formal definition and formulation of Speculative Decoding. Then, we organize in-depth discussions on its key facets, such as drafter selection and verification strategies. Furthermore, we present a comparative analysis of leading methods under third-party testing environments. We aim for this work to serve as a catalyst for further research on Speculative Decoding, ultimately contributing to more efficient LLM inference.1

heading: 1 Introduction
text: Large Language Models (LLMs) have achieved remarkable proficiency in a range of downstream tasks (OpenAI, 2023; Touvron et al., 2023a,b; Chiang et al., 2023; Jiang et al., 2023). They are progressively evolving as the cornerstone of comprehensive API interfaces (e.g., ChatGPT2), offering human life services and guidance through real-time human-machine interactions. However, the inference latency of these sizable models has emerged as a substantial obstacle restricting their broader applications. This latency primarily arises from the token-by-token generation necessitated by autoregressive decoding, resulting in an escalation of the inference latency with both the length of the generated sequence and the model’s scale.
1The relevant papers will be regularly updated at https: //github.com/hemingkx/SpeculativeDecodingPapers.
2https://chat.openai.com
To accelerate LLM inference, an innovative inference paradigm, Speculative Decoding has been introduced (Stern et al., 2018; Xia et al., 2023; Leviathan et al., 2023; Chen et al., 2023a). As shown in Figure 1, in each decoding step, Speculative Decoding first efficiently drafts multiple tokens as speculation of future decoding steps of the target LLM and then utilizes the LLM to verify all drafted tokens in parallel. Only those tokens that meet the LLM’s verification criterion are accepted as final outputs to guarantee generation quality.
Speculative Decoding is founded upon two key observations about LLM inference: 1) many easy tokens can be predicted with less computational overhead (e.g., using a smaller model), and 2) LLM inference is highly memory bandwidth bound (Patterson, 2004; Shazeer, 2019) with the main latency bottleneck arising from memory reads/writes of LLM parameters rather than arithmetic computations. Drawing on these observations, Speculative Decoding adapts the concept of speculative execution3 to focus LLMs’ efforts on the validation of
3Speculative execution (Burton, 1985; Hennessy and Patterson, 2012) is an optimization technique used in computer architecture where tasks are performed in advance and subsequently verified for their necessity, thereby circumventing the delays inherent in sequential task execution.
7655
pre-drafted tokens, substantially diminishing the need for frequent memory operations of LLM parameters, thereby improving inference efficiency.
While Speculative Decoding shows promise, it raises several critical questions that warrant further investigation. For instance, how to design an optimal drafter to strike a balance between speculation accuracy and drafting efficiency (Xia et al., 2023; Zhou et al., 2023; Li et al., 2024). Additionally, it is essential to assess whether the verification criterion can maintain both generation parallelism and output quality (Miao et al., 2024; Cai et al., 2024). Furthermore, since existing methods are evaluated under disparate testing conditions, a unified benchmark is needed to facilitate realistic speedup expectations within the research community.
Amid the rapid expansion of research in Speculative Decoding, this work makes the first attempt to present a survey of this field, aiming to raise awareness among academics about the latest advancements. We provide a systematic categorization of current research and an in-depth analysis of relevant studies. Moreover, we introduce Spec-Bench, a comprehensive benchmark to assess Speculative Decoding methods in diverse application scenarios. Our contributions can be summarized as follows:
(1) First survey: To our knowledge, we are the first to present a comprehensive survey on Speculative Decoding; (2) Formal definition: We furnish a formal definition and formulation of Speculative Decoding, laying the groundwork for future research. (3) New taxonomy: We provide a systematic taxonomy for Speculative Decoding, offering an organized categorization of existing work. (4) Spec-Bench: We introduce Spec-Bench, an extensive benchmark designed for assessing Speculative Decoding, enabling a comparative evaluation of leading methodologies.
We hope that this work can serve as an essential guide for newcomers and motivate future research.

heading: 2 Overview
text: This paper offers a comprehensive survey of Speculative Decoding. We commence by introducing the early stages of Speculative Decoding research (§3), illustrated by a timeline of its evolution (as shown in Figure 2). This is followed by a formal definition and formulation of Speculative Decoding (§4). Then, we delve into a detailed discussion of leading techniques, including the selection of draft models (§5), verification strategies (§6), and alignment between the drafter and the target LLM (§7). Moreover, we introduce Spec-Bench, an extensive evaluation benchmark designed for assessing the acceleration effect of Speculative Decoding (§8).

heading: 3 Evolution of Speculative Decoding
text: This section discusses the motivation behind Speculative Decoding (§3.1) and then provides a detailed introduction to early attempts in this field (§3.2).

heading: 3.1 Motivation
text: The widespread adoption of LLMs has established autoregressive decoding as the de facto standard to LLM inference (Chowdhery et al., 2023; OpenAI, 2023; Jiang et al., 2024). However, autoregressive decoding is limited by its inference latency, which primarily stems from the memory-bound computation of LLMs (Patterson, 2004; Shazeer, 2019). Specifically, the main latency bottleneck of each decoding step is not due to computational operations but arises from the necessity to transfer all LLM parameters from High-Bandwidth Memory (HBM) to the on-chip cache of modern accelerators like GPUs. This process, which generates only one token per step, leads to the underutilization of these accelerators and results in inefficiencies.
Algorithm 1 Autoregressive Decoding Require: Language modelMq , input sequence x1, . . . , xt,
and target sequence length T ; 1: initialize n← t 2: while n < T do 3: Set qn+1 ←Mq (x | x<n+1) 4: Sample xn+1 ∼ qn+1 5: n← n+ 1 6: end while
3.2 Pioneering Draft-then-Verify Efforts
To mitigate the above issue, an intuitive way involves leveraging idle computational resources to enhance parallelism in LLM inference. To this end, Stern et al. (2018) introduced Blockwise Decoding, an approach that incorporates extra feedforward neural (FFN) heads atop the Transformer decoder, enabling the simultaneous drafting of multiple tokens per step. These tokens are then verified by the original LLM in parallel, ensuring that the outputs align with those of the original LLM. As a pioneering work proposing the Draft-then-Verify paradigm, Blockwise Decoding effectively reduces the number of required LLM calls by increasing generation parallelism, thereby accelerating inference.
To further unleash the potential of this paradigm, Xia et al. (2023) introduced Speculative Decoding (SpecDec), which utilizes an independent drafter, notably a specialized Non-Autoregressive Transformer, to perform the drafting task both accurately and efficiently. Moreover, this method presented an innovative strategy that relaxes the rigid verification criterion, thereby increasing the acceptance rate of drafted tokens. Impressively, SpecDec achieves around 5× speedup over autoregressive decoding with comparable quality, underscoring the substantial potential of Speculative Decoding.
Following SpecDec, Leviathan et al. (2023) and Chen et al. (2023a) made concurrent contributions by proposing Speculative Sampling, expanding this paradigm to encompass the lossless acceleration of various sampling methods. These approaches employed smaller LMs from the same series (e.g., T5-small) to speed up the inference of their larger counterparts (e.g., T5-XXL). Unlike previous work, these off-the-shelf small LMs do not require additional training, enabling the rapid adoption of Speculative Decoding in LLM acceleration. This advancement has elevated Speculative Decoding to the forefront of LLM efficiency research, attracting widespread interest within the NLP community.
To sum up, these pioneering efforts in Specula-
Algorithm 2 Speculative Decoding Require: Target language modelMq , draft modelMp, in-
put sequence x1, . . . , xt, block size K, target sequence length T , drafting strategy DRAFT, verification criterion VERIFY, and correction strategy CORRECT;
1: initialize n← t 2: while n < T do // Drafting: obtain distributions fromMp efficiently 3: Set p1, . . . , pK ← DRAFT (x≤n,Mp) // Drafting: sample K drafted tokens 4: Sample x̃i ∼ pi, i = 1, . . . ,K // Verification: compute K+1 distributions in parallel 5: Set qi ←Mq (x | x≤n, x̃<i) , i = 1, . . . ,K + 1 // Verification: verify each drafted token 6: for i = 1 : K do 7: if VERIFY (x̃i, pi, qi) then 8: Set xn+i ← x̃i and n← n+ 1 9: else 10: xn+i ← CORRECT (pi, qi) 11: and Exit for loop. 12: end if 13: end for 14: If all drafted tokens are accepted, sample next token
xn+1 ∼ qK+1 and set n← n+ 1. 15: end while
tive Decoding have gradually solidified the Draftthen-Verify paradigm, showcasing its promising potential in LLM acceleration. We provide a detailed categorization and discussion of these studies and subsequent research in the following sections.

heading: 4 Formulation and Definition
text: In this section, we first provide a concise overview of standard autoregressive decoding (§4.1). Then, we offer an in-depth exposition of Speculative Decoding (§4.2), which encompasses a formal definition, a comprehensive description of the methodology, and a detailed elaboration of the algorithm.

heading: 4.1 Autoregressive Decoding
text: Transformer-based LLMs typically make generations in an autoregressive manner. Given an input sequence x1, . . . , xt, an autoregressive language modelMq generates the next token according to:
xt+1 ∼ qt+1 =Mq (x | x<t+1) , (1)
where q is the conditional probability distribution calculated byMq and xt+1 denotes the next token sampled from qt+1. We illustrate a detailed process in Algorithm 1.
As discussed in Section 3, while the standard autoregressive decoding offers desirable generation quality, it is bounded by memory bandwidth, resulting in low utilization of modern accelerators. In this process, each memory-bound LLM call (i.e., an LLM forward step) produces merely a single
token for the entire sequence, making the whole generation inefficient and time-consuming.

heading: 4.2 Speculative Decoding
text: Following Xia et al. (2023), Leviathan et al. (2023), and Chen et al. (2023a), we here provide a formal definition of Speculative Decoding:
Speculative Decoding is a Draft-thenVerify decoding paradigm in which, at each decoding step, it first efficiently drafts multiple future tokens and then verifies all these tokens in parallel using the target LLM to speed up inference.
We formulate a detailed Speculative Decoding process in Algorithm 2. Subsequently, we delve into the two fundamental substeps integral to this paradigm – drafting and verification:
Drafting At each decoding step, Speculative Decoding first efficiently drafts multiple future tokens, as a speculation of the target LLM’s output. Formally, given an input sequence x1, . . . , xt and the target LLM Mq, this paradigm employs an efficient draft modelMp (e.g., a smaller LM) to decode the next K drafted tokens:
p1, . . . , pK = DRAFT (x≤t,Mp) , x̃i ∼ pi, i = 1, . . . ,K,
(2)
where DRAFT(·) denotes various drafting strategies that we will discuss in Section 5, p is the conditional probability distribution calculated byMp, and x̃i denotes the drafted token sampled from pi.
Verification Subsequently, these drafted tokens are verified by the target LLMMq in parallel. Formally, given the input sequence x1, . . . , xt and the draft x̃1, . . . , x̃K , Speculative Decoding utilizes Mq to compute K + 1 probability distributions simultaneously:
qi =Mq (x | x≤t, x̃<i) , i = 1, . . . ,K + 1. (3) Then, each drafted token x̃i is verified by a specific criterion VERIFY (x̃i, pi, qi). Only those tokens that meet the criterion are selected as final outputs, ensuring quality consistent with the target LLM’s standards. Otherwise, the first drafted token x̃c that fails the verification will be corrected by the strategy CORRECT (pc, qc). All drafted tokens after position c will be discarded, to guarantee the high quality of the final outputs. If all tokens pass verification, an additional token xt+K+1 will be sampled from qK+1 as Eq. (1).
The drafting and verification substeps will be iterated until the termination condition is met, i.e., the [EOS] token is decoded or the sentence reaches the maximal length.
Notably, the acceleration effect of Speculative Decoding primarily hinges on the acceptance rate
of drafted tokens at each step. This rate is influenced by several factors, including the draft quality, verification criteria, and the behavior alignment between the drafter and the target LLM. Additionally, the intrinsic efficiency of the drafter itself also contributes to the overall speedup. In subsequent sections, we will delve into these pivotal components of Speculative Decoding, as depicted in Figure 3, to systematically categorize recent research trends within this promising paradigm.

heading: 5 Drafting
text: As a vital component of Speculative Decoding, the drafting process has a crucial impact on the speedup of the paradigm. The impact is determined by two key factors: the speculation accuracy of the drafterMp, measured by the average number of accepted tokens per step, and the drafting latency (Stern et al., 2018; Xia et al., 2023). How to trade off high speculation accuracy and low drafting latency presents a major challenge in this process. In this section, we classify various drafting strategies into two categories: independent drafting (§5.1) and self-drafting (§5.2), and summarize their formulations DRAFT (x≤t,Mp) in Table 1.

heading: 5.1 Independent Drafting
text: To strike a balance between speculation accuracy and efficiency, SpecDec (Xia et al., 2023) first proposed utilizing an independent model for drafting. Specifically, it employed a specialized NonAutoregressive Transformer that drafts multiple tokens simultaneously per step. This model has a deep-shallow encoder-decoder architecture to run efficiently. Despite its strengths, SpecDec requires training a draft model from scratch, which demands an increased computational budget.
Considering the available models in existing LLM series (e.g., OPT (Zhang et al., 2022) and LLaMA (Touvron et al., 2023a,b)), a more straightforward and efficient approach is directly employing a small LM from the same series as the drafter
to accelerate the inference of its larger counterparts (Leviathan et al., 2023; Chen et al., 2023a; Spector and Re, 2023; Sun et al., 2023; Chen et al., 2023b). For instance, Leviathan et al. (2023) utilized T5-small as the drafter, to accelerate the inference of T5-XXL. These off-the-shelf small LMs do not require additional training or any modification on model architectures, facilitating the quick adoption of Speculative Decoding. Moreover, since models in the same series share tokenizers, pretraining corpora, and similar training processes, they inherently have an alignment in prediction behaviors.

heading: 5.2 Self-Drafting
text: While leveraging an external draft model offers considerable advantages, this approach necessitates extra effort to either train or identify a draft model that closely aligns with the target LLM. This challenge is intensified when no smaller counterparts of the LLM are available, e.g., LLaMA-7B (Touvron et al., 2023a,b). Furthermore, integrating two distinct models within a single system introduces additional computational complexity, particularly in distributed settings (Cai et al., 2024).
To address the above issues, numerous studies have suggested leveraging the target LLM itself for efficient drafting (Stern et al., 2018; Santilli et al., 2023; Hooper et al., 2023; Cai et al., 2024; Fu et al., 2024; Du et al., 2024). Particularly, Blockwise Decoding (Stern et al., 2018) and Medusa (Cai et al., 2024) incorporated FFN heads atop the Transformer decoder, enabling the parallel token generation per step. Compared with external drafters, these lightweight heads reduce extra computational overhead and are friendly to distributed inference. Another line of research has explored the potential of early exiting and layer skipping within the target LLM for drafting (Yang et al., 2023b; Zhang et al., 2023a; Hooper et al., 2023). For instance, Yang et al. (2023b) introduced additional subprocesses that exit early during the current decoding step, thereby initiating the drafting of future tokens in
Methods VERIFY (x̃i, pi, qi) CORRECT (pc, qc) Representative Work
Greedy Decoding x̃i = argmax qi xt+c ← argmax qc
Blockwise Decoding (Stern et al., 2018), SpecDec (Xia et al., 2023)
Speculative Sampling r < min
( 1, qi(x̃i)
pi(x̃i)
) , r ∼ U [0, 1] xt+c ∼ norm(max (0, qc − pc)) Speculative Decoding (Leviathanet al., 2023), SpS (Chen et al., 2023a)
Table 2: Summary of formulations for various verification strategies in Speculative Decoding.
advance. Similarly, Self-Speculative (Zhang et al., 2023a) proposed to adaptively skip several intermediate layers during inference to draft efficiently.
In contrast to prior work that focused on extending model architectures or altering the inference process, Santilli et al. (2023) introduced a simple drafting strategy that directly appends multiple [PAD] tokens to the end of the input prompt to enable parallel generation. However, this approach deviates from LLMs’ autoregressive pretraining pattern, leading to suboptimal drafting quality. To tackle this, Fu et al. (2024) proposed transforming low-quality drafts into multiple n-grams to improve the speculation accuracy; Monea et al. (2023) introduced multiple learnable [LA] tokens and finetuned these token embeddings on a small training dataset to enhance the parallel decoding performance.
6 Verification
In each decoding step, the drafted tokens are verified in parallel to ensure the outputs align with the target LLM. This process also determines the number of tokens accepted per step, a vital factor impacting the speedup. This section summarizes various verification criteria VERIFY (x̃i, pi, qi) (as shown in Table 2), encompassing those supporting greedy decoding (§6.1) and speculative sampling (§6.2) in LLM inference. Besides, we introduce token tree verification (§6.3), an effective strategy to increase the token acceptance rate.

heading: 6.1 Greedy Decoding
text: Early attempts at Speculative Decoding focused on the verification criterion supporting greedy decoding, which guarantees that the outputs are exactly the same as the greedy decoding results of the target LLM (Stern et al., 2019; Sun et al., 2021; Xia et al., 2023). Formally, given the input sequence x1, . . . , xt, the drafted tokens x̃1, . . . , x̃K , and the computed probability distributions p1, . . . , pK , q1, . . . , qK as obtained from Eq. (2) and (3), respectively, the verification criterion on the ith drafted
token is formulated as
x̃i = argmax qi, (4)
where i = 1, . . . ,K. The first position c that the drafted token x̃c fails the verification denotes the bifurcation position. The output token at this position xt+c will be adjusted by the correction strategy, which simply replaces the drafted token with the LLM’s top-1 prediction:
xt+c ← argmax qc. (5)
The verification criterion of greedy decoding is straightforward and clear. Thus, multiple subsequent studies have adopted this criterion to demonstrate the efficacy of their methodologies (Santilli et al., 2023; Yang et al., 2023b; Hooper et al., 2023; Zhang et al., 2023a; Fu et al., 2024). However, the strict matching requirement of this criterion often results in the rejection of high-quality drafted tokens, simply because they differ from the top-1 predictions of the target LLM, thereby constraining the speedup of the paradigm.
To tackle this problem, multiple studies have proposed various approximate verification criteria (Stern et al., 2018; Xia et al., 2023; Kim et al., 2023). Compared with the lossless criterion, these methods slightly relax the matching requirement to trust the drafts more, leading to higher acceptance of drafted tokens. For instance, SpecDec (Xia et al., 2023) only requires the drafted tokens to fall in top-k candidates of the target LLM; BiLD (Kim et al., 2023) proposed a rollback criterion that only rejects drafted tokens when the number of consecutive mismatch tokens exceeds a fixed threshold.

heading: 6.2 Speculative Sampling
text: Following Stern et al. (2019), subsequent work extended Speculative Decoding to support various sampling methods (Leviathan et al., 2023; Chen et al., 2023a), accelerating the target LLM’s inference without changing its output distribution. Formally, given the initial sequence x1, . . . , xt, the
drafted tokens x̃1, . . . , x̃K and the computed distributions p1, . . . , pK , q1, . . . , qK , the verification criterion on the ith drafted token is
r < min ( 1, qi(x̃i)
pi(x̃i)
) , r ∼ U [0, 1] , (6)
where r denotes a random number drawn from a uniform distribution U [0, 1]; qi(x̃i) and pi(x̃i) are the probability of x̃i according toMq andMp, respectively; and i = 1, . . . ,K. In other words, this criterion accepts the token x̃i if qi(x̃i) ≥ pi(x̃i), and in case qi(x̃i) < pi(x̃i) it rejects the token with probability 1− qi(x̃i)pi(x̃i) . The correction strategy resamples the output token at the bifurcation position c from an adjusted distribution:
xt+c ∼ norm(max (0, qc − pc)). (7)
Leviathan et al. (2023) and Chen et al. (2023a) have theoretically proved that this criterion maintains identical output distributions to the target LLM. Thus, it has been widely adopted in subsequent research (Liu et al., 2023; Zhou et al., 2023; Monea et al., 2023; Chen et al., 2023b). In addition to the strict requirement, some work has also explored approximate strategies to improve the token acceptance rate (Leviathan et al., 2023; Zhou et al., 2023). For instance, Leviathan et al. (2023) proposed multiplying pi(x̃i) in Eq. (6) by a lenience parameter l ∈ [0, 1] to slightly relax the criterion.

heading: 6.3 Token Tree Verification
text: Contrary to prior verification strategies that focused on a single draft sequence, SpecInfer (Miao et al., 2024) proposed token tree verification, an effective strategy enabling the target LLM to verify multiple draft sequences in parallel. As illustrated in
Figure 4, this method first merges multiple candidate draft sequences into a token tree by sharing prefixes. It then utilizes a specially designed tree attention mask to facilitate the LLM verifying the whole structure in parallel. Recent research has explored various approaches to obtain these candidate draft sequences (Miao et al., 2024; Cai et al., 2024; He et al., 2023; Li et al., 2024). For instance, Miao et al. (2024) generated diverse draft sequences from different boost-tuned LMs; Cai et al. (2024) considered the top-k predictions from each FFN head to obtain multiple candidate sequences.

heading: 7 Alignment
text: As illustrated in Section 5, the speedup of Speculative Decoding primarily depends on the speculation accuracy, which in turn is influenced by the behavior similarity between the drafter and the target LLM. To enhance this, existing research has explored various knowledge distillation (KD) strategies to align the drafter’s outputs with those of the target LLM (Stern et al., 2018; Xia et al., 2023; Miao et al., 2024; Liu et al., 2023; Kim et al., 2023;
Zhou et al., 2023). Particularly, Blockwise Decoding adopted sequence-level knowledge distillation (Seq-KD) (Kim and Rush, 2016) for alignment, which trained the drafter on the sentences generated by the target LLM. Miao et al. (2024) proposed a collective boost-tuning (Col-BT) strategy, applying Seq-KD to finetune multiple small LMs on the training data and utilizing their aggregated output as drafts to improve the speculation accuracy.
Although Seq-KD is effective, it ignores the probability distributions of the target LLM, leading to performance degradation with sampling methods. To rectify this, recent studies have explored other KD strategies for Speculative Decoding (Zhou et al., 2023; Liu et al., 2023). Notably, DistillSpec (Zhou et al., 2023) conducted a comprehensive comparison of different KD strategies on Speculative Decoding across various downstream tasks. Liu et al. (2023) proposed an online KD strategy that dynamically aligns the drafter with the target LLM on the fly using the query data.
We summarize the main features of existing Speculative Decoding methods in Table 3, including the drafter type or the drafting strategy, the alignment approach, supported verification strategies, and the reported speedup, etc.

heading: 8 Spec-Bench
text: With the rapid research progress in Speculative Decoding, there is an increasing demand for comparative analysis of leading methods. However, existing approaches are tested using disparate benchmarks, devices, and environments, making fair comparisons impractical. To address this gap, we introduce Spec-Bench – a comprehensive benchmark for Speculative Decoding covering diverse application scenarios. Based on Spec-Bench, we present a systematic comparison of open-source approaches under third-party testing conditions. Experiments were executed on the same device and testing environment to ensure a fair comparison.

heading: 8.1 Benchmark Construction
text: To assess Speculative Decoding methods across various scenarios, Spec-Bench encompasses six distinct subtasks: multi-turn conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation. We composed Spec-Bench by randomly selecting 80 instances from each of six widely used datasets, including MT-bench (Zheng et al.,
1 1.2 1.4 1.6 1.8 2 2.2 2.4
Multi-turn Conversation
Translation
Summarization
Question Answering
Mathematical Reasoning
Retrievalaugmented Generation
EAGLE SpS Medusa PLD REST Lookahead
Figure 5: Speedup comparison of various Speculative Decoding methods on Spec-Bench with greedy settings (T = 0). Evaluations were conducted on Vicuna-7B with a batch size of 1. We present the mean speedup over 3 runs. The detailed results are shown in Appendix C.
2023), WMT14 DE-EN, CNN/Daily Mail (Nallapati et al., 2016), Natural Questions (Kwiatkowski et al., 2019), GSM8K (Cobbe et al., 2021), and DPR (Karpukhin et al., 2020). For details on SpecBench and the specific experimental setup, please refer to Appendix B.

heading: 8.2 Comparative Evaluation
text: Our main evaluations were conducted on Vicuna7B at FP16 precision using a single consumergrade 3090 GPU4. As depicted in Figure 5, under greedy settings, EAGLE (Li et al., 2024) achieves the highest speedup ratio (1.8×∼2.4×) over autoregressive decoding across most subtasks, especially in mathematical reasoning (with a∼2.4× speedup). EAGLE’s success is mainly due to two factors: 1) it reuses the KV cache of LLMs to predict drafted tokens, substantially reducing the drafting computational overhead; 2) compared with Medusa (Cai et al., 2024), EAGLE drafts in an autoregressive way, providing more stable and accurate speculation results. PLD (Saxena, 2023) excels in subtasks with high similarities between input and output, such as summarization (with a ∼2.4× speedup). However, its performance diminishes in other subtasks like translation and question answering, with speedup ratios falling between 1.1×∼1.3×.
We also compare the speedups of Speculative Decoding methods at different sampling temperatures. As illustrated in Figure 6, EAGLE consis-
4For comparative analysis on a more powerful A100 GPU, please refer to Appendix D.
tently outperforms other methods across various settings, achieving a speedup ratio ranging from 1.7× to 2.1×. Besides, it is observed that the acceleration effect of all methods decreases with an increase in sampling temperature. This is attributed to the increased computational complexity of the speculative sampling criterion at higher temperatures, as revealed in prior research (Joao Gante, 2023; Spector and Re, 2023).

heading: 9 Challenges and Future Directions
text: How to trade off speculation accuracy and drafting efficiency? As discussed in Sections 5, scaling up the drafter can effectively enhance speculation accuracy, yet it largely reduces the drafting efficiency and even the overall speedup. Therefore, it is essential to strike a balance between speculation accuracy and drafting latency. Among existing strategies, behavior alignment is a promising approach to address this issue, as it improves speculation accuracy without increasing latency. However, despite recent advancements (Miao et al., 2024; Zhou et al., 2023; Liu et al., 2023), there is still considerable room for improvement to align the drafter with the target LLM. For example, given that the drafted tokens after the bifurcation position are all discarded, one potential direction could involve encouraging the drafter to prioritize the generation quality of early-position tokens. Beyond alignment, other factors such as the quality of drafting (Fu et al., 2024) and the determination of speculation length (Su et al., 2023) also influence speculation accuracy and merit further exploration.
How to apply Speculative Decoding in batched inference scenarios? Currently, only a few Speculative Decoding implementations have supported batched inference, such as EAGLE5 and SpS6.
5https://github.com/SafeAILab/EAGLE 6https://github.com/lucidrains/
speculative-decoding
However, batched inference is a crucial technique for efficiently managing user inputs in LLM realtime services. The primary challenges in batched Speculative Decoding lie in two aspects: (1) Each decoded sentence in Speculative Decoding varies in decoding steps due to different speculation accuracy. Thus, the inference latency of a batch depends on the slowest sample in the batch; (2) The extra computational complexity introduced by Speculative Decoding, especially in sampling settings, increases with larger batch sizes. How to maintain a promising speedup of Speculative Decoding in batched inference, and combine it with advanced techniques such as continuous batching (Yu et al., 2022), warrants further investigation.
How to integrate Speculative Decoding with other leading techniques? As a general decoding paradigm, Speculative Decoding has already demonstrated its potential in conjunction with other advanced techniques (Yang et al., 2023a; Zhang et al., 2023b; Li et al., 2023). For instance, Yuan et al. (2023) combined Speculative Decoding with Contrastive Decoding (Li et al., 2023), which not only speeds up the inference but also substantially improves the generation quality. In addition to the acceleration of text-only LLMs, applying Speculative Decoding in multimodal inference, such as image synthesis, text-to-speech synthesis, and video generation, is also an intriguing and valuable direction for future research. Another promising research direction is to integrate Speculative Decoding with other efficient methods such as vLLM (Kwon et al., 2023), Non-Auregressive Generation (Du et al., 2021, 2022) and FlashAttention (Dao et al., 2022; Dao, 2023), further boosting the inference efficiency of LLM services.

heading: 10 Conclusion
text: This paper presents a comprehensive survey of Speculative Decoding, including the evolution of this promising paradigm, its formal definition and formulation, a systematic categorization of existing methods, and an in-depth review of leading techniques. Moreover, we introduce Spec-Bench, an extensive evaluation benchmark for Speculative Decoding methods, and present a comparative evaluation of prominent methods. To our knowledge, this is the first survey dedicated to Speculative Decoding. Our aim for this paper is to clarify the current research landscape and provide insights into future research directions.
Limitations
This paper provides a thorough examination and categorization of current methodologies and emerging trends in Speculative Decoding. We have also conducted a comparative analysis of leading opensource methods to offer researchers deeper insights into the advantages and limitations of different models. Beyond Speculative Decoding, we acknowledge additional efficient NLP strategies such as vLLM (Kwon et al., 2023) and continuous batching (Yu et al., 2022). In the future, we intend to expand the discussion to encompass the integration of Speculative Decoding with these advanced techniques. Moreover, due to the absence of an available implementation of batched Speculative Decoding, our evaluations could not cover this aspect. We plan to undertake subsequent experiments to assess the speedup of Speculative Decoding methods across various batch sizes.
Ethics Statement
The datasets used in our experiment are publicly released and labeled through interaction with humans in English. In this process, user privacy is protected, and no personal information is contained in the dataset. The scientific artifacts that we used are available for research with permissive licenses. And the use of these artifacts in this paper is consistent with their intended use. Therefore, we believe that our research work meets the ethics of ACL.

heading: Acknowledgements
text: We thank all anonymous reviewers for their valuable comments during the review process. This work is partially supported by Research Grants Council of Hong Kong (15207122 and 15213323).

heading: A Applications
text: In addition to serving as a general paradigm, recent work has revealed that some variants of Speculative Decoding demonstrate extraordinary effectiveness in specific tasks. Furthermore, other research has applied this paradigm to address latency issues unique to certain application scenarios, achieving inference acceleration. Below, we will provide a detailed introduction to these promising works.
Recent studies have highlighted Speculative Decoding is particularly well suited for tasks where model inputs and outputs are highly similar (Sun et al., 2021; Ge et al., 2022; Yang et al., 2023a), such as Grammatical Error Correction (Wang et al., 2021; Bryant et al., 2023) and Retrieval-augmented Generation (Cai et al., 2022). These methods introduced a specialized form of Speculative Decoding, where the initial user input or the retrieved context is directly employed as drafts. For instance, SAD (Sun et al., 2021), an early attempt at Speculative Decoding on Grammatical Error Correction, utilized the input sentence with grammatical errors as a draft and leveraged the LLM to verify the whole sentence in parallel, achieving a 9×∼12× speedup. Similarly, LLMA (Yang et al., 2023a) selected text spans from the reference as drafts, demonstrating a 2×∼3× speedup across various practical application scenarios including Retrievalaugmented Generation, Cache-assisted Generation, and Multi-turn Conversations.
Beyond these works, RaLMSpec (Zhang et al., 2023b) adopted Speculative Decoding to accelerate retrieval-augmented language models (RaLMs). It pointed out that the main latency bottleneck of iterative RaLMs is the frequent retrieval from a vast knowledge base. To accelerate inference, this method proposed to maintain a local cache for speculative retrieval, achieving around 2× speedup with identical model outputs. LLMCad (Xu et al., 2023) applied Speculative Decoding to on-device LLM inference. Concretely, it proposed to generate drafts with a smaller real-time LM that can be hosted in device memory, and only utilize the target LLM for parallel verification. This approach effectively reduces repetitive releasing and loading of model weights, achieving a 9.3× speedup compared to existing inference engines.

heading: B Experimental Details
text: B.1 Details of Spec-Bench
To assess the acceleration performance of Speculative Decoding methods in various scenarios, we developed Spec-Bench, a comprehensive benchmark encompassing six distinct tasks. Spec-Bench integrates MT-bench (Zheng et al., 2023), a multiturn conversation benchmark previously adopted in research (Cai et al., 2024; Li et al., 2024), to provide a basis for comparison with earlier studies. Additionally, it includes two input-guided tasks: summarization and retrieval-augmented generation (RAG), both of which exhibit a significant overlap between the input prompts and the target outputs. We selected CNN/Daily Mail (Nallapati et al., 2016) and Natural Questions (Kwiatkowski et al., 2019) as the dataset for these two tasks, respectively. Specifically, in the RAG subtask, the top-5 documents retrieved from DPR (Karpukhin et al., 2020) were concatenated with each question to construct the input prompt.
Moreover, Spec-Bench incorporates three further subtasks – translation, question answering, and mathematical reasoning – to provide a thorough evaluation of Speculative Decoding’s speedup capabilities in diverse contexts. We utilized WMT14 DE-EN, Natural Questions, and GSM8K (Cobbe et al., 2021) as the primary datasets for these tasks, respectively. We randomly selected 80 instances from each subtask’s test set for evaluation. The detailed composition is summarized in Table 4.
B.2 Implementation Details
We have selected six representative Speculative Decoding methods for our comparative analysis on Spec-Bench. These methods are open-source and free of bugs. Specifically, SpS (Chen et al., 2023a) stands as the pioneering work in this field, utilizing a smaller LM from the same model series as the
drafter to accelerate LLM inference. Medusa (Cai et al., 2024) and EAGLE (Li et al., 2024) integrate additional lightweight heads into the target LLM to facilitate efficient drafting. Lookahead (Fu et al., 2024) introduces multiple special tokens to the end of the input prompt for parallel drafting and transforms the drafts into n-gram candidates. PLD (Saxena, 2023) is the code implementation7 of LLMA (Yang et al., 2023a), which selects text spans from the input as drafts. REST (He et al., 2023) retrieves relevant drafts from text corpora based on the input prompt.
We conducted our experimental evaluations using the Vicuna-v1.3 model series (Zheng et al., 2023). For SpS, we employed the Huggingface implementation8 and utilized the vicuna-68m-v1.3 model provided by Yang et al. (2024) as the drafter. We followed the default parameters of Lookahead9 and PLD for our evaluations. The main experiments were conducted using Pytorch 2.0.1 with a single consumer-grade NVIDIA GeForce RTX 3090 GPU (24GB) of 12 CPU cores under CUDA 11.8. Further analysis was performed on a more powerful NVIDIA A100 GPU (80GB) of 64 CPU cores under CUDA 11.4.

heading: C Details of Main Experimental Results
text: The detailed results of our main analysis are shown in Table 5, including the experimental settings of greedy decoding (T = 0) and speculative sampling (T = 1). The findings indicate that EAGLE (Li et al., 2024) excels across various Spec-Bench subtasks, achieving an overall speedup ranging from
7https://github.com/apoorvumang/ prompt-lookup-decoding
8https://huggingface.co/blog/ assisted-generation
9https://github.com/hao-ai-lab/ LookaheadDecoding
1.6× to 2.4×. PLD (Saxena, 2023) shows notable efficiency in scenarios where the input and output have a significant overlap. For instance, the speedup ratio of PLD increases from 1.27× in the question answering subtask to 1.66× in the retrieval-augmented generation subtask, highlighting its effectiveness when the input includes relevant documents. Notably, most methods achieve a suboptimal speedup on the translation subtask. We suspect that it is due to the potential lack of multilingual data in the pretraining corpora.

heading: D Further Analysis on A100
text: This section presents a comprehensive analysis of leading Speculative Decoding methods on SpecBench, utilizing a single NVIDIA A100 GPU. The discussion delves into the influence of computational hardware, model scale, and computational precision on the performance of Speculative Decoding. All experiments were performed on the same device and environment to ensure fair comparison.
D.1 Computational Devices
We first discuss the impact of evolving computational devices on Speculative Decoding. As depicted in Figure 7, the acceleration effect of most Speculative Decoding methods is notably enhanced when employed on high-performance GPUs, such
as NVIDIA A100s. This enhancement is primarily due to the increased availability of idle computational resources on more advanced computational devices, which Speculative Decoding can leverage to accelerate inference processes. Among the methods evaluated, Medusa (Cai et al., 2024) and Lookahead (Fu et al., 2024) demonstrate the most significant improvements. Specifically, the speedup ratio for Medusa escalates from 1.48× to 2.42×, and for Lookahead, it rises from 1.11× to 1.77×. This finding underscores that Speculative Decoding methods will benefit more from evolving computational hardware, such as H100 GPUs.
We illustrate the comparison of various Speculative Decoding methods evaluated with a single A100 GPU in Figure 8. The detailed experimental results are shown in Table 6. The results indicate that Medusa (Cai et al., 2024) and EAGLE (Li et al., 2024) excel in this experimental setting, achieving an overall speedup of 2.4×. These two methods perform particularly well on the multi-turn conversation and mathematical reasoning subtasks, with a ∼2.8× speedup.
D.2 Model Scale
We present the speedup comparison of Speculative Decoding methods across various model scales in Figure 9. The detailed experimental results are shown in Table 6. Among all the evaluated methods, EAGLE (Li et al., 2024) maintains a high speedup ratio over autoregressive decoding across all model scales, achieving a speedup ratio ranging from 2.4× to 2.5×. While Medusa (Cai et al., 2024) demonstrates superior acceleration performance on Vicuna-7B, its speedup ratio degrades
from 2.4× to 2.0× as the model scale increases.
D.3 Computational Precision It is noteworthy that most Speculative Decoding approaches are predominantly evaluated using FP16 precision (Fu et al., 2024; Cai et al., 2024; Li et al., 2024; He et al., 2023). However, it is critical to underscore that the outputs generated by Speculative Decoding in FP16 precision may not consistently align with those derived from autoregressive decoding. This divergence stems from the accumulation of floating-point errors inherent in FP16 computations, which can result in discrepancies between the outputs of the two decoding methods, particularly in the context of longer sequences. In FP32 precision, the outputs of Speculative Decoding are guaranteed to be exactly the same as autoregressive decoding.
We compare the speedup performance of Speculative Decoding methods with FP16/FP32 precision in Figure 10. The experimental results reveal a noticeable reduction in speedup for all methods under FP32 precision. Specifically, PLD (Saxena, 2023) achieves merely 1.01× speedup in FP32 precision, and the acceleration effect of EAGLE (Li et al., 2024) also diminishes, with its speedup falling from 2.39× to 1.74×. To furnish the research community with a comprehensive understanding of the acceleration impact, we advocate for future studies to report speedup metrics across both precision settings.

heading: Future Work
text: To furnish the research community with a comprehensive understanding of the acceleration impact, we advocate for future studies to report speedup metrics across both precision settings