Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Authors: Heming Xia
Tao Ge
Peiyi Wang
Si-Qing Chen
Furu Wei
Zhifang Sui
Venue: EMNLP 2023 Findings (2023)
Keywords: speculative decoding
efficient seq2seq generation

Abstract: We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only 1.4x~2x speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Heming Xial.24 Tao Ge
Peiyi Wang Si-Qing Chen*
Furu Wei Zhifang Sui
Nattioal Key Laboratory for Multimedia Information Processing, Peking University Schooo of Software & Microelectronicss Peking University
School of Compuler Science, Peking University "Microsoft Research Asia {xiaheming, szf edu. cn; wangpeiyi9979@gmai.. com
{tage, fuwei laeeeie@mccooofffec com
Abstract
x
8
9z
Verify
executed
We propose Speculative Decoding (SpecDec), for the first time ever', to formally study ex- ploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Spec- ulative Decoding has two innovations: Spec- Drafter an independent model specially op- timized for efficient and accurate drafting reliable method
parallel
Autoregressive Decoding
92
9. g.
Draft
9:
92
Figure I: Compared with autoregressive decoding (left) that generates token by token, the traft-then-vert paradigm (right) first drafts multiple tokens efficiently and then verifies these tokens in parallel. Drafted tokens after the bifurcation position (e.g., 3h) will be discarded to guarantee the generation quality.
and Verifieeiiicttion for verifying the drafted tokens efficiently the decoding paradigm. Experimental re- sults various seq2seq tasks including ma- chine translation and abstractive summariza- tion show our approach can achieve around 5x speedup for the popular Transformer archi- tectures with comparable generation quality to beam search decoding, refreshing the impres- sion that the draft-then-verify paradigm intro- duces only 1.4x~22 x speedup. In addition to the remarkable speedup, also demonstrate additional advantages of SpecDec, revealing its practical value for accelerating generative mod- els in real- world applications. Our models and codes are available at https://g /ggi thub. com/ hemingkx/ /SpeDDe.
costs but also limits the application of advanced AR models in real-time scenarios.
In this work, we study the traft-then-verify paradigm for accelerating seq2seq generation of an existing AR model?. As shown in Figure 1, the draft-then-verify paradigm first generates num- ber of drafted tokens efficiently and then verifies these tokens using the existing AR model paral- lel to ensure the decoding result matches AR de- coding. However, previous attempts in the "draft- then-verify" paradigm such as Blockwise Decod- ing (Stern et al., 2018) and Aggressive Decod- ing (Sun et al., 2021) tend to lack in-depth inves- tigation of this paradigm. Their modest speedup (i.e., 1.4x~2.0x) or limitation to certain seq2seq tasks like Grammatical Error Correction (GEC) has caused this paradigm to be underestimated, result- ing in it not receiving much attention and remaining dormant for years.
1 Introduction
As the de facto method for text generation, Au- toRegressive (AR) decoding is widely blamed for its poor inference efficiency due to its low level of parallelism, which fails to utilize the full potential of modern parallel computing devices like GPUs. This inefficiency not only leads to high deployment
This work was done during the author's 's internship MSR Asia. Correspondence: Tao Ge (lage@ microsoft.com) Co-first authors with equal contributions
To fully exploit the draft-then-verij paradigm, propose Speculative Decoding (SpecDec), draw- ing inspiration from speculative execution in com-
'This work was initially announced in March 2022 inttps://arxit http//axxv.oogggbbb/aa/a wwt.or/gg//g......////////// 111/22223..166 16487) under the name Generalized Aggressive Decoding. It has been formally re- named Speculative Decoding in our submission to ICLR'23 which has been publicly available since September 2022 at https:/ / ///pp//ppeeee..../////// wwwviie.net... www.ee.weee/dffff/ hetp/ddf?d=Ht-Viie VlwsYvVi. This marks the first time "Speculative Decoding". which explic- itly studies the idea of speculative execution to accelerate Transformer inference, has been proposed.
"The existing AR model in this paper refers to the targeted Transformer using AR decoding that we want to accelerate. "Speculative execution is an optimization technique used in computer architecture where a system performs some task in advance to avoid delays that would have to be incurred by doing the task after it is known that it is required (https: / //wiki //wikipedia. aaciia..rr//// org/wiki/ www.ww//peeuuateeee _eeevveuiii

---

puter architecture, with two key innovations that improve drafting and verification processes respec- tively. For drafting, we derive two principles for designing the drafting model": the Capability Prin- ciple and the Latency Principle. Following these two principles. we propose Spec-Drafter - spe- cialized independent model optimized in the draft- then-verify paradigm, which can accurately and efficiently fulfill the drafting task.
2 Background: draft-then-verify decoding
The "draft- foeffttten-verff" paradigm first drafts mul- tiple tokens efficiently, as a speculation AR de- coding results; then, it verifies these tokens in par- allel to ensure they match the AR decoding result, as illustrated in Figure 1. It is implicit imple- mentation of speculative execution in Transformer inference.
For verification, we propose an advanced method Sspe--erificationn that relaxes the vanilla verifica- tion strategy. Spec- Verification allows the decoding results of SpecDec to be slightly different from AR greedy decoding, offering opportunity to accept more drafted tokens without sacrificing generation quality and leading higher decoding efficiency.
Draft There are different approaches to drafting tokens, including model-based (Stern et al., 2018) and
based methods (Sun et al., 2021; Yang et al., 2023). Take Blockwise Decod- ing the most representative work attempting the draft-then-verij paradigm - as an example (illus- trated in Figure 2(a)): introduces additional k feedforward network (FFN) heads on top an ex- isting AR model, enabling the model to predict the next h drafted tokens in parallel during inference.
We conduct extensive experiments on various seq2seq generation tasks like machine translation and abstractive summarization. Results show our approach can achieve around 5x speedup for the popular Transformer architectures with compa- rable generation quality to beam search decod- ing, largely outperforming previous draft-then- verify work (1.4x~2.00 speedup). Moreover, we demonstrate that SpecDec has several additional advantages that enhance its practicality for acceler- ating generative models in real-world applications.
Verify The generated drafted tokens are fed into the original AR model and verified in parallel. Specifically, it finds the bifurcation position c, the largest index that ensures all previous c 1 drafted tokens and the corresponding AR decoded tokens are identical:
1(1)+4 1544)
arg max
1stsk
i
Our contributions can be summarized as follows:
4544 = arg arg log P(y ÚS 1:41 m: AR)
We are the first work that explicitly exploits the idea of speculative exccution to accelerate Transformer inference. Our proposed two key innovations the independent Spec-Drafis Drafter and Spec-Verteee Verification strategy allow See to achieve over 5x lossless speedup over au- toregressive decoding in seq2seq tasks, re- freshing the impression that the "draft-then- verify" paradigm only has a limited 1.5x ~ 2x acceleration potential.
where II(-) is the indicator function, is the source sentence, Dsy is the previously generated tokens and Mi+s is the i-th drafted token. Drafted tokens after the position e are all discarded. The final decoded tokens in the current iteration are:
Bj+l..ite 1j+1--+oo-1 0i+c)
The above draft and verification steps are iterated until the termination condition is met.
Speculative Decoding
We demonstrate advantages of Spec with extensive empirical results in addition to its remarkable acceleration performance: bet- trade-off, easy adapt-
To fully exploit speculative execution for Trans- former inference, we propose Speculative Decod- ing (SpecDec) with two innovations Spec- Drrc-Drrfter and verc-Veriiiiation that substantially improve drafting (Section 3.1) and verification (Section 3.2) respectively.
ter aatency-throughe ability for existing models and retaining the behavior of the original model, revealing its huge practical value and bringing the long- dormant traff-then-wee vverify paradigm back into the spotlight.
'This kind of method usually limited to special tasks like GEC and retrieval-augmentt we mainly focus the model- based methods.
generation. In this paper.
"We use to denote drafted tokens, while we use ÿ to denote AR decodod/verified generation results.
"The drafting model is also called the drafter in this paper.

---

3/3
36
34
What
Ahardd
dlstect
the
beskc
Transformer Block
Transformer Block
physical
O
1964
univerte
Transforme Block
Transformer Block
7
O
What
0.8
basi
0.5
Y1
3/
32
[M]
[M]
[M]
physkca
Blockwise Decoding
(b) Sper-rratter
[MaSK]
[MASK]
-04
Blockwise Decoding that introduces & FFN heads on top of the target AR model for drafting the next & tokens with shared attention; (b) Spec-Drafter is independent model for drafted token prediction. It employs distinct attention queries for predicting each drafted token. Modules colored in yellow belong to the original AR model while those colored in red denote newly introduced modules.
Figure 2:
[MASK]
[MasK]
Whs
sing
Figure 3: Upper: An AR model's attention heatmap showing that different target positions should attend different source tokens; Lower: The Spec-Drafter attention heatmap showing its capability of modeling drafted tokens in different positions, which highly aligns with the AR counterpart.
3.1 Spec-Drafter
3.1.1 Design Principles
As a crucial ingredient in the draft-then-verißf paradigm, the drafting process has drastic impact on end-to-end acceleration performance. However, there are very limited explorations of the design- ing principles for the drafter by previous studies most of them arbitrarily implement drafter, which accounts for their undesirable acceleration results.
vested to guarantee its capability of accu- rate drafting.
Principle I the most crucial principle in de- termining the end-to-end speedup, as directly influences the value of Tok. which affects both total drafting and verification latency. Surprisingly, little previous work adheres to this seemingly sim- ple and straightforward principle maybe due to the concern of increasing the drafting latency. For in- stance, the drafter in Blockwise Decoding not properly invested: Its drafter not only has limited parameters (FFN prediction heads), making it dif- ficult to fit the challenging drafting task, but more importantly, it employs a shared attention mecha- nism that forces all drafted tokens to share single set of attentions (only differentiating at the final prediction head), as shown in Figure 2(a). How- ever, different target positions should attend dif- ferent context tokens, as illustrated in Figure 3. Despite its computational efficiency, the shared at- tention mechanism in Blockwise decoding severely constrains the drafter's ' capability, resulting in low drafting accuracy and consequently leading to most drafted tokens being discarded.
To understand the effect drafting, we look into the overall latency in the drafi-then-verij paradigm for one sample of the length as follows:
1.
1. Tok.
T
Tok.
total drafting latency
total verification latency
where Tok. denotes the average number of drafted tokens accepted per iteration, la and i, are the time costs of drafting and verification each iteration respectively.
According to Eq (4), Tok. inversely propor- tional to the number of iterations, which is primar- ily influenced by drafting accuracy: A drafter that is more capable of drafting can attain greater Tok. values, consequently completing the decoding pro- cess in fewer iterations. This observation leads us derive the first principle for designing the drafter:
In addition the drafter's accuracy, its latency also impacts the end-to-end speedup result but from another perspective - by affecting the latency of each iteration (i.c., fu in Eq (4)) from which we derive Principle for designing the drafter:
Principle I (Capability Principle): The drafter model should seriously in-
"We don't discuss f, in this paper because is determined by the existing AR model and thus regarded constant.

---

Source Sentence
Machen sich Hunderte Millionen von Autofahrern sorgen über ihre Privatsphäre.
Decoder Input
Millions
of
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
Draft
Millions
of drivers
will
be
concerned
about
Spec-Verify
Millions
of
drivers
will
worry
concerned
about
their
motorg
[BLANK]
be
[BLAK]]
[BLANK]
[BLANK] [BLANK]
[BLANK]
[BLANK]
[BLANK]
Output
Millions
drivers
will
concerned
about
their
Input
Millions
drivers
will
be
about
their
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
Verification. Compared to the vamilla verification strategy strictly requiring the drafted tokens to match the AR top-1 result, Opec-Verification slightly relaxes the criterion to trust the drafts more, by only requiring the drafted tokens to fall in the top-B AR candidates with tolerable log-likelihood gap (not shown this Figure; see Fg (9)). As result, Spec- Verification allows more drafted tokens to be accepted even if they are slightly different from the AR top-1 result, leading to a higher inference speedup.
Figure 4: Illustration of Spee--erifieeee
Principle II (Latency Principle): The drafter should be fast at generating drafted tokens minimize the latency overhead of each iteration.
its encoder (by increasing its depth), which is moti- vated by the fact that the encoder is forwarded only once, while the decoder is frequently forwarded for iterative decoding. This encoder-favored modeling has been demonstrated by previous work to im- prove latency with little generation quality degrada- tion (Kasai et al., 2021; Sun et al., 2021; Ge et al., 2022a). We find it also highly effective for the drafter in the draft-then-verijy decoding paradigm.
Designing a fast drafter solely based on Prin- ciple II is not difficult, done in most previous work. The real challenge lies in designing low- latency drafter without compromising its capability (Principle 1), since it difficult achieve both low latency and high capability simultaneously.
3.1.3 Training and Inference
3.1.2 Model Architecture
Formally, given the source sentence and the ran- domly sampled prefix YsP (0 S < m) of the target sentence, Spec-Drafter appends l special " "[MASK]" tokens to Ysp, and trained to predict these masked tokens in parallel:
We propose Spec-Drafter, which adheres to both principles for accurate and fast drafting. To ensure the drafter is sufficiently capable of accurate draft- ing (Principle I), Spec-Drafter employs an indepen- dent encoder-decode model architecture, which generates drafted tokens conditioned on the left- ward context and source tokens in mask-predict manner (Ghazvininejad al., 2019), as illustrated in Figure 2(b). This independent model design facilitates Spec-Drafter predict each drafted to- ken using distinct attention queries, in contrast to Blockwise Decoding employing a shared attention query for predicting all drafted tokens (as illus- trated in Figure 2). In this way, Spec- Dnec-Drrfter could better align with the AR model's behavior, thereby increasing the chances of its drafted tokens being accepted during verification, as shown in Figure 3. To make Spec-Drafter fast (Principle II) without compromising capability, we design its decoder to be lightweight by reducing the number of de- coder layers and reallocating the freed-up budget to
n+
Crpte-Dratte
Σ 1mpo1
log P I m. Ospec
ySp = (11. Mp- [MASK]. ... [MASK]
(6)
In addition, we leverage the glancing strategy following Qian et al. (2021), which exploits cur- riculum learning during training to get better gen- eration performance.
During inference, Spec-Drafter appends k " "[MASK]" tokens the previously decoded tokens Úæソ and simultaneously predict these masked to- kens as a drafted block:
U+ = arg max log 1 m: (spec-Drateer) (7)
where i = 1....kk

---

3.2
Model Configuration The primary target model we accelerate in our experiments is the Transformer-base model with a 6-layer encoder and a 6-layer decoder of 512/2048 embedding/FFN dimension, which can achieve stateoof-the-an results on the benchmarks under comparable model size conditions. For the Spec-Drafter, we adopt similar architecture to the AR model except with 12 encoder layers and 2 decoder layers to make sure it adheres to both the Capability and Latency principles. We apply sequence-level knowledge distillation (Kim and Rush, 2016) by the AR teacher to the Spec-Drafter to align its behavior with the AR model as much as possible. We include model training details in Appendix A. For the Spee-Verificcate 3 and leading the best generation quality on the validation set. Besides, we re-implement Spec- Verrfeiifiiatinnn
as the joint source-target dictionary. We evaluate performance with BLEU (Papineni et al., 2002) for both language pairs".
As introduced in Section 2, the vanilla verification strategy of preliminary studies only accepts the drafted tokens that match the top-1 result of the AR model, which guarantees that the decoding results are identical to AR greedy decoding. However, the top-1 results are not necessarily better than the drafted tokens, especially when the paradigm is equipped with high-quality drafter. Therefore, the strict verification criterion (i.e., top-1 matching) will result in many good drafted tokens being dis- carded just because they are different from the top-1 result of the AR model, which limits the speedup of the paradigm.
For inference efficiency, we report decoding speedup over beam search. Specifically, we test the inference speed by running the model with one sentence at time (batch=1). We perform model inference with fairseq implementation" 10 using Py- torch 1.10.1 with 1 Nvidia Tesla P100-PCIe of 16GB GPU memory under CUDA 11.1.
To make better of the drafting results, propose advanced verification strategy named Spec- necc-Verification, which is illustrated in Figure Instead of the rigid matching requirement shown in Eq (2), Spec-Verification Verification relaxes the criterion to trust the drafting results more, by only requiring the drafted tokens to fall in 1op-3 candidates with a tolerable (log-likelibnn liglliik)i)ood) score gap т (away from the top-1 result). Formally, it will accept the i-th drafted token 7i++ if all previous - 1 tokens are accepted, and Eq (8) and (9) are both true:
we find the hyperparameters
log P(8)+:14:000. 2 log P(s)fila
(8)
log P() P00 AAk) log P |A; Ax) xx) ST
(9) Blockwise Decoding"l using the same device and environment as ours to facilitate fair comparison.
(10)
4.2 Results
where log P())) |4;
AR) is the top-3 ranked re-
We present the performance and the acceleration effect of SpeeDec to Transformer in Table As reported in the previous work (Stern et al., 2018), Blockwise Decoding (I- - 10) can only achieve 1.4x~2x speedup without affecting the genera- tion results over the Transformer base model. Fur- ther increasing the parallel capability of Block- wise Decoding (e.g.. A 25) will not introduce more speedup its limited drafting accuracy pre- vents more drafted tokens from being accepted. In contrast, our SpecDec shows consistent perfor- mance improvement with increased parallel capa- bilities (k = 10 + 25), resulting in around
sult's log-likelihood score by the AR model.
4 Experiments
4.1
Experimental Settings
Datasets and Evaluation We mainly evaluate our approach on two standard machine translation benchmarks: WMT14 EN+DE (4.5M pairs) and WMT16 EN (610K pairs). Following prior work (Ott al., 2018), for WMT14 EN+DE trans- lation, adopt newstest- 13 as validation set for finding the best hyperparameters, and test on newstest-14. For WMT16 EN++RO translation, we use the dataset released by Lee et al. (2018), where newsdev2016 and newstest2016 are taken as validation and test sets. We use 32K Byte Pair Encoding (BPE) (Sennrich et al., 2016) subwords*
"We also report sacreBI LFU (Post, 2018) and COMET (Rei al., 2020) scores in Appendix C. "https:/ thub. httm/pytocch/fflllo
"In the original paper of Blockwise Decoding, there is also a variation that allows the AR model to be fine-tumed for hetter drafting tokens. We don't discuss this variation because it severely affects the generation quality.
WWo use the same BPF tokenization and vocabulary as Ghazvininejad et al. (2019).

---

EN
DE DE-EN
EN-+RO
RO -EE
Models
Speed
BLEU
Speed
BLEU
Speed
BLEU
Speed
BLEU
Transformer-bas
1.0x
28.89
1.0x
32.53
1.0x
34.96
1.0x
34.86
Transformer-buse
1.1x
28.73
1.1x
32.18
1.1x
34,83
1.1x
34.65
Blockwise Decoding (& 10)
1.9x
28.73
2.0x
32.18
1.4x
34,83
144x
34,65
Blockwise Decoding (A: 25)
1.6x
28.73
1.7x
32.18
1.2x
34,83
1.2x
34.65
SpeeDee (l: = 10)
4.2x
28.90
4.6x
32.61
3.9x
35.29
4.1x x
34.88
SpecDee (A = 25)
5.1x
28.93
5.5x
32.55
4.6x
35,45
4.8%
35.03
12+2 Transformer-base 12+2 Transformer-base 1)
1.0x
29.13
1.0x
32.45
1.0x
34,93
1.0x
34.80
1.1x
28.99
1.1x
32.08
1.1x
34.79
1.1x
34.55
Blockwise Decoding (f: 10)
1.6x
28.99
1.7x
32.08
1.2x
34.79
1.2x
34.55
Blockwise Decoding (k 25)
1.xx
28.99
1.5x
32.08
1.1x
34.79
1.1x
34.55
SpecDec (A- = 10)
2.7x
29.08
3.0x
32.40
2.3x
35.12
2.4X
34.85
SpecDec (l: = 25)
3.0x
29.13
3.3x
32.48
2.5x
35.07
2.6x
34.91
Table I: The performance of Speculative Decoding (SpecDecc speed up the Transformer-base the WMT
benchmarks, We re-implement and evaluate Blockwise Decoding using the same device and environment as ours.
Models
Tok.
BLEU
ta
Speed
Transformer-bel (b 5)
1.00
28.89
1.0x
SpecDec
w/ head-based drafter
2.32
28.02
0.81
1.7x
w/ Spee-Drafter (w/o Principle I)
7.05
28.56
2.29
4.4x
w/ Spee-Drafter (w/o Principle II)
28.95
10.87
4.0x
w/ Spec Dpec-Drafter
8.23
28.93
5.21
5.1x
Table 2: Ablation studies of the drafter in the SpecDec on WMT14 EN-DEE. Tok. denotes the average number of drafted tokens accepted in each iteration. f denotes the average time cost drafting per iteration. The head-based drafter is the one used by Blockwise Decoding (Stern et al., 2018). The Spec-Drafter (w/o Principle reduces the model/FFN dimension to 256/1024, while the Spec-Drafter (wfo Principle II) does not use a deep encoder and shallow decoder instead utilizes more balanced architecture with equal depth of encoder and decoder layers.
~.6xx speedup across the translation bench- marks; moreover, it even achieves an improvement in generation quality (by the BLEU metric) com- pared with AR greedy decoding.
Models
k
Tok.
BLEU
Speed
AR-base (b = 5)
1.00
26.72
1.00
10
6.05
26.80
3.99
15
6.93
27.12
4.54x
Similar results are also observed when acceler- ating the Transformer with a 12-layer encoder and 2-layer decoder- SpecDec can still achieve around 2.5x 33.3 speedup while Blockwise Decoding's acceleration effect becomes nearly negligible over the fast AR baseline.
SpecDec
20
7.41
27.05
4.72x
25
7.89
26.97
5.04x
30
7.67
26.89
4.82
Table 3: The mean accepted tokens (Toks.), the genera- tion quality (BLEU), and the efficiency (Speed) when decoding with various number of block size & on the development set of WMT14 EN-DE..
4.3 Analysis
this section, we conduct comprehensive and thorough analysis, to demonstrate that the signif- icant improvement of the SpecDec arises from both the Spec-Drafter (Section 4.3.1) and Spec- Verification (Section 4.3.2).
end generation quality and efficiency. To further investigate the Spec-Drafter, we conduct an abla- tion study on the principles it follows. Ablating the Capability Principle by reducing its size results in drastic drop in end-to-end acceleration perfor- mance, more iterations are needed complete the decoding process, indicated by a lower Tok.. When we ablate the Latency Principle by using a balanced (6+6) encoder-decoder architecture for
4.3.1 Drafting
According to Table 2, the Spec-Drafter signifi- cantly outperforms the head-based drafter (as used in Blockwise Decoding) terms of both end-to-

---

Models
Top-3 =/=3)
Top-5 (3 5)
700
Al
600
SpecDe:
Vanilla
6.41/26.62
32
500
7.89/26.97 (5.0x)
7.92/26.88
8.75/26.84
8.83/26.79
300
SpecDec
9.51/26.71
9.64/26.68
Latend 2
200
10.11/26.60
10.63/26.59
16
10.46/26.58
11.01/26.58 (6.8x)
100
150
200
Through put (samples/s)
Table 4: Results of SpeeDec (k- = 25) on the develop- ment set of WMT14 EN-DE with different hyperpa- rameters. Each cell lists the mean accepted tokens and BLEU score. Among
Figure 5: The latency-throughuut curve with various batch sizes WMT14 (1T14ENN>111 EN-
runs the table, high- est BLEU score of 26.97 is achieved when == 1.0, with a 5x speedup. On the other hand, when 3 5 and 5 = the highest Tok. (i.e., 11.01) reached, resulting in almost 7x speedup, though the BLEU score slightly decreases to 26.58.
and
enhance its practical value in the following three aspects:
Better aateny-thhougghuut trade-off SpecDec achieves inference acceleration by increasing the GPU computing parallelism. Although increas- ing the batch size can also increase the computing parallelism improve throughput, it results in- creased latency, which is not desirable real-world application scenarios. Therefore, a smaller batch size is often employed during inference, but this in turn results in the underutilization of GPU com- puting resources, leading the dilemma of low throughput for small batches and high latency for large batches, as illustrated by Figure 5. SpeeDec effectively addresses this dilemma. Even by main- taining a small batch size, SpecDec can fully utilize the computing performance of the GPU, signifi- cantly improving both efficiency and throughput.
the drafter, it also experiences a substantial decline in end-to-end acceleration performance due to in- creased latency in each iteration (reflected by a higher tz).
Moreover, analyze the block size k's effect on the end-to-end acceleration performance of the paradigm in Table contrast to the blockwise decoding achieving its best acceleration perfor- mance at = 10 as shown in Table 1, Seec achieves its best performance at k 25 with 7.89 mean accepted tokens each iteration. Further in- creasing k has adverse effect, because it will become very hard for the model learn to draft too many tokens simultaneously given the model capacity, resulting in a drop of Tok..
Easily adaptable for existing models In many practical applications, generative models often pretrained with massive data, which exhibits very high performance. Developing a faster model from scratch to replace the pretrained model is highly challenging and typically requires substantial com- putational costs to reiterate the pretraining process. Otherwise, the quality of the new models is very likely to be compromised despite the increased speed, Table 5 shows. However, SpecDec be casily adapted accelerate existing pretrained models. Taking the BART-base (Lewis et al., 2020) model for the abstractive summarization task as an example, we can easily achieve 5.1 x speedup without compromising the generation quality only by initializing the Spec-Drafter with the BART- base encoder and training it with the BART-base distilled summarization training set.
4.3.2 Verification
We study the effect of Spec-Verification on the development set (i.e., newstest-2011 2013) of WMT14 EN >E in Table 4. Moderately increasing and 3 in the Spec- Verificciiaa not only leads to in- crease of mean accepted tokens (Tok.) and speed since AR verification becomes less strict but also improves the generation quality over greedy decod- ing. However, the generation quality may decrease if over-relaxed: the BLEU score will degrade from the peak of 26.97 to 26.58 when decoding with top-5 selection (i.e., / 5) and r = 5.0. Based on the results in the development set, select B = 3, 1.0 as Aprcc-Veifiicttinn hyperpa- rameters.
4.4 Practical Value
Retaining the behavior of the original model As introduced Section 1, one significant advan- tage of SpecDec is that does not develop a new
addition to the remarkable speedup results, we demonstrate SpecDec's additional advantages that

---

Models
Rouge-1
Rouge-2
Rouge-L
Speed
BART-base (b 5)
43.08
20.41
40.15
1.0x
AR
BART-base (b 1)
43.00
20.28
39.96
1.1x
GLAT+CTC (Qian et al., 2021)
37.76
45.24
14.08
46.20
33.69
[6.27
14.5x
DAT (Huang et al., 2022)
38.95
|4.05
16.11
44.17
35.43
44.53
14.1x
NAR
CMLM (Ghazvininejad et al., 2019)
37.59
|5.41
15.17
45.11
34,22
45.54
1.8x
RewriteNAT (Geng al., 2021)
39.12
43.88
16.24
44.04
35.74
44.22
3.1x
SpecDee
SpeeDec (k 25)
43.11 10.11
20.43
10.15
40.19
10.23
5.1x
Table 5: Results of different methods for Abstractive Summarization on CNN-DM (Hermann et al., 2015), The results on par with BART-base performance highlighted in orange, while blue denotes performance degradation. Compared to prior NAR methods, SpecDec can be easily adapted to accelerate the BART model only by downstream task fine-tuning.
Models
BLEU
coding. Since this work was proposed, we are pleased to see an increasing number of following studies (Leviathan al., 2023; Chen et al., 2023; Kim et al., 2023; Spector and Re, 2023; Zhang et al., 2023) acknowledge, explore and adopt this methodology to accelerate Transformer inference. Among them, Leviathan et al. (2023) use the same name as ours (i.e., Speculative Decoding), employ- ing a small AR model a drafter as well as advanced sampling algorithm. Chen et al. (2023) is similar to Leviathan et al. (2023) but it was the first to validate this methodology accelerate a large language model (i.e., 70B Chinchilla) with a 4B drafter model, thus receiving the most atten- tion. SpecInfer (Miao et al., 2023) proposed to uti- lize various boost-tuned small language models for joint drafting, to improve the speculation accuracy of the LLM's outputs. Besides, it introduces an advanced token tree verification strategy to verify all candidate token sequences in parallel. Distill- Spec (Zhou et al., 2023) further investigated the efficacy of knowledge distillation enhancing the alignment between the target model and the drafter in speculative decoding. addition to employ- ing additional models as drafters, there has also been some research that proposes various strate- gies to efficiently generate drafts from the LLM itself (Santilli et al., 2023; Zhang et al., 2023). All the following research strongly backs up the value of this original work.
Transformer-hase (greedy)
100.00
GLAT+CTC (Qian et al., 2021)
59.10
DAT (Huang et al., 2022)
63.79
CMLM (Ghazvininejad et al., 2019)
60.15
RewriteNAT (Geng et al., 2021)
65.42
Deep-Shallow (Kasai et al., 2020)
64.66
SpecDec 25)
86.52
Table 6: Relative BLEU score computed between the generation of the existing Transformer (i.c., the target model) and other models/approaches. SpecDec shows much better alignment with the target model's behavior than others.
faster model to replace the existing model. In- stead, it accelerates the existing model with mini- mal changes to its behavior. As shown in Table 6, the consistency (BLEU) of SpeeDec's generated results with the original model exceeds 85%, while that of a newly built fast NAR model is only around 55%. The characteristic of maintaining the behav- ior of the original model makes Spec even more valuable in practical applications because transi- tioning from a well-tested and mature model to one with substantially different behavior is risky, re- quiring extensive recalibration and various offline evaluations and online feedback in practice.
5 Related Work
Speculative Decoding We have demonstrated since early 2022 (see our arXiv preprints in 2022) that our proposed methodology, which formally introduces an independent model as a drafter com- bined with an advanced verification strategy to fully exploit speculative execution, promising and has potential to evolve into a de facto stan- dard in the future for efficient and lossless de-
Early Draft-then-verffy attempts This work is generalized version of our previously proposed (Input- Aggressive Decoding (Sun et al.,
Pee provide a detailed comparison between Leviathan (2023) and our work in Appendix G. l3 As our technical report (Ge et al., 2022b) in May 2022 discusses, the Input-guides Aggressive Decoding is indced a special case of Speculative Decoding.

---

2021) in Grammatical Error Correction (GEC), which assumes that the input is exactly the sen- tence to be generated the future and then verifies the whole sentence in parallel. Blockwise Decod- ing (Stern et al., 2018) inserted l feedforward heads on top of the Transformer decoder to gen- erate k positions in parallel and used the original head to verify these outputs. However, both the above studies did not fully investigate the poten- tial of this paradigm and thus failed to uncover its great value for efficient seq2seq generation: Sun et al. (2021) only works for tasks whose inputs and outputs are highly similar (e.g., GEC). Stern et al. (2018) overlooked the importance of drafting ac- curacy; as result, their underinvested prediction heads severely limit the acceleration results. In contrast, conduct thorough investigations and fully exploit speculative execution, refreshing the impression of its limited acceleration potential and revealing its real value in practice.
Conclusion
We present Speculative Decoding (SpecDec), the first work to explicitly embrace the idea of specula- tive execution for seq2seq generation acceleration with formal study and extensive discussion of both drafting and verification phases. Contrary to the common belief that an increase in model com- plexity tends to hamper inference speed, SpecDec's 's introduction of an appropriately invested auxiliary drafter model substantially speeds up Transformer inference, owing higher computational paral- lelism introduced by speculative execution better utilize computing resourees.
The remarkable acceleration performance, com- bined with the advantages demonstrated in our ex- periments, clearly illustrates that SpecDec is prac- tical acceleration method for model deployment in real-world applications. We hope that our pre- liminary study could draw more attention to this promising decoding paradigm that may potentially evolve into a de facto standard for efficient Trans- former decoding in the near future.
Non-autoregressive Decoding There is also an- other line of work named Non-Autoreeresssvv Decoding (NAR) (Gu et al., 2018), which de- codes multiple tokens in parallel compared with conventional AR, thus showing remarkable su- periority in inference efficiency. Recently, var- ious attempts have been made to improve the performance of NAR models, including training with alignment-based objectives (Libovický and Helcl, 2018; Ghazvininejad et al., 2020; Saharia et al., 2020; Gu and Kong, 2021; Shao and Feng, 2022), modeling dependencies between target to- kens (Ghazvininejad et al., 2019; Shu et al., 2020; Qian et al., 2021; Bao et al., 2021) and designing various model architectures (Zheng et al., 2021; Huang et al., 2022). As discussed in Section 4.4. replacing powerful pretrained model with NAR models in practice is challenging due the sub- stantial computational costs required to reiterate the pretraining process. Additionally, transition- ing from a well-tested and mature model to a new NAR model with significantly different behavior poses risks in practical applications. In contrast, our proposed SpecDec can be conveniently adapted to speed up existing AR models including high- performance pretrained models like BART with little effort. Moreover, SpecDec minimally alters the behavior of existing models, showeasing its ability to preserve reliable generation performance in real-world practical applications.
Limitations
Compared with conventional autoregressive de- coding, SpecDec introduces an extra Spec-Drafter module for ensuring its drafting accuracy, which brings additional memory cost at test time. There- fore, SpecDec is particularly suitable for infer- ence scenarios where GPU memory is abundant but there is an urgent need to improve latency it provides solution trading the surplus GPU memory for speed improvements. As thoroughly discussed Appendix B, such scenarios are very common in practice. Most importantly, memory is no longer the bottleneck for practical model de- ployment. With the emergence and maturity of various data/tensor/pipeline parallelism techniques, the addition of more GPUs can easily address mem- ory issues, which is also why models continue to grow larger. In contrast, latency remains an in- escapable bottleneck in model deployment that can- not resolved merely by increasing the number of machines. Therefore, we believe the increased memory consumption may not severely affect its practical value.
Acknowledgements
We thank Fan Yang, Lingxiao Ma, and Lidong Zhou in Microsoft Research for their constructive

---

comments on this work. This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.
Marjan Ghazvininejad. Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel de- coding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIP-IICNIL 2019, Hong Kong. China, November 3-7, 2019, pages 6111- 6120. Association for Computational Linguisties.
References
Yu Bao, Shujian Huang. Tong Xiao, Dongqi Wang, Xinyu Dai, and Jiajun Chen. 2021. Non- autoregressive translation by learning target categor- ical codes. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11. 2021, pages 5749-5759 Association for Com- putational Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non- Non-autoregressive neural machine translation. In International Conference on Learning Representations.
Jiatao Gu and Xiang Kong. 2021. Fully autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics:
ACL/ICCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IICLLP 2021 of Findings of ACL, pages Association for Computational Linguisties.
Charlic Chen, Sebastian Borgeaud, Geoffrey Irv- ing. Jean-Bapte -aptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. CoRR, abs/2302.01318.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay. Mustafa Suleyman, and Phil Blunsom. 2015. Teach- ing machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693-1701
Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, and Ming Zhou. 2020. Improving the efficiency of grammatical error correction with erroneous span detection and correction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7162-7169.
Fei Huang. Hao Zhou, Yang Liu, Hang Li, and Minlie Huang 2022. Directed acyclic transformer for non- autoregressive machine translation, In International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA. vol- ume 162 of Proceedings Machine Learning Research, pages 9420-9428. PMLR.
Tao Ge, Si-Qing Chen, and Furu Wei. 2022a. Edge- former: A parameter-efffiient transformer for on- device seq2seq generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, 2022, pages 10786-10798. Association for Computational Lin- guistics.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Non-autoregressive machine trans- lation with disentangled context transformer. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5144-5155. PMLR
Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, and Furu Wei. 2022b. Lossless acceleration for seq2seq gen- eration with aggressive decoding. arXiv preprint arXiv:2205.10350.
Xinwei Geng, Xiaocheng Feng, and Bing Qin. 2021. Learning rewrite for nun-attoreer neu- ral machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event 7 Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3297-3308. Association for Computa- tional Linguisties.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2021. Deep encoder, shal- low decoder: Reevaluating pon-autoregreessvve ma- chine translation. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non-autoregressive machine translation. Proceedings of the 37th International Conference on Machine Learning, ICML, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3515-3523,
Sehoon Kim, Karttikeya Mangalam, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. 2023, Speculative decoding with big lit- decoder. CoRR, abs/2302.07863..
Yoon Kim and Alexander M Rush. 2016. Sequence- level knowledge distillation. arXiv:1606.07947.
arXiv preprint
PMLR.

---

Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4. 2018, 8, pages 1173-1182. Association for Computational Linguis- tics.
for Computational Linguistics, July 6-12, 2002. Philadelphia, PA, USA, pages 311-318. ACL.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018. Belgium, Brussels, October 31 November 2018, pages 186-191, Association for Computational Lin- guistics.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via spec- ulative decoding. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. volume 202 of Proceedings Machine Learning Research, pages 19274-19286. PMLR.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang. Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for autoregressive neural machine translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLJUCNNP 2021, (Volume
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequene-to-sequencc pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. pages 7871-7880. Association for Computational Linguistics.
1: Long Papers). Virtual Event, August 1-6, 2021, pages 1993-2003, Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural frame- work for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing, FMNLP 2020, Online, November 16-20, 2020, pages 2685-2702 Associ- ation for Computational Linguistics.
Jindrich Libovický and Jindrich Helel. 2018. End- to-end non-autoregressive neural machine transla- tion with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4. 2018, pages 3016-3021 Association for Computational Linguis- tics.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. eooeeuvvreee machine translation with latent alignments. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1098- 1108. Association for Computational Linguisties.
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize: figh-precision text editing, arXiv preprint 1rXiv:1909.00 187.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. 2023. Accelerating transformer inference for translation via parallel de- coding. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages I 2235-12355. As- sociation for Computational Linguisties.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng. Zeyu Wang, Rae Ying Yee Wong, Zhuom- ing Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2023. Specinfer: Accelerating generative LLM serving with speculative inference and token tree verification. CoRR, 00s/2230500781.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. Gector gratmaagaammtic error correction: tag, not rewrite. arXiv preprint arXiv:2005.1 125922.
Robin M. Schmidt, Telmo Pires, Stephan Peitz, and Jonas Lööf. 2022. Non-autoregreesive neural ma- chine translation: call for clarity. CoRR, abs/2205.10577.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium. Brussels, October 31 November 1, 2018, pages 1-9. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association
Chenze Shao and Yang Feng. 2022. Non-monotonic latent alignments for ete-based non- autorreggeeeeeeeeee machine translation. CoRR, abs/2210.03953.

---

Raphael Shu, Jason Lee. Hidcki Nakayama, and Kyunghyun Cho. 2020. varinlev non- autoregressive neural machine translation with deterministic inference using a delta poste- rior. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conterence, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAT 2020, New York, NY. USA, February 7-12, 2020, pages 1846-8855 AAAI Press.
Jun Zhang, Jue Wang, Huan Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023. Draft & ver- ify: Lossless large language model acceleration via self-speculative decoding. CoRR, abs/2309 161800
Zaixiang Zheng, Hao Zhou, Shujian Huang. Jia- jun Chen, Jingjing Xu, and Lei Li. 2021. Du- plex sequence-fo-sequenee learning for reversible machine translation. In Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 14, 2021, virtual, pages 21070-21084.
Benjamin Spector and Chris Re. 2023. Accelerating LLM inference with staged speculative decoding. CoRR, abs/2308.046633.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean- cannFFrançois Kagy, and Rishabh Agarwal. 2023. Distillspec: Improving speculative decoding via knowledge distillation.
Mitchell Stern, Noam Shazeer, and Jakob Uszko- reit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 10107-10116.
Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021. Instantaneous grammatical error correc- tion with shallow aggressive decoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Natural
11th International Joint Conference Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5937-5947. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ou, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien- Hsin S. Lee, Bugra Akyildiz, Maximilian Balan- dat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim M. Hazelwood. 2022. Sustainable ronmental implications. challenges and opportunities. In Proceedings of Machine Learning and Systems 2022, ML.Sss 2022, Santa Clara, CA, USA, August 29 September 1, 2022, misys.org.
Nan Yang. Tao Ge. Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Inference with reference: Lossless ac- celeration large language models. arXiv preprint arXiv:2304.0448

---

Hyperparameter
Value
The largest intermediate variables/results dur- ing inference:
devices
8 Nvidia V100 GPU
label smoothing
0.1
For short sequence (c.g.. sentence-level inputs/outputs in MT tasks), the largest intermediate variable is the output tensor after the Spec-Drafter's/Ar model's vo- cabulary projection layer - - [V] k where B is the batch size, [V] is the vocabulary size and k is the block size. Compared with the memory cost for stor- ing the Spec-Drafter" weights, this part is usually smaller. Also B k tokens be easily divided into small batches (e.g., -sottma-- bbtch in fairseq) for vocabulary projection to avoid massive memory cost in case B V k large.
max tokens
4096
update frequeney
4
dropout rate
10.1. 0.2, 0.3]
max source positions
1000
max target positions
1000
Adam Ir
5 x 10-4
Adam 8:
0.9
Adam A²2
0.999
Adam
1 X 10-6
Ir-scheduler
inverse square
warm-up Ir
1 x 10
weight decay
0.01
clip norm warmup updates
5.0
10000
max updates
300K
Table 7: Hyper-parameten and settings Spec-Drafter.
For long sequence (e.g., paragraph- level inputs/outputs in summarization tasks), the largest intermediate variable becomes the tensor for storing self- attention computation whose size in- creases quadratically with S (S is the sequence length). This variable accounts for the largest memory cost for stor- ing intermediate results in both AR and SpecDec. Therefore, in this case, this part does not introduce additional mem- ory cost compared with AR.
Appendix
A Hyperparameters
Hyper-parameters of training our proposed Spec- Drafter are listed in Table 7. Following Vaswani et al. (2017) and Ott et al. (2018), we also average model parameters from the last 10 checkpoints.
B Memory Analysis
B.1 Additional Memory Cost by SpeeDec
The peak memory footnote of Spec during in- ference mainly comes from three parts:
Table 8 and Table 9 show the comparisons of peak GPU memory footprint (MB) between SpecDec and AR (during inference) on the above two scenarios (i.e., MT and summarization). The results are consistent with our analysis above:
Static AR model's weights
Static Spec-Drafter's weights
Intermediate variables/results
The majority of the additional mem- ory cost (i.e., AMemory) is for storing the Spec-Drafter's weights and the ad- ditional memory cost is not very likely to significantly increase as the batch size or sequence length increases.
Compared with AR, the additional memory cost of SpecDec comes from the last two parts. While the static Spec-Drafter's weights account for the ma- jority of the additional memory cost, the additional cost for storing intermediate variables negligible because the Spec-Drafter and AR model decode alternatively during inference. Compared with AR, SpecDec's is additional intermediate variables/results include:
Our experiments above pre-loaded both the Spec- Drafter and AR model. In fact, it is also possible to load the static weights of the AR model and Spec- Drafter in lazy loading manner in the meantime of GPU computation to save memory they run alternatively. However, is usually unnecessary in practice, because for a seq2seq model deployed on modern GPUs for online service, is latency rather than memory that is the performance bot- tleneck. See the next section for more discussion. TTTsseed with torch. cuda. www.aaaaa_aeeeeeeeeeeaee max__emorr_allocatede
The Spec-Drafter's ' last encoder layer's repre- sentation that will not be freed until decoding finishes, which is equal to B- S d where B is the batch size, S is the sequence length and d is the dimension of the model. This part is ac- tually negligible: for example, when B 32, S = 128, d = 512, this part's memory cost is only 8MB (fp32)/ / 4MB (fp16).

---

Batch Size
Models
Model Weights
32
AR (greedy)
232.4
243.2
271.7
301.4
366.4
494.6
SpecDee (ls 25)
477.8
491.5
519.8
559.2
634.1
782.5
AMeorry
245.4
248.3
248.1
247.8
267.7
287.9
Table 8: Peak GPU memory utilization WMT14 EN-DE translation dataset. The results obtained with fp32
single Nvidia P100 GPU.
Models
Model Weights
Memory Cost
only less than 2GB far below A40 GPU's memory capacity (i.e., 48GB "))
AR (greedy) 534.6
696.9
SpeeDec
1089.6
1264.6
For abstractive summarization, even if the batch size increases to 32, its memory cost is still less than 50% utilization of 1 A40 GPU but its latency is already close up to 5 seconds that is too long for online service in practice.
AMemory
555.0
567.7
Table 9: Peak GPU memory utilization CNN-DM with batch size 1 with fp32 on single Nvidia P100 GPU.
To sum up, we now understand latency the bot- tleneck of seq2seq models for online deployment in most cases. Therefore, we do not think addi- tional memory cost by SpecDec will undermine its practical value; instead, we think a significant lossless acceleration even at the cost of memory (i.e., time-memory trade-off) is much more mean- ingful than the acceleration at the cost of quality, which should be the right path that we need to pay more attention to given much memory headroom on modern GPUs.
B.2 Memory Is Rarely the Bottleneck
To understand the performance bottleneck of on- line deployed seq2seq models, test the latency and memory cost of T5- T (around 770M pa- rameters) with fp16 on Nvidia A40 GPU running greedy decoding in the machine translation and ab- stractive summarization task, and show results in
Table 10 and 11.
Batch Size
Statistics
32
Models
BLEU
SacreBL.EU
COMET
Latency (8)
1.0.A
1.4A
AR-buse (b = 5)
28.89
28.2
51.90
AR-buse (b = 1)
28.73
28.0
51.53
Memory Util. (MB)
1482
2003
SpecDec 25)
28.93
52.10
Memory Util. (%)
3.0
4.0
Table 12: SacreBLEU and COMET scores on WMT14 EN-DE.
Table 10: Latency and peak GPU memory utilization of L5-Large on WMT14 EN-DE.
C SacreBLEU and COMET Scores
Batch Size
Statistics
32
Despite tokenized BLEU scores, we also report SacreBLEu (Post, 2018) and COMET (Rei et al., 2020) scores in Table 12 and 13 to provide reference for future research. SpecDec can also achieve performances par with the AR model with the evaluation in sacreBLEU and COMET. Schmidt al. (2022) pointed out that inconsisten- cies in the use of tokenized BLEU lead to devi- ations of up to 1.8 BLEU points. Therefore, we
Latency (s)
2.7A
4.7
Memory Util. (MB)
2999 6.2
7230
Memory Util. (%)
15,0
Table 11: Latency and peak GPU memory utilization of Ta-Large on CNN-DM.
For MT, T5-large's latency is over 1 second which is actually too long to be accepted because most MT engines in practice require the latency to be less than 100ms. However, its memory cost is
"It also easily scale 96GB NVLink connection of multiple GPUs multi-node connec- tion.
with NVIDIA
In practice, T5-large is rarely deployed for online service hecause it is too large and expensive to serve.
17 Whttps: /gi thub. httpm/mpmsssstttiiiioes 19 "Obaained with 20 come-t-aaaaa da from version 1.1.0.

---

Models
EN-DE
DE~EN
EN-+R
RO~EN
Transformer-Be base
2
32.53(3.11)
34.96(44.0)
34.86(34.21)
Transformer-base (b
28.73(28.0")
32.18(31.7)
34.83(33.9))
34.65(33.91)
SpeeDec (k 25)
28.93(28.2))
32.55(32.1')
35.03(34.3)) 35.45(34.5")
Table 13: BLEU and SacreBLEU (denoted by !) scores WMT14 EN-DE and WMT16 EN-RO benchmarks.
Decoding Algorithm
BLEU
Speed
AR (greedy)
32.23
1.0x
Leviathan al. (2023) (0.IB drafter, T5-Small)
32.23
2.8x
Leviathan al. (2023) (0.8B drafter, TS-Large)
32.23
1.6x
SpeeDec (k = 25) (0.5B drafter)
32.31
4.9x
Table 14: Comparison of SpecDec and Leviathan al. (2023) in accelerating T5-XXL (IIB) for WMT14 EN-DE translation.
student models and greedy decoding much more common than beam search because it is more cost-effective it not only runs faster than beam search but also achieves decent performance with a student model trained through KD (as Point addressed)
Sample Number
200
50
Sperddup
3. Beam search is also an approximate and heuristic solution, which is not golden rule. In fact, Spec- Verification works in similar way beam search is also an approximate and heuristic solution by considering n-best and scores, which can be considered as an approximation of beam search. As shown Table 1, it achieves comparable performance to beam search but much faster (4x ~ 6x).
Figure 6: Single sentence speedup distribution by SpecDec 25) compared with Transformer-hase (b 5). The results obtained with WMT14 EN->DE.
recommend that future research use sacreBLEU when comparing with our work.
Speedup Distribution
Figure presents SpecDec's speedup distribution of single sentence on the WMT14 EN-DE test set (which has 3,003 sentences in total), showing that most sentences are translated with a 3x~77 speedup compared to AR beam search, while some rare cases can even achieve over 10x~11x speedup.
F Carbon Emission
SpecDee introduces extra computational overhead, which leads to an increase GPU power consump- tion. However, it can substantially reduce GPU hours owing to its high efficiency. We compare GPU power consumption and GPU hours of autore- gressive decoding (AR) and SpecDec for translat- ing 3000 sentences in Table 15.
E Discussions of Beam Search
We follow formula for Wu et al. (2022) to calculate the total energy consumption and carbon emitted: = P x PUE, where we set PUE (Power Usage Effectiveness) at 1.1. COzeq
For possible concerns that SpecDec may not apply beam search, we make three points here:
1. As Kim and Rush (2016) mentioned, knowl- edge distillation largely decreases the perfor- mance gap of beam search and greedy decod- ing. In practice, greedy decoding can actually be comparable beam search results after KD.
E, where 0.38 COzee/Wh is the carbon intensity factor that is set based on the US national average.
0.385g COzeq/Wh
According Table 15, while SpecDec's GPU power consumption is 28% higher, its GPU hours are 540% shorter than autoregressive decoding. As a result, SpecDec's total energy consumption
2. In practical online deployment, KD almost used by default for enhancing the results for

---

GPU Avg.
Total Energy
Carbon emitted
Models
GPU Type
GPU-hours (1)
Power consumption (P)
consumption (B)
(COzeq)
AR (greedy)
Nvidia P100
86W
0.27h
25.54Wh
9.83g
SpecDec (k 25
Nvidia P100
110W
0.05h
6.05Wh
2.33g
Table Carbon footprint autoregressive decoding and SpecDec using same P1OD GPU device, follow Wu et al. (2022) compute the carbon emission of the two decoding strategies under the same device. The results were obtained on WMT14 EN-DE.
SouRCE
According the details provided, the tunnel had yet been put into
Draft
Nach den Angahen Angahen war der Tunnel noch nicht
Verify
Nach den vorliegenden war-war-der Tannel www.-oihhh--h-eeeeeeeeeeeeeeeeee in Betrieb
Draft
Nach den vorliegenden Angaben war der Tunnel noch nicht in Betrieb genommen [EOS] Nach den vorliegenden Angaben war der Tunnel noch nicht in Betrieb genommen worden
Verify
Draft
Nach den vorliegenden Angaben war der Tunnel noch nicht in Betrieb genommen worden, [EOS]
Verify
Nach den vorliegenden Angahen war der Tunnel noch nicht in Betrieh genommen worden [EOS]
RESULTS
Nach den vorliegenden Angaben war der Tunnel noch nicht in Betrieb genommen worden.
Table 16: Examples from the WMT14 English-German translation task. At each iteration, Draft and Verify are the outputs the Spec-Drafter and the AR verification, respectively. Tokens within red blocks are the bifurcation positions. The verification pieces after the bifurcation annotated strikethrough. The highlighted parts are translations of previous iterations. The hyperparameters are h 10, top-3, T = 1.0. The output pieces after the [EOS] token is omitted in the table.
is 23.7% of autoregressive decoding, resulting in 4.2x reduction of carbon emission.
ity degradation, while Leviathan et al. (2023)'s 's speedup results are lower than 3x. The rea- for our approach significantly outperforming Leviathan et al. (2023) are twofold:
Therefore, Spec not only does not increase carbon emission but actually significantly reduces it, making it a more environmentally friendly op- tion for inference.
Our Spec-Drafter is specially learned draft for accelerating the target model with the spec- ulation execution idea: its draft results are better aligned with the target model's results.
G Comparison with Other Work
G.1 Speculative Decoding with an Autoregressive Drafter
Our approach adopts fast deep-encoder- shallow-decoder architecture as well as a non- autoregressive approach to generate drafts, which is significantly more efficient than their autoregressive drafting method.
Following this work, some subsequent research (Leviathan et al., 2023; Chen et al., 2023) has also explored using AR models (e.g., smaller language models) independent drafters to accelerate in- ference. However, for seq2seq generation, the ac- celeration effect of AR drafting is severely limited, resulting in an end-to-end speedup at 1.6 1.6xx2.88. which is far lower than our 4.9x speedup.
G.2 Speculative Decoding in Special Cases
Our previous paper (Sun et al., 2021) and techni- cal reports (Ge et al., 2022b; Yang et al., 2023) present extensive empirical studies of Speculative Decoding in special cases (e.g., text editing eetrieval-augmentte generation) and compare it with competitive approaches (Malmi al., 2019; Omelianchuk et al., 2020; Chen al., 2020) in those scenarios. We recommend interested readers to refer to our previous work for further insights into these specific applications.
Table 14 illustrates the detailed comparison of our re-implemented Leviathan et al. (2023) and ours in accelerating T5-XXL (11B) for WMT14 EN-DE translation. We implemented our Spec- Drafter with 24-layer encoder and 6-layer de- coder for comparison. Its embedding/FTN dimen- sion/theads are 1024/4096/16. As shown in Ta- ble 14, our approach uses a 0.5B Spec-Drafter to achieve an almost 5x speedup without qual-

---

H
Case Study
Table 16, we represent an example to illustrate how SpecDec generates translations. In the first iteration, the outputs of the Spec-Drafter are non- autoregressive with multi-modality problems like "Angaben Angaben". The verifier accepts tokens of "Nach den" and replaces the inappropriate trans- lation "Angaben" with "vorliegenden". All the drafted tokens after the bifurcation position (i.e. marked as red tokens Table 16) are all discarded. the second iteration, Spec- Verifieeiiic finds the bifurcation at the last position, thus all tokens be- fore this position are accepted. After 3 iterations, the decoding is finished since the [EOS] token is found.


Reviews:
[{"review_id": "660TmPkeXk", "reviewer": "Reviewer_u3hL", "summary": "", "questions": "1. Is this method also applicable to decoder-only models such as GPT or Llama?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review comments. We appreciate your recognition of our work (very well written + novel and sound method + promising and significant results). We understand your questions and concerns in your review comments, but we believe **these minor issues should not overshadow our merits and contributions (with a score of 3)**. We hope that our following responses will help you better interpret our **merits, originality, and significance** and resolve your **misunderstanding**.\n\n&nbsp;\n\n***Q1: There's a lack of comparison to other fast decoding algorithms such as \"early-exiting\" (Confident Adaptive Language Modeling, Shuster, 2022), or \"speculative decoding\" which uses one small transformer (draft) and one larger transformer (verify). Please refer to \"Fast Inference from Transformers via Speculative Decoding (Leviathan et al 2023), and \"Accelerating Large Language Model Decoding with Speculative Sampling\" (Chen et al 2023) for speculative decoding.***\n\nR1: As you mentioned Leviathan et al (2023) and Chen et al (2023), we believe you must be aware of the methodology’s significance and **the surge in the Speculative Decoding/Sampling/Inference research** in the recent year:\n\n[1]. Leviathan et al (2023): Fast Inference from Transformers via Speculative Decoding\n\n[2]. Chen et al (2023): Accelerating Large Language Model Decoding with Speculative Sampling\n\n[3]. SpecTr: Fast Speculative Decoding via Optimal Transport\n\n[4]. SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification\n\n[5] . ...\n\nHowever, we believe you may **misunderstand** that we followed Leviathan et al (2023) and Chen et al (2023). Instead, the fact is that **this work is much earlier than Leviathan et al (2023) and Chen et al (2023). It is the first work** that proposes the name \"Speculative Decoding\" which explicitly **leverages the idea of speculative execution** to accelerate decoding. It is **the origin of** the above Speculative Decoding/Sampling/Inference research surge. Both Leviathan et al (2023) and Chen et al (2023) as well as other above papers **followed and cited** this work (You can consult with the AC/SAC to confirm this fact). We hope this can address your misunderstanding in terms of this work's originality and significance and **reconsider your score on this work based on the context that it is prior to Leviathan et al (2023) and Chen et al (2023)**.\n\nAlthough **this work indeed predates** Shuster et al (2022), Leviathan et al. (2023) and Chen et al. (2023), we understand the importance of benchmarking our approach against these more recent methods for a comprehensive evaluation, as you suggested. To address your comment and further demonstrate the effectiveness of our method, **we have conducted additional experiments comparing our SpecDec approach with the mentioned algorithms**:\n\nThe following results are the comparison of our reimplemented Leviathan et al. (2023) and ours in accelerating T5-XXL (11B) for WMT EN-DE translation:\n\n| Decoding algorithm (T5-XXL) | BLEU      | Speed    |\n| :--------------------------------------- | :-------- | :------- |\n| Autoregressive Decoding         | 32.23     | 1.0x     |\n| Leviathan et al (0.1B drafter, T5-Small) | 32.23     | 2.8x     |\n| Leviathan et al (0.8B drafter, T5-Large) | 32.23     | 1.6x     |\n| Ours (0.5B drafter)                   | **32.31** | **4.9x** |\n\n| Models                 | Enc,Dec | d_model | d_ff | #heads | #Params |\n| :--------------------- | :------ | :------ | :--- | :----- | :------ |\n| SpecDec (0.5B drafter) | 24,6    | 1024    | 4096 | 16     | 0.5B    |\n\nOur approach uses a 0.5B Spec-Drafter to achieve an almost 5x speedup without quality degradation, while Leviathan et al. (2023)'s speedup results are lower than 3x. The reasons for our approach significantly outperforming Leviathan et al. (2023) are **twofold**:\n\n1. Our Spec-Drafter is specially learned to draft for accelerating the target model with the speculation execution idea: its draft results are **better aligned** with the target model's results.\n2. Our approach uses a fast deep-encoder-shallow-decoder architecture and a non-autoregressive approach to generate drafts, which is significantly **more efficient** than their autoregressive drafting method.\n\nThe following results are the comparison of our reimplemented Shuster et al. (2022) and ours in accelerating a Transformer-base model for WMT DE-EN translation:\n\n| Models               | BLEU      | Speed    |\n| :------------------- | :-------- | :------- |\n| Transformer-base     | 28.89     | 1.0x     |\n| Shuster et al. (2022) | 28.16     | 2.4x     |\n| SpecDec              | **28.93** | **5.1x** |\n\nWhile Shuster et al. (2022) achieve 2.4x speedup, it suffers from serious quality degradation. In contrast, our SpecDec can achieve 5.1x speedup without quality degradation, **showing the superiority over Shuster et al. (2022) in terms of both generation quality and speedup**.\n\n\n&nbsp;\n\n***Q2: It is hard to implement the SpecDec of this paper because the method needs training while other methods usually only need to infer the existing models.***\n\nR2: All SpecDec methods, including this work, Leviathan et al (2023), Chen et al (2023) and SpecInfer[5], require a draft model for speculation to speed up the process. However, **it is not true** that other methods only need to infer the existing models -- The fact is **only Leviathan et al. (2023) employ the existing models as a drafter**. As we show the results in our response to Q1, **using an existing model as a drafter is not the best choice** because existing models **fall short in terms of both draft quality and draft efficiency** compared to a specially designed drafter for speculation. This is also **the reason why not only our work but also Chen et al (2023) and SpecInfer[5] use a specialized model for drafting**: e.g., Chen et al (2023) specially train **a 4B model with only 8 layers** -- a shallow and wide model -- as a drafter, as shallow and wide models have higher draft efficiency. (FYI, existing models at the 4B scale are typically much deeper -- e.g., 32 layers or more.)\n\nMoreover, even the GPT-4 is rumored to exploit the idea of speculative execution to accelerate inference, according to the famous leak of the GPT-4 details (https://news.ycombinator.com/item?id=36710170). If this rumor is true, we believe its drafter model would not be an existing model (deep and slow); instead, it is more likely to be a specialized drafter model that is fast and better aligned with the target model.\n\nThe feasibility, reliability, and practicality of training a new model for drafting have already been corroborated by the multiple papers mentioned above. We believe that **offline training of a specialized drafter model to better accelerate the online inference** has significant practical significance and value. Our comprehensive **opensourced** code, data, and models can help anyone easily implement our idea for easy reproduction.\n\n&nbsp;\n\n***Q3: It is not sure that this would work on a larger scale such as a model larger than or around 10B (scalability). They only tried very small models (12 enc, 2 dec) Would this be also 5X speed up at 10B or 100B models?***\n\nR3: This work tests on 6+6-512 (Transformer-base), 6+6-1024 (Transformer-big), 12+2-512/1024, and BART for fair comparison because most previous work on seq2seq generation acceleration reports results with these model sizes. **Our approach can of course scale to large models**:\n\nOur results in our response to Q1 already indicate that **when applied to 11B models, SpecDec can also achieve a promising 5x speedup** even with a much smaller Spec-Drafter (0.5B), demonstrating that **the effectiveness of SpecDec is not affected by the model size.**\n\nThese results are also consistent with the observations of Chen et al. (2023), which uses the speculation idea to accelerate a 70B decoder-only model, confirming that the speculative execution methodology's effectiveness is not affected by the model size. **We'll add these results in the revised version.**\n\n&nbsp;\n\n***Q4: Is this method also applicable to decoder-only models such as GPT or Llama?***\n\nR4: The target model to be accelerated by SpecDec **can be both enc-dec and dec-only** architectures because **the target model is used as the verifier in SpecDec and it is forwarded in the same way as it is trained**. However, it is notable that the Spec-drafter in our submission that uses a non-autoregressive model better fits seq2seq tasks for maximizing the acceleration effect. When applied to non-seq2seq tasks like next-word prediction, the Spec-drafter may need to be accordingly adjusted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is very well written and well structured. It is easy to understand what they did if you have enough knowledge on Transformers.\n2. Changing the AR decoding side to be NAR (or semi-AR) and increasing a fast encoder's burden makes sense.\n3. Combining the 3 contributions I wrote above is a novel approach.\n4. The results looks promising as it shows around 5 times speed up", "reasons_to_reject": "1. There's a lack of comparison to other fast decoding algorithms such as \"early-exiting\" (Confident Adaptive Language Modeling, Shuster, 2022), or \"speculative decoding\" which uses one small transformer (draft) and one larger transformer (verify). Please refer to \"Fast Inference from Transformers via Speculative Decoding (Leviathan et al 2023), and \"Accelerating Large Language Model Decoding with Speculative Sampling\" (Chen et al 2023) for speculative decoding.\n2.  It is hard to implement the SpecDec of this paper because the method needs training while other methods usually only need to infer the existing models.\n3. It is not sure that this would work on a larger scale such as a model larger than or around 10B (scalability). They only tried very small models (12 enc, 2 dec) Would this be also 5X speed up at 10B or 100B models?", "questions_to_authors": "1. Is this method also applicable to decoder-only models such as GPT or Llama?", "paper_topic_and_main_contributions": "This paper describes how to reduce the latency of AR decoding (especially in seq2seq format.)\nThey suggest draft-and-verify speculative decoding.\nWhat they contributed are\n1. Accurate draft by Masked Language Modeling in decoder conditioned by a few leftward contexts\n2. Fast draft by reducing the number of decoder layers and reallocating them to the encoder side.\n3. Alleviating the rigid verify that only checks top-1 AR result to top-beta so verify wouldn't be an obstacle to fast decoding.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "63fuF6Gj7b", "reviewer": "Reviewer_c6y8", "summary": "", "questions": "1. Latency is one of the two major design principles claimed mentioned in this paper. However, there is no system profiling analysis to support this thrust. There are only a few latency numbers shown in the paper without discussions on what the computing platform is. What is the computing system configuration? GPU types? GPU numbers? GPU cache memory? CPU types & Numbers? Threads? Parallelism? Software package? Compiler support? Please provide the details.\n\n2. This paper lacks a main discussion on the comparison with state-of-the-art acceleration techniques. Please provide quantitative analysis with other acceleration techniques, including weight pruning, head pruning, token pruning, knowledge distillation, and quantization. There are many works on each technique.\n\n3. Due to the introduction of an additional Spec-Drafter module, SpecDec may incur additional memory costs at test time. Could you provide a detailed analysis and comparison of the memory overhead introduced by the Spec-Drafter module？eg. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding (ICLR 2016)\n\n4. The performance of SpecDec might be sensitive to the choice of hyperparameters, possibly adding to the complexity of tuning and optimization. Please investigate the impact of different hyperparameters on performance through a series of experiments.\n\n5. Although SpecDec is suitable for a variety of tasks, its memory requirements may make it unsuitable for inference scenarios where memory is constrained. Please do some experiments to Study how SpecDec could be adapted to fit memory-constrained environments. Such as Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks (ISSCC 2016)\n\n6. Could you do some In-depth analysis of the relationship between speed gains and potential quality loss. Extremely Fast Neural Machine Translation with Lightweight Convolutions\" (EMNLP 2019)\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review comments. However, there is **a serious misunderstanding** regarding the memory cost issue and other concerns you raised. We hope our explanation and responses can resolve your misunderstanding and concerns, and improve your impression and evaluation of our paper:\n\n&nbsp;\n\n***Q1: Due to the introduction of an additional Spec-Drafter module, SpecDec may incur additional memory costs at test time. Although SpecDec is suitable for a variety of tasks, its memory requirements may make it unsuitable for inference scenarios where memory is constrained.***\n\nR1: We have acknowledged in the limitations section that SpecDec introduces memory overhead as a trade-off for accelerating decoding. However, the benefits of our approach, including significant inference speedup and substantial improvements in the latency-throughput curve, **far outweigh the memory overhead**. As extensively discussed in Appendix B, the memory overhead introduced by SpecDec primarily stems from storing the weights of the draft model. Generally, this additional memory consumption is not excessive and does not increase significantly with the sequence length or batch size. Most importantly, memory is no longer the bottleneck for practical model deployment. With the emergence and maturity of **various data/tensor/pipeline parallelism techniques**, the addition of more GPUs **can easily address memory issues**, which is also why models continue to grow larger. In contrast, latency remains an inescapable bottleneck in model deployment that cannot be resolved merely by increasing the number of machines.\n\n**Precisely because memory is no longer the primary bottleneck in practical model deployment**, and the **urgency of addressing latency** issues has increased, there has been a growing interest in research **following our work to explore the methodology** of introducing a draft model to improve transformer inference. As **the first work** to propose the name \"Speculative Decoding\" and explicitly leverage the idea of speculative execution (You can consulate with SAC/AC to confirm this fact), **our approach has been acknowledged** by several recent papers studying this methodology:\n\n1. Accelerating Large Language Model Decoding with **Speculative** Sampling. (Deepmind)\n2. SpecTr: Fast **Speculative** Decoding via Optimal Transport. (Google Research)\n3. SpecInfer: Accelerating Generative LLM Serving with **Speculative** Inference and Token Tree Verification\n\n**All these papers follow our work** to introduce a drafter model for speculation to accelerate inference, and none of them consider the additional memory consumption for increased speed as an unjustifiable trade-off --  they don't even provide a comprehensive analysis of memory consumption as we did in Appendix B. Among them, **Deepmind's work [1] even proposed to employ a 4-billion parameter model as a drafter** to boost the inference speed of a 70B model. Moreover, even **GPT-4 is rumored to exploit our idea of speculative execution** to accelerate inference, according to the famous leak of the GPT-4 details (https://news.ycombinator.com/item?id=36710170). \n\nAll the evidence clearly demonstrates that increasing memory consumption to enhance inference speed **is not an issue** in practice; otherwise, there would not be so many studies following this work to study the use of this approach to accelerate inference. Instead, we believe it will become **a highly practical solution and a key research direction** for optimizing inference speed in the future, as we suggested at the end of Appendix B.\n\nWe sincerely hope our explanation **can resolve your concerns** regarding memory cost. We also hope that you can **reevaluate the value and significance** of our work, taking into consideration **the latest technological advancements and developments following our proposed idea of speculative execution**.\n\n&nbsp;\n\n***Q2: Latency is one of the two major design principles mentioned in this paper. However, there is no system profiling analysis to support this thrust. There are only a few latency numbers shown in the paper without discussions on what the computing platform is. What is the computing system configuration? GPU types? GPU numbers? GPU cache memory? CPU types & Numbers? Threads? Parallelism? Software package? Compiler support? Please provide the details.***\n\nR2: Thank you for your comments. As we introduced in Section 4.1, we use 1 Nvidia P100 GPU based on fairseq implementation in our evaluation. We mainly report speedup instead of absolute latency because absolute latency is very sensitive to the computing devices and runtime. In contrast, the speedup is more insensitive. As we report in Table 1, Blockwise Decoding’s speedup is almost the same in both its original paper and our reimplementation. Therefore, we skip other details of computing devices and runtime. Specifically, the machine we mainly use for evaluation is **Azure Standard_NC24s_v2** (https://learn.microsoft.com/en-us/azure/virtual-machines/ncv2-series). It has 24 vCPU (Intel Xeon E5-2690 v4), 448GB memory. We use Pytorch 1.10 with cuda 11 as our runtime. **We are committed to adding the details in the revised version**.\n\n&nbsp;\n\n***Q3: This paper lacks a main discussion on the comparison with state-of-the-art acceleration techniques. Please provide quantitative analysis with other acceleration techniques, including weight pruning, head pruning, token pruning, knowledge distillation, and quantization. There are many works on each technique.***\n\nR3: While we acknowledge the relevance of various acceleration methods, such as weight pruning, head pruning, token pruning, knowledge distillation, and quantization, we position SpecDec as a novel decoding algorithm that is **orthogonal to these techniques**. Therefore, our main baselines and comparable counterparts are **those decoding approaches such as Autoregressive Decoding and Blockwise Decoding**.\n\nWe understand your interest in seeing the comparison to techniques of other categories (e.g., pruning, quantization, or distillation). However, we hope you can understand that it is a common practice for **a paper focusing on a specific acceleration category not to compare with techniques from other categories**. This is because different categories of acceleration techniques often address different aspects of model optimization, and directly comparing them might not provide meaningful insights into the specific improvement brought by the proposed method in its targeted category. For example, you would **rarely see** a paper on quantization comparing its results to pruning methods, a paper on pruning comparing its results to distillation methods, or a paper on distillation comparing its results to decoding algorithms. Some recent examples include:\n\n1. Quantization: \"Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference\" (CVPR 2018) - This paper focuses on comparing different **quantization** methods and **does not compare with pruning or distillation techniques**.\n2. Pruning: \"The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks\" (ICLR 2019) - This paper compares various **pruning** approaches and **does not compare with distillation or quantization techniques**.\n3. Distillation: \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\" (NeurIPS 2019) - This paper focuses on **knowledge distillation** methods and **does not compare with pruning or quantization techniques**.\n4. Decoding algorithm: \"Blockwise Parallel Decoding for Deep Autoregressive Models\" (NeurIPS 2021) - This paper presents a **decoding algorithm** and compares it with other decoding approaches, but **does not compare with pruning, distillation, or quantization methods**.\n\nWe hope this clarification helps in understanding **our choice of baselines and comparable counterparts** for our proposed SpecDec method.\n\n&nbsp;\n\n***Q4: The performance of SpecDec might be sensitive to the choice of hyperparameters, possibly adding to the complexity of tuning and optimization. Please investigate the impact of different hyperparameters on performance through a series of experiments.***\n\nR4: Our submission has **already provided detailed ablation studies and analyses of various hyperparameters in Sections 4.3.1 and 4.3.2** of the paper. Please refer to **Table 3 and Table 4** in Section 4.3.1 and 4.3.2 for the details.\n\n**It is not true** that it adds to the complexity of tuning and optimization. SpecDec only introduces 3 hyperparameters. Only k is involved during training, and it can be **safely set to 20~30 (see Table 3)**.\nFor beta and tau, **they are only used during inference**, like the beam b in beam search. Rather than increasing the tuning difficulty, SpecDec provides a solution to any quality-speed trade-off. By adjusting these hyperparameters, users can easily find a balance between the desired acceleration and the quality of the generated outputs. **This flexibility allows SpecDec to be adaptable to various application scenarios, where different speed and quality requirements might be needed.**\n\n&nbsp;\n\n***Q5: Could you do some In-depth analysis of the relationship between speed gains and potential quality loss?***\n\nR5: We have **already performed a detailed analysis of the speedup-quality tradeoff of SpecDec in Table 4 of the manuscript**. As Table 4 clearly demonstrates, as AR verification becomes less strict (i.e., the beta and tau increase), more tokens (Tok.) will be accepted and the speedup of SpecDec will accordingly increase. When AR verification becomes too relaxed, there will be a risk of degradation in quality. Therefore, as long as we do not make AR verification too relaxed (high beta and tau), **SpecDec can achieve significant speedup with almost no loss in generation quality**. This is the main difference between SpecDec and previous methods that lead to a loss in generation quality. Please refer to Table 4 and 5 for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. SpecDec achieves significant speed improvements while maintaining a generation quality comparable to traditional AR methods. This balance makes it an appealing solution for various applications.\n2. SpecDec demonstrates good performance across different seq2seq tasks and datasets, showing its wide applicability and robustness.\n3. The paper clearly explains the workings of the method through charts and detailed descriptions, aiding in the understanding of the novel approach.\n", "reasons_to_reject": "1. Due to the introduction of an additional Spec-Drafter module, SpecDec may incur additional memory costs at test time.\n2. The performance of SpecDec might be sensitive to the choice of hyperparameters, possibly adding to the complexity of tuning and optimization.\n3. Although SpecDec is suitable for a variety of tasks, its memory requirements may make it unsuitable for inference scenarios where memory is constrained.\n3. This paper lacks a main discussion on the comparison with state-of-the-art acceleration techniques.\n4. Latency is one of the two major design principles claimed mentioned in this paper. However, there is no system profiling analysis to support this thrust.\n", "questions_to_authors": "1. Latency is one of the two major design principles claimed mentioned in this paper. However, there is no system profiling analysis to support this thrust. There are only a few latency numbers shown in the paper without discussions on what the computing platform is. What is the computing system configuration? GPU types? GPU numbers? GPU cache memory? CPU types & Numbers? Threads? Parallelism? Software package? Compiler support? Please provide the details.\n\n2. This paper lacks a main discussion on the comparison with state-of-the-art acceleration techniques. Please provide quantitative analysis with other acceleration techniques, including weight pruning, head pruning, token pruning, knowledge distillation, and quantization. There are many works on each technique.\n\n3. Due to the introduction of an additional Spec-Drafter module, SpecDec may incur additional memory costs at test time. Could you provide a detailed analysis and comparison of the memory overhead introduced by the Spec-Drafter module？eg. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding (ICLR 2016)\n\n4. The performance of SpecDec might be sensitive to the choice of hyperparameters, possibly adding to the complexity of tuning and optimization. Please investigate the impact of different hyperparameters on performance through a series of experiments.\n\n5. Although SpecDec is suitable for a variety of tasks, its memory requirements may make it unsuitable for inference scenarios where memory is constrained. Please do some experiments to Study how SpecDec could be adapted to fit memory-constrained environments. Such as Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks (ISSCC 2016)\n\n6. Could you do some In-depth analysis of the relationship between speed gains and potential quality loss. Extremely Fast Neural Machine Translation with Lightweight Convolutions\" (EMNLP 2019)\n", "paper_topic_and_main_contributions": "1. The paper introduces a novel method for accelerating AR decoding in seq2seq generation, known as Speculative Decoding (SpecDec). This approach divides the decoding process into two stages: drafting and verifying.\n2. Extensive experiments validate the effectiveness and efficiency of SpecDec across various seq2seq tasks such as machine translation and summarization.\n3. Through ablation studies, the authors reveal the importance of different components within SpecDec, shedding light on how each part contributes to the overall performance.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding (ICLR 2016)\nEyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks (ISSCC 2016)\nExtremely Fast Neural Machine Translation with Lightweight Convolutions\" (EMNLP 2019)\n"}}, {"review_id": "62FQKk5zfH", "reviewer": "Reviewer_PLjh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review comments. However, there is a serious **misunderstanding regarding the memory cost issue** and other concerns you raised. We hope our explanation and responses can resolve your misunderstanding and concerns, and improve your impression and evaluation of our paper:\n\n&nbsp;\n\n***Q1: The proposed approach consumes more memory during decoding. I cannot agree that memory is rarely the bottleneck. It is indeed okay in this paper because of the setup. However, for larger models and longer sequences (which are exactly the current large transformer use case), why memory is not a problem?***\n\nR1: We have acknowledged in the limitations section that SpecDec introduces memory overhead as a trade-off for accelerating decoding. However, the benefits of our approach, including significant inference speedup and substantial improvements in the latency-throughput curve, **far outweigh the memory overhead**. As extensively discussed in Appendix B, the memory overhead introduced by SpecDec primarily stems from storing the weights of the draft model. Generally, this additional memory consumption is not excessive and does not increase significantly with the sequence length or batch size. Most importantly, memory is no longer the bottleneck for practical model deployment. With the emergence and maturity of **various data/tensor/pipeline parallelism techniques**, the addition of more GPUs **can easily address memory issues**, which is also why models continue to grow larger. In contrast, latency remains an inescapable bottleneck in model deployment that cannot be resolved merely by increasing the number of machines.\n\n**Precisely because memory is no longer the primary bottleneck in practical model deployment**, and the **urgency of addressing latency** issues has increased, there has been a growing interest in research **following our work to explore the methodology** of introducing a draft model to improve transformer inference. As **the first work** to propose the name \"Speculative Decoding\" and explicitly leverage the idea of speculative execution (You can consulate with SAC/AC to confirm this fact), **our approach has been acknowledged** by several recent papers studying this methodology:\n\n1. Accelerating Large Language Model Decoding with **Speculative** Sampling. (Deepmind)\n2. SpecTr: Fast **Speculative** Decoding via Optimal Transport. (Google Research)\n3. SpecInfer: Accelerating Generative LLM Serving with **Speculative** Inference and Token Tree Verification\n\n**All these papers follow our work** to introduce a drafter model for speculation to accelerate inference, and none of them consider the additional memory consumption for increased speed as an unjustifiable trade-off --  they don't even provide a comprehensive analysis of memory consumption as we did in Appendix B. Among them, **Deepmind's work [1] even proposed to employ a 4-billion parameter model as a drafter** to boost the inference speed of a 70B model. Moreover, even **GPT-4 is rumored to exploit our idea of speculative execution** to accelerate inference, according to the famous leak of the GPT-4 details (https://news.ycombinator.com/item?id=36710170). \n\nAll the evidence clearly demonstrates that increasing memory consumption to enhance inference speed **is not an issue** in practice; otherwise, there would not be so many studies following this work to study the use of this approach to accelerate inference. Instead, we believe it will become **a highly practical solution and a key research direction** for optimizing inference speed in the future, as we suggested at the end of Appendix B.\n\nWe sincerely hope our explanation **can resolve your concerns** regarding memory cost. We also hope that you can **reevaluate the value and significance** of our work, taking into consideration **the latest technological advancements and developments following our proposed idea of speculative execution**.\n\n&nbsp;\n\n***Q2: It seems that the proposed approach introduces more computational overhead during inference. Is there any comparison about Carbon Emission? My concern is, even if the approach can generate faster, it actually consumes much more energy than before. I may save some money but actually, it is not that good for the environment.***\n\nR2: Thank you for your very interesting question about carbon emission. As you commented, SpecDec introduces extra computational overhead, which leads to an increase in GPU power consumption. However, **it can substantially reduce GPU hours owing to its high efficiency**. We compare GPU power consumption and GPU hours of autoregressive decoding and SpecDec for translating 3000 sentences in WMT14:\n\n| Decoding approaches                | GPU Type | GPU Avg. Power consumption (P)  | GPU-hours (t) | Total Energy consumption (E) | Carbon emitted (CO2eq) |\n| ---------------------- | -------- | --------------------- | --------- | ----------------------- | ---------------------- |\n| Autoregressive Decoding |  Nvidia P100     | 86W                   | 0.27h      | 25.54Wh                  | 9.83 g                  |\n| SpecDec                |  Nvidia P100     | 110W                   | 0.05h      | 6.05Wh                  | 2.33g                   |\n\nThe total energy consumption and carbon emitted can be calculated as follows:\n$E=P \\times t \\times \\text{PUE}$ where we set PUE (Power Usage Effectiveness) at 1.1. $\\text{CO2eq}=0.385g*\\text{CO2eq/Wh} \\times E$ where 0.385g*CO2eq/Wh is the carbon intensity factor that is set based on the US national average.\n\nAccording to the above table, while SpecDec’s GPU power consumption is 28% higher, its GPU hours are **540% shorter than autoregressive decoding**. As a result, SpecDec’s total energy consumption is 23.7% of autoregressive decoding, resulting in **4.2x reduction of carbon emission**.\n\nTherefore, **SpecDec not only does not increase carbon emission but actually significantly reduces it, making it a more environmentally friendly option for inference.**\n\nWe appreciate your suggestion and will add these details in the revised version to further strengthen our work's merits.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Clear motivation and writing. Accelerating AR is a very important research topic in the sequence generation community.\n2) Strong results on machine translation and abstractive summarization show the effectiveness of the approach used.\n3) The two modifications Spec-Drafter and Spec-Verification are interesting and smartly designed.", "reasons_to_reject": "1) The proposed approach consumes more memory during decoding.\n2) I cannot agree that memory is rarely the bottleneck. It is indeed okay in this paper because of the setup. However, for larger models and longer sequences (which are exactly the current large transformer use case), why memory is not a problem?\n3) It seems that the proposed approach introduces more computational overhead during inference. Is there any comparison about Carbon Emission? My concern is, even if the approach can generate faster, it actually consumes much more energy than before. I may save some money but actually, it is not that good for the environment.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Speculative Decoding to speed up autoregressive decoding in seq2seq generation. The key contributions include Spec-Drafter -- a novel model specially optimized for drafting -- and Spec-Verification -- an advanced method for verification in the decoding paradigm. Authors conduct experiments on machine translation and abstractive summarization and show the proposed SpecDec is more efficient than baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a speculative decoding approach for LLMs by using a separate smaller encoder-decoder drafting model to quickly generate multiple tokens and then verify these tokens in parallel using a larger model. The paper is generally well written, has a clearly motivated idea, and is a part of an emerging research area.

The framework of speculative decoding has seen a resurgence in interest over the 18 months (as indicated by the authors in the comments below). One of the reviewers mentions several other speculative decoding works that have appeared in that time frame were not compared against. The authors provide additional extensive results in the rebuttal compared against these other works. The results suggest that their approach outperforms other existing approaches.

Two reviewers raised concerns around the memory/computational overhead of requiring two models (a drafting model and validation model) compared to the conventional approach. In the rebuttal, the reviewers provide additional results to show that Speculative Decoding requires more power but requires less energy due to the increased tokens per second of the draft-then-verify approach. Generally, I think this issue of how this approach scales to larger models is something that could be further explored. For instance, how would this comparison look under a setting where both the drafter and verification models could not both fit in GPU memory? My assumption is that IO for weight transfer would then become a limiting factor.
Decision: Accept-Findings