ParallelSpec: Parallel Drafter for Efficient Speculative Decoding

## Abstract

Abstract Speculative decoding has proven to be an efficient solution to large language model (LLM) inference, where the small drafter predicts future tokens at a low cost, and the target model is leveraged to verify them in parallel.
However, most existing works still draft tokens auto-regressively to maintain sequential dependency in language modeling, which we consider a huge computational burden in speculative decoding.
We present ParallelSpec , an alternative to auto-regressive drafting strategies in state-of-the-art speculative decoding approaches.
In contrast to auto-regressive drafting in the speculative stage, we train a parallel drafter to serve as an efficient speculative model. ParallelSpec learns to efficiently predict multiple future tokens in parallel using a single model, and it can be integrated into any speculative decoding framework that requires aligning the output distributions of the drafter and the target model with minimal training cost. Experimental results show that ParallelSpec accelerates baseline methods in latency up to 62% on text generation benchmarks from different domains, and it achieves 2.84 × \times × overall speedup on the Llama-2-13B model using third-party evaluation criteria.

## 1 Introduction

Large language models (LLMs) such as GPT-4 (OpenAI, ) and Llama (Touvron et al., ) have shown dominant abilities across various domains, such as question answering (Zhuang et al., ), code synthesis (Rozière et al., ), machine translation (Zhang et al., ) and beyond.
However, their auto-regressive nature requires multiple forward passes on models with billions or trillions of parameters, bringing substantial inference latency, thus prohibiting real-time applications.
In the pursuit of accelerating LLM inference, various strategies have been explored, including utilizing model sparsity (Liu et al., ; Sun et al., ; Schuster et al., ; Cai et al., ), exploiting redundancy in KV Cache (Cai. et al., ; Zhang et al., ; Li et al., ), and distilling model capabilities to smaller models (Gu et al., ; Agarwal et al., ). While these approaches can lead to faster inference, they often come at the cost of reduced generation quality and do not preserve the generation distributions of the original models.
Speculative decoding (SD) (Leviathan et al., ; Chen et al., ) has been proposed as one of the compelling alternatives to auto-regressive generation in a lossless manner. The key motivation behind SD is to utilize a low-cost small model to generate draft tokens efficiently and then use the target model to verify them in parallel to ensure sampling integrity, known as draft-then-verify framework.
While promising, we observe that draft models in most draft-then-verify frameworks still generate token by token, resulting in a low arithmetic intensity during the drafting stage.
Moreover, the forward latency of the drafting stage still grows linearly with respect to the draft length, *i.e*., the number of tokens each draft step generates. As empirically profiled in the right part of Figure 1, the draft latency still accounts for a significant proportion of the overall SD latency.

Figure: Figure 1: Illustration of ParallelSpec. Left: comparison between auto-regressive drafting and our proposed parallel drafting. Blocks in green indicate normal draft tokens. Blocks in yellow denote the mask tokens used to prompt the draft model to generate multiple future tokens in a single forward pass. Right: wall time trace diagrams for two drafting styles integrated with EAGLE (Li et al., ) in two rounds of speculative sampling.
Refer to caption: x1.png

Researchers have spotted the draft latency issue and proposed several solutions to alleviate it by dynamically determining the draft length either through learnable policy (Huang et al., ), confidence-guided heuristics (Li et al., ), or optimized draft tree structures (Wang et al., ).
Nevertheless, these solutions do not tackle the underlying issue of drafting latency scaling linearly with the draft length, but operate only for maximumly reducing the compute wasted in drafting tokens that are unlikely to be accepted.

To fundamentally solve the draft latency issue, we propose building a parallel-decoding drafter as the replacement for auto-regressive drafters in popular SD frameworks.
Unlike Medusa-style frameworks (Cai et al., ; Ankner et al., ) that rely on separate language model heads to decode future tokens, we propose to use a single lightweight model to decode the next $k$ tokens simultaneously.
We argue that using a single model for multi-token prediction can effectively leverage parameter sharing to achieve efficient drafter alignment rather than learning several independent language model heads.
The latter design would struggle even more with memory and computation in the large vocabulary size (128,000+) of recently introduced language models such as Llama-3 (AI@Meta, ).
For efficient multi-token alignment training, we introduce a group-wise parallel training strategy that mitigates possible training-inference mismatches by dynamically adjusting the attention mask, positional indexes, and token layout.

Our method still adheres to the draft-and-verify framework at inference time: At each draft step, the drafter generates $k$ tokens with a single forward pass, and then they are sent to the target model for parallel verification.
Since our method incorporates token-level parallelization in the drafting stage, both stages in the SD pipeline now benefit from this parallelization. Therefore, we name our approach ParallelSpec.
ParallelSpec works as an individual module, ready to replace any drafter in existing SD frameworks that require distilling drafters from their target models.
We experiment with the popular speculative decoding framework Medusa (Cai et al., ) and state-of-the-art solution EAGLE (Li et al., ) by replacing their drafters.
Experimental results show that ParallelSpec is able to bring consistent acceleration improvement in all task domains and different combinations of models.
For instance, incorporating ParallelSpec into Vicuna-7B Medusa increases the average speedup ratio from 1.42$\times$ to 2.31$\times$, leading to a 62.7% relative improvement. This empirically validates the superiority of parallel drafter design.
ParallelSpec integration with EAGLE also achieves extra speedups across all target model settings, ranging from 2.55$\times$ to 2.84$\times$. In summary, our key contributions are as follows:

- •
We propose ParallelSpec as a parallel multi-token drafter to replace the auto-regressive drafter design in existing SD frameworks that require aligning drafters with their target models.
- •
We design a group-wise training strategy that allows efficient and accurate parallel drafter training.
- •
We integrate the proposed method into two popular SD frameworks. Extensive experiments demonstrate the compatibility and performance superiority of ParallelSpec.

## 2 Related Works

Accelerating Large Language Model (LLM) Inference has attracted considerable research attention from both machine learning system and the natural language processing community and even led the trend of hardware-software co-design.
These research efforts include model compression (Sun et al., ; Huang et al., ; Ma et al., ), novel architecture design (Gu & Dao, ; Peng et al., ) and hardware optimization (Dao et al., ; Hong et al., ).
However, some of these methods could lead to generation discrepancies compared to the target model, representing a trade-off between model performance and inference speed.
We consider those methods that cannot generate with the target model’s original distribution as lossy acceleration methods.

Speculative Decoding (SD) (Leviathan et al., ; Chen et al., ) arises as one of the lossless acceleration methods for LLM inference. It is based on the observation that the latency bottleneck of LLM inference is brought by memory bandwidth instead of arithmetic computation.
SD alleviates the bandwidth bottleneck by utilizing a small model to draft multiple tokens and verifying them in parallel with the target model, thereby reducing the frequency of language model calls and decreasing the memory access density during decoding.
The community has witnessed many improvements in efficiently and accurately drafting tokens.
Self-speculative decoding methods (Hooper et al., ; Elhoushi et al., ; Zhang et al., ; Bhendawade et al., ) do not explicitly rely on draft models but use some of the intermediate layers of the target model to draft.
Medusa-style methods add independent (Cai et al., ) or sequential (Ankner et al., ) decoding heads on the target model to draft tokens.
Lookahead decoding and its variants (Fu et al., ; Zhao et al., ) use n-gram trajectory as drafts.
DistillSpec (Zhou et al., ) leverages knowledge distillation to closely align distributions between the draft and target models.
PEARL (Liu et al., ) proposes two-stage verification to alleviate the mutual waiting problem.
Apart from efficient draft methods, token tree verification (Miao et al., ) has been widely adopted for verifying top candidate sequences that share common prefixes in parallel.
Specialized SD frameworks for long-context generation (Chen et al., ; ; Sun et al., ), retrieval-augmented generation (He et al., ; Wang et al., ; Zhang et al., ) and beyond (Chen et al., ) have been proposed to better fit individual use cases.

Parallel Decoding was first known for its efficiency in machine translation system (Ghazvininejad et al., ) and code generation (Gloeckle et al., ) as an alternative to auto-regressive generation. However, its usage in SD frameworks remains under-explored.
Santilli et al. () proposed to use fixed-point iterations to replace auto-regressive decoding.
Monea et al. (); Yi et al. () pioneered the use of parallel decoding in SD, but it is limited to a self-speculative framework and results in different generation sampling. BiTA (Lin et al., ) proposed using prompt tuning to train a small number of prompt parameters on a frozen target LM for semi-autoregressive generation.
Wu et al. () suggested using trainable linear projection to regress intermediate hidden states of target models, thereby enabling multi-token prediction.
However, these methods either fail to effectively learn the draft distribution due to the limited number of learnable parameters or cannot achieve lossless acceleration due to the method design.

## 3 Background: Speculative Decoding

Notation. Speculative decoding (SD) frameworks maintain two models: the target model, denoted as $\mathcal{M}_{\text{T}}$, is the one which we want the SD frameworks to sample from; the draft model, denoted as $\mathcal{M}_{\text{D}}$, is the one that proposes candidate tokens which are later being verified by the target model.
Let $x_{<t}$ be the prompt sequence we are running the SD framework on, $p\left(x_{t}\mid x_{<t}\right)$, $q\left(x_{t}\mid x_{<t}\right)$ be the inference distribution of $\mathcal{M}_{\text{T}}$ and $\mathcal{M}_{\text{D}}$ given the prompt $x_{<t}$, respectively. We use the denotations of $p\left(y_{t}\right)$ and $q\left(y_{t}\right)$ to indicate $p\left(y_{t}\mid x,y_{<t}\right)$ and $q\left(y_{t}\mid x,y_{<t}\right)$ whenever they do not lead to confusion.

Speculative Decoding Procedures. One round of speculative decoding can be divided into the drafting and verification stages, each governed by the corresponding model. The drafting stage auto-regressively calls $\mathcal{M}_{\text{D}}$ to sample $\gamma$ candidate token distributions $q\left(y_{t}\right),\ldots,q\left(y_{t+\gamma-1}\right)$. The verification stage calls $\mathcal{M}_{\text{T}}$ once to sample $\gamma$ distributions from the target model, $p\left(y_{t}\right),\ldots,p\left(y_{t+\gamma-1}\right)$, given $y_{t+i}$ is the concatenation of $y_{<t}$ and drafted token sequence $x_{1},\ldots,x_{i}$, where $x_{t+i}\sim q\left(y_{t+i}\right)$. The verification stage determines whether the token $y_{t+i}$ is accepted via speculative sampling, where its acceptance rate $\alpha_{i}$ is defined as:

$$ $\alpha_{i}=\begin{cases}1&p\left(y_{t+i}\right)\geq q\left(y_{t+i}\right)\\ \frac{p\left(y_{t+i}\right)}{q\left(y_{t+i}\right)}&p\left(y_{t+i}\right)<q \left(y_{t+i}\right)\end{cases}$ (1) $$

If the token $y_{t+i}$ is rejected before $\gamma$ candidate tokens are all accepted, the remaining draft tokens will be discarded, and $y_{t+i}$ will be resampled from $\max\left(0,p\left(y_{t+i}\right)-q\left(y_{t+i}\right)\right)$. Otherwise, drafted tokens are all accepted and SD samples an extra token from $y_{t+\gamma}$ and appends it to the end of the sequence.
Each round of speculative decoding generates at least $1$ and at most $\gamma+1$ tokens, and Leviathan et al. () theoretically proves that the sequence from SD and the sequence from the target model follow the same distribution.

Average Acceptance Length.
One important efficiency metric to measure an SD system is the acceptance rate of drafted tokens in each round of speculative decoding.
Since each drafting step takes a constant amount of time and each round of conventional SD takes the same drafting steps, the overall efficiency of an SD system will be determined by the average acceptance length $\tau$ measured on some prompt sequences.

Token Tree Verification.
Prior studies (Miao et al., ; Leviathan et al., ) suggest that verifying multiple candidate sequences within the same verification step could greatly improve the expected acceptance length.
This is achieved by the tree attention mechanism. By properly arranging the top predicted tokens at different token positions and manipulating the attention mask based on a tree structure, we enable the processing of multiple candidate sequences with only one verification step. We refer to the details of token tree verification in Appendix A.3.

## 4 Methodology

In this section, we describe our parallel drafting method (§4.1), the algorithm that preserves the output distribution of the target model with parallel drafter (§4.2) and its integration into popular speculative sampling methods (§4.3).

### 4.1 Parallel Drafting

Figure: Figure 2: Illustration of parallel drafter inference, training, and the difference between training auto-regressive drafter and parallel one. Left: Parallel drafter proposes multiple candidate tokens with a single forward pass. Middle: Training the parallel drafter to align with the target model is a process of knowledge distillation (KD). Right: The input, labels, and position indices for training a parallel drafter need special treatment. ${\dagger}$ refers to Figure 3 for the special attention mask design of training.
Refer to caption: x2.png

Inference. Let $\mathcal{M}_{\text{D}}^{\theta}$ be the parallel drafter parameterized by $\theta$, and $q^{\theta}\left(x_{t},x_{t+1},\ldots,x_{t+k}\mid x_{<t}\right)$ be the multi-token output distribution of the parallel drafter.
Naive draft models do not support predicting multiple tokens in a single forward pass. In order to equip a drafter with such abilities, we propose to use customized [MASK] tokens as prompt tokens to produce contextualized token-level representations that are used to enforce multi-token training.
The parallel drafter introduces $k$ special tokens in its vocabulary, $\texttt{[MASK]}_{1},\ldots,\texttt{[MASK]}_{k}$.
At each drafting step where the drafter is invoked, $k$ special tokens are concatenated after the original input sequence $x_{<t}$.
Apart from the last token representation that is used to decode the next token $x_{t}$, the last-layer representations corresponding to [MASK] tokens are leveraged to sample future tokens $x_{t+1},\ldots,x_{t+k}$.
The left part of Figure 2 demonstrates how the parallel draft model proposes 3 future tokens with 2 [MASK] prompt tokens.

Figure: Figure 3: Attention mask illustration of parallel drafter training. denotes activated attention. denotes attention suppressed to prevent access across parallel groups. -100 denotes ignored tokens in the target sequence that do not contribute to training loss. Blocks with yellow and the legend illustrate one of the next-next token prediction training objectives.
Refer to caption: x3.png

Group-wise Parallel Training.
Given the access to either the ground truth future tokens $y_{t},y_{t+1},\ldots,y_{t+k}$ or their output distributions of the target model $p(y_{t}),p(y_{t+1}),\ldots,p(y_{t+k})$, training a parallel drafter is a process of knowledge distillation (KD) in either online or offline setting (Zhou et al., ), which we defer to §4.3 for detailed discussion.
This process is not as trivial as training an auto-regressive drafter, where teacher-forcing supervision is enforced via shift-one-token in labels as depicted in the upper right of Figure 2.
Simply shifting the label position by $k$ tokens would result in discrepancies between training and inference.
Therefore, We carefully design a group-wise training paradigm that eliminates training-inference mismatch by manipulating the token layout, attention masks, and position indices, illustrated in the lower right of Figure 2.
Specifically, we introduce the concept of parallel group, illustrated in Figure 3, where each parallel group in the source sequence consists of an input token and several [MASK] tokens.
At each training step, a customized causal attention mask guarantees that all training token pairs ignore [MASK] tokens from previous parallel groups to ensure the model behaves the same in training and evaluation.

### 4.2 Speculative Sampling with Parallel Drafting

In order to preserve the output distribution of the target model as standard speculative sampling does, following Monea et al. (), we present a modified version of speculative sampling (Chen et al., ) that drafts with a parallel decoder and verifies with the target model
using token tree verification method (Miao et al., ), detailed in Algorithm 1.

Figure: Algorithm 1 Speculative Sampling with Parallel Draft Models and Token Tree Verification

### 4.3 Integration with Popular Speculative Decoding Frameworks

ParallelSpec can be inserted into any speculative decoding (SD) framework that requires aligning output distributions of the drafter and the target model.
We choose two popular SD frameworks as testbeds, Medusa (Cai et al., ) and EAGLE (Li et al., ).

Medusa leverages an offline knowledge distillation method, where cross-entropy loss between Medusa heads and ground truth tokens is used for alignment. Specifically, $K$ extra decoding heads are added to decode the last hidden states of the target model, and the $k$-th head is used to predict the $(t+k+1)$-th token given a prefix sequence of length $t$. The final training objective of Medusa is expressed as:

$$ $\mathcal{L}_{\text{MEDUSA}}=\sum_{k=1}^{K}-\lambda_{k}\log p_{t}^{(k)}\left(y_ {t+k+1}\right),$ (2) $$

where $\lambda_{k}$ is a coefficient to balance learning difficulties, $p_{t}^{(k)}$ is the output distribution of the $k$-th head and $y_{t+k+1}$ is the oracle token.

To integrate ParallelSpec into Medusa, we introduce a Transformer model specialized in multi-token prediction to replace $K$ Medusa heads as the new draft model.
The draft model shares the embedding layer and the language model head with the target model to minimize memory overhead, and they remain frozen to preserve the target model’s output distribution.
$K$ trainable [MASK] tokens are added to the embedding layer of the draft model to facilitate parallel training.
During the training, each parallel group is trained with a similar objective denoted in Equation 2, except that the log term now denotes the output distribution of the parallel drafter at position $k$, and we need to consider the non-mask token at the beginning of each parallel group:

$$ $\mathcal{L}_{\text{MEDUSA-Parallel}}=-\log q\left(y_{t+1}|x_{<t}\right)-\sum_{ k=1}^{K}\lambda_{k}\log q\left(y_{t+k+1}|x_{<t},\texttt{[MASK]}_{1},\dots, \texttt{[MASK]}_{k}\right).$ (3) $$

EAGLE utilizes an online knowledge distillation method that directly regresses the last-layer hidden states at the feature level.
Specifically, we denote the last-layer hidden states of the target model at $t$-th position as $f_{t}$, the embedding of $t$-th token as $e_{t}$, and the oracle token as $y_{t}$. EAGLE proposed to use a fully connected layer and an auto-regression head $\pi\left(\tilde{f_{t}}|f_{<t},e_{<t}\right)$ as the draft model to predict the next feature that is used to decode draft tokens.
The training objective of EAGLE on each token position is a linear combination of the regression loss $\mathcal{L}_{\text{reg}}$ and the cross-entropy loss $\mathcal{L}_{\text{cls}}$ between draft tokens and oracle tokens:

$$ $\displaystyle\mathcal{L}_{\text{reg}}$ $\displaystyle=\operatorname{SmoothL1}\left(f_{t+1},\pi\left(\tilde{f_{t}}|f_{< t},e_{<t}\right)\right),$ (4) $\displaystyle\mathcal{L}_{\text{cls}}$ $\displaystyle=-\log\mu\left(y_{t+1}\right),$ $$

where $\mu$ denotes the language model head distribution conditioned on drafted feature $\tilde{f_{t}}$.

Integrating ParallelSpec into EAGLE is more intuitive, as it only requires minor modifications to turn the auto-regression head into a parallel head with the method outlined in §4.1, without extra adaptation.
For a parallel group of size $K$ starting at token position $t$, the training loss of EAGLE-Parallel is the sum of EAGLE losses (defined in Equation 4) over $K$ tokens within the same group.

## 5 Experiments

This section describes the experimental settings (§5.1), including training and evaluation datasets, metrics, and involved baselines.
§5.2 reports the main results for ParallelSpec compared with baselines. Finally, we discuss the impact of different experiment settings in §5.3.

**Table 1: Speedup ratios and average acceptance lengths $\tau$ of different methods tested on an A100-PCIE-40GB GPU using third-party benchmark toolkit SpecBench (Xia et al., ). V: Vicuna-v1.3. L2: LLaMA2-Chat. We report the mean and standard deviation of speedup ratios on 3 different runs. Best metrics for each model are marked in boldface.**
| Model | Method | Multi-turn<br>Conversation | Translation | Summarization | Question<br>Answering | Mathematical<br>Reasoning | Retrieval-aug.<br>Generation | Avg. | $\tau$ (tokens) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Temperature = 0.0 |  |  |  |  |  |  |  |  |  |
| V 7B | SpS | 1.67$\times$±0.04 | 1.13$\times$±0.02 | 1.71$\times$±0.01 | 1.49$\times$±0.04 | 1.50$\times$±0.03 | 1.67$\times$±0.02 | 1.53$\times$±0.03 | 2.27 |
| PLD | 1.61$\times$±0.02 | 1.02$\times$±0.01 | 2.57$\times$±0.02 | 1.14$\times$±0.02 | 1.61$\times$±0.01 | 1.82$\times$±0.06 | 1.62$\times$±0.01 | 1.75 |  |
| Hydra | 2.50$\times$±0.02 | 1.94$\times$±0.03 | 1.89$\times$±0.04 | 2.02$\times$±0.04 | 2.53$\times$±0.02 | 1.86$\times$±0.07 | 2.13$\times$±0.04 | 3.26 |  |
| Lookahead | 1.48$\times$±0.02 | 1.15$\times$±0.02 | 1.36$\times$±0.02 | 1.27$\times$±0.02 | 1.59$\times$±0.03 | 1.23$\times$±0.03 | 1.35$\times$±0.02 | 1.64 |  |
| Medusa | 1.60$\times$±0.01 | 1.39$\times$±0.01 | 1.22$\times$±0.03 | 1.37$\times$±0.00 | 1.68$\times$±0.01 | 1.20$\times$±0.06 | 1.42$\times$±0.01 | 2.39 |  |
| +ParallelSpec | 2.63$\times$±0.02 | 1.97$\times$±0.03 | 2.32$\times$±0.04 | 2.20$\times$±0.02 | 2.78$\times$±0.03 | 1.98$\times$±0.03 | 2.31$\times$±0.02 | 3.31 |  |
| EAGLE | 2.57$\times$±0.02 | 1.85$\times$±0.04 | 2.17$\times$±0.05 | 2.03$\times$±0.04 | 2.57$\times$±0.05 | 1.92$\times$±0.04 | 2.18$\times$±0.04 | 3.58 |  |
| +ParallelSpec | 3.01$\times$±0.04 | 2.09$\times$±0.01 | 2.62$\times$±0.06 | 2.40$\times$±0.03 | 2.84$\times$±0.05 | 2.36$\times$±0.02 | 2.55$\times$±0.03 | 3.52 |  |
| L2 7B | SpS | 1.33$\times$±0.03 | 1.25$\times$±0.03 | 1.21$\times$±0.02 | 1.30$\times$±0.01 | 1.34$\times$±0.02 | 1.43$\times$±0.03 | 1.31$\times$±0.02 | 1.67 |
| PLD | 1.42$\times$±0.01 | 1.17$\times$±0.02 | 1.44$\times$±0.02 | 1.07$\times$±0.02 | 1.31$\times$±0.02 | 1.57$\times$±0.01 | 1.33$\times$±0.01 | 1.42 |  |
| Lookahead | 1.46$\times$±0.05 | 1.36$\times$±0.04 | 1.34$\times$±0.04 | 1.32$\times$±0.03 | 1.47$\times$±0.04 | 1.37$\times$±0.03 | 1.39$\times$±0.04 | 1.60 |  |
| EAGLE | 2.61$\times$±0.02 | 2.38$\times$±0.02 | 2.25$\times$±0.02 | 2.30$\times$±0.05 | 2.66$\times$±0.06 | 2.23$\times$±0.01 | 2.40$\times$±0.02 | 3.55 |  |
| +ParallelSpec | 2.95$\times$±0.03 | 2.67$\times$±0.01 | 2.64$\times$±0.03 | 2.76$\times$±0.04 | 2.88$\times$±0.02 | 2.52$\times$±0.03 | 2.74$\times$±0.03 | 3.49 |  |
| V 13B | SpS | 1.69$\times$±0.01 | 1.16$\times$±0.01 | 1.78$\times$±0.00 | 1.45$\times$±0.01 | 1.60$\times$±0.01 | 1.76$\times$±0.04 | 1.57$\times$±0.01 | 2.18 |
| Medusa | 2.06$\times$±0.01 | 1.77$\times$±0.02 | 1.66$\times$±0.04 | 1.74$\times$±0.01 | 2.12$\times$±0.01 | 1.62$\times$±0.05 | 1.84$\times$±0.02 | 2.39 |  |
| +ParallelSpec | 2.76$\times$±0.04 | 2.37$\times$±0.05 | 2.16$\times$±0.02 | 2.33$\times$±0.03 | 2.62$\times$±0.03 | 2.27$\times$±0.04 | 2.52$\times$±0.05 | 3.34 |  |
| EAGLE | 2.78$\times$±0.02 | 2.03$\times$±0.03 | 2.41$\times$±0.02 | 2.11$\times$±0.03 | 2.78$\times$±0.04 | 2.20$\times$±0.03 | 2.39$\times$±0.02 | 3.64 |  |
| +ParallelSpec | 3.03$\times$±0.04 | 2.30$\times$±0.03 | 2.65$\times$±0.02 | 2.36$\times$±0.03 | 3.04$\times$±0.05 | 2.46$\times$±0.04 | 2.64$\times$±0.02 | 3.56 |  |
| L2 13B | SpS | 1.38$\times$±0.03 | 1.30$\times$±0.03 | 1.26$\times$±0.04 | 1.36$\times$±0.03 | 1.41$\times$±0.02 | 1.47$\times$±0.06 | 1.36$\times$±0.03 | 1.66 |
| EAGLE | 2.80$\times$±0.01 | 2.60$\times$±0.02 | 2.53$\times$±0.06 | 2.42$\times$±0.03 | 2.85$\times$±0.03 | 2.39$\times$±0.12 | 2.60$\times$±0.03 | 3.66 |  |
| +ParallelSpec | 3.02$\times$±0.02 | 2.81$\times$±0.03 | 2.77$\times$±0.07 | 2.68$\times$±0.03 | 3.00$\times$±0.02 | 2.74$\times$±0.04 | 2.84$\times$±0.04 | 3.60 |  |
| Temperature = 1.0 |  |  |  |  |  |  |  |  |  |
| V 7B | SpS | 1.35$\times$±0.00 | 1.01$\times$±0.00 | 1.39$\times$±0.02 | 1.25$\times$±0.01 | 1.29$\times$±0.02 | 1.38$\times$±0.05 | 1.28$\times$±0.01 | 1.82 |
| EAGLE | 2.10$\times$±0.01 | 1.59$\times$±0.02 | 1.83$\times$±0.05 | 1.70$\times$±0.02 | 2.04$\times$±0.02 | 1.78$\times$±0.06 | 1.84$\times$±0.01 | 3.18 |  |
| +ParallelSpec | 2.32$\times$±0.02 | 1.78$\times$±0.03 | 2.06$\times$±0.04 | 1.89$\times$±0.02 | 2.10$\times$±0.01 | 1.96$\times$±0.02 | 2.02$\times$±0.03 | 3.09 |  |
| L2 7B | SpS | 1.11$\times$±0.00 | 1.07$\times$±0.01 | 1.04$\times$±0.02 | 1.09$\times$±0.01 | 1.13$\times$±0.01 | 1.15$\times$±0.01 | 1.10$\times$±0.01 | 1.47 |
| EAGLE | 2.19$\times$±0.02 | 1.92$\times$±0.05 | 1.91$\times$±0.03 | 1.93$\times$±0.05 | 2.31$\times$±0.05 | 1.87$\times$±0.07 | 2.02$\times$±0.04 | 3.30 |  |
| +ParallelSpec | 2.47$\times$±0.03 | 2.15$\times$±0.02 | 2.08$\times$±0.04 | 2.11$\times$±0.03 | 2.42$\times$±0.01 | 2.06$\times$±0.03 | 2.22$\times$±0.02 | 3.25 |  |

### 5.1 Settings

Datasets, Tasks, and Training.
Following the setup of SpecBench (Xia et al., ), we conduct evaluations on six types of text generation tasks, including MT-bench (Zheng et al., ) for multi-turn conversation, CNN/Daily Mail (Nallapati et al., ) for text summarization, Natural Questions (Karpukhin et al., ) for retrieval-augmented generation and question answering, WMT14 DE-EN (Bojar et al., ) for machine translation, GSM8K (Cobbe et al., ) for mathematical reasoning.
As ParallelSpec falls into the category of speculative decoding methods that require an extra alignment stage, supervised fine-tuning (i.e., SFT) data are needed for aligning distributional similarity between the drafter and the target model. To ensure a fair comparison with baselines in this category, we follow Li et al. () to use 68,000 ShareGPT (Tay et al., ) conversations as training data without self-distillation.
For the self-distillation setting where multi-turn conversations are distilled from the target model given the prompts from the dataset, we refer to §5.3.
Training ParallelSpec on 7B models takes 13 hours on 8 A100-PCIE-40GB GPUs for 40 epochs.
The size of parallel group is set to 5, *i.e*., the number of [MASK] tokens $k=4$ unless stated otherwise.
We refer to Appendix A.2 for details such as computing environment, other hyper-parameters, *etc*.

Baselines and Metrics.
We select seven competitive speculative decoding methods as baselines. Some of them work as plugin modules like Speculative Sampling (SpS) (Chen et al., ), Prompt Lookup Decoding (PLD) (Saxena, ; Yang et al., ) and Lookahead Decoding (Fu et al., ), which do not need additional training.
For SpS methods, we use vicuna-68m(^1^11[https://huggingface.co/double7/vicuna-68m](https://huggingface.co/double7/vicuna-68m)) and llama-68m(^2^22[https://huggingface.co/JackFram/llama-68m](https://huggingface.co/JackFram/llama-68m)) as the drafter for Vicuna and Llama target models, respectively.
For PLD, we follow the default settings of n-gram size $=3$ and number of lookup tokens $=10$.
For Lookahead Decoding, we use the official recommended configuration of level $=5$, window size $=7$, and n-gram size $=7$.
The remaining methods, including Medusa (Cai et al., ), Hydra (Ankner et al., ), and EAGLE (Li et al., ), that require extra training while preserving the output distributions, are the main peer works for comparison. We use their official drafter checkpoints to report results.
We primarily focus on the end-to-end wall-time speedup ratio compared to naive auto-regressive decoding.
We also report the average acceptance length $\tau$ in each round of speculative decoding.

Models.
We conduct experiments on the Vicuna series (7B, 13B) (Zheng et al., ) and the Llama-2-Chat series (7B, 13B) (Touvron et al., ). We chose these models as they are highly representative, and most prior methods built their drafters upon these models, allowing a fair comparison. We provide results for more recent target models in §5.3.
All parallel drafters are constructed with a single Transformer layer, with hyper-parameters identical to those of layers in their target models.
This results in a 202M drafter for 7B models and a 317M one for 13B models.
We keep the same draft token tree structure with the selected two baseline methods in ParallelSpec.

### 5.2 Main Results

Table 1 gives a performance overview of ParallelSpec and established prior approaches on different types of text generation benchmarks.
We refer to Appendix A.1 for qualitative case studies.
In general, ParallelSpec integration brings a consistent acceleration improvement in all domains, two selected methods, and different decoding temperatures.
Based on the results, we have the following key observations:

ParallelSpec significantly accelerates Medusa frameworks by a large margin.
For example, integrating ParallelSpec into Vicuna-7B Medusa boosts the average speedup ratio from 1.42$\times$ to 2.31$\times$, resulting in a 62.7% relative improvement.
This even outperforms EAGLE, which can be attributed to the parallel drafter design. It not only increases the average acceptance tokens from 2.39 to 3.31 but also significantly reduces drafter runtime overhead.
Similar improvements are also observed on Vicuna-13B Medusa, showing ParallelSpec is not sensitive to the target model size.

Equipping EAGLE with ParallelSpec also achieves considerable speedups.
For instance, the average speedup ratio on Vicuna-7B increased from 2.18$\times$ to 2.55$\times$; on Llama2-7B-Chat, it rose from 2.40$\times$ to 2.74$\times$; and on LLaMA-13B-Chat, it went from 2.60$\times$ to 2.84$\times$.
It is worth noting that incorporating ParallelSpec into EAGLE causes a slight drop in average acceptance length.
This is fully expected, as we no longer preserve the sequence dependency during the draft stage, and the parallel decoding leads to a small decline in drafting accuracy.
In addition, we also conduct experiments on EAGLE-ParallelSpec with temperature decoding settings and observe substantial overall speedup up to 2.22$\times$.
Still, the speculative decoding efficiency at high temperature degrades compared with greedy decoding, echoing conclusions in previous studies (Xia et al., ; Ouyang et al., ).

### 5.3 Ablation Studies

Figure: Figure 4: Ablations on speedup ratio and average acceptance length $\tau$ with respect to the number of [MASK] tokens $K$ on all three test datasets.
Refer to caption: x4.png

**Table 2: Ablations on training data and recent advanced target models.**
| Model & Method | Multi-turn<br>Conversation | Translation | Summarization | Question<br>Answering | Mathematical<br>Reasoning | Retrieval-aug.<br>Generation | Avg. | $\tau$ (tokens) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA 2 7B w/ EAGLE | 2.61$\times$ | 2.38$\times$ | 2.25$\times$ | 2.30$\times$ | 2.66$\times$ | 2.23$\times$ | 2.40$\times$ | 3.55 |
| +ParallelSpec | 2.95$\times$+13.0% | 2.67$\times$+12.2% | 2.64$\times$+17.3% | 2.76$\times$+20.0% | 2.88$\times$+8.3% | 2.52$\times$+13.0% | 2.74$\times$+14.2% | 3.49 |
| +ParallelSpec +self-distillation | 3.02$\times$+15.7% | 2.77$\times$+17.3% | 2.71$\times$+20.4% | 2.87$\times$+24.8% | 2.98$\times$+12.0% | 2.56$\times$+14.8% | 2.82$\times$+17.5% | 3.78 |
| LLaMA 3 8B w/ EAGLE | 2.66$\times$ | 2.42$\times$ | 2.33$\times$ | 2.32$\times$ | 2.75$\times$ | 2.32$\times$ | 2.47$\times$ | 3.42 |
| +ParallelSpec | 3.03$\times$+13.9% | 2.63$\times$+8.7% | 2.61$\times$+12.0% | 2.83$\times$+22.0% | 2.92$\times$+6.2% | 2.45$\times$+5.6% | 2.75$\times$+11.3% | 3.36 |

Training Data.
Prior works often assume the availability of training data that aligns with the output distribution of target models, which is not usually the case. Thus, directly using ShareGPT conversations as SFT data for drafter training may suffer from the domain shift.
In order to bypass this, we follow Cai et al. () to employ the self-distillation technique to build the training dataset that matches the target model.
Specifically, we employ vLLM (Kwon et al., ) to obtain distilled multi-turn conversations by feeding the prompts from ShareGPT in a greedy decoding setting.
The upper part of Table 2 demonstrates that the self-distillation technique can further improve the speedup ratio, with the average acceptance tokens greatly increased to 3.78.

Size of Parallel Group.
The size of the parallel group is an important hyper-parameter of our method, and it determines how many tokens the parallel drafter will predict at each step. It equals the number of [MASK] tokens $K$ plus one.
To investigate its impact on the speedup performance, we conduct ablation studies to re-train parallel drafter with different $K$ and report the speedup ratio and average acceptance length in Figure 4.
We notice the increase in the average acceptance length $\tau$ resulting from larger parallel group size is steady for small $K$ but saturates after $K=4$, leading the speedup ratio to no longer improve beyond that point.
We believe this could be attributed to the difficulty of predicting distant future tokens using parallel decoding.
The benefit of increasing the parallel group size cannot counterbalance the overhead of predicting distant tokens, resulting in a speedup ratio sweet spot around $K=4$.

Advanced Target Model.
The lower part of Table 2 reflects that more advanced models like LLaMA3-8B-Instruct (AI@Meta, ) can still benefit from the design of ParallelSpec. However, the relative improvements on LLaMA3-Instruct series are slightly lower than those on LLaMA2-Chat and Vicuna, possibly because of the larger misalignment between ShareGPT SFT data and LLaMA3-Instruct.

## 6 Conclusion

In this paper, we introduce ParallelSpec, a powerful speculative decoding solution that could be inserted into popular speculative decoding frameworks.
It proposes to use a single lightweight model and several trainable [MASK] tokens to facilitate fast multi-token prediction as drafters, thereby mitigating the issue of drafting latency scaling linearly with the draft length.
Compared with the Medusa-style multi-head multi-token draft strategy, ParallelSpec demonstrates significant advantages in drafting accuracy, latency, and parameter efficiency.
Extensive experiments on various benchmarks and different target models demonstrate the compatibility and performance superiority of ParallelSpec.

## Appendix A Appendix

### A.1 Case Study

Figure: Figure 5: Upper: Visualization of accelerated tokens in generation from (a) Vicuna-7B Medusa and (b) Vicuna-7B Medusa-ParallelSpec given an input prompt from GSM8K (Cobbe et al., ). Lower: Simulated wall-time trace of two different methods generating the text in the highlighted box. We only consider the forward pass latency of draft and verification while ignoring the negligible post-processing overhead. : accepted draft tokens. : rejected draft tokens. : tokens without speculative acceleration.
Refer to caption: x5.png

We provide an illustration of two different speculative decoding methods running on the same prompt from GSM8K in Figure 5.
The side-by-side comparison indicates that Vicuna-7B Medusa equipped with ParallelSpec not only achieves a higher average acceptance length (3.71 vs. 2.66) but also shows a significant advantage in end-to-end latency, thanks to the one-pass decoding nature of parallel-decoding drafter.
The lower part of Figure 5 even reveals that ParallelSpec nearly cut half of the time cost when decoding the same text span compared with the baseline method.

### A.2 Experimental Details

**Table 3: Hyper parameters of ParallelSpec variants for 7B models.**
| Hyper Parameter | Medusa-ParallelSpec | EAGLE-ParallelSpec |
| --- | --- | --- |
| Global Batch Size | 32 | 32 |
| Learning Rate | $5\times 10^{-4}$ | $5\times 10^{-5}$ |
| Optimizer | AdamW ($\beta_{1}=0.9,\beta_{2}=0.999$) | AdamW($\beta_{1}=0.9,\beta_{2}=0.95$) |
| Weight Decay | 0.0 | 0.0 |
| Epochs | 20 | 40 |

All training was conducted using a node with 8 NVIDIA A100-PCIE-40GB GPUs, 2 AMD EPYC 7282 CPUs, and 512GB RAM.
Evaluations were conducted using one GPU of the above node.
PyTorch 2.2.0 with CUDA 12.1 version was used in all experiments.
To avoid the discrepancies brought by computing environment differences, we re-ran all baseline methods and our method 3 times and reported the mean speedup ratio.
We list hyperparameters used in ParallelSpec in Table 3.

### A.3 Token Tree Verification

Initially proposed in SpecInfer (Miao et al., ), and also explored in several follow-up works (Cai et al., ; Li et al., ; Spector & Re, ), tree-based structure for token verification is proved to be useful. Following existing studies, ParallelSpec leverages tree attention to realize this process. Specifically, to guarantee that each token only accesses its predecessors, we use an attention mask that exclusively permits attention flow from the current token back to its antecedent tokens. The positional indices for positional encoding are adjusted in line with this structure.
A conceptual view of this process is visualized in Figure 6, with the draft tree structure in the figure being adopted in all experiments. Starting from “Root”, every node is expanded with $k$ tokens with top-$k$ highest probabilities. $k$ is designed based on manually crafted rules, and is dynamically changing as the tree depth grows.

Figure: Figure 6: An example of the tree-structured verification process. Each circle represents a token, and the shading of the color indicates the probability of each token in the distribution. Tokens with the highest probability are selected and gradually expanded with dynamically designed $k$.
Refer to caption: x6.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Could the authors elaborate on the sensitivity of PARALLELSPEC’s performance to the number of tokens generated in parallel (i.e., the parameter k)? Are there guidelines for selecting k based on specific model or task characteristics?
2. How does PARALLELSPEC’s memory usage compare with traditional auto-regressive drafters, particularly in large models? Does the use of multiple [MASK] tokens introduce significant memory overhead?
3. In high-temperature settings, where acceptance rates may decrease, what are possible strategies to maintain drafter alignment without significantly impacting speedup?
soundness: 3
strengths: + The paper’s parallel drafter introduces a novel alternative to sequential SD models, advancing speculative decoding with efficient multi-token generation.
+ Empirical evaluations across multiple benchmarks and models, along with sound theoretical justifications, confirm the validity of the approach.
+ The work has strong implications for real-time, large-scale LLM applications, especially with the demonstrated integration into established frameworks like Medusa and EAGLE.
+ By employing rejection sampling, PARALLELSPEC maintains a lossless decoding process, preserving the original output distribution of the target model.
summary: The paper introduces a parallel drafting model as an alternative to traditional auto-regressive speculative decoding (SD) drafters for large language models (LLMs). Traditional SD models often draft tokens sequentially, resulting in latency that scales linearly with sequence length. PARALLELSPEC innovates by using a parallel drafter that predicts multiple tokens in a single forward pass, achieving notable reductions in draft latency (up to 62%) and overall speedups of up to 2.84x on the LLaMA-2-13B model. The model’s integration into frameworks like Medusa and EAGLE showcases its compatibility and effectiveness, while a group-wise training strategy ensures alignment with the target model’s output distribution. This method maintains theoretical losslessness through a rejection sampling algorithm, preserving the original LLM distribution.
weaknesses: - Unlike speculative decoding approaches that can use pre-trained models or lightweight modifications, PARALLELSPEC requires a dedicated training process for the parallel drafter. This additional training step may increase setup time and computational cost, which could limit the method’s immediate applicability in certain use cases.
- The absence of publicly available code for the training and experimental setups raises concerns about reproducibility. Without the code, it may be challenging for others in the community to replicate the experiments or validate the results independently, limiting the impact and accessibility of this work.
- The group-wise parallel training could use clearer explanations or diagrams, particularly the role of attention masks and mask tokens in aligning the drafter with the target model’s distribution.
- Handling of Token Uncertainty: While effective in low-temperature settings, PARALLELSPEC may experience reduced alignment at higher temperatures. Further discussion on managing token misalignment under these conditions would enhance robustness, particularly compared to EAGLE's feature-based uncertainty handling.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Can authors complete the evaluations in Table 1?

2. Is the code open-sourced?

3. The EAGLE uses autoregressive decoding as drafers. Is it possible to combine Medusa and EAGLE, meaning it also trains different decoding heads for different token positions?

4. It would be helpful if authors could provide detailed breakdown (e.g., draft time, verification time) of different methods. And compare how much does the propsoed method accelerate the draft time of EAGLE.
soundness: 2
strengths: 1. The intuition of why ParallelSpec can accelerate EAGLE is convincing. Based on the experiment results shown in Table 1, although ParallelSpec generates less tokens per iteration, it improves the drafting efficiency via parallel drafing, so it improves the overall efficiency.

2. The paper is well-written and easy to understand.

3. The paper includes comprehensive discussion on how to integrate the proposed method with state-of-the-art framework such as EAGLE and Medusa.
summary: This paper proposes ParallelSpec, which replaces the traditional autoregressive draft model in speculative decoding with parallel draft model to achieve better inference efficiency. Previous work such as Medusa already considers parallel draft model. The difference between ParallelSpec and Medusa is: Medusa trains additional heads to predict tokens at different positions, while ParallelSpec uses different mask tokens to do that. The authors claim their approach can maximally exploit the parameter sharing and hance is more effective. The experiment results in temperature=0 cases support this claim. But the experiments do not include the comparison with Medusa in temperature=1 cases. The main challenge in ParallelSpec is that it requires a different training mask and attention mechanism. The authors address this challenge reasonably.
weaknesses: 1. One of my main concern is the comparison between the proposed method and Medusa. Although authors provides an intuitive explanation that ParallelSpec has better parameter sharing, I am not fully convinced. So I expect to see a comprehensive comparison between Medusa and ParallelSpec in the experiments. However, in Table 1, the comparison between Medusa and Medusa+ParallelSpec only covers the temperature=0, but not temperature=1.

2. In Table 1, I believe the two settings (temp=0 and temp=1) are equally important. But temp=1 only covers a subset of baselines and target models. So the evaluation is not comprehensive to me.

3. Although the proposed method could accelerate EAGLE, the improvement is somewhat marginal (only about 1.1$\times$ speed up). This limits the contribution of this paper.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: **Concrete action items for the experimental results:**
1. I think it would be more accurate to temper the claim in the abstract, as the speedup for other baselines is around 9%-17% which is still notable.
2. Why are the reported speedup results for Medusa and EAGLE differ from those in the SpecBench paper, especially for Medusa where the difference is especially significant? I think it would strengthen the paper to add an explanation for this discrepancy.

**List of more minor things that could be improved in the paper:**
1. Figure 1 left - I think the arrows could be bolded or colored to clarify the distinction between “Auto-regressive Drafting” and “Parallel Drafting”. Currently the difference is a bit hard to spot.
2. End of intro section - since relative improvement (62.7%) is mentioned for Medusa, it should also be mentioned for Eagle (where I think it is 9-17%).
3. Related Work / Parallel Decoding - I believe "Blockwise Parallel Decoding for Deep Autoregressive Models" can also be mentioned here and possibly Medusa as well
4. Background / Speculative Decoding Procedures -  inconsistent notation for $x_k$. “Drafted token sequence $x_1,...,x_i$ where $x_{t+i} \sim q(y_{t+i})$” should be “Where $x_k \sim q(y_{t+k})$”.
5. I found algorithm 1 a bit hard to follow and suggest a few corrections below. However, since AFAIU this algorithm is simply standard tree speculative decoding as presented e.g. in SpecInfer, I believe it can be omitted or moved to the appendix if needed.
6. Experiments / Baselines - For SpS, it is not clear which gamma was chosen (how many draft tokens are generated on each iteration). Was gamma optimized to maximize speed-up or was it chosen based on some other criteria?

**Algorithm 1 issues:**
1. “Given … initial prefix sequence $x_{<n}$” should be $x_{<t}$, otherwise the line “Initialise n ← t” doesn’t make sense (since n can’t be initialized after it’s already used).
2. As written above, “Draft future token tree N” the details of how this tree is generated are not precisely written in the paper.
3. “O = TreeParallelDecode(N, p)”: O is set but never used. Instead, this should say something like - compute all logits p(x|x<n,{all paths from root to any node}) with the target model using N and tree attention.
4. “$\tilde{x}_s=H(s)$”: this notation is not clear, as H is a set of tree nodes and s is a node in the tree selected from $H$. I understand from the context $\tilde{x}_s$ is the token corresponding to node s in the tree but it should be made clearer.
5. I understand the notation $q(x|x_{<n},...,MASK_t)$ (since the … denotes the other mask tokens), but why $p(x|x_{<n},...,V)$? Why not just $p(x|x_{<n},V)$?
6. Normalize the residual line has an extra “+” at the end of the line.
7. The term $x_{<n}$ uses the n that keeps growing (initialized to t and then grows until it’s T or above). However, x never changes and it’s always the initial prefix of only t tokens. Should probably be changed to $x_{<t}$ everywhere.
soundness: 2
strengths: **Originality:** While the idea of parallel drafting in speculative decoding was already explored, this paper is the first time the effectiveness of this method is demonstrated with a separate drafter.

**Clarity**: Overall the method is simple, elegant, and clearly presented.

**Significance**: Speculative decoding is a widely used technique for accelerating language model inference, and speeding up state of the art speculative decoding methods is an important problem with clear implications for both research and industry. The demonstrated speed-up of ~15% on top of EAGLE is notable, highlighting the practical value of this work.
summary: This paper proposes a new drafter architecture for speculative decoding methods, which drafts several tokens in parallel rather than auto-regressively. It suggests a process for training these drafter models, shows how to incorporate it into several state of the art speculative decoding methods such as Medusa and EAGLE, and shows that it manages to achieve around 37-62% additional speedup for Medusa, and 9-17% additional speedups for EAGLE.
weaknesses: While accelerating speculative decoding methods is an important problem, and the ~15% speedup over existing methods is commendable, I believe the paper should be rejected for the following reasons: (1) the method is not novel enough as it is very similar to ideas already present in existing works (2) the results are not overly convincing and their magnitude is generally far below the 62% reported in the abstract. (3) There are some reproducibility issues, though I think these can be easily clarified by the authors in the follow up discussions.

**Novelty**: The core idea of drafting tokens in parallel instead of autoregressively was already presented by Stern et al (“Blockwise Parallel Decoding for Deep Autoregressive Models”), Monea et al (“PaSS: Parallel Speculative Sampling”) and to some extent Medusa, and the architecture presented in the paper closely resembles Monea et al, though their work was limited to the case where the same target model also generates the look-ahead drafts. This paper, while incrementally different, applies the concept with a separate drafter.

**Reproducibility**: The paper lacks sufficient details regarding the construction of the token tree. The main text refers to Appendix A.3, which only states that each node is expanded with “top-k highest probabilities”, where k is based on “manually crafted rules”. Without more specific details for how k is chosen, this process cannot be fully reproduced. Also, figure 6 may be unintentionally misleading, as from the text description the number of children of each node on a given depth should be the same (the “k” chosen for that depth). To enhance reproducibility, I suggest including a more detailed explanation in Appendix A.3 of how k is chosen for each depth level, and perhaps including pseudo-code.

**Clarity**: There are some mistakes and inaccuracies, mainly in “algorithm 1” which I believe should be revised. I’ll outline these in the Questions/Suggestions section. These issues can likely be addressed before publications, so they did not impact my score.

**Experimental results**: 
* A speed-up of ~15% over a state of the art method like EAGLE is notable, and could offer important inference improvements. However, there is some discrepancy between the results presented here and the ones presented in the SpecBench paper.
* Most importantly, for Medusa (which this paper claims to gain significant speed-ups of 30-60% over), the results presented in table 1 show significantly worse speed-ups than those in table 6 from the SpecBench paper (e.g. on 7B, SpecBench reports 2.42x while ParallelSpec paper reports 1.42x, on 13B SpecBench reports 2.16x while ParallelSpec paper reports 1.84x). I think it’s important to clarify these differences, as the 62% additional speedup reported in the abstract is the one achieved on top of these sub-par Medusa results. 
* The EAGLE reported speedups are also a bit lower than in the SpecBench paper, but by a smaller magnitude (e.g. for 7B, 2.39x in SpecBench paper vs 2.18x in ParallelSpec).Still, this ~10% difference is significant since ParallelSpec only reports an improvement of 15% over EAGLE.
* The 62% speedup claim in the abstract is based on results from the Medusa 7B baseline, which demonstrates particularly subpar performance compared to SpecBench (and according to table 1, seems to be slower than even vanilla SpS or lookahead methods).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. **Technical**-wise, how is your work different from [1] and [2] given that they both are parallel speculation frameworks and [2] even has the same [MASK] token design?
2. **Empirical**-wise, it is important to have evaluation results for a larger model (LLaMA-2-70B) where the inference speed-up is more crucial. Although the current results can show the potential of your draft model design when scaled to a larger model, it would be great if your method could still consistently achieve the best speed-up ratio on a 70B model. If time allows, it is more promising if you can compare it with EAGLE-2.

Minor issue: The wall time trace diagram (right) in Figure 1 is a bit misleading. The latency saving shown in the diagram is only guaranteed if the parallel decoded draft model has the same speculation accuracy as the auto-regressive decoded draft model. So, I would suggest adding this assumption in the caption and the paragraph that mentions the figure (the second paragraph in the introduction section).



**references**

[1]. Stern, Mitchell, Noam Shazeer, and Jakob Uszkoreit. "Blockwise parallel decoding for deep autoregressive models." Advances in Neural Information Processing Systems 31 (2018).

[2]. Xia, Heming, et al. "Speculative decoding: Lossless speedup of autoregressive translation." (2022).
soundness: 2
strengths: 1. The problem is well-motivated as the speculation overhead can indeed play a crucial role in the end-to-end speed-up ratio brought by speculative decoding. 
2. The paper's empirical evaluations show promising results for the proposed ParallelSpec approach, where the EAGLE+ParallelSoec and Medusa+ParallelSpec can consistently achieve a better speed-up ratio than the base version. 
3. The presentation is clear and illustrative.
summary: To deal with the speculation overhead from the auto-regressive draft model in speculative decoding, the paper proposes a parallel decoding approach for speculation by adding additional mask tokens in the vocabulary. Within a single draft model forward step, several future tokens can be speculated in parallel with the [mask] tokens used as temporary input tokens. The parallel speculated tokens are then fed into the target model for verification.  Empirical evaluations show that, assisted with this parallel decoding scheme proposed by the paper, current state-of-the-art speculative decoding frameworks, including EAGLE and Medusa, can further achieve a higher speed-up ratio due to the reduction in the speculation overhead while preserving a similar speculation accuracy.
weaknesses: 1. The concept of parallel decoding is not new. In fact, [1] has proposed a similar idea based on a parallel-decoded drafter and a target model as a verifier for speculative decoding. Additionally, Medusa itself is a parallel decoded speculation approach. Compared with these existing works, this paper mainly differentiates in its draft model architecture design that leverages the [MASK] tokens as a placeholder to predict future tokens within one model forward pass. However, this design is similar to the approach proposed in [2].

2. Though the empirical evaluations in the paper demonstrate promising end-to-end speed-up results on Vicuna-7B/13B and LLaMA-2-7B/13B, the evaluations on a larger model (e.g., LLaMA-2-70B) are missing. I guess this is due to the speculation overhead for a larger target model being less dominant as the theoretical improvement of minimizing speculation overhead is $\frac{\text{Latency(Verification)+Latency(Speculation)}}{Latency(Verification)}$ assuming that the speculation accuracy is the same. However, the speculation accuracy and overhead are usually a trade-off, so achieving this theoretical speed-up in practice is hard. 

3. If I understand correctly, even using the same dataset, the way you trained your draft model results in a larger processed dataset due to the existence of parallel groups, making the training of your drafter less efficient than EAGLE. 

4. Minor: as the EAGLE results in the SpecBench are a bit outdated, it would be interesting to compare your methods against the latest version (EAGLE-2).



**references**

[1]. Stern, Mitchell, Noam Shazeer, and Jakob Uszkoreit. "Blockwise parallel decoding for deep autoregressive models." Advances in Neural Information Processing Systems 31 (2018).

[2]. Xia, Heming, et al. "Speculative decoding: Lossless speedup of autoregressive translation." (2022).

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: Here are my questions and some suggestions for this paper:

1. EAGLE adaptation. In the standalone EAGLE algorithm, assuming the input embeddings are $e_{< t}$, the first step of drafting proceeds as $\tilde f_{t} = \pi (f_{<t}; e_{<t})$, and the second step goes as $\tilde f_{t+1} = \pi([f_{<t}, \tilde f_{t}]; [e_{<t}, \tilde e_t])$, where $[\cdot, \cdot]$ represents concatenation, and all other notations follow this paper. Given that the hidden states $f$ are also generated autoregressively, how is this handled in ParallelSpec? In my understanding, the draft model should forward once to generate all the draft tokens, i.e., $\tilde f_t, \dots, \tilde f_{t+K+1} = \pi ([f_{<t}, ?\dots ?]; [e_{<t}, {[MASK_1]}, \dots, {[MASK_K]}])$. What should be placed in the question marks, or is my understanding of EAGLE+ParallelSpec's drafting process incorrect?

2. Inconsistency results with official SpecBench. The baseline results of Medusa and EAGLE in Table 1 seems to be lower than the official SpecBench [1] results. The speedup of Medusa on Vicuna-7B is 1.42 in Table 1 while the official SpecBench gives 1.78, and speedup of EAGLE on Vicuna-7B in Table 1 is 2.18 while the official result is 2.29. Could you provide an explanation on the reason why the speedup gap occurs?

3. The organization of Table 2. The upper and lower halves of Table 2 should be presented in separate tables. They cover different topics: the upper half focuses on self-distillation, while the lower half discusses Llama-3.

4. Compatibility with dynamic draft trees. I am curious about the speedup gain of ParallelSpec when applied to dynamic draft trees, such as in EAGLE-2. Is this method compatible with dynamic trees, and would its strong performance persist? This is just a question for discussion, and the authors are not required to conduct such experiments during the review process.

[1] https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md#vicuna-7b-v13
soundness: 3
strengths: This paper makes significant contributions to the topic of speculative decoding, offering a technique that can enhance speedup ratios. There are several strengths of this paper (which I personally find very inspiring), and the most important ones are listed below:

1. Clear presentation. The paper clearly outlines the background for developing this technique, and the introduction to related works is distinct and comprehensive. Figures and tables are presented well, making them easy for readers to understand. Important hyperparameters and experimental settings are clearly stated. The way experimental results are presented is highly readable.

2. Simple but effective method. The idea of transforming the autoregressive decoding process of the draft model into parallel decoding is brilliant, and the implementation of group-wise parallel training is simple, clear, and effective. The adaptations to existing methods are minor, and the proposed method can be integrated with most prior work.

3. Strong performance. The experimental results show a substantial speedup gain when ParallelSpec is applied. The proposed method outperforms state-of-the-art methods without significant training costs.

4. Well-designed experiments. ParallelSpec is compared with popular baselines in this field using mainstream benchmark datasets. The ablation studies focus on key components of the system's design.
summary: This paper introduces a new technique, ParallelSpec, to advance the topic of speculative decoding. The proposed method addresses the efficiency barrier in autoregressive decoding within the draft-then-verify paradigm by adding new [MASK] tokens and predicting the next $K$ tokens in a single draft model call. ParallelSpec is orthogonal to methods that require training a draft model through distillation with the target model. To improve the ability to predict multiple subsequent tokens, the paper proposes "group-wise parallel training," where groups of [MASK_1] ... [MASK_K] are inserted after each token in the training corpus, with predicting the correct token at the corresponding position as the training objective. Extensive experiments evaluate the effectiveness of ParallelSpec's design. ParallelSpec is implemented on Llama-2-chat-7B/13B and Vicuna-7B/13B, trained on top of Medusa and EAGLE, and evaluated on SpecBench, where it is compared to popular baselines for speculative decoding. ParallelSpec outperforms all baselines on the tested data, achieving greater wall-time speedup ratios. The paper also includes comprehensive ablation studies, investigating the effects of different $K$ values and self-distillation. Overall, ParallelSpec is an effective technique that can be combined with most speculative decoding algorithms to improve efficiency.
weaknesses: Although this paper makes valuable contributions, there are certain flaws that prevent it from receiving a higher rating. The weaknesses are listed as follows, and the overall rating could be improved if these issues are addressed:

1. Unfair competition between standalone Medusa and Medusa+ParallelSpec. In the original Medusa setup, the Medusa heads are organized in an MLP-like architecture. However, when Medusa is adapted with ParallelSpec, the draft model is a single-layer Transformer model (lines 266-267). To what extent is the speedup gain due to the change in draft model architecture, as opposed to predicting draft tokens in parallel? A baseline should be provided where the only alteration is the draft model architecture. Otherwise, the modified setting should be given a different name, as the current name might create confusion regarding the draft model architecture.

2. Unclear presentation of how EAGLE is adapted with ParallelSpec. The description in lines 355-357 is insufficient. It would greatly improve clarity to provide a more detailed explanation of the training and inference process when adapting EAGLE, even in the Appendix if space in the main text is limited. This weakness is further elaborated in Question Q1.

Meta-review: (a) Summary of Scientific Claims and Findings

This paper introduces ParallelSpec, which leverages mask tokens to enable the drafter model to predict multiple tokens simultaneously within the speculative decoding framework.

(b) Strengths of the Paper

1. Demonstrates notable speed improvements for speculative decoding frameworks.

2. Provides robust empirical validation supported by strong theoretical foundations.

(c) Weaknesses and Missing Elements

1. The approach is considered incremental, bearing significant resemblance to prior works (e.g., Xia et al., Stern et al.).

2. Lacks evaluations on larger models, which are crucial for assessing the effectiveness of speculative decoding techniques.

(d) Decision and Rationale
Reviewers expressed significant concerns regarding the novelty of this work, citing substantial similarities with existing approaches.

Decision: Reject