EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Authors: Yize Wu
KE GAO
Ling Li
Yanjun Wu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
inference acceleration
speculative decoding
Primary area: deep_learning
TL;DR: This paper presents a layer-parallel speculation strategy for optimizing the effiency of multi-GPU utilization during the drafting stage of speculative decoding.

Abstract: Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, a layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependencies in the draft model, enabling multiple layers to run simultaneously across multiple devices as ``fuzzy'' speculation. After each drafting-and-verification iteration, the draft model’s key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation at minimal additional latency. EasySpec is a training-free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve a peak speedup of 4.17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with a maximum speculation accuracy drop of only 7\%. The code is available at https://github.com/Yize-Wu/EasySpec.

EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Yize Wul.2
Ke Gao'
Ling Li
Yanjun Wul
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China University of Chinese Academy of Sciences, Beijing, China {wuyize2021, gaoke, liling. saaiiig.yanuuu fanjun]@iscas.....
Abstract
Speculative decoding is effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependency in the draft model, enabling multiple layers to run simultancously across multiple devices as "fuzzy" speculation. After each trafting-and--i verificciiiiti iteration, the draft model's 's key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation minimal additional latency. EasySpec is training-fres free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve peak speedup of 4. 17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with maximum speculation accuracy drop of only 7%. The code is available at http:://gitubb.com/.ii Wizs-W//aa/SSSpe...
Introduction
Transformer- based Large Language Models (LLMs) have demonstrated remarkable problem--olving abilities across various domains (1)5). However, as the parameter size continues to grow, the time- consuming process of auto-regressive decoding poses a significant barrier to deploying large models in latency-sensitive applications (6] 7
Various effective approaches [8 have been proposed to reduce inference latency, including speculative decoding [1] and lensor-paralle, (TP) distribution [33. Speculative decoding employs a smaller model to generate a draft token sequence, and uses token-level parallelism to conduct non-autorcgressive verification, ensuring no shifting of the original model's output distribution. In contrast, TP distribution leverages cross-device parallelism by partitioning computational workloads across multiple devices (usually GPUs) and synchronizing the results subsequently, which is also lossless.
Combining speculative decoding with TP distribution achieves even greater acceleration ratio. How- ever, integrating draft model into distributed system is not trivial 7 Since the parameter size of
*Coreespodingg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Speculative Decoding
EasySpec
Fuzzy Speculation
Speculation
Draft
M
fuzzy
items
MLP
Atinoutpuxx
(ttr((a)
Attention,
draft model
MI.PP
MLP,
Attention
Attention,
Attention;
Attention,
Verify
Discard fuzzy KV items of ALL token(s)
Discard rejected token(s)
Bonux coleen
2
base model
Speculation
Bonus Calibration
4
I-forward pass
MLP,
precise KV items
0
draft model
Attention,
draft model
ML.P
2
3
Attention
Figure I: Overview of EasySpec: comparison between standard Speculative Decoding and EasySpec. The differences are mainly about: (1) Fuzzy Speculation uses layer parallelization for drafting acceleration. (2) The fuzzy KV items are discarded regardless of acceptance length, while Speculative Decoding preserves accepted items. (3) Speculative Decoding is 2-staged and EasySpec is 3-staged, with additional stage for bonus calibration.
the draft model is typically smaller than that of the base model, the optimal TP size (the number of
segments of workload distribution) correspondingly smaller [12], meaning that the draft model would run fastest when dispatched on one or a subset of GPUs, leaving other GPUs idle (see Fig- ure 2 and Table 6). Consequently, multi-GPU computational resources are under-utilized during the drafting stage.
We identify the primary cause of such inefficiency as the lack of parallelism between the draft model's layers: while tensor operations within one layer can be parallelized by TP, the layers themselves are restricted to be executed sequentially, one after another and from bottom to top, for generating "precise" result of inference. However, the drafting result is never required to be precise, as it is only used for token parallelism and does not directly impact the final outpur-the verification result does (see Section 1.... Therefore, strictly following the execution order is unnecessary, while "fuzzy" but faster layer execution strategy could be preferable than the precise one, as long as it can sufficiently approximate the drafting result.

---

draft model
base model
ille
GPUD
GPC1
TP
GPU2
GPU3
GPU0
TP
GPU1
GPU2
SD
GPU0
Easy
GPU1
Spee
GPU3
Figure 2: GPU runtime of different decoding algorithms in the distributed system. Speculative Decoding (SD) enables token-level parallelism (fewer base-model blocks), while causing multi-GPU under-utilization during the drafting stage. EasySpec solves the problem by layer-parallel speculation.
Based on this insight, we propose EasySpec, a layer-parallel speculation strategy for optimizing the efficiency of drafting-stage multi-GPU utilization. EasySpec introduces fuzzy speculation, which breaks the data dependencies between some consecutive attention layers by running them with the most recent hidden state as input all of them (Figure and Algorithm 2 As the data dependencies are eliminated, multiple layers of the draft model can execute simultaneously on separate devices, achieving layer-level parallelism, more efficient multi-GPU utilization and hence faster speculation inference. Mcanwhile, the speculation outputs can be well- vell-approximated retaining high speculation accuracy.
The approximation errors in the key-value (KV) cache of the draft model may accumulate during the inference procedure. To prevent this, perform bonus calibration after each iteration of drafting- and-verification. Firstly, the accepted tokens are concatenated to the bonus token to form token sequence. This sequence is then re-input the draft model, where conventional layer-sequential forward pass is executed, and the KV cache is updated with the precise values. This calibration step applies token parallelism, which is typically used only in the verification stage, the drafting stage as well. Therefore, it incurs minimal additional latency to the speculation procedure, while significantly enhancing uuturaaccyulimmtlllyyy contributing improved overall inference speed.
We evaluate our method on several widely-used spencsource LLMs and task-specific models. The draft models selected from the same series their corresponding large models. The evaluation results show that EasySpec can achieve peak speedup 4. 4x over vanilla decoding. Specifically, the drafting stage can be accelerated by up 1.62x with greater than 7% drop of speculation accuracy, requiring no additional training or fine-tuning on the existing draft models.
2
Preliminary
2.1 Speculative decoding
Speculative decoding is two-stage non-autoregressive decoding method for inference acceleration. The two stages are namely drafting and verification, which are iteratively executed as inference proceeds. At time step I, the input token sequence is X. In the drafting stage, a smaller and faster model M' employed as the draft model. M' auto-regressively runs times, generating speculation token sequence X, x .., I'4n and probability distributions P'+1..... P/+m. The original large model M then takes the whole sequence as input and conducts a fongle-forward verification, the outputs of which are P++1, Pt..++- The acceptance probability of token af is
min(1, (x()) After the verification sampling, the sequence of m accepted tokens E Fttm

---

(the same as ".+,,,.., m) will be appended to the final output, along with a "bonus" token 2++m++1 (called bonus because is by-product of the verification process) which can be sampled from the distribution p:
norm(max (0,aaxx000000 Pt+m+1 P(4m--1) (m n)
(m < n)
(1)
P++++1
Through this whole verification process, the final output token sequence aligns with the original distribution the base model. Details of this algorithm are in Section A
2.2 Tensor-parallel distributed inference
Tensor parallelism is technique for deploying large models across multiple devices (GPUs). As for TP distributed inference, tensors split up into multiple chunks, and each of these chunks is dispatched on a unique GPU. During computation operations, each GPU works on its respective chunk independently at first, and the results are synchronized afterwards to ensure that the final result is identical to the original. In modern multi-GPP systems, advanced hardware and communication protocols enable TP be applied for inference acceleration.
The optimal TP size is primarily influenced by the parameter size. Firstly, the TP size must evenly divide both the hidden dimension and the number of attention heads. For example, in Qwen-2-0.5B- Instruct these numbers are 4864 and 14 respectively, which limits the maximum feasible TP size 2. Secondly, the computational workload varies with parameter size, which further impacts TP efficiency. A detailed discussion of these device-level characteristics is provided in Section 2.3
2.3 Device characteristics
Modern GPUs feature powerful computation capacities and hence short computation latency. There- fore, matrix computation involved LLM inference is usually bottlenecked by memory bandwidth control-flow overheads [14.. In multi-GPU systems, the computation also be bounded by communication and synchronization overheads.
Formally, we denote the workload of 1-token forward W, the latency of executing W on single GPU as Teme (W), and the additional overhead of TP distribution as Taddi- The workload of s-token parallelism is sW, and W/s s the workload of s-sized TP. If W is relatively small compared to the computation power, we can expect the following relationships:
Teze Teze(W) Teze (sW) (W)
Teze (W/s) + Tadai > Texe (W)
(3)
The effectiveness token-level parallelism can be illustrated by Equation 2 which is leveraged by speculative decoding for acceleration. Equation (3) highlights the impact of communication and control-flow overheads in distributed inference, which constrains the TP size not to be too large and limits multi-GPU utilization of the draft model.
3 Method
In this section, we present the details how EasySpec enhances multi-GPU parallelism to accel- erate the drafting stage of speculative decoding. The two key steps steps-fuzzy fuzzy speculation and bonus calibration- are specified within each segment.
3.1 Fuzzy speculation
As mentioned in Section I the conventional speculation process restricts the layers to be executed sequentially. Algorithm I demonstrates the execution procedure of N consecutive layers. we eliminate hi in the algorithm, we have
h4+1 h; + Altnoutpul; + M LPuutput;
which illustrates the data dependencies between the input of layer i + and the output of layer In other words, before the outputs of the previous attention layer (Attnoutput.) and MLP layer

---

Algorithm 1 Layer-Sequential Speculation
Algorithm 2 Layer-Parallel Fuzzy Speculation
Input: hidden state h. N consecutive atten- tion layers Allmn.... and MLP layers
Input: hidden state h. N consecutive atten- tion layers Allmm...,AAln and MLP layers
M.P MMLN h1 = h
MLP1. MMLN h1 = for to N do Attnoutput; = (tin,(h1) (parallel)
for i 1 to N do
Attnoutput; = Attn;(h;) (sequential) := h; + Attooutput;
end for for
MLPoutput; = MM (h))(h))
= to N h h4 Attnoutput,
h4+1 K2 + MLPouppt;;
end for
M L Poulpuut = MLP,(h)
h4+1 M4 + MLPoulpul,
end for
(MLPoutput.) are computed, nothing can be done with the upper layer i (and + 2 N as well).
Such restriction the execution order is necessary when the draft model is directly used for inference. Executing the model differently would shift the output probability distributions p' to p". undermining the quality of generated content. However, the situation is different when the draft model is just used for speculation. Regardless of how p" deviate from the original, as long as we set the acceptance probabilities min(1,p/v"), the final output distribution will theoretically remain the same as the base model. This leads to a potential optimization of speculation strategy: if the output distribution p' can be efficiently approximated by p", such that p" p', the overall process may speedup, while the final outputs remain lossless. That is to say, slightly fuzzy but faster approach for speculation could outperform the precise one.
We propose an effective approximation method for fuzzy speculation. Inspired by [13), observe that the attention layers' outputs can be well approximated by substituting the in- put h; with hn for all < i S N. In Equa- tion (4), the hidden state update hi+1 h; Attnoutpul; + . L.P Poulput,, is relatively less significant, so the cosine similarity between h, and h441 can approach 1, and consequently the similarity between all h; and h1 also high (Figure 3 is a 2-D demonstration of this phe- nomenon). Such high similarities of the input allow other hidden states to be accurately ap- proximated as well (see Section E. for theo- retical explanations and Table 4 for empirical results).
hg
Attnoulpult
MLPoudpul,
Figure 3: 2-D demonstration of high cosine simi- larity between h, and h
Algorithm| 2 shows the detailed steps of this method. From the perspective of model architectures, the input of the attention layers is not the most recent hidden states, but rather those from several layers prior (Figure []. With this modification of layer execution strategy, we break the data dependencies between N consecutive attention layers: once h, is computed for some j. all attention layers between j to j + N can be executed with no data dependencies among them, thereby achieving a layer-level concurrency. In a multi-GPU system, these attention layers can run simultaneously on different devices (Figure 2
Denote the execution time of sequential and fuzzy speculation over these N attention layers Tsag and T Tuzzz, we have
T Kegg = NTeze (A) Tuuzy = Teze (A) + T Taddi
where A is the workload of one attention layer. Unlike the traditional TP method, the latency of running an attention layer is long enough to compensate for the additional overhead, that is
5

---

Table 1: Example layer- parrr-p strategies for drafter models. The strategies for other drafters is the same as specified in Section 3.1.1
Model
of Layers
N
Strategy
Llama-3-8B-Instre Llama-3.22 22-1BIInstruct
32
4
0, 1-3, 4-7, 8-11, 24-27, 28-30, 31
16
2
0, 1-2, 5-6, ...
11-12, 13-14, 15
Qwen2-7B-Instee IIIntructt
28
4
1-3, 4-7,8-11, 20.223,
24-26, 27
(N - 1)Teze >> Taddi for N > 1, resulting in Tiuzzy Tseq. The MLP layers are executed in the original sequential order, as parallelizing them results in significant approximation errors and degraded end-to-end performance (Section B
In order to achieve a higher acceptance rate, EasySpec also incorporates tree attention [16 in the drafting stage, a technique commonly used in existing speculative decoding methods 18) Tree attention can boost the acceptance rate by increasing the number of draft token sequences. As shown in Figure [5] the drafted tokens are structured Speculation and verification for all the sequences can be performed in a single forward pass using a 2D tree-attention mask.
tree, with each path corresponding to sequence.
3.1.1 Layer-parallel strategy
For the layer-parallel size N, we exclude the first and last layer and parallelize 1 ~ N 1, N ~ 2N 1, ..., until the remaining layers (except for the last) not enough to compose Neset. For instance, for Llama-3-8B-Instte parallelized, while layer 28,29, and 30 have compose a 3-set because layer 31 is the last (see Table D The reason of such strategy is that, parallelizing intermediate layers (excluding the first and last layer) yields best performances in our experiments. We hypothesize that this is because the first and last layer are more sensitive to approximation errors.
(32 layers) and N = 4, we have ~ 3, 4 ~ 7, .... 28 ~ 30
Note that there could be better strategy for models and/or tasks, but this strategy has already yielded sufficiently satisfactory acceleration. Moreover, trying other strategies is simple and lightweight, as requires no modifications to the original model architecture. A change in the configuration is enough.
3.2 Bonus calibration
(' Formally, the speculation of token j can be denoted as koj_ - kolej 1 (2//-1)
Fuzzy speculation induces errors to the value of ,', p' and kv'. The errors x' and p' are limited within single draft-verify iteration, since the rejected tokens and all output distributions will be discarded after verification. However, the errors kv will affect all subsequent generations after leading to long-term error accumulation that can severely degrade the accuracy.
The precise values of kv' are required for preventing such error accumulation. Nevertheless, obtaining these values demands running the model's 's layers sequentially, which is exactly what we try to optimize through fuzzy speculation. To solve this dilemma, we propose an efficient way of obtaining precise KV values with minimal additional latency. At the beginning of a standard drafting stage, the bonus token F1+m+1 from the last iteration is input to the draft model, for the speculation of 2'+ +2- This is a 1-token forward pass and also nemory-boundedd as discussed in Section 2. Therefore, similar to the verification stage, token parallelism can also be applied here. By concatenating the accepted token sequence T1+1, Jtt+ with the bonus token F++m++ and re-inputting this entire sequence Zt+1, ... Jt+m: 21+m+11 to the draft model, the precise KV values kv/=1..... koy (and kv'+m+1) ) can be produced through single sequential forward pass. According to Equation the latency of such a forward is nearly equal to a I-token sequential forward.
We propose bonus calibration based on the method above. As shown in Figure I we firstly discard all fuzzy KV items, regardless of whether the corresponding tokens are accepted or rejected (e.g. the accepted items 1 and 2 are also discarded). Then, token-parale forward is conducted to refill the KV cache with precise values. From the perspective of execution, the precise KV values are by-products of the next-token speculation of the bonus token, similar to how the bonus token itself is produced by the verification. This is the reason why we call this process "bonus". Bonus calibration can generate precise candidate token and distribution, while subsequent rounds of speculation can

---

revert to be fuzzy. The overall latency increases slightly. from nT Juzzy to Tseg + (n 1)T (n the speculation length), while the elimination of errors is crucial and substantial. More detailed theoretical backing is in Section F.2
4
Experiment
Models and benchmarks. We evaluated EasySpec on Llama-3-70B-Instrun and Qwen2-72B: Instruct, as well as 1a-tma---3-70--Intttee and task-specific models Owee2-Matt-722-nnstte
and [19]. The SBB/L5///B//B/8 models in the same series serve as the draft models for their respective base models. The benchmarks include variety of tasks: language understanding (MLU((0), code generation (HumanEval (21)., math reasoning (MATH(22), in- struction following (IFEval (3 and multilingual language usage (MGSM(24), All the experiments were conducted using chain-of of- ofe-off-thhhe thought reasoning with maximum of 128 tokens. We also evaluated our method on Spec-Bench (25 for comparison with related work.
Dwen2.5-Coder-32B--
Environments and configurations. The experiments were conducted on 8xA100 GPUs. Unless specified otherwise, the layer-parallel size N is set 4, as it is optimal for most of the tested models. The tensor-parallel sizes for drafter and base models are and 8, as they are the optimal in most cases (see Table 6). The optimal and applied speculation length is
Baseline settings. We use the following baselines for comparison: tensor-paralle distributed decoding (TP), speculative decoding in the distributed system (+SD), and tree attention (+tree) [16] Additionally, we compare our work with EAGLE-2 [8].
Metrics. We use token throughput of single-batch inference the performance metric. We also measure the running time per 100 tokens of the drafting and verification stages, which demonstrates the effectiveness of acceleration more clearly. Acceptance rates are also recorded and presented.
4.1 Main results
4.1.1 Effectiveness
To illustrate the robust effectiveness of EasySpec, we conducted experiments on Llama-3-70B- Instruct and Swen2-72B-Instrrue across datasets, with temperatures=0 and 0.8. The results are shown in Table 2 We measure the running time per 100 tokens of the drafting (d) and verification (v) stages and the total acceleration ratio (total) compared to vanilla decoding. represents the acceptance rates. The exact total running time of all baselines are in Table
The results show that EasySpec consistently accelerates the overall execution of speculative decoding across all datasets. While the verification stage (the large models) can accelerated by TP, the drafting stage often becomes the primary bottleneck, accounting for large portion of the total running time. EasySpec accelerates the drafting stage by up to 1.62x, thereby improving the overall throughput. Meanwhile, the drop of speculation accuracy is no more than 7%, suggesting that fuzzy speculation with bonus calibration sufficienlly accurate across all evaluated cases.
We also evaluated EasySpec on two task-specific models: Qwen2-Math Barth-72B-Instiee and Qwen2.5- Coder-322 Instructt along with some other combinations drafting and base models. The effective- ness of EpsySpec in these cases are shown in Table 7 and Table S
4.1.2 Generalization and stability
An intermediate question is that whether we can use an even shallower model to reduce drafting latency. Firstly, such a model often does not exist within the same series, while training draft model from scratch is non-trivial and the trained drafter often lacks generalization capability across base models and datasets [26). Secondly, performance, while smaller versions of model in the same series inherently have alignments in behaviors with the larger, due to their shared tokenizer, pretraining corpora and similar training process 25
overly shallower drafter tends to exhibit unstable speculation
To clarify the above statements, we compare our method with EAGLE-2 18] typical tiny-drafter speculative decoding method. FAGLE-2 uses a well-trained one-layer draft model, aiming for extreme drafting latency reduction. We evaluated both methods on Spec-Bench (25). Besides average token
7

---

Table 2: Running time 100 tokens, total acceleration ratio and acceptance rates of two mainstream models across datasets. 'd', 'v' stands for drafting and verification running time (s/100 tokens) respectively, and 'total" and stands for total acceleration ratio and acceptance rates. More detailed numbers are Table
Llama-3-708B 3-700(88))---ssiii
(wven2-22B77B)
Dataset
Method
total
d
total
temperature=0
TP +sd
1.53x
1.56x
3.52
2.15
2.05x
0.57
3.32
2.25
2.13x
0.52
MMLU
+tree EasySpec
2.52
1.59
2.82x
0.88
2.38
1.62
2.96x
0.85
1.70(11.48x)
1.73
3.38x
0.82
1.65(11.44x)
1.70
3.55x
0.80
TP
1.55x
1.57x
+sd
2,93
1.79
2,50x
0.74
2.82
1.83
2.58x
0.69
HE
+tree
2.53
1.58
2.87x
0.92
2.26
1.51
3.18x
0.95
EasySpec
1.6((11.57x)
1.63
3.64x
0.87
1.48(+1.52x)
1.54
3.97x
0.91
TP
1.52x
1.54x
+sd
2.96
1.74
2,45x
0.73
2.48
1.65
2.86x
0.78
MATH
+tree
2.50
1.47
2,90x
0.95
2.20
1.45
3.24x
0.96
EasySpec
1,8((1158x))
1.55
3.68x
0.91
1.44(11.2x))
1.47
4.06x
0.95
TP
1.50x
1.52x
+sd
3.68
2.16
1.93x
0.55
4,24
2.80
1.64x
0.39
IFEval
+tree
2.53
1.55
2.76x
0.89
2.80
1.84
2.49x
0.72
EasySpec
1.8((11.51)
1.65
3.39x
0.82
1.87(11.00))
1.94
3.04x
0.67
1.54x
1.56x
+sd 2.66
1.61
2.73x
0.80
2.62
1.75
2.73x
0.72
MGSM
+tree 2.45
1,48
2.96x
0.96
2.12
1.50
3.29x
0.94
EasySpec
1.55(+1.58x)
1.51
3.80x
0.93
1.54(+1.37x)
1.57
3.83x
0.88
temperature=0.5
TP +sd 3.09
1.53x
1.56x
1.93
2.32x
0.67
2.90
1.93
2.47x
0.65
MMLU
+tree EasySpec
2.49
1.54
2.89x
0.94
2.14
1.48
3.30x
0.95
1.59(11.56x)
1.63
3.61x
0.89
1.49(11.43x)
1.54
3.94x
0.93
TP
1,54x
1.59x
+sd 2.94
1.81
2.48x
0.73
2.54
1.63
2.91x
0.80
HE
+tree EasySpec
2.47
1.49
2.98x
0.96
2.08
1.45
3.44x
0.97
1.53(+1.62x)
1.49
3.90x
0.94
1.45(+1.44x)
1.46
4.17x
0.97
TP
1.53x
1.55x
MATH +sd 2.81
1.65
2.59x
0.78
2.44
1.54
2.98x
0.85
+tree EasySpec
2.44
1.45
2,97x
0.97
2.09
1.40
3,39x
0.99
1.52(+1.61x)
1.48
3.86x
0.94
1.43(11.47x)
1.44
4.13x
0.98
TP
1.50x
1.51x
IFEval +sd
3.08
1.87
2.29x
0.67
3.24
2.14
2.16x
0.56
+tree 2.45
1,46
2.89x
0.97
2.17
1.51
3.16x
0.91
EasySpec
1.52(11.61x)
1.50
3.73x
0.92
1.53(11.42x)
1.58
3.73x
0.87
TP
1,55x
1.57x
+sd
2.54
1.55
2.85x
0.84
2.68
1.72
2.73x
0.76
MGSM
+tree
2.30
1.44
3.13x
0.98
2.08
1.45
3.41x
0.97
EasySpec
1.52(+1.51x)
1.49
3.88x
0.96
1.46(11.42x)
1.50
4.06x
0.94
throughput, also calculate the variances of throughput across data items, for better demonstration of performance stability. The temperature is set to 0.8 for both methods.
The results in Table 3 show that, for Llama-3-708 P7B--Instruch the FasySpec--cceleratte 1B drafter achieves higher average token throughput than EAGLE-2 (=1 across all tasks, suggesting that smaller model from the same series, if existing, is good enough to deliver satisfactory performance with Epey for performance stability, EasySpee exhibits significantly smaller variances, which should be attributed to its remarkably higher speculation accuracies (nearly 0.8 vs less than 0.4), For --B-
the performance gap becomes even more evident, further indicating that trained drafters lack generalization capability to other base models, whereas smaller models from the

---

Table 3: Average token (broughpu((variancc) and acceptance rates of EasySpec and EAGLE-2 for Llama-3-70B- 3---0---nntt-tet on Spec-Bench, The format of throughput is mean(variance) stands for acceptance rates.
EAGLE-2
EasySpec
EAGLE-2
EasySpec
Llama-3-70B-Instee
Llama-3.3-700-Insttu
Task
Throughput
Throughput
Throughput
Throughput
MT
37.15(31.27)
0.38
(1.8g
0.85
25.92(14.28)
0.24
38.69(23.1)
0.84
TI.
32.97(17.28)
0.38
35.31(15.41)
0.72
19.05(8.46)
0.15
34.14(16.06)
0.71
SUM
36.34(9.30)
0.40
39.18(4.88)
0.80
24.23(3.63)
0.20
37.67(5.03)
0.77
QA
31.85(20.15)
0.34
40.38(9.82)
0.84
22.50(10.36)
0.19
40.30(3.41)
0.84
MR
38.10(14.97)
0.45
43.52(4.44)
0.93
27.09(6.14)
0.25
44.59(2.49)
0.96
RAG
37.26(38.15)
0.47
38.15(12.56)
0.83
23.25(15.40)
0.22
36.79(13.75)
0.80
Table Average cosine similarities between precise and approximated hidden states. The tested model and task are 1aama-3-8B-Instreee and MMLUU.
LP size
h
k
Attnoutput
0.93
0.98
0.99
0.92
0.93
0.89
0.97
0.98
0.86
0.88
0.86
0.96
0.97
0.82
0.83
same series remain compatible. Note that EasySpec requires training or fine-tuning on either the target distribution or fuzzy approximation, making it a plug-in method of higher adaptability.
4.2 Approximation precision
The feasibility of fuzzy speculation relies on the approximation precision of the draft model, which is closely related the cosine similarities between the precise and fuzzy-approximated hidden states. Table/4shove average cosine similarities between these hidden states. The tested drafter is Llama-3- 8B-Instruct and the benchmark is MMLU. The evaluated hidden states include input hidden states h. outputs of attention layers Allnoulpul, and queries(g), keys(k) and values(v) inside attention heads. We set layer-parallel sizes to 2,3,4 for diverse approximation granularity. The results show that, as more layers are parallelized, the cosine similarity decreases but stays above 0.8, indicating a sufficiently high precision of approximation. Notably, the queries and keys are well-approximated with cosine similarities approaching 1. This is critical for maintaining precision of other hidden states and outputs, as errors in attention weights scale exponentially.
4.3 Ablation study
We conducted an ablation study aspects of EasySpec: tree attention width, layer-parallel (LP) size and the presence or absence of bonus calibration. We varied the tree width across 1,4,8, 12 and I.P size from 1 to 5. The results of these different configurations are presented in Figure 4
For token throughput (left part), as the LP size increases from I, the throughput improves due to better multi-GPU utilization of fuzzy speculation. However, when the LP size becomes too large (c.g. 5), the speculation results are excessively fuzzy, leading to a noticeable decline in token throughput. The optimal LP size is influenced by the tree width: wider tree increases the likelihood of existence of correct tokens, thus allowing for a fuzzier and faster speculation (a larger LP size). Moreover, even without tree attention (width=1), EasySpec can also accelerate the inference with layer parallelism (see the green line in Figure 4).
For speculation accuracy, the right part shows that bonus calibration can significantly improve acceptance rates for every combination tree width and I.P size. Bonus calibration also slows the decline in acceptance rates with increasing LP sizes (see the right part of Figure 4). All the above

---

30
Token throughput
Acceptance rate
25
0.6
20
0.4
Layer-parallel
Layer-parallel
1 calibration
width=1 calibration
d4 w/o calibration
width-4 w/ callbratian
calibration
wi 12 w/o calibration
width-12 w/ calibration
Figure 4: Token throughput and acceptance rates Baeimaa--70OBBB------ on MMLU under different configurations of EasySpec. Left: token throughput. Right: acceptance rates.
results prove that the trade-off between precision of KV values and additional latency is reasonable and effective.
Related work
Methods focusing on the drafter stage can be broadly categorized into training drafters, self- speculation or algorithmic optimization. Specinfer [16 uses boost-trained draft models and tree- structured attention for efficient speculation. Medusa 7 trains set of extra MLP heads for future
token prediction using the original L.LM's features. Self-Speculative Decoding 277 and LayerSkip
(28 skips some layers of the target LLM for self-speculation. Lookahead (26) uses n-gram Jacobi decoding to increase acceptance rates. REST [29] leverages data storage for retrieval speculation. Recent work also explores optimized stuuctruct attention mechanisms. SpecExec [30] takes the most probable continuations from the draft model to build cache tree, improving inference efficiency essource-constraingg GPUs. OPT-Tree [1] proposes adaptive and scalable draft-tree construction algorithm that maximizes the expected acceptance length. Beyond the drafter itself, other studies investigate stage-level parallelism and theoretical explanations of speculative decod- ing. For instance, PEARL 32 overlaps the drafting and verification stages to achieve stage-level pipeline, while SpecTr optimization.
33 adopts optimal transport theory provide theoretical explanation and
The discrepancy between optimal distribution sizes is discovered by the authors of [12], who suggested training a wider but shallower draft model for better performance under tensor parallelism. There are currently few works focusing this problem, yet it is important and worth-solving, To the best of our knowledge, we are the first to effectively optimize multi-GPU utilization of surrentl--vvailatt draft models.
6 Conclusion
In this paper, we propose EasySpec, layer-pan aayer-parallel speculation strategy for efficienct multi-GPL utilization during the drafting stage of speculative decoding. EasySpec introduces two modifications original speculation process: fuzzy speculation and bonus calibration. Fuzzy speculation breaks the sequential layer execution order and enables multi-layer parallelization, while bonus calibration applies token parallelism to the drafting stage to eliminate long-term error accumulation. EasySpec Training-free and consistently achieves acceleration across models and datasets.
10

---

Acknowledgments
This work partially supported by the NSF of China (under Grant 92364202), and Major Program of ISCAS (Grant No. ISCAS-ZDD-202002)
References
[1] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems,
2017.
[2] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[3] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arxia: org/ abs/ 2407 10671
[4] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[5] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303300877,,20 2023.
[6] Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024.
[7] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10771 2024.
[8] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint 133.511503.02531, 2015.
[9] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference machine learning, pages 5506- 5518. PMLR, 2021.
[10] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[11] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[12] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.001318 2023.
[13] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint 1rXiv:1909.0803: 2019.

---

[14] Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sye -naars quantization. arXiv preprint arXiv:2306.07629, 2023.
[15] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. Infinigen: Efficient generative inference of large language models with dynamic kv cache management. In 18th USENIX Symposim on Operating Systems Design and Implementation (OSDI 24), pages 155-172, 2024.
[16] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 332-949, 2024.
[17] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024.
[18] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024.
[19] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang. Yang Fan, Yichang Zhang, An Yang. Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URL https //arxiv. org/ abs /4099. 12186
[20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 1rXiv:2009003300 2020.
[21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint vxXiv:2107.03374 2021.
[22] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 2021.
[23] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cettuuttionfolloviiii evaluation for large language models. arXiv preprint 2tf1::2/11.09/// 2023.
[24] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual thain-of-thought reasoners. arXiv preprint arXiv:2210.0300. 2022.
(25) Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.0788. 2024.
[26] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of lmm inference using lookahead decoding. In International Conference on Machine Learning, pages 14060-14079. PMLR, 2024.
[27] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08166 2023.
[28] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculatiie decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 12622-12642, 2024.
12

---

[29] Zhenyu He. Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-basee speculative decoding. arXiv preprint arXiv:2311.08252, 2023.
[30] Ruslan Svinschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive Ilm inference on consumer devices. Advances in Neural Information Processing Systems, 57:16342-166
2024.
[31] Jikai Wang. Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, and Min Zhang. Opt-tree: Speculative decoding with adaptive draft tree structure. Transactions the Association for Compulational Linguistics, 13:188-199 2025.
[32] Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. Pearl: Parallel speculative decoding with adaptive draft length. The Thirteenth International Conference on Learning Representations, 2025.
[33] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: fast speculative decoding via optimal transport. In Proceedings of the 37th International Conference Neural Information Processing Systems, pages 30222-30242. 2023.
[34] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
13

---

A Speculative decoding
Speculative decoding consists of iterations of drafting and verification. The detailed algorithm of single iteration is present in Algorithm 3
Algorithm An lteration of Speculative Decoding
Input: input token sequence X, drafting length draft model M', base model M
for i m do
E.P. = M (X, jKi)
end for
P1. Pm Pm+1 (X, .a...... zz'm)
sample r1. T' independently from (0,
for i 1 to do
if
p((m)) ( then
n
(n the accepted length)
break (rejected)
else
= (accepted)
end if
end for
< m then
Phonus norm(max Pr. pl())
else
Phonus Pn+1
end
sample Thonus from Phonus return X, ... Ens Tbonus
For any distribution p', the final output token sequence is equivalent to sampling them directly from The proof is specified in many existing works (e.g. (II).
B Strategy for MLP layers
In EasySpec, the MLP layers are executed sequentially, rather than parallelized like the attention layers. This design choice stems from the empirical results that parallelizing all layers (both attention and MLP) introduces substantial approximation errors, leading to degraded overall performance. As shown in Table 55 even parallelizing just N 2 full layers results in lower token throughput and reduced speculation accuracies compared to the optimal attention-only parallelization strategy, with performance deteriorating further as N increases.
C Tree attention
In the context of speculative decoding, tree attention can leveraged to speculation and verification for multiple possible token sequences in single forward pass. With 2-dimensional attention mask
Table 5: Token throughput and speculation accuracies of models with different parallel strategies on MMLU.
Model
Parallelized Layer
Token throughput
a
Only Attention
2
34.37
0.77
Llama-3-70B(IBB-nnnttrcc
Full Layer
29.71
0.58
Only Attention
29.94
0.76
Qween2-72B(1.5BB)Innnm
Full Layer
2
27.37
0.68
Full Layer
24.10
0.51
14

---

10
10
20
2.1
21
222
233
23
24
25
Figure 5: Illustration of attention. draft token sequences are generated within forward passes, which increases parallelism and potentially acceptance rates. The whole tree can be verified in single-forward pass of the base model.
Table 6: Token throughput models at different TP sizes on MMLU. '*' means the results are from vLL..
Model
TP=1
TP=2
TP=4
TP=8
L3-70B
8.39
13.00
13.23
13.23
L3-70B
OOM
19.5
28.47
28.25
Q2-72B
8.49
13.03
13.01
12.89
Q2-72B
OOM
18.91
28.63
29.39
L3-8B
36.76
33.90
32.31
32.39
L3-8B
79.39
68.86
68.35
67.76
Q2-7B
37.16
36.46
37.12
Q2-7B*
83.38
73.9
73.81
applied to the flattened token tree, token sequences with the same prefix can share the KV-items, thus increasing token-level parallelism. As an illustration Figure 5 sequences are generated.
with 2 forward passes, 6 draft token
D Additional results
D.1 Results different TP sizes
Table the token throughput of some typical tested models different TP sizes. The benchmark consists of 40 data items from MML.U. While the large models benefit from multi-GPU distribution, the draft models experience slowdown when the TP size is greater than 1 (due to Equation (3),. confirming that the under-utilization of multi-GPU resources cannot be solved by traditional TP. We also tested the throughput in another widely-used inference engine vLLM [4], and the results are consistent with the above claim. This consistency supports the conclusion that the acceleration and
slowdown are indeed attributable to multi-GPU configurations, rather than other possible factors. Based on these findings, use 1 and the test-time distribution size for all small and large models respectively, without compromising generalization.
Note that Qwen-2-7B- 2-B--Instuttt does not support TP size=8, as its internal dimension size is not divisible by 8. That is also an aspect of lacking parallelism (other than the slowdown of inference speed) and can be solved by EasySpec as well.
15

---

Table 7: Running time per 100 tokens and acceptance rates of two task-specific models across datasets.
The denotation consistent with Table 2
Q255-Coder-322B(HH--nmee
Datasets
Methods
total
vanilla
7.13
Human
+tree
2.15
1.23
3.38
0.97
Eval
EasySpec
1.43
1.23
2.66
0.97
-0.72
+0.00
-0.72
-0.00
Q2-Math-72BB(BB)-Instrccc
vanilla
11.55
MATH
+tree
2.05
1.51
3.55
0.99
EasySpec
1,40
1.52
2.91
0.98
-0.65
+0.01
-0.64
-0.01
Table 8: Token throughput of other model combinations w/ and w/o EasySpec. The temperature is O.
The left and right numbers are throughput w/o and w/ EasySpec respectively. 'w/o EasySpec' the same baseline as +tree' in Table 2
Models
MMLU
HumenEval
MATH
IFEval
MGSM
Q2-72B-1.5B
24.02
29.94
26.28
33.78
27.54
36.11
20.65
25.17
25.92
33.26
Q2-72B-0:5B
24.57
29.29
28.80
33.15
29.61
35.82
20.57
24.08
27.48
32.63
L3-70B-3B
25.41
31.32
26.81
32.52
27.49
35.78
25.20
30.29
27.23
35.18
L3-70B-1B
32.10
34.37
34.25
37.25
35.70
40.00
28.60
34.04
35.78
40.25
L3-8B-1B
47.97
56,49
50.99
61.88
54.04
64.97
49.54
56.01
52.68
63.68
D.2 Results of task-specific models
Table 7 shows the results of two task-specific models the corresponding datasets MATH and Hvaa these datasets, original speculation results of the draft models is already highly accurate (with the acceptance rates greater than 97%), indicating that the speculation is inherently simple. As a result, fuzzy speculation incurs virtually no drop in accuracies, achieving an ideal drafting- stage acceleration.
D.3 Results of other combinations of draft and base models
We further tested EasySpec on other combinations of draft and base models. The results in Table illustrates that EasySpec shows acceleration across all model combinations and datasets, highlighting the wide adaptability of our method. By 'without EasySpec', we mean that the baseline is 'ttree' in Table 2 'L3'' and 'Q2' stand for Llama-3 and Qwen2 respectively. For the optimal overall performance, the layer-parallel size of Llama-3.2-1 B.2-1BBInntrrue is 2, and the size of others
As stated in Section 4.1.2, using smaller drafters does not necessarily result in overall optimal performance. Furthermore, it should be noted that Qwen-2-1.5B and Qwen-2-0.5B only support TP size < 4 and 2, since the dimension size is not divisible by a larger number (similar to the Qwen 7B model), while EasySpec increases the parallel sizes by introducing extra layer-parallelism.
D.4 Detailed numbers of results
Table 9 shows detailed numbers of results of EasySpee and baselines. Compared to Table 2 the total running time per 100 tokens, the running time of vanilla decoding are listed.
16

---

E
Theoretical backing
E.1
Connection between cosine similarity and approximation precision
Current transformer-bas models typically apply layer normalization (LayerNorm) operation before the attention block. Given two input hidden states, h1 and h2 (where h2 is an approximation of f2), the LayerNorm operator transforms them into normalized representations h and fig, which are then used in subsequent attention computations. Importantly, the LayerNorm operation preserves ( 12). When this cosine similarity is close to 1, the
cosine similarity, i.e., cos(A1, h2) corresponding vectors are also close in Euclidean distance (lh, h2] 0), ensuring that downstream computations maintain high numerical precision. For models without LayerNorm operation, the demonstration Figure also implies that the approximated hidden states can remain precise (in
principle).
E.2 Balance between speculation latency and accuracy
The strategy of speculative decoding (SD) is about finding an optimal balance between the running time of drafters Taraft and the speculation accuracy Denote the running time of base model as Thase: the overall running time of decoding N step without speculative decoding is N Thase. The overall running time with SD can be seen function of Taraft and a: Tan( ()) = N * Täraft + (N/mn * a)) Thase (n is the speculation length), since the base model only needs to run N/(n= 0:) times. The key our method is reducing Taraft largely while maintaining a high which will lead to decrease in the overall time.
Denote the running time of fuzzy speculation as Tuuse and original sequential run as Tori. If we just run fuzzy speculation for every token, the time will be T Tzraft n T'uzz Bonus calibration runs at every beginning of the speculation. As token parallelism is applied (just like being applied to verification in normal SD), it has basically the same running time as Tori. The full speculation time then becomes Taraft (n 1) T Juzz Tori
fuuzz Tdraft- If that were all the story, Tall would increase. However, the is largely increased by the calibration in the meantime (sce in Figure 4. since the fuzzy KV items the cache are replaced by precise ones. Consequently, bonus calibration helps to find better balance between Taraft and
The theoretical analyses above complement the empirical evidence, further supporting that our algorithm can effectively accelerate the inference process.
F
Limitations
The experiments are conducted on a specific hardware configuration of 8xA100 GPUs and a specific inference system, and the acceleration rate is likely to vary across different platforms due to variations in computation speeds and consor-paralle efficiency. However, as hardware continues to improve. computational resources may become increasingly redundant, further reducing the benefits of tensor parallelism. Since this issue going to get worse with faster hardware and operators, we believe effective solution by optimizing the utilization of multi-GPU that layer parallelism offers resources now and in future.
G Broader impacts
Our work enables faster inference by accelerating speculative decoding in a layer-parallel approach. We believe it has positive broader impact for the area of LLM inference acceleration. To the best of our knowledge, there no specific negative impacts be discussed.
17

---

Table 9: Detailed results of running time of drafting and verification per 100 tokens, total running time and acceleration ratio, and acceptance rates of two mainstream models across datasets, at temperature=0 and 0.8. The denotation basically consistent with Table 2
Llama-3- 3eae----77BBB)--mmiiee
Qwen2- 72B(7))BBsBBitt) Instruct
Dataset
Method
total
total
temperature=0
vanilla
11.61
11.88
TP +sd +ice
7,57(1.53x)
7.59(1.56x, 16x))
MMLU
2.15
5.67(2.05x)
3.32
5.57(2.135 13x)
2.52
1.59
4.11(2.82x)
0.88
2.38
1.62
4.01(2.96x)
0.85
EasySpec 1.70(11.8xx)
1.73
3.44(3.38x)
0.82
1.65(11.4x)
1.70
3.35(3.55x)
0.80
vanilla
11.79
11.98
TP
7.58(1.55x)
7.64().57x)
HE
+sd
2.93
1.79
4.72(2.50x)
0.74
2.82
1.83
(.65().58x)
0.69
+trce
2.53
1.58
4.11(2.87x)
2.26
1.51
3.77(3. 18x)
EpsySpec
161(71.57x)
1.63
3.24(3.64x)
1.8((11.22k)
3.02(3.97x)
vanilla
11.52
11.82
TP 2.96
7.56(1.524
7.68(1.54x)
MATH
+sd
1.74
4.70(2.45x)
0.73
2.48
1.65
4.13(2. 86x)
0.78
+tree
2,50
1.47
3,97(2 (.9xx
0.95
2,20
1,45
3.65(3.24% (3x))))
0.96
EasySpec
1.58()1.58x)
1.55
3.13(3.68))
0.91
1.4((1152x)
1.47
2.91(4.06x)
0.95
vanilla
11.29
11.59
TP 3.68
7.55(1
7.64(1.52x)
IFEval
+dd
2,16
5.84(1.93x)
0.55
4.24
2.80
7,04(1.64x)
0.39
+iree
2.53
1.55
4.09(2.76x)
0.89
2,80
1.84
4.65(2.49x)
0.72
EasySpec
168((1151x)
1.65
3.33(3
0.82
1.8(()1500))
1.94
3.81(3.04x)
0.67
vanilla
11.65
11.92
TP 2.66
7.55(
7.63(1.56x)
MGSM
+sd
1.61
4.27(2.73x)
0.80
2.62
1.75
(.37k)
0.72
+tree
2.45
1.48
3.93(2.96x)
0.96
2.12
1.50
3.62(3.29x)
0.94
EasySpec
1.55()1.58x)
1,51
3.06(3.80x)
0.93
1.54(t1.37x)
1.57
3.11(3.83x)
0.88
temperature=0.8
vanilla
11.62
TP 3.09
7.59(1.53x)
7.66(1.56x)
MMLU
+sd
1,93
5.02(2.32x)
0.67
2,90
1.93
4.82(2.47x)
0.65
+tree
2.49
1.54
4.02(2.89x)
0.94
2.14
1.48
3.62(3.300 30x)
0.95
EasySpec 1.56()
1.63
3.22(3.61x)
0.89
1.49(+1433x)
1.54
3.03(3.94x)
0.93
vanilla
11.79
12.14
TP
7. 5.64().54x)
7.65(1.59x)
HE
+sd
2.94
1.81
4.75(2.48x)
0.73
2.54
1.63
4.17(2.91x)
0.80
+tree
2.47
1.49
3.95(2.98x)
0.96
2,08
1.45
3.53(3.44x)
0.97
EasySpec
1.53(11.62x)
1.49
3.02(3.90x)
0.94
1.45(1144k)
1.46
2.91(4.17x)
0.97
vanilla
TP
7.54(1.53x)
(.63(1
MATH
+sd
2.81
1.65
4.46(2.59x)
0.78
2.44
1.54
3.98(2.98x)
0.85
+tree
2.44
1.45
3.89(2.97x)
0.97
2.09
1.40
3.5K(
0.99
EasySpec
1.52(11.61x)
1.48
3.00(3.86x)
0.94
1.43($1.47))
1.44
2.87(4.13x)
0.98
vanilla
11.30
11.63
7.53(1.50x) 50x)
1.72(
IFEval
+sd
1.87
4.94(2.29x)
3,24
2.14
5,38(2. 16x)
2.45
1.46
3.91(2.89x)
2.17
1.51
3.68(3. 16x)
EasySpec
1.52(11.61x)
1.50
3.03(3.73x)
0.92
1.53(11.42))
1.58
3.11(3.73x)
0.87
vanilla
11.67
12.02
TP
7.55(1
7.65()
MGSM
+sd
2,54
1.55
4.10(2.85x)
2.68
1.72
4.41(2.73x)
0.76
+tree
2,30
1.44
13xx)
2.08
1.45
3.53(3.41x)
EasySpec
1.52(11.51x))
3.01(3.88x)
1.46(11.42x)
1.50
2.96(4.06x)
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper provides EasySpec, a layer-parallel speculation strategy for efficienct multi-GPU utilization during the drafting stage of speculative decoding, introducing two modifications to the original speculation process: fuzzy speculation and bonus calibration. The abstract and introduction clearly state these contributions
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses the limitation of the work in Section where the limitation of specific software and hardware configuration is discussed.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theoretical result in the paper. We did provide theoretical backing for our method in Section E while it is stated for further clarification of the empirical results. Therefore, we do not think it is in any way theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We believe that the detailed experimental settings are provided, in Section and some places in the paper. They include model types, datasets, layer parallel strategies, speculation length, and so on.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
20

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code and the benchmarks are all available in the repository link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https (oodeSummissiooooal
/ //nips. httpps.cc/publicgguu.eeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: As mentioned in 'Experimental result reproducibility', believe that detailed hyperparameters have been provided for understanding.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Similar many existing works in this area, do not include error bars confidence intervals in the empirical results, as repeated trials show negligible variation. System-level fluctuations have only marginal effects on the outcomes.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The system configuration is specified Section and Section F
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurr: httphherripp.ccppuleieeeeeeeeeee cewubllc/Ethiceeo ctciideeeeelllesss
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts discussed in Section
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To the best of our knowledge, this paper poses no issues on safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited original papers for code, datasets content. The license and terms use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide assets the form of code, which is well documented and com- mented alongside.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips hett::///earrrpssccc//ooerrreeee cc///oofeereeeee//22////... for what should or should not be described.
25


Reviews:
[{"review_id": "73Fufp9yPu", "reviewer": "Reviewer_d76v", "summary": "This paper proposes a more parallelized speculative decoding method, made by modifying draft models to run multiple layers in parallel. They demonstrate that this leads to a real improvement in distributed systems compared to distributed normal specdec. They compare against competitive baselines like Eagle 2, with detailed benchmarking of the time spent in drafting and verification.", "questions": "How costly is training new draft heads for Eagle 2? Could you generate draft heads for Llama 3.3, and compare? \n\nHow well do distilled models perform as draft models in this use case? Would it be possible to use one of the Llama models trained with distillation to see if it gives superior draft performance?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nTheir method is interesting. Its a essentially \"free\" modification of existing inference pipelines to enable faster decoding. There are definite applications in cases where you do not want to fine tune an additional model. Furthermore, it addresses a real problem with draft model based approaches to speculative decoding, which is their poor utilization of hardware.\n\nNeutral:\n\nThere is an existing literature in parallelized LLMs (see, for example, ladder transformers and diffusion LLMs)\n\nWeaknesses:\nI do think that the Llama 3.3 comparison against Eagle 2 listed here is unfair. Eagle 2 achieves better performance than EasySpec given a draft model in the same family and trained heads, as shown in the Llama 3 comparison. When adapting to llama 3.3, the performance of easyspec is better, but Eagle 2's draft heads are much easier to train than an entire smaller draft model. \n\nFurthermore, the cost of this type of method is that two different KV caches need to be maintained, which increases memory usage.", "was_revised": true, "final_justification": "New experiments with 1B draft models noticeably increased the throughput. Before, the paper was a borderline rejection due to being a highly empirical method without clear empirical superiority. With the improved results, I am not only convinced the method is more practically useful but that it represents an interesting avenue for future research.", "rebuttal": "We sincerely thank you for the review! \n\n> I do think that the Llama 3.3 comparison against Eagle 2 listed here is unfair. Eagle 2 achieves better performance than EasySpec given a draft model in the same family and trained heads, as shown in the Llama 3 comparison. When adapting to llama 3.3, the performance of easyspec is better, but Eagle 2's draft heads are much easier to train than an entire smaller draft model.\n\nWe believe that your concern is whether EasySpec can outperform existing state-of-the-art (SOTA) methods like EAGLE-2 on already-trained models LLaMA-3-70B, rather than LLaMA-3.3.\n\nIn Section 4.1.2, our goal is to show that, EasySpec can improve the performance of non-trained 8B drafter to 1B trained EAGLE-2 head. To emphasize the gap of drafter sizes (8B vs 1B), we choose the 8B model as drafter, which is quite larger than 3B or 1B one, and hence the performance could also be inferior (but comparable enough to EAGLE-2 already). \n\nTo address your concern regarding SOTA performance, we conducted additional experiments using LLaMA-3.2-1B as the drafter on SpecBench:\n\n| | mt-bench | translation | summarization | question-answering | math-reasoning | rag |\n| ---- | ----  | ----  | ----  | ----  | ---- | ---- |\n| 1B w/o EasySpec |33.83 | 30.36 | 33.01 | 31.39 | 37.16 | 33.74 |\n| EAGLE-2 |33.81 | 30.44 | 33.26 | 30.98 | 34.37 | 33.7 |\n| 1B w/ EasySpec |35.15 | 35.03 | 36.11 | 36.12 | 39.17 | 36.32 |\n\nWithout EasySpec, the 1B drafter only achieves comparable performance to EAGLE-2, and with EasySpec it clearly outperforms EAGLE-2 across all tasks. This confirms that EasySpec is effective in improving SOTA average throughput as well. Both methods are evaluated under the same hardware and software conditions, for fairness of comparison.\n\n>How well do distilled models perform as draft models in this use case? Would it be possible to use one of the Llama models trained with distillation to see if it gives superior draft performance?\n\nFirst of all, the smaller model in the same series can be treated as a 'distilled' model indeed. It is a 'distilled' version of the larger, since they share the same training dataset and configurations, and therefore have quite aligned behavior.\n\nEasySpec is orthogonal to distillation (at least theoretically). It leverages the intrinsic property of hidden-state similarity across consecutive layers, which is commonly observed in various models. Distilled models often deviate just slightly from the non-trained original, so we believe this phenomenon will preserve.\n\nWe also observe that distilled drafters often have unstable performance, as shown in EAGLE-2's experiment. This problem arises from overfitting to specific training datasets, leading to loss of generality. \n\nFurthermore, distillation can only improve the acceptance rate, rather than reducing the runtime of drafting stage. It is demonstrated that the smaller model in the family (as a 'distilled' version upon the vast pre-training corpora) already performs sufficiently well on acceptance rate (e.g., exceeding 0.8), so it is not the bottleneck, but the runtime of drafting stage is. Further distillation may yield marginal improvements while requiring substantial extra effort.\n\nSince EasySpec already improve the performance of non-distilled 1B drafter better than other baselines, we did not conduct any further experiments on whether to distill the model, and it is indeed out of the scope of our method: training-free and plug-in."}, {"review_id": "kIR9jwsFrf", "reviewer": "Reviewer_RLfo", "summary": "This paper proposes EasySpec, a layer-parallel speculative decoding method designed to accelerate the drafting stage in large language model inference. In conventional speculative decoding, the draft model is typically small and cannot fully utilize multi-GPU resources. EasySpec addresses this by breaking the sequential dependency between attention layers in the draft model, allowing them to run in parallel across multiple GPUs. To mitigate the KV cache errors introduced by this fuzzy execution, it introduces bonus calibration. Experiments are conducted on several large models and tasks, measuring drafting and verification time, overall speedup, and acceptance rate.", "questions": "1. While bonus calibration is described as lightweight, is there any scenario in which the cost of re-running the draft model outweighs the benefits of layer-parallelism? Some further clarification would be useful.\n2. In fuzzy speculation, the attention layers are executed in parallel using the same input, which generates a set of KV caches. It remains somewhat unclear whether these fuzzy KV caches are meaningfully used in subsequent token generation before being replaced.\n3. EasySpec shows strong performance with medium-sized draft models. However, could it extend to significantly smaller drafters?\n4. PEARL [1] also focuses on accelerating the drafting stage and overlaps the drafting and verification stages to achieve stage-level parallelism. It would be helpful if the key differences between the two approaches could be clarified. \n\n[1] Liu, T., Li, Y., Lv, Q., Liu, K., Zhu, J., Hu, W., & Sun, X. (2025). Pearl: Parallel speculative decoding with adaptive draft length. In The Thirteenth International Conference on Learning Representations.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n+ The paper focuses on a practical performance bottleneck in current large language model inference pipelines—the under-utilization of the drafting stage in multi-GPU settings, making the topic of practical importance.\n+ The paper is well-organized and easy to follow. Background concepts such as speculative decoding and tensor parallelism are explained with sufficient clarity. The method is described clearly, aided by formulas, algorithm pseudocode, and illustrations.\n\nWeaknesses:\n\n- While the paper identifies the drafting stage as a performance bottleneck, the motivation could be communicated more compellingly. The Introduction currently lacks concrete system-level profiling, visual illustrations, or quantitative comparisons that would help emphasize the severity and relevance of the problem.\n- Table 3 presents cosine similarities of intermediate hidden states to support the approximation quality of fuzzy speculation. It would be helpful to include output-level comparisons (e.g., token distributions or generation divergence) to complement the current analysis.\n- Figure 1 illustrates the core idea of layer-parallel attention execution but does not provide a full depiction of the entire EasySpec workflow. In particular, bonus calibration and KV cache correction are critical components of the method and would benefit from visual inclusion in a complete process diagram.", "was_revised": true, "final_justification": "Many thanks to the authors for their extensive response and great effort. I feel more positive about this work and would like to upgrade my score.", "rebuttal": "We sincerely thank you and the ACs for your time and effort.  \n\n>While bonus calibration is described as lightweight, is there any scenario in which the cost of re-running the draft model outweighs the benefits of layer-parallelism? Some further clarification would be useful.\n\nThanks for your suggestion! During our experiments, we did not observe such a scenario in any of our experiments. As shown in our ablation study, bonus calibration consistently improves performance under all tested conditions. This is because, without calibration, the fuzzy errors will accumulate as the generation length increases, eventually leading to a significant drop in accuracy. Our setting of generation length is averagely 128, so this degradation is particularly harmful to overall performance. Consequently, the computational cost of bonus calibration is much outweighed by the gains in speed and accuracy, making it a consistently valuable component of our approach.\n\n>In fuzzy speculation, the attention layers are executed in parallel using the same input, which generates a set of KV caches. It remains somewhat unclear whether these fuzzy KV caches are meaningfully used in subsequent token generation before being replaced.\n\nAs shown in Figure 2, the fuzzy KV caches are used only during a single speculation phase, where they serve to generate N speculated tokens (N = 5 in our experiments). Once these tokens are passed to the verification model, the acceptance decisions are returned, and the corresponding calibrated KV cache entries are generated to replace the fuzzy ones. Therefore, the impact of cache fuzziness is strictly limited to a single drafting stage, and does not propagate beyond that phase.\n\n>EasySpec shows strong performance with medium-sized draft models. However, could it extend to significantly smaller drafters?\n\nThe results on smaller drafters, such as 0.5B, 1B, 1.5B or 3B model, are in Table 8 of the appendix. As shown, EasySpec is effective in all the drafter-base model combinations across various tasks.\n\n>PEARL also focuses on accelerating the drafting stage and overlaps the drafting and verification stages to achieve stage-level parallelism. It would be helpful if the key differences between the two approaches could be clarified.\n\nIn fact, EasySpec and PEARL are fundamentally orthogonal in design. Speculative decoding consists of two stages: drafting and verification. PEARL focuses on inter-stage acceleration, proposing strategies to overlap these two stages for multi-GPU efficiency—either by initiating verification before all draft tokens are generated, or by continuing drafting before verification results are returned. In contrast, EasySpec targets at intra-stage acceleration. It accelerates the drafting stage alone, using fuzzy speculation and then calibration. Thus, EasySpec and PEARL can be seen as complementary approaches that optimize different aspects of the speculative decoding pipeline. We will cite PEARL in the paper for more clarification.\n\n\n>Table 3 presents cosine similarities of intermediate hidden states to support the approximation quality of fuzzy speculation. It would be helpful to include output-level comparisons (e.g., token distributions or generation divergence) to complement the current analysis.\n\nThe cosine similarity provides a model-internal perspective on the behavior of EasySpec. For output-level evaluation, we think the most compelling metric is the acceptance accuracy, which directly reflects how well the fuzzy speculation approximates the precise generation. As noted, the speculation accuracy decreases by only 7% at most, and in certain cases—such as code and math tasks—the accuracy drop is even 0. This indicates that layer-parallel fuzzy speculation can closely approximate the true token distribution.\n\nWe give further clarification about why cosine similarity is a good metric: for input hidden states $h_1$ and $h_2$, as there is a normalization layer at the entry of the attention module, $h_1$ and $h_2$ become normalized vectors $h'_1$ and $h'_2$, which are input to the subsequent matrices. It is easy to see that cos($h_1$, $h_2$) = cos($h'_1$, $h'_2$), so the cosine similarity between $h_1$ and $h_2$ is indeed reflecting the approximation precision of modules' input.\n\n>Figure 1 illustrates the core idea of layer-parallel attention execution but does not provide a full depiction of the entire EasySpec workflow. In particular, bonus calibration and KV cache correction are critical components of the method and would benefit from visual inclusion in a complete process diagram.\n\nThat is a very useful suggestion! We did improve Figure 1 and some other figures to strenthen the clarity, but we cannot link or post them here. It will be put into further versions."}, {"review_id": "CwFj1hnx2S", "reviewer": "Reviewer_Ubkp", "summary": "Conventional speculative decoding has low multi-gpu utilization when the draft model is small. The paper considered the sequential multi-layer inference pipeline as the main bottleneck and proposed a “fuzzy” speculation method, EasySpec, that runs multiple model layers in parallel. To avoid accumulated KV cache errors and potential performance drop, EasySpec introduced Bonus Calibration that recomputed the KV cache using the accepted tokens after each draft-verification cycle. The performance of EasySpec has been evaluated across different LLMs, e.g. LlaMA-3-70B (8B), Qwen2-72B(7B), etc, and different benchmarks, e.g. MMLU, Spec-Bench, etc. Experiments showed that EasySpec achieved lower inference latency compared to several baseline approaches, e.g. TP, tree-attention, etc.", "questions": "N.A.", "limitations": "Limitations were included in the Appendix", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "---\nStrengths\n\nMethod: EasySpec proposed to “break” the sequential dependency of the hidden states across consecutive blocks/layers to a certain extent, which is technically sound. Prior research in LLMs has indicated that the hidden states in consecutive layers may exhibit limited changes in several scenarios, given the use of skip connections [1]\n\nExperiment: the paper provided extensive LATENCY evaluations on auto-regressive inference, including comparisons to alternative approaches on various benchmarks.\n\n---\n\nWeaknesses\n\nAccuracy: it was indicated in the paper that EasySpec may lead to 7% accuracy drop, but no further discussions were included in the paper. This is a main concern. From the paper, EasySpec is not lossless. While EasySpec demonstrated superior throughput, it is not clear to the reader if EasySpec achieved at least comparable accuracies to the baselines. The authors are encouraged to include accuracy comparisons to the baselines across different benchmarks, similar to the latency evaluations. Ablations on the tradeoff between the speedup and the accuracy drop are also highly recommended, as 7% accuracy drop is high.\n\n\nBaselines: the reader may expect at least one representative TP baseline. However, from L192, it looks like the baseline drafter did not use TP (i.e. TP size == 1). From Table 6, the optimal TP size was determined on MMLU. The reviewer has two questions: i) since MMLU involves only classification tasks that requires no autoregressive generation, the reviewer wondered if the optimal TP sized was determined under the autoregressive generation setting or prefilling setting? ii) Any chance the optimal TP size varies across benchmarks?  \n\nPresentation: several figures/tables were referred in the main paper but presented in the appendix, e.g. Figure4, Table 6. The authors are encouraged to reposition the figures and tables\n\n\n[1] The Unreasonable Ineffectiveness of the Deeper Layers. ICLR 2025.", "was_revised": true, "final_justification": "The reviewer appreciated the authors' responses and clarification that resolved the reviewer's concerns. The reviewer raised the rating the paper accordingly.", "rebuttal": "We sincerely thank you for the review. \n \n>It was indicated in the paper that EasySpec may lead to 7% accuracy drop, but no further discussions were included in the paper. This is a main concern. From the paper, EasySpec is not lossless. While EasySpec demonstrated superior throughput, it is not clear to the reader if EasySpec achieved at least comparable accuracies to the baselines. The authors are encouraged to include accuracy comparisons to the baselines across different benchmarks, similar to the latency evaluations. Ablations on the tradeoff between the speedup and the accuracy drop are also highly recommended, as 7% accuracy drop is high.\n\nFirstly, there is a huge misunderstanding here. The term 'accuracy' in speculative decoding are referring to the acceptance rate of speculated tokens—rather than the task-level accuracy. Speculative decoding is inherently a lossless inference acceleration technique, which has been both theoretically and empirically verified in many prior works. The decrease in speculation acceptance rate has no impact on the output quality. \n\nSecondly, we are trading a small drop in speculation accuracy not for overall better performance, but for faster drafting stage. Since the final output quality is not affected, the drop in speculation accuracy is completely reasonable if the drafting stage becomes faster. We believe the effectiveness of this trade-off has been thoroughly demonstrated and justified by our experimental results. Moreover, as a lossless inference acceleration method, the acceptance rate itself is not the ultimate goal, and the overall speed is. The only metric of a speculative decoding method should be the overall performance, which EasySpec achieve higher than previous TP or non-TP baselines.\n\n>The reader may expect at least one representative TP baseline. However, from L192, it looks like the baseline drafter did not use TP (i.e. TP size == 1).\n\nWe conducted extensive experiments with TP=4, and it demonstrates that TP>1 is harmful for the drafter and hence the overall speed. EaspSpec is indeed targeted at solving this problem.\n\n| Base-Draft | MMLU | IFEval | mgsm |\n| ----  | ----  | ----  | ----  | \n| TP=4 w/o EasySpec | | | |\n| 70B-3B | 22.96 | 21.89 | 23.50 |\n| 70B-1B | 29.25 | 27.02 | 29.88 |\n| 8B-1B | 43.38 | 41.56 | 47.75 |\n| TP=1 w/o EasySpec | | | |\n| 70B-3B | 23.06 | 22.33 | 23.81 |\n| 70B-1B | 29.46 | 27.44 | 30.80 |\n| 8B-1B | 45.12 | 42.83 | 49.90 |\n| EasySpec | | | |\n| 70B-3B | 29.72 | 26.95 | 29.54 |\n| 70B-1B | 32.01 | 29.22 | 33.92 |\n| 8B-1B | 52.23 | 50.45 | 53.88 |\n\n>Since MMLU involves only classification tasks that requires no autoregressive generation, the reviewer wondered if the optimal TP sized was determined under the autoregressive generation setting or prefilling setting?\n\nIn all experiments, including MMLU, we adopt chain-of-thought (CoT) inferece, which naturally results in multi-token autoregressive generation, not single-token outputs. The generation length is restricted to no more than 128. There should be no doubt about this experimental setting: speculative decoding is about a sequence of tokens to be generated and verified, so the experiments can never be purely 1-token generation.\n\n>Any chance the optimal TP size varies across benchmarks?\n\nThe optimal tensor parallelism (TP) size may vary across different tasks. However, in our experiments, this influence is negligible. For auto-regressive generation tasks, the prefilling stage typically accounts for only a small portion of inference time, with the majority dominated by the decoding stage. Decoding is basically a 1-token process, so it has similar computation-memory ratio across tasks, and the optimal TP size is also similar."}, {"review_id": "Mt9YNPD94r", "reviewer": "Reviewer_Pf1A", "summary": "This paper introduces EasySpec, a training-free, plug-in strategy for accelerating speculative decoding in multi-GPU systems. By breaking inter-layer dependencies, EasySpec enables parallel execution of draft model layers to maximize GPU utilization. It also calibrates KVcache errors with minimal overhead, and experiments demonstrate its effectiveness.", "questions": "The minor concerns are as follows:\n\n1. In Figure 1(b), EasySpec appears to achieve balanced GPU utilization. However, my understanding is that only the attention layers are distributed across GPUs, while the MLP layers remain sequentially executed on a single GPU — is this correct? Additionally, the attention layers do not seem to be evenly distributed across GPUs. For example, in Table 4 (N=4), Strategy=0 uses only one GPU while the other three are idle, and Strategies=1–3 utilize only three GPUs, leaving one unused. This seems inconsistent with the balanced load shown in Figure 1(b), which is somewhat confusing.\n\n2. Table 2 shows that EasySpec outperforms EAGLE-2 in throughput. Is EAGLE-2 using tensor-parallel (TP) decoding in this comparison? Also, are both methods evaluated under the same hardware conditions?\n\n3. During the drafting stage, given that multiple GPUs are available, is it feasible to use multiple distinct draft models simultaneously to generate draft candidates? For example, could both LLaMA-3-8B and Qwen2-7B be used in parallel to accelerate LLaMA-3-70B inference?\n\n**Suggestions**\n\nThere are many concurrent works on speculative decoding (e.g., [1–8]); please expand the Related Work section to include more of these, which will help readers better understand the development of this field and EasySpec's unique contributions.\n\n[1] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding\n\n[2] SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices​\n\n[3] Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding\n\n[4] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding\n\n[5] EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding\n\n[6] A Theoretical Perspective for Speculative Decoding Algorithm\n\n[7] BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models\n\n[8] OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper presents a meaningful attempt to explore speculative decoding in multi-GPU settings.\n\n2. The addressed problem is important and highly relevant for real-world LLM applications.\n\n3. The idea of fuzzy speculation is intriguing; while it may appear methodologically imperfect due to potential performance degradation, the paper convincingly argues that \"the drafting result is never required to be precise,\" and experimental results confirm its limited negative impact.\n\n4. The paper is well-organized and clearly written, making it easy to follow.\n\n\n**Weakness**\n\nThe main concerns are as follows:\n\n1. The layer-parallel (LP) configuration is currently based on empirical heuristics, lacking a systematic approach for partitioning attention layer sets. While the method is training-free, achieving optimal performance requires extensive manual tuning (as shown in Table 4). It would be valuable to explore automated strategies for configuring distributed attention layers.\n\n2. The applicability of EasySpec is somewhat limited, as it depends on the availability of a suitable draft model. This constraint makes it suitable mainly for LLMs where a compatible drafter is readily available (e.g., LLaMA-3-8B for LLaMA-3-70B, Qwen2-7B for Qwen2-72B). For other LLMs, significant output distribution gaps may exist. The paper reports results only on LLaMA-3-70B-Instruct and Qwen2-72B-Instruct. Including evaluations on additional models—such as using LLaMA-3-8B-Instruct to accelerate Vicuna-33B, LLaMA-2-70B, or Qwen2-72B-Instruct—would greatly enhance the paper's value.\n\n3. Key baseline comparisons are missing. While the paper notes that integrating a draft model into a distributed system is non-trivial, tensor-parallel (TP) inference for the draft model is theoretically feasible. Even though Table 6 shows limited speedup for TP > 1, including a direct comparison between a TP-enabled draft model and EasySpec would improve the experimental rigor.\n\n4. Further issues are discussed in the \"Questions\" section.\n\nOverall, despite these concerns, I find the paper refreshing and creative. My current score falls between borderline reject and borderline accept, but I am open to raising it following constructive author feedback.", "was_revised": true, "final_justification": "The response addresses the concerns I raised, and I find the work overall both meaningful and promising.", "rebuttal": "We sincerely thank you for the review! \n\nWe appreciate your positive feedback on the originality, quality and clarity of our paper.\n \n>While the method is training-free, achieving optimal performance requires extensive manual tuning (as shown in Table 4). It would be valuable to explore automated strategies for configuring distributed attention layers.\n\nThanks for your suggestion! In fact, the strategy is not manually tuned and heuristic, and it is quite easy to follow. For the layer-parallel size N, we parallelize 1 ~ N, N ~ 2N-1, ..., until the remaining layers (except for the last) are not enough to compose a N-set. For example, for N=4, we have 0, 1 ~ 4, 4 ~ 7, ..., 28 ~ 30 parallelized, while layer 28,29, and 30 have to compose a 3-set because layer 31 is the last. We will add such explanation to the paper.\n\nThe reason of such strategy is that, we observe the first and last layers should not be parallelized with other layers, and parallelizing other intermediate layers yields the best performance. We hypothesize that this is because the first and last layers are more sensitive to approximation errors.\n\nAs stated in appendix B.1, we are not sure wheter this strategy is the best, but it already has sufficiently satisfactory results. Moreover, simply trying other combinations is lightweight, as it requires no modifications to the original model architecture. A change in configuration is enough.\n\n>The paper reports results only on LLaMA-3-70B-Instruct and Qwen2-72B-Instruct. Including evaluations on additional models—such as using LLaMA-3-8B-Instruct to accelerate Vicuna-33B, LLaMA-2-70B, or Qwen2-72B-Instruct—would greatly enhance the paper's value.\n\nSpeculative decoding with heterogeneous models is not feasible, as it requires both models to share the same tokenizer. Specifically, token IDs must be identical across models, so cross-family speculation is impossible in principle (e.g., LLaMA to Vicuna or LLaMA-3 to LLaMA-2).\n\nWhile it is true that EasySpec requires access to a smaller drafter model from the same family, we think that our method still offers great adaptability. In practice, smaller variants are commonly available in major open-source families like LLaMA and Qwen. In contrast, training a new speculation model from scratch is resource-intensive and non-trivial, requiring data, infrastructure, and tuning expertise. Trained drafter can also have unstable performance due to overfitting, as we show that the variance of throughput is much higher.\n\nLastly, the smaller model does not have to be the exact same version as the larger one, as long as they share the same tokenizer. For example, Llama-3.2-1B/3B can be used to accelerate Llama-3-70B, as we already experimented on. This largely mitigate the alleged limitation.\n\n>While the paper notes that integrating a draft model into a distributed system is non-trivial, tensor-parallel (TP) inference for the draft model is theoretically feasible. Even though Table 6 shows limited speedup for TP > 1, including a direct comparison between a TP-enabled draft model and EasySpec would improve the experimental rigor.\n\nAs shown in Table 6, small models cannot benefit from TP>1; in fact, they often experience a performance drop compared to TP=1. This is not suprising, as small models are not computation-bound enough to benefit from tensor parallelism, and the additional communication overhead leads to inefficiencies. Consequently, the overall performance with TP>1 for drafter is often worse than TP=1,and even more worse than EasySpec.\n\nTo support this observation, we conducted experiments with TP=4 under various base-draft settings, and the results are presented below:\n\n| Base-Draft | MMLU | IFEval | mgsm |\n| ----  | ----  | ----  | ----  | \n| TP=4 w/o EasySpec | | | |\n| 70B-3B | 22.96 | 21.89 | 23.50 |\n| 70B-1B | 29.25 | 27.02 | 29.88 |\n| 8B-1B | 43.38 | 41.56 | 47.75 |\n| TP=1 w/o EasySpec | | | |\n| 70B-3B | 23.06 | 22.33 | 23.81 |\n| 70B-1B | 29.46 | 27.44 | 30.80 |\n| 8B-1B | 45.12 | 42.83 | 49.90 |\n| EasySpec | | | |\n| 70B-3B | 29.72 | 26.95 | 29.54 |\n| 70B-1B | 32.01 | 29.22 | 33.92 |\n| 8B-1B | 52.23 | 50.45 | 53.88 |\n\nThese results clearly demonstrate that EasySpec consistently outperforms both TP = 1 and TP > 1 baselines, reinforcing our claim that layer-parallel EasySpec is better suited than standard TP-based approaches.\n\n>My understanding is that only the attention layers are distributed across GPUs, while the MLP layers remain sequentially executed on a single GPU — is this correct? Additionally, the attention layers do not seem to be evenly distributed across GPUs. For example, in Table 4 (N=4), Strategy=0 uses only one GPU while the other three are idle, and Strategies=1–3 utilize only three GPUs, leaving one unused. This seems inconsistent with the balanced load shown in Figure 1(b), which is somewhat confusing.\n\nYour understanding of MLP sequential running is correct.\n\nFigure 1(b) illustrates the most typical configuration of layer parallelism in our approach, which is N=4 for intermediate layers. Special cases involving the first and last layers are not shown. We will add some extra explanation for it.\n\nDue to the first and last layer not parallelized, EasySpec does not achieve full GPU utilization (although it is significantly more efficient than using TP=1). However, our ultimate goal is not maximizing utilization, but achieving optimal performance. Therefore, if a slight under-utilization caused by the first and last layer ultimately results in better performance, it would be an undoubtedly better option.\n\n>Is EAGLE-2 using tensor-parallel (TP) decoding in this comparison? Also, are both methods evaluated under the same hardware conditions?\n\nYes, we evaluated EAGLE-2 under tensor parallelism (TP=8, the same as EasySpec), and all experiments were conducted under identical hardware and software conditions for fairness of comparison. EAGLE-2 does not natively support TP, so we re-implemented TP support for it by reusing components from our own codebase, without any modification to its core algorithm.\n\n>During the drafting stage, given that multiple GPUs are available, is it feasible to use multiple distinct draft models simultaneously to generate draft candidates? For example, could both LLaMA-3-8B and Qwen2-7B be used in parallel to accelerate LLaMA-3-70B inference?\n\nNo, it is impossible to do so, because the tokenizer of Qwen2 models is different from Llama's. \n\nLastly, we appreciate that you provided us more related work to enhance the contribution of our paper. Thanks to your detailed review!"}]

Author rebuttal:

Meta-review: This paper proposes EasySpec, a layer-parallel speculative decoding method designed to accelerate the drafting stage in large language model inference.  All reviewers agree that this work makes solid contributions, and the AC recommends acceptance.
Decision: Accept (poster)