Title: EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

inference acceleration

speculative decoding
Abstract: Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, a layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependencies in the draft model, enabling multiple layers to run simultaneously across multiple devices as ``fuzzy'' speculation. After each drafting-and-verification iteration, the draft model’s key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation at minimal additional latency. EasySpec is a training-free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve a peak speedup of 4.17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with a maximum speculation accuracy drop of only 7\%. The code is available at https://github.com/Yize-Wu/EasySpec.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EasySpec, a training-free, plug-in strategy for accelerating speculative decoding in multi-GPU systems. By breaking inter-layer dependencies, EasySpec enables parallel execution of draft model layers to maximize GPU utilization. It also calibrates KVcache errors with minimal overhead, and experiments demonstrate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This paper presents a meaningful attempt to explore speculative decoding in multi-GPU settings.\n\n2. The addressed problem is important and highly relevant for real-world LLM applications.\n\n3. The idea of fuzzy speculation is intriguing; while it may appear methodologically imperfect due to potential performance degradation, the paper convincingly argues that \"the drafting result is never required to be precise,\" and experimental results confirm its limited negative impact.\n\n4. The paper is well-organized and clearly written, making it easy to follow.\n\n\n**Weakness**\n\nThe main concerns are as follows:\n\n1. The layer-parallel (LP) configuration is currently based on empirical heuristics, lacking a systematic approach for partitioning attention layer sets. While the method is training-free, achieving optimal performance requires extensive manual tuning (as shown in Table 4). It would be valuable to explore automated strategies for configuring distributed attention layers.\n\n2. The applicability of EasySpec is somewhat limited, as it depends on the availability of a suitable draft model. This constraint makes it suitable mainly for LLMs where a compatible drafter is readily available (e.g., LLaMA-3-8B for LLaMA-3-70B, Qwen2-7B for Qwen2-72B). For other LLMs, significant output distribution gaps may exist. The paper reports results only on LLaMA-3-70B-Instruct and Qwen2-72B-Instruct. Including evaluations on additional models—such as using LLaMA-3-8B-Instruct to accelerate Vicuna-33B, LLaMA-2-70B, or Qwen2-72B-Instruct—would greatly enhance the paper's value.\n\n3. Key baseline comparisons are missing. While the paper notes that integrating a draft model into a distributed system is non-trivial, tensor-parallel (TP) inference for the draft model is theoretically feasible. Even though Table 6 shows limited speedup for TP > 1, including a direct comparison between a TP-enabled draft model and EasySpec would improve the experimental rigor.\n\n4. Further issues are discussed in the \"Questions\" section.\n\nOverall, despite these concerns, I find the paper refreshing and creative. My current score falls between borderline reject and borderline accept, but I am open to raising it following constructive author feedback.\n\nQUESTIONS:\nThe minor concerns are as follows:\n\n1. In Figure 1(b), EasySpec appears to achieve balanced GPU utilization. However, my understanding is that only the attention layers are distributed across GPUs, while the MLP layers remain sequentially executed on a single GPU — is this correct? Additionally, the attention layers do not seem to be evenly distributed across GPUs. For example, in Table 4 (N=4), Strategy=0 uses only one GPU while the other three are idle, and Strategies=1–3 utilize only three GPUs, leaving one unused. This seems inconsistent with the balanced load shown in Figure 1(b), which is somewhat confusing.\n\n2. Table 2 shows that EasySpec outperforms EAGLE-2 in throughput. Is EAGLE-2 using tensor-parallel (TP) decoding in this comparison? Also, are both methods evaluated under the same hardware conditions?\n\n3. During the drafting stage, given that multiple GPUs are available, is it feasible to use multiple distinct draft models simultaneously to generate draft candidates? For example, could both LLaMA-3-8B and Qwen2-7B be used in parallel to accelerate LLaMA-3-70B inference?\n\n**Suggestions**\n\nThere are many concurrent works on speculative decoding (e.g., [1–8]); please expand the Related Work section to include more of these, which will help readers better understand the development of this field and EasySpec's unique contributions.\n\n[1] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding\n\n[2] SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices​\n\n[3] Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding\n\n[4] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding\n\n[5] EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding\n\n[6] A Theoretical Perspective for Speculative Decoding Algorithm\n\n[7] BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models\n\n[8] OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConventional speculative decoding has low multi-gpu utilization when the draft model is small. The paper considered the sequential multi-layer inference pipeline as the main bottleneck and proposed a “fuzzy” speculation method, EasySpec, that runs multiple model layers in parallel. To avoid accumulated KV cache errors and potential performance drop, EasySpec introduced Bonus Calibration that recomputed the KV cache using the accepted tokens after each draft-verification cycle. The performance of EasySpec has been evaluated across different LLMs, e.g. LlaMA-3-70B (8B), Qwen2-72B(7B), etc, and different benchmarks, e.g. MMLU, Spec-Bench, etc. Experiments showed that EasySpec achieved lower inference latency compared to several baseline approaches, e.g. TP, tree-attention, etc.\n\nSTRENGTHS AND WEAKNESSES:\n---\nStrengths\n\nMethod: EasySpec proposed to “break” the sequential dependency of the hidden states across consecutive blocks/layers to a certain extent, which is technically sound. Prior research in LLMs has indicated that the hidden states in consecutive layers may exhibit limited changes in several scenarios, given the use of skip connections [1]\n\nExperiment: the paper provided extensive LATENCY evaluations on auto-regressive inference, including comparisons to alternative approaches on various benchmarks.\n\n---\n\nWeaknesses\n\nAccuracy: it was indicated in the paper that EasySpec may lead to 7% accuracy drop, but no further discussions were included in the paper. This is a main concern. From the paper, EasySpec is not lossless. While EasySpec demonstrated superior throughput, it is not clear to the reader if EasySpec achieved at least comparable accuracies to the baselines. The authors are encouraged to include accuracy comparisons to the baselines across different benchmarks, similar to the latency evaluations. Ablations on the tradeoff between the speedup and the accuracy drop are also highly recommended, as 7% accuracy drop is high.\n\n\nBaselines: the reader may expect at least one representative TP baseline. However, from L192, it looks like the baseline drafter did not use TP (i.e. TP size == 1). From Table 6, the optimal TP size was determined on MMLU. The reviewer has two questions: i) since MMLU involves only classification tasks that requires no autoregressive generation, the reviewer wondered if the optimal TP sized was determined under the autoregressive generation setting or prefilling setting? ii) Any chance the optimal TP size varies across benchmarks?  \n\nPresentation: several figures/tables were referred in the main paper but presented in the appendix, e.g. Figure4, Table 6. The authors are encouraged to reposition the figures and tables\n\n\n[1] The Unreasonable Ineffectiveness of the Deeper Layers. ICLR 2025.\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nLimitations were included in the Appendix", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EasySpec, a layer-parallel speculative decoding method designed to accelerate the drafting stage in large language model inference. In conventional speculative decoding, the draft model is typically small and cannot fully utilize multi-GPU resources. EasySpec addresses this by breaking the sequential dependency between attention layers in the draft model, allowing them to run in parallel across multiple GPUs. To mitigate the KV cache errors introduced by this fuzzy execution, it introduces bonus calibration. Experiments are conducted on several large models and tasks, measuring drafting and verification time, overall speedup, and acceptance rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The paper focuses on a practical performance bottleneck in current large language model inference pipelines—the under-utilization of the drafting stage in multi-GPU settings, making the topic of practical importance.\n+ The paper is well-organized and easy to follow. Background concepts such as speculative decoding and tensor parallelism are explained with sufficient clarity. The method is described clearly, aided by formulas, algorithm pseudocode, and illustrations.\n\nWeaknesses:\n\n- While the paper identifies the drafting stage as a performance bottleneck, the motivation could be communicated more compellingly. The Introduction currently lacks concrete system-level profiling, visual illustrations, or quantitative comparisons that would help emphasize the severity and relevance of the problem.\n- Table 3 presents cosine similarities of intermediate hidden states to support the approximation quality of fuzzy speculation. It would be helpful to include output-level comparisons (e.g., token distributions or generation divergence) to complement the current analysis.\n- Figure 1 illustrates the core idea of layer-parallel attention execution but does not provide a full depiction of the entire EasySpec workflow. In particular, bonus calibration and KV cache correction are critical components of the method and would benefit from visual inclusion in a complete process diagram.\n\nQUESTIONS:\n1. While bonus calibration is described as lightweight, is there any scenario in which the cost of re-running the draft model outweighs the benefits of layer-parallelism? Some further clarification would be useful.\n2. In fuzzy speculation, the attention layers are executed in parallel using the same input, which generates a set of KV caches. It remains somewhat unclear whether these fuzzy KV caches are meaningfully used in subsequent token generation before being replaced.\n3. EasySpec shows strong performance with medium-sized draft models. However, could it extend to significantly smaller drafters?\n4. PEARL [1] also focuses on accelerating the drafting stage and overlaps the drafting and verification stages to achieve stage-level parallelism. It would be helpful if the key differences between the two approaches could be clarified. \n\n[1] Liu, T., Li, Y., Lv, Q., Liu, K., Zhu, J., Hu, W., & Sun, X. (2025). Pearl: Parallel speculative decoding with adaptive draft length. In The Thirteenth International Conference on Learning Representations.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a more parallelized speculative decoding method, made by modifying draft models to run multiple layers in parallel. They demonstrate that this leads to a real improvement in distributed systems compared to distributed normal specdec. They compare against competitive baselines like Eagle 2, with detailed benchmarking of the time spent in drafting and verification.

STRENGTHS AND WEAKNESSES:
Strengths:

Their method is interesting. Its a essentially "free" modification of existing inference pipelines to enable faster decoding. There are definite applications in cases where you do not want to fine tune an additional model. Furthermore, it addresses a real problem with draft model based approaches to speculative decoding, which is their poor utilization of hardware.

Neutral:

There is an existing literature in parallelized LLMs (see, for example, ladder transformers and diffusion LLMs)

Weaknesses:
I do think that the Llama 3.3 comparison against Eagle 2 listed here is unfair. Eagle 2 achieves better performance than EasySpec given a draft model in the same family and trained heads, as shown in the Llama 3 comparison. When adapting to llama 3.3, the performance of easyspec is better, but Eagle 2's draft heads are much easier to train than an entire smaller draft model. 

Furthermore, the cost of this type of method is that two different KV caches need to be maintained, which increases memory usage.

QUESTIONS:
How costly is training new draft heads for Eagle 2? Could you generate draft heads for Llama 3.3, and compare? 

How well do distilled models perform as draft models in this use case? Would it be possible to use one of the Llama models trained with distillation to see if it gives superior draft performance?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03