PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation
Authors: Zihao An
Huajun Bai
Ziqiong Liu
Dong Li
Emad Barsoum
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Speculative Decoding
AI Infra
Low Cost Training
Primary area: generative models

Abstract: The autoregressive nature of large language models (LLMs) fundamentally limits inference speed, as each forward pass generates only a single token and is often bottlenecked by memory bandwidth. Speculative decoding has emerged as a promising solution, adopting a draft-then-verify strategy to accelerate token generation. While the EAGLE series achieves strong acceleration, its requirement of training a separate draft head for each target model introduces substantial adaptation costs. In this work, we propose **PARD (PARallel Draft)**, a novel speculative decoding method featuring target-independence and parallel token prediction. Specifically, PARD enables a single draft model to be applied across an entire family of target models without requiring separate training for each variant, thereby minimizing adaptation costs. Meanwhile, PARD substantially accelerates inference by predicting multiple future tokens within a single forward pass of the draft phase. To further reduce the training adaptation cost of PARD, we propose a COnditional Drop-token (COD) mechanism based on the integrity of prefix key-value states, enabling autoregressive draft models to be adapted into parallel draft models at low-cost.  Our experiments show that the proposed COD method improves draft model training efficiency by 3x compared with traditional masked prediction training. On the vLLM inference framework, PARD achieves up to 3.67x speedup on LLaMA3.1-8B, reaching 264.88 tokens per second, which is 1.15x faster than EAGLE-3. Our code is available at https://github.com/AMD-AGI/PARD.

Published as a conference paper at ICLR 2026
PARD: ACCELERATING LLM INFERENCE WITH Low-COST PARALLEL DRAFT MODEL ADAPTATION
Zihao An
Huajun Bai
Ziqiong Liu' Tsinghua University @ amm..com
Dong Li'
Emad Barsoum'
1 Advanced Micro Devices, Inc.
An, Huajun.Bai, Ziqiong.Liu, d.li,
ABSTRACT
The autoregressive nature of large language models (LLMs) fundamentally limits inference speed, each forward pass generates only single token and is often bot- tlenecked by memory bandwidth. Speculative decoding has emerged as a promising solution, adopting draft-then- then-verify strategy to accelerate token generation. While the EAGLE series achieves strong acceleration, its requirement of training a sepa- rate draft head for each target model introduces substantial adaptation costs. In this work, we propose PARD Rallel Draft), a novel speculative decoding method featuring larget-independenc and parallel token prediction. Specifically, PARD enables a single draft model to be applied across an entire family of target models without requiring separate training for each variant, thereby minimizing adaptation costs. Mcanwhile, PARD substantially accelerates inference by predicting multiple future tokens within a single forward pass of the draft phase. To further reduce the training adaptation cost of PARD, we propose COnditional Drop-token (COD) mechanism based on the integrity of prefix key-value states, enabling autoregressive draft models to be adapted into parallel draft models at low-cost. Our experiments show that the proposed COD method improves draft model training efficiency by 3x compared with traditional masked prediction training. On the vLLM inference framework, PARD achieves up to 3.67 x speedup on LLaMA3.1 reaching 264.88 tokens per second, which is 1.15x x faster than EAGLE-3. Our code is available at
com/ AMD AGT/PARD
INTRODUCTION
Rapid advancements in LLMs such as GPT-4 (OpenAl 2023). ) LLaMA3 (Llama Team 2024 and DeepSeek-RI (DeepSeek-AI et al. 2025) have fueled an explosion of applications such as content generation, code generation, and AI agents. However, as the counts of model parameters and the lengths of the context continue to grow, inference efficiency has become critical challenge. Due to the auto-regressive (AR) nature of LLMs, tokens are generated sequentially, leading to substantial memory bandwidth consumption and high inference latency. Speculative Decoding (SD) (Chen et al. 2023a Leviathan et al. 2023) has emerged as a promising technique to mitigate bandwidth overhead and reduce decoding latency during LLM inference. The core idea is to a lightweight draft model to predict multiple candidate tokens, which are then verified in parallel by the target model. When combined with speculative sampling, this approach allows the model to generate multiple tokens within a single forward pass, significantly improving efficiency without compromising output quality.
The effectiveness of SD is determined jointly by the accuracy of the draft model and its overhead. To improve draft model accuracy, various methods have been proposed, primarily falling into the category of target-dependen approaches, where the draft model leverages information from the target model. For example, Medusa (Cai et al.) 2024a) and EAGLE (Li et al. 2024a) incorporate features from the target model's 's outputs into the draft model's input, while LayerSkip (Elhoushi et al.) 2024) and Kangaroo (Liu et al.) (024b) reuse selected layers of the target model as the draft model. Although these methods improve token prediction accuracy, they also introduce major drawback: the draft model becomes tightly coupled with the target model. Consequently, any target model requires a separately trained draft model, which highly increases adaptation and deployment costs.
"qqual contribution.

---

Published as a conference paper at ICLR 2026
EAGLE E/GLE3
0.34
VSD
VED
PARD
0.9
BABE 0.0G
Rato
PARD
2x1x faster
0.00
OUES
PARD
PARO
0.827
Time
0.8
0,7C
(b)
Target Independence
Training Cost
Draft model Intency
Draft Aec
Vanilla SD
True
Free
High
High
0.75
False
High
Medium
Medium
EAGLE-3
Falsc
High
Medium
High
gemBk
PARD
True
Low
Low
Very High
(a)
(c)
Figure 1: PARD achieves low latency while maintaining high accuracy. (a) Comparison of first-token acceptance rates using LLaMA3.1 1-BB as the target model. EAGLE and EAGLE-3 use their official model, vanilla speculative decoding (VSD) employs LLLMMA32-11 1B as the draft model, and PARD) represents the adapted version of VSD. (b) Comparison of actual inference time between VSD and PARD. VSD generates candidate tokens autoregressivell during the draft stage, requiring multiple forward passes. In contrast, PARD completes drafting with a single forward pass. The draft model used is LLaMA3.2-1B and the target model is LLaMA3.1-8B (c) Illustrative comparison of training and inference efficiency between PARD and other methods.
Unlike target-dependent methods, vanilla SD (Chen et al., 2023a Leviathan al. 2023 represents class of larget-independen approaches, where single draft model can be applied across an entire family of target models without requiring separate training for each variant. For validation, compare using LLaMA3. 3.2-1B (Llama Team 2024) as draft model against the EAGLE method, both for the target model LLaMA3.1-801 Figure La shows that LLaMA3.2-IB achieves significantly higher accuracy than the EAGLE head. However, due to its increased computational cost, the overall speedup ratio of vanilla SD may be lower than that of EAGLE.
To further accelerate inference, Mask-Predict (Ghazvininejad et al. 2019) provides a form of parallel decoding by using masked tokens as placeholders and employing specialized training to enable multi- ple token predictions in single forward pass. Several recent studies have integrated SD with mask prediction. For example, PaSS (Monea et al., 2023) and BiTA (Gloeckle et al. (024) fine-tune the target model into parallel decoding model that functions as draft model, while ParallelSpec (Xiao)
ct al. 2024) extends EAGLE and Medusa into a parallel decoding framework. However, these approaches still use target model information, making them inherently target-dependett.
The goal of this paper is to break away from the target-dependent design paradigm, by deveploping a target- indeeendend SD approach called Parallel Draft (PARD) with stronger generalization, higher inference efficiency, and lower adaptation cost. PARD builds upon existing high-accuracy small language models. A single draft model with low training cost can be applied across entire family of target models, thereby significantly reducing adaptation overhead. PARD introduces mask tokens for parallel token predictions in the draft phase accelerate inference, shown in Figure lb] To further improve training efficiency, we design conditional drop-token (COD) mechanism based on the integrity of prefix key-value states, reducing the cost of adapting autoregressive draft models into parallel draft models by factor of 3. The advantages of PARD are shown in Figure Ic
To align our evaluation with real-world scenarios, all experiments are conducted on the widely adopted industrial inference framework vLLM. We also find that the inference performance of the Transformers library not highly efficient, and thus apply corresponding software optimizations. Extensive experiments on LLaMA3 and Qwen familys show the consistent speedup of PARD over vanilla SD and EAGLE-3, as shown in Figure
To summarize, we make the following contributions:
We propose PARD, a novel speculative decoding method featuring target- ungee--ineeeendece and parallel token prediction. PARD is highly generalizable: its target-independen design allows
2

---

Published as a conference paper at ICLR 2026
Elagled
300
200
130 2:30
002 Pd
Q2.5 25
Q2.5 75
QU.5 14B
BS
L331 5B
1333 POB
Figure 2: Performance comparison of different methods on the HumanEval task under vLLM. AR denotes the regresseeee baseline, and VSD denotes vanilla speculative decoding, where the draft models used are LLaMA3.2-1B and Qwen2.5-0.5B.
single draft model to accelerate entire family of target models, in contrast to target- dependent methods such as Medusa and EAGLE. This significantly reduces deployment and adaptation costs. Through mask tokens, PARD substantially accelerates inference by predicting multiple future tokens within a single forward pass the draft phase.
We propose COnditional Drop-token (COD) strategy for the effient training of PARD. Leveraging the integrity of prefix key-value states, COD enables low-cost adaptation of autoregressive draft models into parallel ones, boosting training efficiency by up 3x while maintaining accuracy.
We integrate PARD into the high-performance inference framework vLL.M. On LLaMA3.1- 8B, PARD achieves 3.67 speedup, reaching state-of-the-art throughput of 264.88 tokens per second on A100-40GB GPU, which 1.72x faster than vanilla SD and 1.15x faster than EAGLE-3.
PRELIMINARIES
2.1 AUTO-REGRESSIVE NATURE OF LLMS
Modern LLMs are based on the GPT architecture (Radford et al. 2018). During the training phase, GPT models leverage highly efficient parallelization, allowing tokens within sequence to be processed simultaneously. This parallelism enables GPUs to fully utilize computational resources by maximizing matrix multiplications and optimizing memory bandwidth usage, thereby improving training efficiency. Given an input sequence X = (Io, ...... xN-11 1) and its corresponding target sequence (01, 22, .NN) the training objective of GPT is minimize the auto-regressive loss, which can be expressed as:
N C C= E log P P(r. It-11)
(I)
However, during inference, due to its auto-regressive nature, GPT must generate tokens sequentially, with each token depending on all previously generated tokens. This sequential generation process leads to significant drop computational efficiency. At each step the model computes:
Ft = arg max Pl.,
(2)
This sequential generation results in memory-bound scenario, where GPU performance is con- strained by the repeated loading of model weights and the KV-cache, rather than being fully utilized for large-scale parallel computations. As result, even though GPUs capable of high-speed matrix multiplications, the decoding phase remains bottlenecked by memory access latency, leading to high latency during the decoding phase.

---

Published as a conference paper at ICLR 2026
Target Model
Target Model
story
about
large
larguage
story
about
large
langwage
AR Draft Model
ParaDraft Model
story
about
large
tell
musk
mask
AR Draft
PARD
Figure 3: Illustration of PARD Inference. Left: Vanilla speculative decoding involves a draft model ruto-regresyvvely generating candidate tokens, which are then validated by the target model in parallel. Right: PARD introduces mask tokens for parallel Draft. All K candidate tokens are generated in one forward pass.
2.2 SPECULATIVE DECODING
SD effectively reduce decoding latency and improve matrix multiplication efficiency. The idea behind SD is to first use smaller draft model to generate set of candidate tokens, and then use the target model verify the candidates. The process
follows:
The goal is to generate the next K candidate tokens C= -z',........ 2 K These candidates can be produced by lightweight model or traditional machine learning algorithms. For the vanilla SD method, the draft model generates the K candidates auto-regressively.
Drafting Stage:
Verification Stage: The target model then verifies the candidate tokens in parallel, improving compu- tational efficiency. When combined with speculative sampling. SD ensures no loss of performance.
We define Tn as the time taken by the draft model for single forward pass, and Tr the time taken by the target model for a single forward pass. The input length to both the draft and target models can vary across different SD methods. However, when the input length is not significantly large, the change in speed is negligible. Therefore, the time taken per iteration for predicting K tokens by vanilla SD is:
TARa K TD TT-
(3)
As shown in Figure lb when using a high acceptance rate model like LLaMA3.2-IB to accelerate LLaMA3.2-8B. the draft model consumes considerable amount of time. Our PARD method fine-tunes the draft model into a parallel decoding model. this case, the time taken is:
TPARD TD + TT.
can be seen that PARD reduces the total time the draft model to 1/K of the original time, thus significantly reducing the overall decoding time.
PARD FRAMEWORK
We begin by introducing how the PARD model predicts multiple tokens in a single forward pass and provide detailed explanation of inference process in Section 3.1 Next in Section 3.2 we describe how to finetune vanilla draft model to acquire the capabilities of PARD while maitaining its target-independent feature. Moreover, we propose conditional drop training token method that significantly accelerates training.
3.1 PARD INFERENCE
We incorporate the inference method Mask-Predich (Ghazvininejad et al. 2019) into PARD. The objective of the draft model is to predict the next K tokens by optimizing the probability distribution:
P(In.
Int -
Ira, ...-.-.. Fn;
(5)
4

---

Published as a conference paper at ICLR 2026
mod
(a)
(d)
(b)
Figure 4: Illustration of Conditional Drop in PARD training. (a) Training data of the standard AR model. (b) Training data of PARD. The diagram divided into three sections by dashed lines. corresponding to training objectives for predicting tokens at positions +1, +2, and +3. The designed attention mask ensures consistency between training and inference. Labels lighter font indicate tokens that are supplemented for context completion and do not contribute to the loss computation. (c) Sparse training data for PARD with Conditional Drop, where shaded areas represent dropped tokens. The retention pattern follows geometric decay with fraction
0.5 of positions retained for mask token my and 22 = 0.25 for m,, ensuring that each retained token maintains complete preceding key-value pairs. (d) The sparse matrix reorganized into compact format by eliminating dropped positions.
For the vanilla SD method, this probability can be factorized as:
K-1
P( In+K - .o.............................. 1:; ..00-....,..,... 1 AlRran)) = II k=0
P(rn+elro...... In+k " AARaran)
(6)
This formulation follows the standard autoregressive approach in Figure where each token is generated sequentially based all previously predicted tokens.
In contrast, PARD introduces a special token my as a placeholder to replace tokens that would otherwise create dependencies. The formulation is follows:
P(Zny..., Fo. ................................ In OPARd)
=P(xn Io,..., 15 15
II k=1
P(rn4k .0....,, 1100 71-13 OPARD).
As shown in the equation, the predictions each step do not depend each other, enabling fully parallel inference. This reduces the number of forward passes required by the draft model from to significantly improving inference efficiency.
3.2 PARD TRAINING
To enable vanilla draft model to predict multiple tokens within single forward pass, adopt a mask-token training strategy, which adapts a standard autoregressive draft model into a parallel draft model. Unlike conventional approaches, we further propose conditional drop-token (COD) mechanism based on the integrity prefix key-value states, which significantly improves training efficiency while maintaining final performance.
3.2.1
MASK TOKENS BASED TRAINING
To ensure consistency between the training and inference processes, divide training into multiple independent subtasks, where each subtask designed to predict .a..... K-+k-1- As shown in Figure 4a 4b these subtasks can be trained simultaneously with minimal preprocessing of the training
5

---

Published as a conference paper at ICLR 2026
data. The loss for each subtask is computed using cross-entropy function, follows:
zo, F0,411,-0 F1,...... 1:-1; OPARD)) log P(a:
A: 1,
Lk
(8)
log P(x; | zo, Tb.... ki ma, mz OPARD), k > 1.
N
k
t=
where La represents the loss function for predicting the k-th next token, and N denotes the length of the sample.
3.2.2 CONDITIONAL DROP OF TOKENS
Compared to standard auto-regressive model training, the mask token training method breaks the task into multiple subtasks, significantly increasing the training cost. For example, if the training sample length is N, then the number of tokens for AR training is also N. However, in the mask token training approach, the number of tokens for training increases from N to K x N, where K is the number tokens the draft model predicts simultaneously in single forward pass.
To address this challenge, propose innovative COnditional Drop token (COD) strategy. This method boosts training efficiency significantly while maintaining prediction accuracy. The core idea behind COD is that tokens in earlier subtasks are more critical, whereas those in later subtasks can be selectively dropped to reduce computation.
During the token dropping process, randomly dropped tokens can result in incomplete key and value states during attention computation. The COD mechanism ensures that the remaining tokens still provide complete key and value information when computing attention, thereby preserving essential contextual representations. Figure illustrates the token dropping process. Although some tokens are removed, the key contextual information remains intact. Figure shows the final reorganized data after token dropping.
To manage the number tokens retained each subtask, we introduce a retention parameter r. For the i-th subtask, the number of retained tokens N; is given by:
N = N -
(9)
Consequently, the total number of training tokens can be expressed as:
Σ N N K
N
NCOD Σ
(10)
For instance, when 0.5, the number of training tokens can be reduced from K N to 2N. significantly lowering the training cost. Additionally, to prevent excessive token dropping in later subtasks, we introduce a minimum retention ratio Tmins ensuring that the retention rate does not fall below predefined threshold. The adjusted number of retained tokens for each subtask is:
N N max(r"-1 T'iia)
(11)
The detailed algorithm of COD, along with the mechanism integrity of prefix key-value states provided in Appendix B By employing the COD strategy, the training cost of PARD can be significantly reduced from O (N- K) to O(N).
3.2.3 TRAINING EFFICIENCY
We evaluate the training cost of different speculative decoding methods in Pflops per 1M tokens, using LLaMA3.3-70B as the target model. PARD achieves a training efficiency that 7 x higher than EAGLE and 10x higher than EAGLE-3. Full derivations of forward, backward, and total training costs for each method are provided in Appendix
In addition efficiency, PARD exhibits target independence, allowing single draft model to accelerate an entire series of target models (c.g.. LLaMA3-8B. 70B, 40SB). contrast, EAGLE-style methods require separate draft model for each target, which significantly increases adaptation cost.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison different methods on the Qwen and LLaMA3 series under the vLLM framework. The EAGLE series uses the official model.
Target
Method
IumannEvll
GSM8K
SpecBench
Averuge
TPS
SpeedUp
TPS
SpeedUp
TPS
SpeedUp
TPS
SpeedUp
Q2
AR
76.34
1.00
76.12
1.00
75.97
1.00
76.14
1.00
VSD
181.18
2.37
201.15
2.64
112.00
1.47
164.78
2.16
PARD
268.98
3.52
221.43
2.91
135.05
1.78
208.49
2.74
Q2.5 3B
AR
130.46
1,00
130.42
1.00
129.95
1.00
130.28
1.00
VSD
231.65
1.78
238.71
1.83
136.26
1.05
202.21
1.55
PARD
425.96
3.27
406.17
3.11
203.05
1.56
345.06
2.65
Q2.5
AR
76.07
1.00
75.93
1.00
75.97
1.00
75.99
1.00
VSD
178.97
2.35
162.85
2.14
107.38
1.41
149.73
1.97
PARD
285.82
3.76
292.56
3.85
145.64
1.92
241.34
3.18
Q2.5 14B
AR
40.78
1.00
40,92
1.00
40.78
1.00
40.83
1.00
VSD
135.88
3.33
146.69
3.58
80.59
1.98
121.05
2.97
PARD
181.19
182.25
4.45
91.33
2.24
151.59
3.71
L3 8B
AR
73.68
1.00
73.57
1.00
72.80
1.00
73.35
1.00
VSD>
144.59
1.96
123.76
1.68
102.76
1.41
123.70
1.69
PARD
235.84
3,20
200.45
2.72
147.80
2.03
194,70
2.65
1331 8B
AR
73.07
1.00
73.38
1.00
72.70
1.00
73.05
1.00
VSD
155.47
2.13
140.93
1.92
106.44
1.46
134.28
1.84
EAGLE
136.05
1.86
110.81
1.51
100.11
1.38
115.66
1.58
EAGLE33
233.43
3.19
192.56
2.62
155.58
2.14
193.86
2.65
PARD
264.88
3.63
235.09
3.20
157.49
2.17
219.15
3.00
L3.3 70B
AR
24.31
1.00
24.27
1.00
23.95
1.00
24.18
1.00
VSD
80.78
3.32
77.07
3.18
53.09
2.22
70.31
2.91
EAGLE-3
95.17
3.91
75.34
3.10
63.14
2.64
77.88
3.22
PARD
99.88
4.11
95.58
3.94
$7.41
2.40
84.29
3.49
EVALUATION
4.1 EXPERIMENTAL SETUP
Models:
We conduct experiments on popular industry models, including LLaMA3 (Grattafiori et al.)
2024), PeepSeek-RIQWeee (Guo et al. 2025). and Qwen (Yang et al.) 2024). For each model series, we select the smallest model variant and train it in the PARD framework.
Datasets: To ensure alignment with the original instruct model training process, we select datasets tailored to each model series. LLaMA3 is trained with Magpie- Magpie-Llama-3.. -l----almm--111-- 1--ma--.--Pr-I111 1M (Xu et al. 2024) and Evol-CodeAlpaca (Luo al. 2023). Qwen2.5 is trained with Magpie-Qwen2- Qwen2-wenrrr Code/lpaca. DeepSeekRR
and Evol- 2025). OpenThoughts--
trained with OpenR1 OpenR1-Math-2220 --Mtth-220k (Face
1 114k [Team] (025)), and Chinese- Chinese-DeepSeekk -i-ecc--R-Disttl---eeee 110k (Liu et al, 2025 For LLaMA
and Qwen, we further enhance training accuracy by regenerating answers.
Tasks: We evaluate the effectiveness PARD on mathematical reasoning and code generation tasks benchmarks including HumanEval (Chen al.] 2021). GSM8K (Cobbe al. 2021) MATH500 (Lightman et al. 2023) and SpecBench (Xia et al. 2024b).
Training: We conduct training on 8xMI250X cards using the TRL framework for total of 4 epochs. During training, the parameters are set as follows: A = 8, r= 0.7, and training hyperparameters are provided in Appendix
Tmin = 0.2. The detailed
Evaluation: To better reflect real-world usage scenarios, all comparative experiments are conducted on the perf-paeformance vLLM framework, and the evaluation performed on A100-40GB GPU.
Metries: Tokens Per Second: The number of tokens generated per second in real-world scenarios. Speedup: The acceleration ratio compared the baseline standard auto-regressive generation method.
4.2 EXPERIMENTAL RESULTS
Table compares the acceleration effects PARD on the Qwen and LLaMA3 series, while Ap- pendix further reports results the
series. On code tasks, PARD achieves
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of acceptance rates for PARD and EAGLE on 1LLMM33.-8B. where k-a denotes the average acceptance rate when the draft length is k.
Table Memory bandwidth usage during the draft phase of LLaMA3. 1-8B model in 6 dtype. PARD bandwidth usage remains constant as k increases.
HvmanEval
GSM8K
k=
A:
Method
Method
1-o
4-o
1-o
4-a
Draft BW Consumption
EAGLE
0.83
0.72
0.79
0.66
EAGLE EAGLE-3
5.94 GB
8.90 GB
11.88 GB
EAGLE-3
0.87
0.85
0.82
0.79
5,94 GB
8,90 GB
11.88 GB
PARD
0.93
0.90
0.88
0.85
PARD
2.48
2.48
2.48 GB
speedups ranging from 3.20 to 4.44x, with average speedups between 2.65 and x. Notably, LLaMA3.1-81 runs 1.9x faster than EAGLE and 1.15x faster than EAGLE-3.
Our experiments further demonstrate the target-independence property of PARD, where a single PARD model can accelcrate an entire series of target models, as shown in Table and Figure 2 Specifically, we evaluate three target models from the LLaMA3 series and three target models from Qwen. In contrast, target-dependent methods such as the EAGLE series require separate training for each individual model. PARD achieves high acceleration without the need for model-specific adaptation, substantially lowering the barrier for deployment.
During our experiments, we observed that inference throughput using the Transformers library is significantly lower than that of vL.LM Appendix For methods such as vanilla SPD, this suboptimal performance of Transformers can lead to relative speedups under the same inference framework being lower than those measured on vLLM (c.g., 1.36x and 2.26x). To better reflect real-world usage scenarios, all final experiments in this paper are conducted on the vLLM framework.
4.3 ABLATION STUDIES
For all ablation study experiments, the target model used is DeepSeck-RI-Qwen-- and the draft model's pretraining model is -1-Seek--11Qwen 1.5B. Training is conducted 93K subset of OpenRI 1-ath Math-220K for one epoch, and testing is performed using the MATH500 dataset.
The lower the retention rate, the greater the acceleration effect. However, excessively low retention may degrade model performance. As shown in 5a when setting = 0.7 and m'nn 0.2, achieve good balance between speed and accuracy. This setting allows to achieve 3x faster training while maintaining the original accuracy. All experiments in this paper adopt these parameters.
Conditional Drop Token:
Shared Mask Token ID Strategy: We compare different prediction strategies and find that using the same mask token ID across all predicted positions, i.e., my m1
m7 K-111 performs better than using distinct token IDs. The corresponding throughput results are 221.97 and 218.05 tokens/s, respectively. This approach not only improves prediction consistency but also enhances the model's ability to generalize beyond its training configuration, property we refer to as extrapolation capability. Specifically, extrapolation capability allows the model to infer with a larger during inference than it was trained on.
Selection of Draft K: We conduct a Cartesian product test for Kunin during training and Kinter during inference in Figure 5b Due to the extrapolation capability of PARD enabled by the shared mask token ID, Kinter can be greater than Kurin- The best performance is achieved at Kinter = 12, while 8.
results remain stable when Kuain
8. Therefore, select Kurin
For SD, higher acceptance ratio combined with lower bandwidth consumption results in superior speedup. In Table and Table we compare PARD and EAGLE series. PARD achieves a higher acceptance ratio while consuming less bandwidth.
Comparison with the Mainstream Method EAGLE:
Large Batch Size Inference: Appendix D reports results with batch sizes ranging from 1 to 16. As the batch size increases, the bottleneck shifts from memory-bound to compute-bound. bound. In this setting, PARD achieves speedups between 1.33 3.63x.
8

---

Published as a conference paper at ICLR 2026
220
218.03
81
217.93e
230
PARI
210
214.80
223
221
216
211.74
Tokenoos
200
PARD 1.1
PARD
0.7_0
PARD 0.5 0.2
200
0.500
184:184; 188
188.11
PARD 0.2.22
PARb_a220
100
196
10
12
15
Training Hours
Kuje
(a)
(b)
Figure 5: (a) Compare the effects of different values of r and Tmins where each experiment is labeled
PARD Tmin- The x-axis represents training time, while the y-axis indicates the final decoding
speed. (b) Present the results under different Kuain and Kinter settings. The x-axis represents Kinter and the experiment names PARD_ K denote different Kuain values.
5 RELATED WORK
Improving the inference efficiency of large language has been extensively studied from multiple perspectives. Quantization techniques such as GPTQ (Frantar et al. 2022), AWQ (Lin et al. 2024). SmoothQuant (Xiao et al. 2023a). and ---QAA (Liu et al. 2023) focus on reducing computational
and memory costs. To address long-context K KV-cache management, approaches such as GQA (Ainslic) cct al. 2023), MLA (Liu et al., 2024a), StreamingLLM (Xiao et al. 2023b), H2O (Zhang et al. 2023).
MoBA (Lu al. 2025), and NSA (Yuan et al. 2025) explicitly balances GPU memory consumption against model accuracy. At the system level, innovations like FlashAttention (Dao et al. 2022)..
FlashDecoding (Hong al.) 2023), MEGABLOCKS (Gale al. 2023). and vLLM (Kwon et al.
2023) deliver optimized kernels and scheduling strategies to maximize hardware utilization and throughput.
Speculative decoding (Leviathan et al., 2023 Chen et al. 2023b) improve GPU parallelism by leveraging draft model generate candidate tokens, which then verified by the target model, achieving speedup without compromising accuracy. Other approaches such as LOOKAHEAD (Fu
2024). PLD+ (Somasundaram et 2024), REST (He al., 2023) and Suffixeecoding (Oliaro al.
et al.
2024 utilize text-based retrieval mechanisms to generate more informed drafts. LayerSkip
houshi et al.
2024). Kangaroo (Liu al. 2024b), and SWIFT (Xia al. (0224) reuse selected layers
of the target model to construct a lightweight draft model. CSDrafting (Chen et al. 2024b) employs
cascade speculative draft model. CL.MMs (Kou al. 2024) and NSN (Wang al. 2024 adopt
Jacobi decoding.
To improve the accuracy of speculative decoding, methods like Medusa (Cai al. 2024b). EAGLE
al. 2024b), EAGLE-3 (Li et al. 2025), Amphista (Li et al. 2024c) and Hydra (Ankner et al. 2024)
incorporate representations from the target model as additional input signals. Approaches such as
BiTA (Gloeckle et al. 2024), ParallelSpec (Xiao et al. [2024), and PaSS (Monea et al. 2023) introduce mask tokens to enable parallel speculative decoding. Further, techniques including Spectr (Sun et al. 2023). SpecInfer (Miao et al. (2023). Sequoia (Chen et 2024a), and EAGLE-2 (Li et al. 2024b)
optimize tree-hased verification structures to enhance token acceptance rates.
CONCLUSION
We presented PARD, a novel speculative decoding method that is target- carget-independett and supports parallel token prediction. Unlike existing target-dependent approaches such Medusa and EAGLE,

---

Published as a conference paper at ICLR 2026
PARD allows single draft model to accelerate entire family of target models, significantly reduc- ing adaptation and deployment costs. To improve training efficiency, we proposed the Conditional Drop-token (COD) mechanism, which leverages the integrity of prefix key-value states to adapt autoregressive draft models into parallel ones at fraction of the cost. Extensive experiments on ILaMA3 and Qwen families show that PARD consistently outperforms vanilla speculative decod- ing and EAGLE-based methods. On LLaMA3. 1-BB, PARD achieves 264.88 tokens per second, corresponding a 3.67 speedup over standard autoregressive inference and 1.15x speedup over EAGLE-3. In summary, PARD offers highly generalizable, efficient, and practical framework for accelerating large language model inference, demonstrating the potential of target ingget-indeeenentt and parallel decoding strategies for scalable LLM deployment.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. It does not involve human subjects, sensitive data, or applications with foreseeable harmful impact. All datasets and methods are used in compliance with ethical and legal standards.
REPRODUCIBILITY STATEMENT
The experimental hyperparameters are detailed in Appendix C All reported results are reproducible, and both code and models will be released.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Ainslie, James Lee- Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arX.v:2305. 2023.
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-deeenddnt draft heads for medusa decoding. arXiv preprint ar X0v:2240..05109 2024.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint 2tt1::4001.107744 2024a. to appear.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024b.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Saan-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.c 30..0.11.1,,,.. 2023a.
01311 c1318 DeepMind
Technical Report.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337 2021.
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-awan speculative decoding. arXiv preprint 2xXi::2402.12374. 2024a.
Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chang, and Jie Huang. Cascade speculative drafting for even faster Ilm inference. Advances in Neural Information Processing Systems, 37:86226-86243 2024b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 16344-16359 2022.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li. H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou. Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu. Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng
11

---

Published as a conference paper at ICLR 2026
Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li. Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
URL hhttps://arxxtt www.xiv.org/aaa. abs/ /25///as//2111...1 2501 12948
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai. Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12622-12642. Association for Computational Linguistics, 2024. doi: 10. 18655v/3/v ////2/22clllcc acl2...cl--onn.688.. URL http: /ttt:://xx.oo. doi. httppp//.kk.d...ooooo........31. 18653/11116333222222
acl ong. 681
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https //github... com hti.thub.ccmmuuuuggggg...... /open-rop/r//
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.173233 2022.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of 1lm inference using lookahead decoding. arXiv preprint arXiv:2402.00 02055, 2024.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-eexpers. Proceedings of Machine Learning and Systems, 5:288-304, 2023.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-Predict: Parallel
Decoding of Conditional Masked Language Models, September 2019. URL UtRthttp://aaxx brgg/bb/ 1904. 09324 arXiv:1904.0933: [cs].
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better and faster large language models via multi-token prediction, 2024. URL http: URLhttp:
//arxiv/
exq/abs// 2404. 19737 arXiv:2404.19737.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.082522 2023.
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding+4: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01222, 2023.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijic Deng, and Hao Zhang. Cllms: Consistency large language models. arXiv preprint S0iv:2403.00833: 2024.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pp. 61 1-626, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41 International Conference Machine Learning (ICML), 2024a. URL htttttt:///ppereeeeeess 1e to appear.
net
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprini arXiv:2401.15077 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference celeration of large language models via training-time test. arXiv preprint arXiv:2503.0168 840, 2025.
Zeping Li, Xinlong Yang. Ziheng Gao, Ji Liu, Guanchen Li, Zhuang Liu, Dong Li, Jinzhang Peng. Lu Tian, and Emad Barsoum. Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference, October 2024c. [es].
13170 arXiv:2406.13170
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 87-100, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Cong Liu, Zhong Wang. Sheng Yu Shen, Jialiang Peng, Xiaoli Zhang, ZhenDong Du and YaFang Wang. The chinese dataset distilled from deepseek-r1-671b. https::/huggingff co/ datasets
2025.
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint 188111 2024b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emnie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint a7X.v:2305.17888, 2023.
AI @ Meta Llama Team. The llama 3 herd of models, 2024. URL wt..:tttpp//aaxi..rr00/ 2407.22 21783
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulum Du, Tao Jiang, Chao Hong. Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. MoBA: Mixture of block attention for long-context llms. arXiv preprint ar 2rXiv:2502.13/88 13189, 2025.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
Meta. Accelerating generative with pytorch ii: Gpt, fast, November 2023. URL https:// pyt.orch.org/bllo//eeee wew.aaamnnnggeerraaeeee------
13

---

Published as a conference paper at ICLR 2026
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305.0978/ 2023.
Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.1358/ 2023.
Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. Suffixdecoding: A model-free approach to speeding up large language model inference. arXiv preprint arXiv:2411.04975, 2024.
OpenAl. Gpt-4 technical report, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Shwetha Somasundaram, Anirudh Phukan, and Apoorv Saxena. Pld+: Accelerating Ilm inference by leveraging language model artifacts. arXiv preprint arXiv:2412.01447 2024.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36:30222-30242. 2023.
OpenThoughts Team. Open Thoughts. https://peen-thoughssaas ai, January 2025.
Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Make some noise: Unlocking language model parallel inference capability through noisy training. arXiv preprint arXiv:2406.17400 2024.
Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. Swift: On-the-fly fly self-speculative decoding for IIm inference acceleration. arXiv preprint arXiv:2410.06916, 2024a.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency large language model inference: A comprehensive survey of speculative decoding, arXiv preprint #rXiv:2401.07833 2024b.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. / /7453, 2023b.
Zilin Xiao, Hongming Zhang, Tao Ge, Siru Ouyang, Vicente Ordonez, and Dong Yu. Parallelspec: Parallel drafter for efficient speculative decoding. arXiv preprint arXiv:2410.05589, 20244
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Aligmment data synthesis from scratch by prompting aligned Ilms with nothing. arXiv preprint arXiv:2406.08-664 084644 2024.
Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025.
Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling. arXiv preprint arXiv: /5766, 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:44611-34770000 2023.
14

---

Published as a conference paper at ICLR 2026
A
TRAINING COST
Comparison setting: We measure training cost in Pflops per 1M tokens, with the target model fixed to LLaMA3-70B.
PARD: For non-long-text scenarios, the forward pass cost is approximated as twice the model size, where the factor 2 accounts for both multiply and add operations:
PARD. 2 ParameterSize InpurTokenNumm/100 10 2 100 10 100 /10 -6.74
where 3.37M the effective input token count after applying mask prediction with COD. The backward pass requires twice the forward cost:
PfloPSPARD,B 22 6.74 13.48,
giving a total training cost of
PoROOSPARD 20.22.
EAGLE: The forward pass includes both the target model (70B) and the draft model (0.8B layer + 1.0IB Im-head):
PfLOPSEAGLE, R x (70 + 0.86 + 1.01) x 10 x 10 109/1015 143.74.
For the backward pass, the target model requires gradient computation, the draft layer costs twice its forward, and the Im-head costs one forward (activation gradients only):
PflOPSEACLE, 2 (2 x 0.86 +.01) 10 10 /10 5.46,
yielding total training cost of
PfloPSEAGLE 149.20.
EAGLE-3: Compared with EAGLE, the training strategy uses HASS (Zhang et al. 2024). and the input tokens for the EAGLE-3 head are times that of EAGLE. For the forward pass:
PfIOPSEAGLEA.F R 2 x (70+7) + x (0.86 + 1.01)) x 10 x 10 109/10/5 166.18.
For the backward pass:
PRIOPSEAGLES. R 7x (2 x 0.86 + 1.01) x 10 x 10 100/10/5 38..22,
yielding total training cost of
PfloPSEAGLES R 204.4.
For training draft models LLaMA3-70B, PARD achieves training efficiency that is 7x higher than EAGLE and 10x higher than EAGLE-3.
Target Independence: PARD's target independence allows single draft model to entire model series (e.g., LLaMA3-SB, 70B, 405B), while EAGLE-style methods require a separate draft for each target.
15

---

Published as a conference paper at ICLR 2026
B CONDITIONAL DROP TOKENS ALGORITHM
naive random token drop could indeed break the completeness of key and value information during training. Algorithm presents the pseudocode of the COD data processing procedure. Below we provide an illustrative example of how tokens are selected for dropping, based on Figure
For clarity. we further index each original token m. in the figure as majj for j E 1,,,,, where j indicates the position within mi.
In Figure 4b without COD, the training sequence is divided into three sections (delineated by dashed lines), corresponding to prediction objectives for positions +1, +2, and +-3 using all tokens.
Figure 4c with COD applied, shaded regions denote dropped tokens. We use geometric- decay retention rate r 0.5:
For position +1, no tokens are dropped in the leftmost section, so the full context [tell, me, a, story] is used.
For position +2, we drop (1 r) 50% of tokens in the middle section. Specifically,
we drop mo,,0 and m0,3. and retain mo, and m0.2-
For position +-3, we drop (1 - )²) = 75% of tokens in the rightmost section but ensure complete prefix key and value context for each retained token. For example, 11.3 (predicting "about" from [tell, me, mo.2)) can be retained because all its prefix tokens remain. Similarly, m1,2 can be retained since its prefixes [tell, m are intact, whereas m1,1 is excluded because its prefix mopo was dropped at position +22 From the valid candidates (m,12, m1,3,, }, we randomly select one token (here m1,2).
-
In Figure 4d new data after COD.
Regarding COD's effect, conducted an ablation study in Figure 5a The results show that COD can speed up training by threefold while maintaining the same inference acceleration on the target model.
Algorithm PARD with Conditional Drop Tokens: Data Processing
1: Input:
Training dataset D, PARD prediction count K, retention decay factor r, minimum retention rate Tiin
2: Output: Processed training data with updated soput_ids, labels, position encodings, and attention masks
3: for each data sample X D do
4: X + [XI...., Xx). where X₃ is the training data for predicting the k-th token, including
input ids, label, position ids, and attention mask
5: for k = 1 to K do
6: Compute retention rate: y max(r*=1 m'in)
7: Decide which tokens in X to retain, ensuring that the preceding KV cache for attention
computation is complete
8: Update X (i.e., input ids, label, position ids, and attention mask) to obtain X end for
9:
10: Merge updated sequences X' +- [X,...,, X'x and update the overall attention mask
11: Store the processed data for this sample
12: end for
C TRAINING HYPERPARAMETERS
Table 4 summarizes the hyperparameters used for training.
D LARGE BATCH SIZE INFERENCE
Table presents results across batch sizes ranging from 1 to 16. As the batch size increases, the bottleneck shifts from memory-bound to compute-boun bound. Under these conditions, PARD achieves a speedup of 1.33x to x.63
16

---

Published as a conference paper at ICLR 2026
Table 4: Selected Hyperparameters for PARD Training
Hyperparameter
Llama3
Deepseek-Rl-Qwen
Qwen
Optimizers
AdamW
AdamW
AdamW
Learning Rate
le-5
3e-5
8e-5
Per Device Train Batch Size
4
4
8
Gradient Accumulation Steps
2
2
1
Num Processes
8
8
Num Train Epochs
4
Training PARD K
8
8
8
Max Seq Length
512
1024
512
PARD Token ID
128020
151665
151665
The Answer Of Training Data
Regenerate Original
Original
Regenerate
Table 5: Performance comparison across different batch sizes on LLaMA3.1-8B in the
vLLM framework, evaluated on HumanEval.
bs=1
bs=2
bs=4
bs=8
bs= 16
Method
Speedup
AR
1.00
1.00
1.00
1.00
1.00
EAGLE
1.86
1.69
1.69
1.44
1.19
VSD
2.13
2.03
1.88
1.61
1.41
PARD
3.63
3.16
2.59
1.90
1.33
E PERFORMANCE ON DEEPSEEK QWEN SERIES
Table reports the acceleration effects of PARD on the DeepSeek- series, where the evaluation benchmarks consist of mathematics and code tasks.
Table 6: Performance comparison of different methods on the DeepSeek Qwen series.
Target
Method
HumanEval
GSM8K
Math500
Average
TPS
SpeedUp
SpeedUp
TPS
SpeedUp
TPS
SpeedUp
DS 7B
AR
75.87
1.00
75.96
1.00
75.92
1.00
75.92
1.00
VSD
97.90
1.29
129.20
1.70
122.51
1.61
116.54
1.54
PARD
162.39
2.14
204.62
2.69
205.23
2.70
190.75
2.51
DS 14B
AR
40.74
1.00
40.78
1.00
40.72
1.00
40.75
1.00
VSD
75.80
1.86
102.88
2.52
95.35
2.34
91.34
2.24
PARD
103.34
2.54
130.17
3.19
133,98
3.29
122.50
3.01
F PERFORMANCE DIFFERENCES ACROSS DIFFERENT INFERENCE FRAMEWORKS
During our experiments, observed that inference throughput using the Transformers library is significantly lower than that of vLLM. For methods such as vanilla SPD, this suboptimal performance of Transformers can lead relative speedups under the same inference framework being lower than those measured on vLLM (e.g.. 1.36x and 2.26x). To better reflect world usage scenarios, all final experiments in this paper conducted the vL..M framework. All experiments conducted
17

---

Published as a conference paper at ICLR 2026
under chain attention. Tree attention is an orthogonal decoding strategy and can be combined with PARD.
Inspired by GPT-Fast (Meta 2023). further optimized Transformers using torch. compil le and static key-value cache, resulting in Transformers+. Table tab:dif-methoo method--ompparion presents comparison of Transformers, Transformers+, and vL..M, showing that Transformers+ is lightweight while approaching the performance of vLLM. We employ Transformers+ for development and testing throughout the study.
Table 7: Comparison of different frameworks and methods HumanEval and GSM8K for LLaMA3. Here, Transformers+ denotes an optimized version of Transformers.
HumanEval
GSM8K
Average
Target
Framework
Method
TPS
Speedup
TPS
Speedup
TPS
Speedup
AR
34.36
1.00
35.90
1.00
35.13
1.00
Transformers
VSD
50.52
1.47
45.24
1.26
47.88
1.36
PARD
145.47
4.23
114.65
3.19
130.06
3.70x
AR
76.34
1.00
76.50
1.00
76.42
1.00
L3.1 8B
Transformers+
VSD
185.29
2.43
160.59
2.10
172.94
2.26
PARD
336.97
4.41
275.03
3.60
306.00
4.00
AR 73.07
1.00
73.38
1.00
73.22
1.00
vLLM
VSD
155.47
2.13
140.93
1.92
148.20
2.02
PARD
264.88
3.63
235.09
3.20
249.98
3.41
G COMPARISON WITH OTHER PARALLEL SPECULATIVE DECODING METHODS
To better align with real-world deployment scenarios, the performance comparisons in this paper are conducted within the vLLM framework. Although not included experimentally due implementa- tion constraints, we summarize below conceptual comparison between PARD and other parallel speculative decoding methods in Table
Table 8: Comparison of draft model design and target-model independence across different specu- lative decoding methods.
Method
Draft Model Design
Parget-Model Independence
ParallelSpec
Parallel draft head built based on the target model
Requires separate tuning for each target
BTTA
Prefix tuning with additional learned prelix lokens
Requires separate tuning for each
PASS
Look-shead token embeddings trained with each target model
Requires separate tuning for each target
PARD (Ours)
Fully decoupled small drail model
One draft supports family of largel models
H LLM USAGE STATEMENT
In the preparation of this paper, Large Language Models (LLMs) were used solely as an assistive tool for language polishing and minor stylistic refinement. LLMs were not involved in research idcation, experimental design, analysis the paper.
18


Reviews:
[{"review_id": "epavWYNq3e", "reviewer": "Reviewer_6iwh", "summary": "The paper introduces PARD, a speculative decoding framework that accelerates LLM inference through a target-independent, parallel draft model. It combines mask-based prediction and a novel Conditional Drop-token (COD) training strategy to enhance both training and inference efficiency. PARD achieves up to 3.67× speedup on LLaMA3.1-8B and outperforms the current state-of-the-art EAGLE-3 model in both speed and generalizability.", "questions": "Could the authors justify the omission of comparing to other recent parallel speculative decoding methods, and if possible, include some comparative results to support the performance and efficiency claims of PARD in the parallel decoding category?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Unlike methods like EAGLE, PARD can accelerate an entire family of target models with a single draft model, significantly reducing adaptation and deployment costs.\n2. Moreover, the use of masked tokens allows PARD to predict multiple tokens in a single pass, improving inference speed without sacrificing quality.\n3. Extensive benchmarks on HumanEval, GSM8K, and SpecBench across multiple LLMs demonstrate consistent speedups and high acceptance rates compared to EAGLE and vanilla SD.", "weaknesses": "The paper introduces PARD as a parallel and target-independent speculative decoding method. However, the experimental section lacks direct comparisons with other recent parallel speculative decoding methods such as ParallelSpec, BiTA, or PaSS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "alLpq1WAks", "reviewer": "Reviewer_Gfez", "summary": "This paper proposes PARD (Parallelized Accelerated Reasoning Decoding), a novel speculative decoding method featuring target-independence and parallel token prediction. PARD is highly generalizable: its target-independent design allows a single draft model to accelerate an entire family of target models. Through mask tokens, PARD substantially accelerates inference by predicting multiple future tokens within a single forward pass of the draft phase. And they propose a COnditional Drop-token (COD) strategy for the effient training of PARD. Leveraging the integrity of prefix key-value states, COD enables low-cost adaptation of autoregressive draft models into parallel ones, boosting training efficiency by up to 3× while maintaining accuracy. In the end, the author integrates PARD into the high-performance inference framework vLLM.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper makes a pivotal conceptual shift by introducing a target-independent speculative decoding framework. Unlike dominant approaches like EAGLE and Medusa that train a draft model tightly coupled to a specific target model, PARD enables a single draft model to accelerate an entire family of target models.\n2. The work innovates by integrating parallel token prediction into speculative decoding via masked language modeling.A key novelty is the Conditional Drop-token (COD) mechanism, which leverages the integrity of prefix key-value states to dramatically reduce training overhead. This allows efficient adaptation of autoregressive models into parallel draft models. The use of a shared mask token ID to enable extrapolation beyond the trained prediction length is another clever and inventive design choice that enhances practicality.\n3. The integration with the industrial-grade vLLM framework underscores its immediate practical significance, providing a scalable and efficient solution that directly addresses real-world inference bottlenecks.", "weaknesses": "Limited Analysis of the Parallel Drafting Mechanism's Limitations: The paper celebrates the efficiency of parallel drafting but fails to critically analyze its inherent constraints. Generating K tokens in one forward pass necessarily uses less contextual information (relying on masks) than an autoregressive model, which may hurt prediction accuracy for long-range dependencies or complex reasoning steps. This likely explains the performance drop on certain tasks (e.g., PARD vs. EAGLE-3 on SpecBench for LLaMA3.3-70B). The work would be strengthened by a dedicated analysis—for instance, measuring how acceptance rate decays with K on different task types—to delineate the practical boundaries of its parallel approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hRVqYpO4tj", "reviewer": "Reviewer_GWvT", "summary": "The paper proposes PARD (PARallel Draft), a novel method for speculative decoding (SD) designed to accelerate Large Language Model (LLM) inference while featuring target-independence and parallel token prediction. PARD addresses the high adaptation cost of target-dependent methods like EAGLE by allowing a single, low-cost draft model to be applied across an entire family of target models. The core mechanism involves fine-tuning a small auto-regressive (AR) model into a parallel draft model, utilizing the Mask-Predict approach with special mask tokens ($m_k$) to predict multiple candidate tokens in a single forward pass. This significantly reduces the draft model's latency from $K \\times T_D$ to $T_D$.\nTo reduce the training cost for this parallel adaptation, the authors introduce the Conditional Drop-token (COD) mechanism. COD enables sparse sampling of training data while ensuring the integrity of prefix key-value states, boosting training efficiency by up to $3\\times$ compared to traditional masked prediction training. Experiments on the industrial vLLM framework show that PARD achieves a state-of-the-art speedup on LLaMA3.1-8B.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes PARD, which allows a single same-family draft model to accelerate an entire family of target models via multi-token prediction with look-head tokens.\n\n2. This paper integrates a token-dropping scheme to reduce the draft model’s training cost.", "weaknesses": "- The paper frames advanced SD methods as \"target-dependent\" and PARD as \"target-independent\", yet PARD crucially assumes the availability of a same-family small model—often unrealistic for fine-tuned production LLMs. In those settings, self-speculative decoding from the target model itself (e.g., LayerSkip/ACL’24, Self-SD/ACL’24, SWIFT/ICLR’25) seems strictly more applicable. The paper should explicitly discuss this constraint, compare against self-SD scheme, and clarify when PARD is preferable.\n   \n- Much of the method resembles transferring look-ahead token prediction to a small model and then boosting with fine-tuning. Given that small-model training is a one-off offline cost, the contribution of the token dropping mechanism reads like a patch rather than a core innovation.\n\n- The concept of parallel decoding is not novel. For instance, [1] has proposed a similar idea based on a parallel-decoded drafter and a target model as a verifier for speculative decoding. Additionally, Medusa, Multi-Token Prediction (MTP) are existing parallel decoded speculation approaches. Compared with the existing works, this paper mainly differentiates in its draft model architecture design that leverages the [MASK] tokens as a placeholder to predict future tokens within one model forward pass. However, this design is similar to the approach proposed in [2] [3].\n\n- PARD comes from converting the sequential prediction to a parallel prediction, reducing the time cost of the draft stage from $$K \\times T_D$$ to $T_D$. However, the unresolved issue is whether the $T_D^$ of the new draft model will change, and whether it is possible that $K \\times T_D$ is less than $T_D^$ when the draft model is too large.\n\n- The paper claims that its draft model is target-independent and can accelerate an entire family of target models. However, for target model families with a large range of parameter quantities, using the same draft model (lacking the adjustment of the draft model to fit the size of the target model) may lead to the problems mentioned in the last weakness. That is, universal draft model may perform slower in parallel decoding than the ideal draft model even if in sequential decoding. The evidence in the paper is insufficient to reveal or fix this issue.\n\n- The Shared Mask Token ID Strategy in ablation study needs concrete details: how mask tokens are inserted across variants, which IDs are shared.\n\n- Presentation issues: \n   * Fig. 4 (especially subfigures (b) and (c) on training-time attention masks) is hard to parse. In addition, for Fig. 4(a), clarify the meaning of green/red. If green denotes permitted attention, the causal mask for query at position t should be upper-triangular over keys/values (e.g., \"Q(story)\" must attend to its history; the last column should be green). \n   * Equations (7) and (8) have an indexing bug: substituting k=0 (or k=1) yields m_{-1}; define base cases and boundary conditions to prevent invalid indices. \n   * The core idea of COD (\"earlier-subtask tokens are more critical; later ones can be dropped\") lacks justification—why are later tokens less important? Is this theoretically motivated or empirically observed (e.g., uncertainty, acceptance rates)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxdnVcTMNT", "reviewer": "Reviewer_MJ3S", "summary": "This paper proposes PARD, which accelerates autoregressive generation by training a target-independent draft model that generate drafts in parallel. Furthermore, the authors propose the CoD method to accelerate the training of the parallel inference model. Owing to parallel generation and target independence, PARD can quickly transfer across different model within the same series and generate drafts at a low cost, achieving greater acceleration compared to the autoregressive EAGLE.", "questions": "1. The experimental results in Table 1 lack reporting on the acceptance length, and the acceleration ratio of EAGLE is lower than that of other benchmark methods. It would be useful to clarify whether Tree validation is employed in this study.\n\n2. In addition to the absence of a comparison with related methods, the related work section overlooks relevant literature on parallel generation techniques, which could offer valuable context for this study.\n\n[1] Kou S, Hu L, He Z, et al. Cllms: Consistency large language models[C]//Forty-first International Conference on Machine Learning. 2024.\n\n[2] Wang Y, Luo X, Wei F, et al. Make some noise: Unlocking language model parallel inference capability through noisy training[J]. arXiv preprint arXiv:2406.17404, 2024.\n\n[3] Gao X, Xie W, Xiang Y, et al. Falcon: Faster and parallel inference of large language models through enhanced semi-autoregressive drafting and custom-designed decoding tree[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(22): 23933-23941.\n\n3. As indicated in Table 2, the high acceptance rate of PARD primarily stems from its consistency with existing homologous models, rather than additional training. Consequently, it is essential to conduct comparisons between PARD and other draft model acceleration approaches to better assess its relative performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured, with clear and intuitive diagrams that effectively support the explanation.\n\n2. It explores the concept of parallel draft generation, providing insightful and innovative perspectives on the topic.", "weaknesses": "1. The paper lacks a comparison with other similar draft acceleration methods, such as hierarchical acceleration [1,2] and draft models in diffusion models [3].\n\n2. The experimental setup is not sufficiently detailed, with insufficient explanation of decoding strategies (e.g., chain or tree).\n\n[1] Chen Z, Yang X, Lin J, et al. Cascade speculative drafting for even faster llm inference[J]. Advances in Neural Information Processing Systems, 2024, 37: 86226-86242.\n\n[2] Zhao W, Huang Y, Han X, et al. Ouroboros: Speculative decoding with large model enhanced drafting[J]. arXiv preprint arXiv:2402.13720, 2024.\n\n[3] Christopher J K, Bartoldson B R, Ben-Nun T, et al. Speculative diffusion decoding: Accelerating language generation through diffusion[J]. arXiv preprint arXiv:2408.05636, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)