PEARL: Parallel Speculative Decoding with Adaptive Draft Length
Authors: Tianyu Liu
Yun Li
Qitan Lv
Kai Liu
Jianchen Zhu
Winston Hu
Xiao Sun
Venue: ICLR 2025 Poster (2025)
Keywords: speculative decoding
inference acceleration
large language models
Primary area: generative models
TL;DR: we introduce PEARL (Parallel spEculative decoding with Adaptive dRaft Length) to further reduce the inference latency of Large Language Models (LLMs).

Abstract: Speculative decoding (SD), where an extra draft model is employed to provide multiple **draft** tokens first and then the original target model verifies these tokens in parallel, has shown great power for LLM inference acceleration.
However, existing SD methods suffer from the mutual waiting problem, i.e., the target model gets stuck when the draft model is *guessing* tokens, and vice versa. This problem is directly incurred by the asynchronous execution of the draft model and the target model, and is exacerbated due to the fixed draft length in speculative decoding.
To address these challenges, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely 
**P**arallel sp**E**culative decoding with **A**daptive d**R**aft **L**ength (PEARL). 
Specifically, PEARL proposes *pre-verify* to verify the first draft token in advance during the drafting phase, and *post-verify* to generate more draft tokens during the verification phase.
PEARL parallels the drafting phase and the verification phase via applying the two strategies, and achieves adaptive draft length for different scenarios, which effectively alleviates the mutual waiting problem.
Experiments on various text generation benchmarks demonstrate the effectiveness of our PEARL, leading to a superior speedup performance up to **4.43$\times$** and **1.50$\times$**, compared to auto-regressive decoding and vanilla speculative decoding, respectively.

Published as a conference paper at ICLR 2025
PEARL: PARALLEL SPECULATIVE DECODING WITH ADAPTIVE DRAFT LENGTH
Tianyu Liu 1.2 Yun Li
Qitan Lv
Kai Liu?
Jianchen Zhu
Winston Hu
Xiao Sun
l University of Science and Technology of China
Shanghai Al Laboratory
Tencent
(tianyu qitanlv)@mail, ust wattc.euuuu.. cn calliicharleeegee com sunxiao@p lb. org.on
ABSTRACT
Speculative decoding (SD), where an extra draft model is employed to provide multiple draft tokens first, and then the original target model verifies these tokens in parallel, has shown great power for LLM inference acceleration. However, ex- isting SD methods suffer from the mutual waiting problem, i.e., the target model gets stuck when the draft model is guessing tokens, and vice versa. This prob- lem is directly incurred by the asynchronous execution of the draft model and the target model and is exacerbated due to the fixed draft length in speculative decod- ing. To address these challenges, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely Parallel spEculative de- coding with Adaptie dRaft Length (PEARL). Specifically, PEARL proposes pre- verify verify the first draft token in advance during the drafting phase, and post- verify to generate more draft tokens during the verification phase. PEARL paral- lels the drafting phase and the verification phase via applying the two strategies. and achieves adaptive draft length for different scenarios, which effectively allevi- ates the mutual waiting problem. Experiments on various text generation bench- marks demonstrate the effectiveness of our PEARL, leading to superior speed up performance up to 4.43 and 1.50 compared to ruto-regressive decoding and vanilla speculative decoding, respectively. Our code is available at https: / //github. com/ smart stt.atttwtt///aaalllleeeeeeeeaaa
INTRODUCTION
Large language models (LLMs) such GPT-4, LlaMA, and DeepSeek (Achiam et al., 2023; DeepSeek-AI, 2024; Bommasani et al., 2021; Touvron et al., 2023) have dominated natural language understanding and generation (Khurana et al., 2023) over wide range of applications. However, the substantial inference latency of these LLMs has emerged as a significant obstacle bounding their broader application in scenarios with restricted computational resources. This latency primarily originates from the auto-regressive token- token-by-token hy-token decoding process wherein decoding tokens requires K serial runs of LLMs, incurring exacerbated latency with both the length of generated tokens and the model scale.
To address this challenge, extensive research efforts have been devoted accelerating LLM in- ference. Given that inference from large models is often constrained more by memory bandwidth and communication than by arithmetic operations Leviathan al. (2023), one innovative inference paradigm, Speculative Decoding (SD), has emerged as new trend and shown superior performance by effectively enabling better GPU utilization. As shown the upper part of Figure. 1, the key idea of the SI algorithm is to employ an extra small model (referred as the draft model) to first generate draft tokens for the original large model (referred as the target model), and then the target model ver- ifies these draft tokens in parallel within single forward. Here, y is fixed hyperparameter window
"This work is done when Tianyu Liu works as an intern in Tencent.
The Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Prefix: In the domain of speculative decoding, PEARL is a
free algonaio
Hury
acceraaccy
mechod
PEARL is a model free algorithm paralle/inference acceleration method that
Step 1 Pre: model free alporthm
Step 1 Pre: paralle
Stop 2 Pri: Aiforonce awcotcattonnaaaat
Stop 2 Pre: intorisies
Step 3 Posti ither isen achieve
Stop 3 Post: intersncc accelivation monthoo
Step 4 Post: adaeeve draft Hhogth
Step 4 Post that can achieve
PEARL model free www.reerrroiiinooaaaeeeeiiininne acceleration method that can achieve adaptive draft length
Figure 1: An overview of speculative decoding (the upper part) and our PEARL (the lower part). SD employs a draft model to provide multiple drafts and then the target model verifies the drafts in parallel. However, SD suffers from the mutual waiting problem, i.c., the target model gets stuck when the draft model is guessing tokens, and vice versa (the dashed dialogue box). PEARL parallels the drafting and verification process alleviate the mutual waiting problem. Moreover, PEARL can leverage adaptive draft length to generate more tokens within the same amount of time to further mitigate the mutual waiting problem. Specifically, PEARL generates fewer draft tokens if they will be rejected (step in the lower part), and more draft tokens they can be accepted (steps 3 and 4).
size. Draft length is the number of tokens generated by the draft model in continuous execution. Therefore, the draft length is set to y in SD. Following-up works effectively extend this framework by either removing the necessity of the draft model (Cai et al., 2024: Fu et al., 2024; Zhang al., 2023) or identifying compact draft model with high distribution alignment (Zhou et al., 2023; Zhao et al., 2024; Miao et al., 2023). Extensive experiments demonstrate that this drafi-then-verify frame- work effectively enhances the concurrency of the target model, thereby significantly accelerating the inference process.
Albeit with multiple benefits of this draft-then-verify framework, it confronts one significant chal- lenge that may hinder its performance and deployment- mutual waiting problem. That is, the target model will be idle when the draft model is generating the draft tokens and the draft model will be idle when the target model is verifying the previously drafted tokens. This mutual waiting problem primarily stems from two limitations inherent in speculative decoding: (i) the asynchronous execution of the draft and verify phases, which directly results in the mutual waiting problem; and (ii) the fixed draft length, which cannot adapt to most decoding steps and thus exacerbate the mutual waiting problem.
Therefore, in this paper, we seek to answer the question: Can we draft and verify in parallel and adaptively adjust draft length? With this consideration, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely Parallel spEculative decoding with Adaptive dRaft Length (PEARL). Specifically, PEARL consists of two strategies pre-verify and post-verify: (i) pre-verify uses the target model to verify the first draft token during drafting phase, which allows the draft model to generate less draft tokens in difficult scenarios; (ii) post-verify uses the draft model to continue generating draft tokens during verification phase, which provides more draft tokens in simple situations. As shown in the lower part of Figure. PEARL effectively alleviates the mutual waiting problem with parallelism and adaptive draft length via these two strategies. We conduct extensive experiments various text generation benchmarks, leading to superior speed up performance up to 4.43x and 1.50x, compared to auto-regressive decoding and speculative decoding, respectively.
2

---

Published as a conference paper at ICLR 2025
speculative decodling
Influence of Draft Length
M
200
stattic optimal droft length
100
Keration optimal draft length
M
Mq
Mp
PEARL
M₁ decoder
M, decoder
Model Stuck
Draft Leagth
20
Iteration
(a)
(b)
Figure 2: Motivated observations. (a) The time of both the drafting phase and verification phase non-negligible, therefore the asynchronous execution of the draft model and the target model directly incurs the mutual waiting problem. (b) We observe that the optimal draft length changes significantly in different iterations, which exacerbates the mutual waiting problem.
BACKGROUND
Mg to denote the draft model and Mp to denote the target model. (,(,),M(() denotes the logits of the next token of single forward of M,. Mp respectively. is hyperparameter control the window size during speculative decoding. We denote the running speed between Mg and M, as c, which is defined as the ratio between the time for a single forward of M, and the time for single forward of Mq. i.e., (-), (M))/T((())),
Notations. In this paper,
Speculative decoding. Given an input sequence as prefix, a speculative decoding step consists of drafting phase and verification phase. During the drafting phase, the draft model M, is employed to give draft tokens F2, By by running y times model forward and sample. Here, we denote M2 + [x1 ..., 11)-11)) as q:. then each draft token is given by I; q. i 1, ..., % During the verification phase, the prefix x together with draft tokens are sent to Mn for verification. The target model M, inputs x + [ri,,.., Fy,] and outputs the logits P1,P2, ... P-+1- 1- Then SD sequentially verifies z, via speculative sampling, where the acceptance rate is given by:
[x[zv] q [zv].
0 = Pa [x:s]
p[[r,] q [x:]]
q: [<(
If SD rejects v,, it will resample a token from norm(max(0,22)
g:)), otherwise, SD accepts all the draft tokens and samples an additional token from 1-+ In this way, each SD step generates tokens with number of least 1 and at most y + 1, leading efficiency acceleration.
Window size and draft length. We emphasize that the window size is hyperparameter that con- trols the drafting behavior. Draft length is the number of tokens generated by the draft model in continuous execution, which is fixed and the same as the window size in SD, while draft length is adaptive and may be not equal window size in PEARL.
METHODOLOGY
3.1 MOTIVATED OBSERVATION
As illustrated in Figure. 2(a), the mutual waiting problem is directly incurred by the asynchronous execution of the draft model and the target model. In our experiments, we observe that the time consumed during the drafting phase and the verification phase is usually nen-negli Take the instance of Codellama 7B & 34B, at each decoding step, although the running speed of Codellama 7B is almost times faster than Codellama 34B, the total time consumption for generating 6 draft
3

---

Published as a conference paper at ICLR 2025
Parallel Speculative Decoding With Adaptive Draft Length
Draft Medel
?
?
Target Model
X4
X5
Pre-Verify: X
Pre-Verify:
Post-Verify:
Post-Verify: X
pre
V
V
X
X
X7
X
X
PM
PS
pk
P
PS
P5
PS.
PS
Y1
T
T = 3t
= 6t
T =
T= 12t
Figure 3: Illustration of our PEARL. At 0, Mg generates 21; #2, 33 and Mp rejects 21 with the pre-verify strategy. At T = 3t, M, accepts Ia and switches to the post-verify strategy. At T = 6t, Mp accepts all draft tokens :4, IG, ze in the last decoding step, while M, continues drafting I7, Ts, Zg. At T = 96, Mp rejects Ig. drops Iı0, I11, and switches to the pre-verify strategy. The final output is [8/, F4. D5, Te, I7, Fg. y2l-
tokens is even times than the time consumption for one verification step. Therefore, the mutual waiting problem exists at any timestamp, and severely affects the acceleration effectiveness of SD.
The asynchronous execution of the draft model and the target model is the direct cause of the mutual waiting problem, which is determined by two requirements of speculative decoding: (1) the drafting phase requires the input prefix to be verified; (2) the verification phase requires the draft model to complete generating draft tokens. This implies the great potential for alleviating the mutual waiting problem through parallelism: if can remove the two requirements and parallel the drafting phase and the verification phase, substantial acceleration can be possible.
Another limitation that aggravates the mutual waiting problem is the fixed draft length in SD, which is not appropriate for all the decoding steps. As shown in Figure 2(b), the optimal draft length changes significantly in different iterations. On the one hand, when the optimal draft length less than the fixed draft length, the draft model will generate meaningless draft tokens that block the target model. On another hand, when the optimal draft length is more than the fixed draft length, the draft model could have generated more draft tokens that can be accepted by the target model with a single forward. However, fixed draft length will interrupt the longer drafting phase and take an additional verification phase, which strengthens the mutual waiting problem as well. This motivates our PEARL to further alleviate the mutual waiting problem with adaptive draft length.
Together with the two motivations, we propose two simple and effective strategies, per-verify and pest-verify. The pre-verify removes requirement and allows the target model to verify the first draft token in advance. The pest-verify removes requirement and allows the draft model to continue generating draft tokens during the verification phase. The two strategies enable parallelism and achieve adaptive draft length effectively alleviate the mutual waiting problem.
3.2 PRE-VERIFY: VERIFY THE FIRST DRAFT TOKEN ADVANCE.
The pre-verify strategy aims at removing the requirement that the verification phase requires the draft model to complete generating draft tokens. Therefore, we seek to verify some draft tokens in advance during the drafting phase. We delve explicitly into the drafting stage. During the drafting phase, the draft model tries to give y draft tokens by running times model forward. We find that the input of the draft model in y times forward is x, x [ma), .... + [#1, T2, -. Zy-1), respectively.
4

---

Published as a conference paper at ICLR 2025
Only the origin prefix can be acquired by the target model for parallel verification. Therefore, propose to run the target model to output the logits Mx(x) in parallel. this way, we can verify the first token 21 before the verification phase. We implement the same lossless verification method following (Leviathan et al., 2023) as illustrated in Section 2.
By applying such pre-verify strategy, we verify the first draft token before the verification phase. If the first token is rejected, all of the following draft tokens are meaningless and should be dropped. Hence we could skip the verification phase and directly conduct the next drafting phase with the prefix x + [. If the first token is accepted, all the draft tokens will be sent to the target model in the verification phase. Figure. at the timestamp of 22, while the target model outputs pa, rejects the first token 21 and sample another token y1- At the timestamp of 7 34, the draft model generates 744, Wis; 2e while the target model accepts the first token T4. Then 34, Rs, is sent to the target model in the next verification phase.
the draft model generates
3.3 POST-VERIPY: CONTINUE DRAFTING DURING VERIFICATION.
The post-verify strategy aims at removing the requirement that the drafting phase requires the input prefix to be verified. However, this assumption brings the limitation that the draft model should be stuck until the target model finishes verification.
Therefore, we discard this assumption and make another assumption: we directly assume that all the draft tokens be accepted. In this way, We find that when all the draft tokens accepted, sampling a new token from Mp + [x1...., x,]) is not necessary, as the draft model could have generated more draft tokens that be accepted. Hence can the draft model to continue drafting I, T2y during the verification phase.
Algorithm Parallel Speculative Decoding with Adaptive Draft Length.
Require: the draft model Mg. the target model Mp. the input prefix x, the max generate tokens L, the window size 7.
Initialization: mode "pre-verify" while (xn(x) < T. do
if mode "pre-verify" then
x, mode + Pre-verify(Mp, Mp, x,
else
mode + Post-verify(M Mp. x. x,
end if
end while
If all the y draft tokens are accepted, we can skip the next drafting phase as we already get the draft tokens in the next drafting phase. The last logit Mp(x + 1y+1, which is a "pre-verify" process as well. In Figure. 3, at the timestamp of T = 67, the target model takes in Fa, 25, 36 and outputs ph, phg Pag. while the draft model continues guess draft tokens 27, Ig, -g. Fortunately, all the draft tokens are accepted, and we can directly conduct the next verification phase with prefix x + [V1, 34, To, IG, I6, Ts, 29). At the timestamp of T' 9t, the target model takes in I7, Ts, Ty and outputs pLa, P's Pag. while the draft model continues to guess the next draft tokens T10, T11, T12- Unfortunately, only as is accepted, and the draft tokens Iio. 1 12 will be dropped. Finally, the prefix x + [31, I4, 25. Re, 27, Fs V2) is input to the next drafting phase.
can be used to verify
3.4 PEARL: PARALLEL SPECULATIVE DECODING WITH ADAPTIVE DRAFT LENGTH
Taking together the two strategies, our PEARL framework consists draft model, a target model, and two strategies to decode tokens. The two strategies are switched according the verification results in the previous decoding step. Algorithm provides a summary of our PEARL. We also provide more details in Algorithm 2. Note that vre-verify and post-verify strategies not executed only once in the process of generating a sentence and will be repeatedly switched according to the token acceptance situation during the whole process of generating. We provide simple step-by-step
5

---

Published as a conference paper at ICLR 2025
profiling example in Appendix B for better understanding. Then show how PEARL achieves parallelism and adaptive draft length to alleviate the mutual waiting problem.
Parallelism. With the two strategies pre-verify and post-verify, At any timestamp, the draft model and the target model are running in parallel, which directly breaks the asynchronous execution of the draft model and the target model.
Adaptive draft length. In our PEARL, the drafting process can be seen as a segmented drafting process. If the draft model cannot generate any "right" tokens, the pre-verify strategy will avoid the additional drafting process. If the draft model could have generated more "right" tokens, the target model would not interrupt the drafting phase, where the draft model can generate more draft tokens with post-verify strategy. Therefore. PEARL can utilize the two simple yet effective strategies to implement adaptive draft length alleviate the mutual waiting problem.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Tasks and Datasets. We conduct experiments on various text generation tasks to evaluate the effec- tiveness of our PEARL, including HumanEval (code generation task) (Chen et al., 2021), GSM8K & MGSM (multilingual arithmetic reasoning task, MGSM is the multilingual translation of GSM8K) (Cobbe al., 2021; Shi et al.), and MT-bench (multi-round dialogue task) (Zheng et al., 2024). These tasks and datasets are representative benchmarks for evaluation. More details can be found in Appendix C.I.
Evaluation Details. We evaluate the effectiveness of our PEARL with some state-of-the-un LLM families, including CodeLlama (Roziere al., 2023), Deepseek-Codee -oder (Guo al., 2024), Llama (Touvron et al., 2023) and Llama 3.1 (Dubey et al., 2024). our experiments, the models with size less than 7B are used as the draft models and the models with size greater than 33B are used as the target models. We report the walltime speedup ratio as the metric. Additional evaluation details are provided in Appendix C.2 and C.3.
Baseline Methods. We implement four training-free inference acceleration methods as our base- lines. (i) Speculate decoding: standalone SD methods (Leviathan al., 2023; Chen et al., 2023) resort to draft model to draft future tokens and then verify them in parallel. (ii) Ouroboros: ouroboros (Zhao et al., 2024) proposes phrase candidate pool from the verification process gen- erate more precise and longer drafts. (iii) Lookahead Decoding: look ahead decoding (Fu et al., 2024) caches the generation trajectory (n-grams) as drafts reduce the number total decoding steps. (iv) Assisted generation: assisted generation (Joao Gante, 2023) employs heuristic ap- proach to determine the number draft tokens in the next iteration, based on the verification results of tokens generated by the draft model in the previous round.
Table 1: Experiment results on the code generation task. Part of the results of Lookahead Decoding and Ouroboros are taken from (Zhao al., 2024). We bold the best results for each model combination. "Some results of ouroboros and lookahead decoding are reproduced in their official implementation with default parameters. Other results are reproduced our implementation. The symbol '_' denote that the methods do not support current model configuration 2
CodeLlama 7&:4B
Code 7&70B
Llama2 7&70B
Llama3.1 8&70B
DeepSeek 1.3833B
DeepSeek
Method
6.7&33B
Auto Regressiive
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
Speculative Decoding (Leviathan al., 2023)
1,76x
3.03x
2.35x
2.60x
2.32x
1.94x
Ourbboras (Zhao et al., 2024)
2.14x
*3.28x
-2.10x
*3.25x
*2,66x
Lookahead Decoding (Fu et al., 2024)
1.72x
*1,57x
*1.80x
*1.82
*1,82
Assisted Generation (Joao Gante, 2023)
1.37x
2.49x
2.27
2.72x
1.88x
1.52x
PEARL (ours)
2.48x
4.43x
3.29
3.87x
3.48x
2.79x
Thheir implementation requires transformers version of 4.36.2, while Ilama 3.1 requires transformers 2
4.43.0

---

Published as a conference paper at ICLR 2025
Table 2: Experiment results the multilingual arithmetic reasoning task with Llama 2 70B. We bold the best results for each category. Results of ouroboros and lookahead decoding are reproduced in their official implementation with default parameters. Other results are reproduced in our implementation.
Method
English (GSM8K)
Bengali
Spanish
French
Jepanese
Russian
Swahili
Togalu
Regreseive
1.00x
1,00x
Speculirive Decoding
2.69
2.77%
2.64x
2.71
2,72x
2.65%
2.70x
Ourubunos
1.60x
1,75x
1.88
1.69%
1,30x
1,95K
1.85%
1,68x
2.45%
1.92x
1.81x
1.84x
Lookahead Decoding
1.25%
1.34x
151x
1.50%
1.48X
1.29
1.43%
1.60%
1.28%
1.23x
1.48x
1.39%
Assisted Ceneration
1.66
1.69%
1.75x
1.70x
1.67
2.00
1.88
1.58
3.07%
2.17%
1.97%
1.93x
PEARL (ours)
3,82
3.94
4.00
381x
3.66x
3,94
3,85x
4.18
4.10x
3,9x
4,06
3.95
Table 3: Experiment results on the multi-round dialogue task with Llama 2 70B. We bold the best results for each category. Results of ouroboros and lookahead decoding are reproduced their official implementation with default parameters. Other results are reproduced in our implementation.
Method
Writing
Roleplay
Reasoning
Math
Coding
Extraction
Stem
Humanities
Avg.
Auto Regressive
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
Speculative Decoding
1.70x
1.73x
1.96x
2.00x
1.93x
2.14x
187x
1.81x
1.89
Ouroboros
1.42x
1.35x
1.40x
1.61x
1.35x
1.67x
1.44x
1.36x
1.45x
Lookahead Decoding
1
1,24x
1,50x
1.51x
1.38x
1.40x
1,29x
1.27x
1.36x
Assisted Generation
141x
1.40x
1.39x
1.64x
1.74x
1.92x
1.57x
1.47x
1.55x
PEARL (ours)
2.40%
2.45x
2.85x
2.79x
2.67
2.92x
2.58x
2.50x
x
4.2
MAIN RESULTS.
We conduct extensive experiments on the aforementioned benchmarks. As shown in Table 1, PEARL significantly outperforms vanilla speculative decoding, Ouroboros, Lookahead decoding and assisted generation in all backbone model configurations the HumanEval dataset, which encompass different scales model configurations including 1.3&33B, 6.7&33B (7&34B) and 7&70B. Specifically, PEARL auto-regressive methods and vanilla speculative decoding, respectively. These results indicate the universal existence of the mutual waiting problem, and demonstrate that PEARL effectively ad- dresses the mutual waiting problem, thereby achieving significant inference acceleration results compared methods based on the traditional teen-rfyy framework. Moreover, shown in Table 2, 3, PEARL can also achieve significant inference acceleration on 12 multilingual arith- metic tasks and 8 multi-round dialogue tasks, whereas PEARL can achieve 1.36 1.55 x speedup compared with vanilla speculative decoding. These results demonstrate the superior potential of the parallel speculative decoding framework to exploit the computation resources more adequately. We provide more evaluation results on MGSM and MT-bench with more advanced Llama 3.1 8&B in Appendix D.
achieve up to 4.43x and 1.50x speed up compared with vanilla
4.3
ABLATION STUDIES
To provide more insights into the two proposed strategies, we conduct the ablation study. We denote PEARL without pre-verify as PEARL w/o pre-verify and PEARL without post-verify as PEARL w/o post-verify and present the main results of ablation studies.
As shown in Table 4, the absence of any strategy of PEARL results in a performance degradation of the entire framework. The absence of the post-verify strategy exhibits a more pronounced impact on the performance of PEARL than the pre-verify strategy. We explain the reason for this phenomenon as follows. Intuitively, the pre-verify strategy makes more contributions when the acceptance rate is relatively low. The pre-verify strategy can save a target model forward when the first draft token is rejected by the target model. Denote the acceptance rate as a, and the pre-verify strategy will take effect with probability 1 - Therefore, better alignment between the draft model and the target model will make pre-verify strategy less effective. However, the post-verify strategy makes more contributions when the two models are aligned, i.e., there are situations in which all draft tokens are accepted by the target model. Therefore, the two strategies are complementary and accelerate inference together.
7

---

Published as a conference paper at ICLR 2025
Table 4: Ablation results of PEARL HvmanEval and GSM8K datasets.
HvmanEval
GSM8K
Methods
CodeLlama 7B&34B
CodeLlama 7B&70B
DeepSeek 1.3B&33B
Llama 2 7B&70B
PFART. w/o pre-verify
2 (1 (1.0.14)
3.53x (1 0.26)
3.19x (L 0.29)
2.51 (1 0.26)
PEARL w/o post-verify
1.64x (1 0.71)
2.57x (1 1.22)
2.37x (1 1.11)
2.15x (1 0.72)
PEARL
2.35x
3.79
3.48x
x.87
In our experiments, all the model combinations show great alignment, which leads to the superiority of the post-verify strategy. As the language models evolve and more speculative decoding methods, the alignment between the draft model and the target model will be better, which further highlights the importance of the post-verify strategy. Meanwhile, we can further improve the pre-verify strat- egy by pre-verifying multiple draft tokens (similar to the cache pool in Ouroboros and Lookahead Decoding) for more acceleration. We leave these future works.
4.4 CASE STUDIES
In this subsection, we present two important case studies to discuss the sensitive analysis of y and the mean accepted tokens of PEARL. We provide more experiment results in Appendix D.
4.4.1 SESSITIVE ANALYSIS OF THE WINDOW SIZE Y
Intuitively, the optimal value of the window size y should be the speed ratio between the draft model and the target model, where the draft model and the target model can achieve best parallelism and fully alleviate the mutual waiting problem. However, integer. Therefore, we propose to select the round integer of as the window size 7, We conduct some case studies see the effect of different y values in different model configurations and differ- ent tasks. As shown in Table 5 (we mark the round integer below each model configuration). directly choosing the round integer of e as the window size can achieve the maximal inference accelcration, which is robust to the model configurations. Meanwhile, as shown in Table 6 (the round integer of c of Llama 70B is 5), setting the window size as can achieve the maximal inference acceleration as well, which is robust to the tasks. These results suggest great convenience of PEARL framework which alleviates the burden of tuning y for different model configurations and different tasks.
often the case that e may be an
Table 5: Optimal y of different model combinations on HumanEval. (unit: tok/sec)
Table 6: Optimal y for different tasks of Llama 2 70B. (c=5)
CodeLlama 7&34
CodeLlama 7&70
DeepSeek 6,7833
HumanEyal
GSM8K
MT-Bench
(cm3)
(cm3)
3
20.39
18.23
17.67
2 33.25
16.28
30.82
4
24.58
21.81
20.69
3 46.06
23.14
48.
4 42
29.65
47.22
5
30.34
26.47
24.25
5 44.93
40.72
46.91
6
28.02
24.59
22.71
6
41,83
35.39
44.36
7
28.09
24.23
22.54
4.4.2 MEAN ACCEPTED TOKENS
In Section 3.4, we claim that PEARL can achieve adaptive draft length for acceleration. To further illustrate the real mean accepted tokens in PEARL under real- world complex conditions, we conduct experiments on the Eval,Eval, GSM8K, and MT-Bench datasets. As shown in Table 7, we still empirically observe that that PEARL obtains more accepted tokens compared to vanilla SD meth- ods, which further demonstrates the effectiveness of the PEARL framework. Specifically, PEARL achieves the max number mean accepted tokens to 39.9, which significantly outperforms vanilla SD methods by large margin. Note that the mean accepted tokens (MAT) and the speed ratio e be- tween the draft model and the target model both influence the final speed-up results. For example, in the case of Deepseek 6.7B3BB. the draft model runs approximately three times faster than the target model. Even if the MAT approaches infinity, where all tokens are generated by the 6.7B model, the theoretical maximum speed-up would be capped 3x. Consequently, with MAT of 39.9, PEARI.
8

---

Published as a conference paper at ICLR 2025
achieves x speed-up, which is close to this theoretical optimum. These results demonstrate that our PEARL can fully exploit the inference ability of the draft model for further acceleration.
Table 7: Comparison of mean accepted tokens of vanilla SD methods and PEARL.
Methods
Code 7B&34B
CodeL.lmaa 7B&70B
DeepSeek 1.3B&33B
DeepSeek 6,7B&33B
Speculative Decoding
5.27
8.32
7.23
5.69
Ours
27.95
26.53
29.65
39.90
4.5
MORE DISCUSSIONS ON PEARL
Clarification of the application scenarios. First, we would like to clarify the application scenar- ios of our PEARL. The key idea of speculative decoding is to exploit the computation redundancy for acceleration Leviathan et al. (2023). Based on this idea, we observe the mutual waiting problem, which hinders speculative decoding to fully utilize the redundant computational resources. There- fore, the main application scenarios of PEARL focus on the scenarios with adequate computational resources, where speculative decoding cannot sufficiently use these resources.
PEARL in resqurce-adequate scenarios. In such scenarios, the draft model and the target model can be deployed separately. Simultaneously running the draft model and the target model would not bring additional latency in either both drafting or verification stages. Our main experiments along with all baseline methods are conducted under these scenarios, where we deploy the draft model and the target model on different devices. Besides, it is feasible to integrate tensor parallelism (TP) with PEARL in these situations. We provide the solution in Appendix F.
PEARL in resource-constraine scenarios. However, we acknowledge that in many situations, the GPU resources are limited, and the draft model and the target model are deployed on the same devices. We refer to this as a "co-locate" setting or resource competitions (RC). The key problem lies in the nature of GPU hardware design- two running processes on the same GPU will compete for GPU resources, which may lead to slowdowns. We provide a solution to address this issue.
Generally, in real-world LLM applications, the large-scale target model is usually placed with more than 1 GPU to handle more requests and long context inference, while the small-scale draft model only needs GPU for inference. In this case, we can apply pipeline parallelism (PP) to serve the target model with multiple GPUs. Inspired by this observation, we propose improved version of PEARL to effectively utilize GPU computation resources with PP without resource competitions. The key idea is to transfer the computation of the draft model another GPU when the target model is running on a specific GPU. Specifically, we transfer the first [ draft token generation to the last device, while the last [ draft tokens are generated with the first device. As the computation of the target model is conducted sequentially with multiple GPUs, this could effectively utilize the GPU resources to avoid RC. We conduct some experiments in Table 8 and find that this strategy allows PEARL to retain 89% ~ 99% of its original performance, demonstrating the effectiveness of our PEARL in such conditions. We provide detailed implementation and additional experiment results of this strategy in Appendix G.
Table 8: Comparisons of Llama models for PEARL on MT-bench with and without RC scenario.
Models
Writing
Roleplay
Reasoning
Math
Coding
Extraction
Stem
Humanities
Avg.
Llama 7k2:00
22.10
22.47
26.16
25.74
24.63
26.11 11
23.84
23.09
24.28
Llama 70h (RC)
19.88
20.24
25.06
24.47
23.47
25.79
21.55
22.03
22,83
performunce retain
89.95%
90.08%
80%
95.07%
95.29%
98.77%
90.39%
95.41%
94.03%
Llama 3.1 8h&
31.23
30.08
35.09
36.59
31.95
34.60
30.06
27.51
32.14
Llama 3.1 7hA (RC)
29.65
27.54
35.01
36.31
29.85
33.99
26.77
26.10
30.78
performance retain
94.94%
91.56%
99.77%
99.23%
93.43%
98.24%
89.06%
94.87%
95.77%
5 RELATED WORK
Transformer inference acceleration. Inference acceleration field that has been extensively studied over a long period of time (Liu al., 2024). There exists extensive works for transformer

---

Published as a conference paper at ICLR 2025
inference acceleration with the rising of LLMs (Xia et al., 2024; Lv et al., 2024; Chen al., 2024). This includes efforts of model compression (Zhu et al., 2023), efficient architecture design (Chitty- Venkata & Somani, 2022), and hardware optimization and implementation (Dao et al., 2022). Model compression methods such as quantization (Choi et al., 2018), knowledge distillation (Hinton et al., 2015), and structure pruning (Han al., 2015) aim reducing the number of computational op- erations. Efficient architecture design is proposed to develop lightweight transformer architectures. Hardware optimization and implementation
proposed for efficient execution to fully exploit the hardware devices. These methods have achieved great success, while they are orthogonal to specu- lative decoding algorithms, which can be integrated for further speedup.
trrrr-y framework. While SD exhibits great acceleration effectiveness and lossless gen- eralization quality, it remains challenge to find compact draft model with high distribution align- ment. Some works focus on removing the necessity of the draft model. Self-speculative decoding (Zhang et al., 2023) proposes to skip some intermediate layers of the target model for drafting- Medusa (Cai et al., 2024) adds extra decoding heads at the top of the target model to generate drafts. Lookahead decoding et al., 2024) caches the generation trajectory (n-grams) as the drafts. Eagle (Li et al., 2024) employs an additional transformer decoder layer generate drafts at the feature level. Glide (Du et al., 2024) reuses the kv cache from the target model to decode more accurate draft tokens. DistillSpec (Zhou et al., 2023) utilizes distillation method to identify a compact draft model. Ouroboros (Zhao et al., 2024) combines the standard SD and lookahead decoding to gener- ate more precise and longer drafts. Besides these works, SpecInfer (Miao al., 2023) proposes tree attention, which is widely used to verify more drafts and increase the acceptance rate. However, all of them do not address the parallelism issue. From this perspective, our PEARL is orthogonal to these methods and can be integrated with these methods, which is left as a future work.
6 CONCLUSION AND FUTURE WORK
Limitations and broader impact. As our PEARL is parallel acceleration framework, it remains challenge to schedule the GPU resources avoid resource competitions, which may potentially increase power consumption. We affirm our commitment to contributing positively to society, avoid- ing harm, and upholding honesty and trustworthiness. We appropriately cite the previous methods and datasets use, and ensure that all data involved is fully public, with private data being uti- lized. Furthermore, we are committed to correctly maintaining the inference acceleration techniques have developed, without incurring any form of discrimination.
Conclusion. In this paper, we propose a novel inference acceleration framework, called PEARL, which significantly improves LLM inference efficiency. PEARL consists of two simple and effective strategies, i.e., pre-verify and post-verify, which effectively alleviates the mutual waiting problem with parallelism and adaptive draft length. Extensive experiments demonstrate that our proposed PEARL outperforms existing state-of-the-art methods on various text generation benchmarks.
Future work. For future research, we aim integrate PEARL with existing accelerated inference methods to explore more efficient and resource-friendly acceleration approaches for LLM inference. Hopefully, PEARL will facilitate the future development of LLM inference acceleration.
ACKNOWLEDGEMMENT
The authors would like to thank all the anonymous reviewers for their insightful comments. This paper is supported by Shanghai Artificial Intelligence Laboratory.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:23033 08774, 2023.
Rishi Bommasani, Drew A Hudson. Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 07258, 2021.
10

---

Published as a conference paper at ICLR 2025
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint 4rXiv:2401.10774, 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptise Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013// 18, 2023.
Hanzhu Chen, Xu Shen, Qitan Lv, Jie Wang. Xiaoqi Ni, and Jieping Ye. Sac-kg: Exploiting large language models as skilled automatic constructors for domain knowledge graph. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4345-4360, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Krishna Teja Chitty-Venkatee and Arun K Somani. Neural architecture search survey: A hardware perspective, ACM Computing Surveys, 55(4):1-36, 2022.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 1805.06888 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1X.v:2110.141666 2021.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35:16344-16359, 2022.
DeepSeek-AI. Deepseek-v3 v3 technical report, 2024. URL /tL.ttttss//aawwwoorr////////// 19437.
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, al. Glide with cape: A low-hassle method to accelerate speculative decoding. arXiv preprint 2xXi:22402.02082. 2024.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.21700 2024.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Il infer- using lookahead decoding. arXiv preprint Sttt::2402.02055., 2024.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi. Y Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming- rise code intelligence. arXiv preprint arXiv:2401. 14196. 2024.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00145 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.02531, 2015.
Joao Gante. Assisted generation: new direction toward low-latency text generation, 2023. URL
co co/coooob//ooooe hb/llg////////ooooooootoooootttt assisted- weweedd-geeetiiioooo
Diksha Khurana, Aditya Koli, Kiran Khatter, and Sukhdev Singh. Natural language processing: State the art, current trends and challenges. Multimedia tools and applications, 82(3):3713- 3744, 2023.
11

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty, arXiv preprint arXiv:2401. 2024.
Tianyu Liu, Qitan Lv, Jie Wang, Shuling Yang, and Hanzhu Chen. Learning rule-induced subgraph representations for inductive relation prediction. Advances in Neural Information Processing Systems, 36, 2024.
Qitan Lv. Jie Wang, Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. Coarse-to-fine high- lighting: Reducing knowledge hallucination in large language models, In Forty-first International Conference on Machine Learning, 2024.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative I1m serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.0978/ 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multi- lingual chain-of tf-thugghh reasoners, 2022. URL http://arxiv. orglabs/2210.03057.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: comprehensive survey of speculative decoding. arXiv preprint arXiv:2401 1.005111111... 2024.
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint 0rXiv:2300.08818, 2023.
Weilin Zhao, Yuxiang Huang, Xu Han, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv preprint a3X1v:240..13720, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.0967 2024.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, dran-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461, 2023.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey model compression for large language models. arXiv preprint arXiv:2308. 2023.
12

---

Published as a conference paper at ICLR 2025
A ALGORITHM OF PEARL
Here, we give the whole algorithm of our PEARL in detail in Algorithm. 2.
Algorithm Parallel Speculative Decoding with Adaptive Draft Length.
Input: the draft model Mg. the target model Mp. the input prefix x, the max generate tokens L, the window size 7.
D The pre-verify strategy is used first. Initialization: mode +- "pre-verify"
2: while len(x) < do
3: if mode = "pre-verify" then
4: > Pre-verify strategy
5: for
to 7 do 6: 9: M, (x (.-....,--.I]]
7:
2; ~ di
8:
end for
9:
> running the target model parallel to verify the first draft token in advance.
p - M,(x)
ifr ~ (), Σ pla]] then
11:
[1[2]]
12:
V accept the first token
< + ..., [y]]
14:
mode + "post-verify"
15:
else
16:
reject the first token
17:
~ norm(mar(0)) p 91))
18:
- + [y]
19:
mode + "pre-verify"
20:
end if
21: else
Post-verify strategy
23:
x, [21, Ø2, F,1 + x for 7 to 2y do
b split the prefix to get the last y draft tokens
25:
> running the draft model in parallel to continue drafting.
26:
qi + M2 + T;-1))
27:
Ji di end for
28:
29:
Pr. P2. ................................ Py t- Mp(x [z1]), Mp (x + [x1, 22]) .... M2 + [I1, zny)
30:
retrival Q1, 02.0....................... from the cache
31:
(0, 1), ...,. P, ((0,1)
32:
n < min((i 111
P.la 2
J (n)
33:
if n then
accept all draft tokens
35:
< + [x1, .... ]2y]
36:
mode +- "post-verify"
37:
else
38:
X reject someone
39:
norm norm(mar Pn-1 gn+1))
40:
x + x + [21, Fn. 8)
41:
mode +- "pre-verify"
42:
end if
43:
end if
44:
end while
13

---

Published as a conference paper at ICLR 2025
B A SIMPLE STEP-BY-STEP PROFILING EXAMPLE
We provide simple step-by-step profiling of PEARL with real data prompt "xty = 47, x*y 4z'2, express x-y z" in Table
Table 9: Simple step-by-step profiling of PEARL with prompt "x+y 4z, x*y 4z'2, express x-y z" We only report the first steps for simplicity. The prompt is selected from MT-bench, and we Llama 70b as our base model pair.
steps
input prefix
current draft
target model output
judging rea- output prefix son
mode model output
0 x+y= 4z, x 4z"2, express x-y in
pre- verify
Great, I'm
great is not I, hence great is rejected, turn to pre-verify
I
pre-
'm
glad
is accepted, r' turn to post- verify
verify
you
2
I'm glad you
post- inter-
is accepted, but glad is re- jected, turn pre-verify
I'm happy
verify ested in exploring
3
I'm happy
pre- to help to verify you with this I'm happy help post- equation! help you the first 3 you with this
to is accepted, I'm happy to turn to post- verify
4
I'm happy help you with
verify However, with tokens are accepted, your but this is rejected, turn to pre-verify
5 I'm happy to help pre- you with your
question! However,
question
question is ac- cepted, turn to post-verify
I'm happy to help you with your question
verify
6 I'm happy help you with your ques- verify tion! However.
post-
notice that the equa- tion you
How- ever, I no- tice
all previous draft tokens accepted, keep post- verify
I'm happy to help you with your question! However, notice
7 I'm happy to help post- you with your ques- verify tion! However, no- tice that the equation you
provided
that the not correct.
equation is re- jected. turn to pre-verify
I'm happy to help you with your question! However, notice that the equations
we give some explanations about the whole process.
1) At step 0, the prompt "x+y 42, x*y = 4z'2, express x-y in z" input to the draft model and the target model simultancously with strategy ver-verify. Within the left of this explanation, omit this original prompt for simplicity. The draft model outputs [Great, I'm], while the target model outputs [I]. Then, will use [I] to verify the first token [Great] in the draft tokens. As it
14

---

Published as a conference paper at ICLR 2025
is not the same, we reject the draft tokens, save verification stage of the other draft tokens for acceleration, and the output prefix is [T]. As there exists a rejected token, the next strategy is still pre-verify.
2) At step 1, the prefix [T] is input, and the draft model outputs ['m glad you] and the target model outputs [']. This time, ['] is accepted by the target model, and the next strategy is tuned to post-verify.
3) At step 2, the prefix together with the other draft tokens [I'm glad you] is input. The draft model outputs [are interested in exploring] while the target model outputs [m]. i... the first draft token [m] is accepted, but the second draft token [glad] is rejected. The target model additionally appends [happy] to the prefix. As there exists a rejected token, the next strategy is pre-verify.
4) At step the prefix [I'm happy] is input, and the draft model outputs [to help you with this] and the target model outputs [to]. This time, [to] is accepted by the target model, and the next strategy is tuned to post-verify.
5) At step 4, the prefix together with the other draft tokens [I'm happy to help you with this] is input. The draft model outputs [equation! However, IJ while the target model outputs [help you with], i.e., the first three draft tokens [help you with] is accepted, but the fourth draft token [you] is rejected. The target model additionally appends [your] to the prefix. As there exists a rejected token, the next strategy is pre-verify.
6) At step 5, the prefix [I'm happy to help you with your] is input, and the draft model outputs [question! However, I] and the target model outputs [question]. This time, [question] is accepted by the target model, and the next strategy is tuned to post-verify.
7) At step 6, the prefix together with the other draft tokens [I' happy to help you with your ques- tion! However, I] input. The draft model outputs [notice that the equation you] while the target model outputs [! However, notice]. All the draft tokens are accepted, and the next strategy is still post-verify. Therefore, we save the times of the draft model forward for acceleration.
8) At step 7, the prefix together with the other draft tokens [I happy to help you with your ques- tion! However, notice that the equation you] is input. The draft model outputs [provided is not correct] while the target model outputs [that the], i.e., the first two draft tokens [that the] accepted, but the third draft token [equation] is rejected. The target model additionally appends [equations] the prefix. As there exists a rejected token, the next strategy is pre-verify.
C EVALUATION DETAILS
C.I DATASET CONFIGURATIONS
In our experiments, we evaluate the effectiveness of our PEARL on 4 categories of text generation tasks, including code generation, arithmetic reasoning, multilingual inference, and multi-round di- alogue. For the code generation task, employ HumanEval (Chen et al., 2021). famous code generation benchmark which is composed of 164 entries. For arithmetic reasoning and multilingual inference, we employ GSM8K and MGSM (Cobbe al., 2021; Shi et al.) the evaluation bench- mark. As the GSM8K is the English version of MGSM, we report their results in the same table. For GSM8K, we sample the first 100 entries for evaluation. For the other 10 categories in MGSM, we select 10 entries for each language. For multi-round dialogue, we employ MT-bench(Zheng et al., 2024) as the benchmark. The maximum generation lengths of these tasks are respectively set to 1024, 256, 256, and 256.
C.2 MODEL CONFIGURATIONS
We select some representative models for evaluation, including Llama 2 Touvron et al. (2023), Codellama Roziere et al. (2023), and Seepseek-Coder Guo et al. (2024). We summarize the model configuration in Table 10. In our experiments, all models are loaded in the precision of bfloat- 16. Our PEARL does not introduce any additional training, and directly uses these models to evaluate algorithm. The running speed is measured the code generation tasks.
15

---

Published as a conference paper at ICLR 2025
Table 10: Detailed model configurations.
Models
Layers
dim
FFN dim
speed (tok/s)
Codellama-77 7B
32
4096
11008
49.34
Codellama- 33BB
8192
22016
18.58
Codellama-70B
80
8192
28672
9.20
Deepseek-1.33 1.3B
24
2048
5504
63.20
Deepseek-6.77
32
4096
11008
50.05
Deepseek-331
62
7168
19200
17.37
Llama-2-7B
32
4096
11008
49.94
Llama-2-700
80
8192
28672
9.22
Llama-3.1-88
32
4096
14336
44.37
-Jama-331-700
80
8192
28672
9.00
C.3 EVALUATION DETAILS
All of our experiments including latency measurement, ablation studies, and case studies are con- ducted NVIDIA A A100-SXM4-80G GPUs. For models with sizes of 1.3B and 7B, we put them on single A100, while 34B models are deployed on 2 A100, and 70B models are deployed on A100. For inference, we use batch size which commonly used in other speculative decoding works. For the compared baselines, including Lookahead decoding and Ouroboros, we reproduce the results of them on the code generation tasks with the default parameters as described in their paper or code. When evaluating these methods, the model configuration and GPU usage are the same as our PEARL.
Table 11: The number of model runs of the draft model and the target model with different model configurations on HumanEyal
Draft Model (SD)
Target Model (SD)
Draft Model (PEARL)
Target Model (PEARL)
Deepseek 1.3B&333
140500
35125
181864 (1.29x)
45466 (1.29x)
Deepseek 6.7B&&33B
128973
42991
174855 (1.35x)
58285 (1.36x)
Codellama 7B&
132054
44018
181020 (1.37x)
60340 (1.37x)
Codellama 7BB
151960
30392
198370 (1.30x)
39674 (1.30x)
I.ama22 7B&70B
175460
35092
248720 (1,41x)
49744 (1.42x)
As our PEARL is parallel inference acceleration framework, implement the parallel algorithm accelerate, which can be further optimized with other parallel techniques. We leave this as potential future work acquire acceleration.
D MORE EXPERIMENT RESULTS
D.1 EVALUATION RESULTS OF LLAMA 3.1 ON MT-BENCH AND MGSM
As illustrated in Section 4.2, provide evaluation results of PEARL in Table 12 and 13 with both Llama 2 7& 70B and Llama 3.1 8&70B. Notably, Llama 3.1 is a more advanced LLM series which requires the transformers version be greater than 4.43.0. Therefore, we cannot reproduce the results of baseline Ouroboros and Lookahead Decoding.
Table 12: Multi-language experiment results using Ilama 3.1 8B& on GSM8K and MGSM
(Cobbe et al., 2021; Shi et al.). We bold the best results for each language.
Method
English (GSMBK)
Bengali
German
Spanish
French
Japanese
Russian
Swahil
Tegain
Thai
Chinese
Avg.
Auto Regressive
1.0x
1.0x
100x
1.00x
1.00x
1.00x
L.0k
1.00x
1.00x
Specularive Desnding
2.69%
2.77x
2/64x
2.72x
2.1%%
2.78
16

---

Published as a conference paper at ICLR 2025
Table 13: Experiment results using Llama2 7B&70B and Llama 3.1 8B& MT-bench (Zheng et al., 2024). We bold the best results for each subtask.
Model Configuration
Method
Writing
Roleplay
Reasoning
Coding
Extraction
Humanities
Avg.
Auto Regresaive
1.00x
1.00x
L00x
1.0%x
1.00%
1,00x
1.00x
1.00x
1.00x
Llama 2 7B&70B
Speculative Devoding
1.70x
1.73x
196x
2,00%
1.93x
2.14x
1887
1.81
1,89%
Ours
2
2
288x
2,79x
2.67x
2.92
2.58x
2.50
2.64%
Regressive
100x
1.00x
1.00x
1.0x
Laamn 3.1 888708
Specularive Decoding
2.29
2.24%
2,60
281x
2.35%
2.64
2.22%
2.12x
2.42x
Ours
3.49
3.38
3.9x
4,6x
3.55x
3,98x
3.34x
3,08%
3.59
D.2 OPTIMAL, OF SPECULATIVE DECODING
In recent speculative decoding papers, the compared results of vanilla speculative decoding are commonly based a fixed window size y 5. We find that vanilla speculative decoding can achieve better results with appropriate 7 Therefore, all the results of speculative decoding in our paper are based on their optimal 7,. We present some searching results of for speculative decoding Table 14.
Table 14: Optimal y values of speculative decoding for each model pair. (Unit: tokens / second)
codellama 7&34B
codellama 7&70B
deepseek 1.3833B
deepseek 6.7833B
30.95 (y 4)
25.92 (y 8)
37.22 (7 = 6)
30.92 (7 4)
31.85 (y 5)
26.02 (y 9)
38.53 (? 7)
31.74 (7 5)
33.57 (y 6)
27.60 (7 10)
39.52 (ty 8)
33.77 (y 6)
33.52 (7 = 7)
27.23 (γ 11)
39.38 (ry 9)
33.55 (7 7)
32.79 (7 = 8)
26.65 (7 = 12)
38.69 (7 = 10)
32.97 (7 = 8)
D.3 TIME CONSUMPTION OF EACH COMPONENT IN ONE PEARL STEP
To investigate influence and potential additional latency of the parallel inference, we measure the time cost of each component in one PEARL step with different sizes of model pairs in Table 15. From these results, we can find that the communication cost negligible. The draft time and the target time are very close, indicating that PEARL effectively addresses the mutual waiting problem.
Table 15: The time of each component in PEARL step. The experiments conducted Hvma
llama 2 78:0BB
codellama 7&:4B
deepseek 1.3833B
communication
0.2 ms
0.3 ms
0.2 ms
verify
1.7 ms
1.6 ms
1.7 ms
draft
105.1 ms
68.9 ms
65.1 ms
target
108.0 ms
71.1 ms
66.1 ms
E FORWARD TIMES COMPARISON OF PEARL AND SD METHODS
Considering that PEARL is a parallel framework, both the draft model and the target model are running simultancously at all times. Therefore, measure the number of model runs for PEARL compared to the traditional SD method to provide a more comprehensive perspective in Table 11. The results show that our PEARL exhibits relatively more forward times of both the draft model and the target model compared to traditional SD. As our PEARL a parallel inference framework, which executes the draft model and the target model in parallel any timestamp, it naturally increases the
17

---

Published as a conference paper at ICLR 2025
forward times of the target model and leads to more power consumption. However, the additional inference time occurs at another process, which will not affect the multi-user throughput.
F INTEGRATING TP WITH PEARL
In resource-adequate scenarios, it possible to integrate tensor parallelism (TP) with PEARL. The key to integrating TP is to deploy the draft model and the target model on separate devices. The most direct way is to deploy the small-scale draft model on GPU and the large-scale target model on the rest GPUs. Take the example of 8 GPUs, we can place the draft model GPU 0, while the target model is set on GPUs 1-7. In this way, the draft model and the target model can conduct parallel inference and achieve the best inference speedup. Meanwhile, it is possible deploy the draft model on CPU / edge computing. The separation idea is similar to PD dis- dis-aggregation (Zhong al., 2024), which dis-aggregates the prefilling and decoding process on different devices to fully exploit the computation resources. PEARL shares the same idea to disaggregate the decoding process of the draft model and the target model on different devices.
However, we notice that in the modern TP framework, layer width should be divisible by TP size (which would not work with TP=7 as shown the example. We propose to use padding techniques to address the division issue directly. For example, given a weight matrix W E R4006 4096 and TP=7, we can append an extra zero matrix pad E R4096x6 to W, and form a padded weight matrix W E R. 1100 In this way, the dimension issue can be effectively addressed. For other W and TP size pairs, we can get the padded matrix similarly. We provide some explanation show this padding technique is lossless to the performance.
1. For MLP layers, padding a zero matrix W pad E Rdxx to the weight matrix does not affect the final results. Directly remove the final columns of the output can get the original output.
2. For attention layers, padding a zero matrix poa E Rdxr to the weight matrix is equivalent to padding zero matrix to Q, K, where Qpad = Q[W9; W padls K Kyad = K[ K, w peal- The attention weight is computed as QpackT pad = Q[W?; w
padl IlW w padl TT... pad are zero matrices, (WQ; W [WV"; Wo QpacKT Redd ped = QKT. Therefore, padding does not affect the attention weight matrix.
As W pad and W
W yT, i.e.,
3. For norm layers, padding zero matrix may change the scaling factor, e.g., variance in RMSNorm. When computing these scaling factors, ignoring the additional zeros can keep the original results.
Due to the complexity of implementing TP with an existing framework (vLLM (Kwon et al., 2023)), we leave this as a promising future work to integrate TP with PEARL.
G EXPERIMENT RESULTS UNDER LIMITED GPU RESOURCES
Although our PEARL parallels the draft model and the target model at the algorithmic level, still remains a challenge for deployment at the hardware level in the GPU-constrained scenarios, which we refer to as "co-locate" setting or resource competitions (RC). The key problem lies in the nature of GPU hardware design - two running processes on the same GPU will compete for GPU resources, which leads to significant slowdowns.
However, in real-world LLM applications, the large-scale target model is usually placed with more than GPU to handle more requests and long context inference, while the small-scale draft model only needs 1 GPU for inference. In this case, pipeline parallelism (PP) is the most common solution to serve the target model with multiple GPUs, which distributes the parameters to different GPUs and conducts computations sequentially with these GPUs.
Inspired by this observation, we propose an improved version of PEARL to effectively utilize GPU computation resources with PP without resource competitions. The key idea is to transfer the com- putation of the draft model to another GPU when the target model is running on a specific GPU. Specifically, we transfer the first [3] draft token generation the last device, while the last [
18

---

Published as a conference paper at ICLR 2025
draft tokens are generated with the first device. As the computation of the target model is conducted sequentially with multiple GPUs, this method could effectively utilize the GPU resources to avoid resource competition.
Take an instance c = 5, the target model is placed with = 4 GPUs, we denote the time for a target model forward and the time that the target model runs at GPU 0 is 1- To analyze the
GPU utilization in detail, we split into ge 20 steps, where each step 1/ 20 During one target model forward, the occupied GPU number in 20 steps is given by:
Mp
0, 0. 1, 1, 1, 1, 1: 2, 2, 2; 3, 3,
(2)
Then we can further analyze the occupied GPU number of the draft model with proposed methods. First, as the draft model can generate tokens 20 steps, it only needs steps to generate draft
token. Taking [ 3, [ = 2, the first 3 x 4 12 steps of the draft model will occupy the GPU 3, while the last x 4 = 8 steps of the draft model will occupy the GPU Therefore, the occupied GPU number of the draft model in 20 steps is given by:
Mg 3, 3; 3, 3, 3, 3, 3, 0, 0,
0,0,0
(3)
In this way, the draft model and the target model will occupy different devices each step, which effectively avoids resource competition. However, in real-world settings. moving the draft model from the last device to the first device non- non-trivial and costly. As a compromise, We propose to load the draft model both at the first device and the last device. During the inference process, we only move the intermediate KV Cache from the last device the first device. Many KV Cache compression methods can help further reduce the cost. As the draft model is relatively small, the KV cache itself does not incur significant memory overhead. For example, with Llama 3.1 8B, batch size=1 and input length= the size of the KV cache is 2 2 32 x x 8 1024 x 128 ~ 0.13GB. With NVlink, the theoretical time cost for transporting the KV cache is 0. 0.13/300 = 0.43ms, which is significantly lower than the computational time cost. We provide empirical results of the KV cache transport time cost in Table 16.
Table 16:
The empirical time cost of transporting KV cache with different input length. The experiments are conducted with Llama 3.1 8B HumanEval.
input length
128
256
512
1024
empirical time cost
1.3 ms
1.4 ms
1.5 ms
1.6 ms
To further evaluate the effectiveness of this method, we conduct some experiments in Table 8, 17 and 18. We found that this strategy allows PEARL to retain 89% ~ 99% of its original performance, demonstrating the effectiveness of our PEARL such conditions.
Table 17: Performance comparison of Llama 2 and Llama 3.1 models for PEARL on benchmarks with and without RC (resource competitions). Using proposed strategy in the appendix, PEARL can work well in RC settings with only a slight performance decrease (< 5%).
Humaneval
GSMBK
MTT-encch
MGSM
llama 2
32.53
30.33
24.28
30.13
llama (RC)
31.52 (0.97x)
29.05 (0.96x)
22.83 (0.94x)
28.56 (0.95x)
llama 3.1
33.56
32.97
32.14
35.02
llama 3.1 (RC)
31.83 (0.95x)
31.51 (0.96x)
30.78 (0.96x)
33.65 (0.96x)
19

---

Published as a conference paper at ICLR 2025
Table 18: Performance comparison of Llama and Llama 3.1 models for PEARL MGSM with and without RC. Using proposed strategy in the appendix, PEARL can work well in RC settings with only a slight performance decrease (< 10%).
english (GSMBK)
bengali
german
spanish
french
japanese
russian
swahili
tegulu
thai
chinese
average
Baama
1.0k
1.00x
1.00x
1,00k
1.00x
1.00x
1.00x
1.00x
1.00x
1.00K
1,00x
1.00%
(RC)
0.94x
Bama
1.00x
1.00x
1.00c
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
1.00x
lama 3.1 (RC)
0.98xc
0.94x
0.95x
0.94x
0.94x
0.96x
0.96x
0.95x
0.98x
0.98x
1.00x
0.96x
20


Reviews:
[{"review_id": "cCduRuqnCT", "reviewer": "Reviewer_htL3", "summary": "This paper aims to accelerate LLM decoding by building on top of the popular Speculative Decoding (SD) algorithm. It addresses a key bottleneck in SD, the mutual waiting problem: the draft model and target model often get stuck waiting for each other because of sequential execution of the two logic with fixed draft lengths.\n\nUnlike traditional SD, the proposed algorithm PEARL (Parallel spEculative decoding with Adaptive dRaft Length) generates variable draft lengths and supports asynchronous execution through two new operations: pre-verify and post-verify. The authors evaluated PEARL across various tasks (e.g., HumanEval, GSM8k), observing significant speedups in performance.", "questions": "* The range of 1.50x to 4.43x represents a significant gap. Is there any analysis explaining the reasons for such large differences?\n* Considering that pre-verifying a single token may not be the most accurate method for estimating difficulty, can the authors provide empirical evidence or analysis on how effectively the pre-verification of the first token compares to that of other tokens? For example, could there be potential pitfalls when using a very large gamma?\n* Could the authors illustrate a few examples of profiling PEARL on real data, similar to Figure 3, but using actual data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The motivation is clear, with strong supporting evidence, such as in Figure 2.\n* Effective visualization of the PEARL algorithm, as shown in Figure 3.\n* Reduction in manual parameter tuning for gamma, which previously required significant effort in SD (Section 4.1), even if the estimation remains somewhat approximate.\n* Extensive evaluation across various tasks, including code generation, reasoning, and multi-round dialogue.", "weaknesses": "* The window size for each chunk (gamma) appears to remain fixed, which means the draft length will still be determined by the multiplication of gamma. While this may be unavoidable in the current algorithm, the abstract and introduction suggest that the authors intend to eliminate the need for gamma entirely.\n* Pre-verify with only a single token might not be the most reliable method. See below for a question.\n* While Section 5 presents a study on various datasets, the paper does not include a detailed analysis, such as step-by-step profiling or the failure rate in pre-verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0cmkavhyyg", "reviewer": "Reviewer_WhQi", "summary": "This paper presents an interesting speculative decoding (SPD) paradigm, PEARL, that attempts to overlap the drafting and verification stages in the standard SPD framework, thereby mitigating the so-called mutual waiting issues. As a training-free SPD method, PERAL achieves a state-of-the-art acceleration ratio with different pairs of LLMs on different domains by switching between pre-verify and post-verify stages to adjust the draft length dynamically.", "questions": "I find the following concerns and would like further clarification from the authors. \n\nThe core idea behind SPD is to utilize hardware computation redundancy; therefore, running forward passes on drafter and target models simultaneously has to bring additional latencies for both the drafting and verification stages. I am glad to see the paper presents theoretical and empirical analysis, but none of them discussed this. Profiling the latency overhead brought by overlapping two stages could strengthen this paper. (I noticed Appendix E, an engineering technique to get around the resource competition issues, but still, some analysis is expected; let’s say we don’t have a multi-GPU environment to implement the PP solution.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The idea is novel to some extent: while the community has noticed that the drafting stage is the bottleneck of the current SPD system and has proposed works to either dynamically adjust the draft length or decode draft tokens in parallel, attempts to pre-verify and post-verity are innovative to lower the proportion of drafting latency in the SPD process.\n2. The illustrations are self-explanatory in Figure 3. \n3. PERAL eliminates the need to tune the drafting window size according to Section 4.1, a desirable property for other SPD frameworks.", "weaknesses": "1. Missing baselines: while comparison with SPD methods that require training (Medusa, EAGLE, etc.) is not expected, there is still a line of works that focus on training-free SPD, such as Self-Speculative [1], Parallel Decoding [2] and REST [3]. Adding these should be able to strengthen this submission, but please do not focus on this during the discussion stage.\n\nOther than these, I don’t find obvious weaknesses in this submission. Please refer to the question section regarding my main concern.\n\n[1]: Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding.\n[2]: Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via parallel decoding.\n[3]: Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D. Lee, and Di He. REST: retrieval-based speculative decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LM0RQkn8o5", "reviewer": "Reviewer_LqAV", "summary": "The paper describes and addresses the problem of \"mutual waiting\" in Speculative decoding (SD) for LLM inference acceleration, where typically, the target model and draft model are mutually blocked by each other since verification can only occur after drafting is completed and vice-versa (draft-then-verify). The authors propose a novel framework to address this mutual waiting problem called \"Parallel Speculative Decoding with Adaptive Draft Length\" (PEARL) that coordinates the drafting and verification steps partially in parallel by using “pre-verify” and “post-verify” strategies for verifying the first draft token in advance during drafting and generating more drafts during verification, respectively. This can be thought of as a shift from the “draft-then-verify” sequential paradigm into a more parallelized “draft-and-verify” paradigm. In addition to the “pre-verify” and “post-verify” strategies, the authors explore the use of an adaptive draft length to further reduce the mutual-waiting scenario where suboptimal draft lengths are used, whether too short or too long. Using these proposed strategies, the authors are able to showcase relative speedups on top of vanilla SD on a variety of text generation tasks.", "questions": "- Is additional communication overhead costs incurred when running drafting and verification on separate accelerators? (e.g. logits transport from 2 devices for rejection sampling/verification)\n- There are 5 baselines listed but Table 2 and Table 3 only report Auto-regressive and SPEED?\n- What is the assisted generation baseline exactly? Does it adjust draft length depending on the number of tokens accepted in the previous iteration?\n- What value of gamma was used for the baselines on each task? Was it fixed according to the optimal gamma values determined for PEARL?\n- Why do Table 5 and Table 6 differ for HumanEval with gamma=5 for Llama2 7B&70B (40.72 in Table 5 and 30.34 in Table 6)?\n- Is there a cap on draft length given the fixed optimal verification window size? Something like 2x? Large gamma values aren’t only detrimental to drafting phase but also incur additional computational cost in verification (verifying 4 tokens vs. 32 can be a significant difference) even for vanilla SD\n- Table 9 should clarify that it’s not inference speed time but is the number of model runs? Perhaps good to report the ratio on the side (PEARL has ?x more model runs than SD)\n- Line 034: remove “the” in “the natural language”\n- Line 046-047: rewrite to “draft tokens that the original large model (referred as the target model) then verifies in parallel…”\n- Line 161: “generating” not “generation”\n- Line 199: “that stucks the target model” ? Do you mean “blocks”?\n- Line 240 remove “have”\n- Line 331 “being” not “been”\n- Line 352: speed up ratio relative to baseline auto-regressive?\n- Line 410: Pearl without post-verify as Pearl w/o “pre-verify”\n- Line 412: “exhibits a more pronounced”\n- Line 691: “reject some” not “someone”", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Authors clearly present their ideas and propose their PEARL framework with solid working examples and motivations. In particular, the analysis into the mutually blocked asynchronous execution of drafting and verification as well as optimal draft length per decode step were helpful in understanding the potential headroom for a parallelized framework of SD.\n- Experiments with PEARL yield a solid speed up of 1.5x over vanilla speculative decoding on common text generation tasks as well as in comparison to other baselines for the HumanEval code generation task\n- The paper examined each of the component strategies of PEARL, pre-verify, post-verify, and adaptive length, independently via ablation studies and analysis to isolate and highlight the relative impacts and relationships between each strategy\n- Authors provide code implementation for reproducibility", "weaknesses": "- The main paper assumes execution scenarios where there are enough resources to run drafting and verification in parallel (e.g. multiple GPUs). However, in many instances, drafting and verification happen in co-located settings (e.g. single GPU) where it is more resource constrained. While the authors do make brief mentions in the main paper about these resource constrained scenarios, their discussion and strategies to address this are limited to a relatively short section in the appendix with not much details on their experimental results. It would be much more fitting to have this section filled with more detail and part of the main paper. For example, the strategy mentioned in the appendix involved copying the drafter model across multiple chips which incurs greater memory cost as well as potential communication cost in having to transport and sync intermediate attention KV caches.\n- Additionally, given the fact that drafting and verification is assumed to occur on separate devices in parallel, it would be good to see a mention into the communication overhead of data transfer (i.e. logits for adjusted sampling during verification rejection) as well as a breakdown into the additional computational and power consumption in practice from added drafting and verification calls that PEARL conducts in comparison to SD\n- While a variety of tasks (HumanEval, GSM8K & MGM, MT-bench) and baselines (SD, Ouroboros, Lookahead Decoding, Distillspec, Assisted Generation) were mentioned in the experiments, the variety of baselines were only used for the HumanEval code generation task while the remaining GSM8K & MGM and MT-bench tasks only used auto-regressive (AR) and SD baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKHz4rxmeH", "reviewer": "Reviewer_dgRU", "summary": "The authors aim to address two challenges: 1. The mutual waiting problem, which arises when the target model becomes idle while waiting for the draft model to generate tokens, and vice versa. The asynchronous execution of the draft and verification phases leads to inefficiencies. 2. Fixed draft model length.\n\nThe authors introduced two strategies to solve this issue: 1. Pre-verification: This strategy involves using the target model to verify the first draft token during the drafting phase. By doing this, PEARL can determine whether the drafted token will likely be accepted or rejected.\nIf the first draft token is verified and accepted, the draft model can generate additional tokens more confidently. Conversely, if the first token is likely rejected, the draft model can generate fewer tokens, thus saving computational resources and time. \n\n2. Post-verification: In this phase, the draft model generates additional draft tokens while the target model verifies the previously drafted tokens. This allows for a more continuous flow of token generation and verification. By enabling the draft model to produce more tokens during the verification phase, PEARL capitalizes on situations where the target model is actively processing the earlier drafts. This strategy ensures that the draft model is not idle while waiting for the target model to complete its verification, thus maximizing throughput.", "questions": "My concerns are raised above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and provides several contributions: \n\n1. PEARL allows the drafting and verification phases to occur simultaneously\n\n2. Instead of using a fixed draft length, PEARL adapts the number of draft tokens generated based on the context and complexity of the task. This flexibility ensures that the draft model generates an appropriate number of tokens, reducing unnecessary computations and improving the acceptance rate of tokens by the target model. This adaptability helps to optimize the inference process for different scenarios.\n\n3. PEARL theoretically demonstrates that it can achieve a higher mean number of accepted tokens compared to existing draft-then-verify methods. This means that more of the generated tokens are useful, leading to better performance overall", "weaknesses": "The author claims that they can continue doing draft generation while doing the verification. This raises many questions:\n\n1. The latency of verifying a 70B model is about generating 2.5 tokens on the 8B model. Based on the speedup that the authors provided, this parallel approach would not work beyond a lookahead length of 3, which is shown to be suboptimal empirically. Therefore it is not clear how much this post-verify step improves the performance. \n\n2. Some parameters are underspecified in the tokens accepted per second table (Table). It is not clear under what lookahead length are the baseline numbers achieved, or if they are optimal.\n\n3. The authors picked pipeline parallelism for their implementation. However, while this is a convenient setup for solving the \"resource contention\" challenge, this is an unreasonable setting and introduces much higher latency in the first place. In deploying a 70B target model with a 7B draft model, using tensor parallelism (TP) can reduce the latency of the target model by leveraging more parallelism in each layer. Therefore, this casts doubt on all the speedup that the authors reported as this causes both the baseline and their reported results to be slower than in a TP setup. Also, with TP, the proposed solution to resource contention in Appendix E would not apply and it is not clear whether the authors can show a similar speedup with their algorithm.\n\n4. Furthermore, using PP instead of TP causes a higher activation memory footprint, reducing the effective batchsize the model can accommodate during decoding, effectively reducing the overall throughput.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)